
## Introduction

Real-time automated transcription is incredibly useful for anyone who needs to capture spoken content quickly and accurately. Whether you're creating subtitles for videos, transcribing podcast episodes, or documenting meeting notes, having an automated system can save you a lot of time and effort.

In practical terms, automated transcription can be used in various real-world scenarios. For instance, journalists can transcribe interviews on the fly, educators can provide real-time captions for their lectures, and businesses can document conference calls and meetings more efficiently.

[OpenAI Whisper](https://github.com/openai/whisper) is an open-source solution built for this purpose. It uses state-of-the-art machine learning algorithms to transcribe speech with high accuracy, even handling different accents and speaking speeds.

In this tutorial, you will learn how to set up a [Streamlit](https://streamlit.io/) application, integrate OpenAI Whisper for real-time podcast transcription, and deploy the application using Docker and Koyeb, creating a scalable transcription service.

You can consult the [project repository](https://github.com/koyeb/example-whisper-transcription) as work through this guide. You can deploy the podcast transcription application as built in this tutorial using the [Deploy to Koyeb](https://www.koyeb.com/docs/build-and-deploy/deploy-to-koyeb-button) button below:

[![Deploy to Koyeb](https://www.koyeb.com/static/images/deploy/button.svg)](https://app.koyeb.com/deploy?name=whisper-transcription&type=git&repository=koyeb%2Fexample-whisper-transcription&branch=main&builder=dockerfile&instance_type=gpu-nvidia-rtx-4000-sff-ada&env%5B%5D=&ports=8000%3Bhttp%3B%2F)

## Requirements

To successfully follow this tutorial, you will need the following:

- [Git](https://git-scm.com/) installed
- [FFmpeg](https://ffmpeg.org/) installed
- [Python 3.6+](https://www.python.org/downloads/release/python-360/) or later
- A [Koyeb](https://app.koyeb.com/) account

## Demo

Before we jump into the technical details, let me give you a sneak peek of what you will be building in this tutorial:

<iframe
  width="560"
  height="315"
  src="https://www.youtube.com/embed/HlVhwUriUnE"
  title="Using OpenAI Whisper to Transcribe Podcasts"
  frameborder="0"
  allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
  referrerpolicy="strict-origin-when-cross-origin"
  allowfullscreen
></iframe>

## Understanding the components

### OpenAI Whisper

OpenAI Whisper is a sophisticated speech-to-text (STT) model designed to transcribe spoken words into written text with high accuracy. Utilizing advanced machine learning algorithms, Whisper is capable of recognizing various accents, dialects, and speaking speeds. It can be integrated into voice assistants, dictation software, and real-time translation services to convert spoken language into text efficiently.

OpenAI Whisper can be used in sectors such as healthcare for medical dictation, in customer service for automated call transcriptions, and in media for generating subtitles for videos and podcasts. Its ability to handle complex speech patterns and languages makes it the go-to service in any application requiring high-quality speech-to-text conversion.

### Streamlit

Streamlit is an open-source Python library designed to create interactive data applications, often referred to as dashboards. It empowers developers to build and share data apps simply and intuitively, eliminating the need for extensive web development expertise.

Streamlit apps are created as Python scripts, which are then executed within the Streamlit environment. The library offers a set of functions that can be used to add interactive elements to the app such as upload file button.

## Steps

To build the transcription service, we'll complete the following steps:

1. [Set up the environment](#set-up-the-environment): Start by setting up your project directory, installing necessary dependencies, and configuring environment variables.
2. [Set up Streamlit](#set-up-streamlit): Next, install Streamlit and create the initial user interface for your application.
3. [Transcribe podcasts with OpenAI Whisper](#transcribe-podcasts-with-open-ai-whisper): Use OpenAI Whisper to transcribe podcasts into text with timestamps.
4. [Dockerize the Streamlit application](#dockerize-the-streamlit-application): Create a Dockerfile to containerize your application for consistent deployment.
5. [Deploy to Koyeb](#deploy-to-koyeb): Finally, deploy your application on the Koyeb platform.

## Set up the environment

Let's start by creating a new Streamlit project. To keep your Python dependencies organized you should create a virtual environment.

First, create and navigate into a local directory:

```bash copy
# Create and move to the new directory
mkdir example-whisper-koyeb-gpu
cd example-whisper-koyeb-gpu
```

Afterwards, create and activate a new virtual environment:

```bash copy
# Create a virtual environment
python -m venv venv

# Active the virtual environment (Windows)
.\venv\Scripts\activate.bat

# Active the virtual environment (Linux)
source ./venv/bin/activate
```

Now, you can install the required dependencies. Open a `requirements.txt` file in your project directory with the following contents:

```none copy
streamlit
openai-whisper
watchdog
```

Pass the file to `pip` to install the dependencies:

```bash copy
pip install -r requirements.txt
```

For the dependencies, we have included Streamlit for building the web app, OpenAI Whisper for real-time transcription, and watchdog to monitor file system events.

Don't forget to save your dependencies to a `requirements.txt` file:

```bash copy
pip freeze > requirements.txt
```

Now, let's move on to creating a new Streamlit project.

## Set up Streamlit

In this step, you will set up the Streamlit UI that will allow users to upload an audio file, click a button to start the transcribing process, and finally present the segmented transcriptions in an user-friendly manner. All of the logic for the project will reside in this file, so you can start by creating a `app.py` file with the following code:

{/* prettier-ignore-start */}

```python copy
# File: app.py

import streamlit

stream_button_styles = """
<style>
    header { display: none !important; }
</style>
"""

page_styles = """
<style>
    h1 { font-size: 2rem; font-weight: 700; }
    h2 { font-size: 1.7rem; font-weight: 600; }
    .timestamp { color: gray; font-size: 0.9rem; }
</style>
"""

page_title = "Using OpenAI Whisper to Transcribe Podcasts"

page_description = "A demo showcasing the use of OpenAI Whisper to accurately and efficiently convert spoken content from podcasts into written text."

koyeb_box = "To deploy Whisper within minutes, <a href=\"https://koyeb.com/ai\">Koyeb GPUs</a> provide the easiest and most efficient way. Koyeb offers a seamless platform for deploying AI models, leveraging high-performance GPUs to ensure fast and reliable transcriptions."

step_1 = "1. Upload Podcast"

step_2 = "2. Invoke OpenAI Whisper to transcribe podcast 👇🏻"

step_3 = "3. Transcription &nbsp; 🎉"

def unsafe_html(tag, text):
    return streamlit.markdown(f"<{tag}>{text}</{tag}>", unsafe_allow_html=True)

def main():
    # Set title for the page
    streamlit.set_page_config(page_title, layout="centered")
    # Inject hide buttons CSS
    streamlit.markdown(stream_button_styles, unsafe_allow_html=True)
    # Inject page CSS
    streamlit.markdown(page_styles, unsafe_allow_html=True)
    # Create a H1 heading on the page
    streamlit.title(page_title)
    unsafe_html("h2", page_description)
    unsafe_html("p", koyeb_box)
    audio_file = streamlit.file_uploader(step_1, type=["mp3", "mp4", "wav", "m4a"])
    if audio_file:
        # If file is received
        # Write the file on the server
        # Show next step
        unsafe_html("small", step_2)
        if streamlit.button("Transcribe"):
            # Get the transcription
            unsafe_html("small", step_3)
            # Showcase the transcription

if __name__ == "__main__":
    main()
```

{/* prettier-ignore-end */}

The code above does the following:

- Begins by importing the Streamlit module
- Defines CSS for hiding the navigation bar and styling the headings
- Defines text values for the page's title, description, and each step
- Defines the `unsafe_html` function to dynamically create the HTML tags with content
- Accepts an audio file using Streamlit's builtin `file_uploader` function

With this, you have setup a UI that is able to accept podcast audio files from the user. Now, let's move on to transcribing the audio file obtained.

## Transcribe podcasts with OpenAI Whisper

In this step, you will invoke OpenAI Whisper's base model to transcribe an audio file. By default, the model is able to return the timestamps along with the transcription. This enables you to use the generated transcriptions as subtitles as well. Make the following additions in the `app.py` file:

{/* prettier-ignore-start */}

```python copy
# File: app.py

# Existing imports
# . . .
import whisper # [!code ++]

model = whisper.load_model("base") # [!code ++]

# ...

def unsafe_html(tag, text):
    # ...

# Generate transcription of each segment
def timestamp_html(segment): # [!code ++]
    return f'<span class="timestamp">[{segment["start"]:.2f} - {segment["end"]:.2f}]</span> {segment["text"]}' # [!code ++]

# Transcribe an audio file
def transcribe_audio(audio_file): # [!code ++]
    return model.transcribe(audio_file.name) # [!code ++]

# Write the audio file on server
def write_audio(audio_file): # [!code ++]
    with open(audio_file.name, "wb") as f: # [!code ++]
        f.write(audio_file.read()) # [!code ++]

def main():
    # ...
    if audio_file:
        # If file is received
        # Write the file on the server
        write_audio(audio_file) # [!code ++]
        # Show next step
        unsafe_html("small", step_2)
        if streamlit.button("Transcribe"):
            # Get the transcription
            transcript_text = transcribe_audio(audio_file) # [!code ++]
            unsafe_html("small", step_3)
            # Showcase the transcription
            for segment in transcript_text["segments"]: # [!code ++]
                unsafe_html("div", timestamp_html(segment)) # [!code ++]

if __name__ == "__main__":
    main()
```

{/* prettier-ignore-end */}

The changes above do the following:

- Import and instantiate the OpenAI Whisper base model
- Define a `timestamp_html` function to display the transcription with **start** and **end** timestamps
- Define a `transcribe_audio` function which invokes the model to generate transcriptions of the audio file
- Define a `write_audio` function to write the audio file on the server
- If an audio file is found, it writes the file on the server
- If the **Transcribe** button is clicked in the UI, `transcribe_audio` and `timestamp_html` functions are invoked to generate and display the transcriptions of the podcast

Now, you can run the Streamlit application with:

```bash copy
streamlit run ./app.py --server.port 8000
```

The application would now be ready on `http://localhost:8000`. Test the application in action by uploading one of your favorite podcasts file and see the transcriptions generated in real-time.

Next, let's dockerize the application to ensure consistency between multiple deployments.

## Dockerize the Streamlit application

Dockerizing deployments creates a consistent and reproducible environment, ensuring that the application runs the same way on any system. It simplifies dependency management and enhances scalability, making deployments more efficient and reliable. To dockerize, create a `Dockerfile` at the root of your project with the following content:

{/* prettier-ignore-start */}

```dockerfile {16,25,27} copy
FROM python:3.9 AS builder

WORKDIR /app

RUN python3 -m venv venv
ENV VIRTUAL_ENV=/app/venv
ENV PATH="$VIRTUAL_ENV/bin:$PATH"

COPY requirements.txt .
RUN pip install -r requirements.txt

FROM python:3.9 AS runner

WORKDIR /app

RUN apt-get update && apt-get install -y ffmpeg && rm -rf /var/lib/apt/lists/*

COPY --from=builder /app/venv venv
COPY app.py app.py

ENV VIRTUAL_ENV=/app/venv
ENV PATH="$VIRTUAL_ENV/bin:$PATH"

EXPOSE 8000

CMD ["streamlit", "run", "./app.py", "--server.port", "8000"]
```

{/* prettier-ignore-end */}

Apart from the usual Dockerfile to deploy Python applications, the following tweaks and additions have been made in this code:

- `RUN apt-get update && apt-get install -y ffmpeg && rm -rf /var/lib/apt/lists/*` is used to install `ffmpeg`, and then clean up package lists to reduce image size
- `EXPOSE 8000` is used to specify the port on which the Streamlit application will run
- `CMD ["streamlit", "run", "./app.py", "--server.port", "8000"]` is used to define the command to start the Streamlit app on port 8000

With everything configured, let's move on to deploy the application to Koyeb.

## Deploy to Koyeb

Now that you have the application running locally you can also deploy it on Koyeb and make it available on the internet.

Create a [new repository on your GitHub account](https://github.com/new) so that you can push your code.

You can download a [standard `.gitignore` file](https://raw.githubusercontent.com/github/gitignore/main/Python.gitignore) for Python from GitHub to exclude certain directories and files from being pushed to the repository:

```bash copy
curl -L https://raw.githubusercontent.com/github/gitignore/main/Python.gitignore -o .gitignore
```

Run the following commands in your terminal to commit and push your code to the repository:

```bash copy
git init
git add app.py Dockerfile requirements.txt .gitignore
git commit -m "first commit"
git branch -M main
git remote add origin [Your GitHub repository URL]
git push -u origin main
```

You should now have all of your local code in your remote repository. Now it is time to deploy the application.

Within the [Koyeb control panel](https://app.koyeb.com/), while on the Overview tab, initiate the app creation and deployment process by clicking **Create Service** and then choosing **Create web service**.

1. Select **GitHub** as the deployment source.
2. Select your repository from the menu. Alternatively, deploy from the [example repository associated with this tutorial](https://github.com/koyeb/example-whisper-transcription) by entering `https://github.com/koyeb/example-whisper-transcription` in the public repository field.
3. In the **Instance** selection, select a GPU Instance.
4. In the **Builder** section, choose **Dockerfile**.
5. Finally, click the **Deploy** button.

Once the application is deployed, you can visit the Koyeb service URL (ending in `.koyeb.app`) to access the Streamlit application.

## Conclusion

In this tutorial, you built a podcast transcription application with the Streamlit framework and OpenAI Whisper. During the process, you learned how to invoke the OpenAI Whisper model in Python to generate transcription with timestamps, and how to use the Streamlit framework to quickly prototype a user interface with a functioning upload button in a few lines of code.

Given that the application was deployed using the Git deployment option, subsequent code push to the deployed branch will automatically initiate a new build for your application. Changes to your application will become live once the deployment is successful. In the event of a failed deployment, Koyeb retains the most recent operational production deployment, ensuring the uninterrupted operation of your application.


Learn how to use OpenAI Whisper to build an app to generate transcription of podcast audio files in real-time.

Using OpenAI Whisper to Transcribe Podcasts on Koyeb

Deploy your AI applications across continents and run on an edge network of 250 locations.

Low latency AI workloads

Learn how to set up a vLLM Instance to run inference workloads and host your own OpenAI-compatible API on Koyeb.

Deploy the vLLM Inference Engine to Run Large Language Models (LLM) on Koyeb

Run RAG-powered chatbots and your favorite cutting-edge AI tech across continents. No credit card required.

Deploy global AI workloads

In this tutorial, we show how to deploy an RAG chatbot built with the OpenAI embedding API, Replicate, pgvector, and the Koyeb Managed PostgreSQL service.

Build a Retrieval-Augmented Generation Chatbot using pgvector


## Introduction

[LlamaIndex](https://www.llamaindex.ai/) is a data framework that makes it simple to build production-ready applications from your data using LLMs. Specifically, LlamaIndex specializes in context augmentation, a technique of providing custom data as context for queries to generalized LLMs allowing you to inject your specific contextual information without the trouble and expense of fine-tuning a dedicated model.

In this guide, we will demonstrate how to build an application with LlamaIndex and [Streamlit](https://streamlit.io/), a Python framework for building and serving data-based applications, and deploy it to Koyeb. The application will deploy an example web app that allows users to ask questions about custom data. In our example, this custom text will be the story ["The Gift of the Magi" by O. Henry](https://en.wikipedia.org/wiki/The_Gift_of_the_Magi).

You can deploy and preview the example application from this guide with [our LlamaIndex One-Click app](https://www.koyeb.com/deploy/llamaindex) or by clicking the [Deploy to Koyeb button](/docs/build-and-deploy/deploy-to-koyeb-button) below:

[![Deploy to Koyeb](/static/images/deploy/button.svg)](https://app.koyeb.com/deploy?name=example-llamaindex&type=git&repository=koyeb%2Fexample-llamaindex&branch=main&instance_type=micro&builder=dockerfile&env%5BOPENAI_API_KEY%5D=CHANGE_ME&ports=8000%3Bhttp%3B%2F)

Be sure to set the `OPENAI_API_KEY` environment variable during configuration. You can consult the [repository on GitHub](https://github.com/koyeb/example-llamaindex) to find out more about the example application that this guide uses.

## Requirements

To successfully follow and complete this guide, you need:

- [Python 3.11](https://www.python.org/downloads/) installed on your local computer.
- A [GitHub account](https://github.com/) to host your LlamaIndex application.
- A [Koyeb account](https://app.koyeb.com) to deploy and run the preview environments for each pull request.
- An [OpenAI](https://openai.com/) API key so that our application can send queries to OpenAI.

## Steps

To complete this guide and deploy a LlamaIndex application, you'll need to follow these steps:

1. [Set up the project directory](#set-up-the-project-directory)
2. [Install project dependencies and fetch custom data](#install-project-dependencies-and-fetch-custom-data)
3. [Create the LlamaIndex application](#create-the-llama-index-application)
4. [Test the application](#test-the-application)
5. [Create a Dockerfile](#create-a-dockerfile)
6. [Deploy to Koyeb](#deploy-to-koyeb)

## Set up the project directory

To get started, create and then move into a project directory that will hold the application and assets we will be creating:

```bash
mkdir example-llamaindex
cd example-llamaindex
```

Next, create and activate a new Python virtual environment for the project. This will isolate our project's dependencies from system packages to avoid conflicts and offer better reproducability:

```bash
python -m venv venv
source venv/bin/activate
```

Your virtual environment should now be activated.

## Install project dependencies and fetch custom data

Now that we are working within a virtual environment, we can begin to install the packages our application will use and set up the project directory.

First, install the LlamaIndex and Streamlit packages so that we can use them to build the application. We can also take this opportunity to make sure that the local copy of `pip` is up-to-date:

```bash
pip install --upgrade pip llama-index streamlit
```

After installing the dependencies, record them in a `requirements.txt` file so that we can install the correct versions for this project at a later time:

```bash
pip freeze > requirements.txt
```

Next we will download the story that we will be using as context for our LLM prompts. We can download a PDF copy of "The Gift of the Magi" by O. Henry from [TSS Publishing](https://theshortstory.co.uk/about/) a platform for short fiction that hosts [free short stories](https://theshortstory.co.uk/resources/free-short-stories/).

Create a `data` directory to hold the contextual data for our application and then download a copy of the story by typing:

```bash
mkdir data
curl -L https://theshortstory.co.uk/devsitegkl/wp-content/uploads/2015/06/Short-stories-O-Henry-The-Gift-of-the-Magi.pdf -o data/gift_of_the_magi.pdf
```

You should now have a PDF file that we cat load into our application and attach as context to our LLM queries.

## Create the LlamaIndex application

We have everything in place to start writing our LlamaIndex application. Create an `app.py` file in your project directory and paste in the following content:

```python
# app.py
import os.path
import streamlit as st

from llama_index.core import (
    VectorStoreIndex,
    SimpleDirectoryReader,
    StorageContext,
    load_index_from_storage,
)


# check if storage already exists
PERSIST_DIR = "./storage"
if not os.path.exists(PERSIST_DIR):
    # load the documents and create the index
    documents = SimpleDirectoryReader("data").load_data()
    index = VectorStoreIndex.from_documents(documents)
    # store it for later
    index.storage_context.persist(persist_dir=PERSIST_DIR)
else:
    # load the existing index
    storage_context = StorageContext.from_defaults(persist_dir=PERSIST_DIR)
    index = load_index_from_storage(storage_context)

query_engine = index.as_query_engine()

# Define a simple Streamlit app
st.title('Ask Llama about "The Gift of the Magi"')
query = st.text_input("What would you like to ask? (source: data/gift_of_the_magi.pdf)", "What happens in the story?")

# If the 'Submit' button is clicked
if st.button("Submit"):
    if not query.strip():
        st.error(f"Please provide the search query.")
    else:
        try:
            response = query_engine.query(query)
            st.success(response)
        except Exception as e:
            st.error(f"An error occurred: {e}")
```

Let's go over what the application is doing.

It begins by importing the basic packages and modules it will use to create the application. This includes Streamlit (aliased as `st`) as well as functionality from LlamaIndex for loading data and indexes from directories, building vector indexes, and attaching contexts.

Next, we set up some semi-persistent storage for the index files. This logic helps us avoid creating an index from our data document every time we run the application by storing index information in a `storage` directory the first time it is evaluated. The application can load the index data from the `storage` directory directly on subsequent runs to increase performance.

Afterward the index is created or loaded, we create a query engine based on it and begin constructing the application frontend with Streamlit. We add a input field that will be translated into our query and then display the results upon submission.

## Test the application

We can test that the application works as expected on your local machine.

First, set and export the `OPENAI_API_KEY` environment variable using your OpenAI API key as the value:

```bash
export OPENAI_API_KEY="<YOUR_OPENAI_API_KEY>"
```

Next, run the application by typing:

```bash
streamlit run app.py
```

This will start the application server. Navigate to `http://127.0.0.1:8501` in your web browser to view the page prompting for your questions about "The Gift of the Magi". You can submit the default query or ask any other questions you have about the story.

When you are finished, press <kbd>CTRL</kbd>-<kbd>C</kbd> to stop the server.

## Create a Dockerfile

During the deployment process, Koyeb will build our project from a `Dockerfile`. This gives us more control over the version of Python, the build process, and the runtime environment. The next step is to create this `Dockerfile` describing how to build and run the project.

Before we begin, create a basic `.dockerignore` command. This will define files and artifacts that we don't want to copy over into the Docker image. In this case, we want to avoid copying the `venv/` and `storage/` directories since the image will install dependencies and manage cached index files at runtime:

```gitignore
# .dockerignore
storage/
venv/
```

Next, create a `Dockerfile` with the following contents:

```dockerfile
# Dockerfile
FROM python:3.11-slim

WORKDIR /app
COPY . .
RUN pip install --requirement requirements.txt && pip cache purge

ARG PORT
EXPOSE ${PORT:-8000}

CMD streamlit run --server.port ${PORT:-8000} app.py
```

The image uses the `3.11-slim` tag of the [official Python Docker image](https://hub.docker.com/_/python) as its starting point. It defines a directory called `/app` as the working directory and copies all of the project files inside. Afterwards, it installs the dependencies from the `requiremnts.txt` file.

The `PORT` environment variable is also defined as a build variable. This allows us to set it at build time to adjust the port that the image will listen on. We use this value in the `EXPOSE` instruction and again in the main `streamlit` command we run with the `CMD` instruction. Both values will use port 8000 as a fallback if `PORT` is not defined explicitly.

## Publish the repository to GitHub

The application is almost ready to deploy. We just need to commit the changes to Git and push the repository to GitHub.

In the project directory, initialize a new Git repository by running the following command:

```bash
git init
```

Next, download a generic `.gitignore` file for Python projects from GitHub:

```bash
curl -L https://raw.githubusercontent.com/github/gitignore/main/Python.gitignore -o .gitignore
```

Add the `storage/` directory to the `.gitignore` file so that Git ignores the cached vector index files:

```bash
echo "storage/" >> .gitignore
```

You can also optionally add the [Python runtime version](https://www.koyeb.com/docs/build-and-deploy/build-from-git/python#runtime) to a `runtime.txt` file if you want to try to build the repository from the Python buildpack instead of the Dockerfile:

```bash
echo "python-3.11.8" > runtime.txt
```

Next, add the project files to the staging area and commit them. If you don't have an existing GitHub repository to push the code to, [create a new one](https://www.github.com/new) and run the following commands to commit and push changes to your GitHub repository:

```bash
git add :/
git commit -m "Initial commit"
git remote add origin git@github.com:<YOUR_GITHUB_USERNAME>/<YOUR_REPOSITORY_NAME>.git
git branch -M main
git push -u origin main
```

**Note:** Make sure to replace `<YOUR_GITHUB_USERNAME>/<YOUR_REPOSITORY_NAME>` with your GitHub username and repository name.

## Deploy to Koyeb

Once the repository is pushed to GitHub, you can deploy the LlamaIndex application to Koyeb. Any changes in the deployed branch of your codebase will automatically trigger a redeploy on Koyeb, ensuring that your application is always up-to-date.

To get started, open the [Koyeb control panel](https://app.koyeb.com/) and complete the following steps:

1. On the **Overview** tab, click **Create Web Service**.
2. Select **GitHub** as the deployment method.
3. Choose the repository containing your application code. Alternatively, you can enter our public [LlamaIndex example repository](https://github.com/koyeb/example-llamaindex) into the **Public GitHub repository** at the bottom of the page: `https://github.com/koyeb/example-llamaindex`.
4. In the **Builder** section, choose **Dockerfile**.
5. Choose an **Instance** of size **micro** or larger.
6. Expand the **Environment variables** section and click **Add variable** to configure a new environment variable. Create a variable called `OPENAI_API_KEY`. Select the **Secret** type and choose **Create secret** in the value. In the form that appears, create a new secret containing your OpenAI API key.
7. Choose a name for your App and Service, for example `example-llamaindex`, and click **Deploy**.

Koyeb will clone the GitHub repository and use the `Dockerfile` file to build a new container image for the project. Once the build is complete, a container will be started from the image to run your application.

Once the deployment is healthy, visit your Koyeb Service's subdomain (you can find this on your Service's detail page). It will have the following format:

```
https://<YOUR_APP_NAME>-<KOYEB_ORG_NAME>.koyeb.app
```

You should see your LlamaIndex application's prompt, allowing you to ask questions about the story and get responses from the OpenAI API.

## Conclusion

In this guide, we discussed how to build and deploy an LLM-based web app to Koyeb using LlamaIndex and Streamlit. The application loads a story from a PDF on disk and sends this as additional context when submitting user-supplied queries. This allows you to customize the focus of the query without having to fine-tune a model for the purpose.

This tutorial demonstrates a very simple implementation of these technologies. To learn more about how LlamaIndex can help you use LLMs to answer questions about your own data, take a look at the [LlamaIndex documentation](https://docs.llamaindex.ai/en/stable/index.html).


Deploy LlamaIndex and your favorite cutting-edge AI tech across continents. No credit card required.

Learn how to use LlamaIndex, a data framework specializing in context augmentation, to quickly query general LLMs with custom data.

Use LlamaIndex, Streamlit, and OpenAI to Query Unstructured Data


## Introduction

[LangChain](https://www.langchain.com/) is a powerful framework for building applications with AI language models. It simplifies the process of interfacing with local or remote LLMs by making it easy to template prompts, configure query contexts, and chain discrete processes together to form complex pipelines.

[LangServe](https://www.langchain.com/langserve) is a LangChain project that helps you build and deliver these applications over a REST API. Under the hood, it uses [FastAPI](https://fastapi.tiangolo.com/) to construct routes and build web services, and leverages [Pydantic](https://docs.pydantic.dev/latest/) to handle data validation.

In this guide, we will demonstrate how to build an application with LangChain and LangServe and deploy it to Koyeb. The application will serve a REST API where users can submit queries. It will pass these, with contextual information, to OpenAI to generate responses.

You can deploy and preview the example application from this guide by clicking the [Deploy to Koyeb button](/docs/build-and-deploy/deploy-to-koyeb-button) below:

[![Deploy to Koyeb](/static/images/deploy/button.svg)](https://app.koyeb.com/deploy?name=example-langserve&type=git&repository=koyeb%2Fexample-langserve&branch=main&builder=dockerfile&env%5BOPENAI_API_KEY%5D=CHANGE_ME&ports=8000%3Bhttp%3B%2F)

Be sure to set the `OPENAI_API_KEY` environment variable during configuration. You can consult the [repository on GitHub](https://github.com/koyeb/example-langserve) to find out more about the example application that this guide uses.

## Requirements

To successfully follow and complete this guide, you need:

- [Python](https://www.python.org/downloads/) installed on your local computer.
- A [GitHub account](https://github.com/) to host your LangServe application.
- A [Koyeb account](https://app.koyeb.com) to deploy and run the preview environments for each pull request.
- An [OpenAI](https://openai.com/) API key so that our application can send queries to OpenAI.

## Steps

To complete this guide and deploy a LangServe application, you'll need to follow these steps:

1. [Set up the project directory](#Set-up-the-project-directory)
2. [Create the app directory structure and install dependencies](#Create-the-app-directory-structure-and-install-dependencies)
3. [Create the LangServe application](#Create-the-LangServe-application)
4. [Test the application](#Test-the-application)
5. [Adjust the Dockerfile](#Adjust-the-Dockerfile)
6. [Publish the repository to GitHub](#Publish-the-repository-to-GitHub)
7. [Deploy to Koyeb](#Deploy-to-Koyeb)

## Set up the project directory

To get started, create and then move into a project directory that will hold the application and assets we will be creating:

```bash
mkdir example-langserve
cd example-langserve
```

Inside, create a file called `.env` in your text editor. Inside, define the `OPENAI_API_KEY` environment variable by setting it to your OpenAI API key:

```ini
# .env
OPENAI_API_KEY="<YOUR_OPENAI_API_KEY>"
```

Our application will read the API key from this file to authenticate its requests to OpenAI services.

Next, create and activate a new Python virtual environment for the project. This will isolate our project's dependencies from system packages to avoid conflicts and offer better reproducability:

```bash
python -m venv venv
source venv/bin/activate
```

Your virtual environment should now be activated.

## Create the app directory structure and install dependencies

Now that we are working within a virtual environment, we can begin to install the packages our application will use and set up the project directory.

Standard Python installations include the [`pip` package manager](https://pypi.org/project/pip/) by default. However, the LangServe project uses the [`poetry`](https://python-poetry.org/) by default. Because of this, we'll install our dependencies in two stages.

First, install the `langchain-cli` package to get access to the `langchain` command line tool. We'll also take this opportunity to install `poetry` itself and make sure `pip` is up-to-date:

```bash
pip install -U pip langchain-cli poetry
```

Next, with the newly installed `langchain` command, initialize a LangChain project in the current directory:

```bash
langchain app new .
```

**Note:** Be sure to include the trailing dot to target the current directory.

You will be asked whether you wish to install any packages. Despite the wording, this prompt actually refers to [LangChain templates](https://python.langchain.com/docs/templates/) and not Python packages. Press <kbd>ENTER</kbd> to continue without adding any templates.

With the new project files, your directory should now look similar to this:

```
.
├── app/
│   ├── __init__.py
│   ├── __pycache__/
│   │   └── . . .
│   └── server.py
├── Dockerfile
├── packages/
│   └── README.md
├── pyproject.toml
├── README.md
└── venv/
    └── . . .
```

The `pyproject.toml` file is the primary file that the `langchain` command and `poetry` both use to record dependency information and configure project metadata. Because this now makes the directory a valid `poetry` project, we can use `poetry` to install the remaining dependencies:

- `langserve[all]`: The server and client components of the LangServe library.
- `langchain-openai`: The package containing OpenAI integration for LangChain.
- `python-decouple`: A package that can be used to read environment variables and `.env` files.

```bash
poetry add "langserve[all]" langchain-openai python-decouple
```

Our project directory now has all of the dependencies and project files necessary for us to begin working.

## Create the LangServe application

To create a basic LangServe application, open the `app/server.py` file in your text editor. Inside, replace the existing contents with the following code:

```python
# app/server.py
from decouple import config
from fastapi import FastAPI
from langchain_openai import ChatOpenAI
from langchain.prompts import ChatPromptTemplate
from langserve import add_routes


app = FastAPI()

model = ChatOpenAI(openai_api_key=config("OPENAI_API_KEY"))
prompt = ChatPromptTemplate.from_template("Give me a summary about {topic} in a paragraph or less.")
chain = prompt | model

add_routes(app, chain, path="/openai")

if __name__ == "__main__":
    import uvicorn

    uvicorn.run(app, host="0.0.0.0", port=8000)
```

Let's take a moment to go over what this application does.

The code begins by importing all of the necessary classes, functions, and other materials from the packages we've installed. Afterwards, it initializes a `FastAPI()` instance that will serve as the primary application object for the application.

Next, we initialize instances of the `ChatOpenAI` and `ChatPromptTemplate` classes and assign them to the `model` and `prompt` variables, respectively. For the `ChatOpenAI` instance, we pass in the OpenAI API key from our `.env` file using the `config` object from `python-decouple`. For the `ChatPromptTemplate`, we set the prompt to ask for a summary of the given topic. We then chain these two together in a `chain` variable.

We add a route to serve the new chain at `/openai`. Afterwards, we use `uvicorn` to serve the applcition on all interfaces using port 8000.

## Test the application

We can test that the application works as expected by typing the following inside the main project directory:

```bash
langchain serve
```

This will start up the application server. Navigate to `127.0.0.1:8000/openai/playground` in your web browser to view the prompt page. You can test that everything is working correctly by entering a question or topic.

When you are finished, press <kbd>CTRL</kbd>-<kbd>C</kbd> to stop the server.

## Adjust the Dockerfile

When we initialized a new LangChain project, it copied a `Dockerfile` to our project directory. Koyeb will build a Docker image from this file when we deploy later on. The generated `Dockerfile` works as expected, but we will adjust it slightly to allow Koyeb to pass in the port to run on during the build process.

Open the `Dockerfile` in your text editor and replace the final two lines (`EXPOSE` and `CMD`) with the following:

```dockerfile
# Dockerfile
. . .

EXPOSE 8000 // [!code --]
CMD exec uvicorn app.server:app --host 0.0.0.0 --port 8000 // [!code --]
ARG PORT // [!code ++]
EXPOSE ${PORT:-8000} // [!code ++]
CMD exec uvicorn app.server:app --host 0.0.0.0 --port ${PORT:-8000} // [!code ++]
```

The `ARG` line marks `PORT` as a build argument. Afterwards, we modify the hardcoded instances of port 8000 with a reference to that build argument, retaining 8000 as the fallback value if no build value is provided.

## Publish the repository to GitHub

The application is almost ready to deploy. We just need to commit the changes to Git and push the repository to GitHub.

In the project directory, initialize a new Git repository by running the following command:

```bash
git init
```

The project creation process includes a very basic `.gitignore` file. Replace it with a more complete one for Python projects by typing:

```bash
curl -L https://raw.githubusercontent.com/github/gitignore/main/Python.gitignore -o .gitignore
```

Next, add the project files to the staging area and commit them. If you don't have an existing GitHub repository to push the code to, [create a new one](https://www.github.com/new) and run the following commands to commit and push changes to your GitHub repository:

```bash
git add :/
git commit -m "Initial commit"
git remote add origin git@github.com:<YOUR_GITHUB_USERNAME>/<YOUR_REPOSITORY_NAME>.git
git branch -M main
git push -u origin main
```

**Note:** Make sure to replace `<YOUR_GITHUB_USERNAME>/<YOUR_REPOSITORY_NAME>` with your GitHub username and repository name.

## Deploy to Koyeb

Once the repository is pushed to GitHub, you can deploy the LangServe application to Koyeb. Any changes in the deployed branch of your codebase will automatically trigger a redeploy on Koyeb, ensuring that your application is always up-to-date.

To get started, open the [Koyeb control panel](https://app.koyeb.com/) and complete the following steps:

1. In the **Overview** tab, click **Create Web Service**.
2. Select **GitHub** as the deployment option.
3. Choose the repository containing your application code. Alternatively, you can enter our public [LangServe example repository](https://github.com/koyeb/example-langserve) into the **Public GitHub repository** field at the bottom of the page: `https://github.com/koyeb/example-langserve`.
4. In the **Builder** section, choose **Dockerfile**.
5. In the **Environment variables** section, click **Add Variable** to configure a new environment variable. Create a variable called `OPENAI_API_KEY`. Select the **Secret** type and choose **Create secret** in the value. In the form that appears, create a new secret containing your OpenAI API key.
6. Name the App and Service, for example `example-langserve`.
7. Click the **Deploy** button.

Koyeb will clone the GitHub repository and use the `Dockerfile` file to build a new container image for the project. Once the build is complete, a container will be started from the image to run your application.

Once the deployment is healthy, visit your Koyeb Service's subdomain (you can find this on your Service's detail page) with `/openai/playground` appended to the end. It will have the following format:

```
https://<YOUR_APP_NAME>-<KOYEB_ORG_NAME>.koyeb.app/openai/playground
```

You should see your LangServe application's prompt, allowing you to ask questions and get responses from the OpenAI API.

## Conclusion

In this tutorial, we demonstrated how to use [LangChain](https://www.langchain.com/), [LangServe](https://www.langchain.com/langserve), and the [OpenAI API](https://platform.openai.com/) to build a basic page with an AI prompt. LangChain provides an ergonomic interface for working with OpenAI and other models, while LangServe makes it easy to create and serve sites exposing these tools.

This tutorial is only a basic guide to get you started with these projects. To learn more about how to build robust services using more advanced features of these frameworks, take a look at the [LangChain documentation](https://python.langchain.com/docs/get_started/introduction) and try out some of their [project templates](https://python.langchain.com/docs/templates/).


Deploy LangServe and LangChain worldwide in seconds. No credit card required.

Accelerate AI workloads

Learn how to build and serve an API for LangChain applications with LangServe.

Using LangServe to build REST APIs for LangChain Applications

Leverage Koyeb's high-end bare metal machines for your AI workloads. No credit card required.

Run high-performance AI workloads

Learn step-by-step how to set up and utilize AutoGen within Chainlit. You'll discover how to create and interact with AI personas that are tailored to your specific needs, be it scriptwriting for YouTube content or ideating SaaS products.

OpenAI

Using OpenAI Whisper to Transcribe Podcasts on Koyeb

Deploy the vLLM Inference Engine to Run Large Language Models (LLM) on Koyeb

Build a Retrieval-Augmented Generation Chatbot using pgvector

Use LlamaIndex, Streamlit, and OpenAI to Query Unstructured Data

Using LangServe to build REST APIs for LangChain Applications

Use AutoGen, Chainlit, and OpenAI to Generate Dynamic AI Personas