Using LangServe to build REST APIs for LangChain Applications
Introduction
LangChain is a powerful framework for building applications with AI language models. It simplifies the process of interfacing with local or remote LLMs by making it easy to template prompts, configure query contexts, and chain discrete processes together to form complex pipelines.
LangServe is a LangChain project that helps you build and deliver these applications over a REST API. Under the hood, it uses FastAPI to construct routes and build web services, and leverages Pydantic to handle data validation.
In this guide, we will demonstrate how to build an application with LangChain and LangServe and deploy it to Koyeb. The application will serve a REST API where users can submit queries. It will pass these, with contextual information, to OpenAI to generate responses.
You can deploy and preview the example application from this guide by clicking the Deploy to Koyeb button below:
Be sure to set the OPENAI_API_KEY environment variable during configuration. You can consult the repository on GitHub to find out more about the example application that this guide uses.
Requirements
To successfully follow and complete this guide, you need:
- Python installed on your local computer.
- A GitHub account to host your LangServe application.
- A Koyeb account to deploy and run the preview environments for each pull request.
- An OpenAI API key so that our application can send queries to OpenAI.
Steps
To complete this guide and deploy a LangServe application, you'll need to follow these steps:
- Set up the project directory
- Create the app directory structure and install dependencies
- Create the LangServe application
- Test the application
- Adjust the Dockerfile
- Publish the repository to GitHub
- Deploy to Koyeb
Set up the project directory
To get started, create and then move into a project directory that will hold the application and assets we will be creating:
mkdir example-langserve
cd example-langserve
Inside, create a file called .env in your text editor. Inside, define the OPENAI_API_KEY environment variable by setting it to your OpenAI API key:
# .env
OPENAI_API_KEY="<YOUR_OPENAI_API_KEY>"
Our application will read the API key from this file to authenticate its requests to OpenAI services.
Next, create and activate a new Python virtual environment for the project. This will isolate our project's dependencies from system packages to avoid conflicts and offer better reproducability:
python -m venv venv
source venv/bin/activate
Your virtual environment should now be activated.
Create the app directory structure and install dependencies
Now that we are working within a virtual environment, we can begin to install the packages our application will use and set up the project directory.
Standard Python installations include the pip package manager by default. However, the LangServe project uses the poetry by default. Because of this, we'll install our dependencies in two stages.
First, install the langchain-cli package to get access to the langchain command line tool. We'll also take this opportunity to install poetry itself and make sure pip is up-to-date:
pip install -U pip langchain-cli poetry
Next, with the newly installed langchain command, initialize a LangChain project in the current directory:
langchain app new .
Note: Be sure to include the trailing dot to target the current directory.
You will be asked whether you wish to install any packages. Despite the wording, this prompt actually refers to LangChain templates and not Python packages. Press ENTER to continue without adding any templates.
With the new project files, your directory should now look similar to this:
.
├── app/
│ ├── __init__.py
│ ├── __pycache__/
│ │ └── . . .
│ └── server.py
├── Dockerfile
├── packages/
│ └── README.md
├── pyproject.toml
├── README.md
└── venv/
└── . . .
The pyproject.toml file is the primary file that the langchain command and poetry both use to record dependency information and configure project metadata. Because this now makes the directory a valid poetry project, we can use poetry to install the remaining dependencies:
langserve[all]: The server and client components of the LangServe library.langchain-openai: The package containing OpenAI integration for LangChain.python-decouple: A package that can be used to read environment variables and.envfiles.
poetry add "langserve[all]" langchain-openai python-decouple
Our project directory now has all of the dependencies and project files necessary for us to begin working.
Create the LangServe application
To create a basic LangServe application, open the app/server.py file in your text editor. Inside, replace the existing contents with the following code:
# app/server.py
from decouple import config
from fastapi import FastAPI
from langchain_openai import ChatOpenAI
from langchain.prompts import ChatPromptTemplate
from langserve import add_routes
app = FastAPI()
model = ChatOpenAI(openai_api_key=config("OPENAI_API_KEY"))
prompt = ChatPromptTemplate.from_template("Give me a summary about {topic} in a paragraph or less.")
chain = prompt | model
add_routes(app, chain, path="/openai")
if __name__ == "__main__":
import uvicorn
uvicorn.run(app, host="0.0.0.0", port=8000)
Let's take a moment to go over what this application does.
The code begins by importing all of the necessary classes, functions, and other materials from the packages we've installed. Afterwards, it initializes a FastAPI() instance that will serve as the primary application object for the application.
Next, we initialize instances of the ChatOpenAI and ChatPromptTemplate classes and assign them to the model and prompt variables, respectively. For the ChatOpenAI instance, we pass in the OpenAI API key from our .env file using the config object from python-decouple. For the ChatPromptTemplate, we set the prompt to ask for a summary of the given topic. We then chain these two together in a chain variable.
We add a route to serve the new chain at /openai. Afterwards, we use uvicorn to serve the applcition on all interfaces using port 8000.
Test the application
We can test that the application works as expected by typing the following inside the main project directory:
langchain serve
This will start up the application server. Navigate to 127.0.0.1:8000/openai/playground in your web browser to view the prompt page. You can test that everything is working correctly by entering a question or topic.
When you are finished, press CTRL-C to stop the server.
Adjust the Dockerfile
When we initialized a new LangChain project, it copied a Dockerfile to our project directory. Koyeb will build a Docker image from this file when we deploy later on. The generated Dockerfile works as expected, but we will adjust it slightly to allow Koyeb to pass in the port to run on during the build process.
Open the Dockerfile in your text editor and replace the final two lines (EXPOSE and CMD) with the following:
# Dockerfile
. . .
EXPOSE 8000 // [!code --]
CMD exec uvicorn app.server:app --host 0.0.0.0 --port 8000 // [!code --]
ARG PORT // [!code ++]
EXPOSE ${PORT:-8000} // [!code ++]
CMD exec uvicorn app.server:app --host 0.0.0.0 --port ${PORT:-8000} // [!code ++]
The ARG line marks PORT as a build argument. Afterwards, we modify the hardcoded instances of port 8000 with a reference to that build argument, retaining 8000 as the fallback value if no build value is provided.
Publish the repository to GitHub
The application is almost ready to deploy. We just need to commit the changes to Git and push the repository to GitHub.
In the project directory, initialize a new Git repository by running the following command:
git init
The project creation process includes a very basic .gitignore file. Replace it with a more complete one for Python projects by typing:
curl -L https://raw.githubusercontent.com/github/gitignore/main/Python.gitignore -o .gitignore
Next, add the project files to the staging area and commit them. If you don't have an existing GitHub repository to push the code to, create a new one and run the following commands to commit and push changes to your GitHub repository:
git add :/
git commit -m "Initial commit"
git remote add origin git@github.com:<YOUR_GITHUB_USERNAME>/<YOUR_REPOSITORY_NAME>.git
git branch -M main
git push -u origin main
Note: Make sure to replace <YOUR_GITHUB_USERNAME>/<YOUR_REPOSITORY_NAME> with your GitHub username and repository name.
Deploy to Koyeb
Once the repository is pushed to GitHub, you can deploy the LangServe application to Koyeb. Any changes in the deployed branch of your codebase will automatically trigger a redeploy on Koyeb, ensuring that your application is always up-to-date.
To get started, open the Koyeb control panel and complete the following steps:
- In the Overview tab, click Create Web Service.
- Select GitHub as the deployment option.
- Choose the repository containing your application code. Alternatively, you can enter our public LangServe example repository into the Public GitHub repository field at the bottom of the page:
https://github.com/koyeb/example-langserve. - In the Builder section, choose Dockerfile.
- In the Environment variables section, click Add Variable to configure a new environment variable. Create a variable called
OPENAI_API_KEY. Select the Secret type and choose Create secret in the value. In the form that appears, create a new secret containing your OpenAI API key. - Name the App and Service, for example
example-langserve. - Click the Deploy button.
Koyeb will clone the GitHub repository and use the Dockerfile file to build a new container image for the project. Once the build is complete, a container will be started from the image to run your application.
Once the deployment is healthy, visit your Koyeb Service's subdomain (you can find this on your Service's detail page) with /openai/playground appended to the end. It will have the following format:
https://<YOUR_APP_NAME>-<KOYEB_ORG_NAME>.koyeb.app/openai/playground
You should see your LangServe application's prompt, allowing you to ask questions and get responses from the OpenAI API.
Conclusion
In this tutorial, we demonstrated how to use LangChain, LangServe, and the OpenAI API to build a basic page with an AI prompt. LangChain provides an ergonomic interface for working with OpenAI and other models, while LangServe makes it easy to create and serve sites exposing these tools.
This tutorial is only a basic guide to get you started with these projects. To learn more about how to build robust services using more advanced features of these frameworks, take a look at the LangChain documentation and try out some of their project templates.
