Deploy a Python Celery Worker

Introduction

Celery is an open-source distributed task queue that focuses on real-time processing and task scheduling. It coordinates tasks through a backing message broker, typically Redis or RabbitMQ, making it straightforward to scale as demand increases. While Celery is written in Python, its messaging protocols are not language specific, allowing for clients in any language.

In this guide, we will go over how to deploy a Celery worker on the Koyeb serverless platform using git-driven deployment. We will create a repository with Celery and its dependencies installed, deploy RabbitMQ to act as our backing message broker, and then connect our GitHub account to Koyeb to automatically deploy the Celery worker to the platform.

At the end of this guide, you should have a Celery worker deployed to Koyeb that you can automatically update by pushing changes to your GitHub account.

Requirements

To follow along with this guide, you will need the following:

Python 3 installed on your local machine
A Koyeb account to deploy and run your Celery worker and RabbitMQ message broker
A GitHub account to store your application code and trigger deployments via repository changes

Steps

This guide will cover how to deploy a Celery worker to Koyeb through the following steps:

Create a Python Virtual Environment
Install Celery
Create a Basic Celery Application
Push the Celery Project to GitHub
Deploy RabbitMQ on Koyeb
Deploy the Celery Worker on Koyeb
Testing the Celery Worker

Create a Python Virtual Environment

To get started, we will create a Python virtual environment on our local machine to isolate our project's dependencies from the rest of the system packages. Python provides this functionality natively through the standard venv module.

Start by creating a new directory to store our project files.

mkdir ~/celery-koyeb

Next, create the virtual environment within the project directory by typing:

python3 -m venv ~/celery-koyeb/venv

This will create a directory called venv within the ~/celery-koyeb containing initialization scripts, copies of essential tools like pip, and links to the local Python interpreters.

To tell Python to use the virtual environment, activate it by sourcing the initialization script:

source ~/celery-koyeb/venv/bin/activate

Your shell prompt will change to reflect that the virtual environment is active for this session. Python will now use this directory structure in place of the system's standard Python file locations, allowing us to isolate dependencies.

Install Celery

Next, we need to install Celery within the virtual environment using pip, the Python package manager:

pip install celery

Python will download Celery and its dependencies and install them within the virtual environment filesystem.

Along with the Python modules, the installation also includes a small wrapper script called celery that lets you interact with it from the command line. Use the command now to verify that the installation was successful:

celery --version

The command should show the version as expected. The exact version returned does not matter, so long as the command returns successfully:

5.2.7 (dawn-chorus)

With Celery installed, record the project's dependencies by typing:

pip freeze > ~/celery-koyeb/requirements.txt

Create a Basic Celery Application

Next, we'll create a basic Celery application in our project directory.

Create a file called tasks.py within the ~/celery-koyeb directory with the following content inside:

import os
from celery import Celery


# Fail if the `BROKER_HOST` environment variable is not defined
BROKER_HOST = os.environ.get('BROKER_HOST', None)
if BROKER_HOST is None:
    raise Exception("BROKER_HOST environmental variable is not defined")
broker_url = 'pyamqp://guest@' + BROKER_HOST + '//'

app = Celery('tasks', broker=broker_url, backend='rpc://')

@app.task
def add(x, y):
    return x + y

Let's talk a bit about what is happening in the code above.

After importing the os module and the Celery class, the code above sets the location of the backing RabbitMQ message broker based on the value of the BROKER_HOST environment variable. If this variable is not set, we will error out to avoid accidentally deploying misconfigured instances.

Next, we create an instance of the Celery class called app. We pass tasks as the name of the main module and include the broker and backend configuration to allow Celery to connect to the RabbitMQ instance indicated by the environment variable mentioned earlier.

Finally, we create a function with the @app.task decorator to add it to Celery's task registry. Our function adds two numbers together and returns the result.

Save and close the file when you are finished.

Push the Celery Project to GitHub

Next, we will commit our project to git and push our changes to our GitHub account.

In your project directory, initialize a git repository to get started:

cd ~/celery-koyeb
git init

Add the tasks.py application and the requirements.txt file to the staging area. You can also create a minimal .gitignore file to make sure your virtual environment does not accidentally get committed. Commit the changes when you are done:

echo 'venv' >> .gitignore
git add tasks.py requirements.txt .gitignore
git commit -m 'Initial commit'

Create a new repository within your GitHub account. This can be either public or private. Add the GitHub repository address as the origin of your project and push your changes to the repository:

git remote add origin git@github.com:<YOUR_GITHUB_USERNAME>/<YOUR_GITHUB_REPOSITORY>.git
git branch -M main
git push -u origin main

If you refresh the GitHub repository page in your browser, your project files will appear.

Deploy RabbitMQ on Koyeb

Before we deploy our Celery worker, we need to deploy a RabbitMQ instance for Celery to use as a message broker. We will use the official RabbitMQ image on DockerHub for this purpose.

Go to the Overview tab of the Koyeb control panel and click Create Web Service to begin:

Select Docker as the deployment method.
In the Docker image field, enter docker.io/rabbitmq.
In the Exposed ports section, change the port number to 5672 and deselect the Public option. This will allow other services within this application (like our Celery application) to access the message broker using its service name without exposing the service to the internet.
Click the Deploy button.

Your new application will be created and the RabbitMQ service will begin to deploy within it. In a few minutes, the RabbitMQ container will finish deploying to Koyeb and the service will be marked as healthy.

Before moving on, copy the Private domain of your RabbitMQ deployment. We will need this later as we configure the Celery worker service.

Deploy the Celery Worker on Koyeb

Now that the RabbitMQ service is deployed, we can deploy the Celery application from our GitHub repository.

In the Apps tab of the Koyeb control panel, click the app that contains your RabbitMQ service. Click the Create Service button to deploy a new service to the app alongside the RabbitMQ container:

Choose GitHub as the deployment method.
Select the repository for your Celery application from the list.
In the Service type section, select Worker.
In the Builder section, click the Override toggle associated with the Run command and enter celery --app=tasks.app worker in the field.
In the Environment variables section, create a new variable called BROKER_HOST to tell the Celery worker where to find the RabbitMQ instance. Set the value to the Private domain you copied from the RabbitMQ service. It will follow this format: <YOUR_SERVICE_NAME>.<YOUR_APP_NAME>.koyeb.
Click the Deploy button.

The Celery application will be pulled from your GitHub repository and deployed. In a few minutes, the Celery worker will begin running within the service and be available for work.

If you want to learn about how Koyeb automatically builds your Python applications from git, make sure to read the how we build from git documentation.

Testing the Celery Worker

Unlike a web application, Celery does not have a web interface to easily show you that the deployment succeeded. We can, however, use Koyeb's built-in Console to test that the worker is running, able to connect with RabbitMQ, and can process tasks.

In the Koyeb control panel, click the Celery service and then select the Console tab.

In the console window, start a bash session by typing:

/bin/bash

Next, source the environment configuration file so that Python can find the modules associated with our application:

source .profile.d/python.sh

Start a Python session so that we can interact with the Celery application:

python

First, import our tasks module:

import tasks

Next, we can run the add function using Celery's delay method:

results = tasks.add.delay(93, 80)

We can then use the results object to check on the task's progress. Call the ready method to see if it has been processed yet:

results.ready()
True

You can check the answer returned by the function using the get method. It's usually a good idea to set a timeout so that it won't hang waiting if the result is not available yet:

results.get(timeout=1)
173

While this test uses trivial calculations, the results validate that the system is working as intended and able to communicate with RabbitMQ using Koyeb's service mesh.

When you are finished, exit the Python session by typing:

exit()

You can then close the bash session by typing:

exit

Conclusion

In this guide, we covered how to deploy a Celery worker on Koyeb using git-driven deployment. Because Celery requires a message broker for coordination, we also discussed how to deploy a RabbitMQ container on Koyeb using Docker-based deployment.

By deploying both of these services within a single application, we were able to configure Celery to connect with RabbitMQ over Koyeb's service mesh with minimal configuration. Our Celery deployment is backed by a GitHub repository, so it will automatically redeploy whenever new changes are pushed to the branch it is tracking. This makes it easy to change your running Celery worker configuration by modifying your project's code.

Questions or suggestions to improve this guide? Join us on the community platform to chat!