
The need for AI-generated images has been growing rapidly in recent years. These images are not only used for artistic purposes, but also for practical applications in various industries. For example, in the fashion industry, AI-generated images can be used to create virtual models for showcasing clothing. In the automotive industry, AI-generated images can be used for designing and testing new car models. And the best part? You can now run your own AI image generation machine on Koyeb GPUs.

The [FLUX.1 \[dev\] model](https://huggingface.co/black-forest-labs/FLUX.1-dev) (by BlackForestLabs) is an advanced AI image generation model that produces outstanding output quality. [It is a 12 billion parameter rectified flow transformer capable of generating images from text descriptions](https://huggingface.co/black-forest-labs/FLUX.1-dev). It features competitive prompt following, and is trained using guidance distillation. Additionally, the generated outputs can be utilized for personal, scientific, and non-commercial purposes as outlined in the [FLUX.1 \[dev\] Non-Commercial License](https://huggingface.co/black-forest-labs/FLUX.1-dev/blob/main/LICENSE.md).

In this tutorial, we will learn how to set up a Streamlit application, integrate the FLUX model for real-time image generation, and deploy the application using Docker and Koyeb, ensuring a scalable image generation service.

You can deploy the FLUX application as built in this tutorial using the [Deploy to Koyeb](https://www.koyeb.com/docs/build-and-deploy/deploy-to-koyeb-button) button below:

[![Deploy to Koyeb](https://www.koyeb.com/static/images/deploy/button.svg)](https://app.koyeb.com/deploy?name=flux-pytorch&type=git&repository=github.com/koyeb/example-flux-1-pytorch&branch=main&builder=dockerfile&instance_type=gpu-nvidia-rtx-4000-sff-ada)

## Requirements

To successfully follow this tutorial, you will need the following:

- [Git](https://git-scm.com/) installed
- [Python 3.6+](https://www.python.org/downloads/release/python-360/) or later
- A [Hugging Face](https://huggingface.co/) account
- A [Koyeb](https://app.koyeb.com/) account

## Understanding of the components

### Text-To-Image Generation and FLUX Model

The process of text-to-image generation involves the model interpreting input text and translating it into visual representations. This process consists of several steps: first, the model encodes the textual input into a latent space, capturing the semantic meaning of the words. Next, it employs a generative process to sample from this latent space, producing images that align with the described concepts. The FLUX model, developed by BlackForestLabs, is a state-of-the-art 12 billion parameter rectified flow transformer trained on extensive datasets containing pairs of images and their corresponding textual descriptions. This training enables the model to learn the intricate relationships between language and visual content. Consequently, users can input detailed prompts, and the FLUX model generates images that reflect those prompts with accuracy (and creativity).

### Streamlit

Streamlit is an open-source Python library designed to create interactive data applications, often referred to as dashboards. It empowers developers to build and share data apps simply and intuitively, eliminating the need for extensive web development expertise.

Streamlit apps are created as Python scripts, which are then executed within the Streamlit environment. The library offers a set of functions that can be utilized to add interactive elements to the app such as upload file button.

## Steps

1. [Set up the environment](#set-up-the-environment): Start by setting up your project directory, installing necessary dependencies, and configuring environment variables.
2. [Set up Streamlit](#set-up-streamlit): Next, install Streamlit and create the initial user interface for your application.
3. [Generate AI Images with FLUX Model](#generate-ai-images-with-flux-model): Using the FLUX model to generate AI images based on the user prompt and configuration.
4. [Dockerize the Streamlit application](#dockerize-the-streamlit-application): Create a Dockerfile to containerize your application for consistent deployment.
5. [Deploy to Koyeb](#deploy-to-koyeb): Finally, deploy your application on the Koyeb platform.

## Set up the environment

Let's start by creating a new Streamlit project. To keep your Python dependencies organized you should create a virtual environment.

First, create and navigate into a local directory:

```bash
# Create and move to the new directory
mkdir example-koyeb-flux-images
cd example-koyeb-flux-images
```

Afterwards, create and activate a new virtual environment:

```bash
# Create a virtual environment
python -m venv venv

# Active the virtual environment (Windows)
.\venv\Scripts\activate.bat

# Active the virtual environment (Linux)
source ./venv/bin/activate
```

Now, create a `requirements.txt` file with the following dependencies:

```txt
streamlit
watchdog
diffusers
torch
torchvision
einops
huggingface_hub[hf_transfer]
safetensors
sentencepiece
transformers
tokenizers
protobuf
requests
invisible-watermark
accelerate
peft
```

In terms of dependencies, we have included Streamlit for building a web app in Python, Hugging Face for real-time local use of the FLUX model, and watchdog to monitor file system events. The `accelerate` package is used to streamline the training and inference processes of machine learning models, making it easier to manage distributed training and optimize performance. The `invisible-watermark` package is utilized to embed invisible watermarks in generated content, ensuring uniqueness of the outputs produced by the model.

Now, you can install the dependencies with the following command:

```bash
pip install -r requirements.txt
```

Now, let's move on to creatng a new Streamlit project.

## Set up Streamlit

In this step, you will set up the Streamlit UI that will define the visual layout of the page, and the ability for users to load the FLUX model to start generating images with AI. All the logic for the project will reside in this file, so you can start by creating an `app.py` file with the following code:

```python
# File: app.py

import os, torch, streamlit
from diffusers import FluxPipeline
from huggingface_hub import login, snapshot_download

# Log in to Hugging Face using the provided token from environment variables
login(token=os.getenv('HF_TOKEN'))

# Set the title of the Streamlit application
streamlit.title("AI Image Generation with FLUX.1-dev")

# Create a text input field for the user to enter a prompt for image generation
prompt = streamlit.text_input("Enter your prompt:", "Astronaut in a jungle, cold color palette, muted colors, detailed, 8k")

# Create four columns for user inputs
col1, col2, col3, col4 = streamlit.columns(4)

# Column 1: Input for image width
with col1:
    num_width = streamlit.number_input("Width:", min_value=1, value=256)

# Column 2: Input for image height
with col2:
    num_height = streamlit.number_input("Height:", min_value=1, value=256)

# Column 3: Input for the number of images to generate
with col3:
    num_images = streamlit.number_input("Images:", min_value=1, value=3)

# Column 4: Input for the number of inference steps
with col4:
    num_inference_steps = streamlit.number_input("Steps:", min_value=1, value=3)

# Button to load the model from Hugging Face
if streamlit.button("Load Model"):
    snapshot_download(repo_id="black-forest-labs/FLUX.1-dev", cache_dir="./FLUX_1_dev")
```

The code above does the following:

- Begins with importing the Streamlit module.
- Logs into Hugging Face using a token stored in environment variables, allowing access to the model repository.
- Defines text values for each step and page's title and description
- Creates four columns to organize user inputs for image dimensions, the number of images, and inference steps. Each column contains a number input field with specified minimum values and default values.
- Creates a `Load Model` button to load the FLUX model from Hugging Face. When clicked, it downloads the model snapshot to a specified cache directory using `snapshot_download` function.

With this, you have setup a UI that is able to load the FLUX model per user request. Now, let's move on to generating AI images based on the user prompt and settings.

## Generate AI Images with FLUX Model

In this step, you will invoke FLUX model to generate AI images based on the user prompt. Per the default values set in the input, the model will be generate 3 images for each prompt by user. Make the following additions in the `app.py` file:

```python
# File: app.py

# Existing code

# Button to generate images using the FLUX model
if streamlit.button("Generate Image"):
    # Load the FLUX model with specified parameters
    pipe = FluxPipeline.from_pretrained("black-forest-labs/FLUX.1-dev", torch_dtype=torch.bfloat16, cache_dir="./FLUX_1_dev")
    
    # Generate images based on the user input parameters
    images = pipe(prompt=prompt, num_inference_steps=num_inference_steps, height=num_height, width=num_width, num_images_per_prompt=num_images).images
    
    # Create three columns to display the generated images
    cols = streamlit.columns(3)
    
    # Loop through the generated images and display them in the columns
    for i, image in enumerate(images):
        cols[i % 3].image(image, caption=prompt)
```

The code addition above does the following:

- Creates a button titled `Generate Image`. When clicked, it will instantiate a Flux pipeline for text-to-image generation.
- Invokes the pipeline with the user configured parameters to generate the desired height and width, number of inference steps to be taken, and number of images to be generated.
- Creates three columns to organize the generated images.

Now, you can run the Streamlit application with:

```bash
streamlit run ./app.py --server.port 8000
```

The application would now be ready on [http://localhost:8000](http://localhost:8000). Test the application by generating AI images by altering the default input values and your own custom prompt.

Now, let's dockerize the application to ensure consistency between multiple deployments.

## Dockerize the Streamlit application

Dockerizing deployments helps by creating a consistent and reproducible environment, ensuring that the application runs the same way on any system. It simplifies dependencies management and enhances scalability, making deployments more efficient and reliable. To dockerize, create a `Dockerfile` at the root of your project with the following code:

```Dockerfile
FROM python:3.12 AS runner

WORKDIR /app

COPY app.py requirements.txt .

RUN pip install -r requirements.txt --root-user-action=ignore
RUN pip install --pre torch --index-url https://download.pytorch.org/whl/nightly/cu121 --root-user-action=ignore # [!code ++]

RUN apt-get update 
RUN apt-get install -y libsm6 libxext6 git git-lfs # [!code ++]
RUN rm -rf /var/lib/apt/lists/*

EXPOSE 8000 # [!code ++]

ENV HF_HUB_ENABLE_HF_TRANSFER=1

ARG HF_TOKEN # [!code ++]
CMD ["streamlit", "run", "./app.py", "--server.port", "8000"] # [!code ++]
```

Apart from the usual Dockerfile to deploy Python applications, following tweaks/additions have been made in this code:

- `pip install --pre torch --index-url https://download.pytorch.org/whl/nightly/cu121 --root-user-action=ignore` to install PyTorch with CUDA support for GPU acceleration.
- `RUN apt-get update && apt-get install -y libsm6 libxext6 git git-lfs && rm -rf /var/lib/apt/lists/*` is used to install `git-lfs` and `git`, and then clean up package lists to reduce image size.
- `EXPOSE 8000` is used to specify the port on which the Streamlit application will run.
- `CMD ["streamlit", "run", "./app.py", "--server.port", "8000"]` is used to define the command to start the Streamlit app on port 8000.

With all configured, let's move on to deploy the application to Koyeb.

## Deploy to Koyeb

Now that you have the application running locally you can also deploy it on Koyeb and make it available on the internet.

Create a [new repository on your GitHub account](https://github.com/new) so that you can push your code.

You can download a [standard `.gitignore` file](https://raw.githubusercontent.com/github/gitignore/main/Python.gitignore) for Python from GitHub to exclude certain directories and files from being pushed to the repository:

```bash
curl -L https://raw.githubusercontent.com/github/gitignore/main/Python.gitignore -o .gitignore
```

Run the following commands in your terminal to commit and push your code to the repository:

```bash
git init
git add .
git commit -m "first commit"
git branch -M main
git remote add origin [Your GitHub repository URL]
git push -u origin main
```

You should now have all your local code in your remote repository. Now it is time to deploy the application.

Within the [Koyeb control panel](https://app.koyeb.com/), while on the Overview tab, initiate the app creation and deployment process by clicking Create Web Service.

1. Select **GitHub** as the deployment method.
2. Select your repository from the menu. Alternatively, deploy from the [example repository associated with this tutorial](https://github.com/koyeb/example-flux-1-pytorch) by entering `https://github.com/koyeb/example-flux-1-pytorch` in the public repository field.
4. In the Instance selection, select a GPU Instance.
5. Set your HuggingFace access token in the `HF_TOKEN` environment variable.
6. Finally, click the **Deploy** button.

Once the application is deployed, you can visit the Koyeb service URL (ending in `.koyeb.app`) to access the Streamlit application.

## Conclusion

In this tutorial, you built an AI Image Generation application using FLUX dev model with Streamlit framework. During the process, you learned how to invoke the Flux pipeline in python to generate AI images on the go, customized with user fed prompt and settings, and use the Streamlit framework to quickly prototype the user interface.

Given that the application was deployed using the Git deployment option, subsequent code push to the deployed branch will automatically initiate a new build for your application. Changes to your application will become live once the deployment is successful. In the event of a failed deployment, Koyeb retains the last operational production deployment, ensuring the uninterrupted operation of your application.

Perform real-time inference on high-performance infrastructure with zero-configuration.

Real-Time Inference with No-Ops

Learn how to use the FLUX model to generate images based on prompts in real-time.

Use FLUX, PyTorch, and Streamlit to Build an AI Image Generation App


One standout feature of Firefly models from [Adobe](https://www.adobe.com/products/firefly.html) is **generative fill: simply outline the area you want to modify, provide a prompt, and the model generates the content for you**.

Generative fill is possible due to **image inpainting**, which restores missing or damaged parts of an image by using information from surrounding pixels.
Common use cases range from personal photo editing to professional image restoration in various industries.

**Stable Diffusion is an open source generative AI model** that creates unique photorealistic images from text and image prompts.
In this tutorial, we will create an image inpaint service using Stable Diffusion. We will deploy the endpoint to Koyeb as a web service, enabling users to upload images and receive modified versions based on their prompts.

You can consult the [repository](https://github.com/koyeb/example-stable-diffusion-inpainting) for this guide to follow along on your own. You can deploy Stable Diffusion by clicking the [Deploy to Koyeb](/docs/build-and-deploy/deploy-to-koyeb-button) button below:

[![Deploy to Koyeb](https://www.koyeb.com/static/images/deploy/button.svg)](https://app.koyeb.com/deploy?name=stable-diffusion&type=git&repository=koyeb%2Fexample-stable-diffusion-inpainting&branch=main&builder=dockerfile&instance_type=gpu-nvidia-rtx-4000-sff-ada&ports=8000%3Bhttp%3B%2F)

<Admonition>Make sure to set the grace period to 300 seconds.</Admonition>

By the end, you will have built and deployed your own image inpainting service that will look like this:

<iframe
  width="560"
  height="315"
  src="https://www.youtube.com/embed/ACpNCwpYMp4?si=QJMPU2e1pmHVBw8z"
  title="YouTube video player"
  frameborder="0"
  allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
  referrerpolicy="strict-origin-when-cross-origin"
  allowfullscreen
></iframe>

## Requirements

Here are the following requirements for the project:

- Python 3.10 or later
- [Pytorch 2.4](https://pytorch.org/get-started/locally/)
- A [Koyeb account](https://app.koyeb.com/)

## Steps

Here are the steps we will follow to build the image inpainting service:

1. **Set up the environment**: Begin by setting up your project directory and installing the necessary dependencies, including Python and Hugging Face's Diffusers library. Configure the environment variables required for the application.
2. **Implement Inpaint**: Use a pre-trained model, like Stable Diffusion, to handle the inpainting process. This function will take the user’s image and mask to generate the output.
3. **Set up the Web Service:** Use Gradio to create the user interface for the inpainting service. The web interface will allow users to upload images and masks for the inpainting process.
4. **Dockerize the Application:** Create a Dockerfile to containerize the application. This step ensures the service is portable and can be deployed consistently across environments.
5. **Deploy to Koyeb GPU:** Deploy your Dockerized inpainting service on Koyeb. This final step brings your service online and makes it accessible to users.

## Overview of implementation of image inpaint service

In this section, we’ll guide you through implementing the image inpainting service and deploying it to Koyeb. Here is an overview of the approach to this project and a diagram of the application’s architecture:

![Image Inpainting Service Architecture](/static/images/tutorials/image-inpainting/image-inpainting-service-architecture.png)

## Set up the project

First, let's create a project directory named `inpaint-koyeb`. This will be the base folder where all the code, models, and configurations for the image inpainting service will reside.

```
mkdir inpaint-koyeb
cd inpaint-koyeb
```

### Create and activate a virtual environment

It's best to use a virtual environment to isolate the project dependencies and avoid conflicts with other projects. In this case, we’ll use Python 3.10

Ensure Python >=3.10 is installed. You can check your Python version by running:

```
python --version
```

Now, create and activate the virtual environment:

```
python -m venv env
```

Next, activate the virtual environment. On **macOS/Linux**:

```
source venv/bin/activate
```

On **Windows**:

```
.\venv\Scripts\activate
```

Once activated, your terminal will be prefixed with `(venv)`, indicating the active environment.

### Initialize Git repository

To track changes and manage your code efficiently, it’s a good idea to initialize your project directory as a Git repository. Run the following command:

```
git init
```

This will create a hidden `.git` folder in your project directory, allowing you to commit changes and work with version control.

### Install requirements

Create `requirements.txt`. The `requirements.txt` file lists the dependencies your project will need. Create the file and add the following dependencies to it:

```
torch==2.4
diffusers==0.30.2
gradio==4.43.0
transformers==4.44.2
gradio_client==1.3.0
```

As we will build a Docker container later, we must add the versions to ensure consistency across environments and prevent breaking changes.

Finally, install the dependencies listed in `requirements.txt` using the following command:

```
pip install -r requirements.txt
```

This will ensure all necessary libraries for the inpainting service are installed within your virtual environment.

### Create a `.gitignore` file

You should exclude specific files and directories, such as your virtual environment, from version control. Create a `.gitignore` file and add the venv folder to it to ensure Git does not track it:

```
# .gitignore
venv/
```

This will prevent your virtual environment from being uploaded to the repository.

## Implement inpaint service

Huggingface has made it much easier for developers to work with the Stable Diffusion model by creating `diffusers` library. It allows you to quickly implement models for tasks like image generation, inpainting, and more, leveraging state-of-the-art architectures like Stable Diffusion.

In this example, we use the `AutoPipelineForInpainting` class from the diffusers library to create an inpainting model.

### Import the necessary libraries

In an `app.py` file, import the necessary libraries:

```app.py
import torch
from diffusers import AutoPipelineForInpainting
```

- `torch`: This is the PyTorch library for tensors and GPU computations.
- `AutoPipelineForInpainting`: A pipeline specifically designed for the task of image inpainting.

### Load the pre-trained inpainting model

Next, load the pre-trained inpainting model in the `app.py` file:

```app.py
pipeline = AutoPipelineForInpainting.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0", torch_dtype=torch.float16
).to("cuda")

pipeline.enable_model_cpu_offload()
```

- **Model:** The model being used is `stable-diffusion-xl-base-1.0`, which supports inpainting. This pre-trained model means you don’t need to train it from scratch.
- **Inference Speed:** To optimize performance, we implemented the following:
  - torch_dtype=torch.float16 ensures the model uses half-precision (FP16) for computations, which speeds up inference on modern GPUs.
  - to("cuda") moves the model to the GPU for faster processing. This is important for large models like stable-diffusion-xl.
- **Memory**: To save memory, enable_model_cpu_offload() ensures the model offloads some parts of its computation to the CPU when GPU memory is full, allowing you to use larger models even on limited GPU resources.

### Defining the inpainting function

Now that the model is ready, we define the function in `app.py` that will handle the inpainting process:

```app.py
def image_inpaint(image_input, mask_input, prompt_input):
    image = pipeline(prompt=prompt_input, image=image_input, mask_image=mask_input).images[0]
    return image
```

The function `image_inpaint` is designed to take three inputs: the original image, a mask indicating the area to be filled in, and a text prompt that guides the model on how to fill in that masked region.

Inside the function, the `pipeline` object is used to perform the inpainting. It processes the image and mask according to the provided prompt, and then generates an output image where the masked part is filled in based on the prompt's instructions. Once the new image is created, the function returns it.

With this setup, we now have the core logic for modifying images by filling in missing parts. The next steps will cover how to create a user interface to make this functionality accessible to end users.

### Set up the web service

To set up the web service, we will use Gradio. Gradio is an easy-to-use library that allows developers to create web interfaces for machine learning models and other Python functions. We will create a function that includes two image input sections, one text input section, and an output image section.

In your `app.py' file, add the following code to set up the Gradio interface:

```app.py
import gradio as gr

def image_inpaint(image_input, mask_input, prompt_input):
    image = pipeline(prompt=prompt_input, image=image_input, mask_image=mask_input).images[0]
    return image

def gradio_interface():
    with gr.Blocks() as demo:
        gr.Markdown("## Image Inpainting Service")

        with gr.Row():
            with gr.Column():
                image_input = gr.Image(type="pil", label="Upload Image")
                mask_input = gr.Image(type="pil", label="Upload Mask")
                prompt_input = gr.Textbox(label="Enter Prompt", placeholder="Describe what you want to inpaint")
                submit_btn = gr.Button("Inpaint")

            with gr.Column():
                result_output = gr.Image(type="pil", label="Inpainted Image")

        # Connect the button click event with the image inpainting
        submit_btn.click(image_inpaint, inputs=[image_input, mask_input, prompt_input], outputs=result_output)

    demo.launch()

# Run the Gradio interface
if __name__ == "__main__":
    gradio_interface()
```

Here’s an explanation of the code above:

- **Import Gradio**: We begin by importing the gradio library to set up the interface for the image inpainting service.

- **Creating the Interface Function (gradio_interface)**: The function gradio_interface() is where we define the layout and logic of the web service.

- **Gradio Blocks**: gr.Blocks() define the overall structure of the interface. Think of it as the container for the entire UI.

- **Markdown Header**: gr.Markdown() adds a header describing the service: "Image Inpainting Service."

- **Creating Input Fields**:

  - **Image Input**: gr.Image(type="pil", label="Upload Image") allows users to upload a base image for inpainting.
  - **Mask Input**: gr.Image(type="pil", label="Upload Mask") lets users upload a mask, which defines the areas to be inpainted.
  - **Prompt Input**: gr.Textbox() enables users to input a description of the changes or details to be inpainted into the image.

- **Submit Button**: gr.Button("Inpaint") adds a button to trigger the inpainting process when clicked.

- **Output Image**: gr.Image(type="pil", label="Inpainted Image") is where the result of the inpainting operation will be displayed.

- **Button Click Action**: submit_btn.click() connects the button to the image_inpaint function, taking the inputs (image, mask, and prompt) and displaying the resulting inpainted image in the output.

- **Launch the Interface**: demo.launch() starts the Gradio interface, making the web service accessible.

To run the application locally, simply execute the file:

```
python app.py
```

## Dockerize the application

Dockerization is an essential step for packaging and deploying modern applications. Creating a Docker container ensures your application runs consistently in any environment. Deploying the application to Koyeb during the build step further streamlines this process.  Now, let’s go through the Dockerfile, which will allow you to containerize your Image Inpainting Service.

```Dockerfile
FROM python:3.10
WORKDIR /app

# Copy the requirements file
COPY requirements.txt app.py

# Install the dependencies
RUN pip install --upgrade pip && pip install -r requirements.txt
ENV GRADIO_SERVER_NAME="0.0.0.0”
ENV GRADIO_SERVER_PORT=${GRADIO_SERVER_PORT:-8000}
CMD ["python", "app.py"]
```

Apart from the usual dockerfile commands, these are some new additions:

- CMD ["python", "app.py"]: This command tells Docker to execute the `app.py` file (which starts the Gradio interface) when the container is run. The CMD instruction sets the container's default command.

## Deploy to Koyeb GPU

With the application now running locally, you can quickly deploy it to Koyeb and make it accessible online.

### Push the code to GitHub

Since we have already set up a GitHub repository, simply commit your changes and push the code to GitHub.

To do this, run the following commands in your terminal:

```bash
git add app.py Dockerfile requirements.txt .gitignore
git commit -m "first commit"
git branch -M main
git remote add origin [Your GitHub repository URL]
git push -u origin main`
```

### Deploy the application on Koyeb

1. In the [Koyeb control panel](https://app.koyeb.com/), go to the **Overview** tab. Click **Create Service** and select **Create Web Service** to begin the app creation and deployment
2. Choose **GitHub** as the deployment source
3. From the menu, select your repository
4. Under **Instance Selection**, choose a **GPU Instance**
5. In the **Builder** section, select **Dockerfile**
6. Adjust the **grace period** for TCP health checks to **300 seconds**
7. Click the **Deploy** button to finalize the process

After deploying the application, you can track the deployment progress through the provided build and runtime logs.
Once the deployment is complete and health checks pass, your Gradio application will be up and running.

Click the provided public URL to access your live application.

Here is an example of the deployed application:

<iframe
  width="560"
  height="315"
  src="https://www.youtube.com/embed/ACpNCwpYMp4?si=QJMPU2e1pmHVBw8z"
  title="YouTube video player"
  frameborder="0"
  allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
  referrerpolicy="strict-origin-when-cross-origin"
  allowfullscreen
></iframe>

# Conclusion

In this guide, you built an image inpainting service using Stable Diffusion and Gradio and deployed it on Koyeb. You learned how to set up a Python environment, implement the inpainting function with Stable Diffusion, and create a user-friendly web interface with Gradio. After containerizing the application with Docker, **Koyeb** enabled scalable and efficient deployment.

The models for image inpaint are computationally expensive, so we want something like Koyeb to handle the deployment and scaling.

With automatic builds and deployments managed through Git, Koyeb ensures your service remains operational, even in the event of a failed deployment, by reverting to the last stable version.


Learn how to create an image inpainting service using Stable Diffusion and Gradio and deploy it on Koyeb.

Use Stable Diffusion and PyTorch to Build an Image Inpainting Service

Fine-tune models like MistralAI using Koyeb's Serverless GPUs.

Fine-Tune LLMs with Serverless GPUs

A step-by-step guide to learn how to fine-tune MistralAI on domain-specific knowledge and evaluate the fine-tuned model's performance.

Fine-Tune MistralAI and Evaluate the Fine-Tuned Model on Koyeb Serverless GPUs


Large Language Models (LLMs) are fantastic tools for getting quick answers on programming questions. However, their knowledge is not always up to date and they may not know about your favourite framework or library.

In this guide, we'll walk you through how to fine-tune an LLM on your favourite project's documentation. This will let the model answer questions with (hopefully) correct, and up-to-date information. We'll be using [Llama 3.1 8B Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct), an open-weights model from Meta and teach it about Apple's new deep learning framework: [MLX](https://github.com/ml-explore/mlx).

We will first generate a custom LLM training dataset from Apple's documentation and publish it on the HuggingFace Hub. Then, we'll fine-tune Llama 3.1 8B using QLORA, a training method which significantly reduces GPU memory usage and training time. Finally, we'll deploy the model on Koyeb's serverless GPUs, enabling you to get answers to your questions in real-time.

Quick disclaimer: This guide is intended as an introductory overview. Fine-tuning a language model involves careful consideration of data distribution, hyperparameters, and continual pre-training. For production-level models, a more rigorous approach is required.

## Requirements

To successfully follow this tutorial, you will need the following:

- [Python 3.8](https://www.python.org/) or later.
- An [OpenAI API key](https://platform.openai.com/api-keys).
- A [HuggingFace access token](https://huggingface.co/settings/tokens) with write permissions and access to [Llama 3.1 8B Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct).
- A [Weights & Biases access token](https://wandb.ai/) (Optional).

## Steps

1. Configure the local environment.
2. Build the Apple MLX documentation from source (Optional).
3. Generate the training dataset with Python and the OpenAI API.
4. Fine-tune the model using Jupyter Notebook on Koyeb.
5. Deploy and use the fine-tuned model.

## Configure the local environment

First, we'll clone the repository for this project, create a Python virtual environment, and install the required dependencies.

```bash
# Clone the repository
git clone https://github.com/koyeb/example-finetune-llama-3-1.git
cd example-finetune-llama-3-1

# Create a virtual environment
python3 -m venv venv

# Activate the virtual environment (Windows)
.\venv\Scripts\activate.bat
#  Activate the virtual environment (Linux & macOS)
source ./venv/bin/activate
```

Now let's install the required dependencies:

```bash
pip install datasets==2.16.1 openai==1.42.0 tqdm
```

The `datasets` library is used to push our dataset to the HuggingFace Hub and the openai library lets us interact with the OpenAI API.

Next, we'll login to the HuggingFace Hub.

```
huggingface-cli login
```

Follow the instructions in the terminal and paste your access token when prompted.

## Build the Apple MLX documentation from source

The repository for this tutorial already contains the Apple MLX documentation in text format. However, if you want to build the documentation from source, you can follow the instructions below. Otherwise, you can skip to the next step.

You'll need to install doxygen to build the Apple MLX documentation from source.

```bash
# Install doxygen (macOS)
brew install doxygen
# Install doxygen (Linux)
sudo apt-get install doxygen
```

Now, you can clone the MLX repository and build the documentation using Doxygen.

```bash
# Clone the MLX repository
rm -r mlx ; git clone https://github.com/ml-explore/mlx.git

# Install the required dependencies and build the documentation in text format
cd mlx/docs
pip install -r requirements.txt
doxygen && make text

# Move back to the project directory
cd ../..
```

If everything went well, the `mlx/docs/build/text' directory should now contain the documentation in text format. If you encounter any issues, you can fall back to using the [pre-built documentation](https://github.com/koyeb/example-finetune-llama-3-1/tree/main/mlx/docs/build/text) from the repository.

## Generate the training dataset

To generate the training dataset, we'll use the OpenAI API. This step should use less than $10 in OpenAI credits and take an hour or so. If you don't have an OpenAI API key or don't want to use it, you can skip to the next step and use [the dataset we published on HuggingFace](https://huggingface.co/datasets/koyeb/Apple-MLX-QA).

In order to generate conversation samples from the MLX documentation, we'll use the `generate_dataset.py` script in this tutorial's repository. There's a lot going on in the script, so let's break it down:

- At the top of the file, we define the prompts used to generate questions and answers.
- After parsing the command-line arguments, we read all the documentation files in an array.
- For each chunk of documentation, we generate N questions using the chat endpoint of the OpenAI API.
  - We use OpenAI's structured output feature to ensure the model generates a list of questions. This is done by specifying a JSON schema in the `response_format` parameter. For more information on structured outputs, see the [OpenAI API documentation](https://platform.openai.com/docs/guides/structured-outputs).
- For each question, we generate an answer using the same chat endpoint.
- Finally, we write the question-answer pairs to a JSONL file and push it to the HuggingFace Hub.

Let's run the script, specifying the input directory, the OpenAI model to use and the HuggingFace repository to push the dataset to. This should be the name of your organization (or HuggingFace account) and the name of the dataset (for example koyeb/Apple-MLX-QA).

```bash
export OPENAI_API_KEY='your-openai-api-key'
export HF_ORG='your-huggingface-organization'
python generate_dataset.py --input mlx/docs/build/text --model gpt-4o --repo $HF_ORG/Apple-MLX-QA
```

Once done, you should see your dataset on the [HuggingFace Hub](https://huggingface.co/).

## Fine-tune the model using Jupyter Notebook on Koyeb

Now that we have our dataset, we can proceed with fine-tuning. The next step involves deploying a Jupyter Notebook server on a Koyeb GPU instance. To do this, you can visit the [One-Click App page for Jupyter Notebook on Koyeb](https://www.koyeb.com/deploy/pytorch-jupyter) and follow the instructions on the page.

Once your notebook is started, visit the URL of your service and connect to Jupyter using the password you set during the deployment process. Once you're logged into Jupyter, import the [notebook.ipynb](https://github.com/koyeb/example-finetune-llama-3-1/blob/main/notebook.ipynb) file that's in this tutorial's repository by clicking on the "Upload" button in the Jupyter interface as shown below:

![Upload the notebook](/static/images/tutorials/fine-tune-llama-3-1/upload-notebook.png)

The rest of the instructions for this step are in the notebook. Once you're done, you can come back here to deploy and use the fine-tuned model on Koyeb.

## Deploy and use the fine-tuned model on Koyeb

This section teaches you how to use the model in your Python code as well as how to deploy it for production use on Koyeb's serverless GPUs.

You can use your LORA adapter in Python code using torch, transformers, and peft. Here's an example:

```
pip install torch transformers peft
```

```python
import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

# Load the base model and tokenizer
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B-Instruct",
    torch_dtype=torch.bfloat16,
    device_map="cuda",
)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")

# Load the LORA adapter
model = PeftModel.from_pretrained(
    model,
    "koyeb/Meta-Llama-3.1-8B-Instruct-Apple-MLX-Adapter",
)

# Utility function for chat completions
def complete_chat(model, tokenizer, messages, **kwargs):
    inputs = tokenizer.apply_chat_template(messages, return_tensors="pt", return_dict=True, add_generation_prompt=True).to(model.device)
    num_input_tokens = len(inputs["input_ids"][0])
    model.eval()
    with torch.inference_mode():
        return tokenizer.decode(model.generate(**inputs, **kwargs)[0][num_input_tokens:], skip_special_tokens=True)

# Generate and print the response
print(complete_chat(model, tokenizer, [
    {"role": "system", "content": "You are a helpful AI coding assistant with expert knowledge of Apple latest machine learning framework: MLX. You can help answer questions about MLX, provide code snippets, and help debug code."},
    {"role": "user", "content": "How do you transpose a matrix in MLX?"}
], max_new_tokens=256))
```

For production, you can deploy your fine-tune on Koyeb's serverless GPUs using vLLM with One-Click Apps.

1. Visit the [One-Click App page for vLLM](/deploy/vllm) and click the "Deploy" button.
2. Override the command args and specify the HuggingFace repository for your merged model: `["--model", "YOUR-ORG/Meta-Llama-3.1-8B-Instruct-Apple-MLX", "--max-model-len", "8192"]`.
3. Set your HuggingFace access token in the `HF_TOKEN` environment variable. Optionally, set `VLLM_DO_NOT_TRACK` to `1` to disable telemetry.

Once deployed, you can interact with the model using the OpenAI API format. Here's an example using curl:

```
export SERVICE_URL='your-service.koyeb.app'
curl https://$SERVICE_URL/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
     "messages": [
        {"role": "system", "content": "You are a helpful AI coding assistant with expert knowledge of Apple latest machine learning framework: MLX. You can help answer questions about MLX, provide code snippets, and help debug code."},
        {"role": "user", "content": "How do you transpose a matrix in MLX?"}
    ],
     "model": "koyeb/Meta-Llama-3.1-8B-Instruct-Apple-MLX",
     "temperature": 0.3
   }'
```

## Conclusion

Congratulations, you've successfully fine-tuned Llama 3.1 8B using QLORA!

Remember, fine-tuning is an iterative process. Feel free to experiment with different hyperparameters and training methods to get the best results. You can also work on increasing the size or improving the quality of your training dataset using additional data sources or data augmentation techniques. Hack on!


Fine-tune custom models using Koyeb's serverless GPUs.

Learn how to fine-tune an Llama 3.1 8B using QLORA on your favourite project's documentation. This will enable the model to answer questions with (hopefully) correct, and up-to-date information.

Fine-Tune Llama 3.1 8B using QLORA on Koyeb Serverless GPUs


Recently, AI image creation has seen significant progress. Many new tools are available to make the creative process easier with flexible, powerful functionality. One of these tools, ComfyUI, has grown to prominence due to its versatile, ease of use, and high-quality image generation.

Whether you're an artist wanting to try new things or a developer wanting to use AI for visual content, ComfyUI is a powerful tool that can help solve your image-based needs. In this guide, we'll explain what ComfyUI is, how it works, and how to set it up on a Koyeb GPU.

We'll also show you how to install ComfyUI Manager to add custom modules and give a step-by-step process for making high-quality images with Flux. By the end of this guide, you'll know everything you need to get the most out of ComfyUI and create amazing visuals.

You can consult the [project repository](https://github.com/koyeb/example-comfyui) as work through this guide. You can deploy ComfyUI as configured in this tutorial using the [Deploy to Koyeb](https://www.koyeb.com/docs/build-and-deploy/deploy-to-koyeb-button) button below:

[![Deploy to Koyeb](https://www.koyeb.com/static/images/deploy/button.svg)](https://app.koyeb.com/deploy?name=example-comfyui&type=git&repository=koyeb%2Fexample-comfyui&branch=main&builder=dockerfile&instance_type=gpu-nvidia-rtx-4000-sff-ada&env%5B%5D=&ports=8501%3Bhttp%3B%2F)

## What is ComfyUI?

[ComfyUI](https://github.com/comfyanonymous/ComfyUI) is a powerful, easy-to-use tool for creating images with AI. Users can design workflows, adjust settings, and see results immediately.

### Key features

- **User-friendly interface:** ComfyUI is designed to be simple so users of all skill levels can easily navigate and use its features.
- **Modular workflow design:** Users can create custom workflows by connecting different components, enabling flexibility in the image creation process.
- **Real-time feedback:** ComfyUI provides instant visual feedback so users can see the effects of their adjustments and fine-tune their workflows as needed.
- **Integration with AI models:** ComfyUI supports a variety of pre-trained AI models, as well as custom models, allowing users to generate images that meet their specific needs.

### Use cases

ComfyUI is great for many different uses, such as:

- **Artistic creation:** Artists can use ComfyUI to explore new creative ideas, make unique visuals, and try out different styles and techniques.
- **Graphic design:** Graphic designers can use ComfyUI to quickly create design elements, make mockups, and streamline their work process.
- **Content creation:** Marketers, bloggers, and content creators can use ComfyUI to make eye-catching visuals for their content, making it more engaging and appealing.
- **Research and development:** AI researchers and developers can use ComfyUI to test and visualize the performance of different models, experiment with new algorithms, and push the limits of AI-driven image generation.

## Basics of a ComfyUI workflow

At its core, a ComfyUI workflow is a series of connected modules, each doing a specific job in the image creation process. You can arrange these modules in different ways to get different results, giving you the flexibility to customize your workflows to fit your needs.

### Key components of the workflow

Let's see the example of the default ComfyUI workflow to understand better the components:

![ComfyUI default workflow](/static/images/tutorials/comfyui-flux/comfyui_default_flow.png)

The components of the workflow can be grouped in the following categories:

- **Input modules:** This is the starting point of any workflow. The input module lets you set the initial settings like image size, model choice, and input data (such as sketches, text prompts, or existing images). In the example above, for instance, the `Load Checkpoint` and `CLIP Text Encode` components are input modules.
- **Processing modules:** These parts handle various aspects of the image creation process, like applying filters, adjusting colors, refining details, and more. You can stack or arrange processing modules in parallel to create complex effects. In the example above, the `KSampler` component is a processing module.
- **Output module:** These are the final stages of the workflow, where the generated image is created and saved. You can specify the output format, resolution, and other details. In the example above, the `Save Image` component is an output module.
- **Control modules:** These parts give you additional control over the workflow, allowing you to tweak settings, adjust weights, and fine-tune the process. Control modules are essential for getting the desired results and ensuring high-quality outputs. In the example above, the `Empty Latent Image` component is a control module.

## Install ComfyUI on Koyeb GPUs

If you want to use the power of cloud computing for your image generation tasks, installing ComfyUI on a Koyeb GPU is a great choice. Koyeb offers powerful GPU instances that can handle the demanding requirements of AI-driven image generation, making sure your processing is fast and efficient.

### Prerequisites

Before starting the installation process, make sure you have the following:

- A **Koyeb account:** [Sign up](https://app.koyeb.com/auth/signup) for a Koyeb account if you don't already have one.

### Create a Dockerfile

We'll start by preparing a Dockerfile so we can make sure we have all of the dependencies installed, especially for GPU support. Create a `Dockerfile` with the following contents:

```dockerfile copy
# Use the official Python base image
FROM python:3.12

# Clone the repository
RUN git clone https://github.com/comfyanonymous/ComfyUI.git

# Set the working directory
WORKDIR /ComfyUI

# Update pip, install GPU dependencies, and install Comfy dependencies
RUN pip install --upgrade pip && pip install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu121 && pip install -r requirements.txt

# Set the entry point for the container
CMD python3 main.py --listen 0.0.0.0 --port ${PORT:-8188}
```

This Dockerfile sets up a container image for running ComfyUI. Here's what it does step-by-step:

- First, it starts with a base Python image, specifically version 3.12.
- Next, it downloads the ComfyUI repository from GitHub. This repository contains all the code and resources needed to run ComfyUI.
- It sets the working directory to where ComfyUI was downloaded, so any subsequent commands are executed in the correct location.
- Next, it updates the `pip` package manager to the latest version to ensure compatibility with new packages. Chained to this `RUN` instruction are additional `pip` commands that install the GPU dependencies including PyTorch, and a command that installs ComfyUI's dependencies from the included `requirements.txt` file.
- Finally, it sets the default command to run the ComfyUI application when the container starts. This command uses **`0.0.0.0`** as the IP address, which means the application will be accessible from any network interface. It listens on the port defined by the `PORT` environment variable, using ComfyUI's default port 8188 as a fallback if the environment variable is unset.

### Create the repository

The final step is to create a [new repository on GitHub](https://github.com/new) to store the project files.

Once you're ready, run the following commands in your terminal to commit and push your code to the repository:

```bash copy
echo "# ComfyUI-Flux" >> README.md
git init
git add .
git commit -m "First Commit"
git branch -M main
git remote add origin [Your GitHub repository URL]
git push -u origin main
```

You should now have all your local code in your remote repository. Now it is time to deploy the Dockerfile.

### Deploy to Koyeb

In the [Koyeb control panel](https://app.koyeb.com/), while on the **Overview** tab, initiate the app creation and deployment process by clicking **Create App**.

On the App deployment page:

1. Select **GitHub** as your deployment method.
2. Choose the repository where your code resides. For example, `ComfyUI-Flux`.
3. Select the GPU you wish to use, for example, `RTX-4000-SFF-ADA`.
4. In the **Builder** section, choose **Dockerfile**.
5. In the **Service name** section, choose an appropriate name.
6. Finally, click **Deploy**.

After a few minutes, your application should be available at the indicated URL.

## What is ComfyUI Manager?

While ComfyUI comes with a set of pre-built modules, its real power comes from its extensibility through custom modules.

[ComfyUI Manager](https://github.com/ltdrdata/ComfyUI-Manager) is a plugin that lets users manage and install custom modules directly within the ComfyUI interface. It provides an easy-to-use interface for browsing available modules, installing them with a single click, and integrating them into your workflows.

## Install ComfyUI Manager

To install ComfyUI Manager, the simplest way is to modify your Dockerfile to include the necessary commands to install it.

Update your `Dockerfile` to add an additional `git clone` command as follows:

```dockerfile copy
# Use the official Python base image
FROM python:3.12

# Clone the repository
RUN git clone https://github.com/comfyanonymous/ComfyUI.git

# Set the working directory
WORKDIR /ComfyUI

# Update pip, install GPU dependencies, and install Comfy dependencies
RUN pip install --upgrade pip && pip install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu121 && pip install -r requirements.txt

# Clone ComfyUI-Manager
RUN git clone https://github.com/ltdrdata/ComfyUI-Manager.git /ComfyUI/custom_nodes/ComfyUI-Manager

# Set the entry point for the container
CMD python3 main.py --listen 0.0.0.0 --port ${PORT:-8188}
```

This Dockerfile sets up a container for running ComfyUI and installs also ComfyUI Manager. The only difference from the previous Dockerfile is that it clones the ComfyUI manager repository into ComfyUI's `custom_nodes` directory.

Once you're done, commit the changes to your repository and push them to GitHub:

```bash copy
git add .
git commit -m "Add ComfyUI Manager"
git push origin main
```

Your Koyeb application will automatically create a new deployment based on your updated Dockerfile.

## Using custom modules

With ComfyUI Manager installed, you can now easily integrate custom modules into your workflows.

Whether you're looking to add new processing techniques, experiment with different models, or create entirely new workflows, the possibilities are endless.

We will use ComfyUI Manager to install Flux in the next section to generate high-quality images.

## Workflow for high-quality image generation with Flux

One of the great features of ComfyUI is that it can work with advanced AI models like Flux, which are specifically made for creating high-quality images. In this section, we'll guide you through a detailed process for generating amazing images using ComfyUI and Flux.

### What is Flux?

[Flux](https://github.com/black-forest-labs/flux) is a powerful AI model made for creating high-quality images. It's one of the latest deep learning models that use large datasets, complex structures, and advanced algorithms to generate images that look realistic and are artistically impressive.

### Key features of Flux

- **High-resolution outputs:** Flux is designed to produce high-resolution images, making it perfect for tasks where detail and clarity are important, like digital art, graphic design, and print media.
- **Artistic flexibility:** One of Flux's standout features is its ability to adapt to different artistic styles. Whether you want photorealism, impressionism, or a more abstract look, Flux can be adjusted to create images that match your desired style.
- **Deep learning architecture:** Flux uses a deep neural network architecture, which includes multiple layers of convolutional, generative, and refinement processes. This complex architecture allows it to understand and replicate intricate patterns, textures, and lighting effects in images.
- **Contextual understanding:** Unlike some earlier models, Flux has a strong ability to understand the context within images. This means it can accurately maintain the coherence of a scene, ensuring that elements in the image are not only visually consistent but also logically related.
- **Versatile input methods:** Flux supports various input methods, including text prompts, sketches, and even other images. This versatility allows users to guide the image generation process more effectively, providing a high degree of creative control.

### Applications of Flux

- **Digital art:** Artists use Flux to create detailed and stylistically diverse pieces of digital art. Its ability to interpret and generate different artistic styles makes it a valuable tool for creators looking to push the boundaries of digital expression.
- **Graphic design:** Graphic designers benefit from Flux's high-resolution outputs and its capability to generate assets that can be directly used in branding, marketing materials, and product designs.
- **Content creation:** For content creators, Flux offers a way to quickly generate unique visuals that can enhance blog posts, social media content, and other digital media.
- **AI research:** Researchers in the field of artificial intelligence use Flux to explore new techniques in image generation, test theories in deep learning, and develop new applications for AI in the creative industries.

## Install Flux on ComfyUI with ComfyUI Manager

To install a Flux model/checkpoint on ComfyUI, you can use the ComfyUI Manager. Open the ComfyUI Manager by clicking **Manager** in the sidebar menu.

This will open the manager window:

![ComfyUI Manager main menu](/static/images/tutorials/comfyui-flux/comfyui_manager_main_menu.png)

From here, click on **Model Manager** and then search for "flux":

![ComfyUI Manager search flux](/static/images/tutorials/comfyui-flux/comfyui_manager_search_flux.png)

For this example, install the "Comfy Org/FLUX.1 [schnell]" checkpoint by click the **install** button.

After a couple of minutes the download and install should complete:

![ComfyUI Manager flux install complete](/static/images/tutorials/comfyui-flux/comfyui_manager_flux_install_complete.png)

Now you can build your workflow to generate a high-quality image. You can use our [sample workflow](https://github.com/koyeb/example-comfyui/blob/main/koyeb-workflow.json) as a starting point. Download the file to your local computer.

To load the workflow into ComfyUI, click the **Load** button in the sidebar menu and select the `koyeb-workflow.json` file you just downloaded. Next, select the Flux checkpoint in the **Load Checkpoint** node and type in your prompt in the **CLIP Text Encode (Prompt)** node.

When you are ready, press **CTRL-Enter** to run the workflow and generate the image:

![ComfyUI run Koyeb workflow](/static/images/tutorials/comfyui-flux/comfyui_run_koyeb_workflow.png)

Here is the prompt included in the Koyeb sample workflow file followed by an example of an image generated from it:

> A serene twilight scene by a calm lake surrounded by tall, evergreen pine trees. The sky is painted with soft shades of pink, orange, and purple as the sun sets in the background. A small wooden dock extends into the water, with a single lantern casting a warm glow. Gentle ripples on the lake's surface reflect the vibrant colors of the sky, while a few fireflies dance around the dock. In the distance, mist rises from the water, adding a mystical quality to the peaceful, nature-filled landscape.

![ComfyUI generated image](/static/images/tutorials/comfyui-flux/comfyui_generated_image.png)

## Conclusion

ComfyUI is a powerful and flexible tool for making images with AI. It has a user-friendly interface, lets you design your own workflows, and can work with advanced models like Flux. Whether you are an artist, designer, or researcher, ComfyUI gives you the tools you need to create amazing visuals easily.

By following the steps in this guide, you learned how to set up ComfyUI on a Koyeb GPU, add custom modules with ComfyUI Manager, and create high-quality images using a detailed workflow. The possibilities are endless, and with ComfyUI, the only limit is your imagination.


Learn how to deploy ComfyUI, an image creation workflow manager, to Koyeb to generate images with Flux, an advanced image generation AI model.

Using ComfyUI, ComfyUI Manager, and Flux to Generate High-Quality Images on Koyeb

Understand how the YOLO algorithm works and use it to identify and manipulate images through computer vision.

Using YOLO for Real-Time Object Detection with Koyeb GPUs


## Introduction

Real-time automated transcription is incredibly useful for anyone who needs to capture spoken content quickly and accurately. Whether you're creating subtitles for videos, transcribing podcast episodes, or documenting meeting notes, having an automated system can save you a lot of time and effort.

In practical terms, automated transcription can be used in various real-world scenarios. For instance, journalists can transcribe interviews on the fly, educators can provide real-time captions for their lectures, and businesses can document conference calls and meetings more efficiently.

[OpenAI Whisper](https://github.com/openai/whisper) is an open-source solution built for this purpose. It uses state-of-the-art machine learning algorithms to transcribe speech with high accuracy, even handling different accents and speaking speeds.

In this tutorial, you will learn how to set up a [Streamlit](https://streamlit.io/) application, integrate OpenAI Whisper for real-time podcast transcription, and deploy the application using Docker and Koyeb, creating a scalable transcription service.

You can consult the [project repository](https://github.com/koyeb/example-whisper-transcription) as work through this guide. You can deploy the podcast transcription application as built in this tutorial using the [Deploy to Koyeb](https://www.koyeb.com/docs/build-and-deploy/deploy-to-koyeb-button) button below:

[![Deploy to Koyeb](https://www.koyeb.com/static/images/deploy/button.svg)](https://app.koyeb.com/deploy?name=whisper-transcription&type=git&repository=koyeb%2Fexample-whisper-transcription&branch=main&builder=dockerfile&instance_type=gpu-nvidia-rtx-4000-sff-ada&env%5B%5D=&ports=8000%3Bhttp%3B%2F)

## Requirements

To successfully follow this tutorial, you will need the following:

- [Git](https://git-scm.com/) installed
- [FFmpeg](https://ffmpeg.org/) installed
- [Python 3.6+](https://www.python.org/downloads/release/python-360/) or later
- A [Koyeb](https://app.koyeb.com/) account

## Demo

Before we jump into the technical details, let me give you a sneak peek of what you will be building in this tutorial:

<iframe
  width="560"
  height="315"
  src="https://www.youtube.com/embed/HlVhwUriUnE"
  title="Using OpenAI Whisper to Transcribe Podcasts"
  frameborder="0"
  allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
  referrerpolicy="strict-origin-when-cross-origin"
  allowfullscreen
></iframe>

## Understanding the components

### OpenAI Whisper

OpenAI Whisper is a sophisticated speech-to-text (STT) model designed to transcribe spoken words into written text with high accuracy. Utilizing advanced machine learning algorithms, Whisper is capable of recognizing various accents, dialects, and speaking speeds. It can be integrated into voice assistants, dictation software, and real-time translation services to convert spoken language into text efficiently.

OpenAI Whisper can be used in sectors such as healthcare for medical dictation, in customer service for automated call transcriptions, and in media for generating subtitles for videos and podcasts. Its ability to handle complex speech patterns and languages makes it the go-to service in any application requiring high-quality speech-to-text conversion.

### Streamlit

Streamlit is an open-source Python library designed to create interactive data applications, often referred to as dashboards. It empowers developers to build and share data apps simply and intuitively, eliminating the need for extensive web development expertise.

Streamlit apps are created as Python scripts, which are then executed within the Streamlit environment. The library offers a set of functions that can be used to add interactive elements to the app such as upload file button.

## Steps

To build the transcription service, we'll complete the following steps:

1. [Set up the environment](#set-up-the-environment): Start by setting up your project directory, installing necessary dependencies, and configuring environment variables.
2. [Set up Streamlit](#set-up-streamlit): Next, install Streamlit and create the initial user interface for your application.
3. [Transcribe podcasts with OpenAI Whisper](#transcribe-podcasts-with-open-ai-whisper): Use OpenAI Whisper to transcribe podcasts into text with timestamps.
4. [Dockerize the Streamlit application](#dockerize-the-streamlit-application): Create a Dockerfile to containerize your application for consistent deployment.
5. [Deploy to Koyeb](#deploy-to-koyeb): Finally, deploy your application on the Koyeb platform.

## Set up the environment

Let's start by creating a new Streamlit project. To keep your Python dependencies organized you should create a virtual environment.

First, create and navigate into a local directory:

```bash copy
# Create and move to the new directory
mkdir example-whisper-koyeb-gpu
cd example-whisper-koyeb-gpu
```

Afterwards, create and activate a new virtual environment:

```bash copy
# Create a virtual environment
python -m venv venv

# Active the virtual environment (Windows)
.\venv\Scripts\activate.bat

# Active the virtual environment (Linux)
source ./venv/bin/activate
```

Now, you can install the required dependencies. Open a `requirements.txt` file in your project directory with the following contents:

```none copy
streamlit
openai-whisper
watchdog
```

Pass the file to `pip` to install the dependencies:

```bash copy
pip install -r requirements.txt
```

For the dependencies, we have included Streamlit for building the web app, OpenAI Whisper for real-time transcription, and watchdog to monitor file system events.

Don't forget to save your dependencies to a `requirements.txt` file:

```bash copy
pip freeze > requirements.txt
```

Now, let's move on to creating a new Streamlit project.

## Set up Streamlit

In this step, you will set up the Streamlit UI that will allow users to upload an audio file, click a button to start the transcribing process, and finally present the segmented transcriptions in an user-friendly manner. All of the logic for the project will reside in this file, so you can start by creating a `app.py` file with the following code:

{/* prettier-ignore-start */}

```python copy
# File: app.py

import streamlit

stream_button_styles = """
<style>
    header { display: none !important; }
</style>
"""

page_styles = """
<style>
    h1 { font-size: 2rem; font-weight: 700; }
    h2 { font-size: 1.7rem; font-weight: 600; }
    .timestamp { color: gray; font-size: 0.9rem; }
</style>
"""

page_title = "Using OpenAI Whisper to Transcribe Podcasts"

page_description = "A demo showcasing the use of OpenAI Whisper to accurately and efficiently convert spoken content from podcasts into written text."

koyeb_box = "To deploy Whisper within minutes, <a href=\"https://koyeb.com/ai\">Koyeb GPUs</a> provide the easiest and most efficient way. Koyeb offers a seamless platform for deploying AI models, leveraging high-performance GPUs to ensure fast and reliable transcriptions."

step_1 = "1. Upload Podcast"

step_2 = "2. Invoke OpenAI Whisper to transcribe podcast 👇🏻"

step_3 = "3. Transcription &nbsp; 🎉"

def unsafe_html(tag, text):
    return streamlit.markdown(f"<{tag}>{text}</{tag}>", unsafe_allow_html=True)

def main():
    # Set title for the page
    streamlit.set_page_config(page_title, layout="centered")
    # Inject hide buttons CSS
    streamlit.markdown(stream_button_styles, unsafe_allow_html=True)
    # Inject page CSS
    streamlit.markdown(page_styles, unsafe_allow_html=True)
    # Create a H1 heading on the page
    streamlit.title(page_title)
    unsafe_html("h2", page_description)
    unsafe_html("p", koyeb_box)
    audio_file = streamlit.file_uploader(step_1, type=["mp3", "mp4", "wav", "m4a"])
    if audio_file:
        # If file is received
        # Write the file on the server
        # Show next step
        unsafe_html("small", step_2)
        if streamlit.button("Transcribe"):
            # Get the transcription
            unsafe_html("small", step_3)
            # Showcase the transcription

if __name__ == "__main__":
    main()
```

{/* prettier-ignore-end */}

The code above does the following:

- Begins by importing the Streamlit module
- Defines CSS for hiding the navigation bar and styling the headings
- Defines text values for the page's title, description, and each step
- Defines the `unsafe_html` function to dynamically create the HTML tags with content
- Accepts an audio file using Streamlit's builtin `file_uploader` function

With this, you have setup a UI that is able to accept podcast audio files from the user. Now, let's move on to transcribing the audio file obtained.

## Transcribe podcasts with OpenAI Whisper

In this step, you will invoke OpenAI Whisper's base model to transcribe an audio file. By default, the model is able to return the timestamps along with the transcription. This enables you to use the generated transcriptions as subtitles as well. Make the following additions in the `app.py` file:

{/* prettier-ignore-start */}

```python copy
# File: app.py

# Existing imports
# . . .
import whisper # [!code ++]

model = whisper.load_model("base") # [!code ++]

# ...

def unsafe_html(tag, text):
    # ...

# Generate transcription of each segment
def timestamp_html(segment): # [!code ++]
    return f'<span class="timestamp">[{segment["start"]:.2f} - {segment["end"]:.2f}]</span> {segment["text"]}' # [!code ++]

# Transcribe an audio file
def transcribe_audio(audio_file): # [!code ++]
    return model.transcribe(audio_file.name) # [!code ++]

# Write the audio file on server
def write_audio(audio_file): # [!code ++]
    with open(audio_file.name, "wb") as f: # [!code ++]
        f.write(audio_file.read()) # [!code ++]

def main():
    # ...
    if audio_file:
        # If file is received
        # Write the file on the server
        write_audio(audio_file) # [!code ++]
        # Show next step
        unsafe_html("small", step_2)
        if streamlit.button("Transcribe"):
            # Get the transcription
            transcript_text = transcribe_audio(audio_file) # [!code ++]
            unsafe_html("small", step_3)
            # Showcase the transcription
            for segment in transcript_text["segments"]: # [!code ++]
                unsafe_html("div", timestamp_html(segment)) # [!code ++]

if __name__ == "__main__":
    main()
```

{/* prettier-ignore-end */}

The changes above do the following:

- Import and instantiate the OpenAI Whisper base model
- Define a `timestamp_html` function to display the transcription with **start** and **end** timestamps
- Define a `transcribe_audio` function which invokes the model to generate transcriptions of the audio file
- Define a `write_audio` function to write the audio file on the server
- If an audio file is found, it writes the file on the server
- If the **Transcribe** button is clicked in the UI, `transcribe_audio` and `timestamp_html` functions are invoked to generate and display the transcriptions of the podcast

Now, you can run the Streamlit application with:

```bash copy
streamlit run ./app.py --server.port 8000
```

The application would now be ready on `http://localhost:8000`. Test the application in action by uploading one of your favorite podcasts file and see the transcriptions generated in real-time.

Next, let's dockerize the application to ensure consistency between multiple deployments.

## Dockerize the Streamlit application

Dockerizing deployments creates a consistent and reproducible environment, ensuring that the application runs the same way on any system. It simplifies dependency management and enhances scalability, making deployments more efficient and reliable. To dockerize, create a `Dockerfile` at the root of your project with the following content:

{/* prettier-ignore-start */}

```dockerfile {16,25,27} copy
FROM python:3.9 AS builder

WORKDIR /app

RUN python3 -m venv venv
ENV VIRTUAL_ENV=/app/venv
ENV PATH="$VIRTUAL_ENV/bin:$PATH"

COPY requirements.txt .
RUN pip install -r requirements.txt

FROM python:3.9 AS runner

WORKDIR /app

RUN apt-get update && apt-get install -y ffmpeg && rm -rf /var/lib/apt/lists/*

COPY --from=builder /app/venv venv
COPY app.py app.py

ENV VIRTUAL_ENV=/app/venv
ENV PATH="$VIRTUAL_ENV/bin:$PATH"

EXPOSE 8000

CMD ["streamlit", "run", "./app.py", "--server.port", "8000"]
```

{/* prettier-ignore-end */}

Apart from the usual Dockerfile to deploy Python applications, the following tweaks and additions have been made in this code:

- `RUN apt-get update && apt-get install -y ffmpeg && rm -rf /var/lib/apt/lists/*` is used to install `ffmpeg`, and then clean up package lists to reduce image size
- `EXPOSE 8000` is used to specify the port on which the Streamlit application will run
- `CMD ["streamlit", "run", "./app.py", "--server.port", "8000"]` is used to define the command to start the Streamlit app on port 8000

With everything configured, let's move on to deploy the application to Koyeb.

## Deploy to Koyeb

Now that you have the application running locally you can also deploy it on Koyeb and make it available on the internet.

Create a [new repository on your GitHub account](https://github.com/new) so that you can push your code.

You can download a [standard `.gitignore` file](https://raw.githubusercontent.com/github/gitignore/main/Python.gitignore) for Python from GitHub to exclude certain directories and files from being pushed to the repository:

```bash copy
curl -L https://raw.githubusercontent.com/github/gitignore/main/Python.gitignore -o .gitignore
```

Run the following commands in your terminal to commit and push your code to the repository:

```bash copy
git init
git add app.py Dockerfile requirements.txt .gitignore
git commit -m "first commit"
git branch -M main
git remote add origin [Your GitHub repository URL]
git push -u origin main
```

You should now have all of your local code in your remote repository. Now it is time to deploy the application.

Within the [Koyeb control panel](https://app.koyeb.com/), while on the Overview tab, initiate the app creation and deployment process by clicking **Create Service** and then choosing **Create web service**.

1. Select **GitHub** as the deployment source.
2. Select your repository from the menu. Alternatively, deploy from the [example repository associated with this tutorial](https://github.com/koyeb/example-whisper-transcription) by entering `https://github.com/koyeb/example-whisper-transcription` in the public repository field.
3. In the **Instance** selection, select a GPU Instance.
4. In the **Builder** section, choose **Dockerfile**.
5. Finally, click the **Deploy** button.

Once the application is deployed, you can visit the Koyeb service URL (ending in `.koyeb.app`) to access the Streamlit application.

## Conclusion

In this tutorial, you built a podcast transcription application with the Streamlit framework and OpenAI Whisper. During the process, you learned how to invoke the OpenAI Whisper model in Python to generate transcription with timestamps, and how to use the Streamlit framework to quickly prototype a user interface with a functioning upload button in a few lines of code.

Given that the application was deployed using the Git deployment option, subsequent code push to the deployed branch will automatically initiate a new build for your application. Changes to your application will become live once the deployment is successful. In the event of a failed deployment, Koyeb retains the most recent operational production deployment, ensuring the uninterrupted operation of your application.


Learn how to use OpenAI Whisper to build an app to generate transcription of podcast audio files in real-time.

Using OpenAI Whisper to Transcribe Podcasts on Koyeb


## Introduction

[Continue](https://continue.dev) is an open-source AI code assistant that connects any models and context to build custom autocomplete prompts and chat experiences inside the IDE, like VS Code and JetBrains.

[Ollama](https://ollama.com/) is a self-hosted AI solution to run open-source large language models on your own infrastructure, and [Codestral](https://ollama.com/library/codestral) is [MistralAI's](https://mistral.ai/) first-ever code model designed for code generation tasks.

In this guide, we will demonstrate how to use Continue with [Ollama](/deploy/ollama), the [Mistral Codestral](https://mistral.ai/news/codestral/) model, and [Koyeb GPUs](/blog/serverless-gpus-public-preview-run-ai-workloads-on-h100-a100-l40s-and-more) to build a custom, self-hosted AI code assistant.

When complete, you will have a private AI code assistant for autocomplete prompts and chat available within VS Code and JetBrains.

## Requirements

To successfully follow and complete this guide, you need:

- A [Koyeb account](https://app.koyeb.com) to deploy and run Ollama
- [VS Code](https://code.visualstudio.com/) or [JetBrains](https://www.jetbrains.com/) installed on your machine

## Steps

To complete this guide and build a custom AI code assistant using Continue, Ollama, Codestral, and Koyeb GPUs, you need to follow these steps:

1. [**Deploy Ollama on Koyeb's GPUs**](#deploy-ollama-on-koyebs-gpus)
2. [**Install and configure the Continue package in VS Code**](#install-and-configure-the-continue-package-in-vs-code)
3. [**Get started with your custom AI code assistant**](#get-started-with-your-custom-ai-code-assistant)

## Deploy Ollama on Koyeb's GPUs

To get started, we will deploy Ollama on Koyeb's GPUs. Ollama will be used to run the Mistral Codestral model on a Koyeb RTX 4000 SFF ADA which is ideal for cost-effective AI inference and running open-source large language models.

To create and deploy Ollama on Koyeb, we will use the [Deploy to Koyeb](https://www.koyeb.com/docs/build-and-deploy/deploy-to-koyeb-button) button below:

[![Deploy to Koyeb](https://www.koyeb.com/static/images/deploy/button.svg)](https://app.koyeb.com/deploy?name=ollama&type=docker&image=ollama%2Follama&command=serve&instance_type=gpu-nvidia-rtx-4000-sff-ada&env%5B%5D=&ports=11434%3Bhttp%3B%2F)

On the service configuration page, you can customize the [Service](/docs/reference/services) name, [Instance](/docs/reference/instances) type, and other settings to match your requirements.

When you are ready, click the **Deploy** button to create the service and start the deployment process.

After a few seconds, your Ollama service will be deployed and running on Koyeb.

The next step is to pull the Mistral Codestral model to use it with Ollama. To do so, retrieve the Service URL from the Koyeb dashboard and run the following command in your terminal:

```bash
curl https://<YOUR_SUBDOMAIN>.koyeb.app/api/pull -d '{
  "name": "codestral"
}'
```

<Admonition>Take care to replace the base URL ending in `koyeb.app` with your actual service URL.</Admonition>

Ollama will pull the Mistral Codestral model and prepare it for use. This might take a few moments. Once it's done, we can move to the next step and configure Continue to use the `ollama` provider.

## Install and configure the Continue package in VS Code

With Ollama deployed, we will show how to configure Continue for VS Code to use `ollama` as a provider. For JetBrains, please refer to the [Continue documentation](https://docs.continue.dev/quickstart#jetbrains).

Get started by installing the [Continue VS Code extension](https://marketplace.visualstudio.com/items?itemName=Continue.continue).
This will open the Continue extension page for VS Code. Click the **Install** button to install the extension.

Once the install has completed, open the `~/.continue/config.json` file on your machine and edit it to match the format below:

```json
{
  "models": [
    {
      "title": "Codestral on Koyeb",
      "apiBase": "https://<YOUR_SUBDOMAIN>.koyeb.app/",
      "provider": "ollama",
      "model": "llama3:8b"
    }
  ]
}
```

The above configuration tells Continue to:

1. use the `ollama` provider
2. use the Mistral Codestral model
3. use the Ollama Instance located at the Koyeb Service URL

<Admonition>Take care to replace the `apiBase` value with your Ollama Service URL.</Admonition>

Restart VS Code to apply the changes and get started using the AI code assistant.

## Get started with your Custom AI code assistant

Use the following shortcuts to access Continue and interact with the AI code assistant:

- cmd+L (MacOS)
- ctrl-L (Windows / Linux)

You can now start asking questions about your codebase, get autocomplete suggestions, and more.

<Banner
  title="Blazing-Fast AI Deployments"
  description="Enjoy automatic continuous deployment, global load balancing, real-time metrics and monitoring, autoscaling, and more."
  type="claim-free"
  buttonText="Deploy Now"
  buttonLink="https://app.koyeb.com/"
/>

## Conclusion

In this guide, we demonstrated how to use Continue, Ollama, MistralAI's Codestral, and Koyeb GPUs to build a custom autocomplete and chat experience inside of VS Code.
This tutorial covers the basics of how to get started using Continue. To go further, be sure to check out the [Continue documentation](https://docs.continue.dev/intro)
to learn more about how to use Continue.


Run your AI workloads worldwide on high-performance infrastructure.

Global deployments of LLMs and AI apps

This guide shows how to use Continue with Ollama, a self-hosted AI solution to run the Mistral Codestral model on Koyeb GPUs

Use Continue, Ollama, Codestral, and Koyeb GPUs to Build a Custom AI Code Assistant


## Introduction

[Fooocus](https://github.com/lllyasviel/Fooocus) is an AI-powered image generation tool that makes it easy to create custom images from descriptive prompts. Built using [Gradio](https://www.gradio.app/), Fooocus uses a prompt-centered approach to image generation, taking care of much of the behind configuration needed to produce high-quality images.

In this guide, we will demonstrate how to deploy and configure Fooocus on Koyeb's GPU instances. We will set up a custom `Dockerfile` to configure authentication and push our project to GitHub. Afterwards, we will deploy the project to Koyeb to build a container image and launch it on GPU-powered hardware.

You can consult the [repository for this guide](https://github.com/koyeb/example-fooocus) to follow along on your own. You can deploy the Fooocus instance by clicking the [Deploy to Koyeb button](/docs/build-and-deploy/deploy-to-koyeb-button) below:

[![Deploy to Koyeb](https://www.koyeb.com/static/images/deploy/button.svg)](https://app.koyeb.com/deploy?name=fooocus&type=git&repository=koyeb%2Fexample-fooocus&branch=main&builder=dockerfile&instance_type=gpu-nvidia-rtx-4000-sff-ada&env%5BCMDARGS%5D=--listen+--port%3D%7B%7B+PORT+%7D%7D&ports=8000%3Bhttp%3B%2F)

You will need to set the **Grace period** in the **Health checks** section to 300 seconds during configuration. If you want to password protect your Fooocus instance, add the `FOOOCUS_USERNAME` and `FOOOCUS_PASSWORD` environment variables with appropriate values. You can consult the appropriate sections of this guide for additional information.

## Requirements

To successfully follow and complete this guide, you need:

- A [Koyeb account](https://app.koyeb.com) to build and run the Fooocus instance on GPUs.
- Access to GPU Instances on Koyeb. [Join the preview today](https://www.koyeb.com/ai) to gain access.

## Steps

To complete this guide and deploy your own Fooocus instance, you'll need to follow these steps:

1. [Create a custom Dockerfile](#create-a-custom-dockerfile)
2. [Push the Dockerfile to GitHub](#push-the-dockerfile-to-git-hub)
3. [Deploy Fooocus on Koyeb](#deploy-fooocus-on-koyeb)

## Create a custom Dockerfile

The Fooocus project provides [Docker images](https://github.com/lllyasviel/Fooocus/pkgs/container/fooocus) based on the [`Dockerfile` included in the repository](https://github.com/lllyasviel/Fooocus/blob/main/Dockerfile). However, the provided images do not configure authentication by default, meaning that, once deployed, the Fooocus instance would be accessible to anyone who visits the URL.

To mitigate this, we will create a custom `Dockerfile` based on the official `Dockefile`. It will conditionally write a [`auth.json` file](https://github.com/lllyasviel/Fooocus?tab=readme-ov-file#ui-access-and-authentication) to the image filesystem if the appropriate variables are configured at build time.

To begin, create a new project directory and navigate inside:

```bash copy
mkdir example-fooocus
cd example-fooocus
```

Inside, create a `Dockerfile` with the following contents:

```dockerfile copy
FROM ghcr.io/lllyasviel/fooocus

ARG FOOOCUS_USERNAME
ARG FOOOCUS_PASSWORD

RUN if [ -n "$FOOOCUS_USERNAME" -a -n "$FOOOCUS_PASSWORD" ]; then echo "[ { \"user\": \"$FOOOCUS_USERNAME\", \"pass\": \"$FOOOCUS_PASSWORD\" } ]" > /content/app/auth.json; fi
```

If the `FOOOCUS_USERNAME` and `FOOOCUS_PASSWORD` variables are set at build time, this Dockerfile will create the `auth.json` file that Fooocus checks to configure authentication. If these values are not set, authentication will be disabled.

## Push the Dockerfile to GitHub

The small `Dockerfile` above is the only thing you need to configure authentication for Fooocus. Next, commit the file to a git repository and push it to GitHub.

[Create a new GitHub repository](https://github.com/new) and then run the following commands to commit and push changes to your GitHub repository:

```bash /<YOUR_GITHUB_USERNAME>/ /<YOUR_REPOSITORY_NAME>/ copy
git add :/
git commit -m "Initial commit"
git remote add origin git@github.com:<YOUR_GITHUB_USERNAME>/<YOUR_REPOSITORY_NAME>.git
git branch -M main
git push -u origin main
```

**Note:** Make sure to replace `<YOUR_GITHUB_USERNAME>` and `<YOUR_REPOSITORY_NAME>` with your GitHub username and repository name.

## Deploy Fooocus on Koyeb

Now that the `Dockerfile` is on GitHub, we can deploy it to Koyeb. On the **Overview** tab of the [Koyeb control panel](https://app.koyeb.com/), click **Create Web Service** to begin:

1. Select **GitHub** as the deployment method.
2. Select your Fooocus project repository. Alternatively, you can enter our public [Fooocus example repository](https://github.com/koyeb/example-fooocus) into the **Public GitHub repository** field at the bottom of the page: `https://github.com/koyeb/example-fooocus`.
3. In the **Environment variables** section, click **Bulk edit** to enter multiple environment variables at once. In the text box that appears, paste the following:

   ```bash copy
   CMDARGS=--listen --port={{ PORT }}
   FOOOCUS_USERNAME=
   FOOOCUS_PASSWORD=
   ```

   Set the variable values to reference your own information as follows:

   - `CMDARGS`: The `--listen --port={{ PORT }}` arguments configure Fooocus to listen for external connections. Koyeb automatically sets the `PORT` environment variable to the port it exposes. This will be automatically substituted with the correct value on deployment.
   - `FOOOCUS_USERNAME`: Set to the username you wish to use to authenticate. Remove this environment variable to deploy without authentication.
   - `FOOOCUS_USERNAME`: Set to the password you wish to use to authenticate. Remove this environment variable to deploy without authentication.

4. In the **Instance** section, select the **GPU** category and choose **RTX-4000-SFF-ADA**. These Instances are available when you request access to the [GPU preview](https://www.koyeb.com/ai).
5. In the **Health checks** section, set the **Grace period** to 300 seconds. This will provide time for the server to download all of the relevant models from Hugging Face and initialize the server.
6. Click **Deploy**.

Koyeb will pull the provided Fooocus repository, build the `Dockerfile` it contains, and run it on a GPU Instance. During deployment, Fooocus will fetch the provided model from Hugging Face and start up the service to expose it to users. If you provided authentication details, these will be configure during the build process.

Once the deployment is complete, access your Fooocus instance by visiting your Koyeb deployment URL. The application URL should have the following format:

```
https://<YOUR_APP_NAME>-<YOUR_KOYEB_ORG>-<HASH>.koyeb.app
```

Enter your provided username and password, if configured. The Fooocus prompt interface will be displayed, allowing you to generate images by describing what you would like to produce. To modify the default configuration, check the **Advanced** box below the prompt. From here, you can change the generator settings, the image styles, the models that will be used, and more.

## Conclusion

In this guide, we discussed how to configure and deploy a Fooocus Instance on Koyeb to generate AI images. We started with the [basic Fooocus Docker image](https://github.com/lllyasviel/Fooocus/pkgs/container/fooocus) and configured authentication based on user-provided variables. Afterwards, we deployed Fooocus to Koyeb by building the `Dockerfile` and launching it on Koyeb's GPU Instances.

This process provides you with a fully-functional AI-powered image generator. You can tweak the settings to change the styles the generator will use, its model configuration, and the formats it outputs. As you experiment, you might want to check out the [Fooocus Docker instructions](https://github.com/lllyasviel/Fooocus/blob/main/docker.md) and [troubleshooting tips](https://github.com/lllyasviel/Fooocus/blob/main/troubleshoot.md).


Go from development to high-throughput inference in minutes

High-performance Severless for Inference

Learn how to deploy and configure Fooocus, an AI-based image generation tool, on Koyeb.

Deploy Fooocus and Generate AI Images on Koyeb GPUs

Deploy your AI applications across continents and run on an edge network of 250 locations.

Low latency AI workloads

Learn how to set up a vLLM Instance to run inference workloads and host your own OpenAI-compatible API on Koyeb.