The Koyeb Serverless Engine: Docker Containers and Continuous Deployment of Functions

In July, we announced the early access of the Koyeb platform to help developers and businesses run serverless data-processing apps in minutes. Since then, we have received a lot of feedback and have been at work improving the product.

Today, we are proud to announce the public availability of the Koyeb Serverless Engine with our latest features to deploy your own code. In addition to the ready-to-use integrations, you can now seamlessly deploy Docker Containers and Code Functions with built-in Continuous Deployment using Git.

This release brings the power of the Koyeb Serverless Engine to all developers and businesses with event-driven processing, native autoscaling, and a complete secret management engine. The Koyeb platform provides strong primitives for data-processing with our Universal S3-Compliant Object Storage API and ready-to-use integrations. We're also happy to share that our catalog is now open-source.

The Koyeb platform offers an efficient solution to deploy your serverless applications. It is the best platform to deploy short and long-running background processing tasks with no time limit for the execution of your jobs. Common use-cases are:

• Media processing: transforming images, videos, audio recordings or PDFs directly at upload
• Web scraping and headless browser requests: fetching data from and interacting with websites that do not have any API
• Interfacing with slow or asynchronous APIs: calling slow APIs or APIs using callbacks
• Asynchronous Computer Vision and Inference: automatic content detection in photos and videos for indexing, metadata enrichment or advanced analysis
• Batch processing: running heavy computations on batches of database records or media
• Data science and report generation: analysing data and generating pre-computed reports
• Notification reception and processing from IoT devices: reacting to events generated by devices and triggering actions
• DevOps: backup, monitoring, build and deployment jobs
• And much more!

We're happy to provide 1000 hours of compute, 1TB of storage, and 5TB of bandwidth per month for free until the end of the year. All the function executions are currently powered by 1GB of RAM and 1 vCPU. Sign up now and start deploying serverless functions!

Continuous Deployment of Python and Node.js functions using Git

One of the recurring requests we got was the ability to deploy your own code on Koyeb. We get it: you need to be able to inject your business logic and pair it with our ready-to-use integrations to build your application faster and better.

When looking for an efficient way to let you manage your stack functions and modifications, we decided to go with the best practice for code and infrastructure management: version everything. We're glad to share that this new release brings a native integration with git and GitHub to seamlessly integrate Koyeb with your development workflows.

Integrating Koyeb into your development environment is a two-step process:

1. Add a koyeb.yaml file in your repository describing your stack configuration. Stacks can now be deployed with a simple YAML syntax which should look familiar. For instance, to deploy a Python 3.8 function with the handler entrypoint in the hello_world package, your koyeb.yaml will look like this:

functions:
  - name: hello-world
    runtime: python3.8
    handler: hello_world.handler

Fork our Hello World in Python on GitHub to see a simple example in action. You can deploy Python and Node.js functions with the same syntax.

2. Connect your GitHub repository to Koyeb.

Now each time you git push, we will build and deploy your code!

For Python and Node.js functions, we take care of the complete build process using standard dependency management tools. If you want to read more about deploying code functions, check out the documentation.

Native support of Docker containers

After looking into the serverless space, we found that serverless solutions were fragmented into two separate generations of products to solve the same problem: containers and code functions. Our research shows that a lot of developers and companies try to use code functions but end up migrating to a container service due to runtime limitations.

We want you to be able to process your data with the technology you know and love, so we decided to provide a unified solution to deploy your applications.

Containers can be deployed with the same simple YAML syntax as functions. For instance, to deploy the koyeb/cowsay container from the Docker Hub, you simply need three lines of configuration:

functions:
  - name: hello-koyeb
    image: koyeb/cowsay

The Koyeb Stacks work in a unified way for containers and code functions. The deployment of containers also integrates with git and lets you benefit from native versioning.

Event-driven processing

The Koyeb Serverless Engine is completely event-driven, allowing seamless integration with various sources and native autoscaling. The platform not only provides strong integration with events coming from our Object Storage gateway, it also lets you invoke your functions using events respecting the CloudEvent specification.

The event system is designed to be powerful with easy filtering of incoming events using the Common Expression Language. Here is a simple example, triggering a container dumping the incoming event with jq each time an event is received on the Koyeb Object Storage gateway:

functions:
  - name: display-koyeb-event
    image: stedolan/jq
    args: ['.', '/koyeb/events/in/raw']
    events:
      - cloudevent:
          expression: event.source == "koyeb.com/gateway"

One of the most challenging parts of serverless technologies is troubleshooting. We decided to provide essential observability features and event tracing as part of the core platform. All stacks have an audit log with all the events received and which function they triggered. The event content is highly accessible, so you can easily understand your functions' executions and failures.

Object Storage API and data-processing

As part of the Koyeb Platform, we provide an S3-compliant object Storage API to store your data. You can use a Koyeb Managed Store or connect your own account cloud service provider. We're happy to share that we already support major cloud service providers including GCP Storage and AWS S3.

We also have an impressive list of cloud service providers in preview: Azure Blob, Wasabi Storage, Backblaze B2, DigitalOcean Spaces, StackPath Object Storage, and Scaleway Object Storage.

Our Serverless Compute Engine is designed to seamlessly integrate with our Object Storage API. You can easily interact with your Stores from your Koyeb Stack functions and access your data with no effort.

When you do so, each function execution will get short-lived credentials in the environment to access your data store and prevent credentials leakage.

Here is an example of a function using the stores with our secret management engine to fetch the content of an object. The object to fetch and bucket location are automatically provided in the incoming event:

import boto3
import os

def handler(event, context):
		obj_name = event["object"]["key"]
		store_name = event["bucket"]["name"]
		boto_session = boto3.Session(region_name=os.environ[f"KOYEB_STORE_{store_name}_REGION"])
    store_client = boto_session.resource(
        "s3",
        aws_access_key_id=os.environ[f"KOYEB_STORE_{store_name}_ACCESS_KEY"],
        aws_secret_access_key=os.environ[f"KOYEB_STORE_{store_name}_SECRET_KEY"],
        endpoint_url=os.environ[f"KOYEB_STORE_{store_name}_ENDPOINT"],
    )
    obj = store_client.Object(obj_key).get()
    content = obj["Body"].read()
    # Add your own processing logic!

Our S3-Compliant object storage API can now also be used as a standalone solution to benefit from a unified API wherever your data is stored.

Serverless: Autoscaling and High-Availability

One of the core benefits of the Koyeb serverless engine is that autoscaling and high availability are provided by design.

On the availability side, you don't need to worry about dealing with failures of the underlying infrastructure, we take care of automatically provisioning your functions on a new server in case of a failure.

On the scaling side, we automatically increase the number of containers according to the number of incoming events. Free accounts have a default scaling limit of 10 to prevent abuse, contact us if you need to scale more!

New Open-Source Catalog

Our function catalog has been completely refreshed with ready-to-use integrations which are now completely open-source: github.com/koyeb-community.

It's easy to combine ready-to-use functions with your own code in Stacks. For instance, to use the image-resize function from the catalog, simply add to your koyeb.yaml:

functions:
  - name: image-resize
    use: image-resize@1.0.0
    with:
      STORE: your-store
      IMAGE_RESIZE_WIDTH: 150

All catalog functions can be easily forked, modified to your needs and deployed thanks to the GitHub integration.

What is next?

This post extensively covers all of the platform's new features. If you want to read about complete examples, head to our new tutorials section where we cover complete end-to-end use cases:

• How to build an application with automatic labeling and indexation of media using Koyeb, AWS Rekognition and Algolia
• How to automatically watermark images uploaded to a Backblaze B2 bucket
• How to process DigitalOcean Spaces images to generate thumbnails

Some of you already spotted some of the new features under development in our documentation: cron to schedule recurring jobs, HTTP event sources, and our CLI are all under construction and scheduled for a release in the coming weeks!

We're happy to provide 1000 hours of compute, 1TB of storage, and 5TB of bandwidth per month for free until the end of the year! Sign up now ;)

As always, we're available through our support channel, our community platform or through our integrated instant messaging system if you have a question or want to share feedback.

We're extremely grateful for all the amazing support we have received from our early users.

Thank you for your trust.

Yann - Co-Founder and CEO