Run & Scale
Log Exporter

Log Exporter

Koyeb's log exporter (opens in a new tab) is a Docker image you can use to forward logs to external log management endpoints. The log exporter works by forwarding logs from your Services' standard output and error streams.

The log exporter images use Vector (opens in a new tab) to collect logs, format messages, and deliver them to the appropriate target.

Koyeb logging

Koyeb automatically collects log lines that your programs write to the standard output (opens in a new tab) and error (opens in a new tab) streams. These are collected and made available both in the control panel (opens in a new tab) and with the CLI (opens in a new tab).

Application output is useful for monitoring process' health and troubleshooting various issues. While Koyeb's logging interface is adequate for many types of debugging, it can be helpful to use more sophisticated log management tools to track changes over time, aggregate logs from multiple services, and visualize changes in behavior.

Deploying the log exporter

Deploy the log exporter as you would a normal Service, through the Koyeb control panel (opens in a new tab) or CLI. The specific configuration depends on where you want to send logs. In all cases, you must select the worker Service type during the deployment configuration process.

Preconfigured export locations

The log exporter images include dedicated functionality for sending logs to the following locations without a Vector sink configuration file:

For other targets, read the custom export locations section.

To export your logs to an Elasticsearch (opens in a new tab) instance, create a service using the docker.io/koyeb/log-exporter image with the <VERSION>-elastic tag:

docker.io/koyeb/log-exporter:<VERSION>-elastic

You can optionally click the Deploy to Koyeb button below to configure the log exporter to ship to an Elasticsearch instance with much of the configuration pre-filled. Update the environment variables to correspond with your values and choose any other customizations before deploying:

Deploy to Koyeb (opens in a new tab)

Currently, only basic authentication is supported.

During the deployment, set the following environment variables to configure forwarding to Elasticsearch:

  • KOYEB_SERVICE: The name or ID of the Koyeb Service to export logs for. This takes the form of either <APPNAME>/<SERVICE_NAME> or <SERVICE_ID>.
  • KOYEB_TOKEN: A Koyeb access token generated in the API section of your Koyeb organization settings (opens in a new tab).
  • ELASTICSEARCH_ENDPOINT: The HTTPS URI where your Elasticsearch instance accepts incoming messages.
  • ELASTICSEARCH_USER: The Elasticsearch username.
  • ELASTICSEARCH_PASSWORD: The Elasticsearch password. We recommend storing this as a Secret.

Additional resources:

Custom export locations

To send your logs to an alternative Vector sink (opens in a new tab), create a service using the docker.io/koyeb/log-exporter image with a generic <VERSION> tag:

docker.io/koyeb/log-exporter:<VERSION>

During the deployment, set the following basic environment variables:

Additionally, you need to set one or more additional environment variables to provide the configuration file for the target sink. The Vector sink configuration file must use the pipe input (inputs = ["pipe"]) in order to correctly pull and forward the logs from your Service.

You can pass configuration file to the log exporter image in a few different ways.

One way to pass a Vector sink configuration is to set an environment variable to a Vector configuration file reformatted as an inline string.

You can optionally click the Deploy to Koyeb button below to configure the log exporter to ship to a custom location using an inline Vector configuration. Much of the Koyeb deployment configuration is pre-filled. Update the environment variables to correspond with your values and choose any other customizations before deploying:

Deploy to Koyeb (opens in a new tab)

To configure the deployment, you should replace the newline characters in your file with \n characters.

The environment variable should use the following format:

SINK_<FORMAT>_<FILE_NAME>

The variable begins with SINK_ and has the following components:

  • <FORMAT>: Specifies the format of the sink configuration file and can be one of JSON, YAML, or TOML.
  • <FILE_NAME>: The name the configuration will be given in the Vector configuration directory, without the extension.

For example, assume that you want to use sink configuration file called sink-http.toml with the following contents:

sink-http.toml
[sinks.hook]
type = "http"
inputs = ["pipe"]
encoding.codec = "json"
uri = "https://my.endpoint.org/logs"
method = "post"

You could use a tool like awk to replace the newline characters with \n by typing:

awk '{printf "%s\\n",$0} END {print ""}' sink-http.toml

A similar alternative using sed (compatible with both Linux and macOS versions) would be:

sed -e ':a' -e 'N' -e '$!ba' -e 's/\n/\\n/g' sink-http.toml

You would then assign the string to the variable SINK_TOML_SINK-HTTP:

SINK_TOML_SINK-HTTP='[sinks.hook]\ntype = "http"\ninputs = ["pipe"]\nencoding.codec = "json"\nuri = "https://my.endpoint.org/logs"\nmethod = "post"\n'

The log exporter image will reconstruct the file from the environment variable and place it in the Vector configuration directory.

Troubleshooting

If you are experiencing issues while using the log exporter, set the DEBUG environment variable to any value to turn on debugging:

DEBUG=1

This will make the log exporter print out each message it processes as it runs. You can see these messages using the control panel (opens in a new tab) or with the CLI (opens in a new tab).

MetricsDatabases