Troubleshooting Deployments

Wondering why your app doesn't respond to the public URL or your Service is in an abnormal state? Below you will find a few common errors and failures that might happen along with steps you can take mitigate them.

When you investigate failures, pay attention to the state of the Instances composing your Service. When you deploy a Service, Koyeb will start one or more Instances depending on your configuration. All of these Instances will be health checked before routing production traffic to them.

Unhealthy Instances

Instances are labeled unhealthy when health checks fail.

Every ten seconds, Koyeb checks all of the network ports opened by your configuration, on each Instance. If any of these ports do not respond after the Instance is booted, the Instance will be declared unhealthy.

Unresponsive network ports are usually caused by one of these problems:

Network port configuration issue: the network port you selected is not the one your Service is listening on.
Binding on localhost instead of 0.0.0.0: if your application is bound to localhost or 127.0.0.1, nobody outside of the container can reach your application. Ensure your application is bound to 0.0.0.0 to make it reachable when deployed.
Service configuration issue: your Service is not starting due to an application-specific issue like a missing environment variable or a connection issue to another Service.
Service failure: your Service might fail at startup or when it's processing requests due to a lack of memory or temporary storage.

If you don't understand why an Instance is unhealthy, check the Instance's logs.

Metrics

To investigate and diagnose performance issues, you can use Koyeb Metrics, which provide a high-level overview of your Services and comes built-in to the platform. With Koyeb Metrics, you can access your Service's CPU usage, memory usage, average response time, request throughput, and public data transfer.

A common source of performance issues is not having enough CPU or memory resources allocated to the Service. If your Service is continuously using a large amount of CPU or memory, like around 70-80%, this may be an indication that you need to scale out your Service or switch to a bigger Instance size.

You can view your Service metrics by going to the console and looking under the Metrics tab on your Service's page. Read the full Koyeb Metrics documentation to learn more.

Logs

Runtime logs allow you to troubleshoot Deployments and follow logs written by your Services on standard streams (stdout/stderr) in real time.

By default, the logs of all of the Instances composing the Service are displayed in a single log stream. For each log entry, you can see the following values:

instance_id: The Instance associated with the entry
stream: The standard stream (stdout/stderr) where the message was written
message: The message itself

For example, if "Hey there, I'm a Service running on Koyeb" was printed on stdout of your Instance named instance-131869ab, you would get a line looking like this:

Koyeb log output

instance-131869ab stdout Hey there, I'm a service running on Koyeb

The default runtime logs view shows logs for currently running Instances and older, destroyed Instances. You can filter logs per Instance.

Each time you apply a filter, we will display the last 1000 log lines available.

Limits

Koyeb logs have the following limitations:

Each instance may emit logs at a rate of 100 lines per second.
Logging rates may burst to 1000 lines.
Log lines longer than 10KB are truncated.
Logs are kept for 30 days.

The logging limitation and burst mechanic work by starting each Instance with a quota of 1000 log lines. The Instance draws from this pool for each line it wants to log. Each second, another 100 lines will be added to the pool of available lines, capped at maximum of 1000 lines.

Below are some scenarios that might help illustrate this behavior:

An Instance that logs at a rate of 50-80 lines per second. This Instance's logging never exceeds its the quota's refresh rate so it will always have the burstable pool available if it needs it.
An Instance that logs 200 lines per second. All of this Instance's logs will be recorded for the first five seconds. For each subsequent second, the first 100 logs will be transmitted using the new quota made available by the refresh, while the second 100 lines each second are dropped.
An Instance that averages 50 lines per second, but emits a burst of 800 lines every 30 seconds. The Instance's regular log rate is low enough that the refresh tops off the log pool every second. When the burst is used, the pool must be refilled using the refresh of 100 lines, minus the 50 lines that the Instance emits at its normal rate. Since the pool is topped off by the time the next burst occurs, no lines are ever dropped.

If you wish to archive your logs or lift the volume limits, consider using the Log Exporter.

Slow Service

Slow Service response time is usually related to a lack of resources or a contention for external resources like a database. Check the Service metrics to see if it is resource related.

Instance shell access

For advanced troubleshooting, you can execute commands directly on your Service's Instances.

For Docker-based deployments, the user is the user defined in your image specification. In Dockerfiles, it is specified with the USER (opens in a new tab) instruction. If no user is defined, the root user is used.

For git-driven deployments, the user is the user assigned by the buildpack. In most cases, this will be koyeb.

$ koyeb instances list
ID      	SERVICE              	STATUS   	DEPLOYMENT ID	DATACENTER	CREATED AT
1bbb7ac7	my-demo-app/main     	HEALTHY  	a6615e32     	par1      	16 Dec 21 10:49 UTC
$ koyeb instances exec 1bbb7ac7 -- netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:3000            0.0.0.0:*               LISTEN      27/node

You may also run short-lived debugging sessions and get access to a shell on the running instance.

$ koyeb instances list
ID      	SERVICE              	STATUS   	DEPLOYMENT ID	DATACENTER	CREATED AT
1bbb7ac7	my-demo-app/main     	HEALTHY  	a6615e32     	par1      	16 Dec 21 10:49 UTC
$ koyeb instances exec 1bbb7ac7 /bin/sh
/app $ ls
next.config.js  node_modules    package.json    public

Exposing your Service Health Checks