Resource limits: memory, CPU, and exit code 137

One container can starve the whole machine

Lesson 1-3 established that containers share the host's kernel, and this unit has to face a consequence of that: by default a container may use all of the machine's memory and CPU. One app with a memory leak can push the host into thrashing and take down every other container on it, including the database.

Platforms therefore require declared limits, and every serious compose file or deploy config carries them.

docker run -d --memory 512m --cpus 1.5 myapp:v1

The asymmetry has a physical reason. The kernel can simply schedule a greedy container less often, but memory cannot be politely taken back from a process, so the kernel's OOM killer, short for out-of-memory killer, terminates it instead.

Reading the symptoms

An OOM-killed container leaves a specific fingerprint, visible in docker ps -a:

STATUS: Exited (137) 2 minutes ago

137 is worth memorizing. Unix reports killed by signal N as exit code 128 plus N, and the OOM killer uses SIGKILL, signal 9, so the total is 137.

When a container dies repeatedly with 137 and docker logs ends mid-sentence with no error message, that is a memory limit being hit rather than a bug in the final log line.

Live usage per container comes from one command:

docker stats

It shows a live table of each container's memory and CPU against its limits. In compose, limits go under the service:

  app:
    build: .
    deploy:
      resources:
        limits:
          memory: 512M

With restart: unless-stopped from lesson 9-2, an OOM-killed app restarts automatically, which keeps the service up while the leak is hunted.

The arithmetic behind 137

Two lines of shell arithmetic make the encoding concrete, since Unix reports killed by signal N as 128 plus N and the OOM killer sends signal 9.

echo "OOM-killed container exit code: $((128 + 9))"
echo "Exited (137) means signal $((137 - 128)), which is SIGKILL"

Output

OOM-killed container exit code: 137
Exited (137) means signal 9, which is SIGKILL

Subtracting 128 turns any of these mystery numbers into a named signal, which is the fastest triage step available when a container will not stay up.

Diagnosing a repeating Exited (137)

A container dying every few minutes with Exited (137) and logs that simply stop most likely exceeded its memory limit, so the kernel's OOM killer terminated it with SIGKILL, giving 128 plus 9 equal to 137.

137 is the SIGKILL fingerprint, and SIGKILL gives the process no chance to log a goodbye, which is why the logs end mid-flight.

A crash from a code bug would normally print a stack trace and use a different exit code, which is what makes this diagnosis reliable. The fixes are raising the limit if it is genuinely too low, or finding the leak with docker stats.

The signal number hiding in 137

The signal is 9, which is 137 minus 128.

SIGKILL is signal 9, the unconditional kill a process cannot catch or clean up after, so 128 plus 9 gives 137. It is the same kill -9 from the terminal course.

The same rule decodes other exits, so 143 is 128 plus 15, SIGTERM, the polite shutdown that docker stop sends first. Reading exit codes this way turns mystery deaths into diagnoses.