Container Logs: stdout, Files, and the Rotation You Forgot

The alert said the API was crash-looping. The application had not crashed. It had written debug logging to /var/log/app.log inside its own writable layer, filled the container filesystem, and failed the next time it tried to write anything at all, including its own health check.

Nothing had rotated the file, because nothing knew it existed. The platform collects stdout. The application was not using stdout.

The contract your platform already has

Every container runtime has an opinion about logs, and it is the same opinion: the process writes to file descriptor 1, and the runtime takes it from there.

# what the runtime expects
CMD ["node", "server.js"]        # inherits stdout/stderr
# NOT:
CMD ["sh", "-c", "node server.js >> /var/log/app.log 2>&1"]

That second line is the one that causes the incident. Redirecting to a file inside the container means the log is now stored on the writable layer, which is bounded by the storage driver, and nobody (not the orchestrator, not docker logs, not your aggregation pipeline) can see it.

Writing to stdout is not a style preference but how the runtime learns that a log line exists.

Where rotation actually happens

With stdout, rotation is the runtime's problem, and runtimes solve it.

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "5"
  }
}

Docker writes stdout to a JSON file on the host, caps it at five rotated copies, and docker logs reads the active one. Kubernetes does the same per container with kubelet, using containerLogMaxSize and containerLogMaxFiles. In both cases the rotation is outside the container, so filling it cannot take the container down.

The important property is that the cap applies to the runtime's file, not the application's. Your process can emit a gigabyte an hour and the container filesystem stays the same size.

If you must write a file

Some software genuinely cannot be convinced to log to stdout: certain vendor agents, some legacy binaries, a JVM library that opens its own handler. In that case the file needs three things:

# sidecar pattern: ship and truncate, never let it grow
volumeMounts:
  - name: app-logs
    mountPath: /var/log/app
containers:
  - name: log-shipper
    image: fluent/fluent-bit
    volumeMounts:
      - name: app-logs
        mountPath: /var/log/app
        readOnly: true
  - name: app
    # ...and the app writes here

A shipped file plus a rotate strategy, or a file that is written, read, and truncated before it can accumulate. The failure mode you are designing against is not "logs are hard to find"; it is "the disk filled and the process died," which presents as an application bug to everyone who was not watching disk usage.

The debug trap

The reason this recurs is that file logging is convenient during development. On your laptop, >> app.log is immediate and greppable. It ships to production in a refactor, and the container filesystem is the wrong place to discover that.

The guard is a build-time check rather than a code review rule, because code review does not reliably catch a redirect buried in a shell wrapper:

# fail the build if an image will write logs into its own filesystem
docker run --rm --read-only --tmpfs /tmp "$IMAGE" sh -c 'true' \
  && echo "runs with a read-only rootfs"

Running the image with --read-only in CI is the single most effective test: the same posture container runtime hardening argues for at the layer below. Anything the application tries to write fails immediately, at build time, rather than at 2am on a box nobody is watching.

Separating the two questions

It helps to name them distinctly, because teams conflate them and then solve the wrong one.

Availability of logs is a stdout question. If the process dies, can its last words still be read? Stdout handled by the runtime gives you that for free; a file inside a dead container does not.

Volume of logs is a rotation question, answered by caps on the runtime's file or by truncation on the path you cannot avoid.

Neither is answered by giving the application a bigger disk. A larger filesystem just delays the same failure and makes the eventual incident harder to attribute.

Summary

Log to stdout and let the runtime own rotation: it caps the file outside the container, so no amount of output can fill your filesystem. Where a binary insists on a file, ship it and truncate it, and run the image with --read-only in CI so a stray write fails at build time rather than as a mystery crash loop. The platform already has a log contract; the incidents come from opting out of it without noticing.