Registry Rate Limits: How Pull Throttling Breaks Builds You Thought Were Fine

The pipeline failed at 04:12 with a message none of us had seen before: toomanyrequests: You have reached your pull rate limit. The retry failed the same way. The image had pulled cleanly the night before, and nothing in the repository had changed.

What changed was the hour. A merge queue released twelve jobs at once. Each started from an empty runner. Each asked the registry for a base image.

Where the limit actually counts

Rate limits are counted by the registry, against an account or an anonymous IP range, over a rolling window. Anonymous pulls are what everyone hits first: a small allowance per IP per several hours.

In CI the detail that matters is caching. A runner with a warm image never contacts the registry; an ephemeral one has no cache, so every job pays a full request. The limit counts pulls rather than unique images, so twelve jobs spending twelve requests works exactly as designed and still throttles you.

bash

# what the registry sees on a fresh runner
docker pull python:3.12-slim   # one request
# what it sees on a warm runner
docker pull python:3.12-slim   # zero: resolved locally

Mirrors and pull-through caches

Two patterns fix most of this, and they solve different problems.

A pull-through cache is a registry you operate that proxies the upstream. Point it at the upstream once and every later request is served locally.

yaml

# registry config.yml
proxy:
  remoteurl: https://registry-1.docker.io

The first pull of library/alpine walks upstream; the twelve that follow hit your own host. Cost: your registry is now a dependency of every build, and it needs disk, eviction policy, and monitoring like any other stateful service.

A mirrored copy is stronger. Sync the handful of base images you depend on into your own registry on a schedule, then reference them by your own hostname. Builds stop touching upstream entirely, which also removes upstream outages from your failure modes.

Both move the trust boundary, which is the part teams skip: the image now comes from a host you operate, so scanning and digest pinning has to supply the attestation upstream used to provide.

Authenticate, and keep the secret out of the image

Logging in raises the quota from an IP allowance to an account allowance, and it is usually enough on its own for a small team. In Kubernetes the credential lives as a pull secret attached to a service account:

yaml

apiVersion: v1
kind: Secret
metadata:
  name: registry-creds
type: kubernetes.io/dockerconfigjson

The failure I see most often is the shortcut: docker login baked into a Dockerfile or a runner startup script, with credentials written to ~/.docker/config.json inside an image that then gets pushed. The rate limit problem gets solved and a much larger one gets created. Credentials belong to the cluster or the runner, never to the artifact.

Failing fast instead of retrying into a wall

Retries are what turn a throttle into an outage. A job that backs off three times spends more quota. It also hides the real message and delays everything queued behind it.

bash

# one explicit attempt, clear message, no hidden backoff loop
docker pull "$BASE_IMAGE" || {
  echo "registry throttled or image missing: $BASE_IMAGE"
  exit 1
}

On a shared self-hosted runner, one preflight pull before the matrix fans out warms the cache for every job that follows. That single line removes the thundering herd, and it belongs next to the rest of the pipeline tuning in build caching and parallelism.

The structural version of the fix

If pulling dominates your build time, the pull is not the real problem. A pipeline that reuses a base layer cache and builds only what changed spends far fewer pulls for the same output.

dockerfile

# dependency layer first: the base pull and package install are the stable part
FROM python:3.12-slim
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .

Ordering the stable layers early means a source change reuses everything above it, including the part that had to come from a registry.

Summary

Rate limits are counted per pull, not per image, and ephemeral runners make every job a fresh one. Put a cache or a mirror in front of your base images, authenticate so the quota belongs to an account, and keep credentials out of anything you build. Fail once with a clear message, warm the cache before the matrix starts, and remember that when you become your own registry you are also responsible for proving the image is the one you expected.