Build Once, Promote Twice: Why Rebuilding per Environment Bites You

Staging passed and production failed forty minutes later. Same commit, same branch, same pipeline template. The error was a missing native dependency, and the reason it existed in one environment and not the other was that we had built the image twice.

Nothing had tampered with the code. The second build had simply resolved a floating tag to a newer patch of a base image, and the two builds were no longer the same artifact.

The commit matched. The artifact did not.

Every rebuild is a fresh roll of the dice against everything outside your repository: base image tags, package registries, transitive dependencies, build tool versions, the clock, the locale, the region of the runner.

# two builds of one commit, two different outputs
FROM node:20            # resolves today to 20.19.4
RUN npm ci              # lockfile pins direct deps, not the registry's TLS stack

When environment A builds at 09:00 and environment B builds at 14:30, you are testing a hypothesis about reproducibility that no build system can actually guarantee. The pipeline reports identical commit SHAs, and the SHAs are telling the truth. The bytes inside the images are not.

One artifact, configuration at the edge

Build once. Push an image, a jar, a wheel, whatever your unit of deployment is. Promote that exact digest through every environment until it reaches production or gets deleted.

IMAGE="registry.internal/api:$GIT_SHA"
docker build -t "$IMAGE" .
docker push "$IMAGE"
# staging and production both reference $IMAGE by digest, never by tag

Tags move. Digests do not. Pinning by digest means the promotion path is a record of which immutable object went where, which is also the record you want during an incident.

Configuration moves to runtime: environment variables, a mounted config file, a secrets manager, a feature flag. Anything that differs between staging and production stops being a build input and becomes a deploy input, where it can be changed without producing a new binary.

What has to be environment-agnostic

This only works if the artifact does not secretly encode its destination. Three things usually break it.

Build-time paths. A compiled binary that writes to /home/ci/app works on one runner and fails on another.

Bundled config. Anything baked in at build time, including a config file copied in with COPY, becomes a reason to rebuild.

Environment checks at start. Code that expects a hostname present only in one cluster will boot in staging and crash in production, or the reverse.

The test for each is simple: could this decision have been made without knowing which environment it is for? If yes, move it to runtime.

Where teams get stuck

The honest difficulty is not technical. It is that rebuilding is easy and promotion needs infrastructure: a registry with retention rules, a pipeline stage that gates on tests rather than on compiling, and a way to roll a digest forward without editing YAML by hand.

Feature flags are the other half. When a build ships dark, the same artifact can sit in production with behaviour switched off, which means the thing you tested in staging is literally the bytes serving traffic. That is the whole argument in deploying on Friday with flags.

Progressive delivery then controls exposure rather than content: canary the same digest, watch it, widen it. The strategies that make sense depend on the platform, and the tradeoffs are laid out in Kubernetes deployment strategies.

# promotion as a digest reference, not a rebuild
spec:
  template:
    spec:
      containers:
        - name: api
          image: registry.internal/api@sha256:9f2c...   # the artifact that passed staging

The cache angle

If you are worried this makes builds slower, the opposite is true. You build once per change instead of once per environment, so the total work drops. The rest of the speed comes from not rebuilding what did not change, which is the caching and layer ordering problem in build caching and parallelism.

# dependency layer first so a source change reuses the install
COPY package.json package-lock.json .
RUN npm ci
COPY . .

Summary

Rebuilding per environment gives two different artifacts one commit message, and the difference shows up in production rather than in review. Build once, push by digest, promote the same digest everywhere, and move every environment-specific decision to runtime where it belongs. Flags decide what is visible, progressive delivery decides how much traffic sees it, and the artifact itself stops changing the moment testing ends.