We removed the shell from our containers. Here's what broke.

Most of the images we were shipping started from a full base like node:20 or python:3.12. They worked fine, but every one of them came with bash, apt, curl and a few hundred packages the app never touched. If someone ever got code execution inside one of those containers, they had everything they needed to download tools, poke around the network and try to move somewhere else.

So I set a simple goal: production images should contain the app and its runtime, and nothing else. No shell, no package manager, no sudo.

The starting point

Before changing anything I ran our images through a scanner to get a baseline. The numbers weren’t a surprise, but they were useful for getting buy in from the team. A lot of the reported CVEs were in packages like perl, tar and openssl CLI tools that the apps didn’t use at all. They were just there because the base image had them.

Multi-stage builds do most of the work

The fix is mostly boring. Build in one stage with all the tools you want, then copy only the output into a tiny final stage.

FROM golang:1.23 AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go build -ldflags="-s -w" -o /app ./cmd/server

FROM gcr.io/distroless/static-debian12:nonroot
COPY --from=build /app /app
USER nonroot:nonroot
ENTRYPOINT ["/app"]

For Go this is easy because you get a static binary. For Node and Python I used the matching distroless runtime images and copied in only the installed dependencies, not the whole build folder.

What broke

This is the part nobody warns you about.

Health checks that called curl. A few Dockerfiles had HEALTHCHECK CMD curl -f localhost:8080/health. No curl, no health check. We moved these to Kubernetes liveness and readiness probes using httpGet, which is where they should have been anyway.

Entrypoint scripts. Some apps used a docker-entrypoint.sh to read env vars, run migrations and then start the server. With no shell, those scripts can’t run. Migrations became a separate Kubernetes Job, and the rest of the logic moved into the app itself.

Debugging. The first time something went wrong in production, someone tried kubectl exec -it pod -- sh and got an error. That was the moment the team really felt the change. The answer is ephemeral debug containers:

kubectl debug -it pod/api-7d9f -n prod --image=busybox --target=api

This attaches a temporary container with tools to the running pod, shares its process namespace, and goes away when you’re done. You get to debug without baking a shell into every image.

Timezones and CA certificates. One service failed TLS calls because the final image had no CA bundle. Another logged times in UTC when the team expected IST. Both were fixed by copying /etc/ssl/certs and the tzdata files from the build stage, or by picking a distroless variant that already has them.

Locking it down further in Kubernetes

A small image is only half of it. I also set a standard securityContext on every deployment:

securityContext:
  runAsNonRoot: true
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: true
  capabilities:
    drop: ["ALL"]

readOnlyRootFilesystem caught a couple of apps writing temp files to random places. We gave them an emptyDir volume mounted at /tmp and moved on.

To make sure nobody slips back to the old way, a Kyverno policy rejects pods that run as root or don’t drop capabilities. That check runs in the cluster and in CI, so people find out on their pull request and not during a deploy.

Was it worth it

Yes. The images are a fraction of their old size, so pulls and rollouts are faster. Scanner reports went from long lists nobody read to short ones we actually act on. And if an attacker does get in, there’s very little for them to work with.

If you’re thinking about doing this, start with one service, write down everything that breaks, and turn that list into your migration guide for the rest.