Old container images can break TLS

A container can look healthy and still be old.

The app starts. The pod is ready. The dashboard is green.

Then the app tries to call another service and TLS fails.

The error might look like this:

x509: certificate signed by unknown authority

Or this:

SSL certificate problem: unable to get local issuer certificate

Or this, if it is a Java app:

PKIX path building failed

When this happens, it is easy to blame the other service.

Maybe their certificate changed. Maybe the load balancer is wrong. Maybe DNS is pointing to the wrong place.

Those things are worth checking.

But there is another simple question we should ask early:

How old is the container image?

Why the image matters

A container image is a snapshot.

When we build it, we freeze the operating system packages, libraries and certificates inside it.

If we do not rebuild the image, those things do not update by themselves.

That includes the CA certificates.

CA certificates help the app decide which certificate authorities to trust. If the CA bundle inside the image is old, the app may reject a certificate that is actually valid.

That is why the same API call can work from your laptop but fail from an old container.

Your laptop may have a newer trust store.

The container may not.

What changes over time

Certificates do not stay the same forever.

Root certificates expire. Certificate chains change. New trusted roots get added. Old ones get removed. Internal platforms may also move from one private CA to another.

The world outside the container keeps changing.

The old container image stays the same.

That is the main problem.

Immutability is useful because it gives us repeatable builds. But immutability does not mean the image stays safe or current. It only means the image stays unchanged.

Sometimes unchanged is the risk.

A useful example

The Let’s Encrypt DST Root CA X3 expiry in 2021 is a good example.

Many modern systems handled the change because they had updated trust stores. Some older clients and devices failed because they did not have the right trust material.

A stale container can behave like one of those older clients.

It may reject a certificate chain that a fresh system accepts.

So the problem is not always the remote certificate. Sometimes the problem is our old image.

Pinning does not solve this on its own

Pinning a base image by digest can be useful.

It helps us know exactly what we built from. That is good for repeatability, audit evidence and supply chain control.

But pinning also creates a responsibility.

If we pin to a digest and nobody updates it, the service stays tied to that old base image. It will not get newer packages or newer CA certificates until someone updates the pin and rebuilds the image.

So the question is not only: should we pin images?

The better question is: who owns the refresh?

If nobody owns it, the image will quietly age.

What I would check first

If a container starts throwing TLS errors, I would check these before changing application code:

1. When was the image last rebuilt?
2. Did the build pull the latest base image, or did it reuse a cached layer?
3. Is the base image pinned to a digest?
4. If it is pinned, when was that digest last reviewed?
5. Is ca-certificates installed in the image?
6. Does the runtime use the operating system trust store?
7. Does the runtime have its own certificate store?
8. Are private CA certificates involved?
9. How are those private CA certificates added or rotated?
10. Did the external certificate chain change recently?

These checks are simple, but they keep the investigation logical.

Start with what changed outside the app. Then check whether the image is too old to understand that change.

What not to do

Do not make “turn off TLS verification” the fix.

In curl, that usually means using -k. In application code, it may be called “disable certificate validation” or “trust all certificates”.

That can be useful for a quick test, but it is not a proper fix.

If the real problem is an old CA bundle, turning verification off only hides the issue. The service may start working again, but now it cannot properly prove who it is talking to.

That turns a maintenance problem into a security problem.

The solution

The solution is to treat container images as things that need regular maintenance.

We should not build an image once and forget it.

We should rebuild it. Refresh the base image. Update the CA certificates. Test the TLS calls that matter.

Containers did not remove operating system maintenance. They moved it into the build and deployment process.

If the rebuild does not happen, the maintenance does not happen.

Action plan

This is the routine I would use:

1. Rebuild production images on a schedule, even when the application code has not changed.
2. Make the pipeline pull fresh base images. For Docker builds, use --pull where it makes sense.
3. If images are pinned by digest, review and update those digests regularly.
4. Scan images for old packages, known vulnerabilities and stale base layers.
5. Add a small smoke test for important outbound TLS calls.
6. Document how each runtime handles certificates. Java, Go, Python, Node.js and .NET can behave differently.
7. Document how private CA certificates are added and rotated.
8. Give the image refresh routine an owner.

That is the practical fix.

Not panic when TLS breaks.

Not disabling certificate checks and hoping for the best.

Just a clear routine: rebuild the image, refresh the base, check the trust store and test the important TLS paths.

Notes I checked while writing

Docker describes images as immutable snapshots and recommends rebuilding images regularly with updated dependencies. Their build guidance also recommends using --pull to check for fresh base images during builds.

Docker’s guidance on digest pinning is useful too. Pinning improves repeatability, but it can also opt you out of automatic base image updates unless you have a process to update the digest.

The Debian update-ca-certificates man page explains how the system updates /etc/ssl/certs and generates the combined ca-certificates.crt bundle.

The Let’s Encrypt DST Root CA X3 expiry is a useful reminder that trust chains change, and older clients or systems without updated trust stores can fail even when the server certificate is valid.