GitOps Only Controls What You Write Down

In the last week of September I moved every app I run to GitOps. Each pipeline now builds an image and writes its digest into a deploy repo. A controller in the cluster polls that repo every 30 seconds and applies it.

The move took a week and took this site down for two days. Three things broke. None of them showed up in the deploy repo, because each one lived somewhere the manifests didn't specify.

1. The pipeline wrote a bad image digest

What happened. After pushing an image, the pipeline has to read its digest back and write it into the manifest. Getting that right took eight commits in 33 minutes on a Thursday night:

• docker inspect returned a digest the registry didn't serve.
• The registry API returned the right digest only with the OCI index Accept header.
• The builder's layer storage got corrupted, so I pruned before every build. Then two concurrent prunes collided.
• The night before, the build containers had timed out on startup. That morning they ran out of disk.

At one point a failed build produced no digest, and the next step wrote an empty one into a manifest. That app stayed up only because Kubernetes keeps the old pod running when the new one can't start. A restart would have taken it down.

Fix. Build with docker buildx build --push. Read the digest from the registry's manifest endpoint with OCI Accept headers. Fail the job unless the value is sha256: plus 64 hex characters.

Rule. The controller applies whatever the repo says. Validate every value before the pipeline writes it.

2. A manifest without a namespace deployed to the wrong place

What happened. fieldmappings.com returned a 502 on every page for about two days. The site's manifest didn't set metadata.namespace. Before the move, the pipeline applied it into default. After the move, the controller applied it, and the controller runs in its own namespace. The Deployment and its PersistentVolumeClaim landed there, so the Service the tunnel points at had no pods behind it.

The AI agent that builds these apps with me guessed the namespace first. A controller log from another app seemed to rule that out, so it moved on to the image, the Service name and the secrets. I found the cause from the cluster side.

Fix. Set namespace: default on every resource.

Rule. Any field a manifest leaves out gets a default from whoever applies it. Change who applies it and you change the defaults. Set the namespace explicitly on every resource.

3. Moving namespaces left every app with an empty volume

What happened. On October 4 I moved every app from default into a dedicated apps namespace, so the controller only needs permissions there. Every resource set its namespace explicitly. Every site came back up.

A PersistentVolumeClaim belongs to one namespace. Changing the namespace in the manifest doesn't move the claim. The cluster created new, empty claims in apps and left the old ones in default, unattached. Every app with a volume in that commit came up with an empty one.

Nothing alerted. A week later the agent's Monday analytics check showed 36 views for the last 30 days, against 292 the week before. That comparison is the only reason anyone noticed.

Fix. For the invoicing app, I restored the database from a backup. For analytics, nothing: a month of page views on a small blog isn't worth recovering, so I let it go. To move data without a backup: scale the app to zero, run a one-off pod that mounts both claims, copy the data, then scale back up.

Rule. Before changing a namespace, a claim name or a storage class, list every PVC the change touches, back up the ones that matter, and decide what to migrate. git revert restores manifests, not data.

Checklist before moving to GitOps

• Set namespace on every resource.
• Pin images by digest. Validate the digest in the pipeline before writing it.
• Monitor the build runner on its own: disk, cache, startup time. The deploy repo can't see it.
• List every PVC before changing namespaces or claim names. Back up what matters and migrate on purpose.
• After a deploy, check the numbers that should only go up: rows, events, page views. Don't count on someone remembering last week's figure.

Was it worth it

Yes. The deploy repo is the most accurate record I've had of what runs on my own hardware. Every change has a commit, a time and an author, and both the agent and I can read it.

The repo controls what it describes. It doesn't describe the build runner, the defaults of whoever applies it, or the data on disk. Cover those yourself.