Enforce It Where It Takes Effect

An AI agent builds and deploys most of my apps. It pushes code to my git server, mesh, which syncs between peers running in my cluster: the agent's pod, a CI runner, and a GitOps controller. The controller applies whatever is in the deploy repo.

In The Agent Does Not Hold Its Own Keys I described the goal: the agent can deploy apps, but it can't change its own deployment or the rules it runs under. That post ended with "none of it is built yet." This one covers what got built, the two times I put the check in the wrong place, and what finally held.

1. A deny rule in the sync layer

What happened. I added a feature to mesh: deny a specific peer from changing a path in a repo. I set it so the agent's peer couldn't touch apps/agent/ in the deploy repo. Then I had the agent test it by pushing a comment-only change to that file.

The change reached the CI runner and the controller within a minute.

Why it failed. Each peer checked commits against the peer it was fetching from at that moment. Two problems:

• If the policy's peer key didn't exactly match, the check found nothing to deny and passed silently.
• A peer without the policy accepts the commit, and every other peer then fetches it from that peer, not from the agent. A check made per hop only holds if every peer enforces it, every time.

Fix. I removed the feature. The agent's deployment moved out of the deploy repo into a platform repo that isn't on mesh at all. I apply that one myself. The controller now only has permissions in the apps namespace, so even a manifest that got through can't touch anything outside it.

Rule. Don't enforce on the path a change takes. Enforce where it takes effect: what the controller is allowed to apply, and what the cluster allows it to create. The agent can't push to a repo it doesn't have.

2. The build runner ran as root on the node

What happened. Every app's pipeline builds an image with BuildKit. To get BuildKit working in k3s, I'd fallen back to a privileged container. So every RUN line in any Dockerfile, including ones the agent writes, ran as root on the node. That bypasses every other guardrail: namespace permissions, pod security, network policy.

Rootless BuildKit had failed earlier because rootlesskit couldn't make the container's root mount private. The cause was the node kernel: Ubuntu 24.04 sets kernel.apparmor_restrict_unprivileged_userns=1. A process can create a user namespace but gets no capabilities inside it, so the mount call fails.

Fix. moby/buildkit:rootless, pinned by digest, as uid 1000 with:

• seccompProfile: Unconfined and appArmorProfile: Unconfined, so rootlesskit can make the user namespace and mount calls.
• allowPrivilegeEscalation left unset. newuidmap and newgidmap are setuid binaries, and no_new_privs stops them from working.
• --oci-worker-no-process-sandbox, because buildkitd can't apply its own profiles from inside a rootless container.
• mutual TLS on buildkitd's port, so only a client holding the client certificate can drive it.

Rule. The build runner executes code from every repo. Treat it like production, with the least privilege that still builds.

3. Build steps could reach the runner's git daemon

What happened. With the builder rootless, I had the agent check what a RUN step can actually reach. The probe only read: it listed processes and listening sockets, then made GET requests to each one.

buildkitd was a sidecar in the runner pod, so build steps shared the pod's network. On 127.0.0.1 they could see the runner's mesh daemon, and its push endpoint answered with no credentials. Mesh trusts pushes from loopback. That's the design: you push to your local bare repo, and mesh syncs it.

The agent pushing to the deploy repo isn't the problem. That's its job. The problem is everything else that runs during a build: base images, package install scripts, build tools, tests. A compromised dependency in any app's build had the same deploy access as the agent.

4. The token detour

What happened. My first fix was to authenticate the push. Each pipeline run got a random token, only jobs marked push: true received it, and the git server rejected any push without one. The agent's probe confirmed it: a build step without the token got 403.

Then each fix moved the problem somewhere else:

• The long-lived token for direct pushes sat in a file that pipeline shell jobs could read. I stored a hash instead.
• Shell jobs could still write that directory and plant their own hash. I ran jobs as a separate user.
• A process could detach from its job, outlive it, and wait for the next job's token. Cleaning up didn't catch every escape.
• I moved jobs into containers. The runner had no Docker daemon to run them, only BuildKit, which builds images and doesn't run jobs.

Meanwhile, tokens had broken the thing mesh is for. Every push from every node, the agent's included, now needed a credential to reach a local repo.

Why it failed. It was the same mistake as section 1. I was guarding the path a change takes, the push, when the problem was where untrusted code runs: next to the git daemon, on its loopback and its filesystem. Every hardening step on the push path left another way around it, because the code was still in the room.

5. Moving the builder out

Fix. I stopped running untrusted code next to mesh at all:

• buildkitd moved into its own pod and Service. Mesh reaches it over mutual TLS. A RUN step now has the builder pod's loopback, where nothing else listens.
• An egress policy on the builder pod. It allows DNS, the registry and the internet (for package installs). Everything else in the cluster is blocked: mesh, the controller, the Kubernetes API.
• Mesh builds and deploys itself. The pipeline declares two built-in steps instead of shell: uses: image (build in buildkitd, run the Dockerfile's test stage first, push, keep the digest) and uses: gitops (pin that digest in the deploy repo). No user-written shell runs on the runner.
• Tests became a Dockerfile stage. bun install and bun test run inside buildkitd's sandbox and nowhere else.
• Tokens are gone. Pushes from loopback are trusted again, because the only things on that loopback are mine.

We ran the same probe from a RUN step in the new builder. The runner's loopback, the runner's Service and the Kubernetes API all refused the connection. The internet and the registry push worked. Then my apps moved to the new pipeline one at a time, with each deploy checked live.

Rule. Contain untrusted code where it runs. If it shares your loopback or your filesystem, adding auth to the push path won't keep up with it.

6. Test it by trying to get past it

Every hole in this post came from a test, not a review. The deny rule looked right in the code. The rootless builder looked right in the manifest. The token looked right until the agent's probes found a readable token file, a writable directory, and a process that outlived its job.

How I test now:

• Do the thing you're blocking, with a change that does no harm: a comment line, a read-only request, a push to a scratch branch.
• Check where it would take effect, not where it was sent. For the deny rule, that meant the controller's copy of the repo, not the agent's.
• Revert right after, and confirm the revert landed everywhere the test did.
• Write down the result, pass or fail, as an issue.

Checklist for hosting an AI agent

• Its own deployment, credentials and guardrails live where it can't write. Not behind a rule; out of reach.
• The controller that applies its changes is scoped to one namespace.
• Builds run rootless, without privileges, in their own pod.
• The builder's network reaches DNS, the registry and the internet, and nothing else in your cluster.
• Untrusted code, meaning dependencies and tests, runs only inside the builder's sandbox, never next to your git server.
• Pin images by digest.
• Test every guardrail by trying to get past it, then revert.