An Agent Built My Kubernetes Lab Alone — and Found Four Gaps in the Runbook

I gave Claude Code a goal and one constraint, then left it alone to see whether it could stand up a three-node kubeadm lab on its own. It could. Four corrections came out of the runbook along the way — but what exposed them was the harness design, not the agent.

What this post covers

The question this run was built to answer — can an agent carry a "build the lab" goal all the way to done on its own — and the conditions that made the answer hold
The run itself: a three-node kubeadm runbook executed on AWS, and the four corrections it produced
Why those four defects and not others, and why human review structurally cannot catch this class

TL;DR

The experiment was about autonomy, not productivity — hand Claude Code the goal "stand up a three-node cluster" and see whether it reaches done. It reached done.
Runbook executed end to end on a Terraform + Claude Code harness on EC2. Under 60 minutes, about $0.19 of EC2, four corrections to the document.
All four defects are omissions — nothing in the document was wrong, something was missing. conntrack, Calico 179/tcp, Calico 5473/tcp, and the scp source file in §7.
179 and 5473 are Calico's own ports and appear nowhere on the upstream Kubernetes ports page. The agent identified both. The 5473 case in particular showed up as one node out of three stuck at 0/1 — a symptom that reads like a broken machine — and it did not take that bait.
What exposed the defects, though, was the harness design rather than the agent — the security group was derived exclusively from the runbook's port table, which turned omissions into failures and gave the agent something to diagnose.
Autonomy held here because verification was cheap. Open a port and a probe answers within 30 seconds — where the gap between hypothesis and confirmation is short, autonomy is safe.

Background — what an unverified runbook actually is

I keep field notes as markdown in a repository. The trouble is that these documents are not binary. They do not sit at "correct" or "wrong" — they sit somewhere between "it worked on my machine" and "it works in anyone's hands."

A runbook is less like a recipe and more like uncompiled code. The syntax can look fine and you still cannot tell whether there is a link error until you run it — and a runbook's link errors are almost always "the thing that was already installed on my machine." The author's environment quietly fills in the document's blanks, and those blanks leave no trace in the document.

So I changed the method. Instead of reviewing the document by reading it, I built the environment from the document alone and executed it as written.

The harness — compiling the document into firewall rules

Terraform stands up three t3.medium instances (2 vCPU / 4 GB / 40 GB, Ubuntu 24.04.4) in one flat 10.10.10.0/24 subnet. So far, an ordinary lab.

The part that matters comes next. The security group rules were derived exclusively from the runbook's "Ports" table. Only ports in the table get opened. Nothing gets added because it is customary, and nothing gets added because it would be convenient.

That is the whole trick. A generously configured lab will never surface an omission — as long as the environment is more permissive than the document, there is no way to learn what the document left out. The moment the security group is made dependent on the table, a blank in the table becomes a cluster outage.

This is where the agent earns its place. Building this harness by hand — writing the Terraform, translating the port table into rules, bootstrapping, destroying and rebuilding after every failure — is half a day of work. That half day is why verification gets postponed, and postponement is how documents go stale.

Two things came from me. The goal — stand up a three-node cluster and verify it with a workload. And the constraint — derive the security group from the runbook's port table and nothing else. Everything below that, Claude Code took.

Harness construction — translate the port table into security group rules, write the Terraform for the VPC, subnet, and three instances.
Execution — apply §1 through §8 in order across all three nodes, collecting the output of each verification command.
Diagnosis and recovery — narrow down the cause behind preflight errors and 0/1 states, amend the rules, confirm again.

Measurements — one hour, four corrections

Recorded on the 2026-08-08 run.

Item	Value
Total elapsed	Under 60 minutes (the runbook's `duration` field says 60–90)
Harness	3× `t3.medium` (2 vCPU / 4 GB / 40 GB), Ubuntu 24.04.4 LTS, single `10.10.10.0/24`
Versions installed	Kubernetes 1.31.14 (kubeadm) · containerd 2.2.1 · Calico 3.28.2
Recovery from missing `conntrack`	About 10 minutes
Recovery after opening 179/tcp	One probe interval (30 s) — two of three nodes flipped to `1/1`
Worker reboot recovery	SSH back within a minute, `Ready` shortly after
EC2 cost	About $0.19 (us-east-1 on-demand $0.0416/hr × 3 × 1.5 hr, EBS excluded)

The missing conntrack cost ten minutes, and not because it was hard to diagnose. The error message is explicit: [ERROR FileExisting-conntrack]: conntrack not found in system path. Most of the cost comes from where the error lands — it arrives only after §1.5 has finished, so all three nodes get fully packaged before kubeadm init refuses.

179/tcp went the other way. It looked like it would take a long time and did not. Opening the port flipped two nodes to 1/1 within a single 30-second probe interval, and that immediacy was the confirmation. When hypothesis and refutation are 30 seconds apart, diagnosis stops being inference and becomes experiment.

The reboot check produced one surprise. The pod on the rebooted worker does not get rescheduled elsewhere. It sits as Unknown on that node for a couple of minutes and is then restarted in place by the returning kubelet — new pod IP, RESTARTS at 1. The service kept answering 200 throughout on the surviving replicas. Watch kubectl get deployment web returning to 3/3, not the pod list.

Something is missing from the numbers too. There is no per-phase breakdown. Nothing in the logs says how much of the 60 minutes went to terraform apply and image pulls, so this post cannot tell you whether "one hour" means an hour of human involvement or an hour that was mostly waiting. The next run gets per-step timestamps.

Version drift — surfaced, but not a defect

One more thing came out of the run. The runbook was drafted against containerd 1.7, and as of 2026-08-08 the Ubuntu 24.04 package was containerd 2.2.1. That is a major-version jump — the generated config becomes version = 3, and the CRI plugin key moves from io.containerd.grpc.v1.cri to io.containerd.cri.v1.runtime.

The runbook's sed was unaffected. The default is still SystemdCgroup = false and the line still matches. So this never registered as a failure and is not one of the four corrections.

The silence is the problem. Any config snippet you find online that references the old plugin path will not apply, and the runbook was not telling readers that. It now carries a version check.

containerd --version
head -1 /etc/containerd/config.toml    # expect: version = 3

Defect verification and drift detection are different jobs. The harness asks whether something fails. Drift makes a document stale without ever failing. This one got caught because it happened to be visible during the run — which is luck, not design.

The shape of the defects — not wrong, missing

The four corrections, in the order they surfaced.

1. conntrack was absent and kubeadm init would not even start. A fatal preflight error, not a warning. The kubelet package declares iptables, kubernetes-cni, mount, util-linux, and libc6 as dependencies, and conntrack is not among them. A full Ubuntu server install usually happens to have it, which is how this gap survived; a minimal or cloud image does not. §1.5 now installs it alongside socat — which is only a warning at init time, but whose absence resurfaces much later as a broken kubectl port-forward.

2. 179/tcp was closed and every calico-node sat at 0/1. The readiness probe is -bird-ready -felix-ready, and BIRD needs BGP port 179. The upstream Kubernetes ports page does not list 179, because it is Calico's. The runbook's port table was built from that page, so 179 was not there either. Opening it flipped two of three nodes to 1/1 within one 30-second probe interval, which is what confirmed the cause.

3. The remaining node was 5473/tcp. Its confd could not reach Typha, so it never wrote bird.cfg, so BIRD never started — the log reads bird: Unable to open configuration file /etc/calico/confd/config/bird.cfg. The operator runs two Typha replicas on a three-node cluster. The two nodes hosting one reached it over loopback and recovered the moment 179 opened; the one without a replica did not.

Both of those diagnoses came from the agent. It went from the probe flags to BIRD, from BIRD to BGP 179, and on to the fact that 179 was absent from the security group; then, for the last node, back down through a missing bird.cfg to confd to Typha to 5473. Neither port is in the upstream documentation, so neither could be read out of the runbook — both have to be derived backwards from how the components behave.

4. The scp in §7 failed on the first attempt with Permission denied. /etc/kubernetes/admin.conf is 0600 root:root and SSH lands as an unprivileged user. §7 now copies the user-owned duplicate that §2.1 already creates.

Line the four up and one thing is common to all of them. None was caused by the document being wrong. Every command is correct and the order is correct. An item was simply absent.

Omissions are the defect class human review structurally cannot catch. A reviewer reads what is written; nobody can read what is not there. And all four were things the author's environment had been supplying — conntrack was already on my machine, my firewall was more permissive than the table, and I had run §2.1 before §7. The environment of the person writing a document is always richer than the document.

Green failure — a cluster that looks finished

Two of the four (179 and 5473) exposed a second problem on top of the first. The verification checklist itself was inadequate.

Here is what a cluster with 179 blocked looks like. All three nodes Ready. Every pod Running. Workloads scheduling normally. Pods get addresses and talk to each other over VXLAN. Both kubectl get nodes and kubectl get pods -A are green. And calico-node sits at 0/1 permanently — the DaemonSet never converges, and no future rollout of it can complete.

A checklist item reading "one calico-node present per node" passes this state. So the item became "one per node, each at 1/1", with kubectl -n calico-system rollout status ds/calico-node added alongside it.

The 5473 case is nastier. Because Typha runs fewer replicas than there are nodes, the symptom appears on one node out of three. That does not read like a firewall problem — it reads like a broken machine, and the instinct is to go debug that node. A missing port disguising itself as a partial node failure is a pattern worth filing away.

Autonomy — what actually made this run work

The original question was not "does this get faster" but "does it get all the way there alone." It did. I supplied a goal and a constraint and stayed out of the rest, and a cluster came up with four runbook corrections behind it.

Stopping there would make this a review of how capable the agent is. Three conditions are what actually made it work.

The goal was self-adjudicating. "Stand up a three-node cluster" is a completion state you can ask the cluster about. kubectl get nodes, pod scheduling, a service call returning 200, a DNS lookup — the runbook already carried a verification checklist, and the agent used it as an answer key. Had adjudication required a human, autonomous execution would not have held.

The feedback loop was short. Open 179 and a probe answers within 30 seconds. When every hypothesis comes with a 30-second refutation attached, diagnosis becomes experiment rather than inference. Work without that property — results that take a day, or results that are probabilistic — will not get the same value out of the same autonomy.

The blast radius was contained. A dedicated VPC, three instances, and a state that disappears with one terraform destroy. Nothing gets hurt if the agent mangles a security group. Autonomy was affordable here not because I trust the agent but because the radius was small.

And here is the part I am most careful about: the agent should not get credit for exposing the defects. The reasoning that pinned down 179 and 5473 is unambiguously the agent's. But that reasoning could only begin because a 0/1 failure existed, and that failure was manufactured by the constraint that tied the security group to the runbook. Put the same agent in a permissive lab and the cluster comes up fine, the runbook gets a passing stamp, and 179 stays missing.

What the agent did was not "find defects" — it was "trace an engineered failure back to its cause." Engineering the failure is still the human's job.

Platform engineering view — cheaper verification changes what a document is

The bottleneck in runbook verification was never execution. It was the cost of repeatedly deriving a clean environment from the document. Typing the commands was always about an hour of work; the half day went to standing the environment up and rebuilding it from scratch after every failure.

When that cost drops to an hour, three things follow.

First, the verification date becomes a real property of the document. This note's frontmatter carries verified: 2026-08-08. When verification is expensive that field is decoration — you fill it in once, never again, and it becomes a lie as time passes. When verification is cheap it starts working as an expiry date.

Second, the document gets promoted to an environment specification. Deriving the security group from the port table turns the document from the thing being verified into the thing verifying. If the port table is wrong the cluster does not come up, so the table cannot stay wrong. Part of the runbook effectively becomes code — and this is where it should stop, in my view. Move the whole runbook into code and you just have IaC, and the value of a document a person reads and understands is gone. What belongs in code is the tables that state environmental premises, not the procedure itself.

Third, the engineer's remaining surface shifts toward goals and constraints. Two sentences in this run were mine: what counts as done, and what gets derived from the document alone. The other 60 minutes belonged to the agent. Bolt an agent onto an unchanged verification design and nothing comes out, so the quality of those two sentences determines the whole result. The more execution the tooling absorbs, the larger the share that design carries — a pattern that repeats at this layer too.

Caveats

This was verified on EC2, not on bare metal. The runbook targets on-prem three-node hardware and the harness is cloud. Four things went unverified because of that gap — disabling swap (the cloud image has none, so §1.2 is a no-op), duplicate product_uuid on cloned templates (EC2 never produces duplicates), ufw (inactive on the AMI, so port rules were verified as security-group rules instead), and real NIC and driver behaviour. The note states the same.
Single run, no control group. "One hour" is one record, not a number placed beside the time the same verification would take by hand. Nothing here supports a claim about a productivity multiple.
Whether these four defects are specific to this runbook or general to kubeadm runbooks is unknown. The sample is one document. The conntrack and Calico port omissions derive from the structure of the upstream documentation and so look reproducible, but I have not confirmed that.
The harness design determines which defect class you find. Deriving the firewall from the document produced port omissions. Deriving the package list would have produced different ones. This method does not find "all defects" — only defects on the axis the harness made dependent on the document.
There is no per-phase timing. Only the 60-minute total, with no record of how much went to provisioning and image pulls. Reading "one hour" as human working time overstates it.
The $0.19 is calculated, not billed. It uses the us-east-1 on-demand rate and excludes EBS and data transfer. A different region gives a different number.
"Completed autonomously" is a claim about this goal, this constraint, and this blast radius. Completion could be adjudicated by the cluster itself, refutation took 30 seconds, and failure ended at terraform destroy. Do not carry the conclusion into work missing any of the three.
The agent's diagnostic path was confirmed by outcome, not audited as reasoning. That 179 and 5473 were the right answers is something I learned by opening the ports. How many wrong turns preceded them is not in this post.