AI 25 min read

Lights Out, part 4: the lessons learned from configuring a coding agent

Dobby, briefly

Dobby is our coding agent, named after the house elf. He is a GitHub App with his own scoped identity, plus the host that runs him. When something arrives, a GitHub event, a Slack message or a scheduled routine, the host classifies it and starts a run over a pipeline graph. Every node in that graph launches Claude Code in its own Docker container, with the target repo cloned and a prompt the host assembles. The run authenticates on the seat of the developer whose work it is and shows up on their dashboard. Dobby can open draft pull requests and comment. He cannot merge, cannot deploy, and cannot touch production. Part 3 covered how that is set up and how he fills his night from a 16:30 handoff.

Part 3 also ended with a promise: the next post would be the unglamorous one. The prompts, the checklists, the failure modes, and the ways Dobby learned to game his own reviewer. This is that post, and it has a single thesis I did not hold when we started.

Deterministic, well built systems are so much better than AI that does whatever it wants. And yet AI can do things I sometimes cannot program. Not because I am dim, but because the world is chaotic and imperfect. So build a platform that gets you to 90%. Use AI for the last 9%, the part that was never possible before.

Some numbers on the platform, because they show where the effort went.

First commit 12 June 2026
Commits to date 993
TypeScript source, excluding tests ~85,000 lines
Test code ~80,000 lines across 261 test files

The test suite is almost as large as the code it tests. The rest of this post is about why.

As for gaming the reviewer, the first fix in the history is from day one. Dobby opened a PR, reviewed it himself, and his review comment contained the trigger word, which fired a webhook, which started another review of the same PR. An infinite loop of self-review before lunch. Every guard below is a descendant of that afternoon.

Six lessons follow. Each item starts with the thing that went wrong.

01. Reliability: an agent you still have to check costs more than no agent

You think the agent is working and find out hours later that nothing happened.

The first version had a thirty minute node timeout and a three hour watchdog meant to kill runaway containers. The ladder was upside down. The timeout always fired first, so the watchdog never could, and every task was silently capped at half an hour. A forty seven minute run doing real work was declared crashed mid-work, with its container still alive and still spending tokens.

Reliability is not something you get from a better prompt. It is a system layer:

  • A flat, total state machine. A run is in exactly one of eight phases: queued, acquiring, running, reply, rate limited, done, error, killed. The transition table is the whole machine and anything not in it resolves to error. In the module’s words, “there is no way to express an illegal phase move.”
  • Watchdogs with a ladder. Three hours per execution, an eight hour coarse net behind it, and a run budget on top: three hours plus three per open sub-issue, capped at twenty four. Spend is derived from the execution ledger, so a nine hour wait on a rate limit costs nothing.
  • Rescue as a node. When a container dies before a terminal node, a rescue node reads the transcript and picks retry, continue-as-outcome, or abort with a plain-words reply. The rescue itself is never rescued.
  • Cleanup every thirty seconds. Leaked run slots are reconciled, warm containers recycled before their token expires, and anything over budget is stopped in the killed phase rather than error, so the dashboard can tell “we stopped it” from “it fell over.”

The rescue node has its own story. For weeks it decided between retry and abort from a section headed “Tail of the dead container’s logs” that never contained any logs, only the one line crash string. It read the absence as evidence that nothing had been done, retried, and then told the user “no transcript or partial output survived.” The dead node’s 1.8 MB transcript was in the archive the whole time and its draft PR had been open for twenty eight minutes. Now the host hands the rescue the real tail and every PR link in it, and the prompt says: “never treat missing evidence as evidence.”

After you give an instruction you cannot tell quickly enough whether it was understood.

So the router confirms. Every event first passes a deterministic gate: zero matching pipelines means ignored, exactly one means no model is consulted, and only with several candidates does the router model pick. “The router LLM is a tiebreaker, not a gate.” Whatever it picks, an acknowledgement goes to the thread before work starts, and that persona is hemmed in. It must “confirm you’re on it and say what you’re about to do,” must “not promise specific results, fixes, or timelines,” and may never promise a new PR on a request made inside an existing one. Most early wrong promises were exactly that: Dobby cheerfully announcing a PR for a question.

An agent should only report when the work is really finished.

Only end nodes answer. Work nodes never post anywhere. They emit one structured outcome with the user-facing answer in a details field, and a reply node turns it into one short message. The basic pipeline puts it bluntly: “anything not in details never reaches the user.” The reply may only include URLs that appear verbatim in the outcome, a few sentences, no headings, no praise. A node posting its own progress is not transparency. It is noise that trains people to stop reading.

You want to see at a glance what is running, and stalled work needs a visible end state.

The dashboard is named after human work: Desk is what Dobby is doing for you now, Tasks the queue and live runs, Routines the standing schedule. It only became legible once every run had an owner, which is lesson six. And every graph has explicit exits for the ways things stop: the coder has a give-up edge, the conflict resolver has “recommended close” and “gave up” as distinct outcomes, each requiring a comment that says why and who can take over. A run that goes quiet is the most expensive failure of all, because you think it is working.

02. From 70% to usable

AI does 70% of the work with ease. If you want to rely on it, 70% is nothing. The road to usable ran through four steps, and I think the order is general.

Step 1: better prompts help until they drown in their own rules.

We prompted our way through the first month and the prompts grew. Then we measured. From a corpus of 300 delivered prompts and transcripts, someone asked of each sentence whether it had ever done anything. One warning appeared in 300 of 300 prompts, was ignored in 2, and failed in 0. Cut, 266 bytes off every prompt. Out of that came the gates for what earns a line its place: not if the model can derive it, not if it can discover it and has reason to look, and not if a check is the durable copy. The moment less text gives better output is the moment to stop prompting and start building structure.

Step 2: the agent delivers work that does not pass the checks.

Bake the ordinary developer tooling into the image so “the tests pass” means something before a reviewer looks. Dobby’s container ships Node with pnpm and yarn, Rust with rust-analyzer and clippy, .NET 10, Go, Python with ruff, Java 17 with the Android SDK and Flutter, Playwright with Chromium, and language servers for TypeScript, Go and C#. Part 2 called this the bowling lane.

Step 3: the research, code, test loop gets reinvented in every prompt.

Make the loop the host’s. Our Code Change pipeline is a graph, and the graph decides who writes, who reviews, who posts findings, and when it is done.

The Code Change pipeline graph: trigger, PR Dobby, Review Dobby, Rules Dobby, Done, with a bounded fix loop and a give-up exit. A GitHub or Slack trigger enters PR Dobby, which opens a draft PR and hands to Review Dobby. Review Dobby analyses and runs the diff and hands findings to Rules Dobby, which merges them with a rule-compliance pass and either loops back to PR Dobby for fixes, at most three times, or marks the PR ready and ends at Done. PR Dobby also has a direct give-up exit to Done for a premise that does not hold. GitHubor Slack PR Dobbyopens a DRAFT PR Review Dobbyruns the diff Rules Dobbyposts, decides Done fix issues: bug-severity only, at most 3 loops give up: premise did not hold, or genuinely stuck. Also a good ending. approve: mark ready The model chooses the payload. The host chooses the route.
The Code Change graph as committed. PR Dobby writes and opens a draft. Review Dobby analyses and actually runs the diff but posts nothing. Rules Dobby merges the findings with a rule check, posts one consolidated review, and is the only node allowed to send the PR back or mark it ready.

Three details matter more than the shape. The reviewer and the rule checker are separate nodes: the reviewer is told “Rule compliance is NOT your job” and spends ten to fifteen minutes actually building and running the code, because “static review catches the obvious mistakes; running the code catches the ones that bite.” The rule checker fans the rules out to cheap Haiku sub-agents and is told to treat everything they return as a candidate: “Haiku over-flags. Drop anything you cannot confirm.” And the loop is bounded in the graph, not the prompt: the fix-issues edge carries a loopback limit of three, and only bug-severity findings send a PR back.

There is also a gate the model cannot see. The coder prompt used to ask, in words, to sync with the base branch and watch CI after pushing. Across 265 Dobby-authored PRs, 8 needed a human to type a re-dispatch asking for a rebase. The paragraphs are gone. When the coder emits its outcome, the host reads the PR it left behind and refuses the outcome if the branch conflicts or CI is red where the base is green, at most twice per PR. A rule you can check should be a check, not a sentence.

Step 4: a new model must not reinterpret the workflow.

Under the first engine each prompt embedded a URL per possible next step, the model picked one, and nothing checked whether the transition was legal. Now the graph is prescriptive, and the module header is the design principle of the whole system:

“A dobby NEVER names a target node. It emits exactly ONE outcome label from its node’s declared set, plus a payload. The HOST owns the map outcome → edge → next node. The LLM controls the payload; the host controls the routing. Illegal transitions are impossible to EXPRESS, not merely vetoed.”

When the vendor ships a new model, the workflow does not change. Triggers, edges and end nodes belong to the engine.

The agent sometimes writes good code for the wrong problem.

Four thrown-away PRs, four different failures. The issue predated by six days the commit Dobby blamed, and the bump he reverted was the fix. Dobby found the dependency unused, wrote that in his own PR body, and opened the PR anyway. The root cause was a Terraform deploy, nothing in the code. The issue was true when filed and stale after twenty four days in the queue.

The coder prompt now has a section called “Confirm the premise before you write the diff”: does the reported behaviour still reproduce, is the code you would change still reachable, did the fix already land? And it changes what counts as finishing: “‘actionable but unnecessary’ is a valid way to finish, and the better outcome: a PR on a dead premise is reviewed and then thrown away.” The test that pins this section lists the four PRs by name.

Without log analysis you keep repairing individual runs.

All of the above came from reading transcripts in bulk. The same corpus revealed that the model alias stored on each run was not the model that answered: opus resolved to four different concrete models in three days. The host now reads the resolved model off the archived transcript. Log the runs, look for patterns, then change the system. Sometimes the change is a prompt cut, sometimes a guard test, sometimes host behaviour.

The agent quickly builds something that already exists in the codebase.

Look for the existing function first is a convention like any other, so it lives in the rule bundle next to our comment-density rule and the other fixed habits.

03. Everything an agent writes, grows

You remove content and it is quickly back.

The whole lesson is the timeline of one file, our root CLAUDE.md, the instruction file Claude Code reads at start.

Date Size What happened
3 August 115,480 → 86,988 B Cut by 24.7% under the prompt gates
20 August 195,010 B Had more than doubled in 17 days
20 August 1,514 B Replaced with a four paragraph orientation file
Since capped at 4,000 B A test fails the build if it grows

The cut held and the file doubled anyway, because the removals had no gate and the prompts had a producer: every work node was told to write what it learned into CLAUDE.md in the same PR. A knowledge base grows exactly as instructed. Find the producer. The guard test now checks both halves: the file’s size, and the absence in any assembled prompt of an instruction to write to it.

A cleanup without a ceiling is undone again.

The successor is a repo-scoped rule bundle, and every number in it is a ceiling. One rule is at most 600 bytes, because the rule checker was already distilling every rule to about 595. Thirty permanent rules that bind everywhere plus ten per repository, in two buckets, so one repo cannot crowd out the rest. A delivered bundle is at most 80,000 bytes. A temporary rule lapses after fourteen days, measured from the last time it failed, so the corpus prunes by irrelevance rather than age.

A self-learning system keeps adding knowledge.

Put a judge on the write path and a filter on the read path. A container that hits a wall no rule warned it about files the fact. That used to become a rule immediately. Now it is a request, and an adjudicator decides add, amend or discard. If the covering rule was in force in that very container, the filing is recorded as a defeat of the rule, not as a new rule. On the read path the host narrows the bundle to the repo the task is in. Once a week a codifier routine proposes changes to the permanent set. It proposes. A human decides. “There is no rule you can change yourself.”

A rule in prose is not an enforceable rule.

Make conventions testable. That is where 261 test files come from, and a fair number of them test prose: that each coder node still has the premise section, that Dobby posts no review on his own PRs, that the orientation file stays small. The people editing prompts are fallible producers too.

A test on individual files misses what the agent actually receives.

The “one fact, one home” test exists because six facts were found stated two or three times inside a single delivered prompt, and nobody had noticed because each copy was the only one in its own file. The test rebuilds the assembled prompt per node and counts. A field can be correct everywhere and still arrive wrong.

A green guard can still catch nothing.

Run a new guard on old history. That test’s header records how it was validated: run it over the tree before the cleanup and every fact reports 2 or 3. If a guard never goes red on the history that motivated it, you do not know whether it guards anything.

A convention test that is too broad turns open PRs red for no reason.

Our dashboard lets someone author a pipeline in the UI, which pushes a commit straight to main. A test over every graph would red-CI every open PR the moment that happened. So the prose guards are scoped to the two committed coder nodes by id. Host-injected text can be tested broadly. Hand-written prompt prose wants a narrower check.

You build a subsystem for work the framework can already do.

Ask first whether the work fits in a graph. A separate campaign subsystem for multi-issue work was deleted in July, superseded by a graph with a chooser node. Rescue was a subsystem with a hardcoded retry count and became a node with a budget on an edge. Reply was two hand-written builders and became one node with three modes. Each new entrance for the same intention had saved a few words of typing and cost a parallel implementation.

04. Delivering context

The agent lacks the information to give the right answer.

The target repo is pre-cloned before the prompt arrives. Rules are scoped per repo before delivery. For UI work the coder must find and work from the repo’s design source and “never introduce a new palette, font stack, radius/shadow language, or icon set into a repo that already has one.” If a decision is not derivable from the source, ask in the PR instead of inventing.

A big job does not fit in one context.

Split it into a superissue and sub-issues. The superissue pipeline has a chooser node that writes no code. Each cycle it lists the sub-issues in order, works out from GitHub which are done, picks the first that is not, and hands it to a coder that opens a stacked draft PR. It is told to “re-derive all progress from GitHub every cycle, so the flow is restart-proof and survives human edits.” Every cycle starts with a fresh context and a fresh review budget.

Even within a run the context window can fill up.

Measure context use and put a threshold on it. Stay well under the window and you rely less on compaction. A node that needs the whole window is a node that should be two. The measurement side also paid for itself: with per-machine sections moved out of the system prompt so the cached prefix is identical across the fleet, about three quarters of every prompt is a cache read.

The same facts end up in several prompts.

Treat prompt text as a bundle. Each fact gets one place, one owner, and one check on the final prompt. The Context section of every node is composed by the host from facts about the walk. Every host-built section has a declared byte budget in one table, and the host-built part of a work node’s prompt is held at 13,000 bytes. A section that wants to grow has to take bytes from a named neighbour. The container talks back through one small CLI, one subcommand per callback, built from the same contract module the host uses.

05. Attention is scarce

The agent quickly does too much and pushes work into review.

Use an open-PR budget per person: three open Dobby PRs coupled to you as reviewer, editable on your own Desk. Work a human asked for always runs, even over budget: “6/3 is honest: they asked for six.” Work Dobby started on his own must find room or not start, and is parked with a comment on the issue when it cannot. The nightly budget-fill routine runs at 03:15 and opens with the principle: “Finishing beats starting: if nobody has room, you do nothing.” A full night is a success.

The agent produces reviews nobody reads.

Measure whether people react. Over 266 Dobby PRs the loop had posted 559 reviews and 349 inline threads. Three ever drew a human reply. So Dobby no longer posts reviews on his own PRs. The loop still runs, but the rules node writes one line into the PR body instead, plus an “Attention:” line per finding it could not fix. If nobody uses the output, stop producing the output. I should add that this was a hand measurement written into a test header. Nothing continuously measures whether a comment was read, and we would like that.

Every nit costs a human reading time and typing.

The first count was 113 inline comments on human PRs and zero GitHub suggestion blocks. After a first fix, 52 of 456 carried one, so about 88% still arrived as prose, “at the cost of a 45-minute container run per nit they wanted.” Hence the current rule: on a person’s PR a suggestion is required, not optional, “on every finding whose fix is a concrete edit to lines already in the diff — the ones the reader accepts in one click.” Beside a suggestion the comment is one line. No Haiku sub-agent may author one, because a suggestion is a one-click patch to somebody’s branch.

Someone on holiday should not be disturbed.

Make holiday a fact on the user: one flag with four readers. Handoffs to that person are accepted automatically, the reviewer picker skips them, the scheduled DMs stop, and colleagues see them as away. The picker’s comment says why: “A review request is a claim on somebody’s WEEK, not on their seat, and parking one on a person who is provably not there is how a PR comes to sit for a fortnight with nothing wrong with it.” An admin may set the flag, logged with both names and DM’d to the person.

One confession. The first vacation mode did something else: the token pool tried an absent colleague’s Claude credential first, so Alice’s seat ran Bob’s work while Alice was on a beach. Efficient, and also one person’s credential doing another’s work, which our provider’s terms rule out and which makes every audit trail a lie. It is gone, guarded by a test that stops it coming back.

PR descriptions that are too long are not read.

Keep the body short. We did not end up with a byte limit, because the body is also where a finding outside the diff lands and a cap would have deleted findings rather than shortened prose. Instead the review body is one line, with an example in the prompt: “Checked 6 changed files against 9 rules; npm test passed (312/312); 1 blocking finding and 2 nits inline.” The findings are the inline comments. The same goes for every message Dobby sends: the reply is a few sentences, and the evening planning DM is at most four suggestions in under twelve lines.

06. Boundaries, cost and identity

An agent with freedom can do irreversible damage.

The container never merges or deploys. A standing rule in every work container reads “Always create DRAFT pull requests, never merge directly,” and there is no merge command anywhere in the graphs, routines or host. Only the rules node flips a PR to ready, only on approve, and it holds the PR in draft if it references a security advisory. The App has no permission to write workflow files, so deploying is structurally out of reach. Every launch also drops all capabilities, forbids new privileges, and runs as an unprivileged user with a process, memory and CPU limit. People decide what lands.

A prompt-injected agent can send a token anywhere.

Enforce egress outside the container. The egress module opens with the threat: a container with permissions skipped “launches with Docker’s default bridge network: unrestricted outbound reach to the entire internet. That is the real exfiltration lever for a prompt-injected agent.” Docker has no domain allowlist, so enforcement is a proxy that only forwards to allowlisted hosts, GitHub, the model API, the npm registry and our own product domains, plus a dedicated network whose firewall makes the proxy the only route out. The comment says the thing people skip: the allowlist only helps once direct egress is actually closed at the network layer. Honesty requires one more sentence: that mode is a deployment switch and the default is still open. The exception you leave is the one that gets used.

Security problems are only found when someone looks.

A routine looks every night at 00:45 with about forty minutes of budget, auditing one repository per night on a rotating cursor so a single night cannot flood the org. Before it may dispatch work it has to ask the host who owns the repo and whether they have room, and otherwise skip. The same reflex bounds every routine that creates work: the weekly rule codifier may have at most ten advisories pending, and the overnight plan dispatch caps at five items a night. The comment on that constant is this post in one line: “an instruction in a prompt is a request, and this one guards how many containers a single evening’s chat can spend.”

The best model is often too expensive for every node.

Model and effort are settable per node and per routine. Work nodes run Sonnet at high effort. The rule checker fans out to Haiku sub-agents and embeds the rules and diff slice in each prompt rather than letting them read the repo, because that “costs ~30× more.” The nightly routines that decide direction run on Opus. Every container’s spend goes to an append-only ledger with owner, requested model, answering model and tokens. We count tokens, not money, and that was enough to find the biggest saving: cache writes were 3.1% of tokens but 27.9% of cost, because the headless CLI takes the one hour cache window at a doubled write price while pipeline nodes run back to back. One environment variable moved every container to the five minute window.

Work runs on a credential without a clear owner.

Decide the seat before work starts. A seat is one person’s Claude credential, and the rule is one credential, one owner: “there is no round-robin and no cross-user failover, so no run can ever be served by another person’s key.” Every trigger has a caller, a human, Dobby himself, or the scheduler, and that decides both whose seat the work spends and whether it counts against a budget. Ownership decides whose seat, never how autonomous the work is.

Two decisions are worth stealing. First, the container names the work and the host names the owner, “so identity never travels as text a container wrote.” An earlier version let bot-authored text carry owner markers, until someone noticed that a Dependabot PR body quotes an upstream changelog verbatim, so an instruction hidden in one would have been an instruction we followed. Second, ownership has to run all the way down. The overnight plan dispatch used to be a routine on one named human’s seat, and when that seat was busy or rate limited, nobody’s plan ran and nobody noticed until morning. One credential had become a single point of failure for five people’s night. Now the host sweeps the stored plans between 22:00 and 06:00 and starts each item on the seat of the developer who planned it. Warm pools, rate limits, routines and dashboards all key on the same question: whose work is this.

The 90, the 9 and the 1

I expected to spend three months on prompts. We spent them on a state machine, watchdogs, a rescue path, a rule store with ceilings, a handoff gate, a PR budget, a seat model, and eighty thousand lines of tests, a good share of which test sentences. Almost none of that is AI. It is the deterministic platform that gets you to 90%, and it is what makes the agent trustworthy enough to leave alone overnight.

The 9% is real and it is the reason to bother. Reading a diff and deciding whether the premise still holds. Resolving a merge conflict so the result is what both sides meant. Turning a developer’s last three days into a concrete suggestion for tonight. Judging whether a filed rule is new, a rewording, or a defeat. I could not have programmed any of that.

The last 1% has not moved since part 3. It is a human clicking merge.

Every guard in this post marks a place where we caught the agent being confidently, plausibly wrong, and decided the fix was not a better sentence but a system that made the wrong thing impossible to express. Old school stays king. The agent is the new part. The engineering is the old part. The old part is what makes the new part work.


Sources & tools

This post draws on the Dobby codebase itself: pipeline graphs, engine modules, guard tests and commit history, June to September 2026. Quoted passages are code comments, test headers, commit messages or node prompts from that repository.

The agent runtime

GitHub mechanics the pipelines rely on

Baked-in skills and standards

Containment

The callbacks

  • Lights Out, part 1: the insane adoption of agentic coding, the accountability question.
  • Lights Out, part 2: old school stays king, the bowling lane and the sandbox this post’s guards sit inside.
  • Lights Out, part 3: running the nightshift, Dobby’s identity, triggers and the 16:30 handoff.