The Orchestrator’s Tax

The Incident Was Not Actually About Parallelism

4 subagents had been launched in a single wave. Three had clear runtimes:
roughly twelve minutes, 5 and a half minutes, and 7 minutes. A fourth
was nonetheless operating. Checked out a method, that already justified the delegation.
Three duties ran concurrently, so wall-clock time was round twelve minutes
as an alternative of one thing nearer to 25 if the work had been
serialized.

However velocity was the seen byproduct, not the fascinating half. What I used to be
looking for was the price, and my first intuition was that it needed to be the
duplicated effort of delegation itself: each subagent studying recordsdata,
reconstructing context, understanding the duty independently.

That was not the place the largest shock turned up.

At one level in the course of the work, the orchestrator steered checking on the
operating brokers. It was a small, throwaway immediate: “test on the brokers.” I
adopted it. As an alternative of a light-weight abstract, the device it used pulled again
the complete uncooked transcript of a background agent: tens of hundreds of tokens of
JSONL, intermediate reasoning, and gear output, imported wholesale into the
most important thread. Then it occurred once more, for a second standing test.

There is a crucial caveat right here. The declare that this polling behaviour
price greater than the duplication tax of 4 brokers was the orchestrator grading
its personal mistake. I did not have actual per-call token accounting, so deal with that
rating because the orchestrator’s account, not a measured truth. The reliable
half is narrower: the transcript dumps have been actual, the wall-clock timings have been
actual, and the status-check path clearly launched a big, avoidable price.
Whether or not it was the one largest price stays a speculation till the tooling
can instrument it correctly.

What mattered extra was that I would discovered a price I hadn’t been in search of.

The Prices Have been Not All of the Identical

As soon as I ended treating the session as one lump “subagent price,” the
image broke into items that did not belong collectively.

First, two of the 4 subagents have been working in the identical space of the
response pipeline. Completely different duties, completely different recordsdata, however each needed to
perceive the identical structure, the identical testing conventions, and far of
the identical surrounding code earlier than both might start. Every paid that
orientation price independently. That is not an argument in opposition to delegation.
It is an argument that the work had been cut up too finely.

Second, one agent ran git stash and git stash pop whereas sibling brokers
have been writing elsewhere in the identical tree. Nothing broke, however the danger was
structural, as a result of repository-wide operations are completely affordable in a
single-threaded session and turn out to be a lot more durable to justify the second a number of
writers are lively without delay.

By now I had a rising listing of culprits: standing polling, duplicated
orientation, unsafe git operations. I discovered myself attempting to rank them. Which
one price essentially the most? The longer I attempted to reply that, the much less satisfied I
grew to become that rating them was the correct query in any respect.

The Scarce Useful resource Is the Orchestrator’s Working Reminiscence

The transcript-polling incident stored bothering me for a purpose that had
nothing to do with token price. A token invoice is one-time, you pay it and it is
over. What occurred right here was completely different. The uncooked transcript stayed within the
orchestrator’s context after the device name accomplished, and each flip after
that carried it ahead, whether or not or not it was nonetheless helpful.

That was the second I spotted I had been treating two very completely different
sorts of price as if they have been the identical. Tokens are spent as soon as. Context
shapes each determination that follows. I wasn’t merely token
consumption anymore. I used to be trying on the high quality of the orchestrator’s
working reminiscence.

That realization was carrying two separate concepts, and I wish to pull them aside
reasonably than allow them to blur collectively. The primary is what I simply described.
Air pollution left in context taxes each later flip. The second is just not about
operating out of area in any respect. The extra that is sitting in context, competing for
consideration, the more durable it will get for a mannequin to select what issues proper now,
even with loads of room nonetheless free. A much bigger context window would not repair that.
It simply provides the noise extra room to pile up earlier than anybody notices.

Context home windows are solely going to get greater, that is a given. What issues
is just not how a lot room there’s, however how a lot of what is sitting in that room is
well worth the mannequin’s consideration. That is the actual downside subagents want to unravel,
in the event that they’re used proper.

Seen this manner, the orchestrator is the one a part of the system that
accumulates understanding throughout an extended session. It remembers why a design
determination was made, carries ahead architectural constraints, and is aware of which
trade-offs have already been mentioned. The subagents do not, and that is by
design. They’re imagined to be disposable. Exploration, repeated file reads,
failed approaches, and noisy intermediate reasoning are supposed to keep in
employee contexts and by no means make the journey again to the primary thread.

Cognitive Locality Modifications What Parallelism Is For

This reframes the duplicated-orientation downside, which wasn’t actually “two
brokers studying the identical recordsdata.” It was two brokers independently reconstructing
the identical psychological mannequin of the codebase, as a result of the work had been partitioned
by process reasonably than by the data every process required. I’ve began calling
that distinction cognitive locality: Duties that want the identical psychological mannequin ought to normally keep collectively.
Splitting them simply forces a number of brokers to rebuild the identical understanding
from scratch.

Parallelism nonetheless issues right here, it is simply not the primary level. Operating 4
brokers concurrently is beneficial, however bizarre. The actual profit is that they
maintain noisy intermediate reasoning out of the primary thread and return solely what
it nonetheless wants. That is the isolation subagents are supposed to supply, and it
solely holds if the primary thread respects it.

My working perception now: that is what subagents are literally for. Not that
they save time, however that they allow you to offload reasoning the orchestrator
would not want to carry onto, so it has much less to hold and fewer competing for its
consideration. Get the isolation proper, maintain issues native by cognitive locality,
and subagents turn out to be the device that protects the orchestrator’s working reminiscence,
not only a price you tolerate for parallelism. That is a perception, although, not a
measurement. What I’ve truly measured is the opposite facet of it, the price of
getting the isolation flawed.

Turning a Session into Standing Guidelines

My subsequent transfer was the plain one. Encode the lesson into CLAUDE.md, the
standing instruction file each session hundreds.

It could have been straightforward to jot down a big corrective coverage right here, however each
additional line in a standing instruction file is a price paid once more on each future
session. So I compressed the repair into the smallest algorithm that addressed
the failures I would seen. Each is absolutely answering the identical query. Does
this piece of data, or this manner the work is cut up, earn a spot within the
orchestrator’s context?

1. Want two to 4 brokers in a single wave. If the orchestrator needs 5 or
   extra, it ought to first ask whether or not duties sharing recordsdata or conventions must be
   merged.
2. Don’t ballot background brokers for standing when the reply will be given
   from what’s already identified. Don’t fetch a full transcript to reply a
   light-weight query.
3. Don’t enable repository-wide git operations inside concurrent agent
   prompts.
4. Deal with overlapping file possession as a consolidation sign, not a cue to
   spawn extra brokers.

None of those inform the orchestrator precisely what to do in each case. Every
one provides it one thing to test or ask itself earlier than performing, not a script to
run, and none of them is profound by itself. Their solely actual worth is that
they’re all aimed on the similar factor, retaining disposable reasoning disposable
and retaining room within the orchestrator’s context for what it wants later within the
session.

The Subsequent Mistake Would Have Been Extra Governance

A later session surfaced a unique hole. I had began the orchestrator
with specific abilities for the sort of work I wished, coding steerage in a single
case, design steerage in one other. I assumed that after a ability was lively in
the primary thread, spawned subagents would comply with it mechanically. They do not.
A subagent would not inherit abilities lively within the father or mother session except the
orchestrator passes them alongside explicitly.

My first intuition was so as to add a confirm-before-spawn gate. The orchestrator
would cease, listing which brokers it needs to launch and which abilities every ought to
load, and watch for my approval.

I am glad I did not maintain that model. It solved the flawed downside. I did not
have proof that dangerous spawn plans have been slipping via for lack of a
affirmation step. I would found a lacking truth about ability propagation, and
that is a unique sort of hole. A common affirmation gate would have added
a round-trip to each related session, and earlier than lengthy I would virtually definitely
have began approving these prompts on autopilot.

At that time, I spotted I wasn’t actually bettering governance. I used to be
simply including one other ritual.

And it nonetheless would not have caught the actual downside from earlier than, the
orchestrator polluting its personal context.

The narrower repair held up higher. Earlier than spawning, the orchestrator states
which lively abilities are related to every agent’s process and factors the subagent
on the ability file to load, reasonably than pasting the entire ability inline.
Affirmation is just required above the identical batch-size threshold already in
place, or when file possession is ambiguous.

That left me with a heuristic I now use greater than the rule itself: Earlier than
including a line to a standing instruction file, ask whether or not a fairly
competent orchestrator would make the correct determination as soon as it knew the one
lacking truth.

If sure, the rule ought to simply state the actual fact. If the repair begins specifying a
determination process, corresponding to approvals, checkpoints, necessary steps, that is
normally an indication I am encoding course of the place a small clarification would have
achieved the job.

I do not know but whether or not that heuristic survives more durable instances. For now it
stops me from turning each fascinating incident right into a miniature
paperwork.

The place This Leaves Me

I haven’t got a settled view of how a lot governance is sufficient, and I do not
suppose this piece earns one.

What I’ve as an alternative is a small flywheel, with a human nonetheless firmly within the
center of it. A session exposes a spot. Somebody has to note that it felt
flawed, cease the work lengthy sufficient to examine it, resolve whether or not the issue is
actual or simply noise, and choose what deserves to turn out to be a standing rule. The
orchestrator can grade its personal session and floor clues, because it did right here, however
it can not make that judgment name itself. The selection of what to codify, what
to depart alone, and what is likely to be an overreaction remains to be mine. The subsequent
session then tells me whether or not that judgment improved the work or simply created
a unique sort of waste.

The artifact of this spherical is the present model of
my CLAUDE.md.
It is not a completed prescription. It is the state of the calibration after this
iteration. The thresholds in it, two to 4 brokers per wave, 5 as a
consolidation sign, match the work I used to be doing once I wrote them. I would not
current them as something like common constants, and I would be suspicious of
any orchestration write-up that did. They have been additionally calibrated in opposition to Claude Sonnet 5,
and I have not examined how they maintain up in opposition to others. A distinct
mannequin would possibly moderately want a unique steadiness. A couple of extra guidelines have accrued
within the file since. Deal with this file as a pattern, not a template, modify and optimize
it for no matter mannequin and workflow you are truly operating. What issues is not the
particular file, it is the behavior behind it: noticing a failure, asking what it truly
price, and writing the rule that might have caught it.

For years we optimized software program methods round CPU, reminiscence, and throughput.
The primary wave of LLM tooling taught us to look at tokens. This session made me
suspect there is a third factor value watching in long-running agent workflows:
the standard of the orchestrator’s personal working reminiscence, the one useful resource that,
as soon as polluted, retains charging lease for the remainder of the session. I do not suppose
that is a settled regulation but. It is a sample that held up within the periods I’ve
checked out up to now.

The tax I went in search of was by no means on the subagents. It was on the
orchestrator, in what it selected to hold ahead. That is the query I carry
into each multi-agent design now: not what number of brokers to run, however what earns
a spot within the orchestrator’s context.

The questions I am left with are genuinely open:

• How do I measure this correctly, as an alternative of counting on the orchestrator’s personal
  account of its errors?
• When ought to a lacking truth go into the directions, and when does that flip
  into an excessive amount of course of?
• What is the subsequent orchestration mistake I am not seeing but?

I anticipate I am going to revise the file once more. That feels much less like a failure of
foresight and extra like the traditional price of working with a system opaque sufficient
that doubt itself turns into a part of the strategy.