The demo movies all the time finish on the similar second. A figma body turns right into a passing take a look at in twelve minutes. Somebody within the room says the phrase “productiveness.” The recording stops.
The components that come after that second are the components I truly get paged about. Who owns the ticket the agent opened at 3:14 a.m.? Which mannequin name produced the assertion in take a look at case 47? What closes the 17 draft tickets a caught run left behind earlier than the subsequent dash planning notices them? None of that reveals up within the demo. All of it reveals up on the on-call rotation. After 20 years of main take a look at automation throughout consumer-scale platforms, I’ve a robust bias about which slide within the deck predicts whether or not a pipeline ships or stalls. It’s by no means the structure slide. It’s the runbook.
This piece is in regards to the runbook. I constructed an unattended agentic take a look at pipeline over the Mannequin Context Protocol — a five-agent SDLC (product supervisor, QA engineer, automation engineer, developer, pull-request reviewer) coordinating by means of MCP servers for Jira, Figma, Confluence, TestRail and GitHub, with hosted Claude because the orchestration mannequin and an open-weights Hermes-3 as a validation baseline — and I ran it as an impartial analysis challenge lengthy sufficient to be taught which manufacturing constraints the agent literature glosses over. What follows is the brief record of issues I now insist on earlier than I let any agentic pipeline contact a shared system.
Composition contracts, or why the agent lied to itself
The costliest failure mode I’ve seen in a multi-agent pipeline will not be the mannequin getting a solution unsuitable. It’s agent A getting a solution proper and agent B misreading it.
I watched this occur inside a run final month. The necessities agent produced a clear output with a discipline referred to as acceptance_criteria that was a listing of strings. The ticket agent anticipated acceptance_criteria to be a single markdown blob. Neither agent was unsuitable in isolation. Each had been passing their very own unit checks. What the pipeline produced was a Jira ticket whose acceptance standards learn, in full, the 4 characters [” — the JSON serialization of an empty first element. The test-plan agent read that, generated one test, marked it green and moved on. It took me until the pull-request stage to notice.
This is a composition fault. It is the natural failure mode of any system where independent components hand off structured data across a boundary they don’t jointly own. Distributed systems people have written about this for years — Martin Fowler’s tolerant reader pattern is the classical treatment — but the LLM literature mostly still treats it as a prompt problem. It isn’t. You cannot fix it with a better system prompt. You fix it with a typed handoff contract that both agents agree to before they run, plus a validator between them that fails loud when the shape drifts.
The heuristic I now use: Every agent-to-agent boundary gets a schema and a validator. Every agent gets a golden-input regression suite that catches contract drift before it reaches a downstream agent. Neither of those is glamorous. Both are the difference between a pipeline that produces work and a pipeline that produces a very expensive game of telephone.
If you want to know whether a team’s agentic pipeline is going to survive the first quarter, ask them what the contract is between agent one and agent two. If the answer is “the model figures it out,” budget for the cleanup.
Provenance, or the audit trail nobody wrote
The second thing I insist on now is that every artifact the pipeline produces has to answer three questions without a human doing archaeology: which agent produced it, which model call produced it and which upstream inputs the agent was looking at when it did.
This sounds like a nice-to-have. It is not.
Somewhere around week three of running the pipeline I hit a subtle case: the requirements agent was quoting a Confluence page back to itself. The MCP server for Confluence had returned an empty result on a first call, the agent had written a placeholder requirement, the second call had succeeded, the agent had retrieved its own placeholder and by the third pass it was citing that placeholder as source truth. The ticket said, entirely in earnest, that the requirement came from the design owner. It hadn’t come from anywhere. It had come from itself, four minutes ago.
You cannot debug that class of failure with logs of what the model said. You need logs of what the model was looking at. That means capturing the tool-call ID for every MCP call, stamping every artifact with the set of tool-call IDs it derived from and refusing to accept any retrieved fact into a downstream stage that cannot be traced to a real external source. I call it the prove-the-source rule. It’s boring. It’s a two-line requirement in the runbook. It is also the single guardrail that has saved me the most on-call time.
There is a nice side effect. When editors, reviewers or auditors ask where a decision came from, the pipeline can show them. That matters if you work anywhere near a regulated stack, and increasingly it matters everywhere else — the AI transparency and provenance framing in NIST’s AI Risk Management Framework is going to become the default expectation faster than most teams are budgeting for.
Cleanup and ownership, or why I now write the shutdown script first
The last thing that will surprise a team shipping their first agentic pipeline is the debris.
A pipeline that runs unattended for a week will leak. It will leak draft Jira tickets whose parent stories were never approved. It will leak GitHub branches for test suites that were never merged. It will leak TestRail runs that started, produced two results and never got a summary. It will leak Confluence page comments the agent posted while asking itself a clarifying question. None of these are bugs. They are the natural output of an autonomous system that starts more work than it finishes.
I learned this by accident. About six weeks into the project I ran a query on Jira for tickets created by the automation user in the past thirty days. I was expecting maybe 20. It was 91. 68 of them were [DRAFT] within the title and had not been touched by any human. I closed them by hand one afternoon, and whereas I used to be doing it, I understood that the pipeline had a silent second job I had by no means designed for: rubbish assortment.
Now each pipeline I run has three issues I write earlier than I write the primary agent. A shutdown script that closes any artifact the present run has orphaned. A nightly reconciliation go that closes any artifact any prior run has orphaned. And a single named proprietor — a human, on the org chart, with a Slack deal with — for each downstream system the pipeline can write into. When the pipeline creates a Jira ticket, the ticket has an actual assignee. When the pipeline opens a pull request, a selected reviewer is on the hook. When the pipeline information a TestRail run, somebody will get pinged if it goes stale. The pipeline will not be allowed to the touch a system that doesn’t have a named proprietor. That rule alone would have saved me the ninety-one-ticket afternoon.
There’s one anti-pattern right here I need to title outright. Don’t, beneath any circumstance, let the agent shut its personal artifacts. I attempted it. The agent, tasked with cleanup, closed the unsuitable thirty tickets after which confidently wrote a abstract saying it had closed the suitable ones. Cleanup is a human loop or a deterministic script. It isn’t a mannequin name.
The pipeline works. It saves actual time on the correct of labor. It additionally prices actual time on a form of labor most groups don’t put within the estimate: Contract validators between brokers, provenance stamping on each artifact, a shutdown script, a reconciliation go, a human proprietor for each downstream system and an unshakable rule in opposition to the mannequin doing its personal cleanup. None of that’s glamorous. None of it’s what the demo reveals. All of it’s the runbook, and the runbook is what determines whether or not a pipeline is a lab experiment or one thing the on-call engineer trusts sufficient to sleep by means of. The groups that write the runbook first ship. The groups that write it after the primary Sunday incident spend the subsequent two quarters catching up.
This text is revealed as a part of the Foundry Skilled Contributor Community.
Need to be part of?
