AI brokers are being judged by the standard of their solutions. That’s the fallacious place to start out. Earlier than asking whether or not an agent hallucinated, misunderstood a request or made a nasty choice, ask: did the work run in any respect?
I function scheduled brokers throughout roughly 18 companies on one machine. They learn telemetry, watch repositories and put together stories. For 18 months, I anticipated critical failures to return from the fashions. I anticipated a nasty choice or a assured reply constructed on a misunderstanding.
I used to be fallacious about the place the hazard lived.
Not one critical outage started with a nasty mannequin output. Each one started within the plumbing across the mannequin. A job was registered with the scheduler incorrectly. A script referred to as a binary that was truly a shell alias. A timeout guard relied on a command the working system didn’t ship. Every failure seemed clear from the surface. Every returned exit standing zero. The agent had not failed in an attention-grabbing manner. It had not run.
The outage that seemed like silence
The primary failure was a scheduled job that by no means loaded. The configuration existed, the command seemed appropriate and the encompassing service was wholesome. The schedule was fallacious, so the runner by no means launched the job. Nothing crashed. Nothing logged an exception. The anticipated report merely didn’t seem.
That may be a troublesome failure to note as a result of an absent output can imply a number of various things. The system could have run and located nothing. It could have run and didn’t authenticate. It could by no means have began. A dashboard that shows solely the most recent successful-looking worth turns all three states into the identical clean area.
The second failure was a shell assumption. A script invoked a command that labored interactively as a result of it was an alias. The scheduler launched it in a non-interactive setting the place the alias didn’t exist, but the wrapper nonetheless accomplished efficiently. The agent was blamed for producing no consequence although it by no means reached the work.
The third failure concerned a timeout guard constructed round a command obtainable on one machine however absent from the working system that truly ran the job. The safeguard didn’t safeguard something. The method once more exited cleanly, leaving a system that seemed idle quite than damaged.
These are abnormal distributed-systems failures: incorrect registration, setting variations and dependency assumptions. AI makes them more durable to see as a result of the mannequin output is probably the most seen a part of the system. Groups examine the immediate and reply whereas the scheduler, shell and course of boundary stay exterior the body.
Richard Prepare dinner’s essay “How Advanced Programs Fail” makes the broader level: complicated techniques not often fail due to one dramatic element. They fail by means of interactions amongst abnormal parts whose habits made sense regionally. An AI agent is one other element in that system, not a substitute for the system round it.
A very good agent stories unknown
Probably the most helpful design rule I reached got here from a unique incident. One in every of my brokers misplaced an API credential throughout a run. It may not see the channel it was purported to summarize.
The tempting response would have been an empty abstract: no new messages, no motion required. That reply would have been true in a slender sense. The agent had seen zero messages. It could even have been utterly deceptive, as a result of zero messages was not the state of the channel. It was the restrict of the agent’s entry.
The agent reported unknown as a substitute.
That distinction is the boundary between a system that’s trustworthy about its proof and one which quietly manufactures certainty. A very good agent stories unknown when it can’t observe the supply. A nasty one stories zero. The identical rule applies when a instrument name occasions out, a scheduler has not loaded the job or a dependency can’t be resolved. “No consequence” and “no information noticed” will not be interchangeable.
The POSIX specification isn’t an AI doc, nevertheless it represents the specific contracts agent techniques want. A human’s interactive shell isn’t the identical setting as a scheduled course of, and a profitable exit isn’t proof that the supposed work occurred.
Observability should embody what didn’t occur
Most observability steering begins after execution: gather logs, traces and metrics for the work carried out. That’s needed, however agent fleets want one other query: what work was anticipated however by no means noticed?
I now distinguish between a job that ran and produced a consequence, a job that ran and located no qualifying information, a job that ran however couldn’t entry its supply and a job that didn’t run or whose execution state is unknown. These states want completely different alerts and completely different language in stories.
“No new objects” ought to require proof that the supply was reachable and assortment accomplished. If that proof is lacking, the right result’s unknown, not zero. The OpenTelemetry observability primer is a helpful place to begin for enthusiastic about traces, metrics and logs as proof of habits. For agent techniques, I lengthen that concept to execution receipts: the schedule that fired, the command that resolved, the credential or connection that was used and the supply that was truly noticed. A last mannequin response is just the final hyperlink in that chain.
The lesson is to not add extra prompts. Make the boundaries testable. Confirm binaries. Take a look at the scheduler’s actual setting. Deal with lacking credentials and dependencies as express states. Add a heartbeat that proves the work ran, not merely that configuration exists.
AI brokers can produce unhealthy solutions. That failure is seen and sometimes reviewable. The extra harmful class is a system that produces a believable empty reply as a result of its execution path disappeared earlier than the mannequin had an opportunity to do something.
Your brokers might not be failing. They might not be operating. Earlier than evaluating the intelligence of the reply, confirm the existence of the work that was supposed to supply it.
How do I detect when an AI agent scheduled job by no means ran?
Distinguish between 4 execution states: job ran and produced a consequence, job ran and located no qualifying information, job ran however couldn’t entry its supply, and job didn’t run or whose execution state is unknown. Implement heartbeat alerts or execution receipts — logging the schedule that fired, the command that resolved, and the credential used — so a lacking heartbeat triggers an alert quite than silent absence.
Why does a scheduled agent return exit code 0 however produce no output?
A zero exit code confirms the wrapper course of accomplished, not that the supposed work ran. Frequent causes embody shell aliases that resolve interactively however not in a non-interactive scheduler setting, misconfigured cron or job-runner schedules that by no means set off, and lacking dependencies (corresponding to a timeout utility) that trigger the agent to skip its work silently.
What’s the distinction between ‘no consequence’ and ‘unknown’ in an AI agent system?
‘No consequence’ implies the agent ran, noticed the supply, and located nothing. ‘Unknown’ means the agent couldn’t confirm whether or not it noticed the supply in any respect — for instance, as a result of a credential was lacking or a instrument name timed out. Conflating the 2 causes techniques to report false certainty, making actual failures indistinguishable from genuinely empty information.
How ought to observability be arrange for AI agent fleets?
Commonplace observability (logs, traces, metrics) covers what occurred throughout execution. Agent fleets moreover want expected-but-not-observed monitoring: alerts that fireside when a scheduled job produces no execution receipt inside a time window. OpenTelemetry gives a basis; lengthen it with agent-specific execution receipts protecting the schedule set off, resolved binary, credentials used, and the info supply truly queried.
