Experiment setup

Borrowing Birgitta Böckeler’s framing, a behavioral harness combines guides and sensors. Guides anticipate the place the agent may go incorrect and steer it earlier than it acts, rising the possibility of an accurate first try. Sensors observe the outcomes after it acts and supply suggestions that helps it self-correct.

For every mannequin, I in contrast brokers working with and with out the behavioral harness. Each situations started with the appliance transient and baseline repository directions in separate, personal workspaces. Harnessed brokers additionally acquired the API specs, examples, and instruments described within the inner-loop part beneath.

The experiment separated growth from analysis into two distinct loops. The inside loop supplied suggestions to harnessed brokers whereas they constructed. As soon as an agent completed, the outer loop ran 178 automated acceptance checks in opposition to its generated service. Each situations confronted the identical analysis, and the inside harness by no means had entry to the analysis suite’s reply key. Lastly, I collated the accepted outcomes, token use, time, and estimated value.

On this experiment, I handled an utility as acceptable provided that it began efficiently and handed all 178 exterior acceptance checks. Reliability meant how persistently a mannequin produced an appropriate utility throughout repeated runs. Token use, elapsed time, and estimated value have been separate measures of run effectivity.

The applying transient

Each try started with an in depth utility transient. I described what the service ought to do, however left the code-level implementation particulars to the agent.

The duty was to construct a backend-for-frontend (BFF) for product discovery. It needed to expose a consumer-facing HTTP API, validate incoming requests and downstream responses, and name an Order API. After a profitable availability search, it additionally needed to publish solely the primary returned product to Kafka. Publishing each product would have allowed the agent to go the info by with out implementing the choice conduct.

Determine 2: The applying structure described by the transient

The transient additionally specified much less apparent conduct. A timed-out product creation needed to grow to be a deferred request with a 202 response code and a hyperlink within the response header that customers may ballot for a response. A transient search failure needed to return 429 with a usable restoration interval; unavailable dependencies or failed occasion publication needed to grow to be 503; and profitable downstream responses nonetheless needed to obey requested product filters or in any other case return 502.

Determine 3: A profitable availability search and its Kafka publication

Briefly, the service was greater than a set of CRUD endpoints: the agent needed to coordinate HTTP semantics, asynchronous occasions, schema boundaries, and fault-tolerance insurance policies concurrently.

This was additionally a positive place to begin for an unaided agent: the transient spelled out the practical necessities, architectural and know-how constraints, detailed descriptions of the exterior interfaces, response shapes, and failure conduct in uncommon element. I don’t but know the way the outcomes would change with a much less full transient.

The inside loop: behavioral harness

On this experiment, the guides included OpenAPI and AsyncAPI specs that expressed the interface agreements in machine-readable kind, offering a foundation for executable checks. They complemented the transient quite than defining a separate set of necessities. The guides additionally contained request-response examples for these interfaces. The examples coated downstream timeouts, malformed payloads, filter violations, retry sequences, and Kafka occasion confirmations. An API-development talent provided implementation practices and steerage on dependency simulation, behavioral verification, and find out how to diagnose and reply to failures.

The sensors checked contract conformance, used API checks to confirm HTTP response values, and asserted observable negative effects, together with calls to downstream mocks and Kafka publication. Take a look at outcomes, CTRF stories, and API protection stories returned these observations to the agent throughout growth.

Birgitta additionally distinguishes computational controls, executed by deterministic instruments, from inferential controls, which depend on mannequin interpretation or judgment. This harness mixed each. The behavioral checks produced computational suggestions with out an LLM judging correctness. Guided by the talent’s directions, the agent nonetheless needed to interpret these outcomes and resolve find out how to repair failures.

Determine 4: The behavioral harness and its suggestions paths

An agent plugin provided the talent and MCP instruments, accompanied by the contracts and examples. The OpenAPI specs coated the BFF and downstream Order API; the AsyncAPI specification coated Kafka.

Earlier than implementing the service, the agent learn the specs and examples alongside the transient. The talent directed it to map the appliance’s elements and hyperlink every required conduct to its downstream interactions and verification checks. Helper scripts scaffolded these paperwork, a document of verification proof, and the instructions for constructing, beginning, and testing the service; the agent accomplished the behavior-specific particulars.

Configuration instruments wired the contracts to the appliance and dependency mocks. Contract-driven dependency mocks made downstream interactions executable, permitting the agent to train its implementation in opposition to outlined downstream responses, failures, and restoration sequences. Different helpers verified timing constraints. Collectively, these instruments supplied each implementation steerage and an executable atmosphere wherein the agent may examine its selections.

The agent invoked the harness by the mission’s check command. That command validated the configuration and examples, ran the required construct and unit checks, began the downstream mocks, and launched the appliance in opposition to them. It then ran the checks described above.

Report summaries recognized particular contract, response worth, and interplay mismatches. The agent used the talent’s directions to diagnose them, change the implementation or execution setup, and use a helper to rerun the affected eventualities. As soon as these eventualities handed, the agent ran the whole suite as a ultimate gate. The suggestions loop relied on the agent invoking these instruments throughout growth; it didn’t run routinely after each code edit.

Determine 5: The inner-loop check and suggestions sequence

Protecting guides and sensors in sync

One of many massive challenges of designing any such behavioral harness is to ensure the guides and sensors do not contradict one another and confuse the brokers. I addressed this challenge with the next strategy:

Earlier than operating any checks, an “examples validate” command checked that the instance payloads complied with the respective API specs. Checks then used these specs and examples to examine the implementation, holding the sensors aligned with their supply guides.

The supply of these checks mattered. If an agent interprets necessities to jot down each the implementation and its checks, the identical misreading or omission can seem in each. Permitting it to vary the anticipated outcomes when checks fail may obscure a defect. Right here, the contracts and examples signify agreed human expectations in executable kind. Contract checks have been generated deterministically from provided specs, whereas provided examples with behavioral fixtures outlined worth and side-effect assertions. The agent was instructed to protect these inputs and the talent’s verification helpers, examine failures, and proper the implementation or execution setup quite than weaken the expectations.

This gave verification expectations a supply impartial of the implementing agent’s interpretation. Used as a deterministic computational management, they make correctness checkable with out counting on the agent’s personal claims or an LLM’s judgment, releasing people to focus evaluation on what the contracts do not cowl.

The identical sensors subsequently serve two distinct functions relying on once they run:

• Throughout growth: Sensor outcomes feed again to the agent as executable consideration, steering building and serving to it get better from lapses.
• At completion: Sensor outcomes grow to be the idea for accepting the work: an impartial examine of the agent’s interpretation in opposition to human expectations, which issues for each mannequin.

The talent additionally required reviewable proof of dependency interactions and the ultimate verification end result, so completion was supported by recorded observations quite than the agent’s declare alone.

On this experiment, I applied the behavioral harness utilizing Specmatic, although equal contract-testing and repair virtualization instruments may present the identical capabilities.

The outer loop: analysis harness

To maintain the comparability truthful, every try ran inside an remoted Docker container with its personal workspace and a recent, attempt-specific Codex dwelling. Aside from shared authentication and language dependency caches, every agent started with out classes, recollections, plugins, abilities, or different state from an earlier run.

The 2 situations differed in what brokers may use throughout growth; inside every comparability, accomplished purposes confronted the identical exterior analysis.

After growth stopped, the outer-loop analysis harness examined every utility in opposition to the identical corpus of 178 mixed practical checks, together with invalid inputs, downstream failures, restoration sequences, and the messages produced by profitable requests. These checks exercised the conduct specified within the utility transient, together with the client-facing API, downstream interactions, and Kafka publication. The analysis harness managed its personal downstream fixtures and message seize quite than counting on the appliance’s growth checks.

Testing a cost-optimized mannequin

OpenAI positions GPT-6 Sol as a flagship mannequin for advanced skilled work, whereas GPT-6 Luna is optimized for cost-sensitive, high-volume workloads and roughly corresponds to the nano tier in earlier GPT-5 households.

With the methodology in place, the core query is whether or not a mannequin with a smaller functionality profile would behave otherwise.

I ran the very same experiment on GPT-6 Luna at low reasoning effort. And not using a harness, it succeeded in 3 of 10 runs. With the harness, it succeeded in all 10.

GPT-6 Luna (10-run averages for tokens, time, and value):

Offering that suggestions added a mean of 17,072,912 tokens, 19 minutes 4 seconds of autonomous runtime, and about $0.187 of estimated API-equivalent value per try. The harnessed try used roughly 3.9 occasions as many tokens general as an unharnessed Luna try. However take a look at the comparative economics: even with the harness overhead, a harnessed Luna try value about 33% lower than an unharnessed Sol try ($0.273 vs $0.406), and each situations achieved the identical 10-for-10 acceptance price on this pattern.

What went higher than anticipated

Trying previous the headline go charges into the codebase, a number of implementations stood out:

• GPT-6 Sol solved the transient unaided in a single shot.

  A single /purpose immediate was sufficient. All 10 unharnessed GPT-6 Sol purposes handed each practical examine on the primary go with out human debugging or corrective follow-ups.

• GPT-6 Luna got here remarkably shut by itself.

  Unharnessed GPT-6 Luna runs handed 94.8% of the practical checks. It was stumbling over refined edge circumstances quite than lacking the duty totally.

• Asynchronous 202 workflows held collectively end-to-end.

  Returning a 202 Accepted standing code is simple. What stunned me was seeing the agent wire up the monitor endpoint, observe pending requests, retry downstream creation with the unique idempotency key, and floor the ultimate payload on the polling URL.

• Kafka publication stayed tied to the search transaction.

  Slightly than firing off a message asynchronously and returning a direct 200, accepted purposes waited for Kafka acknowledgement earlier than responding, and accurately translated publication failures or timeouts into 503 Service Unavailable.

• Boundary validation was persistently defensive.

  Each fashions validated incoming payloads in opposition to schemas and validated downstream response constructions and enterprise constraints earlier than utilizing the info, quite than blindly trusting downstream providers.

The place unharnessed runs slipped up

The failed unharnessed Java purposes didn’t include invalid syntax or basically misunderstand the transient. Every nonetheless handed a minimum of 144 of the 178 mixed practical checks. Their failures clustered across the deferred creation workflow and a smaller set of protocol particulars:

• Deferred creation broke on the timeout boundary.

  When a downstream product creation exceeded the configured timeout, the BFF was speculated to return 202 Accepted with a monitor hyperlink, retry within the background, and finally expose the downstream 201 response. As an alternative, 4 unharnessed purposes returned 503, 2 returned 400, and 1 returned 429. An utility already counted above additionally created displays in some eventualities, however its retries by no means reached the terminal 201 response. A number of implementations acknowledged timeouts by matching a slender exception kind or cause-chain form. The artifacts set up the wrong responses, however typically don’t document the precise shopper exception, in order that mechanism is a probable clarification quite than a confirmed widespread trigger.

• Lacking monitor hyperlinks brought on secondary failures.

  In 7 runs, the wrong response to the timed-out creation additionally meant that no monitor URL was returned. A later evaluator examine then failed whereas attempting to make use of that lacking URL. I deal with these errors as penalties of the damaged 202 move, not as separate utility defects.

• Kafka supply succeeded whereas contract conformance failed.

  In 3 runs, each anticipated message reached Kafka, however the evaluator nonetheless rejected the publication. Two purposes used Spring’s JSON serializer and added an surprising __TypeId__ header. One other emitted “classes”: null, though the non-compulsory area needed to include an array when current. The mixing labored mechanically whereas nonetheless violating the wire contract.

• Error responses missed actual HTTP contracts.

  Different noticed misses have been narrower. Though 415 Unsupported Media Kind is a traditional response for an unsupported media kind, the appliance transient explicitly required 400 for invalid requests of this type, so the evaluator counted 415 as a contract violation. In different circumstances, invalid requests reached a generic 500 handler, and a lacking monitor returned the incorrect illustration. These have been small variations in code, however sufficient to make the appliance unacceptable beneath the exterior analysis.

• Configuration names diverged at an integration seam.

  Unusual product creation returned 503 as a substitute of 201. The generated shopper learn ORDER_API_API_KEY, whereas the recorded launch provided ORDER_API_KEY. That mismatch is a believable clarification for the downstream failures. In harnessed runs, contract-driven dependency mocks and configuration checks enabled validations of those atmosphere variables earlier than deployment.

The harness as executable consideration

I discover myself occupied with this dynamic when it comes to executable consideration.

The behaviors behind these failures have been specified within the provided necessities. The applying transient clearly specified the deferred creation workflow, Kafka message contract, and actual error responses. A requirement might be specific within the transient and nonetheless lose affect throughout implementation. I consider this as an consideration lapse: the ensuing code omits or incompletely implements a requirement the agent was given.

Juggling library selections, boilerplate setup, dependency injection, and asynchronous wiring throughout a rising context could contribute to such lapses, though this experiment didn’t set up the trigger. Executable consideration makes these gaps observable, giving the agent one other alternative to behave on the requirement.

• A timed-out creation is not deferred till the response features a monitor hyperlink and polling finally reaches the downstream end result.
• A Kafka publication is not legitimate merely as a result of a message arrived; its payload and headers should each conform to the wire contract.
• An non-compulsory area can’t be serialized as null when the contract permits solely an array or omission.
• An error response is not appropriate except its standing, content material kind, and physique all match the client-facing contract.

GPT-6 Sol preserved these particulars from the written transient in all noticed runs. GPT-6 Luna additionally applied a lot of the conduct accurately with out the harness, however its accepted outcomes diverse. The misses in Luna are in keeping with necessities dropping affect throughout implementation. My interpretation is that the harness served as an exterior reminiscence and a focus assist, pulling dropped necessities again into focus every time a sensor went crimson.

The economics of constant outcomes

Placing each mannequin samples aspect by aspect makes the price of that suggestions seen. These are API-equivalent estimates based mostly on recorded token use and mannequin costs, not precise billed prices.

Determine 6: Common estimated value and time throughout each fashions

Inside the GPT-6 runs, harnessed Luna averaged about $0.273 per try, in contrast with $0.406 for unharnessed Sol; each had 10 accepted purposes in 10 makes an attempt. The harnessed Luna try subsequently value about 33% much less on common on this pattern.

All 10 harnessed GPT-6 Luna purposes handed the exterior analysis and achieved a 100% success price. This small pattern can’t inform us the long-run failure price or the full value of reaching an appropriate utility.

These figures measure solely autonomous era value and machine time. They exclude the upfront effort of authoring, reviewing, and sustaining the contracts, examples, assertions, and harness. Additionally they pass over the hassle of discovering, diagnosing, and repairing an unacceptable end result. Reusing interface contracts and checks throughout implementations may unfold the setup value over time, however I didn’t measure that reuse or the ensuing whole value.

Sol’s 10-for-10 end result nonetheless relied on impartial analysis. That implies one other comparability: make the sensors out there to the agent solely at a ultimate verification gate, quite than all through growth. A final-gate workflow would supply an impartial acceptance examine whereas minimizing overhead when the primary implementation passes. Failures, nevertheless, may nonetheless require analysis and restore cycles. I didn’t measure that workflow’s value.

The boundaries of this experiment

This experiment coated 10 paired runs per mannequin on 1 greenfield Java service, with each fashions at low reasoning effort. That pattern describes the noticed outcomes; it can’t set up long-run reliability.

A extra demanding structure or a trickier resiliency state of affairs may expose gaps in Sol that require a harness. Conversely, an easier job may run reliably on Luna with out this behavioral harness.

The transient specified practical conduct, structure, know-how, and interfaces in uncommon element. I’ve not examined whether or not the sample holds when modifying legacy code, utilizing different languages or reasoning settings, or working from imprecise or incomplete necessities. A much less full transient may have an effect on each the unaided implementation and the standard of the contracts and examples provided to the harness. I might be cautious about extrapolating these actual figures to completely different briefs, architectures, codebases, or workforce workflows.

As a result of the harness added guides and sensors collectively, this experiment additionally can’t isolate their particular person contributions.

What this implies in follow

I might deal with these outcomes as a place to begin for selecting a mannequin and suggestions cadence for a specific job. What modifications with mannequin functionality is when and the way typically the sensors run, not whether or not an impartial examine is required.

• For cost-optimized fashions: When a cost-optimized mannequin produces inconsistent outcomes, run a behavioral harness all through growth. It serves as executable consideration, catching drift because it occurs and turning inconsistent makes an attempt into reliable supply.
• For frontier fashions: Retain an impartial acceptance examine even when a mannequin passes persistently on the primary try. As an alternative of operating steady checks throughout growth, run the automated checks as soon as as a deterministic ultimate verification gate earlier than acceptance, offering an impartial examine with minimal overhead.
• Spend money on clear specs and examples: The checks can implement solely the expectations people encode. Overview and keep these expectations alongside the implementation, and use human evaluation to deal with gaps in protection.

Conclusion

My authentic assumption was too blunt. A behavioral harness will not be an computerized win for each mannequin on each job. For Sol, operating the harness all through growth added overhead with out a detected acceptance profit; for Luna, harnessed makes an attempt have been accepted extra persistently and value much less on common than unharnessed Sol makes an attempt on this pattern.

That leaves me with a revised, conditional speculation:

What modifications with mannequin functionality will not be whether or not impartial sensors are wanted, however once they run. For fashions whose unaided makes an attempt produce inconsistent outcomes, operating sensors constantly throughout growth gives executable consideration that broadens the set of fashions a workforce can use efficiently. For frontier fashions able to fixing the transient in a single go, operating those self same sensors as soon as as an impartial ultimate acceptance gate gives the verification each system requires with out the event overhead.

In follow, engineering groups hardly ever decide a mannequin on uncooked benchmark intelligence alone. Unit economics, latency budgets, knowledge privateness, and the power to run on premises typically dominate the choice. If a behavioral harness can take a less expensive, lighter mannequin that already suits these operational constraints and make it reliably ship the products, that could be a compelling choice.

I’ve examined this thus far on a single greenfield service with an unusually detailed transient. The subsequent step is to check a “ultimate gate solely” situation for frontier fashions to measure its overhead and the way typically the gate catches discrepancies. Past that, I plan to differ the standard of the transient and the results of a missed requirement, then check how properly this speculation holds up when modifying an current codebase, switching programming languages, or operating open-weights fashions domestically.