They mentioned that the benchmark comprises 310 work environments throughout 52 skilled domains together with coding, crystallography, family tree and music sheet notation. Every setting consists of actual paperwork totaling round 15K tokens in size, and 5 to 10 complicated modifying duties {that a} consumer may ask an LLM to carry out.
And, they acknowledged within the paper’s summary: “Our evaluation reveals that present LLMs are unreliable delegates: they introduce sparse however extreme errors that silently corrupt paperwork, compounding over lengthy interplay.”
These errors are vital, they mentioned. “The findings present that present LLMs introduce substantial errors when modifying work paperwork, with frontier fashions (Gemini 3.1 Professional, Claude 4.6 Opus, and GPT 5.4) dropping a median 25% of doc content material over 20 delegated interactions, and a median degradation throughout all fashions of fifty%.”
Benchmark train receives a thumbs up
Brian Jackson, principal analysis director at Information-Tech Analysis Group, discovered the findings very fascinating. “Placing a listing of LLMs to the take a look at throughout completely different work domains yields a whole lot of helpful insights,” he mentioned. “I believe this sort of benchmark train could possibly be useful to enterprise builders who want to leverage agentic AI to automate particular workflows and perceive the bounds of what will be achieved.”
