Microsoft's DELEGATE-52 Benchmark: AI Agents Silently Corrupt 25% of Documents
TL;DR: Microsoft Research's DELEGATE-52 benchmark tests 19 LLMs across 52 professional domains and finds even frontier models silently corrupt 25% of document content during long editing workflows. The errors are sparse but catastrophic — entire sections rewritten with fabricated data — and agentic tools don't help. Gemini 3.1 Pro, the top performer, was deemed ready in only 11 of 52 domains. The paper is a sobering reality check for anyone betting on autonomous AI agents to replace human oversight in document-intensive workflows.
The Delegation Dream Meets Reality
The pitch is seductive: hand your messy documents to an AI agent, walk away, and come back to perfectly organized, error-free results. This vision — dubbed "vibe coding" in software engineering circles and "delegated work" by researchers — is driving massive investment in autonomous AI agents across enterprise workflows.
But a landmark study from Microsoft Research, published in April 2026 and widely covered this month, suggests the dream has a serious blind spot. Their new benchmark, DELEGATE-52, systematically measures how well AI models preserve document fidelity during multi-step editing tasks. The results are sobering: even the most advanced frontier models silently corrupt their own work, introducing errors that are nearly impossible for a human to catch without meticulous line-by-line review.
The researchers — Philippe Laban, Tobias Schnabel, and Jennifer Neville — designed DELEGATE-52 to answer a straightforward question: can you trust an AI to edit your documents over 20 consecutive interactions without messing them up?
The answer, across 19 models and 52 professional domains, is a resounding "not yet."
How DELEGATE-52 Works: The Round-Trip Relay
What makes DELEGATE-52 clever is its evaluation methodology. Grading multi-step document editing usually requires expensive human review — someone has to check that every change was applied correctly. The researchers bypassed this bottleneck with a "round-trip relay" approach inspired by backtranslation in machine translation.
Here's how it works:
- Each task is designed to be fully reversible — every forward editing instruction has a precise inverse
- A model is given a seed document (2,000 to 5,000 tokens from real-world sources) plus 5-10 editing tasks
- The model performs all tasks in sequence, then researchers give it the inverse tasks in a new session (the model doesn't know it's reversing its own work)
- After the full relay, they compare the final document to the original
The difference between the original and the round-tripped document measures how much the model silently corrupted the content. This method simulates 20 consecutive interactions, with 8,000 to 12,000 tokens of distractor files thrown in to mimic realistic work environments.
As Laban explained to VentureBeat, "The models do not know whether a task is a forward or backward step and are unaware of the overall experiment design. They are simply attempting each task as thoroughly as they can at each step."
The Numbers: Frontier Models Fail at Surprising Rates
The headline finding is stark. Across all 19 models tested — including top-tier systems like Gemini 3.1 Pro, Claude 4.6 Opus, and GPT 5.4 — documents suffered an average degradation of 50% by the end of the 20-round simulation.
Even the best frontier models corrupted an average of 25% of document content.
Let that sink in. The world's most capable AI systems lose or distort a quarter of your document's content over 20 editing rounds.
Python coding tasks were the notable exception — most models achieved "ready" scores of 98% or higher in this domain. The structured, testable nature of code means errors are easier to detect and fix. But for virtually every other domain — financial accounting, legal documents, fiction editing, music notation, earnings statements — performance dropped off a cliff.
The top overall performer, Gemini 3.1 Pro, was deemed "ready" for delegated work in only 11 out of 52 domains.
The Corruption Pattern: Not Death by a Thousand Cuts
Perhaps the most unsettling finding is how the corruption happens. You might expect incremental fidelity loss — tiny errors that gradually accumulate. That's not what the data shows.
Approximately 80% of total document degradation comes from sparse but catastrophic failures. In a single interaction, a model can drop 10% or more of a document's content — entire paragraphs, tables, or data sections silently erased or rewritten.
The frontier models don't necessarily make fewer small errors. They just delay the catastrophic failures to later rounds. This makes them more dangerous in some ways: they maintain an illusion of reliability for the first few interactions, then silently wreck everything when you're not looking.
Even more concerning, the nature of the errors differs between model tiers:
- Weaker models corrupt primarily through content deletion — text simply vanishes
- Frontier models corrupt through active rewriting — the text is still there, but subtly distorted with hallucinated facts or fabricated data
The second type is far harder to catch. A deleted paragraph stands out. A rewritten one that sounds plausible but contains wrong information requires expert-level domain knowledge to identify.
"Frontier models don't just delete document content — they rewrite it, and the errors are nearly impossible to catch," as VentureBeat summarized.
Agentic Tools Don't Help — They Make Things Worse
This finding flips a common assumption on its head. If you give an AI agent tools — code execution, file read/write access, a development environment — shouldn't that improve its editing accuracy?
Not according to DELEGATE-52. Providing models with an agentic harness actually added an average of 6% more degradation.
The reason, according to Laban, is that models "lack the capability to write effective programs on the fly that can manipulate files across diverse domains without mistakes." When the model can't figure out a programmatic approach, it falls back to reading and rewriting entire files — a far more error-prone strategy.
The lesson for developers: generic tools don't help. What works is building tightly scoped, domain-specific functions — a purpose-built function to calculate entries in a ledger file, for example — rather than giving an agent free rein with a file system.
Distractor Files and the RAG Warning
The DELEGATE-52 benchmark included distractor files (8,000-12,000 tokens of related but irrelevant content) to simulate real-world workspaces. The impact was dramatic.
While a noisy context window might cause only a 1% performance drop over two interactions, that degradation compounds to 2-8% over longer simulations. For teams building retrieval-augmented generation (RAG) pipelines, this is a direct warning: single-turn RAG evaluations systematically underestimate the harm of imprecise retrieval.
Laban put it bluntly: "RAG pipelines should be evaluated over multi-step workflows, not just single-turn retrieval benchmarks. Single-turn measurements systematically underestimate the harm of imprecise retrieval."
What This Means for Enterprise AI Adoption
The timing of DELEGATE-52 is critical. May 2026 has been a banner month for agentic AI announcements — OpenAI's Symphony, Anthropic's Claude Managed Agents, Microsoft's Office AI agents, and SAP's 200+ Joule assistants at Sapphire 2026. Every one of these announcements sells the promise of delegation: hand off the work, trust the agent. DELEGATE-52 is the strongest empirical challenge to that promise to date.
This doesn't mean AI agents are useless. What it means is that the "fire and forget" model is dangerous for document-centric workflows. Enterprises need human-in-the-loop verification, incremental review points, and domain-specific guardrails — not just a single final check.
Laban recommends building applications around "short, transparent tasks rather than complex long-horizon agents." He notes that the methodology itself can serve as a practical blueprint for teams wanting to test their own AI pipelines.
A Path Forward: Testing Your Own AI Pipeline
The researchers emphasize that DELEGATE-52 isn't meant to discourage AI adoption — it's meant to make it safer. They've open-sourced the benchmark and data, and they believe the rate of improvement is encouraging.
"Looking at the GPT family alone, models go from scoring below 20% to around 70% in 18 months," Laban noted. "If that trajectory continues, models will soon be able to achieve saturated scores on DELEGATE-52."
For organizations deploying AI agents today, the DELEGATE-52 framework provides a template for stress-testing in-house pipelines:
- Build reversible editing tasks — Design forward/backward task pairs that represent your actual workflows
- Create document parsers — Convert your domain documents into structured, comparable representations
- Measure round-trip fidelity — Run the relay and measure how much content degrades
- Fix the gaps — Invest in domain-specific tools, not generic agent harnesses
The Microsoft team found that existing parsing libraries worked for 30 of the 52 domains in DELEGATE-52, meaning most teams won't need to build parsers from scratch.
The Bottom Line
DELEGATE-52 is the most rigorous demonstration yet that current AI models are not reliable delegates for document-intensive work. The errors aren't minor typos — they're silent, catastrophic rewrites that compound over time. The best model in the study earned passing marks in only 21% of tested domains.
The practical takeaways are clear:
- Don't fire-and-forget on documents. Human review is essential at every stage, not just at the end
- Short, scoped tasks beat long autonomous workflows. Break delegation into small, verifiable steps
- Domain-specific tools beat generic agent harnesses. Invest in purpose-built functions over open-ended agent capabilities
- Test your own pipelines. Use the DELEGATE-52 methodology to measure fidelity before trusting an agent with production documents
The race toward autonomous AI agents is accelerating fast. But DELEGATE-52 is an essential reality check: before we hand over the keys, we need to make sure the car can drive straight.
Frequently Asked Questions
What is DELEGATE-52?
DELEGATE-52 is a benchmark created by Microsoft Research to measure how accurately AI models handle multi-step document editing tasks. It tests 19 LLMs across 52 professional domains using a round-trip relay methodology that automatically detects content corruption.
Which models were tested in DELEGATE-52?
The study tested 19 models from OpenAI, Anthropic, Google, Mistral, xAI, and Moonshot, including frontier models like Gemini 3.1 Pro, Claude 4.6 Opus, and GPT 5.4.
How bad is the document corruption?
Even the best frontier models silently corrupt an average of 25% of document content over 20 editing rounds. Across all tested models, the average degradation was 50%. The top performer, Gemini 3.1 Pro, was deemed ready in only 11 of 52 domains.
Do giving AI agents tools help?
No. According to the study, providing models with agentic tool harnesses actually worsened performance by an average of 6%. Generic tools without domain-specific scoping are counterproductive.
How can I test my own AI agent pipelines?
Microsoft has open-sourced the DELEGATE-52 code and data. The researchers recommend building reversible editing tasks that represent your workflows, creating document parsers for structured comparison, and measuring round-trip fidelity before trusting agents with production documents.
← Back to all posts
