The core problem isn't that enterprises lack enough tests — it's that the tests they run don't reflect what happens in production. A June 2026 VentureBeat Pulse survey of 157 organizations (100+ employees) found that 50% had deployed an agent or LLM feature in the past year that passed internal evaluations and then caused a customer-facing failure. A quarter saw it happen more than once. Only 36% reported a clean record. The evaluation is clearing agents that aren't ready, and most teams don't find out until a customer does.
Trust in automated evaluation is nearly nonexistent. Just 5% of respondents say they fully trust their automated evals today. The most common complaint — cited by 29% — is that evaluations align poorly with real-world outcomes. Inconsistency and bias (21%) and lack of explainability (18%) follow. In other words, the tools meant to certify agents are themselves uncertified. Builders need to treat a passing eval score as a necessary condition, not a sufficient one, and design their release process accordingly.
Despite this, the autonomy ceiling is rising fast. Two-thirds of organizations either already allow zero-human-in-the-loop deployment for low-risk agents (34%) or are actively building pipelines to enable it within twelve months (33%). Larger enterprises are slightly ahead of smaller ones on this path — not behind, as many assume. The mechanism is straightforward and concerning: as human review gets engineered out of the deployment loop, the burden shifts entirely onto evaluations that most teams openly admit they don't trust.
The tooling picture makes this worse. The most common primary evaluation tool is provider-native evals from OpenAI or Anthropic — tied, at 17% each, with having no dedicated evaluation tooling at all. Independent specialists like DeepEval, Braintrust, LangSmith, and Arize each hold single-digit to low double-digit shares. Only about a quarter of organizations run real-time quality checks on live production traffic; the majority monitor for uptime and latency but not for whether the agent's answers are actually correct. A confidently wrong response is invisible to infrastructure monitoring.
Where is the money going next? Production observability leads planned investment, followed by human review workflows at 26% — more than the 16% planning to grow automated evaluation pipelines. That's the report's sharpest contradiction: the same organizations removing humans from deployment decisions are simultaneously budgeting to add more human reviewers. They're hedging, which is rational given the gap, but it's not a long-term fix. Sixty-four percent also plan to adopt a new or replacement evaluation platform within twelve months, signaling that the current tooling layer is about to consolidate around whoever can demonstrate real-world alignment.
What should builders actually do? First, track eval-to-production failure rate as a first-class metric — if half your peers have already shipped a false-passing agent, you need to know whether you're in that group. Second, add output-quality monitoring to production, not just uptime checks; a request that completes fast with a wrong answer is a silent failure. Third, before extending any agent's autonomy, audit whether your evaluation suite actually covers the failure modes you've seen in production, not just the ones that are easy to test. More evals won't close this gap — evals that reflect reality will.