Anthropic recently published a case study on how they've internally automated 95% of all business analytics queries using Claude — with ~95% accuracy in aggregate.
That's an impressive result. But the most important insight is buried in a footnote: without the right infrastructure around the AI agent, accuracy sat at 21%.
This isn't a model problem. It's a data governance problem.
The honest picture of AI analytics today
Most organizations testing AI for data analysis make the same mistake: they point a model at a data warehouse and hope for the best.
The outcome is predictable. The agent retrieves the wrong table. It uses a stale metric definition. It returns an answer that looks correct but is built on the wrong data — and no one knows.
Anthropic identified three systemic failure modes:
- Entity ambiguity — the agent can't map "revenue for product X" to the correct table and column, because there are fifteen tables with similar names and no clear owner
- Data staleness — schemas, business definitions, and logic change constantly; the agent doesn't know what's current
- Retrieval failure — the right information exists, but the agent can't find it in a warehouse with millions of fields
The solution: an agentic data stack
What took Anthropic from 21% to 95% wasn't a better model — it was a layer of structure around the model:
1. Canonical datasets
Instead of dozens of near-identical tables: a small number of well-governed, clearly owned datasets that serve as the single source of truth. Near-duplicates are aggressively deprecated.
2. Semantic layer as primary source
Compiled metric and dimension definitions that the agent must consult first — before writing a single line of SQL. If a metric is covered by the semantic layer, a consistent answer is returned.
3. Skills — structured context documents
Markdown files colocated with dbt models in a single repo. CI checks ensure that a change in the data model always triggers a skill update. Without skills: 21%. With skills: 95–99%.
4. Evals and provenance
Every answer carries a "provenance footer" — which data source it came from, how fresh the data is, who owns the model. Evals run automatically on every PR merge.
The most surprising finding
Anthropic tested giving the agent direct access to thousands of historical SQL files — thinking "every right answer already exists in there." It barely helped. The agent performed only marginally better.
This reinforces the point: raw data is not knowledge. Structured, maintained, owned context is knowledge.
What this means for SMBs
There's a clear pattern in how AI implementations fail: teams invest in a good model or tool, then wonder why the agent gives wrong answers — or why the team stopped trusting it.
The answer is almost always the same: the context around the agent wasn't built for machine consumption.
That's exactly what we build at VasthavM. Not a new AI model — but the infrastructure that makes that model consistently deliver correct, trusted answers in your specific business.
This means:
- Data governance — clear ownership, canonical definitions, deprecated duplicates
- Context layers — structured documents the agent can navigate without guessing
- Validation — automated checks that catch errors before they reach end users
- Maintenance — processes that keep context current as the business evolves
A critical detail from Anthropic: without active maintenance, accuracy dropped from 95% to 65% within one month. Today, 90% of all data PRs update agent skills.
This isn't just a tech project — it's an organizational commitment.
The real AI moat
Prashant Goela, data engineering expert, summarized it best in his analysis of Anthropic's case study:
"The real enterprise AI moat isn't the LLM. It's the legibility of the data infrastructure supporting it."
Anyone can call Claude via an API. Not everyone can build a system where Claude consistently finds the right answer in your specific business data.
That's the difference between an AI demo and AI that creates value in production.
Want to know where your organization stands? Book an Automation Audit — we map where context gaps are costing you time and money, and build a concrete plan to close them.