Anthropic just wrote down the playbook we've been building (and we couldn't agree more)

Anthropic just wrote down the playbook we've been building. I want to walk through it, partly because it's one of the clearest explanations I've seen of a problem we've bet our careers trying to solve, and partly because reading it felt like watching one of the best teams on the planet independently arrive at the same conclusions we did and then write them down.

The thing that changed this year

There was a lot of skepticism that the models couldn't write correct queries on complicated data domains. What changed this year is that this stopped being true: assuming the business and data context is actually present, text-to-SQL is essentially a solved problem now. Anthropic's data team says so about as plainly as anyone could, noting that once you've correctly mapped a person's question to the specific, up-to-date entities in your data model, the resulting execution and SQL becomes almost trivial. The intelligence was never in the query. It lives in knowing what the question actually means in the particular dialect of your business, and knowing it is still true today rather than as it was defined in the past.

They frame it like this: analytics accuracy is a context and verification problem, not a code generation issue. And they draw a sharp distinction between coding, which is an open-ended space that rewards a model's creativity and ships with its own built-in guardrails in the form of tests and types, and analytics, where there is usually exactly one correct answer drawn from exactly one correct source and no deterministic way to prove you got it right. Data, as they put it, is not software, and that single sentence explains why so many "just connect an LLM to your warehouse" demos look like magic for ten minutes and then fall apart the moment they meet a real organization.

The three ways it actually breaks

Anthropic names three failure modes, and rather than flatten them into a tidy list, I want to explore each, because anyone who has ever owned a number that ended up in an important deck will appreciate learning about them.

The first is the ambiguity between a concept and an entity, which is really the difference between answering the question someone typed and answering the question they actually meant. You ask for "active users," and buried in your data model are hundreds of plausible ways to answer that, which actions qualify as active, whether you count the flagged or fraudulent accounts, whether the window is seven days or thirty. A seasoned analyst resolves this without even noticing, either by asking a quick clarifying question or by simply knowing the house rules. While an ungrounded agent just picks an interpretation and commits to it, handing you a confident, formatted answer to a question you never asked.

The second is staleness, which is the one I think gets underappreciated the most, because your schemas and definitions and business logic are changing constantly, so whatever an agent learned last month is already subtly out of date, and nobody tends to notice until a figure stops tying out somewhere. Anthropic measured this directly, and the number is sobering. When they treated their guidance as a static artifact rather than something to actively maintain, their accuracy drifted from around 95% at launch down to roughly 65% over a single month.

The third is retrieval failure, which is almost the saddest of the three, because the correct answer genuinely exists somewhere in your stack; it's just buried deep in the accumulated pile of tables, notebooks, and dashboards that neither a person nor an agent can reliably surface it at the moment they need it.

What they had to build, and what we already ship

Their solution felt like reading our own roadmap with slightly different labels and ordering.

They start with strong data foundations, a deliberately small set of canonical, single-source-of-truth datasets, clearly owned and consumption-ready, with the near-duplicates aggressively deprecated, because every lookalike table is just another way for an agent to grab the wrong thing. They treat the metadata around those datasets, the column descriptions, metric definitions, and documented grain and ownership, as a first-class product rather than an afterthought. Orion generally sits on top of the warehouse you already have, rather than materializing a brand-new canonical store underneath you, so this is the one layer where their internal setup and our product diverge. But the spirit of it is pulling clean, governed definitions out of the semantics you already maintain, naming things precisely enough that "YTD Active Customers" never gets mixed up with "YTD Registered Users". This is exactly what Orion does.

On top of that, they put their sources of truth, ranked explicitly by how much they trust each one, and at the very top sits human-curated semantics that the agent is structurally required to consult before it does anything else, so the numbers are always calculated using agreed definitions, and the joins and grain and filters are already baked in. This is the layer that most directly mirrors Orion's metrics system, where a metric is a first-class, owned, versioned definition of a KPI rather than something the agent reinvents on every question, and where governance rules like always-filter constraints are enforced across every path the agent can take, right down to the data it extracts. There's one place where the comparison gets harder for us: Anthropic's answer to staleness.

Colocating the skill docs in the same repo as the transformation models, so that the PR that changes a model is the same PR that updates the doc describing it, is elegant precisely because they own both sides of that equation. We don't. Orion sits on top of data foundations our clients own, which means the definitions, schemas, and business logic we've encoded in skills and knowledge base page can drift out of sync whenever a pipeline changes upstream, without anyone necessarily knowing to update Orion's side of the picture. This is the harder version of the problem. When Anthropic says they watched accuracy fall from 95% to 65% in a month of neglect, they're describing an engineering discipline problem they solved by owning the whole stack. For us, solving it means building the connectors, the change detection, and the maintenance workflows that keep Orion's knowledge current even when the underlying data model lives somewhere else and changes on someone else's schedule. That's the version of this problem we've had to think hardest about, and it's the reason we've built CI/CD support into Orion rather than just telling customers to remember to update their docs.It was interesting that they tried to bootstrap a semantic layer by having an LLM auto-generate the definitions from raw tables and query logs, and it failed, because it produced plausible-looking definitions that simply re-encoded the very ambiguity they were trying to eliminate. Human curation turned out to be essential, which is precisely why, in Orion, we've built UI to make those definitions managed and editable by the people who actually own them.

They make a similar and almost counterintuitive point about the corpus of historical queries, which turned out to be far less valuable than they expected: giving the agent raw retrieval access to thousands of prior queries moved accuracy by less than a single percentage point, and when they dug into why, they found the precedent answer was already present about 80% of the time, and yet its mere presence didn't predict whether the agent got the question right, because unstructured retrieval simply couldn't map a new question onto the right piece of old work. The bottleneck, again, was structure rather than access, which is exactly why Orion leans on structured, searchable surfaces, a cross-conversation search that lets the agent find and reuse the right prior analysis, @-mentions that pull specific context into a question, and a knowledge base it's required to reference. Rather than a giant undifferentiated pile of past SQL.

Then there's the layer both Gravity and Anthropic call business context, the company knowledge graph of docs and roadmaps, decision logs, and org structure, which is the ambient understanding a new hire would otherwise spend their first few months absorbing, and which lets an agent resolve a vague reference or ask a smart clarifying question. This is, almost one-to-one, Orion's knowledge base: a company wiki with version history and an audit trail, that syncs from the places your context already lives like Confluence and Notion and GitHub, that the agent consults automatically during analysis, and that, crucially, is wrapped in the machinery that makes a knowledge base safe to actually use inside a real company: folder-level access grants, an approval workflow that auto-approves the low-risk edits and holds the risky ones for a human, and enough provenance that you can trust what's in there in the first place.

And finally, there's the layer that made the single biggest difference according to Ant: skills, which are folders of markdown that encode procedural knowledge, which sources to check, how to navigate ambiguity, and what a finished, trustworthy piece of analysis actually looks like. They describe a clever pairwise structure, a thin top-level knowledge skill that acts as a router and loads domain detail only when it's needed, paired with a playbook skill that captures how a senior analyst actually moves through a problem and bundles a dozen reusable patterns like retention curves and funnel analysis. This is the exact architecture Orion already uses, with a progressive-disclosure skill system where around twenty skills cover everything from data extraction to notebook analysis to per-datasource expertise, each loaded only when it's relevant, so the agent isn't drowning in instructions it doesn't need. You can add your own from there. The number they attach to this is one I'd underline: without skills, their agent was accurate 21% of the time, and with them it consistently cleared 95% and reached around 99% in some domains, with the same underlying model, the entire difference coming from the scaffolding around it.

There is an important point about maintenance, because skills describe data models that change every single day, and so they say treat skill maintenance as an engineering discipline rather than a documentation chore. The markdown lives in the same repository as the data transformations, so the pull request that changes a model is the same pull request that updates the doc describing it. This is the antidote to accuracy decay, and it's why we built native CI/CD support into Orion.

Keeping it working over time

The last sections of their post are about verification, which is the part most people skip and the part that actually separates a demo from something you'd let near a real decision. They run offline evaluations built from real questions, some auto-generated from existing dashboards and then validated by a human, some generated from business context to cover the long tail of rarer questions. They are careful to anchor each eval to a snapshot date or a stable fact table so the test itself doesn't quietly drift out from under them. A domain owner can't even announce to their stakeholders until that domain's slice of the evals clears a threshold they initially set around 90%.

In Orion, we do this with something called Observatory, a pipeline that generates claims from the material things it observes across your metrics, your knowledge base, and your conversations. Orion then verifies those claims against your live data before it trusts them, surfacing the ones it can't verify as open questions for a human rather than quietly asserting them. That instinct, that an answer should carry its own verification, and that the system should know the difference between what it has confirmed and what it is merely guessing, is the same instinct running underneath Anthropic's entire validation layer.

There's a second piece to this that I find delightfully recursive: as people use Orion, we stream product analytics back about how they're using it, and every Orion tenant mounts their own slice of that data as a data source, so our customers can analyze how their stakeholders are actually using Orion with Orion itself. It's Orion-on-Orion, and it's both quantitative and qualitative, not just how many questions got asked and how fast they came back, but the texture of what people asked and whether they walked away satisfied. The reason this matters is that it closes exactly the loop Anthropic spends the back half of their post describing: a data team can use the data Orion generates about itself to curate the experience their stakeholders are having, to spot the questions that aren't landing well, and to make sure the whole thing is meeting expectations and SLOs rather than quietly drifting. The verification isn't just happening inside a single answer; it's happening at the level of the whole deployment, and your data team gets to own it.

They also do a few smaller things worth calling out as sensible. Like Orion, they run an adversarial review step, a sub-agent that aggressively challenges the assumptions behind a final answer, and they measured its cost at about 6% more accuracy in exchange for 32% more tokens and 72% more latency, which is exactly the kind of trade-off you want stated plainly rather than buried. And every answer ships with a provenance footer telling you which tier the data came from and how fresh it is, so that a "raw table, freshness unknown" footer becomes a signal to verify before you forward something upstream. Orion does the equivalent in its own idiom, where every finding carries clickable citations back to the exact notebook cell that produced it, stale references get struck through automatically, Looker results come with deterministic links to view the result in Looker yourself, and the agent is explicitly prompted to distinguish an observation from a hypothesis and to refuse to assert causation it can't actually support. Anthropic is honest that even with all of this they don't yet have a robust solution for silent failures, and I'd rather point you to that than pretend anyone has fully solved it, because the real answer for now is exactly what both of us have landed on: make the provenance and the freshness and the uncertainty visible, so that a human always knows when to look twice.

Why I'm telling you all this

Most of what they describe building is what you get when you buy Orion.

Everything in that stack: the curated definitions, the knowledge base of business context, the skills that encode how a good analyst thinks, the evals that catch regressions before they reach a stakeholder, the citations and provenance that let someone trust a number. That is the exact set of things we decided should simply be Orion, rather than a months-long internal engineering PoC you have to bet on and then keep staffing forever. And underneath all of that sits the layer nobody writes blog posts about because it isn't glamorous, but which is the part you least want to build yourself: the roles and groups and SSO claim mapping, the per-project schema access controls, the time-bounded support access, the global search across every knowledge base page and metric and past conversation, the @-mentions that pull the right context into a question.

When a company I admire this much publishes a long, careful piece that lands on the same conclusions we built our company around, I feel relief, and a little bit of pride, and mostly gratitude that someone with their credibility said it so plainly, because we've been telling people for years that the hard part of AI analytics is context and trust rather than code.

This is the conversation we'd love to have.