Your AI demo works. Then someone uploads a crooked scan, a table that spans three pages or a contract with an exception buried in a footnote.
When the answer comes back wrong, the instinct is to adjust the prompt or swap models. It’s worth asking a different question first: did the information the model needed survive document processing?
Reliable agents run on accurate, structured, traceable data.
Document intelligence keeps the meaning of a source file intact, retrieval puts the right slice of it in front of the model, and verification and execution controls decide whether the agent has enough evidence, and enough permission, to act.
Documents become data, data becomes usable context, and context powers agents. But a strong data foundation only gets you to the starting line. Production agents also need tested application logic, integrations that don’t fall over, monitoring, and hard limits on what they’re allowed to do.
AI failures often start before the model runs
Business documents were designed for people. We know a column heading governs the numbers beneath it, and that a handwritten note in the margin changes the instruction next to it.
Optical Character Recognition (OCR) converts images of text into machine-readable text. It doesn’t necessarily carry those relationships with it. Tables come back as loose numbers, reading order shifts, footnotes drift away from the fields they qualify, and drawings and handwriting vanish altogether. Even a value that extracts perfectly gets harder to defend once you’ve lost track of where on the page it came from.
Document intelligence takes on the bigger job: pulling out content while preserving structure, relationships and a pointer back to the source. Plenty of workflows still need real human review and cleanup on top of that. Leave that work out of your project scope and you’ll overestimate how much you’ve actually automated.
Extracted text is not usable context
Once documents are data, your system has to find what each task needs.
With a small, well-organized set of documents, handing the full text to a Large Language Model (LLM) may be enough. Anything larger or messier needs a deliberate retrieval strategy. Either way you’ll run into conflicting versions, broken relationships and access restrictions you have to respect.
Structured storage gives fields a consistent home, and metadata tracks the things that decide which version wins: effective date, document version, permissions. From there, the model retrieves only the most pertinent information, all while respecting conflict rules and boundaries enforced by your metadata.
How you prepare that information matters. In its 2024 contextual retrieval experiments, Anthropic reported a 49% relative reduction in top-20 retrieval failure rate – from 5.7% to 2.9% – by enriching chunks with context before both semantic and keyword search. That measured whether the right information showed up in the first 20 chunks retrieved, not whether the final answer was correct. Which is rather the point: retrieval needs its own evaluation, separate from the model’s.
A knowledge graph, which makes the connections between entities explicit, earns its keep when a task depends on links between materials, specifications or projects. Plenty of other applications need nothing more than a database query or a document search.
Citations point a reader at supporting material. Provenance records where information came from and what happened to it along the way. You want both the first time an answer looks wrong.
You don’t have to trust the model blindly
You can pick a model and you can adapt it, but you can’t assume it will be right every time. What you can control is the evidence going in, the verification on the way out, and the permissions around both.
Confidence scoring flags uncertainty – in an extracted field, in a retrieval result, in the final output. Those three scores measure different things, and none of them is trustworthy until you’ve tested whether it actually predicts errors on documents that look like yours. Do that before you let a score route work.
Verification is the concrete part: check that required fields are present, validate units, reconcile totals, compare claims against the source. A citation gives you somewhere to look. It doesn’t prove the conclusion is right.
From there the policy writes itself. Low-confidence extraction triggers a retry or a human. Missing evidence stops any action that depends on it. High-risk decisions need approval no matter how confident the system is. Permissions and execution policy define which records the agent can reach and which actions it can take, and those limits belong in the application, outside the model.
From vibe-coded prototype to production
Vibe coding and AI-assisted development have made ideas cheap to test, and that’s a real gain. A working prototype gives users something to react to, and it surfaces requirements nobody could see on paper.
Production is a different animal. Documents arrive unvetted, uploads come in half-finished, and users show up with wildly different permissions. Someone has to chase down failed extractions, absorb service outages and make sure a retry doesn’t fire the same action twice.
A promising first version is still worth building, because it tells your team where it needs experienced help. That’s the real return on an AI proof of concept: not the demo, but a clear read on what the production system will take.
If you’re a consultant or an MSP, scope document discovery, representative testing, exception handling, integrations and support alongside the prototype. Be specific about which decisions the system will make and how you’ll judge whether it got them right. Wiring an LLM to a folder of files is a fraction of that engagement.
Build what differentiates you
Every project needs a data foundation. Whether you should build yours is a different question.
Meibel, a trusted York IE partner, provides document intelligence and AI orchestration technology. Through our AI, ML and data science practice, we work with Meibel to turn clients’ complex documents into structured, traceable data. The platform absorbs the foundational infrastructure work, which frees more of a client’s budget for the things that actually differentiate them: proprietary workflows, industry logic, user experience and integrations, plus the evaluation, deployment and adoption work that decides whether any of it gets used.
Build infrastructure when it creates an advantage, or when nothing on the market meets your requirements. Otherwise, evaluate platforms against your real documents, your traceability obligations and your operating constraints. The question is always where your engineering effort turns into customer value.
Questions to answer before you deploy
Turn the framework into deployment criteria:
- Does extraction preserve the tables, reading order and relationships that carry meaning?
- Can you trace an important output back to its source document and version?
- Are your confidence signals tested, and what happens when confidence is low?
- Which actions can the agent take on its own, and where are permissions enforced?
- Which outputs require human review, and can the reviewer see the evidence?
- Does custom infrastructure differentiate your product, or would an existing platform do it?
The agent is the last thing to run and the first thing anyone blames. Parsing sets the floor, retrieval and verification decide whether that floor holds weight.
Lose meaning in the first step and every layer downstream inherits the problem.
Join York IE and Meibel on Wednesday, September 16, 2026, at 1 p.m. ET / 10 a.m. PT for “Documents Become Data. Data Becomes Agents.” Kevin McGrath, Meibel’s Co-Founder and CEO, and I will walk through the framework and how it plays out in practice.
Can’t make it live? Register anyway and we’ll send you the recording.
