Every document intelligence vendor pitch sounds the same: "Our extraction accuracy is best-in-class." What they don't show you is the harness. And the harness is where the real decision gets made.
The trap: comparing outputs instead of interfaces
When we started evaluating extraction approaches for structured regulatory documents — think FAA maintenance records, NHTSA incident reports — the instinct was to run each candidate model on a sample set and eyeball the results. Tesseract-plus-LayoutLM here, Unstructured.io-plus-LayoutLM there, Donut somewhere else, an LLM-based extractor as a fourth option. Four pipelines, four sets of glue code, four ways to get burned when you switch vendors six months later.
That's backwards. The extraction method is an implementation detail. What actually matters is whether every method answers to the same contract.
One harness, two shapes
The diagram above shows the pattern we landed on, and it splits into two architectures that look almost identical from the test harness's point of view but are structurally very different underneath.
Top: the LLM path. The test harness talks to an API. The API is a single round-trip conversation with an LLM — request in, structured extraction out. Simple, linear, one dependency.
Bottom: the GPU-backed path. Same test harness, same API surface, but now the API is fanning out to specialized models — LayoutLM with OCR pre-processing, or Donut running end-to-end on document images — each on its own GPU-backed service. The API absorbs the complexity of routing to the right extractor and normalizing what comes back.
The test harness never knows the difference. That's the point.
The contract that makes it work
Both paths return the exact same shape:
{
doc_id: string
confidence: float
inputs: JSONB { ... }
outputs: JSONB { ... }
metadata: JSONB
}That's a deliberately thin contract. doc_id and confidence are queryable, indexable, comparable across runs. inputs and outputs are JSONB precisely because the shape of what goes in and comes out varies by extractor — a Donut run and an LLM run don't produce structurally identical payloads, and forcing them into a rigid schema would mean lossy normalization at exactly the point where you want to preserve detail for later analysis. metadata catches everything else: model version, latency, pipeline stage, whatever you need for debugging without polluting the core fields.
The result is that confidence scores, doc IDs, and timing data are comparable across every extractor in the harness, even though the extractors themselves have nothing in common architecturally. One's a single API call to a hosted LLM. The other is a GPU inference fleet. From the outside, they're interchangeable.
Why this matters more than accuracy numbers
Here's the uncomfortable truth about document intelligence evaluations: accuracy on your sample set today tells you almost nothing about total cost of ownership over eighteen months. What tells you that is how expensive it is to add a fifth extractor, swap out a vendor, or run a controlled A/B between two approaches on production traffic.
With a shared output contract:
- Adding a new extractor means writing an adapter, not rebuilding the harness.
- Swapping Tesseract-plus-LayoutLM for a newer OCR front-end doesn't touch anything downstream.
- Confidence-based routing — send high-confidence docs straight through, kick low-confidence ones to a heavier model or a human — becomes a query against
confidence, not a bespoke integration. - You can genuinely compare an LLM-based approach against a specialized vision-language model on cost, latency, and accuracy, because the measurement layer doesn't care which one produced the result.
That last point is the one most teams skip. It's very easy to fall in love with an architecture — GPU-backed models feel more "real," LLM calls feel more "modern" — before you've actually measured which one wins for your document types and your volume. A Protocol-based extractor interface, with mock extractors standing in during early development, lets you build and test the harness itself before any real model is wired in. You're validating the contract, not the vendor.
The mid-market version of this problem
If you're a PE-backed operating company sitting on a document backlog — inspection reports, maintenance logs, incident forms, whatever your industry's version of "unstructured paperwork nobody wants to touch" happens to be — you don't need to pick the perfect extraction model on day one. You need an architecture that doesn't punish you for picking wrong.
The document is different every time. The problem — get structured, confident, comparable output regardless of what's producing it — isn't. Build the harness first. Let the extractors compete inside it.
Clarity through the chaos.