What this pipeline does — and how this guide was assembled
This guide is assembled from official documentation and community reports — we did not run this build ourselves. It documents a pipeline that turns PDFs and scans into validated structured JSON using vision-capable models, covering the four documented lanes (Gemini, Claude, Mistral OCR, and OpenAI), prompt design, validation, and a cost method you compute yourself.
No “best model” claim appears anywhere below: the sources document what each lane does, not which one wins. The pipeline follows a five-stage chain — parse, understand, schema, extract, validate — drawn from the TokSpan write-up and expanded with prompt-placement, preprocessing, and failure-mode details from the provider pages themselves.
How each vision lane sees your document
Each provider converts your document differently before the model sees it. Gemini reads PDFs natively; Claude renders every page as an image plus extracted text; Mistral OCR returns page-level JSON with markdown, tables and confidence scores; OpenAI documents file inputs for the Responses API. The sections below give each lane’s documented behavior, limits, and gotchas.
For wider context beyond these four providers, the side-by-side model comparisons on ToolBrain cover how vendors position similar document capabilities against each other.
Gemini: native PDF vision
Per Google’s Gemini document-understanding documentation, Gemini processes PDFs with native vision — text, images, diagrams, charts and tables — in documents up to 1,000 pages. Multiple PDFs fit one request only if their combined size and your prompt fit the model’s context window, and extraction into structured output formats is supported. Two routing rules matter for beginners: non-PDF types (TXT, Markdown, HTML, XML) are sent as plain text, so layout and visual elements are lost — convert them to PDF before the vision stage. The page’s best practices: rotate pages upright before upload, avoid blurry pages, and for a single page place the text prompt after the page. For large PDFs reused across turns, it recommends the Files API. On billing, tokens from a PDF’s extracted native text are not charged, while tokens from pages rendered as images are reported under the IMAGE modality in usage_metadata.
Claude: image plus text on every page
Anthropic’s PDF support documentation describes a hybrid approach: the system converts each page to an image and extracts that page’s text, and Claude analyzes both together, so charts, diagrams and other non-text content are visible. Documented limits: request size up to 128 MB on the first-party API, up to 600 pages per request, dropping to 100 pages when the request’s context window is under 1M tokens, and standard PDFs only — passwords and encryption are not supported. Three input routes exist: a URL reference, a base64 document block, or a file_id from the Files API. The Files API route requires a hosted-on-Anthropic deployment; Bedrock and Google-managed routes accept base64 only. One compliance nuance: HIPAA eligibility applies to PDFs sent inline through the Messages API, not through the Files API.
Mistral OCR: page-level JSON with confidence scores
Mistral’s OCR documentation describes the alias mistral-ocr-latest, which accepts PDFs and images by public URL, base64, or uploaded file. Output is JSON: pages[], one entry per page carrying index, markdown (the main output), images, tables, hyperlinks, header, footer, dimensions, and confidence_scores. Options are version-gated: table_format (null for inline markdown, or markdown, or html) and the extract_header / extract_footer flags need OCR 2512 or newer, while include_blocks needs OCR 4 or newer. The confidence_scores_granularity setting accepts page, block, or word. For high-volume jobs the page points to Batch Inference; for structured output it points to Annotations.
OpenAI: file inputs for the Responses API
OpenAI’s file inputs guide documents file inputs for the Responses API. That is the extent of what this guide claims about the lane: the fetched documentation excerpt did not include size or page limits, parameter names, prices, or request shapes, so none are repeated here. Check their guide directly before building against it.
Pipeline design: parse, understand, schema, extract, validate
The TokSpan write-up argues that reliable extraction runs five stages in order: parse, understand, schema, extract, validate. Its central claim is that failures are silent — a model can return well-formed JSON with wrong values — so the schema, with required fields and enums, is validated after extraction, not before. Treat this as vendor practice, not a benchmark.
Three provider facts shape the design. First, prompt placement: for a single page, the Gemini documentation says to put the text prompt after the page. Second, preprocessing: rotate pages upright and avoid blurry pages before upload. Third, routing: because non-PDF inputs lose layout and visuals, convert anything that is not a PDF into PDF form before the vision stage. The hand-labelled sampling and field-accuracy work recommended by TokSpan is an evaluation-maturity step — the ToolBrain roadmap on building evaluation programmes sketches how to grow that capability over time.
Build walkthrough: a minimal scan-to-JSON pipeline
The build described here uses Python or any HTTP-capable language and three pieces: a preprocessor that rotates pages upright and rescans blurry ones, an extraction lane chosen from the four documented routes, and a validator that checks JSON Schema, triages Mistral confidence scores, and scores fields against a hand-labelled sample. The code below is illustrative pseudocode.
For SDK names, current releases, and setup notes for each lane, ToolBrain’s tools directory for document-processing SDKs is the maintained reference.
Config guardrails, each traced to its source page:
- Gemini: up to 1,000 pages per document; combine multiple PDFs in one request only when the combined size plus prompt fit the context window; use the Files API for large PDFs reused across turns (Gemini docs).
- Claude: keep requests within 128 MB and 600 pages — or 100 pages when the context window is under 1M tokens; fail fast on password-protected or encrypted PDFs (Claude docs).
- Claude routing:
file_idonly on hosted-on-Anthropic deployments; base64 on Bedrock or Google-managed routes; remember HIPAA eligibility covers inline PDFs, not Files API uploads (Claude docs). - Mistral: pin or verify the OCR version before relying on
table_formatand header/footer flags (OCR 2512+) orinclude_blocks(OCR 4+); route high volume to Batch Inference (Mistral docs).
# Stages 1-2: preprocess + OCR pass (illustrative pseudocode)
# Rotate pages upright and drop or rescan blurry pages first (a documented
# Gemini best practice), and convert non-PDF inputs to PDF before this stage.
ocr_request = {
"model": "mistral-ocr-latest",
"table_format": "html", # null = inline markdown; needs OCR 2512+
"extract_header": True, # header/footer flags need OCR 2512+
"extract_footer": True,
"confidence_scores_granularity": "word", # page, block, or word
# attach the PDF by public URL, base64, or uploaded file (request key per Mistral docs)
}
# Response: pages[], each with index, markdown, images, tables, hyperlinks,
# header, footer, dimensions, confidence_scores
# Stages 3-4: structure with a vision model + JSON Schema (illustrative pseudocode)
schema = {
"type": "object",
"required": ["invoice_number", "vendor", "currency", "line_items"],
"properties": {
"invoice_number": {"type": "string"},
"vendor": {"type": "string"},
"currency": {"enum": ["USD", "EUR", "GBP"]},
"line_items": {"type": "array"},
},
}
# Gemini: send the PDF natively; structured output is supported; for a single
# page, place the prompt after the page.
# Claude: every page arrives as image + extracted text; route by URL, base64
# document block, or file_id (deployment caveats apply).
# OpenAI: file inputs are documented for the Responses API; request shape per their guide.
# Stage 5: validate AFTER extraction (illustrative pseudocode)
def validate_page(extracted, page_confidences, schema, threshold):
structurally_ok = json_schema_check(extracted, schema) # any JSON Schema validator
confident = min(page_confidences) >= threshold # Mistral confidence_scores
return structurally_ok and confident # else route to human review
Validation runs in four layers:
- Structural: validate the extracted JSON against a schema with
requiredfields andenumconstraints, after extraction (TokSpan). - Signal triage: Mistral’s
confidence_scores, at page, block, or word granularity, route low-confidence pages to human review — the signal is documented (Mistral docs); the triage design is ours. - Accuracy: measure field-level accuracy against a hand-labelled sample, a practice TokSpan recommends as vendor opinion, publishing no independent numbers.
- Cost audit: read Gemini’s
usage_metadata— the IMAGE modality for rendered pages — to feed the cost formula in the next section (Gemini docs).
Documented failure modes to design against:
- Silent wrong-but-valid JSON: schema checks catch shapes, not wrong values — pair them with sampled field checks (TokSpan write-up).
- Layout loss on TXT, Markdown, HTML and XML inputs, which arrive as plain text — convert to PDF first (Gemini docs).
- Rotated or blurry scans degrading vision — preprocess before upload (Gemini docs).
- Oversized documents — chunk at 1,000 pages on Gemini, and at 600 pages or 128 MB on Claude, or 100 pages under a 1M-token window (Claude docs).
- Password-protected or encrypted PDFs rejected by Claude — fail fast (Claude docs).
- Version-gated Mistral options silently unavailable on older builds — pin the OCR version (Mistral docs).
- Deployment mismatch: Files API route needs hosted-on-Anthropic; Bedrock and Google-managed take base64 only (Claude docs).
- Multi-PDF requests failing when combined size plus prompt exceed the context window (Gemini docs).
- Cost surprise: image-rendered pages are reported as image tokens — measure one document before scaling (Gemini docs).
This guide is assembled from official documentation and community reports — we did not run this build ourselves.
Computing cost per page from your own price sheet
Nothing below is a price: this guide verified no dollar figures, latency numbers, or accuracy percentages. The method is reader-computed. Anchor fact: Gemini does not charge for tokens from a PDF’s extracted native text, and reports tokens from pages rendered as images under the IMAGE modality in usage_metadata, so your provider’s own reporting drives the math.
The method, as a repeatable box:
- Pick one representative document from your real workload — whatever dominates it.
- Run it once through your chosen lane.
- Read the provider-reported token usage; on Gemini, check
usage_metadata, where rendered pages appear under the IMAGE modality while native-text tokens are not charged (Gemini docs). - Multiply by the per-token prices on your own current price sheet.
- Divide by the page count.
cost_per_page = (reported_tokens × your_price_per_token) / pages
Re-run the measurement per document class — dense tables, forms, and plain prose will not report the same usage. Every cost number you publish should come from this loop, never from a blog post, including this one.
Tradeoffs at a glance
Every cell below traces to the provider documentation cited in this guide; where a fact was not documented in the fetched sources, the cell says so rather than guessing. Use the table to shortlist a lane, then re-verify limits on the provider’s page before you build, because documented caps change.
| Lane | How the document is seen | Documented limits | Structured-output path | Extra signals | Watch-outs |
|---|---|---|---|---|---|
| Gemini | Native PDF vision covering text, images, diagrams, charts and tables | Up to 1,000 pages per document; multiple PDFs per request only if combined size and prompt fit the context window | Structured output supported, per the docs page | Native-text tokens not charged; rendered-page tokens reported under the IMAGE modality in usage_metadata | TXT, Markdown, HTML and XML arrive as plain text and lose layout; Files API recommended for large PDFs reused across turns |
| Claude | Each page converted to an image plus extracted text, analyzed together | 128 MB request, 600 pages per request, 100 pages when the context window is under 1M tokens, standard unencrypted PDF only | Not documented on the cited page | Charts and diagrams visible to the model | Files API route needs hosted-on-Anthropic; Bedrock and Google-managed routes take base64 only; HIPAA eligibility covers inline PDFs, not Files API uploads |
| Mistral OCR | OCR pass returning JSON with one entry per page: markdown plus images, tables, hyperlinks, header, footer, dimensions | PDFs and images by public URL, base64 or uploaded file; table_format and header or footer flags need OCR 2512 or newer; include_blocks needs OCR 4 or newer | Annotations, as the pointer from the OCR page | confidence_scores at page, block or word granularity | Version-gated options silently unavailable on older builds — pin the OCR version; Batch Inference is the documented high-volume route |
| OpenAI | File inputs for the Responses API, per their guide | Not stated here — the fetched guide excerpt omitted size and page limits | See their guide | Not verified for this guide | No parameter names, prices or request shapes stated here |
| OCR-then-structure pattern | Mistral OCR emits markdown and tables, then a second pass structures them | The combination of both lanes’ caps applies | Schema validation after extraction | Confidence scores plus schema checks | More moving parts than a single native-vision call; no winner claimed by this guide |
Vision model vs. dedicated OCR: which should you pick?
There is no documented winner, and this guide makes no best-model claim. Both patterns are officially documented: Gemini and Claude take whole PDFs natively, while Mistral OCR extracts markdown, tables and confidence scores that a second structuring pass can consume. Choose based on your pages’ layout complexity, your volume, and the signals you need — not vendor benchmarks.
As decision heuristics: if per-word or per-block confidence routing matters to your review workflow, Mistral’s documented confidence scores provide that signal directly. If your documents carry charts, diagrams, or other non-text content you need understood, the native vision lanes see those elements by design. If your inputs arrive in non-PDF formats, conversion is required regardless of lane. Open-weight vision models exist, but this guide’s sources did not cover them, so nothing specific is claimed here.
FAQ
Three questions cover what beginners ask most: which architecture to choose, how to get a real cost-per-page number, and what breaks on odd file types. Each answer cites only the documentation in this guide; anything not documented is flagged as unverified rather than estimated. For standing questions about relative model strength, community-reported results are the closest thing to a leaderboard.
Q1. Whole PDF into one vision model, or OCR first and then structure?
Both are documented patterns. Gemini and Claude accept whole PDFs natively — pages as images plus extracted text — while Mistral OCR emits page-level markdown, tables and confidence scores, with Annotations as its structured-output pointer. The deeper treatment lives in the vision-versus-OCR section above; for community-reported standings rather than our testing, see the ToolBrain arena results.
Q2. How do I get a real cost-per-page number?
Use the method from the cost section: run one representative document, read the provider-reported token usage — on Gemini, usage_metadata with the IMAGE modality for rendered pages — multiply by the per-token price from your own current price sheet, and divide by pages. This guide publishes no figures, and none of its sources were verified for price, latency, or accuracy.
Q3. What breaks with DOCX or TIFF exports, or password-protected scans?
On Gemini’s documented behavior, non-PDF inputs — the docs list TXT, Markdown, HTML and XML — arrive as plain text and lose layout and visuals, so convert to PDF before the vision stage. Claude’s documented format rule is standard PDF with no passwords or encryption, so check for protected scans before upload. Either way, rotate pages upright and avoid blurry uploads first.
Where to go next on ToolBrain
Start small: pick one lane from the lane section, preprocess a few real pages, run the cost method once, and only then scale. The comparisons hub, tools directory, evaluation roadmap, and arena standings linked throughout this guide are the four anchors; the sibling topics worth scheduling next are structured-output prompting, batch inference for high-volume jobs, and building your first hand-labelled evaluation sample set.
📖 Related Reads
- CodeIntel Log — code quality, debugging, and software engineering benchmarks
Cross-links automatically generated from None.
Back to all posts