Good Enough for What?

The Fifth Elephant

Good Enough
for What?

Is the cheaper model
good enough?

I've had to answer this three times this year.
I got it wrong, every single time.

Three cases, in order

1 · OCR engine · document pipeline 2 · embedding model · retrieval 3 · vision verifier · CCTV compliance

Nothing in common between the three systems - except this question, asked once per case.

One thing held constant: every model below, hosted or small, is used exactly as downloaded or called via API. No fine-tuning, no custom training, on either side.

Case one · a document pipeline

Hosted OCR usually wins,
publicly, by ~9 points.

Tesseract, general OCR benchmark

87.2

Google Cloud Vision, same benchmark

95.9

Sarvam, Hindi-specific OCR benchmark

95.91

Two different roundups: a generic one, and Sarvam's own Indic bench, where it beats Gemini 3 Pro (95.12) and GPT-5.2 (84.86).

Real, official, Indic-specific OCR done well. It just wasn't part of this ten-engine comparison.

aimultiple.com OCR Accuracy Benchmark; sarvam.ai/blogs/sarvam-vision

So I tested it on my own corpus

Tesseract.

Free. Local. No API cost.

A real scanned government document page in Hindi, showing starred question 1883, the kind of source material this benchmark runs on

One real page from the corpus - the exact page behind the *1883 example on the next slide.

Word accuracy, Hindi

hosted 92-95

90.03

The gap nearly vanished - which is exactly what made me trust it.

The question I actually asked

Could it still read
the question number?

question *1883. small model -> +883. hosted model -> *1883.

The string that links a question to its answer.

A junk token and a dropped leading digit - the source says *1883, the small model returns +883. Rate on a genuine scan: 2.2% -> 5.2% junk-token rate, and it lands exactly on the question numbers, circled above.

Benchmark: word accuracy. Metric that mattered: can the question number survive.

Question numbers recovered · recovery rate, by engine

figures.md, question-number recovery

Same benchmark. One axis called it close. This one didn't.

Not a script-difficulty gap - the Hindi and English subsets use different question-numbering conventions, and the corruption hits the delimiter format, not the language.

Rejected.

Benchmark: word accuracy

90.03

vs. hosted 92-95 - close call

Metric that mattered: question numbers

68%

vs. hosted 100% - not close at all

Case two · retrieval feeding generation

bge-m3.

Open. ~570M parameters. One modest GPU.
Against hosted text-embedding-3-large.

The public number that made it worth trying

Beat OpenAI's larger model,
publicly, by ~15 points.

bge-m3, avg across 18 languages

70.0

OpenAI text-embedding-3-large

54.9

nDCG@10, MIRACL multilingual retrieval - bge-m3's own published benchmark, not this corpus.

On Hindi specifically: 59.3. That's OpenAI's text-embedding-3-large - the same model we're testing against throughout this case, on both a public benchmark and our own data.

BGE-M3 paper, arXiv 2402.03216, Table 2; OpenAI embedding-models announcement

A real scanned page from the same bilingual Hindi-English archive this retrieval system runs on

Same archive as case one - texture, not the actual retrieved passage

The tell · the hosted model, text-embedding-3-large

OpenAI's hosted model ranked
the wrong passage higher.

Cosine similarity: how close two pieces of text sit in the model's meaning-space, from 0 to 1. Higher should mean "closer in meaning" - the right passage should score higher than a wrong one.

distractor passage

0.328

correct Hindi passage

0.262

same content, in English

0.082

It understands English fine. In Devanagari, it fell back to surface token overlap.

bge-m3 · written down before I ran it

Beat the baseline.
Without cheating to do it.

MRR: how high up the correct passage ranks, on average. A "hard negative": a wrong passage built to look right - the trap a model can pass by matching words instead of meaning.

MRR, higher is better

win line 0.403

0.461

hard-negative MRR, must stay low

ceiling 0.074

0.020

Without the second bar, a model that wins by matching surface tokens still passes.

Benchmark: beat the baseline on MRR. Metric that mattered: beat it without winning on hard negatives.

Chosen.

BM25 - a classic keyword-matching search ranker, no embeddings involved. The baseline to beat. Recall@10: of the top 10 results returned, how often the right passage shows up at all.

bge-m3 recall@10

64.4

BM25 recall@10

58.4

bge-m3 MRR

0.461

BM25 MRR

0.403

Benchmark: beat BM25 on recall and MRR. Metric that mattered: beat it without winning on hard negatives - and it did, at 0.020 against a 0.074 ceiling. All of this straight off the shelf - no fine-tuning on this corpus.

Case three · a different kind of system

Grading CCTV compliance,
against a client's rulebook.

An overhead warehouse camera view from a public dataset, illustrating the kind of top-down CCTV coverage this case is about

Not the client's footage or facility - a public overhead-warehouse dataset, same kind of coverage. "Warehouse Safety" dataset (CC BY 4.0), Roboflow Universe.

Long, unedited recordings. A model watches the footage and grades compliance - and that grade drives a real decision.

Hard constraint going in: this footage can't leave the client's infrastructure to any hosted model.

Third case, a completely different kind of system, and I want to explain what it actually is before any numbers, because this one isn't self-explanatory the way OCR or search are. This started as a manual process - someone sitting and watching long, unedited CCTV recordings by hand, to check whether the footage matches the client's own compliance rulebook. We proposed automating that with AI. That's not the client's footage, by the way, I can't show that - it's a public dataset, same kind of overhead camera coverage, different facility. So the idea is: a model watches the footage and grades compliance, and that grade drives a real decision, so a wrong one has real consequences. And there was a hard constraint going in from day one - the client didn't want their raw CCTV footage leaving their infrastructure and going to any hosted model. That shapes everything about what I could actually build here.

So before building anything, two public numbers

Frontier misses this sometimes.
SAM is a real detector.

Gemini-2.5-Pro, loose scoring (IoU 0.3)

74.1%

Gemini-2.5-Pro, strict scoring (IoU 0.5)

61.1%

TimeLens-Bench - can a model say when something happens in a video. Not what this pipeline was measured on - the closest public analog.

SAM 3 (cgF1, SA-Co/Gold)

54.1

best prior baseline (OWLv2)

24.6

Promptable concept segmentation, Meta's own benchmark. Neither number above ever asked whether a conveyor is extended into a trailer - no one publishes that benchmark, it's the client's own rule.

TimeLens-Bench, arXiv 2512.14698 Meta AI, arXiv 2511.16719

So here's the pattern I built

SAM 3.1.

Frontier model proposes.
Self-hosted SAM 3.1 verifies every cited frame.

Frontier model

reads footage,
cites frames

cites

SAM 3.1

self-hosted,
one GPU

verifies

Interface

shows only what's
defended

Self-hosted, because the footage can't leave the client's infrastructure. And to be precise: SAM 3.1 is a speed and video-tracking patch on SAM 3, not a re-benchmark of that 54.1 figure.

Measured three ways.
Right on one of three.

disagreed with the frontier model, 38 of 65 scored checks - it could only ask for "conveyor," never "extended into the trailer"

disagreed, 38agreed, 27

SAM's guess

right at the start

What actually happened

25 minutes in

Worst of three sessions on conveyor timing - right on one, off by up to 25 minutes on the other two.

Agreeing with the frontier model isn't the same question as being right.

The obvious objection: bad threshold, not a bad model

Minutes from ground truth · same prompt, three sessions

figures.md, threshold sweep, run 2026-08-05

Peak scores: 0.226 to 0.586. Same prompt. Same object class.

What the prompt asked for

"conveyor"

What the rule needed

"conveyor, extended into the trailer"

No score threshold fixes a prompt that's pointed at the wrong region of the frame.

Demoted.

Best config
in the sweep

correctwrongwrong

Excluding its
output instead

correctcorrectwrong

No threshold beat excluding it. So it stopped deciding - and kept running in the background.

Not a tuning problem - no cutoff fixes a detector answering the wrong question.

Benchmark instinct: does it agree with the frontier model. Metric that mattered: is it right against independent ground truth - and no threshold on the wrong prompt ever got there.

Not a tuning problem. Demoted. I tried every operating point in the sweep. Best case: one session out of three, correct. Excluding its output instead: two out of three, correct. That's what took it out of the verdict - it stopped being trusted alone, but it kept running. And this wasn't a tuning problem - no confidence cutoff fixes a detector that's answering the wrong question. My instinct going in was to check agreement with the frontier model. The metric that actually mattered was correctness against independent ground truth - and once the prompt itself was pointed at the wrong region, no amount of threshold tuning was ever going to close that gap. What was missing wasn't a better threshold, it was judgment: something that could read the same evidence and reason about it, not just score it against a fixed number.

Then a third layer

A Claude checker,
over both.

Frontier model

cites frames

Claude checker

checks, doesn't detect -
downgrades, never upgrades

Interface

what the user
sees

SAM 3.1

demoted - a grey trace line

Not one API call on the whole clip - a coding agent, in a sandbox, free to crop, rotate, and zoom into any frame before it answers.

132 callsabout 17s each, ~$1.50 total for the session

So that's what got added: a third layer, sitting over both. A Claude checker - it doesn't look at the footage itself, it looks at what the other two already said and decides whether that holds up. Here's the part that actually made it work, and it's not the model - it's how it was run. This wasn't a single API call fed the whole clip. It was Claude Code, a coding agent, put in a sandbox with the frames themselves. That meant it could crop into a corner of the image, flip it, zoom in, rerun the same frame from a different angle - actually go look, the way a person reviewing footage would, instead of committing to one answer from one fixed view. That's the entire reason it beat a plain API call: not a bigger model, a model with hands. 132 calls, about 17 seconds each, about a dollar fifty for the session - cheap, given what it replaced. Its contract stayed disciplined throughout: allowed to downgrade severity, never to upgrade it.

The canonical win, written up that same evening

SME ground truth, independent

00:25:00

Recorded before the model ran. Disagrees with the frontier model on a different session, so it isn't a copy of it.

Verifier rule 3, mandated

"Could not be confirmed"

Timestamp claims must return unverifiable absent a burnt-in clock - rule 3 names 00:25:00 as its own example.

A rule-produced category
got read as a finding.

The verdict was fixed before the model saw a pixel - "could not be confirmed" got filed as "caught."

Not the same mistake

Three different failures.

OCR: wrong metric Embeddings: wrong slice Verifier: faulty evaluator

One thing in common: the evaluation signal didn't represent the production question.

Every public number here was directional, not decisive - what caught each failure was a check I built for that task, not a bigger benchmark.

These are three different failures, not the same mistake - OCR was the wrong metric, embeddings was the wrong evaluation slice, the verifier was a faulty evaluator. What's actually common across all three: the evaluation signal didn't represent the production question. That's the thing to check for, before you trust any number. And notice what all three fixes have in common: none of them was a better public benchmark. Word accuracy, cosine similarity on a standard eval set, agreement-with-a-frontier-model - all directionally useful, none of them decisive. What caught the failure, every time, was a small, task-specific check I built myself. That's the actual argument for having your own internal benchmark: not instead of public ones, but because a public benchmark tells you the model is in the right neighborhood, and only your own check tells you if it does the one thing your product needs.

The honest complication

The model was rarely
the lever.

~2 pointsseparated the entire ten-engine OCR field

3-4xHindi misread improved, from one font-table fix - on all ten engines at once

If the question was whether a smaller model was good enough, that comparison is the answer. It was never about the model.

Every Hindi score I'd published before that fix was measured against a corrupted answer key. The numbers were wrong. No decision made on them was.

A runnable artifact, not a diagram

The bound, on one page.

$ uv run --script metric_blindspot.py the metric everyone reports character error rate 0.0340 reads as 96.6% accurate edits 79 over 2322 characters the metric the task needs identifiers expected 5 identifiers recovered 0 identifier recall 0.0% lost 1247, 1251, 1268, 1273, 1289 why the first number could not have caught it identifiers are 20 of 2322 characters, 0.86% of the page so CER can move at most 0.86 points between every identifier being perfect and every identifier being destroyed identifier recall moved 100 points on this same page

Identifiers are 0.86% of the page. Compute that ratio on your own corpus - that's the bound, before you run anything.

QR code linking to a browser page that runs the same identifier-recall check live

Scan to try it live - edit the text, watch recall move while CER doesn't.

Takeaways

Picking the number before you benchmark is the baseline. Two things past that:

1. The model was rarely the lever.

~2 points across ten engines · one font-table fix moved it 3-4x.

2. The verifier's cleanest win can be its own rule.

"Unverifiable by policy" filed as "caught a hallucination."

Two things, past the obvious one. Picking the number before you benchmark - that's the baseline, you already know that one. Two things past it. One: the model was rarely the actual lever. Ten OCR engines, about two points of accuracy apart - one font-table fix moved Hindi misread three to four times, on all ten engines, at once. Half of this talk was a model comparison. Check whether the leverage is even in the model before you spend that effort on the comparison. Two: a verifier's most confident-looking catch can be its own rule firing, not a finding. "Unverifiable by policy" got filed as "caught a hallucination" - on a timestamp that was actually correct. If you're stacking a verifier on a verifier, which is exactly what an agent harness is, the cleanest, most reassuring-looking catch is the one most worth doubting.