# Good Enough for What? - Transcript

Hi everyone. I hope you have all the refreshments you need, and refreshed enough, so that we can see how refreshing benchmarking can be. So, I'm Prudhvi. I work as a data scientist at Straive - well, the title says data scientist, but I mostly get to explore and exploit what LLMs and GenAI, or data science things in general, can do, partly to get some benefit out of the table that we can pitch to clients. So that's my day job. Title says data scientist, but my work is partly inclined towards GenAI - partly because of business needs, partly because I'm interested.

So while I was exploring and exploiting things, I get these questions from people - what does this do, what does that do. But this year, an interesting pattern came up: the same question was being asked thrice, which is, is the cheaper model, or are the small models, good enough that we can use. And I got the answer wrong for the first iteration every time - not because the model was doing badly, but because I was reading the wrong number that I needed to evaluate on for the use case.

So I'll get started with what the use cases are, high level for now, and we'll go into detail while talking through them. There are three use cases where this question was being asked. One is an OCR engine - we're working with a government client where the corpus of their documents is scanned, bilingual, mostly PDFs, English plus Hindi. So the question is, can we use small models, or a cheaper model, to get that accurate - and accurate is subjective, I'll come to that. The second use case is building a RAG pipeline on top of the same corpus I just talked about, which again is bilingual, Hindi plus English, so the RAG pipeline needs to reason in Hindi as well. Third is an independent and interesting use case we've done - it's CCTV compliance. Think of it as a client where they have n number of warehouses, n being in the thousands, where they have CCTV cameras across their warehouses and their own compliance rules that they check against. That's usually how the client's warehouses work, and we did something interesting there that I'll share.

## Case one: OCR

So let's jump into the first one. As I mentioned, we're working with a government client where the corpus is Hindi plus English, bilingual, so the OCR needs to handle that. Someone challenged me once, saying Tesseract is a free, local OCR engine you can directly use against the hosted models as an option. So, when someone comes to me and asks which are the better models for OCR, I'd immediately say Gemini, because most of the data Google's models are trained on is this kind of image and document data. So I'd say Gemini, apart from benchmarking - Tesseract recently has also been doing a pretty good job on Hindi OCR, and apparently the cost is also less, partly because it's costing us directly in rupees, not dollars, and partly because it's OCR priced per page rather than per token - all the hosted, Gemini-style models will cost per token, input and output.

So there's a public benchmark where Tesseract, on general OCR, gets almost 87% accuracy, where hosted models are in the 92 to 95% range. But Tesseract is a free model - remember that, and we can train Tesseract. Across this talk, whenever I say "small model," it's not a model fine-tuned yet with our data - it's a model available as-is, on Hugging Face or via an API, or on a modest GPU that I have a machine running. So it's not trained - that's one factor to remember. Fine-tuning itself is a different topic, worth its own discussion on a different track.

Coming to the point: this is the benchmark, and I was almost going to say Tesseract would be a good model - but I don't trust myself on just the numbers. So my practice is to test with the actual data - the client usually gives some sample data, so I test on that to see what's working, at least for the first iteration. What I did is use Tesseract and Gemini models to check the accuracy on the actual corpus. This is what the corpus looks like - it has questions, it has question numbers, questions and answers, questions in one PDF and answers in another, and the way it links up is by the question number. If you look, there's a "*1883" - that's the question number, along with it being a scanned PDF. For context, these are questions asked in the assembly, and the answers get linked up later as part of the document, so it needs some kind of RAG on top of it to ease things for their users.

What I realized is: Tesseract tested at 87% - good to start with, and maybe fine-tuning would do a pretty good job free, not considering training costs. But the metric the internet quotes, that 87%, is word accuracy - the ability to get the word right. What it misses entirely is the number. If you remember, "*1883" is the question number, but Tesseract returned it as "+883" - a completely different question, almost a thousand numbers off. If I'd taken that into production, the question would have been completely disconnected from its answer. I didn't realize this until I took a closer look at what was actually being retrieved. The hosted model, meanwhile, was doing a pretty good job getting the numbers right - it got the star correct, it got the 1883 correct. Word accuracy is a different part of the story, but on question numbers specifically, the hosted models - Gemini, even Claude - were doing 100%.

So the metric that mattered here wasn't the accuracy the internet reports for me - it was the accuracy of the numbers. In my case, I passed a three-page sample PDF that had both embedded text and content that needed OCR - image to text. The question-number retrieval accuracy came out to 68% for Hindi and 39% for English - not because the numbers themselves differ between English and Hindi, but because how they're encoded in the English versus Hindi corpus is different. Hindi does fine; English is pretty bad at retrieving the numbers. The hosted models, meanwhile, were 100% - never missed a number, never missed a star before it. So if I'd trusted the benchmark and gone ahead, I'd have missed the numbers, and it would have collapsed the entire purpose of having a RAG corpus on top of it. The benchmark that mattered publicly is word accuracy; what mattered for my case was the retrieval of the numbers. So I rejected Tesseract - because it didn't get the numbers right, and that collapses the entire purpose. That's case one.

## Case two: embeddings

Now for case two. The first part is done - I have the data, OCR'd into text, so now I can do embeddings. Same question again: what are the smaller or cheaper models I can use? When you say embeddings and come to me, I'd say OpenAI's text-embedding models, because I've used them and they're accurate. But one that's well known for good accuracy on multilingual, bilingual work is bge-m3 - it's a 570-million-parameter model that runs on a modest GPU, even a laptop GPU, and I have one to run this. I compared it against text-embedding-3-large from OpenAI, because OpenAI claims almost 55% accuracy on non-English languages - German, Spanish, and apparently Hindi, since Hindi is one of the most common languages in India. bge-m3 claims 70% - and that's a public number, not my own measurement - so it's winning by a wide margin on paper.

My obvious instinct was to just go with it, but again, I wanted to test on the same corpus I'd just retrieved. What happened is, testing both bge-m3 and the OpenAI model, the interesting case I didn't expect: the OpenAI model was doing fine on plain word accuracy, around 55%. But what I hadn't considered enough is the one thing that actually matters in RAG - cosine similarity, the distance between two points, 0 to 1, 1 being closest to what you're looking for. What text-embedding-3-large was doing was ranking the wrong passage higher - a false positive, which is exactly what you want to keep at zero. When I looked closer, it was matching keywords rather than meaning. For example - I don't know Hindi myself, but if there's a Hindi word, it was matching the keyword rather than what the sentence actually means. That's not what I want - I want a RAG that answers by meaning. It was doing a pretty good job in English, though - ranked around 0.08, very close to zero, meaning it correctly rejected the non-relevant passages there. So the OpenAI model was fine in English, just not in Hindi.

Then I tested bge-m3 the same way, with a few metrics - MRR, mean reciprocal rank, how high on average the correct passage ranks. It passed well on the first iteration itself, without any training - and if it does this well untrained, I could use the client's data to train it further and do even better. So what did I choose? bge-m3 - not because the benchmark said it's 15 points ahead of the OpenAI model, but because I saw the text-embedding model ranking the wrong passage, and bge-m3 doing a genuinely good job on the actual corpus. That's case two.

## Case three: CCTV compliance

The third, and I'd say the most interesting thing I've done: a client came up with this - they have warehouses, they have CCTV cameras. I believe we have CCTV cameras here too, fixed in one position, just capturing the whole room. So someone at their end, sitting from morning to evening, just watches CCTV footage to check compliance. Compliance can be as simple as a person wearing a vest of their branded colour, or more complex. For example - imagine a conveyor belt like this, and it needs to be extended from here to there to unload packages from the containers it receives from. I couldn't show the actual footage since it's client data and sensitive, so I'm using a public dataset image just to give the sense. That's the use case I'll refer to for the next few slides: "conveyor extended to the trailer." One CCTV camera per store captures everything.

So again, at this point if you ask me which models - hosted or small - do well at detection, I'd say Gemini again, for the same reason: the data Gemini's trained on, text and images both, is good for this. Not that Claude or open models are bad - Gemini is just good here. There's a benchmark called TimeLens-Bench that I found while exploring - it benchmarks exactly this skill: can a model tell you when something happened, rather than just what happened. You give a video to an LLM and ask what happened, it does a pretty good job - but what I need here is for it to trigger something the moment something happens. If someone crosses the belt left to right when it's extended, that's against the client's own rule, and it needs to actually trigger on that.

On TimeLens-Bench, an older Gemini 2.5 Pro, from a few months back, scores about 74% under loose scoring - meaning roughly two out of three times it gets it right, one out of three it fails - and under strict scoring it drops to about 60%, so 40% of the time it's still failing. Similarly, SAM 3 - a Meta model, SAM 3.1 being the latest, extended version - scores about 54% loose, against a previous best baseline of around 24%. That's roughly double the previous best. But 54% is still low - not the accuracy we actually want. So looking at these numbers, I had an idea: the frontier model does about 74% of the job, SAM does 54% - what if I combine them? What if the frontier model detects when something's happening, and SAM verifies it, so the combined accuracy lands around 80%? That was my first instinct.

So what I did: I used a frontier Gemini Pro model - 2.5 Pro I believe, a few months back - as the frontier layer, passing it frame by frame, passing the compliance rules and tools, to detect if something happened - say, someone crossed the belt left to right when it shouldn't have, and to trigger on that. In this setup, the frontier model detects, SAM verifies, and it comes to a human-in-the-loop review.

What I found: I measured this three ways. First, SAM disagreed with the frontier model on 38 of 65 scored checks. Looking closer at SAM's outputs, I realized it's not checking the rule I actually meant - it's checking object names, not relations. When I say "conveyor belt extended to the trailer," it's only detecting "conveyor" or "trailer" - not "conveyor extended to the trailer," which is a completely different thing. SAM doesn't know English, in that sense - it only knows object names. So even training SAM further, or training a YOLO-style CV model, for that matter, wouldn't make sense for me. And I also saw a hallucination case: SAM and Gemini together flagged an incident, but the actual event was 25 minutes after the timestamp it cited - so partly it's hallucinating the timestamp rather than the event itself.

The obvious question is: is the setup bad, or is the model bad? Easy enough to check - I had the CCTV footage the client had sent, and I reran three sessions of about four hours each, offline, to see how the accuracy drifts. And it confirmed exactly what I'd said - when I say "conveyor extended to the trailer," I mean the relation; what SAM assumes is just "conveyor." So whenever the pixels hit SAM, it's just looking for a conveyor and reporting back, agreeing or disagreeing with the frontier model. At first glance, 38 out of 65 disagreements looked like either Gemini or SAM was hallucinating. But then I realized SAM is only ever detecting the conveyor - so on other rules, say a vest rule, it's only looking for the vest, not for a human actually wearing the vest, or whether the vest is visible at all. So I demoted it - not because it's a bad model, but because it wasn't functionally working for me. Even training it further wouldn't help, because it can only ever look at objects, not the relations I actually need it to judge.

## The third layer

So, me being me - I'm a heavy Claude Code user - the thing with coding agents is the intelligence of the LLM combined with tool calling, which is what makes them useful with the right guardrails. So what I suggested: try Claude as a checker on top of both. Now, Claude here isn't just an API call - that would be exactly what I did in the first iteration with Gemini, just swapping the model or provider rather than changing the approach, so that wouldn't make sense either. I chose Claude Code specifically for its tool-calling ability. We put it in a sandbox, we put in guardrails, we configured hooks - mostly because hooks can actively deny what shouldn't be done, rather than relying on a memory file or a learnings file that it might just ignore. Hooks can reject access directly and continuously.

The benefit here: with tool calling and agentic ability, it can take a frame, resize it, rotate it, look from another angle - for example, if the CCTV camera view makes it ambiguous, it can rotate and look from a different perspective to check if something's actually happening, and it does correctly understand "conveyor extended to the trailer" rather than just "conveyor." It went surprisingly well. But it still has the same kind of timestamp issue, still has false positives, still returns "could not be confirmed," still has some hallucinations - but we came up with solutions for that, and we're suggesting this instead of an API call or a small trained model: have an agentic model, or agentic tool capability, or your own agent with tool abilities to resize, crop, edit, or look from another perspective in your own environment. That's the best thing you can do, because if you just go with an API call, the human reviewing it ends up with more work than before - before, they only had to watch the CCTV and check a list; now they have to do both, because the output isn't accurate enough on its own, and that undermines the whole point of automating their job. So we suggested this, and we're trying to get it into production - though we do have challenges, since coding agents at scale mostly cap out at 12 to 15 parallel instances, not more. That's what we're working around. But it's a good finding - one we couldn't have found in any benchmark, because no benchmark says a coding agent does this well at finding objects in this way. So that's the direction we're pursuing.

## The pattern, and two things past the obvious one

So these are three different failures. OCR: I was looking at the wrong metric. Embeddings: I was looking at the wrong slice, because wrong passages were being ranked highly. And third: a faulty evaluator itself - I was treating SAM as an evaluator rather than a detector, and that's a faulty way to use it. So the one thing in common between all three is the evaluation signal - the signal itself never actually asked our production question, which is: which model is good for this. It never answered that directly. But the exploration around it does answer the question, across the whole process. The model was rarely the lever - it's never really about the model, it's about the use case you have, and the approach you take.

We're almost out of time, so I'll skip past the QR code slide - it just gives you an in-browser check on your phone for the recall example, feel free to grab it later. Two things, past the obvious one. Picking the number before you benchmark - that's the baseline, you already know that one from the public benchmarks. Two things past that: one, the model is rarely the actual lever. The model is a problem, sure, but it's not the only problem - check whether the leverage is even in the model before you spend the effort on a comparison. Half of this talk went into comparing Gemini models against other models, this model against that model - but you have to evaluate whether that effort is even worth it for your case. And second: a verifier's most confident-looking catch can be its own rule firing, not a finding. SAM agreed with the frontier model as a reviewer, and the confidence it came with was good enough that I nearly shipped it - but a verifier can make mistakes too, even when its confidence looks fine. You have to look at that as well.

So coming back to the title - good enough for what. Ask what made you reject it, ask what made you accept it, and do that before you run the benchmark. That's what tells you what to keep and what not to keep, for your particular problem and your particular problem statement. I'm open for questions, or feel free to find me later.

## Q&A

**Host:** Yeah, you can take questions.

**Audience member:** [inaudible]

**Speaker:** Great question. We did evaluate on a bigger set just for this talk - I picked the top two performing candidates into clusters, one hosted and one small model, for the slides.