Picking the Right Model: Cost, Latency, and Quality
No public benchmark can tell you which model fits your task, a tiny private eval can. How to choose: optimize cost per successful outcome, use the thinking and effort dials, and shift the curve with caching and context hygiene.
Guillaume Rufenacht
AI Product Manager · Lisbon
Every model launch comes with benchmarks, prompting guides, and a wall of hot takes ranging from “AGI is here” to “they’re cooked.” None of it answers the only question that matters for you: should I use this model for my task? The honest answer is that no public benchmark can tell you, and a tiny private eval can. Picking the right model is a measurement problem, not a vibes problem.
Making this call well is a core part of running production LLM pipelines at Geonimo. Here’s the framework I use.
Key takeaways
- A small, well-designed private eval beats any public benchmark for choosing a model for your use case.
- Optimize for cost per successful outcome, not cost per token. A pricier model can be cheaper once it succeeds more.
- More intelligent models are often faster and use fewer tokens, they plan better and take fewer turns. 'Smaller = faster' is a trap.
- Use the dials: model tier, thinking (a scratchpad to reason), and effort (how hard it works) to move along the cost/quality curve.
- Shift the whole curve with prompt caching and context hygiene, often Opus quality at Sonnet cost.
Build the eval first
Public benchmarks (coding, research, and so on) are directional at best. Your real workload is messier and more specific, mixed tasks, niche languages, your own data, and a model can top a leaderboard while underperforming on exactly your job. So build a small eval: a set of tasks, each with inputs and clear success criteria. Grade the outcome and the working, like a math exam, an agent can reach the right answer the wrong way. Mix LLM-as-judge (for fuzzy correctness) with deterministic checks (did it call the right tool with the right arguments). It’s real work, and it’s the highest-leverage thing you can do before betting a product on a model.
The eval gotchas that bite everyone
Optimize cost per successful outcome
This is the reframe that changes decisions. The right model isn’t the cheapest per token, it’s the cheapest per successful outcome. Counterintuitively, a more capable model is frequently faster and cheaper in practice: it plans better, takes fewer turns, and doesn’t burn tokens flailing. Teams routinely find a top-tier model finishing a task in less time and fewer tokens than a smaller one that needed five attempts. “Smaller equals faster” is a vibe, measure it.
Use the dials: tier, thinking, effort
You have more control than just the model name. Beyond choosing a tier (more intelligence vs lower latency/cost), modern models expose thinking (an adaptive scratchpad to reason before acting) and effort (how much work to spend across thinking, tool calls, and responses). You can mix them, low thinking with high effort, or no thinking with a high effort budget, to land precisely where you want on the cost/quality curve instead of taking whatever a default gives you.
Shift the curve, don’t just move along it
The biggest wins change the trade-off entirely:
Prompt caching. Reusing a cached prompt prefix costs a fraction of the input-token price, in effect, top-tier quality at mid-tier cost. The trick is an append-only message history (no volatile variables like a timestamp in the system prompt, which silently breaks the cache). Aim for an 80-90% cache hit rate.
Context hygiene. Clean up what you feed the model, markdown instead of bloated JSON, dedup results, trim noise, and you can cut tokens by half or more and improve accuracy, because the model reasons over cleaner data. This is the practical edge of context engineering, and it compounds every turn.
The takeaway
This is the daily work of shipping LLM products, part of building with Claude Code and what I do at Geonimo. See how I build or get in touch.
Frequently asked questions
How do I choose which AI model to use?
Build a small private eval for your actual task, a set of inputs with clear success criteria, and run candidate models against it. A well-designed eval beats any public benchmark for your specific use case.
Is the cheapest model the most cost-effective?
No. Optimize for cost per successful outcome, not cost per token. A more capable model often finishes in fewer turns and fewer tokens, making it cheaper and faster in practice despite a higher per-token price.
What do the thinking and effort settings do?
Thinking is an adaptive scratchpad that lets the model reason before acting; effort controls how much work it spends across thinking, tool calls, and responses. You can mix them to land precisely on your cost and quality target.
How do I get higher quality without paying more?
Shift the curve with prompt caching (a cached prefix costs a fraction of input-token price) and context hygiene (markdown over JSON, dedup, trimming), which cut tokens and often raise accuracy at once.
Work with me
Want a system like this built for your pipeline?
I help teams take AI from a clever prototype to dependable production, outbound engines, lead intelligence, and the LLM pipelines underneath. See what I have shipped or get in touch.