Stop picking models by benchmark

Everyone who starts routing AI falls into the same temptation: open a table, pick the first place in the ranking, and call that a technical decision. It looks objective. It is also often the most expensive way to solve a common task.

The problem is that a benchmark measures an average. Your task is not an average. It either has files or it does not, needs tools or does not, runs once a day or runs all day, tolerates latency or needs to answer while the person is still looking at the screen.

What I do today is stop asking "which model is best?" and ask "which work band am I trying to execute?". That question removes most of the confusion.

The work band

Writing and editing code in a large repository sits in its own band. Returning good text is not enough. The model needs to handle long context, understand several files, and use tools to change the disk. If it cannot write the file, it is not doing the work; it is describing the work.

Hard reasoning and architecture sit in another band. These are few calls, with high value per decision. Here the expensive model can pay for itself, because one mistake changes the whole direction of the system. This is not where I save per call if the consequence is making the decision again.

Classification, summarization, and extraction are the opposite. High volume, narrow task, short answer. The cheap model wins when it is consistently good enough, because the real cost is repetition.

Embedding also does not belong to conversation. It turns text into a vector. If the content is in Portuguese, I want a dedicated model that is good with multilingual content. Rerank comes after that: vector search approximates, the reranker decides better what actually serves the question.

Vision is another band again. Reading a print, screenshot, or scanned document is not the same as talking about clean text. And support chat, WhatsApp, and a virtual receptionist do not ask for the smartest model by default either. They ask for predictable latency and a cost per message that survives volume.

File or text

The distinction that organizes my routing is simpler: a task that writes a file versus a task that returns text.

When the output is text, the band can be much cheaper. A classification, a short answer, an internal summary, a controlled extraction: all of that can accept degradation if the acceptance criterion is clear. If it fails, you repeat it or mark it for review.

When the task writes a file, the game changes. It needs a tool, permission, repository context, and the ability to confirm what changed. Sending that work to a text-only model is not a slower option. It is a dead route. It returns something plausible, maybe even in diff format, but nothing changed on disk.

This difference also explains why a personal assistant and marketing or sales do not improve only by switching models. The bottleneck is context. Without your files, your decisions, your tone, and your history, the best model in the world returns generic text. It may write better, but it does not know more.

How I choose

I split first by type of work, not by ranking:

• act on files: model with tools and long context;
• decide architecture: strong model, few calls;
• process volume: cheap model that passes the criterion;
• search knowledge: embedding and rerank before the generator;
• handle repetitive conversation: latency and cost per message.

After that, benchmark becomes secondary data. It helps compare inside the same band, but it does not decide between different bands. A model that is excellent at reasoning may be waste for labeling messages. A good chat model may be useless for editing a repository.

The durable criterion is the fit between capability and task. The table changes. The price changes. The model name changes. The question stays: can this call do exactly the work I am asking for, with the cost and failure mode I accept?

Where it breaks

Every model table is a dated snapshot. If you turn that snapshot into a fixed rule, the system ages silently. The criterion lasts, not the ranking position.

There is also false economy. The cheapest model that fails does not only cost the call. It costs the whole round again: another attempt, another review, more context, more waiting. Cheap is measured by the result, not by the invoice line.