Stop choosing models by ranking position
Tests compare average tasks. In practice, each model needs to fit the kind of work.
People who start choosing AI models (programs that respond to requests) for different tasks often fall into the same trap. They open a table, pick the top-ranked model, and call it a technical decision. It looks objective, but often costs too much for a common task.
The problem is that a benchmark (a test comparing models on set tasks) measures an average. Your task is not an average. It may involve files or not, need tools or not, run once a day or all day. Maybe a slow answer is fine, or it must arrive while the person is still looking at the screen.
I stopped asking which model is best. Now I ask what kind of work I want it to do. That clears up most confusion.
Types of work
Writing and editing code in a large repository (a collection of project files) is its own type of work. Returning good text is not enough. The model needs to handle a lot of information, understand several files, and use tools to save changes. If it cannot write to the file, it only describes the work.
Hard reasoning and decisions about the system's structure fit another type. There are few calls, but each decision matters a lot. An expensive model can pay for itself because one mistake changes the direction of the whole system. Saving on each call costs more if the decision has to be made again.
Classification, summarizing, and extracting information are the opposite: many tasks, each narrow, with a short answer. A cheap model pays off when it is right often enough. Repeating the task is where the real cost comes from.
Embedding is a way to turn text into numbers so content can be compared. It happens outside the conversation. For Portuguese content, I want a dedicated model that works well across languages. Rerank comes next: vector search (using lists of numbers that represent text) finds nearby results. The re-ordering step picks which results help answer the question.
Reading images is another type of work. Looking at a screenshot or a scanned document differs from talking about clean text. Customer support chat, WhatsApp, and a virtual receptionist also do not need the smartest model by default. They need predictable response time and a per-message cost that holds up at volume.
File or text
The difference that guides my choices is simple: does the task save a file or return text?
When the answer is text, a much cheaper model may be enough. A classification, short answer, internal summary, or controlled extraction can be less accurate if the acceptance rule is clear. If it fails, you try again or ask for a review.
To save a file, the model needs a tool. It also needs permission, access to the project files, and a way to confirm what changed. Sending that work to a model that only returns text does not make the result slower. It creates a dead end. The model may return something plausible, perhaps a diff that shows changes. But no file changes on disk.
This also explains why switching models alone does not improve a personal assistant or marketing and sales work. What is missing is information about your business. Without your files, decisions, tone, and history, even the best model returns generic text. It may write better, but it does not know more.
How I choose
First I sort by type of work, not ranking:
- change files: a model with tools and room to consider lots of information;
- decide system structure: a strong model, with few calls;
- process high volume: a cheap model that meets the rule;
- search knowledge: embedding and rerank before generating an answer;
- handle repetitive conversations: response time and cost per message.
After that, the benchmark is secondary evidence. It helps compare models of the same type, but does not choose between different types. A model great at reasoning may be wasteful for labeling messages. A good chat model may be useless for editing a project.
The lasting rule is the fit between model ability and task. The table, price, and model name change. The question stays the same: can this call do the work I ask for? Can it do so at a cost and with a failure mode I accept?
Where it fails
Every model table is a snapshot in time. If you turn that snapshot into a fixed rule, the system ages without anyone noticing. The criterion stays useful, but the ranking position does not.
There is also false economy. The cheapest model that fails costs more than the call. You repeat the task, review it again, load the information again, and wait. Low price is measured by the result, not one line on the invoice.