Your fallback probably is not a fallback

Setting up fallback for AI looks easy. You take the preferred model, put another below it, maybe a third one, and that is it: if the first one fails, the system tries the next.

That list is almost always false.

The mistake is treating a model name as the unit of resilience. In practice, a quota fails, a subscription fails, a capability fails, a tool fails. If the next step does not separate those things, you do not have fallback. You have a nice-looking queue that collapses in the same place.

Redundancy is not a different name

The first trap is counting redundancy by the name in configuration. Two different clients may point to the same subscription. Two models may consume the same meter. Two routes may depend on the same administrative resource.

When the quota ends, both fail together.

That is why I started thinking about fallback by resource and meter, not by label. The question is not "do I have two models?". It is "if this limit runs out, does the next step still have an independent limit?". If not, the list only created an illusion of depth.

This applies to provider, project, account type, and any layer that can deny the call before the model reasons. The system does not need every private infrastructure detail. It needs to record which steps share the same failure point.

The same thing

The second trap is worse because it passes as success. The first step edits files. The second only returns text. The call fails on the first, falls to the second, the second answers with a plan, maybe even with a formatted patch, and the automation marks it as done.

Nothing happened.

Fallback only counts if the next step knows how to do the same thing. If the task needed to open a tool, write a file, read an image, or call an external system, the fallback needs that same capability. Degrading quality may be acceptable. Silently changing the task is not.

Failing loudly is better than degrading in hiding. A clear error stops the pipeline and says a capability was missing. A false success moves forward and delivers absence as if it were finished work.

That is the distinction I use: model fallback is not capability fallback. If the capability changes, the route needs to change state. It can become "answer in text", "request review", or "block". It cannot keep the same contract.

Temporary is not definitive

The third trap is confusing a momentary failure with absence. A rate limit says the call was not measured at that moment. It does not say the resource is gone forever. An unauthorized or not found response speaks about another class of problem.

When the router treats every failure the same way, it knocks out a healthy step. A temporary spike becomes false proof that the route died. Then the system uses a worse path, even after the first one has recovered.

That is why fallback needs to record cause, not only destination. "Fell to the second model" is not enough. I need to know whether it fell because of a limit, unavailability, permission, missing resource, or task incapability. Without that, you can only hope.

One passage point

The most boring way, and the one that works best, is to have a single gateway. Every call passes through one point that knows the requested task, required capabilities, independent steps, and which model actually answered.

That last record is essential. Without it, you think the system is using the first step. In reality, it may have been surviving for weeks on the last one. The product still works, but the margin is gone.

Fallback that works is not a list. It is a contract: this work requires these capabilities; these steps are independent; this failure can retry; that other one must stop.

Where it breaks

A single gateway also becomes a single point of failure. If it goes down, everything that depended on it goes down too. The fix is treating the gateway as real infrastructure, with observability and a manual path for critical operations.

Route records are also sensitive data. They reveal what you send out, when you send it, and which class of provider it passes through. They need to be stored as operational data, not as loose logs.

And automatic routing hides degradation. If nobody watches the telemetry, you discover the good step died on the day the last step dies too.