Edition 4 · 05 / 06
Build for the Model You Haven't Chosen Yet
Practical engineering patterns for model substitution, including internal contracts, evaluation suites, shadow testing and a proportionate fallback path.
The objective is not to pretend that every model is interchangeable. They are not. The objective is to make substitution possible where the business consequence warrants it, and to know when an alternative has enough evidence to be used.
That is a practical engineering discipline. It begins by separating the organisation’s own business rules from the model-specific details around them.
Keep provider detail away from business logic
Provider abstraction is useful when it is honest about its limits. Keep SDK calls, model names, provider-specific request shapes and authentication at the edge of the application. Let the service own its internal representations of messages, tools, structured outputs, errors, usage, retries and observability.
An abstraction layer limits API coupling. Evaluation evidence is what makes behavioural portability measurable.
This reduces API coupling. It means a provider change has fewer places to reach into the system. It does not automatically eliminate behavioural coupling. A clean internal interface cannot guarantee that two models will interpret a prompt, call a tool or meet a schema in the same way.
Make the evaluation suite the centre of portability
The most valuable portability asset is not an adapter. It is a representative evaluation suite.
It should include golden datasets, real classes of production work, edge cases, tool-use scenarios, expected structured outputs, safety cases, latency thresholds and cost thresholds. The contents will differ for each service. A document extraction workflow may need field accuracy and traceable confidence. An agentic support route may need correct escalation, tool arguments and a safe failure path.
The key is that the suite belongs to the application. It describes the outcome that matters, rather than rehearsing a single model’s preferred answer. Run it against the primary model and against alternatives before an incident, then repeat it as models, prompts, tools and data conditions change.
This turns a claim that a service is portable into something that can be assessed. OpenAI’s model optimisation guidance similarly identifies evaluations as essential to measuring an LLM application.
Test alternatives without promoting them too early
Shadow evaluation lets teams compare an alternative against representative workload without exposing its output to users. It can reveal whether the model meets the required structure, latency and safety behaviour before it is put on a routing path.
It is not a licence to send every production request to another provider. Review privacy, contractual and geographical constraints first. Use synthetic or de-identified cases where practical, and establish who approves the secondary route before a disruptive event turns this into an urgent decision.
Choose a fallback path deliberately
A practical path often has three layers: a primary model, a validated secondary model and a reduced-function or human route. The last layer is important. Automatic fallback is not safe where the alternative has insufficient evidence or the action itself carries material risk.
The reduced mode might provide deterministic rules, pause a decision while retaining the case, ask a person to assess it or clearly withdraw the affected feature. What matters is that the organisation has designed the behaviour, rather than letting an unvalidated model make the decision because it happens to return a response.
The portability ladder
The following Beyond the Model framework distinguishes increasingly strong evidence. It is not an external industry standard and it is not a requirement that every workload reach the top level.
- Level 0Model captiveOne model, no fallback, no substitution test.
- Level 1API portableAnother provider could theoretically be integrated.
- Level 2Technically portableAn alternative has actually been integrated.
- Level 3Behaviourally portableCritical evaluations pass on an alternative.
- Level 4Operationally portableA tested fallback can restore capability within a defined objective.
This ladder describes progressively stronger evidence, not a requirement for every workload.
Level 0 is model captive: there is no known substitute. At Level 1, another provider could theoretically be integrated. At Level 2, an alternative has actually been integrated. Level 3 adds evidence that critical evaluations pass. Level 4 means the organisation has tested a fallback against a defined recovery objective.
The distinction prevents a common mistake: confusing an architecture diagram with a recovery capability. A second route only becomes resilience when it has passed the work that the business needs it to do.
Observe the dependency while it is healthy
For each significant model call, know which model and version answered, what tools it invoked, what latency and cost it incurred, whether a fallback was used and what evaluation or failure signals were present. Maintain a clear owner for the service and a record of the data, contracts and approvals needed to change the route.
Those records support day-to-day improvement as well as contingency planning. They allow a team to choose a better model when it is beneficial, while retaining evidence about what a change will alter.
Architecture can make switching possible. But would the organisation itself survive the interruption?
References
- OpenAI, model optimisation — official guidance on evaluation-driven LLM development; checked 12 October 2026.