Edition 4 · 02 / 06
Same API. Different Machine.
Why changing a model name does not necessarily restore the behaviour, controls or service level an AI product depends on.
At first sight, replacing an AI model can look trivial:
model = "provider-a/model"becomes:
model = "provider-b/model"The request is accepted. The endpoint returns a response. What could possibly go wrong?
Quite a lot, because a production system does not depend on an endpoint alone. It depends on a particular model doing a particular kind of work within a collection of surrounding assumptions.
The implicit behavioural contract
An ordinary API exposes a technical contract: request fields, authentication, response formats and documented errors. A model has that visible contract, but it also develops a much larger implicit behavioural contract with the application that uses it.
The endpoint is visible. The production contract beneath it is often only partly specified.
- Prompt and system-instruction interpretation
- Schema and structured-output reliability
- Tool selection, call format and retry behaviour
- Context, retrieval, safety, latency and cost
The endpoint may be replaceable. The behavioural contract may not be.
That contract may include how the model interprets the system prompt, how sensitive it is to a change in wording, whether it follows a JSON schema, how it chooses and formats tool calls, how it handles long context, its latency, tokenisation, caching, output length and refusal behaviour. Multimodal capability, retrieval behaviour and rate limits can matter too.
Some of those characteristics are documented. Many are discovered in production. They become visible when a model sees an unusual customer record, a badly scanned document, an ambiguous request or a tool result that contradicts the initial prompt.
This is behavioural lock-in. The organisation has not necessarily made a poor choice or been trapped by a provider. It has built a service around a capability whose important behaviour is only partly represented by an API specification.
A swap can work and still fail the service
Consider a workflow that receives a customer complaint, extracts information, classifies severity, decides whether to escalate and calls a tool. Both Model A and Model B below accept the same shaped request. The figures are deliberately fictional: this is not a comparison of real providers.
Illustrative demonstration
A compatible API is only the start
Fictional customer-complaint workflow: extract the complaint, classify severity, decide escalation and call the correct tool.
Model A is the validated production baseline.
All figures are fictional. A lower cost or compatible request shape does not prove operational equivalence.
The illustration makes a simple point. A working connection is not evidence that the substitute is ready to carry the same customer commitment. If a different model selects the wrong tool more often, takes longer, returns a different structured shape or applies a different safety boundary, the business has work to do before it can claim a recovery.
That work may be modest for a low-risk drafting task. It can be substantial for a process that triggers action, handles sensitive data or sits inside an operational service level.
Where the dependency hides
The dangerous dependencies are often the convenient ones. A prompt may have acquired examples that suit a specific model. A parser may quietly tolerate its characteristic formatting. A support team may know how to interpret its uncertain answers. A safety control may assume a familiar refusal pattern. None of that appears in model = "...".
Agentic systems add another layer. Tool selection, tool arguments, retries and hand-offs create paths that are hard to inspect from a single answer. A replacement can produce prose that looks credible while invoking the wrong action, omitting a condition or approaching a workflow in a different order.
This does not mean that models are incomparable. It means comparison must be made against the work that matters. The right question is not “Which model is best?” It is “Which model can safely perform this particular service to the level we have promised?”
Evaluation is portability evidence
An evaluation suite makes that question measurable. It can contain representative production cases, golden datasets, difficult edge cases, structured-output checks, tool-use scenarios, safety cases, latency thresholds and cost thresholds. It should be owned by the organisation, not merely inferred from one provider’s examples.
The same suite can be run against the primary model and a realistic alternative. That turns a hypothetical replacement into evidence. OpenAI’s model optimisation guidance similarly treats evaluation as the way to measure and improve an LLM application.
For some workloads, shadow evaluation is useful: run representative requests through an alternative without using its output in the live service. That needs care. Sending customer data to another provider may introduce fresh privacy, security, contractual and cross-border-processing questions. Synthetic or de-identified cases may be the appropriate first step.
Not all differences require an automatic fallback
The answer is rarely “route everything to a second model”. Automatic fallback is appropriate only when the alternative has enough evidence for the task and the system can safely make the change. Otherwise the right action may be reduced functionality, deterministic rules, a visible pause or human escalation.
An abstraction layer can make a swap easier to attempt. It cannot make the difference disappear. API compatibility is technical. Behavioural compatibility is operational.
If models are not interchangeable, how often do they actually change or disappear?
References
- OpenAI, model optimisation — official guidance on measuring LLM application performance with evaluations; checked 12 October 2026.