Edition 1 · 06 / 08
Is Your Supplier Replacing Experience With AI?
Watch for patterns in review capacity, service quality and technical ownership, rather than treating layoffs or faster releases as proof.
Smaller teams and faster releases do not prove that a supplier is taking unacceptable risks with AI. Redundancies can reflect restructuring, falling demand or a merger. Frequent releases can reflect excellent engineering.
The concern is a pattern: output expectations rise while experienced judgement, review capacity or operational knowledge declines. Customers need evidence about that combination, rather than a conclusion drawn from a single headline.
Measure outcomes alongside activity
DORA’s 2025 review describes AI as improving throughput while potentially costing stability where the underlying engineering foundation is weak. That supports examining the delivery system as a whole. It does not establish that AI caused a particular supplier’s incident or that smaller teams necessarily perform worse.
Track release frequency with rollback rates, hotfixes, serious incidents and escaped defects. A faster cadence accompanied by stable quality may be welcome. A faster cadence with mounting regressions deserves investigation.
Use consistent definitions and compare a supplier with its own baseline. Changes in workload, product scope or incident reporting can distort simple comparisons.
Preserve the judgement layer
Ask who owns architecture, security, release approval and incident response. Are those responsibilities still held by people with the experience and time to discharge them?
Human review is not meaningful merely because a person clicks an approval button. Reviewers need context, competence and enough capacity to challenge the output. An expanding queue of generated changes can overwhelm a nominally unchanged review process.
Other indicators may include recurring support misunderstandings, slow escalation to specialists and difficulty explaining previous design decisions. These are reasons to ask further questions. They are not reliable proof that a supplier secretly uses AI.
The warning sign is a weakening ability to recognise, explain and recover from mistakes.
Use a proportionate scorecard
A useful review can consider three broad states:
- Green: AI adoption grows while expert ownership, testing and service outcomes remain healthy.
- Amber: review capacity or critical expertise appears thinner and delivery expectations are increasing; further evidence is needed.
- Red: worsening incidents and rework coincide with weakened ownership and inadequate evidence of effective controls.
This is a discussion aid, not a validated prediction model. Avoid turning weak signals into an automatic judgement. Record the evidence, the supplier’s explanation and the agreed follow-up.
Ask what has changed in your service
Focus the conversation on the team and process delivering the work you buy. Request an explanation of how AI has changed task allocation, human review, testing and escalation. Where staffing information is commercially sensitive, seek proportionate assurance about the capabilities that remain.
The most revealing question is: what evidence shows that increased output has not been purchased by removing the experience needed to recognise when the AI is wrong?
References
- DORA: 2025 year in review. Research context for throughput and stability. The warning signals and scorecard are editorial assessment proposals, not causal findings about individual suppliers.