The Most Dangerous AI Systems Are the Ones That Look Healthy

Most writing about AI reliability treats “healthy” as a measurement problem. Pick the right metrics, instrument the right signals, and the truth will emerge from the data.

That framing understates the problem. In every team I have seen ship a generative system into production, “healthy” is not a measurement. It is a claim — made by someone, accepted by someone else, and rarely re‑opened once the launch is over. The dashboard is the artefact. The claim is the thing actually running the system.

This is the layer where the most dangerous AI systems live. Not the ones whose behavior degrades after launch — those are the subject of a separate problem. The dangerous ones are the systems that were never healthy in the sense the organisation believes they are, and whose definition of healthy has been quietly inherited from a different kind of software.

A representative launch

A new internal copilot ships to a 400-person operations team. The launch review goes well:

• 99.95% successful API calls in the staging soak.
• A 50-prompt eval set passes at 92%, up from 87% the previous quarter.
• The product manager demos five workflows; all five complete cleanly.
• The model provider’s status page is green.
• The Slack channel for the launch fills with thumbs-up.

The system is declared healthy. Headcount on the project drops by half within two weeks. The dashboard is moved to a TV in the office. Nobody opens it after the second month.

Six months later, an internal audit finds that the copilot has been quietly producing incorrect figures in roughly 8% of pricing-related conversations since the second week after launch. The eval set never covered pricing intents at the granularity the ops team actually used them. No metric on the dashboard would have surfaced it. The system has been “healthy” the entire time.

Nothing failed. The definition of healthy was wrong on day one, and the organisation had no mechanism to notice.

Four patterns to recognise

These are organisational, not technical. They describe the team and the process around the system, not the model.

• Health by inheritance. The team adopts the reliability vocabulary of web services — uptime, latency, error rate — without asking whether those words describe success for a generative system. The metrics are real; they are just measuring the wrong thing. This is the default state of most AI launches.
• Green-dashboard consensus. Once a dashboard exists and is green, “healthy” becomes the social default. Disagreeing with it requires producing counter-evidence under time pressure, and most people do not. The dashboard becomes a coordination device for not investigating.
• Owner ambiguity. Infrastructure is owned by SRE. The model is owned by an ML team. The prompt is owned by a product manager. The eval set is owned by whoever wrote it. Behavioural quality is owned by no one — which means “is the system healthy?” has no one to ask, so it gets answered by whichever metric is loudest. The operational version of this problem is treated in a separate post; here the issue is who is allowed to make the claim.
• Trust debt. Each small unaddressed degradation slightly lowers user trust. Trust does not recover on its own. A system can accumulate months of trust debt while every individual incident looks too small to act on, and then experience a sudden, unexplained drop in usage that the team treats as a product problem rather than a reliability one.

A team can have excellent telemetry and still exhibit all four. The patterns are about who is allowed to say the system is broken, with what evidence, and how easily that claim can be ignored.

Heuristics for a more honest definition of healthy

The heuristics here are not metrics. They are questions the team should be able to answer at any time without preparation.

• Who is the named owner of behavioural quality for this system? If the answer is more than one person, or more than one team, the answer is effectively no one.
• When was the eval set last refreshed against live traffic? If it was written before launch and has not changed since, the eval is testing a system that no longer exists.
• What evidence would cause the team to declare the system unhealthy? If the team cannot name a specific signal and threshold in advance, “healthy” is unfalsifiable, and unfalsifiable claims are not engineering claims.
• Who is allowed to call an incident on behaviour alone, with no infrastructure signal? If only SRE can declare an incident, and SRE only watches infrastructure, behavioural incidents will not exist as a category.
• How often does the team look at production conversations directly, not summarised, not sampled by a dashboard? Teams that read raw output weekly find regressions weeks before teams that only read charts.
• What is the half-life of the launch dashboard? If no one has changed a panel in three months, the dashboard is no longer instrumentation. It is decoration left over from a launch.

These questions are uncomfortable on purpose. The discomfort is the diagnostic.

What to watch for

Organisational signals that “healthy” is doing more work than it should:

• The system has not had a declared incident since launch, and the team treats this as a success rather than a question.
• Metrics on the dashboard have not moved in 90 days, and no one finds that suspicious.
• User complaints arrive through customer success, not through the on-call channel.
• Behavioural regressions, when found, are described as “product feedback” rather than incidents.
• The original eval set is still in use, written by someone who has since left the team.
• Headcount on the system dropped sharply after launch, and the remaining owner also owns three other things.
• Nobody on the team can answer “what does healthy mean for this system?” without first opening the dashboard.

The dangerous AI systems in your organisation are not the ones currently on fire. They are the ones that look fine, have looked fine for a long time, and whose definition of “fine” has not been examined since the launch deck.

Healthy is a claim. Treat it like one. Ask who made it, on what evidence, and when it was last re-examined — and be willing to discover that the most stable system you operate has been quietly wrong for months.