The discipline of reliability engineering is among the most successful of our era for keeping an institution’s beliefs about its own systems honest. While the frontier labs’ concentration on AI safety genuinely does embrace this flavor of discipline, systems work outside of the models themselves remains a second class concern. But safety claims, which are the subject of so much discussion, need to also be true of actual deployments of AI products. The industry needs to be honest about that divide, and be transparent with users about how they close it. Here’s how it can happen.

Micro-policy at midnight

Consider an AI infrastructure scenario where a safety monitor in the serving path goes down in the middle of the night. The design of this hypothetical system gives deference to the on-call engineer: whether to wall off the serving stack from the user base until the safety monitor comes back online, or to keep the product exposed to the public without it.

Outside of a good reliability engineering discipline, the reasoning in such scenarios is typically that the cost of shutting down is immediate and certain (customer tasks fail, revenue halts, and metrics turn red) while staying online seems cheap: “Just a small chance that something bad occurs during the degraded window, until the safety monitor comes back up!” On this reasoning, the on-call engineer’s decision winds up being to keep serving the product to the world. Perhaps a month later, the same event leads to the same reasoning and the same decision again. Then, after enough such short outages of this kind, something interesting happens: the safety monitor becomes a component whose intermittent absence is tolerated, which is to say, an ornament.

Today we make safety claims about models, but the world doesn’t consume models, it consumes deployed products. The latter adds routers, classifiers, samplers, compilers, hardware platforms, and partner clouds. What is publicized in the form of model cards is only true when the serving stack is governed by policies that force the model to be conveyed faithfully. A deployed agent that can’t be observed, corrected, and revoked is an agent that lacks operational safety, regardless of whether the model-in-the-lab meets stated safety goals.

Where we are

The most concerning kinds of AI product incidents are less than an outage but more than a hiccup. Subtly altered product behavior is something that happens on a regular basis, but there are no norms for disclosing these events. To their credit however, Anthropic has actually delivered two incident postmortems on the topic. They describe software defects that altered Claude but allowed it to keep running. These incidents did not alter the model weights, yet users still wound up interacting with something different than intended.

One of these incidents is especially interesting to study. The lab had found a defect that occasionally caused the most probable next token to be ignored, and patched it. Later it rewrote that part of the code to address what it believed was the root cause, and removed the patch. But the patch had also been masking a second defect, in an approximate top-k optimization. The lab believed the patch had one purpose when it had two, and removing it caused a regression.

The first easy lesson here is that an institution’s beliefs about its own systems can turn out to be wrong, especially when there isn’t sufficient process in place to question them. This can happen anywhere, not just in an AI lab. There’s also a second, stranger lesson. Because the model was good at recovering from isolated mistakes, alarms that were based on running evaluations of the product remained green. The robustness of the model had the effect of obscuring faults in the serving layer. Better models make it harder to build good alarms for the serving layer.

The stack is supposed to faithfully convey a model to users, but that’s not the only objective in play. The production team is also under pressures which are measured in cost and latency. The approximate top-k optimization had been adopted under those pressures, based on a seemingly good theory that the errors it introduced would only affect low-probability tokens. That theory was correct, but the implementation in the deployed product was fraught.

The Anthropic incident wasn’t a safety failure, but it’s an existence proof: Model and product can diverge for weeks, even at a safety focused lab. A slightly different operational defect could have effects on safety guarantees. It could quell a refusal behavior, or cause the system to ignore the output of a classifier, or route a request to a degraded server that’s unable to trigger a manual review. An incident like this hasn’t appeared in the public record yet. Does that mean it hasn’t happened? There are currently no norms of disclosure, so right now we can’t say.

The place of reliability engineering

The AI safety field has a branch focusing on control research, which designs monitoring systems and protocols based on assumptions that once seemed radical: The model might have a low opinion of its creator, a cunning plan to cause a failure, or otherwise be working against its operator. The field also works on standards for the security of model weights, to prevent their exfiltration and misuse. And safety policies at frontier labs include deployment safeguards.

So I’m not claiming that no one is minding the shop. However all of the work I just mentioned depends on being able to effectively govern the serving infrastructure. If a safety monitor isn’t running, or a classifier’s output isn’t being honored, or a shutdown command fails to reach every replica of a model, then the system as a whole will fall out of alignment with its safety goals. These scenarios, as a category, are surprisingly common, and reliability engineering is the discipline for controlling their occurrence. It’s how you ensure your software is actually running, that the particular code and configuration match what you intend, and that those components will timely obey your commands at runtime. It’s the discipline that makes every important operational behavior measurable, not just deduced.

It’s actually auspicious how strong the symmetry is between the philosophy of the AI safety field and the doctrines of SRE. Both strive to define and engrain patterns for performing high-quality reasoning under conditions of uncertainty.

• Service level objective: Committing in advance to the amount of failure that will be tolerated, and scoring the system against that commitment on a regular cadence.

• Error budgets: You should define in advance the right action for when it becomes apparent that your expectations were too rosy.

• Blameless postmortems: Commitment to the view that failures are the product of the systems where they occur, and so the valuable thing to interrogate is the system.

• Chaos engineering: Forcing beliefs to be validated on a cadence. Either the resilience mechanism works when it’s exercised, or the belief in it was just a comfort.

• Runbooks: Quality decisions, written by your rested self, for your exhausted self.

• Canary deployment: Running an inexpensive experiment before a more expensive one.

Everything above could describe the approach of either discipline; we are not strangers to one another.

Two commitments to serve users

Every frontier lab and model-serving operation should make two basic commitments that will facilitate the effectiveness of AI safety at the infrastructure layer.

1. Commit to taking your product offline when safety isn’t operational. For every infrastructure component that’s required to realize a safety promise, make a clear internal policy about whether or not the product is allowed to be exposed to users without it. Then make that policy into crystal clear operational doctrine.

It’s important to acknowledge that such a policy and doctrine requires something else, more than commitment alone: headroom. That means spare capacity to provision the safety monitor on another cluster, and engineers with time to build the failover mechanism, for example. In a hypercompetitive environment, slack reads to many simply as waste; and even if a given lab took a different position, no single lab can competitively hold that line alone. The industry must hold it as a group, which leads me to the second commitment:

2. Publish three metrics, with an objective and a current measurement for each:

1. Fidelity. The share of production requests that are served in a way that matches the behavior of the evaluated and documented model. This should be measured by feeding identical inputs, both organic and synthetic, to reference implementations and production serving stacks and comparing the outputs.

2. Time to detect. How long a behavioral regression, relative to the reference implementation, may persist in production before it’s detected and triggers an investigation. This should be measured by regular drills, not organic incidents alone. (In the Anthropic postmortem, the first regression persisted for about three weeks before an investigation began.)

3. Time to revoke. How much latency is allowed between the decision to withdraw a model or service artifact (an API endpoint, a web service, an app, model weights, a sampler build, or a system prompt) and the action taking effect across every serving path.

An organization should make a commitment not only to targets, but to what happens when an observed measurement falls behind its target. At minimum, that should be a timely notification to users. When it’s novel, it should be a published postmortem that the industry can learn from.

The trick here is that the second commitment, the publication of the 3 metrics, makes affordable the first commitment: to suspend products at those times they can’t be operated safely. The commonly adopted language of “nines” has enabled buyers of services to price availability without understanding a vendor’s architecture. Published operational-safety numbers will let customers and regulators price the slack that’s needed for safety, turning a normally invisible backstop into a visible feature. The first lab to publish such metrics will define the norms against which their competitors will be initially compared.

Sooner or later

Every important technology eventually gets its own reliability engineering discipline. In many cases it happens only after something catastrophic occurs. The mandate for an aircraft pilot’s checklist only emerged following the crash of the Boeing Model 299, along with the once incredible idea that a given plane might actually be too complex to fly from memory. Decades later, Margaret Hamilton and her team advocated for better controls against maloperation, but the work wasn’t greenlighted until a preventable mistake occurred onboard Apollo 8. Medical software had to wait two decades more, until Therac-25 killed or gravely injured six people.

History repeats itself, and AI will get its reliability engineering discipline. The open question is whether that will come to pass before or after a serious incident. If we want to see AI adopted and embedded in real world processes that must be reliable, then reliability engineering at the frontier labs needs to get ahead of the curve.