When Your Production AI Fails, the Clock Is Already Running

A hospital customer emails on a Monday morning. It’s not a routine support ticket. For the past week, your AI tool has been quietly flagging normal cases as abnormal. The review queue backed up, clinical staff got frustrated, and a technician finally picked up the phone.

By the time the issue escalates to your engineering team, the room gets quiet. The questions starting to fly are none of the ones your sales demo was built to answer:

• Which exact version of the model was running for that specific customer on Thursday at 2:00 PM?
• What did the pipeline actually "see" before it started calling normal cases abnormal?
• Has it been doing the same thing at other customer sites, just more quietly?
• And the question nobody on the team has had to answer against a clock before: Do we have to notify the FDA, and how many hours do we have left to do it?

We’ve written before that the model itself is not the deliverable. We’ve also argued that earning the right to keep improving a cleared model is a far harder engineering problem than getting cleared the first time.

This is the chapter that comes immediately after both. It’s the one that opens the moment a model in production makes a consequential, real-world mistake.

A surprising amount of your company’s systemic exposure lives right here. Yet, most teams walk into production equipped with nothing but an on-call rotation and a Slack channel. That’s a reasonable setup for a website outage and a badly inadequate one for a medical device that clinicians rely on to make critical decisions. 

The failure has no stack trace

Every software engineering org expects a familiar shape of failure. Something throws an exception, the logs point directly to a line of code, you write a patch, deploy, and that specific bug never surprises you the same way again.

An AI model gives you none of that comfort.

When a model returns a wrong answer, nothing crashed. No exception was thrown. The system did exactly what its mathematical parameters told it to do on an input that sat just outside its training distribution. It delivers a failure with the exact same composure and confidence it shows when it is 100% right.

There is no broken line of code to fix. Often, there is no way to even reproduce the failure unless your system captured the precise input, the exact model version, the preprocessing state, and the environment variables at the exact millisecond of inference.

Worse, model failures are rarely isolated incidents. A model that misinterprets one unusual case is highly likely to misinterpret an entire class of cases that resemble it, most of which nobody has looked at yet. This is why the classic engineering posture of "we'll dig into the logs when something breaks" collapses on first contact with production AI. By the time you realize there’s a pattern, the evidence has usually aged out of your logging window.

The FDA clock you're probably already on

If you hold FDA clearance for a device, you already live under a reporting regime that predates this wave of AI and applies to your model regardless of how anyone framed it during development. 

Under 21 CFR Part 803 (Medical Device Reporting), you must report a device-related death, serious injury, or a malfunction that could plausibly cause one, within 30 calendar days of becoming aware of it. If the event demands remedial action to prevent an unreasonable risk to public health, that window shrinks to 5 working days.

The trigger isn’t definitive proof of harm. The legal standard is merely information that "reasonably suggests" your device may have caused or contributed to it. You do not get to wait for a clean internal investigation. The agency grants no extensions; you file with what you have and supplement the report later.

One wrinkle constantly catches software teams off guard: the fact that a human doctor made the final clinical call does not take your software off the hook. If your model's output, or even a confusing way of presenting that output on the UI, shaped the doctor's decision-making, you are still legally obligated to report.

You can only investigate what you thought to record

The ability to respond to an AI failure is a core infrastructure requirement. It must be designed into the architecture of your system while you build it, not bolted on as an afterthought.

To survive production, your architecture needs five unglamorous pillars:

1. An Immutable Decision Trail: A durable, privacy-safe record of every prediction, including the raw input, exact pipeline version, preprocessing state, output, and confidence. Without this, you cannot conduct a compliant investigation.
2. Version Pinning and Replay: The ability to instantly spin up the exact model artifact and pipeline environment that served a past case, allowing you to watch it decide all over again.
3. Closed-Loop Feedback Telemetry: A reliable pipeline for real-world clinical outcomes—like user overrides and corrected ground truths—to route back and attach to the original prediction metadata.
4. Deterministic Fallbacks: When a model misbehaves, your first move shouldn't be a panicked, rushed retraining cycle. You need the architectural capability to "freeze" the model, preserve the forensic evidence, and immediately failover to a safe, deterministic behavior or manual human review.
5. A Continuous V&V Pipeline: Before you deploy a mitigation, you must be able to run automated regression tests against historic edge cases to prove the new version resolves the issue without degrading overall model performance.

When the root cause is that nothing broke

In traditional software, root-cause analysis assumes there is a bug waiting to be squashed. AI investigations usually land somewhere far less satisfying: the system worked exactly as designed, and harm resulted anyway.

The root cause is rarely a broken line of code but data drift that slipped past weak monitoring, an unrepresentative training dataset, an edge case outside your validated operating envelope, or a user interface design that invited too much blind trust from the clinician.

Each of these challenges requires an active engineering response, not a quick, one-line hotfix.

Where does your system stand?

You don't need a live incident to find out if you're ready for one. Take your highest-stakes model, pick a specific prediction it made last week, and see whether you can retrieve the exact input, model version, and output behind it. 

If your team can't produce that complete dataset, you do not have a forensic trail. You are operating on hope, and your response to a real failure will be entirely improvised.

Then ask who gets the call if that prediction harmed someone and whether they can actually freeze the model and notify the FDA; whether you know which reporting window applies; whether you could prove a fix safe without re-validating from scratch; and whether you could contain a misbehaving model without trampling the evidence you'd need to understand it. Specific answers mean you've built a real post-market capability. Vague ones mean you've just found the work standing between you and your next bad afternoon.

The pattern behind all of it is simple enough: the forensic trail comes first, before the model ships, because the same logging, versioning, and replay that let you investigate a failure are what let you validate the model in the first place. 

Whether you can contain, investigate, and report an incident in days is a property of the pipeline you designed months earlier, not something you assemble in the war room after the call comes in.

How we build for high-stakes environments

At SpiceFactory, this level of systemic integrity is built directly into how our product studio operates. We don't just build models; we build the production-grade, highly resilient software systems required to run them safely in regulated industries.

We work alongside engineering teams to implement the decision logging, version replay, and containment protocols required to turn AI from an unpredictable black box into a compliant, manageable product.

If your AI system operates in a regulated or high-stakes environment and you want to ensure your engineering infrastructure is ready for production, , get in touch with our engineering team, or book a strategy call. Let’s look at the gaps in your current architecture and build a system that is resilient by design.