A better model does not close the system boundary
An AI model can improve substantially and still be an imperfect component. The engineering question is what the surrounding system does with its mistakes.
Better prompts, grounding, evaluations, and models can reduce error. They do not remove the need to decide which outputs may become actions, which claims need evidence, or what happens when validation cannot complete. Those obligations survive a model upgrade.
Model failure should be expected. System failure should not be accepted as the inevitable consequence of putting a model in the architecture. The useful goal is containment, detection, and recovery—not a promise that every error can be prevented.
The dangerous transition is from output to authority
A generated answer starts as a proposal. It becomes consequential when another component treats it as a fact, updates a record, invokes a tool, or presents it as authoritative guidance. That transition deserves an explicit design.
Consider a hypothetical system drafting instructions for a regulated workflow. Fluent text can contain an unsupported requirement. If the application forwards it solely because generation completed, a component error has crossed into system behavior without an evidence check.
The release boundary should ask what the output will be used for and what must be true before that use is allowed. A draft for an expert to inspect and an instruction executed automatically require different controls.
Layer the controls around different failure modes
A single guardrail is attractive because it offers one place to put the problem. It also becomes one place to misunderstand the problem. A filter may catch prohibited language while missing an unsupported claim. Grounding may supply an authoritative source that the model then misinterprets.
Controls should have distinct responsibilities. Classify the requested use and risk. Establish an evidence basis. Check the relevant constraints. Restrict action authority. Route unresolved cases to someone who can decide. Preserve enough of the path to investigate the result.
Several checks built on the same mistaken assumption are not independent protection. Ask which failure each layer can detect, what it cannot see, and whether a shared dependency could defeat several layers together.
LSAS makes the evidence boundary explicit
I developed LSAS—the Layered Safety and Accuracy System—as a specification for governing generative AI outputs in regulated healthcare and medtech contexts. Its central discipline is to treat an output as untrusted until the relevant evidence and validation support its release.
The published specification describes a safety-orchestration sequence: intent and risk classification, grounding to authoritative sources, real-time validation, escalation and human review, then continuous learning. This is a specification and a set of implementation contracts; its publication does not by itself prove a deployment safe.
In a wider agent architecture, bounded tool authority and auditability support that sequence. The model's confidence cannot grant a permission, waive a failed gate, or stand in for an absent reviewer. The system owns those decisions.
Escalation is an operating path
A review queue is not a safety mechanism merely because a human appears at its end. Someone must have the expertise, evidence, time, and authority to resolve the case. The system also needs an answer for when that person is unavailable.
For a consequential action, abstaining or deferring may be the correct result. For a low-risk task, a constrained fallback may be enough. A timeout should lead to a defined state, rather than quietly converting a failed check into permission to continue.
This is where reliability becomes an operational commitment. A policy that requires review has to be matched by staffing, routing, and response expectations. Otherwise the architecture has specified a dependency that the organization cannot supply.
Audit the decision, then improve the system
A useful trace explains why an action was allowed: the relevant input, evidence, policy and model versions, validation result, and escalation decision. Collect what is necessary for that explanation while respecting the sensitivity of the data. Storing everything creates its own exposure.
When a failure occurs, turn it into a regression case and decide which boundary should have caught it. The repair might belong in a policy, a tool contract, an evidence source, the user experience, or the operating procedure. Another prompt is only one possible response.
A change to one layer can alter the behavior of another. Version the policies and test representative paths again. Continuous learning here means reviewed changes to the system, not silently granting the model more authority after an incident.
A reliability claim needs a scope
No architecture makes AI infallible, and a collection of controls does not certify a system for every use. Reliability claims need a task, operating conditions, acceptance criteria, known limits, and evidence.
The same principle applies to my own frameworks. LSAS provides a way to make these responsibilities explicit. Teams still have to implement, test, and operate them for their setting.
The question to carry into a design review is this: where can a wrong model output become a consequential system action without anyone noticing? Start there.
More questions. Better conversations.
Bring this perspective to your event