Your patient intake agent works in dev and hallucinates allergies in prod. The 5-layer architecture that fixes it. Schema-constrained tool use, BAA boundaries, audit trails, eval harness, human-in-the-loop on EHR writes.
It is written for HealthTech CTOs, VP Engs, and Heads of AI/ML who have prototyped an agent in Cursor or Claude, watched it work in staging, and then watched it fabricate a structured field the moment a real patient note hit it. The goal here is not to scare you off agents. The goal is to show you the production architecture our team uses to keep agents inside the HIPAA fence and out of the hallucination zone.
AI agents in healthcare are autonomous, LLM-driven workflows that use tools (APIs, retrieval, structured-output validators) to execute multi-step tasks against clinical data: patient intake summarization, charting assistance, prior-authorization drafting, claims triage. The unit is not a single prompt; it is a loop of model calls, tool calls, and validation steps that produces a clinical-grade outcome. Most prototypes work on the happy path and break the moment a real patient note arrives, which is the production-vs-demo gap. The five layers below are the architecture that closes it.
Every HealthTech engineering leader we talk to has watched the same movie. An engineer wires up an LLM call to a patient intake form, summarization prompt, or charting assistant. The first 10 test cases look good. The eleventh test case is a patient with a long medication list and a confusing chief complaint, and the agent emits an allergy that the patient never reported. The team patches the prompt. The agent emits a different invented field a week later. Eventually somebody asks the obvious question: how is this thing ever going to ship under HIPAA?
The reliability wall is not a model problem. The model is doing what it was trained to do, which is generate plausible text. The wall is an architecture problem. A raw LLM call is the wrong primitive to drop into a clinical workflow, in the same way a raw database connection is the wrong primitive to drop into a payment workflow. You need the layers around it.
There is also a regulatory layer many teams underweight until late. The HIPAA Security Rule’s technical safeguards in 45 CFR §164.312 require access control, audit controls, integrity, person-or-entity authentication, and transmission security on any system that touches electronic protected health information. None of those requirements go away because there is an LLM in the call graph. They get harder, because LLMs introduce a new failure mode (fabrication) the Security Rule was not written to address.
The five layers below are how we keep that failure mode contained.
The first failure mode to kill in any production rollout of AI agents in healthcare is unconstrained generation on any step that produces structured data. If your agent is reading a patient note and emitting an “allergies” field, that emission cannot be a free-text completion. It needs to be a tool call against a schema you control.
Modern model APIs support this directly. Anthropic’s tool-use system, for example, lets you define a strict JSON schema for each tool’s input, and the model is constrained to produce a call that matches. As Anthropic’s tool-use documentation explains, you can add strict: true to a tool definition and the model uses grammar-constrained sampling to guarantee the tool input matches your schema. OpenAI’s function calling and structured outputs do the same thing under different names. The mechanics matter less than the discipline.
In practice, that means an extraction step looks something like this:
# Strict tool definition for an Anthropic Messages API call
# (claude-opus-4-7 family, 2026 strict-mode tool use).
ALLERGIES_TOOL = {
"name": "record_allergies",
"description": (
"Record allergies explicitly stated by the patient. "
"Do not infer from medications or diagnoses."
),
"strict": True,
"input_schema": {
"type": "object",
"properties": {
"allergies": {
"type": "array",
"items": {
"type": "object",
"properties": {
"substance": {"type": "string"},
"reaction": {
"type": "string",
"enum": [
"rash", "anaphylaxis", "gi_upset",
"unknown", "other",
],
},
"source_span": {
"type": "string",
"description": (
"Verbatim quote from the note where "
"this allergy was stated."
),
},
},
"required": ["substance", "reaction", "source_span"],
"additionalProperties": False,
},
},
"no_allergies_stated": {"type": "boolean"},
},
# Anthropic strict tool use accepts anyOf, not oneOf.
# This forces the model to either return allergies or
# explicitly state none were found.
"anyOf": [
{"required": ["allergies"]},
{"required": ["no_allergies_stated"]},
],
"additionalProperties": False,
},
}
Two things to note. First, the schema is doing real work. The anyOf forces the model to either return allergies or explicitly say none were stated. There is no third option where it just hallucinates. (Anthropic’s strict tool use does not currently accept oneOf; use anyOf for the same union-shape constraint, or you’ll get a 400.) Second, the source_span field is a citation requirement. If the model can’t quote the patient note, it can’t claim the allergy. We post-validate every source_span against the source document with a substring check before the call returns. If the quote is not literally present, the call fails and goes to retry or human-in-the-loop (HITL) review.
Schema-constrained tool use does not eliminate hallucination. It eliminates a category of hallucination, the kind where the agent invents a structured field out of thin air. That is the category that gets you in regulatory trouble.
A Business Associate Agreement is the legal seam between your covered-entity client and any vendor that touches PHI. HHS’s guidance on Business Associate Contracts makes the requirement explicit: a covered entity must have a written contract with any business associate that uses or discloses PHI on its behalf, and that contract has to spell out permitted uses, safeguards, and breach reporting.
For an agent stack, that means every external API your agent calls is either inside the BAA fence or it is not. There is no “we’ll figure that out later.” The architectural discipline is to mark each tool the agent has access to with its BAA status, and route requests accordingly.
Concretely:
Two implementation patterns worth naming. Every tool definition carries a phi_zone flag (inside or outside), and the agent runtime refuses to dispatch a call where any field traceable to a PHI input lands in an outside-zone tool. Prompts are templated so PHI never appears in any string constructed for an outside-zone call. If a developer adds a new outside-zone tool, the type system makes them prove no PHI-tainted variable can pass in.
The mistake to avoid: assuming the agent will “just know” not to send PHI to a public API. It will not. The architecture has to make it impossible.
45 CFR §164.312(b) requires audit controls on systems that contain or use ePHI: hardware, software, or procedural mechanisms that record and examine activity. Translated for agents, every model decision that influences a clinical workflow needs to be logged in a way that a security officer can later reconstruct.
What “every model decision” means in an agent context is more than just the prompt and the response. The minimum log envelope we ship looks like this:
{
"trace_id": "tr_2026-04-27_a91f...",
"step_id": 3,
"agent_version": "intake-v1.4.2",
"actor": "agent",
"model": "claude-opus-4-7",
"model_revision": "anthropic-2026-04-15",
"prompt_template_id": "intake_extract_v7",
"tool_call": "record_allergies",
"input_hash": "sha256:7c1...",
"output_hash": "sha256:9b4...",
"redacted_input_preview": "[PHI redacted, 2,134 chars]",
"redacted_output_preview": "{...allergies: 2 records...}",
"latency_ms": 1842,
"tokens": {"input": 2014, "output": 187},
"validation_results": [
{"check": "source_span_substring", "result": "pass"},
{"check": "schema_conformance", "result": "pass"}
],
"outcome": "tool_call_executed",
"downstream_action": "queued_for_clinician_review",
"phi_zone": "inside"
}
A few load-bearing details. The model revision is captured because “Claude Opus 4.7 in March” and “Claude Opus 4.7 in May” are different artifacts and a regression three months from now will need that to be reconstructable. The prompt template is versioned because changing the prompt is a model change, even if the weights are identical. The validation results are captured because the audit also has to show the guardrails ran. And the trace id stitches every step of the agent’s run together so a single patient interaction can be reviewed start to finish.
We store these in an append-only log, encrypted at rest, with retention that matches the client’s HIPAA retention policy (typically 6 years). The redacted previews give a security officer enough signal to investigate without the log itself becoming a PHI repository to manage. For the cross-vertical view of these layers (eval harness, observability, retries, audit trails, compliance integration), see the broader 5-layer production pattern.
A blog post about reliability for AI agents in healthcare that doesn’t mention evals is selling you something. Evals are not optional and they are not something you bolt on after launch. The NIST AI Risk Management Framework names “Measure” as one of its four core functions specifically because you cannot manage what you do not measure, and AI systems drift in ways traditional software does not.
What an eval harness for a HealthTech agent actually looks like:
The PHI-redacted test cases are non-trivial to build. The shortcut some teams take, running real PHI through an eval suite that lives outside the BAA boundary, is exactly the wrong shortcut. Build the de-identification pipeline once, build the eval set against it, and accept the upfront cost.
Frontier model improvements are real and welcome. OpenAI’s GPT-5.5 system card reports OpenAI’s strongest set of safeguards to date alongside meaningful gains on agentic-task and tool-use benchmarks, which tells you the providers are pushing the floor up. It tells you nothing about your workflow. Build the suite.
The last layer is the one that gets the most pushback and the most upside. Any action the agent takes that writes to a system of record (the EHR, the billing system, the e-prescribing platform) sits behind a clinician review step until you have measured the agent’s accuracy on that action class for long enough to defend the alternative in a deposition.
That is not over-engineering. That is the calibration loop. The agent proposes; a credentialed human disposes. Every disposition becomes a labeled training case for the next eval run. Over months, you build the dataset and the statistical confidence that lets you graduate certain action classes from “human approves each” to “human spot-checks a sample” to “human reviews exceptions only.” Skipping that loop is how an agent ends up writing a fabricated allergy directly into a chart and how a regulator ends up writing a press release about it.
Implementation specifics: the HITL UI matters more than people expect. The reviewing clinician needs to see (a) the source document, (b) the agent’s structured proposal, (c) the source-span citation for each field, and (d) a one-click affirm-or-edit interface. If you make the clinician retype things, they will batch-approve to clear the queue and you will lose the calibration signal. The point of the loop is the data, not the friction.
The order to ship a production-grade healthcare AI agent from a working dev prototype is six weeks of focused work: schema-constrained extraction first (Week 1), then BAA-boundary enforcement (Week 2), audit log shipping (Week 3), eval set v1 with a regression gate (Week 4), and human-in-the-loop UI on the highest-stakes write action (Week 5). A 6-12 week graduation curve follows, moving actions from “review each” to “review exceptions” as calibration data accumulates. Most teams get to this floor with 2 engineers running for 6-8 weeks. The week-by-week plan if a HealthTech client came to us tomorrow with a working dev prototype:
source_span requirement and the substring validator. Confirm the validator fails closed on a curated set of 20 known-fabrication test cases.phi_zone. Add the runtime check that refuses outside-zone calls with PHI-tainted inputs. Confirm with red-team prompts.That is 6 weeks of focused work to get to the floor of what we would call production-grade. Beyond that floor, the work compounds. The eval set grows, the audit log feeds incident response, and the calibration data feeds continual improvement. The point is that there is a floor, and most teams get to roughly the floor with 2 engineers running for 6 to 8 weeks.
Are HIPAA-grade AI agents in healthcare possible without a BAA from the model provider?
Not for any workflow that sends real PHI to the model. The BAA from the model provider is a hard prerequisite if PHI is in the prompt. The major cloud LLM providers (Anthropic, OpenAI, AWS Bedrock, Azure OpenAI, GCP Vertex) all offer BAAs on appropriate tiers. If your model provider does not offer one, you either de-identify everything before it reaches the model or you change providers.
How do you handle hallucinations in clinical workflows?
Three layers stacked: schema-constrained tool use on extraction (kills the invented-field category), source-span citation requirements with substring validation (kills the unsupported-claim category), and HITL review on any write to a system of record (catches what the first two layers miss). Frontier model improvements help but do not replace these layers.
What does an agent eval harness for healthcare actually contain?
A graded test set of 200–2,000 de-identified cases, per-step metrics, hallucination-specific metrics (invented-field rate, source-span fidelity, schema conformance), an adversarial slice of hard cases, and a CI regression gate. Scaled appropriately to the workflow’s risk profile.
How long does it take to ship a production-grade HealthTech agent?
For a focused workflow with one or two write actions, 6 to 8 weeks of 2-engineer time gets you to the floor of production-grade. That floor includes schema-constrained extraction, BAA fence, audit logs, eval suite, and HITL on writes. Multi-workflow agents and EHR integrations extend the timeline. The “demo to deployment” gap is real and it is mostly the layers above.
Do we need an internal compliance team to do this, or can a partner ship it?
You need someone on the client side who can sign on the BAA chain, define retention policy, and credential the clinicians who staff the HITL queue. The engineering work (schema design, boundary enforcement, audit logging, eval suite, HITL UI) is the work we do. We coordinate with your compliance and clinical leads; we do not replace them.
Deeper looks at the buy-or-build, integration, and outgrowth decisions in this space:
Book a 15-min discovery call and we will walk through the layer-by-layer state of your current agent: https://cal.com/mirgen-motomtech/quick-intro
