Your prototype patient summarizer works because GPT-5.5 reliably emits valid JSON when the input is clean. Then a real chart note arrives with a numbered medication list inside a free-text “history of present illness” block, and the model emits a hallucinated allergy under a key that does not exist in your schema. The fix is not a better prompt. The fix is schema-constrained tool use, plus PHI handling at the tool boundary, plus an audit trail the Security Rule can read. This post walks through what HIPAA-grade LLM tool use actually looks like in production and why “the model can refuse” is not a control any auditor will accept.
In a free-form LLM call, you prompt the model and it returns text. You then parse that text and hope it matches the shape you wanted. In a schema-constrained tool call, you define a tool with a strict JSON Schema for its inputs, and the provider’s API guarantees that whatever the model emits will match that schema before you ever see it.
The mechanism varies by provider. Anthropic’s tool-use API constrains tool inputs to the declared input_schema JSON Schema, with the model’s outputs sampled against that schema before the response returns. OpenAI’s structured outputs do the same under the response_format and tools interfaces. Azure OpenAI inherits the OpenAI behavior. AWS Bedrock supports tool use across the Anthropic, Cohere, and Mistral models it hosts, with the strictness semantics inherited from each model.
The HIPAA-relevant point is not that schema constraint exists. The point is that it converts an unbounded text-generation call into a typed, validatable function call. When your agent extracts allergies from a chart note, it is now calling record_allergies(allergies: list[Allergy]) with each Allergy matching a defined object shape, not emitting JSON-flavored text that a regex might or might not parse. That shape is the contract you can audit, log, and refuse on.
The discipline that follows is more important than the API call itself. You still need to decide which fields are required vs optional, whether enums are closed or open, what your fallback is when the model legitimately cannot fill a field, and what happens when a tool call schema-validates but is semantically wrong (the model said the patient is allergic to penicillin, the note says no such thing). Schema constraint kills the parsing failure mode. It does not kill the fabrication failure mode. Both have to be designed for.
Free-form LLM output is fine for some things. It is not fine for any step in a HIPAA workflow that produces a structured field a downstream system will act on. There are three failure modes operators see repeatedly.
The first is silent schema drift. The model returns JSON that looks right, your parser accepts it, your downstream system ingests it, and three weeks later somebody notices that the severity field has been receiving values like “moderate-to-severe” instead of the documented enum of mild | moderate | severe | life-threatening. Nobody set out to break the contract. The model just chose words that read like clinical English in the absence of a constraint.
The second is fabrication under prompt pressure. When the input is ambiguous or the prompt asks for a complete extraction and the data is incomplete, free-form models often fill in plausible-but-invented detail rather than emit an empty field. A patient note that says “patient reports drug sensitivity, see prior chart” can produce an allergy list with three confidently-named drugs the patient never mentioned.
The third is failure to refuse. Operators sometimes argue that the model can be prompted to refuse when it is uncertain. This is true on average and unreliable on any individual call. Refusal is a probability, not a guarantee, and the Security Rule does not grade systems on probability. It requires audit controls, integrity, and access control. A workflow that depends on the model choosing to abstain is a workflow with a soft control where it needs a hard one.
Schema-constrained tool use does not solve fabrication, but it does provide the seams where you can layer the controls that do. You can require non-empty source spans for each extracted field. You can require a confidence value. You can require that any extracted allergy point to a character offset in the source note. None of that is possible inside a free-text completion.
Even with schema constraint and a BAA, most HealthTech engineering teams want to minimize the PHI that hits the model. The right place to do redaction is not the prompt template. It is the tool boundary.
The pattern looks like this. Your agent receives a request to summarize a patient note. Instead of passing the raw note into the model, you pass a redacted version where direct identifiers (patient name, MRN, date of birth, address, phone, exact dates outside the clinical window) are replaced with stable tokens like [PATIENT_NAME_1] or [DOB]. The model summarizes the redacted note. When the summary comes back, your tool layer un-redacts the tokens before the summary is persisted or sent.
The reason to do this at the tool layer and not at the prompt layer is reproducibility. Prompt-layer redaction tends to be ad-hoc, applied differently by different developers, and impossible to audit. Tool-layer redaction is a single function that every model call passes through. You can log every redaction operation. You can test the redactor as a unit. You can refuse to make a model call if the redactor errored out.
Two practical notes. First, the redactor is itself a control surface and needs evals. A redactor that misses a patient name in even a fraction of a percent of notes compounds to thousands of PHI exposures once note volume scales. Second, “redaction” here does not necessarily mean removing all PHI. The HIPAA Privacy Rule’s Safe Harbor de-identification standard lists 18 identifier categories, and removing all of them often destroys the clinical signal. Most HealthTech teams redact direct identifiers (categories that uniquely identify the patient), leave clinical content intact, and rely on the BAA plus the model provider’s data-handling guarantees for residual PHI that survives redaction.
A Business Associate Agreement (BAA) is the legal instrument that lets a covered entity share PHI with a vendor without violating HIPAA. The major LLM providers all offer BAAs, but the terms are not identical and the engineering implications differ.
OpenAI offers a BAA on its Enterprise and API plans. Per OpenAI’s Enterprise privacy documentation, data sent to the API is not used for training by default, and BAA-covered customers get additional handling guarantees. Note that ChatGPT.com (the consumer product) is not BAA-eligible, regardless of plan tier.
Anthropic offers a BAA on commercial API plans. Per Anthropic’s HIPAA and data handling documentation, the BAA applies to direct API use and certain partner deployments. Claude.ai (the consumer product) is not in scope.
Azure OpenAI Service is BAA-eligible as part of Azure’s broader HIPAA-compliant services. Per Microsoft’s Azure HIPAA documentation, the Azure BAA covers a list of in-scope services that includes Azure OpenAI, and the terms differ from the consumer Microsoft account BAA program.
AWS Bedrock is BAA-eligible under the broader AWS BAA. AWS’s HIPAA-eligible services list includes Bedrock and most of the foundation models it hosts, including Anthropic’s Claude models, with the BAA applying to the underlying AWS infrastructure and the model invocations.
The engineering implication is that you need to verify three things before any model call carries PHI. First, that the specific account and plan tier is BAA-covered (Enterprise vs Pro vs free is not the same). Second, that the specific model and region are in scope (some new models are not added to the BAA on day one). Third, that no logging or retention setting is sending data to a non-BAA endpoint (default retention windows vary, and some providers offer zero-retention modes for BAA customers).
A signed BAA is necessary and not sufficient. The Security Rule still requires audit controls and access controls on your end. The BAA just makes it legal for the provider to receive the data; it does not relieve you of building the controls.
The HIPAA Security Rule at 45 CFR §164.312(b) requires audit controls: “hardware, software, and/or procedural mechanisms that record and examine activity in information systems that contain or use electronic protected health information.” For LLM-based agents, that means logging the tool calls.
The minimum useful audit log for a tool call has six fields. The agent identifier (which workflow ran). The user or service identity on whose behalf the call ran. The timestamp. The tool name. The input arguments (post-redaction; the audit log itself is PHI-bearing if you log un-redacted inputs and needs to live in BAA-covered storage). The output, including success vs error, and any downstream effect (e.g., an EHR write ID).
The reason to log the tool call rather than the full model conversation is twofold. First, the tool call is the moment of effect: it is what touched a downstream system. The model’s internal reasoning between tool calls is incidental from an audit perspective. Second, full conversation logs are expensive once traffic ramps, often contain redundant context, and can balloon storage costs without adding investigative value. Tool-call logs are the right granularity for incident response.
Two practical patterns. First, treat the tool call log as append-only with cryptographic integrity (a Merkle log, a signed write-ahead log, or a managed service like AWS QLDB). If somebody can edit the audit log, it is not a HIPAA-grade audit log. Second, retain logs for at least six years to align with HIPAA’s general record-retention expectation, and put them under the same access controls as the underlying PHI.
A recurring objection from teams new to clinical workflows goes something like this: “We do not need a hard schema. We tell the model to refuse if the data is ambiguous, and it does.” This argument always loses at the architecture review. Here is why.
The HIPAA Security Rule grades controls on their technical implementation, not on their average behavior. An access control is either enforced by the system or it is not. An audit control either records the event or it does not. There is no rule that grants partial credit for “usually records.” A refusal that depends on the model’s choice is a control whose enforcement rate is a probability between zero and one, varying with prompt phrasing, input distribution, model version, and the latent factors no one can introspect. That is not what an auditor accepts.
The architectural alternative is to build the refusal into the structure of the system. Schema constraint forces the model to emit fields that match a defined shape, with required fields marked required. Tool definitions can refuse to execute when confidence values are below threshold. Downstream consumers can reject tool outputs that fail business-rule validation. Human review queues can hold any tool call above a risk threshold. Each of these is a hard control. Each is auditable. None depends on the model choosing well.
The model still benefits from instruction. Prompting the model to abstain when uncertain reduces the rate at which the hard controls have to fire. But the instruction is a performance optimization, not a HIPAA control. The hard controls are what the Security Rule scores you on.
Q: What does HIPAA require for AI agent audit trails?
HIPAA’s Security Rule at 45 CFR §164.312(b) requires hardware, software, or procedural mechanisms that record and examine activity in any system that contains or uses electronic PHI. For LLM agents, the practical minimum is an append-only log of every tool call, capturing agent identity, user identity, timestamp, tool name, post-redaction arguments, and the call result. Logs must be tamper-evident and live in BAA-covered storage, since they themselves contain PHI.
Q: Can ChatGPT be used in a HIPAA-covered workflow?
ChatGPT.com, the consumer product, is not HIPAA-eligible regardless of plan tier. OpenAI’s API and Enterprise products are BAA-eligible with a signed Business Associate Agreement and the right account configuration. Most HealthTech production deployments call the OpenAI API directly or through Azure OpenAI Service, not through ChatGPT.com, because the BAA terms and data-handling controls are tied to the API surface.
Q: How is schema-constrained tool use different from JSON mode?
JSON mode tells the model to return syntactically valid JSON but does not constrain the shape of the object. The model can emit any keys with any value types. Schema-constrained tool use enforces a specific JSON Schema, so required fields must be present, enums are bounded, and types are checked before the API returns. JSON mode catches malformed JSON; schema-constrained tool use catches malformed semantics.
Q: Does a BAA with an LLM provider mean I can send raw PHI in prompts?
The BAA makes it legal for the provider to receive and process PHI under HIPAA. It does not relieve the covered entity of the Security Rule’s other technical safeguards on its own systems. Most production HealthTech systems redact direct identifiers at the tool boundary regardless, both to minimize the PHI surface area and to maintain reproducibility in the redaction layer. The BAA covers the residual PHI that survives redaction.
Q: How do I redact PHI from an LLM prompt without losing clinical meaning?
Redact direct identifiers (patient name, MRN, exact date of birth, address, phone, account numbers) at the tool boundary using stable tokens like [PATIENT_NAME_1]. Leave the clinical content (symptoms, medications, diagnoses, lab values) intact, since redacting them destroys the signal the model needs. On the return path, un-redact tokens before persisting or displaying the output. Treat the redactor itself as a control surface that needs unit tests and eval coverage; a redactor that misses identifiers in 0.5 percent of notes is a breach risk at any meaningful scale.
If your team is wiring up an LLM-driven workflow that will touch PHI and you are not sure whether the architecture will hold up under a security review, that is a conversation worth having before the prototype ships, not after. The patterns above are what we build into every HealthTech Agentic AI Development engagement at Motomtech, and the same architecture also informs how we extend existing Custom Software Development codebases when an AI layer gets added to a HIPAA-bound system. If you want to walk through the specifics of your stack, you can grab time directly on my calendar at cal.com/mirgen-motomtech/quick-intro.
Mirgen Hoxha, CEO, Motomtech.
