Building HIPAA-Compliant AI Systems: Why Budget Governance Matters

The Problem With Traditional LLM Monitoring

Most LLM observability platforms work like this:

1. Your app makes a call to OpenAI, Claude, or Gemini
2. The observability SDK intercepts the request
3. Prompts, responses, and metadata are sent to an external server
4. You get dashboards, analytics, and cost tracking

This architecture works fine for most SaaS applications. But in healthcare, it's a compliance liability.

Why This Breaks HIPAA

HIPAA defines Protected Health Information (PHI) broadly:

Any health information that can identify a patient — including clinical notes, lab results, diagnosis codes, medication lists, or any context that could identify a person.

When a patient's medical record is mentioned in a prompt, that's PHI. When a doctor dictates clinical notes into your AI system, that's PHI. Sending it to an external server for monitoring — even with encryption in transit — creates a data handling chain that HIPAA regulators scrutinize closely.

The regulatory reality:

• You need a Business Associate Agreement (BAA) with any service handling PHI
• Not all vendors offer BAAs
• Those who do impose strict audit requirements
• One data breach can trigger HIPAA penalties of $100–$50,000 per violation

What HIPAA Actually Requires (And What It Doesn't)

One of the biggest misconceptions about healthcare AI is that HIPAA prohibits cloud services, large language models (LLMs), or third-party vendors.

HIPAA does not ban:

• AI systems
• Cloud infrastructure
• LLM providers such as OpenAI, Anthropic, or Google
• Third-party software vendors
• External observability platforms

Instead, HIPAA requires healthcare organizations to implement appropriate administrative, technical, and physical safeguards to protect Protected Health Information (PHI).

The compliance challenge is not whether AI is used. The challenge is understanding where PHI flows, who can access it, how it is protected, and which organizations become part of the PHI handling chain.

For example, if patient information is transmitted to a third-party service, organizations may need additional controls such as Business Associate Agreements (BAAs), vendor risk assessments, audit procedures, access controls, and breach notification processes.

As healthcare AI systems become more complex, every additional system that receives PHI increases the scope of compliance reviews and ongoing governance obligations. This is why many healthcare engineering teams focus on minimizing PHI exposure wherever possible. Reducing the number of systems that process patient data can simplify compliance efforts, reduce operational risk, and make security reviews easier to manage.

The Current Workarounds (And Why They Don't Scale)

Healthcare teams facing this use three imperfect solutions:

Option 1: Monitor Nothing

You deploy LLM applications blind. You don't know if costs are spiraling. You can't detect anomalies or ai model drift. You manage budgets manually by checking provider dashboards.

Option 2: Redact and Send

Your team manually removes PHI before sending monitoring data to external tools. This is error-prone, slow, and expensive at scale.

Option 3: Build Your Own

You hire engineers to build internal observability. This costs months of engineering time and doesn't solve budget governance — just visibility.

How Local-First Monitoring Works

DoCoreAI intercepts every LLM call locally inside your infrastructure. Your prompts and responses never leave your servers. Only non-sensitive metadata (token counts, cost calculations, latency, model name) is transmitted to DoCoreAI's secure servers — never the content of your LLM calls, never patient data.

Here's the architecture:

1. You deploy DoCoreAI as a .pth file in your Python environment
2. DoCoreAI wraps all LLM SDK calls (OpenAI, Anthropic, Cohere, Bedrock, etc.)
3. Before and after each call, DoCoreAI extracts metadata locally:
   • Token counts
   • Cost calculations
   • Latency
   • Model used
   • Request timestamp
4. NO prompts or responses are captured — only metadata

   Think of it like your phone bill. Your carrier knows you called a number, for how long, and at what time — but they can't hear the conversation. DoCoreAI works the same way: it records that an LLM call happened, how many tokens were used, and what it cost — but never reads what you sent or what the model replied. Your patient data stays in your conversation. DoCoreAI only sees the receipt.

5. Metadata is stored locally in SQLite on your server
6. Budget decisions fire autonomously in real-time:
   • Soft limits trigger warnings
   • Hard limits block requests
   • Pacing engines distribute budget across hours/days
   • Governance rules fire without human approval

Result: Complete observability, cost governance, reduced PHI exposure and a simplified HIPAA compliance posture.

What Gets Captured (And What Doesn't)

✓ DoCoreAI DOES capture:

• Input token count
• Output token count
• Model name
• Provider
• Request timestamp
• Response time
• Cost per request
• Cumulative cost per team/department
• Error codes

✗ DoCoreAI NEVER captures:

• Prompt text
• Response text
• Patient identifiers
• Any content from the message
• User identifiers
• IP addresses (configurable)

HIPAA Implication: Because DoCoreAI never captures prompt content, response text, or any patient-identifiable information, it operates outside the PHI data chain. Most healthcare legal teams find no BAA is required — but we recommend confirming this with your compliance team for your specific deployment. Your metadata (tokens, costs, timestamps) stays within your control. You own your data entirely.