Enterprise AI Governance: A Framework for PHI/PII Protection

Traditional data privacy risks—unauthorized access, insecure storage, data breach—are well-understood and well-mitigated in modern enterprise environments. AI systems introduce a new category of privacy risk: emergent disclosure. An AI system may reveal sensitive information not because it was directly queried for that information, but because it synthesizes non-sensitive pieces of information in ways that reveal sensitive facts. A system that knows an employee's age, their recent medical leave dates, and their role in a project team may, in response to an innocuous question, generate output that reveals the employee's health condition without any single data field being sensitive in isolation.

This emergent disclosure risk requires privacy governance approaches that analyze data in combination, not just in isolation. Traditional data classification (PII, non-PII) is insufficient; AI governance requires contextual privacy analysis that considers what information can be inferred from a combination of fields and how that inference risk changes as the AI's knowledge base expands.

Effective AI privacy governance addresses risk at each stage of the AI pipeline. At the Ingestion Stage, data is classified upon entry: PII/PHI fields are identified, sensitive documents are tagged with access control metadata, and data lineage is established. At the Storage Stage, sensitive data at rest is encrypted with field-level encryption for PII/PHI fields, and vector embeddings are stored in a tenant-isolated namespace to prevent cross-tenant information leakage.

At the Retrieval Stage, row-level security ensures that only data the requesting user is authorized to access is included in retrieval results. At the Context Assembly Stage, the assembled prompt is scanned for PII/PHI before being sent to the LLM, with sensitive data replaced by pseudonymous tokens. At the Reasoning Stage, the LLM is instructed via system prompt to avoid disclosing PII/PHI in its outputs. At the Output Generation Stage, generated outputs are scanned for sensitive data using both pattern matching and NER before delivery. At the Action Stage, actions that would expose sensitive data to external systems are intercepted and reviewed before execution.

HIPAA's Privacy Rule requires that AI systems processing PHI implement technical safeguards equivalent to those required for any PHI-handling system: access controls, audit controls, integrity controls, and transmission security. The key HIPAA-specific requirement for AI systems is the Minimum Necessary standard: the system should use only the minimum PHI required to accomplish the task. In RAG systems, this requires retrieval configurations that limit context to data relevant to the current query, preventing the LLM from accessing a broader patient record than the query requires.

GDPR Article 22 imposes additional constraints on automated decision-making: individuals have the right to object to decisions made solely by automated systems, the right to explanation of automated decisions, and the right to human review of significant automated decisions. AI systems that make decisions affecting individuals must implement these rights technically—providing explainable outputs, logging decision reasoning, and routing flagged cases to human reviewers. CCPA imposes similar consent and transparency requirements for AI systems processing California residents' personal information.

Implementing this framework begins with a data flow mapping exercise: tracing the complete journey of each sensitive data type through the AI pipeline, from source system to final destination, identifying every point where sensitive data could be exposed to unauthorized systems or personnel. This mapping typically requires 2-4 weeks for a mature enterprise AI system and produces the privacy impact assessment documentation required by GDPR Article 35.

With the data flow map in place, the technical controls can be implemented in priority order: access controls and PII/PHI classification first (these prevent the most severe exposure risks), followed by audit trail generation (required for regulatory investigation), followed by output scanning and context window management (these address the AI-specific emergent disclosure risks). Organizations that implement controls in this order consistently achieve faster time to compliance than those that implement them in architectural rather than risk-prioritized order.

Regulatory audits of AI systems increasingly request evidence of privacy controls that is specific to the AI context: proof that access controls are enforced at the retrieval level (not just the storage level), evidence that context windows do not persist sensitive data across sessions, and documentation of the AI-specific incident response procedures. Organizations that prepare this evidence proactively—maintaining a continuously updated compliance evidence library—consistently experience shorter and less disruptive audit processes.

AI-specific incident response procedures should cover four scenarios: a model that begins generating sensitive information it should not have access to (requiring model output recall and re-evaluation), a vector store that leaks cross-tenant data (requiring tenant isolation audit and remediation), a governance gate failure that allows PII to reach an external system (requiring exposure impact assessment and regulator notification if required), and a training data contamination incident (requiring training data audit and model retraining).