Artificial intelligence is rapidly changing how healthcare organizations operate. From clinical documentation and patient communication to coding and administrative automation, AI has the potential to improve efficiency across the board. But with those benefits comes an important responsibility. If an AI tool processes Protected Health Information (PHI), it must be evaluated as part of your HIPAA compliance program.

A common mistake is treating AI like any other software purchase. AI systems introduce unique risks that traditional applications do not, including prompt injection attacks, inaccurate outputs, model misuse, and new data sharing concerns. That is why every healthcare organization should conduct a HIPAA risk assessment before deploying AI in workflows involving patient information.

In this guide, we'll walk through the key steps to perform a practical HIPAA risk assessment for AI.

Why AI Needs a Separate Risk Assessment

Under the HIPAA Security Rule, covered entities and business associates are required to conduct an accurate and thorough assessment of potential risks to electronic Protected Health Information (ePHI). While HIPAA does not have AI-specific requirements, organizations should evaluate the unique ways AI systems collect, process, store, and transmit sensitive data.

Unlike conventional software, AI tools may generate new content, interact with users through natural language, rely on third-party models, or use cloud infrastructure that requires additional vendor oversight. These characteristics can introduce risks that may not exist in traditional healthcare applications.

A dedicated AI risk assessment helps identify those risks before they become compliance issues.

Step 1: Create an Inventory of AI Tools

You cannot protect what you do not know exists.

Start by identifying every AI application used within your organization. This includes both officially approved tools and unauthorized applications employees may have adopted independently.

Your inventory should include:

• AI documentation assistants
• Medical transcription platforms
• Clinical decision support tools
• Coding and billing assistants
• Customer service chatbots
• Productivity tools with built-in AI
• Image analysis systems
• Employee-facing AI assistants

Many organizations are surprised to discover that multiple departments are already using AI without formal approval. This practice, often referred to as shadow AI, can create significant compliance risks if patient information is involved.

Step 2: Determine Whether the AI Processes PHI

Not every AI application falls under HIPAA.

Review each tool and determine whether it creates, receives, maintains, or transmits Protected Health Information. PHI includes individually identifiable health information such as patient names, medical record numbers, diagnoses, treatment details, insurance information, and other identifiers.

Ask questions such as:

• Can users enter patient information?
• Does the AI access electronic health records?
• Are AI-generated outputs linked to identifiable patients?
• Is any patient information stored by the vendor?

If the answer is yes, the AI system should be evaluated as part of your HIPAA risk analysis.

Step 3: Map the Data Flow

Understanding how information moves through an AI system is critical.

Document:

• Where patient information originates
• How data enters the AI system
• Whether information is encrypted during transmission
• Where data is processed
• Whether data is stored after processing
• Who has access to the information
• Whether subcontractors are involved
• How data is deleted when no longer needed

A visual data flow diagram can help identify unnecessary exposure points and simplify future audits.

Step 4: Evaluate the AI Vendor

Vendor due diligence is one of the most important parts of an AI risk assessment.

Review questions such as:

• Does the vendor offer a Business Associate Agreement?
• Does the agreement cover the specific AI product you intend to use?
• Is customer data used to train AI models?
• What are the vendor's data retention policies?
• Where is data stored?
• What security certifications does the vendor maintain?
• Does the vendor rely on subcontractors?
• How are security incidents reported?

Do not assume every version of an AI platform follows the same privacy practices. Consumer products and enterprise offerings often have very different contractual protections.

Step 5: Assess Technical Safeguards

HIPAA requires organizations to implement appropriate safeguards to protect electronic Protected Health Information.

For AI systems, evaluate whether appropriate controls are in place, including:

• Encryption for data at rest and in transit
• Multi-factor authentication
• Role-based access controls
• Unique user accounts
• Audit logging
• Automatic session timeouts
• Secure backups
• Data loss prevention controls

Access should follow the principle of minimum necessary, meaning users only have access to the information required for their role.

Step 6: Identify AI-Specific Risks

Traditional HIPAA assessments often focus on unauthorized access and data breaches. AI introduces additional concerns that deserve separate evaluation.

These may include:

Prompt Injection

Attackers attempt to manipulate AI prompts to expose confidential information or bypass security controls.

Hallucinations

AI systems can generate inaccurate or fabricated information that appears credible. Clinical or operational decisions should never rely solely on AI-generated content.

Model Inversion

Researchers have demonstrated that some AI models may reveal information about training data under specific circumstances.

Excessive Data Collection

Employees may unknowingly enter more patient information than necessary when interacting with AI tools.

Unauthorized AI Usage

Staff members may use personal AI accounts instead of approved enterprise solutions.

Each identified risk should be documented along with its likelihood, potential impact, and proposed mitigation strategy.

Step 7: Review Organizational Policies

Technology alone does not create compliance.

Your organization should have documented policies covering:

• Approved AI platforms
• Acceptable use
• Employee responsibilities
• Handling of Protected Health Information
• Incident reporting
• Vendor approval procedures
• Ongoing monitoring

Employees should understand exactly when AI may be used and what information can never be entered into unauthorized systems.

Step 8: Train Employees

Even the strongest technical safeguards can fail if employees do not understand them.

Training should explain:

• What qualifies as Protected Health Information
• Which AI tools are approved
• When a Business Associate Agreement is required
• How to recognize risky AI practices
• How to report suspected privacy incidents

Regular refresher training helps reinforce expectations as AI technology evolves.

Step 9: Document Your Findings

A HIPAA risk assessment should produce more than a checklist.

Document:

• AI systems evaluated
• Data flow diagrams
• Identified risks
• Existing safeguards
• Risk ratings
• Recommended corrective actions
• Assigned owners
• Target completion dates

Clear documentation demonstrates that your organization has taken a structured approach to identifying and managing risks.

Step 10: Make AI Risk Assessments an Ongoing Process

AI technology changes quickly.

New features, integrations, vendors, and regulations can all affect your risk profile. Risk assessments should be reviewed whenever significant changes occur and incorporated into your organization's regular HIPAA risk management process.

A one-time review is not enough.

Common Mistakes to Avoid

Healthcare organizations often make the same mistakes when introducing AI.

These include:

• Assuming enterprise AI automatically equals HIPAA compliance
• Using AI before completing a risk assessment
• Failing to sign a Business Associate Agreement when required
• Ignoring unauthorized employee use of AI
• Overlooking subcontractors that process patient information
• Skipping audit logging and monitoring
• Relying on AI-generated content without human review

Avoiding these mistakes can significantly reduce compliance and security risks.

Final Thoughts

AI has enormous potential to improve healthcare operations, but successful adoption requires thoughtful governance.

A HIPAA risk assessment helps organizations understand how AI affects patient privacy, information security, and regulatory compliance. By identifying risks early, evaluating vendors carefully, documenting safeguards, and training employees, healthcare providers can adopt AI with greater confidence while protecting sensitive patient information.

Remember that HIPAA compliance is not determined by the AI tool alone. It depends on how the technology is implemented, managed, and monitored within your organization. Because every healthcare environment is different, AI risk assessments should be reviewed by your privacy, compliance, legal, and security teams before new AI systems are deployed.