Generative AI is quickly becoming part of everyday healthcare operations. Organizations are using AI to automate documentation, summarize medical records, assist with coding, support patient communication, and improve administrative workflows.

Choosing the right AI platform, however, involves much more than comparing features and pricing. If an AI vendor will create, receive, maintain, or transmit Protected Health Information (PHI), healthcare organizations have a responsibility to evaluate the vendor's security, privacy, and compliance practices before signing a contract.

A thorough vendor due diligence process helps reduce compliance risks, strengthen security, and build confidence that the AI solution can support healthcare requirements.

This checklist outlines the practical questions every healthcare organization should ask when evaluating a generative AI vendor.

Why Vendor Due Diligence Matters

Not every AI platform is designed for healthcare.

Many consumer AI services have different privacy practices, data retention policies, and contractual terms than enterprise offerings. Assuming every version of an AI product provides the same protections can expose organizations to unnecessary risk.

Vendor due diligence helps answer critical questions such as:

• Can this AI system process PHI securely?
• Does the vendor understand HIPAA requirements?
• How is customer data protected?
• Who has access to the information?
• What happens if a security incident occurs?

The goal is to verify vendor claims with documentation rather than relying solely on marketing materials.

1. Business Associate Agreement (BAA)

A Business Associate Agreement is one of the first items to discuss if the AI system will handle PHI.

Ask the vendor:

• Will you sign a Business Associate Agreement?
• Which products and services are covered by the agreement?
• Are all AI features included?
• Does the BAA cover future product updates?
• Are subcontractors included within the agreement?
• What are each party's responsibilities during a security incident?

Request evidence such as:

• A sample Business Associate Agreement
• Standard contractual language
• Documentation describing covered services

Do not assume a vendor offers a BAA for every product or subscription tier.

2. Data Collection and Usage

Understanding how the vendor handles customer data is essential.

Questions to ask include:

• What data is collected?
• Is customer content used to train AI models?
• Can training be disabled?
• Is customer data shared with third parties?
• How is metadata handled?
• Are prompts and responses stored?

Request evidence such as:

• Data processing documentation
• Privacy policies
• Product documentation
• Contractual commitments regarding model training

Healthcare organizations should seek clear answers rather than relying on general statements about privacy.

3. Data Retention and Deletion

Knowing how long information remains in vendor systems is critical.

Ask:

• How long is customer data retained?
• Can retention settings be customized?
• Is zero-retention available?
• How are deleted records removed?
• Are backups also deleted?
• How quickly is data permanently erased after deletion?

Request evidence including:

• Data retention schedules
• Data deletion procedures
• Documentation describing retention controls

Organizations should ensure retention practices align with their own policies and regulatory obligations.

4. Security Controls

Security should be evaluated using objective evidence.

Questions include:

• Is data encrypted in transit?
• Is data encrypted at rest?
• Is multi-factor authentication supported?
• Are role-based access controls available?
• Are customer environments logically separated?
• How are encryption keys managed?

Ask for supporting evidence such as:

• Security white papers
• Independent security assessments
• Compliance certifications
• Architecture documentation

Strong technical safeguards help reduce the likelihood of unauthorized access.

5. Audit Logging and Monitoring

Audit logs are important for investigating incidents and supporting compliance efforts.

Ask whether the platform records:

• User logins
• Prompt submissions
• AI-generated responses
• Administrative actions
• Permission changes
• API activity
• Failed authentication attempts

Also ask:

• How long are logs retained?
• Can customers export logs?
• Are logs searchable?
• Can logs integrate with SIEM platforms?

Request sample audit logs or documentation showing available logging capabilities.

6. Subprocessors and Third-Party Services

Many AI vendors rely on additional service providers.

Healthcare organizations should understand exactly who participates in processing customer data.

Questions include:

• Who are your subprocessors?
• What services do they provide?
• Where are they located?
• Do they process PHI?
• How are they evaluated?
• How are customers notified when subprocessors change?

Request:

• Current subprocessor lists
• Vendor management policies
• Security review procedures

Visibility into the vendor's supply chain is an important part of due diligence.

7. AI Security Testing

Security testing should extend beyond traditional software assessments.

Ask the vendor:

• Do you perform AI red teaming?
• How frequently is testing conducted?
• Are prompt injection attacks evaluated?
• Is jailbreak resistance tested?
• Do you assess data leakage risks?
• How are vulnerabilities prioritized?

Request evidence such as:

• Red team summaries
• Security testing reports
• Penetration testing summaries
• Vulnerability management documentation

Vendors may not share full reports, but they should be prepared to explain their testing approach and remediation process.

8. Risk Management Practices

Ask how AI risks are identified and managed throughout the product lifecycle.

Questions may include:

• Do you maintain a formal AI risk management program?
• How are new risks identified?
• How are security updates prioritized?
• How are AI model changes reviewed?
• How are customers informed of significant updates?

Evidence may include:

• Risk management policies
• Governance documentation
• AI security procedures
• Internal review processes

A mature governance program demonstrates that AI risks are managed continuously rather than only during product development.

9. Privacy and Compliance

Healthcare organizations should understand how privacy is integrated into the vendor's operations.

Questions include:

• What privacy frameworks do you follow?
• How do you support HIPAA requirements?
• How are privacy incidents investigated?
• Do you conduct privacy impact assessments?
• How do you handle customer requests related to stored data?

Supporting evidence may include:

• Privacy documentation
• Compliance certifications
• Independent audit reports
• Internal privacy policies

Remember that no vendor can make your organization HIPAA compliant through technology alone. Compliance depends on both the vendor's capabilities and your organization's implementation.

10. Incident Response

Every vendor should have a documented process for responding to security events.

Ask:

• How are incidents detected?
• What is the customer notification process?
• What information is included in incident reports?
• What are the expected response timelines?
• How are investigations conducted?
• Are customers involved in remediation activities?

Request:

• Incident response policies
• Breach notification procedures
• Business continuity documentation

Clear communication during an incident can significantly reduce operational disruption.

11. AI Governance

Responsible AI extends beyond cybersecurity.

Ask vendors how they manage:

• Human oversight
• Model updates
• Accuracy monitoring
• Bias evaluations
• Performance testing
• Responsible AI policies

Organizations should understand how governance supports the ongoing reliability of the AI system.

12. Documentation to Request

A strong due diligence package often includes:

• Business Associate Agreement
• Security white paper
• Privacy documentation
• Data processing agreement
• Data retention policy
• Subprocessor list
• Architecture overview
• Audit logging documentation
• Incident response policy
• Independent security assessment summaries
• AI governance documentation
• Red team or AI security testing summaries

Not every vendor will provide every document, but reputable vendors should be transparent about their security and privacy practices.

Common Mistakes to Avoid

Healthcare organizations often make avoidable mistakes during vendor evaluations.

These include:

• Assuming every subscription tier includes HIPAA support
• Accepting marketing claims without supporting evidence
• Ignoring subcontractor risks
• Overlooking data retention settings
• Failing to review audit logging capabilities
• Skipping AI-specific security testing questions
• Treating vendor compliance as a substitute for internal governance

A structured review process helps reduce these risks.

Final Thoughts

Selecting a generative AI vendor is not simply a technology decision. It is also a privacy, security, and risk management decision.

A thorough due diligence process helps healthcare organizations verify that vendors have appropriate contractual protections, technical safeguards, governance practices, and security controls before patient information is involved. Requesting evidence such as Business Associate Agreements, data retention policies, audit logging documentation, subprocessor lists, and AI security testing summaries provides a stronger basis for evaluating vendor claims.

Vendor due diligence should also be viewed as an ongoing process. As AI platforms evolve, organizations should periodically review contracts, security documentation, governance practices, and product updates to ensure the solution continues to meet operational and compliance expectations. Building this discipline into your vendor management program can reduce risk while supporting the responsible adoption of AI across healthcare.