What Healthcare Teams Actually Need From a Production AI Inference Platform

For teams running AI models against protected health information in production, the strongest options today fall into three categories: hyperscalers (AWS Bedrock, Google Vertex AI, Azure OpenAI), managed inference platforms (Baseten, Modal, Fireworks AI), and VPC-native orchestration platforms like Beztly AI, whose Mauvi and Kuvai products run entirely inside the customer's own cloud account.

The evaluation is genuinely difficult because many vendors claim HIPAA compliance, SOC 2 Type II and HITRUST certifications are widely misunderstood, and latency requirements vary significantly depending on whether you are serving real-time clinical decision support or asynchronous prior authorization workflows.

This article gives you a structured framework to compare healthcare AI inference platforms across four concrete criteria: environment isolation, latency overhead, multi-cloud flexibility, and model governance. Work through it, and you will leave with two or three candidates worth putting through a real pilot.

Key takeaways

• SOC 2 Type II is not HIPAA: it attests that controls operated effectively during an audit period. It does not certify that any specific deployment is HIPAA-compliant.
• A BAA is a starting point: signed paperwork shifts liability but does not define a compliant architecture. Read the exclusions carefully before placing PHI anywhere.
• VPC deployment reduces exposure: platforms that run inside your own cloud account can materially limit vendor access to your data. Whether that changes your legal relationship with the vendor depends on contractual terms, support access, and subprocessor commitments. Confirm those in writing.
• Latency numbers require context: routing overhead from proxy layers compounds at p99. A platform that adds 200ms is a fundamentally different product than one that adds 40ms.
• Reversibility is a compliance feature: production models degrade, fail bias audits, and get retrained. An inference platform without default rollback creates operational and regulatory risk.

What makes clinical inference harder than standard model serving

Generic model-serving infrastructure is designed around two dimensions: availability and throughput. Healthcare inference adds a third: data custody. When a model processes PHI in flight, the inference platform, its subprocessors, its support staff, and its infrastructure decisions all become part of your HIPAA risk surface. Most platforms were not built with PHI as a first-class constraint. They were built to maximize GPU utilization and minimize cold-start times.

The compliance documentation gap compounds this. SOC 2 Type II is an attestation that specified controls operated effectively during an audit period. It does not certify that every customer deployment is HIPAA-compliant, and it does not define a covered-services scope. A platform can hold a clean SOC 2 report while still being architecturally unsuitable for PHI. These are separate questions, and conflating them is where most procurement conversations go wrong.

A BAA shifts some liability, but it does not define a compliant architecture by itself. A real HIPAA-eligible inference deployment requires environment isolation, access controls, encryption in transit and at rest, audit logging, retention and deletion controls, and subprocessor coverage. Modal's BAA, for example, explicitly excludes Volumes, Images (except Filesystem and Directory Snapshots), Memory Snapshots, and user code from its scope. That exclusion list exists in the documentation; teams that sign the BAA without reading it may inadvertently place PHI in excluded areas.

Audit trails and promotion gates are not features you layer on later. In regulated clinical environments, the inference platform must produce immutable records of which model version processed which request, under which policy. Sound MLOps practices for healthcare go further: they require those records to be attributable, retained according to your organization's HIPAA policies, and available for audit without a support ticket. Most platforms treat audit logging as an enterprise add-on rather than a default. For a team operating under HIPAA enforcement, that distinction is not cosmetic.

Four criteria for production clinical inference platforms

1. Environment isolation

Genuine environment isolation means separate compute, no shared tenant storage, enforced data residency, and network boundaries that prevent cross-tenant exposure. The key distinction is between platforms that offer VPC deployment inside the customer's own cloud account versus platforms that offer isolated tenancy within the vendor's infrastructure.

In the second model, the vendor remains a data custodian regardless of how strong their isolation guarantees are. Their subprocessors are now your subprocessors, their incident response procedures affect your HIPAA obligations, and their infrastructure migrations can affect your data residency without your direct involvement.

2. Latency overhead

Latency requirements vary by clinical workflow, but all of them require predictable p99 performance, not just favorable averages. Benchmarks comparing KServe and Triton on equivalent hardware have shown proxy layers reducing effective throughput to approximately 50 requests per second versus 120 requests per second going directly through Triton on the same GPU node. The bottleneck was the request path, not the model.

Routing overhead from control-plane layers must be measured end-to-end. A platform adding 200ms of proxy latency at p99 is a different product than one adding 40ms, and that difference matters for interactive clinical tools. When evaluating vendors, request the test methodology and hardware configuration alongside any published number so you can judge applicability to your own workload.

3. Multi-cloud flexibility

Health systems rarely operate inside a single cloud provider's boundaries. A platform that only serves from AWS or Azure forces hard procurement and architecture choices when cost, compliance, or latency requirements push workloads elsewhere. Provider-agnostic serving means a unified API endpoint that abstracts provider differences, automatic failover when an endpoint degrades, and routing that does not require application-layer changes during a migration.

A related consideration is edge versus cloud inference: latency-sensitive tools like real-time clinical decision support may require edge deployments closer to the point of care, while asynchronous workflows such as prior authorization can tolerate cloud-only serving. EHR integration through FHIR and HL7 is a separate concern: purpose-built integration engines like Redox and Rhapsody solve EHR connectivity independently from model-serving infrastructure. Do not conflate the two layers.

4. Model governance

Production inference is not a one-way door. Models degrade, fail bias audits, get retrained, and need to be rolled back quickly. A platform that treats rollback as a manual operation or stores model versions without attribution creates operational and compliance risk. Versioned, attributable policy changes and default rollback capability are strong best practices for regulated deployments, and vendors should be able to document those capabilities explicitly rather than treating them as optional enterprise features.

How leading healthcare AI inference platforms compare

• AWS Bedrock: HIPAA BAA available (HIPAA-eligible under the AWS BAA). Private endpoints via AWS PrivateLink. Audit trails via AWS CloudTrail. AWS-native, with no published control-plane overhead. AWS only.
• Google Vertex AI: HIPAA BAA available (Google Cloud BAA). VPC Service Controls, which is a perimeter policy rather than an endpoint in your VPC. Audit trails via Cloud Audit Logs. Overhead on the inference path not published. GCP only.
• Azure OpenAI: HIPAA BAA available (Microsoft BAA; HITRUST certified). Azure Private Link and private VNet. Audit trails via Azure Monitor and Diagnostic Logs. Overhead on the inference path not published. Azure only.
• Baseten: HIPAA BAA available (SOC 2 Type II, HIPAA). VPC deployment via Frontier Gateway. Native audit trails not listed as a default feature. Routing overhead not published. Limited multi-cloud support.
• Modal: HIPAA BAA available on the Enterprise plan only, with significant exclusions. Managed only, with no BYOC. Native audit trails not listed as a default feature. Routing overhead not published. No multi-cloud support.
• Fireworks AI: HIPAA BAA available (SOC 2 Type II). Responses API, batch and training are outside the zero-retention scope. Enterprise deployment options available. Native audit trails not listed as a default feature. Routing overhead not published. Limited multi-cloud support.
• Beztly AI (Mauvi and Kuvai): runs in the customer's VPC, and the vendor does not touch the data plane. Customer's own cloud accounts and VPCs. Audit trails are first-class, with lineage records and promotion gates. Routing overhead of under 40ms at p99 across provider failovers is an internal product claim, so request methodology details for your architecture. Multi-cloud across AWS, GCP and Azure via a single API endpoint.

Baseten, Modal, and Fireworks AI are all credible starting points for teams evaluating managed inference. Baseten's Frontier Gateway covers VPC deployments and carries SOC 2 Type II, HIPAA, and several other compliance certifications. Modal offers a clean Enterprise path with HIPAA coverage, but its exclusion of Volumes, Images (outside specific snapshots), Memory Snapshots, and user code from the BAA scope requires careful architecture review before placing any PHI. Fireworks AI supports HIPAA for healthcare and life sciences with zero data retention as a default for open-model inference, but Responses API, batch jobs, and training involve designed storage and should be treated separately.

Modal is managed-only with no BYOC option; Baseten and Fireworks offer VPC or enterprise deployment configurations that can be set up for BYOC, though the specifics vary. Confirm current capabilities directly with each vendor.

The hyperscalers bring strong compliance pedigrees. AWS Bedrock is HIPAA-eligible under the AWS BAA with private endpoints via PrivateLink. Google Vertex AI covers HIPAA under the Google Cloud BAA with VPC Service Controls. Azure OpenAI is HITRUST certified and HIPAA-covered under Microsoft's BAA, with Private Link support. The consistent trade-off across all three is provider lock-in: each ties your inference architecture to one cloud, which creates fragility when health systems operate across providers or need flexibility to move workloads based on cost or regional requirements.

Why running inside your own VPC reduces a whole class of compliance risk

When an inference platform runs in the vendor's infrastructure, even with strong isolation guarantees, the vendor is a data custodian. That creates concrete exposure. The vendor's subprocessors are now your subprocessors. Their incident response procedures affect your HIPAA obligations. Their infrastructure decisions, including region changes and hardware migrations, can affect your data residency controls without your direct involvement.

Moving to a customer VPC architecture can materially reduce that exposure by limiting vendor access to your data plane, but it does not automatically change the legal relationship. Whether it does depends on whether the vendor has any actual access to PHI or systems containing it, and on the contractual terms: the BAA, support access provisions, monitoring and telemetry practices, and subprocessor commitments. Confirm those in writing before treating VPC deployment as a categorical compliance solution.

Beztly AI's Mauvi deploys the control plane into the customer's own cloud accounts. According to Beztly's product specifications, the model endpoints, the routing layer, and the inference traffic stay within the customer's VPC, and Beztly does not access the data plane. Kuvai enforces environment isolation at the infrastructure level: sandbox, staging, and production are separated by policy, with explicit promotion gates and audit records at each boundary. Every rollout is versioned and reversible by default. Teams evaluating this architecture should review Beztly's BAA, support access model, and subprocessor list to confirm that the operational claim holds under their specific compliance requirements.

Because Mauvi operates as a provider-agnostic control plane, teams can serve from AWS in one region and GCP in another without changing their application's API surface. Failover between providers happens automatically. According to Beztly's internal benchmarks, routing overhead adds less than 40ms at p99. Request the test methodology and configuration details to evaluate how that applies to your specific workload and concurrency profile. Achieving equivalent multi-cloud flexibility with hyperscaler endpoints requires building the abstraction layer in-house, which is a significant engineering commitment that most teams underestimate until they are already locked in.

What to actually ask before committing to a platform

Teams evaluating inference platforms for healthcare AI use should request and review specific documentation before any contract conversation. Ask for the current SOC 2 Type II report: the full report, not a one-page summary. Ask for the HIPAA BAA with its exclusions listed explicitly. Then request three more things separately: the subprocessor list with individual compliance status, data retention and deletion policies per environment, and a written description of which features fall outside the BAA's scope.

Modal's exclusion of Volumes and Memory Snapshots is a concrete example of a fine-print pattern that exists in the documentation but does not appear in marketing materials. Every vendor with a 'HIPAA compliant' badge has a document like this somewhere. Read it.

A productive 30-day pilot should measure four things against predefined thresholds: p99 end-to-end latency under realistic clinical concurrency, audit log completeness and format for a sample request batch, promotion gate behavior between staging and production environments, and failover time when a provider endpoint degrades. Run the pilot with synthetic PHI-equivalent data, not real PHI, until compliance documentation is reviewed and signed. Define your latency SLO before you start measuring; a p99 result without a declared threshold is just a number.

Making the shortlist with confidence

The right platform for production clinical inference is not a universal answer. It depends on whether your primary constraint is compliance purity, latency, multi-cloud flexibility, or all three. For teams already committed to a single hyperscaler, AWS Bedrock and Azure OpenAI are credible choices if provider lock-in is acceptable and the compliance documentation matches your actual architecture.

For teams that handle PHI directly and need both a compliant development environment and a provider-agnostic serving layer, Beztly AI's Mauvi and Kuvai are worth evaluating as a combined option. Confirm the architecture against your specific compliance requirements and review the supporting documentation as you would with any vendor. The teams that pick well at this stage avoid a far more expensive re-architecture 18 months later.

Request the SOC 2 report. Read the BAA exclusions. Run the latency pilot with declared thresholds. Get your shortlist to two or three candidates before you open a contract conversation, and evaluate them against your specific clinical workload, not against each other's marketing pages. Teams building on open-weight models or operating across multiple cloud providers can contact Beztly AI to walk through how Mauvi and Kuvai fit their specific compliance and serving requirements.

Frequently asked questions

Is SOC 2 Type II enough to prove HIPAA compliance for our deployment?

No. SOC 2 Type II attests that specified controls operated effectively during an audit period but does not certify that any specific deployment is HIPAA-compliant or define a covered-services scope. You must review BAAs, architecture, and exclusions to determine HIPAA eligibility for your specific setup.

Can we run a pilot using synthetic PHI-equivalent data instead of real PHI?

Yes, and you should, until compliance documentation is reviewed and signed. This protects you from placing real PHI into environments before BAAs and exclusions are confirmed. Define your latency SLO and measure p99 end-to-end latency, audit log completeness, promotion gate behavior, and failover time during the 30-day pilot.

Does deploying a platform into our VPC eliminate all compliance risk automatically?

No. Deploying into your VPC reduces a whole class of exposure but does not automatically eliminate compliance risk. VPC deployment can limit vendor access to the data plane, but whether the legal relationship changes depends on contractual terms, support access provisions, monitoring practices, and subprocessor commitments. Confirm the BAA, support access model, and subprocessor list in writing before treating VPC deployment as a categorical solution.

What latency information should we require from vendors to judge applicability to our clinical workflows?

Require p99 end-to-end latency measurements along with the vendors' test methodology and hardware configuration. Routing overhead from proxy layers compounds at p99, and a platform adding 200ms at p99 is materially different from one adding 40ms. Request the test details so you can judge applicability to your concurrency profile and declared latency SLOs.