Developing Healthcare Robotics with GPU-Native Medical Physics Simulation

Unlike autonomous driving or industrial robotics, healthcare robotics can’t rely on internet-scale data collection or unlimited real-world experimentation. Every demonstration requires specialized equipment, clinical expertise, and access to patients or laboratory environments. This creates three fundamental challenges for developers.

First is the data gap. Training modern robotic policies requires demonstrations across diverse anatomies and procedures. Most teams have only hundreds of demonstrations—not the tens of thousands needed to build robust systems. But even if collecting millions of demonstrations were practical, the most important cases would still be missing. This is because healthcare is dominated by the long tail. Rare anatomies, challenging patient physiology, complications, and failure modes occur too infrequently to be adequately represented in real-world datasets. Yet, these are the cases that matter most for clinical safety. 

This data gap leads directly to the second challenge: generalization. Imitation learning inevitably plateaus at the edges of the data distribution. Reinforcement learning (RL) offers a path beyond that plateau. RL can explore millions of interactions, stress-test policies, and learn from failures. To do so effectively, it needs simulation that is realistic enough to produce meaningful policies—and fast and scalable enough to train at scale.

The third challenge is development velocity. Medical robotics development relies on benchtop phantoms, cadaver studies, animal models, and limited clinical evaluations. These remain essential, but they are inherently sequential, expensive, and difficult to scale. Iterating on designs and algorithms currently takes months, pushing total development cycles to 4–7 years.

These three challenges point to the same missing infrastructure: an open, GPU-native simulation framework capable of modeling device–anatomy interactions with the fidelity required for robot training. 

NVIDIA Medical Physics Simulation framework—an open source, GPU-accelerated capability within NVIDIA Isaac for Healthcare, was built to fill that gap. Developers can generate anatomical digital twins, simulate device–anatomy interactions and medical imaging, and train reinforcement learning policies at GPU scale—all within NVIDIA Isaac Sim and NVIDIA Isaac Lab.

A GPU-native simulation framework for healthcare robotics

Medical Physics Simulation provides both classical physics-based solvers and generative world-model-based simulators for real-time simulation of device–anatomy interactions—enabling RL-ready policy training across healthcare robotics segments. Both modules provide a modular research and engineering platform for interactive surgical and interventional simulation, robot learning, synthetic data generation, and procedure development. They are well suited for machine learning workflows because both simulation and learning run on the same GPU, avoiding the overhead of repeated CPU–GPU memory transfers.

Classical solvers  

The Endoluminal Simulation Module is released for general availability in Isaac for Healthcare. It enables real-time simulation of diagnostic and interventional procedures involving long, flexible surgical instruments navigating through endoluminal cavities. It is implemented as a standalone package, so it can be integrated independently into different workflows and environments. This initial release shows catheter navigation through the vascular system under fluoroscopic guidance.

The module is implemented in Python using NVIDIA Warp and Newton Physics. The flexible instruments are modelled as Cosserat rods, providing a solid theoretical foundation for simulating materials’ bending, twisting, and stretching deformation. To capture the complex nonlinear dynamics of these one-dimensional rods efficiently on the GPU, extended position-based dynamics (XPBD) was selected as the primary simulation method. 

XPBD typically relies on local, iterative constraint projections. For long instruments, however, local projections may require many iterations to propagate motion from the controlled proximal end to the distal tip. This module instead assembles the coupled rod constraints into a matrix system. 

Each rod segment contributes six constraint equations—three for stretch and shear and three for bending and twisting—producing a block-tridiagonal XPBD system with 6 × 6 blocks. The Thomas algorithm solves this system in linear time with respect to instrument length, while independent instruments are processed concurrently across vectorized environments on the GPU. 

This globally coupled solver propagates inputs such as proximal insertion and rotation along the entire instrument within each simulation step, even for very long instruments, enabling immediate response to user manipulations at the distal end.

Simulating catheter-vessel interactions

Catheter–vessel interaction is also evaluated in parallel on the GPU. Signed closest-point queries are evaluated directly against the patient-specific triangle mesh, avoiding a separately precomputed distance field. Instrument samples are projected back into the lumen when they penetrate the wall while retaining tangential sliding motion. The current release supports rigid, one-way containment; two-way coupling to deformable vessel walls is planned for a future release.

Control inputs are expressed directly through the rod’s boundary conditions and rest state. Insertion advances the constrained proximal end along the introducer axis, hub rotation sets its orientation, and distal steering modifies the rest curvature of the final rod segments rather than applying an artificial external force. The bend-and-twist constraints then propagate these inputs through the instrument.

Connecting physics, imaging, and robot learning

The Endoluminal Simulation Module maintains compatibility with other Newton solvers through a common API. Torch–Warp interoperability, provided through wp.from_torch and wp.to_torch keeps the control and reinforcement-learning interface zero-copy and fully GPU-resident. This enables straightforward integration with NVIDIA Isaac Lab, a framework designed for large-scale robot learning. Isaac Lab manages environment orchestration and rollout control, while the solver executes the GPU-based dynamics and returns device-resident tensors for observation, reward, and control loops.

This co-design of physics and imaging supports high-throughput policy training at scale, including experiments with 512 parallel environments and 1,500 training iterations. Reported performance is approximately 1,300 Hz for single-environment physics, 60 Hz for physics simulation across 512 environments, and 63 frames per second for the complete simulation and rendering loop at a resolution of 256 × 256 pixels. The platform also provides a foundation for future policy training directly from fluoroscopic images and other pixel-based observations.

The catheter navigation workflow runs as a unified simulation loop that combines two coupled patient-specific components. The vasculature-digital-twin package generates the anatomical attenuation/geometry artifacts (mu_volume, vessel mask/mesh, and centerline), and the endoluminal simulation package advances catheter mechanics under XPBD constraints inside that anatomy. For each frame, the solver state is transformed into renderable catheter segments and composited by fluorosim into the same CT-derived imaging domain. This keeps the physics and fluoroscopy synchronized in one end-to-end loop while additional detector-response effects improve the realism of the resulting fluoroscopic images.

Surgical Simulation Module 

The Surgical Simulation Module available through early access is a real-time GPU-accelerated surgical simulation framework, designed to model soft-tissue deformation and surgical interactions with haptic feedback and rendering within a unified simulation pipeline.

Similar to the Endoluminal Simulation Module, it is also implemented in Python using NVIDIA Warp and Newton Physics. Unlike monolithic physics engines, it organizes its functionality as explicitly ordered simulation systems. Each procedure is configured through the sequence of deformation, collision, grasping, cutting, clipping, diathermy, and rendering operations. This makes the physical GPU pipeline inspectable, configurable, and easier to extend.

Soft tissue is represented using tetrahedral meshes. Deformation is solved with position-based dynamics using distance- and volume-preserving constraints. Corrections are accumulated and averaged on the GPU, reducing constraint-ordering artifacts and supporting efficient parallel execution (Jacobi solver). 

Additionally, both simulation and visualization loops remain on the GPU, including deformation, collision queries, grasping, thermal diffusion, topology updates, and surface reconstruction. NVIDIA CUDA graph capture further reduces kernel-launch overhead, while direct GPU-to-renderer data transfer (CUDA interop) avoids unnecessary CPU/GPU copies. This enables real-time simulation at more than 30 frames per second with eight physics substeps and eight internal deformation constraint iterations on one consumer-grade GPU. 

The initial release of this module supports an end-to-end cholecystectomy procedure (gallbladder removal). The simulation begins with the robotic instruments inserted into the inflated abdominal cavity, visualizing key structures like the liver, gallbladder, and connective fatty tissue. It involves dissection of the hepatocystic triangle, clipping and cutting the cystic duct and artery, and separating the gallbladder from the liver.

Learned world dynamics and generative physics for surgical simulator 

The generative approach to medical physics simulation is a complementary technology to physics solvers. Rather than explicitly programming every interaction, a generative world model learns from video and other data how real scenes look and how they change over time. During training, it learns relationships between visual observations, motion, and conditioning signals such as robot actions. At inference time, it uses this learned world knowledge to predict the next video frames. 

Because the result is produced in the visual space seen by cameras and robotic policies, the model can reproduce scene detail and appearance that would be difficult to build by hand. It also avoids much of the manual work needed to create every asset, material, and lighting setup for a conventional simulator. The tradeoff is that these predictions are learned from data: they can be visually realistic without providing the explicit physical guarantees of a numerical solver.

NVIDIA Cosmos provides the world foundation models for this approach. Cosmos video models use a flow-based diffusion-transformer architecture with a rectified-flow formulation. During generation, the model iteratively transforms noise in a compressed video representation into a coherent sample from the learned data distribution. A video tokenizer compresses pixel-space frames into this latent representation for efficient modeling and decodes the generated latents back into video. Multimodal conditioning makes the generated future controllable. For medical applications, post-training on domain data adapts the broad visual model before specialized anatomy, devices, procedures, and dynamics.

Generative simulation with Cosmos-H

The Cosmos-H family shows several forms of generative medical physics simulation. Cosmos-H-Surgical-Predict forecasts plausible future surgical video from an initial image and text. Cosmos-H-Surgical-Transfer generates surgical video guided by controls such as depth, edges, segmentation, or blur. Cosmos-H-Surgical-Simulator predicts future frames conditioned directly on surgical-robot kinematics. 

Together, these models support use cases ranging from synthetic-data generation and controlled domain transfer to action-conditioned robot training and policy evaluation. They do not replace explicit physics in every setting, but reduce scene-authoring burden and provide realistic, observation-level simulations where complete first-principles modeling is impractical.

Cosmos-H-Dreams extends this approach to interactive simulation. It distills a teacher fine-tuned from Cosmos-H-Surgical-Simulator into a causal, few-step student that autoregressively streams future surgical video in response to robot actions. Served through the FlashDreams inference engine, it turns a high-quality offline world model into a real-time environment with frame generation at more than 30 frames per second. The environment can be controlled by a human operator or queried in closed loop by a learned robotics policy on a single workstation GPU.

Why healthcare robotics needs both approaches

Isaac for Healthcare supports both classical and generative simulation to address the complementary requirements of healthcare robotics. To provide user trust and quantitative quality assessment, which are the foundations of any healthcare solution, the physically consistent and deterministically modeled environments provided by classical simulation are essential. Generative simulation, in turn, produces diverse and visually realistic environments at scale that classical simulation cannot, enabling clinical applications in both data synthesis and real-time clinical workflows.

Get started with medical physics simulation

Medical Physics Simulation in Isaac for Healthcare provides developers with the building blocks to create patient-specific digital twins, simulate device–anatomy interactions, generate synthetic experiences, and train intelligent healthcare robotics systems. The following resources help you get started with each simulation capability.

Build an Endoluminal Robotics Workflow

Build patient-specific catheter navigation simulations using GPU-accelerated physics, anatomical digital twins, and simulated medical imaging.

1. Generate and create a vasculature digital twin.
2. Incorporate a catheter-vasculature interaction solver.
3. Simulate X-ray perception using X-ray sensor simulation.
4. Build the end-to-end catheter navigation workflow. 

Cosmos-H Dreams generative surgical simulation

Cosmos-H Dreams enables scalable generation of realistic surgical experiences using world foundation models. Developers can use pretrained models, run interactive simulation, or adapt the framework to new procedures and robotic embodiments.

1. Run Cosmos-H Dreams. Get started with pretrained checkpoints, examples, and interactive simulation workflows in the GitHub repository.
2. Download the Cosmos-H Dreams model. Access pretrained model checkpoints and configurations from the Hugging Face checkpoint.
3. Create a custom Cosmos-H Dreams model. Cosmos-H Dreams is not limited to the procedures and robots included with the framework. Choose one of two paths to build a real-time simulator for your own embodiment:
   • Extend Cosmos-H:
     1. Fine-tune Cosmos-H-Surgical-Simulator on your data by following its training guide.
     2. Distill the fine-tuned checkpoint to a few-step diffusion schedule for real-time throughput using the distillation guide.
     3. Deploy the distilled checkpoint with --pipeline.diffusion-model.transformer.checkpoint-path.
   • Bring your own model: If you are starting with a different video diffusion architecture or a model trained on a proprietary dataset, integrate it with FlashDreams, the high-performance inference runtime that powers Cosmos-H Dreams.