NVIDIA Cosmos Framework Boosts GR00T World Model Training

NVIDIA Cosmos Framework Expands GR00T Capabilities

NVIDIA introduced the Cosmos Framework on November 2, 2026, through its developer blog. The open modular toolkit supports training, reinforcement learning, optimization, and inference of world models. It directly ties into ongoing GR00T physical AI work by providing recipes that address sensor adaptation and sim-to-real transfer gaps across robotics, autonomous vehicles, and surgical applications.

The release includes support for multiple models beyond the core Cosmos 3 family. Developers can now work with NVIDIA Alpamayo, GR00T, Qwen, and pi0.5 through compatible Hugging Face checkpoints. Training pipelines accept data in JSONL, WebDataset, LeRobot, or custom formats, allowing full supervised fine-tuning or parameter-efficient LoRA adaptation across GPU clusters.

Reinforcement learning receives explicit coverage in the new framework. Supported methods include DPO for offline preference optimization, GRPO for online RL with rule or model rewards, PPO for actor-critic training with frozen references, OPD for on-policy distillation, and DiffusionNFT for diffusion-based world models. Rollout, reward, and training roles can share GPUs or run on separate pools while connecting to supported simulators.

Simulation Recipes Target Robotics and AV Workflows

Cosmos 3 Nano simulation checkpoints enable synchronized camera video and LiDAR generation for closed-loop evaluation. The Transfer-Auto variant supports bidirectional denoising of multi-view clips up to 400 frames at 30 fps in 480p or 720p resolution. The Sim-Auto variant provides causal, few-step-distilled autoregressive streaming for interactive policy rollouts.

Reference runs require four GPUs and produce seven 720p videos plus 99 LiDAR sweeps when using published sample configurations. GPU memory footprints are documented at 58 GiB in BF16 and 44 GiB in FP8 with offloading for Cosmos3-Nano generation tasks. These numbers come directly from the NVIDIA technical blog and allow developers to plan hardware allocation before starting runs.

The framework supplies agent skills under a .agents/skills directory for automated post-training, inference, and troubleshooting. A bounded prompt example instructs an agent to post-train a checkpoint on a dataset using a specific recipe, validate data splits, run smoke tests, and compare outputs against the base model using three seeds while recording quality, peak memory, and latency.

NIM Microservices Deliver Accelerated Inference

Cosmos NIM microservices package models for GPU-accelerated serving. Configurations incorporate quantization, speculative decoding, and KV-cache-aware routing. Speculative decoding delivers measured mean speedups of roughly 2–3x on H200 GPUs according to the November 2 release notes.

Developers select a NIM configuration matching their GPU, launch the container, verify readiness, and send inference requests through application APIs. Compatible runtimes include Hugging Face Diffusers, PyTorch, SGLang, vLLM-Omni, vLLM, TRT-LLM, and TRT-Edge-LLM. Distillation and quantization steps further reduce model size or generation steps before deployment.

Integration with GR00T Physical AI Pipeline

The Cosmos Framework release strengthens the existing Isaac GR00T stack. GR00T 1.7, the latest open VLA model, already incorporates a Cosmos backend and runs on Jetson Thor with TensorRT NVFP4 quantization at 25 Hz. The new tools let teams post-train GR00T alongside other models using the shared RL and optimization recipes.

Developers can capture demonstrations through teleoperation in simulation or on physical robots, convert the data, post-train policies, evaluate in Isaac Lab-Arena, and deploy for real-time control. The framework’s multimodal data processing and agent skills reduce manual scripting when adapting GR00T to new embodiments or tasks.

Related coverage of the GR00T family appears in [NVIDIA Isaac GR00T N1.7 VLA Model Ships with Human Video Pretraining](/articles/nvidia-isaac-gr00t-n1-7-vla-model-ships-with-human-video-pretraining). Earlier previews of GR00T N2 at GTC 2026 are detailed in [NVIDIA GR00T N2 Preview at GTC 2026 Advances Physical AI](/articles/nvidia-gr00t-n2-preview-at-gtc-2026-advances-physical-ai). Additional context on GR00T 1.7 launches and ecosystem partners is available in [NVIDIA GR00T 1.7 and Agile ONE SN Launch at GTC Berlin](/articles/nvidia-gr00t-1-7-and-agile-one-sn-launch-at-gtc-berlin).

Broader Ecosystem and Next Steps

NVIDIA also points to a technical report and a livestream scheduled for November 5, 2026, to demonstrate framework usage. The GitHub repository at https://github.com/NVIDIA/cosmos-framework hosts the code, recipes, and agent skills. Checkpoints are available on Hugging Face under collections such as nvidia/Cosmos3-Nano-Transfer-Auto.

Further reading on related NVIDIA robotics software includes [NVIDIA Isaac ROS 5.0 Pushes Agentic Robotics Toward Factory Floors](/articles/nvidia-isaac-ros-5-0-pushes-agentic-robotics-toward-factory-floors) and [NVIDIA SONIC Scales Humanoid Control Beyond Task-Specific Limits](/articles/nvidia-sonic-scales-humanoid-control-beyond-task-specific-limits). These pieces examine complementary layers of the physical AI stack that now benefit from Cosmos tooling.

The framework’s emphasis on reproducible agent-driven workflows and documented memory requirements gives teams concrete starting points for scaling world model development. By releasing both the training infrastructure and optimized inference paths on the same day, NVIDIA lowers barriers for developers working on sim-to-real transfer in manipulation and navigation tasks.