NVIDIA Cosmos for Developers
NVIDIA Cosmosâ„¢ is a platform of state-of-the-art generative world foundation models (WFMs), advanced tokenizers, guardrails, and an accelerated data processing and curation pipeline for autonomous vehicles (AVs) and robotics developers.
Build, evaluate, deploy, and simulate physical AI models faster while minimizing testing and validation risks in the real world.
How It Works
Cosmos WFMs accelerate physical AI development, helping developers augment datasets and post-train downstream world models for robots and autonomous vehicles.
Cosmos Predict generates next frames based on input to build datasets predicting various edge cases and serves as the foundation for all world models.
Cosmos Reason acts as a critic, using chain-of-thought reasoning to evaluate synthetic visuals and reward outcomes. It can also generate captions to speed up data curation.
Cosmos Transfer amplifies structured video across various environments and lighting conditions.
Developers can use the available PyTorch inference and post-training scripts along with model checkpoints. Cosmos NIM microservices are in development—Cosmos Predict NIM microservices are available here.
NVIDIA Cosmos World Foundation Models
A family of pretrained models for world generation as videos for accelerating physical AI development. Available openly to developers on NGC, Hugging Face, and GitHub.
Cosmos Predict-2
Our best-performing world foundation model yet—higher fidelity, flexible frame rates and resolutions, fewer hallucinations, and better text, object, and motion control in the video.
Generate previews from text in under 4s and up to 30s of future world video from reference image, or preview.
Cosmos Transfer
For controllable and photorealistic synthetic data at scale.
Input: Segmentation maps, depth signals, lidar scans, key points, trajectories, HD maps, and ground-truth simulations from NVIDIA Omniverseâ„¢.
Output: Photorealistic world scenes, conditioned based on inputs, mirroring layout, object placement, and motion.
Cosmos Reason
For physical AI reasoning.
Fully customizable, multimodal reasoning model trained using visual-language fine-tuning and reinforcement learning that uses a chain of thoughts to plan responses.
The model enables intelligent decision-making by reasoning and rewarding optimal responses.
Cosmos Predict-1
For out-of-the box world generation and post-training.
A generalist model that generates world states from text or video prompts and synthesizes continuous motion by predicting frames between a given start and end frame.
These models range from 4 billion to 15 billion parameters and can be used based on inference requirements.
Cosmos Tokenizers
A suite of image and video tokenizers that advances the state of the art in visual tokenization for world model training.
Cosmos WFM Post-Training Samples
Post-trained Cosmos Predict WFM generates predictive world states for autonomous vehicles, creating single or multi-view videos from ground-truth input for 360° environmental awareness in AV training.
Cosmos Guardrails
A set of guardrails, including a pre-guard to block harmful inputs and a post-guard to ensure safety and consistency in generations.
Cosmos Prompt Upsampler
Transform original input prompts into more detailed and enriched versions for higher-quality outputs from Cosmos WFMs.
Introductory Resources
Develop Custom Physical AI Foundation Models With NVIDIA Cosmos Predict-2
Cosmos Predict-2 is a suite of improved physical AI foundation models designed to generate realistic, physics-aware simulation data for training robots and AVs.
End-to-End AV Development With New Cosmos WFMs
Cosmos Predict-2 and Cosmos Transfer, accelerate end-to-end AV development by enabling high-quality SDG and unlocking new data sources such as generating multi-view videos from single-view footage.
Scale SDG With the NVIDIA NeMo Agent Toolkit
Agent toolkit is built using NVIDIA Omniverse, OpenUSD, Cosmos WFMs, and NVIDIA NIM microservices to automate and scale the generation of high-quality SDG, and accelerate the training and deployment of physical AI systems.
Starter Kits
Start solving physical AI challenges by developing custom world models with Cosmos or using Cosmos WFMs for downstream use cases. Explore implementation scripts, explainer blogs, and more how-to documentation for various stages of physical AI development.
Post-Training Cosmos WFMs
Cosmos WFMs are purpose-built for post-training. Use domain-specific datasets to build world models or post-train for different types of output, such as action generation for policy models.
Synthetic Data Generation
Build and deploy world models for infinite domain-specific synthetic data. Use NVIDIA Omniverse for physics-based conditioning.
Cosmos Learning Library
More Resources
Ethical Considerations
NVIDIA believes Trustworthy AI is a shared responsibility, and we have established policies and practices to enable development for a wide array of AI applications. When downloading or using this model in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.
NVIDIA has collaborated with Google Deepmind to watermark generated videos from the NVIDIA API catalog.
For more detailed information on ethical considerations for this model, please see the System Card, Model Card++ Explainability, Bias, Safety & Security, and Privacy Subcards. Please report security vulnerabilities or NVIDIA AI concerns here.
Get Started With NVIDIA Cosmos Today