NVIDIA announced new USD-based generative AI and NVIDIA-accelerated development tools built on NVIDIA Omniverse at SIGGRAPH 2024. These advancements will expand adoption of Universal Scene Description (OpenUSD) to robotics, industrial design, and engineering, so developers can quickly build highly accurate virtual worlds for the next evolution of AI.
OpenUSD is an open-source framework and data interchange for 3D and simulation workflows. This versatility enables developers to build highly accurate, AI-enabled virtual worlds and the tools that will make these worlds possible.
With new OpenUSD-based generative AI and NVIDIA-accelerated development tools built on NVIDIA Omniverse, developers in more industries can create applications for visualizing industrial design and engineering projects, or for simulating environments to build the next wave of physical AI and robots. New developer tools include:
- NVIDIA NIM microservices and workflow guides for OpenUSD language, materials, spatial intelligence, and physical AI.
- OpenUSD Exchange SDK and USD connectors to support industry-specific data including robotics and industrial simulation formats.
- NVIDIA Omniverse Kit SDK and developer tooling to build and deploy OpenUSD-based applications from scratch.
Bringing generative AI to OpenUSD workflows
Today, generative AI is relatively easy for text- and numeric-based applications. But for generative AI to work on 3D scenes, the AI needs to have both spatial intelligence and physical intelligence.
New NVIDIA NIM USD microservices for OpenUSD language, materials, spatial intelligence, and physical AI bring generative AI to OpenUSD workflows. As a developer, you can use these microservices to incorporate fine-tuned generative AI copilots and agents into your 3D workflows. This expands the adoption of USD into new industries including manufacturing, automotive, and robotics.
The microservices available in preview are:
- USD Code NIM microservice: State-of-the-art LLM that answers OpenUSD knowledge queries and generates USD-Python code.
- USD Search NIM microservice: AI-powered search for OpenUSD data, 3D models, images, and assets using text- or image-based inputs.
- USD Validate NIM microservice: Verify compatibility of OpenUSD assets with RTX render and rule-based validation.
WPP, the world’s largest marketing and communications company, is one of the early adopters for integrating NIM microservices. Working with The Coca-Cola Company, WPP is using USD Search and Code to accelerate iteration on creative campaigns at global scale.
Other early adopters include innovative NVIDIA Inception members HOMEE AI and Livvy by Aireal. HOMEE AI is an early adopter of USD Search. They plan to use the NIM microservice to enable intelligent search for users of their AI-powered spatial planning. Another early adopter, Aireal, plans to use USD NIM microservices including USD Search to help them develop AI-powered tools to revolutionize the interior design industry.
Additional NIM microservices for USD development are coming soon. These include USD Layout NIM, a generative AI model for assembling OpenUSD-based scenes from a series of text prompts, and USD SmartMaterial NIM for applying realistic material to 3D objects.
NIM microservices are also coming soon for fVDB, a deep-learning framework for building physical AI with spatial intelligence. fVDB (in early access) builds NVIDIA-accelerated AI operators on top of OpenVDB to generate AI-ready virtual environments that capture the massive spatial scale and high resolution of the real world. To learn more, see Building Spatial Intelligence from Real-World 3D Data Using Deep-Learning Framework fVDB.
Get started with NIM reference workflows, including the ChatUSD workflow, for building an interactive chat interface with USD Code or accelerating synthetic 3D data generation pipelines with USD Search and USD Code.
Build USD-native applications with NVIDIA Omniverse
You can build OpenUSD-native applications from scratch with Omniverse—leverage the Omniverse Kit SDK and developer tooling to get started quickly. The latest release, Omniverse Kit SDK 106, is packed with new tooling and enhanced functionality, delivering greater ease and control to development workflows. Omniverse Kit is now in general availability and fully enterprise supported with Omniverse Enterprise licensing.
Included with this release is the Omniverse Kit App Template repository, which provides a variety of application templates that can be used as a starting point and customized to fit the specific needs of any project. This template-based tooling significantly reduces development time, so you can focus on integrating the technologies, data pipelines, and functionality to solve previously intractable problems with the power of NVIDIA AI.
Also available is the Omniverse Kit App Streaming API, now in early access. This API enables you to deploy Kit-based applications to and stream interactively from the cloud. You can self-host these applications on any public cloud and integrate them seamlessly into web-based solutions.
Connect your industry-specific data to OpenUSD
To further advance the OpenUSD ecosystem expansion, NVIDIA announced the OpenUSD Exchange SDK, available later this year on GitHub. With this SDK, you can build your own robust OpenUSD data plugins, converters, and exchangers. Developers everywhere will be empowered to more easily access and contribute to the growing list of third-party solutions that can be used to connect common data sources with OpenUSD.
OpenUSD Exchange will be especially useful to developers who need to interchange industry-specific data with USD. To help jumpstart this, NVIDIA continues to build and support connectors from key formats to OpenUSD such as the Unified Robotics Description Format (URDF). Connecting URDF to OpenUSD enables roboticists to seamlessly bring their robot data across applications from design, to simulation, to reinforcement learning.
Additionally, NVIDIA announced new developer tools and APIs to stream large-scale OpenUSD scenes to Apple Vision Pro (in early access) through the NVIDIA Graphics Delivery Network (GDN).
Get started developing on Omniverse
We’re excited to see how you use these new tools and capabilities to develop your own OpenUSD workflows and applications.
- Get developer access to NVIDIA Omniverse for all the essentials you need to start developing on Omniverse.
- Access a collection of OpenUSD resources, including USD examples and samples.
- Connect with the Omniverse Developer Community.
For industry-specific use cases, check out the reference architectures for Generative AI Content Creation, Virtual Factories and Product Configurators. Stay tuned for additional reference architectures coming soon for popular Omniverse use cases.
At SIGGRAPH 2024, NVIDIA CEO Jensen Huang sat down for fireside chats with Meta founder and CEO Mark Zuckerberg and WIRED Sr. Writer Lauren Goode. Watch the fireside chats and other sessions from NVIDIA at SIGGRAPH 2024 on demand.
Stay up to date by subscribing to the newsletter, and following NVIDIA Omniverse on YouTube, Discord, and Medium.
