Generative AI

Rapidly Generate 3D Assets for Virtual Worlds with Generative AI

A group of different animals standing together.

To accelerate the development of 3D worlds and the metaverse, NVIDIA has launched numerous AI research projects to help creators across industries unlock new possibilities with generative AI.

Generative AI will touch every aspect of the metaverse and it is already being leveraged for use cases like bringing AI avatars to life with Omniverse ACE. Many of these projects, like Audio2Face and Audio2Gesture, which generate animations from audio, have turned into widely loved tools in the Omniverse community.

At CES, NVIDIA unveiled new generative AI technologies coming to Omniverse to help create virtual worlds faster and easier than ever. These models are available as both third-party Connectors from NVIDIA partners and internal AI projects published by the NVIDIA research team as extensions in AI Toy Box.

All generative AI research projects and Connectors available in NVIDIA Omniverse use legally licensed and purchased datasets.

Accelerating 3D content creation with generative AI Connectors and extensions

NVIDIA Omniverse is bringing in the latest and greatest generative AI technologies with Connectors and extensions for third-party technologies. generates animations from body movements. enables you to capture human motion anywhere and export directly into Omniverse.

Video 1. An example of an individual boxing in NVIDIA Omniverse, with motion data powering the character.

Lumirithmic generates 3D mesh for heads from facial scans. The connector enables you to create movie-grade avatars from facial scans easily and enables 3D scanning at scale for any industry. “We’re excited to be working with NVIDIA to bring our 3D facial scanning products through Omniverse to creators, gamers, and developers, democratizing access to the highest-quality 3D faces for gaming, adtech, metaverse, and other applications,” said Gaurav Chawla, co-founder and CEO of Lumirithmic.

Video 2. Omniverse – Lumirithmic 3D Avatar Capture Integration

Elevate3D generates photorealistic 3D visualizations of products from 360° video recordings, leveraging neural radiance fields to create 3D meshes with unmatched visual fidelity. “Omniverse enables us to offer our technology to the widest range of workflows and industries. Additionally, the excellent documentation and developer experience have made it a pleasure to build our extension on the platform,” said Alexandre Carlier and Martin Danelljan, co-founders of Elevate3D. “This extension will make the challenges of scanning physical objects a thing of the past.”

Video 3. Elevate3D extension for Omniverse

Explore and experiment with the latest generative AI research

AI workflows are central to the Omniverse experience. That’s why NVIDIA is constantly undertaking new research projects to help 3D artists and creators accelerate their workflows with AI. Many of these AI projects turn into widely used features like pose estimation in Machinima and NVIDIA Canvas, which generate 360-degree images from a few 2D brushstrokes.

Announced at CES, GET3D is a new generative AI model that generates 3D shapes with topology, rich geometric details, and textures from data like 2D images, text, and numbers. These 3D shapes or textured meshes can take the form of animals and humans with cars, chairs, motorbikes, and buildings coming soon. This gives creators in several industries the ability to scale the quantity and diversity of content in their virtual worlds quickly.

GIF showing a field being populated with a herd of cows using GET3D
Figure 1. GET3D enables you to rapidly generate 3D animals and other shapes and meshes to populate a scene.

GET3D will be available in Omniverse AI Toy Box along with existing generative AI research projects published by NVIDIA, such as GANVerse3D Image2Car and AI Animal Explorer. AI ToyBox houses all the latest NVIDIA AI projects, empowering content creators to explore new possibilities in virtual worlds.

AI and the metaverse: a symbiotic relationship

Just as AI is crucial to the expansion of the metaverse, so too is the metaverse to the expansion of AI. Global industries are becoming software-defined, from cars to medical instruments to warehouse robots. These software-defined technologies cannot be developed or deployed without thorough testing in real-world environments.

Digital twins in the metaverse are providing physically accurate virtual environments for developers to simulate and test AI for software-defined technologies such as intelligent robots faster than ever before. Synthetic data created in the metaverse is further accelerating the training of cutting-edge AI applications by enabling AI to be trained on more data than was previously possible.

Generative AI is key to scaling these digital twins and virtual environments that will usher in a new era of AI and the metaverse.

Developers can build generative AI tools for 3D worlds with Omniverse’s modular development framework and enterprises can leverage the latest generative AI technologies to scale digital twin simulations with NVIDIA Omniverse Enterprise.

Subscribe to the Omniverse newsletter to receive updates about generative AI in Omniverse. Follow us on Instagram, Twitter, and YouTube to stay up to date with the next releases and use cases.

Visit the Omniverse Developer Resource Center for additional resources, view the latest tutorials on Omniverse, and check out the forums for support. Join the Omniverse community, Discord server, and Twitch Channel to chat with the community.

Discuss (0)