Today’s immersive extended reality (XR) devices require heavy optics and displays to be secured with head straps, a necessity that adds bulk and presents a social barrier. In the envisioned future, head straps will not be required. Over the past few years, NVIDIA Research has been researching how to develop smaller and lighter XR glasses in collaboration with the Computational Imaging Group at Stanford under Professor Gordon Wetzstein.
The most important rule in the optical design of XR glasses is that the display should be closer, and the image should appear farther away. As the display moves farther from the eyes, the center of gravity also shifts, necessitating the display being positioned close to the user’s face for a compact and lightweight design.
However, human eyes cannot properly perceive a display that is too close, so optical systems like lenses must be used to relay the image to a comfortable viewing distance. For this system to be effective, the display needs to be located near the lens’s focal length, which paradoxically requires placing the display farther away (Figure 1).
One solution is a pancake lens that uses a folded optical path. However, this approach introduces new issues such as aberrations and additional weight. Optical waveguides are also often used to achieve compact XR glasses, but they come with their own challenges, including limited viewing angles and a fixed focus at infinity.
Holographic glasses for XR
Holographic near-eye displays are gaining attention as a technology capable of overcoming these limitations. The new display element, the Spatial Light Modulator (SLM), manipulates not the intensity but the phase of light, enabling the reconstruction of 3D holograms in front of or behind the SLM. Using this feature, 3D images can be positioned near the lens’s focal length, even if the display is very close to the lens.
Employing this principle, our team introduced holographic glasses for virtual reality (VR) that are only 2.5 mm thick (Figure 2), as presented in Holographic Glasses for Virtual Reality. With waveguides, holographic near-eye displays, and geometric phase lenses, VR glasses can be manufactured with minimal thickness without free-space propagation. The team implemented benchtop and wearable prototypes for testing.
Our binocular wearable prototype supported 3D focus cues. It provided a diagonal field of view of 22.8 degrees with a 2.3 mm static eye box and additional capabilities of dynamic eye box with beam steering while weighing only 60 g, excluding the driving board. With an idea utilizing the user’s pupil as a natural Fourier filter, a true glasses-form factor holographic VR was presented for the first time.
The journal Nature recently featured the study, Full-Colour 3D Holographic Augmented Reality Displays with Metasurface Waveguides. This research introduces unique holographic augmented reality (AR) glasses that combine inverse-designed full-color metasurface gratings, compact dispersion-compensating waveguide geometries, and AI-driven holography algorithms.
Employing a similar approach as our previous work, this research eliminates lenses from the optical path and strategically uses full-color metasurface gratings as in-couplers and out-couplers. As demonstrated in Figure 3, conventional AR glasses use amplitude SLMs, such as organic light-emitting diodes or micro light-emitting diodes, which require a projector-based light engine that is typically at least as thick as the focal length of the projection lens.
The design of our holographic AR glasses uses a phase-only SLM that can be mounted very close to the in-coupling grating, thereby minimizing the device form factor. Unlike conventional AR glasses, our holographic design can provide full 3D depth cues for virtual content. Figure 4 shows experimental results captured through our compact holographic display prototype. This innovative work is the first to successfully implement full-color 3D holography in a perfectly transparent glasses structure.
Another important feature of our design is the use of an AI-driven holography algorithm, which enables the use of the previously introduced compact and lightweight holographic XR glasses. SLMs are powered by coherent light sources such as lasers, where precise manipulation of coherent wavefronts in a waveguide system is crucial for holographic displays yet very challenging due to the interfering nature of coherent light. To tackle this challenge, we have developed a mathematical model that describes the propagation of coherent waves in a waveguide using a combination of physically accurate modeling techniques and AI.
As illustrated in Figure 5, the physical aspects of the waveguide (highlighted in green) are combined with AI components learned from camera feedback (highlighted in orange). In our model, the input phase pattern (left) applies a per-pixel phase delay, from 0 to 2π, to the converging illumination before the wavefront is modulated by the learned in-coupler efficiency. Note that efficiency in this context refers to the physical quantity that characterizes metasurfaces.
This wavefront is then sent through a CNN at the in-coupler plane and propagated through the waveguide, using its physically motivated transfer function, before an additional learned out-coupler efficiency is used to determine the out-coupled wavefront (center). The latter is propagated to the target scene at various distances from the user, where a CNN is applied, converting the complex-valued field into observed intensities (right).
When trained on a captured dataset, the learned parameters of the CNNs, the coupler efficiencies, and the waveguide propagation enable this model to predict the output of our holographic AR glasses accurately. The model is fully differentiable, enabling simple gradient descent computer-generated holography algorithms to compute the phase pattern for a target scene at runtime.
This model can be parameterized by neural network components that can be automatically learned from camera feedback. In actual holographic displays, even wavelength-scale misalignments can cause significant degradation in image quality, making AI-driven calibration essential, especially in complex structures such as waveguides. The proposed wave propagation model accurately represents the physical optics, significantly improving image quality over alternative models (Figures 2 and 4).
Summary
Once thought impossible, AI has made practical holographic displays feasible and enabled the creation of 3D XR glasses that support focus cues. This field of research opens new possibilities for AI in displays. While AI is already used in content creation and human-computer interaction, it can also significantly reduce the size and weight of displays themselves. This represents a truly groundbreaking achievement that surpasses traditional expectations. These innovations also promise the potential for new display form factors, beyond the conventional 2D rectangular displays that have dominated since their inception.
To learn more, see Holographic AI Glasses with Metasurface Waveguides.
