Platform Software Stacks

This section describes the platform component software stacks.

Use the DRIVE OS software stack to build your autonomous vehicle applications.

Foundation Services Stack

The NVIDIA DRIVE AGX™ platform Foundation services runtime software stack provides the infrastructure for all the components of the platform. With this infrastructure, guest OS systems can run on the hardware, with the Hypervisor managing use of hardware resources.

media/image2.png

Foundation Components

Component Description
Hypervisor

Trusted Software server that separates the system into partitions. Each partition can contain an operating system or a bare-metal application. The Hypervisor manages:

  • Guest OS partitions and the isolation between them.

    Note: Multiple Guest OSs are not supported. In addition, you can run QNX or Linux, but not both.

  • Partitions' virtual views of the CPU and memory resources.

  • Hardware interactions

  • Run-lists

  • Channel recovery

Hypervisor is optimized to run on the ARMv8.2 Architecture.
guest OS

Allocates peripherals that Guest OS needs to control.

Services

Services for DRIVE Update.
Bootloader

Firmware that runs during boot to load firmware components, such as boot images, partition images, and other firmware.
Trusted OS

Trusted OS configuration in the PCT describes the configuration of the virtual Trusted OS device.
Orin SoC

System on a Chip hardware resources.

Virtualized Servers

The virtualized configurable servers are as follows.

Component Description
Boot and Power Manager Processor (BPMP) Server

Facilitates communication between Guest Virtual Machines (VM) and BPMP firmware.
VM Server

host1X virtualizes NvHost.

Security Engine (SE) Server

Para-virtualizes and allows multiple Guest Virtual Machines access to the security engine cryptographic hardware accelerator.
Trusted Applications (TA)

Through Trusted Applications, Trusted OS exposes a set of core services that use managed security assets in cryptographic operations without exposing them to non-secure guest software.
GPU

Runs on top of the virtualization core and handles sharing of GPU between multiple client Guest Virtual Machines. The GPU virtualization also includes the para-virtualized GPU client driver running inside of each Guest Virtual Machine.
Storage Server

Para-virtualizes storage access to enable sharing physical storage devices among multiple Guest Virtual Machines.
Debug Server Provides support for kernel-level debugging of Guest Virtual Machines (VM).

NvMedia Architecture

NvMedia provides powerful processing of multimedia data for true hardware acceleration across NVIDIA DRIVE® Orin™ devices. With the NvMedia and Orin firmware components, multimedia applications support multiple simultaneous camera feeds for simultaneous processing. The NvMedia features include:

  • Robust image processing and sensor control blocks for use by applications.
  • Hardware surface to handle all types of image formats such as RGB, YUV and RAW.
  • Functional components for image capture, processing, acceleration, encoding, and interoperating with other libraries.

Applications leverage the NvMedia Application Programming Interface (API) to process image and video data. Additionally, NvMedia can route image data to/from other components, such as OpenGL ES and NVIDIA® CUDA®.

NvMedia Stack

The NvMedia software stack supports these types of interactions:

  • Applications call the NvMedia Framework components to string together a sequence of processing steps for images.
  • The NvMedia Framework calls low-level hardware drivers to interact with the SoC components on the SoC chip.

Consult building and running NvMedia sample applications to build an NvMedia sample application.

media/image2.png

Each component functionality is as follows. The drivers expose a subset of the functionality available.

Tegra Hardware

Description

Video Input (VI)

Receives CSI data from the camera.

Image Signal Processor

(ISP)

Produces a processed image from image data captured from an image sensor. For example, it can make pixel-level changes such as inverting pixel bits, auto exposure, and white balance correction.

NVIDIA Encoder (NVENC)

Converts raw image data into one of the supported image formats.

NVIDIA Decoder (NVDEC)

Converts encoded image data into raw image data.

Video Interlace Compositor (VIC)

Converts video data for deinterlacing, composition, and format conversion.

Optical Flow Accelerator (OFA)

Accelerate optical flow and stereo disparity computation between the frames.

NvMedia API Architecture

The NvMedia API library is a frame-level, driver-level, threadless library that provides video and image processing pipeline acceleration across NVIDIA® Orin devices. The NvMedia API is consistent across all Orin devices while being operating system middleware and framework agnostic.

  • The Video API library supports interlaced video for human-machine interface (HMI) applications.
  • The Image API library supports RAW camera processing using Image Signal Processing (ISP) and progressive image processing for Advanced Driver Assistance Systems (ADAS) development.

NvMedia Video Surface

The NvMedia video surface:

  • Defines and handles YUV, RGB, Progressive, and Interlaced processing.
  • Does not imply a buffer context by the process handling which is required for hardware synchronization.
  • Is not required to access the CPU of the NVIDIA hardware buffer format.

The video surface components are as follows:

Video Component

Description

Decode

Decodes frame level bitstream; must be parsed externally to provide sequence and slice headers. Supports progressive and interlaced formats.

NvMedia Image Surface

The NvMedia image surface:

  • Defines and handles YUV, RGB, and RAW (progressive only) image processing.
  • Carries the image sensor register data for embedded line information.
  • Carries per image specific metadata that does not have to be per processing handle. The handle is required for hardware synchronization.
  • Allocation is signaled for CPU access of pitch linear buffer format.

The image surface components are as follows:

Image Component

Description

SIPL

The SIPL framework processes incoming image data with AE, AWB controls and includes the ISP processing capability. It supports camera tuning tools.

ISP

The Image Signal Processor produces a processed image from image data captured from an image sensor. For example, it can make pixel-level changes such as inverting pixel bits, auto exposure, and white balance correction.

ICP

Image Capture Processing that provides the ability to capture data over the CSI interface.

2D

Image 2D provides the ability to perform manipulation of image data; such as cropping, scaling, copying and converting format.

IEP

Image Encode Processing provides the ability to encode processed YUV surface inputs to H.264 and H.265.

Interoperability of EGLStream

Provides OpenGL and CUDA interoperability to share camera data processed by the GPU (GL or CUDA API).

IJE

Image JPEG Encoding provides the ability to encode YUV surfaces to JPEG format.

IJD

Image JPEG Decoding provides the ability to decode images compressed inJPEG format to raw YUV surfaces.

NvMedia APIs and Thread Safety

NvMedia APIs are not designed to be thread safe. It is the responsibility of the application to maintain thread safety. To ensure thread safety:

  • NvMedia components can be created and used in any thread but the APIs cannot be used from different threads concurrently.
  • Different instances of the same component can be used in parallel from different threads.
  • Encoders are designed to be fed from one thread and get the encoded bitstream from another thread.