Jetson AGX Xavier: Deep Learning Inference Benchmarks

This page provides initial benchmarking results of deep learning inference performance and energy efficiency for Jetson AGX Xavier on networks including ResNet-18 FCN, ResNet-50, VGG19, GoogleNet, and AlexNet using JetPack 4.1.1 Developer Preview software. Performance and power characteristics will continue to improve over time as NVIDIA releases software updates containing additional features and optimizations for Jetson AGX Xavier.

ResNet-18 FCN (Fully Convolutional Network) for semantic segmentation operates at full HD resolution (2048x1024) and is representative of autonomous machine workloads involved with perception, path planning, and navigation. ResNet-50, VGG19, GoogleNet, and AlexNet perform recognition and classification on image patches with 224x224 resolution, and are commonly used as the encoder backbones of various object detection and segmentation networks. Using a batch size of 8 or higher at the lower resolution can be used to approximate the performance and latency of a batch size of 1 at higher resolutions. Robotic platforms and autonomous machines often incorporate multiple cameras and sensors which can be batch processed for increased performance, in addition to performing detection of regions-of-interest (ROIs) followed by further classification of the ROIs in batches.

With the recent availability of substantial computational resources at the edge, applications are deploying increasingly complex networks, such as variants of ResNet and VGG, for improved accuracy. Here we provide GoogleNet and AlexNet for historical completeness (denoted in the tables in gray), and in the future we will be updating this document to include additional networks for tasks like object detection, motion planning, and those that incorporate Recurrent Neural Networks (RNNs) for processes such as speech recognition and image captioning.

The benchmark measurements below were collected with the following environment:

• Jetson AGX Xavier Developer Kit
• JetPack 4.1.1 Developer Preview
• TensorRT 5.0, using trtexec^** tool
• Concurrent use of GPU (INT8) and two DLA’s (FP16)
• MAX-N and 15W power modes (nvpmodel)

Total module power consumption and energy efficiency measurements include the usage of CPU, GPU, DLAs, memory, miscellaneous SoC power, I/O, and regulator efficiency losses on all rails. Power consumption was measured using INA voltage and current monitors onboard the module.

Estimates of future performance, incorporating software enhancements such as INT8 support for the DLA’s and additional GPU optimizations, are provided for various network configurations. The future performance estimates presume concurrent use of GPU (INT8) and two DLAs (INT8).

15W Mode (HD)

MAX-N Mode^* (HD)

15W Mode

MAX-N Mode^*

15W Mode (Per-Rail)

The following provides a breakdown of the individual power rails, with each rail including regulator efficiency losses, for a subset of ResNet-50:

Notes

^* Module power in MAX-N mode with the JetPack 4.1.1 Developer Preview release may exceed TDP for some configurations. Users should tune the power profile and configuration to stay within the TDP for their application. Future versions of JetPack will further optimize performance and power.

^** Example trtexec launch commands:

For GPU

$ ./trtexec --avgRuns=100 --deploy=resnet50.prototxt --int8 --batch=8 --iterations=10000 --output=prob --useSpinWait

For DLA (Core 0)

$ ./trtexec --avgRuns=100 --deploy=resnet50.prototxt --fp16 --batch=8 --iterations=10000 --output=prob --useDLACore=0 --useSpinWait --allowGPUFallback

For DLA (Core 1)

$ ./trtexec --avgRuns=100 --deploy=resnet50.prototxt --fp16 --batch=8 --iterations=10000 --output=prob --useDLACore=1 --useSpinWait --allowGPUFallback

Multiple instances of trtexec can be launched simultaneously in this fashion for concurrent execution of the GPU and DLA’s. DLA supports a maximum batch size of 32 depending on the network, while the GPU can run higher batch sizes concurrently.