NVIDIA Data Center Deep Learning Product Performance

Review the latest GPU acceleration factors of popular HPC applications.

Please refer to Measuring Training and Inferencing Performance on NVIDIA AI Platforms Reviewer’s Guide for instructions on how to reproduce these performance claims.

Training
Inference

NVIDIA’s complete solution stack, from GPUs to libraries, and containers on NVIDIA GPU Cloud (NGC), allows data scientists to quickly get up and running with deep learning. NVIDIA® V100 Tensor Core GPUs leverage mixed precision to accelerate deep learning training throughputs across every framework and every type of neural network. NVIDIA breaks performance records on MLPerf, the AI’s first industry-wide benchmark, a testament to our GPU-accelerated platform approach.

NVIDIA Performance on MLPerf 0.6 AI Benchmarks

ResNet-50 v1.5 Time to Solution on V100

MXNet | Batch Size refer to CNN V100 Training table below | Precision: Mixed | Dataset: ImageNet2012 | Convergence criteria - refer to MLPerf requirements

Training Performance

NVIDIA Performance on MLPerf 0.6 AI Benchmarks

Framework	Network	Network Type	Time to Solution	GPU	Server	MLPerf-ID	Precision	Dataset	GPU Version
MXNet	ResNet-50 v1.5	CNN	115.22 minutes	8x V100	DGX-1	0.6-8	Mixed	ImageNet2012	V100-SXM2-16GB
		CNN	57.87 minutes	16x V100	DGX-2	0.6-17	Mixed	ImageNet2012	V100-SXM3-32GB
		CNN	52.74 minutes	16x V100	DGX-2H	0.6-19	Mixed	ImageNet2012	V100-SXM3-32GB-H
		CNN	2.59 minutes	512x V100	DGX-2H	0.6-29	Mixed	ImageNet2012	V100-SXM3-32GB-H
		CNN	1.69 minutes	1040x V100	DGX-1	0.6-16	Mixed	ImageNet2012	V100-SXM2-16GB
		CNN	1.33 minutes	1536x V100	DGX-2H	0.6-30	Mixed	ImageNet2012	V100-SXM3-32GB-H

PyTorch	SSD-ResNet-34	CNN	22.36 minutes	8x V100	DGX-1	0.6-9	Mixed	COCO2017	V100-SXM2-16GB
		CNN	12.21 minutes	16x V100	DGX-2	0.6-18	Mixed	COCO2017	V100-SXM3-32GB
		CNN	11.41 minutes	16x V100	DGX-2H	0.6-20	Mixed	COCO2017	V100-SXM3-32GB-H
		CNN	4.78 minutes	64x V100	DGX-2H	0.6-21	Mixed	COCO2017	V100-SXM3-32GB-H
		CNN	2.67 minutes	240x V100	DGX-1	0.6-13	Mixed	COCO2017	V100-SXM2-16GB
		CNN	2.56 minutes	240x V100	DGX-2H	0.6-24	Mixed	COCO2017	V100-SXM3-32GB-H
		CNN	2.23 minutes	240x V100	DGX-2H	0.6-27	Mixed	COCO2017	V100-SXM3-32GB-H
	Mask R-CNN	CNN	207.48 minutes	8x V100	DGX-1	0.6-9	Mixed	COCO2017	V100-SXM2-16GB
		CNN	101 minutes	16x V100	DGX-2	0.6-18	Mixed	COCO2017	V100-SXM3-32GB
		CNN	95.2 minutes	16x V100	DGX-2H	0.6-20	Mixed	COCO2017	V100-SXM3-32GB-H
		CNN	32.72 minutes	64x V100	DGX-2H	0.6-21	Mixed	COCO2017	V100-SXM3-32GB-H
		CNN	22.03 minutes	192x V100	DGX-1	0.6-12	Mixed	COCO2017	V100-SXM2-16GB
		CNN	18.47 minutes	192x V100	DGX-2H	0.6-23	Mixed	COCO2017	V100-SXM3-32GB-H

PyTorch	GNMT	RNN	20.55 minutes	8x V100	DGX-1	0.6-9	Mixed	WMT16 English-German	V100-SXM2-16GB
		RNN	10.94 minutes	16x V100	DGX-2	0.6-18	Mixed	WMT16 English-German	V100-SXM3-32GB
		RNN	9.87 minutes	16x V100	DGX-2H	0.6-20	Mixed	WMT16 English-German	V100-SXM3-32GB-H
		RNN	2.12 minutes	256x V100	DGX-2H	0.6-25	Mixed	WMT16 English-German	V100-SXM3-32GB-H
		RNN	1.99 minutes	384x V100	DGX-1	0.6-14	Mixed	WMT16 English-German	V100-SXM2-16GB
		RNN	1.8 minutes	384x V100	DGX-2H	0.6-26	Mixed	WMT16 English-German	V100-SXM3-32GB-H

PyTorch	Transformer	Attention	20.34 minutes	8x V100	DGX-1	0.6-9	Mixed	WMT17 English-German	V100-SXM2-16GB
		Attention	11.04 minutes	16x V100	DGX-2	0.6-18	Mixed	WMT17 English-German	V100-SXM3-32GB
		Attention	9.8 minutes	16x V100	DGX-2H	0.6-20	Mixed	WMT17 English-German	V100-SXM3-32GB-H
		Attention	2.41 minutes	160x V100	DGX-2H	0.6-22	Mixed	WMT17 English-German	V100-SXM3-32GB-H
		Attention	2.05 minutes	480x V100	DGX-1	0.6-15	Mixed	WMT17 English-German	V100-SXM2-16GB
		Attention	1.59 minutes	480x V100	DGX-2H	0.6-28	Mixed	WMT17 English-German	V100-SXM3-32GB-H

TensorFlow	MiniGo	Reinforcement Learning	27.39 minutes	8x V100	DGX-1	0.6-10	Mixed	N/A	V100-SXM2-16GB
		Reinforcement Learning	13.57 minutes	24x V100	DGX-1	0.6-11	Mixed	N/A	V100-SXM2-16GB

Training Image Classification on CNNs

ResNet-50 V1.5 Throughput on V100

DGX-1: 8x NVIDIA V100-SXM2-16GB for MXNet and PyTorch. NVIDIA V100-SXM2-32GB for TensorFlow, E5-2698 v4 2.2 GHz | Batch Size: MXNet = 208, PyTorch = 256 and TensorFlow = 512 | MXNet, PyTorch, TensorFlow = 19.12-py3 | Precision: Mixed | Dataset: ImageNet2012

ResNet-50 V1.5 Throughput on T4

Supermicro SYS-4029GP-TRT T4: 8x NVIDIA T4, Gold 6240 2.6 GHz for MXNet, PyTorch and TensorFlow | Batch Size: MXNet = 208, PyTorch and TensorFlow = 256 | MXNet, PyTorch, TensorFlow = 19.12-py3 | Precision: Mixed | Dataset: ImageNet2012

V100 Training Performance

Framework	Network	Network Type	Throughput	GPU	Server	Container	Precision	Batch Size	Dataset	GPU Version
MXNet	Inception V3	CNN	553 images/sec	1x V100	DGX-1	19.12-py3	Mixed	208	ImageNet2012	V100-SXM2-16GB
		CNN	623.28 images/sec	1x V100	DGX-2H	19.12-py3	Mixed	384	ImageNet2012	V100-SXM3-32GB-H
		CNN	4,261.95 images/sec	8x V100	DGX-1	19.12-py3	Mixed	208	ImageNet2012	V100-SXM2-16GB
		CNN	4,756.47 images/sec	8x V100	DGX-2H	19.12-py3	Mixed	384	ImageNet2012	V100-SXM3-32GB-H
	ResNet-50	CNN	1,382 images/sec	1x V100	DGX-1	19.11-py3	Mixed	208	ImageNet2012	V100-SXM2-16GB
		CNN	1,445 images/sec	1x V100	DGX-2	19.11-py3	Mixed	512	ImageNet2012	V100-SXM3-32GB
		CNN	1,551 images/sec	1x V100	DGX-2H	19.11-py3	Mixed	512	ImageNet2012	V100-SXM3-32GB-H
		CNN	10,358 images/sec	8x V100	DGX-1	19.11-py3	Mixed	192	ImageNet2012	V100-SXM2-16GB
		CNN	10,805 images/sec	8x V100	DGX-2	19.11-py3	Mixed	256	ImageNet2012	V100-SXM3-32GB
		CNN	11,507 images/sec	8x V100	DGX-2H	19.11-py3	Mixed	256	ImageNet2012	V100-SXM3-32GB-H
	ResNet-50 v1.5	CNN	1,468.55 images/sec	1x V100	DGX-1	19.12-py3	Mixed	208	ImageNet2012	V100-SXM2-16GB
		CNN	1,658.31 images/sec	1x V100	DGX-2H	19.12-py3	Mixed	256	ImageNet2012	V100-SXM3-32GB-H
		CNN	10,787.44 images/sec	8x V100	DGX-1	19.12-py3	Mixed	208	ImageNet2012	V100-SXM2-16GB
		CNN	11,486.48 images/sec	8x V100	DGX-2	19.12-py3	Mixed	256	ImageNet2012	V100-SXM3-32GB
		CNN	12,228.38 images/sec	8x V100	DGX-2H	19.12-py3	Mixed	256	ImageNet2012	V100-SXM3-32GB-H
PyTorch	Inception V3	CNN	534.39 images/sec	1x V100	DGX-1	19.12-py3	Mixed	256	ImageNet2012	V100-SXM2-16GB
		CNN	596.54 images/sec	1x V100	DGX-2H	19.12-py3	Mixed	512	ImageNet2012	V100-SXM3-32GB-H
		CNN	4,161.36 images/sec	8x V100	DGX-1	19.12-py3	Mixed	256	ImageNet2012	V100-SXM2-16GB
	Mask R-CNN	CNN	16 images/sec	1x V100	DGX-1	19.11-py3	Mixed	16	COCO 2014	V100-SXM2-32GB
		CNN	16.52 images/sec	1x V100	DGX-2	19.12-py3	Mixed	16	COCO 2014	V100-SXM3-32GB
		CNN	86.89 images/sec	8x V100	DGX-1	19.12-py3	Mixed	4	COCO 2014	V100-SXM2-16GB
	ResNet-50	CNN	905 images/sec	1x V100	DGX-1	19.11-py3	Mixed	256	ImageNet2012	V100-SXM2-16GB
		CNN	926 images/sec	1x V100	DGX-2	19.11-py3	Mixed	512	ImageNet2012	V100-SXM3-32GB
		CNN	1,025 images/sec	1x V100	DGX-2H	19.11-py3	Mixed	512	ImageNet2012	V100-SXM3-32GB-H
		CNN	6,179 images/sec	8x V100	DGX-1	19.11-py3	Mixed	256	ImageNet2012	V100-SXM2-16GB
		CNN	6,595 images/sec	8x V100	DGX-2	19.11-py3	Mixed	512	ImageNet2012	V100-SXM3-32GB
		CNN	7,151 images/sec	8x V100	DGX-2H	19.11-py3	Mixed	512	ImageNet2012	V100-SXM3-32GB-H
	ResNet-50 V1.5	CNN	912.74 images/sec	1x V100	DGX-1	19.12-py3	Mixed	256	ImageNet2012	V100-SXM2-16GB
		CNN	1,007.19 images/sec	1x V100	DGX-2H	19.12-py3	Mixed	256	ImageNet2012	V100-SXM3-32GB-H
		CNN	6,881.12 images/sec	8x V100	DGX-1	19.12-py3	Mixed	256	ImageNet2012	V100-SXM2-16GB
	SSD v1.1	CNN	235.29 images/sec	1x V100	DGX-1	19.12-py3	Mixed	64	COCO 2017	V100-SXM2-16GB
		CNN	262.1 images/sec	1x V100	DGX-2H	19.12-py3	Mixed	64	COCO 2017	V100-SXM3-32GB-H
		CNN	1,830 images/sec	8x V100	DGX-1	19.12-py3	Mixed	64	COCO 2017	V100-SXM2-16GB
	Tacotron2	CNN	19,621 total output mels/sec	1x V100	DGX-1	19.11-py3	Mixed	128	LJSpeech 1.1	V100-SXM2-32GB
		CNN	24,269 total output mels/sec	1x V100	DGX-2H	19.11-py3	Mixed	128	LJSpeech 1.1	V100-SXM3-32GB-H
		CNN	116,696 total output mels/sec	8x V100	DGX-1	19.11-py3	Mixed	128	LJSpeech 1.1	V100-SXM2-32GB
		CNN	150,263 total output mels/sec	8x V100	DGX-2H	19.11-py3	Mixed	128	LJSpeech 1.1	V100-SXM3-32GB-H
	WaveGlow	CNN	77,732 output samples/sec	1x V100	DGX-1	19.09-py3	Mixed	10	LJSpeech 1.1	V100-SXM2-16GB
		CNN	92,280 output samples/sec	1x V100	DGX-2H	19.10-py3	Mixed	10	LJSpeech 1.1	V100-SXM3-32GB-H
		CNN	557,814 output samples/sec	8x V100	DGX-1	19.10-py3	Mixed	10	LJSpeech 1.1	V100-SXM2-16GB
	Jasper	CNN	41 sequences/sec	1x V100	DGX-1	19.12-py3	Mixed	64	LibriSpeech	V100-SXM2-32GB
		CNN	49 sequences/sec	1x V100	DGX-2H	19.12-py3	Mixed	64	LibriSpeech	V100-SXM3-32GB-H
		CNN	305 sequences/sec	8x V100	DGX-1	19.12-py3	Mixed	64	LibriSpeech	V100-SXM2-32GB
TensorFlow	Inception V3	CNN	727.15 images/sec	1x V100	DGX-1	19.12-py3	Mixed	384	ImageNet2012	V100-SXM2-32GB
		CNN	830.94 images/sec	1x V100	DGX-2H	19.12-py3	Mixed	384	ImageNet2012	V100-SXM3-32GB-H
		CNN	5,516.26 images/sec	8x V100	DGX-1	19.12-py3	Mixed	384	ImageNet2012	V100-SXM2-32GB
		CNN	5,682.88 images/sec	8x V100	DGX-2H	19.12-py3	Mixed	384	ImageNet2012	V100-SXM3-32GB-H
	ResNet-50 V1.5	CNN	1064.75 images/sec	1x V100	DGX-1	19.12-py3	Mixed	256	ImageNet2012	V100-SXM2-16GB
		CNN	1,240.33 images/sec	1x V100	DGX-2H	19.12-py3	Mixed	512	ImageNet2012	V100-SXM3-32GB-H
		CNN	8,116.45 images/sec	8x V100	DGX-1	19.12-py3	Mixed	512	ImageNet2012	V100-SXM2-32GB
		CNN	8,445 images/sec	8x V100	DGX-2H	19.12-py3	Mixed	512	ImageNet2012	V100-SXM3-32GB-H
	SSD v1.2	CNN	124.99 images/sec	1x V100	DGX-1	19.12-py3	Mixed	32	COCO 2017	V100-SXM2-16GB
		CNN	140.34 images/sec	1x V100	DGX-2H	19.12-py3	Mixed	32	COCO 2017	V100-SXM3-32GB-H
		CNN	599.5 images/sec	8x V100	DGX-1	19.12-py3	Mixed	32	COCO 2017	V100-SXM2-16GB
		CNN	726.54 images/sec	8x V100	DGX-2H	19.12-py3	Mixed	32	COCO 2017	V100-SXM3-32GB-H
	U-Net Industrial	CNN	100 images/sec	1x V100	DGX-1	19.12-py3	Mixed	16	DAGM2007	V100-SXM2-16GB
		CNN	112 images/sec	1x V100	DGX-2H	19.12-py3	Mixed	16	DAGM2007	V100-SXM3-32GB-H
		CNN	528 images/sec	8x V100	DGX-1	19.12-py3	Mixed	2	DAGM2007	V100-SXM2-16GB
		CNN	570 images/sec	8x V100	DGX-2H	19.12-py3	Mixed	2	DAGM2007	V100-SXM3-32GB-H
PyTorch	GNMT V2	RNN	56,250 total tokens/sec	1x V100	DGX-1	19.12-py3	Mixed	128	WMT16 English-German	V100-SXM2-16GB
		RNN	64,515 total tokens/sec	1x V100	DGX-2H	19.12-py3	Mixed	128	WMT16 English-German	V100-SXM3-32GB-H
		RNN	407,068 total tokens/sec	8x V100	DGX-1	19.12-py3	Mixed	128	WMT16 English-German	V100-SXM2-16GB
		RNN	445,451 total tokens/sec	8x V100	DGX-2H	19.12-py3	Mixed	128	WMT16 English-German	V100-SXM3-32GB-H
TensorFlow	GNMT V2	RNN	25,159 total tokens/sec	1x V100	DGX-1	19.10-py3	Mixed	192	WMT16 English-German	V100-SXM2-32GB
		RNN	29,972 total tokens/sec	1x V100	DGX-2H	19.10-py3	Mixed	192	WMT16 English-German	V100-SXM3-32GB-H
		RNN	163,445 total tokens/sec	8x V100	DGX-1	19.10-py3	Mixed	192	WMT16 English-German	V100-SXM2-32GB
PyTorch	NCF	Recommender	21,395,676 samples/sec	1x V100	DGX-1	19.12-py3	Mixed	1,048,576	MovieLens 20 Million	V100-SXM2-16GB
		Recommender	23,683,277 samples/sec	1x V100	DGX-2H	19.12-py3	Mixed	1,048,576	MovieLens 20 Million	V100-SXM3-32GB-H
		Recommender	99,257,568 samples/sec	8x V100	DGX-1	19.12-py3	Mixed	1,048,576	MovieLens 20 Million	V100-SXM2-16GB
		Recommender	105,607,286 samples/sec	8x V100	DGX-2H	19.12-py3	Mixed	1,048,576	MovieLens 20 Million	V100-SXM3-32GB-H
TensorFlow	NCF	Recommender	26,470,121 samples/sec	1x V100	DGX-1	19.12-py3	Mixed	1,048,576	MovieLens 20 Million	V100-SXM2-16GB
		Recommender	28,113,739 samples/sec	1x V100	DGX-2H	19.12-py3	Mixed	1,048,576	MovieLens 20 Million	V100-SXM3-32GB-H
		Recommender	76,749,507 samples/sec	8x V100	DGX-1	19.12-py3	Mixed	1,048,576	MovieLens 20 Million	V100-SXM2-16GB
PyTorch	BERT-LARGE	Attention	44.8 items/sec	1x V100	DGX-1	19.12-py3	Mixed	10	SQuaD v1.1	V100-SXM2-32GB
		Attention	52.81 items/sec	1x V100	DGX-2H	19.12-py3	Mixed	10	SQuaD v1.1	V100-SXM3-32GB-H
		Attention	326.06 items/sec	8xV100	DGX-1	19.12-py3	Mixed	10	SQuaD v1.1	V100-SXM2-32GB
		Attention	369.83 items/sec	8xV100	DGX-2H	19.12-py3	Mixed	10	SQuaD v1.1	V100-SXM3-32GB-H
TensorFlow	BERT-LARGE	Attention	38.04 sentences/sec	1x V100	DGX-1	19.12-py3	Mixed	10	SQuaD v1.1	V100-SXM2-32GB
		Attention	44.32 sentences/sec	1x V100	DGX-2H	19.12-py3	Mixed	10	SQuaD v1.1	V100-SXM3-32GB-H
		Attention	195.44 sentences/sec	8x V100	DGX-1	19.12-py3	Mixed	10	SQuaD v1.1	V100-SXM2-32GB
		Attention	221.14 sentences/sec	8x V100	DGX-2H	19.12-py3	Mixed	10	SQuaD v1.1	V100-SXM3-32GB-H

T4 Training Performance

Framework	Network	Network Type	Throughput	GPU	Server	Container	Precision	Batch Size	Dataset	GPU Version
MXNet	Inception V3	CNN	180.13 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.12-py3	Mixed	208	ImageNet2012	NVIDIA T4
		CNN	1,420.91 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.12-py3	Mixed	208	ImageNet2012	NVIDIA T4
	ResNet-50 v1.5	CNN	486.59 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.12-py3	Mixed	208	ImageNet2012	NVIDIA T4
		CNN	3,758.94 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.12-py3	Mixed	208	ImageNet2012	NVIDIA T4
PyTorch	Inception V3	CNN	172.41 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.12-py3	Mixed	256	ImageNet2012	NVIDIA T4
		CNN	1,348.85 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.12-py3	Mixed	256	ImageNet2012	NVIDIA T4
	Mask R-CNN	CNN	6.28 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.12-py3	Mixed	4	COCO 2014	NVIDIA T4
		CNN	39.04 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.12-py3	Mixed	4	COCO 2014	NVIDIA T4
	ResNet-50 V1.5	CNN	280.21 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.12-py3	Mixed	256	ImageNet2012	NVIDIA T4
		CNN	2,245.7 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.12-py3	Mixed	256	ImageNet2012	NVIDIA T4
	SSD v1.1	CNN	76.89 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.12-py3	Mixed	64	COCO 2017	NVIDIA T4
		CNN	637.25 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.12-py3	Mixed	64	COCO 2017	NVIDIA T4
	Tacotron2	CNN	13,451 total output mels/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.12-py3	Mixed	104	LJSpeech 1.1	NVIDIA T4
		CNN	103,679 total output mels/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.09-py3	Mixed	128	LJSpeech 1.1	NVIDIA T4
	WaveGlow	CNN	33,719 output samples/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.12-py3	Mixed	10	LJSpeech 1.1	NVIDIA T4
		CNN	247,439 output samples/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.10-py3	Mixed	10	LJSpeech 1.1	NVIDIA T4
	Jasper	CNN	13 sequences/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.12-py3	Mixed	32	LibriSpeech	NVIDIA T4
		CNN	93 sequences/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.12-py3	Mixed	32	LibriSpeech	NVIDIA T4
TensorFlow	Inception V3	CNN	235.05 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.12-py3	Mixed	192	ImageNet2012	NVIDIA T4
		CNN	1,767.35 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.12-py3	Mixed	192	ImageNet2012	NVIDIA T4
	ResNet-50 V1.5	CNN	355.13 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.12-py3	Mixed	256	ImageNet2012	NVIDIA T4
		CNN	2,684.89 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.12-py3	Mixed	256	ImageNet2012	NVIDIA T4
	SSD v1.2	CNN	62.52 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.12-py3	Mixed	32	COCO 2017	NVIDIA T4
		CNN	286 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.11-py3	Mixed	32	COCO 2017	NVIDIA T4
	U-Net Industrial	CNN	31 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.12-py3	Mixed	16	DAGM2007	NVIDIA T4
		CNN	210 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.12-py3	Mixed	2	DAGM2007	NVIDIA T4
PyTorch	GNMT V2	RNN	22,204 total tokens/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.12-py3	Mixed	128	WMT16 English-German	NVIDIA T4
		RNN	135,243 total tokens/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.12-py3	Mixed	128	WMT16 English-German	NVIDIA T4
TensorFlow	GNMT V2	RNN	11,771 total tokens/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.09-py3	Mixed	192	WMT16 English-German	NVIDIA T4
		RNN	54,389 total tokens/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.12-py3	Mixed	128	WMT16 English-German	NVIDIA T4
PyTorch	NCF	Recommender	7,228,830 samples/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.12-py3	Mixed	1,048,576	MovieLens 20 Million	NVIDIA T4
		Recommender	24,394,706 samples/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.12-py3	Mixed	1,048,576	MovieLens 20 Million	NVIDIA T4
TensorFlow	NCF	Recommender	10,375,820 samples/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.12-py3	Mixed	1,048,576	MovieLens 20 Million	NVIDIA T4
		Recommender	18,957,309 samples/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.12-py3	Mixed	1,048,576	MovieLens 20 Million	NVIDIA T4
TensorFlow	BERT	Attention	9.98 sentences/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.12-py3	Mixed	3	SQuaD v1.1	NVIDIA T4
		Attention	45.16 sentences/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.12-py3	Mixed	3	SQuaD v1.1	NVIDIA T4

NVIDIA® TensorRT™ running on NVIDIA GPUs enable the most efficient deep learning inference performance across multiple application areas and models. This versatility provides wide latitude to data scientists to create the optimal low-latency solution. Visit NVIDIA GPU Cloud (NGC) to download any of these containers.

NVIDIA V100 Tensor Cores GPUs leverage mixed-precision to combine high throughput with low latencies across every type of neural network. NVIDIA P4 is an inference GPU, designed for optimal power consumption and latency, for ultra-efficient scale-out servers. Read the inference whitepaper to learn more about NVIDIA’s inference platform.

Measuring the inference performance involves balancing a lot of variables. PLASTER is an acronym that describes the key elements for measuring deep learning performance. Each letter identifies a factor (Programmability, Latency, Accuracy, Size of Model, Throughput, Energy Efficiency, Rate of Learning) that must be considered to arrive at the right set of tradeoffs and to produce a successful deep learning implementation. Refer to NVIDIA’s PLASTER whitepaper for more details.

NVIDIA landed top performance spots on all five MLPerf Inference 0.5 benchmarks with the best per-accelerator performance among commercially available products.

NVIDIA Performance on MLPerf Inference v0.5 Benchmarks (Server)

NVIDIA Performance on MLPerf Inference v0.5 Benchmarks (Offline)

MLPerf v0.5 Inference results for data center server form factors and offline and server scenarios retrieved from www.mlperf.org on Nov. 6, 2019, from entries Inf-0.5-15,Inf-0. 5-16, Inf-0.5-19, Inf-0.5-21. Inf-0.5-22, Inf-0.5-23, Inf-0.5-25, Inf-0.5-26, Inf-0.5-27. Per-processor performance is calculated by dividing the primary metric of total performance by number of accelerators reported.

MLPerf name and logo are trademarks.

NVIDIA Performance on MLPerf Inference v0.5 Benchmarks (ResNet-50 V1.5 Offline Scenario)

MLPerf v0.5 Inference results for data center server form factors and offline scenario retrieved from www.mlperf.org on Nov. 6, 2019 (Closed Inf-0.5-25 and Inf-0.5-27 for INT8, Open Inf-0.5-460 and Inf-0.5-462 for INT4). Per-processor performance is calculated by dividing the primary metric of total performance by number of accelerators reported. MLPerf name and logo are trademarks.

MLPerf Inference Performance

NVIDIA Turing 70W

Network	Network Type	Batch Size	Throughput	Efficiency	Latency	GPU	Server	Container	Precision	Dataset	GPU Version
MobileNet v1	Server	-	16,884 queries/sec	-	-	1x T4	Supermicro 4029GP-TRT-OTO-28	-	-	ImageNet [224x224]	NVIDIA T4
MobileNet v1	Offline	-	17,726 inputs/sec	-	-	1x T4	Supermicro 4029GP-TRT-OTO-28	-	-	ImageNet [224x224]	NVIDIA T4
ResNet-50 v1.5	Server	-	5,193 queries/sec	-	-	1x T4	Supermicro 4029GP-TRT-OTO-28	-	-	ImageNet [224x224]	NVIDIA T4
ResNet-50 v1.5	Offline	-	5,622 inputs/sec	-	-	1x T4	Supermicro 4029GP-TRT-OTO-28	-	-	ImageNet [224x224]	NVIDIA T4
SSD MobileNet v1	Server	-	7,078 queries/sec	-	-	1x T4	Supermicro 4029GP-TRT-OTO-28	-	-	COCO [300x300]	NVIDIA T4
SSD MobileNet v1	Offline	-	7,609 inputs/sec	-	-	1x T4	Supermicro 4029GP-TRT-OTO-28	-	-	COCO [300x300]	NVIDIA T4
SSD ResNet-34	Server	-	126 queries/sec	-	-	1x T4	Supermicro 4029GP-TRT-OTO-28	-	-	COCO [1200x1200]	NVIDIA T4
SSD ResNet-34	Offline	-	137 inputs/sec	-	-	1x T4	Supermicro 4029GP-TRT-OTO-28	-	-	COCO [1200x1200]	NVIDIA T4
GNMT	Server	-	198 queries/sec	-	-	1x T4	Supermicro 4029GP-TRT-OTO-28	-	-	WMT16	NVIDIA T4
GNMT	Offline	-	354 inputs/sec	-	-	1x T4	Supermicro 4029GP-TRT-OTO-28	-	-	WMT16	NVIDIA T4

Supermicro 4029GP-TRT-OTO-28: 1x NVIDIA T4, 2x Intel(R) Xeon(R) Platinum 8280 CPU @ 2.7GHz | TensorRT 6.0 | CUDA 10.1 | cuDNN 7.6.3

NVIDIA Turing 280W

Network	Network Type	Batch Size	Throughput	Efficiency	Latency	GPU	Server	Container	Precision	Dataset	GPU Version
MobileNet v1	Server	-	49,775 queries/sec	-	-	1x TitanRTX	SCAN 3XS DBP T496X2 Fluid	-	-	ImageNet [224x224]	TitanRTX
MobileNet v1	Offline	-	55,597 inputs/sec	-	-	1x TitanRTX	SCAN 3XS DBP T496X2 Fluid	-	-	ImageNet [224x224]	TitanRTX
ResNet-50 v1.5	Server	-	15,008 queries/sec	-	-	1x TitanRTX	SCAN 3XS DBP T496X2 Fluid	-	-	ImageNet [224x224]	TitanRTX
ResNet-50 v1.5	Offline	-	16,563 inputs/sec	-	-	1x TitanRTX	SCAN 3XS DBP T496X2 Fluid	-	-	ImageNet [224x224]	TitanRTX
SSD MobileNet v1	Server	-	20,503 queries/sec	-	-	1x TitanRTX	SCAN 3XS DBP T496X2 Fluid	-	-	COCO [300x300]	TitanRTX
SSD MobileNet v1	Offline	-	22,945 inputs/sec	-	-	1x TitanRTX	SCAN 3XS DBP T496X2 Fluid	-	-	COCO [300x300]	TitanRTX
SSD ResNet-34	Server	-	388 queries/sec	-	-	1x TitanRTX	SCAN 3XS DBP T496X2 Fluid	-	-	COCO [1200x1200]	TitanRTX
SSD ResNet-34	Offline	-	415 inputs/sec	-	-	1x TitanRTX	SCAN 3XS DBP T496X2 Fluid	-	-	COCO [1200x1200]	TitanRTX
GNMT	Server	-	654 queries/sec	-	-	1x TitanRTX	SCAN 3XS DBP T496X2 Fluid	-	-	WMT16	TitanRTX
GNMT	Offline	-	1,061 inputs/sec	-	-	1x TitanRTX	SCAN 3XS DBP T496X2 Fluid	-	-	WMT16	TitanRTX

SCAN 3XS DBP T496X2 Fluid: 1x TitanRTX, 2x Intel(R) Xeon(R) 8268 | TensorRT 6.0 | CUDA 10.1 | cuDNN 7.6.3

Inference Image Classification on CNNs with TensorRT

ResNet-50 v1.5 Throughput

ResNet-50 v1.5 Latency

ResNet-50 v1.5 Power Efficiency

Inference Performance

V100 Inference Performance

Network	Network Type	Batch Size	Throughput	Efficiency	Latency	GPU	Server	Container	Precision	Dataset	Framework	GPU Version
GoogleNet	CNN	1	1,539 images/sec	14 images/sec/watt	0.65	1x V100	DGX-1	19.12-py3	INT8	Synthetic	TensorRT 6.0	V100-SXM2-16GB
	CNN	2	2,080 images/sec	18 images/sec/watt	0.96	1x V100	DGX-1	19.12-py3	INT8	Synthetic	TensorRT 6.0	V100-SXM2-16GB
	CNN	8	5,155 images/sec	36 images/sec/watt	1.6	1x V100	DGX-1	19.12-py3	INT8	Synthetic	TensorRT 6.0	V100-SXM2-16GB
	CNN	83	12,174 images/sec	45 images/sec/watt	6.8	1x V100	DGX-1	19.12-py3	Mixed	Synthetic	TensorRT 6.0	V100-SXM2-16GB
	CNN	128	12,446 images/sec	45 images/sec/watt	10	1x V100	DGX-1	19.12-py3	Mixed	Synthetic	TensorRT 6.0	V100-SXM2-16GB
MobileNet V1	CNN	1	4,695 images/sec	35 images/sec/watt	0.21	1x V100	DGX-1	19.12-py3	INT8	Synthetic	TensorRT 6.0	V100-SXM2-16GB
	CNN	2	6,493 images/sec	45 images/sec/watt	0.31	1x V100	DGX-1	19.12-py3	INT8	Synthetic	TensorRT 6.0	V100-SXM2-16GB
	CNN	8	15,621 images/sec	73 images/sec/watt	0.51	1x V100	DGX-2	19.12-py3	INT8	Synthetic	TensorRT 6.0	V100-SXM3-32GB
	CNN	220	30,224 images/sec	104 images/sec/watt	6.9	1x V100	DGX-1	19.12-py3	INT8	Synthetic	TensorRT 6.0	V100-SXM2-16GB
	CNN	128	29,481 images/sec	101 images/sec/watt	4.3	1x V100	DGX-1	19.12-py3	INT8	Synthetic	TensorRT 6.0	V100-SXM2-16GB
ResNet-50	CNN	1	1,147 images/sec	8.4 images/sec/watt	0.87	1x V100	DGX-1	19.12-py3	INT8	Synthetic	TensorRT 6.0	V100-SXM2-16GB
	CNN	2	1,529 images/sec	9.5 images/sec/watt	1.3	1x V100	DGX-1	19.12-py3	INT8	Synthetic	TensorRT 6.0	V100-SXM2-16GB
	CNN	8	3,394 images/sec	19 images/sec/watt	2.4	1x V100	DGX-1	19.12-py3	Mixed	Synthetic	TensorRT 6.0	V100-SXM2-16GB
	CNN	52	7,617 images/sec	27 images/sec/watt	6.8	1x V100	DGX-1	19.12-py3	Mixed	Synthetic	TensorRT 6.0	V100-SXM2-16GB
	CNN	128	7,744 images/sec	27 images/sec/watt	17	1x V100	DGX-1	19.12-py3	Mixed	Synthetic	TensorRT 6.0	V100-SXM2-16GB
	CNN	128	7,924 images/sec	24 images/sec/watt	16	1x V100	DGX-2	19.12-py3	Mixed	Synthetic	TensorRT 6.0	V100-SXM3-32GB
ResNet-50v1.5	CNN	1	1,011 images/sec	6.1 images/sec/watt	0.99	1x V100	DGX-2	19.12-py3	INT8	Synthetic	TensorRT 6.0	V100-SXM3-32GB
	CNN	2	1,404 images/sec	9.4 images/sec/watt	1.4	1x V100	DGX-1	19.12-py3	INT8	Synthetic	TensorRT 6.0	V100-SXM2-16GB
	CNN	8	3,269 images/sec	18 images/sec/watt	2.5	1x V100	DGX-1	19.12-py3	Mixed	Synthetic	TensorRT 6.0	V100-SXM2-16GB
	CNN	51	7,379 images/sec	23 images/sec/watt	6.9	1x V100	DGX-2	19.12-py3	Mixed	Synthetic	TensorRT 6.0	V100-SXM3-32GB
	CNN	128	7,191 images/sec	25 images/sec/watt	18	1x V100	DGX-1	19.12-py3	Mixed	Synthetic	TensorRT 6.0	V100-SXM2-16GB
	CNN	128	7,542 images/sec	23 images/sec/watt	17	1x V100	DGX-2	19.12-py3	Mixed	Synthetic	TensorRT 6.0	V100-SXM3-32GB
VGG16	CNN	1	868 images/sec	3.6 images/sec/watt	1.2	1x V100	DGX-2	19.12-py3	INT8	Synthetic	TensorRT 6.0	V100-SXM3-32GB
	CNN	2	960 images/sec	4.7 images/sec/watt	2.1	1x V100	DGX-1	19.12-py3	Mixed	Synthetic	TensorRT 6.0	V100-SXM2-16GB
	CNN	8	1,923 images/sec	6.8 images/sec/watt	4.2	1x V100	DGX-2	19.12-py3	Mixed	Synthetic	TensorRT 6.0	V100-SXM3-32GB
	CNN	16	2,379 images/sec	7.6 images/sec/watt	6.7	1x V100	DGX-2	19.12-py3	Mixed	Synthetic	TensorRT 6.0	V100-SXM3-32GB
	CNN	128	2,797 images/sec	9.6 images/sec/watt	46	1x V100	DGX-1	19.12-py3	Mixed	Synthetic	TensorRT 6.0	V100-SXM2-32GB
NMT	RNN	1	4,013 total tokens/sec	-	13	1x V100	DGX-1	-	Mixed	wmt16-English-German	TensorRT 5.1	V100-SXM2-32GB
	RNN	2	6,290 total tokens/sec	-	16	1x V100	DGX-1	-	Mixed	wmt16-English-German	TensorRT 5.1	V100-SXM2-32GB
	RNN	64	56,531 total tokens/sec	-	58	1x V100	DGX-1	-	Mixed	wmt16-English-German	TensorRT 5.1	V100-SXM2-32GB
	RNN	128	73,375 total tokens/sec	-	89	1x V100	DGX-1	-	Mixed	wmt16-English-German	TensorRT 5.1	V100-SXM2-32GB
NCF	Recommender	1	26,418 samples/sec	335 samples/sec/watt	0.04	1x V100	DGX-2	19.12-py3	FP32	Synthetic	TensorRT 6.0	V100-SXM3-32GB
	Recommender	64	1,630,561 samples/sec	17,549 samples/sec/watt	0.04	1x V100	DGX-2	19.12-py3	FP32	Synthetic	TensorRT 6.0	V100-SXM3-32GB
	Recommender	25,000	91,055,121 samples/sec	609,642 samples/sec/watt	0.27	1x V100	DGX-1	19.12-py3	INT8	Synthetic	TensorRT 6.0	V100-SXM2-16GB
	Recommender	100,000	121,238,130 samples/sec	582,641 samples/sec/watt	0.82	1x V100	DGX-2	19.12-py3	INT8	Synthetic	TensorRT 6.0	V100-SXM3-32GB
BERT-BASE	Attention	1	557 sentences/sec	10.3 sentences/sec/watt	1.8	1x V100	Supermicro SYS-4029GP-TRT	-	Mixed	SQuAD v1.1	TensorRT 6.0	V100-PCIE-16GB
	Attention	2	978 sentences/sec	18.8 sentences/sec/watt	2	1x V100	Supermicro SYS-4029GP-TRT	-	Mixed	SQuAD v1.1	TensorRT 6.0	V100-PCIE-16GB
	Attention	8	1,847 sentences/sec	34.1 sentences/sec/watt	4.3	1x V100	Supermicro SYS-4029GP-TRT	-	Mixed	SQuAD v1.1	TensorRT 6.0	V100-PCIE-16GB
	Attention	24	2,419 sentences/sec	43.7 sentences/sec/watt	9.9	1x V100	Supermicro SYS-4029GP-TRT	-	Mixed	SQuAD v1.1	TensorRT 6.0	V100-PCIE-16GB
	Attention	128	2,645 sentences/sec	46 sentences/sec/watt	48.4	1x V100	Supermicro SYS-4029GP-TRT	-	Mixed	SQuAD v1.1	TensorRT 6.0	V100-PCIE-16GB
BERT-LARGE	Attention	1	239 sentences/sec	4.3 sentences/sec/watt	4.2	1x V100	Supermicro SYS-4029GP-TRT	-	Mixed	SQuAD v1.1	TensorRT 6.0	V100-PCIE-16GB
	Attention	2	407 sentences/sec	7.5 sentences/sec/watt	4.9	1x V100	Supermicro SYS-4029GP-TRT	-	Mixed	SQuAD v1.1	TensorRT 6.0	V100-PCIE-16GB
	Attention	4	562 sentences/sec	10.6 sentences/sec/watt	7.1	1x V101	Supermicro SYS-4029GP-TRT	-	Mixed	SQuAD v1.1	TensorRT 6.0	V100-PCIE-16GB
	Attention	8	636 sentences/sec	11.8 sentences/sec/watt	12.6	1x V100	Supermicro SYS-4029GP-TRT	-	Mixed	SQuAD v1.1	TensorRT 6.0	V100-PCIE-16GB
	Attention	128	823 sentences/sec	13.6 sentences/sec/watt	155.5	1x V100	Supermicro SYS-4029GP-TRT	-	Mixed	SQuAD v1.1	TensorRT 6.0	V100-PCIE-16GB

NGC: TensorRT Container | Download TensorRT 6
Sequence length=128 for BERT-BASE and BERT-LARGE | Efficiency based on board power
Containers with a hyphen indicates a pre-release container

T4 Inference Performance

Network	Network Type	Batch Size	Throughput	Efficiency	Latency	GPU	Server	Container	Precision	Dataset	Framework	GPU Version
GoogleNet	CNN	1	1,694 images/sec	28 images/sec/watt	0.59	1x T4	Supermicro SYS-4029GP-TRT T4	19.12-py3	INT8	Synthetic	TensorRT 6.0	NVIDIA T4
	CNN	2	2,473 images/sec	40 images/sec/watt	0.81	1x T4	Supermicro SYS-4029GP-TRT T4	19.12-py3	INT8	Synthetic	TensorRT 6.0	NVIDIA T4
	CNN	8	6,282 images/sec	90 images/sec/watt	1.3	1x T4	Supermicro SYS-4029GP-TRT T4	19.12-py3	INT8	Synthetic	TensorRT 6.0	NVIDIA T4
	CNN	59	8,659 images/sec	124 images/sec/watt	6.8	1x T4	Supermicro SYS-4029GP-TRT T4	19.12-py3	INT8	Synthetic	TensorRT 6.0	NVIDIA T4
	CNN	128	8,839 images/sec	127 images/sec/watt	14	1x T4	Supermicro SYS-4029GP-TRT T4	19.12-py3	INT8	Synthetic	TensorRT 6.0	NVIDIA T4
MobileNet V1	CNN	1	4,542 images/sec	84 images/sec/watt	0.22	1x T4	Supermicro SYS-4029GP-TRT T4	19.10-py3	INT8	Synthetic	TensorRT 6.0	NVIDIA T4
	CNN	2	7,539 images/sec	123 images/sec/watt	0.27	1x T4	Supermicro SYS-4029GP-TRT T4	19.12-py3	INT8	Synthetic	TensorRT 6.0	NVIDIA T4
	CNN	8	14,360 images/sec	206 images/sec/watt	0.56	1x T4	Supermicro SYS-4029GP-TRT T4	19.12-py3	INT8	Synthetic	TensorRT 6.0	NVIDIA T4
	CNN	120	17,770 images/sec	254 images/sec/watt	6.8	1x T4	Supermicro SYS-4029GP-TRT T4	19.12-py3	INT8	Synthetic	TensorRT 6.0	NVIDIA T4
	CNN	128	17,697 images/sec	254 images/sec/watt	7.2	1x T4	Supermicro SYS-4029GP-TRT T4	19.12-py3	INT8	Synthetic	TensorRT 6.0	NVIDIA T4
ResNet-50	CNN	1	1,152 images/sec	16 images/sec/watt	0.87	1x T4	Supermicro SYS-4029GP-TRT T4	19.12-py3	INT8	Synthetic	TensorRT 6.0	NVIDIA T4
	CNN	2	1,685 images/sec	24 images/sec/watt	1.2	1x T4	Supermicro SYS-4029GP-TRT T4	19.12-py3	INT8	Synthetic	TensorRT 6.0	NVIDIA T4
	CNN	8	4,049 images/sec	58 images/sec/watt	2	1x T4	Supermicro SYS-4029GP-TRT T4	19.12-py3	INT8	Synthetic	TensorRT 6.0	NVIDIA T4
	CNN	35	5,240 images/sec	75 images/sec/watt	6.7	1x T4	Supermicro SYS-4029GP-TRT T4	19.12-py3	INT8	Synthetic	TensorRT 6.0	NVIDIA T4
	CNN	128	5,758 images/sec	82 images/sec/watt	22	1x T4	Supermicro SYS-4029GP-TRT T4	19.12-py3	INT8	Synthetic	TensorRT 6.0	NVIDIA T4
ResNet-50v1.5	CNN	1	1,094 images/sec	16 images/sec/watt	0.91	1x T4	Supermicro SYS-4029GP-TRT T4	19.12-py3	INT8	Synthetic	TensorRT 6.0	NVIDIA T4
	CNN	2	1,714 images/sec	25 images/sec/watt	1.2	1x T4	Supermicro SYS-4029GP-TRT T4	19.12-py3	INT8	Synthetic	TensorRT 6.0	NVIDIA T4
	CNN	8	3,869 images/sec	56 images/sec/watt	2	1x T4	Supermicro SYS-4029GP-TRT T4	19.12-py3	INT8	Synthetic	TensorRT 6.0	NVIDIA T4
	CNN	34	5,040 images/sec	72 images/sec/watt	6.8	1x T4	Supermicro SYS-4029GP-TRT T4	19.12-py3	INT8	Synthetic	TensorRT 6.0	NVIDIA T4
	CNN	128	5,259 images/sec	75 images/sec/watt	24	1x T4	Supermicro SYS-4029GP-TRT T4	19.12-py3	INT8	Synthetic	TensorRT 6.0	NVIDIA T4
VGG16	CNN	1	799 images/sec	11 images/sec/watt	1.3	1x T4	Supermicro SYS-4029GP-TRT T4	19.12-py3	INT8	Synthetic	TensorRT 6.0	NVIDIA T4
	CNN	2	1,093 images/sec	16 images/sec/watt	1.8	1x T4	Supermicro SYS-4029GP-TRT T4	19.12-py3	INT8	Synthetic	TensorRT 6.0	NVIDIA T4
	CNN	8	1,628 images/sec	23 images/sec/watt	4.9	1x T4	Supermicro SYS-4029GP-TRT T4	19.12-py3	INT8	Synthetic	TensorRT 6.0	NVIDIA T4
	CNN	11	1,723 images/sec	25 images/sec/watt	6.4	1x T4	Supermicro SYS-4029GP-TRT T4	19.12-py3	INT8	Synthetic	TensorRT 6.0	NVIDIA T4
	CNN	128	1,885 images/sec	27 images/sec/watt	68	1x T4	Supermicro SYS-4029GP-TRT T4	19.12-py3	INT8	Synthetic	TensorRT 6.0	NVIDIA T4
NCF	Recommender	1	13,169 samples/sec	345 samples/sec/watt	0.08	1x T4	Supermicro SYS-4029GP-TRT T4	19.12-py3	INT8	Synthetic	TensorRT 6.0	NVIDIA T4
	Recommender	64	1,026,669 samples/sec	26,713 samples/sec/watt	0.06	1x T4	Supermicro SYS-4029GP-TRT T4	19.12-py3	INT8	Synthetic	TensorRT 6.0	NVIDIA T4
	Recommender	25,000	51,189,454 samples/sec	733,884 samples/sec/watt	0.49	1x T4	Supermicro SYS-4029GP-TRT T4	19.12-py3	INT8	Synthetic	TensorRT 6.0	NVIDIA T4
	Recommender	100,000	54,236,289 samples/sec	776,160 samples/sec/watt	1.8	1x T4	Supermicro SYS-4029GP-TRT T4	19.12-py3	INT8	Synthetic	TensorRT 6.0	NVIDIA T4
BERT-BASE	Attention	1	484 sentences/sec	11 sentences/sec/watt	2.07	1x T4	Supermicro SYS-4029GP-TRT T4	-	Mixed	SQuAD v1.1	TensorRT 6.0	NVIDIA T4
	Attention	2	754 sentences/sec	17 sentences/sec/watt	2.65	1x T4	Supermicro SYS-4029GP-TRT T4	-	Mixed	SQuAD v1.1	TensorRT 6.0	NVIDIA T4
	Attention	8	827 sentences/sec	20 sentences/sec/watt	9.67	1x T4	Supermicro SYS-4029GP-TRT T4	-	Mixed	SQuAD v1.1	TensorRT 6.0	NVIDIA T4
	Attention	128	800 sentences/sec	16 sentences/sec/watt	160.02	1x T4	Supermicro SYS-4029GP-TRT T4	-	Mixed	SQuAD v1.1	TensorRT 6.0	NVIDIA T4
BERT-LARGE	Attention	1	171 sentences/sec	4 sentences/sec/watt	5.84	1x T4	Supermicro SYS-4029GP-TRT T4	-	Mixed	SQuAD v1.1	TensorRT 6.0	NVIDIA T4
	Attention	2	168 sentences/sec	4 sentences/sec/watt	11.88	1x T4	Supermicro SYS-4029GP-TRT T4	-	Mixed	SQuAD v1.1	TensorRT 6.0	NVIDIA T4
	Attention	8	244 sentences/sec	6 sentences/sec/watt	32.74	1x T4	Supermicro SYS-4029GP-TRT T4	-	Mixed	SQuAD v1.1	TensorRT 6.0	NVIDIA T4
	Attention	128	254 sentences/sec	5 sentences/sec/watt	504.33	1x T4	Supermicro SYS-4029GP-TRT T4	-	Mixed	SQuAD v1.1	TensorRT 6.0	NVIDIA T4

NGC: TensorRT Container | Download TensorRT 6
Sequence length=128 for BERT-BASE and BERT-LARGE | Efficiency based on board power
Containers with a hyphen indicates a pre-release container

Last updated: Feb 9th, 2020

Search form

NVIDIA Data Center Deep Learning Product Performance

Training

Inference

NVIDIA Performance on MLPerf 0.6 AI Benchmarks

Training Performance

NVIDIA Performance on MLPerf 0.6 AI Benchmarks

Training Image Classification on CNNs

V100 Training Performance

T4 Training Performance

NVIDIA Performance on MLPerf Inference v0.5 Benchmarks (Server)

NVIDIA Performance on MLPerf Inference v0.5 Benchmarks (Offline)

NVIDIA Performance on MLPerf Inference v0.5 Benchmarks (ResNet-50 V1.5 Offline Scenario)

MLPerf Inference Performance

NVIDIA Turing 70W

NVIDIA Turing 280W

Inference Image Classification on CNNs with TensorRT

Inference Performance

V100 Inference Performance

T4 Inference Performance