NVIDIA Tesla Deep Learning Product Performance

For HPC performance, please go here.

Deep Learning Training

NVIDIA’s complete solution stack, from GPUs to libraries, and containers on NVIDIA GPU Cloud (NGC), allows data scientists to quickly get up and running with deep learning. NVIDIA® Tesla® V100 Tensor Core GPUs leverage mixed precision to accelerate deep learning training throughputs across every framework and every type of neural network. NVIDIA captured all the top spots on 6 benchmarks submitted to MLPerf, the AI’s first industry-wide benchmark, a testament to our GPU-accelerated platform approach.

NVIDIA Performance on MLPerf AI Benchmarks

ResNet-50 Time to Solution on V100

Training Image Classification on CNNs

ResNet-50 V1.5 Throughput on NVIDIA Tesla V100

DGX-1: 8x Tesla V100 32GB, E5-2698 v4 2.2 GHz (PyTorch, TensorFlow), DGX-2: 8x Tesla V100 32GB, Platinum 8168 2.7 GHz (MXNet) | Batch Size = 256 for PyTorch, 128 for all others | 19.05-py3 | Precision: Mixed | Dataset: ImageNet2012

ResNet-50 V1.5 Throughput on NVIDIA Tesla T4

Supermicro SYS-4029GP-TRT T4: 8x Tesla T4 16GB, Gold 6140 2.3 GHz | Batch Size = 208 for MXNet, 256 for PyTorch, 128 for TensorFlow | 19.05-py3 | Precision: Mixed | Dataset: ImageNet2012

Training Performance

NVIDIA Performance on MLPerf AI Benchmarks

FrameWork	Network	Network Type	Time to Solution	GPU	Server	Container	Precision	BatchSize	DataSet	GPU Version
MXNet	ResNet-50 v1.5	CNN	135 minutes	8x V100	DGX-1V	18.11-py3	Mixed	208	ImageNet2012	V100-SXM2-16GB
		CNN	73.9 minutes	16x V100	DGX-2	18.11-py3	Mixed	256	ImageNet2012	V100-SXM3-32GB
		CNN	70 minutes	16x V100	DGX-2H	18.11-py3	Mixed	256	ImageNet2012	V100-SXM3-32GB
		CNN	6.3 minutes	640x V100	DGX-1V Saturn	18.11-py3	Mixed	26	ImageNet2012	V100-SXM2-16GB
		CNN	7.4 minutes	512x V100	DGX-2H	18.11-py3	Mixed	32	ImageNet2012	V100-SXM3-32GB

PyTorch	SSD	CNN	27 minutes	8x V100	DGX-1V	18.11-py3	Mixed	152	COCO2017	V100-SXM2-16GB
		CNN	15.9 minutes	16x V100	DGX-2	18.11-py3	Mixed	128	COCO2017	V100-SXM3-32GB
		CNN	14.1 minutes	16x V100	DGX-2H	18.11-py3	Mixed	128	COCO2017	V100-SXM3-32GB
		CNN	6.5 minutes	64x V100	DGX-1V Saturn	18.11-py3	Mixed	32	COCO2017	V100-SXM2-16GB
		CNN	5.6 minutes	64x V100	DGX-2H	18.11-py3	Mixed	32	COCO2017	V100-SXM3-32GB
	Mask R-CNN	CNN	323 minutes	8x V100	DGX-1V	18.11-py3	Mixed	4	COCO2014	V100-SXM2-16GB
		CNN	176.3 minutes	16x V100	DGX-2	18.11-py3	Mixed	4	COCO2014	V100-SXM3-32GB
		CNN	166.9 minutes	16x V100	DGX-2H	18.11-py3	Mixed	4	COCO2014	V100-SXM3-32GB
		CNN	81 minutes	64x V100	DGX-1V Saturn	18.11-py3	Mixed	2	COCO2014	V100-SXM2-16GB
		CNN	72.1 minutes	64x V100	DGX-2H	18.11-py3	Mixed	2	COCO2014	V100-SXM3-32GB
	NCF	CNN	0.47 minutes	8x V100	DGX-1V	18.11-py3	Mixed	131072	MovieLens 20 Million	V100-SXM2-16GB
		CNN	0.4 minutes	16x V100	DGX-2	18.11-py3	Mixed	65536	MovieLens 20 Million	V100-SXM3-32GB
		CNN	0.4 minutes	16x V100	DGX-2H	18.11-py3	Mixed	65536	MovieLens 20 Million	V100-SXM3-32GB

PyTorch	GNMT	RNN	18 minutes	8x V100	DGX-1V	18.11-py3	Mixed	128	WMT16 English-German	V100-SXM2-16GB
		RNN	10.5 minutes	16x V100	DGX-2	18.11-py3	Mixed	64	WMT16 English-German	V100-SXM3-32GB
		RNN	9.8 minutes	16x V100	DGX-2H	18.11-py3	Mixed	64	WMT16 English-German	V100-SXM3-32GB
		RNN	2.8 minutes	256x V100	DGX-1V Saturn	18.11-py3	Mixed	32	WMT16 English-German	V100-SXM2-16GB
		RNN	2.7 minutes	256x V100	DGX-2H	18.11-py3	Mixed	32	WMT16 English-German	V100-SXM3-32GB

PyTorch	Transformer	Attention	33 minutes	8x V100	DGX-1V	18.11-py3	Mixed	5120	WMT14 English-German	V100-SXM2-16GB
		Attention	21.2 minutes	16x V100	DGX-2	18.11-py3	Mixed	10240	WMT14 English-German	V100-SXM3-32GB
		Attention	19.2 minutes	16x V100	DGX-2H	18.11-py3	Mixed	10240	WMT14 English-German	V100-SXM3-32GB
		Attention	6.2 minutes	192x V100	DGX-1V Saturn	18.11-py3	Mixed	2560	WMT14 English-German	V100-SXM2-16GB

V100 Training Performance

FrameWork	Network	Network Type	Throughput	GPU	Server	Container	Precision	BatchSize	DataSet	GPU Version
MXNet	Inception V3	CNN	511 images/sec	1x V100	DGX-1	19.05-py3	Mixed	256	ImageNet2012	V100-SXM2-32GB
		CNN	584 images/sec	1x V100	DGX-2H	19.05-py3	Mixed	256	ImageNet2012	V100-SXM3-32GB-H
		CNN	4063 images/sec	8x V100	DGX-1	19.05-py3	Mixed	256	ImageNet2012	V100-SXM2-32GB
		CNN	4520 images/sec	8x V100	DGX-2H	19.05-py3	Mixed	256	ImageNet2012	V100-SXM3-32GB-H
	ResNet-50	CNN	1409 images/sec	1x V100	DGX-1	19.02-py3	Mixed	128	ImageNet2012	V100-SXM2-16GB
		CNN	1442 images/sec	1x V100	DGX-2	19.02-py3	Mixed	256	ImageNet2012	V100-SXM3-32GB
		CNN	10380 images/sec	8x V100	DGX-1	19.02-py3	Mixed	128	ImageNet2012	V100-SXM2-16GB
		CNN	10530 images/sec	8x V100	DGX-2	19.02-py3	Mixed	256	ImageNet2012	V100-SXM3-32GB
	ResNet-50 V1.5	CNN	1419 images/sec	1x V100	DGX-1	19.05-py3	Mixed	208	ImageNet2012	V100-SXM2-16GB
		CNN	1580 images/sec	1x V100	DGX-2H	19.05-py3	Mixed	256	ImageNet2012	V100-SXM3-32GB-H
		CNN	11056 images/sec	8x V100	DGX-2	19.05-py3	Mixed	128	ImageNet2012	V100-SXM3-32GB
		CNN	11507 images/sec	8x V100	DGX-2H	19.05-py3	Mixed	256	ImageNet2012	V100-SXM3-32GB-H

PyTorch	Inception V3	CNN	537 images/sec	1x V100	DGX-1	19.03-py3	Mixed	256	ImageNet2012	V100-SXM2-32GB
		CNN	572 images/sec	1x V100	DGX-2	19.03-py3	Mixed	256	ImageNet2012	V100-SXM3-32GB
		CNN	4156 images/sec	8x V100	DGX-1	19.03-py3	Mixed	256	ImageNet2012	V100-SXM2-32GB
	Mask R-CNN	CNN	14 images/sec	1x V100	DGX-1	19.05-py3	Mixed	16	COCO2014	V100-SXM2-32GB
		CNN	17 images/sec	1x V100	DGX-2H	19.05-py3	Mixed	16	COCO2014	V100-SXM3-32GB-H
		CNN	84 images/sec	8x V100	DGX-1	19.05-py3	Mixed	16	COCO2014	V100-SXM2-32GB
	NCF	CNN	19839034 samples/sec	1x V100	DGX-1	19.05-py3	Mixed	1048576	MovieLens 20 Million	V100-SXM2-32GB
		CNN	23398907 samples/sec	1x V100	DGX-2H	19.05-py3	Mixed	1048576	MovieLens 20 Million	V100-SXM3-32GB-H
		CNN	95524496 samples/sec	8x V100	DGX-1	19.05-py3	Mixed	1048576	MovieLens 20 Million	V100-SXM2-32GB
		CNN	106267011 samples/sec	8x V100	DGX-2H	19.05-py3	Mixed	1048576	MovieLens 20 Million	V100-SXM3-32GB-H
	ResNet-50	CNN	849 images/sec	1x V100	DGX-1	18.10-py3	Mixed	256	ImageNet2012	V100-SXM2-32GB
		CNN	898 images/sec	1x V100	DGX-2	18.10-py3	Mixed	256	ImageNet2012	V100-SXM3-32GB
		CNN	6675 images/sec	8x V100	DGX-1	18.10-py3	Mixed	256	ImageNet2012	V100-SXM2-32GB
	ResNet-50 V1.5	CNN	855 images/sec	1x V100	DGX-1	19.05-py3	Mixed	256	ImageNet2012	V100-SXM2-32GB
		CNN	920 images/sec	1x V100	DGX-2	19.05-py3	Mixed	128	ImageNet2012	V100-SXM3-32GB
		CNN	6791 images/sec	8x V100	DGX-1	19.05-py3	Mixed	256	ImageNet2012	V100-SXM2-32GB
	SSD	CNN	257 images/sec	1x V100	DGX-1	19.05-py3	Mixed	64	COCO 2017	V100-SXM2-32GB
		CNN	275 images/sec	1x V100	DGX-2	19.05-py3	Mixed	64	COCO 2017	V100-SXM3-32GB
		CNN	2017 images/sec	8x V100	DGX-1	19.05-py3	Mixed	64	COCO 2017	V100-SXM2-32GB
	Tacotron2	CNN	2553 tokens/sec	1x V100	DGX-1	19.05-py3	Mixed	80	LJ Speech 1.1	V100-SXM2-16GB
		CNN	2745 tokens/sec	1x V100	DGX-2	19.05-py3	Mixed	80	LJ Speech 1.1	V100-SXM3-32GB
		CNN	3054 tokens/sec	1x V100	DGX-2H	19.05-py3	Mixed	80	LJ Speech 1.1	V100-SXM3-32GB-H
		CNN	17185 tokens/sec	8x V100	DGX-1	19.04-py3	Mixed	80	LJ Speech 1.1	V100-SXM2-32GB
		CNN	18265 tokens/sec	8x V100	DGX-2	19.04-py3	Mixed	80	LJ Speech 1.1	V100-SXM3-32GB
	WaveGlow	CNN	73120 output samples/sec	1x V100	DGX-1	19.05-py3	Mixed	8	LJ Speech 1.1	V100-SXM2-32GB
		CNN	83975 output samples/sec	1x V100	DGX-2	19.05-py3	Mixed	8	LJ Speech 1.1	V100-SXM3-32GB
		CNN	533364 output samples/sec	8x V100	DGX-1	19.05-py3	Mixed	8	LJ Speech 1.1	V100-SXM2-32GB

TensorFlow	Inception V3	CNN	537 images/sec	1x V100	DGX-1	19.05-py3	Mixed	256	ImageNet2012	V100-SXM2-32GB
		CNN	572 images/sec	1x V100	DGX-2	19.05-py3	Mixed	256	ImageNet2012	V100-SXM3-32GB
		CNN	4078 images/sec	8x V100	DGX-1	19.05-py3	Mixed	256	ImageNet2012	V100-SXM2-32GB
	NCF	CNN	24393781 samples/sec	1x V100	DGX-1	19.05-py3	Mixed	1048576	MovieLens 20 Million	V100-SXM2-32GB
		CNN	65260213 samples/sec	8x V100	DGX-1	19.05-py3	Mixed	1048576	MovieLens 20 Million	V100-SXM2-32GB
	ResNet-50	CNN	857 images/sec	1x V100	DGX-1	19.02-py3	Mixed	256	ImageNet2012	V100-SXM2-32GB
		CNN	921 images/sec	1x V100	DGX-2	19.02-py3	Mixed	256	ImageNet2012	V100-SXM3-32GB
		CNN	6704 images/sec	8x V100	DGX-1	19.02-py3	Mixed	128	ImageNet2012	V100-SXM2-16GB
		CNN	1000 images/sec	8x V100	DGX-2H	19.01-py3	Mixed	256	ImageNet2012	V100-SXM3-32GB-H
		CNN	7022 images/sec	8x V100	DGX-2H	19.01-py3	Mixed	256	ImageNet2012	V100-SXM3-32GB-H
	ResNet-50 V1.5	CNN	785 images/sec	1x V100	DGX-1	19.05-py3	Mixed	128	ImageNet2012	V100-SXM2-32GB
		CNN	846 images/sec	1x V100	DGX-2	19.05-py3	Mixed	128	ImageNet2012	V100-SXM3-32GB
		CNN	6295 images/sec	8x V100	DGX-1	19.05-py3	Mixed	128	ImageNet2012	V100-SXM2-32GB
	SSD	CNN	123 images/sec	1x V100	DGX-1	19.05-py3	Mixed	32	COCO 2017	V100-SXM2-32GB
		CNN	136 images/sec	1x V100	DGX-2	19.05-py3	Mixed	32	COCO 2017	V100-SXM3-32GB
		CNN	665 images/sec	8x V100	DGX-1	19.05-py3	Mixed	32	COCO 2017	V100-SXM2-32GB
		CNN	770 images/sec	8x V100	DGX-2	19.05-py3	Mixed	32	COCO 2017	V100-SXM3-32GB
	U-Net Industrial	CNN	100 images/sec	1x V100	DGX-1	19.05-py3	Mixed	16	DAGM2007	V100-SXM2-32GB
		CNN	107 images/sec	1x V100	DGX-2	19.05-py3	Mixed	16	DAGM2007	V100-SXM3-32GB
		CNN	517 images/sec	8x V100	DGX-1	19.05-py3	Mixed	2	DAGM2007	V100-SXM2-32GB
		CNN	546 images/sec	8x V100	DGX-2	19.05-py3	Mixed	2	DAGM2007	V100-SXM3-32GB

PyTorch	GNMT V2	RNN	78347 tokens/sec	1x V100	DGX-1	19.05-py3	Mixed	128	WMT16 English-German	V100-SXM2-32GB
		RNN	83860 tokens/sec	1x V100	DGX-2	19.05-py3	Mixed	128	WMT16 English-German	V100-SXM3-32GB
		RNN	598342 tokens/sec	8x V100	DGX-1	19.05-py3	Mixed	128	WMT16 English-German	V100-SXM2-32GB

TensorFlow	GNMT V2	RNN	24899 tokens/sec	1x V100	DGX-1	19.05-py3	Mixed	192	WMT16 English-German	V100-SXM2-16GB
		RNN	25733 tokens/sec	1x V100	DGX-2H	19.05-py3	Mixed	192	WMT16 English-German	V100-SXM3-32GB-H
		RNN	138216 tokens/sec	8x V100	DGX-1	19.05-py3	Mixed	192	WMT16 English-German	V100-SXM2-32GB

TensorFlow	BERT	Attention	29 sentences/sec	1x V100	DGX-1	19.05-py3	Mixed	10	SQuaD v1.1	V100-SXM2-32GB
		Attention	34.5 sentences/sec	1x V100	DGX-2H	19.05-py3	Mixed	10	SQuaD v1.1	V100-SXM3-32GB-H
		Attention	161.6 sentences/sec	8x V100	DGX-2	19.05-py3	Mixed	10	SQuaD v1.1	V100-SXM3-32GB

T4 Training Performance

FrameWork	Network	Network Type	Throughput	GPU	Server	Container	Precision	BatchSize	DataSet	GPU Version
MXNet	Inception V3	CNN	172 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.05-py3	Mixed	128	ImageNet2012	Tesla T4
		CNN	1359 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.05-py3	Mixed	128	ImageNet2012	Tesla T4
	ResNet-50	CNN	425 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.02-py3	Mixed	128	ImageNet2012	Tesla T4
		CNN	3329 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.02-py3	Mixed	128	ImageNet2012	Tesla T4
	ResNet-50 V1.5	CNN	447 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.05-py3	Mixed	208	ImageNet2012	Tesla T4
		CNN	4116 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.05-py3	Mixed	208	ImageNet2012	Tesla T4

PyTorch	Inception V3	CNN	164 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.05-py3	Mixed	128	ImageNet2012	Tesla T4
		CNN	1253 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.03-py3	Mixed	128	ImageNet2012	Tesla T4
	Mask R-CNN	CNN	6 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.05-py3	Mixed	4	ImageNet2012	Tesla T4
		CNN	39 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.05-py3	Mixed	4	ImageNet2012	Tesla T4
	NCF	CNN	7134482 samples/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.05-py3	Mixed	1048576	ImageNet2012	Tesla T4
		CNN	25428142 samples/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.05-py3	Mixed	1048576	ImageNet2012	Tesla T4
	ResNet-50	CNN	252 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.02-py3	Mixed	256	ImageNet2012	Tesla T4
		CNN	2054 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.01-py3	Mixed	256	ImageNet2012	Tesla T4
	ResNet-50 V1.5	CNN	282 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.05-py3	Mixed	256	ImageNet2012	Tesla T4
		CNN	2241 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.05-py3	Mixed	256	ImageNet2012	Tesla T4
	SSD	CNN	86 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.05-py3	Mixed	64	COCO 2017	Tesla T4
		CNN	693 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.05-py3	Mixed	64	COCO 2017	Tesla T4
	Tacotron2	CNN	1466 tokens/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.05-py3	Mixed	80	LJ Speech 1.1	Tesla T4
		CNN	9390 tokens/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.05-py3	Mixed	80	LJ Speech 1.1	Tesla T4
	WaveGlow	CNN	33256 output samples/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.05-py3	Mixed	8	LJ Speech 1.1	Tesla T4
		CNN	250173 output samples/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.05-py3	Mixed	8	LJ Speech 1.1	Tesla T4

TensorFlow	Inception V3	CNN	181 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.05-py3	Mixed	128	ImageNet2012	Tesla T4
		CNN	1358 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.05-py3	Mixed	128	ImageNet2012	Tesla T4
	NCF	CNN	9573658 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.05-py3	Mixed	1048576	ImageNet2012	Tesla T4
		CNN	19069773 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.05-py3	Mixed	1048576	ImageNet2012	Tesla T4
	ResNet-50	CNN	294 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.02-py3	Mixed	128	ImageNet2012	Tesla T4
		CNN	2262 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.02-py3	Mixed	128	ImageNet2012	Tesla T4
	ResNet-50 V1.5	CNN	272 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.05-py3	Mixed	128	ImageNet2012	Tesla T4
		CNN	2151 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.05-py3	Mixed	128	ImageNet2012	Tesla T4
	SSD	CNN	52 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.05-py3	Mixed	32	COCO 2017	Tesla T4
		CNN	281 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.05-py3	Mixed	32	COCO 2017	Tesla T4
	U-Net Industrial	CNN	29 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.05-py3	Mixed	16	DAGM2007	Tesla T4
		CNN	196 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.05-py3	Mixed	2	DAGM2007	Tesla T4

PyTorch	GNMT V2	RNN	25992 tokens/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.05-py3	Mixed	128	WMT16 English-German	Tesla T4
		RNN	184525 tokens/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.05-py3	Mixed	128	WMT16 English-German	Tesla T4

TensorFlow	GNMT V2	RNN	9654 tokens/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.05-py3	Mixed	192	WMT16 English-German	Tesla T4
		RNN	53570 tokens/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.05-py3	Mixed	192	WMT16 English-German	Tesla T4

TensorFlow	BERT	Attention	8 sentences/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.05-py3	Mixed	3	SQuaD v1.1	Tesla T4
			31 sentences/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.05-py3	Mixed	3	SQuaD v1.1	Tesla T4

NVIDIA Deep Learning Inference Performance

NVIDIA® TensorRT™ running on NVIDIA GPUs enable the most efficient deep learning inference performance across multiple application areas and models. This versatility provides wide latitude to data scientists to create the optimal low-latency solution. Visit NVIDIA GPU Cloud (NGC) to download any of these containers.

NVIDIA Tesla® V100 Tensor Cores GPUs leverage mixed-precision to combine high throughput with low latencies across every type of neural network. Tesla P4 is an inference GPU, designed for optimal power consumption and latency, for ultra-efficient scale-out servers. Read the inference whitepaper to learn more about NVIDIA’s inference platform.

Measuring the inference performance involves balancing a lot of variables. PLASTER is an acronym that describes the key elements for measuring deep learning performance. Each letter identifies a factor (Programmability, Latency, Accuracy, Size of Model, Throughput, Energy Efficiency, Rate of Learning) that must be considered to arrive at the right set of tradeoffs and to produce a successful deep learning implementation. Refer to NVIDIA’s PLASTER whitepaper for more details.

Inference Image Classification on CNNs with TensorRT

ResNet-50 Throughput

ResNet-50 Latency

ResNet-50 Power Efficiency

Inference Performance

V100 Inference Performance

Network	Network Type	Batch Size	Throughput	Efficiency	Latency	GPU	Server	Container	Precision	Dataset	GPU Version
GoogleNet	CNN	1	1579 images/sec	14 images/sec/watt	0.63	1x V100	DGX-1	19.05-py3	INT8	Synthetic	V100-SXM2-16GB
			1686 images/sec	12 images/sec/watt	0.59	1x V100	DGX-2	19.05-py3	INT8	Synthetic	V100-SXM3-32GB
	CNN	2	2129 images/sec	18 images/sec/watt	0.94	1x V100	DGX-1	19.05-py3	INT8	Synthetic	V100-SXM2-16GB
	CNN	8	5140 images/sec	35 images/sec/watt	1.6	1x V100	DGX-1	19.05-py3	INT8	Synthetic	V100-SXM2-16GB
	CNN	82	11805 images/sec	44 images/sec/watt	7	1x V100	DGX-1	19.05-py3	INT8	Synthetic	V100-SXM2-16GB
	CNN	128	12345 images/sec	45 images/sec/watt	10	1x V100	DGX-1	19.05-py3	INT8	Synthetic	V100-SXM2-16GB

MobileNet V2	CNN	1	1737 images/sec	18 images/sec/watt	0.58	1x V100	DGX-1	-	INT8	Synthetic	V100-SXM2-16GB
	CNN	2	2886 images/sec	30 images/sec/watt	0.69	1x V100	DGX-1	-	INT8	Synthetic	V100-SXM2-16GB
	CNN	8	8848 images/sec	63 images/sec/watt	0.9	1x V100	DGX-1	-	INT8	Synthetic	V100-SXM2-16GB
	CNN	32	17649 images/sec	77 images/sec/watt	1.8	1x V100	DGX-1	-	INT8	Synthetic	V100-SXM2-16GB
	CNN	128	23262 images/sec	81 images/sec/watt	5.5	1x V100	DGX-1	-	INT8	Synthetic	V100-SXM2-16GB

ResNet-50	CNN	1	1118 images/sec	8.2 images/sec/watt	0.89	1x V100	DGX-1	19.05-py3	INT8	Synthetic	V100-SXM2-16GB
			1176 images/sec	7.1 images/sec/watt	0.85	1x V100	DGX-2	19.05-py3	INT8	Synthetic	V100-SXM3-32GB
	CNN	2	1551 images/sec	11 images/sec/watt	1.3	1x V100	DGX-1	19.05-py3	INT8	Synthetic	V100-SXM2-16GB
	CNN	8	3308 images/sec	21 images/sec/watt	2.4	1x V100	DGX-1	19.05-py3	Mixed	Synthetic	V100-SXM2-16GB
	CNN	39	5821 images/sec	22 images/sec/watt	6.7	1x V100	DGX-1	19.05-py3	INT8	Synthetic	V100-SXM2-16GB
	CNN	128	7636 images/sec	27 images/sec/watt	17	1x V100	DGX-2	19.05-py3	Mixed	Synthetic	V100-SXM2-16GB

ResNet-50v1.5	CNN	1	934 images/sec	7.1 images/sec/watt	1.1	1x V100	DGX-1	19.05-py3	INT8	Synthetic	V100-SXM2-16GB
	CNN	2	1396 images/sec	9.8 images/sec/watt	1.4	1x V100	DGX-1	19.05-py3	INT8	Synthetic	V100-SXM2-16GB
	CNN	8	3239 images/sec	20 images/sec/watt	2.5	1x V100	DGX-1	19.05-py3	Mixed	Synthetic	V100-SXM2-16GB
	CNN	128	7309 images/sec	25 images/sec/watt	18	1x V100	DGX-1	19.05-py3	Mixed	Synthetic	V100-SXM2-16GB

VGG16	CNN	1	724 images/sec	3.8 images/sec/watt	1.4	1x V100	DGX-1	19.05-py3	Mixed	Synthetic	V100-SXM2-16GB
	CNN	2	1133 images/sec	5.5 images/sec/watt	1.8	1x V100	DGX-1	19.05-py3	Mixed	Synthetic	V100-SXM2-16GB
	CNN	8	2044 images/sec	8 images/sec/watt	3.9	1x V100	DGX-1	19.05-py3	Mixed	Synthetic	V100-SXM2-16GB
	CNN	128	2855 images/sec	9.9 images/sec/watt	45	1x V100	DGX-1	19.05-py3	Mixed	Synthetic	V100-SXM2-16GB

NMT	RNN	1	4013 tokens/sec	tokens/sec/watt	13	1x V100	DGX-1	-	Mixed	wmt16-English-German	V100-SXM2-32GB
	RNN	2	6290 tokens/sec	tokens/sec/watt	16	1x V100	DGX-1	-	Mixed	wmt16-English-German	V100-SXM2-32GB
	RNN	64	56531 tokens/sec	tokens/sec/watt	58	1x V100	DGX-1	-	Mixed	wmt16-English-German	V100-SXM2-32GB
	RNN	128	73375 tokens/sec	tokens/sec/watt	89	1x V100	DGX-1	-	Mixed	wmt16-English-German	V100-SXM2-32GB

Deep Recommender	Recommender	1	5541 images/sec	48 images/sec/watt	0.18	1x V100	DGX-1	-	Mixed	Synthetic	V100-SXM2-16GB
Deep Recommender	Recommender	2	10066 images/sec	88 images/sec/watt	0.2	1x V100	DGX-1	-	Mixed	Synthetic	V100-SXM2-16GB
Deep Recommender	Recommender	64	282209 images/sec	1874 images/sec/watt	0.23	1x V100	DGX-1	-	Mixed	Synthetic	V100-SXM2-16GB
Deep Recommender	Recommender	128	393347 images/sec	2461 images/sec/watt	0.33	1x V100	DGX-1	-	Mixed	Synthetic	V100-SXM2-16GB

NCF	Recommender	1	21824 images/sec	310 images/sec/watt	0.05	1x V100	DGX-1	19.05-py3	Mixed	Synthetic	V100-SXM2-16GB
	Recommender	64	1335854 images/sec	16998 images/sec/watt	0.05	1x V100	DGX-1	19.05-py3	Mixed	Synthetic	V100-SXM2-16GB
	Recommender	25000	91680666 images/sec	730549 images/sec/watt	0.27	1x V100	DGX-1	19.05-py3	Mixed	Synthetic	V100-SXM2-16GB
	Recommender	100000	114022270 images/sec	694185 images/sec/watt	0.88	1x V100	DGX-1	19.05-py3	Mixed	Synthetic	V100-SXM2-16GB

TensorRT 5.1, except TensorRT 5.0 for MobileNet V2 and Deep Recommender

T4 Inference Performance

Network	Network Type	Batch Size	Throughput	Efficiency	Latency	GPU	Server	Container	Precision	Dataset	GPU Version
GoogleNet	CNN	1	1687 images/sec	25 images/sec/watt	0.59	1x T4	Supermicro SYS-4029GP-TRT T4	19.06-py3	INT8	Synthetic	Tesla T4
	CNN	2	2356 images/sec	35 images/sec/watt	0.85	1x T4	Supermicro SYS-4029GP-TRT T4	19.06-py3	INT8	Synthetic	Tesla T4
	CNN	8	5458 images/sec	78 images/sec/watt	1.5	1x T4	Supermicro SYS-4029GP-TRT T4	19.06-py3	INT8	Synthetic	Tesla T4
	CNN	52	7335 images/sec	105 images/sec/watt	7.1	1x T4	Supermicro SYS-4029GP-TRT T4	19.06-py3	INT8	Synthetic	Tesla T4
	CNN	128	7516 images/sec	108 images/sec/watt	17	1x T4	Supermicro SYS-4029GP-TRT T4	19.06-py3	INT8	Synthetic	Tesla T4

MobileNet V2	CNN	1	1766 images/sec	34 images/sec/watt	0.57	1x T4	Supermicro SYS-4029GP-TRT T4	-	INT8	Synthetic	Tesla T4
	CNN	2	3235 images/sec	55 images/sec/watt	0.62	1x T4	Supermicro SYS-4029GP-TRT T4	-	INT8	Synthetic	Tesla T4
	CNN	8	7251 images/sec	106 images/sec/watt	1.1	1x T4	Supermicro SYS-4029GP-TRT T4	-	INT8	Synthetic	Tesla T4
	CNN	32	8869 images/sec	129 images/sec/watt	3.6	1x T4	Supermicro SYS-4029GP-TRT T4	-	INT8	Synthetic	Tesla T4
	CNN	128	9059 images/sec	131 images/sec/watt	14	1x T4	Supermicro SYS-4029GP-TRT T4	-	INT8	Synthetic	Tesla T4

ResNet-50	CNN	1	1065 images/sec	15 images/sec/watt	0.94	1x T4	Supermicro SYS-4029GP-TRT T4	19.06-py3	INT8	Synthetic	Tesla T4
	CNN	2	1725 images/sec	25 images/sec/watt	1.2	1x T4	Supermicro SYS-4029GP-TRT T4	19.06-py3	INT8	Synthetic	Tesla T4
	CNN	8	3778 images/sec	55 images/sec/watt	2.1	1x T4	Supermicro SYS-4029GP-TRT T4	19.06-py3	INT8	Synthetic	Tesla T4
	CNN	33	4723 images/sec	68 images/sec/watt	7	1x T4	Supermicro SYS-4029GP-TRT T4	19.06-py3	INT8	Synthetic	Tesla T4
	CNN	128	5106 images/sec	74 images/sec/watt	25	1x T4	Supermicro SYS-4029GP-TRT T4	19.06-py3	INT8	Synthetic	Tesla T4

VGG16	CNN	1	407 images/sec	5.8 images/sec/watt	2.5	1x T4	Supermicro SYS-4029GP-TRT T4	19.05-py3	INT8	Synthetic	Tesla T4
	CNN	2	656 images/sec	9.5 images/sec/watt	3.1	1x T4	Supermicro SYS-4029GP-TRT T4	19.05-py3	INT8	Synthetic	Tesla T4
	CNN	8	1379 images/sec	20 images/sec/watt	5.8	1x T4	Supermicro SYS-4029GP-TRT T4	19.05-py3	INT8	Synthetic	Tesla T4
	CNN	32	1598 images/sec	23 images/sec/watt	20	1x T4	Supermicro SYS-4029GP-TRT T4	19.02-py3	INT8	Synthetic	Tesla T4
	CNN	128	1895 images/sec	27 images/sec/watt	68	1x T4	Supermicro SYS-4029GP-TRT T4	19.05-py3	INT8	Synthetic	Tesla T4

NMT	RNN	1	2511 tokens/sec	38 tokens/sec/watt	20	1x T4	Supermicro SYS-4029GP-TRT T4	-	Mixed	wmt16-English-German	Tesla T4
	RNN	2	3768 tokens/sec	58 tokens/sec/watt	27	1x T4	Supermicro SYS-4029GP-TRT T4	-	Mixed	wmt16-English-German	Tesla T4
	RNN	8	9975 tokens/sec	160 tokens/sec/watt	41	1x T4	Supermicro SYS-4029GP-TRT T4	-	Mixed	wmt16-English-German	Tesla T4
	RNN	128	34124 tokens/sec	647 tokens/sec/watt	192	1x T4	Supermicro SYS-4029GP-TRT T4	-	Mixed	wmt16-English-German	Tesla T4

Deep Recommender	Recommender	1	3252 images/sec	47 images/sec/watt	0.31	1x T4	Supermicro SYS-4029GP-TRT T4	-	Mixed	Synthetic	Tesla T4
	Recommender	2	6417 images/sec	94 images/sec/watt	0.31	1x T4	Supermicro SYS-4029GP-TRT T4	-	Mixed	Synthetic	Tesla T4
	Recommender	8	23650 images/sec	344 images/sec/watt	0.34	1x T4	Supermicro SYS-4029GP-TRT T4	-	Mixed	Synthetic	Tesla T4
	Recommender	128	219511 images/sec	3186 images/sec/watt	0.58	1x T4	Supermicro SYS-4029GP-TRT T4	-	Mixed	Synthetic	Tesla T4

NCF	Recommender	1	7623 images/sec	274 images/sec/watt	0.14	1x T4	Supermicro SYS-4029GP-TRT T4	19.05-py3	Mixed	Synthetic	Tesla T4
	Recommender	64	488930 images/sec	16539 images/sec/watt	0.14	1x T4	Supermicro SYS-4029GP-TRT T4	19.05-py3	Mixed	Synthetic	Tesla T4
	Recommender	25000	50381198 images/sec	726308 images/sec/watt	0.5	1x T4	Supermicro SYS-4029GP-TRT T4	19.05-py3	Mixed	Synthetic	Tesla T4
	Recommender	100000	53374338 images/sec	726308 images/sec/watt	0.5	1x T4	Supermicro SYS-4029GP-TRT T4	19.05-py3	Mixed	Synthetic	Tesla T4

TensorRT 5.1, except TensorRT 5.0 for MobileNet V2, NMT and Deep Recommender

Last updated: June 19th, 2019

Search form

NVIDIA Tesla Deep Learning Product Performance

Deep Learning Training

NVIDIA Performance on MLPerf AI Benchmarks

Training Image Classification on CNNs

Training Performance

NVIDIA Performance on MLPerf AI Benchmarks

V100 Training Performance

T4 Training Performance

NVIDIA Deep Learning Inference Performance

Inference Image Classification on CNNs with TensorRT

Inference Performance

V100 Inference Performance

T4 Inference Performance