NVIDIA Tesla Deep Learning Product Performance

Review the latest GPU acceleration factors of popular HPC applications.

Training
Inference

NVIDIA’s complete solution stack, from GPUs to libraries, and containers on NVIDIA GPU Cloud (NGC), allows data scientists to quickly get up and running with deep learning. NVIDIA® Tesla® V100 Tensor Core GPUs leverage mixed precision to accelerate deep learning training throughputs across every framework and every type of neural network. NVIDIA breaks performance records on MLPerf, the AI’s first industry-wide benchmark, a testament to our GPU-accelerated platform approach.

NVIDIA Performance on MLPerf 0.6 AI Benchmarks

ResNet-50 v1.5 Time to Solution on V100

MXNet | Batch Size refer to CNN V100 Training table below | Precision: Mixed | Dataset: ImageNet2012 | Convergence criteria - refer to MLPerf requirements

Training Image Classification on CNNs

ResNet-50 V1.5 Throughput on V100

DGX-1: 8x Tesla V100-SXM2-16GB for MXNet, PyTorch and TensorFlow, E5-2698 v4 2.2 GHz | Batch Size = 208 for MXNet, PyTorch and TensorFlow = 256 | MXNet and TensorFlow = 19.09-py3, PyTorch = 19.10-py3 | Precision: Mixed | Dataset: ImageNet2012

ResNet-50 V1.5 Throughput on T4

Supermicro SYS-4029GP-TRT T4: 8x Tesla T4, Gold 6140 2.3 GHz for MXNet and TensorFlow, Gold 6240 2.6GHz for PyTorch | Batch Size = 208 for MXNet, PyTorch and TensorFlow = 256 | MXNet = 19.05-py3, PyTorch = 19.09-py3, TensorFlow = 19.06-py3 | Precision: Mixed | Dataset: ImageNet2012

Training Performance

NVIDIA Performance on MLPerf 0.6 AI Benchmarks

Framework	Network	Network Type	Time to Solution	GPU	Server	MLPerf-ID	Precision	Dataset	GPU Version
MXNet	ResNet-50 v1.5	CNN	115.22 minutes	8x V100	DGX-1	0.6-8	Mixed	ImageNet2012	V100-SXM2-16GB
		CNN	57.87 minutes	16x V100	DGX-2	0.6-17	Mixed	ImageNet2012	V100-SXM3-32GB
		CNN	52.74 minutes	16x V100	DGX-2H	0.6-19	Mixed	ImageNet2012	V100-SXM3-32GB-H
		CNN	2.59 minutes	512x V100	DGX-2H	0.6-29	Mixed	ImageNet2012	V100-SXM3-32GB-H
		CNN	1.69 minutes	1040x V100	DGX-1	0.6-16	Mixed	ImageNet2012	V100-SXM2-16GB
		CNN	1.33 minutes	1536x V100	DGX-2H	0.6-30	Mixed	ImageNet2012	V100-SXM3-32GB-H

PyTorch	SSD-ResNet-34	CNN	22.36 minutes	8x V100	DGX-1	0.6-9	Mixed	COCO2017	V100-SXM2-16GB
		CNN	12.21 minutes	16x V100	DGX-2	0.6-18	Mixed	COCO2017	V100-SXM3-32GB
		CNN	11.41 minutes	16x V100	DGX-2H	0.6-20	Mixed	COCO2017	V100-SXM3-32GB-H
		CNN	4.78 minutes	64x V100	DGX-2H	0.6-21	Mixed	COCO2017	V100-SXM3-32GB-H
		CNN	2.67 minutes	240x V100	DGX-1	0.6-13	Mixed	COCO2017	V100-SXM2-16GB
		CNN	2.56 minutes	240x V100	DGX-2H	0.6-24	Mixed	COCO2017	V100-SXM3-32GB-H
		CNN	2.23 minutes	240x V100	DGX-2H	0.6-27	Mixed	COCO2017	V100-SXM3-32GB-H
	Mask R-CNN	CNN	207.48 minutes	8x V100	DGX-1	0.6-9	Mixed	COCO2017	V100-SXM2-16GB
		CNN	101 minutes	16x V100	DGX-2	0.6-18	Mixed	COCO2017	V100-SXM3-32GB
		CNN	95.2 minutes	16x V100	DGX-2H	0.6-20	Mixed	COCO2017	V100-SXM3-32GB-H
		CNN	32.72 minutes	64x V100	DGX-2H	0.6-21	Mixed	COCO2017	V100-SXM3-32GB-H
		CNN	22.03 minutes	192x V100	DGX-1	0.6-12	Mixed	COCO2017	V100-SXM2-16GB
		CNN	18.47 minutes	192x V100	DGX-2H	0.6-23	Mixed	COCO2017	V100-SXM3-32GB-H

PyTorch	GNMT	RNN	20.55 minutes	8x V100	DGX-1	0.6-9	Mixed	WMT16 English-German	V100-SXM2-16GB
		RNN	10.94 minutes	16x V100	DGX-2	0.6-18	Mixed	WMT16 English-German	V100-SXM3-32GB
		RNN	9.87 minutes	16x V100	DGX-2H	0.6-20	Mixed	WMT16 English-German	V100-SXM3-32GB-H
		RNN	2.12 minutes	256x V100	DGX-2H	0.6-25	Mixed	WMT16 English-German	V100-SXM3-32GB-H
		RNN	1.99 minutes	384x V100	DGX-1	0.6-14	Mixed	WMT16 English-German	V100-SXM2-16GB
		RNN	1.8 minutes	384x V100	DGX-2H	0.6-26	Mixed	WMT16 English-German	V100-SXM3-32GB-H

PyTorch	Transformer	Attention	20.34 minutes	8x V100	DGX-1	0.6-9	Mixed	WMT17 English-German	V100-SXM2-16GB
		Attention	11.04 minutes	16x V100	DGX-2	0.6-18	Mixed	WMT17 English-German	V100-SXM3-32GB
		Attention	9.8 minutes	16x V100	DGX-2H	0.6-20	Mixed	WMT17 English-German	V100-SXM3-32GB-H
		Attention	2.41 minutes	160x V100	DGX-2H	0.6-22	Mixed	WMT17 English-German	V100-SXM3-32GB-H
		Attention	2.05 minutes	480x V100	DGX-1	0.6-15	Mixed	WMT17 English-German	V100-SXM2-16GB
		Attention	1.59 minutes	480x V100	DGX-2H	0.6-28	Mixed	WMT17 English-German	V100-SXM3-32GB-H

TensorFlow	MiniGo	Reinforcement Learning	27.39 minutes	8x V100	DGX-1	0.6-10	Mixed	N/A	V100-SXM2-16GB
		Reinforcement Learning	13.57 minutes	24x V100	DGX-1	0.6-11	Mixed	N/A	V100-SXM2-16GB

V100 Training Performance

Framework	Network	Network Type	Throughput	GPU	Server	Container	Precision	Batch Size	Dataset	GPU Version
MXNet	Inception V3	CNN	552 images/sec	1x V100	DGX-1	19.09-py3	Mixed	208	ImageNet2012	V100-SXM2-16GB
		CNN	563 images/sec	1x V100	DGX-2	19.09-py3	Mixed	384	ImageNet2012	V100-SXM3-32GB
		CNN	4253 images/sec	8x V100	DGX-1	19.09-py3	Mixed	208	ImageNet2012	V100-SXM2-16GB
		CNN	4345 images/sec	8x V100	DGX-2	19.09-py3	Mixed	384	ImageNet2012	V100-SXM3-32GB
	ResNet-50	CNN	1409 images/sec	1x V100	DGX-1	19.02-py3	Mixed	128	ImageNet2012	V100-SXM2-16GB
		CNN	1442 images/sec	1x V100	DGX-2	19.02-py3	Mixed	256	ImageNet2012	V100-SXM3-32GB
		CNN	10380 images/sec	8x V100	DGX-1	19.02-py3	Mixed	128	ImageNet2012	V100-SXM2-16GB
		CNN	10530 images/sec	8x V100	DGX-2	19.02-py3	Mixed	256	ImageNet2012	V100-SXM3-32GB
	ResNet-50 v1.5	CNN	1461 images/sec	1x V100	DGX-1	19.10-py3	Mixed	208	ImageNet2012	V100-SXM2-16GB
		CNN	1660 images/sec	1x V100	DGX-2H	19.10-py3	Mixed	256	ImageNet2012	V100-SXM3-32GB-H
		CNN	10824 images/sec	8x V100	DGX-1	19.09-py3	Mixed	208	ImageNet2012	V100-SXM2-16GB
		CNN	11479 images/sec	8x V100	DGX-2	19.10-py3	Mixed	256	ImageNet2012	V100-SXM3-32GB
		CNN	12241 images/sec	8x V100	DGX-2H	19.10-py3	Mixed	256	ImageNet2012	V100-SXM3-32GB-H

PyTorch	Inception V3	CNN	571 images/sec	1x V100	DGX-1	19.09-py3	Mixed	256	ImageNet2012	V100-SXM2-16GB
		CNN	582 images/sec	1x V100	DGX-2	19.09-py3	Mixed	512	ImageNet2012	V100-SXM3-32GB
		CNN	4466 images/sec	8x V100	DGX-1	19.09-py3	Mixed	256	ImageNet2012	V100-SXM2-16GB
	Mask R-CNN	CNN	15 images/sec	1x V100	DGX-1	19.10-py3	Mixed	16	COCO2014	V100-SXM2-32GB
		CNN	17 images/sec	1x V100	DGX-2	19.09-py3	Mixed	16	COCO2014	V100-SXM3-32GB
		CNN	95 images/sec	8x V100	DGX-1	19.10-py3	Mixed	16	COCO2014	V100-SXM2-32GB
	ResNet-50	CNN	819 images/sec	1x V100	DGX-1	19.02_py3	Mixed	256	ImageNet2012	V100-SXM2-16GB
		CNN	820 images/sec	1x V100	DGX-2	19.02-py3	Mixed	256	ImageNet2012	V100-SXM3-32GB
		CNN	6218 images/sec	8x V100	DGX-1	19.02-py3	Mixed	256	ImageNet2012	V100-SXM2-16GB
	ResNet-50 v1.5	CNN	931 images/sec	1x V100	DGX-1	19.10-py3	Mixed	256	ImageNet2012	V100-SXM2-16GB
		CNN	1046 images/sec	1x V100	DGX-2H	19.10-py3	Mixed	512	ImageNet2012	V100-SXM3-32GB-H
		CNN	7308 images/sec	8x V100	DGX-1	19.10-py3	Mixed	256	ImageNet2012	V100-SXM2-16GB
	SSD v1.1	CNN	237 images/sec	1x V100	DGX-1	19.09-py3	Mixed	64	COCO 2017	V100-SXM2-16GB
		CNN	276 images/sec	1x V100	DGX-2	19.06-py3	Mixed	64	COCO 2017	V100-SXM3-32GB
		CNN	2132 images/sec	8x V100	DGX-1	19.06-py3	Mixed	64	COCO 2017	V100-SXM2-16GB
	Tacotron2	CNN	19988 total output mels/sec	1x V100	DGX-1	19.09-py3	Mixed	128	LJ Speech 1.1	V100-SXM2-32GB
		CNN	23105 total output mels/sec	1x V100	DGX-2	19.09-py3	Mixed	128	LJ Speech 1.1	V100-SXM3-32GB
		CNN	120654 total output mels/sec	8x V100	DGX-1	19.10-py3	Mixed	128	LJ Speech 1.1	V100-SXM2-32GB
		CNN	140622 total output mels/sec	8x V100	DGX-2	19.10-py3	Mixed	128	LJ Speech 1.1	V100-SXM3-32GB
	WaveGlow	CNN	82692 output samples/sec	1x V100	DGX-1	19.08-py3	Mixed	10	LJ Speech 1.1	V100-SXM2-16GB
		CNN	90150 output samples/sec	1x V100	DGX-2	19.08-py3	Mixed	10	LJ Speech 1.1	V100-SXM3-32GB
		CNN	594479 output samples/sec	8x V100	DGX-1	19.08-py3	Mixed	10	LJ Speech 1.1	V100-SXM2-16GB

TensorFlow	Inception V3	CNN	544 images/sec	1x V100	DGX-1	19.09-py3	Mixed	384	ImageNet2012	V100-SXM2-32GB
		CNN	576 images/sec	1x V100	DGX-2	19.09-py3	Mixed	384	ImageNet2012	V100-SXM3-32GB
		CNN	4154 images/sec	8x V100	DGX-1	19.09-py3	Mixed	384	ImageNet2012	V100-SXM2-32GB
	ResNet-50 V1.5	CNN	845 images/sec	1x V100	DGX-1	19.10-py3	Mixed	256	ImageNet2012	V100-SXM2-16GB
		CNN	967 images/sec	1x V100	DGX-2H	19.10-py3	Mixed	512	ImageNet2012	V100-SXM3-32GB-H
		CNN	6484 images/sec	8x V100	DGX-1	19.09-py3	Mixed	256	ImageNet2012	V100-SXM2-16GB
	SSD v1.1	CNN	117 images/sec	1x V100	DGX-1	19.10-py3	Mixed	32	COCO 2017	V100-SXM2-16GB
		CNN	127 images/sec	1x V100	DGX-2	19.10-py3	Mixed	32	COCO 2017	V100-SXM3-32GB
		CNN	672 images/sec	8x V100	DGX-1	19.06-py3	Mixed	32	COCO 2017	V100-SXM2-16GB
		CNN	768 images/sec	8x V100	DGX-2	19.06-py3	Mixed	32	COCO 2017	V100-SXM3-32GB
	U-Net Industrial	CNN	104 images/sec	1x V100	DGX-1	19.10-py3	Mixed	16	DAGM2007	V100-SXM2-16GB
		CNN	106 images/sec	1x V100	DGX-2	19.10-py3	Mixed	16	DAGM2007	V100-SXM3-32GB
		CNN	516 images/sec	8x V100	DGX-1	19.10-py3	Mixed	2	DAGM2007	V100-SXM2-16GB
		CNN	543 images/sec	8x V100	DGX-2	19.10-py3	Mixed	2	DAGM2007	V100-SXM3-32GB

PyTorch	GNMT V2	RNN	76722 total tokens/sec	1x V100	DGX-1	19.08-py3	Mixed	512	WMT16 English-German	V100-SXM2-32GB
		RNN	83948 total tokens/sec	1x V100	DGX-2	19.08-py3	Mixed	512	WMT16 English-German	V100-SXM3-32GB
		RNN	585249 total tokens/sec	8x V100	DGX-1	19.08-py3	Mixed	512	WMT16 English-German	V100-SXM2-32GB
		RNN	606769 total tokens/sec	8x V100	DGX-2	19.08-py3	Mixed	512	WMT16 English-German	V100-SXM3-32GB

TensorFlow	GNMT V2	RNN	26263 total tokens/sec	1x V100	DGX-1	19.09-py3	Mixed	192	WMT16 English-German	V100-SXM2-32GB
		RNN	29384 total tokens/sec	1x V100	DGX-2	19.09-py3	Mixed	192	WMT16 English-German	V100-SXM3-32GB
		RNN	174733 total tokens/sec	8x V100	DGX-1	19.09-py3	Mixed	192	WMT16 English-German	V100-SXM2-32GB

PyTorch	NCF	Recommender	21922278 samples/sec	1x V100	DGX-1	19.08-py3	Mixed	1048576	MovieLens 20 Million	V100-SXM2-16GB
		Recommender	22009822 samples/sec	1x V100	DGX-2	19.08-py3	Mixed	1048576	MovieLens 20 Million	V100-SXM3-32GB
		Recommender	104122673 samples/sec	8x V100	DGX-1	19.07-py3	Mixed	1048576	MovieLens 20 Million	V100-SXM2-16GB
		Recommender	109969915 samples/sec	8x V100	DGX-2H	19.07-py3	Mixed	1048576	MovieLens 20 Million	V100-SXM3-32GB-H

TensorFlow	NCF	Recommender	26071208 samples/sec	1x V100	DGX-1	19.08-py3	Mixed	1048576	MovieLens 20 Million	V100-SXM2-16GB
		Recommender	57083723 samples/sec	8x V100	DGX-1	19.10-py3	Mixed	1048576	MovieLens 20 Million	V100-SXM2-16GB

PyTorch	BERT-LARGE	Attention	50 sentences/sec	1x V100	DGX-1	19.09-py3	Mixed	10	SQuaD v1.1	V100-SXM2-32GB
		Attention	372 sentences/sec	8xV100	DGX-2	19.10-py3	Mixed	10	SQuaD v1.1	V100-SXM3-32GB

TensorFlow	BERT-LARGE	Attention	34 sentences/sec	1x V100	DGX-1	19.10-py3	Mixed	10	SQuaD v1.1	V100-SXM2-32GB
		Attention	37 sentences/sec	1x V100	DGX-2	19.10-py3	Mixed	10	SQuaD v1.1	V100-SXM3-32GB
		Attention	182 sentences/sec	8xV100	DGX-1	19.10-py3	Mixed	10	SQuaD v1.1	V100-SXM2-32GB
		Attention	195 sentences/sec	8x V100	DGX-2	19.10-py3	Mixed	10	SQuaD v1.1	V100-SXM3-32GB

T4 Training Performance

Framework	Network	Network Type	Throughput	GPU	Server	Container	Precision	Batch Size	Dataset	GPU Version
MXNet	Inception V3	CNN	180 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.08-py3	Mixed	208	ImageNet2012	Tesla T4
		CNN	1379 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.10-py3	Mixed	208	ImageNet2012	Tesla T4
	ResNet-50 v1.5	CNN	480 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.10-py3	Mixed	208	ImageNet2012	Tesla T4
		CNN	4116 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.05-py3	Mixed	208	ImageNet2012	Tesla T4

PyTorch	Inception V3	CNN	185 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.08-py3	Mixed	256	ImageNet2012	Tesla T4
		CNN	1372 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.10-py3	Mixed	256	ImageNet2012	Tesla T4
	Mask R-CNN	CNN	7 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.08-py3	Mixed	4	COCO2014	Tesla T4
		CNN	40 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.08-py3	Mixed	4	COCO2014	Tesla T4
	ResNet-50 v1.5	CNN	288 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.10-py3	Mixed	256	ImageNet2012	Tesla T4
		CNN	2307 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.09-py3	Mixed	256	ImageNet2012	Tesla T4
	SSD v1.1	CNN	85 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.06-py3	Mixed	64	COCO 2017	Tesla T4
		CNN	691 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.06-py3	Mixed	64	COCO 2017	Tesla T4
	Tacotron2	CNN	14538 total output mels/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.10-py3	Mixed	128	LJ Speech 1.1	Tesla T4
		CNN	103679 total output mels/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.09-py3	Mixed	128	LJ Speech 1.1	Tesla T4
	WaveGlow	CNN	33271 output samples/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.10-py3	Mixed	10	LJ Speech 1.1	Tesla T4
		CNN	247439 output samples/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.10-py3	Mixed	10	LJ Speech 1.1	Tesla T4

TensorFlow	Inception V3	CNN	177 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.10-py3	Mixed	192	ImageNet2012	Tesla T4
		CNN	1334 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.10-py3	Mixed	192	ImageNet2012	Tesla T4
	ResNet-50 V1.5	CNN	265 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.10-py3	Mixed	256	ImageNet2012	Tesla T4
		CNN	2121 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.06-py3	Mixed	256	ImageNet2012	Tesla T4
	SSD v1.1	CNN	51 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.10-py3	Mixed	32	COCO 2017	Tesla T4
		CNN	279 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.10-py3	Mixed	32	COCO 2017	Tesla T4
	U-Net Industrial	CNN	28 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.10-py3	Mixed	16	DAGM2007	Tesla T4
		CNN	191 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.10-py3	Mixed	2	DAGM2007	Tesla T4

PyTorch	GNMT V2	RNN	26083 total tokens/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.08-py3	Mixed	256	WMT16 English-German	Tesla T4
		RNN	181049 total tokens/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.08-py3	Mixed	256	WMT16 English-German	Tesla T4

TensorFlow	GNMT V2	RNN	11771 total tokens/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.09-py3	Mixed	192	WMT16 English-German	Tesla T4
		RNN	58338 total tokens/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.10-py3	Mixed	128	WMT16 English-German	Tesla T4

PyTorch	NCF	Recommender	7502907 samples/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.08-py3	Mixed	1048576	MovieLens 20 Million	Tesla T4
		Recommender	26544719 samples/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.08-py3	Mixed	1048576	MovieLens 20 Million	Tesla T4

TensorFlow	NCF	Recommender	10025545 samples/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.08-py3	Mixed	1048576	MovieLens 20 Million	Tesla T4
		Recommender	19050484 samples/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.06-py3	Mixed	1048576	MovieLens 20 Million	Tesla T4

TensorFlow	BERT	Attention	9 sentences/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.09-py3	Mixed	3	SQuaD v1.1	Tesla T4
		Attention	39 sentences/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.10-py3	Mixed	3	SQuaD v1.1	Tesla T4

NVIDIA® TensorRT™ running on NVIDIA GPUs enable the most efficient deep learning inference performance across multiple application areas and models. This versatility provides wide latitude to data scientists to create the optimal low-latency solution. Visit NVIDIA GPU Cloud (NGC) to download any of these containers.

NVIDIA Tesla® V100 Tensor Cores GPUs leverage mixed-precision to combine high throughput with low latencies across every type of neural network. Tesla P4 is an inference GPU, designed for optimal power consumption and latency, for ultra-efficient scale-out servers. Read the inference whitepaper to learn more about NVIDIA’s inference platform.

Measuring the inference performance involves balancing a lot of variables. PLASTER is an acronym that describes the key elements for measuring deep learning performance. Each letter identifies a factor (Programmability, Latency, Accuracy, Size of Model, Throughput, Energy Efficiency, Rate of Learning) that must be considered to arrive at the right set of tradeoffs and to produce a successful deep learning implementation. Refer to NVIDIA’s PLASTER whitepaper for more details.

Inference Image Classification on CNNs with TensorRT

ResNet-50 v1.5 Throughput

ResNet-50 v1.5 Latency

ResNet-50 v1.5 Power Efficiency

Inference Performance

V100 Inference Performance

Network	Network Type	Batch Size	Throughput	Efficiency	Latency	GPU	Server	Container	Precision	Dataset	GPU Version
GoogleNet	CNN	1	1610 images/sec	15 images/sec/watt	0.62	1x V100	DGX-1	19.08-py3	INT8	Synthetic	V100-SXM2-16GB
	CNN	2	2162 images/sec	18 images/sec/watt	0.93	1x V100	DGX-1	19.07-py3	INT8	Synthetic	V100-SXM2-16GB
	CNN	8	5368 images/sec	35 images/sec/watt	1.5	1x V100	DGX-2	19.09-py3	INT8	Synthetic	V100-SXM3-32GB
	CNN	82	11869 images/sec	45 images/sec/watt	6.9	1x V100	DGX-1	19.07-py3	INT8	Synthetic	V100-SXM2-16GB
	CNN	128	12697 images/sec	47 images/sec/watt	10	1x V100	DGX-2	19.09-py3	INT8	Synthetic	V100-SXM3-32GB

MobileNet V1	CNN	1	4543 images/sec	30 images/sec/watt	0.22	1x V100	DGX-2	19.09-py3	INT8	Synthetic	V100-SXM3-32GB
	CNN	2	6426 images/sec	47 images/sec/watt	0.31	1x V100	DGX-1	19.09-py3	INT8	Synthetic	V100-SXM2-16GB
	CNN	8	14788 images/sec	96 images/sec/watt	0.54	1x V100	DGX-1	19.07-py3	INT8	Synthetic	V100-SXM2-32GB
	CNN	128	29914 images/sec	104 images/sec/watt	4.3	1x V100	DGX-1	19.06-py3	INT8	Synthetic	V100-SXM2-16GB

ResNet-50	CNN	1	1156 images/sec	8.7 images/sec/watt	0.87	1x V100	DGX-1	19.07-py3	INT8	Synthetic	V100-SXM2-16GB
	CNN	2	1612 images/sec	10 images/sec/watt	1.2	1x V100	DGX-2	19.08-py3	INT8	Synthetic	V100-SXM3-32GB
	CNN	8	3315 images/sec	21 images/sec/watt	2.4	1x V100	DGX-1	19.07-py3	Mixed	Synthetic	V100-SXM2-16GB
	CNN	128	7720 images/sec	27 images/sec/watt	17	1x V100	DGX-1	19.06-py3	Mixed	Synthetic	V100-SXM2-16GB
	CNN	128	7907 images/sec	23 images/sec/watt	16	1x V100	DGX-2	19.09-py3	Mixed	Synthetic	V100-SXM3-32GB

ResNet-50v1.5	CNN	1	955 images/sec	7.6 images/sec/watt	1.1	1x V100	DGX-1	19.09-py3	INT8	Synthetic	V100-SXM2-16GB
	CNN	2	1407 images/sec	9.8 images/sec/watt	1.4	1x V100	DGX-1	19.06-py3	INT8	Synthetic	V100-SXM2-16GB
	CNN	8	3226 images/sec	20 images/sec/watt	2.5	1x V100	DGX-1	19.08-py3	Mixed	Synthetic	V100-SXM2-16GB
	CNN	128	7226 images/sec	25 images/sec/watt	18	1x V100	DGX-1	19.09-py3	Mixed	Synthetic	V100-SXM2-16GB
	CNN	128	7517 images/sec	22 images/sec/watt	17	1x V100	DGX-2	19.09-py3	Mixed	Synthetic	V100-SXM3-32GB

VGG16	CNN	1	821 images/sec	4 images/sec/watt	1.2	1x V100	DGX-1	19.07-py3	INT8	Synthetic	V100-SXM2-16GB
	CNN	2	1145 images/sec	5.5 images/sec/watt	1.8	1x V100	DGX-1	19.06-py3	Mixed	Synthetic	V100-SXM2-16GB
	CNN	8	2067 images/sec	8.2 images/sec/watt	3.9	1x V100	DGX-1	19.06-py3	Mixed	Synthetic	V100-SXM2-16GB
	CNN	128	2845 images/sec	9.7 images/sec/watt	45	1x V100	DGX-1	19.07-py3	Mixed	Synthetic	V100-SXM2-16GB

NMT	RNN	1	4013 total tokens/sec	tokens/sec/watt	13	1x V100	DGX-1	-	Mixed	wmt16-English-German	V100-SXM2-32GB
	RNN	2	6290 total tokens/sec	tokens/sec/watt	16	1x V100	DGX-1	-	Mixed	wmt16-English-German	V100-SXM2-32GB
	RNN	64	56531 total tokens/sec	tokens/sec/watt	58	1x V100	DGX-1	-	Mixed	wmt16-English-German	V100-SXM2-32GB
	RNN	128	73375 total tokens/sec	tokens/sec/watt	89	1x V100	DGX-1	-	Mixed	wmt16-English-German	V100-SXM2-32GB

NCF	Recommender	1048576	61130538 samples/sec	samples/sec/watt		1x V100	DGX-1	19.08-py3	Mixed	MovieLens 20 Million	V100-SXM2-16GB

BERT-BASE	Attention	1	557 sentences/sec	10.3 sentences/sec/watt	1.8	1x V100	Supermicro SYS-4029GP-TRT	-	Mixed	SQuAD v1.1	V100-PCIE-16GB
	Attention	2	978 sentences/sec	18.8 sentences/sec/watt	2	1x V100	Supermicro SYS-4029GP-TRT	-	Mixed	SQuAD v1.1	V100-PCIE-16GB
	Attention	8	1847 sentences/sec	34.1 sentences/sec/watt	4.3	1x V100	Supermicro SYS-4029GP-TRT	-	Mixed	SQuAD v1.1	V100-PCIE-16GB
	Attention	24	2419 sentences/sec	43.7 sentences/sec/watt	9.9	1x V100	Supermicro SYS-4029GP-TRT	-	Mixed	SQuAD v1.1	V100-PCIE-16GB
	Attention	128	2645 sentences/sec	46 sentences/sec/watt	48.4	1x V100	Supermicro SYS-4029GP-TRT	-	Mixed	SQuAD v1.1	V100-PCIE-16GB

BERT-LARGE	Attention	1	239 sentences/sec	4.3 sentences/sec/watt	4.2	1x V100	Supermicro SYS-4029GP-TRT	-	Mixed	SQuAD v1.1	V100-PCIE-16GB
	Attention	2	407 sentences/sec	7.5 sentences/sec/watt	4.9	1x V100	Supermicro SYS-4029GP-TRT	-	Mixed	SQuAD v1.1	V100-PCIE-16GB
	Attention	4	562 sentences/sec	10.6 sentences/sec/watt	7.1	1x V101	Supermicro SYS-4029GP-TRT	-	Mixed	SQuAD v1.1	V100-PCIE-16GB
	Attention	8	636 sentences/sec	11.8 sentences/sec/watt	12.6	1x V100	Supermicro SYS-4029GP-TRT	-	Mixed	SQuAD v1.1	V100-PCIE-16GB
	Attention	128	823 sentences/sec	13.6 sentences/sec/watt	155.5	1x V100	Supermicro SYS-4029GP-TRT	-	Mixed	SQuAD v1.1	V100-PCIE-16GB

TensorRT 6.0 and sequence length=128 for BERT-BASE and BERT-LARGE | PyTorch for NCF | TensorRT 5.1 and TensorRT 6.0 for all other models | Efficiency based on board power

T4 Inference Performance

Network	Network Type	Batch Size	Throughput	Efficiency	Latency	GPU	Server	Container	Precision	Dataset	GPU Version
GoogleNet	CNN	1	1745 images/sec	28 images/sec/watt	0.58	1x T4	Supermicro SYS-4029GP-TRT T4	19.09-py3	INT8	Synthetic	Tesla T4
	CNN	2	2409 images/sec	38 images/sec/watt	0.83	1x T4	Supermicro SYS-4029GP-TRT T4	19.09-py3	INT8	Synthetic	Tesla T4
	CNN	8	6282 images/sec	91 images/sec/watt	1.3	1x T4	Supermicro SYS-4029GP-TRT T4	19.09-py3	INT8	Synthetic	Tesla T4
	CNN	52	7580 images/sec	109 images/sec/watt	6.9	1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	INT8	Synthetic	Tesla T4
	CNN	128	8968 images/sec	128 images/sec/watt	14	1x T4	Supermicro SYS-4029GP-TRT T4	19.09-py3	INT8	Synthetic	Tesla T4

MobileNet V1	CNN	1	4442 images/sec	80 images/sec/watt	0.23	1x T4	Supermicro SYS-4029GP-TRT T4	19.09-py3	INT8	Synthetic	Tesla T4
	CNN	2	8057 images/sec	129 images/sec/watt	0.25	1x T4	Supermicro SYS-4029GP-TRT T4	19.09-py3	INT8	Synthetic	Tesla T4
	CNN	8	14660 images/sec	210 images/sec/watt	0.55	1x T4	Supermicro SYS-4029GP-TRT T4	19.09-py3	INT8	Synthetic	Tesla T4
	CNN	128	17768 images/sec	254 images/sec/watt	7.2	1x T4	Supermicro SYS-4029GP-TRT T4	19.09-py3	INT8	Synthetic	Tesla T4

ResNet-50	CNN	1	1162 images/sec	17 images/sec/watt	0.86	1x T4	Supermicro SYS-4029GP-TRT T4	19.09-py3	INT8	Synthetic	Tesla T4
	CNN	2	1804 images/sec	26 images/sec/watt	1.1	1x T4	Supermicro SYS-4029GP-TRT T4	19.09-py3	INT8	Synthetic	Tesla T4
	CNN	8	4060 images/sec	58 images/sec/watt	2	1x T4	Supermicro SYS-4029GP-TRT T4	19.09-py3	INT8	Synthetic	Tesla T4
	CNN	128	5681 images/sec	81 images/sec/watt	23	1x T4	Supermicro SYS-4029GP-TRT T4	19.09-py3	INT8	Synthetic	Tesla T4

ResNet-50v1.5	CNN	1	1109 images/sec	16 images/sec/watt	0.9	1x T4	Supermicro SYS-4029GP-TRT T4	19.09-py3	INT8	Synthetic	Tesla T4
	CNN	2	1730 images/sec	26 images/sec/watt	1.2	1x T4	Supermicro SYS-4029GP-TRT T4	19.08-py3	INT8	Synthetic	Tesla T4
	CNN	8	3922 images/sec	56 images/sec/watt	2	1x T4	Supermicro SYS-4029GP-TRT T4	19.09-py3	INT8	Synthetic	Tesla T4
	CNN	128	5344 images/sec	77 images/sec/watt	24	1x T4	Supermicro SYS-4029GP-TRT T4	19.09-py3	INT8	Synthetic	Tesla T4

VGG16	CNN	1	792 images/sec	11 images/sec/watt	1.3	1x T4	Supermicro SYS-4029GP-TRT T4	19.09-py3	INT8	Synthetic	Tesla T4
	CNN	2	1083 images/sec	16 images/sec/watt	1.9	1x T4	Supermicro SYS-4029GP-TRT T4	19.09-py3	INT8	Synthetic	Tesla T4
	CNN	8	1670 images/sec	24 images/sec/watt	4.8	1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	INT8	Synthetic	Tesla T4
	CNN	128	1956 images/sec	28 images/sec/watt	65	1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	INT8	Synthetic	Tesla T4

NCF	Recommender	1	12394 samples/sec	442 samples/sec/watt	0.08	1x T4	Supermicro SYS-4029GP-TRT T4	19.09-py3	Mixed	Synthetic	Tesla T4
	Recommender	64	697677 samples/sec	23933 samples/sec/watt	0.09	1x T4	Supermicro SYS-4029GP-TRT T4	19.09-py3	Mixed	Synthetic	Tesla T4
	Recommender	25000	51102744 samples/sec	731187 samples/sec/watt	0.49	1x T4	Supermicro SYS-4029GP-TRT T4	19.09-py3	INT8	Synthetic	Tesla T4
	Recommender	100000	55301530 samples/sec	795155 samples/sec/watt	1.8	1x T4	Supermicro SYS-4029GP-TRT T4	19.09-py3	INT8	Synthetic	Tesla T4

BERT-BASE	Attention	1	484 sentences/sec	11 sentences/sec/watt	2.07	1x T4	Supermicro SYS-4029GP-TRT T4	-	Mixed	SQuAD v1.1	Tesla T4
	Attention	2	754 sentences/sec	17 sentences/sec/watt	2.65	1x T4	Supermicro SYS-4029GP-TRT T4	-	Mixed	SQuAD v1.1	Tesla T4
	Attention	8	827 sentences/sec	20 sentences/sec/watt	9.67	1x T4	Supermicro SYS-4029GP-TRT T4	-	Mixed	SQuAD v1.1	Tesla T4
	Attention	128	800 sentences/sec	16 sentences/sec/watt	160.02	1x T4	Supermicro SYS-4029GP-TRT T4	-	Mixed	SQuAD v1.1	Tesla T4

BERT-LARGE	Attention	1	171 sentences/sec	4 sentences/sec/watt	5.84	1x T4	Supermicro SYS-4029GP-TRT T4	-	Mixed	SQuAD v1.1	Tesla T4
	Attention	2	168 sentences/sec	4 sentences/sec/watt	11.88	1x T4	Supermicro SYS-4029GP-TRT T4	-	Mixed	SQuAD v1.1	Tesla T4
	Attention	8	244 sentences/sec	6 sentences/sec/watt	32.74	1x T4	Supermicro SYS-4029GP-TRT T4	-	Mixed	SQuAD v1.1	Tesla T4
	Attention	128	254 sentences/sec	5 sentences/sec/watt	504.33	1x T4	Supermicro SYS-4029GP-TRT T4	-	Mixed	SQuAD v1.1	Tesla T4

TensorRT 6.0 and sequence length=128 for BERT-BASE and BERT-LARGE | TensorRT 5.1 and TensorRT 6.0 for all other models | Efficiency based on board power | NCF uses facebook dataset

Last updated: Nov 6th, 2019

Search form

NVIDIA Tesla Deep Learning Product Performance

Training

Inference

NVIDIA Performance on MLPerf 0.6 AI Benchmarks

Training Image Classification on CNNs

Training Performance

NVIDIA Performance on MLPerf 0.6 AI Benchmarks

V100 Training Performance

T4 Training Performance

Inference Image Classification on CNNs with TensorRT

Inference Performance

V100 Inference Performance

T4 Inference Performance