NVIDIA Tesla Deep Learning Product Performance

Review the latest GPU acceleration factors of popular HPC applications.

Training
Inference

NVIDIA’s complete solution stack, from GPUs to libraries, and containers on NVIDIA GPU Cloud (NGC), allows data scientists to quickly get up and running with deep learning. NVIDIA® Tesla® V100 Tensor Core GPUs leverage mixed precision to accelerate deep learning training throughputs across every framework and every type of neural network. NVIDIA breaks performance records on MLPerf, the AI’s first industry-wide benchmark, a testament to our GPU-accelerated platform approach.

NVIDIA Performance on MLPerf 0.6 AI Benchmarks

ResNet-50 v1.5 Time to Solution on V100

MXNet | Batch Size refer to CNN V100 Training table below | Precision: Mixed | Dataset: ImageNet2012 | Convergence criteria - refer to MLPerf requirements

Training Image Classification on CNNs

ResNet-50 V1.5 Throughput on V100

DGX-1: 8x Tesla V100-SXM2-32GB for MXNet, PyTorch and TensorFlow use V100-SXM2-16GB, E5-2698 v4 2.2 GHz | Batch Size = 256 | MXNet = 19.06-py3, PyTorch and Tensorflow = 19.07_py3 | Precision: Mixed | Dataset: ImageNet2012

ResNet-50 V1.5 Throughput on T4

Supermicro SYS-4029GP-TRT T4: 8x Tesla T4, Gold 6140 2.3 GHz | Batch Size = 208 for MXNet, PyTorch and TensorFlow = 256 | MXNet = 19.05-py3, PyTorch = 19.07-py3, TensorFlow = 19.06-py3 | Precision: Mixed | Dataset: ImageNet2012

Training Performance

NVIDIA Performance on MLPerf 0.6 AI Benchmarks

Framework	Network	Network Type	Time to Solution	GPU	Server	MLPerf-ID	Precision	Dataset	GPU Version
MXNet	ResNet-50 v1.5	CNN	115.22 minutes	8x V100	DGX-1	0.6-8	Mixed	ImageNet2012	V100-SXM2-16GB
		CNN	57.87 minutes	16x V100	DGX-2	0.6-17	Mixed	ImageNet2012	V100-SXM3-32GB
		CNN	52.74 minutes	16x V100	DGX-2H	0.6-19	Mixed	ImageNet2012	V100-SXM3-32GB-H
		CNN	2.59 minutes	512x V100	DGX-2H	0.6-29	Mixed	ImageNet2012	V100-SXM3-32GB-H
		CNN	1.69 minutes	1040x V100	DGX-1	0.6-16	Mixed	ImageNet2012	V100-SXM2-16GB
		CNN	1.33 minutes	1536x V100	DGX-2H	0.6-30	Mixed	ImageNet2012	V100-SXM3-32GB-H

PyTorch	SSD-ResNet-34	CNN	22.36 minutes	8x V100	DGX-1	0.6-9	Mixed	COCO2017	V100-SXM2-16GB
		CNN	12.21 minutes	16x V100	DGX-2	0.6-18	Mixed	COCO2017	V100-SXM3-32GB
		CNN	11.41 minutes	16x V100	DGX-2H	0.6-20	Mixed	COCO2017	V100-SXM3-32GB-H
		CNN	4.78 minutes	64x V100	DGX-2H	0.6-21	Mixed	COCO2017	V100-SXM3-32GB-H
		CNN	2.67 minutes	240x V100	DGX-1	0.6-13	Mixed	COCO2017	V100-SXM2-16GB
		CNN	2.56 minutes	240x V100	DGX-2H	0.6-24	Mixed	COCO2017	V100-SXM3-32GB-H
		CNN	2.23 minutes	240x V100	DGX-2H	0.6-27	Mixed	COCO2017	V100-SXM3-32GB-H
	Mask R-CNN	CNN	207.48 minutes	8x V100	DGX-1	0.6-9	Mixed	COCO2017	V100-SXM2-16GB
		CNN	101 minutes	16x V100	DGX-2	0.6-18	Mixed	COCO2017	V100-SXM3-32GB
		CNN	95.2 minutes	16x V100	DGX-2H	0.6-20	Mixed	COCO2017	V100-SXM3-32GB-H
		CNN	32.72 minutes	64x V100	DGX-2H	0.6-21	Mixed	COCO2017	V100-SXM3-32GB-H
		CNN	22.03 minutes	192x V100	DGX-1	0.6-12	Mixed	COCO2017	V100-SXM2-16GB
		CNN	18.47 minutes	192x V100	DGX-2H	0.6-23	Mixed	COCO2017	V100-SXM3-32GB-H

PyTorch	GNMT	RNN	20.55 minutes	8x V100	DGX-1	0.6-9	Mixed	WMT16 English-German	V100-SXM2-16GB
		RNN	10.94 minutes	16x V100	DGX-2	0.6-18	Mixed	WMT16 English-German	V100-SXM3-32GB
		RNN	9.87 minutes	16x V100	DGX-2H	0.6-20	Mixed	WMT16 English-German	V100-SXM3-32GB-H
		RNN	2.12 minutes	256x V100	DGX-2H	0.6-25	Mixed	WMT16 English-German	V100-SXM3-32GB-H
		RNN	1.99 minutes	384x V100	DGX-1	0.6-14	Mixed	WMT16 English-German	V100-SXM2-16GB
		RNN	1.8 minutes	384x V100	DGX-2H	0.6-26	Mixed	WMT16 English-German	V100-SXM3-32GB-H

PyTorch	Transformer	Attention	20.34 minutes	8x V100	DGX-1	0.6-9	Mixed	WMT17 English-German	V100-SXM2-16GB
		Attention	11.04 minutes	16x V100	DGX-2	0.6-18	Mixed	WMT17 English-German	V100-SXM3-32GB
		Attention	9.8 minutes	16x V100	DGX-2H	0.6-20	Mixed	WMT17 English-German	V100-SXM3-32GB-H
		Attention	2.41 minutes	160x V100	DGX-2H	0.6-22	Mixed	WMT17 English-German	V100-SXM3-32GB-H
		Attention	2.05 minutes	480x V100	DGX-1	0.6-15	Mixed	WMT17 English-German	V100-SXM2-16GB
		Attention	1.59 minutes	480x V100	DGX-2H	0.6-28	Mixed	WMT17 English-German	V100-SXM3-32GB-H

TensorFlow	MiniGo	Reinforcement Learning	27.39 minutes	8x V100	DGX-1	0.6-10	Mixed	N/A	V100-SXM2-16GB
		Reinforcement Learning	13.57 minutes	24x V100	DGX-1	0.6-11	Mixed	N/A	V100-SXM2-16GB

V100 Training Performance

Framework	Network	Network Type	Throughput	GPU	Server	Container	Precision	Batch Size	Dataset	GPU Version
MXNet	Inception V3	CNN	547 images/sec	1x V100	DGX-1	19.08-py3	Mixed	128	ImageNet2012	V100-SXM2-16GB
		CNN	607 images/sec	1x V100	DGX-2H	19.08-py3	Mixed	256	ImageNet2012	V100-SXM3-32GB-H
		CNN	4216 images/sec	8x V100	DGX-1	19.08-py3	Mixed	128	ImageNet2012	V100-SXM2-16GB
		CNN	4652 images/sec	8x V100	DGX-2H	19.08-py3	Mixed	256	ImageNet2012	V100-SXM3-32GB-H
	ResNet-50	CNN	1409 images/sec	1x V100	DGX-1	19.02-py3	Mixed	128	ImageNet2012	V100-SXM2-16GB
		CNN	1442 images/sec	1x V100	DGX-2	19.02-py3	Mixed	256	ImageNet2012	V100-SXM3-32GB
		CNN	10380 images/sec	8x V100	DGX-1	19.02-py3	Mixed	128	ImageNet2012	V100-SXM2-16GB
		CNN	10530 images/sec	8x V100	DGX-2	19.02-py3	Mixed	256	ImageNet2012	V100-SXM3-32GB
	ResNet-50 v1.5	CNN	1437 images/sec	1x V100	DGX-1	19.08-py3	Mixed	208	ImageNet2012	V100-SXM2-16GB
		CNN	1604 images/sec	1x V100	DGX-2H	19.08-py3	Mixed	256	ImageNet2012	V100-SXM3-32GB-H
		CNN	9566 images/sec	8x V100	DGX-1	19.06-py3	Mixed	256	ImageNet2012	V100-SXM2-32GB
		CNN	11056 images/sec	8x V100	DGX-2	19.05-py3	Mixed	128	ImageNet2012	V100-SXM3-32GB
		CNN	11507 images/sec	8x V100	DGX-2H	19.05-py3	Mixed	256	ImageNet2012	V100-SXM3-32GB-H

PyTorch	Inception V3	CNN	569 images/sec	1x V100	DGX-1	19.08-py3	Mixed	128	ImageNet2012	V100-SXM2-32GB
		CNN	632 images/sec	1x V100	DGX-2H	19.08-py3	Mixed	256	ImageNet2012	V100-SXM3-32GB-H
		CNN	4269 images/sec	8x V100	DGX-1	19.08-py3	Mixed	128	ImageNet2012	V100-SXM2-16GB
	Mask R-CNN	CNN	14 images/sec	1x V100	DGX-1	19.07-py3	Mixed	4	COCO2014	V100-SXM2-32GB
		CNN	17 images/sec	1x V100	DGX-2H	19.05-py3	Mixed	16	COCO2014	V100-SXM3-32GB-H
		CNN	93 images/sec	8x V100	DGX-1	19.08-py3	Mixed	16	COCO2014	V100-SXM2-32GB
	ResNet-50	CNN	819 images/sec	1x V100	DGX-1	19.02_py3	Mixed	256	ImageNet2012	V100-SXM2-16GB
		CNN	820 images/sec	1x V100	DGX-2	19.02-py3	Mixed	256	ImageNet2012	V100-SXM3-32GB
		CNN	6218 images/sec	8x V100	DGX-1	19.02-py3	Mixed	256	ImageNet2012	V100-SXM2-16GB
	ResNet-50 v1.5	CNN	928 images/sec	1x V100	DGX-1	19.07-py3	Mixed	256	ImageNet2012	V100-SXM2-16GB
		CNN	1036 images/sec	1x V100	DGX-2H	19.07-py3	Mixed	256	ImageNet2012	V100-SXM3-32GB-H
		CNN	7288 images/sec	8x V100	DGX-1	19.07-py3	Mixed	256	ImageNet2012	V100-SXM2-16GB
	SSD v1.1	CNN	237 images/sec	1x V100	DGX-1	19.08-py3	Mixed	64	COCO 2017	V100-SXM2-16GB
		CNN	299 images/sec	1x V100	DGX-2H	19.05-py3	Mixed	64	COCO 2017	V100-SXM3-32GB-H
		CNN	2132 images/sec	8x V100	DGX-1	19.06-py3	Mixed	64	COCO 2017	V100-SXM2-16GB
	Tacotron2	CNN	19931 total input tokens/sec	1x V100	DGX-1	19.08-py3	Mixed	128	LJ Speech 1.1	V100-SXM2-32GB
		CNN	24590 total input tokens/sec	1x V100	DGX-2H	19.08-py3	Mixed	128	LJ Speech 1.1	V100-SXM3-32GB-H
		CNN	110370 total input tokens/sec	8x V100	DGX-1	19.08-py3	Mixed	128	LJ Speech 1.1	V100-SXM2-32GB
		CNN	144330 total input tokens/sec	8x V100	DGX-2H	19.08-py3	Mixed	128	LJ Speech 1.1	V100-SXM3-32GB-H
	WaveGlow	CNN	82692 output samples/sec	1x V100	DGX-1	19.08-py3	Mixed	10	LJ Speech 1.1	V100-SXM2-16GB
		CNN	96370 output samples/sec	1x V100	DGX-2H	19.08-py3	Mixed	10	LJ Speech 1.1	V100-SXM3-32GB-H
		CNN	594479 output samples/sec	8x V100	DGX-1	19.08-py3	Mixed	10	LJ Speech 1.1	V100-SXM2-16GB

TensorFlow	Inception V3	CNN	541 images/sec	1x V100	DGX-1	19.08-py3	Mixed	256	ImageNet2012	V100-SXM2-32GB
		CNN	627 images/sec	1x V100	DGX-2H	19.08-py3	Mixed	256	ImageNet2012	V100-SXM3-32GB-H
		CNN	4041 images/sec	8x V100	DGX-1	19.08-py3	Mixed	256	ImageNet2012	V100-SXM2-32GB
	ResNet-50 V1.5	CNN	841 images/sec	1x V100	DGX-1	19.08-py3	Mixed	256	ImageNet2012	V100-SXM2-16GB
		CNN	964 images/sec	1x V100	DGX-2H	19.08-py3	Mixed	256	ImageNet2012	V100-SXM3-32GB-H
		CNN	6474 images/sec	8x V100	DGX-1	19.07-py3	Mixed	256	ImageNet2012	V100-SXM2-16GB
	SSD v1.1	CNN	115 images/sec	1x V100	DGX-1	19.08-py3	Mixed	32	COCO 2017	V100-SXM2-32GB
		CNN	124 images/sec	1x V100	DGX-2	19.08-py3	Mixed	32	COCO 2017	V100-SXM3-32GB
		CNN	672 images/sec	8x V100	DGX-1	19.06-py3	Mixed	32	COCO 2017	V100-SXM2-16GB
		CNN	770 images/sec	8x V100	DGX-2	19.05-py3	Mixed	32	COCO 2017	V100-SXM3-32GB
	U-Net Industrial	CNN	104 images/sec	1x V100	DGX-1	19.06-py3	Mixed	16	DAGM2007	V100-SXM2-16GB
		CNN	116 images/sec	1x V100	DGX-2H	19.06-py3	Mixed	16	DAGM2007	V100-SXM3-32GB-H
		CNN	503 images/sec	8x V100	DGX-1	19.07-py3	Mixed	2	DAGM2007	V100-SXM2-16GB
		CNN	544 images/sec	8x V100	DGX-2H	19.08-py3	Mixed	2	DAGM2007	V100-SXM3-32GB-H

PyTorch	GNMT V2	RNN	76722 total tokens/sec	1x V100	DGX-1	19.08-py3	Mixed	128	WMT16 English-German	V100-SXM2-32GB
		RNN	92364 total tokens/sec	1x V100	DGX-2H	19.08-py3	Mixed	128	WMT16 English-German	V100-SXM3-32GB-H
		RNN	585249 total tokens/sec	8x V100	DGX-1	19.08-py3	Mixed	128	WMT16 English-German	V100-SXM2-32GB
		RNN	663657 total tokens/sec	8x V100	DGX-2H	19.08-py3	Mixed	128	WMT16 English-German	V100-SXM3-32GB-H

TensorFlow	GNMT V2	RNN	22471 total tokens/sec	1x V100	DGX-1	19.07-py3	Mixed	192	WMT16 English-German	V100-SXM2-16GB
		RNN	26039 total tokens/sec	1x V100	DGX-2H	19.07-py3	Mixed	192	WMT16 English-German	V100-SXM3-32GB-H
		RNN	149008 total tokens/sec	8x V100	DGX-1	19.07-py3	Mixed	192	WMT16 English-German	V100-SXM2-16GB

PyTorch	NCF	Recommender	22093850 samples/sec	1x V100	DGX-1	19.07-py3	Mixed	1048576	MovieLens 20 Million	V100-SXM2-16GB
		Recommender	24473776 samples/sec	1x V100	DGX-2H	19.07-py3	Mixed	1048576	MovieLens 20 Million	V100-SXM3-32GB-H
		Recommender	104122673 samples/sec	8x V100	DGX-1	19.07-py3	Mixed	1048576	MovieLens 20 Million	V100-SXM2-16GB
		Recommender	109969915 samples/sec	8x V100	DGX-2H	19.07-py3	Mixed	1048576	MovieLens 20 Million	V100-SXM3-32GB-H

TensorFlow	NCF	Recommender	26415693 samples/sec	1x V100	DGX-1	19.07-py3	Mixed	1048576	MovieLens 20 Million	V100-SXM2-16GB
		Recommender	56991307 samples/sec	8x V100	DGX-1	19.08-py3	Mixed	1048576	MovieLens 20 Million	V100-SXM2-32GB

PyTorch	BERT-LARGE	Attention	50 sentences/sec	1x V100	DGX-1	19.09-py3	Mixed	10	SQuaD v1.1	V100-SXM2-32GB
		Attention	353 sentences/sec	8xV100	DGX-2	19.09-py3	Mixed	10	SQuaD v1.1	V100-SXM2-32GB

TensorFlow	BERT-LARGE	Attention	32 sentences/sec	1x V100	DGX-1	19.07-py3	Mixed	10	SQuaD v1.1	V100-SXM2-32GB
		Attention	37 sentences/sec	1x V100	DGX-2H	19.08-py3	Mixed	10	SQuaD v1.1	V100-SXM3-32GB-H
		Attention	158 sentences/sec	8xV100	DGX-1	19.08-py3	Mixed	10	SQuaD v1.1	V100-SXM2-32GB
		Attention	189 sentences/sec	8x V100	DGX-2H	19.07-py3	Mixed	10	SQuaD v1.1	V100-SXM3-32GB-H

T4 Training Performance

Framework	Network	Network Type	Throughput	GPU	Server	Container	Precision	Batch Size	Dataset	GPU Version
MXNet	Inception V3	CNN	180 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.08-py3	Mixed	128	ImageNet2012	Tesla T4
		CNN	1391 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.06-py3	Mixed	128	ImageNet2012	Tesla T4
	ResNet-50 v1.5	CNN	446 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	Mixed	208	ImageNet2012	Tesla T4
		CNN	4116 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.05-py3	Mixed	208	ImageNet2012	Tesla T4

PyTorch	Inception V3	CNN	185 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.08-py3	Mixed	128	ImageNet2012	Tesla T4
		CNN	1386 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.08-py3	Mixed	128	ImageNet2012	Tesla T4
	Mask R-CNN	CNN	7 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.08-py3	Mixed	4	COCO2014	Tesla T4
		CNN	40 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.08-py3	Mixed	4	COCO2014	Tesla T4
	ResNet-50 v1.5	CNN	287 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.09-py3	Mixed	256	ImageNet2012	Tesla T4
		CNN	2307 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.09-py3	Mixed	256	ImageNet2012	Tesla T4
	SSD v1.1	CNN	85 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.06-py3	Mixed	64	COCO 2017	Tesla T4
		CNN	691 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.06-py3	Mixed	64	COCO 2017	Tesla T4
	Tacotron2	CNN	14864 total input tokens/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.09-py3	Mixed	128	LJ Speech 1.1	Tesla T4
		CNN	103679 total input tokens/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.09-py3	Mixed	128	LJ Speech 1.1	Tesla T4
	WaveGlow	CNN	34577 output samples/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.08-py3	Mixed	10	LJ Speech 1.1	Tesla T4
		CNN	244716 output samples/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.08-py3	Mixed	10	LJ Speech 1.1	Tesla T4

TensorFlow	Inception V3	CNN	182 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.08-py3	Mixed	128	ImageNet2012	Tesla T4
		CNN	1336 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.08-py3	Mixed	128	ImageNet2012	Tesla T4
	ResNet-50 V1.5	CNN	274 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.08-py3	Mixed	256	ImageNet2012	Tesla T4
		CNN	2121 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.06-py3	Mixed	256	ImageNet2012	Tesla T4
	SSD v1.1	CNN	52 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.08-py3	Mixed	32	COCO 2017	Tesla T4
		CNN	280 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.08-py3	Mixed	32	COCO 2017	Tesla T4
	U-Net Industrial	CNN	29 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.08-py3	Mixed	16	DAGM2007	Tesla T4
		CNN	191 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.08-py3	Mixed	2	DAGM2007	Tesla T4

PyTorch	GNMT V2	RNN	26083 total tokens/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.08-py3	Mixed	128	WMT16 English-German	Tesla T4
		RNN	183225 total tokens/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.06-py3	Mixed	128	WMT16 English-German	Tesla T4

TensorFlow	GNMT V2	RNN	9862 total tokens/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.08-py3	Mixed	128	WMT16 English-German	Tesla T4
		RNN	58118 total tokens/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.08-py3	Mixed	128	WMT16 English-German	Tesla T4

PyTorch	NCF	Recommender	7584587 samples/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	Mixed	1048576	MovieLens 20 Million	Tesla T4
		Recommender	27011297 samples/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	Mixed	1048576	MovieLens 20 Million	Tesla T4

TensorFlow	NCF	Recommender	10297010 samples/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	Mixed	1048576	MovieLens 20 Million	Tesla T4
		Recommender	19050484 samples/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.06-py3	Mixed	1048576	MovieLens 20 Million	Tesla T4

TensorFlow	BERT	Attention	9 sentences/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.08-py3	Mixed	3	SQuaD v1.1	Tesla T4
		Attention	32 sentences/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.08-py3	Mixed	3	SQuaD v1.1	Tesla T4

NVIDIA® TensorRT™ running on NVIDIA GPUs enable the most efficient deep learning inference performance across multiple application areas and models. This versatility provides wide latitude to data scientists to create the optimal low-latency solution. Visit NVIDIA GPU Cloud (NGC) to download any of these containers.

NVIDIA Tesla® V100 Tensor Cores GPUs leverage mixed-precision to combine high throughput with low latencies across every type of neural network. Tesla P4 is an inference GPU, designed for optimal power consumption and latency, for ultra-efficient scale-out servers. Read the inference whitepaper to learn more about NVIDIA’s inference platform.

Measuring the inference performance involves balancing a lot of variables. PLASTER is an acronym that describes the key elements for measuring deep learning performance. Each letter identifies a factor (Programmability, Latency, Accuracy, Size of Model, Throughput, Energy Efficiency, Rate of Learning) that must be considered to arrive at the right set of tradeoffs and to produce a successful deep learning implementation. Refer to NVIDIA’s PLASTER whitepaper for more details.

Inference Image Classification on CNNs with TensorRT

ResNet-50 v1.5 Throughput

ResNet-50 v1.5 Latency

ResNet-50 v1.5 Power Efficiency

Inference Performance

V100 Inference Performance

Network	Network Type	Batch Size	Throughput	Efficiency	Latency	GPU	Server	Container	Precision	Dataset	GPU Version
GoogleNet	CNN	1	1610 images/sec	15 images/sec/watt	0.62	1x V100	DGX-1	19.08-py3	INT8	Synthetic	V100-SXM2-16GB
	CNN	2	2162 images/sec	18 images/sec/watt	0.93	1x V100	DGX-1	19.07-py3	INT8	Synthetic	V100-SXM2-16GB
	CNN	8	5228 images/sec	35 images/sec/watt	1.5	1x V100	DGX-1	19.06-py3	INT8	Synthetic	V100-SXM2-16GB
	CNN	82	11869 images/sec	45 images/sec/watt	6.9	1x V100	DGX-1	19.07-py3	INT8	Synthetic	V100-SXM2-16GB
	CNN	128	12400 images/sec	44 images/sec/watt	10	1x V100	DGX-1	19.07-py3	INT8	Synthetic	V100-SXM2-16GB

MobileNet V1	CNN	1	3814 images/sec	29 images/sec/watt	0.26	1x V100	DGX-1	19.07-py3	INT8	Synthetic	V100-SXM2-32GB
	CNN	2	5594 images/sec	45 images/sec/watt	0.36	1x V100	DGX-1	19.07-py3	INT8	Synthetic	V100-SXM2-16GB
	CNN	8	14788 images/sec	96 images/sec/watt	0.54	1x V100	DGX-1	19.07-py3	INT8	Synthetic	V100-SXM2-32GB
	CNN	128	29914 images/sec	104 images/sec/watt	4.3	1x V100	DGX-1	19.06-py3	INT8	Synthetic	V100-SXM2-16GB

ResNet-50	CNN	1	1156 images/sec	8.7 images/sec/watt	0.87	1x V100	DGX-1	19.07-py3	INT8	Synthetic	V100-SXM2-16GB
	CNN	2	1580 images/sec	10 images/sec/watt	1.3	1x V100	DGX-1	19.07-py3	INT8	Synthetic	V100-SXM2-16GB
	CNN	8	3315 images/sec	21 images/sec/watt	2.4	1x V100	DGX-1	19.07-py3	Mixed	Synthetic	V100-SXM2-16GB
	CNN	128	7720 images/sec	27 images/sec/watt	17	1x V100	DGX-1	19.06-py3	Mixed	Synthetic	V100-SXM2-16GB
	CNN	128	7830 images/sec	23 images/sec/watt	16	1x V100	DGX-2	19.06-py3	Mixed	Synthetic	V100-SXM3-32GB

ResNet-50v1.5	CNN	1	949 images/sec	7.1 images/sec/watt	1.1	1x V100	DGX-1	19.08-py3	INT8	Synthetic	V100-SXM2-16GB
	CNN	2	1407 images/sec	9.8 images/sec/watt	1.4	1x V100	DGX-1	19.06-py3	INT8	Synthetic	V100-SXM2-16GB
	CNN	8	3226 images/sec	20 images/sec/watt	2.5	1x V100	DGX-1	19.08-py3	Mixed	Synthetic	V100-SXM2-16GB
	CNN	128	7223 images/sec	25 images/sec/watt	18	1x V100	DGX-1	19.07-py3	Mixed	Synthetic	V100-SXM2-16GB
	CNN	128	7454 images/sec	22 images/sec/watt	17	1x V100	DGX-2	19.08-py3	Mixed	Synthetic	V100-SXM3-32GB

VGG16	CNN	1	821 images/sec	4 images/sec/watt	1.2	1x V100	DGX-1	19.07-py3	INT8	Synthetic	V100-SXM2-16GB
	CNN	2	1145 images/sec	5.5 images/sec/watt	1.8	1x V100	DGX-1	19.06-py3	Mixed	Synthetic	V100-SXM2-16GB
	CNN	8	2067 images/sec	8.2 images/sec/watt	3.9	1x V100	DGX-1	19.06-py3	Mixed	Synthetic	V100-SXM2-16GB
	CNN	128	2845 images/sec	9.7 images/sec/watt	45	1x V100	DGX-1	19.07-py3	Mixed	Synthetic	V100-SXM2-16GB

NMT	RNN	1	4013 total tokens/sec	tokens/sec/watt	13	1x V100	DGX-1	-	Mixed	wmt16-English-German	V100-SXM2-32GB
	RNN	2	6290 total tokens/sec	tokens/sec/watt	16	1x V100	DGX-1	-	Mixed	wmt16-English-German	V100-SXM2-32GB
	RNN	64	56531 total tokens/sec	tokens/sec/watt	58	1x V100	DGX-1	-	Mixed	wmt16-English-German	V100-SXM2-32GB
	RNN	128	73375 total tokens/sec	tokens/sec/watt	89	1x V100	DGX-1	-	Mixed	wmt16-English-German	V100-SXM2-32GB

NCF	Recommender	1048576	61130538 samples/sec	samples/sec/watt		1x V100	DGX-1	19.08-py3	Mixed	MovieLens 20 Million	V100-SXM2-16GB

BERT-BASE	Attention	1	557 sentences/sec	10.3 sentences/sec/watt	1.8	1x V100	Supermicro SYS-4029GP-TRT	-	Mixed	SQuAD v1.1	V100-PCIE-16GB
	Attention	2	978 sentences/sec	18.8 sentences/sec/watt	2	1x V100	Supermicro SYS-4029GP-TRT	-	Mixed	SQuAD v1.1	V100-PCIE-16GB
	Attention	8	1847 sentences/sec	34.1 sentences/sec/watt	4.3	1x V100	Supermicro SYS-4029GP-TRT	-	Mixed	SQuAD v1.1	V100-PCIE-16GB
	Attention	24	2419 sentences/sec	43.7 sentences/sec/watt	9.9	1x V100	Supermicro SYS-4029GP-TRT	-	Mixed	SQuAD v1.1	V100-PCIE-16GB
	Attention	128	2645 sentences/sec	46 sentences/sec/watt	48.4	1x V100	Supermicro SYS-4029GP-TRT	-	Mixed	SQuAD v1.1	V100-PCIE-16GB

BERT-LARGE	Attention	1	239 sentences/sec	4.3 sentences/sec/watt	4.2	1x V100	Supermicro SYS-4029GP-TRT	-	Mixed	SQuAD v1.1	V100-PCIE-16GB
	Attention	2	407 sentences/sec	7.5 sentences/sec/watt	4.9	1x V100	Supermicro SYS-4029GP-TRT	-	Mixed	SQuAD v1.1	V100-PCIE-16GB
	Attention	4	562 sentences/sec	10.6 sentences/sec/watt	7.1	1x V101	Supermicro SYS-4029GP-TRT	-	Mixed	SQuAD v1.1	V100-PCIE-16GB
	Attention	8	636 sentences/sec	11.8 sentences/sec/watt	12.6	1x V100	Supermicro SYS-4029GP-TRT	-	Mixed	SQuAD v1.1	V100-PCIE-16GB
	Attention	128	823 sentences/sec	13.6 sentences/sec/watt	155.5	1x V100	Supermicro SYS-4029GP-TRT	-	Mixed	SQuAD v1.1	V100-PCIE-16GB

TensorRT 6.0 and sequence length=128 for BERT-BASE and BERT-LARGE | PyTorch for NCF | TensorRT 5.1 for all other models | Efficiency based on board power

T4 Inference Performance

Network	Network Type	Batch Size	Throughput	Efficiency	Latency	GPU	Server	Container	Precision	Dataset	GPU Version
GoogleNet	CNN	1	1703 images/sec	26 images/sec/watt	0.59	1x T4	Supermicro SYS-4029GP-TRT T4	19.08-py3	INT8	Synthetic	Tesla T4
	CNN	2	2356 images/sec	35 images/sec/watt	0.85	1x T4	Supermicro SYS-4029GP-TRT T4	19.06-py3	INT8	Synthetic	Tesla T4
	CNN	8	5816 images/sec	84 images/sec/watt	1.4	1x T4	Supermicro SYS-4029GP-TRT T4	19.08-py3	INT8	Synthetic	Tesla T4
	CNN	52	7580 images/sec	109 images/sec/watt	6.9	1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	INT8	Synthetic	Tesla T4
	CNN	128	7745 images/sec	111 images/sec/watt	17	1x T4	Supermicro SYS-4029GP-TRT T4	19.08-py3	INT8	Synthetic	Tesla T4

MobileNet V1	CNN	1	3685 images/sec	70 images/sec/watt	0.27	1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	INT8	Synthetic	Tesla T4
	CNN	2	6006 images/sec	97 images/sec/watt	0.33	1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	INT8	Synthetic	Tesla T4
	CNN	8	13859 images/sec	199 images/sec/watt	0.58	1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	INT8	Synthetic	Tesla T4
	CNN	128	17508 images/sec	251 images/sec/watt	7.3	1x T4	Supermicro SYS-4029GP-TRT T4	19.08-py3	INT8	Synthetic	Tesla T4

ResNet-50	CNN	1	1080 images/sec	16 images/sec/watt	0.93	1x T4	Supermicro SYS-4029GP-TRT T4	19.08-py3	INT8	Synthetic	Tesla T4
	CNN	2	1731 images/sec	26 images/sec/watt	1.2	1x T4	Supermicro SYS-4029GP-TRT T4	19.08-py3	INT8	Synthetic	Tesla T4
	CNN	8	3907 images/sec	56 images/sec/watt	2.1	1x T4	Supermicro SYS-4029GP-TRT T4	19.08-py3	INT8	Synthetic	Tesla T4
	CNN	128	5402 images/sec	78 images/sec/watt	24	1x T4	Supermicro SYS-4029GP-TRT T4	19.08-py3	INT8	Synthetic	Tesla T4

ResNet-50v1.5	CNN	1	1032 images/sec	15 images/sec/watt	0.97	1x T4	Supermicro SYS-4029GP-TRT T4	19.08-py3	INT8	Synthetic	Tesla T4
	CNN	2	1730 images/sec	26 images/sec/watt	1.2	1x T4	Supermicro SYS-4029GP-TRT T4	19.08-py3	INT8	Synthetic	Tesla T4
	CNN	8	3730 images/sec	53 images/sec/watt	2.2	1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	INT8	Synthetic	Tesla T4
	CNN	128	5024 images/sec	72 images/sec/watt	25	1x T4	Supermicro SYS-4029GP-TRT T4	19.08-py3	INT8	Synthetic	Tesla T4

VGG16	CNN	1	726 images/sec	10 images/sec/watt	1.4	1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	INT8	Synthetic	Tesla T4
	CNN	2	1064 images/sec	15 images/sec/watt	1.9	1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	INT8	Synthetic	Tesla T4
	CNN	8	1670 images/sec	24 images/sec/watt	4.8	1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	INT8	Synthetic	Tesla T4
	CNN	128	1956 images/sec	28 images/sec/watt	65	1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	INT8	Synthetic	Tesla T4

NCF	Recommender	1	7716 samples/sec	281 samples/sec/watt	0.14	1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	INT8	Synthetic	Tesla T4
	Recommender	64	491050 samples/sec	16957 samples/sec/watt	0.14	1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	INT8	Synthetic	Tesla T4
	Recommender	25000	50828702 samples/sec	728654 samples/sec/watt	1.9	1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	INT8	Synthetic	Tesla T4
	Recommender	100000	54034709 samples/sec	777433 samples/sec/watt	1.9	1x T4	Supermicro SYS-4029GP-TRT T4	19.08-py3	INT8	Synthetic	Tesla T4

BERT-BASE	Attention	1	484 sentences/sec	11 sentences/sec/watt	2.07	1x T4	Supermicro SYS-4029GP-TRT T4	-	Mixed	SQuAD v1.1	Tesla T4
	Attention	2	754 sentences/sec	17 sentences/sec/watt	2.65	1x T4	Supermicro SYS-4029GP-TRT T4	-	Mixed	SQuAD v1.1	Tesla T4
	Attention	8	827 sentences/sec	20 sentences/sec/watt	9.67	1x T4	Supermicro SYS-4029GP-TRT T4	-	Mixed	SQuAD v1.1	Tesla T4
	Attention	128	800 sentences/sec	16 sentences/sec/watt	160.02	1x T4	Supermicro SYS-4029GP-TRT T4	-	Mixed	SQuAD v1.1	Tesla T4

BERT-LARGE	Attention	1	171 sentences/sec	4 sentences/sec/watt	5.84	1x T4	Supermicro SYS-4029GP-TRT T4	-	Mixed	SQuAD v1.1	Tesla T4
	Attention	2	168 sentences/sec	4 sentences/sec/watt	11.88	1x T4	Supermicro SYS-4029GP-TRT T4	-	Mixed	SQuAD v1.1	Tesla T4
	Attention	8	244 sentences/sec	6 sentences/sec/watt	32.74	1x T4	Supermicro SYS-4029GP-TRT T4	-	Mixed	SQuAD v1.1	Tesla T4
	Attention	128	254 sentences/sec	5 sentences/sec/watt	504.33	1x T4	Supermicro SYS-4029GP-TRT T4	-	Mixed	SQuAD v1.1	Tesla T4

TensorRT 6.0 and sequence length=128 for BERT-BASE and BERT-LARGE | TensorRT 5.1 for all other models | Efficiency based on board power | NCF uses facebook dataset

Last updated: Sept 13th, 2019

Search form

NVIDIA Tesla Deep Learning Product Performance

Training

Inference

NVIDIA Performance on MLPerf 0.6 AI Benchmarks

Training Image Classification on CNNs

Training Performance

NVIDIA Performance on MLPerf 0.6 AI Benchmarks

V100 Training Performance

T4 Training Performance

Inference Image Classification on CNNs with TensorRT

Inference Performance

V100 Inference Performance

T4 Inference Performance