NVIDIA Tesla Deep Learning Product Performance

Deep Learning Training

NVIDIA’s complete solution stack, from GPUs to libraries, and containers on NVIDIA GPU Cloud (NGC), allows data scientists to quickly get up and running with deep learning. NVIDIA® Tesla® V100 Tensor Core GPUs leverage mixed precision to accelerate deep learning training throughputs across every framework and every type of neural network. NVIDIA captured all the top spots on 6 benchmarks submitted to MLPerf, the AI’s first industry-wide benchmark, a testament to our GPU-accelerated platform approach.

NVIDIA Performance on MLPerf AI Benchmarks

ResNet-50 Time to Solution on V100

Training Image Classification on CNNs

ResNet-50 Throughput on NVIDIA Tesla V100

DGX-1: 8x Tesla V100 32GB, E5-2698 v4 2.2 GHz | Batch Size = 256 for PyTorch, 512 for all others | PyTorch uses 18.10_py3, all others use 18.09_py3 | Precision: Mixed | Dataset: ImageNet2012DGX-1

ResNet-50 Throughput on NVIDIA Tesla T4

DGX-1: 8x Tesla T4, Gold 6140 2.3 GHz | Batch Size = 96 for MXnet, 256 for all others | 18.11-py3 | Precision: Mixed | Dataset: ImageNet2012

Training Performance

NVIDIA Performance on MLPerf AI Benchmarks

FrameWork	Network	Network Type	Time to Solution	GPU	Server	Container	Precision	BatchSize	DataSet	GPU Version
MXNet	ResNet-50 v1.5	CNN	135 minutes	8x V100	DGX-1V	18.11-py3	Mixed	208	ImageNet2012	V100-SXM2-16GB
	ResNet-50 v1.5	CNN	73.9 minutes	16x V100	DGX-2	18.11-py3	Mixed	256	ImageNet2012	V100-SXM3-32GB
	ResNet-50 v1.5	CNN	70 minutes	16x V100	DGX-2H	18.11-py3	Mixed	256	ImageNet2012	V100-SXM3-32GB
	ResNet-50 v1.5	CNN	6.3 minutes	640x V100	DGX-1V Saturn	18.11-py3	Mixed	26	ImageNet2012	V100-SXM2-16GB
	ResNet-50 v1.5	CNN	7.4 minutes	512x V100	DGX-2H Pod	18.11-py3	Mixed	32	ImageNet2012	V100-SXM3-32GB

PyTorch	SSD	CNN	27 minutes	8x V100	DGX-1V	18.11-py3	Mixed	152	COCO2017	V100-SXM2-16GB
	SSD	CNN	15.9 minutes	16x V100	DGX-2	18.11-py3	Mixed	128	COCO2017	V100-SXM3-32GB
	SSD	CNN	14.1 minutes	16x V100	DGX-2H	18.11-py3	Mixed	128	COCO2017	V100-SXM3-32GB
	SSD	CNN	6.5 minutes	64x V100	DGX-1V Saturn	18.11-py3	Mixed	32	COCO2017	V100-SXM2-16GB
	SSD	CNN	5.6 minutes	64x V100	DGX-2H Pod	18.11-py3	Mixed	32	COCO2017	V100-SXM3-32GB
	Mask R-CNN	CNN	323 minutes	8x V100	DGX-1V	18.11-py3	Mixed	4	COCO2014	V100-SXM2-16GB
	Mask R-CNN	CNN	176.3 minutes	16x V100	DGX-2	18.11-py3	Mixed	4	COCO2014	V100-SXM3-32GB
	Mask R-CNN	CNN	166.9 minutes	16x V100	DGX-2H	18.11-py3	Mixed	4	COCO2014	V100-SXM3-32GB
	Mask R-CNN	CNN	81 minutes	64x V100	DGX-1V Saturn	18.11-py3	Mixed	2	COCO2014	V100-SXM2-16GB
	Mask R-CNN	CNN	72.1 minutes	64x V100	DGX-2H Pod	18.11-py3	Mixed	2	COCO2014	V100-SXM3-32GB
	NCF	CNN	0.47 minutes	8x V100	DGX-1V	18.11-py3	Mixed	131072	MovieLens 20 Million	V100-SXM2-16GB
	NCF	CNN	0.4 minutes	16x V100	DGX-2	18.11-py3	Mixed	65536	MovieLens 20 Million	V100-SXM3-32GB
	NCF	CNN	0.4 minutes	16x V100	DGX-2H	18.11-py3	Mixed	65536	MovieLens 20 Million	V100-SXM3-32GB

PyTorch	GNMT	RNN	18 minutes	8x V100	DGX-1V	18.11-py3	Mixed	128	WMT16 English-German	V100-SXM2-16GB
	GNMT	RNN	10.5 minutes	16x V100	DGX-2	18.11-py3	Mixed	64	WMT16 English-German	V100-SXM3-32GB
	GNMT	RNN	9.8 minutes	16x V100	DGX-2H	18.11-py3	Mixed	64	WMT16 English-German	V100-SXM3-32GB
	GNMT	RNN	2.8 minutes	256x V100	DGX-1V Saturn	18.11-py3	Mixed	32	WMT16 English-German	V100-SXM2-16GB
	GNMT	RNN	2.7 minutes	256x V100	DGX-2H Pod	18.11-py3	Mixed	32	WMT16 English-German	V100-SXM3-32GB

PyTorch	Transformer	Attention	33 minutes	8x V100	DGX-1V	18.11-py3	Mixed	5120	WMT17 English-German	V100-SXM2-16GB
	Transformer	Attention	21.2 minutes	16x V100	DGX-2	18.11-py3	Mixed	10240	WMT17 English-German	V100-SXM3-32GB
	Transformer	Attention	19.2 minutes	16x V100	DGX-2H	18.11-py3	Mixed	10240	WMT17 English-German	V100-SXM3-32GB
	Transformer	Attention	6.2 minutes	192x V100	DGX-1V Saturn	18.11-py3	Mixed	2560	WMT17 English-German	V100-SXM2-16GB

V100 Training Performance

FrameWork	Network	Network Type	Throughput	GPU	Server	Container	Precision	BatchSize	DataSet	GPU Version
MXNet	Inception V3	CNN	506 images/sec	1x V100	DGX-1	18.08-py3	Mixed	256	ImageNet2012	V100-SXM2-32GB
		CNN	3818 images/sec	8x V100	DGX-1	18.08-py3	Mixed	512	ImageNet2012	V100-SXM2-32GB
	ResNet-50	CNN	1309 images/sec	1x V100	DGX-1	18.12-py3	Mixed	256	ImageNet2012	V100-SXM2-32GB
		CNN	10185 images/sec	8x V100	DGX-1	18.12-py3	Mixed	256	ImageNet2012	V100-SXM2-32GB

PyTorch	Inception V3	CNN	534 images/sec	1x V100	DGX-1	18.12-py3	Mixed	512	ImageNet2012	V100-SXM2-32GB
		CNN	4038 images/sec	8x V100	DGX-1	18.12-py3	Mixed	512	ImageNet2012	V100-SXM2-32GB
	ResNet-50	CNN	875 images/sec	1x V100	DGX-1	18.08-py3	Mixed	512	ImageNet2012	V100-SXM2-32GB
		CNN	6675 images/sec	8x V100	DGX-1	18.10-py3	Mixed	256	ImageNet2012	V100-SXM2-32GB

TensorFlow	GoogleNet	CNN	1761 images/sec	1x V100	DGX-1	18.12-py3	Mixed	1024	ImageNet2012	V100-SXM2-32GB
		CNN	11645 images/sec	8x V100	DGX-1	18.11-py3	Mixed	1024	ImageNet2012	V100-SXM2-32GB
	Inception V3	CNN	519 images/sec	1x V100	DGX-1	18.11-py3	Mixed	384	ImageNet2012	V100-SXM2-32GB
		CNN	3971 images/sec	8x V100	DGX-1	18.11-py3	Mixed	384	ImageNet2012	V100-SXM2-32GB
	ResNet-50	CNN	870 images/sec	1x V100	DGX-1	18.09-py3	Mixed	512	ImageNet2012	V100-SXM2-32GB
		CNN	6720 images/sec	8x V100	DGX-1	18.09-py3	Mixed	512	ImageNet2012	V100-SXM2-32GB

MXNet	Sockeye (OpenNMT)	RNN	36088 total tokens/sec	1x V100	DGX-1	18.08-py3	FP32	512	WMT15 German-English	V100-SXM2-32GB
		RNN	250396 total tokens/sec	8x V100	DGX-1	18.07-py3	FP32	512	WMT15 German-English	V100-SXM2-32GB

PyTorch	GNMT	RNN	65542 total tokens/sec	1x V100	DGX-1	18.12-py3	Mixed	512	WMT16 English-German	V100-SXM2-32GB
		RNN	503936 total tokens/sec	8x V100	DGX-1	18.09-py3	Mixed	384	WMT16 English-German	V100-SXM2-32GB

T4 Training Performance

FrameWork	Network	Network Type	Throughput	GPU	Server	Container	Precision	BatchSize	DataSet	GPU Version
MXNet	Inception V3	CNN	160 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	18.11-py3	Mixed	192	ImageNet2012	Tesla T4
		CNN	1130 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	18.11-py3	Mixed	192	ImageNet2012	Tesla T4
	ResNet-50	CNN	432 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	18.12-py3	Mixed	256	ImageNet2012	Tesla T4
		CNN	3417 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	18.12-py3	Mixed	256	ImageNet2012	Tesla T4

PyTorch	Inception V3	CNN	168 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	18.12-py3	Mixed	256	ImageNet2012	Tesla T4
		CNN	1333 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	18.12-py3	Mixed	256	ImageNet2012	Tesla T4
	ResNet-50	CNN	278 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	18.11-py3	Mixed	256	ImageNet2012	Tesla T4
		CNN	2179 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	18.11-py3	Mixed	256	ImageNet2012	Tesla T4

TensorFlow	GoogleNet	CNN	561 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	18.12-py3	Mixed	768	ImageNet2012	Tesla T4
		CNN	4406 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	18.12-py3	Mixed	768	ImageNet2012	Tesla T4
	Inception V3	CNN	173 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	18.12-py3	Mixed	192	ImageNet2012	Tesla T4
		CNN	1320 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	18.12-py3	Mixed	192	ImageNet2012	Tesla T4
	ResNet-50	CNN	293 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	18.12-py3	Mixed	256	ImageNet2012	Tesla T4
		CNN	2259 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	18.12-py3	Mixed	256	ImageNet2012	Tesla T4

MXnet	OpenNMT	RNN	13650 total tokens/sec	1x T4	Supermicro SYS-4029GP-TRT T4	18.11-py3	FP32	16384	WMT15 German-English	Tesla T4
		RNN	97003 total tokens/sec	8x T4	Supermicro SYS-4029GP-TRT T4	18.11-py3	FP32	16384	WMT15 German-English	Tesla T4

PyTorch	GNMT	RNN	21188 total tokens/sec	1x T4	Supermicro SYS-4029GP-TRT T4	18.12-py3	Mixed	192	WMT16 English-German	Tesla T4
		RNN	146642 total tokens/sec	8x T4	Supermicro SYS-4029GP-TRT T4	18.12-py3	FP32	192	WMT16 English-German	Tesla T4

NVIDIA Deep Learning Inference Performance

NVIDIA® TensorRT™ running on NVIDIA GPUs enable the most efficient deep learning inference performance across multiple application areas and models. This versatility provides wide latitude to data scientists to create the optimal low-latency solution. Visit NVIDIA GPU Cloud (NGC) to download any of these containers.

NVIDIA Tesla® V100 Tensor Cores GPUs leverage mixed-precision to combine high throughput with low latencies across every type of neural network. Tesla P4 is an inference GPU, designed for optimal power consumption and latency, for ultra-efficient scale-out servers. Read the inference whitepaper to learn more about NVIDIA’s inference platform.

Measuring the inference performance involves balancing a lot of variables. PLASTER is an acronym that describes the key elements for measuring deep learning performance. Each letter identifies a factor (Programmability, Latency, Accuracy, Size of Model, Throughput, Energy Efficiency, Rate of Learning) that must be considered to arrive at the right set of tradeoffs and to produce a successful deep learning implementation. Refer to NVIDIA’s PLASTER whitepaper for more details.

Inference Image Classification on CNNs with TensorRT

ResNet-50 Throughput

ResNet-50 Latency

ResNet-50 Power Efficiency

Inference Performance

V100 Inference Performance

Network	Network Type	Batch Size	Throughput	Efficiency	Latency	GPU	Server	Container	Precision	Dataset	GPU Version
ResNet-50	CNN	1	1143 images/sec	8.6 images/sec/watt	0.88	1x V100	DGX-1	18.09-py3	INT8	Synthetic	V100-SXM2-16GB
	CNN	2	1548 images/sec	10 images/sec/watt	1.3	1x V100	DGX-1	18.12-py3	INT8	Synthetic	V100-SXM2-16GB
	CNN	4	1989 images/sec	12 images/sec/watt	2	1x V100	DGX-1	18.11-py3	INT8	Synthetic	V100-SXM2-16GB
	CNN	8	3247 images/sec	18 images/sec/watt	2.5	1x V100	DGX-1	18.10-py3	INT8	Synthetic	V100-SXM2-16GB
	CNN	41	5925 images/sec	21 images/sec/watt	6.9	1x V100	DGX-1	18.10-py3	Mixed	Synthetic	V100-SXM2-16GB
	CNN	64	6524 images/sec	23 images/sec/watt	9.8	1x V100	DGX-1	18.12-py3	Mixed	Synthetic	V100-SXM2-16GB
	CNN	128	6954 images/sec	25 images/sec/watt	18	1x V100	DGX-1	18.12-py3	Mixed	Synthetic	V100-SXM2-16GB

GoogleNet	CNN	1	1622 images/sec	15 images/sec/watt	0.62	1x V100	DGX-1	18.09-py3	INT8	Synthetic	V100-SXM2-16GB
	CNN	2	2142 images/sec	18 images/sec/watt	0.93	1x V100	DGX-1	18.10-py3	INT8	Synthetic	V100-SXM2-16GB
	CNN	4	3174 images/sec	27 images/sec/watt	1.3	1x V100	DGX-1	18.11-py3	INT8	Synthetic	V100-SXM2-16GB
	CNN	8	5246 images/sec	33 images/sec/watt	1.5	1x V100	DGX-1	18.12-py3	INT8	Synthetic	V100-SXM2-32GB
	CNN	64	11454 images/sec	42 images/sec/watt	5.6	1x V100	DGX-1	18.11-py3	INT8	Synthetic	V100-SXM2-16GB
	CNN	82	11916 images/sec	41 images/sec/watt	6.9	1x V100	DGX-1	18.09-py3	INT8	Synthetic	V100-SXM2-16GB
	CNN	128	12352 images/sec	43 images/sec/watt	10	1x V100	DGX-1	18.12-py3	INT8	Synthetic	V100-SXM2-16GB

NMT	RNN	1	3961 tokens/sec	32 tokens/sec/watt	13	1x V100	DGX-1	18.09-py3	Mixed	WMT16 English-German	V100-SXM2-16GB
	RNN	2	5711 tokens/sec	48 tokens/sec/watt	18	1x V100	DGX-1	18.09-py3	Mixed	WMT16 English-German	V100-SXM2-16GB
	RNN	4	9380 tokens/sec	79 tokens/sec/watt	22	1x V100	DGX-1	18.09-py3	Mixed	WMT16 English-German	V100-SXM2-16GB
	RNN	8	14452 tokens/sec	119 tokens/sec/watt	28	1x V100	DGX-1	18.09-py3	Mixed	WMT16 English-German	V100-SXM2-16GB
	RNN	64	55497 tokens/sec	487 tokens/sec/watt	59	1x V100	DGX-1	18.09-py3	Mixed	WMT16 English-German	V100-SXM2-16GB
	RNN	128	75217 tokens/sec	1068 tokens/sec/watt	87	1x V100	DGX-1	18.09-py3	Mixed	WMT16 English-German	V100-SXM2-16GB

T4 Inference Performance

Network	Network Type	Batch Size	Throughput	Efficiency	Latency	GPU	Server	Container	Precision	Dataset	GPU Version
ResNet-50	CNN	1	961 images/sec	14 images/sec/watt	1	1x T4	Supermicro SYS-4029GP-TRT T4	18.12-py3	INT8	Synthetic	Tesla T4
	CNN	2	1409 images/sec	20 images/sec/watt	1.4	1x T4	Supermicro SYS-4029GP-TRT T4	18.12-py3	INT8	Synthetic	Tesla T4
	CNN	4	2590 images/sec	38 images/sec/watt	1.5	1x T4	Supermicro SYS-4029GP-TRT T4	18.11-py3	INT8	Synthetic	Tesla T4
	CNN	8	3494 images/sec	50 images/sec/watt	2.3	1x T4	Supermicro SYS-4029GP-TRT T4	18.12-py3	INT8	Synthetic	Tesla T4
	CNN	16	4037 images/sec	58 images/sec/watt	4	1x T4	Supermicro SYS-4029GP-TRT T4	18.12-py3	INT8	Synthetic	Tesla T4
	CNN	30	4238 images/sec	61 images/sec/watt	7.1	1x T4	Supermicro SYS-4029GP-TRT T4	18.12-py3	INT8	Synthetic	Tesla T4
	CNN	128	4365 images/sec	63 images/sec/watt	29	1x T4	Supermicro SYS-4029GP-TRT T4	18.12-py3	INT8	Synthetic	Tesla T4

GoogleNet	CNN	1	1571 images/sec	23 images/sec/watt	0.64	1x T4	Supermicro SYS-4029GP-TRT T4	18.10-py3	INT8	Synthetic	Tesla T4
	CNN	2	2124 images/sec	31 images/sec/watt	0.94	1x T4	Supermicro SYS-4029GP-TRT T4	18.11-py3	INT8	Synthetic	Tesla T4
	CNN	4	3533 images/sec	52 images/sec/watt	1.1	1x T4	Supermicro SYS-4029GP-TRT T4	18.11-py3	INT8	Synthetic	Tesla T4
	CNN	8	4919 images/sec	71 images/sec/watt	1.6	1x T4	Supermicro SYS-4029GP-TRT T4	18.12-py3	INT8	Synthetic	Tesla T4
	CNN	43	6269 images/sec	90 images/sec/watt	6.9	1x T4	Supermicro SYS-4029GP-TRT T4	18.12-py3	INT8	Synthetic	Tesla T4
	CNN	64	6359 images/sec	91 images/sec/watt	10	1x T4	Supermicro SYS-4029GP-TRT T4	18.12-py3	INT8	Synthetic	Tesla T4
	CNN	128	6366 images/sec	91 images/sec/watt	20	1x T4	Supermicro SYS-4029GP-TRT T4	18.12-py3	INT8	Synthetic	Tesla T4

NMT	RNN	1	2535 tokens/sec	38 tokens/sec/watt	20	1x T4	Supermicro SYS-4029GP-TRT T4	18.12-py3	Mixed	WMT16 English-German	Tesla T4
	RNN	2	3827 tokens/sec	57 tokens/sec/watt	27	1x T4	Supermicro SYS-4029GP-TRT T4	18.12-py3	Mixed	WMT16 English-German	Tesla T4
	RNN	4	6183 tokens/sec	95 tokens/sec/watt	33	1x T4	Supermicro SYS-4029GP-TRT T4	18.12-py3	Mixed	WMT16 English-German	Tesla T4
	RNN	8	10189 tokens/sec	159 tokens/sec/watt	40	1x T4	Supermicro SYS-4029GP-TRT T4	18.11-py3	Mixed	WMT16 English-German	Tesla T4
	RNN	64	29763 tokens/sec	531 tokens/sec/watt	110	1x T4	Supermicro SYS-4029GP-TRT T4	18.12-py3	Mixed	WMT16 English-German	Tesla T4
	RNN	128	35418 tokens/sec	648 tokens/sec/watt	185	1x T4	Supermicro SYS-4029GP-TRT T4	18.12-py3	Mixed	WMT16 English-German	Tesla T4

Last updated: Jan 22, 2019

Search form

NVIDIA Tesla Deep Learning Product Performance

Deep Learning Training

NVIDIA Performance on MLPerf AI Benchmarks

Training Image Classification on CNNs

Training Performance

NVIDIA Performance on MLPerf AI Benchmarks

V100 Training Performance

T4 Training Performance

NVIDIA Deep Learning Inference Performance

Inference Image Classification on CNNs with TensorRT

Inference Performance

V100 Inference Performance

T4 Inference Performance