NVIDIA Tesla Deep Learning Product Performance

Deep Learning Training

NVIDIA’s complete solution stack, from GPUs to libraries, and containers on NVIDIA GPU Cloud (NGC), allows data scientists to quickly get up and running with deep learning. NVIDIA® Tesla® V100 Tensor Core GPUs leverage mixed precision to accelerate deep learning training throughputs across every framework and every type of neural network. NVIDIA captured all the top spots on 6 benchmarks submitted to MLPerf, the AI’s first industry-wide benchmark, a testament to our GPU-accelerated platform approach.

NVIDIA Performance on MLPerf AI Benchmarks

ResNet-50 Time to Solution on V100

Training Image Classification on CNNs

ResNet-50 Throughput on NVIDIA Tesla V100

DGX-1: 8x Tesla V100 32GB, E5-2698 v4 2.2 GHz | Batch Size = 256 for PyTorch, 512 for all others | PyTorch uses 18.10_py3, all others use 18.09_py3 | Precision: Mixed | Dataset: ImageNet2012DGX-1

ResNet-50 Throughput on NVIDIA Tesla T4

DGX-1: 8x Tesla T4, Gold 6140 2.3 GHz | Batch Size = 96 for MXnet, 256 for all others | 18.11-py3 | Precision: Mixed | Dataset: ImageNet2012

Training Performance

NVIDIA Performance on MLPerf AI Benchmarks

FrameWork	Network	Network Type	Time to Solution	GPU	Server	Container	Precision	BatchSize	DataSet	GPU Version
MXNet	Resnet-50 v1.5	CNN	135 minutes	8x V100	DGX-1V	18.11-py3	Mixed	208	ImageNet2012	V100-SXM2-16GB
	Resnet-50 v1.5	CNN	73.9 minutes	16x V100	DGX-2	18.11-py3	Mixed	256	ImageNet2012	V100-SXM3-32GB
	Resnet-50 v1.5	CNN	70 minutes	16x V100	DGX-2H	18.11-py3	Mixed	256	ImageNet2012	V100-SXM3-32GB
	Resnet-50 v1.5	CNN	6.3 minutes	640x V100	DGX-1V Saturn	18.11-py3	Mixed	26	ImageNet2012	V100-SXM2-16GB
	Resnet-50 v1.5	CNN	7.4 minutes	512x V100	DGX-2H System	18.11-py3	Mixed	32	ImageNet2012	V100-SXM3-32GB

PyTorch	SSD	CNN	27 minutes	8x V100	DGX-1V	18.11-py3	Mixed	152	COCO2017	V100-SXM2-16GB
	SSD	CNN	15.9 minutes	16x V100	DGX-2	18.11-py3	Mixed	128	COCO2017	V100-SXM3-32GB
	SSD	CNN	14.1 minutes	16x V100	DGX-2H	18.11-py3	Mixed	128	COCO2017	V100-SXM3-32GB
	SSD	CNN	6.5 minutes	64x V100	DGX-1V Saturn	18.11-py3	Mixed	32	COCO2017	V100-SXM2-16GB
	SSD	CNN	5.6 minutes	64x V100	DGX-2H System	18.11-py3	Mixed	32	COCO2017	V100-SXM3-32GB
	Mask R-CNN	CNN	323 minutes	8x V100	DGX-1V	18.11-py3	Mixed	4	COCO2014	V100-SXM2-16GB
	Mask R-CNN	CNN	176.3 minutes	16x V100	DGX-2	18.11-py3	Mixed	4	COCO2014	V100-SXM3-32GB
	Mask R-CNN	CNN	166.9 minutes	16x V100	DGX-2H	18.11-py3	Mixed	4	COCO2014	V100-SXM3-32GB
	Mask R-CNN	CNN	81 minutes	64x V100	DGX-1V Saturn	18.11-py3	Mixed	2	COCO2014	V100-SXM2-16GB
	Mask R-CNN	CNN	72.1 minutes	64x V100	DGX-2H System	18.11-py3	Mixed	2	COCO2014	V100-SXM3-32GB
	NCF	CNN	0.47 minutes	8x V100	DGX-1V	18.11-py3	Mixed	131072	MovieLens 20 Million	V100-SXM2-16GB
	NCF	CNN	0.4 minutes	16x V100	DGX-2	18.11-py3	Mixed	65536	MovieLens 20 Million	V100-SXM3-32GB
	NCF	CNN	0.4 minutes	16x V100	DGX-2H	18.11-py3	Mixed	65536	MovieLens 20 Million	V100-SXM3-32GB

PyTorch	GNMT	RNN	18 minutes	8x V100	DGX-1V	18.11-py3	Mixed	128	WMT16 English-German	V100-SXM2-16GB
	GNMT	RNN	10.5 minutes	16x V100	DGX-2	18.11-py3	Mixed	64	WMT16 English-German	V100-SXM3-32GB
	GNMT	RNN	9.8 minutes	16x V100	DGX-2H	18.11-py3	Mixed	64	WMT16 English-German	V100-SXM3-32GB
	GNMT	RNN	2.8 minutes	256x V100	DGX-1V Saturn	18.11-py3	Mixed	32	WMT16 English-German	V100-SXM2-16GB
	GNMT	RNN	2.7 minutes	256x V100	DGX-2H System	18.11-py3	Mixed	32	WMT16 English-German	V100-SXM3-32GB

PyTorch	Transformer	Attention	33 minutes	8x V100	DGX-1V	18.11-py3	Mixed	5120	WMT17 English-German	V100-SXM2-16GB
	Transformer	Attention	21.2 minutes	16x V100	DGX-2	18.11-py3	Mixed	10240	WMT17 English-German	V100-SXM3-32GB
	Transformer	Attention	19.2 minutes	16x V100	DGX-2H	18.11-py3	Mixed	10240	WMT17 English-German	V100-SXM3-32GB
	Transformer	Attention	6.2 minutes	192x V100	DGX-1V Saturn	18.11-py3	Mixed	2560	WMT17 English-German	V100-SXM2-16GB

V100 Training Performance

FrameWork	Network	Network Type	Throughput	GPU	Server	Container	Precision	BatchSize	DataSet	GPU Version
MXNet	Inception V3	CNN	506 images/sec	1x V100	DGX-1	18.08-py3	Mixed	256	ImageNet2012	V100-SXM2-32GB
		CNN	3818 images/sec	8x V100	DGX-1	18.08-py3	Mixed	512	ImageNet2012	V100-SXM2-32GB
	Resnet-50	CNN	1232 images/sec	1x V100	DGX-1	18.09-py3	Mixed	512	ImageNet2012	V100-SXM2-32GB
		CNN	9598 images/sec	8x V100	DGX-1	18.09-py3	Mixed	512	ImageNet2012	V100-SXM2-32GB

PyTorch	Inception V3	CNN	509 images/sec	1x V100	DGX-1	18.11-py3	Mixed	384	ImageNet2012	V100-SXM2-32GB
		CNN	3909 images/sec	8x V100	DGX-1	18.10-py3	Mixed	384	ImageNet2012	V100-SXM2-32GB
	Resnet-50	CNN	875 images/sec	1x V100	DGX-1	18.08-py3	Mixed	512	ImageNet2012	V100-SXM2-32GB
		CNN	6675 images/sec	8x V100	DGX-1	18.10-py3	Mixed	256	ImageNet2012	V100-SXM2-32GB

TensorFlow	GoogleNet	CNN	1738 images/sec	1x V100	DGX-1	18.11-py3	Mixed	1024	ImageNet2012	V100-SXM2-32GB
		CNN	11645 images/sec	8x V100	DGX-1	18.11-py3	Mixed	1024	ImageNet2012	V100-SXM2-32GB
	Inception V3	CNN	519 images/sec	1x V100	DGX-1	18.11-py3	Mixed	384	ImageNet2012	V100-SXM2-32GB
		CNN	3971 images/sec	8x V100	DGX-1	18.11-py3	Mixed	384	ImageNet2012	V100-SXM2-32GB
		CNN	870 images/sec	1x V100	DGX-1	18.09-py3	Mixed	512	ImageNet2012	V100-SXM2-32GB
		CNN	6720 images/sec	8x V100	DGX-1	18.09-py3	Mixed	512	ImageNet2012	V100-SXM2-32GB

MXNet	Sockeye (OpenNMT)	RNN	36088 total tokens/sec	1x V100	DGX-1	18.08-py3	FP32	512	WMT15 German-English	V100-SXM2-32GB
		RNN	250396 total tokens/sec	8x V100	DGX-1	18.07-py3	FP32	512	WMT15 German-English	V100-SXM2-32GB

PyTorch	GNMT	RNN	64763 total tokens/sec	1x V100	DGX-1	18.09-py3	Mixed	512	WMT16 English-German	V100-SXM2-32GB
		RNN	503936 total tokens/sec	8x V100	DGX-1	18.09-py3	Mixed	384	WMT16 English-German	V100-SXM2-32GB

T4 Training Performance

FrameWork	Network	Network Type	Throughput	GPU	Server	Container	Precision	BatchSize	DataSet	GPU Version
MXNet	Inception V3	CNN	160 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	18.11-py3	Mixed	192	ImageNet2012	Tesla T4
		CNN	1130 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	18.11-py3	Mixed	192	ImageNet2012	Tesla T4
	Resnet50	CNN	416 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	18.11-py3	Mixed	96	ImageNet2012	Tesla T4
		CNN	3212 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	18.11-py3	Mixed	96	ImageNet2012	Tesla T4

PyTorch	Inception V3	CNN	164 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	18.11-py3	Mixed	192	ImageNet2012	Tesla T4
		CNN	1282 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	18.11-py3	Mixed	192	ImageNet2012	Tesla T4
	Resnet50	CNN	278 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	18.11-py3	Mixed	256	ImageNet2012	Tesla T4
		CNN	2179 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	18.11-py3	Mixed	256	ImageNet2012	Tesla T4

TensorFlow	GoogleNet	CNN	554 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	18.11-py3	Mixed	512	ImageNet2012	Tesla T4
		CNN	4269 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	18.11-py3	Mixed	512	ImageNet2012	Tesla T4
	Inception V3	CNN	170 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	18.11-py3	Mixed	192	ImageNet2012	Tesla T4
		CNN	1026 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	18.11-py3	Mixed	128	ImageNet2012	Tesla T4
	Resnet50	CNN	289 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	18.11-py3	Mixed	256	ImageNet2012	Tesla T4
		CNN	2205 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	18.11-py3	Mixed	256	ImageNet2012	Tesla T4

MXnet	OpenNMT	RNN	13650 total tokens/sec	1x T4	Supermicro SYS-4029GP-TRT T4	18.11-py3	FP32	16384	WMT15 German-English	Tesla T4
		RNN	97003 total tokens/sec	8x T4	Supermicro SYS-4029GP-TRT T4	18.11-py3	FP32	16384	WMT15 German-English	Tesla T4

PyTorch	GNMT	RNN	20139 total tokens/sec	1x T4	Supermicro SYS-4029GP-TRT T4	18.09-py3	Mixed	192	WMT16 English-German	Tesla T4
		RNN	50925 total tokens/sec	8x T4	Supermicro SYS-4029GP-TRT T4	18.11-py3	FP32	128	WMT16 English-German	Tesla T4

NVIDIA Deep Learning Inference Performance

NVIDIA® TensorRT™ running on NVIDIA GPUs enable the most efficient deep learning inference performance across multiple application areas and models. This versatility provides wide latitude to data scientists to create the optimal low-latency solution. Visit NVIDIA GPU Cloud (NGC) to download any of these containers.

NVIDIA Tesla® V100 Tensor Cores GPUs leverage mixed-precision to combine high throughput with low latencies across every type of neural network. Tesla P4 is an inference GPU, designed for optimal power consumption and latency, for ultra-efficient scale-out servers. Read the inference whitepaper to learn more about NVIDIA’s inference platform.

Measuring the inference performance involves balancing a lot of variables. PLASTER is an acronym that describes the key elements for measuring deep learning performance. Each letter identifies a factor (Programmability, Latency, Accuracy, Size of Model, Throughput, Energy Efficiency, Rate of Learning) that must be considered to arrive at the right set of tradeoffs and to produce a successful deep learning implementation. Refer to NVIDIA’s PLASTER whitepaper for more details.

Inference Image Classification on CNNs with TensorRT

ResNet-50 Throughput

ResNet-50 Latency

ResNet-50 Power Efficiency

Inference Performance

V100 Inference Performance

Network	Network Type	Batch Size	Throughput	Efficiency	Latency	GPU	Server	Container	Precision	Dataset	GPU Version
Resnet-50	CNN	1	1143 images/sec	8.6 images/sec/watt	0.88	1x V100	DGX-1	18.09-py3	INT8	Synthetic	V100-SXM2-16GB
	CNN	2	1544 images/sec	11 images/sec/watt	1.3	1x V100	DGX-1	18.11-py3	INT8	Synthetic	V100-SXM2-16GB
	CNN	4	1989 images/sec	12 images/sec/watt	2	1x V100	DGX-1	18.11-py3	INT8	Synthetic	V100-SXM2-16GB
	CNN	8	3247 images/sec	18 images/sec/watt	2.5	1x V100	DGX-1	18.10-py3	INT8	Synthetic	V100-SXM2-16GB
	CNN	41	5925 images/sec	21 images/sec/watt	6.9	1x V100	DGX-1	18.10-py3	Mixed	Synthetic	V100-SXM2-16GB
	CNN	64	6506 images/sec	24 images/sec/watt	9.8	1x V100	DGX-1	18.11-py3	Mixed	Synthetic	V100-SXM2-16GB
	CNN	128	6954 images/sec	24 images/sec/watt	18	1x V100	DGX-1	18.11-py3	Mixed	Synthetic	V100-SXM2-16GB

GoogleNet	CNN	1	1622 images/sec	15 images/sec/watt	0.62	1x V100	DGX-1	18.09-py3	INT8	Synthetic	V100-SXM2-16GB
	CNN	2	2142 images/sec	18 images/sec/watt	0.93	1x V100	DGX-1	18.10-py3	INT8	Synthetic	V100-SXM2-16GB
	CNN	4	3174 images/sec	27 images/sec/watt	1.3	1x V100	DGX-1	18.11-py3	INT8	Synthetic	V100-SXM2-16GB
	CNN	8	5262 images/sec	34 images/sec/watt	1.5	1x V100	DGX-1	18.11-py3	INT8	Synthetic	V100-SXM2-32GB
	CNN	64	11454 images/sec	42 images/sec/watt	5.6	1x V100	DGX-1	18.11-py3	INT8	Synthetic	V100-SXM2-16GB
	CNN	82	11916 images/sec	41 images/sec/watt	6.9	1x V100	DGX-1	18.09-py3	INT8	Synthetic	V100-SXM2-16GB
	CNN	128	12350 images/sec	42 images/sec/watt	10	1x V100	DGX-1	18.10-py3	INT8	Synthetic	V100-SXM2-16GB

NMT	RNN	1	3961 tokens/sec	32 tokens/sec/watt	13	1x V100	DGX-1	18.09-py3	Mixed	WMT16 English-German	V100-SXM2-16GB
	RNN	2	5711 tokens/sec	48 tokens/sec/watt	18	1x V100	DGX-1	18.09-py3	Mixed	WMT16 English-German	V100-SXM2-16GB
	RNN	4	9380 tokens/sec	79 tokens/sec/watt	22	1x V100	DGX-1	18.09-py3	Mixed	WMT16 English-German	V100-SXM2-16GB
	RNN	8	14452 tokens/sec	119 tokens/sec/watt	28	1x V100	DGX-1	18.09-py3	Mixed	WMT16 English-German	V100-SXM2-16GB
	RNN	64	55497 tokens/sec	487 tokens/sec/watt	59	1x V100	DGX-1	18.09-py3	Mixed	WMT16 English-German	V100-SXM2-16GB
	RNN	128	75217 tokens/sec	1068 tokens/sec/watt	87	1x V100	DGX-1	18.09-py3	Mixed	WMT16 English-German	V100-SXM2-16GB

T4 Inference Performance

Network	Network Type	Batch Size	Throughput	Efficiency	Latency	GPU	Server	Container	Precision	Dataset	GPU Version
Resnet-50	CNN	1	961 images/sec	14 images/sec/watt	1	1x T4	Supermicro SYS-4029GP-TRT T4	18.10-py3	INT8	Synthetic	Tesla T4
	CNN	2	1404 images/sec	21 images/sec/watt	1.4	1x T4	Supermicro SYS-4029GP-TRT T4	18.10-py3	INT8	Synthetic	Tesla T4
	CNN	4	2590 images/sec	38 images/sec/watt	1.5	1x T4	Supermicro SYS-4029GP-TRT T4	18.11-py3	INT8	Synthetic	Tesla T4
	CNN	8	3427 images/sec	50 images/sec/watt	2.3	1x T4	Supermicro SYS-4029GP-TRT T4	18.10-py3	INT8	Synthetic	Tesla T4
	CNN	16	3888 images/sec	56 images/sec/watt	4.1	1x T4	Supermicro SYS-4029GP-TRT T4	18.10-py3	INT8	Synthetic	Tesla T4
	CNN	29	4092 images/sec	60 images/sec/watt	7.1	1x T4	Supermicro SYS-4029GP-TRT T4	18.10-py3	INT8	Synthetic	Tesla T4
	CNN	128	4198 images/sec	60 images/sec/watt	31	1x T4	Supermicro SYS-4029GP-TRT T4	18.10-py3	INT8	Synthetic	Tesla T4

GoogleNet	CNN	1	1571 images/sec	23 images/sec/watt	0.64	1x T4	Supermicro SYS-4029GP-TRT T4	18.10-py3	INT8	Synthetic	Tesla T4
	CNN	2	2124 images/sec	31 images/sec/watt	0.94	1x T4	Supermicro SYS-4029GP-TRT T4	18.11-py3	INT8	Synthetic	Tesla T4
	CNN	4	3533 images/sec	52 images/sec/watt	1.1	1x T4	Supermicro SYS-4029GP-TRT T4	18.11-py3	INT8	Synthetic	Tesla T4
	CNN	8	4738 images/sec	69 images/sec/watt	1.7	1x T4	Supermicro SYS-4029GP-TRT T4	18.11-py3	INT8	Synthetic	Tesla T4
	CNN	40	5746 images/sec	83 images/sec/watt	7	1x T4	Supermicro SYS-4029GP-TRT T4	18.11-py3	INT8	Synthetic	Tesla T4
	CNN	64	5860 images/sec	85 images/sec/watt	11	1x T4	Supermicro SYS-4029GP-TRT T4	18.11-py3	INT8	Synthetic	Tesla T4
	CNN	128	6290 images/sec	91 images/sec/watt	20	1x T4	Supermicro SYS-4029GP-TRT T4	18.11-py3	INT8	Synthetic	Tesla T4

NMT	RNN	1	2505 tokens/sec	38 tokens/sec/watt	21	1x T4	Supermicro SYS-4029GP-TRT T4	18.11-py3	Mixed	WMT16 English-German	Tesla T4
	RNN	2	3737 tokens/sec	57 tokens/sec/watt	28	1x T4	Supermicro SYS-4029GP-TRT T4	18.11-py3	Mixed	WMT16 English-German	Tesla T4
	RNN	4	6168 tokens/sec	97 tokens/sec/watt	33	1x T4	Supermicro SYS-4029GP-TRT T4	18.10-py3	Mixed	WMT16 English-German	Tesla T4
	RNN	8	10189 tokens/sec	159 tokens/sec/watt	40	1x T4	Supermicro SYS-4029GP-TRT T4	18.11-py3	Mixed	WMT16 English-German	Tesla T4
	RNN	64	29012 tokens/sec	646 tokens/sec/watt	113	1x T4	Supermicro SYS-4029GP-TRT T4	18.10-py3	Mixed	WMT16 English-German	Tesla T4
	RNN	128	34435 tokens/sec	628 tokens/sec/watt	191	1x T4	Supermicro SYS-4029GP-TRT T4	18.10-py3	Mixed	WMT16 English-German	Tesla T4

Last updated: Dec 12, 2018

Search form

NVIDIA Tesla Deep Learning Product Performance

Deep Learning Training

NVIDIA Performance on MLPerf AI Benchmarks

Training Image Classification on CNNs

Training Performance

NVIDIA Performance on MLPerf AI Benchmarks

V100 Training Performance

T4 Training Performance

NVIDIA Deep Learning Inference Performance

Inference Image Classification on CNNs with TensorRT

Inference Performance

V100 Inference Performance

T4 Inference Performance