NVIDIA Tesla Deep Learning Product Performance

For HPC performance, please go here.

Training
Inference

NVIDIA’s complete solution stack, from GPUs to libraries, and containers on NVIDIA GPU Cloud (NGC), allows data scientists to quickly get up and running with deep learning. NVIDIA® Tesla® V100 Tensor Core GPUs leverage mixed precision to accelerate deep learning training throughputs across every framework and every type of neural network. NVIDIA breaks performance records on MLPerf, the AI’s first industry-wide benchmark, a testament to our GPU-accelerated platform approach.

NVIDIA Performance on MLPerf 0.6 AI Benchmarks

ResNet-50 v1.5 Time to Solution on V100

MXNet | Batch Size refer to CNN V100 Training table below | Precision: Mixed | Dataset: ImageNet2012 | Convergence criteria - refer to MLPerf requirements

Training Image Classification on CNNs

ResNet-50 V1.5 Throughput on V100

DGX-1: 8x Tesla V100-SXM2-32GB, E5-2698 v4 2.2 GHz | Batch Size = 256 | MXNet = 19.06-py3, Tensorflow and PyTorch = 19.07_py3 | Precision: Mixed | Dataset: ImageNet2012

ResNet-50 V1.5 Throughput on T4

Supermicro SYS-4029GP-TRT T4: 8x Tesla T4 16GB, Gold 6140 2.3 GHz | Batch Size = 208 for MXNet, PyTorch = 256, TensorFlow = 128 | MXNet and TensorFlow = 19.05-py3, PyTorch = 19.07_py3 | Precision: Mixed | Dataset: ImageNet2012

Training Performance

NVIDIA Performance on MLPerf 0.6 AI Benchmarks

Framework	Network	Network Type	Time to Solution	GPU	Server	MLPerf-ID	Precision	Dataset	GPU Version
MXNet	ResNet-50 v1.5	CNN	115.22 minutes	8x V100	DGX-1	0.6-8	Mixed	ImageNet2012	V100-SXM2-16GB
		CNN	57.87 minutes	16x V100	DGX-2	0.6-17	Mixed	ImageNet2012	V100-SXM3-32GB
		CNN	52.74 minutes	16x V100	DGX-2H	0.6-19	Mixed	ImageNet2012	V100-SXM3-32GB-H
		CNN	2.59 minutes	512x V100	DGX-2H	0.6-29	Mixed	ImageNet2012	V100-SXM3-32GB-H
		CNN	1.69 minutes	1040x V100	DGX-1	0.6-16	Mixed	ImageNet2012	V100-SXM2-16GB
		CNN	1.33 minutes	1536x V100	DGX-2H	0.6-30	Mixed	ImageNet2012	V100-SXM3-32GB-H

PyTorch	SSD-ResNet-34	CNN	22.36 minutes	8x V100	DGX-1	0.6-9	Mixed	COCO2017	V100-SXM2-16GB
		CNN	12.21 minutes	16x V100	DGX-2	0.6-18	Mixed	COCO2017	V100-SXM3-32GB
		CNN	11.41 minutes	16x V100	DGX-2H	0.6-20	Mixed	COCO2017	V100-SXM3-32GB-H
		CNN	4.78 minutes	64x V100	DGX-2H	0.6-21	Mixed	COCO2017	V100-SXM3-32GB-H
		CNN	2.67 minutes	240x V100	DGX-1	0.6-13	Mixed	COCO2017	V100-SXM2-16GB
		CNN	2.56 minutes	240x V100	DGX-2H	0.6-24	Mixed	COCO2017	V100-SXM3-32GB-H
		CNN	2.23 minutes	240x V100	DGX-2H	0.6-27	Mixed	COCO2017	V100-SXM3-32GB-H
	Mask R-CNN	CNN	207.48 minutes	8x V100	DGX-1	0.6-9	Mixed	COCO2017	V100-SXM2-16GB
		CNN	101 minutes	16x V100	DGX-2	0.6-18	Mixed	COCO2017	V100-SXM3-32GB
		CNN	95.2 minutes	16x V100	DGX-2H	0.6-20	Mixed	COCO2017	V100-SXM3-32GB-H
		CNN	32.72 minutes	64x V100	DGX-2H	0.6-21	Mixed	COCO2017	V100-SXM3-32GB-H
		CNN	22.03 minutes	192x V100	DGX-1	0.6-12	Mixed	COCO2017	V100-SXM2-16GB
		CNN	18.47 minutes	192x V100	DGX-2H	0.6-23	Mixed	COCO2017	V100-SXM3-32GB-H

PyTorch	GNMT	RNN	20.55 minutes	8x V100	DGX-1	0.6-9	Mixed	WMT16 English-German	V100-SXM2-16GB
		RNN	10.94 minutes	16x V100	DGX-2	0.6-18	Mixed	WMT16 English-German	V100-SXM3-32GB
		RNN	9.87 minutes	16x V100	DGX-2H	0.6-20	Mixed	WMT16 English-German	V100-SXM3-32GB-H
		RNN	2.12 minutes	256x V100	DGX-2H	0.6-25	Mixed	WMT16 English-German	V100-SXM3-32GB-H
		RNN	1.99 minutes	384x V100	DGX-1	0.6-14	Mixed	WMT16 English-German	V100-SXM2-16GB
		RNN	1.8 minutes	384x V100	DGX-2H	0.6-26	Mixed	WMT16 English-German	V100-SXM3-32GB-H

PyTorch	Transformer	Attention	20.34 minutes	8x V100	DGX-1	0.6-9	Mixed	WMT17 English-German	V100-SXM2-16GB
		Attention	11.04 minutes	16x V100	DGX-2	0.6-18	Mixed	WMT17 English-German	V100-SXM3-32GB
		Attention	9.8 minutes	16x V100	DGX-2H	0.6-20	Mixed	WMT17 English-German	V100-SXM3-32GB-H
		Attention	2.41 minutes	160x V100	DGX-2H	0.6-22	Mixed	WMT17 English-German	V100-SXM3-32GB-H
		Attention	2.05 minutes	480x V100	DGX-1	0.6-15	Mixed	WMT17 English-German	V100-SXM2-16GB
		Attention	1.59 minutes	480x V100	DGX-2H	0.6-28	Mixed	WMT17 English-German	V100-SXM3-32GB-H

TensorFlow	MiniGo	Reinforcement Learning	27.39 minutes	8x V100	DGX-1	0.6-10	Mixed	N/A	V100-SXM2-16GB
		Reinforcement Learning	13.57 minutes	24x V100	DGX-1	0.6-11	Mixed	N/A	V100-SXM2-16GB

Refer to MLPerf requirements for convergence criteria

V100 Training Performance

Framework	Network	Network Type	Throughput	GPU	Server	Container	Precision	Batch Size	Dataset	GPU Version
MXNet	Inception V3	CNN	527 images/sec	1x V100	DGX-1	19.07-py3	Mixed	256	ImageNet2012	V100-SXM2-32GB
		CNN	606 images/sec	1x V100	DGX-2H	19.07-py3	Mixed	256	ImageNet2012	V100-SXM3-32GB-H
		CNN	4105 images/sec	8x V100	DGX-1	19.07-py3	Mixed	256	ImageNet2012	V100-SXM2-32GB
		CNN	4648 images/sec	8x V100	DGX-2H	19.07-py3	Mixed	256	ImageNet2012	V100-SXM3-32GB-H
	ResNet-50	CNN	1409 images/sec	1x V100	DGX-1	19.02-py3	Mixed	128	ImageNet2012	V100-SXM2-16GB
		CNN	1442 images/sec	1x V100	DGX-2	19.02-py3	Mixed	256	ImageNet2012	V100-SXM3-32GB
		CNN	10380 images/sec	8x V100	DGX-1	19.02-py3	Mixed	128	ImageNet2012	V100-SXM2-16GB
		CNN	10530 images/sec	8x V100	DGX-2	19.02-py3	Mixed	256	ImageNet2012	V100-SXM3-32GB
	ResNet-50 v1.5	CNN	1422 images/sec	1x V100	DGX-1	19.07-py3	Mixed	208	ImageNet2012	V100-SXM2-16GB
		CNN	1597 images/sec	1x V100	DGX-2H	19.07-py3	Mixed	256	ImageNet2012	V100-SXM3-32GB-H
		CNN	9566 images/sec	8x V100	DGX-1	19.06-py3	Mixed	256	ImageNet2012	V100-SXM2-32GB
		CNN	11056 images/sec	8x V100	DGX-2	19.05-py3	Mixed	128	ImageNet2012	V100-SXM3-32GB
		CNN	11507 images/sec	8x V100	DGX-2H	19.05-py3	Mixed	256	ImageNet2012	V100-SXM3-32GB-H

PyTorch	Inception V3	CNN	543 images/sec	1x V100	DGX-1	19.07-py3	Mixed	256	ImageNet2012	V100-SXM2-32GB
		CNN	629 images/sec	1x V100	DGX-2H	19.07-py3	Mixed	256	ImageNet2012	V100-SXM3-32GB-H
		CNN	4156 images/sec	8x V100	DGX-1	19.03-py3	Mixed	256	ImageNet2012	V100-SXM2-32GB
	Mask R-CNN	CNN	14 images/sec	1x V100	DGX-1	19.07-py3	Mixed	4	COCO2014	V100-SXM2-32GB
		CNN	17 images/sec	1x V100	DGX-2H	19.05-py3	Mixed	16	COCO2014	V100-SXM3-32GB-H
		CNN	88 images/sec	8x V100	DGX-1	19.06-py3	Mixed	16	COCO2014	V100-SXM2-32GB
	ResNet-50	CNN	819 images/sec	1x V100	DGX-1	19.02_py3	Mixed	256	ImageNet2012	V100-SXM2-16GB
		CNN	820 images/sec	1x V100	DGX-2	19.02-py3	Mixed	256	ImageNet2012	V100-SXM3-32GB
		CNN	6218 images/sec	8x V100	DGX-1	19.02-py3	Mixed	256	ImageNet2012	V100-SXM2-16GB
	ResNet-50 v1.5	CNN	928 images/sec	1x V100	DGX-1	19.07-py3	Mixed	256	ImageNet2012	V100-SXM2-16GB
		CNN	1036 images/sec	1x V100	DGX-2H	19.07-py3	Mixed	256	ImageNet2012	V100-SXM3-32GB-H
		CNN	7288 images/sec	8x V100	DGX-1	19.07-py3	Mixed	256	ImageNet2012	V100-SXM2-16GB
	SSD v1.1	CNN	225 images/sec	1x V100	DGX-1	19.07-py3	Mixed	64	COCO 2017	V100-SXM2-32GB
		CNN	299 images/sec	1x V100	DGX-2H	19.05-py3	Mixed	64	COCO 2017	V100-SXM3-32GB-H
		CNN	2018 images/sec	8x V100	DGX-1	19.06-py3	Mixed	64	COCO 2017	V100-SXM2-32GB
	Tacotron2	CNN	11743 total input tokens/sec	1x V100	DGX-1	19.07-py3	Mixed	80	LJ Speech 1.1	V100-SXM2-16GB
		CNN	16847 total input tokens/sec	1x V100	DGX-2H	19.07-py3	Mixed	80	LJ Speech 1.1	V100-SXM3-32GB-H
		CNN	81410 total input tokens/sec	8x V100	DGX-1	19.07-py3	Mixed	80	LJ Speech 1.1	V100-SXM2-32GB
		CNN	104447 total input tokens/sec	8x V100	DGX-2H	19.07-py3	Mixed	80	LJ Speech 1.1	V100-SXM3-32GB-H
	WaveGlow	CNN	77780 output samples/sec	1x V100	DGX-1	19.05-py3	Mixed	8	LJ Speech 1.1	V100-SXM2-16GB
		CNN	91052 output samples/sec	1x V100	DGX-2H	19.05-py3	Mixed	8	LJ Speech 1.1	V100-SXM3-32GB-H
		CNN	533364 output samples/sec	8x V100	DGX-1	19.05-py3	Mixed	8	LJ Speech 1.1	V100-SXM2-32GB

TensorFlow	Inception V3	CNN	542 images/sec	1x V100	DGX-1	19.07-py3	Mixed	256	ImageNet2012	V100-SXM2-32GB
		CNN	626 images/sec	1x V100	DGX-2H	19.07-py3	Mixed	256	ImageNet2012	V100-SXM3-32GB-H
		CNN	4097 images/sec	8x V100	DGX-1	19.07-py3	Mixed	256	ImageNet2012	V100-SXM2-32GB
	ResNet-50 v1.5	CNN	840 images/sec	1x V100	DGX-1	19.07-py3	Mixed	256	ImageNet2012	V100-SXM2-16GB
		CNN	965 images/sec	1x V100	DGX-2H	19.07-py3	Mixed	256	ImageNet2012	V100-SXM3-32GB-H
		CNN	6474 images/sec	8x V100	DGX-1	19.07-py3	Mixed	256	ImageNet2012	V100-SXM2-16GB
	SSD v1.1	CNN	114 images/sec	1x V100	DGX-1	19.07-py3	Mixed	32	COCO 2017	V100-SXM2-32GB
		CNN	125 images/sec	1x V100	DGX-2	19.07-py3	Mixed	32	COCO 2017	V100-SXM3-32GB
		CNN	665 images/sec	8x V100	DGX-1	19.05-py3	Mixed	32	COCO 2017	V100-SXM2-32GB
		CNN	770 images/sec	8x V100	DGX-2	19.05-py3	Mixed	32	COCO 2017	V100-SXM3-32GB
	U-Net Industrial	CNN	95 images/sec	1x V100	DGX-1	19.07-py3	Mixed	16	DAGM2007	V100-SXM2-32GB
		CNN	110 images/sec	1x V100	DGX-2H	19.07-py3	Mixed	16	DAGM2007	V100-SXM3-32GB-H
		CNN	492 images/sec	8x V100	DGX-1	19.07-py3	Mixed	2	DAGM2007	V100-SXM2-32GB
		CNN	515 images/sec	8x V100	DGX-2	19.07-py3	Mixed	2	DAGM2007	V100-SXM3-32GB

PyTorch	GNMT V2	RNN	75975 total tokens/sec	1x V100	DGX-1	19.07-py3	Mixed	128	WMT16 English-German	V100-SXM2-32GB
		RNN	82766 total tokens/sec	1x V100	DGX-2	19.07-py3	Mixed	128	WMT16 English-German	V100-SXM3-32GB
		RNN	582605 total tokens/sec	8x V100	DGX-1	19.07-py3	Mixed	128	WMT16 English-German	V100-SXM2-32GB

TensorFlow	GNMT V2	RNN	22471 total tokens/sec	1x V100	DGX-1	19.07-py3	Mixed	192	WMT16 English-German	V100-SXM2-16GB
		RNN	26039 total tokens/sec	1x V100	DGX-2H	19.07-py3	Mixed	192	WMT16 English-German	V100-SXM3-32GB-H
		RNN	149008 total tokens/sec	8x V100	DGX-1	19.07-py3	Mixed	192	WMT16 English-German	V100-SXM2-16GB

PyTorch	NCF	Recommender	22093850 samples/sec	1x V100	DGX-1	19.07-py3	Mixed	1048576	MovieLens 20 Million	V100-SXM2-16GB
		Recommender	24473776 samples/sec	1x V100	DGX-2H	19.07-py3	Mixed	1048576	MovieLens 20 Million	V100-SXM3-32GB-H
		Recommender	104122673 samples/sec	8x V100	DGX-1	19.07-py3	Mixed	1048576	MovieLens 20 Million	V100-SXM2-16GB
		Recommender	109969915 samples/sec	8x V100	DGX-2H	19.07-py3	Mixed	1048576	MovieLens 20 Million	V100-SXM3-32GB-H

TensorFlow	NCF	Recommender	26415693 samples/sec	1x V100	DGX-1	19.07-py3	Mixed	1048576	MovieLens 20 Million	V100-SXM2-16GB
		Recommender	56183685 samples/sec	8x V100	DGX-1	19.07-py3	Mixed	1048576	MovieLens 20 Million	V100-SXM2-32GB

TensorFlow	BERT-LARGE	Attention	32 sentences/sec	1x V100	DGX-1	19.07-py3	Mixed	10	SQuaD v1.1	V100-SXM2-32GB
		Attention	37 sentences/sec	1x V100	DGX-2H	19.07-py3	Mixed	10	SQuaD v1.1	V100-SXM3-32GB-H
		Attention	147 sentences/sec	8xV100	DGX-1	19.06-py3	Mixed	10	SQuaD v1.1	V100-SXM2-32GB
		Attention	189 sentences/sec	8x V100	DGX-2H	19.07-py3	Mixed	10	SQuaD v1.1	V100-SXM3-32GB-H

T4 Training Performance

Framework	Network	Network Type	Throughput	GPU	Server	Container	Precision	Batch Size	Dataset	GPU Version
MXNet	Inception V3	CNN	174 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	Mixed	128	ImageNet2012	Tesla T4
		CNN	1381 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	Mixed	128	ImageNet2012	Tesla T4
	ResNet-50 v1.5	CNN	446 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	Mixed	208	ImageNet2012	Tesla T4
		CNN	4116 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.05-py3	Mixed	208	ImageNet2012	Tesla T4

PyTorch	Inception V3	CNN	176 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	Mixed	128	ImageNet2012	Tesla T4
		CNN	1337 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	Mixed	128	ImageNet2012	Tesla T4
	Mask R-CNN	CNN	6 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.05-py3	Mixed	4	COCO2014	Tesla T4
		CNN	38 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	Mixed	4	COCO2014	Tesla T4
	ResNet-50 v1.5	CNN	286 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	Mixed	256	ImageNet2012	Tesla T4
		CNN	2295 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	Mixed	256	ImageNet2012	Tesla T4
	SSD v1.1	CNN	76 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	Mixed	64	COCO 2017	Tesla T4
		CNN	622 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	Mixed	64	COCO 2017	Tesla T4
	Tacotron2	CNN	11236 total input tokens/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	Mixed	80	LJ Speech 1.1	Tesla T4
		CNN	78803 total input tokens/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	Mixed	80	LJ Speech 1.1	Tesla T4
	WaveGlow	CNN	32167 output samples/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	Mixed	8	LJ Speech 1.1	Tesla T4
		CNN	250173 output samples/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.05-py3	Mixed	8	LJ Speech 1.1	Tesla T4

TensorFlow	Inception V3	CNN	177 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	Mixed	128	ImageNet2012	Tesla T4
		CNN	1345 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	Mixed	128	ImageNet2012	Tesla T4
	ResNet-50 v1.5	CNN	263 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.05-py3	Mixed	256	ImageNet2012	Tesla T4
		CNN	2057 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	Mixed	256	ImageNet2012	Tesla T4
	SSD v1.1	CNN	51 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	Mixed	32	COCO 2017	Tesla T4
		CNN	281 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.06-py3	Mixed	32	COCO 2017	Tesla T4
	U-Net Industrial	CNN	28 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	Mixed	16	DAGM2007	Tesla T4
		CNN	190 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	Mixed	2	DAGM2007	Tesla T4

PyTorch	GNMT V2	RNN	25288 total tokens/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	Mixed	128	WMT16 English-German	Tesla T4
		RNN	182072 total tokens/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	Mixed	128	WMT16 English-German	Tesla T4

TensorFlow	GNMT V2	RNN	9679 total tokens/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	Mixed	192	WMT16 English-German	Tesla T4
		RNN	57464 total tokens/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	Mixed	192	WMT16 English-German	Tesla T4

PyTorch	NCF	Recommender	7584587 samples/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	Mixed	1048576	MovieLens 20 Million	Tesla T4
		Recommender	27011297 samples/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	Mixed	1048576	MovieLens 20 Million	Tesla T4

TensorFlow	NCF	Recommender	10297010 samples/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	Mixed	1048576	MovieLens 20 Million	Tesla T4
		Recommender	16872638 samples/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	Mixed	1048576	MovieLens 20 Million	Tesla T4

TensorFlow	BERT	Attention	9 sentences/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	Mixed	3	SQuaD v1.1	Tesla T4
		Attention	30 sentences/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	Mixed	3	SQuaD v1.1	Tesla T4

NVIDIA® TensorRT™ running on NVIDIA GPUs enable the most efficient deep learning inference performance across multiple application areas and models. This versatility provides wide latitude to data scientists to create the optimal low-latency solution. Visit NVIDIA GPU Cloud (NGC) to download any of these containers.

NVIDIA Tesla® V100 Tensor Cores GPUs leverage mixed-precision to combine high throughput with low latencies across every type of neural network. Tesla P4 is an inference GPU, designed for optimal power consumption and latency, for ultra-efficient scale-out servers. Read the inference whitepaper to learn more about NVIDIA’s inference platform.

Measuring the inference performance involves balancing a lot of variables. PLASTER is an acronym that describes the key elements for measuring deep learning performance. Each letter identifies a factor (Programmability, Latency, Accuracy, Size of Model, Throughput, Energy Efficiency, Rate of Learning) that must be considered to arrive at the right set of tradeoffs and to produce a successful deep learning implementation. Refer to NVIDIA’s PLASTER whitepaper for more details.

Inference Image Classification on CNNs with TensorRT

ResNet-50 v1.5 Throughput

ResNet-50 v1.5 Latency

ResNet-50 v1.5 Power Efficiency

Inference Performance

V100 Inference Performance

Network	Network Type	Batch Size	Throughput	Efficiency	Latency	GPU	Server	Container	Precision	Dataset	GPU Version
GoogleNet	CNN	1	1602 images/sec	15 images/sec/watt	0.62	1x V100	DGX-1	19.07-py3	INT8	Synthetic	V100-SXM2-16GB
	CNN	2	2162 images/sec	18 images/sec/watt	0.93	1x V100	DGX-1	19.07-py3	INT8	Synthetic	V100-SXM2-16GB
	CNN	8	5159 images/sec	35 images/sec/watt	1.6	1x V100	DGX-1	19.07-py3	INT8	Synthetic	V100-SXM2-16GB
	CNN	82	11869 images/sec	45 images/sec/watt	6.9	1x V100	DGX-1	19.07-py3	INT8	Synthetic	V100-SXM2-16GB
	CNN	128	12400 images/sec	44 images/sec/watt	10	1x V100	DGX-1	19.07-py3	INT8	Synthetic	V100-SXM2-16GB

MobileNet V1	CNN	1	3623 images/sec	29 images/sec/watt	0.28	1x V100	DGX-1	-	INT8	Synthetic	V100-SXM2-16GB
	CNN	2	5594 images/sec	45 images/sec/watt	0.36	1x V100	DGX-1	-	INT8	Synthetic	V100-SXM2-16GB
	CNN	8	14532 images/sec	89 images/sec/watt	0.55	1x V100	DGX-1	-	INT8	Synthetic	V100-SXM2-16GB
	CNN	128	29567 images/sec	101 images/sec/watt	4.3	1x V100	DGX-1	-	INT8	Synthetic	V100-SXM2-16GB

ResNet-50	CNN	1	1156 images/sec	8.7 images/sec/watt	0.87	1x V100	DGX-1	19.07-py3	INT8	Synthetic	V100-SXM2-16GB
	CNN	2	1580 images/sec	10 images/sec/watt	1.3	1x V100	DGX-1	19.07-py3	INT8	Synthetic	V100-SXM2-16GB
	CNN	8	3315 images/sec	21 images/sec/watt	2.4	1x V100	DGX-1	19.07-py3	Mixed	Synthetic	V100-SXM2-16GB
	CNN	128	7595 images/sec	27 images/sec/watt	17	1x V100	DGX-1	19.07-py3	Mixed	Synthetic	V100-SXM2-16GB
	CNN	128	7803 images/sec	23 images/sec/watt	16	1x V100	DGX-2	19.07-py3	Mixed	Synthetic	V100-SXM3-32GB

ResNet-50v1.5	CNN	1	943 images/sec	7.1 images/sec/watt	1.1	1x V100	DGX-1	19.07-py3	INT8	Synthetic	V100-SXM2-16GB
	CNN	2	1396 images/sec	9.7 images/sec/watt	1.4	1x V100	DGX-1	19.07-py3	INT8	Synthetic	V100-SXM2-16GB
	CNN	8	3226 images/sec	19 images/sec/watt	2.5	1x V100	DGX-1	19.07-py3	Mixed	Synthetic	V100-SXM2-16GB
	CNN	128	7223 images/sec	25 images/sec/watt	18	1x V100	DGX-1	19.07-py3	Mixed	Synthetic	V100-SXM2-16GB
	CNN	128	7,459 images/sec	22 images/sec/watt	17	1x V100	DGX-2	19.07-py3	Mixed	Synthetic	V100-SXM2-32GB

VGG16	CNN	1	821 images/sec	4 images/sec/watt	1.2	1x V100	DGX-1	19.07-py3	INT8	Synthetic	V100-SXM2-16GB
	CNN	2	893 images/sec	4.3 images/sec/watt	2.2	1x V100	DGX-1	19.07-py3	Mixed	Synthetic	V100-SXM2-16GB
	CNN	8	1846 images/sec	7.4 images/sec/watt	4.3	1x V100	DGX-1	19.07-py3	Mixed	Synthetic	V100-SXM2-16GB
	CNN	128	2845 images/sec	9.7 images/sec/watt	45	1x V100	DGX-1	19.07-py3	Mixed	Synthetic	V100-SXM2-16GB

NMT	RNN	1	4124 total tokens/sec	tokens/sec/watt	12	1x V100	DGX-1	-	Mixed	wmt16-English-German	V100-SXM2-32GB
	RNN	2	6680 total tokens/sec	tokens/sec/watt	15	1x V100	DGX-1	-	Mixed	wmt16-English-German	V100-SXM2-32GB
	RNN	64	58591 total tokens/sec	tokens/sec/watt	56	1x V100	DGX-1	-	Mixed	wmt16-English-German	V100-SXM2-32GB
	RNN	128	73426 total tokens/sec	tokens/sec/watt	90	1x V100	DGX-1	-	Mixed	wmt16-English-German	V100-SXM2-32GB

NCF	Recommender	1048576	56452298 samples/sec	125923 samples/sec/watt		1x V100	DGX-1	19.07-py3	Mixed	Synthetic	V100-SXM2-16GB

BERT-LARGE	Attention	8	105 sentences/sec	0.76 sentences/sec/watt		1x V100	DGX-1	19.07-py3	Mixed	SQuaD v1.1	V100-SXM2-16GB

TensorRT 5.1 | TensorFlow for NCF and BERT-LARGE

T4 Inference Performance

Network	Network Type	Batch Size	Throughput	Efficiency	Latency	GPU	Server	Container	Precision	Dataset	GPU Version
GoogleNet	CNN	1	1681 images/sec	25 images/sec/watt	0.6	1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	INT8	Synthetic	Tesla T4
	CNN	2	2319 images/sec	35 images/sec/watt	0.86	1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	INT8	Synthetic	Tesla T4
	CNN	8	5723 images/sec	83 images/sec/watt	1.4	1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	INT8	Synthetic	Tesla T4
	CNN	52	7580 images/sec	109 images/sec/watt	6.9	1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	INT8	Synthetic	Tesla T4
	CNN	128	7699 images/sec	111 images/sec/watt	17	1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	INT8	Synthetic	Tesla T4

MobileNet V1	CNN	1	3685 images/sec	70 images/sec/watt	0.27	1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	INT8	Synthetic	Tesla T4
	CNN	2	6006 images/sec	97 images/sec/watt	0.33	1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	INT8	Synthetic	Tesla T4
	CNN	8	13859 images/sec	199 images/sec/watt	0.58	1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	INT8	Synthetic	Tesla T4
	CNN	128	17438 images/sec	251 images/sec/watt	7.3	1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	INT8	Synthetic	Tesla T4

ResNet-50	CNN	1	1077 images/sec	15 images/sec/watt	0.93	1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	INT8	Synthetic	Tesla T4
	CNN	2	1702 images/sec	26 images/sec/watt	1.2	1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	INT8	Synthetic	Tesla T4
	CNN	8	3854 images/sec	55 images/sec/watt	2.1	1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	INT8	Synthetic	Tesla T4
	CNN	128	5288 images/sec	76 images/sec/watt	24	1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	INT8	Synthetic	Tesla T4

ResNet-50 v1.5	CNN	1	1037 images/sec	15 images/sec/watt	0.96	1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	INT8	Synthetic	Tesla T4
	CNN	2	1710 images/sec	26 images/sec/watt	1.2	1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	INT8	Synthetic	Tesla T4
	CNN	8	3730 images/sec	53 images/sec/watt	2.2	1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	INT8	Synthetic	Tesla T4
	CNN	128	5013 images/sec	71 images/sec/watt	26	1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	INT8	Synthetic	Tesla T4

VGG16	CNN	1	726 images/sec	10 images/sec/watt	1.4	1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	INT8	Synthetic	Tesla T4
	CNN	2	1064 images/sec	15 images/sec/watt	1.9	1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	INT8	Synthetic	Tesla T4
	CNN	8	1670 images/sec	24 images/sec/watt	4.8	1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	INT8	Synthetic	Tesla T4
	CNN	128	1956 images/sec	28 images/sec/watt	65	1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	INT8	Synthetic	Tesla T4

NCF	Recommender	1	7716 samples/sec	281 samples/sec/watt	0.14	1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	INT8	Synthetic	Tesla T4
	Recommender	64	491050 samples/sec	16957 samples/sec/watt	0.14	1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	INT8	Synthetic	Tesla T4
	Recommender	25000	53908841 samples/sec	782056 samples/sec/watt	1.9	1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	INT8	Synthetic	Tesla T4
	Recommender	100000	53908841 samples/sec	782056 samples/sec/watt	1.9	1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	INT8	Synthetic	Tesla T4

BERT-LARGE	Attention	8	42 sentences/sec	0.80 sentences/sec/watt		1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	Mixed	SQuaD v1.1	Tesla T4

TensorRT 5.1 | TensorFlow for BERT-LARGE

Last updated: August 19th, 2019

Search form

NVIDIA Tesla Deep Learning Product Performance

Training

Inference

NVIDIA Performance on MLPerf 0.6 AI Benchmarks

Training Image Classification on CNNs

Training Performance

NVIDIA Performance on MLPerf 0.6 AI Benchmarks

V100 Training Performance

T4 Training Performance

Inference Image Classification on CNNs with TensorRT

Inference Performance

V100 Inference Performance

T4 Inference Performance