test | NVIDIA Developer

NVIDIA Performance on MLPerf 0.6 AI Benchmarks

ResNet-50 v1.5 Time to Solution on V100

MXNet | Batch Size refer to CNN V100 Training table below | Precision: Mixed | Dataset: ImageNet2012 | Convergence criteria - refer to MLPerf requirements

Training Image Classification on CNNs

ResNet-50 V1.5 Throughput on V100

DGX-1: 8x Tesla V100-SXM2-32GB, E5-2698 v4 2.2 GHz | Batch Size = 256 | MXNet = 19.06-py3, Tensorflow and PyTorch = 19.07_py3 | Precision: Mixed | Dataset: ImageNet2012

ResNet-50 V1.5 Throughput on T4

Supermicro SYS-4029GP-TRT T4: 8x Tesla T4 16GB, Gold 6140 2.3 GHz | Batch Size = 208 for MXNet, PyTorch = 256, TensorFlow = 128 | MXNet and TensorFlow = 19.05-py3, PyTorch = 19.07_py3 | Precision: Mixed | Dataset: ImageNet2012

Training Performance

NVIDIA Performance on MLPerf 0.6 AI Benchmarks

Framework	Network	Network Type	Time to Solution	GPU	Server	MLPerf-ID	Precision	Dataset	GPU Version
MXNet	ResNet-50 v1.5	CNN	115.22 minutes	8x V100	DGX-1	0.6-8	Mixed	ImageNet2012	V100-SXM2-16GB
		CNN	57.87 minutes	16x V100	DGX-2	0.6-17	Mixed	ImageNet2012	V100-SXM3-32GB
		CNN	52.74 minutes	16x V100	DGX-2H	0.6-19	Mixed	ImageNet2012	V100-SXM3-32GB-H
		CNN	2.59 minutes	512x V100	DGX-2H	0.6-29	Mixed	ImageNet2012	V100-SXM3-32GB-H
		CNN	1.69 minutes	1040x V100	DGX-1	0.6-16	Mixed	ImageNet2012	V100-SXM2-16GB
		CNN	1.33 minutes	1536x V100	DGX-2H	0.6-30	Mixed	ImageNet2012	V100-SXM3-32GB-H

PyTorch	SSD-ResNet-34	CNN	22.36 minutes	8x V100	DGX-1	0.6-9	Mixed	COCO2017	V100-SXM2-16GB
		CNN	12.21 minutes	16x V100	DGX-2	0.6-18	Mixed	COCO2017	V100-SXM3-32GB
		CNN	11.41 minutes	16x V100	DGX-2H	0.6-20	Mixed	COCO2017	V100-SXM3-32GB-H
		CNN	4.78 minutes	64x V100	DGX-2H	0.6-21	Mixed	COCO2017	V100-SXM3-32GB-H
		CNN	2.67 minutes	240x V100	DGX-1	0.6-13	Mixed	COCO2017	V100-SXM2-16GB
		CNN	2.56 minutes	240x V100	DGX-2H	0.6-24	Mixed	COCO2017	V100-SXM3-32GB-H
		CNN	2.23 minutes	240x V100	DGX-2H	0.6-27	Mixed	COCO2017	V100-SXM3-32GB-H
	Mask R-CNN	CNN	207.48 minutes	8x V100	DGX-1	0.6-9	Mixed	COCO2017	V100-SXM2-16GB
		CNN	101 minutes	16x V100	DGX-2	0.6-18	Mixed	COCO2017	V100-SXM3-32GB
		CNN	95.2 minutes	16x V100	DGX-2H	0.6-20	Mixed	COCO2017	V100-SXM3-32GB-H
		CNN	32.72 minutes	64x V100	DGX-2H	0.6-21	Mixed	COCO2017	V100-SXM3-32GB-H
		CNN	22.03 minutes	192x V100	DGX-1	0.6-12	Mixed	COCO2017	V100-SXM2-16GB
		CNN	18.47 minutes	192x V100	DGX-2H	0.6-23	Mixed	COCO2017	V100-SXM3-32GB-H

PyTorch	GNMT	RNN	20.55 minutes	8x V100	DGX-1	0.6-9	Mixed	WMT16 English-German	V100-SXM2-16GB
		RNN	10.94 minutes	16x V100	DGX-2	0.6-18	Mixed	WMT16 English-German	V100-SXM3-32GB
		RNN	9.87 minutes	16x V100	DGX-2H	0.6-20	Mixed	WMT16 English-German	V100-SXM3-32GB-H
		RNN	2.12 minutes	256x V100	DGX-2H	0.6-25	Mixed	WMT16 English-German	V100-SXM3-32GB-H
		RNN	1.99 minutes	384x V100	DGX-1	0.6-14	Mixed	WMT16 English-German	V100-SXM2-16GB
		RNN	1.8 minutes	384x V100	DGX-2H	0.6-26	Mixed	WMT16 English-German	V100-SXM3-32GB-H

PyTorch	Transformer	Attention	20.34 minutes	8x V100	DGX-1	0.6-9	Mixed	WMT17 English-German	V100-SXM2-16GB
		Attention	11.04 minutes	16x V100	DGX-2	0.6-18	Mixed	WMT17 English-German	V100-SXM3-32GB
		Attention	9.8 minutes	16x V100	DGX-2H	0.6-20	Mixed	WMT17 English-German	V100-SXM3-32GB-H
		Attention	2.41 minutes	160x V100	DGX-2H	0.6-22	Mixed	WMT17 English-German	V100-SXM3-32GB-H
		Attention	2.05 minutes	480x V100	DGX-1	0.6-15	Mixed	WMT17 English-German	V100-SXM2-16GB
		Attention	1.59 minutes	480x V100	DGX-2H	0.6-28	Mixed	WMT17 English-German	V100-SXM3-32GB-H

TensorFlow	MiniGo	Reinforcement Learning	27.39 minutes	8x V100	DGX-1	0.6-10	Mixed	N/A	V100-SXM2-16GB
		Reinforcement Learning	13.57 minutes	24x V100	DGX-1	0.6-11	Mixed	N/A	V100-SXM2-16GB

V100 Training Performance

Framework	Network	Network Type	Throughput	GPU	Server	Container	Precision	Batch Size	Dataset	GPU Version
MXNet	Inception V3	CNN	527 images/sec	1x V100	DGX-1	19.07-py3	Mixed	256	ImageNet2012	V100-SXM2-32GB
		CNN	606 images/sec	1x V100	DGX-2H	19.07-py3	Mixed	256	ImageNet2012	V100-SXM3-32GB-H
		CNN	4105 images/sec	8x V100	DGX-1	19.07-py3	Mixed	256	ImageNet2012	V100-SXM2-32GB
		CNN	4648 images/sec	8x V100	DGX-2H	19.07-py3	Mixed	256	ImageNet2012	V100-SXM3-32GB-H
	ResNet-50	CNN	1409 images/sec	1x V100	DGX-1	19.02-py3	Mixed	128	ImageNet2012	V100-SXM2-16GB
		CNN	1442 images/sec	1x V100	DGX-2	19.02-py3	Mixed	256	ImageNet2012	V100-SXM3-32GB
		CNN	10380 images/sec	8x V100	DGX-1	19.02-py3	Mixed	128	ImageNet2012	V100-SXM2-16GB
		CNN	10530 images/sec	8x V100	DGX-2	19.02-py3	Mixed	256	ImageNet2012	V100-SXM3-32GB
	ResNet-50 v1.5	CNN	1422 images/sec	1x V100	DGX-1	19.07-py3	Mixed	208	ImageNet2012	V100-SXM2-16GB
		CNN	1597 images/sec	1x V100	DGX-2H	19.07-py3	Mixed	256	ImageNet2012	V100-SXM3-32GB-H
		CNN	9566 images/sec	8x V100	DGX-1	19.06-py3	Mixed	256	ImageNet2012	V100-SXM2-32GB
		CNN	11056 images/sec	8x V100	DGX-2	19.05-py3	Mixed	128	ImageNet2012	V100-SXM3-32GB
		CNN	11507 images/sec	8x V100	DGX-2H	19.05-py3	Mixed	256	ImageNet2012	V100-SXM3-32GB-H

PyTorch	Inception V3	CNN	543 images/sec	1x V100	DGX-1	19.07-py3	Mixed	256	ImageNet2012	V100-SXM2-32GB
		CNN	629 images/sec	1x V100	DGX-2H	19.07-py3	Mixed	256	ImageNet2012	V100-SXM3-32GB-H
		CNN	4156 images/sec	8x V100	DGX-1	19.03-py3	Mixed	256	ImageNet2012	V100-SXM2-32GB
	Mask R-CNN	CNN	14 images/sec	1x V100	DGX-1	19.07-py3	Mixed	4	COCO2014	V100-SXM2-32GB
		CNN	17 images/sec	1x V100	DGX-2H	19.05-py3	Mixed	16	COCO2014	V100-SXM3-32GB-H
		CNN	88 images/sec	8x V100	DGX-1	19.06-py3	Mixed	16	COCO2014	V100-SXM2-32GB
	ResNet-50	CNN	819 images/sec	1x V100	DGX-1	19.02_py3	Mixed	256	ImageNet2012	V100-SXM2-16GB
		CNN	820 images/sec	1x V100	DGX-2	19.02-py3	Mixed	256	ImageNet2012	V100-SXM3-32GB
		CNN	6218 images/sec	8x V100	DGX-1	19.02-py3	Mixed	256	ImageNet2012	V100-SXM2-16GB
	ResNet-50 v1.5	CNN	928 images/sec	1x V100	DGX-1	19.07-py3	Mixed	256	ImageNet2012	V100-SXM2-16GB
		CNN	1036 images/sec	1x V100	DGX-2H	19.07-py3	Mixed	256	ImageNet2012	V100-SXM3-32GB-H
		CNN	7288 images/sec	8x V100	DGX-1	19.07-py3	Mixed	256	ImageNet2012	V100-SXM2-16GB
	SSD v1.1	CNN	225 images/sec	1x V100	DGX-1	19.07-py3	Mixed	64	COCO 2017	V100-SXM2-32GB
		CNN	299 images/sec	1x V100	DGX-2H	19.05-py3	Mixed	64	COCO 2017	V100-SXM3-32GB-H
		CNN	2018 images/sec	8x V100	DGX-1	19.06-py3	Mixed	64	COCO 2017	V100-SXM2-32GB
	Tacotron2	CNN	11743 total input tokens/sec	1x V100	DGX-1	19.07-py3	Mixed	80	LJ Speech 1.1	V100-SXM2-16GB
		CNN	16847 total input tokens/sec	1x V100	DGX-2H	19.07-py3	Mixed	80	LJ Speech 1.1	V100-SXM3-32GB-H
		CNN	81410 total input tokens/sec	8x V100	DGX-1	19.07-py3	Mixed	80	LJ Speech 1.1	V100-SXM2-32GB
		CNN	104447 total input tokens/sec	8x V100	DGX-2H	19.07-py3	Mixed	80	LJ Speech 1.1	V100-SXM3-32GB-H
	WaveGlow	CNN	77780 output samples/sec	1x V100	DGX-1	19.05-py3	Mixed	8	LJ Speech 1.1	V100-SXM2-16GB
		CNN	91052 output samples/sec	1x V100	DGX-2H	19.05-py3	Mixed	8	LJ Speech 1.1	V100-SXM3-32GB-H
		CNN	533364 output samples/sec	8x V100	DGX-1	19.05-py3	Mixed	8	LJ Speech 1.1	V100-SXM2-32GB

TensorFlow	Inception V3	CNN	542 images/sec	1x V100	DGX-1	19.07-py3	Mixed	256	ImageNet2012	V100-SXM2-32GB
		CNN	626 images/sec	1x V100	DGX-2H	19.07-py3	Mixed	256	ImageNet2012	V100-SXM3-32GB-H
		CNN	4097 images/sec	8x V100	DGX-1	19.07-py3	Mixed	256	ImageNet2012	V100-SXM2-32GB
	ResNet-50 v1.5	CNN	840 images/sec	1x V100	DGX-1	19.07-py3	Mixed	256	ImageNet2012	V100-SXM2-16GB
		CNN	965 images/sec	1x V100	DGX-2H	19.07-py3	Mixed	256	ImageNet2012	V100-SXM3-32GB-H
		CNN	6474 images/sec	8x V100	DGX-1	19.07-py3	Mixed	256	ImageNet2012	V100-SXM2-16GB
	SSD v1.1	CNN	114 images/sec	1x V100	DGX-1	19.07-py3	Mixed	32	COCO 2017	V100-SXM2-32GB
		CNN	125 images/sec	1x V100	DGX-2	19.07-py3	Mixed	32	COCO 2017	V100-SXM3-32GB
		CNN	665 images/sec	8x V100	DGX-1	19.05-py3	Mixed	32	COCO 2017	V100-SXM2-32GB
		CNN	770 images/sec	8x V100	DGX-2	19.05-py3	Mixed	32	COCO 2017	V100-SXM3-32GB
	U-Net Industrial	CNN	95 images/sec	1x V100	DGX-1	19.07-py3	Mixed	16	DAGM2007	V100-SXM2-32GB
		CNN	110 images/sec	1x V100	DGX-2H	19.07-py3	Mixed	16	DAGM2007	V100-SXM3-32GB-H
		CNN	492 images/sec	8x V100	DGX-1	19.07-py3	Mixed	2	DAGM2007	V100-SXM2-32GB
		CNN	515 images/sec	8x V100	DGX-2	19.07-py3	Mixed	2	DAGM2007	V100-SXM3-32GB

PyTorch	GNMT V2	RNN	75975 total tokens/sec	1x V100	DGX-1	19.07-py3	Mixed	128	WMT16 English-German	V100-SXM2-32GB
		RNN	82766 total tokens/sec	1x V100	DGX-2	19.07-py3	Mixed	128	WMT16 English-German	V100-SXM3-32GB
		RNN	582605 total tokens/sec	8x V100	DGX-1	19.07-py3	Mixed	128	WMT16 English-German	V100-SXM2-32GB

TensorFlow	GNMT V2	RNN	22471 total tokens/sec	1x V100	DGX-1	19.07-py3	Mixed	192	WMT16 English-German	V100-SXM2-16GB
		RNN	26039 total tokens/sec	1x V100	DGX-2H	19.07-py3	Mixed	192	WMT16 English-German	V100-SXM3-32GB-H
		RNN	149008 total tokens/sec	8x V100	DGX-1	19.07-py3	Mixed	192	WMT16 English-German	V100-SXM2-16GB

PyTorch	NCF	Recommender	22093850 samples/sec	1x V100	DGX-1	19.07-py3	Mixed	1048576	MovieLens 20 Million	V100-SXM2-16GB
		Recommender	24473776 samples/sec	1x V100	DGX-2H	19.07-py3	Mixed	1048576	MovieLens 20 Million	V100-SXM3-32GB-H
		Recommender	104122673 samples/sec	8x V100	DGX-1	19.07-py3	Mixed	1048576	MovieLens 20 Million	V100-SXM2-16GB
		Recommender	109969915 samples/sec	8x V100	DGX-2H	19.07-py3	Mixed	1048576	MovieLens 20 Million	V100-SXM3-32GB-H

TensorFlow	NCF	Recommender	26415693 samples/sec	1x V100	DGX-1	19.07-py3	Mixed	1048576	MovieLens 20 Million	V100-SXM2-16GB
		Recommender	56183685 samples/sec	8x V100	DGX-1	19.07-py3	Mixed	1048576	MovieLens 20 Million	V100-SXM2-32GB

TensorFlow	BERT-LARGE	Attention	32 sentences/sec	1x V100	DGX-1	19.07-py3	Mixed	10	SQuaD v1.1	V100-SXM2-32GB
		Attention	37 sentences/sec	1x V100	DGX-2H	19.07-py3	Mixed	10	SQuaD v1.1	V100-SXM3-32GB-H
		Attention	147 sentences/sec	8xV100	DGX-1	19.06-py3	Mixed	10	SQuaD v1.1	V100-SXM2-32GB
		Attention	189 sentences/sec	8x V100	DGX-2H	19.07-py3	Mixed	10	SQuaD v1.1	V100-SXM3-32GB-H

T4 Training Performance

Framework	Network	Network Type	Throughput	GPU	Server	Container	Precision	Batch Size	Dataset	GPU Version
MXNet	Inception V3	CNN	174 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	Mixed	128	ImageNet2012	Tesla T4
		CNN	1381 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	Mixed	128	ImageNet2012	Tesla T4
	ResNet-50 v1.5	CNN	446 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	Mixed	208	ImageNet2012	Tesla T4
		CNN	4116 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.05-py3	Mixed	208	ImageNet2012	Tesla T4

PyTorch	Inception V3	CNN	176 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	Mixed	128	ImageNet2012	Tesla T4
		CNN	1337 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	Mixed	128	ImageNet2012	Tesla T4
	Mask R-CNN	CNN	6 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.05-py3	Mixed	4	COCO2014	Tesla T4
		CNN	38 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	Mixed	4	COCO2014	Tesla T4
	ResNet-50 v1.5	CNN	286 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	Mixed	256	ImageNet2012	Tesla T4
		CNN	2295 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	Mixed	256	ImageNet2012	Tesla T4
	SSD v1.1	CNN	76 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	Mixed	64	COCO 2017	Tesla T4
		CNN	622 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	Mixed	64	COCO 2017	Tesla T4
	Tacotron2	CNN	11236 total input tokens/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	Mixed	80	LJ Speech 1.1	Tesla T4
		CNN	78803 total input tokens/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	Mixed	80	LJ Speech 1.1	Tesla T4
	WaveGlow	CNN	32167 output samples/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	Mixed	8	LJ Speech 1.1	Tesla T4
		CNN	250173 output samples/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.05-py3	Mixed	8	LJ Speech 1.1	Tesla T4

TensorFlow	Inception V3	CNN	177 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	Mixed	128	ImageNet2012	Tesla T4
		CNN	1345 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	Mixed	128	ImageNet2012	Tesla T4
	ResNet-50 v1.5	CNN	263 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.05-py3	Mixed	256	ImageNet2012	Tesla T4
		CNN	2057 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	Mixed	256	ImageNet2012	Tesla T4
	SSD v1.1	CNN	51 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	Mixed	32	COCO 2017	Tesla T4
		CNN	281 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.06-py3	Mixed	32	COCO 2017	Tesla T4
	U-Net Industrial	CNN	28 images/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	Mixed	16	DAGM2007	Tesla T4
		CNN	190 images/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	Mixed	2	DAGM2007	Tesla T4

PyTorch	GNMT V2	RNN	25288 total tokens/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	Mixed	128	WMT16 English-German	Tesla T4
		RNN	182072 total tokens/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	Mixed	128	WMT16 English-German	Tesla T4

TensorFlow	GNMT V2	RNN	9679 total tokens/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	Mixed	192	WMT16 English-German	Tesla T4
		RNN	57464 total tokens/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	Mixed	192	WMT16 English-German	Tesla T4

PyTorch	NCF	Recommender	7584587 samples/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	Mixed	1048576	MovieLens 20 Million	Tesla T4
		Recommender	27011297 samples/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	Mixed	1048576	MovieLens 20 Million	Tesla T4

TensorFlow	NCF	Recommender	10297010 samples/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	Mixed	1048576	MovieLens 20 Million	Tesla T4
		Recommender	16872638 samples/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	Mixed	1048576	MovieLens 20 Million	Tesla T4

TensorFlow	BERT	Attention	9 sentences/sec	1x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	Mixed	3	SQuaD v1.1	Tesla T4
		Attention	30 sentences/sec	8x T4	Supermicro SYS-4029GP-TRT T4	19.07-py3	Mixed	3	SQuaD v1.1	Tesla T4