NVIDIA Data Center Deep Learning Product Performance

Review the latest GPU acceleration factors of popular HPC applications.

Please refer to Measuring Training and Inferencing Performance on NVIDIA AI Platforms Reviewer’s Guide for instructions on how to reproduce these performance claims.

Deploying AI in real world applications, requires training the networks to convergence at a specified accuracy. This is the best methodology to test AI systems- where they are ready to be deployed in the field, as the networks can then deliver meaningful results (for example, correctly performing image recognition on video streams). Read our blog on convergence for more details. Training that does not converge is a measurement of hardware’s throughput capabilities on the specified AI network, but is not representative of real world applications.

NVIDIA’s complete solution stack, from GPUs to libraries, and containers on NVIDIA GPU Cloud (NGC), allows data scientists to quickly get up and running with deep learning. NVIDIA A100 Tensor Core GPUs provides unprecedented acceleration at every scale, setting records in MLPerf™, the AI industry’s leading benchmark and a testament to our accelerated platform approach.

NVIDIA Performance on MLPerf 0.7 AI Benchmarks

BERT Time to Train on A100

PyTorch | Precision: Mixed | Dataset: Wikipedia 2020/01/01 | Convergence criteria - refer to MLPerf requirements

MLPerf Training Performance

NVIDIA A100 Performance on MLPerf 0.7 AI Benchmarks - Closed Division

Framework	Network	Time to Train (mins)	MLPerf Quality Target	GPU	Server	MLPerf-ID	Precision	Dataset	GPU Version
MXNet	ResNet-50 v1.5	39.78	75.90% classification	8x A100	DGX A100	0.7-18	Mixed	ImageNet2012	A100-SXM4-40GB
		23.75	75.90% classification	16x A100	DGX A100	0.7-21	Mixed	ImageNet2012	A100-SXM4-40GB
		1.06	75.90% classification	768x A100	DGX A100	0.7-32	Mixed	ImageNet2012	A100-SXM4-40GB
		0.83	75.90% classification	1536x A100	DGX A100	0.7-35	Mixed	ImageNet2012	A100-SXM4-40GB
		0.76	75.90% classification	1840x A100	DGX A100	0.7-37	Mixed	ImageNet2012	A100-SXM4-40GB
	SSD	2.25	23.0% mAP	64x A100	DGX A100	0.7-25	Mixed	COCO2017	A100-SXM4-40GB
		0.89	23.0% mAP	512x A100	DGX A100	0.7-31	Mixed	COCO2017	A100-SXM4-40GB
		0.82	23.0% mAP	1024x A100	DGX A100	0.7-33	Mixed	COCO2017	A100-SXM4-40GB
PyTorch	BERT	49.01	0.712 Mask-LM accuracy	8x A100	DGX A100	0.7-19	Mixed	Wikipedia 2020/01/01	A100-SXM4-40GB
		30.63	0.712 Mask-LM accuracy	16x A100	DGX A100	0.7-22	Mixed	Wikipedia 2020/01/01	A100-SXM4-40GB
		3.36	0.712 Mask-LM accuracy	256x A100	DGX A100	0.7-28	Mixed	Wikipedia 2020/01/01	A100-SXM4-40GB
		1.48	0.712 Mask-LM accuracy	1024x A100	DGX A100	0.7-34	Mixed	Wikipedia 2020/01/01	A100-SXM4-40GB
		0.81	0.712 Mask-LM accuracy	2048x A100	DGX A100	0.7-38	Mixed	Wikipedia 2020/01/01	A100-SXM4-40GB
	DLRM	4.43	0.8025 AUC	8x A100	DGX A100	0.7-19	Mixed	Criteo AI Lab’s Terabyte Click-Through-Rate (CTR)	A100-SXM4-40GB
	GNMT	7.81	24.0 Sacre BLEU	8x A100	DGX A100	0.7-19	Mixed	WMT16 English-German	A100-SXM4-40GB
		4.94	24.0 Sacre BLEU	16x A100	DGX A100	0.7-22	Mixed	WMT16 English-German	A100-SXM4-40GB
		0.98	24.0 Sacre BLEU	256x A100	DGX A100	0.7-28	Mixed	WMT16 English-German	A100-SXM4-40GB
		0.71	24.0 Sacre BLEU	1024x A100	DGX A100	0.7-34	Mixed	WMT16 English-German	A100-SXM4-40GB
	Mask R-CNN	82.16	0.377 Box min AP and 0.339 Mask min AP	8x A100	DGX A100	0.7-19	Mixed	COCO2017	A100-SXM4-40GB
		44.21	0.377 Box min AP and 0.339 Mask min AP	16x A100	DGX A100	0.7-22	Mixed	COCO2017	A100-SXM4-40GB
		28.46	0.377 Box min AP and 0.339 Mask min AP	32x A100	DGX A100	0.7-24	Mixed	COCO2017	A100-SXM4-40GB
		10.46	0.377 Box min AP and 0.339 Mask min AP	256x A100	DGX A100	0.7-28	Mixed	COCO2017	A100-SXM4-40GB
	SSD	10.21	23.0% mAP	8x A100	DGX A100	0.7-19	Mixed	COCO2017	A100-SXM4-40GB
		5.68	23.0% mAP	16x A100	DGX A100	0.7-22	Mixed	COCO2017	A100-SXM4-40GB
	Transformer	7.84	25.00 BLEU	8x A100	DGX A100	0.7-19	Mixed	WMT17 English-German	A100-SXM4-40GB
		4.35	25.00 BLEU	16x A100	DGX A100	0.7-22	Mixed	WMT17 English-German	A100-SXM4-40GB
		1.8	25.00 BLEU	80x A100	DGX A100	0.7-26	Mixed	WMT17 English-German	A100-SXM4-40GB
		1.02	25.00 BLEU	160x A100	DGX A100	0.7-27	Mixed	WMT17 English-German	A100-SXM4-40GB
		0.62	25.00 BLEU	480x A100	DGX A100	0.7-30	Mixed	WMT17 English-German	A100-SXM4-40GB
TensorFlow	MiniGo	299.73	50% win rate vs. checkpoint	8x A100	DGX A100	0.7-20	Mixed	N/A	A100-SXM4-40GB
		165.72	50% win rate vs. checkpoint	16x A100	DGX A100	0.7-23	Mixed	N/A	A100-SXM4-40GB
		29.7	50% win rate vs. checkpoint	256x A100	DGX A100	0.7-29	Mixed	N/A	A100-SXM4-40GB
		17.07	50% win rate vs. checkpoint	1792x A100	DGX A100	0.7-36	Mixed	N/A	A100-SXM4-40GB
NVIDIA Merlin HugeCTR	DLRM	3.33	0.8025 AUC	8x A100	DGX A100	0.7-17	Mixed	Criteo AI Lab’s Terabyte Click-Through-Rate (CTR)	A100-SXM4-40GB

Converged Training Performance of NVIDIA A100, V100 and T4

Benchmarks are reproducible by following links to NGC scripts

A100 Training Performance

Framework	Network	Time to Train (mins)	Accuracy	Throughput	GPU	Server	Container	Precision	Batch Size	Dataset	GPU Version
MXNet	ResNet-50 v1.5	40	75.9 Top 1 Accuracy	22,008 images/sec	8x A100	DGX A100	20.06-py3	Mixed	408	ImageNet2012	A100-SXM4-40GB
PyTorch	Mask R-CNN	176	.34 AP Segm	167 images/sec	8x A100	DGX A100	20.12-py3	TF32	8	COCO 2014	A100-SXM-80GB
	ResNeXt101	300	79.37 Top 1	6,888 images/sec	8x A100	DGX A100	-	Mixed	128	ImageNet2012	A100-SXM4-40GB
	SE-ResNeXt101	341	78.82 Top 1	5,737 images/sec	8x A100	DGX A100	21.03-py3	Mixed	128	Imagenet2012	A100-SXM-80GB
	SSD v1.1	43	.25 mAP	3,092 images/sec	8x A100	DGX A100	20.06-py3	Mixed	128	COCO 2017	A100-SXM-80GB
	Tacotron2	103	.57 Training Loss	299,005 total output mels/sec	8x A100	DGX A100	21.02-py3	TF32	128	LJSpeech 1.1	A100-SXM-80GB
	WaveGlow	293	-5.85 Training Loss	1,434,954 output samples/sec	8x A100	DGX A100	21.03-py3	Mixed	10	LJSpeech 1.1	A100-SXM-80GB
	Jasper	3,600	3.53 dev-clean WER	603 sequences/sec	8x A100	DGX A100	20.06-py3	Mixed	64	LibriSpeech	A100 SXM4-40GB
	Transformer	167	27.76	582,721 words/sec	8x A100	DGX A100	20.06-py3	Mixed	10240	wmt14-en-de	A100 SXM4-40GB
	FastPitch	216	.18 Training Loss	1,040,206 frames/sec	8x A100	DGX A100	20.06-py3	Mixed	32	LJSpeech 1.1	A100 SXM4-40GB
	GNMT V2	17	24.02 BLEU Score	882,331 total tokens/sec	8x A100	DGX A100	20.12-py3	Mixed	128	wmt16-en-de	A100-SXM-80GB
	NCF	0.37	.96 Hit Rate at 10	153,085,874 samples/sec	8x A100	DGX A100	21.03-py3	Mixed	131072	MovieLens 20M	A100-SXM-80GB
	BERT-LARGE	3	91.03 F1	938 sequences/sec	8x A100	DGX A100	21.03-py3	Mixed	32	SQuaD v1.1	A100-SXM-80GB
	Transformer-XL Large	429	14.07 Perplexity	192,038 total tokens/sec	8x A100	DGX A100	20.12-py3	Mixed	16	WikiText-103	A100-SXM-80GB
	Transformer-XL Base	214	16.92 Perplexity	614,404 total tokens/sec	8x A100	DGX A100	20.12-py3	Mixed	128	WikiText-103	A100-SXM-80GB
	BERT-Large Pre-Training P1	2,379	-	3,231 sequences/sec	8x A100	DGX A100	20.06-py3	Mixed	-	Wikipedia+BookCorpus	A100-SXM4-40GB
	BERT-Large Pre-Training P2	1,377	1.34 Final Loss	630 sequences/sec	8x A100	DGX A100	20.06-py3	Mixed	-	Wikipedia+BookCorpus	A100-SXM4-40GB
	BERT-Large Pre-Training E2E	3,756	1.34 Final Loss	-	8x A100	DGX A100	20.06-py3	Mixed	-	Wikipedia+BookCorpus	A100-SXM4-40GB
Tensorflow	ResNext101	195	79.19 Top 1	9,939 images/sec	8x A100	DGX A100	21.02-py3	Mixed	256	Imagenet2012	A100-SXM-80GB
	Mask R-CNN	193	.34 AP Segm	137 samples/sec	8x A100	DGX A100	21.03-py3	Mixed	4	COCO 2014	A100-SXM-80GB
	U-Net Industrial	1	.99 IoU Threshold	1,027 images/sec	8x A100	DGX A100	21.03-py3	Mixed	2	DAGM2007	A100-SXM-80GB
	U-Net Medical	6	.9 DICE Score	952 images/sec	8x A100	DGX A100	21.03-py3	Mixed	8	EM segmentation challenge	A100-SXM-80GB
	VAE-CF	1	.43 NDCG@100	1,534,868 users processed/sec	8x A100	DGX A100	21.03-py3	TF32	3072	MovieLens 20M	A100-SXM-80GB
	Wide and Deep	107	.68 MAP at 12	1,111,976 samples/sec	8x A100	DGX A100	20.10-py3	Mixed	16384	Kaggle Outbrain Click Prediction	A100 SXM4-40GB
	BERT-LARGE	11	91.2 F1	847 sequences/sec	8x A100	DGX A100	21.03-py3	Mixed	24	SQuaD v1.1	A100-SXM-80GB
	Electra Fine Tuning	3	92.56 F1	2,461 sequences/sec	8x A100	DGX A100	21.03-py3	Mixed	32	SQuaD v1.1	A100 SXM 80GB
	EfficientNet-B4	4,231	82.81 Top 1	2,535 images/sec	8x A100	DGX A100	20.08-py3	Mixed	160	ImageNet2012	A100-SXM-80GB

FastPitch throughput metric frames/sec refers to mel-scale spectrogram frames/sec | Server with a hyphen is a pre-production server
BERT-Large = BERT-Large Fine Tuning (Squadv1.1) with Sequence Length of 384
BERT-Large Pre-Training Sequence Length for Phase 1 = 128 and Phase 2 = 512 | Batch Size for Phase 1 = 65,536 and Phase 2 = 32,768
EfficientNet-B4: Mixup = 0.2 | Auto-Augmentation | cuDNN Version = 8.0.5.39 | NCCL Version = 2.7.8

A30 Training Performance

Framework	Network	Time to Train (mins)	Accuracy	Throughput	GPU	Server	Container	Precision	Batch Size	Dataset	GPU Version
PyTorch	ResNet-50 v1.5	662	76.78 Top 1	2,963 images/sec	4x A30	GIGABYTE G482-Z52-SW-QZ-001	21.02-py3	Mixed	256	ImageNet2012	A30

A10 Training Performance

Framework	Network	Time to Train (mins)	Accuracy	Throughput	GPU	Server	Container	Precision	Batch Size	Dataset	GPU Version
PyTorch	NCF	1	.96 Hit Rate at 10	47,713,385 samples/sec	8x A10	GIGABYTE G482-Z52-00	21.03-py3	Mixed	131072	MovieLens 20M	A10
	BERT-LARGE	13	91.15 F1	235 sequences/sec	8x A10	GIGABYTE G482-Z52-00	21.03-py3	Mixed	10	SQuaD v1.1	A10
Tensorflow	U-Net Industrial	1	.99 IoU Threshold 0.95	541 images/sec	8x A10	GIGABYTE G482-Z52-00	21.03-py3	Mixed	2	DAGM2007	A10
	VAE-CF	1	.43 NDCG@100	637,683 users processed/sec	8x A10	GIGABYTE G482-Z52-00	21.03-py3	Mixed	3072	MovieLens 20M	A10

Server with a hyphen is a pre-production server
BERT-Large = BERT-Large Fine Tuning (Squadv1.1) with Sequence Length of 384

V100 Training Performance

Framework	Network	Time to Train (mins)	Accuracy	Throughput	GPU	Server	Container	Precision	Batch Size	Dataset	GPU Version
MXNet	ResNet-50 v1.5	168	77.03 Top 1	11,802 images/sec	8x V100	DGX-2	21.03-py3	Mixed	256	ImageNet2012	V100-SXM3-32GB
PyTorch	Mask R-CNN	269	.34 AP Segm	109 images/sec	8x V100	DGX-2	20.12-py3	Mixed	8	COCO 2014	V100-SXM3-32GB
	Tacotron2	192	.53 Training Loss	163,818 total output mels/sec	8x V100	DGX-2	21.03-py3	Mixed	104	LJSpeech 1.1	V100-SXM3-32GB
	WaveGlow	472	-5.75 Training Loss	904,322 output samples/sec	8x V100	DGX-2	21.03-py3	Mixed	10	LJSpeech 1.1	V100-SXM3-32GB
	Jasper	6,300	3.49 dev-clean WER	312 sequences/sec	8x V100	DGX-2	20.06-py3	Mixed	64	LibriSpeech	V100 SXM2-32GB
	Transformer	276	27.82 BLEU Score	223,245 words/sec	8x V100	DGX-1	20.06-py3	Mixed	5120	wmt14-en-de	V100 SXM2-16GB
	FastPitch	354	.18 Training Loss	570,968 frames/sec	8x V100	DGX-1	20.06-py3	Mixed	32	LJSpeech 1.1	V100 SXM2-16GB
	GNMT V2	31	24.03 BLEU Score	479,039 total tokens/sec	8x V100	DGX-2	20.10-py3	Mixed	128	wmt16-en-de	V100-SXM3-32GB
	NCF	0.55	.96 Hit Rate at 10	99,061,003 samples/sec	8x V100	DGX-2	21.03-py3	Mixed	131072	MovieLens 20M	V100-SXM3-32GB
	BERT-LARGE	8	91.23 F1	367 sequences/sec	8x V100	DGX-2	21.03-py3	Mixed	10	SQuaD v1.1	V100-SXM3-32GB
	Transformer-XL Base	118	22.15 Perplexity	279,880 total tokens/sec	8x V100	DGX-2	20.12-py3	Mixed	32	WikiText-103	V100-SXM3-32GB
Tensorflow	ResNet-50 v1.5	184	76.99 Top 1	10,491 images/sec	8x V100	DGX-2	21.03-py3	Mixed	256	ImageNet2012	V100-SXM3-32GB
	ResNext101	413	79.43 Top 1	4,697 images/sec	8x V100	DGX-2	21.02-py3	Mixed	128	Imagenet2012	V100-SXM3-32GB
	SE-ResNext101	498	79.96 Top 1	3,915 images/sec	8x V100	DGX-2	21.03-py3	Mixed	96	Imagenet2012	V100-SXM3-32GB
	U-Net Industrial	0.97	.99 IoU Threshold	661 images/sec	8x V100	DGX-2	21.03-py3	Mixed	2	DAGM2007	V100-SXM3-32GB
	U-Net Medical	14	.9 DICE Score	470 images/sec	8x V100	DGX-2	21.03-py3	Mixed	8	EM segmentation challenge	V100-SXM3-32GB
	VAE-CF	0.70	.43 NDCG@100	1,001,210 users processed/sec	8x V100	DGX-1	20.06-py3	Mixed	3072	MovieLens 20M	V100 SXM2-32GB
	Wide and Deep	185	.68 MAP at 12	643,334 samples/sec	8x V100	DGX-1	20.10-py3	Mixed	16384	Kaggle Outbrain Click Prediction	V100 SXM2-16GB
	BERT-LARGE	19	91.13 F1	331 sequences/sec	8x V100	DGX-2	21.03-py3	Mixed	10	SQuaD v1.1	V100-SXM3-32GB
	Electra Fine-Tuning	6	92.72 F1	1,051 images/sec	8x V100	DGX-1	20.07-py3	Mixed	16	SQuaD v1.1	V100 SXM2-16GB

FastPitch throughput metric frames/sec refers to mel-scale spectrogram frames/sec
BERT-Large = BERT-Large Fine Tuning (Squadv1.1) with Sequence Length of 384

T4 Training Performance

Framework	Network	Time to Train (mins)	Accuracy	Throughput	GPU	Server	Container	Precision	Batch Size	Dataset	GPU Version
MXNet	ResNet-50 v1.5	478	77.26 Top 1	4,085 images/sec	8x T4	Supermicro SYS-4029GP-TRT	21.03-py3	Mixed	192	ImageNet2012	NVIDIA T4
PyTorch	ResNeXt101	1,738	78.75 Top 1	1,124 images/sec	8x T4	Supermicro SYS-4029GP-TRT	20.10-py3	Mixed	128	Imagenet2012	NVIDIA T4
	Tacotron2	247	.53 Training Loss	123,614 total output mels/sec	8x T4	Supermicro SYS-4029GP-TRT	21.03-py3	Mixed	104	LJSpeech 1.1	NVIDIA T4
	WaveGlow	1,034	-5.74 Training Loss	406,920 output samples/sec	8x T4	Supermicro SYS-4029GP-TRT	21.03-py3	Mixed	10	LJSpeech 1.1	NVIDIA T4
	Transformer	2,288	27.65 BLEU Score	42,030 words/sec	8x T4	Supermicro SYS-4029GP-TRT	21.03-py3	Mixed	2560	wmt14-en-de	NVIDIA T4
	FastPitch	319	.21 Training Loss	281,406 frames/sec	8x T4	Supermicro SYS-4029GP-TRT	20.10-py3	Mixed	32	LJSpeech 1.1	NVIDIA T4
	GNMT V2	90	24.27 BLEU Score	160,009 total tokens/sec	8x T4	Supermicro SYS-4029GP-TRT	20.10-py3	Mixed	128	wmt16-en-de	NVIDIA T4
	NCF	2	.96 Hit Rate at 10	26,354,810 samples/sec	8x T4	Supermicro SYS-4029GP-TRT	21.03-py3	Mixed	131072	MovieLens 20M	NVIDIA T4
	BERT-LARGE	23	91.25 F1	127 sequences/sec	8x T4	Supermicro SYS-4029GP-TRT	21.03-py3	Mixed	10	SQuaD v1.1	NVIDIA T4
	Transformer-XL Base	333	22.21 Perplexity	98,917 total tokens/sec	8x T4	Supermicro SYS-4029GP-TRT	20.12-py3	Mixed	32	WikiText-103	NVIDIA T4
Tensorflow	U-Net Industrial	2	.99 IoU Threshold 0.95	292 images/sec	8x T4	Supermicro SYS-4029GP-TRT	21.03-py3	Mixed	2	DAGM2007	NVIDIA T4
	U-Net Medical	13	.89 DICE Score	151 images/sec	8x T4	Supermicro SYS-4029GP-TRT	21.02-py3	Mixed	8	EM segmentation challenge	NVIDIA T4
	VAE-CF	2	.43 NDCG@100	373,317 users processed/sec	8x T4	Supermicro SYS-4029GP-TRT	21.03-py3	Mixed	3072	MovieLens 20M	NVIDIA T4
	SSD	112	.28 mAP	549 images/sec	8x T4	Supermicro SYS-4029GP-TRT	20.12-py3	Mixed	32	COCO 2017	NVIDIA T4
	Mask R-CNN	492	.34 AP Segm	52 samples/sec	8x T4	Supermicro SYS-4029GP-TRT	21.03-py3	Mixed	4	COCO 2014	NVIDIA T4

FastPitch throughput metric frames/sec refers to mel-scale spectrogram frames/sec
BERT-Large = BERT-Large Fine Tuning (Squadv1.1) with Sequence Length of 384

Deploying AI in real world applications, requires training the networks to convergence at a specified accuracy. This is the best methodology to test AI systems, and is typically done on multi-accelerator systems (see the ‘Training-Convergence’ tab or read our blog on convergence for more details) to shorten training-to-convergence times, especially for recurrent monthly container builds.

Scenarios that are not typically used in real-world training, such as single GPU throughput are illustrated in the table below, and provided for reference as an indication of single chip throughput of the platform.

NVIDIA’s complete solution stack, from hardware to software, allows data scientists to deliver unprecedented acceleration at every scale. Visit NVIDIA GPU Cloud (NGC) to pull containers and quickly get up and running with deep learning.

Single GPU Training Performance of NVIDIA A100, V100 and T4

Benchmarks are reproducible by following links to NGC scripts

A100 Training Performance

Framework	Network	Throughput	GPU	Server	Container	Precision	Batch Size	Dataset	GPU Version
MXNet	ResNet-50 v1.5	2,751 images/sec	1x A100	-	-	Mixed	408	ImageNet2012	A100-SXM-80GB
PyTorch	Mask R-CNN	29 images/sec	1x A100	DGX A100	21.03-py3	TF32	8	COCO 2014	A100-SXM-80GB
	SSD v1.1	442 images/sec	1x A100	DGX A100	21.03-py3	Mixed	128	COCO 2017	A100-SXM-80GB
	Tacotron2	39,288 total output mels/sec	1x A100	DGX A100	21.03-py3	TF32	128	LJSpeech 1.1	A100-SXM-80GB
	WaveGlow	206,623 output samples/sec	1x A100	DGX A100	21.03-py3	Mixed	10	LJSpeech 1.1	A100-SXM-80GB
	Jasper	83 sequences/sec	1x A100	DGX A100	20.12-py3	Mixed	64	LibriSpeech	A100-SXM4-40GB
	Transformer	82,618 words/sec	1x A100	DGX A100	20.06-py3	Mixed	10240	wmt14-en-de	A100 SXM4-40GB
	FastPitch	169,838 frames/sec	1x A100	DGX A100	21.03-py3	Mixed	128	LJSpeech 1.1	A100-SXM-80GB
	GNMT V2	148,754 total tokens/sec	1x A100	DGX A100	20.12-py3	Mixed	128	wmt16-en-de	A100-SXM-80GB
	NCF	36,090,563 samples/sec	1x A100	DGX A100	21.03-py3	Mixed	1048576	MovieLens 20M	A100-SXM-80GB
	BERT-LARGE	123 sequences/sec	1x A100	DGX A100	21.03-py3	Mixed	32	SQuaD v1.1	A100-SXM-80GB
	Transformer-XL Large	27,364 total tokens/sec	1x A100	DGX A100	20.12-py3	Mixed	16	WikiText-103	A100-SXM-80GB
	Transformer-XL Base	80,692 total tokens/sec	1x A100	DGX A100	20.12-py3	Mixed	128	WikiText-103	A100-SXM-80GB
Tensorflow	ResNet-50 v1.5	2,678 images/sec	1x A100	DGX A100	21.03-py3	Mixed	256	ImageNet2012	A100-SXM-80GB
	ResNext101	1,325 images/sec	1x A100	DGX A100	21.03-py3	Mixed	256	Imagenet2012	A100-SXM-80GB
	SE-ResNext101	1,132 images/sec	1x A100	DGX A100	21.03-py3	Mixed	256	Imagenet2012	A100-SXM-80GB
	U-Net Industrial	319 images/sec	1x A100	DGX A100	20.11-py3	Mixed	16	DAGM2007	A100-SXM4-40GB
	U-Net Medical	146 images/sec	1x A100	DGX A100	21.03-py3	Mixed	8	EM segmentation challenge	A100-SXM-80GB
	VAE-CF	395,263 users processed/sec	1x A100	DGX A100	21.03-py3	Mixed	24576	MovieLens 20M	A100-SXM-80GB
	Wide and Deep	333,751 samples/sec	1x A100	DGX A100	21.03-py3	TF32	131072	Kaggle Outbrain Click Prediction	A100-SXM-80GB
	BERT-LARGE	117 sequences/sec	1x A100	DGX A100	21.03-py3	Mixed	24	SQuaD v1.1	A100-SXM-80GB
	Electra Fine Tuning	343 sequences/sec	1x A100	DGX A100	21.03-py3	Mixed	32	SQuaD v1.1	A100 SXM 80GB
	EfficientNet-B4	332 images/sec	1x A100	DGX A100	-	Mixed	160	ImageNet2012	A100-SXM-80GB
	NCF	40,477,425 samples/sec	1x A100	DGX A100	20.11-py3	Mixed	1048576	MovieLens 20M	A100-SXM4-40GB

FastPitch throughput metric frames/sec refers to mel-scale spectrogram frames/sec | Server with a hyphen indicates a pre-production server
BERT-Large = BERT-Large Fine Tuning (Squadv1.1) with Sequence Length of 384
EfficientNet-B4: Basic Augmentation | cuDNN Version = 8.0.5.32 | NCCL Version = 2.7.8 | Installation Source = NGC

A30 Training Performance

Framework	Network	Throughput	GPU	Server	Container	Precision	Batch Size	Dataset	GPU Version
MXNet	ResNet-50 v1.5	1,334 images/sec	1x A30	GIGABYTE G482-Z52-SW-QZ-001	21.03-py3	Mixed	64	ImageNet2012	A30
PyTorch	Mask R-CNN	16 images/sec	1x A30	GIGABYTE G482-Z52-SW-QZ-001	21.03-py3	Mixed	8	COCO 2014	A30
	SSD v1.1	224 images/sec	1x A30	GIGABYTE G482-Z52-SW-QZ-001	21.03-py3	Mixed	64	COCO 2017	A30
	Tacotron2	18,980 total output mels/sec	1x A30	GIGABYTE G482-Z52-SW-QZ-001	21.03-py3	Mixed	104	LJSpeech 1.1	A30
	WaveGlow	119,513 output samples/sec	1x A30	GIGABYTE G482-Z52-SW-QZ-001	21.03-py3	Mixed	10	LJSpeech 1.1	A30
	Transformer	25,314 words/sec	1x A30	GIGABYTE G482-Z52-SW-QZ-001	21.03-py3	Mixed	2560	wmt14-en-de	A30
	FastPitch	95,100 frames/sec	1x A30	GIGABYTE G482-Z52-SW-QZ-001	21.03-py3	Mixed	64	LJSpeech 1.1	A30
	NCF	19,126,726 samples/sec	1x A30	GIGABYTE G482-Z52-SW-QZ-001	21.03-py3	Mixed	1048576	MovieLens 20M	A30
	ResNeXt101	561 images/sec	1x A30	GIGABYTE G482-Z52-SW-QZ-001	21.03-py3	Mixed	128	Imagenet2012	A30
Tensorflow	ResNet-50 v1.5	1,341 images/sec	1x A30	GIGABYTE G482-Z52-SW-QZ-001	21.03-py3	Mixed	256	ImageNet2012	A30
	ResNext101	597 images/sec	1x A30	GIGABYTE G482-Z52-SW-QZ-001	21.03-py3	Mixed	128	Imagenet2012	A30
	SE-ResNext101	488 images/sec	1x A30	GIGABYTE G482-Z52-SW-QZ-001	21.03-py3	Mixed	96	Imagenet2012	A30
	U-Net Industrial	102 images/sec	1x A30	GIGABYTE G482-Z52-SW-QZ-001	21.03-py3	Mixed	16	DAGM2007	A30
	U-Net Medical	70 images/sec	1x A30	GIGABYTE G482-Z52-SW-QZ-001	21.03-py3	Mixed	8	EM segmentation challenge	A30
	VAE-CF	201,658 users processed/sec	1x A30	GIGABYTE G482-Z52-SW-QZ-001	21.03-py3	Mixed	24576	MovieLens 20M	A30
	Wide and Deep	229,387 samples/sec	1x A30	GIGABYTE G482-Z52-SW-QZ-001	21.03-py3	TF32	131072	Kaggle Outbrain Click Prediction	A30
	Electra Fine Tuning	152 sequences/sec	1x A30	GIGABYTE G482-Z52-SW-QZ-001	21.03-py3	Mixed	16	SQuaD v1.1	A30
	NCF	13,574,785 samples/sec	1x A30	GIGABYTE G482-Z52-SW-QZ-001	21.03-py3	Mixed	1048576	MovieLens 20M	A30
	Transformer XL Base	15,976 total tokens/sec	1x A30	GIGABYTE G482-Z52-SW-QZ-001	21.03-py3	Mixed	16	WikiText-103	A30

FastPitch throughput metric frames/sec refers to mel-scale spectrogram frames/sec | Server with a hyphen indicates a pre-production server

A10 Training Performance

Framework	Network	Throughput	GPU	Server	Container	Precision	Batch Size	Dataset	GPU Version
MXNet	ResNet-50 v1.5	1,075 images/sec	1x A10	GIGABYTE G482-Z52-00	21.03-py3	Mixed	192	ImageNet2012	A10
PyTorch	Mask R-CNN	13 images/sec	1x A10	GIGABYTE G482-Z52-00	21.03-py3	Mixed	8	COCO 2014	A10
	SSD v1.1	174 images/sec	1x A10	GIGABYTE G482-Z52-00	21.03-py3	Mixed	64	COCO 2017	A10
	Tacotron2	19,520 total output mels/sec	1x A10	GIGABYTE G482-Z52-00	21.03-py3	Mixed	104	LJSpeech 1.1	A10
	WaveGlow	99,358 output samples/sec	1x A10	GIGABYTE G482-Z52-00	21.03-py3	Mixed	10	LJSpeech 1.1	A10
	Transformer	22,365 words/sec	1x A10	GIGABYTE G482-Z52-00	21.03-py3	Mixed	2560	wmt14-en-de	A10
	FastPitch	89,154 frames/sec	1x A10	GIGABYTE G482-Z52-00	21.03-py3	Mixed	64	LJSpeech 1.1	A10
	NCF	16,478,978 samples/sec	1x A10	GIGABYTE G482-Z52-00	21.03-py3	Mixed	1048576	MovieLens 20M	A10
Tensorflow	ResNet-50 v1.5	1,006 images/sec	1x A10	GIGABYTE G482-Z52-00	21.03-py3	Mixed	256	ImageNet2012	A10
	ResNext101	460 images/sec	1x A10	GIGABYTE G482-Z52-00	21.03-py3	Mixed	128	Imagenet2012	A10
	SE-ResNext101	342 images/sec	1x A10	GIGABYTE G482-Z52-00	21.03-py3	Mixed	96	Imagenet2012	A10
	U-Net Industrial	94 images/sec	1x A10	GIGABYTE G482-Z52-00	21.03-py3	Mixed	16	DAGM2007	A10
	U-Net Medical	50 images/sec	1x A10	GIGABYTE G482-Z52-00	21.03-py3	Mixed	8	EM segmentation challenge	A10
	VAE-CF	174,443 users processed/sec	1x A10	GIGABYTE G482-Z52-00	21.03-py3	Mixed	24576	MovieLens 20M	A10
	Wide and Deep	249,905 samples/sec	1x A10	Supermicro SYS-1029GQ-TRT	20.11-py3	Mixed	131072	Kaggle Outbrain Click Prediction	A10
	Electra Fine Tuning	133 sequences/sec	1x A10	GIGABYTE G482-Z52-00	21.03-py3	Mixed	16	SQuaD v1.1	A10
	NCF	11,962,242 samples/sec	1x A10	GIGABYTE G482-Z52-00	21.03-py3	Mixed	1048576	MovieLens 20M	A10
	Transformer XL Base	13,220 total tokens/sec	1x A10	GIGABYTE G482-Z52-00	21.03-py3	Mixed	16	WikiText-103	A10

FastPitch throughput metric frames/sec refers to mel-scale spectrogram frames/sec | Server with a hyphen indicates a pre-production server

V100 Training Performance

Framework	Network	Throughput	GPU	Server	Container	Precision	Batch Size	Dataset	GPU Version
MXNet	ResNet-50 v1.5	1,510 images/sec	1x V100	DGX-2	21.03-py3	Mixed	256	ImageNet2012	V100-SXM3-32GB
PyTorch	Mask R-CNN	18 images/sec	1x V100	DGX-2	21.03-py3	Mixed	8	COCO 2014	V100-SXM3-32GB
	ResNeXt101	575 images/sec	1x V100	DGX-2	21.03-py3	Mixed	128	Imagenet2012	V100-SXM3-32GB
	SSD v1.1	234 images/sec	1x V100	DGX-2	21.03-py3	Mixed	64	COCO 2017	V100-SXM3-32GB
	Tacotron2	22,545 total output mels/sec	1x V100	DGX-2	21.02-py3	Mixed	104	LJSpeech 1.1	V100-SXM3-32GB
	WaveGlow	134,178 output samples/sec	1x V100	DGX-2	21.03-py3	Mixed	10	LJSpeech 1.1	V100-SXM3-32GB
	Jasper	47 sequences/sec	1x V100	DGX-2	20.12-py3	Mixed	64	LibriSpeech	V100-SXM3-32GB
	Transformer	33,468 words/sec	1x V100	DGX-1	20.06-py3	Mixed	5120	wmt14-en-de	V100 SXM2-16GB
	FastPitch	110,370 frames/sec	1x V100	DGX-1	20.06-py3	Mixed	64	LJSpeech 1.1	V100 SXM2-16GB
	GNMT V2	83,200 total tokens/sec	1x V100	DGX-2	20.09-py3	Mixed	128	wmt16-en-de	V100-SXM3-32GB
	NCF	22,123,689 samples/sec	1x V100	DGX-2	21.03-py3	Mixed	1048576	MovieLens 20M	V100-SXM3-32GB
	BERT-LARGE	53 sequences/sec	1x V100	DGX-2	21.03-py3	Mixed	10	SQuaD v1.1	V100-SXM3-32GB
	Transformer-XL Base	42,292 total tokens/sec	1x V100	DGX-2	20.12-py3	Mixed	32	WikiText-103	V100-SXM3-32GB
	Transformer-XL Large	14,499 total tokens/sec	1x V100	DGX-2	20.12-py3	Mixed	8	WikiText-103	V100-SXM3-32GB
Tensorflow	ResNet-50 v1.5	1,396 images/sec	1x V100	DGX-2	21.03-py3	Mixed	256	ImageNet2012	V100-SXM3-32GB
	ResNext101	636 images/sec	1x V100	DGX-2	21.03-py3	Mixed	128	Imagenet2012	V100-SXM3-32GB
	SE-ResNext101	550 images/sec	1x V100	DGX-2	21.03-py3	Mixed	96	Imagenet2012	V100-SXM3-32GB
	U-Net Industrial	169 images/sec	1x V100	DGX-1	20.06-py3	Mixed	16	DAGM2007	V100 SXM2-16GB
	U-Net Medical	68 images/sec	1x V100	DGX-2	21.03-py3	Mixed	8	EM segmentation challenge	V100-SXM3-32GB
	VAE-CF	223,120 users processed/sec	1x V100	DGX-2	21.03-py3	Mixed	24576	MovieLens 20M	V100-SXM3-32GB
	Wide and Deep	276,971 samples/sec	1x V100	DGX-2	21.03-py3	Mixed	131072	Kaggle Outbrain Click Prediction	V100-SXM3-32GB
	BERT-LARGE	48 sequences/sec	1x V100	DGX-2	21.03-py3	Mixed	10	SQuaD v1.1	V100-SXM3-32GB
	Electra Fine-Tuning	194 sequences/sec	1x V100	DGX-2	20.12-py3	Mixed	32	SQuaD v1.1	V100-SXM3-32GB
	SSD	233 images/sec	1x V100	DGX-2	21.03-py3	Mixed	32	COCO 2017	V100-SXM3-32GB
	Mask R-CNN	15 samples/sec	1x V100	DGX-2	21.03-py3	Mixed	4	COCO 2014	V100-SXM3-32GB

FastPitch throughput metric frames/sec refers to mel-scale spectrogram frames/sec
BERT-Large = BERT-Large Fine Tuning (Squadv1.1) with Sequence Length of 384

T4 Training Performance

Framework	Network	Throughput	GPU	Server	Container	Precision	Batch Size	Dataset	GPU Version
MXNet	ResNet-50 v1.5	514 images/sec	1x T4	Supermicro SYS-1029GQ-TRT	21.03-py3	Mixed	64	ImageNet2012	NVIDIA T4
PyTorch	ResNeXt101	208 images/sec	1x T4	Supermicro SYS-1029GQ-TRT	21.03-py3	Mixed	128	Imagenet2012	NVIDIA T4
	Tacotron2	16,844 total output mels/sec	1x T4	Supermicro SYS-1029GQ-TRT	21.03-py3	Mixed	104	LJSpeech 1.1	NVIDIA T4
	WaveGlow	57,673 output samples/sec	1x T4	Supermicro SYS-1029GQ-TRT	21.03-py3	Mixed	10	LJSpeech 1.1	NVIDIA T4
	Transformer	10,602 words/sec	1x T4	Supermicro SYS-1029GQ-TRT	21.03-py3	Mixed	2560	wmt14-en-de	NVIDIA T4
	FastPitch	38,985 frames/sec	1x T4	Supermicro SYS-1029GQ-TRT	21.03-py3	Mixed	64	LJSpeech 1.1	NVIDIA T4
	GNMT V2	31,592 total tokens/sec	1x T4	Supermicro SYS-4029GP-TRT	20.12-py3	Mixed	128	wmt16-en-de	NVIDIA T4
	NCF	8,075,883 samples/sec	1x T4	Supermicro SYS-1029GQ-TRT	21.03-py3	Mixed	1048576	MovieLens 20M	NVIDIA T4
	BERT-LARGE	18 sequences/sec	1x T4	Supermicro SYS-1029GQ-TRT	21.03-py3	Mixed	10	SQuaD v1.1	NVIDIA T4
	Transformer-XL Base	17,062 total tokens/sec	1x T4	Supermicro SYS-4029GP-TRT	20.12-py3	Mixed	32	WikiText-103	NVIDIA T4
	Mask R-CNN	7 images/sec	1x T4	Supermicro SYS-1029GQ-TRT	21.03-py3	Mixed	4	COCO 2014	NVIDIA T4
	Jasper	14 sequences/sec	1x T4	Supermicro SYS-4029GP-TRT	20.12-py3	Mixed	32	LibriSpeech	NVIDIA T4
	SE-ResNeXt101	162 images/sec	1x T4	Supermicro SYS-1029GQ-TRT	21.03-py3	Mixed	128	Imagenet2012	NVIDIA T4
	Transformer-XL Large	5,064 total tokens/sec	1x T4	Supermicro SYS-4029GP-TRT	20.12-py3	Mixed	4	WikiText-103	NVIDIA T4
Tensorflow	U-Net Industrial	42 images/sec	1x T4	Supermicro SYS-1029GQ-TRT	21.03-py3	Mixed	16	DAGM2007	NVIDIA T4
	U-Net Medical	22 images/sec	1x T4	Supermicro SYS-1029GQ-TRT	21.03-py3	Mixed	8	EM segmentation challenge	NVIDIA T4
	VAE-CF	82,118 users processed/sec	1x T4	Supermicro SYS-1029GQ-TRT	21.03-py3	Mixed	24576	MovieLens 20M	NVIDIA T4
	SSD	98 images/sec	1x T4	Supermicro SYS-1029GQ-TRT	21.03-py3	Mixed	32	COCO 2017	NVIDIA T4
	Mask R-CNN	8 samples/sec	1x T4	Supermicro SYS-1029GQ-TRT	21.03-py3	Mixed	4	COCO 2014	NVIDIA T4
	Wide and Deep	203,867 samples/sec	1x T4	Supermicro SYS-1029GQ-TRT	21.03-py3	Mixed	131072	Kaggle Outbrain Click Prediction	NVIDIA T4
	SE-ResNext101	172 images/sec	1x T4	Supermicro SYS-1029GQ-TRT	21.03-py3	Mixed	96	Imagenet2012	NVIDIA T4
	Electra Fine-Tuning	62 sequences/sec	1x T4	Supermicro SYS-1029GQ-TRT	21.03-py3	Mixed	16	SQuaD v1.1	NVIDIA T4

FastPitch throughput metric frames/sec refers to mel-scale spectrogram frames/sec
BERT-Large = BERT-Large Fine Tuning (Squadv1.1) with Sequence Length of 384

Real-world AI inferencing demands high throughput and low latencies with maximum efficiency across use cases. An industry leading solution enables customers to quickly deploy AI models into real-world production with the highest performance from data centers to the edge.

NVIDIA landed top performance spots on all MLPerf™ Inference 1.0 tests, the AI-industry’s leading benchmark. NVIDIA TensorRT™ running on NVIDIA Tensor Core GPUs enable the most efficient deep learning inference performance across multiple application areas and models. This versatility provides wide latitude to data scientists to create the optimal low-latency solution. Visit NVIDIA® GPU Cloud (NGC) to download any of these containers and immediately race into production. The inference whitepaper provides an overview of inference platforms.

Measuring inference performance involves balancing a lot of variables. PLASTER is an acronym that describes the key elements for measuring deep learning performance. Each letter identifies a factor (Programmability, Latency, Accuracy, Size of Model, Throughput, Energy Efficiency, Rate of Learning) that must be considered to arrive at the right set of tradeoffs and to produce a successful deep learning implementation. Refer to the PLASTER whitepaper for more details.

MLPerf Inference v1.0 Performance Benchmarks

Offline Scenario - Closed Division

Network	Throughput	Target Accuracy	GPU	Server	Dataset	GPU Version
ResNet-50 v1.5	105,677 samples/sec	76.46% Top1	8x A10	Supermicro 4029GP-TRT-OTO-28	ImageNet	A10
	141,518 samples/sec	76.46% Top1	8x A30	Gigabyte G482-Z54	ImageNet	A30
	5,108 samples/sec	76.46% Top1	1x1g.10gb A100	DGX A100	ImageNet	A100 SXM-80GB
	38,010 samples/sec	76.46% Top1	1x A100	DGX A100	ImageNet	A100 SXM-80GB
	304,876 samples/sec	76.46% Top1	8x A100	DGX A100	ImageNet	A100 SXM-80GB
	248,179 samples/sec	76.46% Top1	8x A100	Gigabyte G482-Z54	ImageNet	A100-PCIe-40GB
	132,926 samples/sec	76.46% Top1	4x A100	DGX-Station-A100	ImageNet	A100 SXM-80GB
SSD ResNet-34	2,496 samples/sec	0.2 mAP	8x A10	Supermicro 4029GP-TRT-OTO-28	COCO	A10
	3,756 samples/sec	0.2 mAP	8x A30	Gigabyte G482-Z54	COCO	A30
	134 samples/sec	0.2 mAP	1x1g.10gb A100	DGX A100	COCO	A100 SXM-80GB
	989 samples/sec	0.2 mAP	1x A100	DGX A100	COCO	A100 SXM-80GB
	7,879 samples/sec	0.2 mAP	8x A100	DGX A100	COCO	A100 SXM-80GB
	6,586 samples/sec	0.2 mAP	8x A100	Gigabyte G482-Z54	COCO	A100-PCIe-40GB
	3,370 samples/sec	0.2 mAP	4x A100	DGX-Station-A100	COCO	A100 SXM-80GB
3D-UNet	172 samples/sec	0.853 DICE mean	8x A10	Supermicro 4029GP-TRT-OTO-28	BraTS 2019	A10
	237 samples/sec	0.853 DICE mean	8x A30	Gigabyte G482-Z54	BraTS 2019	A30
	7 samples/sec	0.853 DICE mean	1x1g.10gb A100	DGX A100	BraTS 2019	A100 SXM-80GB
	61 samples/sec	0.853 DICE mean	1x A100	DGX A100	BraTS 2019	A100 SXM-80GB
	480 samples/sec	0.853 DICE mean	8x A100	DGX A100	BraTS 2019	A100 SXM-80GB
	412 samples/sec	0.853 DICE mean	8x A100	Gigabyte G482-Z54	BraTS 2019	A100-PCIe-40GB
	214 samples/sec	0.853 DICE mean	4x A100	DGX-Station-A100	BraTS 2019	A100 SXM-80GB
RNN-T	36,116 samples/sec	7.45% WER	8x A10	Supermicro 4029GP-TRT-OTO-28	LibriSpeech	A10
	51,690 samples/sec	7.45% WER	8x A30	Gigabyte G482-Z54	LibriSpeech	A30
	1,553 samples/sec	7.45% WER	1x1g.10gb A100	DGX A100	LibriSpeech	A100 SXM-80GB
	14,008 samples/sec	7.45% WER	1x A100	DGX A100	LibriSpeech	A100 SXM-80GB
	105,677 samples/sec	7.45% WER	8x A100	DGX A100	LibriSpeech	A100 SXM-80GB
	90,853 samples/sec	7.45% WER	8x A100	Gigabyte G482-Z54	LibriSpeech	A100-PCIe-40GB
	48,886 samples/sec	7.45% WER	4x A100	DGX-Station-A100	LibriSpeech	A100 SXM-80GB
BERT	8,454 samples/sec	90.07% f1	8x A10	Supermicro 4029GP-TRT-OTO-28	SQuAD v1.1	A10
	13,260 samples/sec	90.07% f1	8x A30	Gigabyte G482-Z54	SQuAD v1.1	A30
	492 samples/sec	90.07% f1	1x1g.10gb A100	DGX A100	SQuAD v1.1	A100 SXM-80GB
	3,602 samples/sec	90.07% f1	1x A100	DGX A100	SQuAD v1.1	A100 SXM-80GB
	28,347 samples/sec	90.07% f1	8x A100	DGX A100	SQuAD v1.1	A100 SXM-80GB
	22,847 samples/sec	90.07% f1	8x A100	Gigabyte G482-Z54	SQuAD v1.1	A100-PCIe-40GB
	11,305 samples/sec	90.07% f1	4x A100	DGX-Station-A100	SQuAD v1.1	A100 SXM-80GB
DLRM	772,378 samples/sec	80.25% AUC	8x A10	Supermicro 4029GP-TRT-OTO-28	Criteo 1TB Click Logs	A10
	1,067,510 samples/sec	80.25% AUC	8x A30	Gigabyte G482-Z54	Criteo 1TB Click Logs	A30
	36,473 samples/sec	80.25% AUC	1x1g.10gb A100	DGX A100	Criteo 1TB Click Logs	A100 SXM-80GB
	311,826 samples/sec	80.25% AUC	1x A100	DGX A100	Criteo 1TB Click Logs	A100 SXM-80GB
	2,462,300 samples/sec	80.25% AUC	8x A100	DGX A100	Criteo 1TB Click Logs	A100 SXM-80GB
	1,057,550 samples/sec	80.25% AUC	4x A100	DGX-Station-A100	Criteo 1TB Click Logs	A100 SXM-80GB

Server Scenario - Closed Division

Network	Throughput	Target Accuracy	MLPerf Server Latency Constraints (ms)	GPU	Server	Dataset	GPU Version
ResNet-50 v1.5	87,984 queries/sec	76.46% Top1	15	8x A10	Supermicro 4029GP-TRT-OTO-28	ImageNet	A10
	115,987 queries/sec	76.46% Top1	15	8x A30	Gigabyte G482-Z54	ImageNet	A30
	3,602 queries/sec	76.46% Top1	15	1x1g.10gb A100	DGX A100	ImageNet	A100 SXM-80GB
	30,794 queries/sec	76.46% Top1	15	1x A100	DGX A100	ImageNet	A100 SXM-80GB
	259,994 queries/sec	76.46% Top1	15	8x A100	DGX A100	ImageNet	A100 SXM-80GB
	207,976 queries/sec	76.46% Top1	15	8x A100	Gigabyte G482-Z54	ImageNet	A100-PCIe-40GB
	106,988 queries/sec	76.46% Top1	15	4x A100	DGX-Station-A100	ImageNet	A100 SXM-80GB
SSD ResNet-34	2,000 queries/sec	0.2 mAP	100	8x A10	Supermicro 4029GP-TRT-OTO-28	COCO	A10
	3,575 queries/sec	0.2 mAP	100	8x A30	Gigabyte G482-Z54	COCO	A30
	100 queries/sec	0.2 mAP	100	1x1g.10gb A100	DGX A100	COCO	A100 SXM-80GB
	926 queries/sec	0.2 mAP	100	1x A100	DGX A100	COCO	A100 SXM-80GB
	7,654 queries/sec	0.2 mAP	100	8x A100	DGX A100	COCO	A100 SXM-80GB
	6,162 queries/sec	0.2 mAP	100	8x A100	Gigabyte G482-Z54	COCO	A100-PCIe-40GB
	3,081 queries/sec	0.2 mAP	100	4x A100	DGX-Station-A100	COCO	A100 SXM-80GB
RNN-T	22,597 queries/sec	7.45% WER	1,000	8x A10	Supermicro 4029GP-TRT-OTO-28	LibriSpeech	A10
	36,991 queries/sec	7.45% WER	1,000	8x A30	Gigabyte G482-Z54	LibriSpeech	A30
	1,303 queries/sec	7.45% WER	1,000	1x1g.10gb A100	DGX A100	LibriSpeech	A100 SXM-80GB
	12,751 queries/sec	7.45% WER	1,000	1x A100	DGX A100	LibriSpeech	A100 SXM-80GB
	103,986 queries/sec	7.45% WER	1,000	8x A100	DGX A100	LibriSpeech	A100 SXM-80GB
	85,985 queries/sec	7.45% WER	1,000	8x A100	Gigabyte G482-Z54	LibriSpeech	A100-PCIe-40GB
	43,389 queries/sec	7.45% WER	1,000	4x A100	DGX-Station-A100	LibriSpeech	A100 SXM-80GB
BERT	7,204 queries/sec	90.07% f1	130	8x A10	Supermicro 4029GP-TRT-OTO-28	SQuAD v1.1	A10
	11,500 queries/sec	90.07% f1	130	8x A30	Gigabyte G482-Z54	SQuAD v1.1	A30
	381 queries/sec	90.07% f1	130	1x1g.10gb A100	DGX A100	SQuAD v1.1	A100 SXM-80GB
	3,202 queries/sec	90.07% f1	130	1x A100	DGX A100	SQuAD v1.1	A100 SXM-80GB
	25,792 queries/sec	90.07% f1	130	8x A100	DGX A100	SQuAD v1.1	A100 SXM-80GB
	20,792 queries/sec	90.07% f1	130	8x A100	Gigabyte G482-Z54	SQuAD v1.1	A100-PCIe-40GB
	10,203 queries/sec	90.07% f1	130	4x A100	DGX-Station-A100	SQuAD v1.1	A100 SXM-80GB
DLRM	680,147 queries/sec	80.25% AUC	30	8x A10	Supermicro 4029GP-TRT-OTO-28	Criteo 1TB Click Logs	A10
	750,204 queries/sec	80.25% AUC	30	8x A30	Gigabyte G482-Z54	Criteo 1TB Click Logs	A30
	35,991 queries/sec	80.25% AUC	30	1x1g.10gb A100	DGX A100	Criteo 1TB Click Logs	A100 SXM-80GB
	286,002 queries/sec	80.25% AUC	30	1x A100	DGX A100	Criteo 1TB Click Logs	A100 SXM-80GB
	2,302,570 queries/sec	80.25% AUC	30	8x A100	DGX A100	Criteo 1TB Click Logs	A100 SXM-80GB
	942,395 queries/sec	80.25% AUC	30	4x A100	DGX-Station-A100	Criteo 1TB Click Logs	A100 SXM-80GB

Power Efficiency Offline Scenario - Closed Division

Network	Throughput	Throughput per Watt	GPU	Server	Dataset	GPU Version
ResNet-50 v1.5	213,599 samples/sec	97.31 samples/sec/watt	8x A100	Gigabyte G482-Z54	ImageNet	A100 PCIe-40GB
	270,706 samples/sec	78.27 samples/sec/watt	8x A100	DGX A100	ImageNet	A100 SXM-80GB
	124,529 samples/sec	98.14 samples/sec/watt	4x A100	DGX-Station-A100	ImageNet	A100 SXM-80GB
SSD ResNet-34	5,824 samples/sec	2.6 samples/sec/watt	8x A100	Gigabyte G482-Z54	COCO	A100 PCIe-40GB
	6,875 samples/sec	1.96 samples/sec/watt	8x A100	DGX A100	COCO	A100 SXM-80GB
	3,110 samples/sec	2.44 samples/sec/watt	4x A100	DGX-Station-A100	COCO	A100 SXM-80GB
3D-UNet	372 samples/sec	0.16 samples/sec/watt	8x A100	Gigabyte G482-Z54	BraTS 2019	A100 PCIe-40GB
	433 samples/sec	0.12 samples/sec/watt	8x A100	DGX A100	BraTS 2019	A100 SXM-80GB
	202 samples/sec	0.16 samples/sec/watt	4x A100	DGX-Station-A100	BraTS 2019	A100 SXM-80GB
RNN-T	82,540 samples/sec	36.23 samples/sec/watt	8x A100	Gigabyte G482-Z54	LibriSpeech	A100 PCIe-40GB
	93,803 samples/sec	26.39 samples/sec/watt	8x A100	DGX A100	LibriSpeech	A100 SXM-80GB
	47,255 samples/sec	36.16 samples/sec/watt	4x A100	DGX-Station-A100	LibriSpeech	A100 SXM-80GB
BERT	17,697 samples/sec	7.73 samples/sec/watt	8x A100	Gigabyte G482-Z54	SQuAD v1.1	A100 PCIe-40GB
	23,406 samples/sec	6.77 samples/sec/watt	8x A100	DGX A100	SQuAD v1.1	A100 SXM-80GB
	9,865 samples/sec	7.76 samples/sec/watt	4x A100	DGX-Station-A100	SQuAD v1.1	A100 SXM-80GB
DLRM	1,577,960 samples/sec	730.76 samples/sec/watt	8x A100	Gigabyte G482-Z54	Criteo 1TB Click Logs	A100 PCIe-40GB
	2,115,950 samples/sec	619.54 samples/sec/watt	8x A100	DGX A100	Criteo 1TB Click Logs	A100 SXM-80GB
	974,571 samples/sec	762.64 samples/sec/watt	4x A100	DGX-Station-A100	Criteo 1TB Click Logs	A100 SXM-80GB

Power Efficiency Server Scenario - Closed Division

Network	Throughput	Throughput per Watt	GPU	Server	Dataset	GPU Version
ResNet-50 v1.5	184,984 queries/sec	82.39 queries/sec/watt	8x A100	Gigabyte G482-Z54	ImageNet	A100 PCIe-40GB
	239,991 queries/sec	69.53 queries/sec/watt	8x A100	DGX A100	ImageNet	A100 SXM-80GB
	106,988 queries/sec	84.51 queries/sec/watt	4x A100	DGX-Station-A100	ImageNet	A100 SXM-80GB
SSD ResNet-34	5,702 queries/sec	2.52 queries/sec/watt	8x A100	Gigabyte G482-Z54	COCO	A100 PCIe-40GB
	6,301 queries/sec	1.82 queries/sec/watt	8x A100	DGX A100	COCO	A100 SXM-80GB
	3,081 queries/sec	2.43 queries/sec/watt	4x A100	DGX-Station-A100	COCO	A100 SXM-80GB
RNN-T	74,974 queries/sec	32.25 queries/sec/watt	8x A100	Gigabyte G482-Z54	LibriSpeech	A100 PCIe-40GB
	87,984 queries/sec	24.78 queries/sec/watt	8x A100	DGX A100	LibriSpeech	A100 SXM-80GB
	43,389 queries/sec	33.03 queries/sec/watt	4x A100	DGX-Station-A100	LibriSpeech	A100 SXM-80GB
BERT	17,499 queries/sec	7.58 queries/sec/watt	8x A100	Gigabyte G482-Z54	SQuAD v1.1	A100 PCIe-40GB
	21,492 queries/sec	6.03 queries/sec/watt	8x A100	DGX A100	SQuAD v1.1	A100 SXM-80GB
	10,203 queries/sec	7.84 queries/sec/watt	4x A100	DGX-Station-A100	SQuAD v1.1	A100 SXM-80GB
DLRM	2,001,940 queries/sec	575.72 queries/sec/watt	8x A100	DGX A100	Criteo 1TB Click Logs	A100 SXM-80GB
	890,334 queries/sec	663.62 queries/sec/watt	4x A100	DGX-Station-A100	Criteo 1TB Click Logs	A100 SXM-80GB

MLPerf™ v1.0 Inference Closed: ResNet-50 v1.5, SSD ResNet-34, RNN-T, BERT 99% of FP32 accuracy target, 3D U-Net, DLRM 99.9% of FP32 accuracy target: 1.0-25, 1.0-26, 1.0-29, 1.0-30, 1.0-32, 1.0-55, 1.0-57. MLPerf name and logo are trademarks. See https://mlcommons.org/ for more information.
BERT-Large sequence length = 384.
DLRM samples refers to 270 pairs/sample average
A10 and A30 results are preview
For MLPerf™ various scenario data, click here
For MLPerf™ latency constraints, click here

Inference Performance of NVIDIA A100, V100 and T4

Benchmarks are reproducible by following links to NGC scripts

Inference Natural Langugage Processing

BERT Inference Throughput

NVIDIA A100 BERT Inference Benchmarks

Network	Network Type	Batch Size	Throughput	Efficiency	Latency (ms)	GPU	Server	Container	Precision	Dataset	Framework	GPU Version
BERT-Large with Sparsity	Attention	94	6,188 sequences/sec	-	-	1x A100	DGX A100	-	INT8	SQuaD v1.1	-	A100 SXM4-40GB

A100 with 7 MIG instances of 1g.5gb | Sequence length=128 | Efficiency based on board power
Containers with a hyphen indicates a pre-release container

Inference Image Classification on CNNs with TensorRT

ResNet-50 v1.5 Throughput

ResNet-50 v1.5 Power Efficiency

A100 1/7 MIG Inference Performance

Network	Batch Size	1/7 MIG Throughput	Latency (ms)	GPU	Server	Container	Precision	Dataset	Framework	GPU Version
ResNet-50	1	1,411 images/sec	0.71	1x A100	DGX A100	20.12-py3	INT8	Synthetic	TensorRT 7.2	A100-SXM-80GB
	2	2,221 images/sec	0.9	1x A100	DGX A100	20.12-py3	INT8	Synthetic	TensorRT 7.2	A100-SXM-80GB
	8	3,418 images/sec	2.34	1x A100	DGX A100	20.12-py3	INT8	Synthetic	TensorRT 7.2	A100-SXM-80GB
	27	4,052 images/sec	6.66	1x A100	DGX A100	20.12-py3	INT8	Synthetic	TensorRT 7.2	A100-SXM-80GB
	128	4,480 images/sec	28.57	1x A100	DGX A100	20.12-py3	INT8	Synthetic	TensorRT 7.2	A100-SXM-80GB
ResNet-50v1.5	1	1,384 images/sec	0.72	1x A100	DGX A100	20.12-py3	INT8	Synthetic	TensorRT 7.2	A100-SXM-80GB
	2	2,150 images/sec	0.93	1x A100	DGX A100	20.12-py3	INT8	Synthetic	TensorRT 7.2	A100-SXM-80GB
	8	3,316 images/sec	2.41	1x A100	DGX A100	20.12-py3	INT8	Synthetic	TensorRT 7.2	A100-SXM-80GB
	27	3,915 images/sec	6.9	1x A100	DGX A100	20.12-py3	INT8	Synthetic	TensorRT 7.2	A100-SXM-80GB
	128	4,323 images/sec	29.61	1x A100	DGX A100	20.12-py3	INT8	Synthetic	TensorRT 7.2	A100-SXM-80GB
BERT-BASE	1	724 sequences/sec	1.38	1x A100	DGX A100	20.11-py3	INT8	Real (Q&A provided as text input)	TensorRT 7.2	A100-SXM-80GB
	2	1,095 sequences/sec	1.83	1x A100	DGX A100	20.11-py3	INT8	Real (Q&A provided as text input)	TensorRT 7.2	A100-SXM-80GB
	8	1,657 sequences/sec	4.83	1x A100	DGX A100	20.11-py3	INT8	Real (Q&A provided as text input)	TensorRT 7.2	A100-SXM-80GB
	19	1,917 sequences/sec	9.91	1x A100	DGX A100	20.11-py3	INT8	Real (Q&A provided as text input)	TensorRT 7.2	A100-SXM-80GB
	128	2,163 sequences/sec	59.17	1x A100	DGX A100	20.11-py3	INT8	Real (Q&A provided as text input)	TensorRT 7.2	A100-SXM-80GB
BERT-LARGE	1	265 sequences/sec	3.78	1x A100	DGX A100	20.11-py3	INT8	Real (Q&A provided as text input)	TensorRT 7.2	A100-SXM-80GB
	2	376 sequences/sec	5.31	1x A100	DGX A100	20.11-py3	INT8	Real (Q&A provided as text input)	TensorRT 7.2	A100-SXM-80GB
	5	517 sequences/sec	9.67	1x A100	DGX A100	20.11-py3	INT8	Real (Q&A provided as text input)	TensorRT 7.2	A100-SXM-80GB
	8	550 sequences/sec	14.56	1x A100	DGX A100	20.11-py3	INT8	Real (Q&A provided as text input)	TensorRT 7.2	A100-SXM-80GB
	128	674 sequences/sec	189.89	1x A100	DGX A100	20.11-py3	INT8	Real (Q&A provided as text input)	TensorRT 7.2	A100-SXM-80GB

Containers with a hyphen indicates a pre-release container | Servers with a hyphen indicates a pre-production server
BERT-Large: Sequence Length = 128

A100 7 MIG Inference Performance

Network	Batch Size	7 MIG Throughput	Latency (ms)	GPU	Server	Container	Precision	Dataset	Framework	GPU Version
ResNet-50	1	9,607 images/sec	0.71	1x A100	DGX A100	20.12-py3	INT8	Synthetic	TensorRT 7.2	A100-SXM-80GB
	2	14,947 images/sec	0.9	1x A100	DGX A100	20.12-py3	INT8	Synthetic	TensorRT 7.2	A100-SXM-80GB
	8	23,533 images/sec	2.34	1x A100	DGX A100	20.12-py3	INT8	Synthetic	TensorRT 7.2	A100-SXM-80GB
	27	28,042 images/sec	6.66	1x A100	DGX A100	20.12-py3	INT8	Synthetic	TensorRT 7.2	A100-SXM-80GB
	128	31,225 images/sec	28.57	1x A100	DGX A100	20.12-py3	INT8	Synthetic	TensorRT 7.2	A100-SXM-80GB
ResNet-50v1.5	1	9,350 images/sec	0.72	1x A100	DGX A100	20.12-py3	INT8	Synthetic	TensorRT 7.2	A100-SXM-80GB
	2	14,734 images/sec	0.93	1x A100	DGX A100	20.12-py3	INT8	Synthetic	TensorRT 7.2	A100-SXM-80GB
	8	22,822 images/sec	2.41	1x A100	DGX A100	20.12-py3	INT8	Synthetic	TensorRT 7.2	A100-SXM-80GB
	26	27,254 images/sec	6.9	1x A100	DGX A100	20.12-py3	INT8	Synthetic	TensorRT 7.2	A100-SXM-80GB
	128	30,120 images/sec	29.61	1x A100	DGX A100	20.12-py3	INT8	Synthetic	TensorRT 7.2	A100-SXM-80GB
BERT-BASE	1	5,144 sequences/sec	1.37	1x A100	DGX A100	20.11-py3	INT8	Real (Q&A provided as text input)	TensorRT 7.2	A100-SXM-80GB
	2	8,265 sequences/sec	1.71	1x A100	DGX A100	20.11-py3	INT8	Real (Q&A provided as text input)	TensorRT 7.2	A100-SXM-80GB
	8	11,665 sequences/sec	4.82	1x A100	DGX A100	20.11-py3	INT8	Real (Q&A provided as text input)	TensorRT 7.2	A100-SXM-80GB
	18	13,250 sequences/sec	9.53	1x A100	DGX A100	20.11-py3	INT8	Real (Q&A provided as text input)	TensorRT 7.2	A100-SXM-80GB
	128	14,420 sequences/sec	62.18	1x A100	DGX A100	20.11-py3	INT8	Real (Q&A provided as text input)	TensorRT 7.2	A100-SXM-80GB
BERT-LARGE	1	1,787 sequences/sec	3.93	1x A100	DGX A100	20.11-py3	INT8	Real (Q&A provided as text input)	TensorRT 7.2	A100-SXM-80GB
	2	2,632 sequences/sec	5.35	1x A100	DGX A100	20.11-py3	INT8	Real (Q&A provided as text input)	TensorRT 7.2	A100-SXM-80GB
	5	3,575 sequences/sec	9.84	1x A100	DGX A100	20.11-py3	INT8	Real (Q&A provided as text input)	TensorRT 7.2	A100-SXM-80GB
	8	3,810 sequences/sec	14.78	1x A100	DGX A100	20.11-py3	INT8	Real (Q&A provided as text input)	TensorRT 7.2	A100-SXM-80GB
	128	4,465 sequences/sec	200.87	1x A100	DGX A100	20.11-py3	INT8	Real (Q&A provided as text input)	TensorRT 7.2	A100-SXM4-40GB

Containers with a hyphen indicates a pre-release container | Servers with a hyphen indicates a pre-production server
BERT-Large: Sequence Length = 128

A100 Full Chip Inference Performance

Network	Batch Size	Full Chip Throughput	Latency (ms)	GPU	Server	Container	Precision	Dataset	Framework	GPU Version
ResNet-50	2	4,049 images/sec	0.49	1x A100	DGX A100	21.03-py3	INT8	Synthetic	TensorRT 7.2	A100-SXM-80GB
	8	11,304 images/sec	0.71	1x A100	DGX A100	21.03-py3	INT8	Synthetic	TensorRT 7.2	A100-SXM-80GB
	128	29,128 images/sec	4.39	1x A100	DGX A100	21.03-py3	INT8	Synthetic	TensorRT 7.2	A100-SXM-80GB
	211	30,899 images/sec	6.8	1x A100	DGX A100	20.12-py3	INT8	Synthetic	TensorRT 7.2	A100-SXM-80GB
ResNet-50v1.5	2	3,898 images/sec	0.51	1x A100	DGX A100	21.03-py3	INT8	Synthetic	TensorRT 7.2	A100-SXM-80GB
	8	10,985 images/sec	0.73	1x A100	DGX A100	21.03-py3	INT8	Synthetic	TensorRT 7.2	A100-SXM-80GB
	128	28,247 images/sec	4.53	1x A100	DGX A100	21.03-py3	INT8	Synthetic	TensorRT 7.2	A100-SXM-80GB
	208	29,896 images/sec	7	1x A100	DGX A100	20.12-py3	INT8	Synthetic	TensorRT 7.2	A100-SXM-80GB
ResNext101	32	7,674 samples/sec	4.17	1x A100	-	-	INT8	Synthetic	TensorRT 7.2	A100-SXM4-40GB
EfficientNet-B0	128	22,346 images/sec	5.73	1x A100	-	-	INT8	Synthetic	TensorRT 7.2	A100-SXM4-40GB
BERT-BASE	2	2,425 sequences/sec	0.82	1x A100	DGX A100	20.11-py3	INT8	Real (Q&A provided as text input)	TensorRT 7.2	A100-SXM4-40GB
	8	6,837 sequences/sec	1.2	1x A100	DGX A100	20.11-py3	INT8	Real (Q&A provided as text input)	TensorRT 7.2	A100-SXM4-40GB
	128	13,700 sequences/sec	9.3	1x A100	DGX A100	20.11-py3	INT8	Real (Q&A provided as text input)	TensorRT 7.2	A100-SXM4-40GB
	256	14,490 sequences/sec	17.67	1x A100	DGX A100	-	INT8	Real (Q&A provided as text input)	TensorRT 7.2	A100-SXM4-40GB
BERT-LARGE	2	1,087 sequences/sec	1.8	1x A100	DGX A100	20.11-py3	INT8	Real (Q&A provided as text input)	TensorRT 7.2	A100-SXM4-40GB
	8	2,232 sequences/sec	3.6	1x A100	DGX A100	20.11-py3	INT8	Real (Q&A provided as text input)	TensorRT 7.2	A100-SXM4-40GB
	128	4,509 sequences/sec	28	1x A100	DGX A100	20.11-py3	INT8	Real (Q&A provided as text input)	TensorRT 7.2	A100-SXM4-40GB
	256	4,679 sequences/sec	54.71	1x A100	-	-	INT8	Real (Q&A provided as text input)	TensorRT 7.2	A100-SXM-80GB

Containers with a hyphen indicates a pre-release container | Servers with a hyphen indicates a pre-production server
BERT-Large: Sequence Length = 128
For BS=1 inference refer to the Triton Inference Server tab

A30 Inference Performance

Network	Batch Size	Throughput	Efficiency	Latency (ms)	GPU	Server	Container	Precision	Dataset	Framework	GPU Version
ResNet-50	1	2,049 images/sec	24 images/sec/watt	0.49	1x A30	GIGABYTE G482-Z52-SW-QZ-001	21.03-py3	INT8	Synthetic	TensorRT 7.2	A30
	2	3,487 images/sec	38 images/sec/watt	0.57	1x A30	GIGABYTE G482-Z52-SW-QZ-001	21.03-py3	INT8	Synthetic	TensorRT 7.2	A30
	8	8,497 images/sec	70 images/sec/watt	0.94	1x A30	GIGABYTE G482-Z52-SW-QZ-001	21.03-py3	INT8	Synthetic	TensorRT 7.2	A30
	128	15,299 images/sec	93 images/sec/watt	8.37	1x A30	GIGABYTE G482-Z52-SW-QZ-001	21.03-py3	INT8	Synthetic	TensorRT 7.2	A30
ResNet-50v1.5	1	2,049 images/sec	24 images/sec/watt	0.49	1x A30	GIGABYTE G482-Z52-SW-QZ-001	21.03-py3	INT8	Synthetic	TensorRT 7.2	A30
	2	3,498 images/sec	37 images/sec/watt	0.57	1x A30	GIGABYTE G482-Z52-SW-QZ-001	21.03-py3	INT8	Synthetic	TensorRT 7.2	A30
	8	8,330 images/sec	68 images/sec/watt	0.96	1x A30	GIGABYTE G482-Z52-SW-QZ-001	21.03-py3	INT8	Synthetic	TensorRT 7.2	A30
	128	14,778 images/sec	90 images/sec/watt	8.66	1x A30	GIGABYTE G482-Z52-SW-QZ-001	21.03-py3	INT8	Synthetic	TensorRT 7.2	A30

NGC: TensorRT Container
Sequence length=128 for BERT-BASE and BERT-LARGE | Efficiency based on board power
Containers with a hyphen indicates a pre-release container

A10 Inference Performance

Network	Batch Size	Throughput	Efficiency	Latency (ms)	GPU	Server	Container	Precision	Dataset	Framework	GPU Version
ResNet-50	1	2,510 images/sec	20 images/sec/watt	0.4	1x A10	GIGABYTE G482-Z52-00	21.03-py3	INT8	Synthetic	TensorRT 7.2	A10
	2	4,362 images/sec	30 images/sec/watt	0.46	1x A10	GIGABYTE G482-Z52-00	21.03-py3	INT8	Synthetic	TensorRT 7.2	A10
	8	7,803 images/sec	52 images/sec/watt	1.03	1x A10	GIGABYTE G482-Z52-00	21.03-py3	INT8	Synthetic	TensorRT 7.2	A10
	128	11,779 images/sec	79 images/sec/watt	10.87	1x A10	GIGABYTE G482-Z52-00	21.03-py3	INT8	Synthetic	TensorRT 7.2	A10
ResNet-50v1.5	1	2,509 images/sec	20 images/sec/watt	0.4	1x A10	GIGABYTE G482-Z52-00	21.03-py3	INT8	Synthetic	TensorRT 7.2	A10
	2	4,256 images/sec	29 images/sec/watt	0.47	1x A10	GIGABYTE G482-Z52-00	21.03-py3	INT8	Synthetic	TensorRT 7.2	A10
	8	7,517 images/sec	50 images/sec/watt	1.06	1x A10	GIGABYTE G482-Z52-00	21.03-py3	INT8	Synthetic	TensorRT 7.2	A10
	128	11,136 images/sec	74 images/sec/watt	11.49	1x A10	GIGABYTE G482-Z52-00	21.03-py3	INT8	Synthetic	TensorRT 7.2	A10
BERT-BASE	1	1,415 sequences/sec	-	0.71	1x A10	Supermicro SYS-1029GQ-TRT	20.11-py3	INT8	Sample Text	TensorRT 7.2	A10
	2	2,109 sequences/sec	15 sequences/sec/watt	0.95	1x A10	Supermicro SYS-1029GQ-TRT	20.11-py3	INT8	Sample Text	TensorRT 7.2	A10
	8	3,634 sequences/sec	27 sequences/sec/watt	2.73	1x A10	Supermicro SYS-1029GQ-TRT	20.11-py3	INT8	Sample Text	TensorRT 7.2	A10
	128	4,496 sequences/sec	34 sequences/sec/watt	28.47	1x A10	Supermicro SYS-1029GQ-TRT	20.11-py3	INT8	Sample Text	TensorRT 7.2	A10
BERT-LARGE	1	576 sequences/sec	5 sequences/sec/watt	1.74	1x A10	Supermicro SYS-1029GQ-TRT	20.11-py3	INT8	Sample Text	TensorRT 7.2	A10
	2	788 sequences/sec	6 sequences/sec/watt	2.54	1x A10	Supermicro SYS-1029GQ-TRT	20.11-py3	INT8	Sample Text	TensorRT 7.2	A10
	8	1,281 sequences/sec	10 sequences/sec/watt	6.25	1x A10	Supermicro SYS-1029GQ-TRT	20.11-py3	INT8	Sample Text	TensorRT 7.2	A10
	128	1,368 sequences/sec	10 sequences/sec/watt	93.57	1x A10	Supermicro SYS-1029GQ-TRT	20.11-py3	INT8	Sample Text	TensorRT 7.2	A10

NGC: TensorRT Container
Sequence length=128 for BERT-BASE and BERT-LARGE | Efficiency based on board power
Containers with a hyphen indicates a pre-release container

V100 Inference Performance

Network	Batch Size	Throughput	Efficiency	Latency (ms)	GPU	Server	Container	Precision	Dataset	Framework	GPU Version
ResNet-50	2	1,976 images/sec	11 images/sec/watt	1	1x V100	DGX-2	21.03-py3	Mixed	Synthetic	TensorRT 7.2	V100-SXM3-32GB
	8	4,304 images/sec	16 images/sec/watt	1.86	1x V100	DGX-2	21.03-py3	Mixed	Synthetic	TensorRT 7.2	V100-SXM3-32GB
	128	8,168 images/sec	24 images/sec/watt	15.67	1x V100	DGX-2	21.03-py3	Mixed	Synthetic	TensorRT 7.2	V100-SXM3-32GB
ResNet-50v1.5	2	1,979 images/sec	11 images/sec/watt	1.01	1x V100	DGX-2	21.03-py3	Mixed	Synthetic	TensorRT 7.2	V100-SXM3-32GB
	8	4,219 images/sec	16 images/sec/watt	1.9	1x V100	DGX-2	21.03-py3	Mixed	Synthetic	TensorRT 7.2	V100-SXM3-32GB
	128	7,815 images/sec	23 images/sec/watt	16.38	1x V100	DGX-2	21.03-py3	Mixed	Synthetic	TensorRT 7.2	V100-SXM3-32GB
BERT-BASE	8	2,315 sequences/sec	8 sequences/sec/watt	3.46	1x V100	DGX-2	20.11-py3	Mixed	Real (Q&A provided as text input)	TensorRT 7.2	V100-SXM3-32GB
	128	3,194 sequences/sec	11 sequences/sec/watt	40.08	1x V100	DGX-2	20.11-py3	INT8	Real (Q&A provided as text input)	TensorRT 7.2	V100-SXM3-32GB
BERT-LARGE	2	523 sequences/sec	2 sequences/sec/watt	3.82	1x V100	DGX-2	20.11-py3	Mixed	Real (Q&A provided as text input)	TensorRT 7.2	V100-SXM3-32GB
	8	792 sequences/sec	3 sequences/sec/watt	10.1	1x V100	DGX-2	20.11-py3	INT8	Real (Q&A provided as text input)	TensorRT 7.2	V100-SXM3-32GB
	128	978 sequences/sec	3 sequences/sec/watt	130.92	1x V100	DGX-2	20.11-py3	INT8	Real (Q&A provided as text input)	TensorRT 7.2	V100-SXM3-32GB

NGC: TensorRT Container
Sequence length=128 for BERT-BASE and BERT-LARGE | Efficiency based on board power
Containers with a hyphen indicates a pre-release container
For BS=1 inference refer to the Triton Inference Server tab

T4 Inference Performance

Network	Batch Size	Throughput	Efficiency	Latency (ms)	GPU	Server	Container	Precision	Dataset	Framework	GPU Version
ResNet-50	2	2,105 images/sec	30 images/sec/watt	0.95	1x T4	Supermicro SYS-1029GQ-TRT	21.03-py3	INT8	Synthetic	TensorRT 7.2	NVIDIA T4
	8	3,916 images/sec	56 images/sec/watt	2.04	1x T4	Supermicro SYS-1029GQ-TRT	21.03-py3	INT8	Synthetic	TensorRT 7.2	NVIDIA T4
	128	5,073 images/sec	75 images/sec/watt	25.23	1x T4	Supermicro SYS-1029GQ-TRT	21.03-py3	INT8	Synthetic	TensorRT 7.2	NVIDIA T4
ResNet-50v1.5	2	2,092 images/sec	30 images/sec/watt	0.96	1x T4	Supermicro SYS-1029GQ-TRT	21.03-py3	INT8	Synthetic	TensorRT 7.2	NVIDIA T4
	8	3,715 images/sec	53 images/sec/watt	2.15	1x T4	Supermicro SYS-1029GQ-TRT	21.03-py3	INT8	Synthetic	TensorRT 7.2	NVIDIA T4
	128	4,824 images/sec	69 images/sec/watt	26.53	1x T4	Supermicro SYS-1029GQ-TRT	21.03-py3	INT8	Synthetic	TensorRT 7.2	NVIDIA T4
BERT-BASE	2	1,079 sequences/sec	17 sequences/sec/watt	1.85	1x T4	Supermicro SYS-1029GQ-TRT	20.11-py3	INT8	Real (Q&A provided as text input)	TensorRT 7.2	NVIDIA T4
	8	1,720 sequences/sec	28 sequences/sec/watt	4.65	1x T4	Supermicro SYS-1029GQ-TRT	20.11-py3	INT8	Real (Q&A provided as text input)	TensorRT 7.2	NVIDIA T4
	128	1,818 sequences/sec	28 sequences/sec/watt	70.4	1x T4	Supermicro SYS-1029GQ-TRT	20.11-py3	INT8	Real (Q&A provided as text input)	TensorRT 7.2	NVIDIA T4
BERT-LARGE	2	390 sequences/sec	6 sequences/sec/watt	5.12	1x T4	Supermicro SYS-1029GQ-TRT	20.11-py3	INT8	Real (Q&A provided as text input)	TensorRT 7.2	NVIDIA T4
	8	555 sequences/sec	9 sequences/sec/watt	14.41	1x T4	Supermicro SYS-1029GQ-TRT	20.11-py3	INT8	Real (Q&A provided as text input)	TensorRT 7.2	NVIDIA T4
	128	561 sequences/sec	8 sequences/sec/watt	227.98	1x T4	Supermicro SYS-1029GQ-TRT	20.11-py3	INT8	Real (Q&A provided as text input)	TensorRT 7.2	NVIDIA T4

The Triton Inference Server is an open source inference serving software which maximizes performance and simplifies the deployment of AI models at scale in production. Triton lets teams deploy trained AI models from multiple model frameworks (TensorFlow, TensorRT, PyTorch, ONNX Runtime, OpenVino, or custom backends). They can deploy from local storage, Google Cloud Platform, or Amazon S3 on any GPU or CPU based infrastructure (in cloud, data center, or embedded devices). Triton is open source on GitHub and available as a docker container on NGC.

NVIDIA landed top performance spots on all MLPerf™ Inference 1.0 tests, the AI-industry’s leading benchmark competition. For inference submissions, we have typically used a custom A100 inference serving harness. This custom harness has been designed and optimized specifically for providing the highest possible inference performance for MLPerf™ workloads, which require running inference on bare metal.

NVIDIA TRITON Inference Delivered Performance vs. MLPerf v1.0

MLPerf™ v1.0 A100 Inference Closed: ResNet-50 v1.5, SSD ResNet-34, RNN-T, BERT 99% of FP32 accuracy target, 3D U-Net, DLRM 99% of FP32 accuracy target: 1.0-30, 1.0-31. MLPerf name and logo are trademarks. See www.mlcommons.org for more information.

Starting in the previous MLPerf™ round (v0.7), Triton Inference Server was used to submit GPU inference results. The chart above compares the performance of Triton to the custom MLPerf™ serving harness across five different TensorRT networks on bare metal. The results show that Triton is highly efficient and delivers nearly equal or identical performance to the highly optimized MLPerf™ harness.

To deliver such performance, the team brought many optimizations to Triton, such as new lightweight data structures for low latency communication with applications, support for variable sequence length inputs to avoid padding, and CUDA graphs for the TensorRT backend for higher inference performance. These enhancements are available in every Triton release starting from 20.09. In the latest MLPerf™ v1.0 submission, we used Triton for both GPU and CPU inference submissions. We added a new OpenVino backend in Triton for high performance inference on CPU.

NVIDIA Client BS=1 Performance with Triton Inference Server

Network	Accelerator	Training Framework	Framework Backend	Precision	Model Instances on Triton	Client Batch Size	Dynamic Batch Size (Triton)	Number of Concurrent Client Requests	Latency	Throughput	Sequence/Input Length	Triton Container Version
ResNet-50 V1.5 Inference	A100-PCIE-40GB	PyTorch	TensorRT	FP16	2	1	64	256	61.02	4,197 inf/sec	-	20.07-py3
ResNet-50 V1.5 Inference	A100-SXM4-40GB	PyTorch	TensorRT	TF32	2	1	64	256	48.35	5,294 inf/sec	-	21.03-py3
ResNet-50 V1.5 Inference	NVIDIA T4	PyTorch	TensorRT	FP16	1	1	64	256	257.91	992 inf/sec	-	20.07-py3
ResNet-50 V1.5 Inference	V100 SXM2-32GB	PyTorch	TensorRT	FP32	4	1	64	384	215.79	1,781 inf/sec	-	21.03-py3
BERT Large Inference	A100-PCIE-40GB	TensorFlow	TensorRT	FP16	1	1	8	16	17.48	915 inf/sec	384	20.09-py3
BERT Large Inference	A100-SXM4-40GB	TensorFlow	TensorRT	FP16	1	1	8	16	16.1	994 inf/sec	384	20.09-py3
BERT Large Inference	NVIDIA T4	TensorFlow	TensorRT	FP16	1	1	8	16	81.14	197 inf/sec	384	20.09-py3
DLRM Inference	A100-PCIE-40GB	PyTorch	Torchscript	FP16	1	1	65,536	9	7.51	1,197 inf/sec	-	20.08-py3
DLRM Inference	A100-SXM4-40GB	PyTorch	Torchscript	FP16	1	1	65,536	20	16.07	1,245 inf/sec	-	20.08-py3
DLRM Inference	V100-SXM2-32GB	PyTorch	Torchscript	FP16	1	1	65,536	22	25.04	879 inf/sec	-	20.08-py3

NVIDIA Jarvis is an application framework for multimodal conversational AI services that delivers real-time performance on GPUs. Jarvis 1.0 Beta includes fully optimized pipelines for Automatic Speech Recognition (ASR), Natural Language Processing (NLP), Text to Speech (TTS) that can be used for deploying real-time conversational AI apps such as transcription, virtual assistants and chatbots. Please visit Jarvis – Getting Started to download and get started with Jarvis.

Jarvis Benchmarks

Automatic Speech Recognition

A100 Best Streaming Throughput Mode (800 ms chunk)

Acoustic model	# of streams	Avg Latency (ms)	Throughput (RTFX)	GPU Version
Quartznet	1	14.4	1	A100 SXM4-40GB
Quartznet	256	254.3	64	A100 SXM4-40GB
Quartznet	512	351.2	506	A100 SXM4-40GB
Quartznet	1024	630.8	1005	A100 SXM4-40GB
Jasper	1	17.6	1	A100 SXM4-40GB
Jasper	256	244.9	254	A100 SXM4-40GB
Jasper	512	381	507	A100 SXM4-40GB
Jasper	1024	749.3	1,004	A100 SXM4-40GB

A100 Best Streaming Latency Mode (100 ms chunk)

Acoustic model	# of streams	Avg Latency (ms)	Throughput (RTFX)	GPU Version
Quartznet	1	9.6	1	A100 SXM4-40GB
Quartznet	16	25.9	16	A100 SXM4-40GB
Quartznet	128	132.4	128	A100 SXM4-40GB
Jasper	1	13.4	1	A100 SXM4-40GB
Jasper	16	26.3	16	A100 SXM4-40GB
Jasper	128	258.9	128	A100 SXM4-40GB

A100 Offline Mode (3200 ms chunk)

Acoustic model	# of streams	Avg Latency (ms)	Throughput (RTFX)	GPU Version
Quartznet	1	28.1	1	A100 SXM4-40GB
Quartznet	512	566.5	505	A100 SXM4-40GB
Quartznet	1,024	899.3	1,000	A100 SXM4-40GB
Quartznet	1,512	1,303.8	1,460	A100 SXM4-40GB
Jasper	1	31	1	A100 SXM4-40GB
Jasper	512	667.5	504	A100 SXM4-40GB
Jasper	1,024	1,089	997	A100 SXM4-40GB
Jasper	1,512	1,753.8	1,449	A100 SXM4-40GB

V100 Best Streaming Throughput Mode (800 ms chunk)

Acoustic model	# of streams	Avg Latency (ms)	Throughput (RTFX)	GPU Version
Quartznet	1	14.4	1	V100 SXM2-16GB
Quartznet	256	222.2	254	V100 SXM2-16GB
Quartznet	512	385.2	505	V100 SXM2-16GB
Quartznet	768	574.5	752	V100 SXM2-16GB
Jasper	1	26.8	1	V100 SXM2-16GB
Jasper	128	239.4	127	V100 SXM2-16GB
Jasper	256	416	253	V100 SXM2-16GB
Jasper	512	969.7	500	V100 SXM2-16GB

V100 Best Streaming Latency Mode (100 ms chunk)

Acoustic model	# of streams	Avg Latency (ms)	Throughput (RTFX)	GPU Version
Quartznet	1	8.8	1	V100 SXM2-16GB
Quartznet	16	22.4	16	V100 SXM2-16GB
Quartznet	128	114.7	127	V100 SXM2-16GB
Jasper	1	21.5	1	V100 SXM2-16GB
Jasper	16	36.9	16	V100 SXM2-16GB
Jasper	64	406.4	64	V100 SXM2-16GB
Jasper	512	969.7	500	V100 SXM2-16GB

V100 Offline Mode (3200 ms chunk)

Acoustic model	# of streams	Avg Latency (ms)	Throughput (RTFX)	GPU Version
Quartznet	1	32.933	1	V100 SXM2-16GB
Quartznet	256	461.44	253	V100 SXM2-16GB
Quartznet	512	784.73	502	V100 SXM2-16GB
Quartznet	768	1,121.6	747	V100 SXM2-16GB
Quartznet	1,024	1,551.5	986	V100 SXM2-16GB
Jasper	1	48.351	1	V100 SXM2-16GB
Jasper	256	734.99	252	V100 SXM2-16GB
Jasper	512	1,423.3	498	V100 SXM2-16GB
Jasper	768	2,190.2	730	V100 SXM2-16GB

T4 Best Streaming Throughput Mode (800 ms chunk)

Acoustic model	# of streams	Avg Latency (ms)	Throughput (RTFX)	GPU Version
Quartznet	1	33.183	1	NVIDIA T4
Quartznet	64	162.63	64	NVIDIA T4
Quartznet	128	263.6	127	NVIDIA T4
Quartznet	256	449.28	253	NVIDIA T4
Quartznet	384	732.75	376	NVIDIA T4
Jasper	1	72.377	1	NVIDIA T4
Jasper	64	259.64	64	NVIDIA T4
Jasper	128	450.81	127	NVIDIA T4
Jasper	256	1,200.8	249	NVIDIA T4

T4 Best Streaming Latency Mode (100 ms chunk)

Acoustic model	# of streams	Avg Latency (ms)	Throughput (RTFX)	GPU Version
Quartznet	1	19.2	1	NVIDIA T4
Quartznet	16	56.4	16	NVIDIA T4
Quartznet	64	242.4	64	NVIDIA T4
Jasper	1	46.9	1	NVIDIA T4
Jasper	8	51.1	8	NVIDIA T4
Jasper	16	84.4	16	NVIDIA T4

T4 Offline Mode (3200 ms chunk)

Acoustic model	# of streams	Latency (ms) (avg)	Throughput (RTFX)	GPU Version
Quartznet	1	157.62	1	NVIDIA T4
Quartznet	256	906.17	251	NVIDIA T4
Quartznet	512	1,515.2	495	NVIDIA T4
Jasper	1	96.201	1	NVIDIA T4
Jasper	256	1,758.4	247	NVIDIA T4

ASR Throughput (RTFX) - Number of seconds of audio processed per second | Audio Chunk Size – Server side configuration indicating the amount of new data to be considered by the acoustic model | ASR Dataset: Librispeech | The latency numbers were measured using the streaming recognition mode, with the BERT-Base punctuation model enabled, a 4-gram language model, a decoder beam width of 128 and timestamps enabled. The client and the server were using audio chunks of the same duration (100ms, 800ms, 3200ms depending on the server configuration). The Jarvis streaming client jarvis_streaming_asr_client, provided in the Jarvis client image was used with the --simulate_realtime flag to simulate transcription from a microphone, where each stream was doing 5 iterations over a sample audio file from the Librispeech dataset (1272-135031-0000.wav) | Jarvis version: v1.0.0-b1 | Hardware: NVIDIA DGX A100 (1x A100 SXM4-40GB), NVIDIA DGX-1 (1x V100-SXM2-16GB), NVIDIA T4 with 2x Intel(R) Xeon(R) Gold 6240 CPU @ 2.60GHz

Natural Language Processing

A100 Benchmarks

Task	# of streams	Avg Latency (ms)	Throughput (seq/sec)	GPU Version
NER	1	3.19	311	A100 SXM4-40GB
NER	256	95.5	2549	A100 SXM4-40GB
Q&A	1	4.95	201	A100 SXM4-40GB
Q&A	128	279	453	A100 SXM4-40GB

V100 Benchmarks

Task	# of streams	Avg Latency (ms)	Throughput (seq/sec)	GPU Version
NER	1	4.87	204	V100 SXM2-16GB
NER	256	135	1,797	V100 SXM2-16GB
Q&A	1	7.47	134	V100 SXM2-16GB
Q&A	128	521	244	V100 SXM2-16GB

T4 Benchmarks

Task	# of streams	Avg Latency (ms)	Throughput (seq/sec)	GPU Version
NER	1	9.31	107	NVIDIA T4
NER	256	255	960	NVIDIA T4
Q&A	1	11.5	87	NVIDIA T4
Q&A	128	571	223	NVIDIA T4

Named Entity Recogniton (NER): 128 seq len, BERT-base | Question Answering (QA): 384 seq len, BERT-large | NLP Throughput (seq/s) - Number of sequences processed per second | Performance of the Jarvis named entity recognition (NER) service (using a BERT-base model, sequence length of 128) and the Jarvis question answering (QA) service (using a BERT-large model, sequence length of 384) was measured in Jarvis. Batch size 1 latency and maximum throughput were measured. Jarvis version: v1.0.0-b1 | Hardware: NVIDIA DGX A100 (1x A100 SXM4-40GB), NVIDIA DGX-1 (1x V100-SXM2-16GB), NVIDIA T4 with 2x Intel(R) Xeon(R) Gold 6240 CPU @ 2.60GHz

Text to Speech

A100 Benchmarks

# of streams	Avg Latency to first audio (sec)	Avg Latency between audio chunks (sec)	Throughput (RTFX)	GPU Version
1	0.06	0.04	20	A100 SXM4-40GB
4	0.48	0.03	37	A100 SXM4-40GB
6	0.69	0.03	42	A100 SXM4-40GB
8	0.88	0.03	46	A100 SXM4-40GB
10	1.06	0.03	49	A100 SXM4-40GB

V100 Benchmarks

# of streams	Avg Latency to first audio (sec)	Avg Latency between audio chunks (sec)	Throughput (RTFX)	GPU Version
1	0.08	0.05	14	V100 SXM2-16GB
4	0.77	0.05	23	V100 SXM2-16GB
6	1.11	0.05	26	V100 SXM2-16GB
8	1.4	0.06	28	V100 SXM2-16GB
10	1.74	0.07	28	V100 SXM2-16GB

T4 Benchmarks

# of streams	Avg Latency to first audio (sec)	Avg Latency between audio chunks (sec)	Throughput (RTFX)	GPU Version
1	0.12	0.07	11	NVIDIA T4
4	1.02	0.07	17	NVIDIA T4
6	1.59	0.07	18	NVIDIA T4
8	2.13	0.08	19	NVIDIA T4
10	2.55	0.1	18	NVIDIA T4

TTS Throughput (RTFX) - Number of seconds of audio generated per second | Dataset: LJSpeech | Performance of the Jarvis text-to-speech (TTS) service was measured for different number of parallel streams. Each parallel stream performed 10 iterations over 10 input strings from the LJSpeech dataset. Latency to first audio chunk and latency between successive audio chunks and throughput were measured. Jarvis version: v1.0.0-b1 | Hardware: NVIDIA DGX A100 (1x A100 SXM4-40GB), NVIDIA DGX-1 (1x V100-SXM2-16GB), NVIDIA T4 with 2x Intel(R) Xeon(R) Gold 6240 CPU @ 2.60GHz

Last updated: April 21th, 2021

NVIDIA Data Center Deep Learning Product Performance

Training - Convergence

Training - Single GPU

Inference

Triton Inference Server

AI Pipeline

NVIDIA Performance on MLPerf 0.7 AI Benchmarks

MLPerf Training Performance

NVIDIA A100 Performance on MLPerf 0.7 AI Benchmarks - Closed Division

Converged Training Performance of NVIDIA A100, V100 and T4

A100 Training Performance

A30 Training Performance

A10 Training Performance

V100 Training Performance

T4 Training Performance

Single GPU Training Performance of NVIDIA A100, V100 and T4

A100 Training Performance

A30 Training Performance

A10 Training Performance

V100 Training Performance

T4 Training Performance

MLPerf Inference v1.0 Performance Benchmarks

Offline Scenario - Closed Division

Server Scenario - Closed Division

Power Efficiency Offline Scenario - Closed Division

Power Efficiency Server Scenario - Closed Division

Inference Performance of NVIDIA A100, V100 and T4

Inference Natural Langugage Processing

NVIDIA A100 BERT Inference Benchmarks

Inference Image Classification on CNNs with TensorRT

A100 1/7 MIG Inference Performance

A100 7 MIG Inference Performance

A100 Full Chip Inference Performance

A30 Inference Performance

A10 Inference Performance

V100 Inference Performance

T4 Inference Performance

NVIDIA TRITON Inference Delivered Performance vs. MLPerf v1.0

NVIDIA Client BS=1 Performance with Triton Inference Server

Jarvis Benchmarks

Automatic Speech Recognition

A100 Best Streaming Throughput Mode (800 ms chunk)

A100 Best Streaming Latency Mode (100 ms chunk)

A100 Offline Mode (3200 ms chunk)

V100 Best Streaming Throughput Mode (800 ms chunk)

V100 Best Streaming Latency Mode (100 ms chunk)

V100 Offline Mode (3200 ms chunk)

T4 Best Streaming Throughput Mode (800 ms chunk)

T4 Best Streaming Latency Mode (100 ms chunk)

T4 Offline Mode (3200 ms chunk)

Natural Language Processing

A100 Benchmarks

V100 Benchmarks

T4 Benchmarks

Text to Speech

A100 Benchmarks

V100 Benchmarks

T4 Benchmarks