NVIDIA Data Center Deep Learning Product Performance

Review the latest GPU acceleration factors of popular HPC applications.

Please refer to Measuring Training and Inferencing Performance on NVIDIA AI Platforms Reviewer’s Guide for instructions on how to reproduce these performance claims.

Deploying AI in real world applications, requires training the networks to convergence at a specified accuracy. This is the best methodology to test AI systems- where they are ready to be deployed in the field, as the networks can then deliver meaningful results (for example, correctly performing image recognition on video streams). Read our blog on convergence for more details. Training that does not converge is a measurement of hardware’s throughput capabilities on the specified AI network, but is not representative of real world applications.

NVIDIA’s complete solution stack, from GPUs to libraries, and containers on NVIDIA GPU Cloud (NGC), allows data scientists to quickly get up and running with deep learning. NVIDIA A100 Tensor Core GPUs provides unprecedented acceleration at every scale, setting records in MLPerf™, the AI industry’s leading benchmark and a testament to our accelerated platform approach.

NVIDIA Performance on MLPerf 1.0 AI Benchmarks

BERT Time to Train on A100

PyTorch | Precision: Mixed | Dataset: Wikipedia 2020/01/01 | Convergence criteria - refer to MLPerf requirements

MLPerf Training Performance

NVIDIA A100 Performance on MLPerf 1.0 AI Benchmarks - Closed Division

Framework	Network	Time to Train (mins)	MLPerf Quality Target	GPU	Server	MLPerf-ID	Precision	Dataset	GPU Version
MXNet	ResNet-50 v1.5	28.77	75.90% classification	8x A100	DGX A100	1.0-1059	Mixed	ImageNet2012	A100-SXM4-80GB
		4.91	75.90% classification	64x A100	DGX A100	1.0-1064	Mixed	ImageNet2012	A100-SXM4-80GB
		0.58	75.90% classification	1024x A100	DGX A100	1.0-1072	Mixed	ImageNet2012	A100-SXM4-80GB
		0.4	75.90% classification	2480x A100	DGX A100	1.0-1076	Mixed	ImageNet2012	A100-SXM4-80GB
	SSD	8.52	23.0% mAP	8x A100	DGX A100	1.0-1059	Mixed	COCO2017	A100-SXM4-80GB
		1.9	23.0% mAP	64x A100	DGX A100	1.0-1064	Mixed	COCO2017	A100-SXM4-80GB
		0.48	23.0% mAP	1024x A100	DGX A100	1.0-1072	Mixed	COCO2017	A100-SXM4-80GB
	UNet-3D	29.16	0.908 Mean DICE score	8x A100	DGX A100	1.0-1059	Mixed	KiTS19	A100-SXM4-80GB
		4.68	0.908 Mean DICE score	104x A100	DGX A100	1.0-1066	Mixed	KiTS19	A100-SXM4-80GB
		3	0.908 Mean DICE score	800x A100	DGX A100	1.0-1071	Mixed	KiTS19	A100-SXM4-80GB
PyTorch	BERT	21.69	0.712 Mask-LM accuracy	8x A100	DGX A100	1.0-1060	Mixed	Wikipedia 2020/01/01	A100-SXM4-80GB
		3.37	0.712 Mask-LM accuracy	64x A100	DGX A100	1.0-1065	Mixed	Wikipedia 2020/01/01	A100-SXM4-80GB
		0.73	0.712 Mask-LM accuracy	1024x A100	DGX A100	1.0-1073	Mixed	Wikipedia 2020/01/01	A100-SXM4-80GB
		0.32	0.712 Mask-LM accuracy	4096x A100	DGX A100	1.0-1077	Mixed	Wikipedia 2020/01/01	A100-SXM4-80GB
	Mask R-CNN	50.39	0.377 Box min AP and 0.339 Mask min AP	8x A100	DGX A100	1.0-1060	Mixed	COCO2017	A100-SXM4-80GB
		15.75	0.377 Box min AP and 0.339 Mask min AP	32x A100	DGX A100	1.0-1062	Mixed	COCO2017	A100-SXM4-80GB
		3.95	0.377 Box min AP and 0.339 Mask min AP	272x A100	DGX A100	1.0-1070	Mixed	COCO2017	A100-SXM4-80GB
	RNN-T	38.7	0.058 Word Error Rate	8x A100	DGX A100	1.0-1060	Mixed	LibriSpeech	A100-SXM4-80GB
		4.41	0.058 Word Error Rate	128x A100	DGX A100	1.0-1068	Mixed	LibriSpeech	A100-SXM4-80GB
		2.75	0.058 Word Error Rate	1536x A100	DGX A100	1.0-1074	Mixed	LibriSpeech	A100-SXM4-80GB
TensorFlow	MiniGo	269.54	50% win rate vs. checkpoint	8x A100	DGX A100	1.0-1061	Mixed	Go	A100-SXM4-80GB
		29.32	50% win rate vs. checkpoint	256x A100	DGX A100	1.0-1069	Mixed	Go	A100-SXM4-80GB
		15.53	50% win rate vs. checkpoint	1792x A100	DGX A100	1.0-1075	Mixed	Go	A100-SXM4-80GB
NVIDIA Merlin HugeCTR	DLRM	1.96	0.8025 AUC	8x A100	DGX A100	1.0-1058	Mixed	Criteo AI Lab’s Terabyte Click-Through-Rate (CTR)	A100-SXM4-80GB
		1.05	0.8025 AUC	64x A100	DGX A100	1.0-1063	Mixed	Criteo AI Lab’s Terabyte Click-Through-Rate (CTR)	A100-SXM4-80GB
		0.99	0.8025 AUC	112x A100	DGX A100	1.0-1067	Mixed	Criteo AI Lab’s Terabyte Click-Through-Rate (CTR)	A100-SXM4-80GB

Converged Training Performance of NVIDIA A100, A40, A30, A10, V100 and T4

Benchmarks are reproducible by following links to NGC scripts

A100 Training Performance

Framework	Framework Version	Network	Time to Train (mins)	Accuracy	Throughput	GPU	Server	Container	Precision	Batch Size	Dataset	GPU Version
MXNet	1.6.0	ResNet-50 v1.5	40	75.9 Top 1 Accuracy	22,008 images/sec	8x A100	DGX A100	20.06-py3	Mixed	408	ImageNet2012	A100-SXM4-40GB
PyTorch	1.8.0a0	Mask R-CNN	176	.34 AP Segm	167 images/sec	8x A100	DGX A100	21.12-py3	TF32	8	COCO 2014	A100-SXM-80GB
	1.6.0a0	SSD v1.1	43	.25 mAP	3,092 images/sec	8x A100	DGX A100	20.06-py3	Mixed	128	COCO 2017	A100-SXM-80GB
	1.9.0a0	Tacotron2	100	.56 Training Loss	308,404 total output mels/sec	8x A100	DGX A100	21.05-py3	TF32	128	LJSpeech 1.1	A100-SXM-80GB
	1.9.0a0	WaveGlow	295	-5.86 Training Loss	1,453,539 output samples/sec	8x A100	DGX A100	21.05-py3	Mixed	10	LJSpeech 1.1	A100-SXM-80GB
	1.6.0a0	Jasper	3,600	3.53 dev-clean WER	603 sequences/sec	8x A100	DGX A100	20.06-py3	Mixed	64	LibriSpeech	A100 SXM4-40GB
	1.6.0a0	Transformer	167	27.76 BLEU Score	582,721 words/sec	8x A100	DGX A100	20.06-py3	Mixed	10240	wmt14-en-de	A100 SXM4-40GB
	1.6.0a0	FastPitch	216	.18 Training Loss	1,040,206 frames/sec	8x A100	DGX A100	20.06-py3	Mixed	32	LJSpeech 1.1	A100 SXM4-40GB
	1.9.0a0	GNMT V2	20	24.51 BLEU Score	913,199 total tokens/sec	8x A100	DGX A100	21.05-py3	Mixed	128	wmt16-en-de	A100-SXM-80GB
	1.9.0a0	NCF	0.37	.96 Hit Rate at 10	154,525,495 samples/sec	8x A100	DGX A100	21.05-py3	Mixed	131072	MovieLens 20M	A100-SXM-80GB
	1.9.0a0	BERT-LARGE	3	91.31 F1	926 sequences/sec	8x A100	DGX A100	21.05-py3	Mixed	32	SQuaD v1.1	A100-SXM-80GB
	1.9.0a0	Transformer-XL Large	405	14.03 Perplexity	203,314 total tokens/sec	8x A100	DGX A100	21.05-py3	Mixed	16	WikiText-103	A100-SXM-80GB
	1.9.0a0	Transformer-XL Base	206	16.94 Perplexity	637,779 total tokens/sec	8x A100	DGX A100	21.05-py3	Mixed	128	WikiText-103	A100-SXM-80GB
	1.6.0a0	BERT-Large Pre-Training P1	2,379	-	3,231 sequences/sec	8x A100	DGX A100	20.06-py3	Mixed	-	Wikipedia+BookCorpus	A100-SXM4-40GB
	1.6.0a0	BERT-Large Pre-Training P2	1,377	1.34 Final Loss	630 sequences/sec	8x A100	DGX A100	20.06-py3	Mixed	-	Wikipedia+BookCorpus	A100-SXM4-40GB
	1.6.0a0	BERT-Large Pre-Training E2E	3,756	1.34 Final Loss	-	8x A100	DGX A100	20.06-py3	Mixed	-	Wikipedia+BookCorpus	A100-SXM4-40GB
Tensorflow	1.15.5	ResNet-50 v1.5	95	77.01 Top1	20,478 images/sec	8x A100	DGX A100	21.05-py3	Mixed	256	ImageNet2012	A100-SXM-80GB
	1.15.5	ResNext101	192	79.18 Top1	10,129 images/sec	8x A100	DGX A100	21.04-py4	Mixed	256	Imagenet2012	A100-SXM-80GB
	1.15.5	SE-ResNext101	222	79.48 Top1	8,743 images/sec	8x A100	DGX A100	21.05-py3	Mixed	256	Imagenet2012	A100-SXM-80GB
	1.15.5	U-Net Industrial	1	.99 IoU Threshold	1,027 images/sec	8x A100	DGX A100	21.03-py3	Mixed	2	DAGM2007	A100-SXM-80GB
	1.15.5	U-Net Medical	6	.9 Dice Score	952 images/sec	8x A100	DGX A100	21.03-py3	Mixed	8	EM segmentation challenge	A100-SXM-80GB
	1.15.5	VAE-CF	1	.43 NDCG@100	1,555,512 users processed/sec	8x A100	DGX A100	21.05-py3	TF32	3072	MovieLens 20M	A100-SXM-80GB
	1.15.4	Wide and Deep	107	.68 MAP at 12	1,111,976 samples/sec	8x A100	DGX A100	20.10-py3	Mixed	16384	Kaggle Outbrain Click Prediction	A100 SXM4-40GB
	1.15.5	BERT-LARGE	11	91.18 F1	841 sequences/sec	8x A100	DGX A100	21.05-py3	Mixed	24	SQuaD v1.1	A100-SXM-80GB
	2.4.0	Electra Fine Tuning	3	92.71 F1	2,457 sequences/sec	8x A100	DGX A100	21.05-py3	Mixed	32	SQuaD v1.1	A100-SXM-80GB
	2.2.0	EfficientNet-B4	4,231	82.81 Top 1	2,535 images/sec	8x A100	DGX A100	20.08-py3	Mixed	160	ImageNet2012	A100-SXM-80GB

FastPitch throughput metric frames/sec refers to mel-scale spectrogram frames/sec | Server with a hyphen is a pre-production server
BERT-Large = BERT-Large Fine Tuning (Squadv1.1) with Sequence Length of 384
BERT-Large Pre-Training Sequence Length for Phase 1 = 128 and Phase 2 = 512 | Batch Size for Phase 1 = 65,536 and Phase 2 = 32,768
EfficientNet-B4: Mixup = 0.2 | Auto-Augmentation | cuDNN Version = 8.0.5.39 | NCCL Version = 2.7.8

A40 Training Performance

Framework	Framework Version	Network	Time to Train (mins)	Accuracy	Throughput	GPU	Server	Container	Precision	Batch Size	Dataset	GPU Version
PyTorch	1.9.0a0	NCF	1	.96 Hit Rate at 10	59,667,265 samples/sec	8x A40	GIGABYTE G482-Z52-00	21.05-py3	Mixed	131072	MovieLens 20M	A40
	1.9.0a0	BERT-LARGE	7	91.18 F1	429 sequences/sec	8x A40	GIGABYTE G482-Z52-00	21.05-py3	Mixed	32	SQuaD v1.1	A40

Server with a hyphen is a pre-production server
BERT-Large = BERT-Large Fine Tuning (Squadv1.1) with Sequence Length of 384

A30 Training Performance

Framework	Framework Version	Network	Time to Train (mins)	Accuracy	Throughput	GPU	Server	Container	Precision	Batch Size	Dataset	GPU Version
MXNet	1.8.0	ResNet-50 v1.5	182	77.34 Top1	10,739 images/sec	8x A30	GIGABYTE G482-Z52-SW-QZ-001	21.05-py3	Mixed	192	ImageNet2012	A30
PyTorch	1.9.0a0	Tacotron2	215	.54 Training Loss	144,326 total output mels/sec	8x A30	GIGABYTE G482-Z52-SW-QZ-001	21.05-py3	Mixed	104	LJSpeech 1.1	A30
	1.9.0a0	WaveGlow	533	-5.82 Training Loss	794,511 output samples/sec	8x A30	GIGABYTE G482-Z52-SW-QZ-001	21.05-py3	Mixed	10	LJSpeech 1.1	A30
	1.9.0a0	Transformer	1,108	27.58 BLEU Score	87,584 words/sec	8x A30	GIGABYTE G482-Z52-SW-QZ-001	21.05-py3	Mixed	2560	wmt14-en-de	A30
	1.9.0a0	GNMT V2	81	24.65 BLEU Score	219,582 total tokens/sec	8x A30	GIGABYTE G482-Z52-SW-QZ-001	21.05-py3	TF32	128	wmt16-en-de	A30
	1.9.0a0	NCF	1	.96 Hit Rate at 10	56,399,904 samples/sec	8x A30	GIGABYTE G482-Z52-SW-QZ-001	21.05-py3	Mixed	131072	MovieLens 20M	A30
	1.9.0a0	BERT-LARGE	10	91.2 F1	294 sequences/sec	8x A30	GIGABYTE G482-Z52-SW-QZ-001	21.05-py3	Mixed	10	SQuaD v1.1	A30
	1.9.0a0	Transformer-XL Base	151	22.16 Perplexity	219,994 total tokens/sec	8x A30	GIGABYTE G482-Z52-SW-QZ-001	21.05-py3	Mixed	32	WikiText-103	A30
Tensorflow	1.15.5	ResNet-50 v1.5	198	76.78 Top1	9,798 images/sec	8x A30	GIGABYTE G482-Z52-SW-QZ-001	21.05-py3	Mixed	256	ImageNet2012	A30
	1.15.5	U-Net Industrial	1	.99 IoU Threshold 0.95	580 images/sec	8x A30	GIGABYTE G482-Z52-SW-QZ-001	21.05-py3	Mixed	2	DAGM2007	A30
	1.15.5	U-Net Medical	9	.9 DICE Score	461 images/sec	8x A30	GIGABYTE G482-Z52-SW-QZ-001	21.05-py3	Mixed	8	EM segmentation challenge	A30
	1.15.5	VAE-CF	1	.43 NDCG@100	861,797 users processed/sec	8x A30	GIGABYTE G482-Z52-SW-QZ-001	21.05-py3	TF32	3072	MovieLens 20M	A30
	1.15.5	SE-ResNext101	573	79.83 Top1	3,399 images/sec	8x A30	GIGABYTE G482-Z52-SW-QZ-001	21.05-py3	Mixed	96	Imagenet2012	A30
	2.4.0	Electra Fine-Tuning	6	92.65 F1	904 sequences/sec	8x A30	GIGABYTE G482-Z52-SW-QZ-001	21.05-py3	Mixed	16	SQuaD v1.1	A30

Server with a hyphen is a pre-production server
BERT-Large = BERT-Large Fine Tuning (Squadv1.1) with Sequence Length of 384

A10 Training Performance

Framework	Framework Version	Network	Time to Train (mins)	Accuracy	Throughput	GPU	Server	Container	Precision	Batch Size	Dataset	GPU Version
MXNet	1.8.0	ResNet-50 v1.5	242	77.25 Top1	8,117 images/sec	8x A10	GIGABYTE G482-Z52-00	21.05-py3	Mixed	192	ImageNet2012	A10
PyTorch	1.9.0a0	SE-ResNeXt101	996	80.24 Top1	1,953 images/sec	8x A10	GIGABYTE G482-Z52-00	21.05-py3	Mixed	112	Imagenet2012	A10
	1.9.0a0	Tacotron2	204	.5 Training Loss	151,946 total output mels/sec	8x A10	GIGABYTE G482-Z52-00	21.05-py3	Mixed	104	LJSpeech 1.1	A10
	1.9.0a0	WaveGlow	637	-5.84 Training Loss	664,022 output samples/sec	8x A10	GIGABYTE G482-Z52-00	21.05-py3	Mixed	10	LJSpeech 1.1	A10
	1.9.0a0	Transformer	1,365	27.8 BLEU Score	70,844 words/sec	8x A10	GIGABYTE G482-Z52-00	21.05-py3	Mixed	2560	wmt14-en-de	A10
	1.9.0a0	FastPitch	177	.25 Training Loss	467,464 frames/sec	8x A10	GIGABYTE G482-Z52-00	21.05-py3	Mixed	32	LJSpeech 1.1	A10
	1.9.0a0	GNMT V2	61	24.49 BLEU Score	292,052 total tokens/sec	8x A10	GIGABYTE G482-Z52-00	21.05-py3	Mixed	128	wmt16-en-de	A10
	1.9.0a0	NCF	1	.96 Hit Rate at 10	47,605,092 samples/sec	8x A10	GIGABYTE G482-Z52-00	21.05-py3	Mixed	131072	MovieLens 20M	A10
	1.9.0a0	BERT-LARGE	13	91.3 F1	236 sequences/sec	8x A10	GIGABYTE G482-Z52-00	21.05-py3	Mixed	10	SQuaD v1.1	A10
	1.9.0a0	Transformer-XL Base	176	22.16 Perplexity	187,731 total tokens/sec	8x A10	GIGABYTE G482-Z52-00	21.05-py3	Mixed	32	WikiText-103	A10
Tensorflow	1.15.5	ResNet-50 v1.5	266	76.74 Top1	7,283 images/sec	8x A10	GIGABYTE G482-Z52-00	21.05-py3	Mixed	256	ImageNet2012	A10
	1.15.5	U-Net Industrial	1	.99 IoU Threshold 0.95	550 images/sec	8x A10	GIGABYTE G482-Z52-00	21.05-py3	Mixed	2	DAGM2007	A10
	1.15.5	U-Net Medical	14	.9 DICE Score	324 images/sec	8x A10	GIGABYTE G482-Z52-00	21.05-py3	Mixed	8	EM segmentation challenge	A10
	1.15.5	VAE-CF	1	.43 NDCG@100	664,902 users processed/sec	8x A10	GIGABYTE G482-Z52-00	21.05-py3	TF32	3072	MovieLens 20M	A10
	1.15.5	SE-ResNext101	866	79.65 Top1	2,240 images/sec	8x A10	GIGABYTE G482-Z52-00	21.05-py3	Mixed	96	Imagenet2012	A10
	2.4.0	Electra Fine-Tuning	6	92.62 F1	745 sequences/sec	8x A10	GIGABYTE G482-Z52-00	21.05-py3	Mixed	16	SQuaD v1.1	A10

Server with a hyphen is a pre-production server
BERT-Large = BERT-Large Fine Tuning (Squadv1.1) with Sequence Length of 384

V100 Training Performance

Framework	Framework Version	Network	Time to Train (mins)	Accuracy	Throughput	GPU	Server	Container	Precision	Batch Size	Dataset	GPU Version
MXNet	1.8.0	ResNet-50 v1.5	169	77.37 Top1	11,740 images/sec	8x V100	DGX-2	21.05-py3	Mixed	256	ImageNet2012	V100-SXM3-32GB
PyTorch	1.8.0a0	Mask R-CNN	269	.34 AP Segm	109 images/sec	8x V100	DGX-2	20.12-py3	Mixed	8	COCO 2014	V100-SXM3-32GB
	1.9.0a0	Tacotron2	185	.5 Training Loss	165,599 total output mels/sec	8x V100	DGX-2	21.05-py3	Mixed	104	LJSpeech 1.1	V100-SXM3-32GB
	1.9.0a0	WaveGlow	458	-5.65 Training Loss	922,820 output samples/sec	8x V100	DGX-2	21.05-py3	Mixed	10	LJSpeech 1.1	V100-SXM3-32GB
	1.6.0a0	Jasper	6,300	3.49 dev-clean WER	312 sequences/sec	8x V100	DGX-2	20.06-py3	Mixed	64	LibriSpeech	V100 SXM2-32GB
	1.9.0a0	Transformer	470	27.6 BLEU Score	210,671 words/sec	8x V100	DGX-2	21.05-py3	Mixed	5120	wmt14-en-de	V100-SXM3-32GB
	1.6.0a0	FastPitch	354	.18 Training Loss	570,968 frames/sec	8x V100	DGX-1	20.06-py3	Mixed	32	LJSpeech 1.1	V100 SXM2-16GB
	1.6.0a0	GNMT V2	39	24.38 BLEU Score	447,832 total tokens/sec	8x V100	DGX-1	20.06-py3	Mixed	128	wmt16-en-de	V100 SXM2-16GB
	1.9.0a0	NCF	1	.96 Hit Rate at 10	98,983,601 samples/sec	8x V100	DGX-2	21.05-py3	Mixed	131072	MovieLens 20M	V100-SXM3-32GB
	1.9.0a0	BERT-LARGE	8	91.31 F1	368 sequences/sec	8x V100	DGX-2	21.05-py3	Mixed	10	SQuaD v1.1	V100-SXM3-32GB
	1.9.0a0	Transformer-XL Base	115	22.05 Perplexity	286,404 total tokens/sec	8x V100	DGX-2	21.05-py3	Mixed	32	WikiText-103	V100-SXM3-32GB
Tensorflow	1.15.5	ResNet-50 v1.5	185	76.92 Top1	10,476 images/sec	8x V100	DGX-2	21.05-py3	Mixed	256	ImageNet2012	V100-SXM3-32GB
	1.15.5	ResNext101	416	79.36 Top1	4,669 images/sec	8x V100	DGX-2	21.04-py3	Mixed	128	Imagenet2012	V100-SXM3-32GB
	1.15.5	SE-ResNext101	504	79.81 Top1	3,867 images/sec	8x V100	DGX-2	21.05-py3	Mixed	96	Imagenet2012	V100-SXM3-32GB
	1.15.5	U-Net Industrial	1	.99 IoU Threshold 0.95	668 images/sec	8x V100	DGX-2	21.05-py3	Mixed	2	DAGM2007	V100-SXM3-32GB
	2.4.0	U-Net Medical	12	.84 DICE Score	473 images/sec	8x V100	DGX-2	21.05-py3	Mixed	8	EM segmentation challenge	V100-SXM3-32GB
	1.15.5	VAE-CF	1	.43 NDCG@100	913,611 users processed/sec	8x V100	DGX-2	21.05-py3	Mixed	3072	MovieLens 20M	V100-SXM3-32GB
	1.15.4	Wide and Deep	185	.68 MAP at 12	643,334 samples/sec	8x V100	DGX-1	20.10-py3	Mixed	16384	Kaggle Outbrain Click Prediction	V100 SXM2-16GB
	1.15.5	BERT-LARGE	18	91.39 F1	325 sequences/sec	8x V100	DGX-2	21.05-py3	Mixed	10	SQuaD v1.1	V100-SXM3-32GB
	2.2.0	Electra Fine-Tuning	6	92.72 F1	1,051 sequences/sec	8x V100	DGX-1	20.07-py3	Mixed	16	SQuaD v1.1	V100 SXM2-16GB

FastPitch throughput metric frames/sec refers to mel-scale spectrogram frames/sec
BERT-Large = BERT-Large Fine Tuning (Squadv1.1) with Sequence Length of 384

T4 Training Performance

Framework	Framework Version	Network	Time to Train (mins)	Accuracy	Throughput	GPU	Server	Container	Precision	Batch Size	Dataset	GPU Version
MXNet	1.8.0	ResNet-50 v1.5	492	77.3 Top1	3,967 images/sec	8x T4	Supermicro SYS-4029GP-TRT	21.05-py3	Mixed	192	ImageNet2012	NVIDIA T4
PyTorch	1.7.0a0	ResNeXt101	1,738	78.75 Top1	1,124 images/sec	8x T4	Supermicro SYS-4029GP-TRT	20.10-py3	Mixed	128	Imagenet2012	NVIDIA T4
	1.9.0a0	SE-ResNeXt101	1,770	79.94 Top1	1,102 images/sec	8x T4	Supermicro SYS-4029GP-TRT	21.05-py3	Mixed	112	Imagenet2012	NVIDIA T4
	1.9.0a0	Tacotron2	245	.52 Training Loss	126,056 total output mels/sec	8x T4	Supermicro SYS-4029GP-TRT	21.05-py3	Mixed	104	LJSpeech 1.1	NVIDIA T4
	1.9.0a0	WaveGlow	1,016	-5.87 Training Loss	411,083 output samples/sec	8x T4	Supermicro SYS-4029GP-TRT	21.05-py3	Mixed	10	LJSpeech 1.1	NVIDIA T4
	1.9.0a0	Transformer	2,288	27.65 BLEU Score	42,030 words/sec	8x T4	Supermicro SYS-4029GP-TRT	21.05-py3	Mixed	2560	wmt14-en-de	NVIDIA T4
	1.7.0a0	FastPitch	319	.21 Training Loss	281,406 frames/sec	8x T4	Supermicro SYS-4029GP-TRT	20.10-py3	Mixed	32	LJSpeech 1.1	NVIDIA T4
	1.9.0a0	GNMT V2	103	24.45 BLEU Score	168,938 total tokens/sec	8x T4	Supermicro SYS-4029GP-TRT	21.05-py3	Mixed	128	wmt16-en-de	NVIDIA T4
	1.9.0a0	NCF	2	.96 Hit Rate at 10	28,185,539 samples/sec	8x T4	Supermicro SYS-4029GP-TRT	21.05-py3	Mixed	131072	MovieLens 20M	NVIDIA T4
	1.9.0a0	BERT-LARGE	22	91.34 F1	136 sequences/sec	8x T4	Supermicro SYS-4029GP-TRT	21.05-py3	Mixed	10	SQuaD v1.1	NVIDIA T4
	1.9.0a0	Transformer-XL Base	320	22.12 Perplexity	102,981 total tokens/sec	8x T4	Supermicro SYS-4029GP-TRT	21.05-py3	Mixed	32	WikiText-103	NVIDIA T4
Tensorflow	1.15.5	ResNet-50 v1.5	563	76.83 Top1	3,416 images/sec	8x T4	Supermicro SYS-4029GP-TRT	21.05-py3	Mixed	256	ImageNet2012	NVIDIA T4
	1.15.5	U-Net Industrial	2	.99 IoU Threshold 0.95	312 images/sec	8x T4	Supermicro SYS-4029GP-TRT	21.05-py3	Mixed	2	DAGM2007	NVIDIA T4
	1.15.5	U-Net Medical	31	.89 DICE Score	158 images/sec	8x T4	Supermicro SYS-4029GP-TRT	21.05-py3	Mixed	8	EM segmentation challenge	NVIDIA T4
	1.15.5	VAE-CF	2	.43 NDCG@100	389,406 users processed/sec	8x T4	Supermicro SYS-4029GP-TRT	21.05-py3	Mixed	3072	MovieLens 20M	NVIDIA T4
	1.15.4	SSD	112	.28 mAP	549 images/sec	8x T4	Supermicro SYS-4029GP-TRT	20.12-py3	Mixed	32	COCO 2017	NVIDIA T4
	1.15.5	Mask R-CNN	492	.34 AP Segm	53 samples/sec	8x T4	Supermicro SYS-4029GP-TRT	21.03-py3	Mixed	4	COCO 2014	NVIDIA T4
	1.15.5	ResNext101	1,224	79.38 Top1	1,575 images/sec	8x T4	Supermicro SYS-4029GP-TRT	21.02-py3	Mixed	128	Imagenet2012	NVIDIA T4
	1.15.5	SE-ResNext101	1,552	79.53 Top1	1,243 images/sec	8x T4	Supermicro SYS-4029GP-TRT	21.05-py3	Mixed	96	Imagenet2012	NVIDIA T4
	2.4.0	Electra Fine-Tuning	9	92.72 F1	412 sequences/sec	8x T4	Supermicro SYS-4029GP-TRT	21.05-py3	Mixed	16	SQuaD v1.1	NVIDIA T4

FastPitch throughput metric frames/sec refers to mel-scale spectrogram frames/sec
BERT-Large = BERT-Large Fine Tuning (Squadv1.1) with Sequence Length of 384

Deploying AI in real world applications, requires training the networks to convergence at a specified accuracy. This is the best methodology to test AI systems, and is typically done on multi-accelerator systems (see the ‘Training-Convergence’ tab or read our blog on convergence for more details) to shorten training-to-convergence times, especially for recurrent monthly container builds.

Scenarios that are not typically used in real-world training, such as single GPU throughput are illustrated in the table below, and provided for reference as an indication of single chip throughput of the platform.

NVIDIA’s complete solution stack, from hardware to software, allows data scientists to deliver unprecedented acceleration at every scale. Visit NVIDIA GPU Cloud (NGC) to pull containers and quickly get up and running with deep learning.

Single GPU Training Performance of NVIDIA A100, A40, A30, A10, V100 and T4

Benchmarks are reproducible by following links to NGC scripts

A100 Training Performance

Framework	Framework Version	Network	Throughput	GPU	Server	Container	Precision	Batch Size	Dataset	GPU Version
MXNet	-	ResNet-50 v1.5	2,751 images/sec	1x A100	-	-	Mixed	408	ImageNet2012	A100-SXM-80GB
PyTorch	1.9.0a0	Mask R-CNN	29 images/sec	1x A100	DGX A100	21.05-py3	TF32	8	COCO 2014	A100-SXM-80GB
	1.9.0a0	SSD v1.1	445 images/sec	1x A100	DGX A100	21.05-py3	Mixed	128	COCO 2017	A100-SXM-80GB
	1.9.0a0	Tacotron2	38,901 total output mels/sec	1x A100	DGX A100	21.05-py3	TF32	128	LJSpeech 1.1	A100-SXM-80GB
	1.9.0a0	WaveGlow	202,226 output samples/sec	1x A100	DGX A100	21.05-py3	Mixed	10	LJSpeech 1.1	A100-SXM-80GB
	1.9.0a0	Jasper	83 sequences/sec	1x A100	DGX A100	21.05-py3	Mixed	64	LibriSpeech	A100-SXM-80GB
	1.6.0a0	Transformer	82,618 words/sec	1x A100	DGX A100	20.06-py3	Mixed	10240	wmt14-en-de	A100 SXM4-40GB
	1.9.0a0	FastPitch	180,170 frames/sec	1x A100	DGX A100	21.05-py3	Mixed	128	LJSpeech 1.1	A100-SXM-80GB
	1.9.0a0	GNMT V2	157,886 total tokens/sec	1x A100	DGX A100	21.05-py3	Mixed	128	wmt16-en-de	A100-SXM-80GB
	1.9.0a0	NCF	37,371,854 samples/sec	1x A100	DGX A100	21.05-py3	Mixed	1048576	MovieLens 20M	A100-SXM-80GB
	1.9.0a0	BERT-LARGE	122 sequences/sec	1x A100	DGX A100	21.05-py3	Mixed	32	SQuaD v1.1	A100-SXM-80GB
	1.9.0a0	Transformer-XL Large	28,503 total tokens/sec	1x A100	DGX A100	21.05-py3	Mixed	16	WikiText-103	A100-SXM-80GB
	1.9.0a0	Transformer-XL Base	83,345 total tokens/sec	1x A100	DGX A100	21.05-py3	Mixed	128	WikiText-103	A100-SXM-80GB
Tensorflow	1.15.5	ResNet-50 v1.5	2,662 images/sec	1x A100	DGX A100	21.05-py3	Mixed	256	ImageNet2012	A100-SXM-80GB
	1.15.5	ResNext101	1,312 images/sec	1x A100	DGX A100	21.05-py3	Mixed	256	Imagenet2012	A100-SXM-80GB
	1.15.5	SE-ResNext101	1,135 images/sec	1x A100	DGX A100	21.05-py3	Mixed	256	Imagenet2012	A100-SXM-80GB
	1.15.5	U-Net Industrial	365 images/sec	1x A100	DGX A100	21.05-py3	Mixed	16	DAGM2007	A100-SXM4-40GB
	2.4.0	U-Net Medical	149 images/sec	1x A100	DGX A100	21.05-py3	Mixed	8	EM segmentation challenge	A100-SXM-80GB
	1.15.5	VAE-CF	393,529 users processed/sec	1x A100	DGX A100	21.05-py3	Mixed	24576	MovieLens 20M	A100-SXM-80GB
	1.15.5	Wide and Deep	321,808 samples/sec	1x A100	DGX A100	21.05-py3	TF32	131072	Kaggle Outbrain Click Prediction	A100-SXM4-40GB
	1.15.5	BERT-LARGE	117 sequences/sec	1x A100	DGX A100	21.05-py3	Mixed	24	SQuaD v1.1	A100-SXM-80GB
	2.4.0	Electra Fine Tuning	348 sequences/sec	1x A100	DGX A100	21.05-py3	Mixed	32	SQuaD v1.1	A100-SXM-80GB
	-	EfficientNet-B4	332 images/sec	1x A100	DGX A100	-	Mixed	160	ImageNet2012	A100-SXM-80GB
	1.15.5	NCF	40,189,461 samples/sec	1x A100	DGX A100	21.05-py3	Mixed	1048576	MovieLens 20M	A100-SXM4-40GB

FastPitch throughput metric frames/sec refers to mel-scale spectrogram frames/sec | Server with a hyphen indicates a pre-production server
BERT-Large = BERT-Large Fine Tuning (Squadv1.1) with Sequence Length of 384
EfficientNet-B4: Basic Augmentation | cuDNN Version = 8.0.5.32 | NCCL Version = 2.7.8 | Installation Source = NGC

A40 Training Performance

Framework	Framework Version	Network	Throughput	GPU	Server	Container	Precision	Batch Size	Dataset	GPU Version
PyTorch	1.9.0a0	NCF	20,386,212 samples/sec	1x A40	GIGABYTE G482-Z52-00	21.05-py3	Mixed	1048576	MovieLens 20M	A40
	1.9.0a0	BERT-LARGE	62 sequences/sec	1x A40	GIGABYTE G482-Z52-00	21.05-py3	Mixed	32	SQuaD v1.1	A40
Tensorflow	1.15.5	BERT-LARGE	55 sequences/sec	1x A40	GIGABYTE G482-Z52-00	21.05-py3	Mixed	24	SQuaD v1.1	A40

FastPitch throughput metric frames/sec refers to mel-scale spectrogram frames/sec | Server with a hyphen indicates a pre-production server

A30 Training Performance

Framework	Framework Version	Network	Throughput	GPU	Server	Container	Precision	Batch Size	Dataset	GPU Version
MXNet	1.8.0	ResNet-50 v1.5	1,254 images/sec	1x A30	GIGABYTE G482-Z52-SW-QZ-001	21.05-py3	Mixed	192	ImageNet2012	A30
PyTorch	1.9.0a0	SSD v1.1	226 images/sec	1x A30	GIGABYTE G482-Z52-SW-QZ-001	21.05-py3	Mixed	64	COCO 2017	A30
	1.9.0a0	Tacotron2	18,541 total output mels/sec	1x A30	GIGABYTE G482-Z52-SW-QZ-001	21.05-py3	Mixed	104	LJSpeech 1.1	A30
	1.9.0a0	WaveGlow	119,858 output samples/sec	1x A30	GIGABYTE G482-Z52-SW-QZ-001	21.05-py3	Mixed	10	LJSpeech 1.1	A30
	1.9.0a0	Transformer	24,662 words/sec	1x A30	GIGABYTE G482-Z52-SW-QZ-001	21.05-py3	Mixed	2560	wmt14-en-de	A30
	1.9.0a0	FastPitch	100,243 frames/sec	1x A30	GIGABYTE G482-Z52-SW-QZ-001	21.05-py3	Mixed	64	LJSpeech 1.1	A30
	1.9.0a0	NCF	19,634,381 samples/sec	1x A30	GIGABYTE G482-Z52-SW-QZ-001	21.05-py3	Mixed	1048576	MovieLens 20M	A30
	1.9.0a0	GNMT V2	55,112 total tokens/sec	1x A30	GIGABYTE G482-Z52-SW-QZ-001	21.05-py3	TF32	128	wmt16-en-de	A30
	1.9.0a0	Transformer-XL Base	40,981 total tokens/sec	1x A30	GIGABYTE G482-Z52-SW-QZ-001	21.05-py3	Mixed	32	WikiText-103	A30
	1.9.0a0	ResNeXt101	544 images/sec	1x A30	GIGABYTE G482-Z52-SW-QZ-001	21.05-py3	Mixed	112	Imagenet2012	A30
	1.9.0a0	Jasper	34 sequences/sec	1x A30	GIGABYTE G482-Z52-SW-QZ-001	21.05-py3	Mixed	16	LibriSpeech	A30
	1.9.0a0	Transformer-XL Large	12,617 total tokens/sec	1x A30	GIGABYTE G482-Z52-SW-QZ-001	21.05-py3	Mixed	4	WikiText-103	A30
	1.9.0a0	BERT-LARGE	50 sequences/sec	1x A30	GIGABYTE G482-Z52-SW-QZ-001	21.05-py3	Mixed	10	SQuaD v1.1	A30
Tensorflow	1.15.5	ResNet-50 v1.5	1,339 images/sec	1x A30	GIGABYTE G482-Z52-SW-QZ-001	21.05-py3	Mixed	256	ImageNet2012	A30
	1.15.5	ResNext101	595 images/sec	1x A30	GIGABYTE G482-Z52-SW-QZ-001	21.05-py3	Mixed	128	Imagenet2012	A30
	1.15.5	SE-ResNext101	495 images/sec	1x A30	GIGABYTE G482-Z52-SW-QZ-001	21.05-py3	Mixed	96	Imagenet2012	A30
	1.15.5	U-Net Industrial	109 images/sec	1x A30	GIGABYTE G482-Z52-SW-QZ-001	21.05-py3	Mixed	16	DAGM2007	A30
	2.4.0	U-Net Medical	71 images/sec	1x A30	GIGABYTE G482-Z52-SW-QZ-001	21.05-py3	Mixed	8	EM segmentation challenge	A30
	1.15.5	VAE-CF	198,703 users processed/sec	1x A30	GIGABYTE G482-Z52-SW-QZ-001	21.05-py3	Mixed	24576	MovieLens 20M	A30
	1.15.5	Wide and Deep	232,356 samples/sec	1x A30	GIGABYTE G482-Z52-SW-QZ-001	21.05-py3	Mixed	131072	Kaggle Outbrain Click Prediction	A30
	2.4.0	Mask R-CNN	21 samples/sec	1x A30	GIGABYTE G482-Z52-SW-QZ-001	21.05-py3	Mixed	4	COCO 2014	A30
	2.4.0	Electra Fine Tuning	143 sequences/sec	1x A30	GIGABYTE G482-Z52-SW-QZ-001	21.05-py3	Mixed	16	SQuaD v1.1	A30
	1.15.5	SSD	201 images/sec	1x A30	GIGABYTE G482-Z52-SW-QZ-001	21.05-py3	Mixed	32	COCO 2017	A30

FastPitch throughput metric frames/sec refers to mel-scale spectrogram frames/sec | Server with a hyphen indicates a pre-production server

A10 Training Performance

Framework	Framework Version	Network	Throughput	GPU	Server	Container	Precision	Batch Size	Dataset	GPU Version
MXNet	1.8.0	ResNet-50 v1.5	1,019 images/sec	1x A10	GIGABYTE G482-Z52-00	21.05-py3	Mixed	192	ImageNet2012	A10
PyTorch	1.9.0a0	SSD v1.1	175 images/sec	1x A10	GIGABYTE G482-Z52-00	21.05-py3	Mixed	64	COCO 2017	A10
	1.9.0a0	Tacotron2	19,741 total output mels/sec	1x A10	GIGABYTE G482-Z52-00	21.05-py3	Mixed	104	LJSpeech 1.1	A10
	1.9.0a0	WaveGlow	100,432 output samples/sec	1x A10	GIGABYTE G482-Z52-00	21.05-py3	Mixed	10	LJSpeech 1.1	A10
	1.9.0a0	Transformer	22,248 words/sec	1x A10	GIGABYTE G482-Z52-00	21.05-py3	Mixed	2560	wmt14-en-de	A10
	1.9.0a0	FastPitch	92,906 frames/sec	1x A10	GIGABYTE G482-Z52-00	21.05-py3	Mixed	64	LJSpeech 1.1	A10
	1.9.0a0	Transformer-XL Base	34,753 total tokens/sec	1x A10	GIGABYTE G482-Z52-00	21.05-py3	Mixed	32	WikiText-103	A10
	1.9.0a0	GNMT V2	66,783 total tokens/sec	1x A10	GIGABYTE G482-Z52-00	21.05-py3	Mixed	128	wmt16-en-de	A10
	1.9.0a0	ResNeXt101	421 images/sec	1x A10	GIGABYTE G482-Z52-00	21.03-py3	Mixed	128	Imagenet2012	A10
	1.9.0a0	SE-ResNeXt101	327 images/sec	1x A10	GIGABYTE G482-Z52-00	21.03-py3	Mixed	128	Imagenet2012	A10
	1.9.0a0	NCF	16,862,317 samples/sec	1x A10	GIGABYTE G482-Z52-00	21.05-py3	Mixed	1048576	MovieLens 20M	A10
	1.8.0a0	Jasper	29 sequences/sec	1x A10	GIGABYTE G482-Z52-00	21.02-py3	Mixed	32	LibriSpeech	A10
	1.9.0a0	Transformer-XL Large	10,699 total tokens/sec	1x A10	GIGABYTE G482-Z52-00	21.05-py3	Mixed	4	WikiText-103	A10
	1.9.0a0	BERT-LARGE	41 sequences/sec	1x A10	GIGABYTE G482-Z52-00	21.05-py3	Mixed	10	SQuaD v1.1	A10
Tensorflow	1.15.5	ResNet-50 v1.5	996 images/sec	1x A10	GIGABYTE G482-Z52-00	21.05-py3	Mixed	256	ImageNet2012	A10
	1.15.5	ResNext101	460 images/sec	1x A10	GIGABYTE G482-Z52-00	21.03-py3	Mixed	128	Imagenet2012	A10
	1.15.5	SE-ResNext101	342 images/sec	1x A10	GIGABYTE G482-Z52-00	21.03-py3	Mixed	96	Imagenet2012	A10
	1.15.5	U-Net Industrial	96 images/sec	1x A10	GIGABYTE G482-Z52-00	21.05-py3	Mixed	16	DAGM2007	A10
	2.4.0	U-Net Medical	49 images/sec	1x A10	GIGABYTE G482-Z52-00	21.05-py3	Mixed	8	EM segmentation challenge	A10
	1.15.5	VAE-CF	174,947 users processed/sec	1x A10	GIGABYTE G482-Z52-00	21.05-py3	Mixed	24576	MovieLens 20M	A10
	1.15.4	Wide and Deep	249,905 samples/sec	1x A10	Supermicro SYS-1029GQ-TRT	20.11-py3	Mixed	131072	Kaggle Outbrain Click Prediction	A10
	2.4.0	Electra Fine Tuning	128 sequences/sec	1x A10	GIGABYTE G482-Z52-00	21.05-py3	Mixed	16	SQuaD v1.1	A10
	2.4.0	Mask R-CNN	18 samples/sec	1x A10	GIGABYTE G482-Z52-00	21.05-py3	Mixed	4	COCO 2014	A10
	1.15.5	SSD	181 images/sec	1x A10	GIGABYTE G482-Z52-00	21.05-py3	Mixed	32	COCO 2017	A10

FastPitch throughput metric frames/sec refers to mel-scale spectrogram frames/sec | Server with a hyphen indicates a pre-production server

V100 Training Performance

Framework	Framework Version	Network	Throughput	GPU	Server	Container	Precision	Batch Size	Dataset	GPU Version
MXNet	1.8.0	ResNet-50 v1.5	1,473 images/sec	1x V100	DGX-2	21.05-py3	Mixed	256	ImageNet2012	V100-SXM3-32GB
PyTorch	1.9.0a0	ResNeXt101	554 images/sec	1x V100	DGX-2	21.05-py3	Mixed	112	Imagenet2012	V100-SXM3-32GB
	1.9.0a0	SSD v1.1	233 images/sec	1x V100	DGX-2	21.05-py3	Mixed	64	COCO 2017	V100-SXM3-32GB
	1.9.0a0	Tacotron2	24,540 total output mels/sec	1x V100	DGX-2	21.05-py3	Mixed	104	LJSpeech 1.1	V100-SXM3-32GB
	1.9.0a0	WaveGlow	136,702 output samples/sec	1x V100	DGX-2	21.05-py3	Mixed	10	LJSpeech 1.1	V100-SXM3-32GB
	1.9.0a0	Jasper	44 sequences/sec	1x V100	DGX-2	21.05-py3	Mixed	64	LibriSpeech	V100-SXM3-32GB
	1.9.0a0	Transformer	32,218 words/sec	1x V100	DGX-2	21.05-py3	Mixed	5120	wmt14-en-de	V100-SXM3-32GB
	1.9.0a0	FastPitch	123,294 frames/sec	1x V100	DGX-2	21.05-py3	Mixed	64	LJSpeech 1.1	V100-SXM3-32GB
	1.7.0a0	GNMT V2	83,200 total tokens/sec	1x V100	DGX-2	20.09-py3	Mixed	128	wmt16-en-de	V100-SXM3-32GB
	1.9.0a0	NCF	22,193,821 samples/sec	1x V100	DGX-2	21.05-py3	Mixed	1048576	MovieLens 20M	V100-SXM3-32GB
	1.9.0a0	BERT-LARGE	53 sequences/sec	1x V100	DGX-2	21.05-py3	Mixed	10	SQuaD v1.1	V100-SXM3-32GB
	1.9.0a0	Transformer-XL Base	44,072 total tokens/sec	1x V100	DGX-2	21.05-py3	Mixed	32	WikiText-103	V100-SXM3-32GB
	1.9.0a0	Transformer-XL Large	15,360 total tokens/sec	1x V100	DGX-2	21.05-py3	Mixed	8	WikiText-103	V100-SXM3-32GB
Tensorflow	1.15.5	ResNet-50 v1.5	1,393 images/sec	1x V100	DGX-2	21.05-py3	Mixed	256	ImageNet2012	V100-SXM3-32GB
	1.15.5	ResNext101	632 images/sec	1x V100	DGX-2	21.05-py3	Mixed	128	Imagenet2012	V100-SXM3-32GB
	1.15.5	SE-ResNext101	545 images/sec	1x V100	DGX-2	21.05-py3	Mixed	96	Imagenet2012	V100-SXM3-32GB
	1.15.2	U-Net Industrial	169 images/sec	1x V100	DGX-1	20.06-py3	Mixed	16	DAGM2007	V100 SXM2-16GB
	1.15.5	U-Net Medical	68 images/sec	1x V100	DGX-2	21.05-py3	Mixed	8	EM segmentation challenge	V100-SXM3-32GB
	1.15.5	VAE-CF	222,573 users processed/sec	1x V100	DGX-2	21.05-py3	Mixed	24576	MovieLens 20M	V100-SXM3-32GB
	1.15.5	Wide and Deep	301,631 samples/sec	1x V100	DGX-2	21.05-py3	Mixed	131072	Kaggle Outbrain Click Prediction	V100-SXM3-32GB
	1.15.5	BERT-LARGE	48 sequences/sec	1x V100	DGX-2	21.05-py3	Mixed	10	SQuaD v1.1	V100-SXM3-32GB
	2.4.0	Electra Fine-Tuning	192 sequences/sec	1x V100	DGX-2	21.05-py3	Mixed	32	SQuaD v1.1	V100-SXM3-32GB
	2.4.0	Mask R-CNN	22 samples/sec	1x V100	DGX-2	21.05-py3	Mixed	4	COCO 2014	V100-SXM3-32GB
	1.15.5	SSD	222 images/sec	1x V100	DGX-2	21.05-py3	Mixed	32	COCO 2017	V100-SXM3-32GB

FastPitch throughput metric frames/sec refers to mel-scale spectrogram frames/sec
BERT-Large = BERT-Large Fine Tuning (Squadv1.1) with Sequence Length of 384

T4 Training Performance

Framework	Framework Version	Network	Throughput	GPU	Server	Container	Precision	Batch Size	Dataset	GPU Version
MXNet	1.8.0	ResNet-50 v1.5	514 images/sec	1x T4	Supermicro SYS-1029GQ-TRT	21.03-py3	Mixed	64	ImageNet2012	NVIDIA T4
PyTorch	1.8.0	ResNeXt101	209 images/sec	1x T4	Supermicro SYS-1029GQ-TRT	21.02-py3	Mixed	128	Imagenet2012	NVIDIA T4
	1.9.0a0	Tacotron2	16,884 total output mels/sec	1x T4	Supermicro SYS-4029GP-TRT	21.05-py3	Mixed	104	LJSpeech 1.1	NVIDIA T4
	1.9.0a0	WaveGlow	55,618 output samples/sec	1x T4	Supermicro SYS-4029GP-TRT	21.05-py3	Mixed	10	LJSpeech 1.1	NVIDIA T4
	1.9.0a0	Transformer	10,512 words/sec	1x T4	Supermicro SYS-4029GP-TRT	21.05-py3	Mixed	2560	wmt14-en-de	NVIDIA T4
	1.9.0a0	FastPitch	41,078 frames/sec	1x T4	Supermicro SYS-4029GP-TRT	21.05-py3	Mixed	64	LJSpeech 1.1	NVIDIA T4
	1.9.0a0	GNMT V2	31,038 total tokens/sec	1x T4	Supermicro SYS-4029GP-TRT	21.05-py3	Mixed	128	wmt16-en-de	NVIDIA T4
	1.9.0a0	NCF	8,104,090 samples/sec	1x T4	Supermicro SYS-4029GP-TRT	21.05-py3	Mixed	1048576	MovieLens 20M	NVIDIA T4
	1.9.0a0	BERT-LARGE	19 sequences/sec	1x T4	Supermicro SYS-4029GP-TRT	21.05-py3	Mixed	10	SQuaD v1.1	NVIDIA T4
	1.9.0a0	Transformer-XL Base	17,119 total tokens/sec	1x T4	Supermicro SYS-4029GP-TRT	21.05-py3	Mixed	32	WikiText-103	NVIDIA T4
	1.8.0a0	Jasper	14 sequences/sec	1x T4	Supermicro SYS-4029GP-TRT	20.12-py3	Mixed	32	LibriSpeech	NVIDIA T4
	1.8.0a0	SE-ResNeXt101	163 images/sec	1x T4	Supermicro SYS-1029GQ-TRT	21.02-py3	Mixed	128	Imagenet2012	NVIDIA T4
	1.9.0a0	Transformer-XL Large	5,231 total tokens/sec	1x T4	Supermicro SYS-4029GP-TRT	21.05-py3	Mixed	4	WikiText-103	NVIDIA T4
Tensorflow	1.15.5	ResNet-50 v1.5	452 images/sec	1x T4	Supermicro SYS-4029GP-TRT	21.05-py3	Mixed	256	ImageNet2012	NVIDIA T4
	1.15.5	U-Net Industrial	47 images/sec	1x T4	Supermicro SYS-4029GP-TRT	21.05-py3	Mixed	16	DAGM2007	NVIDIA T4
	1.15.5	U-Net Medical	22 images/sec	1x T4	Supermicro SYS-4029GP-TRT	21.05-py3	Mixed	8	EM segmentation challenge	NVIDIA T4
	1.15.5	VAE-CF	83,310 users processed/sec	1x T4	Supermicro SYS-4029GP-TRT	21.05-py3	Mixed	24576	MovieLens 20M	NVIDIA T4
	1.15.5	SSD	97 images/sec	1x T4	Supermicro SYS-4029GP-TRT	21.05-py3	Mixed	32	COCO 2017	NVIDIA T4
	2.4.0	Mask R-CNN	9 samples/sec	1x T4	Supermicro SYS-4029GP-TRT	21.05-py3	Mixed	4	COCO 2014	NVIDIA T4
	1.15.5	Wide and Deep	203,867 samples/sec	1x T4	Supermicro SYS-1029GQ-TRT	21.03-py3	Mixed	131072	Kaggle Outbrain Click Prediction	NVIDIA T4
	1.15.5	SE-ResNext101	170 images/sec	1x T4	Supermicro SYS-4029GP-TRT	21.05-py3	Mixed	96	Imagenet2012	NVIDIA T4
	1.15.5	ResNext101	207 images/sec	1x T4	Supermicro SYS-4029GP-TRT	21.05-py3	Mixed	128	Imagenet2012	NVIDIA T4
	2.4.0	Electra Fine-Tuning	62 sequences/sec	1x T4	Supermicro SYS-4029GP-TRT	21.05-py3	Mixed	16	SQuaD v1.1	NVIDIA T4

FastPitch throughput metric frames/sec refers to mel-scale spectrogram frames/sec
BERT-Large = BERT-Large Fine Tuning (Squadv1.1) with Sequence Length of 384

Real-world AI inferencing demands high throughput and low latencies with maximum efficiency across use cases. An industry leading solution enables customers to quickly deploy AI models into real-world production with the highest performance from data centers to the edge.

NVIDIA landed top performance spots on all MLPerf™ Inference 1.0 tests, the AI-industry’s leading benchmark. NVIDIA TensorRT™ running on NVIDIA Tensor Core GPUs enable the most efficient deep learning inference performance across multiple application areas and models. This versatility provides wide latitude to data scientists to create the optimal low-latency solution. Visit NVIDIA® GPU Cloud (NGC) to download any of these containers and immediately race into production. The inference whitepaper provides an overview of inference platforms.

The Triton Inference Server is an open source inference serving software which maximizes performance and simplifies the deployment of AI models at scale in production. Triton lets teams deploy trained AI models from multiple model frameworks (TensorFlow, TensorRT, PyTorch, ONNX Runtime, OpenVino, or custom backends). They can deploy from local storage, Google Cloud Platform, Azure Storage, or Amazon S3 on any GPU or CPU based infrastructure (in cloud, data center, or embedded devices). Triton is open source on GitHub and available as a docker container on NGC.

MLPerf Inference v1.0 Performance Benchmarks

Offline Scenario - Closed Division

Network	Throughput	Target Accuracy	GPU	Server	Dataset	GPU Version
ResNet-50 v1.5	105,677 samples/sec	76.46% Top1	8x A10	Supermicro 4029GP-TRT-OTO-28	ImageNet	A10
	141,518 samples/sec	76.46% Top1	8x A30	Gigabyte G482-Z54	ImageNet	A30
	5,108 samples/sec	76.46% Top1	1x1g.10gb A100	DGX A100	ImageNet	A100 SXM-80GB
	38,010 samples/sec	76.46% Top1	1x A100	DGX A100	ImageNet	A100 SXM-80GB
	304,876 samples/sec	76.46% Top1	8x A100	DGX A100	ImageNet	A100 SXM-80GB
	248,179 samples/sec	76.46% Top1	8x A100	Gigabyte G482-Z54	ImageNet	A100-PCIe-40GB
	132,926 samples/sec	76.46% Top1	4x A100	DGX-Station-A100	ImageNet	A100 SXM-80GB
SSD ResNet-34	2,496 samples/sec	0.2 mAP	8x A10	Supermicro 4029GP-TRT-OTO-28	COCO	A10
	3,756 samples/sec	0.2 mAP	8x A30	Gigabyte G482-Z54	COCO	A30
	134 samples/sec	0.2 mAP	1x1g.10gb A100	DGX A100	COCO	A100 SXM-80GB
	989 samples/sec	0.2 mAP	1x A100	DGX A100	COCO	A100 SXM-80GB
	7,879 samples/sec	0.2 mAP	8x A100	DGX A100	COCO	A100 SXM-80GB
	6,586 samples/sec	0.2 mAP	8x A100	Gigabyte G482-Z54	COCO	A100-PCIe-40GB
	3,370 samples/sec	0.2 mAP	4x A100	DGX-Station-A100	COCO	A100 SXM-80GB
3D-UNet	172 samples/sec	0.853 DICE mean	8x A10	Supermicro 4029GP-TRT-OTO-28	BraTS 2019	A10
	237 samples/sec	0.853 DICE mean	8x A30	Gigabyte G482-Z54	BraTS 2019	A30
	7 samples/sec	0.853 DICE mean	1x1g.10gb A100	DGX A100	BraTS 2019	A100 SXM-80GB
	61 samples/sec	0.853 DICE mean	1x A100	DGX A100	BraTS 2019	A100 SXM-80GB
	480 samples/sec	0.853 DICE mean	8x A100	DGX A100	BraTS 2019	A100 SXM-80GB
	412 samples/sec	0.853 DICE mean	8x A100	Gigabyte G482-Z54	BraTS 2019	A100-PCIe-40GB
	214 samples/sec	0.853 DICE mean	4x A100	DGX-Station-A100	BraTS 2019	A100 SXM-80GB
RNN-T	36,116 samples/sec	7.45% WER	8x A10	Supermicro 4029GP-TRT-OTO-28	LibriSpeech	A10
	51,690 samples/sec	7.45% WER	8x A30	Gigabyte G482-Z54	LibriSpeech	A30
	1,553 samples/sec	7.45% WER	1x1g.10gb A100	DGX A100	LibriSpeech	A100 SXM-80GB
	14,008 samples/sec	7.45% WER	1x A100	DGX A100	LibriSpeech	A100 SXM-80GB
	105,677 samples/sec	7.45% WER	8x A100	DGX A100	LibriSpeech	A100 SXM-80GB
	90,853 samples/sec	7.45% WER	8x A100	Gigabyte G482-Z54	LibriSpeech	A100-PCIe-40GB
	48,886 samples/sec	7.45% WER	4x A100	DGX-Station-A100	LibriSpeech	A100 SXM-80GB
BERT	8,454 samples/sec	90.07% f1	8x A10	Supermicro 4029GP-TRT-OTO-28	SQuAD v1.1	A10
	13,260 samples/sec	90.07% f1	8x A30	Gigabyte G482-Z54	SQuAD v1.1	A30
	492 samples/sec	90.07% f1	1x1g.10gb A100	DGX A100	SQuAD v1.1	A100 SXM-80GB
	3,602 samples/sec	90.07% f1	1x A100	DGX A100	SQuAD v1.1	A100 SXM-80GB
	28,347 samples/sec	90.07% f1	8x A100	DGX A100	SQuAD v1.1	A100 SXM-80GB
	22,847 samples/sec	90.07% f1	8x A100	Gigabyte G482-Z54	SQuAD v1.1	A100-PCIe-40GB
	11,305 samples/sec	90.07% f1	4x A100	DGX-Station-A100	SQuAD v1.1	A100 SXM-80GB
DLRM	772,378 samples/sec	80.25% AUC	8x A10	Supermicro 4029GP-TRT-OTO-28	Criteo 1TB Click Logs	A10
	1,067,510 samples/sec	80.25% AUC	8x A30	Gigabyte G482-Z54	Criteo 1TB Click Logs	A30
	36,473 samples/sec	80.25% AUC	1x1g.10gb A100	DGX A100	Criteo 1TB Click Logs	A100 SXM-80GB
	311,826 samples/sec	80.25% AUC	1x A100	DGX A100	Criteo 1TB Click Logs	A100 SXM-80GB
	2,462,300 samples/sec	80.25% AUC	8x A100	DGX A100	Criteo 1TB Click Logs	A100 SXM-80GB
	1,057,550 samples/sec	80.25% AUC	4x A100	DGX-Station-A100	Criteo 1TB Click Logs	A100 SXM-80GB

Server Scenario - Closed Division

Network	Throughput	Target Accuracy	MLPerf Server Latency Constraints (ms)	GPU	Server	Dataset	GPU Version
ResNet-50 v1.5	87,984 queries/sec	76.46% Top1	15	8x A10	Supermicro 4029GP-TRT-OTO-28	ImageNet	A10
	115,987 queries/sec	76.46% Top1	15	8x A30	Gigabyte G482-Z54	ImageNet	A30
	3,602 queries/sec	76.46% Top1	15	1x1g.10gb A100	DGX A100	ImageNet	A100 SXM-80GB
	30,794 queries/sec	76.46% Top1	15	1x A100	DGX A100	ImageNet	A100 SXM-80GB
	259,994 queries/sec	76.46% Top1	15	8x A100	DGX A100	ImageNet	A100 SXM-80GB
	207,976 queries/sec	76.46% Top1	15	8x A100	Gigabyte G482-Z54	ImageNet	A100-PCIe-40GB
	106,988 queries/sec	76.46% Top1	15	4x A100	DGX-Station-A100	ImageNet	A100 SXM-80GB
SSD ResNet-34	2,000 queries/sec	0.2 mAP	100	8x A10	Supermicro 4029GP-TRT-OTO-28	COCO	A10
	3,575 queries/sec	0.2 mAP	100	8x A30	Gigabyte G482-Z54	COCO	A30
	100 queries/sec	0.2 mAP	100	1x1g.10gb A100	DGX A100	COCO	A100 SXM-80GB
	926 queries/sec	0.2 mAP	100	1x A100	DGX A100	COCO	A100 SXM-80GB
	7,654 queries/sec	0.2 mAP	100	8x A100	DGX A100	COCO	A100 SXM-80GB
	6,162 queries/sec	0.2 mAP	100	8x A100	Gigabyte G482-Z54	COCO	A100-PCIe-40GB
	3,081 queries/sec	0.2 mAP	100	4x A100	DGX-Station-A100	COCO	A100 SXM-80GB
RNN-T	22,597 queries/sec	7.45% WER	1,000	8x A10	Supermicro 4029GP-TRT-OTO-28	LibriSpeech	A10
	36,991 queries/sec	7.45% WER	1,000	8x A30	Gigabyte G482-Z54	LibriSpeech	A30
	1,303 queries/sec	7.45% WER	1,000	1x1g.10gb A100	DGX A100	LibriSpeech	A100 SXM-80GB
	12,751 queries/sec	7.45% WER	1,000	1x A100	DGX A100	LibriSpeech	A100 SXM-80GB
	103,986 queries/sec	7.45% WER	1,000	8x A100	DGX A100	LibriSpeech	A100 SXM-80GB
	85,985 queries/sec	7.45% WER	1,000	8x A100	Gigabyte G482-Z54	LibriSpeech	A100-PCIe-40GB
	43,389 queries/sec	7.45% WER	1,000	4x A100	DGX-Station-A100	LibriSpeech	A100 SXM-80GB
BERT	7,204 queries/sec	90.07% f1	130	8x A10	Supermicro 4029GP-TRT-OTO-28	SQuAD v1.1	A10
	11,500 queries/sec	90.07% f1	130	8x A30	Gigabyte G482-Z54	SQuAD v1.1	A30
	381 queries/sec	90.07% f1	130	1x1g.10gb A100	DGX A100	SQuAD v1.1	A100 SXM-80GB
	3,202 queries/sec	90.07% f1	130	1x A100	DGX A100	SQuAD v1.1	A100 SXM-80GB
	25,792 queries/sec	90.07% f1	130	8x A100	DGX A100	SQuAD v1.1	A100 SXM-80GB
	20,792 queries/sec	90.07% f1	130	8x A100	Gigabyte G482-Z54	SQuAD v1.1	A100-PCIe-40GB
	10,203 queries/sec	90.07% f1	130	4x A100	DGX-Station-A100	SQuAD v1.1	A100 SXM-80GB
DLRM	680,147 queries/sec	80.25% AUC	30	8x A10	Supermicro 4029GP-TRT-OTO-28	Criteo 1TB Click Logs	A10
	750,204 queries/sec	80.25% AUC	30	8x A30	Gigabyte G482-Z54	Criteo 1TB Click Logs	A30
	35,991 queries/sec	80.25% AUC	30	1x1g.10gb A100	DGX A100	Criteo 1TB Click Logs	A100 SXM-80GB
	286,002 queries/sec	80.25% AUC	30	1x A100	DGX A100	Criteo 1TB Click Logs	A100 SXM-80GB
	2,302,570 queries/sec	80.25% AUC	30	8x A100	DGX A100	Criteo 1TB Click Logs	A100 SXM-80GB
	942,395 queries/sec	80.25% AUC	30	4x A100	DGX-Station-A100	Criteo 1TB Click Logs	A100 SXM-80GB

Power Efficiency Offline Scenario - Closed Division

Network	Throughput	Throughput per Watt	GPU	Server	Dataset	GPU Version
ResNet-50 v1.5	213,599 samples/sec	97.31 samples/sec/watt	8x A100	Gigabyte G482-Z54	ImageNet	A100 PCIe-40GB
	270,706 samples/sec	78.27 samples/sec/watt	8x A100	DGX A100	ImageNet	A100 SXM-80GB
	124,529 samples/sec	98.14 samples/sec/watt	4x A100	DGX-Station-A100	ImageNet	A100 SXM-80GB
SSD ResNet-34	5,824 samples/sec	2.6 samples/sec/watt	8x A100	Gigabyte G482-Z54	COCO	A100 PCIe-40GB
	6,875 samples/sec	1.96 samples/sec/watt	8x A100	DGX A100	COCO	A100 SXM-80GB
	3,110 samples/sec	2.44 samples/sec/watt	4x A100	DGX-Station-A100	COCO	A100 SXM-80GB
3D-UNet	372 samples/sec	0.16 samples/sec/watt	8x A100	Gigabyte G482-Z54	BraTS 2019	A100 PCIe-40GB
	433 samples/sec	0.12 samples/sec/watt	8x A100	DGX A100	BraTS 2019	A100 SXM-80GB
	202 samples/sec	0.16 samples/sec/watt	4x A100	DGX-Station-A100	BraTS 2019	A100 SXM-80GB
RNN-T	82,540 samples/sec	36.23 samples/sec/watt	8x A100	Gigabyte G482-Z54	LibriSpeech	A100 PCIe-40GB
	93,803 samples/sec	26.39 samples/sec/watt	8x A100	DGX A100	LibriSpeech	A100 SXM-80GB
	47,255 samples/sec	36.16 samples/sec/watt	4x A100	DGX-Station-A100	LibriSpeech	A100 SXM-80GB
BERT	17,697 samples/sec	7.73 samples/sec/watt	8x A100	Gigabyte G482-Z54	SQuAD v1.1	A100 PCIe-40GB
	23,406 samples/sec	6.77 samples/sec/watt	8x A100	DGX A100	SQuAD v1.1	A100 SXM-80GB
	9,865 samples/sec	7.76 samples/sec/watt	4x A100	DGX-Station-A100	SQuAD v1.1	A100 SXM-80GB
DLRM	1,577,960 samples/sec	730.76 samples/sec/watt	8x A100	Gigabyte G482-Z54	Criteo 1TB Click Logs	A100 PCIe-40GB
	2,115,950 samples/sec	619.54 samples/sec/watt	8x A100	DGX A100	Criteo 1TB Click Logs	A100 SXM-80GB
	974,571 samples/sec	762.64 samples/sec/watt	4x A100	DGX-Station-A100	Criteo 1TB Click Logs	A100 SXM-80GB

Power Efficiency Server Scenario - Closed Division

Network	Throughput	Throughput per Watt	GPU	Server	Dataset	GPU Version
ResNet-50 v1.5	184,984 queries/sec	82.39 queries/sec/watt	8x A100	Gigabyte G482-Z54	ImageNet	A100 PCIe-40GB
	239,991 queries/sec	69.53 queries/sec/watt	8x A100	DGX A100	ImageNet	A100 SXM-80GB
	106,988 queries/sec	84.51 queries/sec/watt	4x A100	DGX-Station-A100	ImageNet	A100 SXM-80GB
SSD ResNet-34	5,702 queries/sec	2.52 queries/sec/watt	8x A100	Gigabyte G482-Z54	COCO	A100 PCIe-40GB
	6,301 queries/sec	1.82 queries/sec/watt	8x A100	DGX A100	COCO	A100 SXM-80GB
	3,081 queries/sec	2.43 queries/sec/watt	4x A100	DGX-Station-A100	COCO	A100 SXM-80GB
RNN-T	74,974 queries/sec	32.25 queries/sec/watt	8x A100	Gigabyte G482-Z54	LibriSpeech	A100 PCIe-40GB
	87,984 queries/sec	24.78 queries/sec/watt	8x A100	DGX A100	LibriSpeech	A100 SXM-80GB
	43,389 queries/sec	33.03 queries/sec/watt	4x A100	DGX-Station-A100	LibriSpeech	A100 SXM-80GB
BERT	17,499 queries/sec	7.58 queries/sec/watt	8x A100	Gigabyte G482-Z54	SQuAD v1.1	A100 PCIe-40GB
	21,492 queries/sec	6.03 queries/sec/watt	8x A100	DGX A100	SQuAD v1.1	A100 SXM-80GB
	10,203 queries/sec	7.84 queries/sec/watt	4x A100	DGX-Station-A100	SQuAD v1.1	A100 SXM-80GB
DLRM	2,001,940 queries/sec	575.72 queries/sec/watt	8x A100	DGX A100	Criteo 1TB Click Logs	A100 SXM-80GB
	890,334 queries/sec	663.62 queries/sec/watt	4x A100	DGX-Station-A100	Criteo 1TB Click Logs	A100 SXM-80GB

MLPerf™ v1.0 Inference Closed: ResNet-50 v1.5, SSD ResNet-34, RNN-T, BERT 99% of FP32 accuracy target, 3D U-Net, DLRM 99.9% of FP32 accuracy target: 1.0-25, 1.0-26, 1.0-29, 1.0-30, 1.0-32, 1.0-55, 1.0-57. MLPerf name and logo are trademarks. See https://mlcommons.org/ for more information.
BERT-Large sequence length = 384.
DLRM samples refers to 270 pairs/sample average
A10 and A30 results are preview
For MLPerf™ various scenario data, click here
For MLPerf™ latency constraints, click here

NVIDIA TRITON Inference Server Delivered Comparable Performance to Custom Harness in MLPerf v1.0

NVIDIA landed top performance spots on all MLPerf™ Inference 1.0 tests, the AI-industry’s leading benchmark competition. For inference submissions, we have typically used a custom A100 inference serving harness. This custom harness has been designed and optimized specifically for providing the highest possible inference performance for MLPerf™ workloads, which require running inference on bare metal.

MLPerf™ v1.0 A100 Inference Closed: ResNet-50 v1.5, SSD ResNet-34, RNN-T, BERT 99% of FP32 accuracy target, 3D U-Net, DLRM 99% of FP32 accuracy target: 1.0-30, 1.0-31. MLPerf name and logo are trademarks. See www.mlcommons.org for more information.

The chart compares the performance of Triton to the custom MLPerf™ serving harness across five different TensorRT networks on A100 SXM-80GB on bare metal. The results show that Triton is highly efficient and delivers nearly equal or identical performance to the highly optimized MLPerf™ harness.

NVIDIA Client Batch Size=1 Performance with Triton Inference Server

Network	Accelerator	Training Framework	Framework Backend	Precision	Model Instances on Triton	Client Batch Size	Dynamic Batch Size (Triton)	Number of Concurrent Client Requests	Latency (ms)	Throughput	Sequence/Input Length	Triton Container Version
ResNet-50 V1.5 Inference	A100-PCIE-40GB	PyTorch	TensorRT	FP16	2	1	64	256	61.02	4,197 inf/sec	-	20.07-py3
ResNet-50 V1.5 Inference	A100-SXM4-40GB	PyTorch	TensorRT	TF32	2	1	64	256	48.35	5,294 inf/sec	-	21.03-py3
ResNet-50 V1.5 Inference	NVIDIA T4	PyTorch	TensorRT	FP16	1	1	64	256	257.91	992 inf/sec	-	20.07-py3
ResNet-50 V1.5 Inference	V100 SXM2-32GB	PyTorch	TensorRT	FP32	4	1	64	384	215.79	1,781 inf/sec	-	21.03-py3
BERT Large Inference	A100-PCIE-40GB	TensorFlow	TensorRT	FP16	1	1	8	16	17.48	915 inf/sec	384	20.09-py3
BERT Large Inference	A100-SXM4-40GB	TensorFlow	TensorRT	INT8	2	1	8	64	56.34	1,136 inf/sec	384	20.09-py3
BERT Large Inference	NVIDIA T4	TensorFlow	TensorRT	FP16	1	1	8	16	81.14	197 inf/sec	384	20.09-py3
DLRM Inference	A100-PCIE-40GB	PyTorch	Torchscript	Mixed	2	1	65,536	24	2.52	9,521 inf/sec	-	21.05-py3
DLRM Inference	A100-SXM4-40GB	PyTorch	Torchscript	FP16	2	1	65,536	30	2.71	11,076 inf/sec	-	21.03-py3
DLRM Inference	V100-SXM2-32GB	PyTorch	Torchscript	Mixed	2	1	65,536	30	4.24	7,068 inf/sec	-	21.05-py3

Inference Performance of NVIDIA A100, A40, A30, A10, V100 and T4

Benchmarks are reproducible by following links to NGC scripts

Inference Natural Langugage Processing

BERT Inference Throughput

NVIDIA A100 BERT Inference Benchmarks

Network	Network Type	Batch Size	Throughput	Efficiency	Latency (ms)	GPU	Server	Container	Precision	Dataset	Framework	GPU Version
BERT-Large with Sparsity	Attention	94	6,188 sequences/sec	-	-	1x A100	DGX A100	-	INT8	SQuaD v1.1	-	A100 SXM4-40GB

A100 with 7 MIG instances of 1g.5gb | Sequence length=128 | Efficiency based on board power
Containers with a hyphen indicates a pre-release container

Inference Image Classification on CNNs with TensorRT

ResNet-50 v1.5 Throughput

ResNet-50 v1.5 Power Efficiency

A100 1/7 MIG Inference Performance

Network	Batch Size	1/7 MIG Throughput	Latency (ms)	GPU	Server	Container	Precision	Dataset	Framework	GPU Version
ResNet-50	1	1,445 images/sec	0.69	1x A100	DGX A100	21.05-py3	INT8	Synthetic	TensorRT 7.2	A100 SXM-80GB
	2	2,250 images/sec	0.89	1x A100	DGX A100	21.05-py3	INT8	Synthetic	TensorRT 7.2	A100 SXM-80GB
	8	3,508 images/sec	2.28	1x A100	DGX A100	21.05-py3	INT8	Synthetic	TensorRT 7.2	A100 SXM-80GB
	28	4,035 images/sec	6.94	1x A100	DGX A100	21.05-py3	INT8	Synthetic	TensorRT 7.2	A100 SXM-80GB
	128	4,492 images/sec	28.5	1x A100	DGX A100	21.05-py3	INT8	Synthetic	TensorRT 7.2	A100 SXM-80GB
ResNet-50v1.5	1	1,406 images/sec	0.71	1x A100	DGX A100	21.05-py3	INT8	Synthetic	TensorRT 7.2	A100 SXM-80GB
	2	2,170 images/sec	0.92	1x A100	DGX A100	21.05-py3	INT8	Synthetic	TensorRT 7.2	A100 SXM-80GB
	8	3,375 images/sec	2.37	1x A100	DGX A100	21.05-py3	INT8	Synthetic	TensorRT 7.2	A100 SXM-80GB
	27	3,956 images/sec	6.83	1x A100	DGX A100	21.05-py3	INT8	Synthetic	TensorRT 7.2	A100 SXM-80GB
	128	4,333 images/sec	29.54	1x A100	DGX A100	21.05-py3	INT8	Synthetic	TensorRT 7.2	A100 SXM-80GB
BERT-BASE	1	800 sequences/sec	1.25	1x A100	DGX A100	21.05-py3	INT8	Synthetic	TensorRT 7.2	A100 SXM-80GB
	2	1,192 sequences/sec	1.68	1x A100	DGX A100	21.05-py3	INT8	Synthetic	TensorRT 7.2	A100 SXM-80GB
	4	474 sequences/sec	8.55	1x A100	DGX A100	20.11-py3	INT8	Synthetic	TensorRT 7.2	A100 SXM-80GB
	8	1,706 sequences/sec	4.69	1x A100	DGX A100	21.05-py3	INT8	Synthetic	TensorRT 7.2	A100 SXM-80GB
	128	2,175 sequences/sec	58.85	1x A100	DGX A100	21.05-py3	INT8	Synthetic	TensorRT 7.2	A100 SXM-80GB
BERT-LARGE	1	268 sequences/sec	3.73	1x A100	DGX A100	21.05-py3	INT8	Synthetic	TensorRT 7.2	A100 SXM-80GB
	2	386 sequences/sec	5.18	1x A100	DGX A100	21.05-py3	INT8	Synthetic	TensorRT 7.2	A100 SXM-80GB
	5	517 sequences/sec	9.84	1x A100	DGX A100	20.11-py3	INT8	Synthetic	TensorRT 7.2	A100 SXM-80GB
	8	558 sequences/sec	14.33	1x A100	DGX A100	21.05-py3	INT8	Synthetic	TensorRT 7.2	A100 SXM-80GB
	128	677 sequences/sec	189.2	1x A100	DGX A100	21.05-py3	INT8	Synthetic	TensorRT 7.2	A100 SXM-80GB

Containers with a hyphen indicates a pre-release container | Servers with a hyphen indicates a pre-production server
BERT-Large: Sequence Length = 128

A100 7 MIG Inference Performance

Network	Batch Size	7 MIG Throughput	Latency (ms)	GPU	Server	Container	Precision	Dataset	Framework	GPU Version
ResNet-50	1	10,017 images/sec	0.7	1x A100	DGX A100	21.05-py3	INT8	Synthetic	TensorRT 7.2	A100 SXM-80GB
	2	15,345 images/sec	0.91	1x A100	DGX A100	21.05-py3	INT8	Synthetic	TensorRT 7.2	A100 SXM-80GB
	8	24,035 images/sec	2.33	1x A100	DGX A100	21.05-py3	INT8	Synthetic	TensorRT 7.2	A100 SXM-80GB
	27	28,369 images/sec	6.67	1x A100	DGX A100	21.05-py3	INT8	Synthetic	TensorRT 7.2	A100 SXM-80GB
	128	31,301 images/sec	28.63	1x A100	DGX A100	21.05-py3	INT8	Synthetic	TensorRT 7.2	A100 SXM-80GB
ResNet-50v1.5	1	9,875 images/sec	0.71	1x A100	DGX A100	21.05-py3	INT8	Synthetic	TensorRT 7.2	A100 SXM-80GB
	2	15,159 images/sec	0.94	1x A100	DGX A100	21.05-py3	INT8	Synthetic	TensorRT 7.2	A100 SXM-80GB
	8	23,306 images/sec	2.4	1x A100	DGX A100	21.05-py3	INT8	Synthetic	TensorRT 7.2	A100 SXM-80GB
	27	27,498 images/sec	6.87	1x A100	DGX A100	21.05-py3	INT8	Synthetic	TensorRT 7.2	A100 SXM-80GB
	128	30,179 images/sec	29.73	1x A100	DGX A100	21.05-py3	INT8	Synthetic	TensorRT 7.2	A100 SXM-80GB
BERT-BASE	1	5,595 sequences/sec	1.27	1x A100	DGX A100	21.05-py3	INT8	Synthetic	TensorRT 7.2	A100 SXM-80GB
	2	8,298 sequences/sec	1.69	1x A100	DGX A100	21.05-py3	INT8	Synthetic	TensorRT 7.2	A100 SXM-80GB
	4	3,284 sequences/sec	8.55	1x A100	DGX A100	20.11-py3	INT8	Synthetic	TensorRT 7.2	A100 SXM-80GB
	8	11,878 sequences/sec	4.73	1x A100	DGX A100	21.05-py3	INT8	Synthetic	TensorRT 7.2	A100 SXM-80GB
	128	14,393 sequences/sec	62.37	1x A100	DGX A100	21.05-py3	INT8	Synthetic	TensorRT 7.2	A100 SXM-80GB
BERT-LARGE	1	1,880 sequences/sec	3.76	1x A100	DGX A100	21.05-py3	INT8	Synthetic	TensorRT 7.2	A100 SXM-80GB
	2	2,694 sequences/sec	5.21	1x A100	DGX A100	21.05-py3	INT8	Synthetic	TensorRT 7.2	A100 SXM-80GB
	5	3,575 sequences/sec	9.84	1x A100	DGX A100	20.11-py3	INT8	Synthetic	TensorRT 7.2	A100 SXM-80GB
	8	3,838 sequences/sec	14.63	1x A100	DGX A100	21.05-py3	INT8	Synthetic	TensorRT 7.2	A100 SXM-80GB
	128	4,467 sequences/sec	200.76	1x A100	DGX A100	21.05-py3	INT8	Synthetic	TensorRT 7.2	A100 SXM4-40GB

Containers with a hyphen indicates a pre-release container | Servers with a hyphen indicates a pre-production server
BERT-Large: Sequence Length = 128

A100 Full Chip Inference Performance

Network	Batch Size	Full Chip Throughput	Latency (ms)	GPU	Server	Container	Precision	Dataset	Framework	GPU Version
ResNet-50	2	3,995 images/sec	0.5	1x A100	DGX A100	21.05-py3	INT8	Synthetic	TensorRT 7.2	A100-SXM4-40GB
	8	11,418 images/sec	0.7	1x A100	DGX A100	21.05-py3	INT8	Synthetic	TensorRT 7.2	A100-SXM4-40GB
	128	29,109 images/sec	4.4	1x A100	DGX A100	21.05-py3	INT8	Synthetic	TensorRT 7.2	A100-SXM-80GB
	211	31,257 images/sec	6.75	1x A100	DGX A100	21.05-py3	INT8	Synthetic	TensorRT 7.2	A100-SXM-80GB
ResNet-50v1.5	2	4,020 images/sec	0.5	1x A100	DGX A100	21.05-py3	INT8	Synthetic	TensorRT 7.2	A100-SXM-80GB
	8	11,095 images/sec	0.72	1x A100	DGX A100	21.05-py3	INT8	Synthetic	TensorRT 7.2	A100-SXM-80GB
	128	28,222 images/sec	4.54	1x A100	DGX A100	21.05-py3	INT8	Synthetic	TensorRT 7.2	A100-SXM-80GB
	208	30,039 images/sec	6.92	1x A100	DGX A100	21.05-py3	INT8	Synthetic	TensorRT 7.2	A100-SXM-80GB
ResNext101	32	7,674 samples/sec	4.17	1x A100	-	-	INT8	Synthetic	TensorRT 7.2	A100-SXM4-40GB
EfficientNet-B0	128	22,346 images/sec	5.73	1x A100	-	-	INT8	Synthetic	TensorRT 7.2	A100-SXM4-40GB
BERT-BASE	2	2,599 sequences/sec	0.77	1x A100	DGX A100	21.05-py3	INT8	Sample Text	TensorRT 7.2	A100-SXM4-40GB
	8	6,889 sequences/sec	1.16	1x A100	DGX A100	21.05-py3	INT8	Sample Text	TensorRT 7.2	A100-SXM4-40GB
	128	13,661 sequences/sec	9.37	1x A100	DGX A100	21.05-py3	INT8	Sample Text	TensorRT 7.2	A100-SXM4-40GB
	256	14,490 sequences/sec	17.67	1x A100	DGX A100	-	INT8	Sample Text	TensorRT 7.2	A100-SXM4-40GB
BERT-LARGE	2	1,093 sequences/sec	1.83	1x A100	DGX A100	21.05-py3	INT8	Sample Text	TensorRT 7.2	A100-SXM4-40GB
	8	2,311 sequences/sec	3.46	1x A100	DGX A100	21.05-py3	INT8	Sample Text	TensorRT 7.2	A100-SXM-80GB
	128	4,515 sequences/sec	28.35	1x A100	DGX A100	21.05-py3	INT8	Sample Text	TensorRT 7.2	A100-SXM-80GB
	256	4,679 sequences/sec	54.71	1x A100	-	-	INT8	Sample Text	TensorRT 7.2	A100-SXM4-80GB

Containers with a hyphen indicates a pre-release container | Servers with a hyphen indicates a pre-production server
BERT-Large: Sequence Length = 128
For BS=1 inference refer to the Triton Inference Server section

A40 Inference Performance

Network	Batch Size	Throughput	Efficiency	Latency (ms)	GPU	Server	Container	Precision	Dataset	Framework	GPU Version
ResNet-50	2	4,523 images/sec	24 images/sec/watt	0.44	1x A40	GIGABYTE G482-Z52-00	21.05-py3	INT8	Synthetic	TensorRT 7.2	A40
	8	9,279 images/sec	40 images/sec/watt	0.86	1x A40	GIGABYTE G482-Z52-00	21.05-py3	INT8	Synthetic	TensorRT 7.2	A40
	109	16,738 images/sec	- images/sec/watt	6.51	1x A40	GIGABYTE G482-Z52-00	21.05-py3	INT8	Synthetic	TensorRT 7.2	A40
	128	16,286 images/sec	54 images/sec/watt	7.86	1x A40	GIGABYTE G482-Z52-00	21.05-py3	INT8	Synthetic	TensorRT 7.2	A40
ResNet-50v1.5	2	4,451 images/sec	24 images/sec/watt	0.45	1x A40	GIGABYTE G482-Z52-00	21.05-py3	INT8	Synthetic	TensorRT 7.2	A40
	8	8,903 images/sec	38 images/sec/watt	0.9	1x A40	GIGABYTE G482-Z52-00	21.05-py3	INT8	Synthetic	TensorRT 7.2	A40
	109	15,892 images/sec	- images/sec/watt	6.86	1x A40	GIGABYTE G482-Z52-00	21.05-py3	INT8	Synthetic	TensorRT 7.2	A40
	128	15,510 images/sec	52 images/sec/watt	8.25	1x A40	GIGABYTE G482-Z52-00	21.05-py3	INT8	Synthetic	TensorRT 7.2	A40
BERT-BASE	2	2,351 sequences/sec	13 sequences/sec/watt	0.85	1x A40	GIGABYTE G482-Z52-00	21.05-py3	INT8	Sample Text	TensorRT 7.2	A40
	8	4,494 sequences/sec	19 sequences/sec/watt	1.78	1x A40	GIGABYTE G482-Z52-00	21.05-py3	INT8	Sample Text	TensorRT 7.2	A40
	128	7,180 sequences/sec	27 sequences/sec/watt	17.83	1x A40	GIGABYTE G482-Z52-00	21.05-py3	INT8	Sample Text	TensorRT 7.2	A40
BERT-LARGE	2	893 sequences/sec	4 sequences/sec/watt	2.24	1x A40	GIGABYTE G482-Z52-00	21.05-py3	INT8	Sample Text	TensorRT 7.2	A40
	8	1,666 sequences/sec	6 sequences/sec/watt	4.8	1x A40	GIGABYTE G482-Z52-00	21.05-py3	INT8	Sample Text	TensorRT 7.2	A40
	128	2,216 sequences/sec	9 sequences/sec/watt	57.76	1x A40	GIGABYTE G482-Z52-00	21.05-py3	INT8	Sample Text	TensorRT 7.2	A40

NGC: TensorRT Container
Sequence length=128 for BERT-BASE and BERT-LARGE | Efficiency based on board power
Containers with a hyphen indicates a pre-release container

A30 Inference Performance

Network	Batch Size	Throughput	Efficiency	Latency (ms)	GPU	Server	Container	Precision	Dataset	Framework	GPU Version
ResNet-50	2	3,487 images/sec	38 images/sec/watt	0.57	1x A30	GIGABYTE G482-Z52-SW-QZ-001	21.03-py3	INT8	Synthetic	TensorRT 7.2	A30
	8	8,497 images/sec	70 images/sec/watt	0.94	1x A30	GIGABYTE G482-Z52-SW-QZ-001	21.03-py3	INT8	Synthetic	TensorRT 7.2	A30
	100	14,963 images/sec	- images/sec/watt	6.68	1x A30	GIGABYTE G482-Z52-SW-QZ-001	21.05-py3	INT8	Synthetic	TensorRT 7.2	A30
	128	15,246 images/sec	93 images/sec/watt	8.4	1x A30	GIGABYTE G482-Z52-SW-QZ-001	21.05-py3	INT8	Synthetic	TensorRT 7.2	A30
ResNet-50v1.5	2	3,498 images/sec	37 images/sec/watt	0.57	1x A30	GIGABYTE G482-Z52-SW-QZ-001	21.03-py3	INT8	Synthetic	TensorRT 7.2	A30
	8	8,330 images/sec	68 images/sec/watt	0.96	1x A30	GIGABYTE G482-Z52-SW-QZ-001	21.03-py3	INT8	Synthetic	TensorRT 7.2	A30
	95	14,191 images/sec	- images/sec/watt	6.69	1x A30	GIGABYTE G482-Z52-SW-QZ-001	21.05-py3	INT8	Synthetic	TensorRT 7.2	A30
	128	14,568 images/sec	88 images/sec/watt	8.79	1x A30	GIGABYTE G482-Z52-SW-QZ-001	21.05-py3	INT8	Synthetic	TensorRT 7.2	A30
BERT-BASE	2	2,132 sequences/sec	22 sequences/sec/watt	0.94	1x A30	GIGABYTE G482-Z52-SW-QZ-001	21.05-py3	INT8	Sample Text	TensorRT 7.2	A30
	8	4,515 sequences/sec	34 sequences/sec/watt	1.77	1x A30	GIGABYTE G482-Z52-SW-QZ-001	21.05-py3	INT8	Sample Text	TensorRT 7.2	A30
	128	6,940 sequences/sec	48 sequences/sec/watt	18.44	1x A30	GIGABYTE G482-Z52-SW-QZ-001	21.05-py3	INT8	Sample Text	TensorRT 7.2	A30
BERT-LARGE	2	837 sequences/sec	7 sequences/sec/watt	2.39	1x A30	GIGABYTE G482-Z52-SW-QZ-001	21.05-py3	INT8	Sample Text	TensorRT 7.2	A30
	8	1,441 sequences/sec	12 sequences/sec/watt	5.55	1x A30	GIGABYTE G482-Z52-SW-QZ-001	21.05-py3	INT8	Sample Text	TensorRT 7.2	A30
	128	2,211 sequences/sec	15 sequences/sec/watt	57.9	1x A30	GIGABYTE G482-Z52-SW-QZ-001	21.05-py3	INT8	Sample Text	TensorRT 7.2	A30

NGC: TensorRT Container
Sequence length=128 for BERT-BASE and BERT-LARGE | Efficiency based on board power
Containers with a hyphen indicates a pre-release container

A10 Inference Performance

Network	Batch Size	Throughput	Efficiency	Latency (ms)	GPU	Server	Container	Precision	Dataset	Framework	GPU Version
ResNet-50	2	4,282 images/sec	29 images/sec/watt	0.47	1x A10	GIGABYTE G482-Z52-00	21.05-py3	INT8	Synthetic	TensorRT 7.2	A10
	8	7,778 images/sec	52 images/sec/watt	1.03	1x A10	GIGABYTE G482-Z52-00	21.05-py3	INT8	Synthetic	TensorRT 7.2	A10
	75	10,846 images/sec	- images/sec/watt	6.82	1x A10	GIGABYTE G482-Z52-00	21.05-py3	INT8	Synthetic	TensorRT 7.2	A10
	128	11,520 images/sec	77 images/sec/watt	11.11	1x A10	GIGABYTE G482-Z52-00	21.05-py3	INT8	Synthetic	TensorRT 7.2	A10
ResNet-50v1.5	2	4,187 images/sec	28 images/sec/watt	0.48	1x A10	GIGABYTE G482-Z52-00	21.05-py3	INT8	Synthetic	TensorRT 7.2	A10
	8	7,434 images/sec	50 images/sec/watt	1.08	1x A10	GIGABYTE G482-Z52-00	21.05-py3	INT8	Synthetic	TensorRT 7.2	A10
	70	10,785 images/sec	- images/sec/watt	6.49	1x A10	GIGABYTE G482-Z52-00	21.05-py3	INT8	Synthetic	TensorRT 7.2	A10
	128	10,928 images/sec	73 images/sec/watt	11.71	1x A10	GIGABYTE G482-Z52-00	21.05-py3	INT8	Synthetic	TensorRT 7.2	A10
BERT-BASE	2	2,095 sequences/sec	17 sequences/sec/watt	0.95	1x A10	GIGABYTE G482-Z52-00	21.05-py3	INT8	Sample Text	TensorRT 7.2	A10
	8	3,598 sequences/sec	26 sequences/sec/watt	2.22	1x A10	GIGABYTE G482-Z52-00	21.05-py3	INT8	Sample Text	TensorRT 7.2	A10
	128	4,764 sequences/sec	35 sequences/sec/watt	26.87	1x A10	GIGABYTE G482-Z52-00	21.05-py3	INT8	Sample Text	TensorRT 7.2	A10
BERT-LARGE	2	779 sequences/sec	6 sequences/sec/watt	2.57	1x A10	GIGABYTE G482-Z52-00	21.05-py3	INT8	Sample Text	TensorRT 7.2	A10
	8	1,288 sequences/sec	10 sequences/sec/watt	6.21	1x A10	GIGABYTE G482-Z52-00	21.05-py3	INT8	Sample Text	TensorRT 7.2	A10
	128	1,368 sequences/sec	10 sequences/sec/watt	93.57	1x A10	Supermicro SYS-1029GQ-TRT	20.11-py3	INT8	Sample Text	TensorRT 7.2	A10

NGC: TensorRT Container
Sequence length=128 for BERT-BASE and BERT-LARGE | Efficiency based on board power
Containers with a hyphen indicates a pre-release container

V100 Inference Performance

Network	Batch Size	Throughput	Efficiency	Latency (ms)	GPU	Server	Container	Precision	Dataset	Framework	GPU Version
ResNet-50	2	1,995 images/sec	11 images/sec/watt	1	1x V100	DGX-2	21.05-py3	Mixed	Synthetic	TensorRT 7.2.3	V100-SXM3-32GB
	8	4,305 images/sec	17 images/sec/watt	1.86	1x V100	DGX-2	21.05-py3	Mixed	Synthetic	TensorRT 7.2.3	V100-SXM3-32GB
	52	7,909 images/sec	- images/sec/watt	6.57	1x V100	DGX-2	21.05-py3	INT8	Synthetic	TensorRT 7.2.3	V100-SXM3-32GB
	128	8,202 images/sec	24 images/sec/watt	15.61	1x V100	DGX-2	21.05-py3	Mixed	Synthetic	TensorRT 7.2.3	V100-SXM3-32GB
ResNet-50v1.5	2	2,005 images/sec	11 images/sec/watt	1	1x V100	DGX-2	21.05-py3	Mixed	Synthetic	TensorRT 7.2	V100-SXM3-32GB
	8	4,221 images/sec	16 images/sec/watt	1.9	1x V100	DGX-2	21.05-py3	INT8	Synthetic	TensorRT 7.2	V100-SXM3-32GB
	52	7,545 images/sec	- images/sec/watt	6.89	1x V100	DGX-2	21.05-py3	Mixed	Synthetic	TensorRT 7.2	V100-SXM3-32GB
	128	7,857 images/sec	23 images/sec/watt	16.29	1x V100	DGX-2	21.05-py3	INT8	Synthetic	TensorRT 7.2	V100-SXM3-32GB
BERT-BASE	2	1,219 sequences/sec	6 sequences/sec/watt	1.64	1x V100	DGX-2	21.05-py3	Mixed	Sample Text	TensorRT 7.2	V100-SXM3-32GB
	8	2,300 sequences/sec	8 sequences/sec/watt	3.48	1x V100	DGX-2	21.05-py3	Mixed	Sample Text	TensorRT 7.2	V100-SXM3-32GB
	128	3,206 sequences/sec	10 sequences/sec/watt	39.93	1x V100	DGX-2	21.05-py3	Mixed	Sample Text	TensorRT 7.2	V100-SXM3-32GB
BERT-LARGE	2	530 sequences/sec	2 sequences/sec/watt	3.78	1x V100	DGX-2	21.05-py3	Mixed	Sample Text	TensorRT 7.2	V100-SXM3-32GB
	8	793 sequences/sec	3 sequences/sec/watt	10.09	1x V100	DGX-2	21.05-py3	INT8	Sample Text	TensorRT 7.2	V100-SXM3-32GB
	128	978 sequences/sec	3 sequences/sec/watt	130.92	1x V100	DGX-2	20.11-py3	INT8	Sample Text	TensorRT 7.2	V100-SXM3-32GB

NGC: TensorRT Container
Sequence length=128 for BERT-BASE and BERT-LARGE | Efficiency based on board power
Containers with a hyphen indicates a pre-release container
For BS=1 inference refer to the Triton Inference Server section

T4 Inference Performance

Network	Batch Size	Throughput	Efficiency	Latency (ms)	GPU	Server	Container	Precision	Dataset	Framework	GPU Version
ResNet-50	2	2,105 images/sec	30 images/sec/watt	0.95	1x T4	Supermicro SYS-1029GQ-TRT	21.03-py3	INT8	Synthetic	TensorRT 7.2	NVIDIA T4
	8	3,846 images/sec	55 images/sec/watt	2.08	1x T4	Supermicro SYS-4029GP-TRT	21.05-py3	INT8	Synthetic	TensorRT 7.2	NVIDIA T4
	32	4,798 images/sec	- images/sec/watt	6.67	1x T4	Supermicro SYS-4029GP-TRT	21.05-py3	INT8	Synthetic	TensorRT 7.2	NVIDIA T4
	128	5,358 images/sec	77 images/sec/watt	23.89	1x T4	Supermicro SYS-4029GP-TRT	21.05-py3	INT8	Synthetic	TensorRT 7.2	NVIDIA T4
ResNet-50v1.5	2	2,092 images/sec	30 images/sec/watt	0.96	1x T4	Supermicro SYS-1029GQ-TRT	21.03-py3	INT8	Synthetic	TensorRT 7.2	NVIDIA T4
	8	3,698 images/sec	53 images/sec/watt	2.16	1x T4	Supermicro SYS-4029GP-TRT	21.05-py3	INT8	Synthetic	TensorRT 7.2	NVIDIA T4
	29	4,540 images/sec	- images/sec/watt	6.39	1x T4	Supermicro SYS-4029GP-TRT	21.05-py3	INT8	Synthetic	TensorRT 7.2	NVIDIA T4
	128	4,960 images/sec	71 images/sec/watt	25.81	1x T4	Supermicro SYS-4029GP-TRT	21.05-py3	INT8	Synthetic	TensorRT 7.2	NVIDIA T4
BERT-BASE	2	1,089 sequences/sec	18 sequences/sec/watt	1.84	1x T4	Supermicro SYS-4029GP-TRT	21.05-py3	INT8	Sample Text	TensorRT 7.2	NVIDIA T4
	8	1,677 sequences/sec	25 sequences/sec/watt	4.77	1x T4	Supermicro SYS-4029GP-TRT	21.05-py3	INT8	Sample Text	TensorRT 7.2	NVIDIA T4
	128	1,818 sequences/sec	28 sequences/sec/watt	70.4	1x T4	Supermicro SYS-1029GQ-TRT	20.11-py3	INT8	Sample Text	TensorRT 7.2	NVIDIA T4
BERT-LARGE	2	386 sequences/sec	6 sequences/sec/watt	5.18	1x T4	Supermicro SYS-4029GP-TRT	21.05-py3	INT8	Sample Text	TensorRT 7.2	NVIDIA T4
	8	551 sequences/sec	9 sequences/sec/watt	14.52	1x T4	Supermicro SYS-4029GP-TRT	21.05-py3	INT8	Sample Text	TensorRT 7.2	NVIDIA T4
	128	561 sequences/sec	8 sequences/sec/watt	227.98	1x T4	Supermicro SYS-1029GQ-TRT	20.11-py3	INT8	Sample Text	TensorRT 7.2	NVIDIA T4

NVIDIA Jarvis is an application framework for multimodal conversational AI services that delivers real-time performance on GPUs. Jarvis 1.0 Beta includes fully optimized pipelines for Automatic Speech Recognition (ASR), Natural Language Processing (NLP), Text to Speech (TTS) that can be used for deploying real-time conversational AI apps such as transcription, virtual assistants and chatbots. Please visit Jarvis – Getting Started to download and get started with Jarvis.

Jarvis Benchmarks

Automatic Speech Recognition

A100 Best Streaming Throughput Mode (800 ms chunk)

Acoustic model	# of streams	Avg Latency (ms)	Throughput (RTFX)	GPU Version
Quartznet	1	14.4	1	A100 SXM4-40GB
Quartznet	256	254.3	64	A100 SXM4-40GB
Quartznet	512	351.2	506	A100 SXM4-40GB
Quartznet	1024	630.8	1005	A100 SXM4-40GB
Jasper	1	17.6	1	A100 SXM4-40GB
Jasper	256	244.9	254	A100 SXM4-40GB
Jasper	512	381	507	A100 SXM4-40GB
Jasper	1024	749.3	1,004	A100 SXM4-40GB

A100 Best Streaming Latency Mode (100 ms chunk)

Acoustic model	# of streams	Avg Latency (ms)	Throughput (RTFX)	GPU Version
Quartznet	1	9.6	1	A100 SXM4-40GB
Quartznet	16	25.9	16	A100 SXM4-40GB
Quartznet	128	132.4	128	A100 SXM4-40GB
Jasper	1	13.4	1	A100 SXM4-40GB
Jasper	16	26.3	16	A100 SXM4-40GB
Jasper	128	258.9	128	A100 SXM4-40GB

A100 Offline Mode (3200 ms chunk)

Acoustic model	# of streams	Avg Latency (ms)	Throughput (RTFX)	GPU Version
Quartznet	1	28.1	1	A100 SXM4-40GB
Quartznet	512	566.5	505	A100 SXM4-40GB
Quartznet	1,024	899.3	1,000	A100 SXM4-40GB
Quartznet	1,512	1,303.8	1,460	A100 SXM4-40GB
Jasper	1	31	1	A100 SXM4-40GB
Jasper	512	667.5	504	A100 SXM4-40GB
Jasper	1,024	1,089	997	A100 SXM4-40GB
Jasper	1,512	1,753.8	1,449	A100 SXM4-40GB

V100 Best Streaming Throughput Mode (800 ms chunk)

Acoustic model	# of streams	Avg Latency (ms)	Throughput (RTFX)	GPU Version
Quartznet	1	14.4	1	V100 SXM2-16GB
Quartznet	256	222.2	254	V100 SXM2-16GB
Quartznet	512	385.2	505	V100 SXM2-16GB
Quartznet	768	574.5	752	V100 SXM2-16GB
Jasper	1	26.8	1	V100 SXM2-16GB
Jasper	128	239.4	127	V100 SXM2-16GB
Jasper	256	416	253	V100 SXM2-16GB
Jasper	512	969.7	500	V100 SXM2-16GB

V100 Best Streaming Latency Mode (100 ms chunk)

Acoustic model	# of streams	Avg Latency (ms)	Throughput (RTFX)	GPU Version
Quartznet	1	8.8	1	V100 SXM2-16GB
Quartznet	16	22.4	16	V100 SXM2-16GB
Quartznet	128	114.7	127	V100 SXM2-16GB
Jasper	1	21.5	1	V100 SXM2-16GB
Jasper	16	36.9	16	V100 SXM2-16GB
Jasper	64	406.4	64	V100 SXM2-16GB
Jasper	512	969.7	500	V100 SXM2-16GB

V100 Offline Mode (3200 ms chunk)

Acoustic model	# of streams	Avg Latency (ms)	Throughput (RTFX)	GPU Version
Quartznet	1	32.933	1	V100 SXM2-16GB
Quartznet	256	461.44	253	V100 SXM2-16GB
Quartznet	512	784.73	502	V100 SXM2-16GB
Quartznet	768	1,121.6	747	V100 SXM2-16GB
Quartznet	1,024	1,551.5	986	V100 SXM2-16GB
Jasper	1	48.351	1	V100 SXM2-16GB
Jasper	256	734.99	252	V100 SXM2-16GB
Jasper	512	1,423.3	498	V100 SXM2-16GB
Jasper	768	2,190.2	730	V100 SXM2-16GB

T4 Best Streaming Throughput Mode (800 ms chunk)

Acoustic model	# of streams	Avg Latency (ms)	Throughput (RTFX)	GPU Version
Quartznet	1	33.183	1	NVIDIA T4
Quartznet	64	162.63	64	NVIDIA T4
Quartznet	128	263.6	127	NVIDIA T4
Quartznet	256	449.28	253	NVIDIA T4
Quartznet	384	732.75	376	NVIDIA T4
Jasper	1	72.377	1	NVIDIA T4
Jasper	64	259.64	64	NVIDIA T4
Jasper	128	450.81	127	NVIDIA T4
Jasper	256	1,200.8	249	NVIDIA T4

T4 Best Streaming Latency Mode (100 ms chunk)

Acoustic model	# of streams	Avg Latency (ms)	Throughput (RTFX)	GPU Version
Quartznet	1	19.2	1	NVIDIA T4
Quartznet	16	56.4	16	NVIDIA T4
Quartznet	64	242.4	64	NVIDIA T4
Jasper	1	46.9	1	NVIDIA T4
Jasper	8	51.1	8	NVIDIA T4
Jasper	16	84.4	16	NVIDIA T4

T4 Offline Mode (3200 ms chunk)

Acoustic model	# of streams	Latency (ms) (avg)	Throughput (RTFX)	GPU Version
Quartznet	1	157.62	1	NVIDIA T4
Quartznet	256	906.17	251	NVIDIA T4
Quartznet	512	1,515.2	495	NVIDIA T4
Jasper	1	96.201	1	NVIDIA T4
Jasper	256	1,758.4	247	NVIDIA T4

ASR Throughput (RTFX) - Number of seconds of audio processed per second | Audio Chunk Size – Server side configuration indicating the amount of new data to be considered by the acoustic model | ASR Dataset: Librispeech | The latency numbers were measured using the streaming recognition mode, with the BERT-Base punctuation model enabled, a 4-gram language model, a decoder beam width of 128 and timestamps enabled. The client and the server were using audio chunks of the same duration (100ms, 800ms, 3200ms depending on the server configuration). The Jarvis streaming client jarvis_streaming_asr_client, provided in the Jarvis client image was used with the --simulate_realtime flag to simulate transcription from a microphone, where each stream was doing 5 iterations over a sample audio file from the Librispeech dataset (1272-135031-0000.wav) | Jarvis version: v1.0.0-b1 | Hardware: NVIDIA DGX A100 (1x A100 SXM4-40GB), NVIDIA DGX-1 (1x V100-SXM2-16GB), NVIDIA T4 with 2x Intel(R) Xeon(R) Gold 6240 CPU @ 2.60GHz

Natural Language Processing

A100 Benchmarks

Task	# of streams	Avg Latency (ms)	Throughput (seq/sec)	GPU Version
NER	1	3.19	311	A100 SXM4-40GB
NER	256	95.5	2549	A100 SXM4-40GB
Q&A	1	4.95	201	A100 SXM4-40GB
Q&A	128	279	453	A100 SXM4-40GB

V100 Benchmarks

Task	# of streams	Avg Latency (ms)	Throughput (seq/sec)	GPU Version
NER	1	4.87	204	V100 SXM2-16GB
NER	256	135	1,797	V100 SXM2-16GB
Q&A	1	7.47	134	V100 SXM2-16GB
Q&A	128	521	244	V100 SXM2-16GB

T4 Benchmarks

Task	# of streams	Avg Latency (ms)	Throughput (seq/sec)	GPU Version
NER	1	9.31	107	NVIDIA T4
NER	256	255	960	NVIDIA T4
Q&A	1	11.5	87	NVIDIA T4
Q&A	128	571	223	NVIDIA T4

Named Entity Recogniton (NER): 128 seq len, BERT-base | Question Answering (QA): 384 seq len, BERT-large | NLP Throughput (seq/s) - Number of sequences processed per second | Performance of the Jarvis named entity recognition (NER) service (using a BERT-base model, sequence length of 128) and the Jarvis question answering (QA) service (using a BERT-large model, sequence length of 384) was measured in Jarvis. Batch size 1 latency and maximum throughput were measured. Jarvis version: v1.0.0-b1 | Hardware: NVIDIA DGX A100 (1x A100 SXM4-40GB), NVIDIA DGX-1 (1x V100-SXM2-16GB), NVIDIA T4 with 2x Intel(R) Xeon(R) Gold 6240 CPU @ 2.60GHz

Text to Speech

A100 Benchmarks

# of streams	Avg Latency to first audio (sec)	Avg Latency between audio chunks (sec)	Throughput (RTFX)	GPU Version
1	0.06	0.04	20	A100 SXM4-40GB
4	0.48	0.03	37	A100 SXM4-40GB
6	0.69	0.03	42	A100 SXM4-40GB
8	0.88	0.03	46	A100 SXM4-40GB
10	1.06	0.03	49	A100 SXM4-40GB

V100 Benchmarks

# of streams	Avg Latency to first audio (sec)	Avg Latency between audio chunks (sec)	Throughput (RTFX)	GPU Version
1	0.08	0.05	14	V100 SXM2-16GB
4	0.77	0.05	23	V100 SXM2-16GB
6	1.11	0.05	26	V100 SXM2-16GB
8	1.4	0.06	28	V100 SXM2-16GB
10	1.74	0.07	28	V100 SXM2-16GB

T4 Benchmarks

# of streams	Avg Latency to first audio (sec)	Avg Latency between audio chunks (sec)	Throughput (RTFX)	GPU Version
1	0.12	0.07	11	NVIDIA T4
4	1.02	0.07	17	NVIDIA T4
6	1.59	0.07	18	NVIDIA T4
8	2.13	0.08	19	NVIDIA T4
10	2.55	0.1	18	NVIDIA T4

TTS Throughput (RTFX) - Number of seconds of audio generated per second | Dataset: LJSpeech | Performance of the Jarvis text-to-speech (TTS) service was measured for different number of parallel streams. Each parallel stream performed 10 iterations over 10 input strings from the LJSpeech dataset. Latency to first audio chunk and latency between successive audio chunks and throughput were measured. Jarvis version: v1.0.0-b1 | Hardware: NVIDIA DGX A100 (1x A100 SXM4-40GB), NVIDIA DGX-1 (1x V100-SXM2-16GB), NVIDIA T4 with 2x Intel(R) Xeon(R) Gold 6240 CPU @ 2.60GHz

Last updated: June 30th, 2021

NVIDIA Data Center Deep Learning Product Performance

Training - Convergence

Training - Single GPU

Inference

AI Pipeline

NVIDIA Performance on MLPerf 1.0 AI Benchmarks

MLPerf Training Performance

NVIDIA A100 Performance on MLPerf 1.0 AI Benchmarks - Closed Division

Converged Training Performance of NVIDIA A100, A40, A30, A10, V100 and T4

A100 Training Performance

A40 Training Performance

A30 Training Performance

A10 Training Performance

V100 Training Performance

T4 Training Performance

Single GPU Training Performance of NVIDIA A100, A40, A30, A10, V100 and T4

A100 Training Performance

A40 Training Performance

A30 Training Performance

A10 Training Performance

V100 Training Performance

T4 Training Performance

MLPerf Inference v1.0 Performance Benchmarks

Offline Scenario - Closed Division

Server Scenario - Closed Division

Power Efficiency Offline Scenario - Closed Division

Power Efficiency Server Scenario - Closed Division

NVIDIA TRITON Inference Server Delivered Comparable Performance to Custom Harness in MLPerf v1.0

NVIDIA Client Batch Size=1 Performance with Triton Inference Server

Inference Performance of NVIDIA A100, A40, A30, A10, V100 and T4

Inference Natural Langugage Processing

NVIDIA A100 BERT Inference Benchmarks

Inference Image Classification on CNNs with TensorRT

A100 1/7 MIG Inference Performance

A100 7 MIG Inference Performance

A100 Full Chip Inference Performance

A40 Inference Performance

A30 Inference Performance

A10 Inference Performance

V100 Inference Performance

T4 Inference Performance

Jarvis Benchmarks

Automatic Speech Recognition

A100 Best Streaming Throughput Mode (800 ms chunk)

A100 Best Streaming Latency Mode (100 ms chunk)

A100 Offline Mode (3200 ms chunk)

V100 Best Streaming Throughput Mode (800 ms chunk)

V100 Best Streaming Latency Mode (100 ms chunk)

V100 Offline Mode (3200 ms chunk)

T4 Best Streaming Throughput Mode (800 ms chunk)

T4 Best Streaming Latency Mode (100 ms chunk)

T4 Offline Mode (3200 ms chunk)

Natural Language Processing

A100 Benchmarks

V100 Benchmarks

T4 Benchmarks

Text to Speech

A100 Benchmarks

V100 Benchmarks

T4 Benchmarks