NVIDIA Data Center Deep Learning Product Performance

Training to Convergence

Deploying AI in real-world applications requires training networks to convergence at a specified accuracy. This is the best methodology to test whether AI systems are ready to be deployed in the field to deliver meaningful results.

Related Resources

Read our blog on convergence for more details.

Get up and running quickly with NVIDIA’s complete solution stack:

Pull software containers from NVIDIA NGC.

NVIDIA Performance on MLPerf 2.1 Training Benchmarks

BERT Time to Train on A100

PyTorch | Precision: Mixed | Dataset: Wikipedia 2020/01/01 | Convergence criteria - refer to MLPerf requirements

MLPerf Training Performance

NVIDIA Performance on MLPerf 2.1 AI Benchmarks: Single Node - Closed Division

Framework	Network	Time to Train (mins)	MLPerf Quality Target	GPU	Server	MLPerf-ID	Precision	Dataset	GPU Version
MXNet	ResNet-50 v1.5	14.746	75.90% classification	8x H100	DGX H100	2.1-2091	Mixed	ImageNet2012	H100-SXM5-80GB
		27.688	75.90% classification	8x A100	GIGABYTE: G492-ZD2	2.1-2038	Mixed	ImageNet2012	A100-SXM4-80GB
	3D U-Net	13.101	0.908 Mean DICE score	8x H100	DGX H100	2.1-2091	Mixed	KiTS 2019	H100-SXM5-80GB
		22.989	0.908 Mean DICE score	8x A100	GIGABYTE: G492-ZD2	2.1-2038	Mixed	KiTS 2019	A100-SXM4-80GB
PyTorch	BERT	6.378	0.72 Mask-LM accuracy	8x H100	DGX H100	2.1-2091	Mixed	Wikipedia 2020/01/01	H100-SXM5-80GB
		16.549	0.72 Mask-LM accuracy	8x A100	GIGABYTE: G492-ZD2	2.1-2039	Mixed	Wikipedia 2020/01/01	A100-SXM4-80GB
	Mask R-CNN	20.348	0.377 Box min AP and 0.339 Mask min AP	8x H100	DGX H100	2.1-2091	Mixed	COCO2017	H100-SXM5-80GB
		37.916	0.377 Box min AP and 0.339 Mask min AP	8x A100	GIGABYTE: G492-ZD2	2.1-2039	Mixed	COCO2017	A100-SXM4-80GB
	RNN-T	18.202	0.058 Word Error Rate	8x H100	DGX H100	2.1-2091	Mixed	LibriSpeech	H100-SXM5-80GB
		29.948	0.058 Word Error Rate	8x A100	GIGABYTE: G492-ZD2	2.1-2039	Mixed	LibriSpeech	A100-SXM4-80GB
	RetinaNet	38.050	34.0% mAP	8x H100	DGX H100	2.1-2091	Mixed	OpenImages	H100-SXM5-80GB
		82.529	34.0% mAP	8x A100	GIGABYTE: G492-ZD2	2.1-2039	Mixed	OpenImages	A100-SXM4-80GB
TensorFlow	MiniGo	174.584	50% win rate vs. checkpoint	8x H100	DGX H100	2.1-2091	Mixed	Go	H100-SXM5-80GB
		161.848	50% win rate vs. checkpoint	8x A100	GIGABYTE: G492-ZD2	2.1-2040	Mixed	Go	A100-SXM4-80GB
NVIDIA Merlin HugeCTR	DLRM	1.063	0.8025 AUC	8x H100	DGX H100	2.1-2091	Mixed	Criteo AI Lab’s Terabyte Click-Through-Rate (CTR)	H100-SXM5-80GB
		1.625	0.8025 AUC	8x A100	Fujitsu: PRIMERGY-GX2570M6-hugectr	2.1-2033	Mixed	Criteo AI Lab’s Terabyte Click-Through-Rate (CTR)	A100-SXM4-80GB

NVIDIA Performance on MLPerf 2.1 AI Benchmarks: Multi Node - Closed Division

Framework	Network	Time to Train (mins)	MLPerf Quality Target	GPU	Server	MLPerf-ID	Precision	Dataset	GPU Version
MXNet	ResNet-50 v1.5	4.508	75.90% classification	32x H100	DGX H100	2.1-2093	Mixed	ImageNet2012	H100-SXM5-80GB
		4.523	75.90% classification	64x A100	DGX A100	2.1-2065	Mixed	ImageNet2012	A100-SXM4-80GB
		0.555	75.90% classification	1,024x A100	DGX A100	2.1-2073	Mixed	ImageNet2012	A100-SXM4-80GB
		0.319	75.90% classification	4,216x A100	DGX A100	2.1-2080	Mixed	ImageNet2012	A100-SXM4-80GB
	3D U-Net	5.347	0.908 Mean DICE score	24x H100	DGX H100	2.1-2092	Mixed	KiTS 2019	H100-SXM5-80GB
		3.437	0.908 Mean DICE score	72x A100	Azure: ND96amsr_A100_v4_n9	2.1-2009	Mixed	KiTS 2019	A100-SXM4-80GB
		1.216	0.908 Mean DICE score	768x A100	DGX A100	2.1-2072	Mixed	KiTS 2019	A100-SXM4-80GB
PyTorch	BERT	1.797	0.72 Mask-LM accuracy	32x H100	DGX H100	2.1-2093	Mixed	Wikipedia 2020/01/01	H100-SXM5-80GB
		2.497	0.72 Mask-LM accuracy	64x A100	DGX A100	2.1-2068	Mixed	Wikipedia 2020/01/01	A100-SXM4-80GB
		0.421	0.72 Mask-LM accuracy	1,024x A100	DGX A100	2.1-2074	Mixed	Wikipedia 2020/01/01	A100-SXM4-80GB
		0.208	0.72 Mask-LM accuracy	4,096x A100	DGX A100	2.1-2079	Mixed	Wikipedia 2020/01/01	A100-SXM4-80GB
	Mask R-CNN	7.338	0.377 Box min AP and 0.339 Mask min AP	32x H100	DGX H100	2.1-2093	Mixed	COCO2017	H100-SXM5-80GB
		8.293	0.377 Box min AP and 0.339 Mask min AP	64x A100	HPE-ProLiant-XL675d-Gen10-Plus_A100-SXM-80GB_pytorch	2.1-2049	Mixed	COCO2017	A100-SXM4-80GB
		2.750	0.377 Box min AP and 0.339 Mask min AP	384x A100	DGX A100	2.1-2071	Mixed	COCO2017	A100-SXM4-80GB
	RNN-T	7.534	0.058 Word Error Rate	32x H100	DGX H100	2.1-2093	Mixed	LibriSpeech	H100-SXM5-80GB
		6.910	0.058 Word Error Rate	64x A100	DGX A100	2.1-2066	Mixed	LibriSpeech	A100-SXM4-80GB
		2.151	0.058 Word Error Rate	1,536x A100	DGX A100	2.1-2076	Mixed	LibriSpeech	A100-SXM4-80GB
	RetinaNet	11.798	34.0% mAP	32x H100	DGX H100	2.1-2093	Mixed	OpenImages	H100-SXM5-80GB
		12.763	34.0% mAP	64x A100	DGX A100	2.1-2068	Mixed	OpenImages	A100-SXM4-80GB
		2.349	34.0% mAP	1,280x A100	DGX A100	2.1-2075	Mixed	OpenImages	A100-SXM4-80GB
		1.843	34.0% mAP	2,048x A100	DGX A100	2.1-2078	Mixed	OpenImages	A100-SXM4-80GB
TensorFlow	MiniGo	92.522	50% win rate vs. checkpoint	32x H100	DGX H100	2.1-2093	Mixed	Go	H100-SXM5-80GB
		73.038	50% win rate vs. checkpoint	64x A100	DGX A100	2.1-2067	Mixed	Go	A100-SXM4-80GB
		16.231	50% win rate vs. checkpoint	1,792x A100	DGX A100	2.1-2077	Mixed	Go	A100-SXM4-80GB
NVIDIA Merlin HugeCTR	DLRM	0.515	0.8025 AUC	32x H100	DGX H100	2.1-2093	Mixed	Criteo AI Lab’s Terabyte Click-Through-Rate (CTR)	H100-SXM5-80GB
		0.653	0.8025 AUC	64x A100	DGX A100	2.1-2064	Mixed	Criteo AI Lab’s Terabyte Click-Through-Rate (CTR)	A100-SXM4-80GB
		0.588	0.8025 AUC	112x A100	DGX A100	2.1-2070	Mixed	Criteo AI Lab’s Terabyte Click-Through-Rate (CTR)	A100-SXM4-80GB

MLPerf™ v2.1 Training Closed: 2.1-2038, 2.1-2039, 2.1-2033, 2.1-2040, 2.1-2065, 2.1-2068, 2.1-2049, 2.1-2066, 2.1-2064, 2.1-2067, 2.1-2009, 2.1-2070, 2.1-2071, 2.1-2072, 2.1-2073, 2.1-2074, 2.1-2075, 2.1-2076, 2.1-2077, 2.1-2078, 2.1-2079, 2.1-2080, 2.1-2091, 2.1-2092, 2.1-2093 | MLPerf name and logo are trademarks. See https://mlcommons.org/ for more information.
H100 SXM5-80GB is a preview submission

NVIDIA A100 Performance on MLPerf 2.0 Training HPC Benchmarks: Strong Scaling - Closed Division

Framework	Network	Time to Train (mins)	MLPerf Quality Target	GPU	Server	MLPerf-ID	Precision	Dataset	GPU Version
PyTorch	CosmoFlow	3.79	Mean average error 0.124	512x A100	DGX A100	2.0-8006	Mixed	CosmoFlow N-body cosmological simulation data with 4 cosmological parameter targets	A100-SXM4-80GB
	DeepCAM	1.57	IOU 0.82	2,048x A100	DGX A100	2.0-8005	Mixed	CAM5+TECA climate simulation with 3 target classes (atmospheric river, tropical cyclone, background)	A100-SXM4-80GB
	OpenCatalyst	21.93	Forces mean absolute error 0.036	512x A100	DGX A100	2.0-8006	Mixed	Open Catalyst 2020 (OC20) S2EF 2M training split, ID validation set	A100-SXM4-80GB

NVIDIA A100 Performance on MLPerf 2.0 Training HPC Benchmarks: Weak Scaling - Closed Division

Framework	Network	Throughput	MLPerf Quality Target	GPU	Server	MLPerf-ID	Precision	Dataset	GPU Version
PyTorch	CosmoFlow	4.21 models/min	Mean average error 0.124	4,096x A100	DGX A100	2.0-8014	Mixed	CosmoFlow N-body cosmological simulation data with 4 cosmological parameter targets	A100-SXM4-80GB
	DeepCAM	6.40 models/min	IOU 0.82	4,096x A100	DGX A100	2.0-8014	Mixed	CAM5+TECA climate simulation with 3 target classes (atmospheric river, tropical cyclone, background)	A100-SXM4-80GB
	OpenCatalyst	0.66 models/min	Forces mean absolute error 0.036	4,096x A100	DGX A100	2.0-8014	Mixed	Open Catalyst 2020 (OC20) S2EF 2M training split, ID validation set	A100-SXM4-80GB

MLPerf™ v2.0 Training HPC Closed: 2.0-8005, 2.0-8006, 2.0-8014 | MLPerf name and logo are trademarks. See https://mlcommons.org/ for more information.
For MLPerf™ v2.0 Training HPC rules and guidelines, click here

Converged Training Performance of NVIDIA A100, A40, A30, A10, T4 and V100

Benchmarks are reproducible by following links to the NGC catalog scripts

A100 Training Performance

Framework	Framework Version	Network	Time to Train (mins)	Accuracy	Throughput	GPU	Server	Container	Precision	Batch Size	Dataset	GPU Version
PyTorch	1.13.0a0	Tacotron2	98	.56 Training Loss	316,564 total output mels/sec	8x A100	DGX A100	22.09-py3	TF32	128	LJSpeech 1.1	A100-SXM4-80GB
	1.13.0a0	WaveGlow	241	-5.72 Training Loss	1,763,443 output samples/sec	8x A100	DGX A100	22.07-py3	Mixed	10	LJSpeech 1.1	A100-SXM4-80GB
	1.13.0a0	GNMT v2	16	24.4 BLEU Score	962,349 total tokens/sec	8x A100	DGX A100	22.08-py3	Mixed	128	wmt16-en-de	A100-SXM4-80GB
	1.13.0a0	NCF	0.35	.96 Hit Rate at 10	160,067,506 samples/sec	8x A100	DGX A100	22.09-py3	Mixed	131072	MovieLens 20M	A100-SXM4-80GB
	1.13.0a0	Transformer XL Base	181	22.34 Perplexity	732,282 total tokens/sec	8x A100	DGX A100	22.09-py3	Mixed	128	WikiText-103	A100-SXM4-80GB
	1.13.0a0	SE3 Transformer	9	.04 MAE	22,339 molecules/sec	8x A100	DGX A100	22.08-py3	Mixed	240	Quantum Machines 9	A100-SXM4-80GB
Tensorflow	1.15.5	U-Net Industrial	1	.99 IoU Threshold 0.99	1,072 images/sec	8x A100	DGX A100	22.09-py3	Mixed	2	DAGM2007	A100-SXM4-80GB
	2.9.1	U-Net Medical	2	.89 DICE Score	1,057 images/sec	8x A100	DGX A100	22.09-py3	Mixed	8	EM segmentation challenge	A100-SXM4-40GB
	2.9.1	Electra Fine Tuning	3	92.35 F1	2,635 sequences/sec	8x A100	DGX A100	22.09-py3	Mixed	32	SQuaD v1.1	A100-SXM4-40GB
	2.9.1	EfficientNet-B0	528	76.48 Top 1	20,347 images/sec	8x A100	DGX A100	22.09-py3	Mixed	1024	Imagenet2012	A100-SXM4-80GB
	2.9.1	SIM	1	.82 AUC	3,097,360 samples/sec	8x A100	DGX A100	22.09-py3	Mixed	16384	Amazon Reviews	A100-SXM4-40GB

A40 Training Performance

Framework	Framework Version	Network	Time to Train (mins)	Accuracy	Throughput	GPU	Server	Container	Precision	Batch Size	Dataset	GPU Version
PyTorch	1.13.0a0	NCF	1	.96 Hit Rate at 10	50,352,046 samples/sec	8x A40	GIGABYTE G482-Z52-00	22.09-py3	Mixed	131072	MovieLens 20M	A40
	1.13.0a0	Tacotron2	115	.56 Training Loss	268,967 total output mels/sec	8x A40	Supermicro AS -4124GS-TNR	22.07-py3	Mixed	128	LJSpeech 1.1	A40
	1.13.0a0	WaveGlow	464	-5.74 Training Loss	907,704 output samples/sec	8x A40	Supermicro AS -4124GS-TNR	22.08-py3	Mixed	10	LJSpeech 1.1	A40
	1.13.0a0	GNMT v2	54	24.24 BLEU Score	324,183 total tokens/sec	8x A40	Supermicro AS -4124GS-TNR	22.08-py3	Mixed	128	wmt16-en-de	A40
	1.13.0a0	Transformer XL Base	438	22.41 Perplexity	304,580 total tokens/sec	8x A40	Supermicro AS -4124GS-TNR	22.09-py3	Mixed	128	WikiText-103	A40
	1.13.0a0	SE3 Transformer	14	.04 MAE	13,306 molecules/sec	8x A40	Supermicro AS -4124GS-TNR	22.09-py3	Mixed	240	Quantum Machines 9	A40
Tensorflow	1.15.5	U-Net Industrial	1	.99 IoU Threshold 0.99	734 images/sec	8x A40	GIGABYTE G482-Z52-00	22.09-py3	Mixed	2	DAGM2007	A40
	2.9.1	Electra Fine Tuning	4	92.66 F1	1,128 sequences/sec	8x A40	Supermicro AS -4124GS-TNR	22.09-py3	Mixed	32	SQuaD v1.1	A40

A30 Training Performance

Framework	Framework Version	Network	Time to Train (mins)	Accuracy	Throughput	GPU	Server	Container	Precision	Batch Size	Dataset	GPU Version
PyTorch	1.13.0a0	Tacotron2	118	.53 Training Loss	262,520 total output mels/sec	8x A30	GIGABYTE G482-Z52-00	22.09-py3	Mixed	104	LJSpeech 1.1	A30
	1.13.0a0	WaveGlow	426	-5.68 Training Loss	986,408 output samples/sec	8x A30	GIGABYTE G482-Z52-00	22.09-py3	Mixed	10	LJSpeech 1.1	A30
	1.13.0a0	GNMT v2	54	24.42 BLEU Score	324,219 total tokens/sec	8x A30	GIGABYTE G482-Z52-00	22.09-py3	Mixed	128	wmt16-en-de	A30
	1.13.0a0	NCF	1	.96 Hit Rate at 10	57,621,616 samples/sec	8x A30	GIGABYTE G482-Z52-00	22.08-py3	Mixed	131072	MovieLens 20M	A30
	1.13.0a0	FastPitch	435	2.7 Training Loss	180,819 frames/sec	8x A30	GIGABYTE G482-Z52-00	22.07-py3	Mixed	16	LJSpeech 1.1	A30
	1.13.0a0	Transformer XL Base	147	23.69 Perplexity	228,197 total tokens/sec	8x A30	GIGABYTE G482-Z52-00	22.07-py3	Mixed	32	WikiText-103	A30
	1.13.0a0	SE3 Transformer	12	.04 MAE	16,339 molecules/sec	8x A30	GIGABYTE G482-Z52-00	22.08-py3	Mixed	240	Quantum Machines 9	A30
Tensorflow	1.15.5	U-Net Industrial	1	.99 IoU Threshold 0.99	674 images/sec	8x A30	GIGABYTE G482-Z52-00	22.09-py3	Mixed	2	DAGM2007	A30
	2.9.1	U-Net Medical	3	.89 DICE Score	469 images/sec	8x A30	GIGABYTE G482-Z52-00	22.09-py3	Mixed	8	EM segmentation challenge	A30
	2.9.1	Electra Fine Tuning	5	92.6 F1	975 sequences/sec	8x A30	GIGABYTE G482-Z52-00	22.09-py3	Mixed	16	SQuaD v1.1	A30
	2.9.1	SIM	3	.82 AUC	2,257,568 samples/sec	8x A30	GIGABYTE G482-Z52-00	22.09-py3	Mixed	16384	Amazon Reviews	A30

FastPitch throughput metric frames/sec refers to mel-scale spectrogram frames/sec

A10 Training Performance

Framework	Framework Version	Network	Time to Train (mins)	Accuracy	Throughput	GPU	Server	Container	Precision	Batch Size	Dataset	GPU Version
PyTorch	1.13.0a0	Tacotron2	141	.54 Training Loss	215,014 total output mels/sec	8x A10	GIGABYTE G482-Z52-00	22.08-py3	Mixed	104	LJSpeech 1.1	A10
	1.13.0a0	WaveGlow	590	-5.75 Training Loss	710,549 output samples/sec	8x A10	GIGABYTE G482-Z52-00	22.07-py3	Mixed	10	LJSpeech 1.1	A10
	1.13.0a0	GNMT V2	56	24.31 BLEU Score	256,704 total tokens/sec	8x A10	GIGABYTE G482-Z52-00	22.08-py3	Mixed	128	wmt16-en-de	A10
	1.13.0a0	NCF	1	.96 Hit Rate at 10	43,387,067 samples/sec	8x A10	GIGABYTE G482-Z52-00	22.09-py3	Mixed	131072	MovieLens 20M	A10
	1.13.0a0	SE3 Transformer	15	.04 MAE	12,019 molecules/sec	8x A10	GIGABYTE G482-Z52-00	22.09-py3	Mixed	240	Quantum Machines 9	A10
Tensorflow	1.15.5	U-Net Industrial	1	.99 IoU Threshold 0.99	645 images/sec	8x A10	GIGABYTE G482-Z52-00	22.08-py3	Mixed	2	DAGM2007	A10
	1.15.5	U-Net Medical	13	.9 DICE Score	344 images/sec	8x A10	GIGABYTE G482-Z52-00	22.07-py3	Mixed	8	EM segmentation challenge	A10
	2.8.0	Electra Base Fine Tuning	6	92.64 F1	753 sequences/sec	8x A10	GIGABYTE G482-Z52-00	22.05-py3	Mixed	16	SQuaD v1.1	A10
	2.9.1	SIM	1	.83 AUC	2,155,776 samples/sec	8x A10	GIGABYTE G482-Z52-00	22.09-py3	Mixed	16384	Amazon Reviews	A10

T4 Training Performance +

Framework	Framework Version	Network	Time to Train (mins)	Accuracy	Throughput	GPU	Server	Container	Precision	Batch Size	Dataset	GPU Version
PyTorch	1.13.0a0	Tacotron2	228	.53 Training Loss	133,890 total output mels/sec	8x T4	Supermicro SYS-4029GP-TRT	22.09-py3	Mixed	104	LJSpeech 1.1	NVIDIA T4
	1.13.0a0	WaveGlow	999	-5.69 Training Loss	420,050 output samples/sec	8x T4	Supermicro SYS-4029GP-TRT	22.07-py3	Mixed	10	LJSpeech 1.1	NVIDIA T4
	1.13.0a0	GNMT v2	109	24.21 BLEU Score	132,047 total tokens/sec	8x T4	Supermicro SYS-4029GP-TRT	22.09-py3	Mixed	128	wmt16-en-de	NVIDIA T4
	1.13.0a0	NCF	2	.96 Hit Rate at 10	26,243,408 samples/sec	8x T4	Supermicro SYS-4029GP-TRT	22.09-py3	Mixed	131072	MovieLens 20M	NVIDIA T4
	1.13.0a0	SE3 Transformer	37	.04 MAE	4,742 molecules/sec	8x T4	Supermicro SYS-4029GP-TRT	22.09-py3	Mixed	240	Quantum Machines 9	NVIDIA T4
Tensorflow	1.15.5	U-Net Industrial	2	.99 IoU Threshold 0.99	300 images/sec	8x T4	Supermicro SYS-4029GP-TRT	22.09-py3	Mixed	2	DAGM2007	NVIDIA T4
	1.15.5	U-Net Medical	39	.9 DICE Score	156 images/sec	8x T4	Supermicro SYS-4029GP-TRT	22.09-py3	Mixed	8	EM segmentation challenge	NVIDIA T4
	2.9.1	Electra Fine Tuning	10	92.74 F1	376 sequences/sec	8x T4	Supermicro SYS-4029GP-TRT	22.09-py3	Mixed	16	SQuaD v1.1	NVIDIA T4
	1.15.5	Transformer XL Base	938	22.87 Perplexity	35,000 total tokens/sec	8x T4	Supermicro SYS-4029GP-TRT	22.09-py3	Mixed	16	WikiText-103	NVIDIA T4
	2.9.1	SIM	2	.81 AUC	1,079,721 samples/sec	8x T4	Supermicro SYS-4029GP-TRT	22.09-py3	Mixed	16384	Amazon Reviews	NVIDIA T4

V100 Training Performance +

Framework	Framework Version	Network	Time to Train (mins)	Accuracy	Throughput	GPU	Server	Container	Precision	Batch Size	Dataset	GPU Version
PyTorch	1.13.0a0	Tacotron2	176	.49 Training Loss	170,958 total output mels/sec	8x V100	DGX-2	22.08-py3	Mixed	104	LJSpeech 1.1	V100-SXM3-32GB
	1.13.0a0	WaveGlow	399	-5.81 Training Loss	1,069,377 output samples/sec	8x V100	DGX-2	22.08-py3	Mixed	10	LJSpeech 1.1	V100-SXM3-32GB
	1.13.0a0	GNMT v2	33	24.1 BLEU Score	443,670 total tokens/sec	8x V100	DGX-2	22.08-py3	Mixed	128	wmt16-en-de	V100-SXM3-32GB
	1.13.0a0	NCF	1	.96 Hit Rate at 10	99,209,033 samples/sec	8x V100	DGX-2	22.09-py3	Mixed	131072	MovieLens 20M	V100-SXM3-32GB
	1.13.0a0	SE3 Transformer	14	.04 MAE	13,382 molecules/sec	8x V100	DGX-2	22.09-py3	Mixed	240	Quantum Machines 9	V100-SXM3-32GB
Tensorflow	1.15.5	U-Net Industrial	1	.99 IoU Threshold 0.99	634 images/sec	8x V100	DGX-2	22.06-py3	Mixed	2	DAGM2007	V100-SXM3-32GB
	1.15.5	U-Net Medical	13	.9 DICE Score	464 images/sec	8x V100	DGX-2	22.09-py3	Mixed	8	EM segmentation challenge	V100-SXM3-32GB
	1.15.5	Transformer XL Base	318	22.32 Perplexity	103,869 total tokens/sec	8x V100	DGX-2	22.08-py3	Mixed	16	WikiText-103	V100-SXM3-32GB
	2.9.1	Electra Fine Tuning	5	92.36 F1	1,365 sequences/sec	8x V100	DGX-2	22.09-py3	Mixed	32	SQuaD v1.1	V100-SXM3-32GB
	2.9.1	SIM	1	.8 AUC	2,190,082 samples/sec	8x V100	DGX-2	22.09-py3	Mixed	16384	Amazon Reviews	V100-SXM3-32GB

Single GPU Training
Single GPU CSP Training

Single-GPU Training

Some scenarios aren’t used in real-world training, such as single-GPU throughput. The table below provides an indication of a platform’s single-chip throughput.

Related Resources

Achieve unprecedented acceleration at every scale with NVIDIA’s complete solution stack.

Pull software containers from NVIDIA NGC.

Scenarios that are not typically used in real-world training, such as single GPU throughput are illustrated in the table below, and provided for reference as an indication of single chip throughput of the platform.

NVIDIA’s complete solution stack, from hardware to software, allows data scientists to deliver unprecedented acceleration at every scale. Visit the NVIDIA NGC catalog to pull containers and quickly get up and running with deep learning.

Single GPU Training Performance of NVIDIA A100, A40, A30, A10, T4 and V100

Benchmarks are reproducible by following links to the NGC catalog scripts

A100 Training Performance

Framework	Framework Version	Network	Throughput	GPU	Server	Container	Precision	Batch Size	Dataset	GPU Version
MXNet	1.9.1	ResNet-50 v1.5	3,307 images/sec	1x A100	DGX A100	22.09-py3	Mixed	256	ImageNet2012	A100-SXM4-80GB
PyTorch	1.13.0a0	Tacotron2	42,388 total output mels/sec	1x A100	DGX A100	22.09-py3	Mixed	128	LJSpeech 1.1	A100-SXM4-80GB
	1.13.0a0	WaveGlow	255,902 output samples/sec	1x A100	DGX A100	22.09-py3	Mixed	10	LJSpeech 1.1	A100-SXM4-80GB
	1.13.0a0	FastPitch	107,080 frames/sec	1x A100	DGX A100	22.09-py3	Mixed	32	LJSpeech 1.1	A100-SXM4-80GB
	1.13.0a0	GNMT v2	170,862 total tokens/sec	1x A100	DGX A100	22.09-py3	Mixed	128	wmt16-en-de	A100-SXM4-80GB
	1.13.0a0	NCF	41,257,218 samples/sec	1x A100	DGX A100	22.09-py3	Mixed	1048576	MovieLens 20M	A100-SXM4-80GB
	1.13.0a0	Transformer XL Large	17,053 total tokens/sec	1x A100	DGX A100	22.09-py3	Mixed	16	WikiText-103	A100-SXM4-80GB
	1.13.0a0	Transformer XL Base	89,417 total tokens/sec	1x A100	DGX A100	22.09-py3	Mixed	128	WikiText-103	A100-SXM4-80GB
	1.13.0a0	nnU-Net	1,128 images/sec	1x A100	DGX A100	22.09-py3	Mixed	64	Medical Segmentation Decathlon	A100-SXM4-80GB
	1.13.0a0	EfficientNet-B4	389 images/sec	1x A100	DGX A100	22.09-py3	Mixed	128	Imagenet2012	A100-SXM4-80GB
	1.13.0a0	BERT Large Pre-Training Phase 2	302 sequences/sec	1x A100	DGX A100	22.07-py3	Mixed	56	Wikipedia 2020/01/01	A100-SXM4-80GB
	1.13.0a0	BERT Large Pre-Training Phase 1	853 sequences/sec	1x A100	DGX A100	22.07-py3	Mixed	512	Wikipedia 2020/01/01	A100-SXM4-80GB
	1.13.0a0	EfficientNet-WideSE-B4	388 images/sec	1x A100	DGX A100	22.09-py3	Mixed	128	Imagenet2012	A100-SXM4-80GB
	1.13.0a0	SE3 Transformer	2,983 molecules/sec	1x A100	DGX A100	22.09-py3	Mixed	240	Quantum Machines 9	A100-SXM4-80GB
	1.13.0a0	TFT - Traffic	17,483 items/sec	1x A100	DGX A100	22.09-py3	Mixed	1024	Traffic	A100-SXM4-80GB
	1.13.0a0	TFT - Electricity	17,355 items/sec	1x A100	DGX A100	22.09-py3	Mixed	1024	Electricity	A100-SXM4-80GB
Tensorflow	1.15.5	U-Net Industrial	351 images/sec	1x A100	DGX A100	22.09-py3	Mixed	16	DAGM2007	A100-SXM4-40GB
	2.9.1	U-Net Medical	150 images/sec	1x A100	DGX A100	22.09-py3	Mixed	8	EM segmentation challenge	A100-SXM4-80GB
	2.9.1	Electra Base Fine Tuning	369 sequences/sec	1x A100	DGX A100	22.09-py3	Mixed	32	SQuaD v1.1	A100-SXM4-80GB
	1.15.5	NCF	47,214,021 samples/sec	1x A100	DGX A100	22.09-py3	Mixed	1048576	MovieLens 20M	A100-SXM4-40GB
	2.9.1	EfficientNet-B0	3,271 images/sec	1x A100	DGX A100	22.09-py3	Mixed	1024	Imagenet2012	A100-SXM4-80GB
	2.9.1	SIM	594,102 samples/sec	1x A100	DGX A100	22.09-py3	Mixed	131072	Amazon Reviews	A100-SXM4-80GB

FastPitch throughput metric frames/sec refers to mel-scale spectrogram frames/sec | Server with a hyphen indicates a pre-production server

A40 Training Performance

Framework	Framework Version	Network	Throughput	GPU	Server	Container	Precision	Batch Size	Dataset	GPU Version
PyTorch	1.13.0a0	Tacotron2	37,587 total output mels/sec	1x A40	GIGABYTE G482-Z52-00	22.09-py3	Mixed	128	LJSpeech 1.1	A40
	1.13.0a0	WaveGlow	149,922 output samples/sec	1x A40	GIGABYTE G482-Z52-00	22.09-py3	Mixed	10	LJSpeech 1.1	A40
	1.13.0a0	GNMT v2	80,500 total tokens/sec	1x A40	GIGABYTE G482-Z52-00	22.09-py3	Mixed	128	wmt16-en-de	A40
	1.13.0a0	NCF	19,607,776 samples/sec	1x A40	GIGABYTE G482-Z52-00	22.09-py3	Mixed	1048576	MovieLens 20M	A40
	1.13.0a0	Transformer XL Large	10,105 total tokens/sec	1x A40	GIGABYTE G482-Z52-00	22.09-py3	Mixed	16	WikiText-103	A40
	1.13.0a0	FastPitch	94,386 frames/sec	1x A40	GIGABYTE G482-Z52-00	22.09-py3	Mixed	32	LJSpeech 1.1	A40
	1.13.0a0	Transformer XL Base	41,964 total tokens/sec	1x A40	GIGABYTE G482-Z52-00	22.09-py3	Mixed	128	WikiText-103	A40
	1.13.0a0	nnU-Net	561 images/sec	1x A40	GIGABYTE G482-Z52-00	22.09-py3	Mixed	64	Medical Segmentation Decathlon	A40
	1.13.0a0	EfficientNet-B4	182 images/sec	1x A40	GIGABYTE G482-Z52-00	22.08-py3	Mixed	64	Imagenet2012	A40
	1.13.0a0	EfficientNet-WideSE-B4	182 images/sec	1x A40	GIGABYTE G482-Z52-00	22.08-py3	Mixed	64	Imagenet2012	A40
	1.13.0a0	SE3 Transformer	1,804 molecules/sec	1x A40	GIGABYTE G482-Z52-00	22.09-py3	Mixed	240	Quantum Machines 9	A40
	1.13.0a0	TFT - Traffic	9,684 items/sec	1x A40	GIGABYTE G482-Z52-00	22.09-py3	Mixed	1024	Traffic	A40
	1.13.0a0	TFT - Electricity	9,480 items/sec	1x A40	GIGABYTE G482-Z52-00	22.09-py3	Mixed	1024	Electricity	A40
Tensorflow	1.15.5	U-Net Industrial	123 images/sec	1x A40	GIGABYTE G482-Z52-00	22.09-py3	Mixed	16	DAGM2007	A40
	1.15.5	U-Net Medical	67 images/sec	1x A40	GIGABYTE G482-Z52-00	22.09-py3	Mixed	8	EM segmentation challenge	A40
	2.9.1	EfficientNet-B0	928 images/sec	1x A40	GIGABYTE G482-Z52-00	22.07-py3	TF32	512	Imagenet2012	A40

FastPitch throughput metric frames/sec refers to mel-scale spectrogram frames/sec | Server with a hyphen indicates a pre-production server

A30 Training Performance

Framework	Framework Version	Network	Throughput	GPU	Server	Container	Precision	Batch Size	Dataset	GPU Version
PyTorch	1.13.0a0	Tacotron2	34,616 total output mels/sec	1x A30	GIGABYTE G482-Z52-00	22.09-py3	Mixed	104	LJSpeech 1.1	A30
	1.13.0a0	WaveGlow	155,908 output samples/sec	1x A30	GIGABYTE G482-Z52-00	22.09-py3	Mixed	10	LJSpeech 1.1	A30
	1.13.0a0	FastPitch	88,937 frames/sec	1x A30	GIGABYTE G482-Z52-00	22.09-py3	Mixed	16	LJSpeech 1.1	A30
	1.13.0a0	NCF	21,661,040 samples/sec	1x A30	GIGABYTE G482-Z52-00	22.09-py3	Mixed	1048576	MovieLens 20M	A30
	1.13.0a0	GNMT v2	91,540 total tokens/sec	1x A30	GIGABYTE G482-Z52-00	22.09-py3	Mixed	128	wmt16-en-de	A30
	1.13.0a0	Transformer XL Base	19,014 total tokens/sec	1x A30	GIGABYTE G482-Z52-00	22.09-py3	Mixed	32	WikiText-103	A30
	1.13.0a0	Transformer XL Large	7,020 total tokens/sec	1x A30	GIGABYTE G482-Z52-00	22.09-py3	Mixed	4	WikiText-103	A30
	1.13.0a0	nnU-Net	597 images/sec	1x A30	GIGABYTE G482-Z52-00	22.09-py3	Mixed	64	Medical Segmentation Decathlon	A30
	1.13.0a0	EfficientNet-B4	189 images/sec	1x A30	GIGABYTE G482-Z52-00	22.09-py3	Mixed	32	Imagenet2012	A30
	1.13.0a0	EfficientNet-WideSE-B4	188 images/sec	1x A30	GIGABYTE G482-Z52-00	22.09-py3	Mixed	32	Imagenet2012	A30
	1.13.0a0	SE3 Transformer	2,135 molecules/sec	1x A30	GIGABYTE G482-Z52-00	22.09-py3	Mixed	240	Quantum Machines 9	A30
	1.13.0a0	TFT - Traffic	10,535 items/sec	1x A30	GIGABYTE G482-Z52-00	22.09-py3	Mixed	1024	Traffic	A30
	1.13.0a0	TFT - Electricity	10,498 items/sec	1x A30	GIGABYTE G482-Z52-00	22.09-py3	Mixed	1024	Electricity	A30
Tensorflow	1.15.5	U-Net Industrial	117 images/sec	1x A30	GIGABYTE G482-Z52-00	22.09-py3	Mixed	16	DAGM2007	A30
	2.9.1	U-Net Medical	74 images/sec	1x A30	GIGABYTE G482-Z52-00	22.09-py3	Mixed	8	EM segmentation challenge	A30
	1.15.5	Transformer XL Base	18,647 total tokens/sec	1x A30	GIGABYTE G482-Z52-00	22.09-py3	Mixed	16	WikiText-103	A30
	2.9.1	Electra Base Fine Tuning	162 sequences/sec	1x A30	GIGABYTE G482-Z52-00	22.09-py3	Mixed	16	SQuaD v1.1	A30
	2.9.1	EfficientNet-B0	1,620 images/sec	1x A30	GIGABYTE G482-Z52-00	22.09-py3	Mixed	512	Imagenet2012	A30
	2.9.1	SIM	396,452 samples/sec	1x A30	GIGABYTE G482-Z52-00	22.09-py3	Mixed	131072	Amazon Reviews	A30

FastPitch throughput metric frames/sec refers to mel-scale spectrogram frames/sec | Server with a hyphen indicates a pre-production server

A10 Training Performance

Framework	Framework Version	Network	Throughput	GPU	Server	Container	Precision	Batch Size	Dataset	GPU Version
PyTorch	1.13.0a0	Tacotron2	28,564 total output mels/sec	1x A10	GIGABYTE G482-Z52-00	22.08-py3	Mixed	104	LJSpeech 1.1	A10
	1.13.0a0	WaveGlow	114,489 output samples/sec	1x A10	GIGABYTE G482-Z52-00	22.08-py3	Mixed	10	LJSpeech 1.1	A10
	1.13.0a0	FastPitch	73,387 frames/sec	1x A10	GIGABYTE G482-Z52-00	22.09-py3	Mixed	16	LJSpeech 1.1	A10
	1.13.0a0	Transformer XL Base	15,619 total tokens/sec	1x A10	GIGABYTE G482-Z52-00	22.09-py3	Mixed	32	WikiText-103	A10
	1.13.0a0	GNMT v2	64,911 total tokens/sec	1x A10	GIGABYTE G482-Z52-00	22.08-py3	Mixed	128	wmt16-en-de	A10
	1.13.0a0	NCF	16,363,689 samples/sec	1x A10	GIGABYTE G482-Z52-00	22.09-py3	Mixed	1048576	MovieLens 20M	A10
	1.13.0a0	Transformer XL Large	5,976 total tokens/sec	1x A10	GIGABYTE G482-Z52-00	22.09-py3	Mixed	4	WikiText-103	A10
	1.13.0a0	nnU-Net	448 images/sec	1x A10	GIGABYTE G482-Z52-00	22.09-py3	Mixed	64	Medical Segmentation Decathlon	A10
	1.13.0a0	EfficientNet-B4	145 images/sec	1x A10	GIGABYTE G482-Z52-00	22.09-py3	Mixed	32	Imagenet2012	A10
	1.13.0a0	EfficientNet-WideSE-B4	145 images/sec	1x A10	GIGABYTE G482-Z52-00	22.09-py3	Mixed	32	Imagenet2012	A10
	1.13.0a0	SE3 Transformer	1,576 molecules/sec	1x A10	GIGABYTE G482-Z52-00	22.09-py3	Mixed	240	Quantum Machines 9	A10
	1.13.0a0	TFT - Traffic	7,938 items/sec	1x A10	GIGABYTE G482-Z52-00	22.09-py3	Mixed	1024	Traffic	A10
	1.13.0a0	TFT - Electricity	7,964 items/sec	1x A10	GIGABYTE G482-Z52-00	22.09-py3	Mixed	1024	Electricity	A10
Tensorflow	1.15.5	U-Net Industrial	99 images/sec	1x A10	GIGABYTE G482-Z52-00	22.08-py3	Mixed	16	DAGM2007	A10
	2.9.1	U-Net Medical	51 images/sec	1x A10	GIGABYTE G482-Z52-00	22.09-py3	Mixed	8	EM segmentation challenge	A10
	2.9.1	Electra Base Fine Tuning	120 sequences/sec	1x A10	GIGABYTE G482-Z52-00	22.09-py3	Mixed	16	SQuaD v1.1	A10
	2.9.1	EfficientNet-B0	1,343 images/sec	1x A10	GIGABYTE G482-Z52-00	22.09-py3	Mixed	512	Imagenet2012	A10
	2.9.1	SIM	369,569 samples/sec	1x A10	GIGABYTE G482-Z52-00	22.09-py3	Mixed	131072	Amazon Reviews	A10

FastPitch throughput metric frames/sec refers to mel-scale spectrogram frames/sec | Server with a hyphen indicates a pre-production server

T4 Training Performance +

Framework	Framework Version	Network	Throughput	GPU	Server	Container	Precision	Batch Size	Dataset	GPU Version
PyTorch	1.13.0a0	Tacotron2	17,962 total output mels/sec	1x T4	Supermicro SYS-4029GP-TRT	22.09-py3	Mixed	104	LJSpeech 1.1	NVIDIA T4
	1.13.0a0	WaveGlow	55,246 output samples/sec	1x T4	Supermicro SYS-4029GP-TRT	22.09-py3	Mixed	10	LJSpeech 1.1	NVIDIA T4
	1.13.0a0	FastPitch	33,297 frames/sec	1x T4	Supermicro SYS-4029GP-TRT	22.09-py3	Mixed	16	LJSpeech 1.1	NVIDIA T4
	1.13.0a0	GNMT v2	30,655 total tokens/sec	1x T4	Supermicro SYS-4029GP-TRT	22.09-py3	Mixed	128	wmt16-en-de	NVIDIA T4
	1.13.0a0	NCF	7,713,244 samples/sec	1x T4	Supermicro SYS-4029GP-TRT	22.09-py3	Mixed	1048576	MovieLens 20M	NVIDIA T4
	1.13.0a0	Transformer XL Base	9,011 total tokens/sec	1x T4	Supermicro SYS-4029GP-TRT	22.09-py3	Mixed	32	WikiText-103	NVIDIA T4
	1.13.0a0	Transformer XL Large	2,729 total tokens/sec	1x T4	Supermicro SYS-4029GP-TRT	22.09-py3	Mixed	4	WikiText-103	NVIDIA T4
	1.13.0a0	nnU-Net	202 images/sec	1x T4	Supermicro SYS-4029GP-TRT	22.09-py3	Mixed	64	Medical Segmentation Decathlon	NVIDIA T4
	1.13.0a0	EfficientNet-B4	68 images/sec	1x T4	Supermicro SYS-4029GP-TRT	22.09-py3	Mixed	32	Imagenet2012	NVIDIA T4
	1.13.0a0	EfficientNet-WideSE-B4	67 images/sec	1x T4	Supermicro SYS-4029GP-TRT	22.09-py3	Mixed	32	Imagenet2012	NVIDIA T4
	1.13.0a0	SE3 Transformer	610 molecules/sec	1x T4	Supermicro SYS-4029GP-TRT	22.09-py3	Mixed	240	Quantum Machines 9	NVIDIA T4
	1.13.0a0	TFT - Traffic	4,297 items/sec	1x T4	Supermicro SYS-4029GP-TRT	22.09-py3	Mixed	1024	Traffic	NVIDIA T4
	1.13.0a0	TFT - Electricity	4,262 items/sec	1x T4	Supermicro SYS-4029GP-TRT	22.09-py3	Mixed	1024	Electricity	NVIDIA T4
Tensorflow	1.15.5	U-Net Industrial	44 images/sec	1x T4	Supermicro SYS-4029GP-TRT	22.09-py3	Mixed	16	DAGM2007	NVIDIA T4
	1.15.5	U-Net Medical	21 images/sec	1x T4	Supermicro SYS-4029GP-TRT	22.09-py3	Mixed	8	EM segmentation challenge	NVIDIA T4
	2.9.1	Electra Base Fine Tuning	56 sequences/sec	1x T4	Supermicro SYS-4029GP-TRT	22.09-py3	Mixed	16	SQuaD v1.1	NVIDIA T4
	2.9.1	EfficientNet-B0	579 images/sec	1x T4	Supermicro SYS-4029GP-TRT	22.09-py3	Mixed	256	Imagenet2012	NVIDIA T4
	2.9.1	SIM	168,087 samples/sec	1x T4	Supermicro SYS-4029GP-TRT	22.09-py3	Mixed	131072	Amazon Reviews	NVIDIA T4

FastPitch throughput metric frames/sec refers to mel-scale spectrogram frames/sec

V100 Training Performance +

Framework	Framework Version	Network	Throughput	GPU	Server	Container	Precision	Batch Size	Dataset	GPU Version
PyTorch	1.13.0a0	Tacotron2	23,774 total output mels/sec	1x V100	DGX-2	22.08-py3	Mixed	104	LJSpeech 1.1	V100-SXM3-32GB
	1.13.0a0	WaveGlow	155,021 output samples/sec	1x V100	DGX-2	22.08-py3	Mixed	10	LJSpeech 1.1	V100-SXM3-32GB
	1.13.0a0	FastPitch	87,274 frames/sec	1x V100	DGX-2	22.09-py3	Mixed	16	LJSpeech 1.1	V100-SXM3-32GB
	1.13.0a0	GNMT v2	78,543 total tokens/sec	1x V100	DGX-2	22.08-py3	Mixed	128	wmt16-en-de	V100-SXM3-32GB
	1.13.0a0	NCF	24,127,863 samples/sec	1x V100	DGX-2	22.09-py3	Mixed	1048576	MovieLens 20M	V100-SXM3-32GB
	1.13.0a0	Transformer XL Base	17,830 total tokens/sec	1x V100	DGX-2	22.09-py3	Mixed	32	WikiText-103	V100-SXM3-32GB
	1.13.0a0	Transformer XL Large	7,211 total tokens/sec	1x V100	DGX-2	22.09-py3	Mixed	8	WikiText-103	V100-SXM3-32GB
	1.13.0a0	nnU-Net	657 images/sec	1x V100	DGX-2	22.09-py3	Mixed	64	Medical Segmentation Decathlon	V100-SXM3-32GB
	1.13.0a0	EfficientNet-B4	220 images/sec	1x V100	DGX-2	22.09-py3	Mixed	64	Imagenet2012	V100-SXM3-32GB
	1.13.0a0	EfficientNet-WideSE-B4	220 images/sec	1x V100	DGX-2	22.09-py3	Mixed	64	Imagenet2012	V100-SXM3-32GB
	1.13.0a0	SE3 Transformer	1,989 molecules/sec	1x V100	DGX-2	22.09-py3	Mixed	240	Quantum Machines 9	V100-SXM3-32GB
	1.13.0a0	TFT - Traffic	11,685 items/sec	1x V100	DGX-2	22.09-py3	Mixed	1024	Traffic	V100-SXM3-32GB
	1.13.0a0	TFT - Electricity	11,602 items/sec	1x V100	DGX-2	22.09-py3	Mixed	1024	Electricity	V100-SXM3-32GB
Tensorflow	1.15.5	U-Net Industrial	118 images/sec	1x V100	DGX-2	22.08-py3	Mixed	16	DAGM2007	V100-SXM3-32GB
	1.15.5	U-Net Medical	68 images/sec	1x V100	DGX-2	22.09-py3	Mixed	8	EM segmentation challenge	V100-SXM3-32GB
	2.9.1	Electra Base Fine Tuning	188 sequences/sec	1x V100	DGX-2	22.09-py3	Mixed	32	SQuaD v1.1	V100-SXM3-32GB
	1.15.5	Transformer XL Base	18,514 total tokens/sec	1x V100	DGX-2	22.08-py3	Mixed	16	WikiText-103	V100-SXM3-32GB
	2.9.1	EfficientNet-B0	621 images/sec	1x V100	DGX-2	22.09-py3	FP32	256	Imagenet2012	V100-SXM3-32GB
	2.9.1	SIM	351,636 samples/sec	1x V100	DGX-2	22.09-py3	Mixed	131072	Amazon Reviews	V100-SXM3-32GB

FastPitch throughput metric frames/sec refers to mel-scale spectrogram frames/sec

Single GPU Training Performance of NVIDIA GPU on Cloud

Benchmarks are reproducible by following links to the NGC catalog scripts

A100 Training Performance on Cloud

Framework	Framework Version	Network	Throughput	GPU	Server	Container	Precision	Batch Size	Dataset	GPU Version
MXNet	-	ResNet-50 v1.5	2,887 images/sec	1x A100	GCP A2-HIGHGPU-1G	22.07-py3	Mixed	192	ImageNet2012	A100-SXM4-40GB
PyTorch	-	DLRM	3,450,000 records/sec	1x A100	GCP A2-HIGHGPU-1G	22.07-py3	Mixed	32768	Criteo Terabyte Dataset	A100-SXM4-40GB

BERT-Large = BERT-Large Fine Tuning (Squadv1.1) with Sequence Length of 384

T4 Training Performance on Cloud +

Framework	Framework Version	Network	Throughput	GPU	Server	Container	Precision	Batch Size	Dataset	GPU Version
MXNet	-	ResNet-50 v1.5	450 images/sec	1x T4	AWS EC2 g4dn.4xlarge	22.06-py3	Mixed	192	ImageNet2012	NVIDIA T4
	-	ResNet-50 v1.5	432 images/sec	1x T4	GCP N1-HIGHMEM-8	22.07-py3	Mixed	192	ImageNet2012	NVIDIA T4
TensorFlow	-	ResNet-50 v1.5	419 images/sec	1x T4	AWS EC2 g4dn.4xlarge	22.06-py3	Mixed	256	Imagenet2012	NVIDIA T4

BERT-Large = BERT-Large Fine Tuning (Squadv1.1) with Sequence Length of 384

V100 Training Performance on Cloud +

Framework	Framework Version	Network	Throughput	GPU	Server	Container	Precision	Batch Size	Dataset	GPU Version
MXNet	-	ResNet-50 v1.5	1,467 images/sec	1x V100	GCP N1-HIGHMEM-8	22.07-py3	Mixed	192	ImageNet2012	V100-SXM2-16GB

BERT-Large = BERT-Large Fine Tuning (Squadv1.1) with Sequence Length of 384

AI Inference

Real-world inferencing demands high throughput and low latencies with maximum efficiency across use cases. An industry-leading solution lets customers quickly deploy AI models into real-world production with the highest performance from data center to edge.

Related Resources

Learn how NVIDIA landed top performance spots on all MLPerf Inference 2.1 tests.
Read the inference whitepape to explore the evolving landscape and get an overview of inference platforms.
Learn how Dynamic Batching can increase throughput on Triton with Benefits of Triton.
For additional data on Triton performance in offline and online server, please refer to ResNet-50 v1.5

Power high-throughput, low-latency inference with NVIDIA’s complete solution stack:

Achieve the most efficient inference performance with NVIDIA® TensorRT™ running on NVIDIA Tensor Core GPUs.
Maximize performance and simplify the deployment of AI models with the NVIDIA Triton™ Inference Server.
Pull software containers from NVIDIA NGC to race into production.

MLPerf Inference v2.1 Performance Benchmarks

Offline Scenario - Closed Division

Network	Throughput	GPU	Server	GPU Version	Dataset	Target Accuracy
ResNet-50 v1.5	81,292 samples/sec	1x H100	NVIDIA H100	H100-SXM-80GB	ImageNet	76.46% Top1
	335,144 samples/sec	8x A100	DGX A100	A100 SXM-80GB	ImageNet	76.46% Top1
	5,589 samples/sec	1x1g.10gb A100	DGX A100	A100 SXM-80GB	ImageNet	76.46% Top1
	316,342 samples/sec	8x A100	Gigabyte G482-Z54	A100 PCIe-80GB	ImageNet	76.46% Top1
RetinaNet	960 samples/sec	1x H100	NVIDIA H100	H100-SXM-80GB	OpenImages	0.3755 mAP
	4,739 samples/sec	8x A100	DGX A100	A100 SXM-80GB	OpenImages	0.3755 mAP
	74 samples/sec	1x1g.10gb A100	DGX A100	A100 SXM-80GB	OpenImages	0.3755 mAP
	4,345 samples/sec	8x A100	Gigabyte G482-Z54	A100 PCIe-80GB	OpenImages	0.3755 mAP
3D-UNet	5 samples/sec	1x H100	NVIDIA H100	H100-SXM-80GB	KiTS 2019	0.863 DICE mean
	26 samples/sec	8x A100	DGX A100	A100 SXM-80GB	KiTS 2019	0.863 DICE mean
	0.51 samples/sec	1x1g.10gb A100	DGX A100	A100 SXM-80GB	KiTS 2019	0.863 DICE mean
	25 samples/sec	8x A100	Gigabyte G482-Z54	A100 PCIe-80GB	KiTS 2019	0.863 DICE mean
RNN-T	22,885 samples/sec	1x H100	NVIDIA H100	H100-SXM-80GB	LibriSpeech	7.45% WER
	106,726 samples/sec	8x A100	DGX A100	A100 SXM-80GB	LibriSpeech	7.45% WER
	1,918 samples/sec	1x1g.10gb A100	DGX A100	A100 SXM-80GB	LibriSpeech	7.45% WER
	102,784 samples/sec	8x A100	Gigabyte G482-Z54	A100 PCIe-80GB	LibriSpeech	7.45% WER
BERT	7,921 samples/sec	1x H100	NVIDIA H100	H100-SXM-80GB	SQuAD v1.1	90.87% f1
	13,968 samples/sec	8x A100	DGX A100	A100 SXM-80GB	SQuAD v1.1	90.87% f1
	1,757 samples/sec	1x A100	DGX A100	A100 SXM-80GB	SQuAD v1.1	90.87% f1
	247 samples/sec	1x1g.10gb A100	DGX A100	A100 SXM-80GB	SQuAD v1.1	90.87% f1
	12,822 samples/sec	8x A100	Gigabyte G482-Z54	A100 PCIe-80GB	SQuAD v1.1	90.87% f1
DLRM	695,298 samples/sec	1x H100	NVIDIA H100	H100-SXM-80GB	Criteo 1TB Click Logs	80.25% AUC
	2,443,220 samples/sec	8x A100	DGX A100	A100 SXM-80GB	Criteo 1TB Click Logs	80.25% AUC
	314,992 samples/sec	1x A100	DGX A100	A100 SXM-80GB	Criteo 1TB Click Logs	80.25% AUC
	38,995 samples/sec	1x1g.10gb A100	DGX A100	A100 SXM-80GB	Criteo 1TB Click Logs	80.25% AUC
	2,291,310 samples/sec	8x A100	Gigabyte G482-Z54	A100 PCIe-80GB	Criteo 1TB Click Logs	80.25% AUC

Server Scenario - Closed Division

Network	Throughput	GPU	Server	GPU Version	Target Accuracy	MLPerf Server Latency Constraints (ms)	Dataset
ResNet-50 v1.5	58,995 queries/sec	1x H100	NVIDIA H100	H100-SXM-80GB	76.46% Top1	15	ImageNet
	300,064 queries/sec	8x A100	DGX A100	A100 SXM-80GB	76.46% Top1	15	ImageNet
	3,527 queries/sec	1x1g.10gb A100	DGX A100	A100 SXM-80GB	76.46% Top1	15	ImageNet
	236,057 queries/sec	8x A100	Gigabyte G482-Z54	A100 PCIe-80GB	76.46% Top1	15	ImageNet
RetinaNet	848 queries/sec	1x H100	NVIDIA H100	H100-SXM-80GB	0.3755 mAP	100	OpenImages
	4,096 queries/sec	8x A100	DGX A100	A100 SXM-80GB	0.3755 mAP	100	OpenImages
	45 queries/sec	1x1g.10gb A100	DGX A100	A100 SXM-80GB	0.3755 mAP	100	OpenImages
	3,997 queries/sec	8x A100	Gigabyte G482-Z54	A100 PCIe-80GB	0.3755 mAP	100	OpenImages
RNN-T	21,488 queries/sec	1x H100	NVIDIA H100	H100-SXM-80GB	7.45% WER	1,000	LibriSpeech
	104,020 queries/sec	8x A100	DGX A100	A100 SXM-80GB	7.45% WER	1,000	LibriSpeech
	1,347 queries/sec	1x1g.10gb A100	DGX A100	A100 SXM-80GB	7.45% WER	1,000	LibriSpeech
	90,005 queries/sec	8x A100	Gigabyte G482-Z54	A100 PCIe-80GB	7.45% WER	1,000	LibriSpeech
BERT	6,195 queries/sec	1x H100	NVIDIA H100	H100-SXM-80GB	90.87% f1	130	SQuAD v1.1
	12,815 queries/sec	8x A100	DGX A100	A100 SXM-80GB	90.87% f1	130	SQuAD v1.1
	1,572 queries/sec	1x A100	DGX A100	A100 SXM-80GB	90.87% f1	130	SQuAD v1.1
	164 queries/sec	1x1g.10gb A100	DGX A100	A100 SXM-80GB	90.87% f1	130	SQuAD v1.1
	10,795 queries/sec	8x A100	Gigabyte G482-Z54	A100 PCIe-80GB	90.87% f1	130	SQuAD v1.1
DLRM	545,174 queries/sec	1x H100	NVIDIA H100	H100-SXM-80GB	80.25% AUC	30	Criteo 1TB Click Logs
	2,390,910 queries/sec	8x A100	DGX A100	A100 SXM-80GB	80.25% AUC	30	Criteo 1TB Click Logs
	298,565 queries/sec	1x A100	DGX A100	A100 SXM-80GB	80.25% AUC	30	Criteo 1TB Click Logs
	35,991 queries/sec	1x1g.10gb A100	DGX A100	A100 SXM-80GB	80.25% AUC	30	Criteo 1TB Click Logs
	1,326,940 queries/sec	8x A100	Gigabyte G482-Z54	A100 PCIe-80GB	80.25% AUC	30	Criteo 1TB Click Logs

Power Efficiency Offline Scenario - Closed Division

Network	Throughput	Throughput per Watt	GPU	Server	GPU Version	Dataset
ResNet-50 v1.5	288,733 samples/sec	93.68 samples/sec/watt	8x A100	DGX A100	A100 SXM-80GB	ImageNet
	252,721 samples/sec	122.19 samples/sec/watt	8x A100	Gigabyte G482-Z54	A100 PCIe-80GB	ImageNet
RetinaNet	4,122 samples/sec	1.32 samples/sec/watt	8x A100	DGX A100	A100 SXM-80GB	OpenImages
	3,805 samples/sec	1.73 samples/sec/watt	8x A100	Gigabyte G482-Z54	A100 PCIe-80GB	OpenImages
3D-UNet	23 samples/sec	0.008 samples/sec/watt	8x A100	DGX A100	A100 SXM-80GB	KiTS 2019
	19 samples/sec	0.011 samples/sec/watt	8x A100	Gigabyte G482-Z54	A100 PCIe-80GB	KiTS 2019
RNN-T	84,508 samples/sec	27.79 samples/sec/watt	8x A100	DGX A100	A100 SXM-80GB	LibriSpeech
	78,750 samples/sec	38.88 samples/sec/watt	8x A100	Gigabyte G482-Z54	A100 PCIe-80GB	LibriSpeech
BERT	11,152 samples/sec	3.33 samples/sec/watt	8x A100	DGX A100	A100 SXM-80GB	SQuAD v1.1
	11,158 samples/sec	4.37 samples/sec/watt	8x A100	Gigabyte G482-Z54	A100 PCIe-80GB	SQuAD v1.1
DLRM	2,128,420 samples/sec	641.77 samples/sec/watt	8x A100	DGX A100	A100 SXM-80GB	Criteo 1TB Click Logs

Power Efficiency Server Scenario - Closed Division

Network	Throughput	Throughput per Watt	GPU	Server	GPU Version	Dataset
ResNet-50 v1.5	229,055 queries/sec	78.93 queries/sec/watt	8x A100	DGX A100	A100 SXM-80GB	ImageNet
	185,047 queries/sec	87.2 queries/sec/watt	8x A100	Gigabyte G482-Z54	A100 PCIe-80GB	ImageNet
RetinaNet	3,896 queries/sec	1.25 queries/sec/watt	8x A100	DGX A100	A100 SXM-80GB	OpenImages
	2,296 queries/sec	1.21 queries/sec/watt	8x A100	Gigabyte G482-Z54	A100 PCIe-80GB	OpenImages
RNN-T	88,003 queries/sec	25.44 queries/sec/watt	8x A100	DGX A100	A100 SXM-80GB	LibriSpeech
	74,995 queries/sec	33.88 queries/sec/watt	8x A100	Gigabyte G482-Z54	A100 PCIe-80GB	LibriSpeech
BERT	9,995 queries/sec	2.93 queries/sec/watt	8x A100	DGX A100	A100 SXM-80GB	SQuAD v1.1
	7,494 queries/sec	3.45 queries/sec/watt	8x A100	Gigabyte G482-Z54	A100 PCIe-80GB	SQuAD v1.1
DLRM	2,002,080 queries/sec	592.73 queries/sec/watt	8x A100	DGX A100	A100 SXM-80GB	Criteo 1TB Click Logs

MLPerf™ v2.1 Inference Closed: ResNet-50 v1.5, RetinaNet, RNN-T, BERT 99.9% of FP32 accuracy target, 3D U-Net 99.9% of FP32 accuracy target, DLRM 99.9% of FP32 accuracy target: 2.1-0082, 2.1-0084, 2.1-0085, 2.1-0087, 2.1-0088, 2.1-0089, 2.1-0121, 2.1-0122. MLPerf name and logo are trademarks. See https://mlcommons.org/ for more information.
H100 SXM-80GB is a preview submission
BERT-Large sequence length = 384.
DLRM samples refers to 270 pairs/sample average
1x1g.10gb is a notation used to refer to the MIG configuration. In this example, the workload is running on a single MIG slice, with 10GB of memory on a single A100.
For MLPerf™ various scenario data, click here
For MLPerf™ latency constraints, click here

NVIDIA TRITON Inference Server Delivered Comparable Performance to Custom Harness in MLPerf v2.1

NVIDIA landed top performance spots on all MLPerf™ Inference 2.1 tests, the AI-industry’s leading benchmark competition. For inference submissions, we have typically used a custom A100 inference serving harness. This custom harness has been designed and optimized specifically for providing the highest possible inference performance for MLPerf™ workloads, which require running inference on bare metal.

MLPerf™ v2.1 A100 Inference Closed: ResNet-50 v1.5, RetinaNet, BERT 99.9% of FP32 accuracy target, DLRM 99.9% of FP32 accuracy target: 2.1-0088, 2.1-0090. MLPerf name and logo are trademarks. See www.mlcommons.org for more information.

NVIDIA Client Batch Size 1 and 2 Performance with Triton Inference Server

A100 Triton Inference Server Performance

Network	Accelerator	Model Format	Framework Backend	Precision	Model Instances on Triton	Client Batch Size	Dynamic Batch Size (Triton)	Number of Concurrent Client Requests	Latency (ms)	Throughput	Sequence/Input Length	Triton Container Version
BERT Large Inference	A100-SXM4-40GB	tensorrt_plan	TensorRT	Mixed	4	1	1	24	32.705	734 inf/sec	384	22.08-py3
BERT Large Inference	A100-SXM4-80GB	tensorrt_plan	TensorRT	Mixed	4	2	1	24	62.407	770 inf/sec	384	22.08-py3
BERT Large Inference	A100-PCIE-40GB	tensorrt_plan	TensorRT	Mixed	4	1	1	24	38.956	616 inf/sec	384	22.08-py3
BERT Large Inference	A100-PCIE-40GB	tensorrt_plan	TensorRT	Mixed	4	2	1	24	74.039	648 inf/sec	384	22.08-py3
BERT Base Inference	A100-SXM4-80GB	tensorrt_plan	TensorRT	Mixed	4	1	1	24	4.194	5,721 inf/sec	128	22.08-py3
BERT Base Inference	A100-SXM4-40GB	tensorrt_plan	TensorRT	Mixed	4	2	1	24	7.009	6,848 inf/sec	128	22.08-py3
BERT Base Inference	A100-PCIE-40GB	tensorrt_plan	TensorRT	Mixed	4	1	1	24	4.819	4,979 inf/sec	128	22.08-py3
BERT Base Inference	A100-PCIE-40GB	tensorrt_plan	TensorRT	Mixed	4	2	1	24	8.186	5,862 inf/sec	128	22.08-py3
DLRM Inference	A100-SXM4-40GB	pytorch_libtorch	PyTorch	Mixed	2	1	65,536	30	2.351	12,756 inf/sec	-	22.08-py3
DLRM Inference	A100-SXM4-40GB	pytorch_libtorch	PyTorch	Mixed	2	2	65,536	28	2.223	25,185 inf/sec	-	22.08-py3
DLRM Inference	A100-PCIE-40GB	pytorch_libtorch	PyTorch	Mixed	4	1	65,536	30	2.316	12,946 inf/sec	-	22.08-py3
DLRM Inference	A100-PCIE-40GB	pytorch_libtorch	PyTorch	Mixed	4	2	65,536	30	2.152	27,875 inf/sec	-	22.07-py3

A30 Triton Inference Server Performance

Network	Accelerator	Model Format	Framework Backend	Precision	Model Instances on Triton	Client Batch Size	Dynamic Batch Size (Triton)	Number of Concurrent Client Requests	Latency (ms)	Throughput	Sequence/Input Length	Triton Container Version
BERT Large Inference	A30	tensorrt_plan	TensorRT	Mixed	4	1	1	24	68.117	352 inf/sec	384	22.08-py3
BERT Large Inference	A30	tensorrt_plan	TensorRT	Mixed	2	2	1	16	87.427	366 inf/sec	384	22.08-py3
BERT Base Inference	A30	tensorrt_plan	TensorRT	Mixed	4	1	1	24	7.679	3,125 inf/sec	128	22.08-py3
BERT Base Inference	A30	tensorrt_plan	TensorRT	Mixed	2	2	1	16	9.502	3,367 inf/sec	128	22.08-py3

A10 Triton Inference Server Performance

Network	Accelerator	Model Format	Framework Backend	Precision	Model Instances on Triton	Client Batch Size	Dynamic Batch Size (Triton)	Number of Concurrent Client Requests	Latency (ms)	Throughput	Sequence/Input Length	Triton Container Version
BERT Large Inference	A10	tensorrt_plan	TensorRT	Mixed	4	1	1	24	107.907	222 inf/sec	384	22.08-py3
BERT Large Inference	A10	tensorrt_plan	TensorRT	Mixed	2	2	1	24	211.233	228 inf/sec	384	22.08-py3
BERT Base Inference	A10	tensorrt_plan	TensorRT	Mixed	2	1	1	24	11.078	2,166 inf/sec	128	22.08-py3
BERT Base Inference	A10	tensorrt_plan	TensorRT	Mixed	4	2	1	24	21.261	2,257 inf/sec	128	22.08-py3

T4 Triton Inference Server Performance +

Network	Accelerator	Model Format	Framework Backend	Precision	Model Instances on Triton	Client Batch Size	Dynamic Batch Size (Triton)	Number of Concurrent Client Requests	Latency (ms)	Throughput	Sequence/Input Length	Triton Container Version
BERT Large Inference	NVIDIA T4	tensorrt_plan	TensorRT	Mixed	1	1	1	8	92.264	87 inf/sec	384	22.08-py3
BERT Large Inference	NVIDIA T4	tensorrt_plan	TensorRT	Mixed	1	2	1	8	183.106	87 inf/sec	384	22.08-py3
BERT Base Inference	NVIDIA T4	tensorrt_plan	TensorRT	Mixed	1	1	1	24	26.479	906 inf/sec	128	22.08-py3
BERT Base Inference	NVIDIA T4	tensorrt_plan	TensorRT	Mixed	1	2	1	20	43.281	924 inf/sec	128	22.08-py3

V100 Triton Inference Server Performance +

Network	Accelerator	Model Format	Framework Backend	Precision	Model Instances on Triton	Client Batch Size	Dynamic Batch Size (Triton)	Number of Concurrent Client Requests	Latency (ms)	Throughput	Sequence/Input Length	Triton Container Version
BERT Large Inference	V100 SXM2-32GB	tensorrt_plan	TensorRT	Mixed	2	1	1	24	96.163	249 inf/sec	384	22.08-py3
BERT Large Inference	V100 SXM2-32GB	tensorrt_plan	TensorRT	Mixed	4	2	1	24	189.991	253 inf/sec	384	22.08-py3
BERT Base Inference	V100 SXM2-32GB	tensorrt_plan	TensorRT	Mixed	4	1	1	24	12.567	1,910 inf/sec	128	22.08-py3
BERT Base Inference	V100 SXM2-32GB	tensorrt_plan	TensorRT	Mixed	4	2	1	24	20.907	2,295 inf/sec	128	22.08-py3
DLRM Inference	V100-SXM2-32GB	pytorch_libtorch	PyTorch	Mixed	2	1	65,536	30	3.358	8,931 inf/sec	-	22.08-py3
DLRM Inference	V100-SXM2-32GB	pytorch_libtorch	PyTorch	Mixed	2	2	65,536	30	3.532	16,983 inf/sec	-	22.08-py3

Inference Performance of NVIDIA A100, A40, A30, A10, A2, T4 and V100

Benchmarks are reproducible by following links to the NGC catalog scripts

Inference Image Classification on CNNs with TensorRT

ResNet-50 v1.5 Throughput

ResNet-50 v1.5 Power Efficiency

A100 Full Chip Inference Performance

Network	Batch Size	Full Chip Throughput	Efficiency	Latency (ms)	GPU	Server	Container	Precision	Dataset	Framework	GPU Version
ResNet-50	8	11,827 images/sec	64 images/sec/watt	0.68	1x A100	DGX A100	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	A100-SXM4-40GB
	128	29,514 images/sec	86 images/sec/watt	4.34	1x A100	DGX A100	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	A100-SXM4-40GB
	219	31,828 images/sec	- images/sec/watt	6.88	1x A100	DGX A100	22.08-py3	INT8	Synthetic	TensorRT 8.4.2	A100-SXM4-80GB
ResNet-50v1.5	8	11,587 images/sec	56 images/sec/watt	0.69	1x A100	DGX A100	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	A100-SXM4-80GB
	128	29,927 images/sec	77 images/sec/watt	4.28	1x A100	DGX A100	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	A100-SXM4-80GB
	213	30,810 images/sec	- images/sec/watt	6.91	1x A100	DGX A100	22.08-py3	INT8	Synthetic	TensorRT 8.4.2	A100-SXM4-80GB
BERT-BASE	1	For Batch Size 1, please refer to Triton Inference Server page
	2	For Batch Size 2, please refer to Triton Inference Server page
	8	7,315 sequences/sec	27 sequences/sec/watt	1.09	1x A100	DGX A100	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	A100-SXM4-80GB
	128	15,191 sequences/sec	38 sequences/sec/watt	8.43	1x A100	DGX A100	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	A100-SXM4-40GB
BERT-LARGE	1	For Batch Size 1, please refer to Triton Inference Server page
	2	For Batch Size 2, please refer to Triton Inference Server page
	8	2,682 sequences/sec	9 sequences/sec/watt	2.98	1x A100	DGX A100	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	A100-SXM4-80GB
	128	4,964 sequences/sec	13 sequences/sec/watt	25.78	1x A100	DGX A100	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	A100-SXM4-40GB
EfficientNet-B0	8	9,152 images/sec	65 images/sec/watt	0.87	1x A100	DGX A100	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	A100-SXM4-40GB
	128	30,289 images/sec	96 images/sec/watt	4.23	1x A100	DGX A100	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	A100-SXM4-80GB
EfficientNet-B4	8	2,586 images/sec	12 images/sec/watt	3.09	1x A100	DGX A100	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	A100-SXM4-40GB
	128	4,394 images/sec	13 images/sec/watt	29.13	1x A100	DGX A100	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	A100-SXM4-40GB

Containers with a hyphen indicates a pre-release container | Servers with a hyphen indicates a pre-production server
BERT-Large: Sequence Length = 128

A100 1/7 MIG Inference Performance

Network	Batch Size	1/7 MIG Throughput	Efficiency	Latency (ms)	GPU	Server	Container	Precision	Dataset	Framework	GPU Version
ResNet-50	8	3,723 images/sec	34 images/sec/watt	2.15	1x A100	DGX A100	22.08-py3	INT8	Synthetic	TensorRT 8.4.2	A100-SXM4-80GB
	30	4,309 images/sec	- images/sec/watt	6.96	1x A100	DGX A100	22.08-py3	INT8	Synthetic	TensorRT 8.4.2	A100-SXM4-80GB
	128	4,671 images/sec	37 images/sec/watt	27.4	1x A100	DGX A100	22.08-py3	INT8	Synthetic	TensorRT 8.4.2	A100-SXM4-80GB
ResNet-50v1.5	8	3,640 images/sec	32 images/sec/watt	2.2	1x A100	DGX A100	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	A100-SXM4-80GB
	28	4,125 images/sec	- images/sec/watt	6.79	1x A100	DGX A100	22.08-py3	INT8	Synthetic	TensorRT 8.4.2	A100-SXM4-80GB
	128	4,532 images/sec	37 images/sec/watt	28.24	1x A100	DGX A100	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	A100-SXM4-80GB
BERT-BASE	8	1,886 sequences/sec	15 sequences/sec/watt	4.24	1x A100	DGX A100	22.08-py3	INT8	Synthetic	TensorRT 8.4.2	A100-SXM4-80GB
	128	2,342 sequences/sec	17 sequences/sec/watt	54.65	1x A100	DGX A100	22.08-py3	INT8	Synthetic	TensorRT 8.4.2	A100-SXM4-80GB
BERT-LARGE	8	618 sequences/sec	5 sequences/sec/watt	12.95	1x A100	DGX A100	22.08-py3	INT8	Synthetic	TensorRT 8.4.2	A100-SXM4-80GB
	128	744 sequences/sec	5 sequences/sec/watt	172.08	1x A100	DGX A100	22.08-py3	INT8	Synthetic	TensorRT 8.4.2	A100-SXM4-80GB

Containers with a hyphen indicates a pre-release container | Servers with a hyphen indicates a pre-production server
BERT-Large: Sequence Length = 128

A100 7 MIG Inference Performance

Network	Batch Size	7 MIG Throughput	Efficiency	Latency (ms)	GPU	Server	Container	Precision	Dataset	Framework	GPU Version
ResNet-50	8	25,682 images/sec	79 images/sec/watt	2.18	1x A100	DGX A100	22.08-py3	INT8	Synthetic	TensorRT 8.4.2	A100-SXM4-80GB
	29	29,917 images/sec	- images/sec/watt	6.79	1x A100	DGX A100	22.08-py3	INT8	Synthetic	TensorRT 8.4.2	A100-SXM4-80GB
	128	32,531 images/sec	88 images/sec/watt	27.62	1x A100	DGX A100	22.08-py3	INT8	Synthetic	TensorRT 8.4.2	A100-SXM4-80GB
ResNet-50v1.5	8	25,056 images/sec	77 images/sec/watt	2.24	1x A100	DGX A100	22.08-py3	INT8	Synthetic	TensorRT 8.4.2	A100-SXM4-80GB
	28	28,819 images/sec	- images/sec/watt	6.8	1x A100	DGX A100	22.08-py3	INT8	Synthetic	TensorRT 8.4.2	A100-SXM4-80GB
	128	31,490 images/sec	82 images/sec/watt	28.54	1x A100	DGX A100	22.08-py3	INT8	Synthetic	TensorRT 8.4.2	A100-SXM4-80GB
BERT-BASE	8	13,095 sequences/sec	34 sequences/sec/watt	4.29	1x A100	DGX A100	22.08-py3	INT8	Synthetic	TensorRT 8.4.2	A100-SXM4-80GB
	128	15,342 sequences/sec	40 sequences/sec/watt	58.53	1x A100	DGX A100	22.08-py3	INT8	Synthetic	TensorRT 8.4.2	A100-SXM4-80GB
BERT-LARGE	8	4,214 sequences/sec	11 sequences/sec/watt	13.31	1x A100	DGX A100	22.08-py3	INT8	Synthetic	TensorRT 8.4.2	A100-SXM4-80GB
	128	4,812 sequences/sec	12 sequences/sec/watt	186.61	1x A100	DGX A100	22.08-py3	INT8	Synthetic	TensorRT 8.4.2	A100-SXM4-80GB

Containers with a hyphen indicates a pre-release container | Servers with a hyphen indicates a pre-production server
BERT-Large: Sequence Length = 128

A40 Inference Performance

Network	Batch Size	Throughput	Efficiency	Latency (ms)	GPU	Server	Container	Precision	Dataset	Framework	GPU Version
ResNet-50	8	9,917 images/sec	38 images/sec/watt	0.81	1x A40	GIGABYTE G482-Z52-00	22.08-py3	INT8	Synthetic	TensorRT 8.4.2	A40
	106	15,867 images/sec	- images/sec/watt	6.68	1x A40	GIGABYTE G482-Z52-00	22.08-py3	INT8	Synthetic	TensorRT 8.4.2	A40
	128	15,867 images/sec	53 images/sec/watt	8.07	1x A40	GIGABYTE G482-Z52-00	22.08-py3	INT8	Synthetic	TensorRT 8.4.2	A40
ResNet-50v1.5	8	9,686 images/sec	37 images/sec/watt	0.83	1x A40	GIGABYTE G482-Z52-00	22.08-py3	INT8	Synthetic	TensorRT 8.4.2	A40
	101	15,016 images/sec	- images/sec/watt	6.73	1x A40	GIGABYTE G482-Z52-00	22.08-py3	INT8	Synthetic	TensorRT 8.4.2	A40
	128	15,171 images/sec	51 images/sec/watt	8.44	1x A40	GIGABYTE G482-Z52-00	22.08-py3	INT8	Synthetic	TensorRT 8.4.2	A40
BERT-BASE	8	5,609 sequences/sec	19 sequences/sec/watt	1.43	1x A40	GIGABYTE G482-Z52-00	22.08-py3	INT8	Synthetic	TensorRT 8.4.2	A40
	128	7,753 sequences/sec	26 sequences/sec/watt	16.51	1x A40	GIGABYTE G482-Z52-00	22.08-py3	INT8	Synthetic	TensorRT 8.4.2	A40
BERT-LARGE	8	1,724 sequences/sec	6 sequences/sec/watt	4.64	1x A40	GIGABYTE G482-Z52-00	22.08-py3	INT8	Synthetic	TensorRT 8.4.2	A40
	128	2,386 sequences/sec	8 sequences/sec/watt	53.65	1x A40	GIGABYTE G482-Z52-00	22.08-py3	INT8	Synthetic	TensorRT 8.4.2	A40
EfficientNet-B0	8	9,194 images/sec	50 images/sec/watt	0.87	1x A40	GIGABYTE G482-Z52-00	22.08-py3	INT8	Synthetic	TensorRT 8.4.2	A40
	128	19,286 images/sec	65 images/sec/watt	6.64	1x A40	GIGABYTE G482-Z52-00	22.08-py3	INT8	Synthetic	TensorRT 8.4.2	A40
EfficientNet-B4	8	1,950 images/sec	7 images/sec/watt	4.1	1x A40	GIGABYTE G482-Z52-00	22.08-py3	INT8	Synthetic	TensorRT 8.4.2	A40
	128	2,648 images/sec	9 images/sec/watt	48.34	1x A40	GIGABYTE G482-Z52-00	22.08-py3	INT8	Synthetic	TensorRT 8.4.2	A40

Sequence length=128 for BERT-BASE and BERT-LARGE | Efficiency based on board power
Containers with a hyphen indicates a pre-release container

A30 Inference Performance

Network	Batch Size	Throughput	Efficiency	Latency (ms)	GPU	Server	Container	Precision	Dataset	Framework	GPU Version
ResNet-50	8	8,915 images/sec	71 images/sec/watt	0.9	1x A30	GIGABYTE G482-Z52-00	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	A30
	105	15,657 images/sec	- images/sec/watt	6.71	1x A30	GIGABYTE G482-Z52-00	22.08-py3	INT8	Synthetic	TensorRT 8.4.2	A30
	128	16,083 images/sec	98 images/sec/watt	7.96	1x A30	GIGABYTE G482-Z52-00	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	A30
ResNet-50v1.5	8	8,737 images/sec	68 images/sec/watt	0.92	1x A30	GIGABYTE G482-Z52-00	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	A30
	101	15,071 images/sec	- images/sec/watt	6.7	1x A30	GIGABYTE G482-Z52-00	22.08-py3	INT8	Synthetic	TensorRT 8.4.2	A30
	128	15,582 images/sec	95 images/sec/watt	8.21	1x A30	GIGABYTE G482-Z52-00	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	A30
BERT-BASE	1	For Batch Size 1, please refer to Triton Inference Server page
	2	For Batch Size 2, please refer to Triton Inference Server page
	8	5,146 sequences/sec	31 sequences/sec/watt	1.55	1x A30	GIGABYTE G482-Z52-00	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	A30
	128	7,327 sequences/sec	45 sequences/sec/watt	17.47	1x A30	GIGABYTE G482-Z52-00	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	A30
BERT-LARGE	1	For Batch Size 1, please refer to Triton Inference Server page
	2	For Batch Size 2, please refer to Triton Inference Server page
	8	1,776 sequences/sec	12 sequences/sec/watt	4.5	1x A30	GIGABYTE G482-Z52-00	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	A30
	128	2,456 sequences/sec	15 sequences/sec/watt	52.12	1x A30	GIGABYTE G482-Z52-00	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	A30
EfficientNet-B0	8	7,595 images/sec	74 images/sec/watt	1.05	1x A30	GIGABYTE G482-Z52-00	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	A30
	128	16,798 images/sec	102 images/sec/watt	7.62	1x A30	GIGABYTE G482-Z52-00	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	A30
EfficientNet-B4	8	1,718 images/sec	12 images/sec/watt	4.66	1x A30	GIGABYTE G482-Z52-00	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	A30
	128	2,357 images/sec	14 images/sec/watt	54.31	1x A30	GIGABYTE G482-Z52-00	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	A30

Sequence length=128 for BERT-BASE and BERT-LARGE | Efficiency based on board power
Containers with a hyphen indicates a pre-release container

A30 1/4 MIG Inference Performance

Network	Batch Size	Throughput	Efficiency	Latency (ms)	GPU	Server	Container	Precision	Dataset	Framework	GPU Version
ResNet-50	8	3,625 images/sec	44 images/sec/watt	2.21	1x A30	GIGABYTE G482-Z52-00	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	A30
	29	4,253 images/sec	- images/sec/watt	6.82	1x A30	GIGABYTE G482-Z52-00	22.08-py3	INT8	Synthetic	TensorRT 8.4.2	A30
	128	4,605 images/sec	54 images/sec/watt	27.79	1x A30	GIGABYTE G482-Z52-00	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	A30
ResNet-50v1.5	8	3,541 images/sec	44 images/sec/watt	2.26	1x A30	GIGABYTE G482-Z52-00	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	A30
	27	4,143 images/sec	- images/sec/watt	6.52	1x A30	GIGABYTE G482-Z52-00	22.08-py3	INT8	Synthetic	TensorRT 8.4.2	A30
	128	4,458 images/sec	51 images/sec/watt	28.71	1x A30	GIGABYTE G482-Z52-00	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	A30
BERT-BASE	8	1,873 sequences/sec	21 sequences/sec/watt	4.27	1x A30	GIGABYTE G482-Z52-00	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	A30
	128	2,276 sequences/sec	23 sequences/sec/watt	56.23	1x A30	GIGABYTE G482-Z52-00	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	A30
BERT-LARGE	8	601 sequences/sec	7 sequences/sec/watt	13.3	1x A30	GIGABYTE G482-Z52-00	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	A30
	128	742 sequences/sec	7 sequences/sec/watt	172.55	1x A30	GIGABYTE G482-Z52-00	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	A30

Sequence length=128 for BERT-BASE and BERT-LARGE | Efficiency based on board power
Containers with a hyphen indicates a pre-release container

A30 4 MIG Inference Performance

Network	Batch Size	Throughput	Efficiency	Latency (ms)	GPU	Server	Container	Precision	Dataset	Framework	GPU Version
ResNet-50	8	13,975 images/sec	85 images/sec/watt	2.29	1x A30	GIGABYTE G482-Z52-00	22.08-py3	INT8	Synthetic	TensorRT 8.4.2	A30
	27	16,175 images/sec	- images/sec/watt	6.7	1x A30	GIGABYTE G482-Z52-00	22.08-py3	INT8	Synthetic	TensorRT 8.4.2	A30
	128	17,191 images/sec	104 images/sec/watt	29.88	1x A30	GIGABYTE G482-Z52-00	22.08-py3	INT8	Synthetic	TensorRT 8.4.2	A30
ResNet-50v1.5	8	13,516 images/sec	82 images/sec/watt	2.38	1x A30	GIGABYTE G482-Z52-00	22.08-py3	INT8	Synthetic	TensorRT 8.4.2	A30
	26	15,607 images/sec	- images/sec/watt	6.7	1x A30	GIGABYTE G482-Z52-00	22.08-py3	INT8	Synthetic	TensorRT 8.4.2	A30
	128	16,658 images/sec	101 images/sec/watt	30.9	1x A30	GIGABYTE G482-Z52-00	22.08-py3	INT8	Synthetic	TensorRT 8.4.2	A30
BERT-BASE	8	6,894 sequences/sec	42 sequences/sec/watt	4.66	1x A30	GIGABYTE G482-Z52-00	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	A30
	128	7,732 sequences/sec	47 sequences/sec/watt	66.45	1x A30	GIGABYTE G482-Z52-00	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	A30
BERT-LARGE	8	2,187 sequences/sec	13 sequences/sec/watt	14.68	1x A30	GIGABYTE G482-Z52-00	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	A30
	128	2,458 sequences/sec	15 sequences/sec/watt	208.61	1x A30	GIGABYTE G482-Z52-00	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	A30

Sequence length=128 for BERT-BASE and BERT-LARGE | Efficiency based on board power
Containers with a hyphen indicates a pre-release container

A10 Inference Performance

Network	Batch Size	Throughput	Efficiency	Latency (ms)	GPU	Server	Container	Precision	Dataset	Framework	GPU Version
ResNet-50	8	7,774 images/sec	52 images/sec/watt	1.03	1x A10	GIGABYTE G482-Z52-00	22.08-py3	INT8	Synthetic	TensorRT 8.4.2	A10
	71	10,857 images/sec	- images/sec/watt	6.63	1x A10	GIGABYTE G482-Z52-00	22.08-py3	INT8	Synthetic	TensorRT 8.4.2	A10
	128	11,284 images/sec	77 images/sec/watt	11.34	1x A10	GIGABYTE G482-Z52-00	22.08-py3	INT8	Synthetic	TensorRT 8.4.2	A10
ResNet-50v1.5	8	7,640 images/sec	51 images/sec/watt	1.05	1x A10	GIGABYTE G482-Z52-00	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	A10
	128	10,668 images/sec	71 images/sec/watt	12	1x A10	GIGABYTE G482-Z52-00	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	A10
BERT-BASE	8	4,154 sequences/sec	28 sequences/sec/watt	1.93	1x A10	GIGABYTE G482-Z52-00	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	A10
	128	5,004 sequences/sec	33 sequences/sec/watt	25.58	1x A10	GIGABYTE G482-Z52-00	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	A10
BERT-LARGE	8	1,251 sequences/sec	8 sequences/sec/watt	6.4	1x A10	GIGABYTE G482-Z52-00	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	A10
	128	1,537 sequences/sec	11 sequences/sec/watt	83.28	1x A10	GIGABYTE G482-Z52-00	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	A10
EfficientNet-B0	8	8,416 images/sec	56 images/sec/watt	0.95	1x A10	GIGABYTE G482-Z52-00	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	A10
	128	13,900 images/sec	93 images/sec/watt	9.21	1x A10	GIGABYTE G482-Z52-00	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	A10
EfficientNet-B4	8	1,531 images/sec	10 images/sec/watt	5.23	1x A10	GIGABYTE G482-Z52-00	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	A10
	128	1,836 images/sec	12 images/sec/watt	69.71	1x A10	GIGABYTE G482-Z52-00	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	A10

Sequence length=128 for BERT-BASE and BERT-LARGE | Efficiency based on board power
Containers with a hyphen indicates a pre-release container

A2 Inference Performance

Network	Batch Size	Throughput	Efficiency	Latency (ms)	GPU	Server	Container	Precision	Dataset	Framework	GPU Version
ResNet-50	8	2,610 images/sec	43 images/sec/watt	3.06	1x A2	GIGABYTE MZ52-G41-00	22.08-py3	INT8	Synthetic	TensorRT 8.4.2	A2
	19	2,901 images/sec	- images/sec/watt	6.55	1x A2	GIGABYTE MZ52-G41-00	22.08-py3	INT8	Synthetic	TensorRT 8.4.2	A2
	128	3,027 images/sec	51 images/sec/watt	42.28	1x A2	GIGABYTE MZ52-G41-00	22.08-py3	INT8	Synthetic	TensorRT 8.4.2	A2
ResNet-50v1.5	8	2,520 images/sec	42 images/sec/watt	3.17	1x A2	GIGABYTE MZ52-G41-00	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	A2
	18	2,761 images/sec	- images/sec/watt	6.52	1x A2	GIGABYTE MZ52-G41-00	22.08-py3	INT8	Synthetic	TensorRT 8.4.2	A2
	128	2,939 images/sec	49 images/sec/watt	43.55	1x A2	GIGABYTE MZ52-G41-00	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	A2
BERT-BASE	8	1,131 sequences/sec	19 sequences/sec/watt	7.07	1x A2	GIGABYTE MZ52-G41-00	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	A2
	128	1,194 sequences/sec	20 sequences/sec/watt	107.17	1x A2	GIGABYTE MZ52-G41-00	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	A2
BERT-LARGE	8	329 sequences/sec	5 sequences/sec/watt	24.35	1x A2	GIGABYTE MZ52-G41-00	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	A2
	128	364 sequences/sec	6 sequences/sec/watt	351.87	1x A2	GIGABYTE MZ52-G41-00	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	A2
EfficientNet-B0	8	3,054 images/sec	59 images/sec/watt	2.62	1x A2	GIGABYTE MZ52-G41-00	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	A2
	128	3,924 images/sec	66 images/sec/watt	32.62	1x A2	GIGABYTE MZ52-G41-00	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	A2
EfficientNet-B4	8	469 images/sec	8 images/sec/watt	17.04	1x A2	GIGABYTE MZ52-G41-00	22.08-py3	INT8	Synthetic	TensorRT 8.4.2	A2
	128	519 images/sec	9 images/sec/watt	246.49	1x A2	GIGABYTE MZ52-G41-00	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	A2

Sequence length=128 for BERT-BASE and BERT-LARGE | Efficiency based on board power

T4 Inference Performance +

Network	Batch Size	Throughput	Efficiency	Latency (ms)	GPU	Server	Container	Precision	Dataset	Framework	GPU Version
ResNet-50	8	3,901 images/sec	56 images/sec/watt	2.05	1x T4	Supermicro SYS-4029GP-TRT	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	NVIDIA T4
	30	4,562 images/sec	- images/sec/watt	6.58	1x T4	Supermicro SYS-1029GQ-TRT	22.08-py3	INT8	Synthetic	TensorRT 8.4.2	NVIDIA T4
	128	4,966 images/sec	71 images/sec/watt	25.78	1x T4	Supermicro SYS-4029GP-TRT	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	NVIDIA T4
ResNet-50v1.5	8	3,639 images/sec	52 images/sec/watt	2.2	1x T4	Supermicro SYS-4029GP-TRT	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	NVIDIA T4
	27	4,213 images/sec	- images/sec/watt	6.65	1x T4	Supermicro SYS-1029GQ-TRT	22.08-py3	INT8	Synthetic	TensorRT 8.4.2	NVIDIA T4
	128	4,918 images/sec	70 images/sec/watt	26.03	1x T4	Supermicro SYS-4029GP-TRT	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	NVIDIA T4
BERT-BASE	1	For Batch Size 1, please refer to Triton Inference Server page
	2	For Batch Size 2, please refer to Triton Inference Server page
	8	1,681 sequences/sec	24 sequences/sec/watt	4.76	1x T4	Supermicro SYS-4029GP-TRT	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	NVIDIA T4
	128	1,856 sequences/sec	27 sequences/sec/watt	69	1x T4	Supermicro SYS-4029GP-TRT	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	NVIDIA T4
BERT-LARGE	1	For Batch Size 1, please refer to Triton Inference Server page
	2	For Batch Size 2, please refer to Triton Inference Server page
	8	559 sequences/sec	8 sequences/sec/watt	14.3	1x T4	Supermicro SYS-4029GP-TRT	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	NVIDIA T4
	128	535 sequences/sec	8 sequences/sec/watt	239.34	1x T4	Supermicro SYS-4029GP-TRT	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	NVIDIA T4
EfficientNet-B0	8	4,812 images/sec	69 images/sec/watt	1.66	1x T4	Supermicro SYS-4029GP-TRT	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	NVIDIA T4
	128	6,313 images/sec	91 images/sec/watt	20.28	1x T4	Supermicro SYS-4029GP-TRT	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	NVIDIA T4
EfficientNet-B4	8	793 images/sec	11 images/sec/watt	10.08	1x T4	Supermicro SYS-4029GP-TRT	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	NVIDIA T4
	128	852 images/sec	12 images/sec/watt	150.19	1x T4	Supermicro SYS-4029GP-TRT	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	NVIDIA T4

Sequence length=128 for BERT-BASE and BERT-LARGE | Efficiency based on board power
Containers with a hyphen indicates a pre-release container

V100 Inference Performance +

Network	Batch Size	Throughput	Efficiency	Latency (ms)	GPU	Server	Container	Precision	Dataset	Framework	GPU Version
ResNet-50	8	4,426 images/sec	16 images/sec/watt	1.81	1x V100	DGX-2	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	V100-SXM3-32GB
	128	7,889 images/sec	23 images/sec/watt	16.23	1x V100	DGX-2	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	V100-SXM3-32GB
ResNet-50v1.5	8	4,320 images/sec	14 images/sec/watt	1.85	1x V100	DGX-2	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	V100-SXM3-32GB
	128	7,502 images/sec	22 images/sec/watt	17.06	1x V100	DGX-2	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	V100-SXM3-32GB
BERT-BASE	1	For Batch Size 1, please refer to Triton Inference Server page
	2	For Batch Size 2, please refer to Triton Inference Server page
	8	2,369 sequences/sec	7 sequences/sec/watt	3.38	1x V100	DGX-2	22.09-py3	Mixed	Synthetic	TensorRT 8.5.0	V100-SXM3-32GB
	128	3,186 sequences/sec	10 sequences/sec/watt	40.17	1x V100	DGX-2	22.09-py3	Mixed	Synthetic	TensorRT 8.5.0	V100-SXM3-32GB
BERT-LARGE	1	For Batch Size 1, please refer to Triton Inference Server page
	2	For Batch Size 2, please refer to Triton Inference Server page
	8	802 sequences/sec	2 sequences/sec/watt	9.97	1x V100	DGX-2	22.09-py3	Mixed	Synthetic	TensorRT 8.5.0	V100-SXM3-32GB
	128	970 sequences/sec	3 sequences/sec/watt	131.9	1x V100	DGX-2	22.09-py3	Mixed	Synthetic	TensorRT 8.5.0	V100-SXM3-32GB
EfficientNet-B0	8	4,567 images/sec	22 images/sec/watt	1.75	1x V100	DGX-2	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	V100-SXM3-32GB
	128	9,390 images/sec	28 images/sec/watt	13.63	1x V100	DGX-2	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	V100-SXM3-32GB
EfficientNet-B4	8	931 images/sec	3 images/sec/watt	8.59	1x V100	DGX-2	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	V100-SXM3-32GB
	128	1,290 images/sec	4 images/sec/watt	99.21	1x V100	DGX-2	22.09-py3	INT8	Synthetic	TensorRT 8.5.0	V100-SXM3-32GB

Sequence length=128 for BERT-BASE and BERT-LARGE | Efficiency based on board power
Containers with a hyphen indicates a pre-release container

Inference Performance of NVIDIA GPU on Cloud

Benchmarks are reproducible by following links to the NGC catalog scripts

A100 Inference Performance on Cloud

Network	Batch Size	Throughput	Efficiency	Latency (ms)	GPU	Server	Container	Precision	Dataset	Framework	GPU Version
ResNet-50v1.5	8	11,495 images/sec	- images/sec/watt	0.7	1x A100	GCP A2-HIGHGPU-1G	22.07-py3	INT8	Synthetic	TensorRT 8.4.1	A100-SXM4-40GB
	128	28,222 images/sec	- images/sec/watt	4.54	1x A100	GCP A2-HIGHGPU-1G	22.07-py3	INT8	Synthetic	TensorRT 8.4.1	A100-SXM4-40GB
	8	11,288 images/sec	- images/sec/watt	0.71	1x A100	AWS EC2 p4d.24xlarge	22.07-py3	INT8	Synthetic	TensorRT 8.4.1	A100-SXM4-40GB
	128	28,211 images/sec	- images/sec/watt	4.54	1x A100	AWS EC2 p4d.24xlarge	22.07-py3	INT8	Synthetic	TensorRT 8.4.1	A100-SXM4-40GB
	8	11,334 images/sec	- images/sec/watt	0.71	1x A100	Azure Standard_ND96amsr_A100_v4	22.08-py3	INT8	Synthetic	-	A100-SXM4-80GB
	128	29,613 images/sec	- images/sec/watt	4.32	1x A100	Azure Standard_ND96amsr_A100_v4	22.08-py3	INT8	Synthetic	-	A100-SXM4-40GB
BERT-LARGE	8	2,569 images/sec	- images/sec/watt	3.11	1x A100	AWS EC2 p4d.24xlarge	22.07-py3	INT8	Synthetic	TensorRT 8.4.1	A100-SXM4-40GB
	128	5,008 images/sec	- images/sec/watt	25.56	1x A100	AWS EC2 p4d.24xlarge	22.07-py3	INT8	Synthetic	TensorRT 8.4.1	A100-SXM4-40GB
	8	2,698 images/sec	- images/sec/watt	2.96	1x A100	Azure Standard_ND96amsr_A100_v4	22.08-py3	INT8	Synthetic	-	A100-SXM4-80GB
	128	4,907 images/sec	- images/sec/watt	26.09	1x A100	Azure Standard_ND96amsr_A100_v4	22.08-py3	INT8	Synthetic	-	A100-SXM4-80GB

BERT-Large: Sequence Length = 128

T4 Inference Performance on Cloud +

Network	Batch Size	Throughput	Efficiency	Latency (ms)	GPU	Server	Container	Precision	Dataset	Framework	GPU Version
ResNet-50v1.5	8	3,351 images/sec	- images/sec/watt	2.39	1x T4	GCP N1-HIGHMEM-8	22.07-py3	INT8	Synthetic	TensorRT 8.4.1	NVIDIA T4
	128	3,885 images/sec	- images/sec/watt	32.95	1x T4	GCP N1-HIGHMEM-8	22.07-py3	INT8	Synthetic	TensorRT 8.4.1	NVIDIA T4
ResNet-50v1.5	8	3,308 images/sec	- images/sec/watt	2.42	1x T4	AWS EC2 g4dn.4xlarge	22.06-py3	INT8	Synthetic	TensorRT 8.2.5	NVIDIA T4
	128	4,143 images/sec	- images/sec/watt	30.89	1x T4	AWS EC2 g4dn.4xlarge	22.06-py3	INT8	Synthetic	TensorRT 8.2.5	NVIDIA T4
BERT-LARGE	8	475 images/sec	- images/sec/watt	16.83	1x T4	GCP N1-HIGHMEM-8	22.07-py3	INT8	Synthetic	TensorRT 8.4.1	NVIDIA T4
	128	430 images/sec	- images/sec/watt	297.77	1x T4	GCP N1-HIGHMEM-8	22.07-py3	INT8	Synthetic	TensorRT 8.4.1	NVIDIA T4

V100 Inference Performance on Cloud +

Network	Batch Size	Throughput	Efficiency	Latency (ms)	GPU	Server	Container	Precision	Dataset	Framework	GPU Version
ResNet-50v1.5	8	4,257 images/sec	- images/sec/watt	1.88	1x V100	GCP N1-HIGHMEM-8	22.07-py3	INT8	Synthetic	TensorRT 8.4.1	V100-SXM2-16GB
	128	7,360 images/sec	- images/sec/watt	17.39	1x V100	GCP N1-HIGHMEM-8	22.07-py3	INT8	Synthetic	TensorRT 8.4.1	V100-SXM2-16GB
	8	3,824 images/sec	- images/sec/watt	2.09	1x V100	Azure Standard_NC6s_v3	22.05-py3	INT8	Synthetic	TensorRT 8.2.3	V100-SXM2-16GB
	128	7,043 images/sec	- images/sec/watt	18.17	1x V100	Azure Standard_NC6s_v3	22.05-py3	INT8	Synthetic	TensorRT 8.2.3	V100-SXM2-16GB
BERT-LARGE	8	684 images/sec	- images/sec/watt	11.69	1x V100	GCP N1-HIGHMEM-8	22.07-py3	INT8	Synthetic	TensorRT 8.4.1	V100-SXM2-16GB
	128	915 images/sec	- images/sec/watt	139.89	1x V100	GCP N1-HIGHMEM-8	22.07-py3	INT8	Synthetic	TensorRT 8.4.1	V100-SXM2-16GB

Automatic Speech Recognition
Text to Speech

Conversational AI

NVIDIA Riva is an application framework for multimodal conversational AI services that deliver real-time performance on GPUs.

Related Resources

Download and get started with NVIDIA Riva.

Riva Benchmarks

A100 ASR Benchmarks

A100 Best Streaming Throughput Mode (800 ms chunk)

Acoustic Model	Language Model	# of Streams	Avg Latency (ms)	Throughput (RTFX)	GPU Version
citrinet	n-gram	1	11.4	1	A100 SXM4-40GB
citrinet	n-gram	64	64.1	64	A100 SXM4-40GB
citrinet	n-gram	128	103	126	A100 SXM4-40GB
citrinet	n-gram	256	166.7	250	A100 SXM4-40GB
citrinet	n-gram	384	235	371	A100 SXM4-40GB
citrinet	n-gram	512	311	490	A100 SXM4-40GB
citrinet	n-gram	768	492	717	A100 SXM4-40GB
conformer	n-gram	1	16.8	1	A100 SXM4-40GB
conformer	n-gram	64	109	64	A100 SXM4-40GB
conformer	n-gram	128	130	126	A100 SXM4-40GB
conformer	n-gram	256	236	249	A100 SXM4-40GB
conformer	n-gram	384	342	369	A100 SXM4-40GB
conformer	n-gram	512	485	486	A100 SXM4-40GB

A100 Best Streaming Latency Mode (160 ms chunk)

Acoustic Model	Language Model	# of Streams	Avg Latency (ms)	Throughput (RTFX)	GPU Version
citrinet	n-gram	1	10.47	1	A100 SXM4-40GB
citrinet	n-gram	8	15.14	8	A100 SXM4-40GB
citrinet	n-gram	16	26.2	16	A100 SXM4-40GB
citrinet	n-gram	32	39.1	32	A100 SXM4-40GB
citrinet	n-gram	48	48	48	A100 SXM4-40GB
citrinet	n-gram	64	55.4	64	A100 SXM4-40GB
conformer	n-gram	1	14.69	1	A100 SXM4-40GB
conformer	n-gram	8	37.7	8	A100 SXM4-40GB
conformer	n-gram	16	41.5	16	A100 SXM4-40GB
conformer	n-gram	32	55.7	32	A100 SXM4-40GB
conformer	n-gram	48	66.8	48	A100 SXM4-40GB
conformer	n-gram	64	82.2	63	A100 SXM4-40GB

A100 Offline Mode (1600 ms chunk)

Acoustic Model	Language Model	# of Streams	Throughput (RTFX)	GPU Version
citrinet	n-gram	32	4390	A100 SXM4-40GB
conformer	n-gram	32	1700	A100 SXM4-40GB

ASR Throughput (RTFX) - Number of seconds of audio processed per second | Riva version: v2.6.0 | ASR Dataset - Librispeech | Hardware: DGX A100 (1x A100 SXM4-40GB) with EPYC 7742@2.25GHz, NVIDIA A30 with EPYC 7742@2.25GHz, NVIDIA A10 with EPYC 7763@2.45GHz, DGX-1 (1x V100-SXM2-16GB) with Xeon E5-2698@2.20GHz, and NVIDIA T4 with Gold 6240@2.60GHz

A30 ASR Benchmarks

A30 Best Streaming Throughput Mode (800 ms chunk)

Acoustic Model	Language Model	# of Streams	Avg Latency (ms)	Throughput (RTFX)	GPU Version
citrinet	n-gram	1	14.64	1	A30
citrinet	n-gram	64	101	63	A30
citrinet	n-gram	128	152	126	A30
citrinet	n-gram	256	272	249	A30
citrinet	n-gram	384	393	368	A30
citrinet	n-gram	512	569	484	A30
conformer	n-gram	1	21.76	1	A30
conformer	n-gram	64	134	63	A30
conformer	n-gram	128	216	126	A30
conformer	n-gram	256	397	248	A30
conformer	n-gram	384	672	364	A30

A30 Best Streaming Latency Mode (160 ms chunk)

Acoustic Model	Language Model	# of Streams	Avg Latency (ms)	Throughput (RTFX)	GPU Version
citrinet	n-gram	1	13.74	1	A30
citrinet	n-gram	8	29.4	8	A30
citrinet	n-gram	16	44.2	16	A30
citrinet	n-gram	32	58.7	32	A30
citrinet	n-gram	48	65.8	48	A30
citrinet	n-gram	64	83	63	A30
conformer	n-gram	1	20.32	1	A30
conformer	n-gram	8	42.2	8	A30
conformer	n-gram	16	51.5	16	A30
conformer	n-gram	32	71.3	32	A30
conformer	n-gram	48	103.9	48	A30
conformer	n-gram	64	126.8	63	A30

A30 Offline Mode (1600 ms chunk)

Acoustic Model	Language Model	# of Streams	Throughput (RTFX)	GPU Version
citrinet	n-gram	32	3142	A30
conformer	n-gram	32	1120	A30

A10 ASR Benchmarks

A10 Best Streaming Throughput Mode (800 ms chunk)

Acoustic Model	Language Model	# of Streams	Avg Latency (ms)	Throughput (RTFX)	GPU Version
citrinet	n-gram	1	12.93	1	A10
citrinet	n-gram	64	88.5	64	A10
citrinet	n-gram	128	162.6	126	A10
citrinet	n-gram	256	316	248	A10
citrinet	n-gram	384	486	367	A10
citrinet	n-gram	512	710	481	A10
conformer	n-gram	1	15.33	1	A10
conformer	n-gram	64	133	63	A10
conformer	n-gram	128	234	126	A10
conformer	n-gram	256	434	247	A10

A10 Best Streaming Latency Mode (160 ms chunk)

Acoustic Model	Language Model	# of Streams	Avg Latency (ms)	Throughput (RTFX)	GPU Version
citrinet	n-gram	1	10.405	1	A10
citrinet	n-gram	8	20.22	8	A10
citrinet	n-gram	16	29.8	16	A10
citrinet	n-gram	32	49.1	32	A10
citrinet	n-gram	48	67.6	48	A10
citrinet	n-gram	64	84.7	63	A10
conformer	n-gram	1	13.49	1	A10
conformer	n-gram	8	33.8	8	A10
conformer	n-gram	16	40.9	16	A10
conformer	n-gram	32	71.5	32	A10
conformer	n-gram	48	108	48	A10
conformer	n-gram	64	140	63	A10

A10 Offline Mode (1600 ms chunk)

Acoustic Model	Language Model	# of Streams	Throughput (RTFX)	GPU Version
citrinet	n-gram	32	2719	A10
conformer	n-gram	32	992	A10

V100 ASR Benchmarks +

V100 Best Streaming Throughput Mode (800 ms chunk)

Acoustic Model	Language Model	# of Streams	Avg Latency (ms)	Throughput (RTFX)	GPU Version
citrinet	n-gram	1	13.91	1	V100 SXM2-16GB
citrinet	n-gram	64	87.9	63	V100 SXM2-16GB
citrinet	n-gram	128	153	125	V100 SXM2-16GB
citrinet	n-gram	256	283.7	246	V100 SXM2-16GB
citrinet	n-gram	384	407	363	V100 SXM2-16GB
citrinet	n-gram	512	590	474	V100 SXM2-16GB
conformer	n-gram	1	22.3	1	V100 SXM2-16GB
conformer	n-gram	64	153	63	V100 SXM2-16GB
conformer	n-gram	128	230.6	125	V100 SXM2-16GB
conformer	n-gram	256	400	245	V100 SXM2-16GB
conformer	n-gram	384	716	359	V100 SXM2-16GB

V100 Best Streaming Latency Mode (160 ms chunk)

Acoustic Model	Language Model	# of Streams	Avg Latency (ms)	Throughput (RTFX)	GPU Version
citrinet	n-gram	1	13.26	1	V100 SXM2-16GB
citrinet	n-gram	8	24.32	8	V100 SXM2-16GB
citrinet	n-gram	16	32.9	16	V100 SXM2-16GB
citrinet	n-gram	32	50.5	32	V100 SXM2-16GB
citrinet	n-gram	48	65	48	V100 SXM2-16GB
citrinet	n-gram	64	84.1	63	V100 SXM2-16GB
conformer	n-gram	1	19.7	1	V100 SXM2-16GB
conformer	n-gram	8	55	8	V100 SXM2-16GB
conformer	n-gram	16	52.3	16	V100 SXM2-16GB
conformer	n-gram	32	76.7	32	V100 SXM2-16GB
conformer	n-gram	48	119.8	47	V100 SXM2-16GB
conformer	n-gram	64	143	63	V100 SXM2-16GB

V100 Offline Mode (1600 ms chunk)

Acoustic Model	Language Model	# of Streams	Throughput (RTFX)	GPU Version
citrinet	n-gram	32	2693	V100 SXM2-16GB
conformer	n-gram	32	964	V100 SXM2-16GB

T4 ASR Benchmarks +

T4 Best Streaming Throughput Mode (800 ms chunk)

Acoustic Model	Language Model	# of Streams	Avg Latency (ms)	Throughput (RTFX)	GPU Version
citrinet	n-gram	1	26.7	1	NVIDIA T4
citrinet	n-gram	64	170.8	63	NVIDIA T4
citrinet	n-gram	128	342	125	NVIDIA T4
citrinet	n-gram	256	736	242	NVIDIA T4
conformer	n-gram	1	59.1	1	NVIDIA T4
conformer	n-gram	64	310	63	NVIDIA T4
conformer	n-gram	128	505	124	NVIDIA T4

T4 Best Streaming Latency Mode (160 ms chunk)

Acoustic Model	Language Model	# of Streams	Avg Latency (ms)	Throughput (RTFX)	GPU Version
citrinet	n-gram	1	25.9	1	NVIDIA T4
citrinet	n-gram	8	57	8	NVIDIA T4
citrinet	n-gram	16	60.5	16	NVIDIA T4
citrinet	n-gram	32	93.1	32	NVIDIA T4
citrinet	n-gram	48	139.7	47	NVIDIA T4
conformer	n-gram	1	53.4	1	NVIDIA T4
conformer	n-gram	8	82	8	NVIDIA T4
conformer	n-gram	16	104.1	16	NVIDIA T4
conformer	n-gram	32	239	32	NVIDIA T4

T4 Offline Mode (1600 ms chunk)

Acoustic Model	Language Model	# of Streams	Throughput (RTFX)	GPU Version
citrinet	n-gram	32	1322	NVIDIA T4
conformer	n-gram	32	488	NVIDIA T4

A100 TTS Benchmarks

Model	# of streams	Avg Latency to first audio (sec)	Avg Latency between audio chunks (sec)	Throughput (RTFX)	GPU Version
FastPitch + Hifi-GAN	1	0.021	0.003	145	A100 SXM4-40GB
FastPitch + Hifi-GAN	4	0.037	0.006	336	A100 SXM4-40GB
FastPitch + Hifi-GAN	6	0.046	0.007	395	A100 SXM4-40GB
FastPitch + Hifi-GAN	8	0.056	0.009	421	A100 SXM4-40GB
FastPitch + Hifi-GAN	10	0.059	0.01	434	A100 SXM4-40GB

TTS Throughput (RTFX) - Number of seconds of audio generated per second | Riva version: v2.6.0 | TTS Dataset - LJSpeech | Hardware: DGX A100 (1x A100 SXM4-40GB) with EPYC 7742@2.25GHz, NVIDIA A30 with EPYC 7742@2.25GHz, NVIDIA A10 with EPYC 7763@2.45GHz, DGX-1 (1x V100-SXM2-16GB) with Xeon E5-2698@2.20GHz, and NVIDIA T4 with Gold 6240@2.60GHz

A30 TTS Benchmarks

Model	# of streams	Avg Latency to first audio (sec)	Avg Latency between audio chunks (sec)	Throughput (RTFX)	GPU Version
FastPitch + Hifi-GAN	1	0.022	0.004	127	A30
FastPitch + Hifi-GAN	4	0.044	0.007	267	A30
FastPitch + Hifi-GAN	6	0.064	0.009	292	A30
FastPitch + Hifi-GAN	8	0.082	0.011	310	A30
FastPitch + Hifi-GAN	10	0.091	0.013	318	A30

A10 TTS Benchmarks

Model	# of streams	Avg Latency to first audio (sec)	Avg Latency between audio chunks (sec)	Throughput (RTFX)	GPU Version
FastPitch + Hifi-GAN	1	0.021	0.004	127	A10
FastPitch + Hifi-GAN	4	0.049	0.008	235	A10
FastPitch + Hifi-GAN	6	0.072	0.011	250	A10
FastPitch + Hifi-GAN	8	0.096	0.014	256	A10

V100 TTS Benchmarks +

Model	# of streams	Avg Latency to first audio (sec)	Avg Latency between audio chunks (sec)	Throughput (RTFX)	GPU Version
FastPitch + Hifi-GAN	1	0.024	0.005	104	V100 SXM2-16GB
FastPitch + Hifi-GAN	4	0.055	0.009	215	V100 SXM2-16GB
FastPitch + Hifi-GAN	6	0.08	0.012	227	V100 SXM2-16GB
FastPitch + Hifi-GAN	8	0.108	0.015	232	V100 SXM2-16GB
FastPitch + Hifi-GAN	10	0.119	0.018	235	V100 SXM2-16GB

T4 TTS Benchmarks +

Model	# of streams	Avg Latency to first audio (sec)	Avg Latency between audio chunks (sec)	Throughput (RTFX)	GPU Version
FastPitch + Hifi-GAN	1	0.05	0.007	64	NVIDIA T4
FastPitch + Hifi-GAN	4	0.096	0.016	121	NVIDIA T4
FastPitch + Hifi-GAN	6	0.142	0.022	127	NVIDIA T4
FastPitch + Hifi-GAN	8	0.188	0.028	132	NVIDIA T4
FastPitch + Hifi-GAN	10	0.218	0.03	134	NVIDIA T4

Last updated: November 9th, 2022

NVIDIA Data Center Deep Learning Product Performance

Reproducible Performance

Related Resources

HPC Performance

Training - Convergence

Training - Single GPU

Inference

AI Pipeline

Training to Convergence

Related Resources

NVIDIA Performance on MLPerf 2.1 Training Benchmarks

MLPerf Training Performance

NVIDIA Performance on MLPerf 2.1 AI Benchmarks: Single Node - Closed Division

NVIDIA Performance on MLPerf 2.1 AI Benchmarks: Multi Node - Closed Division

NVIDIA A100 Performance on MLPerf 2.0 Training HPC Benchmarks: Strong Scaling - Closed Division

NVIDIA A100 Performance on MLPerf 2.0 Training HPC Benchmarks: Weak Scaling - Closed Division

Converged Training Performance of NVIDIA A100, A40, A30, A10, T4 and V100

A100 Training Performance

A40 Training Performance

A30 Training Performance

A10 Training Performance

T4 Training Performance +

V100 Training Performance +

Single-GPU Training

Related Resources

Single GPU Training Performance of NVIDIA A100, A40, A30, A10, T4 and V100

A100 Training Performance

A40 Training Performance

A30 Training Performance

A10 Training Performance

T4 Training Performance +

V100 Training Performance +

Single GPU Training Performance of NVIDIA GPU on Cloud

A100 Training Performance on Cloud

T4 Training Performance on Cloud +

V100 Training Performance on Cloud +

AI Inference

Related Resources

MLPerf Inference v2.1 Performance Benchmarks

Offline Scenario - Closed Division

Server Scenario - Closed Division

Power Efficiency Offline Scenario - Closed Division

Power Efficiency Server Scenario - Closed Division

NVIDIA TRITON Inference Server Delivered Comparable Performance to Custom Harness in MLPerf v2.1

NVIDIA Client Batch Size 1 and 2 Performance with Triton Inference Server

A100 Triton Inference Server Performance

A30 Triton Inference Server Performance

A10 Triton Inference Server Performance

T4 Triton Inference Server Performance +

V100 Triton Inference Server Performance +

Inference Performance of NVIDIA A100, A40, A30, A10, A2, T4 and V100

Inference Image Classification on CNNs with TensorRT

A100 Full Chip Inference Performance

A100 1/7 MIG Inference Performance

A100 7 MIG Inference Performance

A40 Inference Performance

A30 Inference Performance

A30 1/4 MIG Inference Performance

A30 4 MIG Inference Performance

A10 Inference Performance

A2 Inference Performance

T4 Inference Performance +

V100 Inference Performance +

Inference Performance of NVIDIA GPU on Cloud

A100 Inference Performance on Cloud

T4 Inference Performance on Cloud +

V100 Inference Performance on Cloud +

Conversational AI

Related Resources

Riva Benchmarks

A100 ASR Benchmarks

A100 Best Streaming Throughput Mode (800 ms chunk)

A100 Best Streaming Latency Mode (160 ms chunk)

A100 Offline Mode (1600 ms chunk)

A30 ASR Benchmarks

A30 Best Streaming Throughput Mode (800 ms chunk)

A30 Best Streaming Latency Mode (160 ms chunk)

A30 Offline Mode (1600 ms chunk)

A10 ASR Benchmarks

A10 Best Streaming Throughput Mode (800 ms chunk)