NVIDIA Tesla Deep Learning Product Performance

Deep Learning Training

NVIDIA’s complete solution stack, from GPUs to libraries, and containers on NVIDIA GPU Cloud (NGC), allows data scientists to quickly get up and running with deep learning. NVIDIA® Tesla® V100 Tensor Core GPUs leverage mixed precision to accelerate deep learning training throughputs across every framework and every type of neural network.

 
 

 

CNN Training Performance

GPU: TESLA V100

FRAMEWORK	NETWORK	THROUGHPUT	METRIC	BATCH SIZE	GPU	CONTAINER	PRECISION
MXnet	Inception V3	3,818	images/sec	512	8x V100	18.08-py3	Mixed
	Resnet50	9,598	images/sec	512	8x V100	18.09-py3	Mixed
PyTorch	Inception V3	3,909	images/sec	384	8x V100	18.10-py3	Mixed
	Resnet50	6,675	images/sec	256	8x V100	18.10-py3	Mixed
TensorFlow	GoogleNet	11,476	images/sec	1,024	8x V100	18.09-py3	Mixed
	Inception V3	3,965	images/sec	384	8x V100	18.10-py3	Mixed
	Resnet50	6,720	images/sec	512	8x V100	18.09-py3	Mixed

GPU Memory: 32 GB  |  CPU: E5-2698 v4 2.2 GHz  |  ImageNet2012

GPU: TESLA T4

FRAMEWORK	NETWORK	THROUGHPUT	METRIC	BATCH SIZE	GPU	CONTAINER	PRECISION
MXnet	Inception V3	528	images/sec	192	4x T4	18.10-py3	Mixed
	Resnet50	1,564	images/sec	128	4x T4	18.10-py3	Mixed
PyTorch	Inception V3	294	images/sec	64	4x T4	18.10-py3	Mixed
	Resnet50	439	images/sec	128	4x T4	18.10-py3	Mixed
TensorFlow	GoogleNet	1,208	images/sec	256	4x T4	18.10-py3	Mixed
	Inception V3	296	images/sec	64	4x T4	18.10-py3	Mixed
	Resnet50	456	images/sec	128	4x T4	18.10-py3	Mixed

CPU: Gold 6140 2.3 GHz  |  ImageNet2012

 

RNN Training Performance

GPU: TESLA V100

FRAMEWORK	NETWORK	THROUGHPUT	METRIC	BATCH SIZE	GPU	CONTAINER	DATASET
MXnet	Sockeye (OpenNMT)	250,396	total tokens/sec	512	8x V100	18.07-py3	WMT15 German-English
PyTorch	GNMT	503,936	total tokens/sec	384	8x V100	18.09-py3	WMT16 English-German

GPU Memory: 32 GB  |  CPU: E5-2698 v4 2.2 GHz  |  MXnet uses FP32, PyTorch uses mixed precision

GPU: TESLA T4

FRAMEWORK	NETWORK	THROUGHPUT	METRIC	BATCH SIZE	GPU	CONTAINER	DATASET
MXnet	OpenNMT	49,884	total tokens/sec	16,384	4x T4	18.10-py3	WMT15 German-English
PyTorch	GNMT	24,118	total tokens/sec	128	4x T4	18.10-py3	WMT16 English-German

CPU: Gold 6140 2.3 GHz  |  FP32 precision

 

NVIDIA Deep Learning Inference Performance

NVIDIA® TensorRT™ running on NVIDIA GPUs enable the most efficient deep learning inference performance across multiple application areas and models. This versatility provides wide latitude to data scientists to create the optimal low-latency solution. Visit NVIDIA GPU Cloud (NGC) to download any of these containers.

NVIDIA Tesla® V100 Tensor Cores GPUs leverage mixed-precision to combine high throughput with low latencies across every type of neural network. Tesla P4 is an inference GPU, designed for optimal power consumption and latency, for ultra-efficient scale-out servers. Read the inference whitepaper to learn more about NVIDIA’s inference platform.

Measuring the inference performance involves balancing a lot of variables. PLASTER is an acronym that describes the key elements for measuring deep learning performance. Each letter identifies a factor (Programmability, Latency, Accuracy, Size of Model, Throughput, Energy Efficiency, Rate of Learning) that must be considered to arrive at the right set of tradeoffs and to produce a successful deep learning implementation. Refer to NVIDIA’s PLASTER whitepaper for more details.

 
 

 
 

 

 

CNN Inference Performance

GPU: TESLA V100

NETWORK	BATCH SIZE	THROUGHPUT (images/sec)	EFFICIENCY (images/sec/watt)	LATENCY (ms)	PRECISION	CONTAINER
Resnet50	1	1,143	8.6	0.88	INT8	18.09-py3
	2	1,538	11	1.3	INT8	18.10-py3
	4	1,975	12	2	INT8	18.10-py3
	8	3,247	18	2.5	INT8	18.10-py3
	41	5,925	21	6.9	Mixed	18.10-py3
	64	6,050	21	11	INT8	18.10-py3
	128	6,359	22	20	Mixed	18.09-py3
GoogleNet	1	1,622	15	0.62	INT8	18.09-py3
	2	2,142	18	0.93	INT8	18.10-py3
	4	3,110	26	1.3	INT8	18.09-py3
	8	5,239	35	1.5	INT8	18.09-py3
	64	11,453	41	5.6	INT8	18.10-py3
	82	11,916	41	6.9	INT8	18.09-py3
	128	12,350	42	10	INT8	18.10-py3

GPU Memory: 16 GB  |  CPU: E5-2698 v4 2.2 GHz  |  TensorRT 5.0  |  Synthetic dataset

 

CNN Inference Performance

GPU: TESLA T4

NETWORK	BATCH SIZE	THROUGHPUT (images/sec)	EFFICIENCY (images/sec/watt)	LATENCY (ms)	PRECISION	CONTAINER
Resnet50	1	961	14	1.0	INT8	18.10-py3
	2	1,404	21	1.4	INT8	18.10-py3
	4	2,553	37	1.6	INT8	18.10-py3
	8	3,427	50	2.3	INT8	18.10-py3
	16	3,888	56	4.1	INT8	18.10-py3
	29	4,092	60	7.1	INT8	18.10-py3
	128	4,198	60	31	INT8	18.10-py3
GoogleNet	1	1,571	23	0.64	INT8	18.10-py3
	2	2,026	29	0.99	INT8	18.10-py3
	4	3,421	50	1.2	INT8	Pre-Release
	8	3,507	51	2.3	INT8	18.10-py3
	32	5,755	85	5.6	INT8	Pre-Release
	64	5,966	88	11	INT8	Pre-Release
	128	5,989	87	21	INT8	Pre-Release

CPU: Gold 6140 2.3 GHz  |  TensorRT 5.0  |  Synthetic dataset

 

RNN Inference Performance

GPU: TESLA V100

NETWORK	BATCH SIZE	THROUGHPUT (total tokens/sec)	EFFICIENCY (tokens/sec/watt)	LATENCY (ms)	PRECISION	CONTAINER
NMT	1	3,961	32	13	Mixed	18.09-py3
	2	5,711	48	18	Mixed	18.09-py3
	4	9,380	79	22	Mixed	18.09-py3
	8	14,452	119	28	Mixed	18.09-py3
	64	55,497	487	59	Mixed	18.09-py3
	128	75,217	1068	87	Mixed	18.09-py3

GPU Memory: 16 GB  |  CPU: E5-2698 v4 2.2 GHz  |  TensorRT 5.0  |  WMT16 English-German

 

RNN Inference Performance

GPU: TESLA T4

NETWORK	BATCH SIZE	THROUGHPUT (total tokens/sec)	EFFICIENCY (tokens/sec/watt)	LATENCY (ms)	PRECISION	CONTAINER
NMT	1	2,430	37	21	Mixed	18.10-py3
	2	3,701	57	28	Mixed	18.10-py3
	4	6,168	97	33	Mixed	18.10-py3
	8	10,101	161	41	Mixed	18.10-py3
	64	29,012	646	113	Mixed	18.10-py3
	128	34,435	628	191	Mixed	18.10-py3

CPU: Gold 6140 2.3 GHz  |  TensorRT 5.0  |  WMT16 English-German

 

Last updated: Nov 20, 2018