# AI Inference Performance Benchmarks

This is NVIDIA's Data Center Deep Learning Product Performance Hub — a centralized resource for reproducible AI performance benchmarks across NVIDIA's latest data center GPUs.

The days of raw speed being the only metric that matters are behind us. Now it’s about throughput, efficiency, and economics at scale. As AI evolves from providing one-shot answers to engaging in multi-step reasoning, demand for inference and its underlying economics is increasing. This shift significantly boosts compute demand because far more tokens are generated per query. Metrics such as tokens per watt, cost per million tokens, and tokens per second per user are crucial alongside throughput.  
  
For power-limited AI factories, NVIDIA's continuous software improvements translate into higher token revenue over time, underscoring the importance of our technological advancements.  
  
Pareto curves illustrate how NVIDIA Blackwell and Vera Rubin provide the best balance across the full spectrum of production priorities:

- 

Cost

- 

Energy efficiency

- 

Throughput

- 

Responsiveness

Optimizing for a single scenario can limit deployment flexibility and create inefficiencies at other points on the curve. NVIDIA’s full-stack design approach ensures efficiency and value across multiple real-life production scenarios. This leadership stems from extreme hardware-software co-design, a full-stack architecture built for speed, efficiency, and scalability.

[View Other Performance Data](/deep-learning-performance-training-inference)

Large Language Model

MLPerf Inference

Vision & Generative AI



The rows below are the raw result records for this component — one row per record. Column names are the source index fields; the full data is also downloadable in the links that follow.

| ep | hw | tp | isl | osl | conc | date | num_gpus | sequence | framework | precision | prefill_ep | prefill_tp | tput_per_mw | tput_per_gpu | model_display | spec_decoding | benchmark_type | date_timestamp | num_decode_gpu | serving_config | num_prefill_gpu | input_tput_per_gpu | output_tput_per_gpu | p50_itl | mean_itl | p50_e2el | p50_tpot | p50_ttft | mean_e2el | mean_tpot | mean_ttft | p50_intvty | mean_intvty | dp | pp | spec_length | dataset_name | duration_seconds | total_input_tokens | total_output_tokens | tput_requests_per_sec | kv_cache_precision | p90_itl | p90_e2el | p90_tpot | p90_ttft | p90_intvty | scenario | decode_ep | decode_tp | total_requests | avg_output_tokens_expected | theoretical_cache_hit_rate | stdev_output_tokens_expected | avg_isl | avg_osl | p75_isl | p75_osl | p75_qps | p90_isl | p90_osl | p90_qps | p95_isl | p95_osl | p95_qps | std_qps | mean_qps | stdev_isl | stdev_osl | median_qps | source_run_url | p75_ttft | p75_intvty | dataset_url |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 8 | B300 | 8 | 1024 | 1024 | 271 | 2026-01-28 | 18 | 1K/1K | Dynamo TensorRT LLM | FP4 | 2 | 2 | 1450622.0190386316 | 2756.1818361734 | DeepSeek-R1 | mtp | single_turn | 1769587200 | 16 | disagg | 2 | 12393.6815508794 | 1551.4943718351 | 2.3075432074256 | 2.0389398121142 | 9.5533172194846 | 0.0097829802084 | 0.5528178260429001 | 9.4291409960486 | 0.0093585429191 | 0.8114229454576 | 102.2183402906 | 106.8542409481 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 4 | 1024 | 1024 | 64 | 2026-03-27 | 4 | 1K/1K | vLLM | FP4 |  |  | 2596000.678832117 | 3556.52093 | gpt-oss-120b | off |  | 1774594800 |  | agg |  | 1746.560356 | 1746.560356 |  |  | 8.250320349999999 | 0.0077816479999999995 | 0.289694052 |  |  |  | 117.0478049 |  | 1 | 1 | 0 | random | 46.90361814 | 327680 | 327680 | 6.822501391 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 8 | 1024 | 1024 | 2 | 2026-03-27 | 8 | 1K/1K | vLLM | FP4 |  |  | 109039.76233576643 | 149.3844744 | gpt-oss-120b | off |  | 1774594800 |  | agg |  | 74.54186922 | 74.84260515 |  |  | 3.341955062 | 0.003241737 | 0.025658196 |  |  |  | 308.4766055 |  | 1 | 1 | 0 | random | 15.37894088 | 9171 | 9208 | 0.650239836 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B200 | 1 | 8192 | 1024 | 1 | 2026-02-05 | 1 | 8K/1K | TensorRT LLM | FP4 |  |  | 2869710.7058479534 | 4907.205307 | gpt-oss-120b | eagle |  | 1770278400 |  | agg |  | 4204.834 | 527.8099 |  |  | 1.631976 | 0.001501 | 0.096344 |  |  |  | 616.7399 |  |  |  | 3 |  | 17.62566 | 74113 | 9303 | 0.567354 | FP8 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 1 | 8192 | 1024 | 1 | 2026-05-06 | 1 | 8K/1K | Dynamo vLLM | FP4 |  |  | 850542.0753173723 | 1165.2426431848 | Nemotron 3 Super 120B | off |  | 1778050800 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 142.6643198006 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B200 | 1 | 8192 | 1024 | 128 | 2026-05-06 | 1 | 8K/1K | vLLM | NVFP4 |  |  | 12760691.631744854 | 21820.7826902837 | Nemotron 3 Super 120B | off |  | 1778050800 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 24.4736869747 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B200 | 2 | 8192 | 1024 | 8 | 2026-05-06 | 2 | 8K/1K | vLLM | NVFP4 |  |  | 1234885.4101906433 | 2111.654051426 | Nemotron 3 Super 120B | off |  | 1778050800 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 140.2238355375 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B200 | 2 | 8192 | 1024 | 2 | 2026-05-06 | 2 | 8K/1K | vLLM | NVFP4 |  |  | 398164.2956880702 | 680.8609456266 | Nemotron 3 Super 120B | off |  | 1778050800 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 185.6194930276 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H100 | 8 | 1024 | 1024 | 2 | 2026-05-06 | 8 | 1K/1K | vLLM | FP8 |  |  | 41039.0700010219 | 56.2235259014 | Nemotron 3 Super 120B | off |  | 1778050800 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 178.7958325421 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H100 | 2 | 1024 | 1024 | 8 | 2026-05-06 | 2 | 1K/1K | vLLM | FP8 |  |  | 367945.6250881752 | 504.0855063708 | Nemotron 3 Super 120B | off |  | 1778050800 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 85.6864643889 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 8 | 8192 | 1024 | 64 | 2026-04-09 | 8 | 8K/1K | Dynamo vLLM | FP4 |  |  | 908740.3263671533 | 1244.974247123 | Kimi K2.5/2.6 | off |  | 1775718000 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 19.1234962726 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 4 | B300 | 4 | 8192 | 1024 | 52 | 2026-01-28 | 18 | 8K/1K | Dynamo TensorRT LLM | FP4 | 2 | 2 | 1178205.6460119998 | 2238.5907274228 | DeepSeek-R1 | mtp | single_turn | 1769587200 | 16 | disagg | 2 | 17900.3846960277 | 280.8664813472 | 1.7532332011033 | 1.5207547109468 | 9.9980011510197 | 0.007056075616400001 | 4.0460016161669 | 10.103149035872601 | 0.0065404267381 | 4.0581494332291 | 141.7218372313 | 152.8952222906 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 8 | B300 | 8 | 1024 | 1024 | 25 | 2026-01-28 | 42 | 1K/1K | Dynamo TensorRT LLM | FP4 | 2 | 2 | 135259.0017931579 | 256.992103407 | DeepSeek-R1 | mtp | single_turn | 1769587200 | 40 | disagg | 2 | 2703.2809379655 | 134.6776616791 | 1.0376697224565001 | 0.9267604010714 | 4.1884287485154 | 0.0044421424412 | 0.12816249742170002 | 4.1128980633141 | 0.0042683287202 | 0.1982623645114 | 225.1165993073 | 234.2837362217 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B200 | 2 | 1024 | 1024 | 256 | 2026-02-05 | 2 | 1K/1K | TensorRT LLM | FP4 |  |  | 16892674.175438598 | 28886.47284 | gpt-oss-120b | eagle |  | 1770278400 |  | agg |  | 14479.36 | 14407.11 |  |  | 8.583983 | 0.008317 | 0.075616 |  |  |  | 120.2346 |  |  |  | 2 |  | 81.9714 | 2373787 | 2361942 | 31.23041 | FP8 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B200 | 1 | 8192 | 1024 | 1024 | 2026-05-06 | 1 | 8K/1K | vLLM | NVFP4 |  |  | 17745122.456125207 | 30344.1593999741 | Nemotron 3 Super 120B | off |  | 1778050800 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 9.152319081 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B200 | 8 | 1024 | 1024 | 2 | 2026-05-06 | 8 | 1K/1K | vLLM | NVFP4 |  |  | 20318.885900877194 | 34.7452948905 | Nemotron 3 Super 120B | off |  | 1778050800 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 216.8267391944 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B200 | 2 | 1024 | 1024 | 256 | 2026-05-06 | 2 | 1K/1K | vLLM | NVFP4 |  |  | 3817669.272998304 | 6528.2144568271 | Nemotron 3 Super 120B | off |  | 1778050800 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 44.1448054466 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H100 | 4 | 1024 | 1024 | 64 | 2026-05-06 | 4 | 1K/1K | vLLM | FP8 |  |  | 873053.8373279562 | 1196.0837571393 | Nemotron 3 Super 120B | off |  | 1778050800 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 48.2524444438 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 8 | 8192 | 1024 | 8 | 2026-07-14 | 8 | 8K/1K | SGLang | FP4 |  |  | 311448.3213751095 | 426.6842002839 | DeepSeek-V4-Pro | off |  | 1784012400 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 50.3579594834 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 4 | 8192 | 1024 | 4 | 2026-07-04 | 4 | 8K/1K | SGLang | FP4 |  |  | 1103492.1542856935 | 1511.7842513714 | Qwen 3.5 | on |  | 1783148400 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 194.821211892 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 4 | 8192 | 1024 | 32 | 2026-07-04 | 4 | 8K/1K | SGLang | FP4 |  |  | 2506588.6238496355 | 3434.026414674 | Qwen 3.5 | on |  | 1783148400 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 53.2946584656 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B200 | 2 | 1024 | 1024 | 384 | 2026-02-05 | 2 | 1K/1K | TensorRT LLM | FP4 |  |  | 19579697.57309942 | 33481.28285 | gpt-oss-120b | eagle |  | 1770278400 |  | agg |  | 16795.64 | 16685.64 |  |  | 11.18067 | 0.010841 | 0.089843 |  |  |  | 92.23836 |  |  |  | 2 |  | 106.0759 | 3563226 | 3539890 | 36.20049 | FP8 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H100 | 4 | 1024 | 1024 | 8 | 2026-05-06 | 4 | 1K/1K | vLLM | FP8 |  |  | 222955.84744357664 | 305.4495109977 | Nemotron 3 Super 120B | off |  | 1778050800 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 117.5282958874 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H100 | 4 | 1024 | 1024 | 512 | 2026-05-06 | 4 | 1K/1K | vLLM | FP8 |  |  | 2798792.1392275915 | 3834.3452307418 | Nemotron 3 Super 120B | off |  | 1778050800 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 16.7031945215 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H100 | 8 | 1024 | 1024 | 1 | 2026-05-06 | 8 | 1K/1K | vLLM | FP8 |  |  | 14882.908635693431 | 20.3895848309 | Nemotron 3 Super 120B | off |  | 1778050800 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 186.0755805282 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H100 | 4 | 1024 | 1024 | 128 | 2026-05-06 | 4 | 1K/1K | vLLM | FP8 |  |  | 1318603.9427931388 | 1806.4874016266 | Nemotron 3 Super 120B | off |  | 1778050800 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 35.0400025015 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 8 | 8192 | 1024 | 16 | 2026-04-09 | 8 | 8K/1K | Dynamo vLLM | FP4 |  |  | 555807.7864120438 | 761.4566673845 | Kimi K2.5/2.6 | off |  | 1775718000 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 46.3318678023 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 8 | 8192 | 1024 | 2 | 2026-04-09 | 8 | 8K/1K | Dynamo vLLM | FP4 |  |  | 181939.76357978102 | 249.2574761043 | Kimi K2.5/2.6 | off |  | 1775718000 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 111.2794521935 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H100 | 2 | 1024 | 1024 | 256 | 2026-05-06 | 2 | 1K/1K | vLLM | FP8 |  |  | 1727017.3984578103 | 2366.0138358872 | Nemotron 3 Super 120B | off |  | 1778050800 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 29.0966840321 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B300 | 4 | 1024 | 1024 | 64 | 2026-06-04 | 4 | 1K/1K | vLLM | FP4 |  |  | 1581800.8336760527 | 3005.4215839845 | MiniMax-M2.5 | off |  | 1780556400 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 97.2725334284 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | GB300 NVL72 |  | 131072 | 8192 | 28 | 2025-10-31 |  | 128K/8K | TensorRT LLM | FP4 |  |  | 20627.358490566035 | 43.73 | DeepSeek-R1 | mtp |  | 1761894000 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 226.7574 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 1 | B300 | 4 | 1024 | 1024 | 4 | 2026-06-12 | 4 | 1K/1K | TensorRT LLM | FP4 |  |  | 116386.58231057895 | 221.1345063901 | DeepSeek-V4-Pro | mtp | single_turn | 1781247600 |  | agg |  | 111.1648978301 | 109.96960856 | 1.8112025520530999 | 1.7548068331175 | 7.1598325664527005 | 0.0080534424274 | 0.1486608764972 | 8.010735235607699 | 0.0084025974489 | 0.33345573021220004 | 124.1705033617 | 119.0108185094 |  |  |  |  |  |  |  |  |  | 2.0116438803962002 | 11.367660816130199 | 0.010400518822700001 | 0.26856373266780004 | 96.1490495857 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 8 | GB300 NVL72 | 8 | 1024 | 1024 | 24 | 2026-02-03 | 34 | 1K/1K | Dynamo TensorRT LLM | FP4 | 2 | 2 | 132901.12090759433 | 281.7503763241 | DeepSeek-R1 | mtp | single_turn | 1770105600 | 32 | disagg | 2 | 2400.1872437431 | 149.3480721105 | 1.0697207389749002 | 0.9500149552184001 | 4.5032773949788 | 0.0045669084706999995 | 0.38739624302250003 | 4.3980563303844 | 0.0043559704721 | 0.4040352574836 | 218.9665079595 | 229.5699675663 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 8 | 8192 | 1024 | 512 | 2026-04-09 | 8 | 8K/1K | Dynamo vLLM | FP4 |  |  | 1077815.880521752 | 1476.6077563148 | Kimi K2.5/2.6 | off |  | 1775718000 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 7.4419305674 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 4 | 8192 | 1024 | 2 | 2026-06-25 | 4 | 8K/1K | Dynamo vLLM | FP4 |  |  | 434683.60491394164 | 595.5165387321 | MiniMax-M3 | off |  | 1782370800 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 134.8498766218 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 4 | 8192 | 1024 | 32 | 2026-06-25 | 4 | 8K/1K | Dynamo vLLM | FP4 |  |  | 2022712.1311768615 | 2771.1156197123 | MiniMax-M3 | off |  | 1782370800 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 44.350265178 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B300 | 4 | 1024 | 1024 | 8 | 2026-06-04 | 4 | 1K/1K | vLLM | FP4 |  |  | 315972.176627 | 600.3471355913 | MiniMax-M2.5 | off |  | 1780556400 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 154.6468069334 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H100 | 4 | 1024 | 1024 | 64 | 2026-04-09 | 4 | 1K/1K | vLLM | NVFP4 |  |  | 1029105.5587245986 | 1409.8746154527 | Qwen 3.5 | on |  | 1775718000 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 90.0582596635 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 4 | 8192 | 1024 | 4 | 2026-06-25 | 4 | 8K/1K | Dynamo vLLM | FP4 |  |  | 715108.9715643796 | 979.6992910432 | MiniMax-M3 | off |  | 1782370800 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 118.5059705797 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 8 | 1024 | 1024 | 2 | 2026-07-14 | 8 | 1K/1K | SGLang | FP4 |  |  | 28757.24417109489 | 39.3974245144 | DeepSeek-V4-Pro | off |  | 1784012400 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 81.3907186517 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | GB300 NVL72 |  | 131072 | 8192 | 1024 | 2025-10-31 |  | 128K/8K | TensorRT LLM | FP4 |  |  | 124231.1320754717 | 263.37 | DeepSeek-R1 | mtp |  | 1761894000 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 26.7666 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 8 | B300 | 8 | 1024 | 1024 | 1024 | 2026-06-12 | 64 | 1K/1K | TensorRT LLM | FP4 |  |  | 3077189.621269789 | 5846.6602804126 | DeepSeek-V4-Pro | mtp | single_turn | 1781247600 |  | agg |  | 2924.443374893 | 2922.2169055197 | 7.6552827929845 | 6.711062567133 | 38.6679582684883 | 0.032407764371699996 | 8.710653852496801 | 38.945492456456606 | 0.0321642369276 | 9.3460364822648 | 30.8568029726 | 31.0904313462 |  |  |  |  |  |  |  |  |  | 7.7116442720173 | 46.290526642813305 | 0.038693496149300004 | 10.9161524203606 | 25.8441365996 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 4 | B300 | 4 | 1024 | 1024 | 128 | 2026-06-12 | 16 | 1K/1K | TensorRT LLM | FP4 |  |  | 1309788.6061921578 | 2488.5983517651 | DeepSeek-V4-Pro | mtp | single_turn | 1781247600 |  | agg |  | 1245.9821013593 | 1242.6162504057 | 5.3474207799882 | 4.8487014851366 | 22.3651209900272 | 0.021960609297600003 | 1.9167421425227 | 22.7508038702928 | 0.022517931201300002 | 2.0680617058561 | 45.5360771847 | 44.4090529925 |  |  |  |  |  |  |  |  |  | 5.7404845469515005 | 27.694980081997297 | 0.0276786263319 | 3.4288185034529 | 36.1289605925 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 1 | B300 | 4 | 1024 | 1024 | 16 | 2026-06-12 | 4 | 1K/1K | TensorRT LLM | FP4 |  |  | 290009.4731864737 | 551.0179990543 | DeepSeek-V4-Pro | mtp | single_turn | 1781247600 |  | agg |  | 276.8985417161 | 274.1194573382 | 3.0831930610583997 | 2.8870980598676 | 12.941892520990201 | 0.014303560332 | 0.1666490869829 | 12.985334434996899 | 0.013895969137 | 0.2640588124385 | 69.9126634762 | 71.9633146953 |  |  |  |  |  |  |  |  |  | 3.3579006591811997 | 15.4113888972322 | 0.0160534144347 | 0.4035498623969 | 62.2920441049 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 8 | GB300 NVL72 | 8 | 8192 | 1024 | 2253 | 2026-02-03 | 28 | 8K/1K | Dynamo TensorRT LLM | FP4 | 2 | 2 | 8552620.647857737 | 18131.5557734584 | DeepSeek-R1 | mtp | single_turn | 1770105600 | 8 | disagg | 20 | 22561.5324552162 | 7056.6140690639 | 6.2325701049995 | 5.4753600121993005 | 34.8104832569952 | 0.033561001648 | 3.3582652070036003 | 35.6466374686204 | 0.0327031351828 | 5.5173329805633005 | 29.7964885104 | 30.5781080135 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 4 | 8192 | 1024 | 8 | 2026-06-25 | 4 | 8K/1K | Dynamo vLLM | FP4 |  |  | 1091072.7456944527 | 1494.7696616014 | MiniMax-M3 | off |  | 1782370800 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 96.6651069752 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 4 | 8192 | 1024 | 128 | 2026-06-25 | 4 | 8K/1K | Dynamo vLLM | FP4 |  |  | 2594780.18101365 | 3554.8488479887 | MiniMax-M3 | off |  | 1782370800 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 13.913272014 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 2 | 8192 | 1024 | 16 | 2026-06-25 | 2 | 8K/1K | Dynamo vLLM | FP4 |  |  | 1815499.6056368614 | 2487.2344597225 | MiniMax-M3 | off |  | 1782370800 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 49.9618900695 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 2 | 8192 | 1024 | 8 | 2026-06-25 | 2 | 8K/1K | Dynamo vLLM | FP4 |  |  | 1643291.762522044 | 2251.3097146552 | MiniMax-M3 | off |  | 1782370800 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 73.4533475827 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 4 | 8192 | 1024 | 64 | 2026-07-04 | 4 | 8K/1K | SGLang | FP4 |  |  | 2900495.9751145258 | 3973.6794859069 | Qwen 3.5 | on |  | 1783148400 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 30.3352731133 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 8 | 1024 | 1024 | 1 | 2026-07-14 | 8 | 1K/1K | SGLang | FP4 |  |  | 15999.78503832117 | 21.9197055025 | DeepSeek-V4-Pro | off |  | 1784012400 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 91.8326038197 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 8 | B300 | 8 | 1024 | 1024 | 512 | 2026-06-12 | 64 | 1K/1K | TensorRT LLM | FP4 |  |  | 2416026.2526028943 | 4590.4498799455 | DeepSeek-V4-Pro | mtp | single_turn | 1781247600 |  | agg |  | 2294.9354675444 | 2295.5144124011 | 5.6500886534923005 | 5.0090752380748 | 24.456593876588197 | 0.023896236045799997 | 2.0368058930035 | 24.6713575412567 | 0.0239293030678 | 2.6372965599966003 | 41.8475946622 | 41.7897670135 |  |  |  |  |  |  |  |  |  | 6.1489046631265 | 30.1070446420345 | 0.0287849468143 | 3.9747144138441 | 34.7403803263 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 1 | B300 | 4 | 1024 | 1024 | 2 | 2026-06-12 | 4 | 1K/1K | TensorRT LLM | FP4 |  |  | 68301.16080873685 | 129.7722055366 | DeepSeek-V4-Pro | mtp | single_turn | 1781247600 |  | agg |  | 64.322758153 | 65.4494473836 | 1.4941808070289 | 1.3736196521622 | 6.8186654859455 | 0.0068330245258 | 0.14026347204339998 | 6.9462655912444 | 0.0067458031629 | 0.6276153707469 | 146.3480770819 | 148.2403171049 |  |  |  |  |  |  |  |  |  | 1.5961899073096 | 8.541498763836 | 0.0076474805077 | 1.7545537921367 | 130.7620201186 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 4 | B300 | 4 | 1024 | 1024 | 64 | 2026-06-12 | 16 | 1K/1K | TensorRT LLM | FP4 |  |  | 763179.8276806315 | 1450.0416725932 | DeepSeek-V4-Pro | mtp | single_turn | 1781247600 |  | agg |  | 725.5459800499 | 724.4956925433 | 4.544449744978899 | 4.0822364333777 | 19.175553192501 | 0.0185536376913 | 1.7517610514769002 | 19.5554179629593 | 0.0189764869763 | 2.1038573255319997 | 53.8977863336 | 52.6967926809 |  |  |  |  |  |  |  |  |  | 4.8183151840349 | 23.907709729473602 | 0.0231280117412 | 3.8986581749865 | 43.2376120867 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 4 | GB300 NVL72 | 4 | 1024 | 1024 | 3226 | 2026-02-03 | 6 | 1K/1K | Dynamo TensorRT LLM | FP4 | 2 | 2 | 8357326.118558963 | 17717.531371345 | DeepSeek-R1 | mtp | single_turn | 1770105600 | 4 | disagg | 2 | 26581.8335725709 | 13285.3802707321 | 9.007933673507099 | 7.8290549755462 | 53.5680546145013 | 0.0498719536243 | 5.9892110720102 | 54.2908676289247 | 0.047827870251299996 | 10.276338087891599 | 20.0513500541 | 20.9083112994 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 4 | 8192 | 1024 | 1 | 2026-06-25 | 4 | 8K/1K | Dynamo vLLM | FP4 |  |  | 223022.09073474456 | 305.5402643066 | MiniMax-M3 | off |  | 1782370800 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 147.5860442873 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 4 | 8192 | 1024 | 64 | 2026-06-25 | 4 | 8K/1K | Dynamo vLLM | FP4 |  |  | 2342436.214838759 | 3209.1376143291 | MiniMax-M3 | off |  | 1782370800 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 24.890937309 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 8 | 1024 | 1024 | 8 | 2026-07-14 | 8 | 1K/1K | SGLang | FP4 |  |  | 84484.40501226278 | 115.7436348668 | DeepSeek-V4-Pro | off |  | 1784012400 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 59.865733731 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B200 | 8 | 1024 | 1024 | 4 | 2026-06-24 | 8 | 1K/1K | TensorRT LLM | FP4 |  |  | 161109.65281304094 | 275.4975063103 | Qwen 3.5 | mtp |  | 1782284400 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 318.5231782776 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B200 | 8 | 1024 | 1024 | 512 | 2026-06-24 | 8 | 1K/1K | TensorRT LLM | FP4 |  |  | 4649408.073352456 | 7950.4878054327 | Qwen 3.5 | mtp |  | 1782284400 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 67.6876925503 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 8 | 1024 | 1024 | 32 | 2026-07-14 | 8 | 1K/1K | SGLang | FP4 |  |  | 175527.12739160584 | 240.4721645265 | DeepSeek-V4-Pro | off |  | 1784012400 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 31.0766654313 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 8 | GB300 NVL72 | 8 | 8192 | 1024 | 5 | 2026-02-03 | 34 | 8K/1K | Dynamo TensorRT LLM | FP4 | 2 | 2 | 148704.27171976416 | 315.2530560459 | DeepSeek-R1 | mtp | single_turn | 1770105600 | 32 | disagg | 2 | 4758.1616900336 | 37.5712664216 | 0.7895317499642001 | 0.6970344406914001 | 3.763185795513 | 0.0032010823524999997 | 0.8078803464887 | 3.6912806261634 | 0.0030393795248000004 | 0.8892023601802 | 312.394337251 | 329.0145214977 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 8 | GB300 NVL72 | 8 | 8192 | 1024 | 12 | 2026-02-03 | 34 | 8K/1K | Dynamo TensorRT LLM | FP4 | 2 | 2 | 342794.63497910375 | 726.7246261557 | DeepSeek-R1 | mtp | single_turn | 1770105600 | 32 | disagg | 2 | 10986.2117180919 | 85.5066829097 | 0.9008071229910001 | 0.811457401638 | 3.9414252254937 | 0.0038037855643 | 0.5019430844986 | 3.8309279630329 | 0.0035232093452000003 | 0.6121470498692 | 262.8959974444 | 283.832126342 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 8 | B300 | 8 | 8192 | 1024 | 181 | 2026-01-28 | 14 | 8K/1K | Dynamo TensorRT LLM | FP4 | 2 | 2 | 4140818.3737821053 | 7867.554910186 | DeepSeek-R1 | mtp | single_turn | 1769587200 | 8 | disagg | 6 | 16321.3943077224 | 1527.1753620337 | 2.5650058488827 | 2.2340110614118998 | 13.253701606416099 | 0.0105083989532 | 3.7521123094484 | 12.9124770842217 | 0.0097231959364 | 3.9702618364799 | 95.1619751456 | 102.8468423899 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B200 | 1 | 1024 | 1024 | 1536 | 2026-02-05 | 1 | 1K/1K | TensorRT LLM | FP4 |  |  | 37095611.1754386 | 63433.49511 | gpt-oss-120b | eagle |  | 1770278400 |  | agg |  | 31826.71 | 31606.78 |  |  | 46.54151 | 0.041247 | 4.345390999999999 |  |  |  | 24.24393 |  |  |  | 1 |  | 448.1003 | 14261558 | 14163009 | 34.27804 | FP8 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 8 | 1024 | 1024 | 16 | 2026-07-14 | 8 | 1K/1K | SGLang | FP4 |  |  | 124798.62561729927 | 170.9741170957 | DeepSeek-V4-Pro | off |  | 1784012400 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 43.24832588 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 8 | 1024 | 1024 | 64 | 2026-07-14 | 8 | 1K/1K | SGLang | FP4 |  |  | 240134.92540109492 | 328.9848477995 | DeepSeek-V4-Pro | off |  | 1784012400 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 22.2767288233 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 8 | 1024 | 1024 | 32 | 2026-07-14 | 8 | 1K/1K | SGLang | FP4 |  |  | 175738.16320846716 | 240.7612835956 | DeepSeek-V4-Pro | off |  | 1784012400 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 30.7241214251 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B200 | 8 | 1024 | 1024 | 8 | 2026-05-06 | 8 | 1K/1K | vLLM | NVFP4 |  |  | 77021.12361292397 | 131.7061213781 | Nemotron 3 Super 120B | off |  | 1778050800 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 195.3116814114 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B200 | 2 | 8192 | 1024 | 4 | 2026-05-06 | 2 | 8K/1K | vLLM | NVFP4 |  |  | 691913.0007065497 | 1183.1712312082 | Nemotron 3 Super 120B | off |  | 1778050800 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 164.7650461411 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B300 | 2 | 1024 | 1024 | 128 | 2026-06-04 | 2 | 1K/1K | vLLM | FP4 |  |  | 3853229.415925526 | 7321.1358902585 | MiniMax-M2.5 | off |  | 1780556400 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 58.8994463717 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 1 | H200 | 8 | 1024 | 1024 | 64 | 2026-05-30 | 8 | 1K/1K | vLLM | INT4 |  |  | 380489.06640255474 | 521.2700209715 | Kimi K2.5/2.6 | off | single_turn | 1780124400 |  | agg |  | 260.6937544424 | 260.576266529 | 0.0256808307022 | 0.0296527988001 | 27.8668094421737 | 0.0300904406837 | 0.14745885320010002 | 27.6633499625692 | 0.029632768275499998 | 0.3586359859823 | 33.2331457193 | 33.7464252649 |  |  |  |  |  |  |  |  |  | 0.027027357369699998 | 30.800893590040502 | 0.031032537995599997 | 0.34905903283510004 | 32.224241541 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 1 | H200 | 8 | 1024 | 1024 | 8 | 2026-05-30 | 8 | 1K/1K | vLLM | INT4 |  |  | 121122.02125379562 | 165.9371691177 | Kimi K2.5/2.6 | off | single_turn | 1780124400 |  | agg |  | 82.6610299994 | 83.2761391183 | 0.0109876892529 | 0.0116111882162 | 11.0147727825679 | 0.011661375634600001 | 0.1333247688599 | 10.943786948826201 | 0.0116068394936 | 0.1677336126217 | 85.7531762403 | 86.1560979244 |  |  |  |  |  |  |  |  |  | 0.011365810409199999 | 11.9296679095365 | 0.0118361043107 | 0.26197249153630003 | 84.4872581173 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 1 | H200 | 8 | 1024 | 1024 | 16 | 2026-05-30 | 8 | 1K/1K | vLLM | INT4 |  |  | 183266.0532170803 | 251.0744929074 | Kimi K2.5/2.6 | off | single_turn | 1780124400 |  | agg |  | 126.2019020216 | 124.8725908857 | 0.0144350668415 | 0.015474647323199999 | 14.3000503134681 | 0.015595146213999999 | 0.1161580585176 | 14.333988135671799 | 0.0154678811459 | 0.17633013413430002 | 64.12251519 | 64.6500959356 |  |  |  |  |  |  |  |  |  | 0.014863688545300001 | 15.855537527659902 | 0.0158545911467 | 0.1894970354158 | 63.0732127211 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 1 | H200 | 8 | 1024 | 1024 | 4 | 2026-05-30 | 8 | 1K/1K | vLLM | INT4 |  |  | 74410.64327175183 | 101.9425812823 | Kimi K2.5/2.6 | off | single_turn | 1780124400 |  | agg |  | 51.2193177138 | 50.7232635685 | 0.0091467984021 | 0.009428594781 | 8.8651502002031 | 0.0094564636767 | 0.1321904123761 | 8.7885827974416 | 0.009426644845 | 0.145154412929 | 105.7477757213 | 106.0822823433 |  |  |  |  |  |  |  |  |  | 0.0093886662275 | 9.6874399858527 | 0.009554688798000002 | 0.16278559956699998 | 104.6606562638 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 1 | H200 | 8 | 1024 | 1024 | 32 | 2026-05-30 | 8 | 1K/1K | vLLM | INT4 |  |  | 258321.67715262776 | 353.9006976991 | Kimi K2.5/2.6 | off | single_turn | 1780124400 |  | agg |  | 176.6700353834 | 177.2306623157 | 0.0194923467934 | 0.021793789255 | 20.3711569327861 | 0.0219070033783 | 0.13636806095020002 | 20.3697125215782 | 0.021787442159 | 0.2607876803464 | 45.647502889 | 45.8979990721 |  |  |  |  |  |  |  |  |  | 0.0203348910436 | 22.4463372718543 | 0.0227651224824 | 0.3565663445741 | 43.9268447062 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 16 | GB300 NVL72 | 16 | 1024 | 1024 | 4301 | 2026-06-15 | 28 | 1K/1K | Dynamo TensorRT LLM | FP4 | 4 | 4 | 4505281.678175189 | 9551.1971577314 | DeepSeek-V4-Pro | mtp | single_turn | 1781506800 | 16 | disagg | 12 | 11149.8462688336 | 8352.2103244046 | 5.1700048234779 | 4.597570581597299 | 27.3020250815316 | 0.0270044123665 | 1.6967013999820002 | 28.4900546606903 | 0.0268333518166 | 3.7952963321803 | 37.030985397 | 37.2670550751 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 32 | GB300 NVL72 | 32 | 1024 | 1024 | 167 | 2026-06-15 | 36 | 1K/1K | Dynamo TensorRT LLM | FP4 | 4 | 4 | 374888.60136056604 | 794.7638348844 | DeepSeek-V4-Pro | mtp | single_turn | 1781506800 | 32 | disagg | 4 | 3582.6101511316 | 446.2830453535 | 2.2360554800835 | 1.9413702742693 | 9.700842658989101 | 0.007865151654 | 2.297005918459 | 10.0203021533914 | 0.008250721397599999 | 2.4443082417097997 | 127.1431300996 | 121.2015230925 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 32 | GB300 NVL72 | 32 | 1024 | 1024 | 2253 | 2026-06-15 | 44 | 1K/1K | Dynamo TensorRT LLM | FP4 | 4 | 4 | 2727633.903797736 | 5782.5838760512 | DeepSeek-V4-Pro | mtp | single_turn | 1781506800 | 32 | disagg | 12 | 10600.9400864138 | 3975.7002971652 | 3.4569424494985 | 3.0124620333579 | 13.9837220779737 | 0.012311166758100001 | 1.4789765500464 | 15.4243435602334 | 0.012845079540500001 | 3.5924411067637 | 81.2270696719 | 77.8508219312 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 32 | GB300 NVL72 | 32 | 1024 | 1024 | 90 | 2026-06-15 | 36 | 1K/1K | Dynamo TensorRT LLM | FP4 | 4 | 4 | 216139.24203301888 | 458.21519311 | DeepSeek-V4-Pro | mtp | single_turn | 1781506800 | 32 | disagg | 4 | 2063.8089833224 | 257.5159693334 | 2.0467672490049 | 1.7847762235442999 | 9.3425197189208 | 0.0072544571491 | 2.7076359684579003 | 9.4964623417392 | 0.0072864619669 | 2.7896700398471 | 137.8462894538 | 137.2408179084 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 4 | 8192 | 1024 | 16 | 2026-07-04 | 4 | 8K/1K | SGLang | FP4 |  |  | 2002916.6404117518 | 2743.9957973641 | Qwen 3.5 | on |  | 1783148400 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 85.0708523227 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 8 | B300 | 8 |  |  | 768 | 2026-10-20 | 12 |  | Dynamo vLLM | FP4 | 2 | 2 | 212713836.90526316 | 404156.29012 | DeepSeek-V4.1-Flash | mtp | agentic_traces | 1792479600 | 8 | disagg | 4 | 401301.32568 | 2854.96445 | 0.005 | 0.0049900000000000005 | 3.6631799999999997 | 0.005 | 1.0478 | 6.5215 | 0.00497 | 1.52207 | 200.0 | 200.4008016032 |  |  |  |  | 3629.46428 | 17481614885 | 124520934 |  |  | 0.00522 | 13.70923 | 0.00522 | 2.97306 | 191.5708812261 | agentic | 8 | 8 | 121011 | 1367.72444 | 0.97251 | 2521.49685 | 144434.02623 | 1027.54211 | 186537.5 | 1024.0 | 39.0 | 316802.0 | 2392.0 | 44.0 | 445854.0 | 3962.5 | 48.0 | 8.85983 | 33.34527 | 141632.84183 | 1992.48633 | 33.0 |  |  |  |  |
| 8 | B300 | 8 |  |  | 256 | 2026-10-20 | 12 |  | Dynamo vLLM | FP4 | 2 | 2 | 79356464.4 | 150777.28236 | DeepSeek-V4.1-Flash | mtp | agentic_traces | 1792479600 | 8 | disagg | 4 | 149767.04632 | 1010.23604 | 0.0039 | 0.00395 | 2.1243000000000003 | 0.0039 | 0.29189 | 4.38199 | 0.00393 | 0.46184 | 256.4102564103 | 253.164556962 |  |  |  |  | 3626.78334 | 6519045779 | 43987430 |  |  | 0.0041600000000000005 | 9.65976 | 0.0041600000000000005 | 0.8287899999999999 | 240.3846153846 | agentic | 8 | 8 | 43250 | 1340.70545 | 0.97397 | 2414.91268 | 150706.85651 | 1016.57542 | 197123.0 | 1016.0 | 15.0 | 329663.5 | 2372.0 | 17.0 | 459656.6 | 3898.55 | 19.0 | 4.27027 | 11.92719 | 146059.8395 | 2032.85491 | 12.0 |  |  |  |  |
| 1 | B300 | 2 |  |  | 36 | 2026-10-14 | 2 |  | Dynamo SGLang | FP4 |  |  | 28621538.12631579 | 54380.92244 | Qwen 3.5 | mtp | agentic_traces | 1791961200 |  | agg |  | 53889.85634 | 491.0661 | 0.00349 | 0.0044599999999999996 | 25.20184 | 0.00349 | 22.18727 | 26.53548 | 0.00447 | 22.850669999999997 | 286.5329512894 | 224.2152466368 |  |  |  |  | 3616.89904 | 390290594 | 3552275 |  |  | 0.0048200000000000005 | 42.950540000000004 | 0.0048200000000000005 | 37.53235 | 207.468879668 | agentic |  |  | 3670 | 1350.55914 | 0.96439 | 2089.06783 | 106220.25586 | 967.9218 | 151118.0 | 983.5 | 2.0 | 203872.1 | 2323.1 | 2.0 | 229259.2 | 3713.3 | 3.0 | 0.97608 | 1.0186 | 64210.87885 | 2292.67972 | 1.0 | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/38390415472/attempts/1 |  |  |  |
| 1 | B300 | 8 |  |  | 1 | 2026-08-16 | 8 |  | vLLM | FP4 |  |  | 1065963.1368421053 | 2025.32996 | Kimi-K3 | mtp | agentic_traces | 1786863600 |  | agg |  | 2010.1993 | 15.13066 | 0.00423 | 0.00454 | 5.26484 | 0.00423 | 1.37168 | 9.62951 | 0.00454 | 1.69433 | 236.4066193853 | 220.2643171806 |  |  |  |  | 3585.92273 | 61747004 | 441439 |  |  | 0.0049299999999999995 | 23.00267 | 0.0049299999999999995 | 3.1356599999999997 | 202.8397565923 | agentic |  |  | 239 | 2032.20502 | 0.97712 | 2822.64994 | 241286.00418 | 1816.14644 | 375084.0 | 2533.5 | 0.0 | 414776.0 | 4819.0 | 0.0 | 443936.6 | 6426.9 | 1.0 | 0.25253 | 0.06652 | 141387.34077 | 2495.40571 | 0.0 | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/31893747354/attempts/2 |  |  |  |
| 1 | B300 | 8 |  |  | 8 | 2026-08-16 | 8 |  | vLLM | FP4 |  |  | 2726546.6263157893 | 5180.43859 | Kimi-K3 | mtp | agentic_traces | 1786863600 |  | agg |  | 5139.80599 | 40.6326 | 0.00782 | 0.009470000000000001 | 5.41529 | 0.00782 | 1.04294 | 10.22597 | 0.009470000000000001 | 1.31593 | 127.8772378517 | 105.5966209081 |  |  |  |  | 3607.96498 | 166565454 | 1172895 |  |  | 0.01565 | 22.839419999999997 | 0.01565 | 2.27809 | 63.8977635783 | agentic |  |  | 1073 | 2037.65977 | 0.97541 | 2704.81495 | 138260.87605 | 1093.01771 | 150994.0 | 1029.0 | 0.0 | 290547.4 | 2653.0 | 1.0 | 384176.8 | 4579.8 | 1.0 | 0.57713 | 0.2972 | 113606.32183 | 1964.7966 | 0.0 | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/31893747354/attempts/2 |  |  |  |
| 1 | B300 | 8 |  |  | 16 | 2026-08-16 | 8 |  | vLLM | FP4 |  |  | 4026683.010526316 | 7650.69772 | Kimi-K3 | mtp | agentic_traces | 1786863600 |  | agg |  | 7592.28324 | 58.41448 | 0.016 | 0.01877 | 7.24245 | 0.016 | 1.11374 | 17.70412 | 0.01877 | 1.67245 | 62.5 | 53.2765050613 |  |  |  |  | 3624.88897 | 259528291 | 1715776 |  |  | 0.031129999999999998 | 38.33878 | 0.031129999999999998 | 3.22504 | 32.1233536781 | agentic |  |  | 1741 | 1722.86491 | 0.96967 | 2728.82211 | 126461.49914 | 972.98564 | 138914.0 | 812.0 | 1.0 | 202028.0 | 2372.0 | 1.0 | 375478.0 | 4561.0 | 2.0 | 0.78031 | 0.48013 | 123628.52302 | 2076.4305 | 0.0 | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/31893747354/attempts/2 |  |  |  |
| 1 | B300 | 8 |  |  | 4 | 2026-08-16 | 8 |  | vLLM | FP4 |  |  | 1642046.4 | 3119.88816 | Kimi-K3 | mtp | agentic_traces | 1786863600 |  | agg |  | 3096.44211 | 23.44605 | 0.00504 | 0.00625 | 3.15423 | 0.00504 | 0.95808 | 8.05834 | 0.00625 | 1.24218 | 198.4126984127 | 160.0 |  |  |  |  | 3604.14142 | 99809606 | 676067 |  |  | 0.00826 | 21.81182 | 0.00826 | 2.30494 | 121.0653753027 | agentic |  |  | 534 | 1904.40824 | 0.9748 | 2731.82341 | 167191.2397 | 1265.96067 | 204130.0 | 1356.75 | 0.0 | 505988.6 | 3838.8 | 1.0 | 561780.45 | 5379.3 | 1.0 | 0.38264 | 0.14839 | 173726.50231 | 2260.74194 | 0.0 | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/31893747354/attempts/2 |  |  |  |
| 1 | B300 | 4 |  |  | 1 | 2026-10-20 | 4 |  | Dynamo vLLM | FP4 |  |  | 3950562.231578947 | 7506.06824 | DeepSeek-V4.1-Flash | mtp | agentic_traces | 1792479600 |  | agg |  | 7460.54206 | 45.52617 | 0.00158 | 0.00158 | 1.8729200000000001 | 0.00158 | 0.13891 | 3.6280799999999997 | 0.00157 | 0.5194099999999999 | 632.9113924051 | 632.9113924051 |  |  |  |  | 3593.00457 | 107888015 | 654305 |  |  | 0.00161 | 9.08038 | 0.00161 | 0.7808200000000001 | 621.1180124224 | agentic |  |  | 329 | 2142.1459 | 0.98476 | 2724.21336 | 325905.91793 | 1988.76292 | 488305.0 | 2682.0 | 0.0 | 593544.8 | 5075.6 | 0.0 | 637954.8 | 6526.4 | 1.0 | 0.29391 | 0.09139 | 192275.3097 | 2494.18229 | 0.0 |  |  |  |  |
| 1 | H200 | 4 | 8192 | 1024 | 32 | 2026-06-05 | 4 | 8K/1K | vLLM | FP8 |  |  | 2103176.6817496354 | 2881.352053997 | MiniMax-M2.5 | off | single_turn | 1780642800 |  | agg |  | 2559.1367265103 | 322.2153274867 | 0.019108154811 | 0.023619761254 | 22.3647715924308 | 0.0239226536836 | 0.2713311072439 | 22.472380074547303 | 0.023591871304699998 | 0.6371657464799 | 41.8013826236 | 42.3874811406 |  |  |  |  |  |  |  |  |  | 0.019779410027 | 25.429598224349302 | 0.0249109853082 | 0.6585218762979 | 40.1429324303 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 8 | GB300 NVL72 | 8 | 8192 | 1024 | 24 | 2026-02-03 | 34 | 8K/1K | Dynamo TensorRT LLM | FP4 | 2 | 2 | 471076.92697122646 | 998.683085179 | DeepSeek-R1 | mtp | single_turn | 1770105600 | 32 | disagg | 2 | 15090.997197889 | 117.9134531347 | 1.01786820899 | 0.8924206767121 | 5.4529116555059 | 0.0042212370276 | 1.9026468619995 | 5.5736468521415 | 0.0039263351987 | 1.9704985239166999 | 236.8973818482 | 254.6904299777 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 8 | B300 | 8 |  |  | 2048 | 2026-10-20 | 16 |  | Dynamo vLLM | FP4 | 2 | 2 | 364384676.2736842 | 692330.88492 | DeepSeek-V4.1-Flash | mtp | agentic_traces | 1792479600 | 8 | disagg | 8 | 687340.20199 | 4990.68293 | 0.00855 | 0.00848 | 5.888439999999999 | 0.008539999999999999 | 1.47574 | 10.72106 | 0.00845 | 2.27821 | 116.9590643275 | 117.9245283019 |  |  |  |  | 3628.75044 | 39924417912 | 290534488 |  |  | 0.00945 | 22.63406 | 0.009439999999999999 | 4.48924 | 105.8201058201 | agentic | 8 | 8 | 284688 | 1357.92731 | 0.97132 | 2534.09449 | 140177.93841 | 1017.81278 | 178513.75 | 1012.0 | 88.0 | 299907.3 | 2362.3 | 96.0 | 430272.0 | 3901.0 | 103.0 | 16.00826 | 78.46885 | 137799.50797 | 1992.80801 | 79.0 |  |  |  |  |
| 8 | B300 | 8 |  |  | 1280 | 2026-10-20 | 14 |  | Dynamo vLLM | FP4 | 2 | 2 | 221960558.27894738 | 421725.06073 | DeepSeek-V4.1-Flash | mtp | agentic_traces | 1792479600 | 8 | disagg | 6 | 418674.78719 | 3050.27354 | 0.00522 | 0.00528 | 4.2142100000000005 | 0.00522 | 1.31843 | 12.4305 | 0.00526 | 7.18499 | 191.5708812261 | 189.3939393939 |  |  |  |  | 3629.25303 | 21277138253 | 155167307 |  |  | 0.00562 | 15.6727 | 0.00562 | 4.13138 | 177.9359430605 | agentic | 8 | 8 | 152422 | 1337.53642 | 0.97115 | 2456.08687 | 139564.33037 | 1016.80206 | 176767.75 | 1013.0 | 48.0 | 299968.8 | 2363.0 | 54.0 | 431015.55 | 3911.0 | 58.0 | 10.13321 | 42.00083 | 136901.96693 | 1989.02938 | 42.0 |  |  |  |  |
| 8 | B300 | 8 |  |  | 3584 | 2026-10-20 | 18 |  | Dynamo vLLM | FP4 | 2 | 2 | 418295556.4473684 | 794761.55725 | DeepSeek-V4.1-Flash | mtp | agentic_traces | 1792479600 | 8 | disagg | 10 | 788812.24848 | 5949.30877 | 0.01182 | 0.01165 | 15.64156 | 0.01182 | 9.252799999999999 | 21.42868 | 0.0116 | 9.85403 | 84.6023688663 | 85.8369098712 |  |  |  |  | 3629.92898 | 51621515233 | 391121535 |  |  | 0.01233 | 38.42618 | 0.01233 | 15.56842 | 81.103000811 | agentic | 8 | 8 | 384991 | 1361.80017 | 0.969 | 2580.18717 | 133873.21772 | 1009.68654 | 166540.0 | 1012.0 | 117.0 | 285756.0 | 2339.0 | 124.0 | 424609.0 | 3855.0 | 128.0 | 17.59557 | 106.08267 | 135679.72414 | 1970.89101 | 108.0 |  |  |  |  |
| 1 | B300 | 2 |  |  | 40 | 2026-10-14 | 2 |  | Dynamo SGLang | FP4 |  |  | 19540894.968421053 | 37127.70044 | Qwen 3.5 | mtp | agentic_traces | 1791961200 |  | agg |  | 36797.4585 | 330.24194 | 0.00257 | 0.0030099999999999997 | 48.65643 | 0.00257 | 45.991730000000004 | 51.137449999999994 | 0.0030099999999999997 | 48.572480000000006 | 389.1050583658 | 332.2259136213 |  |  |  |  | 3629.28313 | 268968594 | 2434643 |  |  | 0.00317 | 74.44484 | 0.00317 | 71.92175 | 315.4574132492 | agentic |  |  | 2557 | 1374.46045 | 0.96138 | 2087.86814 | 104457.09464 | 937.45913 | 145778.0 | 994.0 | 1.0 | 199477.2 | 2214.8 | 2.0 | 228889.0 | 3718.8 | 2.0 | 0.8377 | 0.70705 | 63167.60019 | 1745.0379 | 0.0 | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/38390415472/attempts/1 |  |  |  |
| 1 | B300 | 2 |  |  | 44 | 2026-10-14 | 2 |  | Dynamo SGLang | FP4 |  |  | 37091459.02105263 | 70473.77214 | Qwen 3.5 | mtp | agentic_traces | 1791961200 |  | agg |  | 69810.76068 | 663.01146 | 0.0051600000000000005 | 0.0057 | 17.887330000000002 | 0.0051600000000000005 | 13.84016 | 19.54411 | 0.00571 | 14.25384 | 193.7984496124 | 175.4385964912 |  |  |  |  | 3629.35658 | 509445576 | 4812625 |  |  | 0.00732 | 31.75757 | 0.00733 | 22.8567 | 136.6120218579 | agentic |  |  | 5011 | 1321.11575 | 0.96058 | 2092.59909 | 101124.78288 | 960.4091 | 142223.5 | 996.5 | 2.0 | 204474.0 | 2311.0 | 3.0 | 232606.0 | 3731.5 | 3.0 | 1.12649 | 1.38666 | 65219.11582 | 2108.73235 | 1.0 | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/38390415472/attempts/1 |  |  |  |
| 1 | B300 | 4 |  |  | 4 | 2026-10-14 | 4 |  | Dynamo SGLang | FP4 |  |  | 3626240.4210526315 | 6889.8568 | Qwen 3.5 | mtp | agentic_traces | 1791961200 |  | agg |  | 6817.35309 | 72.50371 | 0.00182 | 0.00182 | 1.9666700000000001 | 0.00182 | 0.561 | 3.77691 | 0.00183 | 2.05748 | 549.4505494505 | 549.4505494505 |  |  |  |  | 3605.38096 | 98366314 | 1045615 |  |  | 0.00188 | 9.93391 | 0.00188 | 5.47677 | 531.914893617 | agentic |  |  | 1117 | 1656.40664 | 0.95571 | 2494.6192 | 88018.46016 | 936.09132 | 108316.0 | 801.0 | 1.0 | 186808.8 | 2469.6 | 1.0 | 222727.0 | 4180.4 | 1.0 | 0.54499 | 0.31041 | 59020.72314 | 1839.18148 | 0.0 | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/38390415472/attempts/1 |  |  |  |
| 1 | B300 | 2 |  |  | 64 | 2026-10-14 | 2 |  | Dynamo SGLang | FP4 |  |  | 59397867.71052632 | 112855.94865 | Qwen 3.5 | mtp | agentic_traces | 1791961200 |  | agg |  | 111661.35603 | 1194.59262 | 0.01642 | 0.01874 | 8.31082 | 0.01642 | 0.61039 | 16.88517 | 0.01874 | 0.84 | 60.9013398295 | 53.3617929562 |  |  |  |  | 3628.41477 | 812558930 | 8711239 |  |  | 0.02656 | 37.07739 | 0.02656 | 1.44391 | 37.6506024096 | agentic |  |  | 9481 | 1154.40591 | 0.95215 | 2067.38231 | 85466.45143 | 914.35028 | 123020.0 | 1006.0 | 4.0 | 185585.0 | 1997.0 | 5.0 | 222201.0 | 3366.0 | 7.0 | 2.15798 | 2.62576 | 64354.13104 | 1785.41377 | 2.0 | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/38390415472/attempts/1 |  |  |  |
| 1 | B300 | 4 |  |  | 16 | 2026-10-20 | 4 |  | Dynamo vLLM | FP4 |  |  | 15619879.842105264 | 29677.7717 | DeepSeek-V4.1-Flash | mtp | agentic_traces | 1792479600 |  | agg |  | 29485.03764 | 192.73406 | 0.00227 | 0.00232 | 0.95196 | 0.00226 | 0.07665999999999999 | 2.3514899999999996 | 0.00231 | 0.11864 | 440.5286343612 | 431.0344827586 |  |  |  |  | 3610.93679 | 426624712 | 2783804 |  |  | 0.00298 | 5.52145 | 0.00298 | 0.20107 | 335.5704697987 | agentic |  |  | 2737 | 1649.8238 | 0.9763 | 2534.0107 | 155598.98758 | 1017.09974 | 184554.0 | 962.0 | 1.0 | 284947.2 | 2555.8 | 2.0 | 526591.2 | 4395.6 | 2.0 | 0.9147 | 0.75789 | 152407.18374 | 1970.26448 | 1.0 |  |  |  |  |
| 16 | GB300 NVL72 | 20 |  |  | 600 | 2026-10-26 | 28 |  | Dynamo TensorRT LLM | FP4 |  |  | 54233975.330188684 | 114976.0277 | Qwen 3.5 | mtp | agentic_traces | 1792998000 |  | disagg |  | 113914.85464 | 1061.17306 | 0.0030499999999999998 | 0.00318 | 2.17627 | 0.0030499999999999998 | 0.76302 | 4.0590399999999995 | 0.00318 | 1.21022 | 327.77878 | 314.54732 |  |  |  | semianalysisai/cc-traces-weka-062126-256k | 3623.11339 |  |  | 31.91082 |  | 0.00348 | 8.532020000000001 | 0.00348 | 1.6174600000000001 | 287.73285 | agentic |  |  | 115583 |  |  |  | 99983.04409 |  |  |  |  |  |  |  |  |  |  |  |  | 65399.92498 |  |  |  | 0.9853 | 313.11093 |  |
| 4 | GB300 NVL72 | 10 |  |  | 192 | 2026-10-26 | 12 |  | Dynamo TensorRT LLM | FP4 |  |  | 42573355.25471698 | 90255.51314 | Qwen 3.5 | mtp | agentic_traces | 1792998000 |  | disagg |  | 89445.60949 | 809.90365 | 0.00211 | 0.00233 | 1.4600799999999998 | 0.00211 | 0.49893 | 2.7594499999999997 | 0.00233 | 0.69353 | 472.9965 | 429.30841 |  |  |  | semianalysisai/cc-traces-weka-062126-256k | 3608.53058 |  |  | 10.50471 |  | 0.00322 | 5.732489999999999 | 0.00322 | 1.0500999999999998 | 310.74721 | agentic |  |  | 37902 |  |  |  | 102190.03224 |  |  |  |  |  |  |  |  |  |  |  |  | 65735.31321 |  |  |  | 0.7442799999999999 | 409.10619 |  |
| 16 | GB300 NVL72 | 20 |  |  | 700 | 2026-10-26 | 28 |  | Dynamo TensorRT LLM | FP4 |  |  | 62174222.858490564 | 131809.35246 | Qwen 3.5 | mtp | agentic_traces | 1792998000 |  | disagg |  | 130583.37899 | 1225.97346 | 0.0033 | 0.0034100000000000003 | 2.3994400000000002 | 0.0033 | 0.82904 | 4.6415 | 0.0034100000000000003 | 1.6003399999999999 | 302.98935 | 293.53215 |  |  |  | semianalysisai/cc-traces-weka-062126-256k | 3627.83502 |  |  | 36.99504 |  | 0.00373 | 10.49072 | 0.00373 | 2.1156599999999997 | 267.86662 | agentic |  |  | 134182 |  |  |  | 98855.12777 |  |  |  |  |  |  |  |  |  |  |  |  | 65339.10412 |  |  |  | 1.0569300000000001 | 288.61071 |  |
| 4 | GB300 NVL72 | 8 |  |  | 704 | 2026-10-26 | 20 |  | Dynamo TensorRT LLM | FP4 |  |  | 77335578.80660377 | 163951.42707 | Qwen 3.5 | mtp | agentic_traces | 1792998000 |  | disagg |  | 162411.53815 | 1539.88892 | 0.00701 | 0.00698 | 4.248060000000001 | 0.00701 | 0.87248 | 7.991149999999999 | 0.00698 | 1.77127 | 142.66976 | 143.21862 |  |  |  | semianalysisai/cc-traces-weka-062126-256k | 3629.48488 |  |  | 33.1871 |  | 0.00743 | 17.05079 | 0.00743 | 3.5957600000000003 | 134.63756 | agentic |  |  | 120437 |  |  |  | 97888.55959 |  |  |  |  |  |  |  |  |  |  |  |  | 64959.04874 |  |  |  | 1.37579 | 138.93634 |  |
| 8 | GB300 NVL72 | 20 |  |  | 8 | 2026-10-26 | 20 |  | Dynamo TensorRT LLM | FP4 |  |  | 1325686.3396226414 | 2810.45504 | Qwen 3.5 | mtp | agentic_traces | 1792998000 |  | disagg |  | 2783.36689 | 27.08814 | 0.0005600000000000001 | 0.00057 | 0.5057699999999999 | 0.0005600000000000001 | 0.15644999999999998 | 1.15524 | 0.00057 | 0.5794400000000001 | 1796.86129 | 1741.46963 |  |  |  | semianalysisai/cc-traces-weka-062126-256k | 3599.5156 |  |  | 0.53085 |  | 0.0006 | 2.82715 | 0.0006 | 1.4031500000000001 | 1667.14379 | agentic |  |  | 1911 |  |  |  | 104853.71586 |  |  |  |  |  |  |  |  |  |  |  |  | 55180.4356 |  |  |  | 0.37083999999999995 | 1736.22939 |  |
| 4 | GB300 NVL72 | 10 |  |  | 176 | 2026-10-26 | 20 |  | Dynamo TensorRT LLM | FP4 |  |  | 24002885.051886793 | 50886.11631 | Qwen 3.5 | mtp | agentic_traces | 1792998000 |  | disagg |  | 50433.92406 | 452.19225 | 0.00133 | 0.0013700000000000001 | 1.0303499999999999 | 0.00133 | 0.41469 | 1.7971300000000001 | 0.0013700000000000001 | 0.5804600000000001 | 750.74187 | 727.76314 |  |  |  | semianalysisai/cc-traces-weka-062126-256k | 3612.78939 |  |  | 9.84884 |  | 0.00161 | 3.53852 | 0.00161 | 0.8965 | 621.37376 | agentic |  |  | 35575 |  |  |  | 102435.50018 |  |  |  |  |  |  |  |  |  |  |  |  | 65555.43539 |  |  |  | 0.66374 | 687.7094 |  |
| 2 | GB300 NVL72 | 6 |  |  | 50 | 2026-10-24 | 22 |  | Dynamo TensorRT LLM | FP4 |  |  | 7851272.759433963 | 16644.69825 | MiniMax-M3 | mtp | agentic_traces | 1792825200 |  | disagg |  | 16533.22325 | 111.475 | 0.00152 | 0.00149 | 3.16725 | 0.00152 | 1.6991800000000001 | 5.247439999999999 | 0.00149 | 3.42516 | 656.12009 | 673.34076 |  |  |  | semianalysisai/cc-traces-weka-062126 | 3606.06281 |  |  | 2.33175 |  | 0.0019 | 11.76339 | 0.0019 | 8.3947 | 525.63459 | agentic |  |  | 8337 |  |  |  | 157327.15737 |  |  |  |  |  |  |  |  |  |  |  |  | 169664.59721 |  |  |  | 4.1265 | 575.58794 |  |
| 4 | GB300 NVL72 | 12 |  |  | 264 | 2026-10-26 | 24 |  | Dynamo TensorRT LLM | FP4 |  |  | 29820016.61320755 | 63218.43522 | Qwen 3.5 | mtp | agentic_traces | 1792998000 |  | disagg |  | 62636.18092 | 582.2543 | 0.00163 | 0.0017 | 1.0958299999999999 | 0.00163 | 0.31685 | 2.11906 | 0.0017 | 0.5974400000000001 | 615.17023 | 587.94314 |  |  |  | semianalysisai/cc-traces-weka-062126-256k | 3614.75918 |  |  | 15.0047 |  | 0.00195 | 4.39393 | 0.00195 | 0.80624 | 513.18175 | agentic |  |  | 54228 |  |  |  | 100205.66947 |  |  |  |  |  |  |  |  |  |  |  |  | 65866.44723 |  |  |  | 0.59047 | 572.26633 |  |
| 32 | GB300 NVL72 | 32 | 1024 | 1024 | 4301 | 2026-06-15 | 48 | 1K/1K | Dynamo TensorRT LLM | FP4 | 4 | 4 | 3757033.8660422172 | 7964.9117960095 | DeepSeek-V4-Pro | mtp | single_turn | 1781506800 | 32 | disagg | 16 | 11954.6402019944 | 5970.047593017 | 3.5304959309869 | 3.1924739909993 | 18.9427911620005 | 0.0185587035779 | 1.5304136075429 | 19.8220946133207 | 0.0186262322202 | 2.6798232272705 | 53.8830740952 | 53.6877232163 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 16 | GB300 NVL72 | 16 | 1024 | 1024 | 8192 | 2026-06-15 | 28 | 1K/1K | Dynamo TensorRT LLM | FP4 | 4 | 4 | 4982356.050025849 | 10562.5948260548 | DeepSeek-V4-Pro | mtp | single_turn | 1781506800 | 16 | disagg | 12 | 12327.6059249834 | 9238.8365018583 | 6.6126093019266 | 5.9884657569642 | 45.1264605454635 | 0.0358247219746 | 12.791490473551699 | 48.7026132673125 | 0.0349526708879 | 16.5302536571748 | 27.9136848769 | 28.610116898 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 32 | GB300 NVL72 | 32 | 1024 | 1024 | 333 | 2026-06-15 | 36 | 1K/1K | Dynamo TensorRT LLM | FP4 | 4 | 4 | 689070.626603632 | 1460.8297283997 | DeepSeek-V4-Pro | mtp | single_turn | 1781506800 | 32 | disagg | 4 | 6570.8402333126 | 822.0784152855 | 2.3557459909935 | 2.046460177921 | 10.4934929174487 | 0.0083209126973 | 2.5023648465285 | 10.9427820971021 | 0.0087299002983 | 2.9038195939834 | 120.1791241392 | 114.5488454426 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 4 | GB300 NVL72 | 4 | 1024 | 1024 | 25 | 2026-06-15 | 24 | 1K/1K | Dynamo TensorRT LLM | FP4 | 4 | 4 | 104334.32389674529 | 221.1887666611 | DeepSeek-V4-Pro | mtp | single_turn | 1781506800 | 20 | disagg | 4 | 665.1951314045 | 132.3874937124 | 1.8264863929944 | 1.5932300670363 | 7.9378351685591 | 0.0065020535374 | 1.5983677944751 | 8.110499807374099 | 0.006800250325100001 | 1.8777839334803 | 153.7975647617 | 147.053410123 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 4 | GB300 NVL72 | 12 |  |  | 128 | 2026-10-26 | 20 |  | Dynamo TensorRT LLM | FP4 |  |  | 16813544.311320756 | 35644.71394 | Qwen 3.5 | mtp | agentic_traces | 1792998000 |  | disagg |  | 35297.73177 | 346.98216 | 0.0012 | 0.00121 | 0.904 | 0.0012 | 0.28432 | 1.5648499999999999 | 0.00121 | 0.45607 | 834.73898 | 823.57847 |  |  |  | semianalysisai/cc-traces-weka-062126-256k | 3604.65157 |  |  | 7.32464 |  | 0.00145 | 3.0452 | 0.00145 | 0.7209500000000001 | 688.43176 | agentic |  |  | 26399 |  |  |  | 96394.57889 |  |  |  |  |  |  |  |  |  |  |  |  | 66263.66112 |  |  |  | 0.56657 | 764.70581 |  |
| 2 | GB300 NVL72 | 6 |  |  | 72 | 2026-10-24 | 10 |  | Dynamo TensorRT LLM | FP4 |  |  | 4245009.028301887 | 8999.41914 | MiniMax-M3 | mtp | agentic_traces | 1792825200 |  | disagg |  | 8925.88006 | 73.53907 | 0.00163 | 0.0016799999999999999 | 114.66278999999999 | 0.00163 | 113.26867 | 105.86325 | 0.0016799999999999999 | 103.91135 | 615.15006 | 596.26193 |  |  |  | semianalysisai/cc-traces-weka-062126 | 3611.64466 |  |  | 0.70701 |  | 0.00237 | 206.54339000000002 | 0.00237 | 204.34692 | 421.17955 | agentic |  |  | 2554 |  |  |  | 126222.0325 |  |  |  |  |  |  |  |  |  |  |  |  | 134673.55459 |  |  |  | 157.90903 | 493.29429 |  |
| 8 | GB300 NVL72 | 20 |  |  | 16 | 2026-10-26 | 20 |  | Dynamo TensorRT LLM | FP4 |  |  | 2544910.306603774 | 5395.20985 | Qwen 3.5 | mtp | agentic_traces | 1792998000 |  | disagg |  | 5349.61769 | 45.59216 | 0.00065 | 0.00067 | 0.47158 | 0.00065 | 0.15632 | 0.93128 | 0.00067 | 0.33197000000000004 | 1539.55894 | 1497.62537 |  |  |  | semianalysisai/cc-traces-weka-062126-256k | 3599.69682 |  |  | 0.98055 |  | 0.00076 | 2.25578 | 0.00076 | 0.64143 | 1317.29465 | agentic |  |  | 3530 |  |  |  | 109104.82606 |  |  |  |  |  |  |  |  |  |  |  |  | 61078.75438 |  |  |  | 0.25461 | 1372.17605 |  |
| 1 | H200 | 4 | 1024 | 1024 | 32 | 2026-06-05 | 4 | 1K/1K | vLLM | FP8 |  |  | 667445.6999467154 | 914.400608927 | MiniMax-M2.5 | off | single_turn | 1780642800 |  | agg |  | 456.4760368772 | 457.9245720498 | 0.0152980890125 | 0.016889833386500002 | 15.756152363494 | 0.0170259808717 | 0.10365770012139999 | 15.7515925351792 | 0.0168827694454 | 0.1675017594534 | 58.7337673838 | 59.2319881662 |  |  |  |  |  |  |  |  |  | 0.015873886644800002 | 17.3711880164221 | 0.0173804414109 | 0.261076760944 | 57.5359380326 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 32 | GB300 NVL72 | 32 | 1024 | 1024 | 615 | 2026-06-15 | 36 | 1K/1K | Dynamo TensorRT LLM | FP4 | 4 | 4 | 1166890.7648407076 | 2473.8084214623 | DeepSeek-V4-Pro | mtp | single_turn | 1781506800 | 32 | disagg | 4 | 11133.3020276572 | 1391.3717206879 | 2.4630343429744 | 2.1417539539614 | 11.2581190629862 | 0.008690328326999999 | 3.0138519444735996 | 11.850302527694101 | 0.0091157498428 | 3.4574131574339 | 115.0704510085 | 109.700245975 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 32 | GB300 NVL72 | 32 | 1024 | 1024 | 1229 | 2026-06-15 | 40 | 1K/1K | Dynamo TensorRT LLM | FP4 | 4 | 4 | 1991748.7501817925 | 4222.5073503854 | DeepSeek-V4-Pro | mtp | single_turn | 1781506800 | 32 | disagg | 8 | 10558.9113354266 | 2638.4063541251 | 2.9104698229348 | 2.5107427418633 | 11.935332480003101 | 0.0102077904718 | 2.1814305121079003 | 12.5770693404617 | 0.0107203529707 | 2.7077462999506 | 97.964393251 | 93.2805107008 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 8 | H200 | 8 | 1024 | 1024 | 896 | 2026-03-23 | 64 | 1K/1K | Dynamo TensorRT LLM | FP8 | 8 | 8 | 721525.2967091971 | 988.4896564916 | DeepSeek-R1 | mtp | single_turn | 1774249200 | 56 | disagg | 8 | 3953.3121321026 | 564.9435885472 | 5.132950935047099 | 4.5345815648242 | 24.0748594070319 | 0.0217920997707 | 4.0113172403071 | 24.846131647857298 | 0.0211989099647 | 5.3357909864089 | 45.8881893218 | 47.1722367643 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 8 | H200 | 8 | 8192 | 1024 | 48 | 2026-03-23 | 32 | 8K/1K | Dynamo TensorRT LLM | FP8 | 8 | 8 | 560918.9472044526 | 768.4589576701 | DeepSeek-R1 | mtp | single_turn | 1774249200 | 24 | disagg | 8 | 2730.966259779 | 114.2898569672 | 2.8706050440669 | 2.5067171161495 | 15.5981510159327 | 0.0128635035437 | 3.7256986449938 | 15.6594304780204 | 0.0123561457489 | 4.2225340591792 | 77.7393185773 | 80.9313859129 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B300 | 1 | 1024 | 1024 | 4 | 2026-06-04 | 1 | 1K/1K | vLLM | FP4 |  |  | 569127.7519297894 | 1081.3427286666 | MiniMax-M2.5 | off |  | 1780556400 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 138.9551934959 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B300 | 8 | 1024 | 1024 | 4 | 2026-06-04 | 8 | 1K/1K | vLLM | FP4 |  |  | 94359.39031457895 | 179.2828415977 | MiniMax-M2.5 | off |  | 1780556400 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 186.8057477814 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 1 | GB300 NVL72 | 8 |  |  | 48 | 2026-10-27 | 24 |  | Dynamo vLLM | FP4 | 8 | 8 | 5562174.4433962265 | 11791.80982 | Kimi-K3 | mtp | agentic_traces | 1793084400 | 16 | disagg | 8 | 11712.57298 | 79.23685 | 0.00749 | 0.0074800000000000005 | 4.644399999999999 | 0.00749 | 0.89358 | 9.235 | 0.00752 | 1.3449200000000001 | 133.5113484646 | 133.6898395722 |  |  |  |  | 3629.90638 | 1021715840 | 6918570 |  |  | 0.00932 | 20.38937 | 0.00932 | 2.4989 | 107.2961373391 | agentic | 1 | 8 | 6690 | 1234.97275 | 0.97454 | 1994.35433 | 152522.12885 | 1031.829 | 199418.25 | 1052.5 | 3.0 | 326770.2 | 2471.3 | 4.0 | 563074.25 | 3985.3 | 4.0 | 1.44751 | 1.84321 | 162059.07138 | 2100.80955 | 2.0 | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/40239967341/attempts/1 |  |  |  |
| 2 | GB300 NVL72 | 6 |  |  | 54 | 2026-10-24 | 14 |  | Dynamo TensorRT LLM | FP4 |  |  | 13193506.745283019 | 27970.2343 | MiniMax-M3 | mtp | agentic_traces | 1792825200 |  | disagg |  | 27778.90241 | 191.33189 | 0.00197 | 0.00191 | 3.50288 | 0.00197 | 1.5621800000000001 | 6.01628 | 0.00191 | 3.7164800000000002 | 506.97789 | 524.17187 |  |  |  | semianalysisai/cc-traces-weka-062126 | 3617.73794 |  |  | 2.5499 |  | 0.0025 | 13.963809999999999 | 0.0025 | 10.159360000000001 | 399.68351 | agentic |  |  | 9226 |  |  |  | 152498.92142 |  |  |  |  |  |  |  |  |  |  |  |  | 167024.11836 |  |  |  | 4.68478 | 433.22775 |  |
| 1 | GB300 NVL72 | 2 |  |  | 35 | 2026-10-24 | 2 |  | Dynamo TensorRT LLM | FP4 |  |  | 47499642.58490566 | 100699.24228 | MiniMax-M3 | mtp | agentic_traces | 1792825200 |  | agg |  | 100034.13149 | 665.11079 | 0.00788 | 0.00811 | 4.46453 | 0.00788 | 0.7293200000000001 | 10.9215 | 0.00811 | 0.91404 | 126.86941 | 123.2286 |  |  |  | semianalysisai/cc-traces-weka-062126 | 3610.66538 |  |  | 1.24557 |  | 0.01188 | 25.438119999999998 | 0.01188 | 1.35323 | 84.18194 | agentic |  |  | 4496 |  |  |  | 160671.60832 |  |  |  |  |  |  |  |  |  |  |  |  | 138793.63695 |  |  |  | 0.93855 | 100.92758 |  |
| 4 | GB300 NVL72 | 12 |  |  | 28 | 2026-10-26 | 12 |  | Dynamo TensorRT LLM | FP4 |  |  | 6692117.471698114 | 14187.28904 | Qwen 3.5 | mtp | agentic_traces | 1792998000 |  | disagg |  | 14061.3521 | 125.93694 | 0.00091 | 0.00094 | 0.62567 | 0.00091 | 0.14353 | 1.2634 | 0.00094 | 0.37861 | 1101.23224 | 1067.14579 |  |  |  | semianalysisai/cc-traces-weka-062126-256k | 3599.08579 |  |  | 1.55642 |  | 0.00114 | 2.98626 | 0.00114 | 0.94956 | 873.47466 | agentic |  |  | 5601 |  |  |  | 108426.37922 |  |  |  |  |  |  |  |  |  |  |  |  | 65299.2544 |  |  |  | 0.28 | 921.13007 |  |
| 2 | GB300 NVL72 | 10 |  |  | 12 | 2026-10-24 | 18 |  | Dynamo TensorRT LLM | FP4 |  |  | 1771993.4056603776 | 3756.62602 | MiniMax-M3 | mtp | agentic_traces | 1792825200 |  | disagg |  | 3728.03144 | 28.59458 | 0.00106 | 0.00102 | 1.5504200000000001 | 0.00106 | 0.74272 | 3.31929 | 0.00102 | 1.98907 | 944.85644 | 981.30028 |  |  |  | semianalysisai/cc-traces-weka-062126 | 3599.7145 |  |  | 0.4743 |  | 0.00128 | 8.07783 | 0.00128 | 4.77574 | 781.42239 | agentic |  |  | 1708 |  |  |  | 141426.97834 |  |  |  |  |  |  |  |  |  |  |  |  | 115860.92252 |  |  |  | 1.77367 | 843.80636 |  |
| 1 | GB300 NVL72 | 2 |  |  | 30 | 2026-10-24 | 2 |  | Dynamo TensorRT LLM | FP4 |  |  | 41762465.02358491 | 88536.42585 | MiniMax-M3 | mtp | agentic_traces | 1792825200 |  | agg |  | 87971.43964 | 564.98621 | 0.00658 | 0.006849999999999999 | 3.75583 | 0.00658 | 0.6335299999999999 | 8.78936 | 0.006849999999999999 | 0.7937000000000001 | 151.99645 | 145.97488 |  |  |  | semianalysisai/cc-traces-weka-062126 | 3614.57227 |  |  | 1.11936 |  | 0.01031 | 20.919919999999998 | 0.01031 | 1.14114 | 96.98055 | agentic |  |  | 4043 |  |  |  | 157298.60326 |  |  |  |  |  |  |  |  |  |  |  |  | 131809.02561 |  |  |  | 0.8141 | 120.79769 |  |
| 1 | GB300 NVL72 | 2 |  |  | 25 | 2026-10-24 | 2 |  | Dynamo TensorRT LLM | FP4 |  |  | 37875203.58962264 | 80295.43161 | MiniMax-M3 | mtp | agentic_traces | 1792825200 |  | agg |  | 79785.14288 | 510.28874 | 0.00588 | 0.0060999999999999995 | 3.11341 | 0.00588 | 0.57298 | 7.82641 | 0.0060999999999999995 | 0.76852 | 170.20844 | 163.98229 |  |  |  | semianalysisai/cc-traces-weka-062126 | 3619.03245 |  |  | 1.03568 |  | 0.008919999999999999 | 19.23836 | 0.008919999999999999 | 1.1554 | 112.08672 | agentic |  |  | 3745 |  |  |  | 154202.94873 |  |  |  |  |  |  |  |  |  |  |  |  | 134420.50683 |  |  |  | 0.7327400000000001 | 134.79398 |  |
| 1 | GB300 NVL72 | 2 |  |  | 20 | 2026-10-24 | 2 |  | Dynamo TensorRT LLM | FP4 |  |  | 30913233.424528304 | 65536.05486 | MiniMax-M3 | mtp | agentic_traces | 1792825200 |  | agg |  | 65101.15925 | 434.89562 | 0.00511 | 0.0053 | 2.84814 | 0.00511 | 0.5159199999999999 | 7.04835 | 0.0053 | 0.6593 | 195.56627 | 188.76472 |  |  |  | semianalysisai/cc-traces-weka-062126 | 3606.80458 |  |  | 0.82741 |  | 0.007730000000000001 | 17.91977 | 0.007730000000000001 | 0.93652 | 129.30494 | agentic |  |  | 2983 |  |  |  | 157430.21086 |  |  |  |  |  |  |  |  |  |  |  |  | 144076.61911 |  |  |  | 0.65763 | 157.65199 |  |
| 1 | GB300 NVL72 | 2 |  |  | 45 | 2026-10-24 | 2 |  | Dynamo TensorRT LLM | FP4 |  |  | 53288713.08962265 | 112972.07175 | MiniMax-M3 | mtp | agentic_traces | 1792825200 |  | agg |  | 112228.33068 | 743.74107 | 0.01022 | 0.01041 | 6.0738900000000005 | 0.01022 | 0.91154 | 13.80091 | 0.01041 | 1.44845 | 97.84262 | 96.10262 |  |  |  | semianalysisai/cc-traces-weka-062126 | 3626.66068 |  |  | 1.44319 |  | 0.01506 | 31.79993 | 0.01506 | 2.47843 | 66.41741 | agentic |  |  | 5234 |  |  |  | 155526.96752 |  |  |  |  |  |  |  |  |  |  |  |  | 160340.2414 |  |  |  | 1.33614 | 79.62004 |  |
| 8 | GB300 NVL72 | 8 | 8192 | 1024 | 4096 | 2026-06-03 | 32 | 8K/1K | Dynamo SGLang | FP4 | 4 | 4 | 5487707.707961368 | 11633.9403408781 | DeepSeek-V4-Pro | mtp | single_turn | 1780470000 | 8 | disagg | 24 | 13788.8367866001 | 5169.2510037122 | 1.7095668970142002 | 3.5249994032292 | 84.8215044959215 | 0.0226665245235 | 62.4801501650363 | 84.404328815736 | 0.024277985922799998 | 62.0613167106897 | 44.1179237233 | 41.1895782122 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 1 | GB300 NVL72 | 8 |  |  | 240 | 2026-10-27 | 56 |  | Dynamo vLLM | FP4 | 8 | 8 | 9609098.740566038 | 20371.28933 | Kimi-K3 | mtp | agentic_traces | 1793084400 | 32 | disagg | 24 | 20225.46522 | 145.82411 | 0.013359999999999999 | 0.012960000000000001 | 7.20104 | 0.013359999999999999 | 1.24201 | 15.246129999999999 | 0.01302 | 1.89049 | 74.8502994012 | 77.1604938272 |  |  |  |  | 3629.67706 | 4119266314 | 29895887 |  |  | 0.01467 | 33.62478 | 0.01466 | 3.0001700000000002 | 68.166325835 | agentic | 1 | 8 | 29635 | 1325.9577 | 0.97055 | 2348.40252 | 138723.36086 | 1000.18519 | 170655.5 | 964.0 | 10.0 | 287461.2 | 2296.6 | 12.0 | 424185.6 | 3846.3 | 14.0 | 3.19744 | 8.16561 | 134476.74311 | 2058.31234 | 8.0 | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/40239967341/attempts/1 |  |  |  |
| 1 | GB300 NVL72 | 8 |  |  | 64 | 2026-10-27 | 24 |  | Dynamo vLLM | FP4 | 8 | 8 | 7227205.70754717 | 15321.6761 | Kimi-K3 | mtp | agentic_traces | 1793084400 | 16 | disagg | 8 | 15210.80035 | 110.87575 | 0.009880000000000002 | 0.00966 | 6.40164 | 0.009890000000000001 | 1.1807 | 11.67403 | 0.009720000000000001 | 1.94546 | 101.2145748988 | 103.5196687371 |  |  |  |  | 3626.68379 | 1326089086 | 9688566 |  |  | 0.011619999999999998 | 24.82246 | 0.011609999999999999 | 3.64182 | 86.0585197935 | agentic | 1 | 8 | 9773 | 1231.56077 | 0.97076 | 2169.81285 | 135470.61424 | 987.48296 | 165519.0 | 1050.0 | 4.0 | 295983.0 | 2232.6 | 5.0 | 511108.4 | 3670.4 | 6.0 | 1.80676 | 2.69498 | 154391.20839 | 1910.60218 | 2.0 | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/40239967341/attempts/1 |  |  |  |
| 1 | GB300 NVL72 | 8 |  |  | 240 | 2026-10-27 | 40 |  | Dynamo vLLM | FP4 | 8 | 8 | 11503497.891509434 | 24387.41553 | Kimi-K3 | mtp | agentic_traces | 1793084400 | 24 | disagg | 16 | 24209.80974 | 177.6058 | 0.01563 | 0.015009999999999999 | 11.659889999999999 | 0.01563 | 3.5989400000000002 | 20.722099999999998 | 0.01509 | 5.3600699999999994 | 63.9795265515 | 66.6222518321 |  |  |  |  | 3628.9559 | 3524111837 | 26051241 |  |  | 0.01715 | 44.11667 | 0.01714 | 10.697899999999999 | 58.3090379009 | agentic | 1 | 8 | 25871 | 1333.32808 | 0.96946 | 2371.97384 | 135837.55081 | 996.51904 | 166256.5 | 962.0 | 9.0 | 283308.0 | 2301.0 | 11.0 | 418481.0 | 3833.0 | 12.0 | 3.10981 | 7.13065 | 133536.57167 | 2038.96832 | 7.0 | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/40239967341/attempts/1 |  |  |  |
| 1 | GB300 NVL72 | 8 |  |  | 32 | 2026-10-27 | 24 |  | Dynamo vLLM | FP4 | 8 | 8 | 4535844.339622642 | 9615.99 | Kimi-K3 | mtp | agentic_traces | 1793084400 | 16 | disagg | 8 | 9560.28457 | 55.70543 | 0.00392 | 0.00421 | 2.9908200000000003 | 0.00392 | 0.84224 | 5.76584 | 0.00423 | 1.32892 | 255.1020408163 | 237.5296912114 |  |  |  |  | 3614.32349 | 829722466 | 4832101 |  |  | 0.00598 | 12.952879999999999 | 0.00597 | 2.1447 | 167.2240802676 | agentic | 1 | 8 | 4649 | 1399.5116 | 0.97993 | 2133.43389 | 178381.38675 | 1039.3846 | 237194.0 | 1119.0 | 2.0 | 353599.6 | 2656.6 | 3.0 | 517298.2 | 4162.6 | 4.0 | 1.23141 | 1.28611 | 159212.98603 | 1720.33078 | 1.0 | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/40239967341/attempts/1 |  |  |  |
| 16 | GB300 NVL72 | 16 |  |  | 336 | 2026-10-27 | 40 |  | Dynamo vLLM | FP4 | 8 | 8 | 12530448.452830188 | 26564.55072 | Kimi-K3 | mtp | agentic_traces | 1793084400 | 16 | disagg | 24 | 26358.7009 | 205.84982 | 0.027 | 0.026010000000000002 | 14.50431 | 0.02699 | 1.80931 | 28.941380000000002 | 0.02614 | 3.53166 | 37.037037037 | 38.4467512495 |  |  |  |  | 3629.59894 | 3854726043 | 30339423 |  |  | 0.02925 | 63.215489999999996 | 0.02924 | 7.08423 | 34.188034188 | agentic | 16 | 16 | 31619 | 1275.69243 | 0.96615 | 2430.64185 | 121030.40944 | 945.19406 | 147254.5 | 971.0 | 11.0 | 253870.0 | 2161.2 | 13.0 | 395004.5 | 3601.1 | 14.6 | 3.307 | 8.71232 | 127858.62446 | 1812.88868 | 8.0 | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/40239967341/attempts/1 |  |  |  |
| 1 | GB300 NVL72 | 8 |  |  | 16 | 2026-10-27 | 32 |  | Dynamo vLLM | FP4 | 8 | 8 | 1675596.1179245284 | 3552.26377 | Kimi-K3 | mtp | agentic_traces | 1793084400 | 24 | disagg | 8 | 3529.82194 | 22.44183 | 0.0029 | 0.0029500000000000004 | 1.906 | 0.0029 | 0.57228 | 3.76236 | 0.0029500000000000004 | 0.80414 | 344.8275862069 | 338.9830508475 |  |  |  |  | 3616.41633 | 409045497 | 2597090 |  |  | 0.0033599999999999997 | 8.342139999999999 | 0.00335 | 1.50424 | 297.619047619 | agentic | 1 | 8 | 2635 | 1612.21111 | 0.97614 | 2505.30901 | 155024.58558 | 985.61214 | 183500.5 | 940.5 | 1.0 | 279914.8 | 2421.4 | 2.0 | 525608.9 | 4215.3 | 3.0 | 0.91378 | 0.72843 | 152201.14193 | 1934.09233 | 0.0 | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/40239967341/attempts/1 |  |  |  |
| 1 | GB300 NVL72 | 8 |  |  | 24 | 2026-10-27 | 24 |  | Dynamo vLLM | FP4 | 8 | 8 | 3502964.4811320757 | 7426.2847 | Kimi-K3 | mtp | agentic_traces | 1793084400 | 16 | disagg | 8 | 7381.57542 | 44.70927 | 0.00342 | 0.00357 | 2.2958000000000003 | 0.00342 | 0.70904 | 4.732 | 0.0035800000000000003 | 1.0666099999999998 | 292.3976608187 | 280.1120448179 |  |  |  |  | 3604.3083 | 638915465 | 3867505 |  |  | 0.00453 | 11.146780000000001 | 0.00452 | 1.70425 | 220.7505518764 | agentic | 1 | 8 | 3842 | 1504.19622 | 0.97899 | 2256.20441 | 166197.64836 | 1006.63821 | 221622.0 | 1038.75 | 2.0 | 331233.5 | 2607.1 | 3.0 | 410275.2 | 4211.5 | 3.0 | 1.11933 | 1.06576 | 143804.44252 | 1761.82447 | 1.0 | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/40239967341/attempts/1 |  |  |  |
| 1 | Vera Rubin NVL72 | 16 |  |  | 48 | 2026-10-23 | 32 |  | Dynamo vLLM | FP4 | 8 | 8 | 3566468.424242424 | 11769.3458 | Kimi-K3 | mtp | agentic_traces | 1792738800 | 16 | disagg | 16 | 11692.6453 | 76.7005 | 0.00239 | 0.00237 | 1.4151500000000001 | 0.0023799999999999997 | 0.24768 | 2.8961900000000003 | 0.00237 | 0.33213 | 418.410041841 | 421.94092827 |  |  |  |  | 3621.81905 | 1355540754 | 8889452 |  |  | 0.00292 | 6.5305100000000005 | 0.0029100000000000003 | 0.5482899999999999 | 342.4657534247 | agentic | 1 | 16 | 8394 | 1295.99632 | 0.97652 | 2190.27508 | 161443.48999 | 1059.02442 | 212939.25 | 1080.75 | 3.0 | 364571.8 | 2547.7 | 4.0 | 605393.9 | 4094.7 | 5.0 | 1.63444 | 2.31787 | 175487.26723 | 2089.04172 | 2.0 |  |  |  |  |
| 1 | Vera Rubin NVL72 | 16 |  |  | 1 | 2026-10-23 | 24 |  | Dynamo vLLM | FP4 | 8 | 8 | 411908.39090909093 | 1359.29769 | Kimi-K3 | mtp | agentic_traces | 1792738800 | 16 | disagg | 8 | 1350.97396 | 8.32373 | 0.00108 | 0.00107 | 1.01164 | 0.00107 | 0.23469 | 2.08504 | 0.00106 | 0.27132999999999996 | 925.9259259259 | 934.5794392523 |  |  |  |  | 3594.80169 | 117589116 | 718134 |  |  | 0.00116 | 5.5425699999999996 | 0.00115 | 0.37338 | 862.0689655172 | agentic | 1 | 16 | 438 | 2119.93607 | 0.98266 | 2576.30726 | 266108.68265 | 1639.57078 | 433350.0 | 2249.75 | 0.0 | 583584.6 | 4782.0 | 1.0 | 643616.1 | 6031.5 | 1.0 | 0.34749 | 0.12163 | 211768.51254 | 2338.22019 | 0.0 |  |  |  |  |
| 1 | GB300 NVL72 | 16 |  |  | 1 | 2026-10-27 | 16 |  | Dynamo vLLM | FP4 |  |  | 735009.2500000001 | 1558.21961 | Kimi-K3 | mtp | agentic_traces | 1793084400 |  | agg |  | 1548.09681 | 10.1228 | 0.00219 | 0.00219 | 1.5155399999999999 | 0.0020499999999999997 | 0.72424 | 3.2183 | 0.00147 | 0.83604 | 456.6210045662 | 456.6210045662 |  |  |  |  | 3594.27369 | 89439952 | 582801 |  |  | 0.0023599999999999997 | 8.3437 | 0.0022299999999999998 | 1.29098 | 423.7288135593 | agentic |  |  | 296 | 2125.15878 | 0.98241 | 2805.83643 | 300772.08784 | 1966.70946 | 436282.75 | 2693.0 | 0.0 | 544789.5 | 5082.5 | 0.0 | 572144.5 | 6696.75 | 1.0 | 0.27965 | 0.08217 | 175607.5237 | 2552.71732 | 0.0 | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/40239967341/attempts/1 |  |  |  |
| 1 | Vera Rubin NVL72 | 16 |  |  | 160 | 2026-10-23 | 32 |  | Dynamo vLLM | FP4 | 8 | 8 | 9607527.56969697 | 31704.84098 | Kimi-K3 | mtp | agentic_traces | 1792738800 | 16 | disagg | 16 | 31479.10282 | 225.73815 | 0.00495 | 0.00491 | 3.02788 | 0.004940000000000001 | 0.63019 | 6.220680000000001 | 0.00492 | 0.88739 | 202.0202020202 | 203.66598778 |  |  |  |  | 3616.22158 | 3644240855 | 26122221 |  |  | 0.00584 | 13.53849 | 0.00583 | 1.52996 | 171.2328767123 | agentic | 1 | 16 | 24525 | 1338.64863 | 0.9729 | 2511.37365 | 148531.4232 | 1065.12595 | 190172.0 | 1018.0 | 9.0 | 315791.4 | 2453.6 | 11.0 | 462757.2 | 4105.8 | 12.0 | 2.98328 | 6.78208 | 146716.88448 | 2150.02482 | 7.0 |  |  |  |  |
| 1 | Vera Rubin NVL72 | 16 |  |  | 256 | 2026-10-23 | 32 |  | Dynamo vLLM | FP4 | 8 | 8 | 13562143.136363637 | 44755.07235 | Kimi-K3 | mtp | agentic_traces | 1792738800 | 16 | disagg | 16 | 44442.34824 | 312.72412 | 0.00824 | 0.00807 | 5.19384 | 0.00822 | 1.19108 | 10.23766 | 0.00807 | 1.96186 | 121.359223301 | 123.9157372986 |  |  |  |  | 3629.38562 | 5165591440 | 36442389 |  |  | 0.00977 | 22.75084 | 0.00973 | 3.4901999999999997 | 102.3541453429 | agentic | 1 | 16 | 36207 | 1298.94239 | 0.97171 | 2328.27729 | 142556.67208 | 1003.11777 | 181396.5 | 997.0 | 12.0 | 298210.4 | 2335.4 | 15.0 | 430262.8 | 3836.0 | 16.0 | 3.84008 | 9.97685 | 138174.91355 | 2052.57254 | 10.0 |  |  |  |  |
| 1 | Vera Rubin NVL72 | 16 |  |  | 64 | 2026-10-23 | 24 |  | Dynamo vLLM | FP4 | 8 | 8 | 6416380.621212121 | 21174.05605 | Kimi-K3 | mtp | agentic_traces | 1792738800 | 16 | disagg | 8 | 21028.55648 | 145.49957 | 0.00315 | 0.0030800000000000003 | 2.0187 | 0.00315 | 0.33096 | 3.8695399999999998 | 0.00309 | 0.6088300000000001 | 317.4603174603 | 324.6753246753 |  |  |  |  | 3603.51144 | 1819701294 | 12583429 |  |  | 0.004 | 8.497530000000001 | 0.004 | 1.15235 | 250.0 | agentic | 1 | 16 | 12108 | 1313.56341 | 0.97435 | 2349.31716 | 150201.47423 | 1039.26536 | 194399.5 | 1104.25 | 5.0 | 353924.5 | 2397.3 | 6.0 | 560743.6 | 3919.55 | 7.0 | 2.08689 | 3.36119 | 170499.66829 | 1964.66434 | 3.0 |  |  |  |  |
| 1 | Vera Rubin NVL72 | 16 |  |  | 8 | 2026-10-23 | 24 |  | Dynamo vLLM | FP4 | 8 | 8 | 908503.4363636364 | 2998.06134 | Kimi-K3 | mtp | agentic_traces | 1792738800 | 16 | disagg | 8 | 2977.84449 | 20.21684 | 0.00116 | 0.00126 | 0.84005 | 0.00116 | 0.17028000000000001 | 1.76142 | 0.00125 | 0.22707 | 862.0689655172 | 793.6507936508 |  |  |  |  | 3596.45868 | 258237796 | 1745020 |  |  | 0.0016899999999999999 | 4.22833 | 0.0016799999999999999 | 0.33704 | 591.7159763314 | agentic | 1 | 16 | 1481 | 1930.09091 | 0.98025 | 2458.81727 | 173553.45847 | 1178.26941 | 210351.0 | 1254.0 | 1.0 | 442025.0 | 3010.0 | 1.0 | 543494.0 | 4596.0 | 2.0 | 0.64991 | 0.41157 | 154851.11451 | 1909.77851 | 0.0 |  |  |  |  |
| 8 | B300 | 8 |  |  | 48 | 2026-11-06 | 8 |  | vLLM | FP4 |  |  | 12367762.7 | 23498.74913 | Kimi-K3.1-Pro | mtp | agentic_traces | 1793952000 |  | agg |  | 23337.5962 | 161.15293 | 0.0198 | 0.020989999999999998 | 9.25009 | 0.0198 | 0.54896 | 21.12101 | 0.020989999999999998 | 0.85505 | 50.49546 | 47.65119 |  | 1 |  | semianalysisai/cc-traces-weka-062126 | 3627.25099 |  |  | 1.30778 |  | 0.02711 | 48.38946 | 0.02711 | 1.40795 | 36.88022 | agentic |  |  | 4743 |  |  |  | 142781.0565 |  |  |  |  |  |  |  |  |  |  |  |  | 147170.94954 |  |  | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/42287685511 | 0.8256100000000001 | 43.80667 | https://huggingface.co/datasets/semianalysisai/cc-traces-weka-062126 |
| 1 | B300 | 8 |  |  | 8 | 2026-11-06 | 8 |  | vLLM | FP4 |  |  | 3679862.6210526316 | 6991.73898 | Kimi-K3.1-Pro | mtp | agentic_traces | 1793952000 |  | agg |  | 6940.63608 | 51.1029 | 0.00436 | 0.00511 | 2.6726300000000003 | 0.00436 | 0.38001999999999997 | 5.58256 | 0.00511 | 0.54648 | 229.33571 | 195.76394 |  | 1 |  | semianalysisai/cc-traces-weka-062126 | 3610.38727 |  |  | 0.36565 |  | 0.00719 | 13.615219999999999 | 0.00719 | 0.76958 | 139.0957 | agentic |  |  | 1321 |  |  |  | 151754.02952 |  |  |  |  |  |  |  |  |  |  |  |  | 128950.87508 |  |  | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/42287685511 | 0.55149 | 175.53899 | https://huggingface.co/datasets/semianalysisai/cc-traces-weka-062126 |
| 8 | B300 | 8 |  |  | 32 | 2026-11-06 | 8 |  | vLLM | FP4 |  |  | 9792260.757894738 | 18605.29544 | Kimi-K3.1-Pro | mtp | agentic_traces | 1793952000 |  | agg |  | 18485.89243 | 119.40301 | 0.01478 | 0.0166 | 6.8485 | 0.01478 | 0.43697 | 16.09806 | 0.0166 | 0.67195 | 67.65912 | 60.25679 |  | 1 |  | semianalysisai/cc-traces-weka-062126 | 3617.62355 |  |  | 0.97122 |  | 0.02255 | 39.710989999999995 | 0.02255 | 1.05478 | 44.34366 | agentic |  |  | 3511 |  |  |  | 152378.23925 |  |  |  |  |  |  |  |  |  |  |  |  | 137693.51551 |  |  | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/42287685511 | 0.621 | 55.98622 | https://huggingface.co/datasets/semianalysisai/cc-traces-weka-062126 |
| 1 | B300 | 8 |  |  | 16 | 2026-11-06 | 8 |  | vLLM | FP4 |  |  | 6185673.878947369 | 11752.78037 | Kimi-K3.1-Pro | mtp | agentic_traces | 1793952000 |  | agg |  | 11673.13036 | 79.65001 | 0.00696 | 0.00851 | 3.25027 | 0.00696 | 0.40212 | 7.72884 | 0.00851 | 0.6369 | 143.71275 | 117.49398 |  | 1 |  | semianalysisai/cc-traces-weka-062126 | 3626.72278 |  |  | 0.65177 |  | 0.01199 | 17.6369 | 0.01199 | 0.97136 | 83.37659 | agentic |  |  | 2363 |  |  |  | 143326.9835 |  |  |  |  |  |  |  |  |  |  |  |  | 143970.49961 |  |  | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/42287685511 | 0.57798 | 109.79567 | https://huggingface.co/datasets/semianalysisai/cc-traces-weka-062126 |
| 8 | GB300 NVL72 | 8 | 1024 | 1024 | 48 | 2026-02-03 | 34 | 1K/1K | Dynamo TensorRT LLM | FP4 | 2 | 2 | 179785.6728078302 | 381.1456263526 | DeepSeek-R1 | mtp | single_turn | 1770105600 | 32 | disagg | 2 | 3228.8293450463 | 203.1653939343 | 1.1304270030232 | 1.0034410328310999 | 6.2847666545131 | 0.0047713961568 | 2.056498050486 | 6.3490602587504 | 0.0045585434158 | 2.1325175706398998 | 209.5822621185 | 219.3683176374 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 8 | GB300 NVL72 | 8 | 8192 | 1024 | 33 | 2026-02-03 | 26 | 8K/1K | Dynamo TensorRT LLM | FP4 | 2 | 2 | 760598.0670237736 | 1612.4679020904 | DeepSeek-R1 | mtp | single_turn | 1770105600 | 24 | disagg | 2 | 18618.1736984106 | 195.3257523971 | 1.2316779949761 | 1.0714976970518 | 6.2377607244707 | 0.0050761286672 | 1.6779240304895 | 6.1745750056781 | 0.0046900364699 | 1.8366329692255 | 197.0005225557 | 213.2179582007 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 1 | B300 | 8 |  |  | 4 | 2026-11-06 | 8 |  | vLLM | FP4 |  |  | 2148548.6315789474 | 4082.2424 | Kimi-K3.1-Pro | mtp | agentic_traces | 1793952000 |  | agg |  | 4054.07326 | 28.16914 | 0.00312 | 0.0036 | 1.8001500000000001 | 0.00312 | 0.36654000000000003 | 4.8283000000000005 | 0.0036 | 0.51281 | 320.59013 | 277.89654 |  | 1 |  | semianalysisai/cc-traces-weka-062126 | 3602.90508 |  |  | 0.17274 |  | 0.00432 | 13.1565 | 0.00432 | 0.79953 | 231.74807 | agentic |  |  | 622 |  |  |  | 187864.19453 |  |  |  |  |  |  |  |  |  |  |  |  | 181944.712 |  |  | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/42287685511 | 0.60527 | 265.21742 | https://huggingface.co/datasets/semianalysisai/cc-traces-weka-062126 |
| 1 | B300 | 8 |  |  | 1 | 2026-11-06 | 8 |  | vLLM | FP4 |  |  | 1450271.2368421054 | 2755.51535 | Kimi-K3.1-Pro | mtp | agentic_traces | 1793952000 |  | agg |  | 2736.62846 | 18.88688 | 0.0028599999999999997 | 0.00299 | 3.28814 | 0.0028599999999999997 | 0.52664 | 6.3419300000000005 | 0.00299 | 0.65522 | 349.177 | 333.96069 |  | 1 |  | semianalysisai/cc-traces-weka-062126 | 3596.94086 |  |  | 0.07739 |  | 0.00344 | 15.71923 | 0.00344 | 0.78873 | 290.56986 | agentic |  |  | 279 |  |  |  | 282250.63082 |  |  |  |  |  |  |  |  |  |  |  |  | 165048.1481 |  |  | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/42287685511 | 0.66343 | 341.08745 | https://huggingface.co/datasets/semianalysisai/cc-traces-weka-062126 |
| 8 | B300 | 8 |  |  | 56 | 2026-11-06 | 8 |  | vLLM | FP4 |  |  | 12988949.173684211 | 24679.00343 | Kimi-K3.1-Pro | mtp | agentic_traces | 1793952000 |  | agg |  | 24490.6997 | 188.30373 | 0.0228 | 0.02389 | 13.04684 | 0.0228 | 0.9141900000000001 | 25.46584 | 0.02389 | 2.25489 | 43.85237 | 41.86331 |  | 1 |  | semianalysisai/cc-traces-weka-062126 | 3629.9388 |  |  | 1.51764 |  | 0.02985 | 55.56483 | 0.02985 | 5.70412 | 33.502 | agentic |  |  | 5509 |  |  |  | 129097.46379 |  |  |  |  |  |  |  |  |  |  |  |  | 142694.81951 |  |  | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/42287685511 | 2.776 | 39.06575 | https://huggingface.co/datasets/semianalysisai/cc-traces-weka-062126 |
| 32 | GB300 NVL72 | 40 |  |  | 384 | 2026-11-08 | 56 |  | Dynamo vLLM | FP4 |  |  | 14344240.669811321 | 30409.79022 | Kimi-K3.1-Pro | mtp | agentic_traces | 1794124800 |  | disagg |  | 30181.34371 | 228.44651 | 0.01302 | 0.01298 | 9.43652 | 0.01302 | 2.2999099999999997 | 16.55209 | 0.01298 | 3.6817699999999998 | 76.78739 | 77.02674 |  |  |  | semianalysisai/cc-traces-weka-062126 | 3629.55145 |  |  | 12.7294 |  | 0.013869999999999999 | 34.89559 | 0.013869999999999999 | 8.07976 | 72.10666 | agentic |  |  | 46196 |  |  |  | 132793.0 |  |  |  |  |  |  |  |  |  |  |  |  | 134628.85055 |  |  | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/42501370812 | 4.28813 | 74.76437 | https://huggingface.co/datasets/semianalysisai/cc-traces-weka-062126 |
| 8 | GB300 NVL72 | 16 |  |  | 32 | 2026-11-08 | 32 |  | Dynamo vLLM | FP4 |  |  | 3558881.1886792453 | 7544.82812 | Kimi-K3.1-Pro | mtp | agentic_traces | 1794124800 |  | disagg |  | 7501.52597 | 43.30215 | 0.00362 | 0.0037 | 2.42061 | 0.00362 | 0.6443300000000001 | 4.69221 | 0.0037 | 0.90907 | 276.42083 | 270.20139 |  |  |  | semianalysisai/cc-traces-weka-062126 | 3612.07639 |  |  | 1.32678 |  | 0.004730000000000001 | 10.50818 | 0.004730000000000001 | 1.47524 | 211.32452 | agentic |  |  | 4792 |  |  |  | 180942.13585 |  |  |  |  |  |  |  |  |  |  |  |  | 161763.9562 |  |  | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/42501370812 | 0.92367 | 243.30279 | https://huggingface.co/datasets/semianalysisai/cc-traces-weka-062126 |
| 32 | GB300 NVL72 | 40 |  |  | 192 | 2026-11-08 | 48 |  | Dynamo vLLM | FP4 |  |  | 9657684.688679244 | 20474.29154 | Kimi-K3.1-Pro | mtp | agentic_traces | 1794124800 |  | disagg |  | 20326.20569 | 148.08586 | 0.01 | 0.01 | 6.213439999999999 | 0.01 | 1.4464000000000001 | 12.19285 | 0.01 | 2.03811 | 100.0001 | 99.97714 |  |  |  | semianalysisai/cc-traces-weka-062126 | 3629.25751 |  |  | 6.93881 |  | 0.01067 | 26.52068 | 0.01067 | 3.49927 | 93.73173 | agentic |  |  | 25177 |  |  |  | 140640.80943 |  |  |  |  |  |  |  |  |  |  |  |  | 139008.34425 |  |  | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/42501370812 | 2.25225 | 97.10338 | https://huggingface.co/datasets/semianalysisai/cc-traces-weka-062126 |
| 8 | GB300 NVL72 | 16 |  |  | 20 | 2026-11-08 | 32 |  | Dynamo vLLM | FP4 |  |  | 2173758.6226415094 | 4608.36828 | Kimi-K3.1-Pro | mtp | agentic_traces | 1794124800 |  | disagg |  | 4579.60457 | 28.76371 | 0.00311 | 0.00318 | 1.99301 | 0.00311 | 0.56945 | 4.0020999999999995 | 0.00318 | 0.7159800000000001 | 321.57193 | 314.43574 |  |  |  | semianalysisai/cc-traces-weka-062126 | 3603.29918 |  |  | 0.87899 |  | 0.00379 | 9.24311 | 0.00379 | 1.17967 | 264.09433 | agentic |  |  | 3168 |  |  |  | 166683.69066 |  |  |  |  |  |  |  |  |  |  |  |  | 148933.53257 |  |  | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/42501370812 | 0.81036 | 283.98507 | https://huggingface.co/datasets/semianalysisai/cc-traces-weka-062126 |
| 8 | GB300 NVL72 | 16 |  |  | 56 | 2026-11-08 | 32 |  | Dynamo vLLM | FP4 |  |  | 5829840.339622642 | 12359.26152 | Kimi-K3.1-Pro | mtp | agentic_traces | 1794124800 |  | disagg |  | 12276.20623 | 83.05528 | 0.00509 | 0.00514 | 3.41309 | 0.00509 | 0.8115 | 6.43799 | 0.00514 | 1.29691 | 196.5244 | 194.39423 |  |  |  | semianalysisai/cc-traces-weka-062126 | 3612.00504 |  |  | 2.59845 |  | 0.00625 | 14.1564 | 0.00625 | 2.20342 | 160.01049 | agentic |  |  | 9384 |  |  |  | 151207.9074 |  |  |  |  |  |  |  |  |  |  |  |  | 167756.86053 |  |  | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/42501370812 | 1.3186900000000001 | 175.29217 | https://huggingface.co/datasets/semianalysisai/cc-traces-weka-062126 |
| 8 | GB300 NVL72 | 8 | 1024 | 1024 | 8 | 2026-02-03 | 34 | 1K/1K | Dynamo TensorRT LLM | FP4 | 2 | 2 | 57577.64638075472 | 122.0646103272 | DeepSeek-R1 | mtp | single_turn | 1770105600 | 32 | disagg | 2 | 1033.7031177888 | 65.0872036109 | 0.8192278609495001 | 0.7204120387978 | 3.5515489155077 | 0.0035179187086 | 0.289771859505 | 3.4310676972855 | 0.00332958796 | 0.3423012341402 | 284.2589845979 | 300.3374627808 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 8 | H200 | 8 | 1024 | 1024 | 512 | 2026-03-23 | 72 | 1K/1K | Dynamo TensorRT LLM | FP8 | 8 | 8 | 471302.08435043803 | 645.6838555601 | DeepSeek-R1 | mtp | single_turn | 1774249200 | 64 | disagg | 8 | 2904.2384548479 | 363.3645306491 | 4.2018386045238 | 3.9501280332956 | 18.861431284924002 | 0.0183580459615 | 1.3376573454589 | 19.2395805270597 | 0.0184382459675 | 2.2648312946147 | 54.4720283465 | 54.2350938241 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 4 | GB300 NVL72 | 4 |  |  | 46 | 2026-11-03 | 4 |  | Dynamo SGLang | FP4 |  |  | 30012161.759433962 | 63625.78293 | Muse-Spark-Open | off | agentic_traces | 1793692800 |  | agg |  | 63204.81883 | 420.96409 | 0.0077599999999999995 | 0.00767 | 5.43372 | 0.0077599999999999995 | 0.98493 | 10.85612 | 0.00767 | 2.28131 | 128.91856 | 130.32292 |  |  |  | semianalysisai/cc-traces-weka-062126 | 3627.2767 |  |  | 1.6163 |  | 0.009859999999999999 | 24.941740000000003 | 0.009859999999999999 | 5.21069 | 101.44731 | agentic |  |  | 5852 |  |  |  | 156706.3339 |  |  |  |  |  |  |  |  |  |  |  |  | 162837.43677 |  |  | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/41611203638 | 1.81302 | 111.88822 | https://huggingface.co/datasets/semianalysisai/cc-traces-weka-062126 |
| 8 | GB300 NVL72 | 16 |  |  | 352 | 2026-11-03 | 40 |  | Dynamo SGLang | FP4 |  |  | 23113366.29245283 | 49000.33654 | Muse-Spark-Open | off | agentic_traces | 1793692800 |  | disagg |  | 48636.72954 | 363.607 | 0.0057 | 0.00558 | 5.19398 | 0.0057 | 2.32594 | 9.11174 | 0.00558 | 3.12617 | 175.32934 | 179.15842 |  |  |  | semianalysisai/cc-traces-weka-062126 | 3628.70731 |  |  | 14.22431 |  | 0.00643 | 18.11302 | 0.00643 | 4.371020000000001 | 155.4401 | agentic |  |  | 51494 |  |  |  | 137094.38451 |  |  |  |  |  |  |  |  |  |  |  |  | 135052.80082 |  |  | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/41611203638 | 3.1053200000000003 | 162.32858 | https://huggingface.co/datasets/semianalysisai/cc-traces-weka-062126 |
| 16 | GB300 NVL72 | 32 |  |  | 352 | 2026-11-08 | 32 |  | Dynamo vLLM | FP4 |  |  | 17403030.235849056 | 36894.4241 | Kimi-K3.1-Pro | mtp | agentic_traces | 1794124800 |  | disagg |  | 36608.18965 | 286.23445 | 0.020030000000000003 | 0.01977 | 14.52332 | 0.020030000000000003 | 4.21866 | 26.83152 | 0.01977 | 7.9144499999999995 | 49.93585 | 50.57251 |  |  |  | semianalysisai/cc-traces-weka-062126 | 3627.71129 |  |  | 9.5244 |  | 0.02079 | 56.35976 | 0.02079 | 11.37865 | 48.10902 | agentic |  |  | 34546 |  |  |  | 123016.44677 |  |  |  |  |  |  |  |  |  |  |  |  | 128046.37286 |  |  | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/42501370812 | 6.167479999999999 | 48.94979 | https://huggingface.co/datasets/semianalysisai/cc-traces-weka-062126 |
| 8 | GB300 NVL72 | 16 |  |  | 112 | 2026-11-03 | 40 |  | Dynamo SGLang | FP4 |  |  | 8862825.882075472 | 18789.19087 | Muse-Spark-Open | off | agentic_traces | 1793692800 |  | disagg |  | 18650.01187 | 139.17899 | 0.00289 | 0.00282 | 3.05849 | 0.00289 | 0.9719800000000001 | 4.89838 | 0.00282 | 1.8363399999999999 | 346.34305 | 354.17012 |  |  |  | semianalysisai/cc-traces-weka-062126 | 3613.72837 |  |  | 5.33537 |  | 0.00326 | 10.4879 | 0.00326 | 4.44158 | 306.82724 | agentic |  |  | 19235 |  |  |  | 140153.00639 |  |  |  |  |  |  |  |  |  |  |  |  | 147691.28383 |  |  | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/41611203638 | 2.27903 | 323.96743 | https://huggingface.co/datasets/semianalysisai/cc-traces-weka-062126 |
| 8 | GB300 NVL72 | 16 |  |  | 672 | 2026-11-03 | 56 |  | Dynamo SGLang | FP4 |  |  | 25458527.811320756 | 53972.07896 | Muse-Spark-Open | off | agentic_traces | 1793692800 |  | disagg |  | 53547.18764 | 424.89131 | 0.00644 | 0.00631 | 5.57932 | 0.00644 | 2.4091 | 10.99639 | 0.00631 | 4.398020000000001 | 155.21663 | 158.42422 |  |  |  | semianalysisai/cc-traces-weka-062126 | 3629.67404 |  |  | 23.95748 |  | 0.00735 | 19.64538 | 0.00735 | 4.44696 | 136.06132 | agentic |  |  | 86775 |  |  |  | 125428.92372 |  |  |  |  |  |  |  |  |  |  |  |  | 110767.4765 |  |  | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/41611203638 | 3.13665 | 143.70315 | https://huggingface.co/datasets/semianalysisai/cc-traces-weka-062126 |
| 4 | GB300 NVL72 | 4 |  |  | 56 | 2026-11-03 | 4 |  | Dynamo SGLang | FP4 |  |  | 33166805.74528302 | 70313.62818 | Muse-Spark-Open | off | agentic_traces | 1793692800 |  | agg |  | 69800.95542 | 512.67277 | 0.01051 | 0.0104 | 7.52838 | 0.01051 | 1.14518 | 14.23533 | 0.0104 | 2.79108 | 95.15224 | 96.17273 |  |  |  | semianalysisai/cc-traces-weka-062126 | 3626.79301 |  |  | 2.00691 |  | 0.013519999999999999 | 32.00854 | 0.013519999999999999 | 6.667470000000001 | 73.94004 | agentic |  |  | 7266 |  |  |  | 139363.4005 |  |  |  |  |  |  |  |  |  |  |  |  | 154213.9544 |  |  | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/41611203638 | 3.2365500000000003 | 84.00648 | https://huggingface.co/datasets/semianalysisai/cc-traces-weka-062126 |
| 8 | GB300 NVL72 | 16 |  |  | 120 | 2026-11-03 | 48 |  | Dynamo SGLang | FP4 |  |  | 7691755.202830189 | 16306.52103 | Muse-Spark-Open | off | agentic_traces | 1793692800 |  | disagg |  | 16183.32282 | 123.19821 | 0.00265 | 0.0026 | 2.7082800000000002 | 0.00265 | 0.8292999999999999 | 4.57346 | 0.0026 | 1.70983 | 377.80497 | 384.1964 |  |  |  | semianalysisai/cc-traces-weka-062126 | 3620.59019 |  |  | 5.57293 |  | 0.00299 | 10.05491 | 0.00299 | 4.18215 | 334.37544 | agentic |  |  | 20136 |  |  |  | 139673.84942 |  |  |  |  |  |  |  |  |  |  |  |  | 146519.43731 |  |  | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/41611203638 | 1.9543 | 350.28465 | https://huggingface.co/datasets/semianalysisai/cc-traces-weka-062126 |
| 8 | GB300 NVL72 | 16 |  |  | 1 | 2026-11-08 | 16 |  | Dynamo vLLM | FP4 |  |  | 655497.4386792453 | 1389.65457 | Kimi-K3.1-Pro | mtp | agentic_traces | 1794124800 |  | disagg |  | 1380.18456 | 9.47001 | 0.00272 | 0.00276 | 3.1691599999999998 | 0.00272 | 0.76202 | 6.14592 | 0.00276 | 0.8065800000000001 | 367.57043 | 361.66544 |  |  |  | semianalysisai/cc-traces-weka-062126 | 3591.71282 |  |  | 0.07778 |  | 0.0029 | 15.35972 | 0.0029 | 1.27748 | 344.56748 | agentic |  |  | 280 |  |  |  | 283270.08929 |  |  |  |  |  |  |  |  |  |  |  |  | 165628.85185 |  |  | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/42501370812 | 1.0811099999999998 | 359.02843 | https://huggingface.co/datasets/semianalysisai/cc-traces-weka-062126 |
| 8 | GB300 NVL72 | 16 |  |  | 48 | 2026-11-08 | 24 |  | Dynamo vLLM | FP4 |  |  | 6204236.976415094 | 13152.98239 | Kimi-K3.1-Pro | mtp | agentic_traces | 1794124800 |  | disagg |  | 13066.74216 | 86.24023 | 0.00591 | 0.00584 | 3.56602 | 0.00591 | 0.73527 | 7.00123 | 0.00584 | 1.08392 | 169.13926 | 171.33855 |  |  |  | semianalysisai/cc-traces-weka-062126 | 3623.24705 |  |  | 2.00166 |  | 0.0070999999999999995 | 15.3339 | 0.0070999999999999995 | 1.86259 | 140.92009 | agentic |  |  | 7251 |  |  |  | 156703.46683 |  |  |  |  |  |  |  |  |  |  |  |  | 170175.91627 |  |  | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/42501370812 | 1.13542 | 152.75112 | https://huggingface.co/datasets/semianalysisai/cc-traces-weka-062126 |
| 4 | GB300 NVL72 | 4 |  |  | 50 | 2026-11-03 | 4 |  | Dynamo SGLang | FP4 |  |  | 31270030.018867925 | 66292.46364 | Muse-Spark-Open | off | agentic_traces | 1793692800 |  | agg |  | 65831.04758 | 461.41606 | 0.00886 | 0.00862 | 6.10205 | 0.00886 | 0.99622 | 11.890049999999999 | 0.00862 | 2.4464200000000003 | 112.91366 | 116.0351 |  |  |  | semianalysisai/cc-traces-weka-062126 | 3629.18718 |  |  | 1.8037 |  | 0.01088 | 27.19696 | 0.01088 | 6.167 | 91.89956 | agentic |  |  | 6534 |  |  |  | 146258.4596 |  |  |  |  |  |  |  |  |  |  |  |  | 155913.52216 |  |  | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/41611203638 | 2.0420599999999998 | 100.29077 | https://huggingface.co/datasets/semianalysisai/cc-traces-weka-062126 |
| 32 | Vera Rubin NVL72 | 32 |  |  | 1229 | 2026-10-05 | 48 |  | Dynamo TensorRT LLM | FP4 | 8 | 8 | 41679510.3969697 | 137542.38431 | DeepSeek-V4-Pro | mtp | agentic_traces | 1791183600 | 32 | disagg | 16 | 409634.33232 | 1496.410305 | 0.00491 | 0.0050999999999999995 | 7.114439999999999 | 0.004889999999999999 | 4.02537 | 10.61668 | 0.0049900000000000005 | 5.58859 | 203.66598778 | 196.0784313725 |  |  |  |  | 3629.12782 | 23793566998 | 174097308 |  |  | 0.00536 | 21.84798 | 0.00523 | 10.43738 | 186.5671641791 | agentic | 32 | 32 | 168229 | 1356.24147 | 0.97096 | 2537.39624 | 141389.68669 | 1033.00416 | 179124.0 | 1032.0 | 54.0 | 307931.2 | 2416.0 | 60.0 | 445835.0 | 3963.0 | 64.0 | 11.54729 | 46.36907 | 139796.20753 | 2009.9222 | 47.0 |  |  |  |  |
| 16 | Vera Rubin NVL72 | 16 |  |  | 2934 | 2026-10-05 | 46 |  | Dynamo TensorRT LLM | FP4 | 6 | 6 | 85076826.92121212 | 280753.52884 | DeepSeek-V4-Pro | mtp | agentic_traces | 1791183600 | 16 | disagg | 30 | 427324.996916 | 5932.02616875 | 0.00967 | 0.00983 | 10.86143 | 0.009640000000000001 | 4.538189999999999 | 17.15205 | 0.009710000000000002 | 7.502689999999999 | 103.4126163392 | 101.7293997965 |  |  |  |  | 3629.88653 | 46585614014 | 345909594 |  |  | 0.0103 | 36.75049 | 0.01016 | 17.10651 | 97.0873786408 | agentic | 16 | 16 | 338941 | 1349.42158 | 0.96968 | 2539.42961 | 137293.0319 | 1016.46396 | 170770.0 | 1017.0 | 106.0 | 296103.0 | 2372.0 | 115.0 | 436015.0 | 3893.0 | 122.0 | 19.39549 | 93.39598 | 137603.59133 | 1981.70018 | 95.0 |  |  |  |  |
| 32 | Vera Rubin NVL72 | 32 |  |  | 368 | 2026-10-05 | 38 |  | Dynamo TensorRT LLM | FP4 | 6 | 6 | 14662177.863636365 | 48385.18695 | DeepSeek-V4-Pro | mtp | agentic_traces | 1791183600 | 32 | disagg | 6 | 304155.4970133333 | 428.253813125 | 0.0044599999999999996 | 0.00458 | 11.15878 | 0.0044599999999999996 | 8.36618 | 13.37284 | 0.00457 | 8.786700000000002 | 224.2152466368 | 218.3406113537 |  |  |  |  | 3629.92091 | 6628971190 | 49875447 |  |  | 0.00466 | 22.129 | 0.0046500000000000005 | 12.85517 | 214.5922746781 | agentic | 32 | 32 | 47949 | 1360.87099 | 0.97008 | 2544.92908 | 138154.33882 | 1037.45394 | 173583.0 | 1048.0 | 16.0 | 298294.2 | 2423.2 | 19.0 | 444072.8 | 3953.6 | 20.0 | 4.08821 | 13.2161 | 140955.32349 | 1979.81466 | 13.0 |  |  |  |  |
| 32 | Vera Rubin NVL72 | 32 |  |  | 2087 | 2026-10-05 | 56 |  | Dynamo TensorRT LLM | FP4 | 6 | 6 | 59153954.13333333 | 195208.04864 | DeepSeek-V4-Pro | mtp | agentic_traces | 1791183600 | 32 | disagg | 24 | 452181.9592266667 | 2477.6157 | 0.005679999999999999 | 0.00584 | 7.12074 | 0.00567 | 3.05868 | 11.47297 | 0.00578 | 5.6712 | 176.0563380282 | 171.2328767123 |  |  |  |  | 3629.47278 | 39402135064 | 288460617 |  |  | 0.00605 | 25.03674 | 0.00597 | 12.92072 | 165.2892561983 | agentic | 32 | 32 | 279170 | 1359.60853 | 0.97086 | 2540.61401 | 141090.98668 | 1030.76276 | 178317.0 | 1025.0 | 89.0 | 304314.5 | 2408.0 | 96.0 | 440032.75 | 3955.0 | 101.6 | 17.15888 | 76.92725 | 139176.5592 | 2008.31513 | 79.0 |  |  |  |  |
| 4 | Vera Rubin NVL72 | 4 |  |  | 48 | 2026-10-05 | 28 |  | Dynamo TensorRT LLM | FP4 | 4 | 4 | 3636870.5212121215 | 12001.67272 | DeepSeek-V4-Pro | mtp | agentic_traces | 1791183600 | 24 | disagg | 4 | 83467.41725 | 90.7152866667 | 0.00377 | 0.00385 | 2.2537 | 0.00376 | 0.39518000000000003 | 4.6452 | 0.0038399999999999997 | 0.6688 | 265.2519893899 | 259.7402597403 |  |  |  |  | 3618.13362 | 1209019139 | 7877285 |  |  | 0.00435 | 10.58005 | 0.00435 | 0.97677 | 229.8850574713 | agentic | 4 | 4 | 7389 | 1295.12816 | 0.9764 | 2136.14098 | 163484.24361 | 1066.08215 | 210933.0 | 1084.0 | 3.0 | 369210.2 | 2587.0 | 4.0 | 609729.4 | 4161.2 | 5.0 | 1.55531 | 2.04258 | 172808.06158 | 2140.86625 | 2.0 |  |  |  |  |
| 4 | Vera Rubin NVL72 | 4 |  |  | 6 | 2026-10-05 | 28 |  | Dynamo TensorRT LLM | FP4 | 4 | 4 | 419821.5363636364 | 1385.41107 | DeepSeek-V4-Pro | mtp | agentic_traces | 1791183600 | 24 | disagg | 4 | 9623.94328 | 12.3223683333 | 0.00315 | 0.0032 | 2.26625 | 0.00315 | 0.32151 | 4.64596 | 0.00319 | 0.7459199999999999 | 317.4603174603 | 312.5 |  |  |  |  | 3602.98627 | 139210803 | 1065538 |  |  | 0.00323 | 11.844790000000001 | 0.00323 | 0.96447 | 309.5975232198 | agentic | 4 | 4 | 864 | 1747.29737 | 0.97738 | 2716.22965 | 160532.1088 | 1233.25926 | 215130.75 | 1314.0 | 0.0 | 442877.9 | 3064.5 | 1.0 | 537932.7 | 4564.35 | 1.0 | 0.50924 | 0.24289 | 154394.73715 | 2062.84297 | 0.0 |  |  |  |  |
| 8 | GB300 NVL72 | 16 |  |  | 160 | 2026-11-03 | 24 |  | Dynamo SGLang | FP4 |  |  | 19375447.429245286 | 41075.94855 | Muse-Spark-Open | off | agentic_traces | 1793692800 |  | disagg |  | 40779.59108 | 296.35747 | 0.00451 | 0.00442 | 3.92281 | 0.00451 | 1.51732 | 6.9276599999999995 | 0.00442 | 2.00756 | 221.53375 | 226.06448 |  |  |  | semianalysisai/cc-traces-weka-062126 | 3628.2568 |  |  | 6.67597 |  | 0.0049900000000000005 | 14.24284 | 0.0049900000000000005 | 3.6625799999999997 | 200.466 | agentic |  |  | 24168 |  |  |  | 146930.31649 |  |  |  |  |  |  |  |  |  |  |  |  | 145149.8807 |  |  | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/41611203638 | 2.46152 | 208.97327 | https://huggingface.co/datasets/semianalysisai/cc-traces-weka-062126 |
| 8 | GB300 NVL72 | 16 |  |  | 224 | 2026-11-03 | 32 |  | Dynamo SGLang | FP4 |  |  | 15935385.783018868 | 33783.01786 | Muse-Spark-Open | off | agentic_traces | 1793692800 |  | disagg |  | 33520.27298 | 262.74488 | 0.00393 | 0.00396 | 4.45628 | 0.00393 | 2.42698 | 8.23682 | 0.00396 | 4.19451 | 254.29066 | 252.5379 |  |  |  | semianalysisai/cc-traces-weka-062126 | 3627.26388 |  |  | 8.38768 |  | 0.00445 | 13.52917 | 0.00445 | 4.44426 | 224.72916 | agentic |  |  | 30356 |  |  |  | 128171.69647 |  |  |  |  |  |  |  |  |  |  |  |  | 102781.07916 |  |  | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/41611203638 | 3.2759699999999996 | 237.70842 | https://huggingface.co/datasets/semianalysisai/cc-traces-weka-062126 |
| 8 | GB300 NVL72 | 16 |  |  | 32 | 2026-11-03 | 40 |  | Dynamo SGLang | FP4 |  |  | 3090150.8679245287 | 6551.11984 | Muse-Spark-Open | off | agentic_traces | 1793692800 |  | disagg |  | 6512.76685 | 38.35298 | 0.00188 | 0.00191 | 1.2658 | 0.00188 | 0.24114 | 2.61769 | 0.00191 | 0.43454000000000004 | 530.93 | 523.54937 |  |  |  | semianalysisai/cc-traces-weka-062126 | 3602.08818 |  |  | 1.41163 |  | 0.0022299999999999998 | 6.21557 | 0.0022299999999999998 | 0.91773 | 448.45775 | agentic |  |  | 5073 |  |  |  | 184975.83698 |  |  |  |  |  |  |  |  |  |  |  |  | 165033.07423 |  |  | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/41611203638 | 0.40194 | 487.84381 | https://huggingface.co/datasets/semianalysisai/cc-traces-weka-062126 |
| 8 | GB300 NVL72 | 16 |  |  | 56 | 2026-11-03 | 40 |  | Dynamo SGLang | FP4 |  |  | 5334148.646226415 | 11308.39513 | Muse-Spark-Open | off | agentic_traces | 1793692800 |  | disagg |  | 11233.56658 | 74.82856 | 0.00222 | 0.00218 | 1.8635599999999999 | 0.00222 | 0.40539 | 3.28534 | 0.00218 | 0.9223 | 451.19302 | 458.38076 |  |  |  | semianalysisai/cc-traces-weka-062126 | 3602.41484 |  |  | 2.87229 |  | 0.00248 | 7.4684 | 0.00248 | 2.2210199999999998 | 402.54784 | agentic |  |  | 10325 |  |  |  | 156776.62731 |  |  |  |  |  |  |  |  |  |  |  |  | 171401.71721 |  |  | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/41611203638 | 1.03403 | 420.38577 | https://huggingface.co/datasets/semianalysisai/cc-traces-weka-062126 |
| 16 | GB300 NVL72 | 16 |  |  | 160 | 2026-10-23 | 24 |  | Dynamo SGLang | FP4 | 1 | 1 | 18207842.08018868 | 38600.62521 | DeepSeek-V4-Pro | off | agentic_traces | 1792738800 | 16 | disagg | 8 | 114956.40762 | 422.734005 | 0.006019999999999999 | 0.00637 | 4.93337 | 0.00601 | 1.7774100000000002 | 8.60041 | 0.00636 | 2.31627 | 166.1129568106 | 156.9858712716 |  |  |  |  | 3628.76019 | 6676806714 | 24654608 |  |  | 0.00652 | 17.53584 | 0.0065 | 4.11364 | 153.3742331288 | agentic | 16 | 16 | 23208 | 1339.56768 | 0.97216 | 2505.5814 | 143794.97962 | 1057.56657 | 183578.75 | 1016.0 | 8.0 | 302704.1 | 2449.9 | 11.0 | 444694.25 | 4054.9 | 12.0 | 3.27654 | 6.40342 | 142931.504 | 2113.60231 | 6.0 | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/39763988786/attempts/2 |  |  |  |
| 16 | GB300 NVL72 | 16 |  |  | 300 | 2026-10-23 | 24 |  | Dynamo SGLang | FP4 | 1 | 1 | 31991827.372641508 | 67822.67403 | DeepSeek-V4-Pro | off | agentic_traces | 1792738800 | 16 | disagg | 8 | 201979.33821 | 744.341925 | 0.00684 | 0.0072699999999999996 | 7.0831800000000005 | 0.00683 | 2.83837 | 10.922379999999999 | 0.00726 | 4.06639 | 146.1988304094 | 137.5515818432 |  |  |  |  | 3629.68755 | 11733649612 | 43327191 |  |  | 0.00735 | 21.41404 | 0.00732 | 7.79959 | 136.0544217687 | agentic | 16 | 16 | 42439 | 1327.53498 | 0.97096 | 2390.33388 | 138197.76908 | 1018.58333 | 174674.5 | 1021.0 | 15.0 | 293491.4 | 2355.4 | 18.0 | 425210.5 | 3852.1 | 20.0 | 5.23139 | 11.7359 | 136706.67869 | 2035.45866 | 12.0 | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/39763988786/attempts/2 |  |  |  |
| 12 | GB300 NVL72 | 12 |  |  | 800 | 2026-10-23 | 28 |  | Dynamo SGLang | FP4 | 1 | 1 | 60716021.405660376 | 128717.96538 | DeepSeek-V4-Pro | off | agentic_traces | 1792738800 | 12 | disagg | 16 | 223588.8411575 | 2223.4643433333 | 0.01126 | 0.011619999999999998 | 10.75307 | 0.01122 | 4.1535 | 16.76172 | 0.011519999999999999 | 5.72103 | 88.809946714 | 86.0585197935 |  |  |  |  | 3629.76685 | 25993955548 | 97310370 |  |  | 0.012539999999999999 | 33.04261 | 0.01239 | 11.25096 | 79.7448165869 | agentic | 12 | 12 | 96115 | 1358.54115 | 0.96993 | 2564.23725 | 135100.72117 | 1007.62509 | 169528.5 | 1011.0 | 32.0 | 287554.8 | 2336.0 | 38.0 | 424607.5 | 3851.3 | 42.0 | 10.73541 | 26.5435 | 134917.59546 | 1954.95863 | 26.0 | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/39763988786/attempts/2 |  |  |  |
| 16 | GB300 NVL72 | 16 |  |  | 200 | 2026-10-23 | 24 |  | Dynamo SGLang | FP4 | 1 | 1 | 22449739.273584906 | 47593.44726 | DeepSeek-V4-Pro | off | agentic_traces | 1792738800 | 16 | disagg | 8 | 141757.9566 | 511.192605 | 0.00628 | 0.006690000000000001 | 5.56813 | 0.00628 | 2.0920199999999998 | 9.28241 | 0.00667 | 2.80447 | 159.2356687898 | 149.4768310912 |  |  |  |  | 3624.51746 | 8224852640 | 29811306 |  |  | 0.0068200000000000005 | 18.800939999999997 | 0.0068 | 5.392810000000001 | 146.6275659824 | agentic | 16 | 16 | 28380 | 1352.13258 | 0.97147 | 2420.48468 | 144835.57118 | 1044.58154 | 179468.0 | 999.0 | 10.0 | 302712.3 | 2430.1 | 13.0 | 449481.3 | 4033.25 | 15.0 | 3.83501 | 7.84817 | 142055.48681 | 2130.54422 | 8.0 | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/39763988786/attempts/2 |  |  |  |
| 16 | GB300 NVL72 | 16 |  |  | 80 | 2026-10-23 | 24 |  | Dynamo SGLang | FP4 | 1 | 1 | 10328144.735849058 | 21895.66684 | DeepSeek-V4-Pro | off | agentic_traces | 1792738800 | 16 | disagg | 8 | 65181.86142 | 252.569565 | 0.00549 | 0.00588 | 4.30471 | 0.00549 | 1.20786 | 7.429270000000001 | 0.00587 | 1.76316 | 182.1493624772 | 170.0680272109 |  |  |  |  | 3618.49796 | 3782510766 | 14662473 |  |  | 0.00595 | 15.29782 | 0.00594 | 3.0754 | 168.0672268908 | agentic | 16 | 16 | 13882 | 1344.007 | 0.96996 | 2501.76227 | 135923.02694 | 1053.36112 | 170727.75 | 1099.75 | 5.0 | 301594.8 | 2380.9 | 7.0 | 498108.85 | 3987.95 | 8.0 | 2.36591 | 3.8409 | 156544.69269 | 1976.67165 | 4.0 | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/39763988786/attempts/2 |  |  |  |
| 16 | GB300 NVL72 | 16 |  |  | 720 | 2026-10-23 | 32 |  | Dynamo SGLang | FP4 | 1 | 1 | 52642945.08962265 | 111603.04359 | DeepSeek-V4-Pro | off | agentic_traces | 1792738800 | 16 | disagg | 16 | 221580.5127 | 1625.5745 | 0.00891 | 0.00937 | 9.07776 | 0.00887 | 3.7264 | 13.84585 | 0.00929 | 5.03162 | 112.2334455668 | 106.7235859125 |  |  |  |  | 3629.60315 | 25753180196 | 94889326 |  |  | 0.01001 | 26.99914 | 0.009890000000000001 | 9.6707 | 99.9000999001 | agentic | 16 | 16 | 93443 | 1356.65063 | 0.97047 | 2526.29443 | 137709.50455 | 1010.27412 | 172919.0 | 1014.0 | 31.0 | 296430.0 | 2350.0 | 37.0 | 437352.8 | 3860.0 | 41.0 | 10.33755 | 25.79078 | 138853.66379 | 1939.03407 | 26.0 | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/39763988786/attempts/2 |  |  |  |
| 8 | Vera Rubin NVL72 | 8 |  |  | 1882 | 2026-10-05 | 26 |  | Dynamo TensorRT LLM | FP4 | 6 | 6 | 86478760.21212122 | 285379.9087 | DeepSeek-V4-Pro | mtp | agentic_traces | 1791183600 | 8 | disagg | 18 | 409164.5520011111 | 6864.461305 | 0.01208 | 0.01209 | 13.759879999999999 | 0.01207 | 5.68183 | 21.2937 | 0.012060000000000001 | 9.412889999999999 | 82.7814569536 | 82.7129859388 |  |  |  |  | 3629.9906 | 26795704250 | 200950401 |  |  | 0.01248 | 44.853519999999996 | 0.01244 | 22.38614 | 80.1282051282 | agentic | 8 | 8 | 198295 | 1336.34258 | 0.96896 | 2526.2679 | 134823.07982 | 1005.28727 | 167010.0 | 1011.5 | 63.0 | 292530.6 | 2341.0 | 70.0 | 433465.6 | 3849.0 | 73.0 | 12.42162 | 54.63985 | 135316.23735 | 1962.82757 | 55.0 |  |  |  |  |
| 1 | GB300 NVL72 | 4 |  |  | 1 | 2026-10-23 | 4 |  | Dynamo SGLang | FP4 |  |  | 2283375.231132075 | 4840.75549 | DeepSeek-V4-Pro | off | agentic_traces | 1792738800 |  | agg |  | 4805.28102 | 35.47447 | 0.0034500000000000004 | 0.0035800000000000003 | 3.60458 | 0.0034500000000000004 | 0.37662 | 7.53637 | 0.00357 | 0.6724 | 289.8550724638 | 279.3296089385 |  |  |  |  | 3619.46256 | 69570139 | 513594 |  |  | 0.0038799999999999998 | 18.28342 | 0.0038799999999999998 | 1.63372 | 257.7319587629 | agentic |  |  | 263 | 2120.04183 | 0.98034 | 2902.07726 | 264525.24335 | 1952.8289 | 386683.5 | 2652.0 | 0.0 | 470981.8 | 5089.4 | 0.0 | 503534.0 | 6909.3 | 1.0 | 0.26781 | 0.07256 | 154473.70587 | 2649.4789 | 0.0 | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/39763988786/attempts/2 |  |  |  |
| 1 | GB300 NVL72 | 8 |  |  | 1 | 2026-10-23 | 8 |  | Dynamo SGLang | FP4 |  |  | 1200388.3632075472 | 2544.82333 | DeepSeek-V4-Pro | off | agentic_traces | 1792738800 |  | agg |  | 2526.4358 | 18.38753 | 0.00307 | 0.00314 | 3.59667 | 0.00307 | 0.6299199999999999 | 6.9716700000000005 | 0.00313 | 0.84649 | 325.7328990228 | 318.4713375796 |  |  |  |  | 3602.17618 | 72805335 | 529881 |  |  | 0.0033399999999999997 | 16.79596 | 0.00333 | 1.75081 | 299.4011976048 | agentic |  |  | 269 | 2161.39033 | 0.98101 | 2908.29948 | 270651.80297 | 1969.81784 | 390520.0 | 2759.0 | 0.0 | 485900.6 | 5256.6 | 0.0 | 517698.8 | 6907.2 | 1.0 | 0.27104 | 0.07457 | 158035.69552 | 2641.71529 | 0.0 | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/39763988786/attempts/2 |  |  |  |
| 1 | GB300 NVL72 | 4 |  |  | 4 | 2026-10-23 | 4 |  | Dynamo SGLang | FP4 |  |  | 3459135.0849056607 | 7333.36638 | DeepSeek-V4-Pro | off | agentic_traces | 1792738800 |  | agg |  | 7281.19804 | 52.16834 | 0.0037 | 0.00404 | 2.2756399999999997 | 0.00369 | 0.40044 | 5.62513 | 0.004030000000000001 | 0.65523 | 270.2702702703 | 247.5247524752 |  |  |  |  | 3598.67227 | 104810582 | 750947 |  |  | 0.00463 | 15.59134 | 0.00462 | 1.2066700000000001 | 215.9827213823 | agentic |  |  | 579 | 1858.59758 | 0.97779 | 2650.02445 | 181020.00345 | 1296.97237 | 233342.5 | 1531.0 | 0.0 | 528823.4 | 3856.6 | 1.0 | 585564.9 | 5417.3 | 1.0 | 0.39889 | 0.16109 | 183335.10369 | 2218.6618 | 0.0 | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/39763988786/attempts/2 |  |  |  |
| 8 | GB300 NVL72 | 8 |  |  | 880 | 2026-10-23 | 24 |  | Dynamo SGLang | FP4 | 1 | 1 | 67818589.11792453 | 143775.40893 | DeepSeek-V4-Pro | off | agentic_traces | 1792738800 | 8 | disagg | 16 | 214047.02169 | 3232.18341 | 0.01587 | 0.01609 | 13.565190000000001 | 0.01583 | 5.2005 | 21.79525 | 0.015960000000000002 | 6.51867 | 63.0119722747 | 62.1504039776 |  |  |  |  | 3629.82694 | 24909589846 | 94757902 |  |  | 0.01758 | 43.1379 | 0.01742 | 11.89887 | 56.8828213879 | agentic | 8 | 8 | 94632 | 1342.92307 | 0.96883 | 2549.11578 | 131364.21426 | 991.82233 | 163264.0 | 998.0 | 31.0 | 279195.9 | 2277.0 | 37.0 | 417944.95 | 3770.45 | 41.0 | 10.35858 | 26.14033 | 132779.2821 | 1929.9353 | 26.0 | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/39763988786/attempts/2 |  |  |  |
| 8 | B300 | 8 |  |  | 64 | 2026-10-26 | 24 |  | Dynamo SGLang | FP4 | 8 | 8 | 9529227.978947368 | 18105.53316 | DeepSeek-V4-Pro | off | agentic_traces | 1792998000 | 16 | disagg | 8 | 53933.874 | 191.362755 | 0.00598 | 0.00753 | 4.78997 | 0.00597 | 1.60627 | 8.044310000000001 | 0.0075 | 2.13 | 167.2240802676 | 132.802124834 |  |  |  |  | 3622.85929 | 3127103882 | 11119619 |  |  | 0.006900000000000001 | 16.997709999999998 | 0.00687 | 3.70601 | 144.9275362319 | agentic | 8 | 8 | 10987 | 1272.27891 | 0.97268 | 2258.86985 | 142273.47711 | 1009.60089 | 177477.5 | 1084.0 | 4.0 | 325371.0 | 2296.6 | 6.0 | 546579.1 | 3751.7 | 7.0 | 2.01677 | 3.03369 | 162714.86195 | 1915.70341 | 3.0 | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/39910889415/attempts/2 |  |  |  |
| 8 | B300 | 8 |  |  | 240 | 2026-10-26 | 16 |  | Dynamo SGLang | FP4 | 8 | 8 | 34786940.67894737 | 66095.18729 | DeepSeek-V4-Pro | off | agentic_traces | 1792998000 | 8 | disagg | 8 | 131227.81006 | 962.56452 | 0.01085 | 0.01286 | 10.663870000000001 | 0.01083 | 4.04582 | 18.02309 | 0.0128 | 7.22555 | 92.1658986175 | 77.7604976672 |  |  |  |  | 3629.50955 | 7627529880 | 28127868 |  |  | 0.01314 | 37.74457 | 0.01304 | 14.56736 | 76.103500761 | agentic | 8 | 8 | 27611 | 1337.60487 | 0.97013 | 2372.80225 | 138000.82264 | 1012.24501 | 169415.0 | 972.0 | 10.0 | 289025.0 | 2339.0 | 13.0 | 425784.0 | 3880.0 | 15.0 | 4.32537 | 7.61235 | 135117.72727 | 2090.13992 | 7.0 | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/39910889415/attempts/2 |  |  |  |
| 8 | B300 | 8 |  |  | 64 | 2026-10-26 | 16 |  | Dynamo SGLang | FP4 | 8 | 8 | 13184726.689473685 | 25050.98071 | DeepSeek-V4-Pro | off | agentic_traces | 1792998000 | 8 | disagg | 8 | 49740.77294 | 361.18848 | 0.00754 | 0.00842 | 5.51779 | 0.00752 | 1.63853 | 9.49949 | 0.00838 | 2.10397 | 132.625994695 | 118.7648456057 |  |  |  |  | 3626.72463 | 2888023896 | 10498512 |  |  | 0.00853 | 19.8217 | 0.008490000000000001 | 3.66244 | 117.2332942556 | agentic | 8 | 8 | 10480 | 1249.89214 | 0.97149 | 2216.76772 | 137706.93597 | 999.94742 | 170735.75 | 1067.0 | 4.0 | 307686.6 | 2260.2 | 5.0 | 524908.5 | 3708.55 | 6.0 | 1.94363 | 2.89156 | 157886.49023 | 1915.13695 | 3.0 | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/39910889415/attempts/2 |  |  |  |
| 8 | B300 | 8 |  |  | 384 | 2026-10-26 | 8 |  | Dynamo SGLang | FP4 |  |  | 47227877.510526314 | 89732.96727 | DeepSeek-V4-Pro | off | agentic_traces | 1792998000 |  | agg |  | 89049.60207 | 683.36521 | 0.020550000000000002 | 0.02097 | 14.84135 | 0.02051 | 3.45397 | 44.53749 | 0.02085 | 24.32345 | 48.6618004866 | 47.6871721507 |  |  |  |  | 3627.40117 | 2592031577 | 20072932 |  |  | 0.0257 | 81.79131 | 0.0256 | 29.048740000000002 | 38.9105058366 | agentic |  |  | 20829 | 1212.08671 | 0.97092 | 2162.39164 | 124064.95958 | 952.0725 | 153746.0 | 991.0 | 9.0 | 276284.8 | 2179.4 | 13.0 | 396037.8 | 3580.4 | 16.0 | 5.26299 | 5.74379 | 125628.10186 | 1892.31103 | 5.0 | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/39910889415/attempts/2 |  |  |  |
| 1 | B300 | 4 |  |  | 4 | 2026-10-26 | 4 |  | Dynamo SGLang | FP4 |  |  | 3943252.3052631575 | 7492.17938 | DeepSeek-V4-Pro | off | agentic_traces | 1792998000 |  | agg |  | 7439.83715 | 52.34223 | 0.00379 | 0.0041600000000000005 | 2.27397 | 0.00377 | 0.45019 | 5.7025 | 0.00414 | 0.65665 | 263.8522427441 | 240.3846153846 |  |  |  |  | 3599.09181 | 107106628 | 753538 |  |  | 0.00496 | 15.67358 | 0.00492 | 1.30179 | 201.6129032258 | agentic |  |  | 588 | 1878.04932 | 0.97818 | 2655.20114 | 182154.12925 | 1281.52721 | 244719.0 | 1489.25 | 0.0 | 526876.8 | 3829.2 | 1.0 | 583890.95 | 5396.7 | 1.0 | 0.39961 | 0.1636 | 182153.66739 | 2205.14543 | 0.0 | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/39910889415/attempts/2 |  |  |  |
| 1 | B300 | 4 |  |  | 8 | 2026-10-26 | 4 |  | Dynamo SGLang | FP4 |  |  | 6633402.384210526 | 12603.46453 | DeepSeek-V4-Pro | off | agentic_traces | 1792998000 |  | agg |  | 12509.36634 | 94.09819 | 0.0047599999999999995 | 0.00517 | 3.1317199999999996 | 0.00475 | 0.58586 | 6.25841 | 0.00515 | 0.7695299999999999 | 210.0840336134 | 193.4235976789 |  |  |  |  | 3611.86532 | 180728586 | 1359480 |  |  | 0.00679 | 14.86714 | 0.00679 | 1.20425 | 147.2754050074 | agentic |  |  | 1205 | 1962.1192 | 0.97695 | 2591.89515 | 149982.22905 | 1128.19917 | 171437.0 | 1132.0 | 1.0 | 339730.8 | 2794.2 | 1.0 | 417177.0 | 4470.8 | 2.0 | 0.60668 | 0.33361 | 124980.1789 | 1913.95935 | 0.0 | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/39910889415/attempts/2 |  |  |  |
| 1 | B300 | 8 |  |  | 1 | 2026-10-26 | 8 |  | Dynamo SGLang | FP4 |  |  | 1330082.3 | 2527.15637 | DeepSeek-V4-Pro | off | agentic_traces | 1792998000 |  | agg |  | 2508.80375 | 18.35262 | 0.00311 | 0.00321 | 3.51602 | 0.00311 | 0.65113 | 7.02149 | 0.00319 | 0.82872 | 321.5434083601 | 311.5264797508 |  |  |  |  | 3600.30388 | 72259647 | 528600 |  |  | 0.0033900000000000002 | 16.99419 | 0.0033799999999999998 | 1.7785199999999999 | 294.9852507375 | agentic |  |  | 268 | 2167.50746 | 0.98091 | 2911.99254 | 269625.54851 | 1972.38806 | 390518.0 | 2763.0 | 0.0 | 482996.0 | 5264.9 | 0.0 | 516148.4 | 6907.55 | 1.0 | 0.26756 | 0.07433 | 157434.58967 | 2646.30354 | 0.0 | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/39910889415/attempts/2 |  |  |  |
| 1 | H200 | 8 |  |  | 1 | 2026-11-09 | 8 |  | vLLM | FP4 |  |  | 2146225.6131386864 | 2940.32909 | DeepSeek-V4.1-Flash | mtp | agentic_traces | 1794211200 |  | agg |  | 2920.58186 | 19.74723 | 0.00266 | 0.00271 | 2.67301 | 0.00266 | 0.35639 | 5.641229999999999 | 0.00271 | 0.39183 | 375.9398496241 | 369.0036900369 |  |  |  |  | 3603.27839 | 84189356 | 569238 |  |  | 0.00292 | 14.05609 | 0.00292 | 0.45547000000000004 | 342.4657534247 | agentic |  |  | 290 | 2129.60345 | 0.98173 | 2824.6987 | 290308.12414 | 1962.88966 | 416685.5 | 2737.0 | 0.0 | 526360.6 | 5108.6 | 0.0 | 555316.85 | 6779.25 | 1.0 | 0.27882 | 0.0803 | 169706.4322 | 2574.10711 | 0.0 | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/42341901521/attempts/1 |  |  |  |
| 8 | B300 | 8 | 1024 | 1024 | 2342 | 2026-01-28 | 12 | 1K/1K | Dynamo TensorRT LLM | FP4 | 2 | 2 | 6658188.810011947 | 12650.5587390227 | DeepSeek-R1 | mtp | single_turn | 1769587200 | 8 | disagg | 4 | 18971.458682817 | 9490.1087671255 | 4.488675082917299 | 4.0658557910685 | 26.5841890994925 | 0.0252666868775 | 3.4066320805577996 | 27.3197190430005 | 0.0248382085416 | 4.4401661453876 | 39.5778047532 | 40.2605525405 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 1 | H200 | 8 |  |  | 128 | 2026-11-09 | 8 |  | vLLM | FP4 |  |  | 25660696.89051095 | 35155.15474 | DeepSeek-V4.1-Flash | mtp | agentic_traces | 1794211200 |  | agg |  | 34841.79715 | 313.35759 | 0.04261 | 0.044020000000000004 | 22.900389999999998 | 0.04261 | 0.71174 | 46.87343 | 0.044020000000000004 | 5.02515 | 23.4686693264 | 22.7169468423 |  |  |  |  | 3629.63366 | 1016791801 | 9241661 |  |  | 0.06559000000000001 | 103.71461000000001 | 0.06559000000000001 | 14.71938 | 15.2462265589 | agentic |  |  | 9525 | 1204.25558 | 0.95998 | 2363.78767 | 106215.60924 | 955.27412 | 129041.0 | 1012.0 | 4.0 | 234796.6 | 2061.6 | 5.0 | 344817.8 | 3508.2 | 6.0 | 1.98688 | 2.62514 | 117055.47482 | 1919.50179 | 2.0 | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/42341901521/attempts/1 |  |  |  |
| 1 | H200 | 8 |  |  | 4 | 2026-11-09 | 8 |  | vLLM | FP4 |  |  | 3098466.474452555 | 4244.89907 | DeepSeek-V4.1-Flash | mtp | agentic_traces | 1794211200 |  | agg |  | 4215.73678 | 29.16229 | 0.0027400000000000002 | 0.0030800000000000003 | 1.5649600000000001 | 0.0027400000000000002 | 0.2553 | 4.0141100000000005 | 0.0030800000000000003 | 0.32503 | 364.9635036496 | 324.6753246753 |  |  |  |  | 3598.11874 | 121349772 | 839435 |  |  | 0.00361 | 10.62788 | 0.00361 | 0.47524 | 277.0083102493 | agentic |  |  | 658 | 1800.73556 | 0.97843 | 2560.22056 | 184422.1459 | 1275.73708 | 249212.5 | 1509.5 | 0.0 | 518664.6 | 3752.6 | 1.0 | 582395.45 | 5367.7 | 1.0 | 0.4165 | 0.18301 | 177204.95084 | 2146.5226 | 0.0 | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/42341901521/attempts/1 |  |  |  |
| 1 | H200 | 8 |  |  | 32 | 2026-11-09 | 8 |  | vLLM | FP4 |  |  | 18890945.058394164 | 25880.59473 | DeepSeek-V4.1-Flash | mtp | agentic_traces | 1794211200 |  | agg |  | 25725.66403 | 154.9307 | 0.007 | 0.007940000000000001 | 3.54725 | 0.007 | 0.24267 | 8.07175 | 0.007940000000000001 | 0.35902999999999996 | 142.8571428571 | 125.9445843829 |  |  |  |  | 3618.60745 | 744728636 | 4485067 |  |  | 0.01283 | 18.87089 | 0.01283 | 0.51647 | 77.9423226812 | agentic |  |  | 4351 | 1393.41667 | 0.97925 | 2125.25605 | 171162.63755 | 1030.81292 | 228525.5 | 1102.0 | 2.0 | 335583.0 | 2631.0 | 3.0 | 446553.5 | 4076.5 | 4.0 | 1.21556 | 1.20265 | 150464.55754 | 1722.56793 | 1.0 | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/42341901521/attempts/1 |  |  |  |
| 1 | H200 | 8 |  |  | 2 | 2026-11-09 | 8 |  | vLLM | FP4 |  |  | 2283572.4963503648 | 3128.49432 | DeepSeek-V4.1-Flash | mtp | agentic_traces | 1794211200 |  | agg |  | 3104.34295 | 24.15136 | 0.00271 | 0.00294 | 1.51895 | 0.0027 | 0.2725 | 4.31859 | 0.00294 | 0.34287 | 369.0036900369 | 340.1360544218 |  |  |  |  | 3599.35707 | 89389110 | 695435 |  |  | 0.00327 | 13.290209999999998 | 0.00327 | 0.45772 | 305.8103975535 | agentic |  |  | 486 | 2084.77572 | 0.97649 | 2494.70745 | 183928.20988 | 1430.93621 | 299492.25 | 1833.0 | 0.0 | 391441.0 | 4624.5 | 1.0 | 443015.25 | 6012.25 | 1.0 | 0.35741 | 0.1348 | 141749.55594 | 2247.79332 | 0.0 | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/42341901521/attempts/1 |  |  |  |
| 1 | H200 | 4 |  |  | 2 | 2026-11-09 | 4 |  | vLLM | FP4 |  |  | 4357039.934306569 | 5969.14471 | DeepSeek-V4.1-Flash | mtp | agentic_traces | 1794211200 |  | agg |  | 5921.88185 | 47.26286 | 0.00289 | 0.00326 | 1.85201 | 0.00289 | 0.29167000000000004 | 4.91558 | 0.00326 | 0.41337999999999997 | 346.0207612457 | 306.7484662577 |  |  |  |  | 3607.51549 | 85453122 | 682006 |  |  | 0.00373 | 14.37367 | 0.00373 | 0.57592 | 268.0965147453 | agentic |  |  | 457 | 2130.84683 | 0.97587 | 2596.33208 | 186987.13786 | 1492.35449 | 337845.0 | 1889.0 | 0.0 | 423551.8 | 4705.2 | 1.0 | 482619.0 | 6088.6 | 1.0 | 0.35496 | 0.12646 | 154396.96108 | 2366.31197 | 0.0 | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/42341901521/attempts/1 |  |  |  |
| 1 | H200 | 8 |  |  | 64 | 2026-11-09 | 8 |  | vLLM | FP4 |  |  | 26906271.547445256 | 36861.59202 | DeepSeek-V4.1-Flash | mtp | agentic_traces | 1794211200 |  | agg |  | 36583.1878 | 278.40423 | 0.016550000000000002 | 0.016890000000000002 | 7.93914 | 0.016550000000000002 | 0.26853 | 16.750049999999998 | 0.016890000000000002 | 0.46863 | 60.4229607251 | 59.2066311427 |  |  |  |  | 3628.55215 | 1062999960 | 8105359 |  |  | 0.02189 | 36.64113 | 0.02189 | 0.7203999999999999 | 45.6829602558 | agentic |  |  | 8247 | 1186.42645 | 0.96931 | 2082.61137 | 128768.28398 | 979.94834 | 151184.5 | 1041.0 | 3.0 | 278787.4 | 2199.2 | 5.0 | 487385.1 | 3651.4 | 5.0 | 1.64232 | 2.27601 | 148329.82028 | 1934.24187 | 2.0 | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/42341901521/attempts/1 |  |  |  |
| 1 | H200 | 8 |  |  | 16 | 2026-11-09 | 8 |  | vLLM | FP4 |  |  | 9890234.416058395 | 13549.62115 | DeepSeek-V4.1-Flash | mtp | agentic_traces | 1794211200 |  | agg |  | 13463.86167 | 85.75948 | 0.0046500000000000005 | 0.00526 | 2.10827 | 0.0046500000000000005 | 0.26618 | 5.05437 | 0.00526 | 0.36149000000000003 | 215.0537634409 | 190.1140684411 |  |  |  |  | 3600.59617 | 388204349 | 2470283 |  |  | 0.0077599999999999995 | 11.42833 | 0.00775 | 0.5290199999999999 | 128.8659793814 | agentic |  |  | 2537 | 1583.92733 | 0.97647 | 2462.13599 | 152866.94127 | 973.70201 | 179049.0 | 914.0 | 1.0 | 278216.8 | 2374.8 | 2.0 | 524592.0 | 4163.6 | 2.0 | 0.91462 | 0.70444 | 152360.00797 | 1934.69583 | 0.0 | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/42341901521/attempts/1 |  |  |  |
| 1 | H200 | 8 |  |  | 8 | 2026-11-09 | 8 |  | vLLM | FP4 |  |  | 5404831.160583942 | 7404.61869 | DeepSeek-V4.1-Flash | mtp | agentic_traces | 1794211200 |  | agg |  | 7350.9714 | 53.64728 | 0.0033799999999999998 | 0.00365 | 1.86612 | 0.0033799999999999998 | 0.24915 | 4.154850000000001 | 0.00365 | 0.31443 | 295.8579881657 | 273.9726027397 |  |  |  |  | 3604.2318 | 211956839 | 1546858 |  |  | 0.00487 | 10.09778 | 0.00487 | 0.43945999999999996 | 205.3388090349 | agentic |  |  | 1373 | 1934.84373 | 0.97811 | 2515.18682 | 154374.97378 | 1126.62637 | 171406.0 | 1150.0 | 1.0 | 364299.4 | 2917.2 | 1.0 | 478921.6 | 4493.8 | 2.0 | 0.64211 | 0.38079 | 132405.23357 | 1877.41616 | 0.0 | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/42341901521/attempts/1 |  |  |  |
| 1 | H200 | 4 |  |  | 1 | 2026-11-09 | 4 |  | vLLM | FP4 |  |  | 4113436.0072992705 | 5635.40733 | DeepSeek-V4.1-Flash | mtp | agentic_traces | 1794211200 |  | agg |  | 5596.95729 | 38.45004 | 0.00283 | 0.00296 | 3.0391399999999997 | 0.00283 | 0.35547 | 6.0673699999999995 | 0.00296 | 0.45275 | 353.3568904594 | 337.8378378378 |  |  |  |  | 3600.3405 | 80603808 | 553733 |  |  | 0.0032 | 14.836139999999999 | 0.0032 | 0.56087 | 312.5 | agentic |  |  | 283 | 2123.5053 | 0.98253 | 2853.71305 | 284819.10954 | 1956.65371 | 406478.5 | 2715.0 | 0.0 | 516411.8 | 5089.4 | 0.0 | 545073.7 | 6875.5 | 1.0 | 0.27396 | 0.07842 | 166217.36705 | 2591.98771 | 0.0 | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/42341901521/attempts/1 |  |  |  |
| 1 | H200 | 4 |  |  | 4 | 2026-11-09 | 4 |  | vLLM | FP4 |  |  | 6061915.445255475 | 8304.82416 | DeepSeek-V4.1-Flash | mtp | agentic_traces | 1794211200 |  | agg |  | 8247.87227 | 56.95189 | 0.00299 | 0.0034300000000000003 | 1.6684 | 0.00299 | 0.26365 | 4.4286 | 0.0034300000000000003 | 0.38699 | 334.4481605351 | 291.5451895044 |  |  |  |  | 3593.42783 | 118552535 | 818610 |  |  | 0.0040999999999999995 | 12.1024 | 0.0040999999999999995 | 0.56743 | 243.9024390244 | agentic |  |  | 643 | 1808.88336 | 0.97865 | 2579.33483 | 184374.08243 | 1273.11042 | 249037.0 | 1508.0 | 0.0 | 517914.4 | 3738.0 | 1.0 | 580167.8 | 5358.3 | 1.0 | 0.41688 | 0.17908 | 176739.0453 | 2156.48977 | 0.0 | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/42341901521/attempts/1 |  |  |  |
| 16 | GB300 NVL72 | 16 |  |  | 700 | 2026-11-06 | 22 |  | Dynamo SGLang | FP4 | 2 | 2 | 106709830.32075472 | 226224.84028 | DeepSeek-V4.1-Flash | mtp | agentic_traces | 1793952000 | 16 | disagg | 6 | 823825.2741033332 | 2124.67761 | 0.0030600000000000002 | 0.0033900000000000002 | 1.74723 | 0.0030499999999999998 | 0.28837999999999997 | 3.61671 | 0.0033799999999999998 | 0.45297000000000004 | 326.7973856209 | 294.9852507375 |  |  |  |  | 3627.87261 | 35871139515 | 123460622 |  |  | 0.00329 | 8.122069999999999 | 0.00328 | 0.75521 | 303.9513677812 | agentic | 16 | 16 | 119274 | 1367.14924 | 0.97383 | 2489.18683 | 150346.252 | 1033.99697 | 196764.75 | 1029.0 | 38.0 | 331829.8 | 2429.0 | 43.0 | 464257.75 | 3987.35 | 47.0 | 9.43849 | 32.92106 | 145867.10643 | 2029.29819 | 33.0 | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/41581664540/attempts/3 |  |  |  |
| 1 | GB300 NVL72 | 4 |  |  | 16 | 2026-11-06 | 4 |  | Dynamo SGLang | FP4 |  |  | 13845718.183962265 | 29352.92255 | DeepSeek-V4.1-Flash | mtp | agentic_traces | 1793952000 |  | agg |  | 29163.44436 | 189.4782 | 0.0018 | 0.00191 | 1.4923 | 0.0017900000000000001 | 0.4792 | 2.72492 | 0.00189 | 0.84215 | 555.5555555556 | 523.5602094241 |  |  |  |  | 3599.43527 | 419887721 | 2728058 |  |  | 0.0024 | 6.309270000000001 | 0.0023799999999999997 | 1.69913 | 416.6666666667 | agentic |  |  | 2712 | 1643.11821 | 0.97626 | 2531.47259 | 154825.85583 | 1005.92109 | 183462.75 | 953.5 | 1.0 | 281967.7 | 2517.2 | 2.0 | 527168.7 | 4378.9 | 2.0 | 0.89321 | 0.75368 | 152278.73187 | 1954.23965 | 1.0 | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/41581664540/attempts/3 |  |  |  |
| 1 | GB300 NVL72 | 4 |  |  | 1 | 2026-11-06 | 4 |  | Dynamo SGLang | FP4 |  |  | 3888824.023584906 | 8244.30693 | DeepSeek-V4.1-Flash | mtp | agentic_traces | 1793952000 |  | agg |  | 8192.73825 | 51.56868 | 0.0010400000000000001 | 0.00106 | 0.86393 | 0.00103 | 0.17864 | 1.9256099999999998 | 0.0010500000000000002 | 0.27337 | 961.5384615385 | 943.3962264151 |  |  |  |  | 3600.06769 | 117977649 | 742603 |  |  | 0.00107 | 5.1691899999999995 | 0.00106 | 0.2515 | 934.5794392523 | agentic |  |  | 463 | 2103.72786 | 0.98228 | 2555.42954 | 254811.33693 | 1603.89417 | 416127.0 | 2193.0 | 0.0 | 576565.2 | 4748.4 | 1.0 | 638436.7 | 6021.1 | 1.0 | 0.35557 | 0.12848 | 210847.89986 | 2325.95645 | 0.0 | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/41581664540/attempts/3 |  |  |  |
| 1 | GB300 NVL72 | 4 |  |  | 4 | 2026-11-06 | 4 |  | Dynamo SGLang | FP4 |  |  | 4892153.900943397 | 10371.36627 | DeepSeek-V4.1-Flash | mtp | agentic_traces | 1793952000 |  | agg |  | 10300.90961 | 70.45666 | 0.0015 | 0.00165 | 0.95438 | 0.0015 | 0.15888 | 2.24993 | 0.00164 | 0.19329 | 666.6666666667 | 606.0606060606 |  |  |  |  | 3599.57811 | 148315715 | 1014457 |  |  | 0.00188 | 6.17774 | 0.00184 | 0.24633000000000002 | 531.914893617 | agentic |  |  | 773 | 1765.15524 | 0.97882 | 2565.85818 | 191870.2652 | 1312.36352 | 274452.0 | 1545.0 | 0.0 | 516412.8 | 3758.6 | 1.0 | 595801.6 | 5354.8 | 1.0 | 0.45565 | 0.2148 | 176880.75715 | 2190.47135 | 0.0 | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/41581664540/attempts/3 |  |  |  |
| 1 | H200 | 4 |  |  | 128 | 2026-11-09 | 4 |  | vLLM | FP4 |  |  | 7677006.226277374 | 10517.49853 | DeepSeek-V4.1-Flash | mtp | agentic_traces | 1794211200 |  | agg |  | 10397.33891 | 120.15963 | 0.03505 | 0.03454 | 671.2901 | 0.03505 | 653.82729 | 419.47228 | 0.03454 | 382.02494 | 28.5306704708 | 28.95193978 |  |  |  |  | 3622.48546 | 152081130 | 1749950 |  |  | 0.039119999999999995 | 767.14624 | 0.039119999999999995 | 731.71528 | 25.5623721881 | agentic |  |  | 1676 | 1023.77506 | 0.94909 | 2009.23486 | 89890.7136 | 1038.84606 | 105050.75 | 1010.5 | 1.0 | 204903.5 | 1871.5 | 1.0 | 325696.25 | 3370.25 | 2.0 | 0.9341 | 0.46218 | 118299.92447 | 2921.29359 | 0.0 | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/42341901521/attempts/1 |  |  |  |
| 1 | H200 | 4 |  |  | 16 | 2026-11-09 | 4 |  | vLLM | FP4 |  |  | 19316204.386861317 | 26463.20001 | DeepSeek-V4.1-Flash | mtp | agentic_traces | 1794211200 |  | agg |  | 26293.81918 | 169.38083 | 0.0053 | 0.00604 | 2.46942 | 0.0053 | 0.27485000000000004 | 5.92101 | 0.00604 | 0.49374 | 188.679245283 | 165.5629139073 |  |  |  |  | 3611.86681 | 379879091 | 2447124 |  |  | 0.00875 | 13.39731 | 0.00874 | 0.72634 | 114.2857142857 | agentic |  |  | 2506 | 1585.01506 | 0.97612 | 2466.18018 | 151587.82562 | 976.50599 | 176634.0 | 914.75 | 1.0 | 275582.0 | 2385.0 | 2.0 | 525650.75 | 4200.5 | 2.0 | 0.9022 | 0.69371 | 152659.47266 | 1944.71686 | 0.0 | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/42341901521/attempts/1 |  |  |  |
| 1 | H200 | 4 |  |  | 32 | 2026-11-09 | 4 |  | vLLM | FP4 |  |  | 36082011.343065694 | 49432.35554 | DeepSeek-V4.1-Flash | mtp | agentic_traces | 1794211200 |  | agg |  | 49130.84976 | 301.50578 | 0.00763 | 0.00885 | 3.8892800000000003 | 0.00763 | 0.2534 | 9.05786 | 0.00885 | 0.44653 | 131.0615989515 | 112.9943502825 |  |  |  |  | 3618.3444 | 711364877 | 4372206 |  |  | 0.0143 | 21.81851 | 0.0143 | 0.6623600000000001 | 69.9300699301 | agentic |  |  | 4237 | 1402.9786 | 0.97883 | 2137.7842 | 167828.49658 | 1029.92849 | 223772.0 | 1105.0 | 2.0 | 330589.4 | 2642.0 | 3.0 | 417423.8 | 4097.0 | 3.0 | 1.18826 | 1.17114 | 147161.25971 | 1718.76295 | 1.0 | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/42341901521/attempts/1 |  |  |  |
| 1 | H200 | 4 |  |  | 64 | 2026-11-09 | 4 |  | vLLM | FP4 |  |  | 47651891.24817518 | 65283.09101 | DeepSeek-V4.1-Flash | mtp | agentic_traces | 1794211200 |  | agg |  | 64779.99534 | 503.09567 | 0.018920000000000003 | 0.01946 | 9.49706 | 0.018920000000000003 | 0.31642000000000003 | 20.04507 | 0.01946 | 1.11623 | 52.8541226216 | 51.3874614594 |  |  |  |  | 3628.84956 | 944977773 | 7400845 |  |  | 0.0257 | 45.541650000000004 | 0.0257 | 1.97492 | 38.9105058366 | agentic |  |  | 7408 | 1134.80011 | 0.96899 | 1905.67054 | 126931.34854 | 985.77673 | 148131.0 | 1041.0 | 3.0 | 275111.6 | 2258.0 | 4.0 | 484391.2 | 3670.0 | 5.0 | 1.57549 | 2.04275 | 146134.27231 | 1959.56425 | 2.0 | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/42341901521/attempts/1 |  |  |  |
| 1 | H200 | 4 |  |  | 8 | 2026-11-09 | 4 |  | vLLM | FP4 |  |  | 10591195.343065694 | 14509.93762 | DeepSeek-V4.1-Flash | mtp | agentic_traces | 1794211200 |  | agg |  | 14405.24206 | 104.69556 | 0.0038399999999999997 | 0.00414 | 2.18654 | 0.0038399999999999997 | 0.25933 | 4.68717 | 0.00414 | 0.38894 | 260.4166666667 | 241.5458937198 |  |  |  |  | 3599.21194 | 207390077 | 1507286 |  |  | 0.00549 | 11.337069999999999 | 0.00549 | 0.5512699999999999 | 182.1493624772 | agentic |  |  | 1341 | 1938.06154 | 0.97822 | 2525.02562 | 154653.30127 | 1124.00149 | 173612.0 | 1155.0 | 1.0 | 364829.0 | 2902.0 | 1.0 | 477274.0 | 4482.0 | 2.0 | 0.63547 | 0.37243 | 132554.69468 | 1867.98697 | 0.0 | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/42341901521/attempts/1 |  |  |  |
| 1 | H200 | 4 | 1024 | 1024 | 1 | 2026-06-05 | 4 | 1K/1K | vLLM | FP8 |  |  | 48167.87524496351 | 65.9899890856 | MiniMax-M2.5 | off | single_turn | 1780642800 |  | agg |  | 32.8782012821 | 33.1117878034 | 0.0073854443617 | 0.0074446282661 | 6.844503028318299 | 0.0074377569385 | 0.1002820008434 | 6.9562054187059 | 0.0074448825433 | 0.10416990946980001 | 134.4491367847 | 134.320453571 |  |  |  |  |  |  |  |  |  | 0.0077260948718 | 7.4845350246876 | 0.0075046042911 | 0.114125463739 | 133.2515294892 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 16 | GB300 NVL72 | 16 |  |  | 7480 | 2026-11-06 | 34 |  | Dynamo SGLang | FP4 | 2 | 2 | 525179520.8679246 | 1113380.58424 | DeepSeek-V4.1-Flash | mtp | agentic_traces | 1793952000 | 16 | disagg | 18 | 2087763.813322222 | 17199.4515225 | 0.00919 | 0.009710000000000002 | 7.46566 | 0.009179999999999999 | 2.28285 | 12.69317 | 0.00968 | 3.55429 | 108.8139281828 | 102.9866117405 |  |  |  |  | 3629.94486 | 272955750606 | 1003291240 |  |  | 0.01013 | 26.50657 | 0.01009 | 7.563680000000001 | 98.7166831194 | agentic | 16 | 16 | 979434 | 1359.8324 | 0.97075 | 2531.98181 | 139276.78165 | 1019.90432 | 176466.0 | 1020.0 | 301.0 | 298241.0 | 2368.0 | 323.0 | 428970.35 | 3903.05 | 337.0 | 55.17865 | 270.18372 | 137876.29026 | 1985.30979 | 276.0 | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/41581664540/attempts/3 |  |  |  |
| 16 | GB300 NVL72 | 16 |  |  | 1228 | 2026-11-06 | 22 |  | Dynamo SGLang | FP4 | 2 | 2 | 180956484.32547173 | 383627.74677 | DeepSeek-V4.1-Flash | mtp | agentic_traces | 1793952000 | 16 | disagg | 6 | 1396913.9852433333 | 3645.4073425 | 0.00328 | 0.00375 | 2.1533200000000003 | 0.00327 | 0.44617 | 4.15688 | 0.0037400000000000003 | 0.7680499999999999 | 304.8780487805 | 266.6666666667 |  |  |  |  | 3628.65502 | 60829401308 | 211975235 |  |  | 0.0035800000000000003 | 9.1349 | 0.0035600000000000002 | 1.3443900000000002 | 279.3296089385 | agentic | 16 | 16 | 205165 | 1366.12328 | 0.97335 | 2522.0964 | 148239.28887 | 1031.59316 | 193821.0 | 1027.0 | 64.0 | 326575.0 | 2421.0 | 72.0 | 460232.8 | 3984.0 | 79.0 | 15.03436 | 56.59697 | 143517.06091 | 2001.46347 | 56.0 | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/41581664540/attempts/3 |  |  |  |
| 16 | GB300 NVL72 | 16 |  |  | 2884 | 2026-11-06 | 26 |  | Dynamo SGLang | FP4 | 2 | 2 | 337557915.1509434 | 715622.78012 | DeepSeek-V4.1-Flash | mtp | agentic_traces | 1793952000 | 16 | disagg | 10 | 1847565.1688 | 8158.787195 | 0.004059999999999999 | 0.00454 | 3.35109 | 0.004059999999999999 | 0.9484600000000001 | 5.89909 | 0.00453 | 1.72836 | 246.3054187192 | 220.2643171806 |  |  |  |  | 3629.70248 | 134133667522 | 474774189 |  |  | 0.00433 | 12.7365 | 0.00431 | 3.7394600000000002 | 230.9468822171 | agentic | 16 | 16 | 458076 | 1371.32521 | 0.97275 | 2534.842 | 146397.36347 | 1034.37753 | 190166.0 | 1027.0 | 142.0 | 318758.5 | 2419.0 | 155.0 | 446177.0 | 3990.0 | 163.0 | 26.24508 | 126.39983 | 141811.11625 | 2014.4135 | 129.0 | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/41581664540/attempts/3 |  |  |  |
| 16 | GB300 NVL72 | 16 |  |  | 5540 | 2026-11-06 | 30 |  | Dynamo SGLang | FP4 | 2 | 2 | 478472891.2783019 | 1014362.52951 | DeepSeek-V4.1-Flash | mtp | agentic_traces | 1793952000 | 16 | disagg | 14 | 2157911.228442857 | 13757.41794375 | 0.00596 | 0.00666 | 6.84624 | 0.00595 | 3.06939 | 10.33286 | 0.00664 | 4.2838199999999995 | 167.7852348993 | 150.1501501502 |  |  |  |  | 3629.60737 | 219362308876 | 801262033 |  |  | 0.0064 | 20.58448 | 0.00636 | 8.970040000000001 | 156.25 | agentic | 16 | 16 | 774273 | 1371.82537 | 0.97139 | 2543.69333 | 141620.8327 | 1031.86397 | 181573.0 | 1027.0 | 238.0 | 304485.0 | 2408.0 | 255.0 | 431162.0 | 3964.0 | 265.0 | 41.10782 | 213.65011 | 138544.51845 | 2009.17051 | 219.0 | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/41581664540/attempts/3 |  |  |  |
| 1 | Vera Rubin NVL72 | 8 |  |  | 16 | 2026-10-23 | 10 |  | Dynamo TensorRT LLM | FP4 | 2 | 2 | 3387653.666666667 | 11179.2571 | MiniMax-M3 | mtp | agentic_traces | 1792738800 | 8 | disagg | 2 | 55546.92835 | 87.3392875 | 0.00088 | 0.00091 | 3.24153 | 0.00087 | 2.3193099999999998 | 5.53188 | 0.0009 | 4.63855 | 1136.3636363636 | 1098.9010989011 |  |  |  |  | 3604.71922 | 400843902 | 2518671 |  |  | 0.0010500000000000002 | 13.57597 | 0.0010400000000000001 | 12.42898 | 952.380952381 | agentic | 1 | 8 | 2571 | 1577.4259 | 0.97705 | 2451.25406 | 155761.24465 | 979.64566 | 182841.5 | 936.0 | 1.0 | 279886.0 | 2401.0 | 2.0 | 537686.0 | 4217.5 | 3.0 | 0.96546 | 0.7131 | 154096.71935 | 1932.76933 | 0.0 |  |  |  |  |
| 1 | Vera Rubin NVL72 | 2 |  |  | 52 | 2026-10-23 | 10 |  | Dynamo TensorRT LLM | FP4 | 2 | 2 | 12038296.803030303 | 39726.37945 | MiniMax-M3 | mtp | agentic_traces | 1792738800 | 8 | disagg | 2 | 197291.02795 | 335.217325 | 0.00186 | 0.00181 | 2.22969 | 0.00186 | 0.55432 | 4.14827 | 0.0018 | 2.21975 | 537.6344086022 | 552.4861878453 |  |  |  |  | 3611.5645 | 1425098669 | 9685274 |  |  | 0.0020499999999999997 | 10.076139999999999 | 0.00204 | 6.35173 | 487.8048780488 | agentic | 1 | 2 | 9254 | 1342.99709 | 0.97598 | 2293.72593 | 153993.79155 | 1046.60385 | 206979.75 | 1086.0 | 4.0 | 350745.6 | 2514.7 | 5.0 | 564671.5 | 4046.4 | 6.0 | 1.76214 | 2.56245 | 168659.30619 | 2024.88902 | 2.0 |  |  |  |  |
| 1 | Vera Rubin NVL72 | 4 |  |  | 24 | 2026-10-23 | 10 |  | Dynamo TensorRT LLM | FP4 | 2 | 2 | 5652926.327272727 | 18654.65688 | MiniMax-M3 | mtp | agentic_traces | 1792738800 | 8 | disagg | 2 | 92720.51335 | 138.19275 | 0.00103 | 0.0010500000000000002 | 2.23542 | 0.00102 | 1.12987 | 4.02781 | 0.0010400000000000001 | 2.95546 | 970.8737864078 | 952.380952381 |  |  |  |  | 3605.51466 | 668992082 | 3986050 |  |  | 0.00115 | 9.35108 | 0.00114 | 7.68196 | 869.5652173913 | agentic | 1 | 4 | 3941 | 1492.56477 | 0.97878 | 2236.32415 | 169654.99645 | 1011.4306 | 227879.0 | 1053.0 | 2.0 | 338802.0 | 2608.0 | 3.0 | 443042.0 | 4223.0 | 3.0 | 1.20008 | 1.09293 | 146267.57974 | 1752.97059 | 1.0 |  |  |  |  |
| 1 | Vera Rubin NVL72 | 2 |  |  | 56 | 2026-10-23 | 2 |  | Dynamo TensorRT LLM | FP4 |  |  | 53111934.38787878 | 175269.38348 | MiniMax-M3 | mtp | agentic_traces | 1792738800 |  | agg |  | 174026.74445 | 1242.63902 | 0.006849999999999999 | 0.0069299999999999995 | 4.4021300000000005 | 0.00684 | 0.34676 | 8.5822 | 0.00694 | 1.36592 | 145.9854014599 | 144.3001443001 |  |  |  |  | 3628.99717 | 1263890011 | 9034244 |  |  | 0.00874 | 19.21943 | 0.00873 | 3.76606 | 114.4164759725 | agentic |  |  | 8708 | 1267.13949 | 0.97408 | 2158.96146 | 145048.82017 | 1035.72198 | 182401.25 | 1059.0 | 3.0 | 337268.4 | 2431.9 | 5.0 | 555007.5 | 4042.3 | 5.0 | 1.6308 | 2.39994 | 162512.88443 | 2031.67499 | 2.0 |  |  |  |  |
| 1 | Vera Rubin NVL72 | 2 |  |  | 20 | 2026-10-23 | 2 |  | Dynamo TensorRT LLM | FP4 |  |  | 22592645.675757576 | 74555.73073 | MiniMax-M3 | mtp | agentic_traces | 1792738800 |  | agg |  | 74090.02664 | 465.70409 | 0.00215 | 0.00221 | 2.23442 | 0.00214 | 0.28901 | 4.118189999999999 | 0.0022 | 1.8235899999999998 | 465.1162790698 | 452.4886877828 |  |  |  |  | 3602.10171 | 534195968 | 3355030 |  |  | 0.00257 | 10.346540000000001 | 0.00256 | 5.9028 | 389.1050583658 | agentic |  |  | 3188 | 1578.49952 | 0.97884 | 2332.8884 | 167427.7362 | 1052.39241 | 215292.0 | 1097.25 | 1.0 | 331004.3 | 2681.9 | 2.0 | 441276.9 | 4379.3 | 3.0 | 1.00601 | 0.88503 | 149119.7886 | 1833.61786 | 1.0 |  |  |  |  |
| 1 | Vera Rubin NVL72 | 2 |  |  | 46 | 2026-10-23 | 2 |  | Dynamo TensorRT LLM | FP4 |  |  | 46316727.61515152 | 152845.20113 | MiniMax-M3 | mtp | agentic_traces | 1792738800 |  | agg |  | 151868.6895 | 976.51163 | 0.00487 | 0.00475 | 4.28107 | 0.004860000000000001 | 0.8889900000000001 | 7.2605699999999995 | 0.00475 | 2.13227 | 205.3388090349 | 210.5263157895 |  |  |  |  | 3620.80123 | 1100469766 | 7085125 |  |  | 0.0058 | 15.62421 | 0.0058 | 5.28659 | 172.4137931034 | agentic |  |  | 6756 | 1277.78788 | 0.9766 | 2047.80171 | 162784.58777 | 1046.70056 | 205277.25 | 1065.0 | 3.0 | 361536.0 | 2520.0 | 4.0 | 622393.25 | 4061.25 | 4.0 | 1.36984 | 1.86575 | 174805.99786 | 2109.68227 | 2.0 |  |  |  |  |
| 8 | Vera Rubin NVL72 | 8 |  |  | 56 | 2026-11-03 | 36 |  | Dynamo TensorRT LLM | FP4 | 4 | 4 | 3929526.384848485 | 12967.43707 | Muse-Spark-Open | off | agentic_traces | 1793692800 | 32 | disagg | 4 | 115939.24326 | 95.96129625 | 0.0009599999999999999 | 0.001 | 1.7572 | 0.0009599999999999999 | 0.8966799999999999 | 3.13494 | 0.00099 | 2.11998 | 1041.6666666667 | 1000.0 |  |  |  |  | 3608.78172 | 1673597686 | 11081708 |  |  | 0.00103 | 7.13983 | 0.00102 | 5.280270000000001 | 970.8737864078 | agentic | 8 | 8 | 10526 | 1363.12619 | 0.97714 | 2353.54565 | 158996.55007 | 1052.79384 | 214297.75 | 1082.75 | 4.0 | 387953.0 | 2520.0 | 6.0 | 574374.5 | 4072.5 | 6.0 | 1.94037 | 2.91713 | 173152.06064 | 2032.00361 | 3.0 |  |  |  |  |
| 4 | Vera Rubin NVL72 | 4 |  |  | 84 | 2026-11-03 | 32 |  | Dynamo TensorRT LLM | FP4 | 4 | 4 | 5964857.687878788 | 19684.03037 | Muse-Spark-Open | off | agentic_traces | 1793692800 | 28 | disagg | 4 | 156318.24216 | 164.8572571429 | 0.00126 | 0.0012900000000000001 | 3.00028 | 0.00125 | 1.76151 | 4.804 | 0.00128 | 3.4677 | 793.6507936508 | 775.1937984496 |  |  |  |  | 3603.4208 | 2253121619 | 16633402 |  |  | 0.00133 | 10.860389999999999 | 0.00132 | 8.49246 | 751.8796992481 | agentic | 4 | 4 | 15704 | 1364.91626 | 0.97279 | 2555.92862 | 143474.37717 | 1059.1825 | 190571.75 | 1094.0 | 6.0 | 329136.8 | 2419.4 | 7.0 | 518643.6 | 3998.55 | 8.0 | 2.27636 | 4.35803 | 160841.54838 | 2009.64976 | 4.0 |  |  |  |  |
| 4 | Vera Rubin NVL72 | 4 |  |  | 100 | 2026-11-03 | 20 |  | Dynamo TensorRT LLM | FP4 | 4 | 4 | 10498624.645454545 | 34645.46133 | Muse-Spark-Open | off | agentic_traces | 1793692800 | 16 | disagg | 4 | 171964.5499 | 315.6891875 | 0.0020299999999999997 | 0.002 | 3.4006 | 0.0020299999999999997 | 1.66227 | 5.15325 | 0.00199 | 3.08948 | 492.6108374384 | 500.0 |  |  |  |  | 3603.34781 | 2478592340 | 18200607 |  |  | 0.00216 | 11.09568 | 0.00215 | 7.73234 | 462.962962963 | agentic | 4 | 4 | 17194 | 1259.13038 | 0.97275 | 2332.25221 | 144154.49226 | 1058.54409 | 192987.0 | 1099.0 | 6.0 | 316988.1 | 2408.0 | 8.0 | 485335.9 | 3999.0 | 9.0 | 2.44487 | 4.77186 | 154705.98729 | 1986.80987 | 5.0 |  |  |  |  |
| 4 | Vera Rubin NVL72 | 4 |  |  | 100 | 2026-11-03 | 24 |  | Dynamo TensorRT LLM | FP4 | 4 | 4 | 8654943.084848486 | 28561.31218 | Muse-Spark-Open | off | agentic_traces | 1793692800 | 20 | disagg | 4 | 170114.80878 | 250.61286 | 0.00175 | 0.00173 | 4.0736799999999995 | 0.00175 | 2.32508 | 5.72693 | 0.00172 | 3.93416 | 571.4285714286 | 578.0346820809 |  |  |  |  | 3603.48999 | 2452028044 | 18061619 |  |  | 0.00185 | 12.19957 | 0.00185 | 9.3353 | 540.5405405405 | agentic | 4 | 4 | 17030 | 1261.7917 | 0.97265 | 2339.52732 | 143982.85637 | 1060.57657 | 192490.25 | 1099.75 | 6.0 | 316481.5 | 2415.0 | 8.0 | 482603.95 | 4016.75 | 9.0 | 2.45471 | 4.72634 | 154466.43335 | 1993.97553 | 5.0 |  |  |  |  |
| 1 | GB300 NVL72 | 1 |  |  | 8 | 2026-11-06 | 1 |  | SGLang | FP4 |  |  | 18570378.108490564 | 39369.20159 | Qwen 3.8 Flash Next | mtp | agentic_traces | 1793952000 |  | agg |  | 38988.1884 | 381.01318 | 0.00342 | 0.0035800000000000003 | 2.62361 | 0.00342 | 0.6645 | 5.75818 | 0.0035800000000000003 | 2.46066 | 292.3976608187 | 279.3296089385 |  |  |  |  | 3609.9092 | 140743820 | 1375423 |  |  | 0.00378 | 14.309629999999999 | 0.00378 | 5.65986 | 264.5502645503 | agentic |  |  | 1452 | 1753.78887 | 0.95932 | 2805.25119 | 96931.00551 | 947.26102 | 122676.5 | 813.75 | 1.0 | 168414.9 | 2161.3 | 1.0 | 195585.35 | 4116.2 | 2.0 | 0.63491 | 0.40261 | 50074.77154 | 2018.06612 | 0.0 | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/42124076518/attempts/5 |  |  |  |
| 1 | GB300 NVL72 | 1 |  |  | 20 | 2026-11-06 | 1 |  | SGLang | FP4 |  |  | 48367113.0754717 | 102538.27972 | Qwen 3.8 Flash Next | mtp | agentic_traces | 1793952000 |  | agg |  | 101673.28556 | 864.99417 | 0.00497 | 0.0054 | 3.8487 | 0.00496 | 1.05511 | 8.01518 | 0.0054 | 3.2327199999999996 | 201.2072434608 | 185.1851851852 |  |  |  |  | 3606.84051 | 366719325 | 3119896 |  |  | 0.0059900000000000005 | 18.34506 | 0.0059900000000000005 | 6.3117399999999995 | 166.9449081803 | agentic |  |  | 3388 | 1445.19138 | 0.96425 | 2262.83039 | 108240.65083 | 920.86659 | 150492.0 | 914.5 | 2.0 | 201410.5 | 2318.1 | 2.0 | 225901.8 | 3834.6 | 3.0 | 0.95283 | 0.94083 | 61644.56761 | 1740.56314 | 1.0 | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/42124076518/attempts/5 |  |  |  |
| 4 | Vera Rubin NVL72 | 4 |  |  | 48 | 2026-11-03 | 4 |  | Dynamo TensorRT LLM | FP4 |  |  | 25762695.53939394 | 85016.89528 | Muse-Spark-Open | off | agentic_traces | 1793692800 |  | agg |  | 84462.04643 | 554.84885 | 0.0044599999999999996 | 0.00443 | 2.64396 | 0.0044599999999999996 | 0.39544999999999997 | 5.485180000000001 | 0.00442 | 0.83847 | 224.2152466368 | 225.7336343115 |  |  |  |  | 3623.87613 | 1224319977 | 8042814 |  |  | 0.00518 | 12.78131 | 0.00517 | 1.267 | 193.0501930502 | agentic |  |  | 7674 | 1286.24337 | 0.97636 | 2145.30573 | 159541.30532 | 1048.0602 | 205624.75 | 1067.0 | 3.0 | 354042.2 | 2494.4 | 4.0 | 605783.85 | 4041.7 | 5.0 | 1.55626 | 2.11786 | 172984.0197 | 2093.28772 | 2.0 |  |  |  |  |
| 8 | Vera Rubin NVL72 | 8 |  |  | 1 | 2026-11-03 | 8 |  | Dynamo TensorRT LLM | FP4 |  |  | 1265908.1909090912 | 4177.49703 | Muse-Spark-Open | off | agentic_traces | 1793692800 |  | agg |  | 4151.4165 | 26.08054 | 0.0009699999999999999 | 0.0009699999999999999 | 0.76022 | 0.0009699999999999999 | 0.14045 | 1.7439500000000001 | 0.0009699999999999999 | 0.2422 | 1030.9278350515 | 1030.9278350515 |  |  |  |  | 3599.93059 | 119558490 | 751105 |  |  | 0.00101 | 4.825060000000001 | 0.00101 | 0.27438999999999997 | 990.099009901 | agentic |  |  | 488 | 2115.86885 | 0.98194 | 2537.93454 | 244996.90574 | 1539.14959 | 400632.5 | 2121.25 | 0.0 | 572132.6 | 4655.7 | 1.0 | 634806.65 | 5950.15 | 1.0 | 0.36642 | 0.13535 | 210269.22687 | 2295.75252 | 0.0 |  |  |  |  |
| 4 | Vera Rubin NVL72 | 4 |  |  | 96 | 2026-11-03 | 4 |  | Dynamo TensorRT LLM | FP4 |  |  | 41711251.2969697 | 137647.12928 | Muse-Spark-Open | off | agentic_traces | 1793692800 |  | agg |  | 136589.52376 | 1057.60552 | 0.008039999999999999 | 0.0077599999999999995 | 4.70131 | 0.008039999999999999 | 0.38817 | 9.31319 | 0.00775 | 1.26683 | 124.3781094527 | 128.8659793814 |  |  |  |  | 3628.15997 | 1982936683 | 15381265 |  |  | 0.00936 | 20.169439999999998 | 0.009349999999999999 | 2.9708699999999997 | 106.8376068376 | agentic |  |  | 14804 | 1256.46398 | 0.97049 | 2358.52779 | 133901.28128 | 1036.7906 | 173963.0 | 1076.25 | 5.0 | 289250.1 | 2321.4 | 7.0 | 445513.2 | 3870.1 | 8.0 | 2.25409 | 4.08021 | 145513.12948 | 1986.21829 | 4.0 |  |  |  |  |
| 4 | Vera Rubin NVL72 | 4 |  |  | 24 | 2026-11-03 | 4 |  | Dynamo TensorRT LLM | FP4 |  |  | 13443524.754545456 | 44363.63169 | Muse-Spark-Open | off | agentic_traces | 1793692800 |  | agg |  | 44096.48429 | 267.1474 | 0.00232 | 0.00227 | 3.61541 | 0.00232 | 1.19275 | 5.4380299999999995 | 0.00227 | 3.09189 | 431.0344827586 | 440.5286343612 |  |  |  |  | 3606.97318 | 636219344 | 3854374 |  |  | 0.00256 | 12.31178 | 0.00256 | 8.1166 | 390.625 | agentic |  |  | 3819 | 1509.90556 | 0.979 | 2261.21719 | 166593.17727 | 1009.26263 | 220814.0 | 1040.5 | 2.0 | 334264.6 | 2625.4 | 2.0 | 434031.5 | 4225.2 | 3.0 | 1.0391 | 1.05879 | 145463.77339 | 1766.48633 | 1.0 |  |  |  |  |
| 1 | Vera Rubin NVL72 | 1 |  |  | 336 | 2026-11-05 | 4 |  | Dynamo SGLang | FP4 | 1 | 1 | 96330392.07878788 | 317890.29386 | Qwen 3.8 Flash Next | mtp | agentic_traces | 1793865600 | 2 | disagg | 2 | 629676.66706 | 6103.92066 | 0.00664 | 0.00723 | 6.07203 | 0.00662 | 2.0576399999999997 | 14.469719999999999 | 0.00721 | 8.465770000000001 | 150.6024096386 | 138.3125864454 |  |  |  |  | 3629.50319 | 9158941658 | 44476499 |  |  | 0.00736 | 33.80232 | 0.00732 | 11.01609 | 135.8695652174 | agentic | 1 | 1 | 47692 | 1290.5117 | 0.95715 | 2430.34647 | 95840.53799 | 929.05307 | 139104.75 | 933.0 | 16.0 | 193524.7 | 2086.9 | 19.0 | 221061.5 | 3512.35 | 21.0 | 4.47149 | 13.18132 | 64289.2707 | 1814.3293 | 13.0 |  |  |  |  |
| 1 | Vera Rubin NVL72 | 1 |  |  | 176 | 2026-11-05 | 3 |  | Dynamo SGLang | FP4 | 1 | 1 | 68994174.73030303 | 227680.77661 | Qwen 3.8 Flash Next | mtp | agentic_traces | 1793865600 | 2 | disagg | 1 | 676744.22481 | 3149.05251 | 0.00417 | 0.00475 | 3.0450999999999997 | 0.00417 | 0.93368 | 13.47634 | 0.00475 | 9.688049999999999 | 239.8081534772 | 210.5263157895 |  |  |  |  | 3629.2353 | 4915723165 | 22917533 |  |  | 0.0048200000000000005 | 22.37418 | 0.0048200000000000005 | 5.21209 | 207.468879668 | agentic | 1 | 1 | 24718 | 1271.15595 | 0.95843 | 2337.72915 | 99363.38001 | 924.72308 | 142815.5 | 909.75 | 9.0 | 199267.6 | 2038.3 | 11.0 | 224989.7 | 3493.75 | 12.0 | 2.86775 | 6.81257 | 64823.68498 | 1896.24784 | 7.0 |  |  |  |  |
| 1 | GB300 NVL72 | 1 |  |  | 36 | 2026-11-06 | 1 |  | SGLang | FP4 |  |  | 69980153.51415095 | 148357.92545 | Qwen 3.8 Flash Next | mtp | agentic_traces | 1793952000 |  | agg |  | 147026.65962 | 1331.26583 | 0.0073 | 0.00785 | 5.16778 | 0.00729 | 1.22071 | 12.34035 | 0.00785 | 4.739520000000001 | 136.9863013699 | 127.3885350318 |  |  |  |  | 3623.19823 | 532706733 | 4823440 |  |  | 0.00932 | 27.07932 | 0.00932 | 6.58829 | 107.2961373391 | agentic |  |  | 4960 | 1332.03465 | 0.96387 | 2100.21223 | 107400.55101 | 972.46774 | 151904.0 | 996.25 | 2.0 | 210620.8 | 2336.3 | 3.0 | 235168.75 | 3783.35 | 3.0 | 1.11599 | 1.37065 | 65624.11292 | 2140.77976 | 1.0 | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/42124076518/attempts/5 |  |  |  |
| 1 | GB300 NVL72 | 1 |  |  | 80 | 2026-11-06 | 1 |  | SGLang | FP4 |  |  | 109729171.58962265 | 232625.84377 | Qwen 3.8 Flash Next | mtp | agentic_traces | 1793952000 |  | agg |  | 230069.66914 | 2556.17463 | 0.02249 | 0.02246 | 13.86957 | 0.02249 | 2.4347 | 24.49542 | 0.02246 | 3.5575900000000003 | 44.4642063139 | 44.5235975067 |  |  |  |  | 3628.12263 | 834720972 | 9274115 |  |  | 0.028489999999999998 | 49.946239999999996 | 0.028489999999999998 | 8.08701 | 35.1000351 | agentic |  |  | 10099 | 1191.38067 | 0.94757 | 2198.38411 | 82653.82434 | 918.32013 | 118792.0 | 1008.5 | 4.0 | 179438.2 | 1970.2 | 5.0 | 216845.9 | 3388.0 | 6.0 | 1.76895 | 2.7895 | 63225.70834 | 1779.53647 | 3.0 | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/42124076518/attempts/5 |  |  |  |
| 1 | GB300 NVL72 | 1 |  |  | 56 | 2026-11-06 | 1 |  | SGLang | FP4 |  |  | 90510661.25471698 | 191882.60186 | Qwen 3.8 Flash Next | mtp | agentic_traces | 1793952000 |  | agg |  | 189884.92427 | 1997.67759 | 0.013460000000000001 | 0.013789999999999998 | 9.05 | 0.013460000000000001 | 1.5366099999999998 | 15.98126 | 0.013779999999999999 | 2.98787 | 74.294205052 | 72.5163161711 |  |  |  |  | 3628.06693 | 689231488 | 7267719 |  |  | 0.01697 | 33.96089 | 0.01697 | 7.6235 | 58.9275191514 | agentic |  |  | 7773 | 1188.78961 | 0.95503 | 2021.98785 | 88629.25691 | 932.42094 | 123833.0 | 981.0 | 3.0 | 193810.4 | 2145.6 | 4.0 | 228039.0 | 3525.2 | 5.0 | 1.53528 | 2.14787 | 64433.22743 | 1870.63601 | 2.0 | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/42124076518/attempts/5 |  |  |  |
| 1 | Vera Rubin NVL72 | 1 |  |  | 10 | 2026-11-05 | 1 |  | Dynamo SGLang | FP4 |  |  | 20247704.97878788 | 66817.42643 | Qwen 3.8 Flash Next | mtp | agentic_traces | 1793865600 |  | agg |  | 66194.44063 | 622.9858 | 0.0019199999999999998 | 0.0020099999999999996 | 1.0673499999999998 | 0.00191 | 0.15514 | 2.3672600000000004 | 0.0020099999999999996 | 0.38091 | 520.8333333333 | 497.5124378109 |  |  |  |  | 3604.71619 | 238612172 | 2245687 |  |  | 0.0024500000000000004 | 5.03095 | 0.0024500000000000004 | 0.84245 | 408.1632653061 | agentic |  |  | 2290 | 1732.82153 | 0.96196 | 2621.32752 | 104197.45502 | 980.64934 | 134181.0 | 884.0 | 1.0 | 187264.8 | 2258.4 | 2.0 | 217583.7 | 4315.45 | 2.0 | 0.84808 | 0.63566 | 55625.16361 | 2050.93232 | 0.0 |  |  |  |  |
| 1 | Vera Rubin NVL72 | 1 |  |  | 120 | 2026-11-05 | 1 |  | Dynamo SGLang | FP4 |  |  | 142742298.84848484 | 471049.5862 | Qwen 3.8 Flash Next | mtp | agentic_traces | 1793865600 |  | agg |  | 466265.59902 | 4783.98718 | 0.01114 | 0.01099 | 5.98361 | 0.011130000000000001 | 0.64871 | 12.468860000000001 | 0.010960000000000001 | 2.04789 | 89.7666068223 | 90.991810737 |  |  |  |  | 3620.68195 | 1688587013 | 17346396 |  |  | 0.01427 | 28.40706 | 0.01426 | 3.85377 | 70.0770847933 | agentic |  |  | 18319 | 1249.32125 | 0.9562 | 2302.495 | 92155.65457 | 945.5372 | 135310.0 | 977.5 | 7.0 | 191807.6 | 2084.0 | 9.0 | 222126.6 | 3532.1 | 10.0 | 2.664 | 5.06582 | 65257.32412 | 1832.49107 | 5.0 |  |  |  |  |
| 1 | Vera Rubin NVL72 | 1 |  |  | 13 | 2026-11-05 | 1 |  | Dynamo SGLang | FP4 |  |  | 25716378.924242422 | 84864.05045 | Qwen 3.8 Flash Next | mtp | agentic_traces | 1793865600 |  | agg |  | 84140.60551 | 723.44494 | 0.00196 | 0.002 | 1.11558 | 0.00195 | 0.20775 | 2.4826599999999996 | 0.00198 | 0.65322 | 510.2040816327 | 500.0 |  |  |  |  | 3603.06897 | 303164405 | 2606622 |  |  | 0.00226 | 5.72499 | 0.00225 | 1.20068 | 442.4778761062 | agentic |  |  | 2889 | 1612.91741 | 0.96248 | 2485.81692 | 104937.48875 | 902.25753 | 141303.0 | 789.0 | 1.0 | 191283.2 | 2079.2 | 2.0 | 219741.2 | 3912.0 | 3.0 | 0.93244 | 0.80217 | 58158.7021 | 1924.27778 | 1.0 |  |  |  |  |
| 1 | Vera Rubin NVL72 | 4 |  |  | 1 | 2026-11-05 | 4 |  | Dynamo SGLang | FP4 |  |  | 1854503.4727272727 | 6119.86146 | Qwen 3.8 Flash Next | mtp | agentic_traces | 1793865600 |  | agg |  | 6062.14748 | 57.71398 | 0.00138 | 0.00138 | 0.55875 | 0.00138 | 0.15409 | 1.58138 | 0.0013700000000000001 | 0.35847 | 724.6376811594 | 724.6376811594 |  |  |  |  | 3598.3701 | 87255401 | 830705 |  |  | 0.0014199999999999998 | 4.44796 | 0.00141 | 0.40826999999999997 | 704.2253521127 | agentic |  |  | 936 | 1594.32265 | 0.95414 | 2435.26488 | 93221.58226 | 887.50534 | 126341.5 | 688.75 | 0.0 | 193096.5 | 2402.5 | 1.0 | 226450.5 | 4248.25 | 1.0 | 0.48796 | 0.26016 | 61291.76998 | 1908.03353 | 0.0 |  |  |  |  |
| 1 | Vera Rubin NVL72 | 1 |  |  | 22 | 2026-11-05 | 1 |  | Dynamo SGLang | FP4 |  |  | 40840575.63636364 | 134773.8996 | Qwen 3.8 Flash Next | mtp | agentic_traces | 1793865600 |  | agg |  | 133674.86489 | 1099.03471 | 0.00226 | 0.00232 | 1.48226 | 0.0022400000000000002 | 0.46168 | 2.9725300000000003 | 0.0023 | 0.81848 | 442.4778761062 | 431.0344827586 |  |  |  |  | 3601.16561 | 481385327 | 3957806 |  |  | 0.00282 | 6.91622 | 0.0028 | 1.33473 | 354.609929078 | agentic |  |  | 4350 | 1371.62273 | 0.96568 | 2087.62854 | 110663.29356 | 909.84046 | 157331.5 | 953.5 | 2.0 | 211111.3 | 2316.7 | 3.0 | 232056.85 | 3742.7 | 3.0 | 1.17318 | 1.2094 | 64280.95195 | 1580.3033 | 1.0 |  |  |  |  |
| 4 | GB300 NVL72 | 4 |  |  | 3 | 2026-11-06 | 4 |  | SGLang | FP4 |  |  | 2462223.0566037735 | 5219.91288 | Qwen 3.8 Flash Next | mtp | agentic_traces | 1793952000 |  | agg |  | 5162.09759 | 57.81528 | 0.00239 | 0.00242 | 1.45959 | 0.00239 | 0.3491 | 4.3128400000000005 | 0.0024100000000000002 | 1.7753299999999999 | 418.410041841 | 413.2231404959 |  |  |  |  | 3598.36514 | 74300448 | 832162 |  |  | 0.00254 | 11.821959999999999 | 0.00254 | 5.03801 | 393.7007874016 | agentic |  |  | 802 | 1638.50623 | 0.95065 | 2643.33212 | 92643.95012 | 1037.60848 | 130052.0 | 830.5 | 0.0 | 203933.8 | 2718.4 | 1.0 | 231492.7 | 5220.05 | 1.0 | 0.46963 | 0.22293 | 65422.23512 | 2152.7154 | 0.0 | https://github.com/SemiAnalysisAI/InferenceX/actions/runs/42124076518/attempts/5 |  |  |  |
| 8 | GB300 NVL72 | 8 | 1024 | 1024 | 8192 | 2026-06-15 | 16 | 1K/1K | Dynamo TensorRT LLM | FP4 | 4 | 4 | 5587543.710053067 | 11845.5926653125 | DeepSeek-V4-Pro | mtp | single_turn | 1781506800 | 8 | disagg | 8 | 11849.9939021185 | 11841.1914285065 | 14.513413675013 | 12.5502723431211 | 76.0489374310127 | 0.07699958096899999 | 2.2581479385262 | 77.5894460005816 | 0.0732407522266 | 10.177539545207 | 12.9870836622 | 13.6536008929 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 4 | GB300 NVL72 | 4 | 1024 | 1024 | 15 | 2026-06-15 | 24 | 1K/1K | Dynamo TensorRT LLM | FP4 | 4 | 4 | 64371.27680094339 | 136.467106818 | DeepSeek-V4-Pro | mtp | single_turn | 1781506800 | 20 | disagg | 4 | 412.7232333795 | 81.2158815057 | 1.5179410870187 | 1.3190271338342 | 8.0136697939597 | 0.0054536146498 | 2.6764435659861 | 8.0119803651391 | 0.0057365783884999995 | 2.779839591013 | 183.3646240541 | 174.3199399142 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 8 | B300 | 8 | 1024 | 1024 | 8609 | 2026-01-28 | 26 | 1K/1K | Dynamo TensorRT LLM | FP4 | 2 | 2 | 7257145.211366736 | 13788.5759015968 | DeepSeek-R1 | mtp | single_turn | 1769587200 | 16 | disagg | 10 | 17928.2711817752 | 11201.2663514853 | 7.5244735160377 | 6.7733019084734 | 41.0306475614198 | 0.042322176760799995 | 2.122890813509 | 42.6891213911476 | 0.0413759694217 | 4.6137865847534005 | 23.6282742651 | 24.1686180161 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 8 | H200 | 8 | 8192 | 1024 | 48 | 2026-03-23 | 56 | 8K/1K | Dynamo TensorRT LLM | FP8 | 8 | 8 | 324737.0684863504 | 444.8897838263 | DeepSeek-R1 | mtp | single_turn | 1774249200 | 48 | disagg | 8 | 2766.8533360698 | 57.8958584523 | 2.1325586325256 | 1.9446091976934 | 15.5658714940073 | 0.009120842769699999 | 7.1218101484701 | 15.2945382871068 | 0.0087040239289 | 7.240615400263501 | 109.6389911825 | 114.8893900296 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 8 | H200 | 8 | 1024 | 1024 | 1024 | 2026-03-23 | 40 | 1K/1K | Dynamo TensorRT LLM | FP8 | 8 | 8 | 1152917.8308645987 | 1579.4974282845 | DeepSeek-R1 | mtp | single_turn | 1774249200 | 32 | disagg | 8 | 3948.6272878215 | 987.2149634002 | 5.7791159139015 | 5.1160261801011 | 27.8011706279358 | 0.024539258218299997 | 5.098565860535 | 28.5420249824912 | 0.023912256876099998 | 6.529623042377899 | 40.7510280507 | 41.819557442 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 8 | B300 | 8 | 1024 | 1024 | 12926 | 2026-01-28 | 26 | 1K/1K | Dynamo TensorRT LLM | FP4 | 2 | 2 | 7671806.9533240525 | 14576.4332113157 | DeepSeek-R1 | mtp | single_turn | 1769587200 | 16 | disagg | 10 | 18950.892090478 | 11842.3964118392 | 10.640068282024 | 9.525670334418301 | 58.4516322450945 | 0.0599475257789 | 3.4536425890401 | 60.7731927356887 | 0.0581497361196 | 7.2595010288409005 | 16.6812555982 | 17.1969825958 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B200 | 2 | 1024 | 1024 | 128 | 2026-02-05 | 2 | 1K/1K | TensorRT LLM | FP4 |  |  | 12370212.81871345 | 21153.06392 | gpt-oss-120b | eagle |  | 1770278400 |  | agg |  | 10628.45 | 10524.62 |  |  | 5.81823 | 0.0056289999999999995 | 0.059314 |  |  |  | 177.6376 |  |  |  | 3 |  | 55.94816 | 1189284 | 1177666 | 22.87832 | FP8 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B200 | 4 | 1024 | 1024 | 8 | 2026-02-05 | 4 | 1K/1K | TensorRT LLM | FP4 |  |  | 1275137.1608187135 | 2180.484545 | gpt-oss-120b | eagle |  | 1770278400 |  | agg |  | 1091.704 | 1088.781 |  |  | 1.7503119999999999 | 0.001674 | 0.037753999999999996 |  |  |  | 597.3522 |  |  |  | 3 |  | 17.01858 | 74317 | 74118 | 4.700745 | FP8 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B200 | 2 | 1024 | 1024 | 10 | 2026-02-05 | 2 | 1K/1K | TensorRT LLM | FP4 |  |  | 2336776.0953216376 | 3995.887123 | gpt-oss-120b | eagle |  | 1770278400 |  | agg |  | 2008.194 | 1987.693 |  |  | 2.36565 | 0.002274 | 0.039445999999999995 |  |  |  | 439.7722 |  |  |  | 3 |  | 23.14442 | 92957 | 92008 | 4.320695 | FP8 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 1 | H200 | 4 | 8192 | 1024 | 1 | 2026-06-05 | 4 | 8K/1K | vLLM | FP8 |  |  | 207789.9680613869 | 284.6722562441 | MiniMax-M2.5 | off | single_turn | 1780642800 |  | agg |  | 252.8707047787 | 31.8015514655 | 0.007609709166 | 0.0076158869826 | 7.1973518542945 | 0.0075993114011 | 0.2245859559625 | 7.312617384176701 | 0.0076139910521 | 0.2366970940493 | 131.5908701746 | 131.3371651157 |  |  |  |  |  |  |  |  |  | 0.0077086752281 | 7.8399149456061 | 0.0076702428642 | 0.2725449006073 | 130.3739682966 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 16 | GB300 NVL72 | 16 | 8192 | 1024 | 256 | 2026-06-03 | 20 | 8K/1K | Dynamo SGLang | FP4 | 4 | 4 | 1441325.4519222172 | 3055.6099580751 | DeepSeek-V4-Pro | mtp | single_turn | 1780470000 | 16 | disagg | 4 | 13580.1441432278 | 424.476411787 | 0.7970074745826 | 1.4635452816841 | 32.361331379041104 | 0.0073370727266 | 24.5801331936382 | 31.7068590470133 | 0.0084445143004 | 23.9340616291867 | 136.2941376294 | 118.4200730118 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 1 | GB300 NVL72 | 4 | 8192 | 1024 | 1 | 2026-06-03 | 8 | 8K/1K | Dynamo SGLang | FP4 | 1 | 4 | 91214.42909212265 | 193.3745896753 | DeepSeek-V4-Pro | mtp | single_turn | 1780470000 | 4 | disagg | 4 | 343.5495105528 | 43.1996687979 | 0.0156661910005 | 0.0156938181042 | 5.056824533036 | 0.0040869804479000005 | 1.0243052095175 | 5.3824310713913 | 0.004300664597 | 1.3710916855838 | 244.6794186442 | 232.5222014983 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 1 | GB300 NVL72 | 4 | 8192 | 1024 | 8 | 2026-06-03 | 28 | 8K/1K | Dynamo SGLang | FP4 | 4 | 4 | 154526.17151966982 | 327.5954836217 | DeepSeek-V4-Pro | mtp | single_turn | 1780470000 | 24 | disagg | 4 | 2035.4088784718 | 42.9599178133 | 0.0172008594964 | 0.0173189096862 | 6.1888493159786 | 0.004934216506500001 | 1.2741082410794 | 6.8101945686154 | 0.0055208793165 | 1.6989514714922 | 202.6664210368 | 181.1305668292 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 8 | B300 | 8 | 1024 | 1024 | 654 | 2026-01-28 | 10 | 1K/1K | Dynamo TensorRT LLM | FP4 | 2 | 2 | 3844119.452105053 | 7303.8269589996 | DeepSeek-R1 | mtp | single_turn | 1769587200 | 8 | disagg | 2 | 18264.315941759 | 4563.7047133098 | 3.2154342789436 | 2.8258965525304003 | 15.862855603103501 | 0.0135885171192 | 3.3765829774784 | 15.7156265550798 | 0.012986853768699999 | 3.7682201236046 | 73.5915472769 | 77.0009440168 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 8 | H200 | 8 | 1024 | 1024 | 352 | 2026-03-23 | 96 | 1K/1K | Dynamo TensorRT LLM | FP8 | 8 | 8 | 294165.8698764964 | 403.0072417308 | DeepSeek-R1 | mtp | single_turn | 1774249200 | 88 | disagg | 8 | 2416.3864516296 | 219.9727681037 | 3.7587406649254 | 3.3297293832575 | 16.0249405568466 | 0.015974612093 | 1.2521666130051001 | 16.1018847523605 | 0.015568193962200001 | 1.7717513356321999 | 62.5993291215 | 64.2335265368 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B200 | 1 | 1024 | 1024 | 1 | 2026-02-05 | 1 | 1K/1K | TensorRT LLM | FP4 |  |  | 826534.7093567251 | 1413.374353 | gpt-oss-120b | eagle |  | 1770278400 |  | agg |  | 705.2645 | 708.1099 |  |  | 1.196685 | 0.001142 | 0.028784 |  |  |  | 875.9301 |  |  |  | 2 |  | 13.00363 | 9171 | 9208 | 0.769016 | FP8 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B200 | 1 | 8192 | 1024 | 64 | 2026-02-05 | 1 | 8K/1K | TensorRT LLM | FP4 |  |  | 26465229.292397663 | 45255.54209 | gpt-oss-120b | eagle |  | 1770278400 |  | agg |  | 40241.05 | 5014.492 |  |  | 12.62134 | 0.012163 | 0.178562 |  |  |  | 82.21638 |  |  |  | 2 |  | 117.6293 | 4733525 | 589851 | 5.440823 | FP8 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | GB300 NVL72 |  | 131072 | 8192 | 256 | 2025-10-31 |  | 128K/8K | TensorRT LLM | FP4 |  |  | 91094.33962264152 | 193.12 | DeepSeek-R1 | mtp |  | 1761894000 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 105.3741 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 1 | H200 | 4 | 8192 | 1024 | 16 | 2026-06-05 | 4 | 8K/1K | vLLM | FP8 |  |  | 1568566.2944001462 | 2148.9358233282 | MiniMax-M2.5 | off | single_turn | 1780642800 |  | agg |  | 1910.8199869818 | 238.1158363464 | 0.013352463487499999 | 0.0158185986026 | 15.020055574365001 | 0.0159954606943 | 0.2520945505239 | 14.9569875109009 | 0.0158037249625 | 0.49760214287090004 | 62.517736695 | 63.2762214208 |  |  |  |  |  |  |  |  |  | 0.0141692050733 | 16.659811503533298 | 0.016420186971500002 | 0.6127600804903 | 60.9006463651 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 32 | GB300 NVL72 | 32 | 1024 | 1024 | 666 | 2026-06-22 | 36 | 1K/1K | Dynamo TensorRT LLM | FP4 | 4 | 4 | 1047018.1896746226 | 2219.6785621102 | Kimi K2.5/2.6 | off | single_turn | 1782111600 | 32 | disagg | 4 | 10002.7126557868 | 1246.7993004007 | 1.1976373725338 | 1.1454662726773999 | 13.717044361867 | 0.0120707270371 | 2.2639850485139 | 14.584715410713398 | 0.0120898283328 | 3.4533976601316 | 82.8450512488 | 82.7141604057 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 4 | GB300 NVL72 | 4 | 1024 | 1024 | 10 | 2026-06-15 | 24 | 1K/1K | Dynamo TensorRT LLM | FP4 | 4 | 4 | 37084.4155265566 | 78.6189609163 | DeepSeek-V4-Pro | mtp | single_turn | 1781506800 | 20 | disagg | 4 | 236.9768571052 | 46.9473816786 | 1.3340574320173 | 1.1577961430254 | 9.5337457649293 | 0.0047630198337 | 4.9097944915411 | 9.214984409968599 | 0.0049420256521 | 4.6880016868922 | 209.9508368467 | 202.3461775391 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 8 | GB300 NVL72 | 8 | 8192 | 1024 | 512 | 2026-06-03 | 16 | 8K/1K | Dynamo SGLang | FP4 | 4 | 4 | 3440833.134963066 | 7294.5662461217 | DeepSeek-V4-Pro | mtp | single_turn | 1780470000 | 8 | disagg | 8 | 12968.2340495524 | 1620.898442691 | 1.3160747655201999 | 2.4050855283754 | 32.0834125680849 | 0.011481770045399999 | 19.633636169834098 | 33.0487133824372 | 0.0130451591652 | 21.035499178599302 | 87.0945852462 | 76.6567879577 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 1 | GB300 NVL72 | 4 | 8192 | 1024 | 64 | 2026-06-03 | 28 | 8K/1K | Dynamo SGLang | FP4 | 4 | 4 | 754412.1020806604 | 1599.353656411 | DeepSeek-V4-Pro | mtp | single_turn | 1780470000 | 24 | disagg | 4 | 9953.5847559571 | 206.9818064866 | 0.0237789065577 | 0.024056054856900003 | 10.5647596861236 | 0.0068259806862 | 3.5513929389417 | 11.1216061303021 | 0.007537346633200001 | 4.1854186884177995 | 146.499096023 | 132.6726829305 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 8 | H200 | 8 | 8192 | 1024 | 256 | 2026-03-23 | 32 | 8K/1K | Dynamo TensorRT LLM | FP8 | 8 | 8 | 1511295.0785126279 | 2070.4742575623 | DeepSeek-R1 | mtp | single_turn | 1774249200 | 8 | disagg | 24 | 2453.8331570085 | 920.3975592235 | 7.128685398958599 | 6.2615932894786 | 30.390580075560102 | 0.031841018426800004 | 1.0001772900577999 | 30.6716454043935 | 0.0306193466795 | 2.4773543244737004 | 31.4060306299 | 32.6590900344 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 8 | H200 | 8 | 1024 | 1024 | 44 | 2026-03-23 | 96 | 1K/1K | Dynamo TensorRT LLM | FP8 | 8 | 8 | 67995.0105530657 | 93.1531644577 | DeepSeek-R1 | mtp | single_turn | 1774249200 | 88 | disagg | 8 | 557.220880693 | 50.9651902545 | 2.0108163801487997 | 1.7882222674759 | 8.573992217425301 | 0.008567553439699999 | 0.28457346255890004 | 8.709044318647301 | 0.0083665543895 | 0.9668549947901001 | 116.7194353714 | 119.5235163062 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 8 | H200 | 8 | 8192 | 1024 | 128 | 2026-03-23 | 24 | 8K/1K | Dynamo TensorRT LLM | FP8 | 8 | 8 | 1331580.96543073 | 1824.2659226401 | DeepSeek-R1 | mtp | single_turn | 1774249200 | 8 | disagg | 16 | 2433.3367769047 | 606.124214111 | 5.3531723900232 | 4.7118089012799 | 23.0621167789213 | 0.0238648674936 | 1.0732285425765 | 23.2944489746889 | 0.023043236521299998 | 2.1170768351476 | 41.9026001409 | 43.3966816717 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 8 | H200 | 8 | 1024 | 1024 | 13 | 2026-03-23 | 96 | 1K/1K | Dynamo TensorRT LLM | FP8 | 8 | 8 | 27076.34636160584 | 37.0945945154 | DeepSeek-R1 | mtp | single_turn | 1774249200 | 88 | disagg | 8 | 224.0171466158 | 20.1016352336 | 1.3468476959970002 | 1.2083276187869 | 6.154064772534199 | 0.0057200187123 | 0.7743423988577 | 6.3153375526836 | 0.0055470999034 | 1.2496566695322 | 174.8246029086 | 180.2743807424 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 1 | H200 | 4 | 8192 | 1024 | 256 | 2026-06-05 | 4 | 8K/1K | vLLM | FP8 |  |  | 4433306.594195986 | 6073.6300340485 | MiniMax-M2.5 | off | single_turn | 1780642800 |  | agg |  | 5398.5881676152 | 675.0418664333 | 0.0502471146174 | 0.09081530394819999 | 86.7299884571694 | 0.0929712108878 | 0.6866031568497 | 86.41183020444879 | 0.0906308234926 | 2.7661818072014 | 10.7560178086 | 11.0337737368 |  |  |  |  |  |  |  |  |  | 0.2216530149803 | 99.90428454484791 | 0.0978414058588 | 1.6750296341255 | 10.2206217421 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 8 | GB300 NVL72 | 8 | 1024 | 1024 | 8 | 2026-06-15 | 36 | 1K/1K | Dynamo TensorRT LLM | FP4 | 4 | 4 | 21174.06645334906 | 44.8890208811 | DeepSeek-V4-Pro | mtp | single_turn | 1781506800 | 32 | disagg | 4 | 201.3612609878 | 25.3299908678 | 1.2762638080167001 | 1.1033146499322 | 7.9454655584996 | 0.0044815368053 | 3.9221377320355 | 8.7067804407649 | 0.0046991139131 | 4.3486877947317994 | 223.1377412372 | 212.8060775902 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 4 | GB300 NVL72 | 4 | 1024 | 1024 | 60 | 2026-06-22 | 24 | 1K/1K | Dynamo TensorRT LLM | FP4 | 4 | 4 | 175749.0638132547 | 372.5880152841 | Kimi K2.5/2.6 | off | single_turn | 1782111600 | 20 | disagg | 4 | 1118.8431964769 | 223.3369790456 | 0.8769973290619 | 0.8295367688758 | 11.702219558996099 | 0.008786612288 | 3.6655160235241 | 11.9723326866185 | 0.0087677160087 | 3.8912621451243 | 113.8095055554 | 114.0547890708 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 32 | GB300 NVL72 | 32 | 1024 | 1024 | 333 | 2026-06-22 | 36 | 1K/1K | Dynamo TensorRT LLM | FP4 | 4 | 4 | 559245.1055881132 | 1185.5996238468 | Kimi K2.5/2.6 | off | single_turn | 1782111600 | 32 | disagg | 4 | 5337.9573304585 | 666.5549105203 | 1.1348010299989 | 1.0809189919059 | 12.7463179579936 | 0.0114036693098 | 1.9344247580010998 | 13.6788629710231 | 0.0114167546954 | 3.1579182523141998 | 87.6910731826 | 87.5905655049 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 8 | GB300 NVL72 | 8 | 1024 | 1024 | 24 | 2026-06-22 | 36 | 1K/1K | Dynamo TensorRT LLM | FP4 | 4 | 4 | 65066.29948650944 | 137.9405549114 | Kimi K2.5/2.6 | off | single_turn | 1782111600 | 32 | disagg | 4 | 622.4512075455 | 77.3767233321 | 0.628494206816 | 0.5977060778618001 | 8.2787137804553 | 0.0062961911547 | 2.4169629674871 | 8.5888644875444 | 0.0063119671970000004 | 2.7985869907842003 | 158.8261816435 | 158.4292137129 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 8 | H200 | 8 | 8192 | 1024 | 64 | 2026-03-23 | 16 | 8K/1K | Dynamo TensorRT LLM | FP8 | 8 | 8 | 1088177.4229932118 | 1490.8030695007 | DeepSeek-R1 | mtp | single_turn | 1774249200 | 8 | disagg | 8 | 2650.8627670103 | 330.743371991 | 4.2325079796136995 | 3.7117418541156 | 20.792238746071202 | 0.0190165639352 | 3.4724854715167996 | 20.9988508990788 | 0.0183384768399 | 4.1220249922619 | 52.585735436 | 54.5301558428 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 8 | H200 | 8 | 1024 | 1024 | 144 | 2026-03-23 | 80 | 1K/1K | Dynamo TensorRT LLM | FP8 | 8 | 8 | 179663.42858649636 | 246.1388971635 | DeepSeek-R1 | mtp | single_turn | 1774249200 | 72 | disagg | 8 | 1232.1161136432 | 136.5858731102 | 3.0903065650491 | 2.7789475023307 | 12.873184332042 | 0.0132927277366 | 0.3139074085047 | 12.925549696154299 | 0.012952013944799999 | 1.033970366326 | 75.2291042004 | 77.2080700547 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 8 | H200 | 8 | 1024 | 1024 | 88 | 2026-03-23 | 96 | 1K/1K | Dynamo TensorRT LLM | FP8 | 8 | 8 | 110953.09988532848 | 152.0057468429 | DeepSeek-R1 | mtp | single_turn | 1774249200 | 88 | disagg | 8 | 913.2566097678 | 82.8011229406 | 2.5276487809606003 | 2.2571328926397 | 10.6023437589174 | 0.0107832621213 | 0.2943238376174 | 10.6261599911141 | 0.0105348434 | 0.9384431952719 | 92.7363156667 | 94.9231006128 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B200 | 2 | 8192 | 1024 | 64 | 2026-02-05 | 2 | 8K/1K | TensorRT LLM | FP4 |  |  | 20099699.56140351 | 34370.48625 | gpt-oss-120b | eagle |  | 1770278400 |  | agg |  | 30303.83 | 3776.202 |  |  | 8.271427 | 0.007989 | 0.098709 |  |  |  | 124.8632 |  |  |  | 1 |  | 78.10109 | 4733525 | 589851 | 8.194507 | FP8 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 1 | H200 | 4 | 8192 | 1024 | 8 | 2026-06-05 | 4 | 8K/1K | vLLM | FP8 |  |  | 1062423.8171978833 | 1455.5206295611 | MiniMax-M2.5 | off | single_turn | 1780642800 |  | agg |  | 1291.8955616259 | 163.6250679351 | 0.0104668010026 | 0.011641347409 | 11.142788457684201 | 0.011645068570799999 | 0.2414523852058 | 11.1153358526179 | 0.011627665454300001 | 0.3498545389157 | 85.8732599054 | 86.0017863371 |  |  |  |  |  |  |  |  |  | 0.0107848839834 | 12.3230052732863 | 0.012127916736599999 | 0.5753446418792 | 82.4543919389 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 32 | GB300 NVL72 | 32 | 1024 | 1024 | 2253 | 2026-06-22 | 40 | 1K/1K | Dynamo TensorRT LLM | FP4 | 4 | 4 | 2989977.4671514155 | 6338.752230361 | Kimi K2.5/2.6 | off | single_turn | 1782111600 | 32 | disagg | 8 | 15863.9866250141 | 3957.4436316978 | 1.4736939379945 | 1.4171433772836002 | 15.2410323570948 | 0.0150028991546 | 1.0940053289523 | 15.6394351068531 | 0.014959815853500001 | 1.8557188238601 | 66.653784025 | 66.8457426074 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 4 | GB300 NVL72 | 4 | 1024 | 1024 | 30 | 2026-06-22 | 24 | 1K/1K | Dynamo TensorRT LLM | FP4 | 4 | 4 | 105093.32180787736 | 222.7978422327 | Kimi K2.5/2.6 | off | single_turn | 1782111600 | 20 | disagg | 4 | 668.7314685816 | 133.6111169629 | 0.720449479064 | 0.6834093139299 | 9.737437851959799 | 0.0072113116552 | 3.1372722201049 | 9.868754026909201 | 0.0072257440509000005 | 3.2100694242244 | 138.671027936 | 138.3940522887 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 8 | H200 | 8 | 8192 | 1024 | 9 | 2026-03-23 | 64 | 8K/1K | Dynamo TensorRT LLM | FP8 | 8 | 8 | 112596.14075576644 | 154.2567128354 | DeepSeek-R1 | mtp | single_turn | 1774249200 | 56 | disagg | 8 | 1096.0663917866 | 19.7124729853 | 1.4488349300809 | 1.2763731882397 | 6.5885202679783 | 0.005981527008199999 | 1.3340362359303999 | 7.0340974044288 | 0.0055731402352 | 1.9144230640492998 | 167.1813900746 | 179.4320540651 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 8 | B300 | 8 | 1024 | 1024 | 20 | 2026-01-28 | 42 | 1K/1K | Dynamo TensorRT LLM | FP4 | 2 | 2 | 116353.03737905264 | 221.0707710202 | DeepSeek-R1 | mtp | single_turn | 1769587200 | 40 | disagg | 2 | 2322.6602891001 | 115.9912951162 | 0.9759253894444999 | 0.870681476059 | 3.9051218144596 | 0.0041424390571 | 0.1181764829671 | 3.8018023827835 | 0.00395034006 | 0.1710609053087 | 241.4036721385 | 253.1427636104 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B200 | 8 | 1024 | 1024 | 1 | 2026-02-05 | 8 | 1K/1K | TensorRT LLM | FP4 |  |  | 118893.6941520468 | 203.308217 | gpt-oss-120b | eagle |  | 1770278400 |  | agg |  | 101.4495 | 101.8588 |  |  | 1.042741 | 0.000983 | 0.037488 |  |  |  | 1017.655 |  |  |  | 3 |  | 11.29996 | 9171 | 9208 | 0.884959 | FP8 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B200 | 8 | 1024 | 1024 | 64 | 2026-06-24 | 8 | 1K/1K | TensorRT LLM | FP4 |  |  | 1193925.0238225148 | 2041.6117907365 | Qwen 3.5 | mtp |  | 1782284400 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 162.2807013289 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B300 | 2 | 1024 | 1024 | 256 | 2026-06-04 | 2 | 1K/1K | vLLM | FP4 |  |  | 5625382.4556179475 | 10688.2266656741 | MiniMax-M2.5 | off |  | 1780556400 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 42.8860162199 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B300 | 2 | 1024 | 1024 | 512 | 2026-06-04 | 2 | 1K/1K | vLLM | FP4 |  |  | 8464149.421807263 | 16081.8839014338 | MiniMax-M2.5 | off |  | 1780556400 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 32.145951166 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 1 | H200 | 4 | 1024 | 1024 | 64 | 2026-06-05 | 4 | 1K/1K | vLLM | FP8 |  |  | 1043891.4142767155 | 1430.1312375591 | MiniMax-M2.5 | off | single_turn | 1780642800 |  | agg |  | 715.2267858602 | 714.9044516989 | 0.0188203100115 | 0.0216253162066 | 20.2547310306691 | 0.021893404842799998 | 0.10790327796709999 | 20.131372951236 | 0.021610286442100002 | 0.2184778272596 | 45.6758556826 | 46.2742593755 |  |  |  |  |  |  |  |  |  | 0.0196715565398 | 22.3922826619819 | 0.0224261134717 | 0.2451570102014 | 44.5908739945 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 1 | H200 | 4 | 1024 | 1024 | 256 | 2026-06-05 | 4 | 1K/1K | vLLM | FP8 |  |  | 2258308.934111168 | 3093.8832397323 | MiniMax-M2.5 | off | single_turn | 1780642800 |  | agg |  | 1546.1937014067 | 1547.6895383255 | 0.032933258451500004 | 0.0399897241566 | 37.4095293073915 | 0.0405754109644 | 0.18450535694140002 | 37.2799390947199 | 0.039947576210199995 | 0.45330854108979995 | 24.6454681846 | 25.0328078664 |  |  |  |  |  |  |  |  |  | 0.0635900789872 | 41.7439469584264 | 0.0416897596001 | 0.30147306909780003 | 23.9867058384 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 16 | GB300 NVL72 | 16 | 1024 | 1024 | 8192 | 2026-06-22 | 24 | 1K/1K | Dynamo TensorRT LLM | FP4 | 4 | 4 | 7598495.802103726 | 16108.8111004599 | Kimi K2.5/2.6 | off | single_turn | 1782111600 | 16 | disagg | 8 | 24198.1134454066 | 12064.1599279866 | 3.8640122381039 | 3.6049946607212 | 36.9142881820444 | 0.0390980583551 | 0.8533751624636 | 37.8563565274574 | 0.0380621323979 | 2.8190385934019 | 25.5767176701 | 26.2728317359 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 16 | GB300 NVL72 | 16 | 1024 | 1024 | 4301 | 2026-06-22 | 24 | 1K/1K | Dynamo TensorRT LLM | FP4 | 4 | 4 | 6049376.303948916 | 12824.6777643717 | Kimi K2.5/2.6 | off | single_turn | 1782111600 | 16 | disagg | 8 | 19270.3332550883 | 9601.8500190135 | 2.4604460659903 | 2.3311795265023 | 24.1655545254471 | 0.0248542673525 | 1.1069707325659999 | 24.7341130698078 | 0.0246105488259 | 2.0830807888024 | 40.2345394381 | 40.6329825097 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 8 | GB300 NVL72 | 8 | 1024 | 1024 | 8 | 2026-06-22 | 36 | 1K/1K | Dynamo TensorRT LLM | FP4 | 4 | 4 | 19510.199518867925 | 41.36162298 | Kimi K2.5/2.6 | off | single_turn | 1782111600 | 32 | disagg | 4 | 186.0431289575 | 23.2764347328 | 0.5228369104443 | 0.49369887496630005 | 9.768830527551499 | 0.0052139920516 | 4.9459781049517995 | 9.4899390953127 | 0.0052288339061 | 4.651690278641801 | 191.7916234047 | 191.2472298729 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 32 | GB300 NVL72 | 32 | 1024 | 1024 | 1229 | 2026-06-22 | 36 | 1K/1K | Dynamo TensorRT LLM | FP4 | 4 | 4 | 1904191.9508375472 | 4036.8869357756 | Kimi K2.5/2.6 | off | single_turn | 1782111600 | 32 | disagg | 4 | 18187.702219706 | 2268.0350252843 | 1.3016033419408 | 1.2435613367717 | 14.462999968091001 | 0.0131334299105 | 2.0422105049947 | 14.858669718022 | 0.0131275487183 | 2.7679955028398 | 76.1415720658 | 76.1756837821 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B200 | 4 | 8192 | 1024 | 1 | 2026-02-05 | 4 | 8K/1K | TensorRT LLM | FP4 |  |  | 952613.0754385965 | 1628.968359 | gpt-oss-120b | eagle |  | 1770278400 |  | agg |  | 1447.297 | 181.6713 |  |  | 1.3571769999999999 | 0.001261 | 0.067069 |  |  |  | 792.9572 |  |  |  | 2 |  | 12.80197 | 74113 | 9303 | 0.78113 | FP8 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B200 | 2 | 8192 | 1024 | 1 | 2026-05-06 | 2 | 8K/1K | vLLM | NVFP4 |  |  | 193282.19080923978 | 330.5125462838 | Nemotron 3 Super 120B | off |  | 1778050800 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 204.8590147099 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B200 | 1 | 8192 | 1024 | 16 | 2026-05-06 | 1 | 8K/1K | vLLM | NVFP4 |  |  | 3568184.4761803513 | 6101.5954542684 | Nemotron 3 Super 120B | off |  | 1778050800 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 79.9594965759 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 1 | B300 | 4 | 1024 | 1024 | 32 | 2026-06-12 | 4 | 1K/1K | TensorRT LLM | FP4 |  |  | 440378.9315434737 | 836.7199699326 | DeepSeek-V4-Pro | mtp | single_turn | 1781247600 |  | agg |  | 417.7080579584 | 419.0119119742 | 4.1819910250488 | 3.7344296226858003 | 17.4931713009719 | 0.0182016729535 | 0.1842511214782 | 17.1439950822452 | 0.0178406720121 | 0.6891647761578 | 54.9400048313 | 56.0517002568 |  |  |  |  |  |  |  |  |  | 4.526558781974001 | 20.772122321336102 | 0.021404050334299998 | 1.8216523942305 | 46.7201293392 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 32 | GB300 NVL72 | 32 | 1024 | 1024 | 333 | 2026-02-03 | 34 | 1K/1K | Dynamo TensorRT LLM | FP4 | 2 | 2 | 1067671.0017690095 | 2263.4625237503 | DeepSeek-R1 | mtp | single_turn | 1770105600 | 32 | disagg | 2 | 19225.5289193477 | 1203.3333740254 | 1.4978534699766999 | 1.3238403965757 | 7.5344988584984005 | 0.0063553865245 | 1.6955248714949 | 7.4894653753443 | 0.0060738859309000005 | 1.894749008377 | 157.3468421059 | 164.6392460079 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 32 | GB300 NVL72 | 32 | 1024 | 1024 | 4301 | 2026-06-22 | 40 | 1K/1K | Dynamo TensorRT LLM | FP4 | 4 | 4 | 4629883.8085933495 | 9815.3536742179 | Kimi K2.5/2.6 | off | single_turn | 1782111600 | 32 | disagg | 8 | 24580.8822378627 | 6123.9715333067 | 1.9056557889562 | 1.7998397884131 | 18.7081139091169 | 0.0192203932701 | 0.9312563450076 | 19.3683190648679 | 0.0190014471277 | 1.8799108034377 | 52.0280717438 | 52.6275705886 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 8 | GB300 NVL72 | 8 | 1024 | 1024 | 12 | 2026-06-22 | 36 | 1K/1K | Dynamo TensorRT LLM | FP4 | 4 | 4 | 36030.7725154717 | 76.3852377328 | Kimi K2.5/2.6 | off | single_turn | 1782111600 | 32 | disagg | 4 | 345.849717964 | 42.7021777039 | 0.5722004710115 | 0.5393864051279 | 7.3411171695043 | 0.0057275536417 | 2.2145244915009 | 7.7798368849673 | 0.005727748941199999 | 2.5343042540329996 | 174.5946109911 | 174.588657825 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 8 | GB300 NVL72 | 8 | 8192 | 1024 | 256 | 2026-06-03 | 12 | 8K/1K | Dynamo SGLang | FP4 | 4 | 4 | 2476243.5455916035 | 5249.6363166542 | DeepSeek-V4-Pro | mtp | single_turn | 1780470000 | 8 | disagg | 4 | 13998.6749993308 | 875.1169753159 | 1.0574833799619001 | 1.945211358993 | 31.7860468844883 | 0.0095659774569 | 21.9614251834573 | 31.566958681454402 | 0.0108575735101 | 21.570396048759 | 104.5371478766 | 92.1016099104 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 8 | GB300 NVL72 | 8 | 8192 | 1024 | 1024 | 2026-06-03 | 24 | 8K/1K | Dynamo SGLang | FP4 | 4 | 4 | 4597214.467291038 | 9746.094670657 | DeepSeek-V4-Pro | mtp | single_turn | 1780470000 | 8 | disagg | 16 | 12994.6555126256 | 3248.9729867197 | 1.6842793959659 | 3.0710790361309 | 32.0160788504872 | 0.0146483649506 | 16.9976140515646 | 32.6632504613388 | 0.016642974846800003 | 17.3447443291106 | 68.2670047731 | 60.0854119653 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 8 | GB300 NVL72 | 8 | 8192 | 1024 | 8192 | 2026-06-03 | 40 | 8K/1K | Dynamo SGLang | FP4 | 4 | 4 | 5770501.649172642 | 12233.463496246 | DeepSeek-V4-Pro | mtp | single_turn | 1780470000 | 8 | disagg | 32 | 13593.0258189676 | 6795.2142053593 | 2.2072680727579 | 4.3511903014226 | 132.3086133077741 | 0.037082161998 | 97.3444898827001 | 131.33281942698937 | 0.0374260289568 | 96.8810617861887 | 26.9671439344 | 26.7193722624 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 8 | H200 | 8 | 8192 | 1024 | 512 | 2026-03-23 | 32 | 8K/1K | Dynamo TensorRT LLM | FP8 | 8 | 8 | 1697750.0129848174 | 2325.9175177892 | DeepSeek-R1 | mtp | single_turn | 1774249200 | 8 | disagg | 24 | 2756.6677014945 | 1033.6669666731 | 7.2004517865134 | 6.5006228946545 | 54.2104033875512 | 0.0400201778799 | 17.608300240593998 | 54.897621553019 | 0.0391853842074 | 18.8090075082845 | 24.9873951836 | 25.5197191562 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B200 | 2 | 8192 | 1024 | 128 | 2026-05-06 | 2 | 8K/1K | vLLM | NVFP4 |  |  | 8538115.107889708 | 14600.1768344914 | Nemotron 3 Super 120B | off |  | 1778050800 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 35.5503075479 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 32 | GB300 NVL72 | 32 | 1024 | 1024 | 1229 | 2026-02-03 | 38 | 1K/1K | Dynamo TensorRT LLM | FP4 | 2 | 2 | 2816161.067549057 | 5970.261463204 | DeepSeek-R1 | mtp | single_turn | 1770105600 | 32 | disagg | 6 | 18902.8626833591 | 3545.3987344249 | 2.0182318245352002 | 1.784913617429 | 9.431017198512599 | 0.0085572260866 | 1.557189260493 | 9.413059297616101 | 0.0082014356228 | 1.8647192877487 | 116.8602991057 | 121.9298725235 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 16 | GB300 NVL72 | 16 | 1024 | 1024 | 2253 | 2026-02-03 | 22 | 1K/1K | Dynamo TensorRT LLM | FP4 | 2 | 2 | 5659108.919853963 | 11997.3109100904 | DeepSeek-R1 | mtp | single_turn | 1770105600 | 16 | disagg | 6 | 21986.6229644838 | 8251.3188896928 | 2.5215269179898 | 2.2966773211464 | 14.483616410027 | 0.0142063192965 | 1.4014003720076 | 14.939291313216799 | 0.0140232911596 | 2.0196914795335 | 70.3912096533 | 71.3099363492 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 1 | GB300 NVL72 | 4 | 8192 | 1024 | 32 | 2026-06-03 | 28 | 8K/1K | Dynamo SGLang | FP4 | 4 | 4 | 450117.60084405664 | 954.2493137894 | DeepSeek-V4-Pro | mtp | single_turn | 1780470000 | 24 | disagg | 4 | 5932.8540847809 | 124.4818519575 | 0.020272064954 | 0.0201495911956 | 8.8828470648732 | 0.0059083762119 | 2.6911695743911 | 9.3414055819609 | 0.0065568274414 | 3.2744267459435 | 169.2512399587 | 152.5127828874 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 8 | B300 | 8 | 8192 | 1024 | 3072 | 2026-07-30 | 40 | 8K/1K | Dynamo SGLang | FP4 | 4 | 4 | 5108581.962134105 | 9706.3057280548 | DeepSeek-V4-Pro | mtp | single_turn | 1785394800 | 8 | disagg | 32 | 10784.9105271054 | 5391.8865318525 | 1.8122211690060999 | 2.9704723552864003 | 60.9360859270091 | 0.028795779821099998 | 34.4934052500175 | 60.9791404501453 | 0.0286640683375 | 34.5929915712194 | 34.7273109536 | 34.886883056 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 8 | B300 | 8 | 8192 | 1024 | 2198 | 2026-01-28 | 28 | 8K/1K | Dynamo TensorRT LLM | FP4 | 2 | 2 | 8351617.862749789 | 15868.0739392246 | DeepSeek-R1 | mtp | single_turn | 1769587200 | 8 | disagg | 20 | 19744.7876281265 | 6176.28971697 | 7.1040376119781 | 6.2812625352905 | 38.5232207069639 | 0.038480326340399995 | 2.7733500495087 | 39.720939040408304 | 0.037474421129200006 | 5.1931420447888 | 25.9873055949 | 26.6848685014 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 8 | B300 | 8 | 8192 | 1024 | 512 | 2026-07-30 | 16 | 8K/1K | Dynamo SGLang | FP4 | 4 | 4 | 3818451.4096879475 | 7255.0576784071 | DeepSeek-V4-Pro | mtp | single_turn | 1785394800 | 8 | disagg | 8 | 12897.9959660533 | 1612.119390761 | 1.5717418513959 | 2.2194849229306 | 33.9472275020089 | 0.0155616967095 | 18.8754059229977 | 33.240407566606805 | 0.0161409227835 | 18.3819964881989 | 64.2603450426 | 61.9543264913 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 8 | B300 | 8 | 8192 | 1024 | 256 | 2026-07-30 | 20 | 8K/1K | Dynamo SGLang | FP4 | 4 | 4 | 1620396.8729692104 | 3078.7540586415 | DeepSeek-V4-Pro | mtp | single_turn | 1785394800 | 16 | disagg | 4 | 13683.0042026165 | 427.6915226477 | 1.1332159534977 | 1.6030485484349002 | 31.725231921998798 | 0.0110943205921 | 20.8943321859988 | 31.6998155883395 | 0.0115753844917 | 21.044552342593 | 90.1362090359 | 86.3902188923 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 8 | B300 | 8 | 8192 | 1024 | 2048 | 2026-07-30 | 32 | 8K/1K | Dynamo SGLang | FP4 | 4 | 4 | 4710759.624564263 | 8950.4432866721 | DeepSeek-V4-Pro | mtp | single_turn | 1785394800 | 8 | disagg | 24 | 10606.8549536394 | 3981.2082857699 | 2.1522925019963 | 2.8342850979808 | 54.2546787114698 | 0.0214378671703 | 33.6257317820564 | 54.306293817405994 | 0.0220398703376 | 34.0011599308729 | 46.6464313849 | 45.3723177443 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 1 | B300 | 4 | 8192 | 1024 | 32 | 2026-07-30 | 28 | 8K/1K | Dynamo SGLang | FP4 | 4 | 4 | 561215.5857516316 | 1066.3096129281 | DeepSeek-V4-Pro | mtp | single_turn | 1785394800 | 24 | disagg | 4 | 6629.566562201 | 139.1001213826 | 0.0173780650366 | 0.017520339783099997 | 8.0680750375031 | 0.006482640527 | 1.9214555880170998 | 8.3803951027243 | 0.0067493535906 | 2.1417854457544 | 154.2581292052 | 148.1623368198 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 8 | 1024 | 1024 | 64 | 2026-03-27 | 8 | 1K/1K | vLLM | FP4 |  |  | 1710434.3416058396 | 2343.295048 | gpt-oss-120b | off |  | 1774594800 |  | agg |  | 1171.647524 | 1171.647524 |  |  | 6.970084015 | 0.006520527999999999 | 0.29958349399999995 |  |  |  | 153.361805 |  | 1 | 1 | 0 | random | 34.95931939 | 327680 | 327680 | 9.15349628 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 8 | B300 | 8 | 8192 | 1024 | 8 | 2026-01-28 | 34 | 8K/1K | Dynamo TensorRT LLM | FP4 | 2 | 2 | 180078.20733578946 | 342.148593938 | DeepSeek-R1 | mtp | single_turn | 1769587200 | 32 | disagg | 2 | 5162.7298436567 | 40.8622658305 | 0.7757313795155 | 0.6886392038410001 | 5.6710503725335 | 0.003138802231 | 2.9765972014283997 | 5.4718045315385 | 0.0029003202073 | 2.7861119946494 | 318.5928664545 | 344.789515819 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 8 | B300 | 8 | 1024 | 1024 | 60 | 2026-01-28 | 42 | 1K/1K | Dynamo TensorRT LLM | FP4 | 2 | 2 | 252643.33433578946 | 480.022335238 | DeepSeek-R1 | mtp | single_turn | 1769587200 | 40 | disagg | 2 | 5032.2650897642 | 252.4101975117 | 1.3597503451165 | 1.2089306918466 | 5.4809079284314 | 0.0057695114784 | 0.1678955900716 | 5.2888067352357 | 0.0054779270897999995 | 0.2374915942849 | 173.3248999915 | 182.5508050047 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 8 | B300 | 8 | 8192 | 1024 | 32 | 2026-01-28 | 34 | 8K/1K | Dynamo TensorRT LLM | FP4 | 2 | 2 | 524194.49295321055 | 995.9695366111 | DeepSeek-R1 | mtp | single_turn | 1769587200 | 32 | disagg | 2 | 15038.3001021433 | 118.3238762653 | 1.0309762844116999 | 0.8967610940178999 | 7.4758541445481 | 0.0042205007609 | 3.8992205590475 | 7.3991770729554 | 0.0038995233628 | 3.7953188362582 | 236.9387086155 | 256.4415973326 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 8 | B300 | 8 | 8192 | 1024 | 256 | 2026-07-30 | 12 | 8K/1K | Dynamo SGLang | FP4 | 4 | 4 | 2667436.0012146314 | 5068.1284023078 | DeepSeek-V4-Pro | mtp | single_turn | 1785394800 | 8 | disagg | 4 | 13514.6661748183 | 844.8595160525 | 1.2634256619494 | 1.7943623617921 | 31.2217776040779 | 0.012631354326900001 | 18.84825535852 | 31.9554837974658 | 0.0131355933148 | 19.858257328400303 | 79.1680744693 | 76.1290317107 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 8 | H200 | 8 | 1024 | 1024 | 1024 | 2026-03-23 | 24 | 1K/1K | Dynamo TensorRT LLM | FP8 | 8 | 8 | 1690299.8447402923 | 2315.7107872942 | DeepSeek-R1 | mtp | single_turn | 1774249200 | 16 | disagg | 8 | 3473.4638911102 | 1736.8342353862 | 5.612508881022199 | 5.2264811988639 | 31.6262722790707 | 0.0319316036048 | 1.831923855003 | 32.5972325270066 | 0.0319947703509 | 3.139069723424 | 31.3169364238 | 31.255107914 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 4 | 1024 | 1024 | 2 | 2026-03-27 | 4 | 1K/1K | vLLM | FP4 |  |  | 201444.7283941606 | 275.9792779 | gpt-oss-120b | off |  | 1774594800 |  | agg |  | 137.9896389 | 137.9896389 |  |  | 3.709422564 | 0.003589207 | 0.037663763 |  |  |  | 278.6130722 |  | 1 | 1 | 0 | random | 18.55211753 | 10240 | 10240 | 0.539022027 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B200 | 1 | 1024 | 1024 | 256 | 2026-02-05 | 1 | 1K/1K | TensorRT LLM | FP4 |  |  | 22605032.59649123 | 38654.60574 | gpt-oss-120b | eagle |  | 1770278400 |  | agg |  | 19375.64 | 19278.96 |  |  | 12.93751 | 0.012546 | 0.10297400000000001 |  |  |  | 79.70679 |  |  |  | 2 |  | 122.514 | 2373787 | 2361942 | 20.89558 | FP8 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B200 | 4 | 1024 | 1024 | 1 | 2026-02-05 | 4 | 1K/1K | TensorRT LLM | FP4 |  |  | 254723.40192982458 | 435.5770173 | gpt-oss-120b | eagle |  | 1770278400 |  | agg |  | 217.3501 | 218.227 |  |  | 1.123263 | 0.00106 | 0.038532 |  |  |  | 943.0912 |  |  |  | 3 |  | 10.54865 | 9171 | 9208 | 0.947989 | FP8 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B300 | 2 | 1024 | 1024 | 1024 | 2026-06-04 | 2 | 1K/1K | vLLM | FP4 |  |  | 11690090.497344999 | 22211.1719449555 | MiniMax-M2.5 | off |  | 1780556400 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 22.1701251209 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B200 | 1 | 8192 | 1024 | 256 | 2026-02-05 | 1 | 8K/1K | TensorRT LLM | FP4 |  |  | 37713538.169590645 | 64490.15027 | gpt-oss-120b | eagle |  | 1770278400 |  | agg |  | 57325.18 | 7164.975 |  |  | 35.28142 | 0.034229 | 0.26506 |  |  |  | 29.21491 |  |  |  | 1 |  | 329.4641 | 18886588 | 2360602 | 7.770194 | FP8 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B200 | 4 | 1024 | 1024 | 32 | 2026-05-06 | 4 | 1K/1K | vLLM | NVFP4 |  |  | 393026.63765713456 | 672.0755503937 | Nemotron 3 Super 120B | off |  | 1778050800 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 130.263866957 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B200 | 8 | 1024 | 1024 | 64 | 2026-05-06 | 8 | 1K/1K | vLLM | NVFP4 |  |  | 393512.9575219883 | 672.9071573626 | Nemotron 3 Super 120B | off |  | 1778050800 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 130.1280332183 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H100 | 4 | 1024 | 1024 | 256 | 2026-05-06 | 4 | 1K/1K | vLLM | FP8 |  |  | 2004879.3551458397 | 2746.6847165498 | Nemotron 3 Super 120B | off |  | 1778050800 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 24.7752260273 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 1 | 8192 | 1024 | 2 | 2026-05-06 | 1 | 8K/1K | Dynamo vLLM | FP4 |  |  | 1497867.5512747448 | 2052.0785452464 | Nemotron 3 Super 120B | off |  | 1778050800 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 117.249997913 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 8 | 8192 | 1024 | 1 | 2026-04-09 | 8 | 8K/1K | Dynamo vLLM | FP4 |  |  | 95645.73099642337 | 131.0346514651 | Kimi K2.5/2.6 | off |  | 1775718000 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 124.9045994283 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 4 | 8192 | 1024 | 8 | 2026-07-04 | 4 | 8K/1K | SGLang | FP4 |  |  | 1534964.1624744528 | 2102.90090259 | Qwen 3.5 | on |  | 1783148400 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 137.5382215279 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H100 | 4 | 1024 | 1024 | 1 | 2026-04-09 | 4 | 1K/1K | vLLM | NVFP4 |  |  | 73059.41003846716 | 100.0913917527 | Qwen 3.5 | on |  | 1775718000 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 300.9730757744 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B200 | 8 | 1024 | 1024 | 1024 | 2026-06-24 | 8 | 1K/1K | TensorRT LLM | FP4 |  |  | 5595799.83895269 | 9568.8177246091 | Qwen 3.5 | mtp |  | 1782284400 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 40.9644202386 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 1 | 8192 | 1024 | 16 | 2026-05-06 | 1 | 8K/1K | Dynamo vLLM | FP4 |  |  | 4035826.7061335766 | 5529.082587403 | Nemotron 3 Super 120B | off |  | 1778050800 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 43.5565945642 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 1 | 8192 | 1024 | 256 | 2026-05-06 | 1 | 8K/1K | Dynamo vLLM | FP4 |  |  | 7759056.902640074 | 10629.9079566169 | Nemotron 3 Super 120B | off |  | 1778050800 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 5.30107158 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 8 | 8192 | 1024 | 4 | 2026-04-09 | 8 | 8K/1K | Dynamo vLLM | FP4 |  |  | 283277.5699298541 | 388.0902708039 | Kimi K2.5/2.6 | off |  | 1775718000 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 90.5563295506 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 8 | 8192 | 1024 | 32 | 2026-04-09 | 8 | 8K/1K | Dynamo vLLM | FP4 |  |  | 720523.4570332847 | 987.1171361356 | Kimi K2.5/2.6 | off |  | 1775718000 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 30.4722272525 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 1 | 8192 | 1024 | 128 | 2026-05-06 | 1 | 8K/1K | Dynamo vLLM | FP4 |  |  | 7308953.617269636 | 10013.2664556594 | Nemotron 3 Super 120B | off |  | 1778050800 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 9.5780493514 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 1 | 8192 | 1024 | 4 | 2026-05-06 | 1 | 8K/1K | Dynamo vLLM | FP4 |  |  | 2190012.6172754746 | 3000.3172856674 | Nemotron 3 Super 120B | off |  | 1778050800 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 90.0229111009 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 8 | 8192 | 1024 | 16 | 2026-07-14 | 8 | 8K/1K | SGLang | FP4 |  |  | 420442.13480051095 | 576.0057246767 | DeepSeek-V4-Pro | off |  | 1784012400 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 34.2915503549 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 8 | 8192 | 1024 | 2 | 2026-07-14 | 8 | 8K/1K | SGLang | FP4 |  |  | 115098.83136445256 | 157.6853989693 | DeepSeek-V4-Pro | off |  | 1784012400 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 77.9984779377 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H100 | 4 | 1024 | 1024 | 4 | 2026-04-09 | 4 | 1K/1K | vLLM | NVFP4 |  |  | 283932.6677031387 | 388.9877547533 | Qwen 3.5 | on |  | 1775718000 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 211.701473006 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 4 | RTX PRO 6000 | 4 | 8192 | 1024 | 16 | 2026-07-30 | 16 | 8K/1K | SGLang | FP4 |  |  | 1239114.6898542566 | 1208.1368226079 | Qwen 3.5 | mtp | single_turn | 1785394800 |  | agg |  | 1074.3052085674 | 133.8316140405 | 0.8999792760005 | 1.4805348748227 | 26.6324077765166 | 0.026606008396299997 | 2.5393989189761 | 26.7934996945991 | 0.025802194617800003 | 3.1697152252371 | 37.5854951673 | 38.7563931988 |  |  |  |  |  |  |  |  |  | 3.4120660832035 | 31.631239744304903 | 0.029170481281699998 | 4.7399697384972 | 34.2812307532 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 1 | H200 | 4 | 8192 | 1024 | 128 | 2026-06-05 | 4 | 8K/1K | vLLM | FP8 |  |  | 3691032.445351533 | 5056.7144501316 | MiniMax-M2.5 | off | single_turn | 1780642800 |  | agg |  | 4496.6049454932 | 560.1095046384 | 0.034847789444 | 0.0546538307951 | 51.6661471868865 | 0.055758145439400005 | 0.4683341686614 | 51.7881275206739 | 0.054571633041399996 | 1.5554502009385 | 17.9345993687 | 18.3245386709 |  |  |  |  |  |  |  |  |  | 0.1736481864937 | 59.845184174086896 | 0.0587158594756 | 1.5199102781713 | 17.0311736715 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B300 | 2 | 1024 | 1024 | 2048 | 2026-06-04 | 2 | 1K/1K | vLLM | FP4 |  |  | 15055701.740882896 | 28605.8333076775 | MiniMax-M2.5 | off |  | 1780556400 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 14.7000137363 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 8 | 8192 | 1024 | 8 | 2026-04-09 | 8 | 8K/1K | Dynamo vLLM | FP4 |  |  | 408822.8859940146 | 560.0873538118 | Kimi K2.5/2.6 | off |  | 1775718000 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 69.4047299976 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 4 | 8192 | 1024 | 512 | 2026-07-04 | 4 | 8K/1K | SGLang | FP4 |  |  | 3157859.02279 | 4326.2668612223 | Qwen 3.5 | on |  | 1783148400 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 16.0071821281 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | GB300 NVL72 |  | 131072 | 8192 | 128 | 2025-10-31 |  | 128K/8K | TensorRT LLM | FP4 |  |  | 67099.05660377358 | 142.25 | DeepSeek-R1 | mtp |  | 1761894000 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 137.1742 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | GB300 NVL72 |  | 131072 | 8192 | 1024 | 2025-10-31 |  | 128K/8K | TensorRT LLM | FP4 |  |  | 124877.35849056605 | 264.74 | DeepSeek-R1 | mtp |  | 1761894000 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 26.4971 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | GB300 NVL72 |  | 131072 | 8192 | 512 | 2025-10-31 |  | 128K/8K | TensorRT LLM | FP4 |  |  | 113825.47169811321 | 241.31 | DeepSeek-R1 | mtp |  | 1761894000 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 44.0723 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | GB300 NVL72 |  | 131072 | 8192 | 128 | 2025-10-31 |  | 128K/8K | TensorRT LLM | FP4 |  |  | 69405.6603773585 | 147.14 | DeepSeek-R1 | mtp |  | 1761894000 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 132.626 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 1 | B300 | 4 | 1024 | 1024 | 8 | 2026-07-06 | 4 | 1K/1K | vLLM | FP4 |  |  | 434948.2664636316 | 826.4017062809 | MiniMax-M3 | mtp | single_turn | 1783321200 |  | agg |  | 417.993657404 | 408.4080488768 | 0.08878568396900001 | 0.0940803728348 | 4.0113132025580995 | 0.0043021694368999995 | 0.057397139025900004 | 4.3038492639433 | 0.004563334859 | 0.1149207528622 | 232.4408684174 | 219.1379837105 |  |  |  |  |  |  |  |  |  | 0.13666583118029998 | 5.602825083374 | 0.005904196991 | 0.1268760807579 | 169.3710425873 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 1 | H200 | 4 | 1024 | 1024 | 4 | 2026-06-05 | 4 | 1K/1K | vLLM | FP8 |  |  | 162641.82249934308 | 222.8192968241 | MiniMax-M2.5 | off | single_turn | 1780642800 |  | agg |  | 111.9517694495 | 110.8675273746 | 0.008314144797599999 | 0.008590959427799999 | 8.0527864224277 | 0.0085476263449 | 0.1055877376348 | 8.0036116940202 | 0.008588038570200001 | 0.1280645821709 | 116.9915435754 | 116.4410233864 |  |  |  |  |  |  |  |  |  | 0.008389848284399998 | 8.816021524742201 | 0.008634618206500001 | 0.1430304615758 | 115.8128797452 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 1 | H200 | 4 | 8192 | 1024 | 64 | 2026-06-05 | 4 | 8K/1K | vLLM | FP8 |  |  | 2836607.196792774 | 3886.1518596061 | MiniMax-M2.5 | off | single_turn | 1780642800 |  | agg |  | 3455.0172266045 | 431.1346330016 | 0.0256819576025 | 0.035495603323799994 | 33.8644896750338 | 0.0362266082779 | 0.3112456756644 | 33.6329334210182 | 0.0354544591299 | 0.9499685740257999 | 27.6040194635 | 28.2051968791 |  |  |  |  |  |  |  |  |  | 0.0267778942361 | 38.22387230657041 | 0.0376516179793 | 1.1354788580909 | 26.5592836024 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 32 | GB300 NVL72 | 32 | 8192 | 1024 | 308 | 2026-02-03 | 48 | 8K/1K | Dynamo TensorRT LLM | FP4 | 2 | 2 | 3291308.7043641983 | 6977.5744532521 | DeepSeek-R1 | mtp | single_turn | 1770105600 | 32 | disagg | 16 | 18606.0611667026 | 1163.3310965269 | 1.6312592354952 | 1.4225727403372 | 7.300424157991101 | 0.0066564639831 | 1.1916593965142999 | 7.1409414125765 | 0.0061745979212 | 1.4561931011256999 | 150.2299122392 | 161.9538652988 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 16 | GB300 NVL72 | 16 | 8192 | 1024 | 1127 | 2026-02-03 | 42 | 8K/1K | Dynamo TensorRT LLM | FP4 | 2 | 2 | 6863618.555416368 | 14550.8713374827 | DeepSeek-R1 | mtp | single_turn | 1770105600 | 16 | disagg | 26 | 20893.1541032793 | 4244.6618430632 | 3.6229923940264 | 3.1212061315337 | 14.8042022324807 | 0.0145575827842 | 1.3658870729560002 | 14.5532579171772 | 0.013527406277300001 | 2.101390356178 | 68.69272288 | 73.9240013572 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 8 | B300 | 8 | 8192 | 1024 | 1024 | 2026-07-30 | 24 | 8K/1K | Dynamo SGLang | FP4 | 4 | 4 | 4202595.190386211 | 7984.9308617338 | DeepSeek-V4-Pro | mtp | single_turn | 1785394800 | 8 | disagg | 16 | 10646.4619262075 | 2661.8687327864 | 1.8693808289972 | 2.6216734800207 | 41.510706752000296 | 0.0182257741754 | 23.981834394500797 | 40.9722175821437 | 0.0189413495581 | 23.5442335834333 | 54.8673538021 | 52.7945486109 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 1 | B300 | 4 | 8192 | 1024 | 64 | 2026-07-30 | 28 | 8K/1K | Dynamo SGLang | FP4 | 4 | 4 | 841501.0886263158 | 1598.85206839 | DeepSeek-V4-Pro | mtp | single_turn | 1785394800 | 24 | disagg | 4 | 9950.4631206267 | 206.9168930172 | 0.0213214129908 | 0.0217153709736 | 10.8408225479652 | 0.0080565869994 | 3.0273493489367 | 11.3294349768914 | 0.008402739584600001 | 3.5943520511837 | 124.1220382869 | 119.0088053942 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 1 | B300 | 4 | 8192 | 1024 | 1 | 2026-07-30 | 8 | 8K/1K | Dynamo SGLang | FP4 | 1 | 4 | 107635.60939178948 | 204.5076578444 | DeepSeek-V4-Pro | mtp | single_turn | 1785394800 | 4 | disagg | 4 | 363.3285318134 | 45.6867838755 | 0.012931571051 | 0.012939900913900002 | 5.1726104334812 | 0.0052060036637 | 0.32396186247930003 | 5.0897878156975 | 0.0051148094463 | 0.3537844012026 | 192.0859193736 | 195.5107048462 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 8 | B300 | 8 | 1024 | 1024 | 200 | 2026-01-28 | 42 | 1K/1K | Dynamo TensorRT LLM | FP4 | 2 | 2 | 501823.7716383158 | 953.4651661128 | DeepSeek-R1 | mtp | single_turn | 1769587200 | 40 | disagg | 2 | 10018.4850214547 | 500.2141733457 | 1.8583798034815 | 1.6333709418746 | 8.869902378530199 | 0.007854211999 | 1.7260119415587 | 8.7824914097455 | 0.007515602613 | 1.8782327033884 | 127.3202200464 | 133.0565293958 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 1 | RTX PRO 6000 | 4 | 8192 | 1024 | 4 | 2026-07-30 | 4 | 8K/1K | SGLang | FP4 |  |  | 748306.920507795 | 729.5992474951 | Qwen 3.5 | mtp | single_turn | 1785394800 |  | agg |  | 648.459423161 | 81.139824334 | 0.38404159800849996 | 0.6108011500878 | 10.824106211017401 | 0.0109199703179 | 0.9744464899995 | 11.136933345873 | 0.010830155758 | 1.1961447821202 | 91.5753404896 | 92.3347754498 |  |  |  |  |  |  |  |  |  | 1.4475693690473999 | 12.6907107649255 | 0.012251778855099999 | 1.8001281432982 | 81.6208006879 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 1 | B300 | 4 | 8192 | 1024 | 2 | 2026-07-29 | 4 | 8K/1K | vLLM | FP4 |  |  | 740194.2536746842 | 1406.3690819819 | MiniMax-M3 | mtp | single_turn | 1785308400 |  | agg |  | 1248.8496655771 | 157.5194164047 | 0.052837271010499995 | 0.062427672528700004 | 2.4746563304216003 | 0.00239724057 | 0.1623500814894 | 2.9428587093018 | 0.0029571002064 | 0.16794876190830002 | 417.1462858305 | 338.1691286092 |  |  |  |  |  |  |  |  |  | 0.1085667668842 | 4.8967628324404 | 0.0049074222966000005 | 0.1765155840199 | 203.772966653 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 1 | H200 | 4 | 8192 | 1024 | 2 | 2026-06-05 | 4 | 8K/1K | vLLM | FP8 |  |  | 372167.1104559124 | 509.8689413246 | MiniMax-M2.5 | off | single_turn | 1780642800 |  | agg |  | 452.6540168397 | 57.214924485 | 0.0083000967279 | 0.0084523734047 | 8.21787916217 | 0.008453278903399999 | 0.2390692718327 | 8.1502754493617 | 0.008448617346100001 | 0.2553363617044 | 118.2972916694 | 118.3625626575 |  |  |  |  |  |  |  |  |  | 0.0083814170212 | 8.7849306316115 | 0.008519682939100002 | 0.33419705508279995 | 117.375259989 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 1 | H200 | 4 | 1024 | 1024 | 8 | 2026-06-05 | 4 | 1K/1K | vLLM | FP8 |  |  | 276106.6887718249 | 378.2661636174 | MiniMax-M2.5 | off | single_turn | 1780642800 |  | agg |  | 188.4319882325 | 189.8341753849 | 0.0097422655672 | 0.0101999634398 | 9.6386724147014 | 0.0102459912124 | 0.1050950782374 | 9.5899122890551 | 0.010196474018 | 0.1235814067884 | 97.5991467556 | 98.0731180445 |  |  |  |  |  |  |  |  |  | 0.010233982931799999 | 10.454299660679 | 0.010332896073200001 | 0.1618029829115 | 96.7782887701 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 1 | H200 | 4 | 1024 | 1024 | 128 | 2026-06-05 | 4 | 1K/1K | vLLM | FP8 |  |  | 1824337.8589446715 | 2499.3428667542 | MiniMax-M2.5 | off | single_turn | 1780642800 |  | agg |  | 1251.9927872426 | 1247.3500795116 | 0.0223638750613 | 0.023831605747000002 | 22.1437897494761 | 0.0241954967864 | 0.0992626720108 | 22.0791437161743 | 0.0238234929857 | 0.1969829663728 | 41.3300048695 | 41.9753728221 |  |  |  |  |  |  |  |  |  | 0.0235716090538 | 24.6073515419383 | 0.0250509520296 | 0.4236580020282 | 39.9186425657 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 8 | B300 | 8 | 1024 | 1024 | 10 | 2026-01-28 | 42 | 1K/1K | Dynamo TensorRT LLM | FP4 | 2 | 2 | 67461.27730263158 | 128.176426875 | DeepSeek-R1 | mtp | single_turn | 1769587200 | 40 | disagg | 2 | 1351.5168321981 | 67.0094066089 | 0.8281047309283 | 0.7350380142327 | 3.368082718458 | 0.0034741356025 | 0.11814102949569999 | 3.2511560030095 | 0.0033617640816 | 0.17134704606610002 | 287.8413839903 | 297.4628723871 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H100 | 4 | 1024 | 1024 | 32 | 2026-04-09 | 4 | 1K/1K | vLLM | NVFP4 |  |  | 1026654.0812368614 | 1406.5160912945 | Qwen 3.5 | on |  | 1775718000 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 90.7973850019 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | GB300 NVL72 |  | 131072 | 8192 | 256 | 2025-10-31 |  | 128K/8K | TensorRT LLM | FP4 |  |  | 99683.96226415095 | 211.33 | DeepSeek-R1 | mtp |  | 1761894000 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 70.8717 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | GB300 NVL72 |  | 131072 | 8192 | 512 | 2025-10-31 |  | 128K/8K | TensorRT LLM | FP4 |  |  | 95424.5283018868 | 202.3 | DeepSeek-R1 | mtp |  | 1761894000 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 85.6898 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 1 | B300 | 2 | 8192 | 1024 | 16 | 2026-07-29 | 2 | 8K/1K | vLLM | FP4 |  |  | 3560183.860142211 | 6764.3493342702 | MiniMax-M3 | mtp | single_turn | 1785308400 |  | agg |  | 6015.4525579199 | 748.8967763503 | 0.1622783179628 | 0.2049194713453 | 8.7741361965018 | 0.009193325569 | 0.2266005139973 | 9.521125786379299 | 0.0099602933619 | 0.4009286496927 | 108.7745661232 | 100.3986492832 |  |  |  |  |  |  |  |  |  | 0.35936415998730004 | 13.305372092902001 | 0.0140033295732 | 0.5437862758001001 | 71.4115878491 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 1 | H200 | 4 | 1024 | 1024 | 64 | 2026-06-13 | 4 | 1K/1K | vLLM | FP8 |  |  | 901710.0204862045 | 1235.3427280661 | MiniMax-M3 | mtp | single_turn | 1781334000 |  | agg |  | 624.7349132528 | 610.6078148133 | 0.4808022361249 | 0.5098206584038 | 22.2675229087472 | 0.0234480924394 | 0.28398462943729996 | 23.3309801179945 | 0.024801244655499998 | 0.5666913128633 | 42.6473924302 | 40.3205570483 |  |  |  |  |  |  |  |  |  | 0.7771133292465999 | 30.7647016445175 | 0.0327709800428 | 0.5465504579246 | 30.5148029963 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 1 | H200 | 4 | 1024 | 1024 | 1 | 2026-06-13 | 4 | 1K/1K | vLLM | FP8 |  |  | 89417.88118708029 | 122.5024972263 | MiniMax-M3 | mtp | single_turn | 1781334000 |  | agg |  | 61.068409065 | 61.4340881612 | 0.0807208530605 | 0.0803455905598 | 4.0695190858096 | 0.004074941704699999 | 0.1198650505394 | 3.7596739953384 | 0.0039413208264 | 0.1200189558789 | 245.4022836333 | 253.7220500522 |  |  |  |  |  |  |  |  |  | 0.11146183647220001 | 4.6432319236919 | 0.0048417562344 | 0.12538996990770002 | 206.5366267103 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 8 | B200 | 8 | 8192 | 1024 | 615 | 2026-07-22 | 16 | 8K/1K | Dynamo TensorRT LLM | FP4 | 4 | 4 | 6545431.261280468 | 11192.6874567896 | Kimi K2.5/2.6 | off | single_turn | 1784703600 | 8 | disagg | 8 | 19897.6019397654 | 2487.7729738139 | 2.4837234345014 | 2.3470633870085 | 27.5861520540002 | 0.024874296682899998 | 4.6577012415 | 27.874815680981598 | 0.0247795628747 | 5.0525821139407 | 40.2021417026 | 40.3558369878 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 8 | B200 | 8 | 8192 | 1024 | 4 | 2026-07-22 | 36 | 8K/1K | Dynamo TensorRT LLM | FP4 | 4 | 4 | 96991.532305731 | 165.8555202428 | Kimi K2.5/2.6 | off | single_turn | 1784703600 | 32 | disagg | 4 | 1324.996872809 | 20.962851172 | 0.5403338299948 | 0.5118868017539 | 5.5282595185054 | 0.005428261617000001 | 0.40693402900799996 | 5.468217146775 | 0.0054238046485 | 0.4517266986875 | 184.2210399129 | 184.3724220911 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 8 | 8192 | 1024 | 256 | 2026-07-14 | 8 | 8K/1K | SGLang | FP4 |  |  | 488482.31295620446 | 669.22076875 | DeepSeek-V4-Pro | off |  | 1784012400 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 27.9659859892 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B200 | 8 | 1024 | 1024 | 256 | 2026-06-24 | 8 | 1K/1K | TensorRT LLM | FP4 |  |  | 3199072.32383731 | 5470.4136737618 | Qwen 3.5 | mtp |  | 1782284400 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 100.2631443962 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H100 | 8 | 1024 | 1024 | 8 | 2026-05-06 | 8 | 1K/1K | vLLM | FP8 |  |  | 124196.79582372264 | 170.1496102785 | Nemotron 3 Super 120B | off |  | 1778050800 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 143.5998097025 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | GB300 NVL72 |  | 131072 | 8192 | 56 | 2025-10-31 |  | 128K/8K | TensorRT LLM | FP4 |  |  | 50683.96226415095 | 107.45 | DeepSeek-R1 | mtp |  | 1761894000 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 173.3102 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 4 | RTX PRO 6000 | 4 | 8192 | 1024 | 64 | 2026-07-30 | 16 | 8K/1K | SGLang | FP4 |  |  | 1773642.2274213333 | 1729.3011717358 | Qwen 3.5 | mtp | single_turn | 1785394800 |  | agg |  | 1537.5054484159 | 191.7957233199 | 2.1665497754584 | 4.078385031871 | 76.3762338670203 | 0.0769460248014 | 5.8141274949885 | 76.016389290229 | 0.07354872872679999 | 8.2642180621348 | 12.9961229652 | 13.5964280731 |  |  |  |  |  |  |  |  |  | 9.992794622929098 | 88.4930852685415 | 0.0850424379888 | 14.8179272119829 | 11.7588350434 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 1 | B300 | 4 | 8192 | 1024 | 8 | 2026-07-29 | 4 | 8K/1K | vLLM | FP4 |  |  | 1749201.7552742632 | 3323.4833350211 | MiniMax-M3 | mtp | single_turn | 1785308400 |  | agg |  | 2950.8582473639 | 372.6250876572 | 0.0825951349689 | 0.1019775946732 | 4.4182118964964 | 0.0045180771853 | 0.1631984959822 | 4.8114571412661 | 0.0049677294164 | 0.2160920196766 | 221.3330934784 | 201.2992085896 |  |  |  |  |  |  |  |  |  | 0.1900543764001 | 6.5917346208356005 | 0.007017312805699999 | 0.2747640188085 | 142.5046919947 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 8 | B200 | 8 | 8192 | 1024 | 24 | 2026-07-22 | 36 | 8K/1K | Dynamo TensorRT LLM | FP4 | 4 | 4 | 335031.6794557895 | 572.9041718694 | Kimi K2.5/2.6 | off | single_turn | 1784703600 | 32 | disagg | 4 | 4581.2755753804 | 71.8577464305 | 0.6656299320002 | 0.6339778713782 | 9.3489414944997 | 0.0067010362976 | 3.0959635114996 | 9.4639190604541 | 0.006693204760100001 | 3.2720686972707997 | 149.2306496474 | 149.4052603858 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 8 | B200 | 8 | 8192 | 1024 | 1127 | 2026-07-22 | 20 | 8K/1K | Dynamo TensorRT LLM | FP4 | 4 | 4 | 8402438.759990761 | 14368.1702795842 | Kimi K2.5/2.6 | off | single_turn | 1784703600 | 8 | disagg | 12 | 21284.6270049268 | 3993.4851915705 | 3.0853700519947003 | 2.9101942365412 | 31.4732554590009 | 0.0308935880857 | 2.9279764940002 | 31.856907381081097 | 0.0307242260378 | 3.5464244174697 | 32.3691763231 | 32.5476058785 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 8 | B200 | 8 | 8192 | 1024 | 2151 | 2026-07-22 | 24 | 8K/1K | Dynamo TensorRT LLM | FP4 | 4 | 4 | 9708670.694792924 | 16601.8268880959 | Kimi K2.5/2.6 | off | single_turn | 1784703600 | 8 | disagg | 16 | 22137.3971802755 | 5530.6863037368 | 4.466269871001701 | 4.1932871731965 | 43.3713285795002 | 0.0447187170177 | 2.1433708479999 | 44.0053452532375 | 0.044267797518 | 3.2614489999496 | 22.3620011192 | 22.5897843595 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 4 | B200 | 4 | 8192 | 1024 | 5 | 2026-07-22 | 24 | 8K/1K | Dynamo TensorRT LLM | FP4 | 4 | 4 | 173595.45872409357 | 296.8482344182 | Kimi K2.5/2.6 | off | single_turn | 1784703600 | 20 | disagg | 4 | 1582.9033500857 | 39.6372112847 | 0.5800941140005 | 0.5488846017574 | 5.7129023834999995 | 0.0058039669036 | 0.33613333399990003 | 5.711121141259899 | 0.0058307766171 | 0.37047417708990005 | 172.2959514774 | 171.5037405251 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H100 | 4 | 1024 | 1024 | 2 | 2026-04-09 | 4 | 1K/1K | vLLM | NVFP4 |  |  | 129082.45485839417 | 176.842963156 | Qwen 3.5 | on |  | 1775718000 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 261.1711228229 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H100 | 4 | 1024 | 1024 | 16 | 2026-04-09 | 4 | 1K/1K | vLLM | NVFP4 |  |  | 663067.9905259855 | 908.4031470206 | Qwen 3.5 | on |  | 1775718000 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 123.7343047309 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H100 | 4 | 1024 | 1024 | 8 | 2026-04-09 | 4 | 1K/1K | vLLM | NVFP4 |  |  | 456507.7837735766 | 625.4156637698 | Qwen 3.5 | on |  | 1775718000 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 163.3362192954 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 2 | B300 | 2 | 8192 | 1024 | 128 | 2026-08-10 | 4 | 8K/1K | SGLang | FP4 |  |  | 9673880.998173157 | 18380.373896529 | Qwen 3.5 | mtp | single_turn | 1786345200 |  | agg |  | 16345.1785629903 | 2035.1953335388 | 1.5145233978983 | 2.2670691192902 | 28.7253149839817 | 0.028999493489199998 | 2.1463043739786 | 28.5222007547879 | 0.027940673487200002 | 2.8281604756869 | 34.4833609033 | 35.7901179604 |  |  |  |  |  |  |  |  |  | 5.0444335522596 | 33.7471807079157 | 0.0322701482748 | 4.2644590886892 | 30.9883918563 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 1 | B300 | 4 | 1024 | 1024 | 16 | 2026-07-06 | 4 | 1K/1K | vLLM | FP4 |  |  | 711822.1617812632 | 1352.4621073844 | MiniMax-M3 | mtp | single_turn | 1783321200 |  | agg |  | 681.6841628475 | 670.777944537 | 0.1146327080205 | 0.1164744320691 | 5.1694164657965 | 0.005593879700099999 | 0.06486689136360001 | 5.3343404652289 | 0.0056555008303000006 | 0.13157326619259999 | 178.7668047257 | 176.8189997686 |  |  |  |  |  |  |  |  |  | 0.1674268919975 | 6.8018064265605 | 0.0072217399779 | 0.0964768314734 | 138.470784473 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 4 | H200 | 4 | 1024 | 1024 | 128 | 2026-06-13 | 16 | 1K/1K | vLLM | FP8 |  |  | 1041295.9266197812 | 1426.5754194691 | MiniMax-M3 | mtp | single_turn | 1781334000 |  | agg |  | 721.2882977528 | 705.2871217162 | 0.8377594547346 | 0.8842741311703001 | 38.7962189232931 | 0.0413710032003 | 0.4048118656501 | 40.533405161750906 | 0.043007722751 | 1.0326035507649 | 24.171519244 | 23.2516379858 |  |  |  |  |  |  |  |  |  | 1.3278530221432 | 54.567698247916994 | 0.0571102055949 | 0.7549702767283 | 17.5100052536 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 1 | H200 | 4 | 1024 | 1024 | 2 | 2026-06-05 | 4 | 1K/1K | vLLM | FP8 |  |  | 89237.7745970073 | 122.2557511979 | MiniMax-M2.5 | off | single_turn | 1780642800 |  | agg |  | 60.752148704 | 61.5036024939 | 0.007892882335 | 0.008070915198 | 7.668909486616 | 0.0079409422279 | 0.07288729201540001 | 7.5071199861413 | 0.0079222952845 | 0.13938341671020002 | 125.9296405007 | 126.2260448632 |  |  |  |  |  |  |  |  |  | 0.0079372975742 | 8.033171174 | 0.0079572950329 | 0.3552852850407 | 125.6708461686 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 1 | H200 | 4 | 8192 | 1024 | 4 | 2026-06-05 | 4 | 8K/1K | vLLM | FP8 |  |  | 654998.4584432117 | 897.3478880672 | MiniMax-M2.5 | off | single_turn | 1780642800 |  | agg |  | 797.5250381169 | 99.8228499503 | 0.0088844699785 | 0.0094369900675 | 8.959557075053501 | 0.009441470707 | 0.2348052817397 | 8.978728340612701 | 0.0094296535617 | 0.3077861146536 | 105.9157022278 | 106.0484347022 |  |  |  |  |  |  |  |  |  | 0.0091796418652 | 9.8436868653633 | 0.0095654887753 | 0.7861975970678 | 104.5424884699 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 1 | B300 | 8 | 8192 | 1024 | 1 | 2026-07-29 | 8 | 8K/1K | vLLM | FP4 |  |  | 248694.47342947367 | 472.519499516 | MiniMax-M3 | mtp | single_turn | 1785308400 |  | agg |  | 419.9539229728 | 52.5655765431 | 0.0419262647629 | 0.0463631660009 | 2.0984918521717 | 0.0021615466186 | 0.1430950150825 | 2.2119091743603 | 0.0022096291083000003 | 0.1441120729782 | 462.6317061136 | 452.5646391228 |  |  |  |  |  |  |  |  |  | 0.0713054453954 | 3.2481147008017 | 0.0030789573797 | 0.15343933245170002 | 324.785268736 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 4 | B200 | 4 | 8192 | 1024 | 60 | 2026-07-22 | 24 | 8K/1K | Dynamo TensorRT LLM | FP4 | 4 | 4 | 914253.2710942106 | 1563.3730935711 | Kimi K2.5/2.6 | off | single_turn | 1784703600 | 20 | disagg | 4 | 8342.0830303226 | 207.6311062208 | 0.9309318710002 | 0.8828757009571 | 12.93190624 | 0.009328516220599999 | 4.3341866069999995 | 12.930386317324999 | 0.0093162293834 | 4.3731137654042005 | 107.1981841864 | 107.3395639847 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 4 | B200 | 4 | 8192 | 1024 | 195 | 2026-07-22 | 24 | 8K/1K | Dynamo TensorRT LLM | FP4 | 4 | 4 | 2185277.9894821052 | 3736.8253620144 | Kimi K2.5/2.6 | off | single_turn | 1784703600 | 20 | disagg | 4 | 19928.6765847309 | 498.4551174711 | 1.5439094949998 | 1.4590739188125001 | 17.4471718980001 | 0.0154783234537 | 3.1697739604999002 | 17.6719075946895 | 0.0154048663193 | 3.4848457142215 | 64.6064803459 | 64.9145522766 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 4 | B200 | 4 | 8192 | 1024 | 115 | 2026-07-22 | 24 | 8K/1K | Dynamo TensorRT LLM | FP4 | 4 | 4 | 1453565.2197523394 | 2485.5965257765 | Kimi K2.5/2.6 | off | single_turn | 1784703600 | 20 | disagg | 4 | 13256.5232206084 | 331.4111868102 | 1.1678482140000999 | 1.1061295004695 | 15.5621356224999 | 0.0117059980486 | 4.7689566789999995 | 15.6413336568083 | 0.0116785282695 | 4.878693802668701 | 85.4262913634 | 85.6272277567 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 2 | B300 | 2 | 8192 | 1024 | 32 | 2026-08-10 | 4 | 8K/1K | SGLang | FP4 |  |  | 5983439.984087158 | 11368.5359697656 | Qwen 3.5 | mtp | single_turn | 1786345200 |  | agg |  | 10097.5936035669 | 1270.9423661987 | 0.6497619119472999 | 0.9068100987223 | 11.5016502371291 | 0.0113443581717 | 1.0055775489891001 | 11.3727106397862 | 0.0110204340402 | 1.1795985977121999 | 88.1495440171 | 90.740527674 |  |  |  |  |  |  |  |  |  | 1.9614596794824999 | 13.1733580695 | 0.012597778795 | 1.7950015180279 | 79.3790727934 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 1 | RTX PRO 6000 | 4 | 8192 | 1024 | 1 | 2026-07-30 | 4 | 8K/1K | SGLang | FP4 |  |  | 319614.98865302565 | 311.6246139367 | Qwen 3.5 | mtp | single_turn | 1785394800 |  | agg |  | 276.8209141221 | 34.8036998146 | 0.265498310473 | 0.3705883637281 | 6.6765127645049 | 0.0065118032475 | 0.8304922664829001 | 6.6821455325931005 | 0.0063000585276 | 0.8268495927798 | 153.5672934205 | 158.7286841256 |  |  |  |  |  |  |  |  |  | 0.8216839288827 | 7.4233098641736 | 0.0066886777493 | 0.8850559312734 | 149.5063804059 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 1 | B300 | 8 | 1024 | 1024 | 1 | 2026-07-06 | 8 | 1K/1K | vLLM | FP4 |  |  | 55563.449513052634 | 105.5705540748 | MiniMax-M3 | mtp | single_turn | 1783321200 |  | agg |  | 52.6277090462 | 52.9428450286 | 0.0464140969852 | 0.0479051239063 | 1.9648873400001001 | 0.002110337162 | 0.035179897502499995 | 2.1812883270904 | 0.0023286963585 | 0.0351388811076 | 473.8579303873 | 429.42481374 |  |  |  |  |  |  |  |  |  | 0.0739135743119 | 2.9709021829533997 | 0.0031074270213 | 0.0359827893262 | 321.8096493111 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 1 | B300 | 4 | 8192 | 1024 | 4 | 2026-07-29 | 4 | 8K/1K | vLLM | FP4 |  |  | 1236126.3190227367 | 2348.6400061432 | MiniMax-M3 | mtp | single_turn | 1785308400 |  | agg |  | 2087.9653674489 | 260.6746386944 | 0.061883019996500005 | 0.0732896424694 | 3.2377525820048 | 0.0032250357246 | 0.1657148399972 | 3.4756513625507 | 0.0035656045227 | 0.1922755676729 | 310.0740845657 | 280.4573512349 |  |  |  |  |  |  |  |  |  | 0.1134991882951 | 4.5271813940984 | 0.0045736610523 | 0.1963182362175 | 218.6432244485 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 1 | H200 | 8 | 1024 | 1024 | 4 | 2026-06-13 | 8 | 1K/1K | vLLM | FP8 |  |  | 143881.66482277375 | 197.1178808072 | MiniMax-M3 | mtp | single_turn | 1781334000 |  | agg |  | 99.6864223465 | 97.4314584607 | 0.0897556636482 | 0.0994740133336 | 4.4551508771256 | 0.0047523246361 | 0.1550979726017 | 4.5926973733585 | 0.0048624729857 | 0.1611304226331 | 210.4233352235 | 205.6566695474 |  |  |  |  |  |  |  |  |  | 0.1449015092105 | 5.4015494091436 | 0.0060419254721 | 0.1613325051963 | 165.5101514607 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 1 | H200 | 8 | 1024 | 1024 | 2 | 2026-06-13 | 8 | 1K/1K | vLLM | FP8 |  |  | 85774.53042583942 | 117.5111066834 | MiniMax-M3 | mtp | single_turn | 1781334000 |  | agg |  | 58.32496006 | 59.1861466234 | 0.0785628259182 | 0.08249013608900001 | 3.6779948063195 | 0.0038983449457 | 0.1501168757677 | 3.9437698011287 | 0.0040001504473 | 0.1780952871311 | 256.5191161713 | 249.9905973958 |  |  |  |  |  |  |  |  |  | 0.1182245377451 | 5.0601628156379 | 0.0050676699151 | 0.18832175126300002 | 197.329347954 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 1 | H200 | 8 | 1024 | 1024 | 1 | 2026-06-13 | 8 | 1K/1K | vLLM | FP8 |  |  | 50008.64495970803 | 68.5118435948 | MiniMax-M3 | mtp | single_turn | 1781334000 |  | agg |  | 34.1536653144 | 34.3581782804 | 0.0695579815656 | 0.0720206276371 | 3.413194777444 | 0.0033985322222000003 | 0.1214528419077 | 3.361127647385 | 0.0034998208496 | 0.1274016674608 | 294.2446722926 | 285.7289109887 |  |  |  |  |  |  |  |  |  | 0.1041271574795 | 4.0537216506898 | 0.004277853394900001 | 0.1431099385023 | 233.7621016174 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 4 | B200 | 4 | 8192 | 1024 | 30 | 2026-07-22 | 24 | 8K/1K | Dynamo TensorRT LLM | FP4 | 4 | 4 | 562816.2470439181 | 962.4157824451 | Kimi K2.5/2.6 | off | single_turn | 1784703600 | 20 | disagg | 4 | 5133.1024237569 | 128.2784541828 | 0.7563211830002 | 0.7162829572899 | 10.4405883985 | 0.007578686539 | 3.4798172000005003 | 10.5271828419983 | 0.0075695806186999995 | 3.5481994122916 | 131.9489854687 | 132.1077151269 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 8 | B200 | 8 | 8192 | 1024 | 308 | 2026-07-22 | 12 | 8K/1K | Dynamo TensorRT LLM | FP4 | 4 | 4 | 5245682.094343392 | 8970.1163813272 | Kimi K2.5/2.6 | off | single_turn | 1784703600 | 8 | disagg | 4 | 23922.0971537364 | 1494.1259951226 | 2.050800728 | 1.940499205227 | 22.8968203734998 | 0.0205363822259 | 3.9463532674999002 | 23.2203901802279 | 0.0204765829484 | 4.3660464587005 | 48.6940683612 | 48.8362732454 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 2 | B300 | 2 | 8192 | 1024 | 16 | 2026-08-10 | 4 | 8K/1K | SGLang | FP4 |  |  | 4390622.07646621 | 8342.1819452858 | Qwen 3.5 | mtp | single_turn | 1786345200 |  | agg |  | 7418.0749621484 | 924.1069831373 | 0.4632717685308 | 0.6236730146727 | 7.777872251113901 | 0.007773719703400001 | 0.6845782310702 | 7.7176628048182 | 0.0075539215053 | 0.8026886011387 | 128.6385460435 | 132.3815715185 |  |  |  |  |  |  |  |  |  | 1.3481292252429 | 8.9584650473669 | 0.0086202572741 | 1.4753814581782 | 116.005818412 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 1 | B300 | 4 | 1024 | 1024 | 32 | 2026-07-06 | 4 | 1K/1K | vLLM | FP4 |  |  | 1104904.032300421 | 2099.3176613708 | MiniMax-M3 | mtp | single_turn | 1783321200 |  | agg |  | 1055.9927875846 | 1043.3248737862 | 0.1391694738995 | 0.1488687770613 | 6.475469812518 | 0.0069246904122000005 | 0.0722286410164 | 6.8078120582068005 | 0.0072468528808999995 | 0.1115084703197 | 144.4107881321 | 137.9909343311 |  |  |  |  |  |  |  |  |  | 0.2229427448008 | 9.0681647268124 | 0.0095287460001 | 0.1093862688169 | 104.9456035443 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 2 | B300 | 2 | 1024 | 1024 | 512 | 2026-07-06 | 4 | 1K/1K | vLLM | FP4 |  |  | 4910480.753864527 | 9329.9134323426 | MiniMax-M3 | mtp | single_turn | 1783321200 |  | agg |  | 4706.6578721105 | 4623.2555602322 | 0.728333469131 | 0.8278899790976 | 45.3559324195376 | 0.035686458577 | 11.6426672624657 | 49.5652248248524 | 0.0403072403666 | 12.4686391525169 | 28.0218334874 | 24.8094384757 |  |  |  |  |  |  |  |  |  | 1.1271875431295 | 73.0168020036538 | 0.054856601034 | 16.1431461957516 | 18.2293467176 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 2 | B300 | 2 | 8192 | 1024 | 64 | 2026-07-29 | 4 | 8K/1K | vLLM | FP4 |  |  | 6143168.280490421 | 11672.0197329318 | MiniMax-M3 | mtp | single_turn | 1785308400 |  | agg |  | 10381.2798472147 | 1290.7398857171 | 0.4110635789111 | 0.4765756609924 | 20.5300070425728 | 0.0215460888866 | 0.27938759641259997 | 22.225353001761 | 0.023211050706999997 | 0.865083026456 | 46.412135644 | 43.082926862 |  |  |  |  |  |  |  |  |  | 0.822921329923 | 31.7589449326741 | 0.0338026730143 | 0.8988701118855 | 29.58345926 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 1 | H200 | 4 | 1024 | 1024 | 4 | 2026-06-13 | 4 | 1K/1K | vLLM | FP8 |  |  | 226287.43215372265 | 310.0137820506 | MiniMax-M3 | mtp | single_turn | 1781334000 |  | agg |  | 156.7801190038 | 153.2336630468 | 0.1217408906668 | 0.128115884746 | 5.600581619888501 | 0.0061441990799 | 0.146030860953 | 5.8602529703639 | 0.0062351643374 | 0.1655021504499 | 162.7551430213 | 160.3806966239 |  |  |  |  |  |  |  |  |  | 0.1896960616112 | 7.0905650891364 | 0.0077858140889999995 | 0.20293485596779998 | 128.4387205455 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 1 | H200 | 4 | 1024 | 1024 | 8 | 2026-06-13 | 4 | 1K/1K | vLLM | FP8 |  |  | 308656.84594372264 | 422.8598789429 | MiniMax-M3 | mtp | single_turn | 1781334000 |  | agg |  | 213.8823601469 | 208.977518796 | 0.1666999962181 | 0.1826783138914 | 8.0841780956835 | 0.0083956054077 | 0.1534733111039 | 8.433337793313 | 0.0088787382795 | 0.2881687375018 | 119.1099332859 | 112.6286155219 |  |  |  |  |  |  |  |  |  | 0.2779013548046 | 11.2690334733576 | 0.0114085950212 | 0.340514888987 | 87.653212174 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 1 | B300 | 8 | 8192 | 1024 | 4 | 2026-06-03 | 9 | 8K/1K | Dynamo vLLM | FP4 | 1 | 1 | 349396.6428258421 | 663.8536213691 | MiniMax-M2.5 | off | single_turn | 1780470000 | 8 | disagg | 1 | 5310.6886531341 | 82.9992423984 | 0.17836537699990002 | 0.17564521905140001 | 5.4128639195004995 | 0.0055736936489 | 0.2282928010009 | 5.3747696062751 | 0.005575299255 | 0.2503204449503 | 179.4142382025 | 179.3625694808 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 1 | B300 | 4 | 8192 | 1024 | 4 | 2026-06-03 | 5 | 8K/1K | Dynamo vLLM | FP4 | 1 | 1 | 592714.2922567368 | 1126.1571552878 | MiniMax-M2.5 | off | single_turn | 1780470000 | 4 | disagg | 1 | 5005.0106711265 | 156.4437763281 | 0.1916070280131 | 0.18800945964799998 | 5.7495760109741 | 0.0059739449302000005 | 0.2152385164518 | 5.730820269818699 | 0.0059681744257 | 0.2456443793199 | 167.3935752131 | 167.555424603 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 1 | B300 | 4 | 8192 | 1024 | 8 | 2026-06-03 | 5 | 8K/1K | Dynamo vLLM | FP4 | 1 | 1 | 1076796.1468549473 | 2045.9126790244 | MiniMax-M2.5 | off | single_turn | 1780470000 | 4 | disagg | 1 | 9080.7038778485 | 287.2148793184 | 0.2080638816115 | 0.20441428557320002 | 6.3601628096076 | 0.0064974169911999995 | 0.2394146390725 | 6.318027295667 | 0.0064923396933 | 0.3133577437111 | 153.9073144527 | 154.0276768073 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 1 | B300 | 2 | 1024 | 1024 | 64 | 2026-07-06 | 2 | 1K/1K | vLLM | FP4 |  |  | 1552388.833501316 | 2949.5387836525 | MiniMax-M3 | mtp | single_turn | 1783321200 |  | agg |  | 1491.634519131 | 1457.9042645215 | 0.2585238087922 | 0.43235715432849997 | 12.106127392034999 | 0.0128564371859 | 0.11709394911299999 | 19.6870949184413 | 0.021033169056199997 | 0.3911303123925 | 77.7820468873 | 47.5439529502 |  |  |  |  |  |  |  |  |  | 0.4051664303057 | 50.3444555340335 | 0.0537270693345 | 0.6807086063083 | 18.6125916114 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 1 | B300 | 2 | 1024 | 1024 | 2 | 2026-07-06 | 2 | 1K/1K | vLLM | FP4 |  |  | 283841.6038554737 | 539.2990473254 | MiniMax-M3 | mtp | single_turn | 1783321200 |  | agg |  | 267.6733824013 | 271.6256649242 | 0.0684900286142 | 0.072724464113 | 3.2155426931566997 | 0.003302929135 | 0.0636922989506 | 3.4211774900788 | 0.0035332336831000004 | 0.1122144904453 | 302.7615668178 | 283.0268500979 |  |  |  |  |  |  |  |  |  | 0.1009544139728 | 4.0902569402475 | 0.0044530717103 | 0.0883482319303 | 224.5640908221 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 1 | B300 | 4 | 8192 | 1024 | 1 | 2026-07-29 | 4 | 8K/1K | vLLM | FP4 |  |  | 428109.615504 | 813.4082694576 | MiniMax-M3 | mtp | single_turn | 1785308400 |  | agg |  | 722.9204172254 | 90.4878522321 | 0.048094120982600005 | 0.0540162499016 | 2.2765202485025 | 0.0022540493351 | 0.1525874404906 | 2.5699896609964 | 0.0026081584541 | 0.1554633843945 | 443.6460127312 | 383.4122878715 |  |  |  |  |  |  |  |  |  | 0.0904362272006 | 3.6684600530978 | 0.0038894782712 | 0.1657196877 | 257.1038916461 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 1 | B300 | 2 | 8192 | 1024 | 8 | 2026-07-29 | 2 | 8K/1K | vLLM | FP4 |  |  | 2538455.3998045265 | 4823.0652596286 | MiniMax-M3 | mtp | single_turn | 1785308400 |  | agg |  | 4282.308789991 | 540.7564696376 | 0.10925342515110001 | 0.1406631325027 | 5.965767402434699 | 0.0063068239926 | 0.22206924180499998 | 6.6101777340693 | 0.0068100641703 | 0.2908868057479 | 158.5584124705 | 146.8414944404 |  |  |  |  |  |  |  |  |  | 0.2585815296508 | 9.7025526224636 | 0.0099546777123 | 0.38769367821510003 | 100.4552863392 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 2 | B300 | 2 | 8192 | 1024 | 64 | 2026-08-10 | 4 | 8K/1K | SGLang | FP4 |  |  | 7780171.902698631 | 14782.3266151274 | Qwen 3.5 | mtp | single_turn | 1786345200 |  | agg |  | 13142.8279136643 | 1639.4987014631 | 0.9765073340386 | 1.3914886670075 | 17.50705184252 | 0.0175432219725 | 1.5404705271358 | 17.6804614629098 | 0.0171984102224 | 1.832710266424 | 57.0020718867 | 58.1449091556 |  |  |  |  |  |  |  |  |  | 3.0602209202479997 | 22.080996382329598 | 0.0205287801412 | 2.5612190366489997 | 48.7121004326 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 1 | B300 | 8 | 1024 | 1024 | 4 | 2026-07-06 | 8 | 1K/1K | vLLM | FP4 |  |  | 160452.37481942106 | 304.8595121569 | MiniMax-M3 | mtp | single_turn | 1783321200 |  | agg |  | 154.1735024787 | 150.6860096782 | 0.056596839975100004 | 0.06338392416500001 | 2.792970567476 | 0.0029358469942 | 0.0448056985042 | 2.9204061855533 | 0.0030985762955 | 0.0966524812509 | 340.6172058602 | 322.7288614659 |  |  |  |  |  |  |  |  |  | 0.0885787180508 | 3.7931623479817 | 0.0042378284589 | 0.0949432009016 | 235.9699099894 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 1 | H200 | 4 | 1024 | 1024 | 16 | 2026-06-13 | 4 | 1K/1K | vLLM | FP8 |  |  | 464012.89977992704 | 635.6976726985 | MiniMax-M3 | mtp | single_turn | 1781334000 |  | agg |  | 320.411960876 | 315.2857118225 | 0.2274928428233 | 0.247650940509 | 10.8395629124716 | 0.0115289502937 | 0.1717239199206 | 11.3561143052881 | 0.012066193705 | 0.2644480105489 | 86.7381656201 | 82.8761765682 |  |  |  |  |  |  |  |  |  | 0.3721059700474 | 14.8862304287031 | 0.015814361417800002 | 0.3672379070893 | 63.2336629711 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 1 | H200 | 4 | 1024 | 1024 | 32 | 2026-06-13 | 4 | 1K/1K | vLLM | FP8 |  |  | 633092.6244513139 | 867.3368954983 | MiniMax-M3 | mtp | single_turn | 1781334000 |  | agg |  | 436.2853335184 | 431.0515619799 | 0.3426201120019 | 0.3629657776107 | 15.8158534085378 | 0.0171720563987 | 0.1956011848524 | 16.6696507231274 | 0.0176359378845 | 0.3486678586283 | 58.234143703 | 56.7023997561 |  |  |  |  |  |  |  |  |  | 0.5432974848896001 | 22.4154028445482 | 0.0237625641841 | 0.3851421944797 | 42.0830004814 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 4 | B300 | 4 | 8192 | 1024 | 128 | 2026-06-03 | 5 | 8K/1K | Dynamo vLLM | FP4 | 1 | 1 | 6766834.063651264 | 12856.9847209374 | MiniMax-M2.5 | off | single_turn | 1780470000 | 4 | disagg | 1 | 57167.6048990748 | 1779.329676403 | 0.44663390098140004 | 0.43268297380590004 | 15.6349003349897 | 0.0139615089116 | 2.7583777974942 | 15.9930348746973 | 0.0137472629064 | 3.3590301513645 | 71.6254959496 | 72.7417527991 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 4 | B300 | 4 | 8192 | 1024 | 256 | 2026-06-03 | 6 | 8K/1K | Dynamo vLLM | FP4 | 1 | 1 | 8846593.562594527 | 16808.5277689296 | MiniMax-M2.5 | off | single_turn | 1780470000 | 4 | disagg | 2 | 44823.4490753546 | 2801.0671157172 | 0.6678273095749 | 0.6510823114127999 | 19.9886890819762 | 0.0211142794219 | 0.4521596624982 | 20.3863323516922 | 0.020669079474699998 | 1.3508220087598 | 47.3613131672 | 48.3814482994 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 1 | B300 | 2 | 8192 | 1024 | 32 | 2026-07-29 | 2 | 8K/1K | vLLM | FP4 |  |  | 4668772.321894684 | 8870.6674115999 | MiniMax-M3 | mtp | single_turn | 1785308400 |  | agg |  | 7882.7037063317 | 987.9637052682 | 0.2556639130053 | 0.3128661091099 | 13.8689041024918 | 0.0144993296485 | 0.2337206204975 | 14.6471138534132 | 0.0153011585268 | 0.5231646680907001 | 68.9687057433 | 65.3545284332 |  |  |  |  |  |  |  |  |  | 0.5295875535085001 | 19.744952636011302 | 0.0206210555748 | 0.552840076093 | 48.4941227364 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 2 | B300 | 2 | 8192 | 1024 | 128 | 2026-07-29 | 4 | 8K/1K | vLLM | FP4 |  |  | 7378817.569681052 | 14019.753382394 | MiniMax-M3 | mtp | single_turn | 1785308400 |  | agg |  | 12468.6090060667 | 1551.1443763274 | 0.7427383398171999 | 0.7989810666736 | 35.5258337795967 | 0.0371367984867 | 0.4397349974606 | 37.3062688846194 | 0.039001024377599995 | 1.4713440217927 | 26.927469269 | 25.6403521692 |  |  |  |  |  |  |  |  |  | 1.3012008849531 | 50.2219498144463 | 0.0522883300667 | 1.0548148811562001 | 19.1247262769 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 1 | B300 | 2 | 1024 | 1024 | 512 | 2026-07-06 | 2 | 1K/1K | vLLM | FP4 |  |  | 5449527.70261779 | 10354.1026349738 | MiniMax-M3 | mtp | single_turn | 1783321200 |  | agg |  | 5223.3302086816 | 5130.7724262922 | 0.8644787638914 | 0.9409920207772 | 41.3937430975493 | 0.0429588137648 | 0.6442798324861 | 44.9577471654935 | 0.0468202560959 | 1.8338473005448 | 23.2781101796 | 21.3582770234 |  |  |  |  |  |  |  |  |  | 1.0482160711194999 | 69.006968006213 | 0.0548827506037 | 4.5767070754431 | 18.2206611185 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 2 | B300 | 2 | 1024 | 1024 | 256 | 2026-07-06 | 4 | 1K/1K | vLLM | FP4 |  |  | 4167138.9024931053 | 7917.5639147369 | MiniMax-M3 | mtp | single_turn | 1783321200 |  | agg |  | 3994.4872556798 | 3923.076659057 | 0.5304610184394001 | 0.6377348783322 | 24.660210461588598 | 0.0264322274013 | 0.24642446427609999 | 29.192920895751 | 0.0310180010372 | 0.6273807255793 | 37.8326042985 | 32.2393438184 |  |  |  |  |  |  |  |  |  | 0.8251452775672 | 47.445927999261094 | 0.0447097504865 | 0.6022770617623 | 22.3664858139 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 2 | B300 | 2 | 8192 | 1024 | 32 | 2026-07-29 | 4 | 8K/1K | vLLM | FP4 |  |  | 4507079.352545789 | 8563.450769837 | MiniMax-M3 | mtp | single_turn | 1785308400 |  | agg |  | 7609.7030798507 | 953.7476899862 | 0.2764961021021 | 0.3214369532129 | 13.058894061483398 | 0.0136194860867 | 0.2524795159698 | 14.842685326433301 | 0.015474267726100001 | 0.5497903852869 | 73.4242095212 | 64.6234133788 |  |  |  |  |  |  |  |  |  | 0.5820451005828 | 23.3021728916559 | 0.0243192168768 | 0.5782461978495 | 41.1197451409 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 1 | B300 | 4 | 8192 | 1024 | 4 | 2026-08-10 | 4 | 8K/1K | SGLang | FP4 |  |  | 1316939.0274272105 | 2502.1841521117 | Qwen 3.5 | mtp | single_turn | 1786345200 |  | agg |  | 2223.9125074372 | 278.2716446746 | 0.1899412034545 | 0.26655206817189997 | 3.2623535091989 | 0.0032596615743 | 0.2577714282088 | 3.2427447278751003 | 0.0032037635322 | 0.2973446046002 | 306.7803135995 | 312.1328993056 |  |  |  |  |  |  |  |  |  | 0.5788378432393 | 3.5993900268804 | 0.0035313095595 | 0.4188071948476 | 283.1810644604 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 2 | B300 | 2 | 8192 | 1024 | 8 | 2026-08-10 | 4 | 8K/1K | SGLang | FP4 |  |  | 3104504.9865488946 | 5898.5594744429 | Qwen 3.5 | mtp | single_turn | 1786345200 |  | agg |  | 5235.6420003663 | 662.9174740766 | 0.3070461555035 | 0.4450977619587 | 5.4196959029941 | 0.0055549588801 | 0.3416801800195 | 5.4535257652263995 | 0.0053998091998 | 0.4461761527993 | 180.0193343618 | 185.1917286333 |  |  |  |  |  |  |  |  |  | 0.9860939435894001 | 6.5703309194068 | 0.0061677740993999995 | 0.5786744616984 | 162.1330457125 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 2 | B300 | 2 | 8192 | 1024 | 4 | 2026-08-10 | 4 | 8K/1K | SGLang | FP4 |  |  | 2169425.1112646842 | 4121.9077114029 | Qwen 3.5 | mtp | single_turn | 1786345200 |  | agg |  | 3663.5041853951 | 458.4035260078 | 0.2242089600768 | 0.3259542044416 | 3.9816020934959 | 0.0039378662036 | 0.3025043049711 | 3.9243017481203 | 0.0039061110067999997 | 0.3388058089971 | 253.9446360761 | 256.0091093848 |  |  |  |  |  |  |  |  |  | 0.7251087985584 | 4.3698760180851 | 0.0042501680811 | 0.5276755547499999 | 235.2848124897 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 1 | B300 | 4 | 8192 | 1024 | 16 | 2026-06-03 | 5 | 8K/1K | Dynamo vLLM | FP4 | 1 | 1 | 1924893.214285737 | 3657.2971071429 | MiniMax-M2.5 | off | single_turn | 1780470000 | 4 | disagg | 1 | 16261.5575270669 | 506.2320021619 | 0.2328742349055 | 0.2284847291975 | 6.9995384019567 | 0.0072794697495999995 | 0.2559295115061 | 7.0050544524609 | 0.0072526454741 | 0.3789973901701 | 137.3726431181 | 137.8807227747 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 2 | B300 | 2 | 1024 | 1024 | 128 | 2026-07-06 | 4 | 1K/1K | vLLM | FP4 |  |  | 3474342.366206158 | 6601.2504957917 | MiniMax-M3 | mtp | single_turn | 1783321200 |  | agg |  | 3337.6466944326 | 3263.6038013591 | 0.3580663129687 | 0.3809956068318 | 16.4851505940314 | 0.0175918912421 | 0.1668793399585 | 17.395716643036 | 0.0185375052282 | 0.37414267829139997 | 56.8443714343 | 53.9446914615 |  |  |  |  |  |  |  |  |  | 0.5622624469688 | 23.2503733819816 | 0.0247621449235 | 0.2829879245488 | 40.3842237047 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 1 | B300 | 8 | 1024 | 1024 | 2 | 2026-07-06 | 8 | 1K/1K | vLLM | FP4 |  |  | 99188.82884221053 | 188.4587748002 | MiniMax-M3 | mtp | single_turn | 1783321200 |  | agg |  | 93.538822188 | 94.9199526121 | 0.049176217056799996 | 0.0525050854653 | 2.3895931504667 | 0.0024206317684 | 0.0430494335014 | 2.4652681525622 | 0.0025450784072 | 0.0815373649471 | 413.1152920639 | 392.9152033919 |  |  |  |  |  |  |  |  |  | 0.0702805303736 | 2.9571145204944 | 0.0028523644062 | 0.0486951844301 | 350.5863408765 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 16 | GB300 NVL72 | 16 | 8192 | 1024 | 1536 | 2026-07-30 | 34 | 8K/1K | Dynamo TensorRT LLM | FP4 | 2 | 2 | 10016013.780024858 | 21233.9492136527 | Qwen 3.5 | mtp | single_turn | 1785394800 | 16 | disagg | 18 | 35649.0381292083 | 5016.9741836526 | 3.1651531190146 | 2.7262007701561 | 17.010709334485 | 0.0110377130672 | 7.0750901300053 | 17.126151157264403 | 0.0108892808902 | 7.0938078711897 | 90.5984775935 | 91.8334286791 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 1 | GB300 NVL72 | 4 | 8192 | 1024 | 90 | 2026-07-30 | 64 | 8K/1K | Dynamo TensorRT LLM | FP4 | 2 | 2 | 1175319.5641411794 | 2491.6774759793 | Qwen 3.5 | mtp | single_turn | 1785394800 | 60 | disagg | 4 | 35450.4072087714 | 294.4288271265 | 0.9123719939962 | 0.7866333039761999 | 4.4366246930149 | 0.0031683500617 | 1.5241092669894 | 4.564161975134399 | 0.0031390024829 | 1.6792937172508 | 315.6216896888 | 318.5725418976 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 16 | GB300 NVL72 | 16 | 8192 | 1024 | 48 | 2026-07-30 | 18 | 8K/1K | Dynamo TensorRT LLM | FP4 | 2 | 2 | 2035862.6274417455 | 4316.0287701765 | Qwen 3.5 | mtp | single_turn | 1785394800 | 16 | disagg | 2 | 34543.8375146409 | 537.5526771185 | 1.0889078034961002 | 0.943381922509 | 4.902269474521701 | 0.0038067211378 | 1.3700259679753999 | 5.0160911601916 | 0.0037773837831999997 | 1.544740614792 | 262.693263783 | 264.7334921155 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 8 | GB300 NVL72 | 8 | 8192 | 1024 | 168 | 2026-07-30 | 62 | 8K/1K | Dynamo TensorRT LLM | FP4 | 2 | 2 | 1812114.9253559907 | 3841.6836417547 | Qwen 3.5 | mtp | single_turn | 1785394800 | 56 | disagg | 6 | 35283.8084815362 | 472.8845517782 | 1.0519453589804 | 0.9039738610217001 | 5.5715910019935 | 0.0036537203455 | 2.2231095839816 | 5.6350106622007 | 0.0036116603886 | 2.3079968478375 | 273.693634277 | 276.8809612182 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 16 | GB300 NVL72 | 16 | 8192 | 1024 | 384 | 2026-07-30 | 24 | 8K/1K | Dynamo TensorRT LLM | FP4 | 2 | 2 | 6281898.550334623 | 13317.6249267094 | Qwen 3.5 | mtp | single_turn | 1785394800 | 16 | disagg | 8 | 35515.8480809257 | 2218.5133496013 | 1.7410581090371 | 1.4992524266370002 | 9.5450866429892 | 0.0060704183109 | 4.0280418315087 | 9.6287801986154 | 0.0060045199694 | 4.1005310010496 | 164.7332932906 | 166.5412064744 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 1 | GB300 NVL72 | 16 | 8192 | 1024 | 4 | 2026-07-30 | 66 | 8K/1K | Dynamo TensorRT LLM | FP4 | 2 | 2 | 95928.17036924529 | 203.3677211828 | Qwen 3.5 | mtp | single_turn | 1785394800 | 64 | disagg | 2 | 5955.6648088464 | 23.6084371933 | 0.6508948060218 | 0.5685635874687001 | 2.3931389609934 | 0.0022988643516 | 0.2425401924702 | 2.4065826944197 | 0.0022908299807 | 0.28335321346910003 | 434.9973930932 | 436.5230106183 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 4 | B300 | 4 | 8192 | 1024 | 32 | 2026-06-03 | 5 | 8K/1K | Dynamo vLLM | FP4 | 1 | 1 | 3056014.131874421 | 5806.4268505614 | MiniMax-M2.5 | off | single_turn | 1780470000 | 4 | disagg | 1 | 25788.2622982264 | 810.9679886451 | 0.2892033749959 | 0.2830215775159 | 8.712319772515901 | 0.009040432213800001 | 0.3201398519741 | 8.8392394871666 | 0.008989160368400001 | 0.5299029346594 | 110.6141804233 | 111.2450950943 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 8 | GB300 NVL72 | 8 | 8192 | 1024 | 480 | 2026-07-30 | 52 | 8K/1K | Dynamo TensorRT LLM | FP4 | 2 | 2 | 4346029.913455944 | 9213.5834165266 | Qwen 3.5 | mtp | single_turn | 1785394800 | 40 | disagg | 12 | 35491.2436168404 | 1330.2853564324 | 1.5220715310134 | 1.3081861913221 | 7.9110792610154 | 0.0052792813598 | 3.1034543934802 | 8.0369609655286 | 0.005210250506 | 3.2395970265064 | 189.4197205729 | 191.9293513507 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 8 | GB300 NVL72 | 8 | 8192 | 1024 | 42 | 2026-07-30 | 58 | 8K/1K | Dynamo TensorRT LLM | FP4 | 2 | 2 | 641362.3011633019 | 1359.6880784662 | Qwen 3.5 | mtp | single_turn | 1785394800 | 56 | disagg | 2 | 35061.5833744068 | 156.048960754 | 0.8302873549983 | 0.7177594403548 | 4.1927020669973 | 0.0029000403765 | 1.5296127694891999 | 4.2823534872895 | 0.0028713664845 | 1.6409561115802 | 344.8227852606 | 348.2662367868 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 8 | GB300 NVL72 | 8 | 8192 | 1024 | 192 | 2026-07-30 | 38 | 8K/1K | Dynamo TensorRT LLM | FP4 | 2 | 2 | 2981485.7440465093 | 6320.7497773786 | Qwen 3.5 | mtp | single_turn | 1785394800 | 32 | disagg | 6 | 35582.1606750644 | 834.2352340625 | 1.2113214879937 | 1.0415574053059 | 6.3641238840064 | 0.0041972869374 | 2.5220369359886 | 6.4388672707173 | 0.0041478030577 | 2.6198235484087 | 238.249139244 | 241.0914853212 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 8 | H200 | 8 | 8192 | 1024 | 160 | 2026-03-23 | 64 | 8K/1K | Dynamo TensorRT LLM | FP8 | 8 | 8 | 824974.5135229927 | 1130.2150835265 | DeepSeek-R1 | mtp | single_turn | 1774249200 | 40 | disagg | 24 | 2679.9624845299 | 200.3666429244 | 3.9621351920068 | 3.432493078043 | 17.796283358940897 | 0.0161986965615 | 3.0017717814771 | 17.620267784017802 | 0.0151711563269 | 3.6779101638748 | 61.7333620766 | 65.9145538054 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 4 | B300 | 4 | 8192 | 1024 | 512 | 2026-06-03 | 6 | 8K/1K | Dynamo vLLM | FP4 | 1 | 1 | 11043272.311915578 | 20982.2173926396 | MiniMax-M2.5 | off | single_turn | 1780470000 | 4 | disagg | 2 | 55956.6274539481 | 3495.0123619854 | 1.0426559259649 | 0.9991397677725 | 32.5871270535281 | 0.0333859396369 | 1.2684790894855 | 32.8995826457392 | 0.0317307722439 | 3.6759153839706 | 29.9527289295 | 31.5151485225 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 4 | B300 | 4 | 8192 | 1024 | 64 | 2026-06-03 | 6 | 8K/1K | Dynamo vLLM | FP4 | 1 | 1 | 4252159.624367316 | 8079.1032862979 | MiniMax-M2.5 | off | single_turn | 1780470000 | 4 | disagg | 2 | 21550.5568176221 | 1343.3765206357 | 0.3560889465734 | 0.3481319445468 | 10.5676364348037 | 0.0111332953461 | 0.2558715546038 | 10.6737646839258 | 0.0110538919422 | 0.4985207849837 | 89.8206657523 | 90.4658743933 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 2 | 8192 | 1024 | 64 | 2026-03-27 | 2 | 8K/1K | vLLM | FP4 |  |  | 9888451.503649635 | 13547.17856 | gpt-oss-120b | off |  | 1774594800 |  | agg |  | 12041.9365 | 1505.242063 |  |  | 21.64914323 | 0.020381063999999997 | 0.7993146809999999 |  |  |  | 49.06515167 |  | 1 | 1 | 0 | random | 108.8462807 | 2621440 | 327680 | 2.939925904 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 8 | 8192 | 1024 | 1 | 2026-03-27 | 8 | 8K/1K | vLLM | FP4 |  |  | 250793.8269343066 | 343.5875429 | gpt-oss-120b | off |  | 1774594800 |  | agg |  | 305.4111493 | 38.17639366 |  |  | 3.353059788 | 0.0031966480000000003 | 0.082888682 |  |  |  | 312.8276677 |  | 1 | 1 | 0 | random | 16.76428648 | 40960 | 5120 | 0.298253075 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 8 | 8192 | 1024 | 2 | 2026-03-27 | 8 | 8K/1K | vLLM | FP4 |  |  | 471869.4991240876 | 646.4612138 | gpt-oss-120b | off |  | 1774594800 |  | agg |  | 574.63219 | 71.82902376 |  |  | 3.560005599 | 0.0033730310000000003 | 0.109395226 |  |  |  | 296.4692879 |  | 1 | 1 | 0 | random | 17.82009462 | 81920 | 10240 | 0.561164248 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 1 | B300 | 4 | 1024 | 1024 | 8 | 2026-06-12 | 4 | 1K/1K | TensorRT LLM | FP4 |  |  | 190757.1469566842 | 362.4385792177 | DeepSeek-V4-Pro | mtp | single_turn | 1781247600 |  | agg |  | 180.6457295728 | 181.7928496449 | 2.3277738749748 | 2.2016222370773 | 9.4568290154566 | 0.0100542820773 | 0.1602477750275 | 9.9405291019517 | 0.010422230130999998 | 0.2809118332749 | 99.4601098626 | 95.9487544821 |  |  |  |  |  |  |  |  |  | 2.5943775309715003 | 12.448685503017598 | 0.013336146039200001 | 0.5262353247148 | 74.9841818665 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 1 | B300 | 4 | 1024 | 1024 | 1 | 2026-06-12 | 4 | 1K/1K | TensorRT LLM | FP4 |  |  | 41531.70073373684 | 78.9102313941 | DeepSeek-V4-Pro | mtp | single_turn | 1781247600 |  | agg |  | 39.3370058966 | 39.5732254975 | 1.3124679725005999 | 1.1774957707722 | 5.9013522439927 | 0.0057952412948 | 0.1172301069892 | 5.8204203886911 | 0.0056148672985 | 0.6394392232964 | 172.555368987 | 178.098599102 |  |  |  |  |  |  |  |  |  | 1.3313824329089 | 7.4866173477756 | 0.0065760190121999995 | 1.6904430321913 | 152.0676868694 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 8 | GB300 NVL72 | 8 | 1024 | 1024 | 12 | 2026-02-03 | 34 | 1K/1K | Dynamo TensorRT LLM | FP4 | 2 | 2 | 81307.33957839622 | 172.3715599062 | DeepSeek-R1 | mtp | single_turn | 1770105600 | 32 | disagg | 2 | 1472.7323262995 | 91.0990120066 | 0.8782400769995999 | 0.7781435256088001 | 3.7379440465011 | 0.003728551464 | 0.3578497510171 | 3.6398164080199 | 0.0035723485751 | 0.3716137303301 | 268.200669794 | 279.9278902858 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 8 | H200 | 8 | 8192 | 1024 | 28 | 2026-03-23 | 64 | 8K/1K | Dynamo TensorRT LLM | FP8 | 8 | 8 | 272436.65092766425 | 373.2382117709 | DeepSeek-R1 | mtp | single_turn | 1774249200 | 56 | disagg | 8 | 2653.8234255888 | 47.4403240827 | 2.0654048461001 | 1.7797130514927 | 9.3852751390077 | 0.0085436350249 | 1.5596411650768 | 9.3035170864106 | 0.0080490565104 | 1.8922836605484 | 117.046198379 | 124.2381636542 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 8 | B300 | 8 | 1024 | 1024 | 120 | 2026-01-28 | 42 | 1K/1K | Dynamo TensorRT LLM | FP4 | 2 | 2 | 397226.95825142105 | 754.7312206777 | DeepSeek-R1 | mtp | single_turn | 1769587200 | 40 | disagg | 2 | 7939.809450794 | 395.4773091719 | 1.7068251119927 | 1.504208827984 | 6.8639008614699994 | 0.0072367782108 | 0.1890602840576 | 6.6649840745679 | 0.0069179351588 | 0.315969554947 | 138.1830382072 | 144.5518029649 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 1 | B300 | 4 | 8192 | 1024 | 8 | 2026-07-30 | 28 | 8K/1K | Dynamo SGLang | FP4 | 4 | 4 | 184235.59157226316 | 350.0476239873 | DeepSeek-V4-Pro | mtp | single_turn | 1785394800 | 24 | disagg | 4 | 2174.9080111697 | 45.9042261236 | 0.0144105819636 | 0.0141727379751 | 5.9701599565451 | 0.0052865521802 | 0.7497390829958 | 6.4927352186627 | 0.0054373437784 | 1.4624762813357 | 189.159203563 | 183.9133298825 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 4 | 8192 | 1024 | 64 | 2026-03-27 | 4 | 8K/1K | vLLM | FP4 |  |  | 7746494.671532848 | 10612.6977 | gpt-oss-120b | off |  | 1774594800 |  | agg |  | 9433.50907 | 1179.188634 |  |  | 14.06502454 | 0.013194243 | 0.567314194 |  |  |  | 75.79063218 |  | 1 | 1 | 0 | random | 69.47149731 | 2621440 | 327680 | 4.606205601 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 2 | 1024 | 1024 | 512 | 2026-03-27 | 2 | 1K/1K | vLLM | FP4 |  |  | 9456430.175182482 | 12955.30934 | gpt-oss-120b | off |  | 1774594800 |  | agg |  | 6477.654671 | 6477.654671 |  |  | 39.91009949 | 0.038500541 | 0.524046413 |  |  |  | 25.97366124 |  | 1 | 1 | 0 | random | 202.3448403 | 2621440 | 2621440 | 12.65166928 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 8 | 8192 | 1024 | 4 | 2026-07-14 | 8 | 8K/1K | SGLang | FP4 |  |  | 205050.48252591243 | 280.9191610605 | DeepSeek-V4-Pro | off |  | 1784012400 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 66.8374425706 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 16 | GB300 NVL72 | 16 | 8192 | 1024 | 666 | 2026-02-03 | 36 | 8K/1K | Dynamo TensorRT LLM | FP4 | 2 | 2 | 5745264.126273821 | 12179.9599477005 | DeepSeek-R1 | mtp | single_turn | 1770105600 | 16 | disagg | 20 | 19489.4672037892 | 3043.0758775896 | 2.5447164129873 | 2.2056625931905 | 12.1429532180191 | 0.010320322778299999 | 2.7212235445040003 | 11.8944292825125 | 0.0095917966742 | 3.0691295953088003 | 96.8961941873 | 104.2557545755 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 32 | GB300 NVL72 | 32 | 8192 | 1024 | 180 | 2026-02-03 | 40 | 8K/1K | Dynamo TensorRT LLM | FP4 | 2 | 2 | 2231514.7829075 | 4730.8113397639 | DeepSeek-R1 | mtp | single_turn | 1770105600 | 32 | disagg | 8 | 21030.8934815715 | 655.790804312 | 1.4644192999986 | 1.2775204831784 | 7.503871290508 | 0.0059842742751 | 2.1143423394823997 | 7.3823316377427 | 0.0055251656252999996 | 2.3020586394334 | 167.1046402675 | 180.9900494967 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 8 | GB300 NVL72 | 8 | 1024 | 1024 | 5 | 2026-02-03 | 34 | 1K/1K | Dynamo TensorRT LLM | FP4 | 2 | 2 | 33725.73300273585 | 71.4985539658 | DeepSeek-R1 | mtp | single_turn | 1770105600 | 32 | disagg | 2 | 608.9784318577 | 37.9060615975 | 0.7575018520001 | 0.6646102177453 | 3.7866537709779 | 0.0032449972976 | 0.537719629996 | 3.7384476805502 | 0.0030970610893 | 0.8806239112839 | 308.1666665028 | 322.8867533353 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 8 | B300 | 8 | 1024 | 1024 | 11 | 2026-01-28 | 42 | 1K/1K | Dynamo TensorRT LLM | FP4 | 2 | 2 | 39475.851645421055 | 75.0041181263 | DeepSeek-R1 | mtp | single_turn | 1769587200 | 40 | disagg | 2 | 789.0391299167 | 39.3023675368 | 0.7440155820223 | 0.6618175429718 | 6.5864598939661 | 0.0031857889095999996 | 3.6856998261064 | 6.123232413671699 | 0.0030864185010999997 | 3.2834338487274 | 313.8939924692 | 324.0001314305 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 |  | 8192 | 1024 | 64 | 2026-10-16 |  | 8K/1K | Dynamo TensorRT LLM | FP8 |  |  | 1096661.5240433577 | 1502.4262879394 | DeepSeek-R1 | mtp | single_turn | 1792134000 |  | disagg |  | 2671.5305249607 | 333.3220509182 | 4.2312404480763 | 3.7196868699648 | 20.842199738370297 | 0.0189690709472 | 3.6001777495258 | 21.1218425950778 | 0.0183198022281 | 4.2611996898399 | 52.7173946885 | 54.5857421139 |  |  |  |  |  |  |  |  |  | 4.2509133289568 | 24.1937452983111 | 0.0204839276039 | 4.9480729513336 | 48.818762658 |  |  |  | 640 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B300 | 8 | 1024 | 1024 | 8 | 2026-06-04 | 8 | 1K/1K | vLLM | FP4 |  |  | 174425.0148681579 | 331.4075282495 | MiniMax-M2.5 | off |  | 1780556400 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 171.1672597871 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B300 | 1 | 1024 | 1024 | 8 | 2026-06-04 | 1 | 1K/1K | vLLM | FP4 |  |  | 923313.0848602105 | 1754.2948612344 | MiniMax-M2.5 | off |  | 1780556400 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 113.0023458725 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 8 | 1024 | 1024 | 4 | 2026-07-14 | 8 | 1K/1K | SGLang | FP4 |  |  | 50906.741010218975 | 69.742235184 | DeepSeek-V4-Pro | off |  | 1784012400 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 72.634450126 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B200 | 2 | 1024 | 1024 | 8 | 2026-06-24 | 2 | 1K/1K | TensorRT LLM | FP4 |  |  | 774867.4462340936 | 1325.0233330603 | Qwen 3.5 | mtp |  | 1782284400 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 179.948624945 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 4 | 8192 | 1024 | 32 | 2026-03-27 | 4 | 8K/1K | vLLM | FP4 |  |  | 5813762.909489051 | 7964.855186 | gpt-oss-120b | off |  | 1774594800 |  | agg |  | 7079.871276 | 884.9839095 |  |  | 9.233065663 | 0.008478001 | 0.5600711979999999 |  |  |  | 117.9523409 |  | 1 | 1 | 0 | random | 46.28332737 | 1310720 | 163840 | 3.456968397 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 1 | 8192 | 1024 | 512 | 2026-03-27 | 1 | 8K/1K | vLLM | FP4 |  |  | 15073755.226277374 | 20651.04466 | gpt-oss-120b | off |  | 1774594800 |  | agg |  | 18356.48414 | 2294.560518 |  |  | 236.8557691 | 0.078611908 | 156.4357867 |  |  |  | 12.72071904 |  | 1 | 1 | 0 | random | 1142.458427 | 20971520 | 2621440 | 2.240781756 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 1 | 8192 | 1024 | 256 | 2026-03-27 | 1 | 8K/1K | vLLM | FP4 |  |  | 14974979.80291971 | 20515.72233 | gpt-oss-120b | off |  | 1774594800 |  | agg |  | 18236.19762 | 2279.524703 |  |  | 99.04666932 | 0.07759542500000001 | 19.6665494 |  |  |  | 12.8873577 |  | 1 | 1 | 0 | random | 574.9970589 | 10485760 | 1310720 | 2.226098343 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 1 | 8192 | 1024 | 64 | 2026-03-27 | 1 | 8K/1K | vLLM | FP4 |  |  | 12396462.81751825 | 16983.15406 | gpt-oss-120b | off |  | 1774594800 |  | agg |  | 15096.13694 | 1887.017118 |  |  | 34.664769740000004 | 0.032848843 | 1.060403298 |  |  |  | 30.44247246 |  | 1 | 1 | 0 | random | 173.6497231 | 2621440 | 327680 | 1.842790154 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 8 | 8192 | 1024 | 8 | 2026-03-27 | 8 | 8K/1K | vLLM | FP4 |  |  | 1476448.9583941607 | 2022.735073 | gpt-oss-120b | off |  | 1774594800 |  | agg |  | 1797.986732 | 224.7483415 |  |  | 4.5500656799999994 | 0.004138305 | 0.31657984699999997 |  |  |  | 241.6448384 |  | 1 | 1 | 0 | random | 22.78103574 | 327680 | 40960 | 1.755846418 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 4 | 8192 | 1024 | 16 | 2026-03-27 | 4 | 8K/1K | vLLM | FP4 |  |  | 3753861.2408759124 | 5142.7899 | gpt-oss-120b | off |  | 1774594800 |  | agg |  | 4571.3688 | 571.4211 |  |  | 7.171786936 | 0.006475682 | 0.5471638249999999 |  |  |  | 154.4238794 |  | 1 | 1 | 0 | random | 35.84046862 | 655360 | 81920 | 2.232113672 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 1 | 8192 | 1024 | 32 | 2026-03-27 | 1 | 8K/1K | vLLM | FP4 |  |  | 9721311.693430657 | 13318.19702 | gpt-oss-120b | off |  | 1774594800 |  | agg |  | 11838.39735 | 1479.799668 |  |  | 21.67807871 | 0.020152486 | 1.062086034 |  |  |  | 49.62167072 |  | 1 | 1 | 0 | random | 110.7176894 | 1310720 | 163840 | 1.445116864 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 4 | 8192 | 1024 | 2 | 2026-03-27 | 4 | 8K/1K | vLLM | FP4 |  |  | 863670.0226277373 | 1183.227931 | gpt-oss-120b | off |  | 1774594800 |  | agg |  | 1051.758161 | 131.4697701 |  |  | 3.890964555 | 0.003665886 | 0.140763429 |  |  |  | 272.7853695 |  | 1 | 1 | 0 | random | 19.47215697 | 81920 | 10240 | 0.51355379 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B300 | 4 | 1024 | 1024 | 128 | 2026-06-04 | 4 | 1K/1K | vLLM | FP4 |  |  | 2579645.1940707364 | 4901.3258687344 | MiniMax-M2.5 | off |  | 1780556400 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 79.0570177071 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 8 | 8192 | 1024 | 16 | 2026-03-27 | 8 | 8K/1K | vLLM | FP4 |  |  | 2398605.535766423 | 3286.089584 | gpt-oss-120b | off |  | 1774594800 |  | agg |  | 2920.968519 | 365.1210649 |  |  | 5.594319984 | 0.004976116 | 0.50375361 |  |  |  | 200.9599571 |  | 1 | 1 | 0 | random | 28.04549226 | 655360 | 81920 | 2.85250832 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 8 | 1024 | 1024 | 1 | 2026-03-27 | 8 | 1K/1K | vLLM | FP4 |  |  | 59796.71977372263 | 81.92150609 | gpt-oss-120b | off |  | 1774594800 |  | agg |  | 42.27661835 | 39.64488774 |  |  | 3.225472636 | 0.003131095 | 0.022362628000000002 |  |  |  | 319.3771046 |  | 1 | 1 | 0 | random | 14.05918977 | 4755 | 4459 | 0.355639271 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 4 | 1024 | 1024 | 16 | 2026-03-27 | 4 | 1K/1K | vLLM | FP4 |  |  | 1047892.8678832118 | 1435.613229 | gpt-oss-120b | off |  | 1774594800 |  | agg |  | 717.8066144 | 717.8066144 |  |  | 5.704254134 | 0.005373723 | 0.20693603200000002 |  |  |  | 186.0907412 |  | 1 | 1 | 0 | random | 28.5313615 | 81920 | 81920 | 2.803932087 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 8 | 1024 | 1024 | 8 | 2026-03-27 | 8 | 1K/1K | vLLM | FP4 |  |  | 363594.04182481754 | 498.1238373 | gpt-oss-120b | off |  | 1774594800 |  | agg |  | 249.0619187 | 249.0619187 |  |  | 4.109740406 | 0.003927476 | 0.09193291 |  |  |  | 254.6164795 |  | 1 | 1 | 0 | random | 20.55713707 | 40960 | 40960 | 1.94579624 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B200 | 1 | 8192 | 1024 | 128 | 2026-02-05 | 1 | 8K/1K | TensorRT LLM | FP4 |  |  | 32276833.058479536 | 55193.38453 | gpt-oss-120b | eagle |  | 1770278400 |  | agg |  | 49082.91 | 6110.477 |  |  | 20.678459999999998 | 0.020027999999999997 | 0.190048 |  |  |  | 49.93066 |  |  |  | 1 |  | 192.6989 | 9458221 | 1177482 | 6.642488 | FP8 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B200 | 1 | 8192 | 1024 | 128 | 2026-02-05 | 1 | 8K/1K | TensorRT LLM | FP4 |  |  | 32409554.555555556 | 55420.33829 | gpt-oss-120b | eagle |  | 1770278400 |  | agg |  | 49082.91 | 6110.477 |  |  | 20.678459999999998 | 0.020027999999999997 | 0.190048 |  |  |  | 49.76625 |  |  |  | 1 |  | 192.6989 | 9458221 | 1177482 | 6.642488 | FP8 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B200 | 1 | 8192 | 1024 | 384 | 2026-02-05 | 1 | 8K/1K | TensorRT LLM | FP4 |  |  | 38147542.61403509 | 65232.29787 | gpt-oss-120b | eagle |  | 1770278400 |  | agg |  | 57984.14 | 7248.162 |  |  | 50.74934 | 0.039427 | 10.41593 |  |  |  | 25.36359 |  |  |  | 1 |  | 488.5011 | 28325314 | 3540735 | 7.860781 | FP8 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B200 | 1 | 1024 | 1024 | 384 | 2026-02-05 | 1 | 1K/1K | TensorRT LLM | FP4 |  |  | 25475204.43859649 | 43562.59959 | gpt-oss-120b | eagle |  | 1770278400 |  | agg |  | 21852.86 | 21709.74 |  |  | 17.088810000000002 | 0.016565999999999997 | 0.14205299999999998 |  |  |  | 60.36553 |  |  |  | 3 |  | 163.0554 | 3563226 | 3539890 | 23.55028 | FP8 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B200 | 1 | 1024 | 1024 | 640 | 2026-02-05 | 1 | 1K/1K | TensorRT LLM | FP4 |  |  | 30405119.023391817 | 51992.75353 | gpt-oss-120b | eagle |  | 1770278400 |  | agg |  | 26090.0 | 25902.75 |  |  | 24.15853 | 0.023439 | 0.180157 |  |  |  | 42.66344 |  |  |  | 2 |  | 227.7071 | 5940879 | 5898240 | 28.10628 | FP8 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 4 | 1024 | 1024 | 256 | 2026-03-27 | 4 | 1K/1K | vLLM | FP4 |  |  | 5956798.322627738 | 8160.813702 | gpt-oss-120b | off |  | 1774594800 |  | agg |  | 4080.406851 | 4080.406851 |  |  | 14.846854230000002 | 0.014211600000000001 | 0.30838764300000004 |  |  |  | 70.36505491 |  | 1 | 1 | 0 | random | 80.30571753 | 1310720 | 1310720 | 15.93908926 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 2 | 1024 | 1024 | 256 | 2026-03-27 | 2 | 1K/1K | vLLM | FP4 |  |  | 7535603.058394161 | 10323.77619 | gpt-oss-120b | off |  | 1774594800 |  | agg |  | 5161.888097 | 5161.888097 |  |  | 25.14870601 | 0.024100329 | 0.49406951400000004 |  |  |  | 41.49320961 |  | 1 | 1 | 0 | random | 126.9612955 | 1310720 | 1310720 | 10.08181269 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B200 | 2 | 8192 | 1024 | 32 | 2026-02-05 | 2 | 8K/1K | TensorRT LLM | FP4 |  |  | 14845451.03508772 | 25385.72127 | gpt-oss-120b | eagle |  | 1770278400 |  | agg |  | 22550.49 | 2835.231 |  |  | 5.5283050000000005 | 0.0053170000000000005 | 0.08912 |  |  |  | 188.0796 |  |  |  | 2 |  | 52.21762 | 2355066 | 296098 | 6.128199 | FP8 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B200 | 1 | 1024 | 1024 | 1024 | 2026-02-05 | 1 | 1K/1K | TensorRT LLM | FP4 |  |  | 35688604.385964915 | 61027.5135 | gpt-oss-120b | eagle |  | 1770278400 |  | agg |  | 30622.66 | 30404.85 |  |  | 33.33883 | 0.032394 | 0.199619 |  |  |  | 30.86977 |  |  |  | 1 |  | 310.4982 | 9508283 | 9440652 | 32.97925 | FP8 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B200 | 4 | 1024 | 1024 | 10 | 2026-02-05 | 4 | 1K/1K | TensorRT LLM | FP4 |  |  | 1456150.0894736843 | 2490.016653 | gpt-oss-120b | eagle |  | 1770278400 |  | agg |  | 1251.396 | 1238.621 |  |  | 1.9160409999999999 | 0.0018360000000000002 | 0.037932 |  |  |  | 544.697 |  |  |  | 3 |  | 18.57066 | 92957 | 92008 | 5.384839 | FP8 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B200 | 2 | 8192 | 1024 | 64 | 2026-02-05 | 2 | 8K/1K | TensorRT LLM | FP4 |  |  | 19929844.795321636 | 34080.0346 | gpt-oss-120b | eagle |  | 1770278400 |  | agg |  | 30303.83 | 3776.202 |  |  | 8.271427 | 0.007989 | 0.098709 |  |  |  | 125.1726 |  |  |  | 1 |  | 78.10109 | 4733525 | 589851 | 8.194507 | FP8 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B200 | 2 | 1024 | 1024 | 64 | 2026-02-05 | 2 | 1K/1K | TensorRT LLM | FP4 |  |  | 8304329.47368421 | 14200.4034 | gpt-oss-120b | eagle |  | 1770278400 |  | agg |  | 7130.409 | 7069.994 |  |  | 4.206697 | 0.004071 | 0.041819 |  |  |  | 245.6255 |  |  |  | 3 |  | 41.70322 | 594722 | 589683 | 15.34654 | FP8 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B200 | 4 | 8192 | 1024 | 10 | 2026-02-05 | 4 | 8K/1K | TensorRT LLM | FP4 |  |  | 5115655.769590644 | 8747.771366 | gpt-oss-120b | eagle |  | 1770278400 |  | agg |  | 7775.448 | 972.3234 |  |  | 2.466698 | 0.002347 | 0.065676 |  |  |  | 426.0686 |  |  |  | 2 |  | 23.57523 | 733232 | 91691 | 4.24174 | FP8 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B200 | 4 | 8192 | 1024 | 8 | 2026-02-05 | 4 | 8K/1K | TensorRT LLM | FP4 |  |  | 4409528.661403509 | 7540.294011 | gpt-oss-120b | eagle |  | 1770278400 |  | agg |  | 6694.076 | 846.2178 |  |  | 2.3138829999999997 | 0.002199 | 0.064514 |  |  |  | 454.7942 |  |  |  | 2 |  | 21.88237 | 585929 | 74069 | 3.655911 | FP8 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B200 | 1 | 8192 | 1024 | 1 | 2026-02-05 | 1 | 8K/1K | TensorRT LLM | FP4 |  |  | 2767628.2128654975 | 4732.644244 | gpt-oss-120b | eagle |  | 1770278400 |  | agg |  | 4204.834 | 527.8099 |  |  | 1.631976 | 0.001501 | 0.096344 |  |  |  | 666.1751 |  |  |  | 3 |  | 17.62566 | 74113 | 9303 | 0.567354 | FP8 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B200 | 4 | 1024 | 1024 | 16 | 2026-02-05 | 4 | 1K/1K | TensorRT LLM | FP4 |  |  | 2124084.11871345 | 3632.183843 | gpt-oss-120b | eagle |  | 1770278400 |  | agg |  | 1830.747 | 1801.437 |  |  | 2.134473 | 0.002048 | 0.039658 |  |  |  | 488.3487 |  |  |  | 3 |  | 20.30837 | 148718 | 146337 | 7.878523 | FP8 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B200 | 1 | 1024 | 1024 | 512 | 2026-02-05 | 1 | 1K/1K | TensorRT LLM | FP4 |  |  | 28498719.292397663 | 48732.80999 | gpt-oss-120b | eagle |  | 1770278400 |  | agg |  | 24448.36 | 24284.45 |  |  | 20.55348 | 0.019942 | 0.15266 |  |  |  | 50.14503 |  |  |  | 2 |  | 194.4037 | 4752851 | 4720987 | 26.33695 | FP8 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B200 | 2 | 1024 | 1024 | 32 | 2026-02-05 | 2 | 1K/1K | TensorRT LLM | FP4 |  |  | 5567275.549122807 | 9520.041189 | gpt-oss-120b | eagle |  | 1770278400 |  | agg |  | 4762.166 | 4757.875 |  |  | 3.213565 | 0.003106 | 0.036338999999999996 |  |  |  | 321.979 |  |  |  | 3 |  | 31.11536 | 296353 | 296086 | 10.28431 | FP8 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B200 | 2 | 1024 | 1024 | 16 | 2026-02-05 | 2 | 1K/1K | TensorRT LLM | FP4 |  |  | 3530984.303508772 | 6037.983159 | gpt-oss-120b | eagle |  | 1770278400 |  | agg |  | 3043.354 | 2994.629 |  |  | 2.610728 | 0.0025150000000000003 | 0.037376 |  |  |  | 397.536 |  |  |  | 3 |  | 24.43324 | 148718 | 146337 | 6.548456 | FP8 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B200 | 2 | 8192 | 1024 | 8 | 2026-02-05 | 2 | 8K/1K | TensorRT LLM | FP4 |  |  | 7020539.50877193 | 12005.12256 | gpt-oss-120b | eagle |  | 1770278400 |  | agg |  | 10657.83 | 1347.288 |  |  | 2.902755 | 0.002764 | 0.075179 |  |  |  | 361.7941 |  |  |  | 2 |  | 27.48818 | 585929 | 74069 | 2.910342 | FP8 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B200 | 2 | 8192 | 1024 | 10 | 2026-02-05 | 2 | 8K/1K | TensorRT LLM | FP4 |  |  | 8068229.777777779 | 13796.67292 | gpt-oss-120b | eagle |  | 1770278400 |  | agg |  | 12263.16 | 1533.514 |  |  | 3.206322 | 0.003058 | 0.077774 |  |  |  | 326.9887 |  |  |  | 2 |  | 29.89572 | 733232 | 91691 | 3.34496 | FP8 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 1 | 8192 | 1024 | 32 | 2026-05-06 | 1 | 8K/1K | Dynamo vLLM | FP4 |  |  | 5176747.134982993 | 7092.1435749267 | Nemotron 3 Super 120B | off |  | 1778050800 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 28.0015287379 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 1 | 8192 | 1024 | 64 | 2026-05-06 | 1 | 8K/1K | Dynamo vLLM | FP4 |  |  | 6345135.088827007 | 8692.835071693 | Nemotron 3 Super 120B | off |  | 1778050800 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 16.5301969423 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 4 | 1024 | 1024 | 512 | 2026-03-27 | 4 | 1K/1K | vLLM | FP4 |  |  | 7589655.394160585 | 10397.82789 | gpt-oss-120b | off |  | 1774594800 |  | agg |  | 5198.913946 | 5198.913946 |  |  | 25.249159329999998 | 0.024208873 | 0.48348214100000003 |  |  |  | 41.30716847 |  | 1 | 1 | 0 | random | 126.0570971 | 2621440 | 2621440 | 20.3082576 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 1 | 8192 | 1024 | 8 | 2026-05-06 | 1 | 8K/1K | Dynamo vLLM | FP4 |  |  | 2988232.65499511 | 4093.8787373433 | Nemotron 3 Super 120B | off |  | 1778050800 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 65.3300512735 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B200 | 1 | 8192 | 1024 | 256 | 2026-05-06 | 1 | 8K/1K | vLLM | NVFP4 |  |  | 15817149.643752575 | 27047.3258908169 | Nemotron 3 Super 120B | off |  | 1778050800 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 14.8658844168 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B200 | 1 | 8192 | 1024 | 32 | 2026-05-06 | 1 | 8K/1K | vLLM | NVFP4 |  |  | 5347550.932021872 | 9144.3120937574 | Nemotron 3 Super 120B | off |  | 1778050800 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 55.0251263419 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B200 | 8 | 8192 | 1024 | 1 | 2026-05-06 | 8 | 8K/1K | vLLM | NVFP4 |  |  | 41514.91393497076 | 70.9905028288 | Nemotron 3 Super 120B | off |  | 1778050800 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 234.1828697817 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B200 | 4 | 1024 | 1024 | 16 | 2026-05-06 | 4 | 1K/1K | vLLM | NVFP4 |  |  | 212517.76575766082 | 363.4053794456 | Nemotron 3 Super 120B | off |  | 1778050800 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 152.6492332397 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B200 | 4 | 1024 | 1024 | 128 | 2026-05-06 | 4 | 1K/1K | vLLM | NVFP4 |  |  | 1343004.1471676608 | 2296.5370916567 | Nemotron 3 Super 120B | off |  | 1778050800 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 87.8400096466 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B200 | 2 | 1024 | 1024 | 512 | 2026-05-06 | 2 | 1K/1K | vLLM | NVFP4 |  |  | 5520810.32491462 | 9440.585655604 | Nemotron 3 Super 120B | off |  | 1778050800 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 26.3495185538 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | GB300 NVL72 |  | 131072 | 8192 | 12 | 2025-10-31 |  | 128K/8K | TensorRT LLM | FP4 |  |  | 11768.867924528302 | 24.95 | DeepSeek-R1 | mtp |  | 1761894000 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 275.4821 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 1 | 1024 | 1024 | 512 | 2026-03-27 | 1 | 1K/1K | vLLM | FP4 |  |  | 11842730.226277372 | 16224.54041 | gpt-oss-120b | off |  | 1774594800 |  | agg |  | 8112.270205 | 8112.270205 |  |  | 63.89037131 | 0.061819658 | 0.648861384 |  |  |  | 16.17608437 |  | 1 | 1 | 0 | random | 323.1450548 | 2621440 | 2621440 | 7.922138872 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 4 | 1024 | 1024 | 128 | 2026-03-27 | 4 | 1K/1K | vLLM | FP4 |  |  | 4099404.340875913 | 5616.183947 | gpt-oss-120b | off |  | 1774594800 |  | agg |  | 2808.091973 | 2808.091973 |  |  | 11.38600505 | 0.010838823 | 0.29788942500000004 |  |  |  | 92.26094265 |  | 1 | 1 | 0 | random | 58.34566729 | 655360 | 655360 | 10.96910927 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 8 | 1024 | 1024 | 16 | 2026-03-27 | 8 | 1K/1K | vLLM | FP4 |  |  | 629618.4547445256 | 862.577283 | gpt-oss-120b | off |  | 1774594800 |  | agg |  | 431.2886415 | 431.2886415 |  |  | 4.736418284 | 0.004496003 | 0.137007163 |  |  |  | 222.4197779 |  | 1 | 1 | 0 | random | 23.7428001 | 81920 | 81920 | 3.369442512 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B200 | 2 | 1024 | 1024 | 128 | 2026-05-06 | 2 | 1K/1K | vLLM | NVFP4 |  |  | 2315016.867045497 | 3958.6788426478 | Nemotron 3 Super 120B | off |  | 1778050800 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 63.0975202207 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B200 | 1 | 1024 | 1024 | 1024 | 2026-05-06 | 1 | 1K/1K | vLLM | NVFP4 |  |  | 7222541.40468655 | 12350.545802014 | Nemotron 3 Super 120B | off |  | 1778050800 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 14.8217520114 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B200 | 8 | 1024 | 1024 | 16 | 2026-05-06 | 8 | 1K/1K | vLLM | NVFP4 |  |  | 110398.13119567253 | 188.7808043446 | Nemotron 3 Super 120B | off |  | 1778050800 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 175.5825758761 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B200 | 4 | 1024 | 1024 | 64 | 2026-05-06 | 4 | 1K/1K | vLLM | NVFP4 |  |  | 740161.0973375438 | 1265.6754764472 | Nemotron 3 Super 120B | off |  | 1778050800 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 107.0933360307 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B200 | 8 | 1024 | 1024 | 1 | 2026-05-06 | 8 | 1K/1K | vLLM | NVFP4 |  |  | 10087.72376380117 | 17.2500076361 | Nemotron 3 Super 120B | off |  | 1778050800 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 223.6048624267 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H100 | 4 | 1024 | 1024 | 32 | 2026-05-06 | 4 | 1K/1K | vLLM | FP8 |  |  | 521927.8067543796 | 715.0410952535 | Nemotron 3 Super 120B | off |  | 1778050800 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 65.7810178646 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 8 | B300 | 8 | 8192 | 1024 | 1197 | 2026-01-28 | 26 | 8K/1K | Dynamo TensorRT LLM | FP4 | 2 | 2 | 7438891.639831789 | 14133.8941156804 | DeepSeek-R1 | mtp | single_turn | 1769587200 | 8 | disagg | 18 | 18146.8913219487 | 5104.6504015766 | 4.3113564939704005 | 3.8760063863751 | 25.350019320612798 | 0.0235543225688 | 3.7911633964395 | 25.982359672289302 | 0.0231368222057 | 4.6798676670795 | 42.4550524464 | 43.2211472738 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B200 | 1 | 1024 | 1024 | 768 | 2026-02-05 | 1 | 1K/1K | TensorRT LLM | FP4 |  |  | 32930397.76023392 | 56310.98017 | gpt-oss-120b | eagle |  | 1770278400 |  | agg |  | 28252.88 | 28058.1 |  |  | 27.14342 | 0.026373 | 0.163626 |  |  |  | 37.91727 |  |  |  | 1 |  | 252.3166 | 7128671 | 7079524 | 30.43795 | FP8 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B200 | 2 | 8192 | 1024 | 16 | 2026-02-05 | 2 | 8K/1K | TensorRT LLM | FP4 |  |  | 10393234.701754386 | 17772.43134 | gpt-oss-120b | eagle |  | 1770278400 |  | agg |  | 15804.16 | 1968.276 |  |  | 3.962087 | 0.003797 | 0.077889 |  |  |  | 263.3748 |  |  |  | 2 |  | 37.17187 | 1174940 | 146329 | 4.30433 | FP8 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H100 | 4 | 1024 | 1024 | 4 | 2026-05-06 | 4 | 1K/1K | vLLM | FP8 |  |  | 135619.27275437958 | 185.7984036735 | Nemotron 3 Super 120B | off |  | 1778050800 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 143.5786861195 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 8 | 8192 | 1024 | 128 | 2026-04-09 | 8 | 8K/1K | Dynamo vLLM | FP4 |  |  | 1072795.4803470804 | 1469.7298080755 | Kimi K2.5/2.6 | off |  | 1775718000 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 11.0832931413 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H100 | 8 | 1024 | 1024 | 4 | 2026-05-06 | 8 | 1K/1K | vLLM | FP8 |  |  | 73527.47475766424 | 100.732640418 | Nemotron 3 Super 120B | off |  | 1778050800 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 171.2552190465 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B200 | 2 | 1024 | 1024 | 4 | 2026-06-24 | 2 | 1K/1K | TensorRT LLM | FP4 |  |  | 474511.42981222225 | 811.4145449789 | Qwen 3.5 | mtp |  | 1782284400 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 226.5558181441 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B200 | 8 | 1024 | 1024 | 128 | 2026-06-24 | 8 | 1K/1K | TensorRT LLM | FP4 |  |  | 2044989.5033012282 | 3496.9320506451 | Qwen 3.5 | mtp |  | 1782284400 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 134.0967949325 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | H200 | 8 | 8192 | 1024 | 1 | 2026-07-14 | 8 | 8K/1K | SGLang | FP4 |  |  | 68342.3419780292 | 93.6290085099 | DeepSeek-V4-Pro | off |  | 1784012400 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 91.5325683267 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B200 | 4 | 8192 | 1024 | 1 | 2026-05-06 | 4 | 8K/1K | vLLM | NVFP4 |  |  | 130492.42864070175 | 223.1420529756 | Nemotron 3 Super 120B | off |  | 1778050800 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 228.006119413 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | B200 | 1 | 8192 | 1024 | 8 | 2026-05-06 | 1 | 8K/1K | vLLM | NVFP4 |  |  | 2260389.7788726315 | 3865.2665218722 | Nemotron 3 Super 120B | off |  | 1778050800 |  | agg |  |  |  |  |  |  |  |  |  |  |  | 109.9829819651 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |

[Download the raw results data (JSON)](https://developer.nvidia.com/search-data/nv_inference_benchmark.json)



[SemiAnalysis InferenceX and AgentX](https://github.com/SemiAnalysisAI/InferenceX) benchmarks, used under the Apache 2.0 license. Explore the [methodology](https://github.com/NVIDIA/srt-slurm-recipes) and learn how to replicate the tests.

## MLPerf Inference v6.1 Performance Benchmarks

## Offline Scenario, Closed Division

| Network | Throughput | GPU | Server | GPU Version | QSL Size | Target Accuracy | Dataset |
| --- | --- | --- | --- | --- | --- | --- | --- |
| DeepSeek R1 | 1,183,327 tokens/sec | 72x VR200† | NVIDIA Vera Rubin NVL72 (72x VR200-288GB\_aarch64) | NVIDIA Vera Rubin NVL72 | 4388 | 99% of FP16 (exact match 81.9132%) | mlperf\_deepseek\_r1 |
| | 2,705,130 tokens/sec | 288x GB300 | NVIDIA GB300 NVL72 (72x GB300-288GB\_aarch64, TensorRT) | NVIDIA GB300 NVL72 | 4388 | 99% of FP16 (exact match 81.9132%) | mlperf\_deepseek\_r1 |
| | 2,091,190 tokens/sec | 288x GB200 | Azure GB200 NVL72 (288x GB200-186GB\_aarch64, TensorRT) | NVIDIA GB200 NVL72 | 4388 | 99% of FP16 (exact match 81.9132%) | mlperf\_deepseek\_r1 |
| | 137,067 tokens/sec | 16x B300 | Cisco\_UCS\_B300x16\_G200\_4x2\_RO | NVIDIA B300 | 4388 | 99% of FP16 (exact match 81.9132%) | mlperf\_deepseek\_r1 |
| | 58,913 tokens/sec | 8x B200 | CoreWeave B200 SXM (8x B200-SXM-180GB) | NVIDIA B200 | 4388 | 99% of FP16 (exact match 81.9132%) | mlperf\_deepseek\_r1 |
| gpt-oss 120B | 1,197,710 tokens/sec | 72x GB300 | CoreWeave GB300 NVL72 (72x GB300-288GB\_aarch64) | NVIDIA GB300 NVL72 | 6396 | 99% of 83.13% | AIME25, GPQA Diamond, LiveCodeBench v6 |
| | 911,566 tokens/sec | 72x GB200 | CoreWeave GB200 NVL72 (72x GB200-186GB\_aarch64) | NVIDIA GB200 NVL72 | 6396 | 99% of 83.13% | AIME25, GPQA Diamond, LiveCodeBench v6 |
| | 220,527 tokens/sec | 16x B300 | Cisco\_UCS\_B300x16\_G200\_4x2\_RO | NVIDIA B300 | 6396 | 99% of 83.13% | AIME25, GPQA Diamond, LiveCodeBench v6 |
| | 91,487 tokens/sec | 8x B200 | CoreWeave B200 SXM (8x B200-SXM-180GB) | NVIDIA B200 | 6396 | 99% of 83.13% | AIME25, GPQA Diamond, LiveCodeBench v6 |
| Qwen3-VL 235B | 2,393 samples/sec | 72x VR200† | NVIDIA Vera Rubin NVL72 (72x VR200-288GB\_aarch64) | NVIDIA Vera Rubin NVL72 | 48289 | 99% of BF16 (Category Hierarchical F1 Score \>= 0.7824) | Shopify Product Catalogue |
| | 1,305 samples/sec | 72x GB300 | NVIDIA GB300 NVL72 (72x GB300-288GB\_aarch64, TensorRT) | NVIDIA GB300 NVL72 | 48289 | 99% of BF16 (Category Hierarchical F1 Score \>= 0.7824) | Shopify Product Catalogue |
| | 230 samples/sec | 16x GB200 | NVIDIA GB200 NVL72 (16x GB200-186GB\_aarch64, Dynamo) | NVIDIA GB200 NVL72 | 48289 | 99% of BF16 (Category Hierarchical F1 Score \>= 0.7824) | Shopify Product Catalogue |
| | 132 samples/sec | 8x B300 | G894-ZD3-AAX7 | NVIDIA B300 | 48289 | 99% of BF16 (Category Hierarchical F1 Score \>= 0.7824) | Shopify Product Catalogue |
| | 102 samples/sec | 8x B200 | Lambda Cloud 8x B200-SXM-180GB | NVIDIA B200 | 48289 | 99% of BF16 (Category Hierarchical F1 Score \>= 0.7824) | Shopify Product Catalogue |
| Llama2 70B | 1,136,100 tokens/sec | 72x GB300 | CoreWeave GB300 NVL72 (72x GB300-288GB\_aarch64) | NVIDIA GB300 NVL72 | 24576 | 99.9% of FP32 (rouge1=44.4312, rouge2=22.0352, rougeL=28.6162). Additionally, the generation length of the tokens per sample should be more than 90% of the reference (tokens\_per\_sample=294.45) | OpenOrca (max\_seq\_len=1024) |
| | 894,002 tokens/sec | 72x GB200 | CoreWeave GB200 NVL72 (72x GB200-186GB\_aarch64) | NVIDIA GB200 NVL72 | 24576 | 99.9% of FP32 (rouge1=44.4312, rouge2=22.0352, rougeL=28.6162). Additionally, the generation length of the tokens per sample should be more than 90% of the reference (tokens\_per\_sample=294.45) | OpenOrca (max\_seq\_len=1024) |
| | 227,570 tokens/sec | 16x B300 | Cisco\_UCS\_B300x16\_G200\_4x2\_RO | NVIDIA B300 | 24576 | 99.9% of FP32 (rouge1=44.4312, rouge2=22.0352, rougeL=28.6162). Additionally, the generation length of the tokens per sample should be more than 90% of the reference (tokens\_per\_sample=294.45) | OpenOrca (max\_seq\_len=1024) |
| | 102,703 tokens/sec | 8x B200 | CoreWeave B200 SXM (8x B200-SXM-180GB) | NVIDIA B200 | 24576 | 99.9% of FP32 (rouge1=44.4312, rouge2=22.0352, rougeL=28.6162). Additionally, the generation length of the tokens per sample should be more than 90% of the reference (tokens\_per\_sample=294.45) | OpenOrca (max\_seq\_len=1024) |
| Llama3.1 8B | 171,114 tokens/sec | 8x B300 | Cisco UCS C880A M8 (8x NVIDIA B300-SXM-270GB, TensorRT) | NVIDIA B300 | 13368 | 99% of FP32 and 99.9% of FP32 (rouge1=42.9865, rouge2=20.1235, rougeL=29.9881). Additionally, for both cases the total generation length of the texts should be more than 90% of the reference (gen\_len=8167644) | CNN Dailymail (v3.0.0, max\_seq\_len=2048) |
| Whisper | 53,267 samples/sec | 8x B300 | NVIDIA DGX B300 (8x B300-SXM-270GB, TensorRT) | NVIDIA B300 | 1633 | 99% of FP32 and 99.9% of FP32 (WER=2.0671%) | LibriSpeech |
| Wan2.2 | 0.655 samples/sec | 72x GB300 | NVIDIA GB300 NVL72 (72x GB300-288GB\_aarch64, TensorRT) | NVIDIA GB300 NVL72 | 248 | 99% of BF16 (VBench score \>= 69.7752) | VBench prompts |
| | 0.496 samples/sec | 72x GB200 | NVIDIA GB200 NVL72 (72x GB200-186GB\_aarch64, TensorRT) | NVIDIA GB200 NVL72 | 248 | 99% of BF16 (VBench score \>= 69.7752) | VBench prompts |
| | 0.078 samples/sec | 8x B300 | NVIDIA DGX B300 (8x B300-SXM-270GB, TensorRT) | NVIDIA B300 | 248 | 99% of BF16 (VBench score \>= 69.7752) | VBench prompts |

### Server Scenario - Closed Division 

| Network | Throughput | GPU | Server | GPU Version | QSL Size | Target Accuracy | MLPerf Server Latency   
Constraints (ms) | Dataset |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| DeepSeek R1 | 1,175,890 tokens/sec | 72x VR200† | NVIDIA Vera Rubin NVL72 (72x VR200-288GB\_aarch64) | NVIDIA Vera Rubin NVL72 | 4388 | 99% of FP16 (exact match 81.9132%) | TTFT/TPOT: 2000 ms/80 ms | mlperf\_deepseek\_r1 |
| | 2,028,030 tokens/sec | 288x GB300 | NVIDIA GB300 NVL72 (72x GB300-288GB\_aarch64, TensorRT) | NVIDIA GB300 NVL72 | 4388 | 99% of FP16 (exact match 81.9132%) | TTFT/TPOT: 2000 ms/80 ms | mlperf\_deepseek\_r1 |
| | 1,598,510 tokens/sec | 288x GB200 | Azure GB200 NVL72 (288x GB200-186GB\_aarch64, TensorRT) | NVIDIA GB200 NVL72 | 4388 | 99% of FP16 (exact match 81.9132%) | TTFT/TPOT: 2000 ms/80 ms | mlperf\_deepseek\_r1 |
| | 88,916 tokens/sec | 16x B300 | Cisco\_UCS\_B300x16\_G200\_4x2\_RO | NVIDIA B300 | 4388 | 99% of FP16 (exact match 81.9132%) | TTFT/TPOT: 2000 ms/80 ms | mlperf\_deepseek\_r1 |
| | 56,121 tokens/sec | 8x B200 | CoreWeave B200 SXM (8x B200-SXM-180GB) | NVIDIA B200 | 4388 | 99% of FP16 (exact match 81.9132%) | TTFT/TPOT: 2000 ms/80 ms | mlperf\_deepseek\_r1 |
| gpt-oss 120B | 1,160,490 tokens/sec | 72x GB300 | CoreWeave GB300 NVL72 (72x GB300-288GB\_aarch64) | NVIDIA GB300 NVL72 | 6396 | 99% of 83.13% | TTFT/TPOT: 3000 ms/80 ms | AIME25, GPQA Diamond, LiveCodeBench v6 |
| | 901,058 tokens/sec | 72x GB200 | CoreWeave GB200 NVL72 (72x GB200-186GB\_aarch64) | NVIDIA GB200 NVL72 | 6396 | 99% of 83.13% | TTFT/TPOT: 3000 ms/80 ms | AIME25, GPQA Diamond, LiveCodeBench v6 |
| | 214,774 tokens/sec | 16x B300 | Cisco\_UCS\_B300x16\_G200\_4x2\_RO | NVIDIA B300 | 6396 | 99% of 83.13% | TTFT/TPOT: 3000 ms/80 ms | AIME25, GPQA Diamond, LiveCodeBench v6 |
| | 90,205 tokens/sec | 8x B200 | CoreWeave B200 SXM (8x B200-SXM-180GB) | NVIDIA B200 | 6396 | 99% of 83.13% | TTFT/TPOT: 3000 ms/80 ms | AIME25, GPQA Diamond, LiveCodeBench v6 |
| Qwen3-VL 235B | 2,323 queries/sec | 72x VR200† | NVIDIA Vera Rubin NVL72 (72x VR200-288GB\_aarch64) | NVIDIA Vera Rubin NVL72 | 48289 | 99% of BF16 (Category Hierarchical F1 Score \>= 0.7824) | 12 s | Shopify Product Catalogue |
| | 1,210 queries/sec | 72x GB300 | NVIDIA GB300 NVL72 (72x GB300-288GB\_aarch64, TensorRT) | NVIDIA GB300 NVL72 | 48289 | 99% of BF16 (Category Hierarchical F1 Score \>= 0.7824) | 12 s | Shopify Product Catalogue |
| | 199 queries/sec | 16x GB200 | NVIDIA GB200 NVL72 (16x GB200-186GB\_aarch64, Dynamo) | NVIDIA GB200 NVL72 | 48289 | 99% of BF16 (Category Hierarchical F1 Score \>= 0.7824) | 12 s | Shopify Product Catalogue |
| | 116 queries/sec | 8x B300 | G894-ZD3-AAX7 | NVIDIA B300 | 48289 | 99% of BF16 (Category Hierarchical F1 Score \>= 0.7824) | 12 s | Shopify Product Catalogue |
| | 69.415 queries/sec | 8x B200 | Lambda Cloud 8x B200-SXM-180GB | NVIDIA B200 | 48289 | 99% of BF16 (Category Hierarchical F1 Score \>= 0.7824) | 12 s | Shopify Product Catalogue |
| Llama2 70B | 944,902 tokens/sec | 72x GB300 | CoreWeave GB300 NVL72 (72x GB300-288GB\_aarch64) | NVIDIA GB300 NVL72 | 24576 | 99.9% of FP32 (rouge1=44.4312, rouge2=22.0352, rougeL=28.6162). Additionally, the generation length of the tokens per sample should be more than 90% of the reference (tokens\_per\_sample=294.45) | TTFT/TPOT: 2000 ms/200 ms | OpenOrca (max\_seq\_len=1024) |
| | 910,621 tokens/sec | 72x GB200 | CoreWeave GB200 NVL72 (72x GB200-186GB\_aarch64) | NVIDIA GB200 NVL72 | 24576 | 99.9% of FP32 (rouge1=44.4312, rouge2=22.0352, rougeL=28.6162). Additionally, the generation length of the tokens per sample should be more than 90% of the reference (tokens\_per\_sample=294.45) | TTFT/TPOT: 2000 ms/200 ms | OpenOrca (max\_seq\_len=1024) |
| | 179,597 tokens/sec | 16x B300 | Cisco\_UCS\_B300x16\_G200\_4x2\_RO | NVIDIA B300 | 24576 | 99.9% of FP32 (rouge1=44.4312, rouge2=22.0352, rougeL=28.6162). Additionally, the generation length of the tokens per sample should be more than 90% of the reference (tokens\_per\_sample=294.45) | TTFT/TPOT: 2000 ms/200 ms | OpenOrca (max\_seq\_len=1024) |
| | 102,398 tokens/sec | 8x B200 | CoreWeave B200 SXM (8x B200-SXM-180GB) | NVIDIA B200 | 24576 | 99.9% of FP32 (rouge1=44.4312, rouge2=22.0352, rougeL=28.6162). Additionally, the generation length of the tokens per sample should be more than 90% of the reference (tokens\_per\_sample=294.45) | TTFT/TPOT: 2000 ms/200 ms | OpenOrca (max\_seq\_len=1024) |
| Llama3.1 8B | 173,950 tokens/sec | 8x B300 | Cisco UCS C880A M8 (8x NVIDIA B300-SXM-270GB, TensorRT) | NVIDIA B300 | 13368 | 99% of FP32 and 99.9% of FP32 (rouge1=42.9865, rouge2=20.1235, rougeL=29.9881). Additionally, for both cases the total generation length of the texts should be more than 90% of the reference (gen\_len=8167644) | TTFT/TPOT: 2000 ms/100 ms | CNN Dailymail (v3.0.0, max\_seq\_len=2048) |
| Wan2.2\*\* | 5.73 seconds | 72x GB200 | NVIDIA GB200 NVL72 (72x GB200-186GB\_aarch64, TensorRT) | NVIDIA GB200 NVL72 | 248 | 99% of BF16 (VBench score \>= 69.7752) | N/A | VBench prompts |
| | 5.81 seconds | 72x GB300 | NVIDIA GB300 NVL72 (72x GB300-288GB\_aarch64, TensorRT) | NVIDIA GB300 NVL72 | 248 | 99% of BF16 (VBench score \>= 69.7752) | N/A | VBench prompts |
| | 17.98 seconds | 8x B300 | NVIDIA DGX B300 (8x B300-SXM-270GB, TensorRT) | NVIDIA B300 | 248 | 99% of BF16 (VBench score \>= 69.7752) | N/A | VBench prompts |

### Interactive Scenario - Closed Division 

| Network | Throughput | GPU | Server | GPU Version | QSL Size | Target Accuracy | MLPerf Server Latency   
Constraints (ms) | Dataset |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| DeepSeek R1 | 652,750 tokens/sec | 72x VR200† | NVIDIA Vera Rubin NVL72 (72x VR200-288GB\_aarch64) | NVIDIA Vera Rubin NVL72 | 4388 | 99% of FP16 (exact match 81.9132%) | TTFT/TPOT: 1500 ms/15 ms | mlperf\_deepseek\_r1 |
| | 260,098 tokens/sec | 72x GB300 | Azure GB300 NVL72 (72x GB300-288GB\_aarch64 TensorRT) | NVIDIA GB300 NVL72 | 4388 | 99% of FP16 (exact match 81.9132%) | TTFT/TPOT: 1500 ms/15 ms | mlperf\_deepseek\_r1 |
| | 240,274 tokens/sec | 72x GB200 | NVIDIA GB200 NVL72 (72x GB200-186GB\_aarch64, TensorRT) | NVIDIA GB200 NVL72 | 4388 | 99% of FP16 (exact match 81.9132%) | TTFT/TPOT: 1500 ms/15 ms | mlperf\_deepseek\_r1 |
| gpt-oss 120B | 669,305 tokens/sec | 72x GB300 | NVIDIA GB300 NVL72 (72x GB300-288GB\_aarch64, TensorRT) | NVIDIA GB300 NVL72 | 6396 | 99% of 83.13% | TTFT/TPOT: 2000 ms/20 ms | AIME25, GPQA Diamond, LiveCodeBench v6 |
| | 625,205 tokens/sec | 72x GB200 | NVIDIA GB200 NVL72 (72x GB200-186GB\_aarch64, TensorRT) | NVIDIA GB200 NVL72 | 6396 | 99% of 83.13% | TTFT/TPOT: 2000 ms/20 ms | AIME25, GPQA Diamond, LiveCodeBench v6 |
| Qwen3-VL 235B | 1,307 queries/sec | 72x VR200† | NVIDIA Vera Rubin NVL72 (72x VR200-288GB\_aarch64) | NVIDIA Vera Rubin NVL72 | 48289 | 99% of BF16 (Category Hierarchical F1 Score \>= 0.7824) | 1.5 s | Shopify Product Catalogue |
| | 349 queries/sec | 72x GB300 | NVIDIA GB300 NVL72 (72x GB300-288GB\_aarch64, TensorRT) | NVIDIA GB300 NVL72 | 48289 | 99% of BF16 (Category Hierarchical F1 Score \>= 0.7824) | 1.5 s | Shopify Product Catalogue |
| | 39.851 queries/sec | 16x GB200 | NVIDIA GB200 NVL72 (16x GB200-186GB\_aarch64, Dynamo) | NVIDIA GB200 NVL72 | 48289 | 99% of BF16 (Category Hierarchical F1 Score \>= 0.7824) | 1.5 s | Shopify Product Catalogue |
| | 12.882 queries/sec | 8x B300 | G894-ZD3-AAX7 | NVIDIA B300 | 48289 | 99% of BF16 (Category Hierarchical F1 Score \>= 0.7824) | 1.5 s | Shopify Product Catalogue |
| Llama3.1 8B | 129,168 tokens/sec | 8x B300 | Cisco UCS C880A M8 (8x NVIDIA B300-SXM-270GB, TensorRT) | NVIDIA B300 | 13368 | 99% of FP32 and 99.9% of FP32 (rouge1=42.9865, rouge2=20.1235, rougeL=29.9881). Additionally, for both cases the total generation length of the texts should be more than 90% of the reference (gen\_len=8167644) | TTFT/TPOT: 500 ms/30 ms | CNN Dailymail (v3.0.0, max\_seq\_len=2048) |

\*\*The primary metric on Wan2.2 in Server Scenario is measured in seconds (lower the better).   
 †Vera Rubin NVL72 results are preview submissions.   
 MLPerf™ v6.1 Inference Closed Division. NVIDIA platform results from the following entries: 6.1-0008, 6.1-0010, 6.1-0014, 6.1-0015, 6.1-0021, 6.1-0023, 6.1-0025, 6.1-0028, 6.1-0046, 6.1-0064, 6.1-0071, 6.1-0072, 6.1-0073, 6.1-0074, 6.1-0104, 6.1-0106. MLPerf name and logo are trademarks. See [https://mlcommons.org/](https://mlcommons.org/) for more information.   
 For MLPerf™ various scenario data, click [here](https://github.com/mlcommons/inference_policies/blob/master/inference_rules.adoc#scenarios)  
 For MLPerf™ latency constraints, click [here](https://github.com/mlcommons/inference_policies/blob/master/inference_rules.adoc#constraints-for-the-closed-division)

## Inference Performance of NVIDIA Data Center Products

## B200 Inference Performance

| Network | Batch Size | Throughput | Efficiency | Latency (ms) | GPU | Server | Container | Precision | Dataset | Framework | GPU Version |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| BEVFusion Head | 1 | 2,634 images/sec | 5.52 images/sec/watt | 0.38 | 1x B200 | DGX B200 | 26.08-py3 | INT8 | Synthetic | TensorRT | NVIDIA B200 |
| ControlNet | 4 | 6.28 images/sec | - | 636.55 | 1x B200 | DGX B200 | 26.07-py3 | Mixed | Synthetic | TensorRT | NVIDIA B200 |
| HF Swin Base | 64 | 4,886 samples/sec | 5.34 samples/sec/watt | 13.10 | 1x B200 | DGX B200 | 26.08-py3 | FP8 | Synthetic | TensorRT | NVIDIA B200 |
| HF Swin Large | 128 | 3,127 samples/sec | 3.21 samples/sec/watt | 40.93 | 1x B200 | DGX B200 | 26.08-py3 | FP8 | Synthetic | TensorRT | NVIDIA B200 |
| HF ViT Base | 2048 | 9,366 samples/sec | 9.84 samples/sec/watt | 218.66 | 1x B200 | DGX B200 | 26.07-py3 | FP8 | Synthetic | TensorRT | NVIDIA B200 |
| HF ViT Large | 2048 | 3,364 samples/sec | 3.54 samples/sec/watt | 608.87 | 1x B200 | DGX B200 | 26.08-py3 | FP8 | Synthetic | TensorRT | NVIDIA B200 |
| Stable Diffusion XL | 4 | 2.07 images/sec | - | 1936.19 | 1x B200 | DGX B200 | 26.07-py3 | Mixed | Synthetic | TensorRT | NVIDIA B200 |
| Stable Video Diffusion | 1 | 7.00 videos/min | - | 8570.96 | 1x B200 | DGX B200 | 26.07-py3 | Mixed | Synthetic | TensorRT | NVIDIA B200 |
| Yolo v10 M | 1 | 865 images/sec | 1.07 images/sec/watt | 1.16 | 1x B200 | DGX B200 | 26.08-py3 | INT8 | Synthetic | TensorRT | NVIDIA B200 |
| Yolo v11 M | 1 | 1,079 images/sec | 1.32 images/sec/watt | 0.93 | 1x B200 | DGX B200 | 26.08-py3 | INT8 | Synthetic | TensorRT | NVIDIA B200 |
| Yolo v11 S | 1 | 1,695 images/sec | 2.34 images/sec/watt | 0.59 | 1x B200 | DGX B200 | 26.08-py3 | INT8 | Synthetic | TensorRT | NVIDIA B200 |

HF Swin Base, HF Swin Large, HF ViT Base, HF ViT Large Sequence Length = 384

### RTX PRO 6000 Blackwell Server Edition Inference Performance 

| Network | Batch Size | Throughput | Efficiency | Latency (ms) | GPU | Server | Container | Precision | Dataset | Framework | GPU Version |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| BEVFusion Head | 1 | 1,759 images/sec | 4.57 images/sec/watt | 0.57 | 1x RTX PRO 6000 | Supermicro SYS-521GE-TNRT | 26.07-py3 | FP8 | Synthetic | TensorRT | RTX PRO 6000 BSE |
| ControlNet | 4 | 2.77 images/sec | - | 1442.47 | 1x RTX PRO 6000 | Supermicro SYS-521GE-TNRT | 26.07-py3 | Mixed | Synthetic | TensorRT | RTX PRO 6000 BSE |
| Flux Image Generator | 1 | 0.20 images/sec | - | 5012.06 | 1x RTX PRO 6000 | Supermicro SYS-521GE-TNRT | 26.07-py3 | FP4 | Synthetic | TensorRT | RTX PRO 6000 BSE |
| HF Swin Base | 32 | 2,713 samples/sec | 4.58 samples/sec/watt | 11.79 | 1x RTX PRO 6000 | Supermicro SYS-521GE-TNRT | 26.07-py3 | FP8 | Synthetic | TensorRT | RTX PRO 6000 BSE |
| HF Swin Large | 32 | 1,512 samples/sec | 2.52 samples/sec/watt | 21.17 | 1x RTX PRO 6000 | Supermicro SYS-521GE-TNRT | 26.07-py3 | FP8 | Synthetic | TensorRT | RTX PRO 6000 BSE |
| HF ViT Base | 512 | 3,426 samples/sec | 5.70 samples/sec/watt | 149.45 | 1x RTX PRO 6000 | Supermicro SYS-521GE-TNRT | 26.07-py3 | FP8 | Synthetic | TensorRT | RTX PRO 6000 BSE |
| HF ViT Large | 1024 | 1,165 samples/sec | 2.00 samples/sec/watt | 878.99 | 1x RTX PRO 6000 | Supermicro SYS-521GE-TNRT | 26.07-py3 | FP8 | Synthetic | TensorRT | RTX PRO 6000 BSE |
| Stable Diffusion XL | 4 | 0.72 images/sec | - | 5577.68 | 1x RTX PRO 6000 | Supermicro SYS-521GE-TNRT | 26.07-py3 | Mixed | Synthetic | TensorRT | RTX PRO 6000 BSE |
| Stable Video Diffusion | 1 | 2.82 videos/min | - | 21294.19 | 1x RTX PRO 6000 | Supermicro SYS-521GE-TNRT | 26.07-py3 | Mixed | Synthetic | TensorRT | RTX PRO 6000 BSE |
| Yolo v10 M | 1 | 448 images/sec | 0.83 images/sec/watt | 2.23 | 1x RTX PRO 6000 | Supermicro SYS-521GE-TNRT | 26.07-py3 | FP8 | Synthetic | TensorRT | RTX PRO 6000 BSE |
| Yolo v11 M | 1 | 466 images/sec | 0.86 images/sec/watt | 2.15 | 1x RTX PRO 6000 | Supermicro SYS-521GE-TNRT | 26.07-py3 | FP8 | Synthetic | TensorRT | RTX PRO 6000 BSE |
| Yolo v11 S | 1 | 942 images/sec | 1.79 images/sec/watt | 1.06 | 1x RTX PRO 6000 | Supermicro SYS-521GE-TNRT | 26.07-py3 | FP8 | Synthetic | TensorRT | RTX PRO 6000 BSE |

HF Swin Base, HF Swin Large, HF ViT Base, HF ViT Large Sequence Length = 384

### H100 Inference Performance 

| Network | Batch Size | Throughput | Efficiency | Latency (ms) | GPU | Server | Container | Precision | Dataset | Framework | GPU Version |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| BEVFusion Head | 1 | 2,015 images/sec | 6.21 images/sec/watt | 0.50 | 1x H100 | DGX H100 | 26.08-py3 | INT8 | Synthetic | TensorRT | H100 80GB HBM3 |
| ControlNet | 4 | 3.54 images/sec | - | 1129.35 | 1x H100 | DGX H100 | 26.07-py3 | Mixed | Synthetic | TensorRT | H100 80GB HBM3 |
| Flux Image Generator | 1 | 0.20 images/sec | - | 5048.89 | 1x H100 | DGX H100 | 26.07-py3 | FP8 | Synthetic | TensorRT | H100 80GB HBM3 |
| HF Swin Base | 128 | 2,971 samples/sec | 4.40 samples/sec/watt | 43.19 | 1x H100 | DGX H100 | 26.07-py3 | FP8 | Synthetic | TensorRT | H100 80GB HBM3 |
| HF Swin Large | 128 | 1,834 samples/sec | 2.66 samples/sec/watt | 70.02 | 1x H100 | DGX H100 | 26.07-py3 | FP8 | Synthetic | TensorRT | H100 80GB HBM3 |
| HF ViT Base | 2048 | 4,992 samples/sec | 7.20 samples/sec/watt | 410.28 | 1x H100 | DGX H100 | 26.07-py3 | FP8 | Synthetic | TensorRT | H100 80GB HBM3 |
| HF ViT Large | 512 | 1,739 samples/sec | 2.51 samples/sec/watt | 299.82 | 1x H100 | DGX H100 | 26.07-py3 | FP8 | Synthetic | TensorRT | H100 80GB HBM3 |
| Stable Diffusion XL | 4 | 0.98 images/sec | - | 4060.83 | 1x H100 | DGX H100 | 26.07-py3 | Mixed | Synthetic | TensorRT | H100 80GB HBM3 |
| Stable Video Diffusion | 1 | 3.76 videos/min | - | 15964.16 | 1x H100 | DGX H100 | 26.07-py3 | Mixed | Synthetic | TensorRT | H100 80GB HBM3 |
| Yolo v10 M | 1 | 407 images/sec | 0.68 images/sec/watt | 2.47 | 1x H100 | DGX H100 | 26.08-py3 | FP8 | Synthetic | TensorRT | H100 80GB HBM3 |
| Yolo v11 M | 1 | 478 images/sec | 0.79 images/sec/watt | 2.10 | 1x H100 | DGX H100 | 26.07-py3 | FP8 | Synthetic | TensorRT | H100 80GB HBM3 |
| Yolo v11 S | 1 | 948 images/sec | 1.91 images/sec/watt | 1.26 | 1x H100 | DGX H100 | 26.08-py3 | INT8 | Synthetic | TensorRT | H100 80GB HBM3 |

HF Swin Base, HF Swin Large, HF ViT Base, HF ViT Large Sequence Length = 384

### L40S Inference Performance 

| Network | Batch Size | Throughput | Efficiency | Latency (ms) | GPU | Server | Container | Precision | Dataset | Framework | GPU Version |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| BEVFusion Head | 1 | 1,953 images/sec | 6.89 images/sec/watt | 0.51 | 1x L40S | Supermicro SYS-521GE-TNRT | 26.08-py3 | INT8 | Synthetic | TensorRT | NVIDIA L40S |
| ControlNet | 4 | 1.59 images/sec | - | 2520.76 | 1x L40S | Supermicro SYS-521GE-TNRT | 26.07-py3 | Mixed | Synthetic | TensorRT | NVIDIA L40S |
| Flux Image Generator | 1 | 0.08 images/sec | - | 12505.61 | 1x L40S | Supermicro SYS-521GE-TNRT | 26.07-py3 | FP8 | Synthetic | TensorRT | NVIDIA L40S |
| HF Swin Base | 32 | 1,392 samples/sec | 4.17 samples/sec/watt | 23.29 | 1x L40S | Supermicro SYS-521GE-TNRT | 26.07-py3 | FP8 | Synthetic | TensorRT | NVIDIA L40S |
| HF Swin Large | 32 | 703 samples/sec | 2.16 samples/sec/watt | 45.80 | 1x L40S | Supermicro SYS-521GE-TNRT | 26.07-py3 | FP8 | Synthetic | TensorRT | NVIDIA L40S |
| HF ViT Base | 1024 | 1,661 samples/sec | 4.81 samples/sec/watt | 616.43 | 1x L40S | Supermicro SYS-521GE-TNRT | 26.07-py3 | FP8 | Synthetic | TensorRT | NVIDIA L40S |
| HF ViT Large | 512 | 592 samples/sec | 1.71 samples/sec/watt | 870.93 | 1x L40S | Supermicro SYS-521GE-TNRT | 26.07-py3 | FP8 | Synthetic | TensorRT | NVIDIA L40S |
| Stable Diffusion XL | 1 | 0.36 images/sec | - | 2738.24 | 1x L40S | Supermicro SYS-521GE-TNRT | 26.07-py3 | Mixed | Synthetic | TensorRT | NVIDIA L40S |
| Stable Video Diffusion | 1 | 1.34 videos/min | - | 44632.28 | 1x L40S | Supermicro SYS-521GE-TNRT | 26.07-py3 | Mixed | Synthetic | TensorRT | NVIDIA L40S |
| Yolo v10 M | 1 | 286 images/sec | 0.83 images/sec/watt | 3.64 | 1x L40S | Supermicro SYS-521GE-TNRT | 26.08-py3 | INT8 | Synthetic | TensorRT | NVIDIA L40S |
| Yolo v11 M | 1 | 328 images/sec | 0.95 images/sec/watt | 3.22 | 1x L40S | Supermicro SYS-521GE-TNRT | 26.08-py3 | INT8 | Synthetic | TensorRT | NVIDIA L40S |
| Yolo v11 S | 1 | 803 images/sec | 2.34 images/sec/watt | 1.35 | 1x L40S | Supermicro SYS-521GE-TNRT | 26.08-py3 | INT8 | Synthetic | TensorRT | NVIDIA L40S |

HF Swin Base, HF Swin Large, HF ViT Base, HF ViT Large Sequence Length = 384

## View More Performance Data

### Training to Convergence  

Deploying AI in real-world applications requires training networks to convergence at a specified accuracy. This is the best methodology to test whether AI systems are ready to be deployed in the field to deliver meaningful results.

[Learn More](/deep-learning-performance-training-inference/training)

### AI Pipeline  

NVIDIA Riva is an application framework for multimodal conversational AI services that deliver real-performance on GPUs.

[Learn More](/deep-learning-performance-training-inference/conversational-ai)

## Frequently Asked Questions About Performance of NVIDIA Data Center Deep Learning Products 

Only looking at compute pricing or FLOPs per dollar gives an incomplete view of inference TCO. The most important metric for AI inference TCO is cost per token, or the price-performance actually delivered. GB300 NVL72 delivers AI inference at $0.123 per million tokens at 116 TPS/user interactivity using Dynamo and TensorRT-LLM — the lowest cost per token among major platforms according to [SemiAnalysis InferenceX benchmark](https://inferencex.semianalysis.com/)s as of April 2026.

| Metric | NVIDIA Hopper (HGX H200) | NVIDIA Blackwell (GB300 NVL72) | NVIDIA Blackwell Relative to Hopper |
| --- | --- | --- | --- |
| Cost per GPU per Hour ($) | $1.41 | $2.65 | 2x |
| FLOP per Dollar (PFLOPS) | 2.8 | 5.6 | 2x |
| Tokens per Second per GPU | 90 | 6,000 | 65x |
| Tokens per Second per MW | 54K | 2.8M | 50x |
| Cost per Million Tokens ($) | $4.20 | $0.12 | 35x lower |

GB300 NVL72 delivers AI inference at $0.123 per million tokens at 116 TPS/user interactivity using Dynamo and TensorRT-LLM — the lowest cost per token among major platforms according to [SemiAnalysis InferenceX benchmarks](https://inferencex.semianalysis.com/) as of April 2026.

NVIDIA inference cost per million tokens has improved dramatically across generations: NVIDIA Blackwell Ultra (GB300 NVL72) delivers up to 50x higher throughput per megawatt and up to 35x lower cost per token than NVIDIA Hopper for low-latency agentic workloads, through hardware–software codesign, according to [SemiAnalysis InferenceX benchmarks](https://inferencex.semianalysis.com/) (Q1 2026). Software optimization drives continuous improvement—GB200 token output improved 4x in three months, resulting in a proportional decrease in token cost.

NVIDIA's TensorRT-LLM and Dynamo software stack delivers continuous inference cost improvements without hardware changes. NVIDIA Blackwell B200 cost per million tokens dropped from $0.11 at launch to $0.02 on GPT-OSS-120B within two months, according to [SemiAnalysis InferenceX benchmarks](https://inferencex.semianalysis.com/) as of April 2026—a 5x improvement from software alone. Each TensorRT-LLM release typically delivers throughput gains through kernel fusion, quantization improvements, and scheduling optimizations.