1. [Home](/) Nsight Perf SDK # NVIDIA Nsight Perf SDK Explore GPU performance profiling for graphics applications and long-running AI workloads. # Choose Your Workflow Always-On AI Profiling Graphics Profiling For long-running, distributed AI workloads, NVIDIA Nsight™ Perf SDK for AI preserves selected GPU and CPU performance metrics and execution context from the original run so developers can investigate intermittent issues later. _Coming Soon_ [![Nsight Perf SDK always-on profiling captures a slowdown 3 hours and 14 minutes into a run. A preserved 120 ms trace shows GPU 2 arriving late, causing the other GPUs to wait at an NCCL AllReduce operation.](https://developer.download.nvidia.com/images/nsight-perf/ai-profiling-hero-1920x1007.svg)](https://developer.download.nvidia.com/images/nsight-perf/ai-profiling-hero-1920x1007.svg) _Click Image to Expand Preserve performance evidence when an issue occurs—not only when it can be reproduced later. Nsight Perf SDK keeps a focused record from the original run so teams can investigate intermittent behavior and determine where deeper profiling is needed._ ## Preserve the Evidence Run the collection layer alongside the workload so selected performance metrics and execution context from intermittent behavior are available after the event. ## Remain Alongside the Workload Collect a focused performance record with a layer designed to remain alongside sustained workloads, without the overhead and data volume of conventional full-detail capture. ## Narrow the Next Investigation Use evidence from the original run to identify the interval and workload activity that merit deeper analysis with Nsight Systems or Nsight Compute. ## Preserve Evidence From the Original Run [![Traditional profiling requires reruns that may fail to reproduce an issue, consuming hours of cluster time. NVIDIA Nsight Perf SDK continuously saves CUDA/GPU profiling evidence for analysis in Google Perfetto and targeted investigation with Nsight Systems or Nsight Compute.](https://developer.download.nvidia.com/images/nsight-perf/preserve-original-run-1920x1063.svg) ](https://developer.download.nvidia.com/images/nsight-perf/preserve-original-run-1920x1063.svg) _Click Image to Expand_ Traditional profiling works when a workload can be paused or replayed and an issue can be reproduced while a detailed profiler is attached. Long-running, distributed training and inference workloads make that workflow difficult. Intermittent behavior may disappear, and recreating the full environment can be costly or impractical. Nsight Perf SDK provides an always-on NVIDIA CUDA™/GPU collection layer that preserves selected performance metrics and execution context from the original run. It is a focused record—not a full system trace or kernel profile. ## Analyze the Original Run in Google Perfetto Google Perfetto is the supported analysis and visualization experience available today for data collected through Nsight Perf SDK. Inspect selected performance metrics and execution context on a timeline, identify the interval surrounding an intermittent event, and narrow the next investigation. [ ![Google Perfetto timeline showing GPU clocks, activity, DRAM bandwidth, PCIe bandwidth, and process context across four GPUs, with aligned activity dips and bandwidth spikes above a GPU event trace.](https://developer.download.nvidia.com/images/nsight-perf/perfetto-gpu-metrics-1920x1213.png)](https://developer.download.nvidia.com/images/nsight-perf/perfetto-gpu-metrics-1920x1213.png) _Click Image to Expand_ ## Move From Signal to Deeper Profiling [ ](https://developer.download.nvidia.com/images/nsight-perf/preserve-original-run-1920x1063.svg)[ ![Google Perfetto trace correlating CPU training operations with GPU execution. A flow arrow links an asynchronous host-to-device memory copy call to its GPU operation, with a preceding-flow delay of approximately 411 ms.](https://developer.download.nvidia.com/images/nsight-perf/deeper-profiling-trace-1920x1375.png)](https://developer.download.nvidia.com/images/nsight-perf/deeper-profiling-trace-1920x1375.png) _Click Image to Expand_ Monitoring systems such as NVIDIA DCGM and OpenTelemetry can signal that a system may be unhealthy. Nsight Perf SDK preserves performance evidence tied to the workload activity that produced it. Use that evidence to investigate what happened in the original run and determine whether deeper system-wide tracing or kernel-level analysis is needed. NVIDIA Nsight Systems provides deeper system-wide analysis; NVIDIA Nsight Compute provides detailed kernel-level analysis. Nsight Perf SDK complements these tools rather than replacing them. Nsight Perf SDK supplies an always-on CUDA/GPU collection layer between workload activity and the compatible tools developers use to investigate it. It complements monitoring and deeper profiling tools rather than replacing them. ### NVIDIA Nsight Compute Provides detailed performance metrics and analysis for individual CUDA kernels after the investigation has narrowed to kernel-level behavior. [Learn More](/nsight-compute) ### NVIDIA Nsight Systems Provides deeper, focused system-wide tracing across CPUs, GPUs, operating-system activity, and application libraries after an investigation has narrowed. [Learn More](/nsight-systems) ### Google Perfetto The supported analysis and visualization experience available today. Use its timeline to inspect selected performance metrics and execution context collected from the original run. [Learn More](https://perfetto.dev/) [Learn More About Nsight Tools](/tools-overview)[Browse Tutorial](http://developer.nvidia.com/tools-tutorials) NVIDIA Nsight™ Perf SDK for Graphics is a profiling toolbox for DirectX, Vulkan, and OpenGL that enables you to collect GPU performance metrics directly from your application. [Get Started ](https://developer.nvidia.com/nsight-perf-sdk/get-started "Download Now") ![C++ code integrating NVIDIA Nsight Perf instrumentation into a Direct3D 12 application, configuring GPU metrics, initializing report generation, and setting frame profiling ranges and GPU clocks.](https://d29g4g2dyqv443.cloudfront.net/sites/default/files/akamai/gamedev/Nsight_Perf_ProductPage_Hero_Updated.jpg) _Just a few lines of code are needed to set up GPU performance metrics collection with the Nsight Perf SDK._ [![GPU metrics panel showing a 31.9 ms frame, 94.2% graphics activity, 74.1% SM activity, 4.7% PCI throughput, and 26.6% VRAM throughput. An engine activity chart shows sustained high graphics activity and low synchronous and asynchronous copy engine activity.](https://developer.download.nvidia.com/images/nsight-perf/custom-metrics-1920x1077.png)](https://developer.download.nvidia.com/images/nsight-perf/custom-metrics-1920x1077.png) ## Real-time Perf Triage Enable high-level performance triage via real-time collection and on-screen visualization of GPU performance metrics. The new GPU Periodic Sampler collects device-level metrics at high sampling rates with low overhead. [![PIX event list showing NVIDIA GPU performance counters alongside Direct3D 12 rendering events, with columns for duration, raster culling, active cycles, and shader stalls, plus a counter selection panel.](https://developer.download.nvidia.com/images/nsight-perf/pix-integration-1920x777.png)](https://developer.download.nvidia.com/images/nsight-perf/pix-integration-1920x777.png) _Microsoft’s PIX on Windows showingNVIDIA GPU performance metrics_ ## Profile In-Application Integrate GPU performance metric collection into your application or graphics developer tool of choice. Activate profiling from your own custom programmatic triggers. Choose the list of GPU metrics to collect, customize your output, and keep control over your workflow ![Stylized pipeline diagram with arrows connecting green and blue processing stages, a puzzle-shaped component entering from above, and a return path looping from the final stage to the first.](https://developer.download.nvidia.com/images/nsight-perf/pipeline-metrics-1920x1076.png) ## Real-time Perf Triage Generate detailed profiler reports on every developer and artist change. Add dedicated perf regression criteria by inspecting GPU metric values. * * * ## Real-Time Performance HUD [![Real-time GPU performance HUD over a rendered 3D scene, displaying raster data flow counters and graphs alongside workload statistics for draw calls, primitives, vertices, compute dispatches, and CTA launches.](https://developer.download.nvidia.com/images/nsight-perf/real-time-hud-1920x1078.png)](https://developer.download.nvidia.com/images/nsight-perf/real-time-hud-1920x1078.png) _Click Image to Expand_ Add continuous performance metrics collection to your code, and leverage the built-in HUD renderer to effortlessly enable real-time, high-level performance triage. Explore panels with metrics on SM, L2 cache, ROP, VRAM, and various other subunits to gain an early understanding of the performance characteristics and potential bottlenecks of the scene as you move through it. The HUD- and Periodic Sampler-utility classes also serve as an example for creating your own powerful, low-overhead, real-time workflows on top of the low-level Nsight Perf SDK API. https://www.youtube-nocookie.com/embed/0gpoWXpOadA?&controls=0&showinfo=0&rel=0 * * * ## Timeline Viewer Examine a snapshot of your application’s performance with the Nsight Perf SDK one-shot sampling mode. This allows you to examine hardware activity with minimal overhead. Nsight Perf SDK’s high-frequency sampling collects key GPU metrics in sharp detail. You can visualize unit throughputs, warp occupancy, draw calls, and more on the new Timeline Viewer. [![NVIDIA Nsight Perf timeline viewer correlating frame events with GPU activity, hardware unit throughput, compute activity, draw and dispatch starts, and SM warp occupancy across a shared time axis.](https://developer.download.nvidia.com/images/nsight-perf/timeline-viewer-1920x1080.png)](https://developer.download.nvidia.com/images/nsight-perf/timeline-viewer-1920x1080.png) _Click Image to Expand_ https://www.youtube-nocookie.com/embed/HHjTvFbfZy0?&controls=0&showinfo=0&rel=0 * * * ## HTML Profiler Report Generator [![HTML profiler report comparing SM instruction throughput by pipeline, with instruction counts, per-SM IPC, and percentage of peak. FMA leads at 57.4% of peak, followed by load/store at 18.4% and transcendental/conversion operations at 17.7%.](https://developer.download.nvidia.com/images/nsight-perf/html-profiler-report-1920x929.png)](https://developer.download.nvidia.com/images/nsight-perf/html-profiler-report-1920x929.png) _Click Image to Expand_ Generate detailed profiler reports with minimal effort. Simply insert a few calls at Graphics API Device Initialization, Present/SwapBuffers, a Keypress handler, or an automated trigger. Insert annotations (PushRange/PopRange) around GPU workloads to collect additional reports per region of execution. The report generator automatically collects 100s of GPU metrics of interest; there is no need to study these complex topics on first usage. The reports provide a top-down representation of GPU performance, with fast navigation to the top performance limiters. Quickly determine the workload type, pipeline activity and utilization, shader latency reasons, and 3D data flow. * * * ## View Other Tools Within the Nsight Suite Nsight Perf SDK is part of the NVIDIA Nsight Developer Tools suite, a collection of powerful tools, libraries, and SDKs that enable developers to build, debug, and profile software utilizing the latest accelerated computing hardware. ![decorative icon](https://developer.download.nvidia.com/images/nvidia-nsight-graphics-icon-gbp-shaded-128.png) ### Nsight Graphics NVIDIA Nsight Graphics is a standalone developer tool with ray-tracing support that enables you to debug, profile, and export frames built with Direct3D, Vulkan, OpenGL, OpenVR, and the Oculus SDK. [Get Started](/nsight-graphics) ![img-alt-text](https://developer.download.nvidia.com/images/nvidia-nsight-compute-icon-gbp-shaded-128.png) ### Nsight Compute Nsight Compute is an interactive kernel profiler for CUDA applications. It provides detailed performance metrics and API debugging via a user interface and command-line tool. It also provides a customizable, data-driven user interface and metric collection that can be extended with analysis scripts for post-processing results.. [Get Started](https://developer.nvidia.com/nsight-compute) ![decorative icon](https://developer.download.nvidia.com/images/nvidia-nsight-systems-icon-gbp-shaded-256.png) ### Nsight Systems NVIDIA Nsight Systems is a system-wide performance analysis tool designed to visualize an application’s algorithms, help you identify the largest opportunities to optimize, and tune to scale efficiently across any quantity or size of CPUs and GPUs. [Get Started](/nsight-systems) * * * ## Partners and Industry Standards [ ![Direct X logo](https://developer.download.nvidia.com/images/nsight-perf/DirectX-NSP.jpg) ](https://docs.microsoft.com/en-us/windows/win32/direct3d12/direct3d-12-graphics) [ ![Microsoft logo](https://developer.download.nvidia.com/images/nsight-perf/Microsoft-NSP.jpg) ](https://aka.ms/pixonwindows) [ ![Khronos Group logo](https://developer.download.nvidia.com/images/nsight-perf/Khronos_Group_NSP.jpg) ](https://www.khronos.org/) [ ![Vulkan logo](https://developer.download.nvidia.com/images/nsight-perf/Vulkan_NSP.jpg) ](https://www.khronos.org/vulkan/) [ ![Open GL logo](https://developer.download.nvidia.com/images/nsight-perf/OpenGL_NSP.jpg) ](https://www.khronos.org/opengl/) * * * ## NVIDIA Nsight Tools News * * * ## Resources - [NVIDIA Developer Forum - Nsight Perf SDK](https://forums.developer.nvidia.com/c/development-tools/nsight-perf-sdk/368) - [Nsight Graphics](/nsight-graphics) - [Nsight Systems](/nsight-systems) - [GTC 2022 Session: GPU Performance Analysis and Improvements utilizing the NVIDIA Nsight Perf SDK](https://www.nvidia.com/en-us/on-demand/session/gtcspring22-s41859/) [Learn More About Nsight Tools ](/tools-overview "Download Now")[Browse Tutorials](/tools-tutorials "Go to Documentation")