AI applications at the edge often need to run multiple neural networks within limited compute and power budgets. NVIDIA Deep Learning Accelerator (DLA) platforms provide dedicated hardware for deep learning inference. On NVIDIA Jetson AGX Orin, DLA executes supported neural network operations, allowing applications to offload inference work from the GPU. This can improve power efficiency and leave GPU capacity available for other application tasks.
NVIDIA cuDLA is the runtime API for managing execution on this accelerator. Applications use it to create DLA devices, load compiled networks, register memory buffers, and submit inference tasks. The cuda.bindings.cudla module exposes these APIs directly to Python, so developers can integrate DLA execution into their applications without writing C++ code to manage the runtime.
This post walks through the inference lifecycle on Jetson AGX Orin using an image classification example. It briefly covers installation and compiling a model into a DLA loadable, then follows execution from creating a device and loading the network to managing buffers, synchronizing execution, and reading the output. The example also shows how GPU and DLA stages work together in a single Python application.
How cuDLA fits within a Python application
DLA is a fixed-function accelerator. This means that it efficiently performs fixed operations, such as convolutions and pooling, that are common in modern neural network architectures. It does not have as many supported layers as the GPU, but it still supports a wide variety of layers used in many popular neural network architectures.. To learn more about when and why to use DLA, see Getting Started with the Deep Learning Accelerator on NVIDIA Jetson Orin.
Introduced in cuda-bindings 13.3.0 as part of the CUDA Python project, cuda.bindings.cudla is a thin, Cython-generated Python layer over the cuDLA C++ API for hybrid-mode workflows. It exposes thirteen functions, nine data types, eight enums for hybrid-mode workflows, where CUDA streams coordinate DLA execution with GPU work. On Jetson Orin, NVIDIA JetPack supplies the underlying libcudla.so runtime.
The Python bindings expose that runtime directly to your application, while inference executes on DLA hardware. This complements the existing C++ workflow by allowing runtime management and inference orchestration to remain in Python. Figure 1 shows where cuda.bindings.cudla fits in the stack.
While this post focuses on Jetson, NVIDIA RTX Spark brings DLA inference to mainstream Windows AI developer laptops for the first time, extending cuDLA well beyond the embedded and robotics space. The cuda.bindings.cudla API is identical on both platforms, so everything in this post related to cuDLA APIs applies directly to RTX Spark as well.
cuda.bindings.cudlaPrerequisites and installation
If you are on an NVIDIA Jetson Orin running JetPack, libcudla.so ships with the JetPack installation and is automatically discoverable by the bindings. There is nothing additional to install for the DLA runtime itself—no separate cuDLA SDK and no extra runtime libraries.
Install the cuDLA Python bindings:
pip install "cuda-bindings>=13.3.0"
The JetPack default Python may be older than 3.10, which is the minimum required version. Install Python 3.12 from the deadsnakes PPA if needed:
sudo apt update && sudo apt install -y software-properties-common
sudo add-apt-repository -y ppa:deadsnakes/ppa
sudo apt update && sudo apt install -y python3.12 python3.12-venv python3.12-dev
python3.12 -m venv ~/dla-env
~/dla-env/bin/pip install "cuda-bindings>=13.3.0"
You only need TensorRT to compile a trained model into a DLA loadable. To run an existing loadable, you only need libcudla.so, which ships with JetPack.
Compiling a DLA loadable
A DLA loadable is the binary artifact produced by the TensorRT trtexec tool. It encodes a trained network in a format DLA hardware can execute directly. This compilation example targets TensorRT 11.4.1 with CUDA 13.4 Update 1 on Jetson AGX Orin running JetPack 7.2.2 or later. For an FP16 CHW16 loadable, using an FP16-typed ONNX model with fixed input dimensions, use the following:
trtexec \
--onnx=your_model_fp16.onnx \
--useDLACore=0 \
--inputIOFormats=chw16 \
--outputIOFormats=chw16 \
--memPoolSize=dlaSRAM:1 \
--saveEngine=model.bin \
--buildDLAStandalone
The key flags are:
--useDLACore=0targets the first DLA engine on Orin--buildDLAStandaloneproduces a raw DLA loadable that does not require TensorRT at inference time- I/O format options select CHW16 layout
TensorRT 11 derives precision from the model’s tensor types; --fp16 and --int8 are no longer supported. This example assumes fixed input dimensions, so --shapes is omitted.
This reflects the sample’s current FP16 path and TensorRT 11 flag changes.
Not every network layer is DLA-compatible, and preprocessing the ONNX graph to trim DLA-incompatible head and tail layers is often necessary. The Python workflow in the next section works with any .bin or .nvdla file you produce. Figure 2 shows the split between the one-time build step and the on-device runtime.
The Python workflow
For the CUDA and cuDLA calls, three imports cover everything:
import numpy as np
from cuda.bindings import cudla
from cuda.bindings import driver, runtime
cudla is the new module. driver and runtime are the existing cuda-python bindings for the CUDA driver and runtime APIs—used here for stream management, memory allocation, and data transfers. Everything runs through a single CUDA stream, the coordination point for DLA and GPU work in hybrid mode.
Step 1: Initialize CUDA and create a DLA device
cudla.Mode.CUDA_DLA selects hybrid mode, where you submit DLA tasks through a CUDA stream. This enables GPU-DLA pipelines: preprocessing kernels, DLA inference, and postprocessing kernels can all be enqueued on the same stream and executed in order.
driver.cuInit(0)
runtime.cudaSetDevice(0)
cudla_device = cudla.create_device(0, int(cudla.Mode.CUDA_DLA))
This release does not support standalone mode (which requires the NvSci buffer and sync primitives). create_device raises CudlaError(ErrorUnsupportedOperation) if you request it.
Step 2: Load the loadable
Pass the loadable as a Python bytes object directly to module_load_from_memory. No file path or filesystem handle—just raw bytes, equally easy to load from memory-mapped files, embedded resources, or network streams.
with open("model.bin", "rb") as f:
loadable_data = f.read()
module_handle = cudla.module_load_from_memory(
cudla_device, loadable_data, len(loadable_data), 0
)
Step 3: Introspect the loaded module
Before allocating any memory, query the module to find out exactly what tensors it expects:
num_inputs = cudla.module_get_attributes(
module_handle, int(cudla.ModuleAttributeType.NUM_INPUT_TENSORS)
)
num_outputs = cudla.module_get_attributes(
module_handle, int(cudla.ModuleAttributeType.NUM_OUTPUT_TENSORS)
)
input_descs = cudla.module_get_attributes(
module_handle, int(cudla.ModuleAttributeType.INPUT_TENSOR_DESCRIPTORS)
)
output_descs = cudla.module_get_attributes(
module_handle, int(cudla.ModuleAttributeType.OUTPUT_TENSOR_DESCRIPTORS)
)
Each descriptor is a cudla.ModuleTensorDescriptor with properties for buffer size (size_), tensor shape (n, c, h, w), name, data format, and stride. All subsequent allocation calls are driven by desc.size_ with no hardcoded sizes anywhere in the application. This introspection step is what makes the workflow model-agnostic: the same code paths handle any loadable regardless of network architecture.
Step 4: Allocate GPU memory
input_gpu = runtime.cudaMalloc(input_descs[0].size_)[1]
output_gpu = runtime.cudaMalloc(output_descs[0].size_)[1]
CUDA driver and runtime calls return a status code as the first element of a tuple. The full sample wraps every such call in a helper that checks the returned status and raises on failure—the snippets here elide that for readability, but error checking is essential in production code. For models with multiple inputs or outputs, iterate over input_descs and output_descs and allocate one buffer per tensor. The size_ field on each descriptor gives the exact byte count.
Step 5: Register GPU buffers with cuDLA
This is the step unique to cuDLA. GPU buffers must be registered with the DLA device before they can appear in a task. Registration returns a DLA-side handle that the hardware uses to access the buffer:
input_registered = cudla.mem_register(
cudla_device, int(input_gpu), input_descs[0].size_, 0
)
output_registered = cudla.mem_register(
cudla_device, int(output_gpu), output_descs[0].size_, 0
)
The int() cast extracts the raw pointer from the CUDA memory object. cuda.bindings returns opaque handle objects, and int() is how you pass them to C-level APIs that expect a pointer.
Step 6: Prepare input data and copy to GPU
Preprocessing—normalizing values, matching the expected tensor stride and data format—is the only model-specific code in the script. The tensor descriptor stride, data_format, and data_type fields tell you exactly how to lay out the data. For a multi-input model, repeat the copy for each input buffer. Everything else before and after this block is identical regardless of what network is inside the loadable.
stream = driver.cuStreamCreate(driver.CUstream_flags.CU_STREAM_NON_BLOCKING)[1]
# Preprocess your input into a numpy array `input_cpu`
# matching the tensor layout described by input_descs[0] ...
runtime.cudaMemcpyAsync(
input_gpu, input_cpu, input_descs[0].size_,
runtime.cudaMemcpyKind.cudaMemcpyHostToDevice, stream
)
runtime.cudaMemsetAsync(output_gpu, 0, output_descs[0].size_, stream)
Step 7: Build the task and submit
cudla.Task wraps the cudlaTask C struct. Assigning a Python list to task.input_tensor or task.output_tensor automatically updates the corresponding count field. submit_task enqueues the inference work onto the CUDA stream, sequencing it with any GPU kernels before or after it on the same stream.
task = cudla.Task()
task.module_handle = module_handle
task.input_tensor = [int(input_registered)]
task.output_tensor = [int(output_registered)]
task.wait_events = 0
task.signal_events = 0
cudla.submit_task(cudla_device, int(task), 1, int(stream), 0)
Step 8: Synchronize and read back
The first sync waits for the DLA task to complete. Allocating output_cpu as a numpy array of the descriptor’s byte size gives the D2H copy a destination buffer; the second sync ensures the transfer is done before you read from it.
runtime.cudaStreamSynchronize(stream)
output_cpu = np.empty(output_descs[0].size_, dtype=np.uint8)
runtime.cudaMemcpyAsync(
output_cpu, output_gpu, output_descs[0].size_,
runtime.cudaMemcpyKind.cudaMemcpyDeviceToHost, stream
)
runtime.cudaStreamSynchronize(stream)
Step 9: Cleanup
Every handle created has a matching destroy call—DLA device, module, registered memory handles, GPU allocations, and stream.
cudla.mem_unregister(cudla_device, input_registered)
cudla.mem_unregister(cudla_device, output_registered)
cudla.module_unload(module_handle, 0)
cudla.destroy_device(cudla_device)
runtime.cudaFree(input_gpu)
runtime.cudaFree(output_gpu)
driver.cuStreamDestroy(stream)
Error handling is native to Python
All cuDLA API calls raise cudla.CudlaError on failure rather than returning error codes. The exception carries both the numeric status and its human-readable name:
try:
device = cudla.create_device(0, int(cudla.Mode.CUDA_DLA))
except cudla.CudlaError as e:
print(e) # e.g. "ErrorInvalidDevice (8)"
DLA workflows fit naturally into standard Python exception handling. I recommend try/finally for cleanup. DLA device handles are opaque C resources and will not release themselves on garbage collection. Branch on e.status against cudla.Status enum values to handle specific failure modes, from parameter errors (ErrorInvalidParam) up to hardware-level faults (ErrorDlaErrEngineTimeout).
What is the resulting output?
After cudaStreamSynchronize returns, output_cpu holds the raw DLA output for your network as a numpy array, ready for any standard Python postprocessing. What you do with it depends entirely on the model, including class scores, feature maps, bounding box coordinates. The cuDLA Python layer hands you the bytes. Interpreting them is the same postprocessing you would write in any Python inference pipeline, and you can chain it directly with NumPy, PyTorch, or OpenCV without leaving the Python process.
To make this concrete: for a ResNet-50 backbone compiled as a DLA loadable, the sample output would look like this:
cuDLA device created: 0x55a1b2c30000
Module loaded: 0x55a5b6c70000
numInputTensors = 1
numOutputTensors = 1
Input 0: size=..., shape=(1,64,56,56), name=resnetv17_pool0_fwd
Output 0: size=..., shape=(1,2048,7,7), name=resnetv17_stage4_activation2
SUBMIT IS DONE (task queued; completion pending)
Output tensor written to host
The sample output tells you the exact buffer sizes and tensor names the loadable expects, which your preprocessing and postprocessing code can read directly rather than hardcoding them. After synchronization, output_cpu is a flat numpy array; interpret or reshape it using the data_type, data_format, stride, and shape fields from output_descs[0] and pass it to whatever downstream stage your pipeline requires.
The workflow from create_device to cudaStreamSynchronize is the same regardless of network architecture. Only the tensor names, sizes, and postprocessing change.
Final steps
The cuda.bindings.cudla module ships in cuda-bindings 13.3.0. Install it, compile a loadable with trtexec, and the nine steps above take you from a .bin file to DLA inference results directly in Python on the same board.
pip install "cuda-bindings>=13.3.0"
The unit tests for enums, POD types, and API surface ship in the NVIDIA/cuda-python GitHub repo under cuda_bindings/tests/cudla/ and run entirely without DLA hardware—no libcudla.so required:
pytest cuda_bindings/tests/cudla/ -v
Get started with cuDLA in Python
For a complete end-to-end inference example, sample inputs, and scripts to prepare the model and compile a DLA loadable locally, refer to the Deep Learning Accelerator samples.
To learn more, visit the NVIDIA/cuda-python GitHub repo.
