DriveWorks SDK Reference
5.14.77 Release
For Test and Development only

DNN Interface

Detailed Description

Defines Deep Neural Network (DNN) module for performing inference using NVIDIA® TensorRT models.

Data Structures

Modules

 DNN Plugin
 Provides an interface for supporting non-standard DNN layers.
 

Typedefs

typedef struct dwDNNObject const * dwConstDNNHandle_t
 
typedef struct dwDNNObject * dwDNNHandle_t
 Handles representing Deep Neural Network interface. More...
 

Functions

DW_API_PUBLIC dwStatus dwDNN_getCUDAStream (cudaStream_t *const stream, dwDNNHandle_t const network)
 Gets the CUDA stream used by the feature list. More...
 
DW_API_PUBLIC dwStatus dwDNN_getInputBlobCount (uint32_t *const count, dwDNNHandle_t const network)
 Gets the input blob count. More...
 
DW_API_PUBLIC dwStatus dwDNN_getInputIndex (uint32_t *const blobIndex, const char8_t *const blobName, dwDNNHandle_t const network)
 Gets the index of an input blob with a given blob name. More...
 
DW_API_PUBLIC dwStatus dwDNN_getInputSize (dwBlobSize *const blobSize, uint32_t const blobIndex, dwDNNHandle_t const network)
 Gets the input blob size at blobIndex. More...
 
DW_API_PUBLIC dwStatus dwDNN_getInputTensorProperties (dwDNNTensorProperties *const tensorProps, uint32_t const blobIndex, dwDNNHandle_t const network)
 Gets the input tensor properties at blobIndex. More...
 
DW_API_PUBLIC dwStatus dwDNN_getMetaData (dwDNNMetaData *const metaData, dwDNNHandle_t const network)
 Returns the metadata for the associated network model. More...
 
DW_API_PUBLIC dwStatus dwDNN_getOutputBlobCount (uint32_t *const count, dwDNNHandle_t const network)
 Gets the output blob count. More...
 
DW_API_PUBLIC dwStatus dwDNN_getOutputIndex (uint32_t *const blobIndex, const char8_t *const blobName, dwDNNHandle_t const network)
 Gets the index of an output blob with a given blob name. More...
 
DW_API_PUBLIC dwStatus dwDNN_getOutputSize (dwBlobSize *const blobSize, uint32_t const blobIndex, dwDNNHandle_t const network)
 Gets the output blob size at blobIndex. More...
 
DW_API_PUBLIC dwStatus dwDNN_getOutputTensorProperties (dwDNNTensorProperties *const tensorProps, uint32_t const blobIndex, dwDNNHandle_t const network)
 Gets the output tensor properties at blobIndex. More...
 
DW_API_PUBLIC dwStatus dwDNN_infer (dwDNNTensorHandle_t *const outputTensors, uint32_t const outputTensorCount, dwConstDNNTensorHandle_t *const inputTensors, uint32_t const inputTensorCount, dwDNNHandle_t const network)
 Runs inference pipeline on the given input. More...
 
DW_API_PUBLIC dwStatus dwDNN_inferRaw (float32_t *const *const dOutput, const float32_t *const *const dInput, uint32_t const batchsize, dwDNNHandle_t const network)
 Forwards pass from all input blobs to all output blobs. More...
 
DW_API_PUBLIC dwStatus dwDNN_inferSIO (float32_t *const dOutput, const float32_t *const dInput, uint32_t const batchsize, dwDNNHandle_t const network)
 Forwards pass from the first input blob to the first output blob (a shortcut for a single input - single output network). More...
 
DW_API_PUBLIC dwStatus dwDNN_initializeTensorRTFromFile (dwDNNHandle_t *const network, const char8_t *const modelFilename, const dwDNNPluginConfiguration *const pluginConfiguration, dwProcessorType const processorType, dwContextHandle_t const context)
 Creates and initializes a TensorRT Network from file. More...
 
DW_API_PUBLIC dwStatus dwDNN_initializeTensorRTFromFileWithEngineId (dwDNNHandle_t *const network, const char8_t *const modelFilename, const dwDNNPluginConfiguration *const pluginConfiguration, dwProcessorType const processorType, uint32_t engineId, dwContextHandle_t const context)
 Creates and initializes a TensorRT Network from file with DLA Engine ID. More...
 
DW_API_PUBLIC dwStatus dwDNN_initializeTensorRTFromMemory (dwDNNHandle_t *const network, const char8_t *const modelContent, uint32_t const modelContentSize, const dwDNNPluginConfiguration *const pluginConfiguration, dwProcessorType const processorType, dwContextHandle_t const context)
 Creates and initializes a TensorRT Network from memory. More...
 
DW_API_PUBLIC dwStatus dwDNN_initializeTensorRTFromMemoryWithEngineId (dwDNNHandle_t *const network, const char8_t *const modelContent, uint32_t const modelContentSize, const dwDNNPluginConfiguration *const pluginConfiguration, dwProcessorType const processorType, uint32_t engineId, dwContextHandle_t const context)
 Creates and initializes a TensorRT Network from memory with DLA Engine ID. More...
 
DW_API_PUBLIC dwStatus dwDNN_release (dwDNNHandle_t const network)
 Releases a given network. More...
 
DW_API_PUBLIC dwStatus dwDNN_reset (dwDNNHandle_t const network)
 Resets a given network. More...
 
DW_API_PUBLIC dwStatus dwDNN_setCUDAStream (cudaStream_t const stream, dwDNNHandle_t const network)
 Sets the CUDA stream for infer operations. More...
 

Data Structure Documentation

◆ dwDNNMetaData

struct dwDNNMetaData
Data Fields
dwDataConditionerParams dataConditionerParams DataConditioner parameters for running this network.

Typedef Documentation

◆ dwConstDNNHandle_t

typedef struct dwDNNObject const* dwConstDNNHandle_t

Definition at line 64 of file DNN.h.

◆ dwDNNHandle_t

typedef struct dwDNNObject* dwDNNHandle_t

Handles representing Deep Neural Network interface.

Definition at line 63 of file DNN.h.

Function Documentation

◆ dwDNN_getCUDAStream()

DW_API_PUBLIC dwStatus dwDNN_getCUDAStream ( cudaStream_t *const  stream,
dwDNNHandle_t const  network 
)

Gets the CUDA stream used by the feature list.

Parameters
[out]streamThe CUDA stream currently used.
[in]networkA handle to the DNN module.
Returns
DW_INVALID_HANDLE if the given network handle or the stream are NULL or of wrong type.
DW_SUCCESS otherwise.

◆ dwDNN_getInputBlobCount()

DW_API_PUBLIC dwStatus dwDNN_getInputBlobCount ( uint32_t *const  count,
dwDNNHandle_t const  network 
)

Gets the input blob count.

Parameters
[out]countA pointer to the number of input blobs.
[in]networkNetwork handle created with dwDNN_initialize().
Returns
DW_INVALID_ARGUMENT - if provided network handle or count are NULL.
DW_SUCCESS otherwise.

◆ dwDNN_getInputIndex()

DW_API_PUBLIC dwStatus dwDNN_getInputIndex ( uint32_t *const  blobIndex,
const char8_t *const  blobName,
dwDNNHandle_t const  network 
)

Gets the index of an input blob with a given blob name.

Parameters
[out]blobIndexA pointer to the index of the blob with the given name.
[in]blobNameA pointer to the name of an input blob.
[in]networkNetwork handle created with dwDNN_initialize().
Returns
DW_INVALID_ARGUMENT - if provided network handle, blobIndex or blobName are null or an input blob with the given name is not found.
DW_SUCCESS otherwise.

◆ dwDNN_getInputSize()

DW_API_PUBLIC dwStatus dwDNN_getInputSize ( dwBlobSize *const  blobSize,
uint32_t const  blobIndex,
dwDNNHandle_t const  network 
)

Gets the input blob size at blobIndex.

Parameters
[out]blobSizeA pointer to the where the input blob size is returned.
[in]blobIndexSpecifies the blob index; must be in the range [0 dwDNN_getInputBlobCount()-1].
[in]networkNetwork handle created with dwDNN_initialize().
Returns
DW_INVALID_ARGUMENT - if provided network handle or the blob size are NULL or blobIndex is not in range [0 dwDNN_getInputBlobCount()-1].
DW_SUCCESS otherwise.

◆ dwDNN_getInputTensorProperties()

DW_API_PUBLIC dwStatus dwDNN_getInputTensorProperties ( dwDNNTensorProperties *const  tensorProps,
uint32_t const  blobIndex,
dwDNNHandle_t const  network 
)

Gets the input tensor properties at blobIndex.

Parameters
[out]tensorPropsTensor properties.
[in]blobIndexSpecifies the blob index; must be in the range [0 dwDNN_getInputBlobCount()-1].
[in]networkNetwork handle created with dwDNN_initialize().
Returns
DW_INVALID_ARGUMENT - if provided network handle or the blob size are NULL or blobIndex is not in range [0 dwDNN_getInputBlobCount()-1].
DW_SUCCESS otherwise.

◆ dwDNN_getMetaData()

DW_API_PUBLIC dwStatus dwDNN_getMetaData ( dwDNNMetaData *const  metaData,
dwDNNHandle_t const  network 
)

Returns the metadata for the associated network model.

Parameters
[out]metaDataA pointer to a metadata structure.
[in]networkNetwork handle created with dwDNN_initialize().
Returns
DW_INVALID_ARGUMENT - if provided network handle or metaData are NULL.
DW_SUCCESS otherwise.

◆ dwDNN_getOutputBlobCount()

DW_API_PUBLIC dwStatus dwDNN_getOutputBlobCount ( uint32_t *const  count,
dwDNNHandle_t const  network 
)

Gets the output blob count.

Parameters
[out]countA pointer to the number of output blobs.
[in]networkNetwork handle created with dwDNN_initialize().
Returns
DW_INVALID_ARGUMENT - if provided network handle or count are NULL.
DW_SUCCESS otherwise.

◆ dwDNN_getOutputIndex()

DW_API_PUBLIC dwStatus dwDNN_getOutputIndex ( uint32_t *const  blobIndex,
const char8_t *const  blobName,
dwDNNHandle_t const  network 
)

Gets the index of an output blob with a given blob name.

Parameters
[out]blobIndexA pointer to the index of the blob with the given name.
[in]blobNameA pointer to the name of an output blob.
[in]networkNetwork handle created with dwDNN_initialize().
Returns
DW_INVALID_HANDLE - if provided network handle, blobIndex or blobName are NULL or an output blob with the given name is not found or the network handle is of wrong type
DW_SUCCESS otherwise.

◆ dwDNN_getOutputSize()

DW_API_PUBLIC dwStatus dwDNN_getOutputSize ( dwBlobSize *const  blobSize,
uint32_t const  blobIndex,
dwDNNHandle_t const  network 
)

Gets the output blob size at blobIndex.

Parameters
[out]blobSizeA pointer to the where the output blob size is returned.
[in]blobIndexSpecifies the blob index; must be in the range [0 dwDNN_getOutputBlobCount()-1].
[in]networkNetwork handle created with dwDNN_initialize().
Returns
DW_INVALID_ARGUMENT - if provided network handle is or blobSize are NULL or blobIndex is not in range [0 dwDNN_getOutputBlobCount()-1].
DW_SUCCESS otherwise.

◆ dwDNN_getOutputTensorProperties()

DW_API_PUBLIC dwStatus dwDNN_getOutputTensorProperties ( dwDNNTensorProperties *const  tensorProps,
uint32_t const  blobIndex,
dwDNNHandle_t const  network 
)

Gets the output tensor properties at blobIndex.

Parameters
[out]tensorPropsTensor properties.
[in]blobIndexSpecifies the blob index; must be in the range [0 dwDNN_getOutputBlobCount()-1].
[in]networkNetwork handle created with dwDNN_initialize().
Returns
DW_INVALID_ARGUMENT - if provided network handle is or blobSize are NULL or blobIndex is not in range [0 dwDNN_getOutputBlobCount()-1].
DW_SUCCESS otherwise.

◆ dwDNN_infer()

DW_API_PUBLIC dwStatus dwDNN_infer ( dwDNNTensorHandle_t *const  outputTensors,
uint32_t const  outputTensorCount,
dwConstDNNTensorHandle_t *const  inputTensors,
uint32_t const  inputTensorCount,
dwDNNHandle_t const  network 
)

Runs inference pipeline on the given input.

Parameters
[out]outputTensorsOutput tensors.
[in]outputTensorCountNumber of output tensors.
[in]inputTensorsInput tensors.
[in]inputTensorCountNumber of input tensors.
[in]networkNetwork handle created with dwDNN_initialize().
Returns
DW_INVALID_ARGUMENT - if provided network handle, input or output are NULL.
DW_SUCCESS otherwise.

◆ dwDNN_inferRaw()

DW_API_PUBLIC dwStatus dwDNN_inferRaw ( float32_t *const *const  dOutput,
const float32_t *const *const  dInput,
uint32_t const  batchsize,
dwDNNHandle_t const  network 
)

Forwards pass from all input blobs to all output blobs.

Note
: The size of the output blob and input blob must match the respective sizes returned by dwDNN_getOutputSize() and dwDNN_getInputSize().
: The umber of blobs must match the respective number of blobs returned by dwDNN_getInputBlobCount() and dwDNN_getOutputBlobCount().
Parameters
[out]dOutputA pointer to an array of pointers to the input blobs in GPU Memory.
[in]dInputA pointer to an array of pointers to the output blobs in GPU Memory.
[in]batchsizeBatch size for inference. Batch size must be equal or less than the one that was given at the time of model generation.
[in]networkNetwork handle created with dwDNN_initialize().
Returns
DW_INVALID_ARGUMENT - if provided network handle , output or input are NULL.
DW_INTERNAL_ERROR - if DNN engine cannot execute inference on the given network.
DW_SUCCESS otherwise.
Note
Inference performance might be suboptimal if the network has been generated for a larger batch size.
If the processor type is DLA, the inference will always perform on maximum batch size. The only advantage of giving a lower batchsize as input is that the output to be copied will be smaller in size.

◆ dwDNN_inferSIO()

DW_API_PUBLIC dwStatus dwDNN_inferSIO ( float32_t *const  dOutput,
const float32_t *const  dInput,
uint32_t const  batchsize,
dwDNNHandle_t const  network 
)

Forwards pass from the first input blob to the first output blob (a shortcut for a single input - single output network).

Note
This method requires the network to be single input - single output.
The size of the output blob and input blob must match the respective sizes returned by dwDNN_getOutputSize() and dwDNN_getInputSize().
Parameters
[out]dOutputA pointer to the output blob in GPU memory.
[in]dInputA pointer to the input blob in GPU memory.
[in]batchsizeBatch size for inference. Batch size must be equal or less than the one that was given at the time of model generation.
[in]networkNetwork handle created with dwDNN_initialize().
Returns
DW_INVALID_ARGUMENT - if provided network handle, output or input are NULL or if provided network is not a single input - single output network.
DW_INTERNAL_ERROR - if DNN engine cannot execute inference on the given network.
DW_SUCCESS otherwise.
Note
Inference performance might be suboptimal if the network has been generated for a larger batch size.
If the processor type is DLA, the inference will always perform on maximum batch size. The only advantage of giving a lower batchsize as input is that the output to be copied will be smaller in size.

◆ dwDNN_initializeTensorRTFromFile()

DW_API_PUBLIC dwStatus dwDNN_initializeTensorRTFromFile ( dwDNNHandle_t *const  network,
const char8_t *const  modelFilename,
const dwDNNPluginConfiguration *const  pluginConfiguration,
dwProcessorType const  processorType,
dwContextHandle_t const  context 
)

Creates and initializes a TensorRT Network from file.

Parameters
[out]networkA pointer to network handle that will be initialized from parameters.
[in]modelFilenameA pointer to the name of the TensorRT model file.
[in]pluginConfigurationAn optional pointer to plugin configuration for custom layers.
[in]processorTypeProcessor that the inference should run on. Note that the model must be generated for this processor type.
[in]contextSpecifies the handle to the context under which the DNN module is created.
Returns
DW_INVALID_ARGUMENT - if pointer to the network handle or the model filename are NULL.
DW_DNN_INVALID_MODEL - if the provided model is invalid.
DW_CUDA_ERROR - if compute capability does not met the network type's requirements.
DW_FILE_NOT_FOUND - if given model file does not exist.
DW_SUCCESS otherwise.
Note
The network file must be created by the TensorRT_optimization tool.
DNN module will look for metadata file named <modelFilename>.json in the same folder. If it is present, metadata will be loaded from that file. Otherwise, it will be filled with default values. Example metadata:
{
    "dataConditionerParams" : {
        "meanValue" : [0.0, 0.0, 0.0],
        "splitPlanes" : true,
        "pixelScaleCoefficient": 1.0,
        "ignoreAspectRatio" : false,
        "doPerPlaneMeanNormalization" : false
    }
    "tonemapType" : "none",
    "__comment": "tonemapType can be one of {none, agtm}"
}
\

◆ dwDNN_initializeTensorRTFromFileWithEngineId()

DW_API_PUBLIC dwStatus dwDNN_initializeTensorRTFromFileWithEngineId ( dwDNNHandle_t *const  network,
const char8_t *const  modelFilename,
const dwDNNPluginConfiguration *const  pluginConfiguration,
dwProcessorType const  processorType,
uint32_t  engineId,
dwContextHandle_t const  context 
)

Creates and initializes a TensorRT Network from file with DLA Engine ID.

Parameters
[out]networkA pointer to network handle that will be initialized from parameters.
[in]modelFilenameA pointer to the name of the TensorRT model file.
[in]pluginConfigurationAn optional pointer to plugin configuration for custom layers.
[in]processorTypeProcessor that the inference should run on. Note that the model must be generated for this processor type.
[in]engineIdSpecifies the DLA engine id if the processorType is DW_PROCESSOR_TYPE_CUDLA.
[in]contextSpecifies the handle to the context under which the DNN module is created.
Returns
DW_INVALID_ARGUMENT - if pointer to the network handle or the model filename are NULL.
DW_DNN_INVALID_MODEL - if the provided model is invalid.
DW_CUDA_ERROR - if compute capability does not met the network type's requirements.
DW_FILE_NOT_FOUND - if given model file does not exist.
DW_SUCCESS otherwise.

◆ dwDNN_initializeTensorRTFromMemory()

DW_API_PUBLIC dwStatus dwDNN_initializeTensorRTFromMemory ( dwDNNHandle_t *const  network,
const char8_t *const  modelContent,
uint32_t const  modelContentSize,
const dwDNNPluginConfiguration *const  pluginConfiguration,
dwProcessorType const  processorType,
dwContextHandle_t const  context 
)

Creates and initializes a TensorRT Network from memory.

Parameters
[out]networkA pointer to network handle that is initialized from parameters.
[in]modelContentA pointer to network content in the memory.
[in]modelContentSizeSpecifies the size of network content in memory, in bytes.
[in]pluginConfigurationAn optional pointer to plugin configuration for custom layers.
[in]processorTypeProcessor that the inference should run on. Note that the model must be generated for this processor type.
[in]contextSpecifies a handle to the context under which the DNN module is created.
Returns
DW_INVALID_ARGUMENT - if pointer to the network handle or the model content are NULL.
DW_DNN_INVALID_MODEL - if the provided model is invalid.
DW_CUDA_ERROR - if compute capability does not met the network type's requirements.
DW_SUCCESS otherwise.
Note
The network file must be created by the TensorRT_optimization tool.
DNN module will fill metadata with default values.

◆ dwDNN_initializeTensorRTFromMemoryWithEngineId()

DW_API_PUBLIC dwStatus dwDNN_initializeTensorRTFromMemoryWithEngineId ( dwDNNHandle_t *const  network,
const char8_t *const  modelContent,
uint32_t const  modelContentSize,
const dwDNNPluginConfiguration *const  pluginConfiguration,
dwProcessorType const  processorType,
uint32_t  engineId,
dwContextHandle_t const  context 
)

Creates and initializes a TensorRT Network from memory with DLA Engine ID.

Parameters
[out]networkA pointer to network handle that is initialized from parameters.
[in]modelContentA pointer to network content in the memory.
[in]modelContentSizeSpecifies the size of network content in memory, in bytes.
[in]pluginConfigurationAn optional pointer to plugin configuration for custom layers.
[in]processorTypeProcessor that the inference should run on. Note that the model must be generated for this processor type.
[in]engineIdSpecifies the DLA engine id if the processorType is DW_PROCESSOR_TYPE_CUDLA.
[in]contextSpecifies a handle to the context under which the DNN module is created.
Returns
DW_INVALID_ARGUMENT - if pointer to the network handle or the model content are NULL.
DW_DNN_INVALID_MODEL - if the provided model is invalid.
DW_CUDA_ERROR - if compute capability does not met the network type's requirements.
DW_SUCCESS otherwise.

◆ dwDNN_release()

DW_API_PUBLIC dwStatus dwDNN_release ( dwDNNHandle_t const  network)

Releases a given network.

Parameters
[in]networkThe network handle to release.
Returns
DW_INVALID_ARGUMENT - if provided network handle is NULL.
DW_SUCCESS otherwise.

◆ dwDNN_reset()

DW_API_PUBLIC dwStatus dwDNN_reset ( dwDNNHandle_t const  network)

Resets a given network.

Parameters
[in]networkNetwork handle to reset.
Returns
DW_INVALID_ARGUMENT - if provided network handle is null.
DW_SUCCESS otherwise.

◆ dwDNN_setCUDAStream()

DW_API_PUBLIC dwStatus dwDNN_setCUDAStream ( cudaStream_t const  stream,
dwDNNHandle_t const  network 
)

Sets the CUDA stream for infer operations.

Note
The ownership of the stream remains by the callee.
Parameters
[in]streamThe CUDA stream to be used. Default is the stream 0, resulting in synchronous operations.
[in]networkA handle to the DNN module to set CUDA stream for.
Returns
DW_INVALID_ARGUMENT if the given network handle is NULL.
DW_SUCCESS otherwise.