Vulkan SC#

Introduction#

This feature demonstrates how to integrate Vulkan SC with NvSciStream, including importing NvSciBuf / NvSciSync objects as VkImage / VkFence, or VkSemaphore, which is used as a render target / synchronization primitive in the rendering task.

Data Flow#

The data flow involves four modules:

Vulkan SC diagram showing the data flow through four modules

  1. The image is captured from the SIPLCamera module in YUV BL/PL format.

  2. The NvMedia2D module then converts or stitches the image format to the ABGR/ARGB BL format image.

  3. Next, the image is imported to the VulkanSC module as a VkImage object. Rendering tasks occur with the imported VkImage object. The timestamp renders to the bottom right of the image and the 3D car model to the center of the image.

  4. Finally, the rendered image is displayed by the Display module.

Workflow#

Vulkan SC applications can be developed from Vulkan applications by creating a Vulkan application with similar functionality and then extracting the offline state (saved as pipeline cache binary) necessary for the Vulkan SC application.

To run and develop Vulkan SC applications, refer to the following Vulkan SC development workflow:

image2

For details on the Vulkan SC workflow, see Running the Vulkan Samples.

The .spv and .json files are located in the features/vksc_scenes/data/ folder and generate the pipeline cache file for multicast. Compile multicast with the provided Makefile.sdk file, and the pipeline cache file pipeline_cache.bin is generated in the features/vksc_scenes/data/ folder.

Next, copy all .json and .bin files under the features/vksc_scenes/data/ folder to the folder where the application will run.

To follow the Vulkan SC development workflow for generating pipline cache files, complete the following steps:

  1. Build nvsipl_multicast with the NV_USE_VULKAN macro enabled by setting NV_USE_VULKAN = 1 in the Makefile and rebuilding the application.

  2. Use glslc to compile vertex and fragment shaders on the host:

    glslc shader.vert -o shader.vert.spv
glslc shader.frag -o shader.frag.spv
glslc timestamp.shader.vert -o timestamp.shader.vert.spv
glslc timestamp.shader.frag -o timestamp.shader.frag.spv
glslc carmodel.shader.vert -o carmodel.shader.vert.spv
glslc carmodel.shader.frag -o carmodel.shader.frag.spv
glslc carmodel_notexture.shader.frag -o carmodel_notexture.shader.frag.spv

  3. Generate the JSON file by running the nvsipl_multicast built in the previous step on the target machine:

    1. Set the environment variables on the target machine:

      export VK_LAYER_PATH=/etc/vulkansc/icd.d
export VK_JSON_FILE_PATH=$PWD/

    2. Copy the compiled shaders from the previous step to the folder where the application runs on the target machine.

    3. Copy all binaries in the features/vksc_scenes/data/ folder to the target folder where the application runs.

    4. Run nvsipl_multicast with the pipeline configuration:

      -p "SIPL, *Nvm2, *VulkanSC=width=1920:height=1080"

      The following files are generated in the current folder:

      * nvsipl_multicast_pipeline_0.json
* nvsipl_multicast_pipeline_0.vert.spv
* nvsipl_multicast_pipeline_0.frag.spv
...
* nvsipl_multicast_pipeline_12.json
* nvsipl_multicast_pipeline_12.vert.spv
* nvsipl_multicast_pipeline_12.vert.spv

  4. Generate the pipeline cache by running the PCC tool on the host:

    1. Copy the files generated in step 3d to the folder where PCC runs on the Linux host machine.

    2. The PCC tool provided in the SDK is an x86_64 binary; therefore, only run the tool on an x86_64 Linux host machine in the folder where the .spv and .json files are located:

      ${NV_WORKSPACE}/drive-qnx/vulkansc/pcc/Linux_x86-64/pcc -chip [gb10b|ga10b] -path ./ -out pipeline_cache.bin

  5. Rebuild nvsipl_multicast with the NV_USE_VULKAN macro off by setting NV_USE_VULKAN = 0 in the Makefile and rebuilding the application.

  6. Run the Vulkan SC version of nvsipl_multicast on the target machine:

    1. Rename nvsipl_multicast_pipeline_0.json to timestamp_render_pipeline.json.

    2. Rename nvsipl_multicast_pipeline_1.json to car_model_render_pipeline.json.

    3. Rename nvsipl_multicast_pipeline_9.json to car_model_notexture_render_pipeline.json.

    4. Replace the .bin and .json files in the features/vksc_scenes/data/ folder with the files generated in steps 3 and 4.

    5. Copy all binaries in the features/vksc_scenes/data/ folder to the target folder where the application will run.

    6. Run nvsipl_multicast without display:

      -p "SIPL, *Nvm2d, *VulkanSC=width=1920:height=1080"

    7. Run nvsipl_multicast with display. You must plug in the display, and the width and height must be adjusted to the actual conditions:

      -p "SIPL,*Nvm2d, *VulkanSC=width=1920:height=1080:colortype=ABGR:passthrough=1,*Display=width=1920:height=1080:colortype=ABGR"

Result#

Four images from different cameras are stitched and color-converted by the NvMedia2D module. The timestamp is rendered to the bottom right of the image, and the 3D car model is rendered to the center of the image.

image3

Constraints#

  1. Vulkan SC only supports importing NvSciBuf with the color format of NvSciColor_A8B8G8R8/NvSciColor_A8R8G8B8 as VkImage (zero copy). If the input color format does not match, the NvMedia2D module might be necessary for color conversion.

  2. Vulkan SC only supports importing NvSciSyncObj as VkFence when NvMedia2D / Display modules are involved in the pipeline. Otherwise, the reconciliation of sync attributes will fail.

  3. The JSON-generated layer of Vulkan SC is currently only supported on Drive OS Linux or Desktop Vulkan (version 1.2 or later) SDK with NVIDIA drivers installed. It is not supported in QNX, therefore step 1 must be executed in Linux.

On this page