SIPL Guidance on Output Image Formats

SIPL supports multiple image formats using the ICP and ISP outputs. This topic describes how the output formats are set and which parameters can be modified to override output formats under specific conditions.

ICP Output Formats

ICP output formats are dependent on the sensor used.

Based on the vendor and sensor module specifics, the output format of ICP is determined using the PlatformCfg::DeviceBlockInfo:: CameraModuleInfo::SensorInfo::VirtualChannelInfo::cfa and inputFormat fields in the PlatformCfg struct provided as input to the INvSIPLCamera::SetPlatformCfg API.

Refer to NvSIPLCapStructs.h for the following:

  1. Supported values for input format that correspond to NvSiplCapInputFormatType enum.
  2. Supported values for cfa that correspond to the valid NVSIPL_PIXEL_ORDER flags.
Note: SIPL does not add restrictions on the use of different input format type/pixel orders because these are specific to sensors, and SIPL does not have a way of determining the expectations of each sensor. A recommendation is to adhere to the specifics of the sensors.

SIPL however does pose restrictions such as rejecting bad combinations of input format and pixel order. Some examples of the invalid combinations are as follows.

  1. NVSIPL_CAP_INPUT_FORMAT_TYPE_YUV422 and NVSIPL_PIXEL_ORDER_RGBA
  2. NVSIPL_CAP_INPUT_FORMAT_TYPE_RAW6 and NVSIPL_PIXEL_ORDER_RGBA
  3. NVSIPL_CAP_INPUT_FORMAT_TYPE_RGB888 and NVSIPL_PIXEL_ORDER_RGGB
  4. NVSIPL_CAP_INPUT_FORMAT_TYPE_YUV422 and NVSIPL_PIXEL_ORDER_RGGB
  5. NVSIPL_CAP_INPUT_FORMAT_TYPE_RAW6 and NVSIPL_PIXEL_ORDER_YUV
  6. NVSIPL_CAP_INPUT_FORMAT_TYPE_RGB888 and NVSIPL_PIXEL_ORDER_YUV

ISP Output Formats

Refer to the table provided as part of NvSIPLCamera.hpp for documentation on INvSIPLCamera::RegisterImages API for the various ISP output formats supported by SIPL.

In 6.0 release, users are expected to call INvSIPLCamera::GetImageAttributes API before calling INvSIPLCamera::RegisterImages API in order to validate the attributes provided by the user. The sequence of API calls is as follows:

  1. Application calls NvSciBufAttrListCreate to create an unreconciled buffer attribute list.
  2. Application implements and calls OverrideImageAttributes(NvSciBufAttrs) function to set the required output format for each ISP output in case it is different from the default.
Note: Sample function is implemented in nvsipl_camera -> CNvSIPLMaster.hpp
  1. Application calls INvSIPLCamera::GetImageAttrs(NvSciBufAttrs), which checks and validates the buffer attributes in case you chose to override the default format. Otherwise, Application adds attributes for the default formats.
Note: NVIDIA maintains the name of the API due to ABI compliance requirements. However, a name such as VerifyandSetImageAttrsthe is more representative of the functionality.
  1. Application calls NvSciBufAttrReconcile to reconcile all buffer attributes across other buffer attribute lists (other consumers for this buffer, which could be any downstream engine, such as VIC, IEP, and CUDA)
  2. Application calls NvSciBufObjAlloc on the reconciled attribute list to create a buffer object
  3. Application calls INvSIPLCamera::RegisterImages(NvSciBufObj), which only accepts the buffer if one of the attribute lists used to reconcile and allocate was verified by SIPL using INvSIPLCamera::GetImageAttrs
Note:
  1. ISP output images will be a binary match if the default NITO provided by NVIDIA is used while requesting the same output formats with same resolution.
  2. The ISP0 and ISP1 outputs can have different formats as long as they both are YUV type.
  3. The camera tuning does not have to be re-run for different formats since formats are just different ways to represent images in the memory. However, the data itself will differ as the representation of the memory varies across formats.

OverrideImageAttributes

ISP supports three types of formats:

  1. YUV Semi Planar Images
  2. YUV Packed Images–Luma is a type of YUV Packed Image
  3. RGBA Packed Images

Creating packed type buffers (2 and 3) are straightforward: you use NvSciBufImageAttrKey_Plane*** type attributes to override Image formats.

For semi-planar type buffers (1), however, NvSciBuf provides attribute keys to create buffers of the above types using two methods:

  1. Using NvSciBufImageAttrKey_Plane*** type attributes
  2. Using NvSciBufImageAttrKey_Surf*** type attributes

NvSciBuf added the Surf type attributes to make the task of creating multi-plane image buffers easy for the user. However, in Plane type attributes you must provide the height and width of each plane to determine the memory layout of the image.

In SIPL the library oversees calculations for height and width of the buffer factoring in multiple criteria such as sensor output resolution, input crop, downscale, and so on. This is complicated if you supply Plane type attributes to SIPL while trying to create multi-plane images. Hence, SIPL restricts you to Surf type attributes when using multi-plane YUV images.

Examples

For YUV 444 SEMI-PLANAR UINT 16 BLOCK LINEAR image, the following attributes should be set to the corresponding values:

Attribute Name Attribute Value
NvSciBufGeneralAttrKey_Types NvSciBufType_Image
NvSciBufImageAttrKey_SurfType NvSciSurfType_YUV
NvSciBufImageAttrKey_SurfBPC NvSciSurfBPC_16
NvSciBufImageAttrKey_SurfMemLayout NvSciSurfMemLayout_SemiPlanar
NvSciBufImageAttrKey_SurfSampleType NvSciSurfSampleType_444
NvSciBufImageAttrKey_SurfComponentOrder NvSciSurfComponentOrder_YUV
NvSciBufImageAttrKey_SurfColorStd NvSciColorStd_REC709_ER
NvSciBufImageAttrKey_Layout NvSciBufImage_BlockLinearType

For YUV 444 PACKED UINT 8 BLOCK LINEAR image, the following attributes should be set to the corresponding values

Attribute Name Attribute Value
NvSciBufGeneralAttrKey_Types NvSciBufType_Image
NvSciBufImageAttrKey_PlaneCount 1
NvSciBufImageAttrKey_Layout NvSciBufImage_BlockLinearType
NvSciBufImageAttrKey_PlaneColorFormat NvSciColor_A8Y8U8V8
NvSciBufImageAttrKey_PlaneColorStd NvSciColorStd_REC709_ER