Optical Flow Accelerator (OFA)
NVIDIA Optical Flow Accelerator (OFA) is hardware accelerator for computing optical flow and stereo disparity between the frames.
Optical flow is useful in various use-case such as object detection and tracking, while Stereo disparity is used in depth estimation.
The hardware capabilities of OFA are exposed through NvMedia IOFA APIs.
OFA can operate in two modes:
- Stereo Disparity Mode: In this mode, OFA generates disparity between rectified left and right images of stereo capture.
- Optical Flow Mode: In this mode, OFA generates optical flow between two given frames, returning X and Y component of flow vectors.
OFA generates disparity / flow vector block-wise, one output for each input block of 8x8 / 4x4 / 2x2 / 1x1 pixels (referred as output grid size). The generated output can be further post-processed to improve accuracy, up sampled to produce dense map.
Stereo Disparity Mode
OFA processes rectified left and right view of stereo captures and generates disparity values between them.
OFA in Stereo Disparity Mode, can process only rectified stereo image pairs. Rectification of stereo image pair can be done using NvMedia Image 2D.
The output stereo disparity format is fixed signed 10.5 (2 bytes per disparity output) and range of disparity is fixed [0, 127] or [0, 255]. We need to divide the output values by 32 to get a disparity value in terms of pixel units.
OFA also provides Cost buffer which contain HW cost of winner disparity candidates. Cost for each output is 8 bit in size and range is 0 – 255.
Disparity values have inverse relationship with depth of the objects (distance between object and sensor).
| Stereo Parameter | Description |
|---|---|
| Input Image Size |
Minimum Size – 32 x 32 Maximum Size – 8192 x 8192 No alignment requirement |
| Input Image format / bit depth |
Luma / Single channel Input Supports bit depth of 8/10/12/16 bits |
| Disparity Output | Disparity Map in fixed S 10.5 format |
| Cost Output | HW cost for winner disparity candidate |
| Output Grid Size | 1x1/2x2/4x4/8x8 |
| Maximum Disparity Range | 128 / 256 |
| Search Direction | Left / Right Disparity Map |
| Region Of Interest Support | Supports maximum 32 ROI per stereo pair |
Stereo Use Case
The high-level block diagram of OFA Stereo use case is as follows
The stereo sensor provides unrectified left and right view images.
The application uses NvMedia 2D API to rectify these input stereo image pair.
OFA, configured in stereo mode processes rectified stereo image pair and generates disparity map and HW cost buffer.
The post-processing module processes the generated disparity map and HW cost buffer to improve accuracy of the disparity map.
Common examples of post processing algorithms are:
- Median filtering of disparity map
- Up sampling to produce a dense disparity map
- left–right consistency check
Stereo rectification and post-processing operations are not supported using OFA and must be implemented on other HW accelerators.
Optical Flow Mode
OFA generates optical flow between two given frames.
The input to OFA in this mode, is image pyramid of input and reference frames. As search range of single layer is small, pyramid approach is required to track large motion. Each pyramid level will search around output of previous pyramid level.
OFA can process image pyramid generated with fixed scale factor of 2. Image pyramid generation needs be done outside OFA.
OFA generates flow vector for input block specified by output Grid size. Each flow vector has X and Y component which represent motion in X and Y direction.
The output flow format is fixed signed 10.5 (4 bytes per flow vector). We need to divide the output values by 32 to get a disparity value in terms of pixel units.
| OF Parameter | Description |
|---|---|
| Input Image Size |
Minimum Size – 32 x 32 Maximum Size – 8192 x 8192 No alignment requirement |
| Input Image format / bit depth |
Luma / Single channel Input Supports bit depth of 8/10/12/16 bits |
| Max Pyramid Levels | 5 |
| Flow Output | Flow Map (mvx, mvy) in fixed S10.5 format |
| Cost Output | HW cost for winner disparity candidate |
| Output Grid Size | 1x1/2x2/4x4/8x8 |
| Region Of Interest Support | Supports maximum 32 ROI per stereo pair |
Optical Flow Use Case
The high-level block diagram of OFA optical flowuse case is as follows
Here stereo sensor provides the input and reference frame.
The application generates the image pyramid of input and reference frames.
OFA, configured in off mode processes pyramids of input and reference frames and generates the flow map and HW cost buffer.
The post-processing module processes the generated flow map and HW cost buffer to improve accuracy of the generated flow map.
Common examples of post-processing algorithms are:
- Median filtering of flow map
- Up sampling to produce dense flow map
- Forward-backward consistency check
Pyramid generation and post-processing operations are not supported using OFA and must be implemented on other HW accelerators.