Video for Linux User Guide
This document provides information on use of the MIPI Camera Serial Interface (CSI) on Tegra® X1, using software from the NVIDIA® Tegra® Linux Driver Package (also referred to as L4T). The MIPI CSI protocol, V4L2 API, Tegra X1 system architecture and method of attaching a CSI camera to Jetson TX1 are outside the scope of this document.
The V4L2 software implementation bypasses the Tegra ISP, and is suitable for use when Tegra ISP support is not required, such as with sensors or input devices that provide data in YUV format.
References to additional resources are provided, but the reader should already be familiar with Tegra X1, and have access to the Tegra Technical Reference Manual (TRM) and other documentation available at the Jetson Embedded Platform portal:
http://developer.nvidia.com/embedded-computing
V4L2/SOC_CAMERA Overview
V4L2 is the second version of Video4Linux or V4L, a video capture and output device API and driver framework in the Linux kernel. It supports many USB webcams, TV tuners, and other devices and is closely integrated with the Linux kernel. For a description of the APIs, see
Linux Media Infrastructure APIs.
soc-camera is a set of drivers and a core module, that implement V4L2 functionality on embedded devices; typically a video-enabled embedded device: SoC with a capture interface and video data sources. The soc-camera includes host driver such as the Tegra V4L2 camera driver and client drivers (sensor drivers).
Note: | Software releases after R23: soc_camera driver is deprecated and replaced with the media-controller driver. The media-controller driver framework is V4L2-compatible and provides greater functionality than the soc_camera driver. Existing V4L2 sensor drivers require minor modifications to be compatible with media-controller. Plan your software development for this transition. |
V4L2 on Jetson TX1
Jetson TX1 is a powerful embedded development board including the NVIDIA Tegra X1 processor. The Tegra X1 processor has a video input interface (VI) and camera serial interface (CSI), so it can communicate with the external video input sources such as camera sensor module or other MIPI CSI compatible devices. VI/CSI of Tegra X1 also has 2 test pattern generators which can generate some data patterns like color bricks for testing purpose.
On the software side, Linux for Tegra (
L4T) latest release R23 provides a Tegra V4L2 camera driver and sample drivers for a camera sensor and a built-in test pattern generator (TPG). With an open source V4L2 user space tool like
Yavta, users can capture data from the TPG and sensors.
Test Pattern Generator
The test pattern generator is a configurable resource introduced to improve hardware verification capability for the Tegra CSI. There are two separate test pattern generators that can be configured to provide for the generation of synthetic image data, which is delivered to the PPA and PPB input FIFOs. The image data is multiplexed into the CSI data patch between lane-merging logic and the data FIFOs.
L4T provides a virtual V4L2 soc_camera sensor driver for exposing TPG functionality (soc_camera_platform driver). It can generate 1280x720 resolution RGBA32 color bricks data. There is no need to rebuild the kernel and the soc_camera_platform driver is provided as a loadable module.
To verify the TPG
1. Remove the nvhost_vi module, an incompatible non-V4L2 VI driver used for other purposes and outside the scope of this document:
$ sudo rmmod nvhost_vi
2. Install V4L2 driver modules:
$ sudo modprobe soc_camera_platform
$ sudo modprobe tegra_camera tpg_mode=2
3. Use the yavta application to capture data (other V4L2 applications can be used, if preferred)
$ ./yavta /dev/video0 -c1 -n1 -s1280x720 -fRGB32 -Ftpg.rgba
4. Copy over tpg.rgba to host and use ImageMagick to show the picture:
$ display -size 1280x720 -depth 8 tpg.rgba
Example Sensor: OV5693
L4T provides a sample V4L2 sensor driver for the OmniVision OV5693 Bayer sensor. This driver can be used as a reference in creating a custom V4L2 sensor driver. NVIDIA provides a reference OV5693 camera module E3326, with Jetson TX1.
The driver for OV5693 is neither built into kernel nor built as module. Please try following steps to test OV5693 in L4T on Jetson TX1.
Hardware setup
•Jetson TX1
•OV5693 camera module
1. Apply patches on top of L4T release
L4T 23.1 Release (19 patches)
$ tar xfz r23.1_v4l2.tgz
$ ls r23.1/
0001-drivers-soc_camera-add-ov23850-OTP-controls.patch
0002-drivers-soc_camera-sensor-drivers-update.patch
0003-drivers-media-platform-tegra-get-gpio-pwr-info.patch
0004-drivers-media-soc_camera-IMX230-driver-to-gain.patch
0005-drivers-Restructure-GRHOST_VI.patch
0006-media-v4l2-core-Migration-from-upstream.patch
0007-media-soc_camera-add-ov13860-v4l2-driver.patch
0008-drivers-media-platform-ov5693-gpio-warning.patch
0009-media-tegra_camera-support-YUV-CSI-input.patch
0010-kernel-changes-for-implementing-sensor-cropping.patch
0011-drivers-soc_camera-give-default-hdr_en.patch
0012-media-tegra_camera-fix-syncpoint-time-out-issue.patch
0013-media-soc-camera-actually-use-default-resolution.patch
0014-media-soc_camera-add-tc358840-v4l2-driver.patch
0015-media-soc_camera-vi2-support-gang-mode.patch
0016-media-camera-common-export-symbols-for-modules.patch
0017-media-tegra_camera-init-mipi-bias-pad.patch
0018-ARM64-t210-add-E3326-camera-and-camera_common.patch
0019-ARM64-adding-OV5693-V4L2-on-E3326-jetson_cv.patch
$ git am r23.1/*
L4T R23.2 Release (One patch)
$ git am 0001-ARM64-adding-OV5693-V4L2-on-E3326-jetson_cv.patch
2. Enable OV5693 kernel driver and disable soc_camera_platform.
•CONFIG_SOC_CAMERA_OV5693=m
•CONFIG_VIDEO_TEGRA_VI=m
•Disable CONFIG_SOC_CAMERA_PLATFORM
•Disable CONFIG_SOC_CAMERA_OV13860
•Disable CONFIG_SOC_CAMERA TC358840
3. Build kernel, flash Jetson TX1 and boot the Linux OS
4. Use Yavta to capture a frame
$ sudo rmmod nvhost-vi
$ sudo modprobe ov5693_v4l2
$ sudo modprobe tegra_camera
$ ./yavta /dev/video0 -c1 -n1 -s1920x1080 -fSRGGB10 -Fov.raw
5. Use raw2bmp to convert raw data to BMP file
$ ./raw2bmp ov.raw ov.bmp 1920 1080 16 3
V4L2 Tegra Driver Overview
As V4L2 is a kernel video input framework, Tegra V4L2 stack contains several components. It controls hardware such as the Tegra VI/CSI hardware controller and external sensors. Additionally, it exports a generic device node named /dev/video<N> to user space, where <N> is a numeric value. User space application can use V4L2 standard API to control real hardware via /dev/video<N>.
This section focuses on Tegra X1-related drivers and code in L4T kernel source.
Tegra V4L2 Camera Driver
Tegra V4L2 camera driver is a part of soc_camera and acts as a host driver. It directly controls Tegra X1 VI/CSI hardware. Normally users don’t need to modify this driver, but developers should become familiar with it; it may require customization for some use cases.
•Source code
drivers/media/platform/soc_camera/Kconfig
drivers/media/platform/soc_camera/Makefile
drivers/media/platform/soc_camera/tegra_camera/*
include/media/tegra_v4l2_camera.h
•Kernel config
CONFIG_VIDEO_TEGRA=m
The module name is tegra_camera.ko and it won’t be loaded by default after booting into L4T. There is another driver named nvhost_vi.ko installed by default and mutually-exclusive with tegra_camera.ko, so users must remove the nvhost_vi.ko before loading tegra_camera.ko.
•Input data format
Tegra X1 VI/CSI hardware supports 3 major input data format: YUV, RGB and Bayer RAW. However in this driver only the following have been tested:
•RGB888
•RAW8
•RAW10
•YUV422
Note: | Note: Some other formats are also supported by hardware, but software support is not present in the driver. Please refer to the Tegra TRM for details on supported input formats. |
Please study the source code then add new input data formats not listed here.
Tegra V4L2 Sensor Driver
V4L2 sensor driver normally is an I2C device driver and in L4T it is also a V4L2 soc_camera client driver. It has several I2C register tables for different resolutions like 1920x1080, 1280x720 etc. When a user space application opens /dev/video<N>, the sensor driver powers on the sensor hardware and programs it with the register table via I2C.
•Real sensor code
drivers/media/i2c/soc_camera/ov5693_v4l2.c
drivers/media/i2c/soc_camera/ov5693_mode_tbls.h
include/media/ov5693.h
drivers/media/i2c/soc_camera/Kconfig
drivers/media/i2c/soc_camera/Makefile
•Test Pattern Generator virtual sensor driver source code
drivers/media/platform/soc_camera/soc_camera_platform.c
The soc_camera_platform driver does not perform any real hardware operations like power control and I2C transactions. It is just a virtual driver to enable use of the TPG for testing.
•Kernel configs
CONFIG_SOC_CAMERA_OV5693
CONFIG_SOC_CAMERA_PLATFORM
•Power controls
Each sensor has its own power on/off sequence, clock settings and other hardware specific operations. L4T sensor driver puts these power controls in the sensor driver itself. For more flexible driver design, these power controls must go to board files since each hardware board may have different power controls. Then, the sensor driver itself can be more generic. Normally, power controls include:
•GPIO for sensor reset, power on or power down
•Regulators for sensor power supply
•Clocks for sensor running like mclk or sensor local clock
Board File
Before fully moving to device tree binding, a board file is the only way to describe platform-specific configurations within the Linux kernel. In L4T R23 releases, most hardware devices use device tree binding but V4L2 soc_camera still uses a board file approach.
•Source code
arch/arm64/mach-tegra/board-t210ref-camera.c
•TPG board configs
soc_camera_platform_info defines data format and resolution which must be matched with our TPG hardware.
static struct soc_camera_platform_info t210ref_soc_camera_info = {
.format_name = "RGB4",
.format_depth = 32,
.format = {
.code = V4L2_MBUS_FMT_RGBA8888_4X8_LE,
.colorspace = V4L2_COLORSPACE_SRGB,
.field = V4L2_FIELD_NONE,
.width = 1280,
.height = 720,
},
.set_capture = t210ref_soc_camera_set_capture,
};
tegra_camera_platform_data is the most important data structure to describe the sensor connection. .port indicates which CSI port the sensor connects to:
TEGRA_CAMERA_PORT_CSI_A means the sensor uses CIL_A.
TEGRA_CAMERA_PORT_CSI_B means the sensor uses CIL_B.
TEGRA_CAMERA_PORT_CSI_C means the sensor uses CIL_C.
TEGRA_CAMERA_PORT_CSI_D means the sensor uses CIL_D.
TEGRA_CAMERA_PORT_CSI_E means the sensor uses CIL_E.
TEGRA_CAMERA_PORT_CSI_F means the sensor uses CIL_F.
Tegra X1 internally has 6 CSI channels (CSI_A to CSI_F). CSI A/C/E channel can support 1, 2 and 4 data lane sensors and CSI B/D/F can support 1 and 2 data lane sensors..
static struct tegra_camera_platform_data t210ref_camera_platform_data = {
.flip_v = 0,
.flip_h = 0,
.port = TEGRA_CAMERA_PORT_CSI_A,
.lanes = 4,
.continuous_clk = 0,
};
•OV5693 board file configs
Real sensors do not require that sensor resolution or data format information be put into the board file like TPG soc_camera_platform driver, because that information is in the sensor driver itself.
•OV5693 connects to port CSI_C via 2 data lanes:
static struct tegra_camera_platform_data
t210ref_ov5693_e3326_camera_platform_data = {
.flip_v = 0,
.flip_h = 0,
.port = TEGRA_CAMERA_PORT_CSI_C,
.lanes = 2,
.continuous_clk = 0,
};
•OV5693 uses I2C bus 6 and it’s I2C address is 0x36:
static struct camera_common_pdata t210ref_ov5693_e3326_data = {
.regulators = {
.avdd = "vana",
.iovdd = "vif",
},
.reset_gpio = 148, /* TEGRA_GPIO_PS4 */
.pwdn_gpio = 151, /* TEGRA_GPIO_PS7 */
};
static struct i2c_board_info t210ref_ov5693_e3326_camera_i2c_device = {
I2C_BOARD_INFO("ov5693_v4l2", 0x36),
};
static struct soc_camera_link ov5693_e3326_iclink = {
.bus_id = 0, /* This must match the .id of tegra_vi01_device */
.board_info = &t210ref_ov5693_e3326_camera_i2c_device,
.module_name = "ov5693_v4l2",
.i2c_adapter_id = 6, /* VI2 I2C controller */
.power = t210ref_ov5693_power,
.priv = &t210ref_ov5693_e3326_camera_platform_data,
.dev_priv = &t210ref_ov5693_e3326_data,
};
•Register OV5693 soc_camera platform device:
platform_device_register(&t210ref_ov5693_e3326_soc_camera_device);
Device Tree File
Device tree provides regulator information required by the V4L2 sensor driver. OV5693 sensor driver use 2 regulators: vana and vif. They are defined in:
arch/arm64/boot/dts/tegra210-platforms/tegra210-jetson-e-pmic-p2530-0930-e03.dtsi
arch/arm64/boot/dts/tegra210-platforms/tegra210-jetson-e-power-fixed-p2530-0930-e03.dtsi
Writing and Integrating a Sensor Driver For L4T
Developers can write their own sensor driver for their specific device. Sensor drivers usually have very similar structures but different I2C register tables. Modification of the board file and the device tree file is required for different boards.
Sensor Driver Development
The OV5693 sensor drivers are a good starting point for writing a new sensor driver. The following steps are recommended for developing a new driver:
•Import new I2C register tables
Sensor vendors provide I2C register settings as tables which must be added to sensor driver. The following struct is a good example:
static const ov5693_reg *mode_table[] = {
[OV5693_MODE_2592X1944] = mode_2592x1944,
[OV5693_MODE_2592X1458] = mode_2592x1458,
[OV5693_MODE_1920X1080] = mode_1920x1080,
[OV5693_MODE_1296X972] = mode_1296x972,
[OV5693_MODE_1280X720_120FPS] = mode_1280x720_120fps,
[OV5693_MODE_2592X1944_HDR] = mode_2592x1944_HDR_24fps,
[OV5693_MODE_1920X1080_HDR] = mode_1920x1080_HDR_30fps,
[OV5693_MODE_1296X972_HDR] = mode_1296x972_HDR_30fps,
[OV5693_MODE_START_STREAM] = ov5693_start,
[OV5693_MODE_STOP_STREAM] = ov5693_stop,
[OV5693_MODE_TEST_PATTERN] = tp_colorbars,
};
•Power controls
Different boards have different sensor power controls. It is better put those power controls into a board file. But it is simpler to implement them in a sensor driver. Please take a look at OV5693_power_on() and OV5693_power_off().
•soc_camera and I2C interface
The sensor driver implements soc_camera_ops functions as well as I2C device probing/removing functions. Normally these are quite similar across different sensor drivers -- just reuse them in your driver and use OV5693_v4l2.c as an example.
•KConfig and Makefile
Add a SOC_CAMERA_OV5693 entry into the Kconfig and Makefile files.
•Header file include/media/ov5693.h
This header contains some information for non-V4L2 NVIDIA camera stacks. The following structures can be reused if necessary:
struct ov5693_regulators {
const char *avdd;
const char *dvdd;
const char *dovdd;
};
struct ov5693_platform_data {
unsigned cfg;
unsigned num;
const char *dev_name;
unsigned gpio_count; /* see nvc.h GPIO notes */
struct nvc_gpio_pdata *gpio; /* see nvc.h GPIO notes */
struct nvc_imager_static_nvc *static_info;
bool use_vcm_vdd;
int (*probe_clock)(unsigned long);
int (*power_on)(struct ov5693_power_rail *);
int (*power_off)(struct ov5693_power_rail *);
const char *mclk_name;
struct nvc_imager_cap *cap;
struct ov5693_regulators regulators;
bool has_eeprom;
bool use_cam_gpio;
};
Board File and Device Tree File Updates
A new project or new hardware board may have a new board file such as board-t210ref*.c for Jetson TX1. If so, the new board file must include those settings for sensor drivers. Follow this template in the board file and replace “SENSOR” with your sensor name:
#if IS_ENABLED(CONFIG_SOC_CAMERA_SENSOR)
static int t210ref_sensor_power(struct device *dev, int enable)
{
return 0;
}
// NOTE: power controls can go here instead of sensor driver itself.
struct sensor_platform_data t210ref_sensor_data;
static struct i2c_board_info t210_sensor_camera_i2c_device = {
I2C_BOARD_INFO("sensor_v4l2_driver_name", sensor_i2c_address),
// sensor_v4l2_driver_name should match the sensor driver’s module name
.platform_data = &t210ref_sensor_data,
};
static struct tegra_camera_platform_data t210ref_sensor_camera_platform_data = {
.flip_v = 0,
.flip_h = 0,
.port = TEGRA_CAMERA_PORT_CSI_X_for_sensor,
.lanes = number_of_sensor_data_lanes,
.continuous_clk = 0,
};
static struct soc_camera_link sensor_iclink = {
.bus_id = 0,
.board_info = &t210ref_sensor_camera_i2c_device,
.module_name = "sensor_v4l2_driver_name",
.i2c_adapter_id = sensor_i2c_bus_number,
.power = t210ref_sensor_power,
.priv = &t210ref_sensor_camera_platform_data,
};
static struct platform_device t210ref_sensor_soc_camera_device = {
.name = "soc-camera-pdrv",
.id = 0,
.dev = {
.platform_data = &sensor_iclink,
},
};
#endif
•Finally register the platform device in t210ref_camera_init():
#if IS_ENABLED(CONFIG_SOC_CAMERA_SENSOR)
platform_device_register(&t210ref_sensor_soc_camera_device);
#endif
•Device tree update
Find the new device tree file for the new board and update regulator information appropriate to the hardware configuration of the new board. A good example to look at is:
arch/arm64/boot/dts/tegra210-platforms/tegra210-jetson-e-pmic-p2530-0930-e03.dtsi
arch/arm64/boot/dts/tegra210-platforms/tegra210-jetson-e-power-fixed-p2530-0930-e03.dtsi
Troubleshooting
I2C transaction timeout error
•I2C information is wrong
Check the sensor I2C bus number and the sensor I2C device address in the board file.
•Sensor power control sequence is wrong
Check sensor MCLK setting.
Check regulator operations.
Check GPIO settings.
Sync point timeout without error
This means Tegra VI/CSI does not receive any data but no error occurs. Verify that the sensor is powered on and streaming data correctly before debugging the Tegra driver.
•Change settle time value to see if there if some error shows up. These registers must be configured with the right values to get data from the sensor.
cil_regs_write(vi2_cam, chan, TEGRA_CSI_CIL_PHY_CONTROL, 0xA);
Sync point timeout with error
Capture the error message and look it up in Tegra X1 TRM for further debugging.
Resources
Good resources for V4L integration are:
•Kernel documentation located in:
Documentation/video4linux/
•Linux TV website:
•soc-camera slides:
•Yavta user space V4L2 tool
•Jetson Embedded Platform page