Logging

Note: The nvlog feature is currently under development. You can enable this feature for testing purposes, but do not enable the feature in production systems until it is enabled by default in DRIVE OS releases.

NVIDIA DRIVE® OS has a logging component that allows logs from multiple DRIVE OS components to be obtained. Currently, support is only available for applications within DRIVE OS Linux Guest VM to send logs that are stored in a file that is created and accessible from Linux Guest VM. This support will be enhanced in future releases to support more execution environments outside of Guest VM.

Terms

Logging Entity : Application threads(s) trying to log, each thread is an entity.

Entity Logging library : entity specific logging library build that is linked to Guest VM entities (libnvlog_guestvm_entity)

nvlogclient_fast : logging client/server combined application.

Logging library : full build of logging library that is linked to nvlogclient_fast (libnvlog.so)

Logging Pipeline

The logging pipeline is as follows:
(Logging Entity + libnvlog_guestvm_entity) -> (nvlogclient_fast + libnvlog.so) -> log storage file

Components

The following lists the source and binary locations within the SDK:
  1. Pre-built binaries:
    <top>/drive-linux/lib-target/libnvlog.so
<top>/drive-linux/filesystem/contents/bin/nvlogclient_fast
<top>/drive-linux/filesystem/contents/bin/nvlogtest
<top>/drive-linux/filesystem/contents/bin/nvlogtest_stdin
  2. Source code:
    Source files for the test applications (nvlogtest*) and the Logging Client (nvlogclient_fast) can be found under:
    <top>/drive-linux/samples/tools/nvlog/
  3. Include files and user API:

    Any application needing to send logs using this new framework must use the <top>/drive-linux/samples/tools/nvlog/include/nvlog.h header file and link to libnvlog_guestvm_entity library. API documentation can be found in the same header file.

In a booted DRIVE OS Linux Guest VM console, the following logging related components are available:
  1. /usr/local/bin/

    nvlogclient_fast : Application component responsible for collecting logs from different Logging Entities within Linux Guest VM and storing them.

    nvlogtest : Test application that has two threads sending log messages every second.

    nvlogtest_stdin : Test application that takes input from stdin and sends each line as a log message.

  2. /usr/local/lib/

    libnvlog.so : Logging library that provides logging APIs.

Execution Steps

Default Behavior

nvlog library is disabled by default on boot. This is to keep the default behavior of the system same as before while allowing incremental testing of nvlog. Following messages can be seen for each process that is trying to log via. nvlog during boot (in syslog):
Dec 21 00:54:11 tegra-ubuntu nvtzvault: NvLogOsSupportEnabled: nvlog DT node is not enabled, disabling library                                                                                                                                    
Dec 21 00:54:11 tegra-ubuntu nvtzvault: NvLogOsInit - NvLog support is not enabled
The following messages (in syslog) also appears during boot to show that nvlogclient_fast was launched but failed due to nvlog library being disabled.
Dec 21 00:54:11 tegra-ubuntu bash[1148]: Launched /usr/local/bin/nvlogclient_fast !!!
Dec 21 00:54:11 tegra-ubuntu nvlogclient_fast: NvLogOsSupportEnabled: nvlog DT node is not enabled, disabling library
Dec 21 00:54:11 tegra-ubuntu nvlogclient_fast: NvLogOsInit - NvLog support is not enabled
Dec 21 00:54:11 tegra-ubuntu bash[1179]: main - NvLogClientInit failed for msgqueue (error : -1)

Behavior with nvlog Enabled

nvlog can be enabled for testing by adding the following DT node in tegra234-p3710-0010-a01-driveav-gos-qnxwrap.dts:
nvlog {
status = "okay";
};
After recompiling/flashing, following messages can be seen during boot:
Dec 21 00:54:11 tegra-ubuntu systemd[1]: Starting nvlog client logging service...
Dec 21 00:54:11 tegra-ubuntu bash[1140]: Launched /usr/local/bin/nvlogclient_fast !!!
Dec 21 00:54:11 tegra-ubuntu nvlogclient_fast: SHMDiscoveryClientThread: Discovery client thread starting..
Dec 21 00:54:11 tegra-ubuntu nvlogclient_fast: NvLogDiscoveryPlugin_nvsciipc_Init - Entry
Dec 21 00:54:11 tegra-ubuntu systemd[1]: Started nvlog client logging service.
Output log file is created in the /home/nvidia/ directory:
nvidia@tegra-ubuntu:~$ ls -al /home/nvidia/nvlog_system_log.txt
-rw-r--r--. 1 root root 20480 Dec 21 00:54 /home/nvidia/nvlog_system_log.txt

nvidia@tegra-ubuntu:~$ cat /home/nvidia/nvlog_system_log.txt
      EUID       PID ENTITY ID    SEQ     TIMESTAMP  MSG
         0      1305      1305              3      346841060    Module_id 51 Severity 5 : created TA endpoint: 1, created CA endpoints: 2
         0      1305      1305              2      346840076    Module_id 51 Severity 5 : Server trying IVC endpoint:  pkcs11_keystore_tk11_ivc_d2
         0      1305      1305              1      346838406    Module_id 51 Severity 5 : Server trying IVC endpoint:  pkcs11_keystore_tk11_ivc_d1
         0      1291      1304              1      346765176    Module_id 51 Severity 5 : Waiting for Suspend Request from Kernel:  pkcs11-keystore-tk10
         0      1291      1291              3      346758455    Module_id 51 Severity 5 : created TA endpoint: 1, created CA endpoints: 2
         0      1291      1291              2      346757262    Module_id 51 Severity 5 : Server trying IVC endpoint:  pkcs11_keystore_tk10_ivc_d2
         0      1291      1291              1      346755441    Module_id 51 Severity 5 : Server trying IVC endpoint:  pkcs11_keystore_tk10_ivc_d1
         0      1284      1299              1      346717953    Module_id 51 Severity 5 : Waiting for Suspend Request from Kernel:  pkcs11-keystore-tk1
         0      1284      1284              3      346711198    Module_id 51 Severity 5 : created TA endpoint: 1, created CA endpoints: 2
         0      1284      1284              2      346710136    Module_id 51 Severity 5 : Server trying IVC endpoint:  pkcs11_keystore_tk1_ivc_d2
         0      1284      1284              1      346708419    Module_id 51 Severity 5 : Server trying IVC endpoint:  pkcs11_keystore_tk1_ivc_d1
         0      1272      1281              1      346543194    Module_id 51 Severity 5 : Waiting for Suspend Request from Kernel:  nvmacsec
         0      1272      1272              3      346537165    Module_id 51 Severity 5 : created TA endpoint: 1, created CA endpoints: 2
         0      1272      1272              2      346536002    Module_id 51 Severity 5 : Server trying IVC endpoint:  nvmacsec_ta_d2
         0      1272      1272              1      346534344    Module_id 51 Severity 5 : Server trying IVC endpoint:  nvmacsec_ta_d1
         0      1260      1270              1      346428824    Module_id 51 Severity 5 : Waiting for Suspend Request from Kernel:  gp-se
…
Note: Raw output from Logging Client is not sequential with regards to generated logs. This is due to the underlying implementation where readers and writers are lockless and the reader retrieves messages in bursts (in reverse order for each burst). The raw output file must be sorted on the basis of timestamp (column 5) to get the logs from multiple sources in order (example: sort -k5 -n).
Note: Log messages can be lost if entity (or producer) is spewing a lot of log messages before nvlogclient_fast (or consumer) is able to read the messages.
Note: The first log message is not currently retrieved/stored by the Logging Client. This is a known issue.