Logging

DRIVE OS has a logging component that allows logs from multiple DRIVE OS components to be stored (or) streamed out of DRIVE AGX hardware. Currently, support is only available for applications within DRIVE OS Linux Guest VM to send logs that can be stored in a file that can be created and is accessible from Linux Guest VM. This support will be enhanced in future releases to add more execution environments to generated logs that can be stored or streamed out of DRIVE AGX hardware.

Components

The logging pipeline is as follows :

Application(s) trying to log (referred to as Logging Entity in this document) -> libnvlog.so (logging library that is linked to Logging Entity) -> Logging Client (nvlogclient_fast) -> user configured storage file. 
In a booted DRIVE OS Linux Guest VM console, the following logging related components are available:
  1. /usr/local/bin

    nvlogclient_fast : Application component responsible for collecting logs from different Logging Entities within Linux Guest VM and storing them.

    nvlogtest : Test application that has two threads sending log messages every second.

    nvlogtest_stdin : Test application that takes input from stdin and sends each line as a log message.

  2. /usr/local/lib/

    libnvlog.so : Logging library that provides logging APIs.

Source and Binary Locations

The following lists the source and binary locations within the PDK:
  1. Pre-build binaries:
    <PDK_TOP>/ drive-linux/lib-target/libnvlog.so
    <PDK_TOP>/ drive-linux/filesystem/contents/bin/nvlogclient_fast
    <PDK_TOP>/ drive-linux/filesystem/contents/bin/nvlogtest
    <PDK_TOP>/ drive-linux/filesystem/contents/bin/nvlogtest_stdin
  2. Source code:

    Source files for the test applications (nvlogtest*) and the Logging Client (nvlogclient_fast) can be found under : <PDK_TOP>/ drive-linux/samples/tools/nvlog/.

  3. Include files and user API:

    Any application needing to send logs using this new framework must use the <PDK_TOP>/drive-linux/samples/tools/nvlog/include/nvlog.h and link to libnvlog.so header file. API documentation can be found in the same header file.

Execution Steps

  1. Logging client needs to be started on each fresh boot:
    nvlogclient_fast &
    Note: nvlogclient_fast stores the output log in the current working directory as output_recv.txt by default. This can be changed by passing nvlogclient_fast <log_file_full_path_and_name> to the Logging Client application.
  2. nvlogtest &
    tail -f -n +1 /var/log/syslog | nvlogtest_stdin > output_sender.txt &
    Note: nvlogtest and nvlogtest_stdin are two test applications that log test messages using the new logging framework.
    • nvlogtest has two threads. The main thread sends the following log message every second : "Hello World <sequence number> from main". The secondary thread sends the following log message every second: "Hello World <sequence number> from thread".
    • nvlogtest_stdin is a single threaded application that takes stdin as input and sends each line as a log message. In the above example, /var/log/syslog is read by nvlogtest_stdin and each line is sent as a log message.
  3. sleep 20
    pkill nvlog

The commands above allow logging components to operate for 20 seconds and then kills them. Output from nvlogclient_fast can now be read from the output_recv.txt file in the current working directory.

Sample Output

sort -k4 -n output_recv.txt > output_recv_sorted.txt
cat output_recv_sorted.txt

0    0    1    1656242248580689    266 - Jun 15 05:30:49 tegra-ubuntu kernel: [    0.000000] Linux version 5.10.104-rt63-tegra (buildbrain@mobile-u64-5385-d8000) (aarch64-buildroot-linux-gnu-gcc.br_real (Buildroot 2020.08) 9.3.0, GNU ld (GNU Binutils) 2.33.1) #1 SMP PREEMPT RT Tue Jun 14 10:46:31 PDT 2022
0    0    2    1656242248580690    108 - Jun 15 05:30:49 tegra-ubuntu kernel: [    0.000000] OF: fdt: memory scan node memory@80000000, reg size 32,
0    0    3    1656242248580691    85 - Jun 15 05:30:49 tegra-ubuntu kernel: [    0.000000] OF: fdt:  - 80000000 ,  80000000
0    0    4    1656242248580691    87 - Jun 15 05:30:49 tegra-ubuntu kernel: [    0.000000] OF: fdt:  - 100000000 ,  70d800000
0    0    5    1656242248580692    78 - Jun 15 05:30:49 tegra-ubuntu kernel: [    0.000000] Machine model: p3710-0010
0    0    6    1656242248580692    73 - Jun 15 05:30:49 tegra-ubuntu kernel: [    0.000000] efi: UEFI not found.
0    0    7    1656242248580693    141 - Jun 15 05:30:49 tegra-ubuntu kernel: [    0.000000] OF: reserved mem: initialized node camdbg_carveout, compatible id nvidia,camdbg_carveout
0    0    8    1656242248580694    129 - Jun 15 05:30:49 tegra-ubuntu kernel: [    0.000000] Reserved memory: created CMA memory pool at 0x00000007fd800000, size 256 MiB
0    0    10    1656242248580695    65 - Jun 15 05:30:49 tegra-ubuntu kernel: [    0.000000] Zone ranges:
0    0    9    1656242248580695    128 - Jun 15 05:30:49 tegra-ubuntu kernel: [    0.000000] OF: reserved mem: initialized node linux,cma, compatible id shared-dma-pool
0    0    11    1656242248580696    107 - Jun 15 05:30:49 tegra-ubuntu kernel: [    0.000000]   DMA      [mem 0x0000000080000000-0x00000000ffffffff]
0    0    12    1656242248580696    69 - Jun 15 05:30:49 tegra-ubuntu kernel: [    0.000000]   DMA32    empty
0    0    13    1656242248580697    107 - Jun 15 05:30:49 tegra-ubuntu kernel: [    0.000000]   Normal   [mem 0x0000000100000000-0x000000080d7fffff]
0    0    14    1656242248580697    85 - Jun 15 05:30:49 tegra-ubuntu kernel: [    0.000000] Movable zone start for each node
0    0    15    1656242248580698    77 - Jun 15 05:30:49 tegra-ubuntu kernel: [    0.000000] Early memory node ranges
0    0    16    1656242248580698    108 - Jun 15 05:30:49 tegra-ubuntu kernel: [    0.000000]   node   0: [mem 0x0000000080000000-0x00000000dd33ffff]
0    0    17    1656242248580699    108 - Jun 15 05:30:49 tegra-ubuntu kernel: [    0.000000]   node   0: [mem 0x00000000dd340000-0x00000000dd37ffff]
0    0    18    1656242248580699    108 - Jun 15 05:30:49 tegra-ubuntu kernel: [    0.000000]   node   0: [mem 0x00000000dd380000-0x0000000199bfffff]
0    0    19    1656242248580700    108 - Jun 15 05:30:49 tegra-ubuntu kernel: [    0.000000]   node   0: [mem 0x00000001f9c00000-0x000000080d7fffff]
0    0    20    1656242248580700    117 - Jun 15 05:30:49 tegra-ubuntu kernel: [    0.000000] Initmem setup node 0 [mem 0x0000000080000000-0x000000080d7fffff]
....
0    0    1312    1656242248628536    88 - Jun 15 05:30:49 tegra-ubuntu systemd-networkd[1317]: mgbe3_0: IPv6 successfully enabled
0    0    1313    1656242248628537    88 - Jun 15 05:30:49 tegra-ubuntu systemd-networkd[1317]: mgbe2_0: IPv6 successfully enabled
0    0    1314    1656242248628538    88 - Jun 15 05:30:49 tegra-ubuntu systemd-networkd[1317]: mgbe1_0: IPv6 successfully enabled
0    0    1315    1656242248628538    88 - Jun 15 05:30:49 tegra-ubuntu systemd-networkd[1317]: mgbe0_0: IPv6 successfully enabled
0    0    1316    1656242248628539    87 - Jun 15 05:30:49 tegra-ubuntu systemd-networkd[1317]: eqos_0: IPv6 successfully enabled
0    0    1317    1656242248628539    104 - Jun 15 05:30:50 tegra-ubuntu kernel: [  149.652480] 8021q: adding VLAN 0 to HW filter on device mgbe3_0
0    0    1318    1656242248628540    133 - Jun 15 05:30:50 tegra-ubuntu kernel: [  149.655826] nvethernet 6b10000.ethernet mgbe3_0: Link is Up - 10Gbps/Full - flow control off
0    0    1319    1656242248628541    111 - Jun 15 05:30:50 tegra-ubuntu kernel: [  149.665770] hwmon hwmon1: temp1_input not attached to any thermal zone
0    0    2    1656242248629107    Hello World 1 from main
0    0    3    1656242248629145    Hello World 1 from thread
0    0    1320    1656242248629609    104 - Jun 15 05:30:50 tegra-ubuntu kernel: [  149.706221] 8021q: adding VLAN 0 to HW filter on device mgbe1_0
0    0    1321    1656242248629612    120 - Jun 15 05:30:50 tegra-ubuntu kernel: [  149.706370] Aquantia AQR113C 6910000.ethernet:00: No AQR phy_mode setting in DT
0    0    1322    1656242248629613    77 - Jun 15 05:30:50 tegra-ubuntu systemd-resolved[1383]: Positive Trust Anchors:
...
0    0    98    1656242296632042    Hello World 49 from thread
0    0    99    1656242296632068    Hello World 49 from main
0    0    100    1656242297632090    Hello World 50 from thread
0    0    101    1656242297632135    Hello World 50 from main
0    0    102    1656242298632127    Hello World 51 from thread
0    0    103    1656242298632199    Hello World 51 from main
0    0    104    1656242299632158    Hello World 52 from thread
0    0    105    1656242299632244    Hello World 52 from main
0    0    106    1656242300632222    Hello World 53 from thread
0    0    107    1656242300632308    Hello World 53 from main
0    0    108    1656242301632286    Hello World 54 from thread
0    0    109    1656242301632373    Hello World 54 from main
0    0    110    1656242302632349    Hello World 55 from thread
0    0    111    1656242302632439    Hello World 55 from main
0    0    112    1656242303632365    Hello World 56 from thread
0    0    113    1656242303632504    Hello World 56 from main
<End of File>
Note: Raw output from Logging Client is not sequential with regards to generated logs. This is due to the underlying implementation where readers and writers are lockless and the reader retrieves messages in bursts (in reverse order for each burst). The raw output file must be sorted on the basis of timestamp (column 4) to get the logs from multiple sources in order.