Debug UART#

NVIDIA Tegra® Combined UART (TCU) Muxer manages debug console output to and from multiple CPU clusters.

TCU works within the Tegra UART architecture, where firmware manages the physical Tegra UART. TCU and Tegra CPU clusters (CCPLEX and R5s) use the combined UART protocol to send their debug console output to SPE/UTC.

The tcu_muxer tool is a host application that helps in muxing and demuxing debug UART output streams to/from these SOC clusters. This is achieved by using a tagging protocol implemented between tcu_muxer tool and SPE/UTC.

The tcu_muxer tool creates pseudo terminals for each SoC cluster (also known as clients) and routes UART streams to the respective pseudo terminal.

The `tcu_muxer` Tool#

Usage: ./tcu_muxer [OPTION]...
Options:
         -h       : Print this help screen
         -m <chip>: Set the client/tag mapping for <chip>. Defaults to T234.
         -c <client0>/g=<guest count>,b=<hypervisor index>/,<client1>/g=<guest count>,b=<hypervisor index>/,...: Setup the clients that can be subdivided. Defaults to g=1.
         -i       : Disable the patch for line ending
         -u       : Use separate uart_muxer tool for Guest console
         -g <int> : [DEPRECATED]Spawn <int> consoles for Hypervisor. Defaults to 1.
         -d <dev> : Select UART device <dev> to communicate with. Default: /dev/ttyUSB3
         -r <rate>: Select UART I/O speed <rate>. Default: 115200
         -b <int> : [DEPRECATED]Print Hypervisor broadcast only on console <int>. This value starts from 0 (zero).
         -s <path>: Save the output to a directory <path>
         -t       : Prefix the output with a timestamp
         -p <int> : Set the timeout for output polling ready status. Default: 300ms
         -l <path>: Save the raw output with tags to a log file <path>

In the developer environment, tcu_muxer is built as part of foundation and can be found in the following directory:

${NV_WORKSPACE}/out/embedded-foundation-*/nvidia/tools/tcu_muxer/tmake-hostcc/tcu_muxer

The tcu_muxer helper scripts can be found in:

${NV_WORKSPACE}/tools/tcu_muxer/

In SDK, tcu_muxer and the helper scripts can be found in:

${NV_WORKSPACE}/drive-foundation/tools/host/tcu_muxer/

The UART device parameter can vary depending on the platforms. For more information, see the NVIDIA DRIVE AGX™ Board Setup topic in the NVIDIA DriveOS OS SDK Developer Guide.

The `tcu_muxer` Usage in NVIDIA Native OS System#

$ /path/to/tcu_muxer -d <device>
/dev/pts/1      RCE
/dev/pts/2      FSI
/dev/pts/3      PSCFW
/dev/pts/4      DCE
/dev/pts/5      BPMP
/dev/pts/6      SCE
/dev/pts/7      SPE
/dev/pts/8      TZ
/dev/pts/9      CCPLEX: 0

This will create multiple pseudo terminals for each R5 and CCPLEX. Native OS shell can be accessed through the terminal corresponding to CCPLEX.

The `tcu_muxer` Usage in NVIDIA Virtualization System#

Virtualization System (VS) executing on CCPLEX cluster additionally has multiple UART streams corresponding to partition(s) running over Hypervisor(s). The tcu_muxer tool supports Virtualization and allows creation of pseudo terminals for each of the partitions.

Usage#

$ /path/to/tcu_muxer -d <device> -m <chip> -c <client0>/g=<guest count>,b=<hypervisor index>/,<client1>/g=<guest count>,b=<hypervisor index>/,...

chip: Represents the chip tcu_muxer is connecting to (e.g. T234,T264).
client0, client1, etc.: Represents the hypervisor client(s) that can be subdivided into guest clients.
guest count: Represents the number of virtual machines for the specified hypervisor (recommended to set to maximum which is currently 17 (16 guests + 1 hypervisor broadcast).
hypervisor index: Represents the pseudo terminal for the specified hypervisor.

Note

Only one tcu_muxer instance per UART device is supported. Running multiple tcu_muxer instances to stream from the same UART device will result in undefined behavior (for example, broken or partial output, serial console freeze, and so on).

Deprecated Usage#

$ /path/to/tcu_muxer -g <guest_nr> -b <hyp_nr> -d <device>

guest_nr: Represents the number of virtual machines (recommended to set to maximum which is currently 17 (16 guests + 1 hypervisor broadcast).
hyp_nr: Represents the pseudo terminal for Hypervisor.

Depending on the requirement, the tcu_muxer tool can be launched in different modes:

Usage Mode #1#

In this mode, tools similar to minicom are used to attach to each of the UART terminal.

$ /path/to/tcu_muxer -c ccplex/g=17,b=16/ -d /dev/ttyACM0 -m T234

/dev/pts/1      RCE
/dev/pts/2      FSI
/dev/pts/3      PSCFW
/dev/pts/4      DCE
/dev/pts/5      BPMP
/dev/pts/6      SCE
/dev/pts/7      SPE
/dev/pts/8      TZ
/dev/pts/9      CCPLEX: 0
/dev/pts/10     CCPLEX: 1
/dev/pts/11     CCPLEX: 2
/dev/pts/12     CCPLEX: 3
/dev/pts/13     CCPLEX: 4
/dev/pts/14     CCPLEX: 5
/dev/pts/15     CCPLEX: 6
/dev/pts/16     CCPLEX: 7
/dev/pts/17     CCPLEX: 8
/dev/pts/18     CCPLEX: 9
/dev/pts/19     CCPLEX: 10
/dev/pts/20     CCPLEX: 11
/dev/pts/21     CCPLEX: 12
/dev/pts/22     CCPLEX: 13
/dev/pts/23     CCPLEX: 14
/dev/pts/24     CCPLEX: 15
/dev/pts/25     CCPLEX: 16
################################# 001 ##
   /dev/pts/9 CCPLEX: [Guest OS 0]
  /dev/pts/10 CCPLEX: [Update service]
  /dev/pts/11 CCPLEX: [sysmgr]
  /dev/pts/12 CCPLEX: [bpmp_server_native]
  /dev/pts/13 CCPLEX: [vsc_server_native_sdmmc]
  /dev/pts/14 CCPLEX: [vsc_server_native_qspi]
  /dev/pts/15 CCPLEX: [vsc_server_native_ufs]
  /dev/pts/16 CCPLEX: [se_server_native]
  /dev/pts/17 CCPLEX: [nvhost_server_native]
  /dev/pts/18 CCPLEX: [audio_server_native]
  /dev/pts/19 CCPLEX: [debug-server]
  /dev/pts/25 CCPLEX: [Hypervisor]

CCPLEX: 0 corresponds to guest0. CCPLEX: 1 corresponds to guest1, and so on. CCPLEX: 16 corresponds to Hypervisor as mentioned by the -b option. On T234, guest0 is the guest OS shell (Guest OS 0). On T264, the guest OS shell can be accessed through UTC0 (VM0) as shown below.

$ /path/to/tcu_muxer -d /dev/ttyACM0 -m T264 -c CCPLEX/g=17,b=16/,TZ/g=17,b=16/
/dev/pts/1      RCE
/dev/pts/2      FSI
/dev/pts/3      PSCFW
/dev/pts/4      DCE
/dev/pts/5      BPMP
/dev/pts/6      AON-F1
/dev/pts/7      HPSE
/dev/pts/8      SB
/dev/pts/9      ADSP0
/dev/pts/10     ADSP1
/dev/pts/11     UTC0
/dev/pts/12     UTC1
/dev/pts/13     UTC2
/dev/pts/14     UTC3
/dev/pts/15     CCPLEX: 0
/dev/pts/16     CCPLEX: 1
/dev/pts/17     CCPLEX: 2
/dev/pts/18     CCPLEX: 3
/dev/pts/19     CCPLEX: 4
/dev/pts/20     CCPLEX: 5
/dev/pts/21     CCPLEX: 6
/dev/pts/22     CCPLEX: 7
/dev/pts/23     CCPLEX: 8
/dev/pts/24     CCPLEX: 9
/dev/pts/25     CCPLEX: 10
/dev/pts/26     CCPLEX: 11
/dev/pts/27     CCPLEX: 12
/dev/pts/28     CCPLEX: 13
/dev/pts/29     CCPLEX: 14
/dev/pts/30     CCPLEX: 15
/dev/pts/31     CCPLEX: 16
/dev/pts/32     TZ: 0
/dev/pts/35     TZ: 1
/dev/pts/36     TZ: 2
/dev/pts/37     TZ: 3
/dev/pts/38     TZ: 4
/dev/pts/39     TZ: 5
/dev/pts/40     TZ: 6
/dev/pts/41     TZ: 7
/dev/pts/42     TZ: 8
/dev/pts/43     TZ: 9
/dev/pts/44     TZ: 10
/dev/pts/45     TZ: 11
/dev/pts/46     TZ: 12
/dev/pts/47     TZ: 13
/dev/pts/48     TZ: 14
/dev/pts/49     TZ: 15
/dev/pts/50     TZ: 16
/dev/pts/51     RAW
################################# 001 ##
/dev/pts/15 CCPLEX: [SK-VM0]
/dev/pts/16 CCPLEX: [sysmgr]
/dev/pts/17 CCPLEX: [bpmp_server_native]
/dev/pts/18 CCPLEX: [vsc_server_native_qspi]
/dev/pts/19 CCPLEX: [vsc_server_native_ufs]
/dev/pts/20 CCPLEX: [se_server_native]
/dev/pts/21 CCPLEX: [audio_server_native]
/dev/pts/22 CCPLEX: [nvhost_server_native]
/dev/pts/23 CCPLEX: [ethernet_server_native]
/dev/pts/24 CCPLEX: [pcie_server_native]
/dev/pts/25 CCPLEX: [driveupdate_server]
/dev/pts/26 CCPLEX: [debug-server]
/dev/pts/31 CCPLEX: [Hypervisor]
/dev/pts/11 [VM0]
/dev/pts/12 [VM1]
/dev/pts/13 [VM2]
/dev/pts/14 [VM3]
################################# 001 ##
/dev/pts/15 CCPLEX: [SK-VM0]
/dev/pts/16 CCPLEX: [sysmgr]
/dev/pts/17 CCPLEX: [bpmp_server_native]
/dev/pts/18 CCPLEX: [vsc_server_native_qspi]
/dev/pts/19 CCPLEX: [vsc_server_native_ufs]
/dev/pts/20 CCPLEX: [se_server_native]
/dev/pts/21 CCPLEX: [audio_server_native]
/dev/pts/22 CCPLEX: [nvhost_server_native]
/dev/pts/23 CCPLEX: [ethernet_server_native]
/dev/pts/24 CCPLEX: [pcie_server_native]
/dev/pts/25 CCPLEX: [driveupdate_server]
/dev/pts/26 CCPLEX: [debug-server]
/dev/pts/31 CCPLEX: [Hypervisor]
/dev/pts/32 TZ: [secure-sysmgr]
/dev/pts/35 TZ: [pscfw-service]
/dev/pts/36 TZ: [crypto-asym-service]
/dev/pts/37 TZ: [pkcs11-keystore]
/dev/pts/38 TZ: [gp-lib-server]
/dev/pts/39 TZ: [testing-service]
/dev/pts/40 TZ: [testing-service-slave]
/dev/pts/41 TZ: [gp-se]
/dev/pts/42 TZ: [nvmacsec]
/dev/pts/50 TZ: [Hypervisor]
/dev/pts/11 [VM0]
/dev/pts/12 [VM1]
/dev/pts/13 [VM2]
/dev/pts/14 [VM3]

Usage Mode #2 (Per Terminal Logging)#

In this mode, output from all the terminals is logged in their respective log file.

$ /path/to/tcu_muxer -c ccplex/g=17,b=16/ -d /dev/ttyUSB2 -s /path/to/logging/dir/

/dev/pts/1      RCE
/dev/pts/2      FSI
/dev/pts/3      PSCFW
/dev/pts/4      DCE
/dev/pts/5      BPMP
/dev/pts/6      SCE
/dev/pts/7      SPE
/dev/pts/8      TZ
/dev/pts/9      CCPLEX: 0
/dev/pts/10     CCPLEX: 1
/dev/pts/11     CCPLEX: 2
/dev/pts/12     CCPLEX: 3
/dev/pts/13     CCPLEX: 4
/dev/pts/14     CCPLEX: 5
/dev/pts/15     CCPLEX: 6
/dev/pts/16     CCPLEX: 7
/dev/pts/17     CCPLEX: 8
/dev/pts/18     CCPLEX: 9
/dev/pts/19     CCPLEX: 10
/dev/pts/20     CCPLEX: 11
/dev/pts/21     CCPLEX: 12
/dev/pts/22     CCPLEX: 13
/dev/pts/23     CCPLEX: 14
/dev/pts/24     CCPLEX: 15
/dev/pts/25     CCPLEX: 16
################################# 001 ##
   /dev/pts/9 CCPLEX: [Guest OS 0]
  /dev/pts/10 CCPLEX: [Update service]
  /dev/pts/11 CCPLEX: [sysmgr]
  /dev/pts/12 CCPLEX: [bpmp_server_native]
  /dev/pts/13 CCPLEX: [vsc_server_native_sdmmc]
  /dev/pts/14 CCPLEX: [vsc_server_native_qspi]
  /dev/pts/15 CCPLEX: [vsc_server_native_ufs]
  /dev/pts/16 CCPLEX: [se_server_native]
  /dev/pts/17 CCPLEX: [nvhost_server_native]
  /dev/pts/18 CCPLEX: [audio_server_native]
  /dev/pts/19 CCPLEX: [debug-server]
  /dev/pts/25 CCPLEX: [Hypervisor]

$ ls /path/to/logging/dir/
BPMP.txt      CCPLEX11.txt  CCPLEX14.txt  CCPLEX1.txt  CCPLEX4.txt  CCPLEX7.txt  DCE.txt    RCE.txt  TZ.txt
CCPLEX0.txt   CCPLEX12.txt  CCPLEX15.txt  CCPLEX2.txt  CCPLEX5.txt  CCPLEX8.txt  FSI.txt    SCE.txt
CCPLEX10.txt  CCPLEX13.txt  CCPLEX16.txt  CCPLEX3.txt  CCPLEX6.txt  CCPLEX9.txt  PSCFW.txt  SPE.txt

Each file contains the respective terminals output.

Usage Mode #3 (Raw Logging)#

In this mode, additionally a raw file gets created that contains output of all the terminals.

$ /path/to/tcu_muxer -c ccplex/b=16,g=17 -d /dev/ttyUSB2 -l /path/to/logging/file

/dev/pts/1      RCE
/dev/pts/2      FSI
/dev/pts/3      PSCFW
/dev/pts/4      DCE
/dev/pts/5      BPMP
/dev/pts/6      SCE
/dev/pts/7      SPE
/dev/pts/8      TZ
/dev/pts/9      CCPLEX: 0
/dev/pts/10     CCPLEX: 1
/dev/pts/11     CCPLEX: 2
/dev/pts/12     CCPLEX: 3
/dev/pts/13     CCPLEX: 4
/dev/pts/14     CCPLEX: 5
/dev/pts/15     CCPLEX: 6
/dev/pts/16     CCPLEX: 7
/dev/pts/17     CCPLEX: 8
/dev/pts/18     CCPLEX: 9
/dev/pts/19     CCPLEX: 10
/dev/pts/20     CCPLEX: 11
/dev/pts/21     CCPLEX: 12
/dev/pts/22     CCPLEX: 13
/dev/pts/23     CCPLEX: 14
/dev/pts/24     CCPLEX: 15
/dev/pts/25     CCPLEX: 16
################################# 001 ##
   /dev/pts/9 CCPLEX: [Guest OS 0]
  /dev/pts/10 CCPLEX: [Update service]
  /dev/pts/11 CCPLEX: [sysmgr]
  /dev/pts/12 CCPLEX: [bpmp_server_native]
  /dev/pts/13 CCPLEX: [vsc_server_native_sdmmc]
  /dev/pts/14 CCPLEX: [vsc_server_native_qspi]
  /dev/pts/15 CCPLEX: [vsc_server_native_ufs]
  /dev/pts/16 CCPLEX: [se_server_native]
  /dev/pts/17 CCPLEX: [nvhost_server_native]
  /dev/pts/18 CCPLEX: [audio_server_native]
  /dev/pts/19 CCPLEX: [debug-server]
  /dev/pts/25 CCPLEX: [Hypervisor]

/path/to/logging/file contains the raw file with the control characters
which can be demuxed offline
using the script /path/to/tcu_muxer_raw_log_dump.py

$ /path/to/tcu_muxer_raw_log_dump.py -h
Usage:  ./tcu_muxer_raw_log_dump.py -l path [-g|-c]

-l, --logfile : Path of the file containing the raw logs captured using tcu_muxer -l option
-g, --guestid : Guest ID. Prints Hypervisor log if no value is passed
-c, --cluster : Cluster Name. One of: "RCE", "BPMP", "SCE", "SPE", "TZ", "CCPLEX"

$ /path/to/tcu_muxer_raw_log_dump.py -l /path/to/logging/file -c BPMP
dumps the BPMP logs on the stdout extracted from /path/to/logging/file

$ /path/to/tcu_muxer_raw_log_dump.py -l /path/to/logging/file -g 5
dump the CCPLEX: 5 logs on the stdout extracted from /path/to/logging/file

The raw log file is helpful in knowing the time line of events across the terminals and also in
representing the state of the system in a single file allowing the recipient
to extract the logs of their interest.

Note: Both "-s" and "-l" option can be used together.

Usage Mode #4 (tmux with `tcu_muxer`)#

This is a wrapper over the tcu_muxer tool that creates a tmux session with a separate window for each terminal.

$ /path/to/tcu_muxer_tmux.sh -c ccplex/b=16,g=17/ -d /dev/ttyUSB2 -s /path/to/logging/dir/ -l /path/to/logging/file

/dev/pts/1      RCE
/dev/pts/2      FSI
/dev/pts/3      PSCFW
/dev/pts/4      DCE
/dev/pts/5      BPMP
/dev/pts/6      SCE
/dev/pts/7      SPE
/dev/pts/8      TZ
/dev/pts/9      CCPLEX: 0
/dev/pts/10     CCPLEX: 1
/dev/pts/11     CCPLEX: 2
/dev/pts/12     CCPLEX: 3
/dev/pts/13     CCPLEX: 4
/dev/pts/14     CCPLEX: 5
/dev/pts/15     CCPLEX: 6
/dev/pts/16     CCPLEX: 7
/dev/pts/17     CCPLEX: 8
/dev/pts/18     CCPLEX: 9
/dev/pts/19     CCPLEX: 10
/dev/pts/20     CCPLEX: 11
/dev/pts/21     CCPLEX: 12
/dev/pts/22     CCPLEX: 13
/dev/pts/23     CCPLEX: 14
/dev/pts/24     CCPLEX: 15
/dev/pts/25     CCPLEX: 16
################################# 001 ##
   /dev/pts/9 CCPLEX: [Guest OS 0]
  /dev/pts/10 CCPLEX: [Update service]
  /dev/pts/11 CCPLEX: [sysmgr]
  /dev/pts/12 CCPLEX: [bpmp_server_native]
  /dev/pts/13 CCPLEX: [vsc_server_native_sdmmc]
  /dev/pts/14 CCPLEX: [vsc_server_native_qspi]
  /dev/pts/15 CCPLEX: [vsc_server_native_ufs]
  /dev/pts/16 CCPLEX: [se_server_native]
  /dev/pts/17 CCPLEX: [nvhost_server_native]
  /dev/pts/18 CCPLEX: [audio_server_native]
  /dev/pts/19 CCPLEX: [debug-server]
  /dev/pts/25 CCPLEX: [Hypervisor]

$ tmux attach
Attaches to the tmux session. There are two windows, one for all the R5 clusters
and the other for all the CCPLEX terminals.
The panes and windows can be selected using the mouse. To select text, hold shift
and select otherwise the selection is captured by tmux and will not be copied
to the system clipboard.

Note: All the options passed to tcu_muxer can be passed to tcu_muxer_tmux.sh too.

`uart_muxer` Tool with `tcu_muxer` (DEPRECATED)#

uart_muxer is the original tool developed to mux/demux UART streams from Virtual Machines. With tcu_muxer, the functionality of uart_muxer is integrated within the tcu_muxer. There are still subtle differences in the features provided by uart_muxer. To maintain the interface provided uart_muxer, it is possible to run tcu_muxer and uart_muxer in cascade form. In this configuration tcu_muxer will be responsible for mux/demux of UART streams from different clusters and then CCPLEX UART stream is forwarded by tcu_muxer to uart_muxer and which then mux/demux UART streams from different Virtual Machines.

`tcu_uart_usage` Usage#

$ /path/to/tcu_muxer -u -d <device>

Open the uart_muxer on the CCPLEX pts:

$ ${NV_WORKSPACE}/hypervisor/tools/uart_muxer/uart_muxer -g guest_nr -b hyp_nr -d <device>

Example#

$ /path/to/tcu_muxer -u -d /dev/ttyUSB2
/dev/pts/1      RCE
/dev/pts/2      FSI
/dev/pts/3      PSCFW
/dev/pts/4      DCE
/dev/pts/5      BPMP
/dev/pts/6      SCE
/dev/pts/7      SPE
/dev/pts/8      TZ
/dev/pts/9      CCPLEX: 0

# ${NV_WORKSPACE}/hypervisor/tools/uart_muxer/uart_muxer -g 9 -b 8 -d /dev/pts/104
################################# 001 ##
 /dev/pts/106 [Guest 0]
 /dev/pts/107 [BPMP]
 /dev/pts/115 [Resource Manager]
 /dev/pts/116 [Monitor Partition]
 /dev/pts/118 [I2C]
 /dev/pts/122 [System Manager]
 /dev/pts/128 [Storage]
 /dev/pts/131 [Security Engine]
 /dev/pts/132 [Hypervisor]