Debug Server#

Overview#

The Debug Server provides an interactive console interface for debugging and profiling HVRTOS partitions. It presents a shell-like interface that accepts commands to inspect system state, view task contexts, monitor CPU and memory usage, and collect profiling data.

The Debug Server can be accessed through:

Serial Interface: Via tcu_muxer tool on the pseudo-terminal belonging to the debug-server partition
IVC Interface: Via a VM console virtual terminal.

The debug console can be recognized by its shell prompt >>.

Default interface is serial, IVC interface is disabled by default.

Drive OS Profiles#

This feature is available in the following Drive OS profiles:

Profile	Availability	Notes
`dev_nsr`, `debug_sr`	✓ Available	Full debug capabilities enabled
`prod_sr`, `prod_nsr`, `test_sr`, `test_nsr`	✗ Not Available	Not available in production and test builds

Configuration#

Platform Configuration Table (PCT)

The Debug Server is controlled via PCT profile makefiles. To enable the Debug Server, set the following variable in your profile makefile:

# In profile makefile (e.g., profile_dev_nsr.mk, profile_debug_sr.mk)
ENABLE_HVRTOS_DEBUG_SERVER := y

To enable IVC interface, set the following variable in your profile makefile:

# In profile makefile (e.g., profile_dev_nsr.mk, profile_debug_sr.mk)
ENABLE_HVRTOS_IVC_DEBUG_CONSOLE := y

Note: IVC interface is disabled by default.

Limitations#

Limitation	Description	Impact
Secure Process VCPU Dump	Secure process VCPU context dump is not supported	Use alternative debugging methods for secure processes
Task Count Limit	Maximum 32 tasks can be displayed (16 per world)	System with more tasks will show truncated output
Task Name Length	Task names are limited to 31 characters	Longer names will be truncated
VCPU Count Limit	Maximum 32 vCPUs per VM	VMs with more vCPUs will show truncated register context
Stack Trace Depth	Limited to 64 frames	Deep call stacks may be truncated
Interrupt Lines	Maximum 1024 interrupt lines supported	Systems with more interrupt lines may show incomplete data
Runtime Max-Proc-Time Updates	Runtime threshold updates apply to non-secure processes only	Secure processes use values from the PCT only
Interface support	IVC and serial (UART) debug-console transports are mutually exclusive.	Enabling both in the same integration is not supported.
Events	events command is not supported for IVC interface.	To use events command, use serial interface.

Detailed Usage#

Command Reference#

The Debug Server provides the following commands:

Command	Description
`help` or `h`	Prints available commands with usage information
`list` or `l`	Lists all tasks (processes and VMs) in the system with their IDs, thread counts, and priorities
`vcpu <task_ID>` or `v <task_ID>`	Prints register context of all vCPUs/threads for the specified task; may include a stack trace (depth limited; see Limitations)
`top [<msec>] [<usec_flag>]` or `t [<msec>] [<usec_flag>]`	Displays CPU utilization with configurable refresh rate
`mem` or `m`	Prints memory usage statistics for partitions and virtualization system
`interrupt [<msec>]` or `i [<msec>]`	Displays per-PCPU interrupt counts for each interrupt source
`events [<trace_ID>]` or `e [<trace_ID>]`	Prints eventlib traces for the specified producer
`proc` or `p`	Displays per-thread counts of max-proc-time violations (see Max-Proc-Time in Glossary) for all tasks
`PROC`	Clears max-proc-time violation counters for all tasks
`PROC <time_us>`	Sets max-proc-time threshold in microseconds (valid range: 10-1000000, default: 40)
`q`	Quits/clears current command (useful for stopping `top` or `interrupt`)
`Ctrl+C`	Interrupts current command

Access Methods#

Serial Interface#

Access via tcu_muxer tool:

tcu_muxer -d /dev/pts/X

Where /dev/pts/X is the pseudo-terminal for the debug-server partition.

IVC interface (Debug Server over IVC)#

A Linux VM can use the same Debug Shell (>> prompt and commands as in Command Reference) over IVC, without attaching tcu_muxer on the host. Enabling this path requires updating both the PCT and the Linux guest device tree as part of the profile build.

Scope

Supported for Linux guests only (as of current integration).
Requires Debug Server and IVC Debug Console enabled in the profile makefile (see Configuration): ENABLE_HVRTOS_DEBUG_SERVER and ENABLE_HVRTOS_IVC_DEBUG_CONSOLE.

After build and flash

The guest should expose a TTY device node:

/dev/ttyTGRHVC0

Use any terminal emulator on the guest that can open that device, for example:

minicom -D /dev/ttyTGRHVC0

Using ``echo`` and ``cat`` (no terminal emulator)

Configure the TTY so output is usable from the default console, send a command, then read the reply:

# Configure the TTY to display correctly on the default console
stty -F /dev/ttyTGRHVC0 -echo -icrnl

# Send a command
echo "mem" > /dev/ttyTGRHVC0

# Read the response
cat /dev/ttyTGRHVC0

The IVC path buffers command output so the Linux VM can consume debug text without direct access to the debug-server partition serial port.

Command Details#

Running the `list` Command#

The list command displays all tasks (native processes and VMs) in both secure and non-secure worlds.

Example Output:

>>list

################## NON-SECURE WORLD ##################

Total Tasks (Native Processes and VMs) in the System: 5

NAME                            TASK ID         THREAD COUNT    PRIORITY

Guest OS 0                       1               4               2
sysmgr                           2               1               1
debug-server                     3               1               0
bpmp_server_native               4               1               1
vsc_server_native_sdmmc          5               1               1

################## SECURE WORLD ##################

Total Secure Processes in the System: 2

NAME                            TASK ID         THREAD COUNT    PRIORITY

secure-sysmgr                    16              1               1
tz-se_server                     17              1               1

Running the `vcpu` Command#

The vcpu command dumps the register context for all threads/vCPUs of a specified task. The output may also include a stack trace for each thread when available.

Usage: vcpu <task_ID>

Example:

>>vcpu 1

################## VCPU CONTEXT FOR TASK ID: 1 ##################

LCPU ID: 0
PC: 0x80001234
LR: 0x80005678
SP: 0x90000000
PSTATE: 0x000003c5
...
[Register dump continues]

Running the `top` Command#

The top command displays CPU utilization in real-time with configurable refresh rate.

Usage: top [<msec_refresh_rate>] [<usec_CPU_utilization_val>]

<msec_refresh_rate>: Refresh interval in milliseconds (default: 3000)
<usec_CPU_utilization_val>: - 0 (default): Display as percentage - Non-zero: Display in microseconds

Examples:

>>top
# Displays CPU percentage usage with 3000ms refresh rate

>>top 2000
# Displays CPU percentage usage with 2000ms refresh rate

>>top 2000 1
# Displays CPU usage in microseconds with 2000ms refresh rate

Example Output:

up time: 3:20:13:37, Tasks: 100 total
>>
NAME                     [PRI] LCPU: 00 LCPU: 01 LCPU: 02 LCPU: 03 LCPU: 04

Guest OS 0               [2]    0.0212%  0.0421%  0.0185%  0.0048%  0.0279%
Update service           [3]    0.0037%  0.0000%  0.0000%  0.0000%  0.0000%
sysmgr                   [1]    0.0000%  0.0000%  0.0000%  0.0000%  0.0000%
bpmp_server_native       [1]    0.0140%  0.0000%  0.0000%  0.0000%  0.0000%
vsc_server_native_sdmmc  [1]    0.0000%  0.0000%  0.0000%  0.0000%  0.0000%
vsc_server_native_qspi   [1]    0.0000%  0.0000%  0.0000%  0.0000%  0.0000%
vsc_server_native_ufs    [1]    0.0002%  0.0000%  0.0046%  0.0000%  0.0017%
se_server_native         [1]    0.0000%  0.0000%  0.0000%  0.0000%  0.0000%
nvhost_server_native     [1]    0.0000%  0.0000%  0.0000%  0.0000%  0.0000%
audio_server_native      [1]    0.0000%  0.0000%  0.0000%  0.0000%  0.0000%
debug-server             [0]    0.0000%  0.0462%  0.0000%  0.0000%  0.0000%
crypto-asym-service      [1]    0.0000%  0.0000%  0.0000%  0.0000%  0.0000%
crypto-sym-service       [1]    0.0000%  0.0000%  0.0000%  0.0000%  0.0000%
pkcs11-keystore          [1]    0.0000%  0.0000%  0.0000%  0.0000%  0.0000%
gp-lib-server            [2]    0.0000%  0.0000%  0.0000%  0.0000%  0.0000%
testing-service          [1]    0.0000%  0.0000%  0.0000%  0.0000%  0.0000%
testing-service-slave    [1]    0.0000%  0.0000%  0.0000%  0.0000%  0.0000%
gp-se                    [1]    0.0000%  0.0000%  0.0000%  0.0000%  0.0000%
pscfw-service            [1]    0.0000%  0.0000%  0.0000%  0.0000%  0.0000%
nvmacsec                 [1]    0.0000%  0.0000%  0.0000%  0.0000%  0.0000%
smmu-se-test             [1]    0.0000%  0.0000%  0.0000%  0.0000%  0.0000%
tz-se_server             [1]    0.0000%  0.0000%  0.0000%  0.0000%  0.0000%
secure-sysmgr            [1]    0.0000%  0.0000%  0.0000%  0.0000%  0.0000%
S-Idle                   [X]    0.0000%  0.0000%  0.0000%  0.0000%  0.0000%
S-Kernel                 [X]    0.0000%  0.0000%  0.0000%  0.0000%  0.0000%
HV                       [X]    0.9139%  0.3098%  0.0041%  0.0028%  0.0085%
NS-Idle                  [X]   99.0480% 99.6188% 99.9725% 99.9763% 99.9699%

Note: Press Ctrl+C or q to stop the command.

Explanation of the preceding example output:

up time: The amount of time that the system has been running. In this example, 3 days, 20 hours, 13 minutes, and 37 seconds.
Tasks: The total number of schedulable Tasks (vCPUs and threads) in the system, in this example 100.
NAME: An entity name that consumes LCPU.
- Guest OS 0 to debug server are partitions running in non-secure world.
- S-Idle and S-Kernel represent LCPU time spent in idle with kernel in secure world.
- HV and Idle represent LCPU time spent in idle with kernel in non-secure world.
PRI: The configured PCT priority value. The lower the number, the higher the priority.
- X: A special case that represents system components priority.
%: CPU utilization in percentage for an entity on LCPU. Calculated based on last refresh interval.

Running the `mem` Command#

The mem command provides detailed memory usage statistics.

Usage: mem

Example Output:

>>mem

################################## MEMORY UTILIZATION ##################################
 1. Total RAM                              -   65536 MB
       1.1 Total Size of Bad Blocks            -   0 KB
       1.2 Total Size of Carveouts             -   2598 MB
               1.2.1 CARVEOUT_CCPLEX_SMMU_PTW            -   160 MB
                       1.2.1.1 NON-SECURE HYPERVISOR           -   144 MB
                               1.2.1.1.1 Text Size                       -   384 KB
                               1.2.1.1.2 RO Size                         -   192 KB
                               1.2.1.1.3 RW Size                         -   64 KB
                               1.2.1.1.4 BSS Size                        -   610 KB
                               1.2.1.1.5 Blob Size                       -   28672 KB
                               1.2.1.1.6 Init Stack Size                 -   24 KB
                               1.2.1.1.7 Page Tables Size                -   24576 KB
                               1.2.1.1.8 Heap Size (Used / Available)    -   13 MB / 90 MB
       1.3 Total Guest VM RAM                  -   61722 MB
               Guest VM 0                      -   61722 MB
       1.4 Allocated RAM (excluding guest VMs) -   1214 MB
       1.5 Unused RAM                          -   1 MB
 2. Total Virtualization Usage             -   1694 MB
 3. HVRTOS Processes
       NAME                            HEAP SIZE (KB)  STACK SIZE (KB)
       sysmgr                          384             896
       debug-server                    1024            896
 4. IVC Queues & Mempools
       4.1 IVC Queue Count                     -   103
       4.2 Total Size of IVC Queues            -   6 MB
 5. SECURE HYPERVISOR
       5.1 RTOS Kernel                         -   61440 KB
               5.1.1 Text Size                           -   256 KB
               5.1.7 Heap Size (Used / Available)        -   5668 KB / 34096 KB

Running the `interrupt` Command#

The interrupt command displays interrupt counters per PCPU in real-time.

Usage: interrupt [<msec_refresh_rate>]

<msec_refresh_rate>: Refresh interval in milliseconds (default: 3000)

Example Output:

>>interrupt

Number of hardware supported interrupt lines available: 991
***************************************************************************
PCPU         0       1       2       3       4       5       6       7
  0          0       1       1       1       1       1       1       1  SGI
  1        537    3838    3505    5424   65611    2044     677    1295  SGI
  3      14384    4968     471     439     439     438     438     438  SGI
 26        965    3246   82085      48     575     960     174     970  PPI
 58          0       0       0       0       6       0       0       0  SPI
168     1281962      0       0       0       0       0       0       0  SPI

Note: Press Ctrl+C or q to stop the command.

This grid is a matrix with each row corresponding to an interrupt line and each column corresponding to a CPU. The matrix can be read as follows:

The first row beneath the ******** shows the CPU ID represented by each column
The first column shows the interrupt ID represented by each row. The last column notes what type of interrupt it is: SGI, SPI, or PPI.
The value at each cell shows the interrupt count for the given interrupt line on the given CPU.

Running the `events` Command#

The events command displays eventlib traces for producers.

Usage: events [<trace_ID>]

No argument: Lists all available eventlib producer names with their trace IDs
With <trace_ID>: Prints event traces for the specified producer with the given trace ID.

Example:

>>events
# Lists all available producers

>>events 4
# Prints traces for task with trace_ID = 4

Running the `proc` Command#

The proc command shows how often each thread exceeded the max-proc-time threshold: a violation is counted when a task runs continuously longer than the configured max-proc-time slice. The display is per PCPU column for each task.

Usage: proc

Example Output:

>>proc

########## Max-proc-timeout: 40 microseconds #########
################## Max-proc-violations ##################

TASK NAME                          LCPU0   LCPU1   LCPU2   LCPU3
sysmgr                              2       0       1       0

Running the `PROC` Command#

The PROC command has two modes:

Clear violation counters: PROC (no arguments) - Clears all max-proc-time violation counters
Set threshold: PROC <time_us> - Sets max-proc-time threshold in microseconds

Usage:

>>PROC
# Clears violation counters

>>PROC 50
# Sets threshold to 50 microseconds (valid range: 10-1000000, default: 40)

Configuration Details:

Valid range: 10-1000000 microseconds
Default value: 40 microseconds
Can also be configured in the PCT using the max_proc_time_us field
When threshold is changed, violation counters are automatically reset
Runtime updates are applicable to non-secure processes only

Debug Server#

Overview#

Drive OS Profiles#

Configuration#

Limitations#

Detailed Usage#

Command Reference#

Access Methods#

Serial Interface#

IVC interface (Debug Server over IVC)#

Command Details#

Running the list Command#

Running the vcpu Command#

Running the top Command#

Running the mem Command#

Running the interrupt Command#

Running the events Command#

Running the proc Command#

Running the PROC Command#

Running the `list` Command#

Running the `vcpu` Command#

Running the `top` Command#

Running the `mem` Command#

Running the `interrupt` Command#

Running the `events` Command#

Running the `proc` Command#

Running the `PROC` Command#