Debug Server#
Hypervisor Debug Console#
The Hypervisor provides a debug console to assist in debugging and profiling of Hypervisor partitions. The debug console presents an interactive shell that accepts the commands described in the rest of this section. It can be accessed using the tcu_muxer tool on the pseudo-terminal belonging to the debug-server partition. Instructions for using tcu_muxer can be found in the Tegra Combined UART section. The debug console can be recognized by its shell prompt >>.
Terminology#
Abbreviation or Term |
Description |
|---|---|
PCPU |
Physical CPU. |
LCPU |
Logical CPU, LCPU, and PCPU are the same if RIST is not enabled in the system. |
vCPU |
Virtual CPU running on LCPU. |
HV |
Hypervisor that runs at Exception Level 2 (NS-EL2) in non-secure world. |
Secure world |
ARM TrustZone. |
Secure application |
Software running at Exception Level 0 (S-EL0) in secure world. |
Secure kernel |
Software running at Exception Level 1 (S-EL1) in secure world. |
Idle |
Unused CPU time. |
Debug Server Commands#
The Hypervisor Debug Console provides the following commands to access system information.
Command |
Description |
|---|---|
|
Prints available commands. |
|
Lists information about all of the partitions (Virtual Machines and Processes) in the non-secure and secure world, including the number of threads in the partition and its priority. |
|
Prints the register context of all vCPUs or threads of |
|
Provides runtime CPU usage with 1 second interval. |
Running the ‘top’ Command#
The top command displays CPU usage in real time while the system is running. This command is similar to the top command found in Unix-like operating systems.
The top command is particularly useful for diagnosing starvation where it is suspected that a partition is not being given enough CPU time. If the system load is too high, it should be immediately apparent from the output of the top command. On the other hand if a high proportion of the system time is spent idle, that is an indication that the problem lies elsewhere.
An example of the top command output is provided below:
Note
To save space the command output is truncated to five CPUs.
up time: 3:20:13:37, Tasks: 100 total
>>
NAME [PRI] LCPU: 00 LCPU: 01 LCPU: 02 LCPU: 03 LCPU: 04
Guest OS 0 [2] 0.0212% 0.0421% 0.0185% 0.0048% 0.0279%
Update service [3] 0.0037% 0.0000% 0.0000% 0.0000% 0.0000%
sysmgr [1] 0.0000% 0.0000% 0.0000% 0.0000% 0.0000%
bpmp_server_native [1] 0.0140% 0.0000% 0.0000% 0.0000% 0.0000%
vsc_server_native_sdmmc [1] 0.0000% 0.0000% 0.0000% 0.0000% 0.0000%
vsc_server_native_qspi [1] 0.0000% 0.0000% 0.0000% 0.0000% 0.0000%
vsc_server_native_ufs [1] 0.0002% 0.0000% 0.0046% 0.0000% 0.0017%
se_server_native [1] 0.0000% 0.0000% 0.0000% 0.0000% 0.0000%
nvhost_server_native [1] 0.0000% 0.0000% 0.0000% 0.0000% 0.0000%
audio_server_native [1] 0.0000% 0.0000% 0.0000% 0.0000% 0.0000%
debug-server [0] 0.0000% 0.0462% 0.0000% 0.0000% 0.0000%
crypto-asym-service [1] 0.0000% 0.0000% 0.0000% 0.0000% 0.0000%
crypto-sym-service [1] 0.0000% 0.0000% 0.0000% 0.0000% 0.0000%
pkcs11-keystore [1] 0.0000% 0.0000% 0.0000% 0.0000% 0.0000%
gp-lib-server [2] 0.0000% 0.0000% 0.0000% 0.0000% 0.0000%
testing-service [1] 0.0000% 0.0000% 0.0000% 0.0000% 0.0000%
testing-service-slave [1] 0.0000% 0.0000% 0.0000% 0.0000% 0.0000%
gp-se [1] 0.0000% 0.0000% 0.0000% 0.0000% 0.0000%
pscfw-service [1] 0.0000% 0.0000% 0.0000% 0.0000% 0.0000%
nvmacsec [1] 0.0000% 0.0000% 0.0000% 0.0000% 0.0000%
smmu-se-test [1] 0.0000% 0.0000% 0.0000% 0.0000% 0.0000%
tz-se_server [1] 0.0000% 0.0000% 0.0000% 0.0000% 0.0000%
secure-sysmgr [1] 0.0000% 0.0000% 0.0000% 0.0000% 0.0000%
S-Idle [X] 0.0000% 0.0000% 0.0000% 0.0000% 0.0000%
S-Kernel [X] 0.0000% 0.0000% 0.0000% 0.0000% 0.0000%
HV [X] 0.9139% 0.3098% 0.0041% 0.0028% 0.0085%
NS-Idle [X] 99.0480% 99.6188% 99.9725% 99.9763% 99.9699%
Explanation of the preceding example:
up time: The amount of time that the system has been running. In this example, 3 days, 20 hours, 13 minutes, and 37 seconds.Tasks: The total number of schedulable Tasks (vCPUs and threads) in the system, in this example 100.NAME: An entity name that consumes LCPU.Guest OS 0todebug serverare partitions running in non-secure world.Entities between
debug-serverandS-Idlerepresent partitions running in secure-world.S-IdleandS-Kernelrepresent LCPU time spent in idle with kernel in secure world.HVandIdlerepresent LCPU time spent in idle with kernel in non-secure world.
PRI: The configured PCT priority value. The lower the number, the higher the priority.X: A special case that represents system components priority.
Running the ‘mem’ Command#
The mem command provides a static view of how memory is being used in the virtualization system. An example of the mem command output is provided below:
Note
To save space the command output is truncated to eight CPUs.
>>mem
################## HYPERVISOR RAM INFO ##################
Total RAM available in system: 32768 MB
Size of RAM allocated for carveout regions: 1357 MB
Total size of bad blocks in the system: 0 MB
Size of HV memory partition: 96 MB
Size of HV image (binary): 832 KB
Space allocated for HV heap: 40 MB
Size of HV heap: 43 MB
Size of CPIO archive: 28 MB
Unused RAM: 12 MB
################## VM RAM INFO ##################
NAME TASK ID TOTAL RAM
Guest OS 0 0 31016 MB
################## PROCESS RAM INFO ##################
NAME TASK ID HEAP SIZE (KB) STACK SIZE (KB)
sysmgr 2 384 768
bpmp_server_native 3 64 4
vsc_server_native_sdmmc 4 2048 768
vsc_server_native_qspi 5 2048 768
vsc_server_native_ufs 6 2048 768
se_server_native 7 8192 768
nvhost_server_native 8 8192 64
audio_server_native 9 8192 4
driveupdate_server 10 16384 64
debug-server 11 1024 768
################## IVC INFO ##################
Number of IVC queues: 116
Total size of IVC queues: 7 MB
Number of IVC mempools: 33
Total size of IVC mempools: 361 MB
################## OTHER ##################
Total Virtualization Usage: 465 MB
The following is an explanation of the statistics in the HYPERVISOR RAM INFO section:
Total RAM available in system: the amount of DRAM available for use by the virtualization system.Size of RAM allocated for carveout regions: the combined size of all carveout regions.Total size of bad blocks in the system: the combined size of pages retired by DRAM ECC .Size of HV memory partition: the size of the HV image after being loaded into memory.Size of HV image (binary): the size of the HV image.Space allocated for HV heap: the amount of memory allocated for use as the hypervisor’s heap.Size of HV heap: the remaining free space in the hypervisor’s heap.Size of CPIO archive: size of the archive containing the all of the ELF images that are loaded by the hypervisor.Unused RAM:the amount of unused memory reserved for the virtualization system.
The following is an explanation of the statistics in the VM RAM INFO section:
NAME: the name of the VM.TASK ID: the Task ID as reported by thelistcommand.TOTAL RAM: the total amount of memory allocated to the VM.
The following is an explanation of the statistics in the PROCESS RAM INFO section:
NAME: the name of the process.TASK ID: the Task ID of the process as reported by thelistcommand.HEAP SIZE (KB): the total amount of memory reserved for heap usage across all threads belonging to the process.STACK SIZE (KB): the total amount of memory reserved for stack usage across all threads belonging to the process.
The following is an explanation of the statistics in the OTHER section:
Total Virtualization Usage: this is the sum of the sizes of the hypervisor partition, and all IVC queues and mempools.
Running the ‘interrupt’ Command#
The interrupt command displays interrupt counters in real time while the system is running. An example of the interrupt command output is provided below:
Number of hardware supported interrupt lines available: 991
***************************************************************************
PCPU 0 1 2 3 4 5 6 7
0 0 1 1 1 1 1 1 1 SGI
1 537 3838 3505 5424 65611 2044 677 1295 SGI
2 7 5 5 4 5 5 5 5 SGI
3 14384 4968 471 439 439 438 438 438 SGI
26 965 3246 82085 48 575 960 174 970 PPI
27 6 23 20 23 73 12 8 453 PPI
58 0 0 0 0 6 0 0 0 SPI
67 0 0 4 0 0 0 0 0 SPI
76 61 0 25 0 0 0 0 0 SPI
97 19 0 20 0 0 0 0 0 SPI
159 161 0 0 0 0 0 0 0 SPI
168 1281962 0 0 0 0 0 0 0 SPI
195 0 0 0 0 91 0 0 0 SPI
208 5804 0 0 0 0 0 0 0 SPI
298 0 0 0 0 2 0 0 0 SPI
416 0 0 0 0 0 0 0 0 SPI
800 1 0 0 0 0 0 0 0 SPI
801 1 0 0 0 0 0 0 0 SPI
802 1 0 0 0 0 0 0 0 SPI
803 1 0 0 0 0 0 0 0 SPI
804 17 0 0 0 0 0 0 0 SPI
805 1 0 0 0 0 0 0 0 SPI
806 1 0 0 0 0 0 0 0 SPI
This grid is a matrix with each row corresponding to an interrupt line and each column corresponding to a CPU. The matrix can be read as follows:
The first row beneath the
********shows the CPU ID represented by each columnThe first column shows the interrupt ID represented by each row. The last column notes what type of interrupt it is: SGI, SPI, or PPI.
The value at each cell shows the interrupt count for the given interrupt line on the given CPU.