Debug Scenarios#

Overview#

This section provides guidance on using HVRTOS debug and profiling capabilities to troubleshoot common issues. Scenarios use a below layout:

  • Problem — What the failure looks like

  • Debug Approach — Ordered checks and Debug Server commands (where applicable)

  • Common Issues — Typical root causes to validate

The scenarios below start with a few high-traffic cases (guest boot and unresponsiveness, native server/process health). Additional scenarios can be expanded over time.

Guest VM Did Not Boot#

Problem: Guest VM fails to boot or hangs during boot.

Debug Approach:

  1. View boot logs:

    • Check Hypervisor, Sidekick and VM UART logs for boot-time errors

    • Look for assertion failures or exceptions in Foundation server logs

    • Check FSI or MCU UART logs for reported errors, Foundation servers reports errors with reporting ID ranging from 0x8200 to 0x82FF

  2. Guest starts but hangs during boot:

    • Use the Debug Server vcpu command with the guest task ID and inspect exception state (for example ESR_ELx) in the register dump

    • vcpu dump will also include the stack trace along with exception state which may help in identifying the root cause

Common Issues:

  • Foundation server or sidekick or Hypervisor hit an assertion or exception which prevents guest VM from booting

  • Invalid memory configuration in the Platform Configuration Table (PCT): guest RAM size, IVC configuration

  • Missing or incorrect device tree for the guest

Guest VM Not Responding#

Problem: Guest VM is running but not responding to user inputs.

Debug Approach (Debug Server commands):

  1. Check CPU usage:

    >>top
# Verify guest is consuming CPU time
# Check if guest is stuck in idle

  2. VCPU context dump (registers for each thread/vCPU of the task):

    >>vcpu <guest_task_id>
# Check PC location - is it in expected code?
# Look for exception state

  3. Interrupt check:

    >>interrupt
# Look for unexpected interrupt counts
# Check for interrupt storms

Common Issues:

  • Guest stuck in an infinite loop (check PC in vcpu dump)

  • Guest did not receive enough CPU time because Foundation servers are stealing CPU from guest VM (check CPU usage in top output)

System State Transitions#

Problem: System fails during state transitions (suspend/resume, power state changes).

Debug Approach:

  1. UART logs:

    • Check System Manager UART logs for state transition messages. Pay attention to latest messages.

    • System Manager logs all partitions progress during state transition.

Common Issues:

  • Foundation server or other partitions are not responding to system state transition requests

  • All Foundation servers are responded, though Guest VM did not resume or shutdown properly

HV RTOS Server Not Running#

Problem: Native HVRTOS server process is not running or crashed.

Debug Approach (Debug Server and logs):

  1. List tasks:

    >>list
# Verify server is in task list

  2. Check UART logs: - Look for server initialization errors and check for assertion failures.

  3. Process context dump (if the task exists but is hung):

    >>vcpu <server_task_id>
# Check register state; stack backtrace may appear in the dump.

Common Issues:

  • Initialization failure (check UART logs)

  • Startup-time fault or assertion (use vcpu if the task is listed)

  • HVRTOS server process may have dependency on other native servers or processes that are not running.

System Not Responding#

Problem: Entire system is unresponsive, no debug output.

Debug Approach:

  1. Check system health:

    • Check Debug server OR other foundation servers console is responsive. If yes, system is responsive.

    • If no, check last UART log messages from Hypervisor, Sidekick and VM.

    • Check FSI or MCU UART logs for fatal errors which contains which component is causing the issue.

  2. JTAG Debugging: - If available, use external JTAG probe to halt system and examine state

  3. Hardware Issues: - Verify power supply and check for thermal issues

Common Issues:

  • Hypervisor or ATF is hit an assertion or exception which prevents system from responding

  • Check FSI or MCU UART logs for reported errors.

    • Non-Secure world reports errors with reporting ID ranging from 0x8200 to 0x82FF

    • Secure world reports errors with reporting ID ranging from 0x8300 to 0x83FF

  • Hardware failure.

Performance Issues#

Problem: System or specific component is experiencing performance degradation.

Debug Approach:

  1. CPU usage (Debug Server):

    >>top
# Identify CPU-intensive components (Foundation servers, Hypervisor, VM)
# Monitor over time to identify patterns

  2. Max-proc-time violations (Debug Server):

    >>proc
# Count violations per task/thread (see :term:`Max-Proc-Time` in the glossary)
# Adjust threshold if appropriate: ``PROC <time_us>`` (non-secure processes; see Debug Server limitations)

  3. Event Traces:

    • Enable event tracing for the components causing the performance issue and capture event traces during the performance issue.

    • Enable VM profiler events to capture VM execution details. Analyze event traces in Nsight or NvLumos for hotspots.

Common Issues:

  • Excessive context switching or interrupt overhead.

  • Foundation server consuming excessive CPU time or priority misconfiguration for Foundation servers.