Profiling#

Overview#

HVRTOS provides several profiling capabilities for performance analysis and debugging:

Boot Profiler: Captures boot-time performance data
VM Profiler: Samples VM execution for performance analysis
PMU (Performance Monitoring Unit): ARM hardware performance counters for detailed performance analysis

Drive OS Profiles#

Profiling feature availability by profile:

Feature	dev_nsr, debug_sr	prod_sr, prod_nsr, test_sr, test_nsr	Notes
Boot Profiler	✓ Available	✓ Available	Available in all profiles
VM Profiler	✓ Available	✗ Not Available	Requires eventlib (not available in production profiles)
PMU Virtualization	✓ Available	✓ Available	Controlled through the Platform Configuration Table (PCT).
Uncore PMU Passthru	✓ Available	✗ Not Available	Controlled through the PCT. Only available in dev profiles.

Configuration#

Boot Profiler

Boot Profiler is available in all profiles and configured via BCT (Boot Configuration Table). It uses a profiler region provided by the bootloader.

VM Profiler

VM Profiler is automatically enabled for non-production builds when eventlib is available. It requires event tracing to be available.

VM Profiler can be configured through the PCT using the following fields in the platform configuration headers (for example platform_config.h):

vmprofiler_period_us: Sampling period in microseconds (default: 500, range: 50 to 60000000)
vmprofiler_stack_frames: Number of stack frames to capture per sample (default: 40, range: 1 to 40)

If these fields in the PCT are set to non-zero values, they override the default profiler parameters. If not specified or set to zero, the profiler uses default values (500 microseconds period, 40 stack frames).

PMU

The hypervisor can virtualize the ARM PMU extension so multiple VMs can use PMU independently with context switching and isolation.

PMU virtualization is configured per-VM through the PCT.

PCT:

To enable PMU virtualization for a specific VM, set the following field in the guest configuration:

enable_pmu_virtualization = 1

Note

The field is_vm_pmu_control_privileged is deprecated but still supported for backward compatibility. Use enable_pmu_virtualization for new configurations.

Uncore Performance Monitors:

Platform-specific performance monitors, such as GPU PMU or Vision PMU, can be passed through directly to a single VM:

enable_uncore_pmu_passthru = 1

Only one VM can have uncore PMU pass-through enabled at a time.

Limitations#

VM Profiler Limitations#

Limitation	Description	Impact
Stack Trace Depth	Stack trace depth limited to configured frames (max 40)	Deeper stacks will be truncated

PMU Limitations#

Limitation	Description	Impact
Hardware Dependency	Requires ARM PMUv3p1 or later (FEAT_PMUv3p1) and FEAT_Debugv8p2 or later. PMU versions up to PMUv3p8 are supported.	Availability depends on platform capabilities
Hypervisor / Sidekick Attribution	PMU counting is not left active while the Hypervisor or sidekicks are executing	A VM with PMU virtualization enabled can only observe events attributable to its own VCPUs.
Uncore PMU Passthru	Only one VM can have uncore PMU pass-through enabled. Uncore PMU state is not context-switched between partitions.	A VM with Uncore PMU access can observe events attributable to all partitions in the system.

Detailed Usage#

Boot Profiler#

The Boot Profiler captures checkpoints during system boot, providing insight into boot performance.

Usage:

B::ProfilerLog("System initialization complete");
B::ProfilerLog("Kernel initialization started");
B::ProfilerLog("Kernel initialization complete");

Best Practices:

Use sparingly due to limited entry count
Log only critical boot milestones
Entries are silently dropped after limit is reached

VM Profiler#

The VM Profiler samples VM execution at regular intervals, capturing program counter (PC) and stack traces for performance analysis.

The VM profiler runs as a per-LCPU task that periodically samples the currently running VM at regular checkpoints.

Configuration Parameters:

Parameter	Minimum	Maximum	Default	Description
Sampling Period	50 microseconds	60 seconds	500 microseconds	Interval between samples
Stack Frames	1	40	40	Number of stack frames per sample

Event Integration:

VM profiler events are logged through the eventlib system when B::Group::VmProfileMask is enabled. Each sample generates:

vm_profile event: Contains first 10 stack frames.
vm_profile_ex : Represent remaining 30 stack frames, though each event contains only 10 stack frames.

Sequence IDs:

Each sample has a sequence ID vm_profile_seqid that allows Nsight to:

Detect missed samples (if vm_profile_seqid > prev_vm_profile_seqid + 1, shows as UNKNOWN usage)
Detect idle periods (if floor((ts2-ts1)/sampling_period) > 1, shows as IDLE)

Format:

<timestamp> Event vm_profile: pcpuid=<pcpu-id> seqid=<seq-id> vm_profile_seqid=<id> vmid=<vm-id> pc=<pc> is_sk=<is_sk> is_el0=<is_el0> num_bt=<num_bt> bt1=<bt1> bt2=<bt2> bt3=<bt3> bt4=<bt4> bt5=<bt5> bt6=<bt6> bt7=<bt7> bt8=<bt8> bt9=<bt9> bt10=<bt10>

Example Events:

[2014241159121] Event vm_profile: pcpuid=3 seqid=3735361 vm_profile_seqid=21752 vmid=0 asid=0 pid=0 is_sk=0 is_el0=1 num_bt=6 pc=2440641e98 bt1=2440641f0c bt2=2440641fc8 bt3=244064202c bt4=2440642070 bt5=24406421b8 bt6=4ccaa5aff0 bt7=0 bt8=0 bt9=0 bt10=0

Best Practices:

Sampling Rate: Use 500us (default) for general profiling. Increase to 1-10ms for lower overhead, decrease to 100-200us for detailed analysis
Stack Frame Depth: Use 40 frames (default) for complete traces. Reduce to 10-20 for lower overhead if full traces aren’t needed
Event Mask: Enable only VmProfileMask when profiling to reduce trace file size
Analysis Tools: Use Nsight for VM profiler analysis - it understands the sequence IDs and can correlate samples

PMU (Performance Monitoring Unit)#

Hypervisor virtualizes the architectural ARM PMU for VMs. VMs can access the PMU registers directly, and PMU state is context-switched so one VM cannot observe architectural PMU events attributable to another VM or to the hypervisor.This enables VMs to monitor hardware performance events such as:

Cycle Counting: Measure CPU cycles consumed
Cache Statistics: Monitor cache hits and misses
Branch Prediction: Track branch prediction accuracy
Instruction Counts: Count instructions executed
Memory Access: Monitor memory access patterns

PMU state is saved and restored across context switches, and PMU overflow interrupts are routed back to the owning VM.

For detailed information regarding PMU refer to the ARM Architectural Reference Manual

Uncore PMU Passthru:

For platform-specific uncore PMU units (e.g., GPU PMU, Vision PMU, CCPLEX SLC PMU), pass-through can be enabled via:

enable_uncore_pmu_passthru = 1 in the guest configuration in the PCT

Uncore PMU pass-through provides direct access to platform-specific PMU hardware. Only one VM can have uncore PMU pass-through enabled at a time, and unlike the architectural PMU, uncore PMU state is not context-switched between partitions. This allows VM to monitor all events from uncore PMU units, including events from other VMs.