Integration Guide#
API Integration#
In this section, an operating point (OP) refers to a SoC or VM power and performance state.
Software Components Interaction#
Sample implementation of an LCM Server on a guest OS and an LCM client on a guest OS, using NvSciIpc-based communication.
IPC is used for communication between the LCM server and LCM client running in the same guest OS.
IVC is used for communication between the LCM server and LCM client running in different guest OSes.
Sample implementation of an LCM Server on a guest OS and an LCM client on the MCU, using socket-over-Ethernet communication.
Between the LCM Server on a guest OS and the PM Server on HVRTOS, IVC channel-based communication is used. The two IVC channels are 495 and 496. If these IVC channels need modification, platform configuration must be updated.
Initialization of PM Server Library by LCM Server#
For PM Server library API details, refer to nvpwrmgmt.h.
Initialization of NvDVMS Client Library by LCM Client#
For NvDVMS client library API details, refer to nvdvms_client.h.
SoC OP Transition Orchestration by LCM Server#
An optional SoC OP request flow is when the LCM client on the MCU dictates state transitions to the LCM server.
For NvDVMS client library API details, refer to nvdvms_client.h. For PM Server library API details, refer to nvpwrmgmt.h.
Error Flow for VM OP Error#
Sample error flow. Complete error handling is explained more as part of guidelines for LCM server and LCM client.
For NvDVMS client library API details, refer to nvdvms_client.h. For PM Server library API details, refer to nvpwrmgmt.h.
System Constraints#
The PM server shall be used only through its published PM server client library interface.
The platform shall ensure that when an LCM client drives a VM power profile change through the NvDVMS, the hypervisor system manager recognizes the change and reports the corresponding VM operation completion to the PM server. Without this reporting, the PM server’s per-VM operation bookkeeping shall not complete, and the PM server shall report a VM operation failure.
The NvDVMS server shall be running and ready before LCM server and LCM clients attempt NvDVMS initialization, so that state transitions and power-profile calls succeed.
Exactly one LCM server shall run in the system, and it shall execute within the privileged guest OS. No other guest OS or domain shall host an LCM server instance.
Each guest OS shall run at most one LCM client instance. A guest OS shall not host more than one LCM client connected to the LCM server.
Once a shutdown SoC operation transition has started, the system shall treat it as best-effort delivery. Errors encountered during the shutdown transition (PM failures, client failures, device-state failures) shall be logged but shall not prevent the shutdown from progressing.
If the PM server notifies an error, the LCM server shall be permitted to trigger system shutdown while the PM server state machine is in any state.
Resume shall happen to the previous VM OP which was set during suspend.
After successful error-mode exit, the LCM server shall reattempt the SoC operation that was in progress before the error by re-issuing the same SoC operation identifier with an error-recovery designation.
Guidelines for LCM Server#
Initialization & Session Lifecycle#
The LCM server shall initialize the PM server client library with a valid configuration (communication endpoints, error notification callback, client context) before issuing any PM commands.
The LCM server shall configure necessary communication channels for the PM server and LCM clients.
The LCM server shall run the PM server client library’s asynchronous event-processing like error and completion notification on a dedicated event listener thread, started before issuing any start SoC OP operation command. This thread shall remain active for the entire duration of the LCM server process.
LCM server shall implement protocol versioning mechanism for all LCM clients. On detecting a protocol version mismatch with a LCM client, the LCM server shall send an error notification indicating version mismatch and shall disconnect that LCM client.
SoC OP Transaction Orchestration and Sequencing#
For each SoC operation transaction, the LCM server shall interact with the PM server in four sequential phases, each carrying the same SoC OP operation identifier:
Notify phase: The LCM server shall notify the PM server of the intended SoC operation. If the PM server rejects the notification, the LCM server shall abort the transaction and shall not proceed to subsequent phases.
Prepare phase: The LCM server shall prepare the SoC operation with the PM server. Prepare commands shall succeed before the LCM server considers starting the SoC operation. If a command fails, the LCM server shall start system shutdown.
Initiate phase: The LCM server shall start the SoC operation with the PM server. Start commands shall succeed before the LCM server considers the SoC operation initiated. If a command fails, the LCM server shall start system shutdown.
Complete phase: The LCM server shall wait for the PM server to deliver a completion notification for the initiated SoC operation. For suspend operations, the wait shall be indefinite (spanning the suspend/resume cycle). For shutdown operations, the wait shall be indefinite. For all other operations, the wait shall be time-bounded.
After sending the start command, the LCM server shall wait for PM server completion notification for that transaction at least once. The completion-wait parameters shall include the same SoC OP operation identifier and an operation-type classification (generic, suspend, or shutdown) consistent with the active SoC OP operation so that timeout and wait semantics match PM behavior.
The LCM server shall handle all notification types that the PM server client library completion-wait might return (success, error, default, and deinitialization).
The LCM server shall not issue concurrent PM server client library send calls. The PM server client library shall reject it.
The MCU and FSI LCM client shall receive SoC operation broadcasts before any guest OS LCM client.
The LCM server shall sequence its PM server interactions and LCM client communications such that no client-facing broadcast to guest for VM OP operation is issued until the PM server has accepted the SoC OP operation initiation (notify, prepare, and start commands).
The LCM server shall wait for PM server completion notification once before concluding the SoC operation transaction.
The LCM server shall track per LCM client acknowledgment or rejection for each phase and shall not advance to the next phase until its aggregation rules are satisfied.
The LCM server shall queue SoC operation requests and process them serially so that at most one PM server transaction is active at a time.
While a SoC operation is in progress, the LCM server shall pause processing of new SoC operation requests unless the new request is a forced system shutdown.
If the LCM server receives a SoC operation request whose identifier matches the SoC operation that has already completed, the LCM server shall discard the request without invoking the PM server. It is only allowed if after completion of recovery, the LCM server makes the same SoC operation request.
For LCM clients that cannot tolerate back-to-back messages, the LCM server shall insert inter-message delays or apply flow control so that no mandatory command is lost.
VM Operation Verification#
Before completing the acknowledge/completion phase, the LCM server shall verify that each connected LCM client with a valid VM identifier has reported a power profile or VM operation that matches the expected VM operation for the requested SoC operation and that client’s VM, as derived from configuration. If any LCM client has not yet reported a matching power profile, the LCM server shall defer concluding the transaction until matching succeeds or product-defined timeout logic intervenes.
The LCM server shall obtain VM identifier, initialization state, operational state, and power profile from each participating LCM client through state and heartbeat reports to perform the verification.
If VM operation matching is not yet satisfied when PM server completion arrives, the LCM server shall defer the acknowledge-phase broadcast until matching succeeds or some defined timeout logic intervenes.
Error Handling#
While handling a PM server error notification, the LCM server shall suppress normal SoC operation commands unless a forced shutdown path overrides that suppression.
The LCM server shall register an error notification callback with the PM server client library. The callback shall return promptly. Resource-intensive processing (recovery, shutdown) shall be deferred to a separate execution context, so the PM server event listener thread is not blocked.
The LCM server shall distinguish errors involving its own VM from errors involving guest VMs (using a configured server VM identifier) and shall apply a different policy for each. It can issue actions like system shutdown for non-recoverable errors and can try to recover the system for recoverable errors. The action for error shall depend on the policy defined by DriveOS users. Errors defined by PM server and corresponding actions:
Error Enum |
Error Interpretation |
Probable Action |
|---|---|---|
|
SoC OP operation acknowledgment failure |
System shutdown |
|
VM operation security violation like unexpected VM OP notification for SoC OP. |
Recovery for non-server (LCM server not running) VMs; System shutdown |
|
VM OP was not received for SoC OP transaction. |
Recovery for non-server (LCM server not running) VMs; System shutdown |
Note
Unknown errors can be logged or some action might be taken based on DriveOS user system policy.
When entering error mode, the LCM server shall issue, in order: error mode entry, notify force VM shutdown, force VM shutdown, and force VM boot commands to PM Server, each with the affected VM identifier. If any step fails, the LCM server shall escalate to system shutdown.
When the recovered client reports readiness, the LCM server shall issue error mode exit to PM server. If the PM server rejects the exit, the LCM server shall escalate to system shutdown.
The LCM server shall enforce a configurable recovery interval. If the affected client does not recover within that interval, the LCM server shall escalate to system shutdown.
After successful error-mode exit, the LCM server shall reattempt the SoC operation that was in progress before the error by re-issuing the same SoC operation identifier with an error-recovery designation. The error-recovery designation shall bypass duplicate SoC operation suppression so that the PM server is invoked even though the identifier matches the previously completed SoC operation. The LCM server shall reset its internal coordination state before processing the reattempted request.
If the PM server delivers an error notification while the LCM server is already in an error-handling cycle, the LCM server shall escalate to system shutdown.
On entering error mode, the LCM server shall broadcast an error notification to all connected LCM clients indicating error-mode entry, the affected client, and forced shutdown. On exiting error mode, it shall broadcast error-mode exit.
Heartbeat and Client Liveness#
If heartbeat-based liveness is enabled, the LCM server shall periodically send heartbeat messages to all active connected LCM clients and shall expect timely responses.
The LCM server shall accept the LCM client’s VM identifier from heartbeat responses and shall use it for VM operation matching and error-recovery identification.
The LCM server shall track consecutive missed heartbeats per LCM client. When a configurable miss limit is exceeded, the LCM server shall trigger error recovery for that LCM client.
The LCM server shall not track missed-heartbeat for a LCM client that is currently the subject of an active error-recovery cycle or has been suspended.
When a LCM client connects, the LCM server shall send an immediate out-of-band heartbeat so the client can confirm connectivity.
The LCM server shall include the safety processor LCM client like FSI in its periodic heartbeat cycle. The heartbeat to the safety processor serves as a liveness signal for the LCM server itself; if the LCM server fails to send heartbeats, the safety processor shall be able to detect the failure. The LCM server shall not exclude the safety processor from heartbeat transmission under any operating condition, including error-handling mode.
System Parameters#
The LCM server can accept system parameters (boot reason, vehicle status, battery level, requested state) from MCU LCM clients and can use boot reason to determine the initial SoC operation and its target state flags.
LCM server can evaluate based on DriveOS user defined rules after relevant inputs (system parameters, state changes) and shall translate into SoC operation requests, including suspend, shutdown, and VM-reboot sequences.
Shutdown and Termination#
The LCM server shall implement system shutdown by issuing the full PM sequence (notify, prepare, start, wait-for-completion with shutdown operation type) using the configured shutdown SoC operation identifier. Failures during this sequence shall not prevent the shutdown from progressing. This ensures shutdown is the best effort delivery.
On shutdown or termination, the LCM server shall deinitialize PM server client library, stop all background threads (heartbeat, error handler), drain pending SoC operation queues, and destroy the server communication handle. Deinitialization shall not be invoked from signal handlers. No PM commands shall be issued after deinitialization.
Configuration#
The LCM server shall load configuration from device tree that defines, for each supported SoC operation, the per-VM expected VM state used for verification.
The LCM server shall store per-SoC-operation, per-VM-identifier VM operation mappings from configuration for use in VM operation verification.
The LCM server shall identify from configuration which SoC operation identifiers represent shutdown, suspend, and boot, so PM completion-wait uses the correct operation-type classification.
If heartbeat is used, the LCM server shall obtain the enable flag, miss limit, and interval from configuration.
SoC operation identifiers and VM operation values in LCM server configuration shall be consistent with LCM client configuration and platform power definitions.
Guidelines for LCM Client#
NvDVMS Initialization and Readiness#
LCM clients running on guest operating systems shall initialize the NvDVMS client library and establish communication with the NvDVMS server before claiming readiness for coordinated transitions. This requirement does not apply to the MCU and FSI LCM client.
Before accepting SoC-operation commands, the LCM client shall wait until the NvDVMS library reports init-done, so that the LCM client shall not attempt state transitions from an incomplete boot state. Exception to this is system shutdown transition if NVDVMS_INIT_DONE is not reached.
LCM Server Communication#
Each LCM client shall accept or explicitly reject every LCM server command. Silent loss of a mandatory command might result in the LCM server and PM server being unable to complete the SoC operation transaction, which may leave the system in an indeterminate power state and compromise system safety.
LCM client acknowledgments shall carry the same SoC operation identifier the server signaled, so the server can correctly track per-LCM client phase completion.
On receiving a SoC-operation notification (advance notice), the LCM client shall acknowledge without starting guest-visible work. On receiving a SoC-operation command (start), the LCM client shall begin VM operations and acknowledge accordingly.
LCM clients shall respond to LCM server heartbeat messages with their VM identifier for matching and correct validation.
LCM clients shall report to the LCM server: initialization-done state, power-profile active (with the applied VM operation), and applicable state transitions, so the LCM server can verify VM operation completion.
LCM clients shall perform a protocol version handshake at connection time. On version mismatch notification from the server, the client can choose to terminate cleanly or take some other action as per the DriveOS user defined policy.
LCM clients shall handle error notifications from the server (error-mode entry, forced shutdown, version mismatch) and can take some action as per the DriveOS user defined policy.
LCM clients can request suspend, shutdown, restart, or system shutdown shall use SoC operation identifiers from configuration consistent with the LCM server and PM Server.
In-VM Application Management#
The LCM client shall act as the sole broker between the LCM server and all in-VM applications for power management events. Each in-VM application shall register with the LCM client to receive event notifications and to issue state-change requests. The LCM client shall maintain per-application connection state and shall track each registered application independently.
The LCM client shall relay power/state-change events, power-profile-change events, prepare for power/state changes events, and error events to all registered in VM applications. No registered application shall be excluded from receiving mandatory event notifications.
The LCM client shall wait for application acknowledgment to prepare for power/state changes events before proceeding with NvDVMS state transitions and LCM server acknowledgment, unless a timeout overrides it.
The LCM client shall support queries from local applications for current VM state and current power profile.
The LCM client can accept power/state-change requests from applications (e.g. to operational, suspend, deinit/shutdown, deinit-prepare, reinit) and shall validate them against the current VM operation’s allowed-states map before processing.
The LCM client shall support concurrent registration of multiple in-VM applications. Each registered application shall receive all mandatory event notifications independently. The LCM client shall not proceed with NvDVMS state transitions requiring application acknowledgment until all active registered applications have acknowledged, unless a timeout overrides it.
NvDVMS Usage#
Initialization Phase#
NvDVMS waits for a timeout value to complete initialization and report completion of initialization to NvDVMS. This timeout value comes from the device tree property timeout_init_done_ms. It is the responsibility of the LCM client application layer to re-configure the timeout value in device tree if the system design requires transition of Init Done state must take place within some other specified amount of time. The default value configured is 3000 milliseconds.
Note
The LCM client application layer can poll with timeout for state transitions to Init Done if the system design requires that the transition to the NVDVMS_INIT_DONE state takes place within a bounded time. It is the responsibility of the LCM client application layer to implement this timeout. Init Done timeout for NvDVMS in device tree needs to be configured so that LCM client application layer polling timeout happens after NvDVMS Init Done timeout.
Constraints#
NvDVMS has the following constraints for the state and power profile transitions:
When transitioning the guest VM state to NVDVMS_OPERATIONAL and NVDVMS_DEINIT_PREPARE, NvDVMS is only able to ensure the synchronized transition of the guest VM state and does not propagate the state transition to other guest VMs. For all required guest VMs, the LCM application layer is expected to sequence the guest VM state transitions via
nvdvms_set_vm_state()and confirm their respective completion vianvdvms_get_vm_state()to ensure the consistency of the guest VM state.When transitioning the guest VM state to NVDVMS_INIT_DONE, NvDVMS is only able to ensure the synchronized transition of the guest VM state and does not propagate the state transition to other guest VMs. For all required guest VMs, the LCM application layer is expected to confirm the completion of the guest VM state transitions via
nvdvms_get_vm_state()to ensure the consistency of the guest VM state.Explicit sequencing of the transition from NVDVMS_INIT to NVDVMS_INIT_DONE is not required as this transition is triggered automatically by boot of DriveOS.
Explicit sequencing of the transition from NVDVMS_SUSPEND to NVDVMS_RESUME and subsequently NVDVMS_INIT_DONE is not required as this transition is triggered automatically by resume of DriveOS.
NVDVMS_DEINIT_PREPARE and NVDVMS_DEINIT transition are best effort for shutdown.
For the transition from NVDVMS_OPERATIONAL to NVDVMS_INIT_DONE state via REINIT of all required guest VMs, the LCM client application layer must sequence the transitions of the guest VM states via
nvdvms_set_vm_state()and confirm their respective completion vianvdvms_get_vm_state()to ensure the system state is consistent. The transition from NVDVMS_DEINIT_PREPARE to NVDVMS_INIT_DONE is performed by NvDVMS in the guest VM that receives the request from the LCM client application layer.A parallel request for a guest VM power profile transition during an ongoing guest VM power profile or a guest VM state transition is rejected.
A parallel request for a guest VM state transition during an ongoing guest VM power profile is rejected.
For system suspend, NvDVMS will consider VM_OP_62 as the power profile while using
nvdvms_set_power_profile().For system shutdown, NvDVMS will consider VM_OP_63 as the power profile while using
nvdvms_set_power_profile().If guest VM does not move to NVDVMS_INIT_DONE from NVDVMS_INIT, then guest VM can be moved to NVDVMS_DEINIT_PREPARE from NVDVMS_INIT to facilitate system shutdown.
VM State Transitions#
The LCM client shall drive VM state transitions only along paths permitted by the NvDVMS library:
Init-done → operational (use API
nvdvms_set_vm_state()).Init-done → deinit-prepare (use API
nvdvms_set_vm_state()).Init-done → suspend (use API
nvdvms_set_power_profile()).Operational → deinit-prepare (use API
nvdvms_set_vm_state())Deinit-prepare → reinit (use API
nvdvms_set_vm_state()).Deinit-prepare → deinit (use API
nvdvms_set_power_profile()).Init → Deinit-prepare (use API
nvdvms_set_vm_state()).
For suspend SoC operations, the LCM client shall set the appropriate pre-suspend state as required by the NvDVMS state machine, acknowledge the LCM server, apply the suspend power profile via the NvDVMS library.
For shutdown SoC operations, the LCM client shall transition through deinit-prepare then deinit via the NvDVMS library. After triggering deinit, the client shall not expect control to return. Errors during shutdown transitions shall be logged but shall not prevent the shutdown from progressing.
When a SoC operation requires transitioning, the LCM client shall first transition through the required intermediate states via the NvDVMS library before applying the new profile.
If any NvDVMS state transition or power-profile call fails during a non-shutdown SoC operation, the LCM client shall report the error to its local applications and to the LCM server.
LCM clients shall complete NvDVMS-backed work within deadlines implied by PM server bookkeeping timers and PM server completion windows; exceeding those deadlines may cause PM server to report VM operation failure.
Configurable Timeouts#
NvDVMS implements configurable timeouts through device tree to detect certain errors. These timeouts are defined in device tree node with compatible nvidia,dvms_startup_command.
- Device tree property
timeout_init_done_ms Timeout in milliseconds for guest VM to report NVDVMS_INIT_DONE to NvDVMS. The default value is 3000 milliseconds.
- Device tree property
init_done_poll_interval Polling interval in milliseconds at which NvDVMS checks whether guest VM has reported NVDVMS_INIT_DONE to NvDVMS. The default value is 1000 milliseconds.
Configuration#
The LCM client shall load device tree configuration providing SoC-operation to VM-operation mappings, consistent with platform power definitions used by the LCM server and PM server.
The LCM client shall load from device tree per VM operation, which VM states are permitted. The LCM client shall reject state transitions not allowed for the current VM operation.
LCM client can monitor VM init done state with timeout configured from device tree.
The LCM client shall identify from configuration which VM operations correspond to suspend and shutdown, so it can select the correct NvDVMS transition path.
MCU LCM Clients#
If the customer’s MCU software is designed to provide system parameters (boot reason, vehicle status, battery level, requested state etc) to the LCM server, the MCU LCM client can transmit these parameters after establishing connection with the LCM server. When provided, the LCM server can use these parameters to determine the initial SoC operation selection.
If the customer’s MCU software is designed to initiate SoC operations, the SoC operation identifiers used by the MCU LCM client shall match those defined in the PM server and LCM server configuration.
Safety Processor LCM Client#
The platform shall support an LCM client on the safety processor (e.g. FSI or equivalent platform-specific safety processor). The safety processor LCM client shall connect to the LCM server and participate in the heartbeat mechanism as a connected client.
The safety processor LCM client shall monitor the periodic heartbeat messages from the LCM server. If the safety processor LCM client does not receive a heartbeat from the LCM server within a configurable timeout, it shall consider the LCM server or its host guest OS to be unresponsive.
When the safety processor LCM client determines the LCM server is unresponsive, it shall take DriveOS user defined safety action (e.g. system reset, error escalation) to ensure the system does not remain in an indeterminate power state.
Restriction for Various Power Mode#
SC7#
The following lists the requirements for user software to integrate SC7:
Diagnostic tests might need to be run before entering Mission Mode and are not shown in the sequence/flow diagrams.
The application needs to ensure there are no outstanding service requests to DriveOS at the point of requesting it to enter Suspend State, or the request might fail.
As the board components are powered off during suspend, the application needs to ensure the on-board devices/sensors are re-initialized/configured on resume.
Applications interacting with secure world trusted applications close their sessions before requesting suspend. This is because secure world (TZVault) is restarted during resume.
SC7 flow entails a reset of the security system on the chip. Re-negotiate, re-program security keys on resume.
TimeSync must be redone after resume.
Start runtime IST in the post init phase (both from cold boot and resume), if enabled.
Start Drive Update service in the post init phase.
Perform user visible actions during or after the post init phase.
Component-Specific Limitations#
NvStreams
Before SC7 entry, ensure the following requirements for the applications using NvStreams framework to stream data between the producer and consumers:
There are no jobs that were submitted by the NvStreams applications pending in the CPU or engine pipelines.
The application is not performing any active NvSciSync fence waits.
To meet the requirements, perform the following tasks before proceeding to SC7 entry, if the NvStreams pipeline is being initialized:
Wait for the buffer and synchronization primitives to be allocated and the NvSciBuf and NvSciSync objects are registered to the UMDs.
For the NvSciStream application, wait for pipeline initialization to complete and all the packets are available in the pool block.
When the application is in the streaming phase, ensure that there are no jobs pending in the CPU or UMD engine pipelines by following the SC7 entry guidelines of the respective UMDs and that there are no packets in flight.
CUDA
Before SC7 entry, ensure the following requirements for the applications:
Any outstanding GPU tasks (for example, kernel, memcopies, and event waits) are completed using appropriate synchronization APIs, such as
cudaEventSynchronize,cudaStreamSynchronize, andcudaDeviceSynchronize.Outstanding GPU tasks can also be cross-engine dependencies (for example, between DLA and GPU) that are built using NvSciSync interop. Waiting on unsignaled NvSciSync can lead to infinite waits, so applications should ensure that the signaler of such fences has unblocked all waiters on the GPU.
Any ongoing CUDA APIs that are synchronous with respect to the host (for example,
cudaMallocandcudaStreamCreate) are complete.Timestamps recorded by
cudaEventafter returning from SC7 state will account for the duration in which the GPU was in a suspended state, so the elapsed time returned bycudaEventElapsedTimeshould be interpreted accordingly.
PVA
Before SC7 entry:
The applications must ensure that there are no pending tasks or commands on the PVA engine by waiting for all the previously submitted commands to finish before entering into SC7 by calling
cupva::Fence::wait().The applications must not submit any new commands to PVA while the DOS is transitioning to SC7 suspend state.
Camera
The SIPL camera application should only request the NVIDIA DRIVE OS to transition into the SC7 power state after SIPL software configuration but before SIPL hardware initialization.
The SIPL camera application should limit requests to NVIDIA DRIVE OS to transition into the SC7 power state once per NVIDIA DRIVE OS system cold boot.
NvDisplay
For information on the guidelines, see OpenWFD Usage Guidelines in the NvDisplay documentation.
Graphics
Before SC7 entry, ensure that any tasks submitted to GPU should not be in pending state. To do so, applications must call
vkQueueWaitIdle()for eachVkQueuecreated.
NvSciIpc
Before SC7 entry, ensure that there are no outstanding tasks for applications using NvSciIpc. If a peer endpoint process goes down and up because it is not running during SC7 (that is, Trust Agent), applications must call
NvSciIpcResetEndpointSafe()and then establish connection using theNvSciIpcGetEventSafe()API after SC7 exit.
NvGPU
Before SC7 entry, ensure that there are no outstanding tasks for applications using NVGPU. There should not be any new request to NVGPU after the suspend callback is issued other than resume callback. Requests can be made after the driver resumes and sets the resume process to done.