Mempool Mapping#

The platform_config.h file shows memory pool (mempool) mapping between virtual machines, services, and servers in the AV PCT.

File path for Thor

<nv_sdk_name_foundation>/platform-config/hardware/nvidia/platform/t264/automotive/pct/drive_av/platform_config.h

Mempool Size for Storage Server#

The storage server provides the following configurable parameters in the platform_config.h file for specifying the mempool size for UFS, SDMMC, and QSPI, which can be configured to reduce memory consumption on memory-constrained platforms:

/* Size of Mempool for UFS virtual partitions */
ufs_vsp_mempool_size_kb

/* Size of Mempool for SDMMC virtual partitions */
sdmmc_vsp_mempool_size_kb

/* Size of Mempool for QSPI virtual partitions */
qspi_vsp_mempool_size_kb

Note

If these parameters are not specified, then the default mempool size is 8 MiB for QSPI, 8 MiB for SDMMC and 512 KiB for UFS.

The mempool size affects the number of storage requests that can be queued from the Guest OS Storage client driver to the storage server. If the mempool size is reduced, it can reduce the number of requests that can be queued to the storage server. As a result, it might affect the storage throughput.

The mempool size should be configured after running experiments to confirm that the storage throughput is acceptable for all the storage use cases of that platform.

Mapping IVC Carveouts#

Inter-VM communication (IVC) carveouts, also called IVM carveouts, are memory pools created by the hypervisor for shared buffer allocation between virtual machines.

To map an IVC carveout to a guest VM, configure the mempool in platform_config.h in the PCT configuration file.

PCT Configuration#

Define the mempool in the .mempools table of platform_config.h. Specify the peer VMs, mempool size, and alignment. The mempool index must be unique across the platform configuration:

.mempools = {
   [Mempool_ID] = { .peers = {VM1_ID, VM2_ID}, .size = (SZ_1MB * 64), .align = (SZ_1MB * 2) },
},

For example, a 256 MiB mempool shared between Guest OS 0 and Guest OS 1:

[GOS0_GOS1_M_91] = { .peers = {GID_GUEST0_VM, GID_GUEST1_VM}, .size = (SZ_1MB * 256), .align = (SZ_1MB * 8), .is_contiguous = 0 },

Note

The .is_contiguous parameter in this example is set to 0, indicating that the carveout is not contiguous. Certain hardware engines may require a contiguous carveout. Configure this based on the specific use case for your application.

The hypervisor allocates the carveout at boot time based on the .size and .align values in PCT.

Device Tree Configuration#

In order for the NvMap kernel driver to allocate memory from an IVC carveout, it must be informed about the carveout.

This is done by adding a nvidia,ivm_carveout device tree node to the guest VM partition. The mempool ID in PCT must match the mempool ID in the ivm property of the DT node.

ivm0: ivm-carveout0 {
   status = "okay";
   compatible = "nvidia,ivm_carveout";
   ivm = <&tegra_hv 91>;
   alloc = <1>;
};

The nvidia,ivm_carveout DT node supports the following properties:

Property

Description

Value

Optional

Customizable

compatible

Holds the compatible name for the IVM carveout.

"nvidia,ivm_carveout"

No

No

ivm

Holds the mempool ID. Must match the index defined in the .mempools table of platform_config.h.

Valid mempool number as <&tegra_hv Mempool_ID>

No

Yes

alloc

If true, the guest VM can use the mempool for allocation.

<1> or <0>

No

Yes

status

Enables or disables the IVM carveout node.

okay or disabled

No

Yes

Note

For any IVC carveout, only one participating VM should set alloc = <1>. All other VMs sharing the same mempool must set alloc = <0>. Take care to set the alloc property correctly for each guest VM.

Constraints#

  • Platform configuration assigns the IVM pool ID based on the pool reserved for the partition.

  • The pool ID in the DT node must match the pool reserved by the hypervisor for VI access.

On Linux, add nvidia,ivm_carveout nodes under the reserved-memory node and reference them from the memory-region property of the tegra-carveouts node so that NvMap initializes the IVM heaps.

Example#

The following example configures two IVM carveouts for inter-VM buffer sharing between two guests.

.mempools = {
   [GOS0_GOS1_M_91] = { .peers = {GID_GUEST0_VM, GID_GUEST1_VM}, .size = (SZ_1MB * 256), .align = (SZ_1MB * 8), .is_contiguous = 0 },
   [GOS0_GOS1_M_92] = { .peers = {GID_GUEST0_VM, GID_GUEST1_VM}, .size = (SZ_1MB * 256), .align = (SZ_1MB * 8), .is_contiguous = 0 },
},

We will assume that Guest OS 0 owns the allocation for mempool 91 and Guest OS 1 owns the allocation for mempool 92.

Guest OS 0#

Guest OS 0 sets:

  • alloc = <1> on mempool 91 because it owns allocation for that pool

  • alloc = <0> on mempool 92 because Guest OS 1 owns allocation for that pool

/ {
   reserved-memory {
      /*
       * This VM owns the allocation for this IVM carveout.
       * The peer VM does not own the allocation for this IVM carveout and
       * will set "alloc = <0>".
       */
      ivm0: ivm-carveout0 {
         status = "okay";
         compatible = "nvidia,ivm_carveout";
         ivm = <&tegra_hv 91>;
         alloc = <1>;
      };

      /*
       * This VM does not own the allocation for this IVM carveout.
       * The peer VM owns the allocation and will set "alloc = <1>".
       */
      ivm1: ivm-carveout1 {
         status = "okay";
         compatible = "nvidia,ivm_carveout";
         ivm = <&tegra_hv 92>;
         alloc = <0>;
      };
   };

   tegra-carveouts {
      /* Reference the IVM carveouts, plus any other carveouts that are used
       *by the guest VM. */
      memory-region = <&ivm0 &ivm1>;
   };
};

Guest OS 1#

Guest OS 1 sets:

  • alloc = <0> on mempool 91 because Guest OS 0 owns allocation for that pool

  • alloc = <1> on mempool 92 because it owns allocation for that pool

reserved-memory {
  /*
   * This VM does not own the allocation for this IVM carveout.
   * The peer VM owns the allocation and will set "alloc = <1>".
   */
  ivm0: ivm-carveout0 {
    status = "okay";
    compatible = "nvidia,ivm_carveout";
    ivm = <&tegra_hv 91>;
    alloc = <0>;
  };

  /*
   * This VM owns the allocation for this IVM carveout.
   * The peer VM does not own the allocation for this IVM carveout and will
   * set "alloc = <0>".
   */
  ivm1: ivm-carveout1 {
    status = "okay";
    compatible = "nvidia,ivm_carveout";
    ivm = <&tegra_hv 92>;
    alloc = <1>;
  };
};

tegra-carveouts {
  /* Reference the IVM carveouts, plus any other carveouts that are used by
   * the guest VM. */
  memory-region = <&ivm0 &ivm1>;
};

Such mempools are exposed to the application layer in DRIVE OS through the NvSciBuf API. For details on when NvSciBuf allocates from IVC carveouts, see IVC Carveout Allocation.

Mempool Size for VI Memory Pool#

The VI memory pool is an IVC carveout used for image capture buffer allocation.

The VI memory pool provides the configurable .size parameter for its mempool size in the platform_config.h file. This parameter specifies the total amount of memory available for image capture buffer allocation and should be configured after the total allocation required for image capture has been confirmed:

.mempools = {
   [CAMERA_VM1_M_3] = { .peers = {GID_GUEST0_VM, GID_GUEST0_VM}, .size = (SZ_1MB * 1024), .align = (SZ_1MB * 2) },
}

Note

If this parameter is not specified, then the default configuration creates a 1GB VI memory pool.