Carveout Customization and Profiling#

NvMap provides a few carveouts to be used for different use cases.

Modify these carveouts according to your system’s needs and for your usescases. Additionally, to ensure efficient and adequate use of the carveouts, profile carveout usage throughout development.

Carveout Customization#

This section will outline how to customize both generic and GPU memory caveouts. This allows you to set aside memory for device isolation and security.

Generic Carveout#

  • This carveout exists both on Orin and Thor.

  • This carveout is a reserved memory carveout inside Linux GuestOS.

  • It is used by display to obtain IPA contiguous buffers to satisfy their use cases.

  • The memory for generic carveout is reserved during boot time and is not available to OS memory allocations. The size of the carveout depends on the use-cases or applications for which the platform is targeted and their cumulative memory usage over the application lifecycle.

  • To update the generic carveout size, the device tree (DT) needs to be modified as follows:

    • In tegra234-linux-gos.dtsi or in the relevant platform-specific DT file, update the size in the generic_carveout DT node that is present under reserved-memory DT node. For example, the following DT node updates the size of generic carveout to 0x80000000 bytes:

reserved-memory {
    generic_carveout {
        compatible = "nvidia,generic_carveout";
        size = <0 0x80000000>;
        status = "okay";
    };
};

GPU Carveout#

  • This carveout exists only on Thor.

  • It is based upon the huge pages provided by hugetlbfs. It provides 2MB physically contiguous pages for any buffer allocated from this carveout.

  • It is used by GPU/GFX for achieving performance in the compression use cases.

  • The initial size is configured by the nv_hugetlbfs_init.sh systemd script.

  • The size of this carveout can be changed by updating the /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages sysfs node with the number of 2 MB huge pages.

  • For example, if you want to allocate 1024 numbers of 2 MB huge pages (2048 MB of the memory for GPU carveout), then use the following command:

echo 1024 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages

Carveout Profiling#

The size of the carveout depends on the use cases or applications for which the platform is targeted, and their memory usage needs cumulatively over the application lifecycle. Set the carveout size based on the total outstanding buffers and sizes that is used at any point in the system.

It is recommended to profile carveout usage throughout development. Repeated profiling:

  • Ensures that carveout usage has not changed as development progresses.

  • Considers carveout usage for applications.

The generic carveout and GPU carveout can be profiled as follows:

Generic Carveout#

Certain debugfs nodes are provided to find out the size, free size of the carveout, allocations from the carveout, etc.

  • /sys/kernel/debug/nvmap/generic-0/free_size debugfs node shows the free size in the generic carveout at the time of query (in bytes) while

  • /sys/kernel/debug/nvmap/generic-0/size debugfs node shows the total size of the generic carveout at the time of query (in bytes).

Example 1 : You can use the following command to get the free size present in generic carveout. The following output shows that the free size at the time of running this command was 0x1000000 bytes.

cat /sys/kernel/debug/nvmap/generic-0/free_size
0x0000000001000000

Example 2 : You can use the following command to get the total size of the generic carveout. The following output shows that the size of the generic carveout at the time of running this command was 0x2000000 bytes.

cat /sys/kernel/debug/nvmap/generic-0/size
0x0000000002000000

With the help of running the application and querying free_size and size debugfs nodes at necessary points in time, the generic carveout memory usage can be understood and the size of the generic carveout can be decided.

GPU Carveout#

To find out the free and total size of GPU carveout, certain sysfs nodes can be used.

  • /sys/devices/system/node/node0/hugepages/hugepages-2048kB/free_hugepages sysfs node shows the total number of free 2MB huge pages in the GPU carveout at the time of query while

  • /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages sysfs node shows the total number of 2MB huge pages in the GPU carveout at the time of query. These values can also be queried from the output of

  • /proc/meminfo with help of HugePages_Free and HugePages_Total entries respectively.

Example 1 : You can use the following command to get the number of free 2 MB huge pages present in GPU carveout. The following output shows that the number of free huge pages at the time of running this command was 1022 (2044 MB of memory).

cat /sys/devices/system/node/node0/hugepages/hugepages-2048kB/free_hugepages
1022

Example 2 : You can use the following command to get the total number of 2 MB huge pages present in the GPU carveout. The following output shows that the total number of 2MB huge pages present in the GPU carveout at the time the command was ran was 1024 (the total size of the GPU carveout was 2048 MB).

cat /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
1024

With the help of running the application and querying the above sysfs nodes at necessary points in time, the GPU carveout memory usage can be understood and the size of the GPU carveout can be decided.

On this page