High Speed Data Compression Using NVIDIA GPUs

DOWNLOAD DOCUMENTATION EXAMPLES SUPPORT FEEDBACK

The nvCOMP library provides fast lossless data compression and decompression using a GPU. It features generic compression interfaces to enable developers to use high-performance GPU compressors in their applications.

Binaries are available under the standard NVIDIA Software License Agreement

nvCOMP Key Features

  • Compression Algorithms
    • Cascaded: Novel high-throughput compressor ideal for analytical or structured/tabular data.
    • LZ4: General-purpose no-entropy byte-level compressor well-suited for a wide range of datasets.
    • Snappy: Similar to LZ4, this byte-level compressor is a popular existing format used for tabular data.
    • GDeflate: Proprietary compressor with entropy encoding and LZ77, high compression ratios on arbitrary data.
    • Deflate: Huffman + LZ77, Provided for compatibility with existing Deflate-compressed datasets.
    • Bitcomp: Proprietary compressor designed for efficient GPU compression in Scientific Computing applications.
    • ANS: Proprietary entropy encoder based on asymmetric numeral systems (ANS).
    • zStandard: Huffman + LZ77 + ANS, popular compression format developed by Meta.
  • Flexible APIs
    • High-level is a good starting point for new users — it’s easy to use and fast. Metadata, chunking and the required scratch space are managed internally by nvCOMP. Using the high-level interface metadata, you can decompress a buffer without knowing how it was compressed. This enables the easiest way to ramp up and use nvCOMP in applications.
    • Low-level is targeting advanced users — metadata and chunking must be managed by the user outside of nvCOMP. Low-level nvCOMP APIs also perform batch compression/decompression of multiple streams. This provides the most flexibility for users with existing compression/decompression pipelines in their application.

nvCOMP Compression Algorithms

  • Cascaded: Novel high-throughput compressor ideal for analytical or structured/tabular data.
  • LZ4: General-purpose no-entropy byte-level compressor well-suited for a wide range of datasets.
  • Snappy: Similar to LZ4, this byte-level compressor is a popular existing format used for tabular data.
  • GDeflate: Proprietary compressor with entropy encoding and LZ77, high compression ratios on arbitrary data.
  • Bitcomp: Proprietary compressor designed for floating point data in Scientific Computing applications.
  • ANS: Proprietary entropy encoder based on asymmetric numeral systems (ANS).

nvCOMP Performance

Compression ratio and performance plots for each of the compression methods available in nvCOMP are now provided. Each column shows results for a single column from an analytical dataset derived from Fannie Mae’s Single-Family Loan Performance Data. The presented results are from the 2009Q2 dataset. Instructions for generating the column files used here are provided in the benchmark section. The numbers were collected on a NVIDIA A100 80GB GPU (with ECC on).


For additional information, please see our technical blogs and presentations:

Also feel free to gather your own results by running the benchmarks from the nvCOMP GitHub page.