r/kernel • • 13d ago

Offloading zram compression to GPU?

Is there a way to offload zram compression & decompression to the GPU?

I have a compilation server with 36 logical cores but only 16GB of RAM (I am unable to upgrade due to the global RAM crisis; DDR4 registered ECC costs a ton). In an effort to squeeze more "RAM" out of my RAM, I decided to start using zram compressed with zstd.

I managed to get 48GB of zram to work (3:1 compression), however it resulted in one of my cores being pinned at 100%, desperately trying to compress the increasing amount of ram being used by 36 compile jobs. Is there a way to offload this to the GPU? I think it would greatly speed up compilation if possible, and would perhaps allow for a greater compression rate (zstd ultra maybe???)

EDIT: The GPU is a GTX 1660 Super with the open-source kernel modules from NVIDIA. It basically never gets used and usage is at a constant 0-1%

19 Upvotes

15 comments sorted by

16

u/_gaff 13d ago edited 13d ago

https://lwn.net/Articles/1102680/ patch under review will dramatically improve zram performance

Moving data between CPU and GPU is too slow to be used for compression acceleration.

Build a kernel and try this patch to see if it helps your use case maybe? If it does you could probably add a Tested-by with your findings to the series

3

u/Thin_Dragonfruit2254 13d ago

Does anyone knows how the tests were executed? I built the kernel with the patch but I would like to compare the stock kernel with the new one.

3

u/Thin_Dragonfruit2254 13d ago

Got it:

fio --name=zram-mixed-test \
--filename=/dev/zram0 \
--direct=1 \
--rw=randrw \
--rwmixread=70 \
--bs=4k \
--ioengine=libaio \
--iodepth=32 \
--numjobs=16 \
--runtime=60 \
--time_based \
--group_reporting

3

u/_gaff 9d ago edited 9d ago

What were your findings? Please consider sending a Tested-by if you have meaningful findings

2

u/Thin_Dragonfruit2254 8d ago

I need a better setup to test it properly.. but there are improvements..
I'll post here when I get it done.

1

u/Silent-Degree-6072 13d ago

Do you know where I can download the patch? I've only ever read a few posts on LWN so I'm a bit lost, sorry if I sound dumb

3

u/_gaff 13d ago edited 13d ago

https://lore.kernel.org/all/[email protected]/

patching and building your own kernel might be a job for an LLM if you've never done this.

0

u/Silent-Degree-6072 13d ago

Nah I've already patched and built my own kernels on several occasions, I'm just a bit a stupid lol. Ty for the link

2

u/desal 11d ago

"moving data between CPU and gpu is too slow" what about with SAM / pci rebar? Gives CPU access to all of gpu vram at once instead of chunking

1

u/g1bber 11d ago

Having the CPU access the GPU BAR is not a good idea to move data faster. For larger data it’s best to use the GPU copy engine (the GPU issues DMA reads to host memory to get the input or DMA writes to host memory to place the output)

2

u/Ok_Green5623 13d ago

I don't think your problem is speed of compression. You will never be able to load your cores with compilation work, because compilation tend to use a lot of ram especially with higher optimization levels. If you need like 32-40GB of RAM to utilize your cores, no matter what you do - you will be heavily throttled by paging in and out. Just limit parallelism and use your CPU at 20-40% before it heats the RAM limit and it will be faster.

1

u/possiblyquestionabl3 13d ago

Are you using zstd for zram? The types of work it does for compression (I'm guessing this is what you want to offload?) isn't very well suited for a GPU - you'll want to have fairly uniform execution (few branches, since the simd programming model will serialize all branches together in most cases) and predictable memory shading.

That said, I believe https://github.com/KhronosGroup/Vulkan-ExtensionLayer/blob/38ced57f063a288ecd4f401523638b7bdb3ac9d2/docs/memory_decompression_layer.md has an example of a DEFLATE shader for to emulate VK_LAYER_KHRONOS_memory_decompression on ICDs / hardware that lacks this support, including an implementation of GInflate (decompression of DEFLATE) - https://github.com/KhronosGroup/Vulkan-ExtensionLayer/blob/38ced57f063a288ecd4f401523638b7bdb3ac9d2/layers/decompression/shaders/GInflate.glsl

1

u/vip17 13d ago

Also try zswap, it's said to behave better under pressure

https://linuxblog.io/zswap-better-than-zram/

1

u/desal 11d ago

What about ramfs? Getting an nvme drive and using it with ramfs

2

u/vip17 11d ago

how on earth is ramfs relevant here? zswap is optimized for compressing pages and zram is optimized to be a swap, both don't have any filesystem overhead