I've been working on adambridge, a drop-in AdamW for PyTorch. It keeps only the weights on the GPU. The FP32 master weights and both Adam moments live in host RAM, and fused SIMD kernels (AVX2/AVX-512/NEON) update them on the CPU. The updates run layer by layer while the GPU is still doing the backward pass.
Version 0.2.0 is out. The main result: Qwen2.5-7B full fine-tuning on an RTX 5090 Laptop (24 GB) ran at about 1,122 tokens/s, using 18.2 GiB of GPU memory and 38 GiB of host memory under WSL 2. This is full-parameter AdamW, not LoRA.
Benchmarks
I compared it against DeepSpeed 0.19.9 (ZeRO-Offload, ZeRO-Infinity, ZenFlow, SuperOffload), torchao 0.18, bitsandbytes 0.50.2 and plain all-GPU PyTorch, all on the same machine:
| result |
|---|
| Largest model trained | 8.02B params (next best library: 2.08B) |
| Qwen2.5-0.5B / 1.5B vs DeepSpeed's best | +113% / +120% tokens/s at the same quality |
| Qwen2.5-3B | only adambridge fit it |
| When the model already fits on the GPU | ties all-GPU AdamW (gpu_state_budget="auto") |
No quality loss
The low-memory options that drop the FP32 master copy all lost quality. That covers BF16-only weights, bitsandbytes 8-bit with BF16 weights, and torchao offload with BF16 weights. At fine-tuning learning rates, most AdamW updates are smaller than half a BF16 step, so they get rounded away. On Qwen2.5-0.5B, only 18% of the weights changed at all, compared with 84% for the reference.
adambridge keeps the FP32 master copy on the host, and its final validation loss stayed within run-to-run noise of torch.optim.AdamW.
Usage
import
adambridge
opt = adambridge.AdamBridge(model.parameters(),
lr
=1e-4,
weight_decay
=0.01,
precision
="bf16")
# 'fp32' | 'bf16' | 'fp8'
opt.attach_hooks(model)
# async: each layer is updated as soon as its grad is ready
for
x, y
in
loader:
loss = loss_fn(model(x), y)
loss.backward()
opt.step_sync()
Without attach_hooks it works as a normal optimizer (step() / zero_grad()), and LR schedulers and clip_grad_norm_ still work.
What's new in 0.2.0
FP24 master weights: 3 bytes per parameter instead of 4, with stochastic rounding. Combined with FP8 moments, the host state is about 5 bytes per parameter, and quality matched the reference at 0.5B, 1.5B and 3B.
Chunked causal-LM loss: a 150k-token vocabulary no longer has to hold its full logits tensor in memory.
Delayed updates: the CPU update overlaps the entire next step.
LionBridge: the Lion optimizer on the same pipeline.
Shared pinned transfer ring: about 30 GB less host RAM for a 7B model.
Compatibility: works with the HF Trainer and torch.compile.
Kernels: written in plain C, shipped as source inside a py3-none-any wheel and compiled on first use. If no compiler is available, it falls back to pure PyTorch kernels.
Limitations
I only benchmarked on one laptop. Desktop and server GPUs hold their clocks better when idle, which changes the numbers for every offload method.
The AVX-512 path has never actually run on real hardware (my CPU is AVX2). NEON was tested only under QEMU.
There's no DDP/FSDP support yet, and FP16 with GradScaler isn't covered.
Quality was measured on short runs at a single learning rate.
Global gradient clipping isn't possible in async mode.
pip install adambridge. It's Apache-2.0 licensed.
GitHub: https://github.com/johnanishg/adambridge
I'd like feedback, especially from anyone with an AVX-512 CPU or a multi-socket server. Running python -m adambridge info and the self-test and posting the results would help a lot.