r/pytorch • • 1d ago

Copying tensors 5x faster

2 Upvotes

I made a tool called cudamemcpy-tuner that copies pageable memory much quicker (up to 5x on a b300!)

B300, copying a 1 GB tensor to the GPU:

import time, torch
x = torch.ones(1 << 30, dtype=torch.uint8)
y = x.to("cuda"); y.cpu()
for name, copy in [("CPU -> GPU", lambda: (x.to("cuda"), torch.cuda.synchronize()))]:
t = time.perf_counter()
for _ in range(5):
copy()
print(f"{name}: {5 * x.nbytes / (time.perf_counter() - t) / 1e9:.1f} GB/s")

$ python bench.py
CPU -> GPU: 10.2 GB/s

$ cmt tune -- python bench.py
CPU -> GPU: 51.5 GB/s

writeup: https://maknee.github.io/blog/2026/CudaMemcpy/
code: https://github.com/Maknee/cudamemcpy-tuner


r/pytorch • • 2d ago

Bfloat16 mantissa drift on high-retention decay gates in a 64-token SSD chunk scan

1 Upvotes

​While writing a State Space Duality/Selective Linear Attention block in PyTorch (SlabV4Model), we ran into a precision issue in the chunked scan that took a while to isolate.

​Our slowest heads learn retention gates (alpha = exp(log_alpha)) around 0.999969 which corresponds to a half-life of roughly 22192 tokens. When the sequential interchunk carry ran in bfloat16, its 7-bit mantissa introduced about 0.78% rounding distortion per step on values that close to 1.0. Over a 4096 token sequence, the state drift broke parity between the parallel chunk scan and single token recurrent decoding.

​Splitting the sequence into 64 tokenn chunks (T % 64 == 0 enforced) and keeping the sequential inter-chunk carry in float32 resolved it. A few other things that helped keep training stable on an 8 GB GPU:

​Masking intra-chunk decay terms with -1e30 before torch.exp() and re-zeroing them afterward.

​Using 0.1 * lr with zero weight decay in AdamW for the recurrent parameters (alpha, beta, mu, conv and v_residual) while keeping standard lr and 0.1 weight decay on the linear projections.

​Chunking the cross-entropy loss (ce_rows=4096) under torch.utils.checkpoint to avoid materializing ~536 MB float32 logit tensors per batch item, alongside per-block gradient checkpointing.

​The scan and boundary masking implementation is in ARCHITECTURE.md if anyone wants to compare notes. Iff anyone is doing linear attention or SSM scans in PyTorch keeping the inter chunk state in float32 or using a different parameterization to avoid bf16 rounding near 1.0?


r/pytorch • • 4d ago

I fine-tuned Qwen2.5-7B with full AdamW on a 24 GB laptop GPU by moving the optimizer state to CPU RAM (adambridge, open source)

10 Upvotes

I've been working on adambridge, a drop-in AdamW for PyTorch. It keeps only the weights on the GPU. The FP32 master weights and both Adam moments live in host RAM, and fused SIMD kernels (AVX2/AVX-512/NEON) update them on the CPU. The updates run layer by layer while the GPU is still doing the backward pass.

Version 0.2.0 is out. The main result: Qwen2.5-7B full fine-tuning on an RTX 5090 Laptop (24 GB) ran at about 1,122 tokens/s, using 18.2 GiB of GPU memory and 38 GiB of host memory under WSL 2. This is full-parameter AdamW, not LoRA.

Benchmarks

I compared it against DeepSpeed 0.19.9 (ZeRO-Offload, ZeRO-Infinity, ZenFlow, SuperOffload), torchao 0.18, bitsandbytes 0.50.2 and plain all-GPU PyTorch, all on the same machine:

result
Largest model trained8.02B params (next best library: 2.08B)
Qwen2.5-0.5B / 1.5B vs DeepSpeed's best+113% / +120% tokens/s at the same quality
Qwen2.5-3Bonly adambridge fit it
When the model already fits on the GPUties all-GPU AdamW (gpu_state_budget="auto")

No quality loss

The low-memory options that drop the FP32 master copy all lost quality. That covers BF16-only weights, bitsandbytes 8-bit with BF16 weights, and torchao offload with BF16 weights. At fine-tuning learning rates, most AdamW updates are smaller than half a BF16 step, so they get rounded away. On Qwen2.5-0.5B, only 18% of the weights changed at all, compared with 84% for the reference.

adambridge keeps the FP32 master copy on the host, and its final validation loss stayed within run-to-run noise of torch.optim.AdamW.

Usage

import
 adambridge
opt = adambridge.AdamBridge(model.parameters(), 
lr
=1e-4, 
weight_decay
=0.01,
                            
precision
="bf16")   
# 'fp32' | 'bf16' | 'fp8'
opt.attach_hooks(model)   
# async: each layer is updated as soon as its grad is ready
for
 x, y 
in
 loader:
    loss = loss_fn(model(x), y)
    loss.backward()
    opt.step_sync()

Without attach_hooks it works as a normal optimizer (step() / zero_grad()), and LR schedulers and clip_grad_norm_ still work.

What's new in 0.2.0

  • FP24 master weights: 3 bytes per parameter instead of 4, with stochastic rounding. Combined with FP8 moments, the host state is about 5 bytes per parameter, and quality matched the reference at 0.5B, 1.5B and 3B.

  • Chunked causal-LM loss: a 150k-token vocabulary no longer has to hold its full logits tensor in memory.

  • Delayed updates: the CPU update overlaps the entire next step.

  • LionBridge: the Lion optimizer on the same pipeline.

  • Shared pinned transfer ring: about 30 GB less host RAM for a 7B model.

  • Compatibility: works with the HF Trainer and torch.compile.

  • Kernels: written in plain C, shipped as source inside a py3-none-any wheel and compiled on first use. If no compiler is available, it falls back to pure PyTorch kernels.

Limitations

  • I only benchmarked on one laptop. Desktop and server GPUs hold their clocks better when idle, which changes the numbers for every offload method.

  • The AVX-512 path has never actually run on real hardware (my CPU is AVX2). NEON was tested only under QEMU.

  • There's no DDP/FSDP support yet, and FP16 with GradScaler isn't covered.

  • Quality was measured on short runs at a single learning rate.

  • Global gradient clipping isn't possible in async mode.

pip install adambridge. It's Apache-2.0 licensed.

GitHub: https://github.com/johnanishg/adambridge

I'd like feedback, especially from anyone with an AVX-512 CPU or a multi-socket server. Running python -m adambridge info and the self-test and posting the results would help a lot.


r/pytorch • • 5d ago

[Open Source] Building DiffusionEngine – A Modern, Modular DDPM Framework in PyTorch. Looking for Contributors

6 Upvotes

What if we could bring modern diffusion research together into one unified, extensible PyTorch framework instead of implementing and maintaining every algorithm in isolation?

That's the idea behind DiffusionEngine, an open-source project I've been building:

https://github.com/Himanshu7921/DiffusionEngine

The goal is to build a modern DDPM framework that brings together established diffusion algorithms, research-based improvements, advanced sampling methods, and training strategies under a single, consistent pipeline.

The project already includes several diffusion implementations, and I'm actively working on extending the existing DDPM framework with more advanced features.

I'm looking for contributors interested in deep learning, generative models, PyTorch, and diffusion research.

Some areas where contributions are welcome include:

  • Implementing diffusion algorithms from research papers.

  • Improving existing DDPM components and training strategies.

  • Developing and integrating advanced sampling methods.

  • Exploring improved model architectures and conditioning mechanisms.

  • Testing, benchmarking, and optimizing existing implementations.

The focus is not just on implementing more algorithms. It's about making them work together seamlessly through a unified, modular, and configurable framework.

If you're interested in contributing, collaborating, or learning through hands-on implementation, I'd be glad to have you involved.

Contributors of all experience levels are welcome, provided they're willing to learn, implement, and test their work.

GitHub: https://github.com/Himanshu7921/DiffusionEngine


r/pytorch • • 6d ago

Accessing a model's individual layer

3 Upvotes

I want to access one of a LLM's individual layers and get output from it (like how nn.Linear() works). I tried using model.layer[0](tokenizer.encode("hi")) but that wouldn't work. If someone can give me the proper way to do this, it would be appreciated. I'm using Python for this with the transformers and pytorch library


r/pytorch • • 6d ago

LeRobot DataLoader had a 0.26 ms median wait and 5 sec stalls.

Post image
3 Upvotes

I wanted to see if comparing two model training runs could tell me where a slowdown came from. So I found a LeRobot a data loading regression that maintainers had already fixed, so I ran the same job before and after their fix.

Before the fix, next(dataloader) took about 1.4 secs per step on average. Then I plotted the waits in order. The median was 0.26 ms, but about every 4th step waited over5 secs. After the fix, those stalls disappeared.

John6666 reran it on a T4 and found that the spacing between stalls changed with the number of workers. I put the plots and reproduction steps on Hugging Face.

PS: I used TraceML for the comparison (I maintain it). If you’ve seen DataLoader stalls repeat like this, I’d be curious what you measured next.


r/pytorch • • 8d ago

A static checker for PyTorch training loops (eval mode, loss inputs, gradient order, checkpoints)

2 Upvotes

I wrote a checker for mistakes in PyTorch training and evaluation scripts that do not crash but change your results. It reads the source with ast and never runs it, so it needs no GPU or data.

The PyTorch checks are:

  • model.eval() left on, or never called before evaluation

  • torch.no_grad() used without eval()

  • softmax or sigmoid in front of CrossEntropyLoss

  • zero_grad() called between backward() and step()

  • the loss added to a running total as a tensor, which keeps every graph alive

  • fp16 autocast without a GradScaler

  • a script that resumes from a checkpoint but never saved optimizer state

  • drop_last=True on a validation or test loader

  • a training DataLoader that never shuffles

  • no seed set

I ran it over the 93 files of pytorch/examples at a pinned commit and read every finding: 22 findings, 20 real, 2 harmless. A false positive on distributed autograd (it needs no zero_grad) is fixed. It goes by names like train_loader and val_loader and reads one file at a time, so it will miss things and be wrong sometimes.

uvx trainlint check train.py

Repo and the full list with examples: https://github.com/Satvik-jain/trainlint If you know of a PyTorch mistake that runs fine and quietly hurts results, tell me and I will see if it can be checked.


r/pytorch • • 10d ago

PyTorch training issue in one dataset while failed in another

1 Upvotes
PyTorch version: 2.14.1+cu132, Torchvision version: 0.29.1+cu132
C:\actions-runner_work\pytorch\pytorch\aten\src\ATen\native\cuda\Loss.cu:244: block: [0,0,0], thread: [8,0,0] Assertion `t >= 0 && t < n_classes` failed.

torch.AcceleratorError: CUDA error: device-side assert triggered
Search for `cudaErrorAssert' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
The CUDA driver logged these messages, which may provide useful details:
Returning 710 (CUDA_ERROR_ASSERT) from cuStreamSynchronize

I find the issue. It seems image error. The dataset is Birds Classification Dataset in NingXia (China). And there is an image images\\082.Calidris_temminckii\\Calidris_temminckii_0034_788526.jpg cannot be load in Pillow.

broken data stream when reading image file

r/pytorch • • 12d ago

Stop typing x.shape: tensor shapes right in the VS Code debugger

Post image
1 Upvotes

Small free extension: the debugger's Variables view shows the shape next to every tensor, so you can step through a model and watch shapes change without printing anything.

Works with the normal Python debugger, also NumPy, pandas and plain lists/dicts. It hooks into the debugger itself, so your prints and logs stay unchanged.

MIT: https://github.com/marekprovaznik/dims

VS Code Marketplace / Open VSX: provaznik.dims (works in Cursor too)


r/pytorch • • 13d ago

Fixing PyTorch CUDA Illegal Memory Access

Thumbnail interconnectd.com
1 Upvotes

Dealing with a CUDA error: an illegal memory access was encountered in PyTorch can be frustrating, especially when it crashes your entire CUDA context and forces a process restart.

​This usually happens when a kernel attempts to read or write to an invalid memory address. Common triggers include out-of-bounds tensor indexing, device mismatches between tensors, or hidden race conditions in custom CUDA extensions.

​To debug this effectively, set CUDA_LAUNCH_BLOCKING=1 in your environment before running your script. By default, CUDA kernel launches are asynchronous, which means errors report long after the actual line of code that caused them. Forcing synchronous execution pinpoints the exact line triggering the fault.

​If you are fighting this issue right now, check out the full troubleshooting guide and code fixes here: https://interconnectd.com/forum/thread/267/fixing-pytorch-cuda-illegal-memory-access-context-corruption/


r/pytorch • • 15d ago

Simulusion: reusable PyTorch building blocks for world models (RSSM, VQ tokenizers, ST-Transformer, DiT/EDM)

3 Upvotes

Sharing a library I've been working on, written in plain PyTorch and MIT-licensed.

Simulusion started as a collection of world-model implementations and turned into a framework: every model is built from exported nn.Module components you can use on their own:

- RSSM / ModularRSSM (swappable priors, posteriors and heads)
- VQ-VAE and video tokenizers
- Spatiotemporal Transformer (Genie-style)
- DiT and EDM diffusion backbones
- ViT encoders for I-JEPA

The full models (Dreamer, PlaNet, IRIS, DIAMOND, Genie) are assembled from those parts and share one Trainer with bf16/fp16, resuming and checkpointing. You can register your own model and train it the same way. Export to ONNX, TorchScript and TensorRT is built in.

No hard pin on a PyTorch wheel, so install whatever CUDA/ROCm/CPU build you need first.

Feedback on the module APIs is very welcome, especially anything that feels un-PyTorch-like.

Github: https://github.com/paramthakkar123/simulusion
Website: https://simulusion.vercel.app/


r/pytorch • • 16d ago

How to Fix PyTorch CUDA Not Available on AMD GPUs (ROCm Guide)

Thumbnail interconnectd.com
3 Upvotes

​If PyTorch keeps telling you that CUDA is not available on your AMD graphics card, you are likely dealing with a broken ROCm runtime environment or missing environment variables. This is one of the most frustrating roadblocks when trying to run local models on Radeon hardware.

​I put together a complete troubleshooting guide that walks through cleaning up conflicting installations, setting the proper device flags, and verifying your ROCm-enabled PyTorch build step by step. It solves the dreaded availability check returning false issue once and for all.

​If you are tired of digging through outdated forums and GitHub issues, check out the full fix at https://interconnectd.com/blog/305/fix-pytorch-cuda-not-available-on-amd-gpus-rocm-setup-guide/

Drop a comment if you run into any specific error codes.


r/pytorch • • 16d ago

torchinfo-viz: added an interactive model view to torchinfo

3 Upvotes

torchinfo-viz allows u to see ur Pytorch model's each layer, how layers connected (including skip connections), the tensor shapes, and which layers hold the most parameters.

pip install torchinfo-viz.

Github: https://github.com/woletee/torchinfo-viz


r/pytorch • • 17d ago

Run the same PyTorch script locally, as DDP, or on several machines by changing flags

Thumbnail github.com
2 Upvotes

You state what a job needs (accelerator, GPU memory, frameworks). GPUlayer picks a worker that has it, ships code and inputs, streams logs and returns outputs and checkpoints. If nothing fits, you get the reason for each worker.

PyTorch side: a small Trainer plus launch() for existing scripts. The same file runs single-process, as DDP, or across nodes by changing flags. On a T4 the Trainer was 1.3 to 1.4x faster per step than Lightning on small models and equal on a compute-bound CNN. Benchmarks with raw logs: [link to perf doc]

Tested on CPU, Apple Silicon (MPS, MLX) and NVIDIA T4. Multi-GPU per node and cross-machine NCCL are implemented, not validated yet. Feedback welcome, especially from people with multi-GPU boxes.

GitHub: https://github.com/antonellof/gpulayer


r/pytorch • • 18d ago

When PyTorch training became unexpectedly slow, how did you investigate?

Thumbnail reddit.com
1 Upvotes

r/pytorch • • 20d ago

When PyTorch training became unexpectedly slow, how did you investigate?

2 Upvotes

I’m a beginner building a small training-timing prototype, and I’m trying to understand developers’ actual debugging experiences before developing it further. So far, I’ve only tested controlled CPU experiments.

Thinking about the last time your PyTorch training became unexpectedly slow, especially with multiple GPUs:

- What happened?

- Which tools or measurements helped you investigate?

- What was hardest to figure out, and roughly how long did it take?

I’d appreciate specific experiences, including cases where existing tools already solved the problem well.


r/pytorch • • 22d ago

Beyond seeds: which determinism flags actually mattered for reproducible fine-tunes on a T4?

1 Upvotes

I set torch, random, and numpy seeds, and my small fine-tune still moves run to run on a T4. Same recipe, same data — the weights at the same step differ between runs.

For those of you who actually got reproducible fine-tunes: which flags mattered in practice — math attention backend only, a cuBLAS workspace config, cudnn.deterministic plus benchmark=False, or torch.use_deterministic_algorithms as a diagnostic? And what slowdown did you accept for it? I'm trying to find out which of these is the one that actually closes the gap, rather than turning them all on and hoping.

Note: this question is posted by my AI specialist on my behalf — I am the founder, this is my real problem, and I read every reply myself.


r/pytorch • • 26d ago

Pytorches Finale

2 Upvotes

So for a big explosive demonstration of pytorches, my plan was to run a 17GB flux model on my machine in various ways to demonstrate how something new could beat any current pytorch implementation.

For background my machine has both an 8gb nvidia chip, and an intel gpu with 32gb shared ram.
The approach was to allow pytorches to break parts of each of the transformer layers

It was supposed to go like this:
Run the demo using pytorch compiled for cuda ->OOM
Run the demo using pytorch compiled for xpu -> pretty slow
Run the demo using pytorches (which uses cuda and xpu and distributes work) and bam, it works relatively fast.

<< YOUR BLOWN AWAY, AMAZING, I"LL SMILE AND WAVE>>>

What actually happened first of all is that the latest cuda driver pulls in shared memory now too, it trys to offset work and move the model within its limited memory. This makes running an oversized model possible, not the dead end I expected.

Then secondly I'm having real problem getting matmul to perform on my device driver version of xpu. It works, but I'm just not getting near native pytorch speeds there I get like 5Tflops vs 35 on pytorch. Its frustrating but probably totally naive on my part to think I could optimize up to pytorch speed on my own writing a driver myself.

So anyway thats where I am. I'm not sure how much more time I'll put into making my own xpu device driver just to prove this concept but I do believe a mixed hardware implementation with a planning component could unlock lots of exciting things in the future. Just have to prove it.


r/pytorch • • 27d ago

Amazon launching tool that lets its AI chips run PyTorch

Thumbnail
linkedin.com
14 Upvotes

r/pytorch • • Oct 15 '26

pytest-torch: PyTorch model-testing now made easier

Thumbnail
reddit.com
2 Upvotes

r/pytorch • • Oct 15 '26

Differentiable simulations, one per sample, on the GPU, feeding PyTorch without writing them as tensors

Post image
0 Upvotes

Putting physics in a training loop often means running many small simulations that end at different times. Written as tensor code, every simulation keeps running until the last one finishes.

hawk lets you write the step for one sample as a typed Python function. It compiles it for the GPU and the CPU, and gives you its forward and reverse derivatives. eagle then runs a million of them and drops the ones that finish. The result plugs into torch.autograd, and tensors move to and from PyTorch without copies.

It is deliberately not a tensor framework: no layers, no training stack. PyTorch keeps doing that part. With a million samples in double precision that stop at different steps, the simulation part runs 9.8× to 130× faster than the same step written for torch.compile (130× on an H100, 11× on a free Colab T4), and 2.5× to 4.2× faster than an NVIDIA Warp kernel. JAX support is being certified for one of the next patch releases, and I have not benchmarked Triton yet.

Personal project; I work at ESA. There is a notebook that simulates, takes derivatives, and fits a drag coefficient with PyTorch: https://amasat01.github.io


r/pytorch • • Oct 15 '26

Tensor Golf - Master PyTorch tensor operations via focused drills

Post image
3 Upvotes

Hey everyone, I’ve just released Tensor Golf: a collection of focused PyTorch challenges for getting comfortable manipulating tensors.

I found that following ML tutorials and writing tensor code from scratch were two very different things. Understanding a model’s architecture doesn’t necessarily mean you can confidently reshape, broadcast, mask, or reduce its tensors.

Tensor Golf gives you an input tensor and a target output. Your job is to write the operations that transform one into the other.

The “golf” part is trying to solve problems in fewer operations, but the main goal is building fluency: recognising which operations fit a problem and understanding why they work.

There are currently many PyTorch problems all free on the platform, with some premium ones on the way to support the development of the platform :)

I'd love feedback from people learning or working with PyTorch.

link: https://tensor.golf


r/pytorch • • Oct 14 '26

Pytorch AMD XDNA2 (ryzen AI processors in new laptops) inference and training backend

Thumbnail
reddit.com
3 Upvotes

r/pytorch • • Oct 13 '26

Introducting pytest-torch!

0 Upvotes

I built pytest-torch to make testing PyTorch models easier

Writing the same device setup, reproducibility checks, and output comparisons across projects gets repetitive, so I built pytest-torch, an open-source pytest plugin.

It includes inference checks, compile/export parity, checkpoint tests, accuracy evaluation, and layer-by-layer diagnostics to help locate where two models diverge.

It’s currently in alpha, and I’d appreciate feedback—especially examples of model bugs that are frustrating to test or reproduce.

GitHub: https://github.com/AKJ7/pytest-torch

Thanks and please drop a star if possible.