r/CUDA • • Sep 20 '26

Do you need a free gpu performance engineer?

0 Upvotes

I lied, I cant assume the “engineer” position yet. Here’s what I am though. My name is Octav, Im 20 freelencer (or agency owner - that I run with my partner) specialising in backend and infra. I do contracts for different businesses (American field system Saas, bureaucracy automation startup, blockchain arbitrage grid bot (random, but made sense as i interned as a smart contract developer for 3 months) and more.

Me and my partner have been getting into the gpu performance engineering / cuda programming “niche”. Read the holy book, some c efficiency projects, and are working our way through nvidia certifications.

Now all of learning/research has been really good. But I am eager to apply it to something real. I am willing to learn and grow, and Im looking for some sort of mentoring. What I can offer back is my time and dedication towards your needs.
If you are working on a project, or are a startup, and have some work that I could do for you, please hit me up. Or just providing me with some directions would be really helpfull.


r/CUDA • • Sep 16 '26

Cuda on Mac

0 Upvotes

I've got Cuda running on an external gpu on my Mac using Tiny GPU for connection and a custom Nvidia driver and shim to run the gpu. Check out the work at https://github.com/Davinchy/macuda

It runs near native speed and actually does some stable diffusion faster than on windows.


r/CUDA • • Mar 26 '25

Unlocked RTX 5080 Benchmarks

Thumbnail
gallery
0 Upvotes

I have included the link to current benchmark for NVIDIA RTX unlocked 5080.

https://www.passmark.com/baselines/V11/display.php?id=250827543712


r/CUDA • • Nov 15 '25

PyTorch fooled everyone. Nightlies are pretending to support sm_120 but they’re silently compiling your RTX 5080 as sm_89.

0 Upvotes

PyTorch has pulled off one of the most effective “nothing to see here” illusions I've ever seen in GPU computing.

People think their RTX 5080 / Blackwell cards are running with true sm_120 support just because the nightly wheels claim to include it. The reality is brutal:

🔍 The nightlies are NOT running your GPU as sm_120.

They’re patching around it by quietly compiling the PTX as sm_89, then handing it off like nothing happened.

Yeah, the wheel “works.”
Yeah, torch.cuda.is_available() returns True.
Yeah, your model trains.
But here’s the hidden tax:

⚠️ You lose 20–30% of your compute power.

Every kernel routed through sm_89 PTX =
• Lower occupancy
• Wasted tensor core paths
• Reduced warp scheduling efficiency
• Artificially throttled FP16/BF16 throughput
• ~20–30% real-world loss vs. native sm_120

I confirmed this by reverse engineering the pipelines and checking the PTX dispatch behavior. The fake “sm_120” support is simply a compatibility shim.

🧬 The cause?

A broken PTX chain:

sm_120 → PTX output → silently downgraded → sm_89 backend

The wheels advertise sm_120, but the generated PTX tells the truth.

I had to manually patch the dispatch path myself to unlock full Blackwell performance. Only after fixing the PTX pathway and bypassing the downgrade did the card hit its real performance ceiling.

Once unlocked, the RTX 5080 jumps into performance territory that PyTorch users haven’t even seen yet.

🧨 Why this matters:

Developers think their 5080 is underperforming.
Benchmarks look “fine but not amazing.”
Performance variation looks random.

It’s not.
It’s the PTX.

Until true sm_120 backend support lands, you are not getting full Blackwell compute—even if the wheel says you are.

This isn't a conspiracy theory. It’s a reproducible, verifiable behavior in the current nightly PTX chain.

If PyTorch wants Blackwell adoption to be smooth, this needs to be fixed at the compiler and dispatch level, not wallpapered over with fake arch tags.

If you want the technical breakdown or proof-of-concept patch, I can share more details.

PyTorch has fooled all of you so well. These nigihtlys are passing sm89 off as sm120, yeah your machine works but its costing you loss of compute power. 20 to 30 percent worth. its all due to the ptx files.

EDIT:

I'm done replying to the noise here — Reddit arguments don’t change facts.
Here’s the only thing that matters if you actually care about performance:

✔ The current PyTorch nightlies do not generate true sm_120 PTX.
✔ They silently dispatch via sm_89.
✔ The throughput penalty is measurable and reproducible.
✔ The patched driver + patched PTX path unlock the missing Tensor Core utilization.

If you’re skeptical, perfect — reproduce it.
Build PyTorch from source with full arch flags, inspect the PTX, run Nsight Compute, and compare Tensor Core saturation.

If you don’t see the downgrade, publish your findings.
If you do, welcome to the party.

This thread won’t be my proof — the repos and the Nsight profiles already are.


r/CUDA • • 10d ago

I built an AI agent that ported Karpathy's llm.c GPT-2 trainer from CUDA to SYCL, so it runs on NVIDIA, AMD and CPUs (open source)

2 Upvotes

For Meta's Global AI Developer Hackathon I built Muse Port, an open-source tool that takes CUDA code (which only runs on NVIDIA GPUs) and ports it to SYCL, an open standard that runs on NVIDIA, AMD and regular CPUs.

How it works:

  1. A scanner (no AI) inventories the repo: every kernel, every NVIDIA library call and its oneAPI equivalent, and the CUDA-only parts that need real redesign.

  2. Meta's Muse model writes a migration plan, which gets checked against that scan.

  3. An agent loop ports each piece: write, compile, verify.

  4. A hidden check the model never sees decides whether a piece is done. Then the ports run on real NVIDIA GPUs against the original CUDA, and on AMD GPUs.

Results (all linked to committed evidence in the repo):

- llm.c's production bf16 trainer (12 files, 4,884 lines) ported in one run; it passes llm.c's own test on CPU, NVIDIA and AMD

- tuned in portable SYCL + oneMath (no NVIDIA-specific code) to 1.40-1.55x CUDA's time per training step on an L4 and RTX A4000

- 18 of 19 llm.c kernel files correct; llama2.cu ported for $0.45 with byte-identical output on CPU, NVIDIA and AMD

Limits: the one-run port worked 1 of 2 times, and the H100 is at 2.90x. Every failure is documented in the repo too.

Code: https://github.com/checocadena/museport

2-min demo: https://youtu.be/uBpT5IuxPOY


r/CUDA • • Jan 21 '26

Run 'gazillion-parameter' LLMs with significantly less VRAM

0 Upvotes

Hey guys, I’m embarking on a test this year to see if I can break the VRAM wall. I’ve been working on a method I call SMoE (Shuffled Mixture of Experts). The idea is to keep the 'Expert Pool' in cheap System RAM and use Dynamic VRAM Shuffling to swap them into a single GPU 'X-Slot' only when needed. This means you can run 'gazillion-parameter' LLMs with significantly less VRAM and less energy, making it a viable solution for both individual users and companies. Can't wait for your remarks and ideas!

https://github.com/lookmanbili/SMoE-architecture/blob/main/README.md


r/CUDA • • Mar 12 '19

CUDA Warp Primitives | __shfl_sync Tutorial | Make your cuda application faster

0 Upvotes

A video tutorial on cuda __shfl_sync at cudaeducation.com/shflsync


r/CUDA • • Jun 18 '25

My RTX 4090 Laptop Keeps Crashing When Compiling Large CUDA Projects

0 Upvotes

I'm running a C++ deep learning project on a Windows-based gaming laptop equipped with an RTX 4090. The project includes a significant amount of CUDA code, and I’ve noticed a frustrating issue: once the codebase grows large enough, compiling with nvcc occasionally causes the system to freeze, crash, or even blue screen. The crashes seem to happen during the compilation process — not during runtime training or inference. When I compile the same project on another workstation laptop with an RTX 5000 Ada, or a cloud GPU instance, everything works smoothly with zero issues. Has anyone else seen this kind of behavior?What is the reason of this issue?

Here’s my current environment on the RTX 4090 laptop:

  • Driver Version: 561.03

  • CUDA Version: 12.6

  • OS: Windows 11

  • nvcc: Cuda compilation tools, release 12.6, V12.6.85


r/CUDA • • Jun 25 '25

Write Mojo kernels & win a 5090, 5080, or 5070

Thumbnail
lu.ma
0 Upvotes

r/CUDA • • Aug 21 '26

I wrote a GPU kernel that speeds up AlphaFold-style protein models.

1 Upvotes

The creators of AlphaFold won a Nobel Prize in 2024. I just made its family of open source models faster.

I created fast_trimul, a drop-in, hardware-agnostic library for Fused Triangle Multiplicative Updates across AlphaFold3 family models, powered by CuTe DSL. In addition, it is licensed with Apache-2.0.

The Triangle Multiplicative Update is a memory-heavy operation in protein-structure models.

Also, it is very simple to integrate fast_trimul with other Python libraries. It has a modular architecture, and it was designed to be easy to use in production.

In OpenFold-3:

● Graph vs. ungraphed: Graphed eliminates kernel launch overhead at small N (~22ms vs ~53ms ungraphed).

● fast_trimul output is identical to the OpenFold-3 version, with an invisible difference of about 0.0006%.

● If it fails, it always falls back to PyTorch. This way it never crashes.

● It is a modular and vendor-agnostic design. Supporting new hardware and new libraries like OpenFold-3 are all plug-ins, not a rewrite.

fast_trimul compared with other libraries' Triangle Multiplicative Updates:

● Runs 4.5–6.8× faster on short sequences

● Uses ~2.2–2.4× less peak GPU VRAM, fitting ~1.4× longer sequences before running out

● Works on any sequence length with zero recompilation. Unlike torch.compile, which recompiles for every new N.

It's written in Python CuTe DSL, so the kernel codebase stays small and easy to adapt for other GPUs like H100 or B200.

GitHub link: https://github.com/tiagomonteiro0715/fast_trimul


r/CUDA • • Feb 28 '22

Our savior

1 Upvotes

r/CUDA • • Jun 10 '26

How I dropped my local LLM VRAM usage by 4GB and permanently fixed CUDA OOM errors

0 Upvotes

If you are building sovereign AI tools locally, hitting the dreaded CUDA Out of Memory error is a daily battle. I recently managed to shave off 4GB of VRAM consumption without degrading output quality. Here is the exact breakdown of how I did it. First, Flash Attention 2 is non-negotiable; it optimizes memory reads and writes directly on the GPU, saving massive overhead. Second, lower your context window during the testing phase. You rarely need a 32k context when testing basic reasoning prompts, so cap it at 4k. Third, force 4-bit precision loading via bitsandbytes on your base models. It is the absolute easiest win for VRAM conservation.

Call to Action: If you want to see the complete code repository and the exact Python scripts I use for automated memory management, I put the sovereign engineer guide together here: https://interconnectd.com/forum/thread/184/fix-cuda-oom-on-local-llms-the-sovereign-engineers-guide/


r/CUDA • • Aug 21 '24

CUDA/ML Role in Sydney-based Trading Firm

0 Upvotes

Hi Team CUDA,

Scott Gilbert here. Headhunter for Westbury Partners. We work with Trading Firms globally.

I'm working with a Sydney-based, Tier 1 Market Maker Trading firm and am looking to fill a lucrative Machine Learning/CUDA role.

This is a very lucrative role that would come with visa and relocation for the right candidate.

If you're interested in a chat then please drop your CV to [email protected] or you can reach out via LinkedIn.

Looking forward to having a chat.

Regards,
Scott


r/CUDA • • Jan 10 '26

Please help me to run a code

0 Upvotes

Hi everyone, I made a Bart project for my uni nlp course( its just summarization code not that much ) , my laptop have a RTX 3060 but idk why vs code can't see it although i tried so many time , and the free GPU limit in colab ends so early l befor the code run , I was wondering if anyone can help me, run the code and send it to me , the submission is tomorrow and I can't submit the code without run , so pls if anyone can help me I will be grateful

ml #Bart


r/CUDA • • May 23 '26

Deterministic GPU Replay Demonstration

1 Upvotes

Deterministic GPU replay demonstration showing reproducible execution outputs across reruns and reboot-stable execution verification.


r/CUDA • • Dec 30 '21

Is it possible to run CUDA on 8 GPUs with mining rig setup? slow Memory Bandwidth is not a problem

1 Upvotes

My apology for dumb question.

Application need to run on many GPU RTX 3080ti, but server case can only hold 2 cards. I am considering open case like mining rig, where it can hold many GPU via raiser. Slow Memory bandwidth is not a problem. Does CUDA programs run on this setting? (3 cards on PCIe x16, rest run via x4 raiser)

Motherboard AMD B550 with 3 x16, few x4, RAM 64GB DDR4 3200, Windows10. GPU price is killing me but show must goes on, help!

Appreciate any assistance


r/CUDA • • Jun 23 '25

Help needed.

0 Upvotes

Can anyone help with a theory + hands-on or even hands-on only starters for getting in CUDA?


r/CUDA • • Dec 29 '25

sm_90 Logic Decay: My forensic audit of H100 stability vs. Isaac Lab simulations

0 Upvotes

I’ve been stress-testing autonomous reasoning models on H100 (sm_90) hardware, and I’m seeing something that simulation completely misses. I’m calling it “Stochastic Logic Drift,” and it seems to be a hardware-level limit that effectively creates a “4-hour barrier” for deterministic autonomy.

In standard Euclidean vector search, thermal noise and floating-point non-determinism accumulate over time. In my last 28,000+ query run, the LCP (Longest Common Prefix) depth decayed from 256 bits down to 244 bits after the chip hit ~72°C. Basically, the hardware entropy started overriding the model's weights.

I managed to "anchor" the logic by switching to p-adic ultrametric invariants. It kept a 100% bit-perfect lock throughout the entire run, even under peak thermal throttling.

I’ve uploaded the raw telemetry, the H100 hardware receipts (JSON), and the CUDA kernel I used to fix the substrate here:

https://gist.github.com/StanByriukov02/3686a8cd3da70effa5d848deb46753e7

My take is that we have a massive "Inference Liability" problem in robotics. If the substrate isn't deterministic, simulation parity is just an illusion.

Has anyone else here seen this kind of logic jitter on Hopper or Blackwell? Or are we just accepting this drift as "normal noise" and patching it with more RL?


r/CUDA • • Jul 23 '25

My GPU is too new for the precompiled CUDA kernels in Pytorch

0 Upvotes

I was giften an Aliemware with an RTX 5080 so I can execute my Master projects in Deep learning. However my GPU runs on sm_120 architecture which is apparently too advanced for the available PyTorch version. How can I bypass it and still use the GPU for training somehow?

Edit: I reinstalled the CUDA 12.8 through Pytorch nightly and now it seems to work. The first try didn't work because this alternative is apparently not compatible with Python 3.13, so I had to downgrade it to Python 3.11. Thanks to everyone.


r/CUDA • • Mar 17 '21

Installed graphics card but program doesn't run faster.

0 Upvotes

Hey,

My friend is working on a R program that imports and deals with a massive dataset. It takes hours for the program to process the data. She was told it would be useful to get a graphics card. So she installed a GeForce RTX 2070 SUPER on a PC running an i9 with 128gb of ram, so a pretty powerful machine.

She noticed that when she runs the program, in task manager, the GPU usage doesn't increase. She thought just installing the graphics card would improve the performance.

What are we missing? She uses R-Studio, is there a setting there to enable using the graphics card


r/CUDA • • Jul 12 '20

How to do a matrix calculation with a double pointer (C++ & Cuda)

0 Upvotes

I'm trying to do a Matrix*vector*vector calculation using CUDA with C++. For the matrix, I have used a double-pointer. In this particular case, I have to use a double pointer for calculation. And I get two errors.

"Program hit cudaErrorInvalidValue (error 1) due to "invalid argument" on CUDA API call to cudaMemcpy."

and

"Invalid __global__ read of size 4" for the [ tempsum += B[k] * *AA[(ix*nx)+k] * C[k]; ] line.

If someone knows how to solve this problem, would be very helpful.

Thank you.

Device code

__global__ void matcalc(double *A, double **AA, double *B, int *C, int ny, int nx){

unsigned int ix = blockDim.x * blockIdx.x + threadIdx.x;

unsigned int iy = blockDim.y * blockIdx.y + threadIdx.y;

unsigned int idx = iy * nx + ix;

double tempsum = 0.0;

if(ix < nx && iy < ny){

for (int k = 0; k < nx; k++){

tempsum += B[k] * *AA[(ix*nx)+k] * C[k];

}

A[(ix*nx)+iy] = tempsum;

printf("A[%d]= %f\n",(ix*nx)+iy, A[(ix*nx)+iy]);

}

}

Host code

const int M = 50;

const int N = 50;

int sizeN = N * sizeof(double);

int sizeM = M * sizeof(double);

int nxy=N*M;

int matsize = nxy * sizeof(double);

double **AA;

AA = (double**)calloc(N, sizeof(double*));

for (int i = 0; i < N; i++) {

*(AA + i) = (double*)calloc(M, sizeof(double));

}

double A[N];

double B[M];

double C[N];

int D[N];

double *d_A;

double *d_B;

double *d_C;

int *d_D;

double **d_AA;

d_AA = (double**)calloc(N, sizeof(double*));

for (int i = 0; i < N; i++) {

*(d_AA + i) = (double*)calloc(M, sizeof(double));

}

cudaMalloc((void**)&d_A, sizeM);

cudaMalloc((void**)&d_AA, matsize);

cudaMalloc((void**)&d_B, sizeN);

cudaMalloc((void**)&d_C, sizeN);

cudaMemcpy(d_AA, AA, matsize, cudaMemcpyHostToDevice);

cudaMemcpy(d_A, A, sizeN, cudaMemcpyHostToDevice);

cudaMemcpy(d_B, B, sizeN, cudaMemcpyHostToDevice);

int dimx=32;

int dimy=32;

dim3 block(dimx,dimy);

dim3 grid((N+block.x-1)/block.x, (M+block.y-1)/block.y);

matcalc<<<grid,block>>>(d_A, d_AA, d_B, d_C, M, N);

cudaDeviceSynchronize();

cudaMemcpy(C, d_C, sizeM, cudaMemcpyDeviceToHost);


r/CUDA • • Feb 29 '20

Any tips for improving performance of my program?

0 Upvotes

I'm trying to write my own ray marching (sphere tracing) algoritm. I'm using Cuda for calculating if the rays hit, but if I time how fast it is I get an average of 50ms for an 1000x1000 image (only using the color red to color). I'm using a GeForce GTX 1060 6GB. This is my code:

#include <math.h>

#include "cuda_runtime.h"

#include "device_launch_parameters.h"

const unsigned int W = 1000;

const unsigned int H = 1000;

__constant__

const float epsilon = 0.01;

struct vec3

{

float x;

float y;

float z;

__host__ __device__

vec3(float x, float y, float z) : x(x), y(y), z(z) {}

__host__ __device__

vec3() {}

__host__ __device__

vec3 operator*(float scalar) { return vec3(x*scalar, y*scalar, z*scalar); }

__host__ __device__

vec3 operator/(float scalar) { return vec3(x/scalar, y/scalar, z/scalar); }

__host__ __device__

vec3 operator+(vec3 v) { return vec3(x + v.x, y + v.y, z + v.z); }

__host__ __device__

vec3 operator-(vec3 v) { return vec3(x - v.x, y - v.y, z - v.z); }

};

__host__ __device__

float length(vec3 v)

{

return sqrtf(v.x*v.x + v.y*v.y + v.z*v.z);

}

__host__ __device__

vec3 norm(vec3 v)

{

return v / length(v);

}

__host__ __device__

float dot(vec3 v1, vec3 v2)

{

return v1.x*v2.x + v1.y*v2.y + v1.z*v2.z;

}

struct Camera

{

vec3 p; //position of camera

vec3 dir; //theta, phi, fov

float width;

float height;

float cos_theta;

float sin_theta;

float cos_phi;

float sin_phi;

__host__ __device__

Camera() {}

__host__ __device__

Camera(vec3 p, vec3 dir) : p(p), dir(dir)

{

cos_theta = cos(dir.x);

sin_theta = sin(dir.x);

cos_phi = cos(dir.y);

sin_phi = sin(dir.y);

}

__host__ __device__

vec3 dir_vec(float i, float j)

{

float r = width / (2.0f*tan(dir.z / 2.0f));

vec3 x(cos_theta * cos_phi, cos_theta*sin_phi, -sin_theta);

vec3 y(-cos_theta * sin_phi, cos_theta*cos_phi, -sin_theta);

vec3 z(sin_theta*cos_phi, sin_theta*sin_phi, cos_theta);

return norm(z * r + x * i + y * j);

}

__host__ __device__

vec3 p0(float i, float j)

{

vec3 x(cos_theta * cos_phi, cos_theta*sin_phi, -sin_theta);

vec3 y(-cos_theta * sin_phi, cos_theta*cos_phi, -sin_theta);

return (p + x * i + y * j);

}

};

__host__ __device__

float dist(vec3 p)

{

return length(p) - 1.0f;

}

__host__ __device__

vec3 gradient(vec3 pos)

{

vec3 dx( 0.0f, 0.0f, epsilon );

vec3 dy( 0.0f, epsilon, 0.0f );

vec3 dz( epsilon, 0.0f, 0.0f );

return vec3(dist(pos + dx) - dist(pos - dx), dist(pos + dy) - dist(pos - dy), dist(pos + dz) - dist(pos - dz));

}

__global__

void render(unsigned char pixels[W*H], Camera camera)

{

unsigned int index = blockIdx.x * blockDim.x + threadIdx.x;

unsigned int stride = blockDim.x * gridDim.x;

camera.width = 1.0f;

for (unsigned int k = index; k < W*H; k += stride)

{

unsigned int j = k / W;

unsigned int i = k - j * W;

float x;

float y;

x = (float)i / W - 0.5f;

y = (float)j / H - 0.5f;

vec3 dir = camera.dir_vec(x, y);

vec3 p = camera.p0(x, y);

pixels[k] = 0x00;

for (unsigned int t = 0; t < 50; t++)

{

p = p + dir * dist(p);

if (dist(p) < 0.01)

{

pixels[k] = 0xff * dot(norm(gradient(p)), norm(vec3(-1.0f, 1.0f, 1.0f)));

break;

}

}

}

}

And:

#include <iostream>

#include <chrono>

unsigned char* pixels = new unsigned char[W*H];

Camera camera(vec3(0.0f, 0.0f, -3.0f) , vec3(0.0f, 0.0f, 0.5f));

cudaMallocManaged(&pixels, W*H * sizeof(unsigned char));

unsigned int threads = 256;

unsigned int blocks = 1000;

unsigned __int64 start = std::chrono::duration_cast<std::chrono::milliseconds>
(std::chrono::system_clock::now().time_since_epoch()).count();

camera.p.z -= 0.01f;

render<<<blocks, threads>>>(pixels, camera);

cudaDeviceSynchronize();

unsigned __int64 end = std::chrono::duration_cast<std::chrono::milliseconds>(std::chrono::system_clock::now().time_since_epoch()).count();

std::cout << end - start << std::endl;

Are there are any optimizations or ways to speed it up?

Thank you in advance.


r/CUDA • • Jul 11 '26

Install Cuda on WSL Ubuntu

Thumbnail
darryldias.me
0 Upvotes

Setting up CUDA on WSL Ubuntu for AI development – no more dual‑booting. 🚀

If you're running AI workloads on Windows and want GPU acceleration inside WSL, this one's for you 👇

https://darryldias.me/blog/install-cuda-on-wsl-ubuntu


r/CUDA • • Nov 07 '24

is 4060 CUDA capable?

0 Upvotes

I just bought a 4060 for my desktop only to be able to use cuda for machine learning task. The CUDA compatibility website does not list the 4060 for desktop as CUDA capable. Does that mean that i will not be able to use CUDA on my 4060?