r/MLQuestions • u/PainterNo8889 • 2d ago
Hardware 🖥️ [Project / Help Needed] Testing GPU detection & TFLOPS benchmark paths (NVIDIA/AMD) for my from-scratch SLM trainer
Hi guys,
I’m a 15-year-old developer building SLM-CREATOR—a from-scratch, hardware-aware trainer for small, hyperspecialized text models (capped at 100M parameters) written in Python and Rust. Instead of relying on high-level libraries or fine-tuning existing models, I’m hand-coding the entire pipeline to learn how Transformers, tokenizers, and hardware-aware training work under the hood.
Project Link: github.com
Tracking Issue (#1): github.com
Why I'm asking for help here:
I’m currently coding this on a CPU-only machine, so my NVIDIA (nvidia-smi), modern AMD (amd-smi), and legacy AMD (rocm-smi/rocminfo) detection and PyTorch device-mapping code has only been tested against mock CLI outputs.
If anyone here has an NVIDIA or AMD GPU (especially multi-GPU setups, laptops with hybrid graphics, or ROCm setups) and a couple of minutes to spare, I’d really appreciate it if you could run architecture.py on real hardware and let me know if anything breaks or gets misidentified!
How the hardware detection & sizing works:
GPU Topology & Detection: Parses vendor CLI tools for architecture, total VRAM (MiB), device index, and GPU name.
PyTorch Device Resolution: Sets
CUDA_DEVICE_ORDER=PCI_BUS_IDand matches the vendor tool's GPU name againsttorch.cuda.get_device_name()so multi-GPU rigs target the right card.Matmul TFLOPS Benchmark: Runs a 12-second thermal warmup soak (to settle past laptop boost clocks), then measures the median TFLOPS across 10 runs of 20
4096x4096matmuls—using nativebf16when supported (including_emulation=False) or falling back tofp32.Rust Sizing Engine (
sizer_core): Takes your detected VRAM, measured TFLOPS, training-time budget, and token count to deterministically compute the max parameter count across memory, Chinchilla compute-time, and data limits.
How to test it:
Clone the repo:
git clonehttps://github.comSet up a venv and install
requirements.txt(with your GPU's PyTorch build).Build the Rust sizer:
cd sizer_core && cargo build --releaseRun
pythonarchitecture.pyfrom the project root.Drop your GPU model, OS, and the terminal output (or any tracebacks) in Issue #1 or in the comments below.
Any feedback on CLI parsing bugs, ROCm edge cases, or my sizing math would be super helpful!
1
u/Able_Region_5459 1d ago
On the detection side, parsing CLI text output from tools like nvidia-smi or amd-smi tends to be pretty fragile across different driver versions and operating systems. Also, matching cards by string name via get_device_name will break as soon as someone has two identical GPUs, like dual 3090s. PyTorch's torch.cuda.get_device_properties already gives you memory, names, and compute capabilities out of the box on both CUDA and ROCm (you can check torch.version.hip to tell them apart), and for NVIDIA-specific metrics nvidia-ml-py is much cleaner than parsing subprocess output.
The 12-second thermal soak before benchmarking is a really smart touch, especially for laptops. Just keep in mind that 4096x4096 matmuls measure peak compute, whereas sub-100M parameter models use much smaller tensors and usually hit memory bandwidth and kernel launch overhead long before peak TFLOPS - so your Chinchilla time estimates might end up looking more optimistic than actual training. Also, consider adding an fp16 step before falling back all the way to fp32 so older cards without native bf16 do not lose out on tensor core speed
2
u/jahangir670508 2d ago
please share the full repository url