r/MLQuestions • • 2d ago

Hardware 🖥️ [Project / Help Needed] Testing GPU detection & TFLOPS benchmark paths (NVIDIA/AMD) for my from-scratch SLM trainer

Hi guys,

I’m a 15-year-old developer building SLM-CREATOR—a from-scratch, hardware-aware trainer for small, hyperspecialized text models (capped at 100M parameters) written in Python and Rust. Instead of relying on high-level libraries or fine-tuning existing models, I’m hand-coding the entire pipeline to learn how Transformers, tokenizers, and hardware-aware training work under the hood.

Project Link: github.com
Tracking Issue (#1): github.com

Why I'm asking for help here:
I’m currently coding this on a CPU-only machine, so my NVIDIA (nvidia-smi), modern AMD (amd-smi), and legacy AMD (rocm-smi/rocminfo) detection and PyTorch device-mapping code has only been tested against mock CLI outputs.

If anyone here has an NVIDIA or AMD GPU (especially multi-GPU setups, laptops with hybrid graphics, or ROCm setups) and a couple of minutes to spare, I’d really appreciate it if you could run architecture.py on real hardware and let me know if anything breaks or gets misidentified!

How the hardware detection & sizing works:

  • GPU Topology & Detection: Parses vendor CLI tools for architecture, total VRAM (MiB), device index, and GPU name.

  • PyTorch Device Resolution: Sets CUDA_DEVICE_ORDER=PCI_BUS_ID and matches the vendor tool's GPU name against torch.cuda.get_device_name() so multi-GPU rigs target the right card.

  • Matmul TFLOPS Benchmark: Runs a 12-second thermal warmup soak (to settle past laptop boost clocks), then measures the median TFLOPS across 10 runs of 20 4096x4096 matmuls—using native bf16 when supported (including_emulation=False) or falling back to fp32.

  • Rust Sizing Engine (sizer_core): Takes your detected VRAM, measured TFLOPS, training-time budget, and token count to deterministically compute the max parameter count across memory, Chinchilla compute-time, and data limits.

How to test it:

  1. Clone the repo: git clone https://github.com

  2. Set up a venv and install requirements.txt (with your GPU's PyTorch build).

  3. Build the Rust sizer: cd sizer_core && cargo build --release

  4. Run python architecture.py from the project root.

  5. Drop your GPU model, OS, and the terminal output (or any tracebacks) in Issue #1 or in the comments below.

Any feedback on CLI parsing bugs, ROCm edge cases, or my sizing math would be super helpful!

1 Upvotes

6 comments sorted by

2

u/jahangir670508 2d ago

please share the full repository url

2

u/jahangir670508 2d ago

got it, let me try to run it

1

u/PainterNo8889 1d ago

would mean a lot sir

1

u/PainterNo8889 1d ago

did it work tho?

1

u/PainterNo8889 1d ago

thanks so much sir

1

u/Able_Region_5459 1d ago

On the detection side, parsing CLI text output from tools like nvidia-smi or amd-smi tends to be pretty fragile across different driver versions and operating systems. Also, matching cards by string name via get_device_name will break as soon as someone has two identical GPUs, like dual 3090s. PyTorch's torch.cuda.get_device_properties already gives you memory, names, and compute capabilities out of the box on both CUDA and ROCm (you can check torch.version.hip to tell them apart), and for NVIDIA-specific metrics nvidia-ml-py is much cleaner than parsing subprocess output.

The 12-second thermal soak before benchmarking is a really smart touch, especially for laptops. Just keep in mind that 4096x4096 matmuls measure peak compute, whereas sub-100M parameter models use much smaller tensors and usually hit memory bandwidth and kernel launch overhead long before peak TFLOPS - so your Chinchilla time estimates might end up looking more optimistic than actual training. Also, consider adding an fp16 step before falling back all the way to fp32 so older cards without native bf16 do not lose out on tensor core speed