r/HPC • u/AugustinesConversion • Dec 15 '25
NVIDIA Acquires Open-Source Workload Management Provider SchedMD
r/HPC • u/AugustinesConversion • Dec 15 '25
r/HPC • u/victotronics • Jul 26 '22
I've decided to put my textbooks under a unified umbrella.
Share and enjoy. Comments always welcome.
r/HPC • u/VanRahim • May 12 '26
We built this at the University of Alberta because we had a pile of L40S, A40, and other GPUs that SLURM couldn't meaningfully slice. Hardware MIG only covers a handful of models, requires draining nodes to reconfigure, and locks you into rigid layouts. Result: full 48GB cards going out for jobs that needed 12GB. Classic HPC waste.
SoftMig is a SLURM-native software slicing layer — a fork of HAMi-core adapted for cluster environments. It enforces per-job memory ceilings and compute throttling via LD_PRELOAD, with prolog/epilog hooks handling the job lifecycle. Works on any CUDA 12+ GPU.
A 48GB L40S becomes:
1 full GPU
2 × 24GB half-slices
4 × 12GB quarter-slices
...or whatever layout your site defines
Change layouts through SLURM policy. No node drain, no reboot.
A few things it does that hardware MIG can't:
Mix slice sizes on the same GPU (e.g. a half + two quarters on one card)
No lost capacity — hardware MIG burns memory to its own infrastructure; SoftMig slices the full pool
Compute is sliced too, not just memory — SM access is throttled proportionally per job
Heads up on build/install: The docs are written for Digital Research Alliance of Canada / Compute Canada cluster environments, so if you're deploying elsewhere you may need to adapt things. Claude Code or Cursor work well for navigating the compilation and integration steps if you're not in that ecosystem.
MIT licensed. GitHub: https://github.com/ualberta-rcg/softmig
Happy to answer questions — we've been running v1 in production on Vulcan and v2 is now in testing.
r/HPC • u/RadicalNation • Apr 11 '26
https://developer.nvidia.com/blog/running-large-scale-gpu-workloads-on-kubernetes-with-slurm/
Disclosure: I work for NVIDIA on Slinky.
Obligatory preface: All comments from me are my views and may not reflect the views of my employer.
I'm very proud to present this blog post to everyone. It's been amazing to build Slinky and see it used in production at scale already!
r/HPC • u/basnijholt • Apr 30 '23
r/HPC • u/tgamblin • Jul 20 '25
Spack v1.0 is out — it’s a major milestone; the core is reworked to add compilers as proper dependencies, and it introduces a stable package API. v1.0 also adds concurrent builds, better includes, and much more.
Check out the very detailed release notes here:
r/HPC • u/polycro • Nov 20 '24
It is our gen 3 cluster from 1993. This may be the third oldest object on the floor behind the Ferrari and the plane.
r/HPC • u/dr0p834r • May 30 '22
r/HPC • u/mastercoder123 • Apr 14 '26
Hey all, I made a post on here i think like a year ago talking about building a home HPC cluster. I just want to report on my current work since someone may find it cool that im doing this in my home and not in a datacenter.
I currently have 20 compute nodes that are r650's using xeon platinum 8630Y's and each compute node has 1024gb of ram split between 16 64gb ddr4 2933mhz ecc sticks. Each node is running a single boss S2 card for boot OS only, as well has has 2 dual port 100gbe connectx-5 mellanox NICs. One of the ports on each nic is connected to my 100gbe Ethernet network for storage utilizing rdma over converged ethernet and the other port on each nic is connected to my 100gb infiniband network.
For networking im currently running 5 arista 7160-32cq switches, 3 as the leaf switches and 2 for spine switching (1 is fine for my setup but i want to upgrade to 40 nodes in the near future and its less of a hassle to setup 2 switches as it is to setup 3). Im running 2 Mellanox SB7800 100Gb infiniband switches for the current infiniband network. Everything is connected via single mode fiber which took alot of finagling because the optics im using are Ethernet in their EEPROM so i had to buy a flex box to flash them to infiniband mode so the mellanox switches would stop bitching.
Lastly for storage and the head server. Im running 4 dell r7525's with 24 U.2 bays, but because of drive prices i currently only have 200tb of nvme drives split across all 4 servers. Everything is running in a software raid10 using BeeGFS (which was a PIA to setup lol). Each of the server has 2 dual port connectx-5 100gbe nics for a combined 400gb uplink per server to the storage network. The servers are currently just running dual socket epyc 7443 cpus since they were cheap and decent. The head server is what I and my few clients use to SSH in and run jobs on the nodes and its just a small r640 with some xeon gold 6240s and a single 100gbe connectx-4 nic to connect to the ethernet network so it can send out jobs and other commands. I do have a 10gbe WAN connection that allows my clients to upload and download their completed projects at decent speed because my 1gbe connection was taking days for people to download their projects.
This was a massive undertaking for me and cost about $100,000 which for a homelab isnt cheap but considering the compute power i have and the amount of customers i have and what they are willing to pay i have already made my money back. Currently I am running openHPC and Slurm to manage the cluster and as of now the only jobs the cluster runs is GROMACs and openFOAM. The main people that use the cluster are phd or masters students who want access to a supercomputer but dont want to wait months or years to access their university supercomputer.
I will say this was an awesome project and I learned so much more than i thought in just the area of procurement. Also 1 last thing, the coolest thing i have done so far on my cluster is render the effects of a 350kt nuclear warhead and its affects of the shockwave, radiation exposure and secondary fires. I had to borrow storage from a friend because this project took months to simulate and i was able to simulate 1 single minute of the effects of said nuclear weapon before me and the customer kinda gave up. It took 4 months for this single minute long simulation and used a total of 768tb of storage, but this simulation was doing the effects of every single microsecond after detonation.
Sorry this is so long winded I just thought sharing this would be cool and someone would find my autism interesting.
Edit: i wanted to add an edit for some more updates. I just recently got my current business approved to become a nonprofit that will be overseeing the use of the supercomputer which will give me access to real educational grants from the likes of states, universities and even the DOE. This is huge because grants are basically free money, you are not required to pay the money back like a loan so in the future i will be able to rapidly expand the size of the cluster without having to sink hundreds of thousands of dollars of my own money into it. It will also give me access to dell, supermicro, nvidia and other hardware company and their grants and other education programs which is huge.
I just wanna thank the HPC subreddit because without that simple question i asked all those months ago this shit wouldn't have existed and i wouldn't be where I am. In the near future i plan on applying for many grants that will allow me to upgrade the hardware and add many more nodes included the sought after gpu nodes. In the future i will be creating a website and a business email that will have the schedule for anyone to sign up to use and if you are a student, free of charge.
r/HPC • u/ElectronicDrop3632 • Nov 29 '25
A lot of recent AI outages highlight how sensitive high-density racks have become to sudden power swings and thermal spikes. HPC clusters face similar load patterns during synchronized compute phases, especially when accelerators ramp up or drop off in unison. Traditional room level UPS and cooling systems weren’t really built around this kind of rapid transient behavior.
I’m seeing more designs push toward putting a small fast-response buffer directly inside the rack to smooth out those spikes. One example is the KULR ONE Max, which integrates a rack level BBU with thermal containment for 800V HVDC architectures. HPC workloads are starting to look similar enough to AI loads that this kind of distributed stabilization might become relevant here too.
Anyone in HPC operations exploring in rack buffering or newer HVDC layouts to handle extreme load variability?
r/HPC • u/Omni-Vector • Jan 08 '26
What Does Nvidia’s Acquisition of SchedMD Mean for Slurm?
"If the HPC community doesn’t like the direction Nvidia takes Slurm, it could lead to a fork of the open source project."
If the fork occurs, is it actually realistic that the community would want to chase down the work contributed by Nvidia and a bunch more? Needless duplication of effort, anyone?
r/HPC • u/awfulalexey • Jun 23 '26
I run Chinese open models on a 4×3090 rig every day. The more I watched these models get tuned for domestic hardware, the more I wanted to know what that hardware actually is, so I mapped it. At least 7 Chinese companies are already shipping AI accelerators, and most of them IPO'd in the last 6 months.
China's own framing is "3 dragons, 4 snakes." The dragons are Big Tech that also builds full-stack GPUs. Huawei alone shipped 812K AI cards last year, 49% of China's domestic supply, with their own HBM and their own fabs. The Ascend 950 reportedly targets H200-class.
The "snakes" are the pure-plays that just IPO'd, and this is the part that surprised me: several were founded by the former chief GPU architects of NVIDIA and AMD. MetaX is basically AMD's old global GPU leadership rebuilt in Shenzhen, revenue up about 3,800x in three years. Alibaba is shipping a server with 16×96GB = 1.5TB of VRAM in one box, enough to hold a frontier model in BF16 fully on-prem.
Meanwhile production moved from TSMC to SMIC, and NVIDIA's China share fell from about 95% to 55% in two years. The metal and the open models are converging.
Full breakdown with all 7 vendors and sources:
r/HPC • u/gordicaleksa • Sep 29 '25
r/HPC • u/TheSquashManHimself • Nov 19 '22
r/HPC • u/throwaway761910 • Nov 11 '24
I'm going to SC24 in Atlanta, GA this weekend. This is my first time attending a tech conference, let alone a supercomputing conference
I recently started working as an HPC system admin and have been learning my job as I go. There's going to be a lot of topics, vendors, skills, and information at this conference and I'm feeling a little overwhelmed on where to start and what to do
Any recommendations for a first timer? Are there any sessions you think I should definitely attend?
r/HPC • u/Various_Protection71 • Sep 08 '26
Hello there!
I believe this material is worth of being share here in the channel.
I analyzed 276 HPC job offerings so you don't have to. Here's what the 2026 HPC job market actually looks like from this data:
🖥️ 60%+ of roles are for HPC Systems Engineers
📍 50% of jobs are in the USA (UK and India are growing hubs)
🔧 Linux, SLURM, and GPU are the bare minimum (80% of roles mention them)
💰 Median max salary: $208,000 (but the range is wide)
🚨 Only 3% of roles are entry-level! Yes, that's a problem
Plus: What skills actually pay off, whether certifications matter (spoiler: they usually don't), and how to stand out in a competitive market.
https://theparallelminds.substack.com/p/hpc-job-market-in-2026-what-276-open
r/HPC • u/ell-hol1 • 22d ago
A lot of CUDA tutorials throw the whole programming model at you almost immediately.
You learn the syntax, then grids and blocks, then memory types, then synchronization, and pretty quickly you’re juggling a bunch of concepts before you have a good feel for what CUDA is enabling you to do.
I wanted to try a slower approach.
Part 1 is mostly about building the basic mental model: what a kernel is, what it means to launch lots of threads, how those threads identify the piece of work they own, and how to start thinking about a problem in GPU terms.
The hardware details come later.
I wrote this mainly for programmers who are curious about CUDA but haven’t spent much time with GPU programming yet.
Would be interested in feedback from people here, especially on whether the progression feels natural or whether I’m introducing something in the wrong order.
Part 1: https://computejunkie.substack.com/p/cuda-programming-for-the-curious
Part 2: https://computejunkie.substack.com/p/cuda-programming-for-the-curious-f8f
r/HPC • u/polycro • Dec 16 '25
EDIT: Thanks for the interest! We have filled the positions.
https://explore.msujobs.msstate.edu/en-us/job/509345/computer-specialist-i-ii-iii-or-senior
Mississippi State had a NSF ERC site in the early 90's and has progressed to a multi site interdisciplinary research center. Still growing and providing more academic resources to the university while being separate from main campus IT. MSU has had a supercomputer appear on 43 TOP500 lists since its first appearance in June 1996, including the most recent November 2025 ranking.
https://www.hpc.msstate.edu/computing/history.php
New data center has been finished with a dedicated substation from TVA. Starting with 5MW and upgrade able to 20+ for 10k sq ft of data room with a 14' raised floor over utilities. Unlike most academic research centers we have power and space to grow for decades with lots of land and "cheap" electricity.
MSU has several positions open and funding to fill multiple positions for research computing. Candidates must be eligible for CUI clearance and have demonstrated experience with Slurm and Perl.
Salary: 60k-100k+ depending on education and experience.
Benefits:
- 99% 8-5 working hours
- 15-16 days of University holidays a year
- 18 days of PTO on year one (accumulated at 12hrs/mo) Grows to 27 days at 18hrs/mo. Is paid out on separation or retirement.
- Medical leave accrues at 8hrs/mo
- Generous travel budget for conferences and training. Yearly representation at SC Conference.
- State retirement system
- Tuition waivers to peruse any MSU degree including MS or PHd in CS, Information Security, or Computational Engineering
- Starkville named best small town town in the South
r/HPC • u/nuclear_knucklehead • Apr 29 '23
r/HPC • u/TheDevilKnownAsTaz • Oct 29 '25
r/HPC • u/Wells1632 • Mar 27 '25
I know that the data center I am at isn't even remotely ready for something like this. We were only just starting to plan for the requirements of 130kW per rack, and this comes along.
As far as I can tell, this kind of hardware in any sort of scale is going to require more land to house cooling and power generation (because power companies aren't going to be able to provide power easily to something like this without building an entire substation next to the datacenter something like this is housed) than the data center housing the computational hardware.
This is going to require a complete restructuring inside the data hall as well... how do you get 600kW of power into a rack in the first place, and how do you extract 600kW of heat out of it? Air cooled is right out the window, obviously, and the chilled water capability of the center is going to be massive (which also takes power). Just what kind of voltages are we going to be seeing going into a rack like this? 600kW coming into a rack at 480V is still 1200+ Amps, which is just nuts. Even if you got to 600V, you are still at 1000A. What kind of services are you going to be bringing into that single rack?
It's just nuts, and I don't even want to think about the build-out timeframes that are going to occur because of systems like this.
r/HPC • u/dreiunddreissig33 • Mar 17 '26
Hey HPC Engineers and Researchers,
I’m trying to understand what working in High Performance Computing actually looks like in real life.
What kind of problems do you usually work on, and what does a typical day look like? Is it mostly writing code, optimizing performance, debugging weird scaling issues, or dealing with clusters and infrastructure?
How important are tools like OpenMP, MPI, C++, and Python in your daily work? What else should I be focusing on — OpenCL, CUDA, OpenACC, SYCL, Fortran, or things like profiling tools (VTune, perf, Valgrind)?
Also curious how much low-level knowledge matters — like memory hierarchies, cache optimization, NUMA, vectorization, networking (InfiniBand), etc. Do you regularly work with schedulers like SLURM or container tools like Singularity/Docker?
For someone who wants to stick with HPC long-term, what skills made the biggest difference for you? And what should I avoid wasting time on?
Would really appreciate hearing your experiences — especially what surprised you about working in HPC vs what you expected going in.