r/openshift • u/Additional-Bowler776 • 2d ago
Fun Shiftmox: OpenShift/OKD 5 on Proxmox VE from one YAML file, with console-driven worker autoscaling
I've been building a homelab project to run **OpenShift 5 (or OKD) on Proxmox VE** completely from code. The part I'm most happy with: **workers that scale themselves**, either from *Compute › MachineSets › Edit Machine count* or the ClusterAutoscaler, with the worker VMs created in Proxmox only when they're actually needed.
* **Repo:** https://github.com/jwklijnsma/shift-mox
* **Docs:** https://jwklijnsma.github.io/shift-mox/
## What it does
* **One file describes the cluster** (`cluster.yml`): counts, sizes, network. VM names, VMIDs, IPs, MACs and Proxmox placement are generated.
* **Agent-based installer**: Ansible renders install-config/agent-config, builds the ISO, uploads it, creates the VMs through the Proxmox API and waits for install-complete. No PXE, no bootstrap VM, no external load balancer.
* SNO, compact or standard; **OpenShift or OKD** with one setting; **Calico** or OVN-Kubernetes.
* `./ocp.sh deploy` runs everything from the Proxmox host itself (in tmux), or use the GitHub Actions workflow with a self-hosted runner in an LXC.
## How the autoscaling works
Proxmox isn't a supported platform, so there's no Machine API provider for it. Instead I reused what OpenShift already has for bare metal:
* **Metal3** (platform `baremetal`) with one `BareMetalHost` per worker slot
* A **Redfish BMC for Proxmox**: the sushy-tools emulator plus a small Proxmox driver, so Ironic can power VMs, set the boot order and mount its ISO as virtual media through the Proxmox API
* **VMs on demand**: a slot has no VM until Ironic first touches it. After scale-down a reaper deletes the VM again, but only after checking the API that the host is really released
* **DHCP with static leases** per MAC, a MachineConfig for `/etc/hosts` + hostname, and a CronJob that re-registers hosts stuck in an error state
* The native **ClusterAutoscaler + MachineAutoscaler** on the worker MachineSet
So: pods go `Pending` → ClusterAutoscaler scales the MachineSet → Metal3 claims a slot → Redfish → Proxmox creates the VM → RHCOS installs → node joins. And back again when it's idle.
## Things I learned the hard way
* **A DNS wildcard on a parent domain** (e.g. `*.example.com` at your DNS provider) makes the agent installer refuse to start (`dns-wildcard-not-configured`), and you only find out after the ~1 h bootstrap timeout. The preflight now checks for it.
* **NIC names on q35 VMs aren't `ens18`.** The agent installer copes because it maps interfaces by MAC; Metal3's `preprovisioningNetworkData` doesn't. The non-matching static config got copied into the installed system, which then skipped DHCP and booted without any network (Ignition: `network is unreachable`, forever). New workers now use DHCP only.
* **Re-applying a BareMetalHost that a worker runs on resets `spec.online` and powers the worker off.** Ask me how I know. :)
* New workers boot the release's older boot image first and update to the cluster OS on first boot, so don't panic when GRUB says RHCOS 9.x on a 10.x cluster.
## Caveats
This is a **lab/test** setup: Proxmox VE is not a supported platform for OpenShift. It runs well for me on a single Proxmox host; multi-host should work but has seen less testing.
Feedback, ideas and "you're doing it wrong" are all welcome, especially from anyone who has run Metal3 with virtual BMCs on other hypervisors.