We had a major outage today in our Proxmox clusters caused by Commvault.
Documentation from Commvault on this: https://kb.commvault.com/article/?id=KA-191658
My org was hit by this. From the Proxmox team, this was the cause and fix:
On each host, we saw a ton of stale NDB devices:
nbd4 43:128 0 30G 1 disk
├─nbd4p1 43:129 0 1G 1 part
├─nbd4p2 43:130 0 2G 1 part
└─nbd4p3 43:131 0 26.9G 1 part
└─ubuntu--vg-ubuntu--lv 252:4 0 26.9G 0 lvm
nbd5 43:160 0 0B 1 disk
nbd6 43:192 0 0B 1 disk
nbd7 43:224 0 0B 1 disk
nbd8 43:256 0 0B 1 disk
nbd9 43:288 0 0B 1 disk
nbd10 43:320 0 0B 1 disk
nbd11 43:352 0 0B 1 disk
nbd12 43:384 0 0B 1 disk
nbd13 43:416 0 0B 1 disk
nbd14 43:448 0 0B 1 disk
nbd15 43:480 0 0B 1 disk
nbd16 43:512 0 0B 1 disk
nbd17 43:544 0 0B 1 disk
nbd18 43:576 0 0B 1 disk
nbd19 43:608 0 0B 1 disk
nbd20 43:640 0 0B 1 disk
nbd21 43:672 0 80G 1 disk
├─nbd21p1 43:673 0 1007K 1 part
├─nbd21p2 43:674 0 512M 1 part
└─nbd21p3 43:675 0 79.5G 1 part
├─pdm-swap 252:5 0 4G 0 lvm
└─pdm-root 252:6 0 65.6G 0 lvmnbd4 43:128 0 30G 1
After this happens, each VM impacted by this stops booting because its disks disappear.
The cause is as follows:
From what I discovered, this issue was caused by a race condition between proxmox and commvault for access to the /dev/nbd devices. Â The solution I was able to find requires that we edit /etc/lvm/lvm.conf on each prox host and add the following exclusion so proxmox does not try to take control of the NBD devices during backups.
global_filter=["r|/dev/zd.*|","r|/dev/rbd.*|","r|/dev/nbd.*|"]
I may need to reinstall and configure the Commvault agent on the POC prox nodes to test this change. Â I'd also like to confirm this with Proxmox Support if we are able to purchase a contract.
The immediate fix to clean up any stale NBD devices is to evacuate and reboot each proxmox node one at a time.
The fix we used was the following:
Locate the VM and save evidence before stopping it
ID=189
PID=$(cat /var/run/qemu-server/$ID.pid)
mkdir -p /root/incident/$ID
ls -l /proc/$PID/fd | grep -E 'cv_staging|\.qcow2|\.raw' > /root/incident/$ID/open_files.txt
journalctl -t QEMU --since "3 days ago" | grep "QEMU\[$PID\]" > /root/incident/$ID/qemu_errors.txt
qm config $ID > /root/incident/$ID/config.txt
grep -c cv_staging /root/incident/$ID/open_files.txt   # staging files held by QEMU
grep cv_staging /root/incident/$ID/config.txt      # should be empty
Optionally, open qm monitor $ID and run info block -v. Copy any cv_staging or copy-before-write nodes into the evidence folder.
2. Take it out of HA (if HA-managed)
ha-manager status | grep "vm:$ID" && ha-manager set vm:$ID --state ignoredha-manager status | grep "vm:$ID" && ha-manager set vm:$ID --state ignored
3. Hard stop
qm stop $ID --skiplock --timeout 30
qm status $IDqm stop $ID --skiplock --timeout 30
qm status $ID
If it’s still running: kill -9 $PID, wait 5 seconds, confirm with ps -p $PID, then qm stop $ID — skiplock. If the process is stuck in D state, escalate.
4. Check the config and disks
check reports corruption, copy the files before doing anything else.
grep cv_staging /etc/pve/qemu-server/$ID.conf # must be empty
for f in $(qm config $ID | grep -E '^(scsi|virtio|sata|ide|efidisk)[0-9]+:' | grep -v cdrom \
| sed -E 's/^[^:]+: ([^,]+).*/\1/' | xargs -n1 pvesm path 2>/dev/null); do
echo "== $f"; qemu-img check "$f"
done
If the config references cv_staging, stop and contact Commvault; don't start the VM. If qemu-img
5. Start, then confirm the new process is clean
qm start $ID
NEWPID=$(cat /var/run/qemu-server/$ID.pid)
ls -l /proc/$NEWPID/fd | grep -c cv_staging # must be 0
Watch the console for fsck or an emergency shell.qm start $ID
6. Verify
Guest: root not read-only, no I/O errors in dmesg, systemctl — failed clean, the service works.
Host, after about 5 minutes: disk mtimes are recent, and journalctl -t QEMU — since -10min | grep “QEMU
$NEWPID
“ shows nothing.
7. Return it to HA
ha-manager set vm:$ID --state startedha-manager set vm:$ID --state started