Root Cause
An extended power outage depleted the UPS battery, causing a hard power loss while ZFS was mid-write. Because both mirror drives receive writes simultaneously, the same blocks were corrupted on both drives, preventing ZFS from self-healing. The corruption affected two Immich video files and the TrueNAS app catalog (which was being cloned at the time of the outage).
Symptoms
Critical alert: "Pool <poolname> state is ONLINE: One or more devices has experienced an error resulting in data corruption"
Critical alert: "Boot pool status is ONLINE: One or more devices has experienced an error resulting in data corruption"
Critical alert: "Failed to start docker for Applications"
Critical alert: "Failed to sync TRUENAS catalog"
Apps page showing catalog sync error
Step 1: Identify the Extent of Corruption
SSH into TrueNAS and run:
sudo zpool status -v
Check both pools for CKSUM errors. The -v flag shows which specific files are corrupted. Note: sudo is required to see the file list.
What to look for:
READ/WRITE/CKSUM error counts per drive
The file list under errors:
Whether the scrub repaired anything (repaired 0B with errors = corruption on both mirror drives, unrecoverable by ZFS)
Step 2: Check Drive Health
Identify the physical drives:
ls -la /dev/disk/by-partuuid/ | grep -E "<partuuid1>|<partuuid2>"
Run SMART diagnostics on each drive:
sudo smartctl -a /dev/sda
sudo smartctl -a /dev/sdb
Key attributes to check (all should be 0):
Reallocated_Sector_Ct (ID 5)
Current_Pending_Sector (ID 197)
Offline_Uncorrectable (ID 198)
Reported_Uncorrect (ID 187)
UDMA_CRC_Error_Count (ID 199)
If drives pass SMART and Power-Off_Retract_Count significantly exceeds Power_Cycle_Count, power loss is the likely culprit.
Step 3: Identify Corrupted Files in Immich
Start Docker if it isn't running:
sudo systemctl start docker
Find the Immich postgres container:
sudo docker ps | grep -i immich
Query the database using the file paths from zpool status -v:
sudo docker exec -it <immich-postgres-container> psql -U immich -d immich -c \
"SELECT id, \"originalFileName\", \"originalPath\", \"fileCreatedAt\", \"ownerId\" FROM asset WHERE \"originalPath\" LIKE '%<corrupted-filename>%';"
Step 4: Delete Corrupted Assets
Verify you still have the original files on your source device (phone, camera, etc.)
In the Immich web UI, navigate to the date of the corrupted files
Select and delete the corrupted assets
Go to Trash → Empty Trash to trigger physical file deletion
Step 5: Clear Pool Error Counters
sudo zpool clear <poolname>
sudo zpool clear boot-pool
Step 6: Check for ZFS Snapshots Holding Corrupted Blocks
If a scrub still shows errors after deleting the files, snapshots may be holding references to the corrupted blocks:
sudo zfs list -t snapshot -r <poolname>
sudo zfs list -t snapshot -r <poolname>/ix-apps/app_mounts/immich/data -o name,creation
Delete snapshots created after the corrupted files were written:
sudo zfs destroy <poolname>/ix-apps/app_mounts/immich/data@<snapshot-name>
Step 7: Fix the TrueNAS Catalog
The catalog directory can be corrupted mid-clone if power is lost during a sync. Check if it's a ZFS dataset:
sudo zfs list | grep catalog
Destroy it — this is safe, it is just a local cache of the public TrueNAS app catalog and contains no personal data:
sudo zfs destroy <poolname>/ix-apps/truenas_catalog
Trigger a fresh sync:
sudo midclt call catalog.sync
Check the job result (replace <job-id> with the number returned by the previous command):
sudo midclt call core.get_jobs '[["id", "=", <job-id>]]' | python3 -c \
"import sys,json; jobs=json.load(sys.stdin); print(jobs[0]['state'], jobs[0].get('error',''))"
Should return SUCCESS None.
Step 8: Final Verification Scrub
sudo zpool clear <poolname>
sudo zpool scrub <poolname>
Wait ~50 minutes, then verify:
sudo zpool status -v <poolname>
Success looks like:
scan: scrub repaired 0B in 00:51:41 with 0 errors
errors: No known data errors
Clear the accumulated CKSUM counters:
sudo zpool clear <poolname>
Step 9: Re-upload Affected Files
Re-upload the original files from your source device to Immich.
Prevention: Configure UPS Auto-Shutdown
In TrueNAS UI: System → Services → UPS
Set Shutdown Timer to 30–45 minutes (well within your UPS runtime)
This ensures a clean ZFS shutdown before the battery dies, regardless of outage length
Even with a 1-hour UPS, a 10+ hour outage will eventually kill the battery. Auto-shutdown guarantees the server powers off gracefully at the 30–45 minute mark, protecting your data.