r/linuxadmin
•
u/CautiousCat3294
•
7d ago
•
1
Linux server has high load average but low CPU usage. What would you investigate first?
I'd like to discuss a Linux troubleshooting scenario and get some input from experienced Linux administrators.
Imagine you're troubleshooting a production Linux server with the following symptoms:
Application response time is increasing.
Load average is continuously rising.
CPU utilization is only around 20–25%.
Several application processes are stuck in D state.
The application uses an NFS-mounted directory.
At first glance, CPU doesn't seem to be the bottleneck.
If you were handling this incident, how would you approach the investigation?
A few things I'd be interested in discussing:
Would you start with
vmstatto check runnable and blocked processes?How would you identify what the D-state processes are waiting for?
What commands would you use to investigate NFS connectivity and mount health?
How would you distinguish an NFS issue from local disk I/O problems?
I'm particularly interested in real production experiences.
Have you encountered a similar situation? What was the root cause, and what helped you identify it?
Looking forward to learning from your experiences.
10
u/ipsirc 7d ago
2
1
1
u/Anne_0Nyme87 7d ago
Very interested at first; then my brain melt.. ^
I'll keep it for my long nights, but still full of knowledge! ;)
1
0
u/CautiousCat3294 7d ago
checkout this one as well https://datadevblog.com/linux-high-load-average-low-cpu-troubleshooting/
Appreciate your feedback
4
u/serverhorror 7d ago
D state points to waiting for IO and that (usually) increases load.
Load is not an indicator of a CPU bound system.
1
u/CautiousCat3294 7d ago
correct however beginners thinks Load = CPU bound and take wrong decisions while fixing the issue.
4
u/FarToe1 7d ago
Run htop, hit F2 and toggle on "Detailed CPU time"
The cpu graphs will now who different colour depending on what's using the cpu. System/Iowait/etc. System is actual cpu use, iowait will probably be high in your case.
As others have said, this is almost certainly IOwait where the cpu is waiting on storage or network, and the above makes it really easy to highlight that.
Newer versions of htop also have have an I/O tab so you can see exactly what process is causing that. (Hit tab to cycle to it)
Also - check "dmesg -T". Excessive iowait is sometimes a big indicator of drive problems and messages there, or in syslog/messages/journalctl will probably have some error messages relating to your storage.
You've mentioned NFS and that can absolutely cause very high sysload too. One indicator of that is that the system is pretty snappy considering the load.
1
3
u/frymaster 7d ago
The application uses an NFS-mounted directory.
While I'd want to be systematic about it, high load + low CPU + networked filesystem means "networked filesystem bottleneck" almost all the time
1
4
u/afristralian 7d ago
Check your iowait times in top. Best way to describe is it's amount of time CPU processing is waiting on disk or other io subsystems.
Load average is number of parallel processes running over time intervals. If your CPU is waiting for IO from a disk read it will keep the thread open causing load average to +1 even if the process is not consuming CPU time. (Low CPU usage and high load average).
(My statements aren't 100% accurate, but just look for iowait .. if it's above 20% all the time you have an issue with disk, network or other IO subsystem).
2
u/NeverMindToday 7d ago
Thanks for bringing it up - I was surprised to see
iowaitmentioned so little in the replies. Maybe I'm just too old and been away from the tools too long?Anyway, for the OP - understand the difference between load average and cpu usage (specifically by usage in this case I mean the user/system times). Load average is analogous to the depth of processes in the cpu queue vs usage being the percentage of time the cpu isn't idle or waiting. You can have lowish user/system usage with high load average (inefficient and high latency) and vice versa (efficient and low latency).
1
u/CautiousCat3294 7d ago
Thanks
Tried to bind my investigation in below article. Looking for feedback to make this strong
https://datadevblog.com/linux-high-load-average-low-cpu-troubleshooting/
69
u/dosman33 7d ago
Load average is driven by outstanding I/O's waiting to complete, there are many ways this can happen that have nothing to do with cpu utilization. File lock contention churn, multiple threads doing lots of i/o on the same file, etc. Can also happen if the disk subsystem is having issues and not able to respond in a timely manor (scsi bus resets, etc).
13
u/symcbean 7d ago
The original post sounds like homework or an interview question.
Load average is driven by outstanding I/O's waiting to complete,
On a Unix system, load average is the average number of R state processes, but Linux includes processes in a D state.
High load/low cpu might be slow NFS. But can also be caused by requests which trigger a cascade of processes (microservices, message handlers are common culprits). iostat would help with identifying how much IO is dependent on NFS. It can also happen if your box is swapping to disk. 'top' would be the first place to look.
That this is reported as an emergent behaviour, is significant. The big question to ask is what do you know has changed around the same time as the observed change in behaviour.
If there is significant NFS activity in the absence of code, data or infrastructure changes then the most likely reason for the behaviour to change is the network. ethtool and the switch metrics would give more info.
15
u/Open-Adhesiveness-86 7d ago
For the D-state procs, cat /proc/<pid>/stack tells you exactly where they're parked, and if you see rpc_wait_bit_killable it's NFS, not local disk. From there nfsstat -rc for retrans count and nfsiostat for per-op RTT, then iostat -x separately on the local disks. iotop is misleading here since NFS waits never show as block IO.
1
u/deleriux0 7d ago
This is the best approach IMO.
Often the stack will even output the operation in it's chain (write/read/rename).
May also be worth doing an
straceon the affected process to soo if it's lots of frequent accesses repeating, failing and retrying.
17
1
u/Morbothegreat 7d ago
Running the command:
echo w > /proc/sysrq-trigger
Will dump the stack trace of processes stuck on io. That may give you some hints. But it’s stuck on read/writes to nfs. Could be a bug. Could be busy.
1
3
u/Ambitious-Wrangler93 7d ago
I'd look for "df -h" output hanging. I've diagnosed hundreds, maybe a thousand hung NFS mounts by running that command first. High CPU of a backup process is another bad filesystem mount indicator, usually flagged by monitoring alarms.
3
6
u/RHCidiiot 7d ago
Unmount the nfs share and see if the load goes down, iowait can be checked too.
5
3
u/CardOk755 7d ago
Of course the load goes down. You've just broken your app. By the way, the phone is ringing.
1
u/Memitim 7d ago
Then they determined that they need to operate with a known-good default state to stop hamstringing it with storage fuckups without being able to test for them. If the phone is ringing just because an app node goes down, looks like they also identified a massive gap in CI/CD. It's 2026 kids. Build your shit right, already. There are a ton of references and the robots are really good at helping with this sort of thing.
2
2
2
u/HoustonBOFH 7d ago
No mention of memory here. Is there a lot of swap going on? That load would cascade all over the place. Especially swap on NFS.
10
u/northrupthebandgeek 7d ago
If I ever encounter a production server using NFS for swap then I'm resigning on the spot and fucking off into the wilderness as far away from electricity as possible.
3
2
u/fatmanwithabeard 7d ago
I've done this. In my defense I was running diskless, stateless servers that had to support code that checked how free swap was.
It was a hack, in that I didn't need a change control meeting to add an nfs share, but did to put a swapfile on a lun from any of the arrays.
In any case it did what was needed, and was left as a space for that lab to run that code. Then over a long weekend, the server rebooted, and a lab user talked the admin on duty into making the nfs share part of the general spec. That ended up being a fun outage.
1
1
1
u/skidleydee 7d ago
Not a linux admin but infra admin here. If its a vm you need to check your hypervisor stats for the vm as well. When a cpu is virtualized it doesn't have a way to report on they physical cpu stats it can only report on the vcpu that it had access to. You can't trust a VM's internal stats 100%, this goes for all virtualized hardware as well not just the cpu.
If a VM is not able to schedule your vCpu on the pCpu the vm will still only show the utalization of the VM it doesn't have a way to account for latency that it doesn't know exist.
On VMware if you delete a large amount of data and are using the default vsan storage policy, throttle threshold and threshold percent it will try to reclaim the space on the physical disk. This causes a swarm on an index file and the VM will hang until the swarm is over. The VM stats show the vCpu will spike to 100% utalization, in vmware you would see that the CPU utilization never moved during the hang. This is because the vCpu can't schedule anything because it's waiting on the storage but the VM doesn't know that it thinks it's waiting on cpu.
There are a ton of examples similar to this that would show low cpu utalization but could still end up being a cpu bottle neck in the hypervisor. I think it's probably not your issue but fyi.
1
u/CyberKiller40 7d ago
I/O operations. Either the storage isn't fast enough, or is close to failing.
1
1
u/northrupthebandgeek 7d ago
It's always I/O. Slow network or slow disk. Measure both, then start cutting load on the more-heavily-impacted one.
1
u/michaelpaoli 7d ago
high load average but low CPU usage. What would you investigate first?
I/O
High load indicates processes waiting to run but can't because their waiting on ... well, if's not CPU, that generally leaves I/O, e.g. storage or network I/O.
how would you approach the investigation?
Depends what commands one has available. One can also inspect /proc and/or /sys filesystems, and particularly in quite minimal environments where many more common commands may not be available.
1
u/Dolapevich 7d ago
D uninterruptible sleep (usually IO)
From ps manual, state D means the pid is waiting for IO.
This is a classic situation where NFS is either having too much IO or it loosing packets, or the server is unable to lock exclusive the file or any of the gazillion things that can happen to NFS.
I've seen this particular scenario on a congested network; during the peak hours we would get alerts for high load average, and upon checking nginx was stuck in IO doing mostly nothing.
At the client side, using a combination of ls /path/to/nfs would stuck for minutes, using lsof on nginx pid it would show it was trying to write to an nfs path.
Doing showmount or rpcinfo -p <server> would also hang.
Turned out to be a particular router dropping udp packets when overloaded. We found it with a combination of traceroute ( mtr ), and ping -R
1
u/MinimallyLoquacious 7d ago
Did the scenario happen to you, are you really just learning or are you working for an AI company and looking to get free unpaid training input (for which you will get paid)?
1
1
u/fubes2000 7d ago
"NFS" and "D state" triggered a 20 year old neuron.
Your NFS mount was and/or still is unstable. Those D state processes are probably waiting for IO that will never come. If your kernel allows you to kill the D procs you might get away with that and remounting the NFS share, otherwise that's a reboot to fix.
Also make damn sure that your connection to NFS is 1000% solid at all times.
2
u/biffbobfred 7d ago
We had Solaris machines with an unstable driver for some serial thing and we had a lot of D states. Good times.
2
u/fubes2000 7d ago
Yeah this gave me a war flashback to virtualized HPUX at gov and the NFS server sneezed and every VM needed its NFS shares remounted and those people were utterly allergic to automation and it was agonizing watching them literally type out every command 200 times.
1
u/biffbobfred 7d ago
Heh
We had machines with cross mounts. There was a catch 22 on boot. They got them into auto mounter after that
1
u/cereal7802 7d ago
First thought, disk access. then i read your post...
The application uses an NFS-mounted directory.
Yeah, you are probably having issues with NFS. Not enough bandwidth. too much latency. default nfs settings. losts of potential there. Take a look at if NFS is needed in this setup, and if it is, start your tuning journey there.
1
u/techforallseasons 4d ago
High load, low CPU --- WAITs (i/o typically - SWAPPING otherwise) / lock contention
1
u/markhahn 4d ago
I don't get why there are all these recommendations for generic system-status tools like sar or htop.
Pick a D process, and fully grok it. Should it be hanging? (What does it do?). Being D we know it's some kind of IO. Look around in it's /proc entry: exe, cwd, fd entries. Look at wchan. The info is all right there.
1
u/MikeZ-FSU 3d ago
I like to start with "dstat". It gives a scrolling, not re-drawing like *top, view of overall system stats (so you can see change over time) without all of the clutter of the per-process details. That lets me look at things like user/sys/idle/wait distribution to see if it's a runaway process hogging cpu vs. other things like a hung syscall. It also shows disk and net read and write along side paging and interrupts.
The overview lets me narrow down the kind of problem to a smaller number of processes that are likely to generate that signature. This works for me because it fits my mental process for troubleshooting; others may find a different approach makes more sense to them. Neither is necessarily better or worse, and people should pick the tools that work for them and their particular setup.
1
0
u/hurikhan1977 6d ago
Check memory fragmentation. I had an issue on a server where load was incredibly high, the system felt slow, but it was thrashing page cache and swap even with memory still free. It came down to THP fragmentation in the end. Memory allocation waits count to loadavg, too, not only IO, but due to page cache thrashing, the effects look quite similar: processes stuck in D state with very low IO bandwidth. You can also check PSI to see what is causing the high load.
0
65
u/JasenkoC 7d ago
Storage is the most likely culprit in 85% of the cases. Tools like top, htop, iotop can give you very useful clues.
As for NFS, I'd check if it's mounted and reporting the file system in df, and then I'd check with iotop for any signs of issues with NFS performance. Iotop can tell you if the I/O wait is high in local storage volumes or not (in which case it's most likely the network attached volume that has an issue).
Also, a quick dd write test on a local volume and NFS volume can show you where to look for your bottleneck.