The system load for Linux is an indication of how busy the system is, and how long you are going to wait to get something done. When the load is 0.0, that means that the system is ready to do whatever you want immediately. If the load is 0.50, then half of the time, the system is busy with something else. If the load is 1.00, the system is doing as much as it can. If the load is higher than 1, then that’s an indication of the backlog of the amount of tasks the system has to do. When the system load is 3.14, that means that there is a backlog of things being done, such that there are always π things before your thing. When the load hits 10, you have trouble. When the load hits 59, you have serious trouble. When the load hits 272, something is horribly broken, and the system is not going to recover by itself.
The thing that loads a system is competition for some scarce resource, e.g:
- CPU time: if you have one CPU in your system, it can only do so much. If you have 16 CPU’s they can only do 16 times as much as that. There’s a limit.
- Disk time: the electronic highway that relays data to and from your disks is only so wide, and you can only go so fast on it. If the highway is full, then queries must wait in line. If you need data from the disk, there’s nothing to do but wait.
- Memory: a computer has only so much memory to do its tasks. If it runs out of memory, then it has to either stop doing some things, or it has to use slower memory (disk access).
If you care whether your systems are working or not, you monitor the load, and if something goes beep, you have a look. You start like this:
root@dogmatix [~]# uptime 07:55:05 up 23 days, 15:36, 1 user, load average: 1.05, 2.57, 2.85
That says that for the last 10 minutes, the load has been 2.85, and for the last 1 minute it’s been 1.05. This is not a serious situation. I found another one:
root@kiarapoint [~]# uptime 16:57:23 up 22 days, 21:42, 1 user, load average: 66.27, 56.02, 36.21
Okay, that’s pretty bad. 66.27 times as many things to do as we can actually do in a minute. Not cool. Next step is to run vmstat:
root@kiarapoint [~]# vmstat 3 10 procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------ r b swpd free buff cache si so bi bo in cs us sy id wa st 2 10 3652 104448 1770072 9313432 0 0 8 63 5 4 8 1 84 8 0 0 4 3652 115168 1770076 9313420 0 0 0 35 1071 476 11 1 18 70 0 1 5 3652 137784 1770076 9313428 0 0 0 63 1086 670 12 1 36 51 0 0 5 3652 154716 1770076 9313428 0 0 0 40 1057 402 8 1 31 60 0 0 5 3652 346420 1770076 9313320 0 0 0 0 1075 529 10 2 22 66 0 0 6 3652 346880 1770076 9313148 0 0 0 28 1047 259 1 0 11 88 0 0 6 3652 346624 1770076 9313148 0 0 1 0 1036 315 5 1 0 94 0 0 6 3652 346168 1770076 9313180 0 0 8 240 1069 173 0 0 0 99 0 0 6 3652 343216 1770076 9313180 0 0 0 23 1079 304 1 1 0 99 0 0 6 3652 343832 1770076 9313180 0 0 0 23 1022 86 0 0 0 100 0
If you examine that carefully, you’ll notice that it makes no sense at all. This is in part because the columns don’t line up. This should be better:
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------ r b swpd free buff cache si so bi bo in cs us sy id wa st 2 10 3652 104448 1770072 0 0 8 63 5 4 8 1 84 8 0 0 4 3652 115168 1770076 0 0 0 35 1071 476 11 1 18 70 0 1 5 3652 137784 1770076 0 0 0 63 1086 670 12 1 36 51 0 0 5 3652 154716 1770076 0 0 0 40 1057 402 8 1 31 60 0 0 5 3652 346420 1770076 0 0 0 0 1075 529 10 2 22 66 0 0 6 3652 346880 1770076 0 0 0 28 1047 259 1 0 11 88 0 0 6 3652 346624 1770076 0 0 1 0 1036 315 5 1 0 94 0 0 6 3652 346168 1770076 0 0 8 240 1069 173 0 0 0 99 0 0 6 3652 343216 1770076 0 0 0 23 1079 304 1 1 0 99 0 0 6 3652 343832 1770076 0 0 0 23 1022 86 0 0 0 100 0
That’s better. The columns of interest here are:
- b — blocked processes (not running) — there are around 6 processes waiting for the disk
- bi and bo — buffers in and buffers out — this is the actual disk throughput in something like kilobytes per second
- wa — waiting percentage — well, we’re mostly waiting because we’re waiting for disk.
So this machine is running slow because its disk can only do so much. It’s busy writing out around 1000 blocks per second, which is roughly 8Mbps towards the disk. For this system, that does not indicate a disk performance problem, so we’re just pushing too hard.
Now, we do this — get a list of the processes, and see which ones are in “Disk-wait” state, identified as D preceeded by a space — hence the grep ' D' in the command below:
root@kiarapoint [~]# ps uaxwf | grep ' D' root 357 0.0 0.0 0 0 ? D Nov19 4:20 \_ [pdflush] root 573 0.0 0.0 0 0 ? D< Nov19 13:45 \_ [kjournald] root 32748 0.0 0.0 61248 756 pts/0 S+ 16:58 0:00 \_ grep --color=auto D nobody 24458 0.0 0.0 88536 14772 ? S Dec06 0:00 \_ /usr/local/apache/bin/httpd -k start -DSSL mailnull 32444 0.0 0.0 67568 2696 ? DN 16:55 0:00 | | \_ /usr/sbin/sendmail -t -i nobody 32514 0.0 0.0 88944 14756 ? D 16:56 0:00 \_ /usr/local/apache/bin/httpd -k start -DSSL mailnull 31948 0.0 0.0 124048 4800 ? D 16:50 0:00 | \_ /usr/sbin/sendmail -FCronDaemon -i -odi -oem -oi -t root 7620 0.0 0.0 44912 1004 ? Ss Dec08 0:00 \_ (ncsa_auth) /etc/squid/passwd root 7624 0.0 0.0 44912 1004 ? Ss Dec08 0:00 \_ (ncsa_auth) /etc/squid/passwd root 7625 0.0 0.0 44912 1000 ? Ss Dec08 0:00 \_ (ncsa_auth) /etc/squid/passwd root 7626 0.0 0.0 44912 1004 ? Ss Dec08 0:00 \_ (ncsa_auth) /etc/squid/passwd root 7627 0.0 0.0 44912 1000 ? Ss Dec08 0:00 \_ (ncsa_auth) /etc/squid/passwd
HOWEVER it is December. December, the only month in the year to have the bad taste to start with D. We have processes that have been running for a few days, which started in December. So, for the entire month, we cannot do this:
ps uaxwf | grep ' D'
but have to do this instead:
ps uaxwf | grep ' D[^e]'
Aargh! Bracket, shift, 6, release shift, e, bracket … Aaaaarrrgh! The injury!
