Dustin Boswell - Brain Dumps about Computers, Programming, and Everything Else
dustwell.com About Me Past Academic Work
Blog Posts How to hash passwords Snapshotting with Rsync "An hour" vs. "A hour" Yes/No proposition bets div, span, and CSS "display:" Pair programming + screen SSH keys in 2 easy steps Fixing Your Flaky Internet How X-over-SSH works Drinking Distilled Water djb-dns installation Vim Cheat Sheet Oil ETFs leak money
Most useful tools for diagnosing UNIX system performance. March 24, 2013


CommandExampleInstallWhat it does
toptop(built-in)Interactive overview of machine
vmstatvmstat 2(built-in)Overview of memory, swap, cpu, disk
iostatiostat -dmx 2sudo apt-get install iostatDisk utilization, throughput
iftopiftopsudo apt-get install iftopInteractive overview of network traffic
tcpflowsudo tcpflow -i any -C -e port 80sudo apt-get install tcpflowSniff live network traffic
lsofsudo lsof -i TCP(built-in)Show processes with open files/sockets

top explained

Ignore the "load". It just confuses people, and a 'bad' value depends on how many cores you have. Instead, just look at the CPU usage numbers ("user", "system", "nice", "idle", "iowait"), which are percentages averaged over all cpus. As long as the "idle" value is above 0%, your system probably isn't overloaded (at least, not in a way that having more CPU would help). The "iowait" value is confusing, so I would ignore it (if it's high, that means that a faster disk would improve throughput of your system, but if it's near-0, then either you don't have much io waiting, or that io wait time was used by some other cpu-busy process).

I usually hit 'M' to sort by memory usage -- the biggest processes are usually the most interesting ones. The "virtual" memory size is misleading -- this is how much memory the process would use if it touched all the memory it was given. The "resident (RES)" memory usage is the one that matters. The "shared" column is mostly useless, so ignore it (for instance, it doesn't take into account the amount "shared" between forked processes that haven't copy-on-write yet.)

vmstat explained

The runnable processes ("--procs-- > r") gives you a sense of how many processes are using (or want) CPU at the moemnt. The idle percent ("--cpu-- > id") lets you know how often the CPU has nothing to do. If this is 0, then your system is cpu limited at the moment. The swap amount ("--swap-- > si so") shows the "swapped in" and "swapped out". If these are constantly above 0, then your system is swapping to disk a lot, which is probably bad. The memory amount ("--memory-- > free buf cache") shows how much memory is free, being used for buffers, or for file cache. If these numbers are low (less than 1000(KB) for each), then your system probably doesn't have enough memory for what it wants.

iostat explained

Ignore the first line of output (those are summary stats since bootup, which is rarely useful). Focus on the last column (utilization). If it's near 100%, then your system has a disk bottleneck at the moment.
blog comments powered by Disqus