Via The Age

Entry is free and there’s a cash prize. Nothing to loose!

Get yourself a bargain!

So when things go wrong, before reverting the last change, before breaking out gdb and strace and before tweaking your software on your production host, spend 5 minutes and run through these quick, simple tests – there’s a high likelihood that you’ll solve your problem quickly. (I’m sure there’s other tests you can do – these are the ones burnt into my mind)

#1 – Disk space

Don’t laugh – running out of disk space can cause you pain in so many ways it’ll spin your head.   Check all partitions, including /tmp, /var/tmp and /var. Running out of tmp means applications won’t be able to write temporary files which, depending on the app, may make it behave very strangely. /var is used for many things including logging in /var/log – not being able to log will make some software cry like a baby – i.e. it may crash and you’ll have no idea why – it certainly won’t be in the log file.  Databases like MySQL don’t like having no room to write in /var/lib/mysql – Don’t be surprised if you get some db corruption. With MySQL, you may be able to start the database and even connect to it with the mysql client, leading you to look elsewhere – but checking disk space will take you seconds.

[sourcecode language='shell']
dkam@vihko:~$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1             9.4G  7.1G  1.8G  80% /
udev                   10M  116K  9.9M   2% /dev
shm                   128M     0  128M   0% /dev/shm
[/sourcecode]

Don’t forget to check iNodes too – running out of inodes can cause the same issues as diskspace but is less obvious – checking for it is just as easy though:

[sourcecode language='shell']
dkam@vihko:~$ df -ih
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda1 1.2M 445K 772K 37% /
udev 32K 1.1K 31K 4% /dev
shm 32K 1 32K 1% /dev/shm
[/sourcecode]

#2 - DNS resolution

DNS resolution problems can cause your system and application to hang or timeout in very strange ways.

Some applications will log the name of inbound network connections, performing reverse lookups. If no NS is available, these connections may start to take a long time to connect, as the software waits for the resolver to timeout. If only the first listed NS has failed, this timeout may be variable in length, but probably around 15 seconds. If you see weird lags or delays, check your name servers. This can happen when you’re trying to ssh into the host – if you’re getting delays connecting via ssh, check DNS.  If your software makes connections to external databases for example, and is configured to address them by name, you’ll see these timeouts.

This one can be tricky because some software will cache the name resolution and some local resolvers may cache – meaning you’ll see delays or timeouts sometimes, but not consistently.

Name lookups should be under a second, preferably in the low 100′s of milliseconds.
[sourcecode language='shell']
dkam@vihko:~$ time host www.apple.com
www.apple.com is an alias for www.apple.com.akadns.net.
www.apple.com.akadns.net has address 17.251.200.32

real 0m0.132s
user 0m0.000s
sys 0m0.000s
dkam@vihko:~$ time host www.apple.com
www.apple.com is an alias for www.apple.com.akadns.net.
www.apple.com.akadns.net has address 17.251.200.32

real 0m0.011s
user 0m0.000s
sys 0m0.000s
[/sourcecode]
You can see that in the second run, the name server had cached the value and returned much faster.

It also pays to check each nameserver listed in /etc/reslov.conf:
[sourcecode language='shell']
dig www.google.com @208.78.97.155
[/sourcecode]
Naturally replace 208.78.97.155 with your name server’s IP.

Check “man resolv.conf” for more information.

#3 – Ulimits

The most common ulimit’s that I’ve come across is max number of open files, but you may see others including max user processes.  This one is generally obvious if the software is running as a regular user – when you try to connect as that user you will see error messages about being unable to allocate resources. Editing a file or trying to read a man page will error out if you’re at the maximum number of open files.  Network connections fall into this category also – so you may not be able to open network connections either.

The more likely scenario is that the software is running as a different user – one that people don’t log in as.  Try logging in as, or su -’ing to the user – if you can’t or you can but the user can’t open files, check the ulimits.  In Bash, try “ulimit -a” to view your limits. Different OSs limit these values in different ways – check your OS doco for details.

#4 - /dev/random

This is a little esoteric and is pretty unlikely, but /dev/random is used for lots of reasons – the most common use that you may have problems with is login with software that uses stuff like CRAM-MD5. Random data is used as part of the authentication process and when there’s not enough random data, logging in will be slow or may timeout completely.  Most software should probably fall back to using /dev/urandom.  You can time how long it takes to read 1kb of random data like this:

[sourcecode language='shell']
dkam@vihko:~$ dd if=/dev/urandom of=/dev/null bs=1K count=10
10+0 records in
10+0 records out
10240 bytes (10 kB) copied, 0.002011 s, 5.1 MB/s
[/sourcecode]

#5 - permissions

Generally this will only bite you when you’ve made changes or updated software – check that config files are readable, data directories are read/writable and that executables are executable.