Learning RAID

What’s this all about?

Learning about RAID in a nice safe manner and getting practice using the Linux software RAID tools. The following tutorial is for those people who’d like to get some practice building RAID devices, but don’t have spare drives or partitions laying about. Making use of the Linux loop device enables us to create an arbitrary number of block devices from which we can build a RAID device from. If you’ve ever mounted an ISO image of a cdrom with the ‘mount -o loop’ command, you’ve used the loop device. I’ll run through what’s needed in the kernel and the software needed to manage the RAID devices. The goal is to become familiar with most of the concepts and tools for building and using RAID devices. Note: This will not show you how to boot from your array.

What is RAID?

RAID – Redundant Array of Inexpensive Disks. Self explanatory? The key idea is to store your data on multiple disks, in a Redundant manner. There’s plenty of good background material on the web.

What will I need?

You’ll need space on your disks to create the files which will be mounted as file systems. If you want to build a 1GB RAID0 device, you’ll need just over 1GB of free disk space, just over 2GB for a 1GB RAID 1 device. You’ll also need your kernel sources installed. Obviously, you’ll need to be the root user.

Setting up your kernel and software

The quick and dirty way is to edit /usr/src/linux/.config. You’ll need the Loop device built in, or as a module. I’ve mine built in:

CONFIG_BLK_DEV_LOOP=y

Multi-device support is also required. I’ve chosen only RAID0, 1 and 5.
#
# Multi-device support (RAID and LVM)
# CONFIG_MD=y CONFIG_BLK_DEV_MD=m
# CONFIG_MD_LINEAR is not set
CONFIG_MD_RAID0=m
CONFIG_MD_RAID1=m
CONFIG_MD_RAID5=m
# CONFIG_MD_MULTIPATH is not set
# CONFIG_BLK_DEV_DM is not set

Once this is setup, build, install and boot into your new kernel. I’ll wait here for you. Don’t forget to modprobe the Loop & RAID stuff if you build it as modules. Eg ‘modprobe raid5′.

Installing Software

To build and manage the RAID devices, I’ve used mdadm which is available as source if your distribution doesn’t include mdadm. If you’re running a Gentoo system, you can install this application with an ‘emerge mdadm’. Next, we’ll create some files which will be used to hold the block device data. We’ll start with three, which we’ll use to create a RAID 5 device.

Creating some test block devices

Make a directory somewhere, and cd into it.

# mkdir raidEx
# cd raidEx

Next, we create two files of 1GB each. Feel free to change this to whatever size you like.

# dd if=/dev/zero of=file1 bs=1M count=1000
# dd if=/dev/zero of=file2 bs=1M count=1000
# dd if=/dev/zero of=file3 bs=1M count=1000

Then we setup the loop devices.

# losetup /dev/loop1 file1
# losetup /dev/loop2 file2
# losetup /dev/loop3 file3


Now we’ve got our block devices. In the scheme of things, you can treat /dev/loop1 the same as any other block device such as /dev/hda1.

Build a RAID device

Using mdadm, we create the md device.

# mdadm --create /dev/md0 --level=5 --raid-devices=3 /dev/loop1 /dev/loop2 /dev/loop3

You can follow the status of the md devices in your system cat’ing /proc/mdstat as so.

# watch -n .1 cat /proc/mdstat
Every 0.1s: cat /proc/mdstat
Mon Jul 2 20:54:22 2007
Personalities : [raid0] [raid1] [raid5] md0 : active raid5 loop3[3] loop2[1] loop1[0]
2047872 blocks level 5, 64k chunk, algorithm 2 [3/2] [UU_]
[=>...................] recovery = 5.7% (58752/1023936)
finish=8.7min speed=1836K/sec

We can see the device is synchronising itself, calculating checksums for the parity blocks, complete with statistics about the operation. Once that the array is built, you can add it to the mdadm.conf file, so that it’s easy to start:

mdadm --detail --scan >> /etc/mdadm.conf

Then format the device with the filesystem of your choice, Reiserfs ( MurderFS as some people call it) in this case, and mount it:

# mkreiserfs /dev/md0
# mkdir mnt0 # mount /dev/md0 mnt0

Try failing a device ( once it’s finished syncing of course – or not depending on what you want to learn ), removing it, then re-adding it.

# mdadm /dev/md0 -f /dev/loop2
# mdadm /dev/md0 -r /dev/loop2
# mdadm /dev/md0 -a /dev/loop2

Check out /proc/mdstat to see how it’s all going.

Growing a RAID 5 Array

The latest version of the Linux kernel allows you to grow RAID sets in various manners. This allows you some very useful flexibility. With RAID 5, you can add extra disks to the array as hot spares, then *grow* the array to include these new disks. Amongst other things, you can:

1. Add disks to a RAID 5
2. Add larger disks to a RAID 1 Array, growing the size of the device.

# mdadm /dev/md0 -a /dev/loop4 mdadm: added /dev/loop4
# mdadm --grow /dev/md0 --raid-devices=4 mdadm: Need to backup 384K of critical section.. mdadm: ... critical section passedd

Now check out /proc/mdstat to see how it’s going. When you’re growing a RAID 5 array, there’s a period where a crash would cause it to loose data. mdadm will backup the data at risk before proceeding.

Growing a RAID 1 Array

To grow a RAID 1 array, the steps are as follows:

1. Add two new larger devices to the array as a spares.
2. Fail one of the existing disks in the array
3. Once the array has repaired itself, fail the next device then remove both failed devices.
4. You now have a RAID 1 array of the original size, but with larger underlying block devices.
5. Use –grow and –size=max to expand the array to the full size of the underlying block devices.
6. Resize your filesystem to match the new larger block device.

Here’s an example: /dev/loop1 & /dev/loop2 are both 1G in size, /dev/loop3 & /dev/loop4 are 1.5GB.

~ # mdadm /dev/md0 -a /dev/loop3 mdadm: added /dev/loop3
~ # mdadm /dev/md0 -a /dev/loop4 mdadm: added /dev/loop4
~ # mdadm /dev/md0 -f /dev/loop1 mdadm: set /dev/loop1 faulty in /dev/md0
~ # mdadm /dev/md0- -f /dev/loop2 mdadm: set /dev/loop2 faulty in /dev/md0
~ # mdadm /dev/md0 -r /dev/loop1 mdadm: hot removed /dev/loop1
~ # mdadm /dev/md0 -r /dev/loop2 mdadm: hot removed /dev/loop2
~ # mdadm --grow /dev/md1 --size=max


Now the underlying RAID device has grown, you need to resize your filesystem to match. This is can be useful in many situations. I’m sure you’re thinking of them now.

Conclusion

Now that the concepts are familiar, it shouldn’t be difficult to start doing some more advanced stuff. The man page for mdadm is very good, but here are some useful commands:

• To query the device in detail: “mdadm -QD /dev/md0″
• To stop the array: “mdadm -S /dev/md0″
•  To fail a device in the array: “mdadm /dev/md0 -f /dev/loop1″
• To remove a device from the array: “mdadm /dev/md0 -r /dev/loop1″ 

Some example losetup commands: 

• To check the Loop back device: “losetup /dev/loop1″
• To detach the loop device: “losetup -d /dev/loop1″ 

Further Reading

The Linux Documentation Project has a RAID HOW-TO here. Ryan Dlugosz has a guide for installing Gentoo onto a RAID array. Ferg’s Gaff has a good article on expanding Linux RAID Arrays. And of course, what kind of ricer would I be without a Gentoo reference on how to build a box on RAID1 ?