I recently ran into a problem that most of us are very familiar with: the fileserver has nearly run out of storage space!

There were a few things compounding the upgrade.

1. The server has only two SATA ports
2. Both SATA ports were in use by the existing RAID1 so I could not connect another RAID array and rsync the files across.
3. I needed to minimize downtime as the fileserver ran an important accounting and project management application. 10 hours of downtime to run a grow operation using a live-cd was unacceptable.

Luckily, both Linux MD RAID and the EXT3 file-system are quite mature products, and have features that help mitigate these problems. You can grow the md device while the device is fully running, and you can expand the ext3 file-system on that device to fill all available space while that file-system is mounted. That means your downtime is cut down to the time you need to shut the server down, unplug the drives, and plug the new ones in.

You will need two shutdowns to do this:

1. Remove one of the two drives in the array, and replace it with one of the larger ones
2. Remove the remaining small drive and replace it with the remaining larger one

I did some searching, and didn’t see any good answers about whether the growing of md devices while live was either possible, or a good idea. Even less documented is whether resize2fs works with a live file-system, or if it’s a good idea. Yes, it’s possible, and yes, it seemed to work out fine for me. Keep good backups before you start this! Don’t blame me if you lose everything!

My file-systems were laid out such that the root file-system and the boot partition were on the same drives as the data partition, so I had to be careful that the boot loaders were in place to boot off of either drive. It should have been done long ago when the system was built, but I double checked just to be sure. Minimizing mistakes minimizes downtime!

• Start by double checking your boot loader. I use grub, so it was as simple as:
  • grub-install /dev/hda
  • grub-install /dev/hdb

• double check that your array is up to date
  • cat /proc/mdstat
  • Should show something like this: 1950620224 blocks [2/2] [UU]
  • [UU] means all good, [_U] means one device has failed, check it out and resync it.

• fail and remove one drive
  • mdadm –manage /dev/md0 –fail /dev/hdb1
  • mdadm –manage /dev/md0 –remove /dev/hdb1

• shut down, remove that failed and removed drive, and replace it with a larger drive. Start the server back up again.
• copy the partition table over to the new drive. I like sfdisk for this:
  • sfdisk -d /dev/hda > hda.table
  • sfdisk /dev/hdb < hda.table
  • rm hda.table
  • delete your raid partition on the new drive (hdb in this case) and recreate it as the final size that you would like, in my case, the entire size of the remaining space of the drive. Make sure the partition type is the same. Use your favorite partition editor, I assume if you’re doing this you know what you’re doing there.

• re-add the new partitions on the new drive to the array
  • mdadm –manage /dev/md0 –add /dev/hdb1
  • cat /proc/mdstat
  • wait for the resync to finish. try this to keep an eye on it: watch -n 10 cat /proc/mdstat
  • this will take a while. It’ll tell you how long. Wait till you get a good [UU] again.

• add the boot loader to the new drive
  • grub-install /dev/hdb

• fail and remove the remaining small drive:
  • mdadm –manage /dev/md0 –fail /dev/hda1
  • mdadm –manage /dev/md0 –remove /dev/hda1
• Shutdown, remove the remaining small drive, and replace it with the remaining free large drive. I had to swap the currently installed large drive to the first SATA port as my bios won’t let a drive boot unless it’s in the first spot. But let’s continue with the how-to as if my original drive was able to boot in the hdb spot.
• Repeat the process with the new large drive
  • sfdisk -d /dev/hdb > hdb.table
  • sfdisk /dev/hda < hdb.table
  • we do not need to delete and resize the raid partition on the second drive since we did that on the first drive, and we’re copying the exact partition table to the new drive
  • mdadm –manage /dev/md0 –add /dev/hda1
  • watch -n 10 cat /proc/mdstat
  • wait until the resync is finished for the greatest safety and speed of the next step

Now we’ve got all our data safely moved over to the new drives, we’ve got the larger partition, and we’re ready to start doing the resizing of the actual array and file-system.

• mdadm –grow /dev/md0

• It’s hopefully as simple as that. Depending on your version of mdadm, you might need to use a –manage –grow command instead.
• wait. this doesn’t take as long as syncing. watch -n 10 cat /proc/mdstat is useful again

Your raid device is now resized. df will still show your mounted file-system as the same size though, it’s just your raid device that has the room to spare. So let’s grow the ext3 file-system, shall we?

• resize2fs /dev/md0

• /dev/md0 can still be mounted read/write while you do this
• wait. a LONG time. it took 8 hours for me to go from a 200GB file-system to a 2TB file-system. If you watch df -h, you’ll see your free space steadily climb.

You’re done! You have more space, and your business didn’t have to stop and wait for an rsync to complete to prevent your data from going out of sync!

This entry was posted by Chris Chatelain on Monday, April 11th, 2011 at 2:15 pm and is filed under Linux System Administration, Tools, Tips and Tricks. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.