Failing to Extend the RAID 10 on My KVM Server

When I built my homelab server in 2015, I built it with future upgrades in mind. Since it was acting as my home NAS, room for more hard drives was important, and I chose a case with plenty of 3.5” hard drive bays.

Netflix, Hulu, and other streaming services have greatly reduced my storage needs. Last year, I had nearly a terabyte of free space on my NAS. The fastest-growing part of my collection of important data was the raw photos from my Canon 6D. I seem to take about 80 GB each year, so I assumed my first hard drive upgrade would be at least three or four years away.

Then I started flying FPV quadcopters. Last year, I recorded 209 GB of FPV quadcopter video. Most of that was low-bitrate, standard-definition DVR video recorded on my goggles. Last month, I started capturing FPV footage with a GoPro HERO5 Session. Those video files are much bigger, and I’ve already collected 151 GB of video this year. I won’t be surprised if I end up with 100 GB of video each month!

I bought another 4 TB 7200 RPM disk for my KVM server. That will give me an additional 2 TB of storage.

How many disks are currently in your RAID 10?

There are two disks in my RAID 10. Linux’s software RAID allows RAID 10 arrays to have just two disks or even odd numbers of disks.

I could have created a RAID 1 with those two disks, but that complicates adding additional disks. You can add a third disk to a RAID 1, but it won’t increase your available space—it will increase your redundancy with a third copy of your data.

If you have a RAID 1, and you want to increase your available storage space, you will have to use another pair of disks to create a mirror. I didn’t want to have to buy my disks in pairs.

With my two-disk RAID 10, I can install a third disk and extend my array. Linux’s software RAID 10 implementation will make sure there are two copies of every block spread across all three disks—two copies of each block spread across as many disks as you have in the array!

What’s involved in adding another disk to a RAID 10?

It is pretty simple, but there are quite a few layers involved.

Partition the new disk
Add the new partition to the RAID 10
Extend the LVM Physical Volume (PV)
Extend the LVM Logical Volume (LV) where my qcow images reside
Grow the file system that resides on this LV

That will take care of the KVM host, but it won’t do anything to directly address my lack of storage in my NAS virtual machine. I will also need to grow the NAS virtual machines’s qcow image.

Resize the NAS qcow image
Grow the NAS’s EXT4 file system to match the larger qcow image

I hit a road-block

This was supposed to be a straightforward blog post. Adding a disk to a RAID 10 is simple. I’ve done it before. No problems. This time, though, I seem to have outsmarted myself. I managed to get to the “Add the new partition to the RAID 10” step when I encountered a problem.

root@kvm:~# mdadm --add /dev/md1 /dev/sde1 
mdadm: added /dev/sde1
root@kvm:~# mdadm --grow /dev/md1 --raid-devices=3
mdadm: Cannot reshape RAID10 in far-mode
root@kvm:~# mdadm --grow /dev/md1 --layout=n
mdadm: Cannot reshape RAID10 in far-mode

I intentionally created this array using far-copies for the increased read performance. I didn’t know that it would take away my ability to reshape the array!

What to do next?

This should be easy to remedy, because I am running LVM on top of my RAID 10. I just need to migrate the data off the existing array, recreate the array using near-copies, then migrate the data back. LVM will do all the heavy lifting for me without any downtime outside of the hardware swap.

I have enough free disk space to make that happen right now. My two-disk RAID 10 is the same size as my new, single 4-TB drive. I could add that 4-TB drive to the Volume Group (VG), remove the RAID 10 from the VG, and then recreate the RAID 10.

I’m not brave enough to do that. I don’t have a lot of trust in a single, brand-new hard disk. I’d feel much more confident using a mirrored pair of disks as my temporary storage.

I asked my friend Brian if he had any spare 4-TB drives around. He is the NAS guy, so he usually has spare parts sitting around. I figured it’d be easier to borrow a drive from Brian than order another disk. It won’t actually be easier, but it’ll save me a few bucks, and it’ll be an excuse to write another blog post. That’ll have to do.

RAID is not a backup!

I try to say this as often as I can. RAID helps you prevent downtime. RAID may increase your disk throughput. RAID is not a replacement for backups.

Your RAID will probably protect you from a disk failure. It may or may not protect you when two or more disks fail. In fact, you may very well lose another disk when the array is rebuilding.

Your RAID won’t protect you from ransomware. It won’t protect you from accidental file deletions. It won’t protect you if you save corrupt data over an important file.

I have good backups, so why am I worried about using a mirror as temporary storage for this data migration?

Pretty much everything in my home directory is synchronized almost instantly to the cloud. My blog posts, my shell scripts, and even my save games are synchronized to my Seafile server.

Until recently, the largest files I had to regularly back up were the raw photographs from my Canon 6D. That adds up to less than 100 GB every year. All those raw files are stored in the cloud, on my laptop, and on my NAS.

Now I fly FPV quadcopters a couple of times a week. Every time I go out now, I come home with around 20 GB of GoPro footage.

Is this footage worth backing up? Almost definitely not. Just browsing through yesterday’s video is a time-consuming process. The older these videos get, the less likely I am to use them. Once I’ve compiled a video and uploaded it to YouTube, I don’t think I’ll ever touch the original video files again.

I’m not syncing these videos to the cloud, but I’d prefer not to put them in jeopardy on purpose.

I’d also be bummed out if I had to restore all these virtual machines. A little extra work up-front may prevent a whole bunch of problems later on!

Conclusion

I’ll get back to this in a few days. Brian used this as an excuse to upgrade his personal NAS to 8-TB disks. His 8-TB drives will be here soon, and I’m sure he’ll start the process of upgrading shortly after they arrive.

In the mean-time, I’m going to zero out the new drive and see if it gives me any problems! Once the drive arrives, I’ll document the process of juggling Physical Volumes (PVs). Once that is done, I should be able to document the process I was hoping to run through in this blog post!

root@kvm:~# dd if=/dev/zero of=/dev/sde1 bs=1M status=progress
1333833760768 bytes (1.3 TB, 1.2 TiB) copied, 7171.01 s, 186 MB/s 

This is the part where I usually ask my audience a question, but I don’t have a good question this time. If you have questions about any of this, please leave a comment below or join our Discord server!