Reducing RAID size (removing drives) WITHOUT DATA LOSS is possible via SSH
In the following process, I explore removing a drive from an array without having to destroy and re-build the array and restore all files from backup. This information is provided by myself, just another ReadyNAS enthusiast, with no warranty of any kind. It worked for me, but your configuration may be different enough that it doesn't for you. Please read all the way through before starting anything. Insure you have a current backup in case things go sideways. Most of these steps will not survive a reboot and neither will your volume if you are in the middle of the process and a power cycles or a soft reboot occurs, so have your NAS on an UPS. If you mess up, Netgear is not going to get you out of it, as they have such warnings about using SSH. And it should go without saying that you can't remove a drive if the amount of data on the NAS won't fit on the drives that will remain, preferably with headroom.
I started with a 3-drive XRAID (RAID5) array consisting of 3x1TB drives and my volume is called data, the XRAID default. This process allowed me to remove drive three and change to RAID1 with the other two drives. This is probably not a typical use case, you are more likely to want to remove a drive if the others are much larger, but my collection of sandbox drives is limited and something went awry with my first experiment of this type and I dind't want to have to wait long to start over. But the process is applicable to other scenarios as well. You just need to be sure you are working on the right drive and partition. Note that is is not applicable for going from RAID to single drive, but I plan to try that next.
Before starting, it's a good idea to do a re-boot to insure everything is healthy. If any expansion has been done since it was last booted, that will also insure that the drive device ID's are put in natural order (drive1=sda, drive2=sdb, drive3=sdc, etc.). And if anything is caught up waiting for a sync, it should trigger it.
First, I verified the configuration:
# btrfs filesystem show /data
Label: '758b3fbc:data' uuid: 116ad59c-74ef-4e73-b888-21ddfe4b3b43
Total devices 1 FS bytes used 2.41MiB
devid 1 size 1.81TiB used 2.02GiB path /dev/md127
# mdadm --detail /dev/md127
/dev/md127:
Version : 1.2
Creation Time : Thu Apr 11 07:47:59 2019
Raid Level : raid5
Array Size : 1943825664 (1853.78 GiB 1990.48 GB)
Used Dev Size : 971912832 (926.89 GiB 995.24 GB)
Raid Devices : 3
Total Devices : 3
Persistence : Superblock is persistent
Update Time : Thu Apr 11 21:29:04 2019
State : clean
Active Devices : 3
Working Devices : 3
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 64K
Name : 758b3fbc:data-0 (local to host 758b3fbc)
UUID : 15262894:2d10cdb5:0263bc96:8d941b03
Events : 146
Number Major Minor RaidDevice State
0 8 3 0 active sync /dev/sda3
1 8 19 1 active sync /dev/sdb3
2 8 35 2 active sync /dev/sdc3
# cat /proc/mdstat
Personalities : [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md127 : active raid5 sdc3[2] sdb3[1] sda3[0]
1943825664 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/3] [UUU]
md1 : active raid1 sdc2[2] sdb2[1] sda2[0]
523712 blocks super 1.2 [3/3] [UUU]
md0 : active raid1 sdc1[5] sdb1[4] sda1[0]
4190208 blocks super 1.2 [4/3] [UUU_]
unused devices: <none>
I've highlighted in red the important data these commands gave me. The first command confirmed that my data volume is on device md127. Using that information for the second command, it told me that md127 consists of the third partition of all three drives (sda3, sdb3, and sdc3) with 1943825664KB storage space. The third just confirms than none of the devices are currently in re-sync, including OS and swap. If you have larger drives in some slots, you'll also have a md126 layer consisting of the fourth partition of those drives (sda4, sdb4, etc.), and possibly others, but you won't need to do anything with them unless a drive to be removed includes one of the partitions in the other RAID layer. You don't need to do anything with the OS partition (md0: sda1, sdb1, sdc1...) or swap partition (md1: sda2, sdb2, sdc2...).
The first thing I had to do is shrink the file system to a size that is no larger than the target RAID array size. Drive 3 is a 1TB. That's a little less than what the file system calls a TB (and drive manufactures call a TiB). So, I just shrank by 1TB (really 1 TiB) instead of doing the math to see what the exact size needs to be. I will re-size it to use all available space at the end.
# btrfs filesystem resize -1t /data
The next step virtually failed and then removed the drive, as one cannot remove an active partition from an MDADM device. Then, I zeroed the superblocks so MDADM won't think the drive still belongs to a RAID group. Note that this needed to be done virtually, not by actually removing the drive, because I will re-use the drive as temporary storage during the re-shape. But the NAS UI is still going to complain about the degraded drive and, for a while, an inactive volume. Just ignore it, it'll all get fixed in the long run.
Next, I created the backup storage. Since the removed partition was large enough for the data on that part of the RAID when it was a part of the RAID, I could just create a new volume using that space to serve as the required backup storage and be assured I had enough. Before I created the file, I marked the partition as normal Linux, not Linux RAID, using cfdisk:
# cdisk /dev/sdc
In CFDISK, I selected the third partition and then selected change type. I selected Linux Filesystem as the partition type, and then write, and answered yes to the confirmation. In my past experiment I discovered cfdisk gives an error if you select Help, so I didn't trust it. It still does get the error, but I've tested other functions, and it seems to otherwise work properly.
So, next I formatted the partition using EXT3:
# mkfs.ext3 -L backupvol /dev/sdc3
Note that I first tried this without changing the partition type and used BTRFS for the file system, and the NAS OS "discovered" the new file system and added it to the UI as an "unknown RAID type". But something else went wrong that made me have to start over, and I didn't get to the end to see if that mattered. I made the change to the partition type and formatting, thinking the NAS would no longer "discover" it, but it made no difference -- my backupval still appeared in the UI as an unknown RAID type. But in the end, it didn't matter.
Now, it was time to re-size md127. From the status information we started with, md127 contains 1943825664KB. So, I divided that by two to get the new size for md127 of 971912832KB. If you have more than just 3 drives, you'll calculate the new size using the current size divided by the number of drives less the parity one(s) multiplied by the number that will remain, less parity. For 4 drives in RAID5 reduced to 3, (current size/3) * 2.
So, to tell MDADM to re-size to that value, decrease the number of drives, and use the new backupvol as temporary storage in the process, I did this:
This triggered a re-sync, which took the same amount of time as originally syncing the 3-drive RAID5. You can use the UI to see sync status progress, use cat /proc/mdstat for current status, watch cat proc/mdstat for constant updates, and/or use top to see all processes in action.
As long as you have shrunk the file system to small enough to fit in the new, smaller space (as demonstrated earlier), this will move everything around and not lose any data. If you didn't shrink it small enough, I don't think anything will prevent you from doing this command and potentially destroying your data volume.
Since I was dropping down to only two drives, I also needed to change the RAID type. If you have three or more remaining, you won't need this:
# mdadm --grow /dev/md127 --level=2
I believe this and the mdadm --grow commands above could all be concatenated together. But I had issues doing something similar when I did experiments to grow a FlexRAID array, so I separated them here as well. And, yes, --grow is the correct option to shrink. I assume it's a leftover from when shrink was not an option and they didn't change it to something more generic like --resize.
I verified all was as I expected with:
# mdadm --detail /dev/md127
/dev/md127:
Version : 1.2
Creation Time : Thu Apr 11 07:47:59 2019
Raid Level : raid1
Array Size : 971912832 (926.89 GiB 995.24 GB)
Used Dev Size : 971912832 (926.89 GiB 995.24 GB)
Raid Devices : 2
Total Devices : 2
Persistence : Superblock is persistent
Update Time : Fri Apr 12 07:20:30 2019
State : clean
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0
Name : 758b3fbc:data-0 (local to host 758b3fbc)
UUID : 15262894:2d10cdb5:0263bc96:8d941b03
Events : 2672
Number Major Minor RaidDevice State
0 8 3 0 active sync /dev/sda3
1 8 19 1 active sync /dev/sdb3
Success! RAID1, 2 of 2 devices, sda3 & sdb3, with the anticipated size.
Finally, I re-sized the BTRFS file system to use the maximum available space:
# btrfs filesystem resize max /data
Oddly, that didn't seem to do anything. But some time had gone by after the re-sync and maybe XRAID had expanded it for me. Even if the BTRFS reduction had interpreted the 1TB reduction as 1TB and not 1TiB (as I believe it does), the resulting file syatem should have been less than the size of a partition because the OS and swap partition take away form the 1TB available. I verified with df that the BTRFS file sysystem was the same size as the mdadm array:
# df /data
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/md127 971912832 102236324 868027484 11% /data
So, maybe the final resize isn't needed, but it can't hurt and takes no time.
Now, I could power down, remove the physical drive and re-boot the system. After re-boot, the OS re-built the OS and swap partitions to use only the available drives (which only takes 4-5 minutes) and everything seemed to be intact. My backupvol was gone from the UI. If the drive you are removing includes another RAID group and corresponding drive partition that are also part of your volume, you'll have to go through this process for that as well before the removal.
While this all works, you can certainly see why Netgear didn't include it in XRAID. Especially if the drive to be removed has multiple partitions, it's just a hard decision process to automate. But it is something I think they could implement in an advanced Volumes tab in the GUI to be used at least in FlexRAID, if not XRAID.
I mention above that something went awry in my first attempt. My sandbox drives have a lot of reallocated sectors, and a drive error appears to be what caused it to lock up in the middle of the RAID reduction resync. The NAS log shows ATA errors on one of the drives in that tmeframe. But the same could still happen to you, so before you start, do insure you have a backup and the time to restore it if something does happen. Re-starting also allowed me to do this in XRAID mode, which I could not do with my previous configuration. This working in XRAID should mean it will work in FlexRAID, but the opposite is not always true.
Re: Reducing RAID size (removing drives) WITHOUT DATA LOSS is possible via SSH
So, now it was time to move from 2 drives in RAID1 to one in JBOD, removing drive 2. Except it turns out that the NAS actually creates a 1-drive "RAID1" when only one drive is installed and calls it "JBOD". So, it's a very simple process:
Note the --force parameter is required because a RAID of 1 drive is not a normal format. But it does much more easily lend itself to further RAID expansion, which is obviously why it's used in the ReadyNAS.
End result is that the NAS UI showed 1 drive green, in JBOD mode. Just to be sure this really was equal to a JBOD created from scratch by the NAS, I switched to FlexRAID, added a drive, and selected "Add Group" to expand the JBOD, and it worked as expected.
Then, I wanted to remove a drive from a JBOD. First to see what's in the data volume:
# btrfs filesystem show /data
Label: '758b3fbc:data' uuid: 116ad59c-74ef-4e73-b888-21ddfe4b3b43
Total devices 2 FS bytes used 97.22GiB
devid 1 size 926.89GiB used 100.02GiB path /dev/md127
devid 2 size 926.89GiB used 0.00B path /dev/md126
# cat /proc/mdstat
Personalities : [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md126 : active raid1 sdb3[0]
971912832 blocks super 1.2 [1/1] [U]
md127 : active raid1 sda3[0]
971912832 blocks super 1.2 [1/1] [U]
md1 : active raid1 sda2[0] sdb2[1]
523712 blocks super 1.2 [2/2] [UU]
md0 : active raid1 sda1[0] sdb1[4]
4190208 blocks super 1.2 [4/2] [UU__]
So, all RAID elements were in sync and I needed to remove md126 and check it's no longer included in /data
# btrfs device delete /dev/md126 /data
# btrfs filesystem show /data
Label: '758b3fbc:data' uuid: 116ad59c-74ef-4e73-b888-21ddfe4b3b43
Total devices 1 FS bytes used 97.22GiB
devid 1 size 926.89GiB used 100.02GiB path /dev/md127
OK, all was well, so I powered down, removed the added drive, and powered back on. Once again, the NAS was happy as a clam with it's one-drive JBOD.
I re-enabled XRAID and re-inserted a clean second drive it started to sync. I ran into a sync problem because my sandbox drives are on their last legs. But that it accepted it and started the re-sync satisfied me all would have completed with better drives.
So, I'm convinced that each of the reduction steps left the NAS in the same condition (or close enough it doesn't matter) as creating an array of that size from the start.
