NETGEAR is aware of a growing number of phone and online scams. To learn how to stay safe click here.
Forum Discussion
Possible Failed RAID5 to RAID6 Converstion
Hi I have a RN528X. I moved a 6 drive RAID5 (XRAID2) array from a RN516. This consisted of 4x14TB and 2x10TB. They have been vertically expanded before (6x6TB -> 6x10TB -> replace 4 with 14TB). ...
powellandy1 wrote:
I'm running an experiment currently - 6x6TB was RAID5. I've resized FS, failed and removed last disk, currently rebuilding. Will then try and grow into RAID6. I haven't turned XRAID off. Will report back.
You probably should turn it off before you add the last disk back. Though you'll need to convert each of the md12x RAID groups to RAID-6 (I think you have three of those). You could do that from SSH, though if you are in FlexRAID you should also be able to do it from the web ui.
Also, I suggest either unformatting or reformatting the disk before before you reinsert it.
powellandy1 wrote:
but you have clarified we can go back (assuming the structure can be handled by XRAID).
You should be able to go back in this case. But the rules Netgear is applying are intentionally conservative, so it is possible that you will be stuck in FlexRAID when this is done.
For you that is likely just an inconvenience, since you understand how the RAID groups work.
My experience is that the existance of any "expanded volume", even if it was originally created by XRAID, cannot be switched back to XRAID once switched to FlexRAID. "Expanded volume" appears to include any multi-layer RAID.
I'm pretty sure that's not the way it always was, but it's the way it is now.
So my test case looks successful.
Starting point was a 6x6TB RAID5 (XRAID) array. Aim was to see if could convert to RAID6 array without data loss.
Background info:
root@MediaMKV:~# btrfs filesystem show /data Label: '0ed5d010:data' uuid: da6cab5c-d4b9-4523-8682-cf159c88d4a6 Total devices 1 FS bytes used 59.16GiB devid 1 size 27.27TiB used 69.02GiB path /dev/md127 root@MediaMKV:~# mdadm --detail /dev/md127 /dev/md127: Version : 1.2 Creation Time : Wed Apr 10 23:10:29 2019 Raid Level : raid5 Array Size : 29278364160 (27922.02 GiB 29981.04 GB) Used Dev Size : 5855672832 (5584.40 GiB 5996.21 GB) Raid Devices : 6 Total Devices : 6 Persistence : Superblock is persistent Update Time : Mon Aug 31 00:28:57 2020 State : clean Active Devices : 6 Working Devices : 6 Failed Devices : 0 Spare Devices : 0 Layout : left-symmetric Chunk Size : 64K Consistency Policy : unknown Name : 0ed5d010:data-0 (local to host 0ed5d010) UUID : b6e0550c:d1163f98:05f4f5ba:7e1c8599 Events : 129842 Number Major Minor RaidDevice State 0 8 3 0 active sync /dev/sda3 6 8 19 1 active sync /dev/sdb3 2 8 35 2 active sync /dev/sdc3 3 8 51 3 active sync /dev/sdd3 4 8 67 4 active sync /dev/sde3 5 8 83 5 active sync /dev/sdf3 root@MediaMKV:~# cat /proc/mdstat Personalities : [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] md127 : active raid5 sda3[0] sdf3[5] sde3[4] sdd3[3] sdc3[2] sdb3[6] 29278364160 blocks super 1.2 level 5, 64k chunk, algorithm 2 [6/6] [UUUUUU] md1 : active raid10 sda2[0] sdb2[5] sdf2[4] sde2[3] sdd2[2] sdc2[1] 1566720 blocks super 1.2 512K chunks 2 near-copies [6/6] [UUUUUU] md0 : active raid1 sda1[0] sdf1[5] sde1[4] sdd1[3] sdc1[2] sdb1[6] 4190208 blocks super 1.2 [6/6] [UUUUUU] unused devices: <none> root@MediaMKV:~# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 5.5T 0 disk ├─sda1 8:1 0 4G 0 part │ └─md0 9:0 0 4G 0 raid1 / ├─sda2 8:2 0 512M 0 part │ └─md1 9:1 0 1.5G 0 raid10 [SWAP] └─sda3 8:3 0 5.5T 0 part └─md127 9:127 0 27.3T 0 raid5 /data sdb 8:16 0 5.5T 0 disk ├─sdb1 8:17 0 4G 0 part │ └─md0 9:0 0 4G 0 raid1 / ├─sdb2 8:18 0 512M 0 part │ └─md1 9:1 0 1.5G 0 raid10 [SWAP] └─sdb3 8:19 0 5.5T 0 part └─md127 9:127 0 27.3T 0 raid5 /data sdc 8:32 0 5.5T 0 disk ├─sdc1 8:33 0 4G 0 part │ └─md0 9:0 0 4G 0 raid1 / ├─sdc2 8:34 0 512M 0 part │ └─md1 9:1 0 1.5G 0 raid10 [SWAP] └─sdc3 8:35 0 5.5T 0 part └─md127 9:127 0 27.3T 0 raid5 /data sdd 8:48 0 5.5T 0 disk ├─sdd1 8:49 0 4G 0 part │ └─md0 9:0 0 4G 0 raid1 / ├─sdd2 8:50 0 512M 0 part │ └─md1 9:1 0 1.5G 0 raid10 [SWAP] └─sdd3 8:51 0 5.5T 0 part └─md127 9:127 0 27.3T 0 raid5 /data sde 8:64 0 5.5T 0 disk ├─sde1 8:65 0 4G 0 part │ └─md0 9:0 0 4G 0 raid1 / ├─sde2 8:66 0 512M 0 part │ └─md1 9:1 0 1.5G 0 raid10 [SWAP] └─sde3 8:67 0 5.5T 0 part └─md127 9:127 0 27.3T 0 raid5 /data sdf 8:80 0 5.5T 0 disk ├─sdf1 8:81 0 4G 0 part │ └─md0 9:0 0 4G 0 raid1 / ├─sdf2 8:82 0 512M 0 part │ └─md1 9:1 0 1.5G 0 raid10 [SWAP] └─sdf3 8:83 0 5.5T 0 part └─md127 9:127 0 27.3T 0 raid5 /data root@MediaMKV:~# df -h Filesystem Size Used Avail Use% Mounted on udev 10M 4.0K 10M 1% /dev /dev/md0 4.0G 649M 3.0G 18% / tmpfs 3.9G 0 3.9G 0% /dev/shm tmpfs 3.9G 6.0M 3.9G 1% /run tmpfs 2.0G 1.4M 2.0G 1% /run/lock tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup /dev/md127 28T 60G 28T 1% /data /dev/md127 28T 60G 28T 1% /apps /dev/md127 28T 60G 28T 1% /homeFirstly I toggled XRAID off (although that may not be necessary - see later)
Then issued the following commands via SSH:
root@MediaMKV:~# btrfs filesystem resize -7t /data Resize '/data' of '-7t' root@MediaMKV:~# mdadm /dev/md127 --fail --verbose /dev/sdf3 mdadm: set /dev/sdf3 faulty in /dev/md127 root@MediaMKV:~# mdadm /dev/md127 --remove --verbose /dev/sdf3 mdadm: hot removed /dev/sdf3 from /dev/md127 root@MediaMKV:~# mdadm --zero-superblock --verbose /dev/sdf3 root@MediaMKV:~# mdadm --grow --verbose /dev/md127 --array-size 23422691328 root@MediaMKV:~# mdadm --grow --verbose /dev/md127 --raid-devices=6 --level=6 --add /dev/sdf3 --backup-file=/backupfile mdadm: level of /dev/md127 changed to raid6 mdadm: added /dev/sdf3 mdadm: Need to backup 1280K of critical section..
I note that you can re-add the 'removed' drive and reshape/covert the array both at the same time. I previously tried shrinking (via grow) to a 5 disk RAID5 with the intent of then issue the grow command but XRAID kicked in and convered back to 6 disk RAID 5 - given it appears you can do it in one go (all the other commands up to and including array-size are instantenous) you may not need to. I also note you must include -the --backup-file paramter - if you don't the mdadm reshape fails and recovers back to RAID5 again.
After finished issue:
root@MediaMKV:~# btrfs fi resize max /data
and reboot. And then you CAN toggle XRAID back on again.
Now I have:
root@MediaMKV:~# cat /proc/mdstat Personalities : [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] md127 : active raid6 sda3[0] sdf3[7] sde3[4] sdd3[3] sdc3[2] sdb3[6] 23422691328 blocks super 1.2 level 6, 64k chunk, algorithm 2 [6/6] [UUUUUU] md1 : active raid10 sda2[0] sdb2[5] sdf2[4] sde2[3] sdd2[2] sdc2[1] 1566720 blocks super 1.2 512K chunks 2 near-copies [6/6] [UUUUUU] md0 : active raid1 sda1[0] sdf1[5] sde1[4] sdd1[3] sdc1[2] sdb1[6] 4190208 blocks super 1.2 [6/6] [UUUUUU] unused devices: <none> root@MediaMKV:~# mdadm --detail /dev/md127 /dev/md127: Version : 1.2 Creation Time : Wed Apr 10 23:10:29 2019 Raid Level : raid6 Array Size : 23422691328 (22337.62 GiB 23984.84 GB) Used Dev Size : 5855672832 (5584.40 GiB 5996.21 GB) Raid Devices : 6 Total Devices : 6 Persistence : Superblock is persistent Update Time : Fri Sep 4 19:14:30 2020 State : clean Active Devices : 6 Working Devices : 6 Failed Devices : 0 Spare Devices : 0 Layout : left-symmetric Chunk Size : 64K Consistency Policy : unknown Name : 0ed5d010:data-0 (local to host 0ed5d010) UUID : b6e0550c:d1163f98:05f4f5ba:7e1c8599 Events : 155906 Number Major Minor RaidDevice State 0 8 3 0 active sync /dev/sda3 6 8 19 1 active sync /dev/sdb3 2 8 35 2 active sync /dev/sdc3 3 8 51 3 active sync /dev/sdd3 4 8 67 4 active sync /dev/sde3 7 8 83 5 active sync /dev/sdf3 root@MediaMKV:~# btrfs fi show /data Label: '0ed5d010:data' uuid: da6cab5c-d4b9-4523-8682-cf159c88d4a6 Total devices 1 FS bytes used 59.16GiB devid 1 size 21.81TiB used 68.02GiB path /dev/md127So now to try on the real thing!!
Success!!
So in summary the issue I had was I took a 6 disk RAID5 array out of a RN516 and put them into a RN528X with a view to adding a 7th disc and benefitting from the dual parity/redundancy of RAID6.
The array had previously been vertically expanded. It started life as 6x6TB disks, then 6x10TB discs and now 4x14TB and 2x10TB. As a result there are 3 'layers' - the 6x6TB layer, the additional 6x4TB layer and then the 4x4TB layer.
ReadyOS correctly reshaped the first two from RAID5 into RAID6 but the final layer remained as RAID5 (using the extra 4TB of the 14TB as space rather than parity).
So in effect I didn't have true dual redundancy. You could see this in the GUI - on the System -> Volumes page it said RAID5 in the 'data' information section (blue box on left) but RAID6 below the graphical representation of all the disks in the unit.
There were two possible options
1) Factory reset and restore from backup
2) Attempt to reshape the abberant array layer from RAID5 to RAID6.
Given I had a good backup (proper RAID6 in a RN428) I decided to thy the latter.
So you can see here that md125 is the array in question:
root@MediaMaster:~# btrfs fi show /data Label: '0ed5d010:data' uuid: f1374f27-204a-4692-96d6-554d85e68f77 Total devices 3 FS bytes used 44.47TiB devid 1 size 27.27TiB used 25.81TiB path /dev/md127 devid 2 size 18.19TiB used 16.73TiB path /dev/md126 devid 3 size 14.55TiB used 2.08TiB path /dev/md125 root@MediaMaster:~# cat /proc/mdstat Personalities : [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] md125 : active raid5 sdc5[0] sdg5[4] sda5[3] sdb5[2] sdd5[1] 15623254016 blocks super 1.2 level 5, 64k chunk, algorithm 2 [5/5] [UUUUU] md126 : active raid6 sda4[9] sdg4[10] sdf4[5] sde4[4] sdd4[6] sdc4[7] sdb4[8] 19528915840 blocks super 1.2 level 6, 64k chunk, algorithm 2 [7/7] [UUUUUUU] md127 : active raid6 sda3[11] sdg3[12] sdf3[7] sde3[6] sdd3[8] sdc3[9] sdb3[10] 29278364160 blocks super 1.2 level 6, 64k chunk, algorithm 2 [7/7] [UUUUUUU] md1 : active raid10 sda2[0] sdg2[6] sdf2[5] sde2[4] sdd2[3] sdc2[2] sdb2[1] 1827840 blocks super 1.2 512K chunks 2 near-copies [7/7] [UUUUUUU] md0 : active raid1 sda1[11] sdg1[12] sdf1[7] sde1[6] sdd1[8] sdc1[9] sdb1[10] 4190208 blocks super 1.2 [7/7] [UUUUUUU] unused devices: <none> root@MediaMaster:~# mdadm --detail /dev/md125 /dev/md125: Version : 1.2 Creation Time : Thu Jun 4 05:01:37 2020 Raid Level : raid5 Array Size : 15623254016 (14899.50 GiB 15998.21 GB) Used Dev Size : 3905813504 (3724.87 GiB 3999.55 GB) Raid Devices : 5 Total Devices : 5 Persistence : Superblock is persistent Update Time : Thu Sep 3 21:10:07 2020 State : clean Active Devices : 5 Working Devices : 5 Failed Devices : 0 Spare Devices : 0 Layout : left-symmetric Chunk Size : 64K Consistency Policy : unknown Name : 0a436c4c:data-2 (local to host 0a436c4c) UUID : 9b299c45:6236c52f:26afdb09:a5615851 Events : 18795 Number Major Minor RaidDevice State 0 8 37 0 active sync /dev/sdc5 1 8 53 1 active sync /dev/sdd5 2 8 21 2 active sync /dev/sdb5 3 8 5 3 active sync /dev/sda5 4 8 101 4 active sync /dev/sdg5So I did the following:
root@MediaMaster:~# btrfs fi resize 3:-5t /data
In reality you should only need to resize the filesystem by 4TB (4t) - as we are in effect removing one of the 4TB parts of the 5x4TB array but just in case there are conversion issues with base 10 v. base 2 etc.. I used -5t. the 3: is taken from the btrfs information above where md125 is devid 3.
You need to shrink the filesystem as you are later going to truncate the array - and if you didn't I believe you could lose data (without warning).
root@MediaMaster:~# mdadm --fail --verbose /dev/md125 /dev/sdg5 mdadm: set /dev/sdg5 faulty in /dev/md125 root@MediaMaster:~# mdadm --remove --verbose /dev/md125 /dev/sdg5 mdadm: hot removed /dev/sdg5 from /dev/md125 root@MediaMaster:~# mdadm --zero-superblock --verbose /dev/sdg5
Now we virtually remove the the last disk added (/dev/sdg) from the array. We can see from the mdstat info above that md125 is made up of /dev/sd[a-g]5.
First you fail it, then you remove it, then you zero it to prevent it being automatically re-added.
root@MediaMaster:~# mdadm --grow --verbose /dev/md125 --array-size 11717440512
Now you resize the array. In the mdadm info above we can see that the array size is 15623254016 (KB). This is a 5x4TB array with 1x4TB redundancy as currently RAID5 - so the available size represents the 4 disks. As we are moving to dual redunancy we need to resize it to 3 disks - so 156232544016 / 4 * 3 = 11717440512.
root@MediaMaster:~# mdadm --grow --verbose /dev/md125 --raid-devices=5 --level=6 --add /dev/sdg5 --backup-file=/backupfile
mdadm: level of /dev/md125 changed to raid6
mdadm: added /dev/sdg5
mdadm: Need to backup 768K of critical section..And this is where the business is! This readds the disk/partition (/dev/sdg5) we just removed - this time in RAID6 (--level=6), adding it (back) as the 5th disk (--raid-devices=5), using all the space available (--size=max) and you MUST specify a backup file - as part of the reshaping it needs to save so critical info elsewhere on the filesystem - if you don't specify this it will fail and remain RAID5.
This will take a long time (hours/days).
When it's finally finished
root@MediaMaster:~# btrfs fi resize 3:max /data
Will expand the filesystem back up. Then reboot (seems to be needed to update GUI etc..)
And I finally had:
root@MediaMaster:~# cat /proc/mdstat Personalities : [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] md125 : active raid6 sdc5[0] sdg5[6] sda5[3] sdb5[2] sdd5[1] 11717442240 blocks super 1.2 level 6, 64k chunk, algorithm 2 [5/5] [UUUUU] md126 : active raid6 sda4[9] sdg4[10] sdf4[5] sde4[4] sdd4[6] sdc4[7] sdb4[8] 19528915840 blocks super 1.2 level 6, 64k chunk, algorithm 2 [7/7] [UUUUUUU] md127 : active raid6 sda3[11] sdg3[12] sdf3[7] sde3[6] sdd3[8] sdc3[9] sdb3[10] 29278364160 blocks super 1.2 level 6, 64k chunk, algorithm 2 [7/7] [UUUUUUU] md1 : active raid10 sda2[0] sdg2[6] sdf2[5] sde2[4] sdd2[3] sdc2[2] sdb2[1] 1827840 blocks super 1.2 512K chunks 2 near-copies [7/7] [UUUUUUU] md0 : active raid1 sda1[11] sdg1[12] sdf1[7] sde1[6] sdd1[8] sdc1[9] sdb1[10] 4190208 blocks super 1.2 [7/7] [UUUUUUU] unused devices: <none> root@MediaMaster:~# mdadm --detail /dev/md125 /dev/md125: Version : 1.2 Creation Time : Thu Jun 4 05:01:37 2020 Raid Level : raid6 Array Size : 11717442240 (11174.62 GiB 11998.66 GB) Used Dev Size : 3905814080 (3724.87 GiB 3999.55 GB) Raid Devices : 5 Total Devices : 5 Persistence : Superblock is persistent Update Time : Sun Sep 6 16:25:14 2020 State : clean Active Devices : 5 Working Devices : 5 Failed Devices : 0 Spare Devices : 0 Layout : left-symmetric Chunk Size : 64K Consistency Policy : unknown Name : 0a436c4c:data-2 (local to host 0a436c4c) UUID : 9b299c45:6236c52f:26afdb09:a5615851 Events : 25162 Number Major Minor RaidDevice State 0 8 37 0 active sync /dev/sdc5 1 8 53 1 active sync /dev/sdd5 2 8 21 2 active sync /dev/sdb5 3 8 5 3 active sync /dev/sda5 6 8 101 4 active sync /dev/sdg5 root@MediaMaster:~#Just to be sure nothing was damanged I also hashed every file before and after and they all checked out.
Enjoy!! Obviously this is not without risk and always ensure you have an up-to-date and reliable backup.
A
It does sound like a fault in the XRAID logic that it didn't properly make the 4 drive layer RAID6. Your growth process is a biut complex. Since automatic conversion to RAID6 would not occur on just 4 drives, that may have something to do with it with the 4 partitions. Not that I think that should be the case or is even how Netgear actually decided it should be, just that it's a flaw based on it. I'm glad you not only figured out the issue, but a fix for it. problem is, many users would not notice the issue and many more would not be comfortable following your fix.
Did you have to turn XRAID off to do this, and were you able to turn it on again when completed? My experience has been yes on needing to turn it off and no on being able to re-enable it.
Hi
I agree. It looks if perhaps it considers each partition in isolation and will only convert one to RAID6 if disk >6 which led to my atypical situation.
What it should really do if any partition has disks>6 is something like:
1-2 disks = spare
3 disks = 3 way RAID 1
4+ disks = RAID6
I didn't need to disable XRAID as I expanded and reshaped in one go. When I was experimenting and I removed the one drive, let it resync RAID5 and then tried to expand - if I didn't turn XRAID off then whenever the resync to n-1 disk RAID5 finished it would automatically add the nth back in RAID5. Doing it all in one go seems to work fine. I think it had issued the btrfs resize command automatically, not that that matters.
NETGEAR Academy
Boost your skills with the Netgear Academy - Get trained, certified and stay ahead with the latest Netgear technology!
Join Us!
