Possible Failed RAID5 to RAID6 Converstion

Success!!

So in summary the issue I had was I took a 6 disk RAID5 array out of a RN516 and put them into a RN528X with a view to adding a 7th disc and benefitting from the dual parity/redundancy of RAID6. 

The array had previously been vertically expanded. It started life as 6x6TB disks, then 6x10TB discs and now 4x14TB and 2x10TB. As a result there are 3 'layers' - the 6x6TB layer, the additional 6x4TB layer and then the 4x4TB layer.

ReadyOS correctly reshaped the first two from RAID5 into RAID6 but the final layer remained as RAID5 (using the extra 4TB of the 14TB as space rather than parity).

So in effect I didn't have true dual redundancy. You could see this in the GUI - on the System -> Volumes page it said RAID5 in the 'data' information section (blue box on left) but RAID6 below the graphical representation of all the disks in the unit.

There were two possible options

1) Factory reset and restore from backup

2) Attempt to reshape the abberant array layer from RAID5 to RAID6.

Given I had a good backup (proper RAID6 in a RN428) I decided to thy the latter.

So you can see here that md125 is the array in question:

root@MediaMaster:~# btrfs fi show /data
Label: '0ed5d010:data'  uuid: f1374f27-204a-4692-96d6-554d85e68f77
        Total devices 3 FS bytes used 44.47TiB
        devid    1 size 27.27TiB used 25.81TiB path /dev/md127
        devid    2 size 18.19TiB used 16.73TiB path /dev/md126
        devid    3 size 14.55TiB used 2.08TiB path /dev/md125

root@MediaMaster:~# cat /proc/mdstat
Personalities : [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md125 : active raid5 sdc5[0] sdg5[4] sda5[3] sdb5[2] sdd5[1]
      15623254016 blocks super 1.2 level 5, 64k chunk, algorithm 2 [5/5] [UUUUU]

md126 : active raid6 sda4[9] sdg4[10] sdf4[5] sde4[4] sdd4[6] sdc4[7] sdb4[8]
      19528915840 blocks super 1.2 level 6, 64k chunk, algorithm 2 [7/7] [UUUUUUU]

md127 : active raid6 sda3[11] sdg3[12] sdf3[7] sde3[6] sdd3[8] sdc3[9] sdb3[10]
      29278364160 blocks super 1.2 level 6, 64k chunk, algorithm 2 [7/7] [UUUUUUU]

md1 : active raid10 sda2[0] sdg2[6] sdf2[5] sde2[4] sdd2[3] sdc2[2] sdb2[1]
      1827840 blocks super 1.2 512K chunks 2 near-copies [7/7] [UUUUUUU]

md0 : active raid1 sda1[11] sdg1[12] sdf1[7] sde1[6] sdd1[8] sdc1[9] sdb1[10]
      4190208 blocks super 1.2 [7/7] [UUUUUUU]

unused devices: <none>
root@MediaMaster:~# mdadm --detail /dev/md125
/dev/md125:
           Version : 1.2
     Creation Time : Thu Jun  4 05:01:37 2020
        Raid Level : raid5
        Array Size : 15623254016 (14899.50 GiB 15998.21 GB)
     Used Dev Size : 3905813504 (3724.87 GiB 3999.55 GB)
      Raid Devices : 5
     Total Devices : 5
       Persistence : Superblock is persistent

       Update Time : Thu Sep  3 21:10:07 2020
             State : clean
    Active Devices : 5
   Working Devices : 5
    Failed Devices : 0
     Spare Devices : 0

            Layout : left-symmetric
        Chunk Size : 64K

Consistency Policy : unknown

              Name : 0a436c4c:data-2  (local to host 0a436c4c)
              UUID : 9b299c45:6236c52f:26afdb09:a5615851
            Events : 18795

    Number   Major   Minor   RaidDevice State
       0       8       37        0      active sync   /dev/sdc5
       1       8       53        1      active sync   /dev/sdd5
       2       8       21        2      active sync   /dev/sdb5
       3       8        5        3      active sync   /dev/sda5
       4       8      101        4      active sync   /dev/sdg5

So I did the following:

root@MediaMaster:~# btrfs fi resize 3:-5t /data

In reality you should only need to resize the filesystem by 4TB (4t) - as we are in effect removing one of the 4TB parts of the 5x4TB array but just in case there are conversion issues with base 10 v. base 2 etc.. I used -5t. the 3: is taken from the btrfs information above where md125 is devid 3.

You need to shrink the filesystem as you are later going to truncate the array - and if you didn't I believe you could lose data (without warning).

root@MediaMaster:~# mdadm --fail --verbose /dev/md125 /dev/sdg5
mdadm: set /dev/sdg5 faulty in /dev/md125
root@MediaMaster:~# mdadm --remove --verbose /dev/md125 /dev/sdg5
mdadm: hot removed /dev/sdg5 from /dev/md125
root@MediaMaster:~# mdadm --zero-superblock --verbose /dev/sdg5

Now we virtually remove the the last disk added (/dev/sdg) from the array. We can see from the mdstat info above that md125 is made up of /dev/sd[a-g]5.

First you fail it, then you remove it, then you zero it to prevent it being automatically re-added.

root@MediaMaster:~# mdadm --grow --verbose /dev/md125 --array-size 11717440512

Now you resize the array. In the mdadm info above we can see that the array size is 15623254016 (KB). This is a 5x4TB array with 1x4TB redundancy as currently RAID5 - so the available size represents the 4 disks. As we are moving to dual redunancy we need to resize it to 3 disks - so 156232544016 / 4 * 3 = 11717440512.

root@MediaMaster:~# mdadm --grow --verbose /dev/md125 --raid-devices=5 --level=6 --add /dev/sdg5 --backup-file=/backupfile
mdadm: level of /dev/md125 changed to raid6
mdadm: added /dev/sdg5
mdadm: Need to backup 768K of critical section..

And this is where the business is! This readds the disk/partition (/dev/sdg5) we just removed - this time in RAID6 (--level=6), adding it (back) as the 5th disk (--raid-devices=5), using all the space available (--size=max) and you MUST specify a backup file - as part of the reshaping it needs to save so critical info elsewhere on the filesystem - if you don't specify this it will fail and remain RAID5.

This will take a long time (hours/days).

When it's finally finished

root@MediaMaster:~# btrfs fi resize 3:max /data

Will expand the filesystem back up. Then reboot (seems to be needed to update GUI etc..)

And I finally had:

root@MediaMaster:~# cat /proc/mdstat
Personalities : [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md125 : active raid6 sdc5[0] sdg5[6] sda5[3] sdb5[2] sdd5[1]
      11717442240 blocks super 1.2 level 6, 64k chunk, algorithm 2 [5/5] [UUUUU]

md126 : active raid6 sda4[9] sdg4[10] sdf4[5] sde4[4] sdd4[6] sdc4[7] sdb4[8]
      19528915840 blocks super 1.2 level 6, 64k chunk, algorithm 2 [7/7] [UUUUUUU]

md127 : active raid6 sda3[11] sdg3[12] sdf3[7] sde3[6] sdd3[8] sdc3[9] sdb3[10]
      29278364160 blocks super 1.2 level 6, 64k chunk, algorithm 2 [7/7] [UUUUUUU]

md1 : active raid10 sda2[0] sdg2[6] sdf2[5] sde2[4] sdd2[3] sdc2[2] sdb2[1]
      1827840 blocks super 1.2 512K chunks 2 near-copies [7/7] [UUUUUUU]

md0 : active raid1 sda1[11] sdg1[12] sdf1[7] sde1[6] sdd1[8] sdc1[9] sdb1[10]
      4190208 blocks super 1.2 [7/7] [UUUUUUU]

unused devices: <none>
root@MediaMaster:~# mdadm --detail /dev/md125
/dev/md125:
           Version : 1.2
     Creation Time : Thu Jun  4 05:01:37 2020
        Raid Level : raid6
        Array Size : 11717442240 (11174.62 GiB 11998.66 GB)
     Used Dev Size : 3905814080 (3724.87 GiB 3999.55 GB)
      Raid Devices : 5
     Total Devices : 5
       Persistence : Superblock is persistent

       Update Time : Sun Sep  6 16:25:14 2020
             State : clean
    Active Devices : 5
   Working Devices : 5
    Failed Devices : 0
     Spare Devices : 0

            Layout : left-symmetric
        Chunk Size : 64K

Consistency Policy : unknown

              Name : 0a436c4c:data-2  (local to host 0a436c4c)
              UUID : 9b299c45:6236c52f:26afdb09:a5615851
            Events : 25162

    Number   Major   Minor   RaidDevice State
       0       8       37        0      active sync   /dev/sdc5
       1       8       53        1      active sync   /dev/sdd5
       2       8       21        2      active sync   /dev/sdb5
       3       8        5        3      active sync   /dev/sda5
       6       8      101        4      active sync   /dev/sdg5
root@MediaMaster:~#

Just to be sure nothing was damanged I also hashed every file before and after and they all checked out.

Enjoy!! Obviously this is not without risk and always ensure you have an up-to-date and reliable backup.

A