XRAID turned RAID5 into RAID6 when adding a drive

[StephenB]

---

@jak0lantash wrote:

Alternative alternative conclusion - Any disk at least as big as the smallest HDD ever in the array can be added

Perhaps, but in some cases that would waste space.  If for instance you added a 750 GB drive to an array of 500 GB and 1 TB drives, it wouldn't repartition the 1 TB drives to accomodate the 750. However, once the partition structure is created, it lasts for the life of the array (with new partitions only created "on top").

[Retired_Member]

Yep, absolutely.

The limitation doesn't determine the best practice or optimum configuration 😉

[Retired_Member]

8f) 1x500GB + 1x1TB, md127 is a RAID1 of 2 partitions of 500GB

leave resync, replace the 1x500GB by 1x1TB: md127 is a RAID1 of 2 partitions of 500GB, md126 is a RAID1 of 2 partitions of 500GB

leave resync, add 1x500GB: I believe md127 should reshape to a RAID5 of 3 partitions of 500GB, md126 remains a RAID1 of 2 partitions of 500GB

-> I'll confirm that

-> Confirmed on 6.5.0 RC3

Now, this is definitely different than what was done before (so it was changed, but I don't know when).

Start with 1x500GB + 1x1TB. md127 is a RAID1 of 2 partitions of 500GB (500GB capacity, 500GB mirror). There is 500GB unused capacity on the second HDD. BTRFS volume capacity is 500GB.

md127 : active raid1 sda3[2] sdb3[1]
      483534784 blocks super 1.2 [2/1] [_U]

Replace the 1x500GB by a 1x1TB. md127 resyncs the RAID1 but also grows its size to 2 partitions of 1TB (1TB capacity, 1TB mirror). BTRFS volume capacity is 1TB.

md127 : active raid1 sda3[2] sdb3[1]
      971912800 blocks super 1.2 [2/2] [UU]

Label: '5e2704ec:data' uuid: f451742b-291b-4856-ab26-39a7b98327e4
 Total devices 1 FS bytes used 2.76MiB
 devid 1 size 926.89GiB used 3.02GiB path /dev/md127

As there is no trace of the old 500GB RAID1, adding a 500GB HDD will not trigger another horizontal expansion, as it would be too small.

So the conclusion is different: For Horizontal Expansion, any disk at least as big as the smallest HDD currently in the array can be added to the array.

(Doesn't necessarily apply to RAIDiator 4 and 5, the target here is a rule for ReadyNAS OS 6)

[Retired_Member]

8g) 1x500GB + 2x1TB. md127 is a RAID5 of 3 partitions of 500GB (1TB capacity, 500GB parity). md126 is a RAID1 of 2 partitions of 500GB (500GB capacity, 500GB mirror). BTRFS volume capacity is 1.5TB.

md126 : active raid5 sda4[2] sdc4[1] sdb4[0]
      488246912 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/3] [UUU]

md127 : active raid5 sda3[3] sdc3[2] sdb3[1]
      967069568 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/3] [UUU]

Replace the 1x500GB by 1x1TB. md127 remains a RAID5 of 3 partitions of 500GB (1TB capacity, 500GB parity). md126 reshapes to a RAID5 of 3 partitions of 500GB (1TB capacity, 500GB parity). BTRFS volume capacity is 2TB.

md126 : active raid5 sda4[2] sdc4[1] sdb4[0]
      976493824 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/3] [UUU]

md127 : active raid5 sda3[3] sdc3[2] sdb3[1]
      967069568 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/3] [UUU]

Both 8f and 8g are Vertical Expansion, not Horizontal Expansion.

But the point was to show that if a Vertical Expansion occurred when there was two HDDs (so in RAID1), you can't add a new HDD, for Horizontal Expansion, that is the same capacity as the smallest initial HDD. This is because, in RAID1, X-RAID extend the existing RAID1 during the Vertical Expansion (there is only one RAID, no secondary one).

If a Vertical Expansion occurred when the volume was using more than two HDs (so in RAID5 or RAID6), you can indeed add a new HDD, for Horizontal Expansion, that is the same capacity as the smallest initial HDD. This is because, in RAID5/6, X-RAID doesn't extend the existing RAID5 during the Vertical Expansion but creates or expand a secondary RAID.

Now, for the conclusion 

For Horizontal Expansion, in X-RAID mode, on ReadyNAS OS6, in RAID5 or 6, the disk added needs to be at least as big as the smallest HDD ever in the array (since the time it's in RAID5 or 6 - so 3 HDDs or more), in RAID1, the disk added needs to be at least as big as the smallest HDD currently in the array (since the time it's in RAID1 - so 2 HDDs).

What do you think StephenB?

[StephenB]

---

@jak0lantash wrote:

Now, for the conclusion 

For Horizontal Expansion, in X-RAID mode, on ReadyNAS OS6, in RAID5 or 6, the disk added needs to be at least as big as the smallest HDD ever in the array (since the time it's in RAID5 or 6 - so 3 HDDs or more), in RAID1, the disk added needs to be at least as big as the smallest HDD currently in the array (since the time it's in RAID1 - so 2 HDDs).

What do you think StephenB?

---

I am very impressed by the amount of testing you've done.

I think that people adding disks do want to know if space will be wasted (e.g. what their final volume size will be).  So I'd be careful about adding disks that are between the smallest and the largest.

Can you summarize the results on conversion to automatic redundancy?  Perhaps also summarize cases you found where a factory reset gives a different result from expansion.

[Retired_Member]

Thanks 🙂

I'll summarize all the results and add a note about your point. There is indeed a difference between what's possible and what's best.

If it's technically possible to add a 3TB HDD to your RAID5 volume that was composed of 3x1TB HDDs but then expanded to 3x4TB HDDs, it's not the best idea as you would waste 2TB capacity.

Thanks

[Retired_Member]

I'm still working on the final conclusion.

I had to do more tests as some of the previous assumptions/conclusions were actually wrong.

For example:

If you Factory Default a 6 bays unit with 6 HDDs, you get a RAID5.

If you Factory Default a 12 bays unit with 6 HDDs, you get a RAID6.

If you Factory Default a 12 bays unit with 5 HDDs, you get a RAID5. If you add a 6th HDD, you expand the RAID5! If you add a 7th HDD, you reshape to RAID6.

So the level of RAID even on the primary array is not necessarilly the same with Factory Default and Horizontal Expansion, even with the same number of HDDs, in the same chassis.

I also discovered situations where you can have the primary RAID array as RAID6 and a secondary RAID array as RAID5 (and "RAID5" displayed in the GUI). Which destroys the rule of XRAID maintains the same level of redundancy accros the RAID arrays.

I'll post once all the tests are complete and the conclusion proven and reliable.

[StephenB]

---

@jak0lantash wrote:

I'll post once all the tests are complete and the conclusion proven and reliable.

---

I'm looking forward to it.  

[Retired_Member]

Long overdue, here it comes... (had to split it as a message cannot contain more than 20,000 characters... really, there are 20,000 characters in there?)

On X-RAID mode, there is no choice on the RAID level, it's automatically determined by the number of HDDs in the chassis (and their capacity in case of mixed capacities), for Volume Creation, Horizontal Expansion and Vertical Expansion.
All the information below is specifically meant for X-RAID on ReadyNAS OS 6 (and more specifically for Firmware 6.5.0 - as all testing was done on that Firmware). So it doesn't necessarily apply to RAIDiator 4 and 5. Also Flex-RAID is TOTALLY different.
X-RAID was called X-RAID on RAIDiator 4.1, X-RAID2 on RAIDiator 4.2. On ReadyNAS OS6, depending on where you look, it's either X-RAID or X-RAID2.

----------

- For Volume Creation (when performing a Factory Default), on ReadyNAS OS6, in X-RAID mode, the level of redundancy is determined by the number of HDDs. The capacity of the HDDs is also important in case of mixed capacities.
For basic information about X-RAID: http://kb.netgear.com/app/answers/detail/a_id/22802

The volume as shown to the user is a BTRFS volume built on RAID array(s) (concatenated if multiple RAID arrays). A primary RAID array (called md127) is created based on all the HDDs (partition sdX3). In a RAID array, all members must be the same size. The smallest HDD will therefore limit the size of all members of the primary RAID array.
If there is unused capacity on some HDDs, and enough members to maintain the same level of redundancy in all the secondary RAID arrays as the primary RAID array, during Volume Creation, X-RAID can create secondary RAID array(s), called md126 (partition sdX4), md125 (partition sdX5), and so on.

For Volume Creation, X-RAID doesn't treat 6 HDDs the same way depending on the number of bays.
If the chassis has 6 bays or less, X-RAID will not use RAID6. If the chassis has more than 6 bays, it starts creating RAID6 arrays at the sixth disk.
X-RAID doesn't create volumes across chassis and expansion chassis (as it's a very bad idea), instead it will create separate volumes (much safer), so there is no situation where you can start with six HDDs RAID5 and add a HDD to the volume.

I repeat "Volume Creation", because some of the above rules don't exactly apply to expansion, see below.

The simple rules for Volume Creation with X-RAID/OS6:
- Chassis with six bays or less:

1 HDD    = JBOD
2 HDDs   = RAID1
3-6 HDDs = RAID5

- Chassis with more than six bays:

1 HDD    = JBOD
2 HDDs   = RAID1
3-5 HDDs = RAID5
6+ HDDs  = RAID6

----------

- For Vertical Expansion, on ReadyNAS OS6, in X-RAID mode, in RAID1, 5 or 6, the replaced disk needs to be bigger than the one it replaced (logical I guess). If you replace the HDD by the same size, it simply resyncs and obviously doesn't expand. There is no other "limitation" for the HDD itself.
For basic information about Vertical Expansion: http://kb.netgear.com/app/answers/detail/a_id/23135

If the RAID level is RAID1 (so two HDDs), the primary (and only) RAID array itself will be grown, it's "extended" once both HDDs were upgraded (one by one).
If the RAID level is RAID5 or 6, secondary RAID arrays will be created if there are enough HDDs with unused capacity to maintain the same level of redundancy as the "weakest" RAID array currently present in the BTRFS volume. In most cases all RAID arrays share the same level of redundancy, expect in complex cases following Horizontal Expansion (please see the example section).
The secondary RAID arrays are called md126 (partition sdX4), md125 (partition sdX5), and so on.

The simple rules for Vertical Expansion with X-RAID/OS6:
- The RAID array is RAID1: Both HDDs must be replaced with larger capacity, one by one. The RAID array itself will be "extended".
- The weakest RAID array is RAID5: At least two HDDs must be replaced with larger capacity, one by one. A secondary RAID array will be created (as RAID1).
- The weakest RAID array is RAID6: At least four HDDs must be replaced with larger capacity, one by one (two by two is possible, though not the best idea). A secondary RAID array will be created.

There is no Expansion limit, neither in overall capacity nor in expanded capacity.

----------

- For Horizontal Expansion, on ReadyNAS OS6, in X-RAID mode, starting in RAID5 or 6, the added disk needs to be at least as big as the smallest HDD ever in the array since the time it's in RAID5 or 6 (so 3 HDDs or more), starting in RAID1, the added disk needs to be at least as big as the smallest HDD in the array since the time it's in RAID1 (so 2 HDDs).
This is because in RAID1, the primary RAID (md127) is extended in case of Vertical Expansion, so the new member needs to be at least as big as the members of the primary (and only) RAID array.
In RAID5 or 6, the primary RAID (md127) is not extended in case of Vertical Expansion. Instead, secondary RAID(s) are created. So the new member needs to be big enough to host the same size partition as all the sdX3's (members of the primary RAID array).
For basic information about Horizontal Expansion: http://kb.netgear.com/app/answers/detail/a_id/23134

It's interesting to note that a Factory Default (Volume Creation) with six HDDs in a chassis with more than six bays would base the RAID arrays on dual redundancy, where with Horizontal Expansion, the dual redundancy kicks in at the seventh HDD. When a RAID array (primary or secondary) is reshaped to RAID6, (other) secondary RAID arrays are not forced to reshape, they follow their own separate Expansion rules.

Horizontally Expanding a volume may also imply a process similar to Vertical Expansion... if the additional HDDs are big enough to allow creation of higher secondary RAID array on unused capacity.
After that, in certain situations, the level of redundancy may not be the same across all the RAID arrays. If the first secondary RAID array (md126) has a lower level of redundancy than the primary RAID array, the GUI will display the RAID level of the secondary RAID array. If there is any other secondary RAID arrays, the RAID level displayed by the GUI will still be the one from md126, regardless of the others.
This way, if the primary RAID array is a RAID5 and the secondary RAID array is a RAID1, the GUI would display RAID5, because the level of redundancy is the same.
But if the primary RAID array is a RAID6 and the secondary RAID array is a RAID5 (or a RAID1), the GUI would display RAID5 (or RAID1), because the level of redundancy is lower.

The simple rules for Horizontal Expansion with X-RAID/OS6:
- Chassis with six bays or less:

1 HDD (JBOD)     + 1 HDD -> RAID1
2 HDDs (RAID1)   + 1 HDD -> RAID5
3-5 HDDs (RAID5) + 1 HDD -> RAID5

- Chassis with more than six bays:

1 HDD (JBOD)     + 1 HDD -> RAID1
2 HDDs (RAID1)   + 1 HDD -> RAID5
3-5 HDDs (RAID5) + 1 HDD -> RAID5
6+ HDDs (RAID6)  + 1 HDD -> RAID6 (unless one-go reshape from RAID1 to RAID5)

Reshape from JBOD must go RAID1 (2 HDDs), regardless of the number additional HDD.
Reshape from RAID1 to RAID5 can be done by adding multiple HDDs at a time, they will all be added at the same time (at the condition that they are detected by X-RAID at the same time).
Reshape from RAID5 to RAID6 can also be done by adding multiple HDDs at a time.
Reshape from RAID1 to RAID6 doesn't occur regardless of the number of additional HDDs simultaneously picked-up by the system. Direct consequence is that you can actually reshape a two HDDs RAID1 to a twelve HDDs RAID5 while using X-RAID if you add ten blank HDDs at the same time (the chassis should be shutdown to ensure that X-RAID picks up all the HDDs at the same time).

It's interesting to note that Horizontally Expanding a JBOD adds redundancy (RAID1 mirror), so, despite having added a HDD, the capacity of the volume does not change.
In the same way, Horizontally Expanding a 6 HDDs RAID5 adds dual redundancy (RAID6), so, despite having added a HDD, the capacity of the volume does not change.

There is no Expansion limit, neither in overall capacity nor in expanded capacity.

[Retired_Member]

Below are some examples of X-RAID Volume Creation, Vertical Expansion, Horizontal Expansion (expansion and reshape).
I use two different notations: a detailed one, and a summarised one. Basic and moderate examples use both notations. Advanced and complex examples only use the summarised notation.
To keep things clear, the capacity is calculated as unformatted, overhead is discarded, in TB not TiB, OS/swap partitions are ignored (because it's complicated enough already).
The primary RAID array is called md127. The secondary RAID arrays are called md126, md125, etc.
Each line correspond to an action: a Factory Default (Volume Creation), the addition or replacement of a HDD.
Replacing HDDs is done one by one, after the volume completed resync. All HDDs are blank (no partitions).
In case of multiple HDD addition in one step, the addition is done while the chassis is shutdown (to ensure X-RAID picking them up all at the same time).

Summarised notation:
The arrow corresponds to the X-RAID action. If there are multiple X-RAID actions on the same line, it means that X-RAID performs the actions one after the other, after completion of previous action and resync (the user does not need to do anything).
For readability, the HDDs are sorted by capacity. In real life, you don't need to (and shouldn't) move HDDs around like that.

If any typo, I proofread it multiple times, but it can still happen, please let me know.

----------

1) Factory Default with one HDD, X-RAID will create a JBOD.
Example:

1a) Factory Default with 1x1TB. md127 us a JBOD of 1 partition of 1TB. BTRFS volume capacity is 1TB.

1 -FD-> JBOD (1)

----------

2) Vertical Expansion from JBOD doesn't exist (no redundancy implies that you can't replace a HDD without losing the volume).

----------

3) Horizontal Expansion from JBOD occurs if the additional HDD is at least as big as the HDD currently in the JBOD.
Horizontal Expansion from JBOD cannot be done multiple HDDs at a time. The first step is the reshape to RAID1 (2 HDDs) regardless of the number of additional HDD. The following steps are described in a following section.
Interestingly, in case of reshape from JBOD to RAID1, adding a single HDD does not increase the capacity of the volume, it simply adds redundancy.
Examples:

3a) Factory Default with 1x2TB. md127 is a JBOD of 1 partition of 2TB. BTRFS volume capacity is 2TB.
Add a 1TB HDD, the chassis now contains 1x1TB + 1x2TB. The additional HDD is too small for X-RAID to use. md127 remains a JBOD of 1 partition of 2TB. The 1TB HDD is unused. BTRFS volume capacity is 2TB.

2          -FD-> JBOD (2)
2 + 1 = 12 -HE-> JBOD (2) + unused (1)

3b) Factory Default with 1x1TB. md127 is a JBOD of 1 partition of 1TB. BTRFS volume capacity is 1TB.
Add a 1TB HDD, the chassis now contains 2x1TB HDDs. md127 reshapes to a RAID1 of 2 partitions of 1TB (1TB capacity, 1TB mirror). BTRFS volume capacity is 1TB.

1          -FD-> JBOD (1)
1 + 1 = 11 -HE-> RAID1 (11)

3c) Factory Default with 1x1TB. md127 is a JBOD of 1 partition of 1TB. BTRFS volume capacity is 1TB.
Add a 2TB HDD, the chassis now contains 1x1TB + 1x2TB HDDs. md127 reshapes to a RAID1 of 2 partitions of 1TB (1TB capacity, 1TB mirror). There is 1TB unused capacity on the 2TB HDD. BTRFS volume capacity is 1TB.

1          -FD-> JBOD (1)
1 + 2 = 12 -HE-> RAID1 (11) + unused (1)

3d) Simultaneous multiple HDDs Horizontal Expansion does not occur in JBOD. Next mandatory step is RAID1 (2 HDDs). Following steps explained in following examples.

1                              -FD-> JBOD (1)
1 + 1           = 11           -HE-> RAID1 (11)
1 + 11          = 111          -HE-> RAID1 (11) + unused (1)          -HE-> ...
1 + 111111      = 1111111      -HE-> RAID1 (11) + unused (11111)      -HE-> ...
1 + 1111111     = 11111111     -HE-> RAID1 (11) + unused (111111)     -HE-> ...
1 + 11111111111 = 111111111111 -HE-> RAID1 (11) + unused (1111111111) -HE-> ...

----------

 
4) Factory Default with two HDDs, X-RAID will create a RAID1. Only one data partition per HDD. The size of the data partition on both HDDs will be limited by the smallest HDD.
Examples:

4a) Factory Default with 2x1TB HDDs. md127 is a RAID1 of 2 partitions of 1TB (1TB capacity, 1TB mirror). BTRFS volume capacity is 1TB.

11 -FD-> RAID1 (11)

4b) Factory Default with 1x1TB + 1x2TB HDDs. md127 is a RAID1 of 2 partitions of 1TB (1TB capacity, 1TB mirror). There is 1TB unused capacity on the 2TB HDD. BTRFS volume capacity is 1TB.

12 -FD-> RAID1 (11) + unused (1)

---------- 

5) Vertical expansion in RAID1 occurs in-place. No secondary RAID array is created, the primary RAID array itself is grown, it's "extended".

5a) Factory Default with 2x1TB. md127 is a RAID1 of 2 partitions of 1TB (1TB capacity, 1TB mirror). BTRFS volume capacity is 1TB.
Replace a 1TB HDD by a 2TB HDD, the chassis now contains 1x1TB + 1x2TB HDDs. md127 remains a RAID1 of 2 partitions of 1TB (1TB capacity, 1TB mirror). There is 1TB unused capacity on the 2TB HDD. BTRFS volume capacity is 1TB.
Replace the second 1TB HDD by a 2TB HDD, the chassis now contains 2x2TB HDDs. md127 is extended to a RAID1 of 2 partitions of 2TB (2TB capacity, 2TB mirror). BTRFS volume capacity is 2TB.

11              -FD-> RAID1 (11)
11 - 1 + 2 = 12 -VE-> RAID1 (11) + unused (1)
12 - 1 + 2 = 22 -VE-> RAID1 (22)

----------

6) Horizontal Expansion from RAID1 implies reshape as X-RAID doesn't use RAID1 on more than two members.
Reshape from RAID1 to RAID5 occurs when adding a third HDD, if it's at least as big as the smallest HDD currently in the volume.
Examples:

6a) Factory Default with 2x1TB HDDs. md127 is a RAID1 of 2 partitions of 1TB (1TB capacity, 1TB mirror). BTRFS volume capacity is 1TB.
Add a third 1TB HDD, the chassis now contains 3x1TB HDDs. md127 reshapes to a RAID5 of 3 partitions of 1TB (2TB capacity, 1TB parity). BTRFS volume capacity is 2TB.

11           -FD-> RAID1 (11)
11 + 1 = 111 -HE-> RAID5 (111)

6b) Factory Default with 2x2TB HDDs. md127 is a RAID1 of 2 partitions of 2TB (2TB capacity, 2TB mirror). BTRFS volume capacity is 2TB.
Add a 1TB HDD, the chassis now contains 1x1TB + 2x2TB HDDs. X-RAID can't expand the volume as the additional HDD is smaller than the smallest member of the primary RAID1. The RAID arrays and BTRFS volumes capacities remain the same. The 1TB HDD is unused.

22           -FD-> RAID1 (22)
22 + 1 = 122 -HE-> RAID1 (22) + unused (1)

6c) Factory Default with 1x1TB + 1x2TB HDDs. md127 is a RAID1 of 2 partitions of 1TB (1TB capacity, 1TB mirror). There is 1TB unused capacity on the 2TB HDD. BTRFS volume capacity is 1TB.
Add a 1TB HDD, the chassis now contains 2x1TB + 1x2TB HDDs. md127 reshapes to a RAID5 of 3 partitions of 1TB (2TB capacity, 1TB parity). There is 1TB unused capacity on the 2TB HDD. BTRFS volume capacity is 2TB.

12           -FD-> RAID1 (12)
12 + 1 = 112 -HE-> RAID5 (111) + unused (1)

6d) Factory Default with 1x1TB + 1x2TB HDDs. md127 is a RAID1 of 2 partitions of 1TB (1TB capacity, 1TB mirror). There is 1TB unused capacity on the 2TB HDD. BTRFS volume capacity is 1TB.
Add a 2TB HDD, the chassis now contains 1x1TB + 2x2TB HDDs. md127 reshapes to a RAID5 of 3 partitions of 1TB (2TB capacity, 1TB parity). md126 is created as a RAID1 of 2 partitions of 1TB (1TB capacity, 1TB mirror). BTRFS volume capacity is 3TB.

12           -FD-> RAID1 (12)
12 + 2 = 122 -HE-> RAID5 (111) + RAID1 (11)

6e) Factory Default with 1x1TB + 1x2TB HDDs. md127 is a RAID1 of 2 partitions of 1TB (1TB capacity, 1TB mirror). There is 1TB unused capacity on the 2TB HDD. BTRFS volume capacity is 1TB.
Replace the 1x1TB by 1x2TB HDD, the chassis now contains 2x2TB HDDs. md127 is extended to a RAID1 of 2 partitions of 2TB (2TB capacity, 2TB mirror). BTRFS volume capacity is 2TB.
Add a 1TB, the chassis now contains 1x1TB + 2x2TB HDDs. X-RAID can't expand the volume as the additional HDD is smaller than the smallest member of the primary RAID. The RAID arrays and BTRFS volumes capacities remain the same. The 1TB HDD is unused.

12               -FD-> RAID1 (12) + unused (1)
12 - 1 + 2 = 22  -VE-> RAID1 (22)
22 + 1     = 122 -HE-> RAID1 (22) + unused (1)

6f) Factory Default with 1x1TB + 1x2TB HDDs. md127 is a RAID1 of 2 partitions of 1TB (1TB capacity, 1TB mirror). There is 1TB unused capacity on the 2TB HDD. BTRFS volume capacity is 1TB.
Replace the 1x1TB by 1x2TB HDD, the chassis now contains 2x2TB HDDs. md127 is extended to a RAID1 of 2 partitions of 2TB (2TB capacity, 2TB mirror). BTRFS volume capacity is 2TB.
Add a 2TB, the chassis now contains 3x2TB HDDs. md127 reshapes to a RAID5 of 3 partitions of 2TB (4TB capacity, 2TB parity). BTRFS volume capacity is 4TB.

12               -FD-> RAID1 (12) + unused (1)
12 - 1 + 2 = 22  -VE-> RAID1 (22)
22 + 2     = 222 -HE-> RAID5 (222)

6g) The Horizontal Expansion can be done multiple HDDs at a time. RAID1 can't directly reshape to RAID6, resulting in a reshape to RAID5 that can break the Horizontal Expansion rule applied to RAID5.

11                             -FD-> RAID1 (11)
11 + 1          = 111          -HE-> RAID5 (111)
11 + 11         = 1111         -HE-> RAID5 (1111)
11 + 1111111111 = 111111111111 -HE-> RAID5 (111111111111)

1                              -FD-> JBOD (1)
1 + 1           = 11           -HE-> RAID1 (11)
1 + 11          = 111          -HE-> RAID1 (11) + unused (1)          -HE-> RAID5 (111)
1 + 111111      = 1111111      -HE-> RAID1 (11) + unused (11111)      -HE-> RAID5 (1111111)
1 + 1111111     = 11111111     -HE-> RAID1 (11) + unused (111111)     -HE-> RAID5 (11111111)
1 + 11111111111 = 111111111111 -HE-> RAID1 (11) + unused (1111111111) -HE-> RAID5 (111111111111)

1 + 111111 = 1111111 -HE-> RAID1 (11) + unused (11111) -HE-> RAID5 (1111111)
RAID5 (1111111) + 1  -HE-> RAID6 (11111111)

[Retired_Member]

7) Factory Default with three HDDs. X-RAID will create a RAID5 using the same approach. The size of the data partition on all the HDDs used by the primary RAID array will be limited by the smallest HDD. If X-RAID detects that two HDDs have unused capacity, it will create a second data partition on these two HDDs and build a secondary RAID1 array. The size of this second partition on both HDDs will be limited by the smallest HDD of the two. The BTRFS volume will then concatenate both RAID arrays.
Examples:

7a) Factory Default with 3x1TB HDDs. md127 is a RAID5 of 3 partitions of 1TB (2TB capacity, 1TB parity). BTRFS volume capacity is 2TB.

111 -FD-> RAID5 (111)

7b) Factory Default with 2x1TB + 1x2TB HDDs. md127 is a RAID5 of 3 partitions of 1TB (2TB capacity, 1TB parity). There is 1TB unused capacity on the 2TB HDD. BTRFS volume capacity is 2TB.

112 -FD-> RAID5 (111) + unused (1)

7c) Factory Default with 1x1TB + 2x2TB HDDs. md127 is a RAID5 of 3 partitions of 1TB (2TB capacity, 1TB parity). md126 is a RAID1 of 2 partitions of 1TB (1TB capacity, 1TB mirror). BTRFS volume capacity is 3TB.

122 -FD-> RAID5 (111) + RAID1 (11)

7d) Factory Default with 1x1TB + 1x2TB + 1x3TB HDDs. md127 is a RAID5 of 3 partitions of 1TB (2TB capacity, 1TB parity). md126 is a RAID1 of 2 partitions of 1TB (1TB capacity, 1TB mirror). There is 1TB unused capacity on the 3TB HDD. BTRFS volume capacity is 3TB.

123 -FD-> RAID5 (111) + RAID1 (11) + unused (1)

7bis) Factory Default with four or five HDDs, the same thing will happen.
Examples:

7e) Factory Default with 4x1TB HDDs. md127 is a RAID5 of 4 partitions of 1TB (3TB capacity, 1TB parity). BTRFS volume capacity is 3TB.

1111 -FD-> RAID5 (1111)

7f) Factory Default with 5x1TB HDDs. md127 is a RAID5 of 5 partitions of 1TB (4TB capacity, 1TB parity). BTRFS volume capacity is 4TB.

11111 -FD-> RAID5 (11111)

7g) Factory Default with 2x1TB + 2x2TB + 1x3TB HDDs. md127 is a RAID5 of 5 partitions of 1TB (4TB capacity, 1TB parity). md126 is a RAID5 of 3 partitions of 1TB (2TB capacity, 1TB parity). There is 1TB unused capacity on the 3TB HDD. BTRFS volume capacity is 6TB.

11223 -FD-> RAID5 (11111) + RAID5 (111) + unused (1)

7h) Factory Default with 2x1TB + 1x2TB + 2x3TB HDDs. md127 is a RAID5 of 5 partitions of 1TB (4TB capacity, 1TB parity). md126 is a RAID5 of 3 partitions of 1TB (2TB capacity, 1TB parity). md125 is a RAID1 of 2 partitions of 1TB (1TB capacity, 1TB mirror). BTRFS volume capacity is 7TB.

11233 -FD-> RAID5 (11111) + RAID5 (111) + RAID1 (11)

---------- 

😎 Vertical expansion in RAID5 creates secondary RAID arrays if possible.
Examples:

8a) Factory Default with 3x1TB. md127 is a RAID5 of 3 partitions of 1TB (2TB capacity, 1TB parity). BTRFS volume capacity is 2TB.
Replace a 1TB HDD by a 2TB HDD, the chassis now contains 2x1TB + 1x2TB HDDs. md127 remains a RAID5 of 3 partitions of 1TB (2TB capacity, 1TB parity). There is 1TB unused capacity on the 2TB HDD. BTRFS volume capacity is 2TB.
Replace a second 1TB HDD by a 2TB HDD, the chassis now contains 1x1TB + 2x2TB HDDs. md127 remains a RAID5 of 3 partitions of 1TB (2TB capacity, 1TB parity). md126 is created as a RAID1 of 2 partitions of 1TB (1TB capacity, 1TB mirror). BTRFS volume capacity is 3TB.
Replace the last 1TB HDD by a 2TB HDD, the chassis now contains 3x2TB HDDs. md127 remains a RAID5 of 3 partitions of 1TB (2TB capacity, 1TB parity). md126 reshapes to a RAID5 of 3 partitions of 1TB (2TB capacity, 1TB parity). BTRFS volume capacity is 4TB.

111               -FD-> RAID5 (111)
111 - 1 + 2 = 112 -VE-> RAID5 (111) + unused (1)
112 - 1 + 2 = 122 -VE-> RAID5 (111) + RAID1 (1)
122 - 1 + 2 = 222 -VE-> RAID5 (111) + RAID5 (111)

8b) RAID5 arrays are not vertically "extended" as RAID1 arrays would.

112               -FD-> RAID5 (111) + unused (1)
112 - 1 + 3 = 123 -HE-> RAID5 (111) + RAID1 (11)  + unused (1)
123 - 1 + 3 = 223 -HE-> RAID5 (111) + RAID5 (111) + RAID (11)

----------

9) Horizontal Expansion in RAID5 occurs if the additional HDD is at least as big as the smallest HDD that was ever in the array since it's a RAID5.
Examples:

9a) Factory Default with 1x1TB + 2x2TB HDDs. md127 is a RAID5 of 3 partitions of 1TB (2TB capacity, 1TB parity). md126 is a RAID1 of 2 partitions of 1TB (1TB capacity, 1TB mirror). BTRFS volume capacity is 3TB.
Add a 1TB HDD, the chassis now contains 2x1TB + 2x2TB HDDs. md127 expands to a RAID5 of 4 partitions of 1TB (3TB capacity, 1TB parity). md126 remains a RAID1 of 2 partitions of 1TB (1TB capacity, 1TB mirror). BTRFS volume capacity is 4TB.

122            -FD-> RAID5 (111)  + RAID1 (11)
122 + 1 = 1122 -HE-> RAID5 (1111) + RAID1 (11)

9b) Factory Default with 2x1TB + 1x2TB HDDs. md127 is a RAID5 of 3 partitions of 1TB (2TB capacity, 1TB parity). There is 1TB unused capacity on the 2TB HDD. BTRFS volume capacity is 2TB.
Add a 2TB HDD, the chassis now contains 2x1TB + 2x2TB HDDs. md127 expands to a RAID5 of 4 partitions of 1TB (3TB capacity, 1TB parity). md126 is created as a RAID1 of 2 partitions of 1TB (1TB capacity, 1TB mirror). BTRFS volume capacity is 4TB.
Add a 1TB HDD, the chassis now contains 3x1TB + 2x2TB HDDs. md127 expands to a RAID5 of 5 partitions of 1TB (4TB capacity, 1TB parity). md126 remains a RAID1 of 2 partitions of 1TB (1TB capacity, 1TB mirror). BTRFS volume capacity is 5TB.
Add a 2TB HDD, the chassis now contains 3x1TB + 3x2TB HDDs. md127 expands to a RAID5 of 6 partitions of 1TB (5TB capacity, 1TB parity). md126 reshapes to a RAID5 of 3 partitions of 1TB (2TB capacity, 1TB parity). BTRFS volume capacity is 7TB.

112                -FD-> RAID5 (111)    + unused (1)
112   + 2 = 1122   -HE-> RAID5 (1111)   + RAID1 (11)
1122  + 1 = 11122  -HE-> RAID5 (11111)  + RAID1 (11)
11122 + 2 = 111222 -HE-> RAID5 (111111) + RAID5 (111)

9c) As RAID5 arrays are not vertically "extended" as RAID1 arrays would, the additional HDD only needs to be at least as big as the size of the smallest HDD ever in the volume since it's at least a RAID5 (determining the size of the primary RAID array partition)

111                -FD-> RAID5 (111)
111 - 1 + 2 = 112  -VE-> RAID5 (111)  + unused (1)
112 - 1 + 2 = 122  -VE-> RAID5 (111)  + RAID1 (11)
122 - 1 + 2 = 222  -VE-> RAID5 (111)  + RAID5 (111)
222 + 1     = 2221 -HE-> RAID5 (1111) + RAID5 (111)

9d) The Horizontal Expansion on RAID5 can be done multiple HDDs at a time.

111                -FD-> RAID5 (111)
111 + 1   = 1111   -HE-> RAID5 (1111)
111 + 111 = 111111 -HE-> RAID5 (111111)

---------- 

10) Reshape from RAID5 to RAID6 occurs when adding a sixth HDD to a chassis with more than six bays, if the additional HDD is at least as big as the smallest HDD ever in the array since the time it's in RAID5 or 6 (so 3 HDDs or more).
Interestingly, in case of reshape from RAID5 to RAID6; adding a single HDD does not increase the capacity of the volume, it simply adds redundancy.
Reshaping one RAID array to RAID6 doesn't force any other reshape. The other secondary RAID arrays follow their own Expansion rules.
If there is a mismatch of RAID level between the primary RAID array and secondary RAID arrays, the GUI will display the RAID level of the first secondary RAID array, regardless of any other array's RAID level.
Examples:

10a) Factory Default with 5x1TB in a twelve bays unit. md127 is a RAID5 of 5 partitions of 1TB (4TB capacity, 1TB parity). BTRFS volume capacity is 4TB.
Add a 1TB HDD, the chassis now contains 6x1TB. md127 expands to a RAID5 of 6 partitions of 1TB (5TB capacity, 1TB parity). BTRFS volume capacity is 5TB.
Add a 1TB HDD, the chassis now contains 7x1TB. md127 reshapes to a RAID6 of 7 partitions of 1TB (5TB capacity, 2TB parity). BTRFS volume capacity is 5TB.

11111                -FD-> RAID5 (11111)
11111  + 1 = 111111  -HE-> RAID5 (111111)
111111 + 1 = 1111111 -HE-> RAID6 (1111111)

10b) Factory Default with 3x1TB + 2x2TB in a twelve bays unit. md127 is a RAID5 of 5 partitions of 1TB (4TB capacity, 1TB parity). md126 is a RAID1 of 2 partitions of 1TB (1TB capacity, 1TB mirror). BTRFS volume capacity is 5TB.
Add a 1TB HDD, the chassis now contains 4x1TB + 2x2TB. md127 expands to a RAID5 of 6 partitions of 1TB (5TB capacity, 1TB parity).md126 remains a RAID1 of 2 partitions of 1TB (1TB capacity, 1TB mirror). BTRFS volume capacity is 6TB.
Add a 1TB HDD, the chassis now contains 5x1TB + 2x2TB. md127 reshapes to a RAID6 of 7 partitions of 1TB (5TB capacity, 2TB parity).md126 remains a RAID1 of 2 partitions of 1TB (1TB capacity, 1TB mirror). BTRFS volume capacity is 6TB.

11122                -FD-> RAID5 (11111)  + RAID1 (11)
11122  + 1 = 111122  -HE-> RAID5 (111111) + RAID1 (11)
111122 + 1 = 1111122 -HE-> RAID6 (111111) + RAID1 (11)

10c) Factory Default with 3x1TB + 2x2TB in a twelve bays unit. md127 is a RAID5 of 5 partitions of 1TB (4TB capacity, 1TB parity). md126 is a RAID1 of 2 partitions of 1TB (1TB capacity, 1TB mirror). BTRFS volume capacity is 5TB.
Add a 2TB HDD, the chassis now contains 3x1TB + 3x2TB. md127 expands to a RAID5 of 6 partitions of 1TB (5TB capacity, 1TB parity).md126 reshapes to a RAID5 of 3 partitions of 1TB (2TB capacity, 1TB parity). BTRFS volume capacity is 7TB.
Add a 2TB HDD, the chassis now contains 3x1TB + 4x2TB. md127 reshapes to a RAID6 of 7 partitions of 1TB (5TB capacity, 2TB parity).md126 expands to a RAID5 of 4 partitions of 1TB (3TB capacity, 1TB parity). BTRFS volume capacity is 8TB.

11122                -FD-> RAID5 (11111)  + RAID1 (11)
11122  + 2 = 111222  -HE-> RAID5 (111111) + RAID5 (111)
111122 + 2 = 1112222 -HE-> RAID6 (111111) + RAID5 (1111)

10d) Reshape from RAID5 to RAID6 can be done multiple HDDs at a time.

111                  -FD-> RAID5 (111)
111 + 1111 = 1111111 -HE-> RAID6 (1111111)

10e)

111                            -FD-> RAID5 (111)
111 + 111111111 = 111111111111 -HE-> RAID6 (111111111111)

10f) Each RAID array follow its own Expansion rules. A reshape occurring in one doesn't force the reshape of another. This can result in a mismatch in RAID level, but most importantly in level of redundancy. If the level of redundancy of the first secondary RAID array is lower than the level of redundancy of the primary RAID array, then the GUI will display the RAID level of the secondary RAID array (don't confuse level of redundancy and RAID level).

11122                       -FD-> RAID5 (11111)    + RAID1 (11)                    : GUI "RAID5"
11122    + 1     = 111122   -HE-> RAID5 (111111)   + RAID1 (11)                    : GUI "RAID5"
111122   + 1     = 1111122  -HE-> RAID6 (1111111)  + RAID1 (11)                    : GUI "RAID1"
1111122  + 1     = 11111122 -HE-> RAID6 (11111111) + RAID1 (11)                    : GUI "RAID1"
11111122 - 1 + 2 = 11111222 -VE-> RAID6 (11111111) + RAID5 (111)                   : GUI "RAID5"
11111222 - 1 + 2 = 11112222 -VE-> RAID6 (11111111) + RAID5 (1111)                  : GUI "RAID5"
11112222 - 1 + 2 = 11122222 -VE-> RAID6 (11111111) + RAID5 (11111)                 : GUI "RAID5"
11122222 - 1 + 2 = 11222222 -VE-> RAID6 (11111111) + RAID5 (111111)                : GUI "RAID5"
11222222 - 1 + 2 = 12222222 -VE-> RAID6 (11111111) + RAID6 (1111111)               : GUI "RAID6"
12222222 - 2 + 3 = 12222223 -VE-> RAID6 (11111111) + RAID6 (1111111) + unused (1)  : GUI "RAID6"
12222223 - 2 + 3 = 12222233 -VE-> RAID6 (11111111) + RAID6 (1111111) + unused (11) : GUI "RAID6"
12222233 - 2 + 3 = 12222333 -VE-> RAID6 (11111111) + RAID6 (1111111) + unused (111): GUI "RAID6"
12222333 - 2 + 3 = 12223333 -VE-> RAID6 (11111111) + RAID6 (1111111) + RAID6 (1111): GUI "RAID6"

10g)

11122                -FD-> RAID5 (11111)   + RAID1 (11)                : GUI "RAID5"
11122  + 3 = 111223  -HE-> RAID5 (111111)  + RAID5 (111)  + unused (1) : GUI "RAID5"
111223 + 3 = 1112233 -HE-> RAID6 (1111111) + RAID5 (1111) + unused (11) -VE-> RAID6 (1111111) + RAID5 (1111) + RAID1 (11): GUI "RAID5"

10h)

11122                  -FD-> RAID5 (11111)   + RAID1 (11)                : GUI "RAID5"
11122   + 2 = 111222   -HE-> RAID5 (111111)  + RAID5 (111)               : GUI "RAID5"
111222  + 3 = 1112223  -HE-> RAID6 (1111111) + RAID5 (1111)  + unused (1): GUI "RAID5"
1112223 + 3 = 11122233 -HE-> RAID6 (1111111) + RAID5 (11111) + RAID1 (11): GUI "RAID5"

[Retired_Member]

11) Factory Default with six HDDs will depend on the number of bays.
11-1) If the unit has 6 bays or less and you Factory Default with six HDDs, it does the same as point 8 (single redundancy), so RAID5.
11-2) If the unit has more than 6 bays and you Factory Default with six HDDs or more, X-RAID will base the RAID arrays on dual redundancy, so RAID6.
Examples:

11a) Factory Default with 3x1TB + 3x2TB HDDs in a six bays unit. md127 is a RAID5 of 6 partitions of 1TB (5TB capacity, 1TB parity). md126 is a RAID5 of 3 partitions of 1TB (2TB capacity, 1TB parity). BTRFS volume capacity is 7TB.

(six bays units) 111222 -FD-> RAID5 (111111) + RAID5 (111)

11b) Factory Default with 3x1TB + 3x2TB HDDs in a twelve bays unit. md127 is a RAID6 of 6 partitions of 1TB (4TB capacity, 2TB parity). There is 1TB unused capacity on the three 2TB HDDs. BTRFS volume capacity is 4TB.

(twelve bays units) 111222 -FD-> RAID6 (111111) + unused (111)

11c) Factory Default with 2x1TB + 4x2TB HDDs in a twelve bays unit. md127 is a RAID6 of 6 partitions of 1TB (4TB capacity, 2TB parity). md126 is a RAID6 of 4 partitions of 1TB (2TB capacity, 2TB parity). BTRFS volume capacity is 6TB.

(twelve bays units) 112222 -FD-> RAID6 (111111) + RAID6 (1111)

11bis) If you Factory Default with seven HDDs or more, the same thing will happen, two devices redundancy so RAID6.
Examples:

11d) Factory Default with 12x1TB HDDs. md127 is a RAID6 of 12 partitions of 1TB (10TB capacity, 2TB parity), BTRFS volume capacity is 10TB.

111111111111 -FD-> RAID6 (111111111111)

11e) Factory Default with 10x1TB + 2x2TB HDDs. md127 is a RAID6 of 12 partitions of 1TB (10TB capacity, 2TB parity). There is 1TB unused capacity on both 2TB HDDs. BTRFS volume capacity is 10TB.

111111111122 -FD-> RAID6 (111111111111) + unused (11)

11f) Factory Default with 9x1TB + 3x2TB HDDs. md127 is a RAID6 of 12 partitions of 1TB (10TB capacity, 2TB parity). There is 1TB unused capacity on the three 2TB HDDs. BTRFS volume capacity is 10TB.

111111111222 -FD-> RAID6 (111111111111) + unused (111)

11g) Factory Default with 8x1TB + 4x2TB HDDs. md127 is a RAID6 of 12 partitions of 1TB (10TB capacity, 2TB parity). md126 is a RAID6 of 4 partitions of 1TB (2TB capacity, 2TB parity). BTRFS volume capacity is 12TB.

111111112222 -FD-> RAID6 (111111111111) + RAID6 (1111)

----------

12) Vertical Expansion in RAID6 occurs when there are enough members to create a secondary RAID6, so four members.
Example:

12a) Factory Default with 12x1TB. md127 is a RAID6 of 12 partitions of 1TB (10TB capacity, 2TB parity). BTRFS volume capacity is 10TB.
Replace a 1TB HDD with a 2TB HDD, the chassis now contains 11x1TB + 1x2TB HDDs. md127 remains a RAID6 of 12 partitions of 1TB (10TB capacity, 2TB parity). There is 1TB unused capacity on the 2TB HDD. BTRFS volume capacity is 10TB.
Replace a second 1TB HDD with a 2TB HDD, the chassis now contains 10x1TB + 2x2TB HDDs. md127 remains a RAID6 of 12 partitions of 1TB (10TB capacity, 2TB parity). There is 1TB unused capacity on both 2TB HDDs. BTRFS volume capacity is 10TB.
Replace a third 1TB HDD with a 2TB HDD, the chassis now contains 9x1TB + 3x2TB HDDs. md127 remains a RAID6 of 12 partitions of 1TB (10TB capacity, 2TB parity). There is 1TB unused capacity on the three 2TB HDDs. BTRFS volume capacity is 10TB.
Replace a fourth 1TB HDD with a 2TB HDD, the chassis now contains 8x1TB + 4x2TB HDDs. md127 remains a RAID6 of 12 partitions of 1TB (10TB capacity, 2TB parity). md126 is created as a RAID6 of 4 partitions of 1TB (2TB capacity, 2TB parity). BTRFS volume capacity is 12TB.

111111111111                        -FD-> RAID6 (111111111111)
111111111111 - 1 + 2 = 111111111112 -VE-> RAID6 (111111111111) + unused (1)
111111111112 - 1 + 2 = 111111111122 -VE-> RAID6 (111111111111) + unused (11)
111111111122 - 1 + 2 = 111111111222 -VE-> RAID6 (111111111111) + unused (111)
111111111222 - 1 + 2 = 111111112222 -VE-> RAID6 (111111111111) + RAID6 (1111)

12b) The Vertical Expansion is possible only if 4 HDDs are added, even if the primary RAID array was RAID5 before being RAID6. More specifically, as seen in 10f, 10g and 10h, this four additional HDDs rule is only true if all RAID arrays are RAID6, otherwise, it's the single redundancy rule that applies (two additional HDDs).

11111                     -FD-> RAID5 (11111)
11111   + 1     = 111111  -HE-> RAID5 (111111)
111111  + 1     = 1111111 -HE-> RAID6 (1111111)
1111111 - 1 + 2 = 1111112 -VE-> RAID6 (1111111) + unused (1)
1111112 - 1 + 2 = 1111122 -VE-> RAID6 (1111111) + unused (11)
1111122 - 1 + 2 = 1111222 -VE-> RAID6 (1111111) + unused (111)
1111222 - 1 + 2 = 1112222 -VE-> RAID6 (1111111) + RAID6 (1111)

----------

13) Horizontal Expansion in RAID6 occurs if the additional HDD is at least as big as the smallest HDD that was ever in the array since it's a RAID5 or 6.
The Horizontal Expansion in RAID6 can be done multiple HDDs at a time.
Example:

13a) Factory Default with 7x1TB in a twelve bays unit. md127 is a RAID6 of 7 partitions of 1TB (5TB capacity, 2TB parity). BTRFS volume capacity is 5TB.
Add a 1TB HDD, the chassis now contains 8x1TB. md127 expands to a RAID6 of 8 partitions of 1TB (6TB capacity, 2TB parity). BTRFS volume capacity is 6TB.
Add a 1TB HDD, the chassis now contains 9x1TB. md127 expands to a RAID6 of 9 partitions of 1TB (7TB capacity, 2TB parity). BTRFS volume capacity is 7TB.

1111111                        -FD-> RAID6 (1111111)
1111111 + 1     = 11111111     -HE-> RAID6 (11111111)
1111111 + 11111 = 111111111111 -HE-> RAID6 (111111111111)

13b) Factory Default with 4x1TB + 2x2TB in a twelve bays unit. md127 is a RAID6 of 6 partitions of 1TB (4TB capacity, 2TB parity). There is 1TB unused capacity on both 2TB HDDs. BTRFS volume capacity is 4TB.
Add a 2TB HDD, the chassis now contains 4x1TB + 3x2TB. md127 expands to a RAID6 of 7 partitions of 1TB (5TB capacity, 2TB parity). There is 1TB unused capacity on the three 2TB HDDs. BTRFS volume capacity is 5TB.
Add a 2TB HDD, the chassis now contains 4x1TB + 4x2TB. md127 expands to a RAID6 of 8 partitions of 1TB (6TB capacity, 2TB parity). md126 is created as a RAID6 of 4 partitions of 1TB (2TB capacity, 2TB parity). BTRFS volume capacity is 8TB.

111122                 -FD-> RAID6 (111111)   + unused (11)
111122  + 2 = 1111222  -HE-> RAID6 (1111111)  + unused (111)
1111222 + 2 = 11112222 -HE-> RAID6 (11111111) + RAID6 (1111)