Re: Pro 6 running OS6: Array doesn't automatically expand

fastfwd · ‎2023-10-16

I'm trying to expand one of my old 6-bay NAS boxes by replacing four of its drives with larger ones:

Started with a Pro 6 running OS 6.10.9, with six 4TB drives in XRAID RAID6 for a "16TB" (really 14.54TB) array.
Replaced one 4TB drive with an 8TB drive. Waited for the resync to complete.
Repeated that replacement three more times, so I ended up with four 8TB drives and two 4TB drives.
I expected that the array would automatically resize to something near 24TB after the fourth drive had been replaced, but it didn't.
I restarted the NAS a couple of times, hoping that that would trigger an automatic resize, but it didn't.

Here's the current status. As you can see, it's as though it still has six 4TB drives:

# btrfs filesystem show
Label: '33ea3f13:root' uuid: dad60fbb-7971-46be-8e32-f2063391a033
Total devices 1 FS bytes used 1.76GiB
devid 1 size 4.00GiB used 4.00GiB path /dev/md0

Label: '33ea3f13:data' uuid: 159d011a-173f-4597-b054-715f06650ab3
Total devices 1 FS bytes used 11.53TiB
devid 1 size 14.54TiB used 13.54TiB path /dev/mapper/data-0

# cat /proc/mdstat
Personalities : [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md127 : active raid6 sda3[9] sdf3[8] sde3[7] sdd3[11] sdc3[6] sdb3[10]
15608675328 blocks super 1.2 level 6, 64k chunk, algorithm 2 [6/6] [UUUUUU]

md1 : active raid10 sde2[0] sdd2[5] sdc2[4] sdb2[3] sda2[2] sdf2[1]
1566720 blocks super 1.2 512K chunks 2 near-copies [6/6] [UUUUUU]

md0 : active raid1 sda1[9] sdf1[8] sde1[7] sdd1[11] sdc1[6] sdb1[10]
4190208 blocks super 1.2 [6/6] [UUUUUU]

unused devices: <none>

# mdadm --detail /dev/md127
/dev/md127:
Version : 1.2
Creation Time : Tue Jan 3 20:54:48 2017
Raid Level : raid6
Array Size : 15608675328 (14885.59 GiB 15983.28 GB)
Used Dev Size : 3902168832 (3721.40 GiB 3995.82 GB)
Raid Devices : 6
Total Devices : 6
Persistence : Superblock is persistent

Update Time : Mon Oct 16 13:14:39 2023
State : clean
Active Devices : 6
Working Devices : 6
Failed Devices : 0
Spare Devices : 0

Layout : left-symmetric
Chunk Size : 64K

Consistency Policy : unknown

Name : 33ea3f13:data-0 (local to host 33ea3f13)
UUID : 4d5f86c3:c41a1bc9:0efddce7:58f1c455
Events : 24063

Number Major Minor RaidDevice State
9 8 3 0 active sync /dev/sda3
10 8 19 1 active sync /dev/sdb3
6 8 35 2 active sync /dev/sdc3
11 8 51 3 active sync /dev/sdd3
7 8 67 4 active sync /dev/sde3
8 8 83 5 active sync /dev/sdf3

What's the best way to expand the array to its maximum size?

StephenB · ‎2023-10-26

@fastfwd wrote:

Yes, the volume is encrypted

That explains it. Encrypted volumes aren't expandable.

View solution in original post

StephenB · ‎2023-10-17

@fastfwd wrote:

I'm trying to expand one of my old 6-bay NAS boxes by replacing four of its drives with larger ones:

Started with a Pro 6 running OS 6.10.9, with six 4TB drives in XRAID RAID6 for a "16TB" (really 14.54TB) array.

Replaced one 4TB drive with an 8TB drive. Waited for the resync to complete.

Repeated that replacement three more times, so I ended up with four 8TB drives and two 4TB drives.

I expected that the array would automatically resize to something near 24TB after the fourth drive had been replaced, but it didn't.

I restarted the NAS a couple of times, hoping that that would trigger an automatic resize, but it didn't.

Here's the current status. As you can see, it's as though it still has six 4TB drives:
# btrfs filesystem show
Label: '33ea3f13:root' uuid: dad60fbb-7971-46be-8e32-f2063391a033
Total devices 1 FS bytes used 1.76GiB
devid 1 size 4.00GiB used 4.00GiB path /dev/md0

Label: '33ea3f13:data' uuid: 159d011a-173f-4597-b054-715f06650ab3
Total devices 1 FS bytes used 11.53TiB
devid 1 size 14.54TiB used 13.54TiB path /dev/mapper/data-0
# cat /proc/mdstat
Personalities : [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md127 : active raid6 sda3[9] sdf3[8] sde3[7] sdd3[11] sdc3[6] sdb3[10]
15608675328 blocks super 1.2 level 6, 64k chunk, algorithm 2 [6/6] [UUUUUU]

md1 : active raid10 sde2[0] sdd2[5] sdc2[4] sdb2[3] sda2[2] sdf2[1]
1566720 blocks super 1.2 512K chunks 2 near-copies [6/6] [UUUUUU]

md0 : active raid1 sda1[9] sdf1[8] sde1[7] sdd1[11] sdc1[6] sdb1[10]
4190208 blocks super 1.2 [6/6] [UUUUUU]

unused devices: <none>
# mdadm --detail /dev/md127
/dev/md127:
Version : 1.2
Creation Time : Tue Jan 3 20:54:48 2017
Raid Level : raid6
Array Size : 15608675328 (14885.59 GiB 15983.28 GB)
Used Dev Size : 3902168832 (3721.40 GiB 3995.82 GB)
Raid Devices : 6
Total Devices : 6
Persistence : Superblock is persistent

Update Time : Mon Oct 16 13:14:39 2023
State : clean
Active Devices : 6
Working Devices : 6
Failed Devices : 0
Spare Devices : 0

Layout : left-symmetric
Chunk Size : 64K

Consistency Policy : unknown

Name : 33ea3f13:data-0 (local to host 33ea3f13)
UUID : 4d5f86c3:c41a1bc9:0efddce7:58f1c455
Events : 24063

Number Major Minor RaidDevice State
9 8 3 0 active sync /dev/sda3
10 8 19 1 active sync /dev/sdb3
6 8 35 2 active sync /dev/sdc3
11 8 51 3 active sync /dev/sdd3
7 8 67 4 active sync /dev/sde3
8 8 83 5 active sync /dev/sdf3
What's the best way to expand the array to its maximum size?

If you haven't rebooted the NAS, then maybe try doing that.

If that doesn't help, then look to see if there is fourth partition on all of the 8 TB drives (sdX4), and let us know if they are all there.

Sandshark · ‎2023-10-17

It sounds like XRAID isn't enabled. In order to get your NAS at RAID6, you likely disabled XRAID. Did you re-enable it? Since the NAS is not recognizing your volume as being "expanded", you should still be able to do so.

If it is enabled, then there are some commands you can issue via SSH that may kick-start the process. But you should not try them until you've verified XRAID is on. Once a second layer is created and you have an "expanded" volume, you'll no longer be able to turn on XRAID. So I can give you those commands once you've verified XRAID is enabled.

StephenB · ‎2023-10-17

@Sandshark wrote:

It sounds like XRAID isn't enabled.

Yeah, makes sense to check that first.

It's easy to do - if you see a green stripe on the XRAID control on the volumes page, then it is enabled.

fastfwd · ‎2023-10-17

Thanks for the help so far.

I've rebooted the NAS a couple times, but that has had no effect.

XRAID is definitely enabled:

StephenB · ‎2023-10-17

First see if the sdX4 partitions have been created on all four of the 8 TB drives.

fastfwd · ‎2023-10-17

There is no fourth partition:

# ll /dev/sd*
brw-rw---- 1 root disk 8, 0 Oct 16 08:03 /dev/sda
brw-rw---- 1 root disk 8, 1 Oct 16 08:03 /dev/sda1
brw-rw---- 1 root disk 8, 2 Oct 16 08:03 /dev/sda2
brw-rw---- 1 root disk 8, 3 Oct 16 08:03 /dev/sda3
brw-rw---- 1 root disk 8, 16 Oct 16 08:03 /dev/sdb
brw-rw---- 1 root disk 8, 17 Oct 16 08:03 /dev/sdb1
brw-rw---- 1 root disk 8, 18 Oct 16 08:03 /dev/sdb2
brw-rw---- 1 root disk 8, 19 Oct 16 08:03 /dev/sdb3
brw-rw---- 1 root disk 8, 32 Oct 16 08:03 /dev/sdc
brw-rw---- 1 root disk 8, 33 Oct 16 08:03 /dev/sdc1
brw-rw---- 1 root disk 8, 34 Oct 16 08:03 /dev/sdc2
brw-rw---- 1 root disk 8, 35 Oct 16 08:03 /dev/sdc3
brw-rw---- 1 root disk 8, 48 Oct 16 08:03 /dev/sdd
brw-rw---- 1 root disk 8, 49 Oct 16 08:03 /dev/sdd1
brw-rw---- 1 root disk 8, 50 Oct 16 08:03 /dev/sdd2
brw-rw---- 1 root disk 8, 51 Oct 16 08:03 /dev/sdd3
brw-rw---- 1 root disk 8, 64 Oct 16 08:03 /dev/sde
brw-rw---- 1 root disk 8, 65 Oct 16 08:03 /dev/sde1
brw-rw---- 1 root disk 8, 66 Oct 16 08:03 /dev/sde2
brw-rw---- 1 root disk 8, 67 Oct 16 08:03 /dev/sde3
brw-rw---- 1 root disk 8, 80 Oct 16 08:03 /dev/sdf
brw-rw---- 1 root disk 8, 81 Oct 16 08:03 /dev/sdf1
brw-rw---- 1 root disk 8, 82 Oct 16 08:03 /dev/sdf2
brw-rw---- 1 root disk 8, 83 Oct 16 08:03 /dev/sdf3
brw-rw---- 1 root disk 8, 112 Oct 16 08:03 /dev/sdh
brw-rw---- 1 root disk 8, 113 Oct 16 08:03 /dev/sdh1
brw-rw---- 1 root disk 8, 128 Oct 16 08:03 /dev/sdi
brw-rw---- 1 root disk 8, 129 Oct 16 08:03 /dev/sdi1
brw-rw---- 1 root disk 8, 144 Oct 16 08:03 /dev/sdj
brw-rw---- 1 root disk 8, 145 Oct 16 08:03 /dev/sdj1
brw-rw---- 1 root disk 8, 146 Oct 16 08:04 /dev/sdj2
brw-rw---- 1 root disk 8, 160 Oct 16 08:03 /dev/sdk
brw-rw---- 1 root disk 8, 161 Oct 16 08:03 /dev/sdk1

StephenB · ‎2023-10-17

@fastfwd wrote:

There is no fourth partition:

Which means that the expansion process hasn't kicked off at all, since that is the first step.

Sandshark · ‎2023-10-18

From SSH, try entering the command volume_util -e auto.

fastfwd · ‎2023-10-24

@Sandshark Apologies for the delay. Before I did anything else, I thought it would be prudent to run a btrfs check and make a backup of the most critical files; the check took 4 days and the backup took another day.

But after all that, I did as you suggested and ran "volume_util -e auto".

Unfortunately, the command had no effect. I rebooted a couple times after executing that command, but the reboots didn't trigger any activity either.

There are no new sd*4 devices, no expansion of the mdraid array, no expansion of the btrfs filesystem. The output of all four commands I posted earlier is unchanged.

What next?

Sandshark · ‎2023-10-25

I'm not sure what's going on. I just took a closer look at the data you've provided, and don't know where these are coming from and whether or not they might be interfering with the expansion:

brw-rw---- 1 root disk 8, 112 Oct 16 08:03 /dev/sdh
brw-rw---- 1 root disk 8, 113 Oct 16 08:03 /dev/sdh1
brw-rw---- 1 root disk 8, 128 Oct 16 08:03 /dev/sdi
brw-rw---- 1 root disk 8, 129 Oct 16 08:03 /dev/sdi1
brw-rw---- 1 root disk 8, 144 Oct 16 08:03 /dev/sdj
brw-rw---- 1 root disk 8, 145 Oct 16 08:03 /dev/sdj1
brw-rw---- 1 root disk 8, 146 Oct 16 08:04 /dev/sdj2
brw-rw---- 1 root disk 8, 160 Oct 16 08:03 /dev/sdk
brw-rw---- 1 root disk 8, 161 Oct 16 08:03 /dev/sdk1

Maybe @StephenB has some thnghts.

StephenB · ‎2023-10-25

@Sandshark wrote:

I'm not sure what's going on. I just took a closer look at the data you've provided, and don't know where these are coming from and whether or not they might be interfering with the expansion:

Maybe @StephenB has some thnghts.

I was assuming those were USB drives - @fastfwd is that the case?

As far as the main problem goes, the most direct path would be to back up the data and do a factory default.

fastfwd · ‎2023-10-25

Really appreciate the help.

Yes, sdh through sdk are USB drives.

The NAS is backed up, so I can do a factory default. But it'll take a few days to verify the backup, then maybe up to a week to restore it.

I can live with the loss of availability, but I won't really enjoy it. Is there anything I can try -- even something a little risky -- before I give up and resign myself to doing the factory default?

Oh, and I just thought of something that might be relevant: I'm not sure, but I think I may have created my "16TB" 6x4TB RAID6 array while the NAS was running OS4, and I don't remember whether it was necessary to factory-default in order to perform the OS4-to-OS6 upgrade.

I vaguely remember that under OS4 an array couldn't be expanded beyond 16TB. If I did create the array under OS4 -- and didn't need to factory-default when I upgraded to OS6 -- would the 16TB expansion limit have carried over to OS6?

StephenB · ‎2023-10-25

@fastfwd wrote:

I think I may have created my "16TB" 6x4TB RAID6 array while the NAS was running OS4, and I don't remember whether it was necessary to factory-default in order to perform the OS4-to-OS6 upgrade.

Conversion to OS-6 requires a factory default, so that isn't the issue.

Is it possible that the volume is encrypted?

fastfwd · ‎2023-10-25

Yes, the volume is encrypted (which I think may be why it's at /dev/mapper/data-0 rather than /dev/md127).

StephenB · ‎2023-10-26

@fastfwd wrote:

Yes, the volume is encrypted

That explains it. Encrypted volumes aren't expandable.

fastfwd · ‎2023-10-26

Ah. Ok, factory default it is.

Thanks very much, @StephenB and @Sandshark .

Sandshark · ‎2023-10-26

When you do so, you may want to re-think encryption. IMHO, given the encryption key has to be kept close to the NAS for re-boots, it serves little purpose.

fastfwd · ‎2023-10-26

I enable encryption in order to make it easy to safely dispose of old drives, so the need for an always-available key isn't a problem for me. But now that I know encryption prevents expansion, I may reconsider. Thanks.

Sandshark · ‎2023-10-27

What I do is have one share for VeraCrypt containers that contain my sensitive data. One drawback that may affect you is that only one computer can access a container at any given time. If you decide to do that, do turn off "strict sync" for that share, as it slows writes to the VeraCrypt containers to a crawl. ReadyDR is also much more efficient in backing up files like those that are written to often because the other methods will always back up the whole file again if there is any change at all.