× NETGEAR will be terminating ReadyCLOUD service by July 1st, 2023. For more details click here.
Orbi WiFi 7 RBE973
Reply

ReadyNAS 424 XRAID Not Expanding

ooCHUMoo
Aspirant

ReadyNAS 424 XRAID Not Expanding

I was running 4x10TB (XRAID/RAID 5).  I replaced two disk with 14TB so that the configuration is 2x10TB and 2x14TB.  Resync completed, but volume size is the same as before installing the larger disks at 27TB.  I expected 31TB after expansion.  Logs indicate that it tried to expand but the only error I see is:

 

ERROR: sdd sectors < 0 (-22)

 

after which, it tries to expand again but no errors.

 

Any ideas?

 

Thx.

Model: RN424|ReadyNAS 424 – High-performance Business Data Storage - 4-Bay
Message 1 of 21
StephenB
Guru

Re: ReadyNAS 424 XRAID Not Expanding

Can you download the log zip file, and then post mdstat.log here?

 

Copy/paste it - don't try to attach it.

Message 2 of 21
ooCHUMoo
Aspirant

Re: ReadyNAS 424 XRAID Not Expanding

Here's mdstat.log:

 

Personalities : [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md1 : active raid10 sde2[3] sdc2[2] sdb2[1] sda2[0]
1044480 blocks super 1.2 512K chunks 2 near-copies [4/4] [UUUU]

md127 : active raid5 sde3[5] sda3[4] sdb3[6] sdc3[1]
29284761408 blocks super 1.2 level 5, 64k chunk, algorithm 2 [4/4] [UUUU]

md0 : active raid1 sde1[5] sdb1[6] sda1[4] sdc1[1]
4190208 blocks super 1.2 [4/4] [UUUU]

unused devices: <none>
/dev/md/0:
Version : 1.2
Creation Time : Thu Apr 19 04:49:57 2018
Raid Level : raid1
Array Size : 4190208 (4.00 GiB 4.29 GB)
Used Dev Size : 4190208 (4.00 GiB 4.29 GB)
Raid Devices : 4
Total Devices : 4
Persistence : Superblock is persistent

Update Time : Sun Jan 17 16:16:18 2021
State : clean
Active Devices : 4
Working Devices : 4
Failed Devices : 0
Spare Devices : 0

Consistency Policy : unknown

Name : 0a440448:0 (local to host 0a440448)
UUID : 05b40dd7:f367115f:7dcea494:58ec9efd
Events : 1138

Number Major Minor RaidDevice State
6 8 17 0 active sync /dev/sdb1
1 8 33 1 active sync /dev/sdc1
5 8 65 2 active sync /dev/sde1
4 8 1 3 active sync /dev/sda1
/dev/md/Data-0:
Version : 1.2
Creation Time : Sat Apr 21 09:39:45 2018
Raid Level : raid5
Array Size : 29284761408 (27928.12 GiB 29987.60 GB)
Used Dev Size : 9761587136 (9309.37 GiB 9995.87 GB)
Raid Devices : 4
Total Devices : 4
Persistence : Superblock is persistent

Update Time : Sun Jan 17 04:22:13 2021
State : clean
Active Devices : 4
Working Devices : 4
Failed Devices : 0
Spare Devices : 0

Layout : left-symmetric
Chunk Size : 64K

Consistency Policy : unknown

Name : 0a440448:Data-0 (local to host 0a440448)
UUID : 479170a6:4dfb3551:bbcb61e9:c0019d9b
Events : 53012

Number Major Minor RaidDevice State
5 8 67 0 active sync /dev/sde3
1 8 35 1 active sync /dev/sdc3
6 8 19 2 active sync /dev/sdb3
4 8 3 3 active sync /dev/sda3

 

Message 3 of 21
Sandshark
Sensei

Re: ReadyNAS 424 XRAID Not Expanding

Is XRAID turned on (shows green stripe)?  If it's not, you'll get the result you have until you manually expand the volume.  But, you should be able to just turn it on and have it expand.

 

If it's already on, partitions.log may also be helpful.

Message 4 of 21
ooCHUMoo
Aspirant

Re: ReadyNAS 424 XRAID Not Expanding

Yes, green stripe under XRAID so it's on.  Furthermore, if I click on it, it asks if I want to turn off XRAID and switch to FLEXRAID.

 

If it matters, the drive/slot configuration is 14/10/14/10.  I replaced the 3rd drive initially, and then the 1st drive.

 

Here's partitions.log:

 

major minor #blocks name

8 0 9766436864 sda
8 1 4194304 sda1
8 2 524288 sda2
8 3 9761718216 sda3
8 16 13672382464 sdb
8 17 4194304 sdb1
8 18 524288 sdb2
8 19 9761718216 sdb3
8 32 9766436864 sdc
8 33 4194304 sdc1
8 34 524288 sdc2
8 35 9761718216 sdc3
8 96 4883770583 sdg
8 97 131072 sdg1
8 98 4883638272 sdg2
9 0 4190208 md0
9 127 29284761408 md127
251 0 29284759360 dm-0
8 64 13672382464 sde
8 65 4194304 sde1
8 66 524288 sde2
8 67 9761718216 sde3
9 1 1044480 md1

Message 5 of 21
StephenB
Guru

Re: ReadyNAS 424 XRAID Not Expanding

Have you tried rebooting?

 

What firmware are you running?

Message 6 of 21
ooCHUMoo
Aspirant

Re: ReadyNAS 424 XRAID Not Expanding

Firmware 6.10.4.

Rebooted it about 10 minutes ago, no change.

 

Message 7 of 21
mdgm
Virtuoso

Re: ReadyNAS 424 XRAID Not Expanding

Maybe try powering down, disconnecting your external drive and powering on again?

 

It doesn't look like it's even created the new partitions for vertical expansion on the 14TB disks let alone the new RAID layer etc.

Message 8 of 21
ooCHUMoo
Aspirant

Re: ReadyNAS 424 XRAID Not Expanding

Tried shutting down, removed external storage, but still not expanding after power up.

Message 9 of 21
Sandshark
Sensei

Re: ReadyNAS 424 XRAID Not Expanding

If you are comfortable with SSH, the following command should start the expansion:  volume_util  -e  xraid.  If it doesn't, maybe it will at least give an error message that's helpful.

 

There are some more specific commands to tell it what drives to use, but I'm afraid that would mess up XRAID knowing the configuration.  I've only used them in a FlexRAID unit, and then only for experimentation.

Message 10 of 21
ooCHUMoo
Aspirant

Re: ReadyNAS 424 XRAID Not Expanding

Interesting.  Tried ssh and got an error:

 

admin@Galactus:~$ volume_util -e xraid
Expanding pools...
sh: 1: btrfs: not found
admin@Galactus:~$

Message 11 of 21
Sandshark
Sensei

Re: ReadyNAS 424 XRAID Not Expanding

Not especially helpful to me, but seems to indicate that some information on the XRAID configuration that it needs to determine how to expand is missing.  Maybe one of the Netgear folks can better interpret it.

 

The logs indicate that MDADM and the rest of the Linux core are aware of everything, so it must be XRAID that's not, and I don't know where that's located.   See if volume_util  -s  pool will provide more information.

 

Is the external; drive still disconnected?  Maybe it's getting stuck looking for how to expand it, which it can't do, and never moves to the main volume.

 

The help on volume_util includes:

 

Usage: volume_util [-c NAME] [-c NAME D1,..,DN] [-c NAME LEVEL D1,..,DN]
[-c NAME LEVEL D1,..,DN KEY_FILE] [-d NAME|UUID]
[-x NAME|UUID] [-e] [-e TYPE] [-v NAME LEVEL D1,...,DN]
[-f NAME DISK] [-S DISK] [-F DISK] [-s TYPE..]

-e Expand pool TYPE is one of 'auto', 'expand', 'extend', or 'xraid' auto - automatically do all forms of expansion expand - expand to fill all space on all current disks (disk replacement) extend - extend pool to use available new disks (disk addition) xraid - grow pool to use available space on some current disks (partial disk replacement)

There is a different command for Flex-RAID expansion and one for "Vdev expand pool", whatever that means, since the only references to Vdev I Google are for ZFS.  Then again, ReadyDataOS used ZFS, I believe, so maybe that's just a leftover in the help.  It appears clear that OS6 borrowed a lot from ReadyDataOS.

 

I think playing around with the flex-raid command option is a bad idea at this point, but maybe try volume_util  -e  expand, though I rather expect a similar error since that's what the xraid expansion type should have determined was needed, if I'm interpreting the command properly.  You could try -e  auto, too, likely with the same result.  It doesn't say what -e by itself does, and it is allowed, so you could try that as well.  My guess is it's just shorthand for -e  auto and will also give the same result.

 

The Flex-RAID (-f) option lets you specify devices and the pool (volume), which is what that error message looks like it says it can't find, but I'm afraid it could make things worse if you are not in Flex-RAID mode.  Of course, it might work and just not make it better for future expansion.  If you want to try it, make sure you have a full backup in case a factory default becomes the only way out of a hole you dug yourself.  volume_util  --help will give you the options.

 

Switching XRAID off and back on has been known to fix issues for some, but it is possible that you will be stuck not being able to go back to XRAID, or possibly worse.  So, again, be sure you have a backup before you try that.

Message 12 of 21
ooCHUMoo
Aspirant

Re: ReadyNAS 424 XRAID Not Expanding

volume_util -e expand (and auto) gave the same error as before:

admin@Galactus:~$ volume_util -e expand
Expanding pools...
sh: 1: btrfs: not found
admin@Galactus:~$

 

Here's pool info:

admin@Galactus:~$ volume_util -s pool
sh: 1: btrfs: not found
==============================================
Pools
==============================================
Pool root:
Device: /dev/md0
Node: 9:0
HostID: 0a440448 (native)
UUID: b1ed4dd4-2e64-464f-b4f6-f670db7c1024
Mount point: /
Size: 3793952KB (3 GB)
Available: 3118324KB (2 GB)
RAID Level: 1
State: redundant
Action: idle
Flags: 0x68
Type: btrfs
Tier Flag: Disable
RAIDs:
md0
Size: 8380416
Level: 1
Action: idle
Members: 4
Type: 3
Flags: 0x28
Tier: 0
Data allocation: 0 GB
Metadata allocation: 0 MB
Unallocated: 0 GB
Disks:
sdd:
HostID: 0a440448
Flags: 0x1
Size: 27344764928 (13039 GB)
Free: 7811891214
Controller 0
Channel: 0
Model: WDC_WD141KFGX-68FH9N0
Serial: X0GZ1HMC
Firmware: 83.00A83
Class: SATA (3)
RPM: 7200
sdc:
HostID: 0a440448
Flags: 0x1
Size: 19532873728 (9314 GB)
Free: 14
Controller 0
Channel: 1
Model: ST10000VN0004-2GS11L
Serial: ZJV0MECF
Firmware: SC60
Class: SATA (3)
RPM: 7200
sdb:
HostID: 0a440448
Flags: 0x1
Size: 27344764928 (13039 GB)
Free: 7811891214
Controller 0
Channel: 2
Model: WDC_WD141KFGX-68FH9N0
Serial: X0GYH9WC
Firmware: 83.00A83
Class: SATA (3)
RPM: 7200
sda:
HostID: 0a440448
Flags: 0x1
Size: 19532873728 (9314 GB)
Free: 14
Controller 0
Channel: 3
Model: ST10000VN0004-1ZD101
Serial: ZA2A4XQC
Firmware: SC60
Class: SATA (3)
RPM: 7200
Pool swap:
Device: /dev/md1
Node: 9:1
HostID: 0a440448 (native)
UUID: cb058d6c-8287-44b6-9a98-9799db9df490
Mount point: none
Size: 0KB (0 GB)
Available: 0KB (0 GB)
RAID Level: 10
State: redundant
Action: idle
Flags: 0x28
Type: swap
Tier Flag: Disable
RAIDs:
md1
Size: 2088960
Level: 10
Action: idle
Members: 4
Type: 3
Flags: 0x28
Tier: 0
Disks:
sdd:
HostID: 0a440448
Flags: 0x1
Size: 27344764928 (13039 GB)
Free: 7811891214
Controller 0
Channel: 0
Model: WDC_WD141KFGX-68FH9N0
Serial: X0GZ1HMC
Firmware: 83.00A83
Class: SATA (3)
RPM: 7200
sdc:
HostID: 0a440448
Flags: 0x1
Size: 19532873728 (9314 GB)
Free: 14
Controller 0
Channel: 1
Model: ST10000VN0004-2GS11L
Serial: ZJV0MECF
Firmware: SC60
Class: SATA (3)
RPM: 7200
sdb:
HostID: 0a440448
Flags: 0x1
Size: 27344764928 (13039 GB)
Free: 7811891214
Controller 0
Channel: 2
Model: WDC_WD141KFGX-68FH9N0
Serial: X0GYH9WC
Firmware: 83.00A83
Class: SATA (3)
RPM: 7200
sda:
HostID: 0a440448
Flags: 0x1
Size: 19532873728 (9314 GB)
Free: 14
Controller 0
Channel: 3
Model: ST10000VN0004-1ZD101
Serial: ZA2A4XQC
Firmware: SC60
Class: SATA (3)
RPM: 7200
Pool data
Device: /dev/mapper/Data-0
Node: 251:0
HostID: 0a440448 (native)
UUID: 3f6282ad-bf0a-45f2-8e1b-8f3c4a49dae7
Mount point: /Data
Size: 29281453696KB (27924 GB)
Available: 7860386656KB (7496 GB)
Snapshot: 0KB (0 MB)
Data allocation: 0 GB
Data used: 0 MB (0 GB)
Metadata allocation: 0 MB
Metadata used: 0 MB
Unallocated: 0 GB
RAID Level: 5
State: redundant
Action: idle
Flags: 0x49
Type: btrfs
Tier Flag: Disable
RAIDs:
md127
Size: 58569522816
Level: 5
Action: idle
Members: 4
Type: 3
Flags: 0x9
Tier: 0
Data allocation: 0 GB
Metadata allocation: 0 MB
Unallocated: 0 GB
Disks:
sdd:
HostID: 0a440448
Flags: 0x1
Size: 27344764928 (13039 GB)
Free: 7811891214
Controller 0
Channel: 0
Model: WDC_WD141KFGX-68FH9N0
Serial: X0GZ1HMC
Firmware: 83.00A83
Class: SATA (3)
RPM: 7200
sdc:
HostID: 0a440448
Flags: 0x1
Size: 19532873728 (9314 GB)
Free: 14
Controller 0
Channel: 1
Model: ST10000VN0004-2GS11L
Serial: ZJV0MECF
Firmware: SC60
Class: SATA (3)
RPM: 7200
sdb:
HostID: 0a440448
Flags: 0x1
Size: 27344764928 (13039 GB)
Free: 7811891214
Controller 0
Channel: 2
Model: WDC_WD141KFGX-68FH9N0
Serial: X0GYH9WC
Firmware: 83.00A83
Class: SATA (3)
RPM: 7200
sda:
HostID: 0a440448
Flags: 0x1
Size: 19532873728 (9314 GB)
Free: 14
Controller 0
Channel: 3
Model: ST10000VN0004-1ZD101
Serial: ZA2A4XQC
Firmware: SC60
Class: SATA (3)
RPM: 7200
admin@Galactus:~$

Message 13 of 21
StephenB
Guru

Re: ReadyNAS 424 XRAID Not Expanding


@ooCHUMoo wrote:

volume_util -e expand (and auto) gave the same error as before:

admin@Galactus:~$ volume_util -e expand

You are logging in as admin, and you should be logging in as root (using the admin password).

Message 14 of 21
ooCHUMoo
Aspirant

Re: ReadyNAS 424 XRAID Not Expanding

Good catch.  Tried again but same results, no change in volume size:

root@Galactus:~# volume_util -e xraid
Expanding pools...
root@Galactus:~# volume_util -e expand
Expanding pools...
root@Galactus:~# volume_util -s pool
==============================================
Pools
==============================================
Pool root:
Device: /dev/md0
Node: 9:0
HostID: 0a440448 (native)
UUID: b1ed4dd4-2e64-464f-b4f6-f670db7c1024
Mount point: /
Size: 3793952KB (3 GB)
Available: 3116196KB (2 GB)
RAID Level: 1
State: redundant
Action: idle
Flags: 0x68
Type: btrfs
Tier Flag: Disable
RAIDs:
md0
Size: 8380416
Level: 1
Action: idle
Members: 4
Type: 3
Flags: 0x28
Tier: 0
Data allocation: 0 GB
Metadata allocation: 0 MB
Unallocated: 0 GB
Disks:
sdd:
HostID: 0a440448
Flags: 0x1
Size: 27344764928 (13039 GB)
Free: 7811891214
Controller 0
Channel: 0
Model: WDC_WD141KFGX-68FH9N0
Serial: X0GZ1HMC
Firmware: 83.00A83
Class: SATA (3)
RPM: 7200
sdc:
HostID: 0a440448
Flags: 0x1
Size: 19532873728 (9314 GB)
Free: 14
Controller 0
Channel: 1
Model: ST10000VN0004-2GS11L
Serial: ZJV0MECF
Firmware: SC60
Class: SATA (3)
RPM: 7200
sdb:
HostID: 0a440448
Flags: 0x1
Size: 27344764928 (13039 GB)
Free: 7811891214
Controller 0
Channel: 2
Model: WDC_WD141KFGX-68FH9N0
Serial: X0GYH9WC
Firmware: 83.00A83
Class: SATA (3)
RPM: 7200
sda:
HostID: 0a440448
Flags: 0x1
Size: 19532873728 (9314 GB)
Free: 14
Controller 0
Channel: 3
Model: ST10000VN0004-1ZD101
Serial: ZA2A4XQC
Firmware: SC60
Class: SATA (3)
RPM: 7200
Pool swap:
Device: /dev/md1
Node: 9:1
HostID: 0a440448 (native)
UUID: cb058d6c-8287-44b6-9a98-9799db9df490
Mount point: none
Size: 0KB (0 GB)
Available: 0KB (0 GB)
RAID Level: 10
State: redundant
Action: idle
Flags: 0x28
Type: swap
Tier Flag: Disable
RAIDs:
md1
Size: 2088960
Level: 10
Action: idle
Members: 4
Type: 3
Flags: 0x28
Tier: 0
Disks:
sdd:
HostID: 0a440448
Flags: 0x1
Size: 27344764928 (13039 GB)
Free: 7811891214
Controller 0
Channel: 0
Model: WDC_WD141KFGX-68FH9N0
Serial: X0GZ1HMC
Firmware: 83.00A83
Class: SATA (3)
RPM: 7200
sdc:
HostID: 0a440448
Flags: 0x1
Size: 19532873728 (9314 GB)
Free: 14
Controller 0
Channel: 1
Model: ST10000VN0004-2GS11L
Serial: ZJV0MECF
Firmware: SC60
Class: SATA (3)
RPM: 7200
sdb:
HostID: 0a440448
Flags: 0x1
Size: 27344764928 (13039 GB)
Free: 7811891214
Controller 0
Channel: 2
Model: WDC_WD141KFGX-68FH9N0
Serial: X0GYH9WC
Firmware: 83.00A83
Class: SATA (3)
RPM: 7200
sda:
HostID: 0a440448
Flags: 0x1
Size: 19532873728 (9314 GB)
Free: 14
Controller 0
Channel: 3
Model: ST10000VN0004-1ZD101
Serial: ZA2A4XQC
Firmware: SC60
Class: SATA (3)
RPM: 7200
Pool data
Device: /dev/mapper/Data-0
Node: 251:0
HostID: 0a440448 (native)
UUID: 3f6282ad-bf0a-45f2-8e1b-8f3c4a49dae7
Mount point: /Data
Size: 29281450624KB (27924 GB)
Available: 7862058724KB (7497 GB)
Snapshot: 0KB (0 MB)
Data allocation: 20471 GB
Data used: 20850909 MB (20362 GB)
Metadata allocation: 34848 MB
Metadata used: 32976 MB
Unallocated: 7389 GB
RAID Level: 5
State: redundant
Action: idle
Flags: 0x49
Type: btrfs
Tier Flag: Disable
RAIDs:
md127
Size: 58569522816
Level: 5
Action: idle
Members: 4
Type: 3
Flags: 0x9
Tier: 0
Data allocation: 20471 GB
Metadata allocation: 69696 MB
Unallocated: 7389 GB
Disks:
sdd:
HostID: 0a440448
Flags: 0x1
Size: 27344764928 (13039 GB)
Free: 7811891214
Controller 0
Channel: 0
Model: WDC_WD141KFGX-68FH9N0
Serial: X0GZ1HMC
Firmware: 83.00A83
Class: SATA (3)
RPM: 7200
sdc:
HostID: 0a440448
Flags: 0x1
Size: 19532873728 (9314 GB)
Free: 14
Controller 0
Channel: 1
Model: ST10000VN0004-2GS11L
Serial: ZJV0MECF
Firmware: SC60
Class: SATA (3)
RPM: 7200
sdb:
HostID: 0a440448
Flags: 0x1
Size: 27344764928 (13039 GB)
Free: 7811891214
Controller 0
Channel: 2
Model: WDC_WD141KFGX-68FH9N0
Serial: X0GYH9WC
Firmware: 83.00A83
Class: SATA (3)
RPM: 7200
sda:
HostID: 0a440448
Flags: 0x1
Size: 19532873728 (9314 GB)
Free: 14
Controller 0
Channel: 3
Model: ST10000VN0004-1ZD101
Serial: ZA2A4XQC
Firmware: SC60
Class: SATA (3)
RPM: 7200
root@Galactus:~#

Message 15 of 21
Sandshark
Sensei

Re: ReadyNAS 424 XRAID Not Expanding

OK, this is odd:

 

     Pool data
          Device: /dev/mapper/Data-0

 

It should be:

     Pool data
          Device: /dev/md127

 

Data-0 is the label of the RAID, but the BTRFS volume should be on a md RAID device.  I've only ever seen anything similar when the OS can't properly mount the file system.

 

I have no idea how to fix it or if it's even fixable sort of a factory default, as it shouldn't ever be that way.  The only thing in /dev/mapper on my NAS is control, which is a character special file one byte long, not a normal file.

 

 

Message 16 of 21
ooCHUMoo
Aspirant

Re: ReadyNAS 424 XRAID Not Expanding

Hm, okay, so no options other than reset to factory?

 

Message 17 of 21
ooCHUMoo
Aspirant

Re: ReadyNAS 424 XRAID Not Expanding

Just did a Factor Default using the boot menu.  XRaid volume is still coming up as 27TB.  Tried Destroying the volume and creating one via the New Volume button, and it's still 27TB under Raid5/XRaid.

 

Really not sure what to do at this point.

Message 18 of 21
Sandshark
Sensei

Re: ReadyNAS 424 XRAID Not Expanding

And you are sure XRAID is turned on?  You would be limited to 27TB in FlexRAID.

Message 19 of 21
ooCHUMoo
Aspirant

Re: ReadyNAS 424 XRAID Not Expanding

Finally got this sorted out.  Doing a Factory Default with all four drives in the system resulted in a XRaid volume that was only 27TB, and wouldn't expand.  I tried deleting the volume and re-creating it manually using the UI (Destroy, and then New Volume), but that still resulted in a 27TB volume.

 

This is what worked:

Insert just the two 10TB drives and create a New Volume via UI, this resulted in a 9TB RAID 1 volume.

 

Once resync completed, I turned on XRaid.  Volume is still RAID 1.

 

Insert first 14TB and let XRaid expand and do it's thing.  After Resync, volume is 18TB RAID 5 as expected.

 

Insert second 14TB and let XRaid expand and do it's thing.  After Resync, voume is 31TB RAID 5 as expected.

 

Whole process took almost 4 days, but seems to be okay.  Doing restore from backup now.

Message 20 of 21
Sandshark
Sensei

Re: ReadyNAS 424 XRAID Not Expanding

Glad you figured out how to get it done, even if "painful".  And thanks for sharing the solution in case anyone else has a similar issue.

 

Definitely seems like a bug to me.  Sounds like Netgear needs to do some testing with larger drives now that they're available.

Message 21 of 21
Top Contributors
Discussion stats
  • 20 replies
  • 2479 views
  • 0 kudos
  • 4 in conversation
Announcements