Re: Volume Expansion Failed

compchick813 · ‎2014-12-15

We are attempting to expand our Ultra 4 from 4 x 3 TB disks to 4 x 4 TB disks. We have been replacing each disk over time and just added the 4th one last night. The resync completed, but the volume won't expand. Below are the most recent logs. We have ~8.5 TB of data that we don't have space to migrate off, so we need to avoid a factory reset if at all possible. We are currently on firmware 4.2.27. If anyone has any ideas, that would be appreciated. Rebooting the NAS results in the same resuming then failed loop.

Mon Dec 15 22:26:11 EST 2014 System is up.
Mon Dec 15 22:25:58 EST 2014 Volume expansion failed
Mon Dec 15 22:25:57 EST 2014 Incompleted file system expansion detected. Resuming...
Mon Dec 15 22:25:01 EST 2014 Rebooting device...
Mon Dec 15 22:25:01 EST 2014 Please close this browser session and use RAIDar to reconnect to the device. System rebooting...
Mon Dec 15 22:24:05 EST 2014 System is up.

mdgm-ntgr · ‎2014-12-15

What disks and how many were installed when you last did a factory reset?

Sounds like you have hit the 8TB limit for online expansion for the life of the unit.

compchick813 · ‎2014-12-15

If I remember correctly the last factory reset was on 4 x 3 TB when we first set up the NAS. Are there any debug logs we can pull?

mdgm-ntgr · ‎2014-12-15

If you download your logs (Status > Logs > Download all logs) have a look at expansion.log

compchick813 · ‎2014-12-15

Thanks! I did notice the 2 lines below. The full log since the last boot is below that. BTW there is currently 441 GB free of 9259 GB.


Not enough reserved gdt blocks for resizing

Not enough disk for a new array, drives with free space: 0, X_level:5


[2014/12/15 22:25:50  2034] Boot, handle mutiple expand_md
[2014/12/15 22:25:51  2035] STAGE_CHECK: saved my_pid 2035 in check mode.
[2014/12/15 22:25:51  2035] RAID MODE: 1, sn=
[2014/12/15 22:25:51  2035] LINE 4591: exec command: lvs > /var/log/frontview/.V_E.snapshotstat
[2014/12/15 22:25:51  2035] Current file system status: ext4
[2014/12/15 22:25:51  2035] LINE 5305: exec command: rm -fr /var/log/frontview/.V_E.*
[2014/12/15 22:25:52  2035] X_level: 5
[2014/12/15 22:25:52  2035]  /usr/sbin/expand_md
[2014/12/15 22:25:52  2035] MD degraded 0, bg_job 0, boot 1
[2014/12/15 22:25:52  2035] Read 1957072 byte from configuration
[2014/12/15 22:25:52  2035] Disk configuration matching with online configuration.
[2014/12/15 22:25:52  2035] Should do expand now...
[2014/12/15 22:25:52  2035] ===== Partition Entry (MD not used) =====
[2014/12/15 22:25:52  2035] ===== Partition Entry (Used by MD) =====
[2014/12/15 22:25:52  2035] 000 /dev/sda,Z300ZZSC,8, 0, 3907018584 3907016478 3907016478 MD_FULL 0
[2014/12/15 22:25:52  2035] partitions: 5 property: 4K
[2014/12/15 22:25:52  2035] 	000 /dev/sda1, 8, 1, 4194304 MD_FULL md=/dev/md0
[2014/12/15 22:25:52  2035] 	001 /dev/sda2, 8, 2, 524288 MD_FULL md=/dev/md1
[2014/12/15 22:25:52  2035] 	002 /dev/sda5, 8, 5, 778061756 MD_FULL md=/dev/md2
[2014/12/15 22:25:52  2035] 	003 /dev/sda6, 8, 6, 2147484131 MD_FULL md=/dev/md3
[2014/12/15 22:25:52  2035] 	004 /dev/sda7, 8, 7, 976751999 MD_FULL md=/dev/md4
[2014/12/15 22:25:52  2035] 001 /dev/sdb,Z300QAHG,8, 16, 3907018584 3907016478 3907016478 MD_FULL 0
[2014/12/15 22:25:52  2035] partitions: 5 property: 4K
[2014/12/15 22:25:52  2035] 	000 /dev/sdb1, 8, 17, 4194304 MD_FULL md=/dev/md0
[2014/12/15 22:25:52  2035] 	001 /dev/sdb2, 8, 18, 524288 MD_FULL md=/dev/md1
[2014/12/15 22:25:52  2035] 	002 /dev/sdb5, 8, 21, 778061756 MD_FULL md=/dev/md2
[2014/12/15 22:25:52  2035] 	003 /dev/sdb6, 8, 22, 2147484131 MD_FULL md=/dev/md3
[2014/12/15 22:25:52  2035] 	004 /dev/sdb7, 8, 23, 976751999 MD_FULL md=/dev/md4
[2014/12/15 22:25:52  2035] 002 /dev/sdc,Z3020FAD,8, 32, 3907018584 3907016478 3907016478 MD_FULL 0
[2014/12/15 22:25:52  2035] partitions: 5 property: 4K
[2014/12/15 22:25:52  2035] 	000 /dev/sdc1, 8, 33, 4194304 MD_FULL md=/dev/md0
[2014/12/15 22:25:52  2035] 	001 /dev/sdc2, 8, 34, 524288 MD_FULL md=/dev/md1
[2014/12/15 22:25:52  2035] 	002 /dev/sdc5, 8, 37, 778061756 MD_FULL md=/dev/md2
[2014/12/15 22:25:52  2035] 	003 /dev/sdc6, 8, 38, 2147484131 MD_FULL md=/dev/md3
[2014/12/15 22:25:52  2035] 	004 /dev/sdc7, 8, 39, 976751999 MD_FULL md=/dev/md4
[2014/12/15 22:25:52  2035] 003 /dev/sdd,Z302F9XQ,8, 48, 3907018584 3907016478 3907016478 MD_FULL 0
[2014/12/15 22:25:52  2035] partitions: 5 property: 4K
[2014/12/15 22:25:52  2035] 	000 /dev/sdd1, 8, 49, 4194304 MD_FULL md=/dev/md0
[2014/12/15 22:25:52  2035] 	001 /dev/sdd2, 8, 50, 524288 MD_FULL md=/dev/md1
[2014/12/15 22:25:52  2035] 	002 /dev/sdd5, 8, 53, 778061756 MD_FULL md=/dev/md2
[2014/12/15 22:25:52  2035] 	003 /dev/sdd6, 8, 54, 2147484131 MD_FULL md=/dev/md3
[2014/12/15 22:25:52  2035] 	004 /dev/sdd7, 8, 55, 976751999 MD_FULL md=/dev/md4
[2014/12/15 22:25:52  2035] Found leftover expansion actions DOING_F_RESIZE
, continue...
sendmail: cannot connect to smtp.gmail.com, port 587: Network is unreachable
sendmail: could not send mail (account default from /etc/msmtprc)
[2014/12/15 22:25:57  2035] LINE 448: exec command: echo "DOING_F_RESIZE" > /.os_V_E_continue
[2014/12/15 22:25:58  2035] LINE 4249: exec command: resize2fs -pf /dev/c/c
resize2fs 1.42.12 (29-Aug-2014)
resize2fs: Not enough reserved gdt blocks for resizing
Filesystem at /dev/c/c is mounted on /c; on-line resizing required
old_desc_blocks = 1163, new_desc_blocks = 1395
[2014/12/15 22:25:58  2035] Expand second phase error 10: resize2fs, err=0x100
sendmail: cannot locate host smtp.gmail.com: Name or service not known
sendmail: could not send mail (account default from /etc/msmtprc)
[2014/12/15 22:25:58  2035] File system expand failed
[2014/12/15 22:25:58  2035] LINE 4852: exec command: /usr/sbin/expand_md -a super >> /var/log/frontview/expand_md.log 2>&1 &
[2014/12/15 22:25:58  2697] Boot, handle mutiple expand_md
[2014/12/15 22:25:58  2035] LINE 4855: exec command: /frontview/bin/volumescan &

[2014/12/15 22:25:58  2035] LINE 4939: exec command: ps -ef | grep expand_md | grep -v grep > /var/log/frontview/.V_E.snapshotstat
[2014/12/15 22:25:58  2035] STAGE_WIPE: Clean my_pid 2035
[2014/12/15 22:25:59  2698] STAGE_CHECK: saved my_pid 2698 in check mode.
[2014/12/15 22:25:59  2698] RAID MODE: 1, sn=
[2014/12/15 22:25:59  2698] LINE 4591: exec command: lvs > /var/log/frontview/.V_E.snapshotstat
[2014/12/15 22:25:59  2698] Current file system status: ext4
[2014/12/15 22:25:59  2698] LINE 5305: exec command: rm -fr /var/log/frontview/.V_E.*
[2014/12/15 22:25:59  2698] X_level: 5
[2014/12/15 22:25:59  2698]  /usr/sbin/expand_md -a super
[2014/12/15 22:25:59  2698] MD degraded 0, bg_job 0, boot 1
[2014/12/15 22:25:59  2698] ++++Write 1957072 byte to configuration++++
[2014/12/15 22:25:59  2698] /var/log/frontview/.known_cfgdir/Z300ZZSC
[2014/12/15 22:25:59  2698] gpt sig:0,mbr sig:0,fake type:0
[2014/12/15 22:25:59  2698] get disk /dev/sda format is (GPT=2,MBR=1,MX=3,MISC=-1): 2
[2014/12/15 22:25:59  2698] LINE 7921: exec command: sgdisk -p /dev/sda | grep '[0-9]    ' > /var/log/frontview/.known_cfgdir/Z300ZZSC
[2014/12/15 22:26:00  2698] /var/log/frontview/.known_cfgdir/Z300QAHG
[2014/12/15 22:26:00  2698] gpt sig:0,mbr sig:0,fake type:0
[2014/12/15 22:26:00  2698] get disk /dev/sdb format is (GPT=2,MBR=1,MX=3,MISC=-1): 2
[2014/12/15 22:26:00  2698] LINE 7921: exec command: sgdisk -p /dev/sdb | grep '[0-9]    ' > /var/log/frontview/.known_cfgdir/Z300QAHG
[2014/12/15 22:26:01  2698] /var/log/frontview/.known_cfgdir/Z3020FAD
[2014/12/15 22:26:01  2698] gpt sig:0,mbr sig:0,fake type:0
[2014/12/15 22:26:01  2698] get disk /dev/sdc format is (GPT=2,MBR=1,MX=3,MISC=-1): 2
[2014/12/15 22:26:01  2698] LINE 7921: exec command: sgdisk -p /dev/sdc | grep '[0-9]    ' > /var/log/frontview/.known_cfgdir/Z3020FAD
[2014/12/15 22:26:02  2698] /var/log/frontview/.known_cfgdir/Z302F9XQ
[2014/12/15 22:26:02  2698] gpt sig:0,mbr sig:0,fake type:0
[2014/12/15 22:26:02  2698] get disk /dev/sdd format is (GPT=2,MBR=1,MX=3,MISC=-1): 2
[2014/12/15 22:26:02  2698] LINE 7921: exec command: sgdisk -p /dev/sdd | grep '[0-9]    ' > /var/log/frontview/.known_cfgdir/Z302F9XQ
[2014/12/15 22:26:02  2698] LINE 4939: exec command: ps -ef | grep expand_md | grep -v grep > /var/log/frontview/.V_E.snapshotstat
[2014/12/15 22:26:02  2698] gpt sig:0,mbr sig:0,fake type:0
[2014/12/15 22:26:02  2698] get disk /dev/sda format is (GPT=2,MBR=1,MX=3,MISC=-1): 2
[2014/12/15 22:26:02  2698] /dev/sda used/total/partitioned = 3907016478/3907018584/3907016478, 60000000
[2014/12/15 22:26:02  2698] gpt sig:0,mbr sig:0,fake type:0
[2014/12/15 22:26:02  2698] get disk /dev/sdb format is (GPT=2,MBR=1,MX=3,MISC=-1): 2
[2014/12/15 22:26:02  2698] /dev/sdb used/total/partitioned = 3907016478/3907018584/3907016478, 60000000
[2014/12/15 22:26:02  2698] gpt sig:0,mbr sig:0,fake type:0
[2014/12/15 22:26:02  2698] get disk /dev/sdc format is (GPT=2,MBR=1,MX=3,MISC=-1): 2
[2014/12/15 22:26:02  2698] /dev/sdc used/total/partitioned = 3907016478/3907018584/3907016478, 60000000
[2014/12/15 22:26:02  2698] gpt sig:0,mbr sig:0,fake type:0
[2014/12/15 22:26:02  2698] get disk /dev/sdd format is (GPT=2,MBR=1,MX=3,MISC=-1): 2
[2014/12/15 22:26:02  2698] /dev/sdd used/total/partitioned = 3907016478/3907018584/3907016478, 60000000
[2014/12/15 22:26:02  2698] Not enough disk for a new array, drives with free space: 0, X_level:5
[2014/12/15 22:26:02  2698] LINE 4939: exec command: ps -ef | grep expand_md | grep -v grep > /var/log/frontview/.V_E.snapshotstat
[2014/12/15 22:26:02  2698] STAGE_WIPE: Clean my_pid 2698

mdgm-ntgr · ‎2014-12-15

Backup, do a factory reset (wipes all data, settings, everything) and restore your data from backup.

compchick813 · ‎2014-12-15

You don't think there are any other options? I'm also trying to understand for sure what the actual issue is. Thanks!

mdgm-ntgr · ‎2014-12-15

If your volume capacity was e.g. 2.7TB when you last did a factory reset you cannot expand past 10.7TB.

Well the first step regardless is to ensure you have an up to date backup.

compchick813 · ‎2014-12-15

Darn, I think you're right. I just found the log below in the expansion.log from the day we set it up. That's 2794 GB, if I'm correct in assuming it's talking about space. We must have put 1 or 2 drives in first and added the others after the initialization.

[2012/10/04 20:29:41 1944] exec command: Selected: 0 total=2930266584 used/resv=782780348/60000000 free=2147486236, cl_at=0, property4k=1

alexofindy · ‎2014-12-16

Many, myself included, have made this mistake, because Netgear's documentation for the 4.2.x systems was faulty. The official documentation did not mention the 8 TB limit, and, in fact suggested a good way to build a system was by adding one disk at a time.

There are two ways to fix this: do an offline expansion, which apparently is not affected by the 8 TB limit, but which is not supported, and requires a fair amount of futzing with command lines, and second, as mgdm suggests, backing all your data up and doing a factory reset with all your disks in place. Assuming of course you have a backup you know is good of all 8TB (or whatever) of your data, since the factory reset will erase everything. (I have not tried the offline expansion myself, but probably will soon now that I have a reliable backup, on a new 316 device)

Netgear themselves could, and frankly should, help out by providing a supported add-on that does an offline expansion. Frankly, since it was their official documentation that led many of us to the problem, I think they "owe" us such an add-on.

But, instead, they have moved on to OS 6. OS 4.2.x is yesterday's news.

This is my single biggest gripe with netgear and the readynas product line. There are other issues as well. But, when all is said and done, I did buy a 316.

mdgm-ntgr · ‎2014-12-16

There can be multiple reasons that expansion may fail and providing an add-on to expand a volume could be dangerous if used inappropriately.

compchick813 · ‎2014-12-17

Thanks! I am still working on backing up all of the data. I'll be able to just barely squeeze it onto other drives. 🙂