4.2.16-T75 problem(s) (ultra 6 w/six 3 TB drives)

[ThomColeman]

While on 4.2.15, I swapped out my six 2 TB drives with six 3 TB drives, allowing full resync between each swap.

4.2.15 was unable to use the extra space.

I installed 4.2.16-T75 at 11 pm, Friday night, and it began a resize operation after it rebooted.

The door display would toggle every few seconds from displaying resize stats to displaying resync stats.

Saturday morning (early) the door display showed either the resize or the resync as 63% complete (I think it was the resize). It would display the one statistic almost continuously, with the other statistic only flashing up on the display for a split second every several seconds, making it impossible to read without watching a few iterations of the split second display.

Came home from work Saturday afternoon, and the door display was continuously displaying Resync C: 99.9%. It's been like that for several hours now. I can hear brief drive activity every couple of seconds or so. The door display never blanks.

Fired up RAIDar and looked at that Ultra 6 (I have two of them). Status->Log shows the last entry occurred Friday night at 2300, System is up. (Second System is up message after 4.2.16-T75 was installed.)

RAIDar acts slightly sluggish when I attempt to access the resized Ultra, and will freeze depending on what I access. RAIDar acts fine when I access the other Ultra.

I went into System->Alerts on the resized Ultra and attempted to send a test email alert. RAIDar just hung with the rotating busy icon onscreen. I never received a message that the alert was sent, and I never received an email. The 4.2.15 Ultra sends test alerts just fine.

I went into Status->Logs and attempted to download all logs. My browser displayed a gradually lengthening progress bar like normal, then the bar went away. A log zip file was never received my computer. This works normal on the 4.2.15 Ultra.

I used putty to access the Ultra and looked at the kern.log. The kern.log had a message at 8:04 am Saturday morning that the resync was done. So it appears the resize/resync only took 9 hours or so, then something hung.

I can still access the Ultra with my TiVos, so the Ultra isn't completely hung.

Weekly volume maintenance and filesystem check is scheduled for tonight, so I'm going to see if they succeed.

- Thom

EDIT: I received the normal email alert when the volume consistency check began. Should receive another in an hour or so, when the consistency check finishes.

EDIT: Never received volume consistency check finished email. Nor did RAID scrubbing begin. Door display stuck displaying Resync C: 99.9%. RAIDar reports this RAID as about 12.5 TB in size. Since it was 9.2 TB before adding another 6 TB of drive space, I would have expected it to now be about 14 TB in size.

I'm not going to reboot it for a couple of days, in case any ReadyNAS software people want to look at it remotely. I can't send any logs because the Download All Logs function of RAIDar isn't working with this ReadyNAS.

EDIT: RAIDar is now completely non-functional with 4.2.16-T75. When I select the ReadyNAS and click Setup, I get a completely blank white window. RAIDar works fine with 4.2.15 on another Ultra 6.

- Thom

[ThomColeman]

4.2.16-T75 got hung during the resizing process. But it kept archiving files from my TiVos, and serving the files it had back to my TiVos.

RAIDar finally wouldn't work to access the Ultra at all.

I let it sit like that for 1.5 days, then power cycled it. When I briefly pressed the power button, then pressed it again, the door display changed to reflect the shutdown. But the Ultra never shutdown. I finally held in the power button for several seconds which forced it to power off.

On power up, it noticed an unfinished expansion had been in progress, and it completed the expansion. The email alerts were working again, and RAIDar will now connect and interact with the Ultra again.

All my files are still there, so no harm, no foul.

[sphardy1]

Hi - how long did the expansion take in the end, from beginning to end?

[ThomColeman]

Expansion began Friday evening at 11 pm, and didn't finish until Monday evening at 9 pm. But there was a good 1.5 to 2 days (or more) where the expansion/resize process was hung until I power-cycled the ReadyNAS.

My scheduled maintenance (file system check, then a scan of the entire RAID) is scheduled to begin late Saturday evening each week. I got an email alert that the filesystem check was beginning, but never got an email that the filesystem check completed, which usually takes an hour or so. The RAID scan kicks off about 3 hours later on Sunday, and usually takes 7.5 hours.

It's possible the expansion/resize process and the filesystem check don't get along.

[sphardy1]

Thanks - So it appears it should take approx 2 days to expand 6 x 2TB to 6 x 3TB (excluding the replacement time per disk) - useful to know a rough estimate, especially vs the 2 weeks that were originally estimated (and would have defeated the purpose if true...)

[mdgm-ntgr]

Was this single redundancy or dual-redundancy? I guess two days does sound about right. The original estimates for resyncs, expansions etc. tend to way overestimate the time they'll take.

[kurfuer]

Hi,
I've a similar topic. Installed T75, after this I exchanged 2x2TB disk by 2x3TB disk (one by one of course), the Raid resync has been successfully for both disk. Then I rebootet to start the resize, the resizing process started, but after a while it stopped (but increased the size a bit).
When I boot the system now, it tells me, there is a unfinished resizing action, and starts to continue. But after ~20min the resizing stops (crash) again, leaving the size untouched. Did this 3x until now, but no other result.

Any idea?
thanks and regards
Dirk

some log entries from /var/log/frontview/status.log-old

# rem resinzing after first boot
1!!Wed Mar 16 07:57:23 CET 2011!!root!!Mit Laufwerkserweiterung oder -Migration fortfahren. Unterbrechen Sie das System nicht. Nach Abschluss des Vorgangs erhalten Sie eine Nachricht per E-Mail.
# rem first resize crashed
1!!Wed Mar 16 09:39:33 CET 2011!!root!!Erweiterung des Laufwerks fehlgeschlagen
# next boot => resize
1!!Wed Mar 16 14:49:02 CET 2011!!root!!Unvollständige Dateisystemerweiterung erfasst. Fortsetzen ...
# next crash
1!!Wed Mar 16 15:08:15 CET 2011!!root!!Erweiterung des Laufwerks fehlgeschlagen
# next boot => resize
1!!Wed Mar 16 15:17:24 CET 2011!!root!!Unvollständige Dateisystemerweiterung erfasst. Fortsetzen ...
# next crash
1!!Wed Mar 16 15:36:27 CET 2011!!root!!Erweiterung des Laufwerks fehlgeschlagen

I found this errors, but how to handle this???:

nas1:/var/log/frontview# grep -i "add group" expand_md.log
resize2fs: Operation not permitted While trying to add group #72960
resize2fs: Operation not permitted While trying to add group #72960
nas1:/var/log/frontview# grep -i gdt ../messages
Mar 16 09:39:29 nas1 kernel: EXT4-fs warning (device dm-0): ext4_group_add:746: No reserved GDT blocks, can't resize
Mar 16 15:08:12 nas1 kernel: EXT4-fs warning (device dm-0): ext4_group_add:746: No reserved GDT blocks, can't resize
Mar 16 15:36:24 nas1 kernel: EXT4-fs warning (device dm-0): ext4_group_add:746: No reserved GDT blocks, can't resize
Mar 16 16:10:39 nas1 kernel: EXT4-fs warning (device dm-0): ext4_group_add:746: No reserved GDT blocks, can't resize

[ThomColeman]

mdgm wrote:
Was this single redundancy or dual-redundancy? I guess two days does sound about right. The original estimates for resyncs, expansions etc. tend to way overestimate the time they'll take.

Single redundancy. After resizing I have about 13.8 TB total size reported.

[kurfuer]

move up; may be someboy can help me???

[bgrubb1]

I just finished building up a 6X3T array on a ultra6 plus and had a similar issue. Installed 4.2.16-T75 did a factory reset, and installed one at a time. Things when fine for the first 4 drives (with ~14 hours or re-stripe / re-sync) for each. Drive 5 ran the process, expanded to ~10.7 T, then reported back "Volume Expansion Failed". Upon re-booting, Raidar woupld report "Incomplete File System Expansion Detected. Resuming..." After 6 minutes, back to the "Volume Expansion Failed". After ~ 10 reboots with the same results, I tried a different drive for disk 5 and 14 hours later "Volume expansion Sucsessfull" Now heres where it gets weird. Upon installing brand new disk 6, 14 hours later I was stuck in the same "Volume Expansion Failed" loop. As a last effort, I installed the previously erroring out drive as drive 6, 14 hours later "Volume Expansion Sucsessfull".
So, my guess is the Hitachi drive is returning a non fatal error that is handled poorly by the Ultra6Plus.
..Barry

[Chewbacca]

We will look into this and get back to you.

ThomColeman: Can you send in your logs? Directions are in my signature.

[bgrubb1]

Chewbacca, do you want my logs too ??
..Barry

[Chewbacca]

bgrubb1: Sure send me your logs. Which Hitachi drive do you have?

[ThomColeman]

Chewbacca wrote:
We will look into this and get back to you. ThomColeman: Can you send in your logs? Directions are in my signature.

I will send my logs later today, Saturday, March 26.

EDIT: Logs sent.

[kurfuer]

I was able to proceed. First i installed lsof using "apt-get install lsof". Then I checked the programs which used the filesystem "lsof /dev/c/c" and stopped these processes (using the stop of the init-script /etc/init.d or using the kill command). If nomore entries in the output of "lsof /dev/c/c" then I was able to umount it "umount /c".
Now the resize ran successfully (resize2fs -pf /dev/c/c), and after the reboot the nas is doing a chkdsk right now (but i will see the result tomorrow in the morning).

[Jedi_Knight]

bgrubb1 wrote:

I just finished building up a 6X3T array on a ultra6 plus and had a similar issue. Installed 4.2.16-T75 did a factory reset, and installed one at a time. Things when fine for the first 4 drives (with ~14 hours or re-stripe / re-sync) for each. Drive 5 ran the process, expanded to ~10.7 T, then reported back "Volume Expansion Failed". Upon re-booting, Raidar woupld report "Incomplete File System Expansion Detected. Resuming..." After 6 minutes, back to the "Volume Expansion Failed". After ~ 10 reboots with the same results, I tried a different drive for disk 5 and 14 hours later "Volume expansion Sucsessfull" Now heres where it gets weird. Upon installing brand new disk 6, 14 hours later I was stuck in the same "Volume Expansion Failed" loop. As a last effort, I installed the previously erroring out drive as drive 6, 14 hours later "Volume Expansion Sucsessfull". So, my guess is the Hitachi drive is returning a non fatal error that is handled poorly by the Ultra6Plus. ..Barry

Barry,

You are seeing the current limitation of EXT4 64-bit filesystems (which we use on all of our x86_64 machines) cannot do online expansion by any more than 8TB over the life of the filesystem.
Horizontal and vertical expansion have no difference.

[Jedi_Knight]

Most of you are probably hitting the current limitation of EXT4 64-bit
filesystems (which we use on all of our x86_64 machines) cannot do
online expansion by any more than 8TB over the life of the filesystem.
Horizontal and vertical expansion have no difference. System hang/crash issue is completely different and it has been address in next firmware release.

[sphardy1]

Jedi Knight wrote:
Most of you are probably hitting the current limitation of EXT4 64-bit filesystems (which we use on all of our x86_64 machines) cannot do online expansion by any more than 8TB over the life of the filesystem. Horizontal and vertical expansion have no difference. System hang/crash issue is completely different and it has been address in next firmware release.

Is this a limitation that will (can?) be addressed in future?

As an aside: there appears to be more and more restrictions becoming known re. Online expansion that are not clearly documented. That and the length of time it takes for online expansion as disk sizes are increasing vs the time to restore data from a backup - I'm beginning to wonder just how worthwhile a feature this is?

Would really like to understand where Netgear is going with this feature

[kurfuer]

Hello Jedi knight,
thank you for the explanation.
I can confirm the offline extension (see my post before) and the following chkdsk solved the problem.
So may be it's a good idea to enable ssh by default :wink:
regards
Dirk

[ThomColeman]

I just expanded a second Ultra 6 from six 2TB drives to six 3 TB drives.

Before any 3 TB drives were added, I loaded 4.2.16-T75 software on the Ultra 6.

Once the system was back up after the software load, I then swapped each 2 TB drive for a 3 TB drive, allowing for full resync inbetween each drive swap. It took about 12.5 hours for each resync after a drive swap.

Once I had all six 3 TB drives installed and resynced, I rebooted the Ultra 6 so it would perform the expansion process to use the extra space.

The expansion process began at 7 pm, and finished 5.5 hours later at 12:35 am.

The final resync process finished at 5:42 am, 5 hours after the expansion ended.

So, once you have your new 3 TB drives installed in your Ultra 6, it will take about 10 hours and 45 minutes to resize and resync to use the extra space.

[Tes1]

Now all this talk about online extension and offline extension/expansion and factory reset.

Just to be sure.

It it posibel to make an offline extention/expansion, without loosing all the data.
And is there an easy way to do it?

I just looked aroud the forum, and I can see people talking about offline extension/expansion, but I can seem to find any guid or explanation for to get the NAS to do it.

"Kurfuer" is the one with the best explanation I have see.
But is that the only way to do it, and will it keep the data?

[Pittspilot]

After adding disks 5 and 6, I'm seeing the loop "Incompleted file system expansion detected. Resuming...", followed immediately by "Volume expansion failed". This cycles every two minutes at the moment.

Is there a way to successfully expand to use all six 3Tb drives?

Cheers

Jedi Knight wrote:

bgrubb1 wrote: I just finished building up a 6X3T array on a ultra6 plus and had a similar issue. Installed 4.2.16-T75 did a factory reset, and installed one at a time. Things when fine for the first 4 drives (with ~14 hours or re-stripe / re-sync) for each. Drive 5 ran the process, expanded to ~10.7 T, then reported back "Volume Expansion Failed". Upon re-booting, Raidar woupld report "Incomplete File System Expansion Detected. Resuming..." After 6 minutes, back to the "Volume Expansion Failed". After ~ 10 reboots with the same results, I tried a different drive for disk 5 and 14 hours later "Volume expansion Sucsessfull" Now heres where it gets weird. Upon installing brand new disk 6, 14 hours later I was stuck in the same "Volume Expansion Failed" loop. As a last effort, I installed the previously erroring out drive as drive 6, 14 hours later "Volume Expansion Sucsessfull". So, my guess is the Hitachi drive is returning a non fatal error that is handled poorly by the Ultra6Plus. ..Barry Barry, You are seeing the current limitation of EXT4 64-bit filesystems (which we use on all of our x86_64 machines) cannot do online expansion by any more than 8TB over the life of the filesystem. Horizontal and vertical expansion have no difference.

[StephenB]

It is best if you start your own thread on this. I also suggest opening a support case at support.netgear.com.

Do you recall how large your initial volume was (initial install or most recent factory default). How big are your drives and what model are they? Any errors in the SMART+ stats?

[BaGRoS]

---

@kurfuer wrote:
I was able to proceed. First i installed lsof using "apt-get install lsof". Then I checked the programs which used the filesystem "lsof /dev/c/c" and stopped these processes (using the stop of the init-script /etc/init.d or using the kill command). If nomore entries in the output of "lsof /dev/c/c" then I was able to umount it "umount /c".
Now the resize ran successfully (resize2fs -pf /dev/c/c), and after the reboot the nas is doing a chkdsk right now (but i will see the result tomorrow in the morning).

---

I need help, because

root@nas-JohnyB:~# apt-get install lsof
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
libupsclient1 libpcap0.8 cracklib-runtime libnss-mdns libcrack2 libcap-ng0
libavahi-client3 libnet1
Use 'apt-get autoremove' to remove them.
The following NEW packages will be installed:
lsof
0 upgraded, 1 newly installed, 0 to remove and 107 not upgraded.
Need to get 282 kB of archives.
After this operation, 455 kB of additional disk space will be used.
Err http://ftp.us.debian.org/debian/ squeeze/main lsof armel 4.81.dfsg.1-1
404 Not Found [IP: 128.61.240.89 80]
Failed to fetch http://ftp.us.debian.org/debian/pool/main/l/lsof/lsof_4.81.dfsg.1-1_armel.deb 404 Not Found [IP: 128.61.240.89 80]
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?