RND2000 Duo v1 - lockup after boot - OS partition full?

Chandler1 · ‎2017-12-19

(NB: reposting since it appears my original post never got published for some reason...? https://community.netgear.com/t5/Using-your-ReadyNAS/RND2000-Duo-v1-lockup-after-boot-OS-partition-f...)

Hello again forum!

TL;DR - think my OS partition is full. go to *** END STORY *** if you don't want to read my treatise Smiley Happy

*** START STORY ***

So my ReadyNAS has decided to stop working for me. Not sure when exactly it stopped working - we weren't home for the last week - but upon returning I found a NAS that I couldn't access (Network Shares, FrontView, RAIDar, ping, etc).

After troubleshooting networking issues, I attempted to power down the device via the power button to no avail. Left with no other option, I pulled the power (not much different to losing power, so...). Upon reconnection and powering up, I was overjoyed to see an email from my nas advising of the improper shutdown and a commencing RAID sync. Huzzah!

I then however find that whilst I was talking to it (and it me), it stopped responding (and access via the aforementioned methods ceases to work).

*** END STORY ***

NAS Details (sketchy as I haven't looked recently and I don't want to power it up just to get latest info)

Hardware: NETGEAR ReadyNAS Duo v1

Drives: 2 x 2TB WD RED WD20EFRX, RAID 1

Radiator Version: 4.1.15 (I think... it's been emailing me all the time to update to 4.1.16)

PC Details

Windows 10 Laptop, typically accessing via wirless (can jack in if required)

After a bit of Googling, I find a few posts that give me the impression that it is a full OS partition.

With this knowledge in hand, I'm now looking at fixing the issue but would like to have all the information at hand before I commence and potentiall brick my NAS. I do know my way around PCs, although am not familiar with *nix commands (have played with it a tiny bit many years a go on Red Hat GNOME & KDE, and I do have an Ubunto Live CD here somewhere...).

Sources:

Proposed process:

Remove Disk 2 (for data safety)
Boot NAS to Tech Support Mode (power button 20 seconds?)
SSH (Putty) to NAS - un: root / pw: infr8ntdebug
Mount partition:

# /bin/start_raid.sh
# mount /dev/hdc1 /sysroot

Confirm full partition:

# /sysroot/bin/df -h
# /sysroot/bin/df -i
# /sysroot/usr/bin/du -csh /sysroot/var/*
# /sysroot/usr/bin/du -csh /sysroot/var/log/*

Copy massive log/s to data parition as appropriate

# cp /sysroot/var/log/frontview/error.log   /???/backup_frontview_error.log
(change /frontview/error.log as appropriate, not sure on destination file/location?)

Empty log/s as required:

# echo > /sysroot/var/log/frontview/error.log
(change /frontview/error.log as appropriate)

Come back here when it doesn't work

mdgm-ntgr · ‎2017-12-19

There are multiple possibilities and a full root volume is one of them and perhaps the most likely.

If the NAS is in the middle of a resync then only one of the disks would be usable for getting at the data anyway.

It's situations like the one you've run into here that show some reasons why it's important to backup your data.

Chandler1 · ‎2017-12-20

Thanks mdgm.

So my plan is good? (presuming it is a full root...)

Can you advise on copying the large logs to the data partition? (I'm not a Linux user, although I'm working on it!)

# cp /sysroot/var/log/frontview/error.log   /???/backup_frontview_error.log

StephenB · ‎2017-12-20

First check that the OS partition is full

# cd //

# df . -h

# df . -i

The first df will tell you the space usage on the OS partition. Normally it would be no more than 25%

The second df will tell you the inode usage on the OS partition. Inodes are used for metadata, and if you run out you won't be able to create new files on the OS partition even if there is free space.

Assuming you are running out of space:

You'd put the oversize logs on the C volume. The best thing to do is park them in a share.

So first create a subfolder in a share

#mkdir /c/sharename/log_files

Then

# cp /var/log/frontview/error.log /c/sharename/log_files/backup_frontview_error.log

Chandler1 · ‎2017-12-20

Thanks StephenB.

I'll give it a go over the weekend and report back!

StephenB · ‎2017-12-20

If you are in tech support mode, you will need to mount the C volume.

I believe the following will do that (hopefully @mdgm-ntgr will correct this if it is in correct).

mdgm-ntgr · ‎2017-12-20

No, not that. Something like:

# mount --bind /proc /sysroot/proc
# mount --bind /dev /sysroot/dev
# mount --bind /dev/pts /sysroot/dev/pts
# chroot /sysroot /bin/bash
# vgscan
# vgchange -a y
# mount /dev/c/c /c

Chandler1 · ‎2017-12-26

Thank you for the assistance mdgm and StephenB.

I've now had the opportunity to investigate further and successfully boot the NAS into tech support mode and connect via PuTTY; df outputs below.

root@:/# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/hdc1             1.9G  359M  1.5G  19% /
sysfs                 1.9G  359M  1.5G  19% /sys
tmpfs                 1.9G  359M  1.5G  19% /ramfs
tmpfs                 1.9G  359M  1.5G  19% /USB
/dev/c/c              1.8T  178G  1.6T  10% /c
/dev/c/c              1.8T  178G  1.6T  10% /c

root@:/#  df -i
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/hdc1             128000    8079  119921    7% /
sysfs                 128000    8079  119921    7% /sys
tmpfs                 128000    8079  119921    7% /ramfs
tmpfs                 128000    8079  119921    7% /USB
/dev/c/c             60981248   37340 60943908    1% /c
/dev/c/c             60981248   37340 60943908    1% /c

Presuming I'm reading that correctly, disc space is not my issue, with 93% inodes and 81% space free.

Next step?

Chandler1 · ‎2017-12-27

Further update:

After shutting down the NAS and booting back up normally, it was visible on RAIDar and I logged in via Frontview successfully.

Frontview logs showed nothing of note - improper shutdown, volume scan found and corrected errors
SMART+ for disk 1 details of note:
- Raw Read Error Rate: 45
- Currently Pending Sector: 1
- Lp stat events: 1

Disk 2 is currently out of the box - could it have been causing the isssue...?

StephenB · ‎2017-12-27

It's certainly not a full OS partition.

Failing disks cause a variety of issues, you probably should test them with lifeguard.

Chandler1 · ‎2017-12-28

I'm baaack!

So I (for some reason) decided to give it the benefit of the doubt yesterday and left it on unresponsive to see if it was just the interface locked up due to the RAID sync, but after ~24hrs still no joy (NB: power light steady on, disk 1 steady on, disk 2 blinking steadily).

So I pulled the plug, removed disk 2 and powered up again. Successful boot (as expected), same lights on front (power steady, D1 steady, D2 blinking... odd).

Looking in RAIDar it reports that disk 2 is dead, even though it's not installed (is that normal?)

Data on drive 1 seems fine after a quick browse and sampling a couple of random files.

With the drive out, I've loaded it into a caddy and run the WD Data Lifeguard software on it, results below.

SMART Status (on main screen) was a PASS upon opening the software.
SMART information in screenshot below.

Looking at the actual results however, it looks like the Reallocated sector count is 200, and the value to call the drive faulty is 140, which to me says the drive is faulty? (but it passed, so I presume I'm reading something wrong?)
QUICK Test Results - PASS.
EXTENDED Test Results - .currently running, will add when complete (ETA 3hrs, ~0500 GMT).

Presuming the extended test comes back fine, thoughts on next step?

Personally I'm thinking use the WD tool to wipe drive 2 and do a hot insert. Or perhaps try a hot insert first?

Chandler1 · ‎2017-12-29

Full test results:

Test Option:	QUICK TEST
Model Number:	WDC WD20EFRX-68EUZN0
Unit Serial Number:
Firmware Number:	82.00A82
Capacity:	2000.40 GB
SMART Status:	PASS
Test Result:	PASS
Test Time:	11:26:10, December 29, 2017

Test Option:	EXTENDED TEST
Model Number:	WDC WD20EFRX-68EUZN0
Unit Serial Number:
Firmware Number:	82.00A82
Capacity:	2000.40 GB
SMART Status:	PASS
Test Result:	PASS
Test Time:	18:18:17, December 29, 2017

StephenB · ‎2017-12-29

I suggest also trying the destructive write test in Lifeguard. I've sometimes found that it will turn up issues that the non-destructive test misses.

Also the PASS/FAIL thresholds are chosen by WDC (and they certainly have an incentive to reduce warranty returns). Generally I look at the raw stats, and determine for myself if I think they are ok. I take out drives if the sum of pending sectors+reallocated sectors exceeds ~50, or if it is regularly growing.

You might want to confirm the reallocated sector count by downloading the log zip file from the NAS and looking in disk_smart.log.

BotanyBay · ‎2017-12-29

I concur, I am more aggressive though. When I but a new drive it gets
1) read all sectors (40 hours for 10tb)
2) erase all tracks (30 hours for 10tb)
3) read all sectors (40 hours for 10tb)
I.e. 110 hours or 5 days of test.

If I have a bad sector or pending it goes back to source.

If I see a single sector pop up bad I watch closely, another one within 3 months and it gets pulled.

Total of 3 bad and it is gone.

Still useful for external backups.

The NAS does count ata errors and associates them with the disk, not slot. I do have one drive which accumulated 5 in one slot over a couple of scrubs, reseated drive, same result, now 10 ata errors. Moved from slot 3 to slot 1 and has been stable at 10 for over a year. Drive in slot 3 shows 0. This info is stored in an is file which is where the NAS keeps track I believe.

Err on the side of caution, replace early and make rotating backups!

StephenB · ‎2017-12-30

@BotanyBay wrote:
I concur, I am more aggressive though. When I but a new drive ...
If I have a bad sector or pending it goes back to source.

I will also return new drives if I see any reallocated or pending sectors.

I'm slower on replacements, but at the moment have no disks with reallocated or pending sectors in my primary or backup NAS. I have some with a couple of ATA errors, but they haven't increased since the initial deployment.

@BotanyBay wrote:
This info is stored in an is file which is where the NAS keeps track I believe.

On OS-6 systems you can find it in disk_info.log. SMART stats are also maintained in the drives themselves (and are detected by the drive firmware, not the NAS).

BotanyBay · ‎2017-12-30

Do you know if the ATA errors are polled from the drive or stored by the NAS? It almost appears that the NAS is keeping that info stored by serial number as I moved one with about 10 AYA errors to a different chassis and it is showing 0 now.

StephenB · ‎2017-12-30

Here's the most complete list I know of: https://en.wikipedia.org/wiki/S.M.A.R.T.

ATA errors aren't on it, so I believe these are detected by the NAS itself.

RND2000 Duo v1 - lockup after boot - OS partition full?

RND2000 Duo v1 - lockup after boot - OS partition full?

Re: RND2000 Duo v1 - lockup after boot - OS partition full?

Re: RND2000 Duo v1 - lockup after boot - OS partition full?

Re: RND2000 Duo v1 - lockup after boot - OS partition full?

Re: RND2000 Duo v1 - lockup after boot - OS partition full?

Re: RND2000 Duo v1 - lockup after boot - OS partition full?

Re: RND2000 Duo v1 - lockup after boot - OS partition full?

Re: RND2000 Duo v1 - lockup after boot - OS partition full?

Re: RND2000 Duo v1 - lockup after boot - OS partition full?

Re: RND2000 Duo v1 - lockup after boot - OS partition full?

Re: RND2000 Duo v1 - lockup after boot - OS partition full?

Re: RND2000 Duo v1 - lockup after boot - OS partition full?

Re: RND2000 Duo v1 - lockup after boot - OS partition full?

Re: RND2000 Duo v1 - lockup after boot - OS partition full?

Re: RND2000 Duo v1 - lockup after boot - OS partition full?

Re: RND2000 Duo v1 - lockup after boot - OS partition full?

Re: RND2000 Duo v1 - lockup after boot - OS partition full?