Re: Readynas NV+ stuck at booting - missing shares found

ClausH42 · ‎2015-10-08

Hello,

i have an issue with on of my NV+ v1 which worked for long without an issue (configuration X-Raid). Some days ago when i checked the display after not being able to access the data i found the message booting on the display and it stayed for hours. I restarted the NV+ and found the system stuck again in booting. I did a OS reinstall and the system booted and performed a FS check. Raidar showed 100%checked and all HDD LEDs in green. I restarted and the system stucked again.

I tried to check if the HDDs are causing this and removed slot 1 booted, no change, reinserted, removed slot 2, and so on. Repeated this for all 4 slots. Booting never finished.

The result now is: System starts, Booting and after some minutes it shows "Booting, done". I can reach it via putty, did a check of the smart info of the HDDs and found nothing obvious ("smartctl"). Since some days,after every boot i receive now a message that shares are not accessible (media, backup, music, etc.).

Any idea how i can access my data again or how to rework this?

I have a second NV+ and a Ultra6 available (ultra6 will not help in this case). The disks in the NV+ are WD20EZRX, the green ones but i switched them all to idle time 300 and load cycle count did not increase significantly during the last years.

Can i transfer the disks to the 2nd NV+ and try if it is running there? If not please give me an information how i can access my data again.

Thank you in advance

Readynas NV+ v1 with 4 WD20EZRX and X-Raid latest firmware 4.14

ClausH42 · ‎2015-10-10

I had a look at the etc/fstab using vi and found a large discrepancy between the fstab of my two NV+

the nv+ that does not finish boot:

***** File system check performed at Sun Oct  4 13:28:32 CEST 2015 *****
fsck 1.40.11 (17-June-2008)
WARNING: bad format on line 2 of /etc/fstab
WARNING: bad format on line 4 of /etc/fstab
WARNING: bad format on line 5 of /etc/fstab
WARNING: bad format on line 6 of /etc/fstab
WARNING: bad format on line 7 of /etc/fstab
WARNING: bad format on line 8 of /etc/fstab
WARNING: bad format on line 9 of /etc/fstab
WARNING: bad format on line 10 of /etc/fstab
WARNING: bad format on line 11 of /etc/fstab
WARNING: bad format on line 12 of /etc/fstab
WARNING: bad format on line 13 of /etc/fstab
WARNING: bad format on line 14 of /etc/fstab
e2fsck 1.40.11 (17-June-2008)
fsck.ext3: No such file or directory while trying to open /dev/c/c^M

The superblock could not be read or does not describe a correct ext2
filesystem.  If the device is valid and it really contains an ext2
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
    e2fsck -b 8193 <device>

- fstab 1/33 3%
fsck 1.40.11 (17-June-2008)
WARNING: bad format on line 2 of /etc/fstab
WARNING: bad format on line 4 of /etc/fstab
WARNING: bad format on line 5 of /etc/fstab
WARNING: bad format on line 6 of /etc/fstab
WARNING: bad format on line 7 of /etc/fstab
WARNING: bad format on line 8 of /etc/fstab
WARNING: bad format on line 9 of /etc/fstab
WARNING: bad format on line 10 of /etc/fstab
WARNING: bad format on line 11 of /etc/fstab
WARNING: bad format on line 12 of /etc/fstab
WARNING: bad format on line 13 of /etc/fstab
WARNING: bad format on line 14 of /etc/fstab
e2fsck 1.40.11 (17-June-2008)
fsck.ext3: No such file or directory while trying to open /dev/c/c^M

The superblock could not be read or does not describe a correct ext2
filesystem.  If the device is valid and it really contains an ext2
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
    e2fsck -b 8193 <device>




^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
/dev/hdc2       none    swap    sw                              0 0
/dev/hde2       none    swap    sw                              0 0
/dev/hdg2       none    swap    sw                              0 0
proc            /proc   proc    defaults                        0 0
/dev/c/c        /c      ext3    defaults,noatime,user_xattr,acl,user_xattr,user_
/dev/hdc1       /       ext3    defaults,noatime                        0 1

and the nv+ that is up and running:

#-----------------------------------------------------------------------------
# <device>         <mount>              <type>     <options>    <freq> <pass>
#-----------------------------------------------------------------------------
/dev/hdc1          /                    ext2       defaults,noatime      0
proc               /proc                proc       defaults          0      0
/dev/hde2          swap                 swap       defaults          0      0
/dev/hdc2          swap                 swap       defaults          0      0
/dev/hdg2          swap                 swap       defaults          0      0
/dev/c/c           /c                   ext2       defaults,acl,user_xattr,usrqu

the one that is not running shows an ext3 filesystem, whereas the running one shows ext2. As far as i know i never changed this, but what i find more interesting is the long text before the fstab data starts. Even if i do not understand what it tries to tell me

View solution in original post

StephenB · ‎2015-10-09

You can migrate the disks to the other NAS for testing. Just label the disks by system/slot, and migrate powered down.

But since you do have ssh still working, I suggest first logging in and entering

df . -h

df . -i

I am thinking a full OS partition

ClausH42 · ‎2015-10-09

Thank you so far,

i will try this as soon as i am at home

ClausH42 · ‎2015-10-09

Hello,

i tried the df command and got a response, but this says nothing to me:

nas-1:/# df . -h
Filesystem            Size Used Avail Use% Mounted on
-                  1.9G 1.9G   66M 97% /
nas-1:/# df . -i
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
-                    128000   14543 113457   12% /

I see that 97% is almost full, but the NAS has by far more diskspace. So i assume this is the room for the OS. Any idea how to correct this.

StephenB · ‎2015-10-09

@ClausH42 wrote:

Hello,

i tried the df command and got a response, but this says nothing to me:

nas-1:/# df . -h
Filesystem            Size Used Avail Use% Mounted on
-                  1.9G 1.9G   66M 97% /
nas-1:/# df . -i
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
-                    128000   14543 113457   12% /

I see that 97% is almost full, but the NAS has by far more diskspace. So i assume this is the room for the OS. Any idea how to correct this.

The OS partition on the NV+ is only 2 GB. Normally it is around 20-25% full. 97% full certainly creates performance issues.

The next step is to find out where space is going. Look in /var/log and /var/cache first. It they are not the culprit, you'll need to poke around until you find them.

You can truncate run-away logs using "echo > filename"

You should be able to simply delete files in the cache, but don't delete folders. You can delete everything in /var/cache/minidlna/art_cache/c/ (including folders)

If you use DLNA, then there is an add-on to shift its cache to the "c" volume. If you want a link to that, let us know.

ClausH42 · ‎2015-10-09

I found besides other files:

total 335344
-rw-r--r--    1 root     root       329383 Okt  9 17:39 auth.log
-rw-r--r--    1 root     root     13256492 Okt  9 15:57 kern.log
-rw-r--r--    1 root     root     13386811 Okt  9 15:57 messages
-rw-r--r--    1 root     root     40807755 Sep 25 20:38 mt-daapd.log
-rw-r--r--    1 root     root      1130496 Sep 27 11:53 ReadyNAS_Remote.log
-rw-r--r--    1 root     root     20817763 Okt  9 15:59 syslog
-rw-r--r--    1 root     root     252985911 Sep 27 11:46 upnp-av.log

and in minidlna

-rw-r--r--    1 root     root     538775552 Sep 25 20:35 files.db

i will truncate them as you recommended and see what happens. Roughly calculated this is more than 1 GB already

Filesystem            Size  Used Avail Use% Mounted on
-                           1.9G 1019M  966M  52% /

Even though it seemed to be more than 1 GB i could only free up 966M

After reboot i still can only access the os partition. FSCK shows errors:

fsck 1.40.11 (17-June-2008)
e2fsck 1.40.11 (17-June-2008)
Warning!  /dev/hdc1 is mounted.
/dev/hdc1 contains a file system with errors, check forced.
Pass 1: Checking inodes, blocks, and sizes
Deleted inode 3002 has zero dtime.  Fix? no

Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Free blocks count wrong (61834, counted=61818).
Fix? no

Inode bitmap differences:  -3002
Fix? no

Free inodes count wrong (116672, counted=116659).
Fix? no


/dev/hdc1: ********** WARNING: Filesystem still has errors **********

/dev/hdc1: 11328/128000 files (1.2% non-contiguous), 66166/128000 blocks

i get a warning message that fsck might cause severe problems if it corrects the errors. I am not sure if i should run this next.

Raidar shows the NV+, Frontview does not work, Windows Explorer shows the NV+ too but it is not possible to access any folder.

What can i try next?

StephenB · ‎2015-10-09

I would reboot next, and probably pass on the fsck repair. You should have enough space to restore services.

It is a bit scary to have a corrupt root FS. If you get back CIFS access you should back things up. Then you can try the fsck, and if it goes wrong you can do a reset.

ClausH42 · ‎2015-10-10

So i rebooted result was the same "Booting done" but only accessible via Putty.

I switched the discs between my 2 NV+ v1. The discs from the nas that still worked are performing a FS check in the new position. The discs from the failed nas are still not accessible. The System says again "booting done" and nothing can be accessed as the raid (X-Raid) is not mounted. I searched the forum but i could not find a howto for mounting the shares from ssh.

Can you please describe what i have to do in putty to mount the shares? I would have enough space to transfer the data to my Ultra6 and then do a factory reset. Did i mention that pulling disc by disc (remaining 3 discs in position) did not lead to a system that booted.

Please tell me howto mount the shares if this is possible. I can restore data from different backups, but i invested a hell of time in sorting and tagging music and pictures.

ClausH42 · ‎2015-10-10

I had a look at the etc/fstab using vi and found a large discrepancy between the fstab of my two NV+

the nv+ that does not finish boot:

***** File system check performed at Sun Oct  4 13:28:32 CEST 2015 *****
fsck 1.40.11 (17-June-2008)
WARNING: bad format on line 2 of /etc/fstab
WARNING: bad format on line 4 of /etc/fstab
WARNING: bad format on line 5 of /etc/fstab
WARNING: bad format on line 6 of /etc/fstab
WARNING: bad format on line 7 of /etc/fstab
WARNING: bad format on line 8 of /etc/fstab
WARNING: bad format on line 9 of /etc/fstab
WARNING: bad format on line 10 of /etc/fstab
WARNING: bad format on line 11 of /etc/fstab
WARNING: bad format on line 12 of /etc/fstab
WARNING: bad format on line 13 of /etc/fstab
WARNING: bad format on line 14 of /etc/fstab
e2fsck 1.40.11 (17-June-2008)
fsck.ext3: No such file or directory while trying to open /dev/c/c^M

The superblock could not be read or does not describe a correct ext2
filesystem.  If the device is valid and it really contains an ext2
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
    e2fsck -b 8193 <device>

- fstab 1/33 3%
fsck 1.40.11 (17-June-2008)
WARNING: bad format on line 2 of /etc/fstab
WARNING: bad format on line 4 of /etc/fstab
WARNING: bad format on line 5 of /etc/fstab
WARNING: bad format on line 6 of /etc/fstab
WARNING: bad format on line 7 of /etc/fstab
WARNING: bad format on line 8 of /etc/fstab
WARNING: bad format on line 9 of /etc/fstab
WARNING: bad format on line 10 of /etc/fstab
WARNING: bad format on line 11 of /etc/fstab
WARNING: bad format on line 12 of /etc/fstab
WARNING: bad format on line 13 of /etc/fstab
WARNING: bad format on line 14 of /etc/fstab
e2fsck 1.40.11 (17-June-2008)
fsck.ext3: No such file or directory while trying to open /dev/c/c^M

The superblock could not be read or does not describe a correct ext2
filesystem.  If the device is valid and it really contains an ext2
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
    e2fsck -b 8193 <device>




^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
/dev/hdc2       none    swap    sw                              0 0
/dev/hde2       none    swap    sw                              0 0
/dev/hdg2       none    swap    sw                              0 0
proc            /proc   proc    defaults                        0 0
/dev/c/c        /c      ext3    defaults,noatime,user_xattr,acl,user_xattr,user_
/dev/hdc1       /       ext3    defaults,noatime                        0 1

and the nv+ that is up and running:

#-----------------------------------------------------------------------------
# <device>         <mount>              <type>     <options>    <freq> <pass>
#-----------------------------------------------------------------------------
/dev/hdc1          /                    ext2       defaults,noatime      0
proc               /proc                proc       defaults          0      0
/dev/hde2          swap                 swap       defaults          0      0
/dev/hdc2          swap                 swap       defaults          0      0
/dev/hdg2          swap                 swap       defaults          0      0
/dev/c/c           /c                   ext2       defaults,acl,user_xattr,usrqu

the one that is not running shows an ext3 filesystem, whereas the running one shows ext2. As far as i know i never changed this, but what i find more interesting is the long text before the fstab data starts. Even if i do not understand what it tries to tell me

ClausH42 · ‎2015-10-27

It seems i am now closer to a solution saving all my data. I rebuild the fstab according to the fstab of my second NV+ and booted it. Raidar recognized it with a dead X-Raid. I did not start the rebuild but i started copying data to my second NV+ and my Readynas Ultra which has enough free space.

After saving all the relevant data i will do a factory reset and rebuild the X-Raid.