× NETGEAR will be terminating ReadyCLOUD service by July 1st, 2023. For more details click here.
Orbi WiFi 7 RBE973
Reply

Re: ReadyNAS NV+ Stuck on Booting

DOLMark
Aspirant

ReadyNAS NV+ Stuck on Booting

Hi

 

Got a clients NV+ that'd hung completely. It responded to pings but nothing else.

 

Have rebooted it and I get;

 

- All Drives led's green

 

- Can Ping device

 

- No HTTP access

 

- No SSH access

 

- No sign of it on raidar (under OSX)

 

- LCD Display says "Booting ..." followed by "Checking FS"

 

No %'age display and no activity light after a short while

 

 

If I reboot using the "Skip filesystem checks" option I get exactly the same apart from;

 

- LCD Display shows Booting ...

 

- Power Blue LED goes from Pulsing to Solid and then Pulsing again etc

 

- When it does this the ping goes;

 

64 bytes from 192.168.1.251: icmp_seq=861 ttl=64 time=1.049 ms
64 bytes from 192.168.1.251: icmp_seq=862 ttl=64 time=9.335 ms
64 bytes from 192.168.1.251: icmp_seq=863 ttl=64 time=1.801 ms
64 bytes from 192.168.1.251: icmp_seq=864 ttl=64 time=1.561 ms
64 bytes from 192.168.1.251: icmp_seq=865 ttl=64 time=1.534 ms
64 bytes from 192.168.1.251: icmp_seq=866 ttl=64 time=0.873 ms
64 bytes from 192.168.1.251: icmp_seq=867 ttl=64 time=2.184 ms
64 bytes from 192.168.1.251: icmp_seq=868 ttl=64 time=1.646 ms
64 bytes from 192.168.1.251: icmp_seq=869 ttl=64 time=6952.255 ms
64 bytes from 192.168.1.251: icmp_seq=870 ttl=64 time=5949.652 ms
64 bytes from 192.168.1.251: icmp_seq=871 ttl=64 time=4948.032 ms
64 bytes from 192.168.1.251: icmp_seq=872 ttl=64 time=3944.158 ms
64 bytes from 192.168.1.251: icmp_seq=873 ttl=64 time=2943.767 ms
64 bytes from 192.168.1.251: icmp_seq=874 ttl=64 time=1943.396 ms
64 bytes from 192.168.1.251: icmp_seq=875 ttl=64 time=943.577 ms
64 bytes from 192.168.1.251: icmp_seq=876 ttl=64 time=27.301 ms
64 bytes from 192.168.1.251: icmp_seq=877 ttl=64 time=4.686 ms
64 bytes from 192.168.1.251: icmp_seq=878 ttl=64 time=0.972 ms
64 bytes from 192.168.1.251: icmp_seq=879 ttl=64 time=0.898 ms
64 bytes from 192.168.1.251: icmp_seq=880 ttl=64 time=1.633 ms

 

Which is possibly it watchdogging?

 

 

Device has 4 drives in it.

 

 

Help?  I've run out of ideas 😞

 

Mark

Model: ReadyNAS-NV+|ReadyNAS NV+
Message 1 of 13
StephenB
Guru

Re: ReadyNAS NV+ Stuck on Booting

Try powering down, removing the disks (labeling by slot).  Then power up, and confirm that RAIDar can see the NAS (no disks status).

 

If you have access to a windows machine, try testing the disks with vendor tools (Lifeguard for Western Digital, Seatools for Seagate)

 

FWIW, the guidance in the readynas led language document says:

ReadyNAS OS is booting, this stage should not last for more than 3 minutes. Otherwise, one of disk may be bad or incompatible. SATA backplane is the next to be questioned, and then NAND or SODIMM.

 

So maybe try running the memory test while the disks are being tested.

 

 

Message 2 of 13
DOLMark
Aspirant

Re: ReadyNAS NV+ Stuck on Booting

Hi Stephen

 

Thanks for the quick reply.

 

With drives removed OSX Raidar does indeed see it in "No disks detected" status

 

Memory test is as far as "checking pat 2" without complaining yet

 

Guess I need to find a windows machine to borrow to test the drives.  Everything here is OSX or Ubuntu.

 

Mark

Message 3 of 13
StephenB
Guru

Re: ReadyNAS NV+ Stuck on Booting

If you can't find a windows machine, you should still be able to run a SMART diag with smartctl.

Message 4 of 13
DOLMark
Aspirant

Re: ReadyNAS NV+ Stuck on Booting

Hi Again

 

Managed to get a USB<->SATA cable to connect the drives to a Ubuntu machine, and pulled off the following;

 

All drives pass the SMARTCTL health request.

 

Drives 1(Left), 2 and 3 show partitions.  Drive 4 (right) doesn't.

 

Now customer doesn't actually know how these were configured. They thought it was standard x-raid across all drives but may have been raid on 3 drives + hot spare.  If that'd explain the drive that looks empty.

 

Is there any way to tell how these were configured?

 

Thanks

 

Mark

————————————————————————————————————

Drive 1 Left

[Fri Jun 30 15:31:30 2017] scsi 2:0:0:0: Direct-Access ST2000VN W1H1ND0N PQ: 0 ANSI: 2 CCS
[Fri Jun 30 15:31:30 2017] sd 2:0:0:0: Attached scsi generic sg2 type 0
[Fri Jun 30 15:31:30 2017] sd 2:0:0:0: [sdb] 3907029168 512-byte logical blocks: (2.00 TB/1.82 TiB)
[Fri Jun 30 15:31:30 2017] sd 2:0:0:0: [sdb] Write Protect is off
[Fri Jun 30 15:31:30 2017] sd 2:0:0:0: [sdb] Mode Sense: 00 38 00 00
[Fri Jun 30 15:31:30 2017] sd 2:0:0:0: [sdb] Asking for cache data failed
[Fri Jun 30 15:31:30 2017] sd 2:0:0:0: [sdb] Assuming drive cache: write through
[Fri Jun 30 15:31:30 2017] sdb: sdb1 sdb2 sdb3 < sdb5 >
[Fri Jun 30 15:31:30 2017] sd 2:0:0:0: [sdb] Attached SCSI disk

# fdisk -l /dev/sdb
Disk /dev/sdb: 1.8 TiB, 2000398934016 bytes, 3907029168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x00000000

Device Boot Start End Sectors Size Id Type
/dev/sdb1 32 4096031 4096000 2G 83 Linux
/dev/sdb2 4096032 4608031 512000 250M 82 Linux swap / Solaris
/dev/sdb3 4608032 3906992335 3902384304 1.8T 5 Extended
/dev/sdb5 4608040 3906992335 3902384296 1.8T 8e Linux LVM


# smartctl -H /dev/sdb
smartctl 6.6 2016-05-31 r4324 [i686-linux-4.8.0-45-generic] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

————————————————————————————————————

Drive 2 Middle-Left

[Fri Jun 30 15:33:57 2017] scsi 2:0:0:0: Direct-Access ST2000VN W1H1NWYC PQ: 0 ANSI: 2 CCS
[Fri Jun 30 15:33:57 2017] sd 2:0:0:0: Attached scsi generic sg2 type 0
[Fri Jun 30 15:33:57 2017] sd 2:0:0:0: [sdb] 3907029168 512-byte logical blocks: (2.00 TB/1.82 TiB)
[Fri Jun 30 15:33:57 2017] sd 2:0:0:0: [sdb] Write Protect is off
[Fri Jun 30 15:33:57 2017] sd 2:0:0:0: [sdb] Mode Sense: 00 38 00 00
[Fri Jun 30 15:33:57 2017] sd 2:0:0:0: [sdb] Asking for cache data failed
[Fri Jun 30 15:33:57 2017] sd 2:0:0:0: [sdb] Assuming drive cache: write through
[Fri Jun 30 15:33:57 2017] sdb: sdb1 sdb2 sdb3 < sdb5 >
[Fri Jun 30 15:33:57 2017] sd 2:0:0:0: [sdb] Attached SCSI disk


# fdisk -l /dev/sdb
Disk /dev/sdb: 1.8 TiB, 2000398934016 bytes, 3907029168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x00000000

Device Boot Start End Sectors Size Id Type
/dev/sdb1 32 4096031 4096000 2G 83 Linux
/dev/sdb2 4096032 4608031 512000 250M 82 Linux swap / Solaris
/dev/sdb3 4608032 3906992335 3902384304 1.8T 5 Extended
/dev/sdb5 4608040 3906992335 3902384296 1.8T 8e Linux LVM

# smartctl -H /dev/sdb
smartctl 6.6 2016-05-31 r4324 [i686-linux-4.8.0-45-generic] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

————————————————————————————————————

Drive 3 Middle Right

[Fri Jun 30 15:36:02 2017] scsi 2:0:0:0: Direct-Access ST2000VN W1H1P1KZ PQ: 0 ANSI: 2 CCS
[Fri Jun 30 15:36:02 2017] sd 2:0:0:0: Attached scsi generic sg2 type 0
[Fri Jun 30 15:36:02 2017] sd 2:0:0:0: [sdb] 3907029168 512-byte logical blocks: (2.00 TB/1.82 TiB)
[Fri Jun 30 15:36:02 2017] sd 2:0:0:0: [sdb] Write Protect is off
[Fri Jun 30 15:36:02 2017] sd 2:0:0:0: [sdb] Mode Sense: 00 38 00 00
[Fri Jun 30 15:36:02 2017] sd 2:0:0:0: [sdb] Asking for cache data failed
[Fri Jun 30 15:36:02 2017] sd 2:0:0:0: [sdb] Assuming drive cache: write through
[Fri Jun 30 15:36:02 2017] sdb: sdb1 sdb2 sdb3 < sdb5 >
[Fri Jun 30 15:36:02 2017] sd 2:0:0:0: [sdb] Attached SCSI disk

# fdisk -l /dev/sdb
Disk /dev/sdb: 1.8 TiB, 2000398934016 bytes, 3907029168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x00000000

Device Boot Start End Sectors Size Id Type
/dev/sdb1 32 4096031 4096000 2G 83 Linux
/dev/sdb2 4096032 4608031 512000 250M 82 Linux swap / Solaris
/dev/sdb3 4608032 3906992335 3902384304 1.8T 5 Extended
/dev/sdb5 4608040 3906992335 3902384296 1.8T 8e Linux LVM

# fdisk -l /dev/sdb
Disk /dev/sdb: 1.8 TiB, 2000398934016 bytes, 3907029168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x00000000

Device Boot Start End Sectors Size Id Type
/dev/sdb1 32 4096031 4096000 2G 83 Linux
/dev/sdb2 4096032 4608031 512000 250M 82 Linux swap / Solaris
/dev/sdb3 4608032 3906992335 3902384304 1.8T 5 Extended
/dev/sdb5 4608040 3906992335 3902384296 1.8T 8e Linux LVM
root@bluestripe:~# smartctl -H /dev/sdb
smartctl 6.6 2016-05-31 r4324 [i686-linux-4.8.0-45-generic] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

————————————————————————————————————

Drive 4 Right

[Fri Jun 30 15:37:36 2017] scsi 2:0:0:0: Direct-Access ST2000VN W1H1N9Q3 PQ: 0 ANSI: 2 CCS
[Fri Jun 30 15:37:36 2017] sd 2:0:0:0: Attached scsi generic sg2 type 0
[Fri Jun 30 15:37:36 2017] sd 2:0:0:0: [sdb] 3907029168 512-byte logical blocks: (2.00 TB/1.82 TiB)
[Fri Jun 30 15:37:37 2017] sd 2:0:0:0: [sdb] Write Protect is off
[Fri Jun 30 15:37:37 2017] sd 2:0:0:0: [sdb] Mode Sense: 00 38 00 00
[Fri Jun 30 15:37:37 2017] sd 2:0:0:0: [sdb] Asking for cache data failed
[Fri Jun 30 15:37:37 2017] sd 2:0:0:0: [sdb] Assuming drive cache: write through
[Fri Jun 30 15:37:37 2017] sd 2:0:0:0: [sdb] Attached SCSI disk


# fdisk -l /dev/sdb
Disk /dev/sdb: 1.8 TiB, 2000398934016 bytes, 3907029168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

# smartctl -H /dev/sdb
smartctl 6.6 2016-05-31 r4324 [i686-linux-4.8.0-45-generic] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

 

Model: ReadyNAS-NV+|ReadyNAS NV+
Message 5 of 13
StephenB
Guru

Re: ReadyNAS NV+ Stuck on Booting

The NV+ uses raid-4 (a parity disk instead of distributed parity blocks).  The parity is written to an unpartitioned section of the disk.

 

So this is consistent with the old XRAID.

 

The NV+ has a boot menu option to skip the volume check on startup.  Did you try that?  As a reminder, be sure to shut down the NAS before reinserting the disk drives.

Message 6 of 13
DOLMark
Aspirant

Re: ReadyNAS NV+ Stuck on Booting

Hi

 

Yup, tried the skip FS Checks and it just hung at "Booting ...".  I gave it about 30 mins before pulling the power.

 

Mark

Message 7 of 13
StephenB
Guru

Re: ReadyNAS NV+ Stuck on Booting

You can try mounting the array on the ubuntu system - there is a guide here: http://home.bott.ca/webserver/?p=306

Message 8 of 13
DOLMark
Aspirant

Re: ReadyNAS NV+ Stuck on Booting

Hi

 

Well after a brief pause to get the right cables etc to arrive so I could get the 4 drives in to another Ubuntu server everything's looking promising... although puzzling 🙂

 

So I now have the existing 4 drive array mounted under standard Ubuntu.  No drive errors that I can see.

 

(command line walk through and details below)

 

So The drives look ok. Mount ok.  I'm still none the wiser about why the redynas wont go past " Checking FS ..."

 

Dare I ask; "Next suggestion?"

 

Mark

 

 

For the benefit of furture visitors and diagnostics sake;

 

OS is Ubuntu Workstation v16.10

 

SDA = Existing Boot drive

SDB to SDE = Drives 1 (left) to 4 (right) from ReadyNAS NV+

 

From root / sudo user;

# apt-get install fuseext2 lvm2 smart

 

**** reboot ****

 

**** check the smart status again, previously was via USB not direct attach *** 

 

# smartctl -H /dev/sdb
smartctl 6.6 2016-05-31 r4324 [x86_64-linux-4.8.0-22-generic] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

 

# smartctl -H /dev/sdc
smartctl 6.6 2016-05-31 r4324 [x86_64-linux-4.8.0-22-generic] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

 

# smartctl -H /dev/sdd
smartctl 6.6 2016-05-31 r4324 [x86_64-linux-4.8.0-22-generic] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

 

# smartctl -H /dev/sde
smartctl 6.6 2016-05-31 r4324 [x86_64-linux-4.8.0-22-generic] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

 

 

*** All looks ok there, lets see what the partitions look like ***

 

# fdisk -l

 

*** SDA (boot drive) and others removed ***


Disk /dev/sdb: 1.8 TiB, 2000398934016 bytes, 3907029168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0x00000000

Device Boot Start End Sectors Size Id Type
/dev/sdb1 32 4096031 4096000 2G 83 Linux
/dev/sdb2 4096032 4608031 512000 250M 82 Linux swap / Solaris
/dev/sdb3 4608032 3906992335 3902384304 1.8T 5 Extended
/dev/sdb5 4608040 3906992335 3902384296 1.8T 8e Linux LVM


Disk /dev/sdc: 1.8 TiB, 2000398934016 bytes, 3907029168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0x00000000

Device Boot Start End Sectors Size Id Type
/dev/sdc1 32 4096031 4096000 2G 83 Linux
/dev/sdc2 4096032 4608031 512000 250M 82 Linux swap / Solaris
/dev/sdc3 4608032 3906992335 3902384304 1.8T 5 Extended
/dev/sdc5 4608040 3906992335 3902384296 1.8T 8e Linux LVM


Disk /dev/sdd: 1.8 TiB, 2000398934016 bytes, 3907029168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0x00000000

Device Boot Start End Sectors Size Id Type
/dev/sdd1 32 4096031 4096000 2G 83 Linux
/dev/sdd2 4096032 4608031 512000 250M 82 Linux swap / Solaris
/dev/sdd3 4608032 3906992335 3902384304 1.8T 5 Extended
/dev/sdd5 4608040 3906992335 3902384296 1.8T 8e Linux LVM


Disk /dev/sde: 1.8 TiB, 2000398934016 bytes, 3907029168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/mapper/c-c: 5.5 TiB, 5988627251200 bytes, 11696537600 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes

 

# lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
c c -wi-a----- 5.45t

 

# lvdisplay /dev/c
--- Logical volume ---
LV Path /dev/c/c
LV Name c
VG Name c
LV UUID sIWnvX-Uphi-eWDp-6J0y-qyZ8-h8e2-Xqk0cR
LV Write Access read/write
LV Creation host, time ,
LV Status available
# open 0
LV Size 5.45 TiB
Current LE 178475
Segments 3
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 252:0

*** -o ro   to mount read only ***

# fuseext2 -o ro -o sync_read /dev/c/c /mnt

fuse-umfuse-ext2: version:'0.4', fuse_version:'29' [main (fuse-ext2.c:331)]
fuse-umfuse-ext2: enter [do_probe (do_probe.c:30)]
fuse-umfuse-ext2: leave [do_probe (do_probe.c:55)]
fuse-umfuse-ext2: opts.device: /dev/c/c [main (fuse-ext2.c:358)]
fuse-umfuse-ext2: opts.mnt_point: /mnt [main (fuse-ext2.c:359)]
fuse-umfuse-ext2: opts.volname: c [main (fuse-ext2.c:360)]
fuse-umfuse-ext2: opts.options: ro,sync_read [main (fuse-ext2.c:361)]
fuse-umfuse-ext2: parsed_options: sync_read,ro,fsname=/dev/c/c [main (fuse-ext2.c:362)]
fuse-umfuse-ext2: mounting read-only [main (fuse-ext2.c:378)]

 

# ls /mnt
aquota.group    aquota.user     backup    home    lost+found    media

 

 

So that worked 🙂

Message 9 of 13
Marty_M
NETGEAR Employee Retired

Re: ReadyNAS NV+ Stuck on Booting

Hello DOLMark,

 

It does appear the NAS chassis is ok and the hard drives did passed the test. Can you still set the NAS chassis to telnet/tech support mode? The concern you have described requires assistance from the engineers and/or higher tier who can remote in to the NAS and fix this from the backend. 

 

Unfortunately, It shows on your records that the support warranty of your product is already expired. You may try contacting support center and avail a support contract. That should allow you to extend the support for your product so they can escalate your issue to L3 or patiently wait for anyone in the community who has the knowledge in the SSH commands and fixing some issues from the backend.

Regards,
Marty_M 
NETGEAR Community Team

Message 10 of 13
DOLMark
Aspirant

Re: ReadyNAS NV+ Stuck on Booting

Hi Marty_M

 

Thanks for that.  Hadn't spotted the remote telnet/ssh diags mode hiding under the "Tech Support" boot option.

 

Is there any docs on how to manually diagnose the boot process from inside this environment?

 

It appears to just run start_raid.sh and mount up the drives but if there's any hints as to where to look for boot logs etc that'd be great 🙂

 

Thanks, Mark

 

Message 11 of 13
Marty_M
NETGEAR Employee Retired

Re: ReadyNAS NV+ Stuck on Booting

Hello DOLMark,

 

Being able to set the NAS to telnet/techsupport mode says that the chassis is working fine. I do not have other manual or diagnostic we can offer thru telnet, the concern you have described requires assistance from the engineers and/or higher tier who can remote in to the NAS and fix this from the backend. 

 

Unfortunately, It shows on your records that the support warranty of your product is already expired. You may try contacting support center and avail a support contract. That should allow you to extend the support for your product so they can escalate your issue to L3 or patiently wait for anyone in the community who has the knowledge in the SSH commands and fixing some issues from the backend.

Regards,
Marty_M 
NETGEAR Community Team

Message 12 of 13
StephenB
Guru

Re: ReadyNAS NV+ Stuck on Booting

You do have two other options.

 

Since you can mount the disks in another linux system, you can offload the data, do a factory reset, and then rebuild then NAS (restoring data from backup).

 

Another thing you can look at is the fullness of the OS partition from tech support mode.  For instance (after starting raid)

# mount /dev/hdc1 /sysroot

# /sysroot/bin/df /sysroot -h

# /sysroot/bin/df /sysroot -i

 

A full (or nearly full) OS partition is a fairly common issue, and it has a lot of possible symptoms (including connection drops, firmware update failures, and boot failures).

Message 13 of 13
Top Contributors
Discussion stats
  • 12 replies
  • 3441 views
  • 0 kudos
  • 3 in conversation
Announcements