Re: How to access disk information in Tech Support mode or Telnet, or get to a T3 support tech

mtmazzitello · ‎2018-11-10

I am having a disk problem with my ReadyNAS NV+ RND4000, I cannot get it to boot past "Kernel Panic" or "Bad Disk Found", and thus can't use the admin web interface to see what is going on. I am fairly sure that a disk is bad but I do not know which one. Apparently answering this simple question is not possible without tech support.
I purchased a support incident and have been playing email & phone tag with Ready NAS support for 2 days, most of which has been waiting for a reply after getting rudamentary questions asked and answered (everyone at Netgear must have my email and phone number by now....) So far all they have been able to tell me is to boot it into Tech Support mode and provide the Debug Code, which I have done. I got an email that said the disk in bay 2 "is having performance issues" (whatever that means) and that I should remove or replace it. I did what they said, and with that disk removed or a replacement for it inserted, I still see "Bad Dosk Found" when booting. I reported these findings several hours ago, and now I am again sitting here waiting again for a call from someone who will tell me what to do next.

I am a 20+ year IT professional and I can understand command line and diagnostic logs; is there somewhere I can find how to Telnet or SSH into my ReadyNAS and find information for myself? Or is there some way to just stop wasting everyone's time and get directly to a T3 support person while they are remotely connected to my NAS and do whatever needs to be done working together?

StephenB · ‎2018-11-11

The best way to check your disk health is to power down the NAS and test the drives in a Windows PC using vendor tools (WDC's Lifeguard or Seagate's Seatools).

Another option is to try the RAIDar utility - it does report some disk status. https://kb.netgear.com/20684/ReadyNAS-Downloads

However, to answer your question:

I'm assuming an NV+ v1 here, but if your NAS says "NV+ v2" on the front panel, check the hardware manual for the v2 for booting into tech support mode.

You can boot up the v1 NAS in tech support mode using the instructions on pages 23-24 here: http://www.downloads.netgear.com/files/GDC/RND2110/Duov1_NV+v1_HW_en_06Dec11.pdf

Then you can telnet into the NAS. The login is root, the password is infr8ntdebug

mtmazzitello · ‎2018-11-11

Thanks Stephen.
I have a ReadyNAS NV+ and I did boot into Tech Support Mode, purchased a support incident, and (after a long wait) had a L3 tech tell me to replace disk #2. That did not solve the problem, it still reports "Bad Disk Found".
I have the latest version of RAIDar software but the NAS will not boot to the point of connecting to the network, unless it is in Tech Support mode, and then the RAIDar does not show any disk info - all 4 disks snow "not available" and grayed out.
I have a suspicion that it is not in fact a disk problem but rather something like a SATA controller card, because I have tried removing and replacing all the disks one by one and booting each time (with one disk removed or with a new disk in its place), and it is always the same behavior when booting. I'd just like to be able to determine exactly what the problem is.

mtmazzitello · ‎2018-11-11

Also, even when booting into "Skip Volume Check" mode, it still displays "Bad Disk Found". This further makes me suspect that it is not a bad disk but some kind of controller or hardware problem.

I am able to use PuTTY to connect and login to the NAS when it is in Tech Support mode, but I don't know what to do once I am connected & logged in. Are there instructions published anywhere to find fault codes. logs, run tests, etc.?

I suppose my next step is to go get a USB enclosure or something and try to test each disk individually outside the NAS, so I can at least eliminate those as problems or find one if it is indeed bad. I am assuming this process will not destroy my RAID array - because if all the data is lost then this entire exercise is for nothing and I would just throw thew whole thing in the lake.

Thanks again for the help.

Sandshark · ‎2018-11-11

As long as you only perform a read test, not a write test (which is the default for the vendor tools), your array will be untouuched. It is recommended you put the drives back in in the original order. And, of course, perform all removals and insertions with power off.

Your PC won't recognize the format of the drives, but that doesn't stop the vendor tools from doing their job.

It is kinda sounding like a motherboard or SATA backplane issue, but testing the drives is a good next step to finding out for sure. Since you don't have a USB dock and it does sound like you purchased a new drive, you could insert just that drive (or any unused drive) into the NAS and let it initialize. If it initializes, power off, move the drive to the next slot, power back on, and see if it boots. Repeat for all slots. If it boots in all slots, the NAS hardware is OK, If it boots in some, the NAS hardware is bad. If it boots in none, it's likely the NAS is bad, but you could be unlucky enough to get a new drive that's bad.

mtmazzitello · ‎2018-11-11

That's actually a great idea. I have 2 brand new 1TB drives (the same size as what is in my 4x 1TB RAID 5 array) so I can try as you suggested and use one or both of them individually in the different slots in the NAS and see what happens.
I suppose I could be unlucky enough to have the one new drive I have tried be bad out of the box, or have more than one of my existing drives fail simultaneously. But I do suspect something in the NAS itself, as the initial problem was it 'froze' - although it was running with no fault indications on the display it would not allow access to the volumes, web admin UI, or even ping, and holding the power button would not shut it down. Tech support instructed me to pull the plug to restart it, and it has not been able to boot since then (other than into Tech Support mode), nor has the power button worked to shut it down.

I don't suppose replacement hardware is available for these, should it be a motherboard or SATA controller issue. I had a hell of a time about 5 years ago finding a replacement power supply, and ended up having to modify a non-OEM one to get it to work.

I am currently waiting for them once again to remote access the unit and hopefully call me while they are connected & watching, so I can try some of these drive swaps and have them tell me what is going on. I'll update with anything else I try.

Sandshark · ‎2018-11-11

Parts are not available. The only one ever available was the power supply, and that was because of the recall. I sometimes see used ones for sale that have a bad power supply, but that's a bit of a crap shoot since you have to rely on their proper diagnosis and a power supply going bad can take other components with it. The NV+ SATA backplane is identical in all legacy 4-drive desktop systems except the NV+v2, BTW; but I would only suspect the backplane if some of the slots are bad.

mtmazzitello · ‎2018-11-11

This is very strange. I tried removing all the disks, and using a brand new disk started up the system with that one new disk in each of the four slots. It did boot just fine in all four cases, but when the disk is in slot 4 the LED on the front is illuminated for slot 3, and when logged into the admin web page and doing "locate" for the volume it flashes LED #3 and says it is blinking #3. The Volume being displayed in the UI says channel is "UNKNOWN" where it displays the correct number 1 or 2 or 3 for the others. RAIDar displays disk 4 with a green checkmark, but the rest of the system apparently thinks the disk in slot 4 is actually in slot 3.

When the lone disk is in slot 3 then all the displays and LEDs etc. correctly indicate one disk in slot #3. Same with #1 or #2.

I updated the firmware and it is still behaving the same. I am now suspecting that channel 4 on the RAID controller is bad.

mtmazzitello · ‎2018-11-11

If I start up with only one or two of my RAID 5 array disks, will the system automatically overwrite them or otherwise destroy the RAID? I'm trying to test various scenarios of one, two, or three disks in various slots (see the goofiness from the previous message), but I only have 2 brand new disks to experiment with and do not want to risk losing my RAID volume.

StephenB · ‎2018-11-11

@mtmazzitello wrote:

If I start up with only one or two of my RAID 5 array disks, will the system automatically overwrite them or otherwise destroy the RAID?

It shouldn't, but you do need to be careful not to remove or insert the disks with the NAS running.

I'd try testing with the two new disks first, perhaps you won't need to try three disk scenarios. I'm thinking that you might want to a two disk XRAID volume in slots 1 and 2 and then try moving them to slots 1, 3 followed by 1,4 and finally slots 3,4. You might also want to install the Enable Root SSH Access add on from here: https://kb.netgear.com/24546/Add-ons-for-RAIDiator-4-1-3-Sparc

mtmazzitello · ‎2018-11-12

Thanks again for the tips and advice. I am purchasing two additional 1TB disks today, so I will have four brand new ones that I can try to configure in this NAS and see what happens. That should tell definitively whether the NAS chassis & hardware is functioning or not.

Should it durn out that more than one of my original RAID5 disks are bad - which would be a remarkable coincidence to have more than one fail simultaneously with no triggering event like a power surge or similar event, but I suppose it is possible that my luck could be just that bad - is there any hope to recover my data? I know the "on paper" anmswer is no, but I am wondering whether there is some way to force the ReadyNAS to get bast the SATA controller disk check and boot anyway, or command line procedures to force the array online or other last ditch attempts to get it accessible long enough to copy data off of it.

Thanks again.

StephenB · ‎2018-11-12

@mtmazzitello wrote:

would be a remarkable coincidence to have more than one fail simultaneously with no triggering event like a power surge or similar event,

It's not as unsual as you might think (based on posts here). One aspect is that we tend to buy all the disks at once - same model, same supplier,etc, and put them into the same chassis and subject them to the same loads. Another aspect is that the disks failures won't be detected until you attempt to access the data - and in many cases that only happens until you need to resync.

Newer ReadyNAS have some better options for maintenance and testing (RAID scrubs for instance), but they aren't available on the NV+ v1.

@mtmazzitello wrote:

but I am wondering whether there is some way to force the ReadyNAS to get bast the SATA controller disk check and boot anyway, or command line procedures to force the array online or other last ditch attempts to get it accessible long enough to copy data off of it.

You can force the array to be mounted in an x86 linux system, and there is RAID recovery software that would work in a Windows PC (particularly R-Studio). You'd likely need to get USB adapter/dock(s) though.

If the only problem is bay 4, then you could try booting with the original disks in bays 1-3 (skipping the file system checks), and perhaps that will work. You could I guess try putting the original disk 4 into bay 2 - the problem there is that we don't know how long bay 4 has been misbehaving, and it might be corrupted. So I think the RAID recovery software might be a better option.

Seagate also offers a data recovery service (where you'd mail them your drives). That's available even if you don't have Seagate drives. I haven't seen any posts here from anyone who's used it, but it might be worth a try. They only charge if they can recover at least some of your data. https://www.seagate.com/services-software/recover/in-lab-recovery/

mtmazzitello · ‎2018-11-12

Here is what I discovered. The drives originally in slot 2 and slot 3 always produce the "Bad Disk" error, no matter which slot they are inserted into. The system will boot with only drive #1 inserted, but of course the RAID volume is not accessible with only 1 drive.
As far as I can tell, I have a RAID 5 volume consisting of drives 1, 2, & 3 with the drive in #4 being un-formatted as a spare. I put each of the 4 drives in a SATA USB dock on my PC and in windows disk manager I can see partitions on all of them except 4, which is un-initialized. I used Seagate "SeaTools" to test them, and all 4 drives pass the "short self test" and "short generic" tests. It seems the drives are OK for my USB SATA dock and SeaTools, but the NAS doesn't think so.

I am wondering if there is some way in the SSH session to the NAS in Tech Support mode at the command line to bypass the disk check (not the volume check but the physical disk check on booting) to try to force it to start those disks?

StephenB · ‎2018-11-13

@mtmazzitello wrote:

As far as I can tell, I have a RAID 5 volume consisting of drives 1, 2, & 3 with the drive in #4 being un-formatted as a spare. I put each of the 4 drives in a SATA USB dock on my PC and in windows disk manager I can see partitions on all of them except 4, which is un-initialized.

The NV+ actually uses RAID-4 (a dedicated parity disk, instead of distributing the parity accross all the disks). There is some hardware accelaration, and the data area of the parity disk doesn't have a partition. It still does contain the parity data. Normally the parity disk would be disk 4.

However, I think you should be seeing the OS partition (2 GB) on disk 4 - so maybe check again for that.

@mtmazzitello wrote:

I am wondering if there is some way in the SSH session to the NAS in Tech Support mode at the command line to bypass the disk check (not the volume check but the physical disk check on booting) to try to force it to start those disks?

In tech support mode the system makes no attempt to mount the data volume at all.

You could try the following:

# start_raid.sh

mount the OS partition (this assumes the default XRAID).
# mount /dev/hdc1 /sysroot

chroot
# mount --bind /proc /sysroot/proc
# mount --bind /dev /sysroot/dev
# mount --bind /dev/pts /sysroot/dev/pts
# chroot /sysroot /bin/bash

Then the data volume can be mounted read-only using

# vgscan
# vgchange -ay
# mount -o ro /dev/c/c /c

I expect something will fail on the last three commands. I haven't had to troubleshoot RAID on my own NV+, so you'll need to research debugging options for lvm (vgscan, vgchange, etc) on your own. If it does work, you can attempt to start Samba manually, or connect/mount a USB disk to offload data.

mtmazzitello · ‎2018-11-13

I did find start_raid.sh over the weekend, when I type that in all it does is delays a second or so and then comes back with a # prompt and does nothing; no errors or returned messages or anything. So I am guessing of that doesn't work then the other subsequent commands won't do anything or be of any use?

I got a suggestion from someone else to try these commands to see what errors the system thinks are occurring, but I am also wondering whether these will be useless if the RAID cannot be started. I am going to try this tonight, along with your suggestions:

# get_ata_errors /dev/hdc
# get_ata_errors /dev/hde
# get_ata_errors /dev/hdg
# get_ata_errors /dev/hdi

mtmazzitello · ‎2018-11-13

Oh, and the other scheme I am thinking of trying since the USB SATA dock can apparently read the drives just fine is to find a "cloner" and clone the 2 supposedly-bad drives to new ones.

StephenB · ‎2018-11-13

@mtmazzitello wrote:

I did find start_raid.sh over the weekend, when I type that in all it does is delays a second or so and then comes back with a # prompt and does nothing; no errors or returned messages or anything. So I am guessing of that doesn't work then the other subsequent commands won't do anything or be of any use?

I haven't needed to do this on my own NV+, but I wouldn't assume it is failing just because it is silent. I'd expect that it would generate an error if it fails.

@mtmazzitello wrote:

I got a suggestion from someone else to try these commands to see what errors the system thinks are occurring, but I am also wondering whether these will be useless if the RAID cannot be started. I am going to try this tonight, along with your suggestions:
# get_ata_errors /dev/hdc
# get_ata_errors /dev/hde
# get_ata_errors /dev/hdg
# get_ata_errors /dev/hdi

get_ata_errors should still work. Whether that will give you useful information to troubleshoot is another question - not sure about that.

mtmazzitello · ‎2018-11-13

Here's what I got:

# start_raid.sh
Killed

# get_ata_errors /dev/hdc
0
# get_ata_errors /dev/hde
0
# get_ata_errors /dev/hdg
246
# get_ata_errors /dev/hdi
0

mtmazzitello · ‎2018-11-13

I think I might be getting somewhere! With the last set of commands, I made it to the last one, and I suspect it is maybe just a typo?

root@000da2019d08:/# vgscan
Reading all physical volumes. This may take a while...
Found volume group "c" using metadata type lvm2
root@000da2019d08:/# vgchange -ay
1 logical volume(s) in volume group "c" now active
root@000da2019d08:/# mount -o ro /dev/c/c /c
mount: mount point /c does not exist
root@000da2019d08:/#

StephenB · ‎2018-11-13

@mtmazzitello wrote:

Oh, and the other scheme I am thinking of trying since the USB SATA dock can apparently read the drives just fine is to find a "cloner" and clone the 2 supposedly-bad drives to new ones.

You can clone in windows, as long as you use a sector by sector copy mode. Clonezilla can do it.

mtmazzitello · ‎2018-11-14

We have one of these in my company's IT department:
https://www.startech.com/HDD/Duplicators/usb-3-esata-6-bay-hard-drive-duplicator-dock~SATDOCK5U3ER

Unfortunately it does not seem to be able to do the clone, it gets stuck somewhere along the way and just stops processing. No errors or indications of failure, just stops copying. 😐

I think I got lucky once with the NAS using PuTTY to get the raid started and mount the C volume, although I had some syntax error or something on that very last command, and not knowing what I am doing didn't help to troubleshoot it. I have not been able to get the start_raid.sh command to succeed since then.

Since my USB SATA dock on my Windows PC seems to be able to read at least the partitions and pass trhe SeaTools tests, I guess I'll try using that as the source location and doing a clone using my Windows PC.
Boy this is getting tedious, but might be my last hope.

mtmazzitello · ‎2018-11-15

No luck with cloning 😞

Looking for any more help/guidance I can get for using the PuTTY connection & console to try to mount & access whatever volumes or parts of volumes I might be able to get at. I think that's my last chance. I am not well-versed in unix commands for volumes & file copies etc., I would need an interactive conversation/chat with someone who does. I have been waiting over 4 days for a reply from my paid support incident with Netgear.