× NETGEAR will be terminating ReadyCLOUD service by July 1st, 2023. For more details click here.
Orbi WiFi 7 RBE973
Reply

Re: Pro 6 on 4.2.30 suddenly reported Volume dead

refles
Aspirant

Pro 6 on 4.2.30 suddenly reported Volume dead

Hi All,

I'm at a complete loss as the current issue I'm experiencing is unlike anything I can lookup. Any help or ideas would greatly greatly be appreciated as from what I can tell the data is there but I can't access it via Windows. 

 

About 48 hours ago, my ReadyNAS Pro 6 set to X-RAID2 w/ dual redundancy middle of streaming a video and the video was suddenly not found, checking on what went wrong RAIDar suddenly showed that the volume was reported "DEAD" and then that Disk 6 failed, I immediately pulled the drive and performed Seatools for Windows on the drives with showed no issues on any test I can run on it (Seagate NAS HDD ST3000VN000 3tb). No other drives reported any issues and I believe that's to be accurate because I can access the data via another means

I insert the drive back and the Pro 6 immediately recognizes this and begins a resync'ing of the drive to the array which just finished this morning and it succeeded to resync the drive but it still shows Volume is still dead. 

 

one thing I noticed though.. 

-- To test if the data was there I was able to log in via Plex and stream all the movies, photos and everything else I have parked on the NAS.

-- Additionally Transmission is also running without any issues and I'm able add/remove torrents

 

I did notice that everything under "Standard File Protocols" was now unchecked and any attempt to toggle it ON just keeps it turned off. So that's probably why I cannot access/map to the NAS in Windows but I'm relieved that the data is there as accessing it via Plex to watch movies assures that its there.  The LED on the physical front door of the Pro6 still reports that the IP address and remaining storage left on the NAS.

 

1) Is there anything I can do to get this back online?

2) If this isn't salvagable, how would I move the data off if I know the data is accesable?

 

I'm not extremely proficient on command line, as I haven't used it in a long time but I was able to SSH via Putty into the box and can see the files and directories on the drive using the listing commands I remember from way back when.

 

 

Below is what I'm seeing in some of the screens, what's weird is that I don't remember seeing FAN CAS or Temp AUX before. 

 

thanks for everyone's time in looking and lending their thoughts!

Pro6 Status-Volume settings.jpg

Pro6 Status-Volume settings.jpg

Model: ReadyNASRNDP6000|ReadyNAS Pro 6 Chassis only,RNDP6000-200 |ReadyNAS® Pro 6 |EOL
Message 1 of 22

Accepted Solutions
StephenB
Guru

Re: Pro 6 on 4.2.30 suddenly reported Volume dead


@refles wrote:

 

EJRE Installer

Crashplan

Plex

Transmission

 

It is almost certainly Crashplan.  The Crashplan people pushed out 4.8.0 to my Pro last week, and that filled my OS partition too.

 

So look in /usr/local/crashplan/upgrade first.  You'll likely see a bunch of failed downloads that need to be cleared out. After that you need to resolve the upgrade problem.

 

To get the upgrade to work on my system I had to turn off certificate checks as described here: https://community.netgear.com/t5/Backing-up-to-your-ReadyNAS/Crashplan-4-6-0-upgrade-failing-and-sol...

 

You can do that with 

cd $home

echo "check_certificate = off" >.wgetrc

 

Also, if you haven't moved /usr/local/crashplan/cache you should.  That can take up a lot of space.

 

I created a cache folder in /c/ and used the procedure here; https://support.code42.com/CrashPlan/4/Troubleshooting/Reassigning_Cache_Folder_To_A_Different_Direc...

 

If you edit the file on a windows PC, be careful that you use an editor that can preserve linux txt file formating.  I use notepad++.

 

View solution in original post

Message 8 of 22

All Replies
mdgm-ntgr
NETGEAR Employee Retired

Re: Pro 6 on 4.2.30 suddenly reported Volume dead

Do you use disk spin-down?

Message 2 of 22
refles
Aspirant

Re: Pro 6 on 4.2.30 suddenly reported Volume dead

Thanks for the reply, No, I don't have any of the power settings enabled.. so no Power Save Spindown or Power Timer, WOL.

 

I've been going through some fo the threads and people are posting logs, is that one of the logs when I SSH into the box? Because on the web UI, my logs all disappeared like there's no record of the disk failure other than the email notice going out saying the array died and that I started a resyncing. Its almost like hardware (?) failure? I'm logging into the web management interface ok just no real details in there that I can tell.

Message 3 of 22
StephenB
Guru

Re: Pro 6 on 4.2.30 suddenly reported Volume dead

You could possibly migrate data to ext-formatted USB drives using the linux command line.

 

Did you check the fullness of the OS partition (df . -h, df . -i)?

Message 4 of 22
refles
Aspirant

Re: Pro 6 on 4.2.30 suddenly reported Volume dead

StephenB,

thanks for the commands... so this is what I show, but I'm at a loss what '/dev/md0' is but one's showing 100%, which is a bit odd b/c I'm not sure what it is or does. The one I recognize is that the primary partition is of /c .. could md0 be causing this weird volume suddenly reporting dead? 

 

df -h
Filesystem Size Used Avail Use% Mounted on
/dev/md0 4.0G 4.0G 0 100% /
tmpfs 16K 0 16K 0% /USB
/dev/c/c 11T 5.3T 5.6T 49% /c

 

df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/md0 65536 13091 52445 20% /
tmpfs 506380 1 506379 1% /USB
/dev/c/c 182845440 1090857 181754583 1% /c

 

Is there a way via the linux cmd line to move the files via network to a new storage location? 

Message 5 of 22
StephenB
Guru

Re: Pro 6 on 4.2.30 suddenly reported Volume dead

/dev/md0 is the virtual device where the operating system partitions live.  Yours is full, so it needs to be cleaned.  The UI (and network file access) can totally freeze if you don't take action.

 

What add-ons do you have installed?  Are you running ReadyDLNA?

 

One option is to contact support at my.netgear.com - they will charge (ask about per-incident support).

 

You can also try to find the problem area yourself.  For instance begin with 

cd /var/log

du . -h

 

If a folder looks large, descend into it with cd, and explore further.

 

If the total in /var/log doesn't seem very much, then try again in /var/cache

 

 

 

Message 6 of 22
refles
Aspirant

Re: Pro 6 on 4.2.30 suddenly reported Volume dead

Thank you for the info, In all the reading random stuff around the forum I guess I overlooked that bit and never knew about the /dev/md0 drive. And thank you for the quick guidance, I'll thumb around the directories and see what I uncover, that's a total surprise, I had no idea that's even possible. 

 

As for apps, I'm running...

EJRE Installer

Crashplan

Plex

Transmission

 

 

but the WebUI is still quite responsive and maybe the only time it freezes is when I'm trying to turn something on or off and its just refresh the page and turn it on again and it'll work. Even now, the webUI is working fine just that the logs and all settings seem to be turned off and I can't turn them back on. But Plex and Transmission are still running perfectly and I can access them with the browser and iphone/droid apps. So from that limited experience you would understand why I haven't had as much experience playing w/ the linux command line.. I'll report back if that's ok what I can find and will see what I can clear out. 

Message 7 of 22
StephenB
Guru

Re: Pro 6 on 4.2.30 suddenly reported Volume dead


@refles wrote:

 

EJRE Installer

Crashplan

Plex

Transmission

 

It is almost certainly Crashplan.  The Crashplan people pushed out 4.8.0 to my Pro last week, and that filled my OS partition too.

 

So look in /usr/local/crashplan/upgrade first.  You'll likely see a bunch of failed downloads that need to be cleared out. After that you need to resolve the upgrade problem.

 

To get the upgrade to work on my system I had to turn off certificate checks as described here: https://community.netgear.com/t5/Backing-up-to-your-ReadyNAS/Crashplan-4-6-0-upgrade-failing-and-sol...

 

You can do that with 

cd $home

echo "check_certificate = off" >.wgetrc

 

Also, if you haven't moved /usr/local/crashplan/cache you should.  That can take up a lot of space.

 

I created a cache folder in /c/ and used the procedure here; https://support.code42.com/CrashPlan/4/Troubleshooting/Reassigning_Cache_Folder_To_A_Different_Direc...

 

If you edit the file on a windows PC, be careful that you use an editor that can preserve linux txt file formating.  I use notepad++.

 

Message 8 of 22
refles
Aspirant

Re: Pro 6 on 4.2.30 suddenly reported Volume dead

Thank you StephenB will try that and update the thread. I remember when Crashplan suddenly stopped working last year because of an update to the way it handled security I wasn't aware about and was tons of digging around to figure out the fix. So I guess its the yearly Crashplan issue haha... 

 

I had to travel for work so when I'm back at end of the week I'll go and do some major spring cleaning... very much appreciate sharing your experience. I wouldn't have thought that its possible a filled boot partitiion could wreck so much havoc! 

Message 9 of 22
refles
Aspirant

Re: Pro 6 on 4.2.30 suddenly reported Volume dead

StephenB Thank you so much for all the guidance, turns out the /usr/local/crashplan/upgrade folder was 2.3gb the Crashplan/Cache folder was about 400mb and system logs was only 800mb.. but was completely surprised it brought the system down! 

 

Once I deleted the files in the upgrade folder and rebooted the system everything was functioning like normal again. 

 

One behavior changed though, wondering if you may know.. but its not a big deal, CrashPlan Transmission and Plex seems to start as disabled everytime I reboot the NAS, is that typical behavior because it didn't do that before.. is it something simple or overlooked? 

Message 10 of 22
StephenB
Guru

Re: Pro 6 on 4.2.30 suddenly reported Volume dead


@refles wrote:

CrashPlan Transmission and Plex seems to start as disabled everytime I reboot the NAS, is that typical behavior because it didn't do that before.. 


I don't use transmission or plex on my pro-6.  I think CrashPlan is starting up normally, but I'm not sure if I restarted the NAS after I upgraded to 4.8.

Message 11 of 22
refles
Aspirant

Re: Pro 6 on 4.2.30 suddenly reported Volume dead

The Pro 6 isn't responding this morning as I can't turn features on or off. More like I can turn it on but it will not stay on. I suspect that maybe I removed up too many items I shouldn't have touched on the log file and its not re-creating missing files (could tie to why things aren't staying on and the option "STATUS --> Logs" is completely clear and doesn't seem to update itself.

 

Any ideas as to what may have happened or how I can put some of these back in? I tried to reinstall the firmware 4.2.30 but it wouldn't allow me to as the error message is "Update File is not valid for this architecture".  And I can't SSH anymore, nor can I add the add-on because I get the same error

Message 12 of 22
StephenB
Guru

Re: Pro 6 on 4.2.30 suddenly reported Volume dead

Yikes.

 

You could try tech support mode I guess.

 

Or byte the bullet and do a fresh factory install/restore data from backup.

Message 13 of 22
refles
Aspirant

Re: Pro 6 on 4.2.30 suddenly reported Volume dead

Thanks for the assist.... I may need to bite the bullet and call for help. 

 

I think I may have screwed up when i deleted the files in the '/var/log' directory in my overzealous attempt to clean out all the stuff I didn't think I needed along side with the '.../crashplan/upgrade' directory. That's the only 2 areas I emptied out and it seems to be the '/var/log' contains the configuration files for the firmware (good to know lol).

 

I tried a OS reinstall located in on this page (http://kb.netgear.com/app/answers/detail/a_id/21107) and that only seemed to wipe out my established PW to the box to default and nothing else, but I don't really want to blow up the data unless I really really really have to as all the important stuff is backed up but the movies the kids watch aren't backed up.. so may try to save the OS if there's a way and nuke the box to factory default only as a last resort...

 

Any other ideas to restore the contents in /var/log or do you know if that's what happened? if you erase the files in that var/log folder your box becomes a brick where the firmware becomes unable to save anything? (because the crashplan/upgrade folder should have been safe to erase from your explanation). 

 

Also... this is the details for the per incident tech support right? http://gearhead.netgear.com/home.html

 

thanks StephenB and all who looked. If they're able to solve it I'll report back here what happened but hoping I can undo what I did to myself.

 

Message 14 of 22
StephenB
Guru

Re: Pro 6 on 4.2.30 suddenly reported Volume dead

I don't think you want gearhead support - note they don't mention ReadyNAS at all.  You should be able to get per-incident support from my.netgear.com though I think the price is a bit higher ($75 in US???).

 

Also, I don't believe any config files are in /var/logs.  Generally it's better to truncate the log files (shrink them to 0 bytes) instead of deleting them.  But I suspect the damage occurred when the OS partition was full (and that cleaning /var/logs didn't cause it).

Message 15 of 22
refles
Aspirant

Re: Pro 6 on 4.2.30 suddenly reported Volume dead

I'm digging through the support contact information and it looks like you can only buy yearly support contracts at the rate of  1y @ $380 // 3y @ $519.. etc. 

 

I thought I kept reading there's a support billed at the per incident level, that doesn't seem to exist. Does anyone know if this is true or is the cost at a mininum of 1 year support?

Message 16 of 22
refles
Aspirant

Re: Pro 6 on 4.2.30 suddenly reported Volume dead

Plus this is the support document they have, I can't seem to find my model or prefix of the pro6 model on here, so it may have come to end of life as well.... think I'm pretty much out of luck... 

 

http://prosupport.netgear.com/pdf/DS_ProSupport_US_Intl.pdf

Message 17 of 22
FramerV
NETGEAR Employee Retired

Re: Pro 6 on 4.2.30 suddenly reported Volume dead

Hi refles,

 

It would be best if you call the support group for the per-incident. I know that its not documented but you can mention it to the support tech.

 

Contact NETGEAR Support

 

Also, there is a chance a data recovery will be needed on your case. Please check our coverage for the date recovery.

 

ReadyNAS: Data Recovery Diagnostics - Scope of Service

 

 

Regards,

Message 18 of 22
FramerV
NETGEAR Employee Retired

Re: Pro 6 on 4.2.30 suddenly reported Volume dead

Hi refles,


We’d greatly appreciate hearing your feedback letting us know if the information we provided has helped resolve your issue or if you need further assistance. 

If your issue is now resolved, we encourage you to mark the appropriate reply as the “Accept as Solution” or post what resolved it and mark it as solution so others can be confident in benefiting from the solution. 
 
The Netgear community looks forward to hearing from you and being a helpful resource in the future!
 
Regards,

Message 19 of 22
refles
Aspirant

Re: Pro 6 on 4.2.30 suddenly reported Volume dead

Hi FarmerV,

Given that the box is out of date and I just want to access the data, I was thinking to upgrade the box and pull the drives and put the drives into a new 6 bay machine. Even possibly testing the data on a friend who has a Ultra 6/Pro 6 first before exploring maybe  RN316 or one of the newer models. I floated this thought in the "Before You Buy" section of this forum. Given the amount projected for just to talk to tech support and in the "My Netgear" section I can't just reach out via email without buying a support contract I thought just plugging the data into a new updated box under the new OS would gain access where I can pull it off and then just rebuild it all... but one comment was saying that if there's a problem with the firmware (which I obvious caused in this case if you see earlier posts) it goes along with the disks?

 

thanks for the contact detail but thought a new box modernized box would have been the solution and just abondon this one, thoughts?

Message 20 of 22
StephenB
Guru

Re: Pro 6 on 4.2.30 suddenly reported Volume dead


@refles wrote:

@I'm digging through the support contact information and it looks like you can only buy yearly support contracts at the rate of  1y @ $380 // 3y @ $519.. etc. 

 

I thought I kept reading there's a support billed at the per incident level, that doesn't seem to exist. Does anyone know if this is true or is the cost at a mininum of 1 year support?


Per-incident support does exist, but unfortunately isn't spelled out anywhere.  The cost does vary some by geography, the last US price I heard was $75.  But if you need data recovery, it would be a lot more ($200 to analyze, and it climbs from there).  

Message 21 of 22
refles
Aspirant

Re: Pro 6 on 4.2.30 suddenly reported Volume dead

Ok, that would explain it because I wasn't seeing it anywhere in the literature but its mentioned in many threads!!  Thanks for clearing that... luckily mdgm reach out and threw me a life line (thank you so much!) and was able to get the system up, it turns out I still had crashplan upgrades still on the /md0 drive as I thought I had removed them.  But non the less he's made the system operable again and I'm trying to get the web UI admin panel back as it won't show up right. I can browse the folders and all so I'm quite happy about that. I'll read through the forum for details as to how to get that working again but I can see it, ping it and browse the folder through the networking... I should be able to enable the webUI.  

 

But seems I didn't do a good job at removal of the failed crashplan upgrades as that reall renedered the NAS to near brick status. 

Message 22 of 22
Top Contributors
Discussion stats
  • 21 replies
  • 4285 views
  • 1 kudo
  • 4 in conversation
Announcements