Reply

ReadyNAS Pro 6 Inactive Volumes - Main Volume Gone

dhl
Luminary
Luminary

ReadyNAS Pro 6 Inactive Volumes - Main Volume Gone

Yesterday our ReadyNAS Pro 6 running firmware version 6.8.1 became inaccessable and upon reboot, shows the main voulume missing:

 

Screen-Shot-2017-10-14-at-7.24.33-PM.jpg

 

 

This problem occured without any error warnings beforehand. As you can imagine this is extremely scary, as we run our business off this server.

 

The good news is we have a full and current backup which ran the night before this happened. We would prefer to recover the existing volume if possible since resync and restore takes about a week.

 

Also, we're very concerned about this happening again and want to diagnose the cause so it can be prevented in the future.

 

Any help appreciated. I've downloaded a full log set and will turn on Secure Diagnostic Mode if needed. We've shut the NAS down for safety until we get further instructions.

 

Thanks for your help.

--dhl

 

 

Model: ReadyNASRNDP6000|ReadyNAS Pro 6 Chassis only
Message 1 of 16
dhl
Luminary
Luminary

Re: ReadyNAS Pro 6 Inactive Volumes - Main Volume Gone

Bumping this topic - we're working with local files pulled from the backup but really need help.

 

@mdgm-ntgr@StephenB - any advice?

 

Thanks!

--dhl

Model: ReadyNAS RNDP6000|ReadyNAS Pro 6 Chassis only
Message 2 of 16
StephenB
Guru

Re: ReadyNAS Pro 6 Inactive Volumes - Main Volume Gone

Unfortunately it's not something I can help with.  Hopefully someone from Netgear will reach out to you and at least check your logs.

Message 3 of 16
dhl
Luminary
Luminary

Re: ReadyNAS Pro 6 Inactive Volumes - Main Volume Gone

Thanks @StephenB. I've tagged @mdgm-ntgr so hopefully, he or another Netgear guru will check in. I've got logs ready to send.

Model: ReadyNAS RNDP6000|ReadyNAS Pro 6 Chassis only
Message 4 of 16
Sandshark
Sensei

Re: ReadyNAS Pro 6 Inactive Volumes - Main Volume Gone

If you click in the box for the upper volume, does it then highlight just the drive in bay 5?

 

I agree that it is best to wait for a Netgear rep to look at your logs.  But what it looks like to me is that your NAS somehow thinks that the drive in bay 5 is a separate volume, the main one at that, and is complaing that the remaining drives are "dirty" with data from some other volume, also named data.  But two volumes with the same name are not allowed.  In that case, booting with just drives 1, 2, 3, 4, & 6 would likely work.

 

While you wait for somebody to look at the logs, it may be a good time to test the drives and see if one has a problem that is creating this situation.  Using a PC and manufacturer's tools is much better, and faster, than using the NAS's disk test.

 

 

Message 5 of 16
dhl
Luminary
Luminary

Re: ReadyNAS Pro 6 Inactive Volumes - Main Volume Gone

@Sandshark - Bay 5 is empty, we're only using bays 1, 2, 3, 4, and 6 in RAID 6 mode. This gives us equal capacity to our 4-bay ReadyNAS 104 backup server which we use in RAID 5.

 

I see what you mean about the OS seeing two volumes named "data". It's a mystery why this is happening and I don't want to change anything until someone from Netgear can look at the logs.

 

I'm currently in SF and the NAS is at my partners' studio in LA. I'll be there later in the week and can pull drives and do other hardware checks then.

Model: ReadyNAS RNDP6000|ReadyNAS Pro 6 Chassis only
Message 6 of 16
Sandshark
Sensei

Re: ReadyNAS Pro 6 Inactive Volumes - Main Volume Gone

It may still help if you tell us what happens when the upper data volume is selected.  What drives are what colors?

Message 7 of 16
dhl
Luminary
Luminary

Re: ReadyNAS Pro 6 Inactive Volumes - Main Volume Gone

The NAS is off right now and my partners are out. I'll have them turn it on and check. IIRC, the other volume showed all disks as red too.

Model: ReadyNAS RNDP6000|ReadyNAS Pro 6 Chassis only
Message 8 of 16
Sandshark
Sensei

Re: ReadyNAS Pro 6 Inactive Volumes - Main Volume Gone

I have seen these symptoms reported here far too often.  While it is true that we don't hear from anyone who is not having issues, I believe that the root cause of this type of issue needs to be investigated by Netgear.  Why does the unit suddenly believe that there are two data volumes?  Is it reading two layers as two volumes?  Reading the same one twice (one from fstab and one in a scan)?

 

This issue started being reported at a specific update, but I'd have to look back and see which one (which is difficult with this forum software's terrible search implementation).

 

While it may not be related, I also noted after an OS update that a directory from one NAS that I have mounted on another NAS via an fstab entry started being reported as another "unknown" volume on the system where the mount occurs.  Perhaps the NAS is being too aggressive in looking for volumes?  It could be in BTRFS itself, not the Netgear portion of the OS.

Message 9 of 16
StephenB
Guru

Re: ReadyNAS Pro 6 Inactive Volumes - Main Volume Gone


@Sandshark wrote:

I have seen these symptoms reported here far too often. 


I agree. and unfortunately I haven't see much useful advice on what to do about it (other than get help from Netgear).

 

It'd be great if they provided some more details on troubleshooting steps, especially for folks that can't use Netgear support.

Message 10 of 16
dhl
Luminary
Luminary

Re: ReadyNAS Pro 6 Inactive Volumes - Main Volume Gone


@StephenB wrote:

@Sandshark wrote:

I have seen these symptoms reported here far too often. 


I agree. and unfortunately I haven't see much useful advice on what to do about it (other than get help from Netgear).

 

It'd be great if they provided some more details on troubleshooting steps, especially for folks that can't use Netgear support.


Yikes. This is very disturbing as our business depends on this server. Our Pro 6 has been rock solid since we first put it into service years ago. We recently upgraded to OS 6 as part of a larger upgrade to replace the previous aged out drive set and increase capacity for the next 5-years.

 

I don't know if these symptoms are hardware related (drives or power supply?) but when I'm in LA at the end of the week I can do any tests that might be helpful. Hopfully the logs would have indications of what to look for if hardware is failing.

 

I've searched the forums here and seen a couple other posts mentioning dissappearing volumes. For us, this happened a couple weeks after upgrading to OS 6.8.1.

 

If it is the firmware, I really hope Netgear takes a thorough look for everyone's sake. Even with backup, no one can afford to lose their NAS volume without warning.

 

 

Model: ReadyNAS RNDP6000|ReadyNAS Pro 6 Chassis only
Message 11 of 16
mdgm-ntgr
NETGEAR Employee Retired

Re: ReadyNAS Pro 6 Inactive Volumes - Main Volume Gone

Have you checked systemd-journal.log for what it has to say about md127 and if it's a multi-layer volume additional layers such as md126?

 

Does smart_history.log show recent disk error count increases?

Message 12 of 16
dhl
Luminary
Luminary

Re: ReadyNAS Pro 6 Inactive Volumes - Main Volume Gone


@mdgm wrote:

Have you checked systemd-journal.log for what it has to say about md127 and if it's a multi-layer volume additional layers such as md126?

 

Does smart_history.log show recent disk error count increases?


smart_history.log shows no errors:

 

time model serial realloc_sect realloc_evnt spin_retry_cnt ioedc cmd_timeouts pending_sect uncorrectable_err ata_errors
------------------- -------------------- -------------------- ------------ ------------ -------------- ---------- ------------ ------------ ----------------- ----------
2017-09-11 19:37:08 ST8000NM0055-1RM112 ZA15SFJ3 0 0 0 0 0 0 0 0
2017-09-11 19:37:08 ST8000NM0055-1RM112 ZA15NSWT 0 0 0 0 0 0 0 0
2017-09-11 19:37:08 ST8000NM0055-1RM112 ZA15L8SG 0 0 0 0 0 0 0 0
2017-09-11 19:37:09 ST8000NM0055-1RM112 ZA15SFXK 0 0 0 0 0 0 0 0
2017-09-11 19:37:09 ST8000NM0055-1RM112 ZA15SFD3 0 0 0 0 0 0 0 0

 

systemd-journal.log appears to be showing BTRFS errors (here are all found references to md127 in systemd-journal.log, no references to md126 or other volumes):

 

Oct 14 19:05:58 PM-ReadyNAS kernel: md: md127 stopped.
Oct 14 19:05:58 PM-ReadyNAS kernel: md/raid:md127: device sda3 operational as raid disk 0
Oct 14 19:05:58 PM-ReadyNAS kernel: md/raid:md127: device sde3 operational as raid disk 4
Oct 14 19:05:58 PM-ReadyNAS kernel: md/raid:md127: device sdd3 operational as raid disk 3
Oct 14 19:05:58 PM-ReadyNAS kernel: md/raid:md127: device sdc3 operational as raid disk 2
Oct 14 19:05:58 PM-ReadyNAS kernel: md/raid:md127: device sdb3 operational as raid disk 1
Oct 14 19:05:58 PM-ReadyNAS kernel: md/raid:md127: allocated 5418kB
Oct 14 19:05:58 PM-ReadyNAS kernel: md/raid:md127: raid level 6 active with 5 out of 5 devices, algorithm 2
Oct 14 19:05:58 PM-ReadyNAS kernel: md127: detected capacity change from 0 to 23989791227904
Oct 14 19:05:58 PM-ReadyNAS kernel: BTRFS: device label 33ea2c83:data devid 1 transid 36327 /dev/md127
Oct 14 19:05:59 PM-ReadyNAS kernel: BTRFS info (device md127): has skinny extents
Oct 14 19:05:59 PM-ReadyNAS kernel: BTRFS critical (device md127): corrupt leaf, slot offset bad: block=3842431975424, root=1, slot=77
Oct 14 19:05:59 PM-ReadyNAS kernel: BTRFS critical (device md127): corrupt leaf, slot offset bad: block=3842431975424, root=1, slot=77
Oct 14 19:05:59 PM-ReadyNAS kernel: BTRFS critical (device md127): corrupt leaf, slot offset bad: block=3842428960768, root=1, slot=77
Oct 14 19:05:59 PM-ReadyNAS kernel: BTRFS critical (device md127): corrupt leaf, slot offset bad: block=3842428239872, root=1, slot=77
Oct 14 19:05:59 PM-ReadyNAS kernel: BTRFS critical (device md127): corrupt leaf, slot offset bad: block=3842427453440, root=1, slot=77
Oct 14 19:05:59 PM-ReadyNAS kernel: BTRFS error (device md127): open_ctree failed
Oct 14 19:05:59 PM-ReadyNAS mount[1460]: mount: wrong fs type, bad option, bad superblock on /dev/md127,
Oct 14 19:06:00 PM-ReadyNAS mdadm[2818]: NewArray event detected on md device /dev/md127

 

I can zip up and send all logs if needed, just let me know. Any clue as to what's going on? Do we have any hope of recoverying our volume?

Thanks for your help!

 

Model: ReadyNAS RNDP6000|ReadyNAS Pro 6 Chassis only
Message 13 of 16
a_carneiro
Tutor

Re: ReadyNAS Pro 6 Inactive Volumes - Main Volume Gone

Exactly the same problem here.

Just upgraded to 6.8.1 and now my main volume is gone!

 

 

Model: ReadyNAS-4200|ReadyNAS 4200
Message 14 of 16
dhl
Luminary
Luminary

Re: ReadyNAS Pro 6 Inactive Volumes - Main Volume Gone

I'm in LA and working with the NAS. The upper volume shows all disks inactive as well:

 

Screen Shot 2017-10-22 at 7.05.38 PM.png

We really need to know if this is a hardware problem or a firmware bug.

 

 

Model: ReadyNAS RNDP6000|ReadyNAS Pro 6 Chassis only
Message 15 of 16
dhl
Luminary
Luminary

Re: ReadyNAS Pro 6 Inactive Volumes - Main Volume Gone

Just got off the phone with Netgear Tier 2 support. After looking around the admin panel with TeamViewer, the agent said that because we're running OS6 he was unable to provide any further support. His advice was to revert to OS4, but we can't do that because we need remote access features that OS4 no longer supports. To his credit, he read this thread and refunded the support fee.

 

So now we're stuck. 

 

Another issue I noticed with the device - it takes 5-10 minutes for it to boot once it's powered on. This is strange as my Pro 6 in San Francisco always starts right up as soon as I press the front button. I'm wondering if we might have a faulty PSU and if that may have been a cause for the inactive volume?

 

@mdgm-ntgr - I've sent you an email with logs. I'm hoping you might have time to take a look. I'm in LA with the device today and tomorrow.

 

Thanks!

--dhl

Model: ReadyNAS RNDP6000|ReadyNAS Pro 6 Chassis only
Message 16 of 16
Discussion stats
  • 15 replies
  • 2345 views
  • 1 kudo
  • 5 in conversation
Announcements