- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page
RN316 Upgraded Firmware, EDA500 Volumes are messed up
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
RN316 Upgraded Firmware, EDA500 Volumes are messed up
I just upgraded my firmware. machine booted and then one of my EDA500s doesn't show up. The other is also messed up.
Doesn't seem that netgear provides support anymore... I used to be able to buy a support contract but have no idea how to do that anymore. Can anyone help me figure this out?
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: RN316 Upgraded Firmware, EDA500 Volumes are messed up
So if I disconnect the two EDA500s, my NAS boots up and all is well.
If I turn one of the EDA500s on, it shows up as a mix of unusableness.
If I boot up the NAS while the two EDA500s are powered on and connected, the NAS shows every drive as red.
I am assuming there is something I can do to fix this. Support tells me to post to the forums and won't tell me if you can purchase a support ticket like back in 2018.
Anyone have any ideas?
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: RN316 Upgraded Firmware, EDA500 Volumes are messed up
more info...
Without the EDA500s on, I see my NAS showing up in the Volumes part with
EDA1
EDA2
EDA3
All with data/free at 0.
I only ever had 2 EDA.
So wondering if anyone can help tell me what to do to get these back.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: RN316 Upgraded Firmware, EDA500 Volumes are messed up
@joe_schmo wrote:
If I turn one of the EDA500s on, it shows up as a mix of unusableness.
Does it matter which one? Or does turning either one on cause this problem?
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: RN316 Upgraded Firmware, EDA500 Volumes are messed up
It's both. Here's a screenshot with what happens when I boot the NAS by itself and then turn on one EDA.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: RN316 Upgraded Firmware, EDA500 Volumes are messed up
@joe_schmo wrote:
It's both. Here's a screenshot with what happens when I boot the NAS by itself and then turn on one EDA.
Have you captured a full log zip file? I am suspecting that the two EDA500 volumes are out of sync. @Sandshark might have some suggestions (he has had EDA500s in the past, I have not).
FWIW, I think you must have vertically expanded your EDA500 volumes (that would explain the three eda1 RAID groups).
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: RN316 Upgraded Firmware, EDA500 Volumes are messed up
I have long ago vertically expanded. Probably 6 or 7 years maybe?
I do have the logs downloaded.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: RN316 Upgraded Firmware, EDA500 Volumes are messed up
I suggest looking in dmesg.log, kernel.log, system.log, and systemd-journal.log. Look at the most recent boot sequence for errors related to disk, mdadm, and btrfs.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: RN316 Upgraded Firmware, EDA500 Volumes are messed up
Not sure exactly what to look for.
Kernel.log shows:
Oct 08 14:09:05 NAS kernel: md/raid:md124: device sda6 operational as raid disk 0
Oct 08 14:09:05 NAS kernel: md/raid:md124: device sdd6 operational as raid disk 5
Oct 08 14:09:05 NAS kernel: md/raid:md124: device sdc6 operational as raid disk 4
Oct 08 14:09:05 NAS kernel: md/raid:md124: device sde6 operational as raid disk 3
Oct 08 14:09:05 NAS kernel: md/raid:md124: device sdb6 operational as raid disk 2
Oct 08 14:09:05 NAS kernel: md/raid:md124: device sdf6 operational as raid disk 1
Oct 08 14:09:05 NAS kernel: md/raid:md124: allocated 6474kB
Oct 08 14:09:05 NAS kernel: md/raid:md124: raid level 5 active with 6 out of 6 devices, algorithm 2
Oct 08 14:09:05 NAS kernel: RAID conf printout:
Oct 08 14:09:05 NAS kernel: --- level:5 rd:6 wd:6
Oct 08 14:09:05 NAS kernel: disk 0, o:1, dev:sda6
Oct 08 14:09:05 NAS kernel: disk 1, o:1, dev:sdf6
Oct 08 14:09:05 NAS kernel: disk 2, o:1, dev:sdb6
Oct 08 14:09:05 NAS kernel: disk 3, o:1, dev:sde6
Oct 08 14:09:05 NAS kernel: disk 4, o:1, dev:sdc6
Oct 08 14:09:05 NAS kernel: disk 5, o:1, dev:sdd6
Oct 08 14:09:05 NAS kernel: md124: detected capacity change from 0 to 10001269391360
Oct 08 14:09:05 NAS kernel: BTRFS: device label 7c6e0558:data devid 4 transid 7395299 /dev/md124
Oct 08 14:09:05 NAS kernel: md: md123 stopped.
Oct 08 14:09:05 NAS kernel: md: bind<sdm3>
Oct 08 14:09:05 NAS kernel: md/raid:md123: device sdm3 operational as raid disk 0
Oct 08 14:09:05 NAS kernel: md/raid:md123: allocated 5418kB
Oct 08 14:09:05 NAS kernel: md/raid:md123: not enough operational devices (4/5 failed)
Oct 08 14:09:05 NAS kernel: RAID conf printout:
Oct 08 14:09:05 NAS kernel: --- level:5 rd:5 wd:1
Oct 08 14:09:05 NAS kernel: disk 0, o:1, dev:sdm3
Oct 08 14:09:05 NAS kernel: md/raid:md123: failed to run raid set.
Nothing in dmesg.log or system.log looks odd.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: RN316 Upgraded Firmware, EDA500 Volumes are messed up
In 2018 this is what they did to fix a similar issue:
This is Brian from NETGEAR Level 3 Support.
I have fixed your issue with one of your EDAs and what caused the problem was two of it's drives have lost its partitions.
I have recreated them and this shouldn't occur again.
If I knew what commands they ran, I'd just do it myself.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: RN316 Upgraded Firmware, EDA500 Volumes are messed up
This is consistent with what you are seeing in the web ui:
Oct 08 14:09:05 NAS kernel: md/raid:md123: device sdm3 operational as raid disk 0
Oct 08 14:09:05 NAS kernel: md/raid:md123: allocated 5418kB
Oct 08 14:09:05 NAS kernel: md/raid:md123: not enough operational devices (4/5 failed)
Oct 08 14:09:05 NAS kernel: RAID conf printout:
Oct 08 14:09:05 NAS kernel: --- level:5 rd:5 wd:1
Oct 08 14:09:05 NAS kernel: disk 0, o:1, dev:sdm3
Oct 08 14:09:05 NAS kernel: md/raid:md123: failed to run raid set.
However, it seems unlikely that four out of five disks actually failed.
Do you have ssh enabled?
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: RN316 Upgraded Firmware, EDA500 Volumes are messed up
I do (once upon a time with another NAS I even logged in tech support mode to fix something). And I have the log files too. The drives are healthy.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: RN316 Upgraded Firmware, EDA500 Volumes are messed up
@joe_schmo wrote:
I do
I suggest running smartctl -x on all five of the EDA500 drives, just to make sure that the RN316 is still "seeing" them.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: RN316 Upgraded Firmware, EDA500 Volumes are messed up
It still sees them.
I booted up with the EDAs unplugged, got this screenshot. Ran smartctl on each of those. Rebooted, the NAS came up fine, plugged in the EDA and smartctl ran fine on those drives as well.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: RN316 Upgraded Firmware, EDA500 Volumes are messed up
I'd get back to the state where the internal array is ok.
Then perhaps try
mdadm --stop /dev/md123
mdadm --assemble /dev/md123
If you get an error on the second command, then try again with --force. This isn't completely safe, but I think you will probably need it.
mdadm --stop /dev/md123
mdadm --force --assemble /dev/md123
After that, try
btrfs device scan
mount /dev/md123 /eda1
If the mount is successful, then try rebooting and make sure both volumes are shown on the volume screen and accessible.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: RN316 Upgraded Firmware, EDA500 Volumes are messed up
Sounds good. Thank you. I'll keep you posted on how this goes.
So I just need to make sure I know which EDA is eda1 and which one is eda2, and then grab the corresponding /dev/X number.
Do I need to be in tech support mode for this or just a normal bash prompt?
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: RN316 Upgraded Firmware, EDA500 Volumes are messed up
@joe_schmo wrote:Do I need to be in tech support mode for this or just a normal bash prompt?
I think you will be able to do it with the normal bash prompt (logging in as root).
@joe_schmo wrote:
So I just need to make sure I know which EDA is eda1 and which one is eda2, and then grab the corresponding /dev/X number.
I'd boot up with just one EDA in place for this (I think you have to do that, otherwise you'll also have a unmounted data volume).
The /dev/md123 was from your log fragment above - I thought that was eda1. You can download the log zip again, and find the mdxxx that fails to assemble at the most recent reboot.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: RN316 Upgraded Firmware, EDA500 Volumes are messed up
Yeah, I am going to have to figure out which one of the boxes is EDA1 and which one is EDA2 before I do anything.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: RN316 Upgraded Firmware, EDA500 Volumes are messed up
So I booted NAS with the EDA connected and saw this:
root@NAS:~# cat /proc/mdstat
Personalities : [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md123 : active raid5 sdg3[0] sdk3[4] sdj3[3] sdi3[2] sdh3[1]
39046344448 blocks super 1.2 level 5, 64k chunk, algorithm 2 [5/5] [UUUUU]
[>....................] resync = 0.1% (14019280/9761586112) finish=48698.8min speed=3328K/sec
md124 : active raid5 sda6[0] sdd6[5] sdc6[4] sde6[6] sdb6[2] sdf6[1]
9766864640 blocks super 1.2 level 5, 64k chunk, algorithm 2 [6/6] [UUUUUU]
md125 : active raid5 sda5[5] sdf5[6] sde5[9] sdc5[7] sdd5[10] sdb5[8]
9766864320 blocks super 1.2 level 5, 64k chunk, algorithm 2 [6/6] [UUUUUU]
md126 : active raid5 sda4[6] sdf4[11] sde4[10] sdd4[8] sdc4[9] sdb4[7]
4883113920 blocks super 1.2 level 5, 64k chunk, algorithm 2 [6/6] [UUUUUU]
md127 : active raid5 sda3[6] sdf3[11] sde3[10] sdd3[9] sdc3[8] sdb3[7]
14627073280 blocks super 1.2 level 5, 64k chunk, algorithm 2 [6/6] [UUUUUU]
md0 : active raid1 sda1[17] sdh1[8](W)(S) sdi1[9](W)(S) sdj1[10](W)(S) sdk1[11](W)(S) sdb1[16] sdc1[18] sdd1[19] sde1[20] sdf1[21] sdg1[7](W)
4192192 blocks super 1.2 [7/7] [UUUUUUU]
md1 : active raid10 sda2[0] sdf2[5] sde2[4] sdd2[3] sdc2[2] sdb2[1]
1566720 blocks super 1.2 512K chunks 2 near-copies [6/6] [UUUUUU]
root@NAS:~# mdadm --examine --scan
ARRAY /dev/md/1 metadata=1.2 UUID=e5669719:f7f40928:5ec19a48:dfa7e111 name=7c6e0558:1
ARRAY /dev/md/data-0 metadata=1.2 UUID=2f589054:264b7840:3326a501:d94222fc name=7c6e0558:data-0
ARRAY /dev/md/data-1 metadata=1.2 UUID=8f11ed86:83fff0ac:15af1aa8:ddce01b6 name=7c6e0558:data-1
ARRAY /dev/md/data-2 metadata=1.2 UUID=5a703268:6ae5a904:30dd4315:4381822d name=7c6e0558:data-2
ARRAY /dev/md/data-3 metadata=1.2 UUID=cf948dc9:d2131105:fb56745a:51033405 name=7c6e0558:data-3
ARRAY /dev/md/0 metadata=1.2 UUID=4baa4c4b:a622187f:542e6b5f:4bdbd583 name=7c6e0558:0
spares=4
ARRAY /dev/md/eda3-0 metadata=1.2 UUID=af898d78:0f5b4ce1:ec357aa6:5b12b8d9 name=7c6e0558:eda3-0
root@NAS:~# mdadm --stop /dev/md123
mdadm: Cannot get exclusive access to /dev/md123:Perhaps a running process, mounted filesystem or active volume group?
root@NAS:~# umount /dev/md123
root@NAS:~# mdadm --stop /dev/md123
mdadm: stopped /dev/md123
root@NAS:~# mdadm --assemble /dev/md123
mdadm: /dev/md123 not identified in config file.
So I am not really sure next step.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: RN316 Upgraded Firmware, EDA500 Volumes are messed up
@joe_schmo wrote:
So I booted NAS with the EDA connected and saw this:
So I am not really sure next step.
It looks the eda volume was resyncing, so different symptoms from before. The data might have actually been accessible (with slow performance).
You could either
- add --scan to the assemble command, and see if that works
- reboot and see if you get the same resyncing status on eda 1. If you do, then check that the data is accessible. Wait for the resync to complete before doing anything with eda2.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: RN316 Upgraded Firmware, EDA500 Volumes are messed up
Before I go any further... when that was resyncing, it was showing no data (0 MB). Will allowing this to resync mess anything up? I am trying to go slow with this to make sure I don't do anything dumb.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: RN316 Upgraded Firmware, EDA500 Volumes are messed up
@joe_schmo wrote:
Before I go any further... when that was resyncing, it was showing no data (0 MB). Will allowing this to resync mess anything up? I am trying to go slow with this to make sure I don't do anything dumb.
Sorry, I missed that in the earlier screen shot. That is not a good sign. I definitely wouldn't let it resync. mdadm --assemble might yield the same result.
I am thinking that you might not have a backup. Is that the case?
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: RN316 Upgraded Firmware, EDA500 Volumes are messed up
I have a fairly recent backup, but even if it takes a couple weeks of troubleshooting this it will be quicker than than the restore will be.
It's frustrating because Netgear won't even respond to how I can purchase a support case like the last time this happened or provide me the steps that they did when this happened before.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: RN316 Upgraded Firmware, EDA500 Volumes are messed up
@joe_schmo wrote:It's frustrating because Netgear won't even respond to how I can purchase a support case like the last time this happened or provide me the steps that they did when this happened before.
Tagging the mods ( @Marc_V and @JeraldM ) to see if they can help with the support aspect. They used to offer per-incident support, but now you need to purchase a 12-month support contract.
Though the fact that resync started is troubling, and suggests that data recovery might be needed (not covered by normal support contract).
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: RN316 Upgraded Firmware, EDA500 Volumes are messed up
I had a 516 with two EDA500's (well, one real one and another vendor's equivalent) and found that booting with one of the EDA's disconnected isn't really an issue. It'll show as missing in the GUI and the OS partition will re-sync with the available drives (both when you boot without one, and again when you have them all), but it works. So booting with just one and then the other EDA should be safe to find which is which if you can't figure it out any other way.
Unfortunately, you are finding out (twice now, it seems) why the EDA500 is not a very good solution. There is just too much data being sent up and down the single eSATA channel, and that can lead to corruption. And the connectors don't latch, which can lead to problems if the cable comes loose (which happened to me).
When the loose cable caused me to have a similar issue, I didn't try to recover the volume, I deleted it, re-created it, and restored from backup. So I can't help a lot on finding the cause of the error.
I actually used a 312 to connect to the EDA and create the new volume and restore the backup so as to reduce the load on the 516. I then exported it from that system and imported to the 516. That only works if you don't want XRAID, but you may already be there with the expanded volume(s). And, of course, you need a "spare" NAS that supports the EDA.