× NETGEAR will be terminating ReadyCLOUD service by July 1st, 2023. For more details click here.
Orbi WiFi 7 RBE973
Reply

RN316 Upgraded Firmware, EDA500 Volumes are messed up

joe_schmo
Tutor

RN316 Upgraded Firmware, EDA500 Volumes are messed up

I just upgraded my firmware.  machine booted and then one of my EDA500s doesn't show up.  The other is also messed up.

 

Doesn't seem that netgear provides support anymore... I used to be able to buy a support contract but have no idea how to do that anymore.  Can anyone help me figure this out?

Message 1 of 42
joe_schmo
Tutor

Re: RN316 Upgraded Firmware, EDA500 Volumes are messed up

So if I disconnect the two EDA500s, my NAS boots up and all is well.
If I turn one of the EDA500s on, it shows up as a mix of unusableness.

 

If I boot up the NAS while the two EDA500s are powered on and connected, the NAS shows every drive as red.

 

I am assuming there is something I can do to fix this.  Support tells me to post to the forums and won't tell me if you can purchase a support ticket like back in 2018.  

Anyone have any ideas?

Message 2 of 42
joe_schmo
Tutor

Re: RN316 Upgraded Firmware, EDA500 Volumes are messed up

more info...

 

Without the EDA500s on, I see my NAS showing up in the Volumes part with 

 

EDA1

EDA2

EDA3

 

All with data/free at 0.

 

I only ever had 2 EDA.

 

So wondering if anyone can help tell me what to do to get these back.

Message 3 of 42
StephenB
Guru

Re: RN316 Upgraded Firmware, EDA500 Volumes are messed up


@joe_schmo wrote:


If I turn one of the EDA500s on, it shows up as a mix of unusableness.

 


Does it matter which one?  Or does turning either one on cause this problem?

Message 4 of 42
joe_schmo
Tutor

Re: RN316 Upgraded Firmware, EDA500 Volumes are messed up

It's both.  Here's a screenshot with what happens when I boot the NAS by itself and then turn on one EDA.naserror.png 

Message 5 of 42
StephenB
Guru

Re: RN316 Upgraded Firmware, EDA500 Volumes are messed up


@joe_schmo wrote:

It's both.  Here's a screenshot with what happens when I boot the NAS by itself and then turn on one EDA. 


Have you captured a full log zip file?  I am suspecting that the two EDA500 volumes are out of sync.  @Sandshark might have some suggestions (he has had EDA500s in the past, I have not).

 

FWIW, I think you must have vertically expanded your EDA500 volumes (that would explain the three eda1 RAID groups).

Message 6 of 42
joe_schmo
Tutor

Re: RN316 Upgraded Firmware, EDA500 Volumes are messed up

I have long ago vertically expanded.  Probably 6 or 7 years maybe?

I do have the logs downloaded.

Message 7 of 42
StephenB
Guru

Re: RN316 Upgraded Firmware, EDA500 Volumes are messed up

I suggest looking in dmesg.log, kernel.log, system.log, and systemd-journal.log.  Look at the most recent boot sequence for errors related to disk, mdadm, and btrfs.

Message 8 of 42
joe_schmo
Tutor

Re: RN316 Upgraded Firmware, EDA500 Volumes are messed up

Not sure exactly what to look for.

 

Kernel.log shows:

Oct 08 14:09:05 NAS kernel: md/raid:md124: device sda6 operational as raid disk 0
Oct 08 14:09:05 NAS kernel: md/raid:md124: device sdd6 operational as raid disk 5
Oct 08 14:09:05 NAS kernel: md/raid:md124: device sdc6 operational as raid disk 4
Oct 08 14:09:05 NAS kernel: md/raid:md124: device sde6 operational as raid disk 3
Oct 08 14:09:05 NAS kernel: md/raid:md124: device sdb6 operational as raid disk 2
Oct 08 14:09:05 NAS kernel: md/raid:md124: device sdf6 operational as raid disk 1
Oct 08 14:09:05 NAS kernel: md/raid:md124: allocated 6474kB
Oct 08 14:09:05 NAS kernel: md/raid:md124: raid level 5 active with 6 out of 6 devices, algorithm 2
Oct 08 14:09:05 NAS kernel: RAID conf printout:
Oct 08 14:09:05 NAS kernel:  --- level:5 rd:6 wd:6
Oct 08 14:09:05 NAS kernel:  disk 0, o:1, dev:sda6
Oct 08 14:09:05 NAS kernel:  disk 1, o:1, dev:sdf6
Oct 08 14:09:05 NAS kernel:  disk 2, o:1, dev:sdb6
Oct 08 14:09:05 NAS kernel:  disk 3, o:1, dev:sde6
Oct 08 14:09:05 NAS kernel:  disk 4, o:1, dev:sdc6
Oct 08 14:09:05 NAS kernel:  disk 5, o:1, dev:sdd6
Oct 08 14:09:05 NAS kernel: md124: detected capacity change from 0 to 10001269391360
Oct 08 14:09:05 NAS kernel: BTRFS: device label 7c6e0558:data devid 4 transid 7395299 /dev/md124
Oct 08 14:09:05 NAS kernel: md: md123 stopped.
Oct 08 14:09:05 NAS kernel: md: bind<sdm3>
Oct 08 14:09:05 NAS kernel: md/raid:md123: device sdm3 operational as raid disk 0
Oct 08 14:09:05 NAS kernel: md/raid:md123: allocated 5418kB
Oct 08 14:09:05 NAS kernel: md/raid:md123: not enough operational devices (4/5 failed)
Oct 08 14:09:05 NAS kernel: RAID conf printout:
Oct 08 14:09:05 NAS kernel:  --- level:5 rd:5 wd:1
Oct 08 14:09:05 NAS kernel:  disk 0, o:1, dev:sdm3
Oct 08 14:09:05 NAS kernel: md/raid:md123: failed to run raid set.

 

Nothing in dmesg.log or system.log looks odd.

 

 

 

Message 9 of 42
joe_schmo
Tutor

Re: RN316 Upgraded Firmware, EDA500 Volumes are messed up

In 2018 this is what they did to fix a similar issue:

This is Brian from NETGEAR Level 3 Support.

I have fixed your issue with one of your EDAs and what caused the problem was two of it's drives have lost its partitions. 
I have recreated them and this shouldn't occur again. 

If I knew what commands they ran, I'd just do it myself.  

Message 10 of 42
StephenB
Guru

Re: RN316 Upgraded Firmware, EDA500 Volumes are messed up

This is consistent with what you are seeing in the web ui:

 

Oct 08 14:09:05 NAS kernel: md/raid:md123: device sdm3 operational as raid disk 0
Oct 08 14:09:05 NAS kernel: md/raid:md123: allocated 5418kB
Oct 08 14:09:05 NAS kernel: md/raid:md123: not enough operational devices (4/5 failed)
Oct 08 14:09:05 NAS kernel: RAID conf printout:
Oct 08 14:09:05 NAS kernel:  --- level:5 rd:5 wd:1
Oct 08 14:09:05 NAS kernel:  disk 0, o:1, dev:sdm3
Oct 08 14:09:05 NAS kernel: md/raid:md123: failed to run raid set.

However, it seems unlikely that four out of five disks actually failed.

 

Do you have ssh enabled?

 

Message 11 of 42
joe_schmo
Tutor

Re: RN316 Upgraded Firmware, EDA500 Volumes are messed up

I do (once upon a time with another NAS I even logged in tech support mode to fix something).  And I have the log files too.  The drives are healthy.


Message 12 of 42
StephenB
Guru

Re: RN316 Upgraded Firmware, EDA500 Volumes are messed up


@joe_schmo wrote:

I do 



I suggest running smartctl -x on all five of the EDA500 drives, just to make sure that the RN316 is still "seeing" them.

Message 13 of 42
joe_schmo
Tutor

Re: RN316 Upgraded Firmware, EDA500 Volumes are messed up

It still sees them.


I booted up with the EDAs unplugged, got this screenshot.  Ran smartctl on each of those.  Rebooted, the NAS came up fine, plugged in the EDA and smartctl ran fine on those drives as well.

joe_schmo_0-1665431923160.png

 

Message 14 of 42
StephenB
Guru

Re: RN316 Upgraded Firmware, EDA500 Volumes are messed up

I'd get back to the state where the internal array is ok.

 

Then perhaps try

 

mdadm --stop /dev/md123
mdadm --assemble /dev/md123 

 

 

If you get an error on the second command, then try again with --force.  This isn't completely safe, but I think you will probably need it.

 

mdadm --stop /dev/md123
mdadm --force --assemble  /dev/md123

 

 

After that, try

btrfs device scan
mount /dev/md123 /eda1

 

If the mount is successful, then try rebooting and make sure both volumes are shown on the volume screen and accessible.

Message 15 of 42
joe_schmo
Tutor

Re: RN316 Upgraded Firmware, EDA500 Volumes are messed up

Sounds good.  Thank you.  I'll keep you posted on how this goes.

So I just need to make sure I know which EDA is eda1 and which one is eda2, and then grab the corresponding  /dev/X number.

Do I need to be in tech support mode for this or just a normal bash prompt?

Message 16 of 42
StephenB
Guru

Re: RN316 Upgraded Firmware, EDA500 Volumes are messed up


@joe_schmo wrote:

Do I need to be in tech support mode for this or just a normal bash prompt?


I think you will be able to do it with the normal bash prompt (logging in as root).

 


@joe_schmo wrote:


So I just need to make sure I know which EDA is eda1 and which one is eda2, and then grab the corresponding  /dev/X number.


I'd boot up with just one EDA in place for this (I think you have to do that, otherwise you'll also have a unmounted data volume).

 

The /dev/md123 was from your log fragment above - I thought that was eda1. You can download the log zip again, and find the mdxxx that fails to assemble at the most recent reboot.

 

 

Message 17 of 42
joe_schmo
Tutor

Re: RN316 Upgraded Firmware, EDA500 Volumes are messed up

Yeah, I am going to have to figure out which one of the boxes is EDA1 and which one is EDA2 before I do anything.

Message 18 of 42
joe_schmo
Tutor

Re: RN316 Upgraded Firmware, EDA500 Volumes are messed up

So I booted NAS with the EDA connected and saw this: 

joe_schmo_0-1665465335558.png

 

root@NAS:~# cat /proc/mdstat
Personalities : [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md123 : active raid5 sdg3[0] sdk3[4] sdj3[3] sdi3[2] sdh3[1]
      39046344448 blocks super 1.2 level 5, 64k chunk, algorithm 2 [5/5] [UUUUU]
      [>....................]  resync =  0.1% (14019280/9761586112) finish=48698.8min speed=3328K/sec

md124 : active raid5 sda6[0] sdd6[5] sdc6[4] sde6[6] sdb6[2] sdf6[1]
      9766864640 blocks super 1.2 level 5, 64k chunk, algorithm 2 [6/6] [UUUUUU]

md125 : active raid5 sda5[5] sdf5[6] sde5[9] sdc5[7] sdd5[10] sdb5[8]
      9766864320 blocks super 1.2 level 5, 64k chunk, algorithm 2 [6/6] [UUUUUU]

md126 : active raid5 sda4[6] sdf4[11] sde4[10] sdd4[8] sdc4[9] sdb4[7]
      4883113920 blocks super 1.2 level 5, 64k chunk, algorithm 2 [6/6] [UUUUUU]

md127 : active raid5 sda3[6] sdf3[11] sde3[10] sdd3[9] sdc3[8] sdb3[7]
      14627073280 blocks super 1.2 level 5, 64k chunk, algorithm 2 [6/6] [UUUUUU]

md0 : active raid1 sda1[17] sdh1[8](W)(S) sdi1[9](W)(S) sdj1[10](W)(S) sdk1[11](W)(S) sdb1[16] sdc1[18] sdd1[19] sde1[20] sdf1[21] sdg1[7](W)
      4192192 blocks super 1.2 [7/7] [UUUUUUU]

md1 : active raid10 sda2[0] sdf2[5] sde2[4] sdd2[3] sdc2[2] sdb2[1]
      1566720 blocks super 1.2 512K chunks 2 near-copies [6/6] [UUUUUU]
root@NAS:~# mdadm --examine --scan
ARRAY /dev/md/1  metadata=1.2 UUID=e5669719:f7f40928:5ec19a48:dfa7e111 name=7c6e0558:1
ARRAY /dev/md/data-0  metadata=1.2 UUID=2f589054:264b7840:3326a501:d94222fc name=7c6e0558:data-0
ARRAY /dev/md/data-1  metadata=1.2 UUID=8f11ed86:83fff0ac:15af1aa8:ddce01b6 name=7c6e0558:data-1
ARRAY /dev/md/data-2  metadata=1.2 UUID=5a703268:6ae5a904:30dd4315:4381822d name=7c6e0558:data-2
ARRAY /dev/md/data-3  metadata=1.2 UUID=cf948dc9:d2131105:fb56745a:51033405 name=7c6e0558:data-3
ARRAY /dev/md/0  metadata=1.2 UUID=4baa4c4b:a622187f:542e6b5f:4bdbd583 name=7c6e0558:0
   spares=4
ARRAY /dev/md/eda3-0  metadata=1.2 UUID=af898d78:0f5b4ce1:ec357aa6:5b12b8d9 name=7c6e0558:eda3-0
root@NAS:~# mdadm --stop /dev/md123
mdadm: Cannot get exclusive access to /dev/md123:Perhaps a running process, mounted filesystem or active volume group?

root@NAS:~# umount  /dev/md123

root@NAS:~# mdadm --stop /dev/md123
mdadm: stopped /dev/md123

root@NAS:~# mdadm --assemble /dev/md123
mdadm: /dev/md123 not identified in config file.

So I am not really sure next step. 

Message 19 of 42
StephenB
Guru

Re: RN316 Upgraded Firmware, EDA500 Volumes are messed up


@joe_schmo wrote:

So I booted NAS with the EDA connected and saw this: 

joe_schmo_0-1665465335558.png

 

So I am not really sure next step. 


It looks the eda volume was resyncing, so different symptoms from before.  The data might have actually been accessible (with slow performance).

 

You could either

  1. add --scan to the assemble command, and see if that works
  2. reboot and see if you get the same resyncing status on eda 1.  If you do, then check that the data is accessible.  Wait for the resync to complete before doing anything with eda2.
Message 20 of 42
joe_schmo
Tutor

Re: RN316 Upgraded Firmware, EDA500 Volumes are messed up

Before I go any further... when that was resyncing, it was showing no data (0 MB).  Will allowing this to resync mess anything up?  I am trying to go slow with this to make sure I don't do anything dumb.

Message 21 of 42
StephenB
Guru

Re: RN316 Upgraded Firmware, EDA500 Volumes are messed up


@joe_schmo wrote:

Before I go any further... when that was resyncing, it was showing no data (0 MB).  Will allowing this to resync mess anything up?  I am trying to go slow with this to make sure I don't do anything dumb.


Sorry, I missed that in the earlier screen shot.  That is not a good sign.  I definitely wouldn't let it resync. mdadm --assemble might yield the same result.

 

I am thinking that you might not have a backup.  Is that the case?

Message 22 of 42
joe_schmo
Tutor

Re: RN316 Upgraded Firmware, EDA500 Volumes are messed up

I have a fairly recent backup, but even if it takes a couple weeks of troubleshooting this it will be quicker than than the restore will be.

 

It's frustrating because Netgear won't even respond to how I can purchase a support case like the last time this happened or provide me the steps that they did when this happened before. 

Message 23 of 42
StephenB
Guru

Re: RN316 Upgraded Firmware, EDA500 Volumes are messed up


@joe_schmo wrote:

It's frustrating because Netgear won't even respond to how I can purchase a support case like the last time this happened or provide me the steps that they did when this happened before. 


Tagging the mods ( @Marc_V and @JeraldM ) to see if they can help with the support aspect.  They used to offer per-incident support, but now you need to purchase a 12-month support contract.

 

Though the fact that resync started is troubling, and suggests that data recovery might be needed (not covered by normal support contract).

 

 

Message 24 of 42
Sandshark
Sensei

Re: RN316 Upgraded Firmware, EDA500 Volumes are messed up

I had a 516 with two EDA500's (well, one real one and another vendor's equivalent) and found that booting with one of the EDA's disconnected isn't really an issue.  It'll show as missing in the GUI and the OS partition will re-sync with the available drives (both when you boot without one, and again when you have them all), but it works.  So booting with just one and then the other EDA should be safe to find which is which if you can't figure it out any other way.

 

Unfortunately, you are finding out (twice now, it seems) why the EDA500 is not a very good solution.  There is just too much data being sent up and down the single eSATA channel, and that can lead to corruption.  And the connectors don't latch, which can lead to problems if the cable comes loose (which happened to me).

 

When the loose cable caused me to have a similar issue, I didn't try to recover the volume, I deleted it, re-created it, and restored from backup.  So I can't help a lot on finding the cause of the error.

 

I actually used a 312 to connect to the EDA and create the new volume and restore the backup so as to reduce the load on the 516.  I then exported it from that system and imported to the 516.  That only works if you don't want XRAID, but you may already be there with the expanded volume(s).  And, of course, you need a "spare" NAS that supports the EDA.

Message 25 of 42
Top Contributors
Discussion stats
  • 41 replies
  • 2049 views
  • 0 kudos
  • 4 in conversation
Announcements