× NETGEAR will be terminating ReadyCLOUD service by July 1st, 2023. For more details click here.
Orbi WiFi 7 RBE973
Reply

Re: Remove inactive volumes on EDA500 expansion chassis

jmaxlow
Aspirant

Remove inactive volumes on EDA500 expansion chassis

I have run an RN516 with three EDA500 expansion chassis for multiple years. Recently, an error where one disk in one of the expansions disconnected and then reconnected (on its own) triggered a resync. That process stalled wherein it would not progress past a certain percentage. So, I rebooted the system.

 

Now, two of the three chassis give the "remove inactive volume" messages. Five out of ten disks total, and none in the chassis that is (again) stalled in a resync. 

 

I can SSH into the unit but the web front end doesn't land on the main page after accepting my username and password. I can see the proper volumes in data and eda1 (the main unit volume and one of the expansion volumes) and those shares are accessible to my Windows machine. Via the SSH command line, I can see directories for eda2 and eda3, but they show no shares / folders / files in them, unlike the data and eda1 volumes.

 

Since I can't get the front end to work, I can't download logs, but there were no disk errors reported via the front end or email. Trouble only started happening on those two expansions after the reboot. And I still don't know why the resync on eda1 keeps stalling.

 

I'm trying to get help via chat, but that is a laborious process due to no technicians being available. Is there anything I can try in the short term to get the volumes in eda2 and eda3 mounted again? Or any way to get the proper logs via the command line to post here?

Model: RN51600|ReadyNAS 516 6-Bay Diskless
Message 1 of 12

Accepted Solutions
Hopchen
Prodigy

Re: Remove inactive volumes on EDA500 expansion chassis

Hi all

 

Just to follow up on this one for those interested.

 

@jmaxlow  was getting inactive volume error on both EDA-2 and EDA-3. The reason being that neither raid can start.
The raids cannot start because several disks in both EDAs are missing the data partition. EDA disks should have two partitions: 1 for the OS and 1 for the data. These are referenced to as partitions 1 and 3. Several disks are missing the data partition and thus the raids cannot run.

=== EDA-2 ===

8 176 5860522584 sdl
8 177 4194304 sdl1

8 192 5860522584 sdm
8 193 4194304 sdm1
8 195 5855803936 sdm3

8 208 5860522584 sdn
8 209 4194304 sdn1

8 224 5860522584 sdo
8 225 4194304 sdo1
8 227 5855803936 sdo3

8 240 5860522584 sdp
8 241 4194304 sdp1

=== EDA-3 === 65 0 4883770584 sdq 65 1 4194304 sdq1 65 3 4879051936 sdq3 65 16 4883770584 sdr 65 17 4194304 sdr1 65 32 4883770584 sds 65 33 4194304 sds1 65 35 4879051936 sds3 65 48 4883770584 sdt 65 49 4194304 sdt1 65 64 4883770584 sdu 65 65 4194304 sdu1 65 67 4879051936 sdu3

 

I am a little baffled as to how this happened on so many disks. Perhaps there is a communications issue between the EDAs and the main chassis... Disks appear healthy.


Anyhow, this should be recoverable by copying over the partition table from "working" disks and thus restore the lost partitions. As long as nothing was over-written on those disks we should be good.


I recommended to reach out to NETGEAR and show them this.

 

 

Cheers

View solution in original post

Message 9 of 12

All Replies
jmaxlow
Aspirant

Re: Remove inactive volumes on EDA500 expansion chassis

And now the web interface is inaccessible, with eda1's resync seemingly stuck at 14.08%. Here is my RAIDar result.

 

Is there an SSH command I can use to "restart" the web interface?

 

 

Message 2 of 12
StephenB
Guru

Re: Remove inactive volumes on EDA500 expansion chassis

Did you try downloading the logs from RAIDar?

Message 3 of 12
jmaxlow
Aspirant

Re: Remove inactive volumes on EDA500 expansion chassis

RAIDar would ask for a username and password for the log download, but would then reject them, and would do the same for the graceful Restart.

 

I had to do a hard restart. At that point (last night), I powered off the two troublesome expansion chassis so that FrontView would load and the resync on eda1 would continue. So now that resync is progressing. I can download the logs but they don't seem to contain any errors specific to eda2 and eda3 - no SMART errors and such. But the content on those two in the logs is not as voluminous as the content on the RN516 and eda1. Once it is finished (Wednesday?) I plan to turn eda2 and eda3 back on and try to get those logs.

 

Would it be helpful to post what logs I have today, or what until the resync is finished and I attempt to turn the other two chassis back on?

 

I am very much hoping that it's the case that the data on eda2 and eda3 is still there, and the arrays just need to be reassembled via the command line, but I pretty much have to wait until the resync on eda1 is done before chat support is willing to get more involved. I'm prepared to use Reclaime if I have to, but perhaps something simpler and faster will work.

Message 4 of 12
StephenB
Guru

Re: Remove inactive volumes on EDA500 expansion chassis


@jmaxlow wrote:

 

Would it be helpful to post what logs I have today, or what until the resync is finished and I attempt to turn the other two chassis back on?


Well, don't post a link to the full log zip file here.

 

Instead, send a private message (PM) to the mods ( @JohnCM_S or @Marc_V ) and ask them if they are willing to take a look.  @Hopchen might also be willing to analyze them.

Message 5 of 12
Hopchen
Prodigy

Re: Remove inactive volumes on EDA500 expansion chassis

Yea I can take a look at the logs, no problem.

 

Upload log zip-file to a Google link or similar and PM me the link.

Message 6 of 12
Sandshark
Sensei

Re: Remove inactive volumes on EDA500 expansion chassis

The existance of "eda2" with "eda2-0"  and "eda3" with "eda3-0" says there is (probably only) one drive in each chassis that is out of sync.  It thinks there are two volumes with the same name, which it can't resolve.  And while it may seem obvious which one is the "true" volume by the amount of space used, that's not always the case.

 

The problem is that you really need to get past the stalled sync.  I'm assuming here that top shows it's eating up all the CPU throughput and that watch cat /proc/mdstat shows no progress in the re-sync and time to go just mounting up.  I've seen that before, and it often keeps other commands issued via SSH from working.

 

Whether something happend that drove all the volumes out of sync or if the resync of eda1 dominoed into messing up eda2 and eda3, it's hard to say.  I had a similar situation as your eda1 recently with an external eSATA 5 drive chassis (works like an EDA, but is from another company) and nothing I did could save that volume.  I resigned myself that it was lost, and I deleted the volume, re-created it, and restored from backup.  But before I did that, I booted with it not connected but my real EDA500 connected, and it booted.  Yours may as well and give you more information on which drive is the out of sync one.  If you can remove that drive and re-boot, the "phantom" volume should disappear and you can then put the drive back in, format it, and then let it re-sync.

 

Since you have three EDA's, you may want to boot with just one, get it working, then boot with the second.  Note that when you boot without it, the volume will still show up in the GUI as missing.  Don't destroy that unless you plan to start over with the EDA.  If you get one EDA volume working and want to keep it "safe" when working on another, do an export.  Note that once re-imported (whihc happens automatically when you boot with it attached), you will need to go back in and set protocols and permissions on the shares.  And this whole thing could move you to FlexRAID mode, where you could be stuck (assuming you aren't there already).

 

Note, too, that every time you boot with a sub-set of total drives, md0 (the OS partition) may need to sync.  It doesn't take long, but it's best not to power down while it's ongoing.

Message 7 of 12
Hopchen
Prodigy

Re: Remove inactive volumes on EDA500 expansion chassis

I am working with @jmaxlow in the background. We will take a look at each EDA in turn.

I will update thread when we get further. Will take some time due to raid syncing, etc.

Message 8 of 12
Hopchen
Prodigy

Re: Remove inactive volumes on EDA500 expansion chassis

Hi all

 

Just to follow up on this one for those interested.

 

@jmaxlow  was getting inactive volume error on both EDA-2 and EDA-3. The reason being that neither raid can start.
The raids cannot start because several disks in both EDAs are missing the data partition. EDA disks should have two partitions: 1 for the OS and 1 for the data. These are referenced to as partitions 1 and 3. Several disks are missing the data partition and thus the raids cannot run.

=== EDA-2 ===

8 176 5860522584 sdl
8 177 4194304 sdl1

8 192 5860522584 sdm
8 193 4194304 sdm1
8 195 5855803936 sdm3

8 208 5860522584 sdn
8 209 4194304 sdn1

8 224 5860522584 sdo
8 225 4194304 sdo1
8 227 5855803936 sdo3

8 240 5860522584 sdp
8 241 4194304 sdp1

=== EDA-3 === 65 0 4883770584 sdq 65 1 4194304 sdq1 65 3 4879051936 sdq3 65 16 4883770584 sdr 65 17 4194304 sdr1 65 32 4883770584 sds 65 33 4194304 sds1 65 35 4879051936 sds3 65 48 4883770584 sdt 65 49 4194304 sdt1 65 64 4883770584 sdu 65 65 4194304 sdu1 65 67 4879051936 sdu3

 

I am a little baffled as to how this happened on so many disks. Perhaps there is a communications issue between the EDAs and the main chassis... Disks appear healthy.


Anyhow, this should be recoverable by copying over the partition table from "working" disks and thus restore the lost partitions. As long as nothing was over-written on those disks we should be good.


I recommended to reach out to NETGEAR and show them this.

 

 

Cheers

Message 9 of 12
Sandshark
Sensei

Re: Remove inactive volumes on EDA500 expansion chassis

Definately baffling.  On my EDA, the data partition starts where the space for the swap partition (sdx2) ends, though the swap partition is empty.  That makes sense.  Swap would be very slow if it used the EDA, but reserving it means the volume could be moved into a main NAS chassis and then the swap volume would be added.

 

Before I had an issue and mounted what was my second EDA by itself, only the first drive in the second EDA had anything actually in the OS partition (sdx1), though it was also present.  I wondered why, but did realize that that meant I had 12 that did -- the maximum number of drives in any ReadyNAS at the time. 

 

CFDISK is a good tool to go in and look at the current partitioning and make changes to it.  The only thing is that the HELP function crashes (on all of my NASes, so it's generic).  But it's pretty interactive and easy to figure out.

Message 10 of 12
jmaxlow
Aspirant

Re: Remove inactive volumes on EDA500 expansion chassis

Resolved! A level 3 engineer recreated the partitions, the volumes and arrays were recognized, and all data was intact.

Message 11 of 12
Hopchen
Prodigy

Re: Remove inactive volumes on EDA500 expansion chassis

Super, thanks for the update @jmaxlow

Message 12 of 12
Top Contributors
Discussion stats
  • 11 replies
  • 1875 views
  • 0 kudos
  • 4 in conversation
Announcements