Re: Rebooted to Degraded Volume and RAID Groups appeared without XRaid

Craigatitus2 · ‎2021-01-11

Hello

I have a RN428 with Firmware 6.9.3 which has 2x6TB, 2x4TB, and 2x3TB disks installed in that order with a single data volume which was in X-Raid and working just fine.

I just backed up data and did a reboot to then do a firmware upgrade but after it booted up I am now getting a Degraded Volume error and in the GUI it is showing that X-Raid is no longer enabled and I now have 3 RAID groups selectable under the data volume.

Group 1 consists of disks 1,2,5,6

Group 2 consists of disks 2,3,4

Group 3 consists of disks 5,6

When i go click on X-Raid button on the right hand side (which used to be green and is now not) I get a message "You are attempting to switch from Flex-RAID to X-RAID. THis is not possible because you have expanded volumes."

The disk_info.log shows zero errors or bad sectors or anything for any of the disks. I didn't receive any warnings or anything before.

The volume.log shows zero errors but shows one of the pool data as state degraded.

I do see some errors in the kernal.log but i don't really know what they mean.

Jan 11 18:32:06 ReadyNAS kernel: ata5.00: status: { DRDY ERR }
Jan 11 18:32:06 ReadyNAS kernel: ata5.00: error: { UNC }
Jan 11 18:32:06 ReadyNAS kernel: ata5.00: configured for UDMA/133
Jan 11 18:32:06 ReadyNAS kernel: sd 4:0:0:0: [sda] tag#4 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jan 11 18:32:06 ReadyNAS kernel: sd 4:0:0:0: [sda] tag#4 Sense Key : Medium Error [current] [descriptor]
Jan 11 18:32:06 ReadyNAS kernel: sd 4:0:0:0: [sda] tag#4 Add. Sense: Unrecovered read error - auto reallocate failed
Jan 11 18:32:06 ReadyNAS kernel: sd 4:0:0:0: [sda] tag#4 CDB: Read(16) 88 00 00 00 00 00 00 90 00 48 00 00 00 08 00 00
Jan 11 18:32:06 ReadyNAS kernel: blk_update_request: I/O error, dev sda, sector 9437257
Jan 11 18:32:06 ReadyNAS kernel: ata5: EH complete
Jan 11 18:32:06 ReadyNAS kernel: do_marvell_9170_recover: ignoring PCI device (8086:19b2) at PCI#0
Jan 11 18:32:06 ReadyNAS kernel: ata5.00: exception Emask 0x0 SAct 0x100 SErr 0x0 action 0x0
Jan 11 18:32:06 ReadyNAS kernel: ata5.00: irq_stat 0x40000008
Jan 11 18:32:06 ReadyNAS kernel: ata5.00: failed command: READ FPDMA QUEUED
Jan 11 18:32:06 ReadyNAS kernel: ata5.00: cmd 60/08:40:48:00:90/00:00:00:00:00/40 tag 8 ncq 4096 in
res 41/40:00:49:00:90/00:00:00:00:00/00 Emask 0x409 (media error) <F>

There are lots of these types of messages in here.

and also:

Buffer I/O error on dev sda3, logical block 1, async page read

So can anybody help me understand what is going on? 🙂 My data appears to be there when I access the shares. I rebooted it again to see if it would try to resync but it didn't.

Thanks

Craig.

StephenB · ‎2021-01-12

The top priority now is to make sure you have a full backup. Your data is at risk.

Fixing the volume issues you are seeing is tricky, and there is no step-by-step available for it. It's safest to use paid Netgear support to troubleshoot and fix it.

If all the messages are related to sda, then disk 1 is misbehaving. Possibly it needs to be replaced - hard to say w/o looking at the full logs. But simple replacement might not be enough.

If you do want someone to look at the logs, then send a private message (PM) to one of the mods ( @JohnCM_S or @Marc_V ) asking them to take a look. Include a link to this thread. You can't attach the log zip, so you'll also need to upload it to cloud storage (dropbox, google drive, etc), and include a download link. Don't post the logs publicly.

You send a PM using the envelope icon in the upper right of the forum page.

Craigatitus2 · ‎2021-01-12

Thank you for the response. I do have a backup luckily as i just finished it prior to the reboot.

All the messages in the kernal.log do appear to all be sda but to me its strange that the smart and disk info show no errors or anything for that disk.

I will try to PM the log files to the users like you suggested to see if anymore can be determined. I guess my last resort is to factory reset the device and rebuild the X-raid single volume and restore all the data and see if that helps me. I'm just concerned that these messages may mean that the controller in the NAS unit is failing or something like that.

Thanks

Craig.

rn_enthusiast · ‎2021-01-12

There is no raid controller in the ReadyNAS. It is all done with software raids (mdadm). Essentially, it is the exact same as a standard PC with 8 HDD bays and in a different form factor of course.

Craigatitus2 · ‎2021-01-18

Well i pulled out sda and ran some WD lifeguard dianostics on it which did come back with a fail status on the extended test. So I have ordered a replacement for the drive.

When it comes I guess i'll see if i can just pop it in and it will rebuild the xraid array or if the three raid groups will stay there for some reason. If it doesn't go back to xraid i guess i'll have to wipe the volume and create a new fresh one.

Sandshark · ‎2021-01-18

If the volume doesn't get mounted as degraded with just the remaining drives when you re-boot, then it is unlikely that it wiull do so with a replacement drive.

Craigatitus2 · ‎2021-01-25

Well after i replaced the bad drive and the resync finished it showed that the volume was healthy and no longer degraded and all data was still there.

The only issue remaining is that my single X-Raid volume is still not there and i have 3 Raid Groups showing instead. When i click on the Xraid button to the right it says "You are attempting to switch from Flex-Raid to X-Raid. This is not possible becuase you have expanded volumes.

So i'm still not sure how i got to here with the Raid groups and being on Flex-Raid instead of X-Raid but does anybody have any ideas how to get it back to a single X-raid volume before i do a last resort of destroying the volumes and making a new one again?

Thanks.

Sandshark · ‎2021-01-25

Unfortunately, there is no return to XRAID in your case. It used to work, but stopped doing so at some point in OS updates. I suspect it got too complex to verify that the particular FlexRAID configuration was compatible with XRAID. How did you get there? Perhaps you could have done so accidently, though I thought you had to verify the change and that would be harder to do unintentionally.

Unless you plan to do some partial vertical expansion soon, there is no rush to switch. Drive replacement still works fine if one dies, it's just expansion that's limited.

There is actually a way to do XRAID-like expansion in FlexRAID, it's just not easy: How-to-do-incremental-vertical-expansion-in-FlexRAID-mode-using-SSH. I just did that on my main NAS (for real this time, I wrote the post doing it on a "sandbox" system).

Craigatitus2 · ‎2021-01-25

Thank you for the response and i did read through your very detailed post.

I actually do have some vertical expansion to do with somes new drives as that was one of the main reasons I decided to reboot and update the OS to start with first.

I'm not thinking i want to go through all the trouble do the expansion at the command line so I guess I'll have to wipe the volume and start fresh on it.

Thanks

Craig.

StephenB · ‎2021-01-26

@Craigatitus2 wrote:

I'm not thinking i want to go through all the trouble do the expansion at the command line so I guess I'll have to wipe the volume and start fresh on it.

I think there's some confusion here. FlexRAID is a superset of XRAID. You can manually do vertical expansion in FlexRAID, and it doesn't require using the linux command line.

That said, if you want the convenience of XRAID, you'll need to start over.

Sandshark · ‎2021-01-26

@StephenB wrote:
I think there's some confusion here. FlexRAID is a superset of XRAID. You can manually do vertical expansion in FlexRAID, and it doesn't require using the linux command line.

OK, so tell me how to do it. The documentation definately says you cannot do FlexRAID partial vertical expansion, you have to replace all drives. And I just put three larger drives into my NAS to replace drives in a 6-drive RAID5 and the GUI gave me no option to do anything with them.

StephenB · ‎2021-01-26

@Sandshark wrote:

@StephenB wrote:
I think there's some confusion here. FlexRAID is a superset of XRAID. You can manually do vertical expansion in FlexRAID, and it doesn't require using the linux command line.

OK, so tell me how to do it. The documentation definately says you cannot do FlexRAID partial vertical expansion, you have to replace all drives. And I just put three larger drives into my NAS to replace drives in a 6-drive RAID5 and the GUI gave me no option to do anything with them.

You are saying that you can't select the drives with unused space, and then add group?

Sandshark · ‎2021-01-26

@StephenB wrote:
You are saying that you can't select the drives with unused space, and then add group?

That is precisely what I am saying. Netgear's volume_util wouldn't do it, either (though the help kinda makes it look like it should). I couldn't even select any drives individually, just each volume (there are two 6-drive volumes in the 12-bay NAS). That's why I went through the process of figuring out how to do it manually via SSH.

The volumes were originally created via XRAID in an RN516 with EDA500, then converted to FlexRAID so I could move them to the 12-bay chassis. Then I added a 6th drive to the one from the EDA that had only 5 to start. Adding the drive worked fine via the GUI. It's a converted RD5200 running OS6, but I don't see why that would matter.

Craigatitus2 · ‎2021-01-27

Ok the strangness continues...

So I have destroyed my volume and the with it the 3 raid groups in hopes to just create a brand new XRaid volume. After destroying the volume and selecting all the disks I still don't have the ability to create an X-Raid volume. The button is not enabled. Clicking on it doesn't do anything.

Any ideas how to get X-Raid as an option again aside from a Factory Reset?

Sandshark · ‎2021-01-27

Creating an XRAID volume is just done by turning XRAID on. Since XRAID is autromatic, there are no menu items to create anything. If XRAID is on (green bar in the indicator), then try a re-boot and see if that'll kickstart it.

Craigatitus2 · ‎2021-01-27

The problem appears to be that I cannot turn X-Raid on. The button will not turn to having a green bar. I have rebooted and it stayed the same. I tried to format each drive and that didn't work either (it did fail on one drive due to OS mirror on it.

Sandshark · ‎2021-01-27

I don't know how to fix that and probably wouldn't trust it to stay that way if I did. I think factory default is your best next step. Unless you have hard to restore user or app data, it's not much more than creating a new volume.

Craigatitus2 · ‎2021-01-27

Well a Factory Reset has returned my beloved X-Raid but for some reason it has decided to create the volume as Raid 6 instead of Raid 5 and now its doing a massive 17hr Resync on the empty volume.

Should i pull out 1 or 2 of the disks and do another Factory Reset in hopes it picks Raid 5 this time and then add in the drives after?

I guess i could destroy this volume and create a new one in hopes that X-Raid stays enabled and i see if i can select Raid 5.

Any quick ideas via the UI before i go pulling disks and factory resetting again?

Craigatitus2 · ‎2021-01-27

Since i only have 6 disks shouldn't X-Raid by default have made a Raid 5 volume? I thought Raid 6 was for 7+ disks by default?

Craigatitus2 · ‎2021-01-27

Well after some reading in the very detailed forum topic https://community.netgear.com/t5/Using-your-ReadyNAS-in-Business/XRAID-turned-RAID5-into-RAID6-when-...

I found out that 6+ bay units like mine will start off with RAID 6 for 6 drives and if I Factory Reset with only 5 drives in it will create the volume as RAID 5. After that i should be able to horizontally expand the volume to 6 drives and still maintain RAID 5.

So now I have factory reset again with 5 drives and it created the RAID 5 volume and is doing a resync for the next 15 hours. After that I'll pop in the 6th drive and keep my fingers crossed that it stays on RAID 5 like it was before all this mess.

StephenB · ‎2021-01-28

@Craigatitus2 wrote:

I found out that 6+ bay units like mine will start off with RAID 6 for 6 drives and if I Factory Reset with only 5 drives in it will create the volume as RAID 5. After that i should be able to horizontally expand the volume to 6 drives and still maintain RAID 5.

Hopefully that is the case. Let us know how that turns out.

Sandshark · ‎2021-01-28

If all the drives were the same size, it would convert to RAID6 on the 6th drive. But with unequal drives already forming multiple RAID groups, it won't be able to do that, so I think it should expand as RAID5 (with one 2TB RAID1 layer from the 6TB drives). You should have started with the 5 smallest drives and added the last larger one as 6th. If you didn't do that, I think it may still expand. That doesn't follow the rules Netgear has published, but I've seen reports here that it works as long as there is or ever was already at least one drive the same size as the added one so the proper layer size already exists. It's certainly possible for it to work (the XRAID "wedding cake" structure isn't violated), it's just a matter of whether the logic looks for that situation.

Those RAID group layers will be the same ones you saw in FlexRAID, it's just that XRAID hides them from you because it's controlling them.

Craigatitus2 · ‎2021-01-28

Well after the resync finsihed on the 5 drives in X-Raid Raid 5 I put in the 6th drive, deleted the old volume off of it, and then X-Raid took over and started a resync again with all 6 drives still in Raid 5. So that makes me very happy.

So If all goes well in about a day or so I'll see if my volume is all back to normal and then i can start thinking about putting some data back on it.

Sandshark · ‎2021-01-29

Good to know it works as we expected. Now that you know you'll get the structure you want, you can start putting data on it. It'll slow the re-sync a bit, so whether you do it or not just depends on how soon you want to be able to use it again.