This topic has been marked solved and closed to new posts due to inactivity. We hope you'll join the conversation by posting to an open topic or starting a new one.
ReadyNAS Ultra-6 data degraded after firmware update
I have an old ReadyNAS Ultra-6 that I upgraded to Firmware 6.9.5. After the upgrade it shows data degraded. However the data is not degraded, all drives are healthy, I can still access the shares, and have copied files off them and they seem to work fine.
When I go into the web interface I can see it has 2x RAID groups listed that share some (but not all) of the same HDDs!?? Is this normal (or even possible)? I think maybe during the upgrade it found an old array config and reactivate it causing my issue...? I dunno.
While I have copied off some of the more important data, I do not have sufficient space to copy off the remainder (non-critical stuff, but I'd rather not have to download it all again). So before I blow it all away and start from scratch, is there a way to use the console through SSH to fix it without wiping the config? I know nothing about Linux so I'm not keen to start blindly trying stuff, but I can follow instructions...
I don't even know which of the RAID groups is correct, if either. It used to have 6x2TB drives, and I'd swapped in 3x 6TB drives a while back, so now I'm not sure what the correct configuration should be as it's been a while since I looked at it.
Anyway, if anyone has any instructions on how to diagnose or fix this weird issue I'd be grateful for the help. If not I guess I will just wipe the config and start from scratch. 😞
Re: ReadyNAS Ultra-6 data degraded after firmware update
Hi again
Thanks for posting the disk info. As suspected, disk sda has seen better days. From what I can see in the logs, the disk should be located in bay 1 (the first disk in the NAS). We can see that the disk has some Current Pending Sector errors.
Device: sda
Controller: 0
Channel: 0 <<<=== Bay 1
Model: WDC WD60EFRX-68MYMN0
Serial:
Firmware: 82.00A82
Class: SATA
RPM: 5700
Sectors: 11721045168
Pool: data
PoolType: RAID 5
PoolState: 3
PoolHostId: 33eac74a
Health data
ATA Error Count: 0
Reallocated Sectors: 0
Reallocation Events: 0
Spin Retry Count: 0
Current Pending Sector Count: 11 <<<=== Bad sectors on the disk
Uncorrectable Sector Count: 0
Temperature: 32
Start/Stop Count: 136
Power-On Hours: 35247
Power Cycle Count: 77
Load Cycle Count: 15929
Pending sectors typically indicates imminent failure of the disk. An Acronis KB, describes the issue particularly well.
Current Pending Sector Count S.M.A.R.T. parameter is a critical parameter and indicates the current count of unstable sectors (waiting for remapping). The raw value of this attribute indicates the total number of sectors waiting for remapping. Later, when some of these sectors are read successfully, the value is decreased. If errors still occur when reading some sector, the hard drive will try to restore the data, transfer it to the reserved disk area (spare area) and mark this sector as remapped.
Please also consult your machines's or hard disks documentation. Recommendations
This is a critical parameter. Degradation of this parameter may indicate imminent drive failure. Urgent data backup and hardware replacement is recommended.
It is quite likely that these bad sectors caused the NAS to kick the disk from the one of the data raids. The fact that those sectors appear stuck in "pending" is an indication that the sectors will probably never recover. Without further examination of the logs I'd say you need to replace that disk asap. Note: you must replace with a disk of same size or larger.
Your other disks appear to be healthy, which is good!
Re: ReadyNAS Ultra-6 data degraded after firmware update
Since you say you "swapped out" the 6TB drives, I'm assuming the other bays stll have the 2TBs. So, yes, you'll have two RAID groups. One 6x2TB and one 3x4TB. Only a factory default will change that (but there is normally no need to do so). "Degraded" means "no redundancy", not "no access", so it's likely true. Since you have no full backup, you need to fix that before you lose another drive and the volume does become "dead". Unfortunately, fixing it could put your other drives at higher risk if a resync is needed (which I think it will). So fixing the no backup issue should also be on your short list.
It is odd that all drives show green if the volume is degraded, unless it is currently re-syncing. @Hopchen should be able to tell you more from the log, but if you hover over the green dot, do all drives say they are part of volume "data"?
Re: ReadyNAS Ultra-6 data degraded after firmware update
It was all 2tb drives then I swapped in 6tb drives one at a time so now it has 3x6tb and 3x2rb.
Yeah I did more reading and saw that multiple raid groups is result of raid-x expansion.
All drives are green, all listed as part of 'data'. On the unit LCD display where it says degraded all the drive bays are shown and flashing, whether this means they are failed or just populated I don't know.
Re: ReadyNAS Ultra-6 data degraded after firmware update
Was gonna post logs, but cant attach zips, can't rename to png or other and attach, cant post the the raw txt as message character limit is 2000chars... i give up for now.
Firstly, let me clarify the behaviour of the NAS with the disk configuration that you have. The NAS will actually raid partitions together, not entire disks. So, when using different size disks as you do, the NAS will make a 2TB partition across all 6 disks and raid those partitions together in a raid 5. That will form one data raid - md126 in this instance.
Next the NAS will take the 3 remaining larger disks and make 4TB partitions on each disk and raid those partitions together in a separate data raid. In this case, md127.
md127 : active raid5 sda4[0] sdc4[2] sdb4[1]
7813753856 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/3] [UUU]
Thereafter, the NAS sticks the two data raids together on the filesystem level in order to make it "one volume". So, what you are seeing with two data raids is perfectly normal when using different sized disks. @Sandshark - FYI this will be same configuration whether he factory defaults or not.
With that out of the way, we can see that md126 is degraded. The partition from sda (one of the disks) is missing in this raid.
md126 : active raid5 sdc3[8] sdf3[5] sde3[4] sdd3[3] sdb3[6] <<<=== "sda3" missing as a paticipant here.
9743324160 blocks super 1.2 level 5, 64k chunk, algorithm 2 [6/5] [_UUUUU] <<<=== raid notifying you that one disk is out of this raid.
This makes the md126 raid degraded - i.e. non-redundant anymore. Another disk failure will render the entire volume dead at this point so it needs to be addressed. There could be several reasons for the disk going missing in the md126 raid but a firmware update is not a likely suspect. What is more likely is that the "sda" disk has some dodgy sectors on the partition used for the md126 raid and thus the NAS might have kicked it from that raid upon boot.
What is the health of the disks overall? Can you post the output of the disk_info.log (masking any serial numbers for your disks)?
Re: ReadyNAS Ultra-6 data degraded after firmware update
Hi again
Thanks for posting the disk info. As suspected, disk sda has seen better days. From what I can see in the logs, the disk should be located in bay 1 (the first disk in the NAS). We can see that the disk has some Current Pending Sector errors.
Device: sda
Controller: 0
Channel: 0 <<<=== Bay 1
Model: WDC WD60EFRX-68MYMN0
Serial:
Firmware: 82.00A82
Class: SATA
RPM: 5700
Sectors: 11721045168
Pool: data
PoolType: RAID 5
PoolState: 3
PoolHostId: 33eac74a
Health data
ATA Error Count: 0
Reallocated Sectors: 0
Reallocation Events: 0
Spin Retry Count: 0
Current Pending Sector Count: 11 <<<=== Bad sectors on the disk
Uncorrectable Sector Count: 0
Temperature: 32
Start/Stop Count: 136
Power-On Hours: 35247
Power Cycle Count: 77
Load Cycle Count: 15929
Pending sectors typically indicates imminent failure of the disk. An Acronis KB, describes the issue particularly well.
Current Pending Sector Count S.M.A.R.T. parameter is a critical parameter and indicates the current count of unstable sectors (waiting for remapping). The raw value of this attribute indicates the total number of sectors waiting for remapping. Later, when some of these sectors are read successfully, the value is decreased. If errors still occur when reading some sector, the hard drive will try to restore the data, transfer it to the reserved disk area (spare area) and mark this sector as remapped.
Please also consult your machines's or hard disks documentation. Recommendations
This is a critical parameter. Degradation of this parameter may indicate imminent drive failure. Urgent data backup and hardware replacement is recommended.
It is quite likely that these bad sectors caused the NAS to kick the disk from the one of the data raids. The fact that those sectors appear stuck in "pending" is an indication that the sectors will probably never recover. Without further examination of the logs I'd say you need to replace that disk asap. Note: you must replace with a disk of same size or larger.
Your other disks appear to be healthy, which is good!
Re: ReadyNAS Ultra-6 data degraded after firmware update
Since many users (myself included) rarely re-boot their NAS except when up-dating the OS, that re-boot can ben the cause of discovery of a problem that has actually persisted for a while. Folks tend to blame the update, but that's rarely the root cause; it's just the catalyst for the requirement to reboot. You are likely in this group; so it's not "coincidence" at all; it's a convergence of circumstance.
11 pending sectors is potentially a problem, and drive replacement seems in order. Note that since this does add additional workload to the other old drives, insuring your backup is up to date before proceeding is good insurance.
Re: ReadyNAS Ultra-6 data degraded after firmware update
Thanks for all the help guys, pity the web interface couldn't tell me this info, it seems the NAS has acces to all the data for it to do so... 😕 Even the LCD panel was flashing all the drives as degraded, surely it should only flash the faulty drive (disk 1) so the user knows what to replace.
I have backed it up, and will replace the faulty drive when I can afford it. 😄
Re: ReadyNAS Ultra-6 data degraded after firmware update
It's the entire volume that is degraded, not a disk.
But I agree that Netgear needs to a better job of identifying which disk(s) have dropped out of the array in the web UI. Forcing users to delve into mdstat.log, etc isn't great.
Also, they should implement user-defined thresholds for disk health (pending sectors, etc). 11 pending sectors are well below their hard-wired thresholds, so you will just see a "green" healthy disk in the web ui. IMO the hard-wired thresholds are too high (and based on posts here, Netgear support agrees that is the case.).
It's the entire volume that is degraded, not a disk.
But I agree that Netgear needs to a better job of identifying which disk(s) have dropped out of the array in the web UI. Forcing users to delve into mdstat.log, etc isn't great.
Also, they should implement user-defined thresholds for disk health (pending sectors, etc). 11 pending sectors are well below their hard-wired thresholds, so you will just see a "green" healthy disk in the web ui. IMO the hard-wired thresholds are too high (and based on posts here, Netgear support agrees that is the case.).
I completely agree with this. It falls in the same category as the incredibly confusing message: "Remove Inactive Volumes". The challenge in this case was that sda dropped from md126 but it was still active in md127. Regardless, there should be an improved way of notifying the user other than the display blinking and showing: "Degraded". I think that user notification is an area where NETGEAR has a long way to go. It is fine if you know the system and can pull stats from logs, etc. However, for the average user, the front-end notifications are sub-par.
IMO, you cannot generalise disks errors to some arbitrary count as in, "it is only critical if you hit X mount of errors". Disks errors can hit different areas of the disk (some more important than others) and ATA errors can affect different aspects in how the disk operate.
