Lost all Volume Data after Firmware 6.9.5 upgrade today
Hi . First time post, so excuse if i make any newbie protocol errors.
Some advice please. I have a ReadyNas 516 unit that i've had since new in August 2014. I have all 6x bays in use, with 6x 3TB WD RED hard drives in there. The unit runs like a dream for what i need it for, ie main area to save files to for the family.
Over the last month i seem to have started to have problems. I got a warning that a drive had failed (BAY 4) over the Christmas period. (27.12.18) and the log message changed to "Amber - volume data health changed from redundant to degraded." As i was on holiday at the time of the failure, and didnt return until 4.1.19 so the unit had managed without any interferance from me. I have used the device sparingly since then, only saving files to it occassionally, awaiting the spare cash to purchase the failed hard drive in bay 4, or so i thought.
Stupidly this morning I got a notice that a new firmware was available for the unit, so without thinking I have run it, moving it to the latetst version 6.9.5. During the installation of this firmware, it seemed to bring online the drive in bay 4, and went into resync mode. after this task completed it rebooted the readynas. After this reboot i'm now getting an error message on the readynas admin page that is telling me "to remove inactive volumes to use the disk Disk #1,2,3,4,5,6.. I dont undertsand this message so have then started to look on line for advice.
So i'm confused. I cannot access the device. I havent removed any drives from the readynas yet. I dont understand how the volumes could have disappreared. I dont really understand what the information is telling me in image "screen grab one" but it is still showing that i have my data 13.62TB on the unit. i have also attached a copy of my log, i hope that is ok forum wise.
Re: Lost all Volume Data after Firmware 6.9.5 upgrade today
Hi WayneDudds,
Welcome to the Community!
You may download the logs and then provide it to us so we can further check what caused the issue. You may upload it to a file sharing site and then PM me download link.
Re: Lost all Volume Data after Firmware 6.9.5 upgrade today
Hi JohnCM_S,
I went through nearly identical steps as WayneDudds did and experienced the same problem, only difference is I am using a RN716x with 6x4Gb WDC disks. Can you help me out as well please?
Re: Lost all Volume Data after Firmware 6.9.5 upgrade today
Hi WayneDudds,
Thank you for providing the logs. Please do not post it publicly next time because the logs contain sensitive data that may compromise privacy. You may just PM (private message) the download link to the forum moderator who requested it. You can use PM by clicking the envelope symbol on the upper right hand corner of the page.
We will review the logs first and then provide you a feedback as soon as possible.
Aside from disk 4, your disk 6 has failed as well. 2 out of 6 drives are not operational anymore which is the reason the volume could not be mounted. Please prepare a replacement disks for the failed disks and you may contact support for data recovery.
Jan 24 14:31:53 TABLA-RASA kernel: md/raid:md127: device sda3 operational as raid disk 0
Jan 24 14:31:53 TABLA-RASA kernel: md/raid:md127: device sde3 operational as raid disk 5
Jan 24 14:31:53 TABLA-RASA kernel: md/raid:md127: device sdc3 operational as raid disk 2
Jan 24 14:31:53 TABLA-RASA kernel: md/raid:md127: device sdb3 operational as raid disk 1
Jan 24 14:31:53 TABLA-RASA kernel: md/raid:md127: allocated 6474kB
Jan 24 14:31:53 TABLA-RASA kernel: md/raid:md127: not enough operational devices (2/6 failed)
You also have multiple disk failure on your NAS, disk 2 and disk 4 has failed. Your disk 1 and disk 3 are not looking good either. Please prepare as well for replacement disks and it will be best for you to contact support as well so they can assist you on the data recovery.
Jan 24 20:44:00 WANKA02-NAS1 kernel: md/raid:md127: device sda3 operational as raid disk 0
Jan 24 20:44:00 WANKA02-NAS1 kernel: md/raid:md127: device sdf3 operational as raid disk 5
Jan 24 20:44:00 WANKA02-NAS1 kernel: md/raid:md127: device sde3 operational as raid disk 4
Jan 24 20:44:00 WANKA02-NAS1 kernel: md/raid:md127: device sdc3 operational as raid disk 2
Jan 24 20:44:00 WANKA02-NAS1 kernel: md/raid:md127: allocated 6474kB
Jan 24 20:44:00 WANKA02-NAS1 kernel: md/raid:md127: not enough operational devices (2/6 failed)
Re: Lost all Volume Data after Firmware 6.9.5 upgrade today
Thank you for the detailed response John. Do you mind if I ask questions, to help me learn from this experience.
The unit was purchased new in August 2014, so it is not 5 yrs old yet. I also remember purchasing all the Red Western Digital drives at the same time too. So I now find myself in the scenario where I have 2x of these HDD fail now, I need to learn from this episode.
Firstly, have I got complacent with the kit sitting in the corner of the room working away, 24/7. Is five years, old for this device? Should I be more proactive in thinking that HDD’s will fail eventually, therefore proactively replace them. Is there a best practice this community could suggest?
Am I right in thinking that Western Digital RED drives are still a good choice for NAS storage?
(You can imagine my faith in them has taken a hit with this recent news. I know there is a compatibility list that Netgear provide, but I normally prefer WD HDD’s as they have been reliable for me in the past.). I appreciate this is a subjective matter, but is there a clear market leader for this type of storage?
Have I caused this failure/downgrading of these HDD by doing anything in particular. Other than file storage, the only other thing I do with the ReadyNAS is watch TV programmes I have downloaded - has this caused the issue?
Could you advise me what I need to look at in the logs that gives me an indication of a HDD health, so I know what to look for in the future. Previously I have used only the RAIDAR user interface, and I have just looked at that again now, and there is a green light symbol against all six HDD units still . Quite a poor show really if you advise me that 2x have now failed.
The ReadyNAS unit is still turned on – can I safely turn it off without causing anymore data damage, or is it best to leave it running. I will be able to purchase 2x new HDD in a couple of weeks time is my hope.
Finally what RAID configuration would you choose in my shoes – reading up last time , I found it a minefield, and eventually opted for RAID 10. Having 18TB of hard drive capacity currently, but in reality only 9TB of usable space. If I went down the RAID 6 route, I’d get more storage capability for example (15/3 TB split I believe)
I hope you will appreciate my genuine questions in this area.
The unit was purchased new in August 2014, so it is not 5 yrs old yet. I also remember purchasing all the Red Western Digital drives at the same time too. So I now find myself in the scenario where I have 2x of these HDD fail now, I need to learn from this episode.
Firstly, have I got complacent with the kit sitting in the corner of the room working away, 24/7. Is five years, old for this device? Should I be more proactive in thinking that HDD’s will fail eventually, therefore proactively replace them. Is there a best practice this community could suggest?
Am I right in thinking that Western Digital RED drives are still a good choice for NAS storage?
NAS-purposed drives (like WDC Red, Seagate Ironwolf) and Enterprise Class drives (like WDC Gold, Seagate Ironwolf Pro) are good choices. I'm using about 12 WDC Reds in my various NAS, and have found them to be quite reliable. Several (WD30EFRX) are over 5 years old - purchased shortly after WDC introduced them in 2012.
Overall, about 22% of disk drives have failed by the time they've seen 4 years of use (or stated positively, about 78% are still working after 4 years). I have several WD green drives that have been running almost 10 years. Personally I don't replace disks until there is some sign of failure. If you don't have email alerts enabled on the NAS, that is one thing you should fix - as that will give you some warning signs. There is also a maintenance schedule you can set - running disk tests and scrubs on a schedule. Both of these are good diagnostics of disk health. I run all four of these tests once every three months on my OS 6 NAS.
One complaint I have with OS 6 is that Netgear is using failure thresholds for disks that are (in my opinion) too high. That's one aspect of the green indicators you are seeing in RAIDar. So I'd also suggest downloading the log zip file from time to time, and taking a look at disk_info.log. Reallocated sectors and pending sectors are important indicators, so if you see rising counts on those you should consider disk replacement. ATA errors and command timeouts are also concerning, but not quite as bad as the sector errors.
I also recommend using a UPS with a USP interface to protect your NAS from unexpected power cuts.
However, the most important lesson is that RAID isn't enough to keep your data safe. Disks and ReadyNAS can fail - and there are other ways that you can lose data. Power surges can damage equipment, and there is of course the possibility of theft, fire, etc. RAID is useful, but you want to protect your data you need to have at least one copy on another device. Personally I have at least 3 copies of everything I care about, with one copy off-site (cloud backup in my case).
Have I caused this failure/downgrading of these HDD by doing anything in particular. Other than file storage, the only other thing I do with the ReadyNAS is watch TV programmes I have downloaded - has this caused the issue?
Power issues (brownouts for example) could damage the drives I guess - as would high temperatures. You're describing normal use, and that shouldn't accelerate disk failure.
Finally what RAID configuration would you choose in my shoes – reading up last time , I found it a minefield, and eventually opted for RAID 10. Having 18TB of hard drive capacity currently, but in reality only 9TB of usable space. If I went down the RAID 6 route, I’d get more storage capability for example (15/3 TB split I believe)
RAID-10 can give better performance than RAID-5 or RAID-6, but in practice you are limited by your ethernet connection speed anyway. So I'd switch to something else if I were in your shoes.
Personally I use XRAID/RAID-5. That gives me reasonable performance and the ability easily expand the array later on. I depend on backups for data safety. RAID-6 is reasonable too, though expansion is more expensive (you need to upgrade at least 4 drives to expand).
