I am seeing a very strange behaviour that results in corrupted files. My NAS (ReadyNAS 214) has been stable for years, until recently I found a single file had an error revealed via the logs following a scrub. I recoved that from a backup.
Over the past two days, I have noticed various newly written files appear fine, but after 'a bit of time', they become corrupted. Yes, weird I know. This has happened with edited Excel (xlsx) and non-edited .jpg and PDF files. Not all files going to the NAS are affected.
To isolate the issue:
I scanned my PC in case of a virus. Nothing found.
Created a checksum file, then copied the files to the NAS. Reread the checksums and all appears good.
A variable time later, re-check checksums. Results bad.
The time to going bad is immediate if I re-copy a file within the NAS folder, i.e. CTRL-C / CTRL-V, then check the checksums, and both the original and copy become corrupted.
I initially noticed this issue with photos as previews no longer showed, but repeated with xls and pdf.
I shutdown my PC in case it was the cause in some way, e.g. undetected virus.
I then tried from my wifes PC and was able to recreate the corrupting of files, almost at will.
I'm now convinced it's a NAS problem, rather than a PC problem, but maybe I am wrong!
So, I dived in to the logs and found quite a few lines like the following line in kernel.log:
No disk errors in the logs, or reported in the dashboard logs page.
Have you run the disk test from the volume settings wheel lately?
One option is to do a factory reset (or destroy/recreate the volume), and see if the sync succeeds. Are you using X-RAID? Or FlexRAID with multiple volumes.
Thanks @StephenB , yes this NAS has been expanded at least twice, although during the last upgrade I did upgrade 1 x 2TB and 1 x 3TB both to 8TB, so expansion probably happened on each upgrade (even though I saw it as one).
Right now, I have 7TB free of 14.54TB reported on the dashboard.
Then smartctl -x for each drive, all arror entries report zero, and smart tests for each drive report passed.
I am ready to rebuild/restore this NAS, but I worry that I have not yet identified the root cause, and it could therefore be some failure that I've not detected, but one that will repeat going forwards. Of course, it could 'just' be a software issue, but that's very strange too.
Of course, any more ideas to poke at would be most welcome!
For testing, I have removed all the disks, and inserted some old 3TB drives.
I did a factory default, copied some files, and so far no corruptions observed. It is early days, as sometimes the corruptions took "a while" to be observable, but so far it appears reliable.
To me, this tends to confirm either the system software had become unstable for some reason, or I have a dodgy drive that is not showing any errors through SMART. I will run the manufacturer tests on each drive in the coming days.
Copying and duplicating files at will currently, and no errors are appearing, no files appear corrupt (checksum checking is passing).
The volume is resyncing, it will take another 18h according to the dashboard.
To me, this tends to confirm either the system software had become unstable for some reason, or I have a dodgy drive that is not showing any errors through SMART. I will run the manufacturer tests on each drive in the coming days.
You can't easily rule out memory, so definitely keep that possibility in mind.
I have been having the same issues, RN204 corrupting excel and word docs, you guys seem way more techie than me, does anyone have a fix, or a recovery option? everything i have tried has not repaired the files.
You should store your files on an additional device for now, i.e. your PC local drive.
I found existing files already on my NAS in general did not get corrupted, only files edited or newly added. I did find copying a file within the same directory on the NAS also corrupted both the source and copy, so avoid that action.
Once a file is corrupted, you will need to go back to a previous good copy from another device. The corrupted files appear unrecoverable. Luckily I did have a backup, and I noticed this problem within a day or two, so could recreate what I needed.
Which version of the OS are you running, i.e. have you recently upgraded?
Have you changed anything else recently?
I decided to rebuild my NAS from factory defaults, and now I'm restoring files - around 7.5TB, so it will take a few days. My NAS seems stable since clearing it, not ideal, and for sure there may still be an underlying issue that I've not seen yet on the rebuild.
While I have no solution, I have an idea of where to look for the problem. Excel (and all Microsoft Office products) create a temporary file in the directory (and thus on the device) from which they were opened. Further, when saving a modified file, they concatenate (not combine) changes with the original, saving an original plus changes instead of a modifed original. So if something happens to that temporary file (which is set as invisible and starts with a tilde (~), at least on a Windows machine, which may or may not be pertinent), the end result can be a corrupted file. Or. if the last part of the saved file, which has the changes, is not properly saved, the entire document can also be corrupted.
This would not seem to explain the issue with "newly created" Excel files, but if you save as you go (manually or automatically), it could still have the original plus changes format.
This experiment might help get to the bottom of it: Copy a file that was created on a local drive to the NAS. Open and make a minor modification to the file on the NAS and save. If you then find the modified file to be corrupt, do a binary compare of the original on the local device to the corrupt one on the NAS.
As a not-so-great work-around, I believe using "save as" always creates a new original, not an original plus changes. (I know it used to, but I've not checked recently).
While I have no solution, I have an idea of where to look for the problem. Excel (and all Microsoft Office products) create a temporary file in the directory (and thus on the device) from which they were opened. Further, when saving a modified file, they concatenate (not combine) changes with the original, saving an original plus changes instead of a modifed original. So if something happens to that temporary file (which is set as invisible and starts with a tilde (~), at least on a Windows machine, which may or may not be pertinent), the end result can be a corrupted file. Or. if the last part of the saved file, which has the changes, is not properly saved, the entire document can also be corrupted.
If I remember correctly, there have been similar symptoms with some Macs related to the use of "apple fruit"
OK, an update after several months without issues.
Following on from installing temp disks to test the stability of my RN214, it had zero apparent errors during that time.
I then re-inserted all the original disks, powered up, and ran some simple copy tests. Immediate file corruptions were again observed. Great, the problem is still reproducible 😉!
I had to wipe the system anyway due to several cloud accounts that couldn't be removed without a full reset/re-installation.
After this complete reset, I did a few tests and all appeared to be well. I obviously had to restore all my data from backups, then I ran rsync with file check to alternative backups, and all was still well.
I have had stability for 3 months, but I still have no idea what on earth happened to create the file corruptions in the first place.
The corrupted (xls/jpg) files I copied back to my PC for analysis were completely filled with zeroes. Fortunately it was only a handful of files, which I recovered all bar one picture.
As this issue was so weird/unexplained, and horrific, due to data corruption, I wanted to share a further update: no further issues experienced, so I think we can safely say this was an odd incident indeed.
The RN214 continues to be used every day, the exact same drives are all good, and no errors to report. Phew 😄!
