NETGEAR is aware of a growing number of phone and online scams. To learn how to stay safe click here.
Forum Discussion
Solved
btrfs corruption AGAIN
So I noticed today after trying to write a file to the nas that it was read only (Access denied) coudl not even edit files from the admin page.
Sure enough i went in to SSH to peek at the dmes...
- So a quick update to this. I know it's been a year since my last post, but since this shows up in google now I may as well put a closing comment.
Since my last post, Volume failures have continued and i've pretty much just come to accept this to be a quirk of bad implementation on Netgear's part and that these devices are simply unreliable due to the well known failings of BTRFS as a file system in general everywhere else in the linux community. I really wish the devices used XFS, but alas; we're stuck with the poor design choices Netgear gave us - short of hacking and flashing them with something else.
In subsequent failures, i've not even bothered SSH'ing in to the devices, and used only factory provided tools (Exluding SSH which is still factory provided too btw) and used instead regular backup options as well as ReadyDR. - the file system still crashes. even at 90% which is a huge waste of otherwise perfectly usable free space.
Regular volume house keeping has always been in place, weekly scrubs, monthly defrags, etc.
Currently in the process of switching the disks over from Seagate Barracudas to WD Reds with X2 the capacity to hopefully mitigate the volume almost full self destruct issue that shouldn't exist in the first place (On any sane file system)
But if even this fails to help and i end up once again with a busted volume later down the line, my advise to the general populace at the moment would be "Don't use these devices for anything mission critical where you expect the free space to become limited. If you do, don't trust the RAID configurations and stick to single disk shares, as bugs in BTRFS regarding RAID will likely leave you with a busted volume 6 months down the line."
Until Netgear either fix the bugs in BTRFS (Unlikly) Or switch to a more mature and reliable Filesystem (Like XFS) these NAS drives are volatile at best.
Quick update:
I rebooted the nas after veryfying the array was sat idle and no scans were going on; sure enough the array has come back up, however the filesystem is totally dead.
root@INT-NAS-1:~# mount /dev/md127 /data
mount: wrong fs type, bad option, bad superblock on /dev/md127,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so.
root@INT-NAS-1:~# btrfsck --repair /dev/md127
enabling repair mode
bytenr mismatch, want=18105397379072, have=16016835313664
ERROR: cannot read chunk root
ERROR: cannot open file system
root@INT-NAS-1:~#
dmesg:
[ 154.994060] BTRFS info (device md127): has skinny extents [ 154.995265] BTRFS critical (device md127): unable to find logical 513211334656 len 4096 [ 154.995278] BTRFS critical (device md127): unable to find logical 513211334656 len 4096 [ 154.995321] BTRFS critical (device md127): unable to find logical 513211334656 len 4096 [ 154.995330] BTRFS critical (device md127): unable to find logical 513211334656 len 4096 [ 154.995358] BTRFS critical (device md127): unable to find logical 513211334656 len 4096 [ 154.995367] BTRFS critical (device md127): unable to find logical 513211334656 len 4096 [ 154.995384] BTRFS error (device md127): failed to read chunk root [ 155.057373] BTRFS error (device md127): open_ctree failed
Please send us your logs (see the Sending Logs link in my sig)
Logs have been sent.
I can see that you've been using SSH, so I can't rule out that contributing to the problem.
Also care needs to be taken when running destructive commands, like a btrfsck --repair. If it's a problem that a repair won't fix, running that command may bake in the problem making any subsequent data recovery attempt much less likely to succeed.It's very odd that you've run into this problem so many times. Have you tried running the memory (RAM) test boot menu option?
Thanks for the reply; I've not tried a RAM test yet no, but im curious what exactly if anything i've done would cause btrfs to completely die on its own accord, I generally only go in there to do non-invasive changes to services or run an iperf test between them, or tweak a share - or set up SSH keys between the 2 units. I don't go in to tweak anything related to btrfs or run any commands against it, these drives spend most of their time running frontview backups jobs on an hourly basis to capture data for backups - Data loss isn't a huge concern, since they are backed up and actually themself a backup of another source of data, it's just annoying that this seems to happen roughly every few months and this time the drive wasn't even at more than 60% capacity when i looked.
Edit: i also want to point out that the file system modules in the kernel seem to have issues. One of the units i rotate a USB HDD for offsite backups on a nightly bases (swap, take one home) and i found if i use EXT4 as the file system the NAS Hangs with an mb_cache_entry_g error. (see attached photo) after weeks of head scratching, this issue stopped when i formatted the drive to btrfs, so i'm more likly to suspect the kernel modules are to blame for random filesystem death than my benign use of ssh.
RAM test did not produce any errors, I am considering a full factory rest however it would be nice to figure out why this keeps happening..
Do you sometimes experience power issues that result in an unclean shutdown of the NAS?
That could result in an out-of-sync raid array - normally I'd expect a different symptom (inactive volume), but it might be possible that lost writes could result in corrupted file system.
Now that you mention it, there had been a couple of occasions where the NAS locked up, in this instance unrelated to the above issue with the other unit and a USB drive, but in those situtations a forced power off is the only option (Power button unresponsive, no network access etc.) .. though there were no immediate issues upon reboot. besides that no, the unit is connected to a UPS supply beside all the company servers.
NETGEAR Academy
Boost your skills with the Netgear Academy - Get trained, certified and stay ahead with the latest Netgear technology!
Join Us!
