4x4TB in X-RAID, holds about 4TB in daily data and 1TB in daily snapshots. Used to work fine, when suddenly last Saturday it crashed. The shares disappeared. In a move to find a solution (and since I'm very familiar with Linux), I opened ssh access and I logged in with ssh. dmesg was reporting corruption on the roots of BTRFS on md127. It does not reported hardware problem with the hard disks thought. It seemed a software issue (corruption) with btrfs.
I googled it a lot and tried whatever I could find in order to solve it. After hours on working with btrfsck, I managed to be able to "see" the filesystem on the device, but now, after reboot it reports: "BTRFS error (device md127): qgroup generation mismatch, marked as inconsistent" and mounts md127 (with the snapshots) read only.
I tried to fix it with a couple of btrfsck commands without success. The data are there and I can read them. I can take them a backup (I already have the data on an USB disk).
The best solution is to find a way to fix the corruption on md127 as it is now.
Alternatively I was thinking to factory reset it and then restore configuration and data. But in this case I will need a way to backup (and then restore) the snapshots also, but as snapshots and not as data copies. Maybe something like "format a big enough external USB in btrfs and do this and that to take a full backup - with snapshots- on it, and after factory reset, restore from it. But I'm not sure how to do this.
Any ideas ?
Model: RN21400|ReadyNAS 214 Series 4- Bay (Diskless)
If you format one (or more) USB drives as btrfs, you should be able to do what you want with btrfs send and btrfs receive. Since each share is it's own BTRFS subvolume, you should be able to use multiple USB disks.
The only catch I see is on the restore - if the readynas database needs records of the snapshots, they might not show up in the web ui. Hopefully that's not the case. Maybe a netgear person (perhaps @mdgm-ntgr or @kohdee) can comment on that.
Thank you Stephen for the ultra-fast answer, it gives me a road to follow, I had seen the commands but I had not realized what they were doing. So, let me see if I got it ok:
1) Format a -let say -6TB USB disk in btrfs and mount it (let say to /media/USB_HDD_6)
2) I'll use something like
btrfs send /data/daily -c /data/daily/snapshot/c_2016_09_24__00_00_19 -c /data/daily/snapshot/c_2016_09_30__00_00_00 (and so on for all of my 30 snapshots) | btrfs receive /media/USB_HDD_6
3) Factory reset. Restore configuration. Maybe make my shares again from scratch (I have only two and only one has snaphots)
4) Restore form USB with something like:
btrfs send /media/USB_HDD_6/data/daily -c /media/USB_HDD_6/data/daily/snapshot/c_2016_09_24__00_00_19 (etc etc for all snapshots) | btrfs receive /data/daily
I haven't tested this either and doing something like this yourself via SSH you'd have to deal with it on your own or perhaps with some help from the community if it doesn't work.
Also, qgroup generation mismatch, marked as inconsistent essentially means your quota table has inconsistent data (btrfs qgroup show /data). String together the mount command and then try btrfs quota disable /path/to/mount (mount /dev/md127 /data; btrfs quota disable /data;) and see if it still throws your ReadyNAS into read-only. Typically, an inconsistent qgroup mismatch will not be the root cause to your file system being marked read-only, it is what comes after btrfs tries to scan your system to recuperate what was inconsistent back to consistency that will cause it to be marked read-only (from experience).
Thank you for the hints. Actually I have tried all of them with mixed success. I managed to mount the system read only and copy all of the data and some of the snaphots. When I tried to get all of the snaphosts the device consistently crashed (and it was not a low memory issue because I had created an extra 4GB swap file on an extrnal hard disk to avoid low memory problems that appeared since the beggining). No matter how many time I btrfsck the system there was no other issue than the qgroup one, which I failed to overcome . Indeed this was the reason why the filesystem was marked as inconsistent and was mounted read only.
Anyway, I could not aford folling around anymore, so I took backup whatever I could and I factory-reset it.
Now seeing this same issue after trying to move disks from a RN516 to a RN526X (see my two related posts).
L3 support has been more than USELESS (and not a soul based in the US, apparently). What is a synonym for "more than worthless", Siri?
The data is still there. I can tell.
Advice? The volumes are currently in the RN516. Last L2(?) tech said it was IMPOSSIBLE. Admin console and SMB unavailable, though Radar still sees the volume.
Tonkinite, we have L3s in different places around the world including our head office in San Jose in the US. A few of our US based L3s are among those that have worked on your recent cases.
Support is working with you to resolve the problem you're facing.
Yes, the current L3 in charge of the case seems quite helpful.
The previous ones have not been, like I was a gnat to push out of the way.
That is not how L3 support should be.
I feel confident in saying that If I was a mid-tier supervisor at Oracle trying to get a critical DB up, I would not have been treated with such disrespect.
