Last night the scheduled bi-weekly online file system check reported:
"The on-line filesystem consistency check detected problems with your data volume. Please reboot the system with the volume scan checkbox enabled to do a full filesystem consistency check and repair."
I've had this error occasionally over the 4+ years I've been using by ReadyNAS NVX (currently running 4.2.22, 3 x WD20EFRX, 1 x ST31000528AS drives) and I've done the reboot with file system check and not had any major problems (minor errors detected and corrected). This time though, I have a major problem. The /dev/c/c can't be mounted or fsck'd.
the dmesg output reports: http://pastebin.com/5Rb3qcDM
lvs output: LV VG Attr LSize Origin Snap% Move Log Copy% c c -wi-a- 4.53T
pvs: PV VG Fmt Attr PSize PFree /dev/md2 c lvm2 a- 2.72T 0 /dev/md3 c lvm2 a- 1.82T 10.00G
mount /dev/c/c /c output: mount: wrong fs type, bad option, bad superblock on /dev/c/c, missing codepage or helper program, or other error In some cases useful info is found in syslog - try dmesg | tail or so
fsck.ext4 /dev/c/c e2fsck 1.41.14 (22-Dec-2010) fsck.ext4: Group descriptors look bad... trying backup blocks... /dev/c/c: recovering journal fsck.ext4: unable to set superblock flags on /dev/c/c
I did the mke2fs -n /dev/c/c -b 4096, noted the superblock backup locations and tried the fsck again... I get the following: http://pastebin.com/sx7VEqDy
What the heck happened? What should I do next? For all my preaching to others about backups and restores, I'm in a situation where I *stupidly* do not have a backup of some of the critical data on this volume. I can't believe I could be so stupid, but here I am. Can anyone help me?
I've also created a snapshot of the device /dev/c/c and am doing some further testing on that snapshot of the original device. Uusing debugfs I am able to see some of the file names on the snapshot device. Any ideas would very much be appreciated!
I have successfully used third party tools to get data back from failed arrays, but I don't know about the forum's attitude to such advice. Oh what the heck...
I used R-TT R-Studio (http://www.data-recovery-software.net/ -- I know the URL looks rather dodgy...:p) to save a failed array on a QNAP (two failed devices in a four-disk RAID5, one disk completely broken, WG Green drives killed by head parking) in Windows. I got everything back from a 4.5 TB array... I cannot recommend it enough. I tried everything up until that, testdisk, forcing the array to mount in a Linux system, etc etc. Several days of work. Then I just attached three of the disks to USB adapters (the fourth disk was simulated by the software), input stripe parameters and off we went... in Windows, too. It read EXT right off the broken disks. I have never seen anything like it. A third disk developed problems during the recovery procedure but everything was saved...
The disks belonged to a friend, he too lacked a backup, I own his spleen now. 😉 Spent a week on everything in total.
I have successfully used third party tools to get data back from failed arrays, but I don't know about the forum's attitude to such advice. Oh what the heck...
Forum attitude: Data loss is bad; advice on data recovery is welcome.
Spamming is of course not welcome, but you are not doing that.
Thanks for the info, folks. Update: the md device checks are still running and will take a long time to complete. In the meantime, debugfs is able to read the snapshot I made:
lvcreate -s -L 5G -n /dev/c/c_snap /dev/c/c (from memory - may be off a bit) debugfs -b 4096 -s 32768 -w /dev/c/c_snap (the blocksize and superblock location were reported using dumpe2fs output)
with debugfs I can ls, cd, dump and 'rdump'. rdump recursively dumps directories to a destination. I've connected a USB drive and am currently 'rdump'ing some data. I'll check it out and see if it appears to be intact.
I'm considering rebuilding this device after this experience to have more confidence in the system's stability. I might also consider replacing that one 1TB drive with the same 2TB WD RED to have everything matching and, I assume, have only one pv extent used for the lv volume. Perhaps this would make it a bit more simple to operate?
Does anyone have any suggestions about changes to make? Any additional information about the unit can certainly be provided!
