I had my system lock up while deleting some luns via the GUI interface. (I'm guessing its snapshots as well) where I had to power off to reboot.
The system booted fine but I figured I should check the disk and when I did I got many errors (with /data unmounted: btrfsck /dev/mapper/data-0). I listed only partially the btrfs erros below since there are about 500,000 lines of error codes. I do have the full log if needed.
I had this happen on another computer before and btrfsck --repair did not solve the problem. I will try but I think btrfsck does not handle this situation. On my other computer I was able to get rid of the erros by removing all snapshots on the machine and running btrfsck again.
So does anyway know a better way of trying to fix this witout factory booting or removing all my snapshots? If not, is it okay to manually remove the snapshots from the /data/._media/subvolume_name/.snapsshots directory (btrfs subvolume delete subvolume_path-name?
root@NAS:/# btrfsck /dev/mapper/data-0 Checking filesystem on /dev/mapper/data-0 UUID: d2d2b9e8-9acb-4fd9-8152-9c7c0bebc294 checking extents Extent back ref already exists for 7389433556992 parent 7252941275136 root 0 Extent back ref already exists for 7659068915712 parent 7252941275136 root 0 Extent back ref already exists for 7389433569280 parent 7252941275136 root 0 Extent back ref already exists for 7389433573376 parent 7252941275136 root 0 Extent back ref already exists for 7389433577472 parent 7252941275136 root 0 Extent back ref already exists for 7389433581568 parent 7252941275136 root 0 Extent back ref already exists for 7389433602048 parent 7252941275136 root 0 Extent back ref already exists for 7198478852096 parent 7252941275136 root 0 Extent back ref already exists for 7659068956672 parent 7252941275136 root 0 Extent back ref already exists for 633252085760 parent 7252991348736 root 0 Extent back ref already exists for 6646881452032 parent 0 root 7258 Extent back ref already exists for 7253045641216 parent 0 root 7258 Extent back ref already exists for 6646917271552 parent 0 root 7258 Extent back ref already exists for 6646881579008 parent 0 root 7258 Extent back ref already exists for 6646881853440 parent 0 root 7258 Extent back ref already exists for 6646917279744 parent 0 root 7258 Extent back ref already exists for 6646881603584 parent 0 root 7258 Extent back ref already exists for 6646917292032 parent 0 root 7258 Extent back ref already exists for 6646881628160 parent 0 root 7258 Extent back ref already exists for 6646917316608 parent 0 root 7258 ... thousands of lines like above ... ref mismatch on [33705984 4096] extent item 2, found 3 Incorrect global backref count on 33705984 found 2 wanted 3 backpointer mismatch on [33705984 4096] ref mismatch on [33722368 4096] extent item 2, found 3 Incorrect global backref count on 33722368 found 2 wanted 3 backpointer mismatch on [33722368 4096] ref mismatch on [33730560 4096] extent item 2, found 3 Incorrect global backref count on 33730560 found 2 wanted 3 backpointer mismatch on [33730560 4096] ref mismatch on [33861632 4096] extent item 2, found 3 Incorrect global backref count on 33861632 found 2 wanted 3 backpointer mismatch on [33861632 4096] ref mismatch on [33914880 4096] extent item 2, found 3 Incorrect global backref count on 33914880 found 2 wanted 3 backpointer mismatch on [33914880 4096] ref mismatch on [34250752 4096] extent item 2, found 3 Incorrect global backref count on 34250752 found 2 wanted 3 backpointer mismatch on [34250752 4096] ref mismatch on [34349056 4096] extent item 2, found 3 Incorrect global backref count on 34349056 found 2 wanted 3 backpointer mismatch on [34349056 4096] ref mismatch on [34353152 4096] extent item 2, found 3 Incorrect global backref count on 34353152 found 2 wanted 3 backpointer mismatch on [34353152 4096] ref mismatch on [34365440 4096] extent item 2, found 3 Incorrect global backref count on 34365440 found 2 wanted 3 backpointer mismatch on [34365440 4096] ref mismatch on [34369536 4096] extent item 2, found 3 Incorrect global backref count on 34369536 found 2 wanted 3 backpointer mismatch on [34369536 4096] ... hundreds more like above again ... Errors found in extent allocation tree or chunk allocation checking free space cache checking fs roots ... still has not completed the fs root pahase
I ran a btrfs disk check on the un-mounted /data directory in repair mode which took 6 hours to complete and once complete did not fix any of the errors. As I stated before I had this problem on another system with same result.
btrfsck --repair /dev/mapper/data-0
So, since that did not work I just removed all snapshots. I made sure I removed them in order of newest first so btrfs would not have to work so hard.
# - Edit /etc/fstab and enable /data mount point mount /data cd /data/._share
# (it looks like we are listing oldest first; we are but the for loop will process newest first) for f in `ls -tdS */.snapshot/?_*`; do btrfs subvolume delete $f; done
# - Wait till load is really low and all btrf_cleaner, etc tasks complete before checking if it worked (took maybe an hour)
# - Do a filesystem check again to see if errors have been fixed cd / umount /data
# - Check disk again btrfsck /dev/mapper/data-0
My disk check came back clean this time. I just canceled it after seeing extents were good now.
root@NAS:/# btrfsck /dev/mapper/data-0 Checking filesystem on /dev/mapper/data-0 UUID: d2d2b9e8-9acb-4fd9-8152-9c7c0bebc294 checking extents checking free space cache
So, I just rebooted the NAS
rn_shutdown -r
I hope the btrfs guys take care of this in the check disk utility since this has happened to me more than once in the last few weeks on 2 seperate linux systems. I know they know about it since I seen another user have same problem which he could only fix by re-formatting (did not look like he tried removing snapshots first like I did).
I actually wonder how many users whose systems crahed have these errors and don't even know they have them since the btrfs filesystem does not get checked on boot?
