- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page
Problem with BTRFS filesystem
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Problem with BTRFS filesystem
Greetings all,
recently I noticed my rsync backups were having errors, so I've investigated and here is what I'v found until now:
from status.log it seems the problem occured just after the defragmentation on 12-09-19:
[19/12/07 23:35:44 CET] notice:backup:LOGMSG_SUCCESS_BACKUP Successfully completed backup job Backup Readynas-Pro-6. [19/12/08 01:00:09 CET] notice:system:LOGMSG_SYSTEM_HALT The system is shutting down. [19/12/08 08:01:47 CET] info:system:LOGMSG_START_READYNASD ReadyNASOS background service started. [19/12/09 00:03:00 CET] err:backup:LOGMSG_ERROR_BACKUP Error backup job Backup Readynas-Pro-6: Failure during copy. [19/12/09 01:00:09 CET] notice:system:LOGMSG_SYSTEM_HALT The system is shutting down. [19/12/09 01:00:09 CET] warning:system:LOGMSG_SENT_ALERT_MESG_FAILED Alert message failed to send. [19/12/09 07:01:45 CET] info:system:LOGMSG_START_READYNASD ReadyNASOS background service started. [19/12/09 08:00:01 CET] notice:volume:LOGMSG_DEFRAGSTART_VOLUME Defragmentation started for volume data. [19/12/09 08:00:01 CET] notice:volume:LOGMSG_DEFRAGSTART_VOLUME Defragmentation started for volume data. [19/12/09 08:02:50 CET] notice:volume:LOGMSG_DEFRAGEND_VOLUME Defragmentation complete for volume data. [19/12/10 01:00:02 CET] notice:system:LOGMSG_SCHEDULED_POWEROFF_DELAY Scheduled power off was delayed. [19/12/10 01:00:28 CET] err:backup:LOGMSG_ERROR_BACKUP Error backup job Backup Readynas-Pro-6: Failure during copy. [19/12/10 01:03:10 CET] notice:system:LOGMSG_SYSTEM_HALT The system is shutting down. [19/12/10 07:01:42 CET] info:system:LOGMSG_START_READYNASD ReadyNASOS background service started.
on 12-06-19 I executed a "balance" on the volume without any problem, here is the volume.log
data balance 2019-12-06 13:05:36 2019-12-06 13:06:37 completed Done, had to relocate 21 out of 5634 chunks
but now BTRFS filesystem seems to be corrupted, kernel.log:
Dec 22 23:38:51 Readynas-Pro-6 kernel: BTRFS critical (device md125): corrupt leaf: root=1 block=5563987623936 slot=63 ino=4814786, invalid dir item type for XATTR key, have 1 expect 8 Dec 22 23:38:51 Readynas-Pro-6 kernel: BTRFS error (device md125): error loading props for ino 4814775 (root 275): -5
systemd-journal.log
Dec 25 00:10:00 Readynas-Pro-6 kernel: BTRFS critical (device md125): corrupt leaf: root=1 block=5563987623936 slot=63 ino=4814786, invalid dir item type for XATTR key, have 1 expect 8
kernel.log
Dec 22 23:38:51 Readynas-Pro-6 kernel: BTRFS critical (device md125): corrupt leaf: root=1 block=5563987623936 slot=63 ino=4814786, invalid dir item type for XATTR key, have 1 expect 8 Dec 22 23:38:51 Readynas-Pro-6 kernel: BTRFS error (device md125): error loading props for ino 4814775 (root 275): -5
So now each rsync backup is terminated with errors ! That's not 100% critical for now, but I thought this filesystem was far more robust !
backup_001_copy.log
rsync: readlink_stat("/tmp/12264/Stockage-Marc/dummy.example1") failed: Input/output error (5) rsync: readlink_stat("/tmp/12264/Stockage-Marc/dummy.example2") failed: Input/output error (5) IO error encountered -- skipping file deletion
My ReadyNAS-Pro-6 have been working like a charm with OS6 for several years without any serious problem, it's on UPS and was not forced to shutdown at this time. It already happened I had to force a reset, but at worst it resynchronized and that's all.
Scanning the disks didn't solve the problem at all.
So if anybody have a suggestion (excepted a factory reset), I'd be glad to hear it.
Thanks in advance !
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Problem with BTRFS filesystem
It looks to me like the error happened quite a while after the balance you ran on the 9th of December. Did you look for disk errors in the log around 22 December (when the corrupt file system problem first showed up)?
Is the volume set to read-only?
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Problem with BTRFS filesystem
Thanks for you answer Stephen. From what I remember the BTRFS errors are the source of the rsync backup errors that began on the 9th of december. I only published the errors of kernel.log from the 22th because I don't have the file from the 9th.
What kind of disk errors are you talking about ? SMART errors or BTRFS ?
I've searched in the log files for read only volume, but I haven't found anything relevant. Could you indicate me in which logfile I could find this info ?
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Problem with BTRFS filesystem
SMART errors are in disk_info.log. You'd look for disk and btrfs errors in system.log, kernel.log, and dmesg.log.
Are the errors happening in multiple shares, or just one? I suspect you can find out by just trying to copy a file into each share, and if that succeeds then deleting it.