Reply
marcusp
Tutor

Problem with BTRFS filesystem

Greetings all,

 

recently I noticed my rsync backups were having errors, so I've investigated and here is what I'v found until now:

 

from status.log it seems the problem occured just after the defragmentation on 12-09-19:

[19/12/07 23:35:44 CET] notice:backup:LOGMSG_SUCCESS_BACKUP Successfully completed backup job Backup Readynas-Pro-6.
[19/12/08 01:00:09 CET] notice:system:LOGMSG_SYSTEM_HALT The system is shutting down.
[19/12/08 08:01:47 CET] info:system:LOGMSG_START_READYNASD ReadyNASOS background service started.
[19/12/09 00:03:00 CET] err:backup:LOGMSG_ERROR_BACKUP Error backup job Backup Readynas-Pro-6: Failure during copy.
[19/12/09 01:00:09 CET] notice:system:LOGMSG_SYSTEM_HALT The system is shutting down.
[19/12/09 01:00:09 CET] warning:system:LOGMSG_SENT_ALERT_MESG_FAILED Alert message failed to send.
[19/12/09 07:01:45 CET] info:system:LOGMSG_START_READYNASD ReadyNASOS background service started.
[19/12/09 08:00:01 CET] notice:volume:LOGMSG_DEFRAGSTART_VOLUME Defragmentation started for volume data.
[19/12/09 08:00:01 CET] notice:volume:LOGMSG_DEFRAGSTART_VOLUME Defragmentation started for volume data.
[19/12/09 08:02:50 CET] notice:volume:LOGMSG_DEFRAGEND_VOLUME Defragmentation complete for volume data.
[19/12/10 01:00:02 CET] notice:system:LOGMSG_SCHEDULED_POWEROFF_DELAY Scheduled power off was delayed.
[19/12/10 01:00:28 CET] err:backup:LOGMSG_ERROR_BACKUP Error backup job Backup Readynas-Pro-6: Failure during copy.
[19/12/10 01:03:10 CET] notice:system:LOGMSG_SYSTEM_HALT The system is shutting down.
[19/12/10 07:01:42 CET] info:system:LOGMSG_START_READYNASD ReadyNASOS background service started.

on 12-06-19 I executed a "balance" on the volume without any problem, here is the volume.log

data        balance    2019-12-06 13:05:36  2019-12-06 13:06:37  completed  Done, had to relocate 21 out of 5634 chunks 

 

but now BTRFS filesystem seems to be corrupted, kernel.log:

Dec 22 23:38:51 Readynas-Pro-6 kernel: BTRFS critical (device md125): corrupt leaf: root=1 block=5563987623936 slot=63 ino=4814786, invalid dir item type for XATTR key, have 1 expect 8
Dec 22 23:38:51 Readynas-Pro-6 kernel: BTRFS error (device md125): error loading props for ino 4814775 (root 275): -5

 

systemd-journal.log

Dec 25 00:10:00 Readynas-Pro-6 kernel: BTRFS critical (device md125): corrupt leaf: root=1 block=5563987623936 slot=63 ino=4814786, invalid dir item type for XATTR key, have 1 expect 8

 

kernel.log

Dec 22 23:38:51 Readynas-Pro-6 kernel: BTRFS critical (device md125): corrupt leaf: root=1 block=5563987623936 slot=63 ino=4814786, invalid dir item type for XATTR key, have 1 expect 8
Dec 22 23:38:51 Readynas-Pro-6 kernel: BTRFS error (device md125): error loading props for ino 4814775 (root 275): -5

 

So now each rsync backup is terminated with errors ! That's not 100% critical for now, but I thought this filesystem was far more robust !

backup_001_copy.log

rsync: readlink_stat("/tmp/12264/Stockage-Marc/dummy.example1") failed: Input/output error (5)
rsync: readlink_stat("/tmp/12264/Stockage-Marc/dummy.example2") failed: Input/output error (5)
IO error encountered -- skipping file deletion

My ReadyNAS-Pro-6 have been working like a charm with OS6 for several years without any serious problem, it's on UPS and was not forced to shutdown at this time. It already happened I had to force a reset, but at worst it resynchronized and that's all.

Scanning the disks didn't solve the problem at all.

 

So if anybody have a suggestion (excepted a factory reset), I'd be glad to hear it.

 

Thanks in advance !

ReadyNAS Pro 6
Core 2 Duo E7600 @ 3GHz
2x4GB Patriot PSD28G800K (6-6-6-18)
HDD1: WDC WD2000FYYZ-01UL1B1 2TB
HDD2: HGST HUS726040ALE610 4TB
HDD3: HGST HUS724040ALA640 4TB
HDD4: HGST HUS726040ALE610 4TB
HDD5: HGST HUS724030ALE640 3TB
HDD6: WDC WD2003FYYS-02W0B1 2TB
Model: RNDP6000v2|ReadyNAS Pro 6 Chassis only
Message 1 of 4
StephenB
Guru

Re: Problem with BTRFS filesystem

It looks to me like the error happened quite a while after the balance you ran on the 9th of December.  Did you look for disk errors in the log around 22 December (when the corrupt file system problem first showed up)? 

 

Is the volume set to read-only?

Message 2 of 4
marcusp
Tutor

Re: Problem with BTRFS filesystem

Thanks for you answer Stephen. From what I remember the BTRFS errors are the source of the rsync backup errors that began on the 9th of december. I only published the errors of kernel.log from the 22th because I don't have the file from the 9th.

 

What kind of disk errors are you talking about ? SMART errors or BTRFS ?

I've searched in the log files for read only volume, but I haven't found anything relevant. Could you indicate me in which logfile I could find this info ?

ReadyNAS Pro 6
Core 2 Duo E7600 @ 3GHz
2x4GB Patriot PSD28G800K (6-6-6-18)
HDD1: WDC WD2000FYYZ-01UL1B1 2TB
HDD2: HGST HUS726040ALE610 4TB
HDD3: HGST HUS724040ALA640 4TB
HDD4: HGST HUS726040ALE610 4TB
HDD5: HGST HUS724030ALE640 3TB
HDD6: WDC WD2003FYYS-02W0B1 2TB
Message 3 of 4
StephenB
Guru

Re: Problem with BTRFS filesystem

SMART errors are in disk_info.log.  You'd look for disk and btrfs errors in system.log, kernel.log, and dmesg.log.

 

Are the errors happening in multiple shares, or just one?  I suspect you can find out by just trying to copy a file into each share, and if that succeeds then deleting it.

 

 

Message 4 of 4
Top Contributors
Discussion stats
  • 3 replies
  • 404 views
  • 0 kudos
  • 2 in conversation
Announcements