NETGEAR is aware of a growing number of phone and online scams. To learn how to stay safe click here.

Forum Discussion

marcusp's avatar
marcusp
Tutor
Jan 18, 2020

Problem with BTRFS filesystem

Greetings all,

 

recently I noticed my rsync backups were having errors, so I've investigated and here is what I'v found until now:

 

from status.log it seems the problem occured just after the defragmentation on 12-09-19:

[19/12/07 23:35:44 CET] notice:backup:LOGMSG_SUCCESS_BACKUP Successfully completed backup job Backup Readynas-Pro-6.
[19/12/08 01:00:09 CET] notice:system:LOGMSG_SYSTEM_HALT The system is shutting down.
[19/12/08 08:01:47 CET] info:system:LOGMSG_START_READYNASD ReadyNASOS background service started.
[19/12/09 00:03:00 CET] err:backup:LOGMSG_ERROR_BACKUP Error backup job Backup Readynas-Pro-6: Failure during copy.
[19/12/09 01:00:09 CET] notice:system:LOGMSG_SYSTEM_HALT The system is shutting down.
[19/12/09 01:00:09 CET] warning:system:LOGMSG_SENT_ALERT_MESG_FAILED Alert message failed to send.
[19/12/09 07:01:45 CET] info:system:LOGMSG_START_READYNASD ReadyNASOS background service started.
[19/12/09 08:00:01 CET] notice:volume:LOGMSG_DEFRAGSTART_VOLUME Defragmentation started for volume data.
[19/12/09 08:00:01 CET] notice:volume:LOGMSG_DEFRAGSTART_VOLUME Defragmentation started for volume data.
[19/12/09 08:02:50 CET] notice:volume:LOGMSG_DEFRAGEND_VOLUME Defragmentation complete for volume data.
[19/12/10 01:00:02 CET] notice:system:LOGMSG_SCHEDULED_POWEROFF_DELAY Scheduled power off was delayed.
[19/12/10 01:00:28 CET] err:backup:LOGMSG_ERROR_BACKUP Error backup job Backup Readynas-Pro-6: Failure during copy.
[19/12/10 01:03:10 CET] notice:system:LOGMSG_SYSTEM_HALT The system is shutting down.
[19/12/10 07:01:42 CET] info:system:LOGMSG_START_READYNASD ReadyNASOS background service started.

on 12-06-19 I executed a "balance" on the volume without any problem, here is the volume.log

data        balance    2019-12-06 13:05:36  2019-12-06 13:06:37  completed  Done, had to relocate 21 out of 5634 chunks 

 

but now BTRFS filesystem seems to be corrupted, kernel.log:

Dec 22 23:38:51 Readynas-Pro-6 kernel: BTRFS critical (device md125): corrupt leaf: root=1 block=5563987623936 slot=63 ino=4814786, invalid dir item type for XATTR key, have 1 expect 8
Dec 22 23:38:51 Readynas-Pro-6 kernel: BTRFS error (device md125): error loading props for ino 4814775 (root 275): -5

 

systemd-journal.log

Dec 25 00:10:00 Readynas-Pro-6 kernel: BTRFS critical (device md125): corrupt leaf: root=1 block=5563987623936 slot=63 ino=4814786, invalid dir item type for XATTR key, have 1 expect 8

 

kernel.log

Dec 22 23:38:51 Readynas-Pro-6 kernel: BTRFS critical (device md125): corrupt leaf: root=1 block=5563987623936 slot=63 ino=4814786, invalid dir item type for XATTR key, have 1 expect 8
Dec 22 23:38:51 Readynas-Pro-6 kernel: BTRFS error (device md125): error loading props for ino 4814775 (root 275): -5

 

So now each rsync backup is terminated with errors ! That's not 100% critical for now, but I thought this filesystem was far more robust !

backup_001_copy.log

rsync: readlink_stat("/tmp/12264/Stockage-Marc/dummy.example1") failed: Input/output error (5)
rsync: readlink_stat("/tmp/12264/Stockage-Marc/dummy.example2") failed: Input/output error (5)
IO error encountered -- skipping file deletion

My ReadyNAS-Pro-6 have been working like a charm with OS6 for several years without any serious problem, it's on UPS and was not forced to shutdown at this time. It already happened I had to force a reset, but at worst it resynchronized and that's all.

Scanning the disks didn't solve the problem at all.

 

So if anybody have a suggestion (excepted a factory reset), I'd be glad to hear it.

 

Thanks in advance !

3 Replies

Replies have been turned off for this discussion
  • StephenB's avatar
    StephenB
    Guru - Experienced User

    It looks to me like the error happened quite a while after the balance you ran on the 9th of December.  Did you look for disk errors in the log around 22 December (when the corrupt file system problem first showed up)? 

     

    Is the volume set to read-only?

    • marcusp's avatar
      marcusp
      Tutor

      Thanks for you answer Stephen. From what I remember the BTRFS errors are the source of the rsync backup errors that began on the 9th of december. I only published the errors of kernel.log from the 22th because I don't have the file from the 9th.

       

      What kind of disk errors are you talking about ? SMART errors or BTRFS ?

      I've searched in the log files for read only volume, but I haven't found anything relevant. Could you indicate me in which logfile I could find this info ?

      • StephenB's avatar
        StephenB
        Guru - Experienced User

        SMART errors are in disk_info.log.  You'd look for disk and btrfs errors in system.log, kernel.log, and dmesg.log.

         

        Are the errors happening in multiple shares, or just one?  I suspect you can find out by just trying to copy a file into each share, and if that succeeds then deleting it.

         

         

NETGEAR Academy

Boost your skills with the Netgear Academy - Get trained, certified and stay ahead with the latest Netgear technology! 

Join Us!

ProSupport for Business

Comprehensive support plans for maximum network uptime and business peace of mind.

 

Learn More