NETGEAR is aware of a growing number of phone and online scams. To learn how to stay safe click here.
Forum Discussion
Rsync error read only filesystem(30)
My rsync backups have been running flawlessly for years being pulled from a 316 to a Ultra 6+. Both units are running 6.7.0. For the last 2 days, backups are failing, all with the same error, - ...
Label: '33ea9931:root' uuid: dff4e5dd-d198-401a-b8bf-09915591d8cd
Total devices 1 FS bytes used 544.79MiB
devid 1 size 4.00GiB used 1.63GiB path /dev/md0
Label: '33ea9931:data' uuid: 5a9c472f-3757-4e79-b6b8-90792440712b
Total devices 1 FS bytes used 6.75TiB
devid 1 size 9.07TiB used 6.76TiB path /dev/md127
=== filesystem /data ===
Data, single: total=6.76TiB, used=6.75TiB
System, DUP: total=32.00MiB, used=768.00KiB
Metadata, DUP: total=2.00GiB, used=1.00GiB
GlobalReserve, single: total=352.00MiB, used=7.00MiB
There are many errors in systemjournal about failing to delete snapshots and this:
Apr 10 21:01:11 backupnas kernel: BTRFS: error (device md127) in create_pending_snapshot:1545: errno=-5 IO failure
Apr 10 21:01:11 backupnas kernel: BTRFS info (device md127): forced readonly
Apr 10 21:01:11 backupnas kernel: BTRFS warning (device md127): Skipping commit of aborted transaction.
Apr 10 21:01:11 backupnas kernel: BTRFS: error (device md127) in cleanup_transaction:1843: errno=-5 IO failure
Apr 10 21:01:11 backupnas kernel: ------------[ cut here ]------------
and this:
Apr 10 21:01:11 backupnas replisync[3057]: daemon: _splice error: Broken pipe
Apr 10 21:01:11 backupnas replisync[2863]: ERROR: creating snapshot KIDNASDR/.snapshots/618/snapshot -> snapshot failed: Input/output error
Apr 10 21:01:11 backupnas replisync[3057]: daemon: _btrfs_receive error: Broken pipe
Apr 10 21:01:11 backupnas replisync[2863]: ERROR: truncate Private practice/Copies of lists/2017/20170323 selvey.pdf failed: Read-only file system
Apr 10 21:01:11 backupnas replisync[3056]: daemon: _splice error: Broken pipe
Apr 10 21:01:11 backupnas replisync[2863]: ERROR: creating snapshot CSDNASDR/.snapshots/1075/snapshot -> snapshot failed: Input/output error
Apr 10 21:01:11 backupnas replisync[3055]: daemon: _splice error: Broken pipe
Apr 10 21:01:11 backupnas replisync[2863]: daemon: _splice error: Broken pipe
Apr 10 21:01:11 backupnas replisync[3055]: daemon: _btrfs_receive error: Broken pipe
Apr 10 21:01:11 backupnas replisync[2863]: ERROR: creating snapshot EADNASDR/.snapshots/1074/snapshot -> snapshot failed: Input/output error
Apr 10 21:01:11 backupnas replisync[3056]: daemon: _btrfs_receive error: Broken pipe
Apr 10 21:01:11 backupnas replisync[2863]: daemon: _splice error: Broken pipe
Apr 10 21:01:11 backupnas replisync[3058]: daemon: _splice error: Broken pipe
Apr 10 21:01:11 backupnas replisync[2863]: daemon: _splice error: Broken pipe
Apr 10 21:01:11 backupnas replisync[3058]: daemon: _btrfs_receive error: Broken pipe
Apr 10 21:01:11 backupnas replisync[2863]: daemon: _splice error: Broken pipe
Apr 10 21:15:51 backupnas readynasd[3078]: ERROR: failed to lookup path for root 7734: Input/output error
Apr 10 21:15:51 backupnas kernel: BTRFS error (device md127): parent transid verify failed on 120651776 wanted 13617 found 13384
Apr 10 21:15:51 backupnas kernel: BTRFS error (device md127): parent transid verify failed on 120651776 wanted 13617 found 13384
Apr 10 21:01:11 backupnas replisync[2863]: ERROR: failed to make subvolume read only: Read-only file system
and then also
Apr 10 21:09:23 backupnas fvbackup-q[3644]: cmd=/frontview/bin/fvbackup -e 2
Apr 10 21:09:29 backupnas readynasd[3078]: Error backup job EADNAS: failure during copy.
I don't understand what these mean, there are numerous similar error messages regarding the readydr jobs and the rsync jobs that are failing.
The backup job error logs all say this:
Failed to run rsync, return rc 5888 (23)
'rsync' command returns rc=5888
vandermerwe wrote:Label: '33ea9931:data' uuid: 5a9c472f-3757-4e79-b6b8-90792440712b
Total devices 1 FS bytes used 6.75TiB
devid 1 size 9.07TiB used 6.76TiB path /dev/md127=== filesystem /data ===
Data, single: total=6.76TiB, used=6.75TiB
System, DUP: total=32.00MiB, used=768.00KiB
Metadata, DUP: total=2.00GiB, used=1.00GiB
GlobalReserve, single: total=352.00MiB, used=7.00MiB
This is OK.
vandermerwe wrote:Apr 10 21:01:11 backupnas kernel: BTRFS: error (device md127) in create_pending_snapshot:1545: errno=-5 IO failure
Apr 10 21:01:11 backupnas kernel: BTRFS info (device md127): forced readonly
Apr 10 21:01:11 backupnas kernel: BTRFS warning (device md127): Skipping commit of aborted transaction.
Apr 10 21:01:11 backupnas kernel: BTRFS: error (device md127) in cleanup_transaction:1843: errno=-5 IO failure
Apr 10 21:01:11 backupnas kernel: ------------[ cut here ]------------
This is not very good, though I'd need a full trace (this is just a few lines of one) to say more.
The second line tells you that BTRFS is falling back to read-only, which is why you have the other issues.
vandermerwe wrote:Apr 10 21:15:51 backupnas kernel: BTRFS error (device md127): parent transid verify failed on 120651776 wanted 13617 found 13384
Apr 10 21:15:51 backupnas kernel: BTRFS error (device md127): parent transid verify failed on 120651776 wanted 13617 found 13384
I'm afraid this is filesystem corruption. Afaik, it's really not easy to fix in place. Instead, an easier way would be to perform a full backup and recreate the volume.
OK.
I can recreate the volume.
I will test all the disks with vendor tools too.
Why would this happen out of the blue - the disks are OK, the nas is supported by an UPS and there have been no power issues.?
I don't know, sorry.
After recreating the entire volume, I have received some more input/output errors with ReadyDR jobs, if I rerun them they work. I am wondering whether this is a problem with 6.7.0 or is related to disk spindown on the source NAS - although the setup has worked normally before 6.7.0.
No problems with Rsync backups though.
NETGEAR Academy
Boost your skills with the Netgear Academy - Get trained, certified and stay ahead with the latest Netgear technology!
Join Us!
