NETGEAR is aware of a growing number of phone and online scams. To learn how to stay safe click here.
Forum Discussion
Rsync error read only filesystem(30)
My rsync backups have been running flawlessly for years being pulled from a 316 to a Ultra 6+.
Both units are running 6.7.0.
For the last 2 days, backups are failing, all with the same error, - filesystem is read only(30)
The problem is clearly with the Ultra 6+. Rsync backups continue to work normally to a NV+
File permissions on the source have been reset, but interestingly the destination (Ultra 6+) will not allow me to change or reset permissions, in fact I can't write to the volume at all.
Ready DR backups to the same Ultra 6+ are also failing with an input/output error.
The files and folders on the source and destination look OK when I ssh in and look at permissions.
The OS partition is not full.
I think this is probably a disk problem but all disks smart stats look OK.
Any ideas?
14 Replies
Replies have been turned off for this discussion
vandermerwe wrote:Ready DR backups to the same Ultra 6+ are also failing with an input/output error.
So, I presume you installed OS6 on it.
If you download the logs and look at btrfs.log, do you see the data volume being fully allocated?
Any error in systemd-journal.log?
Label: '33ea9931:root' uuid: dff4e5dd-d198-401a-b8bf-09915591d8cd
Total devices 1 FS bytes used 544.79MiB
devid 1 size 4.00GiB used 1.63GiB path /dev/md0Label: '33ea9931:data' uuid: 5a9c472f-3757-4e79-b6b8-90792440712b
Total devices 1 FS bytes used 6.75TiB
devid 1 size 9.07TiB used 6.76TiB path /dev/md127=== filesystem /data ===
Data, single: total=6.76TiB, used=6.75TiB
System, DUP: total=32.00MiB, used=768.00KiB
Metadata, DUP: total=2.00GiB, used=1.00GiB
GlobalReserve, single: total=352.00MiB, used=7.00MiBThere are many errors in systemjournal about failing to delete snapshots and this:
Apr 10 21:01:11 backupnas kernel: BTRFS: error (device md127) in create_pending_snapshot:1545: errno=-5 IO failure
Apr 10 21:01:11 backupnas kernel: BTRFS info (device md127): forced readonly
Apr 10 21:01:11 backupnas kernel: BTRFS warning (device md127): Skipping commit of aborted transaction.
Apr 10 21:01:11 backupnas kernel: BTRFS: error (device md127) in cleanup_transaction:1843: errno=-5 IO failure
Apr 10 21:01:11 backupnas kernel: ------------[ cut here ]------------and this:
Apr 10 21:01:11 backupnas replisync[3057]: daemon: _splice error: Broken pipe
Apr 10 21:01:11 backupnas replisync[2863]: ERROR: creating snapshot KIDNASDR/.snapshots/618/snapshot -> snapshot failed: Input/output error
Apr 10 21:01:11 backupnas replisync[3057]: daemon: _btrfs_receive error: Broken pipe
Apr 10 21:01:11 backupnas replisync[2863]: ERROR: truncate Private practice/Copies of lists/2017/20170323 selvey.pdf failed: Read-only file system
Apr 10 21:01:11 backupnas replisync[3056]: daemon: _splice error: Broken pipe
Apr 10 21:01:11 backupnas replisync[2863]: ERROR: creating snapshot CSDNASDR/.snapshots/1075/snapshot -> snapshot failed: Input/output error
Apr 10 21:01:11 backupnas replisync[3055]: daemon: _splice error: Broken pipe
Apr 10 21:01:11 backupnas replisync[2863]: daemon: _splice error: Broken pipe
Apr 10 21:01:11 backupnas replisync[3055]: daemon: _btrfs_receive error: Broken pipe
Apr 10 21:01:11 backupnas replisync[2863]: ERROR: creating snapshot EADNASDR/.snapshots/1074/snapshot -> snapshot failed: Input/output error
Apr 10 21:01:11 backupnas replisync[3056]: daemon: _btrfs_receive error: Broken pipe
Apr 10 21:01:11 backupnas replisync[2863]: daemon: _splice error: Broken pipe
Apr 10 21:01:11 backupnas replisync[3058]: daemon: _splice error: Broken pipe
Apr 10 21:01:11 backupnas replisync[2863]: daemon: _splice error: Broken pipe
Apr 10 21:01:11 backupnas replisync[3058]: daemon: _btrfs_receive error: Broken pipe
Apr 10 21:01:11 backupnas replisync[2863]: daemon: _splice error: Broken pipeApr 10 21:15:51 backupnas readynasd[3078]: ERROR: failed to lookup path for root 7734: Input/output error
Apr 10 21:15:51 backupnas kernel: BTRFS error (device md127): parent transid verify failed on 120651776 wanted 13617 found 13384
Apr 10 21:15:51 backupnas kernel: BTRFS error (device md127): parent transid verify failed on 120651776 wanted 13617 found 13384
Apr 10 21:01:11 backupnas replisync[2863]: ERROR: failed to make subvolume read only: Read-only file systemand then also
Apr 10 21:09:23 backupnas fvbackup-q[3644]: cmd=/frontview/bin/fvbackup -e 2
Apr 10 21:09:29 backupnas readynasd[3078]: Error backup job EADNAS: failure during copy.I don't understand what these mean, there are numerous similar error messages regarding the readydr jobs and the rsync jobs that are failing.
The backup job error logs all say this:
Failed to run rsync, return rc 5888 (23)
'rsync' command returns rc=5888
vandermerwe wrote:Label: '33ea9931:data' uuid: 5a9c472f-3757-4e79-b6b8-90792440712b
Total devices 1 FS bytes used 6.75TiB
devid 1 size 9.07TiB used 6.76TiB path /dev/md127=== filesystem /data ===
Data, single: total=6.76TiB, used=6.75TiB
System, DUP: total=32.00MiB, used=768.00KiB
Metadata, DUP: total=2.00GiB, used=1.00GiB
GlobalReserve, single: total=352.00MiB, used=7.00MiBThis is OK.
vandermerwe wrote:Apr 10 21:01:11 backupnas kernel: BTRFS: error (device md127) in create_pending_snapshot:1545: errno=-5 IO failure
Apr 10 21:01:11 backupnas kernel: BTRFS info (device md127): forced readonly
Apr 10 21:01:11 backupnas kernel: BTRFS warning (device md127): Skipping commit of aborted transaction.
Apr 10 21:01:11 backupnas kernel: BTRFS: error (device md127) in cleanup_transaction:1843: errno=-5 IO failure
Apr 10 21:01:11 backupnas kernel: ------------[ cut here ]------------This is not very good, though I'd need a full trace (this is just a few lines of one) to say more.
The second line tells you that BTRFS is falling back to read-only, which is why you have the other issues.
vandermerwe wrote:Apr 10 21:15:51 backupnas kernel: BTRFS error (device md127): parent transid verify failed on 120651776 wanted 13617 found 13384
Apr 10 21:15:51 backupnas kernel: BTRFS error (device md127): parent transid verify failed on 120651776 wanted 13617 found 13384I'm afraid this is filesystem corruption. Afaik, it's really not easy to fix in place. Instead, an easier way would be to perform a full backup and recreate the volume.
- The disk health is ok although I haven't run extended tests on some of them.
I will run a memory test
NETGEAR Academy
Boost your skills with the Netgear Academy - Get trained, certified and stay ahead with the latest Netgear technology!
Join Us!
