While I've fixed the problem (more on that toward the bottom), I had an odd problem with the rsync backup of one share. The job got "stuck" and top on the source NAS showed the rsync process at 100%, but there was almost no network activity, The destination NAS showed almost no rsync CPU activity, even though this was a "pull" backup. This being a rack-mount ReadyNAS, that's a lot of CPU for rsync to use. I cancelled the job and the copy log said there is no log when the share is in read-only mode. But the share was not in read-only mode on either the source or destination. I manually started the job again, and there was a flurry of network activity after which the source again showed 100% CPU use and network activity went to near zero. I let if go for a day, and nothing changed. When I cancelled it, I got the same message about a read-only share. I could see that some files did not update on the destination, but none of them seemed special in any way.
The share contains 3.7TB of data, so it could take over a day for the backup if there were a lot of changes, but there weren't. I created and deleted files in the share on both NAS and even deleted a couple snapshots to convince myself the shares were healthy, and nothing changed. I ran a scrub and balance on both (which takes days on my nearly 100TB main NAS), and nothing changed. I ran a btrfs quota rescan on each share, and nothing changed. None of those indicated any error, either.
So has anyone seen anything like that? I'd like to know what happened to keep it from happening again. We've seen the latest OS versions getting some error messages wrong (I'm running 6.10.9), so the message in the log may just be wrong, but something went wrong.
So, how did I fix it? With a sledgehammer. I created a new share on each. Via SSH, I did a cp /a of the contents of the share on the source NAS to it's new share. On the destination NAS, I mounted the new share on the source via NFS and then did a cp /a to the new destination share. I then updated and ran the backup job, which finished almost immediately because the contents of the shares were the same. A week later, the subsequent backup jobs are running fine, I have a week's worth of snapshots of the new share, and I'm as convinced as I can be that none of the files in the new share are corrupt, so I deleted the old share to re-gain the space.
In all of this, the only error message I got was the backup log saying the volume was read-only. Problem ultimately solved, but what in the world went wrong?
