How does bitrot protection actually work?

StephenB wrote:
BaJohn wrote: mdgm wrote: You need to be using a md raid level that provides redundancy. Does that include RAID10 as like what I have on my RN516 with 6 x 4TB. Or are you talking about RAID5 and RAID6 only. Just to clarify your question - I think you are actually wanting to know if the bitrot protection is supported for RAID-10?

By default the bitrot protection (BRP) on my RN516 is OFF.
I assumed that as the button was there, it was supported regardless of RAID although it MAY not make sense for certain RAID levels.
I was trying to clarify whether RAID-10 has redundancy in the way you mean it, and hence it is a good thing to have BRP ON.
Perhaps that is the question to be answered :)

RAID-10 certainly has redundancy. Netgear will need to confirm that their bitrot feature supports RAID-10 (seems likely though).

RAID-10 would work.

I have read and re-read this post many times and it is full of useful info.
I am just concerned about the following quote.

Nhellie wrote:
As per release notes, "Support bitrot data protection. Automatically detect and correct corruption due to media degradation.", for me this means that as soon as you enable BitRot protection, the NAS will scan-detect-fix corruptions on the data.

To me this reads as proactively scanning the data, and I am not convinced this is how it works.
i.e. If I put data on my ReadyNAS (with snapshots) and never updated it for 5 years, would bitrot be detected?
The above suggests yes, I think it might be no, BUT I really don't know :?

There are several posts on this topic, mostly speculative on the details.

What we know for sure is that

(a) BTRFS includes a checksum feature, and that is enabled when bitrot protection is on.
(b) if a checksum error occurs, then bitrot is detected, and the Netgear algorithm attempts to repair it from the RAID parity blocks. This is different from normal RAID repair, which is triggered by a read failure (not a checksum error).

One or two users have reported cases there this algorithm failed to recover data with a correct checksum. So far I have not seen any users reporting a case where it did recover data.

StephenB wrote:
There are several posts on this topic, mostly speculative on the details.

Hence my comments in this forum about having a definitive technical source.

StephenB wrote:

What we know for sure is that (a) BTRFS includes a checksum feature, and that is enabled when bitrot protection is on. (b) if a checksum error occurs, then bitrot is detected, and the Netgear algorithm attempts to repair it from the RAID parity blocks. This is different from normal RAID repair, which is triggered by a read failure (not a checksum error). One or two users have reported cases there this algorithm failed to recover data with a correct checksum. So far I have not seen any users reporting a case where it did recover data.

BUT what prompts for a checksum error to be discovered?
i.e. Does BTRFS regularly do checksum testing unprompted? Is it only on a data write? Is it on a data read? etc

BTRFS generates the checksums on writes, and verifies them on reads. This feature is built into BTRFS itself.

Netgear's protection algorithm is their own, and they aren't saying much about how it works, other than what I said above. It's a unique feature for Netgear, and my guess is that they want to keep it that way.

There is a similar bitrot protection being built into BTRFS (using a raid-like mode that is integrated into the file system). But that raid-like mode is still experimental, and OS6 is using traditional software raid instead.

Thanks Stephen.
So to answer my own question:-
"If I put data on my ReadyNAS (with snapshots) and never updated it for 5 years, would bitrot be detected?"
Yes BUT only when I go to read some data, and more significantly only those blocks that are being read would be checked.
Then the BTRFS passes the error to the RNOS which (in my case - RAID10) would go off and repair with data from the mirror.
Thanks again.

Hi, I've got an RN102 with only one disk installed.
The NAS initialized with a single volume under XRAID2 JBOD.
Checksums are enabled on the volume because the default set of shares have bitrot protection enabled.
On one of the shares, I disabled bitrot protection (since bitrot can't be fixed without a redundant copy) and set snapshots to never and then restored data on to it over usb.

I presume BTRFS has created checksums for these files? Does that mean that, even with bitrot protection disabled, BTRFS will detect bitrot and alert me to the problem?
Is there any benefit in enabling bitrot protection in this scenario?
Or should I be switching checksums off as well? (although I might add a second disk later...then I might have the option of enabling automatic bitrot protection on the existing data)

Thanks
Paul.

I believe checksums are enabled/disabled at the volume level only (at least I am not seeing any subvolume controls). If so, then btrfs should alert you to checksum failures.

I run jbod also, but have bitrot protection and snapshots enabled on shares. (if you want CoW on all the time, then you do need bitrot protection enabled, since the features are coupled).