Forum Discussion

milind2021

Aspirant

Oct 07, 2023

btrfs errors for no apparent reason on RN212

I have been using this Readynas for the past 1.5 years, but all of a sudden on Oct 4, I found I couldn't write to the share. Digging revealed a btrfs problem in dmesg log: BTRFS critical (device md1...

Other Discussions

milind2021

Aspirant

Oct 29, 2023

Finally my memtester logs - BTW, it is built in into the firmware image of the readynas, so it is highly convenient to use.

# memtester 1900M 1
memtester version 4.3.0 (32-bit)
Copyright (C) 2001-2012 Charles Cazabon.
Licensed under the GNU General Public License version 2 (only).

pagesize is 4096
pagesizemask is 0xfffff000
want 1900MB (1992294400 bytes)
got  1896MB (1988739072 bytes), trying mlock ...locked.
Loop 1/1:
  Stuck Address       : ok         
  Random Value        : ok
FAILURE: 0xa8b536b8 != 0xa8b536bc at offset 0x18ef7df4.
  Compare XOR         : FAILURE: 0xe8fde78a != 0xe8fde792 at offset 0x18ef7df4.
  Compare SUB         : FAILURE: 0x760a25c8 != 0xb4f2633c at offset 0x18ef7df4.
  Compare MUL         :   Compare DIV         : ok
  Compare OR          : ok
  Compare AND         : ok
  Sequential Increment: ok
  Solid Bits          : testing   1FAILURE: 0xfffffffb != 0xffffffff at offset 0x18ef7df4.
  Block Sequential    : testing   4FAILURE: 0x04040400 != 0x04040404 at offset 0x18ef7df4.
  Checkerboard        : testing   1FAILURE: 0x55555551 != 0x55555555 at offset 0x18ef7df4.
  Bit Spread          : testing   1FAILURE: 0xfffffff1 != 0xfffffff5 at offset 0x18ef7df4.
  Bit Flip            : testing   1FAILURE: 0xfffffffa != 0xfffffffe at offset 0x18ef7df4.
  Walking Ones        : testing   0FAILURE: 0xfffffffa != 0xfffffffe at offset 0x18ef7df4.
  Walking Zeroes      : testing   2FAILURE: 0x00000000 != 0x00000004 at offset 0x18ef7df4.
  8-bit Writes        : ok
  16-bit Writes       : ok

Done.

You can even see here how the last hex digit is modified by 4, or basically the bit corresponding to 2^2 is flipped between writing and then subsequent reading. The fact that it failed in only one try was a bit scary though.

milind2021

Aspirant

Oct 29, 2023

I missed out another thing that the experts at the btrfs IRC channel brought to my notice - multiple architectural flaws which fail to mitigate data corruption. Keep in mind that I'm not an expert in this, and is heavily based on what I learned in the IRC channel from users Zygo, Darkling, Forza, multicore.

First of all, a data storing machine should definitely have ECC memory. I am also leaning to this way of thinking now and will make sure to use ECC in machines where data integrity is important, wherever possible.
The fact that btrfs is layered over mdraid rather than btrfs itself handling the RAID bit. The problem with this lies in the fact that when a block is read by the btrfs driver, it asks md which arbitrarily chooses which disk to read from. So if there was corruption in one disk but not another, there's a 50% chance that btrfs would not even notice. Worse, apparently, in some conditions, there's a 25% chance that the good copy would be overwritten with the bad. Synology apparently has patches to hook the two layers together, so that btrfs can know which disk the data was read from.
The fact that bitrot protection is disabled by default, and with it, both CoW and data-sums are disabled. According to the experts, that pretty much makes btrfs toothless. There is also apparently not much use of scrub unless CoW is enabled.

I think these are fairly serious accusations, and Netgear seems to have failed users on data integrity. It is entirely because I assumed that Netgear knows more than me that I decided to buy a readyNAS than make my own.

StephenB
Guru - Experienced User
Oct 29, 2023
FWIW, the decision to layer btrfs over mdraid was made back in 2013 before btrfs RAID was considered stable. Changing the approach after that would have required everyone in the installed base to do a factory reset. Given that all ReadyNAS are now end-of-life, I expect any future updates will be limited to selected bug fixes and security fixes.

Most entry-level NAS (including for instance the Synology DS223j) do not have ECC memory. Some higher end ReadyNAS were sold with this (including my RN526x).