Forum Discussion

Aspirant

May 27, 2021

Solved

Сhecksum errors in files on RAID5 Netgear ReadyNAS Pro 6 RNDP6000

Netgear ReadyNAS Pro 6 RNDP6000 RAID-5 (of 6 disks) 2TB FW 6.10.2 I got a checksum error message in the backup software. I was check disks and found on one many bad sectors. (using HDAT2 softw...

StephenB
Jun 04, 2021
Roman304 wrote:

I have partitioned jbod raid drives into each drive. RAID number 4 checksum error.

That's a clear indication that your issue is either linked to that disk or to that slot. The next step is to figure out which.

I suggest destroying RAID 1,2,5,6, and removing those disks. Then power down the NAS, and swap RAID 3 and RAID 4. Power up and re-run the test on both volumes. That will let you know if the problem is linked to the disk or the slot.

If the problem disappears on both disks, then it could be power-related. You can confirm that by adding the removed disks back one at a time, and see when the problem starts happening again.

StephenB

Guru - Experienced User

May 30, 2021

Roman304 wrote:

What log file do you recommend watching?

system.log, kernel.log, and system-journal.log

Roman304

Aspirant

May 31, 2021

StephenB wrote:
Roman304 wrote:
What log file do you recommend watching?
system.log, kernel.log, and system-journal.log

in kernel.log

May 30 14:35:29 NAS-2 kernel: mdcsrepair[13627]: segfault at 2325238 ip 00000000004048df sp 00007fff08ffa500 error 4 in mdcsrepair[400000+10000]
May 30 14:35:29 NAS-2 kernel: sh (13626): drop_caches: 3
May 30 14:35:29 NAS-2 kernel: sh (13625): drop_caches: 3
May 30 14:35:29 NAS-2 kernel: mdcsrepair[13639]: segfault at 1cd6230 ip 00000000004048df sp 00007ffeba060c50 error 4 in mdcsrepair[400000+10000]
May 30 14:35:29 NAS-2 kernel: mdcsrepair[13640]: segfault at bfe230 ip 00000000004048df sp 00007ffe6f442530 error 4 in mdcsrepair[400000+10000]
May 30 14:35:30 NAS-2 kernel: sh (13647): drop_caches: 3
May 30 14:35:30 NAS-2 kernel: sh (13648): drop_caches: 3
May 30 17:47:16 NAS-2 kernel: BTRFS warning (device md0): csum failed ino 26800 off 6852608 csum 846329252 expected csum 429493038
May 30 17:47:16 NAS-2 kernel: sh (9521): drop_caches: 3
May 30 17:47:17 NAS-2 kernel: sh (9522): drop_caches: 3
May 30 20:41:32 NAS-2 kernel: BTRFS warning (device md0): csum failed ino 26800 off 2220032 csum 4138936066 expected csum 3680012373
May 30 20:41:32 NAS-2 kernel: BTRFS warning (device md0): csum failed ino 26800 off 5521408 csum 988873707 expected csum 286883169
May 30 20:41:32 NAS-2 kernel: BTRFS warning (device md0): csum failed ino 26800 off 5603328 csum 2031798327 expected csum 1425106784
May 30 20:41:32 NAS-2 kernel: sh (2563): drop_caches: 3
May 30 20:41:32 NAS-2 kernel: BTRFS warning (device md0): csum failed ino 26800 off 6266880 csum 1389706193 expected csum 2134807686
May 30 20:41:32 NAS-2 kernel: BTRFS warning (device md0): csum failed ino 26800 off 6754304 csum 1977957496 expected csum 1477350191
May 30 20:41:32 NAS-2 kernel: BTRFS warning (device md0): csum failed ino 26800 off 6754304 csum 1977957496 expected csum 1477350191
May 30 20:41:32 NAS-2 kernel: sh (2567): drop_caches: 3
May 30 20:41:32 NAS-2 kernel: sh (2569): drop_caches: 3
May 30 20:41:32 NAS-2 kernel: mdcsrepair[2571]: segfault at 1fd5230 ip 00000000004048df sp 00007ffe294f0c30 error 4 in mdcsrepair[400000+10000]

what is mean - mdcsrepair segfault ?

i found problems in logs on root device /dev/md0

in system-journal.log

May 30 20:41:32 NAS-2 kernel: BTRFS warning (device md0): csum failed ino 26800 off 2220032 csum 4138936066 expected csum 3680012373
May 30 20:41:32 NAS-2 mdcsrepaird[2862]: mdcsrepaird: mdcsrepair /dev/md0 622764032 4096 fdd84c09 aa77a724 //var/readynasd/db.sq3
May 30 20:41:32 NAS-2 kernel: BTRFS warning (device md0): csum failed ino 26800 off 5521408 csum 988873707 expected csum 286883169
May 30 20:41:32 NAS-2 kernel: BTRFS warning (device md0): csum failed ino 26800 off 5603328 csum 2031798327 expected csum 1425106784
May 30 20:41:32 NAS-2 kernel: sh (2563): drop_caches: 3
May 30 20:41:32 NAS-2 kernel: BTRFS warning (device md0): csum failed ino 26800 off 6266880 csum 1389706193 expected csum 2134807686
May 30 20:41:32 NAS-2 kernel: BTRFS warning (device md0): csum failed ino 26800 off 6754304 csum 1977957496 expected csum 1477350191
May 30 20:41:32 NAS-2 kernel: BTRFS warning (device md0): csum failed ino 26800 off 6754304 csum 1977957496 expected csum 1477350191
May 30 20:41:32 NAS-2 kernel: sh (2567): drop_caches: 3
May 30 20:41:32 NAS-2 mdcsrepaird[2862]: mdcsrepair: repairing /dev/md0 @ 622764032 [//var/readynasd/db.sq3] succeeded.
May 30 20:41:32 NAS-2 kernel: sh (2569): drop_caches: 3
May 30 20:41:32 NAS-2 kernel: mdcsrepair[2571]: segfault at 1fd5230 ip 00000000004048df sp 00007ffe294f0c30 error 4 in mdcsrepair[400000+10000]
May 30 20:41:32 NAS-2 mdcsrepaird[2862]: mdcsrepaird: mdcsrepair /dev/md0 626065408 4096 14fc0ec5 9e82e6ee //var/readynasd/db.sq3
May 30 20:41:32 NAS-2 mdcsrepaird[2862]: mdcsrepaird: mdcsrepair /dev/md0 626147328 4096 c837e586 9f980eab //var/readynasd/db.sq3
May 30 20:41:32 NAS-2 mdcsrepaird[2862]: mdcsrepaird: mdcsrepair /dev/md0 626810880 4096 2ec42aad 796bc180 //var/readynasd/db.sq3
May 30 20:41:32 NAS-2 mdcsrepaird[2862]: mdcsrepaird: mdcsrepair /dev/md0 627298304 4096 87c31a8a d06cf1a7 //var/readynasd/db.sq3
May 30 20:41:33 NAS-2 kernel: sh (2616): drop_caches: 3
May 30 20:41:33 NAS-2 kernel: sh (2617): drop_caches: 3
May 30 20:41:33 NAS-2 mdcsrepaird[2862]: mdcsrepair: designated data mismatches bad checksum /dev/md0 @ 626065408
May 30 20:41:33 NAS-2 kernel: sh (2620): drop_caches: 3
May 30 20:41:33 NAS-2 kernel: sh (2619): drop_caches: 3
May 30 20:41:33 NAS-2 mdcsrepaird[2862]: mdcsrepair: designated data mismatches bad checksum /dev/md0 @ 627298304
May 30 20:41:33 NAS-2 kernel: sh (2618): drop_caches: 3
May 30 20:41:33 NAS-2 kernel: sh (2622): drop_caches: 3
May 30 20:41:33 NAS-2 mdcsrepaird[2862]: mdcsrepair: designated data mismatches bad checksum /dev/md0 @ 626147328
May 30 20:41:33 NAS-2 kernel: sh (2621): drop_caches: 3
May 30 20:41:33 NAS-2 kernel: sh (2623): drop_caches: 3
May 30 20:41:33 NAS-2 mdcsrepaird[2862]: mdcsrepair: designated data mismatches bad checksum /dev/md0 @ 626810880

Can we conclude that my device is completely defective and the operating system does not work correctly? What can I do about it?

StephenB
Guru - Experienced User
May 31, 2021
Roman304 wrote: Can we conclude that my device is completely defective and the operating system does not work correctly? What can I do about it?

It looks that way to me. The csum errors say that files are damaged in the OS partition, and the segmentation faults mean that the repair attempt was either reading or writing to an illegal memory location.

Failing memory is a possibility, and could explain the symptoms. You could try running the memory test from the boot menu. Note the test will keep running until it finds an error. I'd stop it at 3 passes (unless of course it finds an error).

How much memory do you have in the Pro-6?
- Roman304
  Aspirant
  May 31, 2021
  StephenB wrote:
  Roman304 wrote: Can we conclude that my device is completely defective and the operating system does not work correctly? What can I do about it?
  It looks that way to me. The csum errors say that files are damaged in the OS partition, and the segmentation faults mean that the repair attempt was either reading or writing to an illegal memory location.
  
  Failing memory is a possibility, and could explain the symptoms. You could try running the memory test from the boot menu. Note the test will keep running until it finds an error. I'd stop it at 3 passes (unless of course it finds an error).
  
  How much memory do you have in the Pro-6?
  2 x DDR2 1Gb Moudile - total 2056Mb ram
  
  i tested from usb boot by memtest86 v5. 7 laps no errors.
  - DEADDEADBEEF
    Apprentice
    May 31, 2021
    I can only imagine this being an issue in memory, it seems seriously strange to me that btrfs is not throwing errors all over the place if a file on disk was suddenly changed!!
    
    I wonder if file is actually fine but it's something that happens when md5sum calculates the checksum.
    
    Does it just happen with these large files? What if you just create a file with some normal text content like "this is a file"? If checksum changes for such a file it would be interesting to compare the contents before and after.
    
    Can you try to copy the file out from the NAS via SMB and then check the checksum again on the machine that downloaded it, does it match the "new" checksum or the original?