OS6 now works on x86 Legacy WARNING: NO NTGR SUPPORT!

This likely is more a BTRFS (filesystem) issue than one of upgrading to OS 6 on legacy hardware but ....

Be prepared to drop down to SSH management with OS 6 and BTRFS, as I have now learned. Specifically, my lightly used, mostly read-only volume had (unbeknowsnt to me) 27 CRC errors on file blocks allocated within files. Once this occurs, you cannot copy the file nor delete it. Not clear if the ReadyNAS Frontview interface has enough functionality to resolve this issue that OS6 and BTRFS introduce.

Problem:
File would not copy (be read) without the host OS receiving a report of a low-level I/O Block error. This is in the host machine mounting the ReadyNAS volume over SMB. Windows File Explorer and Robocopy simply reports it as a "file copy error". BeyondCompare reports it as "Unable to copy; a device attached to the system is not functioning". There is no alert reported by the ReadyNAS or in the log. It is not treated by the disk as a sector fail to be reallocated as the disk CRC was fine. BTRFS, at a higher level, has added CRC checking to blocks. But a failure at this filesystem level gets reported up simply as an I/O error when trying to read the file block; which is not expected by the OS and normally indicates a basic disc failure that should generate an alert from the READYNAS.

Diagnosis:
The CRC for a block within a file gets corrupted (actually 27 blocks in 23 files for me) . Likely at creation time. The file contents (after fixing) appears to be fine. This can only be detected with a BTRFS Scrub or by looking at the internal Linux message log (dmesg). Key is the error is not detected and reported during operation by the ReadyNAS. You have to go look for the problem and diagnose it yourself. Maybe something Netgear will fix in the future?

Note: not clear if the Frontview Volume option to "Scrub" is the same function as given below. Not sure how Frontview would report errors if Scrub is run from there either. For me, the "checksum" checkbox on the Volume properties pop-up was cleared. Not sure if this is supposed to turn off BTRFS block checksum function and checking. It did not for me. Not sure if it was always cleared or unchecked by default.

Solution:
DISCLAIMER: Dropping down to SSH access with your ReadyNAS generally voids your warranty. And unmounting an MD RAID volume to fix Filesystem errors is definitely core, kernel level activity. Do so with extreme caution and only if very comfortable with Linux/Unix low-level functions.

1. Determine the MD RAID device identifier of the VOLUME with errors. I simply used "df" to figure out the VOLUME_NAME and device of "md127":
   

   root@READYNAS:~# df
   Filesystem       1K-blocks        Used Available Use% Mounted on
   rootfs             4190208      336380   3427300   9% /
   tmpfs                10240           4     10236   1% /dev
   /dev/md0           4190208      336380   3427300   9% /
   tmpfs              2020340           0   2020340   0% /dev/shm
   tmpfs              2020340        6000   2014340   1% /run
   tmpfs              2020340           0   2020340   0% /sys/fs/cgroup
   tmpfs              2020340           0   2020340   0% /media
   /dev/md127     11706500352 11119192052 585113420  96% /VOLUME_NAME
   /dev/md127     11706500352 11119192052 585113420  96% /home
   /dev/md127     11706500352 11119192052 585113420  96% /apps
   /dev/md127     11706500352 11119192052 585113420  96% /run/nfs4/data/SHARE_NAME
   



2. Run a scrub on the mounted VOLUME_NAME to verify the errors; wait for it to finish before proceeding further
   

   
   root@READYNAS:~# btrfs scrub start /VOLUME_NAME
   scrub started on /VOLUME_NAME, fsid c4da3ed4-d080-4e0e-83d9-b223001eef05 (pid=4353)
   ###
   ### Repeat the [b]Status[/b] command below as needed until a final report as shown is given.  
   ###   Frontview detects the Scrub and will report its progress but simply no longer indicates the operation once done.
   ###
   root@READYNAS:~# btrfs scrub status /VOLUME_NAME
   scrub status for c4da3ed4-d080-4e0e-83d9-b223001eef05
           scrub started at Fri Aug 29 22:08:27 2014 and finished after 33028 seconds
           total bytes scrubbed: 10.34TiB with 27 errors
           error details: csum=27
           corrected errors: 0, uncorrectable errors: 27, unverified errors: 0
   #
   # If desired, confirm details of errors by looking at message log from Linux kernal and utilities:
   #
   root@READYNAS:~# dmesg 
   ...
   btrfs csum failed ino 297 off 9038946304 csum 2266718443 private 71772232
   ### Sample message entry above for when trying to read a block with a bad checksum; one or more entries for each attempt to read the bad block
   ....
   btrfs: checksum error at logical 197623537664 on dev /dev/md127, sector 388097008, root 266, inode 297, offset 9038946304, length 4096, links 1 (path: <<directory path and file name within share>>)
   btrfs: bdev /dev/md127 errs: wr 0, rd 0, flush 0, corrupt 1, gen 0
   btrfs: unable to fixup (regular) error at logical 197623537664 on dev /dev/md127
   ### Sample message for each block with a checksum error found in the scrub; identifies the paths and files that will not read/copy
   ....
   



3. Prepare to fix the problem by unmounting the MD RAID volume (cannot fix file system while mounted; cannot do this in Frontview to my knowledge):
   

   
   ###
   ### Just to verify the MD knows about the volume
   ###
   root@READYNAS:~# mdadm --detail /dev/md127
   /dev/md127:
           Version : 1.2
     Creation Time : Sun Jul 27 12:58:41 2014
        Raid Level : raid5
        Array Size : 11706500352 (11164.19 GiB 11987.46 GB)
     Used Dev Size : 3902166784 (3721.40 GiB 3995.82 GB)
      Raid Devices : 4
     Total Devices : 4
       Persistence : Superblock is persistent
   
     Intent Bitmap : Internal
   
       Update Time : Fri Aug 29 23:21:49 2014
             State : active
    Active Devices : 4
   Working Devices : 4
    Failed Devices : 0
     Spare Devices : 0
   
            Layout : left-symmetric
        Chunk Size : 64K
   
              Name : 5e24f95e:data-0  (local to host 5e24f95e)
              UUID : 9c032697:d91707d1:65d32704:66dc17d2
            Events : 16927
   
       Number   Major   Minor   RaidDevice State
          0       8        3        0      active sync   /dev/sda3
          1       8       19        1      active sync   /dev/sdb3
          2       8       35        2      active sync   /dev/sdc3
          3       8       51        3      active sync   /dev/sdd3
   ###
   ### OK, lets unmount
   ###
   root@READYNAS:~# umount /dev/md127
   ###
   ### repeated umount command 4 times as there were four mount points for this filesystem shown in "df"; not just the main volume
   ### have to make sure nothing is attached to or using the volumes otherwise umount will fail (e.g. your login may be using /home)
   ###
   



4. Fix the filesystem by running BTRFS Check (formerly known as btfsck):
   

   
   root@READYNAS:~# btrfs check --fix-crc /dev/md127
   Fixing bad CRC
   Checking filesystem on /dev/md127
   UUID: c4da3ed4-d080-4e0e-83d9-b223001eef05
   checking extents
   ref mismatch on [2700638683136 32768] extent item 33, found 1
   Errors found in extent allocation tree or chunk allocation
   checking free space cache
   cache and super generation don't match, space cache will be invalidated
   checking fs roots
   checking csums
   checking root refs
   found 5942559407194 bytes used err is 0
   total csum bytes: 11083458464
   total tree bytes: 11775672320
   total fs tree bytes: 27557888
   total extent tree bytes: 55345152
   btree space waste bytes: 358773385
   file data blocks allocated: 11349461471232
    referenced 11349461430272
   Btrfs v0.20-rc1
   ###
   ### NOTE: Fix-CRC was not enough for me.  Had to recreate the Checksum tree.
   ###
   root@READYNAS:~# btrfs check --init-csum-tree /dev/md127
   Creating a new CRC tree
   Checking filesystem on /dev/md127
   UUID: c4da3ed4-d080-4e0e-83d9-b223001eef05
   checking extents
   Reinit crc root
   found 0 bytes used err is 0
   total csum bytes: 0
   total tree bytes: 0
   total fs tree bytes: 0
   total extent tree bytes: 0
   btree space waste bytes: 0
   file data blocks allocated: 0
    referenced 0
   Btrfs v0.20-rc1
   ###
   ###  At this point, need to remount the various file systems you unmounted before.  I found it simplest to reboot.  
   ###     Either use Frontview to restart the ReadyNAS or the shell if you feel comfortable ...
   ###
   root@READYNAS:~# shutdown -r
   



5. Verify the problems are fixed (although you should be able to copy the file now as a verification as well):
   

   
   ###
   ### Maybe just do the Volume Scrub using Frontview after a reboot?  Not sure if the same but likely.
   ###
   root@READYNAS:~# btrfs scrub start /VOLUME_NAME
   ....
   ###
   ### Wait for scrub to finish and report all ok
   ###
   root@READYNAS:~# btrfs scrub status /VOLUME_NAME
   ....
   



My setup:
Ultra4 Plus with 4GB DRAM, 6.1.8 firmware (straight to this latest version from 4.2.x), 4x 4TB Hitachi disks, X-Raid RAID5 single volume, single share
After upgrading to 6.1.8, recreated my 10.33 TB in a couple thousand files on the 10.90 TB available. Checked result by comparing file stats only; not binary compare that would read every block. Copied using Win8.1 File Explorer "drag and drop" (with BeyondCompare "touch") or Robocopy; both on a Win8.1 host as this was much, much faster than Linux Rsync or NFS mounts and cp/scp between ReadyNAS boxes. Everything seemed fine. A week later decided to move 2 TB off this share onto another device / volume. But got copy errors in 5 files. This was undetected before as I had not read back each, individual block after writing.

I have an Ultra+4, Pro4, and Pro2 with upgraded memory; all running OS6 . Had 3 NV+ before.

I believe the frontview scrub should have repaired the damage unless unless the parity blocks were also damaged.

StephenB wrote:
I believe the frontview scrub should have repaired the damage unless unless the parity blocks were also damaged.

No damage to data (RAID or otherwise) if that is what you were implying by "parity blocks". Only BTRFS Checksum's were corrupted. Checksum and CRC fixes appear to require the filesystem be offline. And Frontview's Scrub does not take the volume offline while working. So suspect they are doing the simple BTRFS SCRUB command, which works on a mounted filesystem, but then cannot correct many errors it detects. See my results of commands in the last post.

Yes I did look at them. So your checksums were wrong, and but the data itself was fine. btrfs scrub could not fix the checksum errors. btrfs check fixed the errors with the volume off line.

You are correct in thinking that the NAS does a simple btrfs scrub when you do the frontview scrub. At least top shows the command as "btrfs scrub start /volume"

Why do you think the checksums were wrong in the first place?

StephenB wrote:
Why do you think the checksums were wrong in the first place?

That is the $10,000 question. Would need a bit of investigating -- hardware as well as software. Many places for it to go wrong. But wanted to at least make people aware that a workaround exists for the corruption that can be introduced. The CRC stored or calculated incorrectly does appear to be a rarer occurrence -- only 27 out of 2.7 billion blocks written had the error -- hmm, that already looks like too much coincidence. Maybe why Netgear chose not to release OS6 with BTRFS on their older hardware?

There are two pieces of good news though. First is that the scrub detected it (though it did not fix). The second one (in your case anyway) there was no data loss involved. People here can verify their own systems with your first step.

Hi fellas,
I read a lot about Pro's and Plus's upgrading to OS6, but not too many people with Ultra's and only one person with an Ultra 6.
I'm wanting to upgrade my Ultra 6 to OS6 but would really like to hear first from people in the know if there any known compatibility issues or known problems (other than non working Genie and Antivirus that might F the system up). I have absolute zero knowledge of Linux so if I'd need to go into SSH then I'm Fubar.

Also;
- Does performance suffer under OS6 since the Ultra's only have a puny Atom CPU.
- And in that light, do temps go up noticably? (Although my CPU is hard stuck at 50c...)
- On my RN516 there is currently no idle drive spin down which simply aggravates me. My RNDU6000 under OS4 DOES have idle drive spin down. Is it safe to assume then that when I upgrade my Ultra 6 to OS6 that I will lose idle drive spin down? (Until 6.2 comes out supposedly).
- The post above somewhat concerns me. If frontview scrubbing is so useless in OS6 then how can I, as nitwit end user, fix any file errors? (The whole SSH spiel is over my head, even if it is interesting).
- Does OS4 have this scrubbing effectiveness limitation? (I might wanna stick with that if it doesn't).
- Is there some form of tool or simpler command to enable effective scrubbing under OS6?
- Can anyone simply confirm the voltage reporting issue and fan issue are no longer issues with 6.1.9 on an Ultra 6?

I'm not afraid to experiment but would like to know a little better what I'm walking into if others already suffered it. Rebuilding a 30TB array under OS4 again and copying the files again will take over a week if it should be a failure.

For the record, "fixing" CRCs is generally very bad advice. I have seen no verified cases of bad CRCs on an x86 machine (although that of course does not mean they don't exist), but I have seen verified cases of bad data with good CRCs. So, if you see an I/O error due to a checksum mismatch, odds are that the data is corrupt. So rewriting the CRCs based on the current data basically "bakes in" data corruption. Starting in 6.1.9, you can remount the volume with the "recovery" option to allow bad data to be read, rather than just returning an I/O error.

All right.. so even IF there should be an error that can only be fixed by un/dis-mounting the filesystem, now the firmware can do it too for basically Linux inept people... (like myself, but only because of ignorance, not stupidity).
Just when my OS4 is at 91.5% resyncing... Now I can do it all over again for OS6 ;)

One teeny tiny thing that's holding me back though... what in case I shoud want to revert back to OS4?
I know of the link with the 4.2.26 image, but it says "you will likely need to do a USB Boot Recovery followed by a factory default to get a working system again."
What is a USB Boot recovery?

And will this OS4 reverting package revert *any* OS6 package that's installed? Or just the 6.1.7 that's mentioned in the beginning of this thread?

There is a downgrade image here (may be updated from time to time): http://www.readynas.com/forum/viewtopic.php?f=35&t=75198

If you have trouble downgrading with that image let me know.