RN104 says inactive volume

So I tried the readonly mode, no effect.  Same error in dmesg.

I did a bit more poking around.  So each disk has 4 partitions, and if I use mdadm --examine on each set, there appear to be 4 raid set ups

/dev/sd*1 is raid 1
/dev/sd*2 is raid 6
/dev/sd*3 is raid 5
/dev/sd*4 is raid 5

If I look at the raid states:

root@nas:~# mdadm --examine /dev/sd*1|grep "State"                       
          State : clean                                                  
   Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)  
          State : clean                                                  
   Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)  
          State : clean                                                  
   Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)  
          State : clean                                                  
   Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)  
root@nas:~# mdadm --examine /dev/sd*2|grep "State"                       
          State : clean                                                  
   Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)  
          State : clean                                                  
   Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)  
          State : clean                                                  
   Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)  
          State : clean                                                  
   Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)  
root@nas:~# mdadm --examine /dev/sd*3|grep "State"                       
          State : clean                                                  
   Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)  
          State : clean                                                  
   Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)  
          State : clean                                                  
   Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)  
          State : clean                                                  
   Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)  
root@nas:~# mdadm --examine /dev/sd*4|grep "State"                       
          State : clean                                                  
   Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)  
          State : clean                                                  
   Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)  
          State : clean                                                  
   Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)  
          State : clean                                                  
   Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)  

So everything is clean.

But if I try to mount the file system, even in degraded mode, it fails:

root@nas:~# mount -t btrfs -o degraded /dev/md127 /var/recover/
mount: wrong fs type, bad option, bad superblock on /dev/md127,
       missing codepage or helper program, or other error
       In some cases useful info is found in syslog - try
       dmesg | tail or so.
root@nas:~# mount -t btrfs /dev/md127 /var/recover/
mount: wrong fs type, bad option, bad superblock on /dev/md127,
       missing codepage or helper program, or other error
       In some cases useful info is found in syslog - try
       dmesg | tail or so.

If I look at dmesg, it is the btrfs ctree error:

[ 3847.552053] BTRFS (device md126): bad tree block start 11637558493067324903 17973882290176
[ 3847.552446] BTRFS (device md126): bad tree block start 11637558493067324903 17973882290176
[ 3847.552481] BTRFS: failed to read tree root on md126
[ 3847.580455] BTRFS: open_ctree failed
[ 3852.027911] BTRFS (device md126): bad tree block start 11637558493067324903 17973882290176
[ 3852.028306] BTRFS (device md126): bad tree block start 11637558493067324903 17973882290176
[ 3852.028342] BTRFS: failed to read tree root on md126
[ 3852.070223] BTRFS: open_ctree failed

So, it seems to me that this is some sort of btrfs corruption issue.

btrfs fi show 

Label: '0e343298:data'  uuid: 92ecd2e0-45da-425b-87a8-0f626e5ef889
        Total devices 2 FS bytes used 5.04TiB
        devid    1 size 2.71TiB used 2.53TiB path /dev/md127
        devid    2 size 2.73TiB used 2.54TiB path /dev/md126

Now I don't know why there are two md raid set ups, from other digging (dry run) I see md127 thinks device 2 is missing, whatever that is, is that the second drive? the second raid set (/dev/md126?)

oot@nas:~# btrfs restore -F -i -D -v /dev/md127 /dev/null
checksum verify failed on 17973882290176 found 28166D04 wanted 8B8262A2
checksum verify failed on 17973882290176 found 28166D04 wanted 8B8262A2
checksum verify failed on 17973882290176 found 46A24E6E wanted 51485848
checksum verify failed on 17973882290176 found 46A24E6E wanted 51485848
bytenr mismatch, want=17973882290176, have=3707086136388172086
Couldn't read tree root
Could not open root, trying backup super
warning, device 2 is missing
checksum verify failed on 17973882290176 found 46A24E6E wanted 51485848
checksum verify failed on 17973882290176 found 46A24E6E wanted 51485848
bytenr mismatch, want=17973882290176, have=3707086136388172086
Couldn't read tree root
Could not open root, trying backup super
warning, device 2 is missing
checksum verify failed on 17973882290176 found 46A24E6E wanted 51485848
checksum verify failed on 17973882290176 found 46A24E6E wanted 51485848
bytenr mismatch, want=17973882290176, have=3707086136388172086
Couldn't read tree root
Could not open root, trying backup super

Any suggestions?

I do have a backup, but I really would like to take this as a learning opportunity.

Thanks!