btrfs_merge_bio_hook+a0 error, recovery possible?

And the RAID is definitely still running?

cat /proc/mdstat

It seems to be...

# cat /proc/mdstat
Personalities : [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md127 : active raid5 sda3[0] sdd3[3] sdc3[2] sdb3[1]
      11706506496 blocks super 1.2 level 5, 64k chunk, algorithm 2 [4/4] [UUUU]

md1 : active raid6 sda2[0] sdd2[3] sdc2[2] sdb2[1]
      1047424 blocks super 1.2 level 6, 64k chunk, algorithm 2 [4/4] [UUUU]

md0 : active raid1 sda1[0] sdd1[3] sdc1[2] sdb1[1]
      4190208 blocks super 1.2 [4/4] [UUUU]

unused devices: <none>

# btrfs fi sho
Label: '1165568a:Volume1'  uuid: 56990628-d1ea-4142-8c3d-5dd32b620f00
        Total devices 1 FS bytes used 8.30TiB
        devid    1 size 10.90TiB used 8.35TiB path /dev/md127

btrfs-progs v4.1.2

Okay, that is a problem then.

Do one thing. To be sure, scan for btrfs volumes first and then try the FS check again.

First:

btrfs device scan

Then:

btrfsck /dev/md127

Does it yield the same result?

Do you have backups of the data? It could be FS issues here.

Can you post output of the this as well:

dmesg | tail -n30

# btrfs device scan
Scanning for Btrfs filesystems

# btrfsck /dev/md127
Couldn't open file system

# dmesg | tail -n30
[<c02dfe9c>] (submit_extent_page) from [<c02e1694>] (__do_readpage+0x31c/0x8f0)
[<c02e1694>] (__do_readpage) from [<c02e1d38>] (__extent_read_full_page+0xd0/0xd8)
[<c02e1d38>] (__extent_read_full_page) from [<c02e41b0>] (read_extent_buffer_pages+0x1ec/0x2f8)
[<c02e41b0>] (read_extent_buffer_pages) from [<c02b6bc4>] (btree_read_extent_buffer_pages.constprop.5+0xb8/0x124)
[<c02b6bc4>] (btree_read_extent_buffer_pages.constprop.5) from [<c02b71c0>] (read_tree_block+0x24/0x44)
[<c02b71c0>] (read_tree_block) from [<c02b8538>] (btrfs_read_tree_root+0xc8/0x118)
[<c02b8538>] (btrfs_read_tree_root) from [<c02bb204>] (open_ctree+0x1474/0x2190)
[<c02bb204>] (open_ctree) from [<c0292204>] (btrfs_mount+0x88c/0xa74)
[<c0292204>] (btrfs_mount) from [<c00df854>] (mount_fs+0x44/0x154)
[<c00df854>] (mount_fs) from [<c00f8590>] (vfs_kern_mount+0x4c/0xfc)
[<c00f8590>] (vfs_kern_mount) from [<c0291c14>] (btrfs_mount+0x29c/0xa74)
[<c0291c14>] (btrfs_mount) from [<c00df854>] (mount_fs+0x44/0x154)
[<c00df854>] (mount_fs) from [<c00f8590>] (vfs_kern_mount+0x4c/0xfc)
[<c00f8590>] (vfs_kern_mount) from [<c00fac84>] (do_mount+0x1c4/0xbe8)
[<c00fac84>] (do_mount) from [<c00fb9ec>] (SyS_mount+0x74/0xa0)
[<c00fb9ec>] (SyS_mount) from [<c000f400>] (ret_fast_syscall+0x0/0x3c)
Code: 01520004 23a00000 33a00001 eaffffe4 (e7f001f2)
---[ end trace dd1049a4dc0f8ed7 ]---
al_eth 0000:00:01.0 eth0: al_eth_down
[eth rx] warn: dma state didn't change to Disable
[eth rx] warn: failed to change state, error -110
configured MAC to RGMII mode:
al_eth 0000:00:01.0 eth0: using MSI-X per Queue interrupt mode
libphy: al mdio bus: probed
al_eth 0000:00:01.0 eth0: phy[4]: device 8:04, driver Atheros 8035 ethernet
al_eth 0000:00:01.0 eth0: phy[4]:supported 2ef adv 2ef
IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
eth [al_eth_0]: set link speed to 1000Mbps. full duplex.
al_eth 0000:00:01.0 eth0: Link is Up - 1Gbps/Full - flow control off
IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

This is a NAS that stores backups, so no catastrophe if the data is lost, but it would be nice to be able to copy some of it.

Cheers for that.

Yes, that call trace shows definite issues with the filesystem. Looks very much like FS corruption. I do not think there is any way you can mount the data here. These things can happen - which is why backups are important.

You should consider one of two options (as you have backups available):

1. Reset NAS. It will erase everything, but also give you a fresh filesystem. Restore from backups afterwards.

2. Contact Netgear support . Level 3 support in Netgear might be able to do something here, but it would be against payment for a data recovery contract - most likely. I think it is like €150-€170 euro or something like that. But even in that case, they might not be able to solve it. I don't know the true damage/state of the filesystem. But, it does not look good :(

This can be recovered (data recovery is dependent on circumstances) using 3rd party software and a Windows or linux pc.  Transferring the discs to the PC or an esata box and the software reconstructs the volume fo to be trnsferred of the volume onto your recovery drives

 A search on this forum should indicate the software that could be used.

Could you link a thread that mentions possible recovery software?

https://community.netgear.com/t5/Using-your-ReadyNAS/Recovering-RAID-Array-after-quot-No-volume-Found-quot-error/m-p/1042748#M103037

Thank you, I'll check that.

I moved the HDDs to a Linux (Ubuntu) PC.

/proc/mdstat shows the arrays correctly.

btrfsck /dev/md127 gives loads of checksum errors, finally dumps core.

The file system can't be mounted with the ro,recovery options.

I started a dry run of btrfs restore and it outputs file names that would be restored but I don't have a big enough target HDD to complete a restore this way.

Do you know of a way that would overcome the checksum errors and let me mount the FS?

I feel your pain

I had exactly the same issues about a year ago with this error. I knew nothing  about linux but  in getting a solution that worked for me  had a crash course

To attempt to translate into common speak with a degree of licence for the simplistic explanation.

The actual volume presented is in fact as you have been found from a number of discs and also a number of separate volumes on the discs.

Also when you expand via X-Raid another volume is create with the free space.

The operatiing system then "meshes", hooks, these separate volumes together into  what is presented as "One Volume"

So what the error message is saying is "that it cannot find the entry/exit points on these separate storage volumes to  generate the one large volume."

The checksum errors are shown when trying to read across the boundaries but with no way of undertstanding where the data is in the volume.  (or something like that)

I delved deep into this and found a range of articles on how to restore  a btfrs volume After 4 weeks of tinkering I gave up.  I contacted a number of commercial companies and was told on more than one occasion that "recovery  cannot be done who would use btfrs in a commercial system anyway.!!"  Others gave a very high initial charge plus  an even higher charge to recover if it was deemed viable.

Anyway I fell back on an old colleague of mine who was well versed in linux and actually he put me on to a "Windows" solution  One was ReclaiMe and the other was UFS Explorer Pro  I tried the trials and found UFS Explorer  saw all my data.  In fact my friend had a colleague who had a copy so I sent my discs to him and he recovered my data.

Job done Then  I came across this article. A similar problem (different NAS supplier) but actually the same underlying explanation that I found out by my amateur investigations and analysis.  Google " anandtech nas recovery"

As for linux  before getting to my solution I tried a lot of Linux programs I would hestitate to say all but tried most of the "forensic tools"  BTRFS support is poor.  What stopped me from a factory reset was when I used PHOTOREC this recovered a multude of  "sections" of my data.  This indicated that my data still existed it was getting it off that was the issue   There is still a lack of Linux recovery programs   when  proprietary expanasion modes are used even though the basic linux commands create these volumes.

Anyway try the Windows solutions  They worked in my case

---

davexyz wrote:

 

As for linux  before getting to my solution I tried a lot of Linux programs I would hestitate to say all but tried most of the "forensic tools"  BTRFS support is poor.  What stopped me from a factory reset was when I used PHOTOREC this recovered a multude of  "sections" of my data.  This indicated that my data still existed it was getting it off that was the issue   There is still a lack of Linux recovery programs   when  proprietary expanasion modes are used even though the basic linux commands create these volumes. 

---

X-RAID uses mdadm for the RAID. This is Linux software RAID that has been used for several years. mdadm is a free Linux tool and the tools for BTRFS are also free.

We automate expansion to make it easy for you, but we use standard RAID levels. Data can be recovered using an ordinary x86 Linux PC.

The tools for Linux may take some getting used to, but as edkedk discovered advanced users can generally figure out how to do what they want. Trying to do it yourself is at your own risk. If you make mistakes in some cases you can make the problem worse.

We do sell data recovery services. Our data recovery services are software based only (we don't do physical repairs to disks) and are attempts that may be unsuccessful. Sometimes data recovery can be straightforward, sometimes there are some more complicated cases where our team may do things such as analyse metadata dumps to determine what can be done.

Once you've got the RAID started you would then use the standard tools for BTRFS recovery.

I think the data recovery tools on Linux for BTRFS are pretty good and much better than what was around for EXT4.

It is still important to note that data recovery should not be relied upon. Backups are important. No important data should be stored on just the one device.

A solution to this error has been "requested in a number of threads.  This is the only time a user has posted a solution As my post in this thread stated I had this over a year ago and got nowhere.

So I am ecstatic that a solution has been posted.

I also stated that I am a complete novice when it comes to linux. As stated in the prior post "advanced users can get a solution"  My issue/problem whatever you wish to call it  is about when I things go wrong we are back too

"have you got a backup" etc. 

The basic user has shelled out for a NAS  and a significant amount of storage and in the main has a turn-key sutuation.  When things go wrong  then the issue of no backup occurs, so once bitten, for example, buy the same system again to use as a backup.  However this does not guarantee safety as  if it is two Netgear NAS' the same this could happen on the Main and the backup, what are the chances?.  So shell out for multiple usb fallback devices and so on.  Yes there is also the recovery option at what cost?.

I know that it is the users reponsibility as to the value of the data but just trying to put this into the context of the basic user not the IT professional or Linux expert.

And yes BTRFS recovery tools in what ever "flavour" are now becoming available as the commercial useage increases

Let's start by saying this isn't really a NAS issue.  No matter where you keep your data, there's a chance you will lose it due to failures.  The data loss I've experienced myself predates my NAS ownership. Generally my loss stories started with a PC or workstation drive failure, followed by a failure of backup media (some optical disks that couldn't be read, some USB drives also - and once my employer's enterprise backup tapes).

Even for the advanced user, backups are much better than attempting data recovery.  When I've done data recovery, I've never gotten everything back. 

My approach to backup was formed in response to my own pre-NAS data loss, and I've simply carried it forward to my NAS.  That approach is simple - keep three copies of everything on different devices (including the original).  Keep one copy offsite if you want disaster protection.  Theoretically of course I could still lose data, but for me this policy lowers the risk enough.  I have yet to lose data when I follow it.

You do need to weigh the economics, but you should also consider the inherent value of the data - for many home users, it includes personal data (videos, pictures of family, etc) that are impossible to replace.  There's a second class of data (for instance commercial music/videos) which generally are replaceable, but not easily.

There are options that I think anyone who can afford a NAS in the first place can manage:

- cloud backup is available for about $60 per year with unlimited storage (both crashplan and amazon cloud are about that price).   

- USB drives of reasonable size are about $40 per TB

Doing both would follow my "3 copy" policy, and provide a local copy which is faster to restore than the cloud. 

---

davexyz wrote:

 

 When things go wrong  then the issue of no backup occurs, so once bitten, for example, buy the same system again to use as a backup.  However this does not guarantee safety as  if it is two Netgear NAS' the same this could happen on the Main and the backup, what are the chances?.  So shell out for multiple usb fallback devices and so on.  Yes there is also the recovery option at what cost?.

 

 

---

Another NAS is of course is more expensive than using external hard drives, but I find it more convenient. It's easy to automate, and easy to cut over to the backup if the primary fails.  Generally the more storage you have, the more attractive NAS->NAS backup becomes.  An RN214 with 4x4TB works out to about $60 per TB (assuming you use 4 jbod volumes, not RAID).  

Note I don't work for Netgear, so I really don't care if you buy ReadyNAS or another brand (or nothing at all).  I've had multiple ReadyNAS running for some years, and I've never seen two fail at the same time.  I think the main failure scenarios for multiple-NAS are power-induced failures like a nearby lightning strike, and disaster like fire/flood/theft.  Another scenario is when the backup NAS failure either isn't noticed or warnings are ignored.  All of those will take out equipment from any vendor.

I do recommend staggering the application of firmware updates (especially if you have two devices that use the same platform - like two OS 6 ReadyNAS).