× NETGEAR will be terminating ReadyCLOUD service by July 1st, 2023. For more details click here.
Orbi WiFi 7 RBE973
Reply

Re: btrfs_merge_bio_hook+a0 error, recovery possible?

edkedk
Tutor

btrfs_merge_bio_hook+a0 error, recovery possible?

Hi,

 

I have a ReadyNAS 204 with the btrfs_merge_bio_hook+a0 error. After reboot I can ssh into the box, but the web interface does not load and the data volume is not mounted.

 

Output of cat /proc/mdstat:

md127 : active raid5 sda3[0] sdd3[3] sdc3[2] sdb3[1]
      11706506496 blocks super 1.2 level 5, 64k chunk, algorithm 2 [4/4] [UUUU]

md1 : active raid6 sda2[0] sdd2[3] sdc2[2] sdb2[1]
      1047424 blocks super 1.2 level 6, 64k chunk, algorithm 2 [4/4] [UUUU]

md0 : active raid1 sda1[0] sdd1[3] sdc1[2] sdb1[1]
      4190208 blocks super 1.2 [4/4] [UUUU]

unused devices: <none>

Output of btrfs fi show:

Label: '1165568a:Volume1'  uuid: 56990628-d1ea-4142-8c3d-5dd32b620f00
        Total devices 1 FS bytes used 8.30TiB
        devid    1 size 10.90TiB used 8.35TiB path /dev/md127

btrfs-progs v4.1.2

It is possible that volume allocation has exceeded 85%. Didn't know it can cause errors. Is there a chance of recovering the volume or is facory reset my only option?

Model: RN204|ReadyNAS204
Message 1 of 23

Accepted Solutions
edkedk
Tutor

Re: btrfs_merge_bio_hook+a0 error, recovery possible?

Eventually I used --path-regex to restore specific directories and it worked fine. Seems that everything I wanted was copied.

View solution in original post

Message 20 of 23

All Replies
JennC
NETGEAR Employee Retired

Re: btrfs_merge_bio_hook+a0 error, recovery possible?

Hello edkedk,

 

What's the model number and FW version?

 

Welcome to the community!

 

Regards,

Message 2 of 23
mdgm-ntgr
NETGEAR Employee Retired

Re: btrfs_merge_bio_hook+a0 error, recovery possible?

Can you try booting your NAS into volume read-only mode (a boot menu option) and see if the data is then accessible?

Message 3 of 23
edkedk
Tutor

Re: btrfs_merge_bio_hook+a0 error, recovery possible?

HI,

 

Model no. is RN204, fw version 6.4.2

 

Booting into read-only mode results in btrfs_merge_bio_hook+a0 being displayed and the data volume not mounted.

 

If I move the disks to a PC running Linux, will I be able to mount the volume?

Message 4 of 23
mdgm-ntgr
NETGEAR Employee Retired

Re: btrfs_merge_bio_hook+a0 error, recovery possible?

Shell commands would have to be used to try and mount it or attempt to recover the data and these could just as easily be done via SSH on the RN204. However attempting to do that would be at your own risk.

 

Have you checked the smart stats of the disks for SMART errors?

Message 5 of 23
edkedk
Tutor

Re: btrfs_merge_bio_hook+a0 error, recovery possible?

I just checked smart values using smartctl and see no error on any disk.

 

I'd like to try to manually mount the volume. Could you assist me with the required shell commands?

Message 6 of 23
Hopchen
Prodigy

Re: btrfs_merge_bio_hook+a0 error, recovery possible?

Hi,

 

As mdgm said, running commands from the terminal can be "dangerous" if you are not experienced/comfortable with Linux and BTRFS. So, just be careful 🙂

 

I think you should wait mounting the volume. First check that it is not something benign (like a 100% full root/OS). Post the output of the command:
df -h

 

If the OS is not full, then I would suggest you do a read-only filesystem check on the data volume before attempting the mount. You can do this by running the command:
btrfsck /dev/md127

 

That might take some time. Please post the result of the FS check. The above command will not try to repair anything, it will just give us info about the state of the filesystem and the command can only be run on a non-mounted filesystem (so don't attempt to mount first).


Thanks

Message 7 of 23
edkedk
Tutor

Re: btrfs_merge_bio_hook+a0 error, recovery possible?

I have some Linux experience but none with btrfs.

 

# df -h
Filesystem      Size  Used Avail Use% Mounted on
tmpfs            10M  4.0K   10M   1% /dev
/dev/md0        3.7G  923M  2.6G  26% /
tmpfs          1011M     0 1011M   0% /dev/shm
tmpfs          1011M  500K 1011M   1% /run
tmpfs          1011M     0 1011M   0% /sys/fs/cgroup
tmpfs          1011M     0 1011M   0% /media
# btrfsck /dev/md127
Couldn't open file system

 

Message 8 of 23
Hopchen
Prodigy

Re: btrfs_merge_bio_hook+a0 error, recovery possible?

And the RAID is definitely still running?

 

cat /proc/mdstat

Message 9 of 23
edkedk
Tutor

Re: btrfs_merge_bio_hook+a0 error, recovery possible?

It seems to be...

 

# cat /proc/mdstat
Personalities : [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md127 : active raid5 sda3[0] sdd3[3] sdc3[2] sdb3[1]
      11706506496 blocks super 1.2 level 5, 64k chunk, algorithm 2 [4/4] [UUUU]

md1 : active raid6 sda2[0] sdd2[3] sdc2[2] sdb2[1]
      1047424 blocks super 1.2 level 6, 64k chunk, algorithm 2 [4/4] [UUUU]

md0 : active raid1 sda1[0] sdd1[3] sdc1[2] sdb1[1]
      4190208 blocks super 1.2 [4/4] [UUUU]

unused devices: <none>
# btrfs fi sho
Label: '1165568a:Volume1'  uuid: 56990628-d1ea-4142-8c3d-5dd32b620f00
        Total devices 1 FS bytes used 8.30TiB
        devid    1 size 10.90TiB used 8.35TiB path /dev/md127

btrfs-progs v4.1.2

 

Message 10 of 23
Hopchen
Prodigy

Re: btrfs_merge_bio_hook+a0 error, recovery possible?

Okay, that is a problem then.

 

Do one thing. To be sure, scan for btrfs volumes first and then try the FS check again.

 

First:

btrfs device scan

 

Then:

btrfsck /dev/md127

 

Does it yield the same result?

Do you have backups of the data? It could be FS issues here.

 

Can you post output of the this as well:

dmesg | tail -n30

Message 11 of 23
edkedk
Tutor

Re: btrfs_merge_bio_hook+a0 error, recovery possible?

# btrfs device scan
Scanning for Btrfs filesystems
# btrfsck /dev/md127
Couldn't open file system
# dmesg | tail -n30
[<c02dfe9c>] (submit_extent_page) from [<c02e1694>] (__do_readpage+0x31c/0x8f0)
[<c02e1694>] (__do_readpage) from [<c02e1d38>] (__extent_read_full_page+0xd0/0xd8)
[<c02e1d38>] (__extent_read_full_page) from [<c02e41b0>] (read_extent_buffer_pages+0x1ec/0x2f8)
[<c02e41b0>] (read_extent_buffer_pages) from [<c02b6bc4>] (btree_read_extent_buffer_pages.constprop.5+0xb8/0x124)
[<c02b6bc4>] (btree_read_extent_buffer_pages.constprop.5) from [<c02b71c0>] (read_tree_block+0x24/0x44)
[<c02b71c0>] (read_tree_block) from [<c02b8538>] (btrfs_read_tree_root+0xc8/0x118)
[<c02b8538>] (btrfs_read_tree_root) from [<c02bb204>] (open_ctree+0x1474/0x2190)
[<c02bb204>] (open_ctree) from [<c0292204>] (btrfs_mount+0x88c/0xa74)
[<c0292204>] (btrfs_mount) from [<c00df854>] (mount_fs+0x44/0x154)
[<c00df854>] (mount_fs) from [<c00f8590>] (vfs_kern_mount+0x4c/0xfc)
[<c00f8590>] (vfs_kern_mount) from [<c0291c14>] (btrfs_mount+0x29c/0xa74)
[<c0291c14>] (btrfs_mount) from [<c00df854>] (mount_fs+0x44/0x154)
[<c00df854>] (mount_fs) from [<c00f8590>] (vfs_kern_mount+0x4c/0xfc)
[<c00f8590>] (vfs_kern_mount) from [<c00fac84>] (do_mount+0x1c4/0xbe8)
[<c00fac84>] (do_mount) from [<c00fb9ec>] (SyS_mount+0x74/0xa0)
[<c00fb9ec>] (SyS_mount) from [<c000f400>] (ret_fast_syscall+0x0/0x3c)
Code: 01520004 23a00000 33a00001 eaffffe4 (e7f001f2)
---[ end trace dd1049a4dc0f8ed7 ]---
al_eth 0000:00:01.0 eth0: al_eth_down
[eth rx] warn: dma state didn't change to Disable
[eth rx] warn: failed to change state, error -110
configured MAC to RGMII mode:
al_eth 0000:00:01.0 eth0: using MSI-X per Queue interrupt mode
libphy: al mdio bus: probed
al_eth 0000:00:01.0 eth0: phy[4]: device 8:04, driver Atheros 8035 ethernet
al_eth 0000:00:01.0 eth0: phy[4]:supported 2ef adv 2ef
IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
eth [al_eth_0]: set link speed to 1000Mbps. full duplex.
al_eth 0000:00:01.0 eth0: Link is Up - 1Gbps/Full - flow control off
IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

This is a NAS that stores backups, so no catastrophe if the data is lost, but it would be nice to be able to copy some of it.

Message 12 of 23
Hopchen
Prodigy

Re: btrfs_merge_bio_hook+a0 error, recovery possible?

Cheers for that.

 

Yes, that call trace shows definite issues with the filesystem. Looks very much like FS corruption. I do not think there is any way you can mount the data here. These things can happen - which is why backups are important.

 

You should consider one of two options (as you have backups available):

 

1. Reset NAS. It will erase everything, but also give you a fresh filesystem. Restore from backups afterwards.

 

2. Contact Netgear support . Level 3 support in Netgear might be able to do something here, but it would be against payment for a data recovery contract - most likely. I think it is like €150-€170 euro or something like that. But even in that case, they might not be able to solve it. I don't know the true damage/state of the filesystem. But, it does not look good 😞

Message 13 of 23
davexyz
Aspirant

Re: btrfs_merge_bio_hook+a0 error, recovery possible?

This can be recovered (data recovery is dependent on circumstances) using 3rd party software and a Windows or linux pc.  Transferring the discs to the PC or an esata box and the software reconstructs the volume fo to be trnsferred of the volume onto your recovery drives

 

 A search on this forum should indicate the software that could be used.

Message 14 of 23
edkedk
Tutor

Re: btrfs_merge_bio_hook+a0 error, recovery possible?

Could you link a thread that mentions possible recovery software?

Message 15 of 23
davexyz
Aspirant

Re: btrfs_merge_bio_hook+a0 error, recovery possible?

Message 16 of 23
edkedk
Tutor

Re: btrfs_merge_bio_hook+a0 error, recovery possible?

Thank you, I'll check that.

Message 17 of 23
edkedk
Tutor

Re: btrfs_merge_bio_hook+a0 error, recovery possible?

I moved the HDDs to a Linux (Ubuntu) PC.

 

/proc/mdstat shows the arrays correctly.

btrfsck /dev/md127 gives loads of checksum errors, finally dumps core.

The file system can't be mounted with the ro,recovery options.

 

I started a dry run of btrfs restore and it outputs file names that would be restored but I don't have a big enough target HDD to complete a restore this way.

 

Do you know of a way that would overcome the checksum errors and let me mount the FS?

 

Message 18 of 23
davexyz
Aspirant

Re: btrfs_merge_bio_hook+a0 error, recovery possible?

I feel your pain

 

I had exactly the same issues about a year ago with this error. I knew nothing  about linux but  in getting a solution that worked for me  had a crash course

 

To attempt to translate into common speak with a degree of licence for the simplistic explanation.

 

The actual volume presented is in fact as you have been found from a number of discs and also a number of separate volumes on the discs.

 

Also when you expand via X-Raid another volume is create with the free space.

 

The operatiing system then "meshes", hooks, these separate volumes together into  what is presented as "One Volume"

 

So what the error message is saying is "that it cannot find the entry/exit points on these separate storage volumes to  generate the one large volume."

 

The checksum errors are shown when trying to read across the boundaries but with no way of undertstanding where the data is in the volume.  (or something like that)

 

I delved deep into this and found a range of articles on how to restore  a btfrs volume After 4 weeks of tinkering I gave up.  I contacted a number of commercial companies and was told on more than one occasion that "recovery  cannot be done who would use btfrs in a commercial system anyway.!!"  Others gave a very high initial charge plus  an even higher charge to recover if it was deemed viable.

 

Anyway I fell back on an old colleague of mine who was well versed in linux and actually he put me on to a "Windows" solution  One was ReclaiMe and the other was UFS Explorer Pro  I tried the trials and found UFS Explorer  saw all my data.  In fact my friend had a colleague who had a copy so I sent my discs to him and he recovered my data.

 

Job done Then  I came across this article. A similar problem (different NAS supplier) but actually the same underlying explanation that I found out by my amateur investigations and analysis.  Google " anandtech nas recovery"

 

As for linux  before getting to my solution I tried a lot of Linux programs I would hestitate to say all but tried most of the "forensic tools"  BTRFS support is poor.  What stopped me from a factory reset was when I used PHOTOREC this recovered a multude of  "sections" of my data.  This indicated that my data still existed it was getting it off that was the issue   There is still a lack of Linux recovery programs   when  proprietary expanasion modes are used even though the basic linux commands create these volumes.

 

Anyway try the Windows solutions  They worked in my case

 

 

 

 

Message 19 of 23
edkedk
Tutor

Re: btrfs_merge_bio_hook+a0 error, recovery possible?

Eventually I used --path-regex to restore specific directories and it worked fine. Seems that everything I wanted was copied.

Message 20 of 23
mdgm-ntgr
NETGEAR Employee Retired

Re: btrfs_merge_bio_hook+a0 error, recovery possible?


@davexyz wrote:

 

As for linux  before getting to my solution I tried a lot of Linux programs I would hestitate to say all but tried most of the "forensic tools"  BTRFS support is poor.  What stopped me from a factory reset was when I used PHOTOREC this recovered a multude of  "sections" of my data.  This indicated that my data still existed it was getting it off that was the issue   There is still a lack of Linux recovery programs   when  proprietary expanasion modes are used even though the basic linux commands create these volumes. 


X-RAID uses mdadm for the RAID. This is Linux software RAID that has been used for several years. mdadm is a free Linux tool and the tools for BTRFS are also free.

We automate expansion to make it easy for you, but we use standard RAID levels. Data can be recovered using an ordinary x86 Linux PC.

The tools for Linux may take some getting used to, but as edkedk discovered advanced users can generally figure out how to do what they want. Trying to do it yourself is at your own risk. If you make mistakes in some cases you can make the problem worse.

We do sell data recovery services. Our data recovery services are software based only (we don't do physical repairs to disks) and are attempts that may be unsuccessful. Sometimes data recovery can be straightforward, sometimes there are some more complicated cases where our team may do things such as analyse metadata dumps to determine what can be done.

 

Once you've got the RAID started you would then use the standard tools for BTRFS recovery.

I think the data recovery tools on Linux for BTRFS are pretty good and much better than what was around for EXT4.

 

It is still important to note that data recovery should not be relied upon. Backups are important. No important data should be stored on just the one device.

Message 21 of 23
davexyz
Aspirant

Re: btrfs_merge_bio_hook+a0 error, recovery possible?

A solution to this error has been "requested in a number of threads.  This is the only time a user has posted a solution As my post in this thread stated I had this over a year ago and got nowhere.

 

So I am ecstatic that a solution has been posted.

 

I also stated that I am a complete novice when it comes to linux. As stated in the prior post "advanced users can get a solution"  My issue/problem whatever you wish to call it  is about when I things go wrong we are back too

 

"have you got a backup" etc. 

 

The basic user has shelled out for a NAS  and a significant amount of storage and in the main has a turn-key sutuation.  When things go wrong  then the issue of no backup occurs, so once bitten, for example, buy the same system again to use as a backup.  However this does not guarantee safety as  if it is two Netgear NAS' the same this could happen on the Main and the backup, what are the chances?.  So shell out for multiple usb fallback devices and so on.  Yes there is also the recovery option at what cost?.

 

I know that it is the users reponsibility as to the value of the data but just trying to put this into the context of the basic user not the IT professional or Linux expert.

 

And yes BTRFS recovery tools in what ever "flavour" are now becoming available as the commercial useage increases

Message 22 of 23
StephenB
Guru

Re: btrfs_merge_bio_hook+a0 error, recovery possible?

Let's start by saying this isn't really a NAS issue.  No matter where you keep your data, there's a chance you will lose it due to failures.  The data loss I've experienced myself predates my NAS ownership. Generally my loss stories started with a PC or workstation drive failure, followed by a failure of backup media (some optical disks that couldn't be read, some USB drives also - and once my employer's enterprise backup tapes).

 

Even for the advanced user, backups are much better than attempting data recovery.  When I've done data recovery, I've never gotten everything back. 

 

 

My approach to backup was formed in response to my own pre-NAS data loss, and I've simply carried it forward to my NAS.  That approach is simple - keep three copies of everything on different devices (including the original).  Keep one copy offsite if you want disaster protection.  Theoretically of course I could still lose data, but for me this policy lowers the risk enough.  I have yet to lose data when I follow it.

 

You do need to weigh the economics, but you should also consider the inherent value of the data - for many home users, it includes personal data (videos, pictures of family, etc) that are impossible to replace.  There's a second class of data (for instance commercial music/videos) which generally are replaceable, but not easily.

 

There are options that I think anyone who can afford a NAS in the first place can manage:

- cloud backup is available for about $60 per year with unlimited storage (both crashplan and amazon cloud are about that price).   

- USB drives of reasonable size are about $40 per TB

 

Doing both would follow my "3 copy" policy, and provide a local copy which is faster to restore than the cloud. 

 

 


@davexyz wrote:

 

 When things go wrong  then the issue of no backup occurs, so once bitten, for example, buy the same system again to use as a backup.  However this does not guarantee safety as  if it is two Netgear NAS' the same this could happen on the Main and the backup, what are the chances?.  So shell out for multiple usb fallback devices and so on.  Yes there is also the recovery option at what cost?.

 

 


 

Another NAS is of course is more expensive than using external hard drives, but I find it more convenient. It's easy to automate, and easy to cut over to the backup if the primary fails.  Generally the more storage you have, the more attractive NAS->NAS backup becomes.  An RN214 with 4x4TB works out to about $60 per TB (assuming you use 4 jbod volumes, not RAID).  

 

Note I don't work for Netgear, so I really don't care if you buy ReadyNAS or another brand (or nothing at all).  I've had multiple ReadyNAS running for some years, and I've never seen two fail at the same time.  I think the main failure scenarios for multiple-NAS are power-induced failures like a nearby lightning strike, and disaster like fire/flood/theft.  Another scenario is when the backup NAS failure either isn't noticed or warnings are ignored.  All of those will take out equipment from any vendor.

 

I do recommend staggering the application of firmware updates (especially if you have two devices that use the same platform - like two OS 6 ReadyNAS).

 

 

 

Message 23 of 23
Top Contributors
Discussion stats
  • 22 replies
  • 4075 views
  • 1 kudo
  • 6 in conversation
Announcements