The volume data encountered an error and was made read-only. It is recommended to backup your data.

[as_]

Was copying data onto my NAS from external hard drives for backup purposes (just had an external drive fail.. so.. I'm a bit paranoid on that right now..). All of a sudden, I couldn't transfer files anymore. Windows said the disk was "write protected" (have the NAS mapped to a drive letter).

Looked at the admin page of the NAS, and it shows it as "healthy". The logs tell a different story:

Nov 08, 2021 03:02:30 PM

Volume: The volume data encountered an error and was made read-only. It is recommended to backup your data.Nov 08, 2021 04:37:10 AM

Volume: Less than 5% of volume data's capacity is free. data's performance is degraded and you risk running out of usable space. To improve performance and stability, you must add capacity or make free space.Nov 07, 2021 11:57:49 PM

Volume: Less than 10% of volume data's capacity is free. data's performance is degraded and you risk running out of usable space. To improve performance and stability, you must add capacity or make free space.Oct 29, 2021 04:27:43 PM

Volume: Less than 20% of volume data's capacity is free. Performance on volume data will degrade if additional capacity is consumed. NETGEAR recommends that you add capacity to avoid performance degradation.

So.. now what?

I'll note I still have 1TB of storage space remaining (at least, that's what's being reported, in Windows and on the admin page..).. but some of the files I thought I had transferred over there appear to have disappeared (specifically, as far as I can tell, the files I transferred towards the end..)

[StephenB]

---

@as_ wrote:

The logs tell a different story:

Nov 08, 2021 03:02:30 PM

Volume: The volume data encountered an error and was made read-only. It is recommended to backup your data.Nov 08, 2021 04:37:10 AM

Volume: Less than 5% of volume data's capacity is free. data's performance is degraded and you risk running out of usable space. To improve performance and stability, you must add capacity or make free space.Nov 07, 2021 11:57:49 PM

So.. now what?

---

Have you backed up your data?

[as_]

No, I have not.  Most of the data on the NAS is data I have stored on other local drives.. but not all of it.

[StephenB]

---

@as_ wrote:

No, I have not.  Most of the data on the NAS is data I have stored on other local drives.. but not all of it.

---

Data loss is very possible (and reformatting/starting over is in fact the simplest way to proceed).  So it'd be good to back up everything you care about first.

[as_]

I'll start there with the backup.. but that said, we're talking about 20TB of data (4x8TB drives).  Formatting the drive and starting over is absolutely the last thing I want to do.

[as_]

Digging a little deeper (figured out how to download the detailed log files).. I see the following in kernel.log (repeated several times for each message):

kernel: BTRFS error (device md127): bad tree block start 14701692634252692971 23322242252800
kernel: BTRFS error (device md127): bad tree block start 18115770242137751560 23322241466368
kernel: BTRFS error (device md127): parent transid verify failed on 23363584 wanted 3050091 found 3050089

[rn_enthusiast]

The filesystem is damaged here, which you have found out. That is the reason for the NAS remonting to prevent any further corruption.

Are you OK to send me the logs? I wouldn't mind poking around in them and see where we go from here.

When you access the NAS Admin web page, go to: System > Logs and here you will see a button called "Download Logs" on the right-hand side. Click this and it will download a zip file with all the NAS logs inside.

Take this zip file and upload to your Dropbox, Google Drive or similar and then make a link where I can download the log zip file. PM me this link please. Then I will have a look to see what is going on.

Cheers

[as_]

So.. I completed the backup.

Now what?

[rn_enthusiast]

Sorry @as_ 

IRL caught up with me, so didn't get a chance to look at this earlier.

The BTRFS filesystem is reporting you being off with just 2 write transactions: 3050091 found 3050089
There is an argument that you could clear the filesystem journal which will likely help the parent transid verify failed errors.

[Thu Nov 11 18:34:14 2021] BTRFS error (device md127): parent transid verify failed on 23363584 wanted 3050091 found 3050089
[Thu Nov 11 18:36:14 2021] BTRFS error (device md127): parent transid verify failed on 23363584 wanted 3050091 found 3050089
[Thu Nov 11 18:36:14 2021] BTRFS error (device md127): parent transid verify failed on 23363584 wanted 3050091 found 3050089

I can 't see in the kernel logs what potentially caused this, because the logs are rolled over with the BTRFS error spam.

I am more concerned about these messages here:

[Thu Nov 11 18:36:34 2021] BTRFS error (device md127): bad tree block start 18115770242137751560 23322241466368
[Thu Nov 11 18:36:34 2021] BTRFS error (device md127): bad tree block start 18115770242137751560 23322241466368
[Thu Nov 11 18:36:34 2021] BTRFS error (device md127): bad tree block start 18115770242137751560 23322241466368
[Thu Nov 11 18:36:34 2021] BTRFS error (device md127): bad tree block start 18115770242137751560 23322241466368
[Thu Nov 11 18:36:35 2021] BTRFS error (device md127): bad tree block start 18115770242137751560 23322241466368
[Thu Nov 11 18:36:35 2021] BTRFS error (device md127): bad tree block start 18115770242137751560 23322241466368
[Thu Nov 11 18:36:35 2021] BTRFS error (device md127): bad tree block start 18115770242137751560 23322241466368
[Thu Nov 11 18:36:35 2021] BTRFS error (device md127): bad tree block start 18115770242137751560 23322241466368
[Thu Nov 11 18:36:35 2021] BTRFS error (device md127): bad tree block start 18115770242137751560 23322241466368
[Thu Nov 11 18:36:35 2021] BTRFS error (device md127): bad tree block start 18115770242137751560 23322241466368
[Thu Nov 11 18:37:08 2021] btree_readpage_end_io_hook: 10 callbacks suppressed

From what I can read, most people end up destroying and rebuilding their BTRFS volume from backups. But seen as you have a backup now, it is probably worth try clearing the FS journal log (you would do this over SSH with the root user).

btrfs rescue zero-log /dev/md127

However, the results of this are hard to predict and if it will actually help. One reason for the FS complaining could be that the volume is very full (95%). BTRFS tends to handle low free space, poorly.

Total devices 1 FS bytes used 20.77TiB
devid 1 size 21.82TiB used 20.78TiB path /dev/md127

There is of course also the BTRFS mailing list (https://btrfs.wiki.kernel.org/index.php/Btrfs_mailing_list ) which might help you but realistically, you could be on a position where you need to reset the system and start over. If you feel adventurous give zero-log a go (followed by NAS reboot) and you could even do a BTRFS fscheck with repair flag, but this can make things a worse as well. Should be used with caution. I am not even sure if the NAS will mount the volume post boot so it is good that you have backup now.

[as_]

Fiddled with it a bit, and figured out how to ssh as root.  Oof madone, I really need to RTFM (but that's another story..)

I think I know what to do here, but I just want to make sure..  when I run the command:

btrfs rescue zero-log /dev/md127

I get an error:

ERROR: /dev/md127 is currently mounted

I'm thinking I should do:

umount /dev/md127

but, I'm worried about doing something irreversably bad.

[StephenB]

umount won't work if the device is busy (has any open files) - so that command might fail.  It's worth a try though.  Make sure you aren't in a folder of the data volume when you try it - that will definitely fail.

The other option is to boot up the NAS in tech support mode, and access the linux command line with telnet.  We can give you instructions on how to do that if you can't get the umount to work.

[as_]

So.. the umount command seems to complete without error (no output to the terminal at all).. but doesn't appear to actually do anything.. since the btrfs rescue command then fails in the exact same way..

[StephenB]

---

@as_ wrote:

So.. the umount command seems to complete without error (no output to the terminal at all).. but doesn't appear to actually do anything.. since the btrfs rescue command then fails in the exact same way..

---

Have you tried looking at the volume with the ls command after you ran umount?

[as_]

ls -l /dev:

(truncated output)
brw-rw---- 1 root disk      9, 127 Mar 23  2021 md127
(truncated output)

cd /dev/md127:

-bash: cd: md127: Not a directory

[StephenB]

You aren't looking at the data volume, you are looking at the device.

The default data volume name is data.  So ls -als /data

[as_]

Ah.  That's one of those things I don't really understand about *nix systems in general..

So /data cursory glance looks like my files are still accessible by that path.

But now.. and I don't know whether this was because of something I did or not..

:~#umount /dev/md127
umount: /apps: target is busy
(In some cases useful info about processes that
use the device is found by lsof(8) or fuser(1).)

[StephenB]

---

@as_ wrote:

Ah.  That's one of those things I don't really understand about *nix systems in general..

---

sda, sdb, ... (/dev/sda, ...) are the physical disks

md127 is the RAID array (which is a virtual disk)

data is the file system that is on the virtual disk.

umount will accept either the device (/dev/md127) or the file system - it's a bit unusual in that regard.

---

@as_ wrote:

So /data cursory glance looks like my files are still accessible by that path.

But now.. and I don't know whether this was because of something I did or not..

:~#umount /dev/md127
umount: /apps: target is busy
(In some cases useful info about processes that
use the device is found by lsof(8) or fuser(1).)

---

/apps is actually a linux mount point - the folder on the data volume (/data/.apps).  

Try umount /data Then check if you can see folders in /data and /apps.

If that doesn't work you'll need to reboot in tech support mode.

[as_]

:~# umount /data
umount: /data  target is busy
        (In some cases useful info about processes that
         use the device is found by lsof(8) or fuser(1).)
:~# fuser -u /data
/data                21509c(root) 21520c(root)
:~# ps -ef | grep 215
root     21509  2009  0 Nov23 ?        00:00:01 /usr/sbin/smbd --foreground --no-process-group
root     21520  2009  0 Nov23 ?        00:00:00 /usr/sbin/smbd --foreground --no-process-group

Would shutting off the SMB service help here?  Or is that a bad idea?

[StephenB]

---

@as_ wrote:
Would shutting off the SMB service help here?  Or is that a bad idea?

---

It might, but likely there are other processes running that will also get in the way.  I think it's best to use tech support mode. 

You do that using the instructions on page 28 here: https://www.downloads.netgear.com/files/GDC/RNDU2000/ReadyNAS_Ultra_UltraPlus_NVX_Pro_HW_en_8July11....

You'll need to access the linux CLI using telnet instead of ssh.  If you use Windows, then one useful tool for that is putty (which can be downloaded from here: https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html )

The login for tech support mode is root, the password is infr8ntdebug

Once in, enter this:

# rnutil chroot
# btrfs device scan

At this point the data volume is unmounted, so you can try your btrfs commands.

If you want to try mount the volume in tech support mode later on, then enter this:

# mount /dev/md127 /data

This is read/write.  If it fails, you could confirm that you can still boot up read-only with

# mount -o ro /dev/md127 /data

When done you can enter

# sync

to make sure everything is written to the disks and then reboot.