Volume on RN4220 read/only - trying to understand btrfs
Hi. One of the volumes on our RN4220 has become read/only, probably due to free space exhaustion. Simply freeing-up space doesn't seem to help, and I suspect a reboot is in our future - this has happened to us before, I think. OS is 6.5.1, but this is a production NAS, and I'd rather work around existing problems than deal with new ones caused by an upgrade.
What I'm trying to understand is what btrfs it telling me about free space, and why things are allocated the way they are (by NetGEAR?). In any case, here's the volume in question:
root@bugs:/backup/VMBackups# btrfs fi usage -T /backup
Overall:
Device size: 5.45TiB
Device allocated: 4.43TiB
Device unallocated: 1.02TiB
Device missing: 0.00B
Used: 4.21TiB
Free (estimated): 1.24TiB (min: 746.62GiB)
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 160.00MiB (used: 0.00B)
Data Metadata System
Id Path single DUP DUP Unallocated
-- ---------------- ------- --------- --------- -----------
1 /dev/md/backup-0 4.43TiB 3.00GiB 16.00MiB 1.02TiB
-- ---------------- ------- --------- --------- -----------
Total 4.43TiB 1.50GiB 8.00MiB 1.02TiB
Used 4.21TiB 395.25MiB 640.00KiB
I guess I'm trying to understand why there would be 1TB "unallocated" by default. Other posters seem to use "btrfs fi resize" to allocate this space, but then I'm curious about the "min" value for free space (746GB, in this example). I had heard that one has to leave some part of the NAS volume free - is that it? Does this unallocated space ever get allocated automatically, or is it up to me to do this?
At least one enquiring mind wants to know...
Also, I'm not using snapshots on this volume, and the UI shows zero snapshots, but "btrfs subvolume list" does show some stuff - perhaps the default things allocated on a new volume (e.g. a .snapshot in each share).
Re: Volume on RN4220 read/only - trying to understand btrfs
Going into read-only mode is usually a last gasp for the volume to remain intact. A re-boot may well kill it completely, so that's not your best next step. It is highly unlikely that you can fix the problem permanently short of destroying and re-creating the volume. So, now is the time to make sure your backup is complete (and can be restored quickly, so probably not just on a cloud based system) and make plans for how and when you are going to do that on a production system. You can try a re-boot then and see if it fixes anything, but it's unlikely.
Once you get this sorted out, you should take a good look at your backup system. A second NAS that mirrors the main one would make it easy to simpy swap it in. While Netgear has not implemented anything that makes that especially easy, it can be done. I even have almost that at home, though I have two smaller NAS that each mirror part of the main one. I just use periodic rsync jobs, but for somethingt that changes more often, lsyncd (https://axkibe.github.io/lsyncd/ ) would be worth looking into.
Re: Volume on RN4220 read/only - trying to understand btrfs
Upon further digging into online info on btrfs, "unallocated" is an unfortunate choice of terminology, particularly when btrfs reports the volume is full. btrfs seems half-baked at best, and it's too bad NetGEAR chose to use it in its infancy.
So, with zero information in any logs I can find about why this volume went R/O, and apparently lots of so-called free space, what am I left to conclude here?
root@bugs:/backup# mkdir foo
mkdir: cannot create directory `foo': No space left on device
I do apprecate the response, but in the end, it doesn't help me to get any closer to solving my problem. I guess, technically, it isn't R/O, as I *can* delete things, but something has decided that, despite there being 20% or so free space, the volume is full.
root@bugs:/backup# mkdir foo
mkdir: cannot create directory `foo': No space left on device
Backup is the volume name?
Normally you don't mkdir in the volume root (instead you create shares via the web ui). Each share is a btrfs subvolume (not the same as an ordinary folder).
Re: Volume on RN4220 read/only - trying to understand btrfs
"Unallocated" is precisely what that space is -- not yet allocated to either data or metadata. It would be allocated as needed as you add files if the volume had not gone read-only. In other words, you are not out of space and should not be trying to do anything regarding that. Even if you were out of space, you can't do anything about it with the volume read-only.
The message you are getting when trying to create a new folder is because it cannot allocate any space for it. It's incorrectly telling you that it's because the volume is full -- it's because the volume is read-only. As far as what you are deleting, I suspect you aren't -- if you were to reboot (which, as I said previously, you should not attempt until you are sure you have a complete backup), I think you'll find the files are still there.
I had a volume do the same thing. In my case, I knew why. It was in an EDA500 eSATA expansion chassis that had the connection come loose. I Googled everything I could about making it read/write again, and nothing I did survived a reboot. Most of the time, it went back to read-only as soon as I tried to add a file. Ultimately, it was dead on a reboot, forcing my had to delete and re-create it. Since the cause of your volume going read-only is obviously different (no external chassis), maybe you'll have better luck. But I rather doubt it.
As for BTRFS being a poor choice, a lot of file systems would, at this point, likely be unavailable. Instead, yours is read-only so you can at least make a backup. I'm not sure why you think that's a bad thing.
Re: Volume on RN4220 read/only - trying to understand btrfs
6.5.1 is quite ancient and thus running an old btrfs version. Likely your system is unbalanced and in need of a balance, have you been running balance on schedule? Do you frequently add a lot of data and then delete it?
If you can delete data you may be able to run balance, but IIRC balance on 6.5.1 runs with full chunks by default so you might need to do it over CLI with -dusage and -musage values set lower.
I agree that Netgear used BTRFS way before it was mature enough. Heck, the oldest RN firmwares uses a release candidate of BRTFS (basically beta) 🙂
That being said, it has grown into a very solid filesystem. I would use it over pretty anything else at this point, maybe besides ZFS. Keeping your NAS up-to-date is important and running a 5-6 year old firmware isn't ideal at all. Remember, the BTRFS version also gets updated as part of the firmware upgrade.
It is more likely that the filesystem ran into some specific issue, prompting the volume to go read-only. It does not look like a space issue to me. I can take a look at the logs for you, if you like.
Pull the logs from the web interface: System > Logs > Download Logs.
That will download a zip file with all the NAS logs. Upload the zip file to a Google drive, Dropbox or similar and create a sharing link from where I can download the zip file. PM me this link and I will have a look at them.
