I just wanted to confirm this is the correct action as we also have a RN316 with a similar set up where we have pretty much filled it up:
Firmware: 6.10.4
6 x 4TB IronWolf (ST12000VN0007) HDDs
X-RAID (RAID-5)
Admin Page > System > Volume is showing:
data 18.17TB
Free Space: 4.86GB
No snapshots have been configured for any of the shares
Admin Page > Shares confirm there are no snapshots
Files have only every been copied (appended) to the NAS
So there should be no fragmentation
So only Scrubs and Disk Tests have ever been scheduled
No Balance operations have ever been run as the theory was there is no need
So same set up as RN428, but less free space.
So should I do a Balance if the Scrub does not solve my problem (which is what I expect)?
I suppose I am surprised that we over 700GB of wasted "chunks", etc on an RN428 where we were only exclusively copying/appending large files (4GB - 100GB),
Are there any downsides to a Balance? It should not consume extra space or introduce fragmentation, right? Obvisouly it will take a very long time give we have 7 x 12TB worth of data,,,
The ReadyNAS file system not only stores the user data, it also stores additional information related to that data, called metadata. Either data is stored in a number of fixed size storage areas called chunks which are reserved by the file system when new data is stored. In normal operation a chunk reserved for user data cannot be converted to hold metadata, even when the chunk becomes empty again. Therefore, a progression from a high number of large files to a higher number of small files stored on the ReadyNAS can lead to a shortage of available metadata chunks. This can lead to a situation where the ReadyNAS device will report that is has run out of disc space even when enough space seems to be available. Running the balance function will combine half-filled chunks and free empty, previously assigned chunks for reuse with either user data or metadata.
Should I use balancing regularly?
Since balancing requires some storage space to move files it should not be postponed until the available storage space is almost consumed completely. The balancing function will be most effective after a large number of files have been deleted or if there has been a transition from generally large files to smaller files. Especially the latter is hard to monitor, therefore balancing should be run at regular intervals.
How do I balance a volume?
Balancing can be configured to run automatically via the local admin page.
Model: RN428|ReadyNAS 428 – High-performance Business Data Storage - 8-Bays
We cannot comment on whether you need to balance without seeing some BTRFS statistics first 🙂
What error are you getting when trying to transfer? Volume full or? If that is the case, then a balance will likely sort it out but I suggest getting output of the commands below and paste it into the thread, please.
btrfs fi sh /data
btrfs fi df /data
Then we can see what the filesystem looks like. You will need to log into to the NAS over SSH in order to run those commands. Cheers.
Re: Cannot copy files to RN428 even though it shows that 700GB+ is free
Thanks.
I am not a SSH person, so never had to do that in past with any of the RaadyNASes.
I am a Wintel guy, please don't shoot me...
So I have enabled SSH, but when I SSH into the RN428 I get the following error:
Creating directory '/home/administrator'.
Unable to create and initialize directory '/home/administrator'.
Welcome to ReadyNASOS 6.10.4
Last login: Wed Dec 23 16:20:14 2020 from p50
Could not chdir to home directory /home/administrator: No such file or directory
(BTW, then tried by explicitly creatting a user called SSH and using that account but it to generated the same error.)
And if I try running the btrfs command I got the following:
administrator@RN428:/$ btrfs fi sh /data
-bash: btrfs: command not found
administrator@RN428:/$
So not sure if the 2 errors are related... Sorry.
What am I doing wrong?
Thanks for your patience.
PS:Is it easier for me to show you the btrfs log from the ZIP file you can generate?
PPS: Yes, getting error message that there is not enough free space, so volume full I expect. (Copying files from Windows environment.)
Re: Cannot copy files to RN428 even though it shows that 700GB+ is free
Hi again
I should have explained that the username to use, when accessing the command line over SSH, is root (not administrator). Anyhow, yes you can also just pull a fresh log-set and show me the btrfs.log file. That wil work too 🙂
So, the filesystem has assigned 76.21TiB worth of blocks to data and 79.00GiB to metadata. As you can see the filesystem has assigned more blocks to data than it really needs. approx 0.7TiB of data blocks too many. And for metadata, it is basically full. What happens when you try and write a file to the NAS, it will first allocate and write metadata but it can't because all metadata blocks are full and it cannot request new free blocks to be used for metadata as there are no free blocks left (since all the remaining blocks are assigned to data).
A balance is what you need as that will help free up blocks which can then be assigned to metadata usage. A few comments here though. Your NAS is very full in general and you are sitting at 99% used capacity. You have approx 700GB left and even though it sounds like a lot, it isn't on such a large volume. I would advise that you keep more free space in general, maybe bring it down to 3-4%.
Balancing a volume of this size will take considerable amount of time. Days or likely more. I think you should off-load some data to bring the volume down to 3-4% free space and then balance it. But this is your choice of course.
Re: Cannot copy files to RN428 even though it shows that 700GB+ is free
Thanks for that.
That makes perfect sense. I know about FAT/NTFS/RerFS file systems and nothing about BTRFS.
I am currently running a Scrub and it will take 2-3, maybe 4 days. I am well aware and used to how long such operations will take on such a large volume. That is perfectly fine. I generally set fans to run in Cool mode and then kick off a long running operation.
I would like to fill up the NAS so that is has perhaps 4GB free. And then we never write to the NAS again. As I said it is for archival / read-only purposes, after we fill it.
(That's why we wanted to buy another RN428, but it seems there is a shortage of them, so there was another post on whether the RN428 is EOL, whether there is a replacement in the works and whether Netgear is getting out of the NAS game, due to to cloud...)
Anyway... I will wait for the Scub to complete and then do the Balance. Perhaps I will remove some files as you suggest.
Thanks for your help and patience.
Out of curiosity, is there any disadvantge to a Balance? Such as fragmentations?
Re: Cannot copy files to RN428 even though it shows that 700GB+ is free
Yes, there is a shortage of ReadyNAS units and that does bring a concern to the future. I am in the planning of making my own NAS server, running Ubuntu Server 20.04 and BTRFS RAID which I will manually configure. This will also act as my Plex Server, etc. It will happen sometime in 2021 and at the point I will demote my current ReadyNAS units to simple backup stations. I, like you, share a concern for the future of the ReadyNAS line-up.
BTRFS is a filesystem that uses Copy-on-Write (CoW). By nature it will fragment the filesystem over time, especially when data changes. A balance can probably contribute to the fragmentation. You can enable auto defrag on your shares which will help over time but fragmentation is a reality of BTRFS and other CoW filesystems (like ZFS). Unless you have performance slowdowns on certain files/directories, I wouldn't worry too much about it.
One thing to mention, is that a balance had trouble running in the past when there were no free blocks/chunks on the filesystem (such as in your scenario). It is one of the reasons that BTRFS now has a Global Reserve but even then I have still seen systems that wouldn't balance due to "no space" errors when in a situation like yours. It is usually fixable with some manual balance parameters from the command line but you can try and see if the balance will play nice with you. If not, let us know.
Out of curiosity, is there any disadvantge to a Balance? Such as fragmentations?
A balance doesn't fragment any files, and I know of no disadvantages. I run them every 3 months on my own systems - if they take a long time, that is a clue that you need to run them more often.
Personally I keep at least 15% free space on my BTRFS volumes. I think 3-4% isn't really enough. Though I do have snapshots enabled, and one aspect here is that snapshot space can grow surprisingly quickly.
Personally I keep at least 15% free space on my BTRFS volumes. I think 3-4% isn't really enough. Though I do have snapshots enabled, and one aspect here is that snapshot space can grow surprisingly quickly.
IMO, you need to consider the volume's size to determine what realistic free space you should keep. OP's volume is very large (relatively speaking) and so if he/she was to keep 10% free, then that's 7.5TiB of free space, which is overkill in my opinion - especially given the use case that this is cold storage essentially.
It all depends on the use case really. If the unit is used for cold storage you can fill it more whereas a unit used for active changing data with snapshots and so on, should have more free space to play with (15% +) as you say.
That is good to know, I thought that since I ONLY every add very large files, and do NOT have any snapshots that there would be no issue with fragmentation / space allocation. That is why I thought there is absolutely no reason to schedule a Balance operation, but I was wrong. Will now Balance on other other ReadyNASes every so often.
Re: Cannot copy files to RN428 even though it shows that 700GB+ is free
Agreed,
That's the nature of percentages.
10% of 80TB is indeed a large amount of free space. Imagine if I had 8 x 16TB HDDs (ie: 112TB of raw capacity). I would waste over 11TB of space. There is no way that that amount of free space would be required.
I am sure there is better guidance around average file size (or internal filesystem chunk size). Surely file size / chunk size is really the important factor here.
If your average file size is 30-50GB for example, why would you need 700GB of free space? These NAS do not have 40 core CPUs so I am sure it will only every allocate a couple of threads to perform Balance operation???
So I have no idea about BTRFS, as I indicated, but if someone come up with a better recommendation like:
< 10TB = 7%
BETWEEN 10TB and 50TB = 3-4%
BETWEEN 50TB AND 100TB = 1-2%
> 100TB = 10-100GB
That would better serve the community. (Obviously I am making those number up.)
As we've discussed in my use case it is purely for archival / ready-only access, I am not running any VMs on it, not are there many small files like photos. And we only every have 1-2 users accessing the NAS every so often.
Re: Cannot copy files to RN428 even though it shows that 700GB+ is free
I think generalising it, is never easy. It really does depend on the use case for your NAS. If it is cold data, you can fill it more but if it is active data that changes a lot, like VMs or database work - and you are using snapshots - then you need to keep more free space to allow snapshots to do their work, etc.
I absolutely love snapshots and they are the main reason I use BTRFS (+ BRTFS has other cool features) but my data isn't very active and so my snapshots are typically small and so it is very much an archive data situation for me. I have them as part of accidental deletion protection and for backup revisions. It is about admins keeping an eye on their space utilization/trends and observing whether they need to leave more free space or not.
It really does depend on the use case for your NAS. If it is cold data, you can fill it more but if it is active data that changes a lot, like VMs or database work - and you are using snapshots - then you need to keep more free space to allow snapshots to do their work, etc.
I agree that it depends on the use case.
But still I wouldn't be comfortable about running over 99% full. There are reports out there saying that BTRFS file systems can go read-only if they get too full (not just on ReadyNAS). And I've seen cases here where balances can't recover any free space, because there are no unallocated chunks for BTRFS to work with.
BTW, there is at least one case where the ReadyNAS application will automatically create snapshots - even if you have them turned off. "Push" backup jobs (where the destination is remote) make a snapshot, and then back up from the snapshot to ensure backup coherency.
While I get that the free space will seem "wasteful" to some, not having enough can lead to needing to destroy/restore a very large volume from backup. 3%-4% might be enough for the "cold storage w/o snapshots" case - I'm not sure, since it's not a use case that applies to me. But I think if you are running that lean on free space, you do need to be mindful of the operational consequences if the system runs out.
Running these one after another and let them finish before running the next one - that MIGHT get you out of trouble. Hopefully one of these balances will "bite" and actually work.
You can always check status of a balance after you started it:
btrfs balance status /data
All of this is done over SSH with the root user, of course 🙂
Keep me posted on the progress.
And don't defrag, it won't help you at all in this scenario.
Should all this be fruitless too then move a whole bunch of data off the NAS and do the balances again as per above. I am talking moving a good few terabytes off it. Then balance it and move data back to the NAS afterwards. There is also some magic that can be done where you can extend the BTRFS filesystem volume by say, adding in a USB, and "glue" that to the data volume to inject from free chunks so a balance can occur. Then the USB can be gracefully balanced out of the volume afterwards. However, this is absolutely not for the uninitiated to attempt 🙂 It is not particularly hard to do but you need to know BTRFS fairly well and it can be bad if messed up. It is something done as a last resort in extreme cases.
And as always, please do ensure you have recent backups - just in case 🙂
enter ssh root@nas-ip-address(real IP address of course) in the windows search bar of a win10 PC, and then enter the admin password of the NAS at the prompt. The root password of the NAS is the same as the admin password.
