NETGEAR is aware of a growing number of phone and online scams. To learn how to stay safe click here.
Forum Discussion
ReadyNas 102 = Unusable frustrating junk! - now what?
Firmware 6.10.4 hotfix 1 - RAID 1 This unit has become a comedy of errors. It seems anything I try to do with it, it just gives errors. Yeah, it's end of life. But its not really that old and it...
chadwixk wrote:I have a nightly robocopy script that attempts to copy files from my laptop to the RN, but gives a Not Enough Space on the Disk error. There is plenty of space.
What does your btrfs.log look like, particularly the top part of it.
chadwixk wrote:Any ideas on where to start? I'd love for it to just do these two things as it should:
- Copy files to it up to full disk capacity
BTRFS loves free space. It's not a good idea to fill up any disk in any system regardless of brand or operating system used up to capacity. You are far more likely to encounter corner case problems with a full filesystem than one with plenty of free space. How full is your data volume?
chadwixk wrote:If I could get it to work, great. If not, I'll replace...with a NON-Netgear alternative as this has been extremely frustrating and I'd expect to be able at least copy files to a NAS device. IF I go that route, can I just plug my 2 - 3TB drives in RAID 1 config into another brand unit?
A different brand would need to format the disks, so you would need to backup your data first.
I didn't really mean I'd fill it to 100%, but I'd like the ability to fill it...i.e. not get errors that it is full when it isn't.
I have about 600GB of free space. And I just freed up more by cleaning up old snapshots.
Here's the top part of btrfs.log
Label: '2fe56952:data' uuid: f3768221-8da7-44f9-8fe1-fc703e7af6fd Total devices 1 FS bytes used 2.14TiB devid 1 size 2.72TiB used 2.72TiB path /dev/md127 === filesystem /data === Data, single: total=2.70TiB, used=2.13TiB System, DUP: total=8.00MiB, used=352.00KiB System, single: total=4.00MiB, used=0.00B Metadata, DUP: total=10.50GiB, used=10.00GiB Metadata, single: total=8.00MiB, used=0.00B GlobalReserve, single: total=512.00MiB, used=0.00B
Ok, that's what I thought....working on scouring up a bunch of old drives now to do so.
chadwixk wrote:
Here's the top part of btrfs.log
Label: '2fe56952:data' uuid: f3768221-8da7-44f9-8fe1-fc703e7af6fd Total devices 1 FS bytes used 2.14TiB devid 1 size 2.72TiB used 2.72TiB path /dev/md127
Note that third line
size 2.72TiB used 2.72TiB
That means that the space on the data volume is fully allocated.
When you delete data if a chunk is emptied it is returned to unallocated space. It can take some time for this to happen. When you delete an old snapshot all the newer snapshots are updated recursively, I think which can take some time.Note to free up space by deleting snapshots, all snapshots that contain some deleted data must be deleted.
So if you added a file on the 1st January and deleted it on the 1st March you'd need to delete all snapshots from and including the time on 1st January when the data was added onwards through to and including the time on 1st March when the data was deleted. If you don't do that then deleting that data doesn't free up space.
If deleting all the snapshots that contain some data doesn't change the allocations being full then you could try a balance from SSH with some special parameters e.g. -dusage=1. Though it could fail due to not enough space in which case the workaround is to temporarily add some storage e.g. a USB disk so that a balance can be run and then when space is freed up to run commands to remove the USB disk from the data volume.chadwixk wrote:
Data, single: total=2.70TiB, used=2.13TiB
Ok, that's what I thought....working on scouring up a bunch of old drives now to do so.
So we have 2.70TiB allocated to data of which 2.13TiB is in use.
chadwixk wrote:
Metadata, DUP: total=10.50GiB, used=10.00GiB
0.5GiB of that metadata allocation is reserved so the metadata is full. What's needed is to return some space from data to unallocated space so that it can be allocated to metadata.
chadwixk wrote:
Ok, that's what I thought....working on scouring up a bunch of old drives now to do so.
What does your initrd.log look like?
If you look further in btrfs.log do you see some of the deleted snapshots are still in purge?I saw that line, but also right above it it shows that only 2.14TB is used, which also corresponded with what the web interface GUI showed...about 25% free space.
Total devices 1 FS bytes used 2.14TiB
Also this seems to support that there is ~600GB of free space:
Data, single: total=2.70TiB, used=2.13TiB
This seems to directly contradict each other? Why does this show it is full? :
size 2.72TiB used 2.72TiB
Here's the initrd.log
[2015/08/27 01:19:39] Factory default initiated due to new disks (no RAID, no partitions)! [2015/08/27 01:19:45] Defaulting to X-RAID2 mode, RAID level 1 [2015/08/27 01:20:17] Factory default initiated on ReadyNASOS 6.2.0 (ReadyNASOS). [2015/08/30 07:20:21] Updated from ReadyNASOS 6.2.0 (ReadyNASOS) to 6.2.5 (ReadyNASOS). [2015/12/02 09:50:47] Updated from ReadyNASOS 6.2.5 (ReadyNASOS) to 6.4.1 (ReadyNASOS). [2016/11/29 20:25:15] Updated from ReadyNASOS 6.4.1 (ReadyNASOS) to 6.5.0 (RC3). [2017/07/15 16:12:29 UTC] Updated from ReadyNASOS 6.5.0 (RC3) to 6.7.5 (ReadyNASOS). [2018/11/15 21:22:23 UTC] Updated from ReadyNASOS 6.7.5 (ReadyNASOS) to 6.9.4 (ReadyNASOS). [2019/02/05 06:55:49 UTC] Updated from ReadyNASOS 6.9.4 (ReadyNASOS) to 6.9.5 (ReadyNASOS). [2019/12/03 10:12:14 UTC] Updated from ReadyNASOS 6.9.5 (ReadyNASOS) to 6.10.2 (ReadyNASOS). [2021/03/04 03:57:07 UTC] Updated from ReadyNASOS 6.10.2 (ReadyNASOS) to 6.10.4 (ReadyNASOS).
So after deleting multiple snapshots, and my son accidentally tripping the breaker that my NAS is on, restarting the NAS which is now reindexing, I am able to now Save config changes. Turning off the Volume Quota, adding Dropbox uploads and last night's robocopy backup script worked, without any disk full errors!
So I considere this mostly solved. My only remaining issue is that I was having these errors when the disk seemed to be reporting that it had ~600GB of 2.7TB of free space, which is a decent amount of space. So I'm not convinced that it was truly full, and hence not sure of what the root problem was, and further, not convinced it won't happen again nor how to fix if it does. Time will tell I guess.
Current btrfs.log:
Label: '2fe56952:data' uuid: f3768221-8da7-44f9-8fe1-fc703e7af6fd Total devices 1 FS bytes used 1.53TiB devid 1 size 2.72TiB used 2.72TiB path /dev/md127 === filesystem /data === Data, single: total=2.70TiB, used=1.52TiB System, DUP: total=8.00MiB, used=352.00KiB System, single: total=4.00MiB, used=0.00B Metadata, DUP: total=10.50GiB, used=3.00GiB Metadata, single: total=8.00MiB, used=0.00B GlobalReserve, single: total=512.00MiB, used=0.00B
It still shows in this one line that it is full, so I don't think that means its full, but that it is fully reserved for use or something
devid 1 size 2.72TiB used 2.72TiB path /dev/md127
It is now reporting only being a bit over 50% used:
Data, single: total=2.70TiB, used=1.52TiB
Metadata, DUP (not sure the diff between DUP and single), is lower, but originally, it still had 500MB of unused space, I'd think that would be plenty and there was 600GB of free space it could've pulled from if it needed more, and I'd hope the logic would be there to do so.
The space is fully allocated. There's a difference between allocation and what is in use of the allocation. That 512MiB in metadata is in the GlobalReserve. The system had nowhere to store additional metadata and with the space on the volume fully allocated it couldn't allocate any more space to metadata. Now that you have metadata allocation available to use you can write to the volume again.
You may wish to do a balance of the data volume. That will move data around so that chunks are emptied and returned to unallocated space.
chadwixk wrote:
Also this seems to support that there is ~600GB of free space:
Data, single: total=2.70TiB, used=2.13TiB
This seems to directly contradict each other? Why does this show it is full? :
size 2.72TiB used 2.72TiB
To add to mdgm's explanation:
With btrfs, there is a difference between used space and allocated space (and there is similarly a difference between unused space and unallocated space). Note I am not using the word "free", because that is part of the confusion.
The btrfs report would be more clear if it was worded as follows
Label: '2fe56952:data' uuid: f3768221-8da7-44f9-8fe1-fc703e7af6fd Total devices 1 FS bytes used 2.14TiB devid 1 size 2.72TiB allocated 2.72TiB path /dev/md127 === filesystem /data === Data, single: allocated=2.70TiB, used=2.13TiB
But the btrfs folks chose not to word it that way.
Line 3 tells you that the volume is completely allocated - there is no unallocated space.The bottom line tells you that have about 583 GiB (.570 TiB) of allocated data space that is unused (not in files). Your problem is that it is generally unusable (that is, not "free").
As mdgm says, the way to convert the unused space into unallocated space is to run a balance from the volume settings wheel. If that fails, you could try running the balance using ssh (which gives you some options that are useful when you have a completely allocated file system).
Thanks for the info to both of you, but I still am not fully understanding.
I get the concept of allocated vs used as I do database work. You can allocate (i.e. reserve) space to be used by the database so other files, etc can't use that space, but the difference it sounds like is in a database, you can start filling that unused space with data.
Similarly, I'm used to working with disks on PCs (not unix) but I think the concepts are the same and even the terminology IIRC. You create a volume and then within that partitions. Say a partition reserves 500GB...then you can start adding files to it. You add a 10GB file, you have 490GB free to add more files.
But you are saying that allocated, yet unused space, is not usable? That...I don't understand.
That's a large chunk of space to not be usable. And the screen stated it was Free so what it is reporting in that case is not in sync with what is reality?
It currently shows this:
Label: '2fe56952:data' uuid: f3768221-8da7-44f9-8fe1-fc703e7af6fd Total devices 1 FS bytes used 1.53TiB devid 1 size 2.72TiB used 2.72TiB path /dev/md127 === filesystem /data === Data, single: total=2.70TiB, used=1.52TiB System, DUP: total=8.00MiB, used=352.00KiB System, single: total=4.00MiB, used=0.00B Metadata, DUP: total=10.50GiB, used=3.00GiB Metadata, single: total=8.00MiB, used=0.00B GlobalReserve, single: total=512.00MiB, used=0.00B
It still shows being fully allocated. I'm assuming it's always going to show fully allocated, even if you had zero data on it. This is the "Volume" size right?
So I'm assuming the fully allocated Volume then has these partitions within it: Data, System, Metadata, and GlobalReserve.
StephenB I think you are saying the disk was reporting full because the Data partion's unused space was not free?
- why is the 600GB in Data that is not used also not "Free". This is like having a bank account with $1,000 in it...yet if you try to draft $750, it says you can't :)
- why the Admin panel still showed it as Free
- and if its not truly free, why would there not be some automated function that free'd it up.
- I didn't run a Balance, yet it accepting about 100GB in my backup last night, what allowed that if the Free space isn't really free?
mdgm I think you are saying the issue was not the data partition that was the issue, but rather the Metadata partition being full.
- Metadata still had 500MB, which is not insignificant...is there some hard stop buffer at 10GB?
- I assume that the index syncing it is doing now is going to continue to fill up the Metadata partition...will it cause these issues again?
- And if so, what sizes the Metadata partition to the correct size? How does it know to make itself larger? The Balance task? 10.5GB is the default value I guess, but does it have no way to auto-grow on it's own without having to manually Balance? Could it not at least give an error or alert.."Metadata is full, run a Balance"?
Metadata is data about data. All data has associated metadata. Whenever you make changes to data, metadata is also changed. The 512MiB of metadata was reserved for the GlobalReserve. This is held back in reserve for things like running a balance, I think.
As the unused space was fully allocated to data, the metadata allocation could not be increased. As metadata could not be added new data could not be written.
Once metadata usage fell, there was then space again to write metadata so new data could be written.
When you delete data if a chunk becomes empty it is returned to unallocated space. However if even a tiny amount is still in the chunk it does not. What a balance does is move data and metadata around so that chunks are emptied and returned to unallocated space. Then as data is added whichever needs it out of data and metadata can have its allocation increased.
NETGEAR Academy
Boost your skills with the Netgear Academy - Get trained, certified and stay ahead with the latest Netgear technology!
Join Us!
