NETGEAR is aware of a growing number of phone and online scams. To learn how to stay safe click here.
Forum Discussion
Cannot copy files to RN316 although I have 22TB free...
Hi all. I have the following RN316: Firmware 6.10.4 Running 6 x 10TB IronWolf HDDs X-RAID 21.9TB free / 23.4TB used History: Last year I replaced all the 8TB Ironwold HDDs (from memory) one by...
StephenB wrote:
Deleting some large files and trying to balance again is a reasonable next step.
And perhaps some of the oldest (and thus likely largest) snapshots as well before the balance. The first balance will normally take a while, so that error explains why it only took seconds. That the ReadyNAS didn't tell you it ended in error is, IMHO, an oversight in the ReadyNAS OS.
I've seen some posts on general Linux forums recommending a balance after a BTRFS expansion, and it does not appear Netgear does that automatically. Apparently, that helps properly allocate data and metadata across the volume. The MAN page says "The primary purpose of the balance feature is to spread block groups across all devices so they match constraints defined by the respective profiles". I've not found a good list of those restraints, but you may have arrived at one of them. So having added to and then expanded one of the "devices" (which is a multi-partition MDADM RAID) in your file system, a balance should help. Of course, it is sometimes hard to pick out what in the general BTRFS forums and wiki are applicable to ReadyNAS because ReadyNAS uses BTRFS on top of MDADM RAID rather than BTRFS for both file system and RAID and the general discussions typically assume you are using BTRFS for both..
Sandshark wrote:
I've seen some posts on general Linux forums recommending a balance after a BTRFS expansion, and it does not appear Netgear does that automatically. Apparently, that helps properly allocate data and metadata across the volume. The MAN page says "The primary purpose of the balance feature is to spread block groups across all devices so they match constraints defined by the respective profiles". I've not found a good list of those restraints, but you may have arrived at one of them.
I'm wondering that also. Looking at the first post, I see this.
Label: 'data' uuid: ... Total devices 2 FS bytes used 23.43TiB devid 1 size 18.17TiB used 18.17TiB path /dev/md127 devid 2 size 27.28TiB used 5.29TiB path /dev/md126
Note all the unallocated space is on md126. md127 is completely full.
Hi all....
So I have not been able to resolve the issue yet, I believe.
BTW is 6.10.4 (Hotfix 1) buggy??? Should I be downgrading to 6.10.3?I've been offline as the RN316 was horrendously unavailabe during my scrub, and even after it failed or completed after 7 days it still feels sluggish to me...
Anyway... I have moved about 500GB off the RN316. But is that enough? It have moved most of the files I have copied across this year, but also some older stuff from late last year, but probably AFTER I replaced all the 4TB HDDs with 10TB HDDs so I am again wondering if everything is going to the old md127 and not new md126???
So right now here are my stats / logs / telemetry:
BTRFS.LOG
Label: 'blah:root' uuid: blah-blah Total devices 1 FS bytes used 1.43GiB devid 1 size 4.00GiB used 3.61GiB path /dev/md0 Label: 'blah:data' uuid: blah-blah Total devices 2 FS bytes used 22.95TiB devid 1 size 18.17TiB used 18.14TiB path /dev/md127 devid 2 size 27.28TiB used 4.84TiB path /dev/md126 === filesystem /data === Data, single: total=22.95TiB, used=22.93TiB System, RAID1: total=32.00MiB, used=2.95MiB Metadata, RAID1: total=5.85GiB, used=5.32GiB Metadata, DUP: total=10.50GiB, used=10.03GiB GlobalReserve, single: total=512.00MiB, used=0.00B === subvolume /data ===
Why does Data, single: showing 22.95TiB, which suspiciously seems to be the limit of the amount of data I can store? Recall the UI is showing data 22.96TB and Free space: 22.49TB. Is this the "smoking gun"???
VOLUME.LOG
data disk test 2020-09-01 01:00:01 2020-09-01 15:20:27 pass data resilver 2020-09-13 15:54:14 2020-09-14 20:35:46 completed data balance 2021-02-18 21:17:16 2021-02-18 21:18:42 completed ERROR: error during balancing '/data': No space left on device T data scrub 2021-02-18 21:29:29 data disk test 2021-03-01 08:15:53 data balance 2021-03-01 21:03:16 2021-03-01 21:04:48 completed ERROR: error during balancing '/data': No space left on device T data balance 2021-03-03 15:34:37 2021-03-04 03:44:36 completed ERROR: error during balancing '/data': No space left on device T data balance 2021-03-05 09:34:32 2021-03-05 10:29:27 completed ERROR: error during balancing '/data': No space left on device T data balance 2021-03-05 19:39:44 2021-03-05 19:49:07 completed ERROR: error during balancing '/data': No space left on device T data balance 2021-03-05 21:09:45 2021-03-05 21:27:23 completed ERROR: error during balancing '/data': No space left on device T data balance 2021-03-05 21:28:15 2021-03-05 21:28:19 completed Done, had to relocate 1 out of 23557 chunks data balance 2021-03-05 21:45:20 2021-03-05 21:46:05 completed Done, had to relocate 29 out of 23557 chunks data balance 2021-03-05 21:57:26 2021-03-05 21:57:31 completed Done, had to relocate 1 out of 23529 chunks data balance 2021-03-05 21:59:22 2021-03-05 21:59:27 completed Done, had to relocate 1 out of 23529 chunks data balance 2021-03-05 21:59:48 2021-03-05 21:59:53 completed Done, had to relocate 1 out of 23529 chunks data balance 2021-03-05 22:25:13 2021-03-05 22:25:18 completed Done, had to relocate 1 out of 23529 chunks
Why does it keep relocate 1 out of 23529 chunks only? That chunk size does not go down? I have no idea. Do I keep doing balances?
Should I Defrag now?
I also have SMB Plus installed and have enabled Preallocate (FYI: Preallocate disk space before writing data. This can slow down write speed slightly, but should result in the file being nicely laid out on the disk, with minimal fragmentation.)
I have removed a lot of snapshots but would like to keep the ones that I have set up for OneDrive and DropBox apps. Both report 19 snapshots with 2 years protection.
I am happy to turn off the snapshots, ie: set them to manual. But can drop all snapshots if people think that's good. Just being a bit nervous...
I'm pulling my hair out... What do I do?
Has the problem been solved? Can you tell? Or should I try to find much older data and remove another 500Gb or 1 TB of older data before I try balancing again?
Any help appreciated!
For what it's worth:
KERNEL.LOG
Mar 05 22:11:19 RN316 systemd[1]: Set hostname to <RN316>. Mar 05 22:11:19 RN316 systemd[1]: systemd-journald-audit.socket: Cannot add dependency job, ignoring: Unit systemd-journald-audit.socket is masked. Mar 05 22:11:19 RN316 systemd[1]: systemd-journald-audit.socket: Cannot add dependency job, ignoring: Unit systemd-journald-audit.socket is masked. Mar 05 22:11:19 RN316 systemd[1]: Started Forward Password Requests to Wall Directory Watch. Mar 05 22:11:19 RN316 systemd[1]: Listening on Journal Socket (/dev/log). Mar 05 22:11:19 RN316 systemd[1]: Set up automount Arbitrary Executable File Formats File System Automount Point. Mar 05 22:11:19 RN316 systemd[1]: Created slice System Slice. Mar 05 22:11:19 RN316 systemd[1]: Created slice system-serial\x2dgetty.slice. Mar 05 22:11:19 RN316 systemd[1]: Created slice system-getty.slice. Mar 05 22:11:19 RN316 systemd[1]: Listening on /dev/initctl Compatibility Named Pipe. Mar 05 22:11:19 RN316 systemd[1]: Started Dispatch Password Requests to Console Directory Watch. Mar 05 22:11:19 RN316 systemd[1]: Reached target Encrypted Volumes. Mar 05 22:11:19 RN316 systemd[1]: Listening on udev Control Socket. Mar 05 22:11:19 RN316 systemd[1]: Reached target Paths. Mar 05 22:11:19 RN316 systemd[1]: Reached target Remote File Systems (Pre). Mar 05 22:11:19 RN316 systemd[1]: Reached target Remote File Systems. Mar 05 22:11:19 RN316 systemd[1]: Listening on udev Kernel Socket. Mar 05 22:11:19 RN316 systemd[1]: Listening on Journal Socket. Mar 05 22:11:19 RN316 systemd[1]: Starting Remount Root and Kernel File Systems... Mar 05 22:11:19 RN316 systemd[1]: Mounting POSIX Message Queue File System... Mar 05 22:11:19 RN316 systemd[1]: Starting Create Static Device Nodes in /dev... Mar 05 22:11:19 RN316 systemd[1]: Mounting Debug File System... Mar 05 22:11:19 RN316 systemd[1]: Created slice User and Session Slice. Mar 05 22:11:19 RN316 systemd[1]: Reached target Slices. Mar 05 22:11:19 RN316 systemd[1]: Listening on Syslog Socket. Mar 05 22:11:19 RN316 systemd[1]: Starting Journal Service... Mar 05 22:11:19 RN316 systemd[1]: Starting Load Kernel Modules... Mar 05 22:11:19 RN316 systemd[1]: Started ReadyNAS LCD splasher. Mar 05 22:11:19 RN316 systemd[1]: Starting ReadyNASOS system prep... Mar 05 22:11:19 RN316 systemd[1]: Mounted POSIX Message Queue File System. Mar 05 22:11:19 RN316 systemd[1]: Mounted Debug File System. Mar 05 22:11:19 RN316 systemd[1]: Started Remount Root and Kernel File Systems. Mar 05 22:11:19 RN316 systemd[1]: Started Create Static Device Nodes in /dev. Mar 05 22:11:19 RN316 systemd[1]: Started Load Kernel Modules. Mar 05 22:11:19 RN316 systemd[1]: Starting Apply Kernel Variables... Mar 05 22:11:19 RN316 systemd[1]: Mounting FUSE Control File System... Mar 05 22:11:19 RN316 systemd[1]: Mounting Configuration File System... Mar 05 22:11:19 RN316 systemd[1]: Starting udev Kernel Device Manager... Mar 05 22:11:19 RN316 systemd[1]: Starting Load/Save Random Seed... Mar 05 22:11:19 RN316 systemd[1]: Starting Rebuild Hardware Database... Mar 05 22:11:19 RN316 systemd[1]: Mounted Configuration File System. Mar 05 22:11:19 RN316 systemd[1]: Mounted FUSE Control File System. Mar 05 22:11:19 RN316 systemd[1]: Started Apply Kernel Variables. Mar 05 22:11:19 RN316 systemd[1]: Started ReadyNASOS system prep. Mar 05 22:11:19 RN316 systemd[1]: Started Load/Save Random Seed. Mar 05 22:11:19 RN316 systemd[1]: Started udev Kernel Device Manager. Mar 05 22:11:19 RN316 systemd[1]: Started Journal Service. Mar 05 22:11:19 RN316 kernel: md: md127 stopped. Mar 05 22:11:19 RN316 kernel: md: bind<sdb3> Mar 05 22:11:19 RN316 kernel: md: bind<sdc3> Mar 05 22:11:19 RN316 kernel: md: bind<sdd3> Mar 05 22:11:19 RN316 kernel: md: bind<sde3> Mar 05 22:11:19 RN316 kernel: md: bind<sdf3> Mar 05 22:11:19 RN316 kernel: md: bind<sda3> Mar 05 22:11:19 RN316 kernel: md/raid:md127: device sda3 operational as raid disk 0 Mar 05 22:11:19 RN316 kernel: md/raid:md127: device sdf3 operational as raid disk 5 Mar 05 22:11:19 RN316 kernel: md/raid:md127: device sde3 operational as raid disk 4 Mar 05 22:11:19 RN316 kernel: md/raid:md127: device sdd3 operational as raid disk 3 Mar 05 22:11:19 RN316 kernel: md/raid:md127: device sdc3 operational as raid disk 2 Mar 05 22:11:19 RN316 kernel: md/raid:md127: device sdb3 operational as raid disk 1 Mar 05 22:11:19 RN316 kernel: md/raid:md127: allocated 6474kB Mar 05 22:11:19 RN316 kernel: md/raid:md127: raid level 5 active with 6 out of 6 devices, algorithm 2 Mar 05 22:11:19 RN316 kernel: RAID conf printout: Mar 05 22:11:19 RN316 kernel: --- level:5 rd:6 wd:6 Mar 05 22:11:19 RN316 kernel: disk 0, o:1, dev:sda3 Mar 05 22:11:19 RN316 kernel: disk 1, o:1, dev:sdb3 Mar 05 22:11:19 RN316 kernel: disk 2, o:1, dev:sdc3 Mar 05 22:11:19 RN316 kernel: disk 3, o:1, dev:sdd3 Mar 05 22:11:19 RN316 kernel: disk 4, o:1, dev:sde3 Mar 05 22:11:19 RN316 kernel: disk 5, o:1, dev:sdf3 Mar 05 22:11:19 RN316 kernel: created bitmap (30 pages) for device md127 Mar 05 22:11:19 RN316 kernel: md127: bitmap initialized from disk: read 2 pages, set 0 of 59543 bits Mar 05 22:11:19 RN316 kernel: md127: detected capacity change from 0 to 19979093934080 Mar 05 22:11:19 RN316 kernel: Adding 1566716k swap on /dev/md1. Priority:-1 extents:1 across:1566716k Mar 05 22:11:20 RN316 kernel: BTRFS: device label 43f5fa04:data devid 1 transid 1895561 /dev/md127 Mar 05 22:11:20 RN316 kernel: md: md126 stopped. Mar 05 22:11:20 RN316 kernel: md: bind<sdb4> Mar 05 22:11:20 RN316 kernel: md: bind<sdc4> Mar 05 22:11:20 RN316 kernel: md: bind<sdd4> Mar 05 22:11:20 RN316 kernel: md: bind<sde4> Mar 05 22:11:20 RN316 kernel: md: bind<sdf4> Mar 05 22:11:20 RN316 kernel: md: bind<sda4> Mar 05 22:11:20 RN316 kernel: md/raid:md126: device sda4 operational as raid disk 0 Mar 05 22:11:20 RN316 kernel: md/raid:md126: device sdf4 operational as raid disk 5 Mar 05 22:11:20 RN316 kernel: md/raid:md126: device sde4 operational as raid disk 4 Mar 05 22:11:20 RN316 kernel: md/raid:md126: device sdd4 operational as raid disk 3 Mar 05 22:11:20 RN316 kernel: md/raid:md126: device sdc4 operational as raid disk 2 Mar 05 22:11:20 RN316 kernel: md/raid:md126: device sdb4 operational as raid disk 1 Mar 05 22:11:20 RN316 kernel: md/raid:md126: allocated 6474kB Mar 05 22:11:20 RN316 kernel: md/raid:md126: raid level 5 active with 6 out of 6 devices, algorithm 2 Mar 05 22:11:20 RN316 kernel: RAID conf printout: Mar 05 22:11:20 RN316 kernel: --- level:5 rd:6 wd:6 Mar 05 22:11:20 RN316 kernel: disk 0, o:1, dev:sda4 Mar 05 22:11:20 RN316 kernel: disk 1, o:1, dev:sdb4 Mar 05 22:11:20 RN316 kernel: disk 2, o:1, dev:sdc4 Mar 05 22:11:20 RN316 kernel: disk 3, o:1, dev:sdd4 Mar 05 22:11:20 RN316 kernel: disk 4, o:1, dev:sde4 Mar 05 22:11:20 RN316 kernel: disk 5, o:1, dev:sdf4 Mar 05 22:11:20 RN316 kernel: md126: detected capacity change from 0 to 29999560785920 Mar 05 22:11:20 RN316 kernel: BTRFS: device label 43f5fa04:data devid 2 transid 1895561 /dev/md126 Mar 05 22:13:08 RN316 kernel: e1000e: eth1 NIC Link is Down Mar 05 22:13:09 RN316 kernel: IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready Mar 05 22:13:09 RN316 kernel: 8021q: adding VLAN 0 to HW filter on device eth1 Mar 05 22:13:12 RN316 kernel: e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None Mar 05 22:13:12 RN316 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready Mar 05 22:13:15 RN316 kernel: Adjusting tsc more than 11% (6835455 vs 8751273) Mar 05 22:16:07 RN316 kernel: nr_pdflush_threads exported in /proc is scheduled for removal Mar 05 22:25:11 RN316 kernel: BTRFS info (device md126): relocating block group 36377627721728 flags system|raid1
Not sure if these snapperd errors are relevant:
SYSTEM.LOG
Mar 05 22:21:54 RN316 dbus[2986]: [system] Activating service name='org.opensuse.Snapper' (using servicehelper) Mar 05 22:21:54 RN316 dbus[2986]: [system] Successfully activated service 'org.opensuse.Snapper' Mar 05 22:21:54 RN316 snapperd[6838]: loading 13409 failed Mar 05 22:21:54 RN316 snapperd[6838]: loading 19029 failed Mar 05 22:21:54 RN316 snapperd[6838]: loading 19504 failed Mar 05 22:21:54 RN316 snapperd[6838]: loading 19543 failed Mar 05 22:21:54 RN316 snapperd[6838]: loading 19557 failed Mar 05 22:21:54 RN316 snapperd[6838]: loading 19608 failed Mar 05 22:21:54 RN316 snapperd[6838]: loading 19614 failed ... Mar 05 22:25:25 RN316 clamd[4134]: SelfCheck: Database status OK. Mar 05 22:25:32 RN316 snapperd[6838]: loading 13409 failed Mar 05 22:25:32 RN316 snapperd[6838]: loading 19029 failed Mar 05 22:25:32 RN316 snapperd[6838]: loading 19504 failed ... Mar 05 22:25:32 RN316 snapperd[6838]: loading 12924 failed Mar 05 22:25:32 RN316 snapperd[6838]: loading 12925 failed Mar 05 22:25:32 RN316 snapperd[6838]: loading 1036 failed
Platypus69 wrote:
BTRFS.LOG
Label: 'blah:root' uuid: blah-blah Total devices 1 FS bytes used 1.43GiB devid 1 size 4.00GiB used 3.61GiB path /dev/md0 Label: 'blah:data' uuid: blah-blah Total devices 2 FS bytes used 22.95TiB devid 1 size 18.17TiB used 18.14TiB path /dev/md127 devid 2 size 27.28TiB used 4.84TiB path /dev/md126 === filesystem /data === Data, single: total=22.95TiB, used=22.93TiB System, RAID1: total=32.00MiB, used=2.95MiB Metadata, RAID1: total=5.85GiB, used=5.32GiB Metadata, DUP: total=10.50GiB, used=10.03GiB GlobalReserve, single: total=512.00MiB, used=0.00B === subvolume /data ===
Why does Data, single: showing 22.95TiB, which suspiciously seems to be the limit of the amount of data I can store? Recall the UI is showing data 22.96TB and Free space: 22.49TB. Is this the "smoking gun"???
The "total=22.95TB" doesn't mean what you think it means. It is the total of the allocated space (and is essentially the same as the 22.96 TB you are seeing in the UI). The two sizes further up are the size of your storage - md127 has size 18.17, md126 has size 27.28. the total size is therefore 45.45TiB, which is the correct size for 6x10TB single redundancy RAID.
The problem here is that for some reason your system has completely filled md127. You can see that by subtracting the "18.14 used" from the "18.17 size" for md127.
There is a brute-force solution - which is to do a factory default, set up the NAS again, and restore your data from backup. That would give you a single RAID group, and you'd have plenty of free space. In addition to being time-consuming, you would lose all your snapshots. Though painful, if it were my own system I'd do the reset and start over.
The other option I see is to delete all your existing snapshots, and see if that frees up space on md127. You'd wait for a while after deletion - then download the log zip again, and look at the "used" space for that RAID group. Hopefully it will drop substantially.
Firstly, thanks a million as always.
Of course I don't know, but I would be surprised if snapshots are the root cause....
I only have set up snapshots for my OneDrive and Dropbox shares. Which represent a fraction of the photos and movies that are stored on the RN316.
Any other snapshots, which likewise were not large anyway, are now gone. So because I only used the free versions of these services that are limited in size. Dropbox = 16GB, OneDrive I can't remember, but probably around 16GB as well. So I thought I would use the snapshot feature of the RN316 for these shares as the free tiers of OneDrive and Dropbox do not have this functionality.
Do you really think it will make a difference if I remove these underlying snapshots? They are small no? But perhaps they take up a lot of meta data???? I don't know...
OneDrive share UI says:
- 7149 files, 98 folders, 13GB
- 20 snapshots (2year(s) protection)
DropBox share UI says:
- 15365 files, 571 folders, 13.9GB
- 19 snapshots (2year(s) protection)
I too have concluded/decided that I will at some point, as soon as I can, buy 8 x 16TB HDDs for my new DS1819+, and do as you suggest with moving all the data off the RN316, refomatting it and movingit back. But I cannot afford the 8 x 16TB HDDs right now, in one hit.
So the frustrating thing is I have run out of space on all my ReadyNASes. I have this 20TB free but I cannot use it!!!! ArggghHh.... :)
So would you suggest an action plan of trying to removing 1TB of old data from md127, then doing a balance, then doing a defrag, then doing a balance and then trying to copy the data back????????
Of course I am very curious as to what the problem is and how to avoid it in the future. It sounds to me that using a strategy of going from (6 x 4TB HDDs) to (6 x 10TB HDDs) to (6 x 16TB HDDs) in the future is not a viable solution for this BTRFS based RAID NASes.
Unless of course I should have had monthly balances/defrags, Which I never did. Netgear never recommended it. I had assumed (incorrectly it seems) that you never needed to run these operatiosn as I only predominanlty only add my family photos and videos.
So I want to learn the lesson here, but am struggling to learn what I did wrong and how to avoid this in the future, other than your "brute force" technique.
So I was planning to fill out my new DS1819+
- Buy 1 x 16TB HDDs in the first month (Yes I know there is no RAID)
- Add 1 x 16TB HDD every month after that, so as to stagger the HDDs lifetime, reduced the chance of them all failing simultaneously, and also staggering the cost
But given all the dramas I am having with BTRFS, I wondering whether this is a horrendous idea, and I would be better off buying 8 x 16TB HDDs and setting up one massive pool. So take the hit on the wallet! :(
Or can I get away with perhaps buying 4 x 16TB HDDs and set up one pool this year. And in 12-24 months buying 4 x 16TB HDDs and set up another pool?
I am begining to suspect that buy the 8 x 16TB HDDs in one hit in the best way to go... Ouch!
NETGEAR Academy
Boost your skills with the Netgear Academy - Get trained, certified and stay ahead with the latest Netgear technology!
Join Us!
