No space left on device (but there is).

[DonOKeeffe]

ReadyNAS is a 626x on firmware 6.8.1

I have one SMB share which becomes inaccessible after a short term after rebooting.

running dmesg I see loads of these entrys. I have run several balances and scrubs.

 
[Fri Feb 2 15:13:47 2018] systemd-journald[2773]: Failed to rotate /var/log/journal/ddd76a6be61b4ae3968785b647c959e4/system.journal: No space left on device
[Fri Feb 2 15:13:47 2018] systemd-journald[2773]: Failed to write entry (20 items, 504 bytes), ignoring: Input/output error

Could any advise me please?

 Thanks

[StephenB]

What apps are installed?  Did you do anything with ssh?

[DonOKeeffe]

Thanks for your response,

I have used ssh to diagnose the issue. No disk space issues. The only app installed is the SMB Plus app which we use to manage locked files better.

[StephenB]

How full was the OS partition?

Did you check it with the btrfs commands?  df,etc  don't always get the free space correct with the btrfs file system.

[DonOKeeffe]

lots of space my the looks of things (see below)

root@VegaNAS:~# df -h
Filesystem Size Used Avail Use% Mounted on
udev 10M 4.0K 10M 1% /dev
/dev/md0 4.0G 704M 3.0G 20% /
tmpfs 3.9G 0 3.9G 0% /dev/shm
tmpfs 3.9G 6.1M 3.9G 1% /run
tmpfs 2.0G 1.1M 2.0G 1% /run/lock
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
/dev/md127 15T 2.8T 12T 19% /Volume1
/dev/md127 15T 2.8T 12T 19% /home
/dev/md127 15T 2.8T 12T 19% /apps
tmpfs 50M 0 50M 0% /var/replicate/shm

[mdgm-ntgr]

Which services are you running?

[DonOKeeffe]

SMB, HTTP, HTTPS, SNMP, SSH &  Antivirus

[StephenB]

---

wrote:

lots of space my the looks of things (see below)

root@VegaNAS:~# df -h 

---

As I said, df can deceive since it doesn't account for metadata properly.  You should also run btrfs filesystem df // to see used space on the OS partition, and post those results here too.

There are a bunch of odd things happening with AntiVirus lately, so you might also try turning that service off.

[DonOKeeffe]

I have disabled AV and will monitor it thanks. Are the AV issues with firmware 6.8.1 specifically?

[StephenB]

---

wrote:

Are the AV issues with firmware 6.8.1 specifically?

---

Not sure.  Some of turned up quite recently (6.9.1/6.9.2), but that might have been evoked by a ClamAV definitions update.  If so, it could be showing up on older firmware too.

[DonOKeeffe]

I ran the command btrfs filesystem df //

root@VegaNAS:~# btrfs filesystem df //
Data, single: total=1.23GiB, used=668.56MiB
System, DUP: total=8.00MiB, used=16.00KiB
Metadata, DUP: total=204.56MiB, used=9.78MiB
GlobalReserve, single: total=16.00MiB, used=0.00B

this looks ok right? 

Also, you can see from the GUI screenshot attached that the disk isn't full. clealy this is right.

[StephenB]

---

wrote:

I ran the command btrfs filesystem df //

root@VegaNAS:~# btrfs filesystem df //
Data, single: total=1.23GiB, used=668.56MiB
System, DUP: total=8.00MiB, used=16.00KiB
Metadata, DUP: total=204.56MiB, used=9.78MiB
GlobalReserve, single: total=16.00MiB, used=0.00B

this looks ok right?  

---

Yes.

---

wrote:

Also, you can see from the GUI screenshot attached that the disk isn't full. clealy this is right.

---

That screenshot isn't relevant, because your out-of-space errors are being reported on the OS partition, and not the data volume.

But the OS partition looks like it has adequate space.

[DonOKeeffe]

so looking at the results of  btrfs filesystem df /Volume1

it appears the main data Volume is full! but how?

root@VegaNAS:~# btrfs filesystem df /Volume1
Data, single: total=2.68TiB, used=2.67TiB
System, DUP: total=32.00MiB, used=352.00KiB
Metadata, DUP: total=49.50GiB, used=29.62GiB
GlobalReserve, single: total=512.00MiB, used=0.00B

[DonOKeeffe]

i have ran another balance after turning off AV but the btrfs filesystem df /Volume1 command still shows the main data volume (volume1) full.

[mdgm-ntgr]

That output says nothing of the kind. All it indicates is that of the ~ 2.7TB allocated for storing data, pretty much all of that's in use (which is what you'd expect having just balanced it). The data volume could be a lot larger (with a lot of unallocated space) and your earlier output suggests that the data volume is a lot larger.

Balancing the data volume when the problem isn't with the data volume isn't going to do anything to deal with the problem you're facing.

[DonOKeeffe]

You're abosoulty correct. sorry.

After turning of the AV service the server has been 100% stable. 

In order to resolve this issue once and for all, what do you think we should do?

Thanks again for your help here.

Don

[mdgm-ntgr]

Update to 6.9.2, wait 15 minutes to allow plenty of time for the hot-fix to be installed and then turn anti-virus back on if you wish.

[DonOKeeffe]

so there is a hotfix in 6.9.2 for this issue?

[mdgm-ntgr]

Yes there’s an anti-virus hot-fix for 6.9.2 that is automatically installed.

I expect that this is what you ran into.

The fix will be included in 6.9.3.