Lost a lot of disk space right after reboot

[acng]

Hi,

I have a very strange problem. Before rebooting it, I have

root@readynas# df -h
Filesystem      Size  Used Avail Use% Mounted on
...
/dev/md127       28T   16T   12T  59% /data
/dev/md127       28T   16T   12T  59% /home
/dev/md127       28T   16T   12T  59% /apps
/dev/md127       28T   16T   12T  59% /run/nfs4/data/Backup
/dev/md127       28T   16T   12T  59% /run/nfs4/data/COMMREPL

After rebooting it, I have:

Welcome to ReadyNASOS 6.1.9

root@readynas# df -h
Filesystem      Size  Used Avail Use% Mounted on
...
/dev/md127       28T   26T  1.3T  96% /data
/dev/md127       28T   26T  1.3T  96% /home
/dev/md127       28T   26T  1.3T  96% /apps
/dev/md127       28T   26T  1.3T  96% /run/nfs4/data/Backup
/dev/md127       28T   26T  1.3T  96% /run/nfs4/data/COMMREPL

I am very puzzled by the sudden loss of space. I had to reboot it because /dev/md127 went to readonly presumably due to low space. This ReadyNAS is used as Commvault backup target and it's mounted via NFS. The reason it had 12TB left before rebooting is because I've deleted many jobs from within Commvault.

Here are clues it went to readonly even though it still had > 1TB left (which is another thing I don't get).

~~~~~~~~

May 09 13:44:50 readynas kernel: WARNING: at fs/btrfs/super.c:255 __btrfs_abort_transaction+0xe5/0xf0()
May 09 13:44:50 readynas kernel: Hardware name: ReadyNAS 3220
May 09 13:44:50 readynas kernel: btrfs: Transaction aborted (error -28)
May 09 13:44:50 readynas kernel: Modules linked in: rn_gpio vpd(P)
May 09 13:44:50 readynas kernel: Pid: 1818, comm: btrfs-delayed-m Tainted: P             3.0.101.RNx86_64.2.1 #1
May 09 13:44:50 readynas kernel: Call Trace:
May 09 13:44:50 readynas kernel:  [<ffffffff88047fdd>] ? warn_slowpath_common+0x7d/0xc0
...
May 09 13:44:50 readynas kernel: ---[ end trace 30355e39ab0bdd7f ]---
May 09 13:44:50 readynas kernel: BTRFS error (device md127) in __btrfs_free_extent:5648: errno=-28 No space left
May 09 13:44:50 readynas kernel: BTRFS info (device md127): forced readonly
May 09 13:44:50 readynas kernel: BTRFS debug (device md127): run_one_delayed_ref returned -28
May 09 13:44:50 readynas kernel: BTRFS error (device md127) in btrfs_run_delayed_refs:2688: errno=-28 No space left
May 09 13:44:50 readynas kernel: BTRFS error (device md127) in __btrfs_free_extent:5648: errno=-28 No space left
May 09 13:44:50 readynas kernel: BTRFS debug (device md127): run_one_delayed_ref returned -28
May 09 13:44:50 readynas kernel: BTRFS error (device md127) in btrfs_run_delayed_refs:2688: errno=-28 No space left

~~~~~~~~

Is the file system doing something like playback somehow even though it was a graceful reboot? I don't believe I have snapshot enabled.

Please help. Thanks,

[mdgm-ntgr]

6.1.9! That's terribly old firmware.

I wouldn't update the firmware with the data volume this full but the firmware should have been updated ages ago.

Can you send in your logs (see the Sending Logs link in my sig)?

[mdgm-ntgr]

I see no snapshots.

status.log shows the volume usage warning alerts.

I do see a lot of metadata on the data volume and the space on the data volume is fully allocated.

Did you run any scheduled/unscheduled volume maintenance?

Have you checked how much space is consumed by different shares, and also Time Machine (if you use that).

Also have you checked if a user been putting a lot of data into a home share? You can login to SMB as the admin user and check the space consumed by home shares.

Or using SSH you could do e.g.
 

# du -csh /data/*

to get a rough idea as to what's consuming the data.

du is not designed for BTRFS, so it'll just give a very rough idea.

Is your backup up to date?

[StephenB]

---

@mdgm-ntgr wrote:

Did you run any scheduled/unscheduled volume maintenance?

Scheduled Maintenance was added in 6.2.0, so it's not included in his (ancient) firmware.