Re: [RNDU6000] How do I recover the space taken up by the 'ghost' snapshots?

SauvageSausage · ‎2022-03-05

Me noticed my NAS is going out of the free space, so I get into the admin panel. Removing all the snapshots of both folders and home folders didn't change anything, actually.

How do I free the space, consumed by the non-existing snapshots?

My NAS overview page:

Volume summary info:

List of shared resources (please notice there are no snapshots):

Configured maintenance:

DATA PARTITION SIZE:

root@databox:~# du -sh /data

4.1T    /data

FREE SPACE:

root@databox:~# df -H

Filesystem      Size  Used Avail Use% Mounted on
...
/dev/md127       10T  7.6T  2.5T  76% /data
...

StephenB · ‎2022-03-08

I'd try deleting some of them manually (one at a time), and confirm that there are no errors.

There is a -c option, which waits for the deletion to be committed before the btrfs subvolume delete returns. Might be worth trying that on one or two.

With some work, you could create a script to extract each snapshot path from the list, and then build the btrfs command to delete it. But as I mentioned earlier, you'd need to be careful not to accidentally end up deleting a share by mistake.

View solution in original post

StephenB · ‎2022-03-05

@SauvageSausage wrote:

How do I free the space, consumed by the non-existing snapshots?

Start by running a balance from the volume settings wheel. That should reclaim the missing space.

@SauvageSausage wrote:

root@databox:~# du -sh /data

4.1T    /data

root@databox:~# df -H

btrfs fi du -s /data will give more accurate results than du (though it will be very slow).

btrfs fi df /data completes quickly and gives more info than df

SauvageSausage · ‎2022-03-06

@StephenB wrote:
@SauvageSausage wrote:

How do I free the space, consumed by the non-existing snapshots?

Start by running a balance from the volume settings wheel. That should reclaim the missing space.

Already done!

btrfs fi du -s /data will give more accurate results than du (though it will be very slow).

btrfs fi df /data completes quickly and gives more info than df

Thanks for your tip! "btrfs fi du" is in progress; 'df' is:

root@databox:/# btrfs fi df /data
Data, single: total=7.20TiB, used=6.95TiB
System, DUP: total=32.00MiB, used=992.00KiB
Metadata, DUP: total=6.50GiB, used=3.06GiB
GlobalReserve, single: total=512.00MiB, used=0.00B

StephenB · ‎2022-03-06

Can you also run this:

btrfs subvolume list -s /data

If there are any snapshots left for some reason, this command should list them.

SauvageSausage · ‎2022-03-06

ARE ALL THOSE SNAPSHOTS? 😮

https://pastebin.com/WUrGhPQG

StephenB · ‎2022-03-06

@SauvageSausage wrote:

ARE ALL THOSE SNAPSHOTS? 😮

https://pastebin.com/WUrGhPQG

Yes. It's not clear to me why they aren't showing up in the web ui though.

SauvageSausage · ‎2022-03-07

is it safe to bulk delete them using something like

btrfs subvolume delete $SNAPPATH

where $SNAPPATH is a path, ending with '/snapshot' ?

StephenB · ‎2022-03-07

@SauvageSausage wrote:

is it safe to bulk delete them using something like
btrfs subvolume delete $SNAPPATH
where $SNAPPATH is a path, ending with '/snapshot' ?

I don't think that will work. You could create a script to delete one at a time, but you'd to be careful, as a mistake could delete your data.

Try this first:

snapper delete 1-100000

and see if that gets rid of them.

SauvageSausage · ‎2022-03-07

root@databox:~# pwd
/root

root@databox:~# snapper delete 1-100000
The config 'root' does not exist. Likely snapper is not configured.
See 'man snapper' for further instructions.

root@databox:~# sudo snapper delete 1-100000
The config 'root' does not exist. Likely snapper is not configured.
See 'man snapper' for further instructions.

UPDATE:

root@databox:~# ls /etc/snapper/configs
0  1  10  11  12  13  14  16  17  2  20  21  3  4  5  6  7  8  9

root@databox:~# cat /etc/snapper/configs/0

# subvolume to snapshot
SUBVOLUME="/data/torrents"

# filesystem type
FSTYPE="btrfs"


# btrfs qgroup for space aware cleanup algorithms
QGROUP=""


# fraction of the filesystems space the snapshots may use
SPACE_LIMIT="0.5"


# users and groups allowed to work with config
ALLOW_USERS="guest"
ALLOW_GROUPS="admin"

# sync users and groups from ALLOW_USERS and ALLOW_GROUPS to .snapshots
# directory
SYNC_ACL="no"


# start comparing pre- and post-snapshot in background after creating
# post-snapshot
BACKGROUND_COMPARISON="yes"


# run daily number cleanup
NUMBER_CLEANUP="yes"

# limit for number cleanup
NUMBER_MIN_AGE="1800"
NUMBER_LIMIT="50"
NUMBER_LIMIT_IMPORTANT="10"


# create hourly snapshots
TIMELINE_CREATE="yes"

# cleanup hourly snapshots after some time
TIMELINE_CLEANUP="yes"

# limits for timeline cleanup
TIMELINE_MIN_AGE="1800"
TIMELINE_LIMIT_HOURLY="10"
TIMELINE_LIMIT_DAILY="10"
TIMELINE_LIMIT_WEEKLY="0"
TIMELINE_LIMIT_MONTHLY="10"
TIMELINE_LIMIT_YEARLY="10"


# cleanup empty pre-post-pairs
EMPTY_PRE_POST_CLEANUP="yes"

# limits for empty pre-post-pair cleanup
EMPTY_PRE_POST_MIN_AGE="1800"

SauvageSausage · ‎2022-03-07

root@databox:~# snapper delete 1-100000
The config 'root' does not exist. Likely snapper is not configured.
See 'man snapper' for further instructions.


root@databox:~# ls /etc/snapper/configs
0  1  10  11  12  13  14  16  17  2  20  21  3  4  5  6  7  8  9


root@databox:~# cat /etc/snapper/configs/0

# subvolume to snapshot
SUBVOLUME="/data/torrents"

# filesystem type
FSTYPE="btrfs"


# btrfs qgroup for space aware cleanup algorithms
QGROUP=""


# fraction of the filesystems space the snapshots may use
SPACE_LIMIT="0.5"


# users and groups allowed to work with config
ALLOW_USERS="guest"
ALLOW_GROUPS="admin"

# sync users and groups from ALLOW_USERS and ALLOW_GROUPS to .snapshots
# directory
SYNC_ACL="no"


# start comparing pre- and post-snapshot in background after creating
# post-snapshot
BACKGROUND_COMPARISON="yes"


# run daily number cleanup
NUMBER_CLEANUP="yes"

# limit for number cleanup
NUMBER_MIN_AGE="1800"
NUMBER_LIMIT="50"
NUMBER_LIMIT_IMPORTANT="10"


# create hourly snapshots
TIMELINE_CREATE="yes"

# cleanup hourly snapshots after some time
TIMELINE_CLEANUP="yes"

# limits for timeline cleanup
TIMELINE_MIN_AGE="1800"
TIMELINE_LIMIT_HOURLY="10"
TIMELINE_LIMIT_DAILY="10"
TIMELINE_LIMIT_WEEKLY="0"
TIMELINE_LIMIT_MONTHLY="10"
TIMELINE_LIMIT_YEARLY="10"


# cleanup empty pre-post-pairs
EMPTY_PRE_POST_CLEANUP="yes"

# limits for empty pre-post-pair cleanup
EMPTY_PRE_POST_MIN_AGE="1800"

StephenB · ‎2022-03-07

@StephenB wrote:
@SauvageSausage wrote:

is it safe to bulk delete them using something like
btrfs subvolume delete $SNAPPATH
where $SNAPPATH is a path, ending with '/snapshot' ?
I don't think that will work. You could create a script to delete one at a time, but you'd to be careful, as a mistake could delete your data.

Try this first:
snapper delete 1-100000
and see if that gets rid of them.

The snapper command will fail because Netgear has set up snapper with a config for each share (and no root config).

It should work if you first list the configs (which are set up as numeric):

snapper list-configs

then delete snapshots for each config

snapper -c 0 delete 1-100000
snapper -c 1 delete 1-100000
...

Of course this will only delete your snapshots if they are in a config - which might not be the case. But it is easy to try, and there should be a lot fewer configs than there are snapshots in your case.

SauvageSausage · ‎2022-03-07

thanks! but it seems you're right and snapshots aren't in the configs:

root@databox:~# snapper -c 0 delete 1-100000
Snapshot '1' not found.

root@databox:~# snapper -c 1 delete 1-100000
Snapshot '1' not found.

root@databox:~# snapper -c 2 delete 1-100000
Snapshot '1' not found.

root@databox:~# snapper -c 3 delete 1-100000
Snapshot '1' not found.

root@databox:~# snapper -c 4 delete 1-100000
Snapshot '1' not found.

root@databox:~# snapper -c 5 delete 1-100000
Snapshot '1' not found.

root@databox:~# snapper -c 6 delete 1-100000
Snapshot '1' not found.

root@databox:~# snapper -c 7 delete 1-100000
Snapshot '1' not found.

root@databox:~# snapper -c 8 delete 1-100000
Snapshot '1' not found.

root@databox:~# snapper -c 9 delete 1-100000
Snapshot '1' not found.

root@databox:~# snapper -c 10 delete 1-100000
Snapshot '1' not found.

root@databox:~# snapper -c 11 delete 1-100000
Snapshot '1' not found.

root@databox:~# snapper -c 12 delete 1-100000
Snapshot '1' not found.

root@databox:~# snapper -c 13 delete 1-100000
Snapshot '1' not found.

root@databox:~# snapper -c 14 delete 1-100000
Snapshot '1' not found.

root@databox:~# snapper -c 15 delete 1-100000
Unknown config.

root@databox:~# snapper -c 16 delete 1-100000
Snapshot '1' not found.

root@databox:~# snapper -c 17 delete 1-100000
Snapshot '1' not found.

root@databox:~# snapper -c 18 delete 1-100000
Unknown config.

root@databox:~# snapper -c 19 delete 1-100000
Unknown config.

root@databox:~# snapper -c 20 delete 1-100000
Snapshot '1' not found.

root@databox:~# snapper -c 21 delete 1-100000
Snapshot '1' not found.

StephenB · ‎2022-03-08

I'd try deleting some of them manually (one at a time), and confirm that there are no errors.

There is a -c option, which waits for the deletion to be committed before the btrfs subvolume delete returns. Might be worth trying that on one or two.

With some work, you could create a script to extract each snapshot path from the list, and then build the btrfs command to delete it. But as I mentioned earlier, you'd need to be careful not to accidentally end up deleting a share by mistake.

SauvageSausage · ‎2022-03-08

Thank you, it did help! There is a strange thing with balancing progress tho but I believe nothing fatal:

StephenB · ‎2022-03-09

@SauvageSausage wrote:

Thank you, it did help! There is a strange thing with balancing progress tho but I believe nothing fatal:

I've seen this myself - I think it happens when the volume has been vertically expanded in the past. But it isn't fatal.

Sandshark · ‎2022-03-09

I hasve found that the ReadyNAS does not automatically do a balance after expansion (which is recommended when expanding a BTRFS volume), and that results in no allocation of metadata space on a vertically expanded volume. So, manually doing a balance afterward is highly recommened.

I've also noticed some inconsistancy as to whether the metedata is in RAID or DUP. You can find out with btrfs filesystem df /data.

From everything I've read, RAID is the best configuration. The following will change that: btrfs balance start -mconvert=raid1 /data

Of course, it's a balance, so it'll take a while.