× NETGEAR will be terminating ReadyCLOUD service by July 1st, 2023. For more details click here.
Orbi WiFi 7 RBE973
Reply

Snapshots have stopped

darthy
Apprentice

Snapshots have stopped

Hi, I have had my ReadyNAS 314 for quite a number of years with Snapshots enabled and have only had one issue during all that time which the ReadyNAS team resolved for me after a firmware update (2 - 3 years ago).

 

I realised recently that my Snapshots had stopped working as of 29 June. Smart Snapshots are enabled yet it is showing me that there are zero snapshots and no further entries in my log files from 30 June onwards.

 

The only thing I can see in the syslog is that it started a Radar Update on 28 June but no idea if it is related as there is one more snapshot prune after this occured and then none. I am running 6.10.1 (recently upgraded to it). Would appreciate any advice.

 

Thanks in advance.

Model: RN31400|ReadyNAS 300 Series 4- Bay (Diskless)
Message 1 of 14

Accepted Solutions
StephenB
Guru

Re: Snapshots have stopped

The default setting was 95%, but they might have dropped it down to 90% after that (my own NAS is set to 90).   If yours is set to 85%, then you probably adjusted it.

 

But I think that explains it.  Your system is over the threshold (~86% full based on your earlier post).  The system probably started deleting the oldest snapshots a while ago - bringing the free space back up to 85%.  But your main volume continued to grow and eventually the system deleted them all. 

 

So the solution is to expand the volume.  Though you could bump the setting up a bit for now, I wouldn't go above 90. 

View solution in original post

Message 13 of 14

All Replies
StephenB
Guru

Re: Snapshots have stopped

How full is your data volume?

Message 2 of 14
darthy
Apprentice

Re: Snapshots have stopped

I have 709 GB of 4.75 TB free - about 15%. 

I see this message in the Logs, but have been comfortable with the space free until it drops below 10% - Volume: Less than 20% of volume data's capacity is free. Performance on volume data will degrade if additional capacity is consumed. NETGEAR recommends that you add capacity to avoid performance degradation.

Message 3 of 14
StephenB
Guru

Re: Snapshots have stopped

Have you looked in the log zip file?  snapper.log has a list of snapshots - you can use that to confirm that you have no recent ones. 

 

Also look for "snapper" in system.log.  You should see some indication that it is running (and hopefully saying why it failed to create a snapshot). 

 


@darthy wrote:

... have been comfortable with the space free until it drops below 10% - Volume: Less than 20% of volume data's capacity is free. 


The on-disk snapshot space can sometimes increase dramatically very quickly, so I personally don't recommend allowing free space to drop to 10%. You easily can end up with problems that require you to destroy your volume and restore it from backup.  I have run at 15% at times. 

 

One problem with the smart snapshots is that the monthly ones are never pruned - so eventually they will fill the volume.  Options to avoid that are

  • manually delete old snapshots from time to time
  • switch to custom snapshots, which have a retention setting.

Personally I use custom snapshots.

 

Anyway, you might want to delete some of the oldest snapshots after you get the main problem resolved (or consider switching to the custom settings).

Message 4 of 14
darthy
Apprentice

Re: Snapshots have stopped

Thanks for the advice, and will look to add more disk space and setup custom snapshots, but would like to get this issue resolved first.

 

I'm not sure where to find log.zip, however, in the rsyslog folder I have a syslog file and can see a few snapper entries in there but not really showing much except for these two entries:

Aug 6 19:03:30 ReadyNAS dbus[2875]: [system] Activating service name='org.opensuse.Snapper' (using servicehelper)

Aug 6 19:03:30 ReadyNAS dbus[2875]: [system] Successfully activated service 'org.opensuse.Snapper'

Message 5 of 14
StephenB
Guru

Re: Snapshots have stopped


@darthy wrote:

I'm not sure where to find log.zip,

 You download it from the log page in the admin web ui.

 


@darthy wrote:

however, in the rsyslog folder


Are you using the LogAnalyzer app?  There are quite a few posters who have run into problems with that app - in particular it can fill the OS partition on the NAS.

 

If you have ssh installed, then perhaps check the space utilitization for the OS.

 

Also, you can also look for snapper via journalctl

# journalctl --no-pager | grep -i snapper
Message 6 of 14
darthy
Apprentice

Re: Snapshots have stopped

Thanks again for the great help. I have run the following commands using ssh (I think they're the correct ones to run):

 

# btrfs fi show /data
Label: '43f61cdc:data' uuid: 3b5a9e34-5b9f-4833-9e0a-de5911368cb8
Total devices 1 FS bytes used 4.74TiB
devid 1 size 5.45TiB used 4.79TiB path /dev/md127

 

# btrfs fi df /data
Data, single: total=4.76TiB, used=4.74TiB
System, DUP: total=32.00MiB, used=608.00KiB
Metadata, DUP: total=10.50GiB, used=6.40GiB
GlobalReserve, single: total=512.00MiB, used=0.00B

Looks like almost no free space available on the data drive, which is not what the GUI is showing me, but not familiar with these commands so might be misinterpreting the results.

 

I downloaded the log files and looked at snapper.log, there is nothing really in here, it shows the following:

 

Config: 0, subvolume: /data/Software
Type | # | Pre # | Date | User | Cleanup | Description | Userdata
-------+---+-------+------+------+---------+-------------+---------
single | 0 | | | root | | current |

Config: 1, subvolume: /data/Backup
Type | # | Pre # | Date | User | Cleanup | Description | Userdata
-------+---+-------+------+------+---------+-------------+---------
single | 0 | | | root | | current |

Config: 2, subvolume: /data/Documents
Type | # | Pre # | Date | User | Cleanup | Description | Userdata
-------+---+-------+------+------+---------+-------------+---------
single | 0 | | | root | | current |

Config: 3, subvolume: /data/Music
Type | # | Pre # | Date | User | Cleanup | Description | Userdata
-------+---+-------+------+------+---------+-------------+---------
single | 0 | | | root | | current |

Config: 4, subvolume: /data/Pictures
Type | # | Pre # | Date | User | Cleanup | Description | Userdata
-------+---+-------+------+------+---------+-------------+---------
single | 0 | | | root | | current |

Config: 5, subvolume: /data/readydrop
Type | # | Pre # | Date | User | Cleanup | Description | Userdata
-------+---+-------+------+------+---------+-------------+---------
single | 0 | | | root | | current |

Config: 6, subvolume: /data/Videos
Type | # | Pre # | Date | User | Cleanup | Description | Userdata
-------+---+-------+------+------+---------+-------------+---------
single | 0 | | | root | | current |

Config: 7, subvolume: /data/istat
Type | # | Pre # | Date | User | Cleanup | Description | Userdata
-------+---+-------+------+------+---------+-------------+---------
single | 0 | | | root | | current |

Config: 8, subvolume: /data/rsyslog
Type | # | Pre # | Date | User | Cleanup | Description | Userdata
-------+---+-------+------+------+---------+-------------+---------
single | 0 | | | root | | current |

 

 

Message 7 of 14
StephenB
Guru

Re: Snapshots have stopped


@darthy wrote:

 

I downloaded the log files and looked at snapper.log, there is nothing really in here, it shows the following:

...


It looks like it's showing no snapshots at all - is that what the GUI shows?  Can you try 

# btrfs subvolume list -s /data

That command will give you a list of snapshots.

 


@darthy wrote:

 

# btrfs fi df /data
Data, single: total=4.76TiB, used=4.74TiB

Looks like almost no free space available on the data drive, which is not what the GUI is showing me, but not familiar with these commands so might be misinterpreting the results.

 

 


"Used" might not mean what you think. It includes data chunks that hold a mix of file space and free space.  Still, the difference between the volume size and used is about 675 GiB. That would be (5.45 - 4.79)*1024

 

At some point you should probably run a balance (which is done from the volume settings wheel).

Message 8 of 14
darthy
Apprentice

Re: Snapshots have stopped

When I run this command nothing is returned.

# btrfs subvolume list -s /data

# btrfs subvolume list -s /data
#

 

This is the same as what the GUI is showing, that there are no snapshots. It's very odd as I had a large amount of snapshots prior to this.

 

I'll run the Balance command once this is resolved, thanks for the tip.

 

I do also have the LogAnalyzer app running, if you think this might be causing a problem I can remove it.

Message 9 of 14
StephenB
Guru

Re: Snapshots have stopped


@darthy wrote:

 

This is the same as what the GUI is showing, that there are no snapshots. It's very odd as I had a large amount of snapshots prior to this.

 


There is a trigger point: if the free space drops too much the system will start deleting old snapshots.  But I wouldn't expect that to delete all snapshots, or disable the snapshot schedule.  Have you tried making a manual snapshot?

 


@darthy wrote:

 

I do also have the LogAnalyzer app running, if you think this might be causing a problem I can remove it.


I've never run it myself.  It has caused problems for some users, but others seem ok with it.

 

If you don't use it much, then perhaps remove it.

Message 10 of 14
darthy
Apprentice

Re: Snapshots have stopped

I see my snapshot pruning is set to occur when the volume reaches 85% capacity which it has done, as you say though, it shouldn't have removed all my snapshots...

 

Do you know if Netgear monitors these forums or do I need to contact them directly?

Message 11 of 14
darthy
Apprentice

Re: Snapshots have stopped

I do see the following in this support article though, so perhaps it has deleted all my snapshots, but one would think there would be an entry somewhere in the logs indicating this has occured? Also, my threshold was set to 85% so one would think there would have been sufficient capacity after a few older snapshots had been removed.

 

https://kb.netgear.com/24478/ReadyNAS-OS-6-Smart-Snapshot-Management

Note: If volume usage exceeds 95% of total capacity, the oldest snapshots will be automatically pruned to attempt to bring the volume usage below 95%. If it was unable to free up some space after deleting some snapshots, it may prune all automatically taken snapshots.

Message 12 of 14
StephenB
Guru

Re: Snapshots have stopped

The default setting was 95%, but they might have dropped it down to 90% after that (my own NAS is set to 90).   If yours is set to 85%, then you probably adjusted it.

 

But I think that explains it.  Your system is over the threshold (~86% full based on your earlier post).  The system probably started deleting the oldest snapshots a while ago - bringing the free space back up to 85%.  But your main volume continued to grow and eventually the system deleted them all. 

 

So the solution is to expand the volume.  Though you could bump the setting up a bit for now, I wouldn't go above 90. 

Message 13 of 14
darthy
Apprentice

Re: Snapshots have stopped

It looks like you are 100% correct, when I search through syslog using Notepad++ for 'snapshot' I can see that last snapshot was created on May 5 and on Jun 29 there was a notice indicating that it was deleted due to regular pruning activity.

 

On May 1, 2 and 5 there are a large number of entries indicating that Snapshots were deleted due to low free space on the volume. In fact if I filter the Logs on the GUI by 'Snapshot' I can see these entries as well.

 

In terms of feedback to the ReadyNAS team, what is really frustrating here is that I receive daily emails from my NAS re the regular Prune activities, but received nothing regarding the Snapshots being deleted due to low free disk space. It would be really good to see these and have something indicate in them that this isn't a normal prune activity.

 

What would be even better is a daily e-mail indicating that Snapshots are not being created due to low free disk space, this would really help close the loop on this.

 

Anyway, many thanks StephenB for helping me track this down, really appreciate all the help and quick responses. I'll add another drive and hopefully things return to normal. I'll also run the various maintenance jobs you suggested to make sure it's all healthy.

Message 14 of 14
Top Contributors
Discussion stats
  • 13 replies
  • 2348 views
  • 1 kudo
  • 2 in conversation
Announcements