× NETGEAR will be terminating ReadyCLOUD service by July 1st, 2023. For more details click here.
Orbi WiFi 7 RBE973
Reply

Re: 626x used as readydr destination deleted all snapshots

BotanyBay
Tutor

626x used as readydr destination deleted all snapshots

Readynas 626 being used as a destination for ReadyDR started deleting all of the snapshots in a destination share.
Primary machine shows 3 years (59 snapshots). I noticed as the machine went from 50% utilized (12TB of 27TB).
I have lots from a few days before and all looks normal (looked in snapper) and then it looks like pruning started happening. Now show about 4TB of data.
Was manually starting ReadyDR for each share (one at a time and wait to complete) on the primary machine, also running 6.9.4. Logged into 626(backup destination) and noted drastic drop in total stored volume.
Verified that the root file system on both machines is normal (no file system over 25% full)
No additional apps installed except SSH.
Primary machine has 5.8TB free of 20TB.
Backup machine is normally off, only turned on for backups.
Any thoughts?

Setting up for another full backup of primary machine to external disk, not overwriting prior backups just in case there is a file system issue on the primary.

Thank you in advance for any recommendations
Model: RN626X|ReadyNAS 626X – 6 Bays with Intel® Xeon® Quad-Core Server Processor
Message 1 of 12
BotanyBay
Tutor

Re: 626x used as readydr destination deleted all snapshots

Additional details:
Appears the "complete" ReadyDR shares on the destination are the ones I had completed earlier in the day. The "empty" ones are the ones yet to go.
I did get the following message on the last ReadyDR backup(source machine): ReadyDR: ReadyDR job failure: Protocol error [share name]
A volume scrub was running on the source machine
Also note that after a reboot I get the following error on each ReadyDR on the source machine: ReadyDR: ReadyDR job failure: Operation not permitted [Share Name]
This happens immediately (and has for as long as I have been using ReadyDR) and I restart, all is well at that point. I thought I localized that first fail to key exchange issues...
I have always had the ReadyDR jobs disabled and initiate them with the "gear", they are labeled as "instant" backups (not periodic)
Source machine data looks healthy at this point.
Message 2 of 12
XtheUnknown
Apprentice

Re: 626x used as readydr destination deleted all snapshots

I believe there is an un-diagnosed bug in 6.10.2 where "smart" snapshot management on a ReadyDR target share will inappropriately delete some or all of the snapshots (thereby rendering the backup useless).

 

My expereience was that (the only) two snapshots on the target, which were taken at hours other than midnight, were deleted under the 48-hour rule... Not so smart after all...

 

Message 3 of 12
BotanyBay
Tutor

Re: 626x used as readydr destination deleted all snapshots

Interesting...
I am running 6.9.4 on both machines at the moment. I wonder if the bug is in that version.
If I turn smart snapshots off on the destination readynas i assume it would not delete snapshots at all.
On the other hand it would have random additional snapshots from the source from whenever it ran (ie hourly, daily, weekly)...
Interesting that it ran fine until last week and then decided to prune all of the snapshots... it in fact deleted ALL of them on the backup server, most recent first it appeared... or at least when I looked there were only a few of the ones from 2016 left at that point.
If I kick off the ReadyDR job on the source it does rebuild with all of the old snapshots ending up on the backup. Just trying to understand why it did it and how to avoid in the future... would be sad for this to happen when the backups were critical!
Message 4 of 12
StephenB
Guru

Re: 626x used as readydr destination deleted all snapshots


@XtheUnknown wrote:

My expereience was that (the only) two snapshots on the target, which were taken at hours other than midnight, were deleted under the 48-hour rule... Not so smart after all...

 


It is possible that the file system reached the 95% full threshold?

Message 5 of 12
XtheUnknown
Apprentice

Re: 626x used as readydr destination deleted all snapshots

No.

Message 6 of 12
XtheUnknown
Apprentice

Re: 626x used as readydr destination deleted all snapshots

Indeed. In my case I had another backup of the data, but I have now turned off smart snapshot pruning on all my ReadyDR shares....

Message 7 of 12
StephenB
Guru

Re: 626x used as readydr destination deleted all snapshots


@XtheUnknown wrote:

Indeed. In my case I had another backup of the data, but I have now turned off smart snapshot pruning on all my ReadyDR shares....


I don't use ReadyDR, but have done that for all my shares anyway.  Custom snapshots give you control over retention for each share - which prevents the snapshots from eventually taking over all the space. I combine that with "only take snapshots when changes".

Message 8 of 12
BotanyBay
Tutor

Re: 626x used as readydr destination deleted all snapshots

Really unfortunate, up to this point the backups on the secondary machine have been wonderful. Very reliably capturing all of the old snapshots and getting rid of everything but the monthly archives. Guess I must have gotten lucky as the destination machine is rarely on...

I guess I will have to reevaluate my backup methodology going forward. Perhaps I will go to having rsync copies on the backup machine having their own snapshots in addition to the readydr shares.

If I turn off smart snapshots on the readydr volumes on the backup machine I assume I can manually delete intermediate readydr snapshots (ie the excess hourly, daily, and weekly snapshots) without impacting the stability of the readydr snapshots on the backup machine.
Message 9 of 12
BotanyBay
Tutor

Re: 626x used as readydr destination deleted all snapshots

So, not a solution but at least a path forward, I did the following on backup server:
--Turn off smart snapshots on all destination ReadyDR shares.
--Set time based snapshot management, tried 600 months (50 years) which the interface converted to 4161 days (11.4 years)
--Turned off access to snapshots on destination 626x readynas (results in not being able to see snapshots even in interface unless turned back on)
--Disable network access completely on ReadyDR shares

Will verify if run readydr backup and then enable snapshot access that the data is there.

Will also clone a snapshot to the local backup machine and make sure that works as expected....

Thoughts on other things to do on a locked down backup machine?
Message 10 of 12
XtheUnknown
Apprentice

Re: 626x used as readydr destination deleted all snapshots

Well (FWIW and IMHO, etc.), that may be overkill.

 

The only problem I've experienced was with the "smart" snapshot management - having reverted to "custom" and set a long retention period (3 years) or large number of snapshots (128) I've had no further issues.

 

Snapshots are always read-only anyway, so I can't see what disabling snapshot access would gain. The only options you have from them (and therefore ReadyDR backups) is to extract/restore individual file(s) to another location or clone the entire share (space permitting).

 

 

 

Message 11 of 12
StephenB
Guru

Re: 626x used as readydr destination deleted all snapshots


@XtheUnknown wrote:

Well (FWIW and IMHO, etc.), that may be overkill.

 


Though I don't use ReadyDR, I agree it sounds like overkill.  Eliminating snapshot pruning should have been enough, since snapshots can't be deleted over the network unless you are logged into the NAS with ssh and using the command line interface.

Message 12 of 12
Top Contributors
Discussion stats
  • 11 replies
  • 1213 views
  • 0 kudos
  • 3 in conversation
Announcements