NETGEAR is aware of a growing number of phone and online scams. To learn how to stay safe click here.
Forum Discussion
Solved
Likely Readynas 3-series
Hello, I plan to upgrade from my ReadyNAS Duo that served me well for 5+ years running 2x2TB in Raid 1. Consider to buy 2- or 4-bay (or even 6-bay hope not) NAS and it is for home use with small o...
You could put a suggestion in the Feature Request & Feedback subforum.
You could also make a request at rnxtras.com (not a netgear site)
Hi again,
I would like to change focus on my last question from ECC memory to integrity of backup which is the key question. And sure I admit I do not understand ECC-memory and BTRFS well enough and may questions maybe obscure.
I do understand that BTRFS is very useful if you interact with the NAS to some extent, but too much then you get disk fragmentation. Right now I am more interested in the low frequency of interaction that you have for an archive, i.e. mainly reading files and occasionally adding files in accumulative way. In this context bit-rot protection (BTRFS and raid-configuration) gives added value especially if we consider storage over long time, say 5-10 years and more.
Central to long term archive is of course to have an external backup procedure that maintain the data integrity you have in the primary storage, the NAS with BTRFS. I see some alternatives with: external HD, another NAS (for me older ReadyNAS with “OS4” and EXT3/4), or even a newer NAS under OS6. These different backup solutions have different levels of long time security of data integrity. But right now I think just of the short term backup-procedure.
1) When a file is backed up externally, what ensure that the backup is correct and that a recovery will give back the correct file?
a) I think of incremental backup. I thought that the use of checksum that can be enabled in OS6 and BTRFS also could be used somehow for external backup. I guess that if you do backup to another ReadyNAS under OS6 you might be able to exploit some usage of these checksums to ensure that the (incremental) backup/recovery is correct. Is that true?
b) Use of checksums (or something else) perhaps can be used even if backup is made to a non-BTRFS media?
c) Is here some difference in this aspect using the standards built in backup procedure in OS6 compared with for instance rsync or some other backup-software that can be used?
2) In this context of making backup/recovery I guess that ECC-memory provides an extra level to ensure data integrity when doing backup/recovery - or do I misunderstand this?
3) Would you say that ECC-memory is mainly interesting for intensive use of NAS as server rather than the low intensity use of NAS as archive?
Thus, as a user I wold like to be able to say that once the file is in the archive NAS then data integrity will be kept both at the primary NAS, during backup/recover to external site, and at the backup site.
---
See that in the new FORUM it is my responsibility to close the discussion and I think we are soon there.
janpeter1 wrote:
I do understand that BTRFS is very useful if you interact with the NAS to some extent, but too much then you get disk fragmentation. Right now I am more interested in the low frequency of interaction that you have for an archive, i.e. mainly reading files and occasionally adding files in accumulative way. In this context bit-rot protection (BTRFS and raid-configuration) gives added value especially if we consider storage over long time, say 5-10 years and more.
If this is your main use, then you won't see much (if any) disk fragmentation with btrfs.
1) When a file is backed up externally, what ensure that the backup is correct and that a recovery will give back the correct file?
I've never seen a case where a backup / restore failed silently - usually there's an error reported (either by the disk or by the backup utility). That is not to say that "silent bitrot" is impossible. Just that if it has happened to me I didn't discover it later.
This is obviously very important to you. So I'd suggest that you put in place your own methods of validating data integrity, and not rely exclusively on the file system or RAID.
There are tools that verify when you copy (teracopy is one example). Ensuring that the restore done years later gives back precisely the original file likely requires saving an independent checksum to verify. I do this manually with some media files (creating an SFV file that checksums all the files in the folder).
a) I think of incremental backup. I thought that the use of checksum that can be enabled in OS6 and BTRFS also could be used somehow for external backup. I guess that if you do backup to another ReadyNAS under OS6 you might be able to exploit some usage of these checksums to ensure that the (incremental) backup/recovery is correct. Is that true?
Well, the checksum is verified when you copy to the backup. But the checksum is purely local, so another ReadyNAS doesn't see it, and cannot use it. BTRFS isn't supported right now on the external USB drives, so the checksum isn't computed on the USB drive either.
b) Use of checksums (or something else) perhaps can be used even if backup is made to a non-BTRFS media?
As I mentioned above, there are tools available that can do this. But the ReadyNAS doesn't (and no other NAS does either).
c) Is here some difference in this aspect using the standards built in backup procedure in OS6 compared with for instance rsync or some other backup-software that can be used?
I think I addressed this above.
2) In this context of making backup/recovery I guess that ECC-memory provides an extra level to ensure data integrity when doing backup/recovery - or do I misunderstand this?3) Would you say that ECC-memory is mainly interesting for intensive use of NAS as server rather than the low intensity use of NAS as archive?
I wouldn't say that ECC is related to intensive use. ECC-memory increases reliability for both applications. Clearly that's a particularly important aspect for you, so you should probably consider a NAS that supports it.
Thus, as a user I wold like to be able to say that once the file is in the archive NAS then data integrity will be kept both at the primary NAS, during backup/recover to external site, and at the backup site.
My main advice is that there is always some risk, and at some point you need to accept it. I think you will need to put some of your own measures in place in order to help ensure that your own data integrity threshold is met.
My own approach is multiple copies (I maintain 2 local independent backups of all data, and a fourth copy on a cloud backup server). That doesn't alert me if one copy is somehow corrupted, but it does give me some recovery options if I discover that down the road. I'm not saying that will work for you. But you'll need to sort out what measures are enough.
- From you answer it seems that NAS external backup is the weak link for ensuring data integrity, although at this forum key people iterate the need for external backups.When I download larger software packages (eg python etc) I have encountered the adviceto calculate checksum of the downloaded file and compare with what it should be.And I have encountered the standard MD5 (and easy to check on a Mac in the terminal).Just thought that this use of checksums likely was automated in more serious backup software. Therefore I wondered if Netgears OS6 use that or similar.For OS6 I also expected some emphasis on ensuring data integrity when say backup to another NAS with OS6 at least, but maybe not there yet.Could also be that all backup software do some kind of checksum checks etc, I just do not know.Of course check of checksums is not perfect either.---Part of my archive is a collection of 50 000+ photos. At least two photos are found corrupted after 4 years of raid1, but I am not sure the corruption happened as bit-rot. At least for one of the photos am pretty sure it happened during file transfer at an early stage. For the other photo I found the backup also corrupted, but backup was made at some later stage, and no backup of original, only of NAS. My concern it to minimize this kind of irritations.
janpeter1 wrote:
From you answer it seems that NAS external backup is the weak link for ensuring data integrity, although at this forum key people iterate the need for external backups.I wouldn't call it the "weak link", as there are other steps that are equally vulnerable. For instance, the original copy of the data from your PC to the NAS.
And yes, we do repeat the need for backing up the NAS. RAID arrays and servers do eventually fail, and sometimes they fail in ways that lose data.
If you simply want to verify your photos, you can use something like quicksfv on each main folder. If you want some ability to repair lost data independently from the NAS or backup software you could use quickpar.(also on each folder, not on each photo).
- HiThank you for the tips on 3rd party (windows) software to do checksum tests.After all I am mainly a Mac user and some search on internet gave me the followingthat may be useful to some other Mac users:* General information valid cross platform:* Apple support: https://discussions.apple.com/thread/5162581?start=0&tstart=0- there is a tip to use a facility in Carbon Copy Cloner that I thought interesting as I have the software* Search on checksum on App store gave 20+ products to my surpriseBut after all what I think would be much better is a function in OS6 that does the workor why not a 3rd party plugin that do the job. Since it may take some time to do thechecksum work for an incremental backup, better that the NAS did the work in the background.
NETGEAR Academy
Boost your skills with the Netgear Academy - Get trained, certified and stay ahead with the latest Netgear technology!
Join Us!
