NETGEAR is aware of a growing number of phone and online scams. To learn how to stay safe click here.
Forum Discussion
Solved
checksumming outside of raid?
I am just curious if anyone here is doing any supplemental checksum checking of their data beyond what btrfs does, for example to check the data before backing it up to an external source...or compar...
after much reflection I have decided that using some kind of filesystem level checksumming is not a bad idea, but only on data that I don't expect to change. I will refer to that here as "archive" data. In which case a simple md5sum checksum saved somewhere will suffice for this purpose, can be created with bash script, should be created once per file and never changed (unless the file is actually updated). That way at any time I can check to see if the file is still as it was originally saved.
There are a variety of ways a file could be corrupted, besides just bit rot. Bit rot will theoretically be found and maybe corrected by btrfs. But other kinds of corruption from a variety of problems might end up as looking like perfectly legit file changes at the block level and would not be spotted. Which is where file system checksums come in. So if I keep these on my important "archive" data, and include it in all backups, at some point in the future I can always check the data and check my backups to see which are still good.
periodically checking them is not a bad idea to make sure its not corrupted and being backed up over the top of good backup. If I use snapshot versioning in the backup, then I can still get back to a good version too that way, and having the checksum will help me know.
so I see it as being useful, but only for "archive" data. Trying to track checksums of data that will be changed intentionally makes it very hard to distinguish between legit changes and corruption because it depends on file date and some forms of corruption would actually update the date. And or the date itself in meta data could also be compromised...so...I think it is quite complicated to keep updating checksums at filesystem level and not reliable anyway, so might as well forget it, just rely on BTRFS to do checksum checking at block level and then use checksums on "archive" data only. and no special tool needed, readynas as md5sum command which can be scripted in bash pretty easily to do that.
over and out on this topic.
I am just curious if anyone here is doing any supplemental checksum checking of their data beyond what btrfs does, for example to check the data before backing it up to an external source...or comparing checksums of the main volume vs backup volume, etc, etc, etc,
FWIW, I use an old utility called winsfv to checksum my media files - one checksum file per folder. I was doing that before BTRFS, and just kept it up after I started using OS-6.
I don't verify the checksums regularly. But I do have them, just in case.
As far as backups go, I have enough of them that it is unlikely for bitrot to hit the same files on all of them.
after much reflection I have decided that using some kind of filesystem level checksumming is not a bad idea, but only on data that I don't expect to change. I will refer to that here as "archive" data. In which case a simple md5sum checksum saved somewhere will suffice for this purpose, can be created with bash script, should be created once per file and never changed (unless the file is actually updated). That way at any time I can check to see if the file is still as it was originally saved.
There are a variety of ways a file could be corrupted, besides just bit rot. Bit rot will theoretically be found and maybe corrected by btrfs. But other kinds of corruption from a variety of problems might end up as looking like perfectly legit file changes at the block level and would not be spotted. Which is where file system checksums come in. So if I keep these on my important "archive" data, and include it in all backups, at some point in the future I can always check the data and check my backups to see which are still good.
periodically checking them is not a bad idea to make sure its not corrupted and being backed up over the top of good backup. If I use snapshot versioning in the backup, then I can still get back to a good version too that way, and having the checksum will help me know.
so I see it as being useful, but only for "archive" data. Trying to track checksums of data that will be changed intentionally makes it very hard to distinguish between legit changes and corruption because it depends on file date and some forms of corruption would actually update the date. And or the date itself in meta data could also be compromised...so...I think it is quite complicated to keep updating checksums at filesystem level and not reliable anyway, so might as well forget it, just rely on BTRFS to do checksum checking at block level and then use checksums on "archive" data only. and no special tool needed, readynas as md5sum command which can be scripted in bash pretty easily to do that.
over and out on this topic.
NETGEAR Academy
Boost your skills with the Netgear Academy - Get trained, certified and stay ahead with the latest Netgear technology!
Join Us!
