2014: Lowest-cost consumer deduplication?

Deduplication is an enterprise feature.

Our ReadyDATA devices have it.

What data would you be looking to deduplicate? Data primarily stored on the NAS or backups from your PC?

Deduplication uses lots of RAM. How much data are you looking to deduplicate?

If it's enterprise only, then I probably can't afford it.

I'd like to dedupe our Minecraft server backups. 8)

We've got a bunch of different modded Minecrafts running on a hosted server, FTB Unleashed, FTB Monster, Sky Factory, etc, about 150gb to backup about once a week.

For each of these, from one day to the next only a couple of the game world region files and player data files will be different, so the actual backups can be deduped to be quite small, accounting only for the changes as people wander around the world.

With dedupe we may be able to increase the Minecraft server backup frequency from weekly to daily without too much increase in needed storage.

Well our ReadyDATA 516 has dedupe as a feature.

Is your Minecraft Server running on Windows or Linux?

The minecraft host is running Ubuntu 64-bit.

*looks at price of ReadyDATA 516* YIPE!

*runs off like a dog with its tail stepped on*

This is a non-profit project, operating off of donations only. I don't see this working out. That box will easily hit $2000 with drives installed. At that price might as well just buy some 4 TB drives for an old NetGear Duo and forget about dedupe.

Looks like OpenDedup would have to be the way for me to go to get this inexpensively, but I would prefer not having to mess around rolling my own NAS from scratch, and then having to worry about reliability.

Crashplan has de-duplication built in. If that were installed on the server and an intel-based NAS or PC, you could potentially do a "friend" backup for free. Files would be encrypted, and you could only restore them using Crashplan. Or backup to their cloud in addition to maintaining your current weekly local backups.

Also, re-reading your original post, I am thinking you might not need full de-duplication. You are suggesting that a couple of files are changing, not that segments of a large file (like a database) are changing.

That suggests that incremental rsync backup to an RN100 series or RN300 series might serve. Only files that are changed are copied (but they are copied in their entirety). Older versions are easily retrieved using the OS6 snapshot feature. The snapshots give you a full view of the original folder at the time the snapshot was taken, but only require the disk space needed to store the files changed in the parent folder after the snapshot was made (thanks to btrfs).