Hi, I am lookingto establish the optimal migration process from my RN102 (Arm based CPU), containing 2 x6TB drives currently set up as individual volumes under JBOD and holding roughly 6TB of data, to a RNDU6000 (Intel based CPU) containing 6x2TB drives currently set up under Raid 5 (drives currently empty )
A goal is to swap out two 2TB drives and replace them with the two 6TB drives from the RN102 as part of the process, to give greater overall capacity in the RNDU6000
I don't have any other interim backup desinations, so need to manage the entire data transfer process safely just using these drives and NAS devices.
I understand I can not simply swap the 6 TB drives between devices without losing the contents.
I understand from my reading that with X-Raid active, I can mix drive sizes using Raid 5 array and that this would minimse lost space compared with other raid options, while maintaining some redundancy, and that effectivily the largest disk gets sacrificed for this?
Would an optimal approach here be to:
1- Remove 2 x 2TB drives from RNDU6000 , recreate a smaller new raid 5 volume from 4x2TB disks - which should give mt 6TB storage space
2 - Rsync or otherwise transfer the data from RN102, pretty much filling up the RNDU6000
3- Pull the 2x6TB disks from RN102, and plug them into RNDU6000, destroying whatever is on them (assuming this doesnt happen automatically), and letting the Raid5 volume automatically rebuild itself to larger size usinng these (presumably result will be 4x2TB+1x6TB=14TB?)
Is this a plausible and optimally efficient approach? and is my data likely to survive it? Is there anything I am missing?
For example inder X-Raid do I need to introduce the 6TB drives one at a time to destination NAS, letting volume expand first before adding the next one?
Does the fact that I am adding larger disks to volume after the smaller ones are already set up (and pretty much full) create any problems for the data already present while volume size is reconfiguring itself?
OK, I cant seem to erdit my own post, but I posted the wrong drive specs for the RN102 I am trying to migrate from
It currently contains 1 8 TB drive and on 4 TB drive - not 2 6TB drives as I had mentioned.
If Raid 5 will cost me the space on the the largest drive, 8TB is a big hit - wondering what strategy might now be optimal to maximise storage given this new info?
Ok. So you have 6x2TB in the Ultra now, and you want 4x2TB+4TB+8TB.
If you use XRAID in the Ultra, then you'd end up with a 12 TB RAID-5 volume (sum the disks and subtract the smallest), which actually does match the total space you have on the RN102 now. But it is wasting 4 TB of space (and is only 2 TB bigger than your current volume size). You can fix that by upgrading one of the remaining 2 TB drives to 8 TB - but when you do that you will run into one of the expansion limits of the 4.2.x firmware. You can overcome that by converting the Ultra to OS-6 now (while it is still empty).
Another option is to switch to flexraid on the Ultra, and create a 4x2TB RAID-5 volume (6 TB capacity). Then create 4 TB "D" RAID-0 volume and an 8 TB "E" RAID-0 volume. Then you'd have 18 TB of space across the 3 volumes, with one volume having RAID redundancy.
A third option is to leave the disk configurations as they are for now, and use the RN102 to back up the Ultra. That assumes that 10 TB is large enough to do what you want.
You should of course first check that your FTP problem is resolved by the Ultra. If it isn't, you could then convert it to OS-6 and retest.
The Ultra6 already has OS6.10.2 set up by previous owner.
Is X-Raid incompatible with manually denoting say one or two of the 6 disks in the Ultra6 chassis as separate JBOD volumes? Or am I stuck with using FlexRaid if I want to do that?
I was thinking maybe have 5x2Tb in raid 5, the 8TB as JBOD volume, and sticking the 4TB disk in my PC, giving me 16TB total in the NAS.
Over time, as and when I upgrade or replace one of the 2TB drives, say another 8Tb drive, that coud be my prompt to kill the 8TB JBOD volume and let it merge with X-Raid volume at same time?
Would be nice if X-Raid could handle the expansion of that volume automatically for me, rather than making me start again (which Flex Raid seems to need?)
Meanwhile I figure that whatever happens with the bottleneck FTP issue, I am probably better off comitting to migrating to the Ultra6 anyway as fundamentally better hardware?
With OS-6, Flexraid is more flexible (also a bit more complicated). You can
switch to flexraid
destroy the current volume
remove two disks
re-create the a new RAID-5 volume. Choose a volume name that doesn't match the ones on the RN102.
You should then be able to export the volumes on the RN102 and then add those two disks to the Ultra. It should import the volumes and the shares. Note I haven't needed to export/import myself - but @Sandshark has played around with this, so hopefully he will chime in.
Do make sure you have a backup of the RN102 files though - there's always a chance something will go wrong.
The Ultra6 already has OS6.10.2 set up by previous owner.
Is X-Raid incompatible with manually denoting say one or two of the 6 disks in the Ultra6 chassis as separate JBOD volumes? Or am I stuck with using FlexRaid if I want to do that?
XRAID only handles one volume. So you will need to use FlexRaid.
Over time, as and when I upgrade or replace one of the 2TB drives, say another 8Tb drive, that coud be my prompt to kill the 8TB JBOD volume and let it merge with X-Raid volume at same time?
Switching back to XRAID doesn't always work. But if you leave the RAID-5 volume untouched, you should be able to destroy the 8 TB jbod volume. Then switch back to XRAID and add the disks. It'd be best to unformat the jbod disk, so the NAS isn't confused when you re-insert it.
If you left a slot empty, you could also leave the 4x2TB volume alone, and convert the 8 TB volume to RAID-1. That path would preserve the data on the jbod volume.
Actually, you can move the drives from the 102 to the Ultra running OS6 and not lose the data. That is one of the advantages of updating a legacy NAS to OS6. You should delete any apps if you are moving between ARM and Intel CPU, as you are. You can re-install them after the migration.
You just can't do it and leave any existing drives in the new unit (Ultr, in your case). And you can't put the original drives form the new one back in and get to any data on them (at least through the GUI) unless you first exported them. But since the 2TB's are empty, you don't care about any data on them, which makes it a lot easier.
I assume you want to add the two 2TB drives to the array, which you can't do if you start with an array of larger drives unless you originally started with at least one 2TB drive.. And, that's the rub. At some point, to get full use of the new drives, you are going to have to start fresh, meaning backing up the data and restoring it. If you think you did at one point start your 102 array with 2TB drives, post the contents of MDSTAT.log from the log ZIP and let us confirm it. In that case, you should be able to add the 2TB's to the existing 102 array once mounted in the Ultra.
I think your best bet (assuming no such luck as having started with 2TB on the 102) is to move the old drives from the 102 to the Ultra, switch to FlexRAID, and then make a separate RAID1 volume out of the 2TB's. That'll give you 2TB more, though you'll have to manually distribute files between the volumes. There is actually a way to make them part of the same volume "under the hood" with SSH, but once you do, the OS may not know how to do additional expansions. And if you're not at last a little familiar with Linux, something could go horribly wrong.
It's not the most efficient use of the drives, so, at some time in the future, add another 8TB, copy the data from the 2TB volume over to the larger one, and retire the 2TB's.
Thank you very much Sandshark, and Stephen for your very helpful and well informed replies.
I had already started and was a day or so into an rsync transfer from RN102 to Untra 6 when I read the suggustion below about possibly just moving the larger drives over, since both enclosures are running OS6.
Cest le via.
At least an opportunity to rationalise folder structures and what gets copied over to new NAS.
The strategy I had already somewhat comitted to by then was to create a Raid 5 array with 5x2TB disks, giving me 8TB, and to move the 8TB from RN102 over to give me the same again . I could then remove a 2TB drive from my PC, and use the 4TB from RN102
This will leave me a pool of 3 spare 2TB drives (counting another one I have sitting around) to swap in to the raid array as drives fail - assuming they don't all need to be identical for raid 5 to use them?
I end up with two 8TB volumes , in conjuction with a cloudbackup service capacity of 5TB (Idrive subsription).
Occured to me that limiting NAS volume size might actually be a good idea if it imposes some sort of discipline/structure that is somewhat aligned with cloudbackup capacity. One volume can serve just as the local backup copy of what gets uploaded to cloud.
My thinking was that the 2TB drives that came with Ultra 6 are nearly 10 years old and are WD green desktop drives , so presumably not going to be hugely reliable for much longer - but that starting with a pool of 3 spares to offset this risk, this might be one way to extract the maximum use from them and to delay new drive purchases for as long as possible, to take advantage of declining cost/TB over time.
This volume might be the logical local backup copy of what goes to cloudbackup service
The 8TB drive is a 3 year old Seagate desktop drive, and while on paper is a lot faster and should be more reliable for the moment than any of the given 2TB drives, there are a lot more eggs in one basket if it fails, with few affordable/practical recovery options as a JBOD - so thinking nothing going to this drive should be irreplacable or vital - until such time as it is pared with another 8TB drive to achieve some redundancy. in a raid array Might end up being general repository for media files and similar in the meantime - which are becoming less precious now that Netflix etc are on the scene.
Interestingly when I tested the newer 8TB drive in RN102 enclosure vs the older 2TB drives in Ultra6 using NAStester benchmark software (checks NAS to windows file read/write speeds), the Ultra6 enclosure tested substantially faster at both reading and writing despite the lower internal I/O speeds - suggusting the enclosure is the key factor, not nominal specs of drive it houses
Also, for the benefit of anyone doing the same, it seems that rsync transfer initiated from the faster NAS enclosure gives best speeds in NAS to NAS transfers, and that this is roughly twice as fast as going via windows (rsync still glacially slow at average 15Mb/s - so moving the whole disk over would definitely have been my first choice. had I not already started)
This will leave me a pool of 3 spare 2TB drives (counting another one I have sitting around) to swap in to the raid array as drives fail - assuming they don't all need to be identical for raid 5 to use them? ...
My thinking was that the 2TB drives that came with Ultra 6 are nearly 10 years old and are WD green desktop drives , so presumably not going to be hugely reliable for much longer - but that starting with a pool of 3 spares to offset this risk, this might be one way to extract the maximum use from them and to delay new drive purchases for as long as possible, to take advantage of declining cost/TB over time.
They don't need to identical models, they just need to be the same size.
FWIW, I have an ancient set of WD20EARS in an NV+ that are still working and are similar age. While I don't deploy green drives now (I use NAS-purposed drives), those particular ones have certainly exceeded my expectations.
As far as cost/TB goes, the market sweet spot there has shifted to larger sizes. 2 TB NAS-purposed drives are about $40 per TB. 8 TB are less than $30.
Interestingly when I tested the newer 8TB drive in RN102 enclosure vs the older 2TB drives in Ultra6 using NAStester benchmark software (checks NAS to windows file read/write speeds), the Ultra6 enclosure tested substantially faster at both reading and writing despite the lower internal I/O speeds - suggusting the enclosure is the key factor, not nominal specs of drive it houses
Even though the RN100 series was newer, it was entry-level. It's performance is limited by it's CPU speed and memory. The current entry-level ReadyNAS is the RN210 series - which is much faster (file transfer performance is similar to the ultra).
Also, for the benefit of anyone doing the same, it seems that rsync transfer initiated from the faster NAS enclosure gives best speeds in NAS to NAS transfers, and that this is roughly twice as fast as going via windows (rsync still glacially slow at average 15Mb/s - so moving the whole disk over would definitely have been my first choice. had I not already started)
The bottleneck for your rsync was again the RN102. NFS or SMB transfer would have been faster, but I think rsync is more robust. So I tend to use that even with the slower NAS.
Moving the disk over would of course been the fastest.
I have the Ultra6 up and running, and in all respects it does seem faster than RN102
However it seems the change over from RN102 has not really solved the FTP problem that motivated the change..
The same two cron jobs initiating ftp sending of large tar archive backups of my websites (>1GB) to ReadyNAS are generating same ftp failed error reports -i.e. report connection being rejected by recepient (now the the Ultra6) while three otherwise identical cron jobs (identical other than sending smaller tar archive filesize ) continue to work fine.
The cron jobs are spaced well apart and have twice as long to complete now as they did on the RN102, while they were all 5 still working normally from there,.
Ultra 6 has no scheduled backups of its own set up, and no apps are running so presume there should not be any resouce conflicts that way.
Ultra6 has 4 GB ram, and I can manually upload the largest filesize to Ultra6 from my PC using FileZilla with no error reports
Weirdly, even though I am being emailed these error reports, I am seeing the files in question nevertheless appearing on the Ultra6.
I can extract the files, and there is no report of archive corruption when I test archive - but a lingering doubt will always hover over them while I continue getting these error reports about them. I really would like to be confident I can restore a corrupted website from one of these archives, with no missing content.
It doesn't sound like the Ultra6 is fundamentally being overtaxed here, even if the RN102 may have been.
Here is an example error report again
<b>Starting ftp transfer:</b><br />
Connected to Normal (my IP address):50000 Mode: Passive</b><br />
Warning: ftp_put(): Opening BINARY mode data connection for /WebsitesLocalBackup/nzarchitecture.tar in /home/n289235/public_html/xcloner/cloner.cron.php on line 265
<b style='color:red'>FTP upload has failed for file /home/n289235/public_html/nzarchitecture/administrator/backups//nzarchitecture.tar ! Stopping ....<br /></b>
Is it possible there is something in OS6.10.1 (Ultra 6 is using this) that just ends an ftp session prematurely, or otherwise causes an ungraceful exit after a cerian file size and/or time limit has passed? Ftp from webhost will be a lot slower than from my PC.
Apparently my C-panel webhost uses PureFTPd for providing its ftp services.
They looked into things at their end but apprently can't find any sign of ftp errors on the server.
They also confirm they have no resource limitations or filesize limitations their end that could cause this error. Only restriction is a speedlimit (which speedlimit is fine for the other smaller files getting ftp'ed to ReadyNAS this way)
I can also confirm that the files still are turning up each day, even when accompanied by error messages - so there is no issue with login creddentials supplied to readyNAS destination, or path supplied, or writability of destination folder on ReadyNAS.
But something appears to be causing the ftp transfer to cut off in a non graceful way at the end, for just these larger files.
The cronjob is being run at my Web hosting company server - and is intiating the sending of sending files via FTP to ReadyNAS.
Yes, I understand the webhost uses PureFTPd for both incoming and outgoing FTP connections to the web server. The client in this context is presumably whatever OS6 uses by default? I do not have any 3rd party FTP ap installed.
Hmm, apparently having all ports fowarded does not help.
I had a response from a developer of the Xcloner application that generates the cron jobs. He believes the error message is consistant with an FTP timeout.
I am not sure where any timout could be comming from, but presumably, because error messages are generated at webhost end, any detected timeout at webhost would relate to lack of some expected response by the ReadyNAS, but only at the end of large enough file.
I have just set the whole range 32768-65535 to forward - will see if that helps - thanks for your suggustions!
How many are needed depends on how many simultaneous uploads the client is trying to make. Since it didn't help, just set it back (making sure to match the forwarding range to the port range configured in the NAS).
I had a response from a developer of the Xcloner application that generates the cron jobs. He believes the error message is consistant with an FTP timeout.
I was thinking that too, and that is why I asked about upload resume. It's possible you are getting timeouts, and upload resume is recovering. Though I would expect them to be logged on the server end, it's possible that they aren't.
Do you have disk spindown enabled? That could cause a timeout at the beginning of the transfer (since the disks might not be spun up). So if that is enabled, you can try disabling it as a test. If it helps, then set up a spindown schedule.
FWIW, if the web host also has an ftp server, you can reverse the client/server relationship by using a NAS backup job to download the folder from the source.
A good idea about reversing roles between ReadyNas and webhost - except that the chron job on Webhost also compresses the website , includes a copy of the website sql database from completely seperate part of webhost (no directly user accessable database files that could be accessed remotely exist, until cron creates them ) before uploading it all as a single TAR file. I don't know how I could instigate that collation and archiving process remotely.
As a test of concept anyway, I set up the Monsta FTP app on ReadyNAS and tried to manually download an example of a 2GB site backup TAR file that generates error reports when sent via cron job.
Though I was able to log on to the web host and view directories via the app, MonstaFTP simply stopped responding once I clicked 'download ' on the file. No error message or progress indicator. Not sure if this is relevent in any way?
I don't know how I could instigate that collation and archiving process remotely.
You could create the tar files with the cron job, but upload the tar to the NAS using a backup job. Deleing the tars automatically might be tricky, but you could potentially create a script that would keep a couple, and delete the oldest before creating a new one.
Though I was able to log on to the web host and view directories via the app, MonstaFTP simply stopped responding once I clicked 'download ' on the file. No error message or progress indicator. Not sure if this is relevent in any way?
Not sure either - it's not a client I've used. Perhaps try installing filezilla or some similar client on a PC, and see if you can download files from the server.
Looks like the issue with Monsta FTP may just have been a built in filesize limit of 128 MB.
No problems struck so far just using a standard system backup job via ftp to do it, though I have to backup whole remote folder rather than individual TAR files this way.
Problems thus seem confined to webhost instigated ftp connection to ReadyNAS
A solution in the end turned out to lie with replacing the standalone Xcloner website backup application I was using on the webhost for a different version of Xcloner (where effectivily the standalone application functionalty was repackaged into a plugin, run from a dummy wordpress website set up for the purpose)
Suddenly the large files made it through again.
I can only assume the original standalone version had hit some sort of undisclosed resource or execution time limit policy on the host that the wordpress based version did not strike - perhaps the host has a more tolerant policy towards Wordpress installations than little known 3rd party app installations.
