I use a RN526X as my primary NAS and use a RN314 for backup and for the ReadyDR jobs.
I have has ongoing issues where the 314 becomes Read Only after about two weeks of backups. I did a factory restore about 2 weeks ago and have not shut off the 314 since and everything was working fine until a couple of days ago.
Now, the backup job is continuing to run correctly but all the DR jobs fail with this error message: ReadyDR job failure: Input/output error [DR Folder Name]
The 314 is fully accessible on the network and via ReadyCloud. I have checked and recycled the router the 314 is connected to and it seems fine.
Any idea what is wrong and how to correct?
Regards,
David
Model: RN526X| ReadyNAS 526X 6-Bay with up to 60TB total storage
As for ReadyDR issues there are a lot fixes included on the 6.8.0 beta firmware. You may wanna try this firmware version, please check the check the beta firmware release notes from here.
The Read Only problem, like the current input/output error only affects the ReadyDR jobs. The backup job has always, and continues to run without issue.
The backup job is set up on the Backup tab of the 526 and backs up the entire 526 data share to a share on the 314 called Backup. Within the backup folder is a folder called Data (written by the backup job), and within Data are all of the Shares and files on the 526. That backup runs each night at 10:00 pm and continues to run properly.
The ReadyDR jobs are set up on the 526 as "push" jobs to the 314. Therre is a dedicated ReadyDR job for each share on the 526 and that corresponds to a dedicated DR share on the 314. The DR shares were created as part of setting up the ReadyDR process.
Previously, they would run for a week or two and then the shares on the 314 would become "Ready Only" and the push jobs would fail.
While the 314 has not yet become read only, the Ready DR jobs are continuing to fail each night and I get the input/output error.
The only thing different now from precious times is that previously, I was letting the 314 power itself down after the Ready DR job ran (they run just after midnight), at about 1:00 am. Since doing a Factory Restore a couple of weeks ago, I did not power down the 314 until after I started getting a No Route to Host mesage for all of the backup and RaedyDR jobs. At that point I rebooted the 314 and the backup job started working again but the ReadyDR jobs fail.
I did check the IP addresses for both units and they are correct I have also rebooted the 526 but that did not correct the issue.
Hope that helps. Let me know if you need anything else.
Regards,
David
Model: RN526X| ReadyNAS 526X 6-Bay with up to 60TB total storage
There are concerns raised exactly similar to this problem we are encountering and the fix will be provided on the beta firmware or on the official release of firmware 6.8.0, you may want to try to beta firmware and you may download it from here.
Thanks for your response. The 314 has plenty of available capacity with well over 1TB free.
I know that the ReadyDR shares are Read-Only for everything but the ReadyDR jobs but it appears something happens to them along the way if the 314 is rebooted.
I continued to get the input/output error message through yesterday's backup attempt. In an effort to resolve that I rebooted both the 526 and the 314 last evening and sure enough. This morning, instead of the input/output error message, I am back to getting the read-only error message.
For each ReadyDR job I receive the following message:
ReadyDR job failure: Read-only file system [Ready DR Share Name]
This has always happened after the ReadyNAS units are powered down and and restart the next day. Now that this has occurrfed, the 314 is basically locked and the entire volume is read only.
I tried to create a new share and get this error:
Error:
1013050001
Could not create requested share.
The 526 volume is still OK and I can create new shares and write data to the shares with no problem but the 314 is locked.
This has happened every time I set up the ReadyDR volumes.
I believe you are referring to firmware on 6.6.5 and 6.8.1 on your last response. It does appear that there was an issue created for this concern and the fix was provided on firmware version 6.8.0 that time. However in your case you are encountering the same problem. Please try the following steps and check if that will make any difference.
1. Disable quota.
2. Don't keep too much snapshots.
3. Set a speed limit.
4. Try not to let two ReadyDR jobs running at the same time.
I believe you are referring to firmware on 6.6.5 and 6.8.1 on your last response. It does appear that there was an issue created for this concern and the fix was provided on firmware version 6.8.0 that time. However in your case you are encountering the same problem. Please try the following steps and check if that will make any difference.
1. Disable quota. 2. Don't keep too much snapshots. 3. Set a speed limit. 4. Try not to let two ReadyDR jobs running at the same time.
Regards, Marty_M NETGEAR Community Team
Thank you for the suggestions.
In this case I don't have quota's running (unless there is something set by defaut
I am not sure what "too many snapshots" would be, I am using the smart snapshots which have been running since about october of last year when I migrated from OS4 -> OS6, so there are probably 10 monthly snapshots and the others as expected.
I will try putting a limit on speed.
I have a series of about 15 ReadyDR jobs which are in the queue and I assume run sequentially, I created them using the web interface, is there something I need to do inorder for them to not run at the same time.
Also, the ones which fail due to being read only filesystems will not start even if run independently, how do I correct for this?
I see that 6.8.1 is now available. Is this issue something which has been addressed in this build?
Of so, is it a "source" machine issue or a "destination" machine issue for readyDR. i.e. can I update the destination machine only and see this issue resolved or do I need to update both machine. The Source machine is my "production" machine, the destination machine is currently in a test configuration.
For the backup not to run at the same make sure they are set to run on a scheduled time, please refer to ReadyDR backup options here. For the 6.8.1 firmware there is nothing on the release notes for the ReadyDR, but it is always good to troubleshoot with the latest firmware running on the device. You have an option to upgrade the firmware and check if that will make a difference.
I got a good readydr of each share before they became read-only. Currently expanding from four 3tb + two 2tb drives to four 3tb drives + two 10tb drives. (A pair of iron wolf drives) final reshape claims 70 hours to complete.
Then I can try some options and see if I can get readydr stable.
I understand you are planning to do a volume expansion and will try to upgrade the firmware afterwards. Please let us know if upgrading the firmware will help with the concern.
We’d greatly appreciate hearing your feedback letting us know if the information we provided has helped resolve your issue or if you need further assistance. If your issue is now resolved, we encourage you to mark the appropriate reply as the “Accept as Solution” or post what resolved it and mark it as solution so others can be confident in benefiting from the solution.
The Netgear community looks forward to hearing from you and being a helpful resource in the future!
We’d greatly appreciate hearing your feedback letting us know if the information we provided has helped resolve your issue or if you need further assistance. If your issue is now resolved, we encourage you to mark the appropriate reply as the “Accept as Solution” or post what resolved it and mark it as solution so others can be confident in benefiting from the solution.
The Netgear community looks forward to hearing from you and being a helpful resource in the future!
Regards, Marty_M NETGEAR Community Team
No solution as of yet, will try update and testing in the next week or so.
Model: RN626X | ReadyNAS 626X 6-Bay with Intel® Xeon® Quad-Core Server Processor
Volume expansion (two new 10TB drives so about 9TB of free space on the production machine) is now complete.
In the process of updating the backup server (the new 626) from 6.7.5 to 6.9.0 the filesystem became unstable for some reason (it may have already had issues based upon trying ((unsuccessfully)) to delete one of the ReadyDR shares).
At this point I have performed a factory default on the 626 and I am starting over with it as the receipient of the ReadyDR shares from the primary machine.
It is interesting that with the old ReadyDR shares gone it appears there is no way to edit the existing ReadyDR jobs on the source machine such that the new ReadyDR shares get created. So, I have disabled all of the old ReadyDR jobs and I am re-creating them one by one.
In this case I am trying to just create them one at a time and then let it finish prior to creating the next one.
One question is if I only want to manually trigger the ReadyDR process should I set them to "Disable" and then use "start" to trigger the copy?
Current copy rate from the Pro6 to the 626x is about 80MBytes/sec, I had tried some bonding experiments and only got marginally better performance with much added complexity.
In this case I have a direct connection between the two boxes on a dedicated subnet which keeps all of the backup traffic off my network.
Based upon the issues I had going from 6.7.5 to 6.9.0 on the backup machine I think I am going to leave the first machine at 6.7.5 for the time being.
Model: RN626X | ReadyNAS 626X 6-Bay with Intel® Xeon® Quad-Core Server Processor
All of my ReadyDR jobs ran correctly (I ran them one by one and waited for one to finish before starting the next one).
I have not yet tried
1) Running them automatically when the snapshots occur
2) rebooting the destination ReadyNAS (the 626 in this case) and seeing if everything remains running ok.
In this case the source (pushing) machine is running 6.7.5 on a Pro6 and the destination is running 6.9.0 on a 626 with a single 10TB drive. (I am about to go to Raid1 and then to Raid 5 after I complete burn in of the new 10TB Seagate Ironwolf drives, my burn is is 1) read all sectors in Seatools, 2) Write all tracks in Seatools, 3) Read all sectors in Seatools, 4) verify no bad sectors on the drive using the ReadyNAS both before and after expanding the array onto the disk)
Model: RN626X | ReadyNAS 626X 6-Bay with Intel® Xeon® Quad-Core Server Processor
So far so good. I have rebooted the destination NAS and it appears that the volumes are still writeable.
In addition I turned off quotas on the destination NAS, it is interesting that when you do that visibility into what part of the data is snapshots and which is user data goes away. For the destination device that works for me (little user data) but for the primary NAS (Currently a Pro6) I would prefer to be able to see what part of the data is just in snapshots but is not part of the current data.
After running some tests I have now gone from one 10TB drive, then two 10TB drives in raid, I am currently testing the next 10TB drive (read all sectors, write all tracks, read all sectors - which takes about a week using USB2) to expand and then finish the backup server. The plan is that in addition to the ReadyDR shares I will have traditional RSYNC shares on the backup machine. Then I have two rotating external drive backup sets for worst case recovery of which one is always stored offsite.
Model: RN626X | ReadyNAS 626X 6-Bay with Intel® Xeon® Quad-Core Server Processor
