As part of a careful backup and verification process performed to upgrade from 6.9.4 to 6.10.5 on a ReadyNAS 626 and a ReadyNAS Pro6 I discovered that the Pro6 does not want to cleanly shutdown. Otherwise both units are working well.
I understand that the Pro6 is not "Supported" using OS6 which is why I am very careful in my upgrade process. The Pro6 is being used purely as a ReadyDR destination.
Prior to the upgrade process I performed a Defrag, Scrub, Balance, Disk Test on both of the units and downloaded logs on both units. In each case the "/" partition was 16% to 17% utilized.
The Pro6 took a very long time to reboot immediately after the update (several hours) but eventually did restart itself without manual intervention.
After the upgrade I again downloaded logs, performed a Defrag, Scrub, Balance on the unit. Then a final "download logs". I thought it shutdown cleanly at this point.
The Pro6 is a simple setup with 4 drives of 10TB and was created cleanly under 6.9.4 as a backup target after the original drives were moved to the new 626 chassis back in 2017? The number of drives has been expanded twice with the same size drive each time (now 4 drives). The Pro6 has 3GB of memory installed. Other than SSH no other applications have been installed, virus scans ARE NOT enabled. I do see that when running ReadyDR from the 626 and pushing 100MBytes/Sec across the interface I do see most of the memory being consumed on the Pro6, however, the ReadyDR appears to be running nicely and 0 swap is being utilized. Total number of snapshots/ReadyDR target is around 100, total number of targets is about 20.
The 626 appears to be running cleanly at this point in time.
The Question: Any idea why the Pro6 might be getting stuck on shutdown. I have tried shutting down from the GUI, Shutting down from the front button. It both cases I get the same result.
I am considering rerunning the Defrag, Scrub, Balance in the event that one of those did not close out cleanly and something is keeping the unit from shutting down.
One observation: When I performed a shutdown last night and left it for 12 hours the snapshot pruning did occur some 10 hours AFTER the shutdown was initiated... So, it appears that the machine is not "hung" but rather "waiting" for something to finish.
FWIW, I didn't have that issue with my own Pro 6 - it is on a power schedule, so a failure to shutdown would have been immediately apparent. What services do you have enabled? Are there any operations that might hold off a shutdown (for instance a raid resync)
Also - are you running the stock memory, or have you increased it?
Re: 6.10.5 not cleanly shutting down on Readynas Pro6
I can say itr does not sound like a hardware problem. When shutdown doesn't occur because of a hardware issue, the OS normally unloads, but it just never shuts the unit off. Since your OS appears to still be running, that's likely not the case. That's not goiong to help you solve the problem, but hopefully does put you at ease a bit.
Have you tried issuing the rn_shutdown command from SSH?
FWIW, I didn't have that issue with my own Pro 6 - it is on a power schedule, so a failure to shutdown would have been immediately apparent. What services do you have enabled? Are there any operations that might hold off a shutdown (for instance a raid resync)
Also - are you running the stock memory, or have you increased it?
Interesting that 6.10.6 is now available 🙂 I am a slow adopter for updates, watching the logs for a period of time to see if people are having trouble, then update the backup server, wait a while and then update the primary machine. I figure if the Pro6 is happy the 626 will definitely be happy!
I will probably try 6.10.6 and see if the behavior changes.
As far as services, the only thing beyond stock which is enabled is SSH. Definitely no RAID Resync in progress.
I am running 3GB of memory which if I remember correctly was adding a 2GB stick to the stock 1GB stick. I could change to two of the 2GB sticks if that becomes a problem.
It is interesting that a full scrub takes about 10 days on this box, the 626 with a similar amount of data takes about 4 days 🙂
I can say itr does not sound like a hardware problem. When shutdown doesn't occur because of a hardware issue, the OS normally unloads, but it just never shuts the unit off. Since your OS appears to still be running, that's likely not the case. That's not goiong to help you solve the problem, but hopefully does put you at ease a bit.
Have you tried issuing the rn_shutdown command from SSH?
Yeah, I don't think it is a hardware problem. Was running cleanly prior to the update and like you say because the logs are showing that it is still doing things 12 hours after issuing the shutdown command the OS is not crashing. Definitely acts like something is not unloading / terminating correctly.
I am not familiar with "rn_shutdown" does that just terminate the readynas tasks rather than a normal shutdown?
I have also not tried issuing a shutdown command from SSH which might be something interesting to try.
I did just try issuing a shutdown from the GUI while SSH was open. I got one ps -elf off and thought that the SSH was going to stay open and then the SSH stopped taking commands but did not close the window. Interesting.
Re: 6.10.5 not cleanly shutting down on Readynas Pro6
A question about updates.
It appears that the update firmware GUI will take either the .zip or the .img within the zip file.
Is there a preference when updating? I would think that the .zip file would provide some additional file integrity checks if the upload process runs the unzip locally on the ReadyNAS?
Re: 6.10.5 not cleanly shutting down on Readynas Pro6
What you ghot wehen you issued just a shutdown is why I suggested rn_shutdown, which properly shuts the NAS OS down and powers off. I was just wondering if there was any additional information it gave during the shutdown that might help determine the issue.
What you ghot wehen you issued just a shutdown is why I suggested rn_shutdown, which properly shuts the NAS OS down and powers off. I was just wondering if there was any additional information it gave during the shutdown that might help determine the issue.
Gave "rn_shutdown" a try.
Looks like the same behavior as shutting down from the GUI or from the power button (i.e. pressing power briefly twice)
I will give it a few hours to try and shutdown and then reboot and check the logs.
Anywhere specific in the logs which you would check other than looking at the GUI log entries?
Thanks again! I really appreciate the suggestions, no data at risk is always a good time to try and clean up little issues like this.
What you ghot wehen you issued just a shutdown is why I suggested rn_shutdown, which properly shuts the NAS OS down and powers off. I was just wondering if there was any additional information it gave during the shutdown that might help determine the issue.
Gave "rn_shutdown" a try.
Looks like the same behavior as shutting down from the GUI or from the power button (i.e. pressing power briefly twice)
I will give it a few hours to try and shutdown and then reboot and check the logs.
Anywhere specific in the logs which you would check other than looking at the GUI log entries?
Thanks again! I really appreciate the suggestions, no data at risk is always a good time to try and clean up little issues like this.
I will try rummaging around in the "download logs" and see if anything is amis. I wonder if any of the "defrag, scrub, balance" got hung up. I will check for the start and stop of each in the logs.
Perhaps one of the three did not exit cleanly causing the system to not be able to shutdown cleanly.
What you ghot wehen you issued just a shutdown is why I suggested rn_shutdown, which properly shuts the NAS OS down and powers off. I was just wondering if there was any additional information it gave during the shutdown that might help determine the issue.
Gave "rn_shutdown" a try.
Looks like the same behavior as shutting down from the GUI or from the power button (i.e. pressing power briefly twice)
I will give it a few hours to try and shutdown and then reboot and check the logs.
Anywhere specific in the logs which you would check other than looking at the GUI log entries?
Thanks again! I really appreciate the suggestions, no data at risk is always a good time to try and clean up little issues like this.
I will try rummaging around in the "download logs" and see if anything is amis. I wonder if any of the "defrag, scrub, balance" got hung up. I will check for the start and stop of each in the logs.
Perhaps one of the three did not exit cleanly causing the system to not be able to shutdown cleanly.
Well, looked through all of the logs and did not see anything obvious to the casual observer.
I upgraded to 6.10.6 and the behavior remains the same at this point, i.e. still stuck on shutdown but no other ill effects observed.
Re: 6.10.5 not cleanly shutting down on Readynas Pro6
Defrag and Balance had no impact
Started a fresh scrub and then stopped it after a bit (takes 10 days on that machine) and still no change in behavior.
Other than not wanting to finish shutting down everything appears fine.
I did see that the 626 does see the network connection to the Pro6 go away (they have a direct connection for doing backups with no switch in between) so it is at least getting to the point of terminating the network on the Pro6 as it is coming down.
Re: 6.10.5 not cleanly shutting down on Readynas Pro6
Updated the 626 to 6.10.6 as everything is nicely backed up at this point and I like having both machines running the same version when possible.
The 626 updated cleanly.
Note that both the Pro6 and the 626 respond in the GUI with "server error" when I upload the update. It is very consistent and does not depend on if the update was done with a .zip or a .img file.
Confirmed that the utilization of / on both boxes is about 17% (no change)
Re: 6.10.5 not cleanly shutting down on Readynas Pro6
Is the NAS still reachable on the network? If so, can you open, or do you still have open an SSH session? What processes are shown as still active with top?
Re: 6.10.5 not cleanly shutting down on Readynas Pro6
Tried the GUI --> No response (network unreachable)
Tried Ping --> No response (network unreachable)
The other ReadyNAS (The 626) shows the point to point network path as down (i.e. the LED display shows "network configuration changed" and the point to point network is down) fairly quickly during the shutdown sequence.
Tried the GUI --> No response (network unreachable)
Tried Ping --> No response (network unreachable)
The other ReadyNAS (The 626) shows the point to point network path as down (i.e. the LED display shows "network configuration changed" and the point to point network is down) fairly quickly during the shutdown sequence.
Does it behave differently if you disconnect the point to point network?
Re: 6.10.5 not cleanly shutting down on Readynas Pro6
Interesting thought... (could be related to secondary network)
Pulled cable while it was trying to shutdown (i.e. it has been trying for a couple of days) and no change -- Would not expect any change...
With secondary network cable disconnected forced power off and allowed to boot up cleanly, then restarted from GUI -- no change, still won't shutdown cleanly... I will let it try for awhile today with the network cable between the two units disconnected.
So does not appear to be related to secondary point to point network being either active or not active when starting the shutdown.
