Looking for thought on SSH performance on RN systems
Having issues with SSH connections on all the ReadyNAS systems running 6.10.10.
The problem is all the connection stages take close to 2 minutes to complete. It's insane and I can't figure out why.
This occurs on 3 systems (RN204 and RN104) so yes low-powered but this is crazy slow.
I've tried with clients (ssh, putty, and python using paramiko) on Win7, Win10, Win11, Linux (Kali, Parrot, and Ubuntu) and same results every time. Slow as can be. Once the session is live the response isn't bad it's just the connecting is soooo slow.
The python script connects to the server at root@IP which takes almost a minute and half to get to a prompt, then issuing an "ls /" which takes another minute to complete.
The putty and ssh connections are very similar, in that the password challenge is presented nearly instantly, but then takes a minutes for the banner it's nearly 3 minutes! Windows SSH is identical timing.
All the demos are connecting to the RN204, and the same happens with the RN104s which are just as fast, ummmm slow. There are a couple of videos showing top on the parrot sessions.
The putty and ssh connections are very similar, in that the password challenge is presented nearly instantly, but then takes a minutes for the banner it's nearly 3 minutes! Windows SSH is identical timing.
If you
connect with Windows ssh
log in
immediately log out when you get the command prompt (after your password entry)
reconnect with Windows ssh
Does the reconnect also take 2-3 minutes?
Also, do you have disk spindown enabled on the NAS?
Re: Looking for thought on SSH performance on RN systems
Thanks @StephenB, good thinking on the disk spin down and while it's enabled it's not a factor. AFAIK the ssh client (always running in the background) doesn't need disk access and is just like any other background task. I confirmed the device doesn't spin up the drive on ssh connect until you start interacting with the OS.
I tried your test and found the exact same response both times, nearly 3 minutes with connect, disconnect, connect:
Last login: Sat Sep 7 10:21:51 2024 from thefactory root@ReadyNAS-204:~# exit logout Connection to 192.168.1.200 closed.
C:\Users\iceweasel>
You may have noticed the first connect message shows the prior connect from the readynas itself. I had been doing testing with tunneling with paramiko to work around the slow connect. But that got me thinking...what if I ssh from the RN to the RN?
If from an active ssh session on the RN I ssh at the LAN address I get the same results
If I try again but accessing with localhost (127.0.0.1) the connection process is nearly instant. Which would suggest there's something on the LAN interfering? I'm not sure of that but may need to try a direct connect PC->RN and see if things go faster but that will take some work and I'm hoping there may be something else easier than reconfiguring the NAS for a test.
I also checked that only the required system tools are enabled in the web interface.
There's an interesting tangent problem I need to address with uninstalling some old apps, but none of those are running there're just taking up disk space and the UI remove fails. But I'll save that until this gets resolved.
Re: Looking for thought on SSH performance on RN systems
Thanks again @StephenB. I did a bit more testing and found some interesting things...
first I was not too sure my PC wasn't a problem so I did some things to clear up some of the "questionable" networks stuff... removed npcap, unistalled wireshark, etc.
I found the password to banner did speed up which I found very odd as there's really nothing on the client side that should be affecting that. But the initial password challenge of 16 to display "password:" challenge was not acceptable.
So I dug a little deeper and looked at the ssh -vvv details a little closer. I found there was a 16 second (or so) timeout from when the client sends an authentication packet to when the server responds with the password challenge. The debug was helpful.
Looking online I see a common issue with this is a default setting to use DNS for something in this process which... doesn't make much sense unless some CA or other 3rd party is involved.... so I followed the advice of others and ignored the note at the top of the /etc/ssh/sshd_config and added this::
UseDNS no
I now get the password challenge immediately!!
One note on my setup, is my NAS devices are internal only so they couldn't reach an external NAS if they had to. I didn't run any tests to see if exposing them to the WAN would change the behavior, but for the knowledge of the greater good I may do that when I have a few minutes to spare.
Thanks for the suggestions and nudges in the right direction to find and remedy this problem!!
Re: Looking for thought on SSH performance on RN systems
In the interest of discovery... confirmed, if the NAS can access the WAN (DNS really) the "UseDNS no" is not needed.
If the ReadyNAS systems are behind a firewall and only available to the LAN then yes, the "UseDNS no" makes a big difference in the authentication timing.
In the interest of discovery... confirmed, if the NAS can access the WAN (DNS really) the "UseDNS no" is not needed.
If the ReadyNAS systems are behind a firewall and only available to the LAN then yes, the "UseDNS no" makes a big difference in the authentication timing.
My own NAS are behind a NAT router, so they can connect to the internet, but not be reached over the internet. As I posted above, I am not seeing the authenication LAG that you were seeing before you made this change.
Re: Looking for thought on SSH performance on RN systems
Well that's interesting! More interesting that when allowed to access the WAN the delay disappears, block WAN and the delay reappears. I wonder what's different. Maybe different version of ssh server? EDIT: reread your NAS is behind the router but can reach the WAN but not be reached from an external connection? That would make sense why yours works, mine cannot access the WAN nor be accessed from the WAN. These are LAN only until I open them up to poll for updates.
I'm now back on the original task which exposed this and need to figure out a way to recover a file which was accidently deleted from both a local copy on a laptop and the original on the NAS. The local copy can't be recovered and at this point I'm expecting the RAID can't be configured so I can try some other linux options. I'm also assuming (because I can't find it listed) the filesystem may be a custom netgear brew.... think I just need to accept it's gone. =(
Re: Looking for thought on SSH performance on RN systems
The OS6 file system is standard BTRFS on top of a standard MDADM RAID. If you've been using the snapshot capability of BTRFS, that's how you recover the file. If you've not, you are likely out of luck.
"XRAID" is not a unique file system. It's a set of rules and automations for managing BTRFS and MDADM at a level more users can understand. In earlier generations, XRAID used LVM.
Re: Looking for thought on SSH performance on RN systems
Well fud, thanks though. I was expecting this would be the final outcome but had hoped for better. Got to learn something new about the ssh connection method so not all bad!
