TLS handshake failure while connecting to OpenVPN server on Nighthawk R7450

[devansh]

I am trying to setup a VPN server on my router so that I can connect to my home internet while traveling. I have enabled the OpenVPN server with the Netgear DDNS and downloaded the configuration files. But when I try to connect to the OpenVPN server using the configuration files on OpenVPN client GUI or android app, it doesnt work. It waits for connection and then there is an error saying TLS negotiation failed.

Mon Aug 22 09:52:58 2022 MANAGEMENT: >STATE:1661154778,WAIT,,,,,,
Mon Aug 22 09:53:58 2022 TLS Error: TLS key negotiation failed to occur within 60 seconds (check your network connectivity)
Mon Aug 22 09:53:58 2022 TLS Error: TLS handshake failed
Mon Aug 22 09:53:58 2022 TCP/UDP: Closing socket
Mon Aug 22 09:53:58 2022 SIGUSR1[soft,tls-error] received, process restarting
Mon Aug 22 09:53:58 2022 MANAGEMENT: >STATE:1661154838,RECONNECTING,tls-error,,,,,
Mon Aug 22 09:53:58 2022 Restart pause, 5 second(s)
Mon Aug 22 09:54:03 2022 Re-using SSL/TLS context
Mon Aug 22 09:54:03 2022 LZO compression initializing
Mon Aug 22 09:54:03 2022 Control Channel MTU parms [ L:1654 D:1212 EF:38 EB:0 ET:0 EL:3 ]
Mon Aug 22 09:54:03 2022 MANAGEMENT: >STATE:1661154843,RESOLVE,,,,,,
Mon Aug 22 09:54:03 2022 Data Channel MTU parms [ L:1654 D:1450 EF:122 EB:411 ET:32 EL:3 ]
Mon Aug 22 09:54:03 2022 Local Options String (VER=V4): 'V4,dev-type tap,link-mtu 1590,tun-mtu 1532,proto UDPv4,comp-lzo,cipher AES-128-CBC,auth SHA1,keysize 128,key-method 2,tls-client'
Mon Aug 22 09:54:03 2022 Expected Remote Options String (VER=V4): 'V4,dev-type tap,link-mtu 1590,tun-mtu 1532,proto UDPv4,comp-lzo,cipher AES-128-CBC,auth SHA1,keysize 128,key-method 2,tls-server'
Mon Aug 22 09:54:03 2022 TCP/UDP: Preserving recently used remote address: [AF_INET]203.194.96.4:22974
Mon Aug 22 09:54:03 2022 Socket Buffers: R=[65536->65536] S=[65536->65536]
Mon Aug 22 09:54:03 2022 UDP link local: (not bound)
Mon Aug 22 09:54:03 2022 UDP link remote: [AF_INET]203.194.96.4:22974
Mon Aug 22 09:54:03 2022 MANAGEMENT: >STATE:1661154843,WAIT,,,,,,
Mon Aug 22 09:55:04 2022 TLS Error: TLS key negotiation failed to occur within 60 seconds (check your network connectivity)
Mon Aug 22 09:55:04 2022 TLS Error: TLS handshake failed

So far I have tried connecting from a different network, connecting from the same network, using older version 2.4.7 of OpenVPN client GUI. I have tried changing port number to another random port as well as open ports like 443 for TUN and TAP. I have tried using TCP instead of UDP. I have tried completely turning of all firewalls. No matter what I do, it simply wont connect. Some help will be really appreciated.

[devansh · Accepted solution]

Hi, unfortunately the only solution is to wait for Netgear to fix it in their next update. And given how old the router is, I dont think it is going to happen. The only workaround I found is to install 3rd party firmware like DD-WRT or wireguard on the router.

[Kitsap]

You cannot test the connection to an Open VPN server from a device that is connected, either by Wi-Fi or Ethernet, to the LAN where the server is running.

Any changes you make to the configuration on the router has to be followed by creation of new configuration files from the router.  Then the new files have to be copied to the client device.  On a computer the configuration files are read each time the client software is started.  On a cell phone you have to import the configuration files into the application.

Sometimes the router configuration is problematic if you do not allow the VPN connection to access all sites on the internet and home network.  The attached has worked well for me for years on Windows and Android client devices.

[devansh]

Yes, I have tried the same configuration you have attached. Also I have tried connecting from inside the LAN as well as from another network. Once even from another city. Still hasn't worked. Also I downloaded new configuration file and used that every time I made any changes. And each and every time it fails the TLS handshake

[Kitsap]

What operating system and version are you running on your computer?  Windows? IOS?

Have you tried to connect with a different client device?  Android?  Different computer?

TLS stands for Transport Layer Security and has been part of internet connections for some time.  It has also evolved and is on version 1.3 or more.  There is a chance of some security or anti-virus setting on your computer causing the problems.

I have found there are some free Wi-Fi access locations, coffee shops, restaurants, hospitals, etc., that have their internet access configured to prevent connection with VPN services.  The VPN simply will not connect.

[devansh]

I am using windows 11 and I have tried another laptop with windows 10 and Android as well. I have tried multiple public WiFis, my home WiFi and mobile internet using a sim card.

Public WiFis block VPNs by blocking known IPs of VPN services. Since this is my own personal IP I am trying to connect, I don't think they should be able to block that.

Anyway TLS version could be an issue. I'll look into that for sure. Thank you.

But I have tried using older versions of OpenVPN as many people have suggested in other posts. That hasn't worked. Maybe I'll try going back to older firmware on my router as well.

[Kitsap]

I use the Open VPN version 2.5.7-I602 client program on Windows 10 Pro computers.  A combination of Dell laptops and desktops.

For verification purposes in Windows, you download from the router page and unpack four configuration files from the zip file.  Then copy the files to the directory,  C:\Program Files\OpenVPN\config\ of your client device.  These files are read when the OpenVPN program initializes.

On Android phones and tablets, I use OpenVPN Connect version 3.3.0.

For Android there is only one configuration file, and it must be unpacked from the zip file and copied as far up in the device directory tree you can go, not in a folder.  This file is used when you import a profile into the OpenVPN application.

Any time you make a change on the VPN configuration page of the router, you save the changes and download a new set of configuration files.

[Sealz]

Hi devansh,

Did you find any solution to your problem? I am facing the same issue now. Please share if you have any fix. Thanks!

[devansh · Accepted solution]

Hi, unfortunately the only solution is to wait for Netgear to fix it in their next update. And given how old the router is, I dont think it is going to happen. The only workaround I found is to install 3rd party firmware like DD-WRT or wireguard on the router.

[Sealz]

Hi Devansh,

Thanks for your reply. 

FYI, I realized that I made a mistake just like what Kitsap pointed out before: I tried to connect to the OpenVPN server from a device within the LAN where the server is running. After I switched the WiFi to the hotspot of one cellphone which used its data plan, I was able to connect aother cellphone to the OpenVPN server.

Thanks, devansh and Kitsap, for your help! Have a good weekend!