Reply

Re: M4300-28G (GSM4328S) HTTPS Admin "ERR_SSL_PROTOCOL_ERROR"

brwyatt
Aspirant

M4300-28G (GSM4328S) HTTPS Admin "ERR_SSL_PROTOCOL_ERROR"

Netgear support seems to be unable to figure this out, so I'm hoping someone in the community can. I'm trying to enable the HTTPS Admin Interface, but the switch seems to only give me protocol errors in Chrome and using OpenSSL. Below is the full information that I gave to support (with the full steps I have taken in setting this up, except later, different DH_Params files were attempted, but with no change). Am I just out of luck here? I'm starting to think this must be a bug, but support hasn't offered anything yet (after 2 months). (Oh, and don't worry, any Private Keys have been scrubbed 😉 )

 

Firmware was recently updated to latest 12.0.4.8, but issue was originally found with 12.0.2.17.

 

=====

 

So, first I generated the keys, generated the CSR (and edited the openssl.cnf to get the SAN attribute into the signed cert), and signed it using the issuing cert, moving the files I needed (the combined PEM-formatted 2048-bit RSA Cert/Key file, the PEM-formatted CA chain (containing the Root, Intermediate, and Issuing CA certs), and the PEM-formatted Diffie-Hellman parameters file) to a temp directory for the TFTP server (later step)

https://s3-us-west-2.amazonaws.com/netgear29767800/Cert001.png 
https://s3-us-west-2.amazonaws.com/netgear29767800/Cert002.png 
https://s3-us-west-2.amazonaws.com/netgear29767800/Cert003.png 

Then I double-checked the Settings to make sure that Admin Mode was Disabled before stating and that there was no Certificate currently present:

https://s3-us-west-2.amazonaws.com/netgear29767800/InitialSettings001.png 
https://s3-us-west-2.amazonaws.com/netgear29767800/InitialSettings002.png 

Following that, I started up the TFTP server:

https://s3-us-west-2.amazonaws.com/netgear29767800/Upload001.png 

... and then transferred each file:

https://s3-us-west-2.amazonaws.com/netgear29767800/TFTP_Trusted_Root_Success.png 
https://s3-us-west-2.amazonaws.com/netgear29767800/TFTP_DH_Strong_Success.png 
https://s3-us-west-2.amazonaws.com/netgear29767800/TFTP_Server_Cert_Success.png 

Then, I validated that there was a cert present, and enabled HTTPS Admin Mode (and yes, I DID click the "Apply" button):

https://s3-us-west-2.amazonaws.com/netgear29767800/FinalSettings001.png 
https://s3-us-west-2.amazonaws.com/netgear29767800/FinalSettings002.png 

After that, I try and access the HTTPS interface in Chrome, and get the following error:

https://s3-us-west-2.amazonaws.com/netgear29767800/ChromeError001.png 

... and this error using OpenSSL's client interface:

https://s3-us-west-2.amazonaws.com/netgear29767800/OpenSSLError001.png 

But, I can validate this certificate and chain using OpenSSL locally (both specifying the Root cert, and defaulting to using my system cert store, additionally validating I have the cert installed, though, as the above errors point out, it isn't a cert issue, this is a protocol issue, but figure I'll include this for completeness and to remove any doubt or confusion):

https://s3-us-west-2.amazonaws.com/netgear29767800/OpenSSL_Verify.png 


Here are the files (with any sensitive secrets like Private keys or the DH params commented out), in case it helps with your investigation (NOTE: Again, the private key and DH Params have been censored, the certificates have been left unchanged, but anything private/secret that MUST be kept private as been censored, but the block remains with the header and footer to show where they were, the files were uploaded uncensored during the process above and have ONLY been censored when uploaded for this case)

* SSL Trusted Root Certificate PEM File: https://s3-us-west-2.amazonaws.com/netgear29767800/chain_with_root.pem 
* (note: I've also tried with the certs in this file in the reverse order, with no success)
* SSL Server Certificate PEM File: https://s3-us-west-2.amazonaws.com/netgear29767800/combined.pem 
* (note: While in this example, I'm using a 2048-bit key pair, I've tried using 1024-bit and had the same errors)
* SSL DH Strong Encryption Parameter PEM File: https://s3-us-west-2.amazonaws.com/netgear29767800/dhparam.pem 
* (note: this is a 1024-bit parameter file, attempting to use 2048-bit causes the file to be rejected by the switch, which is unfortunate)

Additionally, here are links to more human-readable versions of the certs used (shows all the X509 data in a human-readable format):

* Root: https://s3-us-west-2.amazonaws.com/netgear29767800/root_HR.pem 
* Intermediate: https://s3-us-west-2.amazonaws.com/netgear29767800/intermediate_HR.pem 
* Issuing: https://s3-us-west-2.amazonaws.com/netgear29767800/issuing_HR.pem 
* Server: https://s3-us-west-2.amazonaws.com/netgear29767800/server_HR.pem 

Message 1 of 8

Accepted Solutions
brwyatt
Aspirant

Re: M4300-28G (GSM4328S) HTTPS Admin "ERR_SSL_PROTOCOL_ERROR"

With atian's help, I have found the solution.

 

In addition to "pem format" and "include the private key with the server certificate", any chain CA certficates must be appended to the server certificate file, not the trusted root file, and the Trusted root pem file should only contain the root. Doing this has resulted in the switch properly serving HTTPS and a complete chain of trust to the root CA in my computer's trust store.

 

I want to be VERY clear here, though. Other than "PEM Formatted", no other guidance or information is offerred in any documentation I was able to find or anything offered by support. Nowhere does it say that the private key is to be prepended to the "Server Certificate PEM file", nor does it offer guidance on where to add or append or upload CA chain files (many systems I've used before use separate files for the server private key, the server certificate, and the CA chain certs). This REALLY needs to be documented, and would have saved me two and a half months struggling with support that doesn't have a clue about how this is supposed to work, either (I'm guessing the internal documentation doesn't mention this, either).

View solution in original post

Message 8 of 8

All Replies
DanielZhang
NETGEAR Expert

Re: M4300-28G (GSM4328S) HTTPS Admin "ERR_SSL_PROTOCOL_ERROR"

Hi brwyatt,
Welcome to NETGEAR community!Smiley Happy


Currently M4300 just support TLSv1.0 and SSLv3.0.

But there will be a new updata to support TLSv1.2 for M4300 soon due to our internal security team are working on it.

 

So could you please change protocol to TLSv1.0 and try again for  certificate download on M4300?

 

Thanks,

Daniel.

NETGEAR Employee.
Message 2 of 8
brwyatt
Aspirant

Re: M4300-28G (GSM4328S) HTTPS Admin "ERR_SSL_PROTOCOL_ERROR"

Unfortunately, TLS 1.0 is enabled by default, as is SSL 3.0. I've attempted the same steps with SSLv3.0 disabled, but the result is the same.

 

It is good to hear that TLS 1.2 is "in the works", though it is worth mentioning that Chrome does support TLS 1.0, so things should be working, but Chrome, FireFox, and OpenSSL s_client all return protocol errors.

 

I've attached the output of running testssl.sh to this ticket. Strangely, this seems to show that the switch is seeming to support TLS 1.1 and TLS 1.2? Either way, double checking OpenSSL s_client and forcing -tls1 still shows the same protocol errors:

 

139847755073176:error:0D07207B:asn1 encoding routines:ASN1_get_object:header too long:asn1_lib.c:157:
139847755073176:error:0D068066:asn1 encoding routines:ASN1_CHECK_TLEN:bad object header:tasn_dec.c:1205:
139847755073176:error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509
139847755073176:error:1409000D:SSL routines:ssl3_get_server_certificate:ASN1 lib:s3_clnt.c:1237:

 

Which would, to me, seem to indicate that the switch is manipulating or munging the certificate in some way before it sends it. I'll re-iterate that prior to uploading the cert/key to the switch, I am able to locally verify the certificate (in PEM format) using OpenSSL, and have generated and installed certs using the same methods to multiple other services/applications (RabbitMQ, Apache, etc).

 

Either the switch is expecting some non-standard format or has some extra requirements that aren't called out in the documentation (and aren't checked for on upload or on attempt to activate the HTTPS Admin interface), or something is severely broken in the SSL/TLS implementation (or some combination of both). I'm really hoping I'm just doing it wrong, but so far I've been unable to verify that, and the documentation offers no guidance on how to do this at all (neither has support, for that matter), so I'm kinda stuck hoping I can find someone who can point me in the right direction.

Message 3 of 8
brwyatt
Aspirant

Re: M4300-28G (GSM4328S) HTTPS Admin "ERR_SSL_PROTOCOL_ERROR"

To add some further detail to this issue, as best I can tell by the errors in OpenSSL and from in Firefox, it looks as though the switch is mangling the certificate it sends to the browser. I've tried generating a new cert and re-uploading to the switch (via TFTP), but have again been met with the same errors. The switch does seem to check/validate the uploaded combined certificate/key file, but I'm guessing it is only checking that it is in valid PEM format. I haven't tried any other signature algorithms or keylengths (I've been sticking with 2048-bit keys and SHA256), but I doubt this is a problem as it should only matter to the client; the server (switch) shouldn't be touching that at all.

 

It would be nice to be pointed in some direction that gives some level of guidance on what formats or restrictions the switch has for the keys and other SSL-related files beyond just "PEM format" and "must be the correct format". For instance, nowhere does it mention how to upload the private key, I had to find that on StackOverflow that you need to append it to the Certificate file. It would be incredibly helpful if it was documented somewhere if there are restrictions/requirements on keylength, signature algorithms, DH params, etc. I've been unable to get this information from support (so far, the closest thing to a "response" I've gotten is "we Googled the error in Chrome, have you tried deleting your Hosts file?" which is pretty unhelpful in this context for this issue), and the documentation/manual is severely lacking anything that might be useful in this situation. At this point, any information that might be in anyway relevant would be appreciated or helpful (like the bit about TLS 1.1 and 1.2, even though it ultimately doesn't help in this case, that is very helpful/useful information).

Message 4 of 8
atian
Aspirant

Re: M4300-28G (GSM4328S) HTTPS Admin "ERR_SSL_PROTOCOL_ERROR"

Hi Brwyatt, 

I didn't find the steps how you generate the chain_with_root.pem. How did you generate it?

 

 

 
 
Message 5 of 8
brwyatt
Aspirant

Re: M4300-28G (GSM4328S) HTTPS Admin "ERR_SSL_PROTOCOL_ERROR"

Strictly speaking, that file is not "generated", it is the concatination of the root and intermediary CA certs, as is the usual standard.

 

You can see the file here: https://s3-us-west-2.amazonaws.com/netgear29767800/chain_with_root.pem

 

To parse all three certs (and also showing they are valid): `curl -s https://s3-us-west-2.amazonaws.com/netgear29767800/chain_with_root.pem |awk -v cmd='openssl x509 -noout -subject' '/BEGIN/{close(cmd)};{print | cmd}'`

 

This same file/format works for Apache and RabbitMQ (though, admittedly, the Root cert is usually ommitted and only the 2 intermediaries are needed (as the client will already have the Root in their trust store), but it works either way in Apache). I do want to stress here, I'm not getting "untrusted" errors in the browser. OpenSSL, FireFox, and Chrome are all returning errors indicating that the server (that is to say the switch) is sending back garbage. It appears to actually properly handshake, but then returns a malformed certificate.

Message 6 of 8
brwyatt
Aspirant

Re: M4300-28G (GSM4328S) HTTPS Admin "ERR_SSL_PROTOCOL_ERROR"

Just heard back from support. No answers yet, but they had me double check the switch-generated key/cert and also a zip archive of key/cert and dhparams they generated.

 

I'm seeing some things standing out:

 

  1. Self-signed certs (I'm using a cert signed by a signing CA that has an intermediary CA and then a root CA)
  2. SHA1 (I'm using SHA256. SHA1 is considered untrusted by modern browsers!)
  3. 1024-bit keys (I'm using 2048-bit keys. Seriously, who uses 1024-bit?)

Nowhere in any documentation I've found has it said anything about restrictions on CAs, signature algorithms, or key length. But, if I had to guess, I'd place money on the key length, but this really should be checked when the key/cert is uploaded to the switch (it already checks it for other errors). If I had to guess, it is probably truncating the key and causing issues when sending the cert.

 

I've replied back to support with this info, so we'll see what they say. I don't really feel like re-generating and signing new certs until I hear back if this is, in fact, the issue. If it is, it would be really REALLY nice if this was actually documented anywhere and save someone else hours of effort and months of waiting for support to figure it out.

Message 7 of 8
brwyatt
Aspirant

Re: M4300-28G (GSM4328S) HTTPS Admin "ERR_SSL_PROTOCOL_ERROR"

With atian's help, I have found the solution.

 

In addition to "pem format" and "include the private key with the server certificate", any chain CA certficates must be appended to the server certificate file, not the trusted root file, and the Trusted root pem file should only contain the root. Doing this has resulted in the switch properly serving HTTPS and a complete chain of trust to the root CA in my computer's trust store.

 

I want to be VERY clear here, though. Other than "PEM Formatted", no other guidance or information is offerred in any documentation I was able to find or anything offered by support. Nowhere does it say that the private key is to be prepended to the "Server Certificate PEM file", nor does it offer guidance on where to add or append or upload CA chain files (many systems I've used before use separate files for the server private key, the server certificate, and the CA chain certs). This REALLY needs to be documented, and would have saved me two and a half months struggling with support that doesn't have a clue about how this is supposed to work, either (I'm guessing the internal documentation doesn't mention this, either).

Message 8 of 8
Top Contributors
Discussion stats
  • 7 replies
  • 3118 views
  • 0 kudos
  • 3 in conversation
Announcements