Who Me Too'd this topic

Nicholi
Guide

PSA: Expired LetsEncrypt Root Certificate Potential Fix

For anyone on ReadyNAS 6 you may have noticed some issues recently with the machine being unable to work with sites utilizing letsencrypt certs.

 

The solution first starts with assuring you have the new root certificate LetsEncrypt is utilizing installed. It should be there already, but just to confirm it can be found here:

/etc/ssl/certs/ISRG_Root_X1.pem
# which should be a symbolic link to here
/usr/share/ca-certificates/mozilla/ISRG_Root_X1.crt

And to confirm you have the correct certificate, you can check its fingerprint via this command

 

$ openssl x509 -in /etc/ssl/certs/ISRG_Root_X1.pem -noout -fingerprint -sha256
SHA256 Fingerprint=96:BC:EC:06:26:49:76:F3:74:60:77:9A:CF:28:C5:A7:CF:E8:A3:C0:AA:E1:1A:8F:FC:EE:05:C0:BD:DF:08:C6

If it is not present that is unfortunately beyond the scope of my TED talk :P, and you got a lot of other things to deal with likely. Now you might think that's enough for everything to work (it is for literally everyone else on modern devices) however since we are stuck on Debian 8 (thanks for the EOL netgear) we are also stuck on an ancient version of OpenSSL (1.0.1t is the latest I have access to). The new LetsEncrypt rollout has 2 intermediate paths to validate the chain of trust in their certificates. Unfortunately one of these paths is using the just recently expired DST Root CA X3 certificate, expired on 2021-09-30T14:01:15Z. Our version of OpenSSL has a bug which will cause it to always fail if one of the intermediate paths fails, in our case the path using DST Root CA X3 because it is expired. Which is why everything was fine up until that day. Anyone on newer devices, or likely just anything with OpenSSL 1.1+, also likely has no issues bcause this bug isn't present for them. The failed intermediate path is ignored, and the good path is trusted.

Thankfully the only thing you should need to do in this situation is simply remove the DST Root CA X3 certificate, and all SSL verification should succeed again. On my system at least that was:

# moved to /tmp rather than deleting, just in case you may still need it
sudo mv /usr/share/ca-certificates/mozilla/DST_Root_CA_X3.crt /tmp
# --fresh not strictly necessary, however it cleans up symbolic links to certificates which no longer exist
sudo /usr/sbin/update-ca-certificates --verbose --fresh

 

After this you should be able to again utilize any site that uses a LetsEncrypt certificate. A very simple test would be to use curl with letsencrypts site (of course they use their own certs :P)

 

$ curl -v -s 'https://letsencrypt.org/' -o /dev/null
.. garbage DEBUG output ...
* Trying 184.72.19.87...
* TCP_NODELAY set
* Expire in 149988 ms for 3 (transfer 0x55826216d410)
* Expire in 200 ms for 4 (transfer 0x55826216d410)
* Connected to letsencrypt.org (184.72.19.87) port 443 (#0)
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [85 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [3897 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [180 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [102 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
{ [1 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-ECDSA-AES256-GCM-SHA384
* Server certificate:
* subject: CN=lencr.org
* start date: Aug 6 01:03:53 2021 GMT
* expire date: Nov 4 01:03:51 2021 GMT
* subjectAltName: host "letsencrypt.org" matched cert's "letsencrypt.org"
* issuer: C=US; O=Let's Encrypt; CN=R3
* SSL certificate verify ok.
> GET / HTTP/1.1
> Host: letsencrypt.org
> User-Agent: curl/7.64.0
> Accept: */*
>
< HTTP/1.1 200 OK
< cache-control: public, max-age=0, must-revalidate
... additional HTTP headers and response

As long as you see the HTTP/1.1 200 OK line in response, you are ok. You've passed TLS/SSL and have actually hit the webserver.

 

 

Just for full sample, here is the expected output from curl while the expired certificate is installed. You can see TLS failing.

$ curl -v 'https://letsencrypt.org/' -o /dev/null
... garbage DEBUG output ...
* Trying 54.241.246.27...
* TCP_NODELAY set
* Expire in 149984 ms for 3 (transfer 0x5644e02b1410)
* Expire in 200 ms for 4 (transfer 0x5644e02b1410)
* Connected to letsencrypt.org (54.241.246.27) port 443 (#0)
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [85 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [3897 bytes data]
* TLSv1.2 (OUT), TLS alert, certificate expired (557):
} [2 bytes data]
* SSL certificate problem: certificate has expired
* Closing connection 0
curl: (60) SSL certificate problem: certificate has expired More details here: https://curl.haxx.se/docs/sslcerts.html curl failed to verify the legitimacy of the server and therefore could not establish a secure connection to it. To learn more about this situation and how to fix it, please visit the web page mentioned above.

 

 

And thankfully we can continue using our ReadyNAS machines for a little while longer. Though I would urge anyone to start jumping ship/planning to jump ship. As the aging Debian 8 packages are becoming extremely cumbersome to deal with.

Who Me Too'd this topic