Reply

Re: PSA: Expired LetsEncrypt Root Certificate Potential Fix

Nicholi
Guide

PSA: Expired LetsEncrypt Root Certificate Potential Fix

For anyone on ReadyNAS 6 you may have noticed some issues recently with the machine being unable to work with sites utilizing letsencrypt certs.

 

The solution first starts with assuring you have the new root certificate LetsEncrypt is utilizing installed. It should be there already, but just to confirm it can be found here:

/etc/ssl/certs/ISRG_Root_X1.pem
# which should be a symbolic link to here
/usr/share/ca-certificates/mozilla/ISRG_Root_X1.crt

And to confirm you have the correct certificate, you can check its fingerprint via this command

 

$ openssl x509 -in /etc/ssl/certs/ISRG_Root_X1.pem -noout -fingerprint -sha256
SHA256 Fingerprint=96:BC:EC:06:26:49:76:F3:74:60:77:9A:CF:28:C5:A7:CF:E8:A3:C0:AA:E1:1A:8F:FC:EE:05:C0:BD:DF:08:C6

If it is not present that is unfortunately beyond the scope of my TED talk :P, and you got a lot of other things to deal with likely. Now you might think that's enough for everything to work (it is for literally everyone else on modern devices) however since we are stuck on Debian 8 (thanks for the EOL netgear) we are also stuck on an ancient version of OpenSSL (1.0.1t is the latest I have access to). The new LetsEncrypt rollout has 2 intermediate paths to validate the chain of trust in their certificates. Unfortunately one of these paths is using the just recently expired DST Root CA X3 certificate, expired on 2021-09-30T14:01:15Z. Our version of OpenSSL has a bug which will cause it to always fail if one of the intermediate paths fails, in our case the path using DST Root CA X3 because it is expired. Which is why everything was fine up until that day. Anyone on newer devices, or likely just anything with OpenSSL 1.1+, also likely has no issues bcause this bug isn't present for them. The failed intermediate path is ignored, and the good path is trusted.

Thankfully the only thing you should need to do in this situation is simply remove the DST Root CA X3 certificate, and all SSL verification should succeed again. On my system at least that was:

# moved to /tmp rather than deleting, just in case you may still need it
sudo mv /usr/share/ca-certificates/mozilla/DST_Root_CA_X3.crt /tmp
# --fresh not strictly necessary, however it cleans up symbolic links to certificates which no longer exist
sudo /usr/sbin/update-ca-certificates --verbose --fresh

 

After this you should be able to again utilize any site that uses a LetsEncrypt certificate. A very simple test would be to use curl with letsencrypts site (of course they use their own certs :P)

 

$ curl -v -s 'https://letsencrypt.org/' -o /dev/null
.. garbage DEBUG output ...
* Trying 184.72.19.87...
* TCP_NODELAY set
* Expire in 149988 ms for 3 (transfer 0x55826216d410)
* Expire in 200 ms for 4 (transfer 0x55826216d410)
* Connected to letsencrypt.org (184.72.19.87) port 443 (#0)
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [85 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [3897 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [180 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [102 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
{ [1 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-ECDSA-AES256-GCM-SHA384
* Server certificate:
* subject: CN=lencr.org
* start date: Aug 6 01:03:53 2021 GMT
* expire date: Nov 4 01:03:51 2021 GMT
* subjectAltName: host "letsencrypt.org" matched cert's "letsencrypt.org"
* issuer: C=US; O=Let's Encrypt; CN=R3
* SSL certificate verify ok.
> GET / HTTP/1.1
> Host: letsencrypt.org
> User-Agent: curl/7.64.0
> Accept: */*
>
< HTTP/1.1 200 OK
< cache-control: public, max-age=0, must-revalidate
... additional HTTP headers and response

As long as you see the HTTP/1.1 200 OK line in response, you are ok. You've passed TLS/SSL and have actually hit the webserver.

 

 

Just for full sample, here is the expected output from curl while the expired certificate is installed. You can see TLS failing.

$ curl -v 'https://letsencrypt.org/' -o /dev/null
... garbage DEBUG output ...
* Trying 54.241.246.27...
* TCP_NODELAY set
* Expire in 149984 ms for 3 (transfer 0x5644e02b1410)
* Expire in 200 ms for 4 (transfer 0x5644e02b1410)
* Connected to letsencrypt.org (54.241.246.27) port 443 (#0)
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [85 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [3897 bytes data]
* TLSv1.2 (OUT), TLS alert, certificate expired (557):
} [2 bytes data]
* SSL certificate problem: certificate has expired
* Closing connection 0
curl: (60) SSL certificate problem: certificate has expired More details here: https://curl.haxx.se/docs/sslcerts.html curl failed to verify the legitimacy of the server and therefore could not establish a secure connection to it. To learn more about this situation and how to fix it, please visit the web page mentioned above.

 

 

And thankfully we can continue using our ReadyNAS machines for a little while longer. Though I would urge anyone to start jumping ship/planning to jump ship. As the aging Debian 8 packages are becoming extremely cumbersome to deal with.

Message 1 of 8
ohiomedic
Aspirant

Re: PSA: Expired LetsEncrypt Root Certificate Potential Fix

Thank you so much this fixed the issues I’ve been having.
Message 2 of 8
Jophus
Luminary

Re: PSA: Expired LetsEncrypt Root Certificate Potential Fix

This fixed Logitech Media Server apps list being empty and failed connection to mysqueezebox.com on Readynas 6.

 

You are a STAR!

 

Message 3 of 8
Nicholi
Guide

Re: PSA: Expired LetsEncrypt Root Certificate Potential Fix

Additional note I just found since updating to ReadyNAS OS 6.10.6. You may need to remove the file again and re-run update-ca-certificates, as I believe some of the base level packages (which likely includes ca-certificates) get re-installed, so the old files get installed again.

 

This is likely a necessary step on any ReadyNAS firmware update.

Also the fingerprint of the old file if it helps anyone identify the bad root cert is:

$ openssl x509 -in DST_Root_CA_X3.crt -noout -fingerprint -sha256
SHA256 Fingerprint=06:87:26:03:31:A7:24:03:D9:09:F1:05:E6:9B:CF:0D:32:E1:BD:24:93:FF:C6:D9:20:6D:11:BC:D6:77:07:39
Message 4 of 8
ncameron
Aspirant

Re: PSA: Expired LetsEncrypt Root Certificate Potential Fix

I have done this, apparently successfully, on my ReadyNAS 314 - but I still can't see my Apps listed in LMS. Is there something else I should be looking for? 

Message 5 of 8
Jophus
Luminary

Re: PSA: Expired LetsEncrypt Root Certificate Potential Fix

May I suggest visiting RNXtras site and downloading his package to correct this. No SSH/commands just upload as an app.

 

Turn off LMS

Install app https://rnxtras.com/addons/letsencryptfix 

Turn on LMS

 

Shout out to whocares as one of only a handfull in the community supporting readynas in a wholly voluntary capacity.

Model: RN31600|ReadyNAS 300 Series 6- Bay (Diskless)
Message 6 of 8
ncameron
Aspirant

Re: PSA: Expired LetsEncrypt Root Certificate Potential Fix

****ing brilliant! That did it; after a system reboot. Been wrestling with this for months. 

 

Happy to shout out to whocares, but never heard of it - how do I do that?

 

Thanks 

Message 7 of 8
ncameron
Aspirant

Re: PSA: Expired LetsEncrypt Root Certificate Potential Fix

Just realised who 'whocares' is! I've sent him a donation.

Message 8 of 8
Top Contributors
Discussion stats
  • 7 replies
  • 1556 views
  • 4 kudos
  • 4 in conversation
Announcements