Let’s Encrypt expired root (X3) certificate fix on a CentOS 6 server

First of all, many thanks to the wonderful Let’s Encrypt community for all the help with this, credit where credit is due: https://community.letsencrypt.org/t/rhel-centos-6-openssl-client-compatibility-after-dst-root-ca-x3-expiration/161032

That is a lengthy thread, and after a lot of trial and error and considerable head scratching I wanted to share the fix that worked for us, from start to finish. Hopefully it’ll save you some time so you can get back to your wgets and API calls.

Here are the steps that worked for us:

Go ahead and yum update everything:

yum -y update

If yum says it can’t find any repositories, then fix it using these instructions https://www.getpagespeed.com/server-setup/how-to-fix-yum-after-centos-6-went-eol

Install needed software:

yum install krb5-devel zlib-devel lksctp-tools-devel util-linux make gcc rpm-build

Build yourself an OpenSSL RPM based on the CentOS 7 version. We have to “fix” an issue with the secure_getenv function. This does not exist on CentOS 6 and RHEL did not see fit to handle this gracefully as this version was never meant to run on 6.

curl -o openssl-1.0.2k-21.el7_9.src.rpm https://vault.centos.org/7.9.2009/updates/Source/SPackages/openssl-1.0.2k-21.el7_9.src.rpm
rpm -i openssl-1.0.2k-21.el7_9.src.rpm
cd ~/rpmbuild/SOURCES/
sed -i 's/secure_getenv(/getenv(/g' *patch
cd ../SPECS/
sed -i 's/%patch68 -p1 -b .secure-getenv/#%patch68 -p1 -b .secure-getenv/g' openssl.spec
rpmbuild -bb openssl.spec
cd ../RPMS/x86_64

The last command:

rpm -U openssl-libs-1.0.2k-21.el6.x86_64.rpm openssl-1.0.2k-21.el6.x86_64.rpm

Gives an error:

error: Failed dependencies:
openssl = 1.0.1e-58.el6_10 is needed by (installed) openssl-devel-1.0.1e-58.el6_10.x86_64

So you try to remove it

rpm -e openssl-devel-1.0.1e-58.el6_10.x86_64

But that also gives an error:

error: Failed dependencies:
openssl-devel is needed by (installed) net-snmp-devel-1:5.5-60.el6_10.2.x86_64

Luckily that package is not in use anyway, so:

yum erase net-snmp-devel
rpm -e openssl-devel-1.0.1e-58.el6_10.x86_64
rpm -U openssl-libs-1.0.2k-21.el6.x86_64.rpm openssl-1.0.2k-21.el6.x86_64.rpm

You may have more difficulties here if you have more packages that depend on openssl-devel. Once that’s resolved, you should have an updated version of OpenSSL:

openssl version
OpenSSL 1.0.2k-fips  26 Jan 2017


But still, errors:

wget https://letsencrypt.org
    --2021-10-04 16:43:14--  https://letsencrypt.org/
    Resolving letsencrypt.org... 54.151.57.158, 138.68.50.15, 2604:a880:2:d0::2082:a001, ...
    Connecting to letsencrypt.org|54.151.57.158|:443... connected.
    ERROR: cannot verify letsencrypt.org's certificate, issued by `/C=US/O=Let\'s Encrypt/CN=R3':
      Issued certificate has expired.
    To connect to letsencrypt.org insecurely, use `--no-check-certificate'.


Remove the X3 root certificate from the trust:

cd /etc/pki
grep -rl " X3" .

Find and remove the certificate from these files:

./tls/certs/ca-bundle.trust.crt
./tls/certs/ca-bundle.crt

./ca-trust/extracted/openssl/ca-bundle.trust.crt
./ca-trust/extracted/pem/tls-ca-bundle.pem

I simply used vim and searched for ” X3″ (including the space, without the quotes)

Repeat for other files listed above until there are no more references to X3.

Try wget again:

wget https://letsencrypt.org
    --2021-10-04 16:53:35--  https://letsencrypt.org/
    Resolving letsencrypt.org... 54.241.246.27, 54.151.57.158, 2600:1f1c:471:9d00:1478:99ac:4b21:1cba, ...
    Connecting to letsencrypt.org|54.241.246.27|:443... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: 32087 (31K) [text/html]
    Saving to: `index.html'


Finally, verify that the X1 certificate is in those same four files:

grep -rl "ISRG Root X1" /etc/pki

It should be. If it is not:

cd /etc/pki/ca-trust/source/anchors/
wget https://letsencrypt.org/certs/isrgrootx1.pem -O isrgrootx1.pem --no-check-certificate
wget https://letsencrypt.org/certs/isrg-root-x1-cross-signed.pem -O isrg-root-x1-cross-signed.pem --no-check-certificate
update-ca-trust extract


And (hopefully) you are done!

About Oban

Oban manages the Brownrice Internet staff, keeps the network humming, and chases his wife and twin boys around during his time off.