Solving the expiring. X.509 root CA certificate-mageddon, partly at least

Thanks to the Internet, even non-technical users are now exposed to Public Key Infrastructure (PKI), for secure and authenticated communications and code signing.

However, there are problems brewing with PKI — largely because it’s getting old and neglected.

The X.509 public key certificate standard that is commonly used to secure protocols like Transport Layer Security (TLS) is ancient (at least by the pace of Internet development).

It was issued in 1988 by the International Telecommunications Union as part of its X.500 series of computer directory standards, and was adapted by the Internet Engineering Task Force for Internet use in 1995-96 and updated over the coming decades.

This is stuff that’s been around for a long time, and it tends to just work. After all, most of the implementation heavy lifting has been done for users.

But this ease of use is a double-edged sword. Few people bother to check under the hood unless it’s absolutely necessary, like when thingsgo wrong.  

Clock ticking on certificates

Self-confessed BBC hacker in residence Scott Helme is one of those who have delved deep into Internet PKI, and he has sounded the alarm about Root Certificate Authorities (CAs) reaching their end of life for the past couple of years.

Root CA self-signed certificates are embedded in software and hardware and Scott noted that there are some very old ones out there that have existed for 20 to 25 years.

They have started to expire, like the AddTrust External CA Root, which reached its end of life on 30 May 2020 (UTC).

The expiry of the old AddTrust Root CA certificate caused service outages for streaming content services and payments processors that used it.

Even though the issuer of the root certificate had given a reasonable amount of warning , some well-known corners of the Internet missed it and didn’t upgrade their software.

Why did that happen? Well, partly because nothing like it has taken place before so the issue hasn’t received enough attention.

“I feel like everyone has built stuff on top of Internet PKI and, for the most part, it ‘just works’. These root certs are among the first expiring and the first time we’ve seen this issue on the web,” Scott said.

It’s not so much that people parked the problem, even though two decades ago 2020 must have seemed such a long time away and the issue could be dealt with there and then.

“To be honest, my guess is that they didn’t even think that much about it,” Scott said.

As the AddTrust legacy root certificate expiry shows, this is not a problem that can be ignored. Worse, when certificate expiry strikes, it could cause breakage for scores of users with no apparent reason as to why their devices no longer work.

As of writing, a large number of Samsung Blu-Ray players around the world are stuck in a reboot loop .

Although customers are still waiting for the official word on the issue, the suspicion is that the problem is caused by an expired certificate that stops the device firmware from booting properly, possibly because of strict digital rights management requirements that require verification and signing before full start-up of the players.

As Scott points out, there are hundreds of ageing Root CAs out there, many more when you include the intermediates. Over the next few years, more will expire.

How do we deal with this then?

On the face of it, the fix for the problem is simple. The Root CA certificates need to be updated.

The reality, as Let’s Encrypt found out last year, is that not every device continues to receive updates. Even if they do, upgrades aren’t always installed.

We’re not talking about really old gear either; Scott mentioned devices that are two to eight-years-old missing out on updates.

Finding the certs that are about to expire is possible in some cases but not all.

“Yes and no, it depends on which part of the connection you control,” Scott said.

“If you’re a vendor making a software product that is the client and connects to another service then, yes, you can make sure the certs in your product are good,” he added.

“If you run a server and clients are connecting to you, like in my post the BBC are the Server (iPlayer API) and the TV is the client, there is no way for them to look into the client like that.

“In that scenario it’d be the responsibility of the TV manufacturer to keep their product up to date,” Scott said.

What’s best practice here then?

“There probably isn’t one; this isn’t an issue we’ve experienced before because the web isn’t old enough. It’d involve looking at your infrastructure and highlighting anywhere you depend on the Internet PKI.

“From there you’d need to work through all of the certificates you depend on,” Scott said.

For anyone who depends on certificates for secure communications, like most of us do these days, taking the time to look at your current validation chain and really understand it is key.

“Could you build an alternative path if needed and might you need to at some point in the future?” Scott asked.

If the answer is yes, have a read of Scott’s suggestions on how to do it.

There are always more ways to skin a cat

If by now you’ve come down with a headache and the realization that countless expiring Root CA certificates will be a recurring problem, especially for retail-oriented devices, you’re not alone.

Maybe there’s a different approach that doesn’t result in a long slog of monitoring expiry dates?

Cryptographer Peter Gutmann, who among other things wrote a developer’s style guide for X.509 , put it succinctly: 

“Assuming that the entire world is the web is the problem.”

“PKI is built around the assumption that everything is the web, with zero consideration paid to any other use,” Peter said.

“Wishing you could magically teleport updates into every piece of IoT and whatnot on the planet and then the problem is solved is a manifestation of this wrong thinking,” Peter added.

Peter’s fix is an easy one:

“For embedded and SCADA devices, alongside anything on an RFC 1918 LAN, the only appropriate expiry date for a certificate is never.”

Likewise, in some situations it’s worth noting what Peter said above, which is that not the entire world is the web.

A recent security scare involving Security Assertion Markup Language (SAML), used for authentication on Palo Alto Networks devices running PAN-OS, had the United States Cyber Command tell users to patch the vulnerability double-quick.

Although the critical vulnerability is easy to exploit, it wasn’t so easy to figure out why the SAML implementation was wrong.

Microsoft identity architect, Ryan Newington, explained why:

Sometimes, overdoing things can be just as bad as not doing enough.

Juha is a technology writer and journalist, based in New Zealand. He is a contracted contributor to the APNIC Blog .

The views expressed by the authors of this blog are their own and do not necessarily reflect the views of APNIC. Please note aCode of Conduct applies to this blog.