These field notes are for website operators who are interested in cross-certificates, or who are troubleshooting why their certificates are not being trusted by a browser. The latter group includes 2022-2023 to people who have bought HARICA-issued certificates after June 1st, like me, who may be experiencing issues with getting their browser (or their visitors’ browsers) to trust the certificate they bought.

It’s also an adequate general-purpose guide to reasoning about certificate chain issues, but only if you have some foundations with applied cryptography already.

Intro

A couple days ago, the Hellenic Academic and Research Institutions Certificate Authority (HARICA) recently announced that it would issue new certificates using its 2021 Root TLS Certification Authorities. HARICA is a less-well-known CA, but is notable for currently being the only affordable1 CA to issue .onion certificates. From their blog post, they seem confident that this will be a seamless transition:

Both HARICA TLS RSA Root CA 2021 and HARICA TLS ECC Root CA 2021 are already pre-installed on Windows operating systems as well on macOS 12 providing the necessary trust anchor for Google Chrome, Microsoft Edge and other popular Internet Browsers. In addition, Mozilla Firefox has updated its Certificate Store with HARICA’s new Root CAs.

After purchasing a brand-new .onion certificate for my website and installing it into my webserver, my browser greeted me with the fateful NET::ERR_CERT_AUTHORITY_INVALID message - my browser couldn’t create a valid chain to anything in my trust store. Time to figure out why.

Assessing the Certificate Chain

Starting from the basics,2 let’s look at what certificate chain we have currently. I’d recommend using either KeyCDN’s online SSL tools, or Square’s certigo tool, or any reasonable equivalent. The certificate chain that HARICA originally gave me is available here, and taking a look at it with certigo we see:

% certigo dump -f PEM -n certificate-chain-original.pem 
** CERTIFICATE 1 **
Input Format: PEM
Valid: 2022-06-04 17:26 UTC to 2023-06-04 17:26 UTC
Subject:
	CN=tweedge32j4ib2hrj57l676twj2rwedkkkbr57xcz5z73vpkolws6vid.onion
Issuer:
	C=GR, O=Hellenic Academic and Research Institutions CA, CN=HARICA DV TLS
	ECC
DNS Names:
	tweedge32j4ib2hrj57l676twj2rwedkkkbr57xcz5z73vpkolws6vid.onion

** CERTIFICATE 2 **
Input Format: PEM
Valid: 2021-03-19 09:22 UTC to 2036-03-15 09:22 UTC
Subject:
	C=GR, O=Hellenic Academic and Research Institutions CA, CN=HARICA DV TLS
	ECC
Issuer:
	C=GR, O=Hellenic Academic and Research Institutions CA, CN=HARICA TLS
	ECC Root CA 2021

Which is OK - nothing out of the ordinary. The first certificate in the chain is my certificate, and the second is an intermediate certificate issued by HARICA. Assuming these are valid certificates, they would confirm the following trust chain:

  • My certificate (#1) is trusted by HARICA’s intermediary DV TLS ECC certificate (#2)
  • HARICA’s intermediary DV TLS ECC certificate (#2) is trusted by HARICA’s ECC Root CA 2021

Where does HARICA’s ECC Root CA come from? My device should have it! HARICA’s ECC Root CA is expected to be present in viewers’ browser or OS trust store, and doesn’t need to be provided as part of the certificate chain.3 So, I tried certigo’s verify function and received:

% certigo verify -f PEM -n tweedge32j4ib2hrj57l676twj2rwedkkkbr57xcz5z73vpkolws6vid.onion certificate-chain-original.pem 
Failed to verify certificate chain:
	x509: certificate signed by unknown authority

Which is confusing. By any account, this should be working fine. I’m using a modern browser and OS, and based on a cursory search I see HARICA has been working for over a year to make sure their 2021 Root CAs are included in various trust stores…

What’s in my Trust Store?

…but unfortunately that didn’t include my trust store for some reason, even though it’s been less than two months since I updated to the “latest and greatest” version of my preferred OS. Ripping a quick command from StackOverflow to check the CAs present in my OS’ trust store, I found out that only HARICA’s 2011 and 2015 Root CAs are present:

CN = Hellenic Academic and Research Institutions RootCA 2011
CN = Hellenic Academic and Research Institutions RootCA 2015
CN = Hellenic Academic and Research Institutions ECC RootCA 2015

For a while, it looked like I might have just bought a ceritificate that my machine itself can’t trust. I need to get my 2021 certificate trusted by machines - such as mine - which have only HARICA’s 2015 Root CA installed. I can’t exactly serve a warning over HTTP saying “hey, install this Root CA to view this site” - that’d be frustrating at best, and a trust-killer at worst.

Thankfully, there’s something I haven’t tried yet. In HARICA’s announcement about the switch to the Root CA 2021, they included:

For older operating systems and browsers, HARICA issued two additional cross-certificates to chain the 2021 hierarchy with the older 2015 one for increased ubiquity.

But what is a cross-certificate, and how would it help my situation?

Understanding Cross-Certificates (in Brief)

Please note the below is brief, essentially correct, but not complete. If you’re looking for a deeper dive into X.509 certificates or cross-certificates, I’ve provided further reading in the Appendix.4

As documented by Microsoft, a cross-certificate is:

… a digital certificate issued by one Certificate Authority (CA) that is used to sign the public key for the root certificate of another Certificate Authority. Cross-certificates provide a means to create a chain of trust from a single, trusted, root CA to multiple other CAs.

It sounds like this would allow me to create a chain of trust between the 2015 Root CA - which my computer trusts - and the 2021 Root CA that my .onion certificate has a chain of trust to! To understand how this works, let’s quickly recap what a certificate is. A certificate contains at minimum:

  • An identity. The identity in certificates used for HTTPS is generally a hostname, or a list of hostnames.
  • A public key. The public key corresponds to a private key that the identified party keeps secret.
  • A signature verifying the integrity of the identity + public key. This signature can be from a certificate authority, or the certificate can be self-signed.

HARICA’s 2015 Root CA in my trust store has the following properties:

  • Identity: Hellenic Academic and Research Institutions ECC RootCA 2015
  • Public key: MHYwEAYHKoZI...C6qACSRsfsFG
  • Signature: Root CAs are self-signed.

But if I download HARICA’s 2021 Root CA, it has the following properties:

  • Identity: HARICA TLS ECC Root CA 2021
  • Public key: MHYwEAYHKoZI...kEFYkBVyp6G3
  • Signature: Root CAs are self-signed.

So something cool can be done here - since HARICA possesses the keypair for both their 2015 and 2021 Root CAs, they can arbitrarily create new certificates that include either root CA’s public key, and then can be signed by either root CA’s public key! HARICA could clone the identity of their 2021 Root CA, include their 2021 Root CA’s private key, and then sign it with their 2015 Root CA to create:

  • Identity: HARICA TLS ECC Root CA 2021
  • Public key: MHYwEAYHKoZI...kEFYkBVyp6G3
  • Signature: Signed by Hellenic Academic and Research Institutions ECC RootCA 2015.

Applying Cross-Certificates in Practice

So if our understanding above is correct, we just need to add 2015-2021 cross-certificate from HARICA’s repository to my certificate chain, as that should bridge the trust between the HARICA’s intermediary DV TLS ECC certificate and their ECC Root CA 2015 in my trust store. Now we have an updated certificate chain, seen below:

% certigo dump -f PEM certificate-chain-with-cross-certificate.pem
** CERTIFICATE 1 **
Input Format: PEM
Valid: 2022-06-04 17:26 UTC to 2023-06-04 17:26 UTC
Subject:
	CN=tweedge32j4ib2hrj57l676twj2rwedkkkbr57xcz5z73vpkolws6vid.onion
Issuer:
	C=GR, O=Hellenic Academic and Research Institutions CA, CN=HARICA DV TLS
	ECC
DNS Names:
	tweedge32j4ib2hrj57l676twj2rwedkkkbr57xcz5z73vpkolws6vid.onion

** CERTIFICATE 2 **
Input Format: PEM
Valid: 2021-03-19 09:22 UTC to 2036-03-15 09:22 UTC
Subject:
	C=GR, O=Hellenic Academic and Research Institutions CA, CN=HARICA DV TLS
	ECC
Issuer:
	C=GR, O=Hellenic Academic and Research Institutions CA, CN=HARICA TLS
	ECC Root CA 2021

** CERTIFICATE 3 **
Input Format: PEM
Valid: 2021-09-02 07:44 UTC to 2029-08-31 07:44 UTC
Subject:
	C=GR, O=Hellenic Academic and Research Institutions CA, CN=HARICA TLS
	ECC Root CA 2021
Issuer:
	C=GR, L=Athens, O=Hellenic Academic and Research Institutions Cert.
	Authority, CN=Hellenic Academic and Research Institutions ECC RootCA
	2015

At this point we are expecting the following trust chain to be valid - and thankfully, it seems to be working as intended! certigo appears to be able to verify it:

% certigo verify -f PEM -n tweedge32j4ib2hrj57l676twj2rwedkkkbr57xcz5z73vpkolws6vid.onion certificate-chain-with-cross-certificate.pem 
Found 1 valid certificate chain(s):
[0] CN=tweedge32j4ib2hrj57l676twj2rwedkkkbr57xcz5z73vpkolws6vid.onion
	=> CN=HARICA DV TLS ECC
	=> CN=HARICA TLS ECC Root CA 2021
	=> CN=Hellenic Academic and Research Institutions ECC RootCA 2015 [self-signed]

And once deployed to my webserver, it worked in practice too!

Interestingly, there aren’t major security concerns with this, to my knowledge. If HARICA’s 2021 CA or intermediary certificate are compromised, my certificate would be revoked. However, if HARICA’s 2015 Root CA is compromised, that should not have a major impact on the ability to trust my certificate - browsers/OSes would jettison the HARICA’s Root CA 2015 from their trust store, because it no longer helps bridge to any trusted certificate. Devices which relied on HARICA’s 2015 Root CA would no longer recognize my domain, sure, but any devices (hopefully more in the future!) which support HARICA’s 2021 Root CA should be completely unaffected (as that part of the trust chain remains valid).5

Conclusion

This was a useful reminder of two things.

First, manual certificate management is a pain, and I’m glad that ‘effortless’/automatic certificate management has had significant traction over the past several years. I’m writing all this partially for myself for next year, in case I have to do this again.

Second, this also served as a reminder of the awkwardness of managing roots of trust. We know very well that expiring CAs can be painful - in recent memory, LetsEncrypt’s X3 Root CA expired and many sites and services experienced outages due to unexpected impact. While creating new CAs is not as painful, thankfully, it’s not without sharp edges.

Thankfully though, HARICA took actions to ensure that customers could build in continuity of trust while the 2021 Root CA rolls out across the globe to a trust store near you, and at the end of the day this was pretty interesting to figure out. Hopefully these notes help someone triage and fix this issue faster than I did :)

Appendix

  1. DigiCert is the other option, and costs exactly what you’d expect of DigiCert. For the lucky (but growing!) few people whose technical careers have coincided with the presence and ubiquity of Lets Encrypt: DigiCert is extremely expensive. ↩

  2. I’m assuming you’ve already made sure that the certificate chain you expected is being presented - ex. there was no update delay with whatever webserver or host you’ve used, you’ve rebooted/restarted any services for sanity’s sake, etc. If you haven’t done those, go do so and come back. ↩

  3. As a reminder, Root CAs are self-signed - they have to be pre-provisioned in a trusted manner (ex. bundled with your operating system) in order to be trustworthy, and are not typically sent as part of certificate chains. There’s nothing to stop you from sending a Root CA in a certificate chain, but they’d just be discarded (under normal circumstances). ↩

  4. For those seeking more detailed resources, I’d recommend Takahiko Kawasaki’s Illustrated X.509 Certificate article to more thoroughly understand digital certificates. For deeper reading on cross-certificates, I recommend looking through Scott Helme’s Cross-Signing and Alternate Trust Paths; How They Work. ↩

  5. Should being the operative word here. There may be implementation details I’m not aware of, such as certain implementations only considering the last certificate in the presented certificate chain for validation against the device’s roots of trust. My use of the cross-certificate in this case is not introducing new confidentiality risk though - an attacker with HARICA’s 2015 Root CA private keys could sign arbitrary certificates and would be trusted by any user device that hadn’t yet revoked HARICA’s 2015 Root CA, so it doesn’t matter whether or not I use the cross-certificate in that context. ↩