Monitoring Certificate Chains From the Inside: Three Probes, Three Failure Modes

A certificate check that returns a single boolean is not a monitor. It is a coin flip with extra steps. Expiry, trust-path validation, and revocation are three different questions, answered from three different data sources, and they fail in three different ways. Collapse them into one green light and the light will be wrong in at least one direction — usually the direction that gets you paged.

This is about probing your own endpoints from inside your network: the mail server, the reverse proxy, the internal API. It is not about watching Certificate Transparency logs for your domains. Those answer a different question — what has been issued in your name — and they fail differently. Keep them separate.

Why one check hides the interesting failures

RFC 5280 profiles the X.509 v3 certificate and the X.509 v2 CRL, and it describes an algorithm for certification path validation. That algorithm is the thing your TLS client actually runs. It checks signatures up the chain, validity periods at every depth, name constraints, key usage, and — if configured — revocation status. A monitor that only reads the leaf’s notAfter is checking one input to that algorithm and calling it a day.

The failure modes you care about are not the same failure:

  • Expiry is a property of each certificate in the chain, not just the leaf. An intermediate that expires before the leaf takes the whole endpoint down.
  • Trust-path failures are structural: a missing intermediate, a cross-signed alternate chain, a root that your trust store does not carry, a name mismatch. These are usually binary, but the reason is what you need.
  • Revocation is a network question with a third answer. The responder can say good, revoked, or unknown — and it can also fail to answer at all, which is not the same as good.

Each of these deserves its own probe, its own state model, and its own alert path.

Probe 1 — chain and trust

Verify the served chain against your own trust store, not the system default, and read the structured output rather than the exit status. The OpenSSL verify command verifies certificate chains and, per its manual, attempts to display all problems if a chain has multiple. That last property matters: a chain can have more than one defect, and fixing the first one you see may just reveal the second.

The output format is documented. The general shape is:

server.pem: /C=AU/ST=Queensland/O=CryptSoft Pty Ltd/CN=Test CA (1024 bit)
error 24 at 1 depth lookup: invalid CA certificate

The manual is explicit about depth: it starts at zero for the target (leaf) certificate, then 1 for the CA that signed it, and so on. That number is the difference between a five-minute fix and an hour of guessing. error 24 at 1 depth tells you the intermediate is the problem. error 10 at 0 depth tells you the leaf has expired. error 20 at 0 depth tells you the issuer cannot be found — usually a missing intermediate in what the server sends.

A probe that only checks $? throws that away. Parse the error number and the depth. Store both. Alert on the pair, not on the exit code.

Two practical notes on running this:

  • Fetch the chain the way a client would — openssl s_client -showcerts against the endpoint — and feed the leaf plus the served intermediates to verify with -untrusted, and your own roots with -CAfile or -CApath. If you verify against a different trust store than your clients use, you are testing a different system.
  • If you want the probe to also exercise revocation, verify accepts -CRLfile (a file or URI containing one or more CRLs) and -crl_download, which attempts to download CRL information via the certificates’ CDP entries. Both are useful, but see the revocation section below before you make them the primary signal.

Probe 2 — expiry, at every depth

The leaf is the certificate everyone watches. The intermediate is the one that surprises people, because it was issued years ago, renewed on someone else’s schedule, and is not in your renewal runbook.

The probe should extract notBefore and notAfter for every certificate in the served chain, compute the earliest expiry, and report which certificate at which depth owns it. A chain where the leaf has 200 days left and the intermediate has 12 is a chain with 12 days left.

This is also where cross-signed and alternate chains complicate things. A server may present one chain; a client may build a different path using a cross-signed root or an alternate intermediate it already has cached. Your probe sees what the server sends. That is the right thing to monitor for your endpoints, because it is what most clients will receive — but be aware that a successful verification on your probe does not prove every client will build the same path. If you care about a specific client population, probe with that population’s trust store.

Probe 3 — revocation, and the state everyone forgets

OCSP is specified in RFC 6960 as a protocol for determining the current status of a certificate without requiring CRLs. It defines three definitive status values: good, revoked, and unknown. Read the definitions before you write alert rules, because two of them do not mean what the words suggest.

good is narrower than it sounds. The RFC states that a good response does not necessarily mean the certificate was ever issued, or that the time the response was produced falls within the certificate’s validity interval. It means the responder has no current record of revocation for that serial. It is not a statement that the certificate is valid for your use.

unknown means the responder does not know the certificate, usually because the request names an issuer that responder does not serve. It is not a revocation. It is not a clean bill of health either. It means the status could not be determined by this responder, and the client is left to decide whether to try another source.

revoked is the only one that is unambiguously bad news, and even that has a wrinkle: RFC 6960 permits a responder to return revoked for a certificate that was never issued, provided it includes the extended revoked definition extension and sets the revocation reason to certificateHold with a revocation time of January 1, 1970. If you are parsing responses by hand, do not assume revoked implies a real issuance history.

Then there is the fourth outcome, which the protocol does not call a status at all: the responder did not answer. RFC 6960 defines error responses — malformedRequest, internalError, tryLater, sigRequired, unauthorized — and states plainly that these messages are not signed. tryLater specifically indicates the service exists but is temporarily unable to respond.

This is where most homegrown probes go wrong. The HTTP request times out, the script catches the exception, logs a warning, and returns success — because the certificate itself is fine, right? The certificate may well be fine. You do not know. That is the point.

The design rule: every probe needs a third state

Every probe should return one of three things: healthy, failed, or could not determine. The third state must never be collapsed into healthy. A probe that cannot reach the OCSP responder has not established that the certificate is unrevoked. A probe that cannot reach the endpoint has not established that the certificate is valid. A probe whose own execution failed has told you nothing about the target.

This is not a stylistic preference. It is the difference between a monitor and a random number generator. The failure mode of a silent probe is identical, from the alerting system’s perspective, to a healthy endpoint: no page. You will discover the difference during the incident, which is the worst possible time.

Concretely, the three-state model means:

  • A timeout or connection error from the OCSP responder is could not determine, not healthy.
  • An OCSP unknown response is could not determine unless you have a documented policy that says otherwise for that issuer.
  • A verify invocation that fails to run — missing binary, unreadable trust store, malformed chain file — is could not determine, not a pass.
  • A verify run that completes with errors is failed, with the error code and depth attached.

Write the policy down before you write the code. What do you do with unknown? What do you do with tryLater? What do you do when the responder is reachable but its response is stale — outside the thisUpdate/nextUpdate window? If you do not decide, the default behavior is whatever your tool happens to do, which is usually to treat the absence of a clear failure as success.

A worked example of the failure this prevents

Consider a probe that does the following: fetch the chain, run verify against the local trust store, then query the OCSP responder named in the leaf’s AIA extension. If the OCSP query throws, catch it and continue. Return success if verify passed.

Now suppose the responder is down for maintenance, or the network path from your probe host to the responder is blocked by a firewall change nobody documented. The probe returns green. The certificate may in fact be revoked — you would not know, because you never got an answer. The monitor has reported health it did not observe.

Under the three-state model, the same probe returns could not determine with the responder URL and the error attached. That is a page, or at minimum a ticket, and it is the correct response: your revocation checking is broken, and you should know that before you need it.

What to log on every run

The value of a probe is not the current status. It is the record that lets you diagnose a failure six months from now without re-running against an endpoint that may have changed. Log, at minimum:

  • The chain as served: subject, issuer, serial, notBefore, notAfter, and depth for each certificate.
  • The verify result: exit status, every error number, and every depth reported.
  • The trust store used, by path and by content hash if you can manage it.
  • For revocation: the responder URL, the response status, thisUpdate, nextUpdate, and producedAt if present.
  • For any indeterminate result: the specific error — timeout, TLS failure, HTTP status, OCSP error name.

The thisUpdate/nextUpdate window is particularly worth keeping. A responder that is reachable but serving stale responses is a different problem from a responder that is unreachable, and the two require different fixes. If you only log “OCSP OK”, you cannot tell them apart later.

What this does not cover

Two things are deliberately out of scope, and it is worth being explicit about why.

Certificate Transparency. CT logs are append-only ledgers of certificates that anyone can query, and a monitor can compute an audit proof to verify that a particular certificate exists in a log. That is a valuable check — it tells you whether someone issued a certificate for your domain that you did not request. But it answers a question about the public record, not about the certificate your server is currently presenting. A CT monitor and a local endpoint probe fail in different ways and should be operated as separate concerns.

CAA records. RFC 8659 is explicit that relying parties must not use CAA records as part of certificate validation. CAA is an issuance-time control that CAs check before issuing. It is not a validation input, and a probe that reads CAA and treats it as a trust signal is doing something the specification forbids.

FAQ

Can I just use openssl verify with -crl_check and be done?

You can, and it will catch some things. But -crl_check depends on having a current CRL available, either via -CRLfile or via -crl_download against the certificate’s CDP entries. If the CRL cannot be fetched, you need to know whether the tool treated that as a failure or as a pass, and you need to know which. The three-state model applies here too: a CRL fetch failure is could not determine.

What is the difference between an OCSP unknown and a responder error?

unknown is a signed, definitive response saying the responder does not know the certificate — usually because it does not serve that issuer. A responder error is an unsigned message saying the request could not be processed. RFC 6960 lists the error types and states that error messages are not signed. They are different failure modes and should be logged differently.

Should I alert on unknown?

That is a policy decision, and you should make it deliberately. If your issuer’s responder returns unknown for certificates it should know about, that is a problem worth investigating. If it returns unknown for a certificate from a different issuer because you queried the wrong responder, that is a probe bug. Either way, treating unknown as healthy by default is the one option that is hard to defend.

Why not just check the leaf’s expiry and call it monitoring?

Because the intermediate can expire first, and because expiry is only one of the three questions. A chain with a valid leaf and an expired intermediate fails path validation. A chain with a valid leaf and a revoked intermediate fails revocation checking. Neither failure is visible from the leaf’s notAfter alone.

Does a successful verify mean the certificate is valid for my clients?

It means the chain validates against the trust store you gave it, at the time you ran it, with the revocation settings you configured. If your clients use a different trust store, a different revocation policy, or a different clock, the answer can differ. Probe with the configuration you actually care about.

Scroll to top