TLS Session Resumption and HTTP/3 on the Edge

For a CDN, the expensive part is often not moving bytes but setting up connections. A full TLS handshake costs asymmetric cryptography plus extra network round trips, and for distant visitors those tens or hundreds of milliseconds show up directly in time to first byte. This post covers two handshake-related pieces of work on our Rust edge: TLS session resumption and HTTP/3.

What session resumption is

When a browser first connects to a site, it completes a full handshake and negotiates session keys. If it needs another connection soon after (a new tab, a recycled connection, a switch between mobile networks), there is no reason to repeat the expensive key exchange. TLS has resumption mechanisms for exactly this:

• TLS 1.2 has two: session IDs (the server keeps session state in memory) and session tickets (the server encrypts the session state and hands it to the client, which brings it back next time).
• TLS 1.3 unifies this as PSK (pre-shared key) resumption: after the handshake the server sends a NewSessionTicket, and the client presents it on its next connection.

Either way, tickets are encrypted with a server-side ticket key. As long as the server can still decrypt the ticket a client brings back, the full handshake is skipped. For visitors that means lower latency; for the edge it means less CPU spent on cryptography.

An easy-to-miss problem: every restart invalidated every ticket

In an early version of the Rust edge we noticed something odd: resumption succeeded far less often than expected, and it dropped after every deployment. The cause was simple but easy to overlook: we had never configured a ticket key on the TLS layer.

Without explicit configuration, BoringSSL and OpenSSL generate a random ticket key at process start. That is fine for a single long-lived process, but edge processes restart for deployments and upgrades. Each restart gave the new process a new key, every ticket held by clients became useless at once, and clients fell back to full handshakes. There was a key ring implementation in the codebase, but it had never been wired into the TLS acceptor.

The fix: persist, rotate, bound

We now treat ticket keys as state that needs careful management, not as a throwaway by-product of the process:

1. Persist to disk. At startup the edge loads the key ring from the node's certificate directory; if the file does not exist, it generates new keys with a CSPRNG. Writes go to a temporary file followed by an atomic rename, and the file is created with 0600 permissions from the start, so there is no window in which it is readable.
2. Wire up a real callback. We added a ticket key callback to our vendored TLS bindings (BoringSSL's SSL_CTX_set_tlsext_ticket_key_cb). New tickets are encrypted with the current key; on decryption, the key is looked up in the ring by the key name embedded in the ticket.
3. Rotate on a schedule. By default keys rotate every 6 hours. The ring keeps the current key plus a few previous ones, which are used only to decrypt, never to issue. Recently issued tickets keep working, while each key's exposure is time-limited, which limits the forward-secrecy impact if a key ever leaked.
4. Fail open. If the key file cannot be read, the edge generates temporary keys and keeps serving; tickets from that run simply will not survive the next restart. Resumption is an optimization and should never be a reason to stop serving.

Separately, the server-side session cache (used for TLS 1.2 session IDs) is a bounded LRU with a default limit of 20,480 entries. When it is full the oldest sessions are evicted, so memory use cannot grow without limit.

How we verified it is worth describing. In tests we simulate a restart by building a brand-new SSL_CTX from the same key file, then connect with a ticket issued by the old context and assert session_reused == true. On a real node we capture a ticket, restart the edge service, reconnect with the old ticket and confirm it is still resumed. One gotcha: send a real request and wait a moment. In TLS 1.3 the NewSessionTicket arrives after the handshake, so if you disconnect too early you never receive a ticket and every connection looks new.

A note on scope: ticket keys are currently per node. If a visitor's next connection lands on a different node, it still performs a full handshake. That is a deliberate trade-off; distributing keys between nodes would widen their exposure.

Fixed along the way: clean restarts

While investigating, we also found that edge restarts were not clean. Pingora's default graceful shutdown period after SIGTERM is long, and the systemd stop timeout was much shorter, so every restart ended with a SIGKILL. We reduced the grace period and shutdown timeout to a few seconds, and restarts now complete cleanly within seconds. Together with persistent ticket keys, deployments are close to invisible to visitors: connections drain smoothly and returning visitors' tickets stay valid.

HTTP/3: the same pipeline, a different transport

HTTP/3 runs over QUIC, which runs over UDP and has TLS 1.3 built in. Its benefits show up mostly on unreliable networks: no TCP-level head-of-line blocking, fewer round trips to establish a connection, and connections that can migrate when the network changes.

Pingora's own HTTP/3 support has not been merged upstream. We did not want our data plane roadmap tied to an upstream release cycle, so we built QUIC and HTTP/3 termination on Cloudflare's tokio-quiche, which is based on quiche.

The key design decision: HTTP/3 handles only the transport; all request logic reuses the existing pipeline. Concretely:

• tokio-quiche terminates QUIC and HTTP/3 on UDP 443 and selects a certificate by SNI during the handshake. Certificates come from exactly the same store as TCP 443; there is no second certificate system.
• Decoded HTTP/3 requests are handed over a local loopback connection to Pingora in the same process, so routing, caching, the WAF, rules and origin fetches are all reused. No feature module had to be rewritten for HTTP/3.
• Loopback requests carry a random secret that exists only in process memory, and are trusted only when they genuinely come from loopback. That is how the real client IP and protocol are restored. Externally supplied headers with the same names are always stripped.
• H1/H2 and H3 share a single BoringSSL in one process, which also lets us compute client fingerprints during the QUIC handshake.

Don't advertise what you can't serve: gating Alt-Svc

Browsers learn that a site supports HTTP/3 from the response header Alt-Svc: h3=":443"; ma=86400. If a server sends that header without listening on UDP, browsers waste an attempt before falling back to HTTP/2. So the Alt-Svc advertisement is gated at both the node and the domain level: it is only sent when the node is actually running a QUIC listener and the domain has HTTP/3 enabled. We also strip any Alt-Svc header returned by the origin, so an origin's own advertisement cannot mislead clients.

We do not enable 0-RTT today. It saves another round trip, but early data can be replayed, and enabling it safely means deciding which requests are idempotent. We will revisit it once we are confident about that.

Summary

Neither session resumption nor HTTP/3 is a feature you simply switch on. Resumption depends on managing the lifecycle of ticket keys; HTTP/3 needs a separate transport stack without letting request logic fork. Our approach: treat keys as state that is persisted and rotated, treat HTTP/3 as a transport-only front end, and implement every feature exactly once.