HTTP

Definition

HTTP (Hypertext Transfer Protocol) is the stateless request/response application protocol that powers the web. A client opens a connection, sends a request (method + target URI + headers + optional body), and the server returns a response (status code + headers + optional body). The semantics — methods, status codes, headers, content negotiation, caching — are defined by RFC 9110 (HTTP Semantics, June 2022) and are deliberately versionless; the same semantics ride over three wire formats (HTTP/1.1 over TCP, HTTP/2 over TCP+TLS, HTTP/3 over QUIC+UDP) that differ only in framing, multiplexing, and connection setup. HTTPS is HTTP run inside TLS — same protocol, encrypted transport.

Why it matters

HTTP is the universal substrate: every browser, every REST/GraphQL/gRPC API, every package registry, every webhook, every download, and every modern LLM streaming endpoint speaks it. Understanding it means understanding how to debug a 401 vs a 403, why POST retries are dangerous, what Cache-Control: stale-while-revalidate=60 actually buys you, and why a slow API call might be a TLS handshake rather than a slow server. The differences between HTTP/1.1, /2, and /3 also matter in production: HTTP/2 ships with multiplexing and header compression that eliminate the per-request TCP overhead of HTTP/1.1, while HTTP/3 replaces TCP entirely with QUIC, killing transport-level head-of-line blocking and enabling 0-RTT resumption — wins that show up most on mobile and high-latency networks. By April 2026, traffic shares sit roughly at HTTP/2 ~51%, HTTP/1.x ~28%, HTTP/3 ~21%; every major CDN and modern server (nginx 1.25+, Caddy, Cloudflare, Fastly) negotiates all three from one configuration.

Beyond the wire format, HTTP is also the place where most reliability and security disciplines manifest: idempotency keys for safe retries, problem-details envelopes for machine-readable errors, conditional requests for cheap revalidation, bearer tokens (never query-string secrets) for auth, and Vary for correct CDN caching. Get these wrong on an API and every client integration becomes a 3 a.m. page.

How it works

A request has four parts: a method (verb), a target (path + query), headers, and an optional body. A response replaces the verb/target with a status code and reason phrase, and otherwise has the same shape. Everything else is convention layered on top.

Methods carry meaning. RFC 9110 defines nine methods and classifies them along two axes: safe (read-only — never changes server state) and idempotent (repeating the request has the same net effect as one).

The non-idempotent verbs (POST, PATCH) are the source of most retry bugs. The standard fix is an Idempotency-Key request header (popularised by Stripe): the server stores the result of the first request keyed by the header and replays it on retry within a TTL — converting an unsafe verb into a safe one without changing the URL design.

Status codes are part of the contract. Five classes: 1xx informational, 2xx success, 3xx redirection, 4xx client error, 5xx server error. A handful do real work: 200 OK, 201 Created (with a Location header), 204 No Content, 301/308 permanent redirect, 302/307 temporary, 304 Not Modified (revalidation hit), 400 Bad Request, 401 Unauthorized (missing/invalid auth — not "forbidden"), 403 Forbidden (authenticated but not allowed), 404 Not Found, 409 Conflict and 412 Precondition Failed (optimistic concurrency), 429 Too Many Requests (with Retry-After), 500 Internal Server Error, 502/503/504 (upstream failures). Returning 200 OK with {"error": ...} in the body breaks every generic client and CDN.

Errors get an envelope. RFC 9457 — Problem Details for HTTP APIs (July 2023) obsoletes RFC 7807 and defines one JSON shape served as application/problem+json:

HTTP/1.1 422 Unprocessable Content
Content-Type: application/problem+json

{
  "type":   "https://example.com/probs/validation",
  "title":  "Your request parameters didn't validate.",
  "status": 422,
  "detail": "name must be at least 3 characters",
  "instance": "/users/alicedev",
  "errors":  [{ "field": "name", "rule": "minLength" }]
}

9457 keeps 7807's five core fields, adds a problem-type registry, and clarifies the language around extension members so multiple problems can be represented. Frameworks (Spring 6, ASP.NET Core, Quarkus, Litestar, FastAPI plugins) ship support out of the box.

Caching is explicit. Two mechanisms layer cleanly: a freshness lifetime set by Cache-Control: max-age=N (plus directives like public, private, no-store, immutable, stale-while-revalidate=N), and revalidation via conditional requests — the server returns an ETag (or Last-Modified), the client echoes it back on the next request as If-None-Match (or If-Modified-Since), and the server replies 304 Not Modified with no body. Mind the trap in names: no-cache does allow storage but mandates revalidation on every use; no-store is the directive that forbids caching outright. For any response whose body depends on a request header (auth, language, encoding), set Vary: listing those headers — without it, shared caches will serve the wrong representation to the wrong user.

Content negotiation is proactive. The client advertises preferences with Accept, Accept-Encoding, Accept-Language, optionally with q-values (Accept: text/html, application/json;q=0.9); the server picks the closest match and echoes the choice in Content-Type plus the input headers in Vary. Modern clients drop Accept-Charset entirely — UTF-8 is universal. Client Hints extend the same idea opt-in (server requests, client sends) to keep per-request header bloat down.

Auth lives in headers, not URLs. Authorization: Bearer <token> is the modern default (RFC 6750). For machine-to-machine, OAuth 2.0 client-credentials grants short-lived tokens; for interactive flows, OAuth 2.1 is now the baseline — it mandates PKCE with S256 for every authorization-code client, removes the implicit and ROPC grants, and bans bearer tokens in query strings (they leak into logs, browser history, and Referer). For mutual auth, mTLS (--cert/--key) operates a layer below at TLS.

Three wire formats, one semantics. HTTP/1.1 is plain-text framing over a single TCP connection — simple, but every request blocks the next (head-of-line blocking) and headers are uncompressed. HTTP/2 multiplexes many streams over one TCP connection, compresses headers with HPACK, and serves binary frames — but a single dropped TCP packet still stalls every stream on that connection. HTTP/3 abandons TCP for QUIC (RFC 9000) over UDP: each stream handles loss independently, the handshake is 1-RTT for first visit and 0-RTT for return visits (vs ~3 RTT for TCP+TLS 1.2), and connections survive an IP change because QUIC identifies them by connection ID — critical for mobile clients switching between Wi-Fi and cellular. HTTP/3 also requires TLS 1.3; there is no cleartext H3.

curl --http2 https://api.example.com/v1/users          # force HTTP/2
curl --http3 https://api.example.com/v1/users          # try HTTP/3
curl -v https://api.example.com/ 2>&1 | grep ALPN      # see what was negotiated

ALPN (Application-Layer Protocol Negotiation) inside the TLS handshake is how the client and server agree which wire format to use — h2, http/1.1, or via the Alt-Svc header advertising h3 for the next connection.

Common pitfalls

1. Retrying POST on a network timeout — you do not know whether the server saw the request. Use an Idempotency-Key so retries are safe, or convert the operation to PUT against a client-chosen ID.
2. 200 OK with an error body — breaks every generic HTTP client, CDN, and observability tool. Use the right status code and a Problem-Details envelope (application/problem+json).
3. 401 vs 403 confusion — 401 means "you are not authenticated" (and the server must send WWW-Authenticate); 403 means "you are authenticated but not allowed". Swapping them breaks auth-aware clients.
4. Bearer tokens in query strings — they leak into access logs, browser history, and Referer headers. Put them in Authorization only; OAuth 2.1 explicitly forbids the query-string form.
5. Caching without Vary — a response that depends on Authorization or Accept-Language but lacks Vary will be served to the wrong user by any shared cache. Set Vary to every request header that influenced the body.
6. no-cache mistaken for "don't cache" — no-cache permits storage but forces revalidation every time. To forbid storage entirely use no-store.
7. No Retry-After on 429 or 503 — clients invent their own backoff, usually badly. Set the header (seconds or HTTP-date) so well-behaved clients self-regulate.
8. HTTP/2 over HTTP/1.x assumptions — opening 6+ parallel TCP connections to "work around HoL" is counter-productive on H2/H3 (one multiplexed connection is faster). Re-tune client pools when you switch.
9. HTTP/3 enabled without UDP reachability — corporate firewalls and some load balancers still block UDP 443. Always keep HTTP/2 as the fallback; the server's Alt-Svc advertisement is a hint, not a contract.
10. PATCH without a media type — JSON Merge Patch (RFC 7396) and JSON Patch (RFC 6902) are different formats with different semantics. Advertise the one you accept in Content-Type.

Where to go next

• /concepts/api — APIs as the contract layer that sits on top of HTTP (REST, GraphQL, gRPC, tRPC tradeoffs).
• /sections/linux/curl — the universal CLI HTTP client; flags for --http2/--http3, headers, auth, TLS, and a recipe collection.
• /sections/linux/wget — non-interactive HTTP/HTTPS/FTP downloader with recursion, resumption, and netrc auth.
• /sections/linux/gh — GitHub CLI; a real-world OAuth + REST client you talk to every day.
• /sections/python/requests and /sections/python/httpx — synchronous and async HTTP clients in Python (httpx also speaks HTTP/2).
• /sections/javascript/fetch — the cross-runtime browser/Node/Workers/Deno HTTP client.

Sources

References consulted while writing this concept page. Links open in a new tab.

• RFC 9110 — HTTP Semantics — Authoritative spec (June 2022) for HTTP methods, status codes, headers, and the formal definitions of safe and idempotent methods used throughout "How it works".
• RFC 9457 — Problem Details for HTTP APIs — Replaces RFC 7807 (July 2023); source for the application/problem+json envelope shape and the problem-type registry.
• HTTP/3 and QUIC in production — 2026 deployment guide — Source for current adoption figures, the 0-RTT / 1-RTT handshake comparison, and the connection-migration story for mobile clients.
• Cloudflare — HTTP/3 vs HTTP/2 performance — Reference for the multiplexing / head-of-line-blocking comparison and the situational nature of HTTP/3 wins.
• MDN — HTTP caching — Source for the Cache-Control directive set, no-cache vs no-store distinction, and the ETag / If-None-Match conditional-request flow.
• MDN — Content negotiation — Used for the Accept / Vary semantics and the note that Client Hints are the modern opt-in successor to header-bloat negotiation.
• OAuth 2.1 — Source for the mandatory-PKCE rule, removal of implicit/ROPC grants, and the ban on bearer tokens in query strings cited in "Auth lives in headers".
• RFC 9000 — QUIC — The transport spec under HTTP/3; reference for stream-level loss independence, connection IDs, and the mandatory TLS 1.3 floor.