IPsec secures IP traffic at the network layer, which means every application above it inherits the protection without being modified. It offers two protocols to do that, and the CISSP exam leans on the difference between them: Authentication Header proves who sent a packet and that nobody altered it, while Encapsulating Security Payload adds the part AH deliberately leaves out, which is confidentiality. The two are defined separately in RFC 4302 and RFC 4303, under the architecture set out in RFC 4301.
Key Insight
Both protocols sign your packets. Only ESP hides them. Authentication Header gives integrity and origin authentication and stops there, so the payload travels in the clear and anyone on the path can read it. Encapsulating Security Payload encrypts the payload and can authenticate it too, which makes AH’s one genuine advantage, protection of the outer IP header, a narrow reason to accept readable data.
Think Like a Manager
Manager thinking means knowing why almost everyone chooses ESP, and being able to say it in one sentence: AH authenticates but leaves the data readable and breaks through NAT, while ESP encrypts, authenticates and traverses NAT. For most VPN deployments, ESP in tunnel mode is the safe default, and a design that reaches for AH should be able to justify what it is buying with the confidentiality it gave up.
That is the same kind of question certificate pinning forces: what is this mechanism actually protecting, and what does it quietly leave exposed? A protocol that authenticates is not automatically a protocol that protects.
AH vs ESP: Quick Comparison
| Property | Authentication Header (AH) | Encapsulating Security Payload (ESP) |
|---|---|---|
| Encryption | None, payload stays readable | Encrypts the payload |
| Integrity coverage | Whole packet, including outer IP header | Payload only, outer IP header unprotected |
| Origin authentication | Yes | Yes, optional but almost always used |
| Works through NAT | No, the integrity check breaks | Yes |
| Protocol number | 51 | 50 |
The NAT row is the one that decided the argument in the real world. AH computes its integrity check across fields that Network Address Translation rewrites in transit, so the check fails on arrival and the packet is dropped. ESP leaves the outer header out of its calculation, so translation is invisible to it. Since almost every internet path crosses at least one NAT device, AH is rare outside closed networks.
Transport Mode vs Tunnel Mode
Mode is a separate choice from protocol, and the exam mixes the two axes deliberately.
- Transport mode protects the payload only and keeps the original IP header, so it suits host to host communication between two endpoints that already know each other’s addresses.
- Tunnel mode wraps the entire original packet, header included, inside a new packet carrying a new IP header. That encapsulation is what lets two gateways carry traffic on behalf of the networks behind them, which is why tunnel mode is the site to site VPN workhorse.
Combining the axes gives the answer most scenarios want: ESP in tunnel mode, encrypting the payload and hiding the original addressing behind the gateway’s own.
How the CISSP Exam Tests IPsec
Watch for the word the scenario uses about the data itself. If it says confidential, private, or must not be readable, the answer involves ESP, because AH cannot deliver confidentiality at any setting. If it says the packet crosses NAT, AH is eliminated. If it describes two gateways joining two networks, the mode is tunnel. Most IPsec questions are decided by one of those three sentences rather than by anything about cryptography.
Protocol pairs like this reward recognition rather than reasoning under time pressure. Our CISSP practice tests drill AH against ESP, and transport against tunnel, in the scenario wording the exam actually uses.