ENGINEERING GUIDE · SECURITY

IPsec VPN Engineering Guide

A practical engineering reference for IKEv2, Child SAs, traffic selectors, cryptographic proposals, PFS, NAT-T, lifetimes and interoperability troubleshooting.

Engineering referenceIPsec · IKEv2

Related Ingenix tool

Build and validate a vendor-neutral VPN baseline in your browser.

VPN Configuration Designer →
The important bit: An IPsec tunnel has distinct negotiation stages. “The VPN is down” is not a useful diagnosis until you know whether peer reachability, IKE, Child SA negotiation, routing, policy or traffic handling is failing.

In this guide

  1. Terminology
  2. IKE and Phase 1
  3. Child SAs and Phase 2
  4. Traffic selectors
  5. Cryptographic proposals
  6. NAT-T and transport
  7. Lifetimes and rekeying
  8. Troubleshooting and evidence

Terminology

TermMeaning
IKE SAThe authenticated control-plane security association between VPN peers.
Child SAThe IPsec security association used to protect data traffic.
ESPEncapsulating Security Payload, the IPsec protocol normally carrying protected traffic.
Traffic selectorSource/destination traffic definition associated with a Child SA.
PFSPerfect Forward Secrecy: additional key exchange for Child SA rekeying.
NAT-TNAT Traversal, which encapsulates ESP inside UDP when required by the path.

IKE / Phase 1

IKE establishes the control relationship between the peers. In IKEv2, the initial exchange negotiates cryptographic parameters and authenticates the peers before creating the IKE SA.

Confirm the following on both sides:

  • IKE version.
  • Peer address and identity.
  • Authentication method and credentials/certificate trust.
  • Encryption algorithm.
  • Integrity/PRF.
  • Diffie-Hellman group.
  • Lifetime and reauthentication/rekey behaviour.

When troubleshooting, find the specific negotiation failure rather than stopping at a generic message such as “no proposal chosen” or “negotiation failed”. Compare the exact proposal offered by one peer with what the other peer accepts.

Child SAs / Phase 2

Once the IKE relationship exists, Child SAs protect user traffic. A Child SA normally defines ESP protection, traffic selectors, lifetimes and optionally a PFS group.

A common mistake is assuming that an established IKE SA means the VPN can carry traffic. It does not. If Child SA negotiation fails, inspect the IPsec proposal, PFS setting, lifetimes and selectors.

Traffic selectors

Selectors identify which source and destination traffic belongs to the Child SA. A simple policy-based tunnel might specify one local subnet and one remote subnet. A route-based design may use broader selectors while the routing table determines which traffic reaches the tunnel.

Check that:

  • Local and remote networks are reversed correctly at opposite ends.
  • There is no unintended overlap.
  • IPv4 and IPv6 selectors are not being mixed unintentionally.
  • Host and subnet masks match the intended policy.
  • Both peers support the selector narrowing behaviour being used.

Cryptographic proposals

ParameterPreferred approachCheck
IKE encryptionAES-GCM where supported.Both peers offer the same usable option.
Integrity/PRFModern SHA-2 family.Do not assume the IKE and ESP settings are identical.
DH/PFSMutually supported modern groups.PFS is configured consistently for Child SAs.
Legacy algorithmsAvoid where possible.Document any temporary compatibility exception.

Authenticated encryption such as AES-GCM combines confidentiality and integrity. With older AES-CBC designs, a separate integrity algorithm is normally required.

NAT-T and transport

Native IKE commonly uses UDP/500. When NAT is detected, NAT-T commonly carries the exchange and ESP traffic over UDP/4500. This is different from ordinary source NAT: NAT-T is a mechanism for allowing IPsec to traverse a translating device.

Check every device between peers. A general outbound NAT rule can also break a tunnel if it translates the protected traffic when the design expects a NAT exemption or different handling.

Lifetimes and rekeying

IKE and Child SAs have lifetimes and rekey behaviour. Large differences can produce repeated reauthentication or apparently random tunnel drops. Rekeying should be tested rather than assumed to work because the initial tunnel established successfully.

Record tunnel establishment and rekey timestamps in logs. If a failure happens at a repeatable interval, compare that interval with IKE/Child SA lifetime, authentication lifetime and upstream session timeouts.

Troubleshooting and evidence

  1. Confirm peer reachability and that UDP/500 or UDP/4500 reaches the expected gateway.
  2. Inspect IKE logs and identify the first negotiation error.
  3. Confirm IKE SA state.
  4. Inspect Child SA negotiation and selectors.
  5. Confirm routes and firewall policy.
  6. Check NAT and session logs.
  7. Compare tunnel byte/packet counters while generating one controlled test.
  8. Take a short packet capture if logs do not explain the failure.

Useful capture points include the WAN interface before encryption, the path between gateways, and the inside interface after decryption. Filter by peer addresses and UDP/500, UDP/4500 or ESP protocol 50. Record the exact time and timezone. Never include private keys, passwords or reusable secrets in evidence shared for troubleshooting.

See the VPN Troubleshooting Guide for packet-capture patterns and a structured evidence checklist.