Site-to-Site IPsec VPN Design: Route-Based and Policy-Based

How is an IPsec VPN between branches designed? We explain the difference between policy-based and route-based tunnels, IKEv2 mismatches, the MTU problem and a backup tunnel with an example configuration.
Technical summary
Technology
IPsec, IKEv2, VTI, BGP, OSPF, Cisco IOS XE
Topic
IPsec tunnel and routing design between locations
Level
Intermediate–advanced
Prerequisites
  • Basic IP routing knowledge
  • Experience configuring VPNs on a firewall/router
In this article8 sections

A site-to-site IPsec tunnel works on the day it’s built; the problem comes months later as “file sharing is stalling” or “the primary line went down but it didn’t switch to the backup tunnel.” Most of these faults come not from encryption but from connectivity and routing design. Below we explain the difference between policy-based and route-based VPNs, IKEv2 parameter matching, dynamic routing over the tunnel, MTU/MSS settings and backup tunnel design, with an example Cisco VTI configuration and verification commands to check in order on the console.

Site-to-site VPN topologies

As the number of locations grows, the choice of topology determines the management burden:

  • Hub-and-spoke: Each branch builds a tunnel only to headquarters (the hub). The number of tunnels equals the number of branches; traffic between branches passes through the hub. The most common model for setups where applications are at headquarters.
  • Full mesh: Each location builds a tunnel to every other location; n locations require n(n−1)/2 tunnels. Worked example: 45 tunnels for 10 locations, 190 for 20 locations. Manual management quickly reaches a dead end.
  • Partial mesh: Locations with heavy traffic exchange (for example headquarters, the data center and the disaster recovery site) connect to each other directly, and branches connect to the hub.
  • Dynamic mesh: Vendor-specific solutions such as Cisco DMVPN and Fortinet ADVPN build direct tunnels between branches on demand. If central policy and application-based path selection are also wanted on top of this, an SD-WAN evaluation comes onto the agenda.

The difference between policy-based and route-based VPNs

In a policy-based VPN, an access list or policy determines which traffic enters the tunnel. For example, the rule “encrypt traffic with source 10.10.0.0/16 and destination 10.20.0.0/16” is passed to the other side in IKE negotiation as a traffic selector (proxy ID on some vendors). In a route-based VPN, on the other hand, a virtual tunnel interface (VTI) is created; traffic selectors are usually 0.0.0.0/0 in both directions, and the routing table decides which traffic enters the tunnel.

Criterion Policy-based Route-based (VTI)
Tunnel entry decision ACL / policy Routing table
Dynamic routing Not in practice BGP, OSPF supported
Adding a new subnet ACL change at both ends Adding a route is enough
Backup tunnel Complex Natural with routing
Monitoring Per SA Interface counters, SNMP
Compatibility Common on older devices On most current devices

The classic field fault of a policy-based setup is a traffic selector mismatch. If one end defines 10.20.0.0/16 and the other 10.20.1.0/24, IKEv2 negotiation fails with a TS_UNACCEPTABLE error or the tunnel works only in one direction. Because some implementations create a separate IPsec SA for each subnet pair, with many subnets the number of SAs grows quickly and troubleshooting gets harder. In new installations, choose route-based if the other end supports it. You don’t do routing with ACLs.

IKEv2 and IPsec parameter matching

IKEv2 (RFC 7296) works in two stages: first the IKE SA is established (encryption, integrity/PRF, Diffie-Hellman group and authentication), then Child SAs (IPsec SAs) that carry traffic are negotiated over it. When building a tunnel between different vendors, preparing a single parameter table for both sides and sharing it with the other team saves hours during commissioning:

Parameter Example value Note
IKE version IKEv2 IKEv1 only if mandatory
IKE encryption / PRF AES-GCM-256 / SHA-384 No separate integrity with GCM
DH group 20 (ECP-384) or 19 Groups 1, 2, 5 shouldn’t be used
Authentication Certificate or strong PSK Identity type the same at both ends
ESP AES-GCM-256 Tunnel mode
PFS On, group 20 The same group at both ends
Lifetime IKE 24 hours, IPsec 1 hour Not negotiated in IKEv2

Something that can be overlooked is an identity (IKE ID) mismatch: if one end sends its identity as an IP address and the other as an FQDN, authentication fails even if the PSK is correct. The log says “authentication failed,” everyone changes the key, and the problem isn’t solved. Check the identity type first.

If one of the ends is behind NAT, IKE starts on UDP 500, switches to UDP 4500 when NAT is detected, and ESP packets are carried inside UDP (NAT-T); without NAT, ESP goes as IP protocol 50. All three flows need to be open in firewall rules. For security controls such as algorithm selection and key management, see the enterprise VPN security controls article.

Dynamic routing over the tunnel

Using dynamic routing instead of static routes in route-based tunnels automates both adding new subnets and switching to the backup tunnel.

  • BGP: It scales in an enterprise network infrastructure with many branches; which prefix is accepted from which branch is tightly controlled with prefix lists, and tunnel preference is managed with local-preference or AS-path prepend. Giving each branch its own private AS number makes the design clear.
  • OSPF: Simple with a small number of tunnels. Verify that the network type on tunnel interfaces is point-to-point (this is the default on Cisco tunnel interfaces; on some firewalls it needs to be set manually), and design branches as stub areas where possible. For OSPF area design on the campus side, see the OSPF design in a campus network article.

The most annoying fault in this section is recursive routing: if the route to the other end’s public IP address is learned through the tunnel, the tunnel tries to carry itself, comes up, goes down and comes up again. If you see “tunnel up/down” several times a minute in the log, this is the first suspect. Filter the tunnel endpoints’ public IPs out of dynamic routing; these addresses must always be reached via the physical internet interface.

MTU, MSS and fragmentation problems

ESP tunnel mode adds a new IP header, an ESP header, an IV, padding and an integrity value to each packet; with NAT-T, an 8-byte UDP header is also added. Depending on the algorithm used, the overhead ranges from about 50 to 80 bytes. A 1500-byte packet exceeds the physical interface’s MTU after encryption. If the packet is fragmented, performance drops; if the DF bit is set and ICMP “fragmentation needed” messages are blocked along the path, the packet is silently lost. The symptom is always the same: ping and small requests work, while large file transfers, RDP and some web applications stall. The user says “the VPN is slow.” In fact, packets are being lost.

The practical solution is to lower the IP MTU on the tunnel interface and set the MSS of TCP sessions accordingly. Commonly used safe values are IP MTU 1400 and MSS 1360 (1400 − 20 bytes IP − 20 bytes TCP). If the underlying line is PPPoE (1492 MTU), or the tunnel includes additional encapsulation such as GRE, lower the values further. The MSS setting only rescues TCP; for applications that send large UDP packets, the MTU side must be right.

Backup tunnel, DPD and failover

If a branch has two internet lines, the cleanest method is to build a separate tunnel over each line and run a dynamic routing adjacency on both. Routes learned from the primary tunnel are given a higher local-preference (BGP) or a lower cost (OSPF). The failover behavior of the lines themselves is explained in the dual internet line article.

Failure detection is done at two layers. The IKEv2 liveness check (commonly called DPD) detects that the other end isn’t responding and clears the SAs; the routing protocol’s timers, on the other hand, withdraw the routes. BGP’s default timers (keepalive 60, hold 180 seconds on Cisco) are long for failover; lower the timers or use BFD if the platform supports it. Waiting three minutes doesn’t count as failover. If there are two separate hub devices at headquarters, branches also protect against device failure by building tunnels to both.

Example VTI configuration and verification

The example below builds a tunnel with IKEv2 and a static VTI from the headquarters side to a branch on Cisco IOS XE, and opens a BGP adjacency over the tunnel. Addresses were chosen from documentation ranges; the headquarters public IP is 203.0.113.10 and the branch public IP 198.51.100.20.

crypto ikev2 proposal PROP-GCM
 encryption aes-gcm-256
 prf sha384
 group 20
crypto ikev2 policy POL-GCM
 proposal PROP-GCM
!
crypto ikev2 keyring KR-BRANCH1
 peer BRANCH1
  address 198.51.100.20
  pre-shared-key STRONG_RANDOM_KEY
!
crypto ikev2 profile PROF-BRANCH1
 match identity remote address 198.51.100.20 255.255.255.255
 identity local address 203.0.113.10
 authentication remote pre-share
 authentication local pre-share
 keyring local KR-BRANCH1
 dpd 10 3 on-demand
!
crypto ipsec transform-set TS-GCM esp-gcm 256
 mode tunnel
crypto ipsec profile IPSEC-BRANCH1
 set transform-set TS-GCM
 set pfs group20
 set ikev2-profile PROF-BRANCH1
!
interface Tunnel10
 description VPN-BRANCH1
 ip address 172.31.255.1 255.255.255.252
 ip mtu 1400
 ip tcp adjust-mss 1360
 tunnel source GigabitEthernet0/0/0
 tunnel mode ipsec ipv4
 tunnel destination 198.51.100.20
 tunnel protection ipsec profile IPSEC-BRANCH1
!
ip prefix-list BRANCH1-IN seq 10 permit 10.20.0.0/16 le 24
!
router bgp 65000
 neighbor 172.31.255.2 remote-as 65001
 neighbor 172.31.255.2 timers 10 30
 address-family ipv4
  network 10.10.0.0 mask 255.255.0.0
  neighbor 172.31.255.2 activate
  neighbor 172.31.255.2 prefix-list BRANCH1-IN in

For the network command to be able to announce the 10.10.0.0/16 prefix, this exact prefix must be in the routing table (for example with a summary static route). The branch side is configured with the same parameters, with the addresses reversed. The verification order on the console:

  1. show crypto ikev2 sa: the IKE SA state must be READY; check the selected encryption and DH group.
  2. show crypto ipsec sa interface Tunnel10: the #pkts encaps and #pkts decaps counters must increase in both directions. If only one is increasing, the problem is usually routing or a firewall rule at the other end.
  3. show ip bgp summary: the adjacency must be established, and the expected number of prefixes received from the branch.
  4. ping 10.20.0.1 source Loopback0 size 1400 df-bit: an unfragmented packet size test; if it fails, review the MTU values.
  5. If there’s a backup tunnel, shut down the primary line in a planned way and measure the switchover time of routes and sessions.

Don’t skip this order. If there’s no IKE SA, there’s no point looking at routing; if the counters are one-way, there’s no point looking at the MTU. Proceed layer by layer.

If the same subnet is used at two locations, traffic can’t be routed even if the tunnel comes up; in this case address translation is needed, and the two-way NAT section in the enterprise NAT design article can be applied. For routing and segmentation design between locations, see the VLAN, routing and network segmentation page, and for the line side, the Metro Ethernet and internet line management page.

Frequently asked questions

Can a route-based VPN be built between devices from different vendors?

Yes. Because IKEv2 and IPsec are standard protocols, a route-based tunnel can be built if both ends support VTI or a tunnel interface. If one end works only policy-based, the traffic selectors on the route-based side need to be defined according to the subnets the other side expects; many vendors offer proxy ID or similar settings for this.

The tunnel comes up but some applications don’t work; what should I check first?

If ping works but applications that send large packets, such as file sharing, web applications or RDP, stall, the first suspect is the MTU and MSS setting. Lower the IP MTU on the tunnel interface, set the TCP MSS value accordingly, and verify the largest packet the tunnel can carry with an unfragmented ping test.

In a hub-and-spoke design, how do branches reach each other?

In classic hub-and-spoke, traffic between branches passes through the hub; the hub announces the routes it learns from branches to the other branches. If traffic between branches is heavy and latency matters, consider vendor-specific solutions that build direct tunnels on demand, such as DMVPN and ADVPN, or SD-WAN.

In IKEv2, do SA lifetimes have to be the same on both sides?

In IKEv2, SA lifetimes aren’t negotiated; each end starts rekeying when its own lifetime expires. Different values don’t make the tunnel impossible to build. Still, using consistent values on both sides makes log interpretation and troubleshooting easier.

Support

Contact us for support or a discovery session on this topic.

Contact form
[email protected]

Eretek Network

We build switching, routing, cabling and wireless networks with VLAN segmentation and redundant uplinks, and provide regular maintenance and fault support.

Knowledge Center

Other articles