Enterprise BGP Multihoming: Two Carriers with Your Own AS and IP Block

How do you connect to two carriers with BGP? We cover the address and AS number process, route registry entries, settings that affect route selection and a filtered example.
Technical summary
Technology
BGP-4, RPKI, IRR, RIPE NCC, Cisco IOS XE
Topic
Dual-carrier internet access with your own AS and IP block
Level
Advanced
Prerequisites
  • Basic BGP concepts
  • Internet lines from two separate carriers
In this article8 sections

The primary line went down, egress switched to the second carrier, and users are getting out to the internet; but the website, VPN gateway and customer portal are unreachable. The reason is simple: the published services were on IP addresses given by the first carrier. Getting lines from two carriers doesn’t by itself provide carrier independence. In a BGP multihoming setup built with your own AS number and IP block, the same addresses are announced through both carriers at once. Below we explain the registration processes to follow when moving to this setup, route options, traffic engineering tools and filters, with a complete configuration example.

When is BGP multihoming needed?

For internal users’ internet access, dual lines with carrier IPs are enough in most cases; failover and load balancing logic is explained in the dual internet line article. BGP multihoming comes up in these cases:

  • If published services (web, email, VPN gateway, customer portal, API) must remain reachable during a line change without IP and DNS changes.
  • If remote locations, business partners or cloud providers define your IP addresses in their firewall rules and these addresses must not be affected by a carrier change.
  • If you want to avoid the burden of renumbering when changing carriers or adding a carrier.

It has a cost too. Responsibility for operating BGP on the router, keeping registry databases current and route security passes to the organization. Don’t start until it’s clear who will do this work. An unowned BGP session is a risk.

AS number and IP block: the RIR process

The regional internet registry (RIR) for Turkey, Europe and the Middle East is RIPE NCC. There are two ways to obtain resources: becoming a RIPE NCC member and an LIR (Local Internet Registry), or obtaining resources as an end user under an LIR’s sponsorship.

  • AS number: Requested through a sponsoring LIR or as an LIR. The request usually expects a plan to connect to at least two separate upstream providers and a unique routing policy. Numbers assigned today are mostly 32-bit ASNs; confirm 32-bit ASN support on your devices and carriers.
  • IPv6: A PI (Provider Independent) block can be obtained as an end user through a sponsoring LIR, or an allocation as an LIR. The smallest generally accepted announceable IPv6 prefix on the internet is /48.
  • IPv4: Because RIPE NCC’s free IPv4 pool is exhausted, no new IPv4 PI assignments are made. The options are obtaining a block as an LIR from the waiting list or the transfer market, or leasing from an LIR. The smallest generally accepted announceable IPv4 prefix is /24; smaller blocks are filtered by many networks.

IRR records and RPKI ROAs

Carriers filter the prefixes they’ll accept from customers based on Internet Routing Registry (IRR) records and RPKI validation. Two records must be ready before announcing. If one is missing, the session comes up but the prefix doesn’t go out.

route / route6 object: A record in the RIPE database that maps a prefix to its origin AS. Example:

route:    198.51.100.0/24
descr:    Company internet block
origin:   AS64500
mnt-by:   COMPANY-MNT
source:   RIPE

An aut-num object is also created for the AS number, and contact information (admin-c, tech-c) is kept current. Some carriers ask for an as-set or just the AS number to build their filters; clarify this with the carrier in advance. Also ask how often the carrier generates filters from the IRR: just because you created the record today doesn’t mean the filter will be updated today.

ROA: A record within RPKI that cryptographically states which AS is authorized to announce a prefix. In RIPE NCC’s hosted RPKI service, the resource holder creates it. Create the ROA only for the prefix length you actually announce; keeping the maxLength value wide weakens protection against prefix hijacking attacks, as explained in RFC 9319. Networks that do RPKI validation classify an announcement as “valid,” “invalid” or “not found”; “invalid” announcements are rejected by more and more networks. A wrong ROA erases your announcement from part of the internet. The typical symptom in the field is the site opening from some carriers and not from others; the first thing to check is the RPKI status.

Routes to receive from the carrier: default, partial, full

Option What is received? Use Router load
Default Only 0.0.0.0/0 Simple active-backup or weighted egress Low
Partial Default + the carrier’s own and customer prefixes Direct egress to destinations at that carrier Intermediate
Full The entire internet routing table Best egress selection per destination High; memory and CPU should be checked

The most commonly used option in enterprise networks is receiving a default (or default + partial) from both carriers. A full table contains prefixes on the order of a million on the IPv4 side and keeps growing; verify the memory capacity of a router that will hold two full tables from the vendor’s hardware documents. Receiving a full table on firewall devices is generally not recommended. The table grows; the hardware doesn’t.

Outbound traffic: local-preference

Which carrier traffic leaving the organization uses is entirely under your control. The main tool is the local-preference attribute: if there’s more than one path to the same destination, the path with the higher local-preference value is chosen, and this value is carried to other BGP routers within the organization over iBGP. For example, if routes from the primary carrier are assigned 200 and those from the secondary 100, in normal conditions all egress goes through the primary line; when the primary BGP session drops, the routes are withdrawn and traffic moves to the secondary. When a partial table is received, a higher value can be given only to these prefixes to send traffic to the secondary carrier’s own customers out through that carrier.

The point to watch is that a session dropping and a line degrading aren’t the same thing. If the line is losing packets but BGP keepalives are getting through, the session stays up and traffic keeps flowing over the bad line. If BFD is supported, enabling it together with the carrier keeps you from leaving failure detection time to BGP timers.

Inbound traffic: AS-path prepend and communities

Inbound traffic depends on how other networks evaluate your announcement; control is indirect. Tools you can use:

  • AS-path prepend: By adding your AS number several times to the announcement made to the secondary carrier, this path is made to look longer. Because external networks can put their own preferences first with local-preference, its effect is partial.
  • BGP communities: Most carriers offer behaviors such as lowering local-preference in their own network or not announcing to certain neighbors, based on community values the customer adds to the announcement. The values are carrier-specific; get them from the carrier’s customer documentation. There are also well-known communities such as NO_EXPORT in RFC 1997.
  • MED: Only when there’s more than one connection to the same carrier does it affect which connection that carrier prefers; it doesn’t work between different carriers.
  • More-specific prefixes: If you have a /23 block, by announcing the /23 to both carriers and one of the /24s only to A and the other only to B, you can split inbound traffic by prefix; when one carrier goes down, the /23 announcement remains as a backup path. This method can’t be used with a single /24, because prefixes more specific than /24 are generally filtered on the internet. ROA records should be created to cover both the /23 and /24 announcements (with separate ROAs or with maxLength 24).

Don’t guess inbound traffic; measure it. Increasing the number of prepends often doesn’t give the expected effect; communities are usually more decisive.

Prefix filters and an example configuration

The most important rule is this: the organization must not turn into a transit AS. Routes learned from one carrier are never announced to the other. For this, a filter in the outbound direction that allows only your own prefix is used together with an AS-path filter that passes only paths originating from your own AS. RFC 7454 (BGP operations and security) describes these filters in detail. Some newer software versions, in line with RFC 8212, don’t send or receive routes on eBGP sessions without a defined policy; still, don’t rely on this behavior, write the filters explicitly. An unfiltered eBGP session makes someone else’s fault your fault.

The example below is in Cisco IOS XE syntax; AS numbers and addresses were chosen from ranges reserved for documentation (organization AS64500, carriers AS64501 and AS64502).

ip route 198.51.100.0 255.255.255.0 Null0
!
ip prefix-list OWN-OUT seq 10 permit 198.51.100.0/24
ip prefix-list DEFAULT-IN seq 10 permit 0.0.0.0/0
ip as-path access-list 10 permit ^$
!
route-map ISP-A-IN permit 10
 match ip address prefix-list DEFAULT-IN
 set local-preference 200
route-map ISP-B-IN permit 10
 match ip address prefix-list DEFAULT-IN
 set local-preference 100
!
route-map ISP-A-OUT permit 10
 match ip address prefix-list OWN-OUT
 match as-path 10
route-map ISP-B-OUT permit 10
 match ip address prefix-list OWN-OUT
 match as-path 10
 set as-path prepend 64500 64500
!
router bgp 64500
 bgp log-neighbor-changes
 neighbor 203.0.113.1 remote-as 64501
 neighbor 203.0.113.1 description ISP-A
 neighbor 192.0.2.1 remote-as 64502
 neighbor 192.0.2.1 description ISP-B
 address-family ipv4
  network 198.51.100.0 mask 255.255.255.0
  neighbor 203.0.113.1 activate
  neighbor 203.0.113.1 route-map ISP-A-IN in
  neighbor 203.0.113.1 route-map ISP-A-OUT out
  neighbor 203.0.113.1 maximum-prefix 100
  neighbor 192.0.2.1 activate
  neighbor 192.0.2.1 route-map ISP-B-IN in
  neighbor 192.0.2.1 route-map ISP-B-OUT out
  neighbor 192.0.2.1 maximum-prefix 100

The static route to Null0 is needed so the network command can always announce the prefix; more specific routes in the internal network keep carrying traffic to the right destination. Because there’s no explicit “permit” at the end of the route-maps, everything that doesn’t match is denied. maximum-prefix protects the router by dropping the session if an error on the carrier side sends you an unexpected number of routes. If two separate routers are used, an iBGP session (usually with next-hop-self) is built between them, and a single gateway is offered on the LAN side with VRRP or HSRP. How your own block is used internally with the NAT pool and published services is explained in the enterprise NAT design article.

Verification

  • show ip bgp summary: the state of the sessions and the number of prefixes received.
  • show ip bgp neighbors 192.0.2.1 advertised-routes: checking that only your own prefix goes to each carrier. This is the one command that must not be skipped during commissioning.
  • show ip bgp 0.0.0.0: the local-preference value of the selected default route.
  • External view: that the announcement is seen via both paths through the carriers’ looking glass pages and RIPEstat, and that the RPKI status is “valid.”
  • Failover test: in a planned maintenance window, the primary session is shut down and the time it takes for outbound and inbound traffic to switch to the secondary line is measured. For continuous monitoring of line quality, see the WAN SLA measurement article.

Untested failover is an assumption. For carrier coordination and the line side, see the internet line management page; for planning the addressing and BGP architecture, the enterprise network consulting page.

Frequently asked questions

Can I set up BGP with two carriers without my own IP block?

Announcing the IP block given by carrier A through carrier B is possible only with A’s explicit permission and with the IRR and ROA records adjusted accordingly; when you change carriers, you lose the block. For a permanent, carrier-independent setup, you need your own AS number and an IP block registered in your name or authorized by the registrant.

Is it necessary to receive the full internet routing table?

In most enterprise scenarios it isn’t. Receiving only a default route from both carriers, or the carriers’ own and customer routes in addition to the default route, provides sufficient load distribution. A full table makes sense when you want to optimize the exit path per destination and your router’s memory can hold two full tables.

Does AS-path prepend fully steer inbound traffic?

No. Prepend only affects AS path length; other networks can override this preference with their own local-preference policies. For more decisive control over inbound traffic, use the BGP community values the carrier offers; the result still depends on external networks’ policies and must be verified by measurement.

How should the maxLength value be set in a ROA?

Create the ROA exactly matching the prefix length you announce, and don’t keep maxLength wider than necessary. For example, if you only announce a /24, maxLength should be 24. A wide maxLength weakens protection against more-specific spoofed announcements made to look as though they originate from the same AS.

Support

Contact us for support or a discovery session on this topic.

Contact form
[email protected]

Eretek Network

We build switching, routing, cabling and wireless networks with VLAN segmentation and redundant uplinks, and provide regular maintenance and fault support.

Knowledge Center

Other articles