LACP and EtherChannel Configuration: A Link Aggregation Guide

How is link aggregation with LACP set up and verified? We cover the difference between LACP, PAgP and static mode, load balancing and matching with the server side.
Technical summary
Technology
IEEE 802.1AX (LACP), Cisco EtherChannel, PAgP, Linux bonding, Windows NIC Teaming
Topic
Combining physical links into a single logical link
Level
Intermediate
Prerequisites
  • Trunk and access port concepts
  • Basic Cisco IOS configuration knowledge
  • Permission to manage network interfaces on the server side
In this article8 sections

You connected two 10G uplinks and built a port channel, but the nightly backup still tops out at 10G. Or one member always shows suspended, and another morning the port has gone err-disabled. Link aggregation turns multiple physical Ethernet links into a single logical link in the eyes of the switch and STP; built correctly, it provides capacity and link redundancy, built wrongly, these symptoms. Below we cover LACP and EtherChannel with matching examples on the Cisco IOS, Linux and Windows sides and verification commands.

Once a port channel is built, STP sees the member ports as a single port; there’s no need to block one of the two links, and both carry traffic. When a member goes down, traffic moves to the remaining members and STP doesn’t recalculate. The interruption is usually limited to the time it takes to detect the physical link loss.

What it doesn’t provide should also be known:

  • A single flow doesn’t get faster. Two 10G links don’t give 20G to a single backup job; all frames of a flow are sent over the same member so their order isn’t disrupted.
  • It doesn’t provide device redundancy. Standard LACP assumes a single device at each end. A port channel spanning two switches at one end requires stacking, MLAG or vPC.
  • Distribution isn’t automatically balanced. If the hash algorithm and traffic mix aren’t suitable, one member fills up and the other stays empty.

A port channel doesn’t add up bandwidth. It distributes flows.

LACP, PAgP and static mode

LACP (Link Aggregation Control Protocol) was first defined in IEEE 802.3ad and later moved to the IEEE 802.1AX standard. The two ends learn the other side’s system ID and key information through an exchange of LACPDUs; only ports connected to the same far-end system and compatible are added to the group. PAgP is Cisco-specific; there’s no reason to use it in a new design.

Mode (Cisco) Protocol Behavior Working combinations
active LACP Sends LACPDUs active–active, active–passive
passive LACP Only responds passive–active
desirable / auto PAgP Cisco-specific negotiation desirable–desirable, desirable–auto
on None (static) Doesn’t negotiate Only on–on

passive–passive and auto–auto combinations never form a group. Setting both sides to active is the choice with the fewest surprises. Static mode (on) shouldn’t be used except in special cases such as old devices that don’t support LACP or the “Route based on IP hash” policy of the VMware standard vSwitch: if one end has formed the group and the other hasn’t, traffic can fall into a black hole or a loop can form.

The LACP timeout is also a design parameter. In slow mode, LACPDUs are sent every 30 seconds; in fast mode, every second; the timeout is three missed PDUs. If the other side’s control plane hangs while the physical link is up, detecting this in slow mode can take up to 90 seconds. On platforms that support it, fast mode is enabled with lacp rate fast on member ports; match the setting on both sides.

Load balancing algorithms and the single-flow limit

For each frame, the switch calculates a hash from certain header fields (MAC, IP, L4 port) and maps the result to a member. The more varied the values the selected fields produce, the more balanced the distribution.

  • src-mac / dst-mac: In traffic going to a router or firewall, the destination MAC is always the same; everything piles onto a single member. Not suitable for campus uplinks in an enterprise network infrastructure.
  • src-dst-ip: A good default for traffic behind L3 boundaries.
  • src-dst-port or options including L4: Improves distribution if there are many sessions between the same two servers (backup, replication).

On Cisco Catalyst, the algorithm is mostly chosen globally and applied to all port channels. The hash only runs on the sending side; having different algorithms at the two ends isn’t an error, but you need to evaluate the distribution in each direction separately.

port-channel load-balance src-dst-ip
show etherchannel load-balance
! See which member a specific flow will land on (on platforms that support it):
test etherchannel load-balance interface Port-channel1 ip 10.10.20.15 10.10.50.8

Worked example: in a port channel with 4 × 10G members, a single backup flow running at night can use at most 10G; 40 different client–server pairs running at the same time, on the other hand, can spread across all four members. If single-flow speed is decisive, you need either a faster single link (25G/40G/100G) or parallel flows at the application layer.

EtherChannel configuration example on Cisco IOS

The example below combines two 10G trunk uplinks from an access switch to a distribution switch with LACP. Configure the member ports while they are shut down; a half-finished configuration can create a momentary loop.

interface range TenGigabitEthernet1/1/1 - 2
 shutdown
 description UPLINK-DIST-SW1
 switchport mode trunk
 switchport trunk allowed vlan 10,20,30,99
 switchport trunk native vlan 999
 channel-group 1 mode active
!
interface Port-channel1
 description UPLINK-DIST-SW1
 switchport mode trunk
 switchport trunk allowed vlan 10,20,30,99
 switchport trunk native vlan 999
!
interface range TenGigabitEthernet1/1/1 - 2
 no shutdown

After the port channel is built, change trunk, VLAN segmentation and STP settings on the Port-channel interface; IOS propagates them to the members. A different setting written on a single member drops that member from the group. On some older Catalyst models, you first need to enter switchport trunk encapsulation dot1q for a trunk. The active member limit in the group and hot-standby behavior (lacp max-bundle, lacp port-priority) vary by platform.

Compatibility with the server and storage side

Setting up LACP on the switch side doesn’t mean the server also speaks LACP. With a mismatched configuration, the switch ports show as (I) stand-alone or (s) suspended. This is a scenario you may encounter: the network team builds a port channel, and the server team enables teaming in a different mode.

Linux bonding

On Linux, LACP is provided by the bonding driver’s 802.3ad (mode 4) mode. The hash on the transmit side is selected with xmit_hash_policy:

nmcli con add type bond con-name bond0 ifname bond0 \
  bond.options "mode=802.3ad,miimon=100,lacp_rate=fast,xmit_hash_policy=layer3+4"
nmcli con add type ethernet con-name bond0-p1 ifname ens1f0 master bond0
nmcli con add type ethernet con-name bond0-p2 ifname ens1f1 master bond0

cat /proc/net/bonding/bond0

In the /proc/net/bonding/bond0 output, both members should carry the same “Aggregator ID” value, and the “Partner Mac Address” field should show the switch’s system MAC. If the members are in different aggregator IDs, the switch side hasn’t formed the group. If the partner MAC is zero, no LACPDUs are arriving at all.

Windows and virtualization

On physical Windows Server, LBFO teaming supports LACP mode:

New-NetLbfoTeam -Name "Team1" -TeamMembers "NIC1","NIC2" `
  -TeamingMode Lacp -LoadBalancingAlgorithm Dynamic
Get-NetLbfoTeamMember -Team "Team1"

For the Hyper-V virtual switch, however, the method Microsoft recommends is Switch Embedded Teaming, and SET works only in switch-independent mode; in this case no port channel is built on the switch side. The server side is explained in the Hyper-V virtual network design and SET article. In VMware, LACP is supported only on a distributed switch; with a standard vSwitch, only static EtherChannel and IP hash can be paired. For iSCSI storage traffic, use multipath I/O (MPIO) over two separate paths instead of link aggregation; this is also the common recommendation of storage vendors.

Member port inconsistencies and err-disable states

A port is added to the group only if these parameters are the same as the other members:

  • Speed and duplex
  • Switchport mode (access or trunk)
  • The access VLAN, or the allowed VLAN list and native VLAN on a trunk
  • Depending on the platform, MTU, QoS and port security settings

An inconsistent member drops into the suspended state. More serious is one side having built a port channel while the other hasn’t: on Cisco, spanning-tree etherchannel guard misconfig, which is enabled by default, detects this and err-disables the ports with the channel-misconfig reason. This is a protection. Before reopening the ports, put the configurations of both ends side by side.

Verification and troubleshooting commands

show etherchannel summary
show etherchannel 1 detail
show lacp neighbor
show lacp counters
show interfaces Port-channel1
show interfaces TenGigabitEthernet1/1/1 counters errors
show interfaces status err-disabled

The first thing to look at is the flags in the show etherchannel summary output: P shows the member is working in the group, I that the port is running standalone because no LACPDUs are being received, s that it’s suspended, D that it’s down. On the Port-channel line you should see SU (L2, in use) or RU (L3, in use). Verify with show lacp counters that the LACPDU send and receive counters are increasing; if the receive counter is static, the other side isn’t speaking LACP.

For capacity, compare the members’ traffic counters individually. If one member consistently carries several times the others, review the hash algorithm. If error counters increase on only one member, the problem is most likely that member’s cable or optical module. A port channel hides such physical problems within the average. To report uplink utilization trends, see the network capacity reporting article.

If the port channel needs to span two separate switches, the topic moves to device redundancy; those architectures are explained in the switch stack, MLAG and vPC comparison. For the interaction of port channels with STP, the STP, RSTP and MSTP article is a complement. All bonding parameters on the Linux side are listed in the kernel bonding documentation. For support with installing uplinks and server connections, see our switch, router and network device installation service page.

Frequently asked questions

No. The hash algorithm places each flow on a single member; a single TCP session reaches at most the speed of one member, i.e. 10G. Total capacity is used only when there are many different flows.

Should I use LACP or static EtherChannel?

LACP, if the other side supports it. LACP verifies that the members are really connected to the same far-end device and doesn’t put a miswired port into the group. Static mode doesn’t do this check; a wrong match can cause traffic loss or a loop.

How many ports should I put in a port channel?

Because many Cisco platforms distribute the hash result across 8 buckets, 2, 4 or 8 members give more balanced load sharing. In a 3-member group, the buckets are distributed as 3, 3 and 2. Verify the behavior of your platform in the vendor documentation.

Can I use LACP on a Hyper-V server?

The method Microsoft recommends for the Hyper-V virtual switch is Switch Embedded Teaming; SET works only in switch-independent mode and doesn’t support LACP. In this case, no port channel is built on the switch side; the ports are configured individually as trunks.

Support

Contact us for support or a discovery session on this topic.

Contact form
[email protected]

Eretek Network

We build switching, routing, cabling and wireless networks with VLAN segmentation and redundant uplinks, and provide regular maintenance and fault support.

Knowledge Center

Other articles