A significant share of network outages comes not from faults but from planned changes: an ACL applied to the wrong port, a trunk command with the word “add” forgotten, a route that cuts off management access. Below we explain the technical execution of a change: capturing the state before the change, a step-by-step implementation plan, automatic rollback mechanisms on Cisco and other platforms, precautions against losing remote access, and verification afterwards. The topic isn’t the approval and audit evidence side, but what the engineer does on the console within the window.
Sources of outage risk in network changes
Risk determines which safety net is needed. Outage sources you may encounter are:
- Cutting off your own access: Losing access to the remote device through changes such as a VTY ACL, the management SVI, the default route or removing the management VLAN from a trunk.
- Commands that overwrite: On Cisco, the command
switchport trunk allowed vlan 30makes the allowed list only VLAN 30; adding to the existing list requiresswitchport trunk allowed vlan add 30. Redefining a prefix-list or ACL can also delete existing lines. - Control plane reconvergence: A change of STP root bridge, OSPF/BGP adjacencies dropping or port-channel memberships breaking cause short but wide-reaching outages.
- Hidden dependencies: An old device running on a VLAN thought to be unused, an application dependent on a static route.
- Missing records: Not knowing what changed and what to roll back to because the previous state wasn’t recorded.
The first two items on the list are among the leading causes of outages that require a midnight trip to a remote location with a console cable. Both are one-line mistakes.
Baseline state capture before the change
All verification rests on comparing “before” and “after.” Immediately before the change, the same set of commands is run on every affected device and saved to a file. A typical set for Cisco IOS/IOS XE:
terminal length 0
show running-config
show version
show interfaces status
show interfaces trunk
show etherchannel summary
show spanning-tree root
show ip interface brief
show ip route summary
show ip ospf neighbor
show ip bgp summary
show mac address-table count
show logging
The outputs are added to the change record. After the change, the same set is taken again and compared on Linux with diff -u before.txt after.txt. Fields that change naturally, such as counters and uptime, produce noise; in the comparison, prioritize state-focused commands (adjacencies, port status, route summary). In an environment where configuration backups are kept continuously and versioned, the configuration diff before and after the change also comes out automatically; this setup is explained in the configuration backup with Oxidized article.
Implementation plan template: step, expected result, rollback
A good plan is clear enough for someone other than the engineer who wrote the change to follow. Four pieces of information are written for each step: what will be done (exact commands), the expected result, how it will be verified and how it will be rolled back if it fails. Below is a sample plan for adding a new VLAN and SVI to a distribution switch:
| Step | Action | Verification | Rollback |
|---|---|---|---|
| 1 | Capture state, set the reload in or revert timer |
Outputs in the record | — |
| 2 | Create VLAN 140 | show vlan id 140 |
no vlan 140 |
| 3 | allowed vlan add 140 on the uplink trunks |
show interfaces trunk |
allowed vlan remove 140 |
| 4 | SVI 140 and IP address | Ping to the gateway | no interface vlan 140 |
| 5 | Advertising the prefix into OSPF | Route on the neighbor device | Remove the relevant network line |
| 6 | Cancel the timer, save | Before/after diff | — |
The plan also includes the affected systems, the estimated duration of the change, the time reserved for rollback and the criterion for the rollback decision (for example, “roll back if the test client can’t reach the gateway after step 4”). If the criterion isn’t written in advance, within the window someone says “let’s try a bit more” and the time melts away. The decision is made beforehand.
Not losing remote access: reload in, archive and configure replace
For every change on a remote device that could cut off access, an automatic rollback mechanism is set up. On Cisco IOS, the oldest and simplest method is a scheduled reload:
reload in 15
! make the changes, DO NOT SAVE
! if access continues and verification is complete:
reload cancel
copy running-config startup-config
The logic is this: if the change cuts off access, the device boots 15 minutes later with the saved (pre-change) configuration. That’s why write memory isn’t done before the timer is cancelled. When reload in is issued while there are unsaved changes, the device may ask “System configuration has been modified. Save?”; the answer must be no, otherwise the safety net is disabled from the start. Typing “yes” by reflex is a mistake you may encounter. The method’s drawback is that the rollback is a full reload, meaning an outage.
A cleaner method is the configuration archive and rollback features in Cisco IOS. First the archive is defined:
archive
path flash:archive/$h-
maximum 10
write-memory
!
configure terminal revert timer 10
! apply the changes
! if everything is fine:
configure confirm
For changes made in the configuration mode entered with configure terminal revert timer, if configure confirm isn’t issued within the specified time, the device is returned to the previous configuration in the archive without a reload. To deliberately return to a specific archive file, configure replace flash:archive/<file> list is used; the list option shows the lines applied. Verify platform and version support in your own device’s configuration guide.
Equivalents on other platforms:
| Platform | Safety net | Note |
|---|---|---|
| Cisco IOS/IOS XE | reload in, configure terminal revert timer, configure replace |
Revert requires archive |
| Cisco NX-OS | checkpoint ve rollback running-config checkpoint |
Take a named checkpoint before the change |
| Juniper Junos | commit confirmed <minutes> |
A second commit is required within the time |
| NETCONF-capable devices | Confirmed commit capability | Defined in RFC 6241 |
In addition to these mechanisms, critical locations have an independent out-of-band access path (console server, management over a separate line). Before the change, it’s tested that this path really works. A situation you may encounter is finding out on the first night the console server is needed that it doesn’t work. An untested backup path isn’t a path.
Choosing the maintenance window and communication
Choosing the window isn’t a technical decision; it’s based on business impact. Production shifts, accounting period-end, backup job times and data replication windows are checked. For example, a change in the enterprise network infrastructure that coincides with the night backup window comes back in the morning as a “backup failed” alarm. The window can’t be shorter than the sum of implementation, verification and rollback times. For a change affecting a live production line, apply the change window engineering in OT approach for plant approval, rollback order and on-site verification.
The communication plan includes: affected user groups and systems, the expected impact (full outage, short outage, no impact expected), the people making the change and making decisions, application owners reachable during the window and, if needed, carrier or vendor support contacts. The start, the end and, if a rollback was done, its reason are posted in the relevant channel.
Post-change verification and observation period
Verification has two layers. In technical verification, the command set from the “before” record is run again; differences should be explainable only by the planned changes. The number of adjacencies, the number of connected ports, port-channel members and the STP root should stay the same. In functional verification, the purpose of the change is tested: does the test client on the new VLAN get an IP, can it reach the gateway and critical servers, does it resolve DNS? A short confirmation from application owners catches user-side problems before the morning shift starts.
Then an observation period is defined; during this period, interface error counters, CPU and log messages in the monitoring system are tracked. The configuration is saved, the old configuration is kept in the archive and the change record is closed with the “before/after” outputs. If information such as a new VLAN, IP or port changed, the network documentation is updated before the record is closed. On the audit side, which evidence should be in the change record is covered in the change management audit evidence guide.
Go/no-go checklist
If all of the items below aren’t “yes” before the window starts, the change is postponed. No exceptions.
- Is the implementation plan written with exact commands and reviewed by a second engineer?
- Is the verification and rollback method defined for each step?
- Have current configuration backups and “before” outputs of the affected devices been taken?
- Has out-of-band or console access been tested for this window?
- Is an automatic rollback mechanism (reload in, revert timer, commit confirmed) included in the plan?
- Does the window duration cover implementation, verification and rollback time?
- Have affected users and application owners been informed, and are they reachable during the window?
- Is there no other conflicting change or backup job?
- For changes that include a firmware update, have image verification and the upgrade path been checked? (For details, see the firmware update policy.)
For planning critical core or WAN changes and executing them within the window, you can use our switch, router and network device installation service; for running regular maintenance changes with the same discipline, our network maintenance, monitoring and periodic checks service.
Frequently asked questions
What is the most common mistake when using the reload in command?
Saving the configuration without cancelling the timer. In that case the scheduled reload boots with the faulty configuration and the safety net is useless. The order is always the same: first reload cancel, then save.
What is the difference between configure terminal revert timer and reload in?
reload in reverts to the saved configuration by restarting the device if no confirmation comes, and causes an outage. The revert timer uses the archive feature; if no confirmation is given, it returns the device to the previous configuration without restarting it.
Does every network change need a maintenance window?
No. Standard changes whose impact is limited to a single access port and that are easy to roll back can be made during business hours with pre-approved templates. Trunk, routing, core and firmware changes are made within a window and with a rollback plan.
How long should the post-change observation period be?
It depends on the type of change. For changes whose effect is visible immediately, a short period may be enough; for changes whose effect depends on load and time, such as routing, QoS or firmware, observation for at least one full business day is recommended.