When a switch fails and a new one is installed in its place, or when a faulty change leaves the distribution layer unreachable, the first question is: where is the device’s last working configuration? Manually taken copies sitting undated on a file server don’t give a reliable answer to this question. Below we explain step by step how to back up switch, router and firewall configurations automatically and with Git versioning using the open-source Oxidized, track change diffs and restore from backup.
Why should configuration backup be automatic and versioned?
In an enterprise network infrastructure, configuration backup answers three separate needs:
- Hardware replacement: Loading the last working configuration onto the device that replaces a failed one.
- Rolling back a faulty change: Seeing the difference between yesterday’s and today’s configuration line by line when someone says “it was working yesterday.”
- Audit and traceability: Being able to compare which change was made when against the change records.
A manual backup meets only the first of these, and only on the assumption that the backup is current. A situation you may encounter is the last manual backup being left over from months ago. Automatic backup provides currency, and Git provides history and diffs. The goal is clear: each device’s configuration is taken at set intervals, a new version is created only when it changes, and every change is reported.
Oxidized architecture: source, model, output
Oxidized is a tool written in Ruby and developed as a modern alternative to RANCID (project repository). Its architecture rests on four concepts:
| Component | Function | Example |
|---|---|---|
| Source | Source of the device list | CSV (router.db), SQL, HTTP API |
| Input | How to connect to the device | SSH, Telnet, FTP/TFTP |
| Model | Vendor-specific commands and output cleanup | ios, nxos, junos, eos, procurve, fortios |
| Output | Where the configuration is stored | Git, file, HTTP |
The tool’s real value is in the model. The model defines which commands are run for each vendor (e.g. show version, show inventory, show running-config on Cisco IOS) and how variable lines in the output (such as uptime, clock, “Current configuration : … bytes”) are cleaned up. Without this cleanup, every backup would produce a false diff; this way only real configuration changes create a new version. Oxidized also offers a REST API and an optional web interface (oxidized-web); monitoring platforms such as LibreNMS can display configurations through this API.
Installation and device inventory with router.db
Oxidized is installed as a Ruby gem or with the official Docker image. Gem installation requires build dependencies depending on the distribution (such as Ruby development packages, libssh2, cmake); get the current list from the project documentation. Do the installation with a separate service user:
useradd -m -s /bin/bash oxidized
gem install oxidized oxidized-web
su - oxidized -c "oxidized" # the first run creates a sample configuration
After the first run, the ~/.config/oxidized/config file is edited. An example with a CSV source and Git output:
---
username: oxidized-ro
password: "SecretPassword"
model: ios
interval: 3600
threads: 20
timeout: 30
retries: 2
rest: 127.0.0.1:8888
vars:
remove_secret: true
input:
default: ssh
ssh:
secure: true
output:
default: git
git:
single_repo: true
user: Oxidized
email: [email protected]
repo: "/home/oxidized/.config/oxidized/configs.git"
source:
default: csv
csv:
file: "/home/oxidized/.config/oxidized/router.db"
delimiter: !ruby/regexp /:/
map:
name: 0
model: 1
group: 2
In the router.db file, each line is a device; columns are read according to the map definition above:
core-sw-01.example.local:ios:campus
dist-sw-02.example.local:ios:campus
dc-leaf-01.example.local:nxos:data-center
fw-01.example.local:fortios:security
branch-rtr-ankara.example.local:ios:branch
An IP address can be written instead of the device name, but a DNS name makes the Git history readable. input.ssh.secure: true enforces SSH host key verification. This is the secure choice; the price is updating the key in known_hosts when a device is replaced. If forgotten, that device’s backup silently stops. remove_secret: true removes password and key lines from the backup; its effect on restore is below. Instead of keeping the device list by hand, pull it from your inventory system (e.g. NetBox) or monitoring platform with the HTTP source; that’s the only way the list stays current.
Read-only backup account and device-side settings
The backup account should only read. A service account that can both read and change the whole network’s configuration is the most valuable target for an attacker. Read-only account design varies by vendor:
- Cisco IOS: Assigning
show running-configto a low privilege level isn’t enough; at that level the command shows only the sections the user has configuration rights to, and produces incomplete output. In practice the most robust method is to give the account level 15 and allow onlyshowandterminalcommands with TACACS+ command authorization. The output comes complete, and configuration commands are rejected. - Juniper Junos: A read-only account is defined by granting
viewandview-configurationpermissions with a login class. - Firewalls and other platforms: Most vendors have a read-only administrator profile; verify in the pilot that the profile can see the entire configuration.
Things to check on the device side: can the Oxidized server reach it via SSH from the management network, do the VTY ACLs allow the server address, is every login of the account logged with AAA? The backup account’s logins form a regular pattern in the central logs. A login outside this pattern (a different time, a different source) should be investigated. For the log collection side, see the central syslog server setup article.
Change diff tracking and notification with Git output
With Git output, each device is a file in the repository, and a new commit is created only when the configuration changes. When groups are used, Oxidized by default opens a separate repository for each group; single_repo: true in the example keeps all groups in one repository, in subfolders named after the group. To look at the history:
cd /home/oxidized/.config/oxidized
git --git-dir=configs.git log --oneline -- campus/core-sw-01.example.local
git --git-dir=configs.git log -p -1 -- campus/core-sw-01.example.local # diff of the last change
# extract the version in a commit chosen from the log output to a file
git --git-dir=configs.git show 3f2a9c1:campus/core-sw-01.example.local > /tmp/core-sw-01-old.cfg
Oxidized’s hook mechanism is used for notifications. For example, to run a script when a backup fails, the following section is added to the configuration:
hooks:
backup_failure:
type: exec
events: [node_fail]
cmd: '/usr/local/bin/oxidized-notify.sh'
async: true
The script can take the device name from environment variables such as OX_NODE_NAME and send a notification to email or a chat channel. The post_store event is used for change notifications. The biggest risk is silent failure. When the account’s password changes or a device’s SSH access is closed, backups stop and nobody notices; on the day of an outage, it turns out the backup is three months old. That’s why you should regularly run a “last successful backup older than 48 hours” check from the REST API (/nodes.json) or the monitoring platform.
Link the automatically taken switch copies to the change and configuration management records that show who made the change and with which approval.
Oxidized and RANCID comparison
RANCID is a Perl- and Expect-based configuration backup tool that has been used for many years. The two tools solve the same problem; their operating models differ:
| Feature | Oxidized | RANCID |
|---|---|---|
| Language | Ruby | Perl + Expect |
| How it works | Continuously running service, multithreaded | rancid-run via cron |
| Version control | Git | CVS, Subversion or Git |
| Device source | CSV, SQL, HTTP API | Group-based router.db |
| Notification | Hooks (exec, email, etc.) | Diff email (built in) |
| API / web | REST API, oxidized-web | None (external tools) |
RANCID’s built-in diff email is a practical advantage in simple environments. Oxidized, thanks to its API, dynamic inventory sources and monitoring platform integration, is chosen more often in new installations. Don’t replace a working RANCID just because the other is newer. Consider moving when the need for inventory integration or an API arises.
Restoring from backup and verification steps
A backup is only as valuable as your ability to restore it. The steps are given for Cisco IOS/IOS XE; equivalent mechanisms are used on other platforms.
- Choose the right version: Identify the commit to roll back to from the Git history and extract the file.
- Fill in the secrets: If
remove_secretis enabled, the backup has no passwords, SNMP communities or keys. Add these values from the password vault. SNMPv3 users aren’t in the running-config anyway and must be defined separately; details are in the SNMPv3 configuration guide. - Transfer the file to the device: Copy it to flash with SCP (the device needs
ip scp server enable). - Preview the diff: Review the lines that will change with
show archive config differences flash:restore.cfg system:running-config. - Apply:
configure replace flash:restore.cfgis used to replace the entire configuration. If you’re working remotely, first set a safety timer withreload in 15, then cancel it withreload cancelafter success. Runningconfigure replacewith a file whose secrets haven’t been filled in also deletes the password and key lines on the device. Don’t skip step 2. - Verify: Compare interface states, adjacencies (
show cdp neighbors, routing adjacencies), VLAN and trunk states with outputs taken beforehand; then make it permanent withwrite memory.
When replacing hardware, also check that the new device’s software version is compatible with the backed-up configuration; on a model with different interface naming, the configuration can’t be applied one-to-one. Verify that backups are usable by restoring to a test device at least once a year. An untested backup isn’t a backup. For carrying out planned changes and rollback steps, the network change window planning article is a complement.
Setting up the configuration backup infrastructure together with the monitoring system falls within the network maintenance, monitoring and periodic checks service, and establishing device configuration standards within the switch, router and network device installation service.
Frequently asked questions
How often does Oxidized back up the configuration?
The backup interval is set in seconds with the interval parameter in the configuration; the value 3600 in the example takes a backup every hour. A new Git version is created only when the configuration actually changes, so frequent backups don’t grow the repository unnecessarily.
Is it safe to have passwords in the backup?
With remove_secret enabled, password, community and key lines are removed from the backup. Security improves, but on restore you need to add the secrets from the password vault. If the option is disabled, the Git repository must be protected at least as well as the devices themselves.
How do you define a read-only backup account on Cisco?
Assigning show running-config to a low privilege level produces incomplete output. A more robust method is to restrict the account to show and terminal commands only with TACACS+ command authorization.
If we use RANCID, should we move to Oxidized?
There is no need to replace a working RANCID installation just to modernize. Consider moving to Oxidized when the need for a dynamic inventory source, a REST API or monitoring platform integration arises.