Many hyperlinks are disabled.
Use anonymous login
to enable hyperlinks.
Artifact ID: | d01e032b0c3c0cfbcb8b5acc53e7c787383c48ee1ec13d5e8f9703f89610b290 |
---|---|
Page Name: | Yet Another Backup Script |
Date: | 2022-03-29 21:53:41 |
Original User: | tangent |
Mimetype: | text/x-markdown |
Parent: | f748b4cb1ae417b32810aec67768ac9dd90cc8e0ab3dadd454b0e0d5f533e406 (diff) |
Next | a915cfb6bc10ce66e251ad30698a150239c2d33f9d598b5b472d569f5fd63705 |
There are many solutions for backing up RouterOS devices. This one is mine.
The script uses SSH to create and pull the backup files, which it then stores in Fossil, the version control system backing this very web site. That means you can get a web view of your RouterOS backups using the Files interface above, browse the history of your backups, get graphical diffs between versions of your backups, download Zip archives of your latest backups, and more.
Requirements
1. POSIX
The script was written and tested under Bash, though it should be portable to any other POSIX shell, needing only a change to the shebang line. It relies on other POSIX user environment stuff: cut
, grep
, tail
, etc. It was written for and tested on a macOS system, but it should run just as well on Linux. It can be made to run on modern Windows systems via WSL, or on legacy Windows systems via Cygwin.
2. Command-Line SSH
The script was written for systems with the OpenSSH commands in mind. OpenSSH is installed by default on virtually every Linux and macOS system, and it's a first-party add-on from Microsoft for modern Windows versions. There may be third-party SSH command line implementations that use the same command formats this script relies on, but no attempt has been made to verify this.
3. Fossil
The Fossil version control system is designed with the same sensibility as RouterOS: self-contained, featureful, compact, and coherent. Fans of other VCSes are encouraged to give the script a try as-is before going in and hacking it to use your current VCS of choice. You might find that you enjoy Fossil for the same reasons you enjoy RouterOS.
4. The Script
You can download the latest version of the script here.
Alternately, you may wish to maintain a local fork of this repository so you can manage your local modifications without needing to overwrite them each time this script changes.
Preparation
Set up SSH on each RouterOS box you want backed up, with host keys for automated login.
Adjust the user-configurable variables at the top of the script:
rdir is the Fossil repository directory. The default is "
~/museum
", being a place where one stores precious Fossils, but you are free to use anything else you like.repo is the name of the actual Fossil repository under
$rdir
where you want your backups stored. They're separate because you may have other Fossil repos under that directory; this script deals with just that one repo.bdir is where your local check-out of the backup repository lives. This can be anywhere else on the system. Wherever you want the current versions of your backups on your local machine, that's where you should point this script.
Run the script by hand, passing the hostnames or IPs of the RouterOS boxes you want it to back up. It will do the initial backup on those boxes.
Usage
Once the backup environment is set up, you can simply run this script without arguments each time you want another backup taken. It iterates over the per-host subdirectory names it finds in $bdir
and updates the repository with fresh backups.
The script prompts the user for a commit message for each backup after the initial one. The idea is that there is some documentable reason behind each backup. Examples:
- upgraded to RouterOS 7.1.4
- fixed an OSPF configuration bug
- enabled WireGuard on the border gateway
- updated the firewall rules to block
$VULN_DU_JOUR
This step does mean the script isn't suited to automated backups. If that bothers you, change the "fossil ci
" line near the end, passing a canned or computed message via the -m
flag.
You can pass one or more hostnames or IPs after the initial setup, in which case what you get is a limited backup of only the named hosts. I use this feature when carefully rolling out a new RouterOS version, since it gives me a diff between the old configuration and the upgraded one. I don't need complete backups taken of all the other RouterOS boxes; they're still current, so why burn the network time or clutter up their local file storage?
Compression
The script detects when router configurations haven't changed since the last backup, to prevent it from ballooning your backup repo with redundant copies. This also eases diffs between versions: any change to the textual /export
file (*.rsc
) between versions will be as substantial as the change made.
Atop this, Fossil has a two-level compression scheme that keeps the repository size small. For uncompressed data like these backups, it computes a binary delta between versions and stores only the changes. It then applies data compression to the diff. You can expect your initial repository size to be about half the size of the check-out directory contents, then to grow slowly from there, not surpassing the check-out directory size until several backups have been taken.
Diffs
The script produces a graphical diff as its last step before committing a backup to the repository, allowing you to abort a commit if you see changes you don't like. Common reasons are:
- You made a temporary change and forgot to revert it before committing.
- You made a change to one router and forgot to roll it out through the rest.
- Someone fat-fingered a change, and this feature caught the problem for you, allowing you to fix it.
Because there is no standard graphical diff program, the script's default behavior is to produce an HTML diff and open it in your web browser. Due to the nature of web browsers, Fossil doesn't block and wait for the browser to close before continuing, so after giving the browser time to open the temporary HTML output file, the script moves on to the "commit" step. Since that step does block, it means the script waits on the user to provide a commit message, per above.
The script will instead use another graphical diff tool if one is configured. Popular options for this are:
The method of configuring each is similar. Assuming they're in the command path, it's usually as simple as:
$ fossil set gdiff-command p4merge
If you give that command from within this script's check-out directory — $bdir
above — it affects only that one Fossil repository. If you want the setting to affect all Fossil repos on the machine, you can run it from anywhere, giving the "-g
" flag to make it affect the global Fossil configuration.
Why Both Binary & Text Backups?
The script uses both the RouterOS /system/backup
feature and the /export
command, yielding binary and text backups, respectively.
RouterOS binary backups include the device's full configuration, but for various reasons they don't always restore properly. The most common reason is that you're trying to restore onto a different hardware platform. That's a hard truth to learn, since it means you can't just buy the best-available replacement for a dead MikroTik device if all you have is a binary backup. I've also seen restoration failure between RouterOS versions: a binary backup made on an older version often won't restore properly atop a newer version of the OS. I've even seen this between two closely-spaced RouterOS beta versions!
Text backups solve both of these problems. The /export
format is fairly compatible between versions, and where there are differences, you can edit the commands to make them work under the new OS version. The same is true among different hardware: if the old box said something about "sfp1" but it's now called "sfpplus1" on the new box, you can edit the commands involved to make it work. Even if you have to rebuild the configuration piecemeal, with a text backup, you at least have a guide to what you did on the prior device and can work through it step-by-step.
Text backups are also version-control friendly. They allow for diffs from one backup to the next, helping the administrator remember what they did since the last backup, understand the changes, and make an informed decision about choosing to commit them as the new version. If a given change turns out to be bad, the version history gives you the option to go back and see what changed since the last time it was known to work, and even roll back to that version if necessary.
Unfortunately, text backups made by RouterOS don't include everything. Thus binary and text.
Supplementing the Text Backup
Because both of RouterOS's backup methods have problems, I recommend that instead of relying on the binary backup, you consider it a last-ditch option. Instead, supplement the text backup with the elements that are normally only backed up via the binary method, stored separately. Then if it comes time to do a restore, you can try the binary method, and if that doesn't work, you have the option to rebuild the configuration with a combination of the text backup and the individual elements you had the foresight to store separately.
Before we get down into those details, we come to the question, "Where should I keep my supplementary backups?" Why, in the Fossil repository managed by this script, of course! The script only treats the top-level subdirectory names it finds in the check-out directory as RouterOS box hostnames if they contain a config.bin
file, being the RouterOS binary configuration backup. Every other file or directory at the top level of the check-out — $bdir
above — it ignores. That gives you the freedom to store any type of file you want in the repository this script manages without interfering with its operation.
Fossil also includes a wiki and many other features you might find helpful. For instance, you might have a preferred method for creating certificates, which you document in a wiki article. Since it's stored in the same Fossil repo as your backups, you could add instructions for loading certificate backups.
This combination of features allows the repo to serve as an all-round disaster recovery kit.
Things to consider including in the kit:
X.509 Certificates
Export a copy of all X.509 certificates via the RouterOS export function, then download them somewhere safe. I suggest the "certs/
" subdirectory of the backup repository.
NOTE: It is absolutely necessary to give the "export-passphrase" parameter to this command if you want the resulting PEM file to include the private half of the key. Without a passphrase, RouterOS gives you only the public half.
SSH Keys
For user keys, you normally upload these keys from some other location (e.g. ~/.ssh/id_*.pub
) which should be backed up by other means. However, storing them in this script's backup repo isn't a terrible idea.
More important than this are the SSH host keys, without which your SSH clients will begin complaining that someone's trying to spoof the connection after you restore from a text backup. For instance, you may have run "/ip ssh set strong-crypto=yes
" (as I recommend) and then ran the subsequently-necessary "/ip ssh regenerate-host-keys
" command. You need to export those regenerated SSH host keys via "/ip ssh export-host-key
" and download them somewhere save. Storing them in this script's backup repo is a perfectly sensible plan.
Backing up the Backup
Fossil repositories are designed for long-term readability, not as an encrypted data store. That means that anyone that can read the repo file this script manages can pull the sensitive information out of your backups. To the extent that you go beyond what this script provides, all of that supplemental information will be in plaintext as well.
While there are ways to get Fossil to encrypt a live Fossil repo file, I think it's better to just make sure it's stored somewhere safe, such as under the encrypted home directory of the one user that needs to maintain the backups.
If you want belt-and-suspenders, I recommend one of two methods:
SSH
Set up autosync with another machine over SSH. Using the variable names from the backup script:
$ scp $repo otherbox:museum/
$ cd $bdir
$ fossil sync ssh://otherbox/museum/routeros-backups.fossil
The first time you do that, the only thing that will happen is that the sync link will be set up, but from that point on, each time the script commits a change to the local repository, Fossil will autosync the change to otherbox
over SSH for you.
Repository Backup
You could instead back up the entire RouterOS backup repository with some other method. I highly recommend studying the relevant Fossil documentation page on this. There's a lot of subtlety you might neglect to consider otherwise.
Down at the bottom of that doc is a method for encrypted off-site backups, which pairs well with this topic.
Using Another VCS
This script relies on no irreplaceable Fossil-specific features. The script makes several calls to Fossil, but each one is readily replaced with a call to any other version control system you prefer.
Git is the most popular VCS at the moment, but if you are one of its fans, do realize that Fossil provides everything Git does that matters for this application without bringing along all of its unnecessary complexities. You are unlikely to have a better justification than simple inertia for using it; none of Git's unique capabilities matter for this application. For instance, no sensible network engineer will publish sensitive router backups on GitHub, and even if one were so foolish as to do so, there is all but zero chance that anyone would fork that repository, make improvements, and push a PR for the repo. Git can be used on a private LAN instead, without involving the public Git hosting services, but what does that buy you over Fossil?
A better alternative would be Subversion, being small and coherent like Fossil, though not nearly so self-contained, featureful, or easy to administer.
For day-to-day use, Fossil is as easy to use as CVS while being nearly as powerful as Git.
Another alternative that would work here, which is more in line with the simplicity of Fossil without all of its features, is Mercurial.
Known Weaknesses
Periodic Cleanup
The current version of this script doesn't purge old binary backups under Files on each RouterOS box. It's possible to run this script enough times that you fill the flash, preventing operations like RouterOS version upgrades from working.
Modifying the script to automatically purge old backups wouldn't be difficult. I simply haven't bothered since I run this tool only at need, not periodically, so they don't build up to that point very fast. Even on a small RouterOS box like a hEX, with its piddlin' 16 MiB of flash, you can do 10 or so backups before you start causing this problem.
Automatic cleanup would matter if you ran this script periodically. Patches thoughtfully considered.
Change Detection
The script attempts to detect whether there are meaningful changes to the router configuration before doing a commit, to avoid ballooning the repo with unnecessary backups. It does this by looking at diffs of the text backup file, which means it can be fooled if you make a change affects only the binary backup file, such as updating a certificate.
(By contrast, adding a certificate is likely to require a change the text configuration as well, since you will need to reference that new certificate from somewhere, else why create it? Possible answer: you're using the RouterOS box as a CA, and you've added the certificate only to copy it to another box, in which case you're back in the soup.)
While the script can certainly detect when the binary backup file changes, RouterOS seems to include pointless tiny changes each time, so we cannot distinguish "it changed" from "it needs to be backed up."
I see multiple possible solutions:
Rip the change detection out and commit a backup each time. Then, either rely on the user to know when a backup needs doing, or cope with the fact that the repo will grow on each backup even if there is no substantial change in the versions. A mitigating factor is that this scheme has strong compression to control the repo size growth.
Add a "
--force
" flag to make it skip the change detection.Make it a practice to change something in the configuration that shows up in the
/export
output each time you make a change that only affects the backup. For instance, if you updated a certificate for one of your IPSec clients, you could document the change in thecomment=
on that client's/interface
entry. Then both the binary and text backups show the same change.