MikroTik Solutions

Update of ”Yet Another Backup Script”
Login

Update of ”Yet Another Backup Script”

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview

Artifact ID: b5cec9712be263b2952c8a2d535404d158ba09e64a5e24e7e64adbabe71949fb
Page Name:Yet Another Backup Script
Date: 2022-03-29 19:36:58
Original User: tangent
Mimetype:text/x-markdown
Parent: 6947b350fe9724fecdc3d9d148d28d859d774cccee7d5c08fbd7277ff38b3353 (diff)
Next f748b4cb1ae417b32810aec67768ac9dd90cc8e0ab3dadd454b0e0d5f533e406
Content

There are many solutions for backing up RouterOS devices. This one is mine.

The script uses SSH to create and pull the backup files, which it then stores in Fossil, the version control system backing this very web site. That means you can get a web view of your RouterOS backups using the Files interface above, browse the history of your backups, get graphical diffs between versions of your backups, download Zip archives of your latest backups, and more.

Requirements

1. POSIX

The script was written and tested under Bash, though it should be portable to any other POSIX shell, needing only a change to the shebang line. It relies on other POSIX user environment stuff: cut, grep, tail, etc. It was written for and tested on a macOS system, but it should run just as well on Linux. It can be made to run on modern Windows systems via WSL, or on legacy Windows systems via Cygwin.

2. Command-Line SSH

The script was written for systems with the OpenSSH commands in mind. OpenSSH is installed by default on virtually every Linux and macOS system, and it's a first-party add-on from Microsoft for modern Windows versions. There may be third-party SSH command line implementations that use the same command formats this script relies on, but no attempt has been made to verify this.

3. Fossil

The Fossil version control system is designed with the same sensibility as RouterOS: self-contained, featureful, compact, and coherent. Fans of other VCSes are encouraged to give the script a try as-is before going in and hacking it to use your current VCS of choice. You might find that you enjoy Fossil for the same reasons you enjoy RouterOS.

4. The Script

You can download the latest version of the script here.

Alternately, you may wish to maintain a local fork of this repository so you can manage your local modifications without needing to overwrite them each time this script changes.

Preparation

  1. Set up SSH on each RouterOS box you want backed up, with host keys for automated login.

  2. Adjust the user-configurable variables at the top of the script:

    rdir is the Fossil repository directory. The default is "~/museum", being a place where one stores precious Fossils, but you are free to use anything else you like.

    repo is the name of the actual Fossil repository under $rdir where you want your backups stored. They're separate because you may have other Fossil repos under that directory; this script deals with just that one repo.

    bdir is where your local check-out of the backup repository lives. This can be anywhere else on the system. Wherever you want the current versions of your backups on your local machine, that's where you should point this script.

  3. Run the script by hand, passing the hostnames or IPs of the RouterOS boxes you want it to back up. It will do the initial backup on those boxes.

Usage

Once the backup environment is set up, you can simply run this script without arguments each time you want another backup taken. It iterates over the per-host subdirectory names it finds in $bdir and updates the repository with fresh backups.

The script prompts the user for a commit message for each backup after the initial one. The idea is that there is some documentable reason behind each backup. Examples:

This step does mean the script isn't suited to automated backups. If that bothers you, change the "fossil ci" line near the end, passing a canned or computed message via the -m flag.

You can pass one or more hostnames or IPs after the initial setup, in which case what you get is a limited backup of only the named hosts. I use this feature when carefully rolling out a new RouterOS version, since it gives me a diff between the old configuration and the upgraded one. I don't need complete backups taken of all the other RouterOS boxes; they're still current, so why burn the network time or clutter up their local file storage?

Compression

The script detects when router configurations haven't changed since the last backup, to prevent it from ballooning your backup repo with redundant copies. This also eases diffs between versions: any change to the textual /export file (*.rsc) between versions will be as substantial as the change made.

Atop this, Fossil has a two-level compression scheme that keeps the repository size small. For uncompressed data like these backups, it computes a binary delta between versions and stores only the changes. It then applies data compression to the diff. You can expect your initial repository size to be about half the size of the check-out directory contents, then to grow slowly from there, not surpassing the check-out directory size until several backups have been taken.

Diffs

The script produces a graphical diff as its last step before committing a backup to the repository, allowing you to abort a commit if you see changes you don't like. Common reasons are:

  1. You made a temporary change and forgot to revert it before committing.
  2. You made a change to one router and forgot to roll it out through the rest.
  3. Someone fat-fingered a change, and this feature caught the problem for you, allowing you to fix it.

Because there is no standard graphical diff program, the script's default behavior is to produce an HTML diff and open it in your web browser. Due to the nature of web browsers, Fossil doesn't block and wait for the browser to close before continuing, so after giving the browser time to open the temporary HTML output file, the script moves on to the "commit" step. Since that step does block, it means the script waits on the user to provide a commit message, per above.

The script will instead use another graphical diff tool if one is configured. Popular options for this are:

The method of configuring each is similar. Assuming they're in the command path, it's usually as simple as:

  $ fossil set gdiff-command p4merge

If you give that command from within this script's check-out directory — $bdir above — it affects only that one Fossil repository. If you want the setting to affect all Fossil repos on the machine, you can run it from anywhere, giving the "-g" flag to make it affect the global Fossil configuration.

Why Both Binary & Text Backups?

The script does both binary and text backups. (The latter are also called "/export" files or RSC files.) This is because text backups don't always include everything you want (e.g. certificates, passwords), but binary backups don't always restore properly. Text backups also permit diffs from one version to the next.

I highly recommend that you backup elements of your configuration that are only stored in the binary backup separately. For instance, you should keep a copy of all X.509 certificates via the RouterOS export function. NOTE: It is necessary to give the "export-passphrase" parameter if you want the resulting PEM file to include the private half of the key. Without a passphrase, RouterOS gives you only the public half.

You may store such files in the same Fossil repository that this script manages. For instance, you may place the downloaded PEM files under a "certs/" subdirectory. The script only treats top-level subdirectory names as RouterOS box hostnames if they contain a config.bin file, being the RouterOS binary configuration backup. Every other file or directory at the top level of the check-out — $bdir above — it ignores.

Using Another VCS

This script relies on no irreplaceable Fossil-specific features. The script makes several calls to Fossil, but each one is readily replaced with a call to any other version control system you prefer.

Git is the most popular VCS at the moment, but if you are one of its fans, do realize that Fossil provides everything Git does that matters for this application without bringing along all of its unnecessary complexities. You are unlikely to have a better justification than simple inertia for using it; none of Git's unique capabilities matter for this application. For instance, no sensible network engineer will publish sensitive router backups on GitHub, and even if one were so foolish as to do so, there is all but zero chance that anyone would fork that repository, make improvements, and push a PR for the repo. Git can be used on a private LAN instead, without involving the public Git hosting services, but what does that buy you over Fossil?

A better alternative would be Subversion, being small and coherent like Fossil, though not nearly so self-contained, featureful, or easy to administer.

For day-to-day use, Fossil is as easy to use as CVS while being nearly as powerful as Git.

Another alternative that would work here, which is more in line with the simplicity of Fossil without all of its features, is Mercurial.

Known Weaknesses

Periodic Cleanup

The current version of this script doesn't purge old binary backups under Files on each RouterOS box. It's possible to run this script enough times that you fill the flash, preventing operations like RouterOS version upgrades from working.

Modifying the script to automatically purge old backups wouldn't be difficult. I simply haven't bothered since I run this tool only at need, not periodically, so they don't build up to that point very fast. Even on a small RouterOS box like a hEX, with its piddlin' 16 MiB of flash, you can do 10 or so backups before you start causing this problem.

Automatic cleanup would matter if you ran this script periodically. Patches thoughtfully considered.

Change Detection

The script attempts to detect whether there are meaningful changes to the router configuration before doing a commit, to avoid ballooning the repo with unnecessary backups. It does this by looking at diffs of the text backup file, which means it can be fooled if you make a change affects only the binary backup file, such as updating a certificate.

(By contrast, adding a certificate is likely to require a change the text configuration as well, since you will need to reference that new certificate from somewhere, else why create it? Possible answer: you're using the RouterOS box as a CA, and you've added the certificate only to copy it to another box, in which case you're back in the soup.)

While the script can certainly detect when the binary backup file changes, RouterOS seems to include pointless tiny changes each time, so we cannot distinguish "it changed" from "it needs to be backed up."

I see multiple possible solutions:

  1. Rip the change detection out and commit a backup each time. Then, either rely on the user to know when a backup needs doing, or cope with the fact that the repo will grow on each backup even if there is no substantial change in the versions. A mitigating factor is that this scheme has strong compression to control the repo size growth.

  2. Add a "--force" flag to make it skip the change detection.

  3. Make it a practice to change something in the configuration that shows up in the /export output each time you make a change that only affects the backup. For instance, if you updated a certificate for one of your IPSec clients, you could document the change in the comment= on that client's /interface entry. Then both the binary and text backups show the same change.