D 2022-06-30T11:17:39.227
L Musings\son\sDocker
N text/x-markdown
P aff34ebed72e36711400a7828c8b13c12e8800d0f00db892f854beff5e165943
U tangent
W 7951
## Motivation
[MikroTik added Docker support to RouterOS in 7.4beta4](https://help.mikrotik.com/docs/display/ROS/Container), which allows you to run third-party background services directly on the router. While there is likely to be a small explosion of services that work quite well in this environment, there are a number that will not. I thought it was interesting to think through one such problem, that of [a `fail2ban` configuration for monitoring SSH login failures](/wiki?name=Using+fail2ban+with+Remote+syslog). In principle, that *could* indeed run on some RouterOS devices, but as it turns out, there are a number of reasons it's impractical, which I find instructive.
For some MikroTik routers, there's a single fatal limitation that takes them out of the running, and for others, an unhappy concatenation of weak workarounds yields the same end result.
## The Many, Many Problems with Docker on RouterOS
### 1. CPU Compatibliity
The [Docker cross-compilation tool chain](https://docs.docker.com/buildx/working-with-buildx/) doesn't support any MIPS CPU type, ruling out a large fraction of the RouterOS devices from the outset. Docker also doesn't support obsolete platforms like 32-bit PowerPC and the TILE CPU architecture, ruling out even more MikroTik products. For our purposes, only routers with ARM or x86 CPUs are in the running from the start.
### 2. Third-Party Requirements
If you're able get past problem #1, you then have to build a Docker container — or find one pre-built — that meets all the run-time requirements of your background service. The ideal case is a single statically-linked executable, but the vast majority of useful software has substantial requirements that one typically doesn't even think about when running the software on laptops, desktops, and big server computers. It's either all just *there*, ready to be shared, or you've got so much storage that it doesn't matter if you cart some third-party dependencies around.
To build a Docker container, you have to provide all of that inside the container, somehow. If it duplicates resources provided by the host, too bad: Docker purposefully isolates the container from the host, so there is no more sharing of programs, libraries, and so forth than you get between any two standalone computers.
To take our example of `fail2ban`, it's historically a fairly portable tool, but any given version runs on only a subset of Python versions. Roughly, those contemporaneous with the release. The practical path out of this trap is to start with an existing Linux distribution that has a version of `fail2ban` ported to it, which then brings us to the next problem.
### 3. Resource Usage
Python is not terribly resource efficient. A Linux distro and all of the dependencies needed to run `fail2ban` may exceed the persistent storage space and/or free RAM available on your router.
For instance, [this `fail2ban` image](https://hub.docker.com/r/crazymax/fail2ban) hosted on DockerHub will take about half the space on the broad class of MikroTik routers with 128 MiB of storage space. This includes all current members of [the CCR2004 line](https://mikrotik.com/products/group/ethernet-routers?filter&s=c&search=ccr2004) and the [RB3011](https://mikrotik.com/product/RB3011UiAS-RM). It even includes otherwise high-end products like the [CCR2116](https://mikrotik.com/product/ccr2116_12g_4splus) and [CCR2216](https://mikrotik.com/product/ccr2216_1g_12xs_2xq). If we step across into MikroTik's SOHO WiFi router range, we find that most of them don't have even 128 MiB of storage space. The only two that do are the [hAP ac³](https://mikrotik.com/product/hap_ac3) and the [Audience](https://mikrotik.com/product/audience).
With `fail2ban` and all of its dependencies taking up over half your storage space, you might not have enough left over to update RouterOS out on the host system!
There are only a few RouterOS models with more than 128 MiB of storage space. In MikroTik's current router product line, only the [RB4011](https://mikrotik.com/product/rb4011igs_rm), [its wireless cousin](https://mikrotik.com/product/rb4011igs_5hacq2hnd_in), and the [RB5009](https://mikrotik.com/product/rb5009ug_s_in) are big enough to comfortably host such a container on the SoC's built-in flash. Add to that the "Dude edition" of the [RB1100](https://mikrotik.com/product/rb1100ahx4), which comes with an m.2 SSD; that's a pretty premium to pay just to run `fail2ban`.
As we will see below, this problem has a way of expanding by surprise: for the `fail2ban` case, we also need a copy of `rsyslog` and an SSH client inside the container. This is true even though the host (RouterOS) has a logging subsystem and an SSH client built-in. We can't share them with the processes running inside the container; we must duplicate them.
### 4. Run-Time Storage Requirements
The ideal use case for Docker on RouterOS is when you have a background service that either requires no additional storage space beyond that covered in the prior section, or at least has a static requirement, such as rarely-changing configuration data. A good example is a DNS server where the local zone files rarely change, and the remote ones can be cached in RAM.
In all other cases, you must also account for the growing storage space for your container.
To return to the `fail2ban` example, you also have to account for the space the logs require, since you still have to redirect log data into the container, else `fail2ban` will have no input to crawl through. That requirement is likely to exclude all of the 128 MiB routers all by itself. For the routers left in the running after that bottle-necking requirement, you're then back to problem #2 at the top of [the `rsyslog` article](/wiki?name=Remote%20Log%20Server): storing logs on flash storage is likely to materially shorten the service life of the device. Only the swappable m.2 SSD in the RB1100 is immune from this.
If MikroTik ever releases my dream device — an ARM-based hEX with a microSD slot and an SFP+ port — it may be a suitable Docker container host, provided I'm willing to replace the SD card from time to time, as each one wears out.
### 5. Host Isolation
If you can fix all of the above, you're left with the fact that Docker purposefully isolates the container from its host.
Take the `fail2ban` example one last time: in order for it to reach out and issue "`/ip/firewall/filter`" commands on the RouterOS host, you'll have to burn even more space in the container by installing an SSH client and configuring it to connect out to the host.
This will work, but it calls into question the logic behind the initial wish to run `fail2ban` on the router itself: surely you were hoping to avoid all this `rsyslog` and SSH stuff? Sorry, it can't be avoided.
## Conclusion
It is my opinion that all of the above forces you to the bare-metal x86 version of RouterOS or to [CHR](https://help.mikrotik.com/docs/pages/viewpage.action?pageId=18350234) for any service requiring substantial amounts of storage. (That, or possibly the RB1100 Dude edition.)
Sites that can justify the hardware expense for that likely have an underutilized Linux box sitting around, or at least a VM hypervisor or Docker host that can much more easily host a small `fail2ban` host than the router.
Services like `fail2ban` were written with the assumption of a big server-class CPU, plenty of RAM, and ludicrous amounts of storage to host logs with. Don't fight the design.
One last thing: I do not point all of this out to say that RouterOS's support of Docker is terrible. Indeed, I think it's wonderful. I'm simply pointing out that it has serious limitations that are worth taking into account when planning what services run, where. For some, Docker on RouterOS will be just the thing. For others, the service is likely better run elsewhere.
Z 63515df15c6e170f241d435eaa88bcb2