1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
|
-
+
-
-
-
+
+
+
-
+
+
+
+
+
+
+
+
+
+
+
-
-
-
-
-
-
-
-
-
-
-
+
|
# Motivation
The [RouterOS `container.npk` feature](https://help.mikrotik.com/docs/display/ROS/Container) is highly useful, but it is a custom development written in-house by MikroTik, not a copy of Docker Engine or any of the other server-grade container engines.(^Podman, LXC/LXD, etc.) Because of the stringent resource constraints on the bulk of MikroTik’s devices, it is exceptionally small, thus unavoidably very thinly featured compared to its big-boy competition. If we can use installed size as a proxy for expected feature set size, we find:
* **Docker Engine**: 422 MiB(^Version 27.1.1, according to `dnf remove docker-ce…` after installing these packages [per the instructions](https://docs.docker.com/engine/install/rhel/#install-docker-engine).)
* **`containerd`+`nerdctl`**: 174 MiB(^Version 2.0.0-rc1 of `nerdctl` plus the `containerd` from the Docker Engine CE install according to `sudo dnf remove containerd` and `du -sh nerdctl`.)
* **`containerd`+`nerdctl`**: 174 MiB(^This is essentially Docker Engine minus the build tooling. The size is for version 2.0.0-rc1 of `nerdctl` plus the `containerd` from the Docker Engine CE install above, according to `sudo dnf remove containerd` and `du -sh nerdctl`.)
* **Podman**: 107 MiB(^Version 4.9.4 on EL9, according to `sudo dnf remove podman conmon crun`.)
* **`container.npk`**: _0.0626 MiB_(^Version 7.15.2, according to `/system/package/print`.)
And this is fine! RouterOS serves a particular market, and its developers are working within those constraints. The intent here is to provide a mapping between what people expect of a fully-featured container engine and what you actually get in RouterOS. Where it makes sense, I try to provide workarounds for missing features and guidance to alternative methods where RouterOS’s way merely *works* differently.
# <a id="global"></a>Global Limitations
Allow me to begin with the major limitations visible at a global level in the RouterOS `container.npk` feature, both to satisfy the **tl;dr** crowd and to set broad expectations for the rest of my readers. This super-minimal container implementation lacks:
* a local image cache(^One knock-on effect of this not covered above is that removing and reinstalling a container requires RouterOS to re-download the image, even when done back-to-back, even if you never start the container between and thereby cause it to make changes to the expanded image’s files.)
* a CoW/overlay file system(^This is not a verified fact, but an inference based on the observation that if RouterOS _did_ have this facility underlying its containers, I would expect to find equivalents to Docker’s `commit` and `diff` commands. This pairs with the lack of an image cache: no CoW means no need for a baseline source to compute deltas against.)
* image building(^RouterOS's container runtime is closer in nature to the `runc` command underlying `containerd` than to the broader Docker Engine product. An even closer match is the lightweight `crun` command at the heart of Podman, and even more so the elementary runner that ships with systemd, variously called either [`systemd-nspawn`][sdnsp] or [`systemd-container`][sdcnt], depending on the tastes of whoever is packaging it.)
* syscall restrictions
Lack of a management daemon(^`containerd` in modern setups, `dockerd` in old ones) is not in that list because a good bit of Docker’s competition also lacks this, on purpose. Between that and the other items on the list, the fairest comparison is not to fully-featured container *engines* like Docker and Podman but to the container *runner* at the heart of these systems:
* **runc**: 14 MiB(^This is the container runner underpinning `containerd`, thus also Docker, although it precedes it. Long before they created `containerd`, it underpinned `dockerd` instead. Because it is so primordial, a good many other container engines are also based on it.)
* **crun**: 0.5 MiB(^This Podman's alternative to `runc`, written in C to make it smaller. Early versions of Podman once relied on `runc`, and it can still be configured to use it, but the new default is to use the slimmer but feature-equivalent `crun`.)
* **systemd-nspawn**: 1.3 MiB(^This is the bare-bones OCI image runner built into systemd, with a feature set fairly close to that of `container.npk`. The size above is for version 252 of this program’s parent `systemd-container` package as shipped on EL9.)
One reason `container.npk` is far smaller than even the smallest of these runners is that the engines delegate much of what RouterOS lacks to the runner, so that even then it's an unbalanced comparison. Docker's `start`, `stop`, `kill`, `ps`, `pause`, and `rm` commands are handled way down at the runner level, not up at the top-level CLI.
With this grounding, let us dive into the details.
The single biggest area of difference between the likes of Docker and the RouterOS `container.npk` feature is how you create containers from OCI images. It combines Docker’s `create` and `load` commands as `/container/add`, the distinction expressed by whether you give it the `remote-image` or `file` option, respectively.
The single biggest area of difference between the likes of Docker and the RouterOS `container.npk` feature is how you create containers from OCI images. It combines Docker’s `create` and `load` commands under `/container/add`, the distinction expressed by whether you give it the `remote-image` or `file` option, respectively.
Given the size of the output from `docker create --help`, it should not be surprising that the bulk of that is either not available in RouterOS or exists in a very different form. Most of these limitations stem from [the list above](#global). For instance, the lack of any CPU usage limit features means there is no equivalent under `/container` for the several `docker create --cpu*` options. Rather than go into these options one by one, I’ll cover the ones where the answers cannot be gleaned through a careful reading of the rest of this article:
* **`--env`**: The equivalent is this RouterOS command pair:
/container/envs/add name=NAME …
/container/add envlist=NAME …
|
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
|
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
|
-
+
-
+
+
+
+
-
+
-
-
+
+
+
+
+
+
+
+
-
+
+
+
-
+
+
-
+
+
+
-
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
-
+
+
+
-
+
-
+
-
+
+
-
+
+
-
+
|
* **`--rm`**: No direct equivalent. There is a manual `/container/remove` command, but nothing like this option, which causes the container runtime to automatically remove the instantiated container after it exits. It’s just as well since this option is most often used when running _ad hoc_ containers made from a previously downloaded image; RouterOS’s lack of an image cache means you have to go out of your way to export a tarball of the image and upload it to the router, then use “`/container/add file=…`” if you want to avoid re-downloading the image from the repository on each relaunch.
That brings us to the related matter of…
[script]: https://help.mikrotik.com/docs/display/ROS/Scripting
# <a id="run"></a>There Is No "Run"
# <a id="run"></a>There Is No “Run”
RouterOS offers no shorthand command akin to `docker run` for creating and starting a container in a single step. Moreover, the lack of Linux-like interactive terminal handling means a simple command like…
RouterOS offers no shorthand command akin to `docker run` for creating and starting a container in a single step. Moreover, the lack of Linux-like interactive terminal handling — covered in the [next section](#terminal) — means a simple command like…
$ docker run --rm -it alpine:latest
…followed by…
sh-5.1# <do something inside the container>
sh-5.1# exit
…is most briefly expressed under RouterOS as…
…may end up expressed under RouterOS as…
> /container
> add remote-image=alpine:latest veth=veth1 entrypoint=sleep cmd=3600
… wait for it to download and extract …
… poll with "print" commands to get the container ID …
… wait for it to download and extract by polling …
> print
… nope, not ready, wait some more …
> print
… nope, wait some more …
> print
… oh, good, now we have the container ID …
> start 0
… wait for it to launch …
> shell 0
sh-5.1# <do something inside the container>
sh-5.1# exit
> stop 0
> remove 0
Whew! 😅
The “sleep 3600” hack is necessary because `container.npk` lacks any notion of interactive mode. In order to keep a container of this type from starting, doing a whole lot of _nothing_, and then stopping,(^…but not cleaning up after itself as with `--rm`, mind!) you have to give it some type of busy-work to keep it alive. This is a common one, but it only lasts an hour. More complicated methods are difficult to pull off due to the lack of host-side Bourne shell command parsing in RouterOS; the `cmd` option gets passed into the container as a single string, but you only have two to play with, `entrypoint` and `cmd`, so how do you say something like the following?
docker run alpine:latest 'while true ; do sleep 5 ; done' &
The answer is that if you want a RouterOS container to do anything tricky like that, you need to write your own `ENTRYPOINT` script.
With all of that to ground us, the rest is far simpler to discuss.
Although RouterOS proper is built atop Linux, and it provides a feature-rich CLI, it is nothing like a Linux command shell, but I am not speaking of skin-level command syntax differences here. The differences go far deeper. When you SSH into a RouterOS box, you’re missing out on a meaningful distinction between stdout and stderr, and the kernel’s underlying termios/pty subsystem is hidden from you. These lacks translate directly into limitations in the ability of `container.npk` to mimic the experience of using Docker at the command line.
Although RouterOS proper is built atop Linux, and it provides a feature-rich CLI, it is nothing like a Linux command shell, and I am not speaking of skin-level command syntax differences here; the differences go far deeper.
When you SSH into a RouterOS box, you’re missing out on a meaningful distinction between stdout and stderr, and the kernel’s underlying termios/pty subsystem is hidden from you. These lacks translate directly into limitations in the ability of `container.npk` to mimic the experience of using Docker at the command line.
One of the core RouterOS design principles is being able to run headlessly for long periods, with the administrator connecting to their virtual terminal via WinBox, WebFig, or SSH briefly, only long enough to reconfigure something before logging back out. The RouterOS CLI never was meant to provide the sort of rich terminal experience you need when you work in a Linux terminal all day, every day.
One of the core RouterOS design principles is being able to run headlessly for long periods, with the administrator connecting to their virtual terminal via WinBox, WebFig, or SSH briefly, only long enough to accomplish some network admin task before logging back out. The RouterOS CLI never was meant to provide the sort of rich terminal experience you need when you work in a Linux terminal all day, every day.
The thing is, Docker _was_ designed around this sensibility.
It is for this inherent reason that `container.npk` cannot provide equivalents of Docker’s `attach` command, nor its “`docker run --attach flag`”, nor the common “`docker run -it`” option pair. The closest it comes to all this is its [`shell`](#shell) command implementation, which can connect your local terminal to a true remote Linux terminal subsystem. Alas, that isn’t a close “`run -it`” alternative because you’re left typing commands at this remote shell, not at the container’s `ENTRYPOINT` process. Even then, it doesn’t always work since a good many containers lack a `/bin/sh` program inside the container in the first place, on purpose, typically to reduce the container’s attack surface.(^Indeed, all of [my public containers](https://hub.docker.com/repositories/tangentsoft) elide the shell for this reason.)
It is for this inherent reason that `container.npk` cannot provide equivalents of Docker’s `attach` command, nor its “`docker run --attach`” flag, nor the common “`docker run -it`” option pair. The closest it comes to all this is its [`shell`](#shell) command implementation, which can connect your local terminal to a true remote Linux terminal subsystem. Alas, that isn’t a close “`run -it`” alternative because you’re left typing commands at this remote shell, not at the container’s `ENTRYPOINT` process. Even then, it doesn’t always work since a good many containers lack a `/bin/sh` program inside the container in the first place, on purpose, typically to reduce the container’s attack surface.(^Indeed, all of [my public containers](https://hub.docker.com/repositories/tangentsoft) elide the shell for this reason.)
Although Docker logging is tied into this same Linux terminal I/O design, we cannot blame the lack of an equivalent to “`docker logs`” on the RouterOS design principles in this same manner. The cause here is different, stemming first from the fact that RouterOS boxes try to keep logging to a minimum by default, whereas Docker logs everything the container says, without restriction. RouterOS logs to memory by default to avoid burning out the flash, and it ignores all messages issued under "topics" other than the four preconfigured by default, which does not include the "container" topic you get access to by installing `container.npk`. To prevent log messages from being sent straight to the bit bucket, you must say:
Although Docker logging is tied into this same Linux terminal I/O design, we cannot blame the lack of an equivalent to “`docker logs`” on the RouterOS design principles in this same manner. The cause here is different, stemming first from the fact that RouterOS boxes try to keep logging to a minimum by default, whereas Docker logs everything the container says, without restriction. RouterOS logs to memory by default to avoid burning out the flash, and it ignores all messages issued under “topics” other than the four preconfigured by default, which does not include the “container” topic you get access to by installing `container.npk`. To prevent log messages from being sent straight to the bit bucket, you must say:
/container/{add,set} … logging=yes
/system/logging add topics=container action=…
Having done so, we have a new limitation to contend with: RouterOS logging isn’t as powerful as the Docker “`logs`” command, which by default works as if you asked it, “Tell me what this particular container logged since the last time I asked.” RouterOS logging, on the other hand, mixes everything together in real time, requiring you to dig through the history manually.
Having done so, we have a new limitation to contend with: RouterOS logging isn't as powerful as the Docker "`logs`" command.(^Nor, for that matter, `podman logs`, which ties into systemd's controversial unified "journal" subsystem, a design choice that paid off handsomely when it comes to pulling up per-container logs.) By default, it works as if you asked it, "Tell me what this particular container logged since the last time I asked." RouterOS logging, on the other hand, mixes everything together in real time, requiring you to dig through the history manually.
For lack of any better organization principle, I’ve chosen to cover the remaining `docker` CLI commands in alphabetical order. I skip over short aliases like `docker rmi` for `docker image rm` in order to cover things only once, and I don’t repeat any of the `create`/`load`/`run` discussion [above](#create). Because Podman cloned the Docker CLI, this matches fairly well with it, except that I do not currently go into any of its pure extensions, such as its eponymous `pod` command.
For lack of any better organization principle, I’ve chosen to cover the remaining `docker` CLI commands in alphabetical order. I skip over short aliases like `docker rmi` for `docker image rm` in order to cover things only once, and I don’t repeat any of the `create`/`load`/`run` discussion [above](#create). Because Podman cloned the Docker CLI, this ordering matches up fairly well with its top-level command structure as well, the primary exception being that I do not currently go into any of Podman’s pure extensions, ones such as its eponymous `pod` command.
## <a id="build"></a>`build`/`buildx`
RouterOS provides a bare-bones container runtime only, not any of the image building toolchain.
|