MikroTik Solutions

Changes To Container Limitations
Login

Changes To Container Limitations

Changes to "Container Limitations" between 2024-07-25 21:36:55 and 2024-07-25 22:41:11

1
2
3
4
5
6

7
8
9
10
11
12
13
14
15
16
17
18
19
20


21

22
23
24
25
26
27
28
29
30

31
32
33
34










35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54

55
56
57
58
59
60
61
1
2
3
4
5

6
7
8
9
10
11
12
13
14
15
16



17
18
19
20
21
22
23
24
25
26
27
28

29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50










51
52
53

54
55
56
57
58
59
60
61





-
+










-
-
-

+
+

+







-

+




+
+
+
+
+
+
+
+
+
+






-
-
-
-
-
-
-
-
-
-



-
+







# Motivation

The [RouterOS `container.npk` feature](https://help.mikrotik.com/docs/display/ROS/Container) is highly useful, but it is a custom development written in-house by MikroTik, not a copy of Docker Engine or any of the other server-grade container engines.(^Podman, LXC/LXD, etc.) Because of the stringent resource constraints on the bulk of MikroTik’s devices, it is exceptionally small, thus unavoidably very thinly featured compared to its big-boy competition. If we can use installed size as a proxy for expected feature set size, we find:

* **Docker Engine**: 422 MiB(^Version 27.1.1, according to `dnf remove docker-ce…` after installing these packages [per the instructions](https://docs.docker.com/engine/install/rhel/#install-docker-engine).)
* **`containerd`+`nerdctl`**: 174 MiB(^Version 2.0.0-rc1 of `nerdctl` plus the `containerd` from the Docker Engine CE install according to `sudo dnf remove containerd` and `du -sh nerdctl`.)
* **`containerd`+`nerdctl`**: 174 MiB(^This is essentially Docker Engine minus the build tooling. The size is for version 2.0.0-rc1 of `nerdctl` plus the `containerd` from the Docker Engine CE install above, according to `sudo dnf remove containerd` and `du -sh nerdctl`.)
* **Podman**: 107 MiB(^Version 4.9.4 on EL9, according to `sudo dnf remove podman conmon crun`.)
* **`container.npk`**: _0.0626 MiB_(^Version 7.15.2, according to `/system/package/print`.)

And this is fine! RouterOS serves a particular market, and its developers are working within those constraints. The intent here is to provide a mapping between what people expect of a fully-featured container engine and what you actually get in RouterOS. Where it makes sense, I try to provide workarounds for missing features and guidance to alternative methods where RouterOS’s way merely *works* differently.


# <a id="global"></a>Global Limitations

Allow me to begin with the major limitations visible at a global level in the RouterOS `container.npk` feature, both to satisfy the **tl;dr** crowd and to set broad expectations for the rest of my readers. This super-minimal container implementation lacks:

*   a local image cache(^One knock-on effect of this not covered above is that removing and reinstalling a container requires RouterOS to re-download the image, even when done back-to-back, even if you never start the container between and thereby cause it to make changes to the expanded image’s files.)
*   a CoW/overlay file system(^This is not a verified fact, but an inference based on the observation that if RouterOS _did_ have this facility underlying its containers, I would expect to find equivalents to Docker’s `commit` and `diff` commands. This pairs with the lack of an image cache: no CoW means no need for a baseline source to compute deltas against.)
*   image building(^RouterOS's container runtime is closer in nature to the `runc` command underlying `containerd` than to the broader Docker Engine product. An even closer match is the lightweight `crun` command at the heart of Podman, and even more so the elementary runner that ships with systemd, variously called either [`systemd-nspawn`][sdnsp] or [`systemd-container`][sdcnt], depending on the tastes of whoever is packaging it.)
*   orchestration
*   image building
*   a local image cache(^One knock-on effect of this not covered above is that removing and reinstalling a container requires RouterOS to re-download the image, even when done back-to-back, even if you never start the container between and thereby cause it to make changes to the expanded image’s files.)
*   JSON and REST APIs
*   a CoW/overlay file system(^This is not a verified fact, but an inference based on the observation that if RouterOS _did_ have this facility underlying its containers, I would expect to find equivalents to Docker’s `commit` and `diff` commands. This pairs with the lack of an image cache: no CoW means no need for a baseline source to compute deltas against.)
*   per-container limit controls:(^The only configurable resource limit is on maximum RAM usage, and it’s global, not settable on a per-container basis.)
    *   FD count
    *   PID limit
    *   CPU usage
    *   storage IOPS
    *   `/dev/shm` size limit
    *   terminal/logging bps
    *   syscall restrictions
    *   [capability][caps] restrictions
    *   syscall blocking
    *   [rlimit]
*   hardware pass-thru:
    *   USB device entries under `/dev` are on the wish list, but not currently available.(^Not unless RouterOS itself sees the USB device, as with storage media, which you can bind-mount into the container with “`/container/add mounts=…`”.)
    *   There is no GPU support, not even for bare-metal x86 installs.

Lack of a management daemon(^`containerd` in modern setups, `dockerd` in old ones) is not in that list because a good bit of Docker’s competition also lacks this, on purpose. Between that and the other items on the list, the fairest comparison is not to fully-featured container *engines* like Docker and Podman but to the container *runner* at their heart:

* **runc**: 14 MiB(^This is the container runner underpinning `containerd`, thus also Docker, although it precedes it. Long before they created `containerd`, it underpinned `dockerd` instead. Because it is so primordial, a good many other container engines are also based on it.)
* **crun**: 0.5 MiB(^This Podman’s alternative to `runc`, written in C to make it smaller. Early versions of Podman once relied on `runc`, and it can still be configured to use it, but the new default is to use the slimmer but feature-equivalent `crun`.)
* **systemd-nspawn**: 1.3 MiB(^[This][sdnsp] is the bare-bones OCI image runner built into systemd, with a feature set fairly close to that of `container.npk`. The size above is for version 252 of this program’s parent [`systemd-container`][sdcnt] package as shipped on EL9.)

One reason `container.npk` is far smaller than even the smallest of these runners is that the engines delegate much of what RouterOS lacks to the runner, so that even then it’s an unbalanced comparison. The [`kill`](#kill), [`ps`](#ps), and [`pause`](#pause) commands missing from `container.npk` are provided in Docker Engine way down at the `runc` level, not up at the top-level CLI.

With this grounding, let us dive into the details.

[caps]:   https://www.man7.org/linux/man-pages/man7/capabilities.7.html
[rlimit]: https://www.man7.org/linux/man-pages/man2/getrlimit.2.html
[sdcnt]: https://packages.fedoraproject.org/pkgs/systemd/systemd-container/
[sdnsp]: https://wiki.archlinux.org/title/Systemd-nspawn

Lack of a management daemon(^`containerd` in modern setups, `dockerd` in old ones) is not in that list because a good bit of Docker’s competition also lacks this, on purpose. Between that and the other items on the list, the fairest comparison is not to fully-featured container *engines* like Docker and Podman but to the container *runner* at the heart of these systems:

* **runc**: 14 MiB(^This is the container runner underpinning `containerd`, thus also Docker, although it precedes it. Long before they created `containerd`, it underpinned `dockerd` instead. Because it is so primordial, a good many other container engines are also based on it.)
* **crun**: 0.5 MiB(^This Podman's alternative to `runc`, written in C to make it smaller. Early versions of Podman once relied on `runc`, and it can still be configured to use it, but the new default is to use the slimmer but feature-equivalent `crun`.)
* **systemd-nspawn**: 1.3 MiB(^This is the bare-bones OCI image runner built into systemd, with a feature set fairly close to that of `container.npk`. The size above is for version 252 of this program’s parent `systemd-container` package as shipped on EL9.)

One reason `container.npk` is far smaller than even the smallest of these runners is that the engines delegate much of what RouterOS lacks to the runner, so that even then it's an unbalanced comparison. Docker's `start`, `stop`, `kill`, `ps`, `pause`, and `rm` commands are handled way down at the runner level, not up at the top-level CLI.

With this grounding, let us dive into the details.


## <a id="create" name="load"></a>Container Creation

The single biggest area of difference between the likes of Docker and the RouterOS `container.npk` feature is how you create containers from OCI images. It combines Docker’s `create` and `load` commands as `/container/add`, the distinction expressed by whether you give it the `remote-image` or `file` option, respectively.
The single biggest area of difference between the likes of Docker and the RouterOS `container.npk` feature is how you create containers from OCI images. It combines Docker’s `create` and `load` commands under `/container/add`, the distinction expressed by whether you give it the `remote-image` or `file` option, respectively.

Given the size of the output from `docker create --help`, it should not be surprising that the bulk of that is either not available in RouterOS or exists in a very different form. Most of these limitations stem from [the list above](#global). For instance, the lack of any CPU usage limit features means there is no equivalent under `/container` for the several `docker create --cpu*` options. Rather than go into these options one by one, I’ll cover the ones where the answers cannot be gleaned through a careful reading of the rest of this article:

*   **`--env`**: The equivalent is this RouterOS command pair:

        /container/envs/add name=NAME …
        /container/add envlist=NAME …
83
84
85
86
87
88
89
90

91
92

93
94



95
96
97
98

99
100
101
102
103







104
105
106
107
108
109
110
111

112


113

114

115

116


117

118


119

120























121
122
123

124


125

126
127
128
129

130
131

132
133
134
135

136


137
138
139
140
141
142
143

144
145
146
147
148
149
150
83
84
85
86
87
88
89

90
91

92
93
94
95
96
97
98
99
100

101
102
103
104


105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120

121
122
123
124

125
126
127

128
129
130
131

132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161

162
163
164
165

166
167
168
169

170
171

172
173
174
175
176
177

178
179
180
181
182
183
184
185

186
187
188
189
190
191
192
193







-
+

-
+


+
+
+



-
+



-
-
+
+
+
+
+
+
+








+
-
+
+

+
-
+

+
-
+
+

+
-
+
+

+

+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+


-
+

+
+
-
+



-
+

-
+




+
-
+
+






-
+







*   **`--rm`**: No direct equivalent. There is a manual `/container/remove` command, but nothing like this option, which causes the container runtime to automatically remove the instantiated container after it exits. It’s just as well since this option is most often used when running _ad hoc_ containers made from a previously downloaded image; RouterOS’s lack of an image cache means you have to go out of your way to export a tarball of the image and upload it to the router, then use “`/container/add file=…`” if you want to avoid re-downloading the image from the repository on each relaunch.

That brings us to the related matter of…

[script]: https://help.mikrotik.com/docs/display/ROS/Scripting


# <a id="run"></a>There Is No "Run"
# <a id="run"></a>There Is No “Run”

RouterOS offers no shorthand command akin to `docker run` for creating and starting a container in a single step. Moreover, the lack of Linux-like interactive terminal handling means a simple command like…
RouterOS offers no shorthand command akin to `docker run` for creating and starting a container in a single step. Moreover, the lack of Linux-like interactive terminal handling — covered in the [next section](#terminal) — means a simple command like…

    $ docker run --rm -it alpine:latest

…followed by…

    sh-5.1# <do something inside the container>
    sh-5.1# exit

is most briefly expressed under RouterOS as…
may end up expressed under RouterOS as…

    > /container
    > add remote-image=alpine:latest veth=veth1 entrypoint=sleep cmd=3600
    … wait for it to download and extract …
    … poll with "print" commands to get the container ID …
    … wait for it to download and extract by polling    > print
    … nope, not ready, wait some more …
    > print
    … nope, wait some more …
    > printoh, good, now we have the container ID …
    > start 0
    … wait for it to launch …
    > shell 0
    sh-5.1# <do something inside the container>
    sh-5.1# exit
    > stop 0
    > remove 0

Whew! 😅
The “sleep 3600” hack is necessary because `container.npk` lacks any notion of interactive mode. In order to keep a container of this type from starting, doing a whole lot of _nothing_, and then stopping,(^…but not cleaning up after itself as with `--rm`, mind!) you have to give it some type of busy-work to keep it alive. This is a common one, but it only lasts an hour. More complicated methods are difficult to pull off due to the lack of host-side Bourne shell command parsing in RouterOS; the `cmd` option gets passed into the container as a single string, but you only have two to play with, `entrypoint` and `cmd`, so how do you say something like the following?

I resorted to that “sleep 3600” hack in order to work around the lack of interactive mode in `container.npk`, without which containers of this type will start, do a whole lot of _nothing_, and then stop. I had to give it some type of busy-work to keep it alive long enough to let me shell in and do my actual work. This sneaky scam is a common one for accomplishing that end, but it has the downside of requiring you to predict how long you want the container to run before stopping; this version only lasts an hour.

If you are imaging more complicated methods for keeping containers running in the background when they were designed to run interactively, you are next liable to fall into the trap that…
    docker run alpine:latest 'while true ; do sleep 5 ; done' &


# <a id="cmd"></a>There Is No Host-Side Command Line Parser
The answer is that if you want a RouterOS container to do anything tricky like that, you need to write your own `ENTRYPOINT` script.

The RouterOS CLI isn't a Bourne shell, and the container feature's `entrypoint` and `cmd` option parsers treats them as simple strings, without any of the parsing you get for free when typing `docker` commands into a Linux command shell. The net effect of all this is that you’re limited to two-word commands, one in `entrypoint` and the other in `cmd`, as in the above “`sleep 3600`” hack.

But how then do you say something akin to the following under RouterOS?
With all of that to ground us, the rest is far simpler to discuss.

    docker run -it alpine:latest ls -lR /etc

You might want to do that in debugging to find out what a given `/etc` file is called and exactly where it is in the hierarchy so that you can target it with a `mount=…` override. If you try to pass it all as…

    /container/add … entrypoint="ls -lR /etc"

…the kernel will complain that there is no command in the container’s `PATH` called “`ls -lR /etc`”.

You may then try to split it as…

    /container/add … entrypoint="ls" cmd="-lR /etc"

…but that will earn you error message from `/bin/ls` complaining that there is no option “`-lR /etc`”!

If you try to “cuddle” the options with the arguments as…

    /container/add … entrypoint="ls" cmd="-lR/etc"

…the `/bin/ls` implementation will certainly attempt to treat `/` as an option and die with an error message.(^And it isn't just one implementation's weakness. This happens with the GNU, BSD, _and_ BusyBox implementations of `ls`.)

Things aren’t always this grim. For instance, you can run [my `iperf3` container](/dir/iperf3) as a client instead of its default server mode by saying something like:

    /container/add … cmd="-c192.168.88.99"

This relies on the fact that the `iperf3` command parser knows how to break the host name part out from the `-c` option itself, something not all command parsers are smart enough to do. There’s 50 years of Unix and Linux history encouraging programs to rely on the shell to do a lot of work before the program’s `main()` function is even called. The command line processing that `container.npk` applies to its `cmd` argument lacks all that power. If you want Bourne shell parsing of your command line, you have to set it via `ENTRYPOINT` or `CMD` in the `Dockerfile`, then rebuild the image.


# <a id="terminal" name="logs"></a>Terminal Handling and Logs

Although RouterOS proper is built atop Linux, and it provides a feature-rich CLI, it is nothing like a Linux command shell, but I am not speaking of skin-level command syntax differences here. The differences go far deeper. When you SSH into a RouterOS box, you’re missing out on a meaningful distinction between stdout and stderr, and the kernel’s underlying termios/pty subsystem is hidden from you. These lacks translate directly into limitations in the ability of `container.npk` to mimic the experience of using Docker at the command line.
Although RouterOS proper is built atop Linux, and it provides a feature-rich CLI, it is nothing like a Linux command shell, and I am not speaking of skin-level command syntax differences here; the differences go far deeper.

When you SSH into a RouterOS box, you’re missing out on a meaningful distinction between stdout and stderr, and the kernel’s underlying termios/pty subsystem is hidden from you. These lacks translate directly into limitations in the ability of `container.npk` to mimic the experience of using Docker at the command line.

One of the core RouterOS design principles is being able to run headlessly for long periods, with the administrator connecting to their virtual terminal via WinBox, WebFig, or SSH briefly, only long enough to reconfigure something before logging back out. The RouterOS CLI never was meant to provide the sort of rich terminal experience you need when you work in a Linux terminal all day, every day.
One of the core RouterOS design principles is being able to run headlessly for long periods, with the administrator connecting to their virtual terminal via WinBox, WebFig, or SSH briefly, only long enough to accomplish some network admin task before logging back out. The RouterOS CLI never was meant to provide the sort of rich terminal experience you need when you work in a Linux terminal all day, every day.

The thing is, Docker _was_ designed around this sensibility.

It is for this inherent reason that `container.npk` cannot provide equivalents of Docker’s `attach` command, nor its “`docker run --attach flag`”, nor the common “`docker run -it`” option pair. The closest it comes to all this is its [`shell`](#shell) command implementation, which can connect your local terminal to a true remote Linux terminal subsystem. Alas, that isn’t a close “`run -it`” alternative because you’re left typing commands at this remote shell, not at the container’s `ENTRYPOINT` process. Even then, it doesn’t always work since a good many containers lack a `/bin/sh` program inside the container in the first place, on purpose, typically to reduce the container’s attack surface.(^Indeed, all of [my public containers](https://hub.docker.com/repositories/tangentsoft) elide the shell for this reason.)
It is for this inherent reason that `container.npk` cannot provide equivalents of Docker’s `attach` command, nor its “`docker run --attach`” flag, nor the common “`docker run -it`” option pair. The closest it comes to all this is its [`shell`](#shell) command implementation, which can connect your local terminal to a true remote Linux terminal subsystem. Alas, that isn’t a close “`run -it`” alternative because you’re left typing commands at this remote shell, not at the container’s `ENTRYPOINT` process. Even then, it doesn’t always work since a good many containers lack a `/bin/sh` program inside the container in the first place, on purpose, typically to reduce the container’s attack surface.(^Indeed, all of [my public containers](https://hub.docker.com/repositories/tangentsoft) elide the shell for this reason.)

Although Docker logging is tied into this same Linux terminal I/O design, we cannot blame the lack of an equivalent to “`docker logs`” on the RouterOS design principles in this same manner. The cause here is different, stemming first from the fact that RouterOS boxes try to keep logging to a minimum by default, whereas Docker logs everything the container says, without restriction. RouterOS logs to memory by default to avoid burning out the flash, and it ignores all messages issued under "topics" other than the four preconfigured by default, which does not include the "container" topic you get access to by installing `container.npk`. To prevent log messages from being sent straight to the bit bucket, you must say:
Although Docker logging is tied into this same Linux terminal I/O design, we cannot blame the lack of an equivalent to “`docker logs`” on the RouterOS design principles in this same manner. The cause here is different, stemming first from the fact that RouterOS boxes try to keep logging to a minimum by default, whereas Docker logs everything the container says, without restriction. RouterOS logs to memory by default to avoid burning out the flash, and it ignores all messages issued under topics other than the four preconfigured by default, which does not include the container topic you get access to by installing `container.npk`. To prevent log messages from being sent straight to the bit bucket, you must say:

    /container/{add,set} … logging=yes
    /system/logging add topics=container action=…

Having done so, we have a new limitation to contend with: RouterOS logging isn’t as powerful as the Docker “`logs`” command, which by default works as if you asked it, “Tell me what this particular container logged since the last time I asked.” RouterOS logging, on the other hand, mixes everything together in real time, requiring you to dig through the history manually.
Having done so, we have a new limitation to contend with: RouterOS logging isn't as powerful as the Docker "`logs`" command.(^Nor, for that matter, `podman logs`, which ties into systemd's controversial unified "journal" subsystem, a design choice that paid off handsomely when it comes to pulling up per-container logs.) By default, it works as if you asked it, "Tell me what this particular container logged since the last time I asked." RouterOS logging, on the other hand, mixes everything together in real time, requiring you to dig through the history manually.

(The same is true of `podman logs`, except that it ties into systemd’s controversial unified “journal” subsystem, a design choice that paid off handsomely when it comes to pulling up per-container logs.)


# <a id="tlc"></a>Remaining Top-Level Commands

So ends my coverage of the heavy points. Everything else we can touch on briefly, often by reference to matters covered previously.

For lack of any better organization principle, I’ve chosen to cover the remaining `docker` CLI commands in alphabetical order. I skip over short aliases like `docker rmi` for `docker image rm` in order to cover things only once, and I don’t repeat any of the `create`/`load`/`run` discussion [above](#create). Because Podman cloned the Docker CLI, this matches fairly well with it, except that I do not currently go into any of its pure extensions, such as its eponymous `pod` command.
For lack of any better organization principle, I’ve chosen to cover the remaining `docker` CLI commands in alphabetical order. I skip over short aliases like `docker rmi` for `docker image rm` in order to cover things only once, and I don’t repeat any of the `create`/`load`/`run` discussion [above](#create). Because Podman cloned the Docker CLI, this ordering matches up fairly well with its top-level command structure as well, the primary exception being that I do not currently go into any of Podman’s pure extensions, ones such as its eponymous `pod` command.


## <a id="build"></a>`build`/`buildx`

RouterOS provides a bare-bones container runtime only, not any of the image building toolchain.