1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
|
-
-
-
+
-
+
-
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
-
+
-
+
-
+
|
# Motivation
The [RouterOS `container.npk` feature](https://help.mikrotik.com/docs/display/ROS/Container) is highly useful, but it is a custom development written in-house by MikroTik, not a copy of Docker Engine or any of the other server-grade container engines.(^Podman, LXC/LXD, etc.) Because of the stringent resource constraints on the bulk of MikroTik's devices, it is exceptionally small, thus unavoidably very thinly featured compared to its big-boy competition. If we can use installed size as a proxy for expected feature set size, we find:
* **Docker Engine**: 422 MiB(^Version 27.1.1, according to `dnf remove docker-ce…` after installing these packages [per the instructions](https://docs.docker.com/engine/install/rhel/#install-docker-engine).)
* **`containerd`+`nerdctl`**: 174 MiB(^Version 2.0.0-rc1 of `nerdctl` plus the `containerd` from the Docker Engine CE install according to `sudo dnf remove containerd` and `du -sh nerdctl`.)
* **Podman**: 107 MiB(^Version 4.9.4 on EL9, according to `sudo dnf remove podman conmon crun`.)
* **systemd-nspawn**: 1.3 MiB(^This is the bare-bones OCI image runner built into systemd, with a feature set fairly close to that of `container.npk`. The size above is for version 252 of this program's parent `systemd-container` package as shipped on EL9.)
* **`container.npk`**: _0.0626 MiB_(^Version 7.15.2, according to `/system/package/print`.)
And this is fine! RouterOS serves a particular market, and its developers are working within those constraints. The intent here is to provide a mapping between what people expect of a fully-featured container engine and what you actually get in RouterOS. Where it makes sense, I try to provide workarounds for missing features and guidance to alternative methods where RouterOS's way merely *works* differently.
<font color=red>This document is a **Work in Progress**.</font>
RouterOS's `container.npk` lacks:
The major things lacking in RouterOS's `container.npk` feature are:
* a local image cache(^One knock-on effect of this not covered above is that removing and reinstalling a container requires RouterOS to re-download the image, even when done back-to-back, even if you never start the container between and thereby cause it to make changes to the expanded image's files.)
* a CoW/overlay file system(^This is not a verified fact, but an inference based on the observation that if RouterOS _did_ have this facility underlying its containers, I would expect to find equivalents to Docker's `commit` and `diff` commands. This pairs with the lack of an image cache: no CoW means no need for a baseline source to compute deltas against.)
* image building
* orchestration
* JSON and REST APIs
* per-container limit controls:(^The only configurable resource limit is on maximum RAM usage, and it's global, not settable on a per-container basis.)
* FD count
* PID limit
* CPU usage
* storage IOPS
* `/dev/shm` size limit
* terminal/logging bps
* syscall restrictions
* [capability][caps] restrictions
* [rlimit]
* hardware pass-thru:
* USB device entries under `/dev` are on the wish list, but not currently available.
* USB device entries under `/dev` are on the wish list, but not currently available.(^Not unless RouterOS itself sees the USB device, as with storage media, which you can bind-mount into the container with "`/container/add mounts=…`".)
* There is no GPU support, not even for bare-metal x86 installs.
[caps]: https://www.man7.org/linux/man-pages/man7/capabilities.7.html
[rlimit]: https://www.man7.org/linux/man-pages/man2/getrlimit.2.html
A good many of these limitations stem from those of RouterOS itself. For instance, while RouterOS proper is built atop Linux, and it provides a feature-rich CLI, it is nothing like a Linux command shell. This means equivalent commands to the likes of "`docker run --attach std…`" would not make a lot of sense on RouterOS, there being nothing like the termios/pty subsystem visible at the RouterOS CLI level.
While I could also point out the lack of a background management daemon(^`containerd` in modern setups, `dockerd` in old ones) a good bit of Docker's competition also lacks this, on purpose, so I cannot ding RouterOS for this same lack.
With this grounding, let us get to the per-command details…
With this grounding, let us dive into the details…
## <a id="create"></a>Container Creation
The single biggest area of difference between the likes of Docker and the RouterOS `container.npk` feature is how you create containers from OCI images.
RouterOS combines Docker's `create` and `load` commands as `/container/add`, the distinction expressed by whether you give it the `remote-image` or `file` option, respectively.
There is no shorthand command like `docker run` for creating and starting a container in a single step. Moreover, the lack of Linux-like interactive terminal handling means a simple command like…
$ docker run --rm -it alpine:latest
…is most briefly expressed under RouterOS as…
> /container
> add remote-image=alpine:latest veth=veth1 entrypoint=sleep cmd=3600
… wait for it to download and extract …
… poll with "print" commands to get the container ID …
> start 0
… wait for it to launch …
> shell 0
sh-5.1# <do something inside the container>
sh-5.1# exit
> stop 0
> remove 0
The "sleep 3600" hack is necessary because RouterOS lacks any notion of interactive mode. In order to keep the container from starting, doing a whole lot of _nothing_, and then stopping(^…but not cleaning up after itself as with `--rm`, mind!) you have to give it some type of busy-work to keep it alive. This is a common one, but it only lasts an hour. More complicated methods are difficult to pull off due to the lack of host-side Bourne shell command parsing in RouterOS; the `cmd` option gets passed into the container as a single string, but you only have two to play with, `entrypoint` and `cmd`, so how do you say something like the following?
docker run alpine:latest 'while true ; do sleep 5 ; done' &
The answer is that if you want a RouterOS container to do anything tricky like that, you need to write your own `ENTRYPOINT` script.
Now, given the size of the output from `docker create --help`, it should not be surprising that the bulk of that is either not available in RouterOS or exists in a very different form. Most of these limitations stem from the list of [generalities above](#general). For instance, the lack of any CPU usage limit features means there is no equivalent under `/container` for the several `docker create --cpu*` options. Rather than go into these options one by one, I'll cover the ones where the answers cannot be gleaned through a careful reading of the rest of this article:
* **`--env`**: The equivalent is this RouterOS command pair:
/container/envs/add name=NAME …
/container/add envlist=NAME …
This is in fact closer to the way the **`--env-file`** option works, except that under RouterOS, this particular "file" isn't stored under `/file`!
* **`--expose`/`--publish`**: <a id="publish"></a>The VETH you attach the container to makes every listening socket visible by default. It is left up to you to manually block off anything exposed against your wishes by use of `/ip/firewall/filter` commands.
* **`--health-cmd`**: Because health-checks are often implemented by periodic API calls to verify that the container continues to run properly, the logical equivalent under RouterOS is to [script] calls to [`/fetch`](https://help.mikrotik.com/docs/display/ROS/Fetch), which then issues `/container/{stop,start}` calls to remediate any problems it finds.
* **`--init`**: Although there is no direct equivalent to this in RouterOS, nothing stops you from doing it the old-school way, creating a container that calls "`ENTRYPOINT /sbin/init`" or similar, which then starts the subordinate services inside that container. It would be somewhat silly to use systemd for this in a container meant to run on RouterOS in particular; a more suitable alternative would be [Alpine's OpenRC](https://wiki.alpinelinux.org/wiki/OpenRC) init system, a popular option for managing in-container services.
* **`--label`**: The closest equivalent is RouterOS's `comment` facility, which you can apply to a running container with "`/container/set 0 comment=MYLABEL`".
* **`--mac-address`**: If RouterOS had this, I would expect it to be offered as "`/interface/veth/set mac-address=…`", but that does not currently exist. As it stands, a VETH interface's MAC address is random, same as the default behavior of Docker.
* **`--network`**: This one is tricky. While there is certainly nothing like "`/container/add network=…`", it's fair to say the equivalent is, "RouterOS." You are, after all, running this container atop a highly featureful network operating system. Bare-bones the `container.npk` runtime may be, but any limitations you run into with the network it attaches to are more a reflection of your imagination and skill than to lack of command options under `/container`.
* **`--pid/userns/uts`**: The RouterOS container runner must use Linux namespaces under the hood, but it does not offer you control over which PID, file, network, etc. namespaces each container uses.
* **`--read-only`**: RouterOS offers precious little in terms of file system permission adjustment. As a rule, it is best to either shell into the container and adjust permissions there or rebuild the container with the permissions you want from go. Any expectations based on being able to adjust any of this between image download time and container creation time are likely to founder.
* **`--restart`**: <a id="restart"></a>The closest RouterOS gets to this is its `start-on-boot` setting, meaning you'd have to reboot the router to get the container to restart. If you want automatic restarts, you will have to [script] it.
* **`--rm`**: No direct equivalent. There is a manual `/container/remove` command, but nothing like this option, which causes the container runtime to automatically remove the instantiated container after it exits. It's just as well since this option is most often used when running _ad hoc_ containers made from a previously downloaded image; RouterOS's lack of an image cache means you have to go out of your way to export a tarball of the image and upload it to the router, then use "`/container/add file=…`" if you want to avoid re-downloading the image from the repository on each relaunch.
With all of that to ground us, the rest is far simpler to discuss…
[script]: https://help.mikrotik.com/docs/display/ROS/Scripting
# <a id="top"></a>Top-Level Commands
# <a id="tlc"></a>Remaining Top-Level Commands
For lack of any better organization principle, I've chosen to structure this document along the lines of the `docker` CLI, duplicating their command hierarchy, sorted alphabetically at each level. I skip over short aliases like `docker rmi` for `docker image rm` in order to cover things only once. Because Podman cloned the Docker CLI, this matches fairly well with it, except that I do not currently go into any of its pure extensions, like its eponymous `pod` command.
For lack of any better organization principle, I've chosen to cover these commands in alphabetical order. I skip over short aliases like `docker rmi` for `docker image rm` in order to cover things only once, and I don't repeat any of the `create`/`load`/`run` discussion [above](#create). Because Podman cloned the Docker CLI, this matches fairly well with it, except that I do not currently go into any of its pure extensions, such as its eponymous `pod` command.
## <a id="attach"></a>`attach`
There is no interactive terminal (stdin/stdout/stderr) in RouterOS to speak of, and you normally run these boxes headless, connecting to their Terminal via WinBox or SSH only long enough to reconfigure something before logging back out. This means it is meant to run containers purely in the background, with logging suppressed by default. If you say `/container/set logging=yes`, the standard output streams go to the configured logging destination, but there is no way to interactively type commands at the container short of `/container/shell`, which requires that `/bin/sh` exist inside the container. Even then, you're typing commands at the shell, not at the container's `ENTRYPOINT` process.
There is no interactive terminal (stdin/stdout/stderr) in RouterOS to speak of, and you normally run these boxes headless, connecting to their virtual terminal via WinBox or SSH only long enough to reconfigure something before logging back out. The `container.npk` feature is designed to run its subordinate processes purely in the background, with logging suppressed by default. If you say `/container/set logging=yes`, the standard output streams go to the configured logging destination, but there is no way to interactively type commands at the container short of `/container/shell`, which carries the requirement that a `/bin/sh` program exist inside the container.(^You can't count on that in every container. Indeed, all of [my public containers](https://hub.docker.com/repositories/tangentsoft) elide the shell to reduce the container's attack surface.) Even then, you're typing commands at the shell, not at the container's `ENTRYPOINT` process.
Because of all this, although RouterOS does have an equivalent of `docker run` — [details here](#run) — it lacks any equivalent to the common `-it` option pair. The closest it comes is its [`shell`](#shell) command implementation.
## <a id="build"></a>`build`/`buildx`
RouterOS provides a bare-bones container runtime only, not any of the image building toolchain. It is closer in nature to the `runc` command underlying `containerd` than to Docker Engine proper. An even closer match is the lightweight `crun` command at the heart of Podman, and even more so the elementary runner that ships with systemd, variously called either [`systemd-nspawn`][sdnsp] or [`systemd-container`][sdcnt], depending on the tastes of whoever is packaging it.
|
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
|
142
143
144
145
146
147
148
149
150
151
152
153
154
155
|
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
|
## <a id="cp"></a>`cp`
RouterOS does let you mount a volume inside a container, then use the regular `/file` facility to copy files in under that volume's mount point, but this is not at all the same thing as the "`docker cp`" command. There is no way to overwrite in-container files with external data short of rebuilding the container or using in-container mechanisms like `/bin/sh` to do the copying for you.
If you come from a Docker or Podman background, their local overlay image stores might lead you into thinking you could drill down into the GUID-named "container store" directories visible under `/file` and perform _ad hoc_ administration operations like overwriting existing config files inside the container, but alas, it does not.
## <a id="create"></a>`create`/`load`/`run`
RouterOS combines these commands as `/container/add`, though with many limitations, most of which stem from the list of [generalities above](#general). For instance, the lack of any CPU usage limit features means there is no equivalent under `/container` for the several `docker create --cpu*` options. Rather than go into these one by one, I'll add a few notes on matters you cannot glean by a careful reading of the rest of this article:
* **`--env`**: The equivalent is this RouterOS command pair:
/container/envs/add name=NAME …
/container/add envlist=NAME …
* **`--expose`/`--publish`**: Every TCP/UDP port published by the container is exposed to the VETH you attach the container to. It is left up to you to manually block off anything the container exposes against your wishes by use of `/ip/firewall/filter` commands.
* **`--health-cmd`**: Because health-checks are often implemented by periodic API calls to verify that the container continues to run properly, the logical equivalent under RouterOS is to [script] calls to [`/fetch`](https://help.mikrotik.com/docs/display/ROS/Fetch), which then issues `/container/{stop,start}` calls to remediate any problems it finds.
* **`--init`**: Although there is no direct equivalent, nothing stops you from doing it the old-school way, creating a container that calls "`ENTRYPOINT /sbin/init`" or similar, which then starts the subordinate services inside that container.
* **`--label`**: The closest equivalent is RouterOS's `comment` facility, which you can apply to a running container with "`/container/set 0 comment=MYLABEL`".
* **`--mac-address`**: If RouterOS had this, I would expect it to be offered as "`/interface/veth/set mac-address=…`", but that does not currently exist. As it stands, a VETH interface's MAC address is random, same as the default behavior of Docker.
* **`--network`**: This one is tricky. While there is certainly nothing like "`/container/add network=…`", it's fair to say the equivalent is, "RouterOS." You are, after all, running this container atop a highly featureful network operating system. Bare-bones the `container.npk` runtime may be, but any limitations you run into with the network it attaches to are more a reflection of your imagination and skill than to lack of command options under `/container`.
* **`--pid/userns/uts`**: The RouterOS container runner must use Linux namespaces under the hood, but it does not offer you control over which PID, file, network, etc. namespaces each container uses.
* **`--read-only`**: RouterOS offers precious little in terms of file system permission adjustment. As a rule, it is best to either shell into the container and adjust permissions there or rebuild the container with the permissions you want from go. Any expectations based on being able to adjust any of this between image download time and container creation time are likely to founder.
* **`--restart`**: The closest RouterOS gets to this is its `start-on-boot` setting, meaning you'd have to reboot the router to get the container to restart. If you want automatic restarts, you will have to [script] it.
* **`--rm`**: No direct equivalent. There is a manual `/container/remove` command, but nothing like this option, which causes the container runtime to automatically remove the instantiated container after it exits. It's just as well since this option is most often used when running _ad hoc_ containers made from a previously downloaded image; RouterOS's lack of an image cache means you have to go out of your way to export a tarball of the image and upload it to the router, then use "`/container/add file=…`" if you want to avoid re-downloading the image from the repository on each relaunch.
RouterOS doesn't have separate top-level commands for creating a container from an OCI image registry versus loading it from a tarball. They're both `/container/add`, differing in whether you give the `remote-image` or `file` options, respectively.
RouterOS has no shorthand command like `docker run` for creating and starting a container in a single step. You must `add` it, then `start` it.
[script]: https://help.mikrotik.com/docs/display/ROS/Scripting
|
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
|
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
|
-
+
-
+
+
+
-
+
-
+
+
+
+
+
+
-
+
-
-
|
## <a id="pause"></a>`pause`/`unpause`
No such feature in RouterOS; a container is running or not.
If the container has a shell, you could try a command sequence like this to get the same effect:
> /container/shell 0
$ pkill -STOP 'name of process'
$ pkill -STOP 'name of entrypoint'
## <a id="port"></a>`port`
RouterOS exposes all ports defined for a container in the `EXPOSE` directive in the `Dockerfile`. The only way to instantiate a container with fewer exposed ports is to rebuild it or override it with a different `EXPOSE` value.
RouterOS exposes all ports defined for a container in the `EXPOSE` directive in the `Dockerfile`. The only ways to instantiate a container with fewer exposed ports is to either rebuild it with a different `EXPOSE` value or to create a derived container with the `FROM` directive and set a new `EXPOSE` value.
(See also the discussion of [`--publish`](#publish) above.)
## <a id="ps"></a>`ps`/`stats`/`top`
The closest thing in RouterOS is the `/container/print follow*` commands.
A more direct alternative would be to shell into the container and run whatever it has for a `top` command, but of course that is contingent on what is available, if indeed there is a shell at all.
A more direct alternative would be to shell into the container and run whatever it has for a `top` command, but of course that is contingent on any of that being available.
## <a id="push"></a>`push`/`pull`
RouterOS maintains no local image cache, thus cannot push or pull images.
While it _can_ pull from an OCI image repo, it does so as part of `/container/add`, which is closer to a `docker create` command than to `docker pull`.
There is no equivalent at all to `docker push`.
## <a id="rename"></a>`rename`
RouterOS doesn't let you set the name on creation, much less rename it later. The closest you can come to this is to add a custom `comment`, which you can both set at "`add`" time and after creation.
## <a id="restart"></a>`restart`
RouterOS doesn't provide this shortcut. You must stop it and then start it again manually.
RouterOS has `/container/start`, but with many limitations relative to `docker start`:
RouterOS has `/container/start`, with limitations you can reasonably infer from the rest of this article.
<font color=red>**TODO**</font>
|