# Motivation

The [RouterOS `container.npk` feature](https://help.mikrotik.com/docs/display/ROS/Container) is highly useful, but it is a custom development written in-house by MikroTik, not a copy of Docker Engine or any of the other server-grade container engines.(^Podman, LXC/LXD, etc.) Because of the stringent resource constraints on the bulk of MikroTik’s devices, it is exceptionally small, thus unavoidably very thinly featured compared to its big-boy competition. If we can use installed size as a proxy for expected feature set size, we find:

* **Docker Engine**: 422 MiB(^Version 27.1.1, according to `dnf remove docker-ce…` after installing these packages [per the instructions](https://docs.docker.com/engine/install/rhel/#install-docker-engine).)
* **`containerd`+`nerdctl`**: 174 MiB(^Version 2.0.0-rc1 of `nerdctl` plus the `containerd` from the Docker Engine CE install according to `sudo dnf remove containerd` and `du -sh nerdctl`.)
* **Podman**: 107 MiB(^Version 4.9.4 on EL9, according to `sudo dnf remove podman conmon crun`.)
* **systemd-nspawn**: 1.3 MiB(^This is the bare-bones OCI image runner built into systemd, with a feature set fairly close to that of `container.npk`. The size above is for version 252 of this program’s parent `systemd-container` package as shipped on EL9.)
* **`container.npk`**: _0.0626 MiB_(^Version 7.15.2, according to `/system/package/print`.)

And this is fine! RouterOS serves a particular market, and its developers are working within those constraints. The intent here is to provide a mapping between what people expect of a fully-featured container engine and what you actually get in RouterOS. Where it makes sense, I try to provide workarounds for missing features and guidance to alternative methods where RouterOS’s way merely *works* differently.


# <a id="global"></a>Global Limitations

    *   There is no GPU support, not even for bare-metal x86 installs.

[caps]:   https://www.man7.org/linux/man-pages/man7/capabilities.7.html
[rlimit]: https://www.man7.org/linux/man-pages/man2/getrlimit.2.html
[sdcnt]: https://packages.fedoraproject.org/pkgs/systemd/systemd-container/
[sdnsp]: https://wiki.archlinux.org/title/Systemd-nspawn

Lack of a management daemon(^`containerd` in modern setups, `dockerd` in old ones) is not in that list because a good bit of Docker’s competition also lacks this, on purpose. Between that and the other items on the list, `container.npk` compares most closely to a container *runner* like `runc`, `crun`, or `systemd-nspawn` than to a full *engine* like Docker or Podman.
Lack of a management daemon(^`containerd` in modern setups, `dockerd` in old ones) is not in that list because a good bit of Docker’s competition also lacks this, on purpose. Between that and the other items on the list, the fairest comparison is not to fully-featured container *engines* like Docker and Podman but to the container *runner* at the heart of these systems:

* **runc**: 14 MiB(^This is the container runner underpinning `containerd`, thus also Docker, although it precedes it. Long before they created `containerd`, it underpinned `dockerd` instead. Because it is so primordial, a good many other container engines are also based on it.)
* **crun**: 0.5 MiB(^This Podman's alternative to `runc`, written in C to make it smaller. Early versions of Podman once relied on `runc`, and it can still be configured to use it, but the new default is to use the slimmer but feature-equivalent `crun`.)
* **systemd-nspawn**: 1.3 MiB(^This is the bare-bones OCI image runner built into systemd, with a feature set fairly close to that of `container.npk`. The size above is for version 252 of this program’s parent `systemd-container` package as shipped on EL9.)

One reason `container.npk` is far smaller than even the smallest of these runners is that the engines delegate much of what RouterOS lacks to the runner, so that even then it's an unbalanced comparison. Docker's `start`, `stop`, `kill`, `ps`, `pause`, and `rm` commands are handled way down at the runner level, not up at the top-level CLI.

With this grounding, let us dive into the details.


## <a id="create" name="load"></a>Container Creation

The single biggest area of difference between the likes of Docker and the RouterOS `container.npk` feature is how you create containers from OCI images. It combines Docker’s `create` and `load` commands as `/container/add`, the distinction expressed by whether you give it the `remote-image` or `file` option, respectively.

    docker run alpine:latest 'while true ; do sleep 5 ; done' &

The answer is that if you want a RouterOS container to do anything tricky like that, you need to write your own `ENTRYPOINT` script.

With all of that to ground us, the rest is far simpler to discuss.


# <a id="tlc"></a>Remaining Top-Level Commands
# <a id="terminal" name="logs"></a>Terminal Handling and Logs

For lack of any better organization principle, I’ve chosen to cover these commands in alphabetical order. I skip over short aliases like `docker rmi` for `docker image rm` in order to cover things only once, and I don’t repeat any of the `create`/`load`/`run` discussion [above](#create). Because Podman cloned the Docker CLI, this matches fairly well with it, except that I do not currently go into any of its pure extensions, such as its eponymous `pod` command.
Although RouterOS proper is built atop Linux, and it provides a feature-rich CLI, it is nothing like a Linux command shell, but I am not speaking of skin-level command syntax differences here. The differences go far deeper. When you SSH into a RouterOS box, you’re missing out on a meaningful distinction between stdout and stderr, and the kernel’s underlying termios/pty subsystem is hidden from you. These lacks translate directly into limitations in the ability of `container.npk` to mimic the experience of using Docker at the command line.

One of the core RouterOS design principles is being able to run headlessly for long periods, with the administrator connecting to their virtual terminal via WinBox, WebFig, or SSH briefly, only long enough to reconfigure something before logging back out. The RouterOS CLI never was meant to provide the sort of rich terminal experience you need when you work in a Linux terminal all day, every day.

The thing is, Docker _was_ designed around this sensibility.
## <a id="attach"></a>`attach`

It is for this inherent reason that `container.npk` cannot provide equivalents of Docker’s `attach` command, nor its “`docker run --attach flag`”, nor the common “`docker run -it`” option pair. The closest it comes to all this is its [`shell`](#shell) command implementation, which can connect your local terminal to a true remote Linux terminal subsystem. Alas, that isn’t a close “`run -it`” alternative because you’re left typing commands at this remote shell, not at the container’s `ENTRYPOINT` process. Even then, it doesn’t always work since a good many containers lack a `/bin/sh` program inside the container in the first place, on purpose, typically to reduce the container’s attack surface.(^Indeed, all of [my public containers](https://hub.docker.com/repositories/tangentsoft) elide the shell for this reason.)

Although Docker logging is tied into this same Linux terminal I/O design, we cannot blame the lack of an equivalent to “`docker logs`” on the RouterOS design principles in this same manner. The cause here is different, stemming first from the fact that RouterOS boxes try to keep logging to a minimum by default, whereas Docker logs everything the container says, without restriction. RouterOS logs to memory by default to avoid burning out the flash, and it ignores all messages issued under "topics" other than the four preconfigured by default, which does not include the "container" topic you get access to by installing `container.npk`. To prevent log messages from being sent straight to the bit bucket, you must say:
Although RouterOS proper is built atop Linux, and it provides a feature-rich CLI, it is nothing like a Linux command shell. Even when you SSH in, you’re missing things like a meaningful distinction between stdout and stderr, and the kernel’s underlying termios/pty subsystem is hidden at the RouterOS CLI level.

    /container/{add,set} … logging=yes
    /system/logging add topics=container action=…

Having done so, we have a new limitation to contend with: RouterOS logging isn't as powerful as the Docker "`logs`" command.(^Nor, for that matter, `podman logs`, which ties into systemd's controversial unified "journal" subsystem, a design choice that paid off handsomely when it comes to pulling up per-container logs.) By default, it works as if you asked it, "Tell me what this particular container logged since the last time I asked." RouterOS logging, on the other hand, mixes everything together in real time, requiring you to dig through the history manually.
This lack of exposed Linux terminal interfaces stems from the design principle that these boxes are meant to run headless, connecting to their virtual terminal via WinBox or SSH only long enough to reconfigure something before logging back out. 


Consequently, `container.npk` expects to run its subordinate processes purely in the background, with logging suppressed by default. If you say `/container/set logging=yes`, the standard output streams go to the configured logging destination, which you must set *in addition* under `/system/logging`.
# <a id="tlc"></a>Remaining Top-Level Commands

All of this explains why there are no equivalents of Docker’s `attach` command, nor its “`docker run --attach flag`”, nor the common “`docker run -it`” option pair. The closest `container.npk` comes to all this is its [`shell`](#shell) command implementation, which can connect your local terminal to a true remote Linux terminal subsystem. That isn’t a close “`run -it`” alternative because you’re left typing commands at this remote shell, not at the container’s `ENTRYPOINT` process.
So ends my coverage of the heavy points. Everything else we can touch on briefly, often by reference to matters covered previously.

And even then it doesn’t always work, since a good many containers lack a `/bin/sh` program inside the container in the first place, on purpose, typically to reduce the container’s attack surface.(^Indeed, all of [my public containers](https://hub.docker.com/repositories/tangentsoft) elide the shell for this reason.)
For lack of any better organization principle, I’ve chosen to cover the remaining `docker` CLI commands in alphabetical order. I skip over short aliases like `docker rmi` for `docker image rm` in order to cover things only once, and I don’t repeat any of the `create`/`load`/`run` discussion [above](#create). Because Podman cloned the Docker CLI, this matches fairly well with it, except that I do not currently go into any of its pure extensions, such as its eponymous `pod` command.


## <a id="build"></a>`build`/`buildx`

RouterOS provides a bare-bones container runtime only, not any of the image building toolchain.

    /container/config/set registry-url=… username=… password=…

The only way to “log out” is to overwrite the username and password via:

    /container/config/set username="" password=""


## <a id="logs"></a>`logs`

By default, RouterOS drops all logging output from a container. To see it, you must enable it on a per-container basis with the `/container/add logging=yes` option, then tell RouterOS where to send those logs via a `/system/logging add topics=container …` command.

Each message is handled in real time, not buffered as with Docker or Podman. Furthermore, RouterOS mixes logs from all sources for a given “topic” set, which in this context means that if you have multiple running containers on the device, their logs all go to the same place. Thus, if you were expecting to be able to set up memory logging for a container, log out of the router, then sometime later come back in and get a dump of everything that one particular container has logged since the last time you asked — as you can with the big-boy container engines — then you will be disappointed.


## <a id="pause"></a>`pause`/`unpause`

No such feature in RouterOS; a container is running or not.

If the container has a shell, you could try a command sequence like this to get the same effect: