Motivation

I’m writing this because the official docs are thin on details for this use case.¹

It documents what worked for me.

The Value of VMs

NetInstall needs to force I/O through a single network path under all conditions in order to do what it does. This might seem like an easy thing to accomplish, but then realize that NetInstall operates at a very low level, and there are multiple stages to the conversation, each of which may have different rules applied by elements of the intervening network.

Once upon a time, a typical computer would have only one “real” network interface plus the loopback interface, giving the OS running NetInstall only one logical choice for all outbound I/O, but modern systems are far more complicated. The one I’m typing this on has twenty-five network interfaces defined. While most of them are virtualized interfaces of some type,² each one is capable of changing the host OS’s routing table, affecting packet flow. If you think you know all the routing rules on such a system without careful inspection and testing, you’re probably wrong.

VMs give us a way to return to those good old days without affecting the host system: configure this NetInstall VM with a single virtual network interface, then bridge that to a specific hardware interface out on the host.

This recommendation holds even for those running Linux natively on the host system. While you can run netinstall-cli directly in that case, setting a subordinate VM up for the sole purpose of forcing traffic down a single path is easier than trying to work out why NetInstall is failing atop the host OS. Like as not, the solution will involve reconfiguring the host to placate NetInstall, then reverting those changes when you’re done with NetInstall in order to get on with what you were doing before. If you instead abstract the problem away with a VM like this, the configuration is isolated to that one VM, requiring no changes to the host OS’s configuration.

Although I happened to prove this setup out with a CentOS Stream 9 VM running inside the latest version of the Parallels virtual machine manager atop the latest version of macOS, I don’t believe any of these details to be critical. You should be able to do the same thing atop Windows with Hyper-V running Debian, or atop Arch Linux with KVM running Ubuntu, or whatever.

The only key configuration choice is bridging the virtual network adapter to the one-and-only host-side Ethernet adapter that netinstall-cli will communicate over.³ Success lies in avoiding cleverness like NAT, “shared” networking, automatic switching between Ethernet and WiFi, etc.

Router Configuration

Only one Ethernet port on your router will participate in an EtherBoot conversation. It might be marked “BOOT,” but if not, it’s generally the one that comes up as “ether1” in the default configuration. NetInstall will get stuck in the “Waiting for RouterBOARD...” step if you have the Ethernet cable plugged into the wrong port.

Intervening Hardware

Another common way to fail with NetInstall is to put too much cleverness between the netinstall-cli host and the target device.

The ideal setup is to run a single cable from the netinstall-cli VM’s hardware port to the target’s Etherboot port. Nearly all hosts support Auto MDI-X these days, including MikroTik’s offerings.

Next-simplest is to interpose a dumb Ethernet switch.

Where you start to run risks is when you want to leave one or more smart switches between the VM host and the target device. A common way to fail here is to have a switch supporting DHCP Snooping in the way, which will (not might, will) break NetInstall. If the intervening switch is a MikroTik and you cannot go with one of the two options above, you need to mark the bridge port toward the VM host as “Trusted,” allowing its BOOTP replies to pass to the target device.

Never interpose anything still smarter, such as a router or a wireless AP. It’s doubtless possible to make it work, but you’re reading this guide to learn the easy path, right? Right.

Server Configuration

The key server-side change is that many Linux OSes ship with a firewall enabled which will block the ports netinstall-cli needs when communicating with the router. The tricky bit is, the minimum set of ports isn’t documented anywhere, that I can see. Red Hattish OSes⁴ use firewalld these days, where the commands to unblock the required ports are:

sudo firewall-cmd --add-port bootps/udp
sudo firewall-cmd --add-port tftp/udp
sudo firewall-cmd --add-port 5000/udp

Other Linuxes use other firewall systems. Some still use raw iptables or nft commands, ufw is popular on Ubuntu, etc.

The first two required ports aren’t much of a surprise given the mention of “BOOTP” in the official docs, but I had to do a packet capture to work out that the last one was required. Without it, you’ll get stuck at the “sendFile” step.⁵

Now you can start the server:

sudo netinstall-cli \
    -i enp0s5 \
    -r -s reset.scr \
    routeros-7.9-arm.npk \
    wifiwave2-7.9-arm.npk \
    container-7.9-arm.npk

The enp0s5 value will vary by OS and virtual hardware configuration. On modern Linuxes, say “ip link” to get a list of possible names. For a VM, there are likely only two; pick the one that isn’t the lo interface.

Resetting the configuration is optional, and you can choose to accept the default config instead of what I’ve done here, but I wanted to show the option. The referenced reset.scr file is given below.

For routers with wired interfaces only, the base routeros-*.npk package is all you require, but for WiFi based routers, if you fail to at least include the appropriate wireless package, the default configuration is likely to come up improperly. Anything else you add to this is purely optional; I’m showing the container package as an example only.

I don’t know how critical it is to use the matching version of netinstall-cli when changing RouterOS versions, but while you’re downloading fresh NPKs, you might as well update it as well.

Default Router Configuration Script

My choice above to reset the configuration and apply a fresh configuration lets us do something fun and useful:

/interface bridge add name=bridge
/interface bridge port
add bridge=bridge interface=ether1
add bridge=bridge interface=ether2
add bridge=bridge interface=ether3
add bridge=bridge interface=ether4
/ip dhcp-client add interface=bridge
/user add name="foo" password="bar" group="full"

This is the reset.scr file referenced above, and although it is intended for any of MikroTik’s small WiFi routers, it should serve as a minimalist starting configuration for a wide range of devices.

The bridge setup configures it as a wired-only smart switch. Even if you have a WiFi device and you do want to use it as such, starting your configuration with the radios disabled is good practice. You want to set up the bands, radio power levels, and encryption secrets before exposing a wireless network that radiates outside your building.

The DHCP client lets you do several of these routers and connect them to a setup LAN without creating IP address collisions.

The default user step gets you around MikroTik’s new policy of resetting the admin user’s password to a random value printed in tiny text on a small sticker on the bottom of the router.⁶

There’s one subtlety to this: netinstall-cli will yell if you name the script other than *.scr. I happen to think it ought to accept the *.rsc extension it uses on /export, but oh, well.

Myths

You do NOT need to…

• disable the VM host’s WiFi⁷
• change your NetInstall VM’s IP address to use the 192.168.88.0/24 subnet
• configure a static IP on the VM; even if you set it up using DHCP before reattaching the Ethernet cable to the target router, the Linux OS you’re using inside that VM should keep the address until the lease expires
• connect the router directly to the server’s copper Ethernet port; there merely needs to be an unimpeded L2 path between the two⁸

Tips

If you get the “Key was rejected” message, hit Ctrl-C to break out of netinstall-cli, then Up-Arrow and Enter to quickly restart it. I’ve seen this bypass the symptom when using a CentOS 8 Stream VM as the server.

License

This work is © 2023-2024 by Warren Young and is licensed under CC BY-NC-SA 4.0

---

1. ^{^} They’d rather burn bandwidth telling you the 14 steps necessary to arm-twist Windows into this role by way of twenty-seven 800-by-400 color glossy screenshots with circles and arrows and a paragraph on the bottom of each one. Always keep telling yourself that Windows is the “easy” OS. You’ll come to believe it, eventually, no matter how much evidence mounts up to the contrary.
2. ^{^} Software bridges, tunnels, phony VM NICs, etc.
3. ^{^} In Parallels, you do that via the Devices → Network → Bridged Network → Ethernet menu choice. Other suitably powerful hypervisors — including VMWare and VirtualBox — can do the same via different command paths. I suspect Hyper-V is among that group, but I’ve had difficulties in the past with its bridged networking capabilities.
4. ^{^} RHEL, CentOS, AlmaLinux, Rocky Linux, Oracle Linux, Fedora…
5. ^{^} Another cause of the “stuck in sendFile symptom is attempting to send packages of the wrong architecture, such as AMD64 to a 32-bit ARM box or vice versa. If the product specs merely say “ARM” it means 32-bit.
6. ^{^} To be fair, I fully support this policy; in fact, I proposed essentially what we eventually got on the forums long before they implemented it. I’m more annoyed that it took European Union legislation to arm-twist MikroTik into doing this than I am irritated by the fact that it now requires use of either a strong magnifier or a small trainable child before you can learn what that default password actually is on a device like the hAP ax lite, where it’s printed at 37 DPI using a 3pt dot matrix font.🙄 I’m showing you how to get around this password not because I believe in discarding this excellent security measure but to show that you don’t need to know the password to reset it. My recommendation is to treat this as a temporary password only; one of your first steps should be to give it your own long, random password, one not printed on any stickers anywhere.
7. ^{^} The VM doesn’t even know about the host’s second interface, and we had you bind the VM to the Ethernet interface above regardless.
8. ^{^} While the simplest way to achieve that is indeed to put a direct cable between the two and rely on MDI-X to do the crossover magic, it’s just as effective to put a dumb switch between the two. If instead you substitute a sufficiently intelligent switch, such as a CRS running RouterOS, there’s a fair chance that one of the clever things you bought it to do will end up blocking the NetInstall packets. The one that tripped me up in testing is RouterOS’s “Trusted” flag, which you’re recommended to disable on bridge ports intended for “edge” devices, ones which have no business sending DHCP replies to other clients. The thing is, netinstall-cli does make legitimate use of the BOOTP port, and since DHCP is an extension of BOOTP, telling the switch that your NetInstall machine is not “Trusted” to send DHCP packets prevents NetInstall from getting started. Only the server port needs to be marked “Trusted”, presumably because it is the one sending these BOOTP packets. The reprogrammed device’s port should continue to be marked non-Trusted in case your default configuration includes a DHCP server, to keep it from confusing the rest of the LAN in the time before you can get it unplugged again.