MikroTik Solutions: Bridge Per Switch Chip

Motivations

It's best to have only one bridge per router if you need/want hardware acceleration. As the Bridging & Switching documentation page correctly puts it:

The CRS1xx/2xx series switches support multiple hardware offloaded bridges per switch chip. All other devices support only one hardware offloaded bridge per switch chip.

Every other series of RouterOS based devices MikroTik currently sell allow you to have multiple bridges configured, but only one will be hardware-accelerated by the switch chip. RouterOS chooses which one automatically by default, but you can force it to choose the right one with the "hw=yes" flag when setting up the bridge and setting the others to "hw=no".

Now, if you're using a router-class device in the mode it was designed to support, hardware acceleration might not matter. You might have firewall rules and other configured features that require the traffic to cross the CPU. In that sort of situation, you need to disable hardware acceleration anyway, at least until the traffic's classified in a way that lets you fast-track the connection or similar. If your router is sufficiently fast, it often isn't worth playing fast-track type games, because it complicates the configuration, makes it harder to reason about, and prevents you from using things like queues to shape traffic beyond the initial connection packet exchange.

Contrast devices like the CRS3xx series, which have relatively weak CPUs, making them unsuited to being pressed into service as a proper router. They're better used as smart switches in the core of a SOHO LAN, using their routing capabilities to handle only a small proportion of the traffic. For instance, they're good as VPN gateways in cases where the connection speed is sufficiently bottlenecked upstream that you don't need enough CPU grunt for gigabit-class forwarding.

(VPNs are, by the way, another good example of traffic that must cross the CPU, and so can't be fully hardware-offloaded.)

You get gigabit speeds on a CRS3xx class device by following the standard advice: one hardware-accelerated bridge for everything to keep as much load off the weak CPU as possible.

VLANs vs Bridges

This then brings us to VLANs. If you're using a MikroTik device primarily as a switch, but you want some level of isolation between groups of ports, the normal advice is to put them all on a single bridge, then establish VLAN boundaries between them.

The problem with this is that VLANs are somewhat complicated to set up, and if you need to make the boundary somewhat porous, you get yourself right back into the problem of needing to set up routing, firewall rules, and such.

VLANs have one key virtue over bridges: a given network port can carry traffic from multiple VLANs at once, whereas a port can be a member of only one bridge.

Bridge Per Chip

But what if you don't need this virtue of VLANs? If each host needs to be a member of only one physical network, what are VLANs buying us?

There's a curious qualifier in the quote up at the top of this article: "…per switch chip." Can we make use of this in any way?

Consider the network block diagram of the RB1100AHx4. It shows three separate RTL8367 switch chips: one for ether1 thru ether5, one for ether6 thru ether10, and one for ether11 thru ether13. It allows a network design like this:

C1: oval "chip 1" fit
    arrow 0.3
R1: circle "🔄" fit
    arrow 0.3 ↔
C2: oval "chip 2" fit
    arrow 0.3 ↔
R2: circle "🔄" fit
    arrow 0.3 ↔
C3: oval "chip 3" fit
    arrow 0.3 ↔
R3: circle "🔄" fit
    arrow 0.3 ↔
IN: ellipse "Internet" fit

text "guest" "subnet" at 0.2 below C1.s
text "R1" at 0.1 below R1.s
text "production" "subnet" at 0.2 below C2.s
text "R2" at 0.1 below R2.s
text "DMZ" "subnet" at 0.2 below C3.s
text "R3" at 0.1 below R3.s

line from R1.n up 0.2
line "Internet bypass" above from previous.end then right to 0.2 above C3.n
arrow from previous.end to C3.n

For each "chip" in that diagram, you can have a separate hardware-accelerated bridge, because each RTL8367 chip in the RB1100 stands alone in this regard. R1 thru R3 aren't separate routers, but instead separate routing rule sets between the chips, gating the traffic allowed to transit the boundary between these separate hardware bridges. It creates the effect of having three small switches separated by routers, but within a single device.

Another pair of products in MikroTik's current lineup with this same characteristic is the RB4011 series, only both versions of it have two RTL8367s instead of three. You still have three physical networks in that case because the SFP+ port runs through the RB4011's CPU, which acts as a third switch chip connecting everything together. There are further distinctions between these two products, but they're outside the scope of this article. For our purposes here, they both permit three separate hardware networks connected together by the CPU.

Contrast the RB4011 series' putative replacement, the RB5009, where all nine ports run through a single switch chip, including the SFP+ port. This means that while the RB5009 is more advanced in many ways, you might actually prefer the RB4011 for some applications, depending on how you perceive those two gaps between the port groups. They impose a condition where traffic between these groups of ports must cross the CPU, but to determine whether that a good thing or a bad thing, we must consider the network design. Depending on our outlook, it may be an isolation advantage, or it may create an unwanted performance hit.

What all this means that within each group of ports controlled by a single switch chip, we can play fast-path games, but when traffic transits between groups, we can impose routing, firewalling, queueing, and other rules to control what traffic moves from one group to the other.

Private Services

In the diagram above, the arrow heads show the direction allowed for connections. There's a unidirectional arrow between the guest subnet and the production subnet because hosts on prod have no reason to talk to guest hosts, so the R1 rules block it. At the same time, we might have a server running on prod that guests need to access, so R1 selectively allows this: guests can talk to the Internet and to this private service, nothing else.

You might then consider that the diagram shows bidirectional communication between the DMZ and the production network. What does this mean? It may be that we have port-forwarding rules from the DMZ back into prod for much the same purpose as the guest case above, allowing selective access to a private LAN service.

Now combine the two cases: the "Internet bypass" path normally takes guest subnet traffic past the production network and drops it into the DMZ, where it's presumably picked up and sent outbound.

"But," you may object, "doesn't that mean guests can get back into prod from the DMZ? The arrows show it, right there!"

The answer is yes, but only within the restrictions you impose on that DMZ. If that barrier isn't well-designed, it isn't much of a DMZ. If you've made it possible for guests to get back into the production network from the DMZ, you've likely also made it possible for randos on the Internet to do the same. If the DMZ is properly configured, dumping guest traffic into the DMZ like this will have the desired effect: unless you're accessing internal services specifically forwarded into prod, the only valid path is outward.

Port Isolation

Another way we can make use of this hardware-level separation of ports is like so:

C1: oval "chip 1" fit
    move 0.1
    text "❌"
    move 0.1
C2: oval "chip 2" fit
    arrow 0.3 ↔
R2: circle "🔄" fit
    arrow 0.3 ↔
C3: oval "chip 3" fit
    arrow 0.3 ↔
R3: circle "🔄" fit
    arrow 0.3 ↔
IN: ellipse "Internet" fit

text "guest" "subnet" at 0.2 below C1.s
text "production" "subnet" at 0.2 below C2.s
text "R2" at 0.1 below R2.s
text "DMZ" "subnet" at 0.2 below C3.s
text "R3" at 0.1 below R3.s

line from C1.n up 0.2
line "port isolation, ether5 to ether11" above from previous.end then right to 0.2 above C3.n
arrow from previous.end to C3.n

Now the Internet bypass path uses enforced port isolation, configured in RouterOS like so:

/interface/ethernet/switch/port-isolation
set ether1 forwarding-override=ether5
set ether2 forwarding-override=ether5
...

The RouterOS docs calls feature this "private VLAN," but it's actually independent of VLANs. The proper term is as you see in the RouterOS commands: port isolation.

The scheme as diagrammed requires a short Ethernet cable from ether5 to one of the ports in the "chip 3" group. This is because you aren't allowed to set a forwarding override between two different switch chips. Although this does burn two ports to construct this bypass path, the advantage is that the guest network consumes no additional CPU resources for Internet traffic.

That doesn't mean zero CPU whatsoever. Your LAN might be behind a NAT layer, separating it from the Internet, so you need CPU resources to get traffic from the DMZ out to the Internet and back. The point of this scheme is that we no longer need explicit routing and firewalling rules to keep the guest network isolated. The switch chip now sees to that, at the cost of two ports to tie the chips together at the hardware level rather than at the CPU level, as in the prior diagram.

Wireless Plan B

That scheme is useful if you need two or more ports in the remaining ether1 thru ether4 group, but what if you have a guest network requiring only one wired connection?

Let's say that instead of a wired guest network, you want guests to use WiFi only, via a single AP. If you plug that AP into ether11 on an RB1100, the above diagrams transform to:

C1: oval "chip 1" fit
    arrow 0.3
R1: circle "🔄" fit
    arrow 0.3 ↔
C2: oval "chip 2" fit
    arrow 0.3 ↔
R2: circle "🔄" fit
    arrow 0.3 ↔
C3: oval "chip 3" fit
    arrow 0.3 ↔
R3: circle "🔄" fit
    arrow 0.3 ↔
IN: ellipse "Internet" fit

    arrow up 0.3 ↔ from C3.n
R4: circle "🔄" fit
    arrow left 0.3 ↔ from R4.w
    oval "WiFi" fit

text "OTHER" italic "subnet" at 0.2 below C1.s
text "R1" at 0.1 below R1.s
text "production" "subnet" at 0.2 below C2.s
text "R2" at 0.1 below R2.s
text "DMZ" "subnet" at 0.2 below C3.s
text "R3" at 0.1 below R3.s
text "R4" at 0.12 e of R4.e

What you do with that "OTHER" subnet is up to you. For instance, you might have an A/V network that needs to send data to the prod network, but not vice versa, and it shouldn't be visible to the guest WLAN. It's a simple matter of configuring the rules at R1 and R2 to achieve this.

Now let us consider the WiFi version of the RB4011. You can think of it as having five separate networks inside a single product, permitting a design like this:

C1: oval "chip 1" fit
    arrow 0.3
R1: circle "🔄" fit
    arrow 0.3 ↔
C2: oval "chip 2" fit
    arrow 0.3 ↔
R2: circle "🔄" fit
    arrow 0.3 ↔
CP: oval "CPU" "chip" width 66%
    arrow 0.3 ↔
R5: circle "🔄" fit
    arrow 0.3 ↔
IN: ellipse "Internet" fit

    arrow up 0.3 ↔ from CP.n
R3: circle "🔄" fit
    arrow left 0.3 ↔ from R3.w
    oval "2.4 GHz" fit
    arrow down 0.3 ↔ from CP.s
R4: circle "🔄" fit
    arrow left 0.3 ↔ from R4.w
    oval "5 GHz" fit

text "A/V" "subnet" at 0.2 below C1.s
text "R1" at 0.1 below R1.s
text "production" "subnet" at 0.2 below C2.s
text "R2" at 0.1 below R2.s
text "R3" at 0.12 e of R3.e
text "R4" at 0.12 e of R4.e
text "R5" at 0.1 below R5.s

Within a single device, we've got the effect of the separate WiFi guest AP above, but we have two separate WiFi networks to play with now. We can bond them together seamlessly, or we can treat them as separate networks, with separate rules for each. For instance, we might use the 2.4 GHz network for local IoT gear and allow guest access only on the 5 GHz band.

Conclusion

RouterOS devices with multiple switch chips inside permit network designs that would otherwise require multiple routers and/or switches. Careful study of the device's block diagram and adherence to the rules of what can happen within each switch chip and what needs CPU bypass can be rewarding, permitting compact and inexpensive solutions.

Bridge Per Switch Chip