Bending Network Reality: Merging Virtual and Physical Networks with Dovesnap

Virtual networks (like overlays and VPNs) running over physical networks are commonplace.  Virtualized server infrastructure is also commonplace; but what about virtual networks and clients that appear to be part of a real physical network, such as virtual office desktop machines, or even routers and firewalls? What if we could mirror traffic and monitor those virtual network additions easily?

If we could do that, then we’d have a useful experimental testbed for enterprise network defense that would allow us to concentrate on experimentation without also having to orchestrate monitoring and mirroring. This would significantly reduce the amount of client hardware required, translating into some real cost savings, too. Ideally, such a testbed should scale down (all the way down to a Raspberry Pi for classroom use for example) as well as up.

In May 2020, we introduced Dovesnap, which extends Docker’s networking and virtualization capabilities into the physical network domain by using SDN (specifically, OVS and Faucet). Dovesnap provides centralized traffic mirroring and packet rewriting and filtering (including ACLs), though in this post we’ll concentrate on the basic reality bending necessary for experimental testbeds, described above.

Here we create a Dovesnap network that brings in an external VLAN 110 network on physical interface eno123 and enables DHCP:

$ docker network create mynet -d ovs --internal -o ovs.bridge.mode=flat -o ovs.bridge.dpid=0x1 -o ovs.bridge.add_ports=eno123 --ipam-driver null -o ovs.bridge.dhcp=true -o ovs.bridge.vlan=110

Dovesnap handles the bridging of the real and Docker networks for you (commanding OVS and Faucet) and it can use your network’s existing DHCP server, something Docker cannot do natively, to assign addresses to containers. This makes it very easy, for example, to create many test “hosts” and bridge them onto a real network for testing, such as to perform load testing, check bandwidth, or verify connectivity. Because the containers are connected to your network at layer 2, they also can provide or use layer 2 based broadcast services,  e.g., mDNS. Docker is already able to specify a container’s MAC address, so you can even manage a pool of containers with predictable MAC addresses.

Dovesnap can support both tagged and untagged VLAN connections. Since a tagged VLAN port (trunk) can handle more than one VLAN, you could potentially have many docker networks each connected to its own real VLAN.

OVS underlies Dovesnap’s flexibility. While it is possible to program OVS bridges from the command line without Dovesnap, it can be complex to do by hand and it requires a lot of orchestration: provisioning veth pairs to connect bridges, managing which OVS port is connected to what resource, configuring the Faucet SDN controller to provide connectivity and apply resources or traffic mirroring, etc. Below is an example of a real Dovesnap OVS network with mirroring ready to go. There’s a lot going on but don’t worry, Dovesnap, fortunately, manages it all for us. While we can view the underlying network, dump OpenFlow flows, and even add ports, we don’t have to – we can deal with Dovesnap at the Docker layer instead. Dovesnap will add and remove ports for containers, VMs, and physical devices as they come and go.

$ sudo ovs-vsctl show
588be375-46fc-4903-b924-1da9a394c2f1
    Bridge ovsbr-31df3
        Controller "tcp:192.168.199.1:6654"
            is_connected: true
        Controller "tcp:192.168.199.1:6653"
            is_connected: true
        fail_mode: secure
        Port vnet0
            Interface vnet0
        Port ovp31NmK4Kseo6
            Interface ovp31NmK4Kseo6
        Port ovs-veth0-8d6c9
            Interface ovs-veth0-8d6c9
        Port VLAN140
            Interface VLAN140
        Port ovsbr-31df3
            Interface ovsbr-31df3
                type: internal
    Bridge mirrorbr
        Port enp4s0
            Interface enp4s0
        Port mirrorbr
            Interface mirrorbr
                type: internal
        Port enp3s0
            Interface enp3s0
        Port ovpKseo631NmK4
            Interface ovpKseo631NmK4
    ovs_version: "2.13.1"

Dovesnap can also generate a graphviz diagram for you, showing a wiring diagram of your virtual network which is a very handy way to verify your network configuration! In this example, we have a container, a virtual machine, and an external VLAN 110 (representing the OVS structure, above) all bridged into the same Dovesnap network. Additionally, an optional connection for mirroring any of those connections, has been wired into an OVS bridge “mirrorbr” which also has physical network interfaces connected to it.

Using Grafana you can graph all the metrics of this activity without doing any extra configuration. Dovesnap commands Faucet (and Faucet’s statistics gathering component, Gauge) to monitor connections as they are assigned. Want to see how much traffic a container is generating? There’s a graph for that and the graph will remain available after the container exits – very convenient for forensics or incident response.

A lot is happening , and we haven’t even covered traffic mirroring and ACLs! Our next post will explore how to more finely control connectivity with Faucet ACLs – which can provide their own graphs – and how to mirror traffic from containers selectively.  Make sure to check back to learn more!