Per-packet Scalable Switch Policy Controls with Dovesnap

In our previous post we showed how Dovesnap enables Docker networks to be connected with other networks at layer 2 – both physical and non-Docker virtual networks. We also showed some examples of how Dovesnap leverages Faucet’s monitoring and graphing capabilities via Prometheus and Grafana (see our previous post to learn more about the Faucet community and its capabilities). In this post, we’ll further explore two other Dovesnap features: centralized granular mirroring and Faucet Access Control Lists (ACLs).

Dovesnap mirroring allows you to select, on a per interface or per container basis, traffic to be mirrored and dispatched to a centralized location. This can be either a physical or virtual interface on the local host or a remote host. This minimizes the amount of cabling and configuration you need to do. A physical stack of hosts running Dovesnap can be connected together in a “mirror river” so that traffic arrives in one place (see diagram below). Further, Dovesnap can mark the traffic with VLAN tags and MAC address prefixes, making it easier to analyze the mirrored traffic and find containers, VLANs, or switch ports of interest.

Here’s an example physical setup using Dovesnap on five machines and a hardware switch. One machine, Ctrl 0, is running the Faucet controller, NFV functions such as DHCP and DNS, and monitoring and analytics which has its own separate dedicated mirror port. There are then three physical machines, Host 1, 2, and 3, which each has its own combinations of defined networks and VLANs and are all attached to the mirror river. The final machine, Laptop, receives whatever comes through the mirror river, which we’ll explain in the Faucet config next.

Here is the corresponding faucet.yaml config file that maps these physical machines and networks into the OpenFlow 1.3 physical network switch:

dps:
  switch1:
    dp_id: 3
    hardware: [OpenFlow 1.3 Switch]
    interfaces:
      2:
        description: controller nfv
        tagged_vlans:
        - management
        - tenant1
        - tenant2
        - tenant3
        - tenant4
      3:
        description: Poseidon local mirror
        output_only: true
      4:
        description: host1 tenant1 tenant2
        tagged_vlans:
        - management
        - tenant1
        - tenant2
      5:
        description: host1 tenant3
        tagged_vlans:
        - tenant3
      6:
        description: host2 tenant4
        tagged_vlans:
        - management
        - tenant4
      7:
        description: host3 all tenants
        tagged_vlans:
        - management
        - tenant1
        - tenant2
        - tenant3
        - tenant4
      8:
        acls_in:
        - patchit
        coprocessor:
          strategy: vlan_vid
      9:
        description: dovesnap mirror
        output_only: true
acls:
  patchit:
  - rule:
      actions:
        output:
          ports:
          - 9
vlans:
  management:
    vid: 199
  tenant1:
    vid: 110
  tenant2:
    vid: 120
  tenant3:
    vid: 130
  tenant4:
    vid: 140

Interfaces 8 and 9 in the above configuration implement the downstream end of the mirror river. The river flows into interface 8 and all packets are forwarded to interface 9, where a host running tcpdump is connected. Interface 8 is marked as a Faucet “coprocessor”, which delegates regular Ethernet switching functionally to an external machine – in this case it’s just being used to implement a simple static patch panel.

Mirroring all the traffic for a container is simple with Dovesnap. When starting Dovesnap, specify the interface where mirrored traffic should be sent on the mirror river, then when starting a container on a Dovesnap network, specify the mirror label:

$ docker run -d --label="dovesnap.faucet.mirror=true" --net=mynet busybox sleep 1h

That’s it. All traffic originated by, and sent to that container, will be mirrored to the interface specified with a VLAN tag per the values set when creating the Dovesnap network. Another useful option when mirroring numerous different containers simultaneously is the ability to set the prefixes of the container’s MAC address. That makes selecting a container’s traffic from all mirrored traffic very easy as it can be matched on VLAN tag and MAC address. Here’s an example of starting a container that will be mirrored and set to a specific MAC address prefix:

$ docker run -d --label="dovesnap.faucet.mirror=true" --label="dovesnap.faucet.mac_prefix=0e:99"  --net=mynet busybox sleep 1h

As discussed in the introduction, multiple Dovesnap hosts can be daisy-chained together with one output mirror interface leading to the next host’s input mirror interface. This uses fewer connections and interfaces than a star topology and the mirror interfaces can be 10G or 100G, if desired. Because mirroring is done at a granular port and container interface, you don’t need to mirror everything – just the resources you’re most interested in.  And, you can run a single capture process that sees traffic from all hosts, which makes logging and correlation much smoother.

Faucet ACLs leverage the OpenFlow programmability of Open vSwitch software (and hardware accelerated with DPDK) switching and physical switches that support OpenFlow. Dovesnap allows you to program ACLs, like mirroring, on a per container or port basis. Faucet ACLs are not limited to just the typical block or allow traffic, they can also be used to rate limit traffic between two machines (or containers) with OpenFlow meters and even dynamically rewrite packets on the fly, such as replacing Ethernet or IP source and destination addresses. Faucet ACLs have the same syntax whether they are programming software or hardware switches, and some Faucet-supported hardware switches can execute those complex rewriting or traffic rate-limiting features in hardware at line rate. Faucet ACLs can also coexist with iptables, which Docker natively uses for NATing container traffic.

In this next example, we’ll use a Faucet ACL to rate limit the traffic that a web server container can serve on port 80 (http). First, we need the Faucet ACL itself, including the OpenFlow meter. This example causes traffic over 100Kbps to be dropped by the switch if it originates from TCP port 80. We can simply add this meter and ACL to our faucet.yaml example we walked through above.

meters:
  lossymeter:
    meter_id: 1
    entry:
        flags: "KBPS"
        bands: [{type: "DROP", rate: 100}]
acls:
  ratelimitit:
  - rule:
      eth_type: 0x0800
      ip_proto: 6
      tcp_src: 80
      actions:
        meter: lossymeter
        allow: 1
  - rule:
      actions:
        allow: 1
 
Then we need to start the web server container and tell Dovesnap to apply the ACL:
 
$ docker run -P -d --label="dovesnap.faucet.portacl=ratelimitit --net=mynet nginxdemos/hello

That’s it.

Now that container will be restricted to only serving HTTP (port 80) traffic at a maximum rate of 100Kbps, while other traffic will still have unrestricted bandwidth. All Faucet ACLs come with built-in monitoring. Faucet will collect statistics on how many packets and bytes match each rule in the ACL, and this can be graphed in Prometheus and Grafana. Faucet ACLs also have a convenient pattern for debugging blocked traffic by using the ACL output action. That action allows you to have the switch drop the packet, but also make a copy and mirror it with a VLAN tag of your choice. This makes it very easy to see exactly what traffic was dropped by the ACL policy rather than disappearing into a “default deny rule”.

acls:
  denyallmirror:
  - rule:
      actions:
        output:
          allow: 0
          ports:
            - 99

This ACL will deny incoming traffic by default but also output the denied packet to port 99.

This post covered a lot of packet switching details – how Dovesnap enables Faucet to apply switching policy and packet rewriting and copying to the networks it controls, which can be virtual or physical switches – all while operating alongside iptables. For an example of more advanced packet processing, see pipette, and for background on the Faucet ACL feature including mirroring, see Faucet’s documentation. All of these capabilities and components do require some administration and orchestration, which is why we will introduce Portunus next. Portunus sits above Dovesnap and associated tools, and orchestrates creating and deleting Dovesnap networks as well as connecting, monitoring, and filtering the traffic those networks generate.