Take the Pain out of Orchestrating & Instrumenting Testbed Environments with Portunus

In previous posts, we introduced Dovesnap and explained how it works with Faucet to extend Docker’s network capabilities by blending physical networks with virtual ones. Dovesnap also enables access control lists (ACLs), per-packet rewriting and switching, and centralized mirroring. 

While these capabilities are powerful and drop natively into Docker’s networking interface, they require manual orchestration. This post introduces Portunus, a tool we’ve created here at IQT Labs that adds automated orchestration to Dovesnap with a simple, terminal-based user interface. Portunus enables the building and testing of different network and infrastructure scenarios in an accessible and inexpensive fashion. Portunus also allows you to set up entire networks with containers and virtual machines (VMs) in a few simple menu-driven operations and clean them all up again when you’re done.

Portunus ensures that you spend your time experimenting instead of setting up your testbed and its instrumentation. Cloud environments are powerful and convenient for applications. But they also require skill and time on the part of the cloud operator, not to mention care to set up and maintain correctly. Portunus aims to make the cloud operator’s job in these regards (particularly research clouds) easier.

When first using Portunus, you’ll need to use the Install Dependencies option as seen above.  This takes care of installing all the necessary infrastructure, including Dovesnap and Faucet. Below is a complete transcript of a Portunus session bringing up two Dovesnap networks and starting containers in them. In the first network care, the user opted to have SSH keys installed in the SSH server containers and for their traffic to be mirrored. In the second network, only one container was started without adding any SSH keys. 

You can run Portunus again later to add additional resources to existing networks, such as adding more containers or adding ports. For example, if you have an interface on the host connected to an external VLAN, you can ask Portunus to add that port to a network. You can even start and connect libvirt-based VMs to Dovesnap networks. Cleaning up afterwards is even easier. Portunus takes care of removing any containers still running, taking Dovesnap networks down, and removing any other resources that were allocated (including Faucet and Open vSwitch (OVS), if desired).

What is Portunus orchestrating for us behind the scenes? Quite a lot! Starting at the lowest layer on the diagram below, Portunus (on the left) is controlling Dovesnap, which directs Docker to create Docker networks and then adds or removes resources to them. The switching within each Dovesnap network is done by OVS, with switching policy (including ACLs and mirroring) administered by Faucet. Dovesnap commands Faucet, though Portunus can also command Faucet (for example to change existing ACLs or mirroring options on a network that is up and running already).

FaucetConfRPC makes this reporting chain possible for multiple clients to command Faucet by presenting a network configuration remote procedure call (RPC) interface to Faucet. Dovesnap, Portunus, and potentially other applications like Poseidon, can then interface with FaucetConfRPC rather than Faucet directly. FaucetConfRPC includes RPCs that allow an application to add a new port to an existing VLAN, or add an ACL to a port, or ask for a port to be mirrored, without having to understand the entire network’s changing topology and configuration. FaucetConfRPC provides Python and Golang client libraries, uses gRPC as a transport, and provides mutual client/server certificate-based encryption and authentication. It even provides a command-line tool that allows network automation with simple DIY shell scripts. Since Faucet supports both virtual and hardware switches, FaucetConfRPC allows you to use the same API for both.

Everything above Faucet in the preceding diagram is strictly optional for the most basic system. But you would very likely want Portunus to provide graphing and monitoring via Gauge, Prometheus, and Grafana working together. Poseidon can also interact directly with Portunus-managed infrastructure. Poseidon receives a real time feed of network events from Faucet (information like which hosts are being learned and from where), and commands Faucet to mirror traffic to a Poseidon managed port. Then applications like NetworkML and Network Tools use that mirrored traffic to make decisions about what kinds of hosts are present and optionally decide how to change the network in response (for example, adding an ACL or rate limiting).

Portunus networks can become large and complex. Portunus was used recently to host a CTF exercise with containers with known vulnerabilities made up of tens of networks and hundreds of hosts. Even on a small experimental network, just keeping track of where all the resources are connected can be challenging. Fortunately, Portunus can ask Dovesnap to draw a network diagram, making it quick and easy to verify that your network is set up the way you expect. Here are some example outputs:

Today, Portunus provides the interactive terminal-style UI described at the beginning, and there is future work planned to allow scriptable configuration. This will make creating and re-creating environments with complex or simple topologies even quicker and easier. Additionally, enhancements to simplify required knowledge, such as knowing a unique ID for each switch, would improve the user experience.

While any project can always use improvement, after already using this tool to create a dozen simultaneous environments used by a few dozen folks across multiple physical machines to support over 100 containers and VMs over a 30-day period, we believe it’s ready for all of you to go give it a try and tell us what you think.