DragonFlow SDN based Distributed Virtual Router for OpenStack Neutron

In my previous posts I presented a PoC implementation of an embedded L3 controller in Neutron that solves the network node bottleneck the SDN way, and overcomes some of the DVR limitations.

In this post I am going to cover the first release of DragonFlow - an SDN based Distributed Virtual Router L3 Service Plugin (aligned with OpenStack Kilo), now an official sub-project of Neutron.

The main purpose of DragonFlow is to simplify the management of the virtual router while improving performance, scale and eliminating single point of failure together with the network node bottleneck.

The DrangonFlow solution is based on separation of the routing control plane from the data plane. This is accomplished by implementing the routing logic in distributed forwarding rules on the virtual switches (called "flows" in OpenFlow terminology). To put this simply, the virtual router is implemented using OpenFlow flows only.

DragonFlow eliminates the use of software stack to act as a virtual router (the Linux network namespaces in the DVR and the legacy L3 architecture), it use purely OVS flow to act as a virtual router. A diagram showing DragonFlow components and overall architecture can be seen here:

DragonFlow High-level Architecture

DragonFlow Features for the Kilo Release:

• East-West traffic is fully distributed using direct flows, reactively installed upon VM-to-VM first connection.

• Support for all ML2 type drivers GRE/VXLAN/VLAN

• Support for centralized shared public network (SNAT) based on the legacy L3 implementation

• Support for centralized floating IP (DNAT) based on the legacy L3 implementation

• Support for HA, in case the connection to the Controller is lost, fall back to the legacy L3 implementation until recovery. Reused all the legacy L3 HA. (Controller HA will be supported in the next release).

Key Advantages:

• Performance improvements for inter-subnet network communication by removing the number of kernel layers (namespaces and their TCP stack overhead)
• Scalability improvements for inter-subnet network communication by offloading L3 East-West routing from the Network Node to all Compute Nodes
• Reliability improvements for inter-subnet network communication
• Simplified virtual routing management, mange only active flows and not all possible 
• Non Intrusive solution, does not rely on ML2 modification

How it works

1. On bootstrap the L3 service plugin sends an RPC message to the L2 service plugin, setting the L3  Controller Agent as the controller of the integration bridge.
2. The Controller queries the OVS  for its port configuration via Openflow, matches the actual ports configured on the OVS to the Neutron tenant networks data model. 
3. Then, it installs the bootstrap flow pipeline that offloads all L2 traffic and local subnet L3 traffic into the NORMAL pipeline, while sending every unmatched VM-to-VM inter-sunbet traffic to the controller.

DragonFlow L3 Service Bootstrap

The following diagram shows the multi-table OpenFlow pipeline installed onto the OVS integration bridge (br-int) in order to represent the virtual router using flows:

bootstrap flows pipeline

The base table pipe-line is installed proactively on bootstrap while the East-West rules on the L3 Forwarding table are installed reactively on each first VM-to-VM communication.

If you would like to try it yourself, install guide is available here.

To join the development effort: 

• Homepage: http://launchpad.net/dragonflow
• Source: http://git.openstack.org/cgit/stackforge/dragonflow
• Bugs: http://bugs.launchpad.net/dragonflow
• Blueprints - https://blueprints.launchpad.net/dragonflow

My next post will cover the L3 reactive OpenFlow application, and how we install the East-West reactive flows. 

Neutron DVR The SDN Way

In my previous posts I covered the existing Openstack virtual L3 implementations, from the base centralized implementation to the state of the art solution, DVR, that distributes the load among the computes nodes.

I also summarized the limitations of DVR and hopefully convinced you of the motivation for yet another L3 implementation in Neutron.

Just to quickly recap, the approach for DVR’s design was to take the existing centralized router implementation based on Linux network namespaces and clone it on all compute nodes. This is an evolutionary approach and a reasonable step to take in distributing L3 from which we've learned a lot. However, this solution is far form optimal.

In this post I’m going to present a PoC implementation of an embedded L3 controller in Neutron that solves the same problem (the network node bottleneck) the SDN way, and overcomes some of the DVR limitations.

I chose to start with just the East-West traffic, and leave the North-South for the next step.

I will share some of my ideas for North-South implementation in upcoming posts, as well as links to the Stackforge project.

If you're interested in joining the effort or take the code for a test drive, feel free to email me or leave a comment below.

SDN Controllers are generally perceived as huge pieces of complex code with hundreds of thousands of lines of code that try to handle everything.  It's natural that people would not deem such software to be capable of running "in line".  However, this is not necessarily so.  As I will demonstrate, the basic SDN Controller code can be separated and deployed in a lean and lightweight manner.

It occurred to me that by combining the well-defined abstraction layers already in Neutron (namely the split into br-tun, br-int and br-ext) and a lightweight SDN controller that can be embedded directly into Neutron, it will be possible to solve the network node bottleneck and the L3 high-availability problems in the virtual overlay network in a simple way.

Solution overview 

The proposed method is based on the separation of the routing control plane from the data plane. This is accomplished by implementing the routing logic in distributed forwarding rules on the virtual switches. In OpenFlow these rules are called flows. To put this simply, the virtual router is implemented in using OpenFlow flows. 
The functionality of the virtual routes that we should address in our solution is:

• L3 routing
• ARP response for the router ports
• OAM, like ICMP (ping) and others

L3 Controller embedded in Neutron POC

We decided to create our controller PoC with Openstack design tenets in mind. Specifically the following:

1. Scalability     - Support thousands of compute nodes
2. Elasticity       - Keep controllers stateless to allow for dynamic growth 
3. Performance  - Improve upon DVR
4. Reliability      - Highly available
5. Non intrusive - Rely on the existing abstraction and plug-able module

What we intend to prove is:

• We indeed simplify the DVR flows
• We reduce resource overhead (e.g. bridges, ports, namespaces)
• We remove existing bottlenecks (compared to L3Agent and DVR)
• We improve performance

We've started benchmarking the different models and I'll post results as soon as we have them ready.

Reactive vs. Proactive Mode

SDN defines two modes for managing a switch: reactive and proactive.
In our work, We decided to combine these two modes so that we could benefit from the advantages of both modes (although if you believe the FUD about the reactive mode performance, it is quite possible to enhance it to be purely proactive). To learn more about this mixed mode see my previous blog

The Proactive part

In the solution we install a flow pipeline in the OVS br-int in order to offload the handling of L2 and intra-subnet L3 traffic by forwarding these to the NORMAL path (utilizing the hybrid OpenFlow switch). This means that we reuse the built in mechanisms in Neutron for all L2 traffic (i.e. ML2 remains untouched and fully functional) and for L3 traffic that does not need routing (between IPs in the same tenant + subnet).

In addition we use the OVS OpenFlow extensions in order to install an ARP responder for every virtual router port. This is done to offload ARP responses to the compute nodes instead of replying from the controller.

The Reactive part

Out of the remaining traffic (i.e. inter-subnet L3 traffic) the only traffic that is handled in a reactive mode is the first packet of inter-subnet communications between VMs or traffic addressed directly to the routers' ports.

The Pipeline

Perhaps the most important part of the solution is the OpenFlow pipeline which we install into the integration bridge upon bootstrap.
This is the flow that controls all traffic in the OVS integration bridge (br-int).
The pipeline works in the following manner:

1. Classify the traffic
2. Forward to the appropriate element:
1. If it is ARP, forward to the ARP Responder table
2. If routing is required (L3), forward to the L3 Forwarding table (which implements a virtual router)
3. Otherwise, offload to NORMAL path

At the end of this post is a detailed explanation of the pipeline and implementation brief in case you're interested in the gory details.

In my following posts I will cover in detail the code modifications that were required to support the L3 controller as well as publish a performance study comparing the L3 Agent, DVR and the L3 controller for inter subnet traffic.

If you want to try it out, the code plus installation guide  are available here.

Again, if you'd like to join the effort, feel free to get in touch.

detailed explanation of the pipeline

The following diagram shows the multi-table OpenFlow pipeline installed into the OVS integration bridge (br-int) in order to represent the virtual router using flows only:

L3 Flows Pipeline

The following table describes the purpose of each of the pipeline tables:

ID	Table Name	Purpose
0	Metadata & Dispatch	Tag traffic with the appropriate metadata by input port, the value is the segmentation ID Input traffic from the tunnel bridge is offloaded to the NORMAL path All other traffic is sent to the "Classifier" table for classification
40	Classifier	All ARP traffic is sent to the ARP responder table All broadcast and multicast traffic is offloaded to the Normal path All L3 traffic is sent to the L3 forwarding table
51	ARP Responder	This table handles ARP responses for the virtual router interfaces. The vlan tag is removed from all other traffic and offloaded to the NORMAL path
52	L3 Forwarding	This table is used by the Controller to install flows to handle VM to VM inter subnet traffic (detailed explanation below) Traffic destined for a virtual router is sent to the controller (e.g. ping, packet_in, etc) All local subnet traffic is offloaded to the NORMAL path All other traffic is sent to the public network table
53	Public Network	will be described in following blogs

PoC Implementation brief

For the first release, we used the Open vSwitch (OVS), with OpenFlow v1.3 as the southbound protocol, using the RYU project implementation of the protocol stack as a base library.
The PoC support Route API for IPV4 East -West traffic.  In the current PoC 
implementation, the L3 controller is embedded into  the  L3 service plugin.  

Openstack Neutron DVR - Summary

Following my 3-post series about Openstack Juno DVR in detail, this post is a summary of DVR, in my point of view.

In the next post I’m going to present a POC implementation of an embedded L3 controller in Neutron that solves the problem differently. In this post I’d like to explain the motivation for yet another L3 implementation in Neutron.

I believe the L3 Agent drawbacks are pretty clear (centralized bottleneck, limited HA, etc), so in this post I’m going to cover the key benefits and drawbacks of DVR as I see them. So without further ado, let’s start.

Pros

First of all, from a functional point of view, DVR successfully distributed the East-West traffic and the DNAT floating IP, which provided significant offload from the Network node contention. This achieved two key benefits, first is that the failure domains are much smaller (Network node failure only affects SNAT) and the second is scalability due to the distribution of the load on all the compute nodes. 

Cons

The approach for DVR’s design was to take the existing centralized router implementation based on Linux network namespaces and clone it on all compute nodes. This is an evolutionary approach and an obvious step to take in distributing L3 from which we’ve learned a lot from. However, it adds load and complexity in three major areas: Management, Performance and Code.

To explain the technical details I will briefly cover two Linux networking constructs used in the solution for East-West communications:

1. Linux network namespaces (which internally load a complete TCP/IP stack with ARP tables and routing tables)

2. OVS flows

OVS flows were needed in order to block the local ARP, redirect return traffic directly to the VMs and to replace the router ports MAC (which cannot be accomplished reasonably using namespaces alone). On the flip side, OVS flows can easily accomplish everything that DVR uses the namespaces for (in East-West communications) and perhaps more importantly it does so in a more efficient way (avoids the overhead of the extra TCP/IP stack etc). To hint at our solution, flows also allow us to further improve the solution by selectively using a reactive approach where relevant.

Extra Flows Install for DVR

So overall it seems that using the Linux network namespace as a black box to emulate a router is an overkill. This approach makes sense in the centralized solution (L3 Agent) where for each tenant’s virtual router we use a single namespace in the DC, however, in the distributed approach where the number of namespaces in the DC is multiplied by the number of compute nodes this needs to be reevaluated.

If you’re not convinced yet, here is a summary of the implications of using this approach:

Resource consumption and Performance

1. DVR adds an additional namespace as well as virtual ports per router on all compute nodes that host routed VMs. This means additional TCP/IP stacks that each and every cross subnet packet traverses. This adds latency and host CPU consumption.

2. ARP tables on all namespaces in the Compute nodes are proactively pre-populated with all the possible entries. Whenever a VM is started, all compute nodes which have running VMs from the same tenant will be updated to keep these tables up to date (which potentially adds latency to VM start time)

3. Flows and routing rules are proactively installed on all compute nodes.

Management complexity

1. Multiple configuration points that need to be synchronized: namespaces, routing tables, flow tables, ARP tables and IPTables.

2. Namespaces on all compute nodes need to be kept in sync all the time with the tenant’s VM ARP tables and routing information and need to be tracked to handle failures etc.

3. The current DVR implementation does not support a reactive mode (i.e. creating a flow just-in-time), thus all possible flows are created on all hosts, even If they’re never used.

Code Complexity

1. The DVR required cross-component changes due to its multi configuration points: The Neutron server Data Model, ML2 Plugin, L3 Plugin , OVS L2 Agent and the L3 Agent.

2. Using all Linux networking constructs in a single solution (namespaces, flows, linux bridges, etc) requiring code that can manage all of it.

3. The solution is tightly coupled with the overlay manager (ML2) which means that the addition of every new type driver (today only vxlan) requires code additions on all levels (as can be seen by the vlan patches)

In my next post I will present an alternative solution, we evaluated and developed (code available) for distributing the virtual router that attempts to overcome these limitations, using SDN technologies.

Comments, questions and corrections are welcome.

3 Openstack Neutron Distributed Virtual Router (DVR) - Part 3 of 3

In this post, the last of a 3-post series about Openstack Juno DVR, I go into the North-South SNAT scenario in details.

Up until Juno, all L3 traffic was sent through the network node.  In Juno, DVR was introduced to distribute the load from the network node onto the compute nodes.

The L3 networking in Neutron is divided into 3 main services:

1. East-West communication: IP traffic between VMs in the data center
2. Floating IP (aka DNAT): The ability to provide a public IP to a VM, making it directly accessible from public networks (i.e. internet)
3. Shared IP (aka SNAT): The ability to provide public network access to VMs in the data center using a shared (public) IP address

In my previous posts, I covered how DVR distributes the the East-West L3 traffic and the DNAT North-South.

In this post I finish covering the North-South traffic with the Shared IP (SNAT).

SNAT Shared Gateway  North-South 

The SNAT North-South shared public gateway functionality was not distributed by DVR.  It remains centralized on the Network Node, as you can see in the diagram below.

In order to configure the SNAT namespaces on the Network node, the SNAT-DVR L3 agent is deployed.

An additional private address is assigned in the SNAT namespace for each of the DVR connected subnets, thus providing the centralized SNAT functionality.

Lets follow a communication flow from VM3 in the App network to the public network. 

1. The packet is forwarded to the DVR namespace via the local DVR default gateway port
2. If the packet destination is not private, then it is routed to the appropriate default public gateway port (based on the source subnet) on the SNAT namespace 
3. An OVS flow converts the local VLAN into the App network segmentation ID
4. The packet is forwarded into the SNAT namespace and NAT-ed using the SNAT public address

Let's look at a concrete example:

[Public Network range]=10.100.100.160/28
[App Network]=192.168.200.0/24
[Shared Public IP (SNAT)]=10.100.100.162
[VM3 private IP]=192.168.200.6

In my next post I will summarize the DVR solution in Openstack Juno.

Questions and comments are welcome.

The configuration files I used to set-up the Openstack  Neutron into DVR mode can be found here.

2 Openstack Neutron Distributed Virtual Router (DVR) - Part 2 of 3

In this post, the 2nd of a 3-post series about DVR, I go into the North-South DNAT scenario in details.

Up until Juno, all L3 traffic was sent through the network node.  In Juno, DVR was introduced to distribute the load from the network node onto the compute nodes.


The L3 networking in Neutron is divided into 3 main services:

1. East-West communication: IP traffic between VMs in the data center
2. Floating IP (aka DNAT): The ability to provide a public IP to a VM, making it directly accessible from public networks (i.e. internet)
3. Shared IP (aka SNAT): The ability to provide public network access to VMs in the data center using a shared (public) IP address

In my previous post, I covered how DVR distributes the the East-West L3 traffic. 
In this post I am going to begin covering the North-South traffic starting with  Floating IP (DNAT).

DNAT Floating IP North-South 

In order to support Juno's DVR local handling of floating IP DNAT traffic in the compute nodes, we now require an additional physical port that connects to the external network, on each compute node. 

The Floating IP functionality enable direct access from the public network (e.g. Internet) to a VM.

Let's follow the example below, where we will assign a floating IP to the web servers.  

When we associate a VM with a floating IP, the following actions take place:

1. The fip-<netid> namespace is created on the local compute node (if it does not yet exist)
2. A new port rfp-<portid> is created on the qrouter-<routerid> namespace (if it does not yet exist
3. The rfp port on the qrouter namespace is assigned the associated floating IP address
4. The fbr port on the fip namespace is created and linked via point-to-point network to the rfp port of the qrouter namespace
5. The fip namespace gateway port fg-<portid> is assigned an additional address from the public network range (the floating IP range)
6. The fg-<portid> is configured as a Proxy ARP

Now, lets take a closer look at VM4 (one of the web servers).

In the diagram below, the red dashed line shows the outbound network traffic flow from VM4 to the public network.

The flow goes through 5 steps:

1. The originating VM sends a packet via default gateway and the integration bridge forwards the traffic to the local DVR gateway port (qr-<portid>).
2. DVR routes the packet using the routing table to the rfp-<portid> port
3. The packet is applied NAT rule using IPTables, changing the source-IP of VM4 to the assigned floating IP, and then it is sent through the rfp-<portid> port, which connects to the fip namespace via point-to-point network (e.g. 169.254.31.28/31)
4. The packet is received on the fbr-<portid> port in the fip namespace and then routed outside through the fg-<portid> port

At this point you may be confused with the descriptions, so let's try to simplify this a bit with a concrete example:

[Public Network range]=10.100.100.160/28
[Web Network]=10.0.0.0/24
[VM4 floating IP]=10.100.100.163
[VM4 private IP]=10.0.0.6

As you can see in the diagram, routing consumes an additional IP from the public range per compute node (e.g. 10.100.100.164).


The reverse flow will go in the same route, the fg-<portid> act as a proxy ARP  for the DVR namespace.  

In the next post, I will go into the North-South scenario using Shared IP (SNAT).

Please feel free to leave comments, questions and corrections.

1 Openstack Neutron Distributed Virtual Router (DVR) - Part 1 of 3

In this 3-post series I cover how DVR works in (just about enough) detail. 

I assume you know how virtual routing is managed in Neutron. 

Up until Juno, all L3 traffic was sent through the network node, including even traffic between VMs residing on the same physical host.
This, of course, created a bottleneck, as well as a single point of failure (SPOF).

In Juno, DVR was introduced to overcome these problems.

As I described in my previous post, L3 networking in Neutron is divided into 3 main services:

1. East-West communication: IP traffic between VMs in the data center
2. Floating IP (aka DNAT): The ability to provide a public IP to a VM, making it directly accessible from public networks (i.e. internet)
3. Shared IP (aka SNAT): The ability to provide public network access to VMs in the data center using a shared (public) IP address

The main idea of DVR is to distribute the L3 Agent functionality to all the compute nodes, distributing the load and eliminating the SPOF.

DVR successfully distributes the first two services. 

However, SNAT distribution is not covered by DVR, and remains to be handled by hitting the Network Node in a centralized manner. 

The main challenges for SNAT distribution were maintaining the FWaaS 
statefulness and conserving IP addresses from the public network address pool (since supporting these will require each compute node to have a public IP address).

Neutron components with DVR entities are shown in red

DVR implements a virtual router element on every compute node using the Linux network namespace (similarly to L3 Agent virtual router). 

This is achieved by cloning the current centralized L3 implementation onto every compute node.
For each service, this is handled in a different manner.

Inter subnet routing East-West 

All the cross subnet traffic (between VMs of the same tenant) in DVR is now handled locally on the compute node using the router namespace.

A Linux namespace is created for every virtual router, on each compute node that hosts VMs that are connected to that router.

In order to configure these local DVR namespaces, the DVR L3 Agent (which used to be deployed only to the network node) is deployed onto all compute nodes as you can see in the diagram below.  

Now, extra OVS flows were required to enable the DVR functionality.  To do that, enhancements were applied to L2 OVS Agent.  These enhancements are described below.

In order to simplify management, the same IP and MAC addresses are reused on all the compute node DVRs, i.e. a single {IP, MAC} per virtual router port). 
Owing to this, ARP traffic for the virtual router ports is kept local to the compute node.

All the ARP entries for networks attached to each DVR are proactively populated by the L3 DVR Agent on all the cloned virtual router namespaces, i.e. on all the compute nodes that participate. 

In order to avoid having traffic from different physical hosts using the same MAC (which, as we said, we are reusing), each of the cloned DVRs is assigned a system-wide unique MAC address. 

I will reuse the example from my previous post: the popular cloud deployment 
3-tier web application design pattern (www-app-db).

In Horizon, we set up 3 subnets (in addition to the public) with one virtual router to bind them all.

On each compute node, the DVR namespace is pre-configured with the following:

• Port MAC and IP addresses
• Routing tables (default route per subnet and static routes)
• ARP table entries for all the VMs in the connected networks

In addition to the namespace, OVS flows are installed into the integration bridge and into the tunnel bridge to: 

• Isolate DVR broadcast domain to be local to the integration bridge  
• Translate the globally assigned MAC of the DVR into the local MAC address and vice versa
• Redirect traffic correctly per VM (new flows are installed for every new VM started on the compute node)

Realization of our topology on 2 Compute nodes

As we can see, VM1 on Web subnet communicates with VM3 on App subnet:

1. The packet reaches the subnet GW port on the local compute node 
• It is routed using the DVR namespace routing table into the destination subnet
• Linux namespace then performs the followings actions on the packet:
• Use the pre-populated ARP table to set the destination MAC address
• Set the source MAC to the local DVR GW port of the destination subnet
2. On the tunnel bridge the source MAC is replaced with the global DVR MAC address
3. The packet reaches the compute node that hosts the destination VM:
• The segmentation ID is replaced with the local vlan
• The packet is matched by the global source MAC address and forwarded to the integration bridge
4. The packet source MAC is replaced with the appropriate local DVR GW MAC and forwarded directly to the destination VM port

In the reverse flow from VM3 to VM1, packets are routed through the local DVR instance of the compute node that hosts VM3.

Note that the reverse flow goes in a different route (see the blue line). 

Due to this behavior, we cannot create stateful East-West rules, hence FWaaS for East-West traffic is not supported by DVR Juno.

In my next posts, I will go into the North-South scenario, covering the DNAT and SNAT services.