Voting for my talks in OpenStack Tokyo 2015

I've submitted several talks to OpenStack Tokyo 2015.

If you find anything here interesting, please vote for it.

(note: you need to have an account in openstack.org to do it)

My submitted talks need your votes:

Dragonflow – L3 Deep dive and hands on lab

https://www.openstack.org/summit/tokyo-2015/vote-for-speakers/presentation/3761     

OpenVSwitch Scale Performance secrets revealed – and open source solutions

https://www.openstack.org/summit/tokyo-2015/vote-for-speakers/presentation/3828 

Network high availability by design

https://www.openstack.org/summit/tokyo-2015/vote-for-speakers/presentation/3780

Scaling Neutron - Distributing Advanced Services using SDN

https://www.openstack.org/summit/tokyo-2015/vote-for-speakers/presentation/3788

Say Hello to 100G OpenStack Networking by Offloading SDN flows using DragonFlow and intelligent NICs (joint talk with Mellanox)

https://www.openstack.org/summit/tokyo-2015/vote-for-speakers/presentation/5020

Multi Site Openstack - Deep dive  (joint talk with Midokura)

https://www.openstack.org/summit/tokyo-2015/vote-for-speakers/presentation/5680

Thanks for your vote, and see you there.

DragonFlow SDN based Distributed Virtual Router for OpenStack Neutron

In my previous posts I presented a PoC implementation of an embedded L3 controller in Neutron that solves the network node bottleneck the SDN way, and overcomes some of the DVR limitations.

In this post I am going to cover the first release of DragonFlow - an SDN based Distributed Virtual Router L3 Service Plugin (aligned with OpenStack Kilo), now an official sub-project of Neutron.

The main purpose of DragonFlow is to simplify the management of the virtual router while improving performance, scale and eliminating single point of failure together with the network node bottleneck.

The DrangonFlow solution is based on separation of the routing control plane from the data plane. This is accomplished by implementing the routing logic in distributed forwarding rules on the virtual switches (called "flows" in OpenFlow terminology). To put this simply, the virtual router is implemented using OpenFlow flows only.

DragonFlow eliminates the use of software stack to act as a virtual router (the Linux network namespaces in the DVR and the legacy L3 architecture), it use purely OVS flow to act as a virtual router. A diagram showing DragonFlow components and overall architecture can be seen here:

DragonFlow High-level Architecture

DragonFlow Features for the Kilo Release:

• East-West traffic is fully distributed using direct flows, reactively installed upon VM-to-VM first connection.

• Support for all ML2 type drivers GRE/VXLAN/VLAN

• Support for centralized shared public network (SNAT) based on the legacy L3 implementation

• Support for centralized floating IP (DNAT) based on the legacy L3 implementation

• Support for HA, in case the connection to the Controller is lost, fall back to the legacy L3 implementation until recovery. Reused all the legacy L3 HA. (Controller HA will be supported in the next release).

Key Advantages:

• Performance improvements for inter-subnet network communication by removing the number of kernel layers (namespaces and their TCP stack overhead)
• Scalability improvements for inter-subnet network communication by offloading L3 East-West routing from the Network Node to all Compute Nodes
• Reliability improvements for inter-subnet network communication
• Simplified virtual routing management, mange only active flows and not all possible 
• Non Intrusive solution, does not rely on ML2 modification

How it works

1. On bootstrap the L3 service plugin sends an RPC message to the L2 service plugin, setting the L3  Controller Agent as the controller of the integration bridge.
2. The Controller queries the OVS  for its port configuration via Openflow, matches the actual ports configured on the OVS to the Neutron tenant networks data model. 
3. Then, it installs the bootstrap flow pipeline that offloads all L2 traffic and local subnet L3 traffic into the NORMAL pipeline, while sending every unmatched VM-to-VM inter-sunbet traffic to the controller.

DragonFlow L3 Service Bootstrap

The following diagram shows the multi-table OpenFlow pipeline installed onto the OVS integration bridge (br-int) in order to represent the virtual router using flows:

bootstrap flows pipeline

The base table pipe-line is installed proactively on bootstrap while the East-West rules on the L3 Forwarding table are installed reactively on each first VM-to-VM communication.

If you would like to try it yourself, install guide is available here.

To join the development effort: 

• Homepage: http://launchpad.net/dragonflow
• Source: http://git.openstack.org/cgit/stackforge/dragonflow
• Bugs: http://bugs.launchpad.net/dragonflow
• Blueprints - https://blueprints.launchpad.net/dragonflow

My next post will cover the L3 reactive OpenFlow application, and how we install the East-West reactive flows. 

Common Misconceptions about SDN Controller Management and Scalabilty

Here are a few common misconceptions about SDN controller management and scalability.

#1: Either Proactive mode or Reactive mode

When an SDN controller wants to enforce a set of rules or a policy on a forwarding element, it uses the southbound API, OVSDB, OpFlex, OpenFlow or others. In OpenFlow this process is called flow installation and it can be done using different methods: proactive, reactive or mixed (aka hybrid approach).  

Proactive

During bootstrap, the controller installs all flows and pipelines (multi-table entries) into all the forwarding elements.  The flows must cover all possible scenarios.

Whenever there is a change in the network, the controller removes or installs flows where necessary.

Reactive

When a packet arrives in a switch, a look-up is performed on the flow tables.
If there is no match and the switch is connected to a controller, it will forward the packet (either with or without its payload) to the controller.
In OpenFlow, this is called a PACKET_IN message.

It is also possible to create rules that forward the packet to the controller when matched and we call that...

Mixed Proactive / Reactive

With the mixed approach, you can benefit from the best of both worlds, achieving a balance between dynamic management and performance.

For frequently-used and rarely-modified flows, you install a pipeline and flows proactively.

For unmatched flows, or for flows that you want to handle in-line in your SDN application, you install a flow to forward the packet to the controller.

This mixed approach allows the controller to focus on making real-time dynamic decisions only on the traffic that requires it (reactive), while leaving the heavy-lifting of the majority of the traffic to the real-time forwarding element (proactive).

In addition, with this approach you can avoid over-provisioning the forwarding element with flows that are rare, thus dramatically reducing the number of entries in the flow tables. 

#2: Lack of Scalability and High Availability

In SDN, the control plane (i.e. the SDN Controller) is separated from the data plane (i.e. the Forwarding Elements).

Due to the centralized nature of control in SDN, we now need to support high availability and redundancy of the Control Plane.

In OpenFlow, the Forwarding Element (the Switch) connects to the controller via TCP or via TLS for secure channels.

Up until OpenFlow v1.2, whenever the connection to the controller was lost, the switch would lose the ability to forward PACKET_IN messages to the controller, thus having to drop all unmatched traffic or handle it in the NORMAL switch (only in switches that implemented the dual nature). 

This non-deterministic approach would yield unpredictable network behavior 
while the controller was unavailable.

OpenFlow v1.2 introduced the capability of working with multiple controllers.
The Master and Slave modes provide a mechanism for Active-Passive high availability, whereas the Equal mode provides an Active-Active model.

OpenFlow multiple controllers

With the multi-controller capability, we gained the control plane high availability we needed, improved reliability, fast recovery from failure and controller load balancing.

#3: You cannot design SDN Applications for Really Big Scale

Utilizing the two mechanisms I covered - Mixed-Mode and Multi-Controller - is the key to designing really big scale SDN applications.

If you take care to design your application to be stateless (whenever possible) and share nothing (or little) with other controller instances you can benefit from the Reactive mode where any instance can handle any flow.

In my opinion, the best approach for achieving minimal controller response latency and maximal bandwidth while keeping the dynamic allocation of flows, is by using all of these mechanisms together.

When you add to that the OVS OpenFlow extensions and the dual-nature Hybrid OpenFlow capability (which I covered in my previous post), you can really gain a dramatic performance and management boost.


In my next post I will demonstrate how we utilized these design guidelines and capabilities in a prototype for an alternative to Neutron's L3 Distributed Virtual Router.

Hybrid OpenFlow Switch

In my last post I summarized the DVR solution and tried to explain the motivation for yet another L3 implementation in Neutron that I am going to present in the coming posts.

This 2-post series is intended to cover basic SDN and OpenFlow mechanisms that we used in the L3 controller:

• Hybrid OpenFlow Switch 
• SDN models for managing the forwarding elements (switches)

Hybrid OpenFlow Switch

Hybrid OpenFlow switch was introduced in OpenFlow/1.1. Hybrid switches support both OpenFlow operation pipeline and normal (legacy) Ethernet switching functionality.

The hybrid switch allows forwarding of packets from the OpenFlow

pipeline to the normal pipeline through the NORMAL and FLOOD reserved ports.

The main reason for introducing the hybrid switch was to optimize the handling of operations like MAC learning, where a reactive approach was just not efficient - Doing MAC learning in the OpenFlow controller poses a significant cost in terms of network bandwidth and latency and does not scale for large networks.

The NORMAL action comes to the rescue and lets us offload legacy non-OpenFlow pipeline (like MAC learning mechanism, VLAN, ACL, QoS and other base features) to the forwarding element kernel module, which is optimized to handle such operations in near-line-rate.

But what happens when the NORMAL action is used in an OpenFlow flow?

Basically, what happens is that the traffic is redirected to a completely separated processing pipeline. This is illustrated in the diagram below.

OVS Hybrid OpenFlow Switch Pipelines

The OpenFlow pipeline and the Normal Pipeline, each act as a completely isolated switch.

There are, however, some issues with this hybrid approach.

• NORMAL pipeline is not standardized, so it behaves differently on switches from different vendors - There is a variance in the supported features, and no standard way to configure them
• NORMAL pipeline does not play well with some OpenFlow actions, for example if the port is tagged for the NORMAL pipeline (using the ovs-vsctrl), you can not tag it using OpenFlow actions and then forward it to the NORMAL path, because it will end-up dropped due to double tagging error.

The Open virtual Switch extensions to OpenFlow were developed to support these extra features using Flows, for example the LEARN action (Open vSwitch extension to OpenFlow) for MAC learning .

In my next post I will cover the SDN models for managing the forwarding elements in reactive, proactive and mix modes.

2 Openstack Neutron Distributed Virtual Router (DVR) - Part 2 of 3

In this post, the 2nd of a 3-post series about DVR, I go into the North-South DNAT scenario in details.

Up until Juno, all L3 traffic was sent through the network node.  In Juno, DVR was introduced to distribute the load from the network node onto the compute nodes.


The L3 networking in Neutron is divided into 3 main services:

1. East-West communication: IP traffic between VMs in the data center
2. Floating IP (aka DNAT): The ability to provide a public IP to a VM, making it directly accessible from public networks (i.e. internet)
3. Shared IP (aka SNAT): The ability to provide public network access to VMs in the data center using a shared (public) IP address

In my previous post, I covered how DVR distributes the the East-West L3 traffic. 
In this post I am going to begin covering the North-South traffic starting with  Floating IP (DNAT).

DNAT Floating IP North-South 

In order to support Juno's DVR local handling of floating IP DNAT traffic in the compute nodes, we now require an additional physical port that connects to the external network, on each compute node. 

The Floating IP functionality enable direct access from the public network (e.g. Internet) to a VM.

Let's follow the example below, where we will assign a floating IP to the web servers.  

When we associate a VM with a floating IP, the following actions take place:

1. The fip-<netid> namespace is created on the local compute node (if it does not yet exist)
2. A new port rfp-<portid> is created on the qrouter-<routerid> namespace (if it does not yet exist
3. The rfp port on the qrouter namespace is assigned the associated floating IP address
4. The fbr port on the fip namespace is created and linked via point-to-point network to the rfp port of the qrouter namespace
5. The fip namespace gateway port fg-<portid> is assigned an additional address from the public network range (the floating IP range)
6. The fg-<portid> is configured as a Proxy ARP

Now, lets take a closer look at VM4 (one of the web servers).

In the diagram below, the red dashed line shows the outbound network traffic flow from VM4 to the public network.

The flow goes through 5 steps:

1. The originating VM sends a packet via default gateway and the integration bridge forwards the traffic to the local DVR gateway port (qr-<portid>).
2. DVR routes the packet using the routing table to the rfp-<portid> port
3. The packet is applied NAT rule using IPTables, changing the source-IP of VM4 to the assigned floating IP, and then it is sent through the rfp-<portid> port, which connects to the fip namespace via point-to-point network (e.g. 169.254.31.28/31)
4. The packet is received on the fbr-<portid> port in the fip namespace and then routed outside through the fg-<portid> port

At this point you may be confused with the descriptions, so let's try to simplify this a bit with a concrete example:

[Public Network range]=10.100.100.160/28
[Web Network]=10.0.0.0/24
[VM4 floating IP]=10.100.100.163
[VM4 private IP]=10.0.0.6

As you can see in the diagram, routing consumes an additional IP from the public range per compute node (e.g. 10.100.100.164).


The reverse flow will go in the same route, the fg-<portid> act as a proxy ARP  for the DVR namespace.  

In the next post, I will go into the North-South scenario using Shared IP (SNAT).

Please feel free to leave comments, questions and corrections.