3 Openstack Neutron Distributed Virtual Router (DVR) - Part 3 of 3

In this post, the last of a 3-post series about Openstack Juno DVR, I go into the North-South SNAT scenario in details.

Up until Juno, all L3 traffic was sent through the network node.  In Juno, DVR was introduced to distribute the load from the network node onto the compute nodes.

The L3 networking in Neutron is divided into 3 main services:

1. East-West communication: IP traffic between VMs in the data center
2. Floating IP (aka DNAT): The ability to provide a public IP to a VM, making it directly accessible from public networks (i.e. internet)
3. Shared IP (aka SNAT): The ability to provide public network access to VMs in the data center using a shared (public) IP address

In my previous posts, I covered how DVR distributes the the East-West L3 traffic and the DNAT North-South.

In this post I finish covering the North-South traffic with the Shared IP (SNAT).

SNAT Shared Gateway  North-South 

The SNAT North-South shared public gateway functionality was not distributed by DVR.  It remains centralized on the Network Node, as you can see in the diagram below.

In order to configure the SNAT namespaces on the Network node, the SNAT-DVR L3 agent is deployed.

An additional private address is assigned in the SNAT namespace for each of the DVR connected subnets, thus providing the centralized SNAT functionality.

Lets follow a communication flow from VM3 in the App network to the public network. 

1. The packet is forwarded to the DVR namespace via the local DVR default gateway port
2. If the packet destination is not private, then it is routed to the appropriate default public gateway port (based on the source subnet) on the SNAT namespace 
3. An OVS flow converts the local VLAN into the App network segmentation ID
4. The packet is forwarded into the SNAT namespace and NAT-ed using the SNAT public address

Let's look at a concrete example:

[Public Network range]=10.100.100.160/28
[App Network]=192.168.200.0/24
[Shared Public IP (SNAT)]=10.100.100.162
[VM3 private IP]=192.168.200.6

In my next post I will summarize the DVR solution in Openstack Juno.

Questions and comments are welcome.

The configuration files I used to set-up the Openstack  Neutron into DVR mode can be found here.

Layer-3 Services in OpenStack Neutron Pre Juno

  

In this multi-part blog series I intend to dive into the L3 Services in Neutron Openstack.  

Neutron defines an API extension that allows Administrators and tenants to create virtual routers.  The purpose of those is to connect several virtual L2 network subnets (which are defined using some other Neutron APIs).
Another API defined by Neutron and implemented in the L3 service is the Floating IP extension that provides public connectivity, whether directly to a VM (aka DNAT) or via shared gateway (aka SNAT).

The L3 Services in Neutron also handle the optional Extra Route extension API.

Neutron Virtual Router

In the above diagram we can see the following:

• Yellow: Inter subnet routing (East/West)

• Cyan: SNAT (port mapping and masquerading the IP address)

• Red: DNAT (floating IPs, public N/S connectivity directly to VM)

• Static routes (Extra Routes), defined inside the virtual router

Now, let's look at how everything gets wired when the system loads.

1. The L3 Service Plugin loads inside the Neutron server (which usually runs on the Controller Node).  It handles the layer 3 RESTful APIs and the data access layer. 

L3 Services API

2. The L3 Agent usually runs on the Network Node.  When started, it registers itself on the Neutron L3 Service Plugin Router Scheduler, via the Message Queue.  It implements the virtual router functionality defined by the Neutron API. 

3. Each L3 Agent locally manages virtual routers that are assigned to it, based on configuration data it receives on the Message Queue from the L3 Service Plugin

4. Openstack Neutron reference implementation of the L3 Agent uses  the Linux network namespaces and IPTables rules to implement the virtual routers.

5. The Linux network namespaces provide an isolated network stack with local routing table and IPTables rules.  It enables reuse of IP addresses with scope limited to the namespace. 

6. The L3 Agent creates a network namespace for each virtual router in the tenant network.  Then, it creates all the virtual router ports inside the namespace.

7. Each virtual router port in the namespace is tagged in the integration bridge (aka br-int) with the local segmentation ID of the subnet.

8. Each virtual router namespace includes a gateway port to the external network (aka br-ex).

9. DNAT (Floating IP) and SNAT are implemented using IPTables rules that are applied on the gateway port.

Let's take the popular cloud deployment 3-tier web application design pattern (www-app-db) as an example .  
In Horizon, we set up 3 subnets (in addition to the external) with one virtual router to bind them all.

• The virtual router is realized by the L3 Agent using a namespace
• For each connected network a port is created on the br-int within the namespace
• Each port is tagged with a local VLAN ID
• Local VLAN IDs are mapped to the Segmentation ID by the Tunnel Bridge (aka br-tun)

For the most part, this solution works well.  However, there is an inherent limitation that affects the overall system performance and scalability:

All cross-subnet traffic hits the Network Node. 

  

The problem becomes more apparent in large deployments.  The Network node quickly becomes a bottleneck in the system.

In Juno release, a solution to this problem was introduced.  
It was called DVR (Distributed Virtual Router).
The main idea was to distribute L3 Agents functionality onto the Compute nodes.

In my next post, I will describe the DVR solution.