Multi-Site Management in OpenStack

Managing multiple Openstack clouds as a single resource pool

In this series of posts, we would be diving into "Tricircle" 
- an open source project that promises to give a "single pane of glass" management over multiple OpenStack environments, using a cascading approach.

As more and more companies are deploying OpenStack, it is becoming clear that there is a need to be able to manage multiple cloud installations. The reasons range from application lifecycle management, through spill-over scenarios and all the way to multi-site Disaster Recovery orchestration.

So why would one care to deploy the same service over multiple environments?

There are multiple reasons, and here are a few:

• Service continuity and geo-redundancy (in case of one site going awry)
• Geo-based load balancing and service locality (in case traffic comes from various places in the globe, or if there are strict quality/latency requirements, or if there are regulatory constraints, etc)
• Cost optimization (in case some resources are cheaper in another place)
• Growth (in case one environment cannot grow enough)
• Resource Utilization (in case you have multiple sites and want to aggregate their resources for better value or easier management)
• Single Configuration (instead of continually synchronizing multiple sub-instances of the service)
• ... (feel free to share additional incentives in the comments).

OpenStack Tricircle

Managing multiple OpenStack instances could be done in several ways, for example by introducing multi-site awareness into each project in OpenStack (which we ruled out due to complexity of evolving all OpenStack projects to do it).

The approach we took in Tricircle was to add a "Top" management OpenStack instance over multiple "Bottom" OpenStack instances.

The "Top" introduces a cascading service layer to delegate APIs downwards, and injects itself into several OpenStack components.

So, how does it feel to use such a "Top" OpenStack instance?

Well, first of all let's define the different users:

1. The Multi-site Tenant Admin (the "User") - Uses the multi-site OpenStack cloud (create VMs, networks, etc.)
2. The Multi-site Admin (the "Admin") - This user can add new sites, and needs to have the necessary credentials on them to put it together

User

For the "User", it is pretty straightforward: when you launch a VM, you get to choose from a list of Data Centers (a new drop box in Horizon), and then from a list of Availability Zones based on your Data Center selection, and that's basically it.

Admin


For the "Admin", you get a new "Add Site" API (in CLI only, at this point), and you need to have substantial knowledge about the "Bottom" sites you are adding (credentials and network-related information which we will cover in the next post).

Some High-Level Architecture

The "Top" Instance

The design principle we took was to reuse OpenStack components in the TOP and bottom layer as much as possible and to mange any OpenStack
deployment without any additional requirements (OpenStack API compatible).

For the top layer we used a non-modified OpenStack API layer to intercept operational requests and handle them in the cascading service.

Doing this required integrating with the different OpenStack core components:

Neutron 

• We introduced a custom core plugin
• Updates are written to the database 
• Operational requests are forwarded to our OpenStack Adaptor service
• Reads and Reports are served directly from the database

Nova

• We implemented a custom Nova Scheduler that runs inside our OpenStack Adaptor service
• We created a Compute Node emulation that runs in the Adaptor service and listens to the Nova Compute service queues
• Our current working assumption is to map "Bottom" sites as "Compute Nodes" that reside on different logical AZs
• The Compute Node emulation instance for each "Bottom" site also aggregates information and statistics that represent the site
• At some point, we plan to let the admin decide how to expose the "Bottom" sites, e.g. different AZs on the "Bottom" site, or all the actual "Compute Nodes", etc. This will create a complete decoupling between the Adaptor and the "Cascading" service.

The "Bottom" Instances

We assume that the "Bottom" sites are unmodified and potentially heterogeneous (in terms of network, configuration and version).

At this stage, our design assumes a centralized Keystone service running on the "Top" (we are planing "Federated Keystone" in the future).

In order to add a "Bottom" site to the multi-site environment, the admin needs to deploy a "Cascaded" service, and register it in the "Top" site, using a special "add site" API.  Then, configure the "Bottom" site to use the "Top" Keystone.

The Full Picture

Here is how the entire system looks:

Is it really that simple?

From the user experience point of view - We hope it is.  
But in order to get it there, we needed to handle quite a few obstacles:

• Resource Synchronization across the Multi-site
• Cross-site network
• Image synchronization
• Metadata synchronization (e.g. flavors)
• Resource Status monitoring and propagation (i.e. so that you can see what's happening from the "Top" dashboard) 

Coming next

In our coming posts, we will dive into resource synchronization, the cross-site networking, explain how we tackled the status and notification updates and share our approach to meeting large scale deployments.

Please share your thoughts about this in the comments.
We will be talking about this project in the upcoming OpenStack Tokyo summit, so if you're coming there, be sure to attend our talk.

To join the development effort: 

• Homepage: https://launchpad.net/tricircle
• Source: https://github.com/stackforge/tricircle 
• Blueprints - https://blueprints.launchpad.net/tricircle
• IRC Channel #openstack-tricircle