During the recent OpenStack Tokyo 2015 summit, we introduced a new project that we've been working on recently, which will provide an Application Data Protection as a Service (video) (slides).
What is Smaug?
Not to be confused with Application Security or DLP, Smaug deals with protecting the Data that comprises an OpenStack-deployed Application (what is referred to as "Project" in Keystone terminology) against loss/damage (e.g. backup, replication).
It does that by providing a standard framework of APIs and services that enables vendors to introduce various data protection services into a coherent and unified flow for the user.
We named it Smaug after the famous dragon from J.R.R. Tolkien’s
“The Hobbit”, which was known to guard the treasures of the kingdom of Erebor, as well as have specific knowledge on every item in its hoard. Unlike its namesake, our Smaug is designed to give a simple and user-friendly experience, and not burn a user to a crisp when they want to recover a protected item.
The main concept behind Smaug is to provide protection of an entire OpenStack project, across OpenStack sites (or with a single local site).
Lets take a typical 3-tier cloud app:
In order to fully protect such a deployment (e.g. Disaster Recovery), we would have to protect many resources, which have some dependency
between them.
The following diagram shows how such a dependency tree might look:
The following diagram shows how such a dependency tree might look:
In Smaug, we defined a plugin engine that loads a protection plugin for each resource type.
Then, we let the user create a Protection Plan, which consists of all the resources she wants to protect.
These
resources can be divided into groups, each of which is handled by a
different plugin in Smaug:
- Volume - Typically, a block of data that is mapped/attached to the VM and used for reading/writing
- VM - A deployed workload unit, usually comprised of some metadata (configuration, preferences) and connected resources (dependencies)
- Virtual Network - The virtual network overlay where the VM runs
- Project - A group of VMs and their shared resources (e.g. networks, volumes, images, etc.)
- Image - A software distribution package that is used to launch a VM
Smaug Highlights
Open
Architecture
In Smaug, we defined a plugin engine that loads a protection plugin for each resource type.
Then, we let the user create a Protection Plan, which consists of all the resources she wants to protect.
- Volume - Typically, a block of data that is mapped/attached to the VM and used for reading/writing
- VM - A deployed workload unit, usually comprised of some metadata (configuration, preferences) and connected resources (dependencies)
- Virtual Network - The virtual network overlay where the VM runs
- Project - A group of VMs and their shared resources (e.g. networks, volumes, images, etc.)
- Image - A software distribution package that is used to launch a VM
Smaug Highlights
Open Architecture
Vendors create plugins that
implement Protection mechanisms for different OpenStack resources.
User perspective: Protect Application Deployment
Users configure and manage custom
protection plans on the deployed resources (topology, VMs, volumes, images, …).
The user selects a "Protection Provider" from a selection of available Protection Providers, which is maintained and managed by the admin.
Admin perspective: Configure Protection Providers
The Admin defines which Protection Providers are available to the users.
A "Protection Provider" is basically a bundle of per-resource protection plugins and a bank, which are curated from the total available protection plugins and bank plugins.
In addition, the Admin configures a Bank Account for each user (tenant).
A "Protection Provider" is basically a bundle of per-resource protection plugins and a bank, which are curated from the total available protection plugins and bank plugins.
In addition, the Admin configures a Bank Account for each user (tenant).
Smaug APIs
We are currently in the process of defining Smaug's set of User Service APIs:
Resource (Protectable) API
Enables the Smaug user to access information about which resource types are protectable (i.e. can be protected by Smaug).
In addition, enables the user to get additional information on each resource type, such as a list of actual instances and their dependencies.
In addition, enables the user to get additional information on each resource type, such as a list of actual instances and their dependencies.
Plan API
Enables the Smaug user to create or edit Protection
Plans using a selected Protection Provider, as well as access all the parameters of the plan.
Provider API
Enables the Smaug user to list available providers and get parameters and result schema super-set for all plugins of a specific Provider.
Checkpoints API
Enables the Smaug user to access and manage Protection Checkpoints, as well as listing and querying of the existing Checkpoints in a Provider.In addition, provides Checkpoint Read Access to the Restore API, when recovering a protected application's data.
Calling the Checkpoint Create (POST) API will start a protection process that will create a Vault in the user's Bank Account, on the Bank that is assigned to the Provider.
The process will then pass the Vault on a call to the Protect action on each of the Protection Plugins assigned to the Provider, so each will write its metadata into the Vault.
It is left up to the Plugin implementation to decide where to store the actual data (i.e. in the Vault or somewhere else).
Calling the Checkpoint Create (POST) API will start a protection process that will create a Vault in the user's Bank Account, on the Bank that is assigned to the Provider.
The process will then pass the Vault on a call to the Protect action on each of the Protection Plugins assigned to the Provider, so each will write its metadata into the Vault.
It is left up to the Plugin implementation to decide where to store the actual data (i.e. in the Vault or somewhere else).
Schedule Operation API
Enables the Smaug user to create a mapping between a Trigger and Operation(s) definitions .Smaug Architecture
We defined three services for Smaug:
Smaug API service
These top-level north-bound APIs expose Application Data
Protection services to the Smaug user.
The purpose of the services is to maximize flexibility and accommodate
for (hopefully) any kind of protection for any type of resource, whether it is
a basic OpenStack resource (such as a VM, Volume, Image, etc.) or some
ancillary resource within an application system that is not managed in
OpenStack (such as a hardware device, an external database, etc.).
Smaug Schedule Service
This subsystem is responsible for scheduling and orchestrating
the execution of Protection Plans.
The implementation can be replaced by any other external
solution.
All actual Protection-related activities are managed via the
Operation northbound APIs, in order to support:
- Record maintaining of all operations in the Smaug database (to drive Operation Status APIs)
- Decoupling the implementation of the Scheduler from the implementation of the Protection Service
Smaug Protection Service
This subsystem is responsible for handling the following
tasks:
- Operation Execution
- protectable(resources) plugin management
- Protection provider management
- Protection Plugin management
- Bank Plugin management
- Bank checkpoints sub-service
Join Smaug
We are currently in the process of reviewing the API definition
- Our IRC (we are always there): #openstack-smaug
- Bi-Weekly IRC meeting at 1400 UTC (09:00 EST, 22:00 CST) every (even) Tuesday in #openstack-meeting
- Launchpad : http://launchpad.net/smaug
