Implementing Multiple Cloud Storage APIs

OpenStack Summit Paris

The beauty of cloud storage APIs is that there are so many to choose from. Of course if you are implementing a cloud storage API for a customer to use, you don’t want to have to implement too many of these. When customers ask for support of a given API, can a vendor survive if they ignore these requests? A strategy many vendors are taking is to support multiple APIs with a single implementation. Besides the Swift API, many support the S3 defacto and CDMI standard APIs in their implementation. What is needed for these APIs to co-exist in an implementation? There are basic operations that are nearly identical between them, but what about semantics that have multiple different expressions such as metadata?

Mark Carlson, Alex McDonald and Glyn Bowden lead the discussion of this at the Paris summit.

SummitSlideFront

For the implementers of a cloud storage solution, it is not just the semantics of the APIs, but also the Authentication and Authorization mechanisms related to those APIs need to be supported as well. This is typically done by hosting the services that are required somewhere on the network and syncronizing them with a back end Directory service.

Multiple APIs

Swift leverages Keystone for authentication, and in order to support Swift Clients, you would need to run a Keystone instance on your Auth Server. If you want to support S3 clients, you need a service that is compatible with Signature Version 4 from Amazon. When creating a client, you might use a common library/proxy to insulate your code from the underlying semantic differences of these APIs. Jclouds is such a tool. The latest version of the CDMI API (version 1.1) has capability metadata (like a service catalog) that shows which Auth APIs any given cloud supports. This allows a CDMI Client to use Keystone, for example, as it’s auth mechanism while using the standard HTTP based storage operations and the advanced metadata standards from CDMI. To address the requirements for multiple APIs with the least amount of code duplication, there are some synergies that can be realized.

Storage Operations

CRUD – All pretty much determined by HTTP standard (common code)
Headers are API unique however (handle in API specific modules)

Security Operations

Client communication with Auth Server (API unique)
Multiple separate services running in Auth Server

Looking at two of the interfaces in particular, this chart shows the relationship of the Swift API model and that from the CDMI standard.

CDMISwift

When an object with a name that includes one or more “/“ characters is stored in a cloud, the model viewed via Swift and the view that CDMI shows are similar. Using CDMI, however, the client has access to additional capabilities to manage each level of “/“ containers and subcontainers. CDMI also standardizes a rich set of metadata that is understood and interpreted by the system implementing the cloud.

If you are looking for information that compares the Amazon S3 API with the CDMI standard one, there is a white paper available.

NewImage

The latest version of CDMI – http://www.snia.org/sites/default/files/CDMI_Spec_v1.1.pdf makes this even easier:

Spec text that explicitly forbid (in 1.0) functionality required for S3/Swift integration has been removed from the spec (“/”s may create intervening CDMI Containers)
Baseline operations (mostly governed by RFC 2616) now documented in Clause 6 (pgs. 28-35)
CDMI now uses content type to indicate CDMI-style operations (as opposed to X-CDMI-Specification-Version)
Specific authentication is no longer mandatory. CDMI implementations can now use S3 or Swift authentication exclusively, if desired.

CDMI 1.1 now includes a standard means of discovering what auth methods are available: cdmi_authentication_methods (Data System Metadata) 12.1.3 “If present, this capability contains a list of server-supported authentication methods that are supported by a domain. The following values for authentication method strings are defined:

• “anonymous”-Absence of authentication supported

• “basic”-HTTP basic authentication supported (RFC2617)

• “digest”-HTTP digest authentication supported (RFC2617)

• “krb5”-Kerberos authentication supported, using the Kerberos Domain specified in the CDMI domain (RFC 4559)

• “x509″-certificate-based authentication via TLS (RFC5246)”

The following values are examples of other widely used authentication methods that may be supported by a CDMI server:

“s3”-S3 API signed header authentication supported

“openstack”-OpenStack Identity API header authentication supported

Interoperability with these authentication methods are not defined by this international standard. Servers may include other authentication methods not included in the above list. In these cases, it is up to the CDMI client and CDMI server (implementations themselves) to ensure interoperability. When present, the cdmi_authentication_methods data system metadata shall be supported for all domains.

NewImage