CDMI – Page 2 – SNIA on Cloud Storage

See SNIA at OpenStack Summit Tokyo

October 6, 2015October 6, 2015 mac Leave a comment

Are you headed to the OpenStack Summit in Tokyo later this month? If so, I encourage you to stop by two “Birds of a Feather” (BoF) sessions I’ll be hosting on behalf of SNIA. Here’s the info on both of them:

Extending OpenStack Swift with S3 and CDMI Interfaces – Tues. Oct. 27^th 11:15 a.m.

Cloud application developers using the OpenStack infrastructure are demanding implementations of not just the Swift API, but also the S3 defacto and CDMI standard APIs. Each of these APIs not only offers features in common, but also offers what appear to be unique and incompatible facilities. At this BoF, we’ll discuss how to: Implement a multi-API strategy simply and effectively, sensibly manage the differences between each of the APIs, map common features to each other, take advantage of each of the APIs’ strengths, avoid lowest common denominator implementations

Object Drive Integration with Swift – Thurs. Oct. 29^th 9:00 a.m.

With the emergence of disk drives and perhaps solid state drives with Key/Value and other object interfaces, what are the implications on solution architectures and systems built around OpenStack Swift. One approach is termed “PACO” where the Object Node speaks Key/Value to the drive and is hosted with other Swift Services. Are there other approaches to this? Are you developing products or solutions based on Object Drives? Come to this BoF to discuss these issues with fellow developers.

I expect both of these BoFs will be full of lively discussions around standards, emerging technologies, challenges, best practices and more. If you have any questions about these sessions or about work that SNIA is doing, do not hesitate to contact me. I hope to see you in Tokyo!

Cloud Storage Development Challenges – An SDC Preview

July 27, 2015July 27, 2015 Alex McDonald Leave a comment

This year’s Storage Developer Conference (SDC) is expected to draw over 400 storage developers and professionals. On August 4^th, you can get a sneak preview of key cloud topics that will be covered at SDC in this live Webcast where David Slik and Mark Carlson Co-Chairs of the SNIA Cloud Technical Work Group, together with Yong Chen, Assistant Professor at Texas Tech University will discuss:

Mobile and Secure – Cloud Encrypted Objects using CDMI
Object Drives: A new Architectural Partitioning
Unistore: A Unified Storage Architecture for Cloud Computing
Using CDMI to Manage Swift, S3, and Ceph Object Repositories

You’ll learn how encrypted objects can be stored, retrieved, and transferred between clouds, how Object Drives allow storage to scale up and down by single drive increments, end-user and vendor use cases of the Cloud Data Management Interface (CDMI), and we’ll introduce Unistore – an innovative unified storage architecture that efficiently integrates heterogeneous HDD and SCM devices for Cloud storage systems.

I’ll be moderating the discussion among this expert panel. It should be an enlightening and lively hour. I hope you’ll register now to join us.

Swift, S3 or CDMI – Your Questions Answered

May 13, 2015May 13, 2015 mac 1 Comment

Last week’s live SNIA Cloud Webcast “Swift, S3 or CDMI – Why Choose?” is now available on demand. Thanks to all the folks who attended the live event. We had some great questions from attendees, in case you missed it, here is a complete Q&A.

Q. How do you tag the data? Is that a manual operation?

A. The data is tagged as part of the CDMI API by supplying key value pairs in the JSON Object. Since it is an API you can put a User Interface in front of it to manually tag the data. But you can also develop software to automatically tag the data. We envision an entire ecosystem of software that would use this interface to better manage data in the future

Q. Which vendors support CDMI today?

A. We have a page that lists all the publically announced CDMI implementations here. We also plan to start testing implementations with standardized tests to certify them as conformant. This will be a separate list.

Q. FC3 Common Services layer vs. SWIFT, S3, & CDMI – Will it fully integrate with encryption at rest vendors?

A. Amazon does offer encryption at rest for example, but does not (yet) allow you choose the algorithm. CDMI allows vendors to show a list of algorithms and pick the one they want.

Q. You’d mentioned NFS, other interfaces for compatibility – but often “native” NFS deployments can be pretty high performance. Object storage doesn’t really focus on performance, does it? How is it addressed for customers moving to the object model?

A. CDMI implementations are responsible for the performance not the standard itself, but there is nothing in an object interface that would make it inherently slower. But if the NFS interface implementation is faster, customers can use that interface for apps with those performance needs. The compatibility means they can use whatever interface makes sense for each application type.

Q. Is it possible to query the user-metadata on a container level for listing all the data objects that have that user-metadata set?

A. Yes. Metadata query is key and it can be scoped however you like. Data system metadata is also hierarchical and inherited – meaning that you can override the parent container settings.

Q. So would it be reasonable to say that any current object storage should be expected to implement one or more of these metadata models? What if the object store wasn’t necessarily meant to play in a cloud? Would it be at a disadvantage if its metadata model was proprietary?

A. Yes, but as an add-on that would not interfere with the existing API/access method. Eventually as CDMI becomes ubiquitous, products would be at a disadvantage if they did not add this type of interface.

Securely Sharing Health Care Data across Different Cloud Services

April 21, 2015April 21, 2015 mac 1 Comment

As more and more health care providers leverage the efficiencies of the cloud, the need to share health care data across different cloud services arises. Sharing health care data across cloud services must ensure the confidentiality, integrity, and availability of the health data and preserve the privacy of the patients in such a way that revealing the data to other data requestors is performed only with patient consent.

The Cloud Data Management Interface (CDMI) international standard is a protocol that has been standardized by SNIA to create interoperable data management services in cloud storage.

The Cloud Storage TWG has just released a technical white paper, “Towards a CDMI Health Care Profile,” that explores the capabilities of CDMI in addressing these requirements, and provides suggestions for possible extensions that are appropriate for a health care profile.

I encourage you to download this paper to learn:

Motivations for protecting health data
Health data protection requirements
A use case that promotes the deployment of health data protection
Requirements and implementation aspects of the use case
Use case architecture
Future use cases roadmap

I hope you’ll find this paper enlightening and welcome feedback and comments on its content here in this blog.

What’s New in the CDMI 1.1 Cloud Storage Standard

November 17, 2014November 17, 2014 Alex McDonald Leave a comment

On December 2, 2014, the CSI is hosting a Developer Tutorial Webcast “Introducing CDMI 1.1” to dive into all the capabilities of CDMI 1.1.

Register now to join David Slik, Co-Chair, SNIA Cloud Storage Technical Work Group and me, Alex McDonald, as we’ll explore what’s in this major new release of the CDMI standard, with highlights on what you need to know when moving from CDMI 1.0.2 to CDMI 1.1.

The latest release – CDMI 1.1 — includes:

Enabling support for other popular industry supported cloud storage protocols such as OpenStack Swift and Amazon S3
A variety of extensions, some part of the core specification and some stand-alone, that include a CIMI standard extension, support for immediate queries , an LTFS Export extension, an OVF extension, along with multi-part MIME and versioning extensions. A full list can be found here.
100% backwards compatibility with ISO certified CDMI v. 1.0.2 to ensure continuity and backward compatibility with existing CDMI systems
And more

This event on December 2^nd will be live, so please bring your specific questions. We’ll do our best to answer them on the spot. I hope you’ll join us!

Implementing Multiple Cloud Storage APIs

November 13, 2014November 13, 2014 mac Leave a comment

OpenStack Summit Paris

The beauty of cloud storage APIs is that there are so many to choose from. Of course if you are implementing a cloud storage API for a customer to use, you don’t want to have to implement too many of these. When customers ask for support of a given API, can a vendor survive if they ignore these requests? A strategy many vendors are taking is to support multiple APIs with a single implementation. Besides the Swift API, many support the S3 defacto and CDMI standard APIs in their implementation. What is needed for these APIs to co-exist in an implementation? There are basic operations that are nearly identical between them, but what about semantics that have multiple different expressions such as metadata?

Mark Carlson, Alex McDonald and Glyn Bowden lead the discussion of this at the Paris summit.

SummitSlideFront

For the implementers of a cloud storage solution, it is not just the semantics of the APIs, but also the Authentication and Authorization mechanisms related to those APIs need to be supported as well. This is typically done by hosting the services that are required somewhere on the network and syncronizing them with a back end Directory service.

Multiple APIs

Swift leverages Keystone for authentication, and in order to support Swift Clients, you would need to run a Keystone instance on your Auth Server. If you want to support S3 clients, you need a service that is compatible with Signature Version 4 from Amazon. When creating a client, you might use a common library/proxy to insulate your code from the underlying semantic differences of these APIs. Jclouds is such a tool. The latest version of the CDMI API (version 1.1) has capability metadata (like a service catalog) that shows which Auth APIs any given cloud supports. This allows a CDMI Client to use Keystone, for example, as it’s auth mechanism while using the standard HTTP based storage operations and the advanced metadata standards from CDMI. To address the requirements for multiple APIs with the least amount of code duplication, there are some synergies that can be realized.

Storage Operations

CRUD – All pretty much determined by HTTP standard (common code)
Headers are API unique however (handle in API specific modules)

Security Operations

Client communication with Auth Server (API unique)
Multiple separate services running in Auth Server

Looking at two of the interfaces in particular, this chart shows the relationship of the Swift API model and that from the CDMI standard.

CDMISwift

When an object with a name that includes one or more “/“ characters is stored in a cloud, the model viewed via Swift and the view that CDMI shows are similar. Using CDMI, however, the client has access to additional capabilities to manage each level of “/“ containers and subcontainers. CDMI also standardizes a rich set of metadata that is understood and interpreted by the system implementing the cloud.

If you are looking for information that compares the Amazon S3 API with the CDMI standard one, there is a white paper available.

NewImage

The latest version of CDMI – http://www.snia.org/sites/default/files/CDMI_Spec_v1.1.pdf makes this even easier:

Spec text that explicitly forbid (in 1.0) functionality required for S3/Swift integration has been removed from the spec (“/”s may create intervening CDMI Containers)
Baseline operations (mostly governed by RFC 2616) now documented in Clause 6 (pgs. 28-35)
CDMI now uses content type to indicate CDMI-style operations (as opposed to X-CDMI-Specification-Version)
Specific authentication is no longer mandatory. CDMI implementations can now use S3 or Swift authentication exclusively, if desired.

CDMI 1.1 now includes a standard means of discovering what auth methods are available: cdmi_authentication_methods (Data System Metadata) 12.1.3 “If present, this capability contains a list of server-supported authentication methods that are supported by a domain. The following values for authentication method strings are defined:

• “anonymous”-Absence of authentication supported

• “basic”-HTTP basic authentication supported (RFC2617)

• “digest”-HTTP digest authentication supported (RFC2617)

• “krb5”-Kerberos authentication supported, using the Kerberos Domain specified in the CDMI domain (RFC 4559)

• “x509″-certificate-based authentication via TLS (RFC5246)”

The following values are examples of other widely used authentication methods that may be supported by a CDMI server:

“s3”-S3 API signed header authentication supported

“openstack”-OpenStack Identity API header authentication supported

Interoperability with these authentication methods are not defined by this international standard. Servers may include other authentication methods not included in the above list. In these cases, it is up to the CDMI client and CDMI server (implementations themselves) to ensure interoperability. When present, the cdmi_authentication_methods data system metadata shall be supported for all domains.

NewImage

Other resources that are available for developers include:

CDMI for S3 Developers

http://snia.org/sites/default/files/S3-like_CDMI_v1.0.pdf

Comparison of S3/Swift functions

https://wiki.openstack.org/wiki/Swift/APIFeatureComparison
Somewhat dated – needs updating

Implementation of CDMI filter driver for Swift

https://github.com/osaddon/cdmi
Needs further development

Implementation of S3 filter driver for Swift

https://github.com/stackforge/swift3
Good community maintenance

For the slides from the talk, the site snia.org/cloud has the slideshare and .pdf links.

What the CSI is Up to at SDC

September 9, 2014September 9, 2014 Alex McDonald Leave a comment

What the Cloud Storage Initiative Is Doing At SDC

The SNIA Storage Developer Conference (SDC) is less than a week away. We’re looking forward to the conference and in particular want to make note of some exciting news and events that pertain to work the CSI is doing to promote standards that will increase the adoption, interoperability and portability of data stored in the cloud.

SDC Conference session: Introducing CDMI v1.1 – Tuesday, September 16^th, 1:00 p.m. by David Silk. This session introduces the new CDMI 1.1 and provides an overview of capabilities the Technical Work Group have added to the standard, and what CDMI implementers need to know when moving from CDMI 1.0.2 to CDMI 1.1.

Cloud Interoperability Plugfest – Participants at the 12th Cloud Interoperability Plugfest will be testing the interoperability of their cloud storage interfaces based on CDMI. We always have a large showing of CDMI implementations at this event, but are also looking for implementations of Amazon S3, and OpenStack Swift, Cinder and Manila interfaces.

It’s not too late to register for this Plugfest. Find out how here.

SDC 2014 is going to be exciting and educational. It’s “one stop shopping” for IT professionals who focus on the tools, technologies and developments needed for understanding and implementing efficient data storage, management and security. The CSI hopes to see you there.

Getting Started with the CDMI Conformance Test Program

May 21, 2014May 21, 2014 Alex McDonald Leave a comment

Together with our partner, TATA Consultancy Services, we recently had a great live Webcast to launch the Conformance Test Program (CTP) for the SNIA Cloud Data Management Interface (CDMI). CDMI is an ISO/IEC standard that offers end users simplicity and data storage interoperability across a wide range of cloud solutions. Interoperability and portability of data stored in the cloud has become a top IT priority. The CTP tests for conformance against the specification, and provides purchasers of certified cloud storage solutions the assurance that these solutions meet CDMI interoperability standards. Our Webcast is now available on demand. It details the benefits of the CDMI CTP program and explains how any cloud storage vendor or ISV can begin the CTP process. I encourage you to check it out to learn:

Key benefits of the CDMI standard for vendors and end users
Growing adoption of the CDMI standard
The suite of conformance tests required to achieve CDMI CTP certification
How to begin the CTP process

In addition to the Webcast replay, I encourage you to check out our CDMI CTP Frequently Asked Questions (FAQ). Getting started is easy. Just fill out the CTP form and you’ll be on your way.