Cloud Storage Initiative – Page 3 – SNIA on Cloud Storage

Swift, S3 or CDMI – Your Questions Answered

May 13, 2015May 13, 2015 mac 1 Comment

Last week’s live SNIA Cloud Webcast “Swift, S3 or CDMI – Why Choose?” is now available on demand. Thanks to all the folks who attended the live event. We had some great questions from attendees, in case you missed it, here is a complete Q&A.

Q. How do you tag the data? Is that a manual operation?

A. The data is tagged as part of the CDMI API by supplying key value pairs in the JSON Object. Since it is an API you can put a User Interface in front of it to manually tag the data. But you can also develop software to automatically tag the data. We envision an entire ecosystem of software that would use this interface to better manage data in the future

Q. Which vendors support CDMI today?

A. We have a page that lists all the publically announced CDMI implementations here. We also plan to start testing implementations with standardized tests to certify them as conformant. This will be a separate list.

Q. FC3 Common Services layer vs. SWIFT, S3, & CDMI – Will it fully integrate with encryption at rest vendors?

A. Amazon does offer encryption at rest for example, but does not (yet) allow you choose the algorithm. CDMI allows vendors to show a list of algorithms and pick the one they want.

Q. You’d mentioned NFS, other interfaces for compatibility – but often “native” NFS deployments can be pretty high performance. Object storage doesn’t really focus on performance, does it? How is it addressed for customers moving to the object model?

A. CDMI implementations are responsible for the performance not the standard itself, but there is nothing in an object interface that would make it inherently slower. But if the NFS interface implementation is faster, customers can use that interface for apps with those performance needs. The compatibility means they can use whatever interface makes sense for each application type.

Q. Is it possible to query the user-metadata on a container level for listing all the data objects that have that user-metadata set?

A. Yes. Metadata query is key and it can be scoped however you like. Data system metadata is also hierarchical and inherited – meaning that you can override the parent container settings.

Q. So would it be reasonable to say that any current object storage should be expected to implement one or more of these metadata models? What if the object store wasn’t necessarily meant to play in a cloud? Would it be at a disadvantage if its metadata model was proprietary?

A. Yes, but as an add-on that would not interfere with the existing API/access method. Eventually as CDMI becomes ubiquitous, products would be at a disadvantage if they did not add this type of interface.

New Webcast: Hierarchical Erasure Coding: Making Erasure Coding Usable

May 11, 2015May 11, 2015 Glyn Bowden Leave a comment

On May 14^th the SNIA-CSI (Cloud Storage Initiative) will be hosting a live Webcast “Hierarchical Erasure Coding: Making erasure coding usable.” This technical talk, presented by Vishnu Vardhan, Sr. Manager, Object Storage, at NetApp and myself, will cover two different approaches to erasure coding – a flat erasure code across JBOD, and a hierarchical code with an inner code and an outer code. This Webcast, part of the SNIA-CSI developer’s series, will compare the two approaches on different parameters that impact the IT business and provide guidance on evaluating object storage solutions. You’ll learn:

Industry dynamics
Erasure coding vs. RAID – Which is better?
When is erasure coding a good fit?
Hierarchical Erasure Coding- The next generation
How hierarchical codes make growth easier
Key areas where hierarchical coding is better than flat erasure codes

Securely Sharing Health Care Data across Different Cloud Services

April 21, 2015April 21, 2015 mac 1 Comment

As more and more health care providers leverage the efficiencies of the cloud, the need to share health care data across different cloud services arises. Sharing health care data across cloud services must ensure the confidentiality, integrity, and availability of the health data and preserve the privacy of the patients in such a way that revealing the data to other data requestors is performed only with patient consent.

The Cloud Data Management Interface (CDMI) international standard is a protocol that has been standardized by SNIA to create interoperable data management services in cloud storage.

The Cloud Storage TWG has just released a technical white paper, “Towards a CDMI Health Care Profile,” that explores the capabilities of CDMI in addressing these requirements, and provides suggestions for possible extensions that are appropriate for a health care profile.

I encourage you to download this paper to learn:

Motivations for protecting health data
Health data protection requirements
A use case that promotes the deployment of health data protection
Requirements and implementation aspects of the use case
Use case architecture
Future use cases roadmap

I hope you’ll find this paper enlightening and welcome feedback and comments on its content here in this blog.

Hybrid Clouds Webcast Preview

March 10, 2015March 10, 2015 Glyn Bowden Leave a comment

On March 18^th, SNIA-CSI will be hosting a live Webcast “Hybrid Clouds: Bridging Private and Public Cloud Infrastructures.”

Every IT consumer is using (or is planning to use) cloud in one form or another. The emphasis on the design and implementation of cloud architectures is often made without consideration of where the cloud storage and compute should be located and the benefits, costs and risks of deciding where the applications will run. Will it be a public cloud? Or a private cloud in the data center or co-location site? Or a hybrid of the two?

This session will be an overview on developing & delivering a cloud architecture with a focus on getting the overall goals correctly specified and defined, understanding the issues that must be addressed, and then making the decision about whether the application is suitable for public, private or some hybrid mixture of the two before undertaking implementation. We’ll also focus on one of the most difficult aspects of the solution, the management of data and storage in the cloud, and present a case study of a successful commercial implementation.

Register now for this live event. I hope you’ll join Alex McDonald and me for what we hope will be an informative and interactive event.

SNIA CSI Welcomes Glyn Bowden

March 2, 2015March 2, 2015 Alex McDonald Leave a comment

At our annual SNIA Members’ Symposium in San Jose, the Cloud Storage Initiative (CSI) elected our 2015 CSI board. I’d like to officially welcome our newest board member, Glyn Bowden from HP. HP now joins our growing list of member companies.

The CSI is committed to the adoption, growth and standardization of cloud storage and related cloud data services to promote interoperability and portability of data stored in the cloud. CSI leads as an industry-neutral authority on cloud storage environments and is committed to educating vendor and end user communities on cloud storage & industry standardization benefits.

It’s only the beginning of March and we’ve already hosted several educational Webcasts on topics ranging from OpenStack Cloud Storage and OpenStack Manila, to CDMI and the LTFS Bulk Transfer Standard. All CSI Webcasts are available on-demand. I encourage you to check then out.

LTFS Bulk Transfer Standard Q&A

February 27, 2015February 27, 2015 David Slik Leave a comment

Our recent live SNIA Cloud Webcast “LTFS Bulk Transfer Standard” is now available on-demand. Thanks to all the folks who attended the live event. We did not have time to address all of the questions, so here are answers to them. If you think of additional questions, please feel free to comment on this blog.

Q. The LTFS standard seems to support shared extents between files, and by extension, deduplicated files. Is this a correct assessment, and how does it play in the bulk transfer standard?

A. The LTFS Bulk Transfer Standard supports shared extents as supported by the LTFS standard, which can transparently reduce space used by having multiple references to common data stored on tape (deduplication). This typically happens below the bulk transfer layer, by the software used to read and write the LTFS volumes. At this point, few software packages support this feature due to the wear and latency consequences of read seeks resulting from using this feature.

Q. What is the state of the standard in its lifecycle? (e.g., working group draft, public review, published, etc.)

A. The LTFS standard has been around for some time; more information can be found here at http://www.snia.org/tech_activities/standards/curr_standards/ltfs. The LTFS Bulk Transfer Standard is here at http://www.snia.org/tech_activities/publicreview#ltfsbulk, and is in public review.

Q. The standard seems to be based on the idea of moving physical tapes to the cloud. Is there a definition of a virtual LTFS image that can be moved between systems over the network?

A. Not yet, but that is a great idea we’ll be taking forward in the next versions of the proposal.

Q. One of the barriers to greater use of LTFS in the Cloud is the relative lack of enterprise grade management software that ensures that the tape media is refreshed / upgraded as it ages, that its integrity is periodically checked, that reclamation and compaction is done. It needs open standards for support in standard volume management systems as well. Until these things are in place, LTFS will be interesting largely to specialized industries like film/entertainment, seismic, and bulk transfer & bulk storage — but not about the steady-state use of tape as a true additional layer of the cloud storage hierarchy. Tape with LTFS plus proper management could fill this role — but not until the full lifecycle tape management is available and integrated.

A. The management that is always required for a physical product with a well-defined and finite lifetime is not a unique requirement of LTFS. Tape has a long history of use as a backup and archive medium, and there are a number of tape management products that are commercially available from LTO tape suppliers and independent software companies, as well as open source products. A Google search for “tape management software” will provide you with a number of alternative solutions.

Q. Do you have a list or people that sell LTFS based solutions?

A. No we don’t, but it’s a very good idea, and we’ll investigate it further.

OpenStack Cloud Storage Q&A

January 21, 2015January 21, 2015 Sam Fineberg Leave a comment

More than 300 people have seen our Webcast “OpenStack Cloud Storage.” If you missed it, it’s now available on demand. It was a great session with a lot of questions from attendees. We did not have time to address them all – so here is a complete Q&A. If you think of any others, please comment on this blog. Also, mark your calendar for January 29^th when the SNIA Cloud Storage Initiative will continue its Developers Tutorial Series with a live Webcast on OpenStack Manila.

Q. Is it correct to say that one can use OpenStack on any vendor’s hardware?

A. Servers, yes, assuming the hardware is supported by Linux. Block storage requires a driver, and not all vendor systems have Cinder drivers.

Q. Is there any OpenStack investigation and/or development in the storage networking area?

A. Cinder includes support for FC and iSCSI. As of Icehouse, the FC support also includes auto-zoning.

Q. Is there any monetization going on around OpenStack, like we see for distros of Linux?

A. Yes, there are already several commercial distributions available.

Q. Is erasure code needed to get a positive business case for Swift, when compared with traditional storage systems?

A. It is a way to reduce the cost of replication. Traditional storage systems typically already have erasure coding, in the form of RAID. Systems without erasure coding end up using more storage to achieve the same level of protection due to their use of 3-way replication.

Q. Is erasure code currently implemented in the current Swift release?

A. No, it is a separate development stream, which has not been merged yet.

Q. Any limitation on the number of objects per container or total number of objects per Swift cluster?

A. Technically there are no limits. However, in practice, the fact that the containers are implemented using SQL lite limits their size to a million or maybe a few million objects per container. However, due to the way that Swift partitions its metadata, each user can also have millions of containers, and there can be millions of users. So practically speaking, the total system can support an unlimited number of objects.

Q. What are some of the technical reasons for an enterprise to select Swift vs. Amazon S3? In other words, are they pretty much direct alternatives, or does each have its own preferred use cases?

A. They are more or less direct alternatives. There are some minor differences, but they are made for the same purpose. That said, S3 is only available from Amazon. There are some S3 compatible systems, but most of those also support Swift. Swift, on the other hand, is available open source or from multiple vendors. So if you want to run it in your own data center, or in a public cloud other than Amazon, you probably want Swift.

Q. If I wanted to play around with Open Stack, Cinder, and Swift in a lab environment (or in my basement), what do I need and how do I get started?

A. openstack.org is the best place to start. The “devstack” distribution is also good for playing around.

Q. Will you be showing any features for Kilo?

A. The “Futures” I showed will likely be Kilo features, though the final decision of what will be in Kilo won’t happen until just before release.

Q. Are there any plans to implement data encryption in Cinder?

A. I believe some of the back ends can support encryption already. Cinder is really just a provisioning and orchestration layer. Encryption is a data path feature, so it would need to be implemented in the back end.

Q. Some time back I heard OpenStack Swift is going to come up with block storage as well, any timeline for that?

A. I haven’t heard this, Swift is object storage.

Q. The performance characteristics of Cinder block services can vary quite widely. Is there any standard measure proposed within OpenStack to inform Nova or the application about the underlying Cinder block performance characteristics?

A. Volume types were designed to enable clouds to provide different levels of service. The meaning of these types is up to the cloud administrator. That said, Cinder does expose QoS features like minimum/maximum IOPS.

Q. Is the hypervisor talking to a cinder volume or to (for example) a NetApp or EMC volume?

A. The hypervisor talks to a volume the same way it does outside of OpenStack. For example, the KVM hypervisor can talk to volumes through LVM, or can mount SAN volumes directly.

Q. Which of these projects are most production-ready?

A. This is a hard question, and depends on your definition of production ready. It’s hard to do much without Nova, Glance, and Horizon. Most people use Cinder too, and Swift has been in production at HP and Rackspace for years. Neutron has a lot of complexity, so some people still use Nova network, but that has many limitations. For toy clouds you can avoid using Keystone, but you need it for a “production” cluster. The best way to get a “production ready” OpenStack is to get a supported commercial distribution.

Q. Are there any Plugfests?

A. No, however, the Cinder team has a fairly extensive and continuous integration process that drivers need to pass through. Swift does not because it doesn’t officially “support” any plugins.