The SNIA Cloud Storage Technologies Initiative (CSTI) recently hosted a live webinar, “Simplified Namespace Management – The Open Standards Way,” where David Slik, Chair of the SNIA Cloud Storage Technical Work Group (TWG) provided a fascinating overview of how to tackle the complexities of dynamic namespaces. If you missed the live webinar, you can view it on-demand and access a copy of the webinar slides at the SNIA Educational Library. Attendees at the live event asked several interesting questions. Here are answers to them all.
Q. How are the queues assigned to individual namespaces? How many queues are assigned for a particular namespace, can we customize it and if so, how? What is the difference between normal namespace and SR-IOV enabled namespace? Can you please explain sets domain and endurance group?
A. For NVMe® namespace management, the Cloud Storage TWG recommends the use of the SNIA Swordfish® Specification. The SNIA Cloud Data Management Interface (CDMI™) is designed to provide a cloud abstraction that hides the complexities and details of the underlying storage implementation and infrastructure from the cloud user. Hiding storage implementation and infrastructure complexities are a key part of what makes a cloud a cloud, in that:
a) knowledge and specification of the underlying infrastructure should not be required in order to consume storage services (simplification of use),
b) the underlying infrastructure will be constantly and dynamically changing, and these changes should not be visible to the end user (to enable operational flexibility for the cloud provider), and,
c) the desired quality of service, data protection, and other client-required storage services should be indicated by intent, rather than as a side-effect of the underlying configuration (ideally via declarative metadata).
As these are also three key principles of CDMI. This guides us to avoid directly exposing or managing the underlying storage infrastructure.
Q. How do the responses behave if the responses are really large – i.e. Can I get just the metadata that might warn me there are 70 billion objects at the top level and I don’t really want to spend the time to get them all before deciding or diving into one of them?
A. When obtaining information about data objects and containers, CDMI allows the request to specify which fields should be returned in the JSON response. This is described in sections 8.4.1 and 9.4.1 of the CDMI specification, and uses standard URI query parameters.
For example, to get the object name, metadata and number of children for a container, the following request URI would be used: GET /cdmi/2.0.0/MyContainer/?objectName&metadata&childrenrange
CDMI also allows range requests for listing a subset of the children of a container. For example, listing the first 200 children of a container would be accomplished by using the following request URI: GET /cdmi/2.0.0/MyContainer/?children=0-199
There also is a draft extension to CDMI to support recursive children listing and obtaining information about children in a single request, which can dramatically reduce the number of requests required to enumerate a container when information about each child is required.
Q. Where can I go to get help using CDMI with my application?
A. SNIA provides free access to the CDMI specification on the SNIA website. Extensions to the CDMI standard are also publicly available here.The SNIA Cloud Storage (TWG) provides implementation assistance and discusses extensions and errata to the standard as part of its weekly work group calls. Interested parties are encouraged to join the TWG.
Q. If I were not using CDMI, what tools or methods would I need to incorporate to do the same kind of operations? What else is out there?
A. The Cloud Storage TWG does not know of any similar standards for namespace management. In order to manage namespaces without using CDMI, one would need to do the following:
a) Define or select an HTTP-based protocol that provides basic request/response semantics and includes authentication. This is provided by all of the cloud providers for their cloud APIs.
b) Define or select a set of APIs for enumerating namespaces, for example, the ListBuckets API in AWS S3, and the Azure Files List Directories and Files API in Microsoft Azure.
c) Define a set of APIs for listing and specifying how namespaces (files, directories, objects and containers) can be exported or imported.
While each of these exists for the major cloud providers, they are unique for each provider and storage type. CDMI provides a common, open and unified way to manage all types of storage namespaces.
Q. How does CDMI help address security in my namespace management?
A. CDMI provides a number of security functions that assist with namespace management:
a) Every object in CDMI, including namespaces, can have an access control list (ACL) that specifies what operations can be performed against that object. This is described in section 17.2 of the CDMI specification. ACLs are based on standard NFSv4 ACLs, and allow metadata modifications (E.g. CDMI exports and CDMI imports) to have separate access control entries (ACEs).
b) CDMI objects can have their access control decisions delegated to a customer-provided system via Delegated Access Control (DAC), which can provide finer-grained access control than ACLs where needed, as needed. This allows policies to take into account the specific import and export requests themselves, and to interface with policy enforcement frameworks such as XACML and open source policy engines such as the Open Policy Agent (OPA).
c) CDMI allows mapping of user credentials to the user principal and group to be performed by external systems, such as Active Directory. This mapping can be on an object-by-object basis, allowing objects managed by different security domains to co-exist within a single unified namespace.