72% of organizations have encountered incompatibility issues between various object storage implementations according to a poll at our recent SNIA Cloud Storage Technologies Initiative webinar, “Navigating the Complexities of Object Storage Compatibility.” If you missed the live presentation or you would like to see the answers to the other poll questions we asked the audience, you can view it on-demand at the SNIA Educational Library.
The audience was highly-engaged during the live event and asked several great questions. Here are answers to them all.
Q. Do you see the need for fast object storage for AI kind of workloads?
A. Yes, the demand for fast object storage in AI workloads is growing. Initially, object storage was mainly used for backup or archival purposes. However, its evolution into Data Lakes and the introduction of features like the S3 SELECT API have made it more suitable for data analytics. The launch of Amazon’s S3 Express, a faster yet more expensive tier, is a clear indication of this trend. Other vendors are following suit, suggesting a shift towards object storage as a primary data storage platform for specific workloads.
Q. As Object Storage becomes more prevalent in the primary storage space, could you talk about data protection, especially functionalities like synchronous replication and multi-site deployments – or is your view that this is not needed for object storage deployments?
A. Data protection, including functionalities like synchronous replication and multi-site deployments, is essential for object storage, especially as it becomes more prevalent in primary storage. Various object storage implementations address this differently. For instance, Amazon S3 supports asynchronous replication. Azure ZRS (Zone-redundant storage) offers something akin to synchronous replication within a specific geographical area. Many on-premises solutions provide multi-site deployment and replication capabilities. It’s crucial for vendors to offer distinct features and value additions, giving customers a range of choices to best meet their specific requirements. Ultimately, customers must define their data availability and durability needs and select the solution that aligns with their use case.
Q. Regarding polling question #3 during the webinar, why did the question only ask “above 10PB?” We look for multi-PB like 100PB … does this mean Object Storage is not suitable for multi PB?
A. Object storage is inherently scalable and can support deployments ranging from petabyte to exabyte scale. However, scalability can vary based on specific implementations. Each object storage solution may have its own limits in terms of capacity. It’s important to review the details of each solution to ensure it meets your specific needs for multi-petabyte scale deployments.
Q. Is Wasabi 100% Compatible with Amazon S3?
A. While we typically avoid discussing specific vendors in a general forum, it’s important to note that most ‘S3-compatible’ object storage implementations have some discrepancies when compared to Amazon S3. These differences can vary in significance. Therefore, we always recommend testing your actual workload against the specific object storage solution to identify any critical issues or incompatibilities.
Q. What are the best ways to see a unified view of different types of storage — including objects, file and blocks? This may be most relevant for enterprise-wide data tracking and multi-cloud deployments.
A. There are various solutions available from different vendors that offer visibility into multiple types of storage, including object, file, and block storage. These solutions are particularly useful for enterprise-wide data management and multi-cloud deployments. However, this topic extends beyond the scope of our current discussion. SNIA might consider addressing this in a separate, dedicated webinar in the future.
Q. Is there any standard object storage implementation against which the S3 compatibility would be defined?
A. Amazon S3 serves as a de facto standard for object storage implementation. Independent software vendors (ISVs) can decide the degree of compatibility they want to achieve with Amazon S3, including which features to implement and to what extent. The objective isn’t necessarily to achieve identical functionality across all implementations, but rather for each ISV to be cognizant of the specific differences and potential incompatibilities in their own solutions. Being aware of these discrepancies is key, even if complete compatibility isn’t achieved.
Q. With the introduction of directory buckets, do you anticipate vendors picking up compatibility there as well or maintaining a strictly flat namespace?
A. That’s an intriguing question. We are putting together an on-going object storage forum, which will delve into more in follow-up calls, and will serve as a platform for these kinds of discussions. We anticipate addressing not only the concept of directory buckets versus a flat namespace, but also exploring other ideas like performance enhancements and alternate transport layers for S3. This forum is intended to be a collaborative space for discussing future directions in object storage. If you’re interested, contact the cloudtwg@snia.org.
Q. How would an incompatibility be categorized as something that is important for clients vs. just something that doesn’t meet the AWS spec/behavior?
A. Incompatibilities should be assessed based on the specific needs and priorities of each implementor. While we don’t set universal compatibility goals, it’s up to every implementor to determine how closely they align with S3 or other protocols. They must decide whether to address any discrepancies in behavior or functionality based on their own objectives and their clients’ requirements. Essentially, the significance of an incompatibility is determined by its impact on the implementor’s goals and client needs.
Q. Have customers experienced incompatibilities around different SDKs with regard to HA behaviors? Load balancers vs. round robin DNS vs. other HA techniques on-prem and in the cloud?
A. Yes, customers do encounter incompatibilities related to different SDKs, particularly concerning high availability (HA) behaviors. Object storage encompasses more than just APIs; it also involves implementation choices like load balancing decisions and HA techniques. Discrepancies often arise due to these differences, especially when object storage solutions are deployed within a customer’s data center and need to integrate with the existing networking infrastructure. These incompatibilities can be due to various factors, including whether load balancing is handled through round robin DNS, dedicated load balancers, or other HA techniques, either on-premises or in the cloud.
Q. Any thoughts on keeping pace with AWS as they evolve the S3 API? I’m specifically thinking about the new Directory Bucket type and the associated API changes to support hierarchy.
A. We at the SNIA Cloud Storage Technical Work Group are in dialogue with Amazon and are encouraging their participation in our planned Plugfest at SDC’24. Their involvement would be invaluable in helping us anticipate upcoming changes and understand new developments, such as the Directory Bucket type and its associated API changes. This new variation of S3 from Amazon, which differs from the original implementation, underscores the importance of compatibility testing. While complete compatibility may not always be achievable, it’s crucial for ISVs to be fully aware of how their implementations differ from S3’s evolving standards.
Q. When it comes to object store data protection with backup software, do you see also some incompatibilities with recovered data?
A. When data is backed up to an object storage system, there’s a fundamental expectation that it can be reliably retrieved later. This reliability is a cornerstone of any storage platform. However, issues can arise when data is initially stored in one specific object storage implementation and later transferred to a different one. If this transfer isn’t executed in accordance with the backup software provider’s requirements, it could lead to difficulties in accessing the data in the future. Therefore, careful planning and adherence to recommended practices are crucial during any data migration process to prevent such compatibility issues.
The SNIA Cloud Storage Technical Work Group is actively working on this topic. If you want to get involved, reach out at cloudtwg@snia.org and follow us @SNIAcloud