Object Storage Model: Key Characteristics and Applications for Cloud Success

Traditional storage approaches were designed for VMs and small databases at the TB scale. Object storage is the only solution that delivers the combination of throughput performance and scale.

September 19, 2022

Cloud-native applications want to talk to cloud-native data stores. They want the attributes outlined above, RESTful APIs, automation, containerization and orchestration. They also want scale – and, more specifically, performance at scale. In the first part of this explainer on object storage, Anand Babu Periasamy, CEO and co-founder of MinIO, talked about why object storage is essential for the cloud operating model. In this second part of the explainer, he discusses in detail the key characteristics of object storage and its applications on the cloud.

The definition of primary storage is the data store on which your application runs. This could include a database, your AI/ML pipeline, or your advanced analytics application. It also means your video streaming platform or your website.

In the public cloud, object storage is almost always primary storage. AWS EMR, Redshift, Databricks, Snowflake, and BigQuery, have run on object storage since Day 1. That makes object storage primary storage for those workloads (and thousands more). 

Object storage is also primary storage in the private cloud. When VMware announced Tanzu, did they have any SAN/NAS vendors in the announcement? No. It was all object storage. Did they expect that Tanzu would only deal with “secondary” storage? Of course not. Do they understand the enterprise better than anyone else? You can make a strong argument for that case. VMware sees object storage as primary storage because the cloud-native applications won’t run without it.

The same goes for RedHat OpenShift, HPE Ezmeral, and SUSE Rancher. Every one of them sees object storage as primary storage. That is not to say they don’t also think of SAN/NAS as primary storage – they do. They just recognize that those technologies are legacy when it comes to the cloud. 

Performance at scale is an ever-evolving challenge because data continues to grow. It is standard to deal with multiple PBs of data, and increasingly the ask is to manage EBs. Traditional storage approaches were designed for VMs and small databases at the TB scale. Object storage is the only solution that delivers the combination of throughput performance and scale. As a result, it is the primary solution in the enterprise storage playbook. Indeed, a recent IDC survey Opens a new window showed that 80% of respondents believe object storage can support their top IT initiatives.

What are the key characteristics of cloud object storage? 

Let’s look at the characteristics that make cloud object storage an essential aspect of digital success today.

  1. Lets Kubernetes orchestrate storage nodes

A truly cloud-native object storage system lets Kubernetes orchestrate the storage nodes performing provisioning, placement, scaling, and upgrade operations. This is achieved via the Kubernetes Operator pattern. Hardware appliances and bare metal storage solutions that run outside the Kubernetes system must be managed manually – thus do not work well. Simply having a CSI driver does not mean the solution is cloud-native, and they will not run in the public cloud. 

  1. Highly multi-tenant

We outlined the requirement for multi-tenancy above. What this means in the cloud-native world is that object storage leverages Kubernetes namespaces and containers to isolate object storage instances. You should be able to simultaneously host multiple customers on different versions of object storage instances. Upgrades and updates of one tenant should not affect the other. More importantly, object storage should leverage Kubernetes for tenant orchestration, scaling and isolation with pods and namespaces. If Kubernetes isn’t managing the object storage multi-tenancy, then the storage infrastructure is not cloud native.

  1. Lightweight

Multi-tenancy isn’t possible unless the storage system is extremely lightweight and can be packaged with the application stack. If the storage system takes too many resources or bloated binaries, it won’t be possible to pack many tenants on the same infrastructure.

  1. S3 compatible

The S3 API is the de facto standard in the cloud. As a result, alternatives to AWS must speak the API fluently to function and interoperate across diverse environments—public cloud, private cloud, data center, multi-cloud, hybrid cloud and at the edge. It is a RESTful API, and adherence is more akin to interpretation than law. The ultimate question is whether or not the application in question will work against the object storage endpoint using standard S3 calls.  You need to test – which means you need access to the software. 

  1. Supports cloud tiering and lifecycle management

The impact of the multi-cloud would be its own post. However, to be successful in the multi-cloud, the object store needs to run on different clouds and speak to different clouds. This, in turn, allows for an application to determine what data ends up where – from NVMe to HDD and even infrequently accessed public cloud storage tiers. 

See More: Object Storage as Primary Storage in the Cloud Operating Model

Core Object Storage Applications for the Cloud Operating Model

Let’s take a look at three use cases for primary storage that have been taken over by object storage.

  • Databases

The availability of high-performance and elastically scalable object storage has risen to the challenges presented by the explosive growth of both structured and unstructured data. Object storage must deliver exceptional throughput at an acceptable latency in order to provide backing storage to a database. The best database architectures are disaggregated, with developers heavily focused on the compute side of distributed low-latency query processing and letting object storage handle the rest.

  • OS images, application artifacts, snapshots, backups

Enterprises have wanted faster backup and restore operations for decades but couldn’t achieve them because of the emphasis on the durability and low cost of backup infrastructure. High-performance object storage allows applications to take advantage of disaggregation to deliver superior performance, scalability and economics. Cloud-native applications read and write to object storage using the S3 API over https, making it a convenient, consistent and portable solution for saving artifacts. When relying on MinIO as a ransomware protection mechanism, you gain the ability to conduct a very fast restore from an immutable snapshot, minimizing operational disruption.

  • AI/ML/DL and stream/log analytics

We live in a data-driven world where the streaming log data is often the dominant part of an enterprise’s data. Data Lakes are now at the center of real-time business decisions. Cutting-edge real-time analytics and visualization applications rely on streaming data and require object storage as a sink. The AI (ML/DL) workloads that drive the most business value today rely on object storage for multiple pipeline stages and for model storage. Even HPC, formerly the realm of scale-up, has embraced the scale-out architecture of the cloud and now depends on object storage for performance at scale.

Summary

We live in a cloud-native world, and the transition is only accelerating. The first step is to understand the attributes of the cloud operating model and the technologies that define it. These are journeys – there are no shortcuts. We know this firsthand from working with hundreds of enterprises in the last few years. 

While technology matters, cultural change is as, if not more important. The enterprises that adopted the cloud-native mentality, staffed accordingly and granted power organizationally are the ones that succeed. Change is hard – it needs to be engineered. 

How are you using cloud object storage? Share with us on  FacebookOpens a new window , TwitterOpens a new window , and LinkedInOpens a new window .

MORE ON STORAGE: 

Anand Babu Periasamy
Anand Babu Periasamy is the CEO and co-founder of MinIO. One of the leading thinkers and technologists in the open source software movement, AB was a co-founder and CTO of GlusterFS which was acquired by RedHat in 2011. Following the acquisition, he served in the office of the CTO at RedHat prior to founding MinIO in late 2015. AB is an active angel investor and serves on the board of H2O.ai and the Free Software Foundation of India. He earned his BE in Computer Science and Engineering from Annamalai University.
Take me to Community
Do you still have questions? Head over to the Spiceworks Community to find answers.