Amazon and NetApp's new data storage options optimized for AI

This week at re:Invent 2023, AWS announced two new unstructured file and object storage options designed for large-scale, high-performance workloads such as artificial intelligence. 

The new Amazon S3 Express One Zone object storage option is designed to provide 10 times better performance than the S3 Standard storage class, while handling hundreds of thousands of requests per second with consistent, single-digit millisecond latency. AWS called out that while this storage class can handle a variety of object sizes, it is particularly adept at handling large numbers of small objects due to the consistent low latency.

The other announcement was scale-out file systems for Amazon FSx for NetApp ONTAP. It will offer up to nine times higher storage performance compared with existing file systems. The scale-out version of FSx for ONTAP will deliver ONTAP's enterprise-level data management features and performance to support intensive workloads such as machine learning and generative AI.

Combined, these two storage options significantly enhance the ability of AWS to support data-intensive applications such electronic design automation, film editing, visual effects, life sciences/genomics and seismic analyses for energy exploration.

Prior to these advancements, limited options for high-capacity, high-performance, unstructured storage held back cloud adoption.

According to research by TechTarget's Enterprise Strategy Group, the most common reason organizations determine that a workload is not a candidate for migration to the public cloud is that its high-performance demands can be met more cost-effectively on premises (cited by 30%). That response rate increases to 50% for organizations in the media and entertainment industry. Film editing and visual effects workloads typically require high levels of performance from high-capacity unstructured storage options. This finding suggests that a need for higher-performance cloud storage is even more pronounced for large, unstructured storage environments.

These announcements complement AWS's services designed to enhance and optimize the development and use of predictive and generative AI. As a result, AWS is in an incredibly strong position to serve an expected massive uptick in generative AI adoption in the coming year. Enterprise Strategy Group research has found that 92% of organizations expect to have generative AI in production within the next year.

In addition, low-latency, high-performance storage is not just required for AI-related training activities. It is also beneficial to the entire data pipeline, helping to accelerate data preparation and data cleansing, which are efforts that comprise a large portion of the work associated with artificial intelligence initiatives. Leveraging low-latency, high-performance cloud storage should help simplify and accelerate those operations, which in turn will accelerate AI initiative rollouts, improve business opportunity, and ultimately, improve business outcomes.

These announcements are big wins for AWS, NetApp and AWS's customers. The expansion of the AWS and NetApp partnership will enable both companies to be in a stronger position to reap the rewards of the expected increase in AI and Generative AI investments in the coming year.

Scott Sinclair is Practice Director with TechTarget's Enterprise Strategy Group, covering the storage industry.

Enterprise Strategy Group is a division of TechTarget. Its analysts have business relationships with technology vendors.