Storage optimisation with FinOps and AWS

One big benefit of moving your business to the cloud is that you can turn services on and off as you need them. But that capability can also make it easy to leave services on longer than needed, which means you can end up paying for things you’re not using.

The good news is that Amazon Web Services (AWS) provides some great tools to help you optimise your resource use and spending. By using cost-effective storage solutions on AWS and following the principles of FinOps – an operating model where everyone in an organisation takes responsibility for their cloud usage to maximise the value of every dollar spent – you can ensure that you’ve built a strong foundation on the cloud.

This is the approach that SoftwareOne follows when we work with customers looking to make the most of their resources on AWS. We’ve found that storage is one area where applying FinOps principles can make a significant difference in reducing cloud consumption waste and unnecessary spending. In this blog post, I’ll look at one AWS tool that’s really effective here: Amazon S3 Storage Lens.

Improving visibility into storage usage with AWS services can deliver significant savings

Before getting into the specifics of how to use Amazon S3 Storage Lens, it’s important to understand why tools like these are so valuable: they provide much-needed visibility into what you’re using in the cloud and what you’re spending on cloud services.

Wasted spend on cloud is a major concern for many businesses. According to survey findings in the Flexera 2022 State of the Cloud Report, organisations estimate that 32% of their cloud budget is wasted – and the real figure is likely even higher.

By following FinOps best practices, businesses can significantly improve cloud consumption efficiency and reduce cloud costs. FinOps isn’t just about cutting expenses. It’s also about gaining transparency into your resource use and getting the most value out of every dollar you spend on your use of the cloud.

Consider the example of Airbnb, as recently reported in the Wall Street Journal. Airbnb cut its cloud spending by 27%, in large part by taking advantage of the different storage tiers available on AWS. Best of all, this kind of strategy not only reduces storage costs, but it does so without any impact on your customer-facing products and services.

Inform, optimise, operate with AWS storage services

The three steps for optimising storage align well with the three iterative phases of FinOps, which are 1) inform, 2) optimise and 3) operate. For storage, this means 1) becoming informed about your storage usage and spending, 2) finding ways to improve your usage through optimisation and 3) implementing those improvements and automating where you can, then repeating the whole process to keep continuously improving.

Amazon S3 Storage Lens helps with the first step – inform – by providing data about what storage you’re using, how you’re using it and how much you’re paying for it. Introduced in late 2020, Amazon S3 Storage Lens is a powerful tool for understanding how your organisation uses and pays for storage on Amazon Simple Storage Service (Amazon S3). There’s a free-of-charge version that provides insights into 15 usage-related metrics to help optimise your cloud storage usage, but the paid Storage Lens option provides 35 additional advanced metrics across four categories, along with other features. With the insights gained from Storage Lens, you can optimise and improve your storage usage in a continuous cycle.

For example, you can use Storage Lens to view something called percent retrieval rate, which shows how often data stored is retrieved from each storage bucket. It’s common to set up different S3 storage buckets for specific purposes, but when that purpose has been fulfilled – say, a project is completed – the bucket can remain forgotten and unused, but it still stores data and thus incurs charges. If you find objects like this in storage using Storage Lens, you can either delete them or move them to a less costly storage tier. Storage Lens can also help you identify other unused or duplicate resources, such as data files that were saved during an incomplete multipart upload or older versions of current resources.

S3 Storage Lens lets you create dashboards that focus on different accounts, users and storage buckets. This makes it possible to sort metrics in a variety of ways and view storage trends. By sharing these insights with different people across your organisation, you can identify how often storage resources must be accessed or whether they’re needed at all.

The benefits of tiering

If you can’t delete unused or rarely accessed storage resources because of regulations or compliance requirements, Amazon S3 provides you with a range of options that can significantly reduce your storage costs. In addition to S3 Standard storage, which is the most expensive storage tier, there are (in descending order of cost) S3 Standard Infrequent Access, S3 One Zone-Infrequent Access, S3 Glacier Instant Retrieval, S3 Glacier Flexible Retrieval and S3 Glacier Deep Archive. There’s also S3 Outposts, which is for on-premises storage, and S3 Intelligent-Tiering, which automatically moves resources to the most cost-effective storage tier based on your usage and access patterns.

For example, if you haven’t accessed an object in 30 days, Intelligent-Tiering will move that to the Infrequent Access tier. If you need to access that object again, Intelligent-Tiering will move it back to the S3 Standard tier. Using Intelligent-Tiering can reduce storage costs significantly without you having to manually manage anything – the Standard Infrequent Access tier is priced 40% lower than the Standard tier.

Amazon S3 also gives you the option to create lifecycle policies with customised storage preferences. This provides fine-grained control over your storage usage and costs – ensuring, for instance, that you comply with important industry and regulatory data retention requirements – without the need for constant, time-consuming manual adjustments.