How to Leverage AI Without Compromising Sustainability Goals

By Patrick Smith, VP, EMEA Field CTO, Pure Storage

There’s no denying that Artificial Intelligence (AI) has become one of the fastest growing and largest areas of enterprise technology investment and innovation in recent years. Given there are so many practical applications for this technology, it’s no surprise that AI is supporting mainstream use cases, ranging from healthcare and life sciences to semiconductor and chip manufacturing, automotive, financial services, and beyond.

While generative AI tools such as ChatGPT have dominated the headlines in recent months, the reality is that AI has been present for a number of years. However, the latest wave of widely accessible generative AI tools is resulting in more machine generated data than ever before, and this is driving the unprecedented growth of unstructured data worldwide. In fact, IDC predicts that by 2025, the total amount of digital data created globally will rise to 175 zettabytes (from approximately 40 zettabytes in 2019). This estimate can actually be considered conservative, given the surge in AI-generated data we are seeing today. 

In a somewhat perpetual cycle, greater volumes of data and the acceleration of AI means a bigger opportunity for businesses to turn this information into actionable intelligence, to innovate faster than their competitors, increase customer satisfaction, streamline operations, and ultimately become a more successful company. However, just as we refine oil into useful products such as fuel and plastics, data must also be refined before it can provide value. This is where data analytics (increasingly AI-based) comes in.

In order to power AI, and AI-based data analytics, organisations need a flexible, reliable, performant, and perhaps most importantly, sustainable data storage infrastructure in place.

1. Performance is key because AI relies on sending massive amounts of data into GPUs, over and over again. The faster organisations do that, the quicker and better results they get. AI resources (GPUs, data scientists) are expensive and in high-demand, so keeping them waiting on access to data can lead to a hefty bill. Just as important as feeding the GPUs, is accelerating the whole data preparation and curation workflows, helping to collect and process the data in the first place.
2. Flexibility comes in as AI is easily the most rapidly evolving space in technology - tools, techniques, data-sets and use-cases are evolving every single day. As a result, it’s critical to invest in technology and infrastructure choices that are going to allow organisations to adapt to changes quickly.
3. Enterprise reliability and controls are more important to organisations than ever with AI environments. These are mission critical environments, and any downtime can lead to exorbitant costs. As a result, availability and reliability are essential. Additionally, AI projects are often large sprawling projects and heavily automated. Having controls around quotas, security, and ease of management is critical. 
4. Last but certainly not least is one of the planet’s most pressing concerns, sustainability.

Current estimates have data centers accounting for between one to four percent of all global energy consumption. In fact, in some countries datacenter expansion has been halted because they cannot access adequate power. AI is not going anywhere, and overall it will be an overwhelmingly positive tool for humanity, helping us automate repetitive tasks, treat diseases more effectively, and better understand our world through weather and climate patterns. However, from an environmental perspective, it only adds to energy consumption and carbon footprint concerns. In the wake of this immense challenge and opportunity, building an efficient and sustainable technology infrastructure for AI is critical to mitigating global warming and the worst impacts of climate change.

As data volumes grow and high performance becomes mainstream as a requirement for AI, sustainability concerns come to the fore. As these needs increase, so do costs in terms of power, cooling and the space to house equipment. In today’s context of soaring energy prices this is not only an environmental issue, but an operational and financial challenge for businesses too. 

Fortunately, some companies are designing and building products and delivering services that allow customers to dramatically decrease their own environmental footprints. For example, all-flash storage solutions are considerably more efficient than their spinning disk (HDD) counterparts. What’s more, flash storage is much better suited to running AI projects. 

This is because the key to results is connecting AI models or AI powered applications to data. To do this successfully you need lots of data, this data can’t be cold, and crucially data needs to be easily accessible, across silos and applications. This simply isn’t possible with HDD based storage underpinning your operations, all-flash is needed. 

To further bolster the adoption of sustainable technology choices, consider whether your organisation has a sustainability officer, someone responsible for the company’s overall carbon footprint. Involve those stakeholders at the beginning of the process to ensure no stone goes unturned on your journey to sustainable AI.

FIGURE 1  The era of digital transformation changes a number of factors impacting storage reliability.

Many companies are already applying these best practices to embark on their AI journeys. Meta, for example, wanted to help its AI researchers build new and better AI models that can learn from trillions of examples, work across hundreds of different languages, seamlessly analyse text, images, and video together, develop new augmented reality tools, and much more. As a result the company set out to create its AI Research SuperCluster (RSC), with the intention of being the world’s fastest AI supercomputer.

Meta chose Pure Storage as it needed a partner that could deliver robust and scalable storage capabilities to power RSC. With FlashArray and FlashBlade systems from Pure Storage, RSC has unparalleled performance to rapidly analyse both structured and unstructured data, underpinned by Pure Storage’s foundation of performance, reliability, flexibility and sustainability. For example, Meta needed a storage solution with a very low power profile so that it could invest more power into the GPUs, which speed up AI models. Pure Storage FlashArray systems can reduce greenhouse gas emissions and energy consumption by over 80% when compared to a competitive all-flash array. With this kind of sustainable technology foundation in place, RSC will pave the way toward building technologies for Meta’s next major computing platform, the metaverse, where AI-driven applications and products will play an important role.

To prepare for a world in which ever-growing amounts of unstructured data will be the subject of much-increased use of AI analytics, companies will need storage in colossal volumes that offers rapid access and is efficient in sustainability terms.

Businesses should look for vendors with a roadmap for high density flash storage capacity that can handle workloads from the most performance-hungry to those currently categorized as secondary but which will gain in importance with the rise of constant AI processing. Companies should also evaluate vendor purchasing options that can build in seamless capacity and technology upgrades for years ahead. 

Lastly, organisations should look for all-flash storage providers that can demonstrate third-party verified ESG metrics, so that AI projects can be executed without damaging the environment, and their bottom line.