The AI compute gap: 83% of enterprises face low GPU utilization as spending accelerates

According to a recent VentureBeat survey of 107 enterprises, AI infrastructure spending is rapidly increasing, outpacing the ability to understand its economic implications. While most organizations rely on established hyperscalers and model-provider APIs, many are shifting focus to specialized compute options not currently in use.

Understanding the AI Compute Gap

The central finding of the survey is a significant compute gap, where heavy investment in AI infrastructure is occurring without adequate visibility into associated costs. Only 21% of respondents reported running AI in production at scale, yet 45% plan to evaluate AI-specialized cloud solutions in the next year. This disparity highlights a growing urgency for enterprises to enhance their infrastructure capabilities.

Despite increased spending, the majority of enterprises are not efficiently utilizing their existing resources. An alarming 83% reported GPU utilization at 50% or less, while only 44% can accurately track their AI compute costs. This lack of clarity complicates decision-making as enterprises ramp up their investments.

Shifting Infrastructure Preferences

The survey results indicate that organizations are not firmly committed to their current infrastructure providers. A striking 64% of enterprises intend to switch or add a provider within the next twelve months, with 38% planning to do so in just three months. When making these decisions, enterprises prioritize integration with existing systems (41%) and total cost of ownership (35%) over headline pricing.

Interestingly, the shift from GPU compute to memory bandwidth as inference scales is not yet a priority for many, with about 20% of enterprises unaware of this emerging trend. This oversight could impact future infrastructure decisions as AI demands evolve.

Current AI Infrastructure Landscape

Most enterprises continue to utilize well-known hyperscalers and model APIs for their AI needs, with Google Cloud leading at 48%. In contrast, specialized GPU cloud providers remain largely unutilized, with only 6% operating on-prem GPU clusters. The findings underscore a reliance on established providers, even as organizations express interest in exploring new options.

• Only 21% of enterprises run AI in production at scale.
• 83% report GPU utilization of 50% or less.
• 64% plan to switch or add an infrastructure provider within 12 months.
• 45% intend to evaluate AI-specialized clouds in the next year.

The survey signals a critical moment for enterprises as they navigate the complexities of AI infrastructure and seek to enhance their operational efficiency amidst a rapidly changing landscape.