Meta’s Next Llama AI Models Are Trained on a ‘Bigger Than Any Other’ GPU Cluster

Managing such an array of chips to develop the Llama 4 is likely to present unique engineering challenges and require significant power. Meta executives on Wednesday deflected an analyst’s question about power access issues in parts of the US that have hampered the company’s efforts to develop more powerful AI.

According to one estimate, a cluster of 100,000 H100 chips would require 150 megawatts of power. The largest computer of the national laboratory in the United States, El Capitan, by contrast requires 30 megawatts of power. Meta expects to spend up to $40 billion this year to provide data centers and other infrastructure, an increase of more than 42 percent from 2023. The company expects even worse growth in that spending next year.

Meta’s total operating expenses have grown nearly 9 percent this year. But overall sales—mostly from advertising—increased by more than 22 percent, leaving the company with more cash and more profit as it pours billions of dollars into Llama’s efforts.

Meanwhile, OpenAI, considered the current leader in developing high-end AI, is burning through cash despite charging developers for access to its models. Currently a non-profit, he said he is training GPT-5, which will follow the model that currently powers ChatGPT. OpenAI said that GPT-5 will be larger than its predecessor, but did not say anything about the computer cluster it uses for training. OpenAI also said that in addition to scale, GPT-5 will include other new features, including a newly developed way of reasoning.

CEO Sam Altman said the GPT-5 will be a “significant step forward” compared to its predecessor. Last week, Altman responded to a news report claiming that OpenAI’s next frontier model would be released in December by writing in X, “outrageous fake news.”

On Tuesday, Google CEO Sundar Pichai said the company’s new version of the Gemini family of artificial intelligence models is still in development.

Meta’s open approach to AI has sometimes proven controversial. Some AI experts worry that making more powerful AI models freely available could be dangerous because it could help criminals launch cyberattacks or automate the creation of chemical or biological weapons. Although Llama is properly configured before its release to limit misbehavior, it is very trivial to remove these restrictions.

Zuckerberg remains bullish on open source strategies, as Google and OpenAI push proprietary programs. “It seems clear to me that open source is going to be the most affordable, customizable, reliable, playable, and easy-to-use option available to developers,” he said Wednesday. “And I’m proud that Llama is at the forefront of this.”

Zuckerberg added that Llama 4’s new capabilities should enable a wider range of features across Meta services. Today, the signature offering based on Llama models is a ChatGPT-like chatbot known as Meta AI that is available on Facebook, Instagram, WhatsApp, and other applications.

More than 500 million people use Meta AI every month, Zuckerberg said. Over time, Meta expects to generate revenue from feature ads. “There’s going to be a lot of questions that people are using, and the monetization opportunities are going to be there over time as we get there,” Meta CFO Susan Li said on a call Wednesday. With the power of ad revenue, Meta may be able to outsource Llama funding for everyone.