What's inside a 100,000 GPU cluster? 🤔

Patrick from ServeTheHome takes us on a tour with his camera through different parts of the server, offering a bird's-eye view of its operations. While some more specific details of the supercomputer, such as its power consumption and the size of its bombs, could not be revealed due to a confidentiality agreement, xAI was responsible for blurring and censoring parts of the video before its release. 🎥

Despite this, the most important thing, like Supermicro's GPU servers, remained largely untouched in the footage. These GPU servers are Nvidia HGX H100s, a powerful server solution that features eight H100 GPUs each. 🚀 The HGX H100 platform is integrated within Supermicro's 4U Universal GPU Liquid Cooled system, which provides easily hot-swappable liquid cooling for each GPU. ❄️

These servers are organized into racks containing eight servers each, totaling 64 GPUs per rack. 1U manifolds are sandwiched between each HGX H100, providing the necessary liquid cooling for the servers. At the bottom of each rack, we find another 4U Supermicro unit, this time equipped with a redundant pump system and a rack monitoring system. 🔍

🖥️ These racks are organized in groups of eight, allowing for 512 GPUs per array. Each server is equipped with four redundant power supplies. At the back of the GPU racks are three-phase power supplies, Ethernet switches, and a rack-sized manifold that provides all of the liquid cooling. 💧

There are over 1,500 GPU racks in the Colossus cluster, spread across nearly 200 rack arrays. According to Nvidia CEO Jensen Huang, the GPUs in these 200 arrays were fully installed in just three weeks. 🚀

Since an AI supercluster constantly training models requires massive amounts of bandwidth, xAI went above and beyond in its network interconnectivity. Each graphics card has a dedicated 400GbE NIC (network interface controller), with an additional 400Gb NIC per server. 🔗 This means that each HGX H100 server has 3.6 Terabits per second of Ethernet. Impressive, right? And yes, the entire cluster runs on Ethernet, rather than InfiniBand or other exotic connections that are standard in the supercomputing realm. 🌐

Sure, a supercomputer like the Grok 3 chatbot, which trains AI models, needs more than just GPUs to perform at its best. 🔥 While details about storage and CPU servers in Colossus are somewhat limited, thanks to Patrick's video and the blog post, we know that these servers are usually in Supermicro chassis. 🚀

1U NVMe-forward servers with x86 platform CPUs inside are used, providing both storage and computing power, and are equipped with liquid cooling at the rear. 💧 Additionally, very compact Tesla Megapack battery banks can be seen outside. ⚡️

The start-stop feature of the array, with its millisecond latency between banks, was too much for the conventional power grid or Musk's diesel generators. That's why several Tesla Megapacks (each with a capacity of 3.9 MWh) are used as an intermediate power source between the power grid and the supercomputer. 🖥️🔋 This ensures optimal and efficient operation, avoiding interruptions. 🚦✨

🌟 Using Colossus and Musk's stable supercomputer 🌟

The xAI Colossus supercomputer is currently, according to Nvidia, the largest AI supercomputer in the world. 🤯 While many of the world’s leading supercomputers are used in research by contractors or academics to study weather patterns, diseases, or other complex tasks, Colossus has sole responsibility for training X’s (formerly Twitter’s) various AI models. Most notably, Grok 3, Elon’s “anti-woke” chatbot that’s available only to X Premium subscribers. 🤖

Additionally, ServeTheHome has been informed that Colossus is training AI models “of the future” – models whose uses and capabilities are supposedly beyond the current capabilities of AI. 🚀 The first phase of Colossus construction is complete and the cluster is fully operational, but it’s not all over yet. The Memphis supercomputer will soon be upgraded to double its GPU capacity, with an additional 50,000 H100 GPUs and 50,000 next-generation H200 GPUs. 🔥

This upgrade will also more than double its power consumption, which is already too much for the 14 diesel generators Musk added to the site in July to handle. ⚡ While it's short of Musk's promise of 300,000 H200s inside Colossus, that could be part of Phase 3 of upgrades. 🔋

On the other hand, the 50,000 GPU Cortex supercomputer at Tesla's "Giga Texas" plant also belongs to a Musk company. Cortex is dedicated to training Tesla's autonomous AI technology through camera streaming and image detection, as well as Tesla's autonomous robots and other AI projects. 🤖🚗

Additionally, Tesla will soon see the construction of the Dojo supercomputer in Buffalo, New York, a $500 million project coming soon. 💸 Meanwhile, industry speculators like Baidu CEO Robin Li predict that 99% of AI companies could collapse when the bubble bursts. Whether Musk's record spending on AI will backfire or pay off remains to be seen. ⏳