Meta’s training of its new Llama 3 405B model on a cluster of 16,384 NVIDIA H100 80GB AI GPUs was a 54-day journey fraught with challenges. The company recently released a study detailing the training process, highlighting the unexpected component failures encountered. A staggering 419 unexpected component failures occurred during the 54-day training run, averaging one failure every three hours. Half of these failures stemmed from issues with the GPUs or their onboard HBM3 memory.
While the massive scale of the supercomputer, with its countless components like CPUs, motherboards, RAM, SSDs, GPUs, power systems, and cooling systems, makes such failures somewhat expected, the sheer volume presented a significant hurdle. The team had to work diligently to ensure the system remained operational despite these frequent breakdowns. A single GPU failure could disrupt the entire AI training process, and restarting after 54 days of training would be a daunting prospect.
Despite the challenges, Meta’s Llama 3 team managed to maintain over 90% effective training time. In the 54-day pre-training snapshot, there were 466 job interruptions: 47 planned and 419 unexpected. Planned interruptions were due to automated maintenance, while unexpected issues were primarily related to hardware problems. GPU issues accounted for 58.7% of expected interruptions, with only three requiring significant manual intervention. The rest were automatically managed.
Of the 419 unexpected problems, 148 (30.1%) were caused by various GPU failures, including NVLink issues. Additionally, 72 (17.2%) were attributed to HBM3 memory failures. The high power consumption of NVIDIA’s H100 AI GPUs (around 700W) and the considerable thermal stress they undergo could explain these issues.
To enhance efficiency, Meta’s team streamlined job startup and checkpointing times and developed proprietary diagnostic tools. PyTorch’s NCCL flight recorder proved instrumental in quickly identifying and resolving performance problems, particularly with NCCLX. This tool captures metadata and stack traces, facilitating swift problem resolution.
Straggling GPUs, which can slow down thousands of other GPUs, were addressed with Meta’s in-house tools, identifying problematic GPUs and prioritizing communication issues. This strategy ensured timely resolution of stragglers, minimizing slowdowns and maintaining overall training efficiency.
While mid-day temperature fluctuations impacted training performance by a slight 1-2% variation, it wasn’t a major concern. Dynamic voltage and frequency scaling of the AI GPUs were affected by these minor temperature changes.
The Llama 3 405B LLM training team anticipated another issue: simultaneous power consumption changes from tens of thousands of AI GPUs, potentially straining the data center’s power grid. These fluctuations, reaching tens of megawatts, could push the grid to its limit. Meta ensured its data centers had sufficient power capacity to handle these demands.
It’s worth noting that Meta’s cluster of 16,384 H100 AI GPUs pales in comparison to Elon Musk’s xAI supercomputer, which boasts 100,000 H100 AI GPUs. This difference highlights the sheer scale of Musk’s AI cluster and the impressive power generators required to fuel it.
