The growing competition in the tech world to create the most powerful artificial intelligence system is evident. Elon Musk seems to have taken a big leap forward in the race. On Monday, Musk announced xAI Colossus is now online. The X boss says it is the most powerful AI training system in the world. xAI is Elon Musk’s AI research and development company.
The CEO launched xAI last year to compete with OpenAI; the startup develops a line of large language models called Grok. In May, xAI raised $6 billion at a $24 billion valuation to finance its AI development efforts.
What is xAI Colossus ?
Colossal-AI is an integrated large-scale deep learning system with efficient parallelisation techniques. The system can accelerate model training on distributed systems with multiple GPUs by applying parallelisation techniques. The system can also run on systems with only one GPU.
Colossal AI can be used to train deep models on systems with only one GPU and achieve baseline performances, or it can also be used to train deep learning models on distributed systems with multiple GPUs and accelerate the training process drastically by applying efficient techniques.
Parallelism options are :
-
Data parallel
-
Hybrid parallel
-
MoE parallel
-
Sequence parallel
Elon Musk detailed that Colossus is equipped with 100,000 of Nvidia’s H100 graphics cards. The H100 debuted in 2022 and ranked as the chipmaker’s most powerful AI processor for more than a year. It can run language models up to 30 times faster than Nvidia’s previous-generation GPUs.
One contributor to the H100’s performance is its so-called Transformer Engine module. It’s a set of circuits optimised to run AI models based on the Transformer neural network architecture. The architecture underpins GPT-4o, Meta Platforms Inc.’s Llama 3.1 405B, and many other frontier LLMs.
Musk detailed that xAI plans to double Colossus’ chip count to 200,000 within a few months. He said 50,000 of the new processors will be H200s. The H200 is an upgraded, significantly faster version of the H100 that Nvidia debuted last November.
Nvidia Data Centre's X handle wrote on the platform, "Exciting to see Colossus, the world’s largest GPU #supercomputer, come online in record time. Colossus is powered by @nvidia's #acceleratedcomputing platform, delivering breakthrough performance with exceptional gains in #energyefficiency. Congratulations to the entire team!"