An unprecedented scale model has emerged in the open-source AI camp! Moonshot AI has announced 'Kimi K3,' a massive Mixture-of-Experts (MoE) model with 2.8 trillion (2.8T) parameters, and stated they will fully open-source the weights on July 27th. It is the largest in open-weight history, and it is not just massive in size; both its architecture and benchmark scores offer some truly intriguing aspects.

The most striking feature is the extreme sparse routing structure. Out of 896 experts, only 16 are activated per token, meaning it uses only about 50 billion (50B) active parameters for actual inference. Coupled with a hybrid attention technique called 'Kimi Delta Attention,' it supports a massive context window of 1 million tokens. By partnering with the vLLM team for Day-0 support and cache optimization, they seem quite serious about the open-source ecosystem. 😄

What about real-world performance? It has taken the #1 spot (1,679 Elo) in the Arena.ai coding category, demonstrating unmatched capability in coding and long-context reasoning. According to Artificial Analysis, a benchmark site, it is now hot on the heels of closed-source flagship models.

However, to avoid it being just 'pie in the sky,' one must overcome the hardware barrier. Even in the quantized MXFP4 format, the model size reaches approximately 1.4TB; it is said that running it properly locally requires an enterprise super-node cluster configured with 64 or more accelerators. While the license terms will be revealed on the 27th when the weights are released, it seems less like a tool for individuals and more like a powerful weapon for securing open-source sovereignty in cloud-based distributed agent environments or at the enterprise level.