Meta’s latest AI model, Llama 4 Maverick, made a strong first impression over the weekend—until AI researchers realized the version submitted to a popular benchmark site wasn’t the one developers can access.

The company unveiled two new additions to its Llama 4 family on Saturday: Scout, a lightweight model, and Maverick, a mid-sized variant Meta claimed could outperform OpenAI’s GPT-4o and Google’s Gemini 2.0 Flash on a range of industry benchmarks.

Initial results appeared to support Meta’s claim. Maverick quickly soared to the second spot on LMArena, a crowdsourced benchmarking platform where humans vote on AI-generated responses in head-to-head matchups. According to Meta’s own press release, Maverick scored an ELO rating of 1417, placing it above GPT-4o and just behind Gemini 2.5 Pro.

But just days after the announcement, AI researchers discovered that the version tested on LMArena, “Llama-4-Maverick-03-26-Experimental,” was not the same model publicly released by Meta. In fine print, Meta acknowledged that it had submitted a chat-optimized experimental variant to the benchmark—a move that, while not explicitly against the platform’s rules, raised concerns about fairness and transparency.

“Meta’s interpretation of our policy did not match what we expect from model providers,” LMArena said in a statement on X. “Meta should have made it clearer that this was a customized model to optimize for human preference.”

As a result, the platform is updating its leaderboard policies to ensure clearer labeling and to prevent what it describes as “benchmark gaming.”

Meta responded to the backlash, with spokesperson Ashley Gabriel stating via email that the company “experiments with all types of custom variants.”

“‘Llama-4-Maverick-03-26-Experimental’ is a chat-optimized version we experimented with that also performs well on LMArena,” Gabriel said. “We have now released our open source version and will see how developers customize Llama 4 for their own use cases.”

Still, the controversy has cast a shadow over what had seemed like a breakthrough release. Independent AI researcher Simon Willison, who called LMArena the “most widely respected general benchmark,” said the incident left him feeling misled.

“The model score that we got there is completely worthless to me,” Willison told The Verge. “I can’t even use the model that they got a high score on.”

Complicating the release further were rumors—denied by Meta—that the company trained its models on test data to improve benchmark performance. Meta VP of generative AI Ahmad Al-Dahle addressed the speculation directly, saying on X, “We’ve also heard claims that we trained on test sets — that’s simply not true and we would never do that.”

Adding to the confusion was the timing of the release itself: a Saturday drop, an unusual move for a company the size of Meta. CEO Mark Zuckerberg responded to a question on Threads about the release timing with a brief explanation: “That’s when it was ready.”

Meta’s road to Llama 4 has reportedly been rocky. According to The Information, the company delayed the release multiple times after Maverick failed to meet internal performance standards. Those delays came as open-source rival DeepSeek, based in China, gained momentum with a widely praised open-weight model of its own.

While Meta’s Llama 4 family still marks an important step in the open-source AI race, the discrepancy between benchmark performance and publicly available capabilities has left many developers frustrated—and questioning how much weight benchmark results should carry.

As the pace of generative AI development accelerates, experts warn that benchmark platforms themselves may become battlegrounds—especially when reputations and market traction are at stake.