Mystery Deepens as Super Powerful AI Appears, Then Disappears

In and Out

AI circles were abuzz over the weekend after users discovered a mysterious new AI model on the website LMSYS Chatbot Arena that seems to rival the capabilities of — or perhaps even surpass in some respects, according to some enthusiastic accounts — OpenAI's GPT-4.

But after just a few days of manic hype and frantic testing, the AI model, known only as "gpt2-chatbot," disappeared on Tuesday. LMSYS later confirmed on X-formerly-Twitter that the AI model had been taken down due to "unexpectedly high traffic."

That could do little to cool the wild speculation. Many believe that "gpt-2" is an early, secret preview of the next model from a major AI company like OpenAI. Fueling the hype, Sam Altman, the CEO of the Microsoft-backed startup, cryptically posted about the mystery model on Monday.

"I do have a soft spot for gpt2," Altman wrote on X. (OpenAI already released a model called GPT-2 back in 2019, which was impressive at the time but now sorely dated compared to more recent offerings.)

No Slouch

After the limited window of testing, those who got their hands on the AI model seem to agree that its capabilities are impressive, especially for something that could have been made outside of the major AI companies.

Ethan Mollick of the University of Pennsylvania wrote on X that "it appears to be in the same rough ability level as GPT-4," then later suggested it might even be better. Some AI researchers have highlighted gpt2's ability to solve tough math problems that still trip up other leading AI models.

What's up for debate, however, is whether it represents a step up from current LLMs like GPT-4.5 or Anthropic's Claude 3 Opus. After conducting its own testing, Ars Technica concluded that "whatever it is, it's probably not GPT-5," OpenAI's long-rumored upcoming model.

Open Secret

Is this model the work of some upshot trying to stamp itself on the world stage? Possibly. But if you take a more cynical view, it reeks of a marketing stunt.

"I think it may well be an OpenAI stealth preview of something," acclaimed programmer and AI researcher Simon Willison told Ars Technica.

Willison bemoaned that if it were a stunt, it would go against the "neutral" ethos of the platform where it appeared: LMSYS, an organization dedicated to the open development of large language models whose "Chatbot Arena" serves as a hub to benchmark and test these models. In response to Willison's probing, LMSYS confirmed that it had "partnered with several model developers to bring their new models" to the platform in the past.

"The whole situation is so infuriatingly representative of LLM research," Willison told Ars. "A completely unannounced, opaque release and now the entire Internet is running non-scientific 'vibe checks' in parallel."

More on AI: Internet Horrified at AI App for Cloning Dead Family Members