This Bot Is the Most Dangerous Thing Meta’s Made Yet

Though the deliciously entertaining chaos at Twitter and the collapse of FTX were the main tech industry stories of last week, they were far from the only disasters to unfold. On Nov. 15, Meta launched a demo of an AI dubbed Galactica. In a statement, Meta told The Daily Beast that the model was trained “on 106 billion tokens of open-access scientific text and data. This includes papers, textbooks, scientific websites, encyclopedias, reference material, knowledge bases, and more.”

Think of it as a kind of academic search engine on steroids. With a simple prompt, Galactica “can summarize academic papers, solve math problems, generate Wiki articles, write scientific code, annotate molecules and proteins, and more,” the company wrote. It was going to be a cheat code for academics and researchers. No more hunting down the right study or paper for your research. No more being delayed by a tricky equation. Meta’s new bot could take care of all that for you with just a few keystrokes.

But just two days after going live, the company took the Galactica demo down.

Anyone even vaguely familiar with other large-language models—or AI that can read and generate texts—could have seen this coming. These bots have a long and sordid track record of producing biased, racist, sexist, and overall problematic results—and Galactica was no exception.

Within just a few hours of going live, Twitter users began posting instances where the new Meta bot would generate completely fake and racist research. One user discovered Galactica making up information about Stanford University researchers creating “gaydar” software to find gay people on Facebook. Another was able to get the bot to create a fake study about the benefits of eating crushed glass.

The bot would also completely filter out queries such as queer theory, AIDs, and racism. But perhaps one of the most disconcerting things about the entire affair, though, was the fact that it would create entirely fake studies and attribute them to actual scientists. Michael Black, the director at the Max Planck Institute for Intelligent Systems in Germany, pointed out in a Twitter thread several instances in which the Galactica would create false citations to real-world researchers.

Meanwhile, these citations would be attributed to very convincing text generated by the model—making it seem, on its face, entirely plausible and real.

Meta took down the demo just two days after launch—but the damage had already been done. It even led to Meta’s chief AI scientist Yann LeCun throwing a hissy fit about those pesky Twitter users pointing out glaring problems with the model. “It’s no longer possible to have some fun by casually misusing it. Happy?” he wrote.

On its face, a model like Galactica seems like a genuinely good idea for solving a big problem. There is a glut of scientific data out in the world, and currently no good way to wrap our tiny human minds around it all. It’s a worthy endeavor to design software able to synthesize decades or even centuries of work and deliver it to researchers in an easy and digestible way. Such a tool could push science and technology development to new heights, and Meta should get credit for trying to make this possible.

But the company completely shit the bed on this one—and, instead, created what is arguably its most dangerous AI model yet.

“I imagine that even with its many predictable flaws, there are desirable uses of such a system,” Vincent Conitzer, a professor of computer science at Carnegie Mellon University in Pittsburgh, told The Daily Beast. “My impression is that Meta would have done better by putting more effort into this public release, by doing more serious user studies first, drawing attention to desirable uses of the system, and being honest and forthcoming about undesirable uses.”

At its core, the problem is that Galactica is trying to be a source of authority. It’s been trained to generate material that reads like something that was written by flesh-and-blood academics. So when it starts to just make things up whole-cloth, that means that it can very quickly turn into a tool for bad actors pushing their own agendas.

I Flirted With Meta’s New Chatbot and Things Got Weird

It’s not a stretch to imagine a world in which someone like a COVID or climate change denier uses Galactica to create “studies” that fit their false narratives and worldviews. Race realists could use it to promote their racist and biased beliefs. Hell, you could start a TikTok trend of people eating crushed glass and point to research to back up why it’s a good idea.

These problems are exacerbated by the fact that the made up papers and studies are sometimes tied back to actual researchers who had nothing to do with it, putting real people’s livelihoods and reputations in danger. Imagine working your whole life as an astrophysicist only to see your name cited in a fake paper about how the Earth is actually flat.

To their credit, Meta does note on Galactica’s website that the model has major limitations and that it “can hallucinate.”

“There are no guarantees for truthful or reliable output from language models, even large ones trained on high-quality data like Galactica,” the website says. “NEVER FOLLOW ADVICE FROM A LANGUAGE MODEL WITHOUT VERIFICATION.” They also add that the generated text from Galactica might appear “very authentic and highly-confident” but could still be wrong.

Image Generators Like DALL-E Are Mimicking Our Worst Biases

While those warnings are well and good, it comes at complete odds with the messaging of Galactica being this massive game-changer for scientific research. In order for this bot to work, it needs to be authoritative and trustworthy. If it isn’t, what is even the point of all this?

"Maybe some of the undesirable uses could have been prevented outright, but in any case presenting it in a more modest way and asking the community for help could have made the public reaction less adversarial as well," Conitzer said. "If you put a system like this out there without taking the risks and shortcomings seriously, Twitter will never be kind," he said.

For now, Meta’s Galactica has been taken away from the public—presumably to wait until the embarrassing launch is largely forgotten and to tighten up the model before a potential formal roll-out. No matter what, though, there’s no real pathway for this going well. Large language models have shown time and again that they’re prone to the same biases and problematic behavior to which we’re prone. When you infuse scientific and academic research into the mix, it can and will become even more dangerous.

In all, Galactica (at least its current rendition) is like having a hand grenade in a locked room. Once the pin is pulled, it’s going to get very messy very quickly.

Read more at The Daily Beast.

Got a tip? Send it to The Daily Beast here

Get the Daily Beast's biggest scoops and scandals delivered right to your inbox. Sign up now.

Stay informed and gain unlimited access to the Daily Beast's unmatched reporting. Subscribe now.