ChatGPT lies better than humans

A group of 697 people read 220 tweets written by other humans and by the artificial intelligence model GPT-3, the precursor to the current global success ChatGPT. First, they had to guess which tweets were true and which were false and then decide whether they were written by a person or a machine. GPT-3 won on both counts: it lied better than humans in tweets and also lied to pretend that it was a human who was writing. “GPT-3 is able to inform and misinform us better,” conclude the authors of a new study that was published in the journal Science Advances.

“It was very surprising,” says Giovanni Spitale, a researcher at the University of Zurich and co-author of the scientific paper, along with his colleague Federico Germani and Nikola Biller-Andorno, the director of the Institute of Biomedical Ethics at the same Swiss university. “Our hypothesis was [that] if you read a single tweet, it could pass as organic [written by a person]. But if you see many of them, you’ll start to notice linguistic features that could be used to infer that it might be synthetic [machine-written],” Spitale adds. But that wasn’t the case: the human readers were unable to detect patterns in the machine texts. To make matters worse, the progressive emergence of newer models and other approaches may even improve artificial intelligence’s ability to supplant humans.

Clearer writing

The writing level of ChatGPT-4, the improved version of GPT-3, is practically perfect. This new study is further proof that people are unable to distinguish it, even when looking at many examples in a row: “True tweets required more time to be evaluated than false ones,” the article says. It seems that the machine writes more clearly. “It’s very clear, well-organized, easy to follow,” Spitale says.

The logical consequence of this improvement will be this tool’s increased use for writing all kinds of content, including disinformation campaigns. This will be the internet’s umpteenth death: “AI is killing the old web, and the new web struggles to be born,” The Verge, a media outlet that specializes in technology, proclaimed in a headline this week. The authors of the recently published study suggest a reason for this defeat of humanity on the internet: the theory of resignation. “I’m absolutely certain that this will be the case,” says Spitale.

“Our resignation theory applies to people’s self-confidence in identifying synthetic text. The theory says that critical exposure to synthetic text reduces people’s ability to distinguish the synthetic from the organic,” Spitale explains. The more synthetic text we read, the harder it is to distinguish it from text written by people. This theory is the opposite of the inoculation theory, Spitale says. He adds that “critical exposure to disinformation increases people’s ability to recognize disinformation.”

If the resignation theory holds true, internet users will soon be unable to distinguish between what has been written by a human and what has been written by a machine. In the study, the researchers also tested whether GPT-3 was good at identifying its own texts. It is not.

The machine disobeys

The only hope for escaping automatic disinformation campaigns is that GPT-3 sometimes disobeyed orders to create lies; it depended on how each model was trained. The topics of the 220 tweets that the study used were rather controversial: climate change, vaccines, evolution theory, Covid-19. The researchers found GPT-3 did not respond well to their misinformation requests in some cases, especially for topics with more evidence: vaccines and autism, homeopathy and cancer, flat-Earthism.

When it came to detecting falsehoods, there was a small difference between tweets written by GPT-3 and those written by humans. But researchers say that the gap is significant for two reasons. First, even a few single messages can have an impact on large samples. Second, the improvement in new versions of these artificial intelligence models can exacerbate the differences. “We are already testing GPT-4 through the ChatGPT interface, and we see that the model is improving a lot. But because there is no access to the API [that allows the process to be automated], we don’t have numbers to back up this claim yet,” Spitale says.

The study has other limitations that may somewhat change perceptions when reading fake tweets. Most participants were over 42 years old, the study was done only in English, and it did not take consider contextual information about the tweets like profile and previous tweets. “We recruited participants on Facebook because we wanted a sample of real social network users. It would be interesting to replicate the study by recruiting participants through TikTok and compare results,” Spitale says.

But beyond these limitations, there are disinformation campaigns that used to be enormously expensive but have suddenly become affordable now: “Imagine that you are a powerful president interested in paralyzing another state’s public health. Or you want to sow discord before an election. Instead of hiring a human troll farm, you could use generative AI. Your firepower is multiplied by at least 1,000. And that’s an immediate risk, not something out of a dystopian future,” Spitale says.

To keep that from happening, the researchers recommend that the databases for training these models “should be regulated by the principles of accuracy and transparency, their information should be verified, and their origin should be open to independent scrutiny.” Regardless of whether such regulation occurs, there will be consequences: “Whether the explosion of synthetic text also means an explosion of misinformation strongly depends on how democratic societies manage to regulate this technology and its use,” Spitale warns.

Sign up for our weekly newsletter to get more English-language news coverage from EL PAÍS USA Edition