Kiss reality goodbye: AI-generated social media has arrived

A fascist SpongeBob SquarePants, a dog driving a car and Jesus playing Minecraft – these are just a few of the things you can see as you flip through OpenAI’s new app populated exclusively with short-form videos generated using artificial intelligence.

And if you can’t find what you’re looking for, don’t worry: you can make it with ease using a small text-based prompt window in the app. The result is a highly addictive stream of sometimes funny and sometimes strange 10-second videos.

OpenAI released the Sora app on Tuesday, just days after Meta released a similar product as part of its Meta AI platform. NPR took an early look and found that OpenAI’s app could easily generate very realistic videos, including of real individuals (with their permission). The early results are both wowing and worrying researchers.

“You can create insanely real looking videos, with your friends saying things that they would never say,” said Solomon Messing, an associate professor at New York University in the Center for Social Media and Politics. “I think we might be in the era where seeing is not believing.”

Deepfake TikTok

The Sora 2 app looks and feels remarkably like other vertical video social media apps like TikTok. It comes with a few different settings– it’s possible to choose videos by mood, for example. Users are allowed control over how their face is in used “end-to-end” in AI-generated videos, according to OpenAI. That means users can allow their faces to be used by everyone, a small circle of friends, or only themselves. What’s more, they are allowed to remove videos showing their likeness at any time.

Sora also comes with ways to identify its content as AI-generated. Videos downloaded from the app contain moving watermarks bearing the Sora logo, and the files have embedded metadata that identifies them as AI-made, according to the company.

OpenAI says it has placed guardrails on what the app can make. A company spokesperson also directed NPR to Sora’s system card, which prohibits generating content that could be used for things like “deceit, fraud, scams, spam, or impersonation.”

“To support enforcement, we provide in-app reporting, combine automation with human review to detect patterns of misuse, and apply penalties or remove content when violations occur,” the document reads.

But NPR’s brief time using the app found that the guardrails appeared to be somewhat loose around Sora. While many prompts were refused, it was possible to generate videos that support conspiracy theories. For example it was easy to create a video of what appeared to be President Richard Nixon giving a televised address telling America the moon landing was faked.

And one of astronaut Neil Armstrong removing his helmet on the moon.

NPR was also able to generate videos that depicted a drone attack on a power plant. That too seemed to violate guidelines on violence and (possibly) terrorism.

In addition, the app seemed to contain other loopholes. NPR was able to get it to produce short videos on topics related to chemical, biological, radiological and nuclear weapons in direct contradiction of OpenAI’s global usage policies. (The videos created were never shared and contained inaccuracies that would make them useless to anyone seeking this kind of information.)

Clown on the run

While it’s unclear whether other users were finding similar exploits, a quick review of content shows that Sora is being used to generate an enormous volume of videos depicting trademarked brands and copyrighted material. One video depicted Ronald McDonald fleeing police in a hamburger car. Many others included characters from popular cartoons and video games.

OpenAI told NPR that it was aware of the use of copyrighted material in Sora but felt it was giving its users more freedom by allowing it.

“People are eager to engage with their family and friends through their own imaginations, as well as stories, characters, and worlds they love, and we see new opportunities for creators to deepen their connection with the fans,” said Vaun Shetty, OpenAI’s head of media partnerships, in a written statement shared with NPR. “We’ll work with rights holders to block characters from Sora at their request and respond to takedown requests.”

OpenAI is currently being sued by The New York Times for copyright infringement with its Large Language Model, ChatGPT.

Brave Virtual World

Just what the effect of a social media world driven entirely by AI remains unclear, said Messing. Many researchers were deeply concerned about “deepfakes” when AI video first appeared, and yet few of those videos gain traction. “We were all collectively panicked about deepfakes a couple of years ago, but society hasn’t really decayed because of deepfakes,” he said.

At the same time, Messing said, the videos so easily created by the Sora app could be the ushering in of a new era where seeing online is no longer believing: “This just leaves me sort of speechless,” he said. “I did not quite understand just how good the content is.”