Deepfake deception: Indian woman's identity stolen for erotic AI content
It took just a few days for Indian Instagram sensation Babydoll Archi to double her following to 1.4 million, thanks to a couple of viral social media moments.
One was a video that showed her in a red sari, dancing seductively to Dame Un Grr – a Romanian song. And a photo posted on the platform showed her posing with American adult film star Kendra Lust.
Suddenly everyone wanted to know about her - and the name Babydoll Archi trended in Google search and spawned countless memes and fan pages. But there was one issue about to emerge – there was no real woman behind the online sensation.
The Instagram account was fake, although the face it used had uncanny likeness to a real woman - a homemaker in Dibrugarh city in Assam, whom we'll call Sanchi.
The truth unravelled after her brother lodged a police complaint. Pratim Bora, Sanchi's ex-boyfriend, was arrested.
Senior police officer Sizal Agarwal who's heading the investigation told the BBC that Sanchi and Bora had a falling out and the AI likeness he created was to exact "pure revenge" on her.
Bora - a mechanical engineer and a self-taught artificial intelligence (AI) enthusiast - used private photos of Sanchi to create a fake profile, Ms Agarwal said.
Bora is in custody and has not made any statements yet. The BBC has reached out to his family and will update the article when they speak.
Babydoll Archi was created in 2020 and the first uploads were made in May 2021. The initial photos were her real pictures that had been morphed, Ms Agarwal said.
"As time passed, Bora used tools such as ChatGPT and Dzine to create an AI version. He then populated the handle with deepfake photos and videos."
The account started picking up likes from last year but it started gaining traction from April this year, she added.
The short two-paragraph complaint to the police submitted by Sanchi's family on the night of 11 July came with printouts of some photos and videos as evidence.
Ms Agarwal says it did not name anyone because they had no idea who could be behind it.
Babydoll Archi was not an unfamiliar name for the police. Ms Agarwal says they had also seen media reports and comments speculating that she was AI generated, but there had been no suggestion that it was based on a real person.
Once they received the complaint, police wrote to Instagram asking for the details of the account's creator.
"Once we received information from Instagram, we asked Sanchi if she knew any Pratim Bora. Once she confirmed, we traced his address in the neighbouring district of Tinsukia. We arrested him on the evening of 12 July."
Ms Agarwal says the police have "seized his laptop, mobile phones and hard drives and his bank documents since he had monetised the account".
"The account had 3,000 subscriptions on linktree and we believe he had earned 1m rupees from it. We believe he made 300,000 rupees in just five days before his arrest," she added.
Ms Agarwal says Sanchi is "extremely distraught - but now she and her family are receiving counselling and they are doing better".
There really is no way to prevent something like this from happening, "but had we acted earlier, we could have prevented it from gaining so much traction", Ms Agarwal said.
"But Sanchi had no idea because she has no social media presence. Her family too had been blocked out from this account. They became aware only once it went viral," she added.
Meta has not responded to the BBC queries on the case, but it generally does not allow posting of nudity or sexual content. And last month, CBS reported that it removed a number of ads promoting AI tools used to create sexually explicit deepfakes using images of real people.
The Instagram account of Babydoll Archi, which had 282 posts, is no longer available to public - although social media is replete with her photos and videos and one Instagram account seems to have all of them. The BBC has asked Meta what they are planning to do about it.
Meghna Bal, AI expert and lawyer, says what happened to Sanchi "is horrible but almost impossible to prevent".
She can go to the court and seek the right to be forgotten, and a court can order the press reports that named her to be taken down but it's hard to scrub all the trace from the internet.
What happened to Sanchi, she says, is what's always been happening to women, where their photos and videos are circulated as revenge.
"It's now a lot easier to do because of AI, but such incidents are still not as common as we expect - or they could be under-reported because of stigma or people being targeted may not even know about it as in the present case," Ms Bal says. And people watching it had no incentive to report it to the social media platform or cybercrime portal, she adds.
In their complaint against Mr Bora, police have invoked sections of law that deal with sexual harassment, distribution of obscene material, defamation, forgery to harm reputation, cheating by personification and cybercrime. If found guilty, Mr Bora could get up to 10 years in prison.
The case which has also led to outrage on social media in recent days has seen some seeking tougher laws to deal with such cases.
Ms Bal believes there are enough laws to take care of such cases, but whether there's scope for new laws to deal with generative AI companies has to be looked at.
"But we also have to remember that deepfakes are not necessarily bad and laws have to be carefully crafted because they can be weaponised to chill free speech."
BBC
