Telegram’s artificial intelligence bot, a core feature of the popular messaging app, is being misused to generate deepfake pornography. In response to this alarming trend, South Korean police and the Ministry of Education have formed a task force to investigate and tackle the problem.
On Wednesday, the Cybercrime Investigation Unit of the Seoul Metropolitan Police Agency announced that it has initiated investigations into eight Telegram AI bots used to illegally edit and create fake nude photos. This announcement came just a day after the police agency revealed a broader crackdown on crimes involving deep learning and AI to produce fake pornographic images and videos.
According to a police officer from the Cybercrime Investigation Unit of the Seoul Metropolitan Police Agency, who requested to remain anonymous, the Telegram bot, known simply as “Telegram bot,” is a program that can create fake pornography by editing ordinary photos with nude images.
The Telegram AI bot is one of the application’s features, originally designed to offer user benefits such as keyword alerts, image searches, and simple games. The company states on its website that creating a new bot takes only “a few hours” and provides users with a straightforward application programming interface (API). However, bad actors are exploiting these easily accessible bot features to generate fake pornography.
The police agency has identified 227,000 users who have utilized the AI bot to create fake pornographic content. However, it is unclear whether this number represents individuals who directly used the bot or simply accessed it, according to a police officer from the Cybercrime Investigation Unit of the Seoul Metropolitan Police Agency. The police have stated that they cannot disclose further details about the investigation at this time.
The agency revealed that between January and July 2024, 178 suspects were identified in connection with deepfake pornography crimes, with 131 of them being teenagers. This group accounts for 73.6 percent of the total suspects.
To address the urgent issue of illegal deepfake pornography being created and distributed within schools, the Ministry of Education held a press briefing on Wednesday. The ministry revealed that from January to August 27, 2024, they received 196 reports of such incidents, with 186 victims being students and 10 being teaching staff members. The ministry has referred 179 of these cases for further investigation.
During a press briefing, Oh Seok-hwan, Vice Minister of Education, announced the creation of a task force to address emergency deepfake pornography crimes in schools. “We aim to protect students who have already been victimized and to strictly punish the perpetrators,” said Oh.
Related article: Telegram CEO arrested in connection with alleged misuse of app for criminal activities
Pavel Durov, CEO of the messaging app Telegram, was arrested in Paris on August 24 for allegedly permitting the platform to be used for criminal activities, according to French media reports.
In a press release on Monday, the Paris judicial court announced that Pavel Durov, founder and CEO of the messaging app Telegram, was detained at Le Bourget Airport on August 24 and taken into custody by Paris police. The arrest and subsequent investigation followed an inquiry conducted by the Paris prosecutor’s office.
The Paris prosecutor’s office stated that they are investigating allegations that Telegram is being used for criminal purposes, including illegal transactions, child pornography, drug trafficking, fraud, and money laundering, and that the platform is facilitating these activities. However, Telegram has denied any involvement in cybercrime.
On the social media platform X, Telegram stated that it adheres to EU laws, including the Digital Services Act. The company emphasized that its moderation practices meet industry standards and are continually improving. Telegram added, “It is absurd to claim that the platform or its owners are responsible for the misuse of the platform.”
Pavel Durov was born in Russia and holds dual citizenship in the United Arab Emirates and France. He founded the Russian social networking service VKontakte in 2006. However, when the Russian government demanded that he hand over the personal data of Ukrainian users, he refused and subsequently sold the platform. READ MORE
