OpenAI has recently taken decisive action against the exploitation of its ChatGPT artificial intelligence (AI) tool by cybercriminals from Russia, North Korea, and China. These threat actors were found leveraging ChatGPT to develop and refine malware, conduct phishing campaigns, and automate various malicious activities.
Russian Cybercriminals’ Exploitation of ChatGPT
A group of Russian-speaking cybercriminals utilized ChatGPT to create and enhance a remote access trojan (RAT) and a credential-stealing tool designed to evade detection. By employing multiple ChatGPT accounts, they prototyped and troubleshooted technical components facilitating post-exploitation and credential theft. Evidence of their activities was observed in a Telegram channel dedicated to such actors.
Although ChatGPT’s large language models (LLMs) refused direct requests to produce malicious content, the threat actors circumvented these restrictions by generating building-block code, which they then assembled into malicious workflows. The outputs included code for obfuscation, clipboard monitoring, and utilities to exfiltrate data using a Telegram bot. While these individual components are not inherently malicious, their combination enabled the development of sophisticated malware.
The operators exhibited a mix of high and low sophistication in their requests. Some prompts required deep knowledge of Windows platforms and iterative debugging, while others automated routine tasks such as mass password generation and scripted job applications. The use of a small number of ChatGPT accounts and iterative development across conversations indicated ongoing development rather than occasional testing.
North Korean Hackers’ Misuse of ChatGPT
A second cluster of activity originated from North Korea, overlapping with a campaign detailed by Trellix in August 2025 that targeted diplomatic missions in South Korea using spear-phishing emails to deliver Xeno RAT. These actors used ChatGPT for malware and command-and-control (C2) development, engaging in specific efforts such as developing macOS Finder extensions, configuring Windows Server VPNs, and converting Chrome extensions to their Safari equivalents.
Additionally, the North Korean threat actors utilized ChatGPT to draft phishing emails, experiment with cloud services and GitHub functions, and explore techniques to facilitate DLL loading, in-memory execution, Windows API hooking, and credential theft.
Chinese Hackers’ Exploitation of ChatGPT
The third set of banned accounts was linked to a Chinese hacking group tracked by Proofpoint under the name UNK_DropPitch (aka UTA0388). This group has been associated with phishing campaigns targeting major investment firms, particularly focusing on the Taiwanese semiconductor industry, using a backdoor dubbed HealthKick (aka GOVERSHELL).
The Chinese threat actors used ChatGPT to generate content for phishing campaigns in English, Chinese, and Japanese; assist with tooling to accelerate routine tasks such as remote execution and traffic protection using HTTPS; and search for information related to installing open-source tools like nuclei and fscan. OpenAI described the threat actor as “technically competent but unsophisticated.”
Additional Malicious Activities Disrupted by OpenAI
Beyond these three clusters, OpenAI also blocked accounts used for scam and influence operations:
– Online Fraud Networks: Networks likely originating in Cambodia, Myanmar, and Nigeria abused ChatGPT to defraud individuals online. These networks used AI for translation, message writing, and creating social media content to advertise investment scams.
– Surveillance Operations: Individuals apparently linked to Chinese government entities used ChatGPT to assist in surveilling individuals, including ethnic minority groups like Uyghurs, and analyzing data from Western or Chinese social media platforms. The users asked the tool to generate promotional materials about such tools but did not use the AI chatbot to implement them.
– Influence Campaigns: A Russian-origin threat actor linked to Stop News and likely run by a marketing company used AI models to generate content and videos for sharing on social media sites. The generated content criticized the role of France and the U.S. in Africa and Russia’s role on the continent. It also produced English-language content promoting anti-Ukraine narratives.
– Covert Influence Operations: A covert influence operation originating from China, codenamed “Nine—emdash Line,” used AI models to generate social media content critical of the Philippines’ President Ferdinand Marcos, as well as create posts about Vietnam’s alleged environmental impact in the South China Sea and political figures and activists involved in Hong Kong’s pro-democracy movement.
In two different cases, suspected Chinese accounts asked ChatGPT to identify organizers of a petition in Mongolia and funding sources for an X account that criticized the Chinese government. OpenAI stated that its models returned only publicly available information as responses and did not include any sensitive information.
Adaptation of Tactics by Threat Actors
One of the most interesting takeaways from the report is that threat actors are trying to adapt their tactics to remove possible signs that could indicate that the content was generated by an AI tool. For instance, one of the scam networks from Cambodia asked the model to remove the em-dashes (long dash, –) from their output or appears to have removed the em-dashes manually before publication. This suggests that the threat actors were aware of online discussions identifying em-dashes as a possible indicator of AI usage.
OpenAI’s Commitment to AI Safety
OpenAI reiterated that its tools provided the threat actors with novel capabilities that they could not otherwise have obtained from multiple publicly available resources online, and that they were used to provide incremental efficiency to their existing workflows.
The findings from OpenAI come as rival Anthropic released an open-source auditing tool called Petri (short for “Parallel Exploration Tool for Risky Interactions”) to accelerate AI safety research and better understand model behavior across various categories like deception, sycophancy, encouragement of user delusion, cooperation with harmful requests, and self-preservation.
Petri deploys an automated agent to test a target AI system through diverse multi-turn conversations involving simulated users and tools. Researchers give Petri a list of seed instructions targeting scenarios and behaviors they want to test. Petri then operates on each seed instruction in parallel. For each seed instruction, an auditor agent makes a plan and interacts with the target model in a tool use loop. At the end, a judge scores each of the resulting transcripts across multiple dimensions so researchers can quickly search and filter for the most interesting transcripts.
Conclusion
OpenAI’s proactive measures to disrupt the misuse of ChatGPT by state-sponsored and criminal actors underscore the dual-use nature of AI technologies. While AI offers significant benefits, it also presents challenges in cybersecurity, necessitating continuous vigilance and adaptive strategies to mitigate potential threats.
