Malicious Intent: Microsoft and OpenAI Identify APT Groups Weaponizing GenAI LLMs

• This week, Microsoft and OpenAI disclosed that nation-state threat groups are actively leveraging large language models to automate malicious operations, translate technical papers, generate believable phishing content, and gain vulnerability knowledge.
• The two companies identified five Advanced Persistent Threat groups and terminated associated accounts.

It’s official: state-affiliated threat actors from North Korea, China, Iran, and Russia are using large language models (LLMs) maliciously. Consequently, OpenAI has terminated accounts associated with these Advanced Persistent Threat (APT) groups.

The discovery is somewhat unsurprising, given the usefulness of generative AI, which appeals to anyone keen on polishing their skills, whether they are used for good or bad. The five APT groups, which OpenAI believes may have access to advanced technology, large financial resources, and skilled personnel, were using the company’s generative AI tools to query open-source information, translate and find coding errors, and run basic coding tasks.

In a separate post, Microsoft said that the state-sponsored threat actors sought to improve software scripts, malware and other coding-related tasks, information on potential victims’ industries, locations, and relationships, and used it to and used it to expand their knowledge of native languages.

The five APT groups, two from China and one each from North Korea, Iran and Russia, whose linked OpenAI accounts were terminated are:

See More: Understanding the Cybersecurity Implications of ChatGPT

The good news is that the APT groups weren’t engaging LLMs to create malware directly and were mainly using it for assistance on various topics. Nevertheless, Microsoft and OpenAI’s report indicates that generative AI’s use to achieve operational efficiency spans white and black hats alike.

“Generative AI is amazing, but it’s mostly helping humans be more efficient rather than making breakthroughs. I believe those threat actors are using LLMs to write code (like malware) faster, but it’s not noticeably impactful because they already had malware. It’s possible they’re able to be more efficient, but at the end of the day, they aren’t doing anything new yet,” Joseph Thacker, principal AI engineer and security researcher at AppOmni, told Spiceworks over email.

“However, if a threat actor found a novel attack use case, it could still be in stealth and not detected by these companies yet, so it’s not impossible. I have seen fully autonomous AI agents that can ‘hack’ and find real vulnerabilities, so if any bad actors have developed something similar, that would be dangerous. And open source models like Mixtral are high quality and could be used at scale in novel ways.”

Additionally, Microsoft also identified the following nine more LLM-themed tactics, techniques, and procedures (TTPs):

• LLM-informed reconnaissance
• LLM-enhanced scripting techniques
• LLM-aided development
• LLM-supported social engineering
• LLM-assisted vulnerability research
• LLM-optimized payload crafting
• LLM-enhanced anomaly detection evasion
• LLM-directed security feature bypass
• LLM-advised resource development

“Companies should remain vigilant. Keep doing the basics right. Bad actors will likely be able to deploy malware at a larger scale or on systems they previously didn’t have support for. LLMs are pretty good at translating code from one language or architecture to another. I can see them converting their malicious code into new languages they previously weren’t proficient in,” Thacker added.

“Allowing AI systems to make decisions is convenient. That means many products will incorporate it without adequate security testing. We will see where this leads very soon.”

“Keep in mind that nearly every AI product is SaaS-based. These apps often handle sensitive data and can be a prime target for cyber attacks, so securing them is extremely important, but that requires an app-centric approach.”

Michael Rinehart, VP of AI at Securiti, agrees with Thacker. “Organizations should adopt application-specific models tailored for specific tasks, possibly supplemented by knowledge bases. These models provide high value for use cases such as Q&A systems,” Rinehart told Spiceworks, which is one of the two-tier security approach he suggested.

“Secondly, an advanced monitoring system should be implemented to scrutinize access to and communications with these models for privacy and security issues. This layered approach provides significant flexibility and improved alignment with governance and data protection principles. It also allows organizations to leverage both traditional and cutting-edge security techniques for LLMs to mitigate the risks associated with Generative AI.”

How can organizations defend against generative AI or LLM-based attacks? Share with us on LinkedInOpens a new window , XOpens a new window , or FacebookOpens a new window . We’d love to hear from you!

Image source: Shutterstock

MORE ON AI AND CYBERSECURITY

• Predictive Security – The New Frontier
• Fry the Phish this Valentine’s Day: How to Thwart Online Scammers Using AI
• How to Combat Deepfakes in the Workplace
• The State of AI in Cybersecurity 2023: The “Lake Wobegon Effect”
• Black Hat 2023: Delving Into the Intersection of Artificial Intelligence and Cybersecurity