Generative AI in Cybersecurity: Synthetic Cyber Teams and Threats
In a session at the Data Science Innovators Conference 2023, Carnegie Mellon's Ravi Starzl said generative AI will empower attackers and defenders alike. He proposed pairing human operators with specialist language models and showed local demos for vulnerability and threat intelligence discovery.
By The AI Enterprise Magazine Team4 min read
Key takeaways
- Attackers gain too. Starzl said AI will make hackers more capable by orders of magnitude, with patience and persistence at scale.
- LLMs are not magic. He described them as vector math and attention, not sentient and not black boxes.
- Synthetic cyber teams. Humans stay in the loop while specialist AI does the heavy lifting.
- Local demos. A self-hosted model over about 198,000 CVE reports surfaced vulnerabilities through conversation.
- Build to purpose. Pick specific use cases, set strategy and evaluate results.
In a session at the Data Science Innovators Conference 2023, Ravi Starzl of Carnegie Mellon University's Language Technologies Institute argued that generative AI will reshape cyber operations for attackers and defenders alike. Starzl, who said he is an adjunct professor at Carnegie Mellon and the University of Colorado Boulder and works with the Army Cyber Institute, explained how large language models work, proposed pairing human operators with specialist AI, and showed demonstrations he built himself.
Why does generative AI matter for cybersecurity?
Starzl said large language models turned automated use of language from science fiction into something anyone can try, citing ChatGPT, Google's Bard and Meta's Llama. He called it probably the fastest-adopted technology in human history, and said it is "literally just getting started."
He pointed to two features that matter for security. Much of the development, he said, is oriented toward malicious activity, including cyber threat activity and information operations. And because large language models descended from translation systems and can translate fluently among roughly 90 languages, language barriers have effectively disappeared for attackers. He added that these models are surpassing the pattern recognition in standard security information and event management systems.
How do large language models actually work?
Starzl gave a deliberately high-level explanation, with what he called artistic license. Training text is converted into numeric representations in a vector space, and an attention mechanism captures how important each word is to every other word. In his example, "I" and "apples" relate more closely than "I" and "red" in "I love red apples."
When a user submits a prompt, the same process is applied to it and the model generates a response one token at a time, feeding its own output back in, which is why replies appear to be typed out. Starzl stressed that these systems are not black boxes and not sentient. They are computing machinery, he said, doing "fairly straightforward linear algebra," and their apparent intelligence comes from the prompt and the training data. He added that human feedback and reinforcement learning models help train them.
He said the bottlenecks are training data and task definition. General chatbots often hallucinate or fail when pushed into specifics, which is why he expects specialized AI for areas such as cyber operations, whose data mixes code, domain language and human language.
What are synthetic cyber teams?
Starzl said AI will make attackers more capable "by orders of magnitude," enabling more offensive planning, coordination and execution at scale with endless patience, while hiding inside the networks used to detect them.
His answer for defenders is what he calls synthetic cyber teams: a human working with a specialist large language model, or a network of them, trained on cyber operations data. The human explains the goal through an interactive interface, and the AI returns assessments, attack plans, remediation steps and threat intelligence. Humans stay in the loop while the AI does the tedious heavy lifting.
Fully autonomous cognitive cyber AI is still a way off, he said, because AI systems tend to "run off the rails" when left to pursue their own objectives.
What did Starzl demonstrate?
Starzl said he built a fully local system using LangChain, Hugging Face embeddings and Google's Flan model over about 198,000 CVE vulnerability reports, with no reliance on OpenAI or any paid API. He said he asked it for Windows Server 2022 vulnerabilities and it surfaced a BitLocker security feature bypass with the highest CVSS score of 6.6 at the time, which he had not known about.
In a second example, he said the same setup helped him discover that an industrial infrared IoT camera shipped with a blank default Telnet root password. He called it an oversight, not an exploit. His point was that a conversational interface over public data makes this kind of discovery fast, and that the same capability is "dangerous" in the wrong hands.
He also flagged automated scam calls and long-running, highly sophisticated phishing campaigns as problems defenders will have to prepare for.
How should security teams get started?
Starzl advised building a team of experts who understand what large language models are good and bad at. Teams should identify specific use cases instead of building in general, set a strategy for each one, implement it and evaluate the results. An open-ended "fishing expedition," he warned, will get a team lost.
His conclusion was that the technology cannot be stopped. Filtering by companies such as OpenAI will not do it, he said, because the architectures are known and actions by Western actors will not affect bad actors elsewhere.
Where to follow AI and security discussions
AI Connect hosts continuing conversations on AI risk and governance. The 2027 schedule includes 24 free live webinars in four series: the AI × Enterprise Panel Series, AI Demo Days, Agentic AI Showcases and the Plug & Play Series. Details are at /webinars/, and registration is coming soon.
Frequently asked questions
How will generative AI change cybersecurity?
Starzl said it will make attackers more capable at scale and cloak them inside monitored networks, while giving defenders new ways to analyze threat intelligence.
What is a synthetic cyber team?
A human operator working with a specialist language model trained on cyber data, with the AI returning assessments, plans and remediation steps.
Why not rely on AI filtering to stop misuse?
Starzl argued that model architectures are known, so filtering by one company cannot stop bad actors elsewhere.
How should security teams adopt generative AI?
Build a team of experts, pick specific use cases, define a strategy, implement and evaluate.
This article is based on remarks by Ravi Starzl, Professor, Language Technologies Institute, Carnegie Mellon University at Data Science Innovators Conference 2023 on April 12, 2023. More coverage: More AI Connect Events.
Filed under: Security & Fraud, AI Use Cases & Applications
