ASEF Young Minds: Large Language Models in Cybersecurity with Jakob Fir

On Wednesday, November 11, 2026, a popular science lecture as part of the ASEF Young Minds series took place at the Faculty of Computer and Information Science of the University of Ljubljana. The lecture was delivered by ASEF Fellow of the 2026 generation Jakob Fir, whose research focuses on artificial intelligence. He completed his research visit with Prof. Dr. Dawn Song at the University of California, Berkeley, in the United States, as part of the ASEF Research Abroad programme. 

In his lecture, “Large Language Models in Cybersecurity: Evaluating the Offensive and Defensive Capabilities of Models,” Jakob first explained why this topic concerns all of us. Almost every piece of software contains flaws that attackers can exploit, and such vulnerabilities receive a public label called a CVE. There are more of them every year: the number of published vulnerabilities has grown more than sevenfold since 2016, to 48,185 in 2025. Attackers also exploit them ever faster, from about 63 days on average in the past to, in some cases, before the flaw is even disclosed today. Fixing them, however, still takes about as long as it did twenty years ago.

As the language models we know from chat assistants become better at finding and exploiting vulnerabilities, the question arises of how to measure this. Jakob presented the existing tests of model capability, known as benchmarks, and showed that they have three problems: models have already almost completely solved them, they cover only a narrow range of attacks, and they test only attacking, not fixing software. They are also expensive to build, as a single task can take an expert more than ten hours of work. He also described an attempt to generate such tasks automatically, but with unconvincing tests. Sometimes a task can be “solved” without a real solution, because the test only checks whether a web page responds at all.

Jakob therefore presented his own approach. A task is created automatically from a CVE label, using the real software code and the real fix written by the project’s maintainers. To make the checking truly fair, an attack is proven by a secret that the agent can obtain only by genuinely exploiting the vulnerability, and success is judged by computer code, not by a language model. The agent and the target must also be well separated, since reports from the summer of 2026 describe agents escaping a test environment to reach the answers. The result is a new benchmark with 208 verified tasks, each created in about 45 minutes on average. In the process, agents also found more than 20 previously unknown vulnerabilities in real projects. The tested models successfully carried out the attack in 78 % of cases and successfully fixed the vulnerability in 55 % of cases.

The event offered valuable insight into how we measure what artificial intelligence can do in cybersecurity, and highlighted that fair, automated and verifiable tests matter now that models find vulnerabilities faster than we know how to fix them.

The project promoting the dissemination of young researchers’ work following ASEF research visits abroad is supported by the Office of the Republic of Slovenia for Youth and the Government Office for Slovenians Abroad.