🔬 Research Summary by Shrestha Rath, a biosecurity researcher at Effective Ventures Foundation in Oxford.
[Original paper by Jonas B. Sandbrink]
Overview: Should ChatGPT be able to give you step-by-step instructions to create the next pandemic virus? As artificial intelligence tools like ChatGPT become more advanced, they may lower barriers to biological weapons and bioterrorism. This article differentiates between the impacts of language models and AI tools trained on biological data and proposes how these risks may be mitigated.
Introduction
Iraq’s bioweapons program in the 1990s never turned its less potent liquid anthrax weapon into a more dangerous powdered form despite having access to the relevant equipment. It’s speculated that this was due to a lack of appropriate expertise. Similarly, Aum Shinrikyo- a Japanese cult bent on civilizational collapse to establish a new society- failed to differentiate between the highly deadly botulinum toxin and the organism that produces it. Living in a world populated with LLM-powered chatbots, it’s worth wondering, had Aum and Iraqi programs used today’s ChatGPT to overcome bottlenecks, might they have succeeded at bioterrorism?
The author explores how new AI tools might enable biological misuse, such as bioterrorism and biological weapons. The paper differentiates between Large Language Models (LLMs) like ChatGPT and Biological Design Tools (BDTs) like the protein design tool ProteinMPNN.
The former are trained on natural language data, such as internet text, while the latter are trained on biological data, such as genetic sequences. The author suggests that LLMs could allow more malicious actors to misuse biological agents while BDTs could assist in developing atypical agents posing novel threats. The article identifies the risk profiles of LLM and BDTs while exploring several risk mitigation strategies, such as access controls and pre-release external audits. Additionally, Sandbrink highlights the pressing need for mandatory gene synthesis screening and the reinforcement of international norms governing the use of these tools as another effective risk mitigation strategy.
Key Insights
Risks from LLMs
Increased accessibility to relevant knowledge
In March 2023, Google, Anthropic, and OpenAI released their respective LLMs– Bard, Claude, and GPT-4. LLMs have grappled public attention with their broad range of capabilities. The article broadly identifies how LLMs may lower barriers to biological weapons development. Sandbrink argues that historical attempts at bioweapons development were limited by technical know-how, like the Iraq and Aum Shinrikyo examples described earlier. LLMs excel at answering high-level and specific questions, amalgamating and synthesizing sources, and conveying the information to augment the user’s existing knowledge. Thus, Sandbrink asserts that today’s GPT-4 could have helped bypass hurdles around drying spores and concentrating toxins these actors encountered. The paper highlights an MIT exercise that showed LLMs instructing students to create pandemic pathogens and find DNA synthesis companies that don’t screen orders. Indeed, this exercise shows the potential of language models for explaining existing relevant knowledge, including “point[ing] specifically towards information on research that could be misused.”
Step-by-step instructions and troubleshooting experiments
Secondly, “AI lab assistants” could go beyond making public knowledge accessible– they have the potential to make relevant knowledge actionable. Thanks to their comprehensive training on formal publications and informal discussion forums, LLMs can instruct and troubleshoot laboratory protocols for untrained actors. These AI lab assistants excel at customizing instructions to fit the user’s specific needs and constraints. Consequently, “tacit knowledge” – applied knowledge normally absent from scientific documentation, such as learned through experience – may decrease as a prominent hurdle against biological misuse. However, whether AI lab assistants can sufficiently lower this barrier for actors with limited laboratory experience remains uncertain.
Autonomous capabilities and perception
In combination with laboratory robot infrastructure, LLM capabilities essentially eliminate programming needs for automating experiments. Future technological breakthroughs in LLMs and autonomous labs may surmount past hurdles faced by Soviet and Iraqi bioweapon programs by reducing socio-organizational challenges to large, covert scientific operations.
Lastly, LLMs may not only increase the bioterror threat because of their actual abilities to scan scientific literature, planning, execute, and troubleshoot life science experiments. Even the perception that LLMs improve accessibility to biotech knowledge may spur more attempts at misuse, albeit actual technical barriers remain unchanged. In this regard, the author references a relevant historical example of al-Qaeda’s serious attempts to acquire anthrax following public concerns by the US government about bioterrorism.
Risks from BDTs
Increased risks of worst-case scenarios
Sandbrink further elaborates on Biological Design Tools (BDTs) –another category of AI tools exacerbating risks at the “design” phase of life science research. BDTs “are trained on biological data [and] can help design new proteins or other biological agents.” While he primarily concentrates on general-purpose protein design tools, he does indicate that “eventually, relevant [specialized] tools likely will be able to create proteins, enzymes, and potentially eventually whole organisms optimized across different functions.” BDTs, unlike LLMs, primarily increase the ceiling of capabilities for sophisticated actors, such as advanced extremist groups and state actors. Sandbrink emphasizes that an increased ceiling of capabilities facilitates the rise of the “ceiling of harm.”
BDTs may enable the design of pathogens more dangerous than any naturally occurring ones. This could mean humanity may face existential threats from pandemics for the first time. Historically, extremist groups like Aum Shinrikyo have aimed to cause indiscriminate harm, and BDTs could enable such groups to engineer catastrophic pandemics. Though bioterrorism with engineered pathogens remains a low-probability scenario due to the need for significant skills, time, resources, and access to AI tools, the barriers to using BDTs may decrease as large language models and AI lab assistants advance. The U.S. and Iraq have previously discounted biological weapons due to their short shelf life, risk of friendly fire, and doubts about effectiveness. Advanced design capabilities could also result in enhanced biological agents with greater appeal to state actors.
Circumventing sequence-based biosecurity measures
Furthermore, Sandbrink warns that “biological design tools may challenge existing measures to control access to dangerous agents based on their taxonomy and genetic sequence.” Currently, preventing illegal access to toxins and pathogens involves export controls and voluntary sequence screening by gene synthesis providers. However, BDTs may simplify the design of novel agents with unique functions that don’t fall into existing taxonomic categories or contain recognizable hazardous sequences.
Mitigating risks from AI tools
Mitigating risks from LLMs demands urgent attention. Firstly, predicting the capabilities of new LLMs is difficult, and even innocuous open-source models could later be fine-tuned for biological misuse. Secondly, LLMs already lower barriers to dual-use knowledge, information that can be used for both beneficial and harmful applications. To mitigate risks, Sandbrink identifies access controls as a key strategy. Access controls might be particularly feasible for cutting-edge LLMs because these are only developed by a handful of companies. Arguably, public versions of LLMs need not brainstorm ideas for misuse or instruct dual-use experiments. However, where to draw the line on what exactly LLMs should not disclose is unclear. Finally, mandating external pre-release evaluations of models would incentivize developers to consider biosecurity during model training and deployment.
The paper outlines the risks from BDTs to be significant because of their potential role in increasing the harm ceiling. However, these capabilities and risks remain largely ill-defined. Currently, BDTs are also majorly developed in collaboration with academia. Thus, the next steps for mitigating risks include creating discussion forums between policymakers, biosecurity experts, and model developers to establish governance strategies. It’s essential that model developers practice dual-use review throughout the development of BDTs. Since BDTs misused by well-resourced and skilled actors are hard to prevent via access controls, alternate interventions may be more effective. For sophisticated non-state actors, reinforcing intelligence and law enforcement can detect and prevent misuse. For state actors, maintaining the unattractiveness of biological weapons, strengthening norms against such weapons, advancing verification regimes, and developing robust methods for attribution and detection of biological attacks could be effective. Despite the value of open-source publishing of computational biology tools, structured access to BDTs should be explored to mitigate biosecurity risks and provide a baseline for risk monitoring and governance.
Between the lines
The paper characterizes the impact of AI tools on the potential misuse of biotechnology, differentiating risks posed by each tool. The author underscores the urgent need for implementing pre-release evaluations and tailored access methods for AI lab assistants, prioritizing authentication and training for legitimate users.
The article highlights the necessity of mandatory DNA synthesis screening, especially considering the rising popularity of benchtop gene synthesis devices. It further emphasizes that such measures should be future-oriented, including the screening for functional equivalents of harmful agents that may be designed by future BDTs.
Sandbrink highlights that these discussions around AI-enabled risks are “a concrete instantiation of a broader set of artificial intelligence risks that could catalyze general AI governance measures.”, notes Sandbrink. The paper also emphasizes the need to foster research exploring the beneficial use of AI tools in mitigating risks of biological misuse, such as AI-enabled DNA synthesis screening.
Identifying red lines around dangerous AI capabilities, strengthening security intelligence, and international norms against bioweapons are crucial avenues to enable risk mitigation strategies. A combination of regulatory and technical safeguards can set “the groundwork for enabling AI to realize its very positive implications for the life sciences and human health.”, signs off Sandbrink.