May 4

May 4 Can too much knowledge be a bad thing?

What are information hazards and how does iGEM deal with them?

Earlier this year, we delivered evidence on information hazards and how we deal with them to the Bipartisan Commission on Biodefense. This blog post is adapted from our testimony.

Early in 2022, iGEM’s Responsibility Program was contacted by the Bipartisan Commission on Biodefense to contribute to their efforts to review and revise the US national blueprint to prevent biotechnology being used as a weapon. We were invited to contribute to their first in-person expert consultation since the beginning of the pandemic. The topic of the meeting was on exploring current and future challenges impacting the threat landscape. We were invited to provide (and subsequently delivered) evidence on information hazards and how we deal with them. This blog post is adapted from our testimony.

What is an information hazard?

There is nothing new about information (info) hazards. We are all familiar with the national security implications of information. Widespread knowledge of certain information can change our risk environment. For example, we keep the precise schedule of our leaders out of the public domain to make it harder for anyone planning them harm to be able to act on it. It is the knowledge of where they will be, when, and what steps we are talking to protect them that poses an information hazard.

The concept of info hazards has rarely been applied to biosecurity or biodefence. Most of our efforts to manage biorisk focuses on things - tangible assets. We have rules to control who can access things like pathogens, and where they can be used. Other rules control what you can do with some of these things. There are other rules, like export controls, that control where things can be sent abroad, and to whom, and for what purposes.

There have been efforts to expand some of the rules to also control information. For example, the Netherlands decided to apply its export controls to information when it used them to control pre-publication access to gain-of-function research with H5N1 in 2011.

Examples of info hazards

In the Netherlands, they were concerned about making public the identity of the small number of mutations needed to make a highly pathogenic virus spread more easily. They were not trying to control access to a thing, or in this case even a methodology; it was the information - the precise mutations - that was of concern. As a result, the early drafts of the research paper identifying the mutations was subject to export controls.

A second example of a biological info hazard happened in 2005 when an article was published modeling how botulinum toxin might be introduced into the US milk supply and what the implications of such an attack might be. Immediately, there was considerable concern that such an article had been published. Commentators argued it was a blueprint for those wishing to cause harm. There were worries that it unnecessarily exposed a vulnerability, increasing the chances of a malign actor exploiting it to cause harm. Again, it was information that was changing the risk environment - not access to a thing.

One example of a biological information hazard is the publication of a 2005 article which modeled how botulinum toxin might be introduced into the US milk supply. Commentators argued that the publication was a blueprint for those wishing to cause harm. Photo by @dslr_newb on Unsplash.

A third example comes from the chemical security perspective. A US-based company specializing in the use of machine learning to develop biologically active molecules for the treatment of neurological diseases, along with experts from a national defense laboratory, published a paper describing how to misuse machine learning tools to create new chemical warfare agents. These included agents predicted to be more deadly than nerve agents like VX and in some cases agents not found on any international control lists. They took steps to remove key information from the methodology they used but, nevertheless, have provided a very real proof of principle that this approach is feasible and yields results of considerable concern. By openly advertising a new approach to chemical warfare agent design, they risk encouraging and enabling malign actors to pursue such capabilities.

Options for managing info hazards

From these three examples, we see that information can change the biosecurity risk environment by:

Highlighting approaches that a malign actor might otherwise have missed
Unlocking capabilities that would otherwise be inaccessible, and
Spotlighting vulnerabilities that might otherwise not have been apparent.

So, what can we do about info hazards? At iGEM we take identifying and managing risks from teams’ projects very seriously. We build on national requirements to manage risks to the teams, their colleagues, societies, and the environment. This includes a focus on biosecurity, dual use, and info hazards.

Based on our experiences of working with teams on these issues, we told the meeting that info hazards in the life sciences are real. We deal with them every year. It seems that doing science and engineering on such a scale inevitably means that some teams generate information that could pose a hazard. We shared our practical experiences in adapting what, how, when, and to whom we communicate to manage this risk:

When the risk is well known, or one for which we have ready countermeasures, we encourage our teams to talk about it freely. It is important that teams not only think about dual use, but are seen to be thinking about it too. This helps normalize core approaches to biosecurity.
In some cases, the information a team has generated may be useful to a specific community as part of our efforts to build better defenses. It may be harmful if discussed more widely. We work with the team on their messaging to balance the competing interests of informed debate among the biosecurity community and in avoiding highlighting risks we cannot manage.
Occasionally we have a team that identifies something genuinely new and innovative in the way their work might be misused, or new exploits or vulnerabilities that are not in the public literature, or a vulnerability that cannot readily be patched. In this case, we discourage the team from discussing this information. Of course, we privately communicate these findings to those that need to know so that they can be fed into appropriate national threat assessments.

This is the start (and not the end) of our journey. We are proud of how much we have accomplished but recognize that our efforts to manage info hazards are not yet perfect. We also know that many other programs have yet to really tackle how they think about and address info hazards. We are hoping that by sharing our experiences we can further the discussion around this important issue. We hope that we can support others in strengthening their arrangements by learning from the lessons we have learned. We also know there is much more we can do to work with our teams on this and other connected biosecurity issues. We are looking forward to running both Values and Risks Workshops and Biosecurity Workshops during the 2022 competition. We are looking forward to exploring these issues with you.

If you or another member of your team is interested in discussing information hazards or biosecurity in more detail, please get in touch with safety [AT] igem [DOT] org.

Cover photo by Siora Photography on Unsplash