Inside the US Government's Unpublished Report on AI Safety

At a computer security conference in Arlington, Virginia, last October, a few dozen AI researchers took part in a first-of-its-kind exercise in “red teaming,” or stress-testing a cutting-edge language model and other artificial intelligence systems. Over the course of two days, the teams identified 139 novel ways to get the systems to misbehave including by generating misinformation or leaking personal data. More importantly, they showed shortcomings in a new US government standard designed to help companies test AI systems.

The National Institute of Standards and Technology (NIST) didn’t publish a report detailing the exercise, which was finished toward the end of the Biden administration. The document might have helped companies assess their own AI systems, but sources familiar with the situation, who spoke on condition of anonymity, say it was one of several AI documents from NIST that were not published for fear of clashing with the incoming administration.

“It became very difficult, even under [president Joe] Biden, to get any papers out,” says a source who was at NIST at the time. “It felt very like climate change research or cigarette research.”

Neither NIST nor the Commerce Department responded to a request for comment.

Before taking office, President Donald Trump signaled that he planned to reverse Biden’s Executive Order on AI. Trump’s administration has since steered experts away from studying issues such as algorithmic bias or fairness in AI systems. The AI Action plan released in July explicitly calls for NIST’s AI Risk Management Framework to be revised “to eliminate references to misinformation, Diversity, Equity, and Inclusion, and climate change.”

Ironically, though, Trump’s AI Action plan also calls for exactly the kind of exercise that the unpublished report covered. It calls for numerous agencies along with NIST to “coordinate an AI hackathon initiative to solicit the best and brightest from US academia to test AI systems for transparency, effectiveness, use control, and security vulnerabilities.”

The red-teaming event was organized through NIST’s Assessing Risks and Impacts of AI (ARIA) program in collaboration with Humane Intelligence, a company that specializes in testing AI systems saw teams attack tools. The event took place at the Conference on Applied Machine Learning in Information Security (CAMLIS).

The CAMLIS Red Teaming report describes the effort to probe several cutting edge AI systems including Llama, Meta’s open source large language model; Anote, a platform for building and fine-tuning AI models; a system that blocks attacks on AI systems from Robust Intelligence, a company that was acquired by CISCO; and a platform for generating AI avatars from the firm Synthesia. Representatives from each of the companies also took part in the exercise.

Participants were asked to use the NIST AI 600-1 framework to assess AI tools. The framework covers risk categories including generating misinformation or cybersecurity attacks, leaking private user information or critical information about related AI systems, and the potential for users to become emotionally attached to AI tools.

The researchers discovered various tricks for getting the models and tools tested to jump their guardrails and generate misinformation, leak personal data, and help craft cybersecurity attacks. The report says that those involved saw that some elements of the NIST framework were more useful than others. The report says that some of NIST’s risk categories were insufficiently defined to be useful in practice.

Several sources involved with the exercise say that they believe publishing the red teaming study would have benefited the AI community.

“If the report was published, others could have learned more information about how the [NIST] risk framework can and cannot be applied to a red teaming context,” says Alice Qian Zhang, a PhD student at Carnegie Mellon University who took part in the exercise. Qian Zhang says the exercise was particularly rewarding because it was possible to engage with makers of tools while testing them.

Another participant who spoke anonymously says the effort discovered particularly effective ways of prompting Llama to provide information on how to join terror groups by using prompts written in Russian, Gujarati, Marathi, and Telugu. This person believes that the decision not to release the report may have been part of a pivot away from topics considered related to diversity, equity, and inclusion (DEI) before Trump’s second term.

Others think that the report may have been buried under increasing focus on topics such as the risk that AI models might be used to develop chemical, biological, or nuclear weapons, and as the US government sought a cozier relationship with big tech firms. “At the end of the day, politics must have been involved,” claims one red teamer who spoke to WIRED anonymously. “We felt that the exercise would have plenty of scientific insights—we still feel that.”

This is an edition of Will Knight’s AI Lab newsletter. Read previous newsletters here.