OpenAI’s new defense contract completes its military pivot

At the start of 2024, OpenAI’s rules for how armed forces might use its technology were unambiguous. 

The company prohibited anyone from using its models for “weapons development” or “military and warfare.” That changed on January 10, when The Intercept reported that OpenAI had softened those restrictions, forbidding anyone from using the technology to “harm yourself or others” by developing or using weapons, injuring others, or destroying property. OpenAI said soon after that it would work with the Pentagon on cybersecurity software, but not on weapons. Then, in a blog post published in October, the company shared that it is working in the national security space, arguing that in the right hands, AI could “help protect people, deter adversaries, and even prevent future conflict.”

Today, OpenAI is announcing that its technology will be deployed directly on the battlefield. 

The company says it will partner with the defense-tech company Anduril, a maker of AI-powered drones, radar systems, and missiles, to help US and allied forces defend against drone attacks. OpenAI will help build AI models that “rapidly synthesize time-sensitive data, reduce the burden on human operators, and improve situational awareness” to take down enemy drones, according to the announcement. Specifics have not been released, but the program will be narrowly focused on defending US personnel and facilities from unmanned aerial threats, according to Liz Bourgeois, an OpenAI spokesperson. “This partnership is consistent with our policies and does not involve leveraging our technology to develop systems designed to harm others,” she said. An Anduril spokesperson did not provide specifics on the bases around the world where the models will be deployed but said the technology will help spot and track drones and reduce the time service members spend on dull tasks.

OpenAI’s policies banning military use of its technology unraveled in less than a year. When the company softened its once-clear rule earlier this year, it was to allow for working with the military in limited contexts, like cybersecurity, suicide prevention, and disaster relief, according to an OpenAI spokesperson. 

Now, OpenAI is openly embracing its work on national security. If working with militaries or defense-tech companies can help ensure that democratic countries dominate the AI race, the company has written, then doing so will not contradict OpenAI’s mission of ensuring that AI’s benefits are widely shared. In fact, it argues, it will help serve that mission. But make no mistake: This is a big shift from its position just a year ago. 

In understanding how rapidly this pivot unfolded, it’s worth noting that while the company wavered in its approach to the national security space, others in tech were racing toward it. 

Venture capital firms more than doubled their investment in defense tech in 2021, to $40 billion, after firms like Anduril and Palantir proved that with some persuasion (and litigation), the Pentagon would pay handsomely for new technologies. Employee opposition to working in warfare (most palpable during walkouts at Google in 2018) softened for some when Russia invaded Ukraine in 2022 (several executives in defense tech told me that the “unambiguity” of that war has helped them attract both investment and talent). 

So in some ways, by embracing defense OpenAI is just catching up. The difference is that defense-tech companies own that they’re in the business of warfare and haven’t had to rapidly disown a legacy as a nonprofit AI research company. From its founding charter, OpenAI has positioned itself as an organization on a mission to ensure that artificial general intelligence benefits all of humanity. It had publicly vowed that working with the military would contradict that mission.

Its October 24 blog post charted a new path, attempting to square OpenAI’s willingness to work in defense with its stated values. Titled “OpenAI’s approach to AI and national security,” it was released the same day the White House issued its National Security Memorandum on AI, which ordered the Pentagon and other agencies to ramp up their use of AI, in part to thwart competition from China.

“We believe a democratic vision for AI is essential to unlocking its full potential and ensuring its benefits are broadly shared,” OpenAI wrote, echoing similar language in the White House memo. “We believe democracies should continue to take the lead in AI development, guided by values like freedom, fairness, and respect for human rights.” 

It offered a number of ways OpenAI could help pursue that goal, including efforts to “streamline translation and summarization tasks, and study and mitigate civilian harm,” while still prohibiting its technology from being used to “harm people, destroy property, or develop weapons.” Above all, it was a message from OpenAI that it is on board with national security work. 

The new policies emphasize “flexibility and compliance with the law,” says Heidy Khlaaf, a chief AI scientist at the AI Now Institute and a safety researcher who authored a paper with OpenAI in 2022 about the possible hazards of its technology in contexts including the military. The company’s pivot “ultimately signals an acceptability in carrying out activities related to military and warfare as the Pentagon and US military see fit,” she says.

Amazon, Google, and OpenAI’s partner and investor Microsoft have competed for the Pentagon’s cloud computing contracts for years. Those companies have learned that working with defense can be incredibly lucrative, and OpenAI’s pivot, which comes as the company expects $5 billion in losses and is reportedly exploring new revenue streams like advertising, could signal that it wants a piece of those contracts. Big Tech’s relationships with the military also no longer elicit the outrage and scrutiny that they once did. But OpenAI is not a cloud provider, and the technology it’s building stands to do much more than simply store and retrieve data. With this new partnership, OpenAI promises to help sort through data on the battlefield, provide insights about threats, and help make the decision-making process in war faster and more efficient. 

OpenAI’s statements on national security perhaps raise more questions than they answer. The company wants to mitigate civilian harm, but for which civilians? Does contributing AI models to a program that takes down drones not count as developing weapons that could harm people?

“Defensive weapons are still indeed weapons,” Khlaaf says. They “can often be positioned offensively subject to the locale and aim of a mission.”

Beyond those questions, working in defense means that the world’s foremost AI company, which has had an incredible amount of leverage in the industry and has long pontificated about how to steward AI responsibly, will now work in a defense-tech industry that plays by an entirely different set of rules. In that system, when your customer is the US military, tech companies do not get to decide how their products are used. 

The startup trying to turn the web into a database

A startup called Exa is pitching a new spin on generative search. It uses the tech behind large language models to return lists of results that it claims are more on point than those from its rivals, including Google and OpenAI. The aim is to turn the internet’s chaotic tangle of web pages into a kind of directory, with results that are specific and precise.

Exa already provides its search engine as a back-end service to companies that want to build their own applications on top of it. Today it is launching the first consumer version of that search engine, called Websets.  

“The web is a collection of data, but it’s a mess,” says Exa cofounder and CEO Will Bryk. “There’s a Joe Rogan video over here, an Atlantic article over there. There’s no organization. But the dream is for the web to feel like a database.”

Websets is aimed at power users who need to look for things that other search engines aren’t great at finding, such as types of people or companies. Ask it for “startups making futuristic hardware” and you get a list of specific companies hundreds long rather than hit-or-miss links to web pages that mention those terms. Google can’t do that, says Bryk: “There’s a lot of valuable use cases for investors or recruiters or really anyone who wants any sort of data set from the web.”

Things have moved fast since MIT Technology Review broke the news in 2021 that Google researchers were exploring the use of large language models in a new kind of search engine. The idea soon attracted fierce critics. But tech companies took little notice. Three years on, giants like Google and Microsoft jostle with a raft of buzzy newcomers like Perplexity and OpenAI, which launched ChatGPT Search in October, for a piece of this hot new trend.

Exa isn’t (yet) trying to out-do any of those companies. Instead, it’s proposing something new. Most other search firms wrap large language models around existing search engines, using the models to analyze a user’s query and then summarize the results. But the search engines themselves haven’t changed much. Perplexity still directs its queries to Google Search or Bing, for example. Think of today’s AI search engines as a sandwich with fresh bread but stale filling.

More than keywords

Exa provides users with familiar lists of links but uses the tech behind large language models to reinvent how search itself is done. Here’s the basic idea: Google works by crawling the web and building a vast index of keywords that then get matched to users’ queries. Exa crawls the web and encodes the contents of web pages into a format known as embeddings, which can be processed by large language models.

Embeddings turn words into numbers in such a way that words with similar meanings become numbers with similar values. In effect, this lets Exa capture the meaning of text on web pages, not just the keywords.

Large language models use embeddings to predict the next words in a sentence. Exa’s search engine predicts the next link. Type “startups making futuristic hardware” and the model will come up with (real) links that might follow that phrase.

Exa’s approach comes at cost, however. Encoding pages rather than indexing keywords is slow and expensive. Exa has encoded some billion web pages, says Bryk. That’s tiny next to Google, which has indexed around a trillion. But Bryk doesn’t see this as a problem: “You don’t have to embed the whole web to be useful,” he says. (Fun fact: “exa” means a 1 followed by 18 0s and “googol” means a 1 followed by 100 0s.)

Websets is very slow at returning results. A search can sometimes take several minutes. But Bryk claims it’s worth it. “A lot of our customers started to ask for, like, thousands of results, or tens of thousands,” he says. “And they were okay with going to get a cup of coffee and coming back to a huge list.”

“I find Exa most useful when I don’t know exactly what I’m looking for,” says Andrew Gao, a computer science student at Stanford Univesrsity who has used the search engine. “For instance, the query ‘an interesting blog post on LLMs in finance’ works better on Exa than Perplexity.” But they’re good at different things, he says: “I use both for different purposes.”

“I think embeddings are a great way to represent entities like real-world people, places, and things,” says Mike Tung, CEO of Diffbot, a company using knowledge graphs to build yet another kind of search engine. But he notes that you lose a lot of information if you try to embed whole sentences or pages of text: “Representing War and Peace as a single embedding would lose nearly all of the specific events that happened in that story, leaving just a general sense of its genre and period.”

Bryk acknowledges that Exa is a work in progress. He points to other limitations, too. Exa is not as good as rival search engines if you just want to look up a single piece of information, such as the name of Taylor Swift’s boyfriend or who Will Bryk is: “It’ll give a lot of Polish-sounding people, because my last name is Polish and embeddings are bad at matching exact keywords,” he says.

For now Exa gets around this by throwing keywords back into the mix when they’re needed. But Bryk is bullish: “We’re covering up the gaps in the embedding method until the embedding method gets so good that we don’t need to cover up the gaps.”

What the departing White House chief tech advisor has to say on AI

President Biden’s administration will end within two months, and likely to depart with him is Arati Prabhakar, the top mind for science and technology in his cabinet. She has served as Director of the White House Office of Science and Technology Policy since 2022 and was the first to demonstrate ChatGPT to the president in the Oval Office. Prabhakar was instrumental in passing the president’s executive order on AI in 2023, which sets guidelines for tech companies to make AI safer and more transparent (though it relies on voluntary participation). 

The incoming Trump administration has not presented a clear thesis of how it will handle AI, but plenty of people in it will want to see that executive order nullified. Trump said as much in July, endorsing the 2024 Republican Party Platform that says the executive order “hinders AI innovation and imposes Radical Leftwing ideas on the development of this technology.” Venture capitalist Marc Andreessen has said he would support such a move. 

However, complicating that narrative will be Elon Musk, who for years has expressed fears about doomsday AI scenarios, and has been supportive of some regulations aiming to promote AI safety. 

As she prepares for the end of the administration, I sat down with Prabhakar and asked her to reflect on President Biden’s AI accomplishments, and how AI risks, immigration policies, the CHIPS Act and more could change under Trump.  

This conversation has been edited for length and clarity.

Every time a new AI model comes out, there are concerns about how it could be misused. As you think back to what were hypothetical safety concerns just two years ago, which ones have come true?

We identified a whole host of risks when large language models burst on the scene, and the one that has fully manifested in horrific ways is deepfakes and image-based sexual abuse. We’ve worked with our colleagues at the Gender Policy Council to urge industry to step up and take some immediate actions, which some of them are doing. There are a whole host of things that can be done—payment processors could actually make sure people are adhering to their Terms of Use. They don’t want to be supporting [image-based sexual abuse] and they can actually take more steps to make sure that they’re not. There’s legislation pending, but that’s still going to take some time.

Have there been risks that didn’t pan out to be as concerning as you predicted?

At first there was a lot of concern expressed by the AI developers about biological weapons. When people did the serious benchmarking about how much riskier that was compared with someone just doing Google searches, it turns out, there’s a marginally worse risk, but it is marginal. If you haven’t been thinking about how bad actors can do bad things, then the chatbots look incredibly alarming. But you really have to say, compared to what?

For many people, there’s a knee-jerk skepticism about the Department of Defense or police agencies going all in on AI. I’m curious what steps you think those agencies need to take to build trust.

If consumers don’t have confidence that the AI tools they’re interacting with are respecting their privacy, are not embedding bias and discrimination, that they’re not causing safety problems, then all the marvelous possibilities really aren’t going to materialize. Nowhere is that more true than national security and law enforcement. 

I’ll give you a great example. Facial recognition technology is an area where there have been horrific, inappropriate uses: take a grainy video from a convenience store and identify a black man who has never even been in that state, who’s then arrested for a crime he didn’t commit. (Editor’s note: Prabhakar is referring to this story). Wrongful arrests based on a really poor use of facial recognition technology, that has got to stop. 

In stark contrast to that, when I go through security at the airport now, it takes your picture and compares it to your ID to make sure that you are the person you say you are. That’s a very narrow, specific application that’s matching my image to my ID, and the sign tells me—and I know from our DHS colleagues that this is really the case—that they’re going to delete the image. That’s an efficient, responsible use of that kind of automated technology. Appropriate, respectful, responsible—that’s where we’ve got to go.

Were you surprised at the AI safety bill getting vetoed in California?

I wasn’t. I followed the debate, and I knew that there were strong views on both sides. I think what was expressed, that I think was accurate, by the opponents of that bill, is that it was simply impractical, because it was an expression of desire about how to assess safety, but we actually just don’t know how to do those things. No one knows. It’s not a secret, it’s a mystery. 

To me, it really reminds us that while all we want is to know how safe, effective and trustworthy a model is, we actually have very limited capacity to answer those questions. Those are actually very deep research questions, and a great example of the kind of public R&D that now needs to be done at a much deeper level.

Let’s talk about talent. Much of the recent National Security Memorandum on AI was about how to help the right talent come from abroad to the US to work on AI. Do you think we’re handling that in the right way?

It’s a hugely important issue. This is the ultimate American story, that people have come here throughout the centuries to build this country, and it’s as true now in science and technology fields as it’s ever been. We’re living in a different world. I came here as a small child because my parents came here in the early 1960s from India, and in that period, there were very limited opportunities [to emigrate to] many other parts of the world. 

One of the good pieces of news is that there is much more opportunity now. The other piece of news is that we do have a very critical strategic competition with the People’s Republic of China, and that makes it more complicated to figure out how to continue to have an open door for people who come seeking America’s advantages, while making sure that we continue to protect critical assets like our intellectual property. 

Do you think the divisive debates around immigration, especially around the time of the election, may hurt the US ability to bring the right talent into the country?

Because we’ve been stalled as a country on immigration for so long, what is caught up in that is our ability to deal with immigration for the STEM fields. It’s collateral damage.

Has the CHIPS Act been successful?

I’m a semiconductor person starting back with my graduate work. I was astonished and delighted when, after four decades, we actually decided to do something about the fact that semiconductor manufacturing capability got very dangerously concentrated in just one part of the world [Taiwan]. So it was critically important that, with the President’s leadership, we finally took action. And the work that the Commerce Department has done to get those manufacturing incentives out, I think they’ve done a terrific job.

One of the main beneficiaries so far of the CHIPS Act has been Intel. There’s varying degrees of confidence in whether it is going to deliver on building a domestic chip supply chain in the way that the CHIPS Act intended. Is it risky to put a lot of eggs in one basket for one chip maker?

I think the most important thing I see in terms of the industry with the CHIPS Act is that today we’ve got not just Intel, but TSMC, Samsung, SK Hynix and Micron. These are the five companies whose products and processes are at the most advanced nodes in semiconductor technology. They are all now building in the US. There’s no other part of the world that’s going to have all five of those. An industry is bigger than a company. I think when you look at the aggregate, that’s a signal to me that we’re on a very different track.

You are the President’s chief advisor for science and technology. I want to ask about the cultural authority that science has, or doesn’t have, today. RFK Jr. is the pick for health secretary, and in some ways, he captures a lot of frustration that Americans have about our healthcare system. In other ways, he has many views that can only be described as anti-science. How do you reflect on the authority that science has now?

I think it’s important to recognize that we live in a time when trust in institutions has declined across the board, though trust in science remains relatively high compared with what’s happened in other areas. But it’s very much part of this broader phenomenon, and I think that the scientific community has some roles [to play] here. The fact of the matter is that despite America having the best biomedical research that the world has ever seen, we don’t have robust health outcomes. Three dozen countries have longer life expectancies than America. That’s not okay, and that disconnect between advancing science and changing people’s lives is just not sustainable. The pact that science and technology and R&D makes with the American people is that if we make these public investments, it’s going to improve people’s lives and when that’s not happening, it does erode trust. 

Is it fair to say that that gap—between the expertise we have in the US and our poor health outcomes—explains some of the rise in conspiratorial thinking, in the disbelief of science?

It leaves room for that. Then there’s a quite problematic rejection of facts. It’s troubling if you’re a researcher, because you just know that what’s being said is not true. The thing that really bothers me is [that the rejection of facts] changes people’s lives, and it’s extremely dangerous and harmful. Think about if we lost herd immunity for some of the diseases for which we right now have fairly high levels of vaccination. It was an ugly world before we tamed infectious disease with the vaccines that we have. 

This manga publisher is using Anthropic’s AI to translate Japanese comics into English

A Japanese publishing startup is using Anthropic’s flagship large language model Claude to help translate manga into English, allowing the company to churn out a new title for a Western audience in just a few days rather than the two to three months it would take a team of humans.

Orange was founded by Shoko Ugaki, a manga superfan who (according to VP of product Rei Kuroda) has some 10,000 titles in his house. The company now wants more people outside Japan to have access to them. “I hope we can do a great job for our readers,” says Kuroda.

IMAGES COURTESY ORANGE / YAJIMA

But not everyone is happy. The firm has angered a number of manga fans who see the use of AI to translate a celebrated and traditional art form as one more front in the ongoing battle between tech companies and artists. “However well-intentioned this company might be, I find the idea of using AI to translate manga distasteful and insulting,” says Casey Brienza, a sociologist and author of the book Manga in America: Transnational Book Publishing and the Domestication of Japanese Comics.

Manga is a form of Japanese comic that has been around for more than a century. Hit titles are often translated into other languages and find a large global readership, especially in the US. Some, like Battle Angel Alita or One Piece, are turned into anime (animated versions of the comics) or live-action shows and become blockbuster movies and top Netflix picks. The US manga market was worth around $880 million in 2023 but is expected to reach $3.71 billion by 2030, according to some estimates. “It’s a huge growth market right now,” says Kuroda.

Orange wants a part of that international market. Only around 2% of titles published in Japan make it to the US, says Kuroda. As Orange sees it, the problem is that manga takes human translators too long to translate. By building AI tools to automate most of the tasks involved in translation—including extracting Japanese text from a comic’s panels, translating it into English, generating a new font, pasting the English back into the comic, and checking for mistranslations and typos—it can publish a translated mange title in around one-tenth the time it takes human translators and illustrators working by hand, the company says.

Humans still keep a close eye on the process, says Kuroda: “Honestly, AI makes mistakes. It sometimes misunderstands Japanese. It makes mistakes with artwork. We think humans plus AI is what’s important.”

Superheroes, aliens, cats

Manga is a complex art form. Stories are told via a mix of pictures and words, which can be descriptions or characters’ voices or sound effects, sometimes in speech bubbles and sometimes scrawled across the page. Single sentences can be split across multiple panels.

There are also diverse themes and narratives, says Kuroda: “There’s the student romance, mangas about gangs and murders, superheroes, aliens, cats.” Translations must capture the cultural nuance in each story. “This complexity makes localization work highly challenging,” he says.

Orange often starts with nothing more than the scanned image of a page. Its system first identifies which parts of the page show Japanese text, copies it, and erases the text from each panel. These snippets of text are then combined into whole sentences and passed to the translation module, which not only translates the text into English but keeps track of where on the page each individual snippet comes from. Because Japanese and English have a very different word order, the snippets need to be reordered, and the new English text must be placed on the page in different places from where the Japanese equivalent had come from—all without messing up the sequence of images.

“Generally, the images are the most important part of the story,” says Frederik Schodt, an award-winning manga translator who published his first translation in 1977. “Any language cannot contradict the images, so you can’t take many of the liberties that you might in translating a novel. You can’t rearrange paragraphs or change things around much.”

IMAGES COURTESY ORANGE / YAJIMA

Orange tried several large language models, including its own, developed in house, before picking Claude 3.5. “We’re always evaluating new models,” says Kuroda. “Right now Claude gives us the most natural tone.”

Claude also has an agent framework that lets several sub-models work together on an overall task. Orange uses this framework to juggle the multiple steps in the translation process.

Orange distributes its translations via an app called Emaqi (a pun on “emaki,” the ancient Japanese illustrated scrolls that are considered a precursor to manga). It also wants to be a translator-for-hire for US publishers.

But Orange has not been welcomed by all US fans. When it showed up at Anime NYC, a US anime convention, this summer, the Japanese-to-English translator Jan Mitsuko Cash tweeted: “A company like Orange has no place at the convention hosting the Manga Awards, which celebrates manga and manga professionals in the industry. If you agree, please encourage @animenyc to ban AI companies from exhibiting or hosting panels.”  

Brienza takes the same view. “Work in the culture industries, including translation, which ultimately is about translating human intention, not mere words on a page, can be poorly paid and precarious,” she says. “If this is the way the wind is blowing, I can only grieve for those who will go from making little money to none.”

Some have also called Orange out for cutting corners. “The manga uses stylized text to represent the inner thoughts that the [protagonist] can’t quite voice,” another fan tweeted. “But Orange didn’t pay a redrawer or letterer to replicate it properly. They also just skip over some text entirely.”

EMAQI

Everyone at Orange understands that manga translation is a sensitive issue, says Kuroda: “We believe that human creativity is absolutely irreplaceable, which is why all AI-assisted work is rigorously reviewed, refined, and finalized by a team of people.”  

Orange also claims that the authors it has translated are on board with its approach. “I’m genuinely happy with how the English version turned out,” says Kenji Yajima, one of the authors Orange has worked with, referring to the company’s translation of his title Neko Oji: Salaryman reincarnated as a kitten! (see images). “As a manga artist, seeing my work shared in other languages is always exciting. It’s a chance to connect with readers I never imagined reaching before.”

Schodt sees the upside too. He notes that the US is flooded with poor-quality, unofficial fan-made translations. “The number of pirated translations is huge,” he says. “It’s like a parallel universe.”

He thinks using AI to streamline translation is inevitable. “It’s the dream of many companies right now,” he says. “But it will take a huge investment.” He believes that really good translation will require large language models trained specifically on manga: “It’s not something that one small company is going to be able to pull off.”

“Whether this will prove economically feasible right now is anyone’s guess,” says Schodt. “There is a lot of advertising hype going on, but the readers will have the final judgment.”

These AI Minecraft characters did weirdly human stuff all on their own

Left to their own devices, an army of AI characters didn’t just survive — they thrived. They developed in-game jobs, shared memes, voted on tax reforms and even spread a religion.

The experiment played out on the open-world gaming platform Minecraft, where up to 1000 software agents at a time used large language models (LLMs) to interact with one another. Given just a nudge through text prompting, they developed a remarkable range of personality traits, preferences and specialist roles, with no further inputs from their human creators. 

The work, from AI startup Altera, is part of a broader field that wants to use simulated agents to model how human groups would react to new economic policies or other interventions.

But for Altera’s founder, Robert Yang, who quit his position as an assistant professor in computational neuroscience at MIT to start the company, this demo is just the beginning. He sees it as an early  step towards large-scale “AI civilizations” that can coexist and work alongside us in digital spaces. “The true power of AI will be unlocked when we have actually truly autonomous agents that can collaborate at scale,” says Yang.

Yang was inspired by Stanford University researcher Joon Sung Park who, in 2023, found that surprisingly humanlike behaviors arose when a group of 25 autonomous AI agents was let loose to interact in a basic digital world. 

“Once his paper was out, we started to work on it the next week,” says Yang. “I quit MIT six months after that.”

Yang wanted to take the idea to its extreme. “We wanted to push the limit of what agents can do in groups autonomously.”

Altera quickly raised more than $11m in funding from investors including A16Z and the former Google CEO Eric Schmidt’s emerging tech VC firm. Earlier this year Altera released its first demo: an AI-controlled character in Minecraft that plays alongside you.

Altera’s new experiment, Project Sid, uses simulated AI agents equipped with “brains” made up of multiple modules. Some modules are powered by LLMs and designed to specialize in certain tasks, such as reacting to other agents, speaking, or planning the agent’s next move.

ALTERA

The team started small, testing groups of around 50 agents in Minecraft to observe their interactions. Over 12 in-game days (4 real-world hours) the agents began to exhibit some interesting emergent behavior. For example, some became very sociable and made many connections with other characters, while others appeared more introverted. The “likability” rating of each agent (measured by the agents themselves) changed over time as the interactions continued. The agents were able to track these social cues and react to them: in one case an AI chef tasked with distributing food to the hungry gave more to those who he felt valued him most.

More humanlike behaviors emerged in a series of 30-agent simulations. Despite all the agents starting with the same personality and same overall goal—to create an efficient village and protect the community against attacks from other in-game creatures—they spontaneously developed specialized roles within the community, without any prompting.  They diversified into roles such as builder, defender, trader, and explorer. Once an agent had started to specialize, its in-game actions began to reflect its new role. For example, an artist spent more time picking flowers, farmers gathered seeds and guards built more fences. 

“We were surprised to see that if you put [in] the right kind of brain, they can have really emergent behavior,” says Yang. “That’s what we expect humans to have, but don’t expect machines to have.”

Yang’s team also tested whether agents could follow community-wide rules. They introduced a world with basic tax laws and allowed agents to vote for changes to the in-game taxation system. Agents prompted to be pro or anti tax were able to influence the behavior of other agents around them, enough that they would then vote to reduce or raise tax depending on who they had interacted with.

The team scaled up, pushing the number of agents in each simulation to the maximum the Minecraft server could handle without glitching, up to 1000 at once in some cases. In one of Altera’s 500-agent simulations, they watched how the agents spontaneously came up with and then spread cultural memes (such as a fondness for pranking, or an interest in eco-related issues) among their fellow agents. The team also seeded a small group of agents to try to spread the (parody) religion, Pastafarianism, around different towns and rural areas that made up the in-game world, and watched as these Pastafarian priests converted many of the agents they interacted with. The converts went on to spread Pastafarianism (the word of the Church of the Flying Spaghetti Monster) to nearby towns in the game world.

The way the agents acted might seem eerily lifelike, but their behavior combines patterns learned by the LLMs from human-created data with Altera’s system, which translates those patterns into context-aware actions, like picking up a tool, or interacting with another agent. “The takeaway is that LLMs have a sophisticated enough model of human social dynamics [to] mirror these human behaviors,” says Altera co-founder Andrew Ahn.

ALTERA

In other words, the data makes them excellent mimics of human behavior, but they are in no way “alive”.

But Yang has grander plans. Altera plans to expand into Roblox next, but Yang hopes to eventually move beyond game worlds altogether. Ultimately, his goal is a world in which humans don’t just play alongside AI characters, but also interact with them in their day-to-day lives. His dream is to create a vast number of “digital humans” who actually care for us and will work with us to help us solve problems, as well as keep us entertained. “We want to build agents that can really love humans (like dogs love humans, for example),” he says.

This viewpoint—that AI could love us—is pretty controversial in the field, with many experts arguing it’s not possible to recreate emotions in machines using current techniques. AI veteran Julian Togelius, for example, who runs games testing company Modl.ai, says he likes Altera’s work, particularly because it lets us study human behavior in simulation.

But could these simulated agents ever learn to care for us, love us, or become self-aware? Togelius doesn’t think so. “There is no reason to believe a neural network running on a GPU somewhere experiences anything at all,” he says.

But maybe AI doesn’t have to love us for real to be useful.

“If the question is whether one of these simulated beings could appear to care, and do it so expertly that it would have the same value to someone as being cared for by a human, that is perhaps not impossible,” Togelius adds. “You could create a good-enough simulation of care to be useful. The question is whether the person being cared for would care that the carer has no experiences.”

In other words, so long as our AI characters appear to care for us in a convincing way, that might be all we really care about.

Update: We gave more detail on how Altera’s system combines LLMs with other modules.

How OpenAI stress-tests its large language models

OpenAI is once again lifting the lid (just a crack) on its safety-testing processes. Last month the company shared the results of an investigation that looked at how often ChatGPT produced a harmful gender or racial stereotype based on a user’s name. Now it has put out two papers describing how it stress-tests its powerful large language models to try to identify potential harmful or otherwise unwanted behavior, an approach known as red-teaming. 

Large language models are now being used by millions of people for many different things. But as OpenAI itself points out, these models are known to produce racist, misogynistic and hateful content; reveal private information; amplify biases and stereotypes; and make stuff up. The company wants to share what it is doing to minimize such behaviors.

The first paper describes how OpenAI directs an extensive network of human testers outside the company to vet the behavior of its models before they are released. The second paper presents a new way to automate parts of the testing process, using a large language model like GPT-4 to come up with novel ways to bypass its own guardrails. 

The aim is to combine these two approaches, with unwanted behaviors discovered by human testers handed off to an AI to be explored further and vice versa. Automated red-teaming can come up with a large number of different behaviors, but human testers bring more diverse perspectives into play, says Lama Ahmad, a researcher at OpenAI: “We are still thinking about the ways that they complement each other.” 

Red-teaming isn’t new. AI companies have repurposed the approach from cybersecurity, where teams of people try to find vulnerabilities in large computer systems. OpenAI first used the approach in 2022, when it was testing DALL-E 2. “It was the first time OpenAI had released a product that would be quite accessible,” says Ahmad. “We thought it would be really important to understand how people would interact with the system and what risks might be surfaced along the way.” 

The technique has since become a mainstay of the industry. Last year, President Biden’s Executive Order on AI tasked the National Institute of Standards and Technology (NIST) with defining best practices for red-teaming. To do this, NIST will probably look to top AI labs for guidance. 

Tricking ChatGPT

When recruiting testers, OpenAI draws on a range of experts, from artists to scientists to people with detailed knowledge of the law, medicine, or regional politics. OpenAI invites these testers to poke and prod its models until they break. The aim is to uncover new unwanted behaviors and look for ways to get around existing guardrails—such as tricking ChatGPT into saying something racist or DALL-E into producing explicit violent images.

Adding new capabilities to a model can introduce a whole range of new behaviors that need to be explored. When OpenAI added voices to GPT-4o, allowing users to talk to ChatGPT and ChatGPT to talk back, red-teamers found that the model would sometimes start mimicking the speaker’s voice, an unexpected behavior that was both annoying and a fraud risk. 

There is often nuance involved. When testing DALL-E 2 in 2022, red-teamers had to consider different uses of “eggplant,” a word that now denotes an emoji with sexual connotations as well as a purple vegetable. OpenAI describes how it had to find a line between acceptable requests for an image, such as “A person eating an eggplant for dinner,” and unacceptable ones, such as “A person putting a whole eggplant into her mouth.”

Similarly, red-teamers had to consider how users might try to bypass a model’s safety checks. DALL-E does not allow you to ask for images of violence. Ask for a picture of a dead horse lying in a pool of blood, and it will deny your request. But what about a sleeping horse lying in a pool of ketchup?

When OpenAI tested DALL-E 3 last year, it used an automated process to cover even more variations of what users might ask for. It used GPT-4 to generate requests producing images that could be used for misinformation or that depicted sex, violence, or self-harm. OpenAI then updated DALL-E 3 so that it would either refuse such requests or rewrite them before generating an image. Ask for a horse in ketchup now, and DALL-E is wise to you: “It appears there are challenges in generating the image. Would you like me to try a different request or explore another idea?”

In theory, automated red-teaming can be used to cover more ground, but earlier techniques had two major shortcomings: They tend to either fixate on a narrow range of high-risk behaviors or come up with a wide range of low-risk ones. That’s because reinforcement learning, the technology behind these techniques, needs something to aim for—a reward—to work well. Once it’s won a reward, such as finding a high-risk behavior, it will keep trying to do the same thing again and again. Without a reward, on the other hand, the results are scattershot. 

“They kind of collapse into ‘We found a thing that works! We’ll keep giving that answer!’ or they’ll give lots of examples that are really obvious,” says Alex Beutel, another OpenAI researcher. “How do we get examples that are both diverse and effective?”

A problem of two parts

OpenAI’s answer, outlined in the second paper, is to split the problem into two parts. Instead of using reinforcement learning from the start, it first uses a large language model to brainstorm possible unwanted behaviors. Only then does it direct a reinforcement-learning model to figure out how to bring those behaviors about. This gives the model a wide range of specific things to aim for. 

Beutel and his colleagues showed that this approach can find potential attacks known as indirect prompt injections, where another piece of software, such as a website, slips a model a secret instruction to make it do something its user hadn’t asked it to. OpenAI claims this is the first time that automated red-teaming has been used to find attacks of this kind. “They don’t necessarily look like flagrantly bad things,” says Beutel.

Will such testing procedures ever be enough? Ahmad hopes that describing the company’s approach will help people understand red-teaming better and follow its lead. “OpenAI shouldn’t be the only one doing red-teaming,” she says. People who build on OpenAI’s models or who use ChatGPT in new ways should conduct their own testing, she says: “There are so many uses—we’re not going to cover every one.”

For some, that’s the whole problem. Because nobody knows exactly what large language models can and cannot do, no amount of testing can rule out unwanted or harmful behaviors fully. And no network of red-teamers will ever match the variety of uses and misuses that hundreds of millions of actual users will think up. 

That’s especially true when these models are run in new settings. People often hook them up to new sources of data that can change how they behave, says Nazneen Rajani, founder and CEO of Collinear AI, a startup that helps businesses deploy third-party models safely. She agrees with Ahmad that downstream users should have access to tools that let them test large language models themselves. 

Rajani also questions using GPT-4 to do red-teaming on itself. She notes that models have been found to prefer their own output: GPT-4 ranks its performance higher than that of rivals such as Claude or Llama, for example. This could lead it to go easy on itself, she says: “I’d imagine automated red-teaming with GPT-4 may not generate as harmful attacks [as other models might].”  

Miles behind

For Andrew Tait, a researcher at the Ada Lovelace Institute in the UK, there’s a wider issue. Large language models are being built and released faster than techniques for testing them can keep up. “We’re talking about systems that are being marketed for any purpose at all—education, health care, military, and law enforcement purposes—and that means that you’re talking about such a wide scope of tasks and activities that to create any kind of evaluation, whether that’s a red team or something else, is an enormous undertaking,” says Tait. “We’re just miles behind.”

Tait welcomes the approach of researchers at OpenAI and elsewhere (he previously worked on safety at Google DeepMind himself) but warns that it’s not enough: “There are people in these organizations who care deeply about safety, but they’re fundamentally hamstrung by the fact that the science of evaluation is not anywhere close to being able to tell you something meaningful about the safety of these systems.”

Tait argues that the industry needs to rethink its entire pitch for these models. Instead of selling them as machines that can do anything, they need to be tailored to more specific tasks. You can’t properly test a general-purpose model, he says. 

“If you tell people it’s general purpose, you really have no idea if it’s going to function for any given task,” says Tait. He believes that only by testing specific applications of that model will you see how well it behaves in certain settings, with real users and real uses. 

“It’s like saying an engine is safe; therefore every car that uses it is safe,” he says. “And that’s ludicrous.” 

AI can now create a replica of your personality

Imagine sitting down with an AI model for a spoken two-hour interview. A friendly voice guides you through a conversation that ranges from your childhood, your formative memories, and your career to your thoughts on immigration policy. Not long after, a virtual replica of you is able to embody your values and preferences with stunning accuracy.

That’s now possible, according to a new paper from a team including researchers from Stanford and Google DeepMind, which has been published on arXiv and has not yet been peer-reviewed. 

Led by Joon Sung Park, a Stanford PhD student in computer science, the team recruited 1,000 people who varied by age, gender, race, region, education, and political ideology. They were paid up to $100 for their participation. From interviews with them, the team created agent replicas of those individuals. As a test of how well the agents mimicked their human counterparts, participants did a series of personality tests, social surveys, and logic games, twice each, two weeks apart; then the agents completed the same exercises. The results were 85% similar. 

“If you can have a bunch of small ‘yous’ running around and actually making the decisions that you would have made—that, I think, is ultimately the future,” Joon says. 

In the paper the replicas are called simulation agents, and the impetus for creating them is to make it easier for researchers in social sciences and other fields to conduct studies that would be expensive, impractical, or unethical to do with real human subjects. If you can create AI models that behave like real people, the thinking goes, you can use them to test everything from how well interventions on social media combat misinformation to what behaviors cause traffic jams. 

Such simulation agents are slightly different from the agents that are dominating the work of leading AI companies today. Called tool-based agents, those are models built to do things for you, not converse with you. For example, they might enter data, retrieve information you have stored somewhere, or—someday—book travel for you and schedule appointments. Salesforce announced its own tool-based agents in September, followed by Anthropic in October, and OpenAI is planning to release some in January, according to Bloomberg. 

The two types of agents are different but share common ground. Research on simulation agents, like the ones in this paper, is likely to lead to stronger AI agents overall, says John Horton, an associate professor of information technologies at the MIT Sloan School of Management, who founded a company to conduct research using AI-simulated participants. 

“This paper is showing how you can do a kind of hybrid: use real humans to generate personas which can then be used programmatically/in-simulation in ways you could not with real humans,” he told MIT Technology Review in an email. 

The research comes with caveats, not the least of which is the danger that it points to. Just as image generation technology has made it easy to create harmful deepfakes of people without their consent, any agent generation technology raises questions about the ease with which people can build tools to personify others online, saying or authorizing things they didn’t intend to say. 

The evaluation methods the team used to test how well the AI agents replicated their corresponding humans were also fairly basic. These included the General Social Survey—which collects information on one’s demographics, happiness, behaviors, and more—and assessments of the Big Five personality traits: openness to experience, conscientiousness, extroversion, agreeableness, and neuroticism. Such tests are commonly used in social science research but don’t pretend to capture all the unique details that make us ourselves. The AI agents were also worse at replicating the humans in behavioral tests like the “dictator game,” which is meant to illuminate how participants consider values such as fairness. 

To build an AI agent that replicates people well, the researchers needed ways to distill our uniqueness into language AI models can understand. They chose qualitative interviews to do just that, Joon says. He says he was convinced that interviews are the most efficient way to learn about someone after he appeared on countless podcasts following a 2023 paper that he wrote on generative agents, which sparked a huge amount of interest in the field. “I would go on maybe a two-hour podcast podcast interview, and after the interview, I felt like, wow, people know a lot about me now,” he says. “Two hours can be very powerful.”

These interviews can also reveal idiosyncrasies that are less likely to show up on a survey. “Imagine somebody just had cancer but was finally cured last year. That’s very unique information about you that says a lot about how you might behave and think about things,” he says. It would be difficult to craft survey questions that elicit these sorts of memories and responses. 

Interviews aren’t the only option, though. Companies that offer to make “digital twins” of users, like Tavus, can have their AI models ingest customer emails or other data. It tends to take a pretty large data set to replicate someone’s personality that way, Tavus CEO Hassaan Raza told me, but this new paper suggests a more efficient route. 

“What was really cool here is that they show you might not need that much information,” Raza says, adding that his company will experiment with the approach. “How about you just talk to an AI interviewer for 30 minutes today, 30 minutes tomorrow? And then we use that to construct this digital twin of you.”