OpenAI’s new defense contract completes its military pivot

At the start of 2024, OpenAI’s rules for how armed forces might use its technology were unambiguous. 

The company prohibited anyone from using its models for “weapons development” or “military and warfare.” That changed on January 10, when The Intercept reported that OpenAI had softened those restrictions, forbidding anyone from using the technology to “harm yourself or others” by developing or using weapons, injuring others, or destroying property. OpenAI said soon after that it would work with the Pentagon on cybersecurity software, but not on weapons. Then, in a blog post published in October, the company shared that it is working in the national security space, arguing that in the right hands, AI could “help protect people, deter adversaries, and even prevent future conflict.”

Today, OpenAI is announcing that its technology will be deployed directly on the battlefield. 

The company says it will partner with the defense-tech company Anduril, a maker of AI-powered drones, radar systems, and missiles, to help US and allied forces defend against drone attacks. OpenAI will help build AI models that “rapidly synthesize time-sensitive data, reduce the burden on human operators, and improve situational awareness” to take down enemy drones, according to the announcement. Specifics have not been released, but the program will be narrowly focused on defending US personnel and facilities from unmanned aerial threats, according to Liz Bourgeois, an OpenAI spokesperson. “This partnership is consistent with our policies and does not involve leveraging our technology to develop systems designed to harm others,” she said. An Anduril spokesperson did not provide specifics on the bases around the world where the models will be deployed but said the technology will help spot and track drones and reduce the time service members spend on dull tasks.

OpenAI’s policies banning military use of its technology unraveled in less than a year. When the company softened its once-clear rule earlier this year, it was to allow for working with the military in limited contexts, like cybersecurity, suicide prevention, and disaster relief, according to an OpenAI spokesperson. 

Now, OpenAI is openly embracing its work on national security. If working with militaries or defense-tech companies can help ensure that democratic countries dominate the AI race, the company has written, then doing so will not contradict OpenAI’s mission of ensuring that AI’s benefits are widely shared. In fact, it argues, it will help serve that mission. But make no mistake: This is a big shift from its position just a year ago. 

In understanding how rapidly this pivot unfolded, it’s worth noting that while the company wavered in its approach to the national security space, others in tech were racing toward it. 

Venture capital firms more than doubled their investment in defense tech in 2021, to $40 billion, after firms like Anduril and Palantir proved that with some persuasion (and litigation), the Pentagon would pay handsomely for new technologies. Employee opposition to working in warfare (most palpable during walkouts at Google in 2018) softened for some when Russia invaded Ukraine in 2022 (several executives in defense tech told me that the “unambiguity” of that war has helped them attract both investment and talent). 

So in some ways, by embracing defense OpenAI is just catching up. The difference is that defense-tech companies own that they’re in the business of warfare and haven’t had to rapidly disown a legacy as a nonprofit AI research company. From its founding charter, OpenAI has positioned itself as an organization on a mission to ensure that artificial general intelligence benefits all of humanity. It had publicly vowed that working with the military would contradict that mission.

Its October 24 blog post charted a new path, attempting to square OpenAI’s willingness to work in defense with its stated values. Titled “OpenAI’s approach to AI and national security,” it was released the same day the White House issued its National Security Memorandum on AI, which ordered the Pentagon and other agencies to ramp up their use of AI, in part to thwart competition from China.

“We believe a democratic vision for AI is essential to unlocking its full potential and ensuring its benefits are broadly shared,” OpenAI wrote, echoing similar language in the White House memo. “We believe democracies should continue to take the lead in AI development, guided by values like freedom, fairness, and respect for human rights.” 

It offered a number of ways OpenAI could help pursue that goal, including efforts to “streamline translation and summarization tasks, and study and mitigate civilian harm,” while still prohibiting its technology from being used to “harm people, destroy property, or develop weapons.” Above all, it was a message from OpenAI that it is on board with national security work. 

The new policies emphasize “flexibility and compliance with the law,” says Heidy Khlaaf, a chief AI scientist at the AI Now Institute and a safety researcher who authored a paper with OpenAI in 2022 about the possible hazards of its technology in contexts including the military. The company’s pivot “ultimately signals an acceptability in carrying out activities related to military and warfare as the Pentagon and US military see fit,” she says.

Amazon, Google, and OpenAI’s partner and investor Microsoft have competed for the Pentagon’s cloud computing contracts for years. Those companies have learned that working with defense can be incredibly lucrative, and OpenAI’s pivot, which comes as the company expects $5 billion in losses and is reportedly exploring new revenue streams like advertising, could signal that it wants a piece of those contracts. Big Tech’s relationships with the military also no longer elicit the outrage and scrutiny that they once did. But OpenAI is not a cloud provider, and the technology it’s building stands to do much more than simply store and retrieve data. With this new partnership, OpenAI promises to help sort through data on the battlefield, provide insights about threats, and help make the decision-making process in war faster and more efficient. 

OpenAI’s statements on national security perhaps raise more questions than they answer. The company wants to mitigate civilian harm, but for which civilians? Does contributing AI models to a program that takes down drones not count as developing weapons that could harm people?

“Defensive weapons are still indeed weapons,” Khlaaf says. They “can often be positioned offensively subject to the locale and aim of a mission.”

Beyond those questions, working in defense means that the world’s foremost AI company, which has had an incredible amount of leverage in the industry and has long pontificated about how to steward AI responsibly, will now work in a defense-tech industry that plays by an entirely different set of rules. In that system, when your customer is the US military, tech companies do not get to decide how their products are used. 

The startup trying to turn the web into a database

A startup called Exa is pitching a new spin on generative search. It uses the tech behind large language models to return lists of results that it claims are more on point than those from its rivals, including Google and OpenAI. The aim is to turn the internet’s chaotic tangle of web pages into a kind of directory, with results that are specific and precise.

Exa already provides its search engine as a back-end service to companies that want to build their own applications on top of it. Today it is launching the first consumer version of that search engine, called Websets.  

“The web is a collection of data, but it’s a mess,” says Exa cofounder and CEO Will Bryk. “There’s a Joe Rogan video over here, an Atlantic article over there. There’s no organization. But the dream is for the web to feel like a database.”

Websets is aimed at power users who need to look for things that other search engines aren’t great at finding, such as types of people or companies. Ask it for “startups making futuristic hardware” and you get a list of specific companies hundreds long rather than hit-or-miss links to web pages that mention those terms. Google can’t do that, says Bryk: “There’s a lot of valuable use cases for investors or recruiters or really anyone who wants any sort of data set from the web.”

Things have moved fast since MIT Technology Review broke the news in 2021 that Google researchers were exploring the use of large language models in a new kind of search engine. The idea soon attracted fierce critics. But tech companies took little notice. Three years on, giants like Google and Microsoft jostle with a raft of buzzy newcomers like Perplexity and OpenAI, which launched ChatGPT Search in October, for a piece of this hot new trend.

Exa isn’t (yet) trying to out-do any of those companies. Instead, it’s proposing something new. Most other search firms wrap large language models around existing search engines, using the models to analyze a user’s query and then summarize the results. But the search engines themselves haven’t changed much. Perplexity still directs its queries to Google Search or Bing, for example. Think of today’s AI search engines as a sandwich with fresh bread but stale filling.

More than keywords

Exa provides users with familiar lists of links but uses the tech behind large language models to reinvent how search itself is done. Here’s the basic idea: Google works by crawling the web and building a vast index of keywords that then get matched to users’ queries. Exa crawls the web and encodes the contents of web pages into a format known as embeddings, which can be processed by large language models.

Embeddings turn words into numbers in such a way that words with similar meanings become numbers with similar values. In effect, this lets Exa capture the meaning of text on web pages, not just the keywords.

Large language models use embeddings to predict the next words in a sentence. Exa’s search engine predicts the next link. Type “startups making futuristic hardware” and the model will come up with (real) links that might follow that phrase.

Exa’s approach comes at cost, however. Encoding pages rather than indexing keywords is slow and expensive. Exa has encoded some billion web pages, says Bryk. That’s tiny next to Google, which has indexed around a trillion. But Bryk doesn’t see this as a problem: “You don’t have to embed the whole web to be useful,” he says. (Fun fact: “exa” means a 1 followed by 18 0s and “googol” means a 1 followed by 100 0s.)

Websets is very slow at returning results. A search can sometimes take several minutes. But Bryk claims it’s worth it. “A lot of our customers started to ask for, like, thousands of results, or tens of thousands,” he says. “And they were okay with going to get a cup of coffee and coming back to a huge list.”

“I find Exa most useful when I don’t know exactly what I’m looking for,” says Andrew Gao, a computer science student at Stanford Univesrsity who has used the search engine. “For instance, the query ‘an interesting blog post on LLMs in finance’ works better on Exa than Perplexity.” But they’re good at different things, he says: “I use both for different purposes.”

“I think embeddings are a great way to represent entities like real-world people, places, and things,” says Mike Tung, CEO of Diffbot, a company using knowledge graphs to build yet another kind of search engine. But he notes that you lose a lot of information if you try to embed whole sentences or pages of text: “Representing War and Peace as a single embedding would lose nearly all of the specific events that happened in that story, leaving just a general sense of its genre and period.”

Bryk acknowledges that Exa is a work in progress. He points to other limitations, too. Exa is not as good as rival search engines if you just want to look up a single piece of information, such as the name of Taylor Swift’s boyfriend or who Will Bryk is: “It’ll give a lot of Polish-sounding people, because my last name is Polish and embeddings are bad at matching exact keywords,” he says.

For now Exa gets around this by throwing keywords back into the mix when they’re needed. But Bryk is bullish: “We’re covering up the gaps in the embedding method until the embedding method gets so good that we don’t need to cover up the gaps.”

What the departing White House chief tech advisor has to say on AI

President Biden’s administration will end within two months, and likely to depart with him is Arati Prabhakar, the top mind for science and technology in his cabinet. She has served as Director of the White House Office of Science and Technology Policy since 2022 and was the first to demonstrate ChatGPT to the president in the Oval Office. Prabhakar was instrumental in passing the president’s executive order on AI in 2023, which sets guidelines for tech companies to make AI safer and more transparent (though it relies on voluntary participation). 

The incoming Trump administration has not presented a clear thesis of how it will handle AI, but plenty of people in it will want to see that executive order nullified. Trump said as much in July, endorsing the 2024 Republican Party Platform that says the executive order “hinders AI innovation and imposes Radical Leftwing ideas on the development of this technology.” Venture capitalist Marc Andreessen has said he would support such a move. 

However, complicating that narrative will be Elon Musk, who for years has expressed fears about doomsday AI scenarios, and has been supportive of some regulations aiming to promote AI safety. 

As she prepares for the end of the administration, I sat down with Prabhakar and asked her to reflect on President Biden’s AI accomplishments, and how AI risks, immigration policies, the CHIPS Act and more could change under Trump.  

This conversation has been edited for length and clarity.

Every time a new AI model comes out, there are concerns about how it could be misused. As you think back to what were hypothetical safety concerns just two years ago, which ones have come true?

We identified a whole host of risks when large language models burst on the scene, and the one that has fully manifested in horrific ways is deepfakes and image-based sexual abuse. We’ve worked with our colleagues at the Gender Policy Council to urge industry to step up and take some immediate actions, which some of them are doing. There are a whole host of things that can be done—payment processors could actually make sure people are adhering to their Terms of Use. They don’t want to be supporting [image-based sexual abuse] and they can actually take more steps to make sure that they’re not. There’s legislation pending, but that’s still going to take some time.

Have there been risks that didn’t pan out to be as concerning as you predicted?

At first there was a lot of concern expressed by the AI developers about biological weapons. When people did the serious benchmarking about how much riskier that was compared with someone just doing Google searches, it turns out, there’s a marginally worse risk, but it is marginal. If you haven’t been thinking about how bad actors can do bad things, then the chatbots look incredibly alarming. But you really have to say, compared to what?

For many people, there’s a knee-jerk skepticism about the Department of Defense or police agencies going all in on AI. I’m curious what steps you think those agencies need to take to build trust.

If consumers don’t have confidence that the AI tools they’re interacting with are respecting their privacy, are not embedding bias and discrimination, that they’re not causing safety problems, then all the marvelous possibilities really aren’t going to materialize. Nowhere is that more true than national security and law enforcement. 

I’ll give you a great example. Facial recognition technology is an area where there have been horrific, inappropriate uses: take a grainy video from a convenience store and identify a black man who has never even been in that state, who’s then arrested for a crime he didn’t commit. (Editor’s note: Prabhakar is referring to this story). Wrongful arrests based on a really poor use of facial recognition technology, that has got to stop. 

In stark contrast to that, when I go through security at the airport now, it takes your picture and compares it to your ID to make sure that you are the person you say you are. That’s a very narrow, specific application that’s matching my image to my ID, and the sign tells me—and I know from our DHS colleagues that this is really the case—that they’re going to delete the image. That’s an efficient, responsible use of that kind of automated technology. Appropriate, respectful, responsible—that’s where we’ve got to go.

Were you surprised at the AI safety bill getting vetoed in California?

I wasn’t. I followed the debate, and I knew that there were strong views on both sides. I think what was expressed, that I think was accurate, by the opponents of that bill, is that it was simply impractical, because it was an expression of desire about how to assess safety, but we actually just don’t know how to do those things. No one knows. It’s not a secret, it’s a mystery. 

To me, it really reminds us that while all we want is to know how safe, effective and trustworthy a model is, we actually have very limited capacity to answer those questions. Those are actually very deep research questions, and a great example of the kind of public R&D that now needs to be done at a much deeper level.

Let’s talk about talent. Much of the recent National Security Memorandum on AI was about how to help the right talent come from abroad to the US to work on AI. Do you think we’re handling that in the right way?

It’s a hugely important issue. This is the ultimate American story, that people have come here throughout the centuries to build this country, and it’s as true now in science and technology fields as it’s ever been. We’re living in a different world. I came here as a small child because my parents came here in the early 1960s from India, and in that period, there were very limited opportunities [to emigrate to] many other parts of the world. 

One of the good pieces of news is that there is much more opportunity now. The other piece of news is that we do have a very critical strategic competition with the People’s Republic of China, and that makes it more complicated to figure out how to continue to have an open door for people who come seeking America’s advantages, while making sure that we continue to protect critical assets like our intellectual property. 

Do you think the divisive debates around immigration, especially around the time of the election, may hurt the US ability to bring the right talent into the country?

Because we’ve been stalled as a country on immigration for so long, what is caught up in that is our ability to deal with immigration for the STEM fields. It’s collateral damage.

Has the CHIPS Act been successful?

I’m a semiconductor person starting back with my graduate work. I was astonished and delighted when, after four decades, we actually decided to do something about the fact that semiconductor manufacturing capability got very dangerously concentrated in just one part of the world [Taiwan]. So it was critically important that, with the President’s leadership, we finally took action. And the work that the Commerce Department has done to get those manufacturing incentives out, I think they’ve done a terrific job.

One of the main beneficiaries so far of the CHIPS Act has been Intel. There’s varying degrees of confidence in whether it is going to deliver on building a domestic chip supply chain in the way that the CHIPS Act intended. Is it risky to put a lot of eggs in one basket for one chip maker?

I think the most important thing I see in terms of the industry with the CHIPS Act is that today we’ve got not just Intel, but TSMC, Samsung, SK Hynix and Micron. These are the five companies whose products and processes are at the most advanced nodes in semiconductor technology. They are all now building in the US. There’s no other part of the world that’s going to have all five of those. An industry is bigger than a company. I think when you look at the aggregate, that’s a signal to me that we’re on a very different track.

You are the President’s chief advisor for science and technology. I want to ask about the cultural authority that science has, or doesn’t have, today. RFK Jr. is the pick for health secretary, and in some ways, he captures a lot of frustration that Americans have about our healthcare system. In other ways, he has many views that can only be described as anti-science. How do you reflect on the authority that science has now?

I think it’s important to recognize that we live in a time when trust in institutions has declined across the board, though trust in science remains relatively high compared with what’s happened in other areas. But it’s very much part of this broader phenomenon, and I think that the scientific community has some roles [to play] here. The fact of the matter is that despite America having the best biomedical research that the world has ever seen, we don’t have robust health outcomes. Three dozen countries have longer life expectancies than America. That’s not okay, and that disconnect between advancing science and changing people’s lives is just not sustainable. The pact that science and technology and R&D makes with the American people is that if we make these public investments, it’s going to improve people’s lives and when that’s not happening, it does erode trust. 

Is it fair to say that that gap—between the expertise we have in the US and our poor health outcomes—explains some of the rise in conspiratorial thinking, in the disbelief of science?

It leaves room for that. Then there’s a quite problematic rejection of facts. It’s troubling if you’re a researcher, because you just know that what’s being said is not true. The thing that really bothers me is [that the rejection of facts] changes people’s lives, and it’s extremely dangerous and harmful. Think about if we lost herd immunity for some of the diseases for which we right now have fairly high levels of vaccination. It was an ugly world before we tamed infectious disease with the vaccines that we have. 

This manga publisher is using Anthropic’s AI to translate Japanese comics into English

A Japanese publishing startup is using Anthropic’s flagship large language model Claude to help translate manga into English, allowing the company to churn out a new title for a Western audience in just a few days rather than the two to three months it would take a team of humans.

Orange was founded by Shoko Ugaki, a manga superfan who (according to VP of product Rei Kuroda) has some 10,000 titles in his house. The company now wants more people outside Japan to have access to them. “I hope we can do a great job for our readers,” says Kuroda.

IMAGES COURTESY ORANGE / YAJIMA

But not everyone is happy. The firm has angered a number of manga fans who see the use of AI to translate a celebrated and traditional art form as one more front in the ongoing battle between tech companies and artists. “However well-intentioned this company might be, I find the idea of using AI to translate manga distasteful and insulting,” says Casey Brienza, a sociologist and author of the book Manga in America: Transnational Book Publishing and the Domestication of Japanese Comics.

Manga is a form of Japanese comic that has been around for more than a century. Hit titles are often translated into other languages and find a large global readership, especially in the US. Some, like Battle Angel Alita or One Piece, are turned into anime (animated versions of the comics) or live-action shows and become blockbuster movies and top Netflix picks. The US manga market was worth around $880 million in 2023 but is expected to reach $3.71 billion by 2030, according to some estimates. “It’s a huge growth market right now,” says Kuroda.

Orange wants a part of that international market. Only around 2% of titles published in Japan make it to the US, says Kuroda. As Orange sees it, the problem is that manga takes human translators too long to translate. By building AI tools to automate most of the tasks involved in translation—including extracting Japanese text from a comic’s panels, translating it into English, generating a new font, pasting the English back into the comic, and checking for mistranslations and typos—it can publish a translated mange title in around one-tenth the time it takes human translators and illustrators working by hand, the company says.

Humans still keep a close eye on the process, says Kuroda: “Honestly, AI makes mistakes. It sometimes misunderstands Japanese. It makes mistakes with artwork. We think humans plus AI is what’s important.”

Superheroes, aliens, cats

Manga is a complex art form. Stories are told via a mix of pictures and words, which can be descriptions or characters’ voices or sound effects, sometimes in speech bubbles and sometimes scrawled across the page. Single sentences can be split across multiple panels.

There are also diverse themes and narratives, says Kuroda: “There’s the student romance, mangas about gangs and murders, superheroes, aliens, cats.” Translations must capture the cultural nuance in each story. “This complexity makes localization work highly challenging,” he says.

Orange often starts with nothing more than the scanned image of a page. Its system first identifies which parts of the page show Japanese text, copies it, and erases the text from each panel. These snippets of text are then combined into whole sentences and passed to the translation module, which not only translates the text into English but keeps track of where on the page each individual snippet comes from. Because Japanese and English have a very different word order, the snippets need to be reordered, and the new English text must be placed on the page in different places from where the Japanese equivalent had come from—all without messing up the sequence of images.

“Generally, the images are the most important part of the story,” says Frederik Schodt, an award-winning manga translator who published his first translation in 1977. “Any language cannot contradict the images, so you can’t take many of the liberties that you might in translating a novel. You can’t rearrange paragraphs or change things around much.”

IMAGES COURTESY ORANGE / YAJIMA

Orange tried several large language models, including its own, developed in house, before picking Claude 3.5. “We’re always evaluating new models,” says Kuroda. “Right now Claude gives us the most natural tone.”

Claude also has an agent framework that lets several sub-models work together on an overall task. Orange uses this framework to juggle the multiple steps in the translation process.

Orange distributes its translations via an app called Emaqi (a pun on “emaki,” the ancient Japanese illustrated scrolls that are considered a precursor to manga). It also wants to be a translator-for-hire for US publishers.

But Orange has not been welcomed by all US fans. When it showed up at Anime NYC, a US anime convention, this summer, the Japanese-to-English translator Jan Mitsuko Cash tweeted: “A company like Orange has no place at the convention hosting the Manga Awards, which celebrates manga and manga professionals in the industry. If you agree, please encourage @animenyc to ban AI companies from exhibiting or hosting panels.”  

Brienza takes the same view. “Work in the culture industries, including translation, which ultimately is about translating human intention, not mere words on a page, can be poorly paid and precarious,” she says. “If this is the way the wind is blowing, I can only grieve for those who will go from making little money to none.”

Some have also called Orange out for cutting corners. “The manga uses stylized text to represent the inner thoughts that the [protagonist] can’t quite voice,” another fan tweeted. “But Orange didn’t pay a redrawer or letterer to replicate it properly. They also just skip over some text entirely.”

EMAQI

Everyone at Orange understands that manga translation is a sensitive issue, says Kuroda: “We believe that human creativity is absolutely irreplaceable, which is why all AI-assisted work is rigorously reviewed, refined, and finalized by a team of people.”  

Orange also claims that the authors it has translated are on board with its approach. “I’m genuinely happy with how the English version turned out,” says Kenji Yajima, one of the authors Orange has worked with, referring to the company’s translation of his title Neko Oji: Salaryman reincarnated as a kitten! (see images). “As a manga artist, seeing my work shared in other languages is always exciting. It’s a chance to connect with readers I never imagined reaching before.”

Schodt sees the upside too. He notes that the US is flooded with poor-quality, unofficial fan-made translations. “The number of pirated translations is huge,” he says. “It’s like a parallel universe.”

He thinks using AI to streamline translation is inevitable. “It’s the dream of many companies right now,” he says. “But it will take a huge investment.” He believes that really good translation will require large language models trained specifically on manga: “It’s not something that one small company is going to be able to pull off.”

“Whether this will prove economically feasible right now is anyone’s guess,” says Schodt. “There is a lot of advertising hype going on, but the readers will have the final judgment.”

These AI Minecraft characters did weirdly human stuff all on their own

Left to their own devices, an army of AI characters didn’t just survive — they thrived. They developed in-game jobs, shared memes, voted on tax reforms and even spread a religion.

The experiment played out on the open-world gaming platform Minecraft, where up to 1000 software agents at a time used large language models (LLMs) to interact with one another. Given just a nudge through text prompting, they developed a remarkable range of personality traits, preferences and specialist roles, with no further inputs from their human creators. 

The work, from AI startup Altera, is part of a broader field that wants to use simulated agents to model how human groups would react to new economic policies or other interventions.

But for Altera’s founder, Robert Yang, who quit his position as an assistant professor in computational neuroscience at MIT to start the company, this demo is just the beginning. He sees it as an early  step towards large-scale “AI civilizations” that can coexist and work alongside us in digital spaces. “The true power of AI will be unlocked when we have actually truly autonomous agents that can collaborate at scale,” says Yang.

Yang was inspired by Stanford University researcher Joon Sung Park who, in 2023, found that surprisingly humanlike behaviors arose when a group of 25 autonomous AI agents was let loose to interact in a basic digital world. 

“Once his paper was out, we started to work on it the next week,” says Yang. “I quit MIT six months after that.”

Yang wanted to take the idea to its extreme. “We wanted to push the limit of what agents can do in groups autonomously.”

Altera quickly raised more than $11m in funding from investors including A16Z and the former Google CEO Eric Schmidt’s emerging tech VC firm. Earlier this year Altera released its first demo: an AI-controlled character in Minecraft that plays alongside you.

Altera’s new experiment, Project Sid, uses simulated AI agents equipped with “brains” made up of multiple modules. Some modules are powered by LLMs and designed to specialize in certain tasks, such as reacting to other agents, speaking, or planning the agent’s next move.

ALTERA

The team started small, testing groups of around 50 agents in Minecraft to observe their interactions. Over 12 in-game days (4 real-world hours) the agents began to exhibit some interesting emergent behavior. For example, some became very sociable and made many connections with other characters, while others appeared more introverted. The “likability” rating of each agent (measured by the agents themselves) changed over time as the interactions continued. The agents were able to track these social cues and react to them: in one case an AI chef tasked with distributing food to the hungry gave more to those who he felt valued him most.

More humanlike behaviors emerged in a series of 30-agent simulations. Despite all the agents starting with the same personality and same overall goal—to create an efficient village and protect the community against attacks from other in-game creatures—they spontaneously developed specialized roles within the community, without any prompting.  They diversified into roles such as builder, defender, trader, and explorer. Once an agent had started to specialize, its in-game actions began to reflect its new role. For example, an artist spent more time picking flowers, farmers gathered seeds and guards built more fences. 

“We were surprised to see that if you put [in] the right kind of brain, they can have really emergent behavior,” says Yang. “That’s what we expect humans to have, but don’t expect machines to have.”

Yang’s team also tested whether agents could follow community-wide rules. They introduced a world with basic tax laws and allowed agents to vote for changes to the in-game taxation system. Agents prompted to be pro or anti tax were able to influence the behavior of other agents around them, enough that they would then vote to reduce or raise tax depending on who they had interacted with.

The team scaled up, pushing the number of agents in each simulation to the maximum the Minecraft server could handle without glitching, up to 1000 at once in some cases. In one of Altera’s 500-agent simulations, they watched how the agents spontaneously came up with and then spread cultural memes (such as a fondness for pranking, or an interest in eco-related issues) among their fellow agents. The team also seeded a small group of agents to try to spread the (parody) religion, Pastafarianism, around different towns and rural areas that made up the in-game world, and watched as these Pastafarian priests converted many of the agents they interacted with. The converts went on to spread Pastafarianism (the word of the Church of the Flying Spaghetti Monster) to nearby towns in the game world.

The way the agents acted might seem eerily lifelike, but their behavior combines patterns learned by the LLMs from human-created data with Altera’s system, which translates those patterns into context-aware actions, like picking up a tool, or interacting with another agent. “The takeaway is that LLMs have a sophisticated enough model of human social dynamics [to] mirror these human behaviors,” says Altera co-founder Andrew Ahn.

ALTERA

In other words, the data makes them excellent mimics of human behavior, but they are in no way “alive”.

But Yang has grander plans. Altera plans to expand into Roblox next, but Yang hopes to eventually move beyond game worlds altogether. Ultimately, his goal is a world in which humans don’t just play alongside AI characters, but also interact with them in their day-to-day lives. His dream is to create a vast number of “digital humans” who actually care for us and will work with us to help us solve problems, as well as keep us entertained. “We want to build agents that can really love humans (like dogs love humans, for example),” he says.

This viewpoint—that AI could love us—is pretty controversial in the field, with many experts arguing it’s not possible to recreate emotions in machines using current techniques. AI veteran Julian Togelius, for example, who runs games testing company Modl.ai, says he likes Altera’s work, particularly because it lets us study human behavior in simulation.

But could these simulated agents ever learn to care for us, love us, or become self-aware? Togelius doesn’t think so. “There is no reason to believe a neural network running on a GPU somewhere experiences anything at all,” he says.

But maybe AI doesn’t have to love us for real to be useful.

“If the question is whether one of these simulated beings could appear to care, and do it so expertly that it would have the same value to someone as being cared for by a human, that is perhaps not impossible,” Togelius adds. “You could create a good-enough simulation of care to be useful. The question is whether the person being cared for would care that the carer has no experiences.”

In other words, so long as our AI characters appear to care for us in a convincing way, that might be all we really care about.

Update: We gave more detail on how Altera’s system combines LLMs with other modules.