Do animals have names? According to the poet T.S. Eliot, cats have three: the name their owner calls them (like George); a second, more noble one (like Quaxo or Cricopat); and, finally, a “deep and inscrutable” name known only to themselves “that no human research can discover.”
But now, researchers armed with audio recorders and pattern-recognition software are making unexpected discoveries about the secrets of animal names—at least with small monkeys called marmosets.
That’s according to a team at Hebrew University in Israel, who claim in the journal Science this week they’ve discovered that marmosets “vocally label” their monkey friends with specific sounds.
Until now, only humans, dolphins, elephants, and probably parrots had been known to use specific sounds to call out to other individuals.
Marmosets are highly social creatures that maintain contact through high-pitched chirps and twitters called “phee-calls.” By recording different pairs of monkeys placed near each other, the team in Israel says they found the animals will adjust their sounds toward a vocal label that’s specific to their conversation partner.
“It’s similar to names in humans,” says David Omer, the neuroscientist who led the project. “There’s a typical time structure to their calls, and what we report is that the monkey fine-tunes it to encode an individual.”
These names aren’t really recognizable to the human ear; instead, they were identified via a “random forest,” the statistical machine learning technique Omer’s team used to cluster, classify, and analyze the sounds.
To prove they’d cracked the monkey code—and learned the secret names—the team played recordings at the marmosets through a speaker and found they responded more often when their label, or name, was in the recording.
This sort of research could provide clues to the origins of human language, which is arguably the most powerful innovation in our species’ evolution, right up there with opposable thumbs. In years past, it’s been argued that human language is unique and that animals lack both the brains and vocal apparatus to converse.
But there’s growing evidence that isn’t the case, especially now that the use of names has been found in at least four distantly related species. “This is very strong evidence that the evolution of language was not a singular event,” says Omer.
Some similar research tactics were reported earlier this year by Mickey Pardo, a postdoctoral researcher, now at Cornell University, who spent 14 months in Kenya recording elephant calls. Elephants sound alarms by trumpeting, but in reality most of their vocalizations are deep rumbles that are only partly audible to humans.
Pardo also found evidence that elephants use vocal labels, and he says he can definitely get an elephant’s attention by playing the sound of another elephant addressing it. But does this mean researchers are now “speaking animal”?
Not quite, says Pardo. Real language, he thinks, would mean the ability to discuss things that happened in the past or string together more complex ideas. Pardo says he’s hoping to determine next if elephants have specific sounds for deciding which watering hole to visit—that is, whether they employ place names.
Several efforts are underway to discover if there’s still more meaning in animal sounds than we thought. This year, a group called Project CETI that’s studying the songs of sperm whales found they are far more complex than previously recognized. It means the animals, in theory, could be using a kind of grammar—although whether they actually are saying anything specific isn’t known.
Another effort, the Earth Species Project, aims to use “artificial intelligence to decode nonhuman communication” and has started helping researchers collect more data on animal sounds to feed into those models.
The team in Israel say they will also be giving the latest types of artificial intelligence a try. Their marmosets live in a laboratory facility, and Omer says he’s already put microphones in monkeys’ living space in order to record everything they say, 24 hours a day.
Their chatter, Omer says, will be used to train a large language model that could, in theory, be used to finish a series of calls that a monkey started, or produce what it predicts is an appropriate reply. But will a primate language model actually make sense, or will it just gibber away without meaning?
Only the monkeys will be able to say for sure.
“I don’t have any delusional expectations that they will talk about Nietzsche,” says Omer. “I don’t expect it to be extremely complex like a human, but I would expect it to help us understand something about how our language developed.”
The power of pornography doesn’t lie in arousal but in questions. What is obscene? What is ethical or safe to watch?
We don’t have to consume or even support it, but porn will still demand answers. The question now is: What is “real” porn?
Anti-porn crusades have been at the heart of the US culture wars for generations, but by the start of the 2000s, the issue had lost its hold. Smartphones made porn too easy to spread and hard to muzzle. Porn became a politically sticky issue, too entangled with free speech and evolving tech. An uneasy truce was made: As long as the imagery was created by consenting adults and stayed on the other side of paywalls and age verification systems, it was to be left alone.
But today, as AI porn infiltrates dinner tables, PTA meetings, and courtrooms, that truce may not endure much longer. The issue is already making its way back into the national discourse; Project 2025, the Heritage Foundation–backed policy plan for a future Republican administration, proposes the criminalization of porn and the arrest of its creators.
But what if porn is wholly created by an algorithm? In that case, whether it’s obscene, ethical, or safe becomes secondary to What does it mean for porn to be “real”—and what will the answer demand from all of us?
During my time as a filmmaker in adult entertainment, I witnessed seismic shifts: the evolution from tape to digital, the introduction of new HIV preventions, and the disruption of the industry by free streaming and social media. An early tech adopter, porn was an industry built on desires, greed, and fantasy, propped up by performances and pharmaceuticals. Its methods and media varied widely, but the one constant was its messy humanity. Until now.
What does it mean for porn to be “real”—and what will the answer demand from all of us?
When AI-generated pornography first emerged, it was easy to keep a forensic distance from the early images and dismiss them as a parlor trick. They were laughable and creepy: cheerleaders with seven fingers and dead, wonky eyes. Then, seemingly overnight, they reached uncanny photorealism. Synthetic erotica, like hentai and CGI, has existed for decades, but I had never seen porn like this. These were the hallucinations of a machine trained on a million pornographic images, both the creation of porn and a distillation of it. Femmes fatales with psychedelic genitalia, straight male celebrities in same-sex scenes, naked girls in crowded grocery stores—posted not in the dark corners of the internet but on social media. The images were glistening and warm, raising fresh questions about consent and privacy. What would these new images turn us into?
In September of 2023, the small Spanish town of Almendralejo was forced to confront this question. Twenty girls returned from summer break to find naked selfies they’d never taken being passed around at school. Boys had rendered the images using an AI “nudify” app with just a few euros and a yearbook photo. The girls were bullied and blackmailed, suffered panic attacks and depression. The youngest was 11. The school and parents were at a loss. The tools had arrived faster than the speed of conversation, and they did not discriminate. By the end of the school year, similar cases had spread to Australia, Quebec, London, and Mexico. Then explicit AI images of Taylor Swift flooded social media. If she couldn’t stop this, a 15-year-old from Michigan stood no chance.
The technology behind pornography never slows down, regardless of controversies. When students return to school this fall, it will be in the shadow of AI video engines like Sora and Runway 3, which produce realistic video from text prompts and photographs. If still images have caused so much global havoc, imagine what video could do and where the footage could end up.
As porn becomes more personal, it’s also becoming more personalized. Users can now check boxes on a list of options as long as the Cheesecake Factory menu to create their ideal scenes: categories like male, female, and trans; ages from 18 to 90; breast and penis size; details like tan lines and underwear color; backdrops like grocery stores, churches, the Eiffel Tower, and Stonehenge; even weather, like tornadoes. It may be 1s and 0s, but AI holds no binary; it holds no judgment or beauty standards. It can render seldom-represented bodies, like those of mature, transgender, and disabled people, in all pairings. Hyper-customizable porn will no longer require performers—only selections and an answer to the question “What is it that I really like?” While Hollywood grapples with the ethics of AI, artificial porn films will become a reality. Celebrities may boost their careers by promoting their synthetic sex tapes on late-night shows.
The progress of AI porn may shift our memories, too. AI is already used to extend home movies and turn vintage photos into live-action scenes. What happens when we apply this to sex? Early sexual images etch themselves on us: glimpses of flesh from our first crush, a lost lover, a stranger on the bus. These erotic memories depend on the specific details for their power: a trail of hair, panties in a specific color, sunlight on wet lips, my PE teacher’s red gym shorts. They are ideal for AI prompts.
Porn and real-life sex affect each other in a loop. If people become accustomed to getting exactly what they want from erotic media, this could further affect their expectations of relationships. A first date may have another layer of awkwardness if each party has already seen an idealized, naked digital doppelganger of the other.
Despite (or because of) this blurring of lines, we may actually start to see a genre of “ethical porn.” Without the need for sets, shoots, or even performers, future porn studios might not deal with humans at all. This may be appealing for some viewers, who can be sure that new actors are not underage, trafficked, or under the influence.
A synergy has been brewing since the ’90s, when CD-ROM games, life-size silicone dolls, and websites introduced “interactivity” to adult entertainment. Thirty years later, AI chatbot “partners” and cheaper, lifelike sex dolls are more accessible than ever. Porn tends to merge all available tech toward complete erotic immersion. The realism of AI models has already broken the dam to the uncanny valley. Soon, these avatars will be powered by chatbots and embodied in three-dimensional prosthetics, all existing in virtual-reality worlds. What follows will be the fabled sex robot.
So what happens when we’ve removed the “messy humanity” from sex itself? Porn is defined by the needs of its era. Ours has been marked by increasing isolation. The pandemic further conditioned us to digitize our most intimate moments, bringing us FaceTime hospital visits and weddings, and caused a deep discharge of our social batteries. Adult entertainment may step into that void. The rise of AI-generated porn may be a symptom of a new synthetic sexuality, not the cause. In the near future, we may find this porn arousing because of its artificiality, not in spite of it.
Leo Herrera is a writer and artist. He explores how tech intersects with sex and culture on Substack at Herrera Words.
Open-source AI is everywhere right now. The problem is, no one agrees on what it actually is. Now we may finally have an answer. The Open Source Initiative (OSI), the self-appointed arbiters of what it means to be open source, has released a new definition, which it hopes will help lawmakers develop regulations to protect consumers from AI risks.
Though OSI has published much about what constitutes open-source technology in other fields, this marks its first attempt to define the term for AI models. It asked a 70-person group of researchers, lawyers, policymakers, and activists, as well as representatives from big tech companies like Meta, Google, and Amazon, to come up with the working definition.
According to the group, an open-source AI system can be used for any purpose without securing permission, and researchers should be able to inspect its components and study how the system works.
It should also be possible to modify the system for any purpose—including to change its output—and to share it with others to use, with or without modifications, for any purpose. In addition, the standard attempts to define a level of transparency for a given model’s training data, source code, and weights.
The previous lack of an open-source standard presented a problem. Although we know that the decisions of OpenAI and Anthropic to keep their models, data sets, and algorithms secret makes their AI closed source, some experts argue that Meta and Google’s freely accessible models, which are open to anyone to inspect and adapt, aren’t truly open source either, because of licenses that restrict what users can do with the models and because the training data sets aren’t made public. Meta, Google, and OpenAI have been contacted for their response to the new definition but did not reply before publication.
“Companies have been known to misuse the term when marketing their models,” says Avijit Ghosh, an applied policy researcher at Hugging Face, a platform for building and sharing AI models. Describing models as open source may cause them to be perceived as more trustworthy, even if researchers aren’t able to independently investigate whether they really are open source.
Ayah Bdeir, a senior advisor to Mozilla and a participant in OSI’s process, says certain parts of the open-source definition were relatively easy to agree upon, including the need to reveal model weights (the parameters that help determine how an AI model generates an output). Other parts of the deliberations were more contentious, particularly the question of how public training data should be.
The lack of transparency about where training data comes from has led to innumerable lawsuits against big AI companies, from makers of large language models like OpenAI to music generators like Suno, which do not disclose much about their training sets beyond saying they contain “publicly accessible information.” In response, some advocates say that open-source models should disclose all their training sets, a standard that Bdeir says would be difficult to enforce because of issues like copyright and data ownership.
Ultimately, the new definition requires that open-source models provide information about the training data to the extent that “a skilled person can recreate a substantially equivalent system using the same or similar data.” It’s not a blanket requirement to share all training data sets, but it also goes further than what many proprietary models or even ostensibly open-source models do today. It’s a compromise.
“Insisting on an ideologically pristine kind of gold standard that actually will not effectively be met by anybody ends up backfiring,” Bdeir says. She adds that OSI is planning some sort of enforcement mechanism, which will flag models that are described as open source but do not meet its definition. It also plans to release a list of AI models that do meet the new definition. Though none are confirmed, the handful of models that Bdeir told MIT Technology Review are expected to land on the list are relatively small names, including Pythia by Eleuther, OLMo by Ai2, and models by the open-source collective LLM360.
As a lifelong disabled person who constantly copes with multiple conditions, I have a natural tendency to view emerging technologies with skepticism. Most new things are built for the majority of people—in this case, people without disabilities—and the truth of the matter is there’s no guarantee I’ll have access to them.
There are certainly exceptions to the rule. A prime example is the iPhone. Although discrete accessibility software did not appear until the device’s third-generation model, in 2009, earlier generations were still revolutionary for me. After I’d spent years using flip phones with postage-stamp-size screens and hard-to-press buttons, the fact that the original iPhone had a relatively large screen and a touch-based UI was accessibility unto itself.
AI could make these kinds of jumps in accessibility more common across a wide range of technologies. But you probably haven’t heard much about that possibility. While the New York Times sues OpenAI over ChatGPT’s scraping of its content and everyone ruminates over the ethics of AI tools, there seems to be less consideration of the good ChatGPT can do for people of various abilities. For someone with visual and motor delays, using ChatGPT to do research can be a lifesaver. Instead of trying to manage a dozen browser tabs with Google searches and other pertinent information, you can have ChatGPT collate everything into one space. Likewise, it’s highly plausible that artists who can’t draw in the conventional manner could use voice prompts to have Midjourney or Adobe Firefly create what they’re thinking of. That might be the only way for such a person to indulge an artistic passion.
For those who, like me, are blind or have low vision, the ability to summon a ride on demand and go anywhere without imposing on anyone else for help is a huge deal.
Of course, data needs to be vetted for accuracy and gathered with permission—there are ample reasons to be wary of AI’s potential to serve up wrong or potentially harmful, ableist information about the disabled community. Still, it feels unappreciated (and underreported) that AI-based software can truly be an assistive technology, enabling people to do things they otherwise would be excluded from. AI could give a disabled person agency and autonomy. That’s the whole point of accessibility—freeing people in a society not designed for their needs.
The ability to automatically generate video captions and image descriptions provides additional examples of how automation can make computers and productivity technology more accessible. And more broadly, it’s hard not to be enthused about ever-burgeoning technologies like autonomous vehicles. Most tech journalists and other industry watchers are interested in self-driving cars for the sheer novelty, but the reality is the AI software behind vehicles like Waymo’s fleet of Jaguar SUVs is quite literally enabling many in the disability community to exert more agency over their transport. For those who, like me, are blind or have low vision, the ability to summon a ride on demand and go anywhere without imposing on anyone else for help is a huge deal. It’s not hard to envision a future in which, as the technology matures, autonomous vehicles are normalized to the point where blind people could buy their own cars.
At the same time, AI is enabling serious advances in technology for people with limb differences. How exciting will it be, decades from now, to have synthetic arms and legs, hands or feet, that more or less function like the real things? Similarly, the team at Boston-based Tatum Robotics is combining hardware with AI to make communication more accessible for deaf-blind people: A robotic hand forms hand signs, or words in American Sign Language that can be read tactilely against the palm. Like autonomous vehicles, these applications have enormous potential to positively influence the everyday lives of countless people. All this goes far beyond mere chatbots.
It should be noted that disabled people historically have been among the earliest adopters of new technologies. AI is no different, yet public discourse routinely fails to meaningfully account for this. After all, AI plays to a computer’s greatest strength: automation. As time marches on, the way AI grows and evolves will be unmistakably and indelibly shaped by disabled people and our myriad needs and tolerances. It will offer us more access to information, to productivity, and most important, to society writ large.
Steven Aquino is a freelance tech journalist covering accessibility and assistive technologies. He is based in San Francisco.
Even the most capable robots aren’t great at sensing human touch; you typically need a computer science degree or at least a tablet to interact with them effectively. That may change, thanks to robots that can now sense and interpret touch without being covered in high-tech artificial skin.It’s a significant step toward robots that can interact more intuitively with humans.
To understand the new approach, led by the German Aerospace Center and published today in Science Robotics, consider the two distinct ways our own bodies sense touch. If you hold your left palm facing up and press lightly on your left pinky finger, you may first recognize that touch through the skin of your fingertip. That makes sense–you have thousands of receptors on your hands and fingers alone. Roboticists often try to replicate that blanket of sensors for robots through artificial skins, but these can be expensive and ineffective at withstanding impacts or harsh environments.
But if you press harder, you may notice a second way of sensing the touch: through your knuckles and other joints. That sensation–a feeling of torque, to use the robotics jargon–is exactly what the researchers have re-created in their new system.
Their robotic arm contains six sensors, each of which can register even incredibly small amounts of pressure against any section of the device. After precisely measuring the amount and angle of that force, a series of algorithms can then map where a person is touching the robot and analyze what exactly they’re trying to communicate. For example, a person could draw letters or numbers anywhere on the robotic arm’s surface with a finger, and the robot could interpret directions from those movements. Any part of the robot could also be used as a virtual button.
It means that every square inch of the robot essentially becomes a touch screen, except without the cost, fragility, and wiring of one, says Maged Iskandar, researcher at the German Aerospace Center and lead author of the study.
“Human-robot interaction, where a human can closely interact with and command a robot, is still not optimal, because the human needs an input device,” Iskandar says. “If you can use the robot itself as a device, the interactions will be more fluid.”
A system like this could provide a cheaper and simpler way of providing not only a sense of touch, but also a new way to communicate with robots. That could be particularly significant for larger robots, like humanoids, which continue to receive billions in venture capital investment.
Calogero Maria Oddo, a roboticist who leads the Neuro-Robotic Touch Laboratory at the BioRobotics Institute but was not involved in the work, says the development is significant, thanks to the way the research combines sensors, elegant use of mathematics to map out touch, and new AI methods to put it all together. Oddo says commercial adoption could be fairly quick, since the investment required is more in software than hardware, which is far more expensive.
There are caveats, though. For one, the new model cannot handle more than two points of contact at once. In a fairly controlled setting like a factory floor that might not be an issue, but in environments where human-robot interactions are less predictable, it could present limitations. And the sorts of sensors needed to communicate touch to a robot, though commercially available, can also cost tens of thousands of dollars.
Overall, though, Oddo envisions a future where skin-based sensors and joint-based ones are merged to give robots a more comprehensive sense of touch.
“We humans and other animals have integrated both solutions,” he says. “I expect robots working in the real world will use both, too, to interact safely and smoothly with the world and learn.”
MIT Technology Review’s What’s Next series looks across industries, trends, and technologies to give you a first look at the future. You can read the rest of them here.
Drones have been a mainstay technology among militaries, hobbyists, and first responders alike for more than a decade, and in that time the range available has skyrocketed. No longer limited to small quadcopters with insufficient battery life, drones are aiding search and rescue efforts, reshaping wars in Ukraine and Gaza, and delivering time-sensitive packages of medical supplies. And billions of dollars are being plowed into building the next generation of fully autonomous systems.
These developments raise a number of questions: Are drones safe enough to be flown in dense neighborhoods and cities? Is it a violation of people’s privacy for police to fly drones overhead at an event or protest? Who decides what level of drone autonomy is acceptable in a war zone?
Those questions are no longer hypothetical. Advancements in drone technology and sensors, falling prices, and easing regulations are making drones cheaper, faster, and more capable than ever. Here’s a look at four of the biggest changes coming to drone technology in the near future.
Police drone fleets
Today more than 1,500 US police departments have drone programs, according to tracking conducted by the Atlas of Surveillance. Trained police pilots use drones for search and rescue operations, monitoring events and crowds, and other purposes. The Scottsdale Police Department in Arizona, for example, successfully used a drone to locate a lost elderly man with dementia, says Rich Slavin, Scottsdale’s assistant chief of police. He says the department has had useful but limited experiences with drones to date, but its pilots have often been hamstrung by the “line of sight” rule from the Federal Aviation Administration (FAA). The rule stipulates that pilots must be able to see their drones at all times, which severely limits the drone’s range.
Soon, that will change. On a rooftop somewhere in the city, Scottsdale police will in the coming months install a new police drone capable of autonomous takeoff, flight, and landing. Slavin says the department is seeking a waiver from the FAA to be able to fly its drone past the line of sight. (Hundreds of police agencies have received a waiver from the FAA since the first was granted in 2019.) The drone, which can fly up to 57 miles per hour, will go on missions as far as three miles from its docking station, and the department says it will be used for things like tracking suspects or providing a visual feed of an officer at a traffic stop who is waiting for backup.
“The FAA has been much more progressive in how we’re moving into this space,” Slavin says. That could mean that around the country, the sight (and sound) of a police drone soaring overhead will become much more common.
The Scottsdale department says the drone, which it is purchasing from Aerodome, will kick off its drone-as-first-responder program and will play a role in the department’s new “real-time crime center.” These sorts of centers are becoming increasingly common in US policing, and allow cities to connect cameras, license plate readers, drones, and other monitoring methods to track situations on the fly. The rise of the centers, and their associated reliance on drones, has drawn criticism from privacy advocates who say they conduct a great deal of surveillance with little transparency about how footage from drones and other sources will be used or shared.
In 2019, the police department in Chula Vista, California, was the first to receive a waiver from the FAA to fly beyond line of sight. The program sparked criticism from members of the community who alleged the department was not transparent about the footage it collected or how it would be used.
Jay Stanley, a senior policy analyst at the American Civil Liberties Union’s Speech, Privacy, and Technology Project, says the waivers exacerbate existing privacy issues related to drones. If the FAA continues to grant them, police departments will be able to cover far more of a city with drones than ever, all while the legal landscape is murky about whether this would constitute an invasion of privacy.
“If there’s an accumulation of different uses of this technology, we’re going to end up in a world where from the moment you step out of your front door, you’re going to feel as though you’re under the constant eye of law enforcement from the sky,” he says. “It may have some real benefits, but it is also in dire need of strong checks and balances.”
Scottsdale police say the drone could be used in a variety of scenarios, such as responding to a burglary in progress or tracking a driver with suspected connection to a kidnapping. But the real benefit, Slavin says, will come from pairing it with other existing technologies, like automatic license plate readers and hundreds of cameras placed around the city. “It can get to places very, very quickly,” he says. “It gives us real-time intelligence and helps us respond faster and smarter.”
While police departments might indeed benefit from drones in those situations, Stanley says the ACLU has found that many deploy them for far more ordinary cases, like reports of a kid throwing a ball against a garage or of “suspicious persons” in an area.
“It raises the question about whether these programs will just end up being another way in which vulnerable communities are over-policed and nickeled and dimed by law enforcement agencies coming down on people for all kinds of minor transgressions,” he says.
Drone deliveries, again
Perhaps no drone technology is more overhyped than home deliveries. For years, tech companies have teased futuristic renderings of a drone dropping off a package on your doorstep just hours after you ordered it. But they’ve never managed to expand them much beyond small-scale pilot projects, at least in the US, again largely due to the FAA’s line of sight rules.
But this year, regulatory changes are coming. Like police departments, Amazon’s Prime Air program was previously limited to flying its drones within the pilot’s line of sight. That’s because drone pilots don’t have radar, air traffic controllers, or any of the other systems commercial flight relies on to monitor airways and keep them safe. To compensate, Amazon spent years developing an onboard system that would allow its drones to detect nearby objects and avoid collisions. The company says it showed the FAA in demonstrations that its drones could fly safely in the same airspace as helicopters, planes, and hot air balloons.
In May, Amazon announced the FAA had granted the company a waiver and permission to expand operations in Texas, more than a decade after the Prime Air project started. And in July, the FAA cleared one more roadblock by allowing two companies—Zipline as well as Google’s Wing Aviation—to fly in the same airspace simultaneously without the need for visual observers.
While all this means your chances of receiving a package via drone have ticked up ever so slightly, the more compelling use case might be medical deliveries. Shakiba Enayati, an assistant professor of supply chains at the University of Missouri–St. Louis, has spent years researching how drones could conduct last-mile deliveries of vaccines, antivenom, organs, and blood in remote places. She says her studies have found drones to be game changers for getting medical supplies to underserved populations, and if the FAA extends these regulatory changes, it could have a real impact.
That’s especially true in the steps leading up to an organ transplant, she says. Before an organ can be transmitted to a recipient, a number of blood tests must be sent back-and-forth to make sure the recipient can accept it, which takes a time if the blood is being transferred by car or even helicopter. “In these cases, the clock is ticking,” Enayati says. If drones were allowed to be used in this step at scale, it would be a significant improvement.
“If the technology is supporting the needs of organ delivery, it’s going to make a big change in such an important arena,” she says.
That development could come sooner than using drones for delivery of the actual organs, which have to be transported under very tightly controlled conditions to preserve them.
Domesticating the drone supply chain
Signed into law last December, the American Security Drone Act bars federal agencies from buying drones from countries thought to pose a threat to US national security, such as Russia and China. That’s significant. China is the undisputed leader when it comes to manufacturing drones and drone parts, with over 90% of law enforcement drones in the US made by Shenzhen-based DJI, and many drones used by both sides of the war in Ukraine are made by Chinese companies.
The American Security Drone Act is part of an effort to curb that reliance on China. (Meanwhile, China is stepping up export restrictions on drones with military uses.) As part of the act, the US Department of Defense’s Defense Innovation Unit has created the Blue UAS Cleared List, a list of drones and parts the agency has investigated and approved for purchase. The list applies to federal agencies as well as programs that receive federal funding, which often means state police departments or other non-federal agencies.
Since the US is set to spend such significant sums on drones—with $1 billion earmarked for the Department of Defense’s Replicator initiative alone—getting on the Blue List is a big deal. It means those federal agencies can make large purchases with little red tape.
Allan Evans, CEO of US-based drone part maker Unusual Machine, says the list has sparked a significant rush of drone companies attempting to conform to the US standards. His company manufactures a first-person view flight controller that he hopes will become the first of its kind to be approved for the Blue List.
The American Security Drone Act is unlikely to affect private purchases in the US of drones used by videographers, drone racers, or hobbyists, which will overwhelmingly still be made by China-based companies like DJI. That means any US-based drone companies, at least in the short term, will only survive by catering to the US defense market.
“Basically any US company that isn’t willing to have ancillary involvement in defense work will lose,” Evans says.
The coming months will show the law’s true impact: Because the US fiscal year ends in September, Evans says he expects to see a host of agencies spending their use-it-or-lose-it funding on US-made drones and drone components in the next month. “That will indicate whether the marketplace is real or not, and how much money is actually being put toward it,” he says.
Autonomous weapons in Ukraine
The drone war in Ukraine has largely been one of attrition. Drones have been used extensively for surveying damage, finding and tracking targets, or dropping weapons since the war began, but on average these quadcopter drones last just three flights before being shot down or rendered unnavigable by GPS jamming. As a result, both Ukraine and Russia prioritized accumulating high volumes of drones with the expectation that they wouldn’t last long in battle.
Now they’re having to rethink that approach, according to Andriy Dovbenko, founder of the UK-Ukraine Tech Exchange, a nonprofit that helps startups involved in Ukraine’s war effort and eventual reconstruction raise capital. While working with drone makers in Ukraine, he says, he has seen the demand for technology shift from big shipments of simple commercial drones to a pressing need for drones that can navigate autonomously in an environment where GPS has been jammed. With 70% of the front lines suffering from jamming, according to Dovbenko, both Russian and Ukrainian drone investment is now focused on autonomous systems.
That’s no small feat. Drone pilots usually rely on video feeds from the drone as well as GPS technology, neither of which is available in a jammed environment. Instead, autonomous drones operate with various types of sensors like LiDAR to navigate, though this can be tricky in fog or other inclement weather. Autonomous drones are a new and rapidly changing technology, still being tested by US-based companies like Shield AI. The evolving war in Ukraine is raising the stakes and the pressure to deploy affordable and reliable autonomous drones.
The transition toward autonomous weapons also raises serious yet largely unanswered questions about how much humans should be taken out of the loop in decision-making. As the war rages on and the need for more capable weaponry rises, Ukraine will likely be the testing ground for if and how the moral line is drawn. But Dovbenko says stopping to find that line during an ongoing war is impossible.
“There is a moral question about how much autonomy you can give to the killing machine,” Dovbenko says. “This question is not being asked right now in Ukraine because it’s more of a matter of survival.”
The US Department of Homeland Security (DHS) plans to collect and analyze photos of the faces of migrant children at the border in a bid to improve facial recognition technology, MIT Technology Review can reveal. This includes children “down to the infant,” according to John Boyd, assistant director of the department’s Office of Biometric Identity Management (OBIM), where a key part of his role is to research and develop future biometric identity services for the government.
As Boyd explained at a conference in June, the key question for OBIM is, “If we pick up someone from Panama at the southern border at age four, say, and then pick them up at age six, are we going to recognize them?”
Facial recognition technology (FRT) has traditionally not been applied to children, largely because training data sets of real children’s faces are few and far between, and consist of either low-quality images drawn from the internet or small sample sizes with little diversity. Such limitations reflect the significant sensitivities regarding privacy and consent when it comes to minors.
In practice, the new DHS plan could effectively solve that problem. According to Syracuse University’s Transactional Records Access Clearinghouse (TRAC), 339,234 children arrived at the US-Mexico border in 2022, the last year for which numbers are currently available. Of those children, 150,000 were unaccompanied—the highest annual number on record. If the face prints of even 1% of those children had been enrolled in OBIM’s craniofacial structural progression program, the resulting data set would dwarf nearly all existing data sets of real children’s faces used for aging research.
It’s unclear to what extent the plan has already been implemented; Boyd tells MIT Technology Review that to the best of his knowledge, the agency has not yet started collecting data under the program, but he adds that as “the senior executive,” he would “have to get with [his] staff to see.” He could only confirm that his office is “funding” it. Despite repeated requests, Boyd did not provide any additional information.
Boyd says OBIM’s plan to collect facial images from children under 14 is possible due to recent “rulemaking” at “some DHS components,” or sub-offices, that have removed age restrictions on the collection of biometric data. US Customs and Border Protection (CBP), the US Transportation Security Administration, and US Immigration and Customs Enforcement declined to comment before publication. US Citizenship and Immigration Services (USCIS) did not respond to multiple requests for comment. OBIM referred MIT Technology Review back to DHS’s main press office.
DHS did not comment on the program prior, but sent an emailed statement following publication: “The Department of Homeland Security uses various forms of technology to execute its mission, including some biometric capabilities. DHS ensures all technologies, regardless of type, are operated under the established authorities and within the scope of the law. We are committed to protecting the privacy, civil rights, and civil liberties of all individuals who may be subject to the technology we use to keep the nation safe and secure.”
Boyd spoke publicly about the plan in June at the Federal Identity Forum and Exposition, an annual identity management conference for federal employees and contractors. But close observers of DHS that we spoke with—including a former official, representatives of two influential lawmakers who have spoken out about the federal government’s use of surveillance technologies, and immigrants’ rights organizations that closely track policies affecting migrants—were unaware of any new policies allowing biometric data collection of children under 14.
That is not to say that all of them are surprised. “That tracks,” says one former CBP official who has visited several migrant processing centers on the US-Mexico border and requested anonymity to speak freely. He says “every center” he visited “had biometric identity collection, and everybody was going through it,” though he was unaware of a specific policy mandating the practice. “I don’t recall them separating out children,” he adds.
“The reports of CBP, as well as DHS more broadly, expanding the use of facial recognition technology to track migrant children is another stride toward a surveillance state and should be a concern to everyone who values privacy,” Justin Krakoff, deputy communications director for Senator Jeff Merkley of Oregon, said in a statement to MIT Technology Review. Merkley has been an outspoken critic of both DHS’s immigration policies and of government use of facial recognition technologies.
Beyond concerns about privacy, transparency, and accountability, some experts also worry about testing and developing new technologies using data from a population that has little recourse to provide—or withhold—consent.
Could consent “actually take into account the vast power differentials that are inherent in the way that this is tested out on people?” asks Petra Molnar, author of The Walls Have Eyes: Surviving Migration in the Age of AI. “And if you arrive at a border … and you are faced with the impossible choice of either: get into a country if you give us your biometrics, or you don’t.”
“That completely vitiates informed consent,” she adds.
This question becomes even more challenging when it comes to children, says Ashley Gorski, a senior staff attorney with the American Civil Liberties Union. DHS “should have to meet an extremely high bar to show that these kids and their legal guardians have meaningfully consented to serve as test subjects,” she says. “There’s a significant intimidation factor, and children aren’t as equipped to consider long-term risks.”
Murky new rules
The Office of Biometric Identity Management, previously known as the US Visitor and Immigrant Status Indicator Technology Program (US-VISIT), was created after 9/11 with the specific mandate of collecting biometric data—initially only fingerprints and photographs—from all non-US citizens who sought to enter the country.
Since then, DHS has begun collecting face prints, iris and retina scans, and even DNA, among other modalities. It is also testing new ways of gathering this data—including through contactless fingerprint collection, which is currently deployed at five sites on the border, as Boyd shared in his conference presentation.
Since 2023, CBP has been using a mobile app, CBP One, for asylum seekers to submit biometric data even before they enter the United States; users are required to take selfies periodically to verify their identity. The app has been riddled with problems, including technical glitches and facial recognition algorithms that are unable to recognize darker-skinned people. This is compounded by the fact that not every asylum seeker has a smartphone.
Then, just after crossing into the United States, migrants must submit to collection of biometric data, including DNA. For a sense of scale, a recent report from Georgetown Law School’s Center on Privacy and Technology found that CBP has added 1.5 million DNA profiles, primarily from migrants crossing the border, to law enforcement databases since it began collecting DNA “from any person in CBP custody subject to fingerprinting” in January 2020. The researchers noted that an overrepresentation of immigrants—the majority of whom are people of color—in a DNA database used by law enforcement could subject them to over-policing and lead to other forms of bias.
Generally, these programs only require information from individuals aged 14 to 79. DHS attempted to change this back in 2020, with proposed rules for USCIS and CBP that would have expanded biometric data collection dramatically, including by age. (USCIS’s proposed rule would have doubled the number of people from whom biometric data would be required, including any US citizen who sponsors an immigrant.) But the USCIS rule was withdrawn in the wake of the Biden administration’s new “priorities to reduce barriers and undue burdens in the immigration system.” Meanwhile, for reasons that remain unclear, the proposed CBP rule was never enacted.
This would make it appear “contradictory” if DHS were now collecting the biometric data of children under 14, says Dinesh McCoy, a staff attorney with Just Futures Law, an immigrant rights group that tracks surveillance technologies.
Neither Boyd nor DHS’s media office would confirm which specific policy changes he was referring to in his presentation, though MIT Technology Review has identified a 2017 memo, issued by then-Secretary of Homeland Security John F. Kelly, that encouraged DHS components to remove “age as a basis for determining when to collect biometrics.”
The DHS’s Office of the Inspector General (OIG) referred to this memo as the “overarching policy for biometrics at DHS” in a September 2023 report, though none of the press offices MIT Technology Review contacted—including the main DHS press office, OIG, and OBIM, among others—would confirm whether this was still the relevant policy; we have not been able to confirm any related policy changes since then.
The OIG audit also found a number of fundamental issues related to DHS’s oversight of biometric data collection and use—including that its 10-year strategic framework for biometrics, covering 2015 to 2025, “did not accurately reflect the current state of biometrics across the Department, such as the use of facial recognition verification and identification.” Nor did it provide clear guidance for the consistent collection and use of biometrics across DHS, including age requirements.
But there is also another potential explanation for the new OBIM program: Boyd says it is being conducted under the auspices of the DHS’s undersecretary of science and technology, the office that leads much of the agency’s research efforts. Because it is for research, rather than to be used “in DHS operations to inform processes or decision making,” many of the standard restrictions for DHS use of face recognition and face capture technologies do not apply, according to a DHS directive.
Do you have any additional information on DHS’s craniofacial structural progression initiative? Please reach out with a non-work email to tips@technologyreview.com or securely on Signal at 626.765.5489.
Some lawyers allege that changing the age limit for data collection via department policy, not by a federal rule, which requires a public comment period, is problematic. McCoy, for instance, says any lack of transparency here amplifies the already “extremely challenging” task of “finding [out] in a systematic way how these technologies are deployed”—even though that is key for accountability.
Advancing the field
At the identity forum and in a subsequent conversation, Boyd explained that this data collection is meant to advance the development of effective FRT algorithms. Boyd leads OBIM’s Future Identity team, whose mission is to “research, review, assess, and develop technology, policy, and human factors that enable rapid, accurate, and secure identity services” and to make OBIM “the preferred provider for identity services within DHS.”
Driven by high-profile cases of missing children, there has long been interest in understanding how children’s faces age. At the same time, there have been technical challenges to doing so, both preceding FRT and with it.
At its core, facial recognition identifies individuals by comparing the geometry of various facial features in an original face print with subsequent images. Based on this comparison, a facial recognition algorithm assigns a percentage likelihood that there is a match.
But as children grow and develop, their bone structure changes significantly, making it difficult for facial recognition algorithms to identify them over time. (These changes tend to be even more pronounced in children under 14. In contrast, as adults age, the changes tend to be in the skin and muscle, and have less variation overall.) More data would help solve this problem, but there is a dearth of high-quality data sets of children’s faces with verifiable ages.
“What we’re trying to do is to get large data sets of known individuals,” Boyd tells MIT Technology Review. That means taking high-quality face prints “under controlled conditions where we know we’ve got the person with the right name [and] the correct birth date”—or, in other words, where they can be certain about the “provenance of the data.”
For example, one data set used for aging research consists of 305 celebrities’ faces as they aged from five to 32. But these photos, scraped from the internet, contain too many other variables—such as differing image qualities, lighting conditions, and distances at which they were taken—to be truly useful. Plus, speaking to the provenance issue that Boyd highlights, their actual ages in each photo can only be estimated.
Another tactic is to use data sets of adult faces that have been synthetically de-aged. Synthetic data is considered more privacy-preserving, but it too has limitations, says Stephanie Schuckers, director of the Center for Identification Technology Research (CITeR). “You can test things with only the generated data,” Schuckers explains, but the question remains: “Would you get similar results to the real data?”
(Hosted at Clarkson University in New York, CITeR brings together a network of academic and government affiliates working on identity technologies. OBIM is a member of the research consortium.)
Schuckers’s team at CITeR has taken another approach: an ongoing longitudinal study of a cohort of 231 elementary and middle school students from the area around Clarkson University. Since 2016, the team has captured biometric data every six months (save for two years of the covid-19 pandemic), including facial images. They have found that the open-source face recognition models they tested can in fact successfully recognize children three to four years after they were initially enrolled.
But the conditions of this study aren’t easily replicable at scale. The study images are taken in a controlled environment, all the participants are volunteers, the researchers sought consent from parents and the subjects themselves, and the research was approved by the university’s Institutional Review Board. Schuckers’s research also promises to protect privacy by requiring other researchers to request access, and by providing facial datasets separately from other data that have been collected.
What’s more, this research still has technical limitations, including that the sample is small, and it is overwhelmingly Caucasian, meaning it might be less accurate when applied to other races.
Schuckers says she was unaware of DHS’s craniofacial structural progression initiative.
Far-reaching implications
Boyd says OBIM takes privacy considerations seriously, and that “we don’t share … data with commercial industries.” Still, OBIM has 144 government partners with which it does share information, and it has been criticized by the Government Accountability Office for poorly documenting who it shares information with, and with what privacy-protecting measures.
Even if the data does stay within the federal government, OBIM’s findings regarding the accuracy of FRT for children over time could neverthelessinfluence how—and when—the rest of the government collects biometric data, as well as whether the broader facial recognition industry may also market its services for children. (Indeed, Boyd says sharing “results,” or the findings of how accurate FRT algorithms are, is different than sharing the data itself.)
That this technology is being tested on people who are offered fewer privacy protections than would be afforded to US citizens is just part of the wider trend of using people from the developing world, whether they are migrants coming to the border or civilians in war zones, to help improve new technologies.
In fact, Boyd previously helped advance the Department of Defense’s biometric systems in Iraq and Afghanistan, where he acknowledged that individuals lacked the privacy protections that would have been granted in many other contexts, despite the incredibly high stakes. Biometric data collected in those war zones—in some areas, from every fighting-age male—was used toidentify and target insurgents, and being misidentified could mean death.
These projects subsequently played a substantial role in influencing the expansion of biometric data collection by the Department of Defense, which now happens globally. And architects of the program, like Boyd, have taken important roles in expanding the use of biometrics at other agencies.
“It’s not an accident” that this testing happens in the context of border zones, says Molnar. Borders are “the perfect laboratory for tech experimentation, because oversight is weak, discretion is baked into the decisions that get made … it allows the state to experiment in ways that it wouldn’t be allowed to in other spaces.”
But, she notes, “just because it happens at the border doesn’t mean that that’s where it’s going to stay.”
Update: This story was updated to include comment from DHS.
Do you have any additional information on DHS’s craniofacial structural progression initiative? Please reach out with a non-work email to tips@technologyreview.com or securely on Signal at 626.765.5489.
This story is from The Algorithm, our weekly newsletter on AI. To get it in your inbox first, sign up here.
When the generative AI boom started with ChatGPT in late 2022, we were sold a vision of superintelligent AI tools that know everything, can replace the boring bits of work, and supercharge productivity and economic gains.
Two years on, most of those productivity gains haven’t materialized. And we’ve seen something peculiar and slightly unexpected happen: People have started forming relationships with AI systems. We talk to them, say please and thank you, and have started to invite AIs into our lives as friends, lovers, mentors, therapists, and teachers.
We’re seeing a giant, real-world experiment unfold, and it’s still uncertain what impact these AI companions will have either on us individually or on society as a whole, argue Robert Mahari, a joint JD-PhD candidate at the MIT Media Lab and Harvard Law School, and Pat Pataranutaporn, a researcher at the MIT Media Lab. They say we need to prepare for “addictive intelligence”, or AI companions that have dark patterns built into them to get us hooked. You can read their piece here. They look at how smart regulation can help us prevent some of the risks associated with AI chatbots that get deep inside our heads.
The idea that we’ll form bonds with AI companions is no longer just hypothetical. Chatbots with even more emotive voices, such as OpenAI’s GPT-4o, are likely to reel us in even deeper. During safety testing, OpenAI observed that users would use language that indicated they had formed connections with AI models, such as “This is our last day together.” The company itself admits that emotional reliance is one risk that might be heightened by its new voice-enabled chatbot.
There’s already evidence that we’re connecting on a deeper level with AI even when it’s just confined to text exchanges. Mahari was part of a group of researchers that analyzed a million ChatGPT interaction logs and found that the second most popular use of AI was sexual role-playing. Aside from that, the overwhelmingly most popular use case for the chatbot was creative composition. People also liked to use it for brainstorming and planning, asking for explanations and general information about stuff.
These sorts of creative and fun tasks are excellent ways to use AI chatbots. AI language models work by predicting the next likely word in a sentence. They are confident liars and often present falsehoods as facts, make stuff up, or hallucinate. This matters less when making stuff up is kind of the entire point. In June, my colleague Rhiannon Williams wrote about how comedians found AI language models to be useful for generating a first “vomit draft” of their material; they then add their own human ingenuity to make it funny.
But these use cases aren’t necessarily productive in the financial sense. I’m pretty sure smutbots weren’t what investors had in mind when they poured billions of dollars into AI companies, and, combined with the fact we still don’t have a killer app for AI,it’s no wonder that Wall Street is feeling a lot less bullish about it recently.
The use cases that would be “productive,” and have thus been the most hyped, have seen less success in AI adoption. Hallucination starts to become a problem in some of these use cases, such as code generation, news and online searches, where it matters a lot to get things right. Some of the most embarrassing failures of chatbots have happened when people have started trusting AI chatbots too much, or considered them sources of factual information. Earlier this year, for example, Google’s AI overview feature, which summarizes online search results, suggested that people eat rocks and add glue on pizza.
And that’s the problem with AI hype. It sets our expectations way too high, and leaves us disappointed and disillusioned when the quite literally incredible promises don’t happen. It also tricks us into thinking AI is a technology that is even mature enough to bring about instant changes. In reality, it might be years until we see its true benefit.
Now read the rest of The Algorithm
Deeper Learning
AI “godfather” Yoshua Bengio has joined a UK project to prevent AI catastrophes
Yoshua Bengio, a Turing Award winner who is considered one of the godfathers of modern AI, is throwing his weight behind a project funded by the UK government to embed safety mechanisms into AI systems. The project, called Safeguarded AI, aims to build an AI system that can check whether other AI systems deployed in critical areas are safe. Bengio is joining the program as scientific director and will provide critical input and advice.
What are they trying to do: Safeguarded AI’s goal is to build AI systems that can offer quantitative guarantees, such as risk scores, about their effect on the real world. The project aims to build AI safety mechanisms by combining scientific world models, which are essentially simulations of the world, with mathematical proofs. These proofs would include explanations of the AI’s work, and humans would be tasked with verifying whether the AI model’s safety checks are correct. Read more from me here.
Bits and Bytes
Google DeepMind trained a robot to beat humans at table tennis
Researchers managed to get a robot wielding a 3D-printed paddle to win 13 of 29 games against human opponents of varying abilities in full games of competitive table tennis. The research represents a small step toward creating robots that can perform useful tasks skillfully and safely in real environments like homes and warehouses, which is a long-standing goal of the robotics community. (MIT Technology Review)
Are we in an AI bubble? Here’s why it’s complex.
There’s been a lot of debate recently, and even some alarm, about whether AI is ever going to live up to its potential, especially thanks to tech stocks’ recent nosedive. This nuanced piece explains why although the sector faces significant challenges, it’s far too soon to write off AI’s transformative potential. (Platformer)
How Microsoft spread its bets beyond OpenAI
Microsoft and OpenAI have one of the most successful partnerships in AI. But following OpenAI’s boardroom drama last year, the tech giant and its CEO, Satya Nadella, have been working on a strategy that will make Microsoft more independent of Sam Altman’s startup. Microsoft has diversified its investments and partnerships in generative AI, built its own smaller, cheaper models, and hired aggressively to develop its consumer AI efforts. (Financial Times)
Humane’s daily returns are outpacing sales
Oof. The extremely hyped AI pin, which was billed as a wearable AI assistant, seems to have flopped. Between May and August, more Humane AI Pins were returned than purchased. Infuriatingly, the company has no way to reuse the returned pins, so they become e-waste.(The Verge)
Do you fancy your chances of beating a robot at a game of table tennis? Google DeepMind has trained a robot to play the game at the equivalent of amateur-level competitive performance, the company has announced. It claims it’s the first time a robot has been taught to play a sport with humans at a human level.
Researchers managed to get a robotic arm wielding a 3D-printed paddle to win 13 of 29 games against human opponents of varying abilities in full games of competitive table tennis. The research was published in an Arxiv paper.
The system is far from perfect. Although the table tennis bot was able to beat all beginner-level human opponents it faced and 55% of those playing at amateur level, it lost all the games against advanced players. Still, it’s an impressive advance.
“Even a few months back, we projected that realistically the robot may not be able to win against people it had not played before. The system certainly exceeded our expectations,” says Pannag Sanketi, a senior staff software engineer at Google DeepMind who led the project. “The way the robot outmaneuvered even strong opponents was mind blowing.”
And the research is not just all fun and games. In fact, it represents a step towards creating robots that can perform useful tasks skillfully and safely in real environments like homes and warehouses, which is a long-standing goal of the robotics community. Google DeepMind’s approach to training machines is applicable to many other areas of the field, says Lerrel Pinto, a computer science researcher at New York University who did not work on the project.
“I’m a big fan of seeing robot systems actually working with and around real humans, and this is a fantastic example of this,” he says. “It may not be a strong player, but the raw ingredients are there to keep improving and eventually get there.”
To become a proficient table tennis player, humans require excellent hand-eye coordination, the ability to move rapidly and make quick decisions reacting to their opponent—all of which are significant challenges for robots. Google DeepMind’s researchers used a two-part approach to train the system to mimic these abilities: they used computer simulations to train the system to master its hitting skills; then fine tuned it using real-world data, which allows it to improve over time.
The researchers compiled a dataset of table tennis ball states, including data on position, spin, and speed. The system drew from this library in a simulated environment designed to accurately reflect the physics of table tennis matches to learn skills such as returning a serve, hitting a forehand topspin, or backhand shot. As the robot’s limitations meant it could not serve the ball, the real-world games were modified to accommodate this.
During its matches against humans, the robot collects data on its performance to help refine its skills. It tracks the ball’s position using data captured by a pair of cameras, and follows its human opponent’s playing style through a motion capture system that uses LEDs on its opponent’s paddle. The ball data is fed back into the simulation for training, creating a continuous feedback loop.
This feedback allows the robot to test out new skills to try and beat its opponent—meaning it can adjust its tactics and behavior just like a human would. This means it becomes progressively better both throughout a given match, and over time the more games it plays.
The system struggled to hit the ball when it was hit either very fast, beyond its field of vision (more than six feet above the table), or very low, because of a protocol that instructs it to avoid collisions that could damage its paddle. Spinning balls proved a challenge because it lacked the capacity to directly measure spin—a limitation that advanced players were quick to take advantage of.
Training a robot for all eventualities in a simulated environment is a real challenge, says Chris Walti, founder of robotics company Mytra and previously head of Tesla’s robotics team, who was not involved in the project.
“It’s very, very difficult to actually simulate the real world because there’s so many variables, like a gust of wind, or even dust [on the table]” he says. “Unless you have very realistic simulations, a robot’s performance is going to be capped.”
Google DeepMind believes these limitations could be addressed in a number of ways, including by developing predictive AI models designed to anticipate the ball’s trajectory, and introducing better collision-detection algorithms.
Crucially, the human players enjoyed their matches against the robotic arm. Even the advanced competitors who were able to beat it said they’d found the experience fun and engaging, and said they felt it had potential as a dynamic practice partner to help them hone their skills.
“I would definitely love to have it as a training partner, someone to play some matches from time to time,” one of the study participants said.
Yoshua Bengio, a Turing Award winner who is considered one of the “godfathers” of modern AI, is throwing his weight behind a project funded by the UK government to embed safety mechanisms into AI systems.
The project, called Safeguarded AI, aims to build an AI system that can check whether other AI systems deployed in critical areas are safe. Bengio is joining the program as scientific director and will provide critical input and scientific advice. The project, which will receive £59 million over the next four years, is being funded by the UK’s Advanced Research and Invention Agency (ARIA), which was launched in January last year to invest in potentially transformational scientific research.
Safeguarded AI’s goal is to build AI systems that can offer quantitative guarantees, such as a risk score, about their effect on the real world, says David “davidad” Dalrymple, the program director for Safeguarded AI at ARIA. The idea is to supplement human testing with mathematical analysis of new systems’ potential for harm.
The project aims to build AI safety mechanisms by combining scientific world models, which are essentially simulations of the world, with mathematical proofs. These proofs would include explanations of the AI’s work, and humans would be tasked with verifying whether the AI model’s safety checks are correct.
Bengio says he wants to help ensure that future AI systems cannot cause serious harm.
“We’re currently racing toward a fog behind which might be a precipice,” he says. “We don’t know how far the precipice is, or if there even is one, so it might be years, decades, and we don’t know how serious it could be … We need to build up the tools to clear that fog and make sure we don’t cross into a precipice if there is one.”
Science and technology companies don’t have a way to give mathematical guarantees that AI systems are going to behave as programmed, he adds. This unreliability, he says, could lead to catastrophic outcomes.
Dalrymple and Bengio argue that current techniques to mitigate the risk of advanced AI systems—such as red-teaming, where people probe AI systems for flaws—have serious limitations and can’t be relied on to ensure that critical systems don’t go off-piste.
Instead, they hope the program will provide new ways to secure AI systems that rely less on human efforts and more on mathematical certainty. The vision is to build a “gatekeeper” AI, which is tasked with understanding and reducing the safety risks of other AI agents. This gatekeeper would ensure that AI agents functioning in high-stakes sectors, such as transport or energy systems, operate as we want them to. The idea is to collaborate with companies early on to understand how AI safety mechanisms could be useful for different sectors, says Dalrymple.
The complexity of advanced systems means we have no choice but to use AI to safeguard AI, argues Bengio. “That’s the only way, because at some point these AIs are just too complicated. Even the ones that we have now, we can’t really break down their answers into human, understandable sequences of reasoning steps,” he says.
The next step—actually building models that can check other AI systems—is also where Safeguarded AI and ARIA hope to change the status quo of the AI industry.
ARIA is also offering funding to people or organizations in high-risk sectors such as transport, telecommunications, supply chains, and medical research to help them build applications that might benefit from AI safety mechanisms. ARIA is offering applicants a total of £5.4 million in the first year, and another £8.2 million in another year. The deadline for applications is October 2.
The agency is also casting a wide net for people who might be interested in building Safeguarded AI’s safety mechanism through a nonprofit organization. ARIA is eyeing up to £18 million to set this organization up and will be accepting funding applications early next year.
The program is looking for proposals to start a nonprofit with a diverse board that encompasses lots of different sectors in order to do this work in a reliable, trustworthy way, Dalrymple says. This is similar to what OpenAI was initially set up to do before changing its strategy to be more product- and profit-oriented.
The organization’s board will not just be responsible for holding the CEO accountable; it will even weigh in on decisions about whether to undertake certain research projects, and whether to release particular papers and APIs, he adds.
The Safeguarded AI project is part of the UK’s mission to position itself as a pioneer in AI safety. In November 2023, the country hosted the very first AI Safety Summit, which gathered world leaders and technologists to discuss how to build the technology in a safe way.
While the funding program has a preference for UK-based applicants, ARIA is looking for global talent that might be interested in coming to the UK, says Dalrymple. ARIA also has an intellectual-property mechanism for funding for-profit companies abroad, which allows royalties to return back to the country.
Bengio says he was drawn to the project to promote international collaboration on AI safety. He chairs the International Scientific Report on the safety of advanced AI, which involves 30 countries as well as the EU and UN. A vocal advocate for AI safety, he has been part of an influential lobby warning that superintelligent AI poses an existential risk.
“We need to bring the discussion of how we are going to address the risks of AI to a global, larger set of actors,” says Bengio. “This program is bringing us closer to this.”
