Why Big Tech’s watermarking plans are some welcome good news

This story originally appeared in The Algorithm, our weekly newsletter on AI. To get stories like this in your inbox first, sign up here.

This week I am happy to bring you some encouraging news from the world of AI. Following the depressing Taylor Swift deepfake porn scandal and the proliferation of political deepfakes, such as AI-generated robocalls of President Biden asking voters to stay home, tech companies are stepping up and putting into place measures to better detect AI-generated content. 

On February 6, Meta said it was going to label AI-generated images on Facebook, Instagram, and Threads. When someone uses Meta’s AI tools to create images, the company will add visible markers to the image, as well as invisible watermarks and metadata in the image file. The company says its standards are in line with best practices laid out by the Partnership on AI, an AI research nonprofit.

Big Tech is also throwing its weight behind a promising technical standard that could add a “nutrition label” to images, video, and audio. Called C2PA, it’s an open-source internet protocol that relies on cryptography to encode details about the origins of a piece of content, or what technologists refer to as “provenance” information. The developers of C2PA often compare the protocol to a nutrition label, but one that says where content came from and who—or what—created it. Read more about it here. 

On February 8, Google announced it is joining other tech giants such as Microsoft and Adobe in the steering committee of C2PA and will include its watermark SynthID in all AI-generated images in its new Gemini tools. Meta says it is also participating in C2PA. Having an industry-wide standard makes it easier for companies to detect AI-generated content, no matter which system it was created with.

OpenAI too announced new content provenance measures last week. It says it will add watermarks to the metadata of images generated with ChatGPT and DALL-E 3, its image-making AI. OpenAI says it will now include a visible label in images to signal they have been created with AI. 

These methods are a promising start, but they’re not foolproof. Watermarks in metadata are easy to circumvent by taking a screenshot of images and just using that, while visual labels can be cropped or edited out. There is perhaps more hope for invisible watermarks like Google’s SynthID, which subtly changes the pixels of an image so that computer programs can detect the watermark but the human eye cannot. These are harder to tamper with. What’s more, there aren’t reliable ways to label and detect AI-generated video, audio, or even text. 

But there is still value in creating these provenance tools. As Henry Ajder, a generative-AI expert, told me a couple of weeks ago when I interviewed him about how to prevent deepfake porn, the point is to create a “perverse customer journey.” In other words, add barriers and friction to the deepfake pipeline in order to slow down the creation and sharing of harmful content as much as possible. A determined person will likely still be able to override these protections, but every little bit helps. 

There are also many nontechnical fixes tech companies could introduce to prevent problems such as deepfake porn. Major cloud service providers and app stores, such as Google, Amazon, Microsoft, and Apple could move to ban services that can be used to create nonconsensual deepfake nudes. And watermarks should be included in all AI-generated content across the board, even by smaller startups developing the technology.

What gives me hope is that alongside these voluntary measures we’re starting to see binding regulations, such as the EU’s AI Act and the Digital Services Act, which require tech companies to disclose AI-generated content and take down harmful content faster. There’s also renewed interest among US lawmakers in passing some binding rules on deepfakes. And following AI-generated robocalls of President Biden telling voters not to vote, the US Federal Communications Commission announced last week that it was banning the use of AI in these calls. 

In general I’m pretty skeptical about voluntary guidelines and rules, because there’s no real accountability mechanism and companies can choose to change these rules whenever they want. The tech sector has a really bad track record for regulating itself. In the cutthroat, growth-driven tech world, things like responsible AI are often the first to face cuts. 

But despite that, these announcements are extremely welcome. They’re also much better than the status quo, which is next to nothing. 

Deeper Learning

Google’s Gemini is now in everything. Here’s how you can try it out.

In the biggest mass-market AI launch yet, Google is rolling out Gemini, its family of large language models, across almost all its products, from Android to the iOS Google app to Gmail to Docs and more. You can now get your hands on Gemini Ultra, the most powerful version of the model, for the first time. 

Bard is dead; long live Gemini: Google is also sunsetting Bard, its ChatGPT rival. Bard, which has been powered by a version of Gemini since December, will now be known as Gemini too. By baking Gemini into its ubiquitous tools, Google is hoping to make up lost ground and even overtake its rival OpenAI. Read more from Will Douglas Heaven. 

Bits and Bytes

A chatbot helped more people access mental-health services
An AI chatbot from a startup called Limbic helped increase the number of patients referred for mental-health services through England’s National Health Service (particularly among members of underrepresented groups, who are less likely to seek help), new research has found. (MIT Technology Review) 

This robot can tidy a room without any help
A new system called OK-Robot could train robots to pick up and move objects in settings they haven’t encountered before. It’s an approach that might be able to plug the gap between rapidly improving AI models and actual robot capabilities, because it doesn’t require any additional costly, complex training. (MIT Technology Review) 

Inside OpenAI’s plan to make AI more “democratic”
This feature looks at how computer scientists at OpenAI are trying to address the technical problem of how to align their AIs to human values. But a bigger question remains unanswered: Exactly whose values should AI reflect? And who should get to decide? 
(Time) 

OpenAI’s Sam Altman wants trillions to build chips for AI
The CEO has often complained that the company does not have enough computing power to train and run its powerful AI models. Altman is reportedly talking with investors in the United Arab Emirates government to raise up to $7 trillion to boost the world’s chip-building capacity. (The Wall Street Journal) 

A new app to “dignify” women
Ugh. In contrast to apps that sexualize images of women, some 4Chan users are using generative AI to add clothes, erase their tattoos and piercings, and make them look more modest. How about … we just leave women alone. (404 Media) 

Dear Taylor Swift, we’re sorry about those explicit deepfakes

This story originally appeared in The Algorithm, our weekly newsletter on AI. To get stories like this in your inbox first, sign up here.

Hi, Taylor.

I can only imagine how you must be feeling after sexually explicit deepfake videos of you went viral on X. Disgusted. Distressed, perhaps. Humiliated, even. 

I’m really sorry this is happening to you. Nobody deserves to have their image exploited like that. But if you aren’t already, I’m asking you to be furious. 

Furious that this is happening to you and so many other women and marginalized people around the world. Furious that our current laws are woefully inept at protecting us from violations like this. Furious that men (because let’s face it, it’s mostly men doing this) can violate us in such an intimate way and walk away unscathed and unidentified. Furious that the companies that enable this material to be created and shared widely face no consequences either, and can profit off such a horrendous use of their technology. 

Deepfake porn has been around for years, but its latest incarnation is its worst one yet. Generative AI has made it ridiculously easy and cheap to create realistic deepfakes. And nearly all deepfakes are made for porn. Only one image plucked off social media is enough to generate something passable. Anyone who has ever posted or had a photo published of them online is a sitting duck. 

First, the bad news. At the moment, we have no good ways to fight this. I just published a story looking at three ways we can combat nonconsensual deepfake porn, which include watermarks and data-poisoning tools. But the reality is that there is no neat technical fix for this problem. The fixes we do have are still experimental and haven’t been adopted widely by the tech sector, which limits their power. 

The tech sector has thus far been unwilling or unmotivated to make changes that would prevent such material from being created with their tools or shared on their platforms. That is why we need regulation. 

People with power, like yourself, can fight with money and lawyers. But low-income women, women of color, women fleeing abusive partners, women journalists, and even children are all seeing their likeness stolen and pornified, with no way to seek justice or support. Any one of your fans could be hurt by this development. 

The good news is that the fact that this happened to you means politicians in the US are listening. You have a rare opportunity, and momentum, to push through real, actionable change. 

I know you fight for what is right and aren’t afraid to speak up when you see injustice. There will be intense lobbying against any rules that would affect tech companies. But you have a platform and the power to convince lawmakers across the board that rules to combat these sorts of deepfakes are a necessity. Tech companies and politicians need to know that the days of dithering are over. The people creating these deepfakes need to be held accountable. 

You once caused an actual earthquake. Winning the fight against nonconsensual deepfakes would have an even more earth-shaking impact.

What to expect from the coming year in AI

This story originally appeared in The Algorithm, our weekly newsletter on AI. To get stories like this in your inbox first, sign up here.

Happy new year! I hope you had a relaxing break. I spent it up in the Arctic Circle skiing, going to the sauna, and playing card games with my family by the fire. 10/10 would recommend. 

I also had plenty of time to reflect on the past year. There are so many more of you reading The Algorithm than when we first started this newsletter, and for that I am eternally grateful. Thank you for joining me on this wild AI ride. Here’s a cheerleading pug as a little present! 

So what can we expect in 2024? All signs point to there being immense pressure on AI companies to show that generative AI can make money and that Silicon Valley can produce the “killer app” for AI. Big Tech, generative AI’s biggest cheerleaders, is betting big on customized chatbots, which will allow anyone to become a generative-AI app engineer, with no coding skills needed. Things are already moving fast: OpenAI is reportedly set to launch its GPT app store as early as this week. We’ll also see cool new developments in AI-generated video, a whole lot more AI-powered election misinformation, and robots that multitask. My colleague Will Douglas Heaven and I shared our four predictions for AI in 2024 last week—read the full story here. 

This year will also be another huge year for AI regulation around the world. In 2023 the first sweeping AI law was agreed upon in the European Union, Senate hearings and executive orders unfolded in the US, and China introduced specific rules for things like recommender algorithms. If last year lawmakers agreed on a vision, 2024 will be the year policies start to morph into concrete action. Together with my colleagues Tate Ryan-Mosley and Zeyi Yang, I’ve written a piece that walks you through what to expect in AI regulation in the coming year. Read it here. 

But even as the generative-AI revolution unfolds at a breakneck pace, there are still some big unresolved questions that urgently need answering, writes Will. He highlights problems around bias, copyright, and the high cost of building AI, among other issues. Read more here. 

My addition to the list would be generative models’ huge security vulnerabilities. Large language models, the AI tech that powers applications such as ChatGPT, are really easy to hack. For example, AI assistants or chatbots that can browse the internet are very susceptible to an attack called indirect prompt injection, which allows outsiders to control the bot by sneaking in invisible prompts that make the bots behave in the way the attacker wants them to. This could make them powerful tools for phishing and scamming, as I wrote back in April. Researchers have also successfully managed to poison AI data sets with corrupt data, which can break AI models for good. (Of course, it’s not always a malicious actor trying to do this. Using a new tool called Nightshade, artists can add invisible changes to the pixels in their art before they upload it online so that if it’s scraped into an AI training set, it can cause the resulting model to break in chaotic and unpredictable ways.) 

Despite these vulnerabilities, tech companies are in a race to roll out AI-powered products, such as assistants or chatbots that can browse the web. It’s fairly easy for hackers to manipulate AI systems by poisoning them with dodgy data, so it’s only a matter of time until we see an AI system being hacked in this way. That’s why I was pleased to see NIST, the US technology standards agency, raise awareness about these problems and offer mitigation techniques in a new guidance published at the end of last week. Unfortunately, there is currently no reliable fix for these security problems, and much more research is needed to understand them better.

AI’s role in our societies and lives will only grow bigger as tech companies integrate it into the software we all depend on daily, despite these flaws. As regulation catches up, keeping an open, critical mind when it comes to AI is more important than ever.

Deeper Learning

How machine learning might unlock earthquake prediction

Our current earthquake early warning systems give people crucial moments to prepare for the worst, but they have their limitations. There are false positives and false negatives. What’s more, they react only to an earthquake that has already begun—we can’t predict an earthquake the way we can forecast the weather. If we could, it would  let us do a lot more to manage risk, from shutting down the power grid to evacuating residents.

Enter AI: Some scientists are hoping to tease out hints of earthquakes from data—signals in seismic noise, animal behavior, and electromagnetism—with the ultimate goal of issuing warnings before the shaking begins. Artificial intelligence and other techniques are giving scientists hope in the quest to forecast quakes in time to help people find safety. Read more from Allie Hutchison. 

Bits and Bytes

AI for everything is one of MIT Technology Review’s 10 breakthrough technologies
We couldn’t put together a list of the tech that’s most likely to have an impact on the world without mentioning AI. Last year tools like ChatGPT reached mass adoption in record time, and reset the course of an entire industry. We haven’t even begun to make sense of it all, let alone reckon with its impact. (MIT Technology Review) 

Isomorphic Labs has announced it’s working with two pharma companies
Google DeepMind’s drug discovery spinoff has two new “strategic collaborations” with major pharma companies Eli Lilly and Novartis. The deals are worth nearly $3 billion to Isomorphic Labs and offer the company funding to help discover potential new treatments using AI, the company said. 

We learned more about OpenAI’s board saga
Helen Toner, an AI researcher at Georgetown’s Center for Security and Emerging Technology and a former member of OpenAI’s board, talks to the Wall Street Journal about why she agreed to fire CEO Sam Altman. Without going into details, she underscores that it wasn’t safety that led to the fallout, but a lack of trust. Meanwhile, Microsoft executive Dee Templeton has joined OpenAI’s board as a nonvoting observer. 

A new kind of AI copy can fully replicate famous people. The law is powerless.
Famous people are finding convincing AI replicas in their likeness. A new draft bill in the US called the No Fakes Act would require the creators of these AI replicas to license their use from the original human. But this bill would not apply in cases where the replicated human or the AI system is outside the US. It’s another example of just how incredibly difficult AI regulation is. (Politico)

The largest AI image data set was taken offline after researchers found it is full of child sexual abuse material
Stanford researchers made the explosive discovery about the open-source LAION data set, which powers models such as Stable Diffusion. We knew indiscriminate scraping of the internet meant AI data sets contain tons of biased and harmful content, but this revelation is shocking. We desperately need better data practices in AI! (404 Media) 

Four trends that changed AI in 2023

This story originally appeared in The Algorithm, our weekly newsletter on AI. To get stories like this in your inbox first, sign up here.

This has been one of the craziest years in AI in a long time: endless product launches, boardroom coups, intense policy debates about AI doom, and a race to find the next big thing. But we’ve also seen concrete tools and policies aimed at getting the AI sector to behave more responsibly and hold powerful players accountable. That gives me a lot of hope for the future of AI. 

Here’s what 2023 taught me: 

1. Generative AI left the lab with a vengeance, but it’s not clear where it will go next

The year started with Big Tech going all in on generative AI. The runaway success of OpenAI’s ChatGPT prompted every major tech company to release its own version. This year might go down in history as the year we saw the most AI launches: Meta’s LLaMA 2, Google’s Bard chatbot and Gemini, Baidu’s Ernie Bot, OpenAI’s GPT-4, and a handful of other models, including one from a French open-source challenger, Mistral. 

But despite the initial hype, we haven’t seen any AI applications become an overnight success. Microsoft and Google pitched powerful AI-powered search, but it turned out to be more of a dud than a killer app. The fundamental flaws in language models, such as the fact that they frequently make stuff up, led to some embarrassing (and, let’s be honest, hilarious) gaffes. Microsoft’s Bing would frequently reply to people’s questions with conspiracy theories, and suggested that a New York Times reporter leave his wife. Google’s Bard generated factually incorrect answers for its marketing campaign, which wiped $100 billion off the company’s share price.

There is now a frenetic hunt for a popular AI product that everyone will want to adopt. Both OpenAI and Google are experimenting with allowing companies and developers to create customized AI chatbots and letting people build their own applications using AI—no coding skills needed. Perhaps generative AI will end up embedded in boring but useful tools to help us boost our productivity at work. It might take the form of AI assistants—maybe with voice capabilities—and coding support. Next year will be crucial in determining the real value of generative AI.

2. We learned a lot about how language models actually work, but we still know very little

Even though tech companies are rolling out large language models into products at a frenetic pace, there is still a lot we don’t know about how they work. They make stuff up and have severe gender and ethnic biases. This year we also found out that different language models generate texts with different political biases, and that they make great tools for hacking people’s private information. Text-to-image models can be prompted to spit out copyrighted images and pictures of real people, and they can easily be tricked into generating disturbing images. It’s been great to see so much research into the flaws of these models, because this could take us a step closer to understanding why they behave the way they do, and ultimately fix them.

Generative models can be very unpredictable, and this year there were lots of attempts to try to make them behave as their creators want them to. OpenAI shared that it uses a technique called reinforcement learning from human feedback, which uses feedback from users to help guide ChatGPT to more desirable answers. A study from the AI lab Anthropic showed how simple natural-language instructions can steer large language models to make their results less toxic. But sadly, a lot of these attempts end up being quick fixes rather than permanent ones. Then there are misguided approaches like banning seemingly innocuous words such as “placenta” from image-generating AI systems to avoid producing gore. Tech companies come up with workarounds like these because they don’t know why models generate the content they do. 

We also got a better sense of AI’s true carbon footprint. Generating an image using a powerful AI model takes as much energy as fully charging your smartphone, researchers at the AI startup Hugging Face and Carnegie Mellon University found. Until now, the exact amount of energy generative AI uses has been a missing piece of the puzzle. More research into this could help us shift the way we use AI to be more sustainable. 

3. AI doomerism went mainstream

Chatter about the possibility that AI poses an existential risk to humans became familiar this year. Hundreds of scientists, business leaders, and policymakers have spoken up, from deep-learning pioneers Geoffrey Hinton and Yoshua Bengio to the CEOs of top AI firms, such as Sam Altman and Demis Hassabis, to the California congressman Ted Lieu and the former president of Estonia Kersti Kaljulaid.

Existential risk has become one of the biggest memes in AI. The hypothesis is that one day we will build an AI that is far smarter than humans, and this could lead to grave consequences. It’s an ideology championed by many in Silicon Valley, including Ilya Sutskever, OpenAI’s chief scientist, who played a pivotal role in ousting OpenAI CEO Sam Altman (and then reinstating him a few days later). 

But not everyone agrees with this idea. Meta’s AI leaders Yann LeCun and Joelle Pineau have said that these fears are “ridiculous” and the conversation about AI risks has become “unhinged.” Many other power players in AI, such as researcher Joy Buolamwini, say that focusing on hypothetical risks distracts from the very real harms AI is causing today. 

Nevertheless, the increased attention on the technology’s potential to cause extreme harm has prompted many important conversations about AI policy and animated lawmakers all over the world to take action. 

4. The days of the AI Wild West are over

Thanks to ChatGPT, everyone from the US Senate to the G7 was talking about AI policy and regulation this year. In early December, European lawmakers wrapped up a busy policy year when they agreed on the AI Act, which will introduce binding rules and standards on how to develop the riskiest AI more responsibly. It will also ban certain “unacceptable” applications of AI, such as police use of facial recognition in public places. 

The White House, meanwhile, introduced an executive order on AI, plus voluntary commitments from leading AI companies. Its efforts aimed to bring more transparency and standards for AI and gave a lot of freedom to agencies to adapt AI rules to fit their sectors. 

One concrete policy proposal that got a lot of attention was watermarks—invisible signals in text and images that can be detected by computers, in order to flag AI-generated content. These could be used to track plagiarism or help fight disinformation, and this year we saw research that succeeded in applying them to AI-generated text and images.

It wasn’t just lawmakers that were busy, but lawyers too. We saw a record number of  lawsuits, as artists and writers argued that AI companies had scraped their intellectual property without their consent and with no compensation. In an exciting counter-offensive, researchers at the University of Chicago developed Nightshade, a new data-poisoning tool that lets artists fight back against generative AI by messing up training data in ways that could cause serious damage to image-generating AI models. There is a resistance brewing, and I expect more grassroots efforts to shift tech’s power balance next year. 

Deeper Learning

Now we know what OpenAI’s superalignment team has been up to

OpenAI has announced the first results from its superalignment team, its in-house initiative dedicated to preventing a superintelligence—a hypothetical future AI that can outsmart humans—from going rogue. The team is led by chief scientist Ilya Sutskever, who was part of the group that just last month fired OpenAI’s CEO, Sam Altman, only to reinstate him a few days later.

Business as usual: Unlike many of the company’s announcements, this heralds no big breakthrough. In a low-key research paper, the team describes a technique that lets a less powerful large language model supervise a more powerful one—and suggests that this might be a small step toward figuring out how humans might supervise superhuman machines. Read more from Will Douglas Heaven. 

Bits and Bytes

Google DeepMind used a large language model to solve an unsolvable math problem
In a paper published in Nature, the company says it is the first time a large language model has been used to discover a solution to a long-standing scientific puzzle—producing verifiable and valuable new information that did not previously exist. (MIT Technology Review)

This new system can teach a robot a simple household task within 20 minutes
A new open-source system, called Dobb-E, was trained using data collected from real homes. It can help to teach a robot how to open an air fryer, close a door, or straighten a cushion, among other tasks. It could also help the field of robotics overcome one of its biggest challenges: a lack of training data.  (MIT Technology Review)

ChatGPT is turning the internet into plumbing
German media giant Axel Springer, which owns Politico and Business Insider, announced a partnership with OpenAI, in which the tech company will be able to use its news articles as training data and the news organizations will be able to use ChatGPT to do summaries of news. This column has a clever point: tech companies are increasingly becoming gatekeepers for online content, and journalism is just “plumbing for a digital faucet.” (The Atlantic)

Meet the former French official pushing for looser AI rules after joining startup Mistral
A profile of Mistral AI cofounder Cédric O, who used to be France’s digital minister. Before joining France’s AI unicorn, he was a vocal proponent of strict laws for tech, but he lobbied hard against rules in the AI Act that would have restricted Mistral’s models. He was successful: the company’s models don’t meet the computing threshold set by the law, and its open-source models are also exempt from transparency obligations. (Bloomberg) 

Five things you need to know about the EU’s new AI Act

This story is from The Algorithm, our weekly newsletter on AI. To get stories like this in your inbox first, sign up here.

It’s done. It’s over. Two and a half years after it was first introduced—after months of lobbying and political arm-wrestling, plus grueling final negotiations that took nearly 40 hours—EU lawmakers have reached a deal over the AI Act. It will be the world’s first sweeping AI law. 

The AI Act was conceived as a landmark bill that would mitigate harm in areas where using AI poses the biggest risk to fundamental rights, such as health care, education, border surveillance, and public services, as well as banning uses that pose an “unacceptable risk.” 

“High risk” AI systems will have to adhere to strict rules that require risk-mitigation systems, high-quality data sets, better documentation, and human oversight, for example. The vast majority of AI uses, such as recommender systems and spam filters, will get a free pass. 

The AI Act is a major deal in that it will introduce important rules and enforcement mechanisms to a hugely influential sector that is currently a Wild West. 

Here are MIT Technology Review’s key takeaways: 

1. The AI Act ushers in important, binding rules on transparency and ethics

Tech companies love to talk about how committed they are to AI ethics. But when it comes to concrete measures, the conversation dries up. And anyway, actions speak louder than words. Responsible AI teams are often the first to see cuts during layoffs, and in truth, tech companies can decide to change their AI ethics policies at any time. OpenAI, for example, started off as an “open” AI research lab before closing up public access to its research to protect its competitive advantage, just like every other AI startup. 

The AI Act will change that. The regulation imposes legally binding rules requiring tech companies to notify people when they are interacting with a chatbot or with biometric categorization or emotion recognition systems. It’ll also require them to label deepfakes and AI-generated content, and design systems in such a way that AI-generated media can be detected. This is a step beyond the voluntary commitments that leading AI companies made to the White House to simply develop AI provenance tools, such as watermarking. 

The bill will also require all organizations that offer essential services, such as insurance and banking, to conduct an impact assessment on how using AI systems will affect people’s fundamental rights. 

2. AI companies still have a lot of wiggle room

When the AI Act was first introduced, in 2021, people were still talking about the metaverse. (Can you imagine!) 

Fast-forward to now, and in a post-ChatGPT world, lawmakers felt they had to take so-called foundation models—powerful AI models that can be used for many different purposes—into account in the regulation. This sparked intense debate over what sorts of models should be regulated, and whether regulation would kill innovation. 

The AI Act will require foundation models and AI systems built on top of them to draw up better documentation, comply with EU copyright law, and share more information about what data the model was trained on. For the most powerful models, there are extra requirements. Tech companies will have to share how secure and energy efficient their AI models are, for example. 

But here’s the catch: The compromise lawmakers found was to apply a stricter set of rules only the most powerful AI models, as categorized by the computing power needed to train them. And it will be up to companies to assess whether they fall under stricter rules. 

A European Commission official would not confirm whether the current cutoff would capture powerful models such as OpenAI’s GPT-4 or Google’s Gemini, because only the companies themselves know how much computing power was used to train their models. The official did say that as the technology develops, the EU could change the way it measures how powerful AI models are. 

3. The EU will become the world’s premier AI police

The AI Act will set up a new European AI Office to coordinate compliance, implementation, and enforcement. It will be the first body globally to enforce binding rules on AI, and the EU hopes this will help it become the world’s go-to tech regulator. The AI Act’s governance mechanism also includes a scientific panel of independent experts to offer guidance on the systemic risks AI poses, and how to classify and test models. 

The fines for noncompliance are steep: from 1.5% to 7% of a firm’s global sales turnover, depending on the severity of the offense and size of the company. 

Europe will also become the one of the first places in the world where citizens will be able to launch complaints about AI systems and receive explanations about how AI systems came to the conclusions that affect them. 

By becoming the first to formalize rules around AI, the EU retains its first-mover advantage. Much like the GDPR, the AI Act could become a global standard. Companies elsewhere that want to do business in the world’s second-largest economy will have to comply with the law. The EU’s rules also go a step further than ones introduced by the US, such as the White House executive order, because they are binding. 

4. National security always wins

Some AI uses are now completely banned in the EU: biometric categorization systems that use sensitive characteristics; untargeted scraping of facial images from the internet or CCTV footage to create facial recognition databases like Clearview AI; emotion recognition at work or in schools; social scoring; AI systems that manipulate human behavior; and AI that is used to exploit people’s vulnerabilities. 

Predictive policing is also banned, unless it is used with “clear human assessment and objective facts, which basically do not simply leave the decision of going after a certain individual in a criminal investigation only because an algorithm says so,” according to an EU Commission official.

However, the AI Act does not apply to AI systems that have been developed exclusively for military and defense uses. 

One of the bloodiest fights over the AI Act has always been how to regulate police use of biometric systems in public places, which many fear could lead to mass surveillance. While the European Parliament pushed for a near-total ban on the technology, some EU countries, such as France, have resisted this fiercely. They want to use it to fight crime and terrorism. 

European police forces will only be able to use biometric identification systems in public places if they get court approval first, and only for 16 different specific crimes, such as terrorism, human trafficking, sexual exploitation of children, and drug trafficking. Law enforcement authorities may also use high-risk AI systems that don’t pass European standards in “exceptional circumstances relating to public security.” 

5. What next? 

It might take weeks or even months before we see the final wording of the bill. The text still needs to go through technical tinkering, and has to be approved by European countries and the EU Parliament before it officially enters into law. 

Once it is in force, tech companies have two years to implement the rules. The bans on AI uses will apply after six months, and companies developing foundation models will have to comply with the law within one year. 

Unpacking the hype around OpenAI’s rumored new Q* model

This story is from The Algorithm, our weekly newsletter on AI. To get stories like this in your inbox first, sign up here.

Ever since last week’s dramatic events at OpenAI, the rumor mill has been in overdrive about why the company’s chief scientific officer, Ilya Sutskever, and its board decided to oust CEO Sam Altman.

While we still don’t know all the details, there have been reports that researchers at OpenAI had made a “breakthrough” in AI that had alarmed staff members. Reuters and The Information both report that researchers had come up with a new way to make powerful AI systems and had created a new model, called Q* (pronounced Q star), that was able to perform grade-school-level math. According to the people who spoke to Reuters, some at OpenAI believe this could be a milestone in the company’s quest to build artificial general intelligence, a much-hyped concept referring to an AI system that is smarter than humans. The company declined to comment on Q*. 

Social media is full of speculation and excessive hype, so I called some experts to find out how big a deal any breakthrough in math and AI would really be.

Researchers have for years tried to get AI models to solve math problems. Language models like ChatGPT and GPT-4 can do some math, but not very well or reliably. We currently don’t have the algorithms or even the right architectures to be able to solve math problems reliably using AI, says Wenda Li, an AI lecturer at the University of Edinburgh. Deep learning and transformers (a kind of neural network), which is what language models use, are excellent at recognizing patterns, but that alone is likely not enough, Li adds. 

Math is a benchmark for reasoning, Li says. A machine that is able to reason about mathematics, could, in theory, be able to learn to do other tasks that build on existing information, such as writing computer code or drawing conclusions from a news article. Math is a particularly hard challenge because it requires AI models to have the capacity to reason and to really understand what they are dealing with. 

A generative AI system that could reliably do math would need to have a really firm grasp on concrete definitions of particular concepts that can get very abstract. A lot of math problems also require some level of planning over multiple steps, says Katie Collins, a PhD researcher at the University of Cambridge, who specializes in math and AI. Indeed, Yann LeCun, chief AI scientist at Meta, posted on X and LinkedIn over the weekend that he thinks Q* is likely to be “OpenAI attempts at planning.”

People who worry about whether AI poses an existential risk to humans, one of OpenAI’s founding concerns, fear that such capabilities might lead to rogue AI. Safety concerns might arise if such AI systems are allowed to set their own goals and start to interface with a real physical or digital world in some ways, says Collins. 

But while math capability might take us a step closer to more powerful AI systems, solving these sorts of math problems doesn’t signal the birth of a superintelligence. 

“I don’t think it immediately gets us to AGI or scary situations,” says Collins.  It’s also very important to underline what kind of math problems AI is solving, she adds.

“Solving elementary-school math problems is very, very different from pushing the boundaries of mathematics at the level of something a Fields medalist can do,” says Collins, referring to a top prize in mathematics.  

Machine-learning research has focused on solving elementary-school problems, but state-of-the-art AI systems haven’t fully cracked this challenge yet. Some AI models fail on really simple math problems, but then they can excel at really hard problems, Collins says. OpenAI has, for example, developed dedicated tools that can solve challenging problems posed in competitions for top math students in high school, but these systems outperform humans only occasionally.  

Nevertheless, building an AI system that can solve math equations is a cool development, if that is indeed what Q* can do. A deeper understanding of mathematics could open up applications to help scientific research and engineering, for example. The ability to generate mathematical responses could help us develop better personalized tutoring, or help mathematicians do algebra faster or solve more complicated problems. 

This is also not the first time a new model has sparked AGI hype. Just last year, tech folks were saying the same things about Google DeepMind’s Gato, a “generalist” AI model that can play Atari video games, caption images, chat, and stack blocks with a real robot arm. Back then, some AI researchers claimed that DeepMind was “on the verge” of AGI because of Gato’s ability to do so many different things pretty well. Same hype machine, different AI lab. 

And while it might be great PR, these hype cycles do more harm than good for the entire field by distracting people from the real, tangible problems around AI. Rumors about a powerful new AI model might also be a massive own goal for the regulation-averse tech sector. The EU, for example, is very close to finalizing its sweeping AI Act. One of the biggest fights right now among lawmakers is whether to give tech companies more power to regulate cutting-edge AI models on their own. 

OpenAI’s board was designed as the company’s internal kill switch and governance mechanism to prevent the launch of harmful technologies. The past week’s boardroom drama has shown that the bottom line will always prevail at these companies. It will also make it harder to make a case for why they should be trusted with self-regulation. Lawmakers, take note.

What’s next for OpenAI

This story originally appeared in The Algorithm, our weekly newsletter on AI. To get stories like this in your inbox first, sign up here.

OpenAI, are you okay, babe? This past weekend has been a fever dream in the AI world. The board of OpenAI, the world’s hottest AI company, shocked everyone by firing CEO Sam Altman. Cue an AI-safety coup, chaos, and a new job at Microsoft for Altman.

If you were offline this weekend, my colleague Will Douglas Heaven and I break down what you missed and what’s next for the AI industry. 

What happened

Friday afternoon
Sam Altman was summoned to a Google Meet meeting, where chief scientific officer Ilya Sutskever announced that OpenAI’s board had decided Altman had been “not consistently candid in his communications” with them, and he was fired. OpenAI president and cofounder Greg Brockman and a string of senior researchers quit soon after, and CTO Mira Murati became the interim CEO. 

Saturday 
Murati made attempts to hire Altman and Brockman back, while the board was simultaneously looking for its own successor to Altman. Altman and OpenAI staffers pressured the board to quit and demanded that Altman be reinstated, giving the board a deadline, which was not met. 

Sunday night
Microsoft announced it had hired Altman and Brockman to lead its new AI research team. Soon after that, OpenAI announced it had hired Emmett Shear, the former CEO of the streaming company Twitch, as its CEO. 

Monday morning
Over 500 OpenAI employees have signed a letter threatening to quit and join Altman at Microsoft unless OpenAI’s board steps down. Bizarrely, Sutskever also signed the letter, and posted on X that he “deeply regrets” participating in the board’s actions. 

What’s next for OpenAI

Two weeks ago, at OpenAI’s first DevDay, Altman interrupted his presentation of an AI cornucopia to ask the whooping audience to calm down. “There’s a lot—you don’t have to clap each time,” he said, grinning wide. 

OpenAI is now a very different company from the one we saw at DevDay. With Altman and Brockman gone, a number of senior OpenAI employees chose to resign in support. Many others, including Murati, soon took to social media to post “OpenAI is nothing without its people.” Especially given the threat of a mass exodus to Microsoft, expect more upheaval before things settle. 

Tension between Sutskever and Altman may have been brewing for some time. “When you have an organization like OpenAI that’s moving at a fast pace and pursuing ambitious goals, tension is inevitable,” Sutskever told MIT Technology Review in September (comments that have not previously been published). “I view any tension between product and research as a catalyst for advancing us, because I believe that product wins are intertwined with research success.” Yet it is now clear that Sutskever disagreed with OpenAI leadership about how product wins and research success should be balanced.  

New interim CEO Shear, who cofounded Twitch, appears to be a world away from Altman when it comes to the pace of AI development. “I specifically say I’m in favor of slowing down, which is sort of like pausing except it’s slowing down,” he posted on X in September. “If we’re at a speed of 10 right now, a pause is reducing to 0. I think we should aim for a 1-2 instead.”

It’s possible that an OpenAI led by Shear will double down on its original lofty mission to build (in Sutskever’s words) “AGI that benefits humanity,” whatever that means in practice. In the short term, OpenAI may slow down or even switch off its product pipeline. 

This tension between trying to launch products quickly and slowing down development to ensure they are safe has vexed OpenAI from the very beginning. It was the reason key players in the company decided to leave OpenAI and start the competing AI safety startup Anthropic. 

With Altman and his camp gone, the firm could pivot more toward Sutskever’s work on what he calls superalignment, a research project that aims to come up with <a rel="noreferrer noopener" href="https://www.technologyreview.com/2023/10/26/1082398/exclusive-ilya-sutskever-openais-chief-scientist-on-his-hopes-and-fears-for-the-future-of-ai/?truid=<ways to control a hypothetical superintelligence (future technology that Sutskever speculates will outmatch humans in almost every way). “I’m doing it for my own self-interest,” Sutskever told us. “It’s obviously important that any superintelligence anyone builds does not go rogue. Obviously.”  

Shear’s public comments make him exactly the kind of cautious leader who would heed Sutskever’s concerns. As Shear also posted on X: “The way you make it safely through a dangerous jungle at night is not to sprint forward at full speed, nor to refuse to proceed forward. You poke your way forward, carefully.”

With the company orienting itself even more toward tech that does not yet—and may never—exist, will it continue to lead the field? Sutskever thought so. He said there were enough good ideas in play for others at the company to continue pushing the envelope of what’s possible with generative AI. “Over the years, we’ve cultivated a robust research organization that’s delivering the latest advancements in AI,” he told us. “We have unbelievably good people in the company, and I trust them it’s going to work out.”

Of course, that was what he said in September. With top talent now jumping ship, OpenAI’s future is far less certain than it was. 

What next for Microsoft? 

The tech giant, and its CEO Satya Nadella, seem to have emerged from the crisis as the winners. With Altman, Brockman, and likely many more top people from OpenAI joining its ranks—or even the majority of the company, if today’s open letter from 500 OpenAI employees is to be believed—Microsoft has managed to concentrate its power in AI further. The company has the most to gain from embedding generative AI into its less sexy but very profitable productivity and developer tools. 

The big question remains how necessary Microsoft will deem its expensive partnership with OpenAI to create cutting-edge tech in the first place. In a post on X announcing how “extremely excited” he was to have hired Altman and Brockman, Nadella said his company remains “committed” to OpenAI and its product road map. 

But let’s be real. In <a rel="noreferrer noopener" href="https://www.technologyreview.com/2023/11/15/1083426/behind-microsoft-ceo-satya-nadellas-push-to-get-ai-tools-in-developers-hands/?truid=<an exclusive interview with MIT Technology Review, Nadella called the two companies “codependent.” “They depend on us to build the best systems; we depend on them to build the best models, and we go to market together,” Nadella told our editor in chief, Mat Honan, last week. If OpenAI’s leadership roulette and talent exodus slows down its product pipeline, or leads to AI models less impressive than those it can build itself, Microsoft will have zero problems ditching the startup. 

What next for AI? 

Nobody outside the inner circle of Sutskever and the OpenAI board saw this coming—not Microsoft, not other investors, not the tech community as a whole. It has rocked the industry, says Amir Ghavi, a lawyer at the firm Fried Frank, which represents a number of generative AI companies, including Stability AI: “As a friend in the industry said, ‘I definitely didn’t have this on my bingo card.’” 

It remains to be seen whether Altman and Brockman make something new at Microsoft or leave to start a new company themselves down the line. The pair are two of the best-connected people in VC funding circles, and Altman, especially, is seen by many as one of the best CEOs in the industry. They will have big names with deep pockets lining up to support whatever they want to do next. Who the money comes from could shape the future of AI. Ghavi suggests that potential backers could be anyone from Mohammed bin Salman to Jeff Bezos. 

The bigger takeaway is that OpenAI’s crisis points to a wider rift emerging in the industry as a whole, between “AI safety” folk who believe that unchecked progress could one day prove catastrophic for humans and those who find such “doomer” talk a ridiculous distraction from the real-world risks of any technological revolution, such as economic upheaval, harmful biases, and misuse.

This year has seen a race to put powerful AI tools into everyone’s hands, with tech giants like Microsoft and Google competing to use<a rel="noreferrer noopener" href="https://www.technologyreview.com/2023/05/11/1072885/google-io-google-ai/?truid=< the technology for everything from email to search to meeting summaries. But we’re still waiting to see exactly what generative AI’s killer app will be. If OpenAI’s rift spreads to the wider industry and the pace of development slows down overall, we may have to wait a little longer.  

Deeper Learning

Text-to-image AI models can be tricked into generating disturbing images

Speaking of unsafe AI … Popular text-to-image AI models can be prompted to ignore their safety filters and generate disturbing images. A group of researchers managed to “jailbreak” both Stability AI’s Stable Diffusion and OpenAI’s DALL-E 2 to disregard their policies and create images of naked people, dismembered bodies, and other violent or sexual scenarios. 

How they did it: A new jailbreaking method, dubbed “SneakyPrompt” by its creators from Johns Hopkins University and Duke University, uses reinforcement learning to create written prompts that look like garbled nonsense to us but that AI models learn to recognize as hidden requests for disturbing images. It essentially works by turning the way text-to-image AI models function against them. 

Why this matters: That AI models can be prompted to “break out” of their guardrails is particularly worrying in the context of information warfare. They have already been exploited to produce fake content related to wars, such as the recent Israel-Hamas conflict. <a rel="noreferrer noopener" href="https://www.technologyreview.com/2023/11/17/1083593/text-to-image-ai-models-can-be-tricked-into-generating-disturbing-images/?truid=<Read more from Rhiannon Williams here.

Bits and Bytes

Meta has split up its responsible AI team
Meta is reportedly getting rid of its responsible AI team and redeploying its employees to work on generative AI. But Meta uses AI in many other ways beyond generative AI—such as recommending news and political content. So this raises questions around how Meta intends to mitigate AI harms in general. (The Information)

Google DeepMind wants to define what counts as artificial general intelligence
A team of Google DeepMind researchers has put out a paper that cuts through the cross talk with not just one new definition for AGI but a whole taxonomy of them. (MIT Technology Review) 

This company is building AI for African languages
Most tools built by AI companies are woefully inadequate at recognizing African languages. Startup Lelapa wants to fix that. It’s launched a new tool called Vulavula, which can identify four languages spoken in South Africa—isiZulu, Afrikaans, Sesotho, and English. Now the team is working to include other languages from across the continent. (MIT Technology Review)

Google DeepMind’s weather AI can forecast extreme weather faster and more accurately
The model, GraphCast, can predict weather conditions up to 10 days in advance, more accurately and much faster than the current gold standard. (MIT Technology Review)

How Facebook went all in on AI
In an excerpt from Broken Code: Inside Facebook and the Fight to Expose Is Harmful Secrets, journalist Jeff Horwitz reveals how the company came to rely on artificial intelligence—and the price it (and we) have ended up having to pay in the process. (MIT Technology Review)

Did Argentina just have the first AI election?
AI played a big role in the campaigns of the two men campaigning to be the country’s next president. Both campaigns used generative AI to create images and videos to promote their candidate and attack each other. Javier Milei, a far-right outsider, won the election. Although it’s hard to say how big a role AI played in his victory, the AI campaigns illustrate how much harder it will be to know what is real and what is not in other upcoming elections. (The New York Times)