Believe it or not, PPC automation was around long before the days of ChatGPT.
But with the rise in popularity of AI, machine learning, and automation – where does that leave PPC management?
Staying tried and true to manual PPC management has been shown to limit efficiency and margins.
But PPC automation layering isn’t just for pros. Whether you’re managing one account or 10 accounts, having a human component alongside automation can be the key to success.
This in-depth guide will teach you:
What PPC automation layering is.
How automation has impacted Google Ads.
How automation has impacted PPC practitioners.
Use cases for PPC automation layering.
What Is Automation Layering?
PPC automation layering is the strategic use of multiple automation tools and rules to manage and optimize PPC campaigns.
The main goal of PPC automation layering is to improve the efficiency and effectiveness of your PPC efforts.
This is where automation layering comes in.
Automation layering leverages different automation features, technologies, and sometimes 3rd party tools at various levels of your campaign management.
Some examples of automation layering include:
Smart Bidding strategies – Ad platforms take care of keyword bidding based on goals input within campaign settings. Examples of Smart Bidding include Target CPA, Target ROAS, Maximize Conversions, and more.
Automated PPC rules – Ad platforms can run specific account rules on a schedule based on the goal of the rule. An example would be to have Google Ads pause time-sensitive sale ads on a specific day and time.
PPC scripts – These are blocks of code that give ad platforms certain parameters to look out for and then have the platform take a specific action if those parameters are met.
Google Ads Recommendationstab – Google reviews campaign performance and puts together recommendations for PPC marketers to either take action on or dismiss if irrelevant.
Third-party automation tools – Tools such as Google Ads Editor, Optmyzr, Adalysis, and more can help take PPC management to the next level with their automated software and additional insights.
See the pattern here?
Automation and machine learning produce outputs of PPC management based on the inputs of PPC marketers to produce better campaign results.
How Has Automation Impacted Google Ads?
Over the years, Google Ads (as well as Microsoft Ads, Meta, etc.) have introduced countless automation features for marketers.
So, what’s their main goal here?
To make paid media management easier for advertisers.
In recent years, Google Ads has made a shift towards small to medium-sized businesses (SMBs), with claims that setting up a campaign is as easy as inputting a few settings and letting Google handle the rest.
For more experienced advertisers, they’ve experienced first-hand the shift from hands-on PPC management to a more “hands-off” approach with automation features.
While everyone can agree that easier paid media management sounds great, the learning curve for marketers has been far from perfect.
Automation has essentially taken over many of the day-to-day management tasks that PPC advertisers were used to doing.
For example, a few automation features built into the Google Ads platform include:
Keyword and campaign bid management.
Audience expansion.
Automated ad asset creation.
Keyword expansion.
And much more.
In theory, having Google Ads take over much of the busy work sounds great.
But in reality, the shift to automation layering has been far from great.
This leads us to the next big question: Will automation replace PPC marketers?
Does Automation Replace PPC Experts?
Job layoffs and restructuring due to automation is certainly a sensitive topic.
Add in a looming economic slowdown, and you’ve got many marketers clinging to their roles as hard as they can.
The marketing industry has seen shifts in job responsibilities, specifically in the content space, thanks to platforms like ChatGPT, Bard, and more.
But it’s time to settle this debate once and for all.
Automation will not replace the need for PPC marketers.
Now, keep in mind that automation has already replaced many of the day-to-day tasks that PPC experts have been used to doing for the past decade or so.
What we have, and will continue to see, is a shift in the role of PPC experts.
Since automation and machine learning take the role of day-to-day management, PPC experts will spend more time doing things such as:
Analyzing data and data quality.
Strategic decision making.
Reviewing and optimizing outputs from automation.
Automation and machines are great at pulling levers, making overall campaign management more efficient.
But automation tools alone cannot replace human touch and creating a story based on data and insights.
This is the beauty of PPC automation layering.
Lean into what automation tools have to offer, which leaves you more time to become a more strategic PPC marketer.
PPC Automation Layering Use Cases
There are many ways that PPC marketers and automation technologies can work together for optimal campaign results.
Below are just a few examples of how to use automation layering to your advantage.
1. Make The Most Of Smart Bidding Capabilities
As mentioned earlier in this guide, Smart Bidding is one of the most useful PPC automation tools.
Google Ads has developed its own automated bidding strategies to take the guesswork out of manual bid management.
However, Smart Bidding is not foolproof and certainly not a “set and forget” strategy.
Smart Bidding outputs can only be as effective as the inputs given to the machine learning system.
So, how should you use automation layering for Smart Bidding?
First, pick a Smart Bidding strategy that best fits an individual campaign goal. You can choose from:
Target cost per action (CPA).
Target return on ad spend (ROAS).
Maximize conversions.
Maximize conversion value.
Whenever starting a Smart Bidding strategy, it’s important to put some safeguards in place to reduce the volatility in campaign performance.
This could mean setting up an automated rule to alert you whenever significant volatility is reported, such as:
Spike in cost per click (CPC) or cost.
Dip in impressions, clicks, or cost.
Either of these scenarios could be due to learning curves in the algorithm, or it could be an indicator that your bids are too low or too high.
For example, say a campaign has a set target CPA goal of $25, but then all of a sudden, impressions and clicks fall off a cliff.
This could mean that the target CPA is set too low, and the algorithm has throttled ad serving to preserve only for individual users the algorithm thinks are most likely to purchase.
Without having an alert system in place, campaign volatility could go unnoticed for hours, days, or even weeks if you’re not checking performance in a timely manner.
2. Interact With Recommendations & Insights To Improve Automated Outputs
The goal of the ad algorithms is to get smarter every day and improve campaign performance.
But again, automated outputs are only as good as the input signals it’s been given at the beginning.
Many experienced PPC marketers tend to write off the Google Ads Recommendations or Insights tab due to perceptions of receiving irrelevant suggestions.
However, these systems were meant to learn from the input of marketers to better learn how to optimize.
Just because a recommendation is given in the platform does not mean you have to implement it.
The beauty of this tool is you have the ability to dismiss the opportunity and then tell Google why you’re dismissing it.
There’s even an option for “this is not relevant.”
Be willing to interact with the Recommendations and Insights tab on a weekly or bi-weekly basis to help better train the algorithms for optimizing performance based on what you signal as important.
Over time, these recommendations and insights can save you significant time over the weeks and months, which in turn gives you more time to focus on strategic measures in your account(s).
3. Automate Competitor Analysis With Tools
It’s one thing to ensure your ads and campaigns are running smoothly at all times.
Next-level strategy is using automation to keep track of your competitors and what they’re doing.
Multiple third-party tools have competitor analysis features to alert you on items such as:
Keyword coverage.
Content marketing.
Social media presence.
Market share.
And more.
Keep in mind that these tools are a paid subscription, but many are useful in many other automation areas outside of competitor analysis.
The goal is to not simply keep up with your competitors and copy what they’re doing.
Setting up automated competitor analysis helps you stay informed and can reinforce your market positioning or react in a way to help set you apart from competitor content.
In Summary
PPC automation layering has been around for a while and continues to mature each year.
While automation has replaced certain aspects of a PPC manager’s job, it can’t replace the necessary human components of an effective PPC strategy.
Use this guide to learn how to use automation layering to your advantage to get the most bang for your buck.
Celebrate the Holidays with some of SEJ’s best articles of 2023.
Our Festive Flashback series runs from December 21 – January 5, featuring daily reads on significant events, fundamentals, actionable strategies, and thought leader opinions.
2023 has been quite eventful in the SEO industry and our contributors produced some outstanding articles to keep pace and reflect these changes.
Catch up on the best reads of 2023 to give you plenty to reflect on as you move into 2024.
As an SEO professional, you’ve heard about ChatGPT and BARD – or even used them yourself.
You might also be familiar with a handful of machine learning models from Google, such as BERT and RankBrain.
These are all great applications of machine learning.
But it isn’t always immediately apparent how machine learning can be useful for your BAU (business as usual) tasks or daily SEO to-dos.
There’s also the question of ethics, as in data privacy and fairness, when it comes to using machine learning in your work.
However, understanding how machine learning works in search (and in real life) can only work to your advantage as an SEO pro – whether you’re technical or not.
In this comprehensive article, learn the types of machine learning and ethics associated with the technology, then take inspiration from seven machine learning examples that have had a great impact on our world.
As an added bonus, at the end of this article, explore eight machine learning examples directly applied to SEO.
What Is Machine Learning?
Machine learning is a subset of artificial intelligence (AI) which enables systems to learn and improve without the need to be explicitly programmed. This is achieved by using large data sets that train the parameters in the model.
Perhaps the most well-known examples of machine learning currently are ChatGPT and BARD – and while this post won’t be focusing on them, I decided to ask them both to explain to me what machine learning is. Here are their responses:
ChatGPT
“Machine learning is a type of artificial intelligence that allows software applications to learn from the data and become more accurate in predicting outcomes without explicit programming. In other words, it is a method of teaching computers to learn from the data they are given, and then to make decisions or predictions based on that learning.
The goal of machine learning is to develop algorithms that can learn patterns in data, and then use those patterns to make decisions or predictions about new data. This is done by training the machine learning algorithm on a dataset of known inputs and outputs, and then using that knowledge to make predictions on new, unseen data.
Some common applications of machine learning include image recognition, natural language processing, fraud detection, and recommendation systems.”
Screenshot from ChatGPT, April 2023
BARD
“Machine learning is the study of computer algorithms that improve automatically through experience and by the use of data. It is seen as a subset of artificial intelligence. Machine learning algorithms build a model based on sample data, known as training data, in order to make predictions or decisions without being explicitly programmed to do so.
Machine learning can be used for a variety of tasks, including:
Classification: Sorting data into categories. For example, a spam filter might classify emails as spam or not spam.
Regression: Predicting a continuous value. For example, a weather forecast might predict the temperature tomorrow.
Clustering: Grouping data points together based on their similarity. For example, a recommender system might cluster users together based on their past purchases.
Anomaly detection: Identifying unusual data points. For example, a fraud detection system might identify transactions that are likely to be fraudulent.
Machine learning is a powerful tool that can be used to solve a variety of problems. However, it is important to note that machine learning algorithms are only as good as the data they are trained on. If the data is biased, the algorithm will be biased as well. Additionally, machine learning algorithms can be susceptible to overfitting, which is when the algorithm learns the training data too well and does not generalize well to new data.
Despite these challenges, machine learning is a rapidly growing field with a wide range of applications. As the amount of data available to us continues to grow, machine learning is likely to play an even greater role in our lives in the future.”
Screenshot from BARD, April 2023
Supervised Learning
As the name suggests, in supervised learning, the model learns under supervision. It is trained with labeled data, and the output is manually reviewed.
The machine learns from the labeled data, which is then used to make future predictions.
Once the output is received, the model remembers it and will use it for the next operation.
Image from author, April 2023
Image from author, April 2023
There are two main types of supervised learning: classification and regression.
Classification
Classification is when the output variable is categorical, with two or more classes that the model can identify; for example, true or false and dog or cat.
Examples of this include predicting whether emails are likely to be spam or whether an image is of a dog or cat.
In both of these examples, the model will be trained on data that is either classified as spam or not spam, and whether an image contains a dog or cat.
Regression
This is when the output variable is a real or continuous value, and there is a relationship between the variables. Essentially, a change in one variable is associated with a change that occurs in the other variable.
The model then learns the relationship between them and predicts what the outcome will be depending on the data it is given.
For example, predicting humidity based on a given temperature value or what the stock price is likely to be at a given time.
Unsupervised Learning
Unsupervised learning is when the model uses unlabeled data and learns by itself, without any supervision. Essentially, unlike supervised learning, the model will act on the input data without any guidance.
It does not require any labeled data, as its job is to look for hidden patterns or structures in the input data and then organize it according to any similarities and differences.
For example, if a model is given pictures of both dogs and cats, it isn’t already trained to know the features that differentiate both. Still, it can categorize them based on patterns of similarities and differences.
Image from author, April 2023
There are also two main types of unsupervised learning: clustering and association.
Clustering
Clustering is the method of sorting objects into clusters that are similar to each other and belong to one cluster, versus objects that are dissimilar to a particular cluster and therefore belong in another.
Examples of this include recommendation systems and image classifying.
Association
Association is rule-based and is used to discover the probability of the co-occurrence of items within a collection of values.
Examples include fraud detection, customer segmentation, and discovering purchasing habits.
Semi-supervised Learning
Semi-supervised learning bridges both supervised and unsupervised learning by using a small section of labeled data, together with unlabeled data, to train the model. It, therefore, works for various problems, from classification and regression to clustering and association.
Semi-supervised learning can be used if there is a large amount of unlabeled data, as it only requires a small portion of the data to be labeled to train the model, which can then be applied to the remaining unlabeled data.
Google has used semi-supervised learning to better understand language used within a search to ensure it serves the most relevant content for a particular query.
Reinforcement Learning
Reinforcement learning is when a model is trained to return the optimum solution to a problem by taking a sequential approach to decision-making.
It uses trial and error from its own experiences to define the output, with rewards for positive behavior and negative reinforcement if it is not working towards the goal.
The model interacts with the environment that has been set up and comes up with solutions without human interference.
Human interference will then be introduced to provide either positive or negative reinforcement depending on how close to the goal the output is.
Examples include robotics – think robots working in a factory assembly line – and gaming, with AlphaGo as the most famous example. This is where the model was trained to beat the AlphaGo champion by using reinforcement learning to define the best approach to win the game.
Machine Learning Ethics
There is no doubt that machine learning has many benefits, and the use of machine learning models is ever-growing.
However, it’s important to consider the ethical concerns that come with using technology of this kind. These concerns include:
The accuracy of a machine learning model and whether it will generate the correct output.
Bias in the data that is used to train models, which causes a bias in the model itself, and, therefore, a bias in the outcome. If there is historical bias in data, that bias will often be replicated throughout.
The fairness in outcomes and the overall process.
Privacy – particularly with data that is used to train machine learning models – as well as the accuracy of the outcomes and predictions.
7 Machine Learning Examples In The Real World
1. Netflix
Netflix uses machine learning in a number of ways to provide the best experience for its users.
The company is also continually collecting large amounts of data, including ratings, the location of users, the length of time for which something is watched, if content is added to a list, and even whether something has been binge-watched.
This data is then used to further improve its machine learning models.
Content Recommendations
TV and movie recommendations on Netflix are personalized to each individual user’s preferences. To do this, Netflix deployed a recommendation system that considers previous content consumed, users’ most viewed genres, and content watched by users with similar preferences.
Auto-generated Thumbnails
Netflix discovered that the images used on the browse screen make a big difference in whether users watch something or not.
It, therefore, uses machine learning to create and display different images according to a user’s individual preferences. It does this by analyzing a user’s previous content choices and learning the kind of image that is more likely to encourage them to click.
These are just two examples of how Netflix uses machine learning on its platform. If you want to learn more about how it is used, you can check out the company’s research areas blog.
2. Airbnb
With millions of listings in locations across the globe at different price points, Airbnb uses machine learning to ensure users can find what they are looking for quickly and to improve conversions.
There are a number of ways the company deploys machine learning, and it shares a lot of details on its engineering blog.
Image Classification
As hosts can upload images for their properties, Airbnb found that a lot of images were mislabeled. To try and optimize user experience, it deployed an image classification model that used computer vision and deep learning.
The project aimed to categorize photos based on different rooms. This enabled Airbnb to show listing images grouped by room type and ensure the listing follows Airbnb’s guidelines.
In order to do this, it retrained the image classification neural network ResNet50, with a small number of labeled photos. This enabled it to accurately classify current and future images uploaded to the site.
Search Ranking
To provide a personalized experience for users, Airbnb deployed a ranking model that optimized search and discovery. The data for this model came from user engagement metrics such as clicks and bookings.
Listings started by being ordered randomly, and then various factors were given a weight within the model – including price, quality, and popularity with users. The more weight a listing had, the higher it would be displayed in listings.
This has since been optimized further, with training data including the number of guests, price, and availability also included within the model to discover patterns and preferences to create a more personalized experience.
Spotify uses a recommendation algorithm that predicts a user’s preference based on a collection of data from other users. This is due to numerous similarities that occur between music types that clusters of people listen to.
Playlists are one way it can do this, using statistical methods to create personalized playlists for users, such as Discover Weekly and daily mixes.
It can then use further data to adjust these depending on a user’s behavior.
With personal playlists also being created in the millions, Spotify has a huge database to work with – particularly if songs are grouped and labeled with semantic meaning.
This has allowed the company to recommend songs to users with similar music tastes. The machine learning model can serve songs to users with a similar listening history to aid music discovery.
Natural Language
With the Natural Processing Language (NLP) algorithm enabling computers to understand text better than ever before, Spotify is able to categorize music based on the language used to describe it.
It can scrape the web for text on a particular song and then use NLP to categorize songs based on this context.
This also helps algorithms identify songs or artists that belong in similar playlists, which further helps the recommendation system.
4. Detecting Fake News
While AI tools such as machine learning content generation can be a source for creating fake news, machine learning models that use natural language processing can also be used to assess articles and determine if they include false information.
Social network platforms use machine learning to find words and patterns in shared content that could indicate fake news is being shared and flag it appropriately.
5. Health Detection
There is an example of a neural network that was trained on over 100,000 images to distinguish dangerous skin lesions from benign ones. When tested against human dermatologists, the model could accurately detect 95% of skin cancer from the images provided, compared to 86.6% by the dermatologists.
As the model missed fewer melanomas, it was determined to have a higher sensitivity and was continually trained throughout the process.
There is hope that machine learning and AI, together with human intelligence, may become a useful tool for faster diagnosis.
Other ways image detection is being used in healthcare include identifying abnormalities in X-rays or scans and identifying key markups that may indicate an underlying illness.
6. Wildlife Security
Protection Assistant for Wildlife Security is an AI system that is being used to evaluate information about poaching activity to create a patrol route for conservationists to help prevent poaching attacks.
The system is continually being provided with more data, such as locations of traps and sightings of animals, which helps it to become smarter.
The predictive analysis enables patrol units to identify areas where it is likely animal poachers will visit.
8 Machine Learning Examples In SEO
1. Content Quality
Machine learning models can be trained to improve the quality of website content by predicting what both users and search engines would prefer to see.
The model can be trained on the most important insights, including search volume and traffic, conversion rate, internal links, and word count.
A content quality score can then be generated for each page, which will help inform where optimizations need to be made and can be particularly useful for content audits.
2. Natural Language Processing
Natural Language Processing (NLP) uses machine learning to reveal the structure and meaning of text. It analyzes text to understand the sentiment and extract key information.
NLP focuses on understanding context rather than just words. It is more about the content around keywords and how they fit together into sentences and paragraphs, than keywords on their own.
The overall sentiment is also taken into account, as it refers to the feeling behind the search query. The types of words used within the search help to determine whether it is classified as having a positive, negative, or neutral sentiment.
The key areas of importance for NLP are;
Entity – Words representing tangible objects such as people, places, and things that are identified and evaluated.
Categories – Text separated into categories.
Salience – How relevant the entity is.
Google has a free NLP API demo that can be used to analyze how text is seen and understood by Google. This enables you to identify improvements to content.
Recommendations In The World Of NLP
NLP is also being used to review and understand anchor text that is used to link pages. Therefore, it is more important than ever to ensure anchor text is relevant and informative.
Ensuring each page has a natural flow, with headings providing hierarchy and readability.
Answering the question the article is querying as quickly as possible. Ensure that users and search engines can discover key information without making too much effort.
Ensure you have the correct spelling and punctuation used to display authority and trustworthiness.
3. Google’s Models
AI and machine learning is used throughout Google’s many products and services. The most popular use of it in the context of search is to understand language and the intent behind search queries.
Previously, the search systems looked for matching words only, which didn’t even consider misspellings. Eventually, algorithms were created to find patterns that identified misspellings and potential typos.
There have been several systems introduced throughout the last few years after Google confirmed in 2016 its intention to become a machine learning first company.
RankBrain
The first of these was RankBrain, which was introduced in 2015 and helps Google to understand how different words are related to different concepts.
This enables Google to take a broad query and better define how it relates to real-world concepts.
Google’s systems learn from seeing words used in a query on the page, which it can then use to understand terms and match them to related concepts to understand what a user is searching for.
Neural Matching
Neural matching was launched in 2018 and introduced to local search in 2019.
This helps Google understand how queries relate to pages by looking at the content on a page, or a search query, and understanding it within the context of the page content or query.
Most queries made today make use of neural matching, and it is used in rankings.
This system enables Google to understand how combinations of words express different meanings and intent by reviewing the whole sequence of words on a page.
BERT is now used in most queries, as it helps Google understand what a user is looking for to surface the best results related to the search.
MUM
MUM, which means Multitask Unified Model, was introduced in 2021 and is used to understand languages and variations in search terms.
This uses the latest advancements to find patterns in sentences and correlations between different words to understand nuanced questions – and even predict which words are likely to come next.
4. Predictive Prefetching
By combining historical website data on user behavior with the capabilities of machine learning, some tools can guess which page a user is likely to navigate to next and begin prefetching the necessary resources to load the page.
This is known as predictive prefetching and can enhance website performance.
Predictive prefetching can also apply to other scenarios, such as forecasting pieces of content or widgets that users are most likely to view or interact with and personalizing the experience based on that information.
5. Testing
Running SEO A/B tests is one of the most effective ways to provide the SEO impact of changes, and the ability to generate statistically significant results is possible with the use of machine learning algorithms and neural networks.
SearchPilot is an example of SEO A/B testing that is powered by machine learning and neural network models.
Starting with a bucketing algorithm that creates statistically similar buckets of control and variant pages to perform tests on, a neural network model then forecasts expected traffic to the pages the test is being run on.
The neural network model, which is trained to account for any and all external influences such as seasonality, competitor activity, and algorithm updates, will also analyze the organic search traffic to the variant pages and identify how they perform against the control group throughout the test.
This also enables users to calculate whether any difference in traffic is statistically significant.
(Disclaimer: I work for SearchPilot.)
6. Internal Linking
Machine learning can help with internal linking in two ways:
Updating broken links: Machine learning can crawl your site to spot any broken internal links and then replace them with a link to the best alternative page.
Suggesting relevant internal linking: These tools can leverage big data to suggest relevant internal links during the article creation process and over time.
The other internal linking task is an internal link audit. This includes analyzing the number of internal links to a page, the placement of the links together with the anchor text, and the overall crawl depth of the page.
Anchor text classification can also be performed to identify the phrases used most frequently in alt text and categorize them based on topics and whether they are branded or non-branded terms.
7. Image Captioning For Alt Text
As SEO pros, we understand the importance of image alt text. They improve accessibility for people who use screen readers while also helping search engine crawlers understand the content of the page they are placed on.
Language vision models can be used to automatically caption images, therefore providing content that can be used as alt text. Image captioning is used to describe what is shown within an image in a single sentence.
Two models are used for image captioning, both as important as the other. The image-based model will start by extracting features from the image, while the language-based model will translate those features into a logical sentence.
If you’re interested in how machine learning can be used in daily SEO tasks, this article by Lazarina Stoy is a must-read – and if you would like to play around with some super interesting scripts, this collection of Colab notebooks from Britney Muller is the perfect place to start.
In Conclusion
Machine learning isn’t limited to just ChatGPT and BARD.
There are many practical applications for machine learning, both in the real world and specifically in the world of SEO – and these are likely just the beginning.
And while it will be vital to remain cognizant of the ethical questions associated with machine learning, it has exciting implications for the future of SEO.
Google has postponed the launch of Gemini, which was initially set for next week. According to sources for The Information, the delay is due to the AI’s inconsistent performance with non-English queries.
Why Was Google Gemini Delayed?
Google’s CEO, Sundar Pichai, reportedly canceled several events in California, New York, and Washington, where Gemini would be unveiled.
These events were crucial to Google’s strategy, marking what could have been its most significant product introduction of the year.
The delay highlights the intense competition in the AI sector, particularly against OpenAI, Microsoft, and Meta.
GPT-4 Outperformed Gemini In Multilingual Tasks
In certain aspects, Gemini, comparable to OpenAI’s GPT-4, has fallen short in handling multilingual tasks effectively.
This shortcoming is particularly notable given Google’s global market presence and the importance of diverse language support in AI technologies.
Implications For Google’s AI Strategy
This development has implications beyond Gemini itself. Other Google products like Bard, Assistant, and Docs, which are expected to be enhanced by Gemini’s capabilities, may delay receiving these updates.
This setback contrasts the growing popularity of Microsoft Copilot, which recently announced integration with OpenAI’s latest new features, including GPTs.
Traditionally a leader in AI, Google is racing to match the pace of innovation set by OpenAI. While Google Bard’s capabilities continuously increase, it still falls behind the more advanced features available to premium users of ChatGPT.
Despite the delay, Google remains committed to advancing Gemini, with Pichai expressing a focus on ensuring its competitiveness and state-of-the-art capabilities.
As the company refines Gemini, how it will reshape the landscape of conversational AI and compete with rapidly advancing rivals like OpenAI remains to be seen.
In its latest announcement, OpenAI introduced Data Partnerships, a collaborative effort aimed at enhancing the training of AI models, particularly as the world edges closer to achieving artificial general intelligence (AGI).
This initiative aims to ensure that AI systems deeply understand the vast array of subject matters, industries, cultures, and languages, a goal only possible through accessing a wide-ranging training dataset.
According to the company, the value of incorporating varied datasets into AI is immense, especially for AGI. By understanding nuanced human intentions, behaviors, and interactions, AGI can serve humanity in a more sophisticated and beneficial manner.
This development could be particularly noteworthy for in-house brand marketers, SEO professionals, and marketing agencies.
The potential of AI to comprehend and analyze complex marketing data and consumer behaviors could be significantly enhanced by integrating industry-specific datasets and domain-specific knowledge.
OpenAI Data Partnerships — an open call for organizations to work with us to represent their data in AI training: https://t.co/aTHBQBRddK
The call for large-scale datasets is not limited to any single modality. OpenAI is open to text, images, audio, and video.
They are particularly interested in data that captures human intentions, like long-form writing or conversations, rather than disconnected snippets. Not your Slack archives.
Data partners have the opportunity to influence the future of AI and stand to gain from AI models that are better attuned to their specific needs.
Organization types invited to participate include academic, research, religious, government, political, media, for-profit, non-profit, grassroots community, and individual.
Screenshot from OpenAI, November 2023
Interested partners must create an open-source dataset that can be utilized publicly for AI training or prepare private datasets for training proprietary AI models.
One approach plays a significant role in fostering an open ecosystem, while the other caters to organizations wishing to keep their data confidential.
To streamline the process, OpenAI is prepared to handle data in almost any form (digital or physical) and will assist with digitization and structuring using its advanced OCR and ASR technologies.
In addition, OpenAI is keen on maintaining privacy and sensitive information, offering to work closely with partners to clean the data, ensuring it is free from personal or third-party information.
As AI continues to intertwine with various sectors, the invitation from OpenAI presents a unique opportunity for the marketing industry.
By contributing to the creation of AGI that truly understands the market and its consumers, organizations can not only guide the trajectory of AI development but also reap the rewards of more sophisticated, domain-aware AI tools.
OpenAI’s Data Partnerships represent a critical step towards realizing AGI that is safe, beneficial, and comprehensively informed by “human society.”
One evening in early September, T, a 28-year-old actor who asked to be identified by his first initial, took his seat in a rented Hollywood studio space in front of three cameras, a director, and a producer for a somewhat unusual gig.
The two-hour shoot produced footage that was not meant to be viewed by the public—at least, not a human public.
Rather, T’s voice, face, movements, and expressions would be fed into an AI database “to better understand and express human emotions.” That database would then help train “virtual avatars” for Meta, as well as algorithms for a London-based emotion AI company called Realeyes. (Realeyes was running the project; participants only learned about Meta’s involvement once they arrived on site.)
The “emotion study” ran from July through September, specifically recruiting actors. The project coincided with Hollywood’s historic dual strikes by the Writers Guild of America and the Screen Actors Guild (SAG-AFTRA). With the industry at a standstill, the larger-than-usual number of out-of-work actors may have been a boon for Meta and Realeyes: here was a new pool of “trainers”—and data points—perfectly suited to teaching their AI to appear more human.
For actors like T, it was a great opportunity too: a way to make good, easy money on the side, without having to cross the picket line.
“There aren’t really clear rules right now.”
“This is fully a research-based project,” the job posting said. It offered $150 per hour for at least two hours of work, and asserted that “your individual likeness will not be used for any commercial purposes.”
The actors may have assumed this meant that their faces and performances wouldn’t turn up in a TV show or movie, but the broad nature of what they signed makes it impossible to know the full implications for sure. In fact, in order to participate, they had to sign away certain rights “in perpetuity” for technologies and use cases that may not yet exist.
And while the job posting insisted that the project “does not qualify as struck work” (that is, work produced by employers against whom the union is striking), it nevertheless speaks to some of the strike’s core issues: how actors’ likenesses can be used, how actors should be compensated for that use, and what informed consent should look like in the age of AI.
“This isn’t a contract battle between a union and a company,” said Duncan Crabtree-Ireland, SAG-AFTRA’s chief negotiator, at a panel on AI in entertainment at San Diego Comic-Con this summer. “It’s existential.”
Many actors across the industry, particularly background actors (also known as extras), worry that AI—much like the models described in the emotion study—could be used to replace them, whether or not their exact faces are copied. And in this case, by providing the facial expressions that will teach AI to appear more human, study participants may in fact have been the ones inadvertently training their own potential replacements.
“Our studies have nothing to do with the strike,” Max Kalehoff, Realeyes’s vice president for growth and marketing, saidin an email. “The vast majority of our work is in evaluating the effectiveness of advertising for clients—which has nothing to do with actors and the entertainment industry except to gauge audience reaction.” The timing, he added, was “an unfortunate coincidence.” Meta did not respond to multiple requests for comment.
Given how technological advancements so often build upon one another, not to mention how quickly the field of artificial intelligence is evolving, experts point out that there’s only so much these companies can truly promise.
In addition to the job posting, MIT Technology Review has obtained and reviewed a copy of the data license agreement, and its potential implications are indeed vast. To put it bluntly: whether the actors who participated knew it or not, for as little as $300, they appear to have authorized Realeyes, Meta, and other parties of the two companies’ choosing to access and use not just their faces but also their expressions, and anything derived from them, almost however and whenever they want—as long as they do not reproduce any individual likenesses.
Some actors, like Jessica, who asked to be identified by just her first name, felt there was something “exploitative” about the project—both in the financial incentives for out-of-work actors and in the fight over AI and the use of an actor’s image.
Jessica, a New York–based background actor, says she has seen a growing number of listings for AI jobs over the past few years. “There aren’t really clear rules right now,” she says, “so I don’t know. Maybe … their intention [is] to get these images before the union signs a contract and sets them.”
Do you have any tips related to how AI is being used in the entertainment industry? Please reach out at tips@technologyreview.com or securely on Signal at 626.765.5489.
All this leaves actors, struggling after three months of limited to no work, primed to accept the terms from Realeyes and Meta—and, intentionally or not, to affect all actors, whether or not they personally choose to engage with AI.
“It’s hurt now or hurt later,” says Maurice Compte, an actor and SAG-AFTRA member who has had principal roles on shows like Narcos and Breaking Bad. After reviewing the job posting, he couldn’t help but see nefarious intent. Yes, he said, of course it’s beneficial to have work, but he sees it as beneficial “in the way that the Native Americans did when they took blankets from white settlers,” adding: “They were getting blankets out of it in a time of cold.”
Humans as data
Artificial intelligence is powered by data, and data, in turn, is provided by humans.
It is human labor that prepares, cleans, and annotates data to make it more understandable to machines; as MIT Technology Review has reported, for example, robot vacuums know to avoid running over dog poop because human data labelers have first clicked through and identified millions of images of pet waste—and other objects—inside homes.
When it comes to facial recognition, other biometric analysis, or generative AI models that aim to generate humans or human-like avatars, it is human faces, movements, and voices that serve as the data.
Initially, these models were powered by data scraped off the internet—including, on several occasions, private surveillance camera footage that was shared or sold without the knowledge of anyone being captured.
But as the need for higher-quality data has grown, alongside concerns about whether data is collected ethically and with proper consent, tech companies have progressed from “scraping data from publicly available sources” to “building data sets with professionals,” explains Julian Posada, an assistant professor at Yale University who studies platforms and labor. Or, at the very least, “with people who have been recruited, compensated, [and] signed [consent] forms.”
But the need for human data, especially in the entertainment industry, runs up against a significant concern in Hollywood: publicity rights, or “the right to control your use of your name and likeness,” according to Corynne McSherry, the legal director of the Electronic Frontier Foundation (EFF), a digital rights group.
This wasan issuelong before AI, but AI has amplified the concern. Generative AI in particular makes it easy to create realistic replicas of anyone by training algorithms on existing data, like photos and videos of the person. The more data that is available, the easier it is to create a realistic image. This has a particularly large effect on performers.
He believes it’s that improvisation requirement that explains why Realeyes and Meta were specifically recruiting actors.
Some actors have been able to monetize the characteristics that make them unique. James Earl Jones, the voice of Darth Vader, signed off on the use of archived recordings of his voice so that AI could continue to generate it for future Star Wars films. Meanwhile, de-aging AI has allowed Harrison Ford, Tom Hanks, and Robin Wright to portray younger versions of themselves on screen. Metaphysic AI, the company behind the de-aging technology, recently signed a deal with Creative Artists Agency to put generative AI to use for its artists.
But many deepfakes, or images of fake events created with deep-learning AI, are generated without consent. Earlier this month, Hanks posted on Instagram that an ad purporting to show him promoting a dental plan was not actually him.
The AI landscape is different for noncelebrities. Background actors are increasingly being asked to undergo digital body scans on set, where they have little power to push back or even get clarity on how those scans will be used in the future. Studios say that scans are used primarily to augment crowd scenes, which they have been doing with other technology in postproduction for years—but according to SAG representatives, once the studios have captured actors’ likenesses, they reserve the rights to use them forever. (There have already been multiple reports from voice actors that their voices have appeared in video games other than the ones they were hired for.)
In the case of the Realeyes and Meta study, it might be “study data” rather than body scans, but actors are dealing with the same uncertainty as to how else their digital likenesses could one day be used.
Teaching AI to appear more human
At $150 per hour, the Realeyes study paid far more than the roughly $200 daily rate in the current Screen Actors Guild contract (nonunion jobs pay even less).
This made the gig an attractive proposition for young actors like T, just starting out in Hollywood—a notoriously challenging environment even had he not arrived just before the SAG-AFTRA strike started. (T has not worked enough union jobs to officially join the union, though he hopes to one day.)
In fact, even more than a standard acting job, T described performing for Realeyes as “like an acting workshop where … you get a chance to work on your acting chops, which I thought helped me a little bit.”
For two hours, T responded to prompts like “Tell us something that makes you angry,” “Share a sad story,” or “Do a scary scene where you’re scared,” improvising an appropriate story or scene for each one. He believes it’s that improvisation requirement that explains why Realeyes and Meta were specifically recruiting actors.
In addition to wanting the pay, T participated in the study because, as he understood it, no one would see the results publicly. Rather, it was research for Meta, as he learned when he arrived at the studio space and signed a data license agreement with the company that he only skimmed through. It was the first he’d heard that Meta was even connected with the project. (He had previously signed a separate contract with Realeyes covering the terms of the job.)
The data license agreement says that Realeyes is the sole owner of the data and has full rights to “license, distribute, reproduce, modify, or otherwise create and use derivative works” generated from it, “irrevocably and in all formats and media existing now or in the future.”
This kind of legalese can be hard to parse, particularly when it deals with technology that is changing at such a rapid pace. But what it essentially means is that “you may be giving away things you didn’t realize … because those things didn’t exist yet,” says Emily Poler, a litigator who represents clients in disputes at the intersection of media, technology, and intellectual property.
“If I was a lawyer for an actor here, I would definitely be looking into whether one can knowingly waive rights where things don’t even exist yet,” she adds.
As Jessica argues, “Once they have your image, they can use it whenever and however.” She thinks that actors’ likenesses could be used in the same way that other artists’ works, like paintings, songs, and poetry, have been used to train generative AI, and she worries that the AI could just “create a composite that looks ‘human,’ like believable as human,” but “it wouldn’t be recognizable as you, so you can’t potentially sue them”—even if that AI-generated human was based on you.
This feels especially plausible to Jessica given her experience as an Asian-American background actor in an industry where representation often amounts to being the token minority. Now, she fears, anyone who hires actors could “recruit a few Asian people” and scan them to create “an Asian avatar” that they could use instead of “hiring one of you to be in a commercial.”
It’s not just images that actors should be worried about, says Adam Harvey, an applied researcher who focuses on computer vision, privacy, and surveillance and is one of the co-creators of Exposing.AI, which catalogues the data sets used to train facial recognition systems.
What constitutes “likeness,” he says, is changing. While the word is now understood primarily to mean a photographic likeness, musicians are challenging that definition to include vocal likenesses. Eventually, he believes, “it will also … be challenged on the emotional frontier”—that is, actors could argue that their microexpressions are unique and should be protected.
Realeyes’s Kalehoff did not say what specifically the company would be using the study results for, though he elaborated in an email that there could be “a variety of use cases, such as building better digital media experiences, in medical diagnoses (i.e. skin/muscle conditions), safety alertness detection, or robotic tools to support medical disorders related to recognition of facial expressions (like autism).”
Now, she fears, anyone who hires actors could “recruit a few Asian people” and scan them to create “an Asian avatar” that they could use instead of “hiring one of you to be in a commercial.”
When asked how Realeyes defined “likeness,” he replied that the company used that term—as well as “commercial,” another word for which there are assumed but no universally agreed-upon definitions—in a manner that is “the same for us as [a] general business.” He added, “We do not have a specific definition different from standard usage.”
But for T, and for other actors, “commercial” would typically mean appearing in some sort of advertisement or a TV spot—“something,” T says, “that’s directly sold to the consumer.”
Outside of the narrow understanding in the entertainment industry, the EFF’s McSherry questions what the company means: “It’s a commercial company doing commercial things.”
Kalehoff also said, “If a client would ask us to use such images [from the study], we would insist on 100% consent, fair pay for participants, and transparency. However, that is not our work or what we do.”
Yet this statement does not align with the language of the data license agreement, which stipulates that while Realeyes is the owner of the intellectual property stemming from the study data, Meta and “Meta parties acting on behalf of Meta” have broad rights to the data—including the rights to share and sell it. This means that, ultimately, how it’s used may be out of Realeyes’s hands.
As explained in the agreement, the rights of Meta and parties acting on its behalf also include:
Asserting certain rights to the participants’ identities (“identifying or recognizing you … creating a unique template of your face and/or voice … and/or protecting against impersonation and identity misuse”)
Allowing other researchers to conduct future research, using the study data however they see fit (“conducting future research studies and activities … in collaboration with third party researchers, who may further use the Study Data beyond the control of Meta”)
Creating derivative works from the study data for any kind of use at any time (“using, distributing, reproducing, publicly performing, publicly displaying, disclosing, and modifying or otherwise creating derivative works from the Study Data, worldwide, irrevocably and in perpetuity, and in all formats and media existing now or in the future”)
The only limit on use was that Meta and parties would “not use Study Data to develop machine learning models that generate your specific face or voice in any Meta product” (emphasis added). Still, the variety of possible use cases—and users—is sweeping. And the agreement does little to quell actors’ specific anxieties that “down the line, that database is used to generate a work and that work ends up seeming a lot like [someone’s] performance,” as McSherry puts it.
When I asked Kalehoff about the apparent gap between his comments and the agreement, he denied any discrepancy: “We believe there are no contradictions in any agreements, and we stand by our commitment to actors as stated in all of our agreements to fully protect their image and their privacy.” Kalehoff declined to comment on Realeyes’s work with clients, or to confirm that the study was in collaboration with Meta.
Meanwhile, Meta has been building photorealistic 3D “Codec avatars,” which go far beyond the cartoonish images in Horizon Worlds and require human training data to perfect. CEO Mark Zuckerberg recently described these avatars on the popular podcast from AI researcher Lex Fridman as core to his vision of the future—where physical, virtual, and augmented reality all coexist. He envisions the avatars “delivering a sense of presence as if you’re there together, no matter where you actually are in the world.”
Despite multiple requests for comment, Meta did not respond to any questions from MIT Technology Review, so we cannot confirm what it would use the data for, or who it means by “parties acting on its behalf.”
Individual choice, collective impact
Throughout the strikes by writers and actors, there has been a palpable sense that Hollywood is charging into a new frontier that will shape how we—all of us—engage with artificial intelligence. Usually, that frontier is described with reference to workers’ rights; the idea is that whatever happens here will affect workers in other industries who are grappling with what AI will mean for their own livelihoods.
Already, the gains won by the Writers Guild have provided a model for how to regulate AI’s impact on creative work. The union’s new contract with studios limits the use of AI in writers’ rooms and stipulates that only human authors can be credited on stories, which prevents studios from copyrighting AI-generated work and further serves as a major disincentive to use AI to write scripts.
In early October, the actors’ union and the studios also returned to the bargaining table, hoping to provide similar guidance for actors. But talks quickly broke down because “it is clear that the gap between the AMPTP [Alliance of Motion Picture and Television Producers] and SAG-AFTRA is too great,” as the studio alliance put it in a press release. Generative AI—specifically, how and when background actors should be expected to consent to body scanning—was reportedly one of the sticking points.
Whatever final agreement they come to won’t forbid the use of AI by studios—that was never the point. Even the actors who took issue with the AI training projects have more nuanced views about the use of the technology. “We’re not going to fully cut out AI,” acknowledges Compte, the Breaking Bad actor. Rather, we “just have to find ways that are going to benefit the larger picture… [It] is really about living wages.”
But a future agreement, which is specifically between the studios and SAG, will not be applicable to tech companies conducting “research” projects, like Meta and Realeyes. Technological advances created for one purpose—perhaps those that come out of a “research” study—will also have broader applications, in film and beyond.
“The likelihood that the technology that is developed is only used for that [audience engagement or Codec avatars] is vanishingly small. That’s not how it works,” says the EFF’s McSherry. For instance, while the data agreement for the emotion study does not explicitly mention using the results for facial recognition AI, McSherry believes that they could be used to improve any kind of AI involving human faces or expressions.
(Besides, emotion detection algorithms are themselves controversial, whether or not they even work the way developers say they do. Do we really want “our faces to be judged all the time [based] on whatever products we’re looking at?” asks Posada, the Yale professor.)
This all makes consent for these broad research studies even trickier: there’s no way for a participant to opt in or out of specific use cases. T, for one, would be happy if his participation meant better avatar options for virtual worlds, like those he uses with his Oculus—though he isn’t agreeing to that specifically.
But what are individual study participants—who may need the income—to do? What power do they really have in this situation? And what power do other people—even people who declined to participate—have to ensure that they are not affected? The decision to train AI may be an individual one, but the impact is not; it’s collective.
“Once they feed your image and … a certain amount of people’s images, they can create an endless variety of similar-looking people,” says Jessica. “It’s not infringing on your face, per se.” But maybe that’s the point: “They’re using your image without … being held liable for it.”
T has considered the possibility that, one day, the research he has contributed to could very wellreplace actors.
But at least for now, it’s a hypothetical.
“I’d be upset,” he acknowledges, “but at the same time, if it wasn’t me doing it, they’d probably figure out a different way—a sneakier way, without getting people’s consent.” Besides, T adds, “they paid really well.”
Do you have any tips related to how AI is being used in the entertainment industry? Please reach out at tips@technologyreview.com or securely on Signal at 626.765.5489.
One evening in early September, T, a 28-year-old actor who asked to be identified by his first initial, took his seat in a rented Hollywood studio space in front of three cameras, a director, and a producer for a somewhat unusual gig.
The two-hour shoot produced footage that was not meant to be viewed by the public—at least, not a human public.
Rather, T’s voice, face, movements, and expressions would be fed into an AI database “to better understand and express human emotions.” That database would then help train “virtual avatars” for Meta, as well as algorithms for a London-based emotion AI company called Realeyes. (Realeyes was running the project; participants only learned about Meta’s involvement once they arrived on site.)
The “emotion study” ran from July through September, specifically recruiting actors. The project coincided with Hollywood’s historic dual strikes by the Writers Guild of America and the Screen Actors Guild (SAG-AFTRA). With the industry at a standstill, the larger-than-usual number of out-of-work actors may have been a boon for Meta and Realeyes: here was a new pool of “trainers”—and data points—perfectly suited to teaching their AI to appear more human.
For actors like T, it was a great opportunity too: a way to make good, easy money on the side, without having to cross the picket line.
“There aren’t really clear rules right now.”
“This is fully a research-based project,” the job posting said. It offered $150 per hour for at least two hours of work, and asserted that “your individual likeness will not be used for any commercial purposes.”
The actors may have assumed this meant that their faces and performances wouldn’t turn up in a TV show or movie, but the broad nature of what they signed makes it impossible to know the full implications for sure. In fact, in order to participate, they had to sign away certain rights “in perpetuity” for technologies and use cases that may not yet exist.
And while the job posting insisted that the project “does not qualify as struck work” (that is, work produced by employers against whom the union is striking), it nevertheless speaks to some of the strike’s core issues: how actors’ likenesses can be used, how actors should be compensated for that use, and what informed consent should look like in the age of AI.
“This isn’t a contract battle between a union and a company,” said Duncan Crabtree-Ireland, SAG-AFTRA’s chief negotiator, at a panel on AI in entertainment at San Diego Comic-Con this summer. “It’s existential.”
Many actors across the industry, particularly background actors (also known as extras), worry that AI—much like the models described in the emotion study—could be used to replace them, whether or not their exact faces are copied. And in this case, by providing the facial expressions that will teach AI to appear more human, study participants may in fact have been the ones inadvertently training their own potential replacements.
“Our studies have nothing to do with the strike,” Max Kalehoff, Realeyes’s vice president for growth and marketing, saidin an email. “The vast majority of our work is in evaluating the effectiveness of advertising for clients—which has nothing to do with actors and the entertainment industry except to gauge audience reaction.” The timing, he added, was “an unfortunate coincidence.” Meta did not respond to multiple requests for comment.
Given how technological advancements so often build upon one another, not to mention how quickly the field of artificial intelligence is evolving, experts point out that there’s only so much these companies can truly promise.
In addition to the job posting, MIT Technology Review has obtained and reviewed a copy of the data license agreement, and its potential implications are indeed vast. To put it bluntly: whether the actors who participated knew it or not, for as little as $300, they appear to have authorized Realeyes, Meta, and other parties of the two companies’ choosing to access and use not just their faces but also their expressions, and anything derived from them, almost however and whenever they want—as long as they do not reproduce any individual likenesses.
Some actors, like Jessica, who asked to be identified by just her first name, felt there was something “exploitative” about the project—both in the financial incentives for out-of-work actors and in the fight over AI and the use of an actor’s image.
Jessica, a New York–based background actor, says she has seen a growing number of listings for AI jobs over the past few years. “There aren’t really clear rules right now,” she says, “so I don’t know. Maybe … their intention [is] to get these images before the union signs a contract and sets them.”
Do you have any tips related to how AI is being used in the entertainment industry? Please reach out at tips@technologyreview.com or securely on Signal at 626.765.5489.
All this leaves actors, struggling after three months of limited to no work, primed to accept the terms from Realeyes and Meta—and, intentionally or not, to affect all actors, whether or not they personally choose to engage with AI.
“It’s hurt now or hurt later,” says Maurice Compte, an actor and SAG-AFTRA member who has had principal roles on shows like Narcos and Breaking Bad. After reviewing the job posting, he couldn’t help but see nefarious intent. Yes, he said, of course it’s beneficial to have work, but he sees it as beneficial “in the way that the Native Americans did when they took blankets from white settlers,” adding: “They were getting blankets out of it in a time of cold.”
Humans as data
Artificial intelligence is powered by data, and data, in turn, is provided by humans.
It is human labor that prepares, cleans, and annotates data to make it more understandable to machines; as MIT Technology Review has reported, for example, robot vacuums know to avoid running over dog poop because human data labelers have first clicked through and identified millions of images of pet waste—and other objects—inside homes.
When it comes to facial recognition, other biometric analysis, or generative AI models that aim to generate humans or human-like avatars, it is human faces, movements, and voices that serve as the data.
Initially, these models were powered by data scraped off the internet—including, on several occasions, private surveillance camera footage that was shared or sold without the knowledge of anyone being captured.
But as the need for higher-quality data has grown, alongside concerns about whether data is collected ethically and with proper consent, tech companies have progressed from “scraping data from publicly available sources” to “building data sets with professionals,” explains Julian Posada, an assistant professor at Yale University who studies platforms and labor. Or, at the very least, “with people who have been recruited, compensated, [and] signed [consent] forms.”
But the need for human data, especially in the entertainment industry, runs up against a significant concern in Hollywood: publicity rights, or “the right to control your use of your name and likeness,” according to Corynne McSherry, the legal director of the Electronic Frontier Foundation (EFF), a digital rights group.
This wasan issuelong before AI, but AI has amplified the concern. Generative AI in particular makes it easy to create realistic replicas of anyone by training algorithms on existing data, like photos and videos of the person. The more data that is available, the easier it is to create a realistic image. This has a particularly large effect on performers.
He believes it’s that improvisation requirement that explains why Realeyes and Meta were specifically recruiting actors.
Some actors have been able to monetize the characteristics that make them unique. James Earl Jones, the voice of Darth Vader, signed off on the use of archived recordings of his voice so that AI could continue to generate it for future Star Wars films. Meanwhile, de-aging AI has allowed Harrison Ford, Tom Hanks, and Robin Wright to portray younger versions of themselves on screen. Metaphysic AI, the company behind the de-aging technology, recently signed a deal with Creative Artists Agency to put generative AI to use for its artists.
But many deepfakes, or images of fake events created with deep-learning AI, are generated without consent. Earlier this month, Hanks posted on Instagram that an ad purporting to show him promoting a dental plan was not actually him.
The AI landscape is different for noncelebrities. Background actors are increasingly being asked to undergo digital body scans on set, where they have little power to push back or even get clarity on how those scans will be used in the future. Studios say that scans are used primarily to augment crowd scenes, which they have been doing with other technology in postproduction for years—but according to SAG representatives, once the studios have captured actors’ likenesses, they reserve the rights to use them forever. (There have already been multiple reports from voice actors that their voices have appeared in video games other than the ones they were hired for.)
In the case of the Realeyes and Meta study, it might be “study data” rather than body scans, but actors are dealing with the same uncertainty as to how else their digital likenesses could one day be used.
Teaching AI to appear more human
At $150 per hour, the Realeyes study paid far more than the roughly $200 daily rate in the current Screen Actors Guild contract (nonunion jobs pay even less).
This made the gig an attractive proposition for young actors like T, just starting out in Hollywood—a notoriously challenging environment even had he not arrived just before the SAG-AFTRA strike started. (T has not worked enough union jobs to officially join the union, though he hopes to one day.)
In fact, even more than a standard acting job, T described performing for Realeyes as “like an acting workshop where … you get a chance to work on your acting chops, which I thought helped me a little bit.”
For two hours, T responded to prompts like “Tell us something that makes you angry,” “Share a sad story,” or “Do a scary scene where you’re scared,” improvising an appropriate story or scene for each one. He believes it’s that improvisation requirement that explains why Realeyes and Meta were specifically recruiting actors.
In addition to wanting the pay, T participated in the study because, as he understood it, no one would see the results publicly. Rather, it was research for Meta, as he learned when he arrived at the studio space and signed a data license agreement with the company that he only skimmed through. It was the first he’d heard that Meta was even connected with the project. (He had previously signed a separate contract with Realeyes covering the terms of the job.)
The data license agreement says that Realeyes is the sole owner of the data and has full rights to “license, distribute, reproduce, modify, or otherwise create and use derivative works” generated from it, “irrevocably and in all formats and media existing now or in the future.”
This kind of legalese can be hard to parse, particularly when it deals with technology that is changing at such a rapid pace. But what it essentially means is that “you may be giving away things you didn’t realize … because those things didn’t exist yet,” says Emily Poler, a litigator who represents clients in disputes at the intersection of media, technology, and intellectual property.
“If I was a lawyer for an actor here, I would definitely be looking into whether one can knowingly waive rights where things don’t even exist yet,” she adds.
As Jessica argues, “Once they have your image, they can use it whenever and however.” She thinks that actors’ likenesses could be used in the same way that other artists’ works, like paintings, songs, and poetry, have been used to train generative AI, and she worries that the AI could just “create a composite that looks ‘human,’ like believable as human,” but “it wouldn’t be recognizable as you, so you can’t potentially sue them”—even if that AI-generated human was based on you.
This feels especially plausible to Jessica given her experience as an Asian-American background actor in an industry where representation often amounts to being the token minority. Now, she fears, anyone who hires actors could “recruit a few Asian people” and scan them to create “an Asian avatar” that they could use instead of “hiring one of you to be in a commercial.”
It’s not just images that actors should be worried about, says Adam Harvey, an applied researcher who focuses on computer vision, privacy, and surveillance and is one of the co-creators of Exposing.AI, which catalogues the data sets used to train facial recognition systems.
What constitutes “likeness,” he says, is changing. While the word is now understood primarily to mean a photographic likeness, musicians are challenging that definition to include vocal likenesses. Eventually, he believes, “it will also … be challenged on the emotional frontier”—that is, actors could argue that their microexpressions are unique and should be protected.
Realeyes’s Kalehoff did not say what specifically the company would be using the study results for, though he elaborated in an email that there could be “a variety of use cases, such as building better digital media experiences, in medical diagnoses (i.e. skin/muscle conditions), safety alertness detection, or robotic tools to support medical disorders related to recognition of facial expressions (like autism).”
Now, she fears, anyone who hires actors could “recruit a few Asian people” and scan them to create “an Asian avatar” that they could use instead of “hiring one of you to be in a commercial.”
When asked how Realeyes defined “likeness,” he replied that the company used that term—as well as “commercial,” another word for which there are assumed but no universally agreed-upon definitions—in a manner that is “the same for us as [a] general business.” He added, “We do not have a specific definition different from standard usage.”
But for T, and for other actors, “commercial” would typically mean appearing in some sort of advertisement or a TV spot—“something,” T says, “that’s directly sold to the consumer.”
Outside of the narrow understanding in the entertainment industry, the EFF’s McSherry questions what the company means: “It’s a commercial company doing commercial things.”
Kalehoff also said, “If a client would ask us to use such images [from the study], we would insist on 100% consent, fair pay for participants, and transparency. However, that is not our work or what we do.”
Yet this statement does not align with the language of the data license agreement, which stipulates that while Realeyes is the owner of the intellectual property stemming from the study data, Meta and “Meta parties acting on behalf of Meta” have broad rights to the data—including the rights to share and sell it. This means that, ultimately, how it’s used may be out of Realeyes’s hands.
As explained in the agreement, the rights of Meta and parties acting on its behalf also include:
Asserting certain rights to the participants’ identities (“identifying or recognizing you … creating a unique template of your face and/or voice … and/or protecting against impersonation and identity misuse”)
Allowing other researchers to conduct future research, using the study data however they see fit (“conducting future research studies and activities … in collaboration with third party researchers, who may further use the Study Data beyond the control of Meta”)
Creating derivative works from the study data for any kind of use at any time (“using, distributing, reproducing, publicly performing, publicly displaying, disclosing, and modifying or otherwise creating derivative works from the Study Data, worldwide, irrevocably and in perpetuity, and in all formats and media existing now or in the future”)
The only limit on use was that Meta and parties would “not use Study Data to develop machine learning models that generate your specific face or voice in any Meta product” (emphasis added). Still, the variety of possible use cases—and users—is sweeping. And the agreement does little to quell actors’ specific anxieties that “down the line, that database is used to generate a work and that work ends up seeming a lot like [someone’s] performance,” as McSherry puts it.
When I asked Kalehoff about the apparent gap between his comments and the agreement, he denied any discrepancy: “We believe there are no contradictions in any agreements, and we stand by our commitment to actors as stated in all of our agreements to fully protect their image and their privacy.” Kalehoff declined to comment on Realeyes’s work with clients, or to confirm that the study was in collaboration with Meta.
Meanwhile, Meta has been building photorealistic 3D “Codec avatars,” which go far beyond the cartoonish images in Horizon Worlds and require human training data to perfect. CEO Mark Zuckerberg recently described these avatars on the popular podcast from AI researcher Lex Fridman as core to his vision of the future—where physical, virtual, and augmented reality all coexist. He envisions the avatars “delivering a sense of presence as if you’re there together, no matter where you actually are in the world.”
Despite multiple requests for comment, Meta did not respond to any questions from MIT Technology Review, so we cannot confirm what it would use the data for, or who it means by “parties acting on its behalf.”
Individual choice, collective impact
Throughout the strikes by writers and actors, there has been a palpable sense that Hollywood is charging into a new frontier that will shape how we—all of us—engage with artificial intelligence. Usually, that frontier is described with reference to workers’ rights; the idea is that whatever happens here will affect workers in other industries who are grappling with what AI will mean for their own livelihoods.
Already, the gains won by the Writers Guild have provided a model for how to regulate AI’s impact on creative work. The union’s new contract with studios limits the use of AI in writers’ rooms and stipulates that only human authors can be credited on stories, which prevents studios from copyrighting AI-generated work and further serves as a major disincentive to use AI to write scripts.
In early October, the actors’ union and the studios also returned to the bargaining table, hoping to provide similar guidance for actors. But talks quickly broke down because “it is clear that the gap between the AMPTP [Alliance of Motion Picture and Television Producers] and SAG-AFTRA is too great,” as the studio alliance put it in a press release. Generative AI—specifically, how and when background actors should be expected to consent to body scanning—was reportedly one of the sticking points.
Whatever final agreement they come to won’t forbid the use of AI by studios—that was never the point. Even the actors who took issue with the AI training projects have more nuanced views about the use of the technology. “We’re not going to fully cut out AI,” acknowledges Compte, the Breaking Bad actor. Rather, we “just have to find ways that are going to benefit the larger picture… [It] is really about living wages.”
But a future agreement, which is specifically between the studios and SAG, will not be applicable to tech companies conducting “research” projects, like Meta and Realeyes. Technological advances created for one purpose—perhaps those that come out of a “research” study—will also have broader applications, in film and beyond.
“The likelihood that the technology that is developed is only used for that [audience engagement or Codec avatars] is vanishingly small. That’s not how it works,” says the EFF’s McSherry. For instance, while the data agreement for the emotion study does not explicitly mention using the results for facial recognition AI, McSherry believes that they could be used to improve any kind of AI involving human faces or expressions.
(Besides, emotion detection algorithms are themselves controversial, whether or not they even work the way developers say they do. Do we really want “our faces to be judged all the time [based] on whatever products we’re looking at?” asks Posada, the Yale professor.)
This all makes consent for these broad research studies even trickier: there’s no way for a participant to opt in or out of specific use cases. T, for one, would be happy if his participation meant better avatar options for virtual worlds, like those he uses with his Oculus—though he isn’t agreeing to that specifically.
But what are individual study participants—who may need the income—to do? What power do they really have in this situation? And what power do other people—even people who declined to participate—have to ensure that they are not affected? The decision to train AI may be an individual one, but the impact is not; it’s collective.
“Once they feed your image and … a certain amount of people’s images, they can create an endless variety of similar-looking people,” says Jessica. “It’s not infringing on your face, per se.” But maybe that’s the point: “They’re using your image without … being held liable for it.”
T has considered the possibility that, one day, the research he has contributed to could very wellreplace actors.
But at least for now, it’s a hypothetical.
“I’d be upset,” he acknowledges, “but at the same time, if it wasn’t me doing it, they’d probably figure out a different way—a sneakier way, without getting people’s consent.” Besides, T adds, “they paid really well.”
Do you have any tips related to how AI is being used in the entertainment industry? Please reach out at tips@technologyreview.com or securely on Signal at 626.765.5489.
Perplexity announced an update to its Copilot feature with GPT-3.5 Turbo fine-tuning from OpenAI and the introduction of Code Llama to Perplexity’s LLaMa Chat.
This means a more responsive and efficient AI-powered search experience without compromising quality.
For developers, it means access to Meta’s latest open-source large language model (LLM) for coding within 24 hours of its release.
Improving Speed And Accuracy With GPT-3.5 Turbo Fine-Tuning
According to Perplexity’s testing, the fine-tuned GPT-3.5 Turbo model is tied with the GPT-4-based model in human ranking.
Our fine-tuned GPT-3.5 model ties with the GPT-4-based model in human ranking on our task. This isn’t just about speed; it’s about delivering precise and accurate responses to your complex queries. Now you get the best of Perplexity at your finger tips. pic.twitter.com/jb86dZg91m
This could give Copilot users more confidence in the answers provided by Perplexity.
One of the notable improvements is the reduction in model latency by 4-5 times. This decreased the time it takes to deliver search results by almost 80%.
Faster response times could significantly improve user experience, especially for those who need almost instant answers to critical questions.
The transition to the fine-tuned GPT-3.5 model should also reduce inference costs – the computational expense of making predictions using a trained machine learning model.
The savings would allow Perplexity to invest in additional enhancements, ensuring that users continue to receive new features and better performance.
Perplexity users can also upload PDF files, focus search results on specific sources, and set up an AI profile to personalize search results.
Screenshot from Perplexity, August 2023
The AI profile is somewhat similar to the first portion of ChatGPT’s Custom Instructions.
Logged-in users can try Copilot free up to five times per hour. With a $20 monthly subscription, Perplexity Pro users can try Copilot 300 times per day with the option to switch to GPT-4.
Introducing Code Llama Intstruct To Perplexity Labs LlaMa Chat
In addition to the updates to Perplexity’s AI-assisted search, Perplexity announced Code Llama in Perplexity Labs LlaMa Chat. Code Llama is a coding LLM from Meta AI integrated into Perplexity’s LlaMa Chat to improve answers to technical questions.
Code LLaMA is now on Perplexity’s LLaMa Chat!
Try asking it to write a function for you, or explain a code snippet: 🔗 https://t.co/gyiDw6u6IJ
This is the fastest way to try @MetaAI’s latest code-specialized LLM. With our model deployment expertise, we are able to provide you… pic.twitter.com/hX90QulMz4
Users can access the open-source LLM and ask it to write a function or clarify a section of code. The Instruct model of Code Llama is specifically tuned to understand natural language prompts.
The quick addition of Code Llama to Perplexity Labs LlaMa Chat will allow developers to test its usefulness almost immediately after Meta announced its availability.
Screenshot from Perplexity Labs, August 2023
The Future Of AI Collaboration
The latest updates to Perplexity’s AI-powered search Copilot with a fine-tuned GPT-3.5 Turbo model and the introduction of Code Llama chat demonstrate how rapidly AI products improve when big tech companies collaborate.
Meta AI has unveiled its latest innovation in artificial intelligence (AI), Code Llama. Meta’s latest large language model (LLM) is poised to change how code is written, understood, and debugged.
Today we’re releasing Code Llama, a large language model built on top of Llama 2, fine-tuned for coding & state-of-the-art for publicly available coding tools.
Keeping with our open approach, Code Llama is publicly-available now for both research & commercial use.
With its specialized versions and promising benchmark performance, Code Llama offers a potential transformation in coding practices across various sectors.
What Is Code Llama?
Code Llama builds on the well-established framework of Llama 2 and offers three distinct models:
The foundational code model.
Codel Llama, a version explicitly fine-tuned for Python programming.
Code Llama – Instruct, designed to handle natural language instructions more effectively.
The model stands out for its coding capabilities and is trained to utilize popular programming languages like Python, C++, Java, PHP, and more.
Screenshot from Meta AI, August 2023
The LLM is available in three different sizes with 7B, 13B, and 34B parameters, catering to varying requirements concerning serving and latency.
While the 7B model is ideal for single GPU serving, the 34B model offers superior results and is more suitable for extensive coding assistance.
Risks And Responsibilities
In benchmark testing, Code Llama has demonstrated remarkable performance. Scoring 53.7% on HumanEval and 56.2% on Mostly Basic Python Programming (MBPP), it surpassed other open-source code-specific LLMs and matched the performance of models like ChatGPT.
While the technology could be valuable to developers across all experience levels and industries, Meta emphasized the importance of responsible AI usage.
The company has conducted extensive evaluations to assess the LLM’s risk of generating malicious code and provided guidelines. These guidelines align with a framework on responsible practices for synthetic media supported by Adobe, Google, Meta, Microsoft, OpenAI, and others in big tech.
The Future Of Generative AI For Coding
Code Llama symbolizes a significant advancement in AI-driven code generation and represents Meta’s commitment to an open approach to technology.
By making Code Llama free for commercial and research use, Meta helps more developers innovate safely in this rapidly evolving field.
Code Llama’s introduction could likely impact everyone in coding, from those entering the field to developers with decades of experience.
This new open-source LLM has the potential to streamline coding processes, aid in debugging, and enhance productivity, aligning with the growing demands of the tech industry.
The popularity of generative artificial intelligence (AI) tools like ChatGPT has led to an explosion of AI content, prompting new guidance from the Associated Press (AP).
The latest advice from AP explores how content marketers should focus on real-world implications without sensationalism when writing about AI and how AI tools fit into the content marketing process.
Understanding AI Systems
In an era where AI is becoming more prevalent across various industries, its influence has transformed how content is researched, crafted, published, promoted, and analyzed.
For content marketers to confidently use and explain AI technologies to their audiences, they must grasp the underlying concepts of AI development.
Because humans create the training data and develop AI models, AI tools have inherited biases.
Content marketers should recognize this when evaluating AI models, tools, and outputs.
Moreover, the content should explore how an AI can predict words and phrases with near human-level comprehension to help readers better understand the rapidly advancing technology.
It’s essential, however, to avoid using language that attributes human characteristics to AI systems. While an AI output may appear human-like, an AI system does not have the same thoughts and emotions.
Treating AI Content As Unvetted Source Material
Content writers and editors should exercise caution and skepticism when using generative AI throughout the content publishing process.
Understanding an AI system’s limitations is critical to creating transparent and reliable content for your audience.
AP stressed the responsibility of content publishers to treat AI-generated content as unvetted source material instead of publishable content.
Many AI models rely on outdated information, resulting in unreliable outputs. Content creators must verify the accuracy of anything generated by AI and encourage their readers to do the same.
Maintaining The Role Of Humans In Content Cration
Ultimately, AI is not intended to replace people in content creation. People remain vital in gathering, evaluating, and confirming facts, constantly focusing on accuracy and fairness.
AP hopes its latest advice helps content marketers efficiently utilize AI to share its potential and limitations with audiences through a better understanding of AI.
The addition of AI guidelines in the AP Stylebook comes a month after AP and OpenAI announced a partnership to explore use cases of generative AI for news publishers. It hopes AI tools will give journalists more time to do meaningful reporting.
NVIDIA hopes to open new doors for developing generative artificial intelligence (AI) models with AI Workbench.
The enterprise toolkit is intended to make AI development more streamlined, efficient, and accessible.
Its reported capabilities range from scaling models across any infrastructure, including PCs, workstations, data centers, and public clouds, to seamless collaboration and deployment.
The complexities involved in tuning, scaling, and deploying AI models may be eased by a unified platform, allowing developers to harness the full potential of AI for specific use cases.
Demonstrations at a recent event showed custom image generation with Stable Diffusion XL and a fine-tuned Llama 2 for medical reasoning developed using AI Workbench.
Challenges In Enterprise AI Development
Developing generative AI models involves multiple stages, each with challenges and demands.
From selecting a pre-trained model, such as a Large Language Model (LLM), developers often want to tune the model for specific applications.
This process requires an infrastructure that can handle various computing demands and seamlessly integrate with tools like GitHub, Hugging Face, NVIDIA NGC, and self-hosted servers.
The journey demands expertise in machine learning, data manipulation techniques, Python, and frameworks like TensorFlow.
The complexity of managing credentials, data access, and dependencies between components is added to that.
With the proliferation of sensitive data, security is paramount, demanding robust measures to ensure confidentiality and integrity.
On top of it all, managing workflows across different machines and platforms adds to the complexity.
AI Workbench Features
AI Workbench aims to simplify the development process by addressing these challenges with:
An easy-to-use development platform with tools like JupyterLab, VS Code, and services like GitHub.
A focus on transparency and reproducibility to foster enhanced collaboration across teams.
Client-server deployment to shift between local and remote resources, making the scaling process easier.
Customization across Text and Image Workflows
For enterprises looking to explore the powerful world of generative AI, it may be a crucial stepping stone in accelerating adoption and integration.
The Future Of Enterprise AI Development
The NVIDIA AI Workbench is particularly significant for enterprises as it offers to streamline the development process with new avenues for customization, scalability, and cost-effective solutions.
By addressing technical expertise, data security, and workflow management challenges, NVIDIA’s toolkit could become a game-changer for businesses harnessing AI for various applications.
