Most AI-powered robots today use cameras to understand their surroundings and learn new tasks, but it’s becoming easier to train robots with sound too, helping them adapt to tasks and environments where visibility is limited.
Though sight is important, there are daily tasks where sound is actually more helpful, like listening to onions sizzling on the stove to see if the pan is at the right temperature. Training robots with audio has only been done in highly controlled lab settings, however, and the techniques have lagged behind other fast robot-teaching methods.
Researchers at the Robotics and Embodied AI Lab at Stanford University set out to change that. They first built a system for collecting audio data, consisting of a GoPro camera and a gripper with a microphone designed to filter out background noise. Human demonstrators used the gripper for a variety of household tasks and then used this data to teach robotic arms how to execute the task on their own. The team’s new training algorithms help robots gather clues from audio signals to perform more effectively.
“Thus far, robots have been training on videos that are muted,” says Zeyi Liu, a PhD student at Stanford and lead author of the study. “But there is so much helpful data in audio.”
To test how much more successful a robot can be if it’s capable of “listening,” the researchers chose four tasks: flipping a bagel in a pan, erasing a whiteboard, putting two Velcro strips together, and pouring dice out of a cup. In each task, sounds provide clues that cameras or tactile sensors struggle with, like knowing if the eraser is properly contacting the whiteboard or whether the cup contains dice.
After demonstrating each task a couple of hundred times, the team compared the success rates of training with audio and training only with vision. The results, published in a paper on arXiv that has not been peer-reviewed, were promising. When using vision alone in the dice test, the robot could tell 27% of the time if there were dice in the cup, but that rose to 94% when sound was included.
It isn’t the first time audio has been used to train robots, says Shuran Song, the head of the lab that produced the study, but it’s a big step toward doing so at scale: “We are making it easier to use audio collected ‘in the wild,’ rather than being restricted to collecting it in the lab, which is more time consuming.”
The research signals that audio might become a more sought-after data source in the race to train robots with AI. Researchers are teaching robots faster than ever before using imitation learning, showing them hundreds of examples of tasks being done instead of hand-coding each one. If audio could be collected at scale using devices like the one in the study, it could give them an entirely new “sense,” helping them more quickly adapt to environments where visibility is limited or not useful.
“It’s safe to say that audio is the most understudied modality for sensing [in robots],” says Dmitry Berenson, associate professor of robotics at the University of Michigan, who was not involved in the study. That’s because the bulk of research on training robots to manipulate objects has been for industrial pick-and-place tasks, like sorting objects into bins. Those tasks don’t benefit much from sound, instead relying on tactile or visual sensors. But as robots broaden into tasks in homes, kitchens, and other environments, audio will become increasingly useful, Berenson says.
Consider a robot trying to find which bag or pocket contains a set of keys, all with limited visibility. “Maybe even before you touch the keys, you hear them kind of jangling,” Berenson says. “That’s a cue that the keys are in that pocket instead of others.”
Still, audio has limits. The team points out sound won’t be as useful with so-called soft or flexible objects like clothes, which don’t create as much usable audio. The robots also struggled with filtering out the audio of their own motor noises during tasks, since that noise was not present in the training data produced by humans. To fix it, the researchers needed to add robot sounds—whirs, hums, and actuator noises—into the training sets so the robots could learn to tune them out.
The next step, Liu says, is to see how much better the models can get with more data, which could mean adding more microphones, collecting spatial audio, and incorporating microphones into other types of data-collection devices.
This is today’s edition of The Download, our weekday newsletter that provides a daily dose of what’s going on in the world of technology.
A way to let robots learn by listening will make them more useful
Most AI-powered robots today use cameras to understand their surroundings and learn new tasks, but it’s becoming easier to train robots with sound too, helping them adapt to tasks and environments where visibility is limited.
Though sight is important, for some of our daily tasks, sound is actually more helpful, like listening to onions sizzling on the stove to see if the pan is at the right temperature. Training robots with audio has only been done in highly controlled lab settings, however, and the techniques have lagged behind other fast robot-teaching methods.
Researchers at the Robotics and Embodied AI Lab at Stanford University set out to change that. They first built a system for collecting audio data, then used this data to train robotic arms how to execute the task on their own. The team’s new training algorithms help robots gather clues from audio signals to perform more effectively. Read the full story.
—James O’Donnell
The must-reads
I’ve combed the internet to find you today’s most fun/important/scary/fascinating stories about technology.
1 Google’s emissions have skyrocketed in the past five years And those pesky AI products are to blame. (The Guardian)+ Microsoft has the same problem, too. (Bloomberg $) + AI is an energy hog. This is what it means for climate change. (MIT Technology Review)
2 The US wants to develop an mRNA vaccine to fight bird flu Moderna already started a vaccine trial last year targeting the main strains. (Ars Technica) + What’s next for bird flu vaccines. (MIT Technology Review)
3 An underground network is smuggling Nvidia chips into China Neatly swerving US export restrictions. (WSJ $) + What’s next in chips. (MIT Technology Review)
4 Fiverr freelancers will dox anyone for a price Some charge as little as $30. (404 Media)
5 OnlyFans’ paywalls are impeding abuse investigations Forensic consultants are increasingly frustrated by its infrastructure. (Reuters)
6 US universities are offering degrees in climate change Many of them focus on developing practical climate solutions. (Fast Company $) + How fish-safe hydropower technology could keep more renewables on the grid. (MIT Technology Review)
7 SpaceX has grand ambitions to launch 120 times a year Unsurprisingly, its competitors aren’t happy about its plans for Florida. (TechCrunch)
8 Scientists are using radio-frequency tech to find dinosaur bones The key card technology helps them to tag and recover fossils. (IEEE Spectrum)
9 The strange allure of a pointless website One Million Checkboxes is exactly that—one million boxes to check. (WP $) + How to fix the internet. (MIT Technology Review)
10 Craigslist will never die The classified listings site looks the same as it did almost 30 years ago. (Slate $)
Quote of the day
“When you chew enough glass, you learn to like the taste of your own blood.”
—A speaker at VC firm Andreessen Horowitz’s crypto startup bootcamp shares some intense insights into enduring hardship, Wired reports.
The big story
Social media is polluting society. Moderation alone won’t fix the problem
August 2022
We all want to be able to speak our minds online—to be heard by our friends and talk (back) to our opponents. At the same time, we don’t want to be exposed to unpleasant speech.
Technology companies address this conundrum by setting standards for free speech, a practice protected under federal law, hiring in-house moderators to examine individual pieces of content and removing them if posts violate predefined rules.
The approach clearly has problems: harassment, misinformation about topics like public health, and false descriptions of legitimate elections run rampant. But even if content moderation were implemented perfectly, it would still miss a whole host of issues. We need a new strategy: treat social media companies as potential polluters of the social fabric, and directly measure and mitigate the effects their choices have on us. Read the full story.
—Nathaniel Lubin & Thomas Krendl Gilbert
We can still have nice things
A place for comfort, fun and distraction to brighten up your day. (Got any ideas? Drop me a line or tweet ’em at me.)
+ Salads don’t have to be boring—especially when there’s boiled eggs and salmon involved. + Whac-a-mole has a surprisingly long and checkered past. + Here’s how to rehydrate quickly if you’re feeling the heat. + Phones are crazy expensive. But you can make them last longer if you’re smart about it.
Less than 1% of clothing is recycled, and most of the rest ends up dumped in a landfill or burned. A team of researchers hopes to change that with a new process that breaks down mixed-fiber clothing into reusable, recyclable parts without any sorting or separation in advance.
“We need a better way to recycle modern garments that are complex, because we are never going to stop buying clothes,” says Erha Andini, a chemical engineer at the University of Delaware and lead author of a study on the process, which is out today in Science Advances. “We are looking to create a closed-loop system for textile recycling.”
Many garments are made of a mix of natural and synthetic fibers. Once these fibers are combined, they are difficult to separate. This presents a problem for recycling, which often needs textiles to be sorted into uniform categories, similar to how we sort glass, aluminum, and paper.
To tackle this problem, Andini and her team used a solvent that breaks the chemical bonds in polyester fabric while leaving cotton and nylon intact. To speed up the process, they power it with microwave energy and add a zinc oxide catalyst. This combination reduces the breakdown time to 15 minutes, whereas traditional plastic recycling methods take over an hour. What the polyester ultimately breaks down into is BHET, an organic compound that can, in theory, be turned into polyester once more. While similar methods have been used to recycle pre-sorted plastic, this is the first time they’ve been used to recycle mixed-fiber textiles without any sorting required.
COURTESY OF THE RESEARCHERS
In addition to speeding things up, the use of microwave energy also reduces the technique’s carbon footprint because it’s quicker and uses less energy, says Andini.
Nevertheless, the process could be difficult to scale, says Bryan Vogt, a chemical engineer at Penn State University, who was not involved in the study. That’s because the solvent used to break down polyester is expensive and difficult to recover after use. Further, according to Andini, even though BHET is easily turned back into clothing, it’s less clear what to do with the leftover fibers. Nylon could be especially tricky, as the fabric is degraded significantly by the team’s chemical recycling technique.
“We are chemical engineers, so we think of this process as a whole,” says Andini. “Hopefully, once we are able to get pure components from each part, we can transform them back into yarn and make clothes again.”
Andini, who just received a fellowship for entrepreneurs, is developing a business plan to commercialize the process. In the coming years, she aims to launch a startup that will take the clothes recycling technique out of the lab and into the real world. That could be a significant step toward reducing the large amounts of textile waste in landfills. “It’ll be a matter of having the capital or not,” she says, “but we’re working on it and excited for it.”
Over a third of workers globally have seen a significant role change in the past year. That’s according to PwC’s fifth annual “Global Workforce Hopes and Fears Survey,” released last week.
In March 2024, PwC surveyed 56,600 individuals across 50 countries and territories who are employed or actively seeking work.
Forty-five percent of respondents stated they had to learn new tools and technology in the previous 12 months to perform their job.
–
Per the survey, CEOs value technology benefits more than employees.
–
Most survey respondents had used generative AI (such as ChatGPT and DALL-E) at least once in the past year, although very few did so routinely.
–
Moreover, nearly all of gen AI’s daily users expect advantages while recognizing possible difficulties.
In a recent Instagram Reel, Adam Mosseri, the head of Instagram, revealed a top signal the platform uses to rank content: sends per reach.
This metric measures the number of people who share a post with friends through direct messages (DMs) relative to the total number of viewers.
Mosseri advises creating content people want to share directly with close friends and family, saying it can improve your reach over time.
This insight helps demystify Instagram’s ranking algorithms and can assist your efforts to improve visibility on the platform.
Instagram’s ‘Sends Per Reach’ Ranking Signal
Mosseri describes the sends per reach ranking signal and its reasoning:
“Some advice: One of the most important signals we use in ranking is sends per reach. So out of all the people who saw your video or photo, how many of them sent it to a friend in a DM? At Instagram we’re trying to be a place where people can be creative, but in a way that brings people together.
We want to not only be a place where you passively consume content, but where you discover things you want to tell your friends about.
A reel that made you laugh so hard you want to send it to your brother or sister. Or a soccer highlight that blew your mind and you want to send it to another fan. That kind of thing.
So, don’t force it as a creator. But if you can, think about making content that people would want to send to a friend, or to someone they care about.”
The emphasis on sends as a ranking factor aligns with Instagram’s desire to become a platform where users discover and share content that resonates with them personally.
Advice For Creators
While encouraging creators to produce shareworthy content, Mosseri cautioned against forced attempts to game the system.
However, prompting users to share photos and videos via DM is said to boost reach
What Does This Mean For You?
Getting people to share posts and reels with friends can improve reach, resulting in more engagement and leads.
Content creators and businesses can use this information to refine their Instagram strategies.
Rather than seeing Instagram’s focus on shareable content as an obstacle, consider it an opportunity to experiment with new approaches.
If your reach has been declining lately, and you can’t figure out why, this may be the factor that brings it back up.
In an interview published on YouTube, Google’s Gary Illyes offered advice on what small sites should consider doing if they want to compete against Reddit, Amazon and other big brand websites.
About Big Brand Dominance
Google’s Gary Illyes answered questions about SEO back in May that went underreported so I’m correcting that oversight this month. Gary answered a question about how to compete against Reddit and big brands.
While it may appear that Gary is skeptical that Reddit is dominating, he’s not disputing that perception and that’s not the context of his answer. The context is larger than Reddit because his answer is about the core issue of competing against big brands in the search engine results pages (SERPs).
This is the question that an audience member asked:
“Since Reddit and big publishers dominate nowadays in the SERPS for many keywords, what can the smaller brands do besides targeting the long tail keywords?”
The History Of Big Brands In The SERPs
Gary’s answer encompasses the entire history of big brands in the SERPs and the SEO response to that. About.com was a website about virtually any topic of interest and it used to rank for just about everything. It was like the Wikipedia of its day and many SEOs resented how About.com used to rank so well.
He first puts that context into his answer, that this complaint about Reddit is part of a long history of various brands ranking at the top of the SERPs then washing out of the SERPs as trends change.
Gary answered:
“So before I joined Google I was doing some SEO stuff for big publishers. …SEO type. Like I was also server manager like a cluster manager.
So, I would have had the same questions and in fact back in the day we saw these kind of questions all the time.
Now it’s Reddit. Back then it was Amazon. A few years before that, it was I think …About.com.
Pretty much every two years the name that you would put there …changes.”
Small Sites Can Outcompete Big Brands
Gary next shares that the history of SEO is also about small sites figuring out how to outcompete the bigger sites. This is also true. Some big sites started as small sites that figured out a way to outcompete larger big brand sites. For example, Reviewed.com, before it was purchased by USA Today, was literally started by a child whose passion for the topic contributed to it becoming massively successful.
Gary says that there are two things to do:
Wait until someone else figures out how to outcompete and then copy them
Or figure it out yourself and lead the way
But of course, if you wait for someone else to show the way it’s probably too late.
He continued:
“It seems that people always figure out ways to compete with whoever would be the second word in that question.
So it’s not like, oh my God, like everything sucks now and we can retire. It’s like, one thing you could do is to wait it out and let someone else come up with something for you that you can use to compete with Reddit and the big publishers that allegedly dominate nowadays the SERPs.
Or you sit down and you start thinking about how can you employ some marketing strategies that will boost you to around the same positions as the big publishers and Reddit and whatnot.
One of the most inspiring presentations I’ve seen was the empathetic marketing… do that. Find a way to compete with these positions in the SERPs because it is possible, you just have to find the the angle to compete with them.”
Gary is right. Big brands are slowed down by bureaucracy and scared to take chances. As I mentioned about Reviewed.com, a good strategy can outrun the big brands all day long, I know this from my own experience and from knowing others who have done the same thing, including the founder of Reviewed.com.
Long Tail Keywords & Other Strategies
Gary next talked about long tail keywords. A lot of newbie SEO gurus define long tail keyword phrases with a lot of words in it. That’s 100% wrong. Long tail keyword phrases are keyword phrases that searches rarely use. It’s the rareness of keyword use that makes them long tail, not how many words are in the keyword phrase.
The context this part of Gary’s answer is that the person asking the question essentially dismissed long tail search queries as the crumbs that the big brands leave behind for small sites.
Gary explains:
“And also the other thing is that, like saying that you are left with the long tail keywords. It’s like we see like 15 to even more percent of new long tail keywords every single day.
There’s lots of traffic in long tail keywords. You you can jump on that bandwagon and capture a ton of traffic.”
Something left unmentioned is that conquering long tail keyword phrases is one way to create awareness that a site is about a topic. People come for the long tail and return for the head phrases (the queries with more traffic).
The problem with some small sites is that they’re trying to hit the big traffic keywords without first showing relevance in the long tail. Starting small and building up toward big is one of the secrets of successful sites.
Small Sites Can Be Powerful
Gary is right, there is a lot of traffic in the long tail and emerging trends. The thing that small sites need to remember is that big sites move slow and have to get through layers of bureaucracy in order to make a strategic decision. The stakes for them are also higher so they’re not prone to take big swings either. Speed and the ability to make bold moves is the small site’s super power. Exercise it.
I know from my own experience and from working with clients that it’s absolutely possible to outrank to big sites that have been around for years. The history of SEO is littered with small sites that outpaced the slower moving bigger sites.
Watch Gary answer this question at the 20 minute mark:
Featured Image by Shutterstock/Volodymyr TVERDOKHLIB
AI models can generate astonishingly creative content. However, their outputs can become cliched, unpredictable, and problematic without proper guardrails. How can we harness their potential while maintaining control? In this article, we’ll show you what you can do to provide guardrails for your AI chatbot. Thanks to these techniques, you can ensure its creative outputs align with your specific needs and objectives.
Table of contents
Understanding the need for guardrails
As AI continues to evolve, so do its capabilities to generate creative content. Generative AI can do everything, from writing articles and creating marketing copy to composing music and generating artwork. However, this comes with great responsibilities. Unchecked creativity in AI can lead to various challenges and risks. It’s very important to implement guardrails.
What is AI creativity?
Generative AI refers to the ability of models to generate new content. This can include text, images, music, and other forms of media. AI models like GPT-4, for instance, can write poetry, draft emails, create fictional stories, and even generate code. At Yoast, we use it to power the AI title and meta description generator in Yoast SEO. There are various ways to determine how creative the chatbot or AI system can get while generating that content. For instance, various AI tools like Copilot and Gemini have options to make the output more or less adventurous.
Where AI gets its creativity from
AI models, particularly Large Language Models (LLMs) like GPT-4, exhibit creativity through their ability to generate content. But where does this creativity come from? The answer lies at the intersection of training data, deep learning architectures, and fine-tuned parameters.
Diverse training data
The foundation of AI creativity is the huge datasets used during training. These datasets contain a range of text sources, including books, articles, websites, and other forms of written content. Exposure to a wide variety helps the model learn patterns, styles, and contextual nuances across different genres and topics. Diversity helps AI generate content that is not only coherent but also varied and imaginative.
Deep neural networks
At the heart of LLMs are deep neural networks, specifically transformer architectures. These consist of multiple layers of attention mechanisms. These layers allow the model to understand and generate complex language structures by focusing on the relationships between words and their context. With billions of parameters fine-tuned during training, these models can produce human-like text that mirrors the creativity found in their training data.
Predictive text generation
LLMs’ predictive text generation capabilities also drive creativity. The models generate text one token (word or subword) at a time, predicting the next token based on the preceding context. This token-by-token generation, influenced by probability distributions, allows the AI to craft coherent and contextually relevant content that can surprise and engage readers.
Influence of parameters
Parameters like temperature and top_p are crucial in modulating the model’s output. Temperature controls the randomness of predictions, with higher values leading to more diverse and “creative” outputs, while lower values result in more deterministic and focused text. Top_p, or nucleus sampling, controls the diversity of the output by sampling from a subset of probable tokens. By fine-tuning these parameters, users can balance creativity with coherence — more on this later. These are helpful tools to guide the AI in producing content that meets your needs.
Pattern recognition and replication
Ultimately, the AI’s creativity stems from its ability to recognize and replicate patterns from its training data. By mimicking the linguistic and stylistic patterns it has learned, the model can generate content that feels original and inspired. This pattern recognition allows LLMs to compose poetry, write stories, create marketing copy, and generate artistic descriptions that resonate with human creativity.
AI creativity is a product of training on diverse datasets, neural network architectures, and calibrated parameters. Understanding these components helps harness AI’s creativity while ensuring the content aligns with your objectives.
Human creativity vs. AI creativity
Various forms of creativity often produce similar outputs but from very different backgrounds. Human creativity is rooted in personal experiences, emotions, and conscious thought. This allows people to create art, literature, and innovations that resonate emotionally and culturally. It involves intuition, inspiration, and the ability to make abstract connections that are uniquely human.
In contrast, AI creativity consists of processing data and recognizing patterns within that data. AI generates new content based on learned patterns and statistical probabilities, not personal experiences or emotions. While AI can mimic human creativity and make coherent and relevant content, it lacks human understanding and emotional depth. Fusing human and AI creativity can lead to interesting results, but it’s crucial to recognize and appreciate each’s distinct nature.
Letting the AI run wild
While AI’s creative capabilities are impressive, they come with inherent risks. With proper guardrails, the outputs can become predictable and manageable.
AI can produce off-topic, irrelevant, or even inappropriate content without proper constraints. As a result, businesses and content creators might get hurt. For instance, an AI writing tool might generate marketing copy that is in the wrong tone or even offensive, which can damage a brand’s reputation.
Controlled creativity can generate content that aligns differently with the brand’s voice or message. The end goal, of course, is clarity and consistency.
Guardrails are critical for generative AI
Given these risks, it’s clear that guardrails help control AI’s creative potential. Here’s why guardrails are crucial:
Maintaining relevance and focus:
Guardrails help keep the AI’s outputs focused on the intended topic, preventing deviations that can dilute the message.
Ensuring appropriateness:
Guardrails protect your brand’s reputation and ensure that the content suits your audience by filtering out inappropriate or offensive content.
Aligning with brand voice:
Guardrails ensure that AI-generated content is consistent with your brand’s voice and tone, maintaining coherence in your messaging.
Enhancing credibility:
By preventing factual inaccuracies, guardrails enhance the credibility and reliability of AI-generated content, especially in fields that require precision.
Optimizing user experience:
Well-implemented guardrails contribute to a better user experience by ensuring the content is engaging, relevant, and valuable to the audience.
The following sections will explore practical techniques for providing these guardrails to manage AI creativity effectively.
Techniques for providing guardrails
Effective guardrails for AI are strategies that can help control the output, ensuring it meets specific requirements and aligns with your objectives.
Keyword filtering
Without limiting what the LLM does, it loves to come up with sentences/words like: “In the ever-evolving landscape of…” and “As we stand on the cusp of this new era, the possibilities are as limitless as our imagination.” It uses long-winded sentences with very expressive language, full of cliches. You can curb this by limiting the words or expressions it can use.
Keyword filtering involves setting up filters to exclude specific words, phrases, or types of content deemed inappropriate, irrelevant, or not aligned with your brand’s voice. This technique is useful for maintaining content suitability and relevance.
It’s not hard to implement:
Identify keywords: List words or phrases that should be excluded. This can include offensive language, jargon, or off-topic terms.
Set up filters: Use AI tools that support keyword filtering. Configure these tools to flag or exclude content containing the identified keywords.
Continuous monitoring: Regularly update the list of keywords based on feedback and new requirements.
Try this as an experiment. You’ll notice it’s fairly easy to influence what chatbots use and don’t use.
Write a short piece on the future of content creation with generative AI. Don't use the following words:
Buckle up
Delve
Dive
Elevate
Embark
Embrace
Explore
Discover
Demystified
but do use:
Unleash
Unlocked
Unveiled
Beacon
Bombastic
Competitive digital world
You can also make this process more proficient and scalable using APIs to communicate with LLMs and chatbots.
Prompt engineering
Prompt engineering involves writing prompts to guide the AI in generating content that meets the criteria. Leo S. Lo from the University of New Mexico developed the CLEAR method (context, limitations, examples, audience, requirements), an effective approach to prompt engineering. Of course, there are plenty of other ways to write great prompts for your content.
A practical example of using the CLEAR framework
Imagine we are creating content for a travel blog. Using the CLEAR framework, we devised the following prompt to inspire the AI chatbot to create a blog post about Kyoto, Japan.
Prompt:“Describe a day in the life of a local in Kyoto, Japan. Focus on their morning routine, interactions with neighbors, and favorite spots in the city. Use a descriptive and engaging tone to captivate travel enthusiasts. Include at least two historical landmarks and one local cuisine.”
Clear: The instructions are straightforward to understand. We specifically ask for a description of a day in the life of a local in Kyoto, including particular elements like their morning routine, interactions, and favorite spots.
Logical: The prompt is logically structured. It begins with a general description of a day in the life and then narrows down to specific details such as the morning routine, interactions with neighbors, and favorite spots. This logical flow helps generate a coherent and comprehensive piece of content.
Engaging: The tone is described as “descriptive and engaging,” which is crucial for captivating travel enthusiasts. The prompt invites the writer to create a vivid and relatable narrative by focusing on personal interactions and favorite spots.
Accurate: The prompt asks for at least two historical landmarks and one local cuisine. This ensures that the description is rooted in Kyoto’s actual cultural and historical elements.
Relevant: The topic is highly relevant to travel enthusiasts interested in different places’ cultural and daily life aspects. The prompt taps into a subject of high interest by focusing on Kyoto, a city known for its rich history and cultural landmarks.
Enhanced prompt
To refine it even further, you can add a few more specific guidelines to enhance clarity and completeness:
“Describe a day in the life of a local in Kyoto, Japan. Focus on their morning routine, interactions with neighbors, and favorite spots in the city. Use a descriptive and engaging tone to captivate travel enthusiasts. Include at least two historical landmarks (e.g., Kinkaku-ji, Fushimi Inari Taisha) and one local cuisine (e.g., yudofu, kaiseki). Ensure the narrative captures the essence of Kyoto’s culture and daily life.”
Why these improvements work:
Clear: Specific examples such as Kinkaku-ji and yudofu provide clarity.
Logical: The flow from morning routine to interactions and favorite spots remains logical.
Engaging: The descriptive and engaging tone is maintained.
Accurate: Named landmarks and cuisines ensure accuracy.
Relevant: Provides a detailed, culturally rich experience relevant to travel enthusiasts.
Now, the prompt is well-crafted and aligns with the CLEAR framework, and the enhanced version provides additional guidance and specificity.
Template usage
Templates provide a structured framework the AI chatbot can follow, ensuring consistency and completeness in the generated content. Templates can be particularly useful for recurring content types like blog posts, reports, product descriptions, etc. Using templates, you can maintain a uniform structure across different pieces of content. As a result, all necessary elements are included and appropriately organized.
Identify common content types: Determine the types of content you frequently generate, such as blog posts, product descriptions, social media posts, etc.
Create templates: Develop templates for each content type. These templates should include sections and prompts for each part of the content.
Provide clear instructions: Include detailed instructions within each template section to guide the AI. This can involve specifying the tone, style, length, and key points to cover.
Consistent use: Use these templates consistently to maintain uniformity across all generated content. Review and update the templates regularly to reflect new requirements or insights.
Parameter tuning
Adjusting parameters like temperature and top_p can control the randomness and creativity of the AI’s output. This might seem like it controls creativity, but that’s not actually the case. Instead, it fine-tunes how the model balances creativity with coherence. Temperature affects the variability of the generated content, while top_p controls the diversity by sampling from a subset of probable tokens.
Understanding temperature and top_p in LLMs
Imagine you’re baking cookies, and you want to experiment with different flavors. You have a big jar of various ingredients (chocolate chips, nuts, dried fruits, etc.), and you can either stick to the classic recipe or get a bit adventurous.
Temperature: Think of temperature as the level of adventurousness in your cookie recipe.
Low temperature (e.g., 0.2): You’re playing it safe. You mostly stick to the classic ingredients like chocolate chips and maybe a few nuts. Your cookies are predictable but reliably good.
High temperature (e.g., 0.8): You’re feeling adventurous! You start throwing in various ingredients, like mango bits, chili flakes, and marshmallows. The cookies are more unpredictable — some might be amazing, while others might be too wild.
In AI text generation, a lower temperature means the model plays it safe and chooses more predictable words. A higher temperature allows for more creativity and variety but with the risk of less coherence.
Top_p (Nucleus sampling): Now, imagine you have a friend who helps you pick the ingredients. Top_p is like telling your friend only to consider the most popular ingredients but with a twist.
Low top_p (e.g., 0.1): Your friend only picks the top 10% of frequently used ingredients. You end up with a very standard and safe mix.
High top_p (e.g., 0.9): Your friend considers a wider variety of ingredients, maybe the top 90%. This allows for more interesting and diverse combinations but still within a reasonable limit, so the cookies don’t turn out too strange.
In AI text generation, a lower top_p value means the model selects from a smaller set of high-probability words. This makes the output more predictable. A higher top_p value lets the model choose from a larger set of words, increasing the output’s diversity and “creativity” while maintaining coherence.
Adjusting temperature and top_p controls how adventurous or safe the AI is in generating text. This is much like how you control the ingredients in your cookie recipe.
A misconception
As we’ve mentioned, the temperature and top_p control the randomness and diversity of AI-generated text. However, they do not create or improve creativity. Instead, they manage how the AI explores different word choices. True creativity in AI comes from the model’s ability to generate new content based on the patterns it has learned from its training data.
Experimenting with and fine-tuning these parameters helps you guide the AI. These tools help it produce imaginative and relevant content without veering off into incoherence or irrelevance.
Generative AI tools like TypingMind let you carefully control the performance of various language models
Combining techniques
Combining the above techniques can provide a more robust framework for controlling AI creativity. Each technique complements the others, creating a comprehensive system of guardrails.
An integrated approach combines keyword filtering, prompt engineering, template usage, and parameter tuning to create a multi-layered control system. You can support this using a feedback loop that considers all aspects of the content generation process, from initial prompts to final outputs.
Conclusion to creativity in AI
It’s important to maintain control while still harnessing AI’s creative potential. Use guardrails such as keyword filtering, prompt engineering with frameworks, template usage, and parameter tuning to help the AI produce relevant, high-quality content that aligns with your objectives.
Remember that parameters like temperature and top_p do not define creativity; they merely influence the randomness and diversity of the output. True creativity in AI is limited and cannot be replicated without outside help from real people.
With some help from these techniques, we can purposefully use generative AI’s creative capabilities. Whether generating blog posts, marketing copy, or educational content, these strategies help the AI to add value and meet desired standards.
Back in May Google’s Gary Illyes sat for an interview at the SERP Conf 2024 conference in Bulgaria and answered a question about the causes of crawled but not indexed, offering multiple reasons that are helpful for debugging and fixing this error.
Although the interview happened in May, the video of the interview went underreported and not many people have actually watched it. I only heard of it because the always awesome Olesia Korobka (@Giridja) recently drew attention to the interview in a Facebook post.
So even though the interview happened in May, the information is still timely and useful.
Reason For Crawled – Currently Not Indexed
Crawled Currently Not Indexed is a reference to an error report in the Google Search Console Page Indexing report which alerts that a page was crawled by Google but was not indexed.
During a live interview someone submitted a question, asking:
“Can crawled but not indexed be a result of a page being too similar to other stuff already indexed?
So is Google suggesting there is enough other stuff already and your stuff is not unique enough?”
Google’s search console documentation doesn’t provide an answer as to why Google may crawl a page and not index it, so it’s a legitimate question.
Gary Illyes answered that yes, one of the reasons could be that there is already other content that is similar. But he also goes on to say that there are other reasons, too.
He answered:
“Yeah, that that could be one thing that it can mean. Crawled but not indexed is, ideally we would break up that category into more granular chunks, but it’s super hard because of how the data internally exists.
It can be a bunch of things, dupe elimination is one of those things, where we crawl the page and then we decide to not index it because there’s already a version of that or an extremely similar version of that content available in our index and it has better signals.
But yeah, but it it can be multiple things.”
General Quality Of Site Can Impact Indexing
Gary then called attention to another reason why Google might crawl but choose not to index a site, saying that it could be a site quality issue.
Illyes then continued his answer:
“And the general quality of the of the site, that can matter a lot of how many of these crawled but not indexed you see in search console. If the number of these URLs is very high that could hint at general quality issues.
And I’ve seen that a lot since February, where suddenly we just decided that we are indexing a vast amount of URLs on a site just because …our perception of the site has changed.”
Other Reasons For Crawled Not Indexed
Gary next offered other reasons for why URLs might be crawled but not indexed, saying that it could be that Google’s perception of the site could have changed but that it could be a technical issue.
Gary explained:
“…And one possibility is that when you see that number rising, that the perception of… Google’s perception of the site has changed, that could be one thing.
But then there could also be that there was an error, for example on the site and then it served the same exact page to every single URL on the site. That could also be one of the reasons that you see that number climbing.
So yeah, there could be many things.”
Takeaways
Gary provided answers that should help debug why a web page might be crawled but not indexed by Google.
Content is similar to content already ranked in the search engine results pages (SERPs)
Exact same content exists on another site that has better signals
General site quality issues
Technical issues
Although Illyes didn’t elaborate on what he meant about another site with better signals, I’m fairly certain that he’s describing the scenario when a site syndicates its content to another site and Google chooses to rank the other site for the content and not the original publisher.
Watch Gary answer this question at the 9 minute mark of the recorded interview:
Generative AI, SGE, and now AI Overviews have been hot topics since the launch of ChatGPT in November 2022, which gave Gen AI an accessible interface to the wide market.
Since then, the SEO industry has been trying to figure out just how much search behavior will change and how much this will impact organic search traffic.
Will we see the catastrophic drops in clicks that are being estimated?
Google’s aim is to integrate Gen AI into search to provide better answers and, in its words:
Sometimes you want a quick answer, but you don’t have time to piece together all the information you need. Search will do the work for you with AI Overviews.
However, there has been much contention and discussion about this as, in practice, the results are somewhat unpredictable – with advice, such as the health benefits of running with scissors, taking a bath with a toaster, and adding glue to pizza to make the cheese stick.
Google is still experimenting with AIO. Recently (June 19), a study from SE Ranking showed the frequency of AIO in SERPs has reduced from 64% to 8%. Meanwhile, BrightEdge reports that Google went from showing AI on 84% of queries to 15%.
Google also keeps experimenting with how AIO results appear in SERPs, and the latest iteration features citations in the top carousel.
Gen AI is disrupting the industry faster than anything else in the 25-year history of SEO. Some of the main discussion points for SEO include: How much is AI plagiarizing content, and how much do we need to pivot our approach to SEO?
In Your Opinion, What Do You Think About AI Overviews, How Will They Impact The Industry, And Where Is This Going?
This is what Pedro had to say:
“As I have mentioned before, Google wants to be your personal assistant and not your friendly librarian.
This is an important distinction, to see Google from this perspective moving forward. Instead of pointing us to the books, they will do the work for us.
If we continue to put out content that only requires a quick answer, this is where we will be disrupted. We need to focus on what people want beyond quick answers.
Google wants to be the personal assistant and caters for this by providing quick answers.
AI Overviews is no more than an evolution of instant answers.
If a site owner wants to target the quick answers, they should also be putting effort into more in-depth content that you can funnel your readers to and ideally, is closed to Google.
By doing this, you can protect the content assets you build.
We need to focus more than ever on building our own communities with users aligned to our brands. And doing more than simply providing a ‘this will do’ snippet, or an instant answer.
Right now, it’s impossible to predict how AIO will develop and what the format will be. Google keeps changing how it is presenting the SERP results and playing with the format much like live beta testing.
But, AIO will trigger different search behaviors.
Before, in SEO, we had ten blue links and no instant answers. From this, users would have to visit your website to get the answer, so a site could get considerable traffic for a basic question.
However, this type of traffic has little value, and these are not your customers – they are Google’s customers.
We need to understand how we can distinguish between instant answer traffic and users who want to consume our content. And this is the area where we should put our efforts.
Focus on building content for the people who don’t want the summary or the quick answer. Those who want to ‘read the book’ and consume the details to augment their knowledge.
In the same way that the web disrupted the music industry and the publisher industry, we are about to go through another change and we have to adapt to it. It’s a matter of time – when and not if.”
How Can You Leverage AIO And Google To Build A Content Community?
I asked Pedro:
“If we want to embrace this new approach, it will require thinking about how to gain users from a ‘take all the traffic you can get’ mentality to a selective one – leveraging Google to provide targeted traffic that you can absorb into your own community.
This will be a big change for some, so how can you leverage Google to achieve this?”
Pedro responded:
“Trying to figure out how much ‘discovery’ traffic Google will take away will be different for all verticals. For example, in the legal industry, or accountancy, the industry is based on consultants who understand and are gatekeepers to complex rules.
You can now ask AI to explain complex legislation on wider topics. But, if you have a specific scenario, you still need to visit a specialist who can deal with this for you.
AI can give you the wider information, but the expert is still needed for the detail.
As professionals in SEO, we can create content that covers broad concepts that AI can tap into. And then, for the specific scenarios and questions, this is where we can build out much more in-depth content.
This in-depth content is kept away from Google and AI and gated for your community or clients.
Every business will need to consider where to draw this line of what they give away for free and what they keep back as a premium.
AI came along to create more distance between those who know something and those who are specialists and will be sources of information.
The middle ground is about to disappear.
The professionals will remain because industries rely on the knowledge and the research these people do. And the rest will just be the rest.
Users will be divided into those who want a little information from AI and then the others who want specialist in-depth knowledge.
Being able to discern where you fit into this scenario and being able to create a strategy around this is how you can adapt.”
Fundamental Rules Never Change
I think we can expect more experimentation from Google before we begin to embrace AI in SERPs and SEO.
During a time of great flux, the best thing we can focus on is the fundamental rules that never change. And those fundamentals are all centered around how a brand builds a direct relationship with their user.
For SEO pros, it could be a challenging shift to adapt to this mindset away from chasing volume keyword traffic. Instead, looking at building user journeys and considering content touches where relevant.
The old days of gaining huge amounts of traffic for ranking from one high-volume keyword are becoming outdated. Moving forward, more effort will be needed to achieve far fewer clicks. However, those clicks should be far more relevant and beneficial.
Thank you to Pedro Dias for offering his opinion and being my guest on IMHO.
