Google On Diagnosing Multi-Domain Crawling Issues via @sejournal, @MattGSouthern

Google’s Search Advocate, John Mueller, shared insights on diagnosing widespread crawling issues.

This guidance was shared in response to a disruption reported by Adrian Schmidt on LinkedIn. Google’s crawler stopped accessing several of his domains at the same time.

Despite the interruption, Schmidt noted that live tests via Search Console continued to function without error messages.

Investigations indicated no increase in 5xx errors or issues with robots.txt requests.

What could the problem be?

Mueller’s Response

Addressing the situation, Mueller pointed to shared infrastructure as the likely cause:

“If it shared across a bunch of domains and focuses on something like crawling, it’s probably an issue with a shared piece of infrastructure. If it’s already recovering, at least it’s not urgent anymore and you have a bit of time to poke at recent changes / infrastructure logs.”

Infrastructure Investigation

All affected sites used Cloudflare as their CDN, which raised some eyebrows.

When asked about debugging, Mueller recommended checking Search Console data to determine whether DNS or failed requests were causing the problem.

Mueller stated:

“The crawl stats in Search Console will also show a bit more, perhaps help decide between say DNS vs requests failing.”

He also pointed out that the timing was a key clue:

“If it’s all at exactly the same time, it wouldn’t be robots.txt, and probably not DNS.”

Impact on Search Results

Regarding search visibility concerns, Mueller reassured this type of disruption wouldn’t cause any problems:

“If this is from today, and it just lasted a few hours, I wouldn’t expect any visible issues in search.”

Why This Matters

When Googlebot suddenly stops crawling across numerous sites simultaneously, it can be challenging to identify the root cause.

While temporary crawling pauses might not immediately impact search rankings, they can disrupt Google’s ability to discover and index new content.

The incident highlights a vulnerability organizations might face without realizing it, especially those relying on shared infrastructure.

How This Can Help You

If time Googlebot stops crawling your sites:

  • Check if the problem hits multiple sites at once
  • Look at your shared infrastructure first
  • Use Search Console data to narrow down the cause
  • Don’t rule out DNS just because regular traffic looks fine
  • Keep an eye on your logs

For anyone running multiple sites behind a CDN, make sure you:

  • Have good logging set up
  • Watch your crawl rates
  • Know who to call when things go sideways
  • Keep tabs on your infrastructure provider

Featured Image: PeopleImages.com – Yuri A/Shutterstock

Google Revises URL Parameter Best Practices via @sejournal, @MattGSouthern

In a recent update to its Search Central documentation, Google has added specific guidelines for URL parameter formatting.

The update brings parameter formatting recommendations from a faceted navigation blog post into the main URL structure documentation, making these guidelines more accessible.

Key Updates

The new documentation specifies that developers should use the following:

  • Equal signs (=) to separate key-value pairs
  • Ampersands (&) to connect multiple parameters

Google recommends against using alternative separators such as:

  • Colons and brackets
  • Single or double commas

Why This Matters

URL parameters play a role in website functionality, particularly for e-commerce sites and content management systems.

They control everything from product filtering and sorting to tracking codes and session IDs.

While powerful, they can create SEO challenges like duplicate content and crawl budget waste.

Proper parameter formatting ensures better crawling efficiency and can help prevent common indexing issues that affect search performance.

The documentation addresses broader URL parameter challenges, such as managing dynamic content generation, handling session IDs, and effectively implementing sorting parameters.

Previous Guidance

Before this update, developers had to reference an old blog post about faceted navigation to find specific URL parameter formatting guidelines.

Consolidating this information into the main guidelines makes it easier to find.

The updated documentation can be found in Google’s Search Central documentation under the Crawling and Indexing section.

Looking Ahead

If you’re using non-standard parameter formats, start planning a migration to the standard format. Ensure proper redirects, and monitor your crawl stats during the switch.

While Google has not said non-standard parameters will hurt rankings, this update clarifies what they prefer. New sites and redesigns should adhere to the standard format to avoid future headaches.


Featured Image: Vibe Images/Shutterstock

Google Now Recommends Higher Resolution Favicons via @sejournal, @martinibuster

Google Search Central updated their favicon documentation to recommend higher-resolution images, exceeding the previous minimum standard. Be aware of the changes described below, as they may impact how your site appears in search results.

Favicon

A favicon is a custom icon that is shown in browser tabs, browser bookmarks, browser favorites and sometimes in the search results. The word “favicon” is short for Favorites Icon.

An attractive favicon is useful for making it easier for users to find links to your site from their bookmarks, folders and browser tabs and can (in theory) help increase clicks from the search results. Thus, a high quality favicon that meets Google’s requirements is important in order to maximize popularity, user interactions and engagements, and visits from the search engine results pages (SERPs).

What Changed?

One of the changes to Google’s documentation is to make it clearer that a favicon must be in a square aspect ratio. The other important change is to strongly emphasize that publishers use a favicon that’s at least a 48×48 pixel size. Eight by eight pixels is still the minimum acceptable size for a favicon but publishers will probably miss out on the opportunity for a better presentation in the search results by going with a an 8×8 pixel favicon.

This is the part of the documentation that changed:

Previous version:

“Your favicon must be a multiple of 48px square, for example: 48x48px, 96x96px, 144x144px and so on (or SVG with a 1:1 (square) aspect ratio).”

New version:

“Your favicon must be a square (1:1 aspect ratio) that’s at least 8x8px. While the minimum size requirement is 8x8px, we recommend using a favicon that’s larger than 48x48px so that it looks good on various surfaces.”

Comparison Of Favicon Sizes

Reason For Documentation Changes

Google’s changelog for documentation says that the change was made to make it clearer what Google’s requirements are. This is an example of Google taking a look at their documentation to see how it can be improved. It’s the kind of thing that all publishers, even ecommerce merchants, should do at least once a year to identify if they overlooked an opportunity to be communicate a clearer message. Even ecommerce or local merchants can benefit from a yearly content review because things change or customer feedback can indicate a gap in necessary information.

This is Google’s official explanation for the change:

“Updated the favicon guidelines to state that favicons must have a 1:1 aspect ratio and be at least 8x8px in size, with a strong recommendation for using a higher resolution favicon of at least 48x48px.

What: Updated the favicon guidelines to state that favicons must have a 1:1 aspect ratio and be at least 8x8px in size, with a strong recommendation for using a higher resolution favicon of at least 48x48px.

Why: To reflect the actual requirements for favicons.”

Read the newly updated favicon guidelines at:

Guidelines

Featured Image by Shutterstock/Elnur

Google Merchant Center: A guide for ecommerce retailers

Ecommerce businesses always look for ways to improve their online presence to drive sales. Google Merchant Center (GMC) is an essential tool for achieving this, as it helps businesses showcase their products across Google’s ecosystem. This guide aims to provide retailers with an understanding of GMC. It offers useful insights on optimizing product listings and enhancing visibility on Google Shopping.

Table of contents

What is Google Merchant Center?

Google Merchant Center (GMC) is a free-to-use platform for ecommerce businesses that want to use Google’s extensive reach. It is a centralized hub where retailers can upload and manage product data. With this, business owners can make their products visible across various services such as Search, Maps, YouTube, and Google Shopping. Merchant Center is an important part of the ecommerce SEO puzzle.

The homepage of Google Merchant Center

Google Merchant Center Next is now live

With the recent upgrade to Merchant Center Next, the platform offers a more streamlined and intuitive experience. Now, it’s much easier to manage products and improve their performance.

After signing up and logging in, retailers get access to features like Product Studio, an AI-powered tool for creating product images and videos. This tool generates tailored product assets, improves image quality, and removes backgrounds. Plus, the unified product list also allows omnichannel merchants to manage online and local inventory seamlessly.

The new analytics tab provides comprehensive insights into market trends and pricing strategies. For instance, the Pricing tab in Merchant Center Analytics provides retailers with valuable insights into how their product prices compare to those seen by customers on Google.

an image showing the new overview page in google merchant center
The Google Merchant Center Next is much easier to use and comes with great new features

This tool allows merchants to evaluate their pricing strategies by seeing if their products are more expensive, cheaper, or comparable to those competitors offer. The tab offers a breakdown across various brands and products, identifying those with significant price differences. In addition, it also provides sale price suggestions designed to make you more competitive.

Merchant Center Next also has a much better design, with intuitive navigation and easy website verification options. Finally, merchants have an easy-to-understand interface to manage their product information and optimize their presence on Google. The platform’s advanced analytics tools offer a better view of business performance.

The benefits of using Merchant Center

Google Merchant Center offers significant advantages for ecommerce businesses that focus on sales growth. It increases visibility by allowing products to appear in Google Shopping results and ads, reaching millions of potential customers. This broader audience increases conversion chances.

The platform enhances the shopping experience by providing detailed product listings with high-quality images and descriptions. This helps customers make quick, informed decisions, boosting conversion rates.

GMC also provides data-driven insights through its analytics tools. These tools help businesses optimize product listings by analyzing metrics like pricing, clicks, and conversions, improving their ecommerce strategy.

Additionally, GMC supports both free and paid listings. Free listings enhance visibility for smaller businesses without extra costs, while paid ads more effectively target specific customer segments.

Integration with Google’s ecosystem is seamless. It connects with Google Analytics and Google Ads, offering a comprehensive view of marketing efforts and improving ROI measurement.

Lastly, GMC gives businesses flexibility and control over product presentation and placement. This ensures marketing efforts align with business goals. An AI-powered tool like the Product Studio also helps build robust product data, enhancing efficiency.

Setting up Google Merchant Center

You’ll need to set up a GMC account to use Google’s network to improve your ecommerce business. Here’s a step-by-step guide to help you through the process:

  1. Sign up for Merchant Center:

    Visit the Google Merchant Center website and click “Sign Up for Free” to begin registering.

  2. Provide business information:

    Enter your business name, address, and contact details. This information will be used across the various features and tools in GMC. Specify your business model, whether you sell products online, in a physical store, or both. This flexibility allows GMC to tailor its features to suit your business needs.

  3. Verify and claim your website:

    Follow the instructions for the method you choose to complete the verification process. You can use an HTML file upload, an HTML tag, Google Analytics, Google Tag Manager, a code via a business email address, or an ecommerce platform (like Shopify, WooCommerce, or Wix).

  4. Configure shipping and tax settings:

    Set up shipping options and tax information. These settings are important as they affect how your products appear in listings and ads. Highlight any special offers like free shipping to attract more customers. Comply with Google’s shipping and tax information requirements to avoid disapprovals.

  5. Add product information:

    Once your account is set up, add your product information. This involves creating a product feed, which we will cover later.

Integrating with third-party platforms

Managing product data manually can be time-consuming and prone to errors. You can automate this process by integrating Google Merchant Center with third-party ecommerce platforms. Using this process helps your product listings be consistently updated and accurate.

It’s easy to connect your ecommerce platform. In your Merchant Center account, navigate to the “Settings” menu and select “Data sources” and “Add product source” to link it. Follow the on-screen instructions to connect your platform (e.g., Shopify). This typically involves authorizing the connection and selecting the data you wish to sync.

an image showing how to  add product data sources or feeds in google merchant center
Adding product data sources or feeds is also easier in Google Merchant Center

After integration, you can manage your products directly within Merchant Center. Make any necessary updates or modifications to ensure your listings remain competitive and compliant with Google’s policies.

Benefits of integrating your ecommerce platform

There are many benefits to integrating your ecommerce platform with Google Merchant Center. For one, you get automatic updates. So product information, including images, prices, and descriptions, is automatically synced from your ecommerce platform to GMC. This eliminates the need for manual data entry and reduces the risk of discrepancies.

Integrating also leads to streamlined management, as you can manage product data directly from your ecommerce platform. As a result, you’ll free up time and resources to focus on other aspects of your business. It should also lead to fewer errors in your product data. Automated syncing minimizes human errors, so the product information displayed on Google is always accurate and up-to-date.

Merchant Center product data feeds

Product data feeds (or data sources as they are now called) are the backbone of your Google Merchant Center account. These serve as the primary source of information that Google uses to display your products across its platforms. You should optimize these feeds to represent your products accurately and reach the right audience.

What is a product data feed/data source?

A product data feed or data source is a structured file containing detailed information about your products. This file includes attributes such as product titles, descriptions, prices, and images. Google uses this data to create Shopping ads and listings, so keeping it accurate and complete is critical.

Methods of adding products

Google Merchant Center offers multiple methods for uploading product data, providing flexibility to fit your business’s unique needs.

Automatic addition from your online store

Google can automatically add products from your online store using the structured data markup (schema.org) on your product pages. This method ensures that any changes made on your website, such as price updates or product availability, are reflected in Merchant Center. Simply provide your store URL and implement structured data markup on all product pages to enable automatic updates.

Connecting ecommerce platforms

We’ve already mentioned this option before. Connect platforms like Shopify and WooCommerce to Google Merchant Center to sync product data automatically. This integration ensures product information is consistently updated without manual intervention, reducing errors and saving time.

File uploads

Upload product data using a file (e.g., TSV, TXT, XML). This versatile method allows you to compile data from multiple sources into a single file. Make sure to use proper formatting to avoid errors. You can also host the file at a URL for daily syncing with Merchant Center.

Google Sheets template

You can also use a Google Sheets template to manage product data. Any changes made in the spreadsheet automatically sync with Merchant Center. This provides a user-friendly interface for managing data, with automatic updates ensuring consistency.

Manual product entry

You can manually enter product details one by one in Google Merchant Center. This is ideal for small inventories or specific product updates. It allows you to add, edit, or delete products as needed easily.

Content API for Shopping

The Content API for Shopping is a powerful tool that allows developers to programmatically manage their Google Merchant Center accounts. It offers a more dynamic and efficient way to handle large or complex product inventories. This interface lets you automate updates and maintain accurate product data across Google’s platforms.

The elements of a product data feed

Optimizing your product data source involves attention to several key elements:

  • Product titles: To improve search visibility, use clear, descriptive titles with relevant keywords. Avoid using promotional text or excessive punctuation.
  • Product descriptions: Provide detailed and accurate descriptions, highlighting unique features and specifications. Include relevant keywords naturally.
  • Images: Use high-quality images with a plain background. Make sure images are clear, professional, and accurately represent the product.
  • Prices: Keep pricing current and competitive. Keep consistency between the price on your website and in the data feed.
  • Availability: Update product availability regularly to reflect current stock levels. Use attributes such as “in stock,” “out of stock,” or “available for pre-order.”
  • Unique product identifiers: Include GTINs, MPNs, and brand information to help Google accurately categorize and display your products.

Maintaining your Merchant Center account

Regularly checking your GMC account is essential for identifying opportunities and fixing issues. It has various account monitoring tools, such as the overview page. This page provides a snapshot of your account’s performance. It highlights key metrics such as product status and performance trends. Use this page to identify items needing attention and assess the impact of recent changes.

You can use Google Merchant Center’s Products tool to identify errors and issues within your product data feed. Regularly check the “Needs attention” section for warnings and errors. Here, you’ll find missing attributes or policy violations so you can fix them to avoid product disapprovals.

The new version of Merchant Center includes a much-improved analytics section. You’ll find detailed performance reports to analyze pricing, clicks, impressions, and conversion rates here. Use these insights to understand customer behavior and make informed decisions to optimize your product listings and campaigns.

Common issues and how to fix them

Addressing issues in your Google Merchant Center account is crucial for keeping products visible and compliant. Common problems include data errors, policy violations, disapprovals, and performance issues.

For data errors, update your product feed to meet Google’s specifications, correcting any missing attributes or pricing discrepancies. Comply with Google’s Shopping policies to prevent account suspension.

Review disapproved products to identify and fix issues, then request a re-review. Optimize product titles, descriptions, and images to boost relevance. Adjust bids and targeting in Google Ads to improve visibility and reach.

Strategies for performance improvement

Implement effective strategies to boost your Google Merchant Center performance. Update your product data feed with accurate pricing and availability. Enhance product listings with detailed descriptions and high-quality images, and consider adding ratings to build trust.

Use Merchant Promotions to highlight special offers and leverage Google Ads’ audience targeting to reach specific customer segments. Test and optimize campaigns with various ad formats and bidding strategies to refine your approach. You can sustain growth and maximize your ecommerce success by actively managing your GMC account.

To maximize Google Merchant Center’s potential, leverage advanced features like Local Inventory Ads to enhance product visibility and attract local shoppers. Try out Product Studio for AI-powered content creation, streamlining the production of high-quality product assets.

Enhance customer experience by providing clear return policies and ensuring secure data collection. Integrate with tools like Google Analytics and Google Ads for comprehensive insights and targeted marketing strategies.

an example of editing product images with google product studio
Google Product Studio uses AI to help you improve your product listings

All about Google Merchant Center

Google Merchant Center is an indispensable tool for ecommerce businesses that aim to boost their online visibility and sales. By following this guide, SEO experts can harness the full potential of GMC, driving growth and achieving ecommerce success. Login to your GMC account today, start optimizing and watch your business thrive.

Coming up next!

Ask An SEO: Why Are My Pages Discovered But Not Indexed? via @sejournal, @HelenPollitt1

Today’s Ask An SEO question comes from Mandeep, who is having trouble with indexing on their site.

Mandeep asks:

“We have redesigned a website and we had added a few new pages. Some pages were indexed successfully and some were not.

I tried multiple times on Google but that is not working. Now, while I submit the URL to index, it is showing this error via Google Search Console: Discovered – currently not indexed […]

I have tried everything but nothing is working. Please help me resolve this issue.”

This warning is coming from the “Pages” section of the “Indexing” report in Google Search Console. This report gives users insight into what pages Google has crawled and indexed and the problems it may have encountered doing so.

The report will give details of pages that have been successfully crawled and indexed. It also lists reasons why the pages on the site have not been indexed.

Is It A Problem If A Page Isn’t Indexed?

Most sites have pages that are not indexed. These are oftentimes at the request of the website owner.

For example, a page might be deliberately excluded from the search engine indexes by way of an HTML “noindex” tag on the page, or perhaps it is being blocked from crawling in the robots.txt file.

URLs that have been purposely excluded from indexing will appear within this report, as well as pages with problematic indexing issues.

In general, it can take some time for a new page on a website to be crawled and indexed. A new page taking time to show up among the “indexed” pages on the report is not always a sign of an issue.

Not every reason within the “Why pages aren’t indexed” report needs to be addressed.

Indexing Issues

Google will not crawl and index every URL it finds. Your main concern as a website manager is that the pages that you wish to be available as a search result are indexed.

Essentially, if they are not indexed, they will not be eligible to be a search result.

There are several reasons within the “Why pages aren’t indexed” report that do suggest an issue on the site that should be investigated. For example, “Server error (500)” and “Soft 404.”

These flags may not necessarily be a problem for the individual URLs if they aren’t ones you want to have indexed, but they can indicate a wider issue with the site.

What Is “Discovered – Currently Not Indexed”?

“Discovered – currently not indexed” is an error that Google flags for URLs that it knows about but has not indexed.

What is important to remember is that URLs will not appear in this bucket if they can fit within another in the report.

For example, a page with a noindex tag may technically have been discovered by Google and not indexed, but it would appear in the “Excluded by ‘noindex’ tag” bucket, so pages within the “Discovered – currently not indexed” bucket are there for another reason.

The explanation Google gives for a URL appearing as “Discovered – currently not indexed” is:

“The page was found by Google, but not crawled yet. Typically, Google wanted to crawl the URL but this was expected to overload the site; therefore Google rescheduled the crawl. This is why the last crawl date is empty on the report.”

Google tries to make its bots crawl conscientiously.

That is, as Googlebot is not the only visitor to a site, and maybe one among many bots crawling it, it doesn’t want to crash the site by sending too many “requests” to the server.

What Might Be Causing A URL To Be “Discovered – Currently Not Indexed”?

There are two main reasons a page is known to Google but not indexed. John Muller gave details about these in 2023.

Essentially, alongside the concerns around the server’s capacity to withstand crawling, page quality is also considered.

Now, if a page has not been crawled, how can Google know its quality? Well, it can’t. What it can do is make assumptions based on the quality of the pages elsewhere on the site.

That’s right – thin, duplicate, low-value pages elsewhere on your website can affect the indexation of your core pages.

How To Fix The Issue

There is no quick fix to move a page from “Discovered – currently not indexed” to “Indexed,” but there are several solutions you can try.

Check If The Page Is Actually Indexed

The first port of call is to determine if the Google Search Console report is accurate and up to date.

In the top right-hand corner of the report, you will see the “Last updated” date. This gives you an idea of whether the report might be outdated.

Next, go to Google and perform a site:[yourwebsitedomain] inurl:[the URL slug of the page you want to index] search.

If the page is returned as a search result, then you know it is actually indexed.

Give the report some time to get updated, and it will start appearing under the “Indexed” section and not in the “Discovered – currently not indexed” report.

Check Your Site’s Page Quality

Next, you may want to consider the overall quality of your website, as this could be the reason why Google is not indexing your page.

Remember, quality is not just a measure of the words on your site, their relevance to search queries, and the overall “E-E-A-T” displayed. Instead, Google’s John Muller described it as:

“When it comes to the quality of the content, we don’t mean like just the text of your articles.

It’s really the quality of your overall website.

And that includes everything from the layout to the design.

Like, how you have things presented on your pages, how you integrate images, how you work with speed, all of those factors they kind of come into play there.”

So, review your website with these criteria in mind. How does the quality of your website compare to that of your competitors?

A thorough website audit is a good place to start.

Check For Duplicate Pages

Sometimes, a website might have low-quality or duplicate pages that the website manager has no knowledge of.

For example, a page might be reached via multiple URLs. You might have a “Contact Us” page that exists on both exampledomain.com/contact-us and exampledomain.com/contact-us/.

The URL with and the URL without the “trailing slash” are considered separate pages by Googlebot if it can reach them both, and the server returns a 200 status code. That is, they are both live pages.

There is a possibility that all of your pages may be duplicated in this same way.

You might also have a lot of URL parameters on your website that you are unaware of. These are URLs that contain “query strings,” such as exampledomain.com/dress?colour=red.

They are usually caused by filtering and sorting options on your website. In an ecommerce website, this might look like a product category page that is filtered down by criteria such as color, and able to be sorted by price.

As a result, the main features of the page do not change with this filtering and sorting, just the products listed. These are technically separate, crawlable pages and may be causing a lot of duplicates on your site.

You may think your website only has 100 high-quality pages on it. However, a Googlebot may see hundreds of thousands of near-duplicate pages as a result of these technical issues.

Ways To Fix “Discovered – Currently Not Indexed”

Once you have identified the likely causes of your URL not being indexed, you can attempt to fix it.

If your website has duplicate pages, low-quality, scraped content, or other quality issues, that is where to begin.

As a side benefit, you are likely to see your rankings improve across your pages as you work to fix these issues.

Signify The Page’s Importance

In the example of our opening question, there is a specific page that Mandeep is struggling to get indexed.

In this scenario, I would suggest trying to bolster the page’s importance in the eyes of the search engines. Give them a reason to crawl it.

Add The Page To The Website’s XML Sitemap

One way of showing Google that it is an important page that deserves to be crawled and indexed is by adding it to your website’s XML sitemap.

This is essentially a signpost to all of the URLs that you believe search bots should crawl.

Remember, Googlebot already knows that the page exists; it just doesn’t believe it is beneficial to crawl and index it.

If it is already in the XML sitemap, do not stop there. Consider these next steps.

Add Internal Links To The Page

Another way to show a page’s importance is by linking to it from internal pages on the site.

For example, adding the page to your primary navigation system, like the main menu.

Or add contextual links to it from within the copy on other pages on your website. These will signify to Googlebot that it is a significant page on your website.

Add External Links To The Page

Backlinks – they are a fundamental part of SEO. We’ve known for a while that Google will use links from other websites to determine a page’s relevance and authority to a subject.

If you struggle to show Google that your page is of enough quality to index, then having external links from reputable, relevant websites pointing to it can give additional reassurance of the page’s value.

For example, if the page you are struggling to get indexed is a specific red dress’s product detail page, then having that dress’s page featured in some fashion blogs may give Google the signal that it is a high-quality page.

Submit It To Be Crawled

Once you have made changes to your website, try resubmitting the page to be crawled via Google Search Console.

If you notice in the Google Search Console “Indexing” report that the URL is still within the “Discovered – currently not crawled” bucket after some time (it can take anywhere from a few days to a few weeks for Google to crawl a submitted page), then you know that you potentially still have some issues with the page.

In Summary

Optimize your website for crawling and indexing. If you do this, you are likely to see those pages move from “Discovered – currently not indexed” to “Indexed.”

Optimizing your particular website will require an in-depth analysis of the overall quality of the site and identifying how to convey the importance of the “Discovered – currently not indexed” pages to Googlebot.

More resources: 


Featured Image: Paulo Bobita/Search Engine Journal

What do jumping spiders find sexy? How DIY tech is offering insights into the animal mind.

In his quest to understand the hermit crab housing market, biologist Mark Laidre of Dartmouth College had to get creative. Crabs are always looking to move into a bigger, better shell, but having really nice digs also comes with risks. Sometimes crabs gang up to pull an inhabitant out of an especially desirable shell. If they succeed, the shell is quickly claimed by the largest gang member, leaving another open shell for a slightly smaller crab to grab, and on down the chain until everyone has upgraded. 

To better gauge the trade-offs between shell size and defensibility, Laidre collaborated with an engineer to create the hermit crab eviction machine, a device that holds onto an occupied shell and measures how much force it takes a scientist to pull the crab out (crabs are not harmed or left homeless). It’s essentially a portable load cell that can survive the sun, sand, and humidity of the field. 

The force required to evict a hermit crab is an important measurement, because hanging on to their homes is a matter of life and death for crabs. “If you are evicted, there’s a real strong probability that what is left at the end of one of those chains is something that’s too small for you to even enter,” Laidre says. In his field area on a beach in Costa Rica, a homeless crab can quickly succumb to predators or heat: “You’re really dead meat in a sense.”

Studying the minds of other animals comes with a challenge that human psychologists don’t usually face: Your subjects can’t tell you what they’re thinking. To get answers from animals, scientists need to come up with creative experiments to learn why they behave the way they do. Sometimes this requires designing and building experimental equipment from scratch.

The DIY contraptions that animal behavior scientists create range from ingeniously simple to incredibly complex. All of them are tailored to help answer questions about the lives and minds of specific species, from insects to elephants. Do honeybees need a good night’s sleep? What do jumping spiders find sexy? Do falcons like puzzles? For queries like these, off-the-shelf gear simply won’t do.

The eviction machine was inspired by Laidre’s curiosity about crabs. But sometimes new questions about animals are inspired by an intriguing device or technology, as was the case with another of Laidre’s inventions: the hermit crab escape room (more on that below). The key, Laidre says, is to be sure the question you’re asking is relevant to the animals’ lives.

Here are five more contraptions custom-built by scientists to help them understand the lives and minds of the animals they study. 

OLY DEMPSTER

The falcon innovation box

The brainy birds in the parrot and crow families are the stars of scientific studies on avian intelligence. Now these smarties have a surprising new rival: a falcon. Raptors are not known for creative problem-solving, but behavioral ecologist Katie Harrington of the University of Veterinary Medicine Vienna suspected the striated caracara falcons she had observed on a remote Falkland Island were different. “They’re really interested in investigating things,” she says. “They’re very intelligent birds in general.”

diagram of the falcon experiment

HARRINGTON, ET AL.

To test their smarts, Harrington took inspiration from an “innovation arena” (left), designed for Goffin’s cockatoos, which are members of the parrot family known for their problem-solving abilities. It’s a semicircular array of 20 clear plastic boxes containing puzzles requiring different solutions to release rewards like cashews or corn kernels. Hauling the seven-foot-wide arena to the Falklands was not an option. So Harrington designed a 16-inch-wide “innovation box” attached to a wooden board, with eight compartments and puzzles adapted from the cockatoo studies. 

The birds loved it. “We were having caracaras run full speed to participate,” Harrington says. The challenge was keeping other birds away while one worked the box. The birds were able to solve the puzzles, which involved things like rattling a plank to knock down a bit of mutton or pulling a twig out from under a platform with mutton on it. They were even able to solve a tricky one that required them to punch a hole in a piece of tissue that obscured the treata task that eluded some cockatoos. 

In fact, 10 of 15 falcons solved all the puzzles, most of them within two sessions with the box. So Harrington designed eight new, harder tasks, but soon learned that some required unnatural movements for caracaras. She plans to keep trying to find tasks that reveal what they’re physically and mentally capable of. “They’re totally willing to show us,” she says, “as long as we can design things that are good enough to allow them to show us.”

The raccoon smart box

Why are raccoons so good at city living? One theory is that it’s because they’re flexible thinkers. To test this idea, UC Berkeley cognitive ecologist Lauren Stanton adapted a classic laboratory experiment, called the reversal learning task. For this test, an animal is rewarded for learning to consistently choose one of two options, but then the correct answer is reversed so that the other option brings the reward. Flexible thinkers are better at reacting to the reversals. “They’re going to be more able to switch their choices, and over time, they should be faster,” Stanton says.

To test the learning skills of wild urban raccoons in Laramie, Wyoming, Stanton and her team built a set of “smart boxes” to deploy on the outskirts of the city, each with an antenna to identify raccoons that had previously been captured and microchipped. Inside the box, raccoons found two large buttonssourced from an arcade supplierthat they could push, one of which delivered a reward. Hidden in a separate compartment, an inexpensive Raspberry Pi computer board, powered by a motorcycle battery, recorded which buttons the raccoons pushed and switched the reward button as soon as they made nine out of 10 correct choices. A motor turned a disc with holes in it below a funnel to dispense the reward of dog kibble. 

Many raccoonsand some skunkswere surprisingly eager to participate, which made getting clean data a challenge. “We had multiple raccoons just shove inside the device at the same time, like, three, four animals all trying to compete to get into it,” Stanton says. She also had to employ stronger adhesive to hold the buttons on after a few particularly enthusiastic raccoons ripped them off. (She had placed some kibble inside the transparent buttons to encourage the animals to push them.) 

Surprisingly, the smart boxes revealed that the shyer, more docile raccoons were the best learners. 

The jumping spider eye tracker

The thing about jumping spiders that intrigues behavioral ecologist Elizabeth Jakob is their demeanor. “They look so curious all the time,” she says. Unlike other arachnids, which spend most of their time motionless in their web, jumping spiders are out and about, hunting prey and courting mates. Jakob is interested in what goes on inside their sesame-seed-size brains. What matters to these tiny spiders? 

BARRETT KLEIN

For clues, Jakob watches their eyes, particularly their two principal ones, which have high-acuity color vision at the center of their boomerang-­shaped retinas. She uses a tool evolved from an ophthalmoscope that was specially modified to study the eyes of jumping spiders more than a half-century ago. Generations of scientists, including Jakob and her students at UMass Amherst, have built on this design, slowly morphing it into a mini movie theater that tracks the retinal tubes moving and twisting behind the spiders’ principal eyes as they watch. 

A spider is tethered in front of the tracker while a video of, say, a cricket silhouette is projected through the tracker’s lenses into the spider’s eyes. A beam of infrared light is simultaneously reflected off the spider’s retinas, back through the lenses, and recorded by a camera. The recording of those reflections is then superimposed on the video, showing exactly what the spider was looking at. Jakob found that just about the only thing more interesting to a jumping spider than a potential cricket dinner is a black spot that is growing larger. Could it be an approaching predator? The spider’s lower-resolution secondary eyes catch a glimpse of the looming spot in the corner of the video screen and prompt the primary eyes to shift away from the cricket to get a better look. 

Jakob’s eye tracker has also inspired other scientists’ creative experiments. Visual ecologist Nate Morehouse of the University of Cincinnati used the tracker to reveal that females of one jumping spider species aren’t all that interested in male suitors’ flashy red masks and brilliant green legsit’s the males’ orange knees that they focus on during courtship displays. “To get this insight into what they actually care about is really cool,” Jakob says.

The hermit crab escape room

Hermit crabs won’t just settle for the best empty snail shell they can findthey also remodel their homes. Hermit crab shells get better with time as each subsequent inhabitant makes home improvements, like widening the entranceway or carving out a more open, spacious interior. 

Dartmouth’s Mark Laidre has been studying crabs and their shell preferences for more than a decade. So when he realized he could use a micro-CT x-ray machine to create a three-­dimensional digital scan of a shell, he immediately began envisioning the experimental possibilities. To better understand the choices crabs make, he scanned shells that crabs clearly favored and then made alterations before 3D-printing them in plastic. “We could add little elements onto those that changed the external or the internal architecture,” Laidre says.

Next, he presented crabs with a dilemma. They were placed alone inside a box with a small exit (as shown below) and given a choice between two shells: a really nice, spacious model but with spikes added to the outside so that the crabs would not fit through the exit, and a shell that they would fit through but with uncomfortable spiny protrusions added to the inside. Could they figure out how to get out? “It’s effectively an escape room,” Laidre says.

When not trapped, crabs preferred the comfy shell with protrusions on the outside, claws down. But hermit crabs are social animals that prefer to be with other crabs, giving them motivation to escape solitary confinement. By the end of the day, more than a third of the trapped crabs had sized up their situation, moved from the crummy shell, and escaped. 

Solving a completely novel problem takes a certain amount of mental wherewithal that crabs don’t often get credit for. And Laidre suspects that cognitive capability may be what separated the successful escapees from the crabs that didn’t make it out of the escape room. 

The bee insominator

Sleepy people tend to be poor communicators. Entomologist Barrett Klein of the University of Wisconsin–La Crosse wanted to know if the same was true for drowsy honeybees. These social insects have a sophisticated communication system, known as the waggle dance, to convey to other bees where to find nectar. Are tired bees worse wagglers? To find out, Klein needed a way to keep bees up all night.

A metal disc attached to the back of a bee, seen on the right side of the photo, is painted yellow to hide whether it is made of steel that will be jostled by the magnets to keep her awake, or copper that won’t react to the magnets.
BARRETT KLEIN

He thought of shaking the hive, but this would just send all the bees angrily flying out. He wanted to keep some bees from sleeping while the rest slumbered peacefully, so that their dances could be compared the next day. Klein considered putting individual bees in vials that would be periodically shaken, but he couldn’t be sure if changes in their dance were due to sleepiness or isolation. He also thought of poking bees, aiming streams of air at individual bees, or even shining focused infrared beams at their faces. “Try to do that on all these bees facing all different directions,” Klein said. “It would be insane.” 

Eventually he landed on using neodymium rare earth magnets to jostle bees that had metal wafers glued between their wings with pine resin. “I had to make a hive that was narrow, with only two-millimeter-thick glass on either side, and have the magnets very close but not touching or scraping the glass,” Klein says. The biggest catch with this contraptiondubbed the Insominatorwas that Klein had to stay up all night rolling the banks of magnets back and forth alongside the hive three times a minute, depriving himself of sleep along with the bees.

But it paid off: He found that sleepy bees are indeed sloppy dancers. They did shorter dances that were less accurate with directiona miscommunication that could send hivemates on a flowerless search. In a follow-up study, Klein showed that other bees were not impressed with the drowsy displays and would promptly leave to find better wagglers. 

Happily, he has since upgraded the Insominator to automatically roll the magnets.

Betsy Mason is a freelance science journalist and editor based in the San Francisco Bay Area.

Google DeepMind is making its AI text watermark open source

Google DeepMind has developed a tool for identifying AI-generated text and is making it available open source. 

The tool, called SynthID, is part of a larger family of watermarking tools for generative AI outputs. The company unveiled a watermark for images last year, and it has since rolled out one for AI-generated video. In May, Google announced it was applying SynthID in its Gemini app and online chatbots and made it freely available on Hugging Face, an open repository of AI data sets and models. Watermarks have emerged as an important tool to help people determine when something is AI generated, which could help counter harms such as misinformation. 

“Now, other [generative] AI developers will be able to use this technology to help them detect whether text outputs have come from their own [large language models], making it easier for more developers to build AI responsibly,” says Pushmeet Kohli, the vice president of research at Google DeepMind. 

SynthID works by adding an invisible watermark directly into the text when it is generated by an AI model. 

Large language models work by breaking down language into “tokens” and then predicting which token is most likely to follow the other. Tokens can be a single character, word, or part of a phrase, and each one gets a percentage score for how likely it is to be the appropriate next word in a sentence. The higher the percentage, the more likely the model is going to use it. 

SynthID introduces additional information at the point of generation by changing the probability that tokens will be generated, explains Kohli. 

To detect the watermark and determine whether text has been generated by an AI tool, SynthID compares the expected probability scores for words in watermarked and unwatermarked text. 

Google DeepMind found that using the SynthID watermark did not compromise the quality, accuracy, creativity, or speed of generated text. That conclusion was drawn from a massive live experiment of SynthID’s performance after the watermark was deployed in its Gemini products and used by millions of people. Gemini allows users to rank the quality of the AI model’s responses with a thumbs-up or a thumbs-down. 

Kohli and his team analyzed the scores for around 20 million watermarked and unwatermarked chatbot responses. They found that users did not notice a difference in quality and usefulness between the two. The results of this experiment are detailed in a paper published in Nature today. Currently SynthID for text only works on content generated by Google’s models, but the hope is that open-sourcing it will expand the range of tools it’s compatible with. 

SynthID does have other limitations. The watermark was resistant to some tampering, such as cropping text and light editing or rewriting, but it was less reliable when AI-generated text had been rewritten or translated from one language into another. It is also less reliable in responses to prompts asking for factual information, such as the capital city of France. This is because there are fewer opportunities to adjust the likelihood of the next possible word in a sentence without changing facts. 

“Achieving reliable and imperceptible watermarking of AI-generated text is fundamentally challenging, especially in scenarios where LLM outputs are near deterministic, such as factual questions or code generation tasks,” says Soheil Feizi, an associate professor at the University of Maryland, who has studied the vulnerabilities of AI watermarking.  

Feizi says Google DeepMind’s decision to open-source its watermarking method is a positive step for the AI community. “It allows the community to test these detectors and evaluate their robustness in different settings, helping to better understand the limitations of these techniques,” he adds. 

There is another benefit too, says João Gante, a machine-learning engineer at Hugging Face. Open-sourcing the tool means anyone can grab the code and incorporate watermarking into their model with no strings attached, Gante says. This will improve the watermark’s privacy, as only the owner will know its cryptographic secrets. 

“With better accessibility and the ability to confirm its capabilities, I want to believe that watermarking will become the standard, which should help us detect malicious use of language models,” Gante says. 

But watermarks are not an all-purpose solution, says Irene Solaiman, Hugging Face’s head of global policy. 

“Watermarking is one aspect of safer models in an ecosystem that needs many complementing safeguards. As a parallel, even for human-generated content, fact-checking has varying effectiveness,” she says. 

Holiday Gift Guides Drive Long-Term Revenue

The holiday season is a prime time to acquire customers, but it can also drive them away. Impulse buying inevitably spikes during the gift-giving period. Such purchases might boost short-term revenue but often lead to higher returns and a damaged brand reputation.

Up to 60% of consumers regret impulse purchases, according to my research. Psychologists call this “post-purchase dissonance,” that sinking feeling when shoppers know they’ve made a poor decision. Others call it “buyer’s remorse.” Regardless, customers who regret first-time purchases will likely never buy again, eliminating a cornerstone of ecommerce profitability.

Landing Pages

The design of most landing and product-detail pages assumes bottom-of-funnel traffic, ready to convert. The pages are typically focused and clutter-free to entice quick purchases. Promotions such as “limited stock” and “limited time” are common for creating urgency.

While they can drive immediate sales, those tactics encourage impulse purchases, which come with higher return rates and frustrated customers.

Yet many merchants don’t realize their holiday advertising could drive both top- and bottom-of-funnel traffic. New shoppers unfamiliar with a brand may not be ready to buy and feel pressured into impulsive decisions.

The key is matching the landing experience with the ad’s context. Traffic from paid search, for example, usually requires a different experience than paid social.

Gift Guides Win

Brands sometimes direct paid social traffic to their social media profile page on, say, Facebook or Instagram. This strategy can undermine the ads’ effectiveness, as the aim of social profiles is to drive followers, not sales.

Another frequent error is sending paid traffic to the advertiser’s own home page. While it may prominently feature holiday deals, a home page is typically too broad and unfocused to drive sales.

To illustrate, consider the results of my A/B/C test for a fashion brand during last year’s Black Friday to Cyber Monday weekend. The test compared traffic from paid social to a home page, a product detail page, and a holiday gift guide microsite.

  • Traffic to the home page generated $1.52 in revenue per ad click.
  • Traffic to a product detail page generated $4.08 per click — 168% more than the home page.
  • The holiday gift guide outperformed both, generating $6.12 in sales per click — 303% higher than the home page and 50% more than the product page.

The holiday gift guide microsite is tailored to that campaign. The home page serves multiple purposes, but the gift guide is laser-focused on helping shoppers. It features curated products with holiday incentives — easy to browse across various categories.

This approach appeals to a variety of visitors, particularly those from paid social, where the intent is more diverse. The gift guide encourages considered shopping rather than impulse buying, leading to lower bounce rates, higher engagement, and longer time on-site. Hence the revenue per click is higher.

Custom Holiday Pages

To capitalize, brands can create custom holiday landing pages or gift guide microsites. Off-the-shelf landing page builders make it easy to craft individual pages tailored to specific holiday promotions. A simpler alternative is a promotional category page, although it won’t likely be as effective as one that’s purpose-built.

The goal for all is a landing experience that encourages thoughtful, non-impulsive shopping, driving immediate holiday revenue and even more in the long term.