Mullenweg Criticized for 1st Amendment Claims via @sejournal, @martinibuster

Matt Mullenweg portrayed himself as a victim in his dispute with WP Engine, claiming in a tweet and blog post that they are ‘trying to curtail’ his free speech. Social media responses ranged from polite debunking of his First Amendment claim to accusations of hypocrisy.

TL/DR Of Dispute With WP Engine

Matt Mullenweg, co-creator of WordPress and CEO of Automattic ignited a dispute with managed WordPress web host WP Engine (WPE), using a Q&A at the WordCamp WordPress conference to denounce WP for not giving enough back to the WordPress open source project. He followed that statement with a post on WordPress.org that called WPE a cancer to WordPress, writing:

“This is one of the many reasons they are a cancer to WordPress, and it’s important to remember that unchecked, cancer will spread. WP Engine is setting a poor standard that others may look at and think is ok to replicate. We must set a higher standard to ensure WordPress is here for the next 100 years.”

He next banned thousands of WP Engine customers, cutting them off from updating their websites. Mullenweg later offered a temporary “reprieve” to prevent further inconvenience by WordPress publishers caught in the middle of the dispute and allow WP Engine to create a workaround.

Banning WP Engine elicited a negative response from WordPress developers and businesses. A tweet by the CEO of Ruby Media Group was representative of the general sentiment:

“My dev team can’t update my plugins because of this. You are destroying people’s lives.”

WP Engine responded with a Cease and Desist letter against Mullenweg and Automattic, followed by a federal lawsuit by WP Engine against Mullenweg and Automattic seeking relief from what they allege is an attempt by Mullenweg to extort millions of dollars from WPE.

Claim Of Attempt To Curtail First Amendment Rights

Mullenweg on Sunday published a blog post claiming that WP Engine’s lawsuit against him and Automattic is an attempt to “curtail” his “First Amendment rights.”

He wrote:

“WP Engine has filed hundreds pages of legal documents seeking an injunction against me and Automattic. They say this about community or some nonsense, but if you look at the core, what they’re trying to do is ask a judge to curtail my First Amendment rights.”

Mullenweg ended the post by stating he will no longer comment on the lawsuit filed by WP Engine but encouraged others to speak up in support of his side of the dispute.

Mullenweg Mocked On Social Media

The First Amendment is a guarantee that the United States government shall not create a law that infringes on a person’s free speech. Many on social media were quick to point out that WP Engine cannot curtain is First Amendment rights because they’re not the government.

A WordPress software developer tweeted:

“You have no first amendment rights in this context. WP Engine is not the government trying to curtail your 1A rights. 1A only applies to government entities.”

Another person followed up with:

“Please reread what the first amendment is, dum dum”

Another developer went further, calling Mullenweg “moronic”:

“WPEngine isn’t the government, how moronic can one man be?”

Another web developer advised Mullenweg to seek legal counsel to explain to him how the First Amendment works:

“Please go talk to your big expensive lawyer. I am sure they can break it down into small words for you.”

Accused Of Hypocrisy

Others on social media accused Mullenweg of hypocrisy for curtailing the free speech of others in the official WordPress Slack channel and banning WP Engine users from accessing plugins from the official repository.

A tweet by a WordPress and open source enthusiast captured the general feeling:

“Yes, “freedom of speech” is so important. I assume you now will be unblocking everyone that was exercising their right to freedom of speech in the WordPress Slack and here on X. Or, did you just mean literally “my freedom of speech” only.”

A WordPress developer from Denver tweeted:

“How many people have you banned from the WP slack channel over the past couple weeks?”

And another tweet:

“This coming from the guy cancelling anyone’s account/ access that disagrees with him. Really?”

Not Much Sympathy For Mullenweg

The overwhelming response to Matt Mullenweg’s post about his First Amendment rights was not sympathetic to his side of the story. A web applications developer’s tweet captured the lack of support:

“Maybe, maybe. But you probably won’t be getting a lot of sympathy from the crowd right now due to this thing the kids these days call “consequences.””

Read the original tweet by Mullenweg:

Matt Mullenweg’s blog post:

My Freedom of Speech

Featured Image by Shutterstock/Wirestock Creators

New Google Shopping Is an SEO Reality Check

Google Shopping’s generative AI makeover is a reality check for ecommerce marketers seeking organic search traffic.

Google launched the transformed version of Shopping to U.S. consumers on October 15, 2024.

“The new Google Shopping experience uses AI to intelligently show the most relevant products, helping to speed up and simplify” product searches, according to Sean Scott, Google’s vice president and general manager of consumer shopping, in a blog post.

Small and midsized business leaders often shudder when Google changes one of its services. Phrases like “show the most relevant products,” for example, are concerning. What sort of t-shirt or puffer jacket is “relevant”?

Shopping Graph

The answer might be in the Shopping Graph, which, according to Scott, powers the new Google Shopping along with Gemini AI.

A knowledge graph, such as Google’s Shopping Graph, is a map that connects ideas or concepts (nodes) via relationships (edges).

Screenshot from Google Shopping of five puffer coats with zippers.

The new Google Shopping can identify puffer coats with zippers. Click image to enlarge.

In a 2023 article, Randy Rockinson, Google’s group product manager for Shopping, described how the Shopping Graph connects concepts.

“Let’s say you’re looking for a puffer jacket,” Rockinson wrote. “That seems easy enough. But what if you have something particular in mind? Maybe you’d love a women’s red puffer coat that’s cropped, shiny, and has a fleece hood.”

In that example, Google’s Shopping Graph understands the connection between a particular jacket on, say, the Nordstrom website and the concepts of red and shiny. It can return a list of products matching the specific request.

Google has used data graphs since at least 2012 and officially announced its Shopping Graph at its 2021 I/O event. Thus retail, direct-to-consumer, and B2B marketers are likely familiar with the concept.

Listings

Google’s Shopping Graph has about 45 billion product listings as of October 2024. Those listings come from several sources, including:

  • Google Merchant Center (think product feeds),
  • Google Manufacturer Center,
  • Google Ads,
  • Ecommerce websites,
  • Product photography,
  • Manufacturer websites,
  • Product content, including PDFs,
  • Blog and article content,
  • Product reviews,
  • Social media,
  • YouTube videos,
  • Product testing.

For many merchants, the connection to Google’s Shopping Graph begins with the ecommerce platform. Shopify, BigCommerce, and similar solutions streamline the submission of a well-formated product feed to the Merchant Center.

Nonetheless, knowing that Google Shopping via the Shopping Graph and AI wants to understand details such as whether a puffer jacket is shiny and has a hood unnerves many marketers.

Is your business optimized for these kinds of details?

Product Content

Merchants relying on Google Shopping should audit their product details on that platform, ensuring plenty of specifications and descriptions to help Gemini show personalized results.

Product feed? Showing up in Google Shopping starts with a quality product feed. Ensure that all required attributes — product title, description, price, availability, images — are included and up to date. Use high-quality images. Google Shopping uses images for Google Lens (the visual search tool) and virtual try-on services. And be certain inventory levels are accurate.

Structured data markup? Confirm that your ecommerce site uses structured data markup to give Google more context about your products. Structured data will help Google list and categorize products correctly in its Shopping Graph.

Optimize for visual shopping? In his post, Google’s Scott stated that a primary goal of the new Google Shopping is working with Google Lens and virtual try-ons.

Product reviews? We know that Google’s Shopping Graph gets at least some of its product data from reviews. Thus enabling and encouraging those reviews is a good idea.

Product-focused content marketing? We also know that Google uses YouTube videos as well as third-party blogs, gift guides, and similar to inform the Shopping Graph. Most stores focus only on feeds, which are essential. But don’t stop there. Tutorials, instructions, examples, and more could enable a competitive advantage in Google Shopping.

What’s Next

Search engine optimization and content creation remain at the core of ecommerce marketing as generative AI becomes more prevalent. The new version of Google Shopping is the most recent example.

Don’t be surprised if more changes are coming.

The quest to figure out farming on Mars

Once upon a time, water flowed across the surface of Mars. Waves lapped against shorelines, strong winds gusted and howled, and driving rain fell from thick, cloudy skies. It wasn’t really so different from our own planet 4 billion years ago, except for one crucial detail—its size. Mars is about half the diameter of Earth, and that’s where things went wrong.

The Martian core cooled quickly, soon leaving the planet without a magnetic field. This, in turn, left it vulnerable to the solar wind, which swept away much of its atmosphere. Without a critical shield from the sun’s ultraviolet rays, Mars could not retain its heat. Some of the oceans evaporated, and the subsurface absorbed the rest, with only a bit of water left behind and frozen at its poles. Unrelenting radiation, along with electrostatic discharge from planet-wide dust storms, drove chemical reactions in the arid Martian dirt, ultimately leaving it rich in pesky toxic salts called perchlorates. If ever a blade of grass grew on Mars, those days are over. 

But could they begin again? What would it take to grow plants to feed future astronauts on Mars? In science fiction, it isn’t much of a problem. Matt Damon’s character in the 2015 movie The Martian simply had to build a greenhouse, spread out human excrement, add water, and wait. The film got a lot of things right—bacteria in the human biome will be useful—but it didn’t account for the perchlorates. The potato plants that sustained him would never have grown, but even if they had, two years of eating contaminated, carcinogenic potatoes would have nuked his thyroid, boxed his kidneys, and damaged his cells—though he might not have realized it, because perchlorates are also neurotoxic. It would have been Matt Damon’s finest death scene.

At the time Andy Weir was writing the book on which the film was based, no one really knew just how plentiful and ubiquitous the chemicals were. Though they were first discovered by NASA’s Phoenix lander in 2008, it took subsequent rovers, and compilation of historic data, to confirm that not only are perchlorates everywhere on Mars, but they are, in fact, abundant. Overall, Mars’s surface has perchlorate concentrations of about 0.5% by weight. On Earth, the concentration is often a millionth that amount.

For NASA, that’s a devastating issue. The ultimate goal of the agency’s Artemis program is to land astronauts on Mars. And for the last decade, the agency has pursued a long-term plan of establishing an “Earth independent” human presence on the Red Planet. More ambitiously, if less plausibly, Elon Musk, the chief executive officer of SpaceX, has stated that he expects a million people to live on Mars in the next 20 years.

Any notion of an independent Mars means the perchlorate problem must be solved, because humans have to eat. Resupply missions are, by definition, Earth dependent, and hydroponics are inadequate for feeding people in large numbers.

“We can sustain crews of 10, maybe 20, very comfortably with hydroponics, but it doesn’t scale much larger than that,” says Rafael Loureiro, an associate professor at Winston-Salem State University who specializes in plant stress physiology. Hydroponics systems must be built on Earth, and they require energy-inefficient pumps and constant monitoring for bacterial and fungal infections. “Once the system is infected, you lose your entire crop, because it’s a closed-loop system,” he says. “You have to discard everything and reset.”

The only real path forward, says Loureiro, is farming the land: “The perchlorate problem is something that we will inevitably have to deal with.” 

There is no soil on Mars. Just dusty, poisonous regolith—the mixture of loose rock, sand, and dust that makes up the planetary surface. On Earth, the regolith is replete with billions of years’ worth of broken-­down organic biomass—soil—that just doesn’t exist on Mars. To grow food there, we can’t just drop seeds in the ground and add water. We will need to create a layer of soil that can support life. And to do that, we first have to get rid of those toxic salts.   

There’s more than one way to remove perchlorates. You can burn them out; the compounds break down around 750 °F, but for that, you’ll likely need power sources like nuclear reactors, and a lot of ancillary equipment. You can literally wash the perchlorates from the regolith, but, Loureiro explains, “the amount of water that you need to do that is ungodly, and water is a limited resource as far as we know.” That process would likewise take a significant amount of energy. “That’s something that’s not feasible long term,” he tells me. The ideal solution is not something dependent on heavy machinery. Rather, it would rely on something small—microscopic, in fact. 

NASA and the National Science Foundation are funding research into how future astronauts on Mars might use microbial life not only to remove perchlorates from the planet’s dirt, but also to shape and enrich the regolith into arable soil. The work builds on years of effort to do the same thing in different places on Earth, and if successful, it will improve farming on two planets for the price of one.

“If we are able to grow plants in Martian regolith, we can do it anywhere on Earth.”

Rafael Loureiro, associate professor, Winston-Salem State University

It’s easy to dismiss the idea of Martian agriculture as a distant issue for a fictional future, but scientists must solve these sorts of problems before the rockets launch, not after humans are on their way. And as with much of NASA’s research, solving problems “up there” applies directly to life “down here.” Simply put, what we learn from Mars could find use here on Earth to turn infertile wastelands into rich agricultural zones. On Earth, natural perchlorate levels are highest in desert regions.  In other areas, high levels are usually due to industrial waste. The toxins harm Earthen plants as much as they would prospective Martian ones. Which means it’s not just NASA that is interested in remediation—even the US Department of Agriculture is paying for such research. 

“If I’m able to grow plants in a completely alien environment, the technology that I create to do that is one hundred [percent] transferrable to places here on Earth that are food insecure. To places that are extremely arid and unfit for agriculture. To places that have been affected by mining companies that have polluted the soil,” says Loureiro.

“If we are able to grow plants in Martian regolith, we can do it anywhere on Earth.”

Thinking small

The science laboratory at Arizona State University’s Biodesign Institute looks like a large version of every biology classroom in America: long black tables, myriad microscopes, racks of vials. When you look closer, though, you realize that the microscopes are a little fancier, and there are high-tech instruments like gas chromatographs and organic carbon analyzers in the mix.

Anca Delgado, a microbiologist, meets me at the entrance, where we don white lab coats and goggles. “We don’t plan to splatter you with anything today, but we want to be safe,” she says. 

Earth’s soil is wet and wiggly, teeming with life, and its mineral composition is highly diverse, thanks in part to tectonic action, microbial activity, and the rock cycle. But you can just look at Mars and tell something is way off: The core of the diminutive planet cooled before much of its iron had a chance to sink to its center. As a result, Martian regolith is packed with iron-rich minerals, which over time have oxidized. The planet’s exterior is literally rusted. Without water, it changes primarily via mechanical weathering, driven by wind and temperature; and without life, it is entirely inorganic.

Despite all this, Delgado, her graduate students, and colleagues across the country have found a possible path to solving the perchlorate problem and making Martian regolith arable. 

Perchlorates are salts made out of a negatively charged ion of chlorine and oxygen, bonded with a positive ion such as sodium. (There is also perchloric acid, which contains that same negatively charged ion.)Where perchlorates are abundant on Earth, it is often because we put them there. Everything from military manufacturing to the fireworks spectacular at Disneyland has contributed. They weren’t the only chlorinated compounds the US went wild about around the time of World War II. The United States for decades made heavy use of organic chlorinated solvents in everything from dry cleaning and metal degreasing to clothing dyes and medicine. 

MEREDITH MIOTKE | PHOTO: NASA/JPL-CALTECH/MSSS

Industry overall had a laissez-­faire attitude toward the management of waste products, leading to contamination of the nation’s groundwater. “After the Clean Water Act and later legislation in the 1970s prevented or banned the use of some of these chemicals, that’s when we discovered the extent of this contamination,” Delgado tells me. Some water pollution was obvious. The Cuyahoga River in Ohio routinely caught on fire. But other contamination remained hidden. Residents of Love Canal, a neighborhood in the city of Niagara Falls, New York, reported abnormally high rates of leukemia markers and birth defects before anyone recognized that the 20,000 tons of chemicals dumped into a canal in the 1940s might be responsible. 

It wasn’t enough to stop dumping toxic chemicals in waterways and landfills, however. Scientists had to find alternatives—Disney developed a fireworks launcher in 2004 that eliminated its perchlorate emissions, for example—and they also had to find ways to clean up the pollution that was already there. In the case of perchlorates, they can do so chemically. Rain and artificial irrigation can wash the compounds away, though this only transfers the problem to groundwater. Another strategy is to grow woody plants like willow and cottonwood in contaminated lands. These draw perchlorates from the ground and can then be harvested, removing them from the contamination cycle.

Another biological approach is to go small, using microorganisms to turn toxic chemicals into harmless ones. The poster child for this concept is a bacterium called Dehalococcoides mccartyi, which specifically feeds on organic chlorinated solvents and spits out dechlorinated ethene (a simple, nontoxic hydrocarbon) and harmless chloride ions, which are found naturally in the environment. Delgado studied D. mccartyi during her doctoral program. Her interests were strictly terrestrial. But the process, while highly effective, was not perfect. It took an incredibly long time to work in nature.

“We were looking at treatment times of months to decades,” Delgado tells me. Her research sought to culture D. mccartyi at much higher densities, which would translate to improved rates of action and faster treatment times for abandoned American waste sites. Her work has since been applied at multiple field sites in Arizona, New Jersey, and California.

Delgado walks me through the lab, which is arranged in an open plan. From the Biodesign Institute’s inception in 2004, she says, the idea was to bring researchers who would not normally interact into the same physical space. That means microbiologists working with wastewater samples, sludges, and soils are next door to scientists who do DNA origami. 

The leap from cleaning up Earth’s toxic waste dumps to making the Martian surface arable began in 2017, a month before Delgado started her new job at the university. She had read an article about Mars and idly looked into the chemical elements that had been detected so far on that planet. “I like microbes, and I wanted to see if Mars could fulfill their nutritional requirements,” she says. “I’m sort of a sci-fi geek.”

While attending a university retreat designed to get researchers talking about their work, she decided to “put it out there” that “I would be interested at some point to look at whether microorganisms would be able to grow in Martian conditions.” 

A paper she read in the journal Nature finally spurred her into action. Soil organic matter, which is necessary for growing plants, is itself made of decomposing plant and animal material. That would seem to preclude Martian agriculture from ever being achieved. But researchers had demonstrated for the first time that you can actually form soil organic matter with microorganisms alone—no decaying plants needed. The microbes themselves, and their tissues and excretions, could synthesize soil.  

Delgado realized that perchlorates could be the initial catalyst, the thing that microbes could thrive on and break down. Eventually the process could make the Martian regolith ready for planting. 

She applied for an Emerging Frontiers in Research and Innovation grant from the National Science Foundation to explore the idea. NASA recognized her proposal’s implications and co-funded the grant; the project received $1.9 million total in 2022. It was intended as a multiyear, multi-institution effort, with Delgado as principal investigator. The plan was that ASU, the lead institution, would explore using microbes to lower the concentration of perchlorates in Mars-like dirt. The University of Arizona in Tucson would investigate the soil organic matter formed by those microbes during their breakdown of the perchlorates, and the Florida Institute of Technology in Melbourne, Florida, would figure out how to grow the plants.

Testing the dirt

One problem with studying Martian regolith is that we simply don’t have any of it here on Earth. NASA’s entire campaign of Mars exploration for 50 years has been in service of characterizing the Red Planet as a possible site for life. The agency has long sought to get a pristine sample of regolith from Mars into a clean room on Earth for analysis. But so far it has failed to develop a credible mission to do so. In April, Bill Nelson, the administrator of NASA, essentially admitted defeat, asking outside research institutions and the private sector for proposals on how an affordable Mars sample return might be achieved.

In the meantime, scientists have to make do with simulated Martian dirt to study ways to diminish levels of perchlorates, including heat, radiation, and microbial methods.

Delgado’s lab at ASU includes an incubator and a confocal microscope inside a custom-built anaerobic chamber, for analyzing microorganisms that are sensitive to oxygen. At a research station lined with sealed glassware of various sizes, plus syringes, pipettes, and other equipment, she introduces me to two of her doctoral students: Alba Medina, who is studying environmental engineering, and Briana Paiz, who studies biological design. Both are lead researchers on the project.

In sealed bottles on the table are solutions of various colors ranging from tan to black. In the more transparent solutions, a red material sits at the bottom that looks suspiciously similar in color to the dirt on Mars. “These are called microcosm bottles,” Delgado says. “To maintain the integrity of the chemicals and composition, anything that needs to be put in or taken out of the bottles has to be done by syringe and needle.”

The bottles contain nutrients, water (a requirement for life), and artificial Mars dirt. With no Martian regolith available, Delgado uses an “analogue” called MGS-1—Mars Global Simulant—with chemical and mineral composition, proportions, and physical properties engineered to match up with the specs measured by the Mars rover Curiosity. The simulant is made by a company called Space Resource Technologies and is publicly available. You can buy it online.

“It’s the most expensive dirt you’ll ever buy,” Delgado says with a laugh. After handing me a latex glove so I don’t get my hands dirty, she offers me a bag. It feels like the sort of sand you’d expect to find at beaches too expensive to visit. It is very fine; it looks and feels like cocoa powder.

All researchers have to add to the microcosm bottles are perchlorates, which come as a white powder. With that, they have Mars in a jar. 

“Then,” says Paiz, “we add the microbes.” She shows me around the various experiments. “Those microcosm bottles have Dechloromonas, and the ones in the back are actually pure cultures of Haloferax denitrificans”—a bacterium that thrives in salty environments. The team is also experimenting with myriad microbes in mixed communities, each interacting with different elements and compounds, yielding different chemical compositions in their respective microcosms. It’s why some bottles are the color of chocolate and others are the color of peanut butter.

“They all started off the same color,” says Medina. “The black color of this one is like a visual signature confirmation of activity by sulfate-reducing microorganisms, for example.” 

Bacteria eat the things they like and ignore the things they don’t. Delgado’s group is looking for the ideal combinations not only to eliminate perchlorates, but to do so efficiently. Perchlorates present opportunities as well. When Delgado’s microbes break those compounds down, they form chloride and oxygen. 

Astronauts could potentially use them to produce a “major source of oxygen on Mars,” says Delgado. “Maybe the biggest source. One of the things we have been thinking about is how we could capture it.”

Microbial cultures and strains need not be brought to Mars in giant vats. Microorganisms grow exponentially fast. With less than a gram of material—not even the weight of a paper clip—a scientist on Mars could propagate it infinitely. A few drops in a test tube could theoretically yield entire orchards.

But the ideal microbial transport systems are astronauts themselves. Our bodies already contain perchlorate-eating microbes in our gut biomes. Delgado’s group does its perchlorate research using microbial communities from sludge acquired from wastewater plants. So Matt Damon’s character in The Martian was, to an extent, on the right track. 

But even if the proper microbes for breaking down perchlorates are present, that does not mean they will be able to do their job. “Those communities already have perchlorate reducers, but they also come in with friends and enemies,” says Delgado. The thousands of strains of bacteria in our microbiomes compete for nutrients, making them inefficient. The trick is to find ways to help microbes that eat the bad stuff and reduce the population of microbes that get in the way. 

For now, regolith in very small batches is prepped at her lab. Successful perchlorate reduction brings the concentration from about five grams per kilogram (the original 0.5%) to five to 20 micrograms per kilogram—or less. Existing literature suggests that this concentration range does not inhibit seed germination. For comparison, soils in the Arizona desert have a background concentration of perchlorate ranging from 0.3 to five micrograms per kilogram. In the Atacama Desert, that figure can be up to 2,500 micrograms per kilogram. 

But removing perchlorates is not enough for Martian plants to thrive. “Once you get perchlorates out, you’re still left with the issue of how to convert Mars regolith to soil,” says Andrew Palmer, an associate professor of biology at the Florida Institute of Technology and a co-­investigator on Delgado’s project. 

MEREDITH MIOTKE | PHOTO: NASA/JPL-CALTECH/MSSS

Palmer explains that regolith, with or without perchlorates, is an inert substrate. Soil, strictly speaking, is a substrate that has been acted on by biology and acts on it in turn. But in the regolith simulant—and one day, perhaps, in actual Martian regolith—the microbial activity responsible for eliminating perchlorates might also be able to transform minerals and release other useful plant nutrients like potassium and phosphorus. Finding the best way to do this is one of the Delgado team’s goals in studying different microbial strains and sludge.

“The biological process for removing perchlorates should not only get rid of them, but should also help us put other nutrients into the soil,” Palmer tells me. “We are trying to put an ecological cycle into the regolith.”

Early signs are promising, but it is a years-long endeavor. Researchers have reduced perchlorates in regolith samples. They have increased concentrations of organics in samples. They have changed the structure of regolith. They have grown plants in it. Their goal is to do all those things at once. “The whole grant, the whole process with everybody involved, is turning regolith with perchlorates into soil that is amenable to plant growth,” says Palmer, “and that’s very powerful.” 

If all goes well, the regolith simulant should have a total organic carbon concentration two to five times higher than it did at the outset, thanks to the organic residues formed by microbes. Ultimately, it should also have better water-­holding capacity as the organic carbons change the physical properties of the otherwise clay-like regolith, making it less dense and more beneficial for plants and their root systems.

Once the regolith is ready and the scientists satisfied, the simulated Mars material heads to Palmer’s laboratory in Florida to see what might grow.

Tomatoes and quinoa

Palmer admits he wasn’t particularly interested in the problem of growing plants on Mars when representatives from NASA first approached him seven years ago. The work seemed boring to him: “Plants grow in dirt—film at 11,” he joked.

The more they talked, however, and the more NASA scientists explained the challenges of working with Mars simulants and issues such as the perchlorate problem, the more his curiosity was piqued. How were we going to provide a sufficient food supply there, anyway? He and his researchers at Florida Tech’s Palmer Lab of Chemical Ecology and Astrobiology began growing plants, fungi, and bacteria in lunar and Martian regolith simulants, exploring how to remodel regolith into soil that is amenable to plant growth. 

“Mars is six to nine months away. If you lose a food source, you may not be able to survive the wait for a resupply mission.”

Andrew Palmer, associate professor of biology, the Florida Institute of Technology

In addition to incubators, they use a room they’ve dubbed “the red house,” which is a semi-controlled environment.

“It’s a giant room, all artificial lighting and artificial environmental controls, and the plants we grow in there have never seen the light of day, which is what we think the situation would be like off Earth,” he says. Anything grown on Mars, which lacks a meaningful atmosphere and is colder than Antarctica, would likewise be grown in enclosed, controlled environments with artificial lighting.

Plants are grown and regrown in Mars simulants so that Palmer and his team can get a sense for how the regolith evolves simply from the growth process over time. Currently the researchers are growing romaine lettuce, bell peppers, tomatoes, and clover “pretty regularly” in commercially available Mars simulants. This semester, he says, they’ve also begun experimenting with peanuts and quinoa.

Because the project is still in its early stages, they do not have results to share related to the preliminary material they received from Arizona. On that, they are presently conducting germination assays. 

“We are still trying to understand how the simulant behaves physically, because when you add water, it can cake up—become really solid and dense—and that can stifle roots. It’s a really challenging problem,” he says.

One of the things they have found is that over time, growing plants in Mars simulant makes its texture “fluffier.” Palmer plans to use electron microscopy to study samples from Delgado’s lab. “Regolith grains are actually pretty jagged,” he tells me. This applies to both Martian and lunar regolith. “After things grow in it for a little while, usually bacteria will kind of make those particles more rounded.” This is because microorganism growth in regolith often results in the deposition of biofilms and other organic compounds, as well as etching or corrosion of grain surfaces. All this is beneficial for plant growth.

Palmer considers food security to be paramount for a mission to Mars, and the project’s research so far leaves him optimistic.

“Mars is six to nine months away. If you lose a food source, you may not be able to survive the wait for a resupply mission,” he says. The solution is diversity. There should be frozen food rations. Some things should be grown hydroponically. Some things should be grown in regolith. If one system fails, you still have the others to help you restart. It’s just good safety practice, he says, but more than that, if we are serious about making Mars a home, we must use the skills that make us special. Agriculture must surely be at the top of that list.

“There is something about cultivating a land that I think speaks to being a human,” Palmer says. “It means you have mastered that place. You only have control over a place when you have control over the soil.”

David W. Brown is a writer based in New Orleans. His next book, The Outside Cats, is about a team of polar explorers and his expeditions with them to Antarctica. It will be published in 2026 by Mariner Books.

This lab robot mixes chemicals

Lab scientists spend much of their time doing laborious and repetitive tasks, be it pipetting liquid samples or running the same analyses over and over again. But what if they could simply tell a robot to do the experiments, analyze the data, and generate a report? 

Enter Organa, a benchtop robotic system devised by researchers at the University of Toronto that can perform chemistry experiments. In a paper posted on the arXiv preprint server, the team reported that the system could automate some chemistry lab tasks using a combination of computer vision and a large language model (LLM) that translates scientists’ verbal cues into an experimental pipeline. 

Imagine having a robot that can collaborate with a human scientist on a chemistry experiment, says Alán Aspuru-Guzik, a chemist, computer scientist, and materials scientist at the University of Toronto, who is one of the project’s leaders. Aspuru-Guzik’s vision is to elevate traditional lab automation to “eventually make an AI scientist,” one that can perform and troubleshoot an experiment and even offer feedback on the results. 

Aspuru-Guzik and his team designed Organa to be flexible. That means that instead of performing only one task or one part of an experiment as a typical fixed automation system would, it can perform a multistep experiment on cue. The system is also equipped with visualization tools that can monitor progress and provide feedback on how the experiment is going.  

“This is one of the early examples of showing how you can have a bidirectional conversation with an AI assistant for a robotic chemistry lab,” says Milad Abolhasani, a chemical and material engineer at North Carolina State University, who was not involved in the project. 

Most automated lab equipment is not easily customizable or reprogrammable to suit the chemists’ needs, says Florian Shkurti, a computer scientist at the University of Toronto and a co-leader of the project. And even if it is, the chemists would need to have programming skills. But with Organa, scientists can simply convey their experiments through speech. As scientists prompt the robot with their experimental objectives and setup, Organa’s LLM translates this natural-language instruction into χDL codes, a standard chemical description language. The algorithm breaks down the codes into steps and goals, with a road map to execute each task. If there is an ambiguous instruction or an unexpected outcome, it can flag the issue for the scientist to resolve.

About two-thirds of Organa’s hardware components are made from off-the-shelf parts, making it easier to replicate across laboratories, Aspuru-Guzik says. The robot has a camera detector that can identify both opaque objects and transparent ones, such as a chemical flask.

Organa’s first task was to characterize the electrochemical properties of quinones, the electroactive molecules used in rechargeable batteries. The experiment has 19 parallel steps, including routine chemistry steps such as pH and solubility tests, recrystallization, and an electrochemical measurement. It also involves a tedious electrode-precleaning step, which takes up to six hours. “Chemists really, really hate this,” says Shkurti.

Organa completed the 19-step experiment in about the same amount of time it would take a human—and with comparable results. While the efficiency was not noticeably better than in a manual run, the robot can be much more productive if it is run overnight. “We always get the advantage of it being able to work 24 hours,” Shkurti says. Abolhasani adds, “That’s going to save a lot of our highly trained scientists time that they can use to focus on thinking about the scientific problem, not doing these routine tasks in the lab.” 

Organa’s most sophisticated feature is perhaps its ability to provide feedback on generated data. “We were surprised to find that this visual language model can spot outliers on chemistry graphs,” explains Shkurti. The system also flags these ambiguities or uncertainties and suggests methods of troubleshooting. 

The group is now working on improving the LLM’s ability to plan tasks and then revise those plans to make the system more amenable to experimental uncertainties. 

“There’s a lot roboticists have to offer to scientists in order to amplify what they can do and get them better data,” Shkurti says. “I am really excited to try to create new possibilities.” 

Kristel Tjandra is a freelance science writer based in Oahu. 

Oropouche virus is spreading. Here’s what we know.

This article first appeared in The Checkup, MIT Technology Review’s weekly biotech newsletter. To receive it in your inbox every Thursday, and read articles like this first, sign up here.

There have been plenty of reports of potentially concerning viruses this last year. Covid is still causing thousands of deaths, and bird flu appears set to make the jump to human-to-human transmission. Now there are new concerns over Oropouche, a virus largely spread by bites from insects called midges (sometimes called no-see-ums in the US).

There have been outbreaks of the Oropouche virus in Latin America for decades. But this one is different. The virus is being detected in all-new environments. It is turning up in countries that have never seen it before. The spread is being described as “unprecedented.”

It may also be causing more severe disease. People with Oropouche fever typically have a sudden fever, aches and pains, and nausea. Most cases are mild, but some people have developed encephalitis and meningitis. And this year, two otherwise healthy young women who caught the virus have died.

Oropouche can be passed from mother to fetus, and it has been linked to stillbirths and birth anomalies. There are no treatments. There are no vaccines, either. This week, let’s take a look at why Oropouche is spreading, and what we can do about it.

Oropouche virus was first identified in 1955, in a person and a pool of mosquitoes from the village of Vega de Oropouche in Trinidad and Tobago. It was found in a sloth in Brazil in 1960. Since then, there have been over 30 outbreaks—in those countries as well as Peru, Panama, Colombia, French Guiana, and Venezuela. At least 500,000 cases have been reported in South America, largely in areas close to forest.

That’s probably because of the way the virus is transmitted. Oropouche virus is thought to be carried by some populations of sloths, and potentially some nonhuman primates. These animals can host the virus, which can then spread to people via insect bites, usually from midges or some types of mosquitoes.

Since late 2023, outbreaks have been reported in a number of countries in South America, Central America, and the Caribbean, including Cuba, a first for the country. 

There has been an especially large surge of cases in Brazil. Since the beginning of this year, 10,275 cases of Oropouche have been confirmed in the Americas, according to a situation summary report published by the Pan American Health Organization (PAHO) earlier this week. And 8,258 of them were in Brazil. Travelers have also imported cases to the US and Europe for the first time—90 such cases have been reported in the US, and 30 in Europe.

Another change is that this time around, the virus has been infecting people in urban settings far from forests. It is not entirely clear why, but there are probably a few reasons. Climate change, for a start, has led to increased temperatures and rainfall, both of which can help create breeding grounds for the insects that transmit the virus. And deforestation and urbanization, both of which have caused people to encroach on the habitats of wild animals, have also raised the risk of transmission to people, says Ana Pereiro do Vale, a veterinarian and microbiologist at University College Dublin in Ireland.

The virus itself also appears to have changed, according to new research published this week. William de Souza at the University of Kentucky and his colleagues analyzed blood samples taken from people with an Oropouche diagnosis between 2015 and 2024, enabling them to compare the form of the virus that is currently circulating with a historical strain.

The team found evidence that the virus has swapped genetic material with a related one, creating a new “virus reassortment.” It is this new form of the virus that has spread since the end of 2023, the team says.

That’s not all. The genetic changes have endowed the virus with new features. The current reassortment appears to be better at replicating in mammalian cells. That might mean that infected people—and sloths—have more of the virus in their blood, making it easier for biting insects to pick it up and pass it on.

The new form of the virus also seems to be more virulent. The team’s lab tests suggest that compared with the historical strain, it appears to cause more damage to the cells it infects.

We are still getting to grips with how the virus can spread, too. We know midges and mosquitoes are responsible for spreading Oropouche, but the virus can also pass to a fetus during pregnancy, with potentially harmful consequences. According to the PAHO report, Brazil has reported “13 fetal deaths, three spontaneous miscarriages, and four cases of birth anomalies” linked to Oropouche infections.

In a separate study published earlier this week, Raimunda do Socorro da Silva Azevedo at the Evandro Chagas Institute in Ananindeua, Brazil, and her colleagues assessed 65 unexplained cases of microcephaly—a birth anomaly in which babies have an unexpectedly small head—that had been recorded in Brazil between 2015 and 2024. The team found evidence of an Oropouche infection in six of the babies—and in all three that had been born in 2024.

It’s still not clear whether or how the virus might affect fetuses and babies, and research is ongoing. But the US Centers for Disease Control and Prevention (CDC) recommends that pregnant travelers “reconsider non-essential travel” to Cuba

Some scientists worry that the virus might also spread via sex. In August, a 42-year-old Italian man who fell ill after returning from a trip to Cuba was found to have Oropouche virus in his semen. And it was still there 58 days later. The CDC currently recommends that men diagnosed with Oropouche should use condoms or not have sex for at least six weeks from the start of their symptoms. They should avoid donating semen, too, according to the organization.

There are a lot of unanswered questions when it comes to Oropouche. Some scientists have suggested that this is because outbreaks have historically been seen in poorer countries in the Global South.

“There is sufficient colonialism in disease research—if it doesn’t affect the industrial world and Western business interests, it’s not important,” Shahid Jameel, a virologist at the University of Oxford, told Gavi, an organization focused on global vaccination efforts. “Now that the virus has been found in Cuba—not far from Miami—the wheels of public health will turn.”

Let’s hope they get in gear quickly. As Vale says: “We don’t know what will happen with the virus, the mutation rate of the virus, or if the virus will jump to another host. We need to be careful and pay attention.”


Now read the rest of The Checkup

Read more from MIT Technology Review‘s archive

Oropouche infections can look similar to dengue—another viral disease, also spread by mosquitoes, that affects people in Brazil. The country is attempting to tackle the problem with bacteria-infected mosquitoes, Cassandra Willyard reported in March.

The spread of bird flu in dairy cattle in the US has virologists worried. The virus could stick around on US farms forever and is raising the risk of outbreaks in mammals—including humans—around the world.

Flu season is officially upon those of us in the Northern Hemisphere. This year, it could enable the creation of an all-new bird flu, too. 

Could gene editing help curb the spread of bird flu? Abdullahi Tsanni explored the possibility of using CRISPR to make chickens resistant to the virus.

Another option, of course, is vaccines. Most flu vaccines are made, ironically, in chicken eggs. mRNA vaccines could provide an alternative, egg-free approach.

From around the web

A fertility clinic in London has helped two transgender individuals have a baby in a process that involved egg freezing, donated sperm, IVF, embryo storage, and surrogacy. “To our knowledge this is the first report of family building by a transgender couple in which both partners had successfully achieved gender reassignment and the creation of a family through surrogacy,” write the team. (Reproductive BioMedicine Online)

“They showed me them in a mirror … and I looked like a witch,” says one woman who has experienced the horror of dental veneers gone wrong. Veneers have become as routine as Botox and lip filler. But what can people do when their dream of a perfect smile turns into a nightmare? (The Guardian)

Thinking about deleting your 23andMe data? The company will hold on to some of it regardless, to comply with legal regulations. Some of your genetic information, your date of birth and your sex, and data linked to your account deletion request will all be retained. (MIT Technology Review)

Pet dogs are spending more time indoors, in environments they aren’t suited to. Service dogs, on the other hand, are uniquely well adapted to life in the 21st century, say two researchers at the Duke Canine Cognition Center. Humans need to breed and train more puppies like service animals, they argue. (The Atlantic)

Azalea: a science-fiction story

“This is simply a question of right and wrong.”

“You can’t deny the costs, though. You keep saying that just one more year of taxes will solve—

“We’re not solving—we’re mitigating!”

“Then what’s the point?”

The shrill back-and-forth fills the kitchen, where Xia is busy making breakfast, some kind of awful cricket-protein smoothie with kale. Tascha squeezes into the small space behind her, kisses her on the cheek. 

“Can you maybe put that in your head?”

Xia doesn’t put it in her head, but she at least lowers the volume with a click of her tongue. ElectoPod’s never-ending shouting match becomes something more akin to ocean noise, where only occasional angry waves splash through the kitchen. 

Tascha digs through the fridge, looking for something that isn’t kale or crickets. Finds a BitterBucketBrew and cracks it open. The coffee comes from gengineered plants that can survive higher latitudes. Caffeine in heaps, plus a proprietary process to filter out almost all endocrine disruptors, phthalates, microplastics, arsenic, and lead. Xia says it isn’t as good as coffee grown from heirloom stocks. It’s not natural, she says. Xia also says that a 99% filtration rate isn’t all that great when hormone mimics are dangerous in parts per trillion. 

For a refugee, Xia can be awfully picky. 

“You should be paying attention to the election,” Xia says. “This is your country.”

Tascha sips her coffee. “That’s why I have you.”

“Why don’t you run for district carbon board?” Xia presses. 

“Because then I’d have to deal with people. Anyway, I don’t have time. I’m trying to make bonus so we can get into Azalea.”

“If you don’t make time, someone stupid will. Gribaldi is running again. In Texas—”

“Don’t worry.” Tascha kisses Xia on the forehead. “Even Gribaldi’s not as stupid as Texas.” 

Xia makes a biting motion at Tascha, deliberately turning up the volume in the kitchen.

Is that passive-aggressive? 

Or aggressive-aggressive? 

Regardless, ElectoPod once again floods the kitchen with the latest depressing news. A new generation of nearly undetectable AI proxies are battling it out for mindshare as November approaches. An ocean of microtargeted content is pouring into people’s feeds, custom-generated on-the-fly ads and entertainment based on mountains of tracking data—all of it illegally obtained offshore, all of it tailored to sway public opinion—and no one knows who or what is generating it. It’s enough to make Tascha think she should have fought through her ADHD and gotten her programming degree. Someone has to be making money off that. 

Instead, Tascha clicks her tongue and turns on her own feed. Peace instantly envelops her, as ElectoMute smothers ElectoPod. Custom-tuned bone-conducted vibrations hum through her skull, perfectly canceling out the sound waves of Xia’s obsession. ElectoMute is Tascha’s only paid subscription. It’s not even sound, Xia complains every time she sees the monthly bill. A symptom of Late-Stage Capitalism. Paying to make the noise of another feed go away. 

Tascha calls it the best $50 a month she’s ever spent. 

Sometimes, it’s just nicer to shut things out. If Tascha’s honest, it’s always nicer to shut things out. Ever since she got her first bone implants on recommendation of the school counselor to help her focus and calm herself, she’s been a fan of shutting things out. People are both distracting and a hassle. Tascha is still sort of amazed that Xia doesn’t get on her nerves more. Sure, she also has another—very secret—mute feed tuned to Xia’s voice … but doesn’t everyone put their relationship on mute sometimes?

Xia is pushing a smoothie across the table at her. Her lips make noise shapes. “No smoothie?” 

Tascha shuts off ElectoMute and XiaMute. “Did you know the plywood they’re using on the worksite is made of mushrooms?” 

“So?”

“It’s like, mushroom-hemp composite. I could bring some back for your smoothies.”

“Very funny.”

“It’s carbon negative. You’d love it.” 

Xia gives her a sharp look. “Don’t be cute. I’ll take it from the kid who lost her whole town to a tornado, not from you.” 

Xia volunteers at the Georgia Displacement Authority. She doesn’t have a full work permit yet, but she can volunteer, so, of course, she does. 

Xia, always looking out for everyone. 

Tascha nurses her smoothie. Her father says that relationships are about compromise. If the worst thing about Xia is kale-cricket smoothies, Tascha knows she’s a winner. She forces down the last of the smoothie and gets up from the table. 

“I’m late.” 

Xia’s lips move again, making more mouth shapes. 

Tascha tunes back in. “What?” 

“I said, make sure your frigrig’s charged. It’s hot today.”

“It’s always charged. They charge them every shift.”

Xia is undeterred. “And swap out your mask filters. Canada’s burning up again.”


“How is there even any forest left?” 

Janet’s voice crackles in Tascha’s ear as they dangle off their rappelling lines, swinging from point to point in their harnesses. The gray soup of Canada’s burning forests envelops them. 

“How are we in Atlanta, sucking smoke from fucking Canada?” Janet continues. “Are we not America? Do we not have a long and honorable tradition of blockades, border checkpoints, and deportation? If we can keep Texas and Florida out, why not Burnt Canadians? Put up a sign: BC: Not Fucking Welcome.” 

They’re stitching solar, dangling 300 feet in the air, working their way down Tower 3 of the new Azalea Arcology. Tying window electrics, solar paint, and cell panels together. The mix is meant to make a lovely pattern (azaleas, in fact) on the face of the arcology structure, but the architect should be shot, because the electronics are a hassle and Tascha’s crew is behind schedule.

Most of the arcology is fast-attach, standardized like Lego blocks, built in factories, then autonomously shipped to the site and popped together, as simple as a kit. In the early stages, Azalea was just swarms of bots digging, grading, and auto-­assembling according to plan, knitting together the bones and skin of an entire new city of 10,000. Now that they’re at final finishing stages, though, humans are taking back the site. The complex patterns of varied electrical components still need a clever human touch, which is why Tascha’s crew is out in the Atlanta swelter, nearly mummified in Day-Glo frigrigs to keep the heat at bay. 

 It’s one thing to bike to work with just a filter mask and a chilled helmet to keep you cool; it’s a whole other thing to hang off the side of a building all day when wet-bulb temps push into the 40s. 

“I heard someone was camping up in Alberta and lit a bunch of beetle kill on fire. Whole state’s going up.” 

“Is it a state or a province?”

“How the fuck would I know?”

“Can we get off the coms, people?” Latoya, their crew lead, interrupts. “Some of us are trying to make bonus.”

“Yeah, Janet, get to work.”

A whole chorus of agreement follows from the rest of the crew.

“Yeah, Janet, get to work.” 

“Yeah, Janet, get to work.”

“Where the fuck is Janet? I can’t even see her in this smoke. It’s like pea soup.” 

Tascha slides across to a bank of PV windows and takes a sip from her frigrig reservoir. Cool water, sucked out of the humid air by the suit. Far below, cyclists zip down the street, their chilled jackets boosted by their bike batteries, popping in and out of view from under the street’s shading solar shelters and tree foliage. They look like schools of minnows. Heavy cargo vans and zip buses string out in a line, following programmed one-way routes to give most of the space to cyclists. AItlantis, the city management software, must have detected an increase in people waiting at bus shelters, because a surge of robotic zip buses is swarming toward Obama Greenway. 

Tascha starts splicing and soldering a cluster of energy-­generating windows to the solar paint that surrounds them, and then hooking the whole into wires that will carry the energy into the main arcology grid. It’s fussy work. Janet keeps bitching about the smoke. 

“I swear,” she says, as she spools down and jerks to a stop next to Tascha. “I’m just going up there and lighting the rest of that fucking forest on fire. I mean, shit, let’s get it over with—Hey. Tascha. You doing okay?” 

“I’m fine, why?”

“You missed a connection.” 

Tascha blinks away sweat that’s dripping in her eyes. “Oh. Thanks.”

Janet reaches into the nested wires. Tightens screws. Runs her own diagnostic. Everything glows green. “Can’t have our pretty solar design fail before it even gets hooked up, right?”

“Right. Yeah.” Tascha takes another sip of water. The frigrig’s reservoir and heat pump should be keeping it ice cold, but it’s more lukewarm. “Hey, can you check my frigrig battery?” 

Janet spins her around, checks her back. “Looks good to me. Seventy percent.” 

“Plugged in tight?” 

Tugs and jerks. “Yeah. All good. All tight.” 

“Let’s keep it moving, people. I want to finish this wall today,” Latoya says over the com. “Let’s get our bonus, right? For once?”

“If they designed these connections for frigrig gloves, this would go faster,” Janet says.

“Get back to work, Janet.” 

They all say it at the same time, and laugh.

Tascha’s father claims people didn’t wear any kind of cooling clothing in the old days. They wore tank tops and shorts, and sure they sweated buckets, but people didn’t have to completely hide from the heat. Tascha can’t imagine it. The only person Tascha knows who spends any time out in the heat willingly is Xia. Sometimes Xia lies nude on their balcony, letting the sun burn down on her, her skin sheened with sweat, salt jewels trickling lazily down the curves of her ribs. 

There’s something seductive about the contrast between Xia’s sun-browned flesh and the pale spiderweb of lines where the filter mask and its straps have hugged her face.

It’s fascinating and horrifying, like watching someone cook in an oven. 

Xia says it’s like a giant natural sauna, so why wouldn’t she take advantage of it? Saunas are good for you. Just ask the Finns. Xia also claims she’s epigenetically advantaged thanks to the Texas grid constantly failing during her childhood. Her body has been trained to survive any heat—which is so categorically bullshit that Tascha doesn’t even bother to argue. But Tascha does like Xia’s tan lines. There’s something seductive about the contrast between Xia’s sun-browned flesh and the pale spiderweb of lines where the filter mask and its straps have hugged her face. When Tascha ran her fingers over the tan lines one night, Xia told her it was a fetish. 

“It’s called FMT,” Xia said. “Filter mask tan lines. Very Rule 34.”

For reasons Tascha can’t fully explain, she’s annoyed that something that felt personal and private is actually a well-trodden porn search. Even when she’s alone in her own mind, tentatively feeling her way into real intimacy with Xia for the first time in her life, she’s still surrounded by people. At this very moment, there’s probably some algorithm custom-tailoring political attack ads based on FMT. It probably already knows about Tascha. 

People ruin everything.


“Goddamn, that looks like the good life.” 

Janet is peering in through another cluster of windows that they’re supposed to be hooking up. “Check it. You can see the waterfall and the river from here. It’s finished!”

Tascha realizes that she’s been leaning her head against the glass. She wipes sweat out of her eyes and peers through the PV glaze. Sure enough, the artificial river meanders along under high glass gallery arches, doing its job of water repurification and cooling as it winds through the arcology, then out under the dome of a semi-wild park, where lots of fast-growing carbon-sink cypress and citrus are growing, then cascading and pooling down through a series of rapids down into the artificial canyon the arcology uses for geocooling. Deep down in the shadows, Tascha glimpses a series of artificial lakes where mercury-free fish are destined to be raised.  

“Someone’s kayaking!”

A bright-yellow kayak has entered the top of the cataracts, some lunatic with a red helmet paddling down through the water features. Now that Tascha is looking closely, she can see that another whole part of the canyon is destined to be climbing walls. 

“I’m definitely buying in,” Janet says. “You buying in? We get top slots in the lottery, since we worked on it.” 

“I guess it depends if we get bonus.” Tascha’s hands feel clumsy. She drops the leads. “Are you hot? I think my suit’s fritzing.” 

“You want to tap out early, get it checked?”

“It’s just another couple hours. I’m fine.”

“Don’t try and muscle through—”

“I’m fine. Xia keeps telling me saunas are good for you. Let’s get our bonus.” 

Now that she’s sure her frigrig is fritzing, the heat becomes more bearable. She just imagines Xia, sunbathing, sweating it out intentionally. If Xia can take it, Tascha can take it. Christ, Xia complains if Tascha even turns up the A/C in the condo. She’s got the poverty mentality of all Texans, where people dying for lack of electricity is one of the independent territory’s founding principles. 

“It’s fine,” Tascha had explained, the first time they got in a fight over what constituted a reasonable temperature. “The grid’s in surplus. We’re doing them a favor by using it.”

“You’re making that up.”

“I can literally air-condition the balcony if I want. If you just open the doors I can knock it down 15 degrees. I can make you comfortable out there.”

“Don’t you dare.”

It makes Tascha want to move into Azalea even more. The whole place is kept at reasonable temperatures all day, every day. Outdoor Living, Indoors! is the arcology’s tagline, and it sounds like heaven. No wildfire smoke. Controlled temps. All those parks and rec trails and outdoor cafés. The energy systems connected to the cooling systems connected to the hydroponics systems, all of it managed by the unfortunately named AIzalIA Management Software that should, according to the brochure, make the entire arcology not only function as a carbon sink but also run an energy surplus that all the residents will profit from. 

Xia hates the idea of it. 

“It’s more privatization. You can’t privatize municipal services. It drains support for centralized government and general infrastructure. The rich live great, and the poor die like flies.”

“This isn’t Texas. That’s not how we do it here.”

“It’s not Texas … yet. If you let the rich live apart from the rest, eventually they start to undermine everything.”

“Can you just enjoy things, for once? Maybe practice a little optimism?” 

“It happened with schools. It will happen with infrastructure if you let it.”

“You know, this is exactly why the High Reverend of Texas has a warrant out for you. You’re lucky we don’t extradite.”

Xia makes a face. Tascha feels bad. Xia worrying is the same as Xia getting involved is the same as Xia making trouble is the same as Xia taking care of people is the same as Xia taking care of Tascha. It’s what she does. Tascha kisses her on the forehead. “Not everything is a plot to destroy the world.”

“This is exactly how Florida drowned itself. The rich got rich, and then they got on their private jets and flew away when Miami drowned. They always planned on kissing off to somewhere else. You can’t let these people undermine everything and then run away to hide with all their wealth.”

“I don’t think that’s what Azalea is about—”

“Yeah? What’s the buy-in?”

“That’s not fair. You know how much it costs to build. This ain’t cheap tech.”

“You know what would have been cheap? Just fixing the problem in the beginning so we all could just have gone on ­outdoor living, you know, outdoors. But rich people figured they’d be protected, so they didn’t give a shit. They’d move to New Zealand, right? They’d make their own personal compounds. They’d hire guards. They’d make Azaleas and they’d be fine—”

“But we can buy in too! If I make bonus, we’ll have enough—” 

Xia bites her teeth, hard. ElectoPod streams into Tascha’s head, bypassing ElectoMute: a pair of commentary hosts, haranguing one another.

“I think we need to remember that people in Florida had incomplete information.”

“Bullshit. They had everything they needed.” 

“Come on, Sunita. No one sets out to drown themselves! The people who drowned weren’t the people who made the disaster plans. Florida’s governor didn’t care how many people died. His real estate donors didn’t care. They had the numbers. They knew how much bigger storm surges were going to get—”

“So no one had a clue at all? They were just sitting in the dark like mushrooms? Come on, Maria. Let’s listen to this.”

A news announcer cuts in, old news coverage:

 “That’s the South Beach seawall. We can see the water coming up, coming through. We don’t know how many people are still in lower Miami. Obviously, this brings to mind the levee break in New Orleans in the early 2000s. Our thoughts and prayers are with the people of Florida in this trying time.” 

The argument between the hosts resumes. 

“Reminiscent of New Orleans! They had 70 years of warning! Literally everyone knew. Not just the governor! Not just his real estate donors. Don’t bother defending them, Maria. People got exactly what they signed up for. They deserved it.”

Tascha wants to argue with Xia. To point out that ElectoPod is saying that it wasn’t just rich people, that everyone was stupid, that everyone went along. Bottom line, people in general are just stupid, but Xia keeps talking at her, and XiaMute doesn’t seem to be working.

“Wake up, Tascha! You can’t just seal yourself off from people.”

“Wake up, Tascha! You have to be involved. If you don’t get involved, stupid people will.”

Wake up, Tascha! If you don’t pay attention, other people will decide for you.”

“Wake up Tascha! I know about XiaMute.”

“Wake up!”

“Wake up!”

“Wake up!”


Tascha comes awake, gasping. Water rushes around her. She thrashes, trying to swim, trying to keep her head above water.

“Whoa, girl! Take it easy!” Janet is cradling her in her arms, along with some woman in a red helmet. 

The kayaker? 

They’re in the river, Tascha realizes. They’re inside Azalea. The kayaker and Janet are supporting her, holding her up as water flows and tugs around her. The rest of the construction crew clusters on the riverbank, peering through the cattails, watching with concern. “Is she okay?” Latoya calls.

“She’s going to be fine,” the kayaker calls back. 

 “I was talking to Xia …”

“Xia’s coming,” Janet says. “Don’t worry about Xia. Just lay back. That’s right. Let’s get you cool.”

“She’ll be pissed.”

“She’ll be glad you’re alive. Quit fussing.”

Tascha lets herself sink back, lets Janet and the kayaker buoy her up. “What happened?”

Tascha stares up at the arching solar glass overhead as the river flows around her. Smoke is thick out there, but she can’t smell it in here.

“You heatstroked. And then you tried to pop your harness.” Janet laughs. “You almost went all the way to the ground before your safeties caught you. Shhh. Relax. You’re fine now. Took us a bit to get you untangled and inside. Just float. Stay easy. Let the water do its thing. You were cooking.”

“I messed up our bonus—”

“Don’t worry about that. We got you. All you need to do is let this nice water chill you out.”

Tascha stares up at the arching solar glass overhead as the river flows around her. Smoke is thick out there, but she can’t smell it in here. Here, she smells orange blossoms. Smells green ferns … cattails … warm mud. Life. 

“I should have tapped out when my suit died. I should have stopped to fix it.”

“Yeah, well.” Janet laughs. “We always see things clearer after we’ve screwed them up.”

Paolo Bacigalupi is an internationally best-selling author of speculative fiction. His most recent novel, Navola, was released in July by Knopf.

The race to find new materials with AI needs more data. Meta is giving massive amounts away for free.

Meta is releasing a massive data set and models, called Open Materials 2024, that could help scientists use AI to discover new materials much faster. OMat24 tackles one of the biggest bottlenecks in the discovery process: data.

To find new materials, scientists calculate the properties of elements across the periodic table and simulate different combinations on computers. This work could help us discover new materials with properties that can help mitigate climate change, for example, by making better batteries or helping create new sustainable fuels. But it requires massive data sets that are hard to come by. Creating them requires a lot of computing power and is very expensive. Many of the top data sets and models available now are also proprietary, and researchers don’t have access to them. That’s where Meta is hoping to help: The company is releasing its new data set and models today for free and is making them open source. The data set and models are available on Hugging Face for anyone to download, tinker with, and use.

 “We’re really firm believers that by contributing to the community and building upon open-source data models, the whole community moves further, faster,” says Larry Zitnick, the lead researcher for the OMat project.

Zitnick says the newOMat24 model will top the Matbench Discovery leaderboard, which ranks the best machine-learning models for materials science. Its data set will also be one of the biggest available. 

“Materials science is having a machine-learning revolution,” says Shyue Ping Ong, a professor of nanoengineering at the University of California, San Diego, who was not involved in the project.

Previously, scientists were limited to doing very accurate calculations of material properties on very small systems or doing less accurate calculations on very big systems, says Ong. The processes were laborious and expensive. Machine learning has bridged that gap, and AI models allow scientists to perform simulations on combinations of any elements in the periodic table much more quickly and cheaply, he says. 

Meta’s decision to make its data set openly available is more significant than the AI model itself, says Gábor Csányi, a professor of molecular modeling at the University of Cambridge, who was not involved in the work. 

“This is in stark contrast to other large industry players such as Google and Microsoft, which also recently published competitive-looking models which were trained on equally large but secret data sets,” Csányi says. 

To create the OMat24 data set, Meta took an existing one called Alexandria and sampled materials from it. Then they ran various simulations and calculations of different atoms to scale it.

Meta’s data set has around 110 million data points, which is many times larger than earlier ones. Others also don’t necessarily have high-quality data, says Ong. 

Meta has significantly expanded the data set beyond what the current materials science community has done, and with high accuracy, says Ong. 

Creating the data sets requires vast computational capacity, and Meta is one of the few companies in the world that can afford that. Zitnick says the company has another motive for this work: It’s hoping to find new materials to make its smart augmented-reality glasses more affordable. 

Previous work on open databases, such as one created by the Materials Project, has transformed computational materials science over the last decade, says Chris Bartel, an assistant professor of chemical engineering and materials science at the University of Minnesota, who was also not involved in Meta’s work. 

Tools such as Google’s GNoME (graphical networks for material exploration) have shown that the potential to find new materials increases with the size of the training set, he adds.  

“The public release of the [OMat24] data set is truly a gift for the community and is certain to immediately accelerate research in this space,” Bartel says. 

Software Founder Pivots to PPE Manufacturing

I’ve interviewed hundreds of entrepreneurs for this podcast. Most are problem solvers and optimists, confident in their ability to fulfill a need.

Take Lloyd Armbrust. He’s an editor turned software founder, having launched OwnLocal, a Y Combinator-backed portal for local newspapers.

When the pandemic hit, he observed doctors and nurses struggling to get protective gear. He thought, “This is ridiculous. How hard could it be to make these things?”

His solution was Armbrust American, an Austin, Texas-based manufacturer of personal protective equipment, which he launched in May 2020 and remains viable despite the dramatic drop in demand.

He and I recently spoke. He shared his lessons in manufacturing, ecommerce, and family-first priorities. Our entire audio is embedded below. The transcript is edited for clarity and length.

Eric Bandholz: Tell us who you are.

Lloyd Armbrust: My background is in media and software, but most prominently, I started a U.S.-based personal protective equipment manufacturing company in May 2020, at the onset of the pandemic. We produce U.S.-approved surgical masks, KN95 facemasks, and gloves. People said manufacturing couldn’t be done here and had to be in China, but we proved them wrong — though it’s been the hardest thing I’ve ever done.

When the pandemic hit in March 2020, my software business, OwnLocal, was slowing down. I saw how doctors and nurses struggled to get protective gear — some wore raincoats in emergency rooms. I thought, “This is ridiculous. How hard could it be to make these things?” So, I started researching and quickly learned it was much more difficult than I assumed.

I come from a software background, and I often say pushing pixels is easier than pushing atoms. If you want to scale software, click a few buttons on Amazon Web Services, and you’re ready. Manufacturing is a different beast. Yet we launched on Shopify and made a $1 million revenue in the first week, mainly because no one had masks in stock.

Bandholz: Did you have inventory ready, or was it a scramble?

Armbrust: We were producing masks but underestimated how hard fulfillment would be. We didn’t have a shipping system or proper label printers. I bought a label printer from Office Depot and tried to fulfill the first 100 orders. We had thousands of orders coming in, and it felt impossible. It took about three months to catch up.

This wasn’t about money. It was about solving a problem. Our mission from the start was to bring strategic manufacturing back to the U.S. All profits have gone back into the company. Today, consumer demand for masks has dropped significantly. We’re down to about $1.8 million in sales over the last 12 months compared to $7 million in January 2022 alone.

Bandholz: How do you manage a company with such a revenue drop?

Armbrust: It’s tough and demotivating. Scaling up and scaling down require the same skills — cutting costs and being ruthless. In the early days, our system was inefficient. We had 27 assembly machines, with 100 people running them to produce about a million units daily. Now, we’ve got five machines, each run by one person to output 200,000 units daily. So, we went from 27 machines and 100 employees to five machines and five employees, with the same production capacity.

We got lucky with our lease. The facility had been used by a defense contractor. When the pandemic hit, no one was leasing manufacturing space, so we got the space at a fraction of the cost. The owner wanted $50,000 a month for the space. It was really beautiful and big. We started at $5,000 monthly and worked up to $20,000, which they agreed on. Still, it was an 18-month lease. But by the time demand for PPE dropped, we were in prime real estate, right next to Amazon and Elon Musk’s Boring Company. We eventually moved to a facility on my ranch to save costs.

Bandholz: When you built that facility, what were your revenue projections, and where are you now?

Armbrust: We’re down to $1.8 million in annual consumer sales, but that’s only part of our business. We also manufacture for the government and other companies, but those contracts come with tighter margins — about 10-15%. The consumer side is more profitable and keeps us afloat.

When I built the facility, I had no idea where the bottom would be. China sells masks at prices lower than what it costs us to buy raw materials. They deliver masks to the U.S. for 1 cent each, while my raw materials cost more than that. The Chinese government subsidizes their manufacturers, covering costs like machines and even offering rebates. We can’t compete with that on price.

Ninety percent of medical gloves in the U.S. come from Malaysia and China. But before the pandemic, the U.S. didn’t produce any of these critical items.

Bandholz: You run the manufacturing business and OwnLocal, the publishing portal. You have six kids and a wife who also runs a business. How do you manage it all?

Armbrust: It’s all about priorities. My wife is at the top of my list, followed by my kids, and then making sure there’s money in the bank. I rank tasks every morning. I don’t focus on something that isn’t on that list. That can annoy some folks, but it keeps me sane. Work came before everything else in my first marriage, and I was unhappy.

Now, my family comes first. If my daughter wants to sit on my lap during an important business call, she’s in the meeting. I don’t apologize. I’m focused on enjoying life.

Bandholz: Where can people support you and reach out?

Armbrust: Armbrust.com. You can find me on X and LinkedIn.

A Guide To Robots.txt: Best Practices For SEO via @sejournal, @vahandev

Understanding how to use the robots.txt file is crucial for any website’s SEO strategy. Mistakes in this file can impact how your website is crawled and your pages’ search appearance. Getting it right, on the other hand, can improve crawling efficiency and mitigate crawling issues.

Google recently reminded website owners about the importance of using robots.txt to block unnecessary URLs.

Those include add-to-cart, login, or checkout pages. But the question is – how do you use it properly?

In this article, we will guide you into every nuance of how to do just so.

What Is Robots.txt?

The robots.txt is a simple text file that sits in the root directory of your site and tells crawlers what should be crawled.

The table below provides a quick reference to the key robots.txt directives.

Directive Description
User-agent Specifies which crawler the rules apply to. See user agent tokens. Using * targets all crawlers.
Disallow Prevents specified URLs from being crawled.
Allow Allows specific URLs to be crawled, even if a parent directory is disallowed.
Sitemap Indicates the location of your XML Sitemap by helping search engines to discover it.

This is an example of robot.txt from ikea.com with multiple rules.

Example of robots.txt from ikea.comExample of robots.txt from ikea.com

Note that robots.txt doesn’t support full regular expressions and only has two wildcards:

  • Asterisks (*), which matches 0 or more sequences of characters.
  • Dollar sign ($), which matches the end of a URL.

Also, note that its rules are case-sensitive, e.g., “filter=” isn’t equal to “Filter=.”

Order Of Precedence In Robots.txt

When setting up a robots.txt file, it’s important to know the order in which search engines decide which rules to apply in case of conflicting rules.

They follow these two key rules:

1. Most Specific Rule

The rule that matches more characters in the URL will be applied. For example:

User-agent: *
Disallow: /downloads/
Allow: /downloads/free/

In this case, the “Allow: /downloads/free/” rule is more specific than “Disallow: /downloads/” because it targets a subdirectory.

Google will allow crawling of subfolder “/downloads/free/” but block everything else under “/downloads/.”

2. Least Restrictive Rule

When multiple rules are equally specific, for example:

User-agent: *
Disallow: /downloads/
Allow: /downloads/

Google will choose the least restrictive one. This means Google will allow access to /downloads/.

Why Is Robots.txt Important In SEO?

Blocking unimportant pages with robots.txt helps Googlebot focus its crawl budget on valuable parts of the website and on crawling new pages. It also helps search engines save computing power, contributing to better sustainability.

Imagine you have an online store with hundreds of thousands of pages. There are sections of websites like filtered pages that may have an infinite number of versions.

Those pages don’t have unique value, essentially contain duplicate content, and may create infinite crawl space, thus wasting your server and Googlebot’s resources.

That is where robots.txt comes in, preventing search engine bots from crawling those pages.

If you don’t do that, Google may try to crawl an infinite number of URLs with different (even non-existent) search parameter values, causing spikes and a waste of crawl budget.

When To Use Robots.txt

As a general rule, you should always ask why certain pages exist, and whether they have anything worth for search engines to crawl and index.

If we come from this principle, certainly, we should always block:

  • URLs that contain query parameters such as:
    • Internal search.
    • Faceted navigation URLs created by filtering or sorting options if they are not part of URL structure and SEO strategy.
    • Action URLs like add to wishlist or add to cart.
  • Private parts of the website, like login pages.
  • JavaScript files not relevant to website content or rendering, such as tracking scripts.
  • Blocking scrapers and AI chatbots to prevent them from using your content for their training purposes.

Let’s dive into how you can use robots.txt for each case.

1. Block Internal Search Pages

The most common and absolutely necessary step is to block internal search URLs from being crawled by Google and other search engines, as almost every website has an internal search functionality.

On WordPress websites, it is usually an “s” parameter, and the URL looks like this:

https://www.example.com/?s=google

Gary Illyes from Google has repeatedly warned to block “action” URLs as they can cause Googlebot to crawl them indefinitely even non-existent URLs with different combinations.

Here is the rule you can use in your robots.txt to block such URLs from being crawled:

User-agent: *
Disallow: *s=*
  1. The User-agent: * line specifies that the rule applies to all web crawlers, including Googlebot, Bingbot, etc.
  2. The Disallow: *s=* line tells all crawlers not to crawl any URLs that contain the query parameter “s=.” The wildcard “*” means it can match any sequence of characters before or after “s= .” However, it will not match URLs with uppercase “S” like “/?S=” since it is case-sensitive.

Here is an example of a website that managed to drastically reduce the crawling of non-existent internal search URLs after blocking them via robots.txt.

Screenshot from crawl stats reportScreenshot from crawl stats report

Note that Google may index those blocked pages, but you don’t need to worry about them as they will be dropped over time.

2. Block Faceted Navigation URLs

Faceted navigation is an integral part of every ecommerce website. There can be cases where faceted navigation is part of an SEO strategy and aimed at ranking for general product searches.

For example, Zalando uses faceted navigation URLs for color options to rank for general product keywords like “gray t-shirt.”

However, in most cases, this is not the case, and filter parameters are used merely for filtering products, creating dozens of pages with duplicate content.

Technically, those parameters are not different from internal search parameters with one difference as there may be multiple parameters. You need to make sure you disallow all of them.

For example, if you have filters with the following parameters “sortby,” “color,” and “price,” you may use this set of rules:

User-agent: *
Disallow: *sortby=*
Disallow: *color=*
Disallow: *price=*

Based on your specific case, there may be more parameters, and you may need to add all of them.

What About UTM Parameters?

UTM parameters are used for tracking purposes.

As John Mueller stated in his Reddit post, you don’t need to worry about URL parameters that link to your pages externally.

John Mueller on UTM parametersJohn Mueller on UTM parameters

Just make sure to block any random parameters you use internally and avoid linking internally to those pages, e.g., linking from your article pages to your search page with a search query page “https://www.example.com/?s=google.”

3. Block PDF URLs

Let’s say you have a lot of PDF documents, such as product guides, brochures, or downloadable papers, and you don’t want them crawled.

Here is a simple robots.txt rule that will block search engine bots from accessing those documents:

User-agent: *
Disallow: /*.pdf$

The “Disallow: /*.pdf$” line tells crawlers not to crawl any URLs that end with .pdf.

By using /*, the rule matches any path on the website. As a result, any URL ending with .pdf will be blocked from crawling.

If you have a WordPress website and want to disallow PDFs from the uploads directory where you upload them via the CMS, you can use the following rule:

User-agent: *
Disallow: /wp-content/uploads/*.pdf$
Allow: /wp-content/uploads/2024/09/allowed-document.pdf$

You can see that we have conflicting rules here.

In case of conflicting rules, the more specific one takes priority, which means the last line ensures that only the specific file located in folder “wp-content/uploads/2024/09/allowed-document.pdf” is allowed to be crawled.

4. Block A Directory

Let’s say you have an API endpoint where you submit your data from the form. It is likely your form has an action attribute like action=”/form/submissions/.”

The issue is that Google will try to crawl that URL, /form/submissions/, which you likely don’t want. You can block these URLs from being crawled with this rule:

User-agent: *
Disallow: /form/

By specifying a directory in the Disallow rule, you are telling the crawlers to avoid crawling all pages under that directory, and you don’t need to use the (*) wildcard anymore, like “/form/*.”

Note that you must always specify relative paths and never absolute URLs, like “https://www.example.com/form/” for Disallow and Allow directives.

Be cautious to avoid malformed rules. For example, using /form without a trailing slash will also match a page /form-design-examples/, which may be a page on your blog that you want to index.

Read: 8 Common Robots.txt Issues And How To Fix Them

5. Block User Account URLs

If you have an ecommerce website, you likely have directories that start with “/myaccount/,” such as “/myaccount/orders/” or “/myaccount/profile/.”

With the top page “/myaccount/” being a sign-in page that you want to be indexed and found by users in search, you may want to disallow the subpages from being crawled by Googlebot.

You can use the Disallow rule in combination with the Allow rule to block everything under the “/myaccount/” directory (except the /myaccount/ page).

User-agent: *
Disallow: /myaccount/
Allow: /myaccount/$

And again, since Google uses the most specific rule, it will disallow everything under the /myaccount/ directory but allow only the /myaccount/ page to be crawled.

Here’s another use case of combining the Disallow and Allow rules: in case you have your search under the /search/ directory and want it to be found and indexed but block actual search URLs:

User-agent: *
Disallow: /search/
Allow: /search/$

6. Block Non-Render Related JavaScript Files

Every website uses JavaScript, and many of these scripts are not related to the rendering of content, such as tracking scripts or those used for loading AdSense.

Googlebot can crawl and render a website’s content without these scripts. Therefore, blocking them is safe and recommended, as it saves requests and resources to fetch and parse them.

Below is a sample line that is disallowing sample JavaScript, which contains tracking pixels.

User-agent: *
Disallow: /assets/js/pixels.js

7. Block AI Chatbots And Scrapers

Many publishers are concerned that their content is being unfairly used to train AI models without their consent, and they wish to prevent this.

#ai chatbots
User-agent: GPTBot
User-agent: ChatGPT-User
User-agent: Claude-Web
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: cohere-ai
User-agent: Bytespider
User-agent: Google-Extended
User-Agent: PerplexityBot
User-agent: Applebot-Extended
User-agent: Diffbot
User-agent: PerplexityBot
Disallow: /
#scrapers
User-agent: Scrapy
User-agent: magpie-crawler
User-agent: CCBot
User-Agent: omgili
User-Agent: omgilibot
User-agent: Node/simplecrawler
Disallow: /

Here, each user agent is listed individually, and the rule Disallow: / tells those bots not to crawl any part of the site.

This, besides preventing AI training on your content, can help reduce the load on your server by minimizing unnecessary crawling.

For ideas on which bots to block, you may want to check your server log files to see which crawlers are exhausting your servers, and remember, robots.txt doesn’t prevent unauthorized access.

8. Specify Sitemaps URLs

Including your sitemap URL in the robots.txt file helps search engines easily discover all the important pages on your website. This is done by adding a specific line that points to your sitemap location, and you can specify multiple sitemaps, each on its own line.

Sitemap: https://www.example.com/sitemap/articles.xml
Sitemap: https://www.example.com/sitemap/news.xml
Sitemap: https://www.example.com/sitemap/video.xml

Unlike Allow or Disallow rules, which allow only a relative path, the Sitemap directive requires a full, absolute URL to indicate the location of the sitemap.

Ensure the sitemaps’ URLs are accessible to search engines and have proper syntax to avoid errors.

Sitemap fetch error in search consoleSitemap fetch error in search console

9. When To Use Crawl-Delay

The crawl-delay directive in robots.txt specifies the number of seconds a bot should wait before crawling the next page. While Googlebot does not recognize the crawl-delay directive, other bots may respect it.

It helps prevent server overload by controlling how frequently bots crawl your site.

For example, if you want ClaudeBot to crawl your content for AI training but want to avoid server overload, you can set a crawl delay to manage the interval between requests.

User-agent: ClaudeBot
Crawl-delay: 60

This instructs the ClaudeBot user agent to wait 60 seconds between requests when crawling the website.

Of course, there may be AI bots that don’t respect crawl delay directives. In that case, you may need to use a web firewall to rate limit them.

Troubleshooting Robots.txt

Once you’ve composed your robots.txt, you can use these tools to troubleshoot if the syntax is correct or if you didn’t accidentally block an important URL.

1. Google Search Console Robots.txt Validator

Once you’ve updated your robots.txt, you must check whether it contains any error or accidentally blocks URLs you want to be crawled, such as resources, images, or website sections.

Navigate Settings > robots.txt, and you will find the built-in robots.txt validator. Below is the video of how to fetch and validate your robots.txt.

2. Google Robots.txt Parser

This parser is official Google’s robots.txt parser which is used in Search Console.

It requires advanced skills to install and run on your local computer. But it is highly recommended to take time and do it as instructed on that page because you can validate your changes in the robots.txt file before uploading to your server in line with the official Google parser.

Centralized Robots.txt Management

Each domain and subdomain must have its own robots.txt, as Googlebot doesn’t recognize root domain robots.txt for a subdomain.

It creates challenges when you have a website with a dozen subdomains, as it means you should maintain a bunch of robots.txt files separately.

However, it is possible to host a robots.txt file on a subdomain, such as https://cdn.example.com/robots.txt, and set up a redirect from  https://www.example.com/robots.txt to it.

You can do vice versa and host it only under the root domain and redirect from subdomains to the root.

Search engines will treat the redirected file as if it were located on the root domain. This approach allows centralized management of robots.txt rules for both your main domain and subdomains.

It helps make updates and maintenance more efficient. Otherwise, you would need to use a separate robots.txt file for each subdomain.

Conclusion

A properly optimized robots.txt file is crucial for managing a website’s crawl budget. It ensures that search engines like Googlebot spend their time on valuable pages rather than wasting resources on unnecessary ones.

On the other hand, blocking AI bots and scrapers using robots.txt can significantly reduce server load and save computing resources.

Make sure you always validate your changes to avoid unexpected crawability issues.

However, remember that while blocking unimportant resources via robots.txt may help increase crawl efficiency, the main factors affecting crawl budget are high-quality content and page loading speed.

Happy crawling!

More resources: 


Featured Image: BestForBest/Shutterstock