Rackspace Hosted Exchange Outage Due to Security Incident via @sejournal, @martinibuster

Rackspace hosted Exchange suffered a catastrophic outage beginning December 2, 2022 and is still ongoing as of 12:37 AM December 4th. Initially described as connectivity and login issues, the guidance was eventually updated to announce that they were dealing with a security incident.

Rackspace Hosted Exchange Issues

The Rackspace system went down in the early morning hours of December 2, 2022. Initially there was no word from Rackspace about what the problem was, much less an ETA of when it would be resolved.

Customers on Twitter reported that Rackspace was not responding to support emails.

A Rackspace customer privately messaged me over social media on Friday to relate their experience:

“All hosted Exchange clients down over the past 16 hours.

Not sure how many companies that is, but it’s significant.

They’re serving a 554 long delay bounce so people emailing in aren’t aware of the bounce for several hours.”

The official Rackspace status page offered a running update of the outage but the initial posts had no information other than there was an outage and it was being investigated.

The first official update was on December 2nd at 2:49 AM:

“We are investigating an issue that is affecting our Hosted Exchange environments. More details will be posted as they become available.”

Thirteen minutes later Rackspace began calling it a “connectivity issue.”

“We are investigating reports of connectivity issues to our Exchange environments.

Users may experience an error upon accessing the Outlook Web App (Webmail) and syncing their email client(s).”

By 6:36 AM the Rackspace updates described the ongoing problem as “connectivity and login issues” then later that afternoon at 1:54 PM Rackspace announced they were still in the “investigation phase” of the outage, still trying to figure out what went wrong.

And they were still calling it “connectivity and login issues” in their Cloud Office environments at 4:51 PM that afternoon.

Rackspace Recommends Migrating to Microsoft 365

Four hours later Rackspace referred to the situation as a “significant failure”and began offering their customers free Microsoft Exchange Plan 1 licenses on Microsoft 365 as a workaround until they understood the problem and could bring the system back online.

The official guidance stated:

“We experienced a significant failure in our Hosted Exchange environment. We proactively shut down the environment to avoid any further issues while we continue work to restore service. As we continue to work through the root cause of the issue, we have an alternate solution that will re-activate your ability to send and receive emails.

At no cost to you, we will be providing you access to Microsoft Exchange Plan 1 licenses on Microsoft 365 until further notice.”

Rackspace Hosted Exchange Security Incident

It was not until nearly 24 hours later at 1:57 AM on December 3rd that Rackspace officially announced that their hosted Exchange service was suffering from a security incident.

The announcement further revealed that the Rackspace technicians had powered down and disconnected the Exchange environment.

Rackspace posted:

“After further analysis, we have determined that this is a security incident.

The known impact is isolated to a portion of our Hosted Exchange platform. We are taking necessary actions to evaluate and protect our environments.”

Twelve hours later that afternoon they updated the status page with more information that their security team and outside experts were still working on solving the outage.

Was Rackspace Service Affected by a Vulnerability?

Rackspace has not released details of the security event.

A security event generally involves a vulnerability and there are two severe vulnerabilities currently in the wile that were patched in November 2022.

These are the two most current vulnerabilities:

  • CVE-2022-41040
    Microsoft Exchange Server Server-Side Request Forgery (SSRF) Vulnerability
    A Server Side Request Forgery (SSRF) attack allows a hacker to read and change data on the server.
  • CVE-2022-41082
    Microsoft Exchange Server Remote Code Execution Vulnerability
    A Remote Code Execution Vulnerability is one in which an attacker is able to run malicious code on a server.

An advisory published in October 2022 described the impact of the vulnerabilities:

“An authenticated remote attacker can perform SSRF attacks to escalate privileges and execute arbtirary PowerShell code on vulnerable Microsoft Exchange servers.

As the attack is targeted against Microsoft Exchange Mailbox server, the attacker can potentially gain access to other resources via lateral movement into Exchange and Active Directory environments.”

The Rackspace outage updates have not indicated what the specific problem was, only that it was a security incident.

The most current status update as of December 4th stated that the service is still down and customers are encouraged to migrate to the Microsoft 365 service.

Rackspace posted the following on December 4, 2022 at 12:37 AM:

“We continue to make progress in addressing the incident. The availability of your service and security of your data is of high importance.

We have committed extensive internal resources and engaged world-class external expertise in our efforts to minimize negative impacts to customers.”

It’s possible that the above noted vulnerabilities are related to the security incident affecting the Rackspace Hosted Exchange service.

There has been no announcement of whether customer information has been compromised. This event is still ongoing.


Featured image by Shutterstock/Orn Rin

How Twitter’s “Teacher Li” became the central hub of China protest information

As protests against rigid covid control measures in China engulfed social media in the past week, one Twitter account has emerged as the central source of information: @李老师不是你老师 (“Teacher Li Is Not Your Teacher”). People everywhere in China have sent protest footage and real-time updates to the account through private messages, and it has posted them on their behalf—taking care to keep the sources anonymous during a period of widespread fear and uncertainty.

There’s just one man behind the account: Li, a Chinese painter based in Italy, who requested to be identified only by his last name in light of the security risks. He has never received training in journalism, but that hasn’t stopped him from operating what’s essentially become a one-person newsroom. 

At the peak of activity over the weekend, Li was receiving dozens of submissions every second, and he did his best to filter out unreliable information in a matter of moments. It was a totally new experience—even though he’d spent the past year posting anonymous submissions from his followers. While he has long talked about Chinese social issues online, sometime in 2021 he started receiving private messages on Weibo, the Chinese equivalent to Twitter (which is banned in China), from people asking him to post on their behalf. They feared exposing their own identities. 

His posts would get removed by Chinese censors, and by February, his account was banned. Over the next two months, another 49 of his accounts were suspended. But his followers generously allowed him to borrow their phone numbers to keep registering for more. In April 2022, after he could no longer access new Weibo accounts, he finally moved to Twitter. There, he quickly grew a large following of international accounts and Chinese people accessing the blocked social media platform via VPN. 

Then, last week, workers in a Foxconn factory in Zhengzhou started a violent confrontation with management, and Li started monitoring the situation through Chinese social media and follower submissions. He slept only three hours that night.

More protests then broke out over the weekend in major Chinese cities, and Li once again posted real-time protest footage—aiming to help people within China get information so they could decide if they wanted to join in, and also to inform people outside China about what was really going on. “Even though they’re not in China right this second, things are happening, and they’re watching,” Li told me. 

His Twitter account is now the hub for information on the protests, having gained over 600,000 followers in the past tumultuous week alone. 

The demanding work, though, has taken a toll. Within China, mentions of his account name have been censored on social media platforms, including Weibo and WeChat. He is getting death threats and insults in DMs. And police have visited his family back in China. 

But the anxiety has been mixed with a feeling of liberation as he feels he’s finally able to say the name of Xi Jinping on social media without fear. Li joked that his Twitter avatar, which is a doodle of a cat, has become the most famous and most dangerous cat of his time. 

Over a long conversation early this week, Li told me about what it’s like to be under such immense pressure and how challenging it can be to remain objective. Though this work has occupied almost every waking minute, he told me, he finally forced himself to start taking breaks on Monday—which led to a surprise encounter that warmed his heart. 

Here’s Li’s story, told in his own words. The following transcript has been translated, lightly edited, and rearranged for clarity. —Zeyi Yang

On lending his voice to people who are afraid 

This account is, in essence, the same as many other ordinary Twitter users’. I talk about my life, some topics related to my profession, and, of course, social issues.

But this account also carries another purpose. I don’t know when it started, but gradually I began receiving submissions. People would contact me through private messages, send me what’s happening, or their own stories, and hope I could post that for them. 

I think this may be a phenomenon that emerged from the increasingly strong internet and speech controls on Chinese digital platforms since Xi Jinping came to power. People are afraid to say things directly on the internet, even if their accounts are anonymous. But they still have the desire for expression, so they want someone else to say it for them. 

It was the same on Weibo. Last year, at a time when I only had less than 10,000 followers, people slowly realized that this person could speak [for them], so they came to me. Then, when news broke [in February] about the mother who gave birth to eight children [Editor’s note: She was a trafficking victim who was found chained in a shed], I helped someone publish a submission about how he wanted to find his sister. That was reposted over 30,000 times on Weibo, and then my account was banned. 

In the months after my account was gone, I kept registering new accounts, and they kept getting banned. In about two months, I had 50 accounts suspended. The fastest record was when it took 10 minutes for an account to disappear [after registration]. As soon as [censors] blew up my account, I would immediately start a new one.

My followers—I don’t know why—were able to immediately find me, so I would gain thousands of followers in an instant. It ended when [the regulators] seemed to find the website where I bought those Weibo accounts and suspended that website, after which I couldn’t access any more accounts.

I was really moved during that period. Weibo verifies your identity through your phone number. But a lot of online friends just lent me their phone numbers and said: “It’s okay, Teacher Li, you can use my number for verification. That’s fine.” That really touched me. But later I couldn’t get a new account, so I had to move to Twitter. 

My [Twitter] account was registered in 2020, but I actually started using it in April [2022]. From the very beginning, I have always been sent the latest news. I don’t know why, but there’s always someone on the ground who can send something to me immediately, including about the incident in Shanghai where people held up a white banner [in October]. Slowly, the number of followers grew.

Before I reported on the Foxconn incident, I had about 140,000 followers, and then it got to 190,000 when I finished reporting on Foxconn. I lost count of how many followers I have now. [Editor’s note: By the time of our interview, Li had over 670,000 followers on Twitter; by the time of publishing, the number had increased to over 784,000.

On becoming a one-man newsroom (on only a few hours of sleep)

These days I sleep for about five hours, and I’m focusing on [Twitter] for the rest of the day. There’s no one else. Even my girlfriend is not involved—just me.

In fact, the day I was online for the longest time was not in the past few days; it was during the Foxconn [protest]. Because the situation was developing so quickly, if they didn’t stop, I couldn’t stop. It didn’t cross my mind that since this had nothing to do with me, I could go to sleep. I never had that thought.

The fire in Urumqi [which sparked the broader wave of protests] has actually triggered a lot of empathy from the public. The possibility of a fire is really a concern for everyone, because all of them have at one point been locked in their home and not allowed to go out. 

In every similar news event in the past, no matter whether the government was responsible, it would always censor the news. After having their mouths sealed again and again, people became furious. There is always going to be a last straw, no matter what it ends up being. If it didn’t happen today, it would happen tomorrow, or the day after tomorrow. 

I thought [the protest in] Xinjiang on the night of the 26th was a moment to be remembered in history, but it turned out that was just the beginning of the story.

Particularly when protesters chanted the slogans that originated from the Sitong Bridge protest, I was like, “Oh no, it’s going to be a very, very serious thing if people are shouting these slogans in the center of Shanghai.” I had to document it in a neutral and objective manner, because if not, it could soon be forgotten, even on Twitter. I thought, “I need to take up the baton immediately,” and then I started doing it without thinking too much. 

It’s hard to describe the feeling that came after. It’s like everyone is coming to you and all kinds of information from all over the world is converging toward you and [people are] telling you: Hey, what’s happening here; hey, what’s happening there; do you know, this is what’s happening in Guangzhou; I’m in Wuhan, Wuhan is doing this; I’m in Beijing, and I’m following the big group and walking together. Suddenly all the real-time information is being submitted to me, and I don’t know how to describe that feeling. But there was also no time to think about it. 

My heart was beating very fast, and my hands and my brain were constantly switching between several software programs—because you know, you can’t save a video with Twitter’s web version. So I was constantly switching software, editing the video, exporting it, and then posting it on Twitter. [Editor’s note: Li adds subtitles, blocks out account information, and compiles shorter videos into one.] By the end, there was no time to edit the videos anymore. If someone shot and sent over a 12-second WeChat video, I would just use it as is. That’s it. 

I got the largest amount of [private messages] around 6:00 p.m. on Sunday night. At that time, there were many people on the street in five major cities in China: Beijing, Shanghai, Chengdu, Wuhan, and Guangzhou. So I basically was receiving a dozen private messages every second. In the end, I couldn’t even screen the information anymore. I saw it, I clicked on it, and if it was worth posting, I posted it.

People all over the country are telling me about their real-time situations. In order for more people not to be in danger, they went to the [protest] sites themselves and sent me what was going on there. Like, some followers were riding bikes near the presidential palace in Nanjing, taking pictures, and telling me about the situation in the city. And then they asked me to inform everyone to be cautious. I think that’s a really moving thing.

It’s like I have gradually become an anchor sitting in a TV studio, getting endless information from reporters on the scene all over the country. For example, on Monday in Hangzhou, there were five or six people updating me on the latest news simultaneously. But there was a break because all of them were fleeing when the police cleared the venue. 

On the importance of staying objective 

There are a lot of tweets that embellish the truth. From their point of view, they think it’s the right thing to do. They think you have to maximize the outrage so that there can be a revolt. But for me, I think we need reliable information. We need to know what’s really going on, and that’s the most important thing. If we were doing it for the emotion, then in the end I really would have been part of the “foreign influence,” right? 

But if there is a news account outside China that can record what’s happening objectively, in real time, and accurately, then people inside the Great Firewall won’t have doubts anymore. At this moment, in this quite extreme situation of a continuous news blackout, to be able to have an account that can keep posting news from all over the country at a speed of almost one tweet every few seconds is actually a morale boost for everyone. 

Chinese people grow up with patriotism, so they become shy or don’t dare to say something directly or oppose something directly. That’s why the crowd was singing the national anthem and waving the red flag, the national flag [during protests]. You have to understand that the Chinese people are patriotic. Even when they are demanding things [from the government], they do it with that sentiment. 

So one reason they are willing to pass on information to me is that they know that I am reporting it in a neutral, objective, and truthful way. But for other accounts, they are afraid of messaging them because what if it’s true—as they are told in China—that they are being taken advantage of by foreign forces? 

You can understand it like this: they want to voice their opposition, but they also don’t want it to be too radical. They want to stay in the middle. So I’m actually that middle point. I will report on what happens, but I will only report on what happens and not say a word more. That’s probably why I’ve become the central hub. Of course, I’ve become the central hub also because I’ve kept posting and posting.

So I try to only report on whatever information I receive, but it’s hard to do that now, because there are so many submissions. And to fact-check one thing, I may need videos from several different angles. 

For example, last night there were rumors of a shooting in Wuhan, a shooting in Chengdu, and a shooting in Xi’an, but I didn’t find any videos that I could use to verify them, so I didn’t end up posting anything. Well, that resulted in some Twitter users thinking I might be deliberately covering up some faults by the police.

And now there’s a somewhat awkward situation where some people in China think I’m inciting these things and some people abroad think I’m a big China propagandist. That’s a very difficult spot to be in. When you choose to stand in the middle, you are definitely under pressure from both sides, but that’s okay.

On dealing with digital chaos and deception

Since I basically had no time to think and was just posting every few seconds, the feed became very dense and very chaotic. Some people sent me the same videos repetitively. There were also many videos that originated from me, and then spread to other platforms like WeChat Moments, and were later sent back to me. Maybe this post was about Beijing, the next was Guangzhou, and the next one was Shanghai. There was no way for people to know at once whether the video in their hands had been sent or not, so they had to resend it to me. Maybe the video was taken at 9:00, but they sent it to me at almost 12:00 and thought it was in real time.

The fake video I got the most Sunday night was probably one where a police car was driving under an overpass and running over people. I must have watched it 60 or 70 times. Every time, it says that it was the Sitong Bridge or something. But that footage was actually not taken in China. Many people are willing to believe these videos, or they just want to believe that something big has happened.

One big crisis I experienced Monday morning was that—I don’t know who it was and whether it was someone [on the Chinese government’s side], but they kept sending me false news. There were some messages about things that happened, but not at the places they claimed. Then there were some messages that you could tell were fake immediately. Maybe they were hoping to take me down in this way. 

There are always people in my private messages who want me to post a call to action, or people who want me to summarize the slogans and post them, or declare what people should do, but I have not crossed that line. I believe everyone has a mission for themselves, and my mission is to report on what happened. If I suddenly joined those [activists], I would have really become—particularly since I wasn’t there on the ground—the one giving commands. If people died in the end, then the blood would be on me, because I directed them to act. So I don’t think that should be the case. I can only do the reporting. 

But I think in the end, I will inevitably be the one to blame. Even if I don’t do it, people will assume I’m guilty. 

If I can keep my independence till the end, then I can be a candle, a torch, just standing there on my own.

On the mental toll the work is taking

​​I just finished graduate school, so technically I’m a recent graduate, and I was just dragged into this thing out of nowhere and suddenly found myself with a role in it. I don’t know how to feel. I’m just anxious. I don’t know what will happen to me. Of course, I’m afraid that one day a car is going to run toward me when I’m crossing the road and fake a traffic accident or something. But I only worry about it when I turn off the computer. When I’m sitting in front of it, I don’t have time to think about myself.

It’s mostly just exhausting. I forced myself to take a break today. Usually, I just sit there, start, and keep going until the end, and I hardly ever get up.

But today I started to get some threats, and I became more stressed. You have to be afraid because you have seen so much and know so much. So today, I’m forcing myself to take a vacation. It’s not much of a vacation, I guess, but I spent a long time just walking.

Today has also been quite amazing. 

I got this death threat last night, but I don’t know where it’s from. It just said, “We already know where you live. You just wait.” I didn’t have time to take a screenshot then, because that message was quickly pushed down by other messages. I took one look and it was gone, immediately. But since then, it has been heavy on my mind.

Then this morning when I went out to buy cat food, I stood in front of the peephole and checked repeatedly whether someone was standing outside. On the way, I kept checking if someone was tailing me. And after I returned home, there was some weird movement in the stairs, so I put down everything by the door and stood in front of the peephole for 10 minutes, but never saw anyone. 

Then I thought to myself, I can’t do this forever—I have to make the person leave. I was thinking I would start livestreaming, find them, and then ask them to leave. But it turned out that there was no one. It was a tiny, tiny, tiny kitten. I don’t know why it was hiding there, but I took it home. And now my girlfriend is feeding it. This is the amazing thing that happened today. I’m considering naming it Urumqi.

I forgot whether it started when Xi Jinping came to power [in 2013], but I’ve been feeling quite aggrieved. All these years, I’ve been constantly, repeatedly censoring myself and staying cautious just so I can keep talking.

But just yesterday, suddenly, I’m not afraid anymore. I had no time to think about it, and I just kept posting. The simple version of what happened is: When they shouted out “Xi Jinping, step down,” I suddenly felt it didn’t matter anymore. I can report this thing. I can type these words. If they aren’t afraid to say it, then I’m also not afraid to type it. That’s it.

You know what these three characters mean when they are typed out. It’s completely different [from other words]. At that moment, I suddenly felt like I’m dead, I’m alive, I’m liberated, and I’m aggrieved, all at the same time. It was a very, very complicated feeling.

Your microbiome ages as you do—and that’s a problem

This article is from The Checkup, MIT Technology Review’s weekly biotech newsletter. To receive it in your inbox every Thursday, sign up here.

We’re all crawling with bugs. Our bodies are home to plenty of distinct ecosystems that are home to microbes, fungi, and other organisms. They are crucial to our well-being. Shifts in the microbiome have been linked to a whole host of diseases. Look after your bugs and they’ll look after you, the theory goes.

These ecosystems appear to change as we age—and these changes can potentially put us at increased risk of age-related diseases. So how can we best look after them as we get old? And could an A-grade ecosystem help fend off diseases and help us lead longer, healthier lives?

It’s a question I’ve been pondering this week, partly because I know a few people who have been put on antibiotics for winter infections. These drugs—lifesaving though they can be—can cause mass destruction of gut microbes, wiping out the good along with the bad. How might people who take them best restore a healthy ecosystem afterwards?

I also came across a recent study in which scientists looked at thousands of samples of people’s gut microbe populations to see how they change with age. The standard approach to working out what microbes are living in a person’s gut is to look at feces. The idea is that when we have a bowel movement, we shed plenty of gut bacteria. Scientists can find out which species and strains of bacteria are present to get an estimate of what’s in your intestines.

In this study, a team based at University College Cork in Ireland analyzed data that had already been collected from 21,000 samples of human feces. These had come from people all over the world, including Europe, North and South America, Asia, and Africa. Nineteen nationalities were represented. The samples were all from adults between 18 and 100. 

The authors of this study wanted to get a better handle on what makes for a “good” microbiome, especially as we get older. It has been difficult for microbiologists to work this out. We do know that some bacteria can produce compounds that are good for our guts. Some seem to aid digestion, for example, while others lower inflammation.
 
But when it comes to the ecosystem as a whole, things get more complicated. At the moment, the accepted wisdom is that variety seems to be a good thing—the more microbial diversity, the better. Some scientists believe that unique microbiomes also have benefits, and that a collection of microbes that differs from the norm can keep you healthy.
 
The team looked at how the microbiomes of younger people compared with those of older people, and how they appeared to change with age. The scientists also looked at how the microbial ecosystems varied with signs of unhealthy aging, such as cognitive decline, frailty, and inflammation.
 
They found that the microbiome does seem to change with age, and that, on the whole, the ecosystems in our guts do tend to become more unique—it looks as though we lose aspects of a general “core” microbiome and stray toward a more individual one.
 
But this isn’t necessarily a good thing. In fact, this uniqueness seems to be linked to unhealthy aging and the development of those age-related symptoms listed above, which we’d all rather stave off for as long as possible. And measuring diversity alone doesn’t tell us much about whether the bugs in our guts are helpful or not in this regard.
 
The findings back up what these researchers and others have seen before, challenging the notion that uniqueness is a good thing. Another team has come up with a good analogy, which is known as the Anna Karenina principle of the microbiome: “All happy microbiomes look alike; each unhappy microbiome is unhappy in its own way.”
 
Of course, the big question is: What can we do to maintain a happy microbiome? And will it actually help us stave off age-related diseases?
 
There’s plenty of evidence to suggest that, on the whole, a diet with plenty of fruit, vegetables, and fiber is good for the gut. A couple of years ago, researchers found that after 12 months on a Mediterranean diet—one rich in olive oil, nuts, legumes, and fish, as well as fruit and veg—older people saw changes in their microbiomes that might benefit their health. These changes have been linked to a lowered risk of developing frailty and cognitive decline.
 
But at the individual level, we can’t really be sure of the impact that changes to our diets will have. Probiotics are a good example; you can chug down millions of microbes, but that doesn’t mean that they’ll survive the journey to your gut. Even if they do get there, we don’t know if they’ll be able to form niches in the existing ecosystem, or if they might cause some kind of unwelcome disruption. Some microbial ecosystems might respond really well to fermented foods like sauerkraut and kimchi, while others might not.
 
I personally love kimchi and sauerkraut. If they do turn out to support my microbiome in a way that protects me against age-related diseases, then that’s just the icing on the less-microbiome-friendly cake.

To read more, check out these stories from the Tech Review archive:
 
At-home microbiome tests can tell you which bugs are in your poo, but not much more than that, as Emily Mullin found.
 
Industrial-scale fermentation is one of the technologies transforming the way we produce and prepare our food, according to these experts.
 
Can restricting your calorie intake help you live longer? It seems to work for monkeys, as Katherine Bourzac wrote in 2009. 
 
Adam Piore bravely tried caloric restriction himself to find out if it might help people, too. Teaser: even if you live longer on the diet, you will be miserable doing so. 

From around the web:

Would you pay $15,000 to save your cat’s life? More people are turning to expensive surgery to extend the lives of their pets. (The Atlantic)
 
The World Health Organization will now start using the term “mpox” in place of “monkeypox,” which will be phased out over the next year. (WHO)
 
After three years in prison, He Jiankui—the scientist behind the infamous “CRISPR babies”—is attempting a comeback. (STAT)
 
Tech that allows scientists to listen in on the natural world is revealing some truly amazing discoveries. Who knew that Amazonian sea turtles make more than 200 distinct sounds? And that they start making sounds before they even hatch? (The Guardian)
 
These recordings provide plenty of inspiration for musicians. Whale song is particularly popular. (The New Yorker)
 
Scientists are using tiny worms to diagnose pancreatic cancer. The test, launched in Japan, could be available in the US next year. (Reuters)