The Halo Effect: Your Paid Media Went Offline, Can You Survive Without It? via @sejournal, @jonkagan
Hello, my fellow digital marketers! This study was born out of a question that gave me a combination of irritation and renewed curiosity: “If I turned off all paid media, would my business actually suffer?”
This is a question that is as old as time (in digital marketing time that is), and just like swallows returning to Capistrano, I am posed this question every Q1, when a brand reviews my annual paid media budget recommendation.
What I thought was going to be a four-week test, actually turned out to be a three-month test with a one-month analysis.
The Scenario
The analysis was done for a fast-casual dining restaurant chain that operates 50+ restaurants across 10+ U.S. States that I was asked to do some auditing on. But, honestly, this repeats for most brands and verticals (much less so in B2B, I will note).
As alluded to earlier, the brand had a noticeable disconnect between the restaurant deciphering the impact of its website revenue and the media dollars spent, vs. wondering if it was just cannibalizing its name recognition and organic efforts. It challenged the belief that media was not contributing, and wanted to turn it off in a trial period. But more so, it just didn’t think the paid media was contributing, and it wanted to save some cash. So, we obliged.
Due to the disconnect in restaurant dining revenue being passed back to digital media, we elected to focus on its direct on-site online order of food to validate the data.
Before and after the test period, it was using search, Performance Max, paid social, digital OOH, and Display. All channels covered both prospecting and retargeting efforts.
It ran limited email efforts to its customers registered for rewards, but it does not have a mobile app, so the customer list is quite small, while its annual digital media investment is around $1.1 million.
For the analysis, we planned to pause media for five weeks in the middle of its low season (which is about four months long), and then compare the overall impact on the site before it was paused and after we brought it back.
Thrilling? Well, let’s just say some folks do get all hot and bothered around a mid-to-low impact media holdout aggregate site activity analysis.
The Important Parameters
So, there are some important things to note around this test:
- Traditional media was never stopped, but always ran at a low level, mostly billboards and radio.
- For unexplainable reasons, they never hooked up Search Console.
- They run a consistent SEO effort.
- The analysis was done on the same restaurants for all three time periods (they had a couple shutdown and one open in this time period, so that data was removed from the assessment).
- Their primary key performance indicator is in-restaurant visits, but they struggle to connect the source back to media initiative (we use three different foot traffic tracking vendors to measure it, but they don’t have the ability to pass back the in-restaurant sales data to the visit).
- Foot traffic leads are actually worth 15-25% more than online order sales, but we do not have true pass-through revenue for them.
Against the recommendation to do this test on an isolated market was not taken, and they did a full blackout.
Hypothesis
In my typical (and often inappropriate snarckastic manner, or so I am told), I referred them to my 2021 article, “How Paid Search Incrementality Impacts SEO (Does 1+1=3?),” and told them that this should be their baseline for anticipated impact. For those who don’t want to click on the link and read the article, my stance was that removing paid media and running organic only would have a grand net loss for the brand in terms of traffic and sales.
To give you a sense of performance, prior to the test, paid media accounted for ~28% of incremental site traffic and ~23% of online orders. Which in turn supports the following beliefs:
- With paid search engine-driven traffic exiting, we expect organic to rise, but not enough to offset what paid drove.
- With paid social out, we expect a net loss of overall social traffic, in addition to any halo impact driven by social awareness (i.e., direct to site, organic search).
- With programmatic traffic out, we expect a decrease in aggregate search traffic and direct to site traffic.
Net-net, the loss of paid media will result in a net loss of site traffic, leading to a net loss of online sales, which will be greater than media cost that would’ve been used to generate those sales.
The Pre-Test Data
Having selected a five-week period as our control period, we reviewed the initial data upfront:
| Spend | Impressions | Clicks/Site Visits | Online Orders | Revenue | |
|---|---|---|---|---|---|
| Search | $30,000 | 395,000+ | 57,000+ | 6,000+ | $250,000+ |
| Performance Max | $20,000+ | 9 million+ | 27,000+ | 275+ | $11,000+ |
| Social | $23,000+ | 12 million+ | 38,000+ | 40+ | $500 |
| Programmatic Display | $450 | 19,000+ | 100+ | 1 | $13 |
| DOOH* | $5,000 | 62,000+ | 0 | 0 | $0.00 |
| Total | $80,000 | 21 million+ | 123,000+ | 6,000+ | $262,000+ |
*Digital Out-of-Home advertising (DOOH)
Additionally, organic search had 131K+ site visits (42% of total), along with 12K+ online orders (46% of total) and $532K+ of revenue (47% of total).
While direct to site traffic had 78K+ site visits (25% of total), along with 8K+ online orders (29% of total) and $315K+ of revenue (28% of total).
Based on pre-test data, every site visit (from all traffic sources) was equal to $3.61 in online order revenue.
The Test Itself
- Organic search site visits rose 14% (+18K), orders rose 31% (+4K), and revenue rose 30% (+$161K)
- Direct to site visits dropped 4% (-3K), orders dropped 3% (-277) and revenue dropped 5% (-$15K)
- The single largest channel loss of traffic was social (organic+paid), which dropped 98% (-39K) in visits, and dropped 55% in orders (note this was from 80 to 36) and 27% in revenue (a loss of $400)
- All other site non-paid media traffic channels remained relatively flat
- Overall site visits dropped 22% (-68K+), orders dropped 9% (-2,500) and revenue dropped 9% (-$105K)
- Since total site visits decreased by 68K+ and not 123K+, this means the halo effect from paid media of site visits is ~55K
This means that, despite organic search growing, as it was not being “cannibalized” by paid media, it could not offset the traffic or sales volume that paid search and performance max contributed.
Additionally, the lack of paid awareness media (i.e., display, social, etc.) leads to a contraction of total related searches to the brand name, as illustrated by the aggregate drop in total search traffic to the site, but also a drop in direct to site traffic as well.
“But Jon, they saved on ad spend, that should be helping them come out ahead?”
Wrong.
While they didn’t spend $80K on ads. Thus, the paid media cost per paid site visit dropped from $0.64 to $0. But they lost an aggregate 68K+ visits. On average a visit to the site in the pretest period (for all traffic sources) had a revenue value back to the brand of $3.61, during the test that rose to $4.20 (increased as more direct to site and organic search took a bigger piece of the traffic contribution pie).
This means, the actual Sales Value Impact=((Avg Revenue per Paid Media Site Visit)*Direct Paid Media Visit Lost/Gained)-/+Ad Spend Saved or Spent
Another way to write that formula is SVI=((ARPMSV)*DPMVLG)-/+Ad Spend)
Meaning on the conservative side it would be:
(($2.12)*-123,572)+$79,626.40)= -$182,346.32
But the reality is, organic search rose as it was no longer being cannibalized by paid. But not by enough to offset the loss of paid, so it requires separating the loss into direct impact and halo impact.
The direct impact is similar to the formula above, but you swap paid media visits out for total site visits change (68,652), which then brings the net loss down to $65,915.84.
Then there is a halo effect of paid media on organic. Which is why non-paid visits couldn’t offset the total loss in visits when paid was out. To calculate out the halo effect impact, we would do a formula of:
Halo Sales Value Impact= (((Avg Revenue per Paid Media Site Visit)*((Paid Media Traffic Lost or Gained-Total Traffic Loss or Gained in Test Period))
Or, written as HSVI=(((ARPMSV)*((PMTLG-(PPMT-TTLGTP)))
Meaning on the conservative side, it would be:
((($2.12)*((123,572-68,652)))=$116,430.40
Combine the 2 outcomes together, and you get your loss of $182,346.24 explained.
This means, that by not running paid media, full impact was a net missed revenue opportunity of $182,346.32, between direct and halo.
This goes to an extremely conservative method, and does not account for store visit revenue, or any shifts in revenue per visit over time.
In addition to the traffic that would’ve stood to gain additions to their email/audience lists.
Bringing paid media back
After the 5 weeks offline, we brought media back, in fact we increased investment by 48% (no change of channels, but all incremental went to social awareness and display). This generated 29% fewer clicks from pre-test, but increased impressions 107%.
In the post-test period, vs the test period, the return of media, at the increased investment, lead to a 21% decrease in organic search traffic. But with paid search and performance max generating enough gain to have a net positive in all of search of +6% in site visits. Overall search driven online orders and revenue both saw a 2% increase when paid was reintroduced.
The only true net loss in growth was direct to site (which was believed that it would rise when media was back in market), which decreased in traffic by 6% and orders by 10%. Overall saw a 38% lift in total site visits, but a drop of 1% in online orders (revenue was flat). The loss in online orders was exclusively direct to site traffic.
What Does This All Mean?
A variety of things.
No matter what way you cut it, the presence of paid media had a halo effect on all activity, most notably, the aggregation of paid and organic search.
But the post-click impact on the site may not follow the same path.
Which means, a larger view must be taken to examine additional impact (i.e., foot traffic, loyalty club sign-ups, and LTV).
It also reinforces the concept of 1+1=3, the theory of incrementality. While no other changes were made beyond the exiting of paid digital media, and the brand remained in low season the whole way through, the actual impact of inbound traffic lost (not covered by organic) was considerable.
It also stands as a reminder: Just because a site visit doesn’t generate immediate sales/revenue, it does not mean it doesn’t serve a purpose (i.e., foot traffic).
The Takeaway
Any brand that has more paid media site traffic than non-paid site traffic, and thinks they can turn off paid and coast equally on just non-paid traffic alone, has the same mindset as any NY Jets fan (the inability to accept a very harsh reality).
But, despite my writing, I am an optimist, and I encourage brands to do a similar study, if for no other reason than to have the data on hand for when the CMO comes in saying they want to turn off paid because they don’t think they should pay for it.
But word to the wise: Don’t do what we did here, do a market holdout, so that if things go south, it isn’t system-wide.
More Resources:
Featured Image: Roman Samborskyi/Shutterstock
