Why the Same Marketing Play Works for One Business and Not Another
A tactic that doubled a competitor’s leads can do nothing for you — not because the tactic is bad, but because effects depend on context. This is marketing heterogeneity, and it’s the reason a case study is never proof that something will work in your business.
By William Walczak — CEO, Hiilite Creative Group Inc.The uncomfortable truth: there is no fixed effect
Marketers love to talk about “what works.” Run loyalty emails, they convert. Add a video to the landing page, conversion goes up. Switch to this bidding strategy, costs drop. The hidden assumption is that a tactic has a single, transferable effect — that if it lifted someone else’s revenue by 20%, it should lift yours by roughly the same.
It rarely does. The size and even the direction of a tactic’s effect depends on who you’re reaching, the market you’re in, the offer you’re making, the timing, and where you started from. Statisticians call this heterogeneity of treatment effects: the same “treatment” produces different outcomes across different units. In plain English — the same play, played in two different contexts, gives two different results.
Five reasons the same play lands differently
When a tactic over-performs for one business and disappoints another, the gap almost always traces back to a handful of context variables. None of them are visible in the headline number a case study reports.
- Audience. A retargeting push works when you have a warm audience to retarget. A brand with thin traffic has almost no one to re-reach, so the same campaign barely moves.
- Market. Demand, competition, and seasonality differ by category and geography. A play that wins in a low-competition niche can get buried in a saturated one.
- Offer. The same email cadence behind a strong, differentiated offer converts; behind a me-too offer it just annoys people. The tactic didn’t change — the thing it was selling did.
- Timing. Launching during a buying spike versus a dead season can swing results several-fold, with the tactic getting all the credit or all the blame.
- Baseline. If you’re already doing well on a channel, there’s little headroom left to capture — the same effort yields a smaller lift than it would for someone starting from zero.
Why a competitor’s win is not your proof
This is the core problem with case-study envy. When a competitor reports a result, you’re seeing the outcome of their audience, market, offer, timing, and baseline — bundled together and labelled as the effect of one tactic. You can copy the tactic exactly and still get a different number, because you can’t copy their context.
There’s a second trap underneath the first. Most reported “wins” aren’t even clean measures of the tactic. They’re before-and-after revenue numbers that quietly take credit for everything else happening at the same time — a seasonal bump, a new product, a PR mention. To know what a play actually caused, you need to compare against what would have happened without it — the counterfactual. A competitor’s testimonial almost never contains one.
So a case study is useful as a hypothesis — “this might be worth trying” — but it is never evidence that it will work for you. Treat borrowed results as ideas to test, not conclusions to copy.
What to do instead: test on your own counterfactual
If effects are context-dependent, the only honest way to know whether a play works for you is to measure it on your own numbers, against your own baseline. That means estimating incrementality — the lift a play causes above what would have happened anyway — rather than trusting reported totals.
You don’t need a research lab to do this. A geo holdout, a staggered rollout, or a simple A/B split gives you a comparison group and turns “revenue went up” into “this play caused X.” Our methods paper walks through how these methods stack up on the Measurement Ladder, and the incrementality calculator lets you sketch the math before you commit budget.
The payoff is compounding. Every test you run on your own context teaches you something a competitor’s case study never could — and it stops you pouring money into plays that look great in someone else’s slide deck and do nothing in yours.
How Hiilite handles heterogeneity
We build heterogeneity into how we work rather than pretending it away. First, we measure incrementality on your numbers, not industry benchmarks or borrowed results — so the lift we report is the lift you actually got. Second, we codify tactics as plays scoped to a context, not universal best practices.
A play in our system carries its conditions: which audience it suits, which offer it assumes, which baseline it expects. That’s the difference between an experiment, a play, and a capability — a one-off test becomes a reusable play only once we know where it works. As you run our Sense → Seize → Transform loop, your own library of context-tagged plays grows, and the guesswork shrinks.
The honest version of “what works in marketing” is “it depends” — and the job is to find out, cheaply and quickly, what it depends on for you.
Keep reading
Questions, answered
What is marketing heterogeneity?
It’s the principle that the same marketing tactic produces different results across different businesses. The size and even direction of a tactic’s effect depends on audience, market, offer, timing, and starting baseline — so there’s no single fixed effect that transfers between companies.
If a tactic worked for a competitor, why might it not work for me?
Because you can copy the tactic but not their context. Their reported result reflects their audience, offer, timing, and baseline all bundled together. Most case studies also lack a counterfactual, so they overstate what the tactic alone caused. Treat a competitor win as a hypothesis to test, not proof.
How do I know if a play actually works for my business?
Measure incrementality on your own numbers using a comparison group — a geo holdout, staggered rollout, or A/B split. That isolates the lift the play caused above your baseline, instead of trusting before-and-after totals that quietly credit everything else happening at the same time.
More from this series
Seven plain-English pieces on measuring marketing and turning tests into durable capability:
Want to know whether your marketing is the reason?
One conversation to map what to measure and how we’d prove it.