Incrementality,explained withoutthe statistics degree.

Why attributed conversions overstate the effect

An attribution report works by matching. A person sees or taps an ad, later makes a purchase, and the system credits the ad. The logic is simple, and it contains a flaw that matters most in retargeting: the people you target are the people most likely to return anyway.

Consider who sits in a retargeting audience. They installed your app, used it, and went quiet. Some of them will open it again next week because of a notification, a payday or a free evening. If an ad happens to reach them shortly before they return, the report credits the ad. The ad may have changed nothing. The person was already on the way.

This is why the same campaign can look excellent in an attribution report and do little for the business. The report answers the question "who converted after exposure". It cannot answer "who would have converted without it". Only the second question tells you whether the money earned anything.

The counterfactual

The missing piece has a name: the counterfactual. It is what would have happened to the same people if you had not advertised to them. You can never observe it for one person, because a person is either shown the ad or is not. You can only observe it for groups.

The idea is to build two groups that are alike in every way that matters, treat one differently, and compare. If the groups are truly alike, any difference in outcome afterwards can only come from the one thing that differed. Everything else in incrementality testing is a method for making that sentence true, and for noticing when it has stopped being true.

Randomised holdouts

The most dependable way to build two alike groups is to let chance do it. Take the audience, assign each person to the exposed group or the holdout group by a random draw, and show the holdout nothing from the campaign. Because the draw ignores everything about the person, how often they used the app, what they spent, whether they are loyal, the two groups end up with the same mix of keen and indifferent people. Both contain the person who would return next week regardless.

Here is an illustration, with invented figures chosen for round arithmetic. Imagine a thousand lapsed shoppers, split at random into two equal halves. Suppose that over the following fortnight a certain share of each half comes back and buys. If the exposed half returns at a higher rate than the holdout half, and the groups are large enough that chance alone is an unlikely explanation, the difference is the lift. The holdout's own return rate is the baseline: the shoppers who would have come back with no ad at all. An attribution report would have claimed all of them.

Intent to treat, ghost ads and placebos

A holdout is simple to describe and slightly awkward to run, because of one fact: assigning someone to be exposed does not guarantee that they see an ad. Many people in the exposed group never open an app that has matching inventory in the period. There are three common ways to handle this.

Intent to treat

Compare everyone assigned to the exposed group with everyone assigned to the holdout, whether or not an ad was actually delivered. This is the cleanest design, because the groups stay identical to the moment of assignment. Its drawback is dilution. People who were never reached sit in the exposed group and pull its result toward the holdout's, so the measured effect is smaller than the effect on people who saw ads. That is an honest number for what the campaign delivered, reach included.

Ghost ads

Here the system also records what it would have done for the holdout. When a holdout member's app sends a request and the bidder would have won it, the system logs a ghost impression and shows something else, or nothing. Now the comparison can be limited to people who were reachable: those who received a real ad on one side and those who generated a ghost ad on the other. It removes the dilution without breaking the randomisation, which is why it is the preferred design when the bidding system can support it.

Public service placebos

The older approach shows the holdout an unrelated message in the same slots, often a public service announcement. Both groups are then reached and both see an ad, so comparing them is fair. The cost is real: you pay for impressions that carry no message of yours, and the placebo itself can occasionally nudge behaviour.

How big a holdout needs to be

There is no universal size, but the reasoning is easy to follow. A small group gives a noisy answer, because a handful of unusual people can swing the result. Noise shrinks with the square root of the group size, so halving the uncertainty takes four times as many people.

Three things set the requirement. The first is the baseline rate: if only a few in a hundred of your holdout would return anyway, you need many people to see a difference in so rare an event. The second is the size of the effect you want to detect: a large effect shows up in a small group, a modest one needs a large group. The third is how much risk you will accept of being fooled by chance.

The trade-off is cost. Every person in the holdout is someone you chose not to reach. A larger holdout gives a sharper answer and forgoes more potential return while the test runs. Most teams settle on a modest share of the audience and let the test run long enough to gather the numbers it needs, rather than enlarging the holdout.

Reading a lift result

A lift result has two parts that must be read together: the estimate and its uncertainty. An estimate of a positive lift with a wide range around it, one that comfortably includes zero, is not evidence of an effect. It is a test that did not collect enough data to say. Read it as "unknown". It supports neither a claim that the campaign works nor a claim that it fails.

Two further habits help. Look at lift in absolute terms as well as relative. A large relative lift on a tiny baseline can mean very few additional conversions. And look at lift on the outcome that pays, such as purchases or subscriptions, since a campaign can raise app opens while changing nothing that matters to revenue.

Common ways tests go wrong

• Contamination. The same person lands in both groups, or a holdout member is reached through another channel or another campaign. Randomise at the level of the person, and suppress holdout members everywhere you advertise, or the groups blur together.
• Peeking early. Early results swing wildly because few people have converted. Stopping the moment the lines separate favourably finds false winners. Decide the duration and the group size before the test starts and read it once.
• Changing the audience mid-test. Adding a new segment, loosening a filter or switching the creative halfway changes who is in each group, or what the exposed group experiences. Treat any change as the start of a new test.
• Unequal treatment. If the exposed group also receives push messages the holdout does not, the test measures the whole bundle of messages, and the ads are only one part of it.

What to do with the answer

Divide spend by incremental conversions to get the cost of an additional result, then compare it with what that result is worth. Hold budgets against that figure. The cost per attributed conversion flatters the spend.

Use the answer to make decisions the attribution report could not. Shift budget toward stages and windows where lift is strong, which is often among lapsed and dormant users rather than recently active ones. Reduce or end spend where lift is close to zero. Then test again, because audiences, creative and seasons change. Incrementality is a habit, repeated, and the first test is the cheapest of the lot.