How do you run a holdout test for retention messaging?
Pick a random share of users, often around 10%, and send them no retention messages at all. Compare their retention or revenue with everyone else's over the same period. The gap is what your messages changed, once both groups are large enough and the gap beats its margin of error.
The test in six steps
| Step | What you do |
|---|---|
| 1. Pick the outcome | Retention at 30, 60 or 90 days, renewals, or revenue per user |
| 2. Pick the share | Often around 10% of users |
| 3. Assign at random, once | A stable hash of the user id, so a user stays in their group |
| 4. Keep the holdout clean | No retention messages of any kind to that group |
| 5. Wait for enough users | Both groups large enough for the gap to beat the noise |
| 6. Compare the groups | The difference, with its margin of error |
1. Pick the outcome before you start
Decide what “kept” means for your app: still subscribed after 30 days, renewed at the next billing date, or revenue per user over 90 days. Pick one main outcome before you look at the results, so you can’t go looking for a number that happens to look good.
Opens and clicks aren’t outcomes. A message can get clicked by a user who would have stayed anyway.
2. Pick the size of the holdout
A larger holdout gives a clearer answer sooner but means more users miss messages. Around 10% is a common starting point. A small app may need a larger share, or a longer test, to get enough users into the holdout.
3. Assign users at random, and keep them there
Use a random, stable assignment, such as a hash of the user id, so the same user always lands in the same group. Don’t let anyone hand-pick the holdout, and don’t reassign users partway through. The groups must differ only in whether they got messages.
4. Keep the holdout clean
The holdout gets no retention messages from any tool. If your email tool sends a win-back campaign to everyone, the holdout is broken. Transactional messages such as receipts and password resets still go to everyone.
5. Wait for enough users
With few users, random noise can make either group look better. Before you read the result, both groups need enough users, often a few hundred each at minimum, and more if the effect is small. Report the gap with its margin of error, for example a 95% confidence interval, not as a single number.
6. Read the result honestly
If the interval for the difference sits clearly above zero, the messages are keeping users. If it straddles zero, you can’t tell yet: keep the test running or accept that the effect is small. If the holdout does better, some messages are hurting, often through fatigue or unsubscribes, and you should send fewer.
Where Whisperr fits
Whisperr runs this test by default. It holds back 10% of users at random, kept stable by a hash of the app and user, and never messages them. Once each group has 200 users, the dashboard shows the difference in retention and revenue kept with a 95% confidence interval. Whisperr records the outcome of every message and measures it against the holdout, so the number you see is what the messages changed, not what they touched.
FAQ
How big should a holdout group be?
Big enough to measure a difference, small enough not to cost much. Around 10% is a common starting point. If your app is small, you need a larger share or a longer test to reach enough users in the holdout.
How long should a holdout test run?
At least one full renewal cycle for the outcome you care about, for example a month for monthly subscriptions. Shorter tests measure opens and sessions, not whether users kept paying.
Isn't it bad to withhold messages from users?
The holdout only misses retention messages, not receipts, password resets or anything they need. Without it you can't know whether the messages help, and messages that don't help cost attention and unsubscribes.
What's the difference between a holdout test and an A/B test?
An A/B test compares two versions of a message. A holdout test compares sending against not sending. A message can win an A/B test and still keep no more users than sending nothing at all.
Does Whisperr run a holdout?
Yes, by default. Whisperr holds back 10% of users, chosen at random and kept stable, and never messages them. The dashboard compares retention and revenue kept between the two groups, with a 95% confidence interval, once each group has 200 users.
Let Whisperr run this for you.
Whisperr watches how people use your app, spots the ones about to leave and brings them back with the right message at the right moment. After you sign up, one prompt in your coding agent sets it up.
You approve until you trust it. Then it runs itself.