Incrementality Testing: How a Bank Knows a Program Worked

Incrementality testing holds back a random slice of eligible customers and does not send them the program. The difference between the two groups, in the share who go on to fund or activate, is the lift.

Take a case, with figures chosen to make the arithmetic easy rather than taken from any client. A regional bank has 60,000 checking accounts that were opened and never funded. The retail team runs a 90-day campaign to get them funded, using email and a text reminder. At day 90, 7,200 of those customers have funded. The team reports 7,200 funded accounts and the deposits that came with them.

Some of those 7,200 would have funded anyway. People open an account with a plan. Payroll changes over, or a house sale closes. Some share of the base funds every quarter with no help from anyone. The campaign report counts those customers too. It cannot say how many of the 7,200 the campaign added.

Why the usual comparisons don't answer it

Before and after on the same group

The obvious check is to compare the funding rate before the campaign with the rate during it. Suppose the bank raised its savings rate in week 3, and quarter end fell in week 10. A before-and-after comparison credits the rate rise and the quarter end to the campaign.

The campaign group against the rest of the base

Comparing the customers who got the campaign with those who didn't fails when the campaign group was chosen. Marketing usually picks the customers most likely to respond, such as recent openers who have already logged in to the app. Those customers fund at a higher rate whether or not anyone contacts them.

This year against last year

Last year's funding rate has the same trouble. Deposit rates moved since then, and this year's customers came in through different channels. If funding runs 2 points above last year, the rate move and the change in channel mix explain it as well as the campaign does, and a year-over-year figure cannot separate the three.

What incrementality testing is

Take every customer who is eligible for the program. Split them at random. One part gets the program. The other part gets nothing and stays in the ordinary flow of the bank. Set a window, say 90 days. At the end of it, measure the share of each group that completed. The difference between the two shares is the lift.

Both groups live through the same rate change and the same quarter end, and both are drawn by the same eligibility rule. The only difference is that one group got the program. So the held-back group funds, on average, at the rate the treated group would have reached on its own, which is the figure a bank otherwise has to assume.

What goes in the message, and why a customer stalled in the first place, is a separate subject. It is covered in a companion piece on growing deposits from customers you already have.

Five conditions the test has to meet

• The bank assigns customers to the two groups at random, and records which group each customer landed in.
• Both groups come from the same eligibility rule, so anyone in one group could have been in the other.
• The team tests one change at a time.
• The team writes down the outcome and the measurement window before the program starts, and reads the result when the window closes rather than weekly.
• The held-back group gets nothing from the program until the window closes, and is kept out of other campaigns aimed at the same outcome.

The arithmetic, once

The figures below are illustrative.

The bank has 40,000 eligible unfunded accounts. It holds back 4,000 at random and treats 36,000. At day 90, 12 percent of the treated group has funded. In the held-back group, 10 percent has funded. The lift is 2 percentage points, which is the distance between 12 and 10.

Apply the lift to the treated group. 36,000 times 2 percentage points is 720 accounts. Those are the accounts the campaign added. Put a value on a funded account. Use $200 a year in net interest and fees, and assume the balance stays for a year. Every bank has its own figure for this and should use it. On $200, the campaign is worth about $144,000.

The campaign report would have counted every funded account in the treated group. 12 percent of 36,000 is 4,320 accounts. At the same $200 each, that is $864,000, six times the $144,000 the test measured. That multiple depends on the base rate: the lower the share who would have funded anyway, the closer the two figures sit. The other 3,600 funded on their own, which is what the 10 percent in the held-back group shows.

The 2 points carries a confidence interval, because customers were assigned to the two groups at random and the split could have fallen differently. With these group sizes the 95 percent interval runs from about 1 point to about 3 points. The dollar figure inherits that range, so the report reads about $144,000, between about $72,000 and $216,000. The $200 per account and the one-year assumption carry their own uncertainty on top of that.

How big the held-back group has to be

The size depends on the base completion rate and on the smallest lift worth detecting. At a 10 percent base rate, an even split needs roughly 3,800 customers in each group to detect a 2-point lift, and close to 58,000 in each group to detect half a point. Those illustrative figures assume equal groups, 95 percent confidence, and an 80 percent chance of detecting the lift if it is there. An uneven split also works, and the small group sets the limit. In the example, almost all the uncertainty comes from the 4,000 held back, and adding more customers to the 36,000 would barely improve it. That design can detect about 1.5 points, which is why 4,000 was held back before the run rather than chosen afterward. Any team proposing a test should state the smallest lift the design can detect before the test runs.

What it costs

The held-back customers do not get the program for the length of the test. If the program works, the bank forgoes the lift on them. In the example above, that is 2 percentage points on 4,000 customers, or about 80 accounts and $16,000 for the quarter. The bank gives up that $16,000 to find out whether the other $144,000 is real. After day 90 the held-back 4,000 get the program like everyone else.

What incrementality testing cannot tell you

• It measures a group average. It cannot say which individual customer the program moved.
• It applies to this program, this population, and this window. A lift measured in spring on unfunded checking accounts says little about a card activation program in the fall.
• It cannot separate two changes made at once. A new message sent on a new schedule produces one lift with two causes.
• A lift can clear the statistical bar and still be too small to pay for the program. A significance test says how unlikely the measured difference would be if the true lift were zero. The dollar figure says whether the lift covers what the program costs.

Where to run the first one

Start with a funding or activation outcome. An account is funded or it isn't, and a card is activated or it isn't. The value of each completion can be stated up front, without an argument about which channel deserves the credit. A funded-account count is already in the bank's daily reporting, so there is nothing new to agree on before the test can start.

Boards are asking harder questions about what technology returns, and most answers are still thin. Evident, a firm that benchmarks banks on their use of AI, looked at 50 of the world's largest banks in 2025. Only 4 of them reported realized ROI from AI use cases, according to Deloitte's 2026 banking and capital markets outlook, published in October 2025. A test with a held-back group is how a program avoids joining that list.

Lending is slower to start, because credit, fair lending and Community Reinvestment Act review pull more people into any test that changes who gets contacted. The retail banking page sets out the funding steps, from opened to funded, where these tests usually start.

Frequently asked questions

What if we cannot hold anyone back?

Some programs have to reach everyone, for a regulatory or a service reason. Those cannot be measured this way, and the honest position is to say the result is unproven rather than report a before-and-after number as if it were the lift.

How is it different from a before-and-after report?

A before-and-after report compares the same customers across two periods, so a rate change or a seasonal swing between those periods is counted as part of the result. An incrementality test compares two groups over the same period, so a rate move or a competitor's promotion hits both groups alike.

How do we stop another team contacting the held-back group?

Put the held-back customers on a suppression list that every campaign owner works from, and check the list against what actually went out at the end of the window. If some of them were contacted anyway, the measured lift is pushed toward zero, so the result becomes a floor rather than an estimate.

Can we run one without new software?

Yes. A bank's own data warehouse can split the list at random, flag who was held back, and count who funded by day 90. The work that takes real effort is keeping the held-back group out of every other campaign for those 90 days.

How Numr does it

We run every PXI program with a random holdout from the first send. We report the lift as treated versus held back, in dollars, with its 95 percent confidence interval, and we give the client the arithmetic behind the figure.