High AUC is not a decision: one churn system, five datasets
What we measured
Every dataset went through the same product path. The test share was 25% of the data, split at random and stratified. Decisions were produced only for customers who had not yet left.
| Dataset | Rows | AUC (95% interval) | Decided | Intervene | Review | Monthly value |
|---|---|---|---|---|---|---|
| IBM Telco (control) | 7,043 | 0.845 (0.825-0.863) | 5,174 | 549 | 439 | In the data |
| Bank customers | 10,000 | 0.763 (0.739-0.786) | 7,963 | 0 | 8 | 30, assumed |
| Credit card | 10,127 | 0.931 (0.919-0.943) | 8,500 | 0 | 0 | 20, assumed |
| Iranian telecom | 3,150 | 0.932 (0.910-0.952) | 2,655 | 0 | 0 | 20, assumed |
| E-commerce | 3,941 | 0.902 (0.875-0.925) | 3,267 | 0 | 67 | 40, assumed |
Action assumptions (cost / success rate): 15 / 0.22, 30 / 0.30, 45 / 0.38. Values are in each dataset's own currency or unitless and are not comparable across datasets.
Why zero?
In simple form, expected value = protected value × risk × intervention effect − cost. The engine also applies margin, a 12-month horizon and discounting, so its net value does not match this simple formula exactly.
Take the riskiest customer in the bank data: 89.5% risk, an assumed value of 30 a month. The best action's net value was only 1.95. There is not enough value to carry the spend. The decision was "review", not "intervene".
What happened when the value column changed
We reran the same data with a money column declared as the value. These columns are not real customer profit. They only show how sensitive the decision is to this input.
| Dataset | Value column | AUC | Intervene |
|---|---|---|---|
| Bank | Fixed 30 | 0.763 | 0 |
| Bank | Balance | 0.758 | 2,405 |
| Bank | Estimated salary | 0.779 | 3,599 |
| Credit card | Fixed 20 | 0.931 | 0 |
| Credit card | Annual spend | 0.899 | 1,032 |
| E-commerce | Fixed 40 | 0.902 | 0 |
| E-commerce | Cashback amount | 0.863 | 439 |
On credit card and e-commerce, AUC fell while intervene decisions rose from zero. In this test, the value input moved the decision more than model performance did.
What this test does not show
- It does not show that interventions keep customers. The effect rates are assumptions.
- It does not show that the intervention counts are right. A balance or salary is not what the bank earns from that customer.
- It does not measure causal effect. That needs a randomised control group, which these datasets lack.
- It does not measure drift over time. The split was random because these datasets had no start date to order customers by.
What to do
- Put the decision next to AUC in the model report: how many customers, at what cost?
- Take customer value from the data instead of assuming one number. Use margin rather than revenue where you can.
- Measure intervention effect with a randomised control group.
The Iranian and e-commerce data had no ID column, so we added row numbers. We rounded the e-commerce cashback amount to two decimals. In the credit card data we kept two ready-made prediction columns out of the model.
Sources
See this arithmetic on your own data.
Vinctum Aegis is looking for its first design partner: one data table, a 30-minute call and a controlled measurement.
info@vinctum.studio