High AUC is not a decision: one churn system, five datasets

2 November 20265 min read

What we measured

Every dataset went through the same product path. The test share was 25% of the data, split at random and stratified. Decisions were produced only for customers who had not yet left.

DatasetRowsAUC (95% interval)DecidedInterveneReviewMonthly value
IBM Telco (control)7,0430.845 (0.825-0.863)5,174549439In the data
Bank customers10,0000.763 (0.739-0.786)7,9630830, assumed
Credit card10,1270.931 (0.919-0.943)8,5000020, assumed
Iranian telecom3,1500.932 (0.910-0.952)2,6550020, assumed
E-commerce3,9410.902 (0.875-0.925)3,26706740, assumed

Action assumptions (cost / success rate): 15 / 0.22, 30 / 0.30, 45 / 0.38. Values are in each dataset's own currency or unitless and are not comparable across datasets.

Why zero?

In simple form, expected value = protected value × risk × intervention effect − cost. The engine also applies margin, a 12-month horizon and discounting, so its net value does not match this simple formula exactly.

Take the riskiest customer in the bank data: 89.5% risk, an assumed value of 30 a month. The best action's net value was only 1.95. There is not enough value to carry the spend. The decision was "review", not "intervene".

What happened when the value column changed

We reran the same data with a money column declared as the value. These columns are not real customer profit. They only show how sensitive the decision is to this input.

DatasetValue columnAUCIntervene
BankFixed 300.7630
BankBalance0.7582,405
BankEstimated salary0.7793,599
Credit cardFixed 200.9310
Credit cardAnnual spend0.8991,032
E-commerceFixed 400.9020
E-commerceCashback amount0.863439

On credit card and e-commerce, AUC fell while intervene decisions rose from zero. In this test, the value input moved the decision more than model performance did.

What this test does not show

  • It does not show that interventions keep customers. The effect rates are assumptions.
  • It does not show that the intervention counts are right. A balance or salary is not what the bank earns from that customer.
  • It does not measure causal effect. That needs a randomised control group, which these datasets lack.
  • It does not measure drift over time. The split was random because these datasets had no start date to order customers by.

What to do

  1. Put the decision next to AUC in the model report: how many customers, at what cost?
  2. Take customer value from the data instead of assuming one number. Use margin rather than revenue where you can.
  3. Measure intervention effect with a randomised control group.

The Iranian and e-commerce data had no ID column, so we added row numbers. We rounded the e-commerce cashback amount to two decimals. In the credit card data we kept two ready-made prediction columns out of the model.

Sources

See this arithmetic on your own data.

Vinctum Aegis is looking for its first design partner: one data table, a 30-minute call and a controlled measurement.

Request a pilot Try the simulator
info@vinctum.studio

More writing

How accurate are weather forecasts? 7.3M forecasts, 30 citiesWe scored 7,358,400 archived forecasts in 30 Turkish cities against a free guess. In eight cities the hourly rain forecast never beat "it will not rain".A churn prediction is not a decision: from risk score to actionA churn model tells you who is likely to leave, not whom to spend on. A worked example of why the same 78% risk means two different decisions for two customers.