Retention Copilot

Overview

Built a churn model on the public Kaggle Bank Customer Churn dataset (10,000 customers) and connected it to a retention decision. Instead of picking the model with the best ROC-AUC, the project selects and evaluates it by the profit of the campaign it recommends.

A Streamlit app scores the customer base and recalculates the contact list when the campaign cost or the assumed success rate changes. Every financial figure is a simulation under explicit assumptions, not observed bank revenue.

Problem

Ranking customers by churn probability ignores how much value a call could preserve, and a fixed 0.5 threshold doesn’t respond to changes in campaign cost or effectiveness. The question is which customers to contact to maximize the expected benefit of a retention campaign.

Approach

Estimated each customer’s annual value as 0.025 × balance + 80 × products + 150 × credit card indicator, with balance capped at the training set’s 99th percentile. The coefficients are illustrative assumptions, and the result is annual revenue, not full customer lifetime value.

A customer is recommended for contact only when P(churn) × value × success rate − contact cost is positive, and recommendations are ranked by that expected benefit. Setting the success rate to zero correctly recommends no campaign at all.

Methodology

1. Stratified 60/20/20 split

   6,000 training, 2,000 validation and 2,000 test customers, reproducible with a fixed seed.

2. Three calibrated candidates

   Logistic Regression, Random Forest and XGBoost, each wrapped in 3-fold sigmoid calibration fitted inside the training set. Imputation, p99 caps, scaling and encoding are fitted within each fold.

3. Selection by economic outcome

   The selected model is the one whose contact policy yields the highest simulated profit on validation. No threshold is tuned on the test set.

4. Held-out evaluation with baselines

   The selected model is evaluated once on the test set against no action, contacting everyone and a random selection of the same size (using its analytical expectation).

5. Tested pipeline

   17 unit tests on synthetic data cover the split, the transformations, the profit definitions and the app; GitHub Actions runs them on Python 3.11.

Results / Key insights

Financial results are simulations under explicit assumptions (US$15 per contact, 30% retention success). They are not observed bank revenue or causal effects, and annual customer value is not full CLV.

customers analyzed

10,000

ROC-AUC (held-out test)

0.855

Average Precision (held-out test)

0.698

fewer contacts than contacting everyone

31.5%

more simulated incremental profit than contacting everyone

US$6,405

• The policy contacts 1,370 of 2,000 test customers, 630 fewer than contacting everyone, for US$6,405 more simulated profit.
• Against a random selection of the same size, it adds US$91,967.78 in expected simulated profit.
• Test Brier score: 0.104.
• XGBoost beat Random Forest by only about US$54 of simulated profit on validation. That gap is too small to claim one model is better: results come from a single fixed split without confidence intervals.

Technology

• Python
• pandas
• NumPy
• Scikit-learn
• XGBoost
• Streamlit
• GitHub Actions

Limitations

• There is no intervention data: retention success is a single assumed rate, not a causal estimate. The project predicts churn and simulates decisions; it doesn’t show that a call causes retention.
• Annual customer value omits lifetime horizon, discounting, full costs and differences between offers.
• The split is random, so it doesn’t demonstrate stability over time or generalization to another bank.
• Calibration is fitted with cross-validation but still needs evaluation by segment and out of sample.
• The model uses geography and gender, and no fairness assessment has been done.
• The app is local, with no authentication or controls for a real customer base.