AI / ML · 01 / 03 · LightGBM · MLflow · FastAPI
Card Fraud Detection
A fraud model on 590,000 real card transactions, built the way production needs it: tested on the future, priced in dollars, explained per decision.
- Python
- LightGBM
- scikit-learn
- Optuna
- MLflow
- FastAPI
- Polars
- SHAP
Why I built this
Card fraud is a race against the clock: a bank has milliseconds to approve or stop a payment, and fraud teams can only review a few percent of transactions. Models that look brilliant in a notebook often fall apart in production because they were tested on data from the same weeks they trained on.
Why do fraud models that score well in notebooks disappoint in production? I wanted to build one the honest way: train on the past, test on the future, only use labels the bank would actually have by then, and turn scores into decisions that save money.
What I built
Vesta's IEEE-CIS data: 590,540 real e-commerce card transactions, 3.5% fraud. LightGBM trained on four months, tuned on the fifth, reported on the sixth. Customer features only look at the past, including known fraud counted only once a chargeback could have arrived, 30 days later.
Scores go through a calibrated expected-cost rule (review when the expected loss beats the review cost), models are versioned in MLflow with champion/challenger promotion, and a FastAPI service returns every decision with its top five SHAP reasons.
How it works
The same features are computed twice, vectorised for training and one event at a time for the API; a replay proves they agree on every value.
How it stays correct
Train days 1-119, tune 120-151, report 152-182. A random split would have claimed PR-AUC 0.87; the honest number is 0.64.
Past fraud counts only after 30 days. With 7-day labels the model reaches 0.71; with 90 days, 0.57.
Review when p × (amount + fee) > review cost: 68.5% of fraud cost saved, vs 34% for a 0.5 cutoff.
14.2M feature values and 5,000 API scores matched the batch pipeline exactly.
Measured
- Model
- Test PR-AUC 0.644, ROC-AUC 0.936; +0.116 over logistic regression (95% CI +0.105 to +0.128)
- Cost
- $540k → $170k on the test month, 9% of transactions reviewed, 85% of fraud dollars caught
- Serving
- 5,000 HTTP requests: p95 1.1 ms, 8.2 ms with SHAP reasons
- Drift
- Score PSI stayed under 0.11 while weekly PR-AUC fell from 0.81 to about 0.6: drift metrics alone would miss it