A 100-Store Causal Inference Experiment
The Predictive Markdown Intelligence model looked great in testing. This project answers the only question that matters: does it actually make more money than the simple rule it is supposed to replace?
Good offline metrics are not proof of business value.
This is a direct extension of Predictive Markdown Intelligence, which built an XGBoost model to optimize markdown timing for fast-fashion retailers.
The model predicts the narrow window between discounting too early and losing profit, or too late and creating dead stock. It performed well in offline evaluation, but a model can look strong in a backtest and still lose money when tested against a real business baseline.
This project designs and executes that controlled experiment.
XGBoost decides when to mark down each product using demand signals.
The industry-standard calendar rule marks products down every three weeks.
1.8M
SKU × Store × Day rows
200 SKUs across 100 stores over 90 days.
Separate geographic regions prevented customer migration between treatment and control stores.
Capacity weights reflected store type, so flagship locations received more demand than small outlets.
Products covered tshirts, jeans, jackets, and shoes with realistic price tiers and demand patterns.
The window included 30 baseline days, 45 active experiment days, and 15 post-experiment days.
Before collecting experiment results, we verified that treatment and control were statistically indistinguishable.
The 50/50 split was clean, with no assignment bugs.
Treatment averaged $11,056/day versus $9,713/day for control. The gap was within random noise.
Both groups had the same mix of flagship, standard, and boutique stores.
Both groups had similar demand potential.
All p-values were above 0.05. Any later divergence can therefore be attributed to the treatment rather than a systematic pre-existing imbalance.
DiD Effect = (Treatment After − Treatment Before) − (Control After − Control Before)
Comparing raw averages would be misleading because treatment stores started with slightly higher revenue. DiD compares each group's change, removing pre-existing differences and market trends affecting both groups.
Parallel trends held.
p = 0.76
Both groups moved together before the experiment. Their later divergence can therefore be interpreted as the treatment effect. DiD is widely used to evaluate pricing and operational experiments.
+$45
per store / day · p = 0.13
More units moved, but the lift was not statistically significant.
−$164
per store / day · p < 0.001
A real, measurable loss on the metric that matters.
−$2,994,713
95% CI [−$3.9M, −$2.1M]
The ML model lost on the metric that matters.
The model discounted four times deeper. It learned its lesson perfectly. It was simply the wrong lesson.
The Tweedie objective optimized units sold, not dollars of profit kept. Inventory cleared quickly, but margin disappeared.
A $120 jacket
ML: discount to $84, sell 10 units, keep $360 margin.
Calendar: keep $120, sell 5 units, keep the same $360 margin.
The model burned through twice the inventory without creating more profit. For dead stock, it also discounted before testing whether customers would pay full price.
Do not launch the model as-is.
The experiment turned an apparent model win into a clear business decision before the loss reached the P&L.
Prevent discounts below cost of goods plus a 25% margin. The model recommends when; the business rule limits how deep.
Use aggressive ML for dead stock, the calendar rule for bestsellers, and a margin-protected hybrid for everything between.
Projected next-phase impact: turn the −$3.0M loss into an estimated +$2.5M gain.
Python, pandas, NumPy, statsmodels, scipy, matplotlib, seaborn, XGBoost with Tweedie loss, DuckDB, Jupyter Notebook, and Git LFS.
A PostgreSQL-dialect star schema with dim_stores, dim_products, dim_experiment_assignments, and fact_daily_sales.
Demand followed a Negative Binomial distribution with capacity-weighted store splitting.
Schema design, store generation, randomization and balance checks, the 1.8M-row fact table generator, and the complete DiD analysis notebook with publication-ready charts.
Companion projectThe XGBoost and Tweedie demand model tested in this controlled experiment.