Data · 03 / 04 · BigQuery · dbt · Statistics

Product Analytics & A/B Testing

Google's GA4 store data modelled in BigQuery, and an A/B testing toolkit tested on thousands of simulated experiments before reading out a real 14M-user one.

source on GitHub ↗

BigQuery
dbt
SQL
Python
SciPy
scikit-learn
Power BI
Evidence

4.3MGA4 events modelled in dbt

314duplicated purchases caught (6.3% of revenue)

29.7% → 1.5%false positives: daily peeking vs sequential test

14Musers in the real experiment read out

Why I built this

In the real world

Product teams ship or kill features on A/B test results. Peek at the dashboard every morning, pick a noisy metric, or miss that the groups weren't comparable, and you'll launch things that do nothing and kill things that work.

What I wanted to find out

Can you actually trust an A/B test result? I wanted to build the statistics myself, prove they behave on thousands of fake experiments, and then read out a real one where the obvious answer turned out to be wrong.

What I built

Google's public GA4 ecommerce export modelled in BigQuery with dbt: sessions, checkout funnel, retention cohorts, channels and products, with tests that reconcile revenue against an independent deduplication of the raw events.

Then an experimentation toolkit (SRM checks, power, CUPED, the delta method, sequential tests, doubly robust estimation), validated by simulation on real GA4 users before reading out Criteo's 14-million-user randomised ad experiment. Results are published on an Evidence dashboard and in a Power BI report with a sample-size calculator.

How it works

GA4 exportBigQuery public data

dbt13 models, 27 tests

Martsfunnel, cohorts, users

SimulationsA/A, power, peeking

ReadoutCriteo, AIPW

The dashboard is a static Evidence site, rebuilt and deployed to GitHub Pages by CI. The same marts and results also feed a 5-page Power BI report.

How it stays correct

Revenue to the cent

dbt tests reconcile purchases against a separate deduplication of the raw events.

Tests that hold 5%

A/A simulations on real users: false positives between 4.1% and 5.7%.

Peeking is priced in

Daily looks push false positives to 29.7%; mSPRT keeps them at 1.5%.

Balance checked, not assumed

Criteo's split passed SRM but covariates were imbalanced, so four adjusted estimators replace the naive lift.

Measured

GA4: 360,129 sessions, 4,922 purchases; 314 duplicate purchases worth 6.3% of revenue caught
Metric choice: Detecting +10%: revenue per user needs 190 days, add-to-cart rate 17
Criteo: Naive visit lift +27.1%; doubly robust +19.5% (four estimators agree, +19.2% to +20.5%)
Tests: 33 Python tests, 93% coverage; sqlfluff and dbt parse in CI

Power BI experiment page — Experiment page: naive vs adjusted lift, peeking, SRM and a what-if sample-size calculator.

Power BI funnel page — Checkout funnel with drop-off at each step, by device and channel.

Power BI overview page — Overview: sessions, purchases, revenue and conversion by acquisition channel.

Power BI retention page — Weekly cohort retention and the pooled retention curve.

Evidence dashboard page on the Criteo experiment — The live Evidence dashboard: the split passes SRM, but features sit up to 60 standard errors apart.