DuckDB → BigQuery-portable Point-in-time features Leakage-checked

392,690 sale lines to scored customers in 34.49s.

Real UCI Online Retail transactions through layered ETL, a point-in-time feature store, a repeat-purchase model and a data-quality engine. Every figure on this page was written by the pipeline itself — this page is rebuilt by running it, not by editing it.

100%

Pipeline success

10/10 models, p95 latency 541 ms.

134,737

Records quarantined

24.86% caught at ingestion, not silently loaded.

0.731

Propensity ROC-AUC

vs 0.684 baseline — +0.047 improvement.

100/100

Data-quality score

11/11 assertions, 0 critical failures.

This page is generated, not written. The workflow runs make all on a clean runner, then renders whatever reports/*.json the pipeline produced. If the pipeline breaks or a quality assertion fails, that shows up here. Last run 2026-07-24 23:07 UTC · commit 445ddf3 · workflow run

Pipeline run

Ten SQL models across three layers — staging, intermediate, marts — executed in dependency order. Bars show relative execution time; the figure is in the next column.

ModelLayerRows TimeSecondsStatus
stg_invoice_linesstaging392,6900.5414s✓ success
stg_invoice_lines_rejectedstaging134,7370.1708s✓ success
stg_productsstaging4,0700.0212s✓ success
stg_returnsstaging9,2880.0194s✓ success
int_ordersintermediate18,5320.0417s✓ success
mart_cohort_retentionmarts910.0127s✓ success
mart_country_revenuemarts370.0084s✓ success
mart_customer_featuresmarts4,3380.0292s✓ success
mart_product_performancemarts3,6650.0557s✓ success
mart_rfm_segmentsmarts4,3380.0090s✓ success

541,909 raw line items in, 392,690 loaded, 14,482 duplicates removed and 134,737 quarantined (mostly guest checkouts with no customer id) — the difference is accounted for, not dropped.

Point-in-time feature store

8 features computed as of each cutoff date, so a training row only ever sees data that existed at that moment. The label is repeat_purchase_next_90d. ✓ leakage check passed

As-of cutoffRows PositivesBase rate
2011-03-312,1341,15754.22%
2011-06-302,9601,46649.53%
2011-08-313,3171,86456.20%

Repeat-purchase model

Predicts whether a customer orders again within 90 days. Trained on cutoffs 2011-03-31, 2011-06-30 and validated on 2011-08-31 — 5,094 training rows, 3,317 validation rows at a 56.20% base rate. The baseline is the obvious heuristic: rank customers by how recently they last ordered.

MetricModelBaselineΔ
ROC-AUC0.7305 0.6838 +0.0467
PR-AUC0.7949 0.7040 +0.0909
Top-decile lift1.683× 1.360× +0.323
Read this honestly: the model beats a recency-only baseline on every metric — ROC-AUC, PR-AUC and top-decile lift — on real, unseen customers. The top predictor is prior order frequency, which is exactly what you'd expect to drive repeat purchase.

Feature weights

Signed model coefficients. Direction is shown by which side of the axis the bar falls as well as by colour, and every bar carries its value.

▲ increases propensity ▼ decreases propensity
FeatureWeightValue
orders_last_180d▲ +1.3942
units_last_90d▲ +1.0106
avg_line_value_90d▼ -0.9569
distinct_products_90d▲ +0.2320
orders_last_90d▲ +0.2188
revenue_last_90d▼ -0.1242
days_since_last_order▼ -0.1010
revenue_last_180d▲ +0.0354

Data-quality assertions

11 YAML-declared assertions run against the warehouse after every pipeline execution, written to an audit table and scored. Critical failures fail the build.

CheckTypeSeverity ObservedThresholdStatus
sales_volumerow_count_mincritical3.927e+051e+05✓ pass
sales_freshnessfreshnesscritical12✓ pass
customer_id_not_nullnot_nullcritical00✓ pass
unit_price_positivemin_valuehigh0.0010✓ pass
quantity_positivemin_valuehigh11✓ pass
orders_reconcile_linesreconcile_counthigh1.853e+041.853e+04✓ pass
customer_features_uniqueuniquehigh00✓ pass
monetary_non_negativemin_valuehigh3.750✓ pass
rfm_segments_not_orphanedrelationshipmedium00✓ pass
propensity_scores_not_orphanedrelationshipmedium00✓ pass
unattributable_rows_within_sloquarantine_ratemedium0.24860.3✓ pass

How it is put together

01
Ingest Real UCI Online Retail invoices; guest checkouts and cancellations quarantined at the boundary.
02
Model Layered SQL: staging → intermediate → marts, run in dependency order.
03
Features Config-driven point-in-time store, backfilled per cutoff.
04
Train Walk-forward fit, scored back into the warehouse as a mart.
05
Assert Quality checks to an audit table; critical failures break the build.

It runs locally on DuckDB with no cloud account and no cost, and the same modelling logic ports to BigQuery and Dataform — see /bigquery.

make setup make all pytest -q