Supply Chain LabConcept Drift · ML operations gameLab Home
Machine-learning lifecycle game

Concept Drift Control Room

Operate a cold-chain delivery-risk model for 60 days. The model begins accurate, but the Australian freight environment changes. Decide when to keep watching, investigate a signal or retrain - without confusing one noisy day with genuine concept drift.

1 · MonitorAccuracy, recall, calibration and feature drift arrive as live batch metrics.
2 · DiagnoseA bad day can be noise; a sustained relationship change is concept drift.
3 · InterveneInvestigate before acting, or retrain when evidence supports a new model.
4 · GovernEvery intervention has cost, delay and a model-version audit trail.

Australian cold-chain risk stream

100 labelled shipments arrive per simulated day.

Day 1 · decision due
Live machine-learning monitoring timelineDaily model accuracy and population stability metrics update as freight conditions change and the operator retrains model versions.
1 / 60Stream day
- Batch accuracy
- Risk recall
v1Production model

Monitoring signal

Choose a response after reading the signal pattern.

Operations log

  • Model v1 online. Baseline period begins.

Today’s 100 shipments

Why recall matters

False negatives are risky shipments the model misses. In cold-chain logistics, these shipments may arrive late or outside temperature limits without preventive action.

Production versus training baseline

Population Stability Index

PSI detects distribution movement, not whether predictions are wrong. Pair feature drift with labelled performance and operational context.

Production model card

Version audit trail

About this tool

The concept

Concept drift occurs when the relationship between inputs and the target changes. A model can remain technically online while its decisions silently become less reliable. An anomaly is a short-lived data issue; drift is a sustained change.

How it works

Each simulated day brings 100 labelled shipments. Read accuracy, recall, PSI and the confusion matrix, then keep monitoring, investigate, or retrain and validate. Every intervention has a cost and a model-version audit trail.

Winning strategy

Do not diagnose from one metric alone. Investigate a bad day before acting, and retrain only when PSI stays high over several days. Retraining on every anomaly makes the model less stable.