Monitor After Launch

The concept

A deployed model encounters conditions that may differ from its training and evaluation data. Input distributions, user behavior, and the relationship between inputs and outcomes can shift. Monitoring should ask whether the system still serves its purpose, rather than merely whether a server is running and producing outputs on time.

Worked example

A fictional bicycle-share operator predicts empty docking stations from past trip patterns. A new bridge opens, changing commuter routes. Predictions remain technically available but become less helpful near the bridge. The operator tracks errors by neighborhood and time of day, compares recent weeks with the pilot, and temporarily uses a simpler fallback in the affected area while collecting new evidence. Overall averages would have obscured the local deterioration.

Limits and judgment

Not every fluctuation means the model has failed. Small samples and seasonal events can produce noisy signals. Define investigation thresholds, identify who can pause the system, and retain records that support diagnosis. Monitoring without a response plan simply creates another dashboard. Any retraining should receive a fresh evaluation, because improving recent performance can damage previously reliable cases.

RETRIEVE. REASON. REMEMBER.

Knowledge check.

Choose one answer for each question, then check your reasoning.