Managing and monitoring AI systems after deployment: inventory, drift, and champion/challenger testing
A model's quiet failure mode is the world changing underneath it. Monitoring after deployment exists because performance at launch says little about performance a year later.
Why this matters for the AIGP exam
Post-deployment scenarios almost always involve some form of drift, and the tested skill is naming it correctly and knowing the governance response.
The essentials
- An AI inventory comes first. You cannot monitor systems you have not catalogued; the inventory — what runs where, doing what, owned by whom — is the foundation of every later control.
- Data drift: the input distribution changes — new customer demographics, new sensors, new vocabulary — so the model sees data unlike its training set.
- Concept drift: the relationship between inputs and outcome changes — fraud patterns evolve, consumer behaviour shifts — so even familiar-looking inputs no longer mean what they meant.
- Champion/challenger testing keeps improvement safe: the incumbent model (champion) serves production while candidates (challengers) run against the same data; a challenger is promoted only when it demonstrably wins.
- Monitoring needs owners and triggers. Metrics, thresholds, an owner, and defined retraining or rollback triggers — monitoring nobody owns is the failure the exam likes to describe.
What the exam asks
Distinguish data drift from concept drift in a scenario, and pick the response: investigate, retrain, roll back, or escalate.
Going deeper
This page is the condensed version. The full topic — with the detail above expanded and practice questions attached — is in the app, inside Domain III. Domain I, a quarter of the course, is free to try first.
Back to the AIGP study guide.
AI Governance Study is an independent study aid. It does not represent a government entity: it is not affiliated with, endorsed by or authorised by any government, government agency or regulatory authority, and it does not provide government services or legal advice. Laws and frameworks are described in our own words — the official texts are listed at official sources. It is also not affiliated with, endorsed by, or sponsored by the IAPP. The AIGP name is used only to identify the exam this material helps you prepare for.