Training, testing and validation: the three data subsets, and how to test an AI system
Every supervised model is built on a three-way split of data — training, validation and test — and a large share of practical AI failures trace back to mixing them up.
Why this matters for the AIGP exam
The three subsets are precise, definitional material the exam tests directly, and data leakage between them is a favourite scenario fault.
The essentials
- Training data is what the model learns from — the examples that set its parameters.
- Validation data tunes the model during development: comparing candidate versions, adjusting hyperparameters, deciding when to stop. It shapes decisions, so the model has indirectly "seen" it.
- Test data is the held-out set used once, at the end, to estimate real-world performance. Its value is its untouchedness.
- Leakage is the cardinal sin. If test data influences training or tuning — directly, through duplicates, or through proxies — reported performance becomes fiction, and the failure surfaces in production instead of in testing.
- Testing the system is bigger than testing the model. Accuracy on a test set says nothing about integration, latency, security, misuse or user comprehension; system-level evaluation covers those separately.
What the exam asks
Define each subset's job, identify leakage in a described pipeline, and distinguish model evaluation from system evaluation.
Going deeper
This page is the condensed version. The full topic — with the detail above expanded and practice questions attached — is in the app, inside Domain III. Domain I, a quarter of the course, is free to try first.
Back to the AIGP study guide.
AI Governance Study is an independent study aid. It does not represent a government entity: it is not affiliated with, endorsed by or authorised by any government, government agency or regulatory authority, and it does not provide government services or legal advice. Laws and frameworks are described in our own words — the official texts are listed at official sources. It is also not affiliated with, endorsed by, or sponsored by the IAPP. The AIGP name is used only to identify the exam this material helps you prepare for.