Domain III is where governance stops being paperwork and becomes operations: rolling systems out safely, watching them degrade, catching incidents, and knowing who answers for what. Every concept here comes with a plain-words example.
Deployment is where a model stops being a promise and starts making real decisions. The exam tests the sequence, the controls at each stage, and the difference between a rollout and a release.
Confirm the model card, test evidence, bias assessment, and sign-off gates are complete. Nothing deploys without a named accountable owner.
Example: a bank's credit model needs the model card signed by the CRO before the API goes live — no signature, no traffic.Run the new model in parallel with the old one, logging its predictions without acting on them. Compare outcomes before switching.
Example: a fraud model runs alongside the existing rules engine for 2 weeks — if it flags 3× more false positives, it doesn't go live.Release to a small slice of users or geographies first, with monitoring thresholds armed, then expand. Canary releases are the standard pattern.
Example: a chatbot launches to 5% of customers, then 25%, then 100% — each step gated on error-rate and escalation metrics.System is live. Monitoring dashboards, drift thresholds, rollback triggers, and the incident route are all operational from day one.
Example: the credit model is live, but a dashboard tracks approval rates by demographic group daily — a drift alert fires if any group's rate moves >2%.| Control | What it does | Plain-words example |
|---|---|---|
| Human-in-the-loop (HITL) | No decision executes without human review/approval | A loan officer must approve every AI-recommended rejection before it is sent |
| Human-on-the-loop (HOTL) | System acts autonomously; humans supervise, sample-audit, and can intervene or halt | A chatbot answers customers, but a supervisor watches a live dashboard and can kill the bot with one switch |
| Human-out-of-the-loop (HOOTL) | Fully automated; humans review aggregate performance only | Spam filtering — no one reviews individual emails, only monthly false-positive stats |
| Post-processing bias controls | Adjust or re-rank outputs to meet fairness targets after the model decides | A hiring model's shortlist is re-balanced so no demographic group is under-represented before HR sees it |
| Override & escalation paths | Users can challenge or escalate a decision to a human | A customer denied by the credit model can request a manual review within 24 hours |
The model works, but the surrounding system breaks — API timeouts, data-format mismatches, downstream systems receiving unexpected outputs.
Example: a chatbot's response format change crashes the ticketing system that parses its output.The model is used in a context it wasn't validated for — the classic deployment-bias failure.
Example: COMPAS recidivism scores designed for sentencing used for bail decisions.Users aren't told they're interacting with AI, or the notice is buried in terms of service.
Example: a chatbot that doesn't disclose it's an AI breaches Art. 50 of the EU AI Act.No tested rollback path — when the model misbehaves, the team can't revert quickly.
Example: a pricing model goes wrong on a Friday; the team can't restore the old version until Monday.Models rot. The world moves, and the model keeps predicting the world it was trained on. This simulator shows a fraud model's accuracy over 12 months — flip the drift switches and watch what happens when you don't monitor.
| Drift type | What changes | Plain-words example |
|---|---|---|
| Data drift (input) | The distribution of input features changes — the world the model sees is different | A fraud model trained on card-present transactions suddenly sees mostly online payments after COVID — the input mix shifted, so the model's assumptions break |
| Concept drift | The relationship between inputs and outputs changes — the "rule" itself moved | A spam model learned that "free" means spam; then a legitimate competitor launches a "free" campaign — the same input now maps to a different truth |
| Label drift | The ground-truth labels shift — what counts as the correct answer changed | A defect-inspection model was trained when "defective" meant visible cracks; the factory now uses a stricter standard — the same photo is now labelled defective |
Accuracy, precision, recall, F1 — tracked over time and disaggregated by subgroup. A drop is the first sign of drift.
Example: recall for the 60+ age group falls from 88% to 71% — the model is silently failing older customers.Feature distributions and prediction distributions compared against training baselines. Catches drift before labels arrive.
Example: average transaction amount shifts from $45 to $120 — the model never saw this range.Model outputs feeding back into future training can amplify bias — the model's own decisions become the new "truth".
Example: a hiring model trained on its own past shortlists keeps rejecting the same profile types — the bias compounds.Pre-defined thresholds that fire retraining or rollback. The trigger must be set BEFORE deployment, not after the damage.
Example: "if fraud recall drops below 80% for 3 consecutive days, auto-flag for retraining."Six incident scenarios — pick the right first move. The exam tests whether you know the incident-response lifecycle: detect → triage → respond → recover → review. Speed matters, but the RIGHT first action matters more.
Monitoring flags an anomaly — a metric breach, an alert, a user complaint, a regulator inquiry.
Example: the drift dashboard shows recall dropped 12% overnight.Assess severity, scope, affected parties, and whether it's a security, safety, bias, or availability incident.
Example: 3,000 loan decisions affected vs 30 — different severity, different response.Kill switch, rollback, or quarantine. Stop the bleeding before investigating the cause.
Example: the chatbot is taken offline within 15 minutes of the harmful-output report.Restore service, notify affected parties and regulators, root-cause analysis, update controls.
Example: after the fix, the team adds a new monitoring threshold so the failure can't recur silently.Data poisoning, model extraction, prompt injection, jailbreaks, adversarial attacks.
Example: an attacker crafts prompts that make a support chatbot leak another customer's order details.The system causes or risks physical or psychological harm.
Example: an autonomous warehouse robot misidentifies a worker and moves toward them.Disparate outcomes across groups surface in production.
Example: a hiring model's shortlist includes 0% of candidates from a particular demographic for 3 weeks straight.System down, degraded, or silently wrong at scale.
Example: a pricing model returns NaN for a product category, and the website shows blank prices.Training-data leakage, membership inference, model inversion, or personal data exposure via outputs.
Example: a user prompts a model and it regurgitates a real person's address from training data.Breach of a legal obligation — transparency, notification, documentation.
Example: a deepfake tool ships without the Art. 50 labelling requirement.Four frameworks, one exam. Learn each one's structure and its exam tell — the mnemonic, the function names, and the plain-words example that makes it stick.
| Framework | Structure | Exam tell | Plain-words example |
|---|---|---|---|
| NIST AI RMF 1.0 | Govern – Map – Measure – Manage (G-M-M-M) | "GMMM" — the four functions; Govern is the core that wraps the others | A bank maps its credit model's risks (Map), measures subgroup performance (Measure), manages mitigations (Manage), and its board sets the risk appetite (Govern) |
| ISO/IEC 42001:2023 | Plan – Do – Check – Act (PDCA) AI management system; certifiable | "PDCA" + it's the certifiable management-system standard (like ISO 27001 for AI) | A consultancy's AI policy: plan the AI programme, do the risk assessments, check via audits, act on findings — then get certified |
| Singapore Model AI Governance Framework | Internal governance + human oversight + operations management + stakeholder interaction (2nd ed.); GenAI framework adds 9 dimensions | "SG MGF" — risk-weighted human oversight; the 9 GenAI dimensions | A Singapore fintech sets human oversight by weighing harm severity × probability against commercial considerations |
| OECD AI Principles | 5 principles: inclusive growth, human-centred values, transparency, robustness, accountability | "5 principles" — the international baseline most frameworks reference | A government's national AI strategy cites OECD principles as its foundation for trustworthy-AI policy |
The core: risk-management culture, policies, roles, and accountability. Everything else sits inside it.
Example: the board approves an AI risk appetite statement and appoints an AI governance officer.Understand the context: use case, stakeholders, benefits, and risks.
Example: the team documents who is affected by the hiring model and what could go wrong for each group.Test and evaluate: metrics, bias testing, red-teaming, monitoring baselines.
Example: the model's accuracy is measured per demographic group, not just overall.Act on findings: mitigate risks, implement controls, monitor continuously.
Example: a bias finding triggers retraining with rebalanced data and a new monitoring threshold.You cannot govern what you have not inventoried. This section covers the roles that carry accountability and the register that makes governance visible.
| Role | Core duties | Plain-words example |
|---|---|---|
| AI governance officer | Owns the AI governance programme: policy, risk register, training, incident oversight | A bank's AI governance officer signs off every new model before it can enter shadow deployment |
| Accountable executive | A named senior leader with final sign-off authority and budget responsibility | The CRO personally approves the credit model's go-live — no delegation to a committee |
| Model owner / product owner | Day-to-day accountability for a specific system's lifecycle | The fraud-model owner is the person paged when the drift alert fires at 3am |
| Data steward | Data quality, lineage, lawful basis, and access control for training data | The steward can answer "where did this training record come from and under what basis?" |
| Independent reviewer / auditor | Challenges the team's claims — red-teaming, bias audits, pre-deployment review | An external auditor runs its own bias tests on the hiring model rather than trusting the vendor's report |
| Legal / compliance | Maps obligations (GDPR, PDPA, EU AI Act), advises on lawful basis and notices | Legal flags that the chatbot needs an Art. 50 disclosure before launch |
Records of processing activities — the privacy cousin of the AI inventory. Both must exist and be kept current.
Example: a DPO audit starts by pulling the RoPA and the AI inventory side by side.Accountability follows roles, not contracts — a deployer can't outsource its duties to a vendor.
Example: a hospital buys a diagnostic AI, but the hospital still owns oversight and monitoring duties.Four pairs the exam loves to blur. If you can fill these from memory, Domain III is largely yours.
| Shadow | Canary | A/B test | |
|---|---|---|---|
| Real users affected? | No — predictions logged, not acted on | Yes — small slice of real traffic | Yes — users split between versions |
| Purpose | Compare outcomes before any risk | Gradual, gated rollout | Measure which version performs better |
| Exam tell | "Runs in parallel, no impact" | "5% of traffic, then expand" | "Two groups compared" |
| Data drift | Concept drift | Label drift | |
|---|---|---|---|
| What changes | Input distribution | Input→output relationship | Ground-truth labels |
| Plain example | Card fraud → online fraud mix | "Free" stops meaning spam | Stricter defect standard |
| Detection | Compare feature distributions | Track performance over time | Re-audit labels periodically |
| NIST AI RMF | ISO/IEC 42001 | |
|---|---|---|
| Type | Risk-management framework (voluntary) | Management-system standard (certifiable) |
| Structure | Govern–Map–Measure–Manage | Plan–Do–Check–Act |
| Best for | Designing risk processes | Formal certification and audits |
| Exam tell | "Functions" | "Certification", "management system" |
| HITL | HOTL | HOOTL | |
|---|---|---|---|
| Human role | Approves every decision | Supervises, can intervene | Reviews aggregates only |
| Speed | Slowest | Medium | Fastest |
| Best for | High-stakes, irreversible | Medium-stakes, reversible | Low-stakes, high-volume |
| Example | Loan rejection approval | Chatbot supervision | Spam filtering |
TRUE / FALSE — instant feedback per question, score tallied at the bottom. Best score persists in this browser.
Tap each item as you master it. The goal: fill every box from memory, not recognition.