High stakes · Obligations
High-risk AI systems
Between "banned" and "anything goes" sits the regulation's centre of gravity: systems that may shape your education, job, health or access to services. They are allowed — under conditions designed to make them trustworthy and contestable.
01What "high risk" means in educational terms
High-risk does not mean evil or forbidden. It means: if this system fails or discriminates, real people lose real things — a place at school, a job interview, a diagnosis, a benefit. So the burden of proof flips: the system must demonstrate it deserves deployment.
02Where the label typically lands
- Education — admission, assessment, adaptive platforms that steer learning paths.
- Employment — CV screening, interview scoring, promotion and termination support.
- Essential services — credit, benefits, housing prioritisation.
- Health — clinical decision support and triage.
Four of the game's cases live here: the interview that does not exist, invisible triage, the profiled classroom, the opaque tender.
03The obligations, in plain words
- Risk management — identify what can go wrong, for whom, and mitigate before deployment.
- Data governance — training data relevant and representative for the actual population affected.
- Documentation & logging — the system must be explainable to an authority and traceable after the fact.
- Human oversight — a person with the competence, information and authority to intervene.
- Accuracy and robustness — appropriate to the stakes.
For the whole picture from a company's side — role, risk, competence, deadlines — see AI Act for business.
When they become enforceable. These obligations are in the regulation, but the Digital Omnibus (Reg. (EU) 2026/1744) moved their application date: 2 December 2027 for Annex III systems — employment, education, essential services, justice — and 2 August 2028 for those embedded in already-regulated products (Annex I). Deferred is not abolished: the obligations are identical in substance, what changed is the date from which an authority can require them, and anyone designing a system today that will fall into these categories has less time than it looks. The full set of dates is on the application timeline.
04The oversight trap
The most teachable failure mode: oversight that exists on paper. An operator who "reviews" 400 automated decisions a day reviews none. The game's evidence keeps returning to this — internal notes admitting staff "follow the score" — because recognising fake oversight is a skill students can use on any system they meet.
05Classroom discussion
Strong prompts: Which of the duties above would have caught the failure in our scenario? Who should verify them — buyer, vendor, authority? What should happen when documentation is missing? The activities page has a ready-made sorting exercise. Use provider and deployer to separate responsibilities, and the fundamental rights impact assessment guide for sensitive public-sector uses.
06The obligations in practice: what to look for
High-risk obligations read better as questions to put to a real system than as a list to memorise. Each row below is a question a student can ask of a concrete case, plus the sign that something is missing.
| Obligation (simplified) | The question to ask | How you can tell it is missing |
|---|---|---|
| Risk management | Has anyone written down what can go wrong, and for whom? | Nobody can say who would be harmed first by an error. |
| Data governance | What data was it trained on, and who was under-represented in it? | Errors keep concentrating on the same groups of people. |
| Technical documentation | Is there a document explaining how it works, kept up to date? | The answer is "the vendor knows" — and the vendor points at the sales brochure. |
| Record-keeping (logging) | If I contested a decision today, would a trace remain? | You cannot reconstruct which version of the system decided, or when. |
| Human oversight | Can the person supervising actually overturn the outcome? | They can only confirm, or they get thirty seconds per file. |
| Accuracy and robustness | Is the error rate measured and published, or merely claimed? | A single aggregate percentage is quoted, with no breakdown by group. |
The right-hand column is the didactically useful one: it moves attention from declared compliance to observable compliance. It is also how the game builds its case files — the exhibits show symptoms, not certificates.
07Three systems compared
Why the "high risk" label depends on context rather than technology becomes clearer when you compare similar uses with different outcomes.
| System | Where it is used | Educational reading |
|---|---|---|
| Automated ranking of applications | Recruitment | Typically high risk: it affects access to work, so it needs real oversight, traceability and information to the people involved. |
| Automated ranking of content | A music playlist | Minimal risk: the same ranking technique touches neither fundamental rights nor access to essential services. |
| Predictive scoring of citizens | Access to social benefits | Boundary territory: it can be high risk when it governs access to an essential service, or fall among prohibited practices if it becomes generalised social scoring. |
The third row produces the best discussions, and it is why the game devotes a mirror case to civic credit: the difference is not in the algorithm, but in what the score decides and how much of a life it reaches into.