High stakes · Obligations
High-risk AI systems
Between "banned" and "anything goes" sits the regulation's centre of gravity: systems that may shape your education, job, health or access to services. They are allowed — under conditions designed to make them trustworthy and contestable.
01What "high risk" means in educational terms
High-risk does not mean evil or forbidden. It means: if this system fails or discriminates, real people lose real things — a place at school, a job interview, a diagnosis, a benefit. So the burden of proof flips: the system must demonstrate it deserves deployment.
02Where the label typically lands
- Education — admission, assessment, adaptive platforms that steer learning paths.
- Employment — CV screening, interview scoring, promotion and termination support.
- Essential services — credit, benefits, housing prioritisation.
- Health — clinical decision support and triage.
Four of the game's cases live here: the interview that does not exist, invisible triage, the profiled classroom, the opaque tender.
03The obligations, in plain words
- Risk management — identify what can go wrong, for whom, and mitigate before deployment.
- Data governance — training data relevant and representative for the actual population affected.
- Documentation & logging — the system must be explainable to an authority and traceable after the fact.
- Human oversight — a person with the competence, information and authority to intervene.
- Accuracy and robustness — appropriate to the stakes.
For the whole picture from a company's side — role, risk, competence, deadlines — see AI Act for business.
04The oversight trap
The most teachable failure mode: oversight that exists on paper. An operator who "reviews" 400 automated decisions a day reviews none. The game's evidence keeps returning to this — internal notes admitting staff "follow the score" — because recognising fake oversight is a skill students can use on any system they meet.
05Classroom discussion
Strong prompts: Which of the duties above would have caught the failure in our scenario? Who should verify them — buyer, vendor, authority? What should happen when documentation is missing? The activities page has a ready-made sorting exercise. Use provider and deployer to separate responsibilities, and the fundamental rights impact assessment guide for sensitive public-sector uses.
06The obligations in practice: what to look for
High-risk obligations read better as questions to put to a real system than as a list to memorise. Each row below is a question a student can ask of a concrete case, plus the sign that something is missing.
| Obligation (simplified) | The question to ask | How you can tell it is missing |
|---|---|---|
| Risk management | Has anyone written down what can go wrong, and for whom? | Nobody can say who would be harmed first by an error. |
| Data governance | What data was it trained on, and who was under-represented in it? | Errors keep concentrating on the same groups of people. |
| Technical documentation | Is there a document explaining how it works, kept up to date? | The answer is "the vendor knows" — and the vendor points at the sales brochure. |
| Record-keeping (logging) | If I contested a decision today, would a trace remain? | You cannot reconstruct which version of the system decided, or when. |
| Human oversight | Can the person supervising actually overturn the outcome? | They can only confirm, or they get thirty seconds per file. |
| Accuracy and robustness | Is the error rate measured and published, or merely claimed? | A single aggregate percentage is quoted, with no breakdown by group. |
The right-hand column is the didactically useful one: it moves attention from declared compliance to observable compliance. It is also how the game builds its case files — the exhibits show symptoms, not certificates.
07Three systems compared
Why the "high risk" label depends on context rather than technology becomes clearer when you compare similar uses with different outcomes.
| System | Where it is used | Educational reading |
|---|---|---|
| Automated ranking of applications | Recruitment | Typically high risk: it affects access to work, so it needs real oversight, traceability and information to the people involved. |
| Automated ranking of content | A music playlist | Minimal risk: the same ranking technique touches neither fundamental rights nor access to essential services. |
| Predictive scoring of citizens | Access to social benefits | Boundary territory: it can be high risk when it governs access to an essential service, or fall among prohibited practices if it becomes generalised social scoring. |
The third row produces the best discussions, and it is why the game devotes a mirror case to civic credit: the difference is not in the algorithm, but in what the score decides and how much of a life it reaches into.