How AI fails and who gets hurt: failure modes and the five levels of harm
Governance exists because AI systems fail, and failures land on real people and organizations. This topic covers the ways systems characteristically fail, and how to classify who bears the harm when they do.
Why this matters for the exam
Governance exists because AI systems fail, and failures land on real people and organizations. This topic covers the ways systems characteristically fail, and how to classify who bears the harm when they do.
What you need to know
The last topic covered the principles responsible AI is supposed to follow. This one covers what happens when systems fall short: the ways AI fails, and the harm that follows when it does.
How AI systems fail
Seven failure patterns come up throughout this guide. Here you only need to be able to tell them apart — each one gets its full treatment in a later topic.
| Failure | What happens | What kind of problem |
|---|---|---|
| Hallucination | A generative model confidently produces content that is not true: invented sources, nonexistent places, fabricated facts. | Inherent behavior of generative models. Not an attack. |
| Overfitting | A model matches its training data too closely and performs poorly on data it has not seen. | Training problem. |
| Data poisoning | Someone corrupts the training data to change how the model behaves. | Attack, on the training data. |
| Prompt injection | Someone crafts an input that manipulates the model into ignoring its instructions. | Attack, on the model's input. |
| Proxy discrimination | A protected attribute such as race or gender is removed from the training data, but bias returns through correlated features: ZIP code, name, purchasing patterns. | Design-time bias. |
| Automation bias | People trust the model's output more than they should, and stop questioning it. | Human factor. |
| Model drift | A model's performance degrades over time as the world changes around it. | Operational problem, after deployment. |
Hallucination is the only entry where the model fabricates content by itself. The others come from somewhere else: bad training, an attacker, a human over-trusting the output, or a world that has changed since training. And proxy discrimination is the reason simply deleting a protected attribute does not make a model fair — the model rebuilds the pattern from whatever correlates with it. The approach has a name: "fairness through unawareness."
Failures become harms when they land on an individual, a group, an organization, society or the environment. The rest of this topic is about classifying exactly that: who bears the harm, and what kind it is.
What a harms taxonomy is
A harms taxonomy is a structured breakdown of the negative consequences that could happen to a person or organization if information is leaked or misused. Instead of treating "harm" as one undifferentiated thing, it separates harm into its dimensions: the capacity, capability and opportunity to cause it. Understanding the taxonomy behind a privacy or AI right helps you see why that right matters, builds empathy for the people affected, and lets an organization do targeted risk reduction instead of generic mitigation.
Several taxonomies exist side by side:
- Calo — Ryan Calo splits privacy harms into subjective privacy harms (an internal, felt sense of harm) and objective privacy harms (external, like a loan denial based on personal data).
- Citron and Solove — seven harm types: physical, reputational, relationship, economic, discrimination, psychological, and autonomy.
- NIST AI Risk Management Framework — an AI-specific three-way split: harm to people, to an organization, and to an ecosystem. NIST defines risk as "the composite measure of an event's probability of occurring and the magnitude of the consequences of that event."
Five populations, and how to classify an incident
Alongside these taxonomies, the course organizes harm by who is affected:
- Individuals — civil rights, economic opportunity, safety; physical injury, economic loss, dignity, and discrimination against protected classes.
- Groups — discrimination against subgroups.
- Society — democratic process, public trust (including erosion of trust in information), educational access.
- Organizations — reputational, cultural, economic and acceleration risk, plus direct financial loss and security compromise.
- Ecosystems — natural resources, environment, supply chain.
An incident is classified by who most directly and immediately bears the harm. An AI voice-clone impersonating an executive to trick an employee into wiring funds is, first and foremost, organizational harm: the direct, immediate loss lands on the company. An incident may also contribute to broader societal erosion over time, but its category follows its most direct, immediate effect.
The categories are not mutually exclusive, and a single AI use case routinely creates harm across several of them at once. The course's worked example is facial recognition: it can harm individuals (journalists, protesters, people misidentified for a crime), groups (marginalized populations tracked at a location), society (deepfakes and spoofing eroding public trust), and organizations (biometric data breaches).
Individual harms: bias, discrimination and civil rights
Bias is where AI harm most often begins, so the individual category gets the most detail. Bias in AI can damage a person's civil liberties, safety and economic opportunity, and it originates with the people who build the system. That means it has to be addressed across the whole development life cycle, and cannot be patched at the end. Five terms name the kinds of bias involved:
- Implicit bias — discrimination toward a group, often unconscious, reflected in design or training data.
- Sampling bias — data skewed toward a subset of a group.
- Temporal bias — a model that worked at training time but degrades as the world changes.
- Underfitting — a model too simple to capture real patterns; the opposite of overfitting from the failure table above.
- Noise — data that actively harms learning; distinct from outliers, atypical but not necessarily bad data points.
A real incident shows how these play out. In 2014, Amazon built an AI recruiting tool that was found, during testing, to be biased against women, because it had been trained on résumés submitted almost entirely by men. Engineers tried to retrain it, but the bias proved difficult to fully remove once learned, and the project was abandoned in 2017. Discrimination shows up the same way across insurance and social benefits, housing (tenant selection, mortgage qualification), education (admissions scoring), and credit (lending decisions, differential pricing).
Beyond discrimination, individual privacy harms fall into four kinds:
- Using personal data as training data without a real basis for doing so.
- Reidentification — reversing deidentification by combining datasets. This is a growing risk, given how much data AI systems can process at once.
- Inference harms — an AI's predictions misattributing data to the wrong person.
- A lack of transparency about when AI is being used at all.
Group and societal harms
Facial recognition again supplies the clearest group-harm example: accuracy varies by demographic group, and a 2019 NIST study (Face Recognition Vendor Test Part 3: Demographic Effects) found facial recognition systems unreliable across many conditions, with darker skin tones and women disproportionately harder to recognize accurately. London's police facial recognition software was once measured at an 81% inaccuracy rate. Mass surveillance compounds this, since historically marginalized groups often receive less privacy protection and more disproportionate targeting.
At the societal level, harms center on the democratic process: the spread of disinformation, ideological "echo chambers" that isolate people from differing views, deepfakes (especially damaging around elections), and the safety risk of lethal autonomous weapons operating with insufficient oversight.
Environmental and organizational harms
The last two populations are ecosystems and organizations. AI's environmental footprint is concrete and quantified: a 2019 University of Massachusetts Amherst study (Energy and Policy Considerations for Deep Learning in NLP) found that training a large model can emit over 626,000 pounds of CO₂, about five times the lifetime emissions of an American car. Training also consumes significant water, and lithium extraction for batteries adds further strain. AI can also help the environment: self-driving cars reducing emissions, higher agricultural yields, satellite-image disaster analysis, and weather forecasting. The environmental impact runs both ways.
Organizational harm breaks into five categories worth naming individually:
- Reputational — lost customers, share-price drops, activist targeting.
- Cultural — the false assumption that AI output is inherently more accurate or objective than a human's, so it goes unchallenged.
- Economic — remediation and litigation costs.
- Acceleration — AI's speed and complexity mean that some risks cannot be anticipated in advance.
- Legal and regulatory — industry-specific law plus privacy, competition, trade and tax exposure, potentially resulting in fines or stop-processing orders.
That completes the foundations of AI: what it is, how it works, and how it fails. Next up: a new competency — who does what in AI governance, starting with the roles across the AI value chain, from developer to provider to deployer.
Remember
- AI fails in seven characteristic ways, and each is a different kind of problem. Only hallucination is the model fabricating on its own; data poisoning and prompt injection are attacks.
- Classify an incident by who most directly and immediately bears the harm. A single use case, like facial recognition, routinely harms several populations at once.
- Bias originates with the data and the people who build a system, and once learned it is hard to remove. It has to be addressed across the whole life cycle, and cannot be patched at the end.
- AI accuracy can differ by demographic group, and those differences turn into discriminatory outcomes when systems are deployed at scale.
- AI's environmental impact runs both ways: real costs in energy, water and minerals, and real benefits in efficiency and forecasting.
- Organizational harm has five named categories: reputational, cultural, economic, acceleration, and legal/regulatory.
Practise this topic
Domain I is free in the app, including its practice questions and flashcards, with progress tracking and no card details.
Previous: Principles of responsible AI: FIPs, OECD AI Principles, ethics by design, and trustworthy AI
Next: Roles across the AI value chain: developer, provider, deployer and user
Back to the AIGP study guide.
AI Governance Study is an independent study aid. It does not represent a government entity: it is not affiliated with, endorsed by or authorised by any government, government agency or regulatory authority, and it does not provide government services or legal advice. Laws and frameworks are described in our own words — the official texts are listed at official sources. It is also not affiliated with, endorsed by, or sponsored by the IAPP. The AIGP name is used only to identify the exam this material helps you prepare for.