AI Bias & Fairness Auditing

What This Method Does

AI bias and fairness auditing evaluates whether AI systems produce discriminatory outcomes — and identifies the mechanisms through which discrimination occurs. It attempts to answer: does this system treat different groups of people differently in ways that are unjust, and if so, why?

“Fairness” has multiple mathematical definitions that are mutually incompatible — a system cannot simultaneously satisfy all reasonable fairness criteria (see the impossibility result). Auditing therefore involves not just measurement but judgment: selecting appropriate fairness criteria for the specific context, measuring system performance against those criteria, and interpreting results in light of legal requirements and social context.

Bias auditing is distinct from general performance evaluation. A model can achieve high overall accuracy while systematically underperforming for specific demographic groups — standard aggregate metrics (accuracy, F1, AUC) mask these disparities. Auditing disaggregates performance to reveal group-level and intersectional disparities.

For a step-by-step workflow, see the How to Detect AI Bias practitioner guide.

Which Threat Patterns It Addresses

Bias auditing is relevant to five documented threat patterns:

• Allocational Harm (PAT-SOC-002) — Unequal distribution of opportunities or penalties across groups. Example: Amazon AI hiring systematically downgraded women’s résumés; Workday hiring discrimination alleges systematic age and disability discrimination.

• Data Imbalance Bias (PAT-SOC-003) — Training data that underrepresents specific populations. Example: Pulse oximeter racial bias — AI trained on non-representative data perpetuated medical device biases across darker skin tones.

• Proxy Discrimination (PAT-SOC-004) — Neutral-seeming features (zip code, browsing behavior) that correlate with protected characteristics. Example: SafeRent housing discrimination and Meta housing ads both used proxy features to produce racially disparate outcomes.

• Algorithmic Amplification (PAT-SOC-001) — Systems that amplify existing societal biases beyond baseline rates in training data.

• Representational Harm (PAT-SOC-005) — Stereotyping, demeaning, or erasure of specific groups. Example: Google Gemini image generation controversy demonstrated both erasure and inappropriate representation.

How It Works

A. Quantitative fairness metrics

Group fairness metrics

• Demographic parity (statistical parity) — Positive outcome probability should be equal across groups. Maps directly to the U.S. “four-fifths rule”: if the selection rate for a protected group falls below 80% of the highest-performing group, disparate impact is presumed.

• Equalized odds — True positive rate and false positive rate should be equal across groups. The COMPAS algorithm failed this criterion — its false positive rate for Black defendants was ~2× that for white defendants.

• Predictive parity — Precision (positive predictive value) should be equal across groups. If a diagnostic system predicts a condition, the probability it’s actually present should be the same regardless of demographic group.

• Calibration — Predicted probabilities should match actual outcome rates across groups. A 70% score should mean ~70% of applicants at that score receive the outcome, regardless of group.

Individual fairness metrics

• Similar individuals, similar outcomes — Individuals similar on relevant features should receive similar predictions. Requires defining a domain-specific similarity metric — inherently context-dependent.

• Counterfactual fairness — Predictions should be the same in the actual world and a counterfactual world where the protected attribute differs. Requires a causal model of how the protected attribute influences features — often unavailable or contested.

B. Disaggregated evaluation

• Subgroup analysis — Evaluate performance separately for each demographic subgroup and their intersections (e.g., Black women, elderly Hispanic men). Intersectional disparities are often larger than single-axis disparities and missed by audits examining only one attribute.

• Slice discovery — Automatically identify underperforming subgroups without pre-specified categories. Can reveal disparities associated with non-demographic features (region, dialect, image quality) that correlate with demographics.

• Error analysis — Examine the types of errors across groups, not just rates. A lending model that denies creditworthy applicants from one group at higher rates produces qualitatively different harm than one that approves non-creditworthy applicants at different rates.

C. Qualitative and process auditing

• Dataset auditing — Examine composition, provenance, and representativeness of training data. Assess whether collection systematically over- or under-represents populations. Evaluate label consistency across groups.

• Feature auditing — Examine whether any features serve as proxies for protected attributes. Even when protected attributes are excluded, correlated features (zip code, name, school) can reproduce discriminatory patterns.

• Decision context auditing — Evaluate whether the system is deployed in a context where its performance characteristics are appropriate. A model validated for one population may produce biased outcomes on a different one. The Dutch childcare benefits scandal demonstrated how an algorithmic system applied in a fraud detection context produced discriminatory outcomes targeting dual-nationality families.

• Stakeholder impact assessment — Identify who is affected, what harms they may experience, and whether they have meaningful recourse.

Five-step audit workflow

1. Define context — Identify the decision domain, affected populations, applicable legal requirements, and appropriate fairness criteria for this specific use case.
2. Collect demographic data — Obtain or infer protected attribute data for the population processed by the system. Where direct collection is restricted, use proxy-based methods (with documented limitations).
3. Compute fairness metrics — Calculate the selected group and individual fairness metrics, disaggregated by each protected attribute and key intersections.
4. Conduct qualitative audit — Examine the dataset, features, decision context, and stakeholder impacts that the quantitative metrics cannot capture.
5. Act on findings — Document disparities, determine whether they exceed acceptable thresholds, and decide: mitigate, constrain the deployment scope, or do not deploy.

When to audit: pre-, in-, and post-deployment

Auditing tools and platforms

Limitations

The impossibility theorem constrains all auditing

Multiple reasonable fairness criteria are mathematically incompatible. No system can be “fair” by all definitions simultaneously. Auditing can measure compliance with specific chosen criteria, but the choice of criteria is a normative decision that the audit cannot resolve.

Auditing is snapshot, not continuous

Most auditing is conducted at a point in time. Model behavior changes over time — data drift, feedback loops, shifting population characteristics — in ways that introduce new biases after the audit. Continuous monitoring (see AI Risk Monitoring Systems) complements periodic auditing.

Auditing does not fix bias

Mitigation strategies — re-balancing data, constrained optimization, post-processing — each introduce tradeoffs (typically reducing overall accuracy to improve group parity). Some biases reflect structural real-world inequalities the AI accurately learned. In those cases, the appropriate response may be to not deploy the system.

Regulatory fragmentation

Anti-discrimination law varies by jurisdiction — the U.S. four-fifths rule, EU AI Act non-discrimination requirements, and sector-specific regulations (ECOA for lending, Fair Housing Act) define fairness differently. Compliance in one jurisdiction does not guarantee compliance in another.

Real-World Usage

Evidence from documented incidents

The documented incidents reveal a pattern: bias is most commonly detected by external parties — investigative journalists, regulators, affected individuals filing complaints, and academic researchers — rather than by the organizations deploying the systems. Internal auditing, when it occurred (as at Amazon), led to the system being shut down rather than “fixed.” This suggests auditing’s greatest value may be preventing deployment of biased systems rather than remediating deployed ones.

Regulatory context

• EU AI Act — Classifies AI in employment, credit, education, and essential services as high-risk, requiring conformity assessments that include bias evaluation.
• NYC Local Law 144 — Requires annual bias audits of automated employment decision tools, with public reporting.
• EEOC guidance — Title VII applies to AI-based employment decisions.
• CFPB — Enforces fair lending requirements against AI lending models.

Where Detection Fits in AI Threat Response

• Auditing (this page) — Evaluate whether an AI system produces discriminatory outcomes.
• Risk monitoring — Detect emerging bias through continuous performance tracking.
• Model governance — Require fairness evaluation before deployment approval.
• Audit logging — Maintain decision records for retrospective fairness analysis.
• Human oversight — Enable meaningful human review of AI decisions.
• Incident response — Respond when discriminatory outcomes are identified.

For a step-by-step auditing workflow, see the How to Detect AI Bias guide.