Case Study 2

Amazon's AI Recruiting Tool

When Machine Learning Learns Gender Bias in Hiring

Background

The Promise of AI in Recruiting

In 2014, Amazon began developing an artificial intelligence recruiting tool designed to revolutionize hiring by automatically screening resumes and identifying the most promising candidates. The tech giant hoped to create a system that could:

  • Review large numbers of resumes quickly and efficiently
  • Identify top talent without human bias or fatigue
  • Make hiring decisions more "objective" and data-driven
  • Reduce time and resources spent on initial screening
  • Scale recruiting to match Amazon's rapid growth

How the System Worked

Amazon's recruiting AI was a machine learning system that worked by:

  1. Learning from Historical Data: The algorithm was trained on resumes submitted to Amazon over a 10-year period, looking at patterns in who was hired
  2. Pattern Recognition: The system identified commonalities among successful candidates
  3. Scoring New Resumes: New applicants would receive scores from 1-5 stars, like rating products on Amazon's retail platform
  4. Ranking Candidates: Recruiters would use these scores to prioritize which candidates to interview

The Tech Industry Context

Understanding this case requires context about the tech industry:

  • Tech companies, including Amazon, have long struggled with gender diversity
  • Technical roles in tech have historically been dominated by men
  • Women make up only about 26% of computing professionals in the U.S.
  • This underrepresentation has persisted despite efforts to increase diversity

The Problem: Gender Bias Discovered

What Amazon Discovered

In 2015, Amazon's team realized their recruiting AI had a serious problem: it was systematically discriminating against women.

Specific Biases Identified:

Penalized "Women" in Resumes

The algorithm learned to downgrade resumes that included the word "women." This affected:

  • Graduates of women's colleges (e.g., "Women's Chess Club Captain")
  • Members of women's professional organizations
  • Participants in women-in-tech programs
  • Any resume mentioning women's leadership or advocacy groups

Impact: Women were penalized for the very activities designed to support women in male-dominated fields.

Preferred "Male" Language

The system learned to favor resumes with language more commonly used by men, such as:

  • Action verbs like "executed" and "captured" (common in military experience, predominantly male)
  • Aggressive, competitive language
  • Certain technical terms and phrases more common in male-dominated spaces

Favored Male-Dominated Fields

The algorithm gave higher scores to applicants with backgrounds in activities and fields where men are overrepresented, even when not directly relevant to the job.

Amazon's Response

When engineers discovered these biases:

  • They attempted to make the system neutral by editing specific terms
  • They removed obvious biased patterns they could identify
  • However, they could not guarantee the system wasn't finding other subtle ways to discriminate
  • In 2017, after several years of trying to fix it, Amazon abandoned the project entirely
  • The tool was never used for actual hiring decisions (only experimental)

The Story Becomes Public

In October 2018, Reuters broke the story of Amazon's failed recruiting tool, bringing national attention to the dangers of algorithmic bias in hiring.

How Did Bias Enter the System?

Root Causes of the Bias:

Biased Training Data

The most fundamental problem: The algorithm learned from 10 years of resumes submitted to Amazon, where the vast majority of technical hires had been men.

The Logic Chain:

  1. Historical data showed more men were hired for technical roles
  2. Algorithm concluded that "male patterns" = "successful candidate"
  3. System learned to prefer resumes that looked like past successful candidates (mostly men)
  4. New female applicants' resumes didn't match these patterns
  5. Women received lower scores

Key Insight: The AI didn't learn to identify the best candidates—it learned to identify candidates who looked like past hires. Past discrimination became "prediction" for the future.

Industry-Wide Patterns

The tech industry's gender imbalance meant that:

  • Most technical employees at Amazon were men
  • Most resumes in the training data came from men
  • Successful hiring patterns reflected male-dominated workforce
  • The AI had few examples of successful female technical hires to learn from

Language and Expression Differences

Research shows that men and women sometimes describe their accomplishments differently:

  • Men may use more assertive language ("led," "drove," "executed")
  • Women may use more collaborative language ("coordinated," "facilitated," "supported")
  • Both styles can describe equally impressive achievements
  • But the AI learned to prefer one style because it appeared more often in "successful" resumes

Educational and Experiential Patterns

The system penalized experiences unique to women's attempts to enter tech:

  • Women's colleges (excellent institutions, but flagged by "women's" label)
  • Women-in-tech organizations and programs
  • Diversity initiatives and mentorship programs

The algorithm couldn't understand that these experiences might indicate strong candidates—it only saw they were different from the historical pattern of male hires.

Lack of Diverse Oversight

Questions to consider:

  • Who designed the system? (Likely male engineers, given tech demographics)
  • Who tested it? (May not have included diverse perspectives)
  • Who raised concerns? (Took time to identify the bias)

Real-World Impact

Who Was Harmed?

Individual Women Applicants

Even though Amazon never fully deployed this tool, its development reveals how easily AI can disadvantage qualified candidates:

  • Women with excellent qualifications could have received low scores
  • Female candidates might never have gotten interviews despite being qualified
  • Women who attended women's colleges were specifically penalized
  • Participants in women's professional organizations were disadvantaged

Broader Implications for Women in Tech

This case highlights systemic challenges:

  • Catch-22: Women are underrepresented in tech, so AI learns to prefer men, making it harder for women to get hired, perpetuating underrepresentation
  • Invisible Barriers: Qualified women might be filtered out before human recruiters ever see their applications
  • Confidence Impact: Knowing AI might be biased against you can discourage women from applying to tech jobs

Company Consequences

Amazon and the tech industry suffered from:

  • Missed Talent: Failed to identify qualified candidates, limiting innovation and growth
  • Reduced Diversity: Tool would have made existing diversity problems worse
  • Reputation Damage: Publicizing this failure harmed Amazon's employer brand
  • Wasted Resources: Years of development time and money spent on a failed tool

Industry-Wide Warning

Amazon's experience revealed that many other companies likely have similar problems in their hiring algorithms that haven't been discovered or disclosed yet.

The Positive Side: A Lesson Learned

Amazon's decision to scrap the tool rather than use it shows responsible behavior:

  • They recognized the problem
  • They didn't deploy a system they knew was biased
  • They chose fairness over efficiency

However, the story's publication also raised concerns about how many other companies might be using similar biased systems without detecting or disclosing the problems.

What Could Have Been Done Differently?

Prevention Strategies:

Diverse Development Team

Include women and diverse perspectives in:

  • System design and development
  • Testing and evaluation
  • Decision-making about deployment

Why: Diverse teams are more likely to identify potential biases early.

Data Audit and Cleaning

Before training the algorithm:

  • Analyze historical hiring data for existing biases
  • Consider whether past hires reflect merit or discrimination
  • Don't assume historical patterns should be replicated
  • Balance training data to include diverse successful candidates

Rigorous Testing Across Groups

Test the algorithm separately for different demographic groups:

  • How does it score equally qualified male and female candidates?
  • Does it penalize or favor certain types of experiences?
  • Are error rates equal across groups?

Transparency and Explainability

  • Make scoring criteria visible and explainable
  • Allow candidates to understand why they received certain scores
  • Enable review and challenge of algorithmic decisions

Fairness Constraints

Build fairness requirements into the algorithm from the start:

  • Prohibit use of gender-related terms in scoring
  • Ensure equal false positive/negative rates across groups
  • Require similar score distributions for equally qualified candidates

Question the Premise

Consider whether AI is appropriate for this task at all:

  • Are there aspects of hiring that algorithms can't capture?
  • Does automation risk encoding existing biases?
  • Would alternative approaches (improved human training, structured interviews) work better?

Continuous Monitoring

After deployment (if it had been deployed):

  • Regularly audit outcomes across demographic groups
  • Monitor for disparate impact
  • Update and retrain as new data becomes available
  • Establish clear protocols for addressing discovered biases

Alternative Approaches Amazon Could Have Taken:

  • Augmentation Not Replacement: Use AI to assist human recruiters rather than replace their judgment
  • Blind Screening: Remove identifying information before human review
  • Structured Interviews: Standardize questions and evaluation criteria
  • Diverse Hiring Panels: Include multiple perspectives in hiring decisions
  • Target Recruitment: Actively recruit from diverse sources

Key Lessons Learned

Past Discrimination Becomes Future "Prediction"

Machine learning algorithms trained on biased historical data will perpetuate and even amplify those biases. Historical hiring patterns reflect discrimination, not merit.

You Can't Fix Bias with More Bias

Amazon tried to patch specific problems (removing the word "women"), but couldn't guarantee the system wasn't finding other ways to discriminate. Addressing bias requires systematic approaches, not just patches.

AI Amplifies Existing Inequalities

If women are underrepresented in tech, AI trained on tech data will learn to prefer men, making the problem worse. This creates a harmful feedback loop.

Intention Doesn't Prevent Bias

Amazon didn't intend to create a discriminatory system, but good intentions aren't enough. Rigorous testing and diverse perspectives are essential.

Efficiency Isn't Worth Unfairness

The appeal of AI recruiting is speed and scale, but if the system is unfair, no efficiency gain justifies its use. Amazon made the right call in scrapping the tool.

Transparency Matters

We only know about Amazon's problem because it became public. How many other companies are using biased hiring algorithms without knowing or disclosing it?

Technical Solutions Require Social Understanding

Understanding this problem requires knowledge of tech industry gender disparities, historical discrimination, and social dynamics—not just coding skills.

Discussion Guide

Small Group Discussion Questions

Question 1: Root Cause Analysis

Amazon's AI learned from 10 years of hiring data. Why did this cause the algorithm to discriminate against women? Explain the chain of reasoning the algorithm followed.

Hint: Think about what "success" looked like in the historical data.

Question 2: The Catch-22

This case reveals a "catch-22" or self-reinforcing cycle. Explain how using this algorithm could have made gender imbalance in tech worse, creating a feedback loop.

Question 3: Evaluating Amazon's Response

Amazon discovered the bias, tried to fix it, and eventually scrapped the tool. Was this the right decision? What would you have done differently?

Question 4: Hidden Bias

The algorithm learned to discriminate against women even though gender wasn't a direct input. How is this possible? What does this tell us about trying to eliminate bias from AI?

Think about: Proxy variables, language patterns, experiences, and what "looks like" a successful candidate.

Question 5: Alternative Approaches

If you were Amazon's head of recruiting, how would you use technology to improve hiring without creating bias? Describe a better approach.

Question 6: Broader Implications

Amazon scrapped this tool, but many companies use AI in hiring. What questions should job seekers ask about how companies use AI in recruiting? What regulations might help?

Whole Class Discussion

  • Why do you think it took Amazon's team time to discover the gender bias? What does this tell us about testing AI systems?
  • Compare this case to COMPAS (racial bias in criminal justice). What similarities and differences do you notice?
  • If AI can't be trusted to screen resumes fairly, what should companies do instead?
  • How does this case relate to broader issues of women's representation in technology fields?
  • Should companies be required to disclose when they use AI in hiring decisions? Why or why not?

Additional Resources

Primary Sources

Related Reading

  • "Invisible Women: Data Bias in a World Designed for Men" by Caroline Criado Perez
  • "Algorithms of Oppression" by Safiya Noble
  • "Weapons of Math Destruction" by Cathy O'Neil (Chapter on hiring algorithms)

Statistics on Women in Tech

  • National Center for Women & Information Technology (NCWIT) statistics
  • U.S. Bureau of Labor Statistics data on women in computing
  • Reports on gender diversity in tech companies

Part of Lesson 3: AI Ethics - Fairness and Bias in Algorithms
Evolve AI Institute