Module 2 of 2 · 45 minutes
Keeping discrimination out, and what to ask a supplier
By the end of this module you will be able to
- Explain how the Equality Act applies to a tool's outcomes
- Describe how a tool can discriminate without being told a protected characteristic
- Monitor outcomes for unfairness
- Make reasonable adjustments where a tool disadvantages a disabled candidate
- Ask a supplier the questions that matter, and say what the EU AI Act adds
Work through it
1 interactive for this module, built on the WAJD Teach engine. Nothing moves until you ask it to, and every one has a written version if you would rather read it.
Watch: Emma and George talk it through
3 minutes. Captions are on, and the same conversation is written out in full below. The voices are computer generated.
Emma George, our supplier says the tool is unbiased because it never sees sex or ethnicity. Is that enough?
George No, and it's the most common misunderstanding. Start with the law. The Equality Act protects nine characteristics, and it doesn't care whether a decision was made by a person or by software.
Emma So what's the risk with a tool?
George Indirect discrimination. A criterion that looks neutral, applied to everyone, which puts people with a protected characteristic at a particular disadvantage. Unlawful unless you can justify it. And a scoring model is a criterion.
Emma But it can't see the characteristic.
George It doesn't need to. Other details stand in. A gap in employment can stand in for maternity or illness. A postcode for ethnicity. A graduation year for age.
Emma That's uncomfortable.
George It gets more so. A tool trained on your past hiring learns your past preferences. If previous hires were mostly of one kind, it favours that kind, and presents it as a score.
Emma So how would I know?
George Look. Compare who applies with who's passed through at each automated stage, by the characteristics you lawfully collect for monitoring. A large unexplained gap at one stage tells you where to look.
Emma Doesn't collecting that data make it worse?
George Not if you keep it separate. The people and the tool making decisions never see it. Do it before go live, using past applications, and regularly afterwards.
Emma Anything simpler?
George Yes. Every so often have a person review a sample of the candidates the tool rejected. It's the cheapest test there is.
Emma What about disabled candidates?
George You must make reasonable adjustments. And automated stages create new disadvantages. A timed online test for someone with dyslexia. A video interview scored on speech or eye contact, for someone who's deaf, who stammers, or who's autistic.
Emma So I need a way round the tool.
George A way to ask for an adjustment before the automated stage, clearly offered, and a human route. A process with no alternative can't meet the duty.
Emma And if it goes wrong, is it the supplier's problem?
George Yours. The employer is liable for discrimination in its recruitment, whoever built the tool. The software chose is not a defence in a tribunal.
Emma What do I ask the supplier, then?
George What it does at each stage and what data it uses. What it was trained on and how it was tested for bias, with the results. Whether it infers any characteristic. And what you can see about why a candidate got their score.
Emma We recruit in Ireland too.
George Then note the EU AI Act. Recruitment AI is high risk there, with duties from 2 December 2027. And emotion recognition at work is already banned.
The written material
The Equality Act applies to the outcome
The Equality Act 2010 protects nine characteristics: age, disability, gender reassignment, marriage and civil partnership, pregnancy and maternity, race, religion or belief, sex, and sexual orientation. It does not care whether a decision was made by a person or by software.
Indirect discrimination is the risk with tools. A rule or criterion that looks neutral, applied to everyone, which puts people with a protected characteristic at a particular disadvantage, is unlawful unless it can be justified as a proportionate means of achieving a legitimate aim. A scoring model is a criterion.
How a tool discriminates without being told
You can remove sex and ethnicity from the data and still get a biased result, because other details stand in for them. A gap in employment can stand in for maternity or illness. A postcode can stand in for ethnicity. A graduation year stands in for age. The names of clubs, schools and previous employers all carry information about who someone is.
A tool trained on a company's past hiring learns that company's past preferences. If previous hires were mostly of one kind, the tool will favour that kind, and present it as a score. The ICO's earlier audit of recruitment tools also found some tools inferring characteristics such as gender and ethnicity from a candidate's name, which is itself a data protection problem.
Monitor your own outcomes
The only way to know is to look. Compare who applies with who is passed through at each automated stage, by the characteristics you lawfully collect for monitoring. A large and unexplained gap at one stage tells you where to look.
Do this before a tool goes live, using past applications, and regularly afterwards. Keep the monitoring data separate from the decision, so that the people and the tool making decisions never see it. Record what you found and what you changed.
Reasonable adjustments
An employer must make reasonable adjustments where a disabled person is put at a substantial disadvantage. Automated stages create new ones. A timed online test disadvantages some people with dyslexia. A video interview scored on speech or eye contact disadvantages people who are deaf, who stammer, or who are autistic.
So there has to be a way to ask for an adjustment before the automated stage, clearly offered, and a human route for anyone who needs one. A process with no alternative to the tool cannot meet the duty.
What to ask a supplier, and the EU
Ask what the tool actually does at each stage, and what data it uses. What it was trained on, and how it was tested for bias, with the results. Whether it infers any characteristic. What a candidate is shown and can challenge. What you can see about why a candidate was scored as they were. And who is responsible for what in the contract.
If you recruit in the European Union, the EU AI Act treats AI used for recruitment and for managing workers as high risk, with duties that apply from 2 December 2027 after the amendment of July 2026. It already bans emotion recognition in the workplace, except for medical or safety reasons.
Knowledge check
The knowledge check and your certificate need a free account, so that your progress and results can be saved as evidence.
The learning itself stays free and open. You are reading all of it right now without an account.
Was this module useful? Tell us in two minutes, it decides what we improve next.