AI bias in recruitment and HR: where discrimination comes from and what the law requires

In brief

AI does not discriminate on purpose; it learns to discriminate from data that carries past choices. In recruitment, workforce management and benefit processing this is a question both of the AI Act and of anti-discrimination law. The AI Act classifies recruitment and workforce management systems as high-risk and requires their provider, in Article 10, to govern data in a way that includes examining and mitigating bias; Article 22 of the GDPR limits decisions based solely on automated processing; Finland's Non-Discrimination Act and Equality Act prohibit discrimination in working life regardless of whether a person or a model made the choice. An employer who buys a system ready-made does not buy the responsibility away.

Where bias comes from

A language model and a classifier learn from examples. If a recruitment model is trained on an organisation's past applications and their outcomes, it learns to predict who would have been chosen before. At the same time it learns everything that influenced those choices: the field's gender balance, age, the familiarity of a name, the prestige of a school. None of this needs to be given to the model directly. It shows indirectly in hobbies, gaps in work history, word choices and the names of institutions, and the model finds it because it predicts the outcome.

The second source is the goal. A model that optimises "succeeds in the role" measures success by what the data holds: perhaps retention, perhaps supervisors' ratings, perhaps promotions. Each of these measures carries its own biases. The third source is representativeness: if a group has few examples, the model predicts them worse, and a poor prediction looks like a rejection.

Bias is therefore not a bug fixed once but a property detected by measuring outcomes by group and managed continuously. That is why the law speaks of data governance, not of the absence of bias.

What the AI Act requires

Recruitment, selection, workforce management decisions and performance evaluation are high-risk areas under Annex III. The provider of such systems is bound by Article 10 on data and data governance. Under paragraph 2, training, validation and testing data sets must be subject to governance practices appropriate for the intended purpose, which include the data collection processes and origin, the formulation of assumptions about what the data is meant to measure and, in particular under points (f) and (g), the examination of possible biases that are likely to affect health and safety, have a negative impact on fundamental rights or lead to discrimination prohibited under Union law, and measures to detect, prevent and mitigate them. Under paragraph 3 the data must be relevant, sufficiently representative and, to the best extent possible, free of errors and complete, where necessary also with regard to the groups of persons on whom the system is intended to be used.

Paragraph 5 allows the provider exceptionally to process special categories of personal data, such as ethnic origin, to detect and correct bias, under strict conditions: only if the same cannot be achieved with other data, pseudonymised, with controlled access, without passing the data on, and deleting it once the bias is corrected. This is an important detail: bias cannot be measured if the group cannot be identified, and the regulation resolves the tension by allowing measurement within limits.

The deployer's, that is the employer's, obligations are in Article 26: use according to the instructions, human oversight by competent and trained persons, relevance of input data to the extent the deployer controls it, monitoring of operation and informing workers before use. These are covered in High-risk AI. The obligations apply to Annex III systems from 2 December 2027, but the prohibitions of discrimination are in force now.

What the GDPR requires

Article 22(1) of the GDPR gives the data subject the right not to be subject to a decision based solely on automated processing, including profiling, which produces legal effects concerning them or similarly significantly affects them. A recruitment decision, a dismissal and a benefit decision are such decisions.

The exceptions in paragraph 2 are necessity for entering into or performing a contract, a law that provides safeguards, and explicit consent. Under the contract and consent exceptions, paragraph 3 requires suitable measures: at least the right to obtain human intervention, to express one's point of view and to contest the decision. Under the legal exception the safeguards are laid down in the law that permits the decision. Paragraph 4 prohibits such decisions based on special categories of data almost entirely.

In recruitment, which no law specifically permits to be automated, this means AI may screen, rank and suggest, but a human makes the decision, and the human must genuinely assess. An overseer who approves every recommendation without checking is not human oversight but a rubber stamp, and the decision is then in fact solely automated. Data subject rights as a whole are covered in Data subject rights.

The prohibition of discrimination does not ask who decided

Finland's Non-Discrimination Act (1325/2014) prohibits discrimination on grounds including age, origin, language, religion, belief, state of health and disability, also in recruitment, and the Equality Act (609/1986) prohibits discrimination based on gender in working life. Both bind the employer regardless of whether a person, a spreadsheet or a model made the choice. Indirect discrimination, where an apparently neutral criterion puts a group at a disadvantage without a justified reason, is exactly what a biased model produces.

An employer cannot plead not knowing how the model decided. That is why the provider's instructions for use and information on performance across groups under Article 13 of the AI Act matter to the employer: they are how the employer can show it assessed the system before use.

What the employer does

Stage	Measure	Basis
Before procurement	Ask the provider for the Article 13 instructions, information on training data and tested performance by group	AI Act Articles 10, 13
Before use	Test on own material: do results differ by gender, age or language	Non-Discrimination Act, Equality Act
Before use	Data protection impact assessment and, in the public sector, a fundamental rights impact assessment	GDPR Article 35, AI Act Article 27
Before use	Inform workers' representatives and applicants	AI Act Articles 26(7), 26(11)
In use	Human oversight with the time, competence and authority to override a recommendation	AI Act Article 26(2), GDPR Article 22(3)
In use	Monitor outcomes by group and record deviations	AI Act Article 26(5)

Our skills catalogue treats AI bias and non-discrimination as a level 3 skill of the AI officer, and it is one of the requirements of the AI Officer Competence learning path. The AI officer's role is described in Who in the organisation is responsible for AI use?. Recruitment AI is also the point where the use policy's line "what may not be done without a separate assessment" is met first; the policy is covered in An AI use policy for the workplace.