The Euro­pean Com­mis­si­on has published, as of [date], draft gui­de­lines on the clas­si­fi­ca­ti­on of AI systems (AIS) as high-risk AI systems (HRAIS) pur­su­ant to Artic­le 6 of the AI Act published. The gui­de­lines con­sist of seve­ral documents:

The draft is available for con­sul­ta­ti­on and will be fur­ther coor­di­na­ted with the Euro­pean AI Board befo­re adop­ti­on; the examp­les are express­ly not exhaus­ti­ve and are inten­ded to be updated on an ongo­ing basis.

Clas­si­fi­ca­ti­on of AIS as HRAIS

Gene­ral

In the first part, the Com­mis­si­on explains seve­ral prin­ci­ples („Hori­zon­tal Issues“) that form the basis of its under­stan­ding of indi­vi­du­al cases.

We should pro­ce­ed on the basis of the distinc­tion set forth in Artic­le 6 of the AI Act: HRAIS are systems that

  • a Pro­duct listed in Annex I or are safe­ty com­pon­ents the­reof (Art. 6, para. 1, Annex I); or
  • one of the eight Are­as Accor­ding to Annex III cor­re­spond to: 
    • Bio­me­trics
    • Cri­ti­cal infrastructure
    • Edu­ca­ti­on and voca­tio­nal training
    • Employment, Work­force Manage­ment, and Access to Self-Employment
    • Access to and use of essen­ti­al pri­va­te ser­vices, as well as essen­ti­al public ser­vices and benefits
    • Pro­se­cu­ti­on
    • Migra­ti­on, Asyl­um, and Bor­der Con­trol Management
    • The Admi­ni­stra­ti­on of Justi­ce and Demo­cra­tic Processes

Use Cases Accor­ding to Annex III

Annex III is of par­ti­cu­lar prac­ti­cal rele­van­ce. When clas­si­fy­ing mate­ri­als, the fol­lo­wing applies first and foremost:

  • Signi­fi­cant risk“: Annex III covers only systems that, by their inten­ded pur­po­se, are a signi­fi­cant risk to health, safe­ty, or fun­da­men­tal rights ent­ail. The final list and the fil­ter are both expres­si­ons of this pro­por­tio­na­te, risk-based approach. Accor­ding to Reci­tal 53 of the AI Act, a sub­stan­ti­al risk is absent if the system does not influence the sub­stance and thus the out­co­me of the decision:

    (53) It is also important to cla­ri­fy that the­re may be spe­ci­fic cases in which AI systems refer­red to in pre­de­fi­ned are­as spe­ci­fi­ed in this Regu­la­ti­on do not pose a signi­fi­cant risk of harm to the legal inte­rests pro­tec­ted under tho­se are­as becau­se they do not mate­ri­al­ly influence decis­i­on-making or do not sub­stan­ti­al­ly harm tho­se inte­rests. For the pur­po­ses of this Regu­la­ti­on, an AI system that does not mate­ri­al­ly influence the out­co­me of decis­i­on-making should be under­s­tood to be an AI system that does not affect the sub­stance — and thus the out­co­me — of decis­i­on-making, whe­ther human or auto­ma­ted. An AI system that does not mate­ri­al­ly influence the out­co­me of decis­i­on-making could include situa­tions in which one or more of the fol­lo­wing con­di­ti­ons are met. […]

  • Inten­ded Use as a Stan­dard: The only deter­mi­ning fac­tor is whe­ther the inten­ded use of the AIS falls under a use case listed in Annex III. This clas­si­fi­ca­ti­on must be made befo­re the pro­duct is pla­ced on the mar­ket or put into service.
  • Human Invol­vement That, in and of its­elf, does not chan­ge anything, becau­se human over­sight under Artic­le 14 of the AI Act is a com­pli­ance requi­re­ment for a high-risk system and not a fac­tor in its classification.

Low-risk excep­ti­ons

Artic­le 6(3) of the AI Act addres­ses the prin­ci­ple that insi­gni­fi­cant risks should not result in a clas­si­fi­ca­ti­on as high-risk under Annex III. An AIS that would other­wi­se fall under Annex III is, excep­tio­nal­ly, not con­side­red high-risk if at least one of the fol­lo­wing con­di­ti­ons is met (the Com­mis­si­on refers to this as the „fil­ter.“ Here, the “human-in-the-loop” fac­tor may well be rele­vant to the final clas­si­fi­ca­ti­on as an HRAIS).

The fol­lo­wing rules app­ly to the appli­ca­ti­on of this excep­ti­on — which applies only to Annex III, not to pro­duct-spe­ci­fic clas­si­fi­ca­ti­on under Annex I:

  • Inter­pre­ta­ti­on: The con­di­ti­ons listed are exhaus­ti­ve but alter­na­ti­ve. As excep­ti­ons rele­vant to fun­da­men­tal rights, they must be inter­pre­ted nar­row­ly. They all app­ly only to the ext­ent that the AIS does not signi­fi­cant­ly influence the out­co­me of a decision.
  • Pro­fil­ing: If the system enga­ges in pro­fil­ing as defi­ned in Artic­le 4(4) of the GDPR, no excep­ti­on applies. The­r­e­fo­re, if per­so­nal data is pro­ce­s­sed to eva­lua­te per­so­nal aspects, the low-risk excep­ti­on does not apply.
  • Agent-based systems: Dis­tri­bu­ted and agent-based archi­tec­tures are eva­lua­ted as a sin­gle system as soon as their shared tasks signi­fi­cant­ly influence a decis­i­on in a spe­ci­fic case. The fact that indi­vi­du­al com­pon­ents, taken on their own, fall under an excep­ti­on does not chan­ge this:

    (75) Whe­re seve­ral AI systems form part of a more com­plex AI system, such that their com­bi­ned inten­ded pur­po­se or joint out­puts mate­ri­al­ly influence an indi­vi­du­al decis­i­on, the com­bi­ned con­fi­gu­ra­ti­on is trea­ted as a sin­gle AI system for the pur­po­se of high-risk clas­si­fi­ca­ti­on. To pre­vent cir­cum­ven­ti­on of the high-risk clas­si­fi­ca­ti­on rules through system design, split archi­tec­tures are asses­sed as a whole.

    (76) By con­trast, strict­ly pro­ce­du­ral or pre­pa­ra­to­ry func­tions of an AI system that are lin­ked to a high-risk system remain eli­gi­ble for exemp­ti­on from high-risk clas­si­fi­ca­ti­on under Artic­le 6(3) of the AI Act whe­re they are genui­ne­ly sepa­ra­ble from the AI system […]

    Self-Assess­ment and Regi­stra­ti­on: The appli­ca­ti­on of an excep­ti­on must be review­ed pri­or to pla­cing the pro­duct on the mar­ket and, if appli­ca­ble, docu­men­ted. The AIS must also be regi­stered in the rele­vant data­ba­se (Art. 6(4), Art. 71 AI Act). Mar­ket sur­veil­lan­ce aut­ho­ri­ties may reclas­si­fy the AIS (Art. 80 of the AI Act) and may impo­se sanc­tions in cases of cir­cum­ven­ti­on (Art. 99 of the AI Act).

All of this can be sum­ma­ri­zed as follows:

Nar­row pro­ce­du­ral task (lit. a; „nar­row pro­ce­du­ral task“)

This invol­ves, for exam­p­le, refor­mat­ting, struc­tu­ring, or cate­go­ri­zing data; making value judgments; clas­si­fy­ing data as „useful“ or „less useful“; and scoring or ranking.

The Com­mis­si­on illu­stra­tes this with two examples:

  • An excep­ti­on applies: Sort­ing inco­ming appli­ca­ti­ons for admis­si­on to schools or uni­ver­si­ties based on the requi­red gra­de or edu­ca­tio­nal level, becau­se — or when — such systems clas­si­fy appli­ca­ti­ons into pre­de­fi­ned cate­go­ries (e.g., ele­men­ta­ry school, secon­da­ry school, high school, or spe­ci­fic clas­ses) based on defi­ned infor­ma­ti­on, wit­hout asses­sing sui­ta­bi­li­ty or making admis­si­on decis­i­ons (an inte­re­st­ing exam­p­le becau­se it invol­ves mini­mal risk).
  • The excep­ti­on does not app­ly:

Impro­ve­ment of a human out­co­me (lit. b; „impro­ve the result of a pre­vious­ly com­ple­ted human activity“)

This excep­ti­on applies to refi­ne­ments or qua­li­ty assu­rance that do not invol­ve a mate­ri­al reva­lua­ti­on. For this to app­ly, the fol­lo­wing con­di­ti­ons must be met:

  • A human review or decis­i­on has been com­ple­ted, and
  • led to a result and
  • This is impro­ved by the AIS wit­hout chan­ging the result—

Here are two examp­les of this as well:

  • An excep­ti­on applies:
    • Iden­ti­fy­ing errors or incon­si­sten­ci­es in the results
    • Sup­ple­men­ting Human Decis­i­ons with Addi­tio­nal Evidence
    • Con­ver­si­on of Human-Gene­ra­ted Content
  • The excep­ti­on does not app­ly: An AIS checks a result and pro­vi­des a signi­fi­cant­ly dif­fe­rent solution.

Iden­ti­fy­ing decis­i­on-making pat­terns or devia­ti­ons (lit. c)

This invol­ves purely ex post reviews, wit­hout influen­cing the decis­i­on its­elf. Unli­ke the other “low-risk” excep­ti­ons, this excep­ti­on can cer­tain­ly cover some­what more com­plex systems and also ser­ve to prepa­re the ground for a human decis­i­on. Howe­ver, three con­di­ti­ons must be met:

  • The human review must alre­a­dy be complete;
  • The AIS con­ducts only an ex-post ana­ly­sis, wit­hout pre­de­ter­mi­ning future decisions;
  • The AIS must not be desi­gned to replace or influence human decis­i­on-making. If such influence occurs, it must be fol­lo­wed by a genui­ne human review.

The­re is also an exam­p­le of this. An AIS would likely fall under the “low risk” excep­ti­on if it ana­ly­zes pre­vious clas­si­fi­ca­ti­ons made by a public admi­ni­stra­ti­on for the pur­po­ses of qua­li­ty assu­rance — through pat­tern reco­gni­ti­on that also iden­ti­fi­es devia­ti­ons in indi­vi­du­al decis­i­ons — and report­ing, but wit­hout pro­po­sing solu­ti­ons for future cases or eva­lua­ting employee performance.

Pre­pa­ra­to­ry Task (lit. d)

In con­trast to the nar­row pro­ce­du­ral excep­ti­on (sub­pa­ra­graph (a)), this con­cerns tasks per­for­med in pre­pa­ra­ti­on for a human decis­i­on (alt­hough such pre­pa­ra­ti­on may fall under both sub­pa­ra­graphs (a) and (d) at the same time). In this con­text, the out­put of the AIS may only be a gene­ral fac­tor that sup­ports decis­i­on-making, wit­hout the AIS con­tri­bu­ting a recom­men­da­ti­on or an assess­ment of an indi­vi­du­al case.

One exam­p­le is an AIS that pro­vi­des users with refe­ren­ces to rele­vant legal pro­vi­si­ons, infor­ma­ti­on on case law, and inter­nal gui­de­lines. The AIS neither ana­ly­zes indi­vi­du­al cases nor gene­ra­tes a result that signi­fi­cant­ly influen­ces the decision.

Reser­va­ti­on Regar­ding Profiling

All “low-risk” excep­ti­ons are sub­ject to pro­fil­ing as defi­ned in Artic­le 4(4) of the GDPR, that is, the auto­ma­ted pro­ce­s­sing of per­so­nal data for the pur­po­se of eva­lua­ting per­so­nal characteristics.

For exam­p­le, when a cus­toms aut­ho­ri­ty uses an AIS to assess the risk that goods may vio­la­te legal regu­la­ti­ons upon import, this does not invol­ve the eva­lua­ti­on of per­so­nal cha­rac­te­ri­stics. Such a risk assess­ment of ship­ments or goods does not con­sti­tu­te pro­fil­ing. This is dif­fe­rent from an AIS that, in the con­text of recruit­ment, iden­ti­fi­es devia­ti­ons from inter­nal recruit­ment gui­de­lines and, for this pur­po­se, eva­lua­tes the decis­i­ons of recrui­ters and per­so­nal cha­rac­te­ri­stics in con­nec­tion with job inter­views. While this could in its­elf con­sti­tu­te a case fal­ling under Artic­le 6(3)(c), the pro­ce­s­sing of per­so­nal data for the pur­po­se of pro­fil­ing the recrui­ters pre­clu­des this.

Spe­ci­fic Annex III Use Cases

Bio­me­trics (Annex III, No. 1)

This group of cases inclu­des three use cases:

  • Sec­tion 1(a): remo­te bio­me­tric identification;
  • Sec­tion 1(b): Bio­me­tric cate­go­rizati­on based on sen­si­ti­ve cha­rac­te­ri­stics within the mea­ning of Artic­le 9(1) of the GDPR (spe­cial-cate­go­ry per­so­nal data). Howe­ver, age and gen­der are not inclu­ded in this defi­ni­ti­on (the inter­pre­ta­ti­on under the GDPR is some­ti­mes broa­der and, accor­ding to some opi­ni­ons, may extend to all unch­an­geable phy­si­cal characteristics).
  • Sec­tion 1(c): Emo­ti­on reco­gni­ti­on, to the ext­ent that it is not alre­a­dy pro­hi­bi­ted under Artic­le 5 of the AI Act (emo­ti­on reco­gni­ti­on in the work­place and in edu­ca­tio­nal institutions).

The defi­ni­ti­on of bio­me­tric data is not pro­vi­ded by the GDPR, but rather by Artic­le 3, No. 34 of the AI Act:

  • no bio­me­tric data: track­ing a per­son in a store based on their clot­hing; emo­ti­on reco­gni­ti­on based on writ­ten text; licen­se pla­te recognition;
  • bio­me­tric data: Emo­ti­on reco­gni­ti­on based on keystrokes, facial expres­si­ons, body postu­re, or movements.

Recor­ded Accor­ding to the Com­mis­si­on, the­se include, for exam­p­le, the fol­lo­wing operations:

  • Com­pa­ring pho­tos or the voices of living indi­vi­du­als with archi­val mate­ri­al for iden­ti­fi­ca­ti­on purposes;
  • Com­pa­ri­son of foota­ge from sports sta­di­ums to iden­ti­fy troublemakers;
  • Detec­tion of dise­a­ses based on move­ment patterns;
  • Cate­go­rizati­on of air­line pas­sen­gers using facial recognition;

Not recor­ded include, for exam­p­le, the fol­lo­wing applications:

  • Bio­me­tric veri­fi­ca­ti­on used sole­ly as a means of matching data to con­firm iden­ti­ty (such as unlocking a device or gran­ting access)
  • Loca­ting a per­son wit­hout bio­me­tric matching;
  • Cate­go­ri­zing cus­to­mers by gender;
  • AI for con­tent moderation.

Spe­ci­fi­cal­ly with regard to the Emo­ti­on reco­gni­ti­on (to the ext­ent that AI is used in this context):

Recor­ded:

  • Use of body came­ras to iden­ti­fy aggres­si­ve individuals;
  • Emo­ti­on reco­gni­ti­on in video games to enhan­ce game­play, based, among other things, on facial expres­si­ons and eye movements;
  • Iden­ti­fy­ing poten­ti­al sources of con­flict at concerts;
  • Voice-based emo­ti­on reco­gni­ti­on of cus­to­mers in call centers.

Not recor­ded:

  • Detec­tion of fati­gue in drivers;
  • Reco­gni­ti­on of simp­le facial expres­si­ons (smi­les, etc.).

Cri­ti­cal Infras­truc­tu­re (Annex III, No. 2)

This cate­go­ry inclu­des only AIS that ser­ve as a safe­ty com­po­nent within the mea­ning of Art. 3, No. 14 of the AI Act, a direct pro­tec­ti­ve func­tion that ensu­re the phy­si­cal inte­gri­ty of infras­truc­tu­re in the sec­tors of digi­tal infras­truc­tu­re, road trans­por­ta­ti­on, and the sup­p­ly of water, gas, heat, or electricity.

Recor­ded are for example:

  • Traf­fic light con­trol to pre­vent phy­si­cal damage
  • Fire Alarm System in a Data Center

Not recor­ded On the other hand, examp­les include:

  • Pure ana­ly­sis of traf­fic flow wit­hout direct con­trol intervention
  • Ticket Manage­ment and Net­work Load Forecasts
  • pure cyber­se­cu­ri­ty components
  • systems that mere­ly opti­mi­ze, pro­vi­de infor­ma­ti­on, or organize

Edu­ca­ti­on and Voca­tio­nal Trai­ning (Annex III, No. 3)

Four use cases are listed here, each of which applies to all levels of edu­ca­ti­on, inclu­ding on-the-job and voca­tio­nal trai­ning and con­ti­nuing education:

  • Sec­tion 3(a): Deter­mi­na­ti­on of Admis­si­on, Enroll­ment, or Pla­ce­ment in Insti­tu­ti­ons and Programs
  • Sec­tion 3(b): Assess­ment of lear­ning out­co­mes, but only sum­ma­ti­ve (affec­ting gra­des or qua­li­fi­ca­ti­ons), not for­ma­ti­ve (to sup­port learning)
  • Sec­tion 3(c): Assess­ment of the appro­pria­te level of edu­ca­ti­on that a per­son recei­ves or can attain
  • Sec­tion 3(d): Moni­to­ring and Detec­tion of Pro­hi­bi­ted Beha­vi­or During Exams

The Com­mis­si­on explains its under­stan­ding as follows:

Access and Allo­ca­ti­on (lit. a)

Recor­ded:

  • auto­ma­ted admis­si­ons systems that eva­lua­te appli­ca­ti­ons, tran­scripts, and test scores
  • Tools for assig­ning lear­ners to a pro­gram or course
  • Systems that match job see­kers or employees with pro­fes­sio­nal deve­lo­p­ment programs

Not inclu­ded:

  • Recom­men­da­ti­on tools that sug­gest degree pro­grams to pro­s­pec­ti­ve stu­dents based on their own pre­fe­ren­ces and are inten­ded sole­ly to sup­port their decision-making
  • Infor­ma­ti­on chat­bots about admis­si­on requi­re­ments, wit­hout per­so­na­li­zed recommendations.

Low-Risk Excep­ti­on:

  • A simp­le data orga­ni­zer that extra­cts and cate­go­ri­zes job appli­ca­ti­ons (nar­row scope of the task)
  • Tool for the ex post review of aut­ho­rizati­on decis­i­ons that have alre­a­dy been made (pat­tern recognition)

Pro­fil­ing (the low-risk excep­ti­on does not apply):

  • Systems who­se data ana­ly­sis con­sti­tu­tes pro­fil­ing, such as auto­ma­ted school assign­ment based on place of resi­dence and per­so­nal characteristics
  • Matching for Con­ti­nuing Edu­ca­ti­on Programs

Assess­ment of Lear­ning Out­co­mes (lit. b)

Recor­ded:

  • Gra­ding and assess­ment systems that con­tri­bu­te to the final or inte­rim gra­de, even if a for­ma­ti­ve feed­back system simul­ta­neous­ly informs the teacher’s grading

Not inclu­ded:

  • adap­ti­ve lear­ning systems, intel­li­gent tuto­ring systems, lear­ning ana­ly­tics platforms
  • Tools to Sup­port Neu­ro­di­ver­gent Learners
  • Lan­guage or pro­nun­cia­ti­on prac­ti­ce apps used sole­ly by learners

Low-Risk Excep­ti­on:

  • Simp­le Gra­de Avera­ge Cal­cu­la­tor (nar­row-scope problem)
  • A system that checks tea­chers’ gra­ding pat­terns for anoma­lies and flags them for human review (pat­tern recognition)

Level of edu­ca­ti­on (lit. c)

Recor­ded:

  • adap­ti­ve pla­ce­ment tests
  • Systems that assign lear­ners with spe­cial needs to a pro­gram and level of support

Not inclu­ded:

  • Aggre­ga­ted trend ana­ly­ses of edu­ca­tio­nal tra­jec­to­ries wit­hout refe­rence to spe­ci­fic individuals

Exam Proc­to­ring (lit. d)

Recor­ded:

  • Proc­to­ring systems used during proc­to­red exams, such as tho­se that use facial reco­gni­ti­on, key­board, screen, or audio/video analysis

Not inclu­ded:

  • Pla­gia­rism and col­la­bo­ra­ti­on checks on sub­mit­ted assign­ments that take place out­side of the super­vi­sed, real-time exam set­ting and after the assign­ment has been tur­ned in

Employment (Annex III, No. 4)

Here, a distinc­tion must be made bet­ween two sce­na­ri­os, both of which should be inter­pre­ted broad­ly and func­tion­al­ly. The per­so­nal scope of appli­ca­ti­on extends bey­ond employees; free­lan­cers, self-employed indi­vi­du­als, ser­vice pro­vi­ders, and plat­form workers also fall under this sce­na­rio — regard­less of the type of con­tract — as soon as an AUS faci­li­ta­tes or con­di­ti­ons their access to work:

  • Sec­tion 4(a): Recruit­ment and sel­ec­tion of indi­vi­du­als, inclu­ding tar­ge­ted job postings, revie­w­ing and scree­ning appli­ca­ti­ons, and eva­lua­ting candidates.
  • Sec­tion 4(b): Con­trol over employment rela­ti­on­ships, spe­ci­fi­cal­ly decis­i­ons regar­ding con­tract terms, pro­mo­ti­ons, and ter­mi­na­ti­on; assign­ment of tasks based on beha­vi­or or cha­rac­te­ri­stics; and moni­to­ring and eva­lua­ti­on of per­for­mance and behavior.

Recruit­ment and Sel­ec­tion (lit. a)

The fol­lo­wing are inclu­ded here

  • preli­mi­na­ry steps (tar­ge­ted out­reach to can­di­da­tes, pre-scree­ning) and
  • the actu­al sel­ec­tion pro­cess (short­li­sting, scoring, ran­king, testing).

This inclu­des, for example:

  • the AI-based crea­ti­on of job postings, unless they fall under the “low risk” excep­ti­on (see below);
  • Systems that ana­ly­ze and fil­ter resu­mes and gene­ra­te scores, ran­kings („Top 5“), or sui­ta­bi­li­ty cate­go­ries („high fit“)
  • Sourcing tools that search for pro­files and crea­te shortlists
  • Came­ra-based apti­tu­de test­ing (such as asses­sing the visu­al abili­ties of pilot can­di­da­tes); eva­lua­ti­on of respon­ses in online assessments
  • App­ren­ti­ce Recruitment
  • Employment Agen­cy Pla­ce­ment Systems

Not inclu­ded:

  • Tools for job see­kers that the indi­vi­du­al con­trols them­sel­ves (resu­me opti­mizati­on, job recom­men­da­ti­ons for the candidate)
  • Employer bran­ding not tied to a spe­ci­fic job opening
  • Anony­mi­zed repu­ta­ti­on moni­to­ring of the employer
  • Tools that check job postings only for pro­ble­ma­tic wording
  • Onboar­ding infor­ma­ti­on chat­bots after hiring (though the­se may fall under sub­pa­ra­graph b if they Sup­p­ly or Moni­tor Power)

Low-Risk Excep­ti­on:

  • A job descrip­ti­on gene­ra­tor that crea­tes a job posting based on tasks and qua­li­fi­ca­ti­ons pro­vi­ded by a user
  • Veri­fi­ca­ti­on of a voca­tio­nal diplo­ma with a bina­ry result („con­firm­ed / not con­firm­ed“) (nar­row pro­ce­du­ral task)
  • Orga­ni­zing inco­ming appli­ca­ti­ons in a searcha­ble data­ba­se (pre­pa­ra­to­ry work)
  • Sche­du­ling inter­views, inclu­ding acce­s­si­bi­li­ty opti­ons (purely logistical)
  • Ex-post bias audit using anony­mi­zed data (pat­tern recognition).

Systems that flag and pre-screen appli­cants as „high risk“ are being intro­du­ced Pro­fil­ing con­ti­n­ue to be high­ly risky.

Regu­la­ti­on of Employment Rela­ti­on­ships (lit. b)

Decis­i­on“ should be under­s­tood in a func­tion­al sen­se and also inclu­des cases in which a per­son for­mal­ly makes a decis­i­on but reli­es pri­ma­ri­ly on the out­put. Only decis­i­ons that exce­ed a mate­ria­li­ty thres­hold are inclu­ded; not every day-to-day ope­ra­tio­nal decis­i­on counts.

Recor­ded:

  • Decis­i­ons regar­ding key terms of the con­tract (com­pen­sa­ti­on, working hours, vaca­ti­on), pro­mo­ti­ons, and ter­mi­na­ti­on (inclu­ding the non-rene­wal of fixed-term con­tracts and the per­ma­nent sus­pen­si­on of plat­form accounts)
  • Assig­ning tasks based on beha­vi­or or per­so­nal cha­rac­te­ri­stics (punc­tua­li­ty, respon­se time, relia­bi­li­ty scores; with­hol­ding deli­very slots for low accep­tance rates; ran­king free­lan­cers by rating)
  • syste­ma­tic moni­to­ring of per­for­mance and con­duct, the results of which are inclu­ded in the per­son­nel file

Not inclu­ded:

  • Assign­ment of tasks based on objec­ti­ve, neu­tral, exter­nal cri­te­ria (avai­la­bi­li­ty accor­ding to the work sche­du­le, requi­red qua­li­fi­ca­ti­ons such as a for­k­lift operator’s licen­se or a law licen­se, geo­gra­phic proximity)
  • Systems used exclu­si­ve­ly to com­ply with exter­nal legal obli­ga­ti­ons (such as tran­sac­tion log­ging to pre­vent mar­ket abu­se), for purely medi­cal or secu­ri­ty pur­po­ses, or to pro­tect com­pa­ny property
  • Systems that only alert the affec­ted indi­vi­du­al to poten­ti­al errors, wit­hout pres­su­ring them to per­form bet­ter and wit­hout noti­fy­ing the employer; day-to-day ope­ra­tio­nal decis­i­ons that do not alter con­trac­tu­al rights (assign­ment of office space, sche­du­ling of breaks during a shift)

Low-Risk Excep­ti­on:

  • a system that sim­ply indi­ca­tes when a fixed-term con­tract is set to expi­re, wit­hout pro­vi­ding an evaluation

For the record: Emo­ti­on reco­gni­ti­on in the work­place is pro­hi­bi­ted under Artic­le 5(1)(f) of the AI Act.

Access to Essen­ti­al Ser­vices (Annex III, No. 5)

The­re are four use cases to distin­gu­ish here:

  • Sec­tion 5(a): Eli­gi­bi­li­ty review for essen­ti­al public bene­fits con­duc­ted by or on behalf of govern­ment agen­ci­es, inclu­ding the gran­ting, reduc­tion, with­dra­wal, or reco­very of such benefits
  • Sec­tion 5(b): Assess­ment of cre­dit­wort­hi­ness or cal­cu­la­ti­on of a cre­dit score for individuals
  • Sec­tion 5(c): Risk Assess­ment and Pri­cing in Life and Health Insurance
  • Sec­tion 5(d): Assess­ment and Clas­si­fi­ca­ti­on of Emer­gen­cy Calls, Dis­patch of Emer­gen­cy Medi­cal Ser­vices, Pati­ent Triage

Cre­dit­wort­hi­ness and Cre­dit Score (lit. b)

This covers both the assess­ment of cre­dit­wort­hi­ness and the cal­cu­la­ti­on of a cre­dit score, each con­side­red sepa­ra­te­ly. The decisi­ve fac­tor in each case is the con­nec­tion to access to cre­dit or to an essen­ti­al pri­va­te service:

  • Key Ser­vices The­se include, for exam­p­le, housing, elec­tri­ci­ty sup­p­ly, and tele­com­mu­ni­ca­ti­ons ser­vices, as well as other ser­vices of com­pa­ra­ble importance to natu­ral per­sons. Fur­ther­mo­re, exclu­si­on from access to such ser­vices must be capa­ble of having com­pa­ra­b­ly serious con­se­quen­ces for the life, health, liveli­hood, or social par­ti­ci­pa­ti­on of the indi­vi­du­als con­cer­ned. Accor­ding to the Com­mis­si­on, the­se include, in par­ti­cu­lar, health and long-term care ser­vices, other uti­li­ty ser­vices such as gas and water sup­p­ly, and trans­por­ta­ti­on services.
  • At Finan­ce Depart­ment Access to the fol­lo­wing ser­vices, for exam­p­le, is essential: 
    • Pro­vi­si­on of a bank account;
    • Payment ser­vices;
    • Gran­ting of loans and credit;
    • Increa­se in a cre­dit line or cre­dit card limit;
    • Gran­ting of mor­tga­ge loans;
    • public finan­cial services.
  • Not signi­fi­cant The­se include the fol­lo­wing services: 
    • Acqui­si­ti­on of stocks and securities;
    • Access to mar­gin tra­ding (tra­ding on cre­dit or with leverage);
    • Access to com­plex finan­cial instruments;
    • Pre­mi­um cre­dit cards;
    • spe­cia­li­zed loan pro­ducts, such as lei­su­re or tra­vel loans.

Cases not included:

  • Cus­to­mer Clas­si­fi­ca­ti­on and Seg­men­ta­ti­on for Infor­ma­ti­on or Mar­ke­ting Purposes
  • Pri­cing Simu­la­ti­ons Wit­hout Cre­dit­wort­hi­ness Considerations
  • Cus­to­mer sup­port in fil­ling out the application
  • Appeal Pro­ce­s­sing Fol­lo­wing the Decision
  • Inter­nal pru­den­ti­al moni­to­ring of cre­dit expo­sure after loan origination

Fraud Detec­tion

As men­tio­ned, a high-risk AIS can be a cre­dit assess­ment. This does not app­ly to systems used pri­ma­ri­ly to detect finan­cial fraud, even if their out­put is also incor­po­ra­ted into a cre­dit assessment.

Insu­rance (lit. c)

  • This covers risk assess­ment and pri­cing for life and health insu­rance, inclu­ding pri­va­te long-term care insu­rance, cer­tain reti­re­ment savings insu­rance poli­ci­es, and resi­du­al debt life insu­rance policies
  • Insu­rance lines such as auto and home con­tents insu­rance are not included

Judi­cia­ry, inclu­ding arbi­tra­ti­on (Annex III, No. 8)

This invol­ves pro­vi­ding sup­port to judi­cial aut­ho­ri­ties or even to Arbi­tra­ti­on Pro­ce­e­dings in deter­mi­ning and inter­pre­ting the facts and the law, as well as in app­ly­ing the law

Recor­ded:

  • Inter­pre­ta­ti­on of mea­ning; appli­ca­ti­on of pre­ce­dents to the spe­ci­fic facts of the case
  • Pre­pa­ra­ti­on of draft decis­i­ons, inclu­ding sup­port­ing rationale.

Not recor­ded:

  • sup­port func­tions such as anony­mizati­on, tran­scrip­ti­on, sche­du­ling, or task assignment
  • Pure rese­arch and key­word search
  • Systems ope­ra­ted by par­ties or their legal repre­sen­ta­ti­ves who are not acting „on behalf of“ a judi­cial authority

Other Are­as

The Com­mis­si­on also comm­ents on other use cases, which are not dis­cus­sed in detail here:

  • Cri­mi­nal Pro­se­cu­ti­on (Annex III, No. 6)
  • Migra­ti­on, Asyl­um, and Bor­der Con­trol (Annex III, No. 7)
  • Demo­cra­tic Pro­ce­s­ses (Annex III, No. 8)