Inter-rater reliability
Inter-rater reliability (IRR) is the consistency of scores when different instructors or examiners assess the same performance. ICAO's glossary definition is short: consistency or stability of scores between different raters. If Instructor A grades Situation Awareness at 2 and Instructor B would have graded the same crew 4 on the same evidence, the grade is not a property of the pilot; it is a property of the roster. Training data, licence outcomes and remediation decisions built on that noise are not trustworthy.
IRR is distinct from, and incomplete without, assessment validity: alignment with the approved terms of reference (the adapted competency model, word pictures and policy thresholds). Perfect agreement on the wrong standard is reliable nonsense. Doc 9995 pairs the two under instructor standardisation: agreement among EBT instructors and alignment with those terms of reference.
Why EBT cannot live without it
Competency-based and evidence-based programmes replace simple manoeuvre pass/fail with multi-competency graded data. That data:
- drives individual remediation and ADDITIONAL TRAINING REQUIRED decisions;
- feeds fleet and airline performance analysis;
- steers scenario design and programme change over time.
Doc 9868 lists calibrated instructor/assessor judgement (high inter-rater reliability) among the principles of competency-related training and assessment. Assessments must be reliable: all assessors should reach the same conclusion; they must be trained and monitored to achieve and maintain an acceptable level of IRR. Summative assessments with legal and safety implications especially need systematic evidence tools so raters converge.
Building a usable grading system, then training instructors and assuring IRR, is among the hardest parts of implementing competency frameworks. Word pictures exist largely for this problem: shared verbal anchors for each grade level so raters converge on the same standard and the grade stays valid.
What produces agreement
| Mechanism | Role |
|---|---|
| Shared competency model and indicator language | Same labels and evidence menu |
| Word pictures / grade cells (VENN-encoded) | Shared verbal anchors for levels 1–5 |
| Observe–Record–Classify–Evaluate discipline | Same process, notes that can be challenged |
| Instructor training on the model and grading | Initial competence to use the tools |
| Standardisation / concordance sessions | Watch the same video or event; compare grades; resolve drift |
| Instructor Concordance Assurance Programme (ICAP) | ICAO name for the ongoing consistency system; terms of reference may be set by a standards group to guard against individual bias |
| Recurrent standardisation | Annual refresh; Doc 9995 also expects re-assessment of specified instructor areas on a multi-year cycle |
| Monitoring / sampling of grades | Detect rater outliers (always-3, never-below-4, harsh on one competency) |
Failure modes
- No record. Grade without timeline notes cannot be compared or coached.
- Checklist grading. Counting indicators instead of applying word pictures produces false precision and false disagreement.
- Personal thesaurus. "Pretty solid" is not level 3; quote the cell.
- Halo / horns. One brilliant (or ugly) manoeuvre colours all nine competencies.
- Phase mix-up. Applying Day-1 checking severity to Day-2 process-focused scenario training, or grading in-seat instruction.
- Threshold confusion. Mixing ICAO example "required ≥3" language with an illustrative COMPETENT = all ≥2 rule (or the approved local floor).
- Untrained facilitators. Strong line pilots who never practiced classify and evidence-based training competency grading method (VENN) will not converge.
Instructor use
- In every grade of 1, 2 or 5, leave indicator evidence another instructor could re-score.
- In standardisation, defend grades with four VENN dimensions, not narrative impressions.
- When your grade and a peer's diverge, find the first broken step (different observation, different classify, different evaluate).
- Self-check for signature bias (soft COM, harsh FPM, middle-box everything).
- Treat ICAP / standardisation as part of the job, not an administrative intrusion: without it your careful grades do not mean the same thing as the next rostered examiner's.
Connections
Evaluation cycle. Reliability is one of the five item-quality criteria and a criterion-referenced testing (CRT) system requirement.
Word pictures. Shared verbal anchors that make grades comparable across raters.
VENN grading. Primary methodological tool for consistent evaluate step.
Behavioural indicators. Shared evidence language for classify.
Core competencies. Shared target set being graded.
Instructor competencies. Unit 5 (trainee assessment) and continuous improvement; ICAO IEC5 grading OBs.
From competency to observable behaviour. IRR is the social/process condition that makes the chain trustworthy across the instructor body.
Evidence-based training. Programme data quality depends on IRR.
Competency-based training. competency-based training and assessment (CBTA) principle: calibrate assessors for high IRR.
Day 1 versus Day 2 grades. Phase mix-up (Day-1 check severity on Day-2 process grades) as a practical IRR failure mode.
Sources
- Doc 9995, Glossary. Definition: consistency or stability of scores between different raters.
- Doc 9995, Part I Ch 6. ICAP: consistency includes IRR and assessment validity; standards group may set terms of reference.
- Doc 9995, Part I Ch 7 (Conduct of evidence-based training). Word pictures enhance IRR and assessment validity; recommended multi-dimension assessment method.
- Doc 9995, App 4 (Word pictures for competency assessment). Dimension scales that stabilise how many / how often / outcome / quality language.
- Doc 9868, Part I Ch 2. Principle of calibrated assessor judgement; reliable assessments; train and monitor assessors for IRR.
- A4.D Core Competencies. Grading system plus instructor training and IRR called out as the major implementation challenges.
- A4.E Performance Grades. Parallel word pictures as standardisation design.
- A4.2.4 Grading Methodology for Recurrent Training and Checking. Shared observe, record, classify, evaluate (ORCE) process and outcome rules raters must apply identically.