Evaluation cycle
The evaluation cycle is the instructor's closed-loop procedure for knowing whether training is working and what to do when it is not. Evaluation is not a gate bolted onto the end of a course. It is continuous monitoring of objectives, trainee response, and the quality of the instruction itself. In airline training the correct philosophy is criterion-referenced testing (CRT): judge performance against job standards, not against the rest of the cohort.
Why evaluate
A trainer must always be able to answer:
- is the course succeeding against its objectives?
- if not, what corrective action is required?
Evaluation tells you how many trainees meet the objectives and whether the instruction or the test items need repair. A cohort that always fails the same item is as likely a design fault as a trainee fault.
Criterion-referenced versus normative
| Approach | Reference | Question answered | Fit for flight training |
|---|---|---|---|
| Normative referenced testing (NRT) | Other students | Who ranks higher? | Poor: rank does not equal safe |
| Criterion-referenced testing (CRT) | Job / syllabus standard | Is this person good enough? | Required: same stance as EBT grading |
Profile reporting keeps CRT information usable: strengths and weaknesses against standards, not a single collapsed mark. Competency × grade word pictures are a profile system, not a league table. EBT competence decisions reference the approved programme standards only.
Three functions of evaluation
An assessment must declare which function it serves; validity does not transfer across functions.
| Function | Question | Timing / action |
|---|---|---|
| Diagnostic | What gaps exist so we can remediate? | During training; must be followed by remedial teaching |
| Achievement | Has the phase standard been met? | End of phase; pass / remediate and retest |
| Predictive | How will this person perform on the job later? | Statistical design against job performance; not a relabelled achievement test |
A diagnostic exam only at course end is a contradiction: diagnosis has training value only when there is still time to act.
Five-step cycle
- Define objectives. Syllabus contract. Fixes function (diagnostic / achievement / predictive) and makes validity checkable.
- Select the criteria. Test specification: coverage, conditions, item format, number of items vs time, weighting and marking. Written before administration, it becomes the audit trail.
- Choose the best method. Obey the fidelity principle: the method should best approximate the behaviour the objective names. Multiple-choice of a rejected-take-off decision is not the same as flying it in the FSTD.
- Administer the evaluation. Conditions match the plan; bias controlled as far as practical.
- Draw conclusions. Feed results to trainee remediation, instructor self-correction, and objective or item redesign. Close the loop.
Item quality bar
Every item should be valid, reliable, objective, differentiating, and comprehensive. Fail any one and the score measures the flaw more than the trainee.
How this sits beside EBT grading
Observe, record, classify, evaluate (ORCE) and the evidence-based training competency grading method (VENN) are the shop-floor realisation of continuous criterion-referenced assessment inside a module. Chapter 11 supplies the design philosophy. Competency frameworks supply the standards. Inter-rater reliability keeps grades usable as data. Do not invent a parallel NRT "who is best in this course" culture alongside EBT profiles.
Instructor use
- State the objective and function before writing any item or scenario grade plan.
- Prefer CRT profiles over single ranks when reporting to the next instructor or line training.
- Match method fidelity to the behaviour: knowledge → written/oral; handling → FSTD/aircraft; behaviour → observed scenario performance.
- After diagnostic evidence, teach the gap the same day if possible.
- When a whole group fails one item, inspect the item and the teaching before blaming the group.
- Align oral and written questions with Bloom level of the objective.
Connections
- Systematic Approach to Training. Systematic Approach to Training (SAT) assessment design and CRT posture.
- Competency-based training. Competencies as the criteria CRT grades against.
- Core competencies. Competency standards behind profile grades.
- VENN grading. Four-dimension method that turns observations into competency grades.
- Word pictures. Competency × grade profile language CRT reports against.
- Inter-rater reliability. Reliability requirement for multi-instructor CRT systems.
- Behavioural indicators. Observable behaviours that make criteria concrete.
- Bloom's taxonomy. Cognitive level for knowledge items and oral checks.
- Questioning technique. Oral examination craft.
- Evidence-based training. Continuous criterion-referenced assessment in EVAL/MT/SBT.
- Fault analysis. Diagnostic reading of performance breakdowns.
- Day 1 versus Day 2 grades. CRT reading of Day 1 versus Day 2 grades; cohort-wide Day 2 collapse as design signal.
- Instructor competencies. Includes fair, consistent assessment practice.
Sources
- 11.1 Purpose of Evaluation. Continuous evaluation; NRT vs CRT; profile reporting; closed-loop framing.
- 11.2 Functions of Evaluation. Diagnostic, achievement, predictive; five item-quality criteria; examination modes.
- 11.3 The Evaluation Cycle. Five-step cycle; test specification; fidelity principle.
- 1.4 Methods of Training and Checking. SAT cycle and CRT alignment.
- A4.2.4 Grading Methodology for Recurrent Training and Checking. EBT operational CRT grading.
- A4.E Performance Grades. Competency × grade profile rubrics.
- Doc 9995, Part I Ch 7. Competence against defined programme standards.