Train-the-Trainer by DataHomeLab

11.2 Functions of Evaluation

The functions of evaluation can be classified under three headings. Each tells the trainer something different; an evaluation programme that uses only one of them is incomplete.

  • Diagnostic: diagnostic tests and examinations are designed to find the gaps in a student's knowledge so that remedial action may be taken.
  • Achievement: achievement tests and examinations are designed to measure achievement of a course aim.
  • Predictive: predictive tests and examinations are designed to give a measure of a student's future performance.

The three are separate and distinct: an examination which is valid for one purpose may not be valid for another. Normally, different examinations will be needed for the different functions. The instructor's first design decision when setting an assessment is which of the three functions the assessment is intended to serve, because that decision drives every subsequent choice (item type, timing, weighting, the action taken on the result).

11.2.1 Diagnostic tests

Effective training presupposes a feedback of information to the student on the effectiveness of his learning. This may be given to the student by:

  • verbal questions and answers;
  • correction of written work; and
  • results of diagnostic practical progress tests.

Formal written diagnostic tests, being part of a teaching technique, are designed to find out what a student does not know so that remedial action is possible.

In terms of usefulness, this first type of assessment is the most important. Diagnostic examinations have a positive training function, but if not followed up by remedial training they tend to fortify incorrect knowledge and concepts. Close scrutiny of remedial examination results enables the instructor to update and improve their instruction in the light of student difficulties.

11.2.2 Achievement tests

Achievement tests should be held at the completion of a phase of training. Their function is to assess whether the phase objectives stated in the syllabus have been attained. Where the diagnostic test asks "what does this trainee not yet know?", the achievement test asks "has this trainee reached the standard the phase declares?". The reference is the syllabus objective, not the cohort and not the trainee's earlier baseline.

The achievement test is therefore a 11.1 Purpose of Evaluation instrument by construction: it produces a pass-or-fail decision against a stated standard, and the action taken on the result is binary (proceed to the next phase, or remediate against the unmet objective and retest).

11.2.3 Predictive tests

While an achievement test is concerned with assessing performance in the course objectives, the predictive examination is more concerned with future performance: it is designed to predict later performance in the skills which the course has been designed to produce. Rather than measure what the student has learned on the course, the predictive examination attempts to assess to what extent he can transfer this training to the actual job situation.

In general, the production of a valid predictive test is a purely statistical operation: the predictive test paper consists of a battery of items which are found by statistical analysis to correlate most highly with job performance. This is a different test-design discipline from the other two: the items are selected for their correlation with downstream job performance, not for their content coverage of the syllabus. A predictive test built without that statistical foundation is not a predictive test; it is an achievement test wearing the wrong label.

How the three functions are examined

The three functions are examined in three main ways:

  1. The written examination.
  2. The practical (performance) examination, which is designed principally for the use of aircraft or other equipment.
  3. The oral examination (questions).

The evaluation material does not prescribe which mode goes with which function: that pairing is set by the 11.3 The Evaluation Cycle introduced in the Evaluation Cycle section, where the method chosen is the one that best approximates the behaviour specified by the objective. A practical handling skill is examined in the simulator or on the aircraft; a knowledge-of-systems item is examined in writing or orally; an attitudinal item often surfaces only in the oral interview.

The full taxonomy of evaluation methods is set out below as a tree (Objective and Subjective branches, with Recall, Recognition, Essay, and Practical Assessment as the leaf nodes).

11.2.4 Essential elements for accurate assessment

The assessment system cannot be divorced from instruction. It should therefore be:

  1. An on-going process.
  2. Systematic and planned.
  3. Integral part of the instructional process.
  4. The items used to make the assessment must be valid, reliable, objective, differentiating, and comprehensive.

The first three points reproduce the 11.1 Purpose of Evaluation in different language: assessment is part of training, not a discrete event tacked onto the end. The fourth point is the test-design quality bar.

The five item-quality criteria

Every test item the instructor sets must be measured against five criteria. A failure on any one of them invalidates the data the item produces; an instructor who grades from a flawed item is not measuring trainee performance, they are measuring the flaw.

Criterion What it requires Failure mode
Valid The test item measures what it is supposed to measure and includes a representative sample of the behaviours to be measured. A "flight handling" item that is actually a memory test for a numerical limit; the score correlates with recall, not handling.
Reliable A test or item is reliable if it gives consistent results. A reliable tape measure should give the same result regardless of the individual using it, or when it is used. An oral question whose grade depends on which examiner administers it; the score reflects the examiner more than the candidate.
Objective The device should not be affected by the evaluator's personal bias, for or against the student. An item where a candidate the examiner already considers strong is given the benefit of the doubt that a candidate they consider weak is not; the score is the bias.
Differentiating The device should clearly distinguish between the masters and non-masters. The item should not be designed so that it is so easy that all students pass or so hard that all students fail. An item every candidate gets right (or wrong); the item carries no information.
Comprehensive The test should take as wide a sample of selected behaviours as possible. It is sometimes possible to evaluate the total behaviours as in a lesson objective, but the overall evaluation of a student's course performance should include as many of these samples as possible. A course-final assessment that examines only one of the eight competencies the syllabus declared; the grade misses 7/8 of the curriculum.

The downstream EBT operational rubric

The five item-quality criteria above set the test-design floor. Downstream of that floor, the EBT programme realises the criterion-referenced posture introduced in 11.1 Purpose of Evaluation through a specific operational tool: a 9-competency × 5-grade word-picture rubric. The rubric pairs each of the nine pilot core competencies (eight ICAO competencies plus KNO, the knowledge competency used in this framework) with a five-level descriptor scale, where each level is anchored to a sentence describing what performance at that level looks like.

The structure is the same across all nine competencies. Grade 5 is exemplary; grade 1 is a failure. Two illustrative descriptors from the rubric:

This rubric is the EBT-specific implementation of the profile-reporting recommendation from 11.1 Purpose of Evaluation: each trainee's grade is recorded as a vector of nine grades, one per competency, against word-picture standards. The grading methodology that drives the rubric (when grade 2 or below is recorded for any competency, additional training is mandatory) is set out in A4.2.4 Grading Methodology for Recurrent Training and Checking.

The taxonomy of evaluation methods

Where the previous sections set out the purpose of an evaluation (diagnostic, achievement, predictive) and the quality of the items (valid, reliable, objective, differentiating, comprehensive), the evaluation material also classifies the form the items take. The evaluation-method tree splits methods into two top-level branches: objective methods (which produce a single correct score regardless of marker) and subjective methods (which require examiner judgement to convert performance into a score).

Evaluation method tree
  • Objective methods are split into:
    • Recall items (short answer, completion). The candidate produces the answer.
    • Recognition items (true/false, multiple choice). The candidate selects the answer from supplied options.
  • Subjective methods are split into:
    • Essays. The candidate produces extended written reasoning; the marker grades against a structured rubric.
    • Practical assessment. The candidate performs the task in the operating environment (or a faithful simulation); the examiner grades behaviour against a structured rubric.

Figure: Reliability versus validity

The reliability and validity criteria above are the most commonly conflated of the five item-quality criteria. The classical visual contrast uses a four-target diagram: a tight cluster off-centre is reliable but not valid (consistent, but consistently wrong); a wide scatter centred on the bullseye is valid but not reliable (right on average, but unpredictable per shot); a wide scatter off-centre is neither; a tight cluster on the bullseye is both.

Reliability vs validity

Reliability vs validity: the four-target diagram. Reliability is consistency of measurement; validity is whether the measurement actually targets the right thing. The two are independent and both are required.

Figure: Kirkpatrick's four levels of training evaluation

The three functions above (diagnostic, achievement, predictive) classify evaluation by purpose within the training programme. A complementary frame, widely used in instructional-design and evaluation literature, classifies evaluation by what is being measured: Donald Kirkpatrick's four levels (1959, refined in subsequent decades).

Kirkpatrick four levels of evaluation

Kirkpatrick's four levels of training evaluation: Reaction (did trainees like the training?), Learning (did they acquire the intended knowledge and skills?), Behaviour (do they apply it on the job?), and Results (does the application produce the operational outcomes the training was designed for?).

The four levels increase in operational significance and in measurement difficulty as the level rises. Reaction (level 1) is cheap to measure (a post-course survey) but weakly diagnostic of training effectiveness; trainees reliably enjoy training that does not change behaviour. Learning (level 2) is what the achievement tests above measure: knowledge and skill at the end of the course. Behaviour (level 3) is what the predictive tests aspire to: does the trainee use the training on the line. Results (level 4) is the operational outcome the training was supposed to produce, expressed in an operator's terms (incident rate reduction, on-time performance, fuel burn reduction, customer-experience scores). EBT recurrent training is, in Kirkpatrick terms, predominantly a level-2 and level-3 instrument; an operator's safety-management system is the place where level-4 outcomes get attributed back to the training programme.

Cross-references