Train-the-Trainer by DataHomeLab

11.3 The Evaluation Cycle

To ensure that evaluation is planned, systematic and that the essential elements are covered, a five-step closed-loop evaluation cycle can be used. The cycle is the operational procedure the instructor walks every assessment through, from the first lesson plan to the final review of results.

Evaluation cycle

The five steps:

  1. Define Objectives.
  2. Select the Criteria.
  3. Choose the best Method.
  4. Administer the Evaluation.
  5. Draw Conclusions.

The cycle is closed: conclusions feed back to the objectives, either confirming the objective has been met or surfacing a deficiency that drives the next training intervention. An instructor who runs steps 1 to 4 without step 5, or who runs step 5 without feeding it back to step 1, has not closed the loop.

11.3.1 Define the objectives

Refer to the appropriate syllabus and determine exactly what is to be tested for. The first step is the smallest in word count and the largest in operational consequence: every subsequent step is conditioned by what is decided here.

The objective also fixes which of the three functions (11.2 Functions of Evaluation) the assessment is serving. A "find the gaps so they can be remediated" objective is diagnostic; a "confirm the phase standard has been met" objective is achievement; a "predict performance on the line" objective is predictive. The same content can be examined under any of the three; the objective is what tells the instructor which one is in play.

11.3.2 Selecting the criteria

Selecting the criteria, or planning the assessment, is the first step in ensuring your evaluation is valid, reliable, objective and comprehensive. The test specification (plan) sets out the parameters (limits) of your evaluation.

The test specifications have three parts:

  1. Written.
  2. Practical.
  3. Oral.

Before questions are written, the test specifications can be used to determine the following:

  • Spread or comprehensiveness of the test items. How wide a sample of the syllabus does the test draw from? A test that examines only one of the syllabus's twelve declared topics is not comprehensive, regardless of how good the items are.
  • Conditions under which an objective is to be tested. What environment, what time pressure, what allowable references, what assistance? An objective tested under conditions different from those it was taught under is testing transfer, not attainment, and the difference must be acknowledged.
  • Item format and fidelity. The form of the question (multiple choice, short answer, essay, practical exercise) and how closely it approximates the operational task. See [[#11.3.3 Choose the best method|the fidelity principle below]].
  • Number of items in the test for time given. The candidate has finite time; the test has a finite number of items. The trade-off between depth of sampling and the time pressure on each item is an explicit design choice.
  • Weighting and marking systems, as related to the difficulty level of items. Not every item is worth the same number of marks; not every error costs the candidate the same. The weighting must be declared in advance (so the marking is reproducible) and must reflect the operational consequence of the behaviour the item is sampling.

11.3.3 Choose the best method

At this step of the cycle, which has been partly completed by construction of the test specifications, the trainer must consider the most appropriate method of evaluation given time and resource constraints. In selecting the method the fidelity principle must be obeyed.

The fidelity principle is the rule that breaks the tie when several methods could plausibly be used. Cost, time, scheduling, and equipment availability all push toward lower-fidelity methods. The fidelity principle says: those constraints can be respected, but only by acknowledging the loss of fidelity that the compromise produces. A multiple-choice test of takeoff abort decision-making is cheaper than a simulator session, but it is not measuring takeoff abort decision-making; it is measuring recognition of a written description of takeoff abort decision-making. Both can be defensible, but the instructor must know which one is being measured.

Where possible, all practical skills must be assessed by practical tests. Knowledge or cognitive skills are tested by the written format. It is very important that written test items are constructed to reflect the level of cognitive skills required (classes of objectives). The concept of a hierarchy of intellectual skills (Bloom's taxonomy) should be applied. The objectives for knowledge-type skills require a different performance and question construction from the objectives for evaluation-type skills.

Annex A (referenced in the source) lists the principles to be observed when preparing written items for different classes of objectives. Annex B (referenced in the source) provides illustrative items for each evaluation format.

The evaluation method tree (continued)

The evaluation-method tree, introduced in 11.2 Functions of Evaluation, is reproduced here as the operational reference for method selection. The tree splits methods into:

  • Objective methods, which produce the same score regardless of marker:
    • Recall items: short answer, completion.
    • Recognition items: true/false, multiple choice.
  • Subjective methods, which require examiner judgement:
    • Essays.
    • Practical assessment.

The fidelity principle determines which leaf of the tree the assessment lands at. A handling-skills objective lands at practical assessment; a recall-of-limits objective lands at recognition or recall items; an extended-reasoning or judgement objective lands at essays or oral practical. The cost of getting the wrong leaf is not just an inaccurate grade: it is an inaccurate grade misrepresented as accurate, because the test instrument was the wrong one for the objective.

11.3.4 Administer the test

Controlling the test environment

The test environment can also alter a student's performance. Care must be taken to reduce the variables that adversely affect the result. These variables can be summarised as:

  1. Environmental variables: light, temperature, noise.
  2. Personal variables: examinee's state of health, time of day.
  3. Instructions given: oral instructions must be standardised.
  4. Examiner's attitude.

Each of the four variables corresponds to a familiar CRM threat that the candidate cannot manage in this context (because the examiner controls the conditions of administration). The instructor's job in step 11.3.4 is to remove the variables they can control and acknowledge the ones they cannot.

11.3.5 Draw conclusions

Once the evaluation device has been administered the final step in the cycle can be completed. This consists of:

  1. Checking on the performance of the evaluation measure used; this may include task analysis.
  2. The effectiveness of training in terms of student's performance compared to standard required.
  3. The effectiveness of the instruction given.

The three points map directly to the three things the 11.1 Purpose of Evaluation routes feedback against. The evaluation measure itself is the first conclusion, not the last: an item every candidate gets wrong may be a sign the instruction failed; equally likely, it is a sign the item is malformed. Task analysis (re-reading the item against the operational task it is supposed to sample) is what disambiguates the two.

However, before meaningful conclusions can be drawn as to the knowledge and skills of the graduate, or the effectiveness of the training organisation, the methods of evaluation used must prove to be valid, reliable, objective, comprehensive and differentiating. The five item-quality criteria from 11.2 Functions of Evaluation are the floor under any conclusion drawn here: a conclusion drawn from a flawed instrument is a flawed conclusion, no matter how confident the analysis.

How the cycle pairs with the EBT module

The five-step cycle is the generic form. The EBT module realises the cycle with EBT-specific instantiations of each step:

  • Define objectives: the EBT competency framework (the eight ICAO competencies plus KNO) and the course syllabus declare the objectives the module is testing for. See A4.D Core Competencies.
  • Select criteria: the test specification is the EBT lesson plan, with manoeuvres and scenarios drawn from the per-generation appendices. See A4.2.1 Guidance for Examiners and A4.2.2 Guidance for Instructors.
  • Choose the best method: the EBT three-phase structure (evaluation phase, manoeuvres training phase, scenario-based training phase) selects the highest-fidelity method available (the simulator with the approved programme scenarios) for each objective. See ICAO Doc 9995 Conduct of EBT.
  • Administer the evaluation: the briefing-and-conduct discipline of the EBT module, including the standardised oral and the controlled simulator environment. See A4.2.3 Conduct of Briefing.
  • Draw conclusions: the per-competency grade against the 9-competency × 5-grade word-picture rubric, with the grading-methodology rule (grade 2 or below triggers additional training) closing the loop. See A4.2.4 Grading Methodology for Recurrent Training and Checking and A4.E Performance Grades.

The cycle is the same. The instantiation is what makes it operational.

Cross-references