Train-the-Trainer by DataHomeLab

11.1 Purpose of Evaluation

Why a trainer evaluates

Evaluation and testing opens with two questions an instructor must be able to answer at any point in a course:

  • How does a trainer know whether the course is successful?
  • If it is not successful, what does the trainer do about it?

To answer either question the trainer needs information on how many trainees are meeting the objectives of the course, and what corrective action is required if trainees are not meeting the objectives. The purpose of evaluation is to ensure that only those trainees who meet the objectives are allowed to graduate. More importantly, evaluation provides a means for the continuous monitoring of the quality and improvement of training when it does not achieve the desired results.

The third item is the one many instructors miss. Evaluation tells the trainer about the trainee's performance, but it also tells the trainer about their own. A cohort that consistently fails the same item is not necessarily a cohort of weak students; it is at least as likely to be a signal that the instruction on that item, or the test item itself, needs revision. The evaluation data is the feedback the instructor uses to update both the trainee and themselves.

The closed-loop diagram

Evaluation and testing introduces the operating shape of any programme of instruction with a single feedback-loop figure: training starts from the Objective, moves through Progress Tests and a Final Test, and the results of those tests feed back to the Objective. The trainee exits the loop only when graduation criteria are met.

Evaluation feedback loop

The figure compresses three things at once. First, evaluation is positioned inside the training cycle, not after it: progress tests and the final test are equally part of the loop, and the results of both are routed back to the objective. Second, the feedback arrow is the operational element: an instructor who administers a progress test but does not act on the result has not closed the loop, and the cycle degenerates into measurement without correction. Third, the objective is the unmoved centre: every test, every grade, every remedial decision has to be referenced against the same objective the syllabus declared.

How students should be assessed: NRT versus CRT

Two assessment philosophies are available, and evaluation and testing is unambiguous about which one belongs in operationally-relevant flight training.

Normative Referenced Testing (NRT)

Assessment of students could be done by using a Normative Referenced Testing System (NRT), where one student's performance is compared to another student's performance. Smith ranks above Jones; Jones ranks above Brown; the cohort produces a ranked list. Comparing Smith's performance to Jones', however, tells us little about how well they can carry out an operational task. NRT answers "who is best?", not "is anyone good enough?". For a domain in which the consequence of being below a fixed standard is operationally significant (a pilot who cannot fly to the defined course standard does not become safe by being ranked above another pilot who also cannot), this is the wrong question.

Criterion Referenced Testing (CRT)

Where syllabi have been prepared in accordance with Systematic Approach to Training (SAT) principles, a Criterion Referenced Testing System (CRT) can be adopted. A criterion-referenced testing system judges a student's performance in terms of job-related criteria. Students are graduated in terms of being able to meet the stipulated job standards. The reference is no longer the rest of the cohort: the reference is the job.

Profile reporting

Using CRT means that the assessment measures, providing they are valid and reliable, generate considerably more usable information about a student's strengths and weaknesses than a single ranked grade. To retain this information, profile reporting of results should be used. Each student's weaknesses and strengths in relation to the set standard for a phase of training are recorded, not collapsed to a single mark. This provides the employing unit with a much better picture of the graduate than the traditional grading. Excellence can still be recorded by ranking in a graduation order from the nominated standard if required: profile reporting and ranking are not exclusive, but the ranking is supplementary to the criterion-referenced profile, not a substitute for it.

Cross-references