A2.7 Unit 6 Perform Course Evaluation
Expanded Performance Descriptor. The competent instructor / evaluator should evaluate the effectiveness of the training system.
Unit 6 is the system-level evaluation Unit. Where Unit 5 graded the trainee, Unit 6 grades the instructor population, the courses themselves, and how that evaluation is fed back into improvement. The Unit applies primarily to instructors and examiners with quality-assurance, standardisation, or course-design responsibilities, but every instructor's recognised-training-opportunities reporting (Unit 5 Element E Desirable Behaviour d) feeds the same system. Three Elements decompose the system-evaluation work.
Element A. Evaluates the effectiveness of instructor's performance
This Element applies where required to assess / coach other instructors. Six Desirable Behaviours cover the dimensions of an instructor's performance the senior instructor or examiner observes when evaluating peers.
Desirable Behaviours
a) Evaluates instructor's communication skills.
b) Evaluates instructor's presentation skills.
c) Evaluates instructor's facilitation skills.
d) Evaluates instructor's use of training media.
e) Evaluates instructor's use of instructional materials.
f) Evaluates instructor's assessment of trainees.
The six behaviours map cleanly onto Unit 4 (Conduct training) and Unit 5 (Perform trainee assessment) as observed from above:
| Element A behaviour | What it observes | Unit / Element being graded |
|---|---|---|
| (a) Communication skills | The observed instructor's verbal and non-verbal communication | Unit 4 Element C Desirable Behaviour a |
| (b) Presentation skills | The observed instructor's voice, body language, pacing, sequencing, training-aid use | Unit 4 Element B in full |
| (c) Facilitation skills | The observed instructor's application of the facilitation technique in instruction and debriefing | Unit 4 Element C Desirable Behaviour g; the LOFT facilitator behaviours of Appendix 1 |
| (d) Use of training media | The observed instructor's effective use of training devices, AV, simulator IOS functions | Unit 4 Element B Desirable Behaviour e and Unit 4 Element G in full |
| (e) Use of instructional materials | The observed instructor's correct and current use of company-approved courseware | Unit 4 Element D Desirable Behaviour c |
| (f) Assessment of trainees | The observed instructor's whole conduct of evaluator role | Unit 5 in full |
The Element's "where required to assess / coach other instructors" qualifier matters: the average line instructor is not running peer evaluations and so will not be graded on Element A in their recurrent evaluations. The Element is for senior instructors, examiners, training-department line managers, and standardisation pilots whose roles include observing and coaching colleagues.
Element B. Evaluates the effectiveness of a course or phase of a course
Element B is the course-level evaluation. Six Desirable Behaviours cover the trainee feedback, the trainees' mastery of objectives, and the four input-condition factors (facilities, equipment, materials, programme management) that conditioned the trainees' performance.
Desirable Behaviours
a) Evaluates trainee's feedback on the training process.
b) Evaluates trainee's mastery of end-of-course objectives.
c) Evaluates the effect of facilities on trainee's performance.
d) Evaluates the effect of equipment on trainee's performance.
e) Evaluates the effect of training materials on trainee's performance.
f) Evaluates the effect of the management of the training programme on trainee's performance.
The six behaviours decompose course evaluation into the trainee voice (a), the assessment-against-objectives evidence (b), and the four input factors that conditioned the result (c, d, e, f). The structure is recognisable as a basic systems-evaluation model: evaluate the outcome (b), evaluate the experience (a), and evaluate each of the inputs that produced the outcome (c through f). A negative trainee outcome on a particular course component is interpretable only against this structure: was the outcome itself measured well (b)? Was the experience adverse in a way that contributed (a)? Were any of the input factors below the level needed to produce the outcome (c, d, e, f)?
Element C. Reports information on course evaluation
Element C is the reporting half of course evaluation. Eight Desirable Behaviours, the largest count in the Unit.
Desirable Behaviours
a) Identifies strengths and / or weaknesses of the training course.
b) Identifies systemic safety issues.
c) Identifies unexpected outcomes.
d) Identifies barriers to the transfer of learning.
e) Makes recommendations for improvements to course design.
f) Makes recommendations for improvements to course documentation.
g) Shares information with other instructors and management.
h) Makes recommendations for improvements to training media and facilities.
The eight behaviours fall into four groups:
| Group | Behaviours | What they produce |
|---|---|---|
| Identification | (a), (b), (c), (d) | The findings: what was observed in the course evaluation that is worth acting on |
| Recommendation | (e), (f), (h) | The proposed actions: improvements to design, documentation, media, facilities |
| Communication | (g) | The closure: the reporting channel by which findings and recommendations reach the audiences who can act on them |
The "transfer of learning" wording in (d) is an instructional-design term: it refers to whether the trainees, having completed the course, actually carry the learning forward into the operational environment the course was preparing them for. A barrier to transfer might be a course component that demonstrably produced post-test mastery but that the trainees did not carry into the line operation; identifying and reporting such barriers is the framework's call-out of an evaluation dimension that rote post-course testing would miss.
Behaviour (g), sharing information with other instructors and management, is the closure that converts the evaluation work into a system-level effect. An evaluation whose findings sit in a personal file does not improve the course; the framework grades the instructor / evaluator on whether the information is shared with the audiences who can act on it.
Connections
- A2.6 Unit 5 Perform Trainee Assessment. The previous Unit: trainee assessment, which feeds Element B Desirable Behaviour b (mastery of end-of-course objectives) and Element E Desirable Behaviour d's recognised-training-opportunities reporting feeds Element C in full.
- A2.8 Unit 7 Continuously Improves Performance. The next Unit: the instructor's responsibility to themselves, which feeds the same continuous-improvement loop Unit 6 operates at the system level.
- 11.1 Purpose of Evaluation. The cluster's depth treatment of evaluation that includes course-level evaluation methodology.
- 11.3 The Evaluation Cycle. The systematic evaluation cycle that Unit 6 operates within.
- 12.12 Quality Processes associated with Operations Training. The quality-assurance dimension of operations training that Unit 6 grades.
- 12.20 Identifies and Recommends Areas for Improvement. Identifying and recommending areas for improvement; Element C in particular draws on that discipline.