Train-the-Trainer by DataHomeLab
Concept

What counts as evidence in evidence-based training

In EBT, "evidence" is operational and training data that both (1) justified replacing event-catalogue recurrent training and (2) ranks what to train, how often, and which competencies matter for which threats. It is not the instructor's anecdote alone, not a single accident video used as curriculum, and not the grade sheet of one crew. It is multi-source, generation-aware, and continuously refreshed so the programme stays coupled to real risk.

Two jobs the same evidence does

  1. Case for change. Data showed hull-loss and serious-incident patterns still driven heavily by crew performance, while legacy regulation tracked early-generation accidents and accumulated tick-box events. Training results differed by manoeuvre and by aircraft generation. That package established that reform was necessary.
  2. Curriculum content. The same analyses produced assessment and training topics, frequencies, manoeuvre lists, and competency maps in Doc 9995's matrices. Evidence is therefore not only a political argument for EBT; it is the bill of materials for the baseline programme.

Training needs vary with manoeuvre and aeroplane generation. The same data both motivated EBT and shaped its curriculum.

Source classes

Source class What it contributes Blind spots
Accidents and serious incidents High-severity patterns, competency failures in worst outcomes, trend over time Rare events; hindsight; not representative of everyday risk
Flight data analysis (FDA) Rates and trends (unstable approaches, go-arounds, exceedances, some threat frequencies) Weak on why; definition-dependent; needs human context
Flight-deck observation (LOSA and similar) TEM in normal ops; threats/errors/UAS; context; monitoring and CRM realities Costly; snapshot bias; observer skill
Safety reports (ASR/MOR/confidential) Narrative threats, route/destination specifics, crew coping strategies Reporting culture bias; incomplete capture
Training data and metrics Competency grade distributions, topic-linked errors, skill retention, instructor calibration Measures the training system; must be read against line data
Training criticality survey Expert judgement of likelihood, severity, training benefit by phase and generation Subjective; needs cross-check with operational data
Manufacturer / shared industry data Fleet-wide trends, type-specific issues, benchmark risk events May not match your network or culture
Scientific and meta studies Supporting human-performance findings Variable operational transfer

Doc 9995's Data Report lineage (2014 first edition and 2021 amendment) combined these under peer review so conclusions rested on mutually reinforcing findings, not a single database. Updates require expert review and, for programme-affecting change, often civil aviation authority (CAA) involvement.

From raw data to a module topic

A practical chain:

  1. Collect across operations and training under safety management system (SMS) data-protection rules.
  2. Normalise and taxonomise (generation, phase of flight, severity, TEM or equivalent).
  3. Prioritise (for example risk products of likelihood × severity × training benefit; medians and training-benefit thresholds in the criticality survey method).
  4. Map to competencies that are critical for managing the threat/error or that are the root cause when management fails.
  5. Assign frequency (A / B / C in matrix language) and example scenario elements.
  6. Deliver in EVAL / MT / SBT; capture new training metrics.
  7. Close the loop with operations data: did the risk move? did grades move? adjust without casual drift from the baseline evidence.

No single system is enough. FDA without observation invents causes; observation without FDA misjudges frequency; training grades without line data optimise for the simulator. Doc 9995 expects integrated analysis and liaison between safety and training departments.

What is not sufficient evidence (for curriculum change)

  • One instructor's preferred scenario because "it always teaches well"
  • A viral accident film without rate, generation, and trainability analysis
  • Mandating a topic only because another operator does it
  • Changing frequency because of a single cycle of low grades without checking device, briefing, or rater effects
  • Using jeopardy check data alone as a picture of line competence (practice-then-check confounds baseline)

Instructor use

  • Defend the lesson plan with the right layer of evidence: generation matrix for baseline topics; operator FDA/LOSA/SMS for local colour and enhanced content.
  • Treat your own session grades as training-system evidence: record OBs and competency grades carefully; garbage grades poison the feedback loop.
  • In facilitation, help crews connect session events to line threats they recognise; that is how abstract "evidence-based" becomes operationally real.
  • When you disagree with a topic, escalate through the design/SMS path with data, not by silently dropping matrix items on the instructor operator station (IOS).

Connections

Sources