Evaluations & feedback
How do you know what you did was any good?
The easiest way is to ask. But don't ask if you're not prepared to listen to what people have to say.
The focus of this page is more about the intervention than feedback on debriefing technique, that is covered elsewhere.
In this day and age its hard to interact with any product or service without being asked for 'feedback', and medical education & simulation are no different. The easiest way to gain feedback is to ask for it, either verbally, or via paper/technology-based forms. But what questions should you ask?
This is where knowing about Intervention Effectiveness can come in handy. Considering Kirkpatrick's evaluation hierarchy the base level is simply whether people liked it or not. Following on from this comes whether people felt they learned something or not, with definitive evidence being a change in results.
An example questionnaire is available here.
Many evaluation forms make use of Likert Scales. Likert scales are often represented on scales of 1-5, or 1-10, but could equally be seen as a visual-analogue scale of )-: to :-)
While Likert scales are useful tools to convert qualitative data around thoughts and feelings to a more finite quantitative score, they are limited by the fact that rich data doesn't translate well into numerical data. For example, my level of satisfaction is different to yours, for a given experience at a variable day, or time of day.
Collecting the 'right' data, in the 'right' way, is essential to get the 'right' feedback. For example, does time to complete a cannula convey more benefit to how efficiently an individual can anticipate and manage complications? One should therefore carefully consider whether one needs quantitative data for statistical analysis, or qualitative data for a richer descriptive analysis, or whether a combined approach is required - these concepts are beyond what can be reviewed on this page and is discussed in greater depth elsewhere.
Stuck on what evaluation method to use? Use the flowchart in this article to decide.
More learning resources are available below:
A handy and exhaustive pool of resources can be found here at Evaluating Healthcare Simulation.
References:
Evaluating Healthcare Simulation - https://sites.google.com/view/evaluatinghealthcaresimulation
https://www.healthysimulation.com/3907/how-to-evaluate-effectiveness-of-simulation/
Evaluation of Medical Simulations - https://pubmed.ncbi.nlm.nih.gov/24084307/
Evaluating the role of simulation in healthcare innovation: recommendations of the Simnovate Medical Technologies Domain Group - https://stel.bmj.com/content/bmjstel/3/Suppl_1/S8.full.pdf
Evaluation in medical education: A topical review of target parameters, data collection tools and confounding factors - https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4576315/pdf/GMS-13-15.pdf
Evaluation in medical education - https://www.monash.edu/__data/assets/pdf_file/0007/935692/evaluation-in-medical-education.pdf
Simulation and Feedback in Health Education: A Mixed Methods Study Comparing Three Simulation Modalities - https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6025072/pdf/pharmacy-06-00041.pdf
Analyzing and Interpreting Data From Likert-Type Scales - https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3886444/pdf/i1949-8357-5-4-541.pdf
Use and Misuse of the Likert Item Responses and Other Ordinal Measures - https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4833473/pdf/ijes_08_03_297.pdf
Assessment Methods in Medical Education - https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3068728/pdf/ijhs-2-2-0003.pdf
https://www.tandfonline.com/doi/pdf/10.1080/20786204.2006.10873311
Quantitative and Qualitative Methods in Medical Education Research: AMEE Guide No 90: Part I - https://pubmed.ncbi.nlm.nih.gov/24846122/
Qualitative research essentials for medical education - https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6301871/pdf/SMJ-59-622.pdf
https://www.sesam-web.org/media/documents/sim-dictionary1.pdf
https://www.coursera.org/lecture/clinicalsimulations/evaluating-the-simulation-UBKcY