Longitudinal Investigation of Medical Student Perceptions of a Video‐based Guided Study Resource used to Facilitate an Eight Week Module in Medicine

Andrew Morris, School of Medicine, Keele University Steve Clipstone, School of Medicine, Keele University William Stockton, School of Medicine, Keele University Russell M. Crawford, School of Medicine, Keele University

Contact: r.crawford@keele.ac.uk

Abstract:

We use an action research approach to evaluate three successive cohorts of medical student perceptions of using a multi‐faceted video‐based guided study resource which provides academic (skills‐based), social (motivation‐based) and professional (clinical‐based) interventions aimed at facilitating and enriching learning across an eight week module in the second year of the MBChB medical degree. Our findings show that whilst students value these video resources both as a revision tool and an aid to learning during the semester, they have specific critiques about several areas which would improve the project. We interpret our data to evidence a tangible beneficial argument for the use video‐based learning objects to support student learning that is reliably reaffirmed by our longitudinal data.

Keywords: Motivation, Guided Study, Medical Education, Web 2.0

Introduction:

Using video technology to aid medical education is an idea that has been around for a long time and there are now a range of video formats available to learners; from professional commercial products to low

budget teaching aids (Lewis et al., 2000). One of the main benefits of videos as an aid to learning is that they are reusable and can ‘incorporate text, graphics, animations, audio, and video to support and enhance learning’ (Ruiz et al., 2006). We developed ‘Keele Basic Bites’ (KBB), a series of guided study video resources intended to facilitate medical student learning. KBB videos provide both local context and rigorously quality controlled resources delivered by professional teachers using an informal and multi‐faceted delivery style and provides a range of resources addressing clinical skills, laboratory skills, student motivation, pharmacological mechanisms and expert interviews which were designed to run parallel to the core medical curriculum and facilitate student learning. The full range of resources that were used in this study are available to view at www.keelebasicbites.com.

The KBB project philosophy is Context – Demonstrate – Innovate because medical students constantly have to perform tasks (demonstrate) in differing clinical and scientific settings (context) under often stressful circumstances, thus requiring innovative methods in teaching them how to perform these tasks. In the 1950`s, D.L. Kirkpatrick proposed an enduring model for evaluating educational projects that has since been referred to as ‘Kirkpatrick’s Triangle’. Whilst there is literature exploring the limitations of this model of evaluation, it is still considered a valid and useful way to benchmark learning gain (Bates, 2004). Principally, Kirkpatrick level 1 (satisfaction) is centered on measurement of user satisfaction with higher levels on the triangle being concerned with measuring long‐term changes in user behavior as a result of an intervention and we with our work here, we aimed to reach Kirkpatrick level 3 (impact), to determine if KBB had an impact on changing user behaviour.

Results:

The video resources were filmed and edited in‐house (Adobe PremierePro CS4, AVI format converted to WMV / MPEG4 for website publication). Three consecutive cohorts of second year medical students, from 2011‐12 to 2013‐14, participated in this study. All were given access to the full range of KBB resources aligned with their weekly problem‐based learning (PBL) cases, laboratory and clinical teaching sessions over an eight week module running from January to March each year. Students were informed weekly of KBB resources applicable to that week’s learning, both by their PBL tutors and via an open KBB Facebook® account. They were provided with an average of three or more aligned resources, each week, for eight weeks.

Student attitude to the resources was assessed at the end of the module each year by an anonymised, voluntary questionnaire on a 5‐point Likert scale, running from 1 (strongly agree) to 5 (strongly disagree). The questions are shown in Table 1.

Data were collated for each year and the difference between cohort responses to each question defined by a Kruskal‐Wallis test, using a Dunn’s post‐test. Free text comments for each year were analysed visually for positive, negative or neutral tone. Themes from the comments from all three years were identified by visual assessment of the data and no comments were excluded from this analysis.


What the intervention does:

The KBB concept builds on a self‐directed learning pedagogy principal with the intention to integrate medical teaching and curriculum design (Chen, 2008). As an allegorical example, emergency medical service providers were shown a video on assessing head injury by the Glasgow Coma Scale and concluded that post‐video performance was better than performance without the video (Lane et al., 2002). Additionally, our approach in this study was further validated by an investigation showing that video learning resources enhanced performance in neurological testing (Raijmakers et al., 1991). Whilst individual KBB resources have been available to these students during their first year of study, this research project marks the first time we have attempted to facilitate an entire eight week module using a sign‐posted blend of KBB resources. We believe there is an important place for video‐based learning support in facilitating core student learning and enhancing student perceptions and learning outcomes across an entire module. For this study, we collected three different types of data (questionnaire, free‐ text comments and KBB website traffic) to investigate how three successive cohorts of our students use, perceive and access KBB resources and to what extent they find them useful in their learning and revision. Our research was carried out within an ethical framework where all participants were given copies of the question instrument (Table 1) over twenty‐four hours in advance of the session as well as provided information about the nature of the research in advance of filling out the instrument. Since all question instruments were anonymous, participants we not obliged to fill them out at all and this was also made clear in the session.

Website usage:

Anonymised website data, presented in Figure 1, shows a representative spread of daily KBB website hits from students accessing KBB resources, with students returning to these resources throughout the eight week run time of the module. As can be seen from these data, website usage rarely falls below twenty page accesses per day and that includes students accessing resources at the weekend for either preparation or revision purposes.

Figure 2 shows the question responses revealing that the students from the 2012 cohort were more positive than any other year and that the cohort from 2014 held a significantly (p<0.001) more negative opinion than their colleagues from 2012 on all questions. We also observed that the 2013 and 2012 cohorts shared very similar opinions across the question instrument on all but Q1 (P>0.05). Overall, it should be noted that whilst the 2014 cohort have a ‘more negative’ opinion of the project than the previous cohorts, they were not actually negative about the KBB resources on the whole, as they still showed a majority located in the positive categories of the Likert scale. The free‐text comments will now be used to lend context to these trends.

Thematic Analysis:

Our rationale for facilitating an entire module using KBB video objects was to increase the facilitation of our student learning in the laboratory whilst also providing additional aids for their learning. We conducted a thematic analysis of all free‐text comments collected from all three years student question instrument returns without excluding any data. Three recurrent longitudinal themes emerged from our data:


  • Coverage of the videos (Figure 3A)
  • Reception of the videos (Figure 3B)
  • Technical comments about the videos (Figure 3C)

Below is a selection of representative positive comments to contextualise the data presented in Figure 3 and to offer examples of the types of recurrent comments our students were making. The negative

comments will be discussed in the next section under the three themes due to specific the nature of these comments:

‘This (KBB) is a great resource’; ‘It’s fun and interesting’;; ‘Like KBB, more useful as a revision tool’; ‘The KBB videos are concise and easy to access and understand’; ‘Really helpful’; ‘Very useful’; ‘Very good, especially in time before revision’; ‘Drug Bites are very useful for revision and also give a small starting point for answers to intended learning outcomes (from PBL)’; ‘Skills bites very useful for OSSE (practical exam) revision’; ‘Excellent, really helps with learning’; ‘Really useful revision tools’; ‘Really useful, especially at exam time’; ‘Drug Bites are very good, motivation bites are also good’; ‘Great when revising for exams’; ‘Brilliant revision tool, especially for OSSE`s (practical exams)’; ‘Motivation Bites are also very good’; ‘Excellent work and much appreciated’; ‘Really good for a refresher after the lab classes’; ‘They`re brilliant!’; ‘Love the motivational bites, more please’.

Our data showed overwhelmingly positive representative comments about the KBB resources, indicating that students both value and appreciate them to facilitate not only their weekly learning as part of the module, but a large portion of the comments indicate that students recognise them as valuable revision resources as well.

Discussion:

The KBB project is, to paraphrase, ‘a dream given form’; a project that provides staff and students a consolidated mechanism and place to go for video‐based guided study objects constructively aligned to core curriculum content (Morris et al, 2013 and 2014). Though laudable as a goal, the execution of such an ambitious goal has proven both challenging and rewarding to develop and this action research was intended to evaluate the progress and learner perception of the KBB project to date.

The reason we adopted a longitudinal action research approach to our evaluation was to attempt to control in some small way for inter‐cohort variability of positive perception in order to get a deeper, more reflective sense of how our students perceived the KBB resource we provide.

From the website hit profile of student use (Figure 1), we can see that the resources are consistently accessed throughout the run of the eight week module and that our dissemination strategy was largely successful in drawing student attention to these resources to facilitate their learning each week. Website usage data also shows spiking in weeks where one or more of the resources are used to support learning (in the laboratory or as part of PBL) and even shows reduced but continued use over the weekend as our learners look to consolidate their learning from the week (Guo et al, 2014). One limitation of our dissemination strategy was perhaps an over‐reliance on social media to keep KBB ‘visible’ to our learners, as it assumed that all students would/could access social media, which may not necessarily be the case (Dabbagh, 2012). The return rates associated with our questionnaire (never less than 80% of each total year cohort) would suggest the vast majority of our learners are aware of and access the project resources but our future work will look to include ancillary dissemination mechanism(s) to ensure we are not missing or indeed, disadvantaging a small portion of our learners.

Another reason for the longitudinal aspects of our experiment approach was to control for ‘erroneous’ positive perceptions from learners simply by proxy of a novel resource being made available to them (Clark, 1983). We do not feel that this is a confounder in our study principally because we have inter‐ cohort data to draw upon to support our conclusions and additionally, all cohorts of students had access to KBB resources in their first year on the course in some way. As such, we are confident that the levels of engagement we observe in our data are essentially independent of a ‘new resource’ factor and we would therefore interpret the sustained positive impressions and usage of KBB to suggest a synthesis of KBB resources into our learners’ personal preferences over an extended period of time. We would further interpret this sustained use of KBB resources as our work realistically reaching Kirkpatrick level 3 (impact), defined as ‘learners behaviours change as a result of the intervention’ (Bates, 2004).

Longitudinal thematic analysis of all free‐text comments revealed three themes emerging from our data:

Coverage of the videos: Within this theme, we identified four trends (Figure 3A); where students would specifically name a positive aspect of the KBB offer, name a negative aspect of the offer, request more

resources and in the interests of balance, we also included any mention of wanting less KBB resources. Figure 3A shows that whist 2014 saw an overall decrease in number of positive comments (including specifically named positive videos) and a slight increase in negative in comparison with 2012 and 2013 data, we also observe an increase in demand for newer, specifically named videos. Interestingly, over all three years of this study no student has even suggested they would like less. Comments such as ‘Drug bites and skills bites were more useful, Lab skills, very useful’ and ‘Drugs, labs and other skills were extremely helpful but interview bites were not’ demonstrate that students are quite forthright in telling us what they value. The vast majority of students across all three years named both Drug and Skills Bites as being useful, whilst our data also reveals that a number of students each year find the Interview bites less useful.

Reception of the videos: Within this theme, we grouped all comments received into one of three types; positive, negative or neutral (Figure 3B). We observe that the 2012 and 2013 cohorts were broadly similar in all three types, with positive comments such as ‘very useful’ and‘very helpful’ being common across the data. Negative comments did not centre on academic content, but rather in all cases on the style aspects of the resources, with representative comments such as ‘music can be a bit loud’ and‘less light hearted’ being repeatedly observed. The 2014 cohort appeared to be less positive about the resources according to our findings, with comments such as‘no music for the drug videos please’ and‘not all drugs were present’ at least showing a critical engagement with the resources if not always an appreciation of the style. Overall, we observed the lack of comparable positivity from the 2014 cohort to directly correlate with a lack of general awareness or a desire for increased awareness of the resources rather than criticism of the resources themselves, with representative comments such as ‘Didn`t know they were available’ and‘Didn`t think to use it’ being recorded from that years cohort. These observations are in agreement with our own feelings about the limitations of our current dissemination strategy for the KBB project and will be addressed in further action research cycles.

Technical comments about the videos: Because we adopted an action research approach to evaluating the KBB project longitudinally, we were particularly interested in the comments about technical aspects of the KBB project, since the ever‐changing use of technology to aid learning means that we need to continually improve upon our resources to keep the learning current and useful (Lewis et al., 2000). Our data shows that some students wanted improvements in video resolution, which was expected since we started the project using relatively low resolution technology in 2010 when compared with high definition recording technology available in 2014. By far the most significant issue for 2013 and 2014 cohorts was an identified need for increased awareness of the KBB resources, with representative comments such as ‘Would be helpful if we was signposted to the videos at the time they are relevant’ and ‘I haven`t used it for this unit as I didn`t know ‐‐> however if I had known I definitely would have’ being reported.

Taken as a whole, the free‐text comments from across the three cohorts of students reveal that the majority of students use KBB resources principally as a revision tool throughout the module and that each successive cohort is largely positive about the use of the KBB project but that the most recent cohort feel that awareness and scaffolding of the resources needs to be improved to better aid their learning. Our data shows that overall our approach in using KBB to facilitate learning across an entire eight week module has engaged student learning in an innovative manner that works well with their revision goals as well. Our action research approach has helped identify future development opportunities within the KBB project, principally improvements to awareness and dissemination strategy, to better take advantage of this innovative way of supporting student learning.

Although beyond the scope of the current article, in the future we look to link our perceptual data with a quantified and statistically significant learning gain measured by assessment performance to reach Kirkpatrick level 4 (results), defined as an intervention having a measurable impact on results (Bates, 2004).

We feel that one of the strengths of the KBB innovation is its ability to be generalised to almost any other type of student learning experience because its focus is on addressing the needs of the students, grounded in best practice and does not require to be confined to the study of medicine. We are currently developing a number of cross‐faculty and cross‐institute applications, where KBB resources are being generated which are applicable to other schools such as Nursing, Pharmacy and Biosciences and to other UK higher education institutes. Further, our KBB resources are of potential benefit to psychology, sociology and other humanities schools as some of our contributors are themselves, humanities professionals. In this manner, we would hope KBB could prove useful to inform teaching in a number of other arenas and our future plans for the project aim to expand on these opportunities.

References

Bates, R., 2004. A critical analysis of evaluation practice: the Kirkpatrick model and the principle of beneficence. Evaluation and program planning , 27(3), pp.341‐347.

Chen, N.C., 2008. An educational approach to problem‐based learning. The Kaohsiung journal of medical sciences, 24(3), pp.S23‐S30.

Clark, R.E., 1983. Reconsidering research on learning from media. Review of educational research, 53(4), pp.445‐459.

Crawford, R.M., Morris, A., Stockton, W., Clipstone, S., Bashford, L. and Mahon, M., 2013. Longitudinal investigation of medical student perception of a video‐based intervention to facilitate laboratory teaching.Compendium of Effective Practice in Higher Education, Volume 2, pp156‐159

Crawford, R., Morris, A., Stockton, M.W. and Clipstone, M.S., 2014, Development, application and evaluation of Web 2.0 technology to enhance the student experience–Keele Student Bites. Journal of Academic Development and Education, 1:41‐45

Dabbagh, N. and Kitsantas, A., 2012. Personal Learning Environments, social media, and self‐regulated learning: A natural formula for connecting formal and informal learning. The Internet and higher education, 15(1), pp.3‐8.

Guo, P.J., Kim, J. and Rubin, R., 2014, March. How video production affects student engagement: An empirical study of mooc videos. In Proceedings of the first ACM conference on Learning@ scale conference (pp. 41‐50). ACM.

Lane, P.L., Báez, A.A., Brabson, T., Burmeister, D.D. and Kelly, J.J., 2002. Effectiveness of a Glasgow Coma Scale instructional video for EMS providers. Prehospital and disaster medicine, 17(03), pp.142‐146.

Lewis, L., Jones, J. and Haynes, E., 2000. Low‐cost video‐films in the teaching of undergraduate and postgraduate medical students. Journal of telemedicine and telecare, 6 (suppl 2), pp.45‐47.

Raijmakers, P.G.H.M., Cabezas, M.C., Smal, J.A. and Van Gijn, J., 1991. Teaching the plantar reflex. Clinical neurology and neurosurgery, 93(3), pp.201‐204.

Ruiz, J.G., Mintzer, M.J. and Issenberg, S.B., 2006. Learning objects in medical education. Medical teacher, 28(7), pp.599‐605.

Refbacks

  • There are currently no refbacks.


Copyright (c) 2017 Andrew Morris, Steve Clipstone, William Stockton, Russell M. Crawford