IME 2015 Conference Keck School of Medine at USC Oral Presenation,

The following is the longer version of the paper submitted to IME 2015 (conference syllabus).  Clicker here to view the oral presentation (googledocs) or here for a youtube version and clicker here to view the Q&A.

The Effect of Audience Response Systems on Metacognition in Graduate Students: A Two-Year Mixed Methods Study   

Melanie Brady, Ed.D., University of Southern California, Rossier School of Education

Jane Rosenthal, Ed.D., Assistant Dean, School of Applied Life Sciences, Keck Graduate Institute

Christopher P Forest, MSHS, DFAAPA, PA-C University of Southern California, Keck School of Medicine, Division of Physician Assistant Studies

Problem Statement

Use of educational technology to engage learners continues to grow at a rapid pace. Studies of effectiveness of clicker use find that when clickers are utilized with research-based instructional strategies the learning experience in large lectures is enhanced². In a study with undergraduates (n=198) metacognitive self-regulation seems to improve when clickers were utilized in this manner. Comparison (low technology) and experimental (clickers) methods each demonstrated significance influence on learner metacognition, clickers with the summer cohort and the comparison method with the fall. However, when performance outcomes and qualitative data were factored in, clickers demonstrated a high degree of significance (p> .01). This current mixed methods study of audience response systems and metacognition investigates whether the experience for graduate health science candidates (e.g., 1^st year Physician Assistant candidates 2013 and 2014) is consistent with the undergraduate experience and to what degree between graduate cohorts (2013-2014). 

Rationale

The importance of these investigations lies in the growing body of research that self-regulated and metacognitively aware learners tend to have improved outcomes and that metacognition and self-regulation are teachable. Research suggests when clickers are utilized with instructional strategies (e.g., questioning and peer instruction), performance outcomes increase and metacognition may be affected. Metacognition, the regulation of cognition and self-knowledge, is an essential component in the learning process in order to become a self-regulated learner. This mixed methods comparative study examines the extent to which high-tech devices (clickers) and low-tech devices (paddles) affect learner metacognition. This study extends our 2013 mixed methods examination of clickers and metacognition conducted with 1^st year Physician Assistant candidates and further comparison between clickers and paddles.,^1,2  If the data generated by the two years is not robust enough, a third cohort is proposed for fall 2015 to increase the strength of results and potential for generalization. 

Hypothesis

The response device that more effectively influenced metacognition would be  associated with higher performance outcomes. Based on the results of the undergraduate study we predicted that use of clickers would lead to less social comparison which could enable more productive learning; use of paddles would lead to more social comparisons that could interfere with the learning process.

Methods 

Data were collected from 54 graduate candidates in 2013 and 51graduate students in 2014 during a behavioral sciences course.  Clickers were used during weeks 1-5 of the course and a low technology response system (paddles) during weeks 8-12. Paddles are handmade signs are held up to indicate preferred answers (A-E); this method was selected for comparison as an analogous system to clickers in that it provides a quick visual check of student responses that allows participants to be polled once as opposed to raising hands several times for a multiple choice question. This comparative, mixed-methods study employs several measurement instruments and a pre- and post-test design to compare the two response systems. The components of metacognition of interest in this study are Metacognitive Judgments and Monitoring and Metacognitive control and self-regulation. 

Quantitative instrument. In the first week of the course, pre-test data and demographic information were collected. Questions from the Motivated Strategies for Learning Questionnaire (MSLQ)³ served as the pre-post-test instrument. Two instruments that measure feedback systems and metacognition¹ were administered at week 5 (experimental/clickers) and at week 10 (comparison/paddles) Metacognition in Lecture Survey² (2013-clickers, α = .910; 2013-paddles α = .935; 2014-clickers α =  .806; 2014-paddles α =  .888) measures metacognitive self-regulation experienced by learners in lecture through changes in learning behavior inside or outside of lecture and Metacognition Attribution to Response Device Scale (2013-clickers, α = .723; 2013-paddles α = .704; 2014-clickers α =  .681; 2014-paddles α =  .827) measures the level of metacognitive influence that learners believe to experience as a direct result of the use of the polling method. Mean quiz scores from the first 5-week session served as the measure of performance outcomes for clicker use, and the mean participation scores for weeks 6-10, for the comparison treatment (paddles). 

Qualitative instrumentation. Participants completed an on-line qualitative survey using Qualtrics^© that consisted of open-ended questions to elicit reflections about response device use.   Interviews were conducted using purposeful sampling using the following criteria: 1) low mean scores indicating little metacognitive influence attributed to clicker/paddle use; 2) mean scores in the median range indicating a moderate-to-neutral influence; and 3) high mean scores indicating a strong influence. 

Results

High comfort level and prior use of audience response systems were reported by 60% of participants from the 2013 cohort and 100% of participants from the fall 2014 cohort on the initial survey.  Two tailed t-test for dependent means were conducted to examine between groups differences in metacognitive self-regulation, the pre-post-post –MSLQ), and performance outcomes (e.g., in lecture clicker quizzes. Significance was not found between the Graduate Health Science 2013 and 2014 cohorts on the pre-post-post-test administration, but significance was found with metacognition instrumentation. Lack of significance between pre-post-post-test between groups indicates group similarities which increases the potential strength of results and ability to generalize. The first post-test administration followed use of clickers, the second post-MSLQ following the comparison method (low-technology). This indicates that learners in both cohorts gauged individual metacognitive self-regulation similarly at the start of the course, following the treatment method (clickers) and following the comparison (low-technology polling). 


Differences were found on formative performance assessments between the 2013 cohort (M = 87.38, SD = 5.86 and the 2014 cohort: (M=75.41, SD=6.27) demonstrating differences in metacognition during lecture of the two groups (t₍₅₂₎ = 10.263, p = .001). Significance was demonstrated between groups for both instruments measuring influence of metacognition during lecture, and the attribution of metacognition to the response device (t₍₅₂₎ = 4.84, p = .001; t₍₄₈₎ = 5.83, p = .001). Qualitative analysis results were similar between groups. Clickers were perceived as a more effective way to monitor learning and the low technology method resulted in conformity and reduced pressure to prepare for lectures. Differences occurred in that a small portion of the 2014 cohort suggested that the low technology system created opportunities for discussion and learning and was enjoyable while the majority of peers in the same cohort did not share this opinion, nor did the 2013 cohort. Reports of positive learning experiences with paddle use tended to accompany indications of relief at the ability to rely on group lieu of individual preparation when schedules were busy

Lessons Learned

Quantitative results indicate that clickers influence learner metacognition more so than low technology response devices, suggesting that conceptual understanding may be clarified through use of clicker items and interactive teaching strategies (e.g., questioning, and peer instruction) leading to improved formative feedback for enhanced learning. Both cohorts indicated that clickers strongly influenced peer comparisons, consistently positively influencing the learning process. Clickers can improve accuracy of metacognitive judgments and influence strategies utilized for learning outside of lecture. Qualitative results suggest that graduate learners are more confident in strategies utilized for note-taking in lecture and for preparation for lecture. Several learners reported changing answers based on peer responses, and feeling less pressure to prepare for lecture when the low technology system was utilized. Focus of clickers on independent learning improved ability to monitor learning and results indicate that learners are more apt to prepare for lecture with individual response component. 

Selected References

1. Brady M, Rosenthal J, Forest C. Metacognition and Performance Outcomes: An Examination of the Influence of Clickers with 1^st Year Graduate; in progress.
2. Brady ML, Seli H, Rosenthal J. Metacognition and the influence of polling
systems: How do clickers compare with low technology systems? Educ Technol Res Dev. 2013;61:885-902.
3. Pintrich, PR, Smith DAR, Garcia T, McKeachie WJ. Reliability and predictive validity of the motivated strategies for learning questionnarie (MSLQ). Educ Psychol Meas. 1993;53(3):801-813.