Calibrated Peer Review (CPR) in CS 20: Discrete Mathematics for Computer Science

Discussion Facilitators: Salma Abu Ayyash, Robert Haussman, PhD; Olivia Miller, PhD

Overview: 
Our upcoming discussion will focus on research conducted by LInc on Calibrated Peer Review (CPR), a pedagogical approach designed to enhance students' critical thinking and fluency in a variety of skills, such as proof writing. through structured peer assessment and self-reflection. In Spring 2023, LInc Pedagogical Fellows Olivia Miller, Robert Haussman, and Salma Abu Ayyash collaborated with LInc Faculty Fellows Rebecca Nesson and Michael Mitzenmacher to implement CPR in CS 20: Discrete Mathematics for Computer Science. Their goal was to examine whether engaging in CPR on homework problems improved students’ performance on isomorphic proof-writing questions in exams. At the heart of this study is the main research question: 

• Does engagement in a Calibrated Peer Review process for solving proofs cause higher scores on proof exam questions in an introductory college discrete mathematics course at Harvard University? 

To investigate this, the team conducted a randomized controlled trial (RCT) using an A/B testing approach. One group of students engaged in the CPR process while another solved homework problems without it. Both groups then took the same exam, and later in the semester, the groups were switched to compare outcomes.  
 

Agenda: 

• Overview: CPR benefits and assignment structure 
• The Experiment: Goals, research design, Canvas implementation, and key findings 
• Reflection: How CPR or similar peer review strategies might be applied in different courses 
• (If time permits) Tools available in LInc to support pedagogical design and assessment 

To Do 
For context, here is a short introduction to peer review: Dawson, J., & Mitchell, I. (2014, November 18). Introduction to student peer review. Computer Science Department and CS-CWSEI, University of British Columbia. Retrieved from https://cwsei.ubc.ca/files/resources/instructor/Student-Peer-Review_Intro.pdfLinks to an external site.

The discussion will also consider broader experiences with peer review in teaching and how similar strategies might be adapted across different courses.

[1] Russell, A. A. (2001). Calibrated Peer Review™: A Writing and Critical-Thinking Instructional Tool. The American Biology Teacher, 63(7), 474–480.

Learning Goals 

1. Identify the assumptions behind Calibrated Peer Review (CPR) as a pedagogical approach and assess how structured peer assessment influences student learning in proof-writing and problem-solving.
2. Analyze the design of the CPR study in CS 20, including the rationale for using a randomized controlled trial (RCT), and evaluate the methodological choices made in assessing its effectiveness.
3. Interpret the findings from the study and discuss their broader implications: Does CPR primarily enhance student performance, conceptual understanding, or metacognitive skills? What are the trade-offs?
4. Compare CPR with other peer learning strategies and examine how its cognitive and social dimensions align with or differ from other approaches in STEM education.
5. Reflect on the faculty role in designing, facilitating, and assessing peer review activities and determine how CPR shifts the role of the instructor, identifying potential benefits and challenges.
6. Design an adaptation of CPR or a similar structured peer assessment model for their own teaching context, considering factors such as student preparation, disciplinary differences, and learning objectives.

7. Identify potential barriers to student engagement in peer assessment and develop strategies to mitigate common pitfalls, such as uneven feedback quality, student resistance, or logistical constraints.
   
    

   ---

   
    

   Takeaways 

   1. CPR Supports Student Learning 
      • Structured peer review encourages students to engage more deeply with the material, improving critical thinking, self-assessment, and conceptual understanding.
   2. Research Findings Suggest Positive Impact on Exam Performance
      • The randomized controlled trial (RCT) found that students who engaged in CPR on homework problems demonstrated higher scores on isomorphic proof-writing questions in exams, suggesting improved transfer of learning.
   3. Peer Review Quality Can Be Calibrated for Consistency
      • The CPR process scaffolds peer review by incorporating training and calibration exercises, ensuring that students learn to provide meaningful and accurate feedback.
   4. Student Engagement and Buy-In Are Key to Success
      • Clear communication of the purpose, benefits, and expectations of CPR helps mitigate student resistance and fosters greater participation in the peer review process.
   5. CPR Can Be Adapted for Different STEM Disciplines
      • While CPR was used for proof-writing in CS 20, structured peer assessment can be applied across disciplines, particularly where students benefit from iterative improvement and reflection on their reasoning.
   6. Faculty Role Shifts from Evaluator to Facilitator
      • Implementing CPR repositions faculty as designers of peer learning experiences, emphasizing guidance, structure, and the development of students' evaluative skills rather than direct grading.
   7. Challenges Exist, but Strategies Can Improve Implementation
      • Issues such as inconsistent feedback, student hesitancy, and logistical constraints can be mitigated through structured rubrics, reflection prompts, and technological support.
   8. Further Exploration Needed for Long-Term Impact
      • While the study shows short-term learning gains, additional research could explore long-term retention, deeper conceptual gains, and the effectiveness of CPR in different course contexts.

   ---