2020 PiR: Observation Validity Working Group Meeting

Collaborations

About

This gathering of mathematics and STEM education researchers was designed to bring together experts in classroom observation protocol development for use in school settings to evaluate fidelity of implementation of interventions/programs and to gather formative input to improve practices.

Over the course of the two-day meeting, the group shared projects and discussed the following topics:

highlights and challenges of measurements in individual projects, identifying similarities and differences,
features of measures and measurement development and pros and cons of their usage across users and time,
validity evidence and the essential claims we put forward about it,
how measures support instructional improvement and what that means for researchers developing and collecting data on the constructs being measured.

Questions based on attendees’ experiences and projects.

Who is our audience? Researchers? Administrators?
Does validity in measurement matter if reliability is in question?
How do you identify good instructional decisions when a single tool doesn’t capture everything?
How does a team define rater agreement? Rater consistency?
How do you differentiate rater agreement vs. rater reliability? What about rater drift?
What opportunities exist to improve rater reliability?
How does a team operationalize the construct being measured?
Is the observation tool the intervention? Does the tool measure instructional quality?
Does the observation tool become too proprietary by the developer to be used effectively?

Considerations around the uses of measures emerged and prompted the following discussion points:

collecting and analyzing observational data (including multiple time points),
using existing measures and development/how-to-guide or approaches to avoid re-inventing the wheel,
using measures across users,
using measures to provide feedback to teachers,
interpreting dis-confirming evidence,
determining relative importance of various sources of data,
embedding technology within a measure, and
ensuring relevant and appropriate use of measures.

Date

March 2-3, 2020

Hosts:

Dr. Leanne Ketterlin Geller, Professor and Texas Instruments Chair in Education; Director, Research in Mathematics Education, Simmons School of Education and Human Development, 澳门六合彩预测
Dr. Elizabeth Adams, STEM Evaluation Researcher at Research in Mathematics Education, Simmons School of Education and Human Development, 澳门六合彩预测

About our Experts

Dr. Courtney Bell, Director, ETS/WCR at Wisconsin-Madison
Dr. Julie Cohen, Assistant Professor of Curriculum, Instruction, and Special Education, University of Virginia
Carlin Conner, Research Assistant, Education Consultant, Simmons School of Education and Human Development, 澳门六合彩预测
Dr. Jillian Conry, Research Coordinator and Instructor, Simmons School of Education and Human Development, 澳门六合彩预测
Dr. Angela Crawford, Postdoctoral Researcher, Research Project Coordinator, Boise State
Dr. Richard Duschl, Executive Director, Caruth Institute for Engineering Education, 澳门六合彩预测
Dr. Evelyn Johnson, Scientific Director, Professor, Department of Special Education, CEO at Lee Pesky Learning Center, Boise State University
Dr. Nathan Jones, Associate Professor, Program Director for RS Courses, Boston University
Erica Mason, Graduate Research Assistant, Department of Special Education, University of Missouri
Karen Pierce, Research Assistant, Research in Mathematics Education, Simmons School of Education and Human Development, 澳门六合彩预测
Robyn Pinilla, Research Assistant, Research in Mathematics Education, Simmons School of Education and Human Development, 澳门六合彩预测
Tiffini Pruitt-Britton, Graduate Research Assistant, Research in Mathematics Education Simmons School of Education and Human Development, 澳门六合彩预测
Dr. Anthony Sparks, Research Coordinator, Research in Mathematics Education, Simmons School of Education and Human Development, 澳门六合彩预测
Dr. Paige Ware, Associate Dean and Professor, Department of Teaching and Learning, Simmons School of Education and Human Development, 澳门六合彩预测
Ann Marie Wernick, Graduate Student, 澳门六合彩预测
Dr. Jeanna Wieselmann, Research Assistant Professor, Caruth Institute for Engineering Education, 澳门六合彩预测
Dr. Annie Wilhelm, Associate Professor of Mathematics Education, Department of Teaching and Learning, Simmons School of Education and Human Development, 澳门六合彩预测
Dr. Jonee Wilson, Assistant Professor, Mathematics Education, NC State
Sumei Wu, Research Assistant, Simmons School of Education and Human Development, 澳门六合彩预测
Dr. Paul Yovanoff, Professor, Department of Teaching & Learning, Simmons School of Education and Human Development, 澳门六合彩预测

澳门六合彩预测