Usable Measures of Teacher Understanding: Exploring Diagnostic Models and Topic Analysis as Tools for Assessing Proportional Reasoning for Teaching

This project seeks to measure the kinds of knowledge developed in professional development (PD) programs that have been shown to matter for teachers' classroom practices and their students' learning. The project aims to develop an assessment that identifies patterns in the teachers' learning in a way that helps drive subsequent PD.The overall goal of this project is to pursue a potentially transformative approach to the assessment of teacher proportional knowledge by developing a measure that is well aligned with the content and skills taught in various PD programs.

Full Description

One of the great challenges related to teachers and their knowledge is measuring their learning in ways that are both formative and meaningful in relation to their likely impact on students. This challenge persists despite efforts to define the knowledge teachers should have and despite previous innovative efforts to create good measures. This project tackles the challenge by specifically aiming to measure the kinds of knowledge developed in professional development (PD) programs that has been shown to matter for teachers' classroom practices and their students' learning. The project aims to develop an assessment that identifies patterns in the teachers' learning in a way that helps drive subsequent professional development.

The overall goal of this project is to pursue a potentially transformative approach to the assessment of teacher proportional knowledge by developing a measure that is well aligned with the content and skills taught in various PD programs. This instrument will be based on a new approach that builds on emerging psychometric models. Specifically, diagnostic classification models (DCMs) will be utilized to diagnose teachers' learning during a PD program as well as employed to identify the progression in teachers' learning.  Statistical topic models (STMs) will be used to look for patterns of understanding that emerge from open-ended responses and provide natural-language insight into teachers' reasoning. A final version of the assessment will be constructed for a national sample based on the results from the predictive validity stage, and this version will be tested with teachers who participate in various types of PD programs targeting proportional reasoning. This project has broad implications for the creation of assessments and for teacher education. It will provide insights about whether there is a clear learning progression for teachers. While much work has been done with students' learning progression, much less is known about how teachers learn. Another implication is that the STM approach allows machine scoring of natural language in a way that highlights strengths and weaknesses in reasoning rather than simply returning a score. For formative use, this is information that is more helpful as it highlights areas for further instruction. A third implication is that DCMs will allow to assess teacher knowledge at a finer-grained understanding than is typically available, thus allowing for careful refinement of PD as well as a tool for showing overall growth in PD. A fourth implication is that a more systematic approach will be followed to capture the kinds of knowledge teachers need. Assessments developed using DCMs and STMs have the potential to serve as models for developing further instruments in other STEM content areas. Such assessments have the potential to not only help identify successful PD programs, but also to provide PD providers with rich data from which they can make instructional decisions.

Project Materials

Title Type Post date Sort ascending
Middle School Mathematics Teachers’ Proportional Reasoning and Its Relation to Their Content and Pedagogical Content Knowledge Resource
A Model Comparison Approach to Posterior Predictive Model Checks in Bayesian Confirmatory Factor Analysis Resource
A Gibbs Sampling Algorithm with Monotonicity Constraints for Diagnostic Classification Models Resource
The Impact of Sample Size and Various Other Factors on Estimation of Dichotomous Mixture IRT Models Resource
Investigating Teachers’ Understanding Through Topic Modeling: A Promising Approach to Studying Teachers’ Knowledge Resource
Estimation of Multidimensional Item Response Theory Models with Correlated Latent Variables Using Variational Autoencoders Resource
Modification Indices for Diagnostic Classification Models Resource
Unpacking Response Process Issues Encountered When Developing a Mathematics Teachers’ Pedagogical Content Knowledge (PCK) Assessment Resource
Integrating a Statistical Topic Model and a Diagnostic Classification Model for Analyzing Items in a Mixed Format Assessment Resource
On the Alignment of Teachers’ Mathematical Content Knowledge Assessments with the Common Core State Standards Resource
Revisiting Purpose and Conceptualisation in the Design of Assessments of Mathematics Teachers’ Knowledge Resource