The MIST project investigates how changes in the school and district settings in which mathematics teachers work influence their instructional practices, students' learning opportunities, and student achievement (ESI-0554535). The project is also developing instruments and methods for analyzing data to document the institutional setting of mathematics teaching that are specific to equity and access for all middle school students to high quality mathematics instruction (DRL-0830029).
Q&A with MIST
The MIST project team answered a few of our questions about their project background, the instruments they're developing, what they're learning, and how their work is influenced by local, state, and national contexts. Note: This is the abridged version of our interview. Read the full article here.
Where did the idea for this project come from?
The overall goal of the project is to understand what it takes to support teachers in improving the quality of their mathematics instruction on a large scale. To this end, we are attempting to develop an empirically grounded theory of action that can inform instructional improvement efforts at the level of large urban districts. The idea for this project emerged in the course of a prior teacher development project that involved a five-year collaboration with groups of middle-school mathematics teachers in two urban districts (REC-0231037: Developing Articulated Models for Supporting and Sustaining Teacher Development Efforts in the Context of Schooling, PIs Paul Cobb and Kay McClain). In preparing for the prior project, we knew both from experience and from the research literature that teachers’ instructional practices are profoundly influenced by the school and district settings in which they work. We therefore planned to document the settings in which the collaborating teachers worked, and attempted to find a researcher in either educational policy or educational leadership who was willing to work with us to develop analyses that would inform a teacher development effort while it was still in progress. However, it soon became apparent that researchers in these fields typically conduct observational studies in which they investigate others' efforts to support instructional improvement. As one prominent leadership researcher put it, providing feedback about school and district settings in order to inform work with teachers involves "messing with the intervention."
The goal of the theory of action for instructional improvement that we are developing in the current project is to inform the development of school and district settings that support teachers’ ongoing improvement of their instructional practices. Furthermore, and in contrast to the prior project, this work involves a genuine collaboration between mathematics educators and policy researchers. As part of this work, we give each of the four collaborating districts detailed feedback about how their improvement policies are actually playing out in schools and classrooms, and make actionable recommendations about how those policies might be adjusted to make them more effective. In doing so, we are attempting to do for district leaders what we had hoped others would do for us in the prior project, but at the level of district-wide improvement rather than a more modest collaboration with groups of teachers.
What are some examples of conjectures that you’ve made about policies and supports needed to implement ambitious instructional practices at the scale of a large, urban district? Based on what you’ve learned thus far, what are your current hypotheses and conjectures?
As part of our preparation for the project, we developed a set of research-based hypotheses and conjectures about the supports necessary for district-wide improvements in the quality of mathematics instruction. These hypotheses and conjectures included shared instructional vision, teacher networks, accountability relations and relations of assistance between instructional leaders and teachers, and relations among central district office units.
We have completed three annual rounds of data collection, analysis, and feedback, and now consider that our starting hypotheses and conjectures were relatively abstract. We currently frame our hypotheses and conjectures in terms of concrete, potentially learnable practices for members of specific role groups (e.g., teachers, mathematics coaches, school leaders).
Our current theory of action for instructional improvement at scale comprises the following five elements:
- a coherent instructional system for supporting teachers’ development of ambitious teaching practices
- teacher networks
- mathematics coaching
- school instructional leadership
- district instructional leadership
Find a detailed description of each element in the full article.
We contend that all five components of the proposed theory of action are necessary for large-scale instructional improvement; the prospects for achieving and sustaining instructional improvement diminish significantly if any one of the components is neglected. For example, we would question an improvement strategy that focuses on high-quality curriculum materials, teacher professional development, and mathematics coaching but does not attend to school leaders’ development as instructional leaders. Such a strategy is suspect because school leaders are unlikely to either press teachers to develop the intended practices or support coaches’ work with teachers.
The MIST website lists several instruments that you have developed in order to do this work. Can you tell us more about these instruments (e.g., the measures of reliability and validity, use and audience information)?
Interviews
We conduct two types of interviews each year. In the early fall, we interview key leaders in each of the collaborating districts (e.g., Chief Academic Officer, head of Curriculum and Instruction, head of Mathematics Department, head of Office of Leadership, head of Office of English Language Learners) to find out the district’s theory of action for improving middle-grades mathematics instruction for the academic year (i.e., the intended design). In January, we interview teachers, coaches, principals, and district leaders to find out how the theory of action is playing out in schools and classrooms. Questions about networks (teacher, coach, principal) were adapted from interview protocols provided by Cynthia Coburn and Jennifer Russell. Learn more about each of the interviews conducted in the full article.
We have created a number of coding schemes to assess constructs central to our hypotheses and conjectures about school and district supports for instructional improvement in mathematics. For example, we have a developed three rubrics to assess the sophistication of interviewees’ understandings of key aspects of high-quality mathematics instruction, have achieved reliability in coding, and have retrospectively coded the 600 interviews that we conducted in the first three rounds of data collection. We have also developed a 3-point scale to assess the 200 participants’ expectations for students’ mathematical learning. This scale assesses the extent to which participants 1) view student motivation and performance as a relation between students and instruction rather than as a fixed student characteristic, and 2) describe specific actions that they and others are taking to support struggling students. We have consensus coded 120 interviews conducted in the first round of data collection. Initial analyses suggest a positive association between participants’ expectations for students’ mathematical learning, the sophistication of their visions of high-quality mathematics instruction, and the quality of teachers’ classroom practices. In addition, we have developed coding schemes to assess mathematics coaches’ practices, their relationships with school leaders, and their legitimacy in schools. We will also develop coding schemes for school leaders’ practices that will include 1) the frequency of classroom observations and quality of feedback to teachers, 2) whether and how they participate in mathematics PLCs, and 3) whether and how they support mathematics coaches’ work.
Surveys
The teacher, principal, and coach surveys are a key aspect of the project’s quantitative data collection. The overall goal of the quantitative data analysis is to test hypotheses about associations between school and district supports and changes in teachers’ content knowledge for teaching, their instructional practices, and student achievement. The surveys provide repeated measures of supports enacted at the school level. Learn more about each of the surveys administered in the full article.
The surveys include items developed and refined specifically for this project, and those already developed and field-tested in other research. Pre-existing items come from work by Bryk, Camburn, and Louis (1999), Bryk and Schneider (2002), Spillane (1996), the Consortium on Chicago School Research, the National Evaluation of the Eisenhower Professional Development Program, and the National Longitudinal Study of No Child Left Behind, Study of Instructional Improvement, and the Study of School Leadership. Some survey items about teacher learning communities and instructional leadership in mathematics come from an instrument developed by Spillane and colleagues (Distributed Leadership for Middle School Mathematics Education: Content Area Leadership Expertise in Practice, HER 0412510). A number of items developed for this project are based on the construct of the “Leadership Content Knowledge” (LCK) introduced by Stein & D’Amico (2000, April) and elaborated by Nelson (2005) and Stein and Nelson (2003). Items that had not been previously used were field-tested by conducting cognitive interviews of middle-school math teachers in two urban districts (American Statistical Association, 1997). This methodology is useful for identifying overly complex items, social desirability response bias, and unknowingly misleading responses (Biemer, Groves, Lyberg, Mathiowetz, & Sudman, 1991; Desimone & Le Floch, 2004).
We are currently developing a series of scales based on the surveys. One scale, principal involvement in math instruction, has been developed based on the work of the Consortium for Chicago School Research in general instructional leadership. The scale has a person separation reliability coefficient of 0.92 and a Chronbach’s alpha of 0.94. Preliminary work on a second scale, school-wide community, indicates an average inter-item covariance of 0.60 and a Chronbach’s alpha of 0.85. A third scale, instructional leadership, has an average inter-item covariance of 0.19 and a Chronbach’s alpha of 0.79.
Additional instrument information (including sample protocols) is available on the MIST website.
How have changes in state or national policy affected the work you’re doing in these districts?
From our perspective, policies that are consequential for mathematics instruction have been relatively stable in all the states in which the collaborating districts are located since we began working with them in 2007. This perception is due, in part, to the skill of district leaders in responding to changes in the external policy environment in ways that have kept their instructional improvement plans for mathematics on track. The most significant exception involves a state mandate for algebra in eighth grade in one of the districts. District leaders responded by deciding to skip a unit in the adopted textbook series in sixth grade, which led us to question the mathematical coherence of the resulting mathematics curriculum.
Leaders in all four districts have to cope with the tension between their ambitious goals for students’ mathematical learning and state assessments that emphasize procedural fluency. States in which three of the four districts are located are in the process of revising assessments to reflect a greater balance between conceptual understanding and procedural fluency. We and the mathematics specialists in these districts anticipate that these changes will reduce this very real tension, which is especially acute for school leaders.
In addition to considering the external policy environment, we have also found it important to take account of the history of each of the districts. For example, two of the districts have a history of site-based management. The implementation of certain aspects of these two districts’ improvement initiatives has been relatively uneven when compared with the other two districts.
Learn More: To learn more about this work, visit the MIST website or the project profiles on cadrek12.org.