The MIST project investigates how changes in the school and district settings in which mathematics teachers work influence their instructional practices, students' learning opportunities, and student achievement. The project is also developing instruments and methods for analyzing data to document the institutional setting of mathematics teaching that are specific to equity and access for all middle school students to high quality mathematics instruction.
Q&A with MIST
The MIST project team answered a few of our questions about their project background, the instruments they're developing, what they're learning, and how their work is influenced by local, state, and national contexts.
Where did the idea for this project come from?
The overall goal of the project is to understand what it takes to support teachers in improving the quality of their mathematics instruction on a large scale. To this end, we are attempting to develop an empirically grounded theory of action that can inform instructional improvement efforts at the level of large urban districts. The idea for this project emerged in the course of a prior teacher development project that involved a five-year collaboration with groups of middle-school mathematics teachers in two urban districts (REC-0231037: Developing Articulated Models for Supporting and Sustaining Teacher Development Efforts in the Context of Schooling, PIs Paul Cobb and Kay McClain). In preparing for the prior project, we knew both from experience and from the research literature that teachers’ instructional practices are profoundly influenced by the school and district settings in which they work. We therefore planned to document the settings in which the collaborating teachers worked, and attempted to find a researcher in either educational policy or educational leadership who was willing to work with us to develop analyses that would inform a teacher development effort while it was still in progress. However, it soon became apparent that researchers in these fields typically conduct observational studies in which they investigate others' efforts to support instructional improvement. As one prominent leadership researcher put it, providing feedback about school and district settings in order to inform work with teachers involves "messing with the intervention."
Against this background, we took the lead ourselves and developed an approach for analyzing the school and district settings of mathematics teaching that is tailored to our purposes as mathematics educators. The resulting accounts of the settings in which the collaborating teachers developed and revised their instructional practices proved to be useful not only in informing our professional development plans but in explaining the teachers’ learning. Furthermore, it became apparent that supporting instructional improvement involves, in part, bringing about changes in the settings in which teachers develop and revise their instructional practices. Activities in one of the two teacher groups in particular focused explicitly on working to influence the organizational conditions in which the teachers worked (e.g., regularly scheduled time for collaboration, longer periods for mathematics instruction). This experience served to emphasize that instructional improvement is a problem of organizational learning as well as teacher learning.
The goal of the theory of action for instructional improvement that we are developing in the current project is to inform the development of school and district settings that support teachers’ ongoing improvement of their instructional practices. Furthermore, and in contrast to the prior project, this work involves a genuine collaboration between mathematics educators and policy researchers. As part of this work, we give each of the four collaborating districts detailed feedback about how their improvement policies are actually playing out in schools and classrooms, and make actionable recommendations about how those policies might be adjusted to make them more effective. In doing so, we are attempting to do for district leaders what we had hoped others would do for us in the prior project, but at the level of district-wide improvement rather than a more modest collaboration with groups of teachers.
What are some examples of conjectures that you’ve made about policies and supports needed to implement ambitious instructional practices at the scale of a large, urban district? Based on what you’ve learned thus far, what are your current hypotheses and conjectures?
As part of our preparation for the project, we developed a set of research-based hypotheses and conjectures about the supports necessary for district-wide improvements in the quality of mathematics instruction. These hypotheses and conjectures included shared instructional vision, teacher networks, accountability relations and relations of assistance between instructional leaders and teachers, and relations among central district office units.
We have completed three annual rounds of data collection, analysis, and feedback, and now consider that our starting hypotheses and conjectures were relatively abstract. We currently frame our hypotheses and conjectures in terms of concrete, potentially learnable practices for members of specific role groups (e.g., teachers, mathematics coaches, school leaders).
Our current theory of action for instructional improvement at scale comprises the following five elements:
- a coherent instructional system for supporting teachers’ development of ambitious teaching practices
- teacher networks
- mathematics coaching
- school instructional leadership
- district instructional leadership
We briefly describe each component below. We contend that instructional improvement at scale requires the simultaneous coordination of all five components.
The first component, a coherent instructional system, includes the following elements: a) explicit goals for students’ mathematical learning; b) a detailed vision of high-quality instruction that specifies concrete instructional practices that will lead to the attainment of the learning goals; c) instructional materials and associated tools designed to support teachers’ development of these practices; d) district professional development that focuses on the specific practices, is organized around the above materials, and is sustained over time; e) school-based professional learning communities (PLCs) that provide ongoing opportunities for mathematics teachers to discuss, rehearse, and adapt the practices that are introduced in district professional development; f) assessments aligned with the goals for students’ mathematical learning that can inform the ongoing improvement of instruction and identify students who are currently struggling; and g) additional supports for struggling students to enable them to succeed in mainstream instruction.
Teachers’ social networks are a key support for school-wide instructional improvement and therefore constitute the second component of our proposed theory of action. Although teacher networks are emergent phenomena and cannot simply be mandated into existence, district and school improvement policies can influence the conditions under which teachers decide whether to turn to a colleague for instructional advice and the types of advice they seek. In this regard, the network analyses that we have conducted in our current work indicate that school-based PLCs can facilitate the emergence of teacher networks. The extent to which a teacher network does in fact support the participating teachers’ learning depends crucially on the nature of their interactions with one another. Our network analyses also indicate that at least one member of the network, which might include a coach or an instructional leader, needs to have developed relatively sophisticated knowledge for teaching and/or accomplished instructional practices if networks are to support teacher learning.
The third component of our proposed theory of action for instructional improvement concerns mathematics coaching. Although US districts are increasingly using coaches as a primary means of supporting teachers’ learning, the designs of their coaching programs vary considerably. Additionally, the research base on how coaches might work with individual teachers in their classrooms and on what constitutes high-quality coach professional development is extremely thin. Our current position is that coaches should spend at least some time leading groups of teachers to investigate and enact desired forms of practice, especially if there are school-based PLCs. Current research on the development of complex practices suggest that it is critical that novices co-participate in activities that approximate as closely as possible the targeted practices with more accomplished others. In the case of one-on-one coaching, this implies that desired activities are ones in which the teacher (i.e., the novice) co-participates with the coach (i.e., the more accomplished other) in activities central to teaching. Based on our work with the districts and on research on teacher learning, we therefore conjecture that the following are “high-leverage” coaching activities: a) co-teaching and b) enacting the coaching cycle of jointly planning a lesson, observing the enactment of the lesson, and then jointly analyzing the lesson. However, our initial analyses indicate that districts have to overcome a number of challenges in order to support the development of coaches’ expertise as more knowledgeable others, and to ensure that this expertise is deployed effectively. For example, we have found that the coaches are often not much more sophisticated in their visions of instruction or practice as compared to the teachers they are coaching. We have also found that unless school leaders understand the goals for students’ mathematical learning and the guiding vision of instruction, they may not support the work of coaches effectively. In our current work, we have documented a significant number of cases in which principals assigned additional duties to coaches that took them away from their work with teachers (e.g., analyzing data to identify struggling students, tutoring struggling students). However, observations indicate that principals protect coaches’ time when they understand the coaches’ role in the improvement effort.
The fourth component of the theory of action for improving the quality of mathematics instruction at scale concerns school instructional leadership. Current research provides contradictory guidance on school instructional leadership. Some researchers argue that it is sufficient for school leaders to understand general, content-independent principles of learning and instruction whereas others contend that school leaders need a deep understanding of mathematics, students’ mathematical learning, and teacher learning. Our initial findings suggest that professional development for school leaders based on the first view is too abstract and that most school leaders are not able to connect general principles to concrete instructional practices. It also appears that the provision of professional development based on the second view is beyond the capacity of most districts. Based on our current work, we envision a distribution of instructional leadership where coaches and district mathematics specialists are primarily responsible for supporting teachers’ learning, and school leaders press and hold teachers accountable for developing the intended instructional practices. Our findings indicate that it is not feasible for school leaders to serve as primary supports for mathematics teachers’ learning. However, initial findings also indicate that it is feasible and indeed necessary for school leaders to communicate instructional expectations to teachers that are aligned with the instructional vision guiding the improvement effort, and to ensure that teachers are provided with supports to enable them to meet those expectations. More specifically, we have identified two concrete instructional leadership practices aimed at pressing teachers to develop the intended forms of practice and providing teachers with adequate support: 1) observing mathematics instruction and providing feedback, and 2) participating in mathematics PLCs. In additional, current research indicates that school leaders play a critical role in enabling coaches to support teachers’ learning effectively.
The fifth and final component of the theory of action for improving the quality of mathematics instruction concerns district instructional leadership. At the outset of the project, we conjectured that the relationship between central office units would influence the success of the collaborating districts’ instructional improvement efforts. This has proved to be the case. First, it appears critical that district leaders in the central office units of Curriculum and Instruction (C&I), Leadership, English language learners, and Special Education share both goals for students’ mathematical learning and a vision of high-quality instruction (i.e., goals for teachers’ learning). The alignment of the agendas of C&I (responsible for teacher and coach professional development) and Leadership (responsible for supporting and assessing school leaders) has proved to be particularly critical. For example, in one of the collaborating districts, we have found that while the efforts of leaders in C&I focus on supporting teachers’ and coaches’ development of ambitious practices, leaders in Leadership hold principals accountable primarily for the improvement of students’ mathematics achievement scores. In turn, school leaders communicate these expectations to teachers, and do not press teachers to improve the quality of instruction. Additionally, school leaders direct resources toward providing supplemental supports for struggling students that are not aligned with mainstream classroom instruction (e.g., directing coaches to coordinate tutoring programs that focus on basic computational skills rather than working with teachers to improve the quality of instruction).
We contend that all five components of the proposed theory of action are necessary for large-scale instructional improvement; the prospects for achieving and sustaining instructional improvement diminish significantly if any one of the components is neglected. For example, we would question an improvement strategy that focuses on high-quality curriculum materials, teacher professional development, and mathematics coaching but does not attend to school leaders’ development as instructional leaders. Such a strategy is suspect because school leaders are unlikely to either press teachers to develop the intended practices or support coaches’ work with teachers.
The MIST website lists several instruments that you have developed in order to do this work. Can you tell us more about these instruments (e.g., the measures of reliability and validity, use and audience information)?
Interviews
We conduct two types of interviews each year. In the early fall, we interview key leaders in each of the collaborating districts (e.g., Chief Academic Officer, head of Curriculum and Instruction, head of Mathematics Department, head of Office of Leadership, head of Office of English Language Learners) to find out the district’s theory of action for improving middle-grades mathematics instruction for the academic year (i.e., the intended design). In January, we interview teachers, coaches, principals, and district leaders to find out how the theory of action is playing out in schools and classrooms. Questions about networks (teacher, coach, principal) were adapted from interview protocols provided by Cynthia Coburn and Jennifer Russell.
Theory of Action (Fall). This protocol includes questions for finding out about key aspects of a district’s design for improving middle-grades mathematics teaching and learning. This protocol is customized based on each district’s plans and is revised each year.
Teacher Interview (January). This protocol focuses on the teacher’s teaching responsibilities, opportunities for teacher collaboration, teacher’s views on high-quality instruction in mathematics, how the teacher works with the coach and principal (or assistant principal), and the supports and resources that have been provided to the teacher. This protocol is then customized based on each district’s theory of action and is revised each year.
Coach Interview (January). This protocol focuses on the coach’s role, how the coach works with teachers and administrators, the coach’s views on high-quality instruction in mathematics, and the supports and resources that have been provided to the coach. This protocol is then customized based on each district’s theory of action and is revised each year.
Principal Interview (January). This protocol focuses on the principal’s understanding of the district’s theory of action for instructional improvement in middle-grades mathematics, his/her goals and the current plan for improvement in mathematics teaching in the school, to whom the principal is accountable and for what, principal’s views on high-quality instruction in mathematics, how the principal works with the coach, and the supports and resources that have been provided to the principal. This protocol is then customized based on each district’s theory of action and is revised each year.
District Leaders Interviews (January). We interview leaders in several key central office units each January to find out how the improvement efforts in middle-grades mathematics are progressing. These protocols focus on the leader’s role in the district, the current middle school math initiative(s), the leader’s views on high-quality instruction and high-quality instructional leadership, and how the different units in the district office work together to promote instructional leadership and improvement in middle school mathematics instruction. We created protocols for leaders in each office that had a stake in the middle-grades mathematics improvement efforts (e.g., Office of Curriculum and Instruction, Office of Leadership, Mathematics Department, Office of English Language Learners, Special Education, Office of Research, Evaluation, and Accountability).
We have created a number of coding schemes to assess constructs central to our hypotheses and conjectures about school and district supports for instructional improvement in mathematics. For example, we have a developed three rubrics to assess the sophistication of interviewees’ understandings of key aspects of high-quality mathematics instruction, have achieved reliability in coding, and have retrospectively coded the 600 interviews that we conducted in the first three rounds of data collection. We have also developed a 3-point scale to assess the 200 participants’ expectations for students’ mathematical learning. This scale assesses the extent to which participants 1) view student motivation and performance as a relation between students and instruction rather than as a fixed student characteristic, and 2) describe specific actions that they and others are taking to support struggling students. We have consensus coded 120 interviews conducted in the first round of data collection. Initial analyses suggest a positive association between participants’ expectations for students’ mathematical learning, the sophistication of their visions of high-quality mathematics instruction, and the quality of teachers’ classroom practices. In addition, we have developed coding schemes to assess mathematics coaches’ practices, their relationships with school leaders, and their legitimacy in schools. We will also develop coding schemes for school leaders’ practices that will include 1) the frequency of classroom observations and quality of feedback to teachers, 2) whether and how they participate in mathematics PLCs, and 3) whether and how they support mathematics coaches’ work.
Surveys
The teacher, principal, and coach surveys are a key aspect of the project’s quantitative data collection. The overall goal of the quantitative data analysis is to test hypotheses about associations between school and district supports and changes in teachers’ content knowledge for teaching, their instructional practices, and student achievement. The surveys provide repeated measures of supports enacted at the school level.
The teacher survey measures teachers’ perceptions of supports for developing ambitious instructional practices in mathematics. It includes items that focus on the level of teachers’ participation in professional learning communities and informal networks, the extent to which they seek instructional advice, and who they seek advice from (e.g., math coaches); the degree to which the interactions between teachers in professional learning communities and networks focus on central mathematical ideas and how to relate them to students’ reasoning; the degree to which teachers and formal and informal leaders have a shared vision for math instruction and student learning; the degree to which instructional leadership is distributed among formal and informal leaders; and the professional development that teachers have received in support of improved instructional practices in mathematics.
The principal survey includes items that focus on the assistance that mathematics coaches provide to support the principal’s work as an instructional leader in mathematics; the principal’s work to both support teachers and hold them accountable for developing ambitious instructional practices in mathematics; and the professional development that the principal has received regarding the district’s instructional program in mathematics.
The coach survey includes items that focus on the coach’s work to both support teachers and hold them accountable for developing ambitious instructional practices in mathematics; assistance provided by mathematics coaches to support the principal’s work as an instructional leader in mathematics; and professional development that the coach has received regarding the district’s instructional program in mathematics.
The surveys include items developed and refined specifically for this project, and those already developed and field-tested in other research. Pre-existing items come from work by Bryk, Camburn, and Louis (1999), Bryk and Schneider (2002), Spillane (1996), the Consortium on Chicago School Research, the National Evaluation of the Eisenhower Professional Development Program, and the National Longitudinal Study of No Child Left Behind, Study of Instructional Improvement, and the Study of School Leadership. Some survey items about teacher learning communities and instructional leadership in mathematics come from an instrument developed by Spillane and colleagues (Distributed Leadership for Middle School Mathematics Education: Content Area Leadership Expertise in Practice, HER 0412510). A number of items developed for this project are based on the construct of the “Leadership Content Knowledge” (LCK) introduced by Stein & D’Amico (2000, April) and elaborated by Nelson (2005) and Stein and Nelson (2003). Items that had not been previously used were field-tested by conducting cognitive interviews of middle-school math teachers in two urban districts (American Statistical Association, 1997). This methodology is useful for identifying overly complex items, social desirability response bias, and unknowingly misleading responses (Biemer, Groves, Lyberg, Mathiowetz, & Sudman, 1991; Desimone & Le Floch, 2004).
We are currently developing a series of scales based on the surveys. One scale, principal involvement in math instruction, has been developed based on the work of the Consortium for Chicago School Research in general instructional leadership. The scale has a person separation reliability coefficient of 0.92 and a Chronbach’s alpha of 0.94. Preliminary work on a second scale, school-wide community, indicates an average inter-item covariance of 0.60 and a Chronbach’s alpha of 0.85. A third scale, instructional leadership, has an average inter-item covariance of 0.19 and a Chronbach’s alpha of 0.79.
Additional instrument information (including sample protocols) is available on the MIST website.
How have changes in state or national policy affected the work you’re doing in these districts?
From our perspective, policies that are consequential for mathematics instruction have been relatively stable in all the states in which the collaborating districts are located since we began working with them in 2007. This perception is due, in part, to the skill of district leaders in responding to changes in the external policy environment in ways that have kept their instructional improvement plans for mathematics on track. The most significant exception involves a state mandate for algebra in eighth grade in one of the districts. District leaders responded by deciding to skip a unit in the adopted textbook series in sixth grade, which led us to question the mathematical coherence of the resulting mathematics curriculum.
Leaders in all four districts have to cope with the tension between their ambitious goals for students’ mathematical learning and state assessments that emphasize procedural fluency. States in which three of the four districts are located are in the process of revising assessments to reflect a greater balance between conceptual understanding and procedural fluency. We and the mathematics specialists in these districts anticipate that these changes will reduce this very real tension, which is especially acute for school leaders.
In addition to considering the external policy environment, we have also found it important to take account of the history of each of the districts. For example, two of the districts have a history of site-based management. The implementation of certain aspects of these two districts’ improvement initiatives has been relatively uneven when compared with the other two districts.
Learn More: To learn more about this work, visit the MIST website or the project profiles on cadrek12.org (links below).