The American Museum of Natural History (AMNH), in collaboration with New York University's Institute for Education and Social Policy and the University of Southern Maine Center for Evaluation and Policy, will develop and evaluate a new teacher education program model to prepare science teachers through a partnership between a world class science museum and high need schools in metropolitan New York City (NYC). This innovative pilot residency model was approved by the New York State (NYS) Board of Regents as part of the state’s Race To The Top award. The program will prepare a total of 50 candidates in two cohorts (2012 and 2013) to earn a Board of Regents-awarded Masters of Arts in Teaching (MAT) degree with a specialization in Earth Science for grades 7-12. The program focuses on Earth Science both because it is one of the greatest areas of science teacher shortages in urban areas and because AMNH has the ability to leverage the required scientific and educational resources in Earth Science and allied disciplines, including paleontology and astrophysics.

The proposed 15-month, 36-credit residency program is followed by two additional years of mentoring for new teachers. In addition to a full academic year of residency in high-needs public schools, teacher candidates will undertake two AMNH-based clinical summer residencies; a Museum Teaching Residency prior to entering their host schools, and a Museum Science Residency prior to entering the teaching profession. All courses will be taught by teams of doctoral-level educators and scientists.

The project’s research and evaluation components will examine the factors and outcomes of a program offered through a science museum working with the formal teacher preparation system in high need schools. Formative and summative evaluations will document all aspects of the program. In light of the NYS requirement that the pilot program be implemented in high-need, low-performing schools, this project has the potential to engage, motivate and improve the Earth Science achievement and interest in STEM careers of thousands of students from traditionally  underrepresented populations including English language learners, special education students, and racial minority groups. In addition, this project will gather meaningful data on the role science museums can play in preparing well-qualified Earth Science teachers. The research component will examine the impact of this new teacher preparation model on student achievement in metropolitan NYC schools. More specifically, this project asks, "How do Earth Science students taught by first year AMNH MAT Earth Science teachers perform academically in comparison with students taught by first year Earth Science teachers not prepared in the AMNH program?.”

This is a two-year exploratory study to identify critical aspects of effective science formative assessment (FA) practices for English Language Learners (ELLs), and the contextual factors influencing such practices. Three institutions join efforts for this purpose: University of Colorado at Boulder, University of Colorado at Denver, and University of Washington. FA, in the context of the study, is viewed as a process contributing to the science learning of ELLs, as opposed to the administration of discrete sets of instruments to collect data from students. The study targets Spanish-speaking, elementary and middle school students. Findings from this study contribute to advance knowledge and understanding of FA as an inherent component of the science learning process in linguistically diverse classrooms, and to define a research agenda aimed at enhancing science teachers' ability to enact equitable and effective assessment practices for this student subpopulation.

Three research questions guide the work: (1) What FA practices are occurring in science classrooms that serve predominantly mainstream students and in those serving predominantly ELLs?; (2) How are teachers' FA practices for mainstream students different from or similar to those used with ELLs?; and (3) How do contextual factors and teachers' cultural and linguistic competencies influence FA practices? To address these questions, two conceptual frameworks are used--one for characterizing FA events; the other for examining FA events as a communication process. The study employs a mixed-methods research approach with emphasis on case studies. The sample size consists of three school districts in Colorado and Washington, 16 classrooms (8 elementary, 8 middle school), 16 teachers, and 96 ELLs. Classrooms are selected to represent a particular combination of four factors: (a) teacher ethnicity, (b) teacher formal academic preparation in teaching ELLs, (c) type of linguistic student background, and (d) grade level. Students are selected through a stratified random sample, identified by achievement level (i.e., low, medium, high), and linguistic background (i.e., mainstream, ELL). Data collection strategies to document the implementation of FA at the beginning, during, and at the end of a science unit include: (a) classroom observation protocols, (b) classroom video-recording, (c) video/artifact simulated recall, (d) assessment artifacts, (e) student interviews, (f) teacher questionnaires, (g) teacher interviews, (h) school principal interviews, and (i) school observations. Reliability and validity of most of the data-gathering instruments is determined through pilot studies. Data interpretation strategies include: (a) coding based on the two conceptual frameworks, (b) scoring rubrics to identify levels of effectiveness, and (c) narratives and profiles to describe FA patterns. Publications and the development of a website constitute the main dissemination strategies. A technical advisory board is responsible for formative and summative evaluation. Key evaluation questions are: (1) To what extent does the project enhance research on ELL FA practices through case studies?, and (2) How effectively do the project dissemination activities facilitate understanding of FA practices?

Major project outcomes include: (1) a description of the patterns of formal and informal FA practices for ELLs; (2) a comparison of the FA practices observed in classrooms that vary on the dimensions of teacher characteristics and linguistic diversity; and (3) an empirically and theoretically informed set of findings and strategies for supporting teachers to enact and enhance FA practices sensitive to cultural and linguistic diversity. Three main products are developed: (1) a monograph describing the FA practices observed across the different classrooms with concrete examples; (2) a description of possible professional development strategies to improve in-service FA practices for linguistically diverse students; and (3) a research-informed approach for analyzing FA practices. Besides filling the existing research gap on FA with ELLs, outcomes and products serve as a foundation for a future research agenda and a comprehensive project aimed at ensuring equitable science learning for all students, including ELLs.

Westat will perform a multi-year evaluation of the DR K-12 program. Five broad evaluation questions guide the evaluation, though others may be forthcoming:

1. What does the portfolio of funded projects look like?
2. What percent of development-intensive projects funded in the DR K-12 program employ appropriate methods to evaluate the efficacy and apply them rigorously? What were the methods used to study these projects? What have been the results of these studies? What is the quality of non-development projects?
3. What percent of resources (instructional programs, models, or interventions) developed by the DR K-12 program are found to be effective and ready for adoption at scale?
4. What are the combined effects of the DR K-12 projects that have been evaluated with rigorous methods? What do these combined effects contribute to the knowledge base about innovative approaches to improving STEM learning and teaching? What are the effects from non-development projects?
5. Do the resources, models, tools, and technologies developed and/or studied in DR K-12 projects lead to significant improvement in student learning? To significant improvement in teacher STEM competency?

The evaluation will:

1. Collect and analyze secondary data including conducting portfolio analysis of all funded projects and a meta-analysis of all development-intensive projects to examine both the portfolio and the effects on students and teachers.
2. Administer a web survey of principal investigators about the perceived impacts of the projects, and evidence to support the claims.
3. Perform bibliometric analyses of publications and citations to examine impacts that go beyond the program to others in the STEM research field.
4. Conduct follow-up studies. Possible examples may include expert panel review to judge the quality and usefulness of potentially promising products identified or exploratory projects, case studies of selected projects or products that either are important but under-studied or are unique from a methodological or substantive perspective, etc.
5. Establish an advisory group with substantive expertise in STEM content area, STEM education, and evaluation research to provide advice and constructive criticism through the duration of the project.

This is a 3.5-year efficacy study of the Developing Mathematical Ideas (DMI) elementary math teacher professional development (PD) program. DMI was developed by staff from Education Development Center (EDC), SummerMath for Teachers, and TERC, the STEM research and development institution responsible for this research. DMI is a well-known, commercially available PD program with substantial prior evidence showing its impact on elementary teachers' mathematical and pedagogical knowledge. However, no studies have yet linked DMI directly with changes in teachers' classroom practice, or with improved student outcomes in math. This study aims to remedy this gap.

The research questions for the study are:

1) Does participation in the Developing Mathematical Ideas (DMI) professional development program lead to increases in reform-oriented teaching?

2) Does participation in DMI lead to increases in students' mathematics learning and achievement, especially in their ability to explain their thinking and justify their answers?

3) What is the process by which a reform-oriented professional development program can influence teaching practice and, thus, student learning? Through what mechanisms does DMI have impact, and with what kinds of support do we see the desired changes on our outcome measures when the larger professional development context is examined?

The dependent variables for this study include a) teachers' pedagogical and mathematics knowledge for teaching; b) the nature of their classroom practice; and c) student learning/ achievement in mathematics.

The study uses experimental and quasi-experimental methods, working with about 195 elementary grades teachers and their students in Boston, Springfield, Leominster, Fitchburg, and other Massachusetts public schools. Volunteer teachers are randomly assigned either to PD with DMI in the first year of the efficacy study, or to a control group that will wait until the second year of the study to receive DMI PD. Both groups of teachers will be followed through two academic years. Analyses use OLS regression, hierarchical modeling, and structural equation modeling, as appropriate, to compare the two groups and to track changes over time. In this way, the project explores several aspects of a conceptual framework hypothesizing relationships among PD, teacher mathematical and pedagogical knowledge, classroom teaching practice, and student outcomes. There are multiple measures of each construct, including video-analysis of teacher practice, and a new video-based measure of teacher knowledge.

The study tests the impact of DMI in a range of districts (large urban, small urban, suburban) serving an ethnically and economically diverse mix of students. It provides much needed, rigorous evidence testing the efficacy of this reform-oriented professional development program. It also directly explores the commonplace theory that teachers' understanding of content and student thinking and their encouragement of rich mathematical discourse for student sense-making lead to improvement on measures of mathematics achievement. Findings from the study are disseminated to both research and practitioner communities. The project provides professional development in mathematics to about 195 teachers to improve their ability to teach important concepts. If the evidence for efficacy is positive, then even larger-scale use of this PD program is likely.

This project conducts interdisciplinary research to advance understanding of embodied learning as it applies to STEM topics across a range of current technology-based learning environments (e.g., desktop simulations, interactive whiteboards, and 3D interactive environments). The project builds on extensive research, including prior work of the PIs, regarding both embodied learning and statistical learning. The PIs describe embodied learning as engaging the neuromuscular systems of learners as they interact with the world around them visually, aurally, and kinesthetically in order to construct new knowledge structures. Statistical learning is described as the ability to learn, often without intent, which sequences of stimuli are consistent with a set of rules. An example of statistical learning is pattern recognition, which is central to mastery of complex topics in many STEM disciplines including physics and mathematics.

The project has two central research questions: How are student knowledge gains impacted by the degree of embodied learning and to what extent do the affordances of different technology-based learning environments constrain or support embodied learning for STEM topics? To investigate these questions, the PIs are conducting three series of experiments in five phases using two physics topics. The first four phases are developmental and the final phase implements and assesses the two modules in schools (20 plus teachers, 700 plus K-12 students) in Arizona and New York (15 total sites, 10 plus public schools, minimum one Title I school).

The aim of this project is to meld these two research trajectories to yield two key outcomes: 1) basic research regarding embodiment and statistical learning that can be applied to create powerful STEM learning experiences, and 2) the realization of exemplary models and principles to aid curriculum and technology designers in creating learning scenarios that take into account the level of embodiment that a given learning environment affords. 

This research and development project studies the impact of Project-Based Inquiry Science (PBIS) on 6th grade students in a large urban school district. PBIS is a comprehensive, 3-year middle school science curriculum that focuses on standards-based science content and that uses project-based inquiry science units to help students learn. NSF funded the development of PBIS over the past two decades, with major investments made in the design of materials and with associated teacher professional development designed to help teachers understand the content of the units and how to teach them. Prior small-scale studies of PBIS have shown positive impact on student achievement and motivation, and on teacher use of reform-based instruction. The research questions explored are:

1. Efficacy. What is the impact of PBIS on student learning? To what extent do students in PBIS perform better than non-PBIS students on measures of learning?
2. Enactment and teacher practice. What is the impact of the curriculum on teaching quality? What is the fidelity of classroom implementation? How does the depth and level of implementation relate to student outcomes?

The study involves both quantitative and qualitative methods; the use of an experimental design allows estimates of causal impacts when combining professional development with the curriculum materials. This is a randomized control trial to test the efficacy of PBIS in 42 middle schools and with ˜120 teachers (21 schools and ˜60 teachers per condition), and affecting approximately 8,500 6th grade students. The dependent variables for students include results on state-level achievement tests and measures of their ability to develop and use models and construct explanations in the context of the curriculum units. Mediational analysis measures the association between contextual factors such as fidelity of implementation and quality of the professional development experience and student learning, allowing a deeper understanding of results.

This work is critical to the ongoing effort to support standards-based curriculum reform in science. PBIS has enjoyed some success in urban settings with diverse groups of students, including those from historically underrepresented groups in science, and now moves to larger scale. This curriculum is among a small number of science curriculum initiatives that are at a stage in the research and development cycle where implementation efforts are focused on scaling to a broader range of schools and districts. The curriculum units are based on design principles drawn from theory and research on how students learn and are aligned with learning goals found in state and national standards. Moreover, its design reflects where the science education field is headed – teaching a few big ideas and integrating scientific practices. Project outcomes will provide evidence about the effects of a published and available inquiry-based science curriculum.

This exploratory research study will bring together two promising innovations that have the potential to help more students meet high standards and prepare for college and 21st century careers. One innovation is a new high school course entitled Energizing Physics, designed to help students with a wide range of capabilities by applying best practices and presenting a relatively small number of key concepts in depth. Another is the BEAR assessment system, designed to provide frequent formative assessment data to students and teachers. The goal is to develop and test a formative assessment system for Energizing Physics that has the potential to enable all students to learn how to learn physics, so they can succeed in their first physics course in college. Partners include course authors Aaron Osowiecki and Jesse Southworth from Boston Latin School in Boston, Massachusetts, Cary Sneider and graduate students at Portland State University in Portland, Oregon, and assessment specialists Mark Wilson and Karen Draney at the Graduate School of Education, University of California at Berkeley.

The project will proceed in five phases. Phase I: During the summer of 2010 project teams from Massachusetts and Oregon will meet with assessment experts in California for training in the BEAR assessment system. Phase II: During the subsequent year the team will collaborate remotely to embed the BEAR system into the course materials, and recruit eight teachers (four in Massachusetts and four in Oregon) who will test the new materials in a variety of high school settings. Phase III: Weeklong workshops will be held in Oregon and Massachusetts during the summer of 2011 to familiarize teachers with the course and assessment system. Phase IV: Teachers will present the course to their students, collect pre-post test data on students' conceptual understanding and problem solving abilities, as well as work samples, and report on successes and challenges. Teams will conduct classroom visits and interview teachers at school sites. Phase V: During the summer of 2012 the teams will analyze the results, modify the course materials as appropriate, and report on findings.

Given the substantial body of research on the value of formative evaluation for supporting learning, this exploratory study has the potential to develop a physics course that could help teachers support learning among students with a wide diversity of capabilities. Further, since this research builds on a similar study of the high school course Living by Chemistry, which also uses the BEAR formative evaluation system, it may be possible to generalize ways that high school science courses can be designed to help more students succeed in college science.

<!--[if gte mso 10]> <! /* Style Definitions */ table.MsoNormalTable {mso-style-name:"Table Normal"; mso-tstyle-rowband-size:0; mso-tstyle-colband-size:0; mso-style-noshow:yes; mso-style-parent:""; mso-padding-alt:0in 5.4pt 0in 5.4pt; mso-para-margin-top:0in; mso-para-margin-right:0in; mso-para-margin-bottom:10.0pt; mso-para-margin-left:0in; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"Times New Roman"; mso-ascii-font-family:Cambria; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"Times New Roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:Cambria; mso-hansi-theme-font:minor-latin;} --> <!--[endif]--> <!--StartFragment-->

The project is conducting repeated randomized control trials of an approach to high school geometry that utilizes dynamic geometry (DG) software and supporting instructional materials to supplement ordinary instructional practices.  It compares effects of that intervention with standard instruction that does not make use of computer drawing/exploraction tools. The basic hypothesis of the study is that use of DG software to engage students in constructing mathematical ideas through experimentation, observation, data recording, conjecturing, conjecture testing, and proof results in better geometry learning for most students. The study tests that hypothesis by assessing student learning in 76 classrooms randomly assigned to treatment and control groups. Student learning is assessed by a geometry standardized test, a conjecturing-proving test, and a measure of student beliefs about the nature of geometry and mathematics in general. Teachers in both treatment and control groups receive relevant professional development, and they are provided with supplementary resource materials for teaching geometry. Fidelity of implementation for the experimental treatment is monitored carefully. Data for answering the several research questions of the study are analyzed by appropriate HLM methods. Results will provide evidence about the effectiveness of DG approach in high school teaching, evidence that can inform school decisions about innovation in that core high school mathematics course. The main research question of the project is: Is the dynamic geometry approach better than the business-as-usual approach in facilitating the geometric learning of our students (and more specifically our economically disadvantaged students) over the course of a full school year?

The main resources/products include geometry teachers’ professional development training materials, suggested dynamic geometry instructional activities to supplement current high school geometry curriculum, instruments such as Conjecturing-Proving Test, Geometry Belief Instrument, Classroom Observation Protocols, DG Implementation Questionnaire and Student Interview Protocols. 

The general plan for the four-year project is as follows:

Year 1: Preparation (All research instruments, professional development training and resource materials, recruitment and training of participants, etc.); 

Year 2: The first implementation of the dynamic geometry treatment, and related data collection and initial data analysis; 

Year 3: The second implementation of the DG treatment, and related data collection and data analysis; 

Year 4: Careful and detailed data analysis and reporting.

We are now in project year 3. Data are collected for the second implementation of the DG treatment. For data collected during project year 2, some initial analysis (the analysis on the geometry pretest and posttest data and the psychometric analysis on the project developed instruments) has been conducted. More thorough analysis of the collected data is still on going. The analysis on the geometry test shows that the experimental group significantly outperformed the control group on geometry performance.

The evaluation will be implemented throughout the project’s four-year duration, with an evolving balance of formative and summative evaluation activities.  In the project’s first three years, the evaluation will emphasize formative functions, designed to inform the project research team of the relative strengths and weaknesses of the research design and execution, and target corrections and improvements of the research components. Summative evaluation activities will also take place in these years with the collection of data on student achievement and teacher change. Evaluation activities for year 4 will focus on the summative evaluation of the project’s accomplishment and especially its impact on participating teachers and students. Evaluation reports will be issued annually with a final summative report presented at the end of year 4.

The research results will be disseminated via the following efforts: 1) Creating and constantly updating the project web site; 2) Publishing the related research articles in research journals such as Journal for Research in Mathematics Education; 3) Presenting at state, regional, national, and international research and professional meetings; 4) Meeting with state and local education agencies, schools, and mathematics teacher educators at other universities for presenting the research findings and using the DG approach in more schools and more mathematics teacher education programs; and 5) Contacting more school districts, with a view to developing relationships and ties that would smooth the way to disseminate the research results.  

<!--EndFragment-->

 The Children's Measurement Project examines children's developing knowledge from PreKindergarten through Grade 5 as they develop the capacity and strategies they need to measure geometric space (length, area and volume), investigating number concepts, early algebra, or variability. We investigate ways children learn to use measures as evidence for scientific or mathematical claims. We began by examining the literature on learning trajectories and progressions to interpret existing research on children's understanding of length, area and volume. Our work engages both Rasch modeling and learning/teaching experiments within clinical and classroom contexts to collect data for longitudinal accounts of children's development of measurement concepts and strategies. The work is being conducted as a collaboration of Illinois State University and the University at Buffalo (State University of New York). We are beginning the fourth year of our project (2010).