Advancing understanding of how early learning of mathematics progress is dependent on the development of good measures. Presenters of this session produced the “Research-based Early Mathematics Assessment” (REMA), and now better understand its limitations. They are now creating and evaluating a substantial revision of the REMA using emerging multidimensional theoretical models, both cognitive and psychometric. The development of this instrument will lead to substantive advances in the fields of mathematics education research, cognitive psychology, and psychometric theory. 

There are two major objectives. First, presenters are producing a cognitively diagnostic adaptive assessment that will yield more useful and detailed information about students’ knowledge of mathematics than previously possible. This instrument will produce data on (a) students’ level of thinking along multiple empirically validated learning trajectories with detailed individual cognitive profiles and (b) individual cognitive data that can be aggregated at the classroom, school, district and (even) state levels, therefore providing specific information about teaching and curricula. The adaptive assessment will be a boon to researchers, teachers, and policymakers who seek large-sample, quantitative indicators of students’ learning.

Second, this advance allows them to subject their learning trajectories to close cognitive diagnosis using cutting-edge psychometric approaches. That is, the NSF has funded a number of researchers to create such trajectories, but few have proposed rigorous models for validating these progressions. We and others have tested them via the Rasch model. However, these models force an artificial and restrictive unidimensional model on subject content that are unsatisfactory both theoretically and practically.

Presenters will share and discuss their first two years’ work, developing the Q-Matrices and analyzing the legacy data from the REMA using Q-matrix theory, the Rule Space Method, and poset-based adaptive testing methodologies. They seek critical commentary from participants at this still-early stage of development. At the general level, that will include their framework, theory, and analyses. A more detailed interaction results from sharing paper copies with all participants of one set of items, attributes, and the Q-Matrix that connects them and allow participants time to react and critique the specific assignments of attributes to items on that Q-Matrix.

This collaborative session brings together four mature projects that use different approaches to develop and validate technology-enhanced STEM assessments. Presenters share their designs, research findings, and implications for measuring STEM standards. All offer evidence addressing four questions: (1) What was the technical quality (reliability and validity) and effectiveness of the assessments for their intended purposes? (2) How feasible were the assessments to implement in classrooms? (3) How can the projects can be scaled up and sustained? (4) What challenges were encountered during the projects and should be addressed in future research and development?

All session participants are asked to raise issues about STEM assessment in their projects.

The Calipers II: Using Simulations to Assess Complex Science Learning project developed simulation-based assessments to be embedded in ongoing curricula intended for formative purposes and unit benchmark assessments as summative measures. The evidence-centered design process is described along with findings from field tests in three states with over 6,000 students. Technical quality of the assessments’ measurement of science-system knowledge and inquiry practices was established by alignments with national standards, cognitive labs, and psychometric analyses. Challenges to broader impact are proposed.

The Assistments for Science Inquiry presentation describes the environment and the physical sciences microworlds for middle school science. The project uses educational data mining on log data to develop detectors to assess students’ inquiry skills in real time.

The third presentation summarizes findings from two projects that apply facet-based approaches to formative assessment practices: Contingent Pedagogies for Middle School Earth Science and Chemistry Facets for high school chemistry. Analyses of student written responses and think alouds established the cognitive and content validity of facet clusters, which describe learning goals and common problematic student ideas, and of diagnostic questions. Both projects tested the value of resources in small-scale, quasi-experimental field trials.

The fourth presentation shares findings from technology-enhanced item types delivered through the UC Berkeley Formative Assessment (FADS) project, which employs automated scoring and the partial credit model. Items were developed based on the Constructing Data, Modeling Worlds curriculum from Vanderbilt University for middle school students in mathematics.

The Partnership for the Assessment of College and Career Readiness (PARCC) is a consortium of 24 states that received a Race to the Top Assessment Grant from the U.S. Department of Education in the fall of 2010 to build a next-generation student assessment system. The assessments will be based on the Common Core State Standards (CCSS) for English Language Arts/Literacy and Mathematics and target students in grades 3–11. This presentation (1) provides an overview of the design of the assessments, including their primary purposes, key features of tests, and the “claims” about student performance that the tests will report; (2) describes the research studies planned for 2013 and 2014 that are designed to inform implementation of the assessments in 2015; and (3) describes the resources PARCC has or plans to develop to support the implementation of the CCSS, including a rubric and protocol for evaluating the quality of local, state, and commercially available instructional materials with respect to alignment to the focus and rigor of the CCSS.

Work is progressing to develop the Next Generation Science Standards (NGSS). With private funding from the Carnegie Corporation and support from National Science Teachers Association (NSTA) and the American Association for the Advancement of Science (AAAS), the National Research Council (NRC) and Achieve, Inc., have embarked on a two-step cooperative process to develop the NGSS. The first step was to develop a conceptual framework that is grounded in current research on science and science learning and identifies the science all K–12 students should know. The NRC released the Framework for K–12 Science Education in July of 2011. The next step is the development of the actual standards, a process led by Achieve involving science experts, science teachers, states, and other science education partners. This two-step process takes a broader and more cooperative approach to the development of science standards. The NGSS are due for completion in early 2013.

The session brings together seven DR K–12 projects focused on game-based STEM learning and assessments of science content and inquiry. The panel provides an overview of what they know about how learning takes place in games and how each of these projects is crafting assessments in virtual and game-based environments. They focus on strategies for leveraging popular game mechanics with research from the learning sciences, psychology, science education, and computer science to support and assess players as they develop robust understandings of core scientific concepts and practices. A synthetic discussion then explores challenges and opportunities for integrating research from these fields synergistically rather than disruptively within popular game-play mechanics. The projects include:

Argumentation: A Middle School Game-based Approach—Marilyn Ault discusses designs for leveraging competition and timed challenges to enhance engagement and fun within the Evidence Game. The discussion explores data collected about the relationship between racing elements of the game and the overarching focus on scientific argumentation at the heart of the study from a disciplinary perspective.

Promoting Problem Solving and Engagement—Jon Rowe discusses how intelligent game-based environments can promote problem solving and engagement in science learning by integrating affective connection with virtual characters (agents) within the game and core challenges in problem solving. He discusses laboratory and classroom studies with the CRYSTAL ISLAND game-based learning environment, investigating engagement (motivation, situational interest, presence) and central issues of problem solving (strategy use, divergent thinking, and collaboration) with respect to achievement as measured by both science content knowledge and transfer.

Socially Engineering Gaming Communities to Bootstrap Formal Understanding—This project scaffolds explanations between players to support articulation of the intuitive understandings players develop through game play. The presenter discusses data from a study on the natural collaborations that occur between students as they play SURGE. These natural collaborations include both spontaneous face-to-face collaborations within the classroom as well as contributions within online strategy forums integrated within the SURGE for each classroom.

Embedding Principled Assessments into Serious Learning Games—This project integrates evidence-centered assessment design within game mechanics to shape game challenges, rules for success and failure, and advancement through game levels. This discussion focuses on approaches for design of embedded, unobtrusive assessments within the game that balance the rigor and structure needed for cognitively principled assessment with the fun, exploration, and challenge of gameplay.

SAVE Science: Situated Assessment Using Virtual Environments for Science Content and Inquiry—This series of virtual-environment-situated assessment modules assesses both science content and inquiry in grades 7 and 8. The modules make use of a novel assessment rubric based on student interactions within an authentic context-based science curriculum—embedded in a virtual environment—and relate the assessments to standardized test achievement.

AutoMentor: Virtual Mentoring and Assessment in Computer Games for STEM Learning—This program uses an automated mentoring system that utilizes natural language conversations to help students learn about science and technology, and an assessment/analysis protocol to quantify students’ STEM behavior. This project combines automated tutoring with an automated mentoring technology, AutoMentor, with Epistemic Network Analysis, which analyzes the framework that compares the way learners solve problems to the epistemic framework that an expert might use. This project is studying middle school students either in after-school or in-school programs.

Leveling Up: Supporting and Measuring High School STEM Knowledge Building in Social Digital Games—This project is developing a set of wireless and Web-based free-choice game elements using game mechanics based on Common Core high school science concepts. The game elements will use the model of layered learning that comes from professional game design to create challenges where completion of the game element is only possible through understanding the basic game mechanics (thus the principles of science). In another grant, Arcadia: The Next Generation—Transforming STEM Learning through Transmedia Games, developers are embedding these game elements in a transmedia social game environment. Arcadia will host cross-disciplinary science inquiry games that make use of the individual game elements from Leveling Up and use the electronic activity logs and artifacts from games to measure the ICT and inquiry-skill development in the context of science investigation.