Illustrations with Graphic Devices in Large-Scale Science Assessments: An Exploratory Cross-Cultural Study of Students’ Interpretations

In this exploratory, cross-cultural study, we examined students’ interpretations of graphic devise-based illustrations used in science tests. Graphic devices are visual components (e.g., arrows, dotted lines) intended to ensure proper understanding of the scientific processes or phenomena represented by the illustrations. We address cultural differences in terms of the interaction of two factors, students’ country of origin and items’ country of origin. We hypothesized that interpretations made by students of device-based illustrations are more accurate for items generated in their own country than items generated in another country. Two matched samples of American college students who lived and studied in the U.S. (n=40) and Chinese college students who lived and studied in mainland China (n=40) were given illustrations from eight science items whose illustrations contained different sorts of graphic devices; four of those items were sampled from Chinese large-scale assessments and four from American large-scale assessments. For each illustration,
students were asked: (1) to describe what they saw in the illustration, and (2) whether they thought the illustration represented a scientific concept and, if so, to describe which scientific concept was represented. The accuracy of the responses was scored based on scoring rubrics developed for each item. The results indicate that: (1) some illustrations were more difficult to interpret accurately than others, regardless of the students’ or the items’ country of origin; (2) Chinese students had more accurate interpretations than their American counterparts of the scientific concepts represented by the illustrations; and (3) students’ interpretations of the scientific concepts illustrated were more accurate for items generated in the students’ own culture than items generated in the other culture. We discuss lessons learned from this exploratory study and future directions for a full study.