Motz, B. A., Carvalho, P. F., de Leeuw, J. R., & Goldstone, R. L. (2018). Embedding experiments: staking causal inference in authentic educational contexts. Journal of Learning Analytics,5, 47-59. doi: 10.18608/jla.2018.52.4
To identify the ways teachers and educational systems can improve learning, researchers need to make causal inferences. Analyses of existing datasets play an important role in detecting causal patterns, but conducting experiments also plays an indispensable role in this research. In this article, we advocate for experiments to be embedded in real educational contexts, allowing researchers to test whether interventions such as a learning activity, new technology, or advising strategy elicit reliable improvements in authentic student behaviours and educational outcomes. Embedded experiments, wherein theoretically relevant variables are systematically manipulated in real learning contexts, carry strong benefits for making causal inferences, particularly when allied with the data rich resources of contemporary e-learning environments. Toward this goal, we offer a field guide to embedded experimentation, reviewing experimental design choices, addressing ethical concerns, discussing the importance of involving teachers, and reviewing how interventions can be deployed in a variety of contexts, at a range of scales. Causal inference is a critical component of a field that aims to improve student learning; including experimentation alongside analyses of existing data in learning analytics is the most compelling way to test causal claims.
Download PDF of article here
McColeman, C., Michal, A., Goldstone, R. l., Schloss, K., Kaminski, J., & Hullman, J. (2018). Data visualization as a domain to research areas in cognitive science. Proceedings of the 40th Annual Conference of the Cognitive Science Society. (pp. 35-36). Madison, Wisconsin: Cognitive Science Society.
How people are able to turn information in the environment into meaning is a critical question for cognitive science. That environment is increasingly data-driven. Using data to inform decisions and improve understanding of the world is a valuable component of critical thinking, and serves as the foundation of evidence-based decision making. Designing graphical representations can make those data more accessible, such that users may engage the visual system and capacity for visual pattern recognition to discern regularities and properties of data. We ultimately want to understand the connection between the initial perception of data visualizations and conceptual understanding of information. Data visualizations, broadly, are the representation of recorded values in visual form, including scientific visualizations such as brain scans, or live visualizations such as stock market monitoring; the work discussed through this symposium is of the type used in science, business, and medical settings to display data abstractly.
Download PDF of paper here
Yu, J., Landy, D., & Goldstone, R. L. (2018). Visual flexibility in arithmetic expressions. Proceedings of the 40th Annual Conference of the Cognitive Science Society. (pp. 2750-2755). Madison, Wisconsin: Cognitive Science Society.
We investigated whether, and in what, ways people use visual structures to evaluate mathematical expressions. We also explored the relationship between strategy use and other common measures in mathematics education. Participants organized long sum/products when visual structure was available in algebraic expressions. Two experiments showed a similar pattern: One group of participants primarily calculated from left to right, or combined identical numbers together. A second group calculated adjacent pairs. A third group tended to group terms which either produced easy sums (e.g., 6+4), or participated in a global structure. These different strategies were associated with different levels of success on the task, and, in Experiment 2, with differential math anxiety and mathematical skill. Specifically, problem solvers with lower math anxiety and higher math ability tend to group by chunks and easy calculation. These results identify an important role for the perception of coherent structure and pattern identification in mathematical reasoning.
Download PDF of article here
Yu, J., Goldstone, R. L., & Landy, D. (2018). Experientially grounded learning about the roles of variability, sample size, and difference between means in statistical reasoning. Proceedings of the 40th Annual Conference of the Cognitive Science Society. (pp. 2744-2749). Madison, Wisconsin: Cognitive Science Society.
Despite its omnipresence in this information-laden society, statistics is hard. The present study explored the applicability of a grounded cognition approach to learning basic statistical concepts. Participants in 2 experiments interacted with perceptually rich computer simulations designed to foster understanding of the relations between fundamental statistical concepts and to promote the ability to reason with statistics. During training, participants were asked to estimate the probability of two samples coming from the same population, with sample size, variability, and difference between means independently manipulated. The amount of learning during training was measured by the difference between participants’ confidence judgments and those of an Ideal Observer. The amount of transfer was assessed by the increase in accuracy from a pretest to a posttest. Learning and transfer were observed when tailored guidance was given along with the perceptually salient properties. Implications of our quantitative measures of human sensitivity to statistical concepts were discussed.
Download PDF of article
Motz, B.A., de Leeuw, J.R., Carvalho, P.F., Liang, K.L., Goldstone, R.L. (2017). A dissociation between engagement and learning: Enthusiastic instructions fail to reliably improve performance on a memory task. PLoS ONE, 12(7): e0181775. doi: 10.1371/journal.pone.0181775
Despite widespread assertions that enthusiasm is an important quality of effective teaching, empirical research on the effect of enthusiasm on learning and memory is mixed and largely inconclusive. To help resolve these inconsistencies, we conducted a carefully-controlled laboratory experiment, investigating whether enthusiastic instructions for a memory task would improve recall accuracy. Scripted videos, either enthusiastic or neutral, were used to manipulate the delivery of task instructions. We also manipulated the sequence of learning items, replicating the spacing effect, a known cognitive technique for memory improvement. Although spaced study reliably improved test performance, we found no reliable effect of enthusiasm on memory performance across two experiments. We did, however, find that enthusiastic instructions caused participants to respond to more item prompts, leaving fewer test questions blank, an outcome typically associated with increased task motivation. We find no support for the popular claim that enthusiastic instruction will improve learning, although it may still improve engagement. This dissociation between motivation and learning is dis- cussed, as well as its implications for education and future research on student learning.
Download PDF of article
Goldstone, R. L., Kersten, A., & Carvalho, P. F. (2017). Categorization and Concepts. In J. Wixted (Ed.) Stevens’ Handbook of Experimental Psychology and Cognitive neuroscience, Fourth Edition, Volume Three: Language & Thought. New Jersey: Wiley. (pp. 275-317).
Concepts are the building blocks of thought. They are critically involved when we reason, make inferences, and try to generalize our previous experiences to new situations. Behind every word in every language lies a concept, although there are concepts, like the small plastic tubes attached to the ends of shoelaces, that we are familiar with and can think about even if we do not know that they are called aglets . Concepts are indispensable to human cognition because they take the “blooming, buzzing confusion” (James, 1890, p. 488) of disorganized sensory experiences and establish order through mental categories. These mental categories allow us to make sense of the world and predict how worldly entities will behave. We see, hear, interpret, remember, understand, and talk about our world through our concepts, and so it is worthy of reflection time to establish where concepts come from, how they work, and how they can best be learned and deployed to suit our cognitive needs.
download PDF of article
Marghetis, T., Goldstone, R. L., & Landy, D. (2017). Even when people are manipulating algebraic equations, they still associate numerical magnitude with space. Proceedings of the 39th Annual Conference of the Cognitive Science Society. (pp. 2675-2680). London, England: Cognitive Science Society.
The development of symbolic algebra transformed civilization. Since algebra is a recent cultural invention, however, algebraic reasoning must build on a foundation of more basic capacities. Past work suggests that spatial representations of number may be part of that foundation, but recent studies have failed to find relations between spatial-numerical associations and higher mathematical skills. One possible explanation of this failure is that spatial representations of number are not activated during complex mathematics. We tested this possibility by collecting dense behavioral recordings while participants manipulated equations. When interacting with an equation’s greatest [/least] number, participants’ movements were deflected upward [/downward] and rightward [/leftward]. This occurred even when the task was purely algebraic and could thus be solved without attending to magnitude (although the deflection was reduced). This is the first evidence that spatial representations of number are activated during algebra. Algebraic reasoning may require coordinating a variety of spatial processes.
Download PDF of article
Carvalho, P. F., & Goldstone, R. L. (2017). The most efficient sequence of study depends on the type of test. Proceedings of the 39th Annual Conference of the Cognitive Science Society. (pp. 198-203). London, England: Cognitive Science Society.
Previous research has shown that the sequence in which concepts are studied changes how well they are learned. In a series of experiments featuring naturalistic concepts (psychology concepts) and naïve learners, we extend previous research by showing that the sequence of study changes the representation the learner creates of the study materials. Interleaved study leads to the creation of relatively interrelated concepts that are represented by contrast to each other and based on discriminating properties. Blocked study, instead, leads to the creation of relatively isolated concepts that are represented in terms of their central and characteristic properties. The relative benefits of these representations depend on whether the test of conceptual knowledge requires contrastive or characteristic information. These results argue for the integrated investigation of the benefits of different sequences of study as depending on the characteristics of the study and testing situation as a whole.
Download PDF of paper
Lara-Dammer, F., Hofstadter, D. R., & Goldstone, R. L. (2017). A computer model of context dependent perception in a very simple world. Journal of Experimental & Theoretical Artificial Intelligence, 29:6, 1247-1282. DOI: 10.1080/0952813X.2017.1328463
We propose the foundations of a computer model of scientic discovery that takes into account certain psychological aspects of human observation of the world. To this end, we simulate two main components of such a system. The first is a dynamic microworld in which physical events take place, and the second is an observer that visually perceives entities and events in the microworld. For reason of space, this paper focuses only on the starting phase of discovery, which is the relatively simple visual inputs of objects and collisions.
Download PDF of article
Tullis, J. G., & Goldstone, R. L. (2017). Instruction in computer modeling can support broad application of complex systems knowledge. Frontiers in Education, 2:4, 1-18. doi: 10.3389/feduc.2017.00004
Learners often struggle to grasp the important, central principles of complex systems, which describe how interactions between individual agents can produce complex, aggre-gate-level patterns. Learners have even more difficulty transferring their understanding of these principles across superficially dissimilar instantiations of the principles. Here, we provide evidence that teaching high school students an agent-based modeling language can enable students to apply complex system principles across superficially different domains. We measured student performance on a complex systems assessment before and after 1 week training in how to program models using NetLogo (Wilensky, 1999a). Instruction in NetLogo helped two classes of high school students apply complex sys-tems principles to a broad array of phenomena not previously encountered. We argue that teaching an agent-based computational modeling language effectively combines the benefits of explicitly defining the abstract principles underlying agent-level interac-tions with the advantages of concretely grounding knowledge through interactions with agent-based models.
Download PDF of article
Meagher, B. J., Carvalho, P. F., Goldstone, R. L., & Nosofsky, R. M. (2017). Organized simultaneous displays facilitate learning of complex natural science categories. Psychonomic Bulletin & Review, DOI 10.3758/s13423-017-1251-6.
Subjects learned to classify images of rocks into the categories igneous, metamorphic, and sedimentary. In accord with the real-world structure of these categories, the to-beclassified rocks in the experiments had a dispersed similarity structure. Our central hypothesis was that learning of these complex categories would be improved through observational study of organized, simultaneous displays of the multiple rock tokens. In support of this hypothesis, a technique that included the presentation of the simultaneous displays during phases of the learning process yielded improved acquisition (Experiment 1) and generalization (Experiment 2) compared to methods that relied solely on sequential forms of study and testing. The technique appears to provide a good starting point for application of cognitive-psychology principles of effective category learning to the science classroom.
Download a PDF of this article
Carvalho, P. F., & Goldstone, R. L. (2017). The sequence of study changes what information is attended to, encoded and remembered during category learning. Journal of Experimental Psychology: Learning, Memory, and Cognition, 43, 1699-1719.
The sequence of study influences how we learn. Previous research has identified different sequences as potentially beneficial for learning in different contexts and with different materials. Here we investigate the mechanisms involved in inductive category learning that give rise to these sequencing effects. Across 3 experiments we show evidence that the sequence of study changes what information learners attend to during learning, what is encoded from the materials studied and, consequently, what is remembered from study. Interleaved study (alternating between presentation of 2 categories) leads to an attentional focus on properties that differ between successive items, leading to relatively better encoding and memory for item properties that discriminate between categories. Conversely, when learners study each category in a separate block (blocked study), learners encode relatively more strongly the characteristic features of the items, which may be the result of a strong attentional focus on sequential similarities. These results provide support for the sequential attention theory proposing that inductive category learning takes place through a process of sequential comparisons between the current and previous items. Different sequences of items change how attention is deployed depending on this basic process. Which sequence results in better or worse learning depends on the match between what is encoded and what is required at test.
Download a PDF of this article
Goldstone, R. L., & Day, S. (2017). Transfer of knowledge. In K. Peppler (Ed.) The SAGE Encyclopedia of Out-of-school Learning. Thousand Oaks, CA. (pp. 784-787).
Transfer of knowledge is the application of knowledge learned in one context to new, dissimilar problems or situations where the knowledge would be useful. Teachers, coaches, camp counselors, parents, and learners often have the experience of a learner showing apparent understanding when questioned about a topic in a way that closely matches how it was initially presented but showing almost no understanding when queried in a new context or with novel examples. This entry further explains the concept of knowledge transfer. It then discusses several different strategies used to support knowledge transfer.
Download a PDF of this article
Goldstone, R. L., Weitnauer, E., Ottmar, E., Marghetis, T., & Landy, D. H. (2016). Modeling Mathematical Reasoning as Trained Perception-Action Procedures. In R. Sottilare, A. Graesser, X. Hu, A. Olney, B. Nye, and A. Sinatra (Eds.) Design Recommendations for Intelligent Tutoring Systems: Volume 4 – Domain Modeling. Orlando, FL: U.S. Army Research Laboratory. (pp. 213-223).
We have observed that when people engage in algebraic reasoning, they often perceptually and spatially transform algebraic notations directly rather than first converting the notation to an internal, non spatial representation. We describe empirical evidence for spatial transformations, such as spatially compact grouping, transposition, spatially overlaid intermediate results, cancelling out, swapping, and splitting. This research has led us to understand domain models in mathematics as the deployment of trained and strategically crafted perceptual-motor processes working on grounded and strategically crafted notations. This approach to domain modeling has also motivated us to develop and assess an algebra tutoring system focused on helping students train their perception and action systems to coordinate with each other and formal mathematics. Overall, our laboratory and classroom investigations emphasize the interplay between explicit mathematical understandings and implicit perception action training as having a high potential payoff for making learning more efficient, robust, and broadly applicable.
Download PDF of paper
Tullis, J. G. & Goldstone, R. L. (2016). Comparison versus reminding. Cognitive Research: Principles and Implications, 1, 1-20, DOI 10.1186/s41235-016-0028-1.
Comparison and reminding have both been shown to support learning and transfer. Comparison is thought to support transfer because it allows learners to disregard non-matching features of superficially different episodes in order to abstract the essential structure of concepts. Remindings promote memory for the individual episodes and generalization because they prompt learners to retrieve earlier episodes during the encoding of later related episodes and to compare across episodes. Across three experiments, we compared the consequences of comparison and reminding on memory and transfer. Participants studied a sequence of related, but superficially different, proverb pairs. In the comparison condition, participants saw proverb pairs presented together and compared their meaning. In the reminding condition, participants viewed proverbs one at a time and retrieved any prior studied proverb that shared the same deep meaning as the current proverb. Experiment 1 revealed that participants in the reminding condition recalled more proverbs than those in the comparison condition. Experiment 2 showed that the mnemonic benefits of reminding persisted over a one-week retention interval. Finally, in Experiment 3, we examined the ability of participants to generalize their remembered information to new items in a task that required participants to identify unstudied proverbs that shared the samemeaning as studied proverbs. Comparison led to worse discrimination between proverbs related to studied proverbs and proverbs unrelated to studied proverbs than reminding. Reminding supported better memory for individual instances and transfer to new situations than comparison.
Download PDF version of paper
Marghetis, T., Landy, D., & Goldstone, R. L. (2016). Mastering algebra retrains the visual system to perceive hierarchical structure in equations. Cognitive Research: Principles and Implications, 1(25), 1-10, DOI 10.1186/s41235-016-0020-9.
Formal mathematics is a paragon of abstractness. It thus seems natural to assume that the mathematical expert should rely more on symbolic or conceptual processes, and less on perception and action. We argue instead that mathematical proficiency relies on perceptual systems that have been retrained to implement mathematical skills. Specifically, we investigated whether the visual system—in particular, object-based attention—is retrained so that parsing algebraic expressions and evaluating algebraic validity are accomplished by visual processing. Object-based attention occurs when the visual system organizes the world into discrete objects, which then guide the deployment of attention. One classic signature of object-based attention is better perceptual discrimination within, rather than between, visual objects. The current study reports that object-based attention occurs not only for simple shapes but also for symbolic mathematical elements within algebraic expressions—but only among individuals who have mastered the hierarchical syntax of algebra. Moreover, among these individuals, increased object-based attention within algebraic expressions is associated with a better ability to evaluate algebraic validity. These results suggest that, in mastering the rules of algebra, people retrain their visual system to represent and evaluate abstract mathematical structure. We thus argue that algebraic expertise involves the regimentation and reuse of evolutionarily ancient perceptual processes. Our findings implicate the visual system as central to learning and reasoning in mathematics, leading us to favor educational approaches to mathematics and related STEM fields that encourage students to adapt, not abandon, their use of perception.
Download a PDF version of this paper
Carvalho, P.F., Braithwaite, D.W., de Leeuw, J.R., Motz, B.A., & Goldstone, R.L. (2016). An in vivo study of self-regulated study sequencing in introductory psychology courses. PLoS ONE 11(3): e0152115.
Study sequence can have a profound influence on learning. In this study we investigated how students decide to sequence their study in a naturalistic context and whether their choices result in improved learning. In the study reported here, 2061 undergraduate students enrolled in an Introductory Psychology course completed an online homework tutorial on measures of central tendency, a topic relevant to an exam that counted towards their grades. One group of students was enabled to choose their own study sequence during the tutorial (Self-Regulated group), while the other group of students studied the same materials in sequences chosen by other students (Yoked group). Students who chose their sequence of study showed a clear tendency to block their study by concept, and this tendency was positively associated with subsequent exam performance. In the Yoked group, study sequence had no effect on exam performance. These results suggest that despite findings that blocked study is maladaptive when assigned by an experimenter, it may actually be adaptive when chosen by the learner in a naturalistic context.
Download PDF of article here