Srinivas Narayanan, Ph.D.

Project

Simulation Semantics: a Computational Framework to Study the Link Between Perception, Action, and Cognition

The vast majority of research in Computational Neuroscience has focused on the modeling of individual neurons or of low-level sensory and motor functions - and with good reason. There is clearly a large gap between what is known about neural functioning and architecture and higher cognitive processes like problem-solving and the ability to understand language. But these higher mental functions are crucial to our life and deficits can be as debilitating as any physical ailment. A better understanding of the neural basis of language and thought will have a profound social, scientific, and clinical impact.
There is accumulating evidence from different fields of inquiry that language and thought are dependent on embodied control processes related to perception, motor control and social cognition. This convergence of evidence from neuroscience, cognitive linguistics, and from neural computation points to embodied simulation as a potential link between perception, action and cognition.
Attempts to bridge the conceptual gap between the recent experimental results on neural functioning and models of cognitive phenomena will need a well developed theory to link computational processes and neural systems. My proposed research represents the first systematic evaluation and consolidation of these experimental results into a computational framework. It is the first phase in a long-term research project designed to develop a neurally-plausible model of language and high-level cognition.

Recommended Reading

Feldman, J. and S. Narayanan. 2004. "Embodied meaning in a neural theory of language." Brain and Language 89, 385-392.
Narayanan, S. 1999. "Moving right along: a computational model of metaphoric reasoning about events." Proceedings of the Sixteenth National Conference on Artificial Intelligence (AAAI '99)." AAAI Press, 121-127.
Narayanan, S. 1999. "Reasoning about actions in narrative understanding." Proceedings of the Sixteenth International Joint Conference on Artificial Intelligence (IJCAI 99), San Francisco: Morgan Kaufmann, 350-357.

Colloquium, 17.02.2009

Simulation in Cognition and Language Understanding

The biology of cognition and language understanding is several decades behind the levels achieved in perception and motor control. This arises partly from a lack of animal models for language, but there are also conceptual issues. Even now, much of the work on language and cognition is formulated in terms that make no contact with biology and are often demonstrably unbiological.
The convergence of results from the cognitive and neural sciences has facilitated the emergence of explicitly neural theories of language including our Neural Theory of Language (NTL) effort (http://www.icsi.berkeley.edu/NTL). One cornerstone of such theories is embodiment; the notion that language and cognition employ much of the same neural circuitry that is involved in perception, action, emotion, etc. This has been known computationally for decades and there is now a huge body of supporting literature using several methodologies. Our work focuses on building and testing detailed biologically plausible computational models of cognition and language understanding.

The presentation will focus on the central role of mental simulation in cognition. Work within the NTL project coupled with a variety of converging evidence from Cognitive Linguistics, Psychology and Neuroscience suggests that language understanding involves embodied enactment which we call "simulation semantics". Simulation semantics hypothesizes the mind as "simulating" the external world while functioning in it. The "simulation" takes sensor input about the state of the world (whether linguistic or perceptual) together with general knowledge and makes new inferences. Monitoring the state of the external world, drawing inferences, and acting, jointly constitute a dynamic ongoing interactive process.

I will report on a neurally plausible, computational realization of the simulation semantics hypothesis. This involves a novel network model of motor action that links behavioral and imaging results and provides a framework that uses sensory-motor imagination for real time inference. A cross-cultural system of metaphors project these inferences to abstract domains potentially explaining the widespread use of motion and manipulation terms in everyday reasoning about complex events, actions, and processes. The talk will outline the central computational model, its ability to tackle vexing problems in linguistics, and preliminary results from behavioral, fMRI imaging, and TMS experiments testing its biological predictions.