Multimodal Perceptual Interface

The next generation of computers is expected to interact and communicate with users in a cooperative and natural manner while users engage in everyday activities. By being situated in users' environments, intelligent computers should not only have basic perceptual abilities but also use the knowledge of associations between different perceptual inputs. Toward this goal, we develop a multimodal perceptual interface in which a virtual agent is able to interact with users in real time, verbally describe what users are doing (action recognition) and what they are looking at (visual object recognition), and perform actions (action generation) according to spoken commands (speech understanding).