Maui: A Multimodal Affective User Interface


Affect recognition; Emotions; Intelligent interfaces; Interface agent


Human intelligence is being increasingly redefined to include the all-encompassing effect of emotions upon what used to be considered 'pure reason'. With the recent progress of research in computer vision, speech/prosody recognition, and bio-feedback, real-time recognition of affect will enhance human-computer interaction considerably, as well as assist further progress in the development of new emotion theories. In this article, we describe how affect, moods and emotions closely interact with cognition and how affect and emotion are the quintesseniial multimodal processes in humans. We then propose an adaptive system architecture designed to sense the user's emotional and affective states via three multimodal subsystems (V, K, A): namely (1) the Visual (from facial images and videos). (2) Kinesthetic (from autonomic nervous system (ANS) signals), and (3) Auditory (from speech). The results of the system sensing are then integrated into the multimodal perceived user's state. A multimodal anthropomorphic interface agent then adapts its interface by responding most appropriately to the current emotional states of its user, and provides intelligent multi-modal feedback to the user.

Publication Date


Publication Title

Proceedings of the ACM International Multimedia Conference and Exhibition

Number of Pages


Document Type

Article; Proceedings Paper

Personal Identifier


Socpus ID

0038376887 (Scopus)

Source API URL


This document is currently not available here.