Multimodal Analysis for Identification and Segmentation of Moving-Sounding Objects

    Authors

    H. Izadinia; I. Saleemi;M. Shah

    Comments

    Authors: contact us about adding a copy of your work at STARS@ucf.edu

    Abbreviated Journal Title

    IEEE Trans. Multimedia

    Keywords

    Audio-visual analysis; audio-visual synchronization; canonical; correlation analysis; video segmentation; CANONICAL CORRELATION-ANALYSIS; FUSION; LOCALIZATION; TRACKING; Computer Science, Information Systems; Computer Science, Software; Engineering; Telecommunications

    Abstract

    In this paper, we propose a novel method that exploits correlation between audio-visual dynamics of a video to segment and localize objects that are the dominant source of audio. Our approach consists of a two-step spatiotemporal segmentation mechanism that relies on velocity and acceleration of moving objects as visual features. Each frame of the video is segmented into regions based on motion and appearance cues using the QuickShift algorithm, which are then clustered over time using K-means, so as to obtain a spatiotemporal video segmentation. The video is represented by motion features computed over individual segments. The Mel-Frequency Cepstral Coefficients (MFCC) of the audio signal, and their first order derivatives are exploited to represent audio. The proposed framework assumes there is a non-trivial correlation between these audio features and the velocity and acceleration of the moving and sounding objects. The canonical correlation analysis (CCA) is utilized to identify the moving objects which are most correlated to the audio signal. In addition to moving-sounding object identification, the same framework is also exploited to solve the problem of audio-video synchronization, and is used to aid interactive segmentation. We evaluate the performance of our proposed method on challenging videos. Our experiments demonstrate significant increase in performance over the state-of-the-art both qualitatively and quantitatively, and validate the feasibility and superiority of our approach.

    Journal Title

    Ieee Transactions on Multimedia

    Volume

    15

    Issue/Number

    2

    Publication Date

    1-1-2013

    Document Type

    Article

    Language

    English

    First Page

    378

    Last Page

    390

    WOS Identifier

    WOS:000313875500013

    ISSN

    1520-9210

    Share

    COinS