Estimating intrinsic dimensionality using the multi-criteria decision weighted model and the average standard estimator

    Authors

    T. Z. Ahram; P. McCauley-Bush;W. Karwowski

    Comments

    Authors: contact us about adding a copy of your work at STARS@ucf.edu

    Abbreviated Journal Title

    Inf. Sci.

    Keywords

    Dimensionality estimation; Average standard estimator; Singular value; decomposition; Market analytics; Business process optimization; LATENT SEMANTIC ANALYSIS; COMMON; Computer Science, Information Systems

    Abstract

    Information retrieval today is much more challenging than traditional small document retrieval. The main difference is the importance of correlations between related concepts in complex data structures. As collections of data grow and contain more entries, they require more complex relationships, links, and groupings between individual entries. This paper introduces two novel methods for estimating data intrinsic dimensionality based on the singular value decomposition (SVD). The average standard estimator (ASE) and the multi-criteria decision weighted model are used to estimate matrix intrinsic dimensionality for large document collections. The multi-criteria weighted model calculates the sum of weighted values of matrix dimensions which demonstrated best performance using all possible dimensions [1]. ASE estimates the level of significance for singular values that resulted from the singular value decomposition. ASE assumes that those variables with deep relations have sufficient correlation and that only those relationships with high singular values are significant and should be maintained [1]. Experimental results indicate that ASE improves precision and relative relevance for MEDLINE document collection by 10.2% and 12.9% respectively compared to the percentage of variance dimensionality estimation. Results based on testing three document collections over all possible dimensions using selected performance measures indicate that ASE improved matrix intrinsic dimensionality estimation by including the effect of both singular values magnitude of decrease and random noise distracters. The multi-criteria weighted model with dimensionality reduction provides a more efficient implementation for information retrieval than using a full rank model. (C) 2010 Elsevier Inc. All rights reserved.

    Journal Title

    Information Sciences

    Volume

    180

    Issue/Number

    15

    Publication Date

    1-1-2010

    Document Type

    Article

    Language

    English

    First Page

    2845

    Last Page

    2855

    WOS Identifier

    WOS:000279206900003

    ISSN

    0020-0255

    Share

    COinS