Title

Estimating intrinsic dimensionality using the multi-criteria decision weighted model and the average standard estimator

Authors

Authors

T. Z. Ahram; P. McCauley-Bush;W. Karwowski

Comments

Authors: contact us about adding a copy of your work at STARS@ucf.edu

Abbreviated Journal Title

Inf. Sci.

Keywords

Dimensionality estimation; Average standard estimator; Singular value; decomposition; Market analytics; Business process optimization; LATENT SEMANTIC ANALYSIS; COMMON; Computer Science, Information Systems

Abstract

Information retrieval today is much more challenging than traditional small document retrieval. The main difference is the importance of correlations between related concepts in complex data structures. As collections of data grow and contain more entries, they require more complex relationships, links, and groupings between individual entries. This paper introduces two novel methods for estimating data intrinsic dimensionality based on the singular value decomposition (SVD). The average standard estimator (ASE) and the multi-criteria decision weighted model are used to estimate matrix intrinsic dimensionality for large document collections. The multi-criteria weighted model calculates the sum of weighted values of matrix dimensions which demonstrated best performance using all possible dimensions [1]. ASE estimates the level of significance for singular values that resulted from the singular value decomposition. ASE assumes that those variables with deep relations have sufficient correlation and that only those relationships with high singular values are significant and should be maintained [1]. Experimental results indicate that ASE improves precision and relative relevance for MEDLINE document collection by 10.2% and 12.9% respectively compared to the percentage of variance dimensionality estimation. Results based on testing three document collections over all possible dimensions using selected performance measures indicate that ASE improved matrix intrinsic dimensionality estimation by including the effect of both singular values magnitude of decrease and random noise distracters. The multi-criteria weighted model with dimensionality reduction provides a more efficient implementation for information retrieval than using a full rank model. (C) 2010 Elsevier Inc. All rights reserved.

Journal Title

Information Sciences

Volume

180

Issue/Number

15

Publication Date

1-1-2010

Document Type

Article

Language

English

First Page

2845

Last Page

2855

WOS Identifier

WOS:000279206900003

ISSN

0020-0255

Share

COinS