Title

Estimating Intrinsic Dimensionality Using The Multi-Criteria Decision Weighted Model And The Average Standard Estimator

Keywords

Average standard estimator; Business process optimization; Dimensionality estimation; Market analytics; Singular value decomposition

Abstract

Information retrieval today is much more challenging than traditional small document retrieval. The main difference is the importance of correlations between related concepts in complex data structures. As collections of data grow and contain more entries, they require more complex relationships, links, and groupings between individual entries. This paper introduces two novel methods for estimating data intrinsic dimensionality based on the singular value decomposition (SVD). The average standard estimator (ASE) and the multi-criteria decision weighted model are used to estimate matrix intrinsic dimensionality for large document collections. The multi-criteria weighted model calculates the sum of weighted values of matrix dimensions which demonstrated best performance using all possible dimensions [1]. ASE estimates the level of significance for singular values that resulted from the singular value decomposition. ASE assumes that those variables with deep relations have sufficient correlation and that only those relationships with high singular values are significant and should be maintained [1]. Experimental results indicate that ASE improves precision and relative relevance for MEDLINE document collection by 10.2% and 12.9% respectively compared to the percentage of variance dimensionality estimation. Results based on testing three document collections over all possible dimensions using selected performance measures indicate that ASE improved matrix intrinsic dimensionality estimation by including the effect of both singular values magnitude of decrease and random noise distracters. The multi-criteria weighted model with dimensionality reduction provides a more efficient implementation for information retrieval than using a full rank model. © 2010 Elsevier Inc. All rights reserved.

Publication Date

8-1-2010

Publication Title

Information Sciences

Volume

180

Issue

15

Number of Pages

2845-2855

Document Type

Article

Personal Identifier

scopus

DOI Link

https://doi.org/10.1016/j.ins.2010.04.006

Socpus ID

78049289000 (Scopus)

Source API URL

https://api.elsevier.com/content/abstract/scopus_id/78049289000

This document is currently not available here.

Share

COinS