Abstract

Deep neural networks are quickly advancing and increasingly used in many applications; however, these networks are often extremely large and require computing and storage power beyond what is available in most embedded and sensor devices. For example, IoT (Internet of Things) devices lack powerful processors or graphical processing units (GPUs) that are commonly used in deep networks. Given the very large-scale deployment of such low power devices, it is desirable to design methods for efficient reduction of computational needs of neural networks. This can be done by reducing input data size or network sizes. Expectedly, such reduction comes at the cost of degraded performance. In this work, we examine how sparsifying the input to a neural network can significantly improve the performance of artificial neural networks (ANN) such as time delay neural networks (TDNN) and convolutional neural networks (CNNs). We show how TDNNs can be enhanced using a sparsifying transform layer that significantly improves learning time and forecasting performance for time series. We mathematically prove that the improvement is the result of sparsification of the input of a fully connected layer of a TDNN. Experiments with several datasets and transforms such as discrete cosine transform (DCT), discrete wavelet transform (DWT) and PCA (Principal Component Analysis) are used to show the improvement and the reason behind it. We also show that this improved performance can be traded off for network size reduction of a TDNN. Similarly, we show that the performance of reduced size CNNs can be improved for image classification when domain transforms are employed in the input. The improvement in CNN performance is found to be related to the better preservation of information when sparsifying transforms are used. We evaluate the proposed concept with low complexity CNNs and common datasets of Fashion MNIST and CIFAR. We constrain the size of CNNs in our tests to under 200K learnable parameters, as opposed to millions in deeper networks. We emphasize that finding optimal hyper parameters or network configurations is not the objective of this study; rather, we focus on studying the impact of projecting data to new domains on the performance of reduced size inputs and networks. It is shown that input size reduction of up to 75% is possible, without loss of classification accuracy in some cases.

Notes

If this is your thesis or dissertation, and want to learn how to access it or for more information about readership statistics, contact us at STARS@ucf.edu

Graduation Date

2021

Semester

Spring

Advisor

Mikhael, Wasfy

Degree

Doctor of Philosophy (Ph.D.)

College

College of Engineering and Computer Science

Department

Electrical and Computer Engineering

Degree Program

Computer Engineering

Format

application/pdf

Identifier

CFE0008928; DP0026207

URL

https://purls.library.ucf.edu/go/DP0026207

Language

English

Release Date

November 2021

Length of Campus-only Access

None

Access Status

Doctoral Dissertation (Open Access)

Download

Share

COinS