Abstract

Knowledge distillation, as a popular compression technique, has been widely used to reduce deep neural network (DNN) size for a variety of applications. However, in recent years, some research had found its potential for improving deep neural network performance. This dissertation focuses on further exploring its power to facilitate accurate and reliable DNN training. First, I explored data-efficient method for blackbox knowledge distillation where the specifics of the DNN for distillation is inaccessible. I integrated active learning and mixup to obtain significant distillation performance gain with limited data. This work reveals the competence of knowledge distillation to facilitate large foundation model application. Next, I extended this work to solve a more challenging practical problem, i.e. COVID-19 infection prediction. Due to extremely limited data at the outbreak, it is very difficult to calibrate any existing epidemic model for practical prediction. I applied blackbox knowledge distillation with sequence mixup to distill a comprehensive physics-based simulation system. With the obtained distilled model, epidemic models are better calibrated to fit limited observation data and provide more accurate and reliable projection. This work validates that knowledge distillation can enhance DNN training for complex time series prediction with limited observation data. Next, I applied knowledge distillation to improve DNN reliability which reflects accurate model prediction confidence. Ensemble modeling and data augmentation had been blended to equip distillation process and obtain a reliable DNN. This work justifies that knowledge distillation can equip training for a more reliable DNN. Furthermore, this dissertation extended my knowledge distillation study to semantic segmentation tasks. The study started with investigation of semantic segmentation models, and then, proposed an approach of adaptive convolution to improve the heterogeneity of local convolution fields. The experiments had been carried out across different scales of segmentation benchmarks and justified that this approach outperforms existing state-of-the-art schemes and successfully boosts the performance of various backbone models. After this investigation study, semantic segmentation models had been calibrated with ensemble knowledge distillation which had been applied to solve image classification calibration. Stronger augmentation had been incorporated into distillation process. The experiments justify the effectiveness for semantic segmentation calibration.

Notes

If this is your thesis or dissertation, and want to learn how to access it or for more information about readership statistics, contact us at STARS@ucf.edu

Graduation Date

2023

Semester

Spring

Advisor

Wang, Liqiang

Degree

Doctor of Philosophy (Ph.D.)

College

College of Engineering and Computer Science

Department

Computer Science

Degree Program

Computer Science

Identifier

CFE0009626; DP0027656

URL

https://purls.library.ucf.edu/go/DP0027656

Language

English

Release Date

May 2023

Length of Campus-only Access

None

Access Status

Doctoral Dissertation (Open Access)

Download

Share

COinS