Knowledge distillation, as a popular compression technique, has been widely used to reduce deep neural network (DNN) size for a variety of applications. However, in recent years, some research had found its potential for improving deep neural network performance. This dissertation focuses on further exploring its power to facilitate accurate and reliable DNN training. First, I explored data-efficient method for blackbox knowledge distillation where the specifics of the DNN for distillation is inaccessible. I integrated active learning and mixup to obtain significant distillation performance gain with limited data. This work reveals the competence of knowledge distillation to facilitate large foundation model application. Next, I extended this work to solve a more challenging practical problem, i.e. COVID-19 infection prediction. Due to extremely limited data at the outbreak, it is very difficult to calibrate any existing epidemic model for practical prediction. I applied blackbox knowledge distillation with sequence mixup to distill a comprehensive physics-based simulation system. With the obtained distilled model, epidemic models are better calibrated to fit limited observation data and provide more accurate and reliable projection. This work validates that knowledge distillation can enhance DNN training for complex time series prediction with limited observation data. Next, I applied knowledge distillation to improve DNN reliability which reflects accurate model prediction confidence. Ensemble modeling and data augmentation had been blended to equip distillation process and obtain a reliable DNN. This work justifies that knowledge distillation can equip training for a more reliable DNN. Furthermore, this dissertation extended my knowledge distillation study to semantic segmentation tasks. The study started with investigation of semantic segmentation models, and then, proposed an approach of adaptive convolution to improve the heterogeneity of local convolution fields. The experiments had been carried out across different scales of segmentation benchmarks and justified that this approach outperforms existing state-of-the-art schemes and successfully boosts the performance of various backbone models. After this investigation study, semantic segmentation models had been calibrated with ensemble knowledge distillation which had been applied to solve image classification calibration. Stronger augmentation had been incorporated into distillation process. The experiments justify the effectiveness for semantic segmentation calibration.
If this is your thesis or dissertation, and want to learn how to access it or for more information about readership statistics, contact us at STARS@ucf.edu
Doctor of Philosophy (Ph.D.)
College of Engineering and Computer Science
Length of Campus-only Access
Doctoral Dissertation (Open Access)
Wang, Dongdong, "Improving Deep Neural Network Training with Knowledge Distillation" (2023). Electronic Theses and Dissertations, 2020-. 1689.