Look-Up Table Unit Activation Function For Deep Convolutional Neural Networks

Abstract

Activation functions provide deep neural networks the non-linearity that is necessary to learn complex distributions. It is still inconclusive what is the optimal shape for the activation function. In this work, we introduce a novel type of activation function of which the shape is learned with network training. The proposed Look-up Table Unit (LuTU) stores a set of anchor points in a look-up table like structure, and the activation function is generated from the anchor points by either linear interpolation or smoothing with a single period cosine mask function. LuTU is in theory able to approximate any univariate function. By observing the learned shapes of LuTU, we further propose a Mixture of Gaussian Unit (MoGU) that can learn similar non-linear shapes with much fewer parameters. Finally, we use a multiple activation function fusion framework that combines multiple types of functions to achieve better performance. The inference complexity of multiple activation function fusion is constant with linear interpolation approximation. Our experiments on a synthetic dataset, ImageNet, and CIFAR-10 demonstrate that the proposed method outperforms traditional ReLU family activation functions. On the ImageNet dataset, our method achieves 1.47% and 1.0% higher accuracy on ResNet-18 and ResNet-34 models, respectively. With the proposed activation function, we can design a network that has the same performance as ResNet-34 but 8 fewer convolutional layers.

Publication Date

5-3-2018

Publication Title

Proceedings - 2018 IEEE Winter Conference on Applications of Computer Vision, WACV 2018

Volume

2018-January

Number of Pages

1225-1233

Document Type

Article; Proceedings Paper

Personal Identifier

scopus

DOI Link

https://doi.org/10.1109/WACV.2018.00139

Socpus ID

85051106064 (Scopus)

Source API URL

https://api.elsevier.com/content/abstract/scopus_id/85051106064

This document is currently not available here.

Share

COinS