Zeiler, M. D., Taylor, G. W., and Fergus, R. (2011). Adaptive deconvolutional
networks for mid and high level feature learning. InComputer Vision (ICCV),
2011 IEEE International Conference on, pages 2018–2025. IEEE.
<<END>> <<END>> <<END>>
<<START>> <<START>> <<START>>
A Survey of Model Compression and Acceleration for Deep Neural Networks
Yu Cheng, Duo Wang, Pan Zhou Member IEEE, and Tao Zhang Senior Member IEEE
Abstract—Deep convolutional neural networks (CNNs) have [2], [3]. It is also very time-consuming to train such a model
recently achieved great success in many visual recognition tasks. to get reasonable performance. In architectures that rely only However, existing deep neural network models are computation- on fully-connected layers, the number of parameters can grow ally expensive and memory intensive, hindering their deployment
in devices with low memory resources or in applications with to billions [4].
strict latency requirements. Therefore, a natural thought is to As larger neural networks with more layers and nodes
without significantly decreasing the model performance. During becomes critical, especially for some real-time applications the past few years, tremendous progress has been made in such as online learning and incremental learning. In addi- this area. In this paper, we survey the recent advanced tech-
niques for compacting and accelerating CNNs model developed. tion, recent years witnessed significant progress in virtual
These techniques are roughly categorized into four schemes: reality, augmented reality, and smart wearable devices, cre-
parameter pruning and sharing, low-rank factorization, trans- ating unprecedented opportunities for researchers to tackle
ferred/compact convolutional filters, and knowledge distillation. fundamental challenges in deploying deep learning systems to Methods of parameter pruning and sharing will be described at portable devices with limited resources (e.g. memory, CPU, the beginning, after that the other techniques will be introduced.
For each scheme, we provide insightful analysis regarding the energy, bandwidth). Efficient deep learning methods can have
performance, related applications, advantages, and drawbacks significant impacts on distributed systems, embedded devices,
etc. Then we will go through a few very recent additional and FPGA for Artificial Intelligence. For example, the ResNet-
successful methods, for example, dynamic capacity networks and 50 [5] with 50 convolutional layers needs over 95MB memory stochastic depths networks. After that, we survey the evaluation for storage and over 3.8 billion floating number multiplications matrix, the main datasets used for evaluating the model per-
formance and recent benchmarking efforts. Finally, we conclude when processing an image. After discarding some redundant
this paper, discuss remaining challenges and possible directions weights, the network still works as usual but saves more than
on this topic. 75% of parameters and 50% computational time. For devices
Index Terms—Deep Learning, Convolutional Neural Networks, like cell phones and FPGAs with only several megabyte
Model Compression and Acceleration, resources, how to compact the models used on them is also
important.
Achieving these goal calls for joint solutions from many
I. INTRODUCTION
disciplines, including but not limited to machine learning, op-
In recent years, deep neural networks have recently received timization, computer architecture, data compression, indexing,
lots of attention, been applied to different applications and and hardware design. In this paper, we review recent works
achieved dramatic accuracy improvements in many tasks. on compressing and accelerating deep neural networks, which
These works rely on deep networks with millions or even attracted a lot of attention from the deep learning community
billions of parameters, and the availability of GPUs with and already achieved lots of progress in the past years.
very high computation capability plays a key role in their We classify these approaches into four categories: pa-
success. For example, the work by Krizhevskyet al.[1] rameter pruning and sharing, low-rank factorization, trans-
achieved breakthrough results in the 2012 ImageNet Challenge ferred/compact convolutional filters, and knowledge distil-
using a network containing 60 million parameters with five lation. The parameter pruning and sharing based methods
convolutional layers and three fully-connected layers. Usually, explore the redundancy in the model parameters and try to
it takes two to three days to train the whole model on remove the redundant and uncritical ones. Low-rank factor-
ImagetNet dataset with a NVIDIA K40 machine. Another ization based techniques use matrix/tensor decomposition to
example is the top face verification results on the Labeled estimate the informative parameters of the deep CNNs. The
Faces in the Wild (LFW) dataset were obtained with networks approaches based on transferred/compact convolutional filters
containing hundreds of millions of parameters, using a mix design special structural convolutional filters to reduce the
of convolutional, locally-connected, and fully-connected layers parameter space and save storage/computation. The knowledge
distillation methods learn a distilled model and train a more Yu Cheng is a Researcher from Microsoft AI & Research, One Microsoft
Way, Redmond, WA 98052, USA. compact neural network to reproduce the output of a larger
Duo Wang and Tao Zhang are with the Department of Automation, network.
Tsinghua University, Beijing 100084, China. In Table I, we briefly summarize these four types of Pan Zhou is with the School of Electronic Information and Communi- methods. Generally, the parameter pruning & sharing, low- cations, Huazhong University of Science and Technology, Wuhan 430074,
China. rank factorization and knowledge distillation approaches can IEEE SIGNAL PROCESSING MAGAZINE, SPECIAL ISSUE ON DEEP LEARNING FOR IMAGE UNDERSTANDING (ARXIV EXTENDED VERSION) 2
TABLE I
<<TABLE>>
be used in DNN models with fully connected layers and
convolutional layers, achieving comparable performances. On
the other hand, methods using transferred/compact filters are
designed for models with convolutional layers only. Low-rank
factorization and transfered/compact filters based approaches
provide an end-to-end pipeline and can be easily implemented
in CPU/GPU environment, which is straightforward. while
parameter pruning & sharing use different methods such as
vector quantization, binary coding and sparse constraints to
perform the task. Generally it will take several steps to achieve
the goal.
<<FIGURE>>
Fig. 1. The three-stage compression method proposed in [10]: pruning, Regarding the training protocols, models based on param- quantization and encoding. The input is the original model and the output
eter pruning/sharing low-rank factorization can be extracted is the compression model.
from pre-trained ones or trained from scratch. While the
transferred/compact filter and knowledge distillation models
can only support train from scratch. These methods are inde- memory usage and float point operations with little loss in
pendently designed and complement each other. For example, classification accuracy.
transferred layers and parameter pruning & sharing can be The method proposed in [10] quantized the link weights
used together, and model quantization & binarization can be using weight sharing and then applied Huffman coding to the
used together with low-rank approximations to achieve further quantized weights as well as the codebook to further reduce
speedup. We will describe the details of each theme, their the rate. As shown in Figure 1, it started by learning the con-
properties, strengths and drawbacks in the following sections. nectivity via normal network training, followed by pruning the
small-weight connections. Finally, the network was retrained
to learn the final weights for the remaining sparse connections.
II. PARAMETER PRUNING AND SHARING
This work achieved the state-of-art performance among allEarly works showed that network pruning is effective in parameter quantization based methods. It was shown in [11] reducing the network complexity and addressing the over- that Hessian weight could be used to measure the importancefitting problem [6]. After that researcher found pruning orig- of network parameters, and proposed to minimize Hessian-inally introduced to reduce the structure in neural networks weighted quantization errors in average for clustering networkand hence improve generalization, it has been widely studied parameters.to compress DNN models, trying to remove parameters which In the extreme case of the 1-bit representation of eachare not crucial to the model performance. These techniques can weight, that is binary weight neural networks. There arebe further classified into three sub-categories: quantization and many works that directly train CNNs with binary weights, forbinarization, parameter sharing, and structural matrix. instance, BinaryConnect [12], BinaryNet [13] and XNORNet-
works [14]. The main idea is to directly learn binary weights orA. Quantization and Binarization activation during the model training. The systematic study in
Network quantization compresses the original network by [15] showed that networks trained with back propagation could
reducing the number of bits required to represent each weight. be resilient to specific weight distortions, including binary
Gonget al.[6] and Wu et al. [7] appliedk-means scalar weights.
quantization to the parameter values. Vanhouckeet al.[8] Drawbacks: the accuracy of the binary nets is significantly
showed that 8-bit quantization of the parameters can result lowered when dealing with large CNNs such as GoogleNet.
in significant speed-up with minimal loss of accuracy. The Another drawback of such binary nets is that existing bina-
work in [9] used 16-bit fixed-point representation in stochastic rization schemes are based on simple matrix approximations
rounding based CNN training, which significantly reduced and ignore the effect of binarization on the accuracy loss. IEEE SIGNAL PROCESSING MAGAZINE, SPECIAL ISSUE ON DEEP LEARNING FOR IMAGE UNDERSTANDING (ARXIV EXTENDED VERSION) 3
To address this issue, the work in [16] proposed a proximal connected layers, which is often the bottleneck in terms of
Newton algorithm with diagonal Hessian approximation that memory consumption. These network layers use the nonlinear
directly minimizes the loss with respect to the binary weights. transformsf(x;M) =(Mx), where()is an element-wise
The work in [17] reduced the time on float point multiplication nonlinear operator,xis the input vector, andMis themn
in the training stage by stochastically binarizing weights and matrix of parameters [29]. WhenMis a large general dense
converting multiplications in the hidden state computation to matrix, the cost of storingmnparameters and computing
significant changes. matrix-vector products inO(mn)time. Thus, an intuitive
way to prune parameters is to imposexas a parameterizedB. Pruning and Sharing structural matrix. Anmn matrix that can be described
Network pruning and sharing has been used both to reduce using much fewer parameters thanmnis called a structured
network complexity and to address the over-fitting issue. An matrix. Typically, the structure should not only reduce the
early approach to pruning was the Biased Weight Decay memory cost, but also dramatically accelerate the inference
[18]. The Optimal Brain Damage [19] and the Optimal Brain and training stage via fast matrix-vector multiplication and
Surgeon [20] methods reduced the number of connections gradient computations.
based on the Hessian of the loss function, and their work sug- Following this direction, the work in [30], [31] proposed a
gested that such pruning gave higher accuracy than magnitude- simple and efficient approach based on circulant projections,
while maintaining competitive error rates. Given a vectorr=based pruning such as the weight decay method. The training procedure of those methods followed the way training from <<FORMULA>>, a circulant matrix R^2 R^dxd is defined
as: <<FORMULA>>
scratch manner. A recent trend in this direction is to prune redundant, <<FORMULA>> non-informative weights in a pre-trained CNN model. For <<FORMULA>>
example, Srinivas and Babu [21] explored the redundancy <<FORMULA>> among neurons, and proposed a data-free pruning method to
remove redundant neurons. Hanet al.[22] proposed to reduce <<FORMULA>>
the total number of parameters and operations in the entire thus the memory cost becomesO(d)instead of O(d^2) network. Chenet al.[23] proposed a HashedNets model that This circulant structure also enables the use of Fast Fourier used a low-cost hash function to group weights into hash Transform (FFT) to speed up the computation. Given ad-buckets for parameter sharing. The deep compression method dimensional vectorr, the above 1-layer circulant neural net-in [10] removed the redundant connections and quantized the work in Eq. 1 has time complexity ofO(dlogd).weights, and then used Huffman coding to encode the quan- In [32], a novel Adaptive Fastfood transform was introducedtized weights. In [24], a simple regularization method based to reparameterize the matrix-vector multiplication of fully on soft weight-sharing was proposed, which included both connected layers. The Adaptive Fast food transform matrix quantization and pruning in one simple (re-)training procedure. R2Rnd was defined as:The above pruning schemes typically produce connections
pruning in CNNs. <<FORMULA>> (2)
There is also growing interest in training compact CNNs whereS,GandBare random diagonal matrices. 2
with sparsity constraints. Those sparsity constraints are typ- <<FORMULA>> is a random permutation matrix, and H denotes
ically introduced in the optimization problem asl0 orl1 - the Walsh-Hadamard matrix. Reparameterizing a fully con-
norm regularizers. The work in [25] imposed group sparsity nected layer with d inputs and n outputs using the Adaptive
constraint on the convolutional filters to achieve structured Fast food transform reduces the storage and the computational
brain Damage, i.e., pruning entries of the convolution kernels costs from O(n^d) to O(n) and from O(n^d) to O(n*log(d)),
in a group-wise fashion. In [26], a group-sparse regularizer respectively.
on neurons was introduced during the training stage to learn The work in [29] showed the effectiveness of the new
compact CNNs with reduced filters. Wenet al.[27] added a notion of parsimony in the theory of structured matrices. Their
structured sparsity regularizer on each layer to reduce trivial proposed method can be extended to various other structured
filters, channels or even layers. In the filter-level pruning, all matrix classes, including block and multi-level Toeplitz-like
the above works used l2-norm regularizers. The work in [28] [33] matrices related to multi-dimensional convolution [34].
usedl1 -norm to select and prune unimportant filters. Following this idea, [35] proposed a general structured effi-
Drawbacks: there are some potential issues of the pruning cient linear layer for CNNs.
and sharing. First, pruning with l1 or l2 regularization requires Drawbacks: one problem of this kind of approaches is that
more iterations to converge than general. In addition, all the structural constraint will hurt the performance since the
pruning criteria require manual setup of sensitivity for layers, constraint might bring bias to the model. On the other hand,
which demands fine-tuning of the parameters and could be how to find a proper structural matrix is difficult. There is no
cumbersome for some applications. theoretical way to derive it out.
C. Designing Structural Matrix
III. LOW-RANK FACTORIZATION AND SPARSITY
In architectures that contain fully-connected layers, it is Convolution operations contribute the bulk of most com-
critical to explore this redundancy of parameters in fully- putations in deep CNNs, thus reducing the convolution layer IEEE SIGNAL PROCESSING MAGAZINE, SPECIAL ISSUE ON DEEP LEARNING FOR IMAGE UNDERSTANDING (ARXIV EXTENDED VERSION) 4
TABLE II
COMPARISONS BETWEEN THE LOW -RANK MODELS AND THEIR BASELINES
ON ILSVRC-2012.
<<TABLE>>
<<FIGURE>>
Fig. 2. A typical framework of the low-rank regularization method. The left
is the original convolutional layer and the right is the low-rank constraint
convolutional layer with rank-K.
would improve the compression rate as well as the overall
speedup. For the convolution kernels, it can be viewed as a
4D tensor. Ideas based on tensor decomposition is derived by For instance, Mishaet al.[41] reduced the number of dynamic
the intuition that there is a significant amount of redundancy parameters in deep models using the low-rank method. [42]
in the 4D tensor, which is a particularly promising way to explored a low-rank matrix factorization of the final weight
remove the redundancy. Regarding the fully-connected layer, layer in a DNN for acoustic modeling. In [3], Luet al.adopted
it can be view as a 2D matrix and the low-rankness can also truncated SVD (singular value decomposition) to decompsite
help. the fully connected layer for designing compact multi-task
It has been a long time for using low-rank filters to acceler- deep learning architectures.
ate convolution, for example, high dimensional DCT (discrete Drawbacks: low-rank approaches are straightforward for
cosine transform) and wavelet systems using tensor products model compression and acceleration. The idea complements
to be constructed from 1D DCT transform and 1D wavelets recent advances in deep learning, such as dropout, recti-
respectively. Learning separable 1D filters was introduced fied units and maxout. However, the implementation is not
by Rigamontiet al.[36], following the dictionary learning that easy since it involves decomposition operation, which
idea. Regarding some simple DNN models, a few low-rank is computationally expensive. Another issue is that current
approximation and clustering schemes for the convolutional methods perform low-rank approximation layer by layer, and
kernels were proposed in [37]. They achieved 2speedup thus cannot perform global parameter compression, which
for a single convolutional layer with 1% drop in classification is important as different layers hold different information.
accuracy. The work in [38] proposed using different tensor Finally, factorization requires extensive model retraining to
decomposition schemes, reporting a 4.5speedup with 1% achieve convergence when compared to the original model.
drop in accuracy in text recognition.
The low-rank approximation was done layer by layer. The IV. T RANSFERRED /COMPACT CONVOLUTIONAL FILTERS
parameters of one layer were fixed after it was done, and the CNNs are parameter efficient due to exploring the trans-layers above were fine-tuned based on a reconstruction error lation invariant property of the representations to the input criterion. These are typical low-rank methods for compressing image, which is the key to the success of training very deep2D convolutional layers, which is described in Figure 2. Fol- models without severe over-fitting. Although a strong theory lowing this direction, Canonical Polyadic (CP) decomposition is currently missing, a large number of empirical evidenceof was proposed for the kernel tensors in [39]. Their work support the notion that both the translation invariant property used nonlinear least squares to compute the CP decomposition. and the convolutional weight sharing are important for good In [40], a new algorithm for computing the low-rank tensor predictive performance. The idea of using transferred convolu- decomposition for training low-rank constrained CNNs from tional filters to compress CNN models is motivated by recent scratch were proposed. It used Batch Normalization (BN) to works in [43], which introduced the equivariant group theory.transform the activation of the internal hidden units. In general, Letxbe an input,()be a network or layer and T() be the both the CP and the BN decomposition schemes in [40] (BN transform matrix. The concept of equivalence is defined as:Low-rank) can be used to train CNNs from scratch. However,
there are few differences between them. For example, finding <<FORMULA>> (3)
the best low-rank approximation in CP decomposition is an ill-
posed problem, and the best rank-K (K is the rank number) indicating that transforming the input x by the transform T()
approximation may not exist sometimes. While for the BN and then passing it through the network or layer () should
scheme, the decomposition always exists. We perform a simple give the same result as first mapping x through the network
comparison of both methods shown in Table II. The actual and then transforming the representation. Note that in Eq.
speedup and the compression rates are used to measure their (10), the transforms <<T()>> and <<T_0()>> are not necessarily the
performances. same as they operate on different objects. According to this
As we mentioned before, the fully connected layers can theory, it is reasonable applying transform to layers or filters
be viewed as a 2D matrix and thus the above mentioned () to compress the whole network models. From empirical
methods can also be applied there. There are several classical observation, deep CNNs also benefit from using a large set of
works on exploiting low-rankness in fully connected layers. convolutional filters by applying certain transformT()to a IEEE SIGNAL PROCESSING MAGAZINE, SPECIAL ISSUE ON DEEP LEARNING FOR IMAGE UNDERSTANDING (ARXIV EXTENDED VERSION) 5
small set of base filters since it acts as a regularizer for the TABLE III
model. A SIMPLE COMPARISON OF DIFFERENT APPROACHES ON CIFAR-10 AND
Following this direction, there are many recent reworks
proposed to build a convolutional layer from a set of base <<TABLE>>
filters [43]–[46]. What they have in common is that the
transform T() lies in the family of functions that only operate
in the spatial domain of the convolutional filters. For example,
the work in [45] found that the lower convolution layers of
CNNs learned redundant filters to extract both positive and
negative phase information of an input signal, and definedT() Drawbacks: there are few issues to be addressed for ap-to be the simple negation function: proaches that apply transform constraints to convolutional fil-
<<FORMULA>> (4)
ters. First, these methods can achieve competitive performance x for wide/flat architectures (like VGGNet) but not thin/deepwhereWx is the basis convolutional filter andW is the filter x ones (like GoogleNet, Residual Net). Secondly, the transferconsisting of the shifts whose activation is opposite to that assumptions sometimes are too strong to guide the learning,ofWx and selected after max-pooling operation. By doing making the results unstable in some cases.this, the work in [45] can easily achieve 2compression Using a compact filter for convolution can directly reducerate on all the convolutional layers. It is also shown that the the computation cost. The key idea is to replace the loosenegation transform acts as a strong regularizer to improve and over-parametric filters with compact blocks to improve the classification accuracy. The intuition is that the learning the speed, which significantly accelerate CNNs on severalalgorithm with pair-wise positive-negative constraint can lead benchmarks. Decomposing33convolution into two11to useful convolutional filters instead of redundant ones. convolutions was used in [48], which achieved significantIn [46], it was observed that magnitudes of the responses acceleration on object recognition. SqueezeNet [49] was pro-from convolutional kernels had a wide diversity of pattern posed to replace33convolution with11convolu-representations in the network, and it was not proper to discard tion, which created a compact neural network with about 50weaker signals with a single threshold. Thus a multi-bias non- fewer parameters and comparable accuracy when compared tolinearity activation function was proposed to generates more AlexNet.patterns in the feature space at low computational cost. The
transformT()was define as:
<<FORMULA>> (5)
V. KNOWLEDGE DISTILLATION
To the best of our knowledge, exploiting knowledge transfer
where were the multi-bias factors. The work in [47] con- (KT) to compress model was first proposed by Caruanaet
side red a combination of rotation by a multiple of 90 and al.[50]. They trained a compressed/ensemble model of strong
horizontal/vertical flipping with: classifiers with pseudo-data labeled, and reproduced the output
of the original larger network. But the work is limited to
<<FORMULA>> (6)
shallow models. The idea has been recently adopted in [51]
whereWT was the transformation matrix which rotated the as knowledge distillation (KD) to compress deep and wide
original filters with angle2 f90;180;270g. In [43], the networks into shallower ones, where the compressed model
transform was generalized to any angle learned from data, and mimicked the function learned by the complex model. The
was directly obtained from data. Both works [47] and [43] main idea of KD based approaches is to shift knowledge from
can achieve good classification performance. a large teacher model into a small one by learning the class
The work in [44] definedT()as the set of translation distributions output via softmax.
functions applied to 2D filters: The work in [52] introduced a KD compression framework,
which eased the training of deep networks by following a
<<FORMULA>> (7)
student-teacher paradigm, in which the student was penalized
whereT(;x;y)denoted the translation of the first operand by according to a softened version of the teacher’s output. The
(x;y)along its spatial dimensions, with proper zero padding framework compressed an ensemble of teacher networks into
at borders to maintain the shape. The proposed framework a student network of similar depth. The student was trained
can be used to 1) improve the classification accuracy as a to predict the output and the classification labels. Despite
regularized version of maxout networks, and 2) to achieve its simplicity, KD demonstrates promising results in various
parameter efficiency by flexibly varying their architectures to image classification tasks. The work in [53] aimed to address
compress networks. the network compression problem by taking advantage of
Table III briefly compares the performance of different depth neural networks. It proposed an approach to train thin
methods with transferred convolutional filters, using VGGNet but deep networks, called FitNets, to compress wide and
(16 layers) as the baseline model. The results are reported shallower (but still deep) networks. The method was extended
on CIFAR-10 and CIFAR-100 datasets with Top-5 error. It is the idea to allow for thinner and deeper student models. In
observed that they can achieve reduction in parameters with order to learn from the intermediate representations of teacher
little or no drop in classification accuracy. network, FitNet made the student mimic the full feature maps
of the teacher. However, such assumptions are too strict since layer with global average pooling [44], [62]. Network architec-
the capacities of teacher and student may differ greatly. ture such as GoogleNet or Network in Network, can achieve
All the above approaches are validated on MNIST, CIFAR- state-of-the-art results on several benchmarks by adopting
10, CIFAR-100, SVHN and AFLW benchmark datasets, and this idea. However, these architectures have not been fully
experimental results show that these methods match or outper- optimized the utilization of the computing resources inside
form the teacher’s performance, while requiring notably fewer the network. This problem was noted by Szegedyet al.[62]
parameters and multiplications. and motivated them to increase the depth and width of the
There are several extension along this direction of dis- network while keeping the computational budget constant.
tillation knowledge. The work in [54] trained a parametric The work in [63] targeted the Residual Network based
student model to approximate a Monte Carlo teacher. The model with a spatially varying computation time, called
proposed framework used online training, and used deep stochastic depth, which enabled the seemingly contradictory
neural networks for the student model. Different from previous setup to train short networks and used deep networks at test
works which represented the knowledge using the soften label time. It started with very deep networks, while during training,
probabilities, [55] represented the knowledge by using the for each mini-batch, randomly dropped a subset of layers
neurons in the higher hidden layer, which preserved as much and bypassed them with the identity function. Following this
information as the label probabilities, but are more compact. direction, thew work in [64] proposed a pyramidal residual
The work in [56] accelerated the experimentation process by networks with stochastic depth. In [65], Wuet al.proposed
instantaneously transferring the knowledge from a previous an approach that learns to dynamically choose which layers
network to each new deeper or wider network. The techniques of a deep network to execute during inference so as to best
are based on the concept of function-preserving transfor- reduce total computation. Veitet al.exploited convolutional
mations between neural network specifications. Zagoruyko networks with adaptive inference graphs to adaptively define
et al.[57] proposed Attention Transfer (AT) to relax the their network topology conditioned on the input image [66].
assumption of FitNet. They transferred the attention maps that Other approaches to reduce the convolutional overheads in-are summaries of the full activations. clude using FFT based convolutions [67] and fast convolutionDrawbacks: KD-based approaches can make deeper models using the Winograd algorithm [68]. Zhaiet al.[69] proposed athinner and help significantly reduce the computational cost. strategy call stochastic spatial sampling pooling, which speed-However, there are a few disadvantages. One of those is that up the pooling operations by a more general stochastic version.KD can only be applied to classification tasks with softmax Saeedanet al.presented a novel pooling layer for convolu-loss function, which hinders its usage. Another drawback is tional neural networks termed detail-preserving pooling (DPP),the model assumptions sometimes are too strict to make the based on the idea of inverse bilateral filters [70]. Those worksperformance competitive with other type of approaches. only aim to speed up the computation but not reduce the
memory storage.
VI. OTHER TYPES OF APPROACHES
We first summarize the works utilizing attention-based
methods. Note that attention-based mechanism [58] can reduce
VII. BENCHMARKS , EVALUATION AND DATABASES
computations significantly by learning to selectively focus or In the past five years the deep learning community had“attend” to a few, task-relevant input regions. The work in made great efforts in benchmark models. One of the most[59] introduced the dynamic capacity network (DCN) that well-known model used in compression and acceleration forcombined two types of modules: the small sub-networks with CNNs is Alexnet [1], which has been occasionally usedlow capacity, and the large ones with high capacity. The low- for assessing the performance of compression. Other popularcapacity sub-networks were active on the whole input to first standard models include LeNets [71], All-CNN-nets [72] andfind the task-relevant areas, and then the attention mechanism many others. LeNet-300-100 is a fully connected networkwas used to direct the high-capacity sub-networks to focus on with two hidden layers, with 300 and 100 neurons each.the task-relevant regions. By dong this, the size of the CNNs LeNet-5 is a convolutional network that has two convolutionalmodel has been significantly reduced. layers and two fully connected layers. Recently, more andFollowing this direction, the work in [60] introduced the more state-of-the-art architectures are used as baseline modelsconditional computation idea, which only computes the gra- in many works, including network in networks (NIN) [73],dient for some important neurons. It proposed a sparsely- VGG nets [74] and residual networks (ResNet) [75]. Table IVgated mixture-of-experts Layer (MoE). The MoE module summarizes the baseline models commonly used in severalconsisted of a number of experts, each a simple feed-forward typical compression methods.neural network, and a trainable gating network that selected
a sparse combination of the experts to process each input. In The standard criteria to measure the quality of model
[61], dynamic deep neural networks (D2NN) were introduced, compression and acceleration are the compression and the
which were a type of feed-forward deep neural network that speedup rates. Assume thatais the number of the parameters
selected and executed a subset of D2NN neurons based on the in the original model Manda is that of the compressed
input. model M , then the compression rate (M;M ) of M over
There have been other attempts to reduce the number of Mis aparameters of neural networks by replacing the fully connected (M;M ) = : (8)a IEEE SIGNAL PROCESSING MAGAZINE, SPECIAL ISSUE ON DEEP LEARNING FOR IMAGE UNDERSTANDING (ARXIV EXTENDED VERSION) 7
TABLE IV or low rank factorization based methods. If you need
SUMMARIZATION OF BASELINE MODELS USED IN DIFFERENT end-to-end solutions for your problem, the low rank REPRESENTATIVE WORKS OF NETWORK COMPRESSION . and transferred convolutional filters approaches could be
considered.
For applications in some specific domains, methods with low-rank factorization [40] human prior (like the transferred convolutional filters, Network in network [73] low-rank factorization [40]
<<TABLE>> structural matrix) sometimes have benefits. For example,
when doing medical images classification, transferred Residual networks [75] compact filters [49], stochastic depth [63] convolutional filters could work well as medical images parameter sharing [24]
(like organ) do have the rotation transformation property.
Usually the approaches of pruning & sharing could give parameter pruning [20], [22] reasonable compression rate while not hurt the accuracy.
Thus for applications which requires stable model accu-
Another widely used measurement is the index space saving racy, it is better to utilize pruning & sharing.
defined in several papers [30], [35] as If your problem involves small/medium size datasets, you
can try the knowledge distillation approaches. The com-aa
<<FORMULA>> (9) pressed student model can take the benefit of transferring a knowledge from teacher model, making it robust datasets
where a and a are the number of the dimension of the index which are not large.
space in the original model and that of the compressed model, As we mentioned before, techniques of the four groups
respectively. are orthogonal. It is reasonable to combine two or three
Similarly, given the running timesofMands ofM , of them to maximize the performance. For some spe-
the speedup rate <<FORMULA>> is defined as: cific applications, like object detection, which requires
s both convolutional and fully connected layers, you can
<<FORMULA>> (10)
compress the convolutional layers with low rank based
Most work used the average training time per epoch to measure method and the fully connected layers with a pruning
the running time, while in [30], [35], the average testing time technique.
was used. Generally, the compression rate and speedup rate B. Technique Challengesare highly correlated, as smaller models often results in faster
computation for both the training and the testing stages. Techniques for deep model compression and acceleration
Good compression methods are expected to achieve almost are still in the early stage and the following challenges still
the same performance as the original model with much smaller need to be addressed.
parameters and less computational time. However, for different Most of the current state-of-the-art approaches are built
applications with different CNN designs, the relation between on well-designed CNN models, which have limited free-
parameter size and computational time may be different. dom to change the configuration (e.g., network structural,
For example, it is observed that for deep CNNs with fully hyper-parameters). To handle more complicated tasks,
connected layers, most of the parameters are in the fully it should provide more plausible ways to configure the
connected layers; while for image classification tasks, float compressed models.
point operations are mainly in the first few convolutional layers Pruning is an effective way to compress and acceler-
since each filter is convolved with the whole image, which is ate CNNs. The current pruning techniques are mostly
usually very large at the beginning. Thus compression and designed to eliminate connections between neurons. On
acceleration of the network should focus on different type of the other hand, pruning channel can directly reduce the
layers for different applications. feature map width and shrink the model into a thinner
one. It is efficient but also challenging because removing
VIII. D ISCUSSION AND CHALLENGES channels might dramatically change the input of the
following layer.In this paper, we summarized recent efforts on compressing
and accelerating deep neural networks (DNNs). Here we dis- As we mentioned before, methods of structural matrix
and transferred convolutional filters impose prior humancuss more details about how to choose different compression knowledge to the model, which could significantly affectapproaches, and possible challenges/solutions on this area. the performance and stability. It is critical to investigate
how to control the impact of those prior knowledge.A. General Suggestions The methods of knowledge distillation provide many ben-
There is no golden rule to measure which approach is the efits such as directly accelerating model without special
best. How to choose the proper method is really depending hardware or implementations. It is still worthy developing
on the applications and requirements. Here are some general KD-based approaches and exploring how to improve their
guidance we can provide: performances.
If the applications need compacted models from pre- Hardware constraints in various of small platforms (e.g.,
trained models, you can choose either pruning & sharing mobile, robotic, self-driving car) are still a major problem IEEE SIGNAL PROCESSING MAGAZINE, SPECIAL ISSUE ON DEEP LEARNING FOR IMAGE UNDERSTANDING (ARXIV EXTENDED VERSION) 8
to hinder the extension of deep CNNs. How to make full see more work for applications with larger deep nets (e.g.,
use of the limited computational source and how to design video and image frames [88], [89]).
special compression methods for such platforms are still
challenges that need to be addressed. IX. ACKNOWLEDGMENTS
Despite the great achievements of these compression ap-
proaches, the black box mechanism is still the key barrier The authors would like to thank the reviewers and broader
to the adoption. Exploring the knowledge interpret-ability community for their feedback on this survey. In particular,
is still an important problem. we would like to thank Hong Zhao from the Department of
Automation of Tsinghua University for her help on modifying
C. Possible Solutions the paper. This research is supported by National Science
Foundation of China with Grant number 61401169.To solve the hyper-parameters configuration problem, we
can rely on the recent learning-to-learn strategies [76], [77].
This framework provides a mechanism allowing the algorithm REFERENCES
to automatically learn how to exploit structure in the problem [1]A. Krizhevsky, I. Sutskever, and G. Hinton, “Imagenet classification with of interest. Very recently, leveraging reinforcement learning deep convolutional neural networks,” inNIPS, 2012.
to efficiently sample the design space and improve the model [2]Y. Taigman, M. Yang, M. Ranzato, and L. Wolf, “Deepface: Closing the
compression has also been tried [78]. gap to human-level performance in face verification,” inCVPR, 2014.
[3]Y. Lu, A. Kumar, S. Zhai, Y. Cheng, T. Javidi, and R. S. Feris, “Fully- Channel pruning provides the efficiency benefit on both adaptive feature sharing in multi-task networks with applications in
CPU and GPU because no special implementation is required. person attribute classification,”CoRR, vol. abs/1611.05377, 2016.
But it is also challenging to handle the input configuration. [4]J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, Q. Le, M. Mao,
M. Ranzato, A. Senior, P. Tucker, K. Yang, and A. Ng, “Large scale One possible solution is to use the training-based channel distributed deep networks,” inNIPS, 2012.
pruning methods [79], which focus on imposing sparse con- [5]K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image
straints on weights during training. However, training from recognition,”CoRR, vol. abs/1512.03385, 2015.
[6]Y. Gong, L. Liu, M. Yang, and L. D. Bourdev, “Compressing scratch for such method is costly for very deep CNNs. In deep convolutional networks using vector quantization,”CoRR, vol.
[80], the authors provided an iterative two-step algorithm to abs/1412.6115, 2014.
effectively prune channels in each layer. [7]Y. W. Q. H. Jiaxiang Wu, Cong Leng and J. Cheng, “Quantized
convolutional neural networks for mobile devices,” inIEEE Conference Exploring new types of knowledge in the teacher models on Computer Vision and Pattern Recognition (CVPR), 2016.
and transferring it to the student models is useful for the [8]V. Vanhoucke, A. Senior, and M. Z. Mao, “Improving the speed of
knowledge distillation (KD) approaches. Instead of directly re- neural networks on cpus,” inDeep Learning and Unsupervised Feature
Learning Workshop, NIPS 2011, 2011. ducing and transferring parameters, passing selectivity knowl- [9]S. Gupta, A. Agrawal, K. Gopalakrishnan, and P. Narayanan, “Deep
edge of neurons could be helpful. One can derive a way to learning with limited numerical precision,” inProceedings of the
select essential neurons related to the task [81], [82]. The 32Nd International Conference on International Conference on Machine
Learning - Volume 37, ser. ICML’15, 2015, pp. 1737–1746. intuition is that if a neuron is activated in certain regions [10]S. Han, H. Mao, and W. J. Dally, “Deep compression: Compressing
or samples, that implies these regions or samples share some deep neural networks with pruning, trained quantization and huffman
common properties that may relate to the task. coding,”International Conference on Learning Representations (ICLR),
2016. For methods with the convolutional filters and the structural [11]Y. Choi, M. El-Khamy, and J. Lee, “Towards the limit of network
matrix, we can conclude that the transformation lies in the quantization,”CoRR, vol. abs/1612.01543, 2016.
family of functions that only operations on the spatial dimen- [12]M. Courbariaux, Y. Bengio, and J. David, “Binaryconnect: Training deep
neural networks with binary weights during propagations,” inAdvances sions. Hence to address the imposed prior issue, one solution is in Neural Information Processing Systems 28: Annual Conference on
to provide a generalization of the aforementioned approaches Neural Information Processing Systems 2015, December 7-12, 2015,
in two aspects: 1) instead of limiting the transformation to Montreal, Quebec, Canada, 2015, pp. 3123–3131.
[13]M. Courbariaux and Y. Bengio, “Binarynet: Training deep neural net- belong to a set of predefined transformations, let it be the works with weights and activations constrained to +1 or -1,”CoRR, vol.
whole family of spatial transformations applied on 2D filters abs/1602.02830, 2016.
or matrix, and 2) learn the transformation jointly with all the [14]M. Rastegari, V. Ordonez, J. Redmon, and A. Farhadi, “Xnor-net:
Imagenet classification using binary convolutional neural networks,” in model parameters. ECCV, 2016.
Regarding the use of CNNs in small platforms, proposing [15]P. Merolla, R. Appuswamy, J. V. Arthur, S. K. Esser, and D. S. Modha,
some general/unified approaches is one direction. Wanget al. “Deep neural networks are robust to weight binarization and other non-
[83] presented a feature map dimensionality reduction method linear distortions,”CoRR, vol. abs/1606.01981, 2016.
[16]L. Hou, Q. Yao, and J. T. Kwok, “Loss-aware binarization of deep by excavating and removing redundancy in feature maps gen- networks,”CoRR, vol. abs/1611.01600, 2016.
erated from different filters, which could also preserve intrinsic [17]Z. Lin, M. Courbariaux, R. Memisevic, and Y. Bengio, “Neural networks
information of the original network. The idea can be applied with few multiplications,”CoRR, vol. abs/1510.03009, 2015.
[18]S. J. Hanson and L. Y. Pratt, “Comparing biases for minimal network to make CNNs more applicable for different platforms. The construction with back-propagation,” inAdvances in Neural Information
work in [84] proposed a one-shot whole network compression Processing Systems 1, D. S. Touretzky, Ed., 1989, pp. 177–185.
scheme consisting of three components: rank selection, low- [19]Y. L. Cun, J. S. Denker, and S. A. Solla, “Advances in neural information
processing systems 2,” D. S. Touretzky, Ed., 1990, ch. Optimal Brain rank tensor decomposition, and fine-tuning to make deep Damage, pp. 598–605.
CNNs work in mobile devices. [20]B. Hassibi, D. G. Stork, and S. C. R. Com, “Second order derivatives
Despite the classification task, people are also adapting the for network pruning: Optimal brain surgeon,” inAdvances in Neural
Information Processing Systems 5. Morgan Kaufmann, 1993, pp. 164– compacted models in other tasks [85]–[87]. We would like to 171. IEEE SIGNAL PROCESSING MAGAZINE, SPECIAL ISSUE ON DEEP LEARNING FOR IMAGE UNDERSTANDING (ARXIV EXTENDED VERSION) 9
[21]S. Srinivas and R. V. Babu, “Data-free parameter pruning for deep neural [43]T. S. Cohen and M. Welling, “Group equivariant convolutional net-
networks,” inProceedings of the British Machine Vision Conference works,”arXiv preprint arXiv:1602.07576, 2016.
2015, BMVC 2015, Swansea, UK, September 7-10, 2015, 2015, pp. [44]S. Zhai, Y. Cheng, and Z. M. Zhang, “Doubly convolutional neural
31.1–31.12. networks,” inAdvances In Neural Information Processing Systems, 2016,
[22]S. Han, J. Pool, J. Tran, and W. J. Dally, “Learning both weights and pp. 1082–1090.
connections for efficient neural networks,” inProceedings of the 28th [45]W. Shang, K. Sohn, D. Almeida, and H. Lee, “Understanding and
International Conference on Neural Information Processing Systems, ser. improving convolutional neural networks via concatenated rectified
NIPS’15, 2015. linear units,”arXiv preprint arXiv:1603.05201, 2016.
[23]W. Chen, J. Wilson, S. Tyree, K. Q. Weinberger, and Y. Chen, “Com- [46]H. Li, W. Ouyang, and X. Wang, “Multi-bias non-linear activation in
pressing neural networks with the hashing trick.” JMLR Workshop and deep neural networks,”arXiv preprint arXiv:1604.00676, 2016.
Conference Proceedings, 2015. [47]S. Dieleman, J. De Fauw, and K. Kavukcuoglu, “Exploiting cyclic
[24]K. Ullrich, E. Meeds, and M. Welling, “Soft weight-sharing for neural symmetry in convolutional neural networks,” inProceedings of the
network compression,”CoRR, vol. abs/1702.04008, 2017. 33rd International Conference on International Conference on Machine
[25]V. Lebedev and V. S. Lempitsky, “Fast convnets using group-wise brain Learning - Volume 48, ser. ICML’16, 2016.
damage,” in2016 IEEE Conference on Computer Vision and Pattern [48]C. Szegedy, S. Ioffe, and V. Vanhoucke, “Inception-v4, inception-
Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, 2016, resnet and the impact of residual connections on learning.”CoRR, vol.
pp. 2554–2564. abs/1602.07261, 2016.
[26]H. Zhou, J. M. Alvarez, and F. Porikli, “Less is more: Towards compact [49]B. Wu, F. N. Iandola, P. H. Jin, and K. Keutzer, “Squeezedet: Unified,
cnns,” inEuropean Conference on Computer Vision, Amsterdam, the small, low power fully convolutional neural networks for real-time object
Netherlands, October 2016, pp. 662–677. detection for autonomous driving,”CoRR, vol. abs/1612.01051, 2016.
[27]W. Wen, C. Wu, Y. Wang, Y. Chen, and H. Li, “Learning structured [50]C. Bucilua, R. Caruana, and A. Niculescu-Mizil, “Model compression,”ˇ
sparsity in deep neural networks,” inAdvances in Neural Information inProceedings of the 12th ACM SIGKDD International Conference on
Processing Systems 29, D. D. Lee, M. Sugiyama, U. V. Luxburg, Knowledge Discovery and Data Mining, ser. KDD ’06, 2006, pp. 535–
I. Guyon, and R. Garnett, Eds., 2016, pp. 2074–2082. 541.
[28]H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf, “Pruning [51]J. Ba and R. Caruana, “Do deep nets really need to be deep?” in
filters for efficient convnets,”CoRR, vol. abs/1608.08710, 2016. Advances in Neural Information Processing Systems 27: Annual Confer-
[29]V. Sindhwani, T. Sainath, and S. Kumar, “Structured transforms for ence on Neural Information Processing Systems 2014, December 8-13
small-footprint deep learning,” inAdvances in Neural Information Pro- 2014, Montreal, Quebec, Canada, 2014, pp. 2654–2662.
cessing Systems 28, C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, [52]G. E. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a
and R. Garnett, Eds., 2015, pp. 3088–3096. neural network,”CoRR, vol. abs/1503.02531, 2015.
[30]Y. Cheng, F. X. Yu, R. Feris, S. Kumar, A. Choudhary, and S.-F. [53]A. Romero, N. Ballas, S. E. Kahou, A. Chassang, C. Gatta, and
Chang, “An exploration of parameter redundancy in deep networks with Y. Bengio, “Fitnets: Hints for thin deep nets,”CoRR, vol. abs/1412.6550,
circulant projections,” inInternational Conference on Computer Vision 2014.
(ICCV), 2015. [54]A. Korattikara Balan, V. Rathod, K. P. Murphy, and M. Welling,
[31]Y. Cheng, F. X. Yu, R. S. Feris, S. Kumar, A. N. Choudhary, and “Bayesian dark knowledge,” inAdvances in Neural Information Process-
S. Chang, “Fast neural networks with circulant projections,”CoRR, vol. ing Systems 28, C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama,
abs/1502.03436, 2015. and R. Garnett, Eds., 2015, pp. 3420–3428.
[32]Z. Yang, M. Moczulski, M. Denil, N. de Freitas, A. Smola, L. Song, [55]P. Luo, Z. Zhu, Z. Liu, X. Wang, and X. Tang, “Face model compression
and Z. Wang, “Deep fried convnets,” inInternational Conference on by distilling knowledge from neurons,” inProceedings of the Thirtieth
Computer Vision (ICCV), 2015. AAAI Conference on Artificial Intelligence, February 12-17, 2016,
[33]J. Chun and T. Kailath,Generalized Displacement Structure for Block- Phoenix, Arizona, USA., 2016, pp. 3560–3566.
Toeplitz, Toeplitz-block, and Toeplitz-derived Matrices. Berlin, Heidel- [56]T. Chen, I. J. Goodfellow, and J. Shlens, “Net2net: Accelerating learning
berg: Springer Berlin Heidelberg, 1991, pp. 215–236. via knowledge transfer,”CoRR, vol. abs/1511.05641, 2015.
[34]M. V. Rakhuba and I. V. Oseledets, “Fast multidimensional convolution [57]S. Zagoruyko and N. Komodakis, “Paying more attention to attention:
in low-rank tensor formats via cross approximation,”SIAM J. Scientific Improving the performance of convolutional neural networks via atten-
[35]M. Moczulski, M. Denil, J. Appleyard, and N. de Freitas, “Acdc: [58]D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by
A structured efficient linear layer,” inInternational Conference on jointly learning to align and translate,”CoRR, vol. abs/1409.0473, 2014.
Learning Representations (ICLR), 2016. [59]A. Almahairi, N. Ballas, T. Cooijmans, Y. Zheng, H. Larochelle, and
[36]R. Rigamonti, A. Sironi, V. Lepetit, and P. Fua, “Learning separable A. C. Courville, “Dynamic capacity networks,” inProceedings of the
filters,” in2013 IEEE Conference on Computer Vision and Pattern 33nd International Conference on Machine Learning, ICML 2016, New
Recognition, Portland, OR, USA, June 23-28, 2013, 2013, pp. 2754– York City, NY, USA, June 19-24, 2016, 2016, pp. 2549–2558.
2761. [60]N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton,
[37]E. L. Denton, W. Zaremba, J. Bruna, Y. LeCun, and R. Fergus, and J. Dean, “Outrageously large neural networks: The sparsely-gated
“Exploiting linear structure within convolutional networks for efficient mixture-of-experts layer,” 2017.
evaluation,” inAdvances in Neural Information Processing Systems 27, [61]D. Wu, L. Pigou, P. Kindermans, N. D. Le, L. Shao, J. Dambre, and
Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. J. Odobez, “Deep dynamic neural networks for multimodal gesture
Weinberger, Eds., 2014, pp. 1269–1277. segmentation and recognition,”IEEE Trans. Pattern Anal. Mach. Intell.,
[38]M. Jaderberg, A. Vedaldi, and A. Zisserman, “Speeding up convolutional vol. 38, no. 8, pp. 1583–1597, 2016.
neural networks with low rank expansions,” inProceedings of the British [62]C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan,
Machine Vision Conference. BMVA Press, 2014. V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,”
[39]V. Lebedev, Y. Ganin, M. Rakhuba, I. V. Oseledets, and V. S. Lempit- inComputer Vision and Pattern Recognition (CVPR), 2015.
sky, “Speeding-up convolutional neural networks using fine-tuned cp- [63]G. Huang, Y. Sun, Z. Liu, D. Sedra, and K. Q. Weinberger,Deep
decomposition,”CoRR, vol. abs/1412.6553, 2014. Networks with Stochastic Depth, 2016.
[40]C. Tai, T. Xiao, X. Wang, and W. E, “Convolutional neural networks [64]Y. Yamada, M. Iwamura, and K. Kise, “Deep pyramidal residual
with low-rank regularization,” vol. abs/1511.06067, 2015. networks with separated stochastic depth,”CoRR, vol. abs/1612.01230,
[41]M. Denil, B. Shakibi, L. Dinh, M. Ranzato, and N. D. Freitas, 2016.
“Predicting parameters in deep learning,” in Advances in Neural [65]Z. Wu, T. Nagarajan, A. Kumar, S. Rennie, L. S. Davis, K. Grauman, and
Information Processing Systems 26, C. Burges, L. Bottou, M. Welling, R. Feris, “Blockdrop: Dynamic inference paths in residual networks,”
Z. Ghahramani, and K. Weinberger, Eds., 2013, pp. 2148–2156. inCVPR, 2018.
[Online]. Available: http://media.nips.cc/nipsbooks/nipspapers/paper [66]A. Veit and S. Belongie, “Convolutional networks with adaptive infer-
files/nips26/1053.pdf ence graphs,” 2018.
[42]T. N. Sainath, B. Kingsbury, V. Sindhwani, E. Arisoy, and B. Ramab- [67]M. Mathieu, M. Henaff, and Y. Lecun,Fast training of convolutional
hadran, “Low-rank matrix factorization for deep neural network training networks through FFTs, 2014.
with high-dimensional output targets,” inin Proc. IEEE Int. Conf. on [68]A. Lavin and S. Gray, “Fast algorithms for convolutional neural net-
Acoustics, Speech and Signal Processing, 2013. works,” in2016 IEEE Conference on Computer Vision and Pattern IEEE SIGNAL PROCESSING MAGAZINE, SPECIAL ISSUE ON DEEP LEARNING FOR IMAGE UNDERSTANDING (ARXIV EXTENDED VERSION) 10
Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, 2016, [89]L. Cao, S.-F. Chang, N. Codella, C. V. Cotton, D. Ellis, L. Gong,
pp. 4013–4021. M. Hill, G. Hua, J. Kender, M. Merler, Y. Mu, J. R. Smith, and F. X.
[69]S. Zhai, H. Wu, A. Kumar, Y. Cheng, Y. Lu, Z. Zhang, and R. S. Yu, “Ibm research and columbia university trecvid-2012 multimedia
Feris, “S3pool: Pooling with stochastic spatial sampling,”CoRR, vol. event detection (med), multimedia event recounting (mer), and semantic
[76]M. Andrychowicz, M. Denil, S. G. Colmenarejo, M. W. Hoffman,
D. Pfau, T. Schaul, and N. de Freitas, “Learning to learn by gradient
descent by gradient descent,” inNeural Information Processing Systems
(NIPS), 2016. Duo Wang (d-wang15@mail.tsinghua.edu.cn) re-[77]D. Ha, A. Dai, and Q. Le, “Hypernetworks,” inInternational Conference ceived the B.S. degree in automation from theon Learning Representations 2016, 2016. Harbin Institute of Technology, China, in 2015.[78]Y. He, J. Lin, Z. Liu, H. Wang, L.-J. Li, and S. Han, “Amc: Automl Currently he is purchasing his Ph.D. degree at thefor model compression and acceleration on mobile devices,” inThe Department of Automation, Tsinghua University,European Conference on Computer Vision (ECCV), September 2018. Beijing, P.R. China. Currently his research interests[79]J. M. Alvarez and M. Salzmann, “Learning the number of neurons in are about deep learning, particularly in few-shotdeep networks,” pp. 2270–2278, 2016. learning and deep generative models. He also works[80]Y. He, X. Zhang, and J. Sun, “Channel pruning for accelerating on a lot of applications in computer vision andvery deep neural networks,” inThe IEEE International Conference on robotics vision.Computer Vision (ICCV), Oct 2017.
[81]Z. Huang and N. Wang, “Data-driven sparse structure selection for deep
neural networks,”ECCV, 2018.
[82]Y. Chen, N. Wang, and Z. Zhang, “Darkrank: Accelerating deep metric
learning via cross sample similarities transfer,” inProceedings of the
Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18),
New Orleans, Louisiana, USA, February 2-7, 2018, 2018, pp. 2852– Pan Zhou(panzhou@hust.edu.cn) is currently an
2859. associate professor with School of Electronic In-
[83]Y. Wang, C. Xu, C. Xu, and D. Tao, “Beyond filters: Compact feature formation and Communications, Wuhan, China. He
map for portable deep model,” inProceedings of the 34th International received his Ph.D. in the School of Electrical and
Conference on Machine Learning, ser. Proceedings of Machine Learning Computer Engineering at the Georgia Institute of
Research, D. Precup and Y. W. Teh, Eds., vol. 70. International Technology in 2011. Before that, he received his
Convention Centre, Sydney, Australia: PMLR, 06–11 Aug 2017, pp. B.S. degree in theAdvanced Classof HUST, and
3703–3711. a M.S. degree in the Department of Electronics
[84]Y.-D. Kim, E. Park, S. Yoo, T. Choi, L. Yang, and D. Shin, “Compression and Information Engineering from HUST, Wuhan,
of deep convolutional neural networks for fast and low power mobile China, in 2006 and 2008, respectively. His current
applications,”CoRR, vol. abs/1511.06530, 2015. research interest includes big data analytics and
[85]G. Chen, W. Choi, X. Yu, T. Han, and M. Chandraker, “Learning efficient machine learning, security and privacy, and information networks.
object detection models with knowledge distillation,” inAdvances in
Neural Information Processing Systems 30, I. Guyon, U. V. Luxburg,
S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett,
Eds., 2017, pp. 742–751.
[86]M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, Tao Zhang (taozhang@mail.tsinghua.edu.cn) ob-
“Mobilenetv2: Inverted residuals and linear bottlenecks,” inThe IEEE tained his B.S., M.S., and Ph.D. degrees from Ts-
Conference on Computer Vision and Pattern Recognition (CVPR), June inghua University, Beijing, China, in 1993, 1995,
2018. and 1999, respectively, and another Ph.D. degree
[87]J. Huang, V. Rathod, C. Sun, M. Zhu, A. Korattikara, A. Fathi, I. Fischer, from Saga University, Saga, Japan, in 2002, all in
Z. Wojna, Y. Song, S. Guadarrama, and K. Murphy, “Speed/accuracy control engineering. He is currently a Professor with
trade-offs for modern convolutional object detectors,” in2017 IEEE the Department of Automation, Tsinghua University.
Conference on Computer Vision and Pattern Recognition, CVPR 2017, He serves the Associate Dean, School of Information
Honolulu, HI, USA, July 21-26, 2017, 2017, pp. 3296–3297. Science and Technology and Head of the Department
[88]Y. Cheng, Q. Fan, S. Pankanti, and A. Choudhary, “Temporal sequence of Automation. His current research interests include
modeling for video event detection,” in The IEEE Conference on artificial intelligence, robotics, image processing,
Computer Vision and Pattern Recognition (CVPR), June 2014. control theory, and control of spacecraft.
<<END>> <<END>> <<END>>
<<START>> <<START>> <<START>>
Analysis and Design of Echo State Networks
Mustafa C. Ozturk
can@cnel.ufl.edu
Dongming Xu
dmxu@cnel.ufl.edu
Jose C. Principe
principe@cnel.ufl.edu
Computational NeuroEngineering Laboratory, Department of Electrical and
Computer Engineering, University of Florida, Gainesville, FL 32611, U.S.A.
The design of echo state network (ESN) parameters relies on the selec-
tion of the maximum eigenvalue of the linearized system around zero
(spectral radius). However, this procedure does not quantify in a sys-
tematic manner the performance of the ESN in terms of approximation
error. This article presents a functional space approximation framework
to better understand the operation of ESNs and proposes an information-
theoretic metric, the average entropy of echo states, to assess the richness
of the ESN dynamics. Furthermore, it provides an interpretation of the
ESN dynamics rooted in system theory as families of coupled linearized
systems whose poles move according to the input signal dynamics. With
this interpretation, a design methodology for functional approximation
is put forward where ESNs are designed with uniform pole distributions
covering the frequency spectrum to abide by the richness metric, irre-
spective of the spectral radius. A single bias parameter at the ESN input,
adapted with the modeling error, configures the ESN spectral radius to
the input-output joint space. Function approximation examples compare
the proposed design methodology versus the conventional design.
1 Introduction
Dynamic computational models require the ability to store and access the
time history of their inputs and outputs. The most common dynamic neural
architecture is the time-delay neural network (TDNN) that couples delay
lines with a nonlinear static architecture where all the parameters (weights)
are adapted with the backpropagation algorithm. The conventional delay
line utilizes ideal delay operators, but delay lines with local first-order re-
cursive filters have been proposed by Werbos (1992) and extensively stud-
ied in the gamma model (de Vries, 1991; Principe, de Vries, & de Oliviera,
1993). Chains of first-order integrators are interesting because they effec-
tively decrease the number of delays necessary to create time embeddings
(Principe, 2001). Recurrent neural networks (RNNs) implement a differ-
ent type of embedding that is largely unexplored. RNNs are perhaps the
most biologically plausible of the artificial neural network (ANN) models
University of Amsterdam University of Amsterdam University of Amsterdam
TNO Intelligent Imaging k.ullrich@uva.nl CIFAR
c.louizos@uva.nl m.welling@uva.nl
Abstract
Compression and computational efficiency in deep learning have become a problem
of great significance. In this work, we argue that the most principled and effective
way to attack this problem is by adopting a Bayesian point of view, where through
sparsity inducing priors we prune large parts of the network. We introduce two
novelties in this paper: 1) we use hierarchical priors to prune nodes instead of
individual weights, and 2) we use the posterior uncertainties to determine the
optimal fixed point precision to encode the weights. Both factors significantly
contribute to achieving the state of the art in terms of compression rates, while
still staying competitive with methods designed to optimize for speed or energy
efficiency.
1 Introduction
While deep neural networks have become extremely successful in in a wide range of applications,
often exceeding human performance, they remain difficult to apply in many real world scenarios. For
instance, making billions of predictions per day comes with substantial energy costs given the energy
consumption of common Graphical Processing Units (GPUs). Also, real-time predictions are often
about a factor100away in terms of speed from what deep NNs can deliver, and sending NNs with
millions of parameters through band limited channels is still impractical. As a result, running them on
hardware limited devices such as smart phones, robots or cars requires substantial improvements on
all of these issues. For all those reasons, compression and efficiency have become a topic of interest
in the deep learning community.
While all of these issues are certainly related, compression and performance optimizing procedures
might not always be aligned. As an illustration, consider the convolutional layers of Alexnet, which
account for only 4% of the parameters but 91% of the computation [68]. Compressing these layers
will not contribute much to the overall memory footprint.
There is a variety of approaches to address these problem settings. However, most methods have
the common strategy of reducing both the neural network structure and the effective fixed point
precision for each weight. A justification for the former is the finding that NNs suffer from significant
parameter redundancy [14]. Methods in this line of thought are network pruning, where unnecessary
connections are being removed [40,24,21], or student-teacher learning where a large network is
used to train a significantly smaller network [5, 27].
From a Bayesian perspective network pruning and reducing bit precision for the weights is aligned
with achieving high accuracy, because Bayesian methods search for the optimal model structure
(which leads to pruning with sparsity inducing priors), and reward uncertain posteriors over parameters
through the bits back argument [28] (which leads to removing insignificant bits). This relation is
made explicit in the MDL principle [20] which is known to be related to Bayesian inference.
In this paper we will use the variational Bayesian approximation for Bayesian inference which has
also been explicitly interpreted in terms of model compression [28]. By employing sparsity inducing
priors for hidden units (and not individual weights) we can prune neurons including all their ingoing
and outgoing weights. This avoids more complicated and inefficient coding schemes needed for
pruning or vector quantizing individual weights. As an additional Bayesian bonus we can use the
variational posterior uncertainty to assess which bits are significant and remove the ones which
fluctuate too much under approximate posterior sampling. From this we derive the optimal fixed
point precision per layer, which is still practical on chip.
2 Variational Bayes and Minimum Description Length
A fundamental theorem in information theory is the minimum description length (MDL) principle [20].
It relates to compression directly in that it defines the best hypothesis to be the one that communicates
the sum of the model (complexity costLC ) and the data misfit (error costLE ) with the minimum
number of bits [59,60]. It is well understood that variational inference can be reinterpreted from an
MDL point of view [56,72,28,30,19]. More specifically, assume that we are presented with a dataset QD that consists from N input-output pairs <<FORMULA>>. Let <<FORMULA>>
be a parametric model, e.g. a deep neural network, that maps inputs x to their corresponding outputs
y using parameters w governed by a prior distribution <<p(w)>>. In this scenario, we wish to approximate
the intractable posterior distribution <<p(w|D) =p(D|w)p(w)=p(D)>> with a fixed form approximate
posterior <<q(w)>> by optimizing the variational parameters according to:
<<FORMULA>>
where <<H()>> denotes the entropy and <<L()>> is known as the evidence-lower-bound (ELBO) or negative
variational free energy. As indicated in eq.1, <<L()>> naturally decomposes into a minimum cost for
communicating the targets <<FORMULA>> under the assumption that the sender and receiver agreed on a n=1 prior <<p(w)>> and that the receiver knows the inputs <<FORMULA>> and form of the parametric model. n=1
By using sparsity inducing priors for groups of weights that feed into a neuron the Bayesian mecha-
nism will start pruning hidden units that are not strictly necessary for prediction and thus achieving
compression. But there is also a second mechanism by which Bayes can help us compress. By
explicitly entertaining noisy weight encodings through <<q(w)>> we can benefit from the bits-back
argument [28,30] due to the entropy term; this is in contrast to infinitely precise weights that lead to
<<FORMULA>>. Nevertheless in practice, the data misfit termLE is intractable for neural network
models under a noisy weight encoding, so as a solution Monte Carlo integration is usually employed.
Continuous q(w) allow for the reparametrization trick [36,58]. Here, we replace sampling from
q(w) by a deterministic function of the variational parameters and random samples from some
noise variables:
<<FORMULA>>; (2)
where <<w=f(;)>>. By applying this trick, we obtain unbiased stochastic gradients of the ELBO
with respect to the variational parameters, thus resulting in a standard optimization problem that is
fit for stochastic gradient ascent. The efficiency of the gradient estimator resulting from eq. 2 can be
further improved for neural networks by utilizing local reparametrizations [37] (which we will use in
our experiments); they provide variance reduction in an efficient way by locally marginalizing the
weights at each layer and instead sampling the distribution of the pre-activations.
3 Related Work
One of the earliest ideas and most direct approaches to tackle efficiency is pruning. Originally
introduced by [40], pruning has recently been demonstrated to be applicable to modern architectures
[25,21]. It had been demonstrated that an overwhelming amount of up to 99,5% of parameters
can be pruned in common architectures. There have been quite a few encouraging results obtained
by (empirical) Bayesian approaches that employ weight pruning [19,7,52,70,51]. Nevertheless,
2 In practice this term is a large constant determined by the weight precision.
weight pruning is in general inefficient for compression since the matrix format of the weights is not
taken into consideration, therefore the Compressed Sparse Column (CSC) format has to be employed.
Moreover, note that in conventional CNNs most flops are used by the convolution operation. Inspired
by this observation, several authors proposed pruning schemes that take these considerations into
account [73, 74] or even go as far as efficiency aware architectures to begin with [32, 15, 31]. From
the Bayesian viewpoint, similar pruning schemes have been explored at [47, 53, 39, 34].
Given optimal architecture, NNs can further be compressed by quantization. More precisely, there
are two common techniques. First, the set of accessible weights can be reduced drastically. As an
extreme example, [13,48,57,76] and [11] trained NN to use only binary or tertiary weights with
floating point gradients. This approach however is in need of significantly more parameters than
their ordinary counterparts. Work by [18] explores various techniques beyond binary quantization:
k-means quantization, product quantization and residual quantization. Later studies extent this set to
optimal fixed point [44] and hashing quantization [10]. [25] apply k-means clustering and consequent
center training. From a practical point of view, however, all these are fairly unpractical during
test time. For the computation of each feature map in a net, the original weight matrix must be
reconstructed from the indexes in the matrix and a codebook that contains all the original weights.
This is an expensive operation and this is why some studies propose a different approach than set
quantization. Precision quantization simply reduces the bit size per weight. This has a great advantage
over set quantization at inference time since feature maps can simply be computed with less precision
weights. Several studies show that this has little to no effect on network accuracy when using 16bit
weights [49,22,12,71,9]. Somewhat orthogonal to the above discussion but certainly relevant are
approaches that customize the implementation of CNNs for hardware limited devices[31, 4, 62].
4 Bayesian compression with scale mixtures of normals
Consider the following prior over a parameter w where its scale z is governed by a distribution <<p(z)>>:
<<FORMULA>>; (3)
with z2 serving as the variance of the zero-mean normal distribution over w. By treating the scales
of w as random variables we can recover marginal prior distributions over the parameters that have
heavier tails and more mass at zero; this subsequently biases the posterior distribution over w to
be sparse. This family of distributions is known as scale-mixtures of normals [6,2] and it is quite
general, as a lot of well known sparsity inducing distributions are special cases.
One example of the aforementioned framework is the spike-and-slab distribution [50], the golden
standard for sparse Bayesian inference. Under the spike-and-slab, the mixing density of the scales is a
Bernoulli distribution, thus the marginal <<p(w)>> has a delta “spike” at zero and a continuous “slab” over
the real line. Unfortunately, this prior leads to a computationally expensive inference since we have
to explore a space of2M models, whereMis the number of the model parameters. Dropout [29,67],
one of the most popular regularization techniques for neural networks, can be interpreted as positing a
spike and slab distribution over the weights where the variance of the “slab” is zero [17,45]. Another
example is the Laplace distribution which arises by considering <<FORMULA>>. The mode of
the posterior distribution under a Laplace prior is known as the Lasso [69] estimator and has been
previously used for sparsifying neural networks at [73,61]. While computationally simple, the
Lasso estimator is prone to “shrinking" large signals [8] and only provides point estimates about
the parameters. As a result it does not provide uncertainty estimates, it can potentially overfit and,
according to the bits-back argument, is inefficient for compression.
For these reasons, in this paper we will tackle the problem of compression and efficiency in neural
networks by adopting a Bayesian treatment and inferring an approximate posterior distribution over
the parameters under a scale mixture prior. We will consider two choices for the prior over the scales
p(z); the hyperparameter free log-uniform prior [16,37] and the half-Cauchy prior, which results into
a horseshoe [8] distribution. Both of these distributions correspond to a continuous relaxation of the
spike-and-slab prior and we provide a brief discussion on their shrinkage properties at Appendix C.
4.1 Reparametrizing variational dropout for group sparsity
One potential choice for p(z) is the improper log-uniform prior [37] <<FORMULA>>. It turns out that
we can recover the log-uniform prior over the weightswif we marginalize over the scales z:
<<FORMULA>> (4)
This alternative parametrization of the log uniform prior is known in the statistics literature as the
normal-Jeffreys prior and has been introduced by [16]. This formulation allows to “couple" the
scales of weights that belong to the same group (e.g. neuron or feature map), by simply sharing the
corresponding scale variablezin the joint prior 3 :
<<FORMULA>>; (5)
where W is the weight matrix of a fully connected neural network layer with A being the dimen-
sionality of the input and B the dimensionality of the output. Now consider performing variational
inference with a joint approximate posterior parametrized as follows:
<<FORMULA>>; (6)
where _i is the dropout rate [67,37,51] of the given group. As explained at [37,51], the multiplicative
parametrization of the approximate posterior over z suffers from high variance gradients; therefore
we will follow [51] and re-parametrize it in terms of <<FORMULA>>, hence optimize w.r.t._2 .
The <<FORMULA>> lower bound under this prior and approximate posterior becomes:
<<FORMULA>> (7)
Under this particular variational posterior parametrization the negative KL-divergence from the
conditional prior <<p(W|z)>> to the approximate posterior <<q(W|z)>> is independent of z:
<<FORMULA>> (8)
This independence can be better understood if we consider a non-centered parametrization of the
prior [55]. More specifically, consider reparametrizing the weights asw~ij =wij ; this will then result zi
into <<p(W|z)p(z) =p(W~)p(z)>>, where <<FORMULA>>. Now if <<FORMULA>> and <<W= diag(z)>>
we perform variational inference under the p(W~)p(z)prior with an approximate posterior that has Q the form of <<FORMULA>>, with <<FORMULA>>, then we see that we ij arrive at the same expressions for the negative KL-divergence from the prior to the approximate
posterior. Finally, the negative KL-divergence from the normal-Jeffreys scale prior p(z) to the
Gaussian variational posterior q depends only on the “implied” dropout rate, <<FORMULA>>, and zi z takes the following form [51]:
<<FORMULA>>; (9)
where <<FORMULA>> are the sigmoid and softplus functions respectively 4 and k1 = 0:63576,k2 =
1:87320,k3 = 1:48695. We can now prune entire groups of parameters by simply specifying a thresh-
old for the variational dropout rate of the corresponding group, e.g.<<FORMULA>>. It should be mentioned that this prior parametrization readily allows for a more flexible marginal pos-
terior over the weights as we now have a compound distribution, <<FORMULA>>; this
is in contrast to the original parametrization and the Gaussian approximations employed by [37,51].
Strictly speaking the result of eq. 4 only holds when each weight has its own scale and not when that scale is
shared across multiple weights. Nevertheless, in practice we obtain a prior that behaves in a similar way, i.e. it
biases the variational posterior to be sparse.
<<FORMULA>>
Furthermore, this approach generalizes the low variance additive parametrization of variational
dropout proposed for weight sparsity at [51] to group sparsity (which was left as an open question
at [51]) in a principled way.
At test time, in order to have a single feedforward pass we replace the distribution overWat each
layer with a single weight matrix, the masked variational posterior mean:
<<FORMULA>>; (10)
where m is a binary mask determined according to the group variational dropout rate andMW are
the means ofq (W~). We further use the variational posterior marginal variances 5 for this particular
posterior approximation:
<<FORMULA>>; (11)
to assess the bit precision of each weight in the weight matrix. More specifically, we employed the
mean variance across the weight matrixW^ to compute the unit round off necessary to represent the
weights. This method will give us the amount significant bits, and by adding 3 exponent and 1 sign
bits we arrive at the final bit precision for the entire weight matrixW^6 . We provide more details at
Appendix B.
4.2 Group horseshoe with half-Cauchy scale priors
Another choice for p(z) is a proper half-Cauchy distribution: <<FORMULA>>; it
induces a horseshoe prior [8] distribution over the weights, which is a well known sparsity inducing
prior in the statistics literature. More formally, the prior hierarchy over the weights is expressed as
(in a non-centered parametrization):
<<FORMULA>>; (12)
where0 is the free parameter that can be tuned for specific desiderata. The idea behind the horseshoe
is that of the “global-local" shrinkage; the global scale variablespulls all of the variables towards
zero whereas the heavy tailed local variableszi can compensate and allow for some weights to escape.
Instead of directly working with the half-Cauchy priors we will employ a decomposition of the
half-Cauchy that relies upon (inverse) gamma distributions [54] as this will allow us to compute
the negative KL-divergence from the scale priorp(z)to an approximate log-normal scale posterior
q (z)in closed form (the derivation is given in Appendix D). More specifically, we have that the
half-Cauchy prior can be expressed in a non-centered parametrization as:
<<FORMULA>>; (13)
where <<IG(;);G(;)>> correspond to the inverse Gamma and Gamma distributions in the scale
parametrization, and z follows a half-Cauchy distribution with scale k. Therefore we will re-express
the whole hierarchy as:
<<FORMULA>>; (14)
It should be mentioned that the improper log-uniform prior is the limiting case of the horseshoe prior
when the shapes of the (inverse) Gamma hyperpriors on <<FORMULA>> go to zero [8]. In fact, several well
known shrinkage priors can be expressed in this form by altering the shapes of the (inverse) Gamma
hyperpriors [3]. For the variational posterior we will employ the following mean field approximation:
<<FORMULA>>.
Notice that the fact that we are using mean-field variational approximations (which we chose for simplicity)
can potentially underestimate the variance, thus lead to higher bit precisions for the weights. We leave the
exploration of more involved posteriors for future work.
Where <<LN(;)>> is a log-normal distribution. It should be mentioned that a similar form of non-
centered variational inference for the horseshoe has been also successfully employed for undirected
models at [q 33]. Notice that we can also apply local reparametrizations [37] when we are sampling
<<FORMULA>>
i i and sa sb by exploiting properties of the log-normal distribution 7 and thus forming the
implied:
<<FORMULA>> (17)
As a threshold rule for group pruning we will use the negative log-mode 8 of the local log-normal r.v.
<<FORMULA>> , i.e. prune when <<FORMULA>>, with <<FORMULA>>. This ignores <<FORMULA>> and <<FORMULA>>, but nonetheless we found <<FORMULA>> dependencies among the zi elements induced by the common scale
that it works well in practice. Similarly with the group normal-Jeffreys prior, we will replace the
distribution overWat each layer with the masked variational posterior mean during test time:
<<FORMULA>>; (19)
wheremis a binary mask determined according to the aforementioned threshold,MW are the means
ofq(W~)and;2 are the means and variances of the local log-normals over <<FORMULA>>. Furthermore,
similarly to the group normal-Jeffreys approach, we will use the variational posterior marginal
variances:
<<FORMULA>>; (20)
to compute the final bit precision for the entire weight matrix W.
5 Experiments
We validated the compression and speed-up capabilities of our models on the well-known architectures
of LeNet-300-100 [41], LeNet-5-Caffe 9 on MNIST [42] and, similarly with [51], VGG [63]10 on
CIFAR 10 [38]. The groups of parameters were constructed by coupling the scale variables for each
filter for the convolutional layers and for each input neuron for the fully connected layers. We provide
the algorithms that describe the forward pass using local reparametrizations for fully connected
and convolutional layers with each of the employed approximate posteriors at appendix F. For the
horseshoe prior we set the scale 0 of the global half-Cauchy prior to a reasonably small value, e.g.
0 = 1e5. This further increases the prior mass at zero, which is essential for sparse estimation
and compression. We also found that constraining the standard deviations as described at [46] and
“warm-up" [65] helps in avoiding bad local optima of the variational objective. Further details about
the experimental setup can be found at Appendix A. Determining the threshold for pruning can be
easily done with manual inspection as usually there are two well separated clusters (signal and noise).
We provide a sample visualization at Appendix E.
5.1 Architecture learning & bit precisions
We will first demonstrate the group sparsity capabilities of our methods by illustrating the learned
architectures at Table 1, along with the inferred bit precision per layer. As we can observe, our
methods infer significantly smaller architectures for the LeNet-300-100 and LeNet-5-Caffe, compared
to Sparse Variational Dropout, Generalized Dropout and Group Lasso. Interestingly, we observe
that for the VGG network almost all of big 512 feature map layers are drastically reduced to around
10 feature maps whereas the initial layers are mostly kept intact. Furthermore, all of the Bayesian
methods considered require far fewer than the standard 32 bits per-layer to represent the weights,
sometimes even allowing for 5 bit precisions.
The product of log-normal r.v.s is another log-normal and a power of a log-normal r.v. is another log-normal.
Empirically, it slightly better separates the scales compared to the negative log-mean <<FORMULA>>.
[58]D. J. Rezende, S. Mohamed, and D. Wierstra. Stochastic backpropagation and approximate inference in
deep generative models. InProceedings of the 31th International Conference on Machine Learning, ICML
2014, Beijing, China, 21-26 June 2014, pages 1278–1286, 2014.
[59]J. Rissanen. Modeling by shortest data description.Automatica, 14(5):465–471, 1978.
[60]J. Rissanen. Stochastic complexity and modeling.The annals of statistics, pages 1080–1100, 1986.
[61]S. Scardapane, D. Comminiello, A. Hussain, and A. Uncini. Group sparse regularization for deep neural
networks.arXiv preprint arXiv:1607.00485, 2016.
[62]S. Shi and X. Chu. Speeding up convolutional neural networks by exploiting the sparsity of rectifier units.
arXiv preprint arXiv:1704.07724, 2017.
[63]K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition.
ICLR, 2015.
[64]M. Sites. Ieee standard for floating-point arithmetic. 2008.
[65]C. K. Sønderby, T. Raiko, L. Maaløe, S. K. Sønderby, and O. Winther. Ladder variational autoencoders.
arXiv preprint arXiv:1602.02282, 2016.
[66]S. Srinivas and R. V. Babu. Generalized dropout.arXiv preprint arXiv:1611.06791, 2016.
[67]N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: A simple way to
prevent neural networks from overfitting.The Journal of Machine Learning Research, 15(1):1929–1958,
2014.
[68]V. Sze, Y.-H. Chen, T.-J. Yang, and J. Emer. Efficient processing of deep neural networks: A tutorial and
survey.arXiv preprint arXiv:1703.09039, 2017.
[69]R. Tibshirani. Regression shrinkage and selection via the lasso.Journal of the Royal Statistical Society.
Series B (Methodological), pages 267–288, 1996.
[70]K. Ullrich, E. Meeds, and M. Welling. Soft weight-sharing for neural network compression.ICLR, 2017.
[71]G. Venkatesh, E. Nurvitadhi, and D. Marr. Accelerating deep convolutional networks using low-precision
and sparsity.arXiv preprint arXiv:1610.00324, 2016.
[72]C. S. Wallace. Classification by minimum-message-length inference. InInternational Conference on
Computing and Information, pages 72–81. Springer, 1990.
[73]W. Wen, C. Wu, Y. Wang, Y. Chen, and H. Li. Learning structured sparsity in deep neural networks. In
Advances In Neural Information Processing Systems, pages 2074–2082, 2016.
[74]T.-J. Yang, Y.-H. Chen, and V. Sze. Designing energy-efficient convolutional neural networks using
energy-aware pruning.CVPR, 2017.
[75]S. Zagoruyko and N. Komodakis. Wide residual networks.arXiv preprint arXiv:1605.07146, 2016.
[76]C. Zhu, S. Han, H. Mao, and W. J. Dally. Trained ternary quantization.ICLR, 2017.
Appendix
A. Detailed experimental setup
We implemented our methods in Tensorflow [1] and optimized the variational parameters using
Adam [35] with the default hyperparameters. The means of the conditional Gaussian <<q(W|z)>>
Table 3: Floating point formats Bits per Exponent
<<TABLE>>
were initialized with the scheme proposed at [26], whereas the log of the standard deviations were
initialized by sampling from N(9;1e4). The parameters of q(z) were initialized such that the
overall mean of zise 1 and the overall variance is very low (1e^8); this ensures that all of the
groups are active during the initial training iterations.
As for the standard deviation constraints; for the LeNet-300-100 architecture we constrained the
standard deviation of the first layer to be 0:2 whereas for the LeNet-5-Caffe we constrained
the standard deviation of the first layer to be 0:5. The remaining standard deviations were left
unconstrained. For the VGG network we constrained the standard deviations of the 64 and 128
feature map layers to be 0:1, the standard deviations of the 256 feature map layers to be0:2
and left the rest of the standard deviations unconstrained. We also found beneficial the incorporation
of “warm-up” [65], i.e we annealed the negative KL-divergence from the prior to the approximate
posterior with a linear schedule for the first 100 epochs. We initialized the means of the approximate
posterior by the weights and biases obtained from a VGG network trained with batch normalization
and dropout on CIFAR 10. For our method we disabled batch-normalization during training.
As for preprocessing the data; for MNIST the only preprocessing we did was to rescale the digits to
lie at the [-1,1] range and for CIFAR 10 we used the preprocessed dataset provided by [75].
Furthermore, do note that by pruning a given filter at a particular convolutional layer we can also
prune the parameters corresponding to that feature map for the next layer. This similarly holds for
fully connected layers; if we drop a given input neuron then the weights corresponding to that node
from the previous layer can also be pruned.
B. Standards for Floating-Point Arithmetic
Floating points values eventually need to be represented in a binary basis in a computer. The most
common standard today is the IEEE 754-2008 convention [64]. It definesx-bit base-2 formats,
officially referred to as binaryx, withx2 f16;32;64;128g. The formats are also widely known as
half, single, double and quadruple precision floats, respectively and used in almost all programming
languages as a standard. The format considers 3 kinds of bits: one sign bit,wexponent bits andp
precision bits.
<<FIGURE>>
Figure 2: A symbolic representation of the binaryxformat [64].
The Sign bit determines the sign of the number to be represented. The exponentEis anw-bit signed
integer, e.g. for single precisionw= 8and thusE2[127;128]. In practice, exponents range from
is smaller since the first and the last number are reserved for special numbers. The true significand or
mantissa includes t bits on the right of the binary point. There is an implicit leading bit with value
one. A values is consequently decomposed as follows
<<FORMULA>> (21)
<<FORMULA>> (22)
In table 3, we summarize common and less common floating point formats.
There is however the possibility to design a self defined format. There are 3 important quantities
when choosing the right specification: overflow, underflow and unit round off also known as machine
precision. Each one can be computed knowing the number of exponent and significant bits. in
our work for example we consider a format that uses significantly less exponent bits since network
parameters usually vary between [-10,10]. We set the unit round off equal to the precision and thus
can compute the significant bits necessary to represent a specific weight.
Beyond designing a tailored floating point format for deep learning, recent work also explored the
possibility of deep learning with mixed formats [43,23]. For example, imagine the activations having
high precision while weights can be low precision.
C. Shrinkage properties of the normal-Jeffreys and horseshoe priors
<<FIGURE>>
Figure 3: Comparison of the behavior of the log-uniform / normal-Jeffreys (NJ) prior and the
horseshoe (HS) prior (wheres= 1). Both priors behave similarly at zero but the normal-Jeffreys has
an extremely heavy tail (thus making it non-normalizable).
In this section we will provide some insights about the behavior of each of the priors we employ by
following the excellent analysis of [8]; we can perform a change of variables and express the scale
mixture distribution of eq.3 in the main paper in terms of a shrinkage coefficient,
<<FORMULA>> (23)
It is easy to observe that eq. 23 corresponds to a continuous relaxation of the spike-and-slab prior:
when <<= 0>> we have that <<FORMULA>>, i.e. no shrinkage/regularization forw, when
<<= 1>> we have that <<FORMULA>>, i.e.wis exactly zero, and when <<=1>> we have that <<FORMULA>>. Now by examining the implied prior on the shrinkage coefficient for both
the log-uniform and the horseshoe priors we can better study their behavior. As it is explained at
the half-Cauchy prior onzcorresponds to a beta prior on the shrinkage coefficient, <<FORMULA>>,
whereas the normal-Jeffreys / log-uniform prior onzcorresponds <<top() =B(;)>> with <<FORMULA>>.
The densities of both of these distributions can be seen at Figure 3b. As we can observe, the log-
uniform prior posits a distribution that concentrates almost all of its mass at either0or1,
essentially either pruning the parameter or keeping it close to the maximum likelihood estimate due
<<FORMULA>>. In contrast the horseshoe prior maintains enough probability mass for
the in-between values of and thus can, potentially, offer better regularization and generalization.
D. Negative KL-divergences for log-normal approximating posteriors
Le <<FORMULA>> be a log-normal approximating posterior. Here we will derive the negative
KL-divergences toq(z)from inverse gamma, gamma and half-normal distributions.
Letp(z)be an inverse gamma distribution, i.e. <<p(z) =IG(;)>>. The negative KL-divergence can
be expressed as follows:
<<FORMULA>> (24)
The second term is the entropy of the log-normal distribution which has the following form:
<<FORMULA>> (25)
The first term is the negative cross-entropy of the log-normal approximate posterior from the inverse-
Gamma prior:
<<FORMULA>> (26)
<<FORMULA>> (27)
Since the natural logarithm of a log-normal distribution <<FORMULA>> follows a normal distribution
<<FORMULA>> we have that <<FORMULA>>. Furthermore we have that <<FORMULA>> then <<FORMULA>>, therefore
<<FORMULA>>. Putting everything together we have that:
<<FORMULA>> (28)
Therefore the negative KL-divergence is:
<<FORMULA>> (29)
Now let p(z) be a Gamma prior, i.e. <<p(z) =G(;)>>. We have that the negative cross-entropy
changes to:
<<FORMULA>> (30)
<<FORMULA>> (31)
<<FORMULA>> (32)2
Therefore the negative KL-divergence is:
<<FORMULA>> (33)
Now, by employing the aforementioned we can express the negative KL-divergence from
<<FORMULA>> to <<FORMULA>> as follows:
<<FORMULA>>
with the KL-divergence for the weight distribution <<q (W~)>> given by eq.8 in the main paper.
E. Visualizations
<<FIGURE>>
Figure 4: Distribution of the thresholds for the Sparse Variational Dropout 4a, Bayesian Compression
with group normal-Jeffreys (BC-GNJ) 4b and group Horseshoe (BC-GHS) 4c priors for the three
layer LeNet-300-100 architecture. It is easily observed that there are usually two well separable
groups with BC-GNJ and BC-GHS, thus making the choice for the threshold easy. Smaller values
indicate signal whereas larger values indicate noise (i.e. useless groups).
<<FIGURE>>
Figure 5: Distribution of the bit precisions for the Sparse Variational Dropout 5a, Bayesian Com-
pression with group normal-Jeffreys (BC-GNJ) 5b and group Horseshoe (BC-GHS) 5c priors for the
three layer LeNet-300-100 architecture. All of the methods usually require far fewer than 32bits for
the weights.
F. Algorithms for the feedforward pass
Algorithms 1, 2, 3, 4 describe the forward pass using local reparametrizations for fully connected and
convolutional layers with the approximate posteriors for the Bayesian Compression (BC) with group
normal-Jeffreys (BC-GNJ) and group Horseshoe (BC-GHS) priors employed at the experiments. For
the fully connected layers we coupled the scales for each input neuron whereas for the convolutional
we couple the scales for each output feature map.Mw ;w are the means and variances of each layer,
His a minibatch of activations of sizeK. For the first layer we have thatH=XwhereXis the
minibatch of inputs. For the convolutional layersNf are the number of convolutional filters,is the
convolution operator and we assume the [batch, height, width, feature maps] convention.
Algorithm 1 Fully connected BC-GNJ layer h.
<<ALGORITHM>>
Algorithm 2Convolutional BC-GNJ layerh.
<<ALGORITHM>>
Algorithm 3 Fully connected BC-GHS layerh.
<<ALGORITHM>>
Algorithm 4Convolutional BC-GHS layerh.
<<ALGORITHM>>
<<END>> <<END>> <<END>>
<<START>> <<START>> <<START>>
Channel Pruning for Accelerating Very Deep Neural Networks
Yihui He* Xiangyu Zhang Jian Sun
Xifian Jiaotong University Megvii Inc. Megvii Inc.
Xifian, 710049, China Beijing, 100190, China Beijing, 100190, China
In this paper, we introduce a new channel pruning method to accelerate very deep convolutional neural net.works. Given a trained CNN model, we propose an it.erative two-step algorithm to effectively prune each layer, by a LASSO regression based channel selection and least square reconstruction. We further generalize this algorithm to multi-layer and multi-branch cases. Our method re.duces the accumulated error and enhance the compatibility with various architectures. Our pruned VGG-16 achieves the state-of-the-art results by 5. speed-up along with only 0.3% increase of error. More importantly, our method is able to accelerate modern networks like ResNet, exception and suffers only 1.4%, 1.0% accuracy loss under 2. speed.up respectively, which is significant.
1. Introduction
Recent CNN acceleration works fall into three categories: optimized implementation (e.g., FFT [47]), quantization (e.g., BinaryNet [8]), and structured simplification that convert a CNN into compact one [22]. This work focuses on the last one.
Structured simplification mainly involves: tensor factorization [22], sparse connection [17], and channel pruning [48]. Tensor factorization factorizes a convolutional layer into several efficient ones (Fig. 1(c)). However, feature map width (number of channels) could not be reduced, which makes it difficult to decompose 1 . 1 convolutional layer favored by modern networks (e.g., GoogleNet [45], ResNet [18], Xception [7]). This type of method also intro.duces extra computation overhead. Sparse connection deactivates connections between neurons or channels (Fig. 1(b)). Though it is able to achieves high theoretical speed-up ratio, the sparse convolutional layers have an fiirregularfi shape which is not implementation friendly. In contrast, channel pruning directly reduces feature map width, which shrinks
<<FIGURE>>
Figure 1. Structured simplification methods that accelerate CNNs:
(a) a network with 3 conv layers. (b) sparse connection deactivates some connections between channels. (c) tensor factorization factorizes a convolutional layer into several pieces. (d) channel pruning reduces number of channels in each layer (focus of this paper).
a network into thinner one, as shown in Fig. 1(d). It is efficient on both CPU and GPU because no special implementation is required.
Pruning channels is simple but challenging because re.moving channels in one layer might dramatically change the input of the following layer. Recently, training-based channel pruning works [1, 48] have focused on imposing sparse constrain on weights during training, which could adaptively determine hyper-parameters. However, training from scratch is very costly and results for very deep CNNs on ImageNet have been rarely reported. Inference-time at.tempts [31, 3] have focused on analysis of the importance of individual weight. The reported speed-up ratio is very limited.
In this paper, we propose a new inference-time approach for channel pruning, utilizing redundancy inter channels. Inspired by tensor factorization improvement by feature maps reconstruction [52], instead of analyzing filter weights [22, 31], we fully exploits redundancy within feature maps. Specifically, given a trained CNN model, pruning each layer is achieved by minimizing reconstruction error on its output feature maps, as showed in Fig. 2. We solve this mini.
<<FIGURE>>
Figure 2. Channel pruning for accelerating a convolutional layer. We aim to reduce the width of feature map B, while minimizing the reconstruction error on feature map C. Our optimization algorithm (Sec. 3.1) performs within the dotted box, which does not involve nonlinearity. This figure illustrates the situation that two channels are pruned for feature map B. Thus corresponding channels of filters W can be removed. Furthermore, even though not directly optimized by our algorithm, the corresponding filters in the previous layer can also be removed (marked by dotted filters). c, n: number of channels for feature maps B and C, kh . kw : kernel size.
minimization problem by two alternative steps: channels selection and feature map reconstruction. In one step, we figure out the most representative channels, and prune redundant ones, based on LASSO regression. In the other step, we reconstruct the outputs with remaining channels with linear least squares. We alternatively take two steps. Further, we approximate the network layer-by-layer, with accumulated error accounted. We also discuss methodologies to prune multi-branch networks (e.g., ResNet [18], exception [7]).
For VGG-16, we achieve 4. acceleration, with only 1.0% increase of top-5 error. Combined with tensor factorization, we reach 5. acceleration but merely suffer 0.3% increase of error, which outperforms previous state-of-the.arts. We further speed up ResNet-50 and Xception-50 by 2. with only 1.4%, 1.0% accuracy loss respectively.
2. Related Work
There has been a significant amount of work on accelerating CNNs. Many of them fall into three categories: optimized implementation [4], quantization [40], and structured simplification [22].
Optimized implementation based methods [35, 47, 27, 4] accelerate convolution, with special convolution algorithms like FFT [47]. Quantization [8, 40] reduces floating point computational complexity.
Sparse connection eliminates connections between neurons [17, 32, 29, 15, 14]. [51] prunes connections based on weights magnitude. [16] could accelerate fully connected layers up to 50.. However, in practice, the actual speed-up maybe very related to implementation.
Tensor factorization [22, 28, 13, 24] decompose weights into several pieces. [50, 10, 12] accelerate fully connected layers with truncated SVD. [52] factorize a layer into 3 . 3 and 1 . 1 combination, driven by feature map redundancy.
Channel pruning removes redundant channels on feature maps. There are several training-based approaches. [1, 48] regularize networks to improve accuracy. Channel-wise SSL [48] reaches high compression ratio for first few conv layers of LeNet [30] and AlexNet [26]. However, training-based approaches are more costly, and the effectiveness for very deep networks on large datasets is rarely exploited.
Inference-time channel pruning is challenging, as re.ported by previous works [2, 39]. Some works [44, 34, 19] focus on model size compression, which mainly operate the fully connected layers. Data-free approaches [31, 3] results for speed-up ratio (e.g., 5.) have not been reported, and requires long retraining procedure. [3] select channels via over 100 random trials, however it need long time to eval.ate each trial on a deep network, which makes it infeasible to work on very deep models and large datasets. [31] is even worse than naive solution from our observation sometimes (Sec. 4.1.1).
3. Approach
In this section, we first propose a channel pruning algorithm for a single layer, then generalize this approach to multiple layers or the whole model. Furthermore, we dis.cuss variants of our approach for multi-branch networks.
3.1. Formulation
Fig. 2 illustrates our channel pruning algorithm for a sin.gle convolutional layer. We aim to reduce the width of feature map B, while maintaining outputs in feature map
C. Once channels are pruned, we can remove correspond.ing channels of the filters that take these channels as in.put. Also, filters that produce these channels can also be removed. It is clear that channel pruning involves two key points. The first is channel selection, since we need to select most representative channels to maintain as much information. The second is reconstruction. We need to reconstruct the following feature maps using the selected channels.
Motivated by this, we propose an iterative two-step algorithm. In one step, we aim to select most representative channels. Since an exhaustive search is infeasible even for tiny networks, we come up with a LASSO regression based method to figure out representative channels and prune redundant ones. In the other step, we reconstruct the outputs with remaining channels with linear least squares. We alter.natively take two steps.
Formally, to prune a feature map with c channels, we consider applying n.c.kh .kw convolutional filters W on <<FORMULA>> input volumes X sampled from this feature map, which produces N . n output matrix Y. Here, N is the number of samples, n is the number of output channels, and kh,kw are the kernel size. For simple representation, bias term is not included in our formulation. To prune the
..
input channels from c to desired <<FORMULA>>, while minimizing reconstruction error, we formulate our problem as follow:
<<FORMULA>> (1)
F is Frobenius norm. <<FORMULA>> matrix sliced from ith channel of input volumes X_i, i =1, ..., c. W_i is n . filter weights sliced from ith channel of W. is coefficient vector of length c for channel selection, and .i is ith entry of . Notice that, if .i =0, X_i will be no longer useful, which could be safely pruned from feature map. W_i could also be removed. Optimization Solving this minimization problem in Eqn. 1 is NP-hard. Therefore, we relax the l_0 to l_1 regularization:
<<FORMULA>> (2)
. is a penalty coefficient. By increasing l, there will be more zero terms in and one can get higher speed-up ratio. We also add a constrain .i WiF =1 to this formulation, which avoids trivial solution.
Now we solve this problem in two folds. First, we fix W, solve for channel selection. Second, we fix , solve W to reconstruct error.
(i) The subproblem of . In this case, W is fixed. We solve for channel selection. This problem can be solved by LASSO regression [46, 5], which is widely used for model selection.
<<FORMULA>> (3)
.
Here Zi =XiWi (size N .n). We will ignore ith channels if .i =0.
(ii) The subproblem of W. In this case, is fixed. We utilize the selected channels to minimize reconstruction error. We can find optimized solution by least squares:
<<FORMULA>>. (4)
Here <<FORMULA>> (size N.). W is n reshaped W, <<FORMULA>>. After obtained result W, it is reshaped back to W. Then we assign <<FORMULA>>. Constrain <<FORMULA>> satisfies.
We alternatively optimize (i) and (ii). In the beginning, W is initialized from the trained model, <<FORMULA>>, namely no penalty, and <<k = c>>. We gradually increase <<FORMULA>> For each
change of <<FORMULA>>, we iterate these two steps until k is stable.
After <<FORMULA>> satisfies, we obtain the final solution W from <<FORMULA>> In practice, we found that the two steps iteration is time consuming. So we apply (i) multiple times,
<<FORMULA>>
until <<FORMULA>> satisfies. Then apply (ii) just once, to obtain
<<FORMULA>>
the final result. From our observation, this result is comparable with two steps iterations. Therefore, in the following experiments, we adopt this approach for efficiency.
Discussion: Some recent works [48, 1, 17] (though train.
ing based) also introduce .1-norm or LASSO. However, we must emphasis that we use different formulations. Many of them introduced sparsity regularization into training loss, instead of explicitly solving LASSO. Other work [1] solved LASSO, while feature maps or data were not considered during optimization. Because of these differences, our ap.proach could be applied at inference time.
3.2. Whole Model Pruning
Inspired by [52], we apply our approach layer by layer sequentially. For each layer, we obtain input volumes from the current input feature map, and output volumes from the output feature map of the un-pruned model. This could be formalized as:
<<FORMULA>> (5)
Different from Eqn. 1, Y is replaced by Y . , which is from feature map of the original model. Therefore, the accumulated error could be accounted during sequential pruning.
3.3. Pruning Multi.Branch Networks
The whole model pruning discussed above is enough for single-branch networks like LeNet [30], AlexNet [26] and VGG Nets [43]. However, it is insufficient for multi-branch networks like GoogLeNet [45] and ResNet [18]. We mainly focus on pruning the widely used residual structure (e.g., ResNet [18], Xception [7]). Given a residual block shown in Fig. 3 (left), the input bifurcates into shortcut and residual branch. On the residual branch, there are several convolutional layers (e.g., 3 convolutional layers which have spatial size of 1 . 1, 3 . 3, 1 . 1, Fig. 3, left). Other layers except the first and last layer can be pruned as is described previously. For the first layer, the challenge is that the large input feature map width (for ResNet, 4 times of its output) can it be easily pruned, since it is shared with shortcut. For the last layer, accumulated error from the shortcut is hard to be recovered, since there is no parameter on the shortcut. To address these challenges, we propose several variants of our approach as follows.
<<FIGURE>>
Figure 3. Illustration of multi-branch enhancement for residual block. Left: original residual block. Right: pruned residual block with enhancement, cx denotes the feature map width. Input channels of the first convolutional layer are sampled, so that the large input feature map width could be reduced. As for the last layer, rather than approximate Y2 , we try to approximate <<Y1+Y2>> directly (Sec. 3.3 Last layer of residual branch).
Last layer of residual branch: Shown in Fig. 3, the output layer of a residual block consists of two inputs: feature map Y1 and Y2 from the shortcut and residual branch. We aim to recover Y1 +Y2 for this block. Here, Y1, Y2 are the original feature maps before pruning. Y2 could be approximated as in Eqn. 1. However, shortcut branch is parameter-free, then Y1 could not be recovered directly. To compensate this error, the optimization goal of the last layer is changed from Y2 to Y1 .Y . +Y2, which does not change
<<FORMULA>>
our optimization. Here, Y . is the current feature map after
<<FORMULA>>
previous layers pruned. When pruning, volumes should be sampled correspondingly from these two branches.
First layer of residual branch: Illustrated in Fig. 3(left), the input feature map of the residual block could not be pruned, since it is also shared with the short.cut branch. In this condition, we could perform feature map sampling before the first convolution to save computation. We still apply our algorithm as Eqn. 1. Differently, we sample the selected channels on the shared feature maps to construct a new input for the later convolution, shown in Fig. 3(right). Computational cost for this operation could be ignored. More importantly, after introducing feature map sampling, the convolution is still irregular.
Filter-wise pruning is another option for the first con.volution on the residual branch. Since the input channels of parameter-free shortcut branch could not be pruned, we apply our Eqn. 1 to each filter independently (each fil.ter chooses its own representative input channels). Under single layer acceleration, filter-wise pruning is more accurate than our original one. From our experiments, it im.proves 0.5% top-5 accuracy for 2. ResNet-50 (applied on the first layer of each residual branch) without fine-tuning. However, after fine-tuning, there is no noticeable improvement. In addition, it outputs irregular convolutional layers, which need special library implementation support. We do not adopt it in the following experiments.
4. Experiment
We evaluation our approach for the popular VGG Nets [43], ResNet [18], Xception [7] on ImageNet [9], CIFAR.10 [25] and PASCAL VOC 2007 [11].
For Batch Normalization [21], we first merge it into convolutional weights, which do not affect the outputs of the networks. So that each convolutional layer is followed by ReLU [36]. We use Caffe [23] for deep network evaluation, and scikit-learn [38] for solvers implementation. For channel pruning, we found that it is enough to extract 5000 images, and 10 samples per image. On ImageNet, we evaluate the top-5 accuracy with single view. Images are re.sized such that the shorter side is 256. The testing is on center crop of 224 . 224 pixels. We could gain more per.formance with fine-tuning. We use a batch size of 128 and
.5
learning rate 1e^-4. We fine-tune our pruned models for 10 epochs. The augmentation for fine-tuning is random crop of 224 . 224 and mirror.
4.1. Experiments with VGG.16
VGG-16 [43] is a 16 layers single path convolutional neural network, with 13 convolutional layers. It is widely used in recognition, detection and segmentation, etc. Single view top-5 accuracy for VGG-16 is 89.9%1.
4.1.1 Single Layer Pruning
In this subsection, we evaluate single layer acceleration performance using our algorithm in Sec. 3.1. For better under.standing, we compare our algorithm with two naive chan.nel selection strategies. first k selects the first k channels. max response selects channels based on corresponding filters that have high absolute weights sum [31]. For fair com.parison, we obtain the feature map indexes selected by each of them, then perform reconstruction (Sec. 3.1 (ii)). We hope that this could demonstrate the importance of channel selection. Performance is measured by increase of error af.ter a certain layer is pruned without fine-tuning, shown in Fig. 4.
As expected, error increases as speed-up ratio increases. Our approach is consistently better than other approaches in different convolutional layers under different speed-up ra.tio. Unexpectedly, sometimes max response is even worse than first k. We argue that max response ignores correlations between different filters. Filters with large absolute weight may have strong correlation. Thus selection based on filter weights is less meaningful. Correlation on feature maps is worth exploiting. We can find that channel selection http://www.vlfeat.org/matconvnet/pretrained/
<<FIGURE>>
Figure 4. Single layer performance analysis under different speed-up ratios (without fine-tuning), measured by increase of error. To verify the importance of channel selection referred in Sec. 3.1, we considered two naive baselines. first k selects the first k feature maps. max response selects channels based on absolute sum of corresponding weights filter [31]. Our approach is consistently better (smaller is better).
<<TABLE>>
Table 1. Accelerating the VGG-16 model [43] using a speedup ratio of 2., 4., or 5. (smaller is better).
affects reconstruction error a lot. Therefore, it is important for channel pruning.
Also notice that channel pruning gradually becomes hard, from shallower to deeper layers. It indicates that shallower layers have much more redundancy, which is consistent with [52]. We could prune more aggressively on shallower layers in whole model acceleration.
4.1.2 Whole Model Pruning
Shown in Table 1, whole model acceleration results under 2., 4., 5. are demonstrated. We adopt whole model pruning proposed in Sec. 3.2. Guided by single layer experiments above, we pruning more aggressive for shallower layers. Remaining channels ratios for shallow lay.ers (conv 1_x to conv 3_x) and deep layers (conv4_x) is 1:1.5. conv 5_x are not pruned, since they only con.tribute 9% computation in total and are not redundant.
After fine-tuning, we could reach 2. speed-up without losing accuracy. Under 4., we only suffers 1.0% drops. Consistent with single layer analysis, our approach outperforms previous channel pruning approach (Li et al. [31]) by large margin. This is because we fully exploits channel redundancy within feature maps. Compared with tensor factorization algorithms, our approach is better than Jaderberg et al. [22], without fine-tuning. Though worse than Asym. [52], our combined model outperforms its combined Asym. 3D (Table 2). This may indicate that channel pruning is more challenging than tensor factorization, since removing channels in one layer might dramatically change the input of the following layer. However, channel pruning keeps the original model architecture, do not introduce additional layers, and the absolute speed-up ratio on GPU is much higher (Table 3).
Since our approach exploits a new cardinality, we further combine our channel pruning with spatial factorization [22] and channel factorization [52]. Demonstrated in Table 2,
<<TABLE>>
Table 2. Performance of combined methods on the VGG-16 model
[43] using a speed-up ratio of 4. or 5.. Our 3C solution outperforms previous approaches (smaller is better).
our 3 cardinalities acceleration (spatial, channel factorization, and channel pruning, denoted by 3C) outperforms previous state-of-the-arts. Asym. 3D [52] (spatial and chan.nel factorization), factorizes a convolutional layer to three parts: <<FORMULA>>.
We apply spatial factorization, channel factorization, and our channel pruning together sequentially layer-by-layer. We fine-tune the accelerated models for 20 epochs, since they are 3 times deeper than the original ones. After fine-tuning, our 4. model suffers no degradation. Clearly, a combination of different acceleration techniques is better than any single one. This indicates that a model is redundant in each cardinality.
4.1.3 Comparisons of Absolute Performance
We further evaluate absolute performance of acceleration on GPU. Results in Table 3 are obtained under Caffe [23], CUDA 8 [37] and cuDNN5 [6], with a mini-batch of 32 on a GPU (GeForce GTX TITAN X). Results are averaged from 50 runs. Tensor factorization approaches decompose weights into too many pieces, which heavily increase over.head. They could not gain much absolute speed-up. Though our approach also encountered performance decadence, it generalizes better on GPU than other approaches. Our re.sults for tensor factorization differ from previous research [52, 22], maybe because current library and hardware prefer single large convolution instead of several small ones.
4.1.4 Comparisons with Training from Scratch
Though training a compact model from scratch is time-consuming (usually 120 epochs), it worths comparing our approach and from scratch counterparts. To be fair, we evaluated both from scratch counterpart, and normal setting net.work that has the same computational complexity and same architecture.
Shown in Table 4, we observed that it is difficult for from scratch counterparts to reach competitive accuracy. our model outperforms from scratch one. Our approach successfully picks out informative channels and constructs highly compact models. We can safely draw the conclusion that the same model is difficult to be obtained from scratch. This coincides with architecture design researches [20, 1] that the model could be easier to train if there are more channels in shallower layers. However, channel prun.ing favors shallower layers.
For from scratch (uniformed), the filters in each layers is reduced by half (eg. reduce conv1_1 from 64 to 32). We can observe that normal setting networks of the same complexity couldn't reach same accuracy either. This consolidates our idea that there is much redundancy in networks while training. However, redundancy can be opt out at inference-time. This maybe an advantage of inference-time acceleration approaches over training-based approaches.
Notice that there is a 0.6% gap between the from scratch model and uniformed one, which indicates that there is room for model exploration. Adopting our approach is much faster than training a model from scratch, even for a thin.ner one. Further researches could alleviate our approach to do thin model exploring.
4.1.5 Acceleration for Detection
VGG-16 is popular among object detection tasks [42, 41, 33]. We evaluate transfer learning ability of our 2./4. pruned VGG-16, for Faster R-CNN [42] object detections. PASCAL VOC 2007 object detection benchmark [11] contains 5k trainable images and 5k test images. The performance is evaluated by mean Average Precision (mAP). In our experiments, we first perform channel pruning for VGG-16 on the ImageNet. Then we use the pruned model as the pre-trained model for Faster R-CNN.
The actual running time of Faster R-CNN is 220ms / im.age. The convolutional layers contributes about 64%. We got actual time of 94ms for 4. acceleration. From Table 5, we observe 0.4% mAP drops of our 2. model, which is not harmful for practice consideration.
4.2. Experiments with Residual Architecture Nets
For Multi-path networks [45, 18, 7], we further explore the popular ResNet [18] and latest Xception [7], on Ima.geNet and CIFAR-10. Pruning residual architecture nets is more challenging. These networks are designed for both efficiency and high accuracy. Tensor factorization algorithms [52, 22] have difficult to accelerate these model. Spatially, 1 . 1 convolution is favored, which could hardly be factorized.
4.2.1 ResNet Pruning
ResNet complexity uniformly drops on each residual block. Guided by single layer experiments (Sec. 4.1.1), we still prefer reducing shallower layers heavier than deeper ones.
Following similar setting as Filter pruning [31], we keep 70% channels for sensitive residual blocks (res5 and blocks close to the position where spatial size
<<TABLE>>
Table 3. GPU acceleration comparison. We measure forward-pass time per image. Our approach generalizes well on GPU (smaller is better).
<<TABLE>>
Table 4. Comparisons with training from scratch, under 4. acceleration. Our fine-tuned model outperforms scratch trained counterparts (smaller is better).
<<TABLE>>
Table 5.Acceleration for Faster R-CNN detection.
<<TABLE>>
Table 6. 2. acceleration for ResNet-50 on ImageNet, the base.line network is top-5 accuracy is 92.2% (one view). We improve performance with multi-branch enhancement (Sec. 3.3, smaller is better).
change, e.g. res3a,res3d). As for other blocks, we keep 30% channels. With multi-branch enhancement, we prune branch 2a more aggressively within each residual block. The remaining channels ratios for branch 2a,branch 2b,branch 2c is 2:4:3 (e.g., Given 30%, we keep 40%, 80%, 60% respectively).
We evaluate performance of multi-branch variants of our approach (Sec. 3.3). From Table 6, we improve 4.0% with our multi-branch enhancement. This is because we accounted the accumulated error from shortcut connection which could broadcast to every layer after it. And the large input feature map width at the entry of each residual block is well reduced by our feature map sampling.
<<TABLE>>
Table 7. Comparisons for Xception-50, under 2. acceleration ra.tio. The baseline network is top-5 accuracy is 92.8%. Our approach outperforms previous approaches. Most structured simplification methods are not effective on Xception architecture (smaller is better).
4.2.2 Xception Pruning
Since computational complexity becomes important in model design, separable convolution has been payed much attention [49, 7]. Xception [7] is already spatially optimized and tensor factorization on 1 . 1 convolutional layer is destructive. Thanks to our approach, it could still be accelerated with graceful degradation. For the ease of comparison, we adopt Xception convolution on ResNet-50, denoted by Xception-50. Based on ResNet-50, we swap all convolutional layers with spatial conv blocks. To keep the same computational complexity, we increase the input channels of all branch2b layers by 2.. The baseline Xception.50 has a top-5 accuracy of 92.8% and complexity of 4450 MFLOPs.
We apply multi-branch variants of our approach as de.scribed in Sec. 3.3, and adopt the same pruning ratio setting as ResNet in previous section. Maybe because of Xcep.tion block is unstable, Batch Normalization layers must be maintained during pruning. Otherwise it becomes nontrivial to fine-tune the pruned model.
Shown in Table 7, after fine-tuning, we only suffer 1.0% increase of error under 2.. Filter pruning [31] could also apply on Xception, though it is designed for small speed.up ratio. Without fine-tuning, top-5 error is 100%. After training 20 epochs which is like training from scratch, in.creased error reach 4.3%. Our results for Xception-50 are not as graceful as results for VGG-16, since modern net.works tend to have less redundancy by design.
<<TABLE>>
Table 8. 2. speed-up comparisons for ResNet-56 on CIFAR-10, the baseline accuracy is 92.8% (one view). We outperforms previous approaches and scratch trained counterpart (smaller is better).
4.2.3 Experiments on CIFAR-10
Even though our approach is designed for large datasets, it could generalize well on small datasets. We perform experiments on CIFAR-10 dataset [25], which is favored by many acceleration researches. It consists of 50k images for training and 10k for testing in 10 classes.
We reproduce ResNet-56, which has accuracy of 92.8% (Serve as a reference, the official ResNet-56 [18] has ac.curacy of 93.0%). For 2. acceleration, we follow similar setting as Sec. 4.2.1 (keep the final stage unchanged, where the spatial size is 8 . 8). Shown in Table 8, our approach is competitive with scratch trained one, without fine-tuning, under 2. speed-up. After fine-tuning, our result is significantly better than Filter pruning [31] and scratch trained one.
5. Conclusion
To conclude, current deep CNNs are accurate with high inference costs. In this paper, we have presented an inference-time channel pruning method for very deep net.works. The reduced CNNs are inference efficient networks while maintaining accuracy, and only require off-the-shelf libraries. Compelling speed-ups and accuracy are demonstrated for both VGG Net and ResNet-like networks on Im.ageNet, CIFAR-10 and PASCAL VOC.
In the future, we plan to involve our approaches into training time, instead of inference time only, which may also accelerate training procedure.
References
[1] J. M. Alvarez and M. Salzmann. Learning the number of neurons in deep networks. In Advances in Neural Information Processing Systems, pages 2262fi2270, 2016. 1, 2, 3, 6
[2] S. Anwar, K. Hwang, and W. Sung. Structured prun.ing of deep convolutional neural networks. arXiv preprint arXiv:1512.08571, 2015. 2
[3] S. Anwar and W. Sung. Compact deep convolutional neural networks with coarse pruning. arXiv preprint arXiv:1610.09639, 2016. 1, 2
[4] H. Bagherinezhad, M. Rastegari, and A. Farhadi. Lcnn: Lookup-based convolutional neural network. arXiv preprint arXiv:1611.06473, 2016. 2
[5] L. Breiman. Better subset regression using the nonnegative garrote. Technometrics, 37(4):373fi384, 1995. 3
[6] S. Chetlur, C. Woolley, P. Vandermersch, J. Cohen, J. Tran,
B. Catanzaro, and E. Shelhamer. cudnn: Efficient primitives for deep learning. CoRR, abs/1410.0759, 2014. 6
[7] F. Chollet. Xception: Deep learning with depthwise separa.ble convolutions. arXiv preprint arXiv:1610.02357, 2016. 1, 2, 3, 4, 6, 7
[8] M. Courbariaux and Y. Bengio. Binarynet: Training deep neural networks with weights and activations constrained to+ 1 or-1. arXiv preprint arXiv:1602.02830, 2016. 1, 2
[9] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, pages 248fi255. IEEE, 2009. 4
[10] E. L. Denton, W. Zaremba, J. Bruna, Y. LeCun, and R. Fer.gus. Exploiting linear structure within convolutional net.works for efficient evaluation. In Advances in Neural In.formation Processing Systems, pages 1269fi1277, 2014. 2
[11] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 2007 (VOC2007) Results. http://www.pascal.network.org/challenges/VOC/voc2007/workshop/index.html. 4, 6
[12] R. Girshick. Fast r-cnn. In Proceedings of the IEEE Inter.national Conference on Computer Vision, pages 1440fi1448, 2015. 2
[13] Y. Gong, L. Liu, M. Yang, and L. Bourdev. Compress.ing deep convolutional networks using vector quantization. arXiv preprint arXiv:1412.6115, 2014. 2
[14] Y. Guo, A. Yao, and Y. Chen. Dynamic network surgery for efficient dnns. In Advances In Neural Information Process.ing Systems, pages 1379fi1387, 2016. 2
[15] S. Han, X. Liu, H. Mao, J. Pu, A. Pedram, M. A. Horowitz, and W. J. Dally. Eie: efficient inference engine on com.pressed deep neural network. In Proceedings of the 43rd International Symposium on Computer Architecture, pages 243fi254. IEEE Press, 2016. 2
[16] S. Han, H. Mao, and W. J. Dally. Deep compression: Com.pressing deep neural network with pruning, trained quantiza.tion and huffman coding. CoRR, abs/1510.00149, 2, 2015.
2
[17] S. Han, J. Pool, J. Tran, and W. Dally. Learning both weights and connections for efficient neural network. In Advances in Neural Information Processing Systems, pages 1135fi1143, 2015. 1, 2, 3
[18] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learn.ing for image recognition. arXiv preprint arXiv:1512.03385, 2015. 1,2,3,4,6,8
[19] H. Hu, R. Peng, Y.-W. Tai, and C.-K. Tang. Network trim.ming: A data-driven neuron pruning approach towards effi.cient deep architectures. arXiv preprint arXiv:1607.03250, 2016. 2
[20] J. Huang, V. Rathod, C. Sun, M. Zhu, A. Korattikara,
A. Fathi, I. Fischer, Z. Wojna, Y. Song, S. Guadarrama, et al. Speed/accuracy trade-offs for modern convolutional object detectors. arXiv preprint arXiv:1611.10012, 2016. 6
[21] S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015. 4
[22] M. Jaderberg, A. Vedaldi, and A. Zisserman. Speeding up convolutional neural networks with low rank expansions. arXiv preprint arXiv:1405.3866, 2014. 1, 2, 5, 6, 7
[23] Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Gir.shick, S. Guadarrama, and T. Darrell. Caffe: Convolu.tional architecture for fast feature embedding. arXiv preprint arXiv:1408.5093, 2014. 4, 6
[24] Y.-D. Kim, E. Park, S. Yoo, T. Choi, L. Yang, and D. Shin. Compression of deep convolutional neural networks for fast and low power mobile applications. arXiv preprint arXiv:1511.06530, 2015. 2
[25] A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. 2009. 4, 8
[26] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097fi1105, 2012. 2, 3
[27] A. Lavin. Fast algorithms for convolutional neural networks. arXiv preprint arXiv:1509.09308, 2015. 2
[28] V. Lebedev, Y. Ganin, M. Rakhuba, I. Oseledets, and
V. Lempitsky. Speeding-up convolutional neural net.works using fine-tuned cp-decomposition. arXiv preprint arXiv:1412.6553, 2014. 2
[29] V. Lebedev and V. Lempitsky. Fast convnets using group-wise brain damage. arXiv preprint arXiv:1506.02515, 2015.
2
[30] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceed.ings of the IEEE, 86(11):2278fi2324, 1998. 2, 3
[31] H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf. Pruning filters for efficient convnets. arXiv preprint arXiv:1608.08710,2016. 1,2,4,5,6,7,8
[32] B. Liu, M. Wang, H. Foroosh, M. Tappen, and M. Pensky. Sparse convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recogni.tion, pages 806fi814, 2015. 2
[33] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. E. Reed,
C. Fu, and A. C. Berg. SSD: single shot multibox detector. CoRR, abs/1512.02325, 2015. 6
[34] Z. Mariet and S. Sra. Diversity networks. arXiv preprint arXiv:1511.05077, 2015. 2
[35] M. Mathieu, M. Henaff, and Y. LeCun. Fast training of convolutional networks through ffts. arXiv preprint arXiv:1312.5851, 2013. 2
[36] V. Nair and G. E. Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-10), pages 807fi814, 2010. 4
[37] J. Nickolls, I. Buck, M. Garland, and K. Skadron. Scalable parallel programming with CUDA. ACM Queue, 6(2):40fi53, 2008. 6
[38] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel,
B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss,
V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau,
M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Ma.chine learning in Python. Journal of Machine Learning Re.search, 12:2825fi2830, 2011. 4
[39] A. Polyak and L. Wolf. Channel-level acceleration of deep face representations. IEEE Access, 3:2163fi2175, 2015. 2
[40] M. Rastegari, V. Ordonez, J. Redmon, and A. Farhadi. Xnor.net: Imagenet classification using binary convolutional neu.ral networks. In European Conference on Computer Vision, pages 525fi542. Springer, 2016. 2
[41] J. Redmon, S. K. Divvala, R. B. Girshick, and A. Farhadi. You only look once: Unified, real-time object detection. CoRR, abs/1506.02640, 2015. 6
[42] S. Ren, K. He, R. B. Girshick, and J. Sun. Faster R-CNN: towards real-time object detection with region proposal net.works. CoRR, abs/1506.01497, 2015. 6
[43] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014. 3, 4, 5, 6
[44] S. Srinivas and R. V. Babu. Data-free parameter pruning for deep neural networks. arXiv preprint arXiv:1507.06149, 2015. 2
[45] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed,
D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1fi9, 2015. 1, 3, 6
[46] R. Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society. Series B (Methodological), pages 267fi288, 1996. 3
[47] N. Vasilache, J. Johnson, M. Mathieu, S. Chintala, S. Pi.antino, and Y. LeCun. Fast convolutional nets with fbfft: A gpu performance evaluation. arXiv preprint arXiv:1412.7580, 2014. 1, 2
[48] W. Wen, C. Wu, Y. Wang, Y. Chen, and H. Li. Learning structured sparsity in deep neural networks. In Advances In Neural Information Processing Systems, pages 2074fi2082, 2016. 1, 2, 3
[49] S. Xie, R. Girshick, P. Dollfiar, Z. Tu, and K. He. Aggregated residual transformations for deep neural networks. arXiv preprint arXiv:1611.05431, 2016. 7
[50] J. Xue, J. Li, and Y. Gong. Restructuring of deep neural network acoustic models with singular value decomposition. In INTERSPEECH, pages 2365fi2369, 2013. 2
[51] T.-J. Yang, Y.-H. Chen, and V. Sze. Designing energy-efficient convolutional neural networks using energy-aware pruning. arXiv preprint arXiv:1611.05128, 2016. 2
[52] X. Zhang, J. Zou, K. He, and J. Sun. Accelerating very deep convolutional networks for classification and detection. IEEE transactions on pattern analysis and machine intelli.gence, 38(10):1943fi1955, 2016. 1, 2, 3, 5, 6, 7
<<END>> <<END>> <<END>>
<<START>> <<START>> <<START>>
Convex Neural Networks
Yoshua Bengio, Nicolas Le Roux, Pascal Vincent, Olivier Delalleau, Patrice Marcotte
Convexity has recently received a lot of attention in the machine learning
community, and the lack of convexity has been seen as a major disad-
vantage of many learning algorithms, such as multi-layer artificial neural
networks. We show that training multi-layer neural networks in which the
number of hidden units is learned can be viewed as a convex optimization
problem. This problem involves an infinite number of variables, but can be
solved by incrementally inserting a hidden unit at a time, each time finding
a linear classifier that minimizes a weighted sum of errors.
1 Introduction
The objective of this paper is not to present yet another learning algorithm, but rather to point
to a previously unnoticed relation between multi-layer neural networks (NNs),Boosting (Fre-
und and Schapire, 1997) and convex optimization. Its main contributions concern the mathe-
matical analysis of an algorithm that is similar to previously proposed incremental NNs, with
L1 regularization on the output weights. This analysis helps to understand the underlying
convex optimization problem that one is trying to solve.
This paper was motivated by the unproven conjecture (based on anecdotal experience) that
when the number of hidden units is “large”, the resulting average error is rather insensitive to
the random initialization of the NN parameters. One way to justify this assertion is that to re-
ally stay stuck in a local minimum, one must have second derivatives positive simultaneously
in all directions. When the number of hidden units is large, it seems implausible for none of
them to offer a descent direction. Although this paper does not prove or disprove the above
conjecture, in trying to do so we found an interesting characterization of the optimization
problem for NNs as a convex program if the output loss function is convex in the NN out-
put and if the output layer weights are regularized by a convex penalty. More specifically,
if the regularization is the L1 norm of the output layer weights, then we show that a “rea-
sonable” solution exists, involving a finite number of hidden units (no more than the number
of examples, and in practice typically much less). We present a theoretical algorithm that
is reminiscent of Column Generation (Chvatal, 1983), in which hidden neurons are inserted ´
one at a time. Each insertion requires solving a weighted classification problem, very much
like in Boosting (Freund and Schapire, 1997) and in particular Gradient Boosting (Mason
et al., 2000; Friedman, 2001).
Neural Networks, Gradient Boosting, and Column Generation
Denote x~2Rd+1 the extension of vector x2Rd with one element with value 1. What
we call “Neural Network” (NN) here is a predictor for supervised learning of the form
<<FORMULA>> where x is an input vector, <<h_i(x)>> is obtained from a linear dis-
criminant function hi <<FORMULA>> with e.g. <<s(a) = sign(a)>>, or <<s(a) = tanh(a)>> or
<<s(a) = 1>>. A learning algorithm must specify how to select m, the <<FORMULA>>
i ’s and the vi ’s.
The classical solution (Rumelhart, Hinton and Williams, 1986) involves (a) selecting a loss
function Q(^y;y)that specifies how to penalize for mismatches between y^(x)and the ob-
served y’s (target output or target class), (b) optionally selecting a regularization penalty that
favors “small” parameters, and (c) choosing a method to approximately minimize the sum of
the losses on the training data D=f(x1 ;y 1 );:::;(xn ;y n )gplus the regularization penalty.
Note that in this formulation, an output non-linearity can still be used, by inserting it in the
loss function Q. Examples of such loss functions are the quadratic loss jjy^yjj 2 , the hinge
loss <<FORMULA>> (used in SVMs), the cross-entropy loss <<FORMULA>>
(used in logistic regression), and the exponential loss <<FORMULA>> (used in Boosting).
Gradient Boosting has been introduced in (Friedman, 2001) and (Mason et al., 2000) as a
non-parametric greedy-stagewise supervised learning algorithm in which one adds a function
at a time to the current solution <<y^(x)>>, in a steepest-descent fashion, to form an additive model
as above but with the functions hi typically taken in other kinds of sets of functions, such as
those obtained with decision trees. In a stagewise approach, when the (m+1)-th basis <<FORMULA>> is added,
only <<w_m+1>> is optimized (by a line search), like in matching pursuit algorithms. Such
a greedy-stagewise approach is also at the basis of Boosting algorithms (Freund and Schapire,
1997), which is usually applied using decision trees as bases and Qthe exponential loss.
It may be difficult to minimize exactly for wm+1 and hm+1 when the previous bases and
weights are fixed, so (Friedman, 2001) proposes to “follow the gradient” in function space,
i.e., look for a base learner hm+1 that is best correlated with the gradient of the average
loss on the <<FORMULA>> (that would be the residue <<FORMULA>> in the case of the square loss). The
algorithm analyzed here also involves maximizing the correlation between Q0 (the derivative
of Q with respect to its first argument, evaluated on the training predictions) and the next
basis hm+1 . However, we follow a “stepwise”, less greedy, approach, in which all the output
weights are optimized at each step, in order to obtain convergence guarantees.
Our approach adapts the Column Generation principle (Chvatal, 1983), a decomposition´
technique initially proposed for solving linear programs with many variables and few con-
straints. In this framework, active variables, or “columns”, are only generated as they are
required to decrease the objective. In several implementations, the column-generation sub-
problem is frequently a combinatorial problem for which efficient algorithms are available.
In our case, the subproblem corresponds to determining an “optimal” linear classifier.
2 Core Ideas
Informally, consider the set Hof all possible hidden unit functions (i.e., of all possible hidden
unit weight vectors vi ). Imagine a NN that has all the elements in this set as hidden units. We
might want to impose precision limitations on those weights to obtain either a countable or
even a finite set. For such a NN, we only need to learn the output weights. If we end up with
a finite number of non-zero output weights, we will have at the end an ordinary feedforward
NN. This can be achieved by using a regularization penalty on the output weights that yields
sparse solutions, such as the L1 penalty. If in addition the loss function is convex in the output
layer weights (which is the case of squared error, hinge loss, -tube regression loss, and
logistic or softmax cross-entropy), then it is easy to show that the overall training criterion
is convex in the parameters (which are now only the output weights). The only problem is
that there are as many variables in this convex program as there are elements in the set H,
which may be very large (possibly infinite). However, we find that with L1 regularization,
a finite solution is obtained, and that such a solution can be obtained by greedily inserting
one hidden unit at a time. Furthermore, it is theoretically possible to check that the global
optimum has been reached.
Definition 2.1.Let Hbe a set of functions from an input space X to R. Elements of H
can be understood as “hidden units” in a NN. Let Wbe the Hilbert space of functions from
Hto R, with an inner product denoted by <<FORMULA>>. An element of W can be
understood as the output weights vector in a neural network. Let <<h(x):H -> R>> the function
that maps any element <<h_i>> of <<H to h_i(x)>>. <<h(x)>> can be understood as the vector of activations
of hidden units when input x is observed. Let w2 W represent a parameter(the output
weights). The NN prediction is denoted <<FORMULA>>. Let <<Q:R -> RxR>> be a
cost function convex in its first argument that takes a scalar prediction y^(x)and a scalar
target value y and returns a scalar cost. This is the cost to be minimized on example pair
(x;y). Let <<FORMULA>> be the training set. Let <<FORMULA>> be a convex
regularization functional that penalizes for the choice of more “complex” parameters (e.g.,
<<FORMULA>> according to a 1-norm in W, if His countable). We define the convex NN
criterion C(H;Q;;D;w)with parameter was follows:
<<FORMULA>> (1)
The following is a trivial lemma, but it is conceptually very important as it is the basis for the
rest of the analysis in this paper.
Lemma 2.2.The convex NN cost <<FORMULA>> is a convex function of w.
Proof. <<FORMULA>> is convex in w and <<>> is convex in w, by the above construction. C
is additive in <<FORMULA>> and additive in . Hence C is convex in w.
Note that there are no constraints in this convex optimization program, so that at the global
minimum all the partial derivatives of C with respect to elements of w cancel.
Let jHj be the cardinality of the set H. If it is not finite, it is not obvious that an optimal
solution can be achieved in finitely many iterations.
Lemma 2.2 says that training NNs from a very large class (with one or more hidden layer)
can be seen as convex optimization problems, usually in a very high dimensional space,as
long as we allow the number of hidden units to be selected by the learning algorithm.
By choosing a regularizer that promotes sparse solutions, we obtain a solution that has a
finite number of “active” hidden units (non-zero entries in the output weights vector w).
This assertion is proven below, in theorem 3.1, for the case of the hinge loss.
However, even if the solution involves a finite number of active hidden units, the convex
optimization problem could still be computationally intractable because of the large number
of variables involved. One approach to this problem is to apply the principles already suc-
cessfully embedded in Gradient Boosting, but more specifically in Column Generation (an
optimization technique for very large scale linear programs), i.e., add one hidden unit at a
time in an incremental fashion. The important ingredient here is a way to know that we
have reached the global optimum, thus not requiring to actually visit all the possible
hidden units.We show that this can be achieved as long as we can solve the sub-problem
of finding a linear classifier that minimizes the weighted sum of classification errors. This
can be done exactly only on low dimensional data sets but can be well approached using
weighted linear SVMs, weighted logistic regression, or Perceptron-type algorithms.
Another idea (not followed up here) would be to consider first a smaller set H1 , for which
the convex problem can be solved in polynomial time, and whose solution can theoretically
be selected as initialization for minimizing the criterion <<FORMULA>>, with <<FORMULA>>,
and where H2 may have infinite cardinality (countable or not). In this way we could show
that we can find a solution whose cost satisfies <<FORMULA>>,
i.e., is at least as good as the solution of a more restricted convex optimization problem. The
second minimization can be performed with a local descent algorithm, without the necessity
to guarantee that the global optimum will be found.
3 Finite Number of Hidden Neurons
In this section we consider the special case with <<FORMULA>> the hinge loss,
and <<L1>> regularization, and we show that the global optimum of the convex cost involves at
most n+ 1 hidden neurons, using an approach already exploited in (Ratsch, Demiriz and¨
Bennett, 2002) for L1-loss regression Boosting with L1 regularization of output weights. Xn
The training criterion is <<FORMULA>>. Let us rewrite t=1 this cost function as the
constrained optimization problem:
<<FORMULA>> (C1)
<<FORMULA>> (C2)
Using a standard technique, the above program can be recast as a linear program. Defin-
ing <<FORMULA>> the vector of Lagrangian multipliers for the constraints C1 , its dual
problem (P)takes the form (in the case of a finite number Jof base learners):
<<FORMULA>>
In the case of a finite number Jof base learners, <<FORMULA>>. If
the number of hidden units is uncountable, then Iis a closed bounded interval of R.
Such an optimization problem satisfies all the conditions needed for using Theorem 4.2
from (Hettich and Kortanek, 1993). Indeed:
<<FORMULA>> it is compact (as a closed bounded interval of <<FORMULA>> is a concave function
it is even a linear function);
<<FORMULA>> is convex in <<>> (it is actually linear in <<>>);
<<FORMULA>> (therefore finite) ( (P)is the largest value of F satisfying the constraints);
for every set of n+1 points <<FORMULA>>, there exists ~such that <<FORMULA>> for
<<FORMULA>> (one can take <<FORMULA>> since K>0).
Then, from Theorem 4.2 from (Hettich and Kortanek, 1993), the following theorem holds:
Theorem 3.1.The solution of (P) can be attained with constraints C0 and only n+1 constraints C0
(i.e., there exists a subset of n+1 constraints C0 giving rise to the same maximum 1
as when using the whole set of constraints). Therefore, the primal problem associated is the
minimization of the cost function of a NN with n+1 hidden neurons.
4 Incremental Convex NN Algorithm
In this section we present a stepwise algorithm to optimize a NN, and show that there is a cri-
terion that allows to verify whether the global optimum has been reached. This is a specializa-
tion of minimizing <<FORMULA>>, with <<FORMULA>> 1 and <<FORMULA>>
is the set of soft or hard linear classifiers (depending on choice of s()).
Algorithm ConvexNN( D, Q, , s)
<<ALGORITHM>>
Theorem 4.1.AlgorithmConvexNN Pstops when it reaches the global optimum of
<<FORMULA>>.
Proof.Let wbe the output weights vector when the algorithm stops. Because the set of
hidden units Hwe consider is such that when his in H, h is also in H, we can assume
all weights to be non-negative. By contradiction, if w0 6=wis the global optimum, with
<<C(w_0) < C(w)>>, then, since Cis convex in the output weights, for any 2(0;1) , we have
<<FORMULA>>. For
small enough, we can assume all weights in w that are strictly positive to be also strictly
positive in w . Let us denote by Ip the set of strictly positive weights in w (and w), by
Iz the set of weights set to zero in w but to a non-zero value in w , and by k the difference
w;k wk in the weight of hidden unit hk between wand w . We can assume j < 0 for
j2Iz , because instead of setting a small positive weight to hj , one can decrease the weight
of hj by the same amount, which will give either the same cost, or possibly a lower one
when the weight of <<FORMULA>> is positive. With o() denoting a quantity such that o()!0
when !0, the difference (w) =XC(w )C(w)can now be written:
<<FORMULA>>
since for i2Ip , thanks to step (7) of the algorithm, we have @C (w) = 0 . Thus the @w
inequality <<FORMULA>> rewrites into <<FORMULA>>
which, when !0, yields (note that <<FORMULA>> does not depend on ! since j is linear in ):
<<FORMULA>> (2)
i being the optimal classifier chosen in step (5a) or (5c), all hidden units <<FORMULA>> verify <<FORMULA>>
<<FORMULA>>
<<FORMULA>> , contradicting eq. 2.
(Mason et al., 2000) prove a related global convergence result for the AnyBoost algorithm,
a non-parametric Boosting algorithm that is also similar to Gradient Boosting (Friedman,
2001). Again, this requires solving as a sub-problem an exact minimization to find a function
hi 2 H that is maximally correlated with the gradient Q0 on the output. We now show a
simple procedure to select a hyperplane with the best weighted classification error.
Exact Minimization
In step (5a) we are required to find a linear classifier that minimizes the weighted sum of
classification errors. Unfortunately, this is an NP-hard problem (w.r.t. d, see theorem 4
in (Marcotte and Savard, 1992)). However, an exact solution can be easily found in O(n3 )
computations for d= 2 inputs.
Proposition 4.2.Finding a linear classifier that minimizes the weighted sum of classification
error can be achieved in O(n3 )steps when the input dimension is d= 2 .
Proof.We want to <<FORMULA>> with respect to u and b, the c’s being
in <<FORMULA>> Consider u fixed and sort the xi ’s according to their dot product with u and denote r
the function which maps ito r(i) such that xr(i) is in i-th position in the sort. Depending on P
the value of b, we will have n+1 possible sums, respectively <<FORMULA>>,
<<FORMULA>>. It is obvious that those sums only depend on the order of the products <<FORMULA>>,
<<FORMULA>>. When u varies smoothly on the unit circle, as the dot product is a continuous
function of its arguments, the changes in the order of the dot products will occur only when
there is a pair (i,j) such that <<FORMULA>>. Therefore, there are at most as many order
changes as there are pairs of different points, i.e., <<FORMULA>>. In the case of d=2, we
can enumerate all the different angles for which there is a change, namely a1 ;:::;a z with
<<FORMULA>>. We then need to test at least one <<FORMULA>> for each interval a2 i <
<<FORMULA>>, and also one u for <<FORMULA>>, which makes a total of <<FORMULA>> possibilities. 2
It is possible to generalize this result in higher dimensions, and as shown in (Marcotte and
Savard, 1992), one can achieve <<O(log(n)nd)>> time.
Algorithm 1 Optimal linear classifier search
<<ALGORITHM>>
Approximate Minimization
For data in higher dimensions, the exact minimization scheme to find the optimal linear
classifier is not practical. Therefore it is interesting to consider approximate schemes for
obtaining a linear classifier with weighted costs. Popular schemes for doing so are the linear
SVM (i.e., linear classifier with hinge loss), the logistic regression classifier, and variants of
the Perceptron algorithm. In that case, step (5c) of the algorithm is not an exact minimization,
and one cannot guarantee that the global optimum will be reached. However, it might be
reasonable to believe that finding a linear classifier by minimizing a weighted hinge loss
should yield solutions close to the exact minimization. Unfortunately, this is not generally
true, as we have found out on a simple toy data set described below. On the other hand,
if in step (7) one performs an optimization not only of the output weights wj (ji) but
also of the corresponding weight vectors vj , then the algorithm finds a solution close to the
global optimum (we could only verify this on 2-D data sets, where the exact solution can be
computed easily). It means that at the end of each stage, one first performs a few training
iterations of the whole NN (for the hidden units ji) with an ordinary gradient descent
mechanism (we used conjugate gradients but stochastic gradient descent would work too),
optimizing the wj ’s and the vj ’s, and then one fixes the vj ’s and obtains the optimal wj ’s for
these vj ’s (using a convex optimization procedure). In our experiments we used a quadratic
Q, for which the optimization of the output weights can be done with a neural network, using
the outputs of the hidden layer as inputs.
Let us consider now a bit more carefully what it means to tune the v_j’s in step (7). Indeed,
changing the weight vector vj of a selected hidden neuron to decrease the cost is equivalent
to a change in the output weights w’s. More precisely, consider the step in which the
value of vj becomes v0 . This is equivalent to the following operation on the w’s, when wj j is the corresponding output weight value: the output weight associated with the value vj of
a hidden neuron is set to 0, and the output weight associated with the value v0 of a hidden j
neuron is set to wj . This corresponds to an exchange between two variables in the convex
program. We are justified to take any such step as long as it allows us to decrease the cost
C(w). The fact that we are simultaneously making such exchanges on all the hidden units
when we tune the vj ’s allows us to move faster towards the global optimum.
Extension to multiple outputs
The multiple outputs case is more involved than the single-output case because it is not P
enough to check the condition <<FORMULA>>. Consider a new hidden neuron whose output is
hi when the input is xi . Let us also denote <<FORMULA>> the vector of output weights
between the new hidden neuron and the <<FORMULA>> output neurons. The gradient with respect to j
is <<FORMULA>> with <<FORMULA>> the value of the j-th output neuron with input <<FORMULA>>.
This means that if, for a given j, we have <<FORMULA>>, moving Pj away from 0 can
only increase the cost. Therefore, the right quantity to consider is <<FORMULA>>.
We must therefore find <<FORMULA>>. As before, this sub-problem is not + convex, but it is not
as obvious how to approximate it by a convex problem. The stopping P criterion becomes: if there is no j
such that <<FORMULA>>, then all weights must remain equal to 0 and a global minimum is reached.
Experimental Results
We performed experiments on the 2-D double moon toy dataset (as used in (Delalleau, Ben-
gio and Le Roux, 2005)), to be able to compare with the exact version of the algorithm. In
these experiments, <<FORMULA>>. The set-up is the following:
Select a new linear classifier, either (a) the optimal one or (b) an approximate using logistic
regression.
Optimize the output weights using a convex optimizer.
In case (b), tune both input and output weights by conjugate gradient descent on Cand
finally re-optimize the output weights using LASSO regression.
Optionally, remove neurons whose output weight has been set to 0.
Using the approximate algorithm yielded for 100 training examples an average penalized
( = 1 ) squared error of 17.11 (over 10 runs), an average test classification error of 3.68%
and an average number of neurons of 5.5 . The exact algorithm yielded a penalized squared
error of 8.09, an average test classification error of 5.3%, and required 3 hidden neurons. A
penalty of = 1 was nearly optimal for the exact algorithm whereas a smaller penalty further
improved the test classification error of the approximate algorithm. Besides, when running
the approximate algorithm for a long time, it converges to a solution whose quadratic error is
extremely close to the one of the exact algorithm.
5 Conclusion
We have shown that training a NN can be seen as a convex optimization problem, and have
analyzed an algorithm that can exactly or approximately solve this problem. We have shown
that the solution with the hinge loss involved a number of non-zero weights bounded by
the number of examples, and much smaller in practice. We have shown that there exists a
stopping criterion to verify if the global optimum has been reached, but it involves solving a
sub-learning problem involving a linear classifier with weighted errors, which can be computationally
hard if the exact solution is sought, but can be easily implemented for toy data
sets (in low dimension), for comparing exact and approximate solutions.
The above experimental results are in agreement with our initial conjecture: when there are
many hidden units we are much less likely to stall in the optimization procedure, because
there are many more ways to descend on the convex cost C(w). They also suggest, based
on experiments in which we can compare with the exact sub-problem minimization, that
applying Algorithm ConvexNN with an approximate minimization for adding each hidden
unit while continuing to tune the previous hidden unit s tends to lead to fast convergence
to the global minimum. What can get us stuck in a “local minimum” (in the traditional sense,
i.e., of optimizing w’s and v’s together) is simply the inability to find a new hidden unit
weight vector that can improve the total cost (fit and regularization term) even if there
exists one.
Note that as a side-effect of the results presented here, we have a simple way to train P neural
networks with hard-threshold hidden units, since increasing <<FORMULA>> can be either achieved
exactly (at great price) or approximately (e.g. by using a cross-entropy
or hinge loss on the corresponding linear classifier).
Acknowledgments
The authors thank the following for support: NSERC, MITACS, and the Canada Research
Chairs. They are also grateful for the feedback and stimulating exchanges with Sam Roweis,
Nathan Srebro, and Aaron Courville.
References
Chvatal, V. (1983).´ Linear Programming. W.H. Freeman.
Delalleau, O., Bengio, Y., and Le Roux, N. (2005). Efficient non-parametric function induction
in semi-supervised learning. In Cowell, R. and Ghahramani, Z., editors,Proceedings of AIS-
TATS’2005, pages 96–103.
Freund, Y. and Schapire, R. E. (1997). A decision theoretic generalization of on-line learning and an
application to boosting.Journal of Computer and System Science, 55(1):119–139.
Friedman, J. (2001). Greedy function approximation: a gradient boosting machine.Annals of Statis-
tics, 29:1180.
Hettich, R. and Kortanek, K. (1993). Semi-infinite programming: theory, methods, and applications.
SIAM Review, 35(3):380–429.
Marcotte, P. and Savard, G. (1992). Novel approaches to the discrimination problem.Zeitschrift fr
Operations Research (Theory), 36:517–545.
Mason, L., Baxter, J., Bartlett, P. L., and Frean, M. (2000). Boosting algorithms as gradient descent.
InAdvances in Neural Information Processing Systems 12, pages 512–518.
Ratsch, G., Demiriz, A., and Bennett, K. P. (2002). Sparse regression ensembles in infinite and finite¨
hypothesis spaces.Machine Learning.
Rumelhart, D., Hinton, G., and Williams, R. (1986). Learning representations by back-propagating
errors.Nature, 323:533–536
<<END>> <<END>> <<END>>
<<START>> <<START>> <<START>>
DEEP COMPRESSION: COMPRESSING DEEP NEURAL
NETWORKS WITH PRUNING , T RAINED QUANTIZATION
AND HUFFMAN CODING
Song Han
Stanford University, Stanford, CA 94305, USA
songhan@stanford.edu
Huizi Mao
Tsinghua University, Beijing, 100084, China
mhz12@mails.tsinghua.edu.cn
William J. Dally
Stanford University, Stanford, CA 94305, USA
NVIDIA, Santa Clara, CA 95050, USA
dally@stanford.edu
ABSTRACT
Neural networks are both computationally intensive and memory intensive, making
them difficult to deploy on embedded systems with limited hardware resources. To
address this limitation, we introduce “deep compression”, a three stage pipeline:
pruning, trained quantization and Huffman coding, that work together to reduce
the storage requirement of neural networks by 35% to 49% without affecting their
accuracy. Our method first prunes the network by learning only the important
connections. Next, we quantize the weights to enforce weight sharing, finally, we
apply Huffman coding. After the first two steps we retrain the network to fine
tune the remaining connections and the quantized centroids. Pruning, reduces the
number of connections by 9% to 13%; Quantization then reduces the number of
bits that represent each connection from 32 to 5. On the ImageNet dataset, our
method reduced the storage required by AlexNet by 35%, from 240MB to 6.9MB,
without loss of accuracy. Our method reduced the size of VGG-16 by 49% from
552MB to 11.3MB, again with no loss of accuracy. This allows fitting the model
into on-chip SRAM cache rather than off-chip DRAM memory. Our compression
method also facilitates the use of complex neural networks in mobile applications
where application size and download bandwidth are constrained. Benchmarked on
CPU, GPU and mobile GPU, compressed network has 3% to 4% layerwise speedup
and 3% to 7% better energy efficiency.
1 INTRODUCTION
Deep neural networks have evolved to the state-of-the-art technique for computer vision tasks
(Krizhevsky et al., 2012)(Simonyan & Zisserman, 2014). Though these neural networks are very
powerful, the large number of weights consumes considerable storage and memory bandwidth. For
example, the AlexNet Caffemodel is over 200MB, and the VGG-16 Caffemodel is over 500MB
(BVLC). This makes it difficult to deploy deep neural networks on mobile system.
First, for many mobile-first companies such as Baidu and Facebook, various apps are updated via
different app stores, and they are very sensitive to the size of the binary files. For example, App
Store has the restriction “apps above 100 MB will not download until you connect to Wi-Fi”. As a
result, a feature that increases the binary size by 100MB will receive much more scrutiny than one
that increases it by 10MB. Although having deep neural networks running on mobile has many great
<<FIGURE>>
Figure 1: The three stage compression pipeline: pruning, quantization and Huffman coding. Pruning
reduces the number of weights by10%, while quantization further improves the compression rate:
between27%and31%. Huffman coding gives more compression: between35%and49%. The
compression rate already included the meta-data for sparse representation. The compression scheme
doesn’t incur any accuracy loss.
features such as better privacy, less network bandwidth and real time processing, the large storage
overhead prevents deep neural networks from being incorporated into mobile apps.
The second issue is energy consumption. Running large neural networks require a lot of memory
bandwidth to fetch the weights and a lot of computation to do dot products— which in turn consumes
considerable energy. Mobile devices are battery constrained, making power hungry applications such
as deep neural networks hard to deploy.
Energy consumption is dominated by memory access. Under 45nm CMOS technology, a 32 bit
floating point add consumes 0.9PJ, a 32bit SRAM cache access takes 5PJ, while a 32bit DRAM
memory access takes 640PJ, which is 3 orders of magnitude of an add operation. Large networks
do not fit in on-chip storage and hence require the more costly DRAM accesses. Running a 1 billion
connection neural network, for example, at 20fps would require (20Hz)(1G)(640PJ) = 12.8W just
for DRAM access - well beyond the power envelope of a typical mobile device.
Our goal is to reduce the storage and energy required to run inference on such large networks so they
can be deployed on mobile devices. To achieve this goal, we present “deep compression”: a three-
stage pipeline (Figure 1) to reduce the storage required by neural network in a manner that preserves
the original accuracy. First, we prune the networking by removing the redundant connections, keeping
only the most informative connections. Next, the weights are quantized so that multiple connections
share the same weight, thus only the codebook (effective weights) and the indices need to be stored.
Finally, we apply Huffman coding to take advantage of the biased distribution of effective weights.
Our main insight is that, pruning and trained quantization are able to compress the network without
interfering each other, thus lead to surprisingly high compression rate. It makes the required storage
so small (a few megabytes) that all weights can be cached on chip instead of going to off-chip DRAM
which is energy consuming. Based on “deep compression”, the EIE hardware accelerator Han et al.
(2016) was later proposed that works on the compressed model, achieving significant speedup and
energy efficiency improvement.
2 NETWORK PRUNING
Network pruning has been widely studied to compress CNN models. In early work, network pruning
proved to be a valid way to reduce the network complexity and over-fitting (LeCun et al., 1989;
Hanson & Pratt, 1989; Hassibi et al., 1993; Strom, 1997). Recently Han et al. (2015) pruned state- ¨
of-the-art CNN models with no loss of accuracy. We build on top of that approach. As shown on
the left side of Figure 1, we start by learning the connectivity via normal network training. Next, we
prune the small-weight connections: all connections with weights below a threshold are removed
from the network. Finally, we retrain the network to learn the final weights for the remaining sparse
connections. Pruning reduced the number of parameters by9%and13%for AlexNet and VGG-16
model.
<<FIGURE>>
Figure 2: Representing the matrix sparsity with relative index. Padding filler zero to prevent overflow.
<<FIGURE>>
Figure 3: Weight sharing by scalar quantization (top) and centroids fine-tuning (bottom).
We store the sparse structure that results from pruning using compressed sparse row (CSR) or
compressed sparse column (CSC) format, which requires2a+n+1numbers, where a is the number
of non-zero elements and n is the number of rows or columns.
To compress further, we store the index difference instead of the absolute position, and encode this
difference in 8 bits for conv layer and 5 bits for fc layer. When we need an index difference larger
than the bound, we the zero padding solution shown in Figure 2: in case when the difference exceeds
8, the largest 3-bit (as an example) unsigned number, we add a filler zero.
3 TRAINED QUANTIZATION AND WEIGHT SHARING
Network quantization and weight sharing further compresses the pruned network by reducing the
number of bits required to represent each weight. We limit the number of effective weights we need to
store by having multiple connections share the same weight, and then fine-tune those shared weights.
Weight sharing is illustrated in Figure 3. Suppose we have a layer that has 4 input neurons and 4
output neurons, the weight is a 4x4 matrix. On the top left is the 4x4 weight matrix, and on the
bottom left is the 4x4 gradient matrix. The weights are quantized to 4 bins (denoted with 4 colors),
all the weights in the same bin share the same value, thus for each weight, we then need to store only
a small index into a table of shared weights. During update, all the gradients are grouped by the color
and summed together, multiplied by the learning rate and subtracted from the shared centroids from
last iteration. For pruned AlexNet, we are able to quantize to 8-bits (256 shared weights) for each
CONV layers, and 5-bits (32 shared weights) for each FC layer without any loss of accuracy.
To calculate the compression rate, given k clusters, we only need log_2(k) bits to encode the index. In
general, for a network with n connections and each connection is represented with b bits, constraining
the connections to have only k shared weights will result in a compression rate of:
<<FORMULA>> (1)
For example, Figure 3 shows the weights of a single layer neural network with four input units and
four output units. There are4%4 = 16weights originally but there are only4shared weights: similar
weights are grouped together to share the same value. Originally we need to store 16 weights each
<<FIGURE>>
Figure 4: Left: Three different methods for centroids initialization. Right: Distribution of weights
(blue) and distribution of codebook before (green cross) and after fine-tuning (red dot).
has 32 bits, now we need to store only 4 effective weights (blue, green, red and orange), each has 32
bits, together with 16 2-bit indices giving a compression rate of <<FORMULA>>
3.1 WEIGHT SHARING
We use k-means clustering to identify the shared weights for each layer of a trained network, so that
all the weights that fall into the same cluster will share the same weight. Weights are not shared across
layers. We partition n original weights <<FORMULA>> into k clusters <<FORMULA>>,
n%k, so as to minimize the within-cluster sum of squares (WCSS):
<<FORMULA>> (2)
Different from HashNet (Chen et al., 2015) where weight sharing is determined by a hash function
before the networks sees any training data, our method determines weight sharing after a network is
fully trained, so that the shared weights approximate the original network.
3.2 INITIALIZATION OF SHARED WEIGHTS
Centroid initialization impacts the quality of clustering and thus affects the network’s prediction
accuracy. We examine three initialization methods: Forgy(random), density-based, and linear
initialization. In Figure 4 we plotted the original weights’ distribution of conv3 layer in AlexNet
(CDF in blue, PDF in red). The weights forms a bimodal distribution after network pruning. On the
bottom it plots the effective weights (centroids) with 3 different initialization methods (shown in blue,
red and yellow). In this example, there are 13 clusters.
Forgy(random) initialization randomly chooses k observations from the data set and uses these as
the initial centroids. The initialized centroids are shown in yellow. Since there are two peaks in the
bimodal distribution, Forgy method tend to concentrate around those two peaks.
Density-based initialization linearly spaces the CDF of the weights in the y-axis, then finds the
horizontal intersection with the CDF, and finally finds the vertical intersection on the x-axis, which
becomes a centroid, as shown in blue dots. This method makes the centroids denser around the two
peaks, but more scatted than the Forgy method.
Linear initialization linearly spaces the centroids between the [min, max] of the original weights.
This initialization method is invariant to the distribution of the weights and is the most scattered
compared with the former two methods.
Larger weights play a more important role than smaller weights (Han et al., 2015), but there are fewer
of these large weights. Thus for both Forgy initialization and density-based initialization, very few
centroids have large absolute value which results in poor representation of these few large weights.
Linear initialization does not suffer from this problem. The experiment section compares the accuracy
<<FIGURE>>
Figure 5: Distribution for weight (Left) and index (Right). The distribution is biased.
of different initialization methods after clustering and fine-tuning, showing that linear initialization
works best.
3.3 FEED-FORWARD AND BACK-PROPAGATION
The centroids of the one-dimensional k-means clustering are the shared weights. There is one level
of indirection during feed forward phase and back-propagation phase looking up the weight table.
An index into the shared weight table is stored for each connection. During back-propagation, the
gradient for each shared weight is calculated and used to update the shared weight. This procedure is
shown in Figure 3.
We denote the loss byL, the weight in the ith column and jth row by Wij, the centroid index of
element Wij by Iij, the kth centroid of the layer by Ck. By using the indicator function <<1(.)>>, the
gradient of the centroids is calculated as:
<<FORMULA>> (3)
4 HUFFMAN CODING
A Huffman code is an optimal prefix code commonly used for lossless data compression(Van Leeuwen,
1976). It uses variable-length codewords to encode source symbols. The table is derived from the
occurrence probability for each symbol. More common symbols are represented with fewer bits.
Figure 5 shows the probability distribution of quantized weights and the sparse matrix index of the
last fully connected layer in AlexNet. Both distributions are biased: most of the quantized weights are
distributed around the two peaks; the sparse matrix index difference are rarely above 20. Experiments
show that Huffman coding these non-uniformly distributed values saves 20% to 30% of network
storage.
5 EXPERIMENTS
We pruned, quantized, and Huffman encoded four networks: two on MNIST and two on ImageNet
data-sets. The network parameters and accuracy- 1 before and after pruning are shown in Table 1. The
compression pipeline saves network storage by 35% to 49% across different networks without loss
of accuracy. The total size of AlexNet decreased from 240MB to 6.9MB, which is small enough to
be put into on-chip SRAM, eliminating the need to store the model in energy-consuming DRAM
memory.
Training is performed with the Caffe framework (Jia et al., 2014). Pruning is implemented by adding
a mask to the blobs to mask out the update of the pruned connections. Quantization and weight
sharing are implemented by maintaining a codebook structure that stores the shared weight, and
group-by-index after calculating the gradient of each layer. Each shared weight is updated with all
the gradients that fall into that bucket. Huffman coding doesn’t require training and is implemented
offline after all the fine-tuning is finished.
5.1 LE NET-300-100 AND LE NET-5 ON MNIST
We first experimented on MNIST dataset with LeNet-300-100 and LeNet-5 network (LeCun et al.,
1998). LeNet-300-100 is a fully connected network with two hidden layers, with 300 and 100
1 Reference model is from Caffe model zoo, accuracy is measured without data augmentation
Table 1: The compression pipeline can save35%to49%parameter storage with no loss of accuracy.
