diff --git a/Corpus/CORPUS.txt b/Corpus/CORPUS.txt index 23fb525..b7e7925 100644 --- a/Corpus/CORPUS.txt +++ b/Corpus/CORPUS.txt @@ -6598,4 +6598,2192 @@ D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convol multiplications consume2xenergy than sparse ones because it is accelerated with multi-threading. <> +<> <> <> + + +<> <> <> + DEEP DOUBLE DESCENT: WHERE BIGGER MODELS AND MORE DATA HURT + + Preetum Nakkiran Gal Kaplun y Yamini Bansal y Tristan Yang + Harvard University Harvard University Harvard University Harvard University + + Boaz Barak Ilya Sutskever + Harvard University OpenAI + + + ABSTRACT + + We show that a variety of modern deep learning tasks exhibit a “double-descent” + phenomenon where, as we increase model size, performance first gets worse and + then gets better. Moreover, we show that double descent occurs not just as a + function of model size, but also as a function of the number of training epochs. + We unify the above phenomena by defining a new complexity measure we call + the effective model complexity and conjecture a generalized double descent with + respect to this measure. Furthermore, our notion of model complexity allows us to + identify certain regimes where increasing (even quadrupling) the number of train + samples actually hurts test performance. + + + 1 INTRODUCTION + + <
> + + Figure 1:Left:Train and test error as a function of model size, for ResNet18s of varying width + on CIFAR-10 with 15% label noise.Right:Test error, shown for varying train epochs. All models + trained using Adam for 4K epochs. The largest model (width64) corresponds to standard ResNet18. + + + The bias-variance trade-off is a fundamental concept in classical statistical learning theory (e.g., + Hastie et al. (2005)). The idea is that models of higher complexity have lower bias but higher vari- + ance. According to this theory, once model complexity passes a certain threshold, models “overfit” + with the variance term dominating the test error, and hence from this point onward, increasing model + complexity will only decrease performance (i.e., increase test error). Hence conventional wisdom + in classical statistics is that, once we pass a certain threshold,“larger models are worse.” + However, modern neural networks exhibit no such phenomenon. Such networks have millions of + parameters, more than enough to fit even random labels (Zhang et al. (2016)), and yet they perform + much better on many tasks than smaller models. Indeed, conventional wisdom among practitioners + is that“larger models are better’’ (Krizhevsky et al. (2012), Huang et al. (2018), Szegedy et al. + + <
> + + Figure 2:Left:Test error as a function of model size and train epochs. The horizontal line corre- + sponds to model-wise double descent–varying model size while training for as long as possible. The + vertical line corresponds to epoch-wise double descent, with test error undergoing double-descent + as train time increases.RightTrain error of the corresponding models. All models are Resnet18s + trained on CIFAR-10 with 15% label noise, data-augmentation, and Adam for up to 4K epochs. + + + (2015), Radford et al. (2019)). The effect of training time on test performance is also up for debate. + In some settings, “early stopping” improves test performance, while in other settings training neu- + ral networks to zero training error only improves performance. Finally, if there is one thing both + classical statisticians and deep learning practitioners agree on is“more data is always better”. + In this paper, we present empirical evidence that both reconcile and challenge some of the above + “conventional wisdoms.” We show that many deep learning settings have two different regimes. + In the under-parameterized regime, where the model complexity is small compared to the number + of samples, the test error as a function of model complexity follows the U-like behavior predicted + by the classical bias/variance tradeoff. However, once model complexity is sufficiently large to + interpolate i.e., achieve (close to) zero training error, then increasing complexity only decreases test + error, following the modern intuition of “bigger models are better”. Similar behavior was previously + observed in Opper (1995; 2001), Advani & Saxe (2017), Spigler et al. (2018), and Geiger et al. + (2019b). This phenomenon was first postulated in generality by Belkin et al. (2018) who named + it “double descent”, and demonstrated it for decision trees, random features, and 2-layer neural + networks with‘2 loss, on a variety of learning tasks including MNIST and CIFAR-10. + + + Main contributions. We show that double descent is a robust phenomenon that occurs in a variety + of tasks, architectures, and optimization methods (see Figure 1 and Section 5; our experiments are + summarized in Table A). Moreover, we propose a much more general notion of “double descent” + that goes beyond varying the number of parameters. We define the effective model complexity (EMC) + of a training procedure as the maximum number of samples on which it can achieve close to zero + training error. The EMC depends not just on the data distribution and the architecture of the classifier + but also on the training procedure—and in particular increasing training time will increase the EMC. + We hypothesize that for many natural models and learning algorithms, double descent occurs as a + function of the EMC. Indeed we observe “epoch-wise double descent” when we keep the model fixed + and increase the training time, with performance following a classical U-like curve in the underfitting + stage (when the EMC is smaller than the number of samples) and then improving with training time + once the EMC is sufficiently larger than the number of samples (see Figure 2). As a corollary, early + stopping only helps in the relatively narrow parameter regime of critically parameterized models. + + + Sample non-monotonicity. Finally, our results shed light on test performance as a function of + the number of train samples. Since the test error peaks around the point where EMC matches the + number of samples (the transition from the under- to over-parameterization), increasing the number + of samples has the effect of shifting this peak to the right. While in most settings increasing the + number of samples decreases error, this shifting effect can sometimes result in a setting wheremore + data is worse!For example, Figure 3 demonstrates cases in which increasing the number of samples + by a factor of4:5results in worse test performance. + + Figure 3: Test loss (per-token perplexity) as a + function of Transformer model size (embed- + ding dimension d model) on language trans- + <
> lation (IWSLT‘14 German-to-English). The + curve for 18k samples is generally lower than + the one for 4k samples, but also shifted to + the right, since fitting 18k samples requires + a larger model. Thus, for some models, the + performance for 18k samples is worse than + for 4k samples. + + + + 2 OUR RESULTS + + To state our hypothesis more precisely, we define the notion of effective model complexity. We define + a training procedure T to be any procedure that takes as input a set <> + of labeled training samples and outputs a classifier <> mapping data to labels. We define the + effective model complexity of T (w.r.t. distributionD) to be the maximum number of samples non + which T achieves on average <> training error. + + Definition 1 (Effective Model Complexity)TheEffective Model Complexity(EMC) of a training + procedureT, with respect to distribution D and parameter <>, is defined as: + + <> + + whereError <> is the mean error of modelMon train samplesS. + + Our main hypothesis can be informally stated as follows: + + Hypothesis 1 (Generalized Double Descent hypothesis, informal)For any natural data distribu- + tion D, neural-network-based training procedureT, and small <>, if we consider the task of + predicting labels based on n samples from D then: + + Under-parametrized regime.If <> is sufficiently smaller than n, any perturbation of T + that increases its effective complexity will decrease the test error. + Over-parameterized regime.If <> is sufficiently larger than n, any perturbation of T + that increases its effective complexity will decrease the test error. + Critically parameterized regime.If <>, then a perturbation of T that increases its + effective complexity might decrease or increase the test error. + + Hypothesis 1 is informal in several ways. We do not have a principled way to choose the parameter + <> (and currently heuristically use <>). We also are yet to have a formal specification for + “sufficiently smaller” and “sufficiently larger”. Our experiments suggest that there is a critical + interval around the interpolation threshold when <>: below and above this interval + increasing complexity helps performance, while within this interval it may hurt performance. The + width of the critical interval depends on both the distribution and the training procedure in ways we + do not yet completely understand. + + We believe Hypothesis 1 sheds light on the interaction between optimization algorithms, model size, + and test performance and helps reconcile some of the competing intuitions about them. The main + result of this paper is an experimental validation of Hypothesis 1 under a variety of settings, where + we considered several natural choices of datasets, architectures, and optimization algorithms, and + we changed the “interpolation threshold” by varying the number of model parameters, the length of + training, the amount of label noise in the distribution, and the number of train samples. + Model-wise Double Descent.In Section 5, we study the test error of models of increasing size, + for a fixed large number of optimization steps. We show that “model-wise double-descent” occurs + for various modern datasets (CIFAR-10, CIFAR-100, IWSLT‘14 de-en, with varying amounts of + label noise), model architectures (CNNs, ResNets, Transformers), optimizers (SGD, Adam), number + of train samples, and training procedures (data-augmentation, and regularization). Moreover, the + peak in test error systematically occurs at the interpolation threshold. In particular, we demonstrate + realistic settings in which bigger models are worse. + + Epoch-wise Double Descent.In Section 6, we study the test error of a fixed, large architecture over + the course of training. We demonstrate, in similar settings as above, a corresponding peak in test + performance when models are trained just long enough to reach <> train error. The test error of a + large model first decreases (at the beginning of training), then increases (around the critical regime), + then decreases once more (at the end of training)—that is,training longer can correct overfitting. + Sample-wise Non-monotonicity.In Section 7, we study the test error of a fixed model and training + procedure, for varying number of train samples. Consistent with our generalized double-descent + hypothesis, we observe distinct test behavior in the “critical regime”, when the number of samples + is near the maximum that the model can fit. This often manifests as a long plateau region, in which + taking significantly more data might not help when training to completion (as is the case for CNNs on + CIFAR-10). Moreover, we show settings (Transformers on IWSLT‘14 en-de), where this manifests + as a peak—and for a fixed architecture and training procedure,more data actually hurts. + Remarks on Label Noise.We observe all forms of double descent most strongly in settings with + label noise in the train set (as is often the case when collecting train data in the real-world). How- + ever, we also show several realistic settings with a test-error peak even without label noise: ResNets + (Figure 4a) and CNNs (Figure 20) on CIFAR-100; Transformers on IWSLT‘14 (Figure 8). More- + over, all our experiments demonstrate distinctly different test behavior in the critical regime— often + manifesting as a “plateau” in the test error in the noiseless case which develops into a peak with + added label noise. See Section 8 for further discussion. + + 3 RELATED WORK + + Model-wise double descent was first proposed as a general phenomenon by Belkin et al. (2018). + Similar behavior had been observed in Opper (1995; 2001), Advani & Saxe (2017), Spigler et al. + (2018), and Geiger et al. (2019b). Subsequently, there has been a large body of work studying the + double descent phenomenon. A growing list of papers that theoretically analyze it in the tractable + setting of linear least squares regression includes Belkin et al. (2019); Hastie et al. (2019); Bartlett + et al. (2019); Muthukumar et al. (2019); Bibas et al. (2019); Mitra (2019); Mei & Montanari (2019). + Moreover, Geiger et al. (2019a) provide preliminary results for model-wise double descent in con- + volutional networks trained on CIFAR-10. Our work differs from the above papers in two crucial + aspects: First, we extend the idea of double-descent beyond the number of parameters to incorpo- + rate the training procedure under a unified notion of “Effective Model Complexity”, leading to novel + insights like epoch-wise double descent and sample non-monotonicity. The notion that increasing + train time corresponds to increasing complexity was also presented in Nakkiran et al. (2019). Sec- + ond, we provide an extensive and rigorous demonstration of double-descent for modern practices + spanning a variety of architectures, datasets optimization procedures. An extended discussion of the + related work is provided in Appendix C. + + 4 EXPERIMENTAL SETUP + + We briefly describe the experimental setup here; full details are in Appendix B1. We consider three + families of architectures: ResNets, standard CNNs, and Transformers.ResNets:We parameterize + a family of ResNet18s (He et al. (2016)) by scaling the width (number of filters) of convolutional + layers. Specifically, we use layer widths [k;2k;4k;8k] for varying k. The standard ResNet18 + corresponds tok= 64. Standard CNNs:We consider a simple family of 5-layer CNNs, with + 4 convolutional layers of widths [k;2k;4k;8k] for varying k, and a fully-connected layer. For + context, the CNN with width k=64, can reach over 90% test accuracy on CIFAR-10 with data- + augmentation.Transformers:We consider the 6 layer encoder-decoder from Vaswani et al. (2017), + as implemented by Ott et al. (2019). We scale the size of the network by modifying the embedding + dimension d model , and setting the width of the fully-connected layers proportionally (<>). + + The raw data from our experiments are available at: https://gitlab.com/harvard-machine-learning/double-descent/tree/master + + For ResNets and CNNs, we train with cross-entropy loss, and the following optimizers: (1) Adam + with learning-rate0:0001for 4K epochs; (2) SGD with learning rate/p1 for 500K gradient steps. T We train Transformers for 80K gradient steps, with 10% label smoothing and no drop-out. + + Label Noise. In our experiments, label noise of probability prefers to training on a samples which + have the correct label with probability (<>), and a uniformly random incorrect label otherwise + (label noise is sampled only once and not per epoch). Figure 1 plots test error on the noisy distribu- + tion, while the remaining figures plot test error with respect to the clean distribution (the two curves + are just linear rescaling of one another). + + 5 MODEL-WISE DOUBLE DESCENT + + <
> + + Figure 4:Model-wise double descent for ResNet18s.Trained on CIFAR-100 and CIFAR-10, with + varying label noise. Optimized using Adam with LR0:0001for 4K epochs, and data-augmentation. + + In this section, we study the test error of models of increasing size, when training to completion + (for a fixed large number of optimization steps). We demonstrate model-wise double descent across + different architectures, datasets, optimizers, and training procedures. The critical region exhibits + distinctly different test behavior around the interpolation point and there is often a peak in test error + that becomes more prominent in settings with label noise. + For the experiments in this section (Figures 4, 5, 6, 7, 8), notice that all modifications which increase + the interpolation threshold (such as adding label noise, using data augmentation, and increasing the + number of train samples) also correspondingly shift the peak in test error towards larger models. + Additional plots showing the early-stopping behavior of these models, and additional experiments + showing double descent in settings with no label noise (e.g. Figure 19) are in Appendix E.2. We + also observed model-wise double descent for adversarial training, with a prominent robust test error + peak even in settings without label noise. See Figure 26 in Appendix E.2. + + Discussion. Fully understanding the mechanisms behind model-wise double descent in deep neu- + ral networks remains an important open question. However, an analog of model-wise double descent + occurs even for linear models. A recent stream of theoretical works analyzes this setting (Bartlett + et al. (2019); Muthukumar et al. (2019); Belkin et al. (2019); Mei & Montanari (2019); Hastie et al. + (2019)). We believe similar mechanisms may be at work in deep neural networks. + Informally, our intuition is that for model-sizes at the interpolation threshold, there is effectively + only one model that fits the train data and this interpolating model is very sensitive to noise in the + + <
> + + Figure 5: Effect of Data Augmentation. 5-layer CNNs on CIFAR10, with and without data- + augmentation. Data-augmentation shifts the interpolation threshold to the right, shifting the test + error peak accordingly. Optimized using SGD for 500K steps. See Figure 27 for larger models. + + <
> <
> + + Figure 6:SGD vs. Adam.5-Layer CNNs Figure 7: Noiseless settings. 5-layer + on CIFAR-10 with no label noise, and no CNNs on CIFAR-100 with no label noise; + data augmentation. Optimized using SGD note the peak in test error. Trained with + for 500K gradient steps, and Adam for 4K SGD and no data augmentation. See Fig- + epochs. ure 20 for the early-stopping behavior of + these models. + + + train set and/or model mis-specification. That is, since the model is just barely able to fit the train + data, forcing it to fit even slightly-noisy or mis-specified labels will destroy its global structure, and + result in high test error. (See Figure 28 in the Appendix for an experiment demonstrating this noise + sensitivity, by showing that ensembling helps significantly in the critically-parameterized regime). + However for over-parameterized models, there are many interpolating models that fit the train set, + and SGD is able to find one that “memorizes” (or “absorbs”) the noise while still performing well + on the distribution. + The above intuition is theoretically justified for linear models. In general, this situation manifests + even without label noise for linear models (Mei & Montanari (2019)), and occurs whenever there + + Figure 8:Transformers on language trans- + lation tasks:Multi-head-attention encoder- + decoder Transformer model trained for + <
> 80k gradient steps with labeled smoothed + cross-entropy loss on IWSLT‘14 German- + to-English (160K sentences) and WMT‘14 + English-to-French (subsampled to 200K sen- + tences) dataset. Test loss is measured as per- + token perplexity. + + + + + + is model mis-specification between the structure of the true distribution and the model family. We + believe this intuition extends to deep learning as well, and it is consistent with our experiments. + + + 6 EPOCH-WISE DOUBLE DESCENT + + In this section, we demonstrate a novel form of double-descent with respect to training epochs, + which is consistent with our unified view of effective model complexity (EMC) and the generalized + double descent hypothesis. Increasing the train time increases the EMC—and thus a sufficiently + large model transitions from under- to over-parameterized over the course of training. + + <
> + + Figure 9:Left:Training dynamics for models in three regimes. Models are ResNet18s on CIFAR10 + with 20% label noise, trained using Adam with learning rate0:0001, and data augmentation.Right: + Test error over (Model size Epochs). Three slices of this plot are shown on the left. + + + As illustrated in Figure 9, sufficiently large models can undergo a “double descent” behavior where + test error first decreases then increases near the interpolation threshold, and then decreases again. In + contrast, for “medium sized” models, for which training to completion will only barely reach 0 + error, the test error as a function of training time will follow a classical U-like curve where it is + better to stop early. Models that are too small to reach the approximation threshold will remain in + the “under parameterized” regime where increasing train time monotonically decreases test error. + Our experiments (Figure 10) show that many settings of dataset and architecture exhibit epoch-wise + double descent, in the presence of label noise. Further, this phenomenon is robust across optimizer + variations and learning rate schedules (see additional experiments in Appendix E.1). As in model- + wise double descent, the test error peak is accentuated with label noise. + Conventional wisdom suggests that training is split into two phases: (1) In the first phase, the net- + work learns a function with a small generalization gap (2) In the second phase, the network starts + to over-fit the data leading to an increase in test error. Our experiments suggest that this is not the + complete picture—in some regimes, the test error decreases again and may achieve a lower value at + the end of training as compared to the first minimum (see Fig 10 for 10% label noise). + + <
> + + Figure 10:Epoch-wise double descent for ResNet18 and CNN (width=128). ResNets trained using + Adam with learning rate0:0001, and CNNs trained with SGD with inverse-square root learning rate. + + + 7 SAMPLE-WISE NON-MONOTONICITY + + In this section, we investigate the effect of varying the number of train samples, for a fixed model and + training procedure. Previously, in model-wise and epoch-wise double descent, we explored behavior + in the critical regime, where <>, by varying the EMC. Here, we explore the critical + regime by varying the number of train samples n. By increasing n, the same training procedure T + can switch from being effectively over-parameterized to effectively under-parameterized. + We show that increasing the number of samples has two different effects on the test error vs. model + complexity graph. On the one hand, (as expected) increasing the number of samples shrinks the area + under the curve. On the other hand, increasing the number of samples also has the effect of “shifting + the curve to the right” and increasing the model complexity at which test error peaks. + + <
> + + Figure 11: Sample-wise non-monotonicity. + + + These twin effects are shown in Figure 11a. Note that there is a range of model sizes where the + effects “cancel out”—and having 4% more train samples does not help test performance when + training to completion. Outside the critically-parameterized regime, for sufficiently under- or over- + parameterized models, having more samples helps. This phenomenon is corroborated in Figure 12, + which shows test error as a function of both model and sample size, in the same setting as Figure 11a. + + <
> + + Figure 12:Left:Test Error as a function of model size and number of train samples, for 5-layer + CNNs on CIFAR-10 +20% noise. Note the ridge of high test error again lies along the interpolation + threshold. Right: Three slices of the left plot, showing the effect of more data for models of + different sizes. Note that, when training to completion, more data helps for small and large models, + but does not help for near-critically-parameterized models (green). + + In some settings, these two effects combine to yield a regime of model sizes where more data actually + hurts test performance as in Figure 3 (see also Figure 11b). Note that this phenomenon is not unique + to DNNs: more data can hurt even for linear models (see Appendix D). + + 8 CONCLUSION AND DISCUSSION + + We introduce a generalized double descent hypothesis: models and training procedures exhibit atyp- + ical behavior when their Effective Model Complexity is comparable to the number of train samples. + We provide extensive evidence for our hypothesis in modern deep learning settings, and show that + it is robust to choices of dataset, architecture, and training procedures. In particular, we demon- + strate “model-wise double descent” for modern deep networks and characterize the regime where + bigger models can perform worse. We also demonstrate “epoch-wise double descent,” which, to the + best of our knowledge, has not been previously proposed. Finally, we show that the double descent + phenomenon can lead to a regime where training on more data leads to worse test performance. + Preliminary results suggest that double descent also holds as we vary the amount of regularization + for a fixed model (see Figure 22). + We also believe our characterization of the critical regime provides a useful way of thinking for + practitioners—if a model and training procedure are just barely able to fit the train set, then small + changes to the model or training procedure may yield unexpected behavior (e.g. making the model + slightly larger or smaller, changing regularization, etc. may hurt test performance). + + Early stopping. We note that many of the phenomena that we highlight often do not occur with + optimal early-stopping. However, this is consistent with our generalized double descent hypothesis: + if early stopping prevents models from reaching0train error then we would not expect to see double- + descent, since the EMC does not reach the number of train samples. Further, we show at least one + + setting where model-wise double descent can still occur even with optimal early stopping (ResNets + on CIFAR-100 with no label noise, see Figure 19). We have not observed settings where more data + hurts when optimal early-stopping is used. However, we are not aware of reasons which preclude + this from occurring. We leave fully understanding the optimal early stopping behavior of double + descent as an important open question for future work. + + Label Noise. In our experiments, we observe double descent most strongly in settings with label + noise. However, we believe this effect is not fundamentally about label noise, but rather about + model mis-specification. For example, consider a setting where the label noise is not truly random, + but rather pseudorandom (with respect to the family of classifiers being trained). In this setting, + the performance of the Bayes optimal classifier would not change (since the pseudorandom noise + is deterministic, and invertible), but we would observe an identical double descent as with truly + random label noise. Thus, we view adding label noise as merely a proxy for making distributions + “harder”— i.e. increasing the amount of model mis-specification. + + Other Notions of Model Complexity. Our notion of Effective Model Complexity is related to + classical complexity notions such as Rademacher complexity, but differs in several crucial ways: + (1) EMC depends on the true labels of the data distribution, and (2) EMC depends on the training + procedure, not just the model architecture. + Other notions of model complexity which do not incorporate features (1) and (2) would not suffice + to characterize the location of the double-descent peak. Rademacher complexity, for example, is + determined by the ability of a model architecture to fit a randomly-labeled train set. But Rademacher + complexity and VC dimension are both insufficient to determine the model-wise double descent + peak location, since they do not depend on the distribution of labels— and our experiments show + that adding label noise shifts the location of the peak. + Moreover, both Rademacher complexity and VC dimension depend only on the model family and + data distribution, and not on the training procedure used to find models. Thus, they are not capable + of capturing train-time double-descent effects, such as “epoch-wise” double descent, and the effect + of data-augmentation on the peak location. + + ACKNOWLEDGMENTS + We thank Mikhail Belkin for extremely useful discussions in the early stages of this work. We + thank Christopher Olah for suggesting the Model SizeEpoch visualization, which led to the + investigation of epoch-wise double descent, as well as for useful discussion and feedback. We also + thank Alec Radford, Jacob Steinhardt, and Vaishaal Shankar for helpful discussion and suggestions. + P.N. thanks OpenAI, the Simons Institute, and the Harvard Theory Group for a research environment + that enabled this kind of work. + We thank Dimitris Kalimeris, Benjamin L. Edelman, and Sharon Qian, and Aditya Ramesh for + comments on an early draft of this work. + This work supported in part by NSF grant CAREER CCF 1452961, BSF grant 2014389, NSF US- + ICCS proposal 1540428, a Google Research award, a Facebook research award, a Simons Investiga- + tor Award, a Simons Investigator Fellowship, and NSF Awards CCF 1715187, CCF 1565264, CCF + 1301976, IIS 1409097, and CNS 1618026. Y.B. would like to thank the MIT-IBM Watson AI Lab + for contributing computational resources for experiments. + + + REFERENCES + Madhu S Advani and Andrew M Saxe. High-dimensional dynamics of generalization error in neural + networks.arXiv preprint arXiv:1710.03667, 2017. + + Peter L Bartlett, Philip M Long, Gabor Lugosi, and Alexander Tsigler. Benign overfitting in linear´ + regression.arXiv preprint arXiv:1906.11300, 2019. + + Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine learning + and the bias-variance trade-off.arXiv preprint arXiv:1812.11118, 2018. + + Mikhail Belkin, Daniel Hsu, and Ji Xu. Two models of double descent for weak features.arXiv + preprint arXiv:1903.07571, 2019. + + Koby Bibas, Yaniv Fogel, and Meir Feder. A new look at an old problem: A universal learning + approach to linear regression.arXiv preprint arXiv:1905.04708, 2019. + + Mauro Cettolo, Christian Girardi, and Marcello Federico. Wit 3 : Web inventory of transcribed and + translated talks. InProceedings of the 16 th Conference of the European Association for Machine + Translation (EAMT), pp. 261–268, Trento, Italy, May 2012. + + Mario Geiger, Arthur Jacot, Stefano Spigler, Franck Gabriel, Levent Sagun, Stephane d’Ascoli,´ + Giulio Biroli, Clement Hongler, and Matthieu Wyart. Scaling description of generalization with´ + number of parameters in deep learning.arXiv preprint arXiv:1901.01608, 2019a. + + Mario Geiger, Stefano Spigler, Stephane d’Ascoli, Levent Sagun, Marco Baity-Jesi, Giulio Biroli,´ + and Matthieu Wyart. Jamming transition as a paradigm to understand the loss landscape of deep + neural networks.Physical Review E, 100(1):012115, 2019b. + + Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial + examples.arXiv preprint arXiv:1412.6572, 2014. + + Trevor Hastie, Robert Tibshirani, Jerome Friedman, and James Franklin. The elements of statistical + learning: data mining, inference and prediction.The Mathematical Intelligencer, 27(2):83–85, + 2005. + + Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan J Tibshirani. Surprises in high- + dimensional ridgeless least squares interpolation.arXiv preprint arXiv:1903.08560, 2019. + + Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual + networks. InEuropean conference on computer vision, pp. 630–645. Springer, 2016. + + Yanping Huang, Yonglong Cheng, Dehao Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, and + Zhifeng Chen. Gpipe: Efficient training of giant neural networks using pipeline parallelism. + CoRR, abs/1811.06965, 2018. URLhttp://arxiv.org/abs/1811.06965. + + Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009. + + Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- + lutional neural networks. InAdvances in neural information processing systems, pp. 1097–1105, + 2012. + + Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. + Towards deep learning models resistant to adversarial attacks.arXiv preprint arXiv:1706.06083, + 2017. + + Song Mei and Andrea Montanari. The generalization error of random features regression: Precise + asymptotics and double descent curve.arXiv preprint arXiv:1908.05355, 2019. + + Partha P. Mitra. Understanding overfitting peaks in generalization error: Analytical risk curves for + l2 and l1 penalized interpolation.ArXiv, abs/1906.03667, 2019. + + Vidya Muthukumar, Kailas Vodrahalli, and Anant Sahai. Harmless interpolation of noisy data in + regression.arXiv preprint arXiv:1903.09139, 2019. + + Preetum Nakkiran, Gal Kaplun, Dimitris Kalimeris, Tristan Yang, Benjamin L Edelman, Fred + Zhang, and Boaz Barak. Sgd on neural networks learns functions of increasing complexity.arXiv + preprint arXiv:1905.11604, 2019. + + Manfred Opper. Statistical mechanics of learning: Generalization.The Handbook of Brain Theory + and Neural Networks, 922-925., 1995. + + Manfred Opper. Learning to generalize.Frontiers of Life, 3(part 2), pp.763-775., 2001. + + Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, + and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. InProceedings of + NAACL-HLT 2019: Demonstrations, 2019. + + David Page. How to train your resnet. https://myrtle.ai/how-to-train-your-resnet-4-architecture/, 2018. + + Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, + Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in + PyTorch. InNeurIPS Autodiff Workshop, 2017. + + Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language + models are unsupervised multitask learners. 2019. + + Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. InAdvances in + neural information processing systems, pp. 1177–1184, 2008. + + Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with + subword units.ArXiv, abs/1508.07909, 2015. + + Stefano Spigler, Mario Geiger, Stephane d’Ascoli, Levent Sagun, Giulio Biroli, and Matthieu Wyart.´ + A jamming transition from under-to over-parametrization affects loss landscape and generaliza- + tion.arXiv preprint arXiv:1810.09665, 2018. + + Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- + mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In + Computer Vision and Pattern Recognition (CVPR), 2015. URLhttp://arxiv.org/abs/ + 1409.4842. + + Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, + Lukasz Kaiser, and Illia Polosukhin. Attention is all you need.CoRR, abs/1706.03762, 2017. + + Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding + deep learning requires rethinking generalization.ICLR, abs/1611.03530, 2016. + + + + A SUMMARY TABLE OF EXPERIMENTAL RESULTS + + <
> + + + B APPENDIX: EXPERIMENTAL DETAILS + + B.1 MODELS + + We use the following families of architectures. The PyTorch Paszke et al. (2017) + specification of our ResNets and CNNs are available at https://gitlab.com/harvard-machine-learning/double-descent/tree/master. + + ResNets. We define a family of ResNet18s of increasing size as follows. We follow the Preac- + tivation ResNet18 architecture of He et al. (2016), using 4 ResNet blocks, each consisting of two + BatchNorm-ReLU-Convolution layers. The layer widths for the 4 blocks are [k;2k;4k;8k] for + varyingk2Nand the strides are [1, 2, 2, 2]. The standard ResNet18 corresponds to k=64 con- + volutional channels in the first layer. The scaling of model size withkis shown in Figure 13b. Our + implementation is adapted from https://github.com/kuangliu/pytorch-cifar. + + Standard CNNs. We consider a simple family of 5-layer CNNs, with four Conv-BatchNorm- + ReLU-MaxPool layers and a fully-connected output layer. We scale the four convolutional layer + widths as [k;2k;4k;8k]. The MaxPool is [1, 2, 2, 8]. For all the convolution layers, the kernel + size = 3, stride = 1 and padding=1. This architecture is based on the “backbone” architecture from + Page (2018). For k=64, this CNN has 1558026 parameters and can reach >90% test accuracy on + CIFAR-10 (Krizhevsky (2009)) with data-augmentation. The scaling of model size with k is shown + in Figure 13a. + + Transformers. We consider the encoder-decoder Transformer model from Vaswani et al. (2017) + with 6 layers and 8 attention heads per layer, as implemented by fairseq Ott et al. (2019). We scale + the size of the network by modifying the embedding dimension (d model), and scale the width of the + fully-connected layers proportionally (dff = 4d model). We train with 10% label smoothing and no + drop-out, for 80 gradient steps. + + <
> + + Figure 13: Scaling of model size with our parameterization of width & embedding dimension. + + + B.2 IMAGE CLASSIFICATION: EXPERIMENTAL SETUP + + We describe the details of training for CNNs and ResNets below. + Loss function:Unless stated otherwise, we use the cross-entropy loss for all the experiments. + Data-augmentation: In experiments where data-augmentation was used, we apply + RandomCrop(32, padding=4)andRandomHorizontalFlip. In experiments with + added label noise, the label for all augmentations of a given training sample are given the same + label. + Regularization:No explicit regularization like weight decay or dropout was applied unless explic- + itly stated. + Initialization:We use the default initialization provided by PyTorch for all the layers. + Optimization: + + Adam: Unless specified otherwise, learning rate was set at constant to 1e^4 and all other + parameters were set to their default PyTorch values. + SGD: Unless specified otherwise, learning rate schedule inverse-square root (defined be- + low) was used with initial learning rate <> and updates every L=512 gradient steps. + No momentum was used. + + We found our results are robust to various other natural choices of optimizers and learning rate + schedule. We used the above settings because (1) they optimize well, and (2) they do not require + experiment-specific hyperparameter tuning, and allow us to use the same optimization across many + experiments. + Batch size: All experiments use a batchsize of 128. + Learning rate schedule descriptions: + + Inverse-square root (<>): At gradient stept, the learning rate is set to <>. We set learning-rate with respect to number of gradient steps, and not epochs, <> + in order to allow comparison between experiments with varying train-set sizes. + Dynamic drop (<>, drop, patience): Starts with an initial learning rate of 0 and drops by + a factor of ’drop’ if the training loss has remained constant or become worse for ’patience’ + number of gradient steps. + + B.3 NEURAL MACHINE TRANSLATION: EXPERIMENTAL SETUP + + Here we describe the experimental setup for the neural machine translation experiments. + Training procedure. + + In this setting, the distributionDconsists of triples + + <> + + where V_src and V_tgt are the source and target vocabularies, the stringxis a sentence in the source + language,yis its translation in the target language, andiis the index of the token to be predicted by + the model. We assume that <> is distributed uniformly on <>. + A standard probabilistic model defines an autoregressive factorization of the likelihood: + + <> + + Given a set of training samplesS, we define + + <> + + In practice,S is not constructed from independent samples from D, but rather by first sampling + <<(x,y)>> and then including all <> in S. + For training transformers, we replicate the optimization procedure specified in Vaswani et al. (2017) + section 5.3, where the learning rate schedule consists of a “warmup” phase with linearly increasing + learning rate followed by a phase with inverse square-root decay. We preprocess the data using byte + pair encoding (BPE) as described in Sennrich et al. (2015). We use the implementation provided by + fairseq (https://github.com/pytorch/fairseq). + + Datasets. The IWSLT’14 German to English dataset contains TED Talks as described in Cettolo + et al. (2012). The WMT’14 English to French dataset is taken from http://www.statmt.org/wmt14/translation-task.html. + + B.4 PER-SECTION EXPERIMENTAL DETAILS + + Here we provide full details for experiments in the body, when not otherwise provided. + Introduction: Experimental Details Figure 1: All models were trained using Adam with learning- + rate 0.0001 for 4K epochs. Plotting means and standard deviations for 5 trials, with random network + initialization. + + Model-wise Double Descent: Experimental Details Figure 7: Plotting means and standard devia- + tions for 5 trials, with random network initialization. + Sample-wise Nonmonotonicity: Experimental DetailsFigure 11a: All models are trained with + SGD for 500K epochs, and data-augmentation. Bottom: Means and standard deviations from 5 + trials with random initialization, and random subsampling of the train set. + + C EXTENDED DISCUSSION OF RELATED WORK + + Belkin et al. (2018): This paper proposed, in very general terms, that the apparent contradiction + between traditional notions of the bias-variance trade-off and empirically successful practices in + deep learning can be reconciled under a double-descent curve—as model complexity increases, the + test error follows the traditional “U-shaped curve”, but beyond the point of interpolation, the error + starts todecrease. This work provides empirical evidence for the double-descent curve with fully + connected networks trained on subsets of MNIST, CIFAR10, SVHN and TIMIT datasets. They use + thel2 loss for their experiments. They demonstrate that neural networks are not an aberration in this + regard—double-descent is a general phenomenon observed also in linear regression with random + features and random forests. + + Theoretical works on linear least squares regression: A variety of papers have attempted to the- + oretically analyze this behavior in restricted settings, particularly the case of least squares regression + under various assumptions on the training data, feature spaces and regularization method. + + 1.Advani & Saxe (2017); Hastie et al. (2019) both consider the linear regression problem + stated above and analyze the generalization behavior in the asymptotic limit <> + using random matrix theory. Hastie et al. (2019) highlight that when the model is mis- + specified, the minimum of training error can occur for over-parameterized models + 2.Belkin et al. (2019) Linear least squares regression for two data models, where the input + data is sampled from a Gaussian and a Fourier series model for functions on a circle. They + provide a finite-sample analysis for these two cases + 3.Bartlett et al. (2019) provides generalization bounds for the minimuml2 -norm interpolant + for Gaussian features + 4.Muthukumar et al. (2019) characterize the fundamental limit of of any interpolating solu- + tion in the presence of noise and provide some interesting Fourier-theoretic interpretations. + 5.Mei & Montanari (2019): This work provides asymptotic analysis for ridge regression over + random features + + Similar double descent behavior was investigated in Opper (1995; 2001) + Geiger et al. (2019b) showed that deep fully connected networks trained on the MNIST dataset with + hinge loss exhibit a “jamming transition” when the number of parameters exceeds a threshold that + allows training to near-zero train loss. Geiger et al. (2019a) provide further experiments on CIFAR- + 10 with a convolutional network. They also highlight interesting behavior with ensembling around + the critical regime, which is consistent with our informal intuitions in Section 5 and our experiments + in Figures 28, 29. + Advani & Saxe (2017); Geiger et al. (2019b;a) also point out that double-descent is not observed + when optimal early-stopping is used. + + D RANDOM FEATURES: A CASE STUDY + + <
> + + Figure 14:Random Fourier Featureson the Fashion MNIST dataset. The setting is equivalent + to two-layer neural network witheix activation, with randomly-initialized first layer that is fixed + throughout training. The second layer is trained using gradient flow. + + + In this section, for completeness sake, we show that both the model- and sample-wise double de- + scent phenomena are not unique to deep neural networks—they exist even in the setting of Random + Fourier Features of Rahimi & Recht (2008). This setting is equivalent to a two-layer neural network + with <> activation. The first layer is initialized with aN(0;1 )Gaussian distribution and then + fixed throughout training. The width (or embedding dimension) d dof the first layer parameterizes + the model size. The second layer is initialized with0s and trained with MSE loss. + Figure 14 shows the grid of Test Error as a function of both number of samplesnand model sized. + Note that in this settingEMC =d(the embedding dimension). As a result, as demonstrated in the + figure, the peak follows the path ofn=d. Both model-wise and sample-wise (see figure 15) double + descent phenomena are captured, by horizontally and vertically crossing the grid, respectively. + + <
> + + Figure 15: Sample-wise double-descent slice for Random Fourier Features on the Fashion MNIST + dataset. In this figure the embedding dimension (number of random features) is 1000. + + E APPENDIX: ADDITIONAL EXPERIMENTS + + + E.1 EPOCH-WISE DOUBLE DESCENT: ADDITIONAL RESULTS + + Here, we provide a rigorous evaluation of epoch-wise double descent for a variety of optimizers and + learning rate schedules. We train ResNet18 on CIFAR-10 with data-augmentation and 20% label + noise with three different optimizers—Adam, SGD, SGD + Momentum (momentum set to 0.9) and + three different learning rate schedules—constant, inverse-square root, dynamic drop for differnet + values of initial learning rate. We observe that double-descent occurs reliably for all optimizers and + learning rate schedules and the peak of the double descent curve shifts with the interpolation point. + + <
> + + Figure 16:Epoch-wise double descentfor ResNet18 trained with Adam and multiple learning rate + schedules + + A practical recommendation resulting from epoch-wise double descent is that stopping the training + when the test error starts to increase may not always be the best strategy. In some cases, the test error + may decrease again after reaching a maximum, and the final value may be lower than the minimum + earlier in training. + + <
> + + Figure 17:Epoch-wise double descentfor ResNet18 trained with SGD and multiple learning rate + schedules + + <
> + + Figure 18:Epoch-wise double descentfor ResNet18 trained with SGD+Momentum and multiple + learning rate schedules + + + E.2 MODEL-WISE DOUBLE DESCENT: ADDITIONAL RESULTS + + E.2.1 CLEAN SETTINGS WITH MODEL-WISE DOUBLE DESCENT + + <
> + + Figure 19:Top:Train and test performance as a function of both model size and train epochs. + Bottom:Test error dynamics of the same model (ResNet18, on CIFAR-100 with no label noise, + data-augmentation and Adam optimizer trained for 4k epochs with learning rate 0.0001). Note that + even with optimal early stopping this setting exhibits double descent. + + <
> + + Figure 20:Top:Train and test performance as a function of both model size and train epochs. + Bottom:Test error dynamics of the same models. 5-Layer CNNs, CIFAR-100 with no label noise, + no data-augmentation Trained with SGD for 1e6 steps. Same experiment as Figure 7. + + + E.2.2 WEIGHT DECAY + + <
> + + Figure 21:Left:Test error dynamics with weight decay of 5e-4 (bottom left) and without weight + decay (top left). Right:Test and train error andtest lossfor models with varying amounts of + weight decay. All models are 5-Layer CNNs on CIFAR-10 with 10% label noise, trained with + data-augmentation and SGD for 500K steps. + + Here, we now study the effect of varying the level of regularization on test error. We train CIFAR10 + with data-augmentation and 20% label noise on ResNet18 for weight decay coefficients <> rang- + ing from 0 to 0.1. We train the networks using SGD + inverse-square root learning rate. Figure + below shows a picture qualitatively very similar to that observed for model-wise double descent + wherein ”model complexity” is now controlled by the regularization parameter. This confirms our + generalized double descent hypothesis along yet another axis of Effective Model Complexity. + + <
> + + Figure 22: Generalized double descent for weight decay. We found that using the same initial + learning rate for all weight decay values led to training instabilities. This resulted in some noise in + the Test Error (Weight DecayEpochs) plot shown above. + + + E.2.3 EARLY STOPPING DOES NOT EXHIBIT DOUBLE DESCENT + + <
> + + Figure 23: Model-wise test error dynamics for a subsampled IWSLT‘14 dataset. Left: 4k samples, + Right: 18k samples. Note that with optimal early-stopping, more samples is always better. + + <
> + + Figure 24: Model-wise test error dynamics for a IWSLT‘14 de-en and subsampled WMT‘14 en-fr + datasets.Left: IWSLT‘14,Right: subsampled (200k samples) WMT‘14. Note that with optimal + early-stopping, the test error is much lower for this task. + + <
> + + Figure 25:Top:Train and test performance as a function of both model size and train epochs. + Bottom:Test error dynamics of the same model (CNN, on CIFAR-10 with 10% label noise, data-paugmentation and SGD optimizer with learning rate/1= T). + + + E.2.4 TRAINING PROCEDURE + + <
> + + Figure 26:Model-wise double descent for adversarial trainingResNet18s on CIFAR-10 (sub- + sampled to 25k train samples) with no label noise. We train for L2 robustness of radius <> and + <>, using 10-step PGD (Goodfellow et al. (2014); Madry et al. (2017)). Trained using SGD + (batch size 128) with learning rate0:1for 400 epochs, then0:01for 400 epochs. + + <
> + + Figure 27 + + + E.3 ENSEMBLING + + <
> + + Figure 28:Effect of Ensembling (ResNets, 15% label noise). Test error of an ensemble of 5 + models, compared to the base models. The ensembled classifier is determined by plurality vote over + the 5 base models. Note that emsembling helps most around the critical regime. All models are + ResNet18s trained on CIFAR-10 with 15% label noise, using Adam for 4K epochs (same setting + as Figure 1). Test error is measured against the original (not noisy) test set, and each model in the + ensemble is trained using a train set with independently-sampled 15% label noise. + + <
> + + Figure 29:Effect of Ensembling (CNNs, no label noise). Test error of an ensemble of 5 models, + compared to the base models. All models are 5-layer CNNs trained on CIFAR-10 with no label + noise, using SGD and no data augmentation. (same setting as Figure 7). +<> <> <> + + +<> <> <> +Deep Residual Learning for Image Recognition +Kaiming He Xiangyu Zhang Shaoqing Ren Jian Sun Microsoft Research {kahe, v-xiangz, v-shren, jiansun}@microsoft.com + +Abstract + +Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substantially deeper than those used previously. We explicitly reformulate the layers as learn.ing residual functions with reference to the layer inputs, in.stead of learning unreferenced functions. We provide comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth. On the ImageNet dataset we evaluate residual nets with a depth of up to 152 layers 8. deeper than VGG nets [41] but still having lower complex. +ity. An ensemble of these residual nets achieves 3.57% error on the ImageNet test set. This result won the 1st place on the ILSVRC 2015 classification task. We also present analysis on CIFAR-10 with 100 and 1000 layers. +The depth of representations is of central importance for many visual recognition tasks. Solely due to our extremely deep representations, we obtain a 28% relative improvement on the COCO object detection dataset. Deep residual nets are foundations of our submissions to ILSVRC & COCO 2015 competitions1 , where we also won the 1st places on the tasks of ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation. + +1. Introduction + +Deep convolutional neural networks [22, 21] have led to a series of breakthroughs for image classification [21, 50, 40]. Deep networks naturally integrate low/mid/high.level features [50] and classifiers in an end-to-end multi.layer fashion, and the levels of features can be enriched by the number of stacked layers (depth). Recent evidence [41, 44] reveals that network depth is of crucial importance, and the leading results [41, 44, 13, 16] on the challenging ImageNet dataset [36] all exploit very deep [41] models, with a depth of sixteen [41] to thirty [16]. Many other non.trivial visual recognition tasks [8, 12, 7, 32, 27] have also + + <
> + +Figure 1. Training error (left) and test error (right) on CIFAR-10 with 20-layer and 56-layer plain networks. The deeper network has higher training error, and thus test error. Similar phenomena on ImageNet is presented in Fig. 4. + +greatly benefited from very deep models. Driven by the significance of depth, a question arises: Is learning better networks as easy as stacking more layers? +An obstacle to answering this question was the notorious problem of vanishing/exploding gradients [1, 9], which hamper convergence from the beginning. This problem, however, has been largely addressed by normalized initialization [23, 9, 37, 13] and intermediate normalization layers [16], which enable networks with tens of layers to start con. +verging for stochastic gradient descent (SGD) with back-propagation [22]. +When deeper networks are able to start converging, a degradation problem has been exposed: with the network depth increasing, accuracy gets saturated (which might be unsurprising) and then degrades rapidly. Unexpectedly, such degradation is not caused by overfitting, and adding more layers to a suitably deep model leads to higher training error, as reported in [11, 42] and thoroughly verified by our experiments. Fig. 1 shows a typical example. +The degradation (of training accuracy) indicates that not all systems are similarly easy to optimize. Let us consider a shallower architecture and its deeper counterpart that adds more layers onto it. There exists a solution by construction to the deeper model: the added layers are identity mapping, and the other layers are copied from the learned shallower model. The existence of this constructed solution indicates that a deeper model should produce no higher training error than its shallower counterpart. But experiments show that our current solvers on hand are unable to find solutions that are comparably good or better than the constructed solution (or unable to do so in feasible time). + +In this paper, we address the degradation problem by introducing a deep residual learning framework. In.stead of hoping each few stacked layers directly fit a desired underlying mapping, we explicitly let these lay.ers fit a residual mapping. Formally, denoting the desired underlying mapping as <>, we let the stacked nonlinear layers fit another mapping of <>. The original mapping is recast into <>. We hypothesize that it is easier to optimize the residual mapping than to optimize the original, unreferenced mapping. To the extreme, if an identity mapping were optimal, it would be easier to push the residual to zero than to fit an identity mapping by a stack of nonlinear layers. +The formulation of <> can be realized by feedforward neural networks with shortcut connections (Fig. 2). Shortcut connections [2, 34, 49] are those skipping one or more layers. In our case, the shortcut connections simply perform identity mapping, and their outputs are added to the outputs of the stacked layers (Fig. 2). Identity short.cut connections add neither extra parameter nor computational complexity. The entire network can still be trained end-to-end by SGD with backpropagation, and can be easily implemented using common libraries (e.g., Caffe [19]) without modifying the solvers. +We present comprehensive experiments on ImageNet +[36] to show the degradation problem and evaluate our method. We show that: 1) Our extremely deep residual nets are easy to optimize, but the counterpart plain nets (that simply stack layers) exhibit higher training error when the depth increases; 2) Our deep residual nets can easily enjoy accuracy gains from greatly increased depth, producing results substantially better than previous networks. +Similar phenomena are also shown on the CIFAR-10 set [20], suggesting that the optimization difficulties and the effects of our method are not just akin to a particular dataset. We present successfully trained models on this dataset with over 100 layers, and explore models with over 1000 layers. +On the ImageNet classification dataset [36], we obtain excellent results by extremely deep residual nets. Our 152.layer residual net is the deepest network ever presented on ImageNet, while still having lower complexity than VGG nets [41]. Our ensemble has 3.57% top-5 error on the ImageNet test set, and won the 1st place in the ILSVRC 2015 classification competition. The extremely deep representations also have excellent generalization performance on other recognition tasks, and lead us to further win the 1st places on: ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation in ILSVRC & COCO 2015 competitions. This strong evidence shows that the residual learning principle is generic, and we expect that it is applicable in other vision and non-vision problems. + +2. Related Work Residual Representations. In image recognition, VLAD + +[18] is a representation that encodes by the residual vectors with respect to a dictionary, and Fisher Vector [30] can be formulated as a probabilistic version [18] of VLAD. Both of them are powerful shallow representations for image retrieval and classification [4, 48]. For vector quantization, encoding residual vectors [17] is shown to be more effective than encoding original vectors. +In low-level vision and computer graphics, for solving Partial Differential Equations (PDEs), the widely used Multigrid method [3] reformulates the system as subproblems at multiple scales, where each subproblem is responsible for the residual solution between a coarser and a finer scale. An alternative to Multigrid is hierarchical basis pre.conditioning [45, 46], which relies on variables that represent residual vectors between two scales. It has been shown [3, 45, 46] that these solvers converge much faster than standard solvers that are unaware of the residual nature of the solutions. These methods suggest that a good reformulation or preconditioning can simplify the optimization. +Shortcut Connections. Practices and theories that lead to shortcut connections [2, 34, 49] have been studied for a long time. An early practice of training multi-layer perceptrons (MLPs) is to add a linear layer connected from the network input to the output [34, 49]. In [44, 24], a few intermediate layers are directly connected to auxiliary classifiers for addressing vanishing/exploding gradients. The papers of [39, 38, 31, 47] propose methods for centering layer responses, gradients, and propagated errors, implemented by shortcut connections. In [44], an inception layer is composed of a shortcut branch and a few deeper branches. +Concurrent with our work, highway networks [42, 43] present shortcut connections with gating functions [15]. These gates are data-dependent and have parameters, in contrast to our identity shortcuts that are parameter-free. When a gated shortcut is closed (approaching zero), the layers in highway networks represent non-residual functions. On the contrary, our formulation always learns residual functions; our identity shortcuts are never closed, and all information is always passed through, with additional residual functions to be learned. In addition, high.way networks have not demonstrated accuracy gains with extremely increased depth (e.g., over 100 layers). + +3. Deep Residual Learning + +3.1. Residual Learning +Let us consider <> as an underlying mapping to be fit by a few stacked layers (not necessarily the entire net), with x denoting the inputs to the first of these layers. If one hypothesizes that multiple nonlinear layers can asymptotically approximate complicated functions, then it is equivalent to hypothesize that they can asymptotically approximate the residual functions, i.e., <> (assuming that the input and output are of the same dimensions). So rather than expect stacked layers to approximate <>, we explicitly let these layers approximate a residual function <>. The original function thus becomes <>. Although both forms should be able to asymptotically approximate the desired functions (as hypothesized), the ease of learning might be different. +This reformulation is motivated by the counterintuitive phenomena about the degradation problem (Fig. 1, left). As we discussed in the introduction, if the added layers can be constructed as identity mappings, a deeper model should have training error no greater than its shallower counterpart. The degradation problem suggests that the solvers might have difficulties in approximating identity mappings by multiple nonlinear layers. With the residual learning reformulation, if identity mappings are optimal, the solvers may simply drive the weights of the multiple nonlinear lay.ers toward zero to approach identity mappings. +In real cases, it is unlikely that identity mappings are optimal, but our reformulation may help to precondition the problem. If the optimal function is closer to an identity mapping than to a zero mapping, it should be easier for the solver to find the perturbations with reference to an identity mapping, than to learn the function as a new one. We show by experiments (Fig. 7) that the learned residual functions in general have small responses, suggesting that identity map.pings provide reasonable preconditioning. + +3.2. Identity Mapping by Shortcuts +We adopt residual learning to every few stacked layers. A building block is shown in Fig. 2. Formally, in this paper we consider a building block defined as: + +<>. (1) + +Here x and y are the input and output vectors of the lay.ers considered. The function <> represents the residual mapping to be learned. For the example in Fig. 2 that has two layers, <> in which <> denotes +ReLU [29] and the biases are omitted for simplifying notations. The operation <> is performed by a shortcut connection and element-wise addition. We adopt the second nonlinearity after the addition (i.e., <>, see Fig. 2). +The shortcut connections in Eqn.(1) introduce neither extra parameter nor computation complexity. This is not only attractive in practice but also important in our comparisons between plain and residual networks. We can fairly compare plain/residual networks that simultaneously have the same number of parameters, depth, width, and computational cost (except for the negligible element-wise addition). +The dimensions of x and F must be equal in Eqn.(1). If this is not the case (e.g., when changing the input/output channels), we can perform a linear projection Ws by the shortcut connections to match the dimensions: + +<>. (2) + +We can also use a square matrix <> in Eqn.(1). But we will show by experiments that the identity mapping is sufficient for addressing the degradation problem and is economical, and thus Ws is only used when matching dimensions. +The form of the residual function F is flexible. Experiments in this paper involve a function F that has two or three layers (Fig. 5), while more layers are possible. But if F has only a single layer, Eqn.(1) is similar to a linear layer: <>, for which we have not observed advantages. +We also note that although the above notations are about fully-connected layers for simplicity, they are applicable to convolutional layers. The function <> can represent multiple convolutional layers. The element-wise addition is performed on two feature maps, channel by channel. + +3.3. Network Architectures +We have tested various plain/residual nets, and have observed consistent phenomena. To provide instances for discussion, we describe two models for ImageNet as follows. +Plain Network. Our plain baselines (Fig. 3, middle) are mainly inspired by the philosophy of VGG nets [41] (Fig. 3, left). The convolutional layers mostly have 3.3 filters and follow two simple design rules: (i) for the same output feature map size, the layers have the same number of filters; and (ii) if the feature map size is halved, the number of filters is doubled so as to preserve the time complexity per layer. We perform downsampling directly by convolutional layers that have a stride of 2. The network ends with a global average pooling layer and a 1000-way fully-connected layer with softmax. The total number of weighted layers is 34 in Fig. 3 (middle). +It is worth noticing that our model has fewer filters and lower complexity than VGG nets [41] (Fig. 3, left). Our 34 layer baseline has 3.6 billion FLOPs (multiply-adds), which is only 18% of VGG-19 (19.6 billion FLOPs). + + +Residual Network. Based on the above plain network, we insert shortcut connections (Fig. 3, right) which turn the network into its counterpart residual version. The identity shortcuts (Eqn.(1)) can be directly used when the input and output are of the same dimensions (solid line shortcuts in Fig. 3). When the dimensions increase (dotted line shortcuts in Fig. 3), we consider two options: (A) The shortcut still performs identity mapping, with extra zero entries padded for increasing dimensions. This option introduces no extra parameter; (B) The projection shortcut in Eqn.(2) is used to match dimensions (done by 1.1 convolutions). For both options, when the shortcuts go across feature maps of two sizes, they are performed with a stride of 2. + +3.4. Implementation +Our implementation for ImageNet follows the practice in [21, 41]. The image is resized with its shorter side randomly sampled in [256, 480] for scale augmentation [41]. A 224.224 crop is randomly sampled from an image or its horizontal flip, with the per-pixel mean subtracted [21]. The standard color augmentation in [21] is used. We adopt batch normalization (BN) [16] right after each convolution and before activation, following [16]. We initialize the weights as in [13] and train all plain/residual nets from scratch. We use SGD with a mini-batch size of 256. The learning rate starts from 0.1 and is divided by 10 when the error plateaus, and the models are trained for up to 60 . 104 iterations. We use a weight decay of 0.0001 and a momentum of 0.9. We do not use dropout [14], following the practice in [16]. +In testing, for comparison studies we adopt the standard 10-crop testing [21]. For best results, we adopt the fully +convolutional form as in [41, 13], and average the scores at multiple scales (images are resized such that the shorter side is in {224, 256, 384, 480, 640}). + +4. Experiments + +4.1. ImageNet Classification +We evaluate our method on the ImageNet 2012 classification dataset [36] that consists of 1000 classes. The models are trained on the 1.28 million training images, and evaluated on the 50k validation images. We also obtain a final result on the 100k test images, reported by the test server. We evaluate both top-1 and top-5 error rates. +Plain Networks. We first evaluate 18-layer and 34-layer plain nets. The 34-layer plain net is in Fig. 3 (middle). The 18-layer plain net is of a similar form. See Table 1 for de. +tailed architectures. +The results in Table 2 show that the deeper 34-layer plain net has higher validation error than the shallower 18-layer plain net. To reveal the reasons, in Fig. 4 (left) we com.pare their training/validation errors during the training procedure. We have observed the degradation problem + + <
> + +Table 1. Architectures for ImageNet. Building blocks are shown in brackets (see also Fig. 5), with the numbers of blocks stacked. Down-sampling is performed by conv3 1, conv4 1, and conv5 1 with a stride of 2. + +Figure 4. Training on ImageNet. Thin curves denote training error, and bold curves denote validation error of the center crops. Left: plain networks of 18 and 34 layers. Right: ResNets of 18 and 34 layers. In this plot, the residual networks have no extra parameter compared to their plain counterparts. + + <
> + +Table 2. Top-1 error (%, 10-crop testing) on ImageNet validation. Here the ResNets have no extra parameter compared to their plain counterparts. Fig. 4 shows the training procedures. +34-layer plain net has higher training error throughout the whole training procedure, even though the solution space of the 18-layer plain network is a subspace of that of the 34-layer one. +We argue that this optimization difficulty is unlikely to be caused by vanishing gradients. These plain networks are trained with BN [16], which ensures forward propagated signals to have non-zero variances. We also verify that the backward propagated gradients exhibit healthy norms with BN. So neither forward nor backward signals vanish. In fact, the 34-layer plain net is still able to achieve compet.itive accuracy (Table 3), suggesting that the solver works to some extent. We conjecture that the deep plain nets may have exponentially low convergence rates, which impact the reducing of the training error3. The reason for such opti.mization difficulties will be studied in the future. +Residual Networks. Next we evaluate 18-layer and 34.layer residual nets (ResNets). The baseline architectures are the same as the above plain nets, expect that a shortcut connection is added to each pair of 3.3 filters as in Fig. 3 (right). In the first comparison (Table 2 and Fig. 4 right), we use identity mapping for all shortcuts and zero-padding for increasing dimensions (option A). So they have no extra parameter compared to the plain counterparts. +We have three major observations from Table 2 and Fig. 4. First, the situation is reversed with residual learn.ing fi the 34-layer ResNet is better than the 18-layer ResNet (by 2.8%). More importantly, the 34-layer ResNet exhibits considerably lower training error and is generalizable to the validation data. This indicates that the degradation problem is well addressed in this setting and we manage to obtain accuracy gains from increased depth. +Second, compared to its plain counterpart, the 34-layer +3We have experimented with more training iterations (3.) and still ob.served the degradation problem, suggesting that this problem cannot be feasibly addressed by simply using more iterations. + + <
> + +Table 3. Error rates (%, 10-crop testing) on ImageNet validation. VGG-16 is based on our test. ResNet-50/101/152 are of option B that only uses projections for increasing dimensions. + + <
> + +Table 4. Error rates (%) of single-model results on the ImageNet validation set (except fi reported on the test set). + + <
> + +Table 5. Error rates (%) of ensembles. The top-5 error is on the test set of ImageNet and reported by the test server. + + <
> + +ResNet reduces the top-1 error by 3.5% (Table 2), resulting from the successfully reduced training error (Fig. 4 right vs. left). This comparison verifies the effectiveness of residual learning on extremely deep systems. +Last, we also note that the 18-layer plain/residual nets are comparably accurate (Table 2), but the 18-layer ResNet converges faster (Fig. 4 right vs. left). When the net is not overly deep (18 layers here), the current SGD solver is still able to find good solutions to the plain net. In this case, the ResNet eases the optimization by providing faster convergence at the early stage. +Identity vs. Projection Shortcuts. We have shown that + +Figure 5. A deeper residual function F for ImageNet. Left: a building block (on 56.56 feature maps) as in Fig. 3 for ResNet. + + <
> + +parameter-free, identity shortcuts help with training. Next we investigate projection shortcuts (Eqn.(2)). In Table 3 we compare three options: (A) zero-padding shortcuts are used for increasing dimensions, and all shortcuts are parameter-free (the same as Table 2 and Fig. 4 right); (B) projection shortcuts are used for increasing dimensions, and other shortcuts are identity; and (C) all shortcuts are projections. +Table 3 shows that all three options are considerably bet. +ter than the plain counterpart. B is slightly better than A. We argue that this is because the zero-padded dimensions in A indeed have no residual learning. C is marginally better than B, and we attribute this to the extra parameters introduced by many (thirteen) projection shortcuts. But the small differences among A/B/C indicate that projection shortcuts are not essential for addressing the degradation problem. So we do not use option C in the rest of this paper, to reduce mem.ory/time complexity and model sizes. Identity shortcuts are particularly important for not increasing the complexity of the bottleneck architectures that are introduced below. +Deeper Bottleneck Architectures. Next we describe our deeper nets for ImageNet. Because of concerns on the train.ing time that we can afford, we modify the building block as a bottleneck design4. For each residual function F, we use a stack of 3 layers instead of 2 (Fig. 5). The three layers are 1.1, 3.3, and 1.1 convolutions, where the 1.1 layers are responsible for reducing and then increasing (restoring) dimensions, leaving the 3.3 layer a bottleneck with smaller input/output dimensions. Fig. 5 shows an example, where both designs have similar time complexity. +The parameter-free identity shortcuts are particularly important for the bottleneck architectures. If the identity short.cut in Fig. 5 (right) is replaced with projection, one can show that the time complexity and model size are doubled, as the shortcut is connected to the two high-dimensional ends. So identity shortcuts lead to more efficient models for the bottleneck designs. +50-layer ResNet: We replace each 2-layer block in the +4Deeper non-bottleneck ResNets (e.g., Fig. 5 left) also gain accuracy from increased depth (as shown on CIFAR-10), but are not as economical as the bottleneck ResNets. So the usage of bottleneck designs is mainly due to practical considerations. We further note that the degradation problem of plain nets is also witnessed for the bottleneck designs. + +34-layer net with this 3-layer bottleneck block, resulting in a 50-layer ResNet (Table 1). We use option B for increasing dimensions. This model has 3.8 billion FLOPs. +101-layer and 152-layer ResNets: We construct 101.layer and 152-layer ResNets by using more 3-layer blocks (Table 1). Remarkably, although the depth is significantly increased, the 152-layer ResNet (11.3 billion FLOPs) still has lower complexity than VGG-16/19 nets (15.3/19.6 bil.lion FLOPs). +The 50/101/152-layer ResNets are more accurate than the 34-layer ones by considerable margins (Table 3 and 4). We do not observe the degradation problem and thus enjoy significant accuracy gains from considerably increased depth. The benefits of depth are witnessed for all evaluation metrics (Table 3 and 4). +Comparisons with State-of-the-art Methods. In Table 4 we compare with the previous best single-model results. Our baseline 34-layer ResNets have achieved very competitive accuracy. Our 152-layer ResNet has a single-model top-5 validation error of 4.49%. This single-model result outperforms all previous ensemble results (Table 5). We combine six models of different depth to form an ensemble (only with two 152-layer ones at the time of submitting). This leads to 3.57% top-5 error on the test set (Table 5). +This entry won the 1st place in ILSVRC 2015. + +4.2. CIFAR-10 and Analysis + +We conducted more studies on the CIFAR-10 dataset [20], which consists of 50k training images and 10k test.ing images in 10 classes. We present experiments trained on the training set and evaluated on the test set. Our focus is on the behaviors of extremely deep networks, but not on pushing the state-of-the-art results, so we intentionally use simple architectures as follows. +The plain/residual architectures follow the form in Fig. 3 (middle/right). The network inputs are 32.32 images, with the per-pixel mean subtracted. The first layer is 3.3 convolutions. Then we use a stack of 6n layers with 3.3 convolutions on the feature maps of sizes {32, 16, 8} respectively, with 2n layers for each feature map size. The numbers of filters are {16, 32, 64} respectively. The subsampling is per.formed by convolutions with a stride of 2. The network ends with a global average pooling, a 10-way fully-connected layer, and softmax. There are totally 6n+2 stacked weighted layers. The following table summarizes the architecture: + +<
> + +When shortcut connections are used, they are connected to the pairs of 3.3 layers (totally 3n shortcuts). On this dataset we use identity shortcuts in all cases (i.e., option A), + +<
> + +Table 6. Classification error on the CIFAR-10 test set. All meth.ods are with data augmentation. For ResNet-110, we run it 5 times and show best (mean std) as in [43]. + +so our residual models have exactly the same depth, width, and number of parameters as the plain counterparts. +We use a weight decay of 0.0001 and momentum of 0.9, and adopt the weight initialization in [13] and BN [16] but with no dropout. These models are trained with a mini-batch size of 128 on two GPUs. We start with a learning rate of 0.1, divide it by 10 at 32k and 48k iterations, and terminate training at 64k iterations, which is determined on a 45k/5k train/val split. We follow the simple data augmen.tation in [24] for training: 4 pixels are padded on each side, and a 32.32 crop is randomly sampled from the padded image or its horizontal fiip. For testing, we only evaluate the single view of the original 32.32 image. +We compare n = {3, 5, 7, 9}, leading to 20, 32, 44, and 56-layer networks. Fig. 6 (left) shows the behaviors of the plain nets. The deep plain nets suffer from increased depth, and exhibit higher training error when going deeper. This phenomenon is similar to that on ImageNet (Fig. 4, left) and on MNIST (see [42]), suggesting that such an optimization difficulty is a fundamental problem. +Fig. 6 (middle) shows the behaviors of ResNets. Also similar to the ImageNet cases (Fig. 4, right), our ResNets manage to overcome the optimization difficulty and demon.strate accuracy gains when the depth increases. +We further explore n = 18 that leads to a 110-layer ResNet. In this case, we find that the initial learning rate of 0.1 is slightly too large to start converging5. So we use +0.01 to warm up the training until the training error is below 80% (about 400 iterations), and then go back to 0.1 and continue training. The rest of the learning schedule is as done previously. This 110-layer network converges well (Fig. 6, middle). It has fewer parameters than other deep and thin +5With an initial learning rate of 0.1, it starts converging (<90% error) after several epochs, but still reaches similar accuracy. + +<
> + +Figure 7. Standard deviations (std) of layer responses on CIFAR. +10. The responses are the outputs of each 3.3 layer, after BN and before nonlinearity. Top: the layers are shown in their original order. Bottom: the responses are ranked in descending order. +networks such as FitNet [35] and Highway [42] (Table 6), yet is among the state-of-the-art results (6.43%, Table 6). +Analysis of Layer Responses. Fig. 7 shows the standard deviations (std) of the layer responses. The responses are the outputs of each 3.3 layer, after BN and before other nonlinearity (ReLU/addition). For ResNets, this analysis reveals the response strength of the residual functions. Fig. 7 shows that ResNets have generally smaller responses than their plain counterparts. These results support our ba.sic motivation (Sec.3.1) that the residual functions might be generally closer to zero than the non-residual functions. We also notice that the deeper ResNet has smaller magnitudes of responses, as evidenced by the comparisons among ResNet-20, 56, and 110 in Fig. 7. When there are more layers, an individual layer of ResNets tends to modify the signal less. +Exploring Over 1000 layers. We explore an aggressively deep model of over 1000 layers. We set n = 200 that leads to a 1202-layer network, which is trained as described above. Our method shows no optimization difficulty, and this 103-layer network is able to achieve training error <0.1% (Fig. 6, right). Its test error is still fairly good (7.93%, Table 6). +But there are still open problems on such aggressively deep models. The testing result of this 1202-layer network is worse than that of our 110-layer network, although both + +<
> + +Table 7. Object detection mAP (%) on the PASCAL VOC 2007/2012 test sets using baseline Faster R-CNN. See also Ta.ble 10 and 11 for better results. + +<
> + +Table 8. Object detection mAP (%) on the COCO validation set using baseline Faster R-CNN. See also Table 9 for better results. +have similar training error. We argue that this is because of overfitting. The 1202-layer network may be unnecessarily large (19.4M) for this small dataset. Strong regularization such as maxout [10] or dropout [14] is applied to obtain the best results ([10, 25, 24, 35]) on this dataset. In this paper, we use no maxout/dropout and just simply impose regularization via deep and thin architectures by design, without distracting from the focus on the difficulties of optimization. But combining with stronger regularization may im.prove results, which we will study in the future. + +4.3. Object Detection on PASCAL and MS COCO +Our method has good generalization performance on other recognition tasks. Table 7 and 8 show the object detection baseline results on PASCAL VOC 2007 and 2012 +[5] and COCO [26]. We adopt Faster R-CNN [32] as the detection method. Here we are interested in the improvements of replacing VGG-16 [41] with ResNet-101. The detection implementation (see appendix) of using both models is the same, so the gains can only be attributed to better networks. Most remarkably, on the challenging COCO dataset we ob.tain a 6.0% increase in COCOfis standard metric (mAP@[.5, .95]), which is a 28% relative improvement. This gain is solely due to the learned representations. +Based on deep residual nets, we won the 1st places in several tracks in ILSVRC & COCO 2015 competitions: Im.ageNet detection, ImageNet localization, COCO detection, and COCO segmentation. The details are in the appendix. + +References +[1] Y. Bengio, P. Simard, and P. Frasconi. Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, 5(2):157fi166, 1994. +[2] C. M. Bishop. Neural networks for pattern recognition. Oxford university press, 1995. +[3] W. L. Briggs, S. F. McCormick, et al. A Multigrid Tutorial. Siam, 2000. +[4] K. Chatfield, V. Lempitsky, A. Vedaldi, and A. Zisserman. The devil is in the details: an evaluation of recent feature encoding methods. In BMVC, 2011. +[5] M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman. The Pascal Visual Object Classes (VOC) Challenge. IJCV, pages 303fi338, 2010. +[6] S. Gidaris and N. Komodakis. Object detection via a multi-region & semantic segmentation-aware cnn model. In ICCV, 2015. +[7] R. Girshick. Fast R-CNN. In ICCV, 2015. +[8] R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hier.archies for accurate object detection and semantic segmentation. In CVPR, 2014. +[9] X. Glorot and Y. Bengio. Understanding the difficulty of training deep feedforward neural networks. In AISTATS, 2010. +[10] I. J. Goodfellow, D. Warde-Farley, M. Mirza, A. Courville, and Y. Bengio. Maxout networks. arXiv:1302.4389, 2013. +[11] K. He and J. Sun. Convolutional neural networks at constrained time cost. In CVPR, 2015. +[12] K. He, X. Zhang, S. Ren, and J. Sun. Spatial pyramid pooling in deep convolutional networks for visual recognition. In ECCV, 2014. +[13] K. He, X. Zhang, S. Ren, and J. Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In ICCV, 2015. +[14] G. E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and R. R. Salakhutdinov. Improving neural networks by preventing co-adaptation of feature detectors. arXiv:1207.0580, 2012. +[15] S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 9(8):1735fi1780, 1997. +[16] S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015. +[17] H. Jegou, M. Douze, and C. Schmid. Product quantization for nearest neighbor search. TPAMI, 33, 2011. +[18] H. Jegou, F. Perronnin, M. Douze, J. Sanchez, P. Perez, and C. Schmid. Aggregating local image descriptors into compact codes. TPAMI, 2012. +[19] Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. Caffe: Convolutional architecture for fast feature embedding. arXiv:1408.5093, 2014. +[20] A. Krizhevsky. Learning multiple layers of features from tiny im.ages. Tech Report, 2009. +[21] A. Krizhevsky, I. Sutskever, and G. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012. +[22] Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel. Backpropagation applied to hand.written zip code recognition. Neural computation, 1989. +[23] Y. LeCun,L.Bottou,G.B.Orr,andK.-R.Mfiuller. Efficientbackprop. In Neural Networks: Tricks of the Trade, pages 9fi50. Springer, 1998. +[24] C.-Y. Lee, S. Xie, P. Gallagher, Z. Zhang, and Z. Tu. Deeply-supervised nets. arXiv:1409.5185, 2014. +[25] M. Lin, Q. Chen, and S. Yan. Network in network. arXiv:1312.4400, 2013. +[26] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollfiar, and C. L. Zitnick. Microsoft COCO: Common objects in context. In ECCV. 2014. +[27] J. Long, E. Shelhamer, and T. Darrell. Fully convolutional networks for semantic segmentation. In CVPR, 2015. +[28] G. Montfiufar, R. Pascanu, K. Cho, and Y. Bengio. On the number of linear regions of deep neural networks. In NIPS, 2014. +[29] V. Nair and G. E. Hinton. Rectified linear units improve restricted boltzmann machines. In ICML, 2010. +[30] F. Perronnin and C. Dance. Fisher kernels on visual vocabularies for image categorization. In CVPR, 2007. +[31] T. Raiko, H. Valpola, and Y. LeCun. Deep learning made easier by linear transformations in perceptrons. In AISTATS, 2012. +[32] S. Ren, K. He, R. Girshick, and J. Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. In NIPS, 2015. +[33] S. Ren, K. He, R. Girshick, X. Zhang, and J. Sun. Object detection networks on convolutional feature maps. arXiv:1504.06066, 2015. +[34] B. D. Ripley. Pattern recognition and neural networks. Cambridge university press, 1996. +[35] A. Romero, N. Ballas, S. E. Kahou, A. Chassang, C. Gatta, and Y. Bengio. Fitnets: Hints for thin deep nets. In ICLR, 2015. +[36] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al. Imagenet large scale visual recognition challenge. arXiv:1409.0575, 2014. +[37] A. M. Saxe, J. L. McClelland, and S. Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv:1312.6120, 2013. +[38] N. N. Schraudolph. Accelerated gradient descent by factor-centering decomposition. Technical report, 1998. +[39] N. N. Schraudolph. Centering neural network gradient factors. In Neural Networks: Tricks of the Trade, pages 207fi226. Springer, 1998. +[40] P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus, and Y. Le-Cun. Overfeat: Integrated recognition, localization and detection using convolutional networks. In ICLR, 2014. +[41] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015. +[42] R. K. Srivastava, K. Greff, and J. Schmidhuber. Highway networks. arXiv:1505.00387, 2015. +[43] R. K. Srivastava, K. Greff, and J. Schmidhuber. Training very deep networks. 1507.06228, 2015. +[44] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Er.han, V. Vanhoucke, and A. Rabinovich. Going deeper with convolu.tions. In CVPR, 2015. +[45] R. Szeliski. Fast surface interpolation using hierarchical basis func.tions. TPAMI, 1990. +[46] R. Szeliski. Locally adapted hierarchical basis preconditioning. In SIGGRAPH, 2006. +[47] T. Vatanen, T. Raiko, H. Valpola, and Y. LeCun. Pushing stochas.tic gradient towards second-order methodsfibackpropagation learn.ing with transformations in nonlinearities. In Neural Information Processing, 2013. +[48] A. Vedaldi and B. Fulkerson. VLFeat: An open and portable library of computer vision algorithms, 2008. +[49] W. Venables and B. Ripley. Modern applied statistics with s-plus. 1999. +[50] M. D. Zeiler and R. Fergus. Visualizing and understanding convolu.tional neural networks. In ECCV, 2014. + +A. Object Detection Baselines + +In this section we introduce our detection method based on the baseline Faster R-CNN [32] system. The models are initialized by the ImageNet classification models, and then fine-tuned on the object detection data. We have experi.mented with ResNet-50/101 at the time of the ILSVRC & COCO 2015 detection competitions. +Unlike VGG-16 used in [32], our ResNet has no hidden fc layers. We adopt the idea of fiNetworks on Conv feature maps (NoC) [33] to address this issue. We compute the full-image shared conv feature maps using those lay.ers whose strides on the image are no greater than 16 pixels (i.e., conv1, conv2 x, conv3 x, and conv4 x, totally 91 conv layers in ResNet-101; Table 1). We consider these layers as analogous to the 13 conv layers in VGG-16, and by doing so, both ResNet and VGG-16 have conv feature maps of the same total stride (16 pixels). These layers are shared by a region proposal network (RPN, generating 300 proposals) +[32] and a Fast R-CNN detection network [7]. RoI pool.ing [7] is performed before conv5 1. On this RoI-pooled feature, all layers of conv5 x and up are adopted for each region, playing the roles of VGG-16fis fc layers. The final classification layer is replaced by two sibling layers (classi.fication and box regression [7]). +For the usage of BN layers, after pre-training, we compute the BN statistics (means and variances) for each layer on the ImageNet training set. Then the BN layers are fixed during fine-tuning for object detection. As such, the BN layers become linear activations with constant offsets and scales, and BN statistics are not updated by fine-tuning. We fix the BN layers mainly for reducing memory consumption in Faster R-CNN training. +PASCAL VOC +Following [7, 32], for the PASCAL VOC 2007 test set, we use the 5k trainval images in VOC 2007 and 16k train-val images in VOC 2012 for training (fi07+12fi). For the PASCAL VOC 2012 test set, we use the 10k trainval+test images in VOC 2007 and 16k trainval images in VOC 2012 for training (fi07++12fi). The hyper-parameters for train.ing Faster R-CNN are the same as in [32]. Table 7 shows the results. ResNet-101 improves the mAP by >3% over VGG-16. This gain is solely because of the improved features learned by ResNet. +MS COCO +The MS COCO dataset [26] involves 80 object cate.gories. We evaluate the PASCAL VOC metric (mAP @ IoU = 0.5) and the standard COCO metric (mAP @ IoU = .5:.05:.95). We use the 80k images on the train set for train.ing and the 40k images on the val set for evaluation. Our detection system for COCO is similar to that for PASCAL VOC. We train the COCO models with an 8-GPU imple.mentation, and thus the RPN step has a mini-batch size of 8 images (i.e., 1 per GPU) and the Fast R-CNN step has a mini-batch size of 16 images. The RPN step and Fast R.CNN step are both trained for 240k iterations with a learn.ing rate of 0.001 and then for 80k iterations with 0.0001. +Table 8 shows the results on the MS COCO validation set. ResNet-101 has a 6% increase of mAP@[.5, .95] over VGG-16, which is a 28% relative improvement, solely con.tributed by the features learned by the better network. Re.markably, the mAP@[.5, .95]fis absolute increase (6.0%) is nearly as big as mAP@.5fis (6.9%). This suggests that a deeper network can improve both recognition and localiza.tion. +B. Object Detection Improvements +For completeness, we report the improvements made for the competitions. These improvements are based on deep features and thus should benefit from residual learning. +MS COCO +Box refinement. Our box refinement partially follows the it.erative localization in [6]. In Faster R-CNN, the final output is a regressed box that is different from its proposal box. So for inference, we pool a new feature from the regressed box and obtain a new classification score and a new regressed box. We combine these 300 new predictions with the orig.inal 300 predictions. Non-maximum suppression (NMS) is applied on the union set of predicted boxes using an IoU threshold of 0.3 [8], followed by box voting [6]. Box re.finement improves mAP by about 2 points (Table 9). +Global context. We combine global context in the Fast R-CNN step. Given the full-image conv feature map, we pool a feature by global Spatial Pyramid Pooling [12] (with a fisingle-levelfi pyramid) which can be implemented as fiRoIfi pooling using the entire imagefis bounding box as the RoI. This pooled feature is fed into the post-RoI layers to obtain a global context feature. This global feature is con.catenated with the original per-region feature, followed by the sibling classification and box regression layers. This new structure is trained end-to-end. Global context im.proves mAP@.5 by about 1 point (Table 9). +Multi-scale testing. In the above, all results are obtained by single-scale training/testing as in [32], where the imagefis shorter side is s = 600 pixels. Multi-scale training/testing has been developed in [12, 7] by selecting a scale from a feature pyramid, and in [33] by using maxout layers. In our current implementation, we have performed multi-scale testing following [33]; we have not performed multi-scale training because of limited time. In addition, we have per.formed multi-scale testing only for the Fast R-CNN step (but not yet for the RPN step). With a trained model, we compute conv feature maps on an image pyramid, where the imagefis shorter sides are s .{200, 400, 600, 800, 1000}. + + <
> + +Table 9. Object detection improvements on MS COCO using Faster R-CNN and ResNet-101. + + <
> + +Table 10. Detection results on the PASCAL VOC 2007 test set. The baseline is the Faster R-CNN system. The system fibaseline+++fi include box refinement, context, and multi-scale testing in Table 9. + + +system net data mAP areo bike bird boat bottle bus car cat chair cow table dog horse mbike person plant sheep sofa train tv +baseline baseline baseline+++ VGG-16 ResNet-101 ResNet-101 07++12 07++12 COCO+07++12 70.4 73.8 83.8 84.9 79.8 74.3 53.9 49.8 77.5 75.9 88.5 45.6 77.1 55.3 86.9 81.7 80.9 79.6 40.1 72.6 60.9 81.2 61.5 86.5 81.6 77.2 58.0 51.0 78.6 76.6 93.2 48.6 80.4 59.0 92.1 85.3 84.8 80.7 48.1 77.3 66.5 84.7 65.6 92.1 88.4 84.8 75.9 71.4 86.3 87.8 94.2 66.8 89.4 69.2 93.9 91.9 90.9 89.6 67.9 88.2 76.8 90.3 80.0 +Table 11. Detection results on the PASCAL VOC 2012 test set (http://host.robots.ox.ac.uk:8080/leaderboard/displaylb.php?challengeid=11&compid=4). The baseline is the Faster R-CNN system. The system baseline+++ include box refinement, context, and multi-scale testing in Table 9. + + +We select two adjacent scales from the pyramid following [33]. RoI pooling and subsequent layers are performed on the feature maps of these two scales [33], which are merged by maxout as in [33]. Multi-scale testing improves the mAP by over 2 points (Table 9). +Using validation data. Next we use the 80k+40k trainval set for training and the 20k test-dev set for evaluation. The test.dev set has no publicly available ground truth and the result is reported by the evaluation server. Under this setting, the results are an mAP@.5 of 55.7% and an mAP@[.5, .95] of 34.9% (Table 9). This is our single-model result. +Ensemble. In Faster R-CNN, the system is designed to learn region proposals and also object classifiers, so an ensemble can be used to boost both tasks. We use an ensemble for proposing regions, and the union set of proposals are pro.cessed by an ensemble of per-region classifiers. Table 9 shows our result based on an ensemble of 3 networks. The mAP is 59.0% and 37.4% on the test-dev set. This result won the 1st place in the detection task in COCO 2015. + +We revisit the PASCAL VOC dataset based on the above model. With the single model on the COCO dataset (55.7% mAP@.5 in Table 9), we fine-tune this model on the PAS.CAL VOC sets. The improvements of box refinement, con.text, and multi-scale testing are also adopted. By doing so we achieve 85.6% mAP on PASCAL VOC 2007 (Table 10) and 83.8% on PASCAL VOC 2012 (Table 11)6. The result on PASCAL VOC 2012 is 10 points higher than the previ.ous state-of-the-art result [6]. + +<
> + +Table 12. Our results (mAP, %) on the ImageNet detection dataset. Our detection system is Faster R-CNN [32] with the improvements in Table 9, using ResNet-101. + + +ImageNet Detection +The ImageNet Detection (DET) task involves 200 object categories. The accuracy is evaluated by mAP@.5. Our object detection algorithm for ImageNet DET is the same as that for MS COCO in Table 9. The networks are pre.trained on the 1000-class ImageNet classification set, and are fine-tuned on the DET data. We split the validation set into two parts (val1/val2) following [8]. We fine-tune the detection models using the DET training set and the val1 set. The val2 set is used for validation. We do not use other ILSVRC 2015 data. Our single model with ResNet-101 has + +<
> + +Table 13. Localization error (%) on the ImageNet validation. In the column of fiLOC error on GT classfi ([41]), the ground truth class is used. In the fitestingfi column, fi1-cropfi denotes testing on a center crop of 224.224 pixels, fidensefi denotes dense (fully convolutional) and multi-scale testing. + +<
> + +Table 14. Comparisons of localization error (%) on the ImageNet dataset with state-of-the-art methods. + + +58.8% mAP and our ensemble of 3 models has 62.1% mAP on the DET test set (Table 12). This result won the 1st place in the ImageNet detection task in ILSVRC 2015, surpassing the second place by 8.5 points (absolute). + +C. ImageNet Localization + +The ImageNet Localization (LOC) task [36] requires to classify and localize the objects. Following [40, 41], we assume that the image-level classifiers are first adopted for predicting the class labels of an image, and the localiza.tion algorithm only accounts for predicting bounding boxes based on the predicted classes. We adopt the fiper-class re.gressionfi (PCR) strategy [40, 41], learning a bounding box regressor for each class. We pre-train the networks for Im.ageNet classification and then fine-tune them for localiza.tion. We train networks on the provided 1000-class Ima.geNet training set. +Our localization algorithm is based on the RPN frame.work of [32] with a few modifications. Unlike the way in +[32] that is category-agnostic, our RPN for localization is designed in a per-class form. This RPN ends with two sib.ling 1.1 convolutional layers for binary classification (cls) and box regression (reg), as in [32]. The cls and reg layers are both in a per-class from, in contrast to [32]. Specifi.cally, the cls layer has a 1000-d output, and each dimension is binary logistic regression for predicting being or not be.ing an object class; the reg layer has a 1000.4-d output consisting of box regressors for 1000 classes. As in [32], our bounding box regression is with reference to multiple translation-invariant fianchorfi boxes at each position. +As in our ImageNet classification training (Sec. 3.4), we randomly sample 224.224 crops for data augmentation. We use a mini-batch size of 256 images for fine-tuning. To avoid negative samples being dominate, 8 anchors are ran.domly sampled for each image, where the sampled positive and negative anchors have a ratio of 1:1 [32]. For testing, the network is applied on the image fully-convolutionally. +Table 13 compares the localization results. Following [41], we first perform fioraclefi testing using the ground truth class as the classification prediction. VGGfis paper [41] re-ports a center-crop error of 33.1% (Table 13) using ground truth classes. Under the same setting, our RPN method us.ing ResNet-101 net significantly reduces the center-crop er.ror to 13.3%. This comparison demonstrates the excellent performance of our framework. With dense (fully convolu.tional) and multi-scale testing, our ResNet-101 has an error of 11.7% using ground truth classes. Using ResNet-101 for predicting classes (4.6% top-5 classification error, Table 4), the top-5 localization error is 14.4%. +The above results are only based on the proposal network (RPN) in Faster R-CNN [32]. One may use the detection network (Fast R-CNN [7]) in Faster R-CNN to improve the results. But we notice that on this dataset, one image usually contains a single dominate object, and the proposal regions highly overlap with each other and thus have very similar RoI-pooled features. As a result, the image-centric training of Fast R-CNN [7] generates samples of small variations, which may not be desired for stochastic training. Motivated by this, in our current experiment we use the original R-CNN [8] that is RoI-centric, in place of Fast R-CNN. +Our R-CNN implementation is as follows. We apply the per-class RPN trained as above on the training images to predict bounding boxes for the ground truth class. These predicted boxes play a role of class-dependent proposals. For each training image, the highest scored 200 proposals are extracted as training samples to train an R-CNN classi.fier. The image region is cropped from a proposal, warped to 224.224 pixels, and fed into the classification network as in R-CNN [8]. The outputs of this network consist of two sibling fc layers for cls and reg, also in a per-class form. This R-CNN network is fine-tuned on the training set us.ing a mini-batch size of 256 in the RoI-centric fashion. For testing, the RPN generates the highest scored 200 proposals for each predicted class, and the R-CNN network is used to update these proposalsfi scores and box positions. +This method reduces the top-5 localization error to 10.6% (Table 13). This is our single-model result on the validation set. Using an ensemble of networks for both clas.sification and localization, we achieve a top-5 localization error of 9.0% on the test set. This number significantly out.performs the ILSVRC 14 results (Table 14), showing a 64% relative reduction of error. This result won the 1st place in the ImageNet localization task in ILSVRC 2015. +<> <> <> + + +<> <> <> + Direct Feedback Alignment Scales to Modern Deep Learning Tasks and Architectures + + Julien Launay 1;2 Iacopo Poli 1 François Boniface 1 Florent Krzakala 1;2 + + 1 LightOn 2 École Normale Supérieure + + Abstract + + Despite being the workhorse of deep learning, the backpropagation algorithm is + no panacea. It enforces sequential layer updates, thus preventing efficient paral- + lelization of the training process. Furthermore, its biological plausibility is being + challenged. Alternative schemes have been devised; yet, under the constraint of + synaptic asymmetry, none have scaled to modern deep learning tasks and architec- + tures. Here, we challenge this perspective, and study the applicability of Direct + Feedback Alignment to neural view synthesis, recommender systems, geometric + learning, and natural language processing. In contrast with previous studies lim- + ited to computer vision tasks, our findings show that it successfully trains a large + range of state-of-the-art deep learning architectures, with performance close to + fine-tuned backpropagation. At variance with common beliefs, our work supports + that challenging tasks can be tackled in the absence of weight transport. + + + 1 Introduction + + While the backpropagation algorithm (BP) [1,2] is at the heart of modern deep learning achievements, + it is not without pitfalls. For one, its weight updates are non-local and rely on upstream layers. Thus, + they cannot be easily parallelized [3], incurring important memory and compute costs. Moreover, + its biological implementation is problematic [4,5]. For instance, BP relies on the transpose of the + weights to evaluate updates. Hence, synaptic symmetry is required between the forward and backward + path: this is implausible in biological brains, and known as the weight transport problem [6]. + Consequently, alternative training algorithms have been developed. Some of these algorithms are + explicitly biologically inspired [7–13], while others focus on making better use of available compute + resources [3,14–19]. Despite these enticing characteristics, none has been widely adopted, as they + are often demonstrated on a limited set of tasks. Moreover, as assessed in [20], their performance on + challenging datasets under the constraint of synaptic asymmetry is disappointing. + We seek to broaden this perspective, and demonstrate the applicability of Direct Feedback Alignment + (DFA) [19] in state-of-the-art settings: from applications of fully connected networks such as neural + view synthesis and recommender systems, to geometric learning with graph convolutions, and natural + language processing with Transformers. Our results define new standards for learning without weight + transport and show that challenging tasks can indeed be tackled under synaptic asymmetry. + All code needed to reproduce our experiments is available at https://github.com/lightonai/dfa-scales-to-modern-deep-learning. + + 1.1 Related work + + Training a neural network is a credit assignment problem: an update is derived for each parameter + from its contribution to a cost function. To solve this problem, a spectrum of algorithms exists [21]. + + Biologically motivated methods Finding a training method applicable under the constraints of + biological brains remains an open problem. End-to-end propagation of gradients is unlikely to occur + [22], implying local learning is required. Furthermore, the weight transport problem enforces synaptic + asymmetry [6]. Inspired by auto-encoders, target propagation methods (TP) [10–12] train distinct + feedback connections to invert the feedforward ones. Feedback alignment (FA) [13] replaces the + transpose of the forward weights used in the backward pass by a random matrix. Throughout training, + the forward weights learn to align with the arbitrary backward weights, eventually approximating BP. + + Beyond biological considerations As deep learning models grow bigger, large-scale distributed + training is increasingly desirable. Greedy layer-wise training [14] allows networks to be built layer + by layer, limiting the depth of backpropagation. To enable parallelization of the backward pass, + updates must only depend on local quantities. Unsupervised learning is naturally suited for this, + as it relies on local losses such as Deep InfoMax [17] and Greedy InfoMax [18]. More broadly, + synthetic gradient methods, like decoupled neural interfaces [3,15] and local error signals (LES) + [16], approximate gradients using layer-wise trainable feedback networks. DFA [19] expands on FA + and directly projects a global error to each layer. A shared feedback path is still needed, but it only + depends on a simple random projection operation. + + Performance of alternative methods Local training methods are successful in unsupervised learn- + ing [18]. Even in a supervised setting, they scale to challenging datasets like CIFAR-100 or ImageNet + [14,16]. Thus, locality is not too penalizing. However, TP, FA, and DFA are unable to scale to these + tasks [20]. In fact, DFA is unable to train convolutional layers [23]. To enable feedback alignment + techniques to perform well on challenging datasets, some form of weight transport is necessary: + either by explicitly sharing sign information [24–26], or by introducing dedicated phases of alignment + for the forward and backward weights where some information is shared [27]. To the best of our + knowledge, no method compatible with the weight transport problem has ever been demonstrated on + challenging tasks. + + 1.2 Motivations and contributions + + We focus on DFA, a compromise between biological and computational considerations. Notably, + DFA is compatible with synaptic asymmetry: this asymmetry raises important challenges, seemingly + preventing learning in demanding settings. Moreover, it allows for asynchronous weight updates, + and puts a single operation at the center of the training stage. This enables new classes of training + co-processors [28, 29], leveraging dedicated hardware to perform the random projection. + + Extensive survey We apply DFA in a large variety of settings matching current trends in machine + learning. Previous works have found that DFA is unsuitable for computer vision tasks [20,23]; but + computer vision alone cannot be the litmus test of a training method. Instead, we consider four vastly + different domains, across eight tasks, and with eleven different architectures. This constitutes a survey + of unprecedented scale for an alternative training method, and makes a strong case for the possibility + of learning without weight transport in demanding scenarios. + + Challenging settings We demonstrate the ability of DFA to tackle challenging tasks. We success- + fully learn and render real-world 3D scenes (section 3.1.1); we perform recommendation at scale + (section 3.1.2); we explore graph-based citation networks (section 3.2); and we consider language + modelling with a Transformer (section 3.3). We study tasks at the state-of-the-art level, that have + only been recently successfully tackled with deep learning. + + Modern architectures We prove that the previously established failure of DFA to train convolutions + does not generalize. By evaluating performance metrics, comparing against a shallow baseline, + measuring alignment, and visualizing t-SNE embeddings, we show that learning indeed occurs in + layers involving graph convolutions and attention. This significantly broadens the applicability of + DFA–previously thought to be limited to simple problems like MNIST and CIFAR-10. + + 2 Methods + + Forward pass In a fully connected network, at layer i out of N, neglecting its biases, with W_i its + weight matrix, f_i its non-linearity, and hi its activations, the forward pass is: + + <> (1) + + <> is the input data, and <> are the predictions. A task-specific cost function + <> is computed to quantify the quality of the predictions with respect to the targets y. + + Backward pass with BP The weight updates are computed by backpropagation of the error vector. + Using the chain-rule of derivatives, each neuron is updated based on its contribution to the cost + function. Leaving aside the specifics of the optimizer used, the equation for the weight updates is: + + <> (2) + + Backward pass with DFA The gradient signal <> of the (i+1)-th layer violates synaptic + asymmetry. DFA replaces it with a random projection of the topmost derivative of the loss, <>. + For common classification and regression losses such as the mean squared error or the negative log + likelihood, this corresponds to a random projection of the global error <>. With B_i, a fixed + random matrix of appropriate shape drawn at initialization for each layers: + + <> (3) + + 3 Experiments + + We study the applicability of DFA to a diverse set of applications requiring state-of-the-art architec- + tures. We start with fully connected networks, where DFA has already been demonstrated, and address + new challenging settings. We then investigate geometric learning: we apply DFA to graph neural net- + works in classification tasks on citation networks, as well as graph autoencoders. These architectures + feature graph convolutions and attention layers. Finally, we use DFA to train a transformer-based + Natural Language Processing (NLP) model on a dataset of more than 100 million tokens. + + 3.1 Fully connected architectures + + DFA has been successful at training fully connected architectures, with performance on-par with + backpropagation [19,20]. However, only computer vision tasks have been considered, where fully + connected networks considerably underperform their convolutional counterpart. Here, we focus on + tasks where fully connected architectures are state-of-the-art. Moreover, the architectures considered + are deeper and more complex than those necessary to solve a simple task like MNIST. + + 3.1.1 Neural view synthesis with Neural Radiance Fields + The most recent state-of-the-art neural view synthesis methods are based on large fully connected + networks: this is an ideal setting for a first evaluation of DFA on a challenging task. + + Background There has been growing interest in methods capable of synthesizing novel renders of + a 3D scene using a dataset of past renders. The network is trained to learn an inner representation of + the scene, and a classical rendering system can then query the model to generate novel views. With + robust enough methods, real-world scenes can also be learned from a set of pictures. + Until recently, most successful neural view synthesis methods were based on sampled volumetric + representations [30–32]. In this context, Convolutional Neural Networks (CNNs) can be used to + smooth out the discrete sampling of 3D space [33,34]. However, these methods scale poorly to + higher resolutions, as they still require finer and finer sampling. Conversely, alternative schemes + based on a continuous volume representation have succeeded in generating high-quality renders [35], + even featuring complex phenomenons such as view-dependant scattering [36]. These schemes make + point-wise predictions, and use fully connected neural networks to encode the scene. + + <
> + + Figure 1: Comparisons of NeRF-DFA with state-of-the-art methods trained with BP on the most + challenging synthetic and real-world scenes. While NeRF-DFA generates render of lower quality, + they maintain multi-view consistency and exhibit no geometric artifacts. BP results from [36]. + + + Setting We employ Neural Radiance Fields (NeRF) [36], the state-of-the-art for neural view + synthesis. NeRF represents scenes as a continuous 5D function of space–three spatial coordinates, + two viewing angles–and outputs a point-wise RGB radiance and opacity. A ray-casting renderer can + then query the network to generate arbitrary views of the scene. The network modeling the continuous + function is 10 layers deep. Two identical networks are trained: the coarse network predictions inform + the renderer about the spatial coordinates that the fine network should preferentially evaluate to avoid + empty space and occluded regions. + + Results We report quantitative results of training NeRF with DFA in Table 1. Neural view synthesis + methods are often better evaluated qualitatively: we showcase some renders in Figure 1. + On a dataset of renders featuring complex scenes with non-Lambertian materials (NeRF-Synthetic + [36]), NeRF-DFA outperforms two previous fine-tuned state-of-the-art methods–Scene Representation + Networks (SRN) [35] and Local Light Field Fusion (LLFF) [32]–and nearly matches the performance + of Neural Volumes (NV) [34]. While DFA underperforms alternative methods trained with BP on + the real world view dataset (LLFF-Real [32]), its performance remains significant: real world view + synthesis is a challenging tasks, and this level of PSNR indicates that learning is indeed happening. + In particular, we find that NeRF-DFA retains the key characteristics of NeRF-BP: it can render view- + dependant effects, and is multi-view consistent. The last point is an especially important achievement, + and most visible in videos, as it is a challenge for most algorithms [30–32,35]. The main drawback + of NeRF-DFA appears to be a seemingly lower render definition. The NeRF architecture has not + + + Table 1: Peak Signal to Noise Ratio (PSNR, higher is better) of neural view synthesis methods + trained with backpropagation against NeRF trained with DFA. Even when trained with DFA, NeRF + outperforms two state-of-the-art methods on a synthetic dataset (NeRF-Synthetic), and achieves fair + performance on a challenging real world views datasets (LLFF-Real). BP results from [36]. + + <
> + + been fine-tuned to achieve these results: DFA works out-of-the-box on this advanced method. Future + research focusing on architectural changes to NeRF could improve performance with DFA; some + preliminary results are included in the supplementary material. + + 3.1.2 Click-through rate prediction with recommender systems + We have demonstrated that DFA can train large fully connected networks on the difficult task of neural + view synthesis. We now seek to use DFA in more complex heterogeneous architectures, combining + the use of fully connected networks with other machine learning methods.Recommender systems are + an ideal application for such considerations. + + Background Recommender systems are used to model the behavior of users and predict future + interactions. In particular, in the context of click-through rate (CTR) prediction, these systems model + the probability of a user clicking on a given item. Building recommender systems is hard [37]: their + input is high-dimensional and sparse, and the model must learn to extract high-order combinatorial + features from the data. Moreover, they need to do so efficiently, as they are used to make millions of + predictions and the training data may contain billions of examples. + Factorization Machines (FM) [38] use inner-products of latent vectors between features to extract + pairwise feature interactions. They constitute an excellent baseline for shallow recommender systems, + but fail to efficiently transcribe higher-level features. To avoid extensive feature engineering, it has + been suggested that deep learning can be used in conjunction with wide shallow models to extract + these higher-level features [39]. In production, these systems are regularly retrained on massive + datasets: the speedup allowed by backward unlocking in DFA is thus of particular interest. + + Setting Deep Factorization Machines (DeepFM) [40] combine FM and a deep fully connected + neural network, which we train with DFA. The input embedding is still trained directly via gradient + descent, as weight transport is not necessary to backpropagate through the FM. Deep & Cross + Networks (DCN) [41] replace the FM with a Cross Network, a deep architecture without non- + linearities capable of extracting high-degree interactions across features. We train the fully connected + network, the deep cross network, and the embeddings with DFA. Finally, Adaptative Factorization + Network (AFN) [42] uses Logarithmic Neural Networks [43] to enhance the representational power + of its deep component. We evaluate these methods on the Criteo dataset [44], which features nearly + 46 million samples of one million sparse features. This is a difficult task, where performance + improvements of the AUC on the 0.001-level can enhance CTR significantly [39]. + + Results Performance metrics are reported in Table 2. To obtain these results, a simple hyperpa- + rameter grid search over optimization and regularization parameters was performed for BP and DFA + independently. DFA successfully trains all methods above the FM baseline, and in fact matches BP + performance in both DeepFM and AFN. Because of their complexity, recommender systems require + intensive tuning and feature engineering to perform at the state-of-the-art level–and reproducing + existing results can be challenging [45]. Hence, it is not surprising that a performance gap exists with + Deep&Cross–further fine-tuning may be necessary for DFA to reach BP performance. + Alignment measurements corroborate that learning is indeed occurring in the special layers of + Deep&Cross and AFN–see supplementary for details. Our results on recommender systems support + that DFA can learn in a large variety of settings, and that weight transport is not necessary to solve a + difficult recommendation task. + + + Table 2: AUC (higher is better) and log loss (lower is better) of recommender systems trained on the + Criteo dataset [44]. Even in complex heterogeneous architectures, DFA performance is in line with + BP. Values in bold indicate DFA AUC within 0.001 from the BP AUC or better. + + <
> + + + 3.2 Geometric Learning with Graph Convolutional Networks + + The use of sophisticated architectures beyond fully connected layers is necessary for certain tasks, + such as geometric learning[46], where information lies in a complex structured domain. To address + geometric learning tasks, methods capable of handling graph-based data are commonly needed. + Graph convolutional neural networks (GCNNs) [47–50] have demonstrated the ability to process + large-scale graph data efficiently. We study the applicability of DFA to these methods, including + recent architectures based on an attention mechanism. Overall, this is an especially interesting setting, + as DFA fails to train more classic 2D image convolutional layers [23]. + + Background Complex data like social networks or brain connections lie on irregular or non- + Euclidean domains. They can be represented as graphs, and efficient processing in the spectral + domain is possible. Non-spectral techniques to apply neural networks to graphs have also been + developed [51–53], but they exhibit unfavorable scaling properties. The success of CNNs in deep + learning can be attributed to their ability to efficiently process structured high-dimensional data + by sharing local filters. Thus, a generalization of the convolution operator to the graph domain is + desirable: [47] first proposed a spectral convolution operation for graphs, and [48] introduced a form + of regularization to enforce spatial locality of the filters. We use DFA to train different such GCNNs + implementations. We study both spectral and non-spectral convolutions, as well as methods inspired + by the attention mechanism. We consider the task of semi-supervised node classification: nodes from + a graph are classified using their relationship to other nodes as well as node-wise features. + + Setting Fast Localized Convolutions (ChebConv) [49] approximate the graph convolution kernel + with Chebyshev polynomials, and are one of the first scalable convolution methods on graph. Graph + Convolutions (GraphConv) [50] remove the need for an explicit parametrization of the kernel by + enforcing linearity of the convolution operation on the graph Laplacian spectrum. It is often considered + as the canonical graph convolution. More recent methods do not operate in the spectral domain. Spline + Convolutions (SplineConv) [54] use a spline-based kernel, enabling the inclusion of information + about the relative positioning of nodes, enhancing their representational power–for instance in the + context of 3D meshes. Graph Attention Networks (GATConv) [55] use self-attention [56] layers to + enable predictions at a given node to attend more specifically to certain parts of its neighborhood. + Finally, building upon Jumping Knowledge Network [57], Just Jump (DNAConv) [58] uses multi- + head attention [59] to enhance the aggregation process in graph convolutions and enable deeper + architectures. We use PyTorch Geometric [60] for reference implementation of all of these methods. + We evaluate performance on three citation network datasets: Cora, CiteSeer, and PubMed [61]. + + Results We report classification accuracy in Table 3. BP and DFA regularization and optimiza- + tion hyperparameters are fine-tuned separately on the Cora dataset. In general, we find that less + regularization and lower learning rates are needed with DFA. DFA successfully trains all graph + methods, independent of whether they use the spectral domain or not, and even if they use attention. + Furthermore, for GraphConv, SplineConv, and GATConv DFA performance nearly matches BP. + As GCNNs struggle with learning meaningful representations when stacking many layers [62], all + architectures but DNAConv are quite shallow (two layers). However, DFA performance is still + significantly higher than that of a shallow training method–see supplementary for details. The lower + performance on DNAConv is not a failure to learn: alignment measurements show that learning is + indeed occurring. It may be explained instead by a need for more in-depth fine-tuning, as this is a + deep architecture with 5 successive attention layers. + + Table 3: Classification accuracy (%, higher is better) of graph convolution methods trained with BP + and DFA, on citation networks [61]. But for ChebConv and DNAConv, DFA performance nearly + matches BP performance. Values in bold when DFA is within 2.5% of BP. + + <
> + + Table 4: AUC and Average Precision Figure 2: t-SNE visualization of the hidden layer + (AP, higher is better) for a Graph- activations of a two-layer GraphConv trained on + Conv GAE trained with BP or DFA Cora with DFA. Classes forms clear clusters, indicating + that a useful intermediary representation is learned. Colors represent different classes. + on citation networks. DFA reproduces BP performance. + + + We further demonstrate that DFA helps graph convolutions learn meaningful representations by + applying t-SNE [63,64] to the hidden layer activations in GraphConv (Figure 2). Cluster of classes + are well-separated, indicating that a useful intermediary representation is derived by the first layer. + + Graph autoencoders We consider one last application of graph convolutions, in the context of + graph autoencoders (GAE). We train a non-probabilistic GAE [65] based on GraphConv with DFA, + and report results in Table 4. DFA performance is always in line with BP. + + 3.3 Natural Language Processing with Transformers + + We complete our study by training a Transformer [59] on a language modelling task. Transformers + have proved successful in text, image, music generation, machine translation, and many supervised + NLP tasks [59,66–69]. Here, we demonstrate that DFA can train them, and we show the influence of + tuning the optimizer hyperparameters in narrowing the gap with BP. + + Background NLP has largely benefited from advances in deep learning. Recurrent Neural Net- + works were responsible for early breakthroughs, but their sequential nature prevented efficient + parallelization of data processing. Transformers are attention-based models that do not rely on + recurrence or convolution. Their ability to scale massively has allowed the training of models with + several billion parameters [70,71], obtaining state-of-the-art results on all NLP tasks: Transformers + now top the prominent SQuAD 2.0 [72,73] and SuperGLUE [74] benchmarks. In parallel, transfer + learning in NLP has leaped forward thanks to language modelling, the unsupervised task of predicting + the next word. It can leverage virtually unlimited data from web scraping [75]. This enabled the + training of universal language models[76] on extremely large and diversified text corpora. These + models are useful across a wide range of domains, and can solve most NLP tasks after fine-tuning. + + Setting The prominence of both language modelling and Transformers gives us the ideal candidate + for our NLP experiments: we train a Transformer to predict the next word on the WikiText-103 + dataset [77], a large collection of good and featured Wikipedia articles. We use byte-pair-encoding + [78] with 32,000 tokens. Our setup is similar to GPT [66]: we adapt the Transformer, originally an + encoder-decoder model designed for machine translation, to language modelling. We keep only the + encoder and mask the tokens to predict. Our architecture consists in 6 layers, 8 attention heads, a + model dimension of 512, and a hidden size of 2048 in the feed-forward blocks. The text is sliced + in chunks of 128 tokens and batches of 64 such chunks, resulting in 8192 tokens per batch. Our + baseline is trained with BP using the optimization setup of [59]. We found perplexity after 20 epochs + to be an excellent indicator of perplexity at convergence; to maximize the number of experiments + we could perform, we report the best validation perplexity after 20 epochs. We study two ways of + implementing DFA: applying the feedback after every encoder block (macro) or after every layer in + those blocks (micro). The input embedding layer receives gradients from the next feedback point + through BP. This leaves some amount of weight transport even in the micro-case. + + Table 5: Best validation perplexity after 20 epochs of a Transformer trained on WikiText-103 (lower + is better). The BP and DFA baselines share all hyper-parameters. In Macro the feedback is applied + after every transformer layer, while in Micro the feedback is applied after every sub-layer. The + learning rate of Adam without the learning rate scheduler is <>. With the scheduler, the initial + learning rate is <> and it is multiplied by 0.2 when performance plateaus, with a patience of 1. + * score after 22 epochs to let the learning rate scheduler take effect + + <
> + + Results Our results are summarized in Table 5. Hyper-parameters fine-tuned for BP did not fare + well with DFA, but changes in the optimizer narrowed the gap between BP and DFA considerably. + The learning rate schedule used on top of Adam [79] in [59] proved detrimental. Using Adam alone + required reducing the learning rate between BP and DFA. Increasing 2 from 0.98 [59] to 0.999 + improved performance significantly. Finally, a simple scheduler that reduces the learning rate when + the validation perplexity plateaus helped reducing it further. Considering that the perplexity of the + shallow baseline is over 400, DFA is clearly able to train Transformers. However, our results are not + on par with BP, especially in the micro setting. A substantial amount of work remains to make DFA + competitive with BP, even more so in a minimal weight transport scenario. The large performance + improvements brought by small changes in the optimizer indicate that intensive fine-tuning, common + in publications introducing state-of-the-art results, could close the gap between BP and DFA. + + 4 Conclusion and outlooks + + We conducted an extensive study demonstrating the ability of DFA to train modern architectures. We + considered a broad selection of domains and tasks, with complex models featuring graph convolutions + and attention. Our results on large networks like NeRF and Transformers are encouraging, suggesting + that with further tuning, such leading architectures can be effectively trained with DFA. Future work + on principled training with DFA–in particular regarding the influence of common practices and + whether new procedures are required–will help close the gap with BP. + More broadly, we verified for the first time that learning under synaptic asymmetry is possible beyond + fully-connected layers, and in tasks significantly more difficult than previously considered. This + addresses a notable concern in biologically-plausible architectures. DFA still requires an implausible + global feedback pathway; however, local training has already been demonstrated at scale. The next + step towards biologically-compatible learning is a local method without weight transport. + While the tasks and architectures we have considered are not biologically inspired, they constitute + a good benchmark for behavioral realism[20]. Any learning algorithm claiming to approximate + the brain should reproduce its ability to solve complex and unseen task. Furthermore, even though + the current implementation of mechanisms like attention is devoid of biological considerations, they + represent broader concepts applicable to human brains [80]. Understanding how our brain learns is a + gradual process, and future research could incorporate further realistic elements, like spiking neurons. + Finally, unlocking the backward pass in large architectures like Transformers is promising. More opti- + mized implementation of DFA–built at a lower-level of existing ML libraries–could unlock significant + speed-up. Leveraging the use of a single random projection as the cornerstone of training, dedicated + accelerators may employ more exotic hardware architectures. This will open new possibilities in the + asynchronous training of massive models. + + Broader Impact + + Of our survey This study is the first experimental validation of DFA as an effective training method + in a wide range of challenging tasks and neural networks architectures. This significantly broadens the + applications of DFA, and more generally brings new insight on training techniques alternative to back- + propagation. From neural rendering and recommender systems, to natural language processing or + geometric learning, each of these applications has its own potential impact. Our task selection process + was motivated by current trends in deep learning, as well as by technically appealing mechanisms + (graph convolutions, attention). A limit of our survey is that our–arguably biased–selection of tasks + cannot be exhaustive. Our experiments required substantial cloud compute resources, with state-of- + the-art GPU hardware. Nevertheless, as this study provides new perspectives for hardware accelerator + technologies, it may favor the application of neural networks in fields previously inaccessible because + of computational limits. Future research on DFA should continue to demonstrate its use in novel + contexts of interest as they are discovered. + + Of the considered applications Each of the applications considered in our study has a wide + potential impact, consider for example the impact of textual bias in pretrained word embeddings [81]. + We refer to [82] and references therein for a discussion of ethical concerns of AI applications. + + Of DFA as a training method DFA enables parallelization of the backward pass and places a + single operation at the center of the training process, opening the prospect of reducing the power + consumption of training chips by an order of magnitude [28]. Not only is more efficient training a + path to more environmentally responsible machine learning [83], but it may lower the barrier of entry, + supporting equality and sustainable development goals. A significant downside of moving from BP to + DFA is a far more limited understanding of how to train models and how the trained models behave. + There is a clear empirical understanding of the impact of techniques such as batch normalization + or skip connections on the performance of BP; new insights need to be obtained for DFA. BP also + enjoys decades of works on topics like adversarial attacks, interpretability, and fairness. Much of + this work has to be cross-checked for alternative training methods, something we encourage further + research to consider as the next step towards safely and responsively scaling up DFA. + + Of biologically motivated method Finally, a key motivation for this study was to demonstrate that + learning challenging tasks was possible without weight transport. Biologically motivated methods + are a more foundational research direction, and as such the possible long-term impact of our findings + is harder to estimate under this light. However, fundamental research of this kind is important to open + new pathways for ML and neuroscience. + + Acknowledgments and Disclosure of Funding + + We thank Igor Carron and Laurent Daudet for the general guidance on the subject of this investigation + and the insightful comments, as well as the larger LightOn team for their support. + + References + [1]P. J. Werbos.Beyond Regression: New Tools for Prediction and Analysis in the Behavioral + Sciences. PhD thesis, Harvard University, 1974. + [2]D. E. Rumelhart, G. E. Hinton, and R. J. Williams. Learning internal representations by error + propagation. InParallel Distributed Processing, volume 1, pages 318–362. MIT Press, 1986. + [3]Max Jaderberg, Wojciech Marian Czarnecki, Simon Osindero, Oriol Vinyals, Alex Graves, + David Silver, and Koray Kavukcuoglu. Decoupled neural interfaces using synthetic gradients. + InProceedings of the 34th International Conference on Machine Learning-Volume 70, pages + 1627–1635, 2017. + [4]Francis Crick. The recent excitement about neural networks.Nature, 337(6203):129–132, 1989. + [5]Adam H Marblestone, Greg Wayne, and Konrad P Kording. Toward an integration of deep + learning and neuroscience.Frontiers in computational neuroscience, 10:94, 2016. + [6]Stephen Grossberg. Competitive learning: From interactive activation to adaptive resonance. + Cognitive science, 11(1):23–63, 1987. + [7]Javier R Movellan. Contrastive hebbian learning in the continuous hopfield model. InConnec- + tionist models, pages 10–17. Elsevier, 1991. + [8]Randall C O’Reilly. Biologically plausible error-driven learning using local activation differ- + ences: The generalized recirculation algorithm.Neural computation, 8(5):895–938, 1996. + [9]Ruslan Salakhutdinov and Geoffrey Hinton. Deep boltzmann machines. InArtificial intelligence + and statistics, pages 448–455, 2009. + [10]Yann Le Cun. Learning process in an asymmetric threshold network. InDisordered systems + and biological organization, pages 233–240. Springer, 1986. + [11]Yoshua Bengio. How auto-encoders could provide credit assignment in deep networks via target + propagation.arXiv preprint arXiv:1407.7906, 2014. + [12]Dong-Hyun Lee, Saizheng Zhang, Asja Fischer, and Yoshua Bengio. Difference target propaga- + tion. InJoint european conference on machine learning and knowledge discovery in databases, + pages 498–515. Springer, 2015. + [13]Timothy P Lillicrap, Daniel Cownden, Douglas B Tweed, and Colin J Akerman. Random synap- + tic feedback weights support error backpropagation for deep learning.Nature communications, + 7(1):1–10, 2016. + [14]Eugene Belilovsky, Michael Eickenberg, and Edouard Oyallon. Greedy layerwise learning can + scale to imagenet. InInternational Conference on Machine Learning, pages 583–593, 2019. + [15]Wojciech M Czarnecki, Simon Osindero, Max Jaderberg, Grzegorz Swirszcz, and Razvan + Pascanu. Sobolev training for neural networks. InAdvances in Neural Information Processing + Systems, pages 4278–4287, 2017. + [16]Arild Nøkland and Lars Hiller Eidnes. Training neural networks with local error signals. In + International Conference on Machine Learning, pages 4839–4850, 2019. + [17]R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, + Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information + estimation and maximization. InInternational Conference on Learning Representations, 2019. + URLhttps://openreview.net/forum?id=Bklr3j0cKX. + [18]Sindy Löwe, Peter O’Connor, and Bastiaan Veeling. Putting an end to end-to-end: Gradient- + isolated learning of representations. InAdvances in Neural Information Processing Systems, + pages 3033–3045, 2019. + [19] Arild Nøkland. Direct feedback alignment provides learning in deep neural networks. In + Advances in neural information processing systems, pages 1037–1045, 2016. + [20]Sergey Bartunov, Adam Santoro, Blake Richards, Luke Marris, Geoffrey E Hinton, and Timothy + Lillicrap. Assessing the scalability of biologically-motivated deep learning algorithms and + architectures. InAdvances in Neural Information Processing Systems, pages 9368–9378, 2018. + [21]Timothy P Lillicrap, Adam Santoro, Luke Marris, Colin J Akerman, and Geoffrey Hinton. + Backpropagation and the brain.Nature Reviews Neuroscience, pages 1–12, 2020. + [22]Natalia Caporale and Yang Dan. Spike timing–dependent plasticity: a hebbian learning rule. + Annu. Rev. Neurosci., 31:25–46, 2008. + [23]Julien Launay, Iacopo Poli, and Florent Krzakala. Principled training of neural networks with + direct feedback alignment.arXiv preprint arXiv:1906.04554, 2019. + [24]Qianli Liao, Joel Z Leibo, and Tomaso Poggio. How important is weight symmetry in back- + propagation? InThirtieth AAAI Conference on Artificial Intelligence, 2016. + [25]Theodore H Moskovitz, Ashok Litwin-Kumar, and LF Abbott. Feedback alignment in deep + convolutional networks.arXiv preprint arXiv:1812.06488, 2018. + [26]Will Xiao, Honglin Chen, Qianli Liao, and Tomaso Poggio. Biologically-plausible learning + algorithms can scale to large datasets. InInternational Conference on Learning Representations, + 2019. URL https://openreview.net/forum?id=SygvZ209F7. + + [27]Mohamed Akrout, Collin Wilson, Peter C Humphreys, Timothy Lillicrap, and Douglas Tweed. + Using weight mirrors to improve feedback alignment.arXiv preprint arXiv:1904.05391, 2019. + + [28]Julien Launay, Iacopo Poli, Kilian Müller, Igor Carron, Laurent Daudet, Florent Krzakala, and + Sylvain Gigan. Light-in-the-loop: using a photonics co-processor for scalable training of neural + networks, 2020. + + [29]Charlotte Frenkel.Bottom-Up and Top-Down Neuromorphic Processor Design: Unveiling + Roads to Embedded Cognition. PhD thesis, UCL-Université Catholique de Louvain, 2020. + + [30]Eric Penner and Li Zhang. Soft 3d reconstruction for view synthesis.ACM Transactions on + Graphics (TOG), 36(6):1–11, 2017. + + [31]John Flynn, Michael Broxton, Paul Debevec, Matthew DuVall, Graham Fyffe, Ryan Overbeck, + Noah Snavely, and Richard Tucker. Deepview: View synthesis with learned gradient descent. + InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages + 2367–2376, 2019. + + [32]Ben Mildenhall, Pratul P Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi + Ramamoorthi, Ren Ng, and Abhishek Kar. Local light field fusion: Practical view synthesis + with prescriptive sampling guidelines.ACM Transactions on Graphics (TOG), 38(4):1–14, + 2019. + + [33]Vincent Sitzmann, Justus Thies, Felix Heide, Matthias Nießner, Gordon Wetzstein, and Michael + Zollhofer. Deepvoxels: Learning persistent 3d feature embeddings. InProceedings of the IEEE + Conference on Computer Vision and Pattern Recognition, pages 2437–2446, 2019. + + [34]Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, and + Yaser Sheikh. Neural volumes: Learning dynamic renderable volumes from images.ACM + Transactions on Graphics (TOG), 38(4):65, 2019. + + [35]Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: + Continuous 3d-structure-aware neural scene representations. InAdvances in Neural Information + Processing Systems, pages 1119–1130, 2019. + + [36]Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, + and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis.arXiv + preprint arXiv:2003.08934, 2020. + + [37]H Brendan McMahan, Gary Holt, David Sculley, Michael Young, Dietmar Ebner, Julian Grady, + Lan Nie, Todd Phillips, Eugene Davydov, Daniel Golovin, et al. Ad click prediction: a view + from the trenches. InProceedings of the 19th ACM SIGKDD international conference on + Knowledge discovery and data mining, pages 1222–1230, 2013. + + [38]Steffen Rendle. Factorization machines. In2010 IEEE International Conference on Data + Mining, pages 995–1000. IEEE, 2010. + + [39]Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, + Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, et al. Wide & deep learning for + recommender systems. InProceedings of the 1st workshop on deep learning for recommender + systems, pages 7–10, 2016. + + [40]Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. Deepfm: a + factorization-machine based neural network for ctr prediction.arXiv preprint arXiv:1703.04247, + 2017. + + [41]Ruoxi Wang, Bin Fu, Gang Fu, and Mingliang Wang. Deep & cross network for ad click + predictions. InProceedings of the ADKDD’17, ADKDD’17, New York, NY, USA, 2017. + Association for Computing Machinery. ISBN 9781450351942. doi: 10.1145/3124749.3124754. + URLhttps://doi.org/10.1145/3124749.3124754. + [42]Weiyu Cheng, Yanyan Shen, and Linpeng Huang. Adaptive factorization network: Learning + adaptive-order feature interactions. InThirty-Fourth AAAI Conference on Artificial Intelligence, + 2020. + [43]J Wesley Hines. A logarithmic neural network architecture for unbounded non-linear function + approximation. InProceedings of International Conference on Neural Networks (ICNN’96), + volume 2, pages 1245–1250. IEEE, 1996. + [44]Criteo. Kaggle contest dataset is now available for academic use!http://labs.criteo.com/ + 2014/09/kaggle-contest-dataset-now-available-academic-use/, 2014. accessed + on the 2020-05-20. + [45]Maurizio Ferrari Dacrema, Paolo Cremonesi, and Dietmar Jannach. Are we really making much + progress? a worrying analysis of recent neural recommendation approaches. InProceedings of + the 13th ACM Conference on Recommender Systems, pages 101–109, 2019. + [46]Michael M Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst. + Geometric deep learning: going beyond euclidean data.IEEE Signal Processing Magazine, 34 + (4):18–42, 2017. + [47]Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann Lecun. Spectral networks and locally + connected networks on graphs. InInternational Conference on Learning Representations, pages + http–openreview, 2014. + [48]Mikael Henaff, Joan Bruna, and Yann LeCun. Deep convolutional networks on graph-structured + data.arXiv preprint arXiv:1506.05163, 2015. + [49]Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks + on graphs with fast localized spectral filtering. InAdvances in neural information processing + systems, pages 3844–3852, 2016. + [50]Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional + networks. InInternational Conference on Learning Representations (ICLR), 2017. + [51]Marco Gori, Gabriele Monfardini, and Franco Scarselli. A new model for learning in graph + domains. InProceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005., + volume 2, pages 729–734. IEEE, 2005. + [52]Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. + The graph neural network model.IEEE Transactions on Neural Networks, 20(1):61–80, 2008. + [53]Yujia Li, Daniel Tarlow, Marc Brockschmidt, and Richard Zemel. Gated graph sequence neural + networks. InInternational Conference on Learning Representations, 2016. + [54]Matthias Fey, Jan Eric Lenssen, Frank Weichert, and Heinrich Müller. Splinecnn: Fast geometric + deep learning with continuous b-spline kernels. InProceedings of the IEEE Conference on + Computer Vision and Pattern Recognition, pages 869–877, 2018. + [55]Petar Velickoviˇ c, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua´ + Bengio. Graph attention networks. InInternational Conference on Learning Representations, + 2018. URLhttps://openreview.net/forum?id=rJXMpikCZ. + [56] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly + learning to align and translate. In3rd International Conference on Learning Representations, + ICLR 2015, 2015. + [57]Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural + networks? InInternational Conference on Machine Learning, 2018. + + [58]Matthias Fey. Just jump: Dynamic neighborhood aggregation in graph neural networks. In + ICLR Workshop on Representation Learning on Graphs and Manifolds, 2019. + + [59]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, + Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. InAdvances in neural information + processing systems, pages 5998–6008, 2017. + + [60]Matthias Fey and Jan E. Lenssen. Fast graph representation learning with PyTorch Geometric. + InICLR Workshop on Representation Learning on Graphs and Manifolds, 2019. + + [61]Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi- + Rad. Collective classification in network data.AI magazine, 29(3):93–93, 2008. + + [62]Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural + networks? InInternational Conference on Learning Representations, 2019. URLhttps: + //openreview.net/forum?id=ryGs6iA5Km. + + [63]Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne.Journal of machine + learning research, 9(Nov):2579–2605, 2008. + + [64]David M Chan, Roshan Rao, Forrest Huang, and John F Canny. Gpu accelerated t-distributed + stochastic neighbor embedding.Journal of Parallel and Distributed Computing, 131:1–13, + 2019. + + [65]Thomas N Kipf and Max Welling. Variational graph auto-encoders.NIPS Workshop on Bayesian + Deep Learning, 2016. + + [66]Alec Radford, Karthik Narasimhan, Time Salimans, and Ilya Sutskever. Improving language + understanding with unsupervised learning.Technical report, OpenAI, 2018. + + [67]Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, + and Dustin Tran. Image transformer.ArXiv, abs/1802.05751, 2018. + + [68]Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya + Sutskever. Jukebox: A generative model for music.arXiv preprint arXiv:2005.00341, 2020. + + [69]Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of + deep bidirectional transformers for language understanding. InProceedings of the 2019 Confer- + ence of the North American Chapter of the Association for Computational Linguistics: Human + Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, + Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1423. + URLhttps://www.aclweb.org/anthology/N19-1423. + + [70]Mohammad Shoeybi, Mostofa Ali Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and + Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model + parallelism.ArXiv, abs/1909.08053, 2019. + + [71]Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, + Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are + few-shot learners.arXiv preprint arXiv:2005.14165, 2020. + + [72]Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ + questions for machine comprehension of text. InProceedings of the 2016 Conference on + Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas, Novem- + ber 2016. Association for Computational Linguistics. doi: 10.18653/v1/D16-1264. URL + https://www.aclweb.org/anthology/D16-1264. + + [73]Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don’t know: Unanswerable + questions for SQuAD. InProceedings of the 56th Annual Meeting of the Association for + Computational Linguistics (Volume 2: Short Papers), pages 784–789, Melbourne, Australia, + July 2018. Association for Computational Linguistics. doi: 10.18653/v1/P18-2124. URL + https://www.aclweb.org/anthology/P18-2124. + + [74]Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix + Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose + language understanding systems. InAdvances in Neural Information Processing Systems, pages + 3261–3275, 2019. + [75]The Common Crawl Team. Common Crawl.https://commoncrawl.org, 2020. + [76]Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classifica- + tion. InACL. Association for Computational Linguistics, 2018. URLhttp://arxiv.org/ + abs/1801.06146. + [77]Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture + models.ArXiv, abs/1609.07843, 2017. + [78]Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare + words with subword units. InProceedings of the 54th Annual Meeting of the Association + for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725, Berlin, Germany, + August 2016. Association for Computational Linguistics. doi: 10.18653/v1/P16-1162. URL + https://www.aclweb.org/anthology/P16-1162. + [79]Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization.International + Conference on Learning Representations, 12 2014. + [80]Grace W Lindsay. Attention in psychology, neuroscience, and machine learning.Frontiers in + Computational Neuroscience, 14:29, 2020. + [81]Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. + Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In + Advances in neural information processing systems, pages 4349–4357, 2016. + [82]Alexandra Luccioni and Yoshua Bengio. On the morality of artificial intelligence.arXiv preprint + arXiv:1912.11945, 2019. + [83]Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for + deep learning in nlp.arXiv preprint arXiv:1906.02243, 2019. + [84]Yi Tay, Dara Bahri, Donald Metzler, Da-Cheng Juan, Zhe Zhao, and Che Zheng. Synthesizer: + Rethinking self-attention in transformer models.arXiv preprint arXiv:2005.00743, 2020. + [85]Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, + and Jiawei Han. On the variance of the adaptive learning rate and beyond.arXiv preprint + arXiv:1908.03265, 2019. + [86]Alessandro Raganato, Yves Scherrer, and Jörg Tiedemann. Fixed encoder self-attention patterns + in transformer-based machine translation.arXiv preprint arXiv:2002.10260, 2020. + [87]Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, + Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas + Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, + Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- + performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- + Buc, E. Fox, and R. Garnett, editors,Advances in Neural Information Processing Systems 32, + pages 8024–8035. Curran Associates, Inc., 2019. URLhttp://papers.neurips.cc/paper/ + 9015-pytorch-an-imperative-style-high-performance-deep-learning-library. + pdf. + + + Appendix + + + We first provide additional elements to corroborate our findings: alignment measurement (Section + A), and shallow baselines (Section B). We then discuss the process of adapting the considered + architectures for DFA (Section C), and the issue of weight transport in attention layers (Section D). + We provide some supplementary results for NeRF (Section E), including details of performance on + each scene of each datatset, and a discussion on possible mitigation of DFA shortcomings. Finally, + we outline steps necessary for reproduction of this work (Section F). + + A Alignment + + Alignment measurement In feedback alignment methods, the forward weights learn toalignwith + the random backward weights, making the delivered updates useful. This alignment can be quantified + by measuring the cosine similarity between the gradient signal delivered by DFABi ay and the + gradient signal BP would have deliveredWT <>. For learning to occur and DFA to work as + a training method, there must be alignment. This can be measured numerically [23]. Measuring + alignments allows to check whether or not the layers are effectively being trained by DFA, regardless + of performance metrics. We note that any alignment value superior to 0 signifies that learning is + occuring. Values closer to 1 indicate a better match with BP, but small alignment values are sufficient + to enable learning. We report values measured at the deepest DFA layer. + + Recommender systems We measure alignment on the Criteo dataset, in the two architectures + featuring non-conventional fully-connected layers: Deep & Cross and AFN. Alignment is measured + after 15 epochs of training, and averaged over a random batch of 512 samples. Results are reported in + table A.1. These alignment measurements indicate that learning is indeed occurring in the cross and + logarithmic layers. High-variance of alignment in the cross layers is unique: it may be explained by + the absence of non-linearity, and account for the difference in performance between BP and DFA on + this architecture–which is higher than on the others. + + Table A.1: Alignment cosine similarity (higher is better, standard deviation in parenthesis) of + recommender systems as measured on the Criteo dataset. Learning occurs in both architectures, and + high variance may explain the larger performance gap on Deep & Cross compared to other methods. + + <
> + + Graph convolutions We measure alignment on the Cora dataset, after 250 epochs of training, + averaging values over every sample available–train, validation, and test split included. Results are + reported in Table A.2. We observe high alignment values in all architectures, indicative that learning + is indeed occuring. Slightly lower values in SplineConv and GATConv may be explained by the use + of the Exponential Linear Unit (ELU) instead of the Rectified Linear Unit (ReLU) used as activation + in other architectures. + Table A.2: Alignment cosine similarity (standard deviation in parenthesis) of various graph convolu- + tions architectures as measured on the Cora dataset. These values corroborate that DFA successfully + trains all architectures considered. + + <
> + + B Shallow baselines + + Shallow learning We compare DFA to BP, but also to shallow learning–where only the topmost + layer is trained. While DFA may not reach the performance level of BP, it should still vastly + + Figure A.1: Comparisons of Tiny-NeRF trained with BP, DFA, and a shallow approach. Shallow + training is insufficient to learn scene geometry. Lego scene from the NeRF synthetic dataset. + + <
> + + outperform shallow learning: failure to do so would mean that the weight updates delivered by DFA + are useless. On a simple task like MNIST, a shallow baseline may be as high as 90%. However, given + the difficulty of the tasks we consider, the shallow baseline is here usually much lower. + + NeRF Because NeRF models are expensive to train–up to 15 hours on a V100–we consider a + simplified setup for the shallow baseline, NeRF-Tiny. This setup operates at half the full resolution + of the training images available, runs for 5000 iterations only, and does away with view-dependant + characteristics. Furthermore, the network is cut down to 3 layers of half the width of NeRF, and + no coarse network is used to inform the sampling. We train this network on the Lego scene of the + NeRF-Synthetic dataset, and compare results. + Figure A.1 presents renders generated by NeRF-Tiny trained with BP, DFA, and a shallow approach. + While BP and DFA delivers similar renders, shallow training fails to reproduce even basic scene + geometry, instead outputting a diffuse cloud of colors. This highlights that while DFA may not reach + a level of performance on-par with BP on NeRF, it nonetheless delivers meaningful updates enabling + the learning of complex features. + + Recommender systems Because recommender systems require fine-tuning, we perform the same + hyperparameter search for shallow learning than for DFA and BP. Results are detailed in Table A.3. + Performance of shallow training is always well under BP and DFA–remember that0.001-levelmatter + in recommender systems. In particular, in Deep & Cross, where there was the biggest gap between + BP and DFA, the performance of the shallow method is extremely poor, well below the FM baseline. + Finally, it is expected to see that DeepFM recovers more or less the performance of FM even with a + shallow baseline. + + Table A.3: Shallow baseline for recommender system models on the Criteo dataset. Performance is + always well below BP and DFA, as expected. + + <
> + + Graph convolutions We use the same hyperparameters as for DFA to produce the shallow baseline + on graph datasets. Results are reported in Table A.4. Performance is always much worse than BP + and DFA. GATConv recovers the best performance: random attention layers may still deliver useful + features [84], as do random convolutions. + + Transformers In the baseline setting (optimizer and hyper-parameters of [59]), a Transformer + trained in the shallow regime yields a perplexity of 428 on WikiText-103. We do not consider + + Table A.4: Shallow baseline for GCNNs on Cora, CiteSeer, and PubMed [61]. Performance is always + well below BP and DFA. + + <
> + + other settings, as the cost of training a Transformer is high and we do not expect any meaningful + improvements–as with NeRF above. + + C Adapting architectures to DFA + + NeRF We use an architecture identical to the one used in [36], but based on the effective code + implementation rather than the description in the paper 1 . During our tests, we have found that + lowering the learning rate to <> rather than <> works best with DFA. + + + Recommender systems For all training methods (BP, DFA, and shallow), we have conducted + independent hyperparameter searches. We performed a grid search over the learning rate, from + <> to <> in <> steps, as well as over the dropout probability, from <> to in <> steps + (where applicable). On DeepFM, this search leads to reduce the learning rate from <> with BP + to <> with DFA, but to keep the 0.5 dropout rate. On Deep & Cross, we reduce learning rate + from <> to <>, with no dropout in both cases. In AFN, we reduce dropout from <> to + <> and dropout from 0.3 to 0. + + Graph convolutions We manually test for a few hyperparameters configuration on the Cora dataset, + focusing on learning rate, weight decay, and dropout. We do not consider architectural changes, such + as changing the number of filters or of attention heads. For ChebConv and GraphConv, we reduce + weight decay to <> instead of <>, and set the dropout rate to 0 and 0.1 respectively, instead + of 0.5 with BP. For SplineConv, we find that no change in the hyperparameters are necessary. For + GATConv, we reduce weight decay to <> instead of <> and reduce dedicated dropout layer + to 0.1 instead of 0.6 but keep the 0.6 dropout rate within the GAT layer. Finally, on DNAConv we + disable weight decay entirely, instead of an original value of <>, double the learning rate from + <> to <>, and disable dropout entirely. In all cases, we share the backward random matrix + across all nodes in a graph. + + Transformers The model hyper-parameters were fixed across all of our experiments, except for + the number of attention heads in one case, that we will precise below, and dropout. We tested several + values of dropout probability between 0 and 0.5, but found the original value of 0.1 to perform + best. We manually tested a number of optimizers, optimizer parameters and attention mechanisms. + We tested four combinations of optimizers and schedulers : Adam with the scheduler used in [59], + Adam alone, RAdam [85] alone, and Adam with a scheduler that reduces the learning rate when + the validation perplexity plateaus. We found it necessary to reduce the initial learning rate of Adam + from <> to <>, although it could be set back to <> with a scheduler. We tried two values + of 0.98 and 0.999. We also tried to change <> and observed some small differences that were + not significant enough for the main text. Finally, we tried three attention mechanisms in addition to + the standard multihead scaled dot-product attention: the dense and random (learnable) Synthesizers + of [84], as well as the fixed attention patterns of [86]. The latter needed to be adapted to language + modelling to prevent attending to future tokens, which led us to reduced the number of attention + heads to 4. The backward random matrix is always shared across all tokens and batches. + + D Weight transport and attention + + We consider an attention layer operating on inputx. The queries, keys, and values are respectively + <>, and <> is the dimension of the queries and keys. The layer + performs: + <> (4) + + When using DFA on attention, we deliver the random feedback to the top of the layer. Accordingly, + to obtain updates toWQ ;WK ;andWV we still to have to backpropagate through the attention + mechanism itself. This involves weight transport onWV , sacrificing some biological realism for + simplicity. Overall weight transport between layers still does not occur, and updating the layers in + parallel remains possible. + Beside using FA or DFA within the attention layer, alternative mechanisms like the synthesizer + [84]–which uses random attention in place of the query and key system–or fixed attention [86] can + remove the need for weight transport. Implementing these mechanisms in DFA-trained Transformers, + or other attention-powered architectures, will require further research. + + + E Supplementary NeRF results + + Quantitative results We report per-scene scores for each dataset in Table A.5. BP values are taken + from [36]. On three scenes of the synthetic datasets, NeRF-DFA even outperforms past state-of-the-art + methods trained with BP. Note that Neural Volumes (NV) is not applicable to forward-facing view + synthesis–as is required in LLFF-Real–and thus no results are reported. + + Qualitative results We report sample renders from the NeRF-Synthetic dataset (Figure A.2) and + the LLFF-Real dataset (Figure A.2), for every scene available. However, we recommend readers to + consult the supplementary video to make better sense of characteristics like multi-view consistency + and view-dependent effects (most visible on the LLFF-Real Room scene). + + + Table A.5: Per-scene PSNR for NeRF DFA and BP against other state-of-the-art methods on the + Nerf-Synthetic and LLFF-Real. DFA performance is fairly homogeneous across each dataset and in + line with the differences in other methods. + + <
> + + Possible future directions Despite retranscribing scene geometry in a multi-view consistent way, + NeRF produces renders of a lower quality when trained with DFA instead of BP. In particular, it + struggles to transcribe small-scale details, resulting in "blurry" renders. Moreover, it displays high- + frequency artefacts: not in the scene geometry, but in individual pixels taking values very distant from + their neighborhood. Interestingly, this noise phenomenon is unique to NeRF-DFA: it is not observed + on NeRF-BP with similar PSNR values (achieved during training) or on other methods with similar + or lower PSNR. This leads us to hypothesize this is an aspect unique to DFA, possibly due to the + alignment process. Indeed, DFA creates a bias on the weights, by encouraging them to be "aligned" + with an arbitrary values dependant on the random matrix used. It is possible this could introduce + random noise in the final renders–though we leave a more principled experiment to future research. + To attempt to alleviate this issue, we first consider NeRF-Dual. In NeRF-Dual, we average the + pixel-wise prediction between the fine and coarse network, to attempt to remove some of the noise. + To do so, we first still use the coarse network to create a probability distribution for the hierarchical + sampling. Then, we evaluate again both the coarse and fine networks at the locations informed by + this probability distribution. Compared to vanilla NeRF, this requires an extra batch of evaluation of + the coarse network for all rays–rougly speaking, this increases inference time by 30-50% depending + on the coarse network architecture considered. We note that this is not applied during training, so that + training times remain identical. + Figure A.2 and Figure A.3 showcase comparisons between NeRF and NeRF-Dual trained with DFA + on all scenes. When viewed at high resolution–such as in our supplementary video–the NeRF-Dual + renders are more pleasing, especially for the full scenes. They remove most of the high-frequency + noise, leading to smoother renders. However, this averaging process further blurs small-scale details in + the render. This is especially visible in the NeRF-Synthetic dataset, on scenes like Ficus. Furthermore, + NeRF-Dual introduces novel artefacts in the Mic and Ship scenes, with areas improperly colored + with a violet tint. The cause for these artefacts is unknown, but they show that NeRF-Dual is far from + a silver bullet. The PSNR is also minimally increased, by less than 0.5 per scene. Nevertheless, this + shows some promise in possibilities to allievate the shortcomings of NeRF-DFA. It is possible that + changes to the overall rendering process, or the use of classic image processing techniques, may help + enhance the NeRF-DFA images. + Finally, we also experimented with increasing the capacity of the fine network, by widening its layers + to 512 neurons. We call this architecture NeRF-XL. However, we have not succeeded in getting + PSNR values higher than with vanilla NeRF on DFA. In particular, the training process becomes + much more cumbersome, as multi-GPU parallelism is needed to fit the model. It is possible that + higher network capacity may help learning both the task at hand and to align simultaneously, but + further work is required. + + + F Reproducibility + + Hardware used All main experiments require at most a single NVIDIA V100 GPU with 16GB + of memory to reproduce. Alignment measurement on large architectures (NeRF and Transformers) + require a second identical GPU to keep a copy of the network to evaluate BP gradients. + We estimate that a total of around 10,000 GPU-hours on V100s were necessary for this paper. + Accordingly, we estimate the cloud-computing carbon impact of this paper to be of 1700 kgCO 2 eq 2 . + However, without hyperparameter searches, our results can be reproduced with less than 500 GPU- + hours on V100s, with most of that budget going to NeRF and Transformers. + + Implementation We use the shared random matrix trick from [23] to reduce memory use in DFA + and enable its scaling to large networks. We use PyTorch [87] for all experiments. For reference + implementation of the methods considered, we relied on various sources. Our NeRF implementation + is based on the PyTorch implementation by Krishna Murthy 3 , with modifications to allow for proper + test and validation, as well as DFA and multi-GPU support. For recommender systems, we use + PyTorch Geometric [60] for all graph operations. Our Transformer implementation is our own. + Our code is available as supplementary material. + + NeRF We provide training, testing, and rendering code along with the configurations used to obtain + our results. An example to reproduce our results is given in the supplementary code repository. Given + the computing cost associated with training a NeRF, we also provide our trained models. + + Recommender systems We provide bash scripts to reproduce the results in Table 2 and A.3, with + the results of our hyperparameter search. We provide code to reproduce the results in Table A.1. + + Graph convolutions We provide the code to reproduce all of our results. Note that the t-SNE + results are not exactly reproducible, as the CUDA implementation used is non-deterministic. + + Transformers We provide bash scripts to reproduce Table 5 and the shallow results. + + <
> + + Figure A.2: Sample renders for every scene of the NeRF-Synthetic dataset, for NeRF and NeRF-Dual + trained with DFA. + + <
> + + Figure A.3: Sample renders for every scene of the LLFF-Real dataset, for NeRF and NeRF-Dual + trained with DFA. +<> <> <> + + +<> <> <> +Efficient Behavior of Small-World Networks + +We introduce the concept of efficiency of a network, measuring how efficiently it exchanges information. By using this simple measure small-world networks are seen as systems that are both globally and locally efficient. This allows to give a clear physical meaning to the concept of small-world, and also to perform a precise quantitative analysis of both weighted and unweighted networks. We study neural networks and man-made communication and transportation systems and we show that the underlying general principle of their construction is in fact a small-world principle of high efficiency. PACS numbers 89.70.+c, 05.90.+m, 87.18.Sn, 89.40.+k +We live in a world of networks. In fact any complex system in nature can be modeled as a network, where vertices are the elements of the system and edges represent the interactions between them. Coupled biological and chemical systems, neural networks, social interacting species, computer networks or the Internet are only few of such examples [1]. Characterizing the structural properties of the networks is then of fundamental importance to understand the complex dynamics of these systems. A recent paper [2] has shown that the connection topology of some biological and social networks is neither completely regular nor completely random. These networks, there named small-worlds, in analogy with the concept of small-world phenomenon developed 30 years ago in social psychology [3], are in fact highly clustered like regular lattices, yet having small characteristics path lengths like random graphs. The original paper has triggered a large interest in the study of the properties of small-worlds (see ref. [4] for a recent review). Researchers have focused their attention on different aspects: study of the inset mechanism [5,7], dynamics [8] and spreading of diseases on small-worlds [9], applications to social net.works [10,11] and to the Internet [12,13]. In this letter we introduce the concept of efficiency of a network, measuring how efficiently information is exchanged over the net.work. By using efficiency small-world networks results as systems that are both globally and locally efficient. This formalization gives a clear physical meaning to the concept of small-world, and also allows a precise quantitative analysis of unweighted and weighted networks. We study several systems, like brains, communication and transportation networks, and show that the underlying general principle of their construction is in fact a small-world principle, provided attention is taken not to ignore an important observational property (closure). We start by reexamining the original formulation pro.posed in ref. [2]. There, a generic graph G with N vertices and K edges is considered. G is assumed to be unweighted, i.e. edges are all equal, sparse <<(K . N (N . 1)/2)>>, and connected. i.e. there exists at least one path connecting any couple of vertices with a infinite number of steps. G is therefore represented by simply giving the adjacency (or connection) matrix, i.e. the NxN matrix whose entry a_ij is 1 if there is an edge joining vertex i to vertex j, and 0 otherwise. An important quantity of G is the degree of vertex i, i.e. the number ki of edges incident with vertex i (the number of neighbors of i). The average value of ki is <>. Once {<>} is given it can be used to calculate the matrix of the short.est path lengths d_ij between two generic vertices i and j. The fact that G is assumed to be connected implies that dij is positive and infinite .i = j. In order to quantify the structural properties of G, [2] proposes to evaluate two different quantities: the characteristic path length L and the clustering coefficient C. L is the average distance be- +tween two generic vertices <>, and C is a local property defined as <>. Here C_i is the number of edges existing in Gi, the subgraph of the neighbors of i, divided by the maximum possible number ki(ki . 1)/2. In [2] a simple method is considered to pro.duce a class of graphs with increasing randomness. The initial graph G is taken to be a one-dimensional lattice with each vertex connected to its k neighbors and with periodic boundary conditions. Rewiring each edge at ran.dom with probability p, G can be tuned in a continuous way from a regular lattice (p = 0) into a random graph (p = 1). For the regular lattice we expect <> and a high clustering coefficient <>, while for a random graph <> and <> [14,5]. Although in the two limit cases a large C is associated to a large L and vice versa a small C to a small L, the numerical experiment reveals an intermediate regime at small p where the system is highly clustered like regular lattices, yet having small characteristics path lengths like random graphs. This behavior is there called small-world and it is found to be a property of some social and +biological networks analyzed [2]. +Now we propose a more general set-up to investigate real networks. We will show that: the definition of small-world behavior can be given in terms of a single variable with a physical meaning, the efficiency E of the network. -1/L and C can be seen as first approximations of E evaluated resp. on a global and on a local scale. -we can drop all the restrictions on the system, like unweightedness, connectedness and sparseness. We represent a real network as a generic weighted (and possibly even non sparse and non connected) graph G. Such a graph needs two matrices to be described: the adjacency matrix {a_ij} defined as for the unweighted graph, and the matrix {<>} of physical distances. The number <> can be the space distance between the two vertices or the strength of their possible interaction: we suppose <> can be the geographical distance between stations in transportation systems (in such a case <> respects the triangle equality, though this is not a necessary assumption), the time taken to ex.change a packet of information between routers in the Internet, or the inverse v<>ity of chemical reactions along a direct connection in a biological system. Of course, in the particular case of an unweighted graph <>. The shortest path length dij between two generic points i and j is the smallest sum of the physical distances throughout all the possible paths in the graph from i to j. The matrix {<>} is therefore calculated by using the information contained both in matrix {a_ij} and in matrix {<>}. We have <>, the equality being valid when there is an edge between i and j. Let us now suppose that the system is parallel, i.e. every vertex sends information concurrently along the network, through its edges. The efficiency <> in the communication between vertex i and j can be then defined to be inversely proportional to the shortest distance: <>. When there is no path in the graph between i and j, <> and consistently <>. The average efficiency of G can be defined as: + +<> + +To normalize E we consider the ideal case G_id in which the graph G has all the <> possible edges. In such a case the information is propagated in the most efficient way since dij = .ij .i, j, and E assumes its maxi- +<>. The efficiency <> +<> considered in the following of the paper is always divided by <> and therefore <>. Though the equality E = 1 is valid when there is an edge between each couple of vertices, real networks can reach a high value of E. +In our formalism, we can define the small-world be.haviour by using the single measure E to analyze both the local and global behavior, rather than two different variables L and C. The quantity in eq. (1) is the global efficiency of G and we therefore name it E_glob. Since E is defined also for a disconnected graph we can characterize the local properties of G by evaluating for each vertex i the efficiency of G_i, the subgraph of the neighbors of i. We define the local efficiency as the average efficiency of the local subgraphs, E loc =1/N E(Gi). + +This quantity plays a role similar to the clustering co.efficient C. Since <>, the local efficiency <> tells how much the system is fault tolerant, thus how efficient is the communication between the first neighbors of i when i is removed [15]. The definition of small-world can now be rephrased and generalized in terms of the information <>: small-world networks have high <> and <>, i.e. are very efficient in global and local communication. This definition is valid both for unweighted and weighted graphs, and can also be applied to disconnected and/or non sparse graphs. +It is interesting to see the correspondence between our measure and the quantities L and C of [2] (or, correspondingly, <<1/L>> and C). The fundamental difference is that E_glob is the efficiency of a parallel systems, where all the nodes in the network concurrently exchange pack.ets of information (such are all the systems in [2], for example), while 1/L measures the efficiency of a sequential system (i.e. only one packet of information goes along the network). <> is a reasonable approximation of <>when there are not huge differences among the distances in the graph, and this can explain why L works reasonably well in the unweighted examples of [2]. But, in general 1/L can significantly depart from E_glob. For instance, in the Internet, having few computers with an extremely slow connection does not mean that the whole Internet diminishes by far its efficiency: in practice, the presence of such very slow computers goes unnoticed, be.cause the other thousands of computers are exchanging packets among them in a very efficient way. Here 1/L would give a number very close to zero (strictly 0 in the particular case when a computer is disconnected from the others and <>, while E_glob gives the correct efficiency measure of the Internet. We turn now our attention to the local properties of a network. C is only one among the many possible intuitive measures [10] of how well connected a cluster is. It can be shown that when in a graph most of its local subgraphs Gi are not sparse, then C is a good approximation of E_loc. Summing up there are not two different kinds of analysis to be done for the global and local scales, but just one with a very precise physical meaning: the efficiency in transporting information. We now illustrate the onset of the small-world in an un.weighted graph by means of the same example used in [2]. A regular lattice with <> and <> is rewired +2 +with probability p and <> and <> are reported in <> as functions of p [16]. For <

> we expect the system to be inefficient on a global scale (E_glob . k/N log(N/K)) but locally efficient. The situation is inverted for the ran.dom graph. In fact at p =1 E_glob assumes a maximum value of 0.4, meaning 40% the efficiency of the ideal graph with an edge between each couple of vertices. This at the expenses of the fault tolerance (<>). + +<

> + +FIG. 1. FIG.1 Global and local efficiency for the graph example considered in [2]. A regular lattice with <> and <> is rewired with probability p. The small-world behavior results from the increase of E_glob caused by the introduction of only a few rewired edges (short cuts), which on the other side do not affect <>. At p � 0.1, E_glob has almost reached the value of the random graph, though <> has only diminished by very little from the value of 0.82 of the regular lattice. Small worlds have high E_glob and <>. +The small-world behavior appears for intermediate values of p. It results from the fast increase of E_glob (for small p we find a linear increase of E_glob in the logarithmic horizontal scale) caused by the introduction of only a few rewired edges (short cuts), which on the other side do not affect <>. At p . 0.1, E_glob has almost reached the maximum value of 0.4, though <> has only diminished by very little from the maximum value of 0.82. For an unweighted case the description in terms of network efficiency resembles the approximation given in [2]. In particular we have checked that a good agreement with curves L(p) and C(p) [2] can be obtained by reporting <> and <>. Of course in such an example the short cuts connect at almost no cost vertices that would otherwise be much farther apart (because <>). On the other hand this is not true when we consider a weighted network. As real networks we consider first different examples of natural systems (neural networks), and then we turn our attention to man-made communication and transportation systems. +1) Neural Networks. Thanks to recent experiments +neural structures can be studied at several levels of scale. Here we focus first on the analysis of the neuro-anatomical structure of cerebral cortex, and then on a simple nervous system at the level of wiring between neurons. The anatomical connections between cortical areas are of particular importance for their intricate relationship with the functional connectivity of the cerebral cortex [18]. We analyze two databases of cortico-cortical connections in the macaque and in the cat [19]. Tab.1 indicates the two networks are small-worlds [16]: they have high E_glob, respectively 52% and 69% the efficiency of the ideal graph with an edge between each couple of vertices (just slightly smaller than the best possible values of 57% and 70% obtained in random graphs) and high <>, respectively 70% and 83%, i.e. high fault tolerance [22]. These results indicate that in neural cortex each region is intermingled with the others and has grown following a perfect balance between local necessities (fault tolerance) and wide-scope interactions. Next we consider the neural network of C. elegans, the only case of a nervous system completely mapped at the level of neurons and chemical synapses [23]. Tab.1 shows that this is also a small-world network: C. elegans achieves both a 50% of global and local efficiency. Moreover the value of E_glob is similar to <>. This is a difference from cortex databases where fault tolerance is slightly privileged with respect to global communication. +2) Communication Networks. We have considered two of the most important large-scale communication net.works present nowadays: the World Wide Web and the Internet. Tab.2 shows that they have relatively high val.ues of E_glob (slightly smaller than the best possible val.ues obtained for random graphs) and <>. Despite the WWW is a virtual network and the Internet is a physical network, at a global scale they transport information essentially in the same way (as their E_glob�s are almost equal). At a local scale, the bigger <> in the WWW case can be explained both by the tendency in the WWW to create Web communities (where pages talking about the same subject tend to link to each other), and by the fact that many pages within the same site are often quickly connected to each other by some root or menu page. +3) Transport Networks. differently from previous databases the Boston subway transportation system (MBTA) can be better described by a weighted graph, the matrix {.ij } being given by the geographical distances between stations. If we consider the MBTA as an unweighted graph we obtain that it is apparently neither locally nor globally efficient (see Tab.3). On the other hand, when we take into account the geographical distances, we have E_glob =0.63: this shows the MBTA is a very efficient transportation system on a global scale, only 37% less efficient than the ideal subway with a di.rect tunnel from each station to the others. Even in the weighted case <> stays low (0.03), indicating a poor local behavior: differently from a neural network the + + +MBTA is not fault tolerant and a damage in a station will dramatically affect the connection between the previous and the next station. The difference with respect to neural networks comes from different needs and priorities in the construction and evolution mechanism: when we build a subway system, the priority is given to the achievement of global efficiency, and not to fault tolerance. In fact a temporary problem in a station can be solved by other means: for example, walking, or taking a bus from the previous to the next station. That is to say, the MBTA is not a closed system: it can be considered, after all, as a subgraph of a wider transportation network. This property is of fundamental importance when we analyze a system: while global efficiency is without doubt the major characteristic, it is closure that somehow leads a system to have high local efficiency (without alternatives, there should be high fault-tolerance). The MBTA is not a closed system, and thus this explains why, unlike in the case of the brain, fault tolerance is not a critical issue. Indeed, if we increase the precision of the analysis and change the MBTA subway network by taking into account, for example, the Boston Bus System, this ex.tended transportation system comes back to be a small-world network (<>). Qualitatively similar results, confirming the similarity of construction principles, have been obtained for other undergrounds and for a wider transportation system consisting of all the main airplane and highway connections throughout the world [25]. Considering all the transportation alter.natives available at that scale makes again the system closed (there are no other reasonable routing alternatives), and so fault-tolerance comes back as a leading construction principle. +Summing up, the introduction of the efficiency mea.sure allows to give a definition of small-world with a clear physical meaning, and provides important hints on why the original formulas of [2] work reasonably well in some cases, and where they fail. The efficiency measure al.lows a precise quantitative analysis of the information flow, and works both in the unweighted abstraction, and in the more realistic assumption of weighted networks. Finally, analysis of real data indicates that various existing (neural, communication and transport) networks exhibit the small-world behavior (even, in some cases, when their unweighted abstractions do not), substantiating the idea that the diffusion of small-world networks can be interpreted as the need to create networks that are both globally and locally efficient. +[1] Y. Bar-Yam, Dynamics of Complex Systems (Addison-Wesley, Reading Mass, 1997). +[2] D.J. Watts and S.H. Strogatz, Nature 393, 440 (1998). +[3] S. Milgram, Physicol. Today, 2, 60 (1967). +[4] M.E.J. Newman, cond-mat/0001118. +[5] A. Barrat, M. Weigt, Europ. Phys. J. B 13, 547 (2000) +[6] M. Marchiori and V. Latora, Physica A285, 539 (2000). +[7] M. Barthelemy, L. Amaral, Phys. Rev. Lett. 82, 3180 (1999). +[8] L. F. Lago-Fernandez et al, Phys. Rev. Lett. 84, 2758 (2000). +[9] C. Moore and M.E.J. Newman, Phys. Rev. E61, 5678 (2000). +[10] M.E.J. Newman, cond-mat/0011144. +[11] L. A. N. Amaral, A. Scala, M. Barth�el�emy, and H. E. Stanley, Proc. Natl. Acad. Sci. 97, 11149 (2000). +[12] R. Albert, H. Jeong, and A.-L. Barab�asi, Nature 401, 130 (1999). +[13] A.-L. Barab�asi and R. Albert, Science 286, 509 (1999). +[14] B. Bollob�as, Random Graphs (Academic, London, 1985). +[15] Our concept of fault tolerance is different from the one adopted in R. Albert, H. Jeong, and A.-L. Barab�asi, Na.ture 406, 378 (2000); R. Cohen et al. Phys. Rev. Lett. 85, 2758 (2000), where the authors consider the response of the entire network to the removal of i. +[16] Here and in the following the matrix {dij }i,j2G has been computed by using two different methods: the Floyd-Warshall (O(N 3 )) [17] and the Dijkstra algorithm (O(N 2 logN )) [10]. +[17] G. Gallo and S. Pallottino, Ann. Oper. Res. 13, 3 (1988). +[18] O. Sporns, G. Tononi, G.M. Edelman, Celebral Cortex 10, 127 (2000). [19] J.W.Scannell, Nature 386, 452 (1997). [20] M.P. Young, Phil.Trans.R.Soc B252, 13 (1993). +[21] J.W. Scannell, M.P. Young and C. Blakemore, J. Neu.rosci. 15, 1463 (1995). +[22] E. Sivan, H. Parnas and D. Dolev, Biol. Cybern. 81, 11.23 (1999). +[23] J.G. White et. al., Phil. Trans. R. Soc. London B314,1 (1986). +[24] T.B. Achacoso and W.S. Yamamoto, AY�s Neuroanatomy of C. elegans for Computation (CRC Press, FL, 1992). +[25] M. Marchiori and V. Latora, in preparation. + +TABLE I. Macaque and cat cortico-cortical connections [19]. The macaque database contains N = 69 cortical areas and K = 413 connections [20]. The cat database has N = 55 cortical areas (including hippocampus, amygdala, entorhinal cortex and subiculum) and K = 564 (revised database and cortical parcellation from [21]). The nervous system of C. elegans consists of N = 282 neurons and K = 2462 links which can be either synaptic connections or gap junctions [24]. + +<
> + +TABLE II. Communication networks. Data on the World Wide Web from http://www.nd.edu/�networks contains N = 325729 documents and K = 1090108 links [12], while the Internet database is taken from http://moat.nlanr.net and has N = 6474 nodes and K = 12572 links. + +<
> + +TABLE III. The Boston underground transportation system (MBTA) consists of N = 124 stations and K = 124 tunnels. The matrix {.ij } of the spatial distances between stations, used for the weighted case, has been calculated us.ing databases from http://www.mbta.com/ and the U.S. Na.tional Mapping Division. + +<
> + <> <> <> \ No newline at end of file diff --git a/Corpus/DEEP DOUBLE DESCENT - Preetum Nakkiran.txt b/Corpus/DEEP DOUBLE DESCENT - Preetum Nakkiran.txt deleted file mode 100644 index 282e671..0000000 Binary files a/Corpus/DEEP DOUBLE DESCENT - Preetum Nakkiran.txt and /dev/null differ diff --git a/Corpus/Deep Residual Learning for Image Recognition.txt b/Corpus/Deep Residual Learning for Image Recognition.txt deleted file mode 100644 index 6cb144d..0000000 Binary files a/Corpus/Deep Residual Learning for Image Recognition.txt and /dev/null differ diff --git a/Corpus/Direct Feedback Alignment Scales toModern Deep Learning Tasks and Architectures.txt b/Corpus/Direct Feedback Alignment Scales toModern Deep Learning Tasks and Architectures.txt deleted file mode 100644 index 8b3ad5c..0000000 --- a/Corpus/Direct Feedback Alignment Scales toModern Deep Learning Tasks and Architectures.txt +++ /dev/null @@ -1,1161 +0,0 @@ - Direct Feedback Alignment Scales to - Modern Deep Learning Tasks and Architectures - - - - - Julien Launay 1;2 Iacopo Poli 1 François Boniface 1 Florent Krzakala 1;2 - - 1 LightOn 2 École Normale Supérieure - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - arXiv:2006.12878v1 [stat.ML] 23 Jun 2020 {julien, iacopo, francois, florent}@lighton.ai - - - - Abstract - - Despite being the workhorse of deep learning, the backpropagation algorithm is - no panacea. It enforces sequential layer updates, thus preventing efficient paral- - lelization of the training process. Furthermore, its biological plausibility is being - challenged. Alternative schemes have been devised; yet, under the constraint of - synaptic asymmetry, none have scaled to modern deep learning tasks and architec- - tures. Here, we challenge this perspective, and study the applicability of Direct - Feedback Alignment to neural view synthesis, recommender systems, geometric - learning, and natural language processing. In contrast with previous studies lim- - ited to computer vision tasks, our findings show that it successfully trains a large - range of state-of-the-art deep learning architectures, with performance close to - fine-tuned backpropagation. At variance with common beliefs, our work supports - that challenging tasks can be tackled in the absence of weight transport. - - - 1 Introduction - - While the backpropagation algorithm (BP) [1,2] is at the heart of modern deep learning achievements, - it is not without pitfalls. For one, its weight updates are non-local and rely on upstream layers. Thus, - they cannot be easily parallelized [3], incurring important memory and compute costs. Moreover, - its biological implementation is problematic [4,5]. For instance, BP relies on the transpose of the - weights to evaluate updates. Hence, synaptic symmetry is required between the forward and backward - path: this is implausible in biological brains, and known as the weight transport problem [6]. - Consequently, alternative training algorithms have been developed. Some of these algorithms are - explicitly biologically inspired [7–13], while others focus on making better use of available compute - resources [3,14–19]. Despite these enticing characteristics, none has been widely adopted, as they - are often demonstrated on a limited set of tasks. Moreover, as assessed in [20], their performance on - challenging datasets under the constraint of synaptic asymmetry is disappointing. - We seek to broaden this perspective, and demonstrate the applicability of Direct Feedback Alignment - (DFA) [19] in state-of-the-art settings: from applications of fully connected networks such as neural - view synthesis and recommender systems, to geometric learning with graph convolutions, and natural - language processing with Transformers. Our results define new standards for learning without weight - transport and show that challenging tasks can indeed be tackled under synaptic asymmetry. - All code needed to reproduce our experiments is available athttps://github.com/lightonai/ - dfa-scales-to-modern-deep-learning. - - - - - - - Preprint. Under review. 1.1 Related work - - Training a neural network is a credit assignment problem: an update is derived for each parameter - from its contribution to a cost function. To solve this problem, a spectrum of algorithms exists [21]. - - Biologically motivated methods Finding a training method applicable under the constraints of - biological brains remains an open problem. End-to-end propagation of gradients is unlikely to occur - [22], implying local learning is required. Furthermore, the weight transport problem enforces synaptic - asymmetry [6]. Inspired by auto-encoders, target propagation methods (TP) [10–12] train distinct - feedback connections to invert the feedforward ones. Feedback alignment (FA) [13] replaces the - transpose of the forward weights used in the backward pass by a random matrix. Throughout training, - the forward weights learn toalignwith the arbitrary backward weights, eventually approximating BP. - - Beyond biological considerations As deep learning models grow bigger, large-scale distributed - training is increasingly desirable. Greedy layer-wise training [14] allows networks to be built layer - by layer, limiting the depth of backpropagation. To enable parallelization of the backward pass, - updates must only depend on local quantities. Unsupervised learning is naturally suited for this, - as it relies on local losses such as Deep InfoMax [17] and Greedy InfoMax [18]. More broadly, - synthetic gradient methods, like decoupled neural interfaces [3,15] and local error signals (LES) - [16], approximate gradients using layer-wise trainable feedback networks. DFA [19] expands on FA - and directly projects a global error to each layer. A shared feedback path is still needed, but it only - depends on a simple random projection operation. - - Performance of alternative methods Local training methods are successful in unsupervised learn- - ing [18]. Even in a supervised setting, they scale to challenging datasets like CIFAR-100 or ImageNet - [14,16]. Thus, locality is not too penalizing. However, TP, FA, and DFA are unable to scale to these - tasks [20]. In fact, DFA is unable to train convolutional layers [23]. To enable feedback alignment - techniques to perform well on challenging datasets, some form of weight transport is necessary: - either by explicitly sharing sign information [24–26], or by introducing dedicated phases of alignment - for the forward and backward weights where some information is shared [27]. To the best of our - knowledge, no method compatible with the weight transport problem has ever been demonstrated on - challenging tasks. - - 1.2 Motivations and contributions - - We focus on DFA, a compromise between biological and computational considerations. Notably, - DFA is compatible with synaptic asymmetry: this asymmetry raises important challenges, seemingly - preventing learning in demanding settings. Moreover, it allows for asynchronous weight updates, - and puts a single operation at the center of the training stage. This enables new classes of training - co-processors [28, 29], leveraging dedicated hardware to perform the random projection. - - Extensive survey We apply DFA in a large variety of settings matching current trends in machine - learning. Previous works have found that DFA is unsuitable for computer vision tasks [20,23]; but - computer vision alone cannot be the litmus test of a training method. Instead, we consider four vastly - different domains, across eight tasks, and with eleven different architectures. This constitutes a survey - of unprecedented scale for an alternative training method, and makes a strong case for the possibility - of learning without weight transport in demanding scenarios. - - Challenging settings We demonstrate the ability of DFA to tackle challenging tasks. We success- - fully learn and render real-world 3D scenes (section 3.1.1); we perform recommendation at scale - (section 3.1.2); we explore graph-based citation networks (section 3.2); and we consider language - modelling with a Transformer (section 3.3). We study tasks at the state-of-the-art level, that have - only been recently successfully tackled with deep learning. - - Modern architectures We prove that the previously established failure of DFA to train convolutions - does not generalize. By evaluating performance metrics, comparing against a shallow baseline, - measuring alignment, and visualizing t-SNE embeddings, we show that learning indeed occurs in - layers involving graph convolutions and attention. This significantly broadens the applicability of - DFA–previously thought to be limited to simple problems like MNIST and CIFAR-10. - - 2 2 Methods - - Forward pass In a fully connected network, at layeriout ofN, neglecting its biases, withWi its - weight matrix,fi its non-linearity, andhi its activations, the forward pass is: - 8i2[i;:::;N] :ai =Wi hi1 ;hi =fi (ai ): (1) - h0 =Xis the input data, andhN =f(aN ) =^yare the predictions. A task-specific cost function - L(^y;y)is computed to quantify the quality of the predictions with respect to the targetsy. - - Backward pass with BP The weight updates are computed by backpropagation of the error vector. - Using the chain-rule of derivatives, each neuron is updated based on its contribution to the cost - function. Leaving aside the specifics of the optimizer used, the equation for the weight updates is: - @L @LW Ti = =[(W a (2)@W i+1 i+1 )f0 (ai i )]hT ;ai1 i = - i @ai - - Backward pass with DFA The gradient signalWT ai+1 i+1 of the (i+1)-th layer violates synaptic - asymmetry. DFA replaces it with a random projection of the topmost derivative of the loss,ay . - For common classification and regression losses such as the mean squared error or the negative log - likelihood, this corresponds to a random projection of the global errore=^yy. WithBi , a fixed - random matrix of appropriate shape drawn at initialization for each layers: - @LWi =[(Bi ay )f0 (a ai i )]hT ; i1 y = (3)@ay - - 3 Experiments - - We study the applicability of DFA to a diverse set of applications requiring state-of-the-art architec- - tures. We start with fully connected networks, where DFA has already been demonstrated, and address - new challenging settings. We then investigate geometric learning: we apply DFA to graph neural net- - works in classification tasks on citation networks, as well as graph autoencoders. These architectures - feature graph convolutions and attention layers. Finally, we use DFA to train a transformer-based - Natural Language Processing (NLP) model on a dataset of more than 100 million tokens. - - 3.1 Fully connected architectures - - DFA has been successful at training fully connected architectures, with performance on-par with - backpropagation [19,20]. However, only computer vision tasks have been considered, where fully - connected networks considerably underperform their convolutional counterpart. Here, we focus on - tasks where fully connected architectures are state-of-the-art. Moreover, the architectures considered - are deeper and more complex than those necessary to solve a simple task like MNIST. - - 3.1.1 Neural view synthesis with Neural Radiance Fields - The most recent state-of-the-artneural view synthesismethods are based on large fully connected - networks: this is an ideal setting for a first evaluation of DFA on a challenging task. - - Background There has been growing interest in methods capable of synthesising novel renders of - a 3D scene using a dataset of past renders. The network is trained to learn an inner representation of - the scene, and a classical rendering system can then query the model to generate novel views. With - robust enough methods, real-world scenes can also be learned from a set of pictures. - Until recently, most successful neural view synthesis methods were based on sampled volumetric - representations [30–32]. In this context, Convolutional Neural Networks (CNNs) can be used to - smooth out the discrete sampling of 3D space [33,34]. However, these methods scale poorly to - higher resolutions, as they still require finer and finer sampling. Conversely, alternative schemes - based on a continuous volume representation have succeeded in generating high-quality renders [35], - even featuring complex phenomenons such as view-dependant scattering [36]. These schemes make - point-wise predictions, and use fully connected neural networks to encode the scene. - - 3 Figure 1: Comparisons of NeRF-DFA with state-of-the-art methods trained with BP on the most - challenging synthetic and real-world scenes. While NeRF-DFA generates render of lower quality, - they maintain multi-view consistency and exhibit no geometric artefacts. BP results from [36]. - - - Setting We employ Neural Radiance Fields (NeRF) [36], the state-of-the-art for neural view - synthesis. NeRF represents scenes as a continuous 5D function of space–three spatial coordinates, - two viewing angles–and outputs a point-wise RGB radiance and opacity. A ray-casting renderer can - then query the network to generate arbitrary views of the scene. The network modeling the continuous - function is 10 layers deep. Two identical networks are trained: thecoarsenetwork predictions inform - the renderer about the spatial coordinates that thefinenetwork should preferentially evaluate to avoid - empty space and occluded regions. - - Results We report quantitative results of training NeRF with DFA in Table 1. Neural view synthesis - methods are often better evaluated qualitatively: we showcase some renders in Figure 1. - On a dataset of renders featuring complex scenes with non-Lambertian materials (NeRF-Synthetic - [36]), NeRF-DFA outperforms two previous fine-tuned state-of-the-art methods–Scene Representation - Networks (SRN) [35] and Local Light Field Fusion (LLFF) [32]–and nearly matches the performance - of Neural Volumes (NV) [34]. While DFA underperforms alternative methods trained with BP on - the real world view dataset (LLFF-Real [32]), its performance remains significant: real world view - synthesis is a challenging tasks, and this level of PSNR indicates that learning is indeed happening. - In particular, we find that NeRF-DFA retains the key characteristics of NeRF-BP: it can render view- - dependant effects, and is multi-view consistent. The last point is an especially important achievement, - and most visible in videos, as it is a challenge for most algorithms [30–32,35]. The main drawback - of NeRF-DFA appears to be a seemingly lower render definition. The NeRF architecture has not - - - Table 1: Peak Signal to Noise Ratio (PSNR, higher is better) of neural view synthesis methods - trained with backpropagation against NeRF trained with DFA. Even when trained with DFA, NeRF - outperforms two state-of-the-art methods on a synthetic dataset (NeRF-Synthetic), and achieves fair - performance on a challenging real world views datasets (LLFF-Real). BP results from [36]. - - NV SRN LLFF NeRF - BP BP BP BP DFA - NeRF-Synthetic 26.05 22.26 24.88 31.01 25.41 - LLFF-Real / 22.84 24.13 26.50 20.77 - - - 4 been fine-tuned to achieve these results: DFA works out-of-the-box on this advanced method. Future - research focusing on architectural changes to NeRF could improve performance with DFA; some - preliminary results are included in the supplementary material. - - 3.1.2 Click-through rate prediction with recommender systems - We have demonstrated that DFA can train large fully connected networks on the difficult task of neural - view synthesis. We now seek to use DFA in more complex heterogeneous architectures, combining - the use of fully connected networks with other machine learning methods.Recommender systemsare - an ideal application for such considerations. - - Background Recommender systems are used to model the behavior of users and predict future - interactions. In particular, in the context of click-through rate (CTR) prediction, these systems model - the probability of a user clicking on a given item. Building recommender systems is hard [37]: their - input is high-dimensional and sparse, and the model must learn to extract high-order combinatorial - features from the data. Moreover, they need to do so efficiently, as they are used to make millions of - predictions and the training data may contain billions of examples. - Factorization Machines (FM) [38] use inner-products of latent vectors between features to extract - pairwise feature interactions. They constitute an excellent baseline for shallow recommender systems, - but fail to efficiently transcribe higher-level features. To avoid extensive feature engineering, it has - been suggested that deep learning can be used in conjunction with wide shallow models to extract - these higher-level features [39]. In production, these systems are regularly retrained on massive - datasets: the speedup allowed by backward unlocking in DFA is thus of particular interest. - - Setting Deep Factorization Machines (DeepFM) [40] combine FM and a deep fully connected - neural network, which we train with DFA. The input embedding is still trained directly via gradient - descent, as weight transport is not necessary to backpropagate through the FM. Deep & Cross - Networks (DCN) [41] replace the FM with a Cross Network, a deep architecture without non- - linearities capable of extracting high-degree interactions across features. We train the fully connected - network, the deep cross network, and the embeddings with DFA. Finally, Adaptative Factorization - Network (AFN) [42] uses Logarithmic Neural Networks [43] to enhance the representational power - of its deep component. We evaluate these methods on the Criteo dataset [44], which features nearly - 46 million samples of one million sparse features. This is a difficult task, where performance - improvements of the AUC on the0.001-levelcan enhance CTR significantly [39]. - - Results Performance metrics are reported in Table 2. To obtain these results, a simple hyperpa- - rameter grid search over optimization and regularization parameters was performed for BP and DFA - independently. DFA successfully trains all methods above the FM baseline, and in fact matches BP - performance in both DeepFM and AFN. Because of their complexity, recommender systems require - intensive tuning and feature engineering to perform at the state-of-the-art level–and reproducing - existing results can be challenging [45]. Hence, it is not surprising that a performance gap exists with - Deep&Cross–further fine-tuning may be necessary for DFA to reach BP performance. - Alignment measurements corroborate that learning is indeed occurring in the special layers of - Deep&Cross and AFN–see supplementary for details. Our results on recommender systems support - that DFA can learn in a large variety of settings, and that weight transport is not necessary to solve a - difficult recommendation task. - - - Table 2: AUC (higher is better) and log loss (lower is better) of recommender systems trained on the - Criteo dataset [44]. Even in complex heterogeneous architectures, DFA performance is in line with - BP. Values inboldindicate DFA AUC within 0.001 from the BP AUC or better. - - FM DeepFM Deep&Cross AFN - BP DFA BP DFA BP DFA - AUC 0.7915 0.7954 0.7956 0.8104 0.8009 0.7933 0.7924 - Loss 0.4687 0.4610 0.4624 0.4414 0.4502 0.4630 0.4621 - - - 5 3.2 Geometric Learning with Graph Convolutional Networks - - The use of sophisticated architectures beyond fully connected layers is necessary for certain tasks, - such asgeometric learning[46], where information lies in a complex structured domain. To address - geometric learning tasks, methods capable of handling graph-based data are commonly needed. - Graph convolutional neural networks (GCNNs) [47–50] have demonstrated the ability to process - large-scale graph data efficiently. We study the applicability of DFA to these methods, including - recent architectures based on an attention mechanism. Overall, this is an especially interesting setting, - as DFA fails to train more classic 2D image convolutional layers [23]. - - Background Complex data like social networks or brain connectomes lie on irregular or non- - Euclidean domains. They can be represented as graphs, and efficient processing in the spectral - domain is possible. Non-spectral techniques to apply neural networks to graphs have also been - developed [51–53], but they exhibit unfavorable scaling properties. The success of CNNs in deep - learning can be attributed to their ability to efficiently process structured high-dimensional data - by sharing local filters. Thus, a generalization of the convolution operator to the graph domain is - desirable: [47] first proposed a spectral convolution operation for graphs, and [48] introduced a form - of regularization to enforce spatial locality of the filters. We use DFA to train different such GCNNs - implementations. We study both spectral and non-spectral convolutions, as well as methods inspired - by the attention mechanism. We consider the task of semi-supervised node classification: nodes from - a graph are classified using their relationship to other nodes as well as node-wise features. - - Setting Fast Localized Convolutions (ChebConv) [49] approximate the graph convolution kernel - with Chebyshev polynomials, and are one of the first scalable convolution methods on graph. Graph - Convolutions (GraphConv) [50] remove the need for an explicit parametrization of the kernel by - enforcing linearity of the convolution operation on the graph Laplacian spectrum. It is often considered - as the canonical graph convolution. More recent methods do not operate in the spectral domain. Spline - Convolutions (SplineConv) [54] use a spline-based kernel, enabling the inclusion of information - about the relative positioning of nodes, enhancing their representational power–for instance in the - context of 3D meshes. Graph Attention Networks (GATConv) [55] use self-attention [56] layers to - enable predictions at a given node toattendmore specifically to certain parts of its neighborhood. - Finally, building upon Jumping Knowledge Network [57], Just Jump (DNAConv) [58] uses multi- - head attention [59] to enhance the aggregation process in graph convolutions and enable deeper - architectures. We use PyTorch Geometric [60] for reference implementation of all of these methods. - We evaluate performance on three citation network datasets: Cora, CiteSeer, and PubMed [61]. - - Results We report classification accuracy in Table 3. BP and DFA regularization and optimiza- - tion hyperparameters are fine-tuned separately on the Cora dataset. In general, we find that less - regularization and lower learning rates are needed with DFA. DFA successfully trains all graph - methods, independent of whether they use the spectral domain or not, and even if they use attention. - Furthermore, for GraphConv, SplineConv, and GATConv DFA performance nearly matches BP. - As GCNNs struggle with learning meaningful representations when stacking many layers [62], all - architectures but DNAConv are quite shallow (two layers). However, DFA performance is still - significantly higher than that of a shallow training method–see supplementary for details. The lower - performance on DNAConv is not a failure to learn: alignment measurements show that learning is - indeed occurring. It may be explained instead by a need for more in-depth fine-tuning, as this is a - deep architecture with 5 successive attention layers. - - Table 3: Classification accuracy (%, higher is better) of graph convolution methods trained with BP - and DFA, on citation networks [61]. But for ChebConv and DNAConv, DFA performance nearly - matches BP performance. Values inboldwhen DFA is within 2.5% of BP. - - ChebConv GraphConv SplineConv GATConv DNAConv - BP DFA BP DFA BP DFA BP DFA BP DFA - Cora 79.2 75.4 80.1 79.9 81.0 77.7 82.6 80.6 84.6 82.9 - CiteSeer 69.5 67.6 71.6 69.4 70.0 69.8 72.0 71.2 73.4 70.8 - PubMed 79.5 75.7 78.8 77.8 77.5 77.2 77.7 77.1 87.2 79.9 - - - 6 GAE - BP DFA - AUC 0.918 0.900Cora AP 0.918 0.900 - AUC 0.886 0.879CiteSeer AP 0.895 0.889 - AUC 0.967 0.945PubMed AP 0.966 0.945 - - Table 4: AUC and Average Precision Figure 2: t-SNE visualization of the hidden layer - (AP, higher is better) for a Graph- activations of a two-layer GraphConv trained on - Conv GAE trained with BP or DFA Cora with DFA. Classes forms clear clusters, in- - on citation networks. DFA repro- dicating that a useful intermediary representation - duces BP performance. is learned. Colors represent different classes. - - - We further demonstrate that DFA helps graph convolutions learn meaningful representations by - aplying t-SNE [63,64] to the hidden layer activations in GraphConv (Figure 2). Cluster of classes - are well-separated, indicating that a useful intermediary representation is derived by the first layer. - - Graph autoencoders We consider one last application of graph convolutions, in the context of - graph autoencoders (GAE). We train a non-probabilistic GAE [65] based on GraphConv with DFA, - and report results in Table 4. DFA performance is always in line with BP. - - 3.3 Natural Language Processing with Transformers - - We complete our study by training a Transformer [59] on a language modelling task. Transformers - have proved successful in text, image, music generation, machine translation, and many supervised - NLP tasks [59,66–69]. Here, we demonstrate that DFA can train them, and we show the influence of - tuning the optimizer hyperparameters in narrowing the gap with BP. - - Background NLP has largely benefited from advances in deep learning. Recurrent Neural Net- - works were responsible for early breakthroughs, but their sequential nature prevented efficient - parallelization of data processing. Transformers are attention-based models that do not rely on - recurrence or convolution. Their ability to scale massively has allowed the training of models with - several billion parameters [70,71], obtaining state-of-the-art results on all NLP tasks: Transformers - now top the prominent SQuAD 2.0 [72,73] and SuperGLUE [74] benchmarks. In parallel, transfer - learning in NLP has leaped forward thanks to language modelling, the unsupervised task of predicting - the next word. It can leverage virtually unlimited data from web scraping [75]. This enabled the - training ofuniversal language models[76] on extremely large and diversified text corpora. These - models are useful across a wide range of domains, and can solve most NLP tasks after fine-tuning. - - Setting The prominence of both language modelling and Transformers gives us the ideal candidate - for our NLP experiments: we train a Transformer to predict the next word on the WikiText-103 - dataset [77], a large collection ofgoodandfeaturedWikipedia articles. We use byte-pair-encoding - [78] with 32,000 tokens. Our setup is similar to GPT [66]: we adapt the Transformer, originally an - encoder-decoder model designed for machine translation, to language modelling. We keep only the - encoder and mask the tokens to predict. Our architecture consists in 6 layers, 8 attention heads, a - model dimension of 512, and a hidden size of 2048 in the feed-forward blocks. The text is sliced - in chunks of 128 tokens and batches of 64 such chunks, resulting in 8192 tokens per batch. Our - baseline is trained with BP using the optimization setup of [59]. We found perplexity after 20 epochs - to be an excellent indicator of perplexity at convergence; to maximize the number of experiments - we could perform, we report the best validation perplexity after 20 epochs. We study two ways of - implementing DFA: applying the feedback after every encoder block (macro) or after every layer in - those blocks (micro). The input embedding layer receives gradients from the next feedback point - through BP. This leaves some amount of weight transport even in themicrocase. - - 7 Table 5: Best validation perplexity after 20 epochs of a Transformer trained on WikiText-103 (lower - is better). The BP and DFA baselines share all hyper-parameters. InMacrothe feedback is applied - after every transformer layer, while inMicrothe feedback is applied after every sub-layer. The - learning rate of Adam without the learning rate scheduler is5:10 5 . With the scheduler, the initial - learning rate is1:10 4 and it is multiplied by 0.2 when performance plateaus, with a patience of 1. - * score after 22 epochs to let the learning rate scheduler take effect - - DFA BP - Baseline + Adam +2 = 0:999 + LR schedule Baseline +2 = 0:999 - Macro 95.0 77.1 55.0 52.0 34.4 29.8Micro 182 166 99.9 93.3* - - - Results Our results are summarized in Table 5. Hyper-parameters fine-tuned for BP did not fare - well with DFA, but changes in the optimizer narrowed the gap between BP and DFA considerably. - The learning rate schedule used on top of Adam [79] in [59] proved detrimental. Using Adam alone - required reducing the learning rate between BP and DFA. Increasing2 from 0.98 [59] to 0.999 - improved performance significantly. Finally, a simple scheduler that reduces the learning rate when - the validation perplexity plateaus helped reducing it further. Considering that the perplexity of the - shallow baseline is over 400, DFA is clearly able to train Transformers. However, our results are not - on par with BP, especially in themicrosetting. A substantial amount of work remains to make DFA - competitive with BP, even more so in a minimal weight transport scenario. The large performance - improvements brought by small changes in the optimizer indicate that intensive fine-tuning, common - in publications introducing state-of-the-art results, could close the gap between BP and DFA. - - 4 Conclusion and outlooks - - We conducted an extensive study demonstrating the ability of DFA to train modern architectures. We - considered a broad selection of domains and tasks, with complex models featuring graph convolutions - and attention. Our results on large networks like NeRF and Transformers are encouraging, suggesting - that with further tuning, such leading architectures can be effectively trained with DFA. Future work - on principled training with DFA–in particular regarding the influence of common practices and - whether new procedures are required–will help close the gap with BP. - More broadly, we verified for the first time that learning under synaptic asymmetry is possible beyond - fully-connected layers, and in tasks significantly more difficult than previously considered. This - addresses a notable concern in biologically-plausible architectures. DFA still requires an implausible - global feedback pathway; however, local training has already been demonstrated at scale. The next - step towards biologically-compatible learning is a local method without weight transport. - While the tasks and architectures we have considered are not biologically inspired, they constitute - a good benchmark forbehavioural realism[20]. Any learning algorithm claiming to approximate - the brain should reproduce its ability to solve complex and unseen task. Furthermore, even though - the current implementation of mechanisms like attention is devoid of biological considerations, they - represent broader concepts applicable to human brains [80]. Understanding how our brain learns is a - gradual process, and future research could incorporate further realistic elements, like spiking neurons. - Finally, unlocking the backward pass in large architectures like Transformers is promising. More opti- - mized implementation of DFA–built at a lower-level of existing ML libraries–could unlock significant - speed-up. Leveraging the use of a single random projection as the cornerstone of training, dedicated - accelerators may employ more exotic hardware architectures. This will open new possibilities in the - asynchronous training of massive models. - - - - - - - - - - - 8 Broader Impact - - Of our survey This study is the first experimental validation of DFA as an effective training method - in a wide range of challenging tasks and neural networks architectures. This significantly broadens the - applications of DFA, and more generally brings new insight on training techniques alternative to back- - propagation. From neural rendering and recommender systems, to natural language processing or - geometric learning, each of these applications has its own potential impact. Our task selection process - was motivated by current trends in deep learning, as well as by technically appealing mechanisms - (graph convolutions, attention). A limit of our survey is that our–arguably biased–selection of tasks - cannot be exhaustive. Our experiments required substantial cloud compute resources, with state-of- - the-art GPU hardware. Nevertheless, as this study provides new perspectives for hardware accelerator - technologies, it may favor the application of neural networks in fields previously inaccessible because - of computational limits. Future research on DFA should continue to demonstrate its use in novel - contexts of interest as they are discovered. - - Of the considered applications Each of the applications considered in our study has a wide - potential impact, consider for example the impact of textual bias in pretrained word embeddings [81]. - We refer to [82] and references therein for a discussion of ethical concerns of AI applications. - - Of DFA as a training method DFA enables parallelization of the backward pass and places a - single operation at the center of the training process, opening the prospect of reducing the power - consumption of training chips by an order of magnitude [28]. Not only is more efficient training a - path to more environmentally responsible machine learning [83], but it may lower the barrier of entry, - supporting equality and sustainable development goals. A significant downside of moving from BP to - DFA is a far more limited understanding of how to train models and how the trained models behave. - There is a clear empirical understanding of the impact of techniques such as batch normalization - or skip connections on the performance of BP; new insights need to be obtained for DFA. BP also - enjoys decades of works on topics like adversarial attacks, interpretability, and fairness. Much of - this work has to be cross-checked for alternative training methods, something we encourage further - research to consider as the next step towards safely and responsively scaling up DFA. - - Of biologically motivated method Finally, a key motivation for this study was to demonstrate that - learning challenging tasks was possible without weight transport. Biologically motivated methods - are a more foundational research direction, and as such the possible long-term impact of our findings - is harder to estimate under this light. However, fundamental research of this kind is important to open - new pathways for ML and neuroscience. - - Acknowledgments and Disclosure of Funding - - We thank Igor Carron and Laurent Daudet for the general guidance on the subject of this investigation - and the insightful comments, as well as the larger LightOn team for their support. - - References - [1]P. J. Werbos.Beyond Regression: New Tools for Prediction and Analysis in the Behavioral - Sciences. PhD thesis, Harvard University, 1974. - [2]D. E. Rumelhart, G. E. Hinton, and R. J. Williams. Learning internal representations by error - propagation. InParallel Distributed Processing, volume 1, pages 318–362. MIT Press, 1986. - [3]Max Jaderberg, Wojciech Marian Czarnecki, Simon Osindero, Oriol Vinyals, Alex Graves, - David Silver, and Koray Kavukcuoglu. Decoupled neural interfaces using synthetic gradients. - InProceedings of the 34th International Conference on Machine Learning-Volume 70, pages - 1627–1635, 2017. - [4]Francis Crick. The recent excitement about neural networks.Nature, 337(6203):129–132, 1989. - [5]Adam H Marblestone, Greg Wayne, and Konrad P Kording. Toward an integration of deep - learning and neuroscience.Frontiers in computational neuroscience, 10:94, 2016. - - 9 [6]Stephen Grossberg. Competitive learning: From interactive activation to adaptive resonance. - Cognitive science, 11(1):23–63, 1987. - [7]Javier R Movellan. Contrastive hebbian learning in the continuous hopfield model. InConnec- - tionist models, pages 10–17. Elsevier, 1991. - [8]Randall C O’Reilly. Biologically plausible error-driven learning using local activation differ- - ences: The generalized recirculation algorithm.Neural computation, 8(5):895–938, 1996. - [9]Ruslan Salakhutdinov and Geoffrey Hinton. Deep boltzmann machines. InArtificial intelligence - and statistics, pages 448–455, 2009. - [10]Yann Le Cun. Learning process in an asymmetric threshold network. InDisordered systems - and biological organization, pages 233–240. Springer, 1986. - [11]Yoshua Bengio. How auto-encoders could provide credit assignment in deep networks via target - propagation.arXiv preprint arXiv:1407.7906, 2014. - [12]Dong-Hyun Lee, Saizheng Zhang, Asja Fischer, and Yoshua Bengio. Difference target propaga- - tion. InJoint european conference on machine learning and knowledge discovery in databases, - pages 498–515. Springer, 2015. - [13]Timothy P Lillicrap, Daniel Cownden, Douglas B Tweed, and Colin J Akerman. Random synap- - tic feedback weights support error backpropagation for deep learning.Nature communications, - 7(1):1–10, 2016. - [14]Eugene Belilovsky, Michael Eickenberg, and Edouard Oyallon. Greedy layerwise learning can - scale to imagenet. InInternational Conference on Machine Learning, pages 583–593, 2019. - [15]Wojciech M Czarnecki, Simon Osindero, Max Jaderberg, Grzegorz Swirszcz, and Razvan - Pascanu. Sobolev training for neural networks. InAdvances in Neural Information Processing - Systems, pages 4278–4287, 2017. - [16]Arild Nøkland and Lars Hiller Eidnes. Training neural networks with local error signals. In - International Conference on Machine Learning, pages 4839–4850, 2019. - [17]R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, - Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information - estimation and maximization. InInternational Conference on Learning Representations, 2019. - URLhttps://openreview.net/forum?id=Bklr3j0cKX. - [18]Sindy Löwe, Peter O’Connor, and Bastiaan Veeling. Putting an end to end-to-end: Gradient- - isolated learning of representations. InAdvances in Neural Information Processing Systems, - pages 3033–3045, 2019. - [19] Arild Nøkland. Direct feedback alignment provides learning in deep neural networks. In - Advances in neural information processing systems, pages 1037–1045, 2016. - [20]Sergey Bartunov, Adam Santoro, Blake Richards, Luke Marris, Geoffrey E Hinton, and Timothy - Lillicrap. Assessing the scalability of biologically-motivated deep learning algorithms and - architectures. InAdvances in Neural Information Processing Systems, pages 9368–9378, 2018. - [21]Timothy P Lillicrap, Adam Santoro, Luke Marris, Colin J Akerman, and Geoffrey Hinton. - Backpropagation and the brain.Nature Reviews Neuroscience, pages 1–12, 2020. - [22]Natalia Caporale and Yang Dan. Spike timing–dependent plasticity: a hebbian learning rule. - Annu. Rev. Neurosci., 31:25–46, 2008. - [23]Julien Launay, Iacopo Poli, and Florent Krzakala. Principled training of neural networks with - direct feedback alignment.arXiv preprint arXiv:1906.04554, 2019. - [24]Qianli Liao, Joel Z Leibo, and Tomaso Poggio. How important is weight symmetry in back- - propagation? InThirtieth AAAI Conference on Artificial Intelligence, 2016. - - 10 [25]Theodore H Moskovitz, Ashok Litwin-Kumar, and LF Abbott. Feedback alignment in deep - convolutional networks.arXiv preprint arXiv:1812.06488, 2018. - - [26]Will Xiao, Honglin Chen, Qianli Liao, and Tomaso Poggio. Biologically-plausible learning - algorithms can scale to large datasets. InInternational Conference on Learning Representations, - 2019. URLhttps://openreview.net/forum?id=SygvZ209F7. - - [27]Mohamed Akrout, Collin Wilson, Peter C Humphreys, Timothy Lillicrap, and Douglas Tweed. - Using weight mirrors to improve feedback alignment.arXiv preprint arXiv:1904.05391, 2019. - - [28]Julien Launay, Iacopo Poli, Kilian Müller, Igor Carron, Laurent Daudet, Florent Krzakala, and - Sylvain Gigan. Light-in-the-loop: using a photonics co-processor for scalable training of neural - networks, 2020. - - [29]Charlotte Frenkel.Bottom-Up and Top-Down Neuromorphic Processor Design: Unveiling - Roads to Embedded Cognition. PhD thesis, UCL-Université Catholique de Louvain, 2020. - - [30]Eric Penner and Li Zhang. Soft 3d reconstruction for view synthesis.ACM Transactions on - Graphics (TOG), 36(6):1–11, 2017. - - [31]John Flynn, Michael Broxton, Paul Debevec, Matthew DuVall, Graham Fyffe, Ryan Overbeck, - Noah Snavely, and Richard Tucker. Deepview: View synthesis with learned gradient descent. - InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages - 2367–2376, 2019. - - [32]Ben Mildenhall, Pratul P Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi - Ramamoorthi, Ren Ng, and Abhishek Kar. Local light field fusion: Practical view synthesis - with prescriptive sampling guidelines.ACM Transactions on Graphics (TOG), 38(4):1–14, - 2019. - - [33]Vincent Sitzmann, Justus Thies, Felix Heide, Matthias Nießner, Gordon Wetzstein, and Michael - Zollhofer. Deepvoxels: Learning persistent 3d feature embeddings. InProceedings of the IEEE - Conference on Computer Vision and Pattern Recognition, pages 2437–2446, 2019. - - [34]Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, and - Yaser Sheikh. Neural volumes: Learning dynamic renderable volumes from images.ACM - Transactions on Graphics (TOG), 38(4):65, 2019. - - [35]Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: - Continuous 3d-structure-aware neural scene representations. InAdvances in Neural Information - Processing Systems, pages 1119–1130, 2019. - - [36]Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, - and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis.arXiv - preprint arXiv:2003.08934, 2020. - - [37]H Brendan McMahan, Gary Holt, David Sculley, Michael Young, Dietmar Ebner, Julian Grady, - Lan Nie, Todd Phillips, Eugene Davydov, Daniel Golovin, et al. Ad click prediction: a view - from the trenches. InProceedings of the 19th ACM SIGKDD international conference on - Knowledge discovery and data mining, pages 1222–1230, 2013. - - [38]Steffen Rendle. Factorization machines. In2010 IEEE International Conference on Data - Mining, pages 995–1000. IEEE, 2010. - - [39]Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, - Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, et al. Wide & deep learning for - recommender systems. InProceedings of the 1st workshop on deep learning for recommender - systems, pages 7–10, 2016. - - [40]Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. Deepfm: a - factorization-machine based neural network for ctr prediction.arXiv preprint arXiv:1703.04247, - 2017. - - 11 [41]Ruoxi Wang, Bin Fu, Gang Fu, and Mingliang Wang. Deep & cross network for ad click - predictions. InProceedings of the ADKDD’17, ADKDD’17, New York, NY, USA, 2017. - Association for Computing Machinery. ISBN 9781450351942. doi: 10.1145/3124749.3124754. - URLhttps://doi.org/10.1145/3124749.3124754. - [42]Weiyu Cheng, Yanyan Shen, and Linpeng Huang. Adaptive factorization network: Learning - adaptive-order feature interactions. InThirty-Fourth AAAI Conference on Artificial Intelligence, - 2020. - [43]J Wesley Hines. A logarithmic neural network architecture for unbounded non-linear function - approximation. InProceedings of International Conference on Neural Networks (ICNN’96), - volume 2, pages 1245–1250. IEEE, 1996. - [44]Criteo. Kaggle contest dataset is now available for academic use!http://labs.criteo.com/ - 2014/09/kaggle-contest-dataset-now-available-academic-use/, 2014. accessed - on the 2020-05-20. - [45]Maurizio Ferrari Dacrema, Paolo Cremonesi, and Dietmar Jannach. Are we really making much - progress? a worrying analysis of recent neural recommendation approaches. InProceedings of - the 13th ACM Conference on Recommender Systems, pages 101–109, 2019. - [46]Michael M Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst. - Geometric deep learning: going beyond euclidean data.IEEE Signal Processing Magazine, 34 - (4):18–42, 2017. - [47]Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann Lecun. Spectral networks and locally - connected networks on graphs. InInternational Conference on Learning Representations, pages - http–openreview, 2014. - [48]Mikael Henaff, Joan Bruna, and Yann LeCun. Deep convolutional networks on graph-structured - data.arXiv preprint arXiv:1506.05163, 2015. - [49]Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks - on graphs with fast localized spectral filtering. InAdvances in neural information processing - systems, pages 3844–3852, 2016. - [50]Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional - networks. InInternational Conference on Learning Representations (ICLR), 2017. - [51]Marco Gori, Gabriele Monfardini, and Franco Scarselli. A new model for learning in graph - domains. InProceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005., - volume 2, pages 729–734. IEEE, 2005. - [52]Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. - The graph neural network model.IEEE Transactions on Neural Networks, 20(1):61–80, 2008. - [53]Yujia Li, Daniel Tarlow, Marc Brockschmidt, and Richard Zemel. Gated graph sequence neural - networks. InInternational Conference on Learning Representations, 2016. - [54]Matthias Fey, Jan Eric Lenssen, Frank Weichert, and Heinrich Müller. Splinecnn: Fast geometric - deep learning with continuous b-spline kernels. InProceedings of the IEEE Conference on - Computer Vision and Pattern Recognition, pages 869–877, 2018. - [55]Petar Velickoviˇ c, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua´ - Bengio. Graph attention networks. InInternational Conference on Learning Representations, - 2018. URLhttps://openreview.net/forum?id=rJXMpikCZ. - [56] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly - learning to align and translate. In3rd International Conference on Learning Representations, - ICLR 2015, 2015. - [57]Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural - networks? InInternational Conference on Machine Learning, 2018. - - 12 [58]Matthias Fey. Just jump: Dynamic neighborhood aggregation in graph neural networks. In - ICLR Workshop on Representation Learning on Graphs and Manifolds, 2019. - - [59]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, - Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. InAdvances in neural information - processing systems, pages 5998–6008, 2017. - - [60]Matthias Fey and Jan E. Lenssen. Fast graph representation learning with PyTorch Geometric. - InICLR Workshop on Representation Learning on Graphs and Manifolds, 2019. - - [61]Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi- - Rad. Collective classification in network data.AI magazine, 29(3):93–93, 2008. - - [62]Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural - networks? InInternational Conference on Learning Representations, 2019. URLhttps: - //openreview.net/forum?id=ryGs6iA5Km. - - [63]Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne.Journal of machine - learning research, 9(Nov):2579–2605, 2008. - - [64]David M Chan, Roshan Rao, Forrest Huang, and John F Canny. Gpu accelerated t-distributed - stochastic neighbor embedding.Journal of Parallel and Distributed Computing, 131:1–13, - 2019. - - [65]Thomas N Kipf and Max Welling. Variational graph auto-encoders.NIPS Workshop on Bayesian - Deep Learning, 2016. - - [66]Alec Radford, Karthik Narasimhan, Time Salimans, and Ilya Sutskever. Improving language - understanding with unsupervised learning.Technical report, OpenAI, 2018. - - [67]Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, - and Dustin Tran. Image transformer.ArXiv, abs/1802.05751, 2018. - - [68]Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya - Sutskever. Jukebox: A generative model for music.arXiv preprint arXiv:2005.00341, 2020. - - [69]Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of - deep bidirectional transformers for language understanding. InProceedings of the 2019 Confer- - ence of the North American Chapter of the Association for Computational Linguistics: Human - Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, - Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1423. - URLhttps://www.aclweb.org/anthology/N19-1423. - - [70]Mohammad Shoeybi, Mostofa Ali Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and - Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model - parallelism.ArXiv, abs/1909.08053, 2019. - - [71]Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, - Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are - few-shot learners.arXiv preprint arXiv:2005.14165, 2020. - - [72]Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ - questions for machine comprehension of text. InProceedings of the 2016 Conference on - Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas, Novem- - ber 2016. Association for Computational Linguistics. doi: 10.18653/v1/D16-1264. URL - https://www.aclweb.org/anthology/D16-1264. - - [73]Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don’t know: Unanswerable - questions for SQuAD. InProceedings of the 56th Annual Meeting of the Association for - Computational Linguistics (Volume 2: Short Papers), pages 784–789, Melbourne, Australia, - July 2018. Association for Computational Linguistics. doi: 10.18653/v1/P18-2124. URL - https://www.aclweb.org/anthology/P18-2124. - - 13 [74]Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix - Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose - language understanding systems. InAdvances in Neural Information Processing Systems, pages - 3261–3275, 2019. - [75]The Common Crawl Team. Common Crawl.https://commoncrawl.org, 2020. - [76]Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classifica- - tion. InACL. Association for Computational Linguistics, 2018. URLhttp://arxiv.org/ - abs/1801.06146. - [77]Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture - models.ArXiv, abs/1609.07843, 2017. - [78]Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare - words with subword units. InProceedings of the 54th Annual Meeting of the Association - for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725, Berlin, Germany, - August 2016. Association for Computational Linguistics. doi: 10.18653/v1/P16-1162. URL - https://www.aclweb.org/anthology/P16-1162. - [79]Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization.International - Conference on Learning Representations, 12 2014. - [80]Grace W Lindsay. Attention in psychology, neuroscience, and machine learning.Frontiers in - Computational Neuroscience, 14:29, 2020. - [81]Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. - Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In - Advances in neural information processing systems, pages 4349–4357, 2016. - [82]Alexandra Luccioni and Yoshua Bengio. On the morality of artificial intelligence.arXiv preprint - arXiv:1912.11945, 2019. - [83]Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for - deep learning in nlp.arXiv preprint arXiv:1906.02243, 2019. - [84]Yi Tay, Dara Bahri, Donald Metzler, Da-Cheng Juan, Zhe Zhao, and Che Zheng. Synthesizer: - Rethinking self-attention in transformer models.arXiv preprint arXiv:2005.00743, 2020. - [85]Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, - and Jiawei Han. On the variance of the adaptive learning rate and beyond.arXiv preprint - arXiv:1908.03265, 2019. - [86]Alessandro Raganato, Yves Scherrer, and Jörg Tiedemann. Fixed encoder self-attention patterns - in transformer-based machine translation.arXiv preprint arXiv:2002.10260, 2020. - [87]Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, - Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas - Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, - Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- - performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- - Buc, E. Fox, and R. Garnett, editors,Advances in Neural Information Processing Systems 32, - pages 8024–8035. Curran Associates, Inc., 2019. URLhttp://papers.neurips.cc/paper/ - 9015-pytorch-an-imperative-style-high-performance-deep-learning-library. - pdf. - - - - - - - - - - - 14 Appendix - - - We first provide additional elements to corroborate our findings: alignment measurement (Section - A), and shallow baselines (Section B). We then discuss the process of adapting the considered - architectures for DFA (Section C), and the issue of weight transport in attention layers (Section D). - We provide some supplementary results for NeRF (Section E), including details of performance on - each scene of each datatset, and a discussion on possible mitigation of DFA shortcomings. Finally, - we outline steps necessary for reproduction of this work (Section F). - - A Alignment - - Alignment measurement In feedback alignment methods, the forward weights learn toalignwith - the random backward weights, making the delivered updates useful. This alignment can be quantified - by measuring the cosine similarity between the gradient signal delivered by DFABi ay and the - gradient signal BP would have deliveredWT ai+1 i+1 . For learning to occur and DFA to work as - a training method, there must be alignment. This can be measured numerically [23]. Measuring - alignments allows to check whether or not the layers are effectively being trained by DFA, regardless - of performance metrics. We note that any alignment value superior to 0 signifies that learning is - occuring. Values closer to 1 indicate a better match with BP, but small alignment values are sufficient - to enable learning. We report values measured at the deepest DFA layer. - - Recommender systems We measure alignment on the Criteo dataset, in the two architectures - featuring non-conventional fully-connected layers: Deep & Cross and AFN. Alignment is measured - after 15 epochs of training, and averaged over a random batch of 512 samples. Results are reported in - table A.1. These alignment measurements indicate that learning is indeed occurring in the cross and - logarithmic layers. High-variance of alignment in the cross layers is unique: it may be explained by - the absence of non-linearity, and account for the difference in performance between BP and DFA on - this architecture–which is higher than on the others. - - Table A.1: Alignment cosine similarity (higher is better, standard deviation in parenthesis) of - recommender systems as measured on the Criteo dataset. Learning occurs in both architectures, and - high variance may explain the larger performance gap on Deep & Cross compared to other methods. - - Deep & Cross AFN - Alignment 0.40 (0.91) 0.49 (0.08) - - - Graph convolutions We measure alignment on the Cora dataset, after 250 epochs of training, - averaging values over every sample available–train, validation, and test split included. Results are - reported in Table A.2. We observe high alignment values in all architectures, indicative that learning - is indeed occuring. Slightly lower values in SplineConv and GATConv may be explained by the use - of the Exponential Linear Unit (ELU) instead of the Rectified Linear Unit (ReLU) used as activation - in other architectures. - Table A.2: Alignment cosine similarity (standard deviation in parenthesis) of various graph convolu- - tions architectures as measured on the Cora dataset. These values corroborate that DFA successfully - trains all architectures considered. - - ChebConv GraphConv SplineConv GATConv DNAConv - Alignment 0.87 (0.12) 0.77 (0.25) 0.56 (0.22) 0.63 (0.18) 0.92 (0.30) - - - B Shallow baselines - - Shallow learning We compare DFA to BP, but also to shallow learning–where only the topmost - layer is trained. While DFA may not reach the performance level of BP, it should still vastly - - 15 Figure A.1: Comparisons of Tiny-NeRF trained with BP, DFA, and a shallow approach. Shallow - training is insufficient to learn scene geometry. Lego scene from the NeRF synthetic dataset. - - - outperform shallow learning: failure to do so would mean that the weight updates delivered by DFA - are useless. On a simple task like MNIST, a shallow baseline may be as high as 90%. However, given - the difficulty of the tasks we consider, the shallow baseline is here usually much lower. - - NeRF Because NeRF models are expensive to train–up to 15 hours on a V100–we consider a - simplified setup for the shallow baseline, NeRF-Tiny. This setup operates at half the full resolution - of the training images available, runs for 5000 iterations only, and does away with view-dependant - characteristics. Furthermore, the network is cut down to 3 layers of half the width of NeRF, and - no coarse network is used to inform the sampling. We train this network on the Lego scene of the - NeRF-Synthetic dataset, and compare results. - Figure A.1 presents renders generated by NeRF-Tiny trained with BP, DFA, and a shallow approach. - While BP and DFA delivers similar renders, shallow training fails to reproduce even basic scene - geometry, instead outputting a diffuse cloud of colors. This highlights that while DFA may not reach - a level of performance on-par with BP on NeRF, it nonetheless delivers meaningful updates enabling - the learning of complex features. - - Recommender systems Because recommender systems require fine-tuning, we perform the same - hyperparameter search for shallow learning than for DFA and BP. Results are detailed in Table A.3. - Performance of shallow training is always well under BP and DFA–remember that0.001-levelmatter - in recommender systems. In particular, in Deep & Cross, where there was the biggest gap between - BP and DFA, the performance of the shallow method is extremely poor, well below the FM baseline. - Finally, it is expected to see that DeepFM recovers more or less the performance of FM even with a - shallow baseline. - - Table A.3: Shallow baseline for recommender system models on the Criteo dataset. Performance is - always well below BP and DFA, as expected. - - DeepFM Deep&Cross AFN - AUC 0.7920 0.7324 0.7859 - Loss 0.4682 0.5010 0.4685 - - - Graph convolutions We use the same hyperparameters as for DFA to produce the shallow baseline - on graph datasets. Results are reported in Table A.4. Performance is always much worse than BP - and DFA. GATConv recovers the best performance: random attention layers may still deliver useful - features [84], as do random convolutions. - - Transformers In the baseline setting (optimizer and hyper-parameters of [59]), a Transformer - trained in the shallow regime yields a perplexity of 428 on WikiText-103. We do not consider - - 16 Table A.4: Shallow baseline for GCNNs on Cora, CiteSeer, and PubMed [61]. Performance is always - well below BP and DFA. - - ChebConv GraphConv SplineConv GATConv DNAConv - Cora 23.3 37.0 39.6 59.4 30.2 - CiteSeer 27.4 33.8 30.1 49.8 24.0 - PubMed 37.6 44.8 44.2 67.8 42.2 - - - - other settings, as the cost of training a Transformer is high and we do not expect any meaningful - improvements–as with NeRF above. - - - C Adapting architectures to DFA - - NeRF We use an architecture identical to the one used in [36], but based on the effective code - implementation rather than the description in the paper 1 . During our tests, we have found that - lowering the learning rate to1:10 4 rather than5:10 4 works best with DFA. - - - Recommender systems For all training methods (BP, DFA, and shallow), we have conducted - independent hyperparameter searches. We performed a grid search over the learning rate, from - 1:10 4 to1:10 3 in1:10 4 steps, as well as over the dropout probability, from0:1to0:5in0:1steps - (where applicable). On DeepFM, this search leads to reduce the learning rate from3:10 4 with BP - to5:10 5 with DFA, but to keep the 0.5 dropout rate. On Deep & Cross, we reduce learning rate - from2:10 4 to5:10 5 , with no dropout in both cases. In AFN, we reduce dropout from4:10 4 to - 3:10 4 and dropout from 0.3 to 0. - - - Graph convolutions We manually test for a few hyperparameters configuration on the Cora dataset, - focusing on learning rate, weight decay, and dropout. We do not consider architectural changes, such - as changing the number of filters or of attention heads. For ChebConv and GraphConv, we reduce - weight decay to1:10 4 instead of5:10 4 , and set the dropout rate to0and0:1respectively, instead - of0:5with BP. For SplineConv, we find that no change in the hyperparameters are necessary. For - GATConv, we reduce weight decay to1:10 4 instead of5:10 4 and reduce dedicated dropout layer - to0:1instead of0:6but keep the0:6dropout rate within the GAT layer. Finally, on DNAConv we - disable weight decay entirely, instead of an original value of5:10 4 , double the learning rate from - 5:10 3 to1:10 2 , and disable dropout entirely. In all cases, we share the backward random matrix - across all nodes in a graph. - - - Transformers The model hyper-parameters were fixed across all of our experiments, except for - the number of attention heads in one case, that we will precise below, and dropout. We tested several - values of dropout probability between 0 and 0.5, but found the original value of 0.1 to perform - best. We manually tested a number of optimizers, optimizer parameters and attention mechanisms. - We tested four combinations of optimizers and schedulers : Adam with the scheduler used in [59], - Adam alone, RAdam [85] alone, and Adam with a scheduler that reduces the learning rate when - the validation perplexity plateaus. We found it necessary to reduce the initial learning rate of Adam - from1:10 4 to5:10 5 , although it could be set back to1:10 4 with a scheduler. We tried two values - of2 : 0.98 and 0.999. We also tried to change1 and observed some small differences that were - not significant enough for the main text. Finally, we tried three attention mechanisms in addition to - the standard multihead scaled dot-product attention: the dense and random (learnable) Synthesizers - of [84], as well as the fixed attention patterns of [86]. The latter needed to be adapted to language - modelling to prevent attending to future tokens, which led us to reduced the number of attention - heads to 4. The backward random matrix is always shared across all tokens and batches. - - - 1 https://github.com/bmild/nerf/issues/11 - - 17 D Weight transport and attention - - We consider an attention layer operating on inputx. The queries, keys, and values are respectively - q=xW Q ;k=xW K ;v=xW V , anddk is the dimension of the queries and keys. The layer - performs: qk T - Attention(q;k;v) =softmax p v (4)dk - - When using DFA on attention, we deliver the random feedback to the top of the layer. Accordingly, - to obtain updates toWQ ;WK ;andWV we still to have to backpropagate through the attention - mechanism itself. This involves weight transport onWV , sacrificing some biological realism for - simplicity. Overall weight transport between layers still does not occur, and updating the layers in - parallel remains possible. - Beside using FA or DFA within the attention layer, alternative mechanisms like the synthesizer - [84]–which uses random attention in place of the query and key system–or fixed attention [86] can - remove the need for weight transport. Implementing these mechanisms in DFA-trained Transformers, - or other attention-powered architectures, will require further research. - - - E Supplementary NeRF results - - Quantitative results We report per-scene scores for each dataset in Table A.5. BP values are taken - from [36]. On three scenes of the synthetic datasets, NeRF-DFA even outperforms past state-of-the-art - methods trained with BP. Note that Neural Volumes (NV) is not applicable to forward-facing view - synthesis–as is required in LLFF-Real–and thus no results are reported. - - Qualitative results We report sample renders from the NeRF-Synthetic dataset (Figure A.2) and - the LLFF-Real dataset (Figure A.2), for every scene available. However, we recommend readers to - consult the supplementary video to make better sense of characteristics like multi-view consistency - and view-dependent effects (most visible on the LLFF-Real Room scene). - - - Table A.5: Per-scene PSNR for NeRF DFA and BP against other state-of-the-art methods on the - Nerf-Synthetic and LLFF-Real. DFA performance is fairly homogeneous across each dataset and in - line with the differences in other methods. - - NV SRN LLFF NeRF - BP BP BP BP DFA - NeRF-Synthetic 26.05 22.26 24.88 31.01 25.41 - Chair 28.33 26.96 28.72 33.00 28.74 - Drums 22.58 17.18 21.13 25.01 22.15 - Ficus 24.79 20.73 21.79 30.13 25.61 - Hotdog 30.71 26.81 31.41 36.18 28.03 - Lego 26.08 20.85 24.54 32.54 24.93 - Materials 24.22 18.09 20.72 29.62 25.15 - Mic 27.78 26.85 27.48 32.91 25.43 - Ship 23.93 20.60 23.22 28.65 23.25 - LLFF-Real 22.84 24.13 26.50 20.77 - Room 27.29 28.42 32.70 24.20 - Fern 21.37 22.95 25.17 21.82 - Leaves 18.24 19.52 20.92 16.50 - Fortress 26.63 29.40 31.16 25.16 - Orchids 17.37 18.52 20.36 16.73 - Flower 26.63 25.46 27.40 21.55 - T-Rex 22.87 24.15 26.80 19.43 - Horns 24.33 24.70 27.45 20.75 - - - 18 Possible future directions Despite retranscribing scene geometry in a multi-view consistent way, - NeRF produces renders of a lower quality when trained with DFA instead of BP. In particular, it - struggles to transcribe small-scale details, resulting in "blurry" renders. Moreover, it displays high- - frequency artefacts: not in the scene geometry, but in individual pixels taking values very distant from - their neighborhood. Interestingly, this noise phenomenon is unique to NeRF-DFA: it is not observed - on NeRF-BP with similar PSNR values (achieved during training) or on other methods with similar - or lower PSNR. This leads us to hypothesize this is an aspect unique to DFA, possibly due to the - alignment process. Indeed, DFA creates a bias on the weights, by encouraging them to be "aligned" - with an arbitrary values dependant on the random matrix used. It is possible this could introduce - random noise in the final renders–though we leave a more principled experiment to future research. - To attempt to alleviate this issue, we first consider NeRF-Dual. In NeRF-Dual, we average the - pixel-wise prediction between the fine and coarse network, to attempt to remove some of the noise. - To do so, we first still use the coarse network to create a probability distribution for the hierarchical - sampling. Then, we evaluate again both the coarse and fine networks at the locations informed by - this probability distribution. Compared to vanilla NeRF, this requires an extra batch of evaluation of - the coarse network for all rays–rougly speaking, this increases inference time by 30-50% depending - on the coarse network architecture considered. We note that this is not applied during training, so that - training times remain identical. - Figure A.2 and Figure A.3 showcase comparisons between NeRF and NeRF-Dual trained with DFA - on all scenes. When viewed at high resolution–such as in our supplementary video–the NeRF-Dual - renders are more pleasing, especially for the full scenes. They remove most of the high-frequency - noise, leading to smoother renders. However, this averaging process further blurs small-scale details in - the render. This is especially visible in the NeRF-Synthetic dataset, on scenes like Ficus. Furthermore, - NeRF-Dual introduces novel artefacts in the Mic and Ship scenes, with areas improperly colored - with a violet tint. The cause for these artefacts is unknown, but they show that NeRF-Dual is far from - a silver bullet. The PSNR is also minimally increased, by less than 0.5 per scene. Nevertheless, this - shows some promise in possibilities to allievate the shortcomings of NeRF-DFA. It is possible that - changes to the overall rendering process, or the use of classic image processing techniques, may help - enhance the NeRF-DFA images. - Finally, we also experimented with increasing the capacity of the fine network, by widening its layers - to 512 neurons. We call this architecture NeRF-XL. However, we have not succeeded in getting - PSNR values higher than with vanilla NeRF on DFA. In particular, the training process becomes - much more cumbersome, as multi-GPU parallelism is needed to fit the model. It is possible that - higher network capacity may help learning both the task at hand and to align simultaneously, but - further work is required. - - - F Reproducibility - - Hardware used All main experiments require at most a single NVIDIA V100 GPU with 16GB - of memory to reproduce. Alignment measurement on large architectures (NeRF and Transformers) - require a second identical GPU to keep a copy of the network to evaluate BP gradients. - We estimate that a total of around 10,000 GPU-hours on V100s were necessary for this paper. - Accordingly, we estimate the cloud-computing carbon impact of this paper to be of 1700 kgCO 2 eq 2 . - However, without hyperparameter searches, our results can be reproduced with less than 500 GPU- - hours on V100s, with most of that budget going to NeRF and Transformers. - - Implementation We use the shared random matrix trick from [23] to reduce memory use in DFA - and enable its scaling to large networks. We use PyTorch [87] for all experiments. For reference - implementation of the methods considered, we relied on various sources. Our NeRF implementation - is based on the PyTorch implementation by Krishna Murthy 3 , with modifications to allow for proper - test and validation, as well as DFA and multi-GPU support. For recommender systems, we use - - 2 https://mlco2.github.io/impact#compute - 3 https://github.com/krrish94/nerf-pytorch - - 19 thetorchfmpackage 4 . Finally, we use PyTorch Geometric [60] for all graph operations. Our - Transformer implementation is our own. Our code is available as supplementary material. - - NeRF We provide training, testing, and rendering code along with the configurations used to obtain - our results. An example to reproduce our results is given in the supplementary code repository. Given - the computing cost associated with training a NeRF, we also provide our trained models. - - Recommender systems We provide bash scripts to reproduce the results in Table 2 and A.3, with - the results of our hyperparameter search. We provide code to reproduce the results in Table A.1. - - Graph convolutions We provide the code to reproduce all of our results. Note that the t-SNE - results are not exactly reproducible, as the CUDA implementation used is non-deterministic. - - Transformers We provide bash scripts to reproduce Table 5 and the shallow results. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 4 https://github.com/rixwew/pytorch-fm - - 20 Figure A.2: Sample renders for every scene of the NeRF-Synthetic dataset, for NeRF and NeRF-Dual - trained with DFA. - - - - - - - - - - - 21 Figure A.3: Sample renders for every scene of the LLFF-Real dataset, for NeRF and NeRF-Dual - trained with DFA. - - - - - - - - - - - - 22 \ No newline at end of file diff --git a/Corpus/Efficient Behavior of Small-World Networks.txt b/Corpus/Efficient Behavior of Small-World Networks.txt deleted file mode 100644 index 18b01f0..0000000 Binary files a/Corpus/Efficient Behavior of Small-World Networks.txt and /dev/null differ diff --git a/Corpus/Efficient Processing of Deep Neural Networks- A Tutorial and Survey.txt b/Corpus/Efficient Processing of Deep Neural Networks- A Tutorial and Survey.txt index 319bda1..bf1fb21 100644 Binary files a/Corpus/Efficient Processing of Deep Neural Networks- A Tutorial and Survey.txt and b/Corpus/Efficient Processing of Deep Neural Networks- A Tutorial and Survey.txt differ