</script><linkhref="https://cdnjs.cloudflare.com/ajax/libs/normalize/4.2.0/normalize.min.css"rel="stylesheet"type="text/css"/><linkhref="https://fonts.googleapis.com/css?family=Lato|Roboto+Mono"rel="stylesheet"type="text/css"/><linkhref="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/4.6.3/css/font-awesome.min.css"rel="stylesheet"type="text/css"/><linkhref="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"rel="stylesheet"type="text/css"/><script>documenterBaseURL=".."</script><scriptsrc="https://cdnjs.cloudflare.com/ajax/libs/require.js/2.2.0/require.min.js"data-main="../assets/documenter.js"></script><scriptsrc="../siteinfo.js"></script><scriptsrc="../../versions.js"></script><linkhref="../assets/documenter.css"rel="stylesheet"type="text/css"/><linkhref="../../flux.css"rel="stylesheet"type="text/css"/></head><body><navclass="toc"><h1>Flux</h1><selectid="version-selector"onChange="window.location.href=this.value"style="visibility: hidden"></select><formclass="search"id="search-form"action="../search.html"><inputid="search-query"name="q"type="text"placeholder="Search docs"/></form><ul><li><aclass="toctext"href="../index.html">Home</a></li><li><spanclass="toctext">Building Models</span><ul><li><aclass="toctext"href="basics.html">Basics</a></li><li><aclass="toctext"href="recurrence.html">Recurrence</a></li><li><aclass="toctext"href="regularisation.html">Regularisation</a></li><liclass="current"><aclass="toctext"href="layers.html">Model Reference</a><ulclass="internal"><li><aclass="toctext"href="#Basic-Layers-1">Basic Layers</a></li><li><aclass="toctext"href="#Recurrent-Layers-1">Recurrent Layers</a></li><li><aclass="toctext"href="#Activation-Functions-1">Activation Functions</a></li><li><aclass="toctext"href="#Normalisation-and-Regularisation-1">Normalisation & Regularisation</a></li></ul></li></ul></li><li><spanclass="toctext">Training Models</span><ul><li><aclass="toctext"href="../training/optimisers.html">Optimisers</a></li><li><aclass="toctext"href="../training/training.html">Training</a></li></ul></li><li><aclass="toctext"href="../data/onehot.html">One-Hot Encoding</a></li><li><aclass="toctext"href="../gpu.html">GPU Support</a></li><li><aclass="toctext"href="../saving.html">Saving & Loading</a></li><li><aclass="toctext"href="../community.html">Community</a></li></ul></nav><articleid="docs"><header><nav><ul><li>Building Models</li><li><ahref="layers.html">Model Reference</a></li></ul><aclass="edit-page"href="https://github.com/FluxML/Flux.jl/blob/master/docs/src/models/layers.md"><spanclass="fa"></span> Edit on GitHub</a></nav><hr/><divid="topbar"><span>Model Reference</span><aclass="fa fa-bars"href="#"></a></div></header><h2><aclass="nav-anchor"id="Basic-Layers-1"href="#Basic-Layers-1">Basic Layers</a></h2><p>These core layers form the foundation of almost all neural networks.</p><pre><codeclass="language-none">Chain
Conv2D</code></pre><h2><aclass="nav-anchor"id="Recurrent-Layers-1"href="#Recurrent-Layers-1">Recurrent Layers</a></h2><p>Much like the core layers above, but can be used to process sequence data (as well as other kinds of structured data).</p><sectionclass="docstring"><divclass="docstring-header"><aclass="docstring-binding"id="Flux.RNN"href="#Flux.RNN"><code>Flux.RNN</code></a> — <spanclass="docstring-category">Function</span>.</div><div><pre><codeclass="language-none">RNN(in::Integer, out::Integer, σ = tanh)</code></pre><p>The most basic recurrent layer; essentially acts as a <code>Dense</code> layer, but with the output fed back into the input each time step.</p></div><aclass="source-link"target="_blank"href="https://github.com/FluxML/Flux.jl/blob/642543808effb5bd79e52c2815646d88ff967c63/src/layers/recurrent.jl#L105-L110">source</a></section><sectionclass="docstring"><divclass="docstring-header"><aclass="docstring-binding"id="Flux.LSTM"href="#Flux.LSTM"><code>Flux.LSTM</code></a> — <spanclass="docstring-category">Function</span>.</div><div><pre><codeclass="language-none">LSTM(in::Integer, out::Integer, σ = tanh)</code></pre><p>Long Short Term Memory recurrent layer. Behaves like an RNN but generally exhibits a longer memory span over sequences.</p><p>See <ahref="http://colah.github.io/posts/2015-08-Understanding-LSTMs/">this article</a> for a good overview of the internals.</p></div><aclass="source-link"target="_blank"href="https://github.com/FluxML/Flux.jl/blob/642543808effb5bd79e52c2815646d88ff967c63/src/layers/recurrent.jl#L151-L159">source</a></section><sectionclass="docstring"><divclass="docstring-header"><aclass="docstring-binding"id="Flux.Recur"href="#Flux.Recur"><code>Flux.Recur</code></a> — <spanclass="docstring-category">Type</span>.</div><div><pre><codeclass="language-none">Recur(cell)</code></pre><p><code>Recur</code> takes a recurrent cell and makes it stateful, managing the hidden state in the background. <code>cell</code> should be a model of the form:</p><pre><codeclass="language-none">h, y = cell(h, x...)</code></pre><p>For example, here's a recurrent network that keeps a running total of its inputs.</p><pre><codeclass="language-julia">accum(h, x) = (h+x, x)
rnn.state # 60</code></pre></div><aclass="source-link"target="_blank"href="https://github.com/FluxML/Flux.jl/blob/642543808effb5bd79e52c2815646d88ff967c63/src/layers/recurrent.jl#L7-L26">source</a></section><h2><aclass="nav-anchor"id="Activation-Functions-1"href="#Activation-Functions-1">Activation Functions</a></h2><p>Non-linearities that go between layers of your model. Most of these functions are defined in <ahref="https://github.com/FluxML/NNlib.jl">NNlib</a> but are available by default in Flux.</p><p>Note that, unless otherwise stated, activation functions operate on scalars. To apply them to an array you can call <code>σ.(xs)</code>, <code>relu.(xs)</code> and so on.</p><sectionclass="docstring"><divclass="docstring-header"><aclass="docstring-binding"id="NNlib.σ"href="#NNlib.σ"><code>NNlib.σ</code></a> — <spanclass="docstring-category">Function</span>.</div><div><pre><codeclass="language-none">σ(x) = 1 / (1 + exp(-x))</code></pre><p>Classic <ahref="https://en.wikipedia.org/wiki/Sigmoid_function">sigmoid</a> activation function.</p></div><aclass="source-link"target="_blank"href="https://github.com/FluxML/NNlib.jl/blob/35938a947f3a35f340861772a7f441169deadaa1/src/activation.jl#L1-L6">source</a></section><sectionclass="docstring"><divclass="docstring-header"><aclass="docstring-binding"id="NNlib.relu"href="#NNlib.relu"><code>NNlib.relu</code></a> — <spanclass="docstring-category">Function</span>.</div><div><pre><codeclass="language-none">relu(x) = max(0, x)</code></pre><p><ahref="https://en.wikipedia.org/wiki/Rectifier_(neural_networks)">Rectified Linear Unit</a> activation function.</p></div><aclass="source-link"target="_blank"href="https://github.com/FluxML/NNlib.jl/blob/35938a947f3a35f340861772a7f441169deadaa1/src/activation.jl#L42-L47">source</a></section><sectionclass="docstring"><divclass="docstring-header"><aclass="docstring-binding"id="NNlib.leakyrelu"href="#NNlib.leakyrelu"><code>NNlib.leakyrelu</code></a> — <spanclass="docstring-category">Function</span>.</div><div><pre><codeclass="language-none">leakyrelu(x) = max(0.01x, x)</code></pre><p>Leaky <ahref="https://en.wikipedia.org/wiki/Rectifier_(neural_networks)">Rectified Linear Unit</a> activation function. You can also specify the coefficient explicitly, e.g. <code>leakyrelu(x, 0.01)</code>.</p></div><aclass="source-link"target="_blank"href="https://github.com/FluxML/NNlib.jl/blob/35938a947f3a35f340861772a7f441169deadaa1/src/activation.jl#L51-L57">source</a></section><sectionclass="docstring"><divclass="docstring-header"><aclass="docstring-binding"id="NNlib.elu"href="#NNlib.elu"><code>NNlib.elu</code></a> — <spanclass="docstring-category">Function</span>.</div><div><pre><codeclass="language-none">elu(x, α = 1) =
x > 0 ? x : α * (exp(x) - 1)</code></pre><p>Exponential Linear Unit activation function. See <ahref="https://arxiv.org/abs/1511.07289">Fast and Accurate Deep Network Learning by Exponential Linear Units</a>. You can also specify the coefficient explicitly, e.g. <code>elu(x, 1)</code>.</p></div><aclass="source-link"target="_blank"href="https://github.com/FluxML/NNlib.jl/blob/35938a947f3a35f340861772a7f441169deadaa1/src/activation.jl#L60-L67">source</a></section><sectionclass="docstring"><divclass="docstring-header"><aclass="docstring-binding"id="NNlib.swish"href="#NNlib.swish"><code>NNlib.swish</code></a> — <spanclass="docstring-category">Function</span>.</div><div><pre><codeclass="language-none">swish(x) = x * σ(x)</code></pre><p>Self-gated actvation function. See <ahref="https://arxiv.org/pdf/1710.05941.pdf">Swish: a Self-Gated Activation Function</a>.</p></div><aclass="source-link"target="_blank"href="https://github.com/FluxML/NNlib.jl/blob/35938a947f3a35f340861772a7f441169deadaa1/src/activation.jl#L70-L75">source</a></section><h2><aclass="nav-anchor"id="Normalisation-and-Regularisation-1"href="#Normalisation-and-Regularisation-1">Normalisation & Regularisation</a></h2><p>These layers don't affect the structure of the network but may improve training times or reduce overfitting.</p><sectionclass="docstring"><divclass="docstring-header"><aclass="docstring-binding"id="Flux.testmode!"href="#Flux.testmode!"><code>Flux.testmode!</code></a> — <spanclass="docstring-category">Function</span>.</div><div><pre><codeclass="language-none">testmode!(m)
testmode!(m, false)</code></pre><p>Put layers like <ahref="layers.html#Flux.Dropout"><code>Dropout</code></a> and <ahref="layers.html#Flux.BatchNorm"><code>BatchNorm</code></a> into testing mode (or back to training mode with <code>false</code>).</p></div><aclass="source-link"target="_blank"href="https://github.com/FluxML/Flux.jl/blob/642543808effb5bd79e52c2815646d88ff967c63/src/layers/normalise.jl#L1-L7">source</a></section><sectionclass="docstring"><divclass="docstring-header"><aclass="docstring-binding"id="Flux.BatchNorm"href="#Flux.BatchNorm"><code>Flux.BatchNorm</code></a> — <spanclass="docstring-category">Type</span>.</div><div><pre><codeclass="language-none">BatchNorm(dims...; λ = identity,
initβ = zeros, initγ = ones, ϵ = 1e-8, momentum = .1)</code></pre><p>Batch Normalization Layer for <ahref="layers.html#Flux.Dense"><code>Dense</code></a> layer.</p><p>See <ahref="https://arxiv.org/pdf/1502.03167.pdf">Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift</a></p><p>In the example of MNIST, in order to normalize the input of other layer, put the <code>BatchNorm</code> layer before activation function.</p><pre><codeclass="language-julia">m = Chain(
softmax)</code></pre></div><aclass="source-link"target="_blank"href="https://github.com/FluxML/Flux.jl/blob/642543808effb5bd79e52c2815646d88ff967c63/src/layers/normalise.jl#L70-L91">source</a></section><sectionclass="docstring"><divclass="docstring-header"><aclass="docstring-binding"id="Flux.Dropout"href="#Flux.Dropout"><code>Flux.Dropout</code></a> — <spanclass="docstring-category">Type</span>.</div><div><pre><codeclass="language-none">Dropout(p)</code></pre><p>A Dropout layer. For each input, either sets that input to <code>0</code> (with probability <code>p</code>) or scales it by <code>1/(1-p)</code>. This is used as a regularisation, i.e. it reduces overfitting during training.</p><p>Does nothing to the input once in <ahref="layers.html#Flux.testmode!"><code>testmode!</code></a>.</p></div><aclass="source-link"target="_blank"href="https://github.com/FluxML/Flux.jl/blob/642543808effb5bd79e52c2815646d88ff967c63/src/layers/normalise.jl#L15-L23">source</a></section><sectionclass="docstring"><divclass="docstring-header"><aclass="docstring-binding"id="Flux.LayerNorm"href="#Flux.LayerNorm"><code>Flux.LayerNorm</code></a> — <spanclass="docstring-category">Type</span>.</div><div><pre><codeclass="language-none">LayerNorm(h::Integer)</code></pre><p>A <ahref="https://arxiv.org/pdf/1607.06450.pdf">normalisation layer</a> designed to be used with recurrent hidden states of size <code>h</code>. Normalises the mean/stddev of each input before applying a per-neuron gain/bias.</p></div><aclass="source-link"target="_blank"href="https://github.com/FluxML/Flux.jl/blob/642543808effb5bd79e52c2815646d88ff967c63/src/layers/normalise.jl#L47-L54">source</a></section><footer><hr/><aclass="previous"href="regularisation.html"><spanclass="direction">Previous</span><spanclass="title">Regularisation</span></a><aclass="next"href="../training/optimisers.html"><spanclass="direction">Next</span><spanclass="title">Optimisers</span></a></footer></article></body></html>
