</script><linkhref="https://cdnjs.cloudflare.com/ajax/libs/normalize/4.2.0/normalize.min.css"rel="stylesheet"type="text/css"/><linkhref="https://fonts.googleapis.com/css?family=Lato|Roboto+Mono"rel="stylesheet"type="text/css"/><linkhref="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/4.6.3/css/font-awesome.min.css"rel="stylesheet"type="text/css"/><linkhref="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"rel="stylesheet"type="text/css"/><script>documenterBaseURL="../.."</script><scriptsrc="https://cdnjs.cloudflare.com/ajax/libs/require.js/2.2.0/require.min.js"data-main="../../assets/documenter.js"></script><scriptsrc="../../siteinfo.js"></script><scriptsrc="../../../versions.js"></script><linkhref="../../assets/documenter.css"rel="stylesheet"type="text/css"/><linkhref="../../assets/flux.css"rel="stylesheet"type="text/css"/></head><body><navclass="toc"><h1>Flux</h1><selectid="version-selector"onChange="window.location.href=this.value"style="visibility: hidden"></select><formclass="search"id="search-form"action="../../search/"><inputid="search-query"name="q"type="text"placeholder="Search docs"/></form><ul><li><aclass="toctext"href="../../">Home</a></li><li><spanclass="toctext">Building Models</span><ul><li><aclass="toctext"href="../basics/">Basics</a></li><li><aclass="toctext"href="../recurrence/">Recurrence</a></li><liclass="current"><aclass="toctext"href>Regularisation</a><ulclass="internal"></ul></li><li><aclass="toctext"href="../layers/">Model Reference</a></li><li><aclass="toctext"href="../nnlib/">NNlib</a></li></ul></li><li><spanclass="toctext">Handling Data</span><ul><li><aclass="toctext"href="../../data/onehot/">One-Hot Encoding</a></li><li><aclass="toctext"href="../../data/dataloader/">DataLoader</a></li></ul></li><li><spanclass="toctext">Training Models</span><ul><li><aclass="toctext"href="../../training/optimisers/">Optimisers</a></li><li><aclass="toctext"href="../../training/training/">Training</a></li></ul></li><li><aclass="toctext"href="../../gpu/">GPU Support</a></li><li><aclass="toctext"href="../../saving/">Saving & Loading</a></li><li><aclass="toctext"href="../../ecosystem/">The Julia Ecosystem</a></li><li><aclass="toctext"href="../../performance/">Performance Tips</a></li><li><aclass="toctext"href="../../community/">Community</a></li></ul></nav><articleid="docs"><header><nav><ul><li>Building Models</li><li><ahref>Regularisation</a></li></ul><aclass="edit-page"href="https://github.com/FluxML/Flux.jl/blob/master/docs/src/models/regularisation.md"><spanclass="fa"></span> Edit on GitHub</a></nav><hr/><divid="topbar"><span>Regularisation</span><aclass="fa fa-bars"href="#"></a></div></header><h1><aclass="nav-anchor"id="Regularisation-1"href="#Regularisation-1">Regularisation</a></h1><p>Applying regularisation to model parameters is straightforward. We just need to apply an appropriate regulariser, such as <code>norm</code>, to each model parameter and add the result to the overall loss.</p><p>For example, say we have a simple regression.</p><pre><codeclass="language-julia">using Flux: crossentropy
loss(x, y) = crossentropy(softmax(m(x)), y)</code></pre><p>We can regularise this by taking the (L2) norm of the parameters, <code>m.W</code> and <code>m.b</code>.</p><pre><codeclass="language-julia">using LinearAlgebra
loss(x, y) = crossentropy(softmax(m(x)), y) + penalty()</code></pre><p>When working with layers, Flux provides the <code>params</code> function to grab all parameters at once. We can easily penalise everything with <code>sum(norm, params)</code>.</p><pre><codeclass="language-julia">julia> params(m)
loss(rand(28^2), rand(10))</code></pre><p>One can also easily add per-layer regularisation via the <code>activations</code> function:</p><pre><codeclass="language-julia">julia> using Flux: activations