</script><linkhref="https://fonts.googleapis.com/css?family=Lato|Roboto+Mono"rel="stylesheet"type="text/css"/><linkhref="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.11.2/css/fontawesome.min.css"rel="stylesheet"type="text/css"/><linkhref="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.11.2/css/solid.min.css"rel="stylesheet"type="text/css"/><linkhref="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.11.2/css/brands.min.css"rel="stylesheet"type="text/css"/><linkhref="https://cdnjs.cloudflare.com/ajax/libs/KaTeX/0.11.1/katex.min.css"rel="stylesheet"type="text/css"/><script>documenterBaseURL="../.."</script><scriptsrc="https://cdnjs.cloudflare.com/ajax/libs/require.js/2.3.6/require.min.js"data-main="../../assets/documenter.js"></script><scriptsrc="../../siteinfo.js"></script><scriptsrc="../../../versions.js"></script><linkhref="../../assets/flux.css"rel="stylesheet"type="text/css"/><linkclass="docs-theme-link"rel="stylesheet"type="text/css"href="../../assets/themes/documenter-dark.css"data-theme-name="documenter-dark"/><linkclass="docs-theme-link"rel="stylesheet"type="text/css"href="../../assets/themes/documenter-light.css"data-theme-name="documenter-light"data-theme-primary/><scriptsrc="../../assets/themeswap.js"></script></head><body><divid="documenter"><navclass="docs-sidebar"><divclass="docs-package-name"><spanclass="docs-autofit">Flux</span></div><formclass="docs-search"action="../../search/"><inputclass="docs-search-query"id="documenter-search-query"name="q"type="text"placeholder="Search docs"/></form><ulclass="docs-menu"><li><aclass="tocitem"href="../../">Home</a></li><li><spanclass="tocitem">Building Models</span><ul><li><aclass="tocitem"href="../basics/">Basics</a></li><liclass="is-active"><aclass="tocitem"href>Recurrence</a><ulclass="internal"><li><aclass="tocitem"href="#Recurrent-Cells-1"><span>Recurrent Cells</span></a></li><li><aclass="tocitem"href="#Stateful-Models-1"><span>Stateful Models</span></a></li><li><aclass="tocitem"href="#Sequences-1"><span>Sequences</span></a></li></ul></li><li><aclass="tocitem"href="../regularisation/">Regularisation</a></li><li><aclass="tocitem"href="../layers/">Model Reference</a></li><li><aclass="tocitem"href="../advanced/">Advanced Model Building</a></li><li><aclass="tocitem"href="../nnlib/">NNlib</a></li></ul></li><li><spanclass="tocitem">Handling Data</span><ul><li><aclass="tocitem"href="../../data/onehot/">One-Hot Encoding</a></li><li><aclass="tocitem"href="../../data/dataloader/">DataLoader</a></li></ul></li><li><spanclass="tocitem">Training Models</span><ul><li><aclass="tocitem"href="../../training/optimisers/">Optimisers</a></li><li><aclass="tocitem"href="../../training/training/">Training</a></li></ul></li><li><aclass="tocitem"href="../../gpu/">GPU Support</a></li><li><aclass="tocitem"href="../../saving/">Saving & Loading</a></li><li><aclass="tocitem"href="../../ecosystem/">The Julia Ecosystem</a></li><li><aclass="tocitem"href="../../performance/">Performance Tips</a></li><li><aclass="tocitem"href="../../community/">Community</a></li></ul><divclass="docs-version-selector field has-addons"><divclass="control"><spanclass="docs-label button is-static is-size-7">Version</span></div><divclass="docs-selector control is-expanded"><divclass="select is-fullwidth is-size-7"><selectid="documenter-version-selector"></select></div></div></div></nav><divclass="docs-main"><headerclass="docs-navbar"><navclass="breadcrumb"><ulclass="is-hidden-mobile"><li><aclass="is-disabled">Building Models</a></li><liclass="is-active"><ahref>Recurrence</a></li></ul><ulclass="is-hidden-tablet"><liclass="is-active"><ahref>Recurrence</a></li></ul></nav><divclass="docs-right"><aclass="docs-edit-link"href="https://github.com/FluxML/Flux.jl/blob/master/docs/src/models/recurrence.md"title="Edit on GitHub"><spanclass="docs-icon fab"></span><spanclass="docs-label is-hidden-touch">Edit on GitHub</span></a><aclass="docs-settings-button fas fa-cog"id="documenter-settings-butt
# ...</code></pre><p>Recurrent networks introduce a <em>hidden state</em> that gets carried over each time we run the model. The model now takes the old <code>h</code> as an input, and produces a new <code>h</code> as output, each time we run it.</p><pre><codeclass="language-julia">h = # ... initial state ...
# ...</code></pre><p>Information stored in <code>h</code> is preserved for the next prediction, allowing it to function as a kind of memory. This also means that the prediction made for a given <code>x</code> depends on all the inputs previously fed into the model.</p><p>(This might be important if, for example, each <code>x</code> represents one word of a sentence; the model's interpretation of the word "bank" should change if the previous input was "river" rather than "investment".)</p><p>Flux's RNN support closely follows this mathematical perspective. The most basic RNN is as close as possible to a standard <code>Dense</code> layer, and the output is also the hidden state.</p><pre><codeclass="language-julia">Wxh = randn(5, 10)
h, y = rnn(h, x)</code></pre><p>If you run the last line a few times, you'll notice the output <code>y</code> changing slightly even though the input <code>x</code> is the same.</p><p>We sometimes refer to functions like <code>rnn</code> above, which explicitly manage state, as recurrent <em>cells</em>. There are various recurrent cells available, which are documented in the <ahref="../layers/">layer reference</a>. The hand-written example above can be replaced with:</p><pre><codeclass="language-julia">using Flux
h, y = rnn2(h, x)</code></pre><h2id="Stateful-Models-1"><aclass="docs-heading-anchor"href="#Stateful-Models-1">Stateful Models</a><aclass="docs-heading-anchor-permalink"href="#Stateful-Models-1"title="Permalink"></a></h2><p>For the most part, we don't want to manage hidden states ourselves, but to treat our models as being stateful. Flux provides the <code>Recur</code> wrapper to do this.</p><pre><codeclass="language-julia">x = rand(10)
y = m(x)</code></pre><p>The <code>Recur</code> wrapper stores the state between runs in the <code>m.state</code> field.</p><p>If you use the <code>RNN(10, 5)</code> constructor – as opposed to <code>RNNCell</code>– you'll see that it's simply a wrapped cell.</p><pre><codeclass="language-julia">julia> RNN(10, 5)
Recur(RNNCell(10, 5, tanh))</code></pre><h2id="Sequences-1"><aclass="docs-heading-anchor"href="#Sequences-1">Sequences</a><aclass="docs-heading-anchor-permalink"href="#Sequences-1"title="Permalink"></a></h2><p>Often we want to work with sequences of inputs, rather than individual <code>x</code>s.</p><pre><codeclass="language-julia">seq = [rand(10) for i = 1:10]</code></pre><p>With <code>Recur</code>, applying our model to each element of a sequence is trivial:</p><pre><codeclass="language-julia">m.(seq) # returns a list of 5-element vectors</code></pre><p>This works even when we've chain recurrent layers into a larger model.</p><pre><codeclass="language-julia">m = Chain(LSTM(10, 15), Dense(15, 5))
m.(seq)</code></pre><p>Finally, we can reset the hidden state of the cell back to its initial value using <code>reset!(m)</code>.</p></article><navclass="docs-footer"><aclass="docs-footer-prevpage"href="../basics/">« Basics</a><aclass="docs-footer-nextpage"href="../regularisation/">Regularisation »</a></nav></div><divclass="modal"id="documenter-settings"><divclass="modal-background"></div><divclass="modal-card"><headerclass="modal-card-head"><pclass="modal-card-title">Settings</p><buttonclass="delete"></button></header><sectionclass="modal-card-body"><p><labelclass="label">Theme</label><divclass="select"><selectid="documenter-themepicker"><optionvalue="documenter-light">documenter-light</option><optionvalue="documenter-dark">documenter-dark</option></select></div></p><hr/><p>This document was generated with <ahref="https://github.com/JuliaDocs/Documenter.jl">Documenter.jl</a> on <spanclass="colophon-date"title="Wednesday 4 March 2020 00:45">Wednesday 4 March 2020</span>. Using Julia version 1.3.1.</p></section><footerclass="modal-card-foot"></footer></div></div></div></body></html>