# ...</code></pre><p>Recurrent networks introduce a <em>hidden state</em> that gets carried over each time we run the model. The model now takes the old <code>h</code> as an input, and produces a new <code>h</code> as output, each time we run it.</p><pre><codeclass="language-julia">h = # ... initial state ...
# ...</code></pre><p>Information stored in <code>h</code> is preserved for the next prediction, allowing it to function as a kind of memory. This also means that the prediction made for a given <code>x</code> depends on all the inputs previously fed into the model.</p><p>(This might be important if, for example, each <code>x</code> represents one word of a sentence; the model's interpretation of the word "bank" should change if the previous input was "river" rather than "investment".)</p><p>Flux's RNN support closely follows this mathematical perspective. The most basic RNN is as close as possible to a standard <code>Dense</code> layer, and the output is also the hidden state.</p><pre><codeclass="language-julia">Wxh = randn(5, 10)
h, y = rnn(h, x)</code></pre><p>If you run the last line a few times, you'll notice the output <code>y</code> changing slightly even though the input <code>x</code> is the same.</p><p>We sometimes refer to functions like <code>rnn</code> above, which explicitly manage state, as recurrent <em>cells</em>. There are various recurrent cells available, which are documented in the <ahref="../layers/">layer reference</a>. The hand-written example above can be replaced with:</p><pre><codeclass="language-julia">using Flux
h, y = rnn2(h, x)</code></pre><h2id="Stateful-Models-1"><aclass="docs-heading-anchor"href="#Stateful-Models-1">Stateful Models</a><aclass="docs-heading-anchor-permalink"href="#Stateful-Models-1"title="Permalink"></a></h2><p>For the most part, we don't want to manage hidden states ourselves, but to treat our models as being stateful. Flux provides the <code>Recur</code> wrapper to do this.</p><pre><codeclass="language-julia">x = rand(10)
y = m(x)</code></pre><p>The <code>Recur</code> wrapper stores the state between runs in the <code>m.state</code> field.</p><p>If you use the <code>RNN(10, 5)</code> constructor – as opposed to <code>RNNCell</code>– you'll see that it's simply a wrapped cell.</p><pre><codeclass="language-julia">julia> RNN(10, 5)
Recur(RNNCell(10, 5, tanh))</code></pre><h2id="Sequences-1"><aclass="docs-heading-anchor"href="#Sequences-1">Sequences</a><aclass="docs-heading-anchor-permalink"href="#Sequences-1"title="Permalink"></a></h2><p>Often we want to work with sequences of inputs, rather than individual <code>x</code>s.</p><pre><codeclass="language-julia">seq = [rand(10) for i = 1:10]</code></pre><p>With <code>Recur</code>, applying our model to each element of a sequence is trivial:</p><pre><codeclass="language-julia">m.(seq) # returns a list of 5-element vectors</code></pre><p>This works even when we've chain recurrent layers into a larger model.</p><pre><codeclass="language-julia">m = Chain(LSTM(10, 15), Dense(15, 5))
m.(seq)</code></pre><p>Finally, we can reset the hidden state of the cell back to its initial value using <code>reset!(m)</code>.</p></article><navclass="docs-footer"><aclass="docs-footer-prevpage"href="../basics/">« Basics</a><aclass="docs-footer-nextpage"href="../regularisation/">Regularisation »</a></nav></div><divclass="modal"id="documenter-settings"><divclass="modal-background"></div><divclass="modal-card"><headerclass="modal-card-head"><pclass="modal-card-title">Settings</p><buttonclass="delete"></button></header><sectionclass="modal-card-body"><p><labelclass="label">Theme</label><divclass="select"><selectid="documenter-themepicker"><optionvalue="documenter-light">documenter-light</option><optionvalue="documenter-dark">documenter-dark</option></select></div></p><hr/><p>This document was generated with <ahref="https://github.com/JuliaDocs/Documenter.jl">Documenter.jl</a> on <spanclass="colophon-date"title="Monday 6 April 2020 14:20">Monday 6 April 2020</span>. Using Julia version 1.4.0.</p></section><footerclass="modal-card-foot"></footer></div></div></div></body></html>
