From d68866a2385cb608ff5b87ab98c3efba4ef93e48 Mon Sep 17 00:00:00 2001 From: Michael Green Date: Wed, 27 Mar 2019 20:22:01 +0100 Subject: [PATCH 1/3] Fixed documentation error. --- docs/src/models/recurrence.md | 12 +----------- 1 file changed, 1 insertion(+), 11 deletions(-) diff --git a/docs/src/models/recurrence.md b/docs/src/models/recurrence.md index 8030f4c4..f796ff92 100644 --- a/docs/src/models/recurrence.md +++ b/docs/src/models/recurrence.md @@ -77,7 +77,7 @@ If you use the `RNN(10, 5)` constructor – as opposed to `RNNCell` – you'll s ```julia julia> RNN(10, 5) -Recur(RNNCell(Dense(15, 5))) +Recur(RNNCell(Dense(10, 5))) ``` ## Sequences @@ -114,13 +114,3 @@ truncate!(m) Calling `truncate!` wipes the slate clean, so we can call the model with more inputs without building up an expensive gradient computation. `truncate!` makes sense when you are working with multiple chunks of a large sequence, but we may also want to work with a set of independent sequences. In this case the hidden state should be completely reset to its original value, throwing away any accumulated information. `reset!` does this for you. - -In general, when training with recurrent layers in your model, you'll want to call `reset!` or `truncate!` for each loss calculation: - -```julia -function loss(x,y) - l = Flux.mse(m(x), y) - Flux.reset!(m) - return l -end -``` From a5c34e832550f203200948a1cca0d0ac64b98832 Mon Sep 17 00:00:00 2001 From: Michael Green Date: Wed, 27 Mar 2019 20:30:31 +0100 Subject: [PATCH 2/3] Fixed merging with upstream Flux. --- docs/src/models/recurrence.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/docs/src/models/recurrence.md b/docs/src/models/recurrence.md index f796ff92..a40a509c 100644 --- a/docs/src/models/recurrence.md +++ b/docs/src/models/recurrence.md @@ -114,3 +114,13 @@ truncate!(m) Calling `truncate!` wipes the slate clean, so we can call the model with more inputs without building up an expensive gradient computation. `truncate!` makes sense when you are working with multiple chunks of a large sequence, but we may also want to work with a set of independent sequences. In this case the hidden state should be completely reset to its original value, throwing away any accumulated information. `reset!` does this for you. + +In general, when training with recurrent layers in your model, you'll want to call `reset!` or `truncate!` for each loss calculation: + +```julia +function loss(x,y) + l = Flux.mse(m(x), y) + Flux.reset!(m) + return l +end +``` From 934f7f932d37d70252c83a3e3f8c4507a6e46439 Mon Sep 17 00:00:00 2001 From: Michael Green Date: Sat, 20 Apr 2019 11:22:48 +0200 Subject: [PATCH 3/3] Updated docs again. --- docs/src/models/recurrence.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/src/models/recurrence.md b/docs/src/models/recurrence.md index a40a509c..1ae7cbd8 100644 --- a/docs/src/models/recurrence.md +++ b/docs/src/models/recurrence.md @@ -77,7 +77,7 @@ If you use the `RNN(10, 5)` constructor – as opposed to `RNNCell` – you'll s ```julia julia> RNN(10, 5) -Recur(RNNCell(Dense(10, 5))) +Recur(RNNCell(10, 5, tanh)) ``` ## Sequences