diff --git a/latest/community.html b/latest/community.html index 73e81d97..3eb998c3 100644 --- a/latest/community.html +++ b/latest/community.html @@ -6,4 +6,4 @@ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) ga('create', 'UA-36890222-9', 'auto'); ga('send', 'pageview'); -
Community

Community

All Flux users are welcome to join our community on the Julia forum, the slack (channel #machine-learning), or Flux's Gitter. If you have questions or issues we'll try to help you out.

If you're interested in hacking on Flux, the source code is open and easy to understand – it's all just the same Julia code you work with normally. You might be interested in our intro issues to get started.

+
Community

Community

All Flux users are welcome to join our community on the Julia forum, the slack (channel #machine-learning), or Flux's Gitter. If you have questions or issues we'll try to help you out.

If you're interested in hacking on Flux, the source code is open and easy to understand – it's all just the same Julia code you work with normally. You might be interested in our intro issues to get started.

diff --git a/latest/data/onehot.html b/latest/data/onehot.html index fbbf7dd3..bde611e1 100644 --- a/latest/data/onehot.html +++ b/latest/data/onehot.html @@ -6,7 +6,7 @@ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) ga('create', 'UA-36890222-9', 'auto'); ga('send', 'pageview'); -
One-Hot Encoding

One-Hot Encoding

It's common to encode categorical variables (like true, false or cat, dog) in "one-of-k" or "one-hot" form. Flux provides the onehot function to make this easy.

julia> using Flux: onehot
+
One-Hot Encoding
One-Hot Encoding
It's common to encode categorical variables (like true, false or cat, dog) in "one-of-k" or "one-hot" form. Flux provides the onehot function to make this easy.
julia> using Flux: onehot
 
 julia> onehot(:b, [:a, :b, :c])
 3-element Flux.OneHotVector:
diff --git a/latest/gpu.html b/latest/gpu.html
index 0de021b6..e33c0e6e 100644
--- a/latest/gpu.html
+++ b/latest/gpu.html
@@ -6,7 +6,7 @@ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
 
 ga('create', 'UA-36890222-9', 'auto');
 ga('send', 'pageview');
-
GPU Support
GPU Support
Support for array operations on other hardware backends, like GPUs, is provided by external packages like CuArrays. Flux is agnostic to array types, so we simply need to move model weights and data to the GPU and Flux will handle it.
For example, we can use CuArrays (with the cu converter) to run our basic example on an NVIDIA GPU.
using CuArrays
+
GPU Support
GPU Support
Support for array operations on other hardware backends, like GPUs, is provided by external packages like CuArrays. Flux is agnostic to array types, so we simply need to move model weights and data to the GPU and Flux will handle it.
For example, we can use CuArrays (with the cu converter) to run our basic example on an NVIDIA GPU.
using CuArrays
 
 W = cu(rand(2, 5)) # a 2×5 CuArray
 b = cu(rand(2))
diff --git a/latest/index.html b/latest/index.html
index 81e39dd4..bb6733e3 100644
--- a/latest/index.html
+++ b/latest/index.html
@@ -6,7 +6,7 @@ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
 
 ga('create', 'UA-36890222-9', 'auto');
 ga('send', 'pageview');
-
Home
Flux: The Julia Machine Learning Library
Flux is a library for machine learning. It comes "batteries-included" with many useful tools built in, but also lets you use the full power of the Julia language where you need it. The whole stack is implemented in clean Julia code (right down to the GPU kernels) and any part can be tweaked to your liking.
Installation
Install Julia 0.6.0 or later, if you haven't already.
Pkg.add("Flux")
+
Home
Flux: The Julia Machine Learning Library
Flux is a library for machine learning. It comes "batteries-included" with many useful tools built in, but also lets you use the full power of the Julia language where you need it. The whole stack is implemented in clean Julia code (right down to the GPU kernels) and any part can be tweaked to your liking.
Installation
Install Julia 0.6.0 or later, if you haven't already.
Pkg.add("Flux")
 # Optional but recommended
 Pkg.update() # Keep your packages up to date
 Pkg.test("Flux") # Check things installed correctly
Start with the basics. The model zoo is also a good starting point for many common kinds of models.

diff --git a/latest/internals/tracker.html b/latest/internals/tracker.html
new file mode 100644
index 00000000..65ba7e58
--- /dev/null
+++ b/latest/internals/tracker.html
@@ -0,0 +1,68 @@
+
+Backpropagation · Flux
Backpropagation
Flux.Tracker
Backpropagation, or reverse-mode automatic differentiation, is handled by the Flux.Tracker module.
julia> using Flux.Tracker
The param function converts a normal Julia array into a new object that, while behaving like an array, tracks extra information that allows us to calculate derivatives. For example, say we multiply two parameters:
julia> W = param([1 2; 3 4])
+Tracked 2×2 Array{Float64,2}:
+ 1.0  2.0
+ 3.0  4.0
+
+julia> x = param([5, 6])
+Tracked 2-element Array{Float64,1}:
+ 5.0
+ 6.0
+
+julia> y = W*x
+Tracked 2-element Array{Float64,1}:
+ 17.0
+ 39.0
The output y is also a TrackedArray object. We can now backpropagate sensitivities to W and x via the back! function, and see the gradients accumulated in the W and x tracked arrays:
julia> Tracker.back!(y, [1, -1])
+
+julia> W.grad
+2×2 Array{Float64,2}:
+ 5.0   6.0
+-5.0  -6.0
+
+julia> x.grad
+2-element Array{Float64,1}:
+ -2.0
+ -2.0
Internals
All Tracked* objects (TrackedArray, TrackedReal) are light wrappers around the Tracked type, which you can access via the .tracker field.
julia> x.tracker
+Flux.Tracker.Tracked{Array{Float64,1}}(0x00000000, Flux.Tracker.Call{Void,Tuple{}}(nothing, ()), true, [5.0, 6.0], [-2.0, -2.0])
The Tracker stores the value and gradient of a given object, which we've seen before.
julia> x.tracker.data
+2-element Array{Float64,1}:
+ 5.0
+ 6.0
+
+julia> x.tracker.grad
+2-element Array{Float64,1}:
+ -2.0
+ -2.0
The tracker also contains a Call object, which simply represents a function call that was made at some point during the forward pass. For example, the + call would look like this:
julia> Tracker.Call(+, 1, 2)
+Flux.Tracker.Call{Base.#+,Tuple{Int64,Int64}}(+, (1, 2))
In the case of the y we produced above, we can see that it stores the call that produced it – that is, W*x.
julia> y.tracker.f
+Flux.Tracker.Call{...}(*, (param([1.0 2.0; 3.0 4.0]), param([5.0, 6.0])))
Notice that because the arguments to the call may also be tracked arrays, storing their own calls, this means that Tracker ends up forming a data structure that records everything that happened during the forward pass (often known as a tape).
When we call back!(y, [1, -1]), the sensitivities [1, -1] simply get forwarded to y's call (*), effectively calling
Tracker.back(*, [1, -1], W, x)
which in turn calculates the sensitivities of the arguments (W and x) and backpropagates through their calls. This is recursive, so it will walk the entire program graph and propagate gradients to the original model parameters.
Custom Gradients
We can hook in to the processes above to implement custom gradients for a function or kernel. For a toy example, imagine a custom implementation of minus:
julia> minus(a, b) = a - b
Firstly, we must tell the tracker system to stop when it sees a call to minus, and record it. We can do this using dispatch:
julia> minus(a::TrackedArray, b::TrackedArray) = Tracker.track(minus, a, b)
+minus (generic function with 2 methods)
Tracker.track does two things: (1) it makes sure minus is called with normal array, not tracked ones (you can use @show inside minus to verify this), and (2) it uses the result to add a minus node to the tape. Look inside the result of calling minus to see what happened:
julia> a, b = param([6,5,4]), param([1,2,3])
+(param([6.0, 5.0, 4.0]), param([1.0, 2.0, 3.0]))
+
+julia> c = minus(a, b)
+Tracked 3-element Array{Float64,1}:
+ 5.0
+ 3.0
+ 1.0
+
+julia> c.tracker.f
+Flux.Tracker.Call{...}(minus, (param([6.0, 5.0, 4.0]), param([1.0, 2.0, 3.0])))
Finally, we have to specify the gradient of minus.
julia> Tracker.back(::typeof(minus), Δ, a, b) =
+        (Tracker.@back(a, Δ); Tracker.@back(b, -Δ))
@back(x, Δ) tells the tracker to continue propagating the sensitivity Δ through x. Now, AD will work with any program that calls minus.
julia> Flux.back!(c, 1)
+
+julia> a.grad
+3-element Array{Float64,1}:
+ 1.0
+ 1.0
+ 1.0
+
+julia> b.grad
+3-element Array{Float64,1}:
+ -1.0
+ -1.0
+ -1.0
Notes
For multi-argument functions with custom gradients, you likely want to catch not just minus(::TrackedArray, ::TrackedArray) but also minus(::Array, TrackedArray) and so on. To do so, just define those extra signatures as needed:
minus(a::AbstractArray, b::TrackedArray) = Tracker.track(minus, a, b)
+minus(a::TrackedArray, b::AbstractArray) = Tracker.track(minus, a, b)
@back must be called exactly once on each tracked input argument. You do not need to do any special handling if one of the arguments is not tracked, as @back will just become a no-op.

diff --git a/latest/models/basics.html b/latest/models/basics.html
index b9e05db6..a9e28f0f 100644
--- a/latest/models/basics.html
+++ b/latest/models/basics.html
@@ -6,7 +6,7 @@ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
 
 ga('create', 'UA-36890222-9', 'auto');
 ga('send', 'pageview');
-
Basics
Model-Building Basics
Taking Gradients
Consider a simple linear regression, which tries to predict an output array y from an input x. (It's a good idea to follow this example in the Julia repl.)
W = rand(2, 5)
+
Basics
Model-Building Basics
Taking Gradients
Consider a simple linear regression, which tries to predict an output array y from an input x. (It's a good idea to follow this example in the Julia repl.)
W = rand(2, 5)
 b = rand(2)
 
 predict(x) = W*x .+ b
diff --git a/latest/models/layers.html b/latest/models/layers.html
index 7eb2263a..6eae3a62 100644
--- a/latest/models/layers.html
+++ b/latest/models/layers.html
@@ -6,31 +6,31 @@ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
 
 ga('create', 'UA-36890222-9', 'auto');
 ga('send', 'pageview');
-
Model Reference
Basic Layers
These core layers form the foundation of almost all neural networks.
Flux.Chain — Type.
Chain(layers...)
Chain multiple layers / functions together, so that they are called in sequence on a given input.
m = Chain(x -> x^2, x -> x+1)
+
Model Reference
Basic Layers
These core layers form the foundation of almost all neural networks.
Flux.Chain — Type.
Chain(layers...)
Chain multiple layers / functions together, so that they are called in sequence on a given input.
m = Chain(x -> x^2, x -> x+1)
 m(5) == 26
 
 m = Chain(Dense(10, 5), Dense(5, 2))
 x = rand(10)
-m(x) == m[2](m[1](x))
Chain also supports indexing and slicing, e.g. m[2] or m[1:end-1]. m[1:3](x) will calculate the output of the first three layers.
source
Flux.Dense — Type.
Dense(in::Integer, out::Integer, σ = identity)
Creates a traditional Dense layer with parameters W and b.
y = σ.(W * x .+ b)
The input x must be a vector of length in, or a batch of vectors represented as an in × N matrix. The out y will be a vector or batch of length out.
julia> d = Dense(5, 2)
+m(x) == m[2](m[1](x))
Chain also supports indexing and slicing, e.g. m[2] or m[1:end-1]. m[1:3](x) will calculate the output of the first three layers.
source
Flux.Dense — Type.
Dense(in::Integer, out::Integer, σ = identity)
Creates a traditional Dense layer with parameters W and b.
y = σ.(W * x .+ b)
The input x must be a vector of length in, or a batch of vectors represented as an in × N matrix. The out y will be a vector or batch of length out.
julia> d = Dense(5, 2)
 Dense(5, 2)
 
 julia> d(rand(5))
 Tracked 2-element Array{Float64,1}:
   0.00257447
-  -0.00449443
source
Flux.Conv — Type.
Conv(size, in=>out)
-Conv(size, in=>out, relu)
Standard convolutional layer. size should be a tuple like (2, 2). in and out specify the number of input and output channels respectively.
Data should be stored in WHCN order. In other words, a 100×100 RGB image would be a 100×100×3 array, and a batch of 50 would be a 100×100×3×50 array.
Takes the keyword arguments pad and stride.
source
Recurrent Layers
Much like the core layers above, but can be used to process sequence data (as well as other kinds of structured data).
Flux.RNN — Function.
RNN(in::Integer, out::Integer, σ = tanh)
The most basic recurrent layer; essentially acts as a Dense layer, but with the output fed back into the input each time step.
source
Flux.LSTM — Function.
LSTM(in::Integer, out::Integer, σ = tanh)
Long Short Term Memory recurrent layer. Behaves like an RNN but generally exhibits a longer memory span over sequences.
See this article for a good overview of the internals.
source
Flux.GRU — Function.
GRU(in::Integer, out::Integer, σ = tanh)
Gated Recurrent Unit layer. Behaves like an RNN but generally exhibits a longer memory span over sequences.
See this article for a good overview of the internals.
source
Flux.Recur — Type.
Recur(cell)
Recur takes a recurrent cell and makes it stateful, managing the hidden state in the background. cell should be a model of the form:
h, y = cell(h, x...)
For example, here's a recurrent network that keeps a running total of its inputs.
accum(h, x) = (h+x, x)
+  -0.00449443
source
Flux.Conv — Type.
Conv(size, in=>out)
+Conv(size, in=>out, relu)
Standard convolutional layer. size should be a tuple like (2, 2). in and out specify the number of input and output channels respectively.
Data should be stored in WHCN order. In other words, a 100×100 RGB image would be a 100×100×3 array, and a batch of 50 would be a 100×100×3×50 array.
Takes the keyword arguments pad and stride.
source
Recurrent Layers
Much like the core layers above, but can be used to process sequence data (as well as other kinds of structured data).
Flux.RNN — Function.
RNN(in::Integer, out::Integer, σ = tanh)
The most basic recurrent layer; essentially acts as a Dense layer, but with the output fed back into the input each time step.
source
Flux.LSTM — Function.
LSTM(in::Integer, out::Integer, σ = tanh)
Long Short Term Memory recurrent layer. Behaves like an RNN but generally exhibits a longer memory span over sequences.
See this article for a good overview of the internals.
source
Flux.GRU — Function.
GRU(in::Integer, out::Integer, σ = tanh)
Gated Recurrent Unit layer. Behaves like an RNN but generally exhibits a longer memory span over sequences.
See this article for a good overview of the internals.
source
Flux.Recur — Type.
Recur(cell)
Recur takes a recurrent cell and makes it stateful, managing the hidden state in the background. cell should be a model of the form:
h, y = cell(h, x...)
For example, here's a recurrent network that keeps a running total of its inputs.
accum(h, x) = (h+x, x)
 rnn = Flux.Recur(accum, 0)
 rnn(2) # 2
 rnn(3) # 3
 rnn.state # 5
 rnn.(1:10) # apply to a sequence
-rnn.state # 60
source
Activation Functions
Non-linearities that go between layers of your model. Most of these functions are defined in NNlib but are available by default in Flux.
Note that, unless otherwise stated, activation functions operate on scalars. To apply them to an array you can call σ.(xs), relu.(xs) and so on.
NNlib.σ — Function.
σ(x) = 1 / (1 + exp(-x))
Classic sigmoid activation function.
source
NNlib.relu — Function.
relu(x) = max(0, x)
Rectified Linear Unit activation function.
source
NNlib.leakyrelu — Function.
leakyrelu(x) = max(0.01x, x)
Leaky Rectified Linear Unit activation function. You can also specify the coefficient explicitly, e.g. leakyrelu(x, 0.01).
source
NNlib.elu — Function.
elu(x, α = 1) =
+rnn.state # 60
source
Activation Functions
Non-linearities that go between layers of your model. Most of these functions are defined in NNlib but are available by default in Flux.
Note that, unless otherwise stated, activation functions operate on scalars. To apply them to an array you can call σ.(xs), relu.(xs) and so on.
NNlib.σ — Function.
σ(x) = 1 / (1 + exp(-x))
Classic sigmoid activation function.
source
NNlib.relu — Function.
relu(x) = max(0, x)
Rectified Linear Unit activation function.
source
NNlib.leakyrelu — Function.
leakyrelu(x) = max(0.01x, x)
Leaky Rectified Linear Unit activation function. You can also specify the coefficient explicitly, e.g. leakyrelu(x, 0.01).
source
NNlib.elu — Function.
elu(x, α = 1) =
   x > 0 ? x : α * (exp(x) - 1)
Exponential Linear Unit activation function. See Fast and Accurate Deep Network Learning by Exponential Linear Units. You can also specify the coefficient explicitly, e.g. elu(x, 1).
source
NNlib.swish — Function.
swish(x) = x * σ(x)
Self-gated actvation function. See Swish: a Self-Gated Activation Function.
source
Normalisation & Regularisation
These layers don't affect the structure of the network but may improve training times or reduce overfitting.
Flux.testmode! — Function.
testmode!(m)
-testmode!(m, false)
Put layers like Dropout and BatchNorm into testing mode (or back to training mode with false).
source
Flux.BatchNorm — Type.
BatchNorm(channels::Integer, σ = identity;
+testmode!(m, false)
Put layers like Dropout and BatchNorm into testing mode (or back to training mode with false).
source
Flux.BatchNorm — Type.
BatchNorm(channels::Integer, σ = identity;
           initβ = zeros, initγ = ones,
           ϵ = 1e-8, momentum = .1)
Batch Normalization layer. The channels input should be the size of the channel dimension in your data (see below).
Given an array with N dimensions, call the N-1th the channel dimension. (For a batch of feature vectors this is just the data dimension, for WHCN images it's the usual channel dimension.)
BatchNorm computes the mean and variance for each each W×H×1×N slice and shifts them to have a new mean and variance (corresponding to the learnable, per-channel bias and scale parameters).
See Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift.
Example:
m = Chain(
   Dense(28^2, 64),
   BatchNorm(64, relu),
   Dense(64, 10),
   BatchNorm(10),
-  softmax)
source
Flux.Dropout — Type.
Dropout(p)
A Dropout layer. For each input, either sets that input to 0 (with probability p) or scales it by 1/(1-p). This is used as a regularisation, i.e. it reduces overfitting during training.
Does nothing to the input once in testmode!.
source
Flux.LayerNorm — Type.
LayerNorm(h::Integer)
A normalisation layer designed to be used with recurrent hidden states of size h. Normalises the mean/stddev of each input before applying a per-neuron gain/bias.
source

+  softmax)
source
Flux.Dropout — Type.
Dropout(p)
A Dropout layer. For each input, either sets that input to 0 (with probability p) or scales it by 1/(1-p). This is used as a regularisation, i.e. it reduces overfitting during training.
Does nothing to the input once in testmode!.
source
Flux.LayerNorm — Type.
LayerNorm(h::Integer)
A normalisation layer designed to be used with recurrent hidden states of size h. Normalises the mean/stddev of each input before applying a per-neuron gain/bias.
source

diff --git a/latest/models/recurrence.html b/latest/models/recurrence.html
index 4a824a8b..550b5c7d 100644
--- a/latest/models/recurrence.html
+++ b/latest/models/recurrence.html
@@ -6,7 +6,7 @@ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
 
 ga('create', 'UA-36890222-9', 'auto');
 ga('send', 'pageview');
-
Recurrence
Recurrent Models
Recurrent Cells
In the simple feedforward case, our model m is a simple function from various inputs xᵢ to predictions yᵢ. (For example, each x might be an MNIST digit and each y a digit label.) Each prediction is completely independent of any others, and using the same x will always produce the same y.
y₁ = f(x₁)
+
Recurrence
Recurrent Models
Recurrent Cells
In the simple feedforward case, our model m is a simple function from various inputs xᵢ to predictions yᵢ. (For example, each x might be an MNIST digit and each y a digit label.) Each prediction is completely independent of any others, and using the same x will always produce the same y.
y₁ = f(x₁)
 y₂ = f(x₂)
 y₃ = f(x₃)
 # ...
Recurrent networks introduce a hidden state that gets carried over each time we run the model. The model now takes the old h as an input, and produces a new h as output, each time we run it.
h = # ... initial state ...
diff --git a/latest/models/regularisation.html b/latest/models/regularisation.html
index 51c54208..b165c1bd 100644
--- a/latest/models/regularisation.html
+++ b/latest/models/regularisation.html
@@ -6,7 +6,7 @@ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
 
 ga('create', 'UA-36890222-9', 'auto');
 ga('send', 'pageview');
-
Regularisation
Regularisation
Applying regularisation to model parameters is straightforward. We just need to apply an appropriate regulariser, such as vecnorm, to each model parameter and add the result to the overall loss.
For example, say we have a simple regression.
m = Dense(10, 5)
+
Regularisation
Regularisation
Applying regularisation to model parameters is straightforward. We just need to apply an appropriate regulariser, such as vecnorm, to each model parameter and add the result to the overall loss.
For example, say we have a simple regression.
m = Dense(10, 5)
 loss(x, y) = crossentropy(softmax(m(x)), y)
We can regularise this by taking the (L2) norm of the parameters, m.W and m.b.
penalty() = vecnorm(m.W) + vecnorm(m.b)
 loss(x, y) = crossentropy(softmax(m(x)), y) + penalty()
When working with layers, Flux provides the params function to grab all parameters at once. We can easily penalise everything with sum(vecnorm, params).
julia> params(m)
 2-element Array{Any,1}:
diff --git a/latest/saving.html b/latest/saving.html
index aa434977..a3694ffa 100644
--- a/latest/saving.html
+++ b/latest/saving.html
@@ -6,7 +6,7 @@ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
 
 ga('create', 'UA-36890222-9', 'auto');
 ga('send', 'pageview');
-
Saving & Loading
Saving and Loading Models
You may wish to save models so that they can be loaded and run in a later session. The easiest way to do this is via BSON.jl.
Save a model:
julia> using Flux
+
Saving & Loading
Saving and Loading Models
You may wish to save models so that they can be loaded and run in a later session. The easiest way to do this is via BSON.jl.
Save a model:
julia> using Flux
 
 julia> model = Chain(Dense(10,5,relu),Dense(5,2),softmax)
 Chain(Dense(10, 5, NNlib.relu), Dense(5, 2), NNlib.softmax)
@@ -47,4 +47,4 @@ evalcb = throttle(30) do
   # Show loss
   @save "model-checkpoint.bson" model
 end
This will update the "model-checkpoint.bson" file every thirty seconds.
You can get more advanced by saving a series of models throughout training, for example
@save "model-$(now()).bson" model
will produce a series of models like "model-2018-03-06T02:57:10.41.bson". You could also store the current test set loss, so that it's easy to (for example) revert to an older copy of the model if it starts to overfit.
@save "model-$(now()).bson" model loss = testloss()
You can even store optimiser state alongside the model, to resume training exactly where you left off.
opt = ADAM(params(model))
-@save "model-$(now()).bson" model opt

+@save "model-$(now()).bson" model opt

diff --git a/latest/search.html b/latest/search.html
index 2fc782db..00437b2e 100644
--- a/latest/search.html
+++ b/latest/search.html
@@ -6,4 +6,4 @@ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
 
 ga('create', 'UA-36890222-9', 'auto');
 ga('send', 'pageview');
-
Search
Search
Number of results: loading...
    
+
    Search
    Search
    Number of results: loading...
      
diff --git a/latest/search_index.js b/latest/search_index.js
index f6ba4bb7..a404f0db 100644
--- a/latest/search_index.js
+++ b/latest/search_index.js
@@ -472,6 +472,46 @@ var documenterSearchIndex = {"docs": [
     "text": "In longer training runs it\'s a good idea to periodically save your model, so that you can resume if training is interrupted (for example, if there\'s a power cut). You can do this by saving the model in the callback provided to train!.using Flux: throttle\nusing BSON: @save\n\nm = Chain(Dense(10,5,relu),Dense(5,2),softmax)\n\nevalcb = throttle(30) do\n  # Show loss\n  @save \"model-checkpoint.bson\" model\nendThis will update the \"model-checkpoint.bson\" file every thirty seconds.You can get more advanced by saving a series of models throughout training, for example@save \"model-$(now()).bson\" modelwill produce a series of models like \"model-2018-03-06T02:57:10.41.bson\". You could also store the current test set loss, so that it\'s easy to (for example) revert to an older copy of the model if it starts to overfit.@save \"model-$(now()).bson\" model loss = testloss()You can even store optimiser state alongside the model, to resume training exactly where you left off.opt = ADAM(params(model))\n@save \"model-$(now()).bson\" model opt"
 },
 
+{
+    "location": "internals/tracker.html#",
+    "page": "Backpropagation",
+    "title": "Backpropagation",
+    "category": "page",
+    "text": ""
+},
+
+{
+    "location": "internals/tracker.html#Flux.Tracker-1",
+    "page": "Backpropagation",
+    "title": "Flux.Tracker",
+    "category": "section",
+    "text": "Backpropagation, or reverse-mode automatic differentiation, is handled by the Flux.Tracker module.julia> using Flux.TrackerThe param function converts a normal Julia array into a new object that, while behaving like an array, tracks extra information that allows us to calculate derivatives. For example, say we multiply two parameters:julia> W = param([1 2; 3 4])\nTracked 2×2 Array{Float64,2}:\n 1.0  2.0\n 3.0  4.0\n\njulia> x = param([5, 6])\nTracked 2-element Array{Float64,1}:\n 5.0\n 6.0\n\njulia> y = W*x\nTracked 2-element Array{Float64,1}:\n 17.0\n 39.0The output y is also a TrackedArray object. We can now backpropagate sensitivities to W and x via the back! function, and see the gradients accumulated in the W and x tracked arrays:julia> Tracker.back!(y, [1, -1])\n\njulia> W.grad\n2×2 Array{Float64,2}:\n 5.0   6.0\n-5.0  -6.0\n\njulia> x.grad\n2-element Array{Float64,1}:\n -2.0\n -2.0"
+},
+
+{
+    "location": "internals/tracker.html#Internals-1",
+    "page": "Backpropagation",
+    "title": "Internals",
+    "category": "section",
+    "text": "All Tracked* objects (TrackedArray, TrackedReal) are light wrappers around the Tracked type, which you can access via the .tracker field.julia> x.tracker\nFlux.Tracker.Tracked{Array{Float64,1}}(0x00000000, Flux.Tracker.Call{Void,Tuple{}}(nothing, ()), true, [5.0, 6.0], [-2.0, -2.0])The Tracker stores the value and gradient of a given object, which we\'ve seen before.julia> x.tracker.data\n2-element Array{Float64,1}:\n 5.0\n 6.0\n\njulia> x.tracker.grad\n2-element Array{Float64,1}:\n -2.0\n -2.0The tracker also contains a Call object, which simply represents a function call that was made at some point during the forward pass. For example, the + call would look like this:julia> Tracker.Call(+, 1, 2)\nFlux.Tracker.Call{Base.#+,Tuple{Int64,Int64}}(+, (1, 2))In the case of the y we produced above, we can see that it stores the call that produced it – that is, W*x.julia> y.tracker.f\nFlux.Tracker.Call{...}(*, (param([1.0 2.0; 3.0 4.0]), param([5.0, 6.0])))Notice that because the arguments to the call may also be tracked arrays, storing their own calls, this means that Tracker ends up forming a data structure that records everything that happened during the forward pass (often known as a tape).When we call back!(y, [1, -1]), the sensitivities [1, -1] simply get forwarded to y\'s call (*), effectively callingTracker.back(*, [1, -1], W, x)which in turn calculates the sensitivities of the arguments (W and x) and backpropagates through their calls. This is recursive, so it will walk the entire program graph and propagate gradients to the original model parameters."
+},
+
+{
+    "location": "internals/tracker.html#Custom-Gradients-1",
+    "page": "Backpropagation",
+    "title": "Custom Gradients",
+    "category": "section",
+    "text": "We can hook in to the processes above to implement custom gradients for a function or kernel. For a toy example, imagine a custom implementation of minus:julia> minus(a, b) = a - bFirstly, we must tell the tracker system to stop when it sees a call to minus, and record it. We can do this using dispatch:julia> minus(a::TrackedArray, b::TrackedArray) = Tracker.track(minus, a, b)\nminus (generic function with 2 methods)Tracker.track does two things: (1) it makes sure minus is called with normal array, not tracked ones (you can use @show inside minus to verify this), and (2) it uses the result to add a minus node to the tape. Look inside the result of calling minus to see what happened:julia> a, b = param([6,5,4]), param([1,2,3])\n(param([6.0, 5.0, 4.0]), param([1.0, 2.0, 3.0]))\n\njulia> c = minus(a, b)\nTracked 3-element Array{Float64,1}:\n 5.0\n 3.0\n 1.0\n\njulia> c.tracker.f\nFlux.Tracker.Call{...}(minus, (param([6.0, 5.0, 4.0]), param([1.0, 2.0, 3.0])))Finally, we have to specify the gradient of minus.julia> Tracker.back(::typeof(minus), Δ, a, b) =\n        (Tracker.@back(a, Δ); Tracker.@back(b, -Δ))@back(x, Δ) tells the tracker to continue propagating the sensitivity Δ through x. Now, AD will work with any program that calls minus.julia> Flux.back!(c, 1)\n\njulia> a.grad\n3-element Array{Float64,1}:\n 1.0\n 1.0\n 1.0\n\njulia> b.grad\n3-element Array{Float64,1}:\n -1.0\n -1.0\n -1.0"
+},
+
+{
+    "location": "internals/tracker.html#Notes-1",
+    "page": "Backpropagation",
+    "title": "Notes",
+    "category": "section",
+    "text": "For multi-argument functions with custom gradients, you likely want to catch not just minus(::TrackedArray, ::TrackedArray) but also minus(::Array, TrackedArray) and so on. To do so, just define those extra signatures as needed:minus(a::AbstractArray, b::TrackedArray) = Tracker.track(minus, a, b)\nminus(a::TrackedArray, b::AbstractArray) = Tracker.track(minus, a, b)@back must be called exactly once on each tracked input argument. You do not need to do any special handling if one of the arguments is not tracked, as @back will just become a no-op."
+},
+
 {
     "location": "community.html#",
     "page": "Community",
diff --git a/latest/training/optimisers.html b/latest/training/optimisers.html
index f905563d..ad9d88ad 100644
--- a/latest/training/optimisers.html
+++ b/latest/training/optimisers.html
@@ -6,7 +6,7 @@ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
 
 ga('create', 'UA-36890222-9', 'auto');
 ga('send', 'pageview');
-
      Optimisers
      Optimisers
      Consider a simple linear regression. We create some dummy data, calculate a loss, and backpropagate to calculate gradients for the parameters W and b.
      W = param(rand(2, 5))
+
      Optimisers
      Optimisers
      Consider a simple linear regression. We create some dummy data, calculate a loss, and backpropagate to calculate gradients for the parameters W and b.
      W = param(rand(2, 5))
 b = param(rand(2))
 
 predict(x) = W*x .+ b
@@ -24,4 +24,4 @@ end
      If we call update, the parameters W
   Dense(10, 5, σ),
   Dense(5, 2), softmax)
      Instead of having to write [m[1].W, m[1].b, ...], Flux provides a params function params(m) that returns a list of all parameters in the model for you.
      For the update step, there's nothing whatsoever wrong with writing the loop above – it'll work just fine – but Flux provides various optimisers that make it more convenient.
      opt = SGD([W, b], 0.1) # Gradient descent with learning rate 0.1
 
-opt() # Carry out the update, modifying `W` and `b`.
      An optimiser takes a parameter list and returns a function that does the same thing as update above. We can pass either opt or update to our training loop, which will then run the optimiser after every mini-batch of data.
      Optimiser Reference
      All optimisers return a function that, when called, will update the parameters passed to it.
      Flux.Optimise.SGD — Function.
      SGD(params, η = 0.1; decay = 0)
      Classic gradient descent optimiser with learning rate η. For each parameter p and its gradient δp, this runs p -= η*δp.
      Supports inverse decaying learning rate if the decay argument is provided.
      source
      Flux.Optimise.Momentum — Function.
      Momentum(params, η = 0.01; ρ = 0.9, decay = 0)
      SGD with learning rate  η, momentum ρ and optional learning rate inverse decay.
      source
      Flux.Optimise.Nesterov — Function.
      Nesterov(params, η = 0.01; ρ = 0.9, decay = 0)
      SGD with learning rate  η, Nesterov momentum ρ and optional learning rate inverse decay.
      source
      Flux.Optimise.ADAM — Function.
      ADAM(params, η = 0.001; β1 = 0.9, β2 = 0.999, ϵ = 1e-08, decay = 0)
      ADAM optimiser.
      source
      
+opt() # Carry out the update, modifying `W` and `b`.
      An optimiser takes a parameter list and returns a function that does the same thing as update above. We can pass either opt or update to our training loop, which will then run the optimiser after every mini-batch of data.
      Optimiser Reference
      All optimisers return a function that, when called, will update the parameters passed to it.
      Flux.Optimise.SGD — Function.
      SGD(params, η = 0.1; decay = 0)
      Classic gradient descent optimiser with learning rate η. For each parameter p and its gradient δp, this runs p -= η*δp.
      Supports inverse decaying learning rate if the decay argument is provided.
      source
      Flux.Optimise.Momentum — Function.
      Momentum(params, η = 0.01; ρ = 0.9, decay = 0)
      SGD with learning rate  η, momentum ρ and optional learning rate inverse decay.
      source
      Flux.Optimise.Nesterov — Function.
      Nesterov(params, η = 0.01; ρ = 0.9, decay = 0)
      SGD with learning rate  η, Nesterov momentum ρ and optional learning rate inverse decay.
      source
      Flux.Optimise.ADAM — Function.
      ADAM(params, η = 0.001; β1 = 0.9, β2 = 0.999, ϵ = 1e-08, decay = 0)
      ADAM optimiser.
      source
      
diff --git a/latest/training/training.html b/latest/training/training.html
index 858d936a..c6f05a6b 100644
--- a/latest/training/training.html
+++ b/latest/training/training.html
@@ -6,7 +6,7 @@ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
 
 ga('create', 'UA-36890222-9', 'auto');
 ga('send', 'pageview');
-
      Training
      Training
      To actually train a model we need three things:
      • A objective function, that evaluates how well a model is doing given some input data.
      • A collection of data points that will be provided to the objective function.
      • An optimiser that will update the model parameters appropriately.
      With these we can call Flux.train!:
      Flux.train!(objective, data, opt)
      There are plenty of examples in the model zoo.
      Loss Functions
      The objective function must return a number representing how far the model is from its target – the loss of the model. The loss function that we defined in basics will work as an objective. We can also define an objective in terms of some model:
      m = Chain(
+
      Training
      Training
      To actually train a model we need three things:
      • A objective function, that evaluates how well a model is doing given some input data.
      • A collection of data points that will be provided to the objective function.
      • An optimiser that will update the model parameters appropriately.
      With these we can call Flux.train!:
      Flux.train!(objective, data, opt)
      There are plenty of examples in the model zoo.
      Loss Functions
      The objective function must return a number representing how far the model is from its target – the loss of the model. The loss function that we defined in basics will work as an objective. We can also define an objective in terms of some model:
      m = Chain(
   Dense(784, 32, σ),
   Dense(32, 10), softmax)