From e8598fa03538e8fd4f262c3d75f328e45fc3119e Mon Sep 17 00:00:00 2001 From: zeptodoctor <44736852+zeptodoctor@users.noreply.github.com> Date: Thu, 26 Mar 2020 10:07:57 +0000 Subject: [PATCH] build based on 6b37ce3 --- dev/assets/search.js | 3 +-- dev/community/index.html | 2 +- dev/data/dataloader/index.html | 2 +- dev/data/onehot/index.html | 2 +- dev/ecosystem/index.html | 2 +- dev/gpu/index.html | 2 +- dev/index.html | 2 +- dev/models/advanced/index.html | 2 +- dev/models/basics/index.html | 2 +- dev/models/layers/index.html | 22 +++++++++++----------- dev/models/nnlib/index.html | 2 +- dev/models/recurrence/index.html | 2 +- dev/models/regularisation/index.html | 2 +- dev/performance/index.html | 2 +- dev/saving/index.html | 2 +- dev/search/index.html | 2 +- dev/training/optimisers/index.html | 26 +++++++++++++------------- dev/training/training/index.html | 4 ++-- 18 files changed, 41 insertions(+), 42 deletions(-) diff --git a/dev/assets/search.js b/dev/assets/search.js index 42a273f5..c32d2f1d 100644 --- a/dev/assets/search.js +++ b/dev/assets/search.js @@ -46,7 +46,7 @@ $(document).ready(function() { }) // list below is the lunr 2.1.3 list minus the intersect with names(Base) - // (all, any, get, in, is, which) and (do, else, for, let, where, while, with) + // (all, any, get, in, is, only, which) and (do, else, for, let, where, while, with) // ideally we'd just filter the original list but it's not available as a variable lunr.stopWordFilter = lunr.generateStopWordFilter([ 'a', @@ -112,7 +112,6 @@ $(document).ready(function() { 'off', 'often', 'on', - 'only', 'or', 'other', 'our', diff --git a/dev/community/index.html b/dev/community/index.html index 8612e8a0..0b8dd746 100644 --- a/dev/community/index.html +++ b/dev/community/index.html @@ -6,4 +6,4 @@ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) ga('create', 'UA-36890222-9', 'auto'); ga('send', 'pageview', {'page': location.pathname + location.search + location.hash}); -
Edit on GitHub

Community

All Flux users are welcome to join our community on the Julia forum, or the slack (channel #machine-learning). If you have questions or issues we'll try to help you out.

If you're interested in hacking on Flux, the source code is open and easy to understand – it's all just the same Julia code you work with normally. You might be interested in our intro issues to get started.

+
Edit on GitHub

Community

All Flux users are welcome to join our community on the Julia forum, or the slack (channel #machine-learning). If you have questions or issues we'll try to help you out.

If you're interested in hacking on Flux, the source code is open and easy to understand – it's all just the same Julia code you work with normally. You might be interested in our intro issues to get started.

diff --git a/dev/data/dataloader/index.html b/dev/data/dataloader/index.html index 3f435b51..ef58668d 100644 --- a/dev/data/dataloader/index.html +++ b/dev/data/dataloader/index.html @@ -29,4 +29,4 @@ end # train for 10 epochs using IterTools: ncycle -Flux.train!(loss, ps, ncycle(train_loader, 10), opt)source +Flux.train!(loss, ps, ncycle(train_loader, 10), opt)source diff --git a/dev/data/onehot/index.html b/dev/data/onehot/index.html index c9deb9b2..1e97153b 100644 --- a/dev/data/onehot/index.html +++ b/dev/data/onehot/index.html @@ -37,4 +37,4 @@ julia> onecold(ans, [:a, :b, :c]) 3-element Array{Symbol,1}: :b :a - :b

Note that these operations returned OneHotVector and OneHotMatrix rather than Arrays. OneHotVectors behave like normal vectors but avoid any unnecessary cost compared to using an integer index directly. For example, multiplying a matrix with a one-hot vector simply slices out the relevant row of the matrix under the hood.

+ :b

Note that these operations returned OneHotVector and OneHotMatrix rather than Arrays. OneHotVectors behave like normal vectors but avoid any unnecessary cost compared to using an integer index directly. For example, multiplying a matrix with a one-hot vector simply slices out the relevant row of the matrix under the hood.

diff --git a/dev/ecosystem/index.html b/dev/ecosystem/index.html index e8fb000c..a33af5be 100644 --- a/dev/ecosystem/index.html +++ b/dev/ecosystem/index.html @@ -6,4 +6,4 @@ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) ga('create', 'UA-36890222-9', 'auto'); ga('send', 'pageview', {'page': location.pathname + location.search + location.hash}); -
Edit on GitHub

The Julia Ecosystem

One of the main strengths of Julia lies in an ecosystem of packages globally providing a rich and consistent user experience.

This is a non-exhaustive list of Julia packages, nicely complementing Flux in typical machine learning and deep learning workflows:

This tight integration among Julia pakages is shown in some of the examples in the model-zoo repository.

+
Edit on GitHub

The Julia Ecosystem

One of the main strengths of Julia lies in an ecosystem of packages globally providing a rich and consistent user experience.

This is a non-exhaustive list of Julia packages, nicely complementing Flux in typical machine learning and deep learning workflows:

This tight integration among Julia pakages is shown in some of the examples in the model-zoo repository.

diff --git a/dev/gpu/index.html b/dev/gpu/index.html index c878f9ea..24eb62de 100644 --- a/dev/gpu/index.html +++ b/dev/gpu/index.html @@ -47,4 +47,4 @@ julia> x |> cpu 10-element Array{Float32,1}: 0.235164 ⋮ - 0.192538 + 0.192538 diff --git a/dev/index.html b/dev/index.html index 47e4b1b4..1ae16894 100644 --- a/dev/index.html +++ b/dev/index.html @@ -6,4 +6,4 @@ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) ga('create', 'UA-36890222-9', 'auto'); ga('send', 'pageview', {'page': location.pathname + location.search + location.hash}); -
Edit on GitHub

Flux: The Julia Machine Learning Library

Flux is a library for machine learning. It comes "batteries-included" with many useful tools built in, but also lets you use the full power of the Julia language where you need it. We follow a few key principles:

  • Doing the obvious thing. Flux has relatively few explicit APIs for features like regularisation or embeddings. Instead, writing down the mathematical form will work – and be fast.
  • You could have written Flux. All of it, from LSTMs to GPU kernels, is straightforward Julia code. When in doubt, it’s well worth looking at the source. If you need something different, you can easily roll your own.
  • Play nicely with others. Flux works well with Julia libraries from data frames and images to differential equation solvers, so you can easily build complex data processing pipelines that integrate Flux models.

Installation

Download Julia 1.0 or later, if you haven't already. You can add Flux from using Julia's package manager, by typing ] add Flux in the Julia prompt.

If you have CUDA you can also run ] add CuArrays to get GPU support; see here for more details.

Learning Flux

There are several different ways to learn Flux. If you just want to get started writing models, the model zoo gives good starting points for many common ones. This documentation provides a reference to all of Flux's APIs, as well as a from-scratch introduction to Flux's take on models and how they work. Once you understand these docs, congratulations, you also understand Flux's source code, which is intended to be concise, legible and a good reference for more advanced concepts.

+
Edit on GitHub

Flux: The Julia Machine Learning Library

Flux is a library for machine learning. It comes "batteries-included" with many useful tools built in, but also lets you use the full power of the Julia language where you need it. We follow a few key principles:

  • Doing the obvious thing. Flux has relatively few explicit APIs for features like regularisation or embeddings. Instead, writing down the mathematical form will work – and be fast.
  • You could have written Flux. All of it, from LSTMs to GPU kernels, is straightforward Julia code. When in doubt, it’s well worth looking at the source. If you need something different, you can easily roll your own.
  • Play nicely with others. Flux works well with Julia libraries from data frames and images to differential equation solvers, so you can easily build complex data processing pipelines that integrate Flux models.

Installation

Download Julia 1.0 or later, if you haven't already. You can add Flux from using Julia's package manager, by typing ] add Flux in the Julia prompt.

If you have CUDA you can also run ] add CuArrays to get GPU support; see here for more details.

Learning Flux

There are several different ways to learn Flux. If you just want to get started writing models, the model zoo gives good starting points for many common ones. This documentation provides a reference to all of Flux's APIs, as well as a from-scratch introduction to Flux's take on models and how they work. Once you understand these docs, congratulations, you also understand Flux's source code, which is intended to be concise, legible and a good reference for more advanced concepts.

diff --git a/dev/models/advanced/index.html b/dev/models/advanced/index.html index 8787d6a3..12cfcf9a 100644 --- a/dev/models/advanced/index.html +++ b/dev/models/advanced/index.html @@ -24,4 +24,4 @@ Params([[0.66722 0.774872 0.249809; 0.843321 0.403843 0.429232; 0.683525 0.66245 ) ps = Flux.params(m[3:end])

The Zygote.Params object ps now holds a reference to only the parameters of the layers passed to it.

During training, the gradients will only be computed for (and applied to) the last Dense layer, therefore only that would have its parameters changed.

Flux.params also takes multiple inputs to make it easy to collect parameters from heterogenous models with a single call. A simple demonstration would be if we wanted to omit optimising the second Dense layer in the previous example. It would look something like this:

Flux.params(m[1], m[3:end])

Sometimes, a more fine-tuned control is needed. We can freeze a specific parameter of a specific layer which already entered a Params object ps, by simply deleting it from ps:

ps = params(m)
-delete!(ps, m[2].b)
+delete!(ps, m[2].b) diff --git a/dev/models/basics/index.html b/dev/models/basics/index.html index 11de7076..52e415e9 100644 --- a/dev/models/basics/index.html +++ b/dev/models/basics/index.html @@ -110,4 +110,4 @@ model2(rand(10)) # => 2-element vector

This quickly starts to m(rand(10))

Likewise, Chain will happily work with any Julia function.

m = Chain(x -> x^2, x -> x+1)
 
-m(5) # => 26

Layer helpers

Flux provides a set of helpers for custom layers, which you can enable by calling

Flux.@functor Affine

This enables a useful extra set of functionality for our Affine layer, such as collecting its parameters or moving it to the GPU.

For some more helpful tricks, including parameter freezing, please checkout the advanced usage guide.

Utility functions

Flux provides some utility functions to help you generate models in an automated fashion.

outdims enables you to calculate the spatial output dimensions of layers like Conv when applied to input images of a given size. Currently limited to the following layers:

Missing docstring.

Missing docstring for outdims. Check Documenter's build log for details.

+m(5) # => 26

Layer helpers

Flux provides a set of helpers for custom layers, which you can enable by calling

Flux.@functor Affine

This enables a useful extra set of functionality for our Affine layer, such as collecting its parameters or moving it to the GPU.

For some more helpful tricks, including parameter freezing, please checkout the advanced usage guide.

Utility functions

Flux provides some utility functions to help you generate models in an automated fashion.

outdims enables you to calculate the spatial output dimensions of layers like Conv when applied to input images of a given size. Currently limited to the following layers:

Missing docstring.

Missing docstring for outdims. Check Documenter's build log for details.

diff --git a/dev/models/layers/index.html b/dev/models/layers/index.html index 6df3bf8d..4dbeb652 100644 --- a/dev/models/layers/index.html +++ b/dev/models/layers/index.html @@ -11,43 +11,43 @@ m(5) == 26 m = Chain(Dense(10, 5), Dense(5, 2)) x = rand(10) -m(x) == m[2](m[1](x))

Chain also supports indexing and slicing, e.g. m[2] or m[1:end-1]. m[1:3](x) will calculate the output of the first three layers.

source
Flux.Dense — Type
Dense(in::Integer, out::Integer, σ = identity)

Creates a traditional Dense layer with parameters W and b.

y = σ.(W * x .+ b)

The input x must be a vector of length in, or a batch of vectors represented as an in × N matrix. The out y will be a vector or batch of length out.

julia> d = Dense(5, 2)
+m(x) == m[2](m[1](x))

Chain also supports indexing and slicing, e.g. m[2] or m[1:end-1]. m[1:3](x) will calculate the output of the first three layers.

source
Flux.Dense — Type
Dense(in::Integer, out::Integer, σ = identity)

Creates a traditional Dense layer with parameters W and b.

y = σ.(W * x .+ b)

The input x must be a vector of length in, or a batch of vectors represented as an in × N matrix. The out y will be a vector or batch of length out.

julia> d = Dense(5, 2)
 Dense(5, 2)
 
 julia> d(rand(5))
 Array{Float64,1}:
   0.00257447
-  -0.00449443
source

Convolution and Pooling Layers

These layers are used to build convolutional neural networks (CNNs).

Flux.Conv — Type
Conv(size, in=>out)
+  -0.00449443
source

Convolution and Pooling Layers

These layers are used to build convolutional neural networks (CNNs).

Flux.Conv — Type
Conv(size, in=>out)
 Conv(size, in=>out, relu)

Standard convolutional layer. size should be a tuple like (2, 2). in and out specify the number of input and output channels respectively.

Example: Applying Conv layer to a 1-channel input using a 2x2 window size, giving us a 16-channel output. Output is activated with ReLU.

size = (2,2)
 in = 1
 out = 16
-Conv((2, 2), 1=>16, relu)

Data should be stored in WHCN order (width, height, # channels, batch size). In other words, a 100×100 RGB image would be a 100×100×3×1 array, and a batch of 50 would be a 100×100×3×50 array.

Takes the keyword arguments pad, stride and dilation.

source
Flux.MaxPool — Type
MaxPool(k)

Max pooling layer. k stands for the size of the window for each dimension of the input.

Takes the keyword arguments pad and stride.

source
Flux.GlobalMaxPool — Type
GlobalMaxPool()

Global max pooling layer.

Transforms (w,h,c,b)-shaped input into (1,1,c,b)-shaped output, by performing max pooling on the complete (w,h)-shaped feature maps.

source
Flux.MeanPool — Type
MeanPool(k)

Mean pooling layer. k stands for the size of the window for each dimension of the input.

Takes the keyword arguments pad and stride.

source
Flux.GlobalMeanPool — Type
GlobalMeanPool()

Global mean pooling layer.

Transforms (w,h,c,b)-shaped input into (1,1,c,b)-shaped output, by performing mean pooling on the complete (w,h)-shaped feature maps.

source
Flux.DepthwiseConv — Type
DepthwiseConv(size, in=>out)
-DepthwiseConv(size, in=>out, relu)

Depthwise convolutional layer. size should be a tuple like (2, 2). in and out specify the number of input and output channels respectively. Note that out must be an integer multiple of in.

Data should be stored in WHCN order (width, height, # channels, # batches). In other words, a 100×100 RGB image would be a 100×100×3×1 array, and a batch of 50 would be a 100×100×3×50 array.

Takes the keyword arguments pad, stride and dilation.

source
Flux.ConvTranspose — Type
ConvTranspose(size, in=>out)
-ConvTranspose(size, in=>out, relu)

Standard convolutional transpose layer. size should be a tuple like (2, 2). in and out specify the number of input and output channels respectively.

Data should be stored in WHCN order (width, height, # channels, # batches). In other words, a 100×100 RGB image would be a 100×100×3×1 array, and a batch of 50 would be a 100×100×3×50 array.

Takes the keyword arguments pad, stride and dilation.

source
Flux.CrossCor — Type
CrossCor(size, in=>out)
+Conv((2, 2), 1=>16, relu)

Data should be stored in WHCN order (width, height, # channels, batch size). In other words, a 100×100 RGB image would be a 100×100×3×1 array, and a batch of 50 would be a 100×100×3×50 array.

Takes the keyword arguments pad, stride and dilation.

source
Flux.MaxPool — Type
MaxPool(k)

Max pooling layer. k stands for the size of the window for each dimension of the input.

Takes the keyword arguments pad and stride.

source
Flux.GlobalMaxPool — Type
GlobalMaxPool()

Global max pooling layer.

Transforms (w,h,c,b)-shaped input into (1,1,c,b)-shaped output, by performing max pooling on the complete (w,h)-shaped feature maps.

source
Flux.MeanPool — Type
MeanPool(k)

Mean pooling layer. k stands for the size of the window for each dimension of the input.

Takes the keyword arguments pad and stride.

source
Flux.GlobalMeanPool — Type
GlobalMeanPool()

Global mean pooling layer.

Transforms (w,h,c,b)-shaped input into (1,1,c,b)-shaped output, by performing mean pooling on the complete (w,h)-shaped feature maps.

source
Flux.DepthwiseConv — Type
DepthwiseConv(size, in=>out)
+DepthwiseConv(size, in=>out, relu)

Depthwise convolutional layer. size should be a tuple like (2, 2). in and out specify the number of input and output channels respectively. Note that out must be an integer multiple of in.

Data should be stored in WHCN order (width, height, # channels, # batches). In other words, a 100×100 RGB image would be a 100×100×3×1 array, and a batch of 50 would be a 100×100×3×50 array.

Takes the keyword arguments pad, stride and dilation.

source
Flux.ConvTranspose — Type
ConvTranspose(size, in=>out)
+ConvTranspose(size, in=>out, relu)

Standard convolutional transpose layer. size should be a tuple like (2, 2). in and out specify the number of input and output channels respectively.

Data should be stored in WHCN order (width, height, # channels, # batches). In other words, a 100×100 RGB image would be a 100×100×3×1 array, and a batch of 50 would be a 100×100×3×50 array.

Takes the keyword arguments pad, stride and dilation.

source
Flux.CrossCor — Type
CrossCor(size, in=>out)
 CrossCor(size, in=>out, relu)

Standard cross convolutional layer. size should be a tuple like (2, 2). in and out specify the number of input and output channels respectively.

Example: Applying CrossCor layer to a 1-channel input using a 2x2 window size, giving us a 16-channel output. Output is activated with ReLU.

size = (2,2)
 in = 1
 out = 16
-CrossCor((2, 2), 1=>16, relu)

Data should be stored in WHCN order (width, height, # channels, # batches). In other words, a 100×100 RGB image would be a 100×100×3×1 array, and a batch of 50 would be a 100×100×3×50 array.

Takes the keyword arguments pad, stride and dilation.

source
Flux.flatten — Function
flatten(x::AbstractArray)

Transforms (w,h,c,b)-shaped input into (w x h x c,b)-shaped output, by linearizing all values for each element in the batch.

source

Recurrent Layers

Much like the core layers above, but can be used to process sequence data (as well as other kinds of structured data).

Flux.RNN — Function
RNN(in::Integer, out::Integer, σ = tanh)

The most basic recurrent layer; essentially acts as a Dense layer, but with the output fed back into the input each time step.

source
Flux.LSTM — Function
LSTM(in::Integer, out::Integer)

Long Short Term Memory recurrent layer. Behaves like an RNN but generally exhibits a longer memory span over sequences.

See this article for a good overview of the internals.

source
Flux.GRU — Function
GRU(in::Integer, out::Integer)

Gated Recurrent Unit layer. Behaves like an RNN but generally exhibits a longer memory span over sequences.

See this article for a good overview of the internals.

source
Flux.Recur — Type
Recur(cell)

Recur takes a recurrent cell and makes it stateful, managing the hidden state in the background. cell should be a model of the form:

h, y = cell(h, x...)

For example, here's a recurrent network that keeps a running total of its inputs.

accum(h, x) = (h+x, x)
+CrossCor((2, 2), 1=>16, relu)

Data should be stored in WHCN order (width, height, # channels, # batches). In other words, a 100×100 RGB image would be a 100×100×3×1 array, and a batch of 50 would be a 100×100×3×50 array.

Takes the keyword arguments pad, stride and dilation.

source
Flux.flatten — Function
flatten(x::AbstractArray)

Transforms (w,h,c,b)-shaped input into (w x h x c,b)-shaped output, by linearizing all values for each element in the batch.

source

Recurrent Layers

Much like the core layers above, but can be used to process sequence data (as well as other kinds of structured data).

Flux.RNN — Function
RNN(in::Integer, out::Integer, σ = tanh)

The most basic recurrent layer; essentially acts as a Dense layer, but with the output fed back into the input each time step.

source
Flux.LSTM — Function
LSTM(in::Integer, out::Integer)

Long Short Term Memory recurrent layer. Behaves like an RNN but generally exhibits a longer memory span over sequences.

See this article for a good overview of the internals.

source
Flux.GRU — Function
GRU(in::Integer, out::Integer)

Gated Recurrent Unit layer. Behaves like an RNN but generally exhibits a longer memory span over sequences.

See this article for a good overview of the internals.

source
Flux.Recur — Type
Recur(cell)

Recur takes a recurrent cell and makes it stateful, managing the hidden state in the background. cell should be a model of the form:

h, y = cell(h, x...)

For example, here's a recurrent network that keeps a running total of its inputs.

accum(h, x) = (h+x, x)
 rnn = Flux.Recur(accum, 0)
 rnn(2) # 2
 rnn(3) # 3
 rnn.state # 5
 rnn.(1:10) # apply to a sequence
-rnn.state # 60
source

Other General Purpose Layers

These are marginally more obscure than the Basic Layers. But in contrast to the layers described in the other sections are not readily grouped around a particular purpose (e.g. CNNs or RNNs).

Flux.Maxout — Type
Maxout(over)

Maxout is a neural network layer, which has a number of internal layers, which all have the same input, and the maxout returns the elementwise maximium of the internal layers' outputs.

Maxout over linear dense layers satisfies the univeral approximation theorem.

Reference: Ian J. Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron Courville, and Yoshua Bengio.

  1. Maxout networks.

In Proceedings of the 30th International Conference on International Conference on Machine Learning - Volume 28 (ICML'13), Sanjoy Dasgupta and David McAllester (Eds.), Vol. 28. JMLR.org III-1319-III-1327. https://arxiv.org/pdf/1302.4389.pdf

source
Flux.SkipConnection — Type
SkipConnection(layers, connection)

Creates a Skip Connection, of a layer or Chain of consecutive layers plus a shortcut connection. The connection function will combine the result of the layers with the original input, to give the final output.

The simplest 'ResNet'-type connection is just SkipConnection(layer, +), and requires the output of the layers to be the same shape as the input. Here is a more complicated example:

m = Conv((3,3), 4=>7, pad=(1,1))
+rnn.state # 60
source

Other General Purpose Layers

These are marginally more obscure than the Basic Layers. But in contrast to the layers described in the other sections are not readily grouped around a particular purpose (e.g. CNNs or RNNs).

Flux.Maxout — Type
Maxout(over)

Maxout is a neural network layer, which has a number of internal layers, which all have the same input, and the maxout returns the elementwise maximium of the internal layers' outputs.

Maxout over linear dense layers satisfies the univeral approximation theorem.

Reference: Ian J. Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron Courville, and Yoshua Bengio.

  1. Maxout networks.

In Proceedings of the 30th International Conference on International Conference on Machine Learning - Volume 28 (ICML'13), Sanjoy Dasgupta and David McAllester (Eds.), Vol. 28. JMLR.org III-1319-III-1327. https://arxiv.org/pdf/1302.4389.pdf

source
Flux.SkipConnection — Type
SkipConnection(layers, connection)

Creates a Skip Connection, of a layer or Chain of consecutive layers plus a shortcut connection. The connection function will combine the result of the layers with the original input, to give the final output.

The simplest 'ResNet'-type connection is just SkipConnection(layer, +), and requires the output of the layers to be the same shape as the input. Here is a more complicated example:

m = Conv((3,3), 4=>7, pad=(1,1))
 x = ones(5,5,4,10);
 size(m(x)) == (5, 5, 7, 10)
 
 sm = SkipConnection(m, (mx, x) -> cat(mx, x, dims=3))
-size(sm(x)) == (5, 5, 11, 10)
source

Normalisation & Regularisation

These layers don't affect the structure of the network but may improve training times or reduce overfitting.

Flux.BatchNorm — Type
BatchNorm(channels::Integer, σ = identity;
+size(sm(x)) == (5, 5, 11, 10)
source

Normalisation & Regularisation

These layers don't affect the structure of the network but may improve training times or reduce overfitting.

Flux.BatchNorm — Type
BatchNorm(channels::Integer, σ = identity;
           initβ = zeros, initγ = ones,
           ϵ = 1e-8, momentum = .1)

Batch Normalization layer. The channels input should be the size of the channel dimension in your data (see below).

Given an array with N dimensions, call the N-1th the channel dimension. (For a batch of feature vectors this is just the data dimension, for WHCN images it's the usual channel dimension.)

BatchNorm computes the mean and variance for each each W×H×1×N slice and shifts them to have a new mean and variance (corresponding to the learnable, per-channel bias and scale parameters).

Use testmode! during inference.

See Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift.

Example:

m = Chain(
   Dense(28^2, 64),
   BatchNorm(64, relu),
   Dense(64, 10),
   BatchNorm(10),
-  softmax)
source
Flux.Dropout — Type
Dropout(p, dims = :)

A Dropout layer. In the forward pass, applies the dropout function on the input.

Does nothing to the input once testmode! is true.

source
Flux.dropout — Function
dropout(p, dims = :)

Dropout function. For each input, either sets that input to 0 (with probability p) or scales it by 1/(1-p). The dims argument is to specify the unbroadcasted dimensions, i.e. dims=1 does dropout along columns and dims=2 along rows. This is used as a regularisation, i.e. it reduces overfitting during training.

See also Dropout.

source
Flux.AlphaDropout — Type
AlphaDropout(p)

A dropout layer. It is used in Self-Normalizing Neural Networks. (https://papers.nips.cc/paper/6698-self-normalizing-neural-networks.pdf) The AlphaDropout layer ensures that mean and variance of activations remains the same as before.

Does nothing to the input once testmode! is true.

source
Flux.LayerNorm — Type
LayerNorm(h::Integer)

A normalisation layer designed to be used with recurrent hidden states of size h. Normalises the mean/stddev of each input before applying a per-neuron gain/bias.

source
Flux.GroupNorm — Type

Group Normalization. This layer can outperform Batch-Normalization and Instance-Normalization.

GroupNorm(chs::Integer, G::Integer, λ = identity;
+  softmax)
source
Flux.Dropout — Type
Dropout(p, dims = :)

A Dropout layer. In the forward pass, applies the dropout function on the input.

Does nothing to the input once testmode! is true.

source
Flux.dropout — Function
dropout(p, dims = :)

Dropout function. For each input, either sets that input to 0 (with probability p) or scales it by 1/(1-p). The dims argument is to specify the unbroadcasted dimensions, i.e. dims=1 does dropout along columns and dims=2 along rows. This is used as a regularisation, i.e. it reduces overfitting during training.

See also Dropout.

source
Flux.AlphaDropout — Type
AlphaDropout(p)

A dropout layer. It is used in Self-Normalizing Neural Networks. (https://papers.nips.cc/paper/6698-self-normalizing-neural-networks.pdf) The AlphaDropout layer ensures that mean and variance of activations remains the same as before.

Does nothing to the input once testmode! is true.

source
Flux.LayerNorm — Type
LayerNorm(h::Integer)

A normalisation layer designed to be used with recurrent hidden states of size h. Normalises the mean/stddev of each input before applying a per-neuron gain/bias.

source
Flux.GroupNorm — Type

Group Normalization. This layer can outperform Batch-Normalization and Instance-Normalization.

GroupNorm(chs::Integer, G::Integer, λ = identity;
           initβ = (i) -> zeros(Float32, i), initγ = (i) -> ones(Float32, i),
           ϵ = 1f-5, momentum = 0.1f0)

$chs$ is the number of channels, the channel dimension of your input. For an array of N dimensions, the (N-1)th index is the channel dimension.

$G$ is the number of groups along which the statistics would be computed. The number of channels must be an integer multiple of the number of groups.

Use testmode! during inference.

Example:

m = Chain(Conv((3,3), 1=>32, leakyrelu;pad = 1),
-          GroupNorm(32,16)) # 32 channels, 16 groups (G = 16), thus 2 channels per group used

Link : https://arxiv.org/pdf/1803.08494.pdf

source

Testmode

Many normalisation layers behave differently under training and inference (testing). By default, Flux will automatically determine when a layer evaluation is part of training or inference. Still, depending on your use case, it may be helpful to manually specify when these layers should be treated as being trained or not. For this, Flux provides testmode!. When called on a model (e.g. a layer or chain of layers), this function will place the model into the mode specified.

Flux.testmode! — Function
testmode!(m, mode = true)

Set a layer or model's test mode (see below). Using :auto mode will treat any gradient computation as training.

Note: if you manually set a model into test mode, you need to manually place it back into train mode during training phase.

Possible values include:

  • false for training
  • true for testing
  • :auto or nothing for Flux to detect the mode automatically
source
Flux.trainmode! — Function
trainmode!(m, mode = true)

Set a layer of model's train mode (see below). Symmetric to testmode! (i.e. `trainmode!(m, mode) == testmode!(m, !mode)).

Note: if you manually set a model into train mode, you need to manually place it into test mode during testing phase.

Possible values include:

  • true for training
  • false for testing
  • :auto or nothing for Flux to detect the mode automatically
source

Cost Functions

Flux.mae — Function
mae(ŷ, y)

Return the mean of absolute error sum(abs.(ŷ .- y)) / length(y)

source
Flux.mse — Function
mse(ŷ, y)

Return the mean squared error sum((ŷ .- y).^2) / length(y).

source
Flux.msle — Function
msle(ŷ, y; ϵ=eps(eltype(ŷ)))

Returns the mean of the squared logarithmic errors sum((log.(ŷ .+ ϵ) .- log.(y .+ ϵ)).^2) / length(y). The ϵ term provides numerical stability.

This error penalizes an under-predicted estimate greater than an over-predicted estimate.

source
Flux.huber_loss — Function
huber_loss(ŷ, y; δ=1.0)

Computes the mean of the Huber loss given the prediction ŷ and true values y. By default, δ is set to 1.0.

                | 0.5*|ŷ - y|,   for |ŷ - y| <= δ
+          GroupNorm(32,16)) # 32 channels, 16 groups (G = 16), thus 2 channels per group used

Link : https://arxiv.org/pdf/1803.08494.pdf

source

Testmode

Many normalisation layers behave differently under training and inference (testing). By default, Flux will automatically determine when a layer evaluation is part of training or inference. Still, depending on your use case, it may be helpful to manually specify when these layers should be treated as being trained or not. For this, Flux provides testmode!. When called on a model (e.g. a layer or chain of layers), this function will place the model into the mode specified.

Flux.testmode! — Function
testmode!(m, mode = true)

Set a layer or model's test mode (see below). Using :auto mode will treat any gradient computation as training.

Note: if you manually set a model into test mode, you need to manually place it back into train mode during training phase.

Possible values include:

  • false for training
  • true for testing
  • :auto or nothing for Flux to detect the mode automatically
source
Flux.trainmode! — Function
trainmode!(m, mode = true)

Set a layer of model's train mode (see below). Symmetric to testmode! (i.e. `trainmode!(m, mode) == testmode!(m, !mode)).

Note: if you manually set a model into train mode, you need to manually place it into test mode during testing phase.

Possible values include:

  • true for training
  • false for testing
  • :auto or nothing for Flux to detect the mode automatically
source

Cost Functions

Flux.mae — Function
mae(ŷ, y)

Return the mean of absolute error sum(abs.(ŷ .- y)) / length(y)

source
Flux.mse — Function
mse(ŷ, y)

Return the mean squared error sum((ŷ .- y).^2) / length(y).

source
Flux.msle — Function
msle(ŷ, y; ϵ=eps(eltype(ŷ)))

Returns the mean of the squared logarithmic errors sum((log.(ŷ .+ ϵ) .- log.(y .+ ϵ)).^2) / length(y). The ϵ term provides numerical stability.

This error penalizes an under-predicted estimate greater than an over-predicted estimate.

source
Flux.huber_loss — Function
huber_loss(ŷ, y; δ=1.0)

Computes the mean of the Huber loss given the prediction ŷ and true values y. By default, δ is set to 1.0.

                | 0.5*|ŷ - y|,   for |ŷ - y| <= δ
   Hubber loss = |
-                |  δ*(|ŷ - y| - 0.5*δ),  otherwise

Huber Loss.

source
Flux.crossentropy — Function
crossentropy(ŷ, y; weight=1)

Return the crossentropy computed as -sum(y .* log.(ŷ) .* weight) / size(y, 2).

See also logitcrossentropy, binarycrossentropy.

source
Flux.logitcrossentropy — Function
logitcrossentropy(ŷ, y; weight=1)

Return the crossentropy computed after a softmax operation:

-sum(y .* logsoftmax(ŷ) .* weight) / size(y, 2)

See also crossentropy, binarycrossentropy.

source
Flux.binarycrossentropy — Function
binarycrossentropy(ŷ, y; ϵ=eps(ŷ))

Return -y*log(ŷ + ϵ) - (1-y)*log(1-ŷ + ϵ). The ϵ term provides numerical stability.

Typically, the prediction ŷ is given by the output of a sigmoid activation.

source
Flux.logitbinarycrossentropy — Function
logitbinarycrossentropy(ŷ, y)

logitbinarycrossentropy(ŷ, y) is mathematically equivalent to binarycrossentropy(σ(ŷ), y) but it is more numerically stable.

See also binarycrossentropy, sigmoid, logsigmoid.

source
Flux.kldivergence — Function
kldivergence(ŷ, y)

KLDivergence is a measure of how much one probability distribution is different from the other. It is always non-negative and zero only when both the distributions are equal everywhere.

KL Divergence.

source
Flux.poisson — Function
poisson(ŷ, y)

Poisson loss function is a measure of how the predicted distribution diverges from the expected distribution. Returns sum(ŷ .- y .* log.(ŷ)) / size(y, 2)

Poisson Loss.

source
Flux.hinge — Function
hinge(ŷ, y)

Measures the loss given the prediction ŷ and true labels y (containing 1 or -1). Returns sum((max.(0, 1 .- ŷ .* y))) / size(y, 2)

Hinge Loss See also squared_hinge.

source
Flux.squared_hinge — Function
squared_hinge(ŷ, y)

Computes squared hinge loss given the prediction ŷ and true labels y (conatining 1 or -1). Returns sum((max.(0, 1 .- ŷ .* y)).^2) / size(y, 2)

See also hinge.

source
Flux.dice_coeff_loss — Function
dice_coeff_loss(ŷ, y; smooth=1)

Loss function used in Image Segmentation. Calculates loss based on dice coefficient. Similar to F1_score. Returns 1 - 2*sum(|ŷ .* y| + smooth) / (sum(ŷ.^2) + sum(y.^2) + smooth)

V-Net: Fully Convolutional Neural Networks forVolumetric Medical Image Segmentation

source
Flux.tversky_loss — Function
tversky_loss(ŷ, y; β=0.7)

Used with imbalanced data to give more weightage to False negatives. Larger β weigh recall higher than precision (by placing more emphasis on false negatives) Returns 1 - sum(|y .* ŷ| + 1) / (sum(y .* ŷ + β*(1 .- y) .* ŷ + (1 - β)*y .* (1 .- ŷ)) + 1)

Tversky loss function for image segmentation using 3D fully convolutional deep networks

source
+ | δ*(|ŷ - y| - 0.5*δ), otherwise

Huber Loss.

source
Flux.crossentropy — Function
crossentropy(ŷ, y; weight=1)

Return the crossentropy computed as -sum(y .* log.(ŷ) .* weight) / size(y, 2).

See also logitcrossentropy, binarycrossentropy.

source
Flux.logitcrossentropy — Function
logitcrossentropy(ŷ, y; weight=1)

Return the crossentropy computed after a softmax operation:

-sum(y .* logsoftmax(ŷ) .* weight) / size(y, 2)

See also crossentropy, binarycrossentropy.

source
Flux.binarycrossentropy — Function
binarycrossentropy(ŷ, y; ϵ=eps(ŷ))

Return -y*log(ŷ + ϵ) - (1-y)*log(1-ŷ + ϵ). The ϵ term provides numerical stability.

Typically, the prediction ŷ is given by the output of a sigmoid activation.

source
Flux.logitbinarycrossentropy — Function
logitbinarycrossentropy(ŷ, y)

logitbinarycrossentropy(ŷ, y) is mathematically equivalent to binarycrossentropy(σ(ŷ), y) but it is more numerically stable.

See also binarycrossentropy, sigmoid, logsigmoid.

source
Flux.kldivergence — Function
kldivergence(ŷ, y)

KLDivergence is a measure of how much one probability distribution is different from the other. It is always non-negative and zero only when both the distributions are equal everywhere.

KL Divergence.

source
Flux.poisson — Function
poisson(ŷ, y)

Poisson loss function is a measure of how the predicted distribution diverges from the expected distribution. Returns sum(ŷ .- y .* log.(ŷ)) / size(y, 2)

Poisson Loss.

source
Flux.hinge — Function
hinge(ŷ, y)

Measures the loss given the prediction ŷ and true labels y (containing 1 or -1). Returns sum((max.(0, 1 .- ŷ .* y))) / size(y, 2)

Hinge Loss See also squared_hinge.

source
Flux.squared_hinge — Function
squared_hinge(ŷ, y)

Computes squared hinge loss given the prediction ŷ and true labels y (conatining 1 or -1). Returns sum((max.(0, 1 .- ŷ .* y)).^2) / size(y, 2)

See also hinge.

source
Flux.dice_coeff_loss — Function
dice_coeff_loss(ŷ, y; smooth=1)

Loss function used in Image Segmentation. Calculates loss based on dice coefficient. Similar to F1_score. Returns 1 - 2*sum(|ŷ .* y| + smooth) / (sum(ŷ.^2) + sum(y.^2) + smooth)

V-Net: Fully Convolutional Neural Networks forVolumetric Medical Image Segmentation

source
Flux.tversky_loss — Function
tversky_loss(ŷ, y; β=0.7)

Used with imbalanced data to give more weightage to False negatives. Larger β weigh recall higher than precision (by placing more emphasis on false negatives) Returns 1 - sum(|y .* ŷ| + 1) / (sum(y .* ŷ + β*(1 .- y) .* ŷ + (1 - β)*y .* (1 .- ŷ)) + 1)

Tversky loss function for image segmentation using 3D fully convolutional deep networks

source
diff --git a/dev/models/nnlib/index.html b/dev/models/nnlib/index.html index e64bcb60..a22217da 100644 --- a/dev/models/nnlib/index.html +++ b/dev/models/nnlib/index.html @@ -28,4 +28,4 @@ a = randomly sampled from uniform distribution U(l, u)

Randomized batched_adjoint(A)

Equivalent to applying transpose or adjoint to each matrix A[:,:,k].

These exist to control how batched_mul behaves, as it operated on such matrix slices of an array with ndims(A)==3.

BatchedTranspose{T, N, S} <: AbstractBatchedMatrix{T, N}
 BatchedAdjoint{T, N, S}

Lazy wrappers analogous to Transpose and Adjoint, returned by batched_transpose

NNlib.batched_transpose — Function
batched_transpose(A::AbstractArray{T,3})
 batched_adjoint(A)

Equivalent to applying transpose or adjoint to each matrix A[:,:,k].

These exist to control how batched_mul behaves, as it operated on such matrix slices of an array with ndims(A)==3.

BatchedTranspose{T, N, S} <: AbstractBatchedMatrix{T, N}
-BatchedAdjoint{T, N, S}

Lazy wrappers analogous to Transpose and Adjoint, returned by batched_transpose

+BatchedAdjoint{T, N, S}

Lazy wrappers analogous to Transpose and Adjoint, returned by batched_transpose

diff --git a/dev/models/recurrence/index.html b/dev/models/recurrence/index.html index 6ab6e79c..bf7c75af 100644 --- a/dev/models/recurrence/index.html +++ b/dev/models/recurrence/index.html @@ -39,4 +39,4 @@ m = Flux.Recur(rnn, h) y = m(x)

The Recur wrapper stores the state between runs in the m.state field.

If you use the RNN(10, 5) constructor – as opposed to RNNCell – you'll see that it's simply a wrapped cell.

julia> RNN(10, 5)
 Recur(RNNCell(10, 5, tanh))

Sequences

Often we want to work with sequences of inputs, rather than individual xs.

seq = [rand(10) for i = 1:10]

With Recur, applying our model to each element of a sequence is trivial:

m.(seq) # returns a list of 5-element vectors

This works even when we've chain recurrent layers into a larger model.

m = Chain(LSTM(10, 15), Dense(15, 5))
-m.(seq)

Finally, we can reset the hidden state of the cell back to its initial value using reset!(m).

+m.(seq)

Finally, we can reset the hidden state of the cell back to its initial value using reset!(m).

diff --git a/dev/models/regularisation/index.html b/dev/models/regularisation/index.html index 8e447fbc..436ca24c 100644 --- a/dev/models/regularisation/index.html +++ b/dev/models/regularisation/index.html @@ -36,4 +36,4 @@ julia> activations(c, rand(10)) Float32[0.5192045, 0.48079553] julia> sum(norm, ans) -2.1166067f0 +2.1166067f0 diff --git a/dev/performance/index.html b/dev/performance/index.html index d6852fee..f78186f8 100644 --- a/dev/performance/index.html +++ b/dev/performance/index.html @@ -17,4 +17,4 @@ y_batch = reduce(hcat, ys) function loss_total(x_batch::Matrix, y_batch::Matrix) y_preds = model(x_batch) sum(loss.(y_preds, y_batch)) -end

When doing this kind of concatenation use reduce(hcat, xs) rather than hcat(xs...). This will avoid the splatting penalty, and will hit the optimised reduce method.

+end

When doing this kind of concatenation use reduce(hcat, xs) rather than hcat(xs...). This will avoid the splatting penalty, and will hit the optimised reduce method.

diff --git a/dev/saving/index.html b/dev/saving/index.html index 6fee2713..f374077c 100644 --- a/dev/saving/index.html +++ b/dev/saving/index.html @@ -47,4 +47,4 @@ evalcb = throttle(30) do # Show loss @save "model-checkpoint.bson" model end

This will update the "model-checkpoint.bson" file every thirty seconds.

You can get more advanced by saving a series of models throughout training, for example

@save "model-$(now()).bson" model

will produce a series of models like "model-2018-03-06T02:57:10.41.bson". You could also store the current test set loss, so that it's easy to (for example) revert to an older copy of the model if it starts to overfit.

@save "model-$(now()).bson" model loss = testloss()

You can even store optimiser state alongside the model, to resume training exactly where you left off.

opt = ADAM()
-@save "model-$(now()).bson" model opt
+@save "model-$(now()).bson" model opt diff --git a/dev/search/index.html b/dev/search/index.html index 58a085bd..6598f4bb 100644 --- a/dev/search/index.html +++ b/dev/search/index.html @@ -6,4 +6,4 @@ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) ga('create', 'UA-36890222-9', 'auto'); ga('send', 'pageview', {'page': location.pathname + location.search + location.hash}); -

Loading search...

    +

    Loading search...

      diff --git a/dev/training/optimisers/index.html b/dev/training/optimisers/index.html index 4da0e259..a13a7805 100644 --- a/dev/training/optimisers/index.html +++ b/dev/training/optimisers/index.html @@ -28,7 +28,7 @@ end

      Running this will alter the parameters W and

      An optimiser update! accepts a parameter and a gradient, and updates the parameter according to the chosen rule. We can also pass opt to our training loop, which will update all parameters of the model in a loop. However, we can now easily replace Descent with a more advanced optimiser such as ADAM.

      Optimiser Reference

      All optimisers return an object that, when passed to train!, will update the parameters passed to it.

      Flux.Optimise.update! — Function
      update!(opt, p, g)
-update!(opt, ps::Params, gs)

      Perform an update step of the parameters ps (or the single parameter p) according to optimizer opt and the gradients gs (the gradient g).

      As a result, the parameters are mutated and the optimizer's internal state may change.

      update!(x, x̄)

      Update the array x according to x .-= x̄.

      source
      Flux.Optimise.Descent — Type
      Descent(η)

      Classic gradient descent optimiser with learning rate η. For each parameter p and its gradient δp, this runs p -= η*δp

      Parameters

      • Learning Rate (η): The amount by which the gradients are discounted before updating the weights. Defaults to 0.1.

      Example

      opt = Descent() # uses default η (0.1)
+update!(opt, ps::Params, gs)

      Perform an update step of the parameters ps (or the single parameter p) according to optimizer opt and the gradients gs (the gradient g).

      As a result, the parameters are mutated and the optimizer's internal state may change.

      update!(x, x̄)

      Update the array x according to x .-= x̄.

      source
      Flux.Optimise.Descent — Type
      Descent(η)

      Classic gradient descent optimiser with learning rate η. For each parameter p and its gradient δp, this runs p -= η*δp

      Parameters

      • Learning Rate (η): The amount by which the gradients are discounted before updating the weights. Defaults to 0.1.

      Example

      opt = Descent() # uses default η (0.1)
 
 opt = Descent(0.3) # use provided η
 
@@ -38,23 +38,23 @@ gs = gradient(ps) do
   loss(x, y)
 end
 
-Flux.Optimise.update!(opt, ps, gs)
      source
      Flux.Optimise.Momentum — Type
      Momentum(η, ρ)

      Gradient descent with learning rate η and momentum ρ.

      Parameters

      • Learning Rate (η): Amount by which gradients are discounted before updating the weights. Defaults to 0.01.
      • Momentum (ρ): Parameter that accelerates descent in the relevant direction and dampens oscillations. Defaults to 0.9.

      Examples

      opt = Momentum() # uses defaults of η = 0.01 and ρ = 0.9
+Flux.Optimise.update!(opt, ps, gs)
      source
      Flux.Optimise.Momentum — Type
      Momentum(η, ρ)

      Gradient descent with learning rate η and momentum ρ.

      Parameters

      • Learning Rate (η): Amount by which gradients are discounted before updating the weights. Defaults to 0.01.
      • Momentum (ρ): Parameter that accelerates descent in the relevant direction and dampens oscillations. Defaults to 0.9.

      Examples

      opt = Momentum() # uses defaults of η = 0.01 and ρ = 0.9
 
-opt = Momentum(0.01, 0.99)
      source
      Flux.Optimise.Nesterov — Type
      Nesterov(η, ρ)

      Gradient descent with learning rate η and Nesterov momentum ρ.

      Parameters

      • Learning Rate (η): Amount by which the gradients are dicsounted berfore updating the weights. Defaults to 0.001.
      • Nesterov Momentum (ρ): Parameters controlling the amount of nesterov momentum to be applied. Defaults to 0.9.

      Examples

      opt = Nesterov() # uses defaults η = 0.001 and ρ = 0.9
+opt = Momentum(0.01, 0.99)
      source
      Flux.Optimise.Nesterov — Type
      Nesterov(η, ρ)

      Gradient descent with learning rate η and Nesterov momentum ρ.

      Parameters

      • Learning Rate (η): Amount by which the gradients are dicsounted berfore updating the weights. Defaults to 0.001.
      • Nesterov Momentum (ρ): Parameters controlling the amount of nesterov momentum to be applied. Defaults to 0.9.

      Examples

      opt = Nesterov() # uses defaults η = 0.001 and ρ = 0.9
 
-opt = Nesterov(0.003, 0.95)
      source
      Flux.Optimise.RMSProp — Type
      RMSProp(η, ρ)

      Implements the RMSProp algortihm. Often a good choice for recurrent networks. Parameters other than learning rate generally don't need tuning.

      Parameters

      • Learning Rate (η): Defaults to 0.001.
      • Rho (ρ): Defaults to 0.9.

      Examples

      opt = RMSProp() # uses default η = 0.001 and ρ = 0.9
+opt = Nesterov(0.003, 0.95)
      source
      Flux.Optimise.RMSProp — Type
      RMSProp(η, ρ)

      Implements the RMSProp algortihm. Often a good choice for recurrent networks. Parameters other than learning rate generally don't need tuning.

      Parameters

      • Learning Rate (η): Defaults to 0.001.
      • Rho (ρ): Defaults to 0.9.

      Examples

      opt = RMSProp() # uses default η = 0.001 and ρ = 0.9
 
-opt = RMSProp(0.002, 0.95)

      References

      RMSProp

      source
      Flux.Optimise.ADAM — Type
      ADAM(η, β::Tuple)

      Implements the ADAM optimiser.

      Paramters

      • Learning Rate (η): Defaults to 0.001.
      • Beta (β::Tuple): The first element refers to β1 and the second to β2. Defaults to (0.9, 0.999).

      Examples

      opt = ADAM() # uses the default η = 0.001 and β = (0.9, 0.999)
+opt = RMSProp(0.002, 0.95)

      References

      RMSProp

      source
      Flux.Optimise.ADAM — Type
      ADAM(η, β::Tuple)

      Implements the ADAM optimiser.

      Paramters

      • Learning Rate (η): Defaults to 0.001.
      • Beta (β::Tuple): The first element refers to β1 and the second to β2. Defaults to (0.9, 0.999).

      Examples

      opt = ADAM() # uses the default η = 0.001 and β = (0.9, 0.999)
 
-opt = ADAM(0.001, (0.9, 0.8))

      References

      ADAM optimiser.

      source
      Flux.Optimise.AdaMax — Type
      AdaMax(η, β::Tuple)

      Variant of ADAM based on ∞-norm.

      Parameters

      • Learning Rate (η): Defaults to 0.001
      • Beta (β::Tuple): The first element refers to β1 and the second to β2. Defaults to (0.9, 0.999).

      Examples

      opt = AdaMax() # uses default η and β
+opt = ADAM(0.001, (0.9, 0.8))

      References

      ADAM optimiser.

      source
      Flux.Optimise.AdaMax — Type
      AdaMax(η, β::Tuple)

      Variant of ADAM based on ∞-norm.

      Parameters

      • Learning Rate (η): Defaults to 0.001
      • Beta (β::Tuple): The first element refers to β1 and the second to β2. Defaults to (0.9, 0.999).

      Examples

      opt = AdaMax() # uses default η and β
 
-opt = AdaMax(0.001, (0.9, 0.995))

      References

      AdaMax optimiser.

      source
      Flux.Optimise.ADAGrad — Type
      ADAGrad(η)

      Implements AdaGrad. It has parameter specific learning rates based on how frequently it is updated.

      Parameters

      • Learning Rate (η): Defaults to 0.1

      Examples

      opt = ADAGrad() # uses default η = 0.1
+opt = AdaMax(0.001, (0.9, 0.995))

      References

      AdaMax optimiser.

      source
      Flux.Optimise.ADAGrad — Type
      ADAGrad(η)

      Implements AdaGrad. It has parameter specific learning rates based on how frequently it is updated.

      Parameters

      • Learning Rate (η): Defaults to 0.1

      Examples

      opt = ADAGrad() # uses default η = 0.1
 
-opt = ADAGrad(0.001)

      References

      ADAGrad optimiser. Parameters don't need tuning.

      source
      Flux.Optimise.ADADelta — Type
      ADADelta(ρ)

      Version of ADAGrad that adapts learning rate based on a window of past gradient updates. Parameters don't need tuning.

      Parameters

      • Rho (ρ): Factor by which gradient is decayed at each time step. Defaults to 0.9.

      Examples

      opt = ADADelta() # uses default ρ = 0.9
-opt = ADADelta(0.89)

      References

      ADADelta optimiser.

      source
      Flux.Optimise.AMSGrad — Type
      AMSGrad(η, β::Tuple)

      Implements AMSGrad version of the ADAM optimiser. Parameters don't need tuning.

      Parameters

      • Learning Rate (η): Defaults to 0.001.
      • Beta (β::Tuple): The first element refers to β1 and the second to β2. Defaults to (0.9, 0.999).

      Examples

      opt = AMSGrad() # uses default η and β
-opt = AMSGrad(0.001, (0.89, 0.995))

      References

      AMSGrad optimiser.

      source
      Flux.Optimise.NADAM — Type
      NADAM(η, β::Tuple)

      Nesterov variant of ADAM. Parameters don't need tuning.

      Parameters

      • Learning Rate (η): Defaults to 0.001.
      • Beta (β::Tuple): The first element refers to β1 and the second to β2. Defaults to (0.9, 0.999).

      Examples

      opt = NADAM() # uses default η and β
-opt = NADAM(0.002, (0.89, 0.995))

      References

      NADAM optimiser.

      source
      Flux.Optimise.ADAMW — Function
      ADAMW(η, β::Tuple, decay)

      Variant of ADAM defined by fixing weight decay regularization.

      Parameters

      • Learning Rate (η): Defaults to 0.001.
      • Beta (β::Tuple): The first element refers to β1 and the second to β2. Defaults to (0.9, 0.999).
      • decay: Decay applied to weights during optimisation. Defaults to 0.

      Examples

      opt = ADAMW() # uses default η, β and decay
-opt = ADAMW(0.001, (0.89, 0.995), 0.1)

      References

      ADAMW

      source

      Optimiser Interface

      Flux's optimisers are built around a struct that holds all the optimiser parameters along with a definition of how to apply the update rule associated with it. We do this via the apply! function which takes the optimiser as the first argument followed by the parameter and its corresponding gradient.

      In this manner Flux also allows one to create custom optimisers to be used seamlessly. Let's work this with a simple example.

      mutable struct Momentum
+opt = ADAGrad(0.001)

      References

      ADAGrad optimiser. Parameters don't need tuning.

      source
      Flux.Optimise.ADADelta — Type
      ADADelta(ρ)

      Version of ADAGrad that adapts learning rate based on a window of past gradient updates. Parameters don't need tuning.

      Parameters

      • Rho (ρ): Factor by which gradient is decayed at each time step. Defaults to 0.9.

      Examples

      opt = ADADelta() # uses default ρ = 0.9
+opt = ADADelta(0.89)

      References

      ADADelta optimiser.

      source
      Flux.Optimise.AMSGrad — Type
      AMSGrad(η, β::Tuple)

      Implements AMSGrad version of the ADAM optimiser. Parameters don't need tuning.

      Parameters

      • Learning Rate (η): Defaults to 0.001.
      • Beta (β::Tuple): The first element refers to β1 and the second to β2. Defaults to (0.9, 0.999).

      Examples

      opt = AMSGrad() # uses default η and β
+opt = AMSGrad(0.001, (0.89, 0.995))

      References

      AMSGrad optimiser.

      source
      Flux.Optimise.NADAM — Type
      NADAM(η, β::Tuple)

      Nesterov variant of ADAM. Parameters don't need tuning.

      Parameters

      • Learning Rate (η): Defaults to 0.001.
      • Beta (β::Tuple): The first element refers to β1 and the second to β2. Defaults to (0.9, 0.999).

      Examples

      opt = NADAM() # uses default η and β
+opt = NADAM(0.002, (0.89, 0.995))

      References

      NADAM optimiser.

      source
      Flux.Optimise.ADAMW — Function
      ADAMW(η, β::Tuple, decay)

      Variant of ADAM defined by fixing weight decay regularization.

      Parameters

      • Learning Rate (η): Defaults to 0.001.
      • Beta (β::Tuple): The first element refers to β1 and the second to β2. Defaults to (0.9, 0.999).
      • decay: Decay applied to weights during optimisation. Defaults to 0.

      Examples

      opt = ADAMW() # uses default η, β and decay
+opt = ADAMW(0.001, (0.89, 0.995), 0.1)

      References

      ADAMW

      source

      Optimiser Interface

      Flux's optimisers are built around a struct that holds all the optimiser parameters along with a definition of how to apply the update rule associated with it. We do this via the apply! function which takes the optimiser as the first argument followed by the parameter and its corresponding gradient.

      In this manner Flux also allows one to create custom optimisers to be used seamlessly. Let's work this with a simple example.

      mutable struct Momentum
   eta
   rho
   velocity
@@ -81,4 +81,4 @@ for t = 1:10^5
 end
 
 loss(rand(10)) # around 0.9

      In this manner it is possible to compose optimisers for some added flexibility.

      Decays

      Similar to optimisers, Flux also defines some simple decays that can be used in conjunction with other optimisers, or standalone.

      Flux.Optimise.ExpDecay — Type
      ExpDecay(eta, decay, decay_step, clip)

      Discount the learning rate eta by a multiplicative factor decay every decay_step till a minimum of clip.

      Parameters

      • Learning Rate (eta): Defaults to 0.001.
      • decay: Factor by which the learning rate is discounted. Defaults to 0.1.
      • decay_step: Schedules decay operations by setting number of steps between two decay operations. Defaults to 1000.
      • clip: Minimum value of learning rate. Defaults to 1e-4.

      Example

      To apply exponential decay to an optimiser:

      Optimiser(ExpDecay(..), Opt(..))
-opt = Optimiser(ExpDecay(), ADAM())
      source
      Flux.Optimise.InvDecay — Type
      InvDecay(γ)

      Applies inverse time decay to an optimiser, i.e., the effective step size at iteration n is eta / (1 + γ * n) where eta is the initial step size. The wrapped optimiser's step size is not modified.

      Parameters

      • gamma (γ): Defaults to 0.001

      Example

      Optimiser(InvDecay(..), Opt(..))
      source
      Flux.Optimise.WeightDecay — Type
      WeightDecay(wd)

      Decays the weight by wd

      Parameters

      • weight decay (wd): 0
      source
      +opt = Optimiser(ExpDecay(), ADAM())source
      Flux.Optimise.InvDecay — Type
      InvDecay(γ)

      Applies inverse time decay to an optimiser, i.e., the effective step size at iteration n is eta / (1 + γ * n) where eta is the initial step size. The wrapped optimiser's step size is not modified.

      Parameters

      • gamma (γ): Defaults to 0.001

      Example

      Optimiser(InvDecay(..), Opt(..))
      source
      Flux.Optimise.WeightDecay — Type
      WeightDecay(wd)

      Decays the weight by wd

      Parameters

      • weight decay (wd): 0
      source
      diff --git a/dev/training/training/index.html b/dev/training/training/index.html index 9cf9e50a..55efbc2e 100644 --- a/dev/training/training/index.html +++ b/dev/training/training/index.html @@ -6,7 +6,7 @@ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) ga('create', 'UA-36890222-9', 'auto'); ga('send', 'pageview', {'page': location.pathname + location.search + location.hash}); -
      Edit on GitHub

      Training

      To actually train a model we need four things:

      • A objective function, that evaluates how well a model is doing given some input data.
      • The trainable parameters of the model.
      • A collection of data points that will be provided to the objective function.
      • An optimiser that will update the model parameters appropriately.

      With these we can call train!:

      Flux.Optimise.train! — Function
      train!(loss, params, data, opt; cb)

      For each datapoint d in data computes the gradient of loss(d...) through backpropagation and calls the optimizer opt.

      In case datapoints d are of numeric array type, assumes no splatting is needed and computes the gradient of loss(d).

      Takes a callback as keyword argument cb. For example, this will print "training" every 10 seconds:

      train!(loss, params, data, opt, cb = throttle(() -> println("training"), 10))

      The callback can call Flux.stop() to interrupt the training loop.

      Multiple optimisers and callbacks can be passed to opt and cb as arrays.

      source

      There are plenty of examples in the model zoo.

      Loss Functions

      The objective function must return a number representing how far the model is from its target – the loss of the model. The loss function that we defined in basics will work as an objective. We can also define an objective in terms of some model:

      m = Chain(
+
      Edit on GitHub
      Training
      To actually train a model we need four things:
      • A objective function, that evaluates how well a model is doing given some input data.
      • The trainable parameters of the model.
      • A collection of data points that will be provided to the objective function.
      • An optimiser that will update the model parameters appropriately.
      With these we can call train!:
      Flux.Optimise.train! — Function
      train!(loss, params, data, opt; cb)
      For each datapoint d in data computes the gradient of loss(d...) through backpropagation and calls the optimizer opt.
      In case datapoints d are of numeric array type, assumes no splatting is needed  and computes the gradient of loss(d).
      Takes a callback as keyword argument cb. For example, this will print "training" every 10 seconds:
      train!(loss, params, data, opt,          cb = throttle(() -> println("training"), 10))
      The callback can call Flux.stop() to interrupt the training loop.
      Multiple optimisers and callbacks can be passed to opt and cb as arrays.
      source
      There are plenty of examples in the model zoo.
      Loss Functions
      The objective function must return a number representing how far the model is from its target – the loss of the model. The loss function that we defined in basics will work as an objective. We can also define an objective in terms of some model:
      m = Chain(
   Dense(784, 32, σ),
   Dense(32, 10), softmax)
 
@@ -51,4 +51,4 @@ end

      You could simplify this further, for example by hard-coding in the loss function.

      +end

      You could simplify this further, for example by hard-coding in the loss function.