diff --git a/previews/PR1150/community/index.html b/previews/PR1150/community/index.html index ba722991..e5095872 100644 --- a/previews/PR1150/community/index.html +++ b/previews/PR1150/community/index.html @@ -6,4 +6,4 @@ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) ga('create', 'UA-36890222-9', 'auto'); ga('send', 'pageview', {'page': location.pathname + location.search + location.hash}); -
Edit on GitHub

Community

All Flux users are welcome to join our community on the Julia forum, or the slack (channel #machine-learning). If you have questions or issues we'll try to help you out.

If you're interested in hacking on Flux, the source code is open and easy to understand – it's all just the same Julia code you work with normally. You might be interested in our intro issues to get started.

+
Edit on GitHub

Community

All Flux users are welcome to join our community on the Julia forum, or the slack (channel #machine-learning). If you have questions or issues we'll try to help you out.

If you're interested in hacking on Flux, the source code is open and easy to understand – it's all just the same Julia code you work with normally. You might be interested in our intro issues to get started.

diff --git a/previews/PR1150/data/dataloader/index.html b/previews/PR1150/data/dataloader/index.html index e5da2a43..ce85e216 100644 --- a/previews/PR1150/data/dataloader/index.html +++ b/previews/PR1150/data/dataloader/index.html @@ -29,4 +29,4 @@ end # train for 10 epochs using IterTools: ncycle -Flux.train!(loss, ps, ncycle(train_loader, 10), opt)source +Flux.train!(loss, ps, ncycle(train_loader, 10), opt)source diff --git a/previews/PR1150/data/onehot/index.html b/previews/PR1150/data/onehot/index.html index 0144ddf8..fa959ee1 100644 --- a/previews/PR1150/data/onehot/index.html +++ b/previews/PR1150/data/onehot/index.html @@ -55,4 +55,4 @@ julia> onecold(ans, [:a, :b, :c]) 3×3 Flux.OneHotMatrix{Array{Flux.OneHotVector,1}}: 0 1 0 1 0 1 - 0 0 0source + 0 0 0source diff --git a/previews/PR1150/datasets/index.html b/previews/PR1150/datasets/index.html index 2194b452..177914d4 100644 --- a/previews/PR1150/datasets/index.html +++ b/previews/PR1150/datasets/index.html @@ -26,4 +26,4 @@ julia> labels[1] images(:test)

Load the MNIST images.

Each image is a 28×28 array of Gray colour values (see Colors.jl).

Return the 60,000 training images by default; pass :test to retrieve the 10,000 test images.

source
Flux.Data.MNIST.labels — Method
labels()
 labels(:test)

Load the labels corresponding to each of the images returned from images(). Each label is a number from 0-9.

Return the 60,000 training labels by default; pass :test to retrieve the 10,000 test labels.

source
Flux.Data.FashionMNIST.images — Method
images()
 images(:test)

Load the Fashion-MNIST images.

Each image is a 28×28 array of Gray colour values (see Colors.jl).

Return the 60,000 training images by default; pass :test to retrieve the 10,000 test images.

source
Flux.Data.FashionMNIST.labels — Method
labels()
-labels(:test)

Load the labels corresponding to each of the images returned from images(). Each label is a number from 0-9.

Return the 60,000 training labels by default; pass :test to retrieve the 10,000 test labels.

source
Flux.Data.CMUDict.phones — Method
phones()

Return a Vector containing the phones used in the CMU Pronouncing Dictionary.

source
Flux.Data.CMUDict.symbols — Method
symbols()

Return a Vector containing the symbols used in the CMU Pronouncing Dictionary. A symbol is a phone with optional auxiliary symbols, indicating for example the amount of stress on the phone.

source
Flux.Data.CMUDict.rawdict — Method
rawdict()

Return the unfiltered CMU Pronouncing Dictionary.

source
Flux.Data.CMUDict.cmudict — Method
cmudict()

Return a filtered CMU Pronouncing Dictionary.

It is filtered so each word contains only ASCII characters and a combination of word characters (as determined by the regex engine using \w), '-' and '.'.

source
Flux.Data.Sentiment.train — Method
train()

Return the train split of the Stanford Sentiment Treebank. The data is in treebank format.

source
Flux.Data.Sentiment.test — Method
test()

Return the test split of the Stanford Sentiment Treebank. The data is in treebank format.

source
Flux.Data.Sentiment.dev — Method
dev()

Return the dev split of the Stanford Sentiment Treebank. The data is in treebank format.

source
+labels(:test)

Load the labels corresponding to each of the images returned from images(). Each label is a number from 0-9.

Return the 60,000 training labels by default; pass :test to retrieve the 10,000 test labels.

source
Flux.Data.CMUDict.phones — Method
phones()

Return a Vector containing the phones used in the CMU Pronouncing Dictionary.

source
Flux.Data.CMUDict.symbols — Method
symbols()

Return a Vector containing the symbols used in the CMU Pronouncing Dictionary. A symbol is a phone with optional auxiliary symbols, indicating for example the amount of stress on the phone.

source
Flux.Data.CMUDict.rawdict — Method
rawdict()

Return the unfiltered CMU Pronouncing Dictionary.

source
Flux.Data.CMUDict.cmudict — Method
cmudict()

Return a filtered CMU Pronouncing Dictionary.

It is filtered so each word contains only ASCII characters and a combination of word characters (as determined by the regex engine using \w), '-' and '.'.

source
Flux.Data.Sentiment.train — Method
train()

Return the train split of the Stanford Sentiment Treebank. The data is in treebank format.

source
Flux.Data.Sentiment.test — Method
test()

Return the test split of the Stanford Sentiment Treebank. The data is in treebank format.

source
Flux.Data.Sentiment.dev — Method
dev()

Return the dev split of the Stanford Sentiment Treebank. The data is in treebank format.

source
diff --git a/previews/PR1150/ecosystem/index.html b/previews/PR1150/ecosystem/index.html index 9d2a271e..3a18de55 100644 --- a/previews/PR1150/ecosystem/index.html +++ b/previews/PR1150/ecosystem/index.html @@ -6,4 +6,4 @@ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) ga('create', 'UA-36890222-9', 'auto'); ga('send', 'pageview', {'page': location.pathname + location.search + location.hash}); -
Edit on GitHub

The Julia Ecosystem

One of the main strengths of Julia lies in an ecosystem of packages globally providing a rich and consistent user experience.

This is a non-exhaustive list of Julia packages, nicely complementing Flux in typical machine learning and deep learning workflows:

This tight integration among Julia pakages is shown in some of the examples in the model-zoo repository.

+
Edit on GitHub

The Julia Ecosystem

One of the main strengths of Julia lies in an ecosystem of packages globally providing a rich and consistent user experience.

This is a non-exhaustive list of Julia packages, nicely complementing Flux in typical machine learning and deep learning workflows:

This tight integration among Julia pakages is shown in some of the examples in the model-zoo repository.

diff --git a/previews/PR1150/gpu/index.html b/previews/PR1150/gpu/index.html index 5109cc56..edea11e0 100644 --- a/previews/PR1150/gpu/index.html +++ b/previews/PR1150/gpu/index.html @@ -47,4 +47,4 @@ julia> x |> cpu 10-element Array{Float32,1}: 0.235164 ⋮ - 0.192538 + 0.192538 diff --git a/previews/PR1150/index.html b/previews/PR1150/index.html index 49b54a5e..cdd41865 100644 --- a/previews/PR1150/index.html +++ b/previews/PR1150/index.html @@ -6,4 +6,4 @@ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) ga('create', 'UA-36890222-9', 'auto'); ga('send', 'pageview', {'page': location.pathname + location.search + location.hash}); -
Edit on GitHub

Flux: The Julia Machine Learning Library

Flux is a library for machine learning. It comes "batteries-included" with many useful tools built in, but also lets you use the full power of the Julia language where you need it. We follow a few key principles:

  • Doing the obvious thing. Flux has relatively few explicit APIs for features like regularisation or embeddings. Instead, writing down the mathematical form will work – and be fast.
  • You could have written Flux. All of it, from LSTMs to GPU kernels, is straightforward Julia code. When in doubt, it’s well worth looking at the source. If you need something different, you can easily roll your own.
  • Play nicely with others. Flux works well with Julia libraries from data frames and images to differential equation solvers, so you can easily build complex data processing pipelines that integrate Flux models.

Installation

Download Julia 1.0 or later, if you haven't already. You can add Flux from using Julia's package manager, by typing ] add Flux in the Julia prompt.

If you have CUDA you can also run ] add CuArrays to get GPU support; see here for more details.

Learning Flux

There are several different ways to learn Flux. If you just want to get started writing models, the model zoo gives good starting points for many common ones. This documentation provides a reference to all of Flux's APIs, as well as a from-scratch introduction to Flux's take on models and how they work. Once you understand these docs, congratulations, you also understand Flux's source code, which is intended to be concise, legible and a good reference for more advanced concepts.

+
Edit on GitHub

Flux: The Julia Machine Learning Library

Flux is a library for machine learning. It comes "batteries-included" with many useful tools built in, but also lets you use the full power of the Julia language where you need it. We follow a few key principles:

  • Doing the obvious thing. Flux has relatively few explicit APIs for features like regularisation or embeddings. Instead, writing down the mathematical form will work – and be fast.
  • You could have written Flux. All of it, from LSTMs to GPU kernels, is straightforward Julia code. When in doubt, it’s well worth looking at the source. If you need something different, you can easily roll your own.
  • Play nicely with others. Flux works well with Julia libraries from data frames and images to differential equation solvers, so you can easily build complex data processing pipelines that integrate Flux models.

Installation

Download Julia 1.0 or later, if you haven't already. You can add Flux from using Julia's package manager, by typing ] add Flux in the Julia prompt.

If you have CUDA you can also run ] add CuArrays to get GPU support; see here for more details.

Learning Flux

There are several different ways to learn Flux. If you just want to get started writing models, the model zoo gives good starting points for many common ones. This documentation provides a reference to all of Flux's APIs, as well as a from-scratch introduction to Flux's take on models and how they work. Once you understand these docs, congratulations, you also understand Flux's source code, which is intended to be concise, legible and a good reference for more advanced concepts.

diff --git a/previews/PR1150/models/advanced/index.html b/previews/PR1150/models/advanced/index.html index 8c835514..6bc86eca 100644 --- a/previews/PR1150/models/advanced/index.html +++ b/previews/PR1150/models/advanced/index.html @@ -24,4 +24,4 @@ Params([[0.66722 0.774872 0.249809; 0.843321 0.403843 0.429232; 0.683525 0.66245 ) ps = Flux.params(m[3:end])

The Zygote.Params object ps now holds a reference to only the parameters of the layers passed to it.

During training, the gradients will only be computed for (and applied to) the last Dense layer, therefore only that would have its parameters changed.

Flux.params also takes multiple inputs to make it easy to collect parameters from heterogenous models with a single call. A simple demonstration would be if we wanted to omit optimising the second Dense layer in the previous example. It would look something like this:

Flux.params(m[1], m[3:end])

Sometimes, a more fine-tuned control is needed. We can freeze a specific parameter of a specific layer which already entered a Params object ps, by simply deleting it from ps:

ps = params(m)
-delete!(ps, m[2].b)
+delete!(ps, m[2].b) diff --git a/previews/PR1150/models/basics/index.html b/previews/PR1150/models/basics/index.html index e178cfae..a233a2bf 100644 --- a/previews/PR1150/models/basics/index.html +++ b/previews/PR1150/models/basics/index.html @@ -115,4 +115,4 @@ outdims(m, (10, 10)) == (6, 6)source
outdims(l::Conv, isize::Tuple)

Calculate the output dimensions given the input dimensions isize. Batch size and channel size are ignored as per NNlib.jl.

m = Conv((3, 3), 3 => 16)
 outdims(m, (10, 10)) == (8, 8)
-outdims(m, (10, 10, 1, 3)) == (8, 8)
source
+outdims(m, (10, 10, 1, 3)) == (8, 8)source diff --git a/previews/PR1150/models/layers/index.html b/previews/PR1150/models/layers/index.html index ca792180..6c157c06 100644 --- a/previews/PR1150/models/layers/index.html +++ b/previews/PR1150/models/layers/index.html @@ -71,4 +71,4 @@ julia> Flux.normalise(a, dims=2) initβ = (i) -> zeros(Float32, i), initγ = (i) -> ones(Float32, i), ϵ = 1f-5, momentum = 0.1f0)

Group Normalization layer. This layer can outperform Batch Normalization and Instance Normalization.

chs is the number of channels, the channel dimension of your input. For an array of N dimensions, the N-1th index is the channel dimension.

G is the number of groups along which the statistics are computed. The number of channels must be an integer multiple of the number of groups.

Use testmode! during inference.

Examples

m = Chain(Conv((3,3), 1=>32, leakyrelu;pad = 1),
           GroupNorm(32,16))
-          # 32 channels, 16 groups (G = 16), thus 2 channels per group used
source

Testmode

Many normalisation layers behave differently under training and inference (testing). By default, Flux will automatically determine when a layer evaluation is part of training or inference. Still, depending on your use case, it may be helpful to manually specify when these layers should be treated as being trained or not. For this, Flux provides Flux.testmode!. When called on a model (e.g. a layer or chain of layers), this function will place the model into the mode specified.

Flux.testmode! — Function
testmode!(m, mode = true)

Set a layer or model's test mode (see below). Using :auto mode will treat any gradient computation as training.

Note: if you manually set a model into test mode, you need to manually place it back into train mode during training phase.

Possible values include:

  • false for training
  • true for testing
  • :auto or nothing for Flux to detect the mode automatically
source
Flux.trainmode! — Function
trainmode!(m, mode = true)

Set a layer of model's train mode (see below). Symmetric to testmode! (i.e. `trainmode!(m, mode) == testmode!(m, !mode)).

Note: if you manually set a model into train mode, you need to manually place it into test mode during testing phase.

Possible values include:

  • true for training
  • false for testing
  • :auto or nothing for Flux to detect the mode automatically
source
+ # 32 channels, 16 groups (G = 16), thus 2 channels per group usedsource

Testmode

Many normalisation layers behave differently under training and inference (testing). By default, Flux will automatically determine when a layer evaluation is part of training or inference. Still, depending on your use case, it may be helpful to manually specify when these layers should be treated as being trained or not. For this, Flux provides Flux.testmode!. When called on a model (e.g. a layer or chain of layers), this function will place the model into the mode specified.

Flux.testmode! — Function
testmode!(m, mode = true)

Set a layer or model's test mode (see below). Using :auto mode will treat any gradient computation as training.

Note: if you manually set a model into test mode, you need to manually place it back into train mode during training phase.

Possible values include:

  • false for training
  • true for testing
  • :auto or nothing for Flux to detect the mode automatically
source
Flux.trainmode! — Function
trainmode!(m, mode = true)

Set a layer of model's train mode (see below). Symmetric to testmode! (i.e. `trainmode!(m, mode) == testmode!(m, !mode)).

Note: if you manually set a model into train mode, you need to manually place it into test mode during testing phase.

Possible values include:

  • true for training
  • false for testing
  • :auto or nothing for Flux to detect the mode automatically
source
diff --git a/previews/PR1150/models/losses/index.html b/previews/PR1150/models/losses/index.html index 01739cb3..891bb97b 100644 --- a/previews/PR1150/models/losses/index.html +++ b/previews/PR1150/models/losses/index.html @@ -15,4 +15,4 @@ Huber loss = | | δ * (|ŷ - y| - 0.5 * δ), otherwisesource
Flux.crossentropy — Function
crossentropy(ŷ, y; weight=nothing, dims=1, ϵ=eps(eltype(ŷ)), 
                    logits=false, agg=mean)

Return the cross entropy between the given probability distributions; calculated as

agg(.-sum(weight .* y .* log.(ŷ .+ ϵ); dims=dims))agg=mean,

weight can be nothing, a number or an array. weight=nothing acts like weight=1 but is faster.

If logits=true, the input ̂y is first fed to a softmax layer.

See also: Flux.logitcrossentropy, Flux.binarycrossentropy, Flux.logitbinarycrossentropy

source
Flux.logitcrossentropy — Function
logitcrossentropy(ŷ, y; weight=nothing, agg=mean, dims=1)

Return the cross[1.0 0.5 0.3 2.4]entropy computed after a Flux.logsoftmax operation; calculated as

agg(.-sum(weight .* y .* logsoftmax(ŷ; dims=dims); dims=dims))

logitcrossentropy(ŷ, y) is mathematically equivalent to Flux.crossentropy(softmax(log.(ŷ)), y) but it is more numerically stable.

See also: Flux.crossentropy, Flux.binarycrossentropy, Flux.logitbinarycrossentropy

source
Flux.binarycrossentropy — Function
binarycrossentropy(ŷ, y; agg=mean, ϵ=epseltype(ŷ), logits=false)

Return $-y*\log(ŷ + ϵ) - (1-y)*\log(1-ŷ + ϵ)$. The ϵ term provides numerical stability.

Typically, the prediction ŷ is given by the output of a sigmoid activation. If logits=true, the input ̂y is first fed to a sigmoid activation. See also: Flux.crossentropy, Flux.logitcrossentropy, Flux.logitbinarycrossentropy

source
Flux.logitbinarycrossentropy — Function
logitbinarycrossentropy(ŷ, y; agg=mean)

logitbinarycrossentropy(ŷ, y) is mathematically equivalent to Flux.binarycrossentropy(σ(log(ŷ)), y) but it is more numerically stable.

See also: Flux.crossentropy, Flux.logitcrossentropy, Flux.binarycrossentropy

source
Flux.kldivergence — Function
kldivergence(ŷ, y; dims=1, agg=mean, ϵ=eps(eltype(ŷ)))

Return the Kullback-Leibler divergence between the given arrays interpreted as probability distributions.

KL divergence is a measure of how much one probability distribution is different from the other. It is always non-negative and zero only when both the distributions are equal everywhere.

source
Flux.poisson_loss — Function
poisson_loss(ŷ, y; agg=mean, ϵ=eps(eltype(ŷ))))

Loss function derived from likelihood for a Poisson random variable with mean ŷ to take value y. It is given by

agg(ŷ .- y .* log.(ŷ .+ ϵ))

More information..

source
Flux.hinge — Function
hinge(ŷ, y; agg=mean)

Return the hinge loss given the prediction ŷ and true labels y (containing 1 or -1); calculated as

agg(max.(0, 1 .- ŷ .* y))

See also: squared_hinge

source
Flux.squared_hinge — Function
squared_hinge(ŷ, y; agg=mean)

Return the squared hinge loss given the prediction ŷ and true labels y (containing 1 or -1); calculated as

agg(max.(0, 1 .- ŷ .* y).^2)

See also: hinge

source
Flux.dice_coeff_loss — Function
dice_coeff_loss(ŷ, y; smooth=1, dims=size(ŷ)[1:end-1], agg=mean)

Return a loss based on the Dice coefficient. Used in the V-Net architecture for image segmentation. Current implementation only works for the binary segmentation case.

The arrays ŷ and y contain the predicted and true probabilities respectively for the foreground to be present in a certain pixel. The loss is computed as

1 - (2*sum(ŷ .* y; dims) .+ smooth) ./ (sum(ŷ.^2 .+ y.^2; dims) .+ smooth)

and then aggregated with agg over the batch.

source
Flux.tversky_loss — Function
tversky_loss(ŷ, y; β=0.7, α=1-β, dims=size(ŷ)[1:end-1] agg=mean)

Return the Tversky loss for binary classification. The arrays ŷ and y contain the predicted and true probabilities respectively. Used with imbalanced data to give more weight to false negatives. Larger β weigh recall higher than precision (by placing more emphasis on false negatives) Calculated as:

num = sum(y .* ŷ, dims=dims)
 den = sum(@.(ŷ*y + α*ŷ*(1-y) + β*(1-ŷ)*y)), dims=dims)
-tversky_loss = 1 - num/den

and then aggregated with agg over the batch.

When α+β=1, it is equal to 1-F_β, where F_β is an F-score.

source
+tversky_loss = 1 - num/den

and then aggregated with agg over the batch.

When α+β=1, it is equal to 1-F_β, where F_β is an F-score.

source diff --git a/previews/PR1150/models/nnlib/index.html b/previews/PR1150/models/nnlib/index.html index b36504b5..f12204c3 100644 --- a/previews/PR1150/models/nnlib/index.html +++ b/previews/PR1150/models/nnlib/index.html @@ -28,4 +28,4 @@ a = randomly sampled from uniform distribution U(l, u)

Randomized batched_adjoint(A)

Equivalent to applying transpose or adjoint to each matrix A[:,:,k].

These exist to control how batched_mul behaves, as it operated on such matrix slices of an array with ndims(A)==3.

BatchedTranspose{T, N, S} <: AbstractBatchedMatrix{T, N}
 BatchedAdjoint{T, N, S}

Lazy wrappers analogous to Transpose and Adjoint, returned by batched_transpose

NNlib.batched_transpose — Function
batched_transpose(A::AbstractArray{T,3})
 batched_adjoint(A)

Equivalent to applying transpose or adjoint to each matrix A[:,:,k].

These exist to control how batched_mul behaves, as it operated on such matrix slices of an array with ndims(A)==3.

BatchedTranspose{T, N, S} <: AbstractBatchedMatrix{T, N}
-BatchedAdjoint{T, N, S}

Lazy wrappers analogous to Transpose and Adjoint, returned by batched_transpose

+BatchedAdjoint{T, N, S}

Lazy wrappers analogous to Transpose and Adjoint, returned by batched_transpose

diff --git a/previews/PR1150/models/recurrence/index.html b/previews/PR1150/models/recurrence/index.html index 13b51bb4..131d81e8 100644 --- a/previews/PR1150/models/recurrence/index.html +++ b/previews/PR1150/models/recurrence/index.html @@ -39,4 +39,4 @@ m = Flux.Recur(rnn, h) y = m(x)

The Recur wrapper stores the state between runs in the m.state field.

If you use the RNN(10, 5) constructor – as opposed to RNNCell – you'll see that it's simply a wrapped cell.

julia> RNN(10, 5)
 Recur(RNNCell(10, 5, tanh))

Sequences

Often we want to work with sequences of inputs, rather than individual xs.

seq = [rand(10) for i = 1:10]

With Recur, applying our model to each element of a sequence is trivial:

m.(seq) # returns a list of 5-element vectors

This works even when we've chain recurrent layers into a larger model.

m = Chain(LSTM(10, 15), Dense(15, 5))
-m.(seq)

Finally, we can reset the hidden state of the cell back to its initial value using reset!(m).

+m.(seq)

Finally, we can reset the hidden state of the cell back to its initial value using reset!(m).

diff --git a/previews/PR1150/models/regularisation/index.html b/previews/PR1150/models/regularisation/index.html index 406af633..0404b878 100644 --- a/previews/PR1150/models/regularisation/index.html +++ b/previews/PR1150/models/regularisation/index.html @@ -37,4 +37,4 @@ julia> activations(c, rand(10)) Float32[0.5192045, 0.48079553] julia> sum(norm, ans) -2.1166067f0
Flux.activations — Function
activations(c::Chain, input)

Calculate the forward results of each layers in Chain c with input as model input.

source
+2.1166067f0
Flux.activations — Function
activations(c::Chain, input)

Calculate the forward results of each layers in Chain c with input as model input.

source
diff --git a/previews/PR1150/performance/index.html b/previews/PR1150/performance/index.html index ec4cc5af..6a1255fd 100644 --- a/previews/PR1150/performance/index.html +++ b/previews/PR1150/performance/index.html @@ -17,4 +17,4 @@ y_batch = reduce(hcat, ys) function loss_total(x_batch::Matrix, y_batch::Matrix) y_preds = model(x_batch) sum(loss.(y_preds, y_batch)) -end

When doing this kind of concatenation use reduce(hcat, xs) rather than hcat(xs...). This will avoid the splatting penalty, and will hit the optimised reduce method.

+end

When doing this kind of concatenation use reduce(hcat, xs) rather than hcat(xs...). This will avoid the splatting penalty, and will hit the optimised reduce method.

diff --git a/previews/PR1150/saving/index.html b/previews/PR1150/saving/index.html index c116d388..3da2fd15 100644 --- a/previews/PR1150/saving/index.html +++ b/previews/PR1150/saving/index.html @@ -47,4 +47,4 @@ evalcb = throttle(30) do # Show loss @save "model-checkpoint.bson" model end

This will update the "model-checkpoint.bson" file every thirty seconds.

You can get more advanced by saving a series of models throughout training, for example

@save "model-$(now()).bson" model

will produce a series of models like "model-2018-03-06T02:57:10.41.bson". You could also store the current test set loss, so that it's easy to (for example) revert to an older copy of the model if it starts to overfit.

@save "model-$(now()).bson" model loss = testloss()

You can even store optimiser state alongside the model, to resume training exactly where you left off.

opt = ADAM()
-@save "model-$(now()).bson" model opt
+@save "model-$(now()).bson" model opt diff --git a/previews/PR1150/search/index.html b/previews/PR1150/search/index.html index ef51aaa4..fd37a202 100644 --- a/previews/PR1150/search/index.html +++ b/previews/PR1150/search/index.html @@ -6,4 +6,4 @@ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) ga('create', 'UA-36890222-9', 'auto'); ga('send', 'pageview', {'page': location.pathname + location.search + location.hash}); -

Loading search...

    +

    Loading search...

      diff --git a/previews/PR1150/training/optimisers/index.html b/previews/PR1150/training/optimisers/index.html index c2c51904..ad9a840c 100644 --- a/previews/PR1150/training/optimisers/index.html +++ b/previews/PR1150/training/optimisers/index.html @@ -88,4 +88,4 @@ end loss(rand(10)) # around 0.9

      In this manner it is possible to compose optimisers for some added flexibility.

      Decays

      Similar to optimisers, Flux also defines some simple decays that can be used in conjunction with other optimisers, or standalone.

      Flux.Optimise.ExpDecay — Type
      ExpDecay(η = 0.001, decay = 0.1, decay_step = 1000, clip = 1e-4)

      Discount the learning rate η by the factor decay every decay_step steps till a minimum of clip.

      Parameters

      • Learning rate (η): Amount by which gradients are discounted before updating the weights.
      • decay: Factor by which the learning rate is discounted.
      • decay_step: Schedule decay operations by setting the number of steps between two decay operations.
      • clip: Minimum value of learning rate.

      Examples

      To apply exponential decay to an optimiser:

      Optimiser(ExpDecay(..), Opt(..))
 
-opt = Optimiser(ExpDecay(), ADAM())
      source
      Flux.Optimise.InvDecay — Type
      InvDecay(γ = 0.001)

      Apply inverse time decay to an optimiser, so that the effective step size at iteration n is eta / (1 + γ * n) where eta is the initial step size. The wrapped optimiser's step size is not modified.

      Examples

      Optimiser(InvDecay(..), Opt(..))
      source
      Flux.Optimise.WeightDecay — Type
      WeightDecay(wd = 0)

      Decay weights by wd.

      Parameters

      • Weight decay (wd)
      source
      +opt = Optimiser(ExpDecay(), ADAM())source
      Flux.Optimise.InvDecay — Type
      InvDecay(γ = 0.001)

      Apply inverse time decay to an optimiser, so that the effective step size at iteration n is eta / (1 + γ * n) where eta is the initial step size. The wrapped optimiser's step size is not modified.

      Examples

      Optimiser(InvDecay(..), Opt(..))
      source
      Flux.Optimise.WeightDecay — Type
      WeightDecay(wd = 0)

      Decay weights by wd.

      Parameters

      • Weight decay (wd)
      source
      diff --git a/previews/PR1150/training/training/index.html b/previews/PR1150/training/training/index.html index ebd9c522..61bf7ee9 100644 --- a/previews/PR1150/training/training/index.html +++ b/previews/PR1150/training/training/index.html @@ -55,4 +55,4 @@ end

      You could simplify this further, for example by hard-coding in the loss function.

       +end

      You could simplify this further, for example by hard-coding in the loss function.

      diff --git a/previews/PR1150/utilities/index.html b/previews/PR1150/utilities/index.html index 52bcf3fe..44be8cc6 100644 --- a/previews/PR1150/utilities/index.html +++ b/previews/PR1150/utilities/index.html @@ -96,4 +96,4 @@ julia> θ ...

      The second return value re allows you to reconstruct the original network after making modifications to the weight vector (for example, with a hypernetwork).

      julia> re(θ .* 2)
 Chain(Dense(10, 5, σ), Dense(5, 2), softmax)
      source

      Callback Helpers

      Flux.throttle — Function
      throttle(f, timeout; leading=true, trailing=false)

      Return a function that when invoked, will only be triggered at most once during timeout seconds.

      Normally, the throttled function will run as much as it can, without ever going more than once per wait duration; but if you'd like to disable the execution on the leading edge, pass leading=false. To enable execution on the trailing edge, pass trailing=true.

      source
      Flux.Optimise.stop — Function
      stop()

      Call Flux.stop() in a callback to indicate when a callback condition is met. This will trigger the train loop to stop and exit.

      Examples

      cb = function ()
   accuracy() > 0.9 && Flux.stop()
-end
      source
      +endsource