From 4d3cd105ec39298470ee1bdb37433d61b08c949d Mon Sep 17 00:00:00 2001 From: autodocs Date: Tue, 21 Feb 2017 21:43:19 +0000 Subject: [PATCH] build based on c05ef62 --- latest/apis/backends.html | 2 +- latest/apis/batching.html | 57 +++++++++++++++++++++++++++++++----- latest/contributing.html | 2 +- latest/examples/logreg.html | 2 +- latest/index.html | 2 +- latest/internals.html | 2 +- latest/models/basics.html | 2 +- latest/models/debugging.html | 2 +- latest/models/recurrent.html | 2 +- latest/models/templates.html | 2 +- latest/search_index.js | 2 +- 11 files changed, 59 insertions(+), 18 deletions(-) diff --git a/latest/apis/backends.html b/latest/apis/backends.html index 3167c5d9..dece983e 100644 --- a/latest/apis/backends.html +++ b/latest/apis/backends.html @@ -140,7 +140,7 @@ Backends - + diff --git a/latest/apis/batching.html b/latest/apis/batching.html index 9d985349..711162b9 100644 --- a/latest/apis/batching.html +++ b/latest/apis/batching.html @@ -145,7 +145,7 @@ Batching - + @@ -298,17 +298,12 @@ Right now, the +

+Here's a more detailed illustration of how it might look for code to be "generic across batching". Take for example a weight matrix +W + times a vector +x +, as used in a logistic regression or a simple neural network: +

+
   W    *   x  =>   y
+(10×28) * (28) => (10)
+

+If we want to work with a batch of 50 +x +s, one option is to stack the data into a matrix of size +28 × 50 +. +

+
   W    *    x    =>    y
+(10×28) * (28×50) => (10×50)
+

+This works, but we may find that it's slow or doesn't fit well with the rest of the model, which batches on the first dimension. For that reason we may instead want to put the data in a +50 × 28 + matrix and alter the code as follows: +

+
   x    *    W'   =>    y
+(50×28) * (28×10) => (50×10)
+

+to make the shapes work out. This code change is not ideal; in more complex cases it can become fiddly and error-prone, and it means that the code is less reusable, tied to a particular implementation strategy. +

+

+There's an alternative. We keep the same code, but represent the batched +x +s as either a +Batch{Vector,1} + or a +Batch{Vector,2} +, depending on how the data is stacked. Then we can simply overload +* + as follows: +

+
*(W::Matrix, x::Batch{Vector,1}) = x * W'
+*(W::Matrix, x::Batch{Vector,2}) = W * x
+

+This means that we can always write +W*x +, and the code is reusable in a larger network regardless of the overall batching approach. Moreover, Julia's type system ensures there's no runtime cost to doing this, and we can compile the code appropriately for backends like TensorFlow as well. +