Fix sample weights generation for validation data

Multiple GPU works

.gitignore

@@ -7,4 +7,5 @@ build/*
 keras/datasets/data/*
 keras/datasets/temp/*
 docs/site/*
-docs/theme/*
+docs/theme/*
+tags

.travis.yml

@@ -1,3 +1,4 @@
+sudo: false
 language: python
 # Setup anaconda
 before_install:
@@ -6,15 +7,12 @@ before_install:
   - ./miniconda.sh -b
   - export PATH=/home/travis/miniconda/bin:$PATH
   - conda update --yes conda
-  # The next couple lines fix a crash with multiprocessing on Travis and are not specific to using Miniconda
-  - sudo rm -rf /dev/shm
-  - sudo ln -s /run/shm /dev/shm
 python:
   - "3.4"
 # command to install dependencies
 install:
-  - conda install --yes python=$TRAVIS_PYTHON_VERSION numpy scipy matplotlib pandas pytest
+  - conda install --yes python=$TRAVIS_PYTHON_VERSION numpy scipy matplotlib pandas pytest h5py
   # Coverage packages are on my binstar channel
   - python setup.py install
 # command to run tests
-script: py.test
+script: py.test tests/

README.md

@@ -6,8 +6,9 @@ Keras is a minimalist, highly modular neural network library in the spirit of To
 
 Use Keras if you need a deep learning library that:
 - allows for easy and fast prototyping (through total modularity, minimalism, and extensibility).
-- supports both convolutional networks (for vision) and recurrent networks (for sequence data). As well as combinations of the two. 
-- runs seamlessly on the CPU and the GPU.
+- supports both convolutional networks and recurrent networks, as well as combinations of the two.
+- supports arbitrary connectivity schemes (including multi-input and multi-output training).
+- runs seamlessly on CPU and GPU.
 
 Read the documentation at [Keras.io](http://keras.io).
 
@@ -15,13 +16,13 @@ Keras is compatible with __Python 2.7-3.4__.
 
 ## Guiding principles
 
-- __Modularity.__ A model is understood as a sequence of standalone, fully-configurable modules that can be plugged together with as little restrictions as possible. In particular, neural layers, cost functions, optimizers, initialization schemes, activation functions and dropout are all standalone modules that you can combine to create new models. 
+- __Modularity.__ A model is understood as a sequence or a graph of standalone, fully-configurable modules that can be plugged together with as little restrictions as possible. In particular, neural layers, cost functions, optimizers, initialization schemes, activation functions, regularization schemes are all standalone modules that you can combine to create new models.
 
-- __Minimalism.__ Each module should be kept short and simple (<100 lines of code). Every piece of code should be transparent upon first reading. No black magic: it hurts iteration speed and ability to innovate. 
+- __Minimalism.__ Each module should be kept short and simple (<100 lines of code). Every piece of code should be transparent upon first reading. No black magic: it hurts iteration speed and ability to innovate.
 
-- __Easy extensibility.__ New features (a new module, per the above definition, or a new way to combine modules together) are dead simple to add (as new classes/functions), and existing modules provide ample examples.
+- __Easy extensibility.__ New modules are dead simple to add (as new classes/functions), and existing modules provide ample examples. To be able to easily create new modules allows for total expressiveness, making Keras suitable for advanced research.
 
-- __Work with Python__. No separate models configuration files in a declarative format (like in Caffe or PyLearn2). Models are described in Python code, which is compact, easier to debug, benefits from syntax highlighting, and most of all, allows for ease of extensibility. See for yourself with the examples below.
+- __Work with Python__. No separate models configuration files in a declarative format (like in Caffe or PyLearn2). Models are described in Python code, which is compact, easier to debug, and allows for ease of extensibility.
 
 ## Examples
 
@@ -155,7 +156,7 @@ model.add(Dense(128*4*4, 256))
 model.add(Activation('relu'))
 model.add(Dropout(0.5))
 
-model.add(Repeat(max_caption_len)) 
+model.add(RepeatVector(max_caption_len)) 
 # the GRU below returns sequences of max_caption_len vectors of size 256 (our word embedding size)
 model.add(GRU(256, 256, return_sequences=True))
 
@@ -171,8 +172,11 @@ model.fit(images, captions, batch_size=16, nb_epoch=100)
 In the examples folder, you will find example models for real datasets:
 - CIFAR10 small images classification: Convnet with realtime data augmentation
 - IMDB movie review sentiment classification: LSTM over sequences of words
-- Reuters newswires topic classification: Multilayer Perceptron
-- MNIST handwritten digits classification: Multilayer Perceptron
+- Reuters newswires topic classification: Multilayer Perceptron (MLP)
+- MNIST handwritten digits classification: MLP & CNN
+- Character-level text generation with LSTM
+
+...and more.
 
 
 ## Current capabilities
@@ -186,12 +190,10 @@ A few highlights: convnets, LSTM, GRU, word2vec-style embeddings, PReLU, batch n
 Keras uses the following dependencies:
 
 - numpy, scipy
-
+- pyyaml
 - Theano
     - See installation instructions: http://deeplearning.net/software/theano/install.html#install
-
 - HDF5 and h5py (optional, required if you use model saving/loading functions)
-
 - Optional but recommended if you use CNNs: cuDNN.
 
 Once you have the dependencies installed, cd to the Keras folder and run the install command:
@@ -199,11 +201,16 @@ Once you have the dependencies installed, cd to the Keras folder and run the ins
 sudo python setup.py install
 ```
 
+You can also install Keras from PyPI:
+```
+sudo pip install keras
+```
+
 ## Why this name, Keras?
 
 Keras (κέρας) means _horn_ in Greek. It is a reference to a literary image from ancient Greek and Latin literature, first found in the _Odyssey_, where dream spirits (_Oneiroi_, singular _Oneiros_) are divided between those who deceive men with false visions, who arrive to Earth through a gate of ivory, and those who announce a future that will come to pass, who arrive through a gate of horn. It's a play on the words κέρας (horn) / κραίνω (fulfill), and ἐλέφας (ivory) / ἐλεφαίρομαι (deceive).
 
 Keras was developed as part of the research effort of project ONEIROS (Open-ended Neuro-Electronic Intelligent Robot Operating System).
 
-_"Oneiroi are beyond our unravelling --who can be sure what tale they tell? Not all that men look for comes to pass. Two gates there are that give passage to fleeting Oneiroi; one is made of horn, one of ivory. The Oneiroi that pass through sawn ivory are deceitful, bearing a message that will not be fulfilled; those that come out through polished horn have truth behind them, to be accomplished for men who see them."_ Homer, Odyssey 19. 562 ff (Shewring translation).
+>_"Oneiroi are beyond our unravelling --who can be sure what tale they tell? Not all that men look for comes to pass. Two gates there are that give passage to fleeting Oneiroi; one is made of horn, one of ivory. The Oneiroi that pass through sawn ivory are deceitful, bearing a message that will not be fulfilled; those that come out through polished horn have truth behind them, to be accomplished for men who see them."_ Homer, Odyssey 19. 562 ff (Shewring translation).
 

docs/mkdocs.yml

@@ -1,9 +1,9 @@
 site_name: Keras Documentation
-#theme: readthedocs
+theme: readthedocs
 docs_dir: sources
 repo_url: http://github.com/fchollet/keras
 site_url: http://keras.io/
-theme_dir: theme
+#theme_dir: theme
 site_description: Documentation for fast and lightweight Keras Deep Learning library.
 
 dev_addr: '0.0.0.0:8000'
@@ -36,6 +36,4 @@ pages:
   - Sequence Preprocessing: preprocessing/sequence.md
   - Text Preprocessing: preprocessing/text.md
   - Image Preprocessing: preprocessing/image.md
-- Utils:
-  - Visualization Utilities: utils/visualization.md
 

docs/sources/callbacks.md

@@ -52,7 +52,7 @@ You can create a custom callback by extending the base class `keras.callbacks.Ca
 Here's a simple example saving a list of losses over each batch during training:
 ```python
 class LossHistory(keras.callbacks.Callback):
-    def on_train_begin(self):
+    def on_train_begin(self, logs={}):
         self.losses = []
 
     def on_batch_end(self, batch, logs={}):

docs/sources/documentation.md

@@ -36,8 +36,3 @@
 - [Sequence](preprocessing/sequence.md)
 - [Text](preprocessing/text.md)
 - [Image](preprocessing/image.md)
-
----
-
-## Utils
-- [Visualization](utils/visualization.md)

docs/sources/examples.md

@@ -141,7 +141,7 @@ model.add(Dense(128*4*4, 256))
 model.add(Activation('relu'))
 model.add(Dropout(0.5))
 
-model.add(Repeat(max_caption_len)) 
+model.add(RepeatVector(max_caption_len)) 
 # the GRU below returns sequences of max_caption_len vectors of size 256 (our word embedding size)
 model.add(GRU(256, 256, return_sequences=True))
 

docs/sources/index.md

@@ -7,8 +7,9 @@ Keras is a minimalist, highly modular neural network library in the spirit of To
 Use Keras if you need a deep learning library that:
 
 - allows for easy and fast prototyping (through total modularity, minimalism, and extensibility).
-- supports both __convolutional networks__ and __recurrent networks__ (LSTM, GRU, etc). As well as combinations of the two. 
-- runs seamlessly on the CPU and the GPU.
+- supports both convolutional networks and recurrent networks, as well as combinations of the two.
+- supports arbitrary connectivity schemes (including multi-input and multi-output training).
+- runs seamlessly on CPU and GPU.
 
 ## Guiding principles
 
@@ -16,7 +17,7 @@ Use Keras if you need a deep learning library that:
 
 - __Minimalism.__ Each module should be kept short and simple (<100 lines of code). Every piece of code should be transparent upon first reading. No black magic: it hurts iteration speed and ability to innovate.
 
-- __Easy extensibility.__ New modules are dead simple to add (as new classes/functions), and existing modules provide ample examples. To be able to easily create new modules allows for total expressiveness, making Keras suitable for adavanced research.
+- __Easy extensibility.__ New modules are dead simple to add (as new classes/functions), and existing modules provide ample examples. To be able to easily create new modules allows for total expressiveness, making Keras suitable for advanced research.
 
 - __Work with Python__. No separate models configuration files in a declarative format (like in Caffe or PyLearn2). Models are described in Python code, which is compact, easier to debug, and allows for ease of extensibility.
 
@@ -92,6 +93,7 @@ Have a look at the [examples](examples.md).
 Keras uses the following dependencies:
 
 - __numpy__, __scipy__
+- __pyyaml__
 - __Theano__
     - See [installation instructions](http://deeplearning.net/software/theano/install.html#install).
 - __HDF5__ and __h5py__ (optional, required if you use model saving/loading functions)
@@ -106,6 +108,10 @@ Go to the Keras folder and run the install command:
 cd keras
 sudo python setup.py install
 ```
+You can also install Keras from PyPI:
+```
+sudo pip install keras
+```
 
 ## Support
 
@@ -119,6 +125,7 @@ Keras welcomes all contributions from the community.
 - New features should be documented. Make sure you update the documentation along with your Pull Request.
 - The documentation for every new feature should include a usage example in the form of a code snippet. 
 - All changes should be tested. Make sure any new feature you add has a corresponding unit test.
+- Please no Pull Requests about coding style.
 - Even if you don't contribute to the Keras source code, if you have an application of Keras that is concise and powerful, please consider adding it to our collection of [examples](https://github.com/fchollet/keras/tree/master/examples).
 
 

docs/sources/layers/advanced_activations.md

@@ -32,4 +32,60 @@ Parametrized linear unit. Similar to a LeakyReLU, where each input unit has its
     - __input_shape__: tuple.
 
 - __References__:
-    - [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](http://arxiv.org/pdf/1502.01852v1.pdf)
+    - [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](http://arxiv.org/pdf/1502.01852v1.pdf)
+
+---
+
+## ParametricSoftplus
+
+```python
+keras.layers.advanced_activations.ParametricSoftplus(input_shape)
+```
+
+Parametric Softplus of the form: (`f(x) = alpha * (1 + exp(beta * x))`). This is essentially a smooth version of ReLU where the parameters control the sharpness of the rectification. The parameters are initialized to more closely approximate a ReLU than the standard `softplus`: `alpha` initialized to `0.2` and `beta`  initialized to `5.0`. The parameters are fit separately for each hidden unit.
+
+- __Input shape__: Same as `input_shape`. This layer cannot be used as first layer in a model.
+
+- __Output shape__: Same as input.
+
+- __Arguments__:
+    - __input_shape__: tuple.
+
+- __References__:
+    - [Inferring Nonlinear Neuronal Computation Based on Physiologically Plausible Inputs](http://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1003143)
+
+## Thresholded Linear
+
+```python
+keras.layers.advanced_activations.ThresholdedLinear(theta)
+```
+
+Parametrized linear unit. provides a threshold near zero where values are zeroed.
+
+- __Input shape__: Same as `input_shape`. This layer cannot be used as first layer in a model.
+
+- __Output shape__: Same as input.
+
+- __Arguments__:
+    - __theta__: float >= 0. Threshold location of activation
+
+- __References__:
+    - [Zero-Bias Autoencoders and the Benefits of Co-Adapting Features](http://arxiv.org/pdf/1402.3337.pdf)
+
+## Thresholded ReLu
+
+```python
+keras.layers.advanced_activations.ThresholdedReLu(theta)
+```
+
+Parametrized rectified linear unit. provides a threshold near zero where values are zeroed.
+
+- __Input shape__: Same as `input_shape`. This layer cannot be used as first layer in a model.
+
+- __Output shape__: Same as input.
+
+- __Arguments__:
+    - __theta__: float >= 0. Threshold location of activation
+
+- __References__:
+    - [Zero-Bias Autoencoders and the Benefits of Co-Adapting Features](http://arxiv.org/pdf/1402.3337.pdf)

docs/sources/layers/convolutional.md

@@ -58,7 +58,7 @@ Convolution operator for filtering windows of two-dimensional inputs.
     - __init__: name of initialization function for the weights of the layer (see: [initializations](../initializations.md)), or alternatively, Theano function to use for weights initialization. This parameter is only relevant if you don't pass a `weights` argument.
     - __activation__: name of activation function to use (see: [activations](../activations.md)), or alternatively, elementwise Theano function. If you don't specify anything, no activation is applied (ie. "linear" activation: a(x) = x).
     - __weights__: list of numpy arrays to set as initial weights.
-    - __border_mode__: 'valid', 'full', or 'same'. See scipy.signal.convolve2d.
+    - __border_mode__: 'valid', 'full', or 'same'. [See scipy.signal.convolve2d](http://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.convolve2d.html).
     - __subsample__: tuple of length 2. Factor by which to subsample output. Also called strides elsewhere.
     - __W_regularizer__: instance of [WeightRegularizer](../regularizers.md) (eg. L1 or L2 regularization), applied to the main weights matrix.
     - __b_regularizer__: instance of [WeightRegularizer](../regularizers.md), applied to the bias.
@@ -77,7 +77,7 @@ keras.layers.convolutional.MaxPooling1D(pool_length=2, stride=None, ignore_borde
 
 - __Input shape__: 3D tensor with shape: `(nb_samples, steps, dim)`.
 
-- __Output shape__: 3D tensor with shape: `(nb_samples, steps, new_dim)`.
+- __Output shape__: 3D tensor with shape: `(nb_samples, downsampled_steps, dim)`.
 
 - __Arguments__:
 

docs/sources/layers/core.md

@@ -7,7 +7,7 @@ keras.layers.core.Layer()
 __Methods__:
 
 ```python
-connect(previous_layer)
+set_previous(previous_layer)
 ```
 
 Connect the input of the current layer to the output of the argument layer.
@@ -20,7 +20,7 @@ Connect the input of the current layer to the output of the argument layer.
 
 
 ```python
-output(train)
+get_output(train)
 ```
 
 Get the output of the layer.
@@ -65,6 +65,12 @@ Set the weights of the parameters of the layer.
     - __weights__: List of numpy arrays (one per layer parameter). Should be in the same order as what `get_weights(self)` returns.
 
 
+```python
+get_config()
+```
+
+- __Return__: Configuration dictionary describing the layer.
+
 
 ---
 
@@ -251,7 +257,27 @@ Note that the output is still a single tensor; `RepeatVector` does not split the
 - __Arguments__:
     - __n__: int. 
 
+---
+
+## Permute
+```python
+keras.layers.core.Permute(dims)
+```
+Permute the dimensions of the input data according to the given tuple. Sometimes useful for connecting RNNs and convnets together.
+
+- __Input shape: This layer does not assume a specific input shape.
+
+- __Output shape: Same as the input shape, but with the dimensions re-ordered according to the ordering specified by the tuple.
+
+- __Argument: tuple specifying the permutation scheme (e.g. `(2, 1)` permutes the first and second dimension of the input).
+
 - __Example__:
+```python
+# input shape: (nb_samples, 10)
+model.add(Dense(10, 50)) # output shape: (nb_samples, 50)
+model.add(Reshape(10, 5)) # output shape: (nb_samples, 10, 5)
+model.add(Permute((2, 1))) #output shape: (nb_samples, 5, 10)
+```
 
 ---
 
@@ -331,3 +357,14 @@ model.compile(loss='categorical_crossentropy', optimizer='rmsprop')
 model.fit([X_train, X_train], Y_train, batch_size=128, nb_epoch=20, validation_data=([X_test, X_test], Y_test))
 ```
 
+## Masking
+```python
+keras.layers.core.Masking(mask_value=0.)
+```
+
+Create a mask for the input data by using `mask_value` as the sentinel value which should be masked out.
+Given an input of dimensions `(nb_samples, timesteps, input_dim)`, return the input untouched as output, and supply a mask of shape `(nb_samples, timesteps)` where all timesteps which had *all* their values equal to `mask_value` are masked out.
+
+- __Input shape__: 3D tensor with shape: `(nb_samples, timesteps, features)`.
+
+- __Output shape__: 3D tensor with shape: `(nb_samples, timesteps, features)`.

docs/sources/layers/recurrent.md

@@ -11,7 +11,7 @@ Fully connected RNN where output is to fed back to input.
 - __Input shape__: 3D tensor with shape: `(nb_samples, timesteps, input_dim)`.
 
 - __Output shape__: 
-    - if `return_sequences`: 3D tensor with shape: `(nb_samples, timesteps, ouput_dim)`.
+    - if `return_sequences`: 3D tensor with shape: `(nb_samples, timesteps, output_dim)`.
     - else: 2D tensor with shape: `(nb_samples, output_dim)`.
 
 - __Masking__: This layer supports masking for input data with a variable number of timesteps To introduce masks to your data, use an [Embedding](embeddings.md) layer with the `mask_zero` parameter set to `True`.
@@ -47,7 +47,7 @@ Not a particularly useful model, included for demonstration purposes.
 - __Input shape__: 3D tensor with shape: `(nb_samples, timesteps, input_dim)`.
 
 - __Output shape__:
-    - if `return_sequences`: 3D tensor with shape: `(nb_samples, timesteps, ouput_dim)`.
+    - if `return_sequences`: 3D tensor with shape: `(nb_samples, timesteps, output_dim)`.
     - else: 2D tensor with shape: `(nb_samples, output_dim)`.
 
 - __Masking__: This layer supports masking for input data with a variable number of timesteps To introduce masks to your data, use an [Embedding](embeddings.md) layer with the `mask_zero` parameter set to `True`.
@@ -82,7 +82,7 @@ Gated Recurrent Unit - Cho et al. 2014.
 - __Input shape__: 3D tensor with shape: `(nb_samples, timesteps, input_dim)`.
 
 - __Output shape__:
-    - if `return_sequences`: 3D tensor with shape: `(nb_samples, timesteps, ouput_dim)`.
+    - if `return_sequences`: 3D tensor with shape: `(nb_samples, timesteps, output_dim)`.
     - else: 2D tensor with shape: `(nb_samples, output_dim)`.
 
 - __Masking__: This layer supports masking for input data with a variable number of timesteps To introduce masks to your data, use an [Embedding](embeddings.md) layer with the `mask_zero` parameter set to true.
@@ -118,7 +118,7 @@ Long-Short Term Memory unit - Hochreiter 1997.
 - __Input shape__: 3D tensor with shape: `(nb_samples, timesteps, input_dim)`.
 
 - __Output shape__:
-    - if `return_sequences`: 3D tensor with shape: `(nb_samples, timesteps, ouput_dim)`.
+    - if `return_sequences`: 3D tensor with shape: `(nb_samples, timesteps, output_dim)`.
     - else: 2D tensor with shape: `(nb_samples, output_dim)`.
 
 - __Masking__: This layer supports masking for input data with a variable number of timesteps To introduce masks to your data, use an [Embedding](embeddings.md) layer with the `mask_zero` parameter set to true.
@@ -156,7 +156,7 @@ Top 3 RNN architectures evolved from the evaluation of thousands of models. Serv
 - __Input shape__: 3D tensor with shape: `(nb_samples, timesteps, input_dim)`.
 
 - __Output shape__:
-    - if `return_sequences`: 3D tensor with shape: `(nb_samples, timesteps, ouput_dim)`.
+    - if `return_sequences`: 3D tensor with shape: `(nb_samples, timesteps, output_dim)`.
     - else: 2D tensor with shape: `(nb_samples, output_dim)`.
 
 - __Masking__: This layer supports masking for input data with a variable number of timesteps To introduce masks to your data, use an [Embedding](embeddings.md) layer with the `mask_zero` parameter set to true.

docs/sources/models.md

@@ -24,10 +24,10 @@ model = keras.models.Sequential()
             - __callbacks__: `keras.callbacks.Callback` list. List of callbacks to apply during training. See [callbacks](callbacks.md).
             - __validation_split__: float (0. < x < 1). Fraction of the data to use as held-out validation data.
             - __validation_data__: tuple (X, y) to be used as held-out validation data. Will override validation_split.
-            - __shuffle__: boolean. Whether to shuffle the samples at each epoch.
+            - __shuffle__: boolean or str (for 'batch'). Whether to shuffle the samples at each epoch. 'batch' is a special option for dealing with the limitations of HDF5 data; it shuffles in batch-sized chunks.
             - __show_accuracy__: boolean. Whether to display class accuracy in the logs to stdout at each epoch.
             - __class_weight__: dictionary mapping classes to a weight value, used for scaling the loss function (during training only).
-            - __sample_weight__: list or numpy array with 1:1 mapping to the training samples, used for scaling the loss function (during training only).
+            - __sample_weight__: list or numpy array with 1:1 mapping to the training samples, used for scaling the loss function (during training only). For time-distributed data, there is one weight per sample *per timestep*, i.e. if your output data is shaped `(nb_samples, timesteps, output_dim)`, your mask should be of shape `(nb_samples, timesteps)`. This allows you to mask out or reweight individual output timesteps, which is useful in sequence to sequence learning.
     - __evaluate__(X, y, batch_size=128, show_accuracy=False, verbose=1): Show performance of the model over some validation data.
         - __Return__: The loss score over the data, or a `(loss, accuracy)` tuple if `show_accuracy=True`.
         - __Arguments__: Same meaning as fit method above. verbose is used as a binary flag (progress bar or nothing).
@@ -202,4 +202,4 @@ graph.compile('rmsprop', {'output':'mse'})
 history = graph.fit({'input1':X_train, 'input2':X2_train, 'output':y_train}, nb_epoch=10)
 predictions = graph.predict({'input1':X_test, 'input2':X2_test}) # {'output':...}
 
-```
+```

docs/sources/optimizers.md

@@ -103,16 +103,15 @@ __Arguments__:
 ## Adam
 
 ```python
-keras.optimizers.Adam(lr=0.001, beta_1=0.9, beta_2=0.999, epsilon=1e-8, kappa=1-1e-8)
+keras.optimizers.Adam(lr=0.001, beta_1=0.9, beta_2=0.999, epsilon=1e-8)
 ```
 
-Adam optimizer, proposed by Kingma and Lei Ba in [Adam: A Method For Stochastic Optimization](http://arxiv.org/pdf/1412.6980v4.pdf). Default parameters are those suggested in the paper. The parameter "lambda" from the paper has been renamed kappa, for syntactic reasons.
+Adam optimizer, proposed by Kingma and Lei Ba in [Adam: A Method For Stochastic Optimization](http://arxiv.org/pdf/1412.6980v8.pdf). Default parameters are those suggested in the paper.
 
 __Arguments__:
 
-- __lr__: float >= 0. Learning rate. 
+- __lr__: float >= 0. Learning rate.
 - __beta_1__, __beta_2__: floats, 0 < beta < 1. Generally close to 1.
 - __epsilon__: float >= 0. Fuzz factor.
-- __kappa__: float 0 < kappa < 1. Lambda parameter in the original paper.
 
----
+---

docs/sources/utils/visualization.md

@@ -1,28 +0,0 @@
-## Grapher
-
-Creates a visualization of the model structure using `pydot`.
-
-```python
-grapher = keras.utils.dot_utils.Grapher()
-```
-- __Methods__:
-    - __plot__(model, to_file): creates a graph visualizing the structure of `model` and writes it to `to_file`.
-      - __Arguments__:
-        - __model__: an instance of a Keras model (e.g. `Sequential`)
-        - __to_file__: the filename to save the visualization png to.
-
-__Examples__:
-
-```python
-from keras.models import Sequential
-from keras.layers.core import Dense, Activation
-from keras.utils.dot_utils import Grapher
-
-grapher = Grapher()
-
-model = Sequential()
-model.add(Dense(64, 2, init='uniform'))
-model.add(Activation('softmax'))
-grapher.plot(model, 'model.png')
-```
-

examples/addition_rnn.py

@@ -0,0 +1,164 @@
+# -*- coding: utf-8 -*-
+from __future__ import print_function
+from keras.models import Sequential, slice_X
+from keras.layers.core import Activation, Dense, RepeatVector
+from keras.layers import recurrent
+from sklearn.utils import shuffle
+import numpy as np
+
+"""
+An implementation of sequence to sequence learning for performing addition
+Input: "535+61"
+Output: "596"
+Padding is handled by using a repeated sentinel character (space)
+
+By default, the JZS1 recurrent neural network is used
+JZS1 was an "evolved" recurrent neural network performing well on arithmetic benchmark in:
+"An Empirical Exploration of Recurrent Network Architectures"
+http://jmlr.org/proceedings/papers/v37/jozefowicz15.pdf
+
+Input may optionally be inverted, shown to increase performance in many tasks in:
+"Learning to Execute"
+http://arxiv.org/abs/1410.4615
+and
+"Sequence to Sequence Learning with Neural Networks"
+http://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks.pdf
+Theoretically it introduces shorter term dependencies between source and target.
+
+
+Two digits inverted:
++ One layer JZS1 (128 HN), 5k training examples = 99% train/test accuracy in 55 epochs
+
+Three digits inverted:
++ One layer JZS1 (128 HN), 50k training examples = 99% train/test accuracy in 100 epochs
+
+
+Four digits inverted:
++ One layer JZS1 (128 HN), 400k training examples = 99% train/test accuracy in 20 epochs
+
+
+Five digits inverted:
++ One layer JZS1 (128 HN), 550k training examples = 99% train/test accuracy in 30 epochs
+
+"""
+
+
+class CharacterTable(object):
+    """
+    Given a set of characters:
+    + Encode them to a one hot integer representation
+    + Decode the one hot integer representation to their character output
+    + Decode a vector of probabilties to their character output
+    """
+    def __init__(self, chars, maxlen):
+        self.chars = sorted(set(chars))
+        self.char_indices = dict((c, i) for i, c in enumerate(self.chars))
+        self.indices_char = dict((i, c) for i, c in enumerate(self.chars))
+        self.maxlen = maxlen
+
+    def encode(self, C, maxlen=None):
+        maxlen = maxlen if maxlen else self.maxlen
+        X = np.zeros((maxlen, len(self.chars)))
+        for i, c in enumerate(C):
+            X[i, self.char_indices[c]] = 1
+        return X
+
+    def decode(self, X, calc_argmax=True):
+        if calc_argmax:
+            X = X.argmax(axis=-1)
+        return ''.join(self.indices_char[x] for x in X)
+
+
+class colors:
+    ok = '\033[92m'
+    fail = '\033[91m'
+    close = '\033[0m'
+
+# Parameters for the model and dataset
+TRAINING_SIZE = 50000
+DIGITS = 3
+INVERT = True
+# Try replacing JZS1 with LSTM, GRU, or SimpleRNN
+RNN = recurrent.JZS1
+HIDDEN_SIZE = 128
+BATCH_SIZE = 128
+LAYERS = 1
+MAXLEN = DIGITS + 1 + DIGITS
+
+chars = '0123456789+ '
+ctable = CharacterTable(chars, MAXLEN)
+
+questions = []
+expected = []
+seen = set()
+print('Generating data...')
+while len(questions) < TRAINING_SIZE:
+    f = lambda: int(''.join(np.random.choice(list('0123456789')) for i in xrange(np.random.randint(1, DIGITS + 1))))
+    a, b = f(), f()
+    # Skip any addition questions we've already seen
+    # Also skip any such that X+Y == Y+X (hence the sorting)
+    key = tuple(sorted((a, b)))
+    if key in seen:
+        continue
+    seen.add(key)
+    # Pad the data with spaces such that it is always MAXLEN
+    q = '{}+{}'.format(a, b)
+    query = q + ' ' * (MAXLEN - len(q))
+    ans = str(a + b)
+    # Answers can be of maximum size DIGITS + 1
+    ans += ' ' * (DIGITS + 1 - len(ans))
+    if INVERT:
+        query = query[::-1]
+    questions.append(query)
+    expected.append(ans)
+print('Total addition questions:', len(questions))
+
+print('Vectorization...')
+X = np.zeros((len(questions), MAXLEN, len(chars)), dtype=np.bool)
+y = np.zeros((len(questions), DIGITS + 1, len(chars)), dtype=np.bool)
+for i, sentence in enumerate(questions):
+    X[i] = ctable.encode(sentence, maxlen=MAXLEN)
+for i, sentence in enumerate(expected):
+    y[i] = ctable.encode(sentence, maxlen=DIGITS + 1)
+
+# Shuffle (X, y) in unison as the later parts of X will almost all be larger digits
+X, y = shuffle(X, y)
+# Explicitly set apart 10% for validation data that we never train over
+split_at = len(X) - len(X) / 10
+(X_train, X_val) = (slice_X(X, 0, split_at), slice_X(X, split_at))
+(y_train, y_val) = (y[:split_at], y[split_at:])
+
+print('Build model...')
+model = Sequential()
+# "Encode" the input sequence using an RNN, producing an output of HIDDEN_SIZE
+model.add(RNN(len(chars), HIDDEN_SIZE))
+# For the decoder's input, we repeat the encoded input for each time step
+model.add(RepeatVector(DIGITS + 1))
+# The decoder RNN could be multiple layers stacked or a single layer
+for _ in xrange(LAYERS):
+    model.add(RNN(HIDDEN_SIZE, HIDDEN_SIZE, return_sequences=True))
+# For each of step of the output sequence, decide which character should be chosen
+model.add(Dense(HIDDEN_SIZE, len(chars)))
+model.add(Activation('softmax'))
+
+model.compile(loss='categorical_crossentropy', optimizer='adam')
+
+# Train the model each generation and show predictions against the validation dataset
+for iteration in range(1, 200):
+    print()
+    print('-' * 50)
+    print('Iteration', iteration)
+    model.fit(X, y, batch_size=BATCH_SIZE, nb_epoch=1, validation_data=(X_val, y_val), show_accuracy=True)
+    ###
+    # Select 10 samples from the validation set at random so we can visualize errors
+    for i in xrange(10):
+        ind = np.random.randint(0, len(X_val))
+        rowX, rowy = X_val[np.array([ind])], y_val[np.array([ind])]
+        preds = model.predict_classes(rowX, verbose=0)
+        q = ctable.decode(rowX[0])
+        correct = ctable.decode(rowy[0])
+        guess = ctable.decode(preds[0], calc_argmax=False)
+        print('Q', q[::-1] if INVERT else q)
+        print('T', correct)
+        print(colors.ok + '☑' + colors.close if correct == guess else colors.fail + '☒' + colors.close, guess)
+        print('---')

examples/babi_rnn.py

@@ -0,0 +1,199 @@
+from __future__ import absolute_import
+from __future__ import print_function
+from functools import reduce
+import re
+import tarfile
+
+import numpy as np
+np.random.seed(1337)  # for reproducibility
+
+from keras.datasets.data_utils import get_file
+from keras.layers.embeddings import Embedding
+from keras.layers.core import Dense, Merge
+from keras.layers import recurrent
+from keras.models import Sequential
+from keras.preprocessing.sequence import pad_sequences
+
+'''
+Trains two recurrent neural networks based upon a story and a question.
+The resulting merged vector is then queried to answer a range of bAbI tasks.
+
+The results are comparable to those for an LSTM model provided in Weston et al.:
+"Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks"
+http://arxiv.org/abs/1502.05698
+
+Task Number                  | FB LSTM Baseline | Keras QA
+---                          | ---              | ---
+QA1 - Single Supporting Fact | 50               | 52.1
+QA2 - Two Supporting Facts   | 20               | 37.0
+QA3 - Three Supporting Facts | 20               | 20.5
+QA4 - Two Arg. Relations     | 61               | 62.9
+QA5 - Three Arg. Relations   | 70               | 61.9
+QA6 - Yes/No Questions       | 48               | 50.7
+QA7 - Counting               | 49               | 78.9
+QA8 - Lists/Sets             | 45               | 77.2
+QA9 - Simple Negation        | 64               | 64.0
+QA10 - Indefinite Knowledge  | 44               | 47.7
+QA11 - Basic Coreference     | 72               | 74.9
+QA12 - Conjunction           | 74               | 76.4
+QA13 - Compound Coreference  | 94               | 94.4
+QA14 - Time Reasoning        | 27               | 34.8
+QA15 - Basic Deduction       | 21               | 32.4
+QA16 - Basic Induction       | 23               | 50.6
+QA17 - Positional Reasoning  | 51               | 49.1
+QA18 - Size Reasoning        | 52               | 90.8
+QA19 - Path Finding          | 8                | 9.0
+QA20 - Agent's Motivations   | 91               | 90.7
+
+For the resources related to the bAbI project, refer to:
+https://research.facebook.com/researchers/1543934539189348
+
+Notes:
+
+- With default word, sentence, and query vector sizes, the GRU model achieves:
+  - 52.1% test accuracy on QA1 in 20 epochs (2 seconds per epoch on CPU)
+  - 37.0% test accuracy on QA2 in 20 epochs (16 seconds per epoch on CPU)
+In comparison, the Facebook paper achieves 50% and 20% for the LSTM baseline.
+
+- The task does not traditionally parse the question separately. This likely
+improves accuracy and is a good example of merging two RNNs.
+
+- The word vector embeddings are not shared between the story and question RNNs.
+
+- See how the accuracy changes given 10,000 training samples (en-10k) instead
+of only 1000. 1000 was used in order to be comparable to the original paper.
+
+- Experiment with GRU, LSTM, and JZS1-3 as they give subtly different results.
+
+- The length and noise (i.e. 'useless' story components) impact the ability for
+LSTMs / GRUs to provide the correct answer. Given only the supporting facts,
+these RNNs can achieve 100% accuracy on many tasks. Memory networks and neural
+networks that use attentional processes can efficiently search through this
+noise to find the relevant statements, improving performance substantially.
+This becomes especially obvious on QA2 and QA3, both far longer than QA1.
+'''
+
+
+def tokenize(sent):
+    '''Return the tokens of a sentence including punctuation.
+
+    >>> tokenize('Bob dropped the apple. Where is the apple?')
+    ['Bob', 'dropped', 'the', 'apple', '.', 'Where', 'is', 'the', 'apple', '?']
+    '''
+    return [x.strip() for x in re.split('(\W+)?', sent) if x.strip()]
+
+
+def parse_stories(lines, only_supporting=False):
+    '''Parse stories provided in the bAbi tasks format
+
+    If only_supporting is true, only the sentences that support the answer are kept.
+    '''
+    data = []
+    story = []
+    for line in lines:
+        line = line.decode('utf-8').strip()
+        nid, line = line.split(' ', 1)
+        nid = int(nid)
+        if nid == 1:
+            story = []
+        if '\t' in line:
+            q, a, supporting = line.split('\t')
+            q = tokenize(q)
+            substory = None
+            if only_supporting:
+                # Only select the related substory
+                supporting = map(int, supporting.split())
+                substory = [story[i - 1] for i in supporting]
+            else:
+                # Provide all the substories
+                substory = [x for x in story if x]
+            data.append((substory, q, a))
+            story.append('')
+        else:
+            sent = tokenize(line)
+            story.append(sent)
+    return data
+
+
+def get_stories(f, only_supporting=False, max_length=None):
+    '''Given a file name, read the file, retrieve the stories, and then convert the sentences into a single story.
+
+    If max_length is supplied, any stories longer than max_length tokens will be discarded.
+    '''
+    data = parse_stories(f.readlines(), only_supporting=only_supporting)
+    flatten = lambda data: reduce(lambda x, y: x + y, data)
+    data = [(flatten(story), q, answer) for story, q, answer in data if not max_length or len(flatten(story)) < max_length]
+    return data
+
+
+def vectorize_stories(data):
+    X = []
+    Xq = []
+    Y = []
+    for story, query, answer in data:
+        x = [word_idx[w] for w in story]
+        xq = [word_idx[w] for w in query]
+        y = np.zeros(vocab_size)
+        y[word_idx[answer]] = 1
+        X.append(x)
+        Xq.append(xq)
+        Y.append(y)
+    return pad_sequences(X, maxlen=story_maxlen), pad_sequences(Xq, maxlen=query_maxlen), np.array(Y)
+
+RNN = recurrent.GRU
+EMBED_HIDDEN_SIZE = 50
+SENT_HIDDEN_SIZE = 100
+QUERY_HIDDEN_SIZE = 100
+BATCH_SIZE = 32
+EPOCHS = 20
+print('RNN / Embed / Sent / Query = {}, {}, {}, {}'.format(RNN, EMBED_HIDDEN_SIZE, SENT_HIDDEN_SIZE, QUERY_HIDDEN_SIZE))
+
+path = get_file('babi-tasks-v1-2.tar.gz', origin='http://www.thespermwhale.com/jaseweston/babi/tasks_1-20_v1-2.tar.gz')
+tar = tarfile.open(path)
+# Default QA1 with 1000 samples
+# challenge = 'tasks_1-20_v1-2/en/qa1_single-supporting-fact_{}.txt'
+# QA1 with 10,000 samples
+# challenge = 'tasks_1-20_v1-2/en-10k/qa1_single-supporting-fact_{}.txt'
+# QA2 with 1000 samples
+challenge = 'tasks_1-20_v1-2/en/qa2_two-supporting-facts_{}.txt'
+# QA2 with 10,000 samples
+# challenge = 'tasks_1-20_v1-2/en-10k/qa2_two-supporting-facts_{}.txt'
+train = get_stories(tar.extractfile(challenge.format('train')))
+test = get_stories(tar.extractfile(challenge.format('test')))
+
+vocab = sorted(reduce(lambda x, y: x | y, (set(story + q + [answer]) for story, q, answer in train + test)))
+# Reserve 0 for masking via pad_sequences
+vocab_size = len(vocab) + 1
+word_idx = dict((c, i + 1) for i, c in enumerate(vocab))
+story_maxlen = max(map(len, (x for x, _, _ in train + test)))
+query_maxlen = max(map(len, (x for _, x, _ in train + test)))
+
+X, Xq, Y = vectorize_stories(train)
+tX, tXq, tY = vectorize_stories(test)
+
+print('vocab = {}'.format(vocab))
+print('X.shape = {}'.format(X.shape))
+print('Xq.shape = {}'.format(Xq.shape))
+print('Y.shape = {}'.format(Y.shape))
+print('story_maxlen, query_maxlen = {}, {}'.format(story_maxlen, query_maxlen))
+
+print('Build model...')
+
+sentrnn = Sequential()
+sentrnn.add(Embedding(vocab_size, EMBED_HIDDEN_SIZE, mask_zero=True))
+sentrnn.add(RNN(EMBED_HIDDEN_SIZE, SENT_HIDDEN_SIZE, return_sequences=False))
+
+qrnn = Sequential()
+qrnn.add(Embedding(vocab_size, EMBED_HIDDEN_SIZE))
+qrnn.add(RNN(EMBED_HIDDEN_SIZE, QUERY_HIDDEN_SIZE, return_sequences=False))
+
+model = Sequential()
+model.add(Merge([sentrnn, qrnn], mode='concat'))
+model.add(Dense(SENT_HIDDEN_SIZE + QUERY_HIDDEN_SIZE, vocab_size, activation='softmax'))
+
+model.compile(optimizer='adam', loss='categorical_crossentropy', class_mode='categorical')
+
+print('Training')
+model.fit([X, Xq], Y, batch_size=BATCH_SIZE, nb_epoch=EPOCHS, validation_split=0.05, show_accuracy=True)
+loss, acc = model.evaluate([tX, tXq], tY, batch_size=BATCH_SIZE, show_accuracy=True)
+print('Test loss / test accuracy = {:.4f} / {:.4f}'.format(loss, acc))

examples/cifar10_cnn.py

@@ -19,7 +19,7 @@ from six.moves import range
     (it's still underfitting at that point, though).
 
     Note: the data was pickled with Python 2, and some encoding issues might prevent you
-    from loading it in Python 3. You might have to load it in Python 2, 
+    from loading it in Python 3. You might have to load it in Python 2,
     save it in a different format, load it in Python 3 and repickle it.
 '''
 
@@ -40,16 +40,16 @@ Y_test = np_utils.to_categorical(y_test, nb_classes)
 
 model = Sequential()
 
-model.add(Convolution2D(32, 3, 3, 3, border_mode='full')) 
+model.add(Convolution2D(32, 3, 3, 3, border_mode='full'))
 model.add(Activation('relu'))
 model.add(Convolution2D(32, 32, 3, 3))
 model.add(Activation('relu'))
 model.add(MaxPooling2D(poolsize=(2, 2)))
 model.add(Dropout(0.25))
 
-model.add(Convolution2D(64, 32, 3, 3, border_mode='full')) 
+model.add(Convolution2D(64, 32, 3, 3, border_mode='full'))
 model.add(Activation('relu'))
-model.add(Convolution2D(64, 64, 3, 3)) 
+model.add(Convolution2D(64, 64, 3, 3))
 model.add(Activation('relu'))
 model.add(MaxPooling2D(poolsize=(2, 2)))
 model.add(Dropout(0.25))
@@ -82,18 +82,18 @@ else:
 
     # this will do preprocessing and realtime data augmentation
     datagen = ImageDataGenerator(
-        featurewise_center=True, # set input mean to 0 over the dataset
-        samplewise_center=False, # set each sample mean to 0
-        featurewise_std_normalization=True, # divide inputs by std of the dataset
-        samplewise_std_normalization=False, # divide each input by its std
-        zca_whitening=False, # apply ZCA whitening
-        rotation_range=20, # randomly rotate images in the range (degrees, 0 to 180)
-        width_shift_range=0.2, # randomly shift images horizontally (fraction of total width)
-        height_shift_range=0.2, # randomly shift images vertically (fraction of total height)
-        horizontal_flip=True, # randomly flip images
-        vertical_flip=False) # randomly flip images
+        featurewise_center=True,  # set input mean to 0 over the dataset
+        samplewise_center=False,  # set each sample mean to 0
+        featurewise_std_normalization=True,  # divide inputs by std of the dataset
+        samplewise_std_normalization=False,  # divide each input by its std
+        zca_whitening=False,  # apply ZCA whitening
+        rotation_range=20,  # randomly rotate images in the range (degrees, 0 to 180)
+        width_shift_range=0.2,  # randomly shift images horizontally (fraction of total width)
+        height_shift_range=0.2,  # randomly shift images vertically (fraction of total height)
+        horizontal_flip=True,  # randomly flip images
+        vertical_flip=False)  # randomly flip images
 
-    # compute quantities required for featurewise normalization 
+    # compute quantities required for featurewise normalization
     # (std, mean, and principal components if ZCA whitening is applied)
     datagen.fit(X_train)
 
@@ -114,11 +114,3 @@ else:
         for X_batch, Y_batch in datagen.flow(X_test, Y_test):
             score = model.test_on_batch(X_batch, Y_batch)
             progbar.add(X_batch.shape[0], values=[("test loss", score)])
-
-            
-
-
-
-
-
-

examples/imdb_lstm.py

@@ -1,7 +1,7 @@
 from __future__ import absolute_import
 from __future__ import print_function
 import numpy as np
-np.random.seed(1337) # for reproducibility
+np.random.seed(1337)  # for reproducibility
 
 from keras.preprocessing import sequence
 from keras.optimizers import SGD, RMSprop, Adagrad
@@ -15,24 +15,24 @@ from keras.datasets import imdb
 '''
     Train a LSTM on the IMDB sentiment classification task.
 
-    The dataset is actually too small for LSTM to be of any advantage 
+    The dataset is actually too small for LSTM to be of any advantage
     compared to simpler, much faster methods such as TF-IDF+LogReg.
 
-    Notes: 
+    Notes:
 
-    - RNNs are tricky. Choice of batch size is important, 
-    choice of loss and optimizer is critical, etc. 
+    - RNNs are tricky. Choice of batch size is important,
+    choice of loss and optimizer is critical, etc.
     Some configurations won't converge.
 
-    - LSTM loss decrease patterns during training can be quite different 
-    from what you see with CNNs/MLPs/etc. 
+    - LSTM loss decrease patterns during training can be quite different
+    from what you see with CNNs/MLPs/etc.
 
     GPU command:
         THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 python imdb_lstm.py
 '''
 
 max_features = 20000
-maxlen = 100 # cut texts after this number of words (among top max_features most common words)
+maxlen = 100  # cut texts after this number of words (among top max_features most common words)
 batch_size = 32
 
 print("Loading data...")
@@ -49,7 +49,7 @@ print('X_test shape:', X_test.shape)
 print('Build model...')
 model = Sequential()
 model.add(Embedding(max_features, 128))
-model.add(LSTM(128, 128)) # try using a GRU instead, for fun
+model.add(LSTM(128, 128))  # try using a GRU instead, for fun
 model.add(Dropout(0.5))
 model.add(Dense(128, 1))
 model.add(Activation('sigmoid'))

examples/kaggle_otto_nn.py

@@ -3,7 +3,7 @@ from __future__ import print_function
 
 import numpy as np
 import pandas as pd
-np.random.seed(1337) # for reproducibility
+np.random.seed(1337)  # for reproducibility
 
 from keras.models import Sequential
 from keras.layers.core import Dense, Dropout, Activation
@@ -36,17 +36,19 @@ from sklearn.preprocessing import StandardScaler
     Get the data from Kaggle: https://www.kaggle.com/c/otto-group-product-classification-challenge/data
 '''
 
+
 def load_data(path, train=True):
     df = pd.read_csv(path)
     X = df.values.copy()
     if train:
-        np.random.shuffle(X) # https://youtu.be/uyUXoap67N8
+        np.random.shuffle(X)  # https://youtu.be/uyUXoap67N8
         X, labels = X[:, 1:-1].astype(np.float32), X[:, -1]
         return X, labels
     else:
         X, ids = X[:, 1:].astype(np.float32), X[:, 0].astype(str)
         return X, ids
 
+
 def preprocess_data(X, scaler=None):
     if not scaler:
         scaler = StandardScaler()
@@ -54,6 +56,7 @@ def preprocess_data(X, scaler=None):
     X = scaler.transform(X)
     return X, scaler
 
+
 def preprocess_labels(labels, encoder=None, categorical=True):
     if not encoder:
         encoder = LabelEncoder()
@@ -63,6 +66,7 @@ def preprocess_labels(labels, encoder=None, categorical=True):
         y = np_utils.to_categorical(y)
     return y, encoder
 
+
 def make_submission(y_prob, ids, encoder, fname):
     with open(fname, 'w') as f:
         f.write('id,')

examples/lstm_text_generation.py

@@ -60,13 +60,10 @@ model.add(Activation('softmax'))
 model.compile(loss='categorical_crossentropy', optimizer='rmsprop')
 
 # helper function to sample an index from a probability array
-def sample(a, diversity=0.75):
-    if random.random() > diversity:
-        return np.argmax(a)
-    while 1:
-        i = random.randint(0, len(a)-1)
-        if a[i] > random.random():
-            return i
+def sample(a, temperature=1.0):
+    a = np.log(a)/temperature
+    a = np.exp(a)/np.sum(np.exp(a))
+    return np.argmax(np.random.multinomial(1,a,1))
 
 # train the model, output generated text after each iteration
 for iteration in range(1, 60):
@@ -77,7 +74,7 @@ for iteration in range(1, 60):
 
     start_index = random.randint(0, len(text) - maxlen - 1)
 
-    for diversity in [0.2, 0.4, 0.6, 0.8]:
+    for diversity in [0.2, 0.5, 1.0, 1.2]:
         print()
         print('----- diversity:', diversity)
 
@@ -101,4 +98,4 @@ for iteration in range(1, 60):
 
             sys.stdout.write(next_char)
             sys.stdout.flush()
-        print()
+        print()

examples/mnist_cnn.py

@@ -1,7 +1,7 @@
 from __future__ import absolute_import
 from __future__ import print_function
 import numpy as np
-np.random.seed(1337) # for reproducibility
+np.random.seed(1337)  # for reproducibility
 
 from keras.datasets import mnist
 from keras.models import Sequential
@@ -41,7 +41,7 @@ Y_test = np_utils.to_categorical(y_test, nb_classes)
 
 model = Sequential()
 
-model.add(Convolution2D(32, 1, 3, 3, border_mode='full')) 
+model.add(Convolution2D(32, 1, 3, 3, border_mode='full'))
 model.add(Activation('relu'))
 model.add(Convolution2D(32, 32, 3, 3))
 model.add(Activation('relu'))

examples/mnist_irnn.py

@@ -1,7 +1,7 @@
 from __future__ import absolute_import
 from __future__ import print_function
 import numpy as np
-np.random.seed(1337) # for reproducibility
+np.random.seed(1337)  # for reproducibility
 
 from keras.datasets import mnist
 from keras.models import Sequential

examples/mnist_mlp.py

@@ -1,7 +1,7 @@
 from __future__ import absolute_import
 from __future__ import print_function
 import numpy as np
-np.random.seed(1337) # for reproducibility
+np.random.seed(1337)  # for reproducibility
 
 from keras.datasets import mnist
 from keras.models import Sequential
@@ -20,8 +20,6 @@ batch_size = 128
 nb_classes = 10
 nb_epoch = 20
 
-
-
 # the data, shuffled and split between tran and test sets
 (X_train, y_train), (X_test, y_test) = mnist.load_data()
 

examples/reuters_mlp.py

@@ -1,7 +1,7 @@
 from __future__ import absolute_import
 from __future__ import print_function
 import numpy as np
-np.random.seed(1337) # for reproducibility
+np.random.seed(1337)  # for reproducibility
 
 from keras.datasets import reuters
 from keras.models import Sequential

examples/skipgram_word_embeddings.py

@@ -1,6 +1,6 @@
 
 '''
-    We loop over words in a dataset, and for each word, we look at a context window around the word. 
+    We loop over words in a dataset, and for each word, we look at a context window around the word.
     We generate pairs of (pivot_word, other_word_from_same_context) with label 1,
     and pairs of (pivot_word, random_word) with label 0 (skip-gram method).
 
@@ -8,23 +8,23 @@
     and compute a proximity score between the embeddings (= p(context|word)),
     trained with our positive and negative labels.
 
-    We then use the weights computed by WordContextProduct to encode words 
-    and demonstrate that the geometry of the embedding space 
+    We then use the weights computed by WordContextProduct to encode words
+    and demonstrate that the geometry of the embedding space
     captures certain useful semantic properties.
 
-    Read more about skip-gram in this particularly gnomic paper by Mikolov et al.: 
+    Read more about skip-gram in this particularly gnomic paper by Mikolov et al.:
         http://arxiv.org/pdf/1301.3781v3.pdf
 
-    Note: you should run this on GPU, otherwise training will be quite slow. 
+    Note: you should run this on GPU, otherwise training will be quite slow.
     On a EC2 GPU instance, expect 3 hours per 10e6 comments (~10e8 words) per epoch with dim_proj=256.
     Should be much faster on a modern GPU.
 
     GPU command:
         THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 python skipgram_word_embeddings.py
 
-    Dataset: 5,845,908 Hacker News comments. 
-    Obtain the dataset at: 
-        https://mega.co.nz/#F!YohlwD7R!wec0yNO86SeaNGIYQBOR0A 
+    Dataset: 5,845,908 Hacker News comments.
+    Obtain the dataset at:
+        https://mega.co.nz/#F!YohlwD7R!wec0yNO86SeaNGIYQBOR0A
         (HNCommentsAll.1perline.json.bz2)
 '''
 from __future__ import absolute_import
@@ -43,10 +43,10 @@ from keras.layers.embeddings import WordContextProduct, Embedding
 from six.moves import range
 from six.moves import zip
 
-max_features = 50000 # vocabulary size: top 50,000 most common words in data
-skip_top = 100 # ignore top 100 most common words
+max_features = 50000  # vocabulary size: top 50,000 most common words in data
+skip_top = 100  # ignore top 100 most common words
 nb_epoch = 1
-dim_proj = 256 # embedding space dimension
+dim_proj = 256  # embedding space dimension
 
 save = True
 load_model = False
@@ -66,6 +66,7 @@ html_tags = re.compile(r'<.*?>')
 to_replace = [('&#x27;', "'")]
 hex_tags = re.compile(r'&.*?;')
 
+
 def clean_comment(comment):
     c = str(comment.encode("utf-8"))
     c = html_tags.sub(' ', c)
@@ -74,6 +75,7 @@ def clean_comment(comment):
     c = hex_tags.sub(' ', c)
     return c
 
+
 def text_generator(path=data_path):
     f = open(path)
     for i, l in enumerate(f):
@@ -120,7 +122,7 @@ if train_model:
         progbar = generic_utils.Progbar(tokenizer.document_count)
         samples_seen = 0
         losses = []
-        
+
         for i, seq in enumerate(tokenizer.texts_to_sequences_generator(text_generator())):
             # get skipgram couples for one text in the dataset
             couples, labels = sequence.skipgrams(seq, max_features, window_size=4, negative_samples=1., sampling_table=sampling_table)
@@ -158,26 +160,29 @@ word_index = tokenizer.word_index
 reverse_word_index = dict([(v, k) for k, v in list(word_index.items())])
 word_index = tokenizer.word_index
 
+
 def embed_word(w):
     i = word_index.get(w)
-    if (not i) or (i<skip_top) or (i>=max_features):
+    if (not i) or (i < skip_top) or (i >= max_features):
         return None
     return norm_weights[i]
 
+
 def closest_to_point(point, nb_closest=10):
     proximities = np.dot(norm_weights, point)
     tups = list(zip(list(range(len(proximities))), proximities))
     tups.sort(key=lambda x: x[1], reverse=True)
-    return [(reverse_word_index.get(t[0]), t[1]) for t in tups[:nb_closest]]  
+    return [(reverse_word_index.get(t[0]), t[1]) for t in tups[:nb_closest]]
+
 
 def closest_to_word(w, nb_closest=10):
     i = word_index.get(w)
-    if (not i) or (i<skip_top) or (i>=max_features):
+    if (not i) or (i < skip_top) or (i >= max_features):
         return []
     return closest_to_point(norm_weights[i].T, nb_closest)
 
 
-''' the resuls in comments below were for: 
+''' the resuls in comments below were for:
     5.8M HN comments
     dim_proj = 256
     nb_epoch = 2
@@ -187,26 +192,27 @@ def closest_to_word(w, nb_closest=10):
     skip_top = 100
     negative_samples = 1.
     window_size = 4
-    and frequency subsampling of factor 10e-5. 
+    and frequency subsampling of factor 10e-5.
 '''
 
-words = ["article", # post, story, hn, read, comments
-"3", # 6, 4, 5, 2
-"two", # three, few, several, each
-"great", # love, nice, working, looking
-"data", # information, memory, database
-"money", # company, pay, customers, spend
-"years", # ago, year, months, hours, week, days
-"android", # ios, release, os, mobile, beta
-"javascript", # js, css, compiler, library, jquery, ruby
-"look", # looks, looking
-"business", # industry, professional, customers
-"company", # companies, startup, founders, startups
-"after", # before, once, until
-"own", # personal, our, having
-"us", # united, country, american, tech, diversity, usa, china, sv
-"using", # javascript, js, tools (lol)
-"here", # hn, post, comments
+words = [
+    "article",  # post, story, hn, read, comments
+    "3",  # 6, 4, 5, 2
+    "two",  # three, few, several, each
+    "great",  # love, nice, working, looking
+    "data",  # information, memory, database
+    "money",  # company, pay, customers, spend
+    "years",  # ago, year, months, hours, week, days
+    "android",  # ios, release, os, mobile, beta
+    "javascript",  # js, css, compiler, library, jquery, ruby
+    "look",  # looks, looking
+    "business",  # industry, professional, customers
+    "company",  # companies, startup, founders, startups
+    "after",  # before, once, until
+    "own",  # personal, our, having
+    "us",  # united, country, american, tech, diversity, usa, china, sv
+    "using",  # javascript, js, tools (lol)
+    "here",  # hn, post, comments
 ]
 
 for w in words:
@@ -214,4 +220,3 @@ for w in words:
     print('====', w)
     for r in res:
         print(r)
-

keras/activations.py

@@ -1,35 +1,44 @@
 from __future__ import absolute_import
 import theano.tensor as T
 
+
 def softmax(x):
     return T.nnet.softmax(x.reshape((-1, x.shape[-1]))).reshape(x.shape)
 
+
 def time_distributed_softmax(x):
     import warnings
     warnings.warn("time_distributed_softmax is deprecated. Just use softmax!", DeprecationWarning)
     return softmax(x)
 
+
 def softplus(x):
     return T.nnet.softplus(x)
 
+
 def relu(x):
     return (x + abs(x)) / 2.0
 
+
 def tanh(x):
     return T.tanh(x)
 
+
 def sigmoid(x):
     return T.nnet.sigmoid(x)
 
+
 def hard_sigmoid(x):
     return T.nnet.hard_sigmoid(x)
 
+
 def linear(x):
     '''
     The function returns the variable that is passed in, so all types work
     '''
     return x
 
+
 from .utils.generic_utils import get_from_module
 def get(identifier):
     return get_from_module(identifier, globals(), 'activation function')

keras/callbacks.py

@@ -1,16 +1,14 @@
 from __future__ import absolute_import
 from __future__ import print_function
-import theano
-import theano.tensor as T
-import numpy as np
-import warnings
-import time, json
-from collections import deque
 
+import numpy as np
+import time, json, warnings
+
+from collections import deque
 from .utils.generic_utils import Progbar
 
-class CallbackList(object):
 
+class CallbackList(object):
     def __init__(self, callbacks=[], queue_length=10):
         self.callbacks = [c for c in callbacks]
         self.queue_length = queue_length
@@ -43,10 +41,9 @@ class CallbackList(object):
             callback.on_batch_begin(batch, logs)
         self._delta_ts_batch_begin.append(time.time() - t_before_callbacks)
         delta_t_median = np.median(self._delta_ts_batch_begin)
-        if self._delta_t_batch > 0. and delta_t_median > 0.95 * self._delta_t_batch \
-            and delta_t_median > 0.1:
+        if self._delta_t_batch > 0. and delta_t_median > 0.95 * self._delta_t_batch and delta_t_median > 0.1:
             warnings.warn('Method on_batch_begin() is slow compared '
-                'to the batch update (%f). Check your callbacks.' % delta_t_median)
+                          'to the batch update (%f). Check your callbacks.' % delta_t_median)
         self._t_enter_batch = time.time()
 
     def on_batch_end(self, batch, logs={}):
@@ -56,10 +53,9 @@ class CallbackList(object):
             callback.on_batch_end(batch, logs)
         self._delta_ts_batch_end.append(time.time() - t_before_callbacks)
         delta_t_median = np.median(self._delta_ts_batch_end)
-        if self._delta_t_batch > 0. and delta_t_median > 0.95 * self._delta_t_batch \
-            and delta_t_median > 0.1:
+        if self._delta_t_batch > 0. and delta_t_median > 0.95 * self._delta_t_batch and delta_t_median > 0.1:
             warnings.warn('Method on_batch_end() is slow compared '
-                'to the batch update (%f). Check your callbacks.' % delta_t_median)
+                          'to the batch update (%f). Check your callbacks.' % delta_t_median)
 
     def on_train_begin(self, logs={}):
         for callback in self.callbacks:
@@ -99,16 +95,16 @@ class Callback(object):
     def on_train_end(self, logs={}):
         pass
 
-class BaseLogger(Callback):
 
+class BaseLogger(Callback):
     def on_train_begin(self, logs={}):
         self.verbose = self.params['verbose']
 
     def on_epoch_begin(self, epoch, logs={}):
         if self.verbose:
             print('Epoch %d' % epoch)
-            self.progbar = Progbar(target=self.params['nb_sample'], \
-                verbose=self.verbose)
+            self.progbar = Progbar(target=self.params['nb_sample'],
+                                   verbose=self.verbose)
         self.seen = 0
         self.totals = {}
 
@@ -177,8 +173,8 @@ class History(Callback):
 
 class ModelCheckpoint(Callback):
     def __init__(self, filepath, monitor='val_loss', verbose=0, save_best_only=False):
+
         super(Callback, self).__init__()
-        
         self.monitor = monitor
         self.verbose = verbose
         self.filepath = filepath
@@ -194,7 +190,7 @@ class ModelCheckpoint(Callback):
                 if current < self.best:
                     if self.verbose > 0:
                         print("Epoch %05d: %s improved from %0.5f to %0.5f, saving model to %s"
-                            % (epoch, self.monitor, self.best, current, self.filepath))
+                              % (epoch, self.monitor, self.best, current, self.filepath))
                     self.best = current
                     self.model.save_weights(self.filepath, overwrite=True)
                 else:
@@ -220,7 +216,7 @@ class EarlyStopping(Callback):
         current = logs.get(self.monitor)
         if current is None:
             warnings.warn("Early stopping requires %s available!" % (self.monitor), RuntimeWarning)
- 
+
         if current < self.best:
             self.best = current
             self.wait = 0
@@ -239,7 +235,6 @@ class RemoteMonitor(Callback):
     def on_epoch_begin(self, epoch, logs={}):
         self.seen = 0
         self.totals = {}
-        
 
     def on_batch_end(self, batch, logs={}):
         batch_size = logs.get('size', 0)
@@ -257,7 +252,23 @@ class RemoteMonitor(Callback):
 
         for k, v in self.totals.items():
             send[k] = v / self.seen
-        for k, v in self.logs:
+        for k, v in logs.items():
             send[k] = v
 
-        r = requests.post(self.root + '/publish/epoch/end/', {'data':json.dumps(send)})
+        try:
+            r = requests.post(self.root + '/publish/epoch/end/', {'data': json.dumps(send)})
+        except:
+            print('Warning: could not reach RemoteMonitor root server at ' + str(self.root))
+
+
+class LearningRateScheduler(Callback):
+    '''LearningRateScheduler
+    schedule is a function that gets an epoch number as input and returns a new
+    learning rate as output.
+    '''
+    def __init__(self, schedule):
+        super(LearningRateScheduler, self).__init__()
+        self.schedule = schedule
+
+    def on_epoch_begin(self, epoch, logs={}):
+        model.lr.set_value(self.schedule(epoch))

keras/constraints.py

@@ -3,12 +3,14 @@ import theano
 import theano.tensor as T
 import numpy as np
 
+
 class Constraint(object):
     def __call__(self, p):
         return p
 
     def get_config(self):
-        return {"name":self.__class__.__name__}
+        return {"name": self.__class__.__name__}
+
 
 class MaxNorm(Constraint):
     def __init__(self, m=2):
@@ -21,14 +23,16 @@ class MaxNorm(Constraint):
         return p
 
     def get_config(self):
-        return {"name":self.__class__.__name__,
-            "m":self.m}
+        return {"name": self.__class__.__name__,
+                "m": self.m}
+
 
 class NonNeg(Constraint):
     def __call__(self, p):
         p *= T.ge(p, 0)
         return p
 
+
 class UnitNorm(Constraint):
     def __call__(self, p):
         return p / T.sqrt(T.sum(p**2, axis=-1, keepdims=True))
@@ -40,4 +44,4 @@ unitnorm = UnitNorm
 
 from .utils.generic_utils import get_from_module
 def get(identifier, kwargs=None):
-    return get_from_module(identifier, globals(), 'constraint', instantiate=True, kwargs=kwargs)
+    return get_from_module(identifier, globals(), 'constraint', instantiate=True, kwargs=kwargs)

keras/datasets/cifar10.py

@@ -4,6 +4,7 @@ from .data_utils import get_file
 import numpy as np
 import os
 
+
 def load_data():
     dirname = "cifar-10-batches-py"
     origin = "http://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz"
@@ -20,11 +21,11 @@ def load_data():
         data, labels = load_batch(fpath)
         X_train[(i-1)*10000:i*10000, :, :, :] = data
         y_train[(i-1)*10000:i*10000] = labels
-    
+
     fpath = os.path.join(path, 'test_batch')
     X_test, y_test = load_batch(fpath)
 
     y_train = np.reshape(y_train, (len(y_train), 1))
     y_test = np.reshape(y_test, (len(y_test), 1))
 
-    return (X_train, y_train), (X_test, y_test) 
+    return (X_train, y_train), (X_test, y_test)

keras/datasets/cifar100.py

@@ -4,6 +4,7 @@ from .data_utils import get_file
 import numpy as np
 import os
 
+
 def load_data(label_mode='fine'):
     if label_mode not in ['fine', 'coarse']:
         raise Exception('label_mode must be one of "fine" "coarse".')
@@ -24,4 +25,4 @@ def load_data(label_mode='fine'):
     y_train = np.reshape(y_train, (len(y_train), 1))
     y_test = np.reshape(y_test, (len(y_test), 1))
 
-    return (X_train, y_train), (X_test, y_test) 
+    return (X_train, y_train), (X_test, y_test)

keras/datasets/data_utils.py

@@ -1,9 +1,15 @@
 from __future__ import absolute_import
 from __future__ import print_function
+
 import tarfile, inspect, os
-from six.moves.urllib.request import urlretrieve
+from six.moves.urllib.request import FancyURLopener
+
 from ..utils.generic_utils import Progbar
 
+class ParanoidURLopener(FancyURLopener):
+  def http_error_default(self, url, fp, errcode, errmsg, headers):
+    raise Exception('URL fetch failure on {}: {} -- {}'.format(url, errcode, errmsg))
+
 def get_file(fname, origin, untar=False):
     datadir = os.path.expanduser(os.path.join('~', '.keras', 'datasets'))
     if not os.path.exists(datadir):
@@ -22,6 +28,7 @@ def get_file(fname, origin, untar=False):
 
         global progbar
         progbar = None
+
         def dl_progress(count, block_size, total_size):
             global progbar
             if progbar is None:
@@ -29,7 +36,7 @@ def get_file(fname, origin, untar=False):
             else:
                 progbar.update(count*block_size)
 
-        urlretrieve(origin, fpath, dl_progress)
+        ParanoidURLopener().retrieve(origin, fpath, dl_progress)
         progbar = None
 
     if untar:
@@ -41,4 +48,3 @@ def get_file(fname, origin, untar=False):
         return untar_fpath
 
     return fpath
-

keras/datasets/imdb.py

@@ -6,8 +6,9 @@ import random
 from six.moves import zip
 import numpy as np
 
-def load_data(path="imdb.pkl", nb_words=None, skip_top=0, maxlen=None, test_split=0.2, seed=113, 
-    start_char=1, oov_char=2, index_from=3):
+
+def load_data(path="imdb.pkl", nb_words=None, skip_top=0, maxlen=None, test_split=0.2, seed=113,
+              start_char=1, oov_char=2, index_from=3):
 
     path = get_file(path, origin="https://s3.amazonaws.com/text-datasets/imdb.pkl")
 
@@ -63,4 +64,3 @@ def load_data(path="imdb.pkl", nb_words=None, skip_top=0, maxlen=None, test_spli
     y_test = labels[int(len(X)*(1-test_split)):]
 
     return (X_train, y_train), (X_test, y_test)
-

keras/datasets/mnist.py

@@ -4,6 +4,7 @@ from .data_utils import get_file
 import six.moves.cPickle
 import sys
 
+
 def load_data(path="mnist.pkl.gz"):
     path = get_file(path, origin="https://s3.amazonaws.com/img-datasets/mnist.pkl.gz")
 
@@ -19,4 +20,4 @@ def load_data(path="mnist.pkl.gz"):
 
     f.close()
 
-    return data # (X_train, y_train), (X_test, y_test)
+    return data  # (X_train, y_train), (X_test, y_test)

keras/datasets/reuters.py

@@ -9,6 +9,7 @@ import six.moves.cPickle
 from six.moves import zip
 import numpy as np
 
+
 def make_reuters_dataset(path=os.path.join('datasets', 'temp', 'reuters21578'), min_samples_per_topic=15):
     import re
     from ..preprocessing.text import Tokenizer
@@ -24,8 +25,7 @@ def make_reuters_dataset(path=os.path.join('datasets', 'temp', 'reuters21578'),
             while tag in s:
                 s = s[s.find(tag)+len(tag):]
                 topics = s[:s.find('</')]
-                
-                if topics and not '</D><D>' in topics:
+                if topics and '</D><D>' not in topics:
                     topic = topics.replace('<D>', '').replace('</D>', '')
                     wire_topics.append(topic)
                     topic_counts[topic] = topic_counts.get(topic, 0) + 1
@@ -39,7 +39,7 @@ def make_reuters_dataset(path=os.path.join('datasets', 'temp', 'reuters21578'),
 
     # only keep most common topics
     items = list(topic_counts.items())
-    items.sort(key = lambda x: x[1])
+    items.sort(key=lambda x: x[1])
     kept_topics = set()
     for x in items:
         print(x[0] + ': ' + str(x[1]))
@@ -75,16 +75,15 @@ def make_reuters_dataset(path=os.path.join('datasets', 'temp', 'reuters21578'),
     reverse_word_index = dict([(v, k) for k, v in tokenizer.word_index.items()])
     print(' '.join(reverse_word_index[i] for i in X[10]))
 
-    dataset = (X, labels) 
+    dataset = (X, labels)
     print('-')
     print('Saving...')
     six.moves.cPickle.dump(dataset, open(os.path.join('datasets', 'data', 'reuters.pkl'), 'w'))
     six.moves.cPickle.dump(tokenizer.word_index, open(os.path.join('datasets', 'data', 'reuters_word_index.pkl'), 'w'))
 
 
-
 def load_data(path="reuters.pkl", nb_words=None, skip_top=0, maxlen=None, test_split=0.2, seed=113,
-    start_char=1, oov_char=2, index_from=3):
+              start_char=1, oov_char=2, index_from=3):
 
     path = get_file(path, origin="https://s3.amazonaws.com/text-datasets/reuters.pkl")
     f = open(path, 'rb')

keras/initializations.py

@@ -5,6 +5,7 @@ import numpy as np
 
 from .utils.theano_utils import sharedX, shared_zeros, shared_ones
 
+
 def get_fans(shape):
     fan_in = shape[0] if len(shape) == 2 else np.prod(shape[1:])
     fan_out = shape[1] if len(shape) == 2 else shape[0]
@@ -14,9 +15,11 @@ def get_fans(shape):
 def uniform(shape, scale=0.05):
     return sharedX(np.random.uniform(low=-scale, high=scale, size=shape))
 
+
 def normal(shape, scale=0.05):
     return sharedX(np.random.randn(*shape) * scale)
 
+
 def lecun_uniform(shape):
     ''' Reference: LeCun 98, Efficient Backprop
         http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf
@@ -25,6 +28,7 @@ def lecun_uniform(shape):
     scale = np.sqrt(3. / fan_in)
     return uniform(shape, scale)
 
+
 def glorot_normal(shape):
     ''' Reference: Glorot & Bengio, AISTATS 2010
     '''
@@ -32,11 +36,13 @@ def glorot_normal(shape):
     s = np.sqrt(2. / (fan_in + fan_out))
     return normal(shape, s)
 
+
 def glorot_uniform(shape):
     fan_in, fan_out = get_fans(shape)
     s = np.sqrt(6. / (fan_in + fan_out))
     return uniform(shape, s)
-    
+
+
 def he_normal(shape):
     ''' Reference:  He et al., http://arxiv.org/abs/1502.01852
     '''
@@ -44,30 +50,36 @@ def he_normal(shape):
     s = np.sqrt(2. / fan_in)
     return normal(shape, s)
 
+
 def he_uniform(shape):
     fan_in, fan_out = get_fans(shape)
     s = np.sqrt(6. / fan_in)
     return uniform(shape, s)
 
+
 def orthogonal(shape, scale=1.1):
     ''' From Lasagne
     '''
     flat_shape = (shape[0], np.prod(shape[1:]))
     a = np.random.normal(0.0, 1.0, flat_shape)
     u, _, v = np.linalg.svd(a, full_matrices=False)
-    q = u if u.shape == flat_shape else v # pick the one with the correct shape
+    # pick the one with the correct shape
+    q = u if u.shape == flat_shape else v
     q = q.reshape(shape)
     return sharedX(scale * q[:shape[0], :shape[1]])
 
+
 def identity(shape, scale=1):
     if len(shape) != 2 or shape[0] != shape[1]:
         raise Exception("Identity matrix initialization can only be used for 2D square matrices")
     else:
         return sharedX(scale * np.identity(shape[0]))
 
+
 def zero(shape):
     return shared_zeros(shape)
 
+
 def one(shape):
     return shared_ones(shape)
 

keras/layers/advanced_activations.py

@@ -1,9 +1,13 @@
-from ..layers.core import Layer
-from ..utils.theano_utils import shared_zeros
+from .. import initializations
+from ..layers.core import Layer, MaskedLayer
+from ..utils.theano_utils import shared_zeros, shared_ones, sharedX
+import theano.tensor as T
+import numpy as np
 
-class LeakyReLU(Layer):
+
+class LeakyReLU(MaskedLayer):
     def __init__(self, alpha=0.3):
-        super(LeakyReLU,self).__init__()
+        super(LeakyReLU, self).__init__()
         self.alpha = alpha
 
     def get_output(self, train):
@@ -11,22 +15,26 @@ class LeakyReLU(Layer):
         return ((X + abs(X)) / 2.0) + self.alpha * ((X - abs(X)) / 2.0)
 
     def get_config(self):
-        return {"name":self.__class__.__name__,
-            "alpha":self.alpha}
+        return {"name": self.__class__.__name__,
+                "alpha": self.alpha}
 
 
-class PReLU(Layer):
+class PReLU(MaskedLayer):
     '''
-        Reference: 
+        Reference:
             Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
                 http://arxiv.org/pdf/1502.01852v1.pdf
     '''
-    def __init__(self, input_shape):
-        super(PReLU,self).__init__()
-        self.alphas = shared_zeros(input_shape)
+    def __init__(self, input_shape, init='zero', weights=None):
+        super(PReLU, self).__init__()
+        self.init = initializations.get(init)
+        self.alphas = self.init(input_shape)
         self.params = [self.alphas]
         self.input_shape = input_shape
 
+        if weights is not None:
+            self.set_weights(weights)
+
     def get_output(self, train):
         X = self.get_input(train)
         pos = ((X + abs(X)) / 2.0)
@@ -34,5 +42,78 @@ class PReLU(Layer):
         return pos + neg
 
     def get_config(self):
-        return {"name":self.__class__.__name__,
-        "input_shape":self.input_shape}
+        return {"name": self.__class__.__name__,
+                "input_shape": self.input_shape,
+                "init": self.init.__name__}
+
+
+class ParametricSoftplus(MaskedLayer):
+    '''
+        Parametric Softplus of the form: alpha * log(1 + exp(beta * X))
+
+        Reference:
+            Inferring Nonlinear Neuronal Computation Based on Physiologically Plausible Inputs
+            http://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1003143
+    '''
+    def __init__(self, input_shape, alpha_init=0.2, beta_init=5.0, weights=None):
+
+        super(ParametricSoftplus, self).__init__()
+        self.alpha_init = alpha_init
+        self.beta_init = beta_init
+        self.alphas = sharedX(alpha_init * np.ones(input_shape))
+        self.betas = sharedX(beta_init * np.ones(input_shape))
+        self.params = [self.alphas, self.betas]
+        self.input_shape = input_shape
+
+        if weights is not None:
+            self.set_weights(weights)
+
+    def get_output(self, train):
+        X = self.get_input(train)
+        return T.nnet.softplus(self.betas * X) * self.alphas
+
+    def get_config(self):
+        return {"name": self.__class__.__name__,
+                "input_shape": self.input_shape,
+                "alpha_init": self.alpha_init,
+                "beta_init": self.beta_init}
+
+class ThresholdedLinear(MaskedLayer):
+    '''
+        Thresholded Linear Activation
+
+        Reference:
+            Zero-Bias Autoencoders and the Benefits of Co-Adapting Features
+            http://arxiv.org/pdf/1402.3337.pdf
+    '''
+    def __init__(self, theta=1.0):
+        super(ThresholdedLinear, self).__init__()
+        self.theta = theta
+    
+    def get_output(self, train):
+        X = self.get_input(train)
+        return T.switch( abs(X) < self.theta, 0, X )
+
+    def get_config(self):
+        return {"name": self.__class__.__name__,
+            "theta": self.theta}
+
+class ThresholdedReLu(MaskedLayer):
+    '''
+        Thresholded Rectified Activation
+
+        Reference:
+            Zero-Bias Autoencoders and the Benefits of Co-Adapting Features
+            http://arxiv.org/pdf/1402.3337.pdf
+    '''
+    def __init__(self, theta=1.0):
+        super(ThresholdedReLu, self).__init__()
+        self.theta = theta
+    
+    def get_output(self, train):
+        X = self.get_input(train)
+        return T.switch( X > self.theta, X, 0 )
+
+    def get_config(self):
+        return {"name": self.__class__.__name__,
+            "theta": self.theta}

keras/layers/containers.py

@@ -4,16 +4,9 @@ from __future__ import print_function
 
 import theano.tensor as T
 from ..layers.core import Layer, Merge
+from ..utils.theano_utils import ndim_tensor
 from six.moves import range
 
-def ndim_tensor(ndim):
-    if ndim == 2:
-        return T.matrix()
-    elif ndim == 3:
-        return T.tensor3()
-    elif ndim == 4:
-        return T.tensor4()
-    return T.matrix()
 
 class Sequential(Layer):
     '''
@@ -30,6 +23,7 @@ class Sequential(Layer):
         self.params = []
         self.regularizers = []
         self.constraints = []
+        self.updates = []
 
         for layer in layers:
             self.add(layer)
@@ -41,11 +35,15 @@ class Sequential(Layer):
         self.layers.append(layer)
         if len(self.layers) > 1:
             self.layers[-1].set_previous(self.layers[-2])
+            if not hasattr(self.layers[0], 'input'):
+                self.set_input()
+        layer.init_updates()
 
-        params, regularizers, constraints = layer.get_params()
+        params, regularizers, constraints, updates = layer.get_params()
         self.params += params
         self.regularizers += regularizers
         self.constraints += constraints
+        self.updates += updates
 
     def get_output(self, train=False):
         return self.layers[-1].get_output(train)
@@ -79,8 +77,8 @@ class Sequential(Layer):
             weights = weights[nb_param:]
 
     def get_config(self):
-        return {"name":self.__class__.__name__,
-            "layers":[layer.get_config() for layer in self.layers]}
+        return {"name": self.__class__.__name__,
+                "layers": [layer.get_config() for layer in self.layers]}
 
 
 class Graph(Layer):
@@ -100,67 +98,84 @@ class Graph(Layer):
             - set_weights
     '''
     def __init__(self):
-        self.namespace = set() # strings
-        self.nodes = {} # layer-like
-        self.inputs = {} # layer-like
-        self.input_order = [] # strings
-        self.outputs = {} # layer-like
-        self.output_order = [] # strings
-        self.input_config = [] # dicts
-        self.output_config = [] # dicts
-        self.node_config = [] # dicts
+        self.namespace = set()  # strings
+        self.nodes = {}  # layer-like
+        self.inputs = {}  # layer-like
+        self.input_order = []  # strings
+        self.outputs = {}  # layer-like
+        self.output_order = []  # strings
+        self.input_config = []  # dicts
+        self.output_config = []  # dicts
+        self.node_config = []  # dicts
 
         self.params = []
         self.regularizers = []
         self.constraints = []
+        self.updates = []
 
-    def set_previous(self, layer):
-        if len(self.inputs) != 1 or len(self.outputs) != 1:
-            raise Exception('The Graph container can only be used as a layer \
-                when it has exactly one input and one output.')
-        self.inputs[self.input_order[0]].set_previous(layer)
+    @property
+    def nb_input(self):
+        return len(self.inputs)
+
+    @property
+    def nb_output(self):
+        return len(self.outputs)
+
+    def set_previous(self, layer, connection_map={}):
+        if self.nb_input != layer.nb_output:
+            raise Exception('Cannot connect layers: input count does not match output count.')
+        if self.nb_input == 1:
+            self.inputs[self.input_order[0]].set_previous(layer)
+        else:
+            if not connection_map:
+                raise Exception('Cannot attach multi-input layer: no connection_map provided.')
+            for k, v in connection_map.items():
+                if k in self.inputs and v in layer.outputs:
+                    self.inputs[k].set_previous(layer.outputs[v])
+                else:
+                    raise Exception('Invalid connection map.')
 
     def get_input(self, train=False):
-        if len(self.inputs) != 1 or len(self.outputs) != 1:
-            raise Exception('The Graph container can only be used as a layer \
-                when it has exactly one input and one output.')
-        return self.inputs[self.input_order[0]].get_input(train)
+        if len(self.inputs) == len(self.outputs) == 1:
+            return self.inputs[self.input_order[0]].get_input(train)
+        else:
+            return dict([(k, v.get_input(train)) for k, v in self.inputs.items()])
 
     @property
     def input(self):
         return self.get_input()
 
     def get_output(self, train=False):
-        if len(self.inputs) != 1 or len(self.outputs) != 1:
-            raise Exception('The Graph container can only be used as a layer \
-                when it has exactly one input and one output.')
-        return self.outputs[self.output_order[0]].get_output(train)
+        if len(self.inputs) == len(self.outputs) == 1:
+            return self.outputs[self.output_order[0]].get_output(train)
+        else:
+            return dict([(k, v.get_output(train)) for k, v in self.outputs.items()])
 
     def add_input(self, name, ndim=2, dtype='float'):
         if name in self.namespace:
             raise Exception('Duplicate node identifier: ' + name)
         self.namespace.add(name)
         self.input_order.append(name)
-        layer = Layer() # empty layer
+        layer = Layer()  # empty layer
         if dtype == 'float':
             layer.input = ndim_tensor(ndim)
         else:
             if ndim == 2:
                 layer.input = T.imatrix()
             else:
-                raise Exception('Type "int" can only be used with ndim==2.')
+                raise Exception('Type "int" can only be used with ndim==2 (Embedding).')
         layer.input.name = name
         self.inputs[name] = layer
-        self.input_config.append({'name':name, 'ndim':ndim, 'dtype':dtype})
+        self.input_config.append({'name': name, 'ndim': ndim, 'dtype': dtype})
 
-    def add_node(self, layer, name, input=None, inputs=[], merge_mode='concat'):
+    def add_node(self, layer, name, input=None, inputs=[], merge_mode='concat', create_output=False):
         if hasattr(layer, 'set_name'):
             layer.set_name(name)
         if name in self.namespace:
             raise Exception('Duplicate node identifier: ' + name)
         if input:
             if input not in self.namespace:
-                raise Exception('Unknown identifier: ' + input)
+                raise Exception('Unknown node/input identifier: ' + input)
             if input in self.nodes:
                 layer.set_previous(self.nodes[input])
             elif input in self.inputs:
@@ -179,22 +194,30 @@ class Graph(Layer):
 
         self.namespace.add(name)
         self.nodes[name] = layer
-        self.node_config.append({'name':name, 'input':input, 'inputs':inputs, 'merge_mode':merge_mode})
-        params, regularizers, constraints = layer.get_params()
+        self.node_config.append({'name': name,
+                                 'input': input,
+                                 'inputs': inputs,
+                                 'merge_mode': merge_mode})
+        layer.init_updates()
+        params, regularizers, constraints, updates = layer.get_params()
         self.params += params
         self.regularizers += regularizers
         self.constraints += constraints
+        self.updates += updates
+
+        if create_output:
+            self.add_output(name, input=name)
 
     def add_output(self, name, input=None, inputs=[], merge_mode='concat'):
-        if name in self.namespace:
-            raise Exception('Duplicate node identifier: ' + name)
+        if name in self.output_order:
+            raise Exception('Duplicate output identifier: ' + name)
         if input:
             if input not in self.namespace:
-                raise Exception('Unknown identifier: ' + input)
+                raise Exception('Unknown node/input identifier: ' + input)
             if input in self.nodes:
                 self.outputs[name] = self.nodes[input]
             elif input in self.inputs:
-                self.ouputs[name] = self.inputs[input]
+                self.outputs[name] = self.inputs[input]
         if inputs:
             to_merge = []
             for n in inputs:
@@ -203,15 +226,18 @@ class Graph(Layer):
                 to_merge.append(self.nodes[n])
             merge = Merge(to_merge, mode=merge_mode)
             self.outputs[name] = merge
-        self.namespace.add(name)
+
         self.output_order.append(name)
-        self.output_config.append({'name':name, 'input':input, 'inputs':inputs, 'merge_mode':merge_mode})
+        self.output_config.append({'name': name,
+                                   'input': input,
+                                   'inputs': inputs,
+                                   'merge_mode': merge_mode})
 
     def get_config(self):
-        return {"name":self.__class__.__name__,
-            "input_config":self.input_config,
-            "node_config":self.node_config,
-            "output_config":self.output_config,
-            "input_order":self.input_order,
-            "output_order":self.output_order,
-            "nodes":dict([(c["name"], self.nodes[c["name"]].get_config()) for c in self.node_config])}
+        return {"name": self.__class__.__name__,
+                "input_config": self.input_config,
+                "node_config": self.node_config,
+                "output_config": self.output_config,
+                "input_order": self.input_order,
+                "output_order": self.output_order,
+                "nodes": dict([(c["name"], self.nodes[c["name"]].get_config()) for c in self.node_config])}

keras/layers/convolutional.py

@@ -3,22 +3,24 @@ from __future__ import absolute_import
 
 import theano
 import theano.tensor as T
-from theano.tensor.signal import downsample
+from theano.sandbox.cuda import dnn
 
 from .. import activations, initializations, regularizers, constraints
 from ..utils.theano_utils import shared_zeros
 from ..layers.core import Layer
 
+
 class Convolution1D(Layer):
     def __init__(self, input_dim, nb_filter, filter_length,
-        init='uniform', activation='linear', weights=None,
-        border_mode='valid', subsample_length=1,
-        W_regularizer=None, b_regularizer=None, activity_regularizer=None, W_constraint=None, b_constraint=None):
+                 init='uniform', activation='linear', weights=None,
+                 border_mode='valid', subsample_length=1,
+                 W_regularizer=None, b_regularizer=None, activity_regularizer=None,
+                 W_constraint=None, b_constraint=None):
 
-        if border_mode not in {'valid', 'full'}:
+        if border_mode not in {'valid', 'full', 'same'}:
             raise Exception('Invalid border mode for Convolution1D:', border_mode)
 
-        super(Convolution1D,self).__init__()
+        super(Convolution1D, self).__init__()
         self.nb_filter = nb_filter
         self.input_dim = input_dim
         self.filter_length = filter_length
@@ -61,67 +63,48 @@ class Convolution1D(Layer):
 
     def get_output(self, train):
         X = self.get_input(train)
-        X = theano.tensor.reshape(X, (X.shape[0], X.shape[1], X.shape[2], 1)).dimshuffle(0, 2, 1, 3)
-        conv_out = theano.tensor.nnet.conv.conv2d(X, self.W, border_mode=self.border_mode, subsample=self.subsample)
+        X = T.reshape(X, (X.shape[0], X.shape[1], X.shape[2], 1)).dimshuffle(0, 2, 1, 3)
+
+        border_mode = self.border_mode
+        if border_mode == 'same':
+            border_mode = 'full'
+
+        conv_out = T.nnet.conv.conv2d(X, self.W, border_mode=border_mode, subsample=self.subsample)
+        if self.border_mode == 'same':
+            shift_x = (self.filter_length - 1) // 2
+            conv_out = conv_out[:, :, shift_x:X.shape[2] + shift_x, :]
+
         output = self.activation(conv_out + self.b.dimshuffle('x', 0, 'x', 'x'))
-        return theano.tensor.reshape(output, (output.shape[0], output.shape[1], output.shape[2])).dimshuffle(0, 2, 1)
+        output = T.reshape(output, (output.shape[0], output.shape[1], output.shape[2])).dimshuffle(0, 2, 1)
+        return output
 
     def get_config(self):
-        return {"name":self.__class__.__name__,
-            "input_dim":self.input_dim,
-            "nb_filter":self.nb_filter,
-            "filter_length":self.filter_length,
-            "init":self.init.__name__,
-            "activation":self.activation.__name__,
-            "border_mode":self.border_mode,
-            "subsample_length":self.subsample_length,
-            "W_regularizer":self.W_regularizer.get_config() if self.W_regularizer else None,
-            "b_regularizer":self.b_regularizer.get_config() if self.b_regularizer else None,
-            "activity_regularizer":self.activity_regularizer.get_config() if self.activity_regularizer else None,
-            "W_constraint":self.W_constraint.get_config() if self.W_constraint else None,
-            "b_constraint":self.b_constraint.get_config() if self.b_constraint else None}
-
-
-class MaxPooling1D(Layer):
-    def __init__(self, pool_length=2, stride=None, ignore_border=True):
-        super(MaxPooling1D,self).__init__()
-        self.pool_length = pool_length
-        self.stride = stride
-        if self.stride:
-            self.st = (1, self.stride)
-        else:
-            self.st = None
-
-        self.input = T.tensor3()
-        self.poolsize = (1, pool_length)
-        self.ignore_border = ignore_border
-
-    def get_output(self, train):
-        X = self.get_input(train)
-        X = theano.tensor.reshape(X, (X.shape[0], X.shape[1], X.shape[2], 1)).dimshuffle(0, 1, 3, 2)
-        output = downsample.max_pool_2d(X, ds=self.poolsize, st=self.st, ignore_border=self.ignore_border)
-        output = output.dimshuffle(0, 1, 3, 2)
-        return theano.tensor.reshape(output, (output.shape[0], output.shape[1], output.shape[2]))
-
-    def get_config(self):
-        return {"name":self.__class__.__name__,
-                "stride":self.stride,
-                "pool_length":self.pool_length,
-                "ignore_border":self.ignore_border,
-                "subsample_length": self.subsample_length}
-
+        return {"name": self.__class__.__name__,
+                "input_dim": self.input_dim,
+                "nb_filter": self.nb_filter,
+                "filter_length": self.filter_length,
+                "init": self.init.__name__,
+                "activation": self.activation.__name__,
+                "border_mode": self.border_mode,
+                "subsample_length": self.subsample_length,
+                "W_regularizer": self.W_regularizer.get_config() if self.W_regularizer else None,
+                "b_regularizer": self.b_regularizer.get_config() if self.b_regularizer else None,
+                "activity_regularizer": self.activity_regularizer.get_config() if self.activity_regularizer else None,
+                "W_constraint": self.W_constraint.get_config() if self.W_constraint else None,
+                "b_constraint": self.b_constraint.get_config() if self.b_constraint else None}
 
 
 class Convolution2D(Layer):
     def __init__(self, nb_filter, stack_size, nb_row, nb_col,
-        init='glorot_uniform', activation='linear', weights=None,
-        border_mode='valid', subsample=(1, 1),
-        W_regularizer=None, b_regularizer=None, activity_regularizer=None, W_constraint=None, b_constraint=None):
+                 init='glorot_uniform', activation='linear', weights=None,
+                 border_mode='valid', subsample=(1, 1),
+                 W_regularizer=None, b_regularizer=None, activity_regularizer=None,
+                 W_constraint=None, b_constraint=None):
 
         if border_mode not in {'valid', 'full', 'same'}:
             raise Exception('Invalid border mode for Convolution2D:', border_mode)
 
-        super(Convolution2D,self).__init__()
+        super(Convolution2D, self).__init__()
         self.init = initializations.get(init)
         self.activation = activations.get(activation)
         self.subsample = subsample
@@ -166,73 +149,146 @@ class Convolution2D(Layer):
     def get_output(self, train):
         X = self.get_input(train)
         border_mode = self.border_mode
-        if border_mode == 'same':
-            border_mode = 'full'
+        if dnn.dnn_available() and theano.config.device[:3] == 'gpu':
+            if border_mode == 'same':
+                assert(self.subsample == (1, 1))
+                pad_x = (self.nb_row - self.subsample[0]) // 2
+                pad_y = (self.nb_col - self.subsample[1]) // 2
+                conv_out = dnn.dnn_conv(img=X,
+                                        kerns=self.W,
+                                        border_mode=(pad_x, pad_y))
+            else:
+                conv_out = dnn.dnn_conv(img=X,
+                                        kerns=self.W,
+                                        border_mode=border_mode,
+                                        subsample=self.subsample)
+        else:
+            if border_mode == 'same':
+                border_mode = 'full'
 
-        conv_out = theano.tensor.nnet.conv.conv2d(X, self.W,
-            border_mode=border_mode, subsample=self.subsample)
-
-        if self.border_mode == 'same':
-            shift_x = (self.nb_row - 1) // 2
-            shift_y = (self.nb_col - 1) // 2
-            conv_out = conv_out[:, :, shift_x:X.shape[2] + shift_x, shift_y:X.shape[3] + shift_y]
+            conv_out = T.nnet.conv.conv2d(X, self.W,
+                                          border_mode=border_mode,
+                                          subsample=self.subsample)
+            if self.border_mode == 'same':
+                shift_x = (self.nb_row - 1) // 2
+                shift_y = (self.nb_col - 1) // 2
+                conv_out = conv_out[:, :, shift_x:X.shape[2] + shift_x, shift_y:X.shape[3] + shift_y]
 
         return self.activation(conv_out + self.b.dimshuffle('x', 0, 'x', 'x'))
 
+    def get_config(self):
+        return {"name": self.__class__.__name__,
+                "nb_filter": self.nb_filter,
+                "stack_size": self.stack_size,
+                "nb_row": self.nb_row,
+                "nb_col": self.nb_col,
+                "init": self.init.__name__,
+                "activation": self.activation.__name__,
+                "border_mode": self.border_mode,
+                "subsample": self.subsample,
+                "W_regularizer": self.W_regularizer.get_config() if self.W_regularizer else None,
+                "b_regularizer": self.b_regularizer.get_config() if self.b_regularizer else None,
+                "activity_regularizer": self.activity_regularizer.get_config() if self.activity_regularizer else None,
+                "W_constraint": self.W_constraint.get_config() if self.W_constraint else None,
+                "b_constraint": self.b_constraint.get_config() if self.b_constraint else None}
+
+
+class MaxPooling1D(Layer):
+    def __init__(self, pool_length=2, stride=None, ignore_border=True):
+        super(MaxPooling1D, self).__init__()
+        self.pool_length = pool_length
+        self.stride = stride
+        if self.stride:
+            self.st = (self.stride, 1)
+        else:
+            self.st = None
+
+        self.input = T.tensor3()
+        self.poolsize = (pool_length, 1)
+        self.ignore_border = ignore_border
+
+    def get_output(self, train):
+        X = self.get_input(train)
+        X = T.reshape(X, (X.shape[0], X.shape[1], X.shape[2], 1)).dimshuffle(0, 2, 1, 3)
+        output = T.signal.downsample.max_pool_2d(X, ds=self.poolsize, st=self.st, ignore_border=self.ignore_border)
+        output = output.dimshuffle(0, 2, 1, 3)
+        return T.reshape(output, (output.shape[0], output.shape[1], output.shape[2]))
 
     def get_config(self):
-        return {"name":self.__class__.__name__,
-            "nb_filter":self.nb_filter,
-            "stack_size":self.stack_size,
-            "nb_row":self.nb_row,
-            "nb_col":self.nb_col,
-            "init":self.init.__name__,
-            "activation":self.activation.__name__,
-            "border_mode":self.border_mode,
-            "subsample":self.subsample,
-            "W_regularizer":self.W_regularizer.get_config() if self.W_regularizer else None,
-            "b_regularizer":self.b_regularizer.get_config() if self.b_regularizer else None,
-            "activity_regularizer":self.activity_regularizer.get_config() if self.activity_regularizer else None,
-            "W_constraint":self.W_constraint.get_config() if self.W_constraint else None,
-            "b_constraint":self.b_constraint.get_config() if self.b_constraint else None}
+        return {"name": self.__class__.__name__,
+                "stride": self.stride,
+                "pool_length": self.pool_length,
+                "ignore_border": self.ignore_border}
 
 
 class MaxPooling2D(Layer):
     def __init__(self, poolsize=(2, 2), stride=None, ignore_border=True):
-        super(MaxPooling2D,self).__init__()
+        super(MaxPooling2D, self).__init__()
         self.input = T.tensor4()
         self.poolsize = poolsize
         self.stride = stride
         self.ignore_border = ignore_border
 
-
     def get_output(self, train):
         X = self.get_input(train)
-        output = downsample.max_pool_2d(X, ds=self.poolsize, st=self.stride, ignore_border=self.ignore_border)
+        output = T.signal.downsample.max_pool_2d(X, ds=self.poolsize, st=self.stride, ignore_border=self.ignore_border)
         return output
 
     def get_config(self):
-        return {"name":self.__class__.__name__,
-                "poolsize":self.poolsize,
-                "ignore_border":self.ignore_border,
+        return {"name": self.__class__.__name__,
+                "poolsize": self.poolsize,
+                "ignore_border": self.ignore_border,
                 "stride": self.stride}
 
 
+class UpSample1D(Layer):
+    def __init__(self, length=2):
+        super(UpSample1D, self).__init__()
+        self.length = length
+        self.input = T.tensor3()
+
+    def get_output(self, train):
+        X = self.get_input(train)
+        output = theano.tensor.extra_ops.repeat(X, self.length, axis=1)
+        return output
+
+    def get_config(self):
+        return {"name": self.__class__.__name__,
+                "length": self.length}
+
+
+class UpSample2D(Layer):
+    def __init__(self, size=(2, 2)):
+        super(UpSample2D, self).__init__()
+        self.input = T.tensor4()
+        self.size = size
+
+    def get_output(self, train):
+        X = self.get_input(train)
+        Y = theano.tensor.extra_ops.repeat(X, self.size[0], axis=2)
+        output = theano.tensor.extra_ops.repeat(Y, self.size[1], axis=3)
+        return output
+
+    def get_config(self):
+        return {"name": self.__class__.__name__,
+                "size": self.size}
+
+
 class ZeroPadding2D(Layer):
-    def __init__(self, width=1):
+    def __init__(self, pad=(1, 1)):
         super(ZeroPadding2D, self).__init__()
-        self.width = width
+        self.pad = pad
         self.input = T.tensor4()
 
     def get_output(self, train):
         X = self.get_input(train)
-        width =  self.width
+        pad = self.pad
         in_shape = X.shape
-        out_shape = (in_shape[0], in_shape[1], in_shape[2] + 2 * width, in_shape[3] + 2 * width)
+        out_shape = (in_shape[0], in_shape[1], in_shape[2] + 2 * pad[0], in_shape[3] + 2 * pad[1])
         out = T.zeros(out_shape)
-        indices = (slice(None), slice(None), slice(width, in_shape[2] + width), slice(width, in_shape[3] + width))
+        indices = (slice(None), slice(None), slice(pad[0], in_shape[2] + pad[0]), slice(pad[1], in_shape[3] + pad[1]))
         return T.set_subtensor(out[indices], X)
 
     def get_config(self):
-        return {"name":self.__class__.__name__,
-                "width":self.width}
+        return {"name": self.__class__.__name__,
+                "pad": self.pad}

keras/layers/core.py

@@ -12,17 +12,29 @@ from ..regularizers import ActivityRegularizer, Regularizer
 
 from theano.sandbox.rng_mrg import MRG_RandomStreams as RandomStreams
 from six.moves import zip
-srng = RandomStreams(seed=np.random.randint(10e6))
+
 
 class Layer(object):
     def __init__(self):
         self.params = []
 
-    def set_previous(self, layer):
+    def init_updates(self):
+        self.updates = []
+
+    def set_previous(self, layer, connection_map={}):
+        assert self.nb_input == layer.nb_output == 1, "Cannot connect layers: input count and output count should be 1."
         if not self.supports_masked_input() and layer.get_output_mask() is not None:
-            raise Exception("Attached non-masking layer to layer with masked output")
+            raise Exception("Cannot connect non-masking layer to layer with masked output")
         self.previous = layer
 
+    @property
+    def nb_input(self):
+        return 1
+
+    @property
+    def nb_output(self):
+        return 1
+
     def get_output(self, train=False):
         return self.get_input(train)
 
@@ -67,10 +79,11 @@ class Layer(object):
         return weights
 
     def get_config(self):
-        return {"name":self.__class__.__name__}
+        return {"name": self.__class__.__name__}
 
     def get_params(self):
         consts = []
+        updates = []
 
         if hasattr(self, 'regularizers'):
             regularizers = self.regularizers
@@ -88,7 +101,14 @@ class Layer(object):
         else:
             consts += [constraints.identity() for _ in range(len(self.params))]
 
-        return self.params, regularizers, consts
+        if hasattr(self, 'updates') and self.updates:
+            updates += self.updates
+
+        return self.params, regularizers, consts, updates
+
+    def set_name(self, name):
+        for i in range(len(self.params)):
+            self.params[i].name = '%s_p%d' % (name, i)
 
 
 class MaskedLayer(Layer):
@@ -111,7 +131,36 @@ class MaskedLayer(Layer):
         return self.get_input_mask(train)
 
 
-class Merge(object): 
+class Masking(MaskedLayer):
+    """Mask an input sequence by using a mask value to identify padding.
+
+    This layer copies the input to the output layer with identified padding
+    replaced with 0s and creates an output mask in the process.
+
+    At each timestep, if the values all equal `mask_value`,
+    then the corresponding mask value for the timestep is 0 (skipped),
+    otherwise it is 1.
+
+    """
+    def __init__(self, mask_value=0.):
+        super(Masking, self).__init__()
+        self.mask_value = mask_value
+        self.input = T.tensor3()
+
+    def get_output_mask(self, train=False):
+        X = self.get_input(train)
+        return T.any(T.ones_like(X) * (1. - T.eq(X, self.mask_value)), axis=-1)
+
+    def get_output(self, train=False):
+        X = self.get_input(train)
+        return X * T.shape_padright(T.any((1. - T.eq(X, self.mask_value)), axis=-1))
+
+    def get_config(self):
+        return {"name": self.__class__.__name__,
+                "mask_value": self.mask_value}
+
+
+class Merge(Layer):
     def __init__(self, layers, mode='sum'):
         ''' Merge the output of a list of layers or containers into a single tensor.
             mode: {'sum', 'concat'}
@@ -123,17 +172,19 @@ class Merge(object):
         self.params = []
         self.regularizers = []
         self.constraints = []
+        self.updates = []
         for l in self.layers:
-            params, regs, consts = l.get_params()
+            params, regs, consts, updates = l.get_params()
             self.regularizers += regs
+            self.updates += updates
             # params and constraints have the same size
             for p, c in zip(params, consts):
-                if not p in self.params:
+                if p not in self.params:
                     self.params.append(p)
                     self.constraints.append(c)
 
     def get_params(self):
-        return self.params, self.regularizers, self.constraints
+        return self.params, self.regularizers, self.constraints, self.updates
 
     def get_output(self, train=False):
         if self.mode == 'sum':
@@ -156,7 +207,7 @@ class Merge(object):
             for output in o:
                 if output not in res:
                     res.append(output)
-        return res   
+        return res
 
     @property
     def input(self):
@@ -181,10 +232,10 @@ class Merge(object):
             weights = weights[nb_param:]
 
     def get_config(self):
-        return {"name":self.__class__.__name__,
-            "layers":[l.get_config() for l in self.layers],
-            "mode":self.mode}
-  
+        return {"name": self.__class__.__name__,
+                "layers": [l.get_config() for l in self.layers],
+                "mode": self.mode}
+
 
 class Dropout(MaskedLayer):
     '''
@@ -193,20 +244,21 @@ class Dropout(MaskedLayer):
     def __init__(self, p):
         super(Dropout, self).__init__()
         self.p = p
+        self.srng = RandomStreams(seed=np.random.randint(10e6))
 
     def get_output(self, train=False):
         X = self.get_input(train)
         if self.p > 0.:
             retain_prob = 1. - self.p
             if train:
-                X *= srng.binomial(X.shape, p=retain_prob, dtype=theano.config.floatX)
+                X *= self.srng.binomial(X.shape, p=retain_prob, dtype=theano.config.floatX)
             else:
                 X *= retain_prob
         return X
 
     def get_config(self):
-        return {"name":self.__class__.__name__,
-            "p":self.p}
+        return {"name": self.__class__.__name__,
+                "p": self.p}
 
 
 class Activation(MaskedLayer):
@@ -224,10 +276,10 @@ class Activation(MaskedLayer):
         return self.activation(X)
 
     def get_config(self):
-        return {"name":self.__class__.__name__,
-            "activation":self.activation.__name__,
-            "target":self.target,
-            "beta":self.beta}
+        return {"name": self.__class__.__name__,
+                "activation": self.activation.__name__,
+                "target": self.target,
+                "beta": self.beta}
 
 
 class Reshape(Layer):
@@ -246,8 +298,25 @@ class Reshape(Layer):
         return theano.tensor.reshape(X, nshape)
 
     def get_config(self):
-        return {"name":self.__class__.__name__,
-            "dims":self.dims}
+        return {"name": self.__class__.__name__,
+                "dims": self.dims}
+
+
+class Permute(Layer):
+    '''
+        Permute the dimensions of the data according to the given tuple
+    '''
+    def __init__(self, dims):
+        super(Permute, self).__init__()
+        self.dims = dims
+
+    def get_output(self, train):
+        X = self.get_input(train)
+        return X.dimshuffle((0,) + self.dims)
+
+    def get_config(self):
+        return {"name": self.__class__.__name__,
+                "dims": self.dims}
 
 
 class Flatten(Layer):
@@ -283,8 +352,8 @@ class RepeatVector(Layer):
         return stacked.dimshuffle((1, 0, 2))
 
     def get_config(self):
-        return {"name":self.__class__.__name__,
-            "n":self.n}
+        return {"name": self.__class__.__name__,
+                "n": self.n}
 
 
 class Dense(Layer):
@@ -292,7 +361,8 @@ class Dense(Layer):
         Just your regular fully connected NN layer.
     '''
     def __init__(self, input_dim, output_dim, init='glorot_uniform', activation='linear', weights=None, name=None,
-        W_regularizer=None, b_regularizer=None, activity_regularizer=None, W_constraint=None, b_constraint=None):
+                 W_regularizer=None, b_regularizer=None, activity_regularizer=None,
+                 W_constraint=None, b_constraint=None):
 
         super(Dense, self).__init__()
         self.init = initializations.get(init)
@@ -307,7 +377,6 @@ class Dense(Layer):
         self.params = [self.W, self.b]
 
         self.regularizers = []
-        
         self.W_regularizer = regularizers.get(W_regularizer)
         if self.W_regularizer:
             self.W_regularizer.set_param(self.W)
@@ -343,16 +412,16 @@ class Dense(Layer):
         return output
 
     def get_config(self):
-        return {"name":self.__class__.__name__,
-            "input_dim":self.input_dim,
-            "output_dim":self.output_dim,
-            "init":self.init.__name__,
-            "activation":self.activation.__name__,
-            "W_regularizer":self.W_regularizer.get_config() if self.W_regularizer else None,
-            "b_regularizer":self.b_regularizer.get_config() if self.b_regularizer else None,
-            "activity_regularizer":self.activity_regularizer.get_config() if self.activity_regularizer else None,
-            "W_constraint":self.W_constraint.get_config() if self.W_constraint else None,
-            "b_constraint":self.b_constraint.get_config() if self.b_constraint else None}
+        return {"name": self.__class__.__name__,
+                "input_dim": self.input_dim,
+                "output_dim": self.output_dim,
+                "init": self.init.__name__,
+                "activation": self.activation.__name__,
+                "W_regularizer": self.W_regularizer.get_config() if self.W_regularizer else None,
+                "b_regularizer": self.b_regularizer.get_config() if self.b_regularizer else None,
+                "activity_regularizer": self.activity_regularizer.get_config() if self.activity_regularizer else None,
+                "W_constraint": self.W_constraint.get_config() if self.W_constraint else None,
+                "b_constraint": self.b_constraint.get_config() if self.b_constraint else None}
 
 
 class ActivityRegularization(Layer):
@@ -373,9 +442,9 @@ class ActivityRegularization(Layer):
         return self.get_input(train)
 
     def get_config(self):
-        return {"name":self.__class__.__name__,
-            "l1":self.l1,
-            "l2":self.l2}
+        return {"name": self.__class__.__name__,
+                "l1": self.l1,
+                "l2": self.l2}
 
 
 class TimeDistributedDense(MaskedLayer):
@@ -386,8 +455,9 @@ class TimeDistributedDense(MaskedLayer):
        Tensor output dimensions:  (nb_sample, shared_dimension, output_dim)
 
     '''
-    def __init__(self, input_dim, output_dim, init='glorot_uniform', activation='linear', weights=None, 
-        W_regularizer=None, b_regularizer=None, activity_regularizer=None, W_constraint=None, b_constraint=None):
+    def __init__(self, input_dim, output_dim, init='glorot_uniform', activation='linear', weights=None,
+                 W_regularizer=None, b_regularizer=None, activity_regularizer=None,
+                 W_constraint=None, b_constraint=None):
 
         super(TimeDistributedDense, self).__init__()
         self.init = initializations.get(init)
@@ -430,18 +500,17 @@ class TimeDistributedDense(MaskedLayer):
         output = self.activation(T.dot(X.dimshuffle(1, 0, 2), self.W) + self.b)
         return output.dimshuffle(1, 0, 2)
 
-
     def get_config(self):
-        return {"name":self.__class__.__name__,
-            "input_dim":self.input_dim,
-            "output_dim":self.output_dim,
-            "init":self.init.__name__,
-            "activation":self.activation.__name__,            
-            "W_regularizer":self.W_regularizer.get_config() if self.W_regularizer else None,
-            "b_regularizer":self.b_regularizer.get_config() if self.b_regularizer else None,
-            "activity_regularizer":self.activity_regularizer.get_config() if self.activity_regularizer else None,
-            "W_constraint":self.W_constraint.get_config() if self.W_constraint else None,
-            "b_constraint":self.b_constraint.get_config() if self.b_constraint else None}
+        return {"name": self.__class__.__name__,
+                "input_dim": self.input_dim,
+                "output_dim": self.output_dim,
+                "init": self.init.__name__,
+                "activation": self.activation.__name__,
+                "W_regularizer": self.W_regularizer.get_config() if self.W_regularizer else None,
+                "b_regularizer": self.b_regularizer.get_config() if self.b_regularizer else None,
+                "activity_regularizer": self.activity_regularizer.get_config() if self.activity_regularizer else None,
+                "W_constraint": self.W_constraint.get_config() if self.W_constraint else None,
+                "b_constraint": self.b_constraint.get_config() if self.b_constraint else None}
 
 
 class AutoEncoder(Layer):
@@ -463,13 +532,15 @@ class AutoEncoder(Layer):
         self.params = []
         self.regularizers = []
         self.constraints = []
+        self.updates = []
         for layer in [self.encoder, self.decoder]:
-            params, regularizers, constraints = layer.get_params()
-            self.constraints += constraints
-            for p, r in zip(params, regularizers):
+            params, regularizers, constraints, updates = layer.get_params()
+            self.regularizers += regularizers
+            self.updates += updates
+            for p, c in zip(params, constraints):
                 if p not in self.params:
                     self.params.append(p)
-                    self.regularizers.append(r)
+                    self.constraints.append(c)
 
         if weights is not None:
             self.set_weights(weights)
@@ -505,11 +576,10 @@ class AutoEncoder(Layer):
         return self.decoder.get_output(train)
 
     def get_config(self):
-        return {"name":self.__class__.__name__,
-                "encoder_config":self.encoder.get_config(),
-                "decoder_config":self.decoder.get_config(),
-                "output_reconstruction":self.output_reconstruction}
-
+        return {"name": self.__class__.__name__,
+                "encoder_config": self.encoder.get_config(),
+                "decoder_config": self.decoder.get_config(),
+                "output_reconstruction": self.output_reconstruction}
 
 
 class MaxoutDense(Layer):
@@ -517,8 +587,9 @@ class MaxoutDense(Layer):
         Max-out layer, nb_feature is the number of pieces in the piecewise linear approx.
         Refer to http://arxiv.org/pdf/1302.4389.pdf
     '''
-    def __init__(self, input_dim, output_dim, nb_feature=4, init='glorot_uniform', weights=None, 
-        W_regularizer=None, b_regularizer=None, activity_regularizer=None, W_constraint=None, b_constraint=None):
+    def __init__(self, input_dim, output_dim, nb_feature=4, init='glorot_uniform', weights=None,
+                 W_regularizer=None, b_regularizer=None, activity_regularizer=None,
+                 W_constraint=None, b_constraint=None):
 
         super(MaxoutDense, self).__init__()
         self.init = initializations.get(init)
@@ -563,13 +634,13 @@ class MaxoutDense(Layer):
         return output
 
     def get_config(self):
-        return {"name":self.__class__.__name__,
-            "input_dim":self.input_dim,
-            "output_dim":self.output_dim,
-            "init":self.init.__name__,
-            "nb_feature" : self.nb_feature,
-            "W_regularizer":self.W_regularizer.get_config() if self.W_regularizer else None,
-            "b_regularizer":self.b_regularizer.get_config() if self.b_regularizer else None,
-            "activity_regularizer":self.activity_regularizer.get_config() if self.activity_regularizer else None,
-            "W_constraint":self.W_constraint.get_config() if self.W_constraint else None,
-            "b_constraint":self.b_constraint.get_config() if self.b_constraint else None}
+        return {"name": self.__class__.__name__,
+                "input_dim": self.input_dim,
+                "output_dim": self.output_dim,
+                "init": self.init.__name__,
+                "nb_feature": self.nb_feature,
+                "W_regularizer": self.W_regularizer.get_config() if self.W_regularizer else None,
+                "b_regularizer": self.b_regularizer.get_config() if self.b_regularizer else None,
+                "activity_regularizer": self.activity_regularizer.get_config() if self.activity_regularizer else None,
+                "W_constraint": self.W_constraint.get_config() if self.W_constraint else None,
+                "b_constraint": self.b_constraint.get_config() if self.b_constraint else None}

keras/layers/embeddings.py

@@ -11,17 +11,17 @@ from ..constraints import unitnorm
 
 class Embedding(Layer):
     '''
-        Turn positive integers (indexes) into denses vectors of fixed size. 
+        Turn positive integers (indexes) into denses vectors of fixed size.
         eg. [[4], [20]] -> [[0.25, 0.1], [0.6, -0.2]]
 
         @input_dim: size of vocabulary (highest input integer + 1)
         @out_dim: size of dense representation
     '''
     def __init__(self, input_dim, output_dim, init='uniform',
-        W_regularizer=None, activity_regularizer=None, W_constraint=None,
-        mask_zero=False, weights=None):
+                 W_regularizer=None, activity_regularizer=None, W_constraint=None,
+                 mask_zero=False, weights=None):
 
-        super(Embedding,self).__init__()
+        super(Embedding, self).__init__()
         self.init = initializations.get(init)
         self.input_dim = input_dim
         self.output_dim = output_dim
@@ -55,7 +55,7 @@ class Embedding(Layer):
         if not self.mask_zero:
             return None
         else:
-            return T.ones_like(X) * (1 - T.eq(X,0))
+            return T.ones_like(X) * (1 - T.eq(X, 0))
 
     def get_output(self, train=False):
         X = self.get_input(train)
@@ -63,24 +63,24 @@ class Embedding(Layer):
         return out
 
     def get_config(self):
-        return {"name":self.__class__.__name__,
-            "input_dim":self.input_dim,
-            "output_dim":self.output_dim,
-            "init":self.init.__name__,
-            "activity_regularizer":self.activity_regularizer.get_config() if self.activity_regularizer else None,
-            "W_regularizer":self.W_regularizer.get_config() if self.W_regularizer else None,
-            "W_constraint":self.W_constraint.get_config() if self.W_constraint else None}
+        return {"name": self.__class__.__name__,
+                "input_dim": self.input_dim,
+                "output_dim": self.output_dim,
+                "init": self.init.__name__,
+                "activity_regularizer": self.activity_regularizer.get_config() if self.activity_regularizer else None,
+                "W_regularizer": self.W_regularizer.get_config() if self.W_regularizer else None,
+                "W_constraint": self.W_constraint.get_config() if self.W_constraint else None}
 
 
 class WordContextProduct(Layer):
     '''
-        This layer turns a pair of words (a pivot word + a context word, 
+        This layer turns a pair of words (a pivot word + a context word,
         ie. a word from the same context, or a random, out-of-context word),
         indentified by their index in a vocabulary, into two dense reprensentations
         (word representation and context representation).
 
         Then it returns activation(dot(pivot_embedding, context_embedding)),
-        which can be trained to encode the probability 
+        which can be trained to encode the probability
         of finding the context word in the context of the pivot word
         (or reciprocally depending on your training procedure).
 
@@ -96,9 +96,10 @@ class WordContextProduct(Layer):
             Efficient Estimation of Word reprensentations in Vector Space
             http://arxiv.org/pdf/1301.3781v3.pdf
     '''
-    def __init__(self, input_dim, proj_dim=128, 
-        init='uniform', activation='sigmoid', weights=None):
-        super(WordContextProduct,self).__init__()
+    def __init__(self, input_dim, proj_dim=128,
+                 init='uniform', activation='sigmoid', weights=None):
+
+        super(WordContextProduct, self).__init__()
         self.input_dim = input_dim
         self.proj_dim = proj_dim
         self.init = initializations.get(init)
@@ -115,20 +116,18 @@ class WordContextProduct(Layer):
         if weights is not None:
             self.set_weights(weights)
 
-
     def get_output(self, train=False):
         X = self.get_input(train)
-        w = self.W_w[X[:, 0]] # nb_samples, proj_dim
-        c = self.W_c[X[:, 1]] # nb_samples, proj_dim
+        w = self.W_w[X[:, 0]]  # nb_samples, proj_dim
+        c = self.W_c[X[:, 1]]  # nb_samples, proj_dim
 
         dot = T.sum(w * c, axis=1)
         dot = theano.tensor.reshape(dot, (X.shape[0], 1))
         return self.activation(dot)
 
     def get_config(self):
-        return {"name":self.__class__.__name__,
-            "input_dim":self.input_dim,
-            "proj_dim":self.proj_dim,
-            "init":self.init.__name__,
-            "activation":self.activation.__name__}
-
+        return {"name": self.__class__.__name__,
+                "input_dim": self.input_dim,
+                "proj_dim": self.proj_dim,
+                "init": self.init.__name__,
+                "activation": self.activation.__name__}

keras/layers/noise.py

@@ -1,7 +1,10 @@
 from __future__ import absolute_import
-from .core import srng, MaskedLayer
+import numpy as np
+from .core import MaskedLayer
 import theano
 import theano.tensor as T
+from theano.sandbox.rng_mrg import MRG_RandomStreams as RandomStreams
+
 
 class GaussianNoise(MaskedLayer):
     '''
@@ -10,38 +13,41 @@ class GaussianNoise(MaskedLayer):
     def __init__(self, sigma):
         super(GaussianNoise, self).__init__()
         self.sigma = sigma
+        self.srng = RandomStreams(seed=np.random.randint(10e6))
 
     def get_output(self, train=False):
         X = self.get_input(train)
         if not train or self.sigma == 0:
             return X
         else:
-            return X + srng.normal(size=X.shape, avg=0.0, std=self.sigma,
-                             dtype=theano.config.floatX)
+            return X + self.srng.normal(size=X.shape, avg=0.0, std=self.sigma,
+                                        dtype=theano.config.floatX)
 
     def get_config(self):
-        return {"name":self.__class__.__name__,
-            "sigma":self.sigma}
+        return {"name": self.__class__.__name__,
+                "sigma": self.sigma}
+
 
 class GaussianDropout(MaskedLayer):
     '''
         Multiplicative Gaussian Noise
-        Reference: 
+        Reference:
             Dropout: A Simple Way to Prevent Neural Networks from Overfitting
             Srivastava, Hinton, et al. 2014
             http://www.cs.toronto.edu/~rsalakhu/papers/srivastava14a.pdf
     '''
     def __init__(self, p):
-        super(GaussianDropout,self).__init__()
+        super(GaussianDropout, self).__init__()
         self.p = p
+        self.srng = RandomStreams(seed=np.random.randint(10e6))
 
     def get_output(self, train):
         X = self.get_input(train)
         if train:
             # self.p refers to drop probability rather than retain probability (as in paper) to match Dropout layer syntax
-            X *= srng.normal(size=X.shape, avg=1.0, std=T.sqrt(self.p / (1.0 - self.p)), dtype=theano.config.floatX)
+            X *= self.srng.normal(size=X.shape, avg=1.0, std=T.sqrt(self.p / (1.0 - self.p)), dtype=theano.config.floatX)
         return X
 
     def get_config(self):
-        return {"name":self.__class__.__name__,
-            "p":self.p}
+        return {"name": self.__class__.__name__,
+                "p": self.p}

keras/layers/normalization.py

@@ -1,9 +1,10 @@
 from ..layers.core import Layer
-from ..utils.theano_utils import shared_zeros
+from ..utils.theano_utils import shared_zeros, shared_ones, ndim_tensor
 from .. import initializations
 
 import theano.tensor as T
 
+
 class BatchNormalization(Layer):
     '''
         Reference:
@@ -16,43 +17,36 @@ class BatchNormalization(Layer):
             momentum: momentum term in the computation of a running estimate of the mean and std of the data
     '''
     def __init__(self, input_shape, epsilon=1e-6, mode=0, momentum=0.9, weights=None):
-        super(BatchNormalization,self).__init__()
+        super(BatchNormalization, self).__init__()
         self.init = initializations.get("uniform")
         self.input_shape = input_shape
         self.epsilon = epsilon
         self.mode = mode
         self.momentum = momentum
+        self.input = ndim_tensor(len(self.input_shape) + 1)
 
         self.gamma = self.init((self.input_shape))
         self.beta = shared_zeros(self.input_shape)
 
-        self.running_mean = None
-        self.running_std = None
-
         self.params = [self.gamma, self.beta]
         if weights is not None:
             self.set_weights(weights)
 
+    def init_updates(self):
+        self.running_mean = shared_zeros(self.input_shape)
+        self.running_std = shared_ones((self.input_shape))
+        X = self.get_input(train=True)
+        m = X.mean(axis=0)
+        std = T.mean((X - m) ** 2 + self.epsilon, axis=0) ** 0.5
+        mean_update = self.momentum * self.running_mean + (1-self.momentum) * m
+        std_update = self.momentum * self.running_std + (1-self.momentum) * std
+        self.updates = [(self.running_mean, mean_update), (self.running_std, std_update)]
+
     def get_output(self, train):
         X = self.get_input(train)
 
         if self.mode == 0:
-            if train:
-                m = X.mean(axis=0)
-                # manual computation of std to prevent NaNs
-                std = T.mean((X-m)**2 + self.epsilon, axis=0) ** 0.5
-                X_normed = (X - m) / (std + self.epsilon)
-
-                if self.running_mean is None:
-                    self.running_mean = m
-                    self.running_std = std
-                else:
-                    self.running_mean *= self.momentum
-                    self.running_mean += (1-self.momentum) * m
-                    self.running_std *= self.momentum
-                    self.running_std += (1-self.momentum) * std
-            else:
-                X_normed = (X - self.running_mean) / (self.running_std + self.epsilon)
+            X_normed = (X - self.running_mean) / (self.running_std + self.epsilon)
 
         elif self.mode == 1:
             m = X.mean(axis=-1, keepdims=True)
@@ -63,10 +57,10 @@ class BatchNormalization(Layer):
         return out
 
     def get_config(self):
-        return {"name":self.__class__.__name__,
-            "input_shape":self.input_shape,
-            "epsilon":self.epsilon,
-            "mode":self.mode}
+        return {"name": self.__class__.__name__,
+                "input_shape": self.input_shape,
+                "epsilon": self.epsilon,
+                "mode": self.mode}
 
 
 class LRN2D(Layer):
@@ -98,8 +92,8 @@ class LRN2D(Layer):
         return X / scale
 
     def get_config(self):
-        return {"name":self.__class__.__name__,
-            "alpha":self.alpha,
-            "k":self.k,
-            "beta":self.beta,
-            "n": self.n}
+        return {"name": self.__class__.__name__,
+                "alpha": self.alpha,
+                "k": self.k,
+                "beta": self.beta,
+                "n": self.n}

keras/layers/recurrent.py

@@ -9,6 +9,7 @@ from ..utils.theano_utils import shared_scalar, shared_zeros, alloc_zeros_matrix
 from ..layers.core import Layer, MaskedLayer
 from six.moves import range
 
+
 class Recurrent(MaskedLayer):
     def get_output_mask(self, train=None):
         if self.return_sequences:
@@ -19,12 +20,12 @@ class Recurrent(MaskedLayer):
     def get_padded_shuffled_mask(self, train, X, pad=0):
         mask = self.get_input_mask(train)
         if mask is None:
-            mask = T.ones_like(X.sum(axis=-1)) # is there a better way to do this without a sum?
+            mask = T.ones_like(X.sum(axis=-1))  # is there a better way to do this without a sum?
 
         # mask is (nb_samples, time)
-        mask = T.shape_padright(mask) # (nb_samples, time, 1)
-        mask = T.addbroadcast(mask, -1) # (time, nb_samples, 1) matrix.
-        mask = mask.dimshuffle(1, 0, 2) # (time, nb_samples, 1)
+        mask = T.shape_padright(mask)  # (nb_samples, time, 1)
+        mask = T.addbroadcast(mask, -1)  # (time, nb_samples, 1) matrix.
+        mask = mask.dimshuffle(1, 0, 2)  # (time, nb_samples, 1)
 
         if pad > 0:
             # left-pad in time with 0
@@ -37,15 +38,15 @@ class SimpleRNN(Recurrent):
     '''
         Fully connected RNN where output is to fed back to input.
 
-        Not a particularly useful model, 
-        included for demonstration purposes 
+        Not a particularly useful model,
+        included for demonstration purposes
         (demonstrates how to use theano.scan to build a basic RNN).
     '''
-    def __init__(self, input_dim, output_dim, 
-        init='glorot_uniform', inner_init='orthogonal', activation='sigmoid', weights=None,
-        truncate_gradient=-1, return_sequences=False):
+    def __init__(self, input_dim, output_dim,
+                 init='glorot_uniform', inner_init='orthogonal', activation='sigmoid', weights=None,
+                 truncate_gradient=-1, return_sequences=False):
 
-        super(SimpleRNN,self).__init__()
+        super(SimpleRNN, self).__init__()
         self.init = initializations.get(init)
         self.inner_init = initializations.get(inner_init)
         self.input_dim = input_dim
@@ -65,62 +66,61 @@ class SimpleRNN(Recurrent):
 
     def _step(self, x_t, mask_tm1, h_tm1, u):
         '''
-            Variable names follow the conventions from: 
+            Variable names follow the conventions from:
             http://deeplearning.net/software/theano/library/scan.html
 
         '''
         return self.activation(x_t + mask_tm1 * T.dot(h_tm1, u))
 
     def get_output(self, train=False):
-        X = self.get_input(train) # shape: (nb_samples, time (padded with zeros), input_dim)
+        X = self.get_input(train)  # shape: (nb_samples, time (padded with zeros), input_dim)
         # new shape: (time, nb_samples, input_dim) -> because theano.scan iterates over main dimension
         padded_mask = self.get_padded_shuffled_mask(train, X, pad=1)
-        X = X.dimshuffle((1, 0, 2)) 
+        X = X.dimshuffle((1, 0, 2))
         x = T.dot(X, self.W) + self.b
-        
+
         # scan = theano symbolic loop.
         # See: http://deeplearning.net/software/theano/library/scan.html
         # Iterate over the first dimension of the x array (=time).
         outputs, updates = theano.scan(
-            self._step, # this will be called with arguments (sequences[i], outputs[i-1], non_sequences[i])
-            sequences=[x, dict(input=padded_mask, taps=[-1])], # tensors to iterate over, inputs to _step
+            self._step,  # this will be called with arguments (sequences[i], outputs[i-1], non_sequences[i])
+            sequences=[x, dict(input=padded_mask, taps=[-1])],  # tensors to iterate over, inputs to _step
             # initialization of the output. Input to _step with default tap=-1.
             outputs_info=T.unbroadcast(alloc_zeros_matrix(X.shape[1], self.output_dim), 1),
-            non_sequences=self.U, # static inputs to _step
-            truncate_gradient=self.truncate_gradient
-        )
+            non_sequences=self.U,  # static inputs to _step
+            truncate_gradient=self.truncate_gradient)
 
         if self.return_sequences:
             return outputs.dimshuffle((1, 0, 2))
         return outputs[-1]
 
     def get_config(self):
-        return {"name":self.__class__.__name__,
-            "input_dim":self.input_dim,
-            "output_dim":self.output_dim,
-            "init":self.init.__name__,
-            "inner_init":self.inner_init.__name__,
-            "activation":self.activation.__name__,
-            "truncate_gradient":self.truncate_gradient,
-            "return_sequences":self.return_sequences}
+        return {"name": self.__class__.__name__,
+                "input_dim": self.input_dim,
+                "output_dim": self.output_dim,
+                "init": self.init.__name__,
+                "inner_init": self.inner_init.__name__,
+                "activation": self.activation.__name__,
+                "truncate_gradient": self.truncate_gradient,
+                "return_sequences": self.return_sequences}
 
 
 class SimpleDeepRNN(Recurrent):
     '''
-        Fully connected RNN where the output of multiple timesteps 
+        Fully connected RNN where the output of multiple timesteps
         (up to "depth" steps in the past) is fed back to the input:
 
         output = activation( W.x_t + b + inner_activation(U_1.h_tm1) + inner_activation(U_2.h_tm2) + ... )
 
-        This demonstrates how to build RNNs with arbitrary lookback. 
+        This demonstrates how to build RNNs with arbitrary lookback.
         Also (probably) not a super useful model.
     '''
     def __init__(self, input_dim, output_dim, depth=3,
-        init='glorot_uniform', inner_init='orthogonal', 
-        activation='sigmoid', inner_activation='hard_sigmoid',
-        weights=None, truncate_gradient=-1, return_sequences=False):
+                 init='glorot_uniform', inner_init='orthogonal',
+                 activation='sigmoid', inner_activation='hard_sigmoid',
+                 weights=None, truncate_gradient=-1, return_sequences=False):
 
-        super(SimpleDeepRNN,self).__init__()
+        super(SimpleDeepRNN, self).__init__()
         self.init = initializations.get(init)
         self.inner_init = initializations.get(inner_init)
         self.input_dim = input_dim
@@ -152,10 +152,10 @@ class SimpleDeepRNN(Recurrent):
     def get_output(self, train=False):
         X = self.get_input(train)
         padded_mask = self.get_padded_shuffled_mask(train, X, pad=self.depth)
-        X = X.dimshuffle((1, 0, 2)) 
+        X = X.dimshuffle((1, 0, 2))
 
         x = T.dot(X, self.W) + self.b
-        
+
         if self.depth == 1:
             initial = T.unbroadcast(alloc_zeros_matrix(X.shape[1], self.output_dim), 1)
         else:
@@ -164,12 +164,12 @@ class SimpleDeepRNN(Recurrent):
         outputs, updates = theano.scan(
             self._step,
             sequences=[x, dict(
-                input = padded_mask,
-                taps = [(-i) for i in range(self.depth)]
+                input=padded_mask,
+                taps=[(-i) for i in range(self.depth)]
             )],
             outputs_info=[dict(
-                initial = initial,
-                taps = [(-i-1) for i in range(self.depth)]
+                initial=initial,
+                taps=[(-i-1) for i in range(self.depth)]
             )],
             non_sequences=self.Us,
             truncate_gradient=self.truncate_gradient
@@ -180,16 +180,16 @@ class SimpleDeepRNN(Recurrent):
         return outputs[-1]
 
     def get_config(self):
-        return {"name":self.__class__.__name__,
-            "input_dim":self.input_dim,
-            "output_dim":self.output_dim,
-            "depth":self.depth,
-            "init":self.init.__name__,
-            "inner_init":self.inner_init.__name__,
-            "activation":self.activation.__name__,
-            "inner_activation":self.inner_activation.__name__,
-            "truncate_gradient":self.truncate_gradient,
-            "return_sequences":self.return_sequences}
+        return {"name": self.__class__.__name__,
+                "input_dim": self.input_dim,
+                "output_dim": self.output_dim,
+                "depth": self.depth,
+                "init": self.init.__name__,
+                "inner_init": self.inner_init.__name__,
+                "activation": self.activation.__name__,
+                "inner_activation": self.inner_activation.__name__,
+                "truncate_gradient": self.truncate_gradient,
+                "return_sequences": self.return_sequences}
 
 
 class GRU(Recurrent):
@@ -214,12 +214,12 @@ class GRU(Recurrent):
             Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
                 http://arxiv.org/pdf/1412.3555v1.pdf
     '''
-    def __init__(self, input_dim, output_dim=128, 
-        init='glorot_uniform', inner_init='orthogonal',
-        activation='sigmoid', inner_activation='hard_sigmoid',
-        weights=None, truncate_gradient=-1, return_sequences=False):
+    def __init__(self, input_dim, output_dim=128,
+                 init='glorot_uniform', inner_init='orthogonal',
+                 activation='sigmoid', inner_activation='hard_sigmoid',
+                 weights=None, truncate_gradient=-1, return_sequences=False):
 
-        super(GRU,self).__init__()
+        super(GRU, self).__init__()
         self.input_dim = input_dim
         self.output_dim = output_dim
         self.truncate_gradient = truncate_gradient
@@ -239,7 +239,7 @@ class GRU(Recurrent):
         self.U_r = self.inner_init((self.output_dim, self.output_dim))
         self.b_r = shared_zeros((self.output_dim))
 
-        self.W_h = self.init((self.input_dim, self.output_dim)) 
+        self.W_h = self.init((self.input_dim, self.output_dim))
         self.U_h = self.inner_init((self.output_dim, self.output_dim))
         self.b_h = shared_zeros((self.output_dim))
 
@@ -252,10 +252,10 @@ class GRU(Recurrent):
         if weights is not None:
             self.set_weights(weights)
 
-    def _step(self, 
-        xz_t, xr_t, xh_t, mask_tm1,
-        h_tm1, 
-        u_z, u_r, u_h):
+    def _step(self,
+              xz_t, xr_t, xh_t, mask_tm1,
+              h_tm1,
+              u_z, u_r, u_h):
         h_mask_tm1 = mask_tm1 * h_tm1
         z = self.inner_activation(xz_t + T.dot(h_mask_tm1, u_z))
         r = self.inner_activation(xr_t + T.dot(h_mask_tm1, u_r))
@@ -264,36 +264,34 @@ class GRU(Recurrent):
         return h_t
 
     def get_output(self, train=False):
-        X = self.get_input(train) 
+        X = self.get_input(train)
         padded_mask = self.get_padded_shuffled_mask(train, X, pad=1)
-        X = X.dimshuffle((1, 0, 2)) 
+        X = X.dimshuffle((1, 0, 2))
 
         x_z = T.dot(X, self.W_z) + self.b_z
         x_r = T.dot(X, self.W_r) + self.b_r
         x_h = T.dot(X, self.W_h) + self.b_h
         outputs, updates = theano.scan(
-            self._step, 
-            sequences=[x_z, x_r, x_h, padded_mask], 
+            self._step,
+            sequences=[x_z, x_r, x_h, padded_mask],
             outputs_info=T.unbroadcast(alloc_zeros_matrix(X.shape[1], self.output_dim), 1),
             non_sequences=[self.U_z, self.U_r, self.U_h],
-            truncate_gradient=self.truncate_gradient
-        )
+            truncate_gradient=self.truncate_gradient)
 
         if self.return_sequences:
             return outputs.dimshuffle((1, 0, 2))
         return outputs[-1]
 
     def get_config(self):
-        return {"name":self.__class__.__name__,
-            "input_dim":self.input_dim,
-            "output_dim":self.output_dim,
-            "init":self.init.__name__,
-            "inner_init":self.inner_init.__name__,
-            "activation":self.activation.__name__,
-            "inner_activation":self.inner_activation.__name__,
-            "truncate_gradient":self.truncate_gradient,
-            "return_sequences":self.return_sequences}
-
+        return {"name": self.__class__.__name__,
+                "input_dim": self.input_dim,
+                "output_dim": self.output_dim,
+                "init": self.init.__name__,
+                "inner_init": self.inner_init.__name__,
+                "activation": self.activation.__name__,
+                "inner_activation": self.inner_activation.__name__,
+                "truncate_gradient": self.truncate_gradient,
+                "return_sequences": self.return_sequences}
 
 
 class LSTM(Recurrent):
@@ -321,12 +319,12 @@ class LSTM(Recurrent):
             Supervised sequence labelling with recurrent neural networks
                 http://www.cs.toronto.edu/~graves/preprint.pdf
     '''
-    def __init__(self, input_dim, output_dim=128, 
-        init='glorot_uniform', inner_init='orthogonal', forget_bias_init='one',
-        activation='tanh', inner_activation='hard_sigmoid',
-        weights=None, truncate_gradient=-1, return_sequences=False):
-    
-        super(LSTM,self).__init__()
+    def __init__(self, input_dim, output_dim=128,
+                 init='glorot_uniform', inner_init='orthogonal', forget_bias_init='one',
+                 activation='tanh', inner_activation='hard_sigmoid',
+                 weights=None, truncate_gradient=-1, return_sequences=False):
+
+        super(LSTM, self).__init__()
         self.input_dim = input_dim
         self.output_dim = output_dim
         self.truncate_gradient = truncate_gradient
@@ -365,10 +363,10 @@ class LSTM(Recurrent):
         if weights is not None:
             self.set_weights(weights)
 
-    def _step(self, 
-        xi_t, xf_t, xo_t, xc_t, mask_tm1,
-        h_tm1, c_tm1, 
-        u_i, u_f, u_o, u_c): 
+    def _step(self,
+              xi_t, xf_t, xo_t, xc_t, mask_tm1,
+              h_tm1, c_tm1,
+              u_i, u_f, u_o, u_c):
         h_mask_tm1 = mask_tm1 * h_tm1
         c_mask_tm1 = mask_tm1 * c_tm1
 
@@ -380,7 +378,7 @@ class LSTM(Recurrent):
         return h_t, c_t
 
     def get_output(self, train=False):
-        X = self.get_input(train) 
+        X = self.get_input(train)
         padded_mask = self.get_padded_shuffled_mask(train, X, pad=1)
         X = X.dimshuffle((1, 0, 2))
 
@@ -388,62 +386,60 @@ class LSTM(Recurrent):
         xf = T.dot(X, self.W_f) + self.b_f
         xc = T.dot(X, self.W_c) + self.b_c
         xo = T.dot(X, self.W_o) + self.b_o
-        
+
         [outputs, memories], updates = theano.scan(
-            self._step, 
+            self._step,
             sequences=[xi, xf, xo, xc, padded_mask],
             outputs_info=[
                 T.unbroadcast(alloc_zeros_matrix(X.shape[1], self.output_dim), 1),
                 T.unbroadcast(alloc_zeros_matrix(X.shape[1], self.output_dim), 1)
-            ], 
-            non_sequences=[self.U_i, self.U_f, self.U_o, self.U_c], 
-            truncate_gradient=self.truncate_gradient 
-        )
+            ],
+            non_sequences=[self.U_i, self.U_f, self.U_o, self.U_c],
+            truncate_gradient=self.truncate_gradient)
 
         if self.return_sequences:
             return outputs.dimshuffle((1, 0, 2))
         return outputs[-1]
 
     def get_config(self):
-        return {"name":self.__class__.__name__,
-            "input_dim":self.input_dim,
-            "output_dim":self.output_dim,
-            "init":self.init.__name__,
-            "inner_init":self.inner_init.__name__,
-            "forget_bias_init":self.forget_bias_init.__name__,
-            "activation":self.activation.__name__,
-            "inner_activation":self.inner_activation.__name__,
-            "truncate_gradient":self.truncate_gradient,
-            "return_sequences":self.return_sequences}
-
+        return {"name": self.__class__.__name__,
+                "input_dim": self.input_dim,
+                "output_dim": self.output_dim,
+                "init": self.init.__name__,
+                "inner_init": self.inner_init.__name__,
+                "forget_bias_init": self.forget_bias_init.__name__,
+                "activation": self.activation.__name__,
+                "inner_activation": self.inner_activation.__name__,
+                "truncate_gradient": self.truncate_gradient,
+                "return_sequences": self.return_sequences}
 
 
 class JZS1(Recurrent):
     '''
         Evolved recurrent neural network architectures from the evaluation of thousands
         of models, serving as alternatives to LSTMs and GRUs. See Jozefowicz et al. 2015.
-        
+
         This corresponds to the `MUT1` architecture described in the paper.
-        
+
         Takes inputs with shape:
         (nb_samples, max_sample_length (samples shorter than this are padded with zeros at the end), input_dim)
-        
+
         and returns outputs with shape:
         if not return_sequences:
             (nb_samples, output_dim)
         if return_sequences:
             (nb_samples, max_sample_length, output_dim)
-        
+
         References:
             An Empirical Exploration of Recurrent Network Architectures
                 http://www.jmlr.org/proceedings/papers/v37/jozefowicz15.pdf
     '''
-    def __init__(self, input_dim, output_dim=128, 
-        init='glorot_uniform', inner_init='orthogonal',
-        activation='tanh', inner_activation='sigmoid',
-        weights=None, truncate_gradient=-1, return_sequences=False):
+    def __init__(self, input_dim, output_dim=128,
+                 init='glorot_uniform', inner_init='orthogonal',
+                 activation='tanh', inner_activation='sigmoid',
+                 weights=None, truncate_gradient=-1, return_sequences=False):
 
-        super(JZS1,self).__init__()
+        super(JZS1, self).__init__()
         self.input_dim = input_dim
         self.output_dim = output_dim
         self.truncate_gradient = truncate_gradient
@@ -482,10 +478,10 @@ class JZS1(Recurrent):
         if weights is not None:
             self.set_weights(weights)
 
-    def _step(self, 
-        xz_t, xr_t, xh_t, mask_tm1,
-        h_tm1, 
-        u_r, u_h):
+    def _step(self,
+              xz_t, xr_t, xh_t, mask_tm1,
+              h_tm1,
+              u_r, u_h):
         h_mask_tm1 = mask_tm1 * h_tm1
         z = self.inner_activation(xz_t)
         r = self.inner_activation(xr_t + T.dot(h_mask_tm1, u_r))
@@ -494,7 +490,7 @@ class JZS1(Recurrent):
         return h_t
 
     def get_output(self, train=False):
-        X = self.get_input(train) 
+        X = self.get_input(train)
         padded_mask = self.get_padded_shuffled_mask(train, X, pad=1)
         X = X.dimshuffle((1, 0, 2))
 
@@ -502,55 +498,53 @@ class JZS1(Recurrent):
         x_r = T.dot(X, self.W_r) + self.b_r
         x_h = T.tanh(T.dot(X, self.Pmat)) + self.b_h
         outputs, updates = theano.scan(
-            self._step, 
+            self._step,
             sequences=[x_z, x_r, x_h, padded_mask],
             outputs_info=T.unbroadcast(alloc_zeros_matrix(X.shape[1], self.output_dim), 1),
             non_sequences=[self.U_r, self.U_h],
-            truncate_gradient=self.truncate_gradient
-        )
+            truncate_gradient=self.truncate_gradient)
         if self.return_sequences:
             return outputs.dimshuffle((1, 0, 2))
         return outputs[-1]
 
     def get_config(self):
-        return {"name":self.__class__.__name__,
-            "input_dim":self.input_dim,
-            "output_dim":self.output_dim,
-            "init":self.init.__name__,
-            "inner_init":self.inner_init.__name__,
-            "activation":self.activation.__name__,
-            "inner_activation":self.inner_activation.__name__,
-            "truncate_gradient":self.truncate_gradient,
-            "return_sequences":self.return_sequences}
-
+        return {"name": self.__class__.__name__,
+                "input_dim": self.input_dim,
+                "output_dim": self.output_dim,
+                "init": self.init.__name__,
+                "inner_init": self.inner_init.__name__,
+                "activation": self.activation.__name__,
+                "inner_activation": self.inner_activation.__name__,
+                "truncate_gradient": self.truncate_gradient,
+                "return_sequences": self.return_sequences}
 
 
 class JZS2(Recurrent):
     '''
         Evolved recurrent neural network architectures from the evaluation of thousands
         of models, serving as alternatives to LSTMs and GRUs. See Jozefowicz et al. 2015.
-        
+
         This corresponds to the `MUT2` architecture described in the paper.
-        
+
         Takes inputs with shape:
         (nb_samples, max_sample_length (samples shorter than this are padded with zeros at the end), input_dim)
-        
+
         and returns outputs with shape:
         if not return_sequences:
             (nb_samples, output_dim)
         if return_sequences:
             (nb_samples, max_sample_length, output_dim)
-        
+
         References:
             An Empirical Exploration of Recurrent Network Architectures
                 http://www.jmlr.org/proceedings/papers/v37/jozefowicz15.pdf
     '''
-    def __init__(self, input_dim, output_dim=128, 
-        init='glorot_uniform', inner_init='orthogonal',
-        activation='tanh', inner_activation='sigmoid',
-        weights=None, truncate_gradient=-1, return_sequences=False):
+    def __init__(self, input_dim, output_dim=128,
+                 init='glorot_uniform', inner_init='orthogonal',
+                 activation='tanh', inner_activation='sigmoid',
+                 weights=None, truncate_gradient=-1, return_sequences=False):
 
-        super(JZS2,self).__init__()
+        super(JZS2, self).__init__()
         self.input_dim = input_dim
         self.output_dim = output_dim
         self.truncate_gradient = truncate_gradient
@@ -590,10 +584,10 @@ class JZS2(Recurrent):
         if weights is not None:
             self.set_weights(weights)
 
-    def _step(self, 
-        xz_t, xr_t, xh_t, mask_tm1,
-        h_tm1, 
-        u_z, u_r, u_h):
+    def _step(self,
+              xz_t, xr_t, xh_t, mask_tm1,
+              h_tm1,
+              u_z, u_r, u_h):
         h_mask_tm1 = mask_tm1 * h_tm1
         z = self.inner_activation(xz_t + T.dot(h_mask_tm1, u_z))
         r = self.inner_activation(xr_t + T.dot(h_mask_tm1, u_r))
@@ -604,61 +598,59 @@ class JZS2(Recurrent):
     def get_output(self, train=False):
         X = self.get_input(train)
         padded_mask = self.get_padded_shuffled_mask(train, X, pad=1)
-        X = X.dimshuffle((1, 0, 2)) 
+        X = X.dimshuffle((1, 0, 2))
 
         x_z = T.dot(X, self.W_z) + self.b_z
         x_r = T.dot(X, self.Pmat) + self.b_r
         x_h = T.dot(X, self.W_h) + self.b_h
         outputs, updates = theano.scan(
-            self._step, 
+            self._step,
             sequences=[x_z, x_r, x_h, padded_mask],
             outputs_info=T.unbroadcast(alloc_zeros_matrix(X.shape[1], self.output_dim), 1),
             non_sequences=[self.U_z, self.U_r, self.U_h],
-            truncate_gradient=self.truncate_gradient
-        )
+            truncate_gradient=self.truncate_gradient)
         if self.return_sequences:
             return outputs.dimshuffle((1, 0, 2))
         return outputs[-1]
 
     def get_config(self):
-        return {"name":self.__class__.__name__,
-            "input_dim":self.input_dim,
-            "output_dim":self.output_dim,
-            "init":self.init.__name__,
-            "inner_init":self.inner_init.__name__,
-            "activation":self.activation.__name__,
-            "inner_activation":self.inner_activation.__name__,
-            "truncate_gradient":self.truncate_gradient,
-            "return_sequences":self.return_sequences}
-
+        return {"name": self.__class__.__name__,
+                "input_dim": self.input_dim,
+                "output_dim": self.output_dim,
+                "init": self.init.__name__,
+                "inner_init": self.inner_init.__name__,
+                "activation": self.activation.__name__,
+                "inner_activation": self.inner_activation.__name__,
+                "truncate_gradient": self.truncate_gradient,
+                "return_sequences": self.return_sequences}
 
 
 class JZS3(Recurrent):
     '''
         Evolved recurrent neural network architectures from the evaluation of thousands
         of models, serving as alternatives to LSTMs and GRUs. See Jozefowicz et al. 2015.
-        
+
         This corresponds to the `MUT3` architecture described in the paper.
-        
+
         Takes inputs with shape:
         (nb_samples, max_sample_length (samples shorter than this are padded with zeros at the end), input_dim)
-        
+
         and returns outputs with shape:
         if not return_sequences:
             (nb_samples, output_dim)
         if return_sequences:
             (nb_samples, max_sample_length, output_dim)
-        
+
         References:
             An Empirical Exploration of Recurrent Network Architectures
                 http://www.jmlr.org/proceedings/papers/v37/jozefowicz15.pdf
     '''
-    def __init__(self, input_dim, output_dim=128, 
-        init='glorot_uniform', inner_init='orthogonal',
-        activation='tanh', inner_activation='sigmoid',
-        weights=None, truncate_gradient=-1, return_sequences=False):
+    def __init__(self, input_dim, output_dim=128,
+                 init='glorot_uniform', inner_init='orthogonal',
+                 activation='tanh', inner_activation='sigmoid',
+                 weights=None, truncate_gradient=-1, return_sequences=False):
 
-        super(JZS3,self).__init__()
+        super(JZS3, self).__init__()
         self.input_dim = input_dim
         self.output_dim = output_dim
         self.truncate_gradient = truncate_gradient
@@ -691,10 +683,10 @@ class JZS3(Recurrent):
         if weights is not None:
             self.set_weights(weights)
 
-    def _step(self, 
-        xz_t, xr_t, xh_t, mask_tm1,
-        h_tm1, 
-        u_z, u_r, u_h):
+    def _step(self,
+              xz_t, xr_t, xh_t, mask_tm1,
+              h_tm1,
+              u_z, u_r, u_h):
         h_mask_tm1 = mask_tm1 * h_tm1
         z = self.inner_activation(xz_t + T.dot(T.tanh(h_mask_tm1), u_z))
         r = self.inner_activation(xr_t + T.dot(h_mask_tm1, u_r))
@@ -705,13 +697,13 @@ class JZS3(Recurrent):
     def get_output(self, train=False):
         X = self.get_input(train)
         padded_mask = self.get_padded_shuffled_mask(train, X, pad=1)
-        X = X.dimshuffle((1, 0, 2)) 
+        X = X.dimshuffle((1, 0, 2))
 
         x_z = T.dot(X, self.W_z) + self.b_z
         x_r = T.dot(X, self.W_r) + self.b_r
         x_h = T.dot(X, self.W_h) + self.b_h
         outputs, updates = theano.scan(
-            self._step, 
+            self._step,
             sequences=[x_z, x_r, x_h, padded_mask],
             outputs_info=T.unbroadcast(alloc_zeros_matrix(X.shape[1], self.output_dim), 1),
             non_sequences=[self.U_z, self.U_r, self.U_h],
@@ -722,16 +714,12 @@ class JZS3(Recurrent):
         return outputs[-1]
 
     def get_config(self):
-        return {"name":self.__class__.__name__,
-            "input_dim":self.input_dim,
-            "output_dim":self.output_dim,
-            "init":self.init.__name__,
-            "inner_init":self.inner_init.__name__,
-            "activation":self.activation.__name__,
-            "inner_activation":self.inner_activation.__name__,
-            "truncate_gradient":self.truncate_gradient,
-            "return_sequences":self.return_sequences}
-
-
-
-
+        return {"name": self.__class__.__name__,
+                "input_dim": self.input_dim,
+                "output_dim": self.output_dim,
+                "init": self.init.__name__,
+                "inner_init": self.inner_init.__name__,
+                "activation": self.activation.__name__,
+                "inner_activation": self.inner_activation.__name__,
+                "truncate_gradient": self.truncate_gradient,
+                "return_sequences": self.return_sequences}

keras/models.py

@@ -3,19 +3,18 @@ from __future__ import print_function
 import theano
 import theano.tensor as T
 import numpy as np
-import warnings, time, copy, yaml
+import warnings, time, copy, pprint
+from six.moves import range
+import six
 
 from . import optimizers
 from . import objectives
 from . import regularizers
 from . import constraints
 from . import callbacks as cbks
-
-import time, copy, pprint
 from .utils.layer_utils import container_from_config
 from .utils.generic_utils import Progbar, printv
 from .layers import containers
-from six.moves import range
 
 
 def standardize_y(y):
@@ -26,6 +25,18 @@ def standardize_y(y):
     return y
 
 
+def batch_shuffle(index_array, batch_size):
+    batch_count = int(len(index_array)/batch_size)
+    # to reshape we need to be cleanly divisible by batch size
+    # we stash extra items and reappend them after shuffling
+    last_batch = index_array[batch_count*batch_size:]
+    index_array = index_array[:batch_count*batch_size]
+    index_array = index_array.reshape((batch_count, batch_size))
+    np.random.shuffle(index_array)
+    index_array = index_array.flatten()
+    return np.append(index_array, last_batch)
+
+
 def make_batches(size, batch_size):
     nb_batch = int(np.ceil(size/float(batch_size)))
     return [(i*batch_size, min(size, (i+1)*batch_size)) for i in range(0, nb_batch)]
@@ -52,13 +63,20 @@ def slice_X(X, start=None, stop=None):
 
 
 def weighted_objective(fn):
-    def weighted(y_true, y_pred, weights):
-        # it's important that 0 * Inf == 0, not NaN, so I need to mask first
-        masked_y_true = y_true[weights.nonzero()[:-1]]
-        masked_y_pred = y_pred[weights.nonzero()[:-1]]
-        masked_weights = weights[weights.nonzero()]
-        obj_output = fn(masked_y_true, masked_y_pred)
-        return (masked_weights.flatten() * obj_output.flatten()).mean()
+    def weighted(y_true, y_pred, weights, mask=None):
+        # it's important that 0 * Inf == 0, not NaN, so we need to filter
+        # those out first
+        filtered_y_true = y_true[weights.nonzero()[:-1]]
+        filtered_y_pred = y_pred[weights.nonzero()[:-1]]
+        filtered_weights = weights[weights.nonzero()]
+        obj_output = fn(filtered_y_true, filtered_y_pred)
+        weighted = filtered_weights * obj_output
+        if mask is None:
+            # Instead of calling mean() here, we divide by the sum of filtered_weights.
+            return weighted.sum() / filtered_weights.sum()
+        else:
+            filtered_mask = mask[weights.nonzero()[:-1]]
+            return weighted.sum() / (filtered_mask * filtered_weights).sum()
     return weighted
 
 
@@ -66,15 +84,18 @@ def standardize_weights(y, sample_weight=None, class_weight=None):
     if sample_weight is not None:
         return standardize_y(sample_weight)
     elif isinstance(class_weight, dict):
-        if len(y.shape) > 2:
-            raise Exception('class_weight not supported for 3+ dimensional targets.')
+        if len(y.shape) > 3:
+            raise Exception('class_weight not supported for 4+ dimensional targets.')
+        yshape = y.shape
+        y = np.reshape(y, (-1, yshape[-1]))  # for time-distributed data, collapse time and sample
         if y.shape[1] > 1:
             y_classes = y.argmax(axis=1)
         elif y.shape[1] == 1:
             y_classes = np.reshape(y, y.shape[0])
         else:
             y_classes = y
-        return np.expand_dims(np.array(list(map(lambda x: class_weight[x], y_classes))), 1)
+        class_weights = np.asarray([class_weight[cls] for cls in y_classes])
+        return np.reshape(class_weights, yshape[:-1] + (1,))  # uncollapse initial dimensions
     else:
         return np.ones(y.shape[:-1] + (1,))
 
@@ -84,21 +105,36 @@ def model_from_yaml(yaml_string):
         Returns a model generated from a local yaml file,
         which is either created by hand or from to_yaml method of Sequential or Graph
     '''
-    model_params = yaml.load(yaml_string)
-    model_name = model_params.get('name')
-    if not model_name in {'Graph', 'Sequential'}:
+    import yaml
+    config = yaml.load(yaml_string)
+    return model_from_config(config)
+
+
+def model_from_json(json_string):
+    import json
+    config = json.loads(json_string)
+    return model_from_config(config)
+
+
+def model_from_config(config):
+    model_name = config.get('name')
+    if model_name not in {'Graph', 'Sequential'}:
         raise Exception('Unrecognized model:', model_name)
 
     # Create a container then set class to appropriate model
-    model = container_from_config(model_params)
-    model.__class__ = get(model_name)
+    model = container_from_config(config)
+    if model_name == 'Graph':
+        model.__class__ = Graph
+    elif model_name == 'Sequential':
+        model.__class__ = Sequential
 
-    if 'optimizer' in model_params: # if it has an optimizer, the model is assumed to be compiled
-        loss = objectives.get(model_params.get('loss'))
-        class_mode = model_params.get('class_mode')
-        theano_mode = model_params.get('theano_mode')
+    if 'optimizer' in config:
+        # if it has an optimizer, the model is assumed to be compiled
+        loss = config.get('loss')
+        class_mode = config.get('class_mode')
+        theano_mode = config.get('theano_mode')
 
-        optimizer_params = model_params.get('optimizer')
+        optimizer_params = dict([(k, v) for k, v in config.get('optimizer').items()])
         optimizer_name = optimizer_params.pop('name')
         optimizer = optimizers.get(optimizer_name, optimizer_params)
 
@@ -110,9 +146,16 @@ def model_from_yaml(yaml_string):
     return model
 
 
+def get_function_name(o):
+    if isinstance(o, six.string_types):
+        return o
+    else:
+        return o.__name__
+
+
 class Model(object):
-    def _fit(self, f, ins, out_labels=[], batch_size=128, nb_epoch=100, verbose=1, callbacks=[], \
-        validation_split=0., val_f=None, val_ins=None, shuffle=True, metrics=[]):
+    def _fit(self, f, ins, out_labels=[], batch_size=128, nb_epoch=100, verbose=1, callbacks=[],
+             val_f=None, val_ins=None, shuffle=True, metrics=[]):
         '''
             Abstract fit function for f(*ins). Assume that f returns a list, labelled by out_labels.
         '''
@@ -121,13 +164,6 @@ class Model(object):
             do_validation = True
             if verbose:
                 print("Train on %d samples, validate on %d samples" % (len(ins[0]), len(val_ins[0])))
-        else:
-            if 0 < validation_split < 1:
-                do_validation = True
-                split_at = int(len(ins[0]) * (1 - validation_split))
-                (ins, val_ins) = (slice_X(ins, 0, split_at), slice_X(ins, split_at))
-                if verbose:
-                    print("Train on %d samples, validate on %d samples" % (len(ins[0]), len(val_ins[0])))
 
         nb_train_sample = len(ins[0])
         index_array = np.arange(nb_train_sample)
@@ -146,20 +182,27 @@ class Model(object):
             'nb_sample': nb_train_sample,
             'verbose': verbose,
             'do_validation': do_validation,
-            'metrics':metrics,
+            'metrics': metrics,
         })
         callbacks.on_train_begin()
 
         self.stop_training = False
         for epoch in range(nb_epoch):
             callbacks.on_epoch_begin(epoch)
-            if shuffle:
+            if shuffle == 'batch':
+                index_array = batch_shuffle(index_array, batch_size)
+            elif shuffle:
                 np.random.shuffle(index_array)
 
             batches = make_batches(nb_train_sample, batch_size)
             for batch_index, (batch_start, batch_end) in enumerate(batches):
                 batch_ids = index_array[batch_start:batch_end]
-                ins_batch = slice_X(ins, batch_ids)
+                try:
+                    ins_batch = slice_X(ins, batch_ids)
+                except TypeError as err:
+                    print('TypeError while preparing batch. \
+                        If using HDF5 input data, pass shuffle="batch".\n')
+                    raise
 
                 batch_logs = {}
                 batch_logs['batch'] = batch_index
@@ -172,10 +215,10 @@ class Model(object):
                     batch_logs[l] = o
 
                 callbacks.on_batch_end(batch_index, batch_logs)
-                
-                if batch_index == len(batches) - 1: # last batch
+
+                epoch_logs = {}
+                if batch_index == len(batches) - 1:  # last batch
                     # validation
-                    epoch_logs = {}
                     if do_validation:
                         # replace with self._evaluate
                         val_outs = self._test_loop(val_f, val_ins, batch_size=batch_size, verbose=0)
@@ -254,11 +297,35 @@ class Model(object):
 
     def get_config(self, verbose=0):
         config = super(Model, self).get_config()
+        for p in ['class_mode', 'theano_mode']:
+            if hasattr(self, p):
+                config[p] = getattr(self, p)
+        if hasattr(self, 'optimizer'):
+            config['optimizer'] = self.optimizer.get_config()
+        if hasattr(self, 'loss'):
+            if type(self.loss) == dict:
+                config['loss'] = dict([(k, get_function_name(v)) for k, v in self.loss.items()])
+            else:
+                config['loss'] = get_function_name(self.loss)
+
         if verbose:
             pp = pprint.PrettyPrinter(indent=4)
             pp.pprint(config)
         return config
 
+    def to_yaml(self):
+        # dump model configuration to yaml string
+        import yaml
+        config = self.get_config()
+        return yaml.dump(config)
+
+    def to_json(self):
+        # dump model configuration to json string
+        import json
+        config = self.get_config()
+        return json.dumps(config)
+
+
 class Sequential(Model, containers.Sequential):
     '''
         Inherits from Model the following methods:
@@ -267,7 +334,7 @@ class Sequential(Model, containers.Sequential):
             - _evaluate
         Inherits from containers.Sequential the following methods:
             - __init__
-            - add 
+            - add
             - get_output
             - get_input
             - get_weights
@@ -277,8 +344,8 @@ class Sequential(Model, containers.Sequential):
     def compile(self, optimizer, loss, class_mode="categorical", theano_mode=None):
         self.optimizer = optimizers.get(optimizer)
 
-        self.unweighted_loss = objectives.get(loss)
-        self.loss = weighted_objective(objectives.get(loss))
+        self.loss = objectives.get(loss)
+        weighted_loss = weighted_objective(objectives.get(loss))
 
         # input of model
         self.X_train = self.get_input(train=True)
@@ -292,8 +359,12 @@ class Sequential(Model, containers.Sequential):
 
         self.weights = T.ones_like(self.y_train)
 
-        train_loss = self.loss(self.y, self.y_train, self.weights)
-        test_loss = self.loss(self.y, self.y_test, self.weights)
+        if hasattr(self.layers[-1], "get_output_mask"):
+            mask = self.layers[-1].get_output_mask()
+        else:
+            mask = None
+        train_loss = weighted_loss(self.y, self.y_train, self.weights, mask)
+        test_loss = weighted_loss(self.y, self.y_test, self.weights, mask)
 
         train_loss.name = 'train_loss'
         test_loss.name = 'test_loss'
@@ -314,6 +385,7 @@ class Sequential(Model, containers.Sequential):
         for r in self.regularizers:
             train_loss = r(train_loss)
         updates = self.optimizer.get_updates(self.params, self.constraints, train_loss)
+        updates += self.updates
 
         if type(self.X_train) == list:
             train_ins = self.X_train + [self.y, self.weights]
@@ -324,64 +396,49 @@ class Sequential(Model, containers.Sequential):
             test_ins = [self.X_test, self.y, self.weights]
             predict_ins = [self.X_test]
 
-        self._train = theano.function(train_ins, train_loss,
-            updates=updates, allow_input_downcast=True, mode=theano_mode)
-        self._train_with_acc = theano.function(train_ins, [train_loss, train_accuracy],
-            updates=updates, allow_input_downcast=True, mode=theano_mode)
+        self._train = theano.function(train_ins, train_loss, updates=updates,
+                                      allow_input_downcast=True, mode=theano_mode)
+        self._train_with_acc = theano.function(train_ins, [train_loss, train_accuracy], updates=updates,
+                                               allow_input_downcast=True, mode=theano_mode)
         self._predict = theano.function(predict_ins, self.y_test,
-            allow_input_downcast=True, mode=theano_mode)
+                                        allow_input_downcast=True, mode=theano_mode)
         self._test = theano.function(test_ins, test_loss,
-            allow_input_downcast=True, mode=theano_mode)
+                                     allow_input_downcast=True, mode=theano_mode)
         self._test_with_acc = theano.function(test_ins, [test_loss, test_accuracy],
-            allow_input_downcast=True, mode=theano_mode)
+                                              allow_input_downcast=True, mode=theano_mode)
 
-    def train(self, X, y, accuracy=False, sample_weight=None):
-        warnings.warn('The "train" method is deprecated, use "train_on_batch" instead.')
-        return self.train_on_batch(X, y, accuracy, sample_weight)
-
-    def test(self, X, y, accuracy=False):
-        warnings.warn('The "test" method is deprecated, use "test_on_batch" instead.')
-        return self.test_on_batch(X, y, accuracy)
-
-    def train_on_batch(self, X, y, accuracy=False, sample_weight=None):
+    def train_on_batch(self, X, y, accuracy=False, class_weight=None, sample_weight=None):
         X = standardize_X(X)
         y = standardize_y(y)
-
-        if sample_weight is None:
-            sample_weight = np.ones(list(y.shape[0:-1]) + [1])
-        else:
-            sample_weight = standardize_y(sample_weight)
+        sample_weight = standardize_weights(y, class_weight=class_weight, sample_weight=sample_weight)
 
         ins = X + [y, sample_weight]
         if accuracy:
             return self._train_with_acc(*ins)
         else:
             return self._train(*ins)
-        
 
-    def test_on_batch(self, X, y, accuracy=False):
+    def test_on_batch(self, X, y, accuracy=False, sample_weight=None):
         X = standardize_X(X)
         y = standardize_y(y)
-        sample_weight = np.ones(y.shape[:-1] + (1,))
+        sample_weight = standardize_weights(y, sample_weight=sample_weight)
+
         ins = X + [y, sample_weight]
         if accuracy:
             return self._test_with_acc(*ins)
         else:
             return self._test(*ins)
 
-
     def predict_on_batch(self, X):
         ins = standardize_X(X)
         return self._predict(*ins)
 
-
     def fit(self, X, y, batch_size=128, nb_epoch=100, verbose=1, callbacks=[],
             validation_split=0., validation_data=None, shuffle=True, show_accuracy=False,
             class_weight=None, sample_weight=None):
 
         X = standardize_X(X)
         y = standardize_y(y)
-        sample_weight = standardize_weights(y, class_weight=class_weight, sample_weight=sample_weight)
 
         val_f = None
         val_ins = None
@@ -391,14 +448,31 @@ class Sequential(Model, containers.Sequential):
             else:
                 val_f = self._test
         if validation_data:
-            try:
+            if len(validation_data) == 2:
                 X_val, y_val = validation_data
-            except:
-                raise Exception("Invalid format for validation data; provide a tuple (X_val, y_val). \
+                X_val = standardize_X(X_val)
+                y_val = standardize_y(y_val)
+                sample_weight_val = np.ones(y_val.shape[:-1] + (1,))
+            elif len(validation_data) == 3:
+                X_val, y_val, sample_weight_val = validation_data
+                X_val = standardize_X(X_val)
+                y_val = standardize_y(y_val)
+                sample_weight_val = standardize_weights(y_val, sample_weight=sample_weight_val)
+            else:
+                raise Exception("Invalid format for validation data; provide a tuple (X_val, y_val) or (X_val, y_val, sample_weight). \
                     X_val may be a numpy array or a list of numpy arrays depending on your model input.")
-            X_val = standardize_X(X_val)
-            y_val = standardize_y(y_val)
-            val_ins = X_val + [y_val, np.ones(y_val.shape[:-1] + (1,))]
+            val_ins = X_val + [y_val, sample_weight_val]
+
+        elif 0 < validation_split < 1:
+            split_at = int(len(X[0]) * (1 - validation_split))
+            X, X_val = (slice_X(X, 0, split_at), slice_X(X, split_at))
+            y, y_val = (slice_X(y, 0, split_at), slice_X(y, split_at))
+            if sample_weight is not None:
+                sample_weight, sample_weight_val = (slice_X(sample_weight, 0, split_at), slice_X(sample_weight, split_at))
+                sample_weight_val = standardize_weights(y_val, sample_weight=sample_weight_val)
+            else:
+                sample_weight_val = np.ones(y_val.shape[:-1] + (1,))
+            val_ins = X_val + [y_val, sample_weight_val]
 
         if show_accuracy:
             f = self._train_with_acc
@@ -407,24 +481,24 @@ class Sequential(Model, containers.Sequential):
             f = self._train
             out_labels = ['loss']
 
+        sample_weight = standardize_weights(y, class_weight=class_weight, sample_weight=sample_weight)
         ins = X + [y, sample_weight]
         metrics = ['loss', 'acc', 'val_loss', 'val_acc']
-        return self._fit(f, ins, out_labels=out_labels, batch_size=batch_size, nb_epoch=nb_epoch, verbose=verbose, callbacks=callbacks, \
-            validation_split=validation_split, val_f=val_f, val_ins=val_ins, shuffle=shuffle, metrics=metrics)
-
+        return self._fit(f, ins, out_labels=out_labels, batch_size=batch_size, nb_epoch=nb_epoch,
+                         verbose=verbose, callbacks=callbacks,
+                         val_f=val_f, val_ins=val_ins,
+                         shuffle=shuffle, metrics=metrics)
 
     def predict(self, X, batch_size=128, verbose=0):
         X = standardize_X(X)
         return self._predict_loop(self._predict, X, batch_size, verbose)[0]
 
-
     def predict_proba(self, X, batch_size=128, verbose=1):
         preds = self.predict(X, batch_size, verbose)
         if preds.min() < 0 or preds.max() > 1:
             warnings.warn("Network returning invalid probability values.")
         return preds
 
-
     def predict_classes(self, X, batch_size=128, verbose=1):
         proba = self.predict(X, batch_size=batch_size, verbose=verbose)
         if self.class_mode == "categorical":
@@ -432,7 +506,6 @@ class Sequential(Model, containers.Sequential):
         else:
             return (proba > 0.5).astype('int32')
 
-
     def evaluate(self, X, y, batch_size=128, show_accuracy=False, verbose=1, sample_weight=None):
         X = standardize_X(X)
         y = standardize_y(y)
@@ -449,7 +522,6 @@ class Sequential(Model, containers.Sequential):
         else:
             return outs[0]
 
-
     def save_weights(self, filepath, overwrite=False):
         # Save weights from all layers to HDF5
         import h5py
@@ -480,7 +552,6 @@ class Sequential(Model, containers.Sequential):
         f.flush()
         f.close()
 
-
     def load_weights(self, filepath):
         '''
             This method does not make use of Sequential.set_weights()
@@ -496,27 +567,13 @@ class Sequential(Model, containers.Sequential):
         f.close()
 
 
-    def to_yaml(self):
-        '''
-            Stores a model to yaml string, optionally with all learnable parameters
-            If the model is compiled, it will also serialize the necessary components
-        '''
-        model_params = self.get_config()
-        if hasattr(self, 'optimizer'):
-            model_params['class_mode'] = self.class_mode
-            model_params['theano_mode'] = self.theano_mode
-            model_params['loss'] = self.unweighted_loss.__name__
-            model_params['optimizer'] = self.optimizer.get_config()
-
-        return yaml.dump(model_params)
-
-
 class Graph(Model, containers.Graph):
     def compile(self, optimizer, loss, theano_mode=None):
         # loss is a dictionary mapping output name to loss functions
         ys = []
         ys_train = []
         ys_test = []
+        weights = []
         train_loss = 0.
         test_loss = 0.
         for output_name in self.output_order:
@@ -528,39 +585,53 @@ class Graph(Model, containers.Graph):
             ys.append(y)
             ys_train.append(y_train)
             ys_test.append(y_test)
-            
-            train_loss += objectives.get(loss_fn)(y, y_train).mean()
-            test_loss += objectives.get(loss_fn)(y, y_test).mean()
+
+            if hasattr(output, "get_output_mask"):
+                mask = output.get_output_mask()
+            else:
+                mask = None
+
+            weight = T.ones_like(y_test)
+            weights.append(weight)
+            weighted_loss = weighted_objective(objectives.get(loss_fn))
+            train_loss += weighted_loss(y, y_train, weight, mask)
+            test_loss += weighted_loss(y, y_test, weight, mask)
 
         train_loss.name = 'train_loss'
         test_loss.name = 'test_loss'
 
         ins = [self.inputs[name].input for name in self.input_order]
-        train_ins = ins + ys
-        test_ins = ins + ys
+        train_ins = ins + ys + weights
+        test_ins = ins + ys + weights
 
         for r in self.regularizers:
             train_loss = r(train_loss)
         self.optimizer = optimizers.get(optimizer)
         updates = self.optimizer.get_updates(self.params, self.constraints, train_loss)
+        updates += self.updates
         self.theano_mode = theano_mode
         self.loss = loss
 
-        self._train = theano.function(train_ins, train_loss, 
-            updates=updates, allow_input_downcast=True, mode=theano_mode)
-        self._test = theano.function(test_ins, test_loss, 
-            allow_input_downcast=True, mode=theano_mode)
-        self._predict = theano.function(inputs=ins, outputs=ys_test, 
-            allow_input_downcast=True, mode=theano_mode)
+        self._train = theano.function(train_ins, train_loss, updates=updates,
+                                      allow_input_downcast=True, mode=theano_mode)
+        self._test = theano.function(test_ins, test_loss,
+                                     allow_input_downcast=True, mode=theano_mode)
+        self._predict = theano.function(inputs=ins, outputs=ys_test,
+                                        allow_input_downcast=True, mode=theano_mode)
 
-    def train_on_batch(self, data):
+    def train_on_batch(self, data, class_weight={}, sample_weight={}):
         # data is a dictionary mapping output and input names to arrays
-        ins = [data[name] for name in self.input_order] + [standardize_y(data[name]) for name in self.output_order]
+        sample_weight = [standardize_weights(data[name],
+                                             sample_weight=sample_weight.get(name),
+                                             class_weight=class_weight.get(name)) for name in self.output_order]
+        ins = [data[name] for name in self.input_order] + [standardize_y(data[name]) for name in self.output_order] + sample_weight
         return self._train(*ins)
 
-    def test_on_batch(self, data):
+    def test_on_batch(self, data, sample_weight={}):
         # data is a dictionary mapping input names to arrays
-        ins = [data[name] for name in self.input_order] + [standardize_y(data[name]) for name in self.output_order]
+        sample_weight = [standardize_weights(data[name],
+                                             sample_weight=sample_weight.get(name)) for name in self.output_order]
+        ins = [data[name] for name in self.input_order] + [standardize_y(data[name]) for name in self.output_order] + sample_weight
         return self._test(*ins)
 
     def predict_on_batch(self, data):
@@ -569,25 +640,49 @@ class Graph(Model, containers.Graph):
         return self._predict(*ins)
 
     def fit(self, data, batch_size=128, nb_epoch=100, verbose=1, callbacks=[],
-            validation_split=0., validation_data=None, shuffle=True):
-        ins = [data[name] for name in self.input_order] + [standardize_y(data[name]) for name in self.output_order]
+            validation_split=0., validation_data=None, shuffle=True, class_weight={}, sample_weight={}):
+        X = [data[name] for name in self.input_order]
+        y = [standardize_y(data[name]) for name in self.output_order]
+        sample_weight_list = [standardize_weights(data[name],
+                                                  sample_weight=sample_weight.get(name)) for name in self.output_order]
+        class_weight_list = [class_weight.get(name) for name in self.output_order]
 
         val_f = None
         val_ins = None
         if validation_data or validation_split:
             val_f = self._test
         if validation_data:
-            val_ins = [validation_data[name] for name in self.input_order] + [standardize_y(validation_data[name]) for name in self.output_order]
+            # can't use sample weights with validation data at this point
+            sample_weight = [standardize_weights(validation_data[name]) for name in self.output_order]
+            val_ins = [validation_data[name] for name in self.input_order] + [standardize_y(validation_data[name]) for name in self.output_order] + sample_weight
+
+        elif 0 < validation_split < 1:
+            split_at = int(len(X[0]) * (1 - validation_split))
+            X, X_val = (slice_X(X, 0, split_at), slice_X(X, split_at))
+            y, y_val = (slice_X(y, 0, split_at), slice_X(y, split_at))
+            sample_weight_list, sample_weight_list_val = (slice_X(sample_weight_list, 0, split_at), slice_X(sample_weight_list, split_at))
+            val_ins = X_val + y_val + sample_weight_list_val
 
         f = self._train
-        out_labels = self.output_order
-        metrics = self.output_order + ['val_' + m for m in self.output_order]
-        history = self._fit(f, ins, out_labels=out_labels, batch_size=batch_size, nb_epoch=nb_epoch, verbose=verbose, callbacks=callbacks, \
-            validation_split=validation_split, val_f=val_f, val_ins=val_ins, shuffle=shuffle, metrics=metrics)
+        out_labels = ['loss']
+        metrics = ['loss', 'val_loss']
+
+        sample_weight_list = [standardize_weights(y[i],
+                                                  sample_weight=sample_weight_list[i],
+                                                  class_weight=class_weight_list[i]) for i in range(len(self.output_order))]
+        ins = X + y + sample_weight_list
+
+        history = self._fit(f, ins, out_labels=out_labels, batch_size=batch_size, nb_epoch=nb_epoch,
+                            verbose=verbose, callbacks=callbacks,
+                            val_f=val_f, val_ins=val_ins,
+                            shuffle=shuffle, metrics=metrics)
         return history
 
-    def evaluate(self, data, batch_size=128, verbose=0):
-        ins = [data[name] for name in self.input_order] + [standardize_y(data[name]) for name in self.output_order]
+    def evaluate(self, data, batch_size=128, verbose=0, sample_weight={}):
+        sample_weight = [standardize_weights(data[name],
+                                             sample_weight=sample_weight.get(name)) for name in self.output_order]
+
+        ins = [data[name] for name in self.input_order] + [standardize_y(data[name]) for name in self.output_order] + sample_weight
         outs = self._test_loop(self._test, ins, batch_size, verbose)
         return outs[0]
 
@@ -632,19 +727,3 @@ class Graph(Model, containers.Graph):
         weights = [g['param_{}'.format(p)] for p in range(g.attrs['nb_params'])]
         self.set_weights(weights)
         f.close()
-
-    def to_yaml(self):
-        '''
-            Stores a model to yaml string, optionally with all learnable parameters
-            If the model is compiled, it will also serialize the necessary components
-        '''
-        model_params = self.get_config()
-        if hasattr(self, 'optimizer'):
-            model_params['theano_mode'] = self.theano_mode
-            model_params['loss'] = self.loss
-            model_params['optimizer'] = self.optimizer.get_config()
-        return yaml.dump(model_params)
-
-from .utils.generic_utils import get_from_module
-def get(identifier):
-    return get_from_module(identifier, globals(), 'model')

keras/objectives.py

@@ -9,38 +9,50 @@ if theano.config.floatX == 'float64':
 else:
     epsilon = 1.0e-7
 
+
 def mean_squared_error(y_true, y_pred):
     return T.sqr(y_pred - y_true).mean(axis=-1)
 
+
 def mean_absolute_error(y_true, y_pred):
     return T.abs_(y_pred - y_true).mean(axis=-1)
 
+
 def mean_absolute_percentage_error(y_true, y_pred):
     return T.abs_((y_true - y_pred) / T.clip(T.abs_(y_true), epsilon, np.inf)).mean(axis=-1) * 100.
 
+
 def mean_squared_logarithmic_error(y_true, y_pred):
     return T.sqr(T.log(T.clip(y_pred, epsilon, np.inf) + 1.) - T.log(T.clip(y_true, epsilon, np.inf) + 1.)).mean(axis=-1)
 
+
 def squared_hinge(y_true, y_pred):
     return T.sqr(T.maximum(1. - y_true * y_pred, 0.)).mean(axis=-1)
 
+
 def hinge(y_true, y_pred):
     return T.maximum(1. - y_true * y_pred, 0.).mean(axis=-1)
 
+
 def categorical_crossentropy(y_true, y_pred):
     '''Expects a binary class matrix instead of a vector of scalar classes
     '''
     y_pred = T.clip(y_pred, epsilon, 1.0 - epsilon)
     # scale preds so that the class probas of each sample sum to 1
-    y_pred /= y_pred.sum(axis=-1, keepdims=True) 
+    y_pred /= y_pred.sum(axis=-1, keepdims=True)
     cce = T.nnet.categorical_crossentropy(y_pred, y_true)
     return cce
 
+
 def binary_crossentropy(y_true, y_pred):
     y_pred = T.clip(y_pred, epsilon, 1.0 - epsilon)
     bce = T.nnet.binary_crossentropy(y_pred, y_true).mean(axis=-1)
     return bce
 
+
+def poisson_loss(y_true, y_pred):
+    return T.mean(y_pred - y_true * T.log(y_pred + epsilon), axis=-1)
+
 # aliases
 mse = MSE = mean_squared_error
 mae = MAE = mean_absolute_error
@@ -50,4 +62,3 @@ msle = MSLE = mean_squared_logarithmic_error
 from .utils.generic_utils import get_from_module
 def get(identifier):
     return get_from_module(identifier, globals(), 'objective')
-

keras/optimizers.py

@@ -1,21 +1,35 @@
 from __future__ import absolute_import
 import theano
 import theano.tensor as T
-import numpy as np
 
-from .utils.theano_utils import shared_zeros, shared_scalar
+from .utils.theano_utils import shared_zeros, shared_scalar, floatX
+from .utils.generic_utils import get_from_module
 from six.moves import zip
 
+
 def clip_norm(g, c, n):
     if c > 0:
         g = T.switch(T.ge(n, c), g * c / n, g)
     return g
 
+
 def kl_divergence(p, p_hat):
     return p_hat - p + p * T.log(p / p_hat)
 
+
 class Optimizer(object):
-    
+    def __init__(self, **kwargs):
+        self.__dict__.update(kwargs)
+        self.updates = []
+
+    def get_state(self):
+        return [u[0].get_value() for u in self.updates]
+
+    def set_state(self, value_list):
+        assert len(self.updates) == len(value_list)
+        for u, v in zip(self.updates, value_list):
+            u[0].set_value(floatX(v))
+
     def get_updates(self, params, constraints, loss):
         raise NotImplementedError
 
@@ -30,173 +44,172 @@ class Optimizer(object):
         return grads
 
     def get_config(self):
-        return {"name":self.__class__.__name__}
+        return {"name": self.__class__.__name__}
+
 
 class SGD(Optimizer):
 
     def __init__(self, lr=0.01, momentum=0., decay=0., nesterov=False, *args, **kwargs):
-        self.__dict__.update(kwargs)
+        super(SGD, self).__init__(**kwargs)
         self.__dict__.update(locals())
         self.iterations = shared_scalar(0)
+        self.lr = shared_scalar(lr)
+        self.momentum = shared_scalar(momentum)
 
     def get_updates(self, params, constraints, loss):
         grads = self.get_gradients(loss, params)
         lr = self.lr * (1.0 / (1.0 + self.decay * self.iterations))
-        updates = [(self.iterations, self.iterations + 1.)]
+        self.updates = [(self.iterations, self.iterations + 1.)]
 
         for p, g, c in zip(params, grads, constraints):
-            m = shared_zeros(p.get_value().shape) # momentum
-            v = self.momentum * m - lr * g # velocity
-            updates.append((m, v)) 
+            m = shared_zeros(p.get_value().shape)  # momentum
+            v = self.momentum * m - lr * g  # velocity
+            self.updates.append((m, v))
 
             if self.nesterov:
                 new_p = p + self.momentum * v - lr * g
             else:
                 new_p = p + v
 
-            updates.append((p, c(new_p))) # apply constraints
-        return updates
+            self.updates.append((p, c(new_p)))  # apply constraints
+        return self.updates
 
     def get_config(self):
-        return {"name":self.__class__.__name__,
-            "lr":self.lr,
-            "momentum":self.momentum,
-            "decay":self.decay,
-            "nesterov":self.nesterov}
+        return {"name": self.__class__.__name__,
+                "lr": float(self.lr.get_value()),
+                "momentum": float(self.momentum.get_value()),
+                "decay": float(self.decay.get_value()),
+                "nesterov": self.nesterov}
 
 
 class RMSprop(Optimizer):
-
     def __init__(self, lr=0.001, rho=0.9, epsilon=1e-6, *args, **kwargs):
-        self.__dict__.update(kwargs)
+        super(RMSprop, self).__init__(**kwargs)
         self.__dict__.update(locals())
+        self.lr = shared_scalar(lr)
+        self.rho = shared_scalar(rho)
 
     def get_updates(self, params, constraints, loss):
         grads = self.get_gradients(loss, params)
         accumulators = [shared_zeros(p.get_value().shape) for p in params]
-        updates = []
+        self.updates = []
 
         for p, g, a, c in zip(params, grads, accumulators, constraints):
-            new_a = self.rho * a + (1 - self.rho) * g ** 2 # update accumulator
-            updates.append((a, new_a))
+            new_a = self.rho * a + (1 - self.rho) * g ** 2  # update accumulator
+            self.updates.append((a, new_a))
 
             new_p = p - self.lr * g / T.sqrt(new_a + self.epsilon)
-            updates.append((p, c(new_p))) # apply constraints
-            
-        return updates
+            self.updates.append((p, c(new_p)))  # apply constraints
+        return self.updates
 
     def get_config(self):
-        return {"name":self.__class__.__name__,
-            "lr":self.lr,
-            "rho":self.rho,
-            "epsilon":self.epsilon}
+        return {"name": self.__class__.__name__,
+                "lr": float(self.lr.get_value()),
+                "rho": float(self.rho.get_value()),
+                "epsilon": self.epsilon}
+
 
 class Adagrad(Optimizer):
-
     def __init__(self, lr=0.01, epsilon=1e-6, *args, **kwargs):
-        self.__dict__.update(kwargs)
+        super(Adagrad, self).__init__(**kwargs)
         self.__dict__.update(locals())
+        self.lr = shared_scalar(lr)
 
     def get_updates(self, params, constraints, loss):
         grads = self.get_gradients(loss, params)
         accumulators = [shared_zeros(p.get_value().shape) for p in params]
-        updates = []
+        self.updates = []
 
         for p, g, a, c in zip(params, grads, accumulators, constraints):
-            new_a = a + g ** 2 # update accumulator
-            updates.append((a, new_a))
-
+            new_a = a + g ** 2  # update accumulator
+            self.updates.append((a, new_a))
             new_p = p - self.lr * g / T.sqrt(new_a + self.epsilon)
-            updates.append((p, c(new_p))) # apply constraints
-        return updates
+            self.updates.append((p, c(new_p)))  # apply constraints
+        return self.updates
 
     def get_config(self):
-        return {"name":self.__class__.__name__,
-            "lr":self.lr,
-            "epsilon":self.epsilon}
+        return {"name": self.__class__.__name__,
+                "lr": float(self.lr.get_value()),
+                "epsilon": self.epsilon}
+
 
 class Adadelta(Optimizer):
     '''
         Reference: http://arxiv.org/abs/1212.5701
     '''
     def __init__(self, lr=1.0, rho=0.95, epsilon=1e-6, *args, **kwargs):
-        self.__dict__.update(kwargs)
+        super(Adadelta, self).__init__(**kwargs)
         self.__dict__.update(locals())
+        self.lr = shared_scalar(lr)
 
     def get_updates(self, params, constraints, loss):
         grads = self.get_gradients(loss, params)
         accumulators = [shared_zeros(p.get_value().shape) for p in params]
         delta_accumulators = [shared_zeros(p.get_value().shape) for p in params]
-        updates = []
+        self.updates = []
 
-        for p, g, a, d_a, c in zip(params, grads, accumulators, delta_accumulators, constraints):
-            new_a = self.rho * a + (1 - self.rho) * g ** 2 # update accumulator
-            updates.append((a, new_a))
+        for p, g, a, d_a, c in zip(params, grads, accumulators,
+                                   delta_accumulators, constraints):
+            new_a = self.rho * a + (1 - self.rho) * g ** 2  # update accumulator
+            self.updates.append((a, new_a))
 
             # use the new accumulator and the *old* delta_accumulator
-            update = g * T.sqrt(d_a + self.epsilon) / T.sqrt(new_a + self.epsilon)
+            update = g * T.sqrt(d_a + self.epsilon) / T.sqrt(new_a +
+                                                             self.epsilon)
 
             new_p = p - self.lr * update
-            updates.append((p, c(new_p))) # apply constraints
+            self.updates.append((p, c(new_p)))  # apply constraints
 
             # update delta_accumulator
             new_d_a = self.rho * d_a + (1 - self.rho) * update ** 2
-            updates.append((d_a, new_d_a))
-        return updates
+            self.updates.append((d_a, new_d_a))
+        return self.updates
 
     def get_config(self):
-        return {"name":self.__class__.__name__,
-            "lr":self.lr,
-            "rho":self.rho,
-            "epsilon":self.epsilon}
+        return {"name": self.__class__.__name__,
+                "lr": float(self.lr.get_value()),
+                "rho": self.rho,
+                "epsilon": self.epsilon}
+
 
 class Adam(Optimizer):
     '''
-        Reference: http://arxiv.org/abs/1412.6980
+        Reference: http://arxiv.org/abs/1412.6980v8
 
-        Default parameters follow those provided in the original paper
-
-        lambda is renamed kappa.
+        Default parameters follow those provided in the original paper.
     '''
-    def __init__(self, lr=0.001, beta_1=0.9, beta_2=0.999, epsilon=1e-8, kappa=1-1e-8, *args, **kwargs):
-        self.__dict__.update(kwargs)
+    def __init__(self, lr=0.001, beta_1=0.9, beta_2=0.999, epsilon=1e-8, *args, **kwargs):
+        super(Adam, self).__init__(**kwargs)
         self.__dict__.update(locals())
         self.iterations = shared_scalar(0)
+        self.lr = shared_scalar(lr)
 
     def get_updates(self, params, constraints, loss):
         grads = self.get_gradients(loss, params)
-        updates = [(self.iterations, self.iterations+1.)]
+        self.updates = [(self.iterations, self.iterations+1.)]
 
-        i = self.iterations
-        beta_1_t = self.beta_1 * (self.kappa**i)
-
-        # the update below seems missing from the paper, but is obviously required
-        beta_2_t = self.beta_2 * (self.kappa**i) 
+        t = self.iterations + 1
+        lr_t = self.lr * T.sqrt(1-self.beta_2**t)/(1-self.beta_1**t)
 
         for p, g, c in zip(params, grads, constraints):
-            m = theano.shared(p.get_value() * 0.) # zero init of moment
-            v = theano.shared(p.get_value() * 0.) # zero init of velocity
+            m = theano.shared(p.get_value() * 0.)  # zero init of moment
+            v = theano.shared(p.get_value() * 0.)  # zero init of velocity
 
-            m_t = (beta_1_t * m) + (1 - beta_1_t) * g
-            v_t = (beta_2_t * v) + (1 - beta_2_t) * (g**2)
+            m_t = (self.beta_1 * m) + (1 - self.beta_1) * g
+            v_t = (self.beta_2 * v) + (1 - self.beta_2) * (g**2)
+            p_t = p - lr_t * m_t / (T.sqrt(v_t) + self.epsilon)
 
-            m_b_t = m_t / (1 - beta_1_t)
-            v_b_t = v_t / (1 - beta_2_t)
-
-            p_t = p - self.lr * m_b_t / (T.sqrt(v_b_t) + self.epsilon)
-            
-            updates.append((m, m_t))
-            updates.append((v, v_t))
-            updates.append((p, c(p_t))) # apply constraints
-        return updates
+            self.updates.append((m, m_t))
+            self.updates.append((v, v_t))
+            self.updates.append((p, c(p_t)))  # apply constraints
+        return self.updates
 
     def get_config(self):
-        return {"name":self.__class__.__name__,
-            "lr":self.lr,
-            "beta_1":self.beta_1,
-            "beta_2":self.beta_2,
-            "epsilon":self.epsilon,
-            "kappa":self.kappa}
+        return {"name": self.__class__.__name__,
+                "lr": float(self.lr.get_value()),
+                "beta_1": self.beta_1,
+                "beta_2": self.beta_2,
+                "epsilon": self.epsilon}
 
 # aliases
 sgd = SGD
@@ -205,6 +218,7 @@ adagrad = Adagrad
 adadelta = Adadelta
 adam = Adam
 
-from .utils.generic_utils import get_from_module
+
 def get(identifier, kwargs=None):
-    return get_from_module(identifier, globals(), 'optimizer', instantiate=True, kwargs=kwargs)
+    return get_from_module(identifier, globals(), 'optimizer', instantiate=True,
+                           kwargs=kwargs)

keras/preprocessing/text.py

@@ -15,12 +15,14 @@ if sys.version_info < (3,):
 else:
     maketrans = str.maketrans
 
+
 def base_filter():
     f = string.punctuation
     f = f.replace("'", '')
     f += '\t\n'
     return f
 
+
 def text_to_word_sequence(text, filters=base_filter(), lower=True, split=" "):
     '''prune: sequence of characters to filter out
     '''
@@ -32,8 +34,8 @@ def text_to_word_sequence(text, filters=base_filter(), lower=True, split=" "):
 
 
 def one_hot(text, n, filters=base_filter(), lower=True, split=" "):
-    seq = text_to_word_sequence(text)
-    return [(abs(hash(w))%(n-1)+1) for w in seq]
+    seq = text_to_word_sequence(text, filters=filters, lower=lower, split=split)
+    return [(abs(hash(w)) % (n - 1) + 1) for w in seq]
 
 
 class Tokenizer(object):
@@ -67,18 +69,17 @@ class Tokenizer(object):
                     self.word_docs[w] = 1
 
         wcounts = list(self.word_counts.items())
-        wcounts.sort(key = lambda x: x[1], reverse=True)
+        wcounts.sort(key=lambda x: x[1], reverse=True)
         sorted_voc = [wc[0] for wc in wcounts]
-        self.word_index = dict(list(zip(sorted_voc, list(range(1, len(sorted_voc)+1)))))
+        self.word_index = dict(list(zip(sorted_voc, list(range(1, len(sorted_voc) + 1)))))
 
         self.index_docs = {}
         for w, c in list(self.word_docs.items()):
             self.index_docs[self.word_index[w]] = c
 
-
     def fit_on_sequences(self, sequences):
         '''
-            required before using sequences_to_matrix 
+            required before using sequences_to_matrix
             (if fit_on_texts was never called)
         '''
         self.document_count = len(sequences)
@@ -91,7 +92,6 @@ class Tokenizer(object):
                 else:
                     self.index_docs[i] += 1
 
-
     def texts_to_sequences(self, texts):
         '''
             Transform each text in texts in a sequence of integers.
@@ -126,7 +126,6 @@ class Tokenizer(object):
                         vect.append(i)
             yield vect
 
-
     def texts_to_matrix(self, texts, mode="binary"):
         '''
             modes: binary, count, tfidf, freq
@@ -140,7 +139,7 @@ class Tokenizer(object):
         '''
         if not self.nb_words:
             if self.word_index:
-                nb_words = len(self.word_index)
+                nb_words = len(self.word_index) + 1
             else:
                 raise Exception("Specify a dimension (nb_words argument), or fit on some text data first")
         else:
@@ -165,21 +164,13 @@ class Tokenizer(object):
                 if mode == "count":
                     X[i][j] = c
                 elif mode == "freq":
-                    X[i][j] = c/len(seq)
+                    X[i][j] = c / len(seq)
                 elif mode == "binary":
                     X[i][j] = 1
                 elif mode == "tfidf":
-                    tf = np.log(c/len(seq))
-                    df = (1 + np.log(1 + self.index_docs.get(j, 0)/(1 + self.document_count)))
+                    tf = np.log(c / len(seq))
+                    df = (1 + np.log(1 + self.index_docs.get(j, 0) / (1 + self.document_count)))
                     X[i][j] = tf / df
                 else:
                     raise Exception("Unknown vectorization mode: " + str(mode))
         return X
-
-
-
-                
-
-
-
-    

keras/regularizers.py

@@ -1,6 +1,7 @@
 from __future__ import absolute_import
 import theano.tensor as T
 
+
 class Regularizer(object):
     def set_param(self, p):
         self.p = p
@@ -12,7 +13,8 @@ class Regularizer(object):
         return loss
 
     def get_config(self):
-        return {"name":self.__class__.__name__}
+        return {"name": self.__class__.__name__}
+
 
 class WeightRegularizer(Regularizer):
     def __init__(self, l1=0., l2=0.):
@@ -28,9 +30,10 @@ class WeightRegularizer(Regularizer):
         return loss
 
     def get_config(self):
-        return {"name":self.__class__.__name__,
-            "l1":self.l1,
-            "l2":self.l2}
+        return {"name": self.__class__.__name__,
+                "l1": self.l1,
+                "l2": self.l2}
+
 
 class ActivityRegularizer(Regularizer):
     def __init__(self, l1=0., l2=0.):
@@ -44,27 +47,33 @@ class ActivityRegularizer(Regularizer):
         loss += self.l1 * T.sum(T.mean(abs(self.layer.get_output(True)), axis=0))
         loss += self.l2 * T.sum(T.mean(self.layer.get_output(True) ** 2, axis=0))
         return loss
-    
+
     def get_config(self):
-        return {"name":self.__class__.__name__,
-            "l1":self.l1,
-            "l2":self.l2}
+        return {"name": self.__class__.__name__,
+                "l1": self.l1,
+                "l2": self.l2}
+
 
 def l1(l=0.01):
     return WeightRegularizer(l1=l)
 
+
 def l2(l=0.01):
     return WeightRegularizer(l2=l)
 
+
 def l1l2(l1=0.01, l2=0.01):
     return WeightRegularizer(l1=l1, l2=l2)
 
+
 def activity_l1(l=0.01):
     return ActivityRegularizer(l1=l)
 
+
 def activity_l2(l=0.01):
     return ActivityRegularizer(l2=l)
 
+
 def activity_l1l2(l1=0.01, l2=0.01):
     return ActivityRegularizer(l1=l1, l2=l2)
 

keras/utils/dot_utils.py

@@ -1,51 +0,0 @@
-import pydot
-from keras.layers.core import Merge
-from keras.models import Model
-from collections import Counter
-
-class Grapher(object):
-
-    def __init__(self):
-        self.names = {}
-        self.class_counts = Counter()
-
-    def get_name(self, model):
-        """
-        returns the name of the model instance. If model does not have a `name` attribute, then it will be assigned
-        a generic (and unique) identifier based on its class
-        """
-        if hasattr(model, 'name'):
-            return model.name
-        clz = model.__class__.__name__
-        if model not in self.names:
-            self.class_counts[clz] += 1
-            self.names[model] = clz + str(self.class_counts[clz])
-        return self.names[model]
-
-    def add_edge(self, f, t, graph):
-        if f: graph.add_edge(pydot.Edge(f, t))
-        return t
-
-    def add_model(self, model, graph, parent=None):
-        """
-        Recursively adds `model` and its components to the pydot graph
-        """
-        this = self.get_name(model)
-        if isinstance(model, Model):
-            parent = self.add_edge(parent, this, graph)
-            for child in reversed(model.layers):
-                parent = self.add_model(child, graph, parent)
-        elif isinstance(model, Merge):
-            for child in model.models:
-                self.add_model(child, graph, this)
-            return self.add_edge(parent, this, graph)
-        else:
-            return self.add_edge(parent, this, graph)
-
-    def plot(self, model, to_file):
-        """
-        creates a graph visualizing the structure of `model` and writes it to `to_file`
-        """
-        graph = pydot.Dot(graph_type='graph')
-        self.add_model(model, graph)
-        graph.write_png(to_file)

keras/utils/generic_utils.py

@@ -4,6 +4,7 @@ import time
 import sys
 import six
 
+
 def get_from_module(identifier, module_params, module_name, instantiate=False, kwargs=None):
     if isinstance(identifier, six.string_types):
         res = module_params.get(identifier)
@@ -17,9 +18,11 @@ def get_from_module(identifier, module_params, module_name, instantiate=False, k
             return res
     return identifier
 
+
 def make_tuple(*args):
     return args
 
+
 def printv(v, prefix=''):
     if type(v) == dict:
         if 'name' in v:
@@ -36,11 +39,12 @@ def printv(v, prefix=''):
         prefix += '...'
         for i, nv in enumerate(v):
             print(prefix + '#' + str(i))
-            printv(nv, prefix) 
+            printv(nv, prefix)
     else:
         prefix += '...'
         print(prefix + str(v))
 
+
 class Progbar(object):
     def __init__(self, target, width=30, verbose=1):
         '''
@@ -63,11 +67,11 @@ class Progbar(object):
         '''
         for k, v in values:
             if k not in self.sum_values:
-                self.sum_values[k] = [v * (current-self.seen_so_far), current-self.seen_so_far]
+                self.sum_values[k] = [v * (current - self.seen_so_far), current - self.seen_so_far]
                 self.unique_values.append(k)
             else:
-                self.sum_values[k][0] += v * (current-self.seen_so_far)
-                self.sum_values[k][1] += (current-self.seen_so_far)
+                self.sum_values[k][0] += v * (current - self.seen_so_far)
+                self.sum_values[k][1] += (current - self.seen_so_far)
         self.seen_so_far = current
 
         now = time.time()
@@ -91,7 +95,7 @@ class Progbar(object):
             bar += ']'
             sys.stdout.write(bar)
             self.total_width = len(bar)
-            
+
             if current:
                 time_per_unit = (now - self.start) / current
             else:
@@ -103,7 +107,7 @@ class Progbar(object):
             else:
                 info += ' - %ds' % (now - self.start)
             for k in self.unique_values:
-                info += ' - %s: %.4f' % (k, self.sum_values[k][0]/ max(1, self.sum_values[k][1]))
+                info += ' - %s: %.4f' % (k, self.sum_values[k][0] / max(1, self.sum_values[k][1]))
 
             self.total_width += len(info)
             if prev_total_width > self.total_width:
@@ -119,9 +123,8 @@ class Progbar(object):
             if current >= self.target:
                 info = '%ds' % (now - self.start)
                 for k in self.unique_values:
-                    info += ' - %s: %.4f' % (k, self.sum_values[k][0]/ max(1, self.sum_values[k][1]))
+                    info += ' - %s: %.4f' % (k, self.sum_values[k][0] / max(1, self.sum_values[k][1]))
                 sys.stdout.write(info + "\n")
 
-
     def add(self, n, values=[]):
         self.update(self.seen_so_far+n, values)

keras/utils/io_utils.py

@@ -3,8 +3,8 @@ import h5py
 import numpy as np
 from collections import defaultdict
 
-class HDF5Matrix:
-    
+
+class HDF5Matrix():
     refs = defaultdict(int)
 
     def __init__(self, datapath, dataset, start, end, normalizer=None):
@@ -17,7 +17,7 @@ class HDF5Matrix:
         self.end = end
         self.data = f[dataset]
         self.normalizer = normalizer
-    
+
     def __len__(self):
         return self.end - self.start
 
@@ -60,11 +60,12 @@ def save_array(array, name):
     ds[:] = array
     f.close()
 
+
 def load_array(name):
     import tables
     f = tables.open_file(name)
     array = f.root.data
-    a=np.empty(shape=array.shape, dtype=array.dtype)
-    a[:]=array[:]
+    a = np.empty(shape=array.shape, dtype=array.dtype)
+    a[:] = array[:]
     f.close()
     return a

keras/utils/layer_utils.py

@@ -1,22 +1,25 @@
+from __future__ import print_function
 import inspect
 import numpy as np
+import theano
+import copy
 
 from ..layers.advanced_activations import LeakyReLU, PReLU
-from ..layers.core import Dense, Merge, Dropout, Activation, Reshape, Flatten, RepeatVector, Layer
+from ..layers.core import Dense, Merge, Dropout, Activation, Reshape, Flatten, RepeatVector, Layer, AutoEncoder
 from ..layers.core import ActivityRegularization, TimeDistributedDense, AutoEncoder, MaxoutDense
+from ..layers.convolutional import Convolution1D, Convolution2D, MaxPooling1D, MaxPooling2D, ZeroPadding2D
 from ..layers.embeddings import Embedding, WordContextProduct
 from ..layers.noise import GaussianNoise, GaussianDropout
 from ..layers.normalization import BatchNormalization
 from ..layers.recurrent import SimpleRNN, SimpleDeepRNN, GRU, LSTM, JZS1, JZS2, JZS3
 from ..layers import containers
-
 from .. import regularizers
 from .. import constraints
 
 
-def container_from_config(layer_dict):
+def container_from_config(original_layer_dict):
+    layer_dict = copy.deepcopy(original_layer_dict)
     name = layer_dict.get('name')
-    hasParams = False
 
     if name == 'Merge':
         mode = layer_dict.get('mode')
@@ -55,34 +58,68 @@ def container_from_config(layer_dict):
             graph_layer.add_output(**output)
         return graph_layer
 
-    else: # The case in which layer_dict represents an "atomic" layer
+    elif name == 'AutoEncoder':
+        kwargs = {'encoder': container_from_config(layer_dict.get('encoder_config')),
+                  'decoder': container_from_config(layer_dict.get('decoder_config'))}
+        for kwarg in ['output_reconstruction', 'weights']:
+            if kwarg in layer_dict:
+                kwargs[kwarg] = layer_dict[kwarg]
+        return AutoEncoder(**kwargs)
+
+    else:
         layer_dict.pop('name')
-        if 'parameters' in layer_dict:
-            params = layer_dict.get('parameters')
-            layer_dict.pop('parameters')
-            hasParams = True
 
         for k, v in layer_dict.items():
-        	# For now, this can only happen for regularizers and constraints
+            # For now, this can only happen for regularizers and constraints
             if isinstance(v, dict):
                 vname = v.get('name')
                 v.pop('name')
-                if vname in [x for x,y in inspect.getmembers(constraints, predicate=inspect.isclass)]:
-                	layer_dict[k] = constraints.get(vname, v)
-                if vname in [x for x,y in inspect.getmembers(regularizers, predicate=inspect.isclass)]:
-                	layer_dict[k] = regularizers.get(vname, v)
-                
+                if vname in [x for x, y in inspect.getmembers(constraints, predicate=inspect.isclass)]:
+                    layer_dict[k] = constraints.get(vname, v)
+                if vname in [x for x, y in inspect.getmembers(regularizers, predicate=inspect.isclass)]:
+                    layer_dict[k] = regularizers.get(vname, v)
+
         base_layer = get_layer(name, layer_dict)
-        if hasParams:
-            shaped_params = []
-            for param in params:
-                data = np.asarray(param.get('data'))
-                shape = tuple(param.get('shape'))
-                shaped_params.append(data.reshape(shape))
-            base_layer.set_weights(shaped_params)
         return base_layer
 
 
+def print_layer_shapes(model, input_shapes):
+    """
+    Utility function to print the shape of the output at each layer of a Model
+
+    Arguments:
+        model: instance of Model / Merge
+        input_shapes: dict (Graph), list of tuples (Merge) or tuple (Sequential)
+    """
+    if model.__class__.__name__ in ['Sequential', 'Merge']:
+        # in this case input_shapes is a tuple, or a list [shape1, shape2]
+        if not isinstance(input_shapes[0], tuple):
+            input_shapes = [input_shapes]
+
+        inputs = model.get_input(train=False)
+        if not isinstance(inputs, list):
+            inputs = [inputs]
+        input_dummy = [np.zeros(shape, dtype=np.float32)
+                       for shape in input_shapes]
+        layers = model.layers
+
+    elif model.__class__.__name__ == 'Graph':
+        # in this case input_shapes is a dictionary
+        inputs = [model.inputs[name].input
+                  for name in model.input_order]
+        input_dummy = [np.zeros(input_shapes[name], dtype=np.float32)
+                       for name in model.input_order]
+        layers = [model.nodes[c['name']] for c in model.node_config]
+
+    print("input shapes : ", input_shapes)
+    for l in layers:
+        shape_f = theano.function(inputs, l.get_output(train=False).shape,
+                                  on_unused_input='ignore')
+        out_shape = tuple(shape_f(*input_dummy))
+        config = l.get_config()
+        print('shape after %s: %s' % (config['name'], out_shape))
+
+
 from .generic_utils import get_from_module
 def get_layer(identifier, kwargs=None):
-    return get_from_module(identifier, globals(), 'layer', instantiate=True, kwargs=kwargs)
+    return get_from_module(identifier, globals(), 'layer', instantiate=True, kwargs=kwargs)

keras/utils/np_utils.py

@@ -4,6 +4,7 @@ import scipy as sp
 from six.moves import range
 from six.moves import zip
 
+
 def to_categorical(y, nb_classes=None):
     '''Convert class vector (integers from 0 to nb_classes)
     to binary class matrix, for use with categorical_crossentropy
@@ -16,32 +17,38 @@ def to_categorical(y, nb_classes=None):
         Y[i, y[i]] = 1.
     return Y
 
+
 def normalize(a, axis=-1, order=2):
     l2 = np.atleast_1d(np.linalg.norm(a, order, axis))
-    l2[l2==0] = 1
+    l2[l2 == 0] = 1
     return a / np.expand_dims(l2, axis)
 
+
 def binary_logloss(p, y):
     epsilon = 1e-15
     p = sp.maximum(epsilon, p)
     p = sp.minimum(1-epsilon, p)
-    res = sum(y*sp.log(p) + sp.subtract(1,y)*sp.log(sp.subtract(1,p)))
+    res = sum(y * sp.log(p) + sp.subtract(1, y) * sp.log(sp.subtract(1, p)))
     res *= -1.0/len(y)
     return res
 
+
 def multiclass_logloss(P, Y):
     score = 0.
     npreds = [P[i][Y[i]-1] for i in range(len(Y))]
-    score = -(1./len(Y)) * np.sum(np.log(npreds))
+    score = -(1. / len(Y)) * np.sum(np.log(npreds))
     return score
 
+
 def accuracy(p, y):
-    return np.mean([a==b for a, b in zip(p, y)])
+    return np.mean([a == b for a, b in zip(p, y)])
+
 
 def probas_to_classes(y_pred):
     if len(y_pred.shape) > 1 and y_pred.shape[1] > 1:
         return categorical_probas_to_classes(y_pred)
     return np.array([1 if p > 0.5 else 0 for p in y_pred])
 
+
 def categorical_probas_to_classes(p):
     return np.argmax(p, axis=1)

keras/utils/test_utils.py

@@ -1,10 +1,11 @@
 import numpy as np
 
-def get_test_data(nb_train=1000, nb_test=500, input_shape=(10,), output_shape=(2,), 
-    classification=True, nb_class=2):
+
+def get_test_data(nb_train=1000, nb_test=500, input_shape=(10,), output_shape=(2,),
+                  classification=True, nb_class=2):
     '''
-        classification=True overrides output_shape 
-        (i.e. output_shape is set to (1,)) and the output 
+        classification=True overrides output_shape
+        (i.e. output_shape is set to (1,)) and the output
         consists in integers in [0, nb_class-1].
 
         Otherwise: float output with shape output_shape.

keras/utils/theano_utils.py

@@ -3,21 +3,38 @@ import numpy as np
 import theano
 import theano.tensor as T
 
+
 def floatX(X):
     return np.asarray(X, dtype=theano.config.floatX)
 
+
 def sharedX(X, dtype=theano.config.floatX, name=None):
     return theano.shared(np.asarray(X, dtype=dtype), name=name)
 
+
 def shared_zeros(shape, dtype=theano.config.floatX, name=None):
     return sharedX(np.zeros(shape), dtype=dtype, name=name)
 
+
 def shared_scalar(val=0., dtype=theano.config.floatX, name=None):
     return theano.shared(np.cast[dtype](val))
 
+
 def shared_ones(shape, dtype=theano.config.floatX, name=None):
     return sharedX(np.ones(shape), dtype=dtype, name=name)
 
+
 def alloc_zeros_matrix(*dims):
     return T.alloc(np.cast[theano.config.floatX](0.), *dims)
 
+
+def ndim_tensor(ndim):
+    if ndim == 1:
+        return T.vector()
+    elif ndim == 2:
+        return T.matrix()
+    elif ndim == 3:
+        return T.tensor3()
+    elif ndim == 4:
+        return T.tensor4()
+    return T.matrix()

keras/wrappers/scikit_learn.py

@@ -1,23 +1,46 @@
 from __future__ import absolute_import
+import abc
 import copy
 import numpy as np
 
 from ..utils.np_utils import to_categorical
 
-class KerasClassifier(object):
+
+class BaseWrapper(object):
     """
-    Implementation of the scikit-learn classifier API for Keras.
+    Base class for the Keras scikit-learn wrapper.
+
+    Warning: This class should not be used directly. Use derived classes instead.
 
     Parameters
     ----------
-    model : object
-        An un-compiled Keras model object is required to use the scikit-learn wrapper.
-    optimizer : string, optional
-        Optimization method used by the model during compilation/training.
-    loss : string, optional
-        Loss function used by the model during compilation/training.
+    train_batch_size : int, optional
+        Number of training samples evaluated at a time.
+    test_batch_size : int, optional
+        Number of test samples evaluated at a time.
+    nb_epochs : int, optional
+        Number of training epochs.
+    shuffle : boolean, optional
+        Whether to shuffle the samples at each epoch.
+    show_accuracy : boolean, optional
+        Whether to display class accuracy in the logs at each epoch.
+    validation_split : float [0, 1], optional
+        Fraction of the data to use as held-out validation data.
+    validation_data : tuple (X, y), optional
+        Data to be used as held-out validation data. Will override validation_split.
+    callbacks : list, optional
+        List of callbacks to apply during training.
+    verbose : int, optional
+        Verbosity level.
     """
-    def __init__(self, model, optimizer='adam', loss='categorical_crossentropy'):
+    __metaclass__ = abc.ABCMeta
+
+    @abc.abstractmethod
+    def __init__(self, model, optimizer, loss,
+                 train_batch_size=128, test_batch_size=128,
+                 nb_epoch=100, shuffle=True, show_accuracy=False,
+                 validation_split=0, validation_data=None, callbacks=None,
+                 verbose=0,):
         self.model = model
         self.optimizer = optimizer
         self.loss = loss
@@ -26,6 +49,17 @@ class KerasClassifier(object):
         self.config_ = []
         self.weights_ = []
 
+        self.train_batch_size = train_batch_size
+        self.test_batch_size = test_batch_size
+        self.nb_epoch = nb_epoch
+        self.shuffle = shuffle
+        self.show_accuracy = show_accuracy
+        self.validation_split = validation_split
+        self.validation_data = validation_data
+        self.callbacks = [] if callbacks is None else callbacks
+
+        self.verbose = verbose
+
     def get_params(self, deep=True):
         """
         Get parameters for this estimator.
@@ -60,7 +94,7 @@ class KerasClassifier(object):
             setattr(self, parameter, value)
         return self
 
-    def fit(self, X, y, batch_size=128, nb_epoch=100, verbose=0, shuffle=True):
+    def fit(self, X, y):
         """
         Fit the model according to the given training data.
 
@@ -75,19 +109,11 @@ class KerasClassifier(object):
             and n_features is the number of features.
         y : array-like, shape = (n_samples) or (n_samples, n_outputs)
             True labels for X.
-        batch_size : int, optional
-            Number of training samples evaluated at a time.
-        nb_epochs : int, optional
-            Number of training epochs.
-        verbose : int, optional
-            Verbosity level.
-        shuffle : boolean, optional
-            Indicator to shuffle the training data.
 
         Returns
         -------
-        self : object
-            Returns self.
+        history : object
+            Returns details about the training history at each epoch.
         """
         if len(y.shape) == 1:
             self.classes_ = list(np.unique(y))
@@ -98,13 +124,71 @@ class KerasClassifier(object):
 
         self.compiled_model_ = copy.deepcopy(self.model)
         self.compiled_model_.compile(optimizer=self.optimizer, loss=self.loss)
-        self.compiled_model_.fit(X, y, batch_size=batch_size, nb_epoch=nb_epoch, verbose=verbose, shuffle=shuffle)
+        history = self.compiled_model_.fit(
+            X, y, batch_size=self.train_batch_size, nb_epoch=self.nb_epoch, verbose=self.verbose,
+            shuffle=self.shuffle, show_accuracy=self.show_accuracy,
+            validation_split=self.validation_split, validation_data=self.validation_data,
+            callbacks=self.callbacks)
+
         self.config_ = self.model.get_config()
         self.weights_ = self.model.get_weights()
 
-        return self
+        return history
 
-    def score(self, X, y, batch_size=128, verbose=0):
+
+class KerasClassifier(BaseWrapper):
+    """
+    Implementation of the scikit-learn classifier API for Keras.
+
+    Parameters
+    ----------
+    model : object
+        An un-compiled Keras model object is required to use the scikit-learn wrapper.
+    optimizer : string
+        Optimization method used by the model during compilation/training.
+    loss : string
+        Loss function used by the model during compilation/training.
+    """
+    def __init__(self, model, optimizer='adam', loss='categorical_crossentropy', **kwargs):
+        super(KerasClassifier, self).__init__(model, optimizer, loss, **kwargs)
+
+    def predict(self, X):
+        """
+        Returns the class predictions for the given test data.
+
+        Parameters
+        ----------
+        X : array-like, shape = (n_samples, n_features)
+            Test samples where n_samples in the number of samples
+            and n_features is the number of features.
+
+        Returns
+        -------
+        preds : array-like, shape = (n_samples)
+            Class predictions.
+        """
+        return self.compiled_model_.predict_classes(
+            X, batch_size=self.test_batch_size, verbose=self.verbose)
+
+    def predict_proba(self, X):
+        """
+        Returns class probability estimates for the given test data.
+
+        Parameters
+        ----------
+        X : array-like, shape = (n_samples, n_features)
+            Test samples where n_samples in the number of samples
+            and n_features is the number of features.
+
+        Returns
+        -------
+        proba : array-like, shape = (n_samples, n_outputs)
+            Class probability estimates.
+        """
+        return self.compiled_model_.predict_proba(
+            X, batch_size=self.test_batch_size, verbose=self.verbose)
+
+    def score(self, X, y):
         """
         Returns the mean accuracy on the given test data and labels.
 
@@ -115,58 +199,68 @@ class KerasClassifier(object):
             and n_features is the number of features.
         y : array-like, shape = (n_samples) or (n_samples, n_outputs)
             True labels for X.
-        batch_size : int, optional
-            Number of test samples evaluated at a time.
-        verbose : int, optional
-            Verbosity level.
 
         Returns
         -------
         score : float
-            Mean accuracy of self.predict(X) wrt. y.
+            Mean accuracy of predictions on X wrt. y.
         """
-        loss, accuracy = self.compiled_model_.evaluate(X, y, batch_size=batch_size,
-                                                       show_accuracy=True, verbose=verbose)
+        loss, accuracy = self.compiled_model_.evaluate(
+            X, y, batch_size=self.test_batch_size, show_accuracy=True, verbose=self.verbose)
         return accuracy
 
-    def predict(self, X, batch_size=128, verbose=0):
+
+class KerasRegressor(BaseWrapper):
+    """
+    Implementation of the scikit-learn regressor API for Keras.
+
+    Parameters
+    ----------
+    model : object
+        An un-compiled Keras model object is required to use the scikit-learn wrapper.
+    optimizer : string
+        Optimization method used by the model during compilation/training.
+    loss : string
+        Loss function used by the model during compilation/training.
+    """
+    def __init__(self, model, optimizer='adam', loss='mean_squared_error', **kwargs):
+        super(KerasRegressor, self).__init__(model, optimizer, loss, **kwargs)
+
+    def predict(self, X):
         """
-        Returns the class predictions for the given test data.
+        Returns predictions for the given test data.
 
         Parameters
         ----------
         X : array-like, shape = (n_samples, n_features)
             Test samples where n_samples in the number of samples
             and n_features is the number of features.
-        batch_size : int, optional
-            Number of test samples evaluated at a time.
-        verbose : int, optional
-            Verbosity level.
 
         Returns
         -------
         preds : array-like, shape = (n_samples)
-            Class predictions.
+            Predictions.
         """
-        return self.compiled_model_.predict_classes(X, batch_size=batch_size, verbose=verbose)
+        return self.compiled_model_.predict(
+            X, batch_size=self.test_batch_size, verbose=self.verbose).ravel()
 
-    def predict_proba(self, X, batch_size=128, verbose=0):
+    def score(self, X, y):
         """
-        Returns class probability estimates for the given test data.
+        Returns the mean accuracy on the given test data and labels.
 
         Parameters
         ----------
         X : array-like, shape = (n_samples, n_features)
             Test samples where n_samples in the number of samples
             and n_features is the number of features.
-        batch_size : int, optional
-            Number of test samples evaluated at a time.
-        verbose : int, optional
-            Verbosity level.
+        y : array-like, shape = (n_samples)
+            True labels for X.
 
         Returns
         -------
-        proba : array-like, shape = (n_samples, n_outputs)
-            Class probability estimates.
+        score : float
+            Loss from predictions on X wrt. y.
         """
-        return self.compiled_model_.predict_proba(X, batch_size=batch_size, verbose=verbose)
+        loss = self.compiled_model_.evaluate(
+            X, y, batch_size=self.test_batch_size, show_accuracy=False, verbose=self.verbose)
+        return loss

setup.py

@@ -1,14 +1,14 @@
 from setuptools import setup
 from setuptools import find_packages
 
-setup(name = 'Keras',
-      version = '0.1.1',
-      description = 'Theano-based Deep Learning library',
-      author = 'Francois Chollet',
-      author_email = 'francois.chollet@gmail.com',
-      url = 'https://github.com/fchollet/keras',
-      download_url = 'https://github.com/fchollet/keras/tarball/0.1.1',
-      license = 'MIT',
-      install_requires = ['theano', 'pyyaml'],
-      packages = find_packages(),
-)
+
+setup(name='Keras',
+      version='0.1.2',
+      description='Theano-based Deep Learning library',
+      author='Francois Chollet',
+      author_email='francois.chollet@gmail.com',
+      url='https://github.com/fchollet/keras',
+      download_url='https://github.com/fchollet/keras/tarball/0.1.2',
+      license='MIT',
+      install_requires=['theano', 'pyyaml', 'h5py'],
+      packages=find_packages())

tests/auto/keras/layers/test_core.py

@@ -0,0 +1,162 @@
+import unittest
+import numpy as np
+from numpy.testing import assert_allclose
+import theano
+
+from keras.layers import core
+
+
+class TestLayerBase(unittest.TestCase):
+    def test_input_output(self):
+        nb_samples = 10
+        input_dim = 5
+        layer = core.Layer()
+
+        # As long as there is no input, an error should be raised.
+        for train in [True, False]:
+            self.assertRaises(AttributeError, layer.get_input, train)
+            self.assertRaises(AttributeError, layer.get_output, train)
+
+        # Once an input is provided, it should be reachable through the
+        # appropriate getters
+        input = np.ones((nb_samples, input_dim))
+        layer.input = theano.shared(value=input)
+        for train in [True, False]:
+            assert_allclose(layer.get_input(train).eval(), input)
+            assert_allclose(layer.get_output(train).eval(), input)
+
+    def test_connections(self):
+        nb_samples = 10
+        input_dim = 5
+        layer1 = core.Layer()
+        layer2 = core.Layer()
+
+        input = np.ones((nb_samples, input_dim))
+        layer1.input = theano.shared(value=input)
+
+        # As long as there is no previous layer, an error should be raised.
+        for train in [True, False]:
+            self.assertRaises(AttributeError, layer2.get_input, train)
+
+        # After connecting, input of layer1 should be passed through
+        layer2.set_previous(layer1)
+        for train in [True, False]:
+            assert_allclose(layer2.get_input(train).eval(), input)
+            assert_allclose(layer2.get_output(train).eval(), input)
+
+
+class TestConfigParams(unittest.TestCase):
+    """
+    Test the constructor, config and params functions of all layers in core.
+    """
+
+    def _runner(self, layer):
+        conf = layer.get_config()
+        assert (type(conf) == dict)
+
+        param = layer.get_params()
+        # Typically a list or a tuple, but may be any iterable
+        assert hasattr(param, '__iter__')
+
+    def test_base(self):
+        layer = core.Layer()
+        self._runner(layer)
+
+    def test_masked(self):
+        layer = core.MaskedLayer()
+        self._runner(layer)
+
+    def test_merge(self):
+        layer_1 = core.Layer()
+        layer_2 = core.Layer()
+        layer = core.Merge([layer_1, layer_2])
+        self._runner(layer)
+
+    def test_dropout(self):
+        layer = core.Dropout(0.5)
+        self._runner(layer)
+
+    def test_activation(self):
+        layer = core.Activation('linear')
+        self._runner(layer)
+
+    def test_reshape(self):
+        layer = core.Reshape(10, 10)
+        self._runner(layer)
+
+    def test_flatten(self):
+        layer = core.Flatten()
+        self._runner(layer)
+
+    def test_repeat_vector(self):
+        layer = core.RepeatVector(10)
+        self._runner(layer)
+
+    def test_dense(self):
+        layer = core.Dense(10, 10)
+        self._runner(layer)
+
+    def test_act_reg(self):
+        layer = core.ActivityRegularization(0.5, 0.5)
+        self._runner(layer)
+
+    def test_time_dist_dense(self):
+        layer = core.TimeDistributedDense(10, 10)
+        self._runner(layer)
+
+    def test_autoencoder(self):
+        layer_1 = core.Layer()
+        layer_2 = core.Layer()
+
+        layer = core.AutoEncoder(layer_1, layer_2)
+        self._runner(layer)
+
+    def test_maxout_dense(self):
+        layer = core.MaxoutDense(10, 10)
+        self._runner(layer)
+
+
+class TestMasking(unittest.TestCase):
+    """Test the Masking class"""
+
+    def test_sequences(self):
+        """Test masking sequences with zeroes as padding"""
+        # integer inputs, one per timestep, like embeddings
+        layer = core.Masking()
+        func = theano.function([layer.input], layer.get_output_mask())
+        self.assertTrue(np.all(
+            # get mask for this input
+            func(np.array(
+            [[[1], [2], [3], [0]],
+             [[0], [4], [5], [0]]], dtype=np.int32)) ==
+            # This is the expected output mask, one dimension less
+            np.array([[1, 1, 1, 0], [0, 1, 1, 0]])))
+
+    def test_non_zero(self):
+        """Test masking with non-zero mask value"""
+        layer = core.Masking(5)
+        func = theano.function([layer.input], layer.get_output_mask())
+        self.assertTrue(np.all(
+            # get mask for this input, if not all the values are 5, shouldn't masked
+            func(np.array(
+            [[[1, 1], [2, 1], [3, 1], [5, 5]],
+             [[1, 5], [5, 0], [0, 0], [0, 0]]], dtype=np.int32)) ==
+            # This is the expected output mask, one dimension less
+            np.array([[1, 1, 1, 0], [1, 1, 1, 1]])))
+
+    def test_non_zero_output(self):
+        """Test output of masking layer with non-zero mask value"""
+        layer = core.Masking(5)
+        func = theano.function([layer.input], layer.get_output())
+        self.assertTrue(np.all(
+            # get output for this input, replace padding with 0
+            func(np.array(
+            [[[1, 1], [2, 1], [3, 1], [5, 5]],
+             [[1, 5], [5, 0], [0, 0], [0, 0]]], dtype=np.int32)) ==
+            # This is the expected output
+            np.array([[[1, 1], [2, 1], [3, 1], [0, 0]],
+             [[1, 5], [5, 0], [0, 0], [0, 0]]])))
+
+
+if __name__ == '__main__':
+    unittest.main()

tests/auto/keras/layers/test_recurrent.py

@@ -0,0 +1,59 @@
+import unittest
+import numpy as np
+import theano
+
+from keras.layers import recurrent
+
+nb_samples, timesteps, input_dim, output_dim = 3, 3, 10, 5
+
+
+def _runner(layer_class):
+    """
+    All the recurrent layers share the same interface, so we can run through them with a single
+    function.
+    """
+    for weights in [None, [np.ones((input_dim, output_dim))]]:
+        for ret_seq in [True, False]:
+            layer = layer_class(input_dim, output_dim, return_sequences=ret_seq, weights=weights)
+            layer.input = theano.shared(value=np.ones((nb_samples, timesteps, input_dim)))
+            config = layer.get_config()
+
+            for train in [True, False]:
+                out = layer.get_output(train).eval()
+                # Make sure the output has the desired shape
+                if ret_seq:
+                    assert(out.shape == (nb_samples, timesteps, output_dim))
+                else:
+                    assert(out.shape == (nb_samples, output_dim))
+
+                mask = layer.get_output_mask(train)
+
+
+class TestRNNS(unittest.TestCase):
+    """
+    Test all the RNNs using a generic test runner function defined above.
+    """
+    def test_simple(self):
+        _runner(recurrent.SimpleRNN)
+
+    def test_simple_deep(self):
+        _runner(recurrent.SimpleDeepRNN)
+
+    def test_gru(self):
+        _runner(recurrent.GRU)
+
+    def test_lstm(self):
+        _runner(recurrent.LSTM)
+
+    def test_jzs1(self):
+        _runner(recurrent.JZS1)
+
+    def test_jzs2(self):
+        _runner(recurrent.JZS2)
+
+    def test_jzs3(self):
+        _runner(recurrent.JZS3)
+
+
+if __name__ == '__main__':
+    unittest.main()

tests/auto/keras/test_normalization.py

@@ -0,0 +1,99 @@
+import unittest
+import numpy as np
+from numpy.testing import assert_allclose
+from theano import tensor as T
+from keras.layers import normalization
+from keras.models import Sequential
+
+
+class TestBatchNormalization(unittest.TestCase):
+    def setUp(self):
+        self.input_1 = np.arange(10)
+        self.input_2 = np.zeros(10)
+        self.input_3 = np.ones((10))
+
+        self.input_shapes = [np.ones((10, 10)), np.ones((10, 10, 10))]
+
+    def test_setup(self):
+        norm_m0 = normalization.BatchNormalization((10, 10))
+        norm_m1 = normalization.BatchNormalization((10, 10), mode=1)
+
+        # mode 3 does not exist
+        self.assertRaises(Exception, normalization.BatchNormalization((10, 10), mode=3))
+
+    def test_mode_0(self):
+        model = Sequential()
+        norm_m0 = normalization.BatchNormalization((10,))
+        model.add(norm_m0)
+        model.compile(loss='mse', optimizer='sgd')
+
+        # centered on 5.0, variance 10.0
+        X = np.random.normal(loc=5.0, scale=10.0, size=(1000, 10))
+        model.fit(X, X, nb_epoch=5, verbose=0)
+        norm_m0.input = X
+        out = (norm_m0.get_output(train=True) - norm_m0.beta) / norm_m0.gamma
+
+        self.assertAlmostEqual(out.mean().eval(), 0.0, places=1)
+        self.assertAlmostEqual(out.std().eval(), 1.0, places=1)
+
+    def test_mode_1(self):
+        norm_m1 = normalization.BatchNormalization((10,), mode=1)
+        norm_m1.init_updates()
+
+        for inp in [self.input_1, self.input_2, self.input_3]:
+            norm_m1.input = inp
+            out = (norm_m1.get_output(train=True) - norm_m1.beta) / norm_m1.gamma
+            self.assertAlmostEqual(out.mean().eval(), 0.0)
+            if inp.std() > 0.:
+                self.assertAlmostEqual(out.std().eval(), 1.0, places=2)
+            else:
+                self.assertAlmostEqual(out.std().eval(), 0.0, places=2)
+
+    def test_shapes(self):
+        """
+        Test batch normalization with various input shapes
+        """
+        for inp in self.input_shapes:
+            norm_m0 = normalization.BatchNormalization(inp.shape, mode=0)
+            norm_m0.init_updates()
+            norm_m0.input = inp
+            out = (norm_m0.get_output(train=True) - norm_m0.beta) / norm_m0.gamma
+
+            norm_m1 = normalization.BatchNormalization(inp.shape, mode=1)
+            norm_m1.input = inp
+            out = (norm_m1.get_output(train=True) - norm_m1.beta) / norm_m1.gamma
+
+    def test_weight_init(self):
+        """
+        Test weight initialization
+        """
+
+        norm_m1 = normalization.BatchNormalization((10,), mode=1, weights=[np.ones(10), np.ones(10)])
+        norm_m1.init_updates()
+
+        for inp in [self.input_1, self.input_2, self.input_3]:
+            norm_m1.input = inp
+            out = (norm_m1.get_output(train=True) - np.ones(10)) / 1.
+            self.assertAlmostEqual(out.mean().eval(), 0.0)
+            if inp.std() > 0.:
+                self.assertAlmostEqual(out.std().eval(), 1.0, places=2)
+            else:
+                self.assertAlmostEqual(out.std().eval(), 0.0, places=2)
+
+        assert_allclose(norm_m1.gamma.eval(), np.ones(10))
+        assert_allclose(norm_m1.beta.eval(), np.ones(10))
+
+        # Weights must be an iterable of gamma AND beta.
+        self.assertRaises(Exception, normalization.BatchNormalization((10,)), weights=np.ones(10))
+
+    def test_config(self):
+        norm = normalization.BatchNormalization((10, 10), mode=1, epsilon=0.1)
+        conf = norm.get_config()
+        conf_target = {"input_shape": (10, 10), "name": normalization.BatchNormalization.__name__,
+                       "epsilon": 0.1, "mode": 1}
+
+        self.assertDictEqual(conf, conf_target)
+
+
+if __name__ == '__main__':
+    unittest.main()

tests/auto/test_datasets.py

@@ -0,0 +1,48 @@
+from __future__ import print_function
+import unittest
+from keras.datasets import cifar10, cifar100, reuters, imdb, mnist
+
+
+class TestDatasets(unittest.TestCase):
+    def test_cifar(self):
+        print('cifar10')
+        (X_train, y_train), (X_test, y_test) = cifar10.load_data()
+        print(X_train.shape)
+        print(X_test.shape)
+        print(y_train.shape)
+        print(y_test.shape)
+
+        print('cifar100 fine')
+        (X_train, y_train), (X_test, y_test) = cifar100.load_data('fine')
+        print(X_train.shape)
+        print(X_test.shape)
+        print(y_train.shape)
+        print(y_test.shape)
+
+        print('cifar100 coarse')
+        (X_train, y_train), (X_test, y_test) = cifar100.load_data('coarse')
+        print(X_train.shape)
+        print(X_test.shape)
+        print(y_train.shape)
+        print(y_test.shape)
+
+    def test_reuters(self):
+        print('reuters')
+        (X_train, y_train), (X_test, y_test) = reuters.load_data()
+
+    def test_mnist(self):
+        print('mnist')
+        (X_train, y_train), (X_test, y_test) = mnist.load_data()
+        print(X_train.shape)
+        print(X_test.shape)
+        print(y_train.shape)
+        print(y_test.shape)
+
+    def test_imdb(self):
+        print('imdb')
+        (X_train, y_train), (X_test, y_test) = imdb.load_data()
+
+
+if __name__ == '__main__':
+    print('Test datasets')
+    unittest.main()

tests/auto/test_embeddings.py

@@ -6,8 +6,8 @@ from keras.layers.embeddings import Embedding
 from theano import function
 from keras.constraints import unitnorm
 
-class TestConcatenation(unittest.TestCase):
 
+class TestEmbedding(unittest.TestCase):
     def setUp(self):
         self.X1 = np.array([[1], [2]], dtype='int32')
         self.W1 = np.array([[0.1, 0.2], [0.3, 0.4], [0.5, 0.6]], dtype='float32')
@@ -19,9 +19,9 @@ class TestConcatenation(unittest.TestCase):
         lookup.add(Dense(2, 1))
         lookup.add(Activation('sigmoid'))
         lookup.compile(loss='binary_crossentropy', optimizer='sgd', class_mode='binary')
-        lookup.train(self.X1, np.array([[1], [0]], dtype='int32'))
+        lookup.train_on_batch(self.X1, np.array([[1], [0]], dtype='int32'))
         norm = np.linalg.norm(lookup.params[0].get_value(), axis=1)
         self.assertTrue(np.allclose(norm, np.ones_like(norm).astype('float32')))
 
 if __name__ == '__main__':
-    unittest.main()
+    unittest.main()

tests/auto/test_graph_model.py

@@ -2,6 +2,7 @@ from __future__ import print_function
 import unittest
 import numpy as np
 np.random.seed(1337)
+
 from keras.models import Graph, Sequential
 from keras.layers import containers
 from keras.layers.core import Dense, Activation
@@ -13,9 +14,10 @@ y = np.random.random((100, 4))
 y2 = np.random.random((100,))
 
 (X_train, y_train), (X_test, y_test) = get_test_data(nb_train=1000, nb_test=200, input_shape=(32,),
-    classification=False, output_shape=(4,))
+                                                     classification=False, output_shape=(4,))
 (X2_train, y2_train), (X2_test, y2_test) = get_test_data(nb_train=1000, nb_test=200, input_shape=(32,),
-    classification=False, output_shape=(1,))
+                                                         classification=False, output_shape=(1,))
+
 
 class TestGraph(unittest.TestCase):
     def test_1o_1i(self):
@@ -28,15 +30,15 @@ class TestGraph(unittest.TestCase):
         graph.add_node(Dense(16, 4), name='dense3', input='dense1')
 
         graph.add_output(name='output1', inputs=['dense2', 'dense3'], merge_mode='sum')
-        graph.compile('rmsprop', {'output1':'mse'})
+        graph.compile('rmsprop', {'output1': 'mse'})
 
-        history = graph.fit({'input1':X_train, 'output1':y_train}, nb_epoch=10)
-        out = graph.predict({'input1':X_test})
+        history = graph.fit({'input1': X_train, 'output1': y_train}, nb_epoch=10)
+        out = graph.predict({'input1': X_test})
         assert(type(out == dict))
         assert(len(out) == 1)
-        loss = graph.test_on_batch({'input1':X_test, 'output1':y_test})
-        loss = graph.train_on_batch({'input1':X_test, 'output1':y_test})
-        loss = graph.evaluate({'input1':X_test, 'output1':y_test})
+        loss = graph.test_on_batch({'input1': X_test, 'output1': y_test})
+        loss = graph.train_on_batch({'input1': X_test, 'output1': y_test})
+        loss = graph.evaluate({'input1': X_test, 'output1': y_test})
         print(loss)
         assert(loss < 2.5)
 
@@ -53,15 +55,15 @@ class TestGraph(unittest.TestCase):
         graph.add_node(Dense(16, 4), name='dense4', inputs=['dense1', 'dense3'], merge_mode='sum')
 
         graph.add_output(name='output1', inputs=['dense2', 'dense4'], merge_mode='sum')
-        graph.compile('rmsprop', {'output1':'mse'})
+        graph.compile('rmsprop', {'output1': 'mse'})
 
-        history = graph.fit({'input1':X_train, 'output1':y_train}, nb_epoch=10)
-        out = graph.predict({'input1':X_train})
+        history = graph.fit({'input1': X_train, 'output1': y_train}, nb_epoch=10)
+        out = graph.predict({'input1': X_train})
         assert(type(out == dict))
         assert(len(out) == 1)
-        loss = graph.test_on_batch({'input1':X_test, 'output1':y_test})
-        loss = graph.train_on_batch({'input1':X_test, 'output1':y_test})
-        loss = graph.evaluate({'input1':X_test, 'output1':y_test})
+        loss = graph.test_on_batch({'input1': X_test, 'output1': y_test})
+        loss = graph.train_on_batch({'input1': X_test, 'output1': y_test})
+        loss = graph.evaluate({'input1': X_test, 'output1': y_test})
         print(loss)
         assert(loss < 2.5)
         graph.get_config(verbose=1)
@@ -77,15 +79,15 @@ class TestGraph(unittest.TestCase):
         graph.add_node(Dense(16, 4), name='dense3', input='dense1')
 
         graph.add_output(name='output1', inputs=['dense2', 'dense3'], merge_mode='sum')
-        graph.compile('rmsprop', {'output1':'mse'})
+        graph.compile('rmsprop', {'output1': 'mse'})
 
-        history = graph.fit({'input1':X_train, 'input2':X2_train, 'output1':y_train}, nb_epoch=10)
-        out = graph.predict({'input1':X_test, 'input2':X2_test})
+        history = graph.fit({'input1': X_train, 'input2': X2_train, 'output1': y_train}, nb_epoch=10)
+        out = graph.predict({'input1': X_test, 'input2': X2_test})
         assert(type(out == dict))
         assert(len(out) == 1)
-        loss = graph.test_on_batch({'input1':X_test, 'input2':X2_test, 'output1':y_test})
-        loss = graph.train_on_batch({'input1':X_test, 'input2':X2_test, 'output1':y_test})
-        loss = graph.evaluate({'input1':X_test, 'input2':X2_test, 'output1':y_test})
+        loss = graph.test_on_batch({'input1': X_test, 'input2': X2_test, 'output1': y_test})
+        loss = graph.train_on_batch({'input1': X_test, 'input2': X2_test, 'output1': y_test})
+        loss = graph.evaluate({'input1': X_test, 'input2': X2_test, 'output1': y_test})
         print(loss)
         assert(loss < 3.0)
         graph.get_config(verbose=1)
@@ -101,15 +103,15 @@ class TestGraph(unittest.TestCase):
 
         graph.add_output(name='output1', input='dense2')
         graph.add_output(name='output2', input='dense3')
-        graph.compile('rmsprop', {'output1':'mse', 'output2':'mse'})
+        graph.compile('rmsprop', {'output1': 'mse', 'output2': 'mse'})
 
-        history = graph.fit({'input1':X_train, 'output1':y_train, 'output2':y2_train}, nb_epoch=10)
-        out = graph.predict({'input1':X_test})
+        history = graph.fit({'input1': X_train, 'output1': y_train, 'output2': y2_train}, nb_epoch=10)
+        out = graph.predict({'input1': X_test})
         assert(type(out == dict))
         assert(len(out) == 2)
-        loss = graph.test_on_batch({'input1':X_test, 'output1':y_test, 'output2':y2_test})
-        loss = graph.train_on_batch({'input1':X_test, 'output1':y_test, 'output2':y2_test})
-        loss = graph.evaluate({'input1':X_test, 'output1':y_test, 'output2':y2_test})
+        loss = graph.test_on_batch({'input1': X_test, 'output1': y_test, 'output2': y2_test})
+        loss = graph.train_on_batch({'input1': X_test, 'output1': y_test, 'output2': y2_test})
+        loss = graph.evaluate({'input1': X_test, 'output1': y_test, 'output2': y2_test})
         print(loss)
         assert(loss < 4.)
 
@@ -122,12 +124,44 @@ class TestGraph(unittest.TestCase):
         graph.add_node(Dense(16, 1), name='dense3', input='dense1')
         graph.add_output(name='output1', input='dense2')
         graph.add_output(name='output2', input='dense3')
-        graph.compile('rmsprop', {'output1':'mse', 'output2':'mse'})
+        graph.compile('rmsprop', {'output1': 'mse', 'output2': 'mse'})
         graph.load_weights('temp.h5')
-        nloss = graph.evaluate({'input1':X_test, 'output1':y_test, 'output2':y2_test})
+        nloss = graph.evaluate({'input1': X_test, 'output1': y_test, 'output2': y2_test})
         print(nloss)
         assert(loss == nloss)
 
+    def test_2o_1i_sample_weights(self):
+        print('test a non-sequential graph with 1 input and 2 outputs with sample weights')
+        graph = Graph()
+        graph.add_input(name='input1', ndim=2)
+
+        graph.add_node(Dense(32, 16), name='dense1', input='input1')
+        graph.add_node(Dense(32, 4), name='dense2', input='input1')
+        graph.add_node(Dense(16, 1), name='dense3', input='dense1')
+
+        graph.add_output(name='output1', input='dense2')
+        graph.add_output(name='output2', input='dense3')
+
+        weights1 = np.random.uniform(size=y_train.shape[0])
+        weights2 = np.random.uniform(size=y2_train.shape[0])
+        weights1_test = np.random.uniform(size=y_test.shape[0])
+        weights2_test = np.random.uniform(size=y2_test.shape[0])
+
+        graph.compile('rmsprop', {'output1': 'mse', 'output2': 'mse'})
+
+        history = graph.fit({'input1': X_train, 'output1': y_train, 'output2': y2_train}, nb_epoch=10,
+                            sample_weight={'output1': weights1, 'output2': weights2})
+        out = graph.predict({'input1': X_test})
+        assert(type(out == dict))
+        assert(len(out) == 2)
+        loss = graph.test_on_batch({'input1': X_test, 'output1': y_test, 'output2': y2_test},
+                                   sample_weight={'output1': weights1_test, 'output2': weights2_test})
+        loss = graph.train_on_batch({'input1': X_train, 'output1': y_train, 'output2': y2_train},
+                                    sample_weight={'output1': weights1, 'output2': weights2})
+        loss = graph.evaluate({'input1': X_train, 'output1': y_train, 'output2': y2_train},
+                              sample_weight={'output1': weights1, 'output2': weights2})
+        print(loss)
+
     def test_recursive(self):
         print('test layer-like API')
 
@@ -154,7 +188,28 @@ class TestGraph(unittest.TestCase):
         pred = seq.predict(X_test)
         seq.get_config(verbose=1)
 
+    def test_create_output(self):
+        print('test create_output argument')
+        graph = Graph()
+        graph.add_input(name='input1', ndim=2)
+
+        graph.add_node(Dense(32, 16), name='dense1', input='input1')
+        graph.add_node(Dense(32, 4), name='dense2', input='input1')
+        graph.add_node(Dense(16, 4), name='dense3', input='dense1')
+        graph.add_node(Dense(4, 4), name='output1', inputs=['dense2', 'dense3'], merge_mode='sum', create_output=True)
+        graph.compile('rmsprop', {'output1': 'mse'})
+
+        history = graph.fit({'input1': X_train, 'output1': y_train}, nb_epoch=10)
+        out = graph.predict({'input1': X_test})
+        assert(type(out == dict))
+        assert(len(out) == 1)
+        loss = graph.test_on_batch({'input1': X_test, 'output1': y_test})
+        loss = graph.train_on_batch({'input1': X_test, 'output1': y_test})
+        loss = graph.evaluate({'input1': X_test, 'output1': y_test})
+        print(loss)
+        assert(loss < 2.5)
+
 
 if __name__ == '__main__':
     print('Test graph model')
-    unittest.main()
+    unittest.main()

tests/auto/test_loss_masking.py

@@ -0,0 +1,46 @@
+import numpy as np
+np.random.seed(1337)
+
+import unittest
+from keras.models import Sequential, weighted_objective
+from keras.layers.core import TimeDistributedDense, Masking
+from keras import objectives
+import theano
+
+
+class TestLossMasking(unittest.TestCase):
+    def test_loss_masking(self):
+        X = np.array(
+            [[[1, 1], [2, 1], [3, 1], [5, 5]],
+             [[1, 5], [5, 0], [0, 0], [0, 0]]], dtype=np.int32)
+        model = Sequential()
+        model.add(Masking(mask_value=0))
+        model.add(TimeDistributedDense(2, 1, init='one'))
+        model.compile(loss='mse', optimizer='sgd')
+        y = model.predict(X)
+        loss = model.fit(X, 4*y, nb_epoch=1, batch_size=2, verbose=1).history['loss'][0]
+        assert loss == 285.
+
+    def test_loss_masking_time(self):
+        theano.config.mode = 'FAST_COMPILE'
+        weighted_loss = weighted_objective(objectives.get('categorical_crossentropy'))
+        shape = (3, 4, 2)
+        X = np.arange(24).reshape(shape)
+        Y = 2 * X
+
+        weights = np.ones((3, 4, 1))  # Normally the trailing 1 is added by standardize_weights
+        weights[0, 0] = 0
+        mask = np.ones((3, 4))
+        mask[1, 0] = 0
+
+        out = weighted_loss(X, Y, weights, mask).eval()
+        weights[0, 0] = 1e-9  # so that nonzero() doesn't remove this weight
+        out2 = weighted_loss(X, Y, weights, mask).eval()
+        print(out)
+        print(out2)
+        assert abs(out - out2) < 1e-8
+
+
+if __name__ == '__main__':
+    print('Test loss masking')
+    unittest.main()

tests/auto/test_loss_weighting.py

@@ -1,23 +1,24 @@
 from __future__ import absolute_import
 from __future__ import print_function
+
+import numpy as np
+np.random.seed(1336)  # for reproducibility
+
 from keras.datasets import mnist
-from keras.models import Sequential
+from keras.models import Sequential, Graph
 from keras.layers.core import Dense, Activation
 from keras.utils import np_utils
-import numpy as np
 import unittest
 
 nb_classes = 10
 batch_size = 128
-nb_epoch = 5
+nb_epoch = 8
 weighted_class = 9
 standard_weight = 1
 high_weight = 5
 max_train_samples = 5000
 max_test_samples = 1000
 
-np.random.seed(1337) # for reproducibility
-
 # the data, shuffled and split between tran and test sets
 (X_train, y_train), (X_test, y_test) = mnist.load_data()
 X_train = X_train.reshape(60000, 784)[:max_train_samples]
@@ -32,7 +33,14 @@ Y_train = np_utils.to_categorical(y_train, nb_classes)
 Y_test = np_utils.to_categorical(y_test, nb_classes)
 test_ids = np.where(y_test == np.array(weighted_class))[0]
 
-def create_model():
+class_weight = dict([(i, standard_weight) for i in range(nb_classes)])
+class_weight[weighted_class] = high_weight
+
+sample_weight = np.ones((y_train.shape[0])) * standard_weight
+sample_weight[y_train == weighted_class] = high_weight
+
+
+def create_sequential_model():
     model = Sequential()
     model.add(Dense(784, 50))
     model.add(Activation('relu'))
@@ -40,40 +48,96 @@ def create_model():
     model.add(Activation('softmax'))
     return model
 
-def _test_weights(model, class_weight=None, sample_weight=None):
-    model.fit(X_train, Y_train, batch_size=batch_size, nb_epoch=nb_epoch, verbose=0, \
-        class_weight=class_weight, sample_weight=sample_weight)
+
+def create_graph_model():
+    model = Graph()
+    model.add_input(name='input')
+    model.add_node(Dense(784, 50, activation='relu'), name='d1', input='input')
+    model.add_node(Dense(50, 10, activation='softmax'), name='d2', input='d1')
+    model.add_output(name='output', input='d2')
+    return model
+
+
+def _test_weights_sequential(model, class_weight=None, sample_weight=None):
+    if sample_weight is not None:
+        model.fit(X_train, Y_train, batch_size=batch_size, nb_epoch=nb_epoch // 3, verbose=0,
+                  class_weight=class_weight, sample_weight=sample_weight)
+        model.fit(X_train, Y_train, batch_size=batch_size, nb_epoch=nb_epoch // 3, verbose=0,
+                  class_weight=class_weight, sample_weight=sample_weight, validation_split=0.1)
+        model.fit(X_train, Y_train, batch_size=batch_size, nb_epoch=nb_epoch // 3, verbose=0,
+                  class_weight=class_weight, sample_weight=sample_weight, validation_data=(X_train, Y_train, sample_weight))
+    else:
+        model.fit(X_train, Y_train, batch_size=batch_size, nb_epoch=nb_epoch // 2, verbose=0,
+                  class_weight=class_weight, sample_weight=sample_weight)
+        model.fit(X_train, Y_train, batch_size=batch_size, nb_epoch=nb_epoch // 2, verbose=0,
+                  class_weight=class_weight, sample_weight=sample_weight, validation_split=0.1)
+
+    model.train_on_batch(X_train[:32], Y_train[:32],
+                         class_weight=class_weight, sample_weight=sample_weight[:32] if sample_weight is not None else None)
+    model.test_on_batch(X_train[:32], Y_train[:32],
+                        sample_weight=sample_weight[:32] if sample_weight is not None else None)
     score = model.evaluate(X_test[test_ids, :], Y_test[test_ids, :], verbose=0)
     return score
 
-class TestConcatenation(unittest.TestCase):
 
-    def test_loss_weighting(self):
-        class_weight = dict([(i, standard_weight) for i in range(nb_classes)])
-        class_weight[weighted_class] = high_weight
+def _test_weights_graph(model, class_weight=None, sample_weight=None):
+    model.fit({'input': X_train, 'output': Y_train}, batch_size=batch_size, nb_epoch=nb_epoch // 2, verbose=0,
+              class_weight={'output': class_weight}, sample_weight={'output': sample_weight})
+    model.fit({'input': X_train, 'output': Y_train}, batch_size=batch_size, nb_epoch=nb_epoch // 2, verbose=0,
+              class_weight={'output': class_weight}, sample_weight={'output': sample_weight}, validation_split=0.1)
 
-        sample_weight = np.ones((y_train.shape[0])) * standard_weight
-        sample_weight[y_train == weighted_class] = high_weight
+    model.train_on_batch({'input': X_train[:32], 'output': Y_train[:32]},
+                         class_weight={'output': class_weight}, sample_weight={'output': sample_weight[:32] if sample_weight is not None else None})
+    model.test_on_batch({'input': X_train[:32], 'output': Y_train[:32]},
+                        sample_weight={'output': sample_weight[:32] if sample_weight is not None else None})
+    score = model.evaluate({'input': X_test[test_ids, :], 'output': Y_test[test_ids, :]}, verbose=0)
+    return score
 
+
+class TestLossWeighting(unittest.TestCase):
+    def test_sequential(self):
         for loss in ['mae', 'mse', 'categorical_crossentropy']:
             print('loss:', loss)
+            print('sequential')
             # no weights: reference point
-            model = create_model()
+            model = create_sequential_model()
             model.compile(loss='categorical_crossentropy', optimizer='rmsprop')
-            standard_score = _test_weights(model)
+            standard_score = _test_weights_sequential(model)
             # test class_weight
-            model = create_model()
+            model = create_sequential_model()
             model.compile(loss=loss, optimizer='rmsprop')
-            score = _test_weights(model, class_weight=class_weight)
+            score = _test_weights_sequential(model, class_weight=class_weight)
             print('score:', score, ' vs.', standard_score)
             self.assertTrue(score < standard_score)
             # test sample_weight
-            model = create_model()
+            model = create_sequential_model()
             model.compile(loss=loss, optimizer='rmsprop')
-            score = _test_weights(model, sample_weight=sample_weight)
+            score = _test_weights_sequential(model, sample_weight=sample_weight)
             print('score:', score, ' vs.', standard_score)
             self.assertTrue(score < standard_score)
 
+    def test_graph(self):
+        for loss in ['mae', 'mse', 'categorical_crossentropy']:
+            print('loss:', loss)
+            print('graph')
+            # no weights: reference point
+            model = create_graph_model()
+            model.compile(loss={'output': 'categorical_crossentropy'}, optimizer='rmsprop')
+            standard_score = _test_weights_graph(model)
+            # test class_weight
+            model = create_graph_model()
+            model.compile(loss={'output': 'categorical_crossentropy'}, optimizer='rmsprop')
+            score = _test_weights_graph(model, class_weight=class_weight)
+            print('score:', score, ' vs.', standard_score)
+            self.assertTrue(score < standard_score)
+            # test sample_weight
+            model = create_graph_model()
+            model.compile(loss={'output': 'categorical_crossentropy'}, optimizer='rmsprop')
+            score = _test_weights_graph(model, sample_weight=sample_weight)
+            print('score:', score, ' vs.', standard_score)
+            self.assertTrue(score < standard_score)
+
+
 if __name__ == '__main__':
     print('Test class_weight and sample_weight')
-    unittest.main()
+    unittest.main()

tests/auto/test_optimizers.py

@@ -0,0 +1,59 @@
+from __future__ import print_function
+import numpy as np
+np.random.seed(1337)
+
+from keras.utils.test_utils import get_test_data
+from keras.optimizers import SGD, RMSprop, Adagrad, Adadelta, Adam
+from keras.models import Sequential
+from keras.layers.core import Dense, Activation
+from keras.utils.np_utils import to_categorical
+import unittest
+
+
+(X_train, y_train), (X_test, y_test) = get_test_data(nb_train=1000, nb_test=200, input_shape=(10,),
+                                                     classification=True, nb_class=2)
+y_train = to_categorical(y_train)
+y_test = to_categorical(y_test)
+
+
+def get_model(input_dim, nb_hidden, output_dim):
+    model = Sequential()
+    model.add(Dense(input_dim, nb_hidden))
+    model.add(Activation('relu'))
+    model.add(Dense(nb_hidden, output_dim))
+    model.add(Activation('softmax'))
+    return model
+
+
+def _test_optimizer(optimizer, target=0.9):
+    model = get_model(X_train.shape[1], 10, y_train.shape[1])
+    model.compile(loss='categorical_crossentropy', optimizer=optimizer)
+    history = model.fit(X_train, y_train, nb_epoch=12, batch_size=16, validation_data=(X_test, y_test), show_accuracy=True, verbose=2)
+    return history.history['val_acc'][-1] > target
+
+
+class TestOptimizers(unittest.TestCase):
+    def test_sgd(self):
+        print('test SGD')
+        sgd = SGD(lr=0.01, momentum=0.9, nesterov=True)
+        self.assertTrue(_test_optimizer(sgd))
+
+    def test_rmsprop(self):
+        print('test RMSprop')
+        self.assertTrue(_test_optimizer(RMSprop()))
+
+    def test_adagrad(self):
+        print('test Adagrad')
+        self.assertTrue(_test_optimizer(Adagrad()))
+
+    def test_adadelta(self):
+        print('test Adadelta')
+        self.assertTrue(_test_optimizer(Adadelta()))
+
+    def test_adam(self):
+        print('test Adam')
+        self.assertTrue(_test_optimizer(Adam()))
+
+if __name__ == '__main__':
+    print('Test optimizers')
+    unittest.main()

tests/auto/test_regularizers.py

@@ -32,6 +32,7 @@ Y_train = np_utils.to_categorical(y_train, nb_classes)
 Y_test = np_utils.to_categorical(y_test, nb_classes)
 test_ids = np.where(y_test == np.array(weighted_class))[0]
 
+
 def create_model(weight_reg=None, activity_reg=None):
     model = Sequential()
     model.add(Dense(784, 50))

tests/auto/test_sequential_model.py

@@ -4,7 +4,7 @@ import unittest
 import numpy as np
 np.random.seed(1337)
 
-from keras.models import Sequential
+from keras.models import Sequential, model_from_json, model_from_yaml
 from keras.layers.core import Dense, Activation, Merge
 from keras.utils import np_utils
 from keras.utils.test_utils import get_test_data
@@ -19,12 +19,13 @@ train_samples = 5000
 test_samples = 1000
 
 (X_train, y_train), (X_test, y_test) = get_test_data(nb_train=train_samples, nb_test=test_samples, input_shape=(input_dim,),
-    classification=True, nb_class=4)
+                                                     classification=True, nb_class=4)
 y_test = np_utils.to_categorical(y_test)
 y_train = np_utils.to_categorical(y_train)
 print(X_train.shape)
 print(y_train.shape)
 
+
 class TestSequential(unittest.TestCase):
     def test_sequential(self):
         print('Test sequential')
@@ -67,6 +68,13 @@ class TestSequential(unittest.TestCase):
         print(nloss)
         assert(loss == nloss)
 
+        # test json serialization
+        json_data = model.to_json()
+        model = model_from_json(json_data)
+
+        # test yaml serialization
+        yaml_data = model.to_yaml()
+        model = model_from_yaml(yaml_data)
 
     def test_merge_sum(self):
         print('Test merge: sum')
@@ -121,7 +129,6 @@ class TestSequential(unittest.TestCase):
         print(nloss)
         assert(loss == nloss)
 
-
     def test_merge_concat(self):
         print('Test merge: concat')
         left = Sequential()
@@ -179,7 +186,6 @@ class TestSequential(unittest.TestCase):
         print(nloss)
         assert(loss == nloss)
 
-
     def test_merge_recursivity(self):
         print('Test merge recursivity')
 
@@ -231,7 +237,6 @@ class TestSequential(unittest.TestCase):
         print(nloss)
         assert(loss == nloss)
 
-
     def test_merge_overlap(self):
         print('Test merge overlap')
         left = Sequential()
@@ -272,7 +277,6 @@ class TestSequential(unittest.TestCase):
         assert(loss == nloss)
 
 
-
 if __name__ == '__main__':
     print('Test Sequential model')
     unittest.main()

tests/auto/test_tasks.py

@@ -1,6 +1,7 @@
 from __future__ import print_function
 import numpy as np
 np.random.seed(1337)
+
 from keras.utils.test_utils import get_test_data
 from keras.models import Sequential
 from keras.layers.core import Dense, Activation, TimeDistributedDense, Flatten
@@ -9,13 +10,14 @@ from keras.layers.convolutional import Convolution2D
 from keras.utils.np_utils import to_categorical
 import unittest
 
+
 class TestRegularizers(unittest.TestCase):
     def test_vector_clf(self):
         nb_hidden = 10
 
         print('vector classification data:')
         (X_train, y_train), (X_test, y_test) = get_test_data(nb_train=1000, nb_test=200, input_shape=(10,),
-            classification=True, nb_class=2)
+                                                             classification=True, nb_class=2)
         print('X_train:', X_train.shape)
         print('X_test:', X_test.shape)
         print('y_train:', y_train.shape)
@@ -38,7 +40,7 @@ class TestRegularizers(unittest.TestCase):
         nb_hidden = 10
         print('vector regression data:')
         (X_train, y_train), (X_test, y_test) = get_test_data(nb_train=1000, nb_test=200, input_shape=(10,), output_shape=(2,),
-            classification=False)
+                                                             classification=False)
         print('X_train:', X_train.shape)
         print('X_test:', X_test.shape)
         print('y_train:', y_train.shape)
@@ -55,7 +57,7 @@ class TestRegularizers(unittest.TestCase):
     def test_temporal_clf(self):
         print('temporal classification data:')
         (X_train, y_train), (X_test, y_test) = get_test_data(nb_train=1000, nb_test=200, input_shape=(5,10), 
-            classification=True, nb_class=2)
+                                                             classification=True, nb_class=2)
         print('X_train:', X_train.shape)
         print('X_test:', X_test.shape)
         print('y_train:', y_train.shape)
@@ -74,7 +76,7 @@ class TestRegularizers(unittest.TestCase):
     def test_temporal_reg(self):
         print('temporal regression data:')
         (X_train, y_train), (X_test, y_test) = get_test_data(nb_train=1000, nb_test=200, input_shape=(5, 10), output_shape=(2,),
-            classification=False)
+                                                             classification=False)
         print('X_train:', X_train.shape)
         print('X_test:', X_test.shape)
         print('y_train:', y_train.shape)
@@ -84,13 +86,12 @@ class TestRegularizers(unittest.TestCase):
         model.add(GRU(X_train.shape[-1], y_train.shape[-1]))
         model.compile(loss='hinge', optimizer='adam')
         history = model.fit(X_train, y_train, nb_epoch=12, batch_size=16, validation_data=(X_test, y_test), verbose=2)
-        self.assertTrue(history.history['val_loss'][-1] < 0.75)
-
+        self.assertTrue(history.history['val_loss'][-1] < 0.8)
 
     def test_seq_to_seq(self):
         print('sequence to sequence data:')
         (X_train, y_train), (X_test, y_test) = get_test_data(nb_train=1000, nb_test=200, input_shape=(5, 10), output_shape=(5, 10),
-            classification=False)
+                                                             classification=False)
         print('X_train:', X_train.shape)
         print('X_test:', X_test.shape)
         print('y_train:', y_train.shape)
@@ -102,11 +103,10 @@ class TestRegularizers(unittest.TestCase):
         history = model.fit(X_train, y_train, nb_epoch=12, batch_size=16, validation_data=(X_test, y_test), verbose=2)
         self.assertTrue(history.history['val_loss'][-1] < 0.75)
 
-
     def test_img_clf(self):
         print('image classification data:')
-        (X_train, y_train), (X_test, y_test) = get_test_data(nb_train=1000, nb_test=200, input_shape=(3, 32, 32), 
-            classification=True, nb_class=2)
+        (X_train, y_train), (X_test, y_test) = get_test_data(nb_train=1000, nb_test=200, input_shape=(3, 32, 32),
+                                                             classification=True, nb_class=2)
         print('X_train:', X_train.shape)
         print('X_test:', X_test.shape)
         print('y_train:', y_train.shape)
@@ -128,4 +128,4 @@ class TestRegularizers(unittest.TestCase):
 
 if __name__ == '__main__':
     print('Test different types of classification and regression tasks')
-    unittest.main()
+    unittest.main()

tests/manual/check_autoencoder.py

@@ -1,7 +1,7 @@
 from __future__ import absolute_import
 from __future__ import print_function
 from keras.datasets import mnist
-from keras.models import Sequential
+from keras.models import Sequential, model_from_config
 from keras.layers.core import AutoEncoder, Dense, Activation, TimeDistributedDense, Flatten
 from keras.layers.recurrent import LSTM
 from keras.layers.embeddings import Embedding
@@ -57,76 +57,78 @@ print('\nclassical_score:', classical_score)
 # autoencoder model test #
 ##########################
 
-def build_lstm_autoencoder(autoencoder, X_train, X_test):
-	X_train = X_train[:, np.newaxis, :] 
-	X_test = X_test[:, np.newaxis, :]
-	print("Modified X_train: ", X_train.shape)
-	print("Modified X_test: ", X_test.shape)
 
-	# The TimeDistributedDense isn't really necessary, however you need a lot of GPU memory to do 784x394-394x784
-	autoencoder.add(TimeDistributedDense(input_dim, 16))
-	autoencoder.add(AutoEncoder(encoder=LSTM(16, 8, activation=activation, return_sequences=True),
-								decoder=LSTM(8, input_dim, activation=activation, return_sequences=True),
-								output_reconstruction=False))
-	return autoencoder, X_train, X_test
+def build_lstm_autoencoder(autoencoder, X_train, X_test):
+    X_train = X_train[:, np.newaxis, :]
+    X_test = X_test[:, np.newaxis, :]
+    print("Modified X_train: ", X_train.shape)
+    print("Modified X_test: ", X_test.shape)
+
+    # The TimeDistributedDense isn't really necessary, however you need a lot of GPU memory to do 784x394-394x784
+    autoencoder.add(TimeDistributedDense(input_dim, 16))
+    autoencoder.add(AutoEncoder(encoder=LSTM(16, 8, activation=activation, return_sequences=True),
+                                decoder=LSTM(8, input_dim, activation=activation, return_sequences=True),
+                                output_reconstruction=False))
+    return autoencoder, X_train, X_test
+
 
 def build_deep_classical_autoencoder(autoencoder):
-	encoder = containers.Sequential([Dense(input_dim, hidden_dim, activation=activation), Dense(hidden_dim, hidden_dim/2, activation=activation)])
-	decoder = containers.Sequential([Dense(hidden_dim/2, hidden_dim, activation=activation), Dense(hidden_dim, input_dim, activation=activation)])
-	autoencoder.add(AutoEncoder(encoder=encoder, decoder=decoder, output_reconstruction=False))
-	return autoencoder
-
-
+    encoder = containers.Sequential([Dense(input_dim, hidden_dim, activation=activation), Dense(hidden_dim, hidden_dim/2, activation=activation)])
+    decoder = containers.Sequential([Dense(hidden_dim/2, hidden_dim, activation=activation), Dense(hidden_dim, input_dim, activation=activation)])
+    autoencoder.add(AutoEncoder(encoder=encoder, decoder=decoder, output_reconstruction=False))
+    return autoencoder
 
 # Try different things here: 'lstm' or 'classical' or 'denoising'
 # or 'deep_denoising'
 
 for autoencoder_type in ['classical', 'lstm']:
-	print(autoencoder_type)
-	print('-'*40)
-	# Build our autoencoder model
-	autoencoder = Sequential()
-	if autoencoder_type == 'lstm':
-		print("Training LSTM AutoEncoder")
-		autoencoder, X_train, X_test = build_lstm_autoencoder(autoencoder, X_train, X_test)
-	elif autoencoder_type == 'classical':
-		print("Training Classical AutoEncoder")
-		autoencoder = build_deep_classical_autoencoder(autoencoder)
-	else:
-		print("Error: unknown autoencoder type!")
-		exit(-1)
+    print(autoencoder_type)
+    print('-'*40)
+    # Build our autoencoder model
+    autoencoder = Sequential()
+    if autoencoder_type == 'lstm':
+        print("Training LSTM AutoEncoder")
+        autoencoder, X_train, X_test = build_lstm_autoencoder(autoencoder, X_train, X_test)
+    elif autoencoder_type == 'classical':
+        print("Training Classical AutoEncoder")
+        autoencoder = build_deep_classical_autoencoder(autoencoder)
+    else:
+        print("Error: unknown autoencoder type!")
+        exit(-1)
 
-	autoencoder.get_config(verbose=1)
-	autoencoder.compile(loss='mean_squared_error', optimizer='adam')
-	# Do NOT use validation data with return output_reconstruction=True
-	autoencoder.fit(X_train, X_train, batch_size=batch_size, nb_epoch=nb_epoch, show_accuracy=False, verbose=1)
+    autoencoder.compile(loss='mean_squared_error', optimizer='adam')
+    # Do NOT use validation data with return output_reconstruction=True
+    autoencoder.fit(X_train, X_train, batch_size=batch_size, nb_epoch=nb_epoch, show_accuracy=False, verbose=1)
 
-	# Do an inference pass
-	prefilter_train = autoencoder.predict(X_train, verbose=0)
-	prefilter_test = autoencoder.predict(X_test, verbose=0)
-	print("prefilter_train: ", prefilter_train.shape)
-	print("prefilter_test: ", prefilter_test.shape)
+    # Do an inference pass
+    prefilter_train = autoencoder.predict(X_train, verbose=0)
+    prefilter_test = autoencoder.predict(X_test, verbose=0)
+    print("prefilter_train: ", prefilter_train.shape)
+    print("prefilter_test: ", prefilter_test.shape)
 
-	# Classify results from Autoencoder
-	print("Building classical fully connected layer for classification")
-	model = Sequential()
-	if autoencoder_type == 'lstm':
-		model.add(TimeDistributedDense(8, nb_classes, activation=activation))
-		model.add(Flatten())
-	elif autoencoder_type == 'classical':
-		model.add(Dense(prefilter_train.shape[1], nb_classes, activation=activation))
-	else:
-		model.add(Dense(prefilter_train.shape[1], nb_classes, activation=activation))
+    # Classify results from Autoencoder
+    print("Building classical fully connected layer for classification")
+    model = Sequential()
+    if autoencoder_type == 'lstm':
+        model.add(TimeDistributedDense(8, nb_classes, activation=activation))
+        model.add(Flatten())
+    elif autoencoder_type == 'classical':
+        model.add(Dense(prefilter_train.shape[1], nb_classes, activation=activation))
+    else:
+        model.add(Dense(prefilter_train.shape[1], nb_classes, activation=activation))
 
-	model.add(Activation('softmax'))
+    model.add(Activation('softmax'))
 
-	model.get_config(verbose=1)
-	model.compile(loss='categorical_crossentropy', optimizer='adam')
-	model.fit(prefilter_train, Y_train, batch_size=batch_size, nb_epoch=nb_epoch, show_accuracy=False, verbose=0, validation_data=(prefilter_test, Y_test))
+    model.get_config(verbose=1)
+    model.compile(loss='categorical_crossentropy', optimizer='adam')
+    model.fit(prefilter_train, Y_train, batch_size=batch_size, nb_epoch=nb_epoch, show_accuracy=False, verbose=0, validation_data=(prefilter_test, Y_test))
 
-	score = model.evaluate(prefilter_test, Y_test, verbose=0, show_accuracy=True)
-	print('\nscore:', score)
+    score = model.evaluate(prefilter_test, Y_test, verbose=0, show_accuracy=True)
+    print('\nscore:', score)
 
-	print('Loss change:', (score[0] - classical_score[0])/classical_score[0], '%')
-	print('Accuracy change:', (score[1] - classical_score[1])/classical_score[1], '%')
+    print('Loss change:', (score[0] - classical_score[0])/classical_score[0], '%')
+    print('Accuracy change:', (score[1] - classical_score[1])/classical_score[1], '%')
 
+    # check serialization
+    config = autoencoder.get_config(verbose=1)
+    autoencoder = model_from_config(config)

tests/manual/check_dot_utils.py

@@ -1,27 +0,0 @@
-from keras.utils.dot_utils import Grapher
-
-from keras.models import Sequential
-from keras.layers.core import Dense, Activation, Merge, Flatten
-from keras.layers.embeddings import Embedding
-from keras.layers.recurrent import GRU
-
-ent_lookup = Sequential()
-ent_lookup.add(Embedding(10, 2))
-ent_lookup.add(Flatten())
-
-rel_lookup = Sequential()
-rel_lookup.add(Embedding(20, 2))
-rel_lookup.add(Flatten())
-
-word_sequence = Sequential()
-word_sequence.add(Embedding(10, 5))
-word_sequence.add(GRU(5, 2))
-
-model = Sequential()
-model.add(Merge([word_sequence, ent_lookup, rel_lookup], mode='concat'))
-model.add(Activation('relu'))
-model.add(Dense(6, 2))
-model.add(Activation('softmax'))
-
-g = Grapher()
-g.plot(model, 'mymodel.png')

tests/manual/check_model_utils.py

@@ -0,0 +1,44 @@
+from __future__ import absolute_import
+from __future__ import print_function
+from keras.models import Sequential, Graph
+from keras.layers.core import Layer, Activation, Dense, Flatten, Reshape, Merge
+from keras.layers.convolutional import Convolution2D, MaxPooling2D
+import keras.utils.layer_utils as layer_utils
+
+print('-- Sequential model')
+left = Sequential()
+left.add(Convolution2D(32, 1, 3, 3, border_mode='valid'))
+left.add(MaxPooling2D(poolsize=(2, 2)))
+left.add(Flatten())
+left.add(Dense(32 * 13 * 13, 50))
+left.add(Activation('relu'))
+
+right = Sequential()
+right.add(Dense(784, 30))
+right.add(Activation('relu'))
+
+model = Sequential()
+model.add(Merge([left, right], mode='concat'))
+
+model.add(Dense(80, 10))
+model.add(Activation('softmax'))
+
+layer_utils.print_layer_shapes(model, [(1, 1, 28, 28), (1, 784)])
+
+print('-- Graph model')
+graph = Graph()
+graph.add_input(name='input1', ndim=2)
+graph.add_input(name='input2', ndim=4)
+graph.add_node(Dense(32, 16), name='dense1', input='input1')
+graph.add_node(Dense(16, 4), name='dense3', input='dense1')
+
+graph.add_node(Convolution2D(32, 1, 3, 3), name='conv1', input='input2')
+graph.add_node(Flatten(), name='flatten1', input='conv1')
+graph.add_node(Dense(32 * 13 * 13, 10), name='dense4', input='flatten1')
+
+graph.add_output(name='output1', inputs=['dense1', 'dense3'], merge_mode='sum')
+graph.add_output(name='output2', inputs=['dense1', 'dense4'], merge_mode='concat')
+
+layer_utils.print_layer_shapes(graph, {'input1': (1, 32), 'input2': (1, 1, 28, 28)})
+
+print('Test script complete')

tests/manual/check_wrappers.py

@@ -4,23 +4,28 @@ from keras.datasets import mnist
 from keras.models import Sequential
 from keras.layers.core import Dense, Activation
 from keras.utils import np_utils
-from keras.wrappers.scikit_learn import KerasClassifier
+from keras.wrappers.scikit_learn import *
 import numpy as np
 
-nb_classes = 10
 batch_size = 128
 nb_epoch = 1
 
+nb_classes = 10
 max_train_samples = 5000
 max_test_samples = 1000
 
 np.random.seed(1337) # for reproducibility
 
-# the data, shuffled and split between tran and test sets
+############################################
+# scikit-learn classification wrapper test #
+############################################
+print('Beginning scikit-learn classification wrapper test')
+
+print('Loading data')
 (X_train, y_train), (X_test, y_test) = mnist.load_data()
 
-X_train = X_train.reshape(60000,784)[:max_train_samples]
-X_test = X_test.reshape(10000,784)[:max_test_samples]
+X_train = X_train.reshape(60000, 784)[:max_train_samples]
+X_test = X_test.reshape(10000, 784)[:max_test_samples]
 X_train = X_train.astype('float32')
 X_test = X_test.astype('float32')
 X_train /= 255
@@ -29,11 +34,6 @@ X_test /= 255
 Y_train = np_utils.to_categorical(y_train, nb_classes)[:max_train_samples]
 Y_test = np_utils.to_categorical(y_test, nb_classes)[:max_test_samples]
 
-#############################
-# scikit-learn wrapper test #
-#############################
-print('Beginning scikit-learn wrapper test')
-
 print('Defining model')
 model = Sequential()
 model.add(Dense(784, 50))
@@ -42,10 +42,10 @@ model.add(Dense(50, 10))
 model.add(Activation('softmax'))
 
 print('Creating wrapper')
-classifier = KerasClassifier(model)
+classifier = KerasClassifier(model, train_batch_size=batch_size, nb_epoch=nb_epoch)
 
 print('Fitting model')
-classifier.fit(X_train, Y_train, batch_size=batch_size, nb_epoch=nb_epoch)
+classifier.fit(X_train, Y_train)
 
 print('Testing score function')
 score = classifier.score(X_train, Y_train)
@@ -63,7 +63,7 @@ print('Testing get params')
 print(classifier.get_params())
 
 print('Testing set params')
-classifier.set_params(optimizer='sgd', loss='mse')
+classifier.set_params(optimizer='sgd', loss='binary_crossentropy')
 print(classifier.get_params())
 
 print('Testing attributes')
@@ -73,5 +73,54 @@ print('Config')
 print(classifier.config_)
 print('Weights')
 print(classifier.weights_)
+print('Compiled model')
+print(classifier.compiled_model_)
+
+########################################
+# scikit-learn regression wrapper test #
+########################################
+print('Beginning scikit-learn regression wrapper test')
+
+print('Generating data')
+X_train = np.random.random((5000, 100))
+X_test = np.random.random((1000, 100))
+y_train = np.random.random(5000)
+y_test = np.random.random(1000)
+
+print('Defining model')
+model = Sequential()
+model.add(Dense(100, 50))
+model.add(Activation('relu'))
+model.add(Dense(50, 1))
+model.add(Activation('linear'))
+
+print('Creating wrapper')
+regressor = KerasRegressor(model, train_batch_size=batch_size, nb_epoch=nb_epoch)
+
+print('Fitting model')
+regressor.fit(X_train, y_train)
+
+print('Testing score function')
+score = regressor.score(X_train, y_train)
+print('Score: ', score)
+
+print('Testing predict function')
+preds = regressor.predict(X_test)
+print('Preds.shape: ', preds.shape)
+
+print('Testing get params')
+print(regressor.get_params())
+
+print('Testing set params')
+regressor.set_params(optimizer='sgd', loss='mean_absolute_error')
+print(regressor.get_params())
+
+print('Testing attributes')
+print('Config')
+print(regressor.config_)
+print('Weights')
+print(regressor.weights_)
+print('Compiled model')
+print(regressor.compiled_model_)
 
 print('Test script complete.')

tests/manual/check_yaml.py

@@ -14,12 +14,12 @@ from keras.datasets import imdb
 from keras.models import model_from_yaml
 
 '''
-This is essentially the IMDB test. Deserialized models should yield 
+This is essentially the IMDB test. Deserialized models should yield
 the same config as the original one.
 '''
 
 max_features = 10000
-maxlen = 100 
+maxlen = 100
 batch_size = 32
 
 (X_train, y_train), (X_test, y_test) = imdb.load_data(nb_words=max_features, test_split=0.2)
@@ -29,7 +29,7 @@ X_test = sequence.pad_sequences(X_test, maxlen=maxlen)
 
 model = Sequential()
 model.add(Embedding(max_features, 128))
-model.add(LSTM(128, 128)) 
+model.add(LSTM(128, 128))
 model.add(Dropout(0.5))
 model.add(Dense(128, 1, W_regularizer='identity', b_constraint='maxnorm'))
 model.add(Activation('sigmoid'))
@@ -70,7 +70,7 @@ y = np.random.random((100, 4))
 y2 = np.random.random((100,))
 
 (X_train, y_train), (X_test, y_test) = get_test_data(nb_train=1000, nb_test=200, input_shape=(32,),
-    classification=False, output_shape=(4,))
+                                                     classification=False, output_shape=(4,))
 
 graph = Graph()
 
@@ -81,12 +81,12 @@ graph.add_node(Dense(32, 4), name='dense2', input='input1')
 graph.add_node(Dense(16, 4), name='dense3', input='dense1')
 
 graph.add_output(name='output1', inputs=['dense2', 'dense3'], merge_mode='sum')
-graph.compile('rmsprop', {'output1':'mse'})
+graph.compile('rmsprop', {'output1': 'mse'})
 
 graph.get_config(verbose=1)
 
-history = graph.fit({'input1':X_train, 'output1':y_train}, nb_epoch=10)
-original_pred = graph.predict({'input1':X_test})
+history = graph.fit({'input1': X_train, 'output1': y_train}, nb_epoch=10)
+original_pred = graph.predict({'input1': X_test})
 
 graph_yaml = graph.to_yaml()
 graph.save_weights('temp.h5', overwrite=True)
@@ -95,7 +95,7 @@ reloaded_graph = model_from_yaml(graph_yaml)
 reloaded_graph.load_weights('temp.h5')
 reloaded_graph.get_config(verbose=1)
 
-reloaded_graph.compile('rmsprop', {'output1':'mse'})
-new_pred = reloaded_graph.predict({'input1':X_test})
+reloaded_graph.compile('rmsprop', {'output1': 'mse'})
+new_pred = reloaded_graph.predict({'input1': X_test})
 
-assert(new_pred['output1'][3][1] == original_pred['output1'][3][1])
+assert(np.sum(new_pred['output1'] - original_pred['output1']) == 0)