Prepare 0.3.3 release (last release before 1.0).

add sign operation to backends

.travis.yml

@@ -12,7 +12,9 @@ matrix:
         - python: 2.7
           env: KERAS_BACKEND=tensorflow
         - python: 2.7
-          env: KERAS_BACKEND=theano INTEGRATION_TESTS=true
+          env: KERAS_BACKEND=theano TEST_MODE=INTEGRATION_TESTS
+        - python: 2.7
+          env: KERAS_BACKEND=theano TEST_MODE=PEP8
 install:
   # code below is taken from http://conda.pydata.org/docs/travis.html
   # We do this conditionally because it saves us some downloading if the
@@ -33,6 +35,7 @@ install:
   - conda create -q -n test-environment python=$TRAVIS_PYTHON_VERSION numpy scipy matplotlib pandas pytest h5py
   - source activate test-environment
   - pip install pytest-cov python-coveralls pytest-xdist coverage==3.7.1 #we need this version of coverage for coveralls.io to work
+  - pip install pep8 pytest-pep8
   - pip install git+git://github.com/Theano/Theano.git
 
   # install PIL for preprocessing tests
@@ -57,8 +60,10 @@ script:
   # set up keras backend
   - sed -i -e 's/"backend":[[:space:]]*"[^"]*/"backend":\ "'$KERAS_BACKEND'/g' ~/.keras/keras.json;
   - echo -e "Running tests with the following config:\n$(cat ~/.keras/keras.json)"
-  - if [[ "$INTEGRATION_TESTS" == "true" ]]; then
+  - if [[ "$TEST_MODE" == "INTEGRATION_TESTS" ]]; then
        PYTHONPATH=$PWD:$PYTHONPATH py.test tests/integration_tests;
+    elif [[ "$TEST_MODE" == "PEP8" ]]; then
+       PYTHONPATH=$PWD:$PYTHONPATH py.test --pep8 -m pep8 -n0;
     else
        PYTHONPATH=$PWD:$PYTHONPATH py.test tests/ --ignore=tests/integration_tests;
     fi

CONTRIBUTING.md

@@ -36,25 +36,30 @@ We love pull requests. Here's a quick guide:
 
 1. If your PR introduces a change in functionality, make sure you start by opening an issue to discuss whether the change should be made, and how to handle it. This will save you from having your PR closed down the road! Of course, if your PR is a simple bug fix, you don't need to do that.
 
-2. Write the code. This is the hard part! We use PEP8 syntax conventions, but we aren't dogmatic when it comes to line length. Make sure your lines stay reasonably sized, though. To make your life easier, we recommend installing a PEP8 linter.
+2. Write the code. This is the hard part!
 
 3. Make sure any new function or class you introduce has proper docstrings. Make sure any code you touch still has up-to-date docstrings and documentation.
 
 4. Write tests. Your code should have full unit test coverage. If you want to see your PR merged promptly, this is crucial.
 
 5. Run our test suite locally. It's easy: from the Keras folder, simply run: `py.test tests/`.
-  - You will need to install `pytest`, `coveralls`, `pytest-cov`, `pytest-xdist`: `pip install pytest pytest-cov python-coveralls pytest-xdist`
+  - You will need to install `pytest`, `coveralls`, `pytest-cov`, `pytest-xdist`: `pip install pytest pytest-cov python-coveralls pytest-xdist pep8 pytest-pep8`
 
 6. Make sure all tests are passing:
   - with the Theano backend, on Python 2.7 and Python 3.5
   - with the TensorFlow backend, on Python 2.7
 
-7. When committing, use appropriate, descriptive commit messages. Make sure that your branch history is not a string of "bug fix", "fix", "oops", etc. When submitting your PR, squash your commits into a single commit with an appropriate commit message, to make sure the project history stays clean and readable. See ['rebase and squash'](http://rebaseandsqua.sh/) for technical help on how to squash your commits.
+7. We use PEP8 syntax conventions, but we aren't dogmatic when it comes to line length. Make sure your lines stay reasonably sized, though. To make your life easier, we recommend running a PEP8 linter:
+  - Install PEP8 packages: `pip install pep8 pytest-pep8 autopep8`
+  - Run a standalone PEP8 check: `py.test --pep8 -m pep8`
+  - You can automatically fix some PEP8 error by running: `autopep8 -i --select <errors> <FILENAME>` for example: `autopep8 -i --select E128 tests/keras/backend/test_backends.py`
 
-8. Update the documentation. If introducing new functionality, make sure you include code snippets demonstrating the usage of your new feature.
+8. When committing, use appropriate, descriptive commit messages. Make sure that your branch history is not a string of "bug fix", "fix", "oops", etc. When submitting your PR, squash your commits into a single commit with an appropriate commit message, to make sure the project history stays clean and readable. See ['rebase and squash'](http://rebaseandsqua.sh/) for technical help on how to squash your commits.
 
-9. Submit your PR. If your changes have been approved in a previous discussion, and if you have have complete (and passing) unit tests, your PR is likely to be merged promptly. Otherwise, well...
+9. Update the documentation. If introducing new functionality, make sure you include code snippets demonstrating the usage of your new feature.
+
+10. Submit your PR. If your changes have been approved in a previous discussion, and if you have complete (and passing) unit tests, your PR is likely to be merged promptly. Otherwise, well...
 
 ## Adding new examples
 
-Even if you don't contribute to the Keras source code, if you have an application of Keras that is concise and powerful, please consider adding it to our collection of examples. Existing examples show idiomatic Keras code: make sure to keep your own script in the same spirit.
+Even if you don't contribute to the Keras source code, if you have an application of Keras that is concise and powerful, please consider adding it to our collection of examples. [Existing examples](https://github.com/fchollet/keras/tree/master/examples) show idiomatic Keras code: make sure to keep your own script in the same spirit.

ISSUE_TEMPLATE.md

@@ -0,0 +1,9 @@
+Please make sure that the boxes below are checked before you submit your issue. Thank you!
+
+- [ ] Check that you are up-to-date with the master branch of Keras. You can update with:
+pip install git+git://github.com/fchollet/keras.git --upgrade --no-deps
+
+- [ ] If running on Theano, check that you are up-to-date with the master branch of Theano. You can update with:
+pip install git+git://github.com/Theano/Theano.git --upgrade --no-deps
+
+- [ ] Provide a link to a GitHub Gist of a Python script that can reproduce your issue (or just copy the script here if it is short).

docs/README.md

@@ -8,4 +8,5 @@ Our documentation uses extended Markdown, as implemented by [MkDocs](http://mkdo
 - install MkDocs: `pip install mkdocs`
 - `cd` to the `docs/` folder and run:
     - `python autogen.py`
-    - `mkdocs serve`
+    - `mkdocs serve`    # Starts a local webserver:  [localhost:8000](localhost:8000)
+    - `mkdocs build`    # Builds a static site in "site" directory

docs/autogen.py

@@ -80,7 +80,7 @@ def get_method_signature(method):
     for a in args:
         st += str(a) + ', '
     for a, v in kwargs:
-        if  type(v) == str:
+        if type(v) == str:
             v = '\'' + v + '\''
         elif type(v) == unicode:
             v = 'u\'' + v + '\''
@@ -117,26 +117,30 @@ def code_snippet(snippet):
 
 
 def process_class_docstring(docstring):
-    docstring = re.sub(r'    # (.*)\n',
-                       r'    __\1__\n\n',
+    docstring = re.sub(r'\n    # (.*)\n',
+                       r'\n    __\1__\n\n',
                        docstring)
 
     docstring = re.sub(r'    ([^\s\\]+):(.*)\n',
                        r'    - __\1__:\2\n',
                        docstring)
+
+    docstring = docstring.replace('    ' * 5, '\t\t')
     docstring = docstring.replace('    ' * 3, '\t')
     docstring = docstring.replace('    ', '')
     return docstring
 
 
 def process_method_docstring(docstring):
-    docstring = re.sub(r'    # (.*)\n',
-                       r'    __\1__\n\n',
+    docstring = re.sub(r'\n        # (.*)\n',
+                       r'\n        __\1__\n\n',
                        docstring)
 
     docstring = re.sub(r'    ([^\s\\]+):(.*)\n',
                        r'    - __\1__:\2\n',
                        docstring)
+
+    docstring = docstring.replace('    ' * 6, '\t\t')
     docstring = docstring.replace('    ' * 4, '\t')
     docstring = docstring.replace('    ', '')
     return docstring

docs/templates/activations.md

@@ -14,11 +14,13 @@ is equivalent to:
 model.add(Dense(64, activation='tanh'))
 ```
 
-You can also pass an element-wise Theano function as an activation:
+You can also pass an element-wise Theano/TensorFlow function as an activation:
 
 ```python
+from keras import backend as K
+
 def tanh(x):
-    return theano.tensor.tanh(x)
+    return K.tanh(x)
 
 model.add(Dense(64, activation=tanh))
 model.add(Activation(tanh))
@@ -36,4 +38,4 @@ model.add(Activation(tanh))
 
 ## On Advanced Activations
 
-Activations that are more complex than a simple Theano function (eg. learnable activations, configurable activations, etc.) are available as [Advanced Activation layers](layers/advanced_activations.md), and can be found in the module `keras.layers.advanced_activations`. These include PReLU and LeakyReLU.
+Activations that are more complex than a simple Theano/TensorFlow function (eg. learnable activations, configurable activations, etc.) are available as [Advanced Activation layers](layers/advanced_activations.md), and can be found in the module `keras.layers.advanced_activations`. These include PReLU and LeakyReLU.

docs/templates/examples.md

@@ -1,11 +1,23 @@
 
 Here are a few examples to get you started!
 
-### Multilayer Perceptron (MLP):
+In the examples folder, you will also find example models for real datasets:
+
+- CIFAR10 small images classification: Convolutional Neural Network (CNN) with realtime data augmentation
+- IMDB movie review sentiment classification: LSTM over sequences of words
+- Reuters newswires topic classification: Multilayer Perceptron (MLP)
+- MNIST handwritten digits classification: MLP & CNN
+- Character-level text generation with LSTM
+
+...and more.
+
+------------------
+
+### Multilayer Perceptron (MLP) for multi-class softmax classification:
 
 ```python
 from keras.models import Sequential
-from keras.layers.core import Dense, Dropout, Activation
+from keras.layers import Dense, Dropout, Activation
 from keras.optimizers import SGD
 
 model = Sequential()
@@ -18,42 +30,64 @@ model.add(Dropout(0.5))
 model.add(Dense(64, init='uniform'))
 model.add(Activation('tanh'))
 model.add(Dropout(0.5))
-model.add(Dense(2, init='uniform'))
+model.add(Dense(10, init='uniform'))
 model.add(Activation('softmax'))
 
 sgd = SGD(lr=0.1, decay=1e-6, momentum=0.9, nesterov=True)
-model.compile(loss='mean_squared_error', optimizer=sgd)
+model.compile(loss='categorical_crossentropy',
+              optimizer=sgd)
 
-model.fit(X_train, y_train, nb_epoch=20, batch_size=16)
+model.fit(X_train, y_train,
+          nb_epoch=20,
+          batch_size=16,
+          show_accuracy=True)
 score = model.evaluate(X_test, y_test, batch_size=16)
 ```
 
-### Alternative implementation of MLP:
+------------------
+
+### Alternative implementation of a similar MLP:
 
 ```python
 model = Sequential()
-model.add(Dense(64, input_dim=20, init='uniform', activation='tanh'))
+model.add(Dense(64, input_dim=20, activation='relu'))
 model.add(Dropout(0.5))
-model.add(Dense(64, init='uniform', activation='tanh'))
+model.add(Dense(64, activation='relu'))
 model.add(Dropout(0.5))
-model.add(Dense(2, init='uniform', activation='softmax'))
+model.add(Dense(10, activation='softmax'))
 
-sgd = SGD(lr=0.1, decay=1e-6, momentum=0.9, nesterov=True)
-model.compile(loss='mean_squared_error', optimizer=sgd)
+model.compile(loss='categorical_crossentropy', optimizer='adadelta')
 ```
 
+------------------
+
+### MLP for binary classification:
+```python
+model = Sequential()
+model.add(Dense(64, input_dim=20, init='uniform', activation='relu'))
+model.add(Dropout(0.5))
+model.add(Dense(64, activation='relu'))
+model.add(Dropout(0.5))
+model.add(Dense(1, activation='sigmoid'))
+
+model.compile(loss='binary_crossentropy',
+              optimizer='rmsprop')
+```
+
+------------------
+
 ### VGG-like convnet:
 
 ```python
 from keras.models import Sequential
-from keras.layers.core import Dense, Dropout, Activation, Flatten
-from keras.layers.convolutional import Convolution2D, MaxPooling2D
+from keras.layers import Dense, Dropout, Activation, Flatten
+from keras.layers import Convolution2D, MaxPooling2D
 from keras.optimizers import SGD
 
 model = Sequential()
 # input: 100x100 images with 3 channels -> (3, 100, 100) tensors.
 # this applies 32 convolution filters of size 3x3 each.
-model.add(Convolution2D(32, 3, 3, border_mode='full', input_shape=(3, 100, 100)))
+model.add(Convolution2D(32, 3, 3, border_mode='valid', input_shape=(3, 100, 100)))
 model.add(Activation('relu'))
 model.add(Convolution2D(32, 3, 3))
 model.add(Activation('relu'))
@@ -83,13 +117,15 @@ model.fit(X_train, Y_train, batch_size=32, nb_epoch=1)
 
 ```
 
+------------------
+
 ### Sequence classification with LSTM:
 
 ```python
 from keras.models import Sequential
-from keras.layers.core import Dense, Dropout, Activation
-from keras.layers.embeddings import Embedding
-from keras.layers.recurrent import LSTM
+from keras.layers import Dense, Dropout, Activation
+from keras.layers import Embedding
+from keras.layers import LSTM
 
 model = Sequential()
 model.add(Embedding(max_features, 256, input_length=maxlen))
@@ -117,13 +153,13 @@ vocab_size = 10000
 # will encode pictures into 128-dimensional vectors.
 # it should be initialized with pre-trained weights.
 image_model = Sequential()
-image_model.add(Convolution2D(32, 3, 3, border_mode='full', input_shape=(3, 100, 100)))
+image_model.add(Convolution2D(32, 3, 3, border_mode='valid', input_shape=(3, 100, 100)))
 image_model.add(Activation('relu'))
 image_model.add(Convolution2D(32, 3, 3))
 image_model.add(Activation('relu'))
 image_model.add(MaxPooling2D(pool_size=(2, 2)))
 
-image_model.add(Convolution2D(64, 3, 3, border_mode='full'))
+image_model.add(Convolution2D(64, 3, 3, border_mode='valid'))
 image_model.add(Activation('relu'))
 image_model.add(Convolution2D(64, 3, 3))
 image_model.add(Activation('relu'))
@@ -147,9 +183,10 @@ image_model.add(RepeatVector(max_caption_len))
 
 # the output of both models will be tensors of shape (samples, max_caption_len, 128).
 # let's concatenate these 2 vector sequences.
-model = Merge([image_model, language_model], mode='concat', concat_axis=-1)
+model = Sequential()
+model.add(Merge([image_model, language_model], mode='concat', concat_axis=-1))
 # let's encode this vector sequence into a single vector
-model.add(GRU(256, 256, return_sequences=False))
+model.add(GRU(256, return_sequences=False))
 # which will be used to compute a probability
 # distribution over what the next word in the caption should be!
 model.add(Dense(vocab_size))
@@ -166,12 +203,188 @@ model.compile(loss='categorical_crossentropy', optimizer='rmsprop')
 model.fit([images, partial_captions], next_words, batch_size=16, nb_epoch=100)
 ```
 
-In the examples folder, you will find example models for real datasets:
+------------------
 
-- CIFAR10 small images classification: Convolutional Neural Network (CNN) with realtime data augmentation
-- IMDB movie review sentiment classification: LSTM over sequences of words
-- Reuters newswires topic classification: Multilayer Perceptron (MLP)
-- MNIST handwritten digits classification: MLP & CNN
-- Character-level text generation with LSTM
+### Stacked LSTM for sequence classification
 
-...and more.
+In this model, we stack 3 LSTM layers on top of each other,
+making the model capable of learning higher-level temporal representations.
+
+The first two LSTMs return their full output sequences, but the last one only returns
+the last step in its output sequence, thus dropping the temporal dimension
+(i.e. converting the input sequence into a single vector).
+
+<img src="http://keras.io/img/regular_stacked_lstm.png" alt="stacked LSTM" style="width: 300px;"/>
+
+(N.B.: in Keras, "None" in an input shape indicates a variable dimension. In the graph above, the batch size is "None",
+meaning that any batch size is allowed for the input data).
+
+```python
+from keras.models import Sequential
+from keras.layers import LSTM, Dense
+import numpy as np
+
+data_dim = 16
+timesteps = 8
+nb_classes = 10
+
+# expected input data shape: (batch_size, timesteps, data_dim)
+model = Sequential()
+model.add(LSTM(32, return_sequences=True,
+               input_shape=(timesteps, data_dim)))  # returns a sequence of vectors of dimension 32
+model.add(LSTM(32, return_sequences=True))  # returns a sequence of vectors of dimension 32
+model.add(LSTM(32))  # return a single vector of dimension 32
+model.add(Dense(10, activation='softmax'))
+
+model.compile(loss='categorical_crossentropy', optimizer='rmsprop')
+
+# generate dummy training data
+x_train = np.random.random((1000, timesteps, data_dim))
+y_train = np.random.random((1000, nb_classes))
+
+# generate dummy validation data
+x_val = np.random.random((100, timesteps, data_dim))
+y_val = np.random.random((100, nb_classes))
+
+model.fit(x_train, y_train,
+          batch_size=64, nb_epoch=5, show_accuracy=True,
+          validation_data=(x_val, y_val))
+```
+
+------------------
+
+### Same stacked LSTM model, rendered "stateful"
+
+A stateful recurrent model is one for which the internal states (memories) obtained after processing a batch
+of samples are reused as initial states for the samples of the next batch. This allows to process longer sequences
+while keeping computational complexity manageable.
+
+[You can read more about stateful RNNs in the FAQ.](/faq/#how-can-i-use-stateful-rnns)
+
+```python
+from keras.models import Sequential
+from keras.layers import LSTM, Dense
+import numpy as np
+
+data_dim = 16
+timesteps = 8
+nb_classes = 10
+batch_size = 32
+
+# expected input batch shape: (batch_size, timesteps, data_dim)
+# note that we have to provide the full batch_input_shape since the network is stateful.
+# the sample of index i in batch k is the follow-up for the sample i in batch k-1.
+model = Sequential()
+model.add(LSTM(32, return_sequences=True, stateful=True,
+               batch_input_shape=(batch_size, timesteps, data_dim)))
+model.add(LSTM(32, return_sequences=True, stateful=True))
+model.add(LSTM(32, stateful=True))
+model.add(Dense(10, activation='softmax'))
+
+model.compile(loss='categorical_crossentropy', optimizer='rmsprop')
+
+# generate dummy training data
+x_train = np.random.random((batch_size * 10, timesteps, data_dim))
+y_train = np.random.random((batch_size * 10, nb_classes))
+
+# generate dummy validation data
+x_val = np.random.random((batch_size * 3, timesteps, data_dim))
+y_val = np.random.random((batch_size * 3, nb_classes))
+
+model.fit(x_train, y_train,
+          batch_size=batch_size, nb_epoch=5, show_accuracy=True,
+          validation_data=(x_val, y_val))
+```
+
+------------------
+
+### Two merged LSTM encoders for classification over two parallel sequences
+
+In this model, two input sequences are encoded into vectors by two separate LSTM modules.
+
+These two vectors are then concatenated, and a fully connected network is trained on top of the concatenated representations.
+
+![Dual LSTM](http://keras.io/img/dual_lstm.png)
+
+```python
+from keras.models import Sequential
+from keras.layers import Merge, LSTM, Dense
+import numpy as np
+
+data_dim = 16
+timesteps = 8
+nb_classes = 10
+
+encoder_a = Sequential()
+encoder_a.add(LSTM(32, input_shape=(timesteps, data_dim)))
+
+encoder_b = Sequential()
+encoder_b.add(LSTM(32, input_shape=(timesteps, data_dim)))
+
+decoder = Sequential()
+decoder.add(Merge([encoder_a, encoder_b], mode='concat'))
+decoder.add(Dense(32, activation='relu'))
+decoder.add(Dense(nb_classes, activation='softmax'))
+
+decoder.compile(loss='categorical_crossentropy', optimizer='rmsprop')
+
+# generate dummy training data
+x_train_a = np.random.random((1000, timesteps, data_dim))
+x_train_b = np.random.random((1000, timesteps, data_dim))
+y_train = np.random.random((1000, nb_classes))
+
+# generate dummy validation data
+x_val_a = np.random.random((100, timesteps, data_dim))
+x_val_b = np.random.random((100, timesteps, data_dim))
+y_val = np.random.random((100, nb_classes))
+
+decoder.fit([x_train_a, x_train_b], y_train,
+            batch_size=64, nb_epoch=5, show_accuracy=True,
+            validation_data=([x_val_a, x_val_b], y_val))
+```
+
+------------------
+
+### Single shared LSTM over two parallel sequences, for classification
+
+This is a similar setup as above, but now a single LSTM encoder is used for both input sequences.
+Such a setup makes sense if the two input sequences are the same type of object.
+
+<img src="http://keras.io/img/shared_lstm.png" alt="Shared LSTM" style="width: 500px;"/>
+
+```python
+from keras.models import Graph
+from keras.layers import LSTM, Dense
+import numpy as np
+
+data_dim = 16
+timesteps = 8
+nb_classes = 10
+
+encoder = Sequential()
+encoder.add(LSTM(32, input_shape=(timesteps, data_dim)))
+
+model = Graph()
+model.add_input(name='input_a', input_shape=(timesteps, data_dim))
+model.add_input(name='input_b', input_shape=(timesteps, data_dim))
+model.add_shared_node(encoder, name='shared_encoder', inputs=['input_a', 'input_b'],
+                      merge_mode='concat')
+model.add_node(Dense(64, activation='relu'), name='fc1', input='shared_encoder')
+model.add_node(Dense(3, activation='softmax'), name='output', input='fc1', create_output=True)
+
+model.compile(optimizer='adam', loss={'output': 'categorical_crossentropy'})
+
+# generate dummy training data
+x_train_a = np.random.random((1000, timesteps, data_dim))
+x_train_b = np.random.random((1000, timesteps, data_dim))
+y_train = np.random.random((1000, 3))
+
+# generate dummy validation data
+x_val_a = np.random.random((100, timesteps, data_dim))
+x_val_b = np.random.random((100, timesteps, data_dim))
+y_val = np.random.random((100, 3))
+
+model.fit({'input_a': x_train_a, 'input_b': x_train_b, 'output': y_train},
+          batch_size=64, nb_epoch=5,
+          validation_data={'input_a': x_val_a, 'input_b': x_val_b, 'output': y_val})
+```

docs/templates/faq.md

@@ -1,5 +1,7 @@
 # Keras FAQ: Frequently Asked Keras Questions
 
+[How should I cite Keras?](#how-should-i-cite-keras)
+
 [How can I run Keras on GPU?](#how-can-i-run-keras-on-gpu)
 
 [How can I save a Keras model?](#how-can-i-save-a-keras-model)
@@ -8,8 +10,6 @@
 
 [How can I visualize the output of an intermediate layer?](#how-can-i-visualize-the-output-of-an-intermediate-layer)
 
-[Isn't there a bug with Merge or Graph related to input concatenation?](#isnt-there-a-bug-with-merge-or-graph-related-to-input-concatenation)
-
 [How can I use Keras with datasets that don't fit in memory?](#how-can-i-use-keras-with-datasets-that-dont-fit-in-memory)
 
 [How can I interrupt training when the validation loss isn't decreasing anymore?](#how-can-i-interrupt-training-when-the-validation-loss-isnt-decreasing-anymore)
@@ -24,8 +24,26 @@
 
 ---
 
+### How should I cite Keras?
+
+Please cite Keras in your publications if it helps your research. Here is an example BibTeX entry:
+
+```
+@misc{chollet2015keras,
+  author = {Chollet, François},
+  title = {Keras},
+  year = {2015},
+  publisher = {GitHub},
+  journal = {GitHub repository},
+  howpublished = {\url{https://github.com/fchollet/keras}}
+}
+```
+
 ### How can I run Keras on GPU?
 
+If you are running on the TensorFlow backend, your code will automatically run on GPU if any available GPU is detected.
+If you are running on the Theano backend, you can use one of the following methods:
+
 Method 1: use Theano flags.
 ```bash
 THEANO_FLAGS=device=gpu,floatX=float32 python my_keras_script.py
@@ -69,7 +87,10 @@ model = model_from_json(json_string)
 model = model_from_yaml(yaml_string)
 ```
 
-If you need to save the weights of a model, you can do so in HDF5:
+If you need to save the weights of a model, you can do so in HDF5 with the code below.
+
+Note that you will first need to install HDF5 and the Python library h5py, which do not come bundled with Keras.
+
 ```python
 model.save_weights('my_model_weights.h5')
 ```
@@ -103,31 +124,23 @@ Besides, the training loss is the average of the losses over each batch of train
 
 ### How can I visualize the output of an intermediate layer?
 
-You can build a Theano function that will return the output of a certain layer given a certain input, for example:
+You can build a Keras function that will return the output of a certain layer given a certain input, for example:
 
 ```python
+from keras import backend as K
+
 # with a Sequential model
-get_3rd_layer_output = theano.function([model.layers[0].input],
-                                       model.layers[3].get_output(train=False))
-layer_output = get_3rd_layer_output(X)
+get_3rd_layer_output = K.function([model.layers[0].input],
+                                  [model.layers[3].get_output(train=False)])
+layer_output = get_3rd_layer_output([X])[0]
 
 # with a Graph model
-get_conv_layer_output = theano.function([model.inputs[i].input for i in model.input_order],
-                                        model.nodes['conv'].get_output(train=False),
-                                        on_unused_input='ignore')
-conv_output = get_conv_layer_output([input_data_dict[i] for i in model.input_order])
+get_conv_layer_output = K.function([model.inputs[i].input for i in model.input_order],
+                                   [model.nodes['conv'].get_output(train=False)])
+conv_output = get_conv_layer_output([input_data_dict[i] for i in model.input_order])[0]
 ```
 
----
-
-### Isn't there a bug with Merge or Graph related to input concatenation?
-
-Yes, there was a known bug with tensor concatenation in Theano that was fixed early 2015.
-Please upgrade to the latest version of Theano:
-
-```bash
-sudo pip install git+git://github.com/Theano/Theano.git
-```
+Similarly, you could build a Theano and TensorFlow function directly.
 
 ---
 
@@ -135,7 +148,9 @@ sudo pip install git+git://github.com/Theano/Theano.git
 
 You can do batch training using `model.train_on_batch(X, y)` and `model.test_on_batch(X, y)`. See the [models documentation](models.md).
 
-You can also see batch training in action in our [CIFAR10 example](https://github.com/fchollet/keras/blob/master/examples/cifar10_cnn.py).
+Alternatively, you can write a generator that yields batches of training data and use the method `model.fit_generator(data_generator, samples_per_epoch, nb_epoch)`.
+
+You can see batch training in action in our [CIFAR10 example](https://github.com/fchollet/keras/blob/master/examples/cifar10_cnn.py).
 
 ---
 

docs/templates/objectives.md

@@ -7,10 +7,10 @@ An objective function (or loss function, or optimization score function) is one
 model.compile(loss='mean_squared_error', optimizer='sgd')
 ```
 
-You can either pass the name of an existing objective, or pass a Theano symbolic function that returns a scalar for each data-point and takes the following two arguments:
+You can either pass the name of an existing objective, or pass a Theano/TensorFlow symbolic function that returns a scalar for each data-point and takes the following two arguments:
 
-- __y_true__: True labels. Theano tensor.
-- __y_pred__: Predictions. Theano tensor of the same shape as y_true.
+- __y_true__: True labels. Theano/TensorFlow tensor.
+- __y_pred__: Predictions. Theano/TensorFlow tensor of the same shape as y_true.
 
 The actual optimized objective is the mean of the output array across all datapoints.
 
@@ -19,7 +19,6 @@ For a few examples of such functions, check out the [objectives source](https://
 ## Available objectives
 
 - __mean_squared_error__ / __mse__
-- __root_mean_squared_error__ / __rmse__
 - __mean_absolute_error__ / __mae__
 - __mean_absolute_percentage_error__ / __mape__
 - __mean_squared_logarithmic_error__ / __msle__
@@ -27,3 +26,5 @@ For a few examples of such functions, check out the [objectives source](https://
 - __hinge__
 - __binary_crossentropy__: Also known as logloss. 
 - __categorical_crossentropy__: Also known as multiclass logloss. __Note__: using this objective requires that your labels are binary arrays of shape `(nb_samples, nb_classes)`.
+- __poisson__: mean of `(predictions - targets * log(predictions))`
+- __cosine_proximity__: the opposite (negative) of the mean cosine proximity between predictions and targets.

docs/templates/preprocessing/image.md

@@ -10,11 +10,12 @@ keras.preprocessing.image.ImageDataGenerator(featurewise_center=True,
     rotation_range=0.,
     width_shift_range=0.,
     height_shift_range=0.,
+    shear_range=0.,
     horizontal_flip=False,
     vertical_flip=False)
 ```
 
-Generate batches of tensor image data with real-time data augmentation.
+Generate batches of tensor image data with real-time data augmentation. The data will be looped over (in batches) indefinitely.
 
 - __Arguments__:
     - __featurewise_center__: Boolean. Set input mean to 0 over the dataset.
@@ -25,24 +26,25 @@ Generate batches of tensor image data with real-time data augmentation.
     - __rotation_range__: Int. Degree range for random rotations.
     - __width_shift_range__: Float (fraction of total width). Range for random horizontal shifts.
     - __height_shift_range__: Float (fraction of total height). Range for random vertical shifts.
+    - __shear_range__: Float. Shear Intensity (Shear angle in counter-clockwise direction as radians)
     - __horizontal_flip__: Boolean. Randomly flip inputs horizontally.
     - __vertical_flip__: Boolean. Randomly flip inputs vertically.
 
 - __Methods__:
     - __fit(X)__: Required if featurewise_center or featurewise_std_normalization or zca_whitening. Compute necessary quantities on some sample data.
-        - __Arguments__: 
+        - __Arguments__:
             - __X__: sample data.
             - __augment__: Boolean (default: False). Whether to fit on randomly augmented samples.
             - __rounds__: int (default: 1). If augment, how many augmentation passes over the data to use.
     - __flow(X, y)__:
-        - __Arguments__: 
+        - __Arguments__:
             - __X__: data.
             - __y__: labels.
             - __batch_size__: int (default: 32).
             - __shuffle__: boolean (defaut: False).
-            - __save_to_dir__: None or str. This allows you to optimally specify a directory to which to save the augmented pictures being generated (useful for visualizing what you are doing). 
+            - __save_to_dir__: None or str. This allows you to optimally specify a directory to which to save the augmented pictures being generated (useful for visualizing what you are doing).
             - __save_prefix__: str. Prefix to use for filenames of saved pictures.
-            - __save_format__: one of "png", jpeg". 
+            - __save_format__: one of "png", jpeg".
 
 - __Example__:
 ```python
@@ -58,13 +60,23 @@ datagen = ImageDataGenerator(
     height_shift_range=0.2,
     horizontal_flip=True)
 
-# compute quantities required for featurewise normalization 
+# compute quantities required for featurewise normalization
 # (std, mean, and principal components if ZCA whitening is applied)
 datagen.fit(X_train)
 
+# fits the model on batches with real-time data augmentation:
+model.fit_generator(datagen.flow(X_train, Y_train, batch_size=32),
+                    samples_per_epoch=len(X_train), nb_epoch=nb_epoch)
+
+# here's a more "manual" example
 for e in range(nb_epoch):
     print 'Epoch', e
-    # batch train with realtime data augmentation
-    for X_batch, Y_batch in datagen.flow(X_train, Y_train):
+    batches = 0
+    for X_batch, Y_batch in datagen.flow(X_train, Y_train, batch_size=32):
         loss = model.train(X_batch, Y_batch)
-```
+        batches += 1
+        if batches >= len(X_train) / 32:
+            # we need to break the loop by hand because
+            # the generator loops indefinitely
+            break
+```

docs/templates/preprocessing/sequence.md

@@ -12,6 +12,9 @@ Transform a list of `nb_samples sequences` (lists of scalars) into a 2D numpy ar
     - __sequences__: List of lists of int or float.
     - __maxlen__: None or int. Maximum sequence length, longer sequences are truncated and shorter sequences are padded with zeros at the end.
     - __dtype__: datatype of the numpy array returned.
+    - __padding__: 'pre' or 'post', pad either before or after each sequence.
+    - __truncating__: 'pre' or 'post', remove values from sequences larger than maxlen either in the beginning or in the end of the sequence
+    - __value__: float, value to pad the sequences to the desired value.
 
 ---
 

docs/templates/visualization.md

@@ -10,6 +10,11 @@ from keras.utils.visualize_util import plot
 plot(model, to_file='model.png')
 ```
 
+`plot` takes two optional arguments:
+
+- `recursive` (defaults to True) controls whether we recursively explore container layers.
+- `show_shape` (defaults to False) controls whether output shapes are shown in the graph.
+
 You can also directly obtain the `pydot.Graph` object and render it yourself,
 for example to show it in an ipython notebook :
 ```python

examples/antirectifier.py

@@ -0,0 +1,106 @@
+'''The example demonstrates how to write custom layers for Keras.
+
+We build a custom activation layer called 'Antirectifier',
+which modifies the shape of the tensor that passes through it.
+We need to specify two methods: `output_shape` and `get_output`.
+
+Note that the same result can also be achieved via a Lambda layer.
+
+Because our custom layer is written with primitives from the Keras
+backend (`K`), our code can run both on TensorFlow and Theano.
+'''
+
+from __future__ import print_function
+import numpy as np
+from keras.models import Sequential
+from keras.layers.core import Dense, Dropout, Layer, Activation
+from keras.datasets import mnist
+from keras import backend as K
+from keras.utils import np_utils
+
+
+class Antirectifier(Layer):
+    '''This is the combination of a sample-wise
+    L2 normalization with the concatenation of the
+    positive part of the input with the negative part
+    of the input. The result is a tensor of samples that are
+    twice as large as the input samples.
+
+    It can be used in place of a ReLU.
+
+    # Input shape
+        2D tensor of shape (samples, n)
+
+    # Output shape
+        2D tensor of shape (samples, 2*n)
+
+    # Theoretical justification
+        When applying ReLU, assuming that the distribution
+        of the previous output is approximately centered around 0.,
+        you are discarding half of your input. This is inefficient.
+
+        Antirectifier allows to return all-positive outputs like ReLU,
+        without discarding any data.
+
+        Tests on MNIST show that Antirectifier allows to train networks
+        with twice less parameters yet with comparable
+        classification accuracy as an equivalent ReLU-based network.
+    '''
+    @property
+    def output_shape(self):
+        shape = list(self.input_shape)
+        assert len(shape) == 2  # only valid for 2D tensors
+        shape[-1] *= 2
+        return tuple(shape)
+
+    def get_output(self, train):
+        x = self.get_input(train)
+        x -= K.mean(x, axis=1, keepdims=True)
+        x = K.l2_normalize(x, axis=1)
+        pos = K.relu(x)
+        neg = K.relu(-x)
+        return K.concatenate([pos, neg], axis=1)
+
+# global parameters
+batch_size = 128
+nb_classes = 10
+nb_epoch = 40
+
+# the data, shuffled and split between train and test sets
+(X_train, y_train), (X_test, y_test) = mnist.load_data()
+
+X_train = X_train.reshape(60000, 784)
+X_test = X_test.reshape(10000, 784)
+X_train = X_train.astype('float32')
+X_test = X_test.astype('float32')
+X_train /= 255
+X_test /= 255
+print(X_train.shape[0], 'train samples')
+print(X_test.shape[0], 'test samples')
+
+# convert class vectors to binary class matrices
+Y_train = np_utils.to_categorical(y_train, nb_classes)
+Y_test = np_utils.to_categorical(y_test, nb_classes)
+
+# build the model
+model = Sequential()
+model.add(Dense(256, input_shape=(784,)))
+model.add(Antirectifier())
+model.add(Dropout(0.1))
+model.add(Dense(256))
+model.add(Antirectifier())
+model.add(Dropout(0.1))
+model.add(Dense(10))
+model.add(Activation('softmax'))
+
+# compile the model
+model.compile(loss='categorical_crossentropy', optimizer='rmsprop')
+
+# train the model
+model.fit(X_train, Y_train,
+          batch_size=batch_size, nb_epoch=nb_epoch,
+          show_accuracy=True, verbose=1,
+          validation_data=(X_test, Y_test))
+
+# next, compare with an equivalent network
+# with2x bigger Dense layers and ReLU

examples/babi_memnn.py

@@ -18,7 +18,7 @@ from keras.models import Sequential
 from keras.layers.embeddings import Embedding
 from keras.layers.core import Activation, Dense, Merge, Permute, Dropout
 from keras.layers.recurrent import LSTM
-from keras.datasets.data_utils import get_file
+from keras.utils.data_utils import get_file
 from keras.preprocessing.sequence import pad_sequences
 from functools import reduce
 import tarfile

examples/babi_rnn.py

@@ -7,8 +7,8 @@ http://arxiv.org/abs/1502.05698
 
 Task Number                  | FB LSTM Baseline | Keras QA
 ---                          | ---              | ---
-QA1 - Single Supporting Fact | 50               | 52.1
-QA2 - Two Supporting Facts   | 20               | 37.0
+QA1 - Single Supporting Fact | 50               | 100.0
+QA2 - Two Supporting Facts   | 20               | 50.0
 QA3 - Three Supporting Facts | 20               | 20.5
 QA4 - Two Arg. Relations     | 61               | 62.9
 QA5 - Three Arg. Relations   | 70               | 61.9
@@ -34,8 +34,8 @@ https://research.facebook.com/researchers/1543934539189348
 Notes:
 
 - With default word, sentence, and query vector sizes, the GRU model achieves:
-  - 52.1% test accuracy on QA1 in 20 epochs (2 seconds per epoch on CPU)
-  - 37.0% test accuracy on QA2 in 20 epochs (16 seconds per epoch on CPU)
+  - 100% test accuracy on QA1 in 20 epochs (2 seconds per epoch on CPU)
+  - 50% test accuracy on QA2 in 20 epochs (16 seconds per epoch on CPU)
 In comparison, the Facebook paper achieves 50% and 20% for the LSTM baseline.
 
 - The task does not traditionally parse the question separately. This likely
@@ -64,9 +64,9 @@ import tarfile
 import numpy as np
 np.random.seed(1337)  # for reproducibility
 
-from keras.datasets.data_utils import get_file
+from keras.utils.data_utils import get_file
 from keras.layers.embeddings import Embedding
-from keras.layers.core import Dense, Merge
+from keras.layers.core import Dense, Merge, Dropout, RepeatVector
 from keras.layers import recurrent
 from keras.models import Sequential
 from keras.preprocessing.sequence import pad_sequences
@@ -138,12 +138,12 @@ def vectorize_stories(data, word_idx, story_maxlen, query_maxlen):
         Y.append(y)
     return pad_sequences(X, maxlen=story_maxlen), pad_sequences(Xq, maxlen=query_maxlen), np.array(Y)
 
-RNN = recurrent.GRU
+RNN = recurrent.LSTM
 EMBED_HIDDEN_SIZE = 50
 SENT_HIDDEN_SIZE = 100
 QUERY_HIDDEN_SIZE = 100
 BATCH_SIZE = 32
-EPOCHS = 20
+EPOCHS = 40
 print('RNN / Embed / Sent / Query = {}, {}, {}, {}'.format(RNN, EMBED_HIDDEN_SIZE, SENT_HIDDEN_SIZE, QUERY_HIDDEN_SIZE))
 
 path = get_file('babi-tasks-v1-2.tar.gz', origin='http://www.thespermwhale.com/jaseweston/babi/tasks_1-20_v1-2.tar.gz')
@@ -178,15 +178,19 @@ print('story_maxlen, query_maxlen = {}, {}'.format(story_maxlen, query_maxlen))
 print('Build model...')
 
 sentrnn = Sequential()
-sentrnn.add(Embedding(vocab_size, EMBED_HIDDEN_SIZE, mask_zero=True))
-sentrnn.add(RNN(SENT_HIDDEN_SIZE, return_sequences=False))
+sentrnn.add(Embedding(vocab_size, EMBED_HIDDEN_SIZE, input_length=story_maxlen, mask_zero=True))
+sentrnn.add(Dropout(0.3))
 
 qrnn = Sequential()
-qrnn.add(Embedding(vocab_size, EMBED_HIDDEN_SIZE))
-qrnn.add(RNN(QUERY_HIDDEN_SIZE, return_sequences=False))
+qrnn.add(Embedding(vocab_size, EMBED_HIDDEN_SIZE, input_length=query_maxlen))
+qrnn.add(Dropout(0.3))
+qrnn.add(RNN(EMBED_HIDDEN_SIZE, return_sequences=False))
+qrnn.add(RepeatVector(story_maxlen))
 
 model = Sequential()
-model.add(Merge([sentrnn, qrnn], mode='concat'))
+model.add(Merge([sentrnn, qrnn], mode='sum'))
+model.add(RNN(EMBED_HIDDEN_SIZE, return_sequences=False))
+model.add(Dropout(0.3))
 model.add(Dense(vocab_size, activation='softmax'))
 
 model.compile(optimizer='adam', loss='categorical_crossentropy', class_mode='categorical')

examples/cifar10_cnn.py

@@ -17,9 +17,8 @@ from keras.preprocessing.image import ImageDataGenerator
 from keras.models import Sequential
 from keras.layers.core import Dense, Dropout, Activation, Flatten
 from keras.layers.convolutional import Convolution2D, MaxPooling2D
-from keras.optimizers import SGD, Adadelta, Adagrad
-from keras.utils import np_utils, generic_utils
-from six.moves import range
+from keras.optimizers import SGD
+from keras.utils import np_utils
 
 batch_size = 32
 nb_classes = 10
@@ -75,24 +74,23 @@ X_train /= 255
 X_test /= 255
 
 if not data_augmentation:
-    print('Not using data augmentation or normalization')
-    model.fit(X_train, Y_train, batch_size=batch_size, nb_epoch=nb_epoch)
-    score = model.evaluate(X_test, Y_test, batch_size=batch_size)
-    print('Test score:', score)
-
+    print('Not using data augmentation.')
+    model.fit(X_train, Y_train, batch_size=batch_size,
+              nb_epoch=nb_epoch, show_accuracy=True,
+              validation_data=(X_test, Y_test), shuffle=True)
 else:
-    print('Using real time data augmentation')
+    print('Using real-time data augmentation.')
 
     # this will do preprocessing and realtime data augmentation
     datagen = ImageDataGenerator(
-        featurewise_center=True,  # set input mean to 0 over the dataset
+        featurewise_center=False,  # set input mean to 0 over the dataset
         samplewise_center=False,  # set each sample mean to 0
-        featurewise_std_normalization=True,  # divide inputs by std of the dataset
+        featurewise_std_normalization=False,  # divide inputs by std of the dataset
         samplewise_std_normalization=False,  # divide each input by its std
         zca_whitening=False,  # apply ZCA whitening
-        rotation_range=20,  # randomly rotate images in the range (degrees, 0 to 180)
-        width_shift_range=0.2,  # randomly shift images horizontally (fraction of total width)
-        height_shift_range=0.2,  # randomly shift images vertically (fraction of total height)
+        rotation_range=0,  # randomly rotate images in the range (degrees, 0 to 180)
+        width_shift_range=0.1,  # randomly shift images horizontally (fraction of total width)
+        height_shift_range=0.1,  # randomly shift images vertically (fraction of total height)
         horizontal_flip=True,  # randomly flip images
         vertical_flip=False)  # randomly flip images
 
@@ -100,20 +98,9 @@ else:
     # (std, mean, and principal components if ZCA whitening is applied)
     datagen.fit(X_train)
 
-    for e in range(nb_epoch):
-        print('-'*40)
-        print('Epoch', e)
-        print('-'*40)
-        print('Training...')
-        # batch train with realtime data augmentation
-        progbar = generic_utils.Progbar(X_train.shape[0])
-        for X_batch, Y_batch in datagen.flow(X_train, Y_train):
-            loss = model.train_on_batch(X_batch, Y_batch)
-            progbar.add(X_batch.shape[0], values=[('train loss', loss[0])])
-
-        print('Testing...')
-        # test time!
-        progbar = generic_utils.Progbar(X_test.shape[0])
-        for X_batch, Y_batch in datagen.flow(X_test, Y_test):
-            score = model.test_on_batch(X_batch, Y_batch)
-            progbar.add(X_batch.shape[0], values=[('test loss', score[0])])
+    # fit the model on the batches generated by datagen.flow()
+    model.fit_generator(datagen.flow(X_train, Y_train, batch_size=batch_size),
+                        samples_per_epoch=X_train.shape[0],
+                        nb_epoch=nb_epoch, show_accuracy=True,
+                        validation_data=(X_test, Y_test),
+                        nb_worker=1)

examples/conv_filter_visualization.py

@@ -0,0 +1,186 @@
+'''Visualization of the filters of VGG16, via gradient ascent in input space.
+
+This script can run on CPU in a few minutes (with the TensorFlow backend).
+
+Results example: http://i.imgur.com/4nj4KjN.jpg
+
+Before running this script, download the weights for the VGG16 model at:
+https://drive.google.com/file/d/0Bz7KyqmuGsilT0J5dmRCM0ROVHc/view?usp=sharing
+(source: https://gist.github.com/baraldilorenzo/07d7802847aaad0a35d3)
+and make sure the variable `weights_path` in this script matches the location of the file.
+'''
+from __future__ import print_function
+from scipy.misc import imsave
+import numpy as np
+import time
+import os
+import h5py
+
+from keras.models import Sequential
+from keras.layers import Convolution2D, ZeroPadding2D, MaxPooling2D
+from keras import backend as K
+
+# dimensions of the generated pictures for each filter.
+img_width = 128
+img_height = 128
+
+# path to the model weights file.
+weights_path = 'vgg16_weights.h5'
+
+# the name of the layer we want to visualize (see model definition below)
+layer_name = 'conv5_1'
+
+# util function to convert a tensor into a valid image
+def deprocess_image(x):
+    # normalize tensor: center on 0., ensure std is 0.1
+    x -= x.mean()
+    x /= (x.std() + 1e-5)
+    x *= 0.1
+
+    # clip to [0, 1]
+    x += 0.5
+    x = np.clip(x, 0, 1)
+
+    # convert to RGB array
+    x *= 255
+    x = x.transpose((1, 2, 0))
+    x = np.clip(x, 0, 255).astype('uint8')
+    return x
+
+# this will contain our generated image
+input_img = K.placeholder((1, 3, img_width, img_height))
+
+# build the VGG16 network with our input_img as input
+first_layer = ZeroPadding2D((1, 1), input_shape=(3, img_width, img_height))
+first_layer.input = input_img
+
+model = Sequential()
+model.add(first_layer)
+model.add(Convolution2D(64, 3, 3, activation='relu', name='conv1_1'))
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(64, 3, 3, activation='relu', name='conv1_2'))
+model.add(MaxPooling2D((2, 2), strides=(2, 2)))
+
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(128, 3, 3, activation='relu', name='conv2_1'))
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(128, 3, 3, activation='relu', name='conv2_2'))
+model.add(MaxPooling2D((2, 2), strides=(2, 2)))
+
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(256, 3, 3, activation='relu', name='conv3_1'))
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(256, 3, 3, activation='relu', name='conv3_2'))
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(256, 3, 3, activation='relu', name='conv3_3'))
+model.add(MaxPooling2D((2, 2), strides=(2, 2)))
+
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(512, 3, 3, activation='relu', name='conv4_1'))
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(512, 3, 3, activation='relu', name='conv4_2'))
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(512, 3, 3, activation='relu', name='conv4_3'))
+model.add(MaxPooling2D((2, 2), strides=(2, 2)))
+
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(512, 3, 3, activation='relu', name='conv5_1'))
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(512, 3, 3, activation='relu', name='conv5_2'))
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(512, 3, 3, activation='relu', name='conv5_3'))
+model.add(MaxPooling2D((2, 2), strides=(2, 2)))
+
+# load the weights of the VGG16 networks
+# (trained on ImageNet, won the ILSVRC competition in 2014)
+# note: when there is a complete match between your model definition
+# and your weight savefile, you can simply call model.load_weights(filename)
+assert os.path.exists(weights_path), 'Model weights not found (see "weights_path" variable in script).'
+f = h5py.File(weights_path)
+for k in range(f.attrs['nb_layers']):
+    if k >= len(model.layers):
+        # we don't look at the last (fully-connected) layers in the savefile
+        break
+    g = f['layer_{}'.format(k)]
+    weights = [g['param_{}'.format(p)] for p in range(g.attrs['nb_params'])]
+    model.layers[k].set_weights(weights)
+f.close()
+print('Model loaded.')
+
+# get the symbolic outputs of each "key" layer (we gave them unique names).
+layer_dict = dict([(layer.name, layer) for layer in model.layers])
+
+
+def normalize(x):
+    # utility function to normalize a tensor by its L2 norm
+    return x / (K.sqrt(K.mean(K.square(x))) + 1e-5)
+
+
+kept_filters = []
+for filter_index in range(0, 200):
+    # we only scan through the first 200 filters,
+    # but there are actually 512 of them
+    print('Processing filter %d' % filter_index)
+    start_time = time.time()
+
+    # we build a loss function that maximizes the activation
+    # of the nth filter of the layer considered
+    layer_output = layer_dict[layer_name].get_output()
+    loss = K.mean(layer_output[:, filter_index, :, :])
+
+    # we compute the gradient of the input picture wrt this loss
+    grads = K.gradients(loss, input_img)[0]
+
+    # normalization trick: we normalize the gradient
+    grads = normalize(grads)
+
+    # this function returns the loss and grads given the input picture
+    iterate = K.function([input_img], [loss, grads])
+
+    # step size for gradient ascent
+    step = 1.
+
+    # we start from a gray image with some random noise
+    input_img_data = np.random.random((1, 3, img_width, img_height)) * 20 + 128.
+
+    # we run gradient ascent for 20 steps
+    for i in range(20):
+        loss_value, grads_value = iterate([input_img_data])
+        input_img_data += grads_value * step
+
+        print('Current loss value:', loss_value)
+        if loss_value <= 0.:
+            # some filters get stuck to 0, we can skip them
+            break
+
+    # decode the resulting input image
+    if loss_value > 0:
+        img = deprocess_image(input_img_data[0])
+        kept_filters.append((img, loss_value))
+    end_time = time.time()
+    print('Filter %d processed in %ds' % (filter_index, end_time - start_time))
+
+# we will stich the best 64 filters on a 8 x 8 grid.
+n = 8
+
+# the filters that have the highest loss are assumed to be better-looking.
+# we will only keep the top 64 filters.
+kept_filters.sort(key=lambda x: x[1], reverse=True)
+kept_filters = kept_filters[:n * n]
+
+# build a black picture with enough space for
+# our 8 x 8 filters of size 128 x 128, with a 5px margin in between
+margin = 5
+width = n * img_width + (n - 1) * margin
+height = n * img_height + (n - 1) * margin
+stitched_filters = np.zeros((width, height, 3))
+
+# fill the picture with our saved filters
+for i in range(n):
+    for j in range(n):
+        img, loss = kept_filters[i * n + j]
+        stitched_filters[(img_width + margin) * i: (img_width + margin) * i + img_width,
+                         (img_height + margin) * j: (img_height + margin) * j + img_height, :] = img
+
+# save the result to disk
+imsave('stitched_filters_%dx%d.png' % (n, n), stitched_filters)

examples/deep_dream.py

@@ -0,0 +1,235 @@
+'''Deep Dreaming in Keras.
+
+Run the script with:
+```
+python deep_dream.py path_to_your_base_image.jpg prefix_for_results
+```
+e.g.:
+```
+python deep_dream.py img/mypic.jpg results/dream
+```
+
+It is preferrable to run this script on GPU, for speed.
+If running on CPU, prefer the TensorFlow backend (much faster).
+
+Example results: http://i.imgur.com/FX6ROg9.jpg
+'''
+from __future__ import print_function
+from scipy.misc import imread, imresize, imsave
+import numpy as np
+from scipy.optimize import fmin_l_bfgs_b
+import time
+import argparse
+import h5py
+import os
+
+from keras.models import Sequential
+from keras.layers.convolutional import Convolution2D, ZeroPadding2D, MaxPooling2D
+from keras import backend as K
+
+parser = argparse.ArgumentParser(description='Deep Dreams with Keras.')
+parser.add_argument('base_image_path', metavar='base', type=str,
+                    help='Path to the image to transform.')
+parser.add_argument('result_prefix', metavar='res_prefix', type=str,
+                    help='Prefix for the saved results.')
+
+args = parser.parse_args()
+base_image_path = args.base_image_path
+result_prefix = args.result_prefix
+
+# dimensions of the generated picture.
+img_width = 600
+img_height = 600
+
+# path to the model weights file.
+weights_path = 'vgg16_weights.h5'
+
+# some settings we found interesting
+saved_settings = {
+    'bad_trip': {'features': {'conv4_1': 0.05,
+                              'conv4_2': 0.01,
+                              'conv4_3': 0.01},
+                 'continuity': 0.1,
+                 'dream_l2': 0.8,
+                 'jitter': 5},
+    'dreamy': {'features': {'conv5_1': 0.05,
+                            'conv5_2': 0.02},
+               'continuity': 0.1,
+               'dream_l2': 0.02,
+               'jitter': 0},
+}
+# the settings we will use in this experiment
+settings = saved_settings['dreamy']
+
+# util function to open, resize and format pictures into appropriate tensors
+def preprocess_image(image_path):
+    img = imresize(imread(image_path), (img_width, img_height))
+    img = img.transpose((2, 0, 1)).astype('float64')
+    img = np.expand_dims(img, axis=0)
+    return img
+
+# util function to convert a tensor into a valid image
+def deprocess_image(x):
+    x = x.transpose((1, 2, 0))
+    x = np.clip(x, 0, 255).astype('uint8')
+    return x
+
+# this will contain our generated image
+dream = K.placeholder((1, 3, img_width, img_height))
+
+# build the VGG16 network with our dream as input
+first_layer = ZeroPadding2D((1, 1), input_shape=(3, img_width, img_height))
+first_layer.input = dream
+
+model = Sequential()
+model.add(first_layer)
+model.add(Convolution2D(64, 3, 3, activation='relu', name='conv1_1'))
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(64, 3, 3, activation='relu', name='conv1_2'))
+model.add(MaxPooling2D((2, 2), strides=(2, 2)))
+
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(128, 3, 3, activation='relu', name='conv2_1'))
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(128, 3, 3, activation='relu', name='conv2_2'))
+model.add(MaxPooling2D((2, 2), strides=(2, 2)))
+
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(256, 3, 3, activation='relu', name='conv3_1'))
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(256, 3, 3, activation='relu', name='conv3_2'))
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(256, 3, 3, activation='relu', name='conv3_3'))
+model.add(MaxPooling2D((2, 2), strides=(2, 2)))
+
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(512, 3, 3, activation='relu', name='conv4_1'))
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(512, 3, 3, activation='relu', name='conv4_2'))
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(512, 3, 3, activation='relu', name='conv4_3'))
+model.add(MaxPooling2D((2, 2), strides=(2, 2)))
+
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(512, 3, 3, activation='relu', name='conv5_1'))
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(512, 3, 3, activation='relu', name='conv5_2'))
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(512, 3, 3, activation='relu', name='conv5_3'))
+model.add(MaxPooling2D((2, 2), strides=(2, 2)))
+
+# load the weights of the VGG16 networks
+# (trained on ImageNet, won the ILSVRC competition in 2014)
+# note: when there is a complete match between your model definition
+# and your weight savefile, you can simply call model.load_weights(filename)
+assert os.path.exists(weights_path), 'Model weights not found (see "weights_path" variable in script).'
+f = h5py.File(weights_path)
+for k in range(f.attrs['nb_layers']):
+    if k >= len(model.layers):
+        # we don't look at the last (fully-connected) layers in the savefile
+        break
+    g = f['layer_{}'.format(k)]
+    weights = [g['param_{}'.format(p)] for p in range(g.attrs['nb_params'])]
+    model.layers[k].set_weights(weights)
+f.close()
+print('Model loaded.')
+
+# get the symbolic outputs of each "key" layer (we gave them unique names).
+layer_dict = dict([(layer.name, layer) for layer in model.layers])
+
+# continuity loss util function
+def continuity_loss(x):
+    assert K.ndim(x) == 4
+    a = K.square(x[:, :, :img_width-1, :img_height-1] - x[:, :, 1:, :img_height-1])
+    b = K.square(x[:, :, :img_width-1, :img_height-1] - x[:, :, :img_width-1, 1:])
+    return K.sum(K.pow(a + b, 1.25))
+
+# define the loss
+loss = K.variable(0.)
+for layer_name in settings['features']:
+    # add the L2 norm of the features of a layer to the loss
+    assert layer_name in layer_dict.keys(), 'Layer ' + layer_name + ' not found in model.'
+    coeff = settings['features'][layer_name]
+    x = layer_dict[layer_name].get_output()
+    shape = layer_dict[layer_name].output_shape
+    # we avoid border artifacts by only involving non-border pixels in the loss
+    loss -= coeff * K.sum(K.square(x[:, :, 2: shape[2]-2, 2: shape[3]-2])) / np.prod(shape[1:])
+
+# add continuity loss (gives image local coherence, can result in an artful blur)
+loss += settings['continuity'] * continuity_loss(dream) / (3 * img_width * img_height)
+# add image L2 norm to loss (prevents pixels from taking very high values, makes image darker)
+loss += settings['dream_l2'] * K.sum(K.square(dream)) / (3 * img_width * img_height)
+
+# feel free to further modify the loss as you see fit, to achieve new effects...
+
+# compute the gradients of the dream wrt the loss
+grads = K.gradients(loss, dream)
+
+outputs = [loss]
+if type(grads) in {list, tuple}:
+    outputs += grads
+else:
+    outputs.append(grads)
+
+f_outputs = K.function([dream], outputs)
+def eval_loss_and_grads(x):
+    x = x.reshape((1, 3, img_width, img_height))
+    outs = f_outputs([x])
+    loss_value = outs[0]
+    if len(outs[1:]) == 1:
+        grad_values = outs[1].flatten().astype('float64')
+    else:
+        grad_values = np.array(outs[1:]).flatten().astype('float64')
+    return loss_value, grad_values
+
+# this Evaluator class makes it possible
+# to compute loss and gradients in one pass
+# while retrieving them via two separate functions,
+# "loss" and "grads". This is done because scipy.optimize
+# requires separate functions for loss and gradients,
+# but computing them separately would be inefficient.
+class Evaluator(object):
+    def __init__(self):
+        self.loss_value = None
+        self.grads_values = None
+
+    def loss(self, x):
+        assert self.loss_value is None
+        loss_value, grad_values = eval_loss_and_grads(x)
+        self.loss_value = loss_value
+        self.grad_values = grad_values
+        return self.loss_value
+
+    def grads(self, x):
+        assert self.loss_value is not None
+        grad_values = np.copy(self.grad_values)
+        self.loss_value = None
+        self.grad_values = None
+        return grad_values
+
+evaluator = Evaluator()
+
+# run scipy-based optimization (L-BFGS) over the pixels of the generated image
+# so as to minimize the loss
+x = preprocess_image(base_image_path)
+for i in range(5):
+    print('Start of iteration', i)
+    start_time = time.time()
+
+    # add a random offset jitter to the initial image. This will be reverted at decoding time
+    ox, oy = np.random.randint(-settings['jitter'], settings['jitter']+1, 2)
+    x = np.roll(np.roll(x, ox, -1), oy, -2)
+
+    # run L-BFGS for 7 steps
+    x, min_val, info = fmin_l_bfgs_b(evaluator.loss, x.flatten(),
+                                     fprime=evaluator.grads, maxfun=7)
+    print('Current loss value:', min_val)
+    # decode the dream and save it
+    x = x.reshape((3, img_width, img_height))
+    x = np.roll(np.roll(x, -ox, -1), -oy, -2) # unshift image
+    img = deprocess_image(x)
+    fname = result_prefix + '_at_iteration_%d.png' % i
+    imsave(fname, img)
+    end_time = time.time()
+    print('Image saved as', fname)
+    print('Iteration %d completed in %ds' % (i, end_time - start_time))

examples/imdb_bidirectional_lstm.py

@@ -55,7 +55,7 @@ model.compile('adam', {'output': 'binary_crossentropy'})
 print('Train...')
 model.fit({'input': X_train, 'output': y_train},
           batch_size=batch_size,
-          nb_epoch=4)
+          nb_epoch=4, show_accuracy=True)
 acc = accuracy(y_test,
                np.round(np.array(model.predict({'input': X_test},
                                                batch_size=batch_size)['output'])))

examples/imdb_cnn.py

@@ -71,8 +71,7 @@ model.add(Dense(1))
 model.add(Activation('sigmoid'))
 
 model.compile(loss='binary_crossentropy',
-              optimizer='rmsprop',
-              class_mode='binary')
+              optimizer='rmsprop')
 model.fit(X_train, y_train, batch_size=batch_size,
           nb_epoch=nb_epoch, show_accuracy=True,
           validation_data=(X_test, y_test))

examples/imdb_cnn_lstm.py

@@ -2,7 +2,7 @@
 classification task.
 
 GPU command:
-    THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 python imdb_lstm.py
+    THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 python imdb_cnn_lstm.py
 
 Get to 0.8498 test accuracy after 2 epochs. 41s/epoch on K520 GPU.
 '''

examples/imdb_lstm.py

@@ -38,7 +38,7 @@ print('Loading data...')
 print(len(X_train), 'train sequences')
 print(len(X_test), 'test sequences')
 
-print("Pad sequences (samples x time)")
+print('Pad sequences (samples x time)')
 X_train = sequence.pad_sequences(X_train, maxlen=maxlen)
 X_test = sequence.pad_sequences(X_test, maxlen=maxlen)
 print('X_train shape:', X_train.shape)
@@ -46,19 +46,18 @@ print('X_test shape:', X_test.shape)
 
 print('Build model...')
 model = Sequential()
-model.add(Embedding(max_features, 128, input_length=maxlen))
-model.add(LSTM(128))  # try using a GRU instead, for fun
+model.add(Embedding(max_features, 128, input_length=maxlen, dropout=0.5))
+model.add(LSTM(128, dropout_W=0.5, dropout_U=0.1))  # try using a GRU instead, for fun
 model.add(Dropout(0.5))
 model.add(Dense(1))
 model.add(Activation('sigmoid'))
 
 # try using different optimizers and different optimizer configs
 model.compile(loss='binary_crossentropy',
-              optimizer='adam',
-              class_mode="binary")
+              optimizer='adam')
 
-print("Train...")
-model.fit(X_train, y_train, batch_size=batch_size, nb_epoch=3,
+print('Train...')
+model.fit(X_train, y_train, batch_size=batch_size, nb_epoch=15,
           validation_data=(X_test, y_test), show_accuracy=True)
 score, acc = model.evaluate(X_test, y_test,
                             batch_size=batch_size,

examples/lstm_text_generation.py

@@ -14,13 +14,19 @@ from __future__ import print_function
 from keras.models import Sequential
 from keras.layers.core import Dense, Activation, Dropout
 from keras.layers.recurrent import LSTM
-from keras.datasets.data_utils import get_file
+from keras.utils.data_utils import get_file
 import numpy as np
 import random
 import sys
 
 path = get_file('nietzsche.txt', origin="https://s3.amazonaws.com/text-datasets/nietzsche.txt")
-text = open(path).read().lower()
+
+try: 
+    text = open(path).read().lower()
+except UnicodeDecodeError:
+    import codecs
+    text = codecs.open(path, encoding='utf-8').read().lower()
+
 print('corpus length:', len(text))
 
 chars = set(text)

examples/mnist_cnn.py

@@ -29,7 +29,7 @@ nb_pool = 2
 # convolution kernel size
 nb_conv = 3
 
-# the data, shuffled and split between tran and test sets
+# the data, shuffled and split between train and test sets
 (X_train, y_train), (X_test, y_test) = mnist.load_data()
 
 X_train = X_train.reshape(X_train.shape[0], 1, img_rows, img_cols)

examples/mnist_mlp.py

@@ -20,7 +20,7 @@ batch_size = 128
 nb_classes = 10
 nb_epoch = 20
 
-# the data, shuffled and split between tran and test sets
+# the data, shuffled and split between train and test sets
 (X_train, y_train), (X_test, y_test) = mnist.load_data()
 
 X_train = X_train.reshape(60000, 784)

examples/mnist_siamese_graph.py

@@ -0,0 +1,124 @@
+'''Train a Siamese MLP on pairs of digits from the MNIST dataset.
+
+It follows Hadsell-et-al.'06 [1] by computing the Euclidean distance on the
+output of the shared network and by optimizing the contrastive loss (see paper
+for mode details).
+
+[1] "Dimensionality Reduction by Learning an Invariant Mapping"
+    http://yann.lecun.com/exdb/publis/pdf/hadsell-chopra-lecun-06.pdf
+
+Run on GPU: THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 python mnist_siamese_graph.py
+
+Gets to 99.5% test accuracy after 20 epochs.
+3 seconds per epoch on a Titan X GPU
+'''
+from __future__ import absolute_import
+from __future__ import print_function
+import numpy as np
+np.random.seed(1337)  # for reproducibility
+
+import random
+from keras.datasets import mnist
+from keras.models import Sequential, Graph
+from keras.layers.core import Dense, Dropout, Lambda
+from keras.optimizers import SGD, RMSprop
+from keras import backend as K
+
+
+def euclidean_distance(inputs):
+    assert len(inputs) == 2, ('Euclidean distance needs '
+                              '2 inputs, %d given' % len(inputs))
+    u, v = inputs.values()
+    return K.sqrt(K.sum(K.square(u - v), axis=1, keepdims=True))
+
+
+def contrastive_loss(y, d):
+    '''Contrastive loss from Hadsell-et-al.'06
+    http://yann.lecun.com/exdb/publis/pdf/hadsell-chopra-lecun-06.pdf
+    '''
+    margin = 1
+    return K.mean(y * K.square(d) + (1 - y) * K.square(K.maximum(margin - d, 0)))
+
+
+def create_pairs(x, digit_indices):
+    '''Positive and negative pair creation.
+    Alternates between positive and negative pairs.
+    '''
+    pairs = []
+    labels = []
+    n = min([len(digit_indices[d]) for d in range(10)]) - 1
+    for d in range(10):
+        for i in range(n):
+            z1, z2 = digit_indices[d][i], digit_indices[d][i+1]
+            pairs += [[x[z1], x[z2]]]
+            inc = random.randrange(1, 10)
+            dn = (d + inc) % 10
+            z1, z2 = digit_indices[d][i], digit_indices[dn][i]
+            pairs += [[x[z1], x[z2]]]
+            labels += [1, 0]
+    return np.array(pairs), np.array(labels)
+
+
+def create_base_network(input_dim):
+    '''Base network to be shared (eq. to feature extraction).
+    '''
+    seq = Sequential()
+    seq.add(Dense(128, input_shape=(input_dim,), activation='relu'))
+    seq.add(Dropout(0.1))
+    seq.add(Dense(128, activation='relu'))
+    seq.add(Dropout(0.1))
+    seq.add(Dense(128, activation='relu'))
+    return seq
+
+
+def compute_accuracy(predictions, labels):
+    '''Compute classification accuracy with a fixed threshold on distances.
+    '''
+    return labels[predictions.ravel() < 0.5].mean()
+
+
+# the data, shuffled and split between train and test sets
+(X_train, y_train), (X_test, y_test) = mnist.load_data()
+X_train = X_train.reshape(60000, 784)
+X_test = X_test.reshape(10000, 784)
+X_train = X_train.astype('float32')
+X_test = X_test.astype('float32')
+X_train /= 255
+X_test /= 255
+input_dim = 784
+nb_epoch = 20
+
+# create training+test positive and negative pairs
+digit_indices = [np.where(y_train == i)[0] for i in range(10)]
+tr_pairs, tr_y = create_pairs(X_train, digit_indices)
+
+digit_indices = [np.where(y_test == i)[0] for i in range(10)]
+te_pairs, te_y = create_pairs(X_test, digit_indices)
+
+# network definition
+base_network = create_base_network(input_dim)
+
+g = Graph()
+g.add_input(name='input_a', input_shape=(input_dim,))
+g.add_input(name='input_b', input_shape=(input_dim,))
+g.add_shared_node(base_network, name='shared', inputs=['input_a', 'input_b'],
+                  merge_mode='join')
+g.add_node(Lambda(euclidean_distance), name='d', input='shared')
+g.add_output(name='output', input='d')
+
+# train
+rms = RMSprop()
+g.compile(loss={'output': contrastive_loss}, optimizer=rms)
+g.fit({'input_a': tr_pairs[:, 0], 'input_b': tr_pairs[:, 1], 'output': tr_y},
+      validation_data={'input_a': te_pairs[:, 0], 'input_b': te_pairs[:, 1], 'output': te_y},
+      batch_size=128,
+      nb_epoch=nb_epoch)
+
+# compute final accuracy on training and test sets
+pred = g.predict({'input_a': tr_pairs[:, 0], 'input_b': tr_pairs[:, 1]})['output']
+tr_acc = compute_accuracy(pred, tr_y)
+pred = g.predict({'input_a': te_pairs[:, 0], 'input_b': te_pairs[:, 1]})['output']
+te_acc = compute_accuracy(pred, te_y)
+
+print('* Accuracy on training set: %0.2f%%' % (100 * tr_acc))
+print('* Accuracy on test set: %0.2f%%' % (100 * te_acc))

examples/mnist_transfer_cnn.py

@@ -4,7 +4,7 @@
 2- Freeze convolutional layers and fine-tune dense layers
    for the classification of digits [5..9].
 
-Run on GPU: THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 python mnist_cnn.py
+Run on GPU: THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 python mnist_transfer_cnn.py
 
 Get to 99.8% test accuracy after 5 epochs
 for the first five digits classifier

examples/neural_style_transfer.py

@@ -0,0 +1,290 @@
+'''Neural style transfer with Keras.
+
+Before running this script, download the weights for the VGG16 model at:
+https://drive.google.com/file/d/0Bz7KyqmuGsilT0J5dmRCM0ROVHc/view?usp=sharing
+(source: https://gist.github.com/baraldilorenzo/07d7802847aaad0a35d3)
+and make sure the variable `weights_path` in this script matches the location of the file.
+
+Run the script with:
+```
+python neural_style_transfer.py path_to_your_base_image.jpg path_to_your_reference.jpg prefix_for_results
+```
+e.g.:
+```
+python neural_style_transfer.py img/tuebingen.jpg img/starry_night.jpg results/my_result
+```
+
+It is preferrable to run this script on GPU, for speed.
+If running on CPU, prefer the TensorFlow backend (much faster).
+
+Example result: https://twitter.com/fchollet/status/686631033085677568
+
+# Details
+
+Style transfer consists in generating an image
+with the same "content" as a base image, but with the
+"style" of a different picture (typically artistic).
+
+This is achieved through the optimization of a loss function
+that has 3 components: "style loss", "content loss",
+and "total variation loss":
+
+- The total variation loss imposes local spatial continuity between
+the pixels of the combination image, giving it visual coherence.
+
+- The style loss is where the deep learning keeps in --that one is defined
+using a deep convolutional neural network. Precisely, it consists in a sum of
+L2 distances betwen the Gram matrices of the representations of
+the base image and the style reference image, extracted from
+different layers of a convnet (trained on ImageNet). The general idea
+is to capture color/texture information at different spatial
+scales (fairly large scales --defined by the depth of the layer considered).
+
+ - The content loss is a L2 distance between the features of the base
+image (extracted from a deep layer) and the features of the combination image,
+keeping the generated image close enough to the original one.
+
+# References
+    - [A Neural Algorithm of Artistic Style](http://arxiv.org/abs/1508.06576)
+'''
+
+from __future__ import print_function
+from scipy.misc import imread, imresize, imsave
+import numpy as np
+from scipy.optimize import fmin_l_bfgs_b
+import time
+import os
+import argparse
+import h5py
+
+from keras.models import Sequential
+from keras.layers.convolutional import Convolution2D, ZeroPadding2D, MaxPooling2D
+from keras import backend as K
+
+parser = argparse.ArgumentParser(description='Neural style transfer with Keras.')
+parser.add_argument('base_image_path', metavar='base', type=str,
+                    help='Path to the image to transform.')
+parser.add_argument('style_reference_image_path', metavar='ref', type=str,
+                    help='Path to the style reference image.')
+parser.add_argument('result_prefix', metavar='res_prefix', type=str,
+                    help='Prefix for the saved results.')
+
+args = parser.parse_args()
+base_image_path = args.base_image_path
+style_reference_image_path = args.style_reference_image_path
+result_prefix = args.result_prefix
+weights_path = 'vgg16_weights.h5'
+
+# these are the weights of the different loss components
+total_variation_weight = 1.
+style_weight = 1.
+content_weight = 0.025
+
+# dimensions of the generated picture.
+img_width = 400
+img_height = 400
+assert img_height == img_width, 'Due to the use of the Gram matrix, width and height must match.'
+
+# util function to open, resize and format pictures into appropriate tensors
+def preprocess_image(image_path):
+    img = imresize(imread(image_path), (img_width, img_height))
+    img = img.transpose((2, 0, 1)).astype('float64')
+    img = np.expand_dims(img, axis=0)
+    return img
+
+# util function to convert a tensor into a valid image
+def deprocess_image(x):
+    x = x.transpose((1, 2, 0))
+    x = np.clip(x, 0, 255).astype('uint8')
+    return x
+
+# get tensor representations of our images
+base_image = K.variable(preprocess_image(base_image_path))
+style_reference_image = K.variable(preprocess_image(style_reference_image_path))
+
+# this will contain our generated image
+combination_image = K.placeholder((1, 3, img_width, img_height))
+
+# combine the 3 images into a single Keras tensor
+input_tensor = K.concatenate([base_image,
+                              style_reference_image,
+                              combination_image], axis=0)
+
+# build the VGG16 network with our 3 images as input
+first_layer = ZeroPadding2D((1, 1), input_shape=(3, img_width, img_height))
+first_layer.input = input_tensor
+
+model = Sequential()
+model.add(first_layer)
+model.add(Convolution2D(64, 3, 3, activation='relu', name='conv1_1'))
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(64, 3, 3, activation='relu'))
+model.add(MaxPooling2D((2, 2), strides=(2, 2)))
+
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(128, 3, 3, activation='relu', name='conv2_1'))
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(128, 3, 3, activation='relu'))
+model.add(MaxPooling2D((2, 2), strides=(2, 2)))
+
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(256, 3, 3, activation='relu', name='conv3_1'))
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(256, 3, 3, activation='relu'))
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(256, 3, 3, activation='relu'))
+model.add(MaxPooling2D((2, 2), strides=(2, 2)))
+
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(512, 3, 3, activation='relu', name='conv4_1'))
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(512, 3, 3, activation='relu', name='conv4_2'))
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(512, 3, 3, activation='relu'))
+model.add(MaxPooling2D((2, 2), strides=(2, 2)))
+
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(512, 3, 3, activation='relu', name='conv5_1'))
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(512, 3, 3, activation='relu'))
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(512, 3, 3, activation='relu'))
+model.add(MaxPooling2D((2, 2), strides=(2, 2)))
+
+# load the weights of the VGG16 networks
+# (trained on ImageNet, won the ILSVRC competition in 2014)
+# note: when there is a complete match between your model definition
+# and your weight savefile, you can simply call model.load_weights(filename)
+assert os.path.exists(weights_path), 'Model weights not found (see "weights_path" variable in script).'
+f = h5py.File(weights_path)
+for k in range(f.attrs['nb_layers']):
+    if k >= len(model.layers):
+        # we don't look at the last (fully-connected) layers in the savefile
+        break
+    g = f['layer_{}'.format(k)]
+    weights = [g['param_{}'.format(p)] for p in range(g.attrs['nb_params'])]
+    model.layers[k].set_weights(weights)
+f.close()
+print('Model loaded.')
+
+# get the symbolic outputs of each "key" layer (we gave them unique names).
+outputs_dict = dict([(layer.name, layer.get_output()) for layer in model.layers])
+
+# compute the neural style loss
+# first we need to define 4 util functions
+
+# the gram matrix of an image tensor (feature-wise outer product)
+def gram_matrix(x):
+    assert K.ndim(x) == 3
+    features = K.batch_flatten(x)
+    gram = K.dot(features, K.transpose(features))
+    return gram
+
+# the "style loss" is designed to maintain
+# the style of the reference image in the generated image.
+# It is based on the gram matrices (which capture style) of
+# feature maps from the style reference image
+# and from the generated image
+def style_loss(style, combination):
+    assert K.ndim(style) == 3
+    assert K.ndim(combination) == 3
+    S = gram_matrix(style)
+    C = gram_matrix(combination)
+    channels = 3
+    size = img_width * img_height
+    return K.sum(K.square(S - C)) / (4. * (channels ** 2) * (size ** 2))
+
+# an auxiliary loss function
+# designed to maintain the "content" of the
+# base image in the generated image
+def content_loss(base, combination):
+    return K.sum(K.square(combination - base))
+
+# the 3rd loss function, total variation loss,
+# designed to keep the generated image locally coherent
+def total_variation_loss(x):
+    assert K.ndim(x) == 4
+    a = K.square(x[:, :, :img_width-1, :img_height-1] - x[:, :, 1:, :img_height-1])
+    b = K.square(x[:, :, :img_width-1, :img_height-1] - x[:, :, :img_width-1, 1:])
+    return K.sum(K.pow(a + b, 1.25))
+
+# combine these loss functions into a single scalar
+loss = K.variable(0.)
+layer_features = outputs_dict['conv4_2']
+base_image_features = layer_features[0, :, :, :]
+combination_features = layer_features[2, :, :, :]
+loss += content_weight * content_loss(base_image_features,
+                                      combination_features)
+
+feature_layers = ['conv1_1', 'conv2_1', 'conv3_1', 'conv4_1', 'conv5_1']
+for layer_name in feature_layers:
+    layer_features = outputs_dict[layer_name]
+    style_reference_features = layer_features[1, :, :, :]
+    combination_features = layer_features[2, :, :, :]
+    sl = style_loss(style_reference_features, combination_features)
+    loss += (style_weight / len(feature_layers)) * sl
+loss += total_variation_weight * total_variation_loss(combination_image)
+
+# get the gradients of the generated image wrt the loss
+grads = K.gradients(loss, combination_image)
+
+outputs = [loss]
+if type(grads) in {list, tuple}:
+    outputs += grads
+else:
+    outputs.append(grads)
+
+f_outputs = K.function([combination_image], outputs)
+def eval_loss_and_grads(x):
+    x = x.reshape((1, 3, img_width, img_height))
+    outs = f_outputs([x])
+    loss_value = outs[0]
+    if len(outs[1:]) == 1:
+        grad_values = outs[1].flatten().astype('float64')
+    else:
+        grad_values = np.array(outs[1:]).flatten().astype('float64')
+    return loss_value, grad_values
+
+# this Evaluator class makes it possible
+# to compute loss and gradients in one pass
+# while retrieving them via two separate functions,
+# "loss" and "grads". This is done because scipy.optimize
+# requires separate functions for loss and gradients,
+# but computing them separately would be inefficient.
+class Evaluator(object):
+    def __init__(self):
+        self.loss_value = None
+        self.grads_values = None
+
+    def loss(self, x):
+        assert self.loss_value is None
+        loss_value, grad_values = eval_loss_and_grads(x)
+        self.loss_value = loss_value
+        self.grad_values = grad_values
+        return self.loss_value
+
+    def grads(self, x):
+        assert self.loss_value is not None
+        grad_values = np.copy(self.grad_values)
+        self.loss_value = None
+        self.grad_values = None
+        return grad_values
+
+evaluator = Evaluator()
+
+# run scipy-based optimization (L-BFGS) over the pixels of the generated image
+# so as to minimize the neural style loss
+x = np.random.uniform(0, 255, (1, 3, img_width, img_height))
+for i in range(10):
+    print('Start of iteration', i)
+    start_time = time.time()
+    x, min_val, info = fmin_l_bfgs_b(evaluator.loss, x.flatten(),
+                                     fprime=evaluator.grads, maxfun=20)
+    print('Current loss value:', min_val)
+    # save current generated image
+    img = deprocess_image(x.reshape((3, img_width, img_height)))
+    fname = result_prefix + '_at_iteration_%d.png' % i
+    imsave(fname, img)
+    end_time = time.time()
+    print('Image saved as', fname)
+    print('Iteration %d completed in %ds' % (i, end_time - start_time))

examples/stateful_lstm.py

@@ -0,0 +1,85 @@
+'''Example script showing how to use stateful RNNs
+to model long sequences efficiently.
+'''
+from __future__ import print_function
+import numpy as np
+import matplotlib.pyplot as plt
+from keras.models import Sequential
+from keras.layers.core import Dense
+from keras.layers.recurrent import LSTM
+
+
+# since we are using stateful rnn tsteps can be set to 1
+tsteps = 1
+batch_size = 25
+epochs = 25
+# number of elements ahead that are used to make the prediction
+lahead = 1
+
+
+def gen_cosine_amp(amp=100, period=25, x0=0, xn=50000, step=1, k=0.0001):
+    """Generates an absolute cosine time series with the amplitude
+    exponentially decreasing
+
+    Arguments:
+        amp: amplitude of the cosine function
+        period: period of the cosine function
+        x0: initial x of the time series
+        xn: final x of the time series
+        step: step of the time series discretization
+        k: exponential rate
+    """
+    cos = np.zeros(((xn - x0) * step, 1, 1))
+    for i in range(len(cos)):
+        idx = x0 + i * step
+        cos[i, 0, 0] = amp * np.cos(idx / (2 * np.pi * period))
+        cos[i, 0, 0] = cos[i, 0, 0] * np.exp(-k * idx)
+    return cos
+
+
+print('Generating Data')
+cos = gen_cosine_amp()
+print('Input shape:', cos.shape)
+
+expected_output = np.zeros((len(cos), 1))
+for i in range(len(cos) - lahead):
+    expected_output[i, 0] = np.mean(cos[i + 1:i + lahead + 1])
+
+print('Output shape')
+print(expected_output.shape)
+
+print('Creating Model')
+model = Sequential()
+model.add(LSTM(50,
+               batch_input_shape=(batch_size, tsteps, 1),
+               return_sequences=True,
+               stateful=True))
+model.add(LSTM(50,
+               batch_input_shape=(batch_size, tsteps, 1),
+               return_sequences=False,
+               stateful=True))
+model.add(Dense(1))
+model.compile(loss='mse', optimizer='rmsprop')
+
+print('Training')
+for i in range(epochs):
+    print('Epoch', i, '/', epochs)
+    model.fit(cos,
+              expected_output,
+              batch_size=batch_size,
+              verbose=1,
+              nb_epoch=1,
+              shuffle=False)
+    model.reset_states()
+
+print('Predicting')
+predicted_output = model.predict(cos, batch_size=batch_size)
+
+print('Ploting Results')
+plt.subplot(2, 1, 1)
+plt.plot(expected_output)
+plt.title('Expected')
+plt.subplot(2, 1, 2)
+plt.plot(predicted_output)
+plt.title('Predicted')
+plt.show()

keras/__init__.py

@@ -1 +1 @@
-__version__ = '0.3.0'
+__version__ = '0.3.3'

keras/activations.py

@@ -7,11 +7,9 @@ def softmax(x):
     if ndim == 2:
         return K.softmax(x)
     elif ndim == 3:
-        # apply softmax to each timestep
-        def step(x, states):
-            return K.softmax(x), []
-        last_output, outputs, states = K.rnn(step, x, [], masking=False)
-        return outputs
+        e = K.exp(x - K.max(x, axis=-1, keepdims=True))
+        s = K.sum(e, axis=-1, keepdims=True)
+        return e / s
     else:
         raise Exception('Cannot apply softmax to a tensor that is not 2D or 3D. ' +
                         'Here, ndim=' + str(ndim))

keras/backend/__init__.py

@@ -2,6 +2,7 @@ from __future__ import absolute_import
 from __future__ import print_function
 import os
 import json
+import sys
 from .common import epsilon, floatx, set_epsilon, set_floatx
 
 _keras_base_dir = os.path.expanduser('~')
@@ -17,7 +18,7 @@ _config_path = os.path.expanduser(os.path.join(_keras_dir, 'keras.json'))
 if os.path.exists(_config_path):
     _config = json.load(open(_config_path))
     _floatx = _config.get('floatx', floatx())
-    assert _floatx in {'float32', 'float64'}
+    assert _floatx in {'float16', 'float32', 'float64'}
     _epsilon = _config.get('epsilon', epsilon())
     assert type(_epsilon) == float
     _backend = _config.get('backend', _BACKEND)
@@ -41,10 +42,10 @@ if 'KERAS_BACKEND' in os.environ:
     _BACKEND = _backend
 
 if _BACKEND == 'theano':
-    print('Using Theano backend.')
+    sys.stderr.write('Using Theano backend.\n')
     from .theano_backend import *
 elif _BACKEND == 'tensorflow':
-    print('Using TensorFlow backend.')
+    sys.stderr.write('Using TensorFlow backend.\n')
     from .tensorflow_backend import *
 else:
     raise Exception('Unknown backend: ' + str(_BACKEND))

keras/backend/common.py

@@ -20,7 +20,7 @@ def floatx():
 
 def set_floatx(floatx):
     global _FLOATX
-    if floatx not in {'float32', 'float64'}:
+    if floatx not in {'float16', 'float32', 'float64'}:
         raise Exception('Unknown floatx type: ' + str(floatx))
     floatx = str(floatx)
     _FLOATX = floatx

keras/backend/tensorflow_backend.py

@@ -1,5 +1,7 @@
 import tensorflow as tf
 import numpy as np
+import os
+import warnings
 from .common import _FLOATX, _EPSILON
 
 # INTERNAL UTILS
@@ -7,14 +9,18 @@ from .common import _FLOATX, _EPSILON
 _SESSION = None
 
 
-def _get_session():
+def get_session():
     global _SESSION
     if _SESSION is None:
-        _SESSION = tf.Session('')
+        if not os.environ.get('OMP_NUM_THREADS'):
+            _SESSION = tf.Session(config=tf.ConfigProto(allow_soft_placement=True))
+        else:
+            nb_thread = int(os.environ.get('OMP_NUM_THREADS'))
+            _SESSION = tf.Session(config=tf.ConfigProto(intra_op_parallelism_threads=nb_thread, allow_soft_placement=True))
     return _SESSION
 
 
-def _set_session(session):
+def set_session(session):
     global _SESSION
     _SESSION = session
 
@@ -23,7 +29,7 @@ def _set_session(session):
 
 def variable(value, dtype=_FLOATX, name=None):
     v = tf.Variable(np.asarray(value, dtype=dtype), name=name)
-    _get_session().run(v.initializer)
+    get_session().run(v.initializer)
     return v
 
 
@@ -35,7 +41,13 @@ def placeholder(shape=None, ndim=None, dtype=_FLOATX, name=None):
 
 
 def shape(x):
-    return x.get_shape()
+    # symbolic shape
+    return tf.shape(x)
+
+
+def int_shape(x):
+    shape = x.get_shape()
+    return tuple([i.__int__() for i in shape])
 
 
 def ndim(x):
@@ -45,7 +57,7 @@ def ndim(x):
 def eval(x):
     '''Run a graph.
     '''
-    return x.eval(session=_get_session())
+    return x.eval(session=get_session())
 
 
 def zeros(shape, dtype=_FLOATX, name=None):
@@ -57,11 +69,11 @@ def ones(shape, dtype=_FLOATX, name=None):
 
 
 def ones_like(x, name=None):
-    return tf.ones_like(x)
+    return tf.ones_like(x, name=name)
 
 
 def zeros_like(x, name=None):
-    return tf.zeros_like(x)
+    return tf.zeros_like(x, name=name)
 
 
 def count_params(x):
@@ -81,53 +93,76 @@ def dot(x, y):
     return tf.matmul(x, y)
 
 
+def batch_dot(x, y, axes=None):
+    if axes:
+        adj_x = None if axes[0][0] == ndim(x)-1 else True
+        adj_y = True if axes[1][0] == ndim(y)-1 else None
+    else:
+        adj_x = None
+        adj_y = None
+    return tf.batch_matmul(x, y, adj_x=adj_x, adj_y=adj_y)
+
+
 def transpose(x):
     return tf.transpose(x)
 
 
 def gather(reference, indices):
-    '''reference: a tensor.
-    indices: an int tensor of indices.
+    '''
+    # Arguments
+        reference: a tensor.
+        indices: an int tensor of indices.
 
-    Return: a tensor of same type as reference.
+    # Returns
+        a tensor of same type as `reference`.
     '''
     return tf.gather(reference, indices)
 
 
 # ELEMENT-WISE OPERATIONS
 
+def normalize_axis(axis, ndim):
+    if type(axis) is tuple:
+        axis = list(axis)
+    if type(axis) is list:
+        for i, a in enumerate(axis):
+            if a is not None and a < 0:
+                axis[i] = a % ndim
+    else:
+        if axis is not None and axis < 0:
+            axis = axis % ndim
+    return axis
+
+
 def max(x, axis=None, keepdims=False):
-    if axis is not None and axis < 0:
-        axis = axis % len(x.get_shape())
+    axis = normalize_axis(axis, ndim(x))
     return tf.reduce_max(x, reduction_indices=axis, keep_dims=keepdims)
 
 
 def min(x, axis=None, keepdims=False):
-    if axis is not None and axis < 0:
-        axis = axis % len(x.get_shape())
+    axis = normalize_axis(axis, ndim(x))
     return tf.reduce_min(x, reduction_indices=axis, keep_dims=keepdims)
 
 
 def sum(x, axis=None, keepdims=False):
     '''Sum of the values in a tensor, alongside the specified axis.
     '''
-    if axis is not None and axis < 0:
-        axis = axis % len(x.get_shape())
+    axis = normalize_axis(axis, ndim(x))
     return tf.reduce_sum(x, reduction_indices=axis, keep_dims=keepdims)
 
 
 def prod(x, axis=None, keepdims=False):
     '''Multiply the values in a tensor, alongside the specified axis.
     '''
+    axis = normalize_axis(axis, ndim(x))
     return tf.reduce_prod(x, reduction_indices=axis, keep_dims=keepdims)
 
 
 def std(x, axis=None, keepdims=False):
-    if axis is not None and axis < 0:
-        axis = axis % len(x.get_shape())
+    axis = normalize_axis(axis, ndim(x))
     if x.dtype.base_dtype == tf.bool:
         x = tf.cast(x, _FLOATX)
-    m = tf.reduce_mean(x, reduction_indices=axis, keep_dims=keepdims)
+    m = tf.reduce_mean(x, reduction_indices=axis, keep_dims=True)
     devs_squared = tf.square(x - m)
     return tf.sqrt(tf.reduce_mean(devs_squared,
                                   reduction_indices=axis,
@@ -135,8 +170,7 @@ def std(x, axis=None, keepdims=False):
 
 
 def mean(x, axis=None, keepdims=False):
-    if axis is not None and axis < 0:
-        axis = axis % len(x.get_shape())
+    axis = normalize_axis(axis, ndim(x))
     if x.dtype.base_dtype == tf.bool:
         x = tf.cast(x, _FLOATX)
     return tf.reduce_mean(x, reduction_indices=axis, keep_dims=keepdims)
@@ -145,13 +179,12 @@ def mean(x, axis=None, keepdims=False):
 def any(x, axis=None, keepdims=False):
     '''Bitwise reduction (logical OR).
 
-    Return array of int8 (0s and 1s).
+    Return array of uint8 (0s and 1s).
     '''
-    if axis is not None and axis < 0:
-        axis = axis % len(x.get_shape())
+    axis = normalize_axis(axis, ndim(x))
     x = tf.cast(x, tf.bool)
     x = tf.reduce_any(x, reduction_indices=axis, keep_dims=keepdims)
-    return tf.cast(x, tf.int8)
+    return tf.cast(x, tf.uint8)
 
 
 def argmax(x, axis=-1):
@@ -192,6 +225,10 @@ def round(x):
     return tf.round(x)
 
 
+def sign(x):
+    return tf.sign(x)
+
+
 def pow(x, a):
     return tf.pow(x, a)
 
@@ -207,6 +244,10 @@ def equal(x, y):
     return tf.equal(x, y)
 
 
+def not_equal(x, y):
+    return tf.not_equal(x, y)
+
+
 def maximum(x, y):
     return tf.maximum(x, y)
 
@@ -219,7 +260,10 @@ def minimum(x, y):
 
 def concatenate(tensors, axis=-1):
     if axis < 0:
-        axis = axis % len(tensors[0].get_shape())
+        if len(tensors[0].get_shape()):
+            axis = axis % len(tensors[0].get_shape())
+        else:
+            axis = 0
     return tf.concat(axis, tensors)
 
 
@@ -230,12 +274,34 @@ def reshape(x, shape):
 def permute_dimensions(x, pattern):
     '''Transpose dimensions.
 
-    pattern should be a tuple or list of
-    dimension indices, e.g. [0, 2, 1].
+    # Arguments
+        pattern: should be a tuple or list of
+            dimension indices, e.g. [0, 2, 1].
     '''
     return tf.transpose(x, perm=pattern)
 
 
+def resize_images(X, height_factor, width_factor, dim_ordering):
+    '''Resize the images contained in a 4D tensor of shape
+    - [batch, channels, height, width] (for 'th' dim_ordering)
+    - [batch, height, width, channels] (for 'tf' dim_ordering)
+    by a factor of (height_factor, width_factor). Both factors should be
+    positive integers.
+    '''
+    if dim_ordering == 'th':
+        new_shape = tf.shape(X)[2:]
+        new_shape *= tf.constant(np.array([height_factor, width_factor]).astype('int32'))
+        X = permute_dimensions(X, [0, 2, 3, 1])
+        X = tf.image.resize_nearest_neighbor(X, new_shape)
+        return permute_dimensions(X, [0, 3, 1, 2])
+    elif dim_ordering == 'tf':
+        new_shape = tf.shape(X)[1:3]
+        new_shape *= tf.constant(np.array([height_factor, width_factor]).astype('int32'))
+        return tf.image.resize_nearest_neighbor(X, new_shape)
+    else:
+        raise Exception('Invalid dim_ordering: ' + dim_ordering)
+
+
 def repeat_elements(x, rep, axis):
     '''Repeats the elements of a tensor along an axis, like np.repeat
 
@@ -256,6 +322,7 @@ def repeat(x, n):
     if x has shape (samples, dim) and n=2,
     the output will have shape (samples, 2, dim)
     '''
+    assert ndim(x) == 2
     tensors = [x] * n
     stacked = tf.pack(tensors)
     return tf.transpose(stacked, (1, 0, 2))
@@ -266,6 +333,10 @@ def tile(x, n):
 
 
 def flatten(x):
+    return tf.reshape(x, [-1])
+
+
+def batch_flatten(x):
     '''Turn a n-D tensor into a 2D tensor where
     the first dimension is conserved.
     '''
@@ -307,16 +378,21 @@ def spatial_2d_padding(x, padding=(1, 1), dim_ordering='th'):
     return tf.pad(x, pattern)
 
 
+def pack(x):
+    return tf.pack(x)
+
+
 # VALUE MANIPULATION
 
+
 def get_value(x):
     '''Technically the same as eval() for TF.
     '''
-    return x.eval(session=_get_session())
+    return x.eval(session=get_session())
 
 
 def set_value(x, value):
-    tf.assign(x, np.asarray(value)).op.run(session=_get_session())
+    tf.assign(x, np.asarray(value)).op.run(session=get_session())
 
 
 # GRAPH MANIPULATION
@@ -324,20 +400,30 @@ def set_value(x, value):
 class Function(object):
 
     def __init__(self, inputs, outputs, updates=[]):
+        assert type(inputs) in {list, tuple}, 'Input to a TensorFlow backend function should be a list or tuple.'
+        assert type(outputs) in {list, tuple}, 'Output to a TensorFlow backend function should be a list or tuple.'
+        assert type(updates) in {list, tuple}, 'Updates in a TensorFlow backend function should be a list or tuple.'
         self.inputs = list(inputs)
         self.outputs = list(outputs)
         with tf.control_dependencies(self.outputs):
             self.updates = [tf.assign(p, new_p) for (p, new_p) in updates]
 
     def __call__(self, inputs):
+        assert type(inputs) in {list, tuple}
         names = [v.name for v in self.inputs]
         feed_dict = dict(zip(names, inputs))
-        session = _get_session()
+        session = get_session()
         updated = session.run(self.outputs + self.updates, feed_dict=feed_dict)
         return updated[:len(self.outputs)]
 
 
-def function(inputs, outputs, updates=[]):
+def function(inputs, outputs, updates=[], **kwargs):
+    if len(kwargs) > 0:
+        msg = [
+            "Expected no kwargs, you passed %s" % len(kwargs),
+            "kwargs passed to function are ignored with Tensorflow backend"
+        ]
+        warnings.warn('\n'.join(msg))
     return Function(inputs, outputs, updates=updates)
 
 
@@ -348,84 +434,114 @@ def gradients(loss, variables):
 # CONTROL FLOW
 
 def rnn(step_function, inputs, initial_states,
-        go_backwards=False, masking=True):
+        go_backwards=False, mask=None, constants=None):
     '''Iterates over the time dimension of a tensor.
 
-    Parameters
-    ----------
-    inputs: tensor of temporal data of shape (samples, time, ...)
-        (at least 3D).
-    step_function:
-        Parameters:
-            input: tensor with shape (samples, ...) (no time dimension),
-                representing input for the batch of samples at a certain
-                time step.
-            states: list of tensors.
-        Returns:
-            output: tensor with shape (samples, ...) (no time dimension),
-            new_states: list of tensors, same length and shapes
-                as 'states'.
-    initial_states: tensor with shape (samples, ...) (no time dimension),
-        containing the initial values for the states used in
-        the step function.
-    go_backwards: boolean. If True, do the iteration over
-        the time dimension in reverse order.
-    masking: boolean. If true, any input timestep inputs[s, i]
-        that is all-zeros will be skipped (states will be passed to
-        the next step unchanged) and the corresponding output will
-        be all zeros.
+    # Arguments
+        inputs: tensor of temporal data of shape (samples, time, ...)
+            (at least 3D).
+        step_function:
+            Parameters:
+                input: tensor with shape (samples, ...) (no time dimension),
+                    representing input for the batch of samples at a certain
+                    time step.
+                states: list of tensors.
+            Returns:
+                output: tensor with shape (samples, ...) (no time dimension),
+                new_states: list of tensors, same length and shapes
+                    as 'states'.
+        initial_states: tensor with shape (samples, ...) (no time dimension),
+            containing the initial values for the states used in
+            the step function.
+        go_backwards: boolean. If True, do the iteration over
+            the time dimension in reverse order.
+        mask: binary tensor with shape (samples, time, 1),
+            with a zero for every element that is masked.
+        constants: a list of constant values passed at each step.
 
-    Returns
-    -------
-    A tuple (last_output, outputs, new_states).
-        last_output: the latest output of the rnn, of shape (samples, ...)
-        outputs: tensor with shape (samples, time, ...) where each
-            entry outputs[s, t] is the output of the step function
-            at time t for sample s.
-        new_states: list of tensors, latest states returned by
-            the step function, of shape (samples, ...).
+    # Returns
+        A tuple (last_output, outputs, new_states).
+            last_output: the latest output of the rnn, of shape (samples, ...)
+            outputs: tensor with shape (samples, time, ...) where each
+                entry outputs[s, t] is the output of the step function
+                at time t for sample s.
+            new_states: list of tensors, latest states returned by
+                the step function, of shape (samples, ...).
     '''
-    inputs = tf.transpose(inputs, (1, 0, 2))
+    ndim = len(inputs.get_shape())
+    assert ndim >= 3, "Input should be at least 3D."
+    axes = [1, 0] + list(range(2, ndim))
+    inputs = tf.transpose(inputs, (axes))
     input_list = tf.unpack(inputs)
+    if constants is None:
+        constants = []
 
     states = initial_states
     successive_states = []
     successive_outputs = []
     if go_backwards:
         input_list.reverse()
-    for input in input_list:
-        output, new_states = step_function(input, states)
-        if masking:
-            # for now we raise an exception because tf.reduce_any will not work
-            raise Exception("Masking is Theano-only for the time being.")
 
-            # if all-zero input timestep, return
-            # all-zero output and unchanged states
-            switch = tf.reduce_any(input)
-            output = tf.python.control_flow_ops.cond(switch,
-                                                     lambda: output,
-                                                     lambda: 0. * output)
+    if mask is not None:
+        # Transpose not supported by bool tensor types, hence round-trip to uint8.
+        mask = tf.cast(mask, tf.uint8)
+        if len(mask.get_shape()) == ndim-1:
+            mask = expand_dims(mask)
+        mask = tf.cast(tf.transpose(mask, axes), tf.bool)
+        mask_list = tf.unpack(mask)
+
+        if go_backwards:
+            mask_list.reverse()
+
+        for input, mask_t in zip(input_list, mask_list):
+            output, new_states = step_function(input, states + constants)
+
+            # tf.select needs its condition tensor to be the same shape as its two
+            # result tensors, but in our case the condition (mask) tensor is
+            # (nsamples, 1), and A and B are (nsamples, ndimensions). So we need to
+            # broadcast the mask to match the shape of A and B. That's what the
+            # tile call does, is just repeat the mask along its second dimension
+            # ndimensions times.
+            tiled_mask_t = tf.tile(mask_t, tf.pack([1, tf.shape(output)[1]]))
+
+            if len(successive_outputs) == 0:
+                prev_output = zeros_like(output)
+            else:
+                prev_output = successive_outputs[-1]
+
+            output = tf.select(tiled_mask_t, output, prev_output)
+
             return_states = []
             for state, new_state in zip(states, new_states):
-                return_states.append(tf.python.control_flow_ops.cond(switch,
-                                                                     lambda: new_state,
-                                                                     lambda: state))
+                # (see earlier comment for tile explanation)
+                tiled_mask_t = tf.tile(mask_t, tf.pack([1, tf.shape(new_state)[1]]))
+                return_states.append(tf.select(tiled_mask_t, new_state, state))
+
             states = return_states
-        else:
-            states = new_states
-        successive_outputs.append(output)
-        successive_states.append(states)
+            successive_outputs.append(output)
+            successive_states.append(states)
+    else:
+        for input in input_list:
+            output, states = step_function(input, states + constants)
+            successive_outputs.append(output)
+            successive_states.append(states)
 
     last_output = successive_outputs[-1]
     outputs = tf.pack(successive_outputs)
     new_states = successive_states[-1]
 
-    outputs = tf.transpose(outputs, (1, 0, 2))
-    return last_output, outputs, states
+    axes = [1, 0] + list(range(2, len(outputs.get_shape())))
+    outputs = tf.transpose(outputs, axes)
+    return last_output, outputs, new_states
 
 
 def switch(condition, then_expression, else_expression):
-    '''condition: scalar tensor.
+    '''Switch between two operations depending on a scalar value.
+
+    # Arguments
+        condition: scalar tensor.
+        then_expression: TensorFlow operation.
+        else_expression: TensorFlow operation.
     '''
     return tf.python.control_flow_ops.cond(condition,
                                            lambda: then_expression,
@@ -437,14 +553,18 @@ def switch(condition, then_expression, else_expression):
 def relu(x, alpha=0., max_value=None):
     '''ReLU.
 
-    alpha: slope of negative section.
+    # Arguments
+        alpha: slope of negative section.
+        max_value: saturation threshold.
     '''
     negative_part = tf.nn.relu(-x)
     x = tf.nn.relu(x)
     if max_value is not None:
         x = tf.clip_by_value(x, tf.cast(0., dtype=_FLOATX),
                              tf.cast(max_value, dtype=_FLOATX))
-    x -= tf.constant(alpha, dtype=_FLOATX) * negative_part
+    if isinstance(alpha, (tuple, list, np.ndarray)) or np.isscalar(alpha):
+        alpha = tf.constant(alpha, dtype=_FLOATX)
+    x -= alpha * negative_part
     return x
 
 
@@ -463,13 +583,13 @@ def categorical_crossentropy(output, target, from_logits=False):
     if not from_logits:
         # scale preds so that the class probas of each sample sum to 1
         output /= tf.reduce_sum(output,
-                                reduction_indices=len(output.get_shape())-1,
+                                reduction_indices=len(output.get_shape()) - 1,
                                 keep_dims=True)
         # manual computation of crossentropy
         output = tf.clip_by_value(output, tf.cast(_EPSILON, dtype=_FLOATX),
-                                  tf.cast(1.-_EPSILON, dtype=_FLOATX))
+                                  tf.cast(1. - _EPSILON, dtype=_FLOATX))
         return - tf.reduce_sum(target * tf.log(output),
-                               reduction_indices=len(output.get_shape())-1)
+                               reduction_indices=len(output.get_shape()) - 1)
     else:
         return tf.nn.softmax_cross_entropy_with_logits(output, target)
 
@@ -510,16 +630,23 @@ def dropout(x, level, seed=None):
     return tf.nn.dropout(x * 1., retain_prob, seed=seed)
 
 
+def l2_normalize(x, axis):
+    if axis < 0:
+        axis = axis % len(x.get_shape())
+    return tf.nn.l2_normalize(x, dim=axis)
+
+
 # CONVOLUTIONS
 
 
 def conv2d(x, kernel, strides=(1, 1), border_mode='valid', dim_ordering='th',
            image_shape=None, filter_shape=None):
-    '''
-    Run on cuDNN if available.
-    border_mode: string, "same" or "valid".
-    dim_ordering: whether to use Theano or TensorFlow dimension ordering
-    in inputs/kernels/ouputs.
+    '''Runs on cuDNN if available.
+
+    # Arguments
+        border_mode: string, "same" or "valid".
+        dim_ordering: whether to use Theano or TensorFlow dimension ordering
+        in inputs/kernels/ouputs.
     '''
     if border_mode == 'same':
         padding = 'SAME'
@@ -559,10 +686,11 @@ def conv2d(x, kernel, strides=(1, 1), border_mode='valid', dim_ordering='th',
 def pool2d(x, pool_size, strides=(1, 1),
            border_mode='valid', dim_ordering='th', pool_mode='max'):
     '''
-    pool_size: tuple of 2 integers.
-    strides: tuple of 2 integers.
-    border_mode: one of "valid", "same".
-    dim_ordering: one of "th", "tf".
+    # Arguments
+        pool_size: tuple of 2 integers.
+        strides: tuple of 2 integers.
+        border_mode: one of "valid", "same".
+        dim_ordering: one of "th", "tf".
     '''
     if border_mode == 'same':
         padding = 'SAME'
@@ -617,3 +745,10 @@ def random_uniform(shape, low=0.0, high=1.0, dtype=_FLOATX, seed=None):
         seed = np.random.randint(10e6)
     return tf.random_uniform(shape, minval=low, maxval=high,
                              dtype=dtype, seed=seed)
+
+
+def random_binomial(shape, p=0.0, dtype=_FLOATX, seed=None):
+    if seed is None:
+        seed = np.random.randint(10e6)
+    return tf.select(tf.random_uniform(shape, dtype=dtype, seed=seed) <= p,
+                     tf.ones(shape), tf.zeros(shape))

keras/backend/theano_backend.py

@@ -1,7 +1,9 @@
 import theano
 from theano import tensor as T
 from theano.sandbox.rng_mrg import MRG_RandomStreams as RandomStreams
-from theano.tensor.signal import downsample
+from theano.tensor.signal import pool
+from theano.tensor.nnet import conv3d2d
+import inspect
 import numpy as np
 from .common import _FLOATX, _EPSILON
 
@@ -10,21 +12,6 @@ from .common import _FLOATX, _EPSILON
 theano.config.floatX = _FLOATX
 
 
-def _on_gpu():
-    '''Return whether the session is set to
-    run on GPU or not (i.e. on CPU).
-    '''
-    return theano.config.device[:3] == 'gpu'
-
-
-if _on_gpu():
-    '''Import cuDNN only if running on GPU:
-    not having Cuda installed should not
-    prevent from running the present code.
-    '''
-    from theano.sandbox.cuda import dnn
-
-
 # VARIABLE MANIPULATION
 
 def variable(value, dtype=_FLOATX, name=None):
@@ -41,18 +28,9 @@ def placeholder(shape=None, ndim=None, dtype=_FLOATX, name=None):
         raise Exception('Specify either a shape or ndim value.')
     if shape is not None:
         ndim = len(shape)
-    if ndim == 0:
-        return T.scalar(name=name, dtype=dtype)
-    elif ndim == 1:
-        return T.vector(name=name, dtype=dtype)
-    elif ndim == 2:
-        return T.matrix(name=name, dtype=dtype)
-    elif ndim == 3:
-        return T.tensor3(name=name, dtype=dtype)
-    elif ndim == 4:
-        return T.tensor4(name=name, dtype=dtype)
-    else:
-        raise Exception('ndim too large: ' + str(ndim))
+
+    broadcast = (False,) * ndim
+    return T.TensorType(dtype, broadcast)(name)
 
 
 def shape(x):
@@ -118,6 +96,13 @@ def dot(x, y):
     return T.dot(x, y)
 
 
+def batch_dot(x, y, axes=None):
+    if axes is None:
+        # behaves like tf.batch_matmul as default
+        axes = [(x.ndim-1,), (y.ndim-2,)]
+    return T.batched_tensordot(x, y, axes=axes)
+
+
 def transpose(x):
     return T.transpose(x)
 
@@ -155,7 +140,10 @@ def prod(x, axis=None, keepdims=False):
 
 
 def mean(x, axis=None, keepdims=False):
-    return T.mean(x, axis=axis, keepdims=keepdims)
+    dtype = None
+    if 'int' in x.dtype:
+        dtype = _FLOATX
+    return T.mean(x, axis=axis, keepdims=keepdims, dtype=dtype)
 
 
 def std(x, axis=None, keepdims=False):
@@ -201,6 +189,10 @@ def round(x):
     return T.round(x)
 
 
+def sign(x):
+    return T.sgn(x)
+
+
 def pow(x, a):
     return T.pow(x, a)
 
@@ -215,6 +207,10 @@ def equal(x, y):
     return T.eq(x, y)
 
 
+def not_equal(x, y):
+    return T.neq(x, y)
+
+
 def maximum(x, y):
     return T.maximum(x, y)
 
@@ -251,15 +247,56 @@ def repeat_elements(x, rep, axis):
     '''
     return T.repeat(x, rep, axis=axis)
 
+
+def resize_images(X, height_factor, width_factor, dim_ordering):
+    '''Resize the images contained in a 4D tensor of shape
+    - [batch, channels, height, width] (for 'th' dim_ordering)
+    - [batch, height, width, channels] (for 'tf' dim_ordering)
+    by a factor of (height_factor, width_factor). Both factors should be
+    positive integers.
+    '''
+    if dim_ordering == 'th':
+        output = repeat_elements(X, height_factor, axis=2)
+        output = repeat_elements(output, width_factor, axis=3)
+        return output
+    elif dim_ordering == 'tf':
+        output = repeat_elements(X, height_factor, axis=1)
+        output = repeat_elements(output, width_factor, axis=2)
+        return output
+    else:
+        raise Exception('Invalid dim_ordering: ' + dim_ordering)
+
+
+def resize_volumes(X, depth_factor, height_factor, width_factor, dim_ordering):
+    '''Resize the volume contained in a 5D tensor of shape
+    - [batch, channels, depth, height, width] (for 'th' dim_ordering)
+    - [batch, depth, height, width, channels] (for 'tf' dim_ordering)
+    by a factor of (depth_factor, height_factor, width_factor).
+    Both factors should be positive integers.
+    '''
+    if dim_ordering == 'th':
+        output = repeat_elements(X, depth_factor, axis=2)
+        output = repeat_elements(output, height_factor, axis=3)
+        output = repeat_elements(output, width_factor, axis=4)
+        return output
+    elif dim_ordering == 'tf':
+        output = repeat_elements(X, depth_factor, axis=1)
+        output = repeat_elements(output, height_factor, axis=2)
+        output = repeat_elements(output, width_factor, axis=3)
+        return output
+    else:
+        raise Exception('Invalid dim_ordering: ' + dim_ordering)
+
+
 def repeat(x, n):
     '''Repeat a 2D tensor.
 
     If x has shape (samples, dim) and n=2,
     the output will have shape (samples, 2, dim).
     '''
-    tensors = [x] * n
-    stacked = T.stack(*tensors)
-    return stacked.dimshuffle((1, 0, 2))
+    assert x.ndim == 2
+    x = x.dimshuffle((0, 'x', 1))
+    return T.extra_ops.repeat(x, n, axis=1)
 
 
 def tile(x, n):
@@ -267,6 +304,10 @@ def tile(x, n):
 
 
 def flatten(x):
+    return T.flatten(x)
+
+
+def batch_flatten(x):
     '''Turn a n-D tensor into a 2D tensor where
     the first dimension is conserved.
     '''
@@ -339,6 +380,45 @@ def spatial_2d_padding(x, padding=(1, 1), dim_ordering='th'):
         raise Exception('Invalid dim_ordering: ' + dim_ordering)
     return T.set_subtensor(output[indices], x)
 
+
+def spatial_3d_padding(x, padding=(1, 1, 1), dim_ordering='th'):
+    '''Pad the 2nd, 3rd and 4th dimensions of a 5D tensor
+    with "padding[0]", "padding[1]" and "padding[2]" (resp.) zeros left and right.
+    '''
+    input_shape = x.shape
+    if dim_ordering == 'th':
+        output_shape = (input_shape[0],
+                        input_shape[1],
+                        input_shape[2] + 2 * padding[0],
+                        input_shape[3] + 2 * padding[1],
+                        input_shape[4] + 2 * padding[2])
+        output = T.zeros(output_shape)
+        indices = (slice(None),
+                   slice(None),
+                   slice(padding[0], input_shape[2] + padding[0]),
+                   slice(padding[1], input_shape[3] + padding[1]),
+                   slice(padding[2], input_shape[4] + padding[2]))
+
+    elif dim_ordering == 'tf':
+        output_shape = (input_shape[0],
+                        input_shape[1] + 2 * padding[0],
+                        input_shape[2] + 2 * padding[1],
+                        input_shape[3] + 2 * padding[2],
+                        input_shape[4])
+        output = T.zeros(output_shape)
+        indices = (slice(None),
+                   slice(padding[0], input_shape[1] + padding[0]),
+                   slice(padding[1], input_shape[2] + padding[1]),
+                   slice(padding[2], input_shape[3] + padding[2]),
+                   slice(None))
+    else:
+        raise Exception('Invalid dim_ordering: ' + dim_ordering)
+    return T.set_subtensor(output[indices], x)
+
+
+def pack(x):
+    return T.stack(*x)
+
 # VALUE MANIPULATION
 
 
@@ -362,11 +442,18 @@ class Function(object):
                                         allow_input_downcast=True, **kwargs)
 
     def __call__(self, inputs):
+        assert type(inputs) in {list, tuple}
         return self.function(*inputs)
 
 
-def function(inputs, outputs, updates=[]):
-    return Function(inputs, outputs, updates=updates)
+def function(inputs, outputs, updates=[], **kwargs):
+    if len(kwargs) > 0:
+        function_args = inspect.getargspec(theano.function)[0]
+        for key in kwargs.keys():
+            if key not in function_args:
+                msg = "Invalid argument '%s' passed to K.function" % key
+                raise ValueError(msg)
+    return Function(inputs, outputs, updates=updates, **kwargs)
 
 
 def gradients(loss, variables):
@@ -376,64 +463,86 @@ def gradients(loss, variables):
 # CONTROL FLOW
 
 def rnn(step_function, inputs, initial_states,
-        go_backwards=False, masking=True):
-    '''Iterate over the time dimension of a tensor.
+        go_backwards=False, mask=None, constants=None):
+    '''Iterates over the time dimension of a tensor.
 
-    Parameters
-    ----------
-    inputs: tensor of temporal data of shape (samples, time, ...)
-        (at least 3D).
-    step_function:
-        Parameters:
-            input: tensor with shape (samples, ...) (no time dimension),
-                representing input for the batch of samples at a certain
-                time step.
-            states: list of tensors.
-        Returns:
-            output: tensor with shape (samples, ...) (no time dimension),
-            new_states: list of tensors, same length and shapes
-                as 'states'.
-    initial_states: tensor with shape (samples, ...) (no time dimension),
-        containing the initial values for the states used in
-        the step function.
-    go_backwards: boolean. If True, do the iteration over
-        the time dimension in reverse order.
-    masking: boolean. If true, any input timestep inputs[s, i]
-        that is all-zeros will be skipped (states will be passed to
-        the next step unchanged) and the corresponding output will
-        be all zeros.
+    # Arguments
+        inputs: tensor of temporal data of shape (samples, time, ...)
+            (at least 3D).
+        step_function:
+            Parameters:
+                input: tensor with shape (samples, ...) (no time dimension),
+                    representing input for the batch of samples at a certain
+                    time step.
+                states: list of tensors.
+            Returns:
+                output: tensor with shape (samples, ...) (no time dimension),
+                new_states: list of tensors, same length and shapes
+                    as 'states'.
+        initial_states: tensor with shape (samples, ...) (no time dimension),
+            containing the initial values for the states used in
+            the step function.
+        go_backwards: boolean. If True, do the iteration over
+            the time dimension in reverse order.
+        mask: binary tensor with shape (samples, time),
+            with a zero for every element that is masked.
+        constants: a list of constant values passed at each step.
 
-    Returns
-    -------
-    A tuple (last_output, outputs, new_states).
-        last_output: the latest output of the rnn, of shape (samples, ...)
-        outputs: tensor with shape (samples, time, ...) where each
-            entry outputs[s, t] is the output of the step function
-            at time t for sample s.
-        new_states: list of tensors, latest states returned by
-            the step function, of shape (samples, ...).
+
+    # Returns
+        A tuple (last_output, outputs, new_states).
+            last_output: the latest output of the rnn, of shape (samples, ...)
+            outputs: tensor with shape (samples, time, ...) where each
+                entry outputs[s, t] is the output of the step function
+                at time t for sample s.
+            new_states: list of tensors, latest states returned by
+                the step function, of shape (samples, ...).
     '''
-    inputs = inputs.dimshuffle((1, 0, 2))
+    ndim = inputs.ndim
+    assert ndim >= 3, 'Input should be at least 3D.'
 
-    def _step(input, *states):
-        output, new_states = step_function(input, states)
-        if masking:
-            # if all-zero input timestep, return
-            # all-zero output and unchanged states
-            switch = T.any(input, axis=-1, keepdims=True)
-            output = T.switch(switch, output, 0. * output)
+    axes = [1, 0] + list(range(2, ndim))
+    inputs = inputs.dimshuffle(axes)
+
+    if mask is not None:
+        if mask.ndim == ndim-1:
+            mask = expand_dims(mask)
+        assert mask.ndim == ndim
+        mask = mask.dimshuffle(axes)
+
+        if constants is None:
+            constants = []
+        # build an all-zero tensor of shape (samples, output_dim)
+        initial_output = step_function(inputs[0], initial_states + constants)[0] * 0
+        # Theano gets confused by broadcasting patterns in the scan op
+        initial_output = T.unbroadcast(initial_output, 0, 1)
+
+        def _step(input, mask, output_tm1, *states):
+            output, new_states = step_function(input, states)
+            # output previous output if masked.
+            output = T.switch(mask, output, output_tm1)
             return_states = []
             for state, new_state in zip(states, new_states):
-                return_states.append(T.switch(switch, new_state, state))
+                return_states.append(T.switch(mask, new_state, state))
             return [output] + return_states
-        else:
+
+        results, _ = theano.scan(
+            _step,
+            sequences=[inputs, mask],
+            outputs_info=[initial_output] + initial_states,
+            non_sequences=constants,
+            go_backwards=go_backwards)
+    else:
+        def _step(input, *states):
+            output, new_states = step_function(input, states)
             return [output] + new_states
 
-    results, _ = theano.scan(
-        _step,
-        sequences=inputs,
-        outputs_info=[None] + initial_states,
-        go_backwards=go_backwards)
+        results, _ = theano.scan(
+            _step,
+            sequences=inputs,
+            outputs_info=[None] + initial_states,
+            non_sequences=constants,
+            go_backwards=go_backwards)
 
     # deal with Theano API inconsistency
     if type(results) is list:
@@ -446,7 +555,8 @@ def rnn(step_function, inputs, initial_states,
     outputs = T.squeeze(outputs)
     last_output = outputs[-1]
 
-    outputs = outputs.dimshuffle((1, 0, 2))
+    axes = [1, 0] + list(range(2, outputs.ndim))
+    outputs = outputs.dimshuffle(axes)
     states = [T.squeeze(state[-1]) for state in states]
     return last_output, outputs, states
 
@@ -460,6 +570,10 @@ def switch(condition, then_expression, else_expression):
 # NN OPERATIONS
 
 def relu(x, alpha=0., max_value=None):
+    assert hasattr(T.nnet, 'relu'), ('It looks like like your version of '
+                                     'Theano is out of date. '
+                                     'Install the latest version with:\n'
+                                     'pip install git+git://github.com/Theano/Theano.git --upgrade --no-deps')
     x = T.nnet.relu(x, alpha)
     if max_value is not None:
         x = T.minimum(x, max_value)
@@ -517,13 +631,16 @@ def dropout(x, level, seed=None):
     return x
 
 
-# CONVOLUTIONS
+def l2_normalize(x, axis):
+    norm = T.sqrt(T.sum(T.square(x), axis=axis, keepdims=True))
+    return x / norm
 
 
+# CONVOLUTIONS
+
 def conv2d(x, kernel, strides=(1, 1), border_mode='valid', dim_ordering='th',
            image_shape=None, filter_shape=None):
     '''
-    Run on cuDNN if available.
     border_mode: string, "same" or "valid".
     '''
     if dim_ordering not in {'th', 'tf'}:
@@ -545,49 +662,153 @@ def conv2d(x, kernel, strides=(1, 1), border_mode='valid', dim_ordering='th',
             filter_shape = (filter_shape[3], filter_shape[2],
                             filter_shape[0], filter_shape[1])
 
-    if _on_gpu() and dnn.dnn_available():
-        if border_mode == 'same':
-            assert(strides == (1, 1))
-            conv_out = dnn.dnn_conv(img=x,
-                                    kerns=kernel,
-                                    border_mode='full')
-            shift_x = (kernel.shape[2] - 1) // 2
-            shift_y = (kernel.shape[3] - 1) // 2
-            conv_out = conv_out[:, :,
-                                shift_x:x.shape[2] + shift_x,
-                                shift_y:x.shape[3] + shift_y]
-        else:
-            conv_out = dnn.dnn_conv(img=x,
-                                    kerns=kernel,
-                                    border_mode=border_mode,
-                                    subsample=strides)
+    if border_mode == 'same':
+        th_border_mode = 'half'
+        np_kernel = kernel.eval()
+        assert strides[0] <= np_kernel.shape[2], 'strides should be smaller than the convolution window.'
+        assert strides[1] <= np_kernel.shape[3], 'strides should be smaller than the convolution window.'
+    elif border_mode == 'valid':
+        th_border_mode = 'valid'
     else:
-        if border_mode == 'same':
-            th_border_mode = 'full'
-            assert(strides == (1, 1))
-        elif border_mode == 'valid':
-            th_border_mode = 'valid'
-        else:
-            raise Exception('Border mode not supported: ' + str(border_mode))
+        raise Exception('Border mode not supported: ' + str(border_mode))
+
+    # Theano might not accept like longs
+    def int_or_none(value):
+        try:
+            return int(value)
+        except TypeError:
+            return None
+
+    if image_shape is not None:
+        image_shape = tuple(int_or_none(v) for v in image_shape)
+
+    if filter_shape is not None:
+        filter_shape = tuple(int_or_none(v) for v in filter_shape)
+
+    conv_out = T.nnet.conv2d(x, kernel,
+                             border_mode=th_border_mode,
+                             subsample=strides,
+                             input_shape=image_shape,
+                             filter_shape=filter_shape)
+
+    if border_mode == 'same':
+        if np_kernel.shape[2] % 2 == 0:
+            conv_out = conv_out[:,:,:(x.shape[2]+strides[0]-1) // strides[0],:]
+        if np_kernel.shape[3] % 2 == 0:
+            conv_out = conv_out[:,:,:,:(x.shape[3]+strides[1]-1) // strides[1]]
 
-        conv_out = T.nnet.conv.conv2d(x, kernel,
-                                      border_mode=th_border_mode,
-                                      subsample=strides,
-                                      image_shape=image_shape,
-                                      filter_shape=filter_shape)
-        if border_mode == 'same':
-            shift_x = (kernel.shape[2] - 1) // 2
-            shift_y = (kernel.shape[3] - 1) // 2
-            conv_out = conv_out[:, :,
-                                shift_x:x.shape[2] + shift_x,
-                                shift_y:x.shape[3] + shift_y]
     if dim_ordering == 'tf':
         conv_out = conv_out.dimshuffle((0, 2, 3, 1))
     return conv_out
 
 
+def conv3d(x, kernel, strides=(1, 1, 1),
+           border_mode='valid', dim_ordering='th',
+           volume_shape=None, filter_shape=None):
+    '''
+    Run on cuDNN if available.
+    border_mode: string, "same" or "valid".
+    '''
+    if dim_ordering not in {'th', 'tf'}:
+        raise Exception('Unknown dim_ordering ' + str(dim_ordering))
+
+    if border_mode not in {'same', 'valid'}:
+        raise Exception('Invalid border mode: ' + str(border_mode))
+
+    if dim_ordering == 'tf':
+        # TF uses the last dimension as channel dimension,
+        # instead of the 2nd one.
+        # TH input shape: (samples, input_depth, conv_dim1, conv_dim2, conv_dim3)
+        # TF input shape: (samples, conv_dim1, conv_dim2, conv_dim3, input_depth)
+        # TH kernel shape: (out_depth, input_depth, kernel_dim1, kernel_dim2, kernel_dim3)
+        # TF kernel shape: (kernel_dim1, kernel_dim2, kernel_dim3, input_depth, out_depth)
+        x = x.dimshuffle((0, 4, 1, 2, 3))
+        kernel = kernel.dimshuffle((4, 3, 0, 1, 2))
+        if volume_shape:
+            volume_shape = (volume_shape[0], volume_shape[4],
+                            volume_shape[1], volume_shape[2], volume_shape[3])
+        if filter_shape:
+            filter_shape = (filter_shape[4], filter_shape[3],
+                            filter_shape[0], filter_shape[1], filter_shape[2])
+
+    if border_mode == 'same':
+        assert(strides == (1, 1, 1))
+        pad_dim1 = (kernel.shape[2] - 1)
+        pad_dim2 = (kernel.shape[3] - 1)
+        pad_dim3 = (kernel.shape[4] - 1)
+        output_shape = (x.shape[0], x.shape[1],
+                        x.shape[2] + pad_dim1,
+                        x.shape[3] + pad_dim2,
+                        x.shape[4] + pad_dim3)
+        output = T.zeros(output_shape)
+        indices = (slice(None), slice(None),
+                   slice(pad_dim1 // 2, x.shape[2] + pad_dim1 // 2),
+                   slice(pad_dim2 // 2, x.shape[3] + pad_dim2 // 2),
+                   slice(pad_dim3 // 2, x.shape[4] + pad_dim3 // 2))
+        x = T.set_subtensor(output[indices], x)
+        border_mode = 'valid'
+
+    border_mode_3d = (border_mode, border_mode, border_mode)
+    conv_out = conv3d2d.conv3d(signals=x.dimshuffle(0, 2, 1, 3, 4),
+                               filters=kernel.dimshuffle(0, 2, 1, 3, 4),
+                               border_mode=border_mode_3d)
+    conv_out = conv_out.dimshuffle(0, 2, 1, 3, 4)
+
+    # support strides by manually slicing the output
+    if strides != (1, 1, 1):
+        conv_out = conv_out[:, :, ::strides[0], ::strides[1], ::strides[2]]
+
+    if dim_ordering == 'tf':
+        conv_out = conv_out.dimshuffle((0, 2, 3, 4, 1))
+
+    return conv_out
+
+
 def pool2d(x, pool_size, strides=(1, 1), border_mode='valid',
            dim_ordering='th', pool_mode='max'):
+    if border_mode == 'same':
+        w_pad = pool_size[0] - 2 if pool_size[0] % 2 == 1 else pool_size[0] - 1
+        h_pad = pool_size[1] - 2 if pool_size[1] % 2 == 1 else pool_size[1] - 1
+        padding = (w_pad, h_pad)
+    elif border_mode == 'valid':
+        padding = (0, 0)
+    else:
+        raise Exception('Invalid border mode: ' + str(border_mode))
+
+    if dim_ordering not in {'th', 'tf'}:
+        raise Exception('Unknown dim_ordering ' + str(dim_ordering))
+
+    if dim_ordering == 'tf':
+        x = x.dimshuffle((0, 3, 1, 2))
+
+    if pool_mode == 'max':
+        pool_out = pool.pool_2d(x, ds=pool_size, st=strides,
+                                ignore_border=True,
+                                padding=padding,
+                                mode='max')
+    elif pool_mode == 'avg':
+        pool_out = pool.pool_2d(x, ds=pool_size, st=strides,
+                                ignore_border=True,
+                                padding=padding,
+                                mode='average_exc_pad')
+    else:
+        raise Exception('Invalid pooling mode: ' + str(pool_mode))
+
+    if border_mode == 'same':
+        expected_width = (x.shape[2] + strides[0] - 1) // strides[0]
+        expected_height = (x.shape[3] + strides[1] - 1) // strides[1]
+
+        pool_out = pool_out[:, :,
+                            : expected_width,
+                            : expected_height]
+
+    if dim_ordering == 'tf':
+        pool_out = pool_out.dimshuffle((0, 2, 3, 1))
+    return pool_out
+
+
+def pool3d(x, pool_size, strides=(1, 1, 1), border_mode='valid',
+           dim_ordering='th', pool_mode='max'):
     if border_mode == 'same':
         # TODO: add implementation for border_mode="same"
         raise Exception('border_mode="same" not supported with Theano.')
@@ -601,25 +822,49 @@ def pool2d(x, pool_size, strides=(1, 1), border_mode='valid',
         raise Exception('Unknown dim_ordering ' + str(dim_ordering))
 
     if dim_ordering == 'tf':
-        x = x.dimshuffle((0, 3, 1, 2))
+        x = x.dimshuffle((0, 4, 1, 2, 3))
 
     if pool_mode == 'max':
-        pool_out = downsample.max_pool_2d(x, ds=pool_size, st=strides,
-                                          ignore_border=ignore_border,
-                                          padding=padding,
-                                          mode='max')
+        # pooling over conv_dim2, conv_dim1 (last two channels)
+        output = pool.pool_2d(input=x.dimshuffle(0, 1, 4, 3, 2),
+                              ds=(pool_size[1], pool_size[0]),
+                              st=(strides[1], strides[0]),
+                              ignore_border=ignore_border,
+                              padding=padding,
+                              mode='max')
+
+        # pooling over conv_dim3
+        pool_out = pool.pool_2d(input=output.dimshuffle(0, 1, 4, 3, 2),
+                                ds=(1, pool_size[2]),
+                                st=(1, strides[2]),
+                                ignore_border=ignore_border,
+                                padding=padding,
+                                mode='max')
+
     elif pool_mode == 'avg':
-        pool_out = downsample.max_pool_2d(x, ds=pool_size, st=strides,
-                                          ignore_border=ignore_border,
-                                          padding=padding,
-                                          mode='average_exc_pad')
+        # pooling over conv_dim2, conv_dim1 (last two channels)
+        output = pool.pool_2d(input=x.dimshuffle(0, 1, 4, 3, 2),
+                              ds=(pool_size[1], pool_size[0]),
+                              st=(strides[1], strides[0]),
+                              ignore_border=ignore_border,
+                              padding=padding,
+                              mode='average_exc_pad')
+
+        # pooling over conv_dim3
+        pool_out = pool.pool_2d(input=output.dimshuffle(0, 1, 4, 3, 2),
+                                ds=(1, pool_size[2]),
+                                st=(1, strides[2]),
+                                ignore_border=ignore_border,
+                                padding=padding,
+                                mode='average_exc_pad')
     else:
         raise Exception('Invalid pooling mode: ' + str(pool_mode))
 
     if dim_ordering == 'tf':
-        pool_out = pool_out.dimshuffle((0, 2, 3, 1))
+        pool_out = pool_out.dimshuffle((0, 2, 3, 4, 1))
     return pool_out
 
+
 # RANDOMNESS
 
 
@@ -636,6 +881,13 @@ def random_uniform(shape, low=0.0, high=1.0, dtype=_FLOATX, seed=None):
     rng = RandomStreams(seed=seed)
     return rng.uniform(shape, low=low, high=high, dtype=dtype)
 
+
+def random_binomial(shape, p=0.0, dtype=_FLOATX, seed=None):
+    if seed is None:
+        seed = np.random.randint(10e6)
+    rng = RandomStreams(seed=seed)
+    return rng.binomial(shape, p=p, dtype=dtype)
+
 '''
 more TODO:
 

keras/callbacks.py

@@ -92,7 +92,8 @@ class Callback(object):
     will include the following quantities in the `logs` that
     it passes to its callbacks:
 
-        on_epoch_end: logs optionally include `val_loss`
+        on_epoch_end: logs include `acc` and `loss`, and
+            optionally include `val_loss`
             (if validation is enabled in `fit`), and `val_acc`
             (if validation and accuracy monitoring are enabled).
         on_batch_begin: logs include `size`,
@@ -129,11 +130,35 @@ class Callback(object):
 
 
 class BaseLogger(Callback):
-    '''Callback that prints events to the standard output.
+    '''Callback that accumulates epoch averages of
+    the metrics being monitored.
 
     This callback is automatically applied to
-    every Keras model (it is the basis of the verbosity modes
-    in models).
+    every Keras model.
+    '''
+    def on_epoch_begin(self, epoch, logs={}):
+        self.seen = 0
+        self.totals = {}
+
+    def on_batch_end(self, batch, logs={}):
+        batch_size = logs.get('size', 0)
+        self.seen += batch_size
+
+        for k, v in logs.items():
+            if k in self.totals:
+                self.totals[k] += v * batch_size
+            else:
+                self.totals[k] = v * batch_size
+
+    def on_epoch_end(self, epoch, logs={}):
+        for k in self.params['metrics']:
+            if k in self.totals:
+                # make value available to next callbacks
+                logs[k] = self.totals[k] / self.seen
+
+
+class ProgbarLogger(Callback):
+    '''Callback that prints metrics to stdout.
     '''
     def on_train_begin(self, logs={}):
         self.verbose = self.params['verbose']
@@ -145,7 +170,6 @@ class BaseLogger(Callback):
             self.progbar = Progbar(target=self.params['nb_sample'],
                                    verbose=self.verbose)
         self.seen = 0
-        self.totals = {}
 
     def on_batch_begin(self, batch, logs={}):
         if self.seen < self.params['nb_sample']:
@@ -155,11 +179,6 @@ class BaseLogger(Callback):
         batch_size = logs.get('size', 0)
         self.seen += batch_size
 
-        for k, v in logs.items():
-            if k in self.totals:
-                self.totals[k] += v * batch_size
-            else:
-                self.totals[k] = v * batch_size
         for k in self.params['metrics']:
             if k in logs:
                 self.log_values.append((k, logs[k]))
@@ -171,8 +190,6 @@ class BaseLogger(Callback):
 
     def on_epoch_end(self, epoch, logs={}):
         for k in self.params['metrics']:
-            if k in self.totals:
-                self.log_values.append((k, self.totals[k] / self.seen))
             if k in logs:
                 self.log_values.append((k, logs[k]))
         if self.verbose:
@@ -191,26 +208,8 @@ class History(Callback):
         self.epoch = []
         self.history = {}
 
-    def on_epoch_begin(self, epoch, logs={}):
-        self.seen = 0
-        self.totals = {}
-
-    def on_batch_end(self, batch, logs={}):
-        batch_size = logs.get('size', 0)
-        self.seen += batch_size
-        for k, v in logs.items():
-            if k in self.totals:
-                self.totals[k] += v * batch_size
-            else:
-                self.totals[k] = v * batch_size
-
     def on_epoch_end(self, epoch, logs={}):
         self.epoch.append(epoch)
-        for k, v in self.totals.items():
-            if k not in self.history:
-                self.history[k] = []
-            self.history[k].append(v / self.seen)
-
         for k, v in logs.items():
             if k not in self.history:
                 self.history[k] = []
@@ -256,7 +255,7 @@ class ModelCheckpoint(Callback):
 
         if mode not in ['auto', 'min', 'max']:
             warnings.warn('ModelCheckpoint mode %s is unknown, '
-                          'fallback to auto mode.' % (self.mode),
+                          'fallback to auto mode.' % (mode),
                           RuntimeWarning)
             mode = 'auto'
 
@@ -373,26 +372,10 @@ class RemoteMonitor(Callback):
     def __init__(self, root='http://localhost:9000'):
         self.root = root
 
-    def on_epoch_begin(self, epoch, logs={}):
-        self.seen = 0
-        self.totals = {}
-
-    def on_batch_end(self, batch, logs={}):
-        batch_size = logs.get('size', 0)
-        self.seen += batch_size
-        for k, v in logs.items():
-            if k in self.totals:
-                self.totals[k] += v * batch_size
-            else:
-                self.totals[k] = v * batch_size
-
     def on_epoch_end(self, epoch, logs={}):
         import requests
         send = {}
         send['epoch'] = epoch
-
-        for k, v in self.totals.items():
-            send[k] = v / self.seen
         for k, v in logs.items():
             send[k] = v
 
@@ -456,14 +439,15 @@ class TensorBoard(Callback):
                             'with the TensorFlow backend.')
         self.log_dir = log_dir
         self.histogram_freq = histogram_freq
+        self.merged = None
 
     def _set_model(self, model):
         import tensorflow as tf
         import keras.backend.tensorflow_backend as KTF
 
         self.model = model
-        self.sess = KTF._get_session()
-        if self.histogram_freq:
+        self.sess = KTF.get_session()
+        if self.histogram_freq and not self.merged:
             mod_type = self.model.get_config()['name']
             if mod_type == 'Sequential':
                 layers = {l.get_config()['name']: l for l in self.model.layers}
@@ -485,19 +469,6 @@ class TensorBoard(Callback):
         self.writer = tf.train.SummaryWriter(self.log_dir,
                                              self.sess.graph_def)
 
-    def on_epoch_begin(self, epoch, logs={}):
-        self.seen = 0
-        self.totals = {}
-
-    def on_batch_end(self, batch, logs={}):
-        batch_size = logs.get('size', 0)
-        self.seen += batch_size
-        for k, v in logs.items():
-            if k in self.totals:
-                self.totals[k] += v * batch_size
-            else:
-                self.totals[k] = v * batch_size
-
     def on_epoch_end(self, epoch, logs={}):
         import tensorflow as tf
 
@@ -508,12 +479,14 @@ class TensorBoard(Callback):
                 else:
                     test_function = self.model._test
                 names = [v.name for v in test_function.inputs]
+                # TODO: implement batched calls to sess.run
+                # (current call will likely go OOM on GPU)
                 feed_dict = dict(zip(names, self.model.validation_data))
                 result = self.sess.run([self.merged], feed_dict=feed_dict)
                 summary_str = result[0]
                 self.writer.add_summary(summary_str, epoch)
 
-        for name, value in self.totals.items() + logs.items():
+        for name, value in logs.items():
             if name in ['batch', 'size']:
                 continue
             summary = tf.Summary()

keras/constraints.py

@@ -11,29 +11,77 @@ class Constraint(object):
 
 
 class MaxNorm(Constraint):
-    def __init__(self, m=2):
+    '''Constrain the weights incident to each hidden unit to have a norm less than or equal to a desired value.
+
+    # Arguments
+        m: the maximum norm for the incoming weights.
+        axis: integer, axis along which to calculate weight norms. For instance,
+            in a `Dense` layer the weight matrix has shape (input_dim, output_dim),
+            set `axis` to `0` to constrain each weight vector of length (input_dim).
+            In a `MaxoutDense` layer the weight tensor has shape (nb_feature, input_dim, output_dim),
+            set `axis` to `1` to constrain each weight vector of length (input_dim),
+            i.e. constrain the filters incident to the `max` operation.
+            In a `Convolution2D` layer with the Theano backend, the weight tensor
+            has shape (nb_filter, stack_size, nb_row, nb_col), set `axis` to `[1,2,3]`
+            to constrain the weights of each filter tensor of size (stack_size, nb_row, nb_col).
+            In a `Convolution2D` layer with the TensorFlow backend, the weight tensor
+            has shape (nb_row, nb_col, stack_size, nb_filter), set `axis` to `[0,1,2]`
+            to constrain the weights of each filter tensor of size (nb_row, nb_col, stack_size).
+
+    # References
+        - [Dropout: A Simple Way to Prevent Neural Networks from Overfitting Srivastava, Hinton, et al. 2014](http://www.cs.toronto.edu/~rsalakhu/papers/srivastava14a.pdf)
+    '''
+    def __init__(self, m=2, axis=0):
         self.m = m
+        self.axis = axis
 
     def __call__(self, p):
-        norms = K.sqrt(K.sum(K.square(p), axis=0))
+        norms = K.sqrt(K.sum(K.square(p), axis=self.axis, keepdims=True))
         desired = K.clip(norms, 0, self.m)
-        p = p * (desired / (1e-7 + norms))
+        p = p * (desired / (K.epsilon() + norms))
         return p
 
     def get_config(self):
         return {"name": self.__class__.__name__,
-                "m": self.m}
+                "m": self.m,
+                "axis": self.axis}
 
 
 class NonNeg(Constraint):
+    '''Constrain the weights to be non-negative.
+    '''
     def __call__(self, p):
         p *= K.cast(p >= 0., K.floatx())
         return p
 
 
 class UnitNorm(Constraint):
+    '''Constrain the weights incident to each hidden unit to have unit norm.
+
+    # Arguments
+        axis: integer, axis along which to calculate weight norms. For instance,
+            in a `Dense` layer the weight matrix has shape (input_dim, output_dim),
+            set `axis` to `0` to constrain each weight vector of length (input_dim).
+            In a `MaxoutDense` layer the weight tensor has shape (nb_feature, input_dim, output_dim),
+            set `axis` to `1` to constrain each weight vector of length (input_dim),
+            i.e. constrain the filters incident to the `max` operation.
+            In a `Convolution2D` layer with the Theano backend, the weight tensor
+            has shape (nb_filter, stack_size, nb_row, nb_col), set `axis` to `[1,2,3]`
+            to constrain the weights of each filter tensor of size (stack_size, nb_row, nb_col).
+            In a `Convolution2D` layer with the TensorFlow backend, the weight tensor
+            has shape (nb_row, nb_col, stack_size, nb_filter), set `axis` to `[0,1,2]`
+            to constrain the weights of each filter tensor of size (nb_row, nb_col, stack_size).
+    '''
+    def __init__(self, axis=0):
+        self.axis = axis
+
     def __call__(self, p):
-        return p / K.sqrt(K.sum(K.square(p), axis=-1, keepdims=True))
+        return p / (K.epsilon() + K.sqrt(K.sum(K.square(p), axis=self.axis, keepdims=True)))
+
+    def get_config(self):
+        return {"name": self.__class__.__name__,
+                "axis": self.axis}
+
 
 identity = Constraint
 maxnorm = MaxNorm
@@ -41,5 +89,7 @@ nonneg = NonNeg
 unitnorm = UnitNorm
 
 from .utils.generic_utils import get_from_module
+
+
 def get(identifier, kwargs=None):
     return get_from_module(identifier, globals(), 'constraint', instantiate=True, kwargs=kwargs)

keras/datasets/cifar.py

@@ -4,6 +4,7 @@ import sys
 from six.moves import cPickle
 from six.moves import range
 
+
 def load_batch(fpath, label_key='labels'):
     f = open(fpath, 'rb')
     if sys.version_info < (3,):

keras/datasets/cifar10.py

@@ -1,6 +1,6 @@
 from __future__ import absolute_import
 from .cifar import load_batch
-from .data_utils import get_file
+from ..utils.data_utils import get_file
 import numpy as np
 import os
 

keras/datasets/cifar100.py

@@ -1,6 +1,6 @@
 from __future__ import absolute_import
 from .cifar import load_batch
-from .data_utils import get_file
+from ..utils.data_utils import get_file
 import numpy as np
 import os
 

keras/datasets/data_utils.py

@@ -1,53 +1,4 @@
-from __future__ import absolute_import
-from __future__ import print_function
+from ..utils.data_utils import *
+import warnings
 
-import tarfile
-import os
-from six.moves.urllib.request import FancyURLopener
-
-from ..utils.generic_utils import Progbar
-
-
-class ParanoidURLopener(FancyURLopener):
-    def http_error_default(self, url, fp, errcode, errmsg, headers):
-        raise Exception('URL fetch failure on {}: {} -- {}'.format(url, errcode, errmsg))
-
-
-def get_file(fname, origin, untar=False):
-    datadir_base = os.path.expanduser(os.path.join('~', '.keras'))
-    if not os.access(datadir_base, os.W_OK):
-        datadir_base = os.path.join('/tmp', '.keras')
-    datadir = os.path.join(datadir_base, 'datasets')
-    if not os.path.exists(datadir):
-        os.makedirs(datadir)
-
-    if untar:
-        untar_fpath = os.path.join(datadir, fname)
-        fpath = untar_fpath + '.tar.gz'
-    else:
-        fpath = os.path.join(datadir, fname)
-
-    if not os.path.exists(fpath):
-        print('Downloading data from',  origin)
-        global progbar
-        progbar = None
-
-        def dl_progress(count, block_size, total_size):
-            global progbar
-            if progbar is None:
-                progbar = Progbar(total_size)
-            else:
-                progbar.update(count*block_size)
-
-        ParanoidURLopener().retrieve(origin, fpath, dl_progress)
-        progbar = None
-
-    if untar:
-        if not os.path.exists(untar_fpath):
-            print('Untaring file...')
-            tfile = tarfile.open(fpath, 'r:gz')
-            tfile.extractall(path=datadir)
-            tfile.close()
-        return untar_fpath
-
-    return fpath
+warnings.warn('data_utils has been moved to keras.utils.data_utils.')

keras/datasets/imdb.py

@@ -1,7 +1,7 @@
 from __future__ import absolute_import
 from six.moves import cPickle
 import gzip
-from .data_utils import get_file
+from ..utils.data_utils import get_file
 from six.moves import zip
 import numpy as np
 

keras/datasets/mnist.py

@@ -1,6 +1,6 @@
 # -*- coding: utf-8 -*-
 import gzip
-from .data_utils import get_file
+from ..utils.data_utils import get_file
 from six.moves import cPickle
 import sys
 

keras/datasets/reuters.py

@@ -1,6 +1,6 @@
 # -*- coding: utf-8 -*-
 from __future__ import absolute_import
-from .data_utils import get_file
+from ..utils.data_utils import get_file
 from six.moves import cPickle
 from six.moves import zip
 import numpy as np

keras/initializations.py

@@ -3,58 +3,77 @@ import numpy as np
 from . import backend as K
 
 
-def get_fans(shape):
-    fan_in = shape[0] if len(shape) == 2 else np.prod(shape[1:])
-    fan_out = shape[1] if len(shape) == 2 else shape[0]
+def get_fans(shape, dim_ordering='th'):
+    if len(shape) == 2:
+        fan_in = shape[0]
+        fan_out = shape[1]
+    elif len(shape) == 4 or len(shape) == 5:
+        # assuming convolution kernels (2D or 3D).
+        # TH kernel shape: (depth, input_depth, ...)
+        # TF kernel shape: (..., input_depth, depth)
+        if dim_ordering == 'th':
+            fan_in = np.prod(shape[1:])
+            fan_out = shape[0]
+        elif dim_ordering == 'tf':
+            fan_in = np.prod(shape[:-1])
+            fan_out = shape[-1]
+        else:
+            raise Exception('Invalid dim_ordering: ' + dim_ordering)
+    else:
+        # no specific assumptions
+        fan_in = np.sqrt(np.prod(shape))
+        fan_out = np.sqrt(np.prod(shape))
     return fan_in, fan_out
 
 
-def uniform(shape, scale=0.05):
-    return K.variable(np.random.uniform(low=-scale, high=scale, size=shape))
+def uniform(shape, scale=0.05, name=None):
+    return K.variable(np.random.uniform(low=-scale, high=scale, size=shape),
+                      name=name)
 
 
-def normal(shape, scale=0.05):
-    return K.variable(np.random.normal(loc=0.0, scale=scale, size=shape))
+def normal(shape, scale=0.05, name=None):
+    return K.variable(np.random.normal(loc=0.0, scale=scale, size=shape),
+                      name=name)
 
 
-def lecun_uniform(shape):
+def lecun_uniform(shape, name=None, dim_ordering='th'):
     ''' Reference: LeCun 98, Efficient Backprop
         http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf
     '''
-    fan_in, fan_out = get_fans(shape)
+    fan_in, fan_out = get_fans(shape, dim_ordering=dim_ordering)
     scale = np.sqrt(3. / fan_in)
-    return uniform(shape, scale)
+    return uniform(shape, scale, name=name)
 
 
-def glorot_normal(shape):
+def glorot_normal(shape, name=None, dim_ordering='th'):
     ''' Reference: Glorot & Bengio, AISTATS 2010
     '''
-    fan_in, fan_out = get_fans(shape)
+    fan_in, fan_out = get_fans(shape, dim_ordering=dim_ordering)
     s = np.sqrt(2. / (fan_in + fan_out))
-    return normal(shape, s)
+    return normal(shape, s, name=name)
 
 
-def glorot_uniform(shape):
-    fan_in, fan_out = get_fans(shape)
+def glorot_uniform(shape, name=None, dim_ordering='th'):
+    fan_in, fan_out = get_fans(shape, dim_ordering=dim_ordering)
     s = np.sqrt(6. / (fan_in + fan_out))
-    return uniform(shape, s)
+    return uniform(shape, s, name=name)
 
 
-def he_normal(shape):
+def he_normal(shape, name=None, dim_ordering='th'):
     ''' Reference:  He et al., http://arxiv.org/abs/1502.01852
     '''
-    fan_in, fan_out = get_fans(shape)
+    fan_in, fan_out = get_fans(shape, dim_ordering=dim_ordering)
     s = np.sqrt(2. / fan_in)
-    return normal(shape, s)
+    return normal(shape, s, name=name)
 
 
-def he_uniform(shape):
-    fan_in, fan_out = get_fans(shape)
+def he_uniform(shape, name=None, dim_ordering='th'):
+    fan_in, fan_out = get_fans(shape, dim_ordering=dim_ordering)
     s = np.sqrt(6. / fan_in)
-    return uniform(shape, s)
+    return uniform(shape, s, name=name)
 
 
-def orthogonal(shape, scale=1.1):
+def orthogonal(shape, scale=1.1, name=None):
     ''' From Lasagne. Reference: Saxe et al., http://arxiv.org/abs/1312.6120
     '''
     flat_shape = (shape[0], np.prod(shape[1:]))
@@ -63,25 +82,26 @@ def orthogonal(shape, scale=1.1):
     # pick the one with the correct shape
     q = u if u.shape == flat_shape else v
     q = q.reshape(shape)
-    return K.variable(scale * q[:shape[0], :shape[1]])
+    return K.variable(scale * q[:shape[0], :shape[1]], name=name)
 
 
-def identity(shape, scale=1):
+def identity(shape, scale=1, name=None):
     if len(shape) != 2 or shape[0] != shape[1]:
         raise Exception('Identity matrix initialization can only be used '
                         'for 2D square matrices.')
     else:
-        return K.variable(scale * np.identity(shape[0]))
+        return K.variable(scale * np.identity(shape[0]), name=name)
 
 
-def zero(shape):
-    return K.zeros(shape)
+def zero(shape, name=None):
+    return K.zeros(shape, name=name)
 
 
-def one(shape):
-    return K.ones(shape)
+def one(shape, name=None):
+    return K.ones(shape, name=name)
 
 
 from .utils.generic_utils import get_from_module
-def get(identifier):
-    return get_from_module(identifier, globals(), 'initialization')
+def get(identifier, **kwargs):
+    return get_from_module(identifier, globals(),
+                           'initialization', kwargs=kwargs)

keras/layers/__init__.py

@@ -0,0 +1,8 @@
+from __future__ import absolute_import
+from .core import *
+from .convolutional import *
+from .recurrent import *
+from .normalization import *
+from .embeddings import *
+from .noise import *
+from .advanced_activations import *

keras/layers/advanced_activations.py

@@ -6,8 +6,8 @@ import numpy as np
 
 class LeakyReLU(MaskedLayer):
     '''Special version of a Rectified Linear Unit
-    that allows a small gradient when the unit is not active
-    (`f(x) = alpha*x for x < 0`).
+    that allows a small gradient when the unit is not active:
+    `f(x) = alpha*x for x < 0`.
 
     # Input shape
         Arbitrary. Use the keyword argument `input_shape`
@@ -29,8 +29,8 @@ class LeakyReLU(MaskedLayer):
         return K.relu(X, alpha=self.alpha)
 
     def get_config(self):
-        config = {"name": self.__class__.__name__,
-                  "alpha": self.alpha}
+        config = {'name': self.__class__.__name__,
+                  'alpha': self.alpha}
         base_config = super(LeakyReLU, self).get_config()
         return dict(list(base_config.items()) + list(config.items()))
 
@@ -59,8 +59,9 @@ class PReLU(MaskedLayer):
 
     def build(self):
         input_shape = self.input_shape[1:]
-        self.alphas = self.init(input_shape)
-        self.params = [self.alphas]
+        self.alphas = self.init(input_shape,
+                                name='{}_alphas'.format(self.name))
+        self.trainable_weights = [self.alphas]
 
         if self.initial_weights is not None:
             self.set_weights(self.initial_weights)
@@ -73,8 +74,8 @@ class PReLU(MaskedLayer):
         return pos + neg
 
     def get_config(self):
-        config = {"name": self.__class__.__name__,
-                  "init": self.init.__name__}
+        config = {'name': self.__class__.__name__,
+                  'init': self.init.__name__}
         base_config = super(PReLU, self).get_config()
         return dict(list(base_config.items()) + list(config.items()))
 
@@ -106,8 +107,8 @@ class ELU(MaskedLayer):
         return pos + self.alpha * (K.exp(neg) - 1.)
 
     def get_config(self):
-        config = {"name": self.__class__.__name__,
-                  "alpha": self.alpha}
+        config = {'name': self.__class__.__name__,
+                  'alpha': self.alpha}
         base_config = super(ELU, self).get_config()
         return dict(list(base_config.items()) + list(config.items()))
 
@@ -140,9 +141,11 @@ class ParametricSoftplus(MaskedLayer):
 
     def build(self):
         input_shape = self.input_shape[1:]
-        self.alphas = K.variable(self.alpha_init * np.ones(input_shape))
-        self.betas = K.variable(self.beta_init * np.ones(input_shape))
-        self.params = [self.alphas, self.betas]
+        self.alphas = K.variable(self.alpha_init * np.ones(input_shape),
+                                 name='{}_alphas'.format(self.name))
+        self.betas = K.variable(self.beta_init * np.ones(input_shape),
+                                name='{}_betas'.format(self.name))
+        self.trainable_weights = [self.alphas, self.betas]
 
         if self.initial_weights is not None:
             self.set_weights(self.initial_weights)
@@ -153,9 +156,9 @@ class ParametricSoftplus(MaskedLayer):
         return K.softplus(self.betas * X) * self.alphas
 
     def get_config(self):
-        config = {"name": self.__class__.__name__,
-                  "alpha_init": self.alpha_init,
-                  "beta_init": self.beta_init}
+        config = {'name': self.__class__.__name__,
+                  'alpha_init': self.alpha_init,
+                  'beta_init': self.beta_init}
         base_config = super(ParametricSoftplus, self).get_config()
         return dict(list(base_config.items()) + list(config.items()))
 
@@ -186,8 +189,8 @@ class ThresholdedLinear(MaskedLayer):
         return K.switch(K.abs(X) < self.theta, 0, X)
 
     def get_config(self):
-        config = {"name": self.__class__.__name__,
-                  "theta": self.theta}
+        config = {'name': self.__class__.__name__,
+                  'theta': self.theta}
         base_config = super(ThresholdedLinear, self).get_config()
         return dict(list(base_config.items()) + list(config.items()))
 
@@ -218,7 +221,66 @@ class ThresholdedReLU(MaskedLayer):
         return K.switch(X > self.theta, X, 0)
 
     def get_config(self):
-        config = {"name": self.__class__.__name__,
-                  "theta": self.theta}
+        config = {'name': self.__class__.__name__,
+                  'theta': self.theta}
         base_config = super(ThresholdedReLU, self).get_config()
         return dict(list(base_config.items()) + list(config.items()))
+
+
+class SReLU(MaskedLayer):
+    '''SReLU
+
+    # Input shape
+        Arbitrary. Use the keyword argument `input_shape`
+        (tuple of integers, does not include the samples axis)
+        when using this layer as the first layer in a model.
+
+    # Output shape
+        Same shape as the input.
+
+    # Arguments
+        t_left_init: initialization function for the left part intercept
+        a_left_init: initialization function for the left part slope
+        t_right_init: initialization function for the right part intercept
+        a_right_init: initialization function for the right part slope
+
+    # References
+        [Deep Learning with S-shaped Rectified Linear Activation Units](http://arxiv.org/abs/1512.07030)
+    '''
+    def __init__(self, t_left_init='zero', a_left_init='glorot_uniform',
+                 t_right_init='glorot_uniform', a_right_init='one', **kwargs):
+        self.t_left_init = initializations.get(t_left_init)
+        self.a_left_init = initializations.get(a_left_init)
+        self.t_right_init = initializations.get(t_right_init)
+        self.a_right_init = initializations.get(a_right_init)
+        super(SReLU, self).__init__(**kwargs)
+
+    def build(self):
+        input_shape = self.input_shape[1:]
+        self.t_left = self.t_left_init(input_shape,
+                                       name='{}_t_left'.format(self.name))
+        self.a_left = self.a_left_init(input_shape,
+                                       name='{}_a_left'.format(self.name))
+        self.t_right = self.t_right_init(input_shape,
+                                         name='{}_t_right'.format(self.name))
+        self.a_right = self.a_right_init(input_shape,
+                                         name='{}_a_right'.format(self.name))
+        # ensure the the right part is always to the right of the left
+        self.t_right_actual = self.t_left + abs(self.t_right)
+        self.trainable_weights = [self.t_left, self.a_left,
+                                  self.t_right, self.a_right]
+
+    def get_output(self, train=False):
+        X = self.get_input(train)
+        Y_left_and_center = self.t_left + K.relu(X - self.t_left,
+                                                 self.a_left,
+                                                 self.t_right_actual - self.t_left)
+        Y_right = K.relu(X - self.t_right_actual) * self.a_right
+        return Y_left_and_center + Y_right
+
+    def get_config(self):
+        return {'name': self.__class__.__name__,
+                't_left_init': self.t_left_init.__name__,
+                'a_left_init': self.a_left_init.__name__,
+                't_right_init': self.t_right_init.__name__,
+                'a_right_init': self.a_right_init.__name__}

keras/layers/containers.py

@@ -21,33 +21,11 @@ class Sequential(Layer):
     def __init__(self, layers=[]):
         self.layers = []
         self.layer_cache = {}
+        self.shape_cache = {}
         for layer in layers:
             self.add(layer)
         self._cache_enabled = True
 
-    def __call__(self, X, mask=None, train=False):
-        # turn off layer cache temporarily
-        tmp_cache_enabled = self.cache_enabled
-        self.cache_enabled = False
-        # recursively search for a layer which is not a Sequential model
-        layer = self
-        while issubclass(layer.__class__, Sequential):
-            layer = layer.layers[0]
-        # set temporary input to first layer
-        tmp_input = layer.get_input
-        tmp_mask = None
-        layer.get_input = lambda _: X
-        if hasattr(layer, 'get_input_mask'):
-            tmp_mask = layer.get_input_mask
-            layer.get_input_mask = lambda _: mask
-        Y = self.get_output(train=train)
-        # return input from first layer to what it was
-        layer.get_input = tmp_input
-        if hasattr(layer, 'get_input_mask'):
-            layer.get_input_mask = tmp_mask
-        self.cache_enabled = tmp_cache_enabled
-        return Y
-
     @property
     def cache_enabled(self):
         return self._cache_enabled
@@ -58,11 +36,35 @@ class Sequential(Layer):
         for l in self.layers:
             l.cache_enabled = value
 
-    def set_previous(self, layer):
-        self.layers[0].previous = layer
+    @property
+    def layer_cache(self):
+        return super(Sequential, self).layer_cache
+
+    @layer_cache.setter
+    def layer_cache(self, value):
+        self._layer_cache = value
+        for layer in self.layers:
+            layer.layer_cache = self._layer_cache
+
+    @property
+    def shape_cache(self):
+        return super(Sequential, self).shape_cache
+
+    @shape_cache.setter
+    def shape_cache(self, value):
+        self._shape_cache = value
+        for layer in self.layers:
+            layer.shape_cache = self._shape_cache
+
+    def set_previous(self, layer, reset_weights=True):
+        self.layers[0].set_previous(layer, reset_weights)
+
+    def clear_previous(self, reset_weights=True):
+        self.layers[0].clear_previous(reset_weights)
 
     def add(self, layer):
         layer.layer_cache = self.layer_cache
+        layer.shape_cache = self.shape_cache
         self.layers.append(layer)
         if len(self.layers) > 1:
             self.layers[-1].set_previous(self.layers[-2])
@@ -70,12 +72,12 @@ class Sequential(Layer):
                 self.set_input()
 
     @property
-    def params(self):
-        params = []
+    def trainable_weights(self):
+        weights = []
         for l in self.layers:
             if l.trainable:
-                params += l.get_params()[0]
-        return params
+                weights += l.get_params()[0]
+        return weights
 
     @property
     def regularizers(self):
@@ -154,9 +156,9 @@ class Sequential(Layer):
         return weights
 
     def set_weights(self, weights):
-        for i in range(len(self.layers)):
-            nb_param = len(self.layers[i].params)
-            self.layers[i].set_weights(weights[:nb_param])
+        for layer in self.layers:
+            nb_param = len(layer.get_weights())
+            layer.set_weights(weights[:nb_param])
             weights = weights[nb_param:]
 
     def get_config(self):
@@ -188,6 +190,72 @@ class Graph(Layer):
         self.output_config = []  # dicts
         self.node_config = []  # dicts
         self.layer_cache = {}
+        self.shape_cache = {}
+        self._cache_enabled = True
+
+    def __call__(self, X, mask=None, train=False):
+        if type(X) != dict:
+            return super(Graph, self).__call__(X, mask, train)
+        else:
+            # turn off layer cache temporarily
+            tmp_cache_enabled = self.cache_enabled
+            self.cache_enabled = False
+            # create a temporary layer for each input
+            tmp_previous = {}
+            for name, input in self.inputs.items():
+                layer = Layer(batch_input_shape=input.input_shape)
+                layer.input = X[name]
+                if hasattr(self, 'get_input_mask'):
+                    layer.get_input_mask = lambda _: mask[name]
+                # set temporary previous
+                if hasattr(input, 'previous'):
+                    tmp_previous[name] = input.previous
+                input.set_previous(layer, False)
+            Y = self.get_output(train=train)
+            # return previous to what it was
+            for name, input in self.inputs.items():
+                if name in tmp_previous:
+                    input.set_previous(tmp_previous[name], False)
+                else:
+                    input.clear_previous(False)
+            self.cache_enabled = tmp_cache_enabled
+            return Y
+
+    @property
+    def cache_enabled(self):
+        return self._cache_enabled
+
+    @cache_enabled.setter
+    def cache_enabled(self, value):
+        self._cache_enabled = value
+        for l in self.nodes.values():
+            l.cache_enabled = value
+        for l in self.inputs.values():
+            l.cache_enabled = value
+
+    @property
+    def layer_cache(self):
+        return super(Graph, self).layer_cache
+
+    @layer_cache.setter
+    def layer_cache(self, value):
+        self._layer_cache = value
+        for layer in self.nodes.values():
+            layer.layer_cache = self._layer_cache
+        for layer in self.inputs.values():
+            layer.layer_cache = self._layer_cache
+
+    @property
+    def shape_cache(self):
+        return super(Graph, self).shape_cache
+
+    @shape_cache.setter
+    def shape_cache(self, value):
+        self._shape_cache = value
+        for layer in self.nodes.values():
+            layer.shape_cache = self._shape_cache
+        for layer in self.inputs.values():
+            layer.shape_cache = self._shape_cache
 
     @property
     def nb_input(self):
@@ -198,12 +266,12 @@ class Graph(Layer):
         return len(self.outputs)
 
     @property
-    def params(self):
-        params = []
+    def trainable_weights(self):
+        weights = []
         for l in self.nodes.values():
             if l.trainable:
-                params += l.get_params()[0]
-        return params
+                weights += l.get_params()[0]
+        return weights
 
     @property
     def regularizers(self):
@@ -248,22 +316,35 @@ class Graph(Layer):
             if hasattr(l, 'reset_states') and getattr(l, 'stateful', False):
                 l.reset_states()
 
-    def set_previous(self, layer, connection_map={}):
+    def set_previous(self, layer, connection_map={}, reset_weights=True):
         if self.nb_input != layer.nb_output:
             raise Exception('Cannot connect layers: '
                             'input count does not match output count.')
         if self.nb_input == 1:
-            self.inputs[self.input_order[0]].set_previous(layer)
+            self.inputs[self.input_order[0]].set_previous(layer, reset_weights)
         else:
             if not connection_map:
                 raise Exception('Cannot attach multi-input layer: '
                                 'no connection_map provided.')
             for k, v in connection_map.items():
                 if k in self.inputs and v in layer.outputs:
-                    self.inputs[k].set_previous(layer.outputs[v])
+                    self.inputs[k].set_previous(layer.outputs[v], reset_weights)
                 else:
                     raise Exception('Invalid connection map.')
 
+    def clear_previous(self, reset_weights=True):
+        for k in self.inputs.values():
+            k.clear_previous(reset_weights)
+
+    @property
+    def input_shape(self):
+        if self.nb_input == 1:
+            # return tuple
+            return self.inputs[self.input_order[0]].input_shape
+        else:
+            # return dictionary mapping input names to shape tuples
+            return dict([(k, v.input_shape) for k, v in self.inputs.items()])
+
     def get_input(self, train=False):
         if len(self.inputs) == len(self.outputs) == 1:
             return self.inputs[self.input_order[0]].get_input(train)
@@ -305,7 +386,7 @@ class Graph(Layer):
             raise Exception('Duplicate node identifier: ' + name)
         self.namespace.add(name)
         self.input_order.append(name)
-        layer = Layer()  # empty layer
+        layer = Layer(name=name)  # empty layer
         if input_shape:
             layer.set_input_shape((None,) + tuple(input_shape))
         elif batch_input_shape:
@@ -320,9 +401,12 @@ class Graph(Layer):
             else:
                 raise Exception('Type "int" can only be used with ndim==2 (Embedding).')
         self.inputs[name] = layer
-        self.input_config.append({'name': name,
-                                  'input_shape': input_shape,
-                                  'dtype': dtype})
+        config = {'name': name, 'dtype': dtype}
+        if batch_input_shape:
+            config['batch_input_shape'] = batch_input_shape
+        else:
+            config['input_shape'] = input_shape
+        self.input_config.append(config)
 
     def add_node(self, layer, name, input=None, inputs=[],
                  merge_mode='concat', concat_axis=-1, dot_axes=-1,
@@ -348,6 +432,7 @@ class Graph(Layer):
         '''
         if name in self.namespace:
             raise Exception('Duplicate node identifier: ' + name)
+        layer.name = name
         if input:
             if input not in self.namespace:
                 raise Exception('Unknown node/input identifier: ' + input)
@@ -370,6 +455,7 @@ class Graph(Layer):
 
         self.namespace.add(name)
         layer.layer_cache = self.layer_cache
+        layer.shape_cache = self.shape_cache
         self.nodes[name] = layer
         self.node_config.append({'name': name,
                                  'input': input,
@@ -446,6 +532,7 @@ class Graph(Layer):
                 sh = SiameseHead(i)
                 sh.previous = s
                 sh_name = outputs[i]
+                sh.name = sh_name
                 self.namespace.add(sh_name)
                 self.nodes[sh_name] = sh
                 self.node_config.append({'name': sh_name,

keras/layers/convolutional.py

@@ -79,7 +79,7 @@ class Convolution1D(Layer):
             raise Exception('Invalid border mode for Convolution1D:', border_mode)
         self.nb_filter = nb_filter
         self.filter_length = filter_length
-        self.init = initializations.get(init)
+        self.init = initializations.get(init, dim_ordering='th')
         self.activation = activations.get(activation)
         assert border_mode in {'valid', 'same'}, 'border_mode must be in {valid, same}'
         self.border_mode = border_mode
@@ -101,15 +101,14 @@ class Convolution1D(Layer):
         self.input_length = input_length
         if self.input_dim:
             kwargs['input_shape'] = (self.input_length, self.input_dim)
-        self.input = K.placeholder(ndim=3)
         super(Convolution1D, self).__init__(**kwargs)
 
     def build(self):
         input_dim = self.input_shape[2]
         self.W_shape = (self.nb_filter, input_dim, self.filter_length, 1)
-        self.W = self.init(self.W_shape)
-        self.b = K.zeros((self.nb_filter,))
-        self.params = [self.W, self.b]
+        self.W = self.init(self.W_shape, name='{}_W'.format(self.name))
+        self.b = K.zeros((self.nb_filter,), name='{}_b'.format(self.name))
+        self.trainable_weights = [self.W, self.b]
         self.regularizers = []
 
         if self.W_regularizer:
@@ -184,9 +183,10 @@ class Convolution2D(Layer):
 
     # Output shape
         4D tensor with shape:
-        `(samples, nb_filter, nb_row, nb_col)` if dim_ordering='th'
+        `(samples, nb_filter, new_rows, new_cols)` if dim_ordering='th'
         or 4D tensor with shape:
-        `(samples, nb_row, nb_col, nb_filter)` if dim_ordering='tf'.
+        `(samples, new_rows, new_cols, nb_filter)` if dim_ordering='tf'.
+        `rows` and `cols` values might have changed due to padding.
 
 
     # Arguments
@@ -233,7 +233,7 @@ class Convolution2D(Layer):
         self.nb_filter = nb_filter
         self.nb_row = nb_row
         self.nb_col = nb_col
-        self.init = initializations.get(init)
+        self.init = initializations.get(init, dim_ordering=dim_ordering)
         self.activation = activations.get(activation)
         assert border_mode in {'valid', 'same'}, 'border_mode must be in {valid, same}'
         self.border_mode = border_mode
@@ -250,7 +250,6 @@ class Convolution2D(Layer):
         self.constraints = [self.W_constraint, self.b_constraint]
 
         self.initial_weights = weights
-        self.input = K.placeholder(ndim=4)
         super(Convolution2D, self).__init__(**kwargs)
 
     def build(self):
@@ -262,9 +261,9 @@ class Convolution2D(Layer):
             self.W_shape = (self.nb_row, self.nb_col, stack_size, self.nb_filter)
         else:
             raise Exception('Invalid dim_ordering: ' + self.dim_ordering)
-        self.W = self.init(self.W_shape)
-        self.b = K.zeros((self.nb_filter,))
-        self.params = [self.W, self.b]
+        self.W = self.init(self.W_shape, name='{}_W'.format(self.name))
+        self.b = K.zeros((self.nb_filter,), name='{}_b'.format(self.name))
+        self.trainable_weights = [self.W, self.b]
         self.regularizers = []
 
         if self.W_regularizer:
@@ -342,6 +341,195 @@ class Convolution2D(Layer):
         return dict(list(base_config.items()) + list(config.items()))
 
 
+class Convolution3D(Layer):
+    '''Convolution operator for filtering windows of three-dimensional inputs.
+    When using this layer as the first layer in a model,
+    provide the keyword argument `input_shape`
+    (tuple of integers, does not include the sample axis),
+    e.g. `input_shape=(3, 10, 128, 128)` for 10 frames of 128x128 RGB pictures.
+
+    Note: this layer will only work with Theano for the time being.
+
+    # Input shape
+        5D tensor with shape:
+        `(samples, channels, conv_dim1, conv_dim2, conv_dim3)` if dim_ordering='th'
+        or 5D tensor with shape:
+        `(samples, conv_dim1, conv_dim2, conv_dim3, channels)` if dim_ordering='tf'.
+
+    # Output shape
+        5D tensor with shape:
+        `(samples, nb_filter, new_conv_dim1, new_conv_dim2, new_conv_dim3)` if dim_ordering='th'
+        or 5D tensor with shape:
+        `(samples, new_conv_dim1, new_conv_dim2, new_conv_dim3, nb_filter)` if dim_ordering='tf'.
+        `new_conv_dim1`, `new_conv_dim2` and `new_conv_dim3` values might have changed due to padding.
+
+    # Arguments
+        nb_filter: Number of convolution filters to use.
+        kernel_dim1: Length of the first dimension in the covolution kernel.
+        kernel_dim2: Length of the second dimension in the convolution kernel.
+        kernel_dim3: Length of the third dimension in the convolution kernel.
+        init: name of initialization function for the weights of the layer
+            (see [initializations](../initializations.md)), or alternatively,
+            Theano function to use for weights initialization.
+            This parameter is only relevant if you don't pass
+            a `weights` argument.
+        activation: name of activation function to use
+            (see [activations](../activations.md)),
+            or alternatively, elementwise Theano function.
+            If you don't specify anything, no activation is applied
+            (ie. "linear" activation: a(x) = x).
+        weights: list of numpy arrays to set as initial weights.
+        border_mode: 'valid' or 'same'.
+        subsample: tuple of length 3. Factor by which to subsample output.
+            Also called strides elsewhere.
+            Note: 'subsample' is implemented by slicing the output of conv3d with strides=(1,1,1).
+        W_regularizer: instance of [WeightRegularizer](../regularizers.md)
+            (eg. L1 or L2 regularization), applied to the main weights matrix.
+        b_regularizer: instance of [WeightRegularizer](../regularizers.md),
+            applied to the bias.
+        activity_regularizer: instance of [ActivityRegularizer](../regularizers.md),
+            applied to the network output.
+        W_constraint: instance of the [constraints](../constraints.md) module
+            (eg. maxnorm, nonneg), applied to the main weights matrix.
+        b_constraint: instance of the [constraints](../constraints.md) module,
+            applied to the bias.
+        dim_ordering: 'th' or 'tf'. In 'th' mode, the channels dimension
+            (the depth) is at index 1, in 'tf' mode is it at index 4.
+    '''
+    input_ndim = 5
+
+    def __init__(self, nb_filter, kernel_dim1, kernel_dim2, kernel_dim3,
+                 init='glorot_uniform', activation='linear', weights=None,
+                 border_mode='valid', subsample=(1, 1, 1), dim_ordering='th',
+                 W_regularizer=None, b_regularizer=None, activity_regularizer=None,
+                 W_constraint=None, b_constraint=None, **kwargs):
+        if K._BACKEND != 'theano':
+            raise Exception(self.__class__.__name__ +
+                            ' is currently only working with Theano backend.')
+        if border_mode not in {'valid', 'same'}:
+            raise Exception('Invalid border mode for Convolution3D:', border_mode)
+        self.nb_filter = nb_filter
+        self.kernel_dim1 = kernel_dim1
+        self.kernel_dim2 = kernel_dim2
+        self.kernel_dim3 = kernel_dim3
+        self.init = initializations.get(init, dim_ordering=dim_ordering)
+        self.activation = activations.get(activation)
+        assert border_mode in {'valid', 'same'}, 'border_mode must be in {valid, same}'
+        self.border_mode = border_mode
+        self.subsample = tuple(subsample)
+        assert dim_ordering in {'tf', 'th'}, 'dim_ordering must be in {tf, th}'
+        self.dim_ordering = dim_ordering
+
+        self.W_regularizer = regularizers.get(W_regularizer)
+        self.b_regularizer = regularizers.get(b_regularizer)
+        self.activity_regularizer = regularizers.get(activity_regularizer)
+
+        self.W_constraint = constraints.get(W_constraint)
+        self.b_constraint = constraints.get(b_constraint)
+        self.constraints = [self.W_constraint, self.b_constraint]
+
+        self.initial_weights = weights
+        super(Convolution3D, self).__init__(**kwargs)
+
+    def build(self):
+
+        if self.dim_ordering == 'th':
+            stack_size = self.input_shape[1]
+            self.W_shape = (self.nb_filter, stack_size,
+                            self.kernel_dim1, self.kernel_dim2, self.kernel_dim3)
+        elif self.dim_ordering == 'tf':
+            stack_size = self.input_shape[4]
+            self.W_shape = (self.kernel_dim1, self.kernel_dim2, self.kernel_dim3,
+                            stack_size, self.nb_filter)
+        else:
+            raise Exception('Invalid dim_ordering: ' + self.dim_ordering)
+
+        self.W = self.init(self.W_shape, name='{}_W'.format(self.name))
+        self.b = K.zeros((self.nb_filter,), name='{}_b'.format(self.name))
+        self.trainable_weights = [self.W, self.b]
+        self.regularizers = []
+
+        if self.W_regularizer:
+            self.W_regularizer.set_param(self.W)
+            self.regularizers.append(self.W_regularizer)
+
+        if self.b_regularizer:
+            self.b_regularizer.set_param(self.b)
+            self.regularizers.append(self.b_regularizer)
+
+        if self.activity_regularizer:
+            self.activity_regularizer.set_layer(self)
+            self.regularizers.append(self.activity_regularizer)
+
+        if self.initial_weights is not None:
+            self.set_weights(self.initial_weights)
+            del self.initial_weights
+
+    @property
+    def output_shape(self):
+        input_shape = self.input_shape
+        if self.dim_ordering == 'th':
+            conv_dim1 = input_shape[2]
+            conv_dim2 = input_shape[3]
+            conv_dim3 = input_shape[4]
+        elif self.dim_ordering == 'tf':
+            conv_dim1 = input_shape[1]
+            conv_dim2 = input_shape[2]
+            conv_dim3 = input_shape[3]
+        else:
+            raise Exception('Invalid dim_ordering: ' + self.dim_ordering)
+
+        conv_dim1 = conv_output_length(conv_dim1, self.kernel_dim1,
+                                       self.border_mode, self.subsample[0])
+        conv_dim2 = conv_output_length(conv_dim2, self.kernel_dim2,
+                                       self.border_mode, self.subsample[1])
+        conv_dim3 = conv_output_length(conv_dim3, self.kernel_dim3,
+                                       self.border_mode, self.subsample[2])
+
+        if self.dim_ordering == 'th':
+            return (input_shape[0], self.nb_filter, conv_dim1, conv_dim2, conv_dim3)
+        elif self.dim_ordering == 'tf':
+            return (input_shape[0], conv_dim1, conv_dim2, conv_dim3, self.nb_filter)
+        else:
+            raise Exception('Invalid dim_ordering: ' + self.dim_ordering)
+
+    def get_output(self, train=False):
+        X = self.get_input(train)
+        conv_out = K.conv3d(X, self.W, strides=self.subsample,
+                            border_mode=self.border_mode,
+                            dim_ordering=self.dim_ordering,
+                            volume_shape=self.input_shape,
+                            filter_shape=self.W_shape)
+
+        if self.dim_ordering == 'th':
+            output = conv_out + K.reshape(self.b, (1, self.nb_filter, 1, 1, 1))
+        elif self.dim_ordering == 'tf':
+            output = conv_out + K.reshape(self.b, (1, 1, 1, 1, self.nb_filter))
+        else:
+            raise Exception('Invalid dim_ordering: ' + self.dim_ordering)
+        output = self.activation(output)
+        return output
+
+    def get_config(self):
+        config = {"name": self.__class__.__name__,
+                  "nb_filter": self.nb_filter,
+                  "kernel_dim1": self.kernel_dim1,
+                  "kernel_dim2": self.kernel_dim2,
+                  "kernel_dim3": self.kernel_dim3,
+                  "dim_ordering": self.dim_ordering,
+                  "init": self.init.__name__,
+                  "activation": self.activation.__name__,
+                  "border_mode": self.border_mode,
+                  "subsample": self.subsample,
+                  "W_regularizer": self.W_regularizer.get_config() if self.W_regularizer else None,
+                  "b_regularizer": self.b_regularizer.get_config() if self.b_regularizer else None,
+                  "activity_regularizer": self.activity_regularizer.get_config() if self.activity_regularizer else None,
+                  "W_constraint": self.W_constraint.get_config() if self.W_constraint else None,
+                  "b_constraint": self.b_constraint.get_config() if self.b_constraint else None}
+        base_config = super(Convolution3D, self).get_config()
+        return dict(list(base_config.items()) + list(config.items()))
+
+
 class _Pooling1D(Layer):
     '''Abstract class for different pooling 1D layers.
     '''
@@ -355,7 +543,6 @@ class _Pooling1D(Layer):
         self.pool_length = pool_length
         self.stride = stride
         self.st = (self.stride, 1)
-        self.input = K.placeholder(ndim=3)
         self.pool_size = (pool_length, 1)
         assert border_mode in {'valid', 'same'}, 'border_mode must be in {valid, same}'
         self.border_mode = border_mode
@@ -406,6 +593,7 @@ class MaxPooling1D(_Pooling1D):
         border_mode: 'valid' or 'same'.
             Note: 'same' will only work with TensorFlow for the time being.
     '''
+
     def __init__(self, pool_length=2, stride=None,
                  border_mode='valid', **kwargs):
         super(MaxPooling1D, self).__init__(pool_length, stride,
@@ -421,7 +609,7 @@ class MaxPooling1D(_Pooling1D):
 class AveragePooling1D(_Pooling1D):
     '''Average pooling for temporal data.
 
-        # Input shape
+    # Input shape
         3D tensor with shape: `(samples, steps, features)`.
 
     # Output shape
@@ -433,6 +621,7 @@ class AveragePooling1D(_Pooling1D):
         border_mode: 'valid' or 'same'.
             Note: 'same' will only work with TensorFlow for the time being.
     '''
+
     def __init__(self, pool_length=2, stride=None,
                  border_mode='valid', **kwargs):
         super(AveragePooling1D, self).__init__(pool_length, stride,
@@ -453,7 +642,6 @@ class _Pooling2D(Layer):
     def __init__(self, pool_size=(2, 2), strides=None, border_mode='valid',
                  dim_ordering='th', **kwargs):
         super(_Pooling2D, self).__init__(**kwargs)
-        self.input = K.placeholder(ndim=4)
         self.pool_size = tuple(pool_size)
         if strides is None:
             strides = self.pool_size
@@ -534,6 +722,7 @@ class MaxPooling2D(_Pooling2D):
         dim_ordering: 'th' or 'tf'. In 'th' mode, the channels dimension
             (the depth) is at index 1, in 'tf' mode is it at index 3.
     '''
+
     def __init__(self, pool_size=(2, 2), strides=None, border_mode='valid',
                  dim_ordering='th', **kwargs):
         super(MaxPooling2D, self).__init__(pool_size, strides, border_mode,
@@ -571,6 +760,7 @@ class AveragePooling2D(_Pooling2D):
         dim_ordering: 'th' or 'tf'. In 'th' mode, the channels dimension
             (the depth) is at index 1, in 'tf' mode is it at index 3.
     '''
+
     def __init__(self, pool_size=(2, 2), strides=None, border_mode='valid',
                  dim_ordering='th', **kwargs):
         super(AveragePooling2D, self).__init__(pool_size, strides, border_mode,
@@ -583,6 +773,157 @@ class AveragePooling2D(_Pooling2D):
         return output
 
 
+class _Pooling3D(Layer):
+    '''Abstract class for different pooling 3D layers.
+    '''
+    input_ndim = 5
+
+    def __init__(self, pool_size=(2, 2, 2), strides=None, border_mode='valid',
+                 dim_ordering='th', **kwargs):
+        super(_Pooling3D, self).__init__(**kwargs)
+        self.pool_size = tuple(pool_size)
+        if strides is None:
+            strides = self.pool_size
+        self.strides = tuple(strides)
+        assert border_mode in {'valid', 'same'}, 'border_mode must be in {valid, same}'
+        self.border_mode = border_mode
+        assert dim_ordering in {'tf', 'th'}, 'dim_ordering must be in {tf, th}'
+        self.dim_ordering = dim_ordering
+
+    @property
+    def output_shape(self):
+        input_shape = self.input_shape
+        if self.dim_ordering == 'th':
+            len_dim1 = input_shape[2]
+            len_dim2 = input_shape[3]
+            len_dim3 = input_shape[4]
+        elif self.dim_ordering == 'tf':
+            len_dim1 = input_shape[1]
+            len_dim2 = input_shape[2]
+            len_dim3 = input_shape[3]
+        else:
+            raise Exception('Invalid dim_ordering: ' + self.dim_ordering)
+
+        len_dim1 = conv_output_length(len_dim1, self.pool_size[0],
+                                      self.border_mode, self.strides[0])
+        len_dim2 = conv_output_length(len_dim2, self.pool_size[1],
+                                      self.border_mode, self.strides[1])
+        len_dim3 = conv_output_length(len_dim3, self.pool_size[2],
+                                      self.border_mode, self.strides[2])
+
+        if self.dim_ordering == 'th':
+            return (input_shape[0], input_shape[1], len_dim1, len_dim2, len_dim3)
+        elif self.dim_ordering == 'tf':
+            return (input_shape[0], len_dim1, len_dim2, len_dim3, input_shape[4])
+        else:
+            raise Exception('Invalid dim_ordering: ' + self.dim_ordering)
+
+    def _pooling_function(self, inputs, pool_size, strides,
+                          border_mode, dim_ordering):
+        raise NotImplementedError
+
+    def get_output(self, train=False):
+        X = self.get_input(train)
+        output = self._pooling_function(inputs=X, pool_size=self.pool_size,
+                                        strides=self.strides,
+                                        border_mode=self.border_mode,
+                                        dim_ordering=self.dim_ordering)
+        return output
+
+    def get_config(self):
+        config = {'name': self.__class__.__name__,
+                  'pool_size': self.pool_size,
+                  'border_mode': self.border_mode,
+                  'strides': self.strides,
+                  'dim_ordering': self.dim_ordering}
+        base_config = super(_Pooling3D, self).get_config()
+        return dict(list(base_config.items()) + list(config.items()))
+
+
+class MaxPooling3D(_Pooling3D):
+    '''Max pooling operation for 3D data (spatial or spatio-temporal).
+
+    Note: this layer will only work with Theano for the time being.
+
+    # Input shape
+        5D tensor with shape:
+        `(samples, channels, len_pool_dim1, len_pool_dim2, len_pool_dim3)` if dim_ordering='th'
+        or 5D tensor with shape:
+        `(samples, len_pool_dim1, len_pool_dim2, len_pool_dim3, channels)` if dim_ordering='tf'.
+
+    # Output shape
+        5D tensor with shape:
+        `(nb_samples, channels, pooled_dim1, pooled_dim2, pooled_dim3)` if dim_ordering='th'
+        or 5D tensor with shape:
+        `(samples, pooled_dim1, pooled_dim2, pooled_dim3, channels)` if dim_ordering='tf'.
+
+    # Arguments
+        pool_size: tuple of 3 integers,
+            factors by which to downscale (dim1, dim2, dim3).
+            (2, 2, 2) will halve the size of the 3D input in each dimension.
+        strides: tuple of 3 integers, or None. Strides values.
+        border_mode: 'valid' or 'same'.
+        dim_ordering: 'th' or 'tf'. In 'th' mode, the channels dimension
+            (the depth) is at index 1, in 'tf' mode is it at index 4.
+    '''
+
+    def __init__(self, pool_size=(2, 2, 2), strides=None, border_mode='valid',
+                 dim_ordering='th', **kwargs):
+        if K._BACKEND != 'theano':
+            raise Exception(self.__class__.__name__ +
+                            ' is currently only working with Theano backend.')
+        super(MaxPooling3D, self).__init__(pool_size, strides, border_mode,
+                                           dim_ordering, **kwargs)
+
+    def _pooling_function(self, inputs, pool_size, strides,
+                          border_mode, dim_ordering):
+        output = K.pool3d(inputs, pool_size, strides,
+                          border_mode, dim_ordering, pool_mode='max')
+        return output
+
+
+class AveragePooling3D(_Pooling3D):
+    '''Average pooling operation for 3D data (spatial or spatio-temporal).
+
+    Note: this layer will only work with Theano for the time being.
+
+    # Input shape
+        5D tensor with shape:
+        `(samples, channels, len_pool_dim1, len_pool_dim2, len_pool_dim3)` if dim_ordering='th'
+        or 5D tensor with shape:
+        `(samples, len_pool_dim1, len_pool_dim2, len_pool_dim3, channels)` if dim_ordering='tf'.
+
+    # Output shape
+        5D tensor with shape:
+        `(nb_samples, channels, pooled_dim1, pooled_dim2, pooled_dim3)` if dim_ordering='th'
+        or 5D tensor with shape:
+        `(samples, pooled_dim1, pooled_dim2, pooled_dim3, channels)` if dim_ordering='tf'.
+
+    # Arguments
+        pool_size: tuple of 3 integers,
+            factors by which to downscale (dim1, dim2, dim3).
+            (2, 2, 2) will halve the size of the 3D input in each dimension.
+        strides: tuple of 3 integers, or None. Strides values.
+        border_mode: 'valid' or 'same'.
+        dim_ordering: 'th' or 'tf'. In 'th' mode, the channels dimension
+            (the depth) is at index 1, in 'tf' mode is it at index 4.
+    '''
+
+    def __init__(self, pool_size=(2, 2, 2), strides=None, border_mode='valid',
+                 dim_ordering='th', **kwargs):
+        if K._BACKEND != 'theano':
+            raise Exception(self.__class__.__name__ +
+                            ' is currently only working with Theano backend.')
+        super(AveragePooling3D, self).__init__(pool_size, strides, border_mode,
+                                               dim_ordering, **kwargs)
+
+    def _pooling_function(self, inputs, pool_size, strides,
+                          border_mode, dim_ordering):
+        output = K.pool3d(inputs, pool_size, strides,
+                          border_mode, dim_ordering, pool_mode='avg')
+        return output
+
+
 class UpSampling1D(Layer):
     '''Repeat each temporal step `length` times along the time axis.
 
@@ -600,7 +941,6 @@ class UpSampling1D(Layer):
     def __init__(self, length=2, **kwargs):
         super(UpSampling1D, self).__init__(**kwargs)
         self.length = length
-        self.input = K.placeholder(ndim=3)
 
     @property
     def output_shape(self):
@@ -645,7 +985,6 @@ class UpSampling2D(Layer):
 
     def __init__(self, size=(2, 2), dim_ordering='th', **kwargs):
         super(UpSampling2D, self).__init__(**kwargs)
-        self.input = K.placeholder(ndim=4)
         self.size = tuple(size)
         assert dim_ordering in {'tf', 'th'}, 'dim_ordering must be in {tf, th}'
         self.dim_ordering = dim_ordering
@@ -668,15 +1007,8 @@ class UpSampling2D(Layer):
 
     def get_output(self, train=False):
         X = self.get_input(train)
-        if self.dim_ordering == 'th':
-            output = K.repeat_elements(X, self.size[0], axis=2)
-            output = K.repeat_elements(output, self.size[1], axis=3)
-        elif self.dim_ordering == 'tf':
-            output = K.repeat_elements(X, self.size[0], axis=1)
-            output = K.repeat_elements(output, self.size[1], axis=2)
-        else:
-            raise Exception('Invalid dim_ordering: ' + self.dim_ordering)
-        return output
+        return K.resize_images(X, self.size[0], self.size[1],
+                               self.dim_ordering)
 
     def get_config(self):
         config = {'name': self.__class__.__name__,
@@ -685,6 +1017,71 @@ class UpSampling2D(Layer):
         return dict(list(base_config.items()) + list(config.items()))
 
 
+class UpSampling3D(Layer):
+    '''Repeat the first, second and third dimension of the data
+    by size[0], size[1] and size[2] respectively.
+
+    Note: this layer will only work with Theano for the time being.
+
+    # Input shape
+        5D tensor with shape:
+        `(samples, channels, dim1, dim2, dim3)` if dim_ordering='th'
+        or 5D tensor with shape:
+        `(samples, dim1, dim2, dim3, channels)` if dim_ordering='tf'.
+
+    # Output shape
+        5D tensor with shape:
+        `(samples, channels, upsampled_dim1, upsampled_dim2, upsampled_dim3)` if dim_ordering='th'
+        or 5D tensor with shape:
+        `(samples, upsampled_dim1, upsampled_dim2, upsampled_dim3, channels)` if dim_ordering='tf'.
+
+    # Arguments
+        size: tuple of 3 integers. The upsampling factors for dim1, dim2 and dim3.
+        dim_ordering: 'th' or 'tf'.
+            In 'th' mode, the channels dimension (the depth)
+            is at index 1, in 'tf' mode is it at index 4.
+    '''
+    input_ndim = 5
+
+    def __init__(self, size=(2, 2, 2), dim_ordering='th', **kwargs):
+        if K._BACKEND != 'theano':
+            raise Exception(self.__class__.__name__ +
+                            ' is currently only working with Theano backend.')
+        super(UpSampling3D, self).__init__(**kwargs)
+        self.size = tuple(size)
+        assert dim_ordering in {'tf', 'th'}, 'dim_ordering must be in {tf, th}'
+        self.dim_ordering = dim_ordering
+
+    @property
+    def output_shape(self):
+        input_shape = self.input_shape
+        if self.dim_ordering == 'th':
+            return (input_shape[0],
+                    input_shape[1],
+                    self.size[0] * input_shape[2],
+                    self.size[1] * input_shape[3],
+                    self.size[2] * input_shape[4])
+        elif self.dim_ordering == 'tf':
+            return (input_shape[0],
+                    self.size[0] * input_shape[1],
+                    self.size[1] * input_shape[2],
+                    self.size[2] * input_shape[3],
+                    input_shape[4])
+        else:
+            raise Exception('Invalid dim_ordering: ' + self.dim_ordering)
+
+    def get_output(self, train=False):
+        X = self.get_input(train)
+        return K.resize_volumes(X, self.size[0], self.size[1], self.size[2],
+                                self.dim_ordering)
+
+    def get_config(self):
+        config = {'name': self.__class__.__name__,
+                  'size': self.size}
+        base_config = super(UpSampling3D, self).get_config()
+        return dict(list(base_config.items()) + list(config.items()))
+
+
 class ZeroPadding1D(Layer):
     '''Zero-padding layer for 1D input (e.g. temporal sequence).
 
@@ -704,13 +1101,13 @@ class ZeroPadding1D(Layer):
     def __init__(self, padding=1, **kwargs):
         super(ZeroPadding1D, self).__init__(**kwargs)
         self.padding = padding
-        self.input = K.placeholder(ndim=3)
 
     @property
     def output_shape(self):
         input_shape = self.input_shape
+        length = input_shape[1] + self.padding * 2 if input_shape[1] is not None else None
         return (input_shape[0],
-                input_shape[1] + self.padding * 2,
+                length,
                 input_shape[2])
 
     def get_output(self, train=False):
@@ -745,7 +1142,6 @@ class ZeroPadding2D(Layer):
     def __init__(self, padding=(1, 1), dim_ordering='th', **kwargs):
         super(ZeroPadding2D, self).__init__(**kwargs)
         self.padding = tuple(padding)
-        self.input = K.placeholder(ndim=4)
         assert dim_ordering in {'tf', 'th'}, 'dim_ordering must be in {tf, th}'
         self.dim_ordering = dim_ordering
 
@@ -753,14 +1149,18 @@ class ZeroPadding2D(Layer):
     def output_shape(self):
         input_shape = self.input_shape
         if self.dim_ordering == 'th':
+            width = input_shape[2] + 2 * self.padding[0] if input_shape[2] is not None else None
+            height = input_shape[3] + 2 * self.padding[1] if input_shape[3] is not None else None
             return (input_shape[0],
                     input_shape[1],
-                    input_shape[2] + 2 * self.padding[0],
-                    input_shape[3] + 2 * self.padding[1])
+                    width,
+                    height)
         elif self.dim_ordering == 'tf':
+            width = input_shape[1] + 2 * self.padding[0] if input_shape[1] is not None else None
+            height = input_shape[2] + 2 * self.padding[1] if input_shape[2] is not None else None
             return (input_shape[0],
-                    input_shape[1] + 2 * self.padding[0],
-                    input_shape[2] + 2 * self.padding[1],
+                    width,
+                    height,
                     input_shape[3])
         else:
             raise Exception('Invalid dim_ordering: ' + self.dim_ordering)
@@ -775,3 +1175,68 @@ class ZeroPadding2D(Layer):
                   'padding': self.padding}
         base_config = super(ZeroPadding2D, self).get_config()
         return dict(list(base_config.items()) + list(config.items()))
+
+
+class ZeroPadding3D(Layer):
+    '''Zero-padding layer for 3D data (spatial or spatio-temporal).
+
+    Note: this layer will only work with Theano for the time being.
+
+    # Input shape
+        5D tensor with shape:
+        (samples, depth, first_axis_to_pad, second_axis_to_pad, third_axis_to_pad)
+
+    # Output shape
+        5D tensor with shape:
+        (samples, depth, first_padded_axis, second_padded_axis, third_axis_to_pad)
+
+    # Arguments
+        padding: tuple of int (length 3)
+            How many zeros to add at the beginning and end of
+            the 3 padding dimensions (axis 3, 4 and 5).
+    '''
+    input_ndim = 5
+
+    def __init__(self, padding=(1, 1, 1), dim_ordering='th', **kwargs):
+        if K._BACKEND != 'theano':
+            raise Exception(self.__class__.__name__ +
+                            ' is currently only working with Theano backend.')
+        super(ZeroPadding3D, self).__init__(**kwargs)
+        self.padding = tuple(padding)
+        assert dim_ordering in {'tf', 'th'}, 'dim_ordering must be in {tf, th}'
+        self.dim_ordering = dim_ordering
+
+    @property
+    def output_shape(self):
+        input_shape = self.input_shape
+        if self.dim_ordering == 'th':
+            dim1 = input_shape[2] + 2 * self.padding[0] if input_shape[2] is not None else None
+            dim2 = input_shape[3] + 2 * self.padding[1] if input_shape[3] is not None else None
+            dim3 = input_shape[4] + 2 * self.padding[2] if input_shape[4] is not None else None
+            return (input_shape[0],
+                    input_shape[1],
+                    dim1,
+                    dim2,
+                    dim3)
+        elif self.dim_ordering == 'tf':
+            dim1 = input_shape[1] + 2 * self.padding[0] if input_shape[1] is not None else None
+            dim2 = input_shape[2] + 2 * self.padding[1] if input_shape[2] is not None else None
+            dim3 = input_shape[3] + 2 * self.padding[2] if input_shape[3] is not None else None
+            return (input_shape[0],
+                    dim1,
+                    dim2,
+                    dim3,
+                    input_shape[4])
+        else:
+            raise Exception('Invalid dim_ordering: ' + self.dim_ordering)
+
+    def get_output(self, train=False):
+        X = self.get_input(train)
+        return K.spatial_3d_padding(X, padding=self.padding,
+                                    dim_ordering=self.dim_ordering)
+
+    def get_config(self):
+        config = {'name': self.__class__.__name__,
+                  'padding': self.padding}
+        base_config = super(ZeroPadding3D, self).get_config()
+        return dict(list(base_config.items()) + list(config.items()))

keras/layers/embeddings.py

@@ -1,10 +1,8 @@
 from __future__ import absolute_import
+
 from .. import backend as K
-
-from .. import activations, initializations, regularizers, constraints
-from ..layers.core import Layer, MaskedLayer
-
-from ..constraints import unitnorm
+from .. import initializations, regularizers, constraints
+from ..layers.core import Layer
 
 
 class Embedding(Layer):
@@ -42,6 +40,10 @@ class Embedding(Layer):
           This argument is required if you are going to connect
           `Flatten` then `Dense` layers upstream
           (without it, the shape of the dense outputs cannot be computed).
+      dropout: float between 0 and 1. Fraction of the embeddings to drop.
+
+    # References
+        - [A Theoretically Grounded Application of Dropout in Recurrent Neural Networks](http://arxiv.org/abs/1512.05287)
     '''
     input_ndim = 2
 
@@ -50,12 +52,13 @@ class Embedding(Layer):
                  W_regularizer=None, activity_regularizer=None,
                  W_constraint=None,
                  mask_zero=False,
-                 weights=None, **kwargs):
+                 weights=None, dropout=0., **kwargs):
         self.input_dim = input_dim
         self.output_dim = output_dim
         self.init = initializations.get(init)
         self.input_length = input_length
         self.mask_zero = mask_zero
+        self.dropout = dropout
 
         self.W_constraint = constraints.get(W_constraint)
         self.constraints = [self.W_constraint]
@@ -70,8 +73,9 @@ class Embedding(Layer):
     def build(self):
         self.input = K.placeholder(shape=(self.input_shape[0], self.input_length),
                                    dtype='int32')
-        self.W = self.init((self.input_dim, self.output_dim))
-        self.params = [self.W]
+        self.W = self.init((self.input_dim, self.output_dim),
+                           name='{}_W'.format(self.name))
+        self.trainable_weights = [self.W]
         self.regularizers = []
         if self.W_regularizer:
             self.W_regularizer.set_param(self.W)
@@ -89,9 +93,7 @@ class Embedding(Layer):
         if not self.mask_zero:
             return None
         else:
-            if K._BACKEND == "tensorflow":
-                raise Exception("Masking is Theano-only for the time being.")
-            return K.ones_like(X) * (1 - K.equal(X, 0))
+            return K.not_equal(X, 0)
 
     @property
     def output_shape(self):
@@ -99,7 +101,13 @@ class Embedding(Layer):
 
     def get_output(self, train=False):
         X = self.get_input(train)
-        out = K.gather(self.W, X)
+        retain_p = 1. - self.dropout
+        if train and self.dropout > 0:
+            B = K.random_binomial((self.input_dim,), p=retain_p)
+        else:
+            B = K.ones((self.input_dim)) * retain_p
+        # we zero-out rows of W at random
+        out = K.gather(self.W * K.expand_dims(B), X)
         return out
 
     def get_config(self):
@@ -111,6 +119,7 @@ class Embedding(Layer):
                   "mask_zero": self.mask_zero,
                   "activity_regularizer": self.activity_regularizer.get_config() if self.activity_regularizer else None,
                   "W_regularizer": self.W_regularizer.get_config() if self.W_regularizer else None,
-                  "W_constraint": self.W_constraint.get_config() if self.W_constraint else None}
+                  "W_constraint": self.W_constraint.get_config() if self.W_constraint else None,
+                  "dropout": self.dropout}
         base_config = super(Embedding, self).get_config()
         return dict(list(base_config.items()) + list(config.items()))

keras/layers/noise.py

@@ -65,7 +65,7 @@ class GaussianDropout(MaskedLayer):
             # self.p refers to drop probability rather than
             # retain probability (as in paper), for consistency
             X *= K.random_normal(shape=K.shape(X), mean=1.0,
-                                 std=self.p / (1.0 - self.p))
+                                 std=K.sqrt(self.p / (1.0 - self.p)))
         return X
 
     def get_config(self):

keras/layers/normalization.py

@@ -20,80 +20,100 @@ class BatchNormalization(Layer):
         epsilon: small float > 0. Fuzz parameter.
         mode: integer, 0 or 1.
             - 0: feature-wise normalization.
-                If the input has multiple feature dimensions,
-                each will be normalized separately
-                (e.g. for an image input with shape
-                `(channels, rows, cols)`,
-                each combination of a channel, row and column
-                will be normalized separately).
+                Each feature map in the input will
+                be normalized separately. The axis on which
+                to normalize is specified by the `axis` argument.
+                Note that if the input is a 4D image tensor
+                using Theano conventions (samples, channels, rows, cols)
+                then you should set `axis` to `1` to normalize along
+                the channels axis.
             - 1: sample-wise normalization. This mode assumes a 2D input.
+        axis: integer, axis along which to normalize in mode 0. For instance,
+            if your input tensor has shape (samples, channels, rows, cols),
+            set axis to 1 to normalize per feature map (channels axis).
         momentum: momentum in the computation of the
             exponential average of the mean and standard deviation
             of the data, for feature-wise normalization.
         weights: Initialization weights.
             List of 2 numpy arrays, with shapes:
             `[(input_shape,), (input_shape,)]`
-
+        beta_init: name of initialization function for shift parameter
+            (see [initializations](../initializations.md)), or alternatively,
+            Theano/TensorFlow function to use for weights initialization.
+            This parameter is only relevant if you don't pass a `weights` argument.
+        gamma_init: name of initialization function for scale parameter (see
+            [initializations](../initializations.md)), or alternatively,
+            Theano/TensorFlow function to use for weights initialization.
+            This parameter is only relevant if you don't pass a `weights` argument.
     # References
         - [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](http://arxiv.org/pdf/1502.03167v3.pdf)
     '''
-    def __init__(self, epsilon=1e-6, mode=0, momentum=0.9,
-                 weights=None, **kwargs):
-        self.init = initializations.get("uniform")
+    def __init__(self, epsilon=1e-6, mode=0, axis=-1, momentum=0.9,
+                 weights=None, beta_init='zero', gamma_init='one', **kwargs):
+        self.beta_init = initializations.get(beta_init)
+        self.gamma_init = initializations.get(gamma_init)
         self.epsilon = epsilon
         self.mode = mode
+        self.axis = axis
         self.momentum = momentum
         self.initial_weights = weights
         super(BatchNormalization, self).__init__(**kwargs)
 
     def build(self):
         input_shape = self.input_shape  # starts with samples axis
-        input_shape = input_shape[1:]
+        shape = (input_shape[self.axis],)
 
-        self.gamma = self.init(input_shape)
-        self.beta = K.zeros(input_shape)
+        self.gamma = self.gamma_init(shape, name='{}_gamma'.format(self.name))
+        self.beta = self.beta_init(shape, name='{}_beta'.format(self.name))
+        self.trainable_weights = [self.gamma, self.beta]
 
-        self.params = [self.gamma, self.beta]
-        self.running_mean = K.zeros(input_shape)
-        self.running_std = K.ones(input_shape)
+        self.running_mean = K.zeros(shape,
+                                    name='{}_running_mean'.format(self.name))
+        self.running_std = K.ones(shape,
+                                  name='{}_running_std'.format(self.name))
+        self.non_trainable_weights = [self.running_mean, self.running_std]
 
         if self.initial_weights is not None:
             self.set_weights(self.initial_weights)
             del self.initial_weights
 
-    def get_weights(self):
-        super_weights = super(BatchNormalization, self).get_weights()
-        return super_weights + [K.get_value(self.running_mean),
-                                K.get_value(self.running_std)]
-
-    def set_weights(self, weights):
-        K.set_value(self.running_mean, weights[-2])
-        K.set_value(self.running_std, weights[-1])
-        super(BatchNormalization, self).set_weights(weights[:-2])
-
     def get_output(self, train):
         X = self.get_input(train)
         if self.mode == 0:
-            m = K.mean(X, axis=0)
-            std = K.mean(K.square(X - m) + self.epsilon, axis=0)
-            std = K.sqrt(std)
-            mean_update = self.momentum * self.running_mean + (1-self.momentum) * m
-            std_update = self.momentum * self.running_std + (1-self.momentum) * std
-            self.updates = [(self.running_mean, mean_update),
-                            (self.running_std, std_update)]
-            X_normed = ((X - self.running_mean) /
-                        (self.running_std + self.epsilon))
+            input_shape = self.input_shape
+            reduction_axes = list(range(len(input_shape)))
+            del reduction_axes[self.axis]
+            broadcast_shape = [1] * len(input_shape)
+            broadcast_shape[self.axis] = input_shape[self.axis]
+            if train:
+                m = K.mean(X, axis=reduction_axes)
+                brodcast_m = K.reshape(m, broadcast_shape)
+                std = K.mean(K.square(X - brodcast_m) + self.epsilon, axis=reduction_axes)
+                std = K.sqrt(std)
+                brodcast_std = K.reshape(std, broadcast_shape)
+                mean_update = self.momentum * self.running_mean + (1-self.momentum) * m
+                std_update = self.momentum * self.running_std + (1-self.momentum) * std
+                self.updates = [(self.running_mean, mean_update),
+                                (self.running_std, std_update)]
+                X_normed = (X - brodcast_m) / (brodcast_std + self.epsilon)
+            else:
+                brodcast_m = K.reshape(self.running_mean, broadcast_shape)
+                brodcast_std = K.reshape(self.running_std, broadcast_shape)
+                X_normed = ((X - brodcast_m) /
+                            (brodcast_std + self.epsilon))
+            out = K.reshape(self.gamma, broadcast_shape) * X_normed + K.reshape(self.beta, broadcast_shape)
         elif self.mode == 1:
             m = K.mean(X, axis=-1, keepdims=True)
             std = K.std(X, axis=-1, keepdims=True)
             X_normed = (X - m) / (std + self.epsilon)
-        out = self.gamma * X_normed + self.beta
+            out = self.gamma * X_normed + self.beta
         return out
 
     def get_config(self):
         config = {"name": self.__class__.__name__,
                   "epsilon": self.epsilon,
                   "mode": self.mode,
+                  "axis": self.axis,
                   "momentum": self.momentum}
         base_config = super(BatchNormalization, self).get_config()
         return dict(list(base_config.items()) + list(config.items()))

keras/layers/recurrent.py

@@ -3,10 +3,42 @@ from __future__ import absolute_import
 import numpy as np
 
 from .. import backend as K
-from .. import activations, initializations
+from .. import activations, initializations, regularizers
 from ..layers.core import MaskedLayer
 
 
+def time_distributed_dense(x, w, b=None, dropout=None,
+                           input_dim=None, output_dim=None, timesteps=None):
+    '''Apply y.w + b for every temporal slice y of x.
+    '''
+    if not input_dim:
+        # won't work with TensorFlow
+        input_dim = K.shape(x)[2]
+    if not timesteps:
+        # won't work with TensorFlow
+        timesteps = K.shape(x)[1]
+    if not output_dim:
+        # won't work with TensorFlow
+        output_dim = K.shape(w)[1]
+
+    if dropout:
+        # apply the same dropout pattern at every timestep
+        ones = K.ones_like(K.reshape(x[:, 0, :], (-1, input_dim)))
+        dropout_matrix = K.dropout(ones, dropout)
+        expanded_dropout_matrix = K.repeat(dropout_matrix, timesteps)
+        x *= expanded_dropout_matrix
+
+    # collapse time dimension and batch dimension together
+    x = K.reshape(x, (-1, input_dim))
+
+    x = K.dot(x, w)
+    if b:
+        x = x + b
+    # reshape to 3D tensor
+    x = K.reshape(x, (-1, timesteps, output_dim))
+    return x
+
+
 class Recurrent(MaskedLayer):
     '''Abstract base class for recurrent layers.
     Do not use in a model -- it's not a functional layer!
@@ -52,7 +84,6 @@ class Recurrent(MaskedLayer):
         of timesteps. To introduce masks to your data,
         use an [Embedding](embeddings.md) layer with the `mask_zero` parameter
         set to `True`.
-        **Note:** for the time being, masking is only supported with Theano.
 
     # TensorFlow warning
         For the time being, when using the TensorFlow backend,
@@ -79,6 +110,10 @@ class Recurrent(MaskedLayer):
 
         To reset the states of your model, call `.reset_states()` on either
         a specific layer, or on your entire model.
+
+    # Note on using dropout with TensorFlow
+        When using the TensorFlow backend, specify a fixed batch size for your model
+        following the notes on statefulness RNNs.
     '''
     input_ndim = 3
 
@@ -113,43 +148,50 @@ class Recurrent(MaskedLayer):
     def step(self, x, states):
         raise NotImplementedError
 
-    def get_initial_states(self, X):
+    def get_constants(self, x, train=False):
+        return []
+
+    def get_initial_states(self, x):
         # build an all-zero tensor of shape (samples, output_dim)
-        initial_state = K.zeros_like(X)  # (samples, timesteps, input_dim)
+        initial_state = K.zeros_like(x)  # (samples, timesteps, input_dim)
         initial_state = K.sum(initial_state, axis=1)  # (samples, input_dim)
         reducer = K.zeros((self.input_dim, self.output_dim))
         initial_state = K.dot(initial_state, reducer)  # (samples, output_dim)
         initial_states = [initial_state for _ in range(len(self.states))]
         return initial_states
 
+    def preprocess_input(self, x, train=False):
+        return x
+
     def get_output(self, train=False):
         # input shape: (nb_samples, time (padded with zeros), input_dim)
         X = self.get_input(train)
+        mask = self.get_input_mask(train)
+
         assert K.ndim(X) == 3
         if K._BACKEND == 'tensorflow':
             if not self.input_shape[1]:
                 raise Exception('When using TensorFlow, you should define ' +
                                 'explicitly the number of timesteps of ' +
-                                'your sequences. Make sure the first layer ' +
-                                'has a "batch_input_shape" argument ' +
-                                'including the samples axis.')
-
-        mask = self.get_output_mask(train)
-        if mask:
-            # apply mask
-            X *= K.cast(K.expand_dims(mask), X.dtype)
-            masking = True
-        else:
-            masking = False
-
+                                'your sequences.\n' +
+                                'If your first layer is an Embedding, ' +
+                                'make sure to pass it an "input_length" ' +
+                                'argument. Otherwise, make sure ' +
+                                'the first layer has ' +
+                                'an "input_shape" or "batch_input_shape" ' +
+                                'argument, including the time axis.')
         if self.stateful:
             initial_states = self.states
         else:
             initial_states = self.get_initial_states(X)
+        constants = self.get_constants(X, train)
+        preprocessed_input = self.preprocess_input(X, train)
 
-        last_output, outputs, states = K.rnn(self.step, X, initial_states,
+        last_output, outputs, states = K.rnn(self.step, preprocessed_input,
+                                             initial_states,
                                              go_backwards=self.go_backwards,
-                                             masking=masking)
+                                             mask=mask,
+                                             constants=constants)
         if self.stateful:
             self.updates = []
             for i in range(len(states)):
@@ -163,16 +205,20 @@ class Recurrent(MaskedLayer):
     def get_config(self):
         config = {"name": self.__class__.__name__,
                   "return_sequences": self.return_sequences,
-                  "input_dim": self.input_dim,
-                  "input_length": self.input_length,
                   "go_backwards": self.go_backwards,
                   "stateful": self.stateful}
+        if self.stateful:
+            config['batch_input_shape'] = self.input_shape
+        else:
+            config['input_dim'] = self.input_dim
+            config['input_length'] = self.input_length
+
         base_config = super(Recurrent, self).get_config()
         return dict(list(base_config.items()) + list(config.items()))
 
 
 class SimpleRNN(Recurrent):
-    '''Fully-connected RNN where the output is to fed back to input.
+    '''Fully-connected RNN where the output is to be fed back to input.
 
     # Arguments
         output_dim: dimension of the internal projections and the final output.
@@ -183,14 +229,31 @@ class SimpleRNN(Recurrent):
         activation: activation function.
             Can be the name of an existing function (str),
             or a Theano function (see: [activations](../activations.md)).
+        W_regularizer: instance of [WeightRegularizer](../regularizers.md)
+            (eg. L1 or L2 regularization), applied to the input weights matrices.
+        U_regularizer: instance of [WeightRegularizer](../regularizers.md)
+            (eg. L1 or L2 regularization), applied to the recurrent weights matrices.
+        b_regularizer: instance of [WeightRegularizer](../regularizers.md),
+            applied to the bias.
+        dropout_W: float between 0 and 1. Fraction of the input units to drop for input gates.
+        dropout_U: float between 0 and 1. Fraction of the input units to drop for recurrent connections.
+
+    # References
+        - [A Theoretically Grounded Application of Dropout in Recurrent Neural Networks](http://arxiv.org/abs/1512.05287)
     '''
     def __init__(self, output_dim,
                  init='glorot_uniform', inner_init='orthogonal',
-                 activation='sigmoid', **kwargs):
+                 activation='tanh',
+                 W_regularizer=None, U_regularizer=None, b_regularizer=None,
+                 dropout_W=0., dropout_U=0., **kwargs):
         self.output_dim = output_dim
         self.init = initializations.get(init)
         self.inner_init = initializations.get(inner_init)
         self.activation = activations.get(activation)
+        self.W_regularizer = regularizers.get(W_regularizer)
+        self.U_regularizer = regularizers.get(U_regularizer)
+        self.b_regularizer = regularizers.get(b_regularizer)
+        self.dropout_W, self.dropout_U = dropout_W, dropout_U
         super(SimpleRNN, self).__init__(**kwargs)
 
     def build(self):
@@ -203,10 +266,24 @@ class SimpleRNN(Recurrent):
         input_dim = input_shape[2]
         self.input_dim = input_dim
 
-        self.W = self.init((input_dim, self.output_dim))
-        self.U = self.inner_init((self.output_dim, self.output_dim))
-        self.b = K.zeros((self.output_dim,))
-        self.params = [self.W, self.U, self.b]
+        self.W = self.init((input_dim, self.output_dim),
+                           name='{}_W'.format(self.name))
+        self.U = self.inner_init((self.output_dim, self.output_dim),
+                                 name='{}_U'.format(self.name))
+        self.b = K.zeros((self.output_dim,), name='{}_b'.format(self.name))
+
+        self.regularizers = []
+        if self.W_regularizer:
+            self.W_regularizer.set_param(self.W)
+            self.regularizers.append(self.W_regularizer)
+        if self.U_regularizer:
+            self.W_regularizer.set_param(self.U)
+            self.regularizers.append(self.U_regularizer)
+        if self.b_regularizer:
+            self.W_regularizer.set_param(self.b)
+            self.regularizers.append(self.b_regularizer)
+
+        self.trainable_weights = [self.W, self.U, self.b]
 
         if self.initial_weights is not None:
             self.set_weights(self.initial_weights)
@@ -217,27 +294,51 @@ class SimpleRNN(Recurrent):
         input_shape = self.input_shape
         if not input_shape[0]:
             raise Exception('If a RNN is stateful, a complete ' +
-                            'input_shape must be provided ' +
-                            '(including batch size).')
+                            'input_shape must be provided (including batch size).')
         if hasattr(self, 'states'):
             K.set_value(self.states[0],
                         np.zeros((input_shape[0], self.output_dim)))
         else:
             self.states = [K.zeros((input_shape[0], self.output_dim))]
 
-    def step(self, x, states):
-        # states only contains the previous output.
-        assert len(states) == 1
+    def preprocess_input(self, x, train=False):
+        if train and (0 < self.dropout_W < 1):
+            dropout = self.dropout_W
+        else:
+            dropout = 0
+        input_shape = self.input_shape
+        input_dim = input_shape[2]
+        timesteps = input_shape[1]
+        return time_distributed_dense(x, self.W, self.b, dropout,
+                                      input_dim, self.output_dim, timesteps)
+
+    def step(self, h, states):
         prev_output = states[0]
-        h = K.dot(x, self.W) + self.b
-        output = self.activation(h + K.dot(prev_output, self.U))
+        if len(states) == 2:
+            B_U = states[1]
+        else:
+            B_U = 1.
+        output = self.activation(h + K.dot(prev_output * B_U, self.U))
         return output, [output]
 
+    def get_constants(self, x, train=False):
+        if train and (0 < self.dropout_U < 1):
+            ones = K.ones_like(K.reshape(x[:, 0, 0], (-1, 1)))
+            ones = K.concatenate([ones] * self.output_dim, 1)
+            B_U = K.dropout(ones, self.dropout_U)
+            return [B_U]
+        return []
+
     def get_config(self):
         config = {"output_dim": self.output_dim,
                   "init": self.init.__name__,
                   "inner_init": self.inner_init.__name__,
-                  "activation": self.activation.__name__}
+                  "activation": self.activation.__name__,
+                  "W_regularizer": self.W_regularizer.get_config() if self.W_regularizer else None,
+                  "U_regularizer": self.U_regularizer.get_config() if self.U_regularizer else None,
+                  "b_regularizer": self.b_regularizer.get_config() if self.b_regularizer else None,
+                  "dropout_W": self.dropout_W,
+                  "dropout_U": self.dropout_U}
         base_config = super(SimpleRNN, self).get_config()
         return dict(list(base_config.items()) + list(config.items()))
 
@@ -255,44 +356,79 @@ class GRU(Recurrent):
             Can be the name of an existing function (str),
             or a Theano function (see: [activations](../activations.md)).
         inner_activation: activation function for the inner cells.
+        W_regularizer: instance of [WeightRegularizer](../regularizers.md)
+            (eg. L1 or L2 regularization), applied to the input weights matrices.
+        U_regularizer: instance of [WeightRegularizer](../regularizers.md)
+            (eg. L1 or L2 regularization), applied to the recurrent weights matrices.
+        b_regularizer: instance of [WeightRegularizer](../regularizers.md),
+            applied to the bias.
+        dropout_W: float between 0 and 1. Fraction of the input units to drop for input gates.
+        dropout_U: float between 0 and 1. Fraction of the input units to drop for recurrent connections.
 
     # References
         - [On the Properties of Neural Machine Translation: Encoder–Decoder Approaches](http://www.aclweb.org/anthology/W14-4012)
         - [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](http://arxiv.org/pdf/1412.3555v1.pdf)
+        - [A Theoretically Grounded Application of Dropout in Recurrent Neural Networks](http://arxiv.org/abs/1512.05287)
     '''
     def __init__(self, output_dim,
                  init='glorot_uniform', inner_init='orthogonal',
-                 activation='sigmoid', inner_activation='hard_sigmoid',
-                 **kwargs):
+                 activation='tanh', inner_activation='hard_sigmoid',
+                 W_regularizer=None, U_regularizer=None, b_regularizer=None,
+                 dropout_W=0., dropout_U=0., **kwargs):
         self.output_dim = output_dim
         self.init = initializations.get(init)
         self.inner_init = initializations.get(inner_init)
         self.activation = activations.get(activation)
         self.inner_activation = activations.get(inner_activation)
+        self.W_regularizer = regularizers.get(W_regularizer)
+        self.U_regularizer = regularizers.get(U_regularizer)
+        self.b_regularizer = regularizers.get(b_regularizer)
+        self.dropout_W, self.dropout_U = dropout_W, dropout_U
         super(GRU, self).__init__(**kwargs)
 
     def build(self):
         input_shape = self.input_shape
         input_dim = input_shape[2]
         self.input_dim = input_dim
-        self.input = K.placeholder(input_shape)
 
-        self.W_z = self.init((input_dim, self.output_dim))
-        self.U_z = self.inner_init((self.output_dim, self.output_dim))
-        self.b_z = K.zeros((self.output_dim,))
+        self.W_z = self.init((input_dim, self.output_dim),
+                             name='{}_W_z'.format(self.name))
+        self.U_z = self.inner_init((self.output_dim, self.output_dim),
+                                   name='{}_U_z'.format(self.name))
+        self.b_z = K.zeros((self.output_dim,), name='{}_b_z'.format(self.name))
 
-        self.W_r = self.init((input_dim, self.output_dim))
-        self.U_r = self.inner_init((self.output_dim, self.output_dim))
-        self.b_r = K.zeros((self.output_dim,))
+        self.W_r = self.init((input_dim, self.output_dim),
+                             name='{}_W_r'.format(self.name))
+        self.U_r = self.inner_init((self.output_dim, self.output_dim),
+                                   name='{}_U_r'.format(self.name))
+        self.b_r = K.zeros((self.output_dim,), name='{}_b_r'.format(self.name))
 
-        self.W_h = self.init((input_dim, self.output_dim))
-        self.U_h = self.inner_init((self.output_dim, self.output_dim))
-        self.b_h = K.zeros((self.output_dim,))
+        self.W_h = self.init((input_dim, self.output_dim),
+                             name='{}_W_h'.format(self.name))
+        self.U_h = self.inner_init((self.output_dim, self.output_dim),
+                                   name='{}_U_h'.format(self.name))
+        self.b_h = K.zeros((self.output_dim,), name='{}_b_h'.format(self.name))
 
-        self.params = [self.W_z, self.U_z, self.b_z,
-                       self.W_r, self.U_r, self.b_r,
-                       self.W_h, self.U_h, self.b_h]
+        self.regularizers = []
+        if self.W_regularizer:
+            self.W_regularizer.set_param(K.concatenate([self.W_z,
+                                                        self.W_r,
+                                                        self.W_h]))
+            self.regularizers.append(self.W_regularizer)
+        if self.U_regularizer:
+            self.U_regularizer.set_param(K.concatenate([self.U_z,
+                                                        self.U_r,
+                                                        self.U_h]))
+            self.regularizers.append(self.U_regularizer)
+        if self.b_regularizer:
+            self.b_regularizer.set_param(K.concatenate([self.b_z,
+                                                        self.b_r,
+                                                        self.b_h]))
+            self.regularizers.append(self.b_regularizer)
 
+        self.trainable_weights = [self.W_z, self.U_z, self.b_z,
+                                  self.W_r, self.U_r, self.b_r,
+                                  self.W_h, self.U_h, self.b_h]
         if self.stateful:
             self.reset_states()
         else:
@@ -308,34 +444,67 @@ class GRU(Recurrent):
         input_shape = self.input_shape
         if not input_shape[0]:
             raise Exception('If a RNN is stateful, a complete ' +
-                            'input_shape must be provided ' +
-                            '(including batch size).')
+                            'input_shape must be provided (including batch size).')
         if hasattr(self, 'states'):
             K.set_value(self.states[0],
                         np.zeros((input_shape[0], self.output_dim)))
         else:
             self.states = [K.zeros((input_shape[0], self.output_dim))]
 
+    def preprocess_input(self, x, train=False):
+        if train and (0 < self.dropout_W < 1):
+            dropout = self.dropout_W
+        else:
+            dropout = 0
+        input_shape = self.input_shape
+        input_dim = input_shape[2]
+        timesteps = input_shape[1]
+
+        x_z = time_distributed_dense(x, self.W_z, self.b_z, dropout,
+                                     input_dim, self.output_dim, timesteps)
+        x_r = time_distributed_dense(x, self.W_r, self.b_r, dropout,
+                                     input_dim, self.output_dim, timesteps)
+        x_h = time_distributed_dense(x, self.W_h, self.b_h, dropout,
+                                     input_dim, self.output_dim, timesteps)
+        return K.concatenate([x_z, x_r, x_h], axis=2)
+
     def step(self, x, states):
-        assert len(states) == 1
-        x_z = K.dot(x, self.W_z) + self.b_z
-        x_r = K.dot(x, self.W_r) + self.b_r
-        x_h = K.dot(x, self.W_h) + self.b_h
+        h_tm1 = states[0]  # previous memory
+        if len(states) == 2:
+            B_U = states[1]  # dropout matrices for recurrent units
+        else:
+            B_U = [1., 1., 1.]
 
-        h_tm1 = states[0]
-        z = self.inner_activation(x_z + K.dot(h_tm1, self.U_z))
-        r = self.inner_activation(x_r + K.dot(h_tm1, self.U_r))
+        x_z = x[:, :self.output_dim]
+        x_r = x[:, self.output_dim: 2 * self.output_dim]
+        x_h = x[:, 2 * self.output_dim:]
 
-        hh = self.activation(x_h + K.dot(r * h_tm1, self.U_h))
+        z = self.inner_activation(x_z + K.dot(h_tm1 * B_U[0], self.U_z))
+        r = self.inner_activation(x_r + K.dot(h_tm1 * B_U[1], self.U_r))
+
+        hh = self.activation(x_h + K.dot(r * h_tm1 * B_U[2], self.U_h))
         h = z * h_tm1 + (1 - z) * hh
         return h, [h]
 
+    def get_constants(self, x, train=False):
+        if train and (0 < self.dropout_U < 1):
+            ones = K.ones_like(K.reshape(x[:, 0, 0], (-1, 1)))
+            ones = K.concatenate([ones] * self.output_dim, 1)
+            B_U = [K.dropout(ones, self.dropout_U) for _ in range(3)]
+            return [B_U]
+        return []
+
     def get_config(self):
         config = {"output_dim": self.output_dim,
                   "init": self.init.__name__,
                   "inner_init": self.inner_init.__name__,
                   "activation": self.activation.__name__,
-                  "inner_activation": self.inner_activation.__name__}
+                  "inner_activation": self.inner_activation.__name__,
+                  "W_regularizer": self.W_regularizer.get_config() if self.W_regularizer else None,
+                  "U_regularizer": self.U_regularizer.get_config() if self.U_regularizer else None,
+                  "b_regularizer": self.b_regularizer.get_config() if self.b_regularizer else None,
+                  "dropout_W": self.dropout_W,
+                  "dropout_U": self.dropout_U}
         base_config = super(GRU, self).get_config()
         return dict(list(base_config.items()) + list(config.items()))
 
@@ -359,56 +528,99 @@ class LSTM(Recurrent):
             Can be the name of an existing function (str),
             or a Theano function (see: [activations](../activations.md)).
         inner_activation: activation function for the inner cells.
+        W_regularizer: instance of [WeightRegularizer](../regularizers.md)
+            (eg. L1 or L2 regularization), applied to the input weights matrices.
+        U_regularizer: instance of [WeightRegularizer](../regularizers.md)
+            (eg. L1 or L2 regularization), applied to the recurrent weights matrices.
+        b_regularizer: instance of [WeightRegularizer](../regularizers.md),
+            applied to the bias.
+        dropout_W: float between 0 and 1. Fraction of the input units to drop for input gates.
+        dropout_U: float between 0 and 1. Fraction of the input units to drop for recurrent connections.
 
     # References
         - [Long short-term memory](http://deeplearning.cs.cmu.edu/pdfs/Hochreiter97_lstm.pdf) (original 1997 paper)
         - [Learning to forget: Continual prediction with LSTM](http://www.mitpressjournals.org/doi/pdf/10.1162/089976600300015015)
         - [Supervised sequence labelling with recurrent neural networks](http://www.cs.toronto.edu/~graves/preprint.pdf)
+        - [A Theoretically Grounded Application of Dropout in Recurrent Neural Networks](http://arxiv.org/abs/1512.05287)
     '''
     def __init__(self, output_dim,
                  init='glorot_uniform', inner_init='orthogonal',
                  forget_bias_init='one', activation='tanh',
-                 inner_activation='hard_sigmoid', **kwargs):
+                 inner_activation='hard_sigmoid',
+                 W_regularizer=None, U_regularizer=None, b_regularizer=None,
+                 dropout_W=0., dropout_U=0., **kwargs):
         self.output_dim = output_dim
         self.init = initializations.get(init)
         self.inner_init = initializations.get(inner_init)
         self.forget_bias_init = initializations.get(forget_bias_init)
         self.activation = activations.get(activation)
         self.inner_activation = activations.get(inner_activation)
+        self.W_regularizer = regularizers.get(W_regularizer)
+        self.U_regularizer = regularizers.get(U_regularizer)
+        self.b_regularizer = regularizers.get(b_regularizer)
+        self.dropout_W, self.dropout_U = dropout_W, dropout_U
         super(LSTM, self).__init__(**kwargs)
 
     def build(self):
         input_shape = self.input_shape
         input_dim = input_shape[2]
         self.input_dim = input_dim
-        self.input = K.placeholder(input_shape)
 
         if self.stateful:
             self.reset_states()
         else:
-            # initial states: 2 all-zero tensor of shape (output_dim)
+            # initial states: 2 all-zero tensors of shape (output_dim)
             self.states = [None, None]
 
-        self.W_i = self.init((input_dim, self.output_dim))
-        self.U_i = self.inner_init((self.output_dim, self.output_dim))
-        self.b_i = K.zeros((self.output_dim,))
+        self.W_i = self.init((input_dim, self.output_dim),
+                             name='{}_W_i'.format(self.name))
+        self.U_i = self.inner_init((self.output_dim, self.output_dim),
+                                   name='{}_U_i'.format(self.name))
+        self.b_i = K.zeros((self.output_dim,), name='{}_b_i'.format(self.name))
 
-        self.W_f = self.init((input_dim, self.output_dim))
-        self.U_f = self.inner_init((self.output_dim, self.output_dim))
-        self.b_f = self.forget_bias_init((self.output_dim,))
+        self.W_f = self.init((input_dim, self.output_dim),
+                             name='{}_W_f'.format(self.name))
+        self.U_f = self.inner_init((self.output_dim, self.output_dim),
+                                   name='{}_U_f'.format(self.name))
+        self.b_f = self.forget_bias_init((self.output_dim,),
+                                         name='{}_b_f'.format(self.name))
 
-        self.W_c = self.init((input_dim, self.output_dim))
-        self.U_c = self.inner_init((self.output_dim, self.output_dim))
-        self.b_c = K.zeros((self.output_dim,))
+        self.W_c = self.init((input_dim, self.output_dim),
+                             name='{}_W_c'.format(self.name))
+        self.U_c = self.inner_init((self.output_dim, self.output_dim),
+                                   name='{}_U_c'.format(self.name))
+        self.b_c = K.zeros((self.output_dim,), name='{}_b_c'.format(self.name))
 
-        self.W_o = self.init((input_dim, self.output_dim))
-        self.U_o = self.inner_init((self.output_dim, self.output_dim))
-        self.b_o = K.zeros((self.output_dim,))
+        self.W_o = self.init((input_dim, self.output_dim),
+                             name='{}_W_o'.format(self.name))
+        self.U_o = self.inner_init((self.output_dim, self.output_dim),
+                                   name='{}_U_o'.format(self.name))
+        self.b_o = K.zeros((self.output_dim,), name='{}_b_o'.format(self.name))
 
-        self.params = [self.W_i, self.U_i, self.b_i,
-                       self.W_c, self.U_c, self.b_c,
-                       self.W_f, self.U_f, self.b_f,
-                       self.W_o, self.U_o, self.b_o]
+        self.regularizers = []
+        if self.W_regularizer:
+            self.W_regularizer.set_param(K.concatenate([self.W_i,
+                                                        self.W_f,
+                                                        self.W_c,
+                                                        self.W_o]))
+            self.regularizers.append(self.W_regularizer)
+        if self.U_regularizer:
+            self.U_regularizer.set_param(K.concatenate([self.U_i,
+                                                        self.U_f,
+                                                        self.U_c,
+                                                        self.U_o]))
+            self.regularizers.append(self.U_regularizer)
+        if self.b_regularizer:
+            self.b_regularizer.set_param(K.concatenate([self.b_i,
+                                                        self.b_f,
+                                                        self.b_c,
+                                                        self.b_o]))
+            self.regularizers.append(self.b_regularizer)
+
+        self.trainable_weights = [self.W_i, self.U_i, self.b_i,
+                                  self.W_c, self.U_c, self.b_c,
+                                  self.W_f, self.U_f, self.b_f,
+                                  self.W_o, self.U_o, self.b_o]
 
         if self.initial_weights is not None:
             self.set_weights(self.initial_weights)
@@ -419,8 +631,7 @@ class LSTM(Recurrent):
         input_shape = self.input_shape
         if not input_shape[0]:
             raise Exception('If a RNN is stateful, a complete ' +
-                            'input_shape must be provided ' +
-                            '(including batch size).')
+                            'input_shape must be provided (including batch size).')
         if hasattr(self, 'states'):
             K.set_value(self.states[0],
                         np.zeros((input_shape[0], self.output_dim)))
@@ -430,29 +641,65 @@ class LSTM(Recurrent):
             self.states = [K.zeros((input_shape[0], self.output_dim)),
                            K.zeros((input_shape[0], self.output_dim))]
 
+    def preprocess_input(self, x, train=False):
+        if train and (0 < self.dropout_W < 1):
+            dropout = self.dropout_W
+        else:
+            dropout = 0
+        input_shape = self.input_shape
+        input_dim = input_shape[2]
+        timesteps = input_shape[1]
+
+        x_i = time_distributed_dense(x, self.W_i, self.b_i, dropout,
+                                     input_dim, self.output_dim, timesteps)
+        x_f = time_distributed_dense(x, self.W_f, self.b_f, dropout,
+                                     input_dim, self.output_dim, timesteps)
+        x_c = time_distributed_dense(x, self.W_c, self.b_c, dropout,
+                                     input_dim, self.output_dim, timesteps)
+        x_o = time_distributed_dense(x, self.W_o, self.b_o, dropout,
+                                     input_dim, self.output_dim, timesteps)
+        return K.concatenate([x_i, x_f, x_c, x_o], axis=2)
+
     def step(self, x, states):
-        assert len(states) == 2
         h_tm1 = states[0]
         c_tm1 = states[1]
+        if len(states) == 3:
+            B_U = states[2]
+        else:
+            B_U = [1. for _ in range(4)]
 
-        x_i = K.dot(x, self.W_i) + self.b_i
-        x_f = K.dot(x, self.W_f) + self.b_f
-        x_c = K.dot(x, self.W_c) + self.b_c
-        x_o = K.dot(x, self.W_o) + self.b_o
+        x_i = x[:, :self.output_dim]
+        x_f = x[:, self.output_dim: 2 * self.output_dim]
+        x_c = x[:, 2 * self.output_dim: 3 * self.output_dim]
+        x_o = x[:, 3 * self.output_dim:]
+
+        i = self.inner_activation(x_i + K.dot(h_tm1 * B_U[0], self.U_i))
+        f = self.inner_activation(x_f + K.dot(h_tm1 * B_U[1], self.U_f))
+        c = f * c_tm1 + i * self.activation(x_c + K.dot(h_tm1 * B_U[2], self.U_c))
+        o = self.inner_activation(x_o + K.dot(h_tm1 * B_U[3], self.U_o))
 
-        i = self.inner_activation(x_i + K.dot(h_tm1, self.U_i))
-        f = self.inner_activation(x_f + K.dot(h_tm1, self.U_f))
-        c = f * c_tm1 + i * self.activation(x_c + K.dot(h_tm1, self.U_c))
-        o = self.inner_activation(x_o + K.dot(h_tm1, self.U_o))
         h = o * self.activation(c)
         return h, [h, c]
 
+    def get_constants(self, x, train=False):
+        if train and (0 < self.dropout_U < 1):
+            ones = K.ones_like(K.reshape(x[:, 0, 0], (-1, 1)))
+            ones = K.concatenate([ones] * self.output_dim, 1)
+            B_U = [K.dropout(ones, self.dropout_U) for _ in range(4)]
+            return [B_U]
+        return []
+
     def get_config(self):
         config = {"output_dim": self.output_dim,
                   "init": self.init.__name__,
                   "inner_init": self.inner_init.__name__,
                   "forget_bias_init": self.forget_bias_init.__name__,
                   "activation": self.activation.__name__,
-                  "inner_activation": self.inner_activation.__name__}
+                  "inner_activation": self.inner_activation.__name__,
+                  "W_regularizer": self.W_regularizer.get_config() if self.W_regularizer else None,
+                  "U_regularizer": self.U_regularizer.get_config() if self.U_regularizer else None,
+                  "b_regularizer": self.b_regularizer.get_config() if self.b_regularizer else None,
+                  "dropout_W": self.dropout_W,
+                  "dropout_U": self.dropout_U}
         base_config = super(LSTM, self).get_config()
         return dict(list(base_config.items()) + list(config.items()))

keras/layers/wrappers.py

@@ -0,0 +1,114 @@
+from .core import MaskedLayer
+from .. import backend as K
+
+
+class TimeDistributed(MaskedLayer):
+    """This wrapper allows to apply a layer to every
+    temporal slice of an input.
+
+    The input should be at least 3D,
+    and the dimension of index one will be considered to be
+    the temporal dimension.
+
+    Consider a batch of 32 samples, where each sample is a sequence of 10
+    vectors of 16 dimensions. The batch input shape of the layer is then `(32, 10, 16)`
+    (and the `input_shape`, not including the samples dimension, is `(10, 16)`).
+
+    You can then use `TimeDistributed` to apply a `Dense` layer to each of the 10 timesteps, independently:
+    ```python
+        model = Sequential()
+        model.add(TimeDistributed(Dense(8), input_shape=(10, 16)))
+    ```
+
+    The output will then have shape `(32, 10, 8)`.
+
+    Note this is strictly equivalent to using `layers.core.TimeDistributedDense`.
+    However what is different about `TimeDistributed`
+    is that it can be used with arbitrary layers, not just `Dense`,
+    for instance with a `Convolution2D` layer:
+
+    ```python
+        model = Sequential()
+        model.add(TimeDistributed(Convolution2D(64, 3, 3), input_shape=(10, 3, 299, 299)))
+    ```
+
+    # Arguments
+        layer: a layer instance.
+    """
+
+    def __init__(self, layer, **kwargs):
+        self.layer = layer
+        super(TimeDistributed, self).__init__(**kwargs)
+
+    def build(self):
+        input_shape = self.input_shape
+        assert len(input_shape) >= 3
+        child_input_shape = (input_shape[0],) + input_shape[2:]
+        self.layer.set_input_shape(child_input_shape)
+        self.layer.build()
+
+        trainable_weights, regularizers, constraints, updates = self.layer.get_params()
+        self.trainable_weights = trainable_weights
+        self.non_trainable_weights = self.layer.non_trainable_weights
+        self.regularizers = regularizers
+        self.constraints = constraints
+        self.updates = updates
+
+    @property
+    def output_shape(self):
+        child_output_shape = self.layer.output_shape
+        timesteps = self.input_shape[1]
+        return (child_output_shape[0], timesteps) + child_output_shape[1:]
+
+    def get_output(self, train=False):
+        X = self.get_input(train)
+        mask = self.get_input_mask(train)
+
+        if K._BACKEND == 'tensorflow':
+            if not self.input_shape[1]:
+                raise Exception('When using TensorFlow, you should define ' +
+                                'explicitly the number of timesteps of ' +
+                                'your sequences.\n' +
+                                'If your first layer is an Embedding, ' +
+                                'make sure to pass it an "input_length" ' +
+                                'argument. Otherwise, make sure ' +
+                                'the first layer has ' +
+                                'an "input_shape" or "batch_input_shape" ' +
+                                'argument, including the time axis.')
+
+        if self.input_shape[0]:
+            # batch size matters, use rnn-based implementation
+            def step(x, states):
+                output = self.layer(x, train=train)
+                return output, []
+
+            last_output, outputs, states = K.rnn(step, X,
+                                                 initial_states=[],
+                                                 mask=mask)
+            y = outputs
+        else:
+            # no batch size specified, therefore the layer will be able
+            # to process batches of any size
+            # we can go with reshape-based implementation for performance
+            input_shape = self.input_shape
+            x = K.reshape(X, (-1, ) + input_shape[2:])  # (nb_samples * timesteps, ...)
+            y = self.layer(x, train=False)  # (nb_samples * timesteps, ...)
+            input_length = input_shape[1]
+            if not input_length:
+                input_length = K.shape(X)[1]
+            # (nb_samples, timesteps, ...)
+            y = K.reshape(y, (-1, input_length) + self.layer.output_shape[1:])
+        return y
+
+    def get_weights(self):
+        weights = self.layer.get_weights()
+        return weights
+
+    def set_weights(self, weights):
+        self.layer.set_weights(weights)
+
+    def get_config(self):
+        config = {'name': self.__class__.__name__,
+                  'layer': self.layer.get_config()}
+        base_config = super(TimeDistributed, self).get_config()
+        return dict(list(base_config.items()) + list(config.items()))

keras/objectives.py

@@ -7,10 +7,6 @@ def mean_squared_error(y_true, y_pred):
     return K.mean(K.square(y_pred - y_true), axis=-1)
 
 
-def root_mean_squared_error(y_true, y_pred):
-    return K.sqrt(K.mean(K.square(y_pred - y_true), axis=-1))
-
-
 def mean_absolute_error(y_true, y_pred):
     return K.mean(K.abs(y_pred - y_true), axis=-1)
 
@@ -37,22 +33,29 @@ def hinge(y_true, y_pred):
 def categorical_crossentropy(y_true, y_pred):
     '''Expects a binary class matrix instead of a vector of scalar classes.
     '''
-    return K.mean(K.categorical_crossentropy(y_pred, y_true), axis=-1)
+    return K.categorical_crossentropy(y_pred, y_true)
 
 
 def binary_crossentropy(y_true, y_pred):
     return K.mean(K.binary_crossentropy(y_pred, y_true), axis=-1)
 
 
-def poisson_loss(y_true, y_pred):
+def poisson(y_true, y_pred):
     return K.mean(y_pred - y_true * K.log(y_pred + K.epsilon()), axis=-1)
 
+
+def cosine_proximity(y_true, y_pred):
+    y_true = K.l2_normalize(y_true, axis=-1)
+    y_pred = K.l2_normalize(y_pred, axis=-1)
+    return -K.mean(y_true * y_pred, axis=-1)
+
+
 # aliases
 mse = MSE = mean_squared_error
-rmse = RMSE = root_mean_squared_error
 mae = MAE = mean_absolute_error
 mape = MAPE = mean_absolute_percentage_error
 msle = MSLE = mean_squared_logarithmic_error
+cosine = cosine_proximity
 
 from .utils.generic_utils import get_from_module
 def get(identifier):

keras/optimizers.py

@@ -275,6 +275,7 @@ class Adam(Optimizer):
                 "beta_2": float(K.get_value(self.beta_2)),
                 "epsilon": self.epsilon}
 
+
 class Adamax(Optimizer):
     '''Adamax optimizer from Adam paper's Section 7. It is a variant
      of Adam based on the infinity norm.

keras/preprocessing/image.py

@@ -1,3 +1,7 @@
+'''Fairly basic set of tools for realtime data augmentation on image data.
+Can easily be extended to include new transformations,
+new preprocessing methods, etc...
+'''
 from __future__ import absolute_import
 
 import numpy as np
@@ -7,48 +11,41 @@ from scipy import linalg
 
 from os import listdir
 from os.path import isfile, join
-import random, math
+import math
 from six.moves import range
+import threading
 
-'''
-    Fairly basic set of tools for realtime data augmentation on image data.
-    Can easily be extended to include new transforms, new preprocessing methods, etc...
-'''
 
-def random_rotation(x, rg, fill_mode="nearest", cval=0.):
-    angle = random.uniform(-rg, rg)
-    x = ndimage.interpolation.rotate(x, angle, axes=(1,2), reshape=False, mode=fill_mode, cval=cval)
+def random_rotation(x, rg, fill_mode='nearest', cval=0.):
+    angle = np.random.uniform(-rg, rg)
+    x = ndimage.interpolation.rotate(x, angle,
+                                     axes=(1, 2),
+                                     reshape=False,
+                                     mode=fill_mode,
+                                     cval=cval)
     return x
 
-def random_shift(x, wrg, hrg, fill_mode="nearest", cval=0.):
-    crop_left_pixels = 0
-    crop_right_pixels = 0
-    crop_top_pixels = 0
-    crop_bottom_pixels = 0
 
-    original_w = x.shape[1]
-    original_h = x.shape[2]
+def random_shift(x, wrg, hrg, fill_mode='nearest', cval=0.):
+    shift_x = shift_y = 0
 
     if wrg:
-        crop = random.uniform(0., wrg)
-        split = random.uniform(0, 1)
-        crop_left_pixels = int(split*crop*x.shape[1])
-        crop_right_pixels = int((1-split)*crop*x.shape[1])
-
+        shift_x = np.random.uniform(-wrg, wrg) * x.shape[2]
     if hrg:
-        crop = random.uniform(0., hrg)
-        split = random.uniform(0, 1)
-        crop_top_pixels = int(split*crop*x.shape[2])
-        crop_bottom_pixels = int((1-split)*crop*x.shape[2])
-
-    x = ndimage.interpolation.shift(x, (0, crop_left_pixels, crop_top_pixels), mode=fill_mode, cval=cval)
+        shift_y = np.random.uniform(-hrg, hrg) * x.shape[1]
+    x = ndimage.interpolation.shift(x, (0, shift_y, shift_x),
+                                    order=0,
+                                    mode=fill_mode,
+                                    cval=cval)
     return x
 
+
 def horizontal_flip(x):
     for i in range(x.shape[0]):
         x[i] = np.fliplr(x[i])
     return x
 
+
 def vertical_flip(x):
     for i in range(x.shape[0]):
         x[i] = np.flipud(x[i])
@@ -59,41 +56,51 @@ def random_barrel_transform(x, intensity):
     # TODO
     pass
 
-def random_shear(x, intensity):
-    # TODO
-    pass
+
+def random_shear(x, intensity, fill_mode='nearest', cval=0.):
+    shear = np.random.uniform(-intensity, intensity)
+    shear_matrix = np.array([[1.0, -math.sin(shear), 0.0],
+                            [0.0, math.cos(shear), 0.0],
+                            [0.0, 0.0, 1.0]])
+    x = ndimage.interpolation.affine_transform(x, shear_matrix,
+                                               mode=fill_mode,
+                                               order=3,
+                                               cval=cval)
+    return x
+
 
 def random_channel_shift(x, rg):
     # TODO
     pass
 
-def random_zoom(x, rg, fill_mode="nearest", cval=0.):
-    zoom_w = random.uniform(1.-rg, 1.)
-    zoom_h = random.uniform(1.-rg, 1.)
-    x = ndimage.interpolation.zoom(x, zoom=(1., zoom_w, zoom_h), mode=fill_mode, cval=cval)
-    return x # shape of result will be different from shape of input!
-
 
+def random_zoom(x, rg, fill_mode='nearest', cval=0.):
+    zoom_w = np.random.uniform(1.-rg, 1.)
+    zoom_h = np.random.uniform(1.-rg, 1.)
+    x = ndimage.interpolation.zoom(x, zoom=(1., zoom_w, zoom_h),
+                                   mode=fill_mode,
+                                   cval=cval)
+    return x  # shape of result will be different from shape of input!
 
 
 def array_to_img(x, scale=True):
     from PIL import Image
-    x = x.transpose(1, 2, 0) 
+    x = x.transpose(1, 2, 0)
     if scale:
         x += max(-np.min(x), 0)
         x /= np.max(x)
         x *= 255
     if x.shape[2] == 3:
         # RGB
-        return Image.fromarray(x.astype("uint8"), "RGB")
+        return Image.fromarray(x.astype('uint8'), 'RGB')
     else:
         # grayscale
-        return Image.fromarray(x[:,:,0].astype("uint8"), "L")
+        return Image.fromarray(x[:, :, 0].astype('uint8'), 'L')
 
 
 def img_to_array(img):
     x = np.asarray(img, dtype='float32')
-    if len(x.shape)==3:
+    if len(x.shape) == 3:
         # RGB: height, width, channel -> channel, height, width
         x = x.transpose(2, 0, 1)
     else:
@@ -107,128 +114,179 @@ def load_img(path, grayscale=False):
     img = Image.open(path)
     if grayscale:
         img = img.convert('L')
-    else: # Assure 3 channel even when loaded image is grayscale
+    else:  # Ensure 3 channel even when loaded image is grayscale
         img = img.convert('RGB')
     return img
 
 
 def list_pictures(directory, ext='jpg|jpeg|bmp|png'):
-    return [join(directory,f) for f in listdir(directory) \
-        if isfile(join(directory,f)) and re.match('([\w]+\.(?:' + ext + '))', f)]
-
+    return [join(directory, f) for f in listdir(directory)
+            if isfile(join(directory, f)) and re.match('([\w]+\.(?:' + ext + '))', f)]
 
 
 class ImageDataGenerator(object):
-    '''
-        Generate minibatches with 
-        realtime data augmentation.
-    '''
-    def __init__(self, 
-            featurewise_center=True, # set input mean to 0 over the dataset
-            samplewise_center=False, # set each sample mean to 0
-            featurewise_std_normalization=True, # divide inputs by std of the dataset
-            samplewise_std_normalization=False, # divide each input by its std
+    '''Generate minibatches with
+    real-time data augmentation.
 
-            zca_whitening=False, # apply ZCA whitening
-            rotation_range=0., # degrees (0 to 180)
-            width_shift_range=0., # fraction of total width
-            height_shift_range=0., # fraction of total height
-            horizontal_flip=False,
-            vertical_flip=False,
-        ):
+    # Arguments
+        featurewise_center: set input mean to 0 over the dataset.
+        samplewise_center: set each sample mean to 0.
+        featurewise_std_normalization: divide inputs by std of the dataset.
+        samplewise_std_normalization: divide each input by its std.
+        zca_whitening: apply ZCA whitening.
+        rotation_range: degrees (0 to 180).
+        width_shift_range: fraction of total width.
+        height_shift_range: fraction of total height.
+        shear_range: shear intensity (shear angle in radians).
+        horizontal_flip: whether to randomly flip images horizontally.
+        vertical_flip: whether to randomly flip images vertically.
+    '''
+    def __init__(self,
+                 featurewise_center=True,
+                 samplewise_center=False,
+                 featurewise_std_normalization=True,
+                 samplewise_std_normalization=False,
+                 zca_whitening=False,
+                 rotation_range=0.,
+                 width_shift_range=0.,
+                 height_shift_range=0.,
+                 shear_range=0.,
+                 horizontal_flip=False,
+                 vertical_flip=False):
         self.__dict__.update(locals())
         self.mean = None
         self.std = None
         self.principal_components = None
+        self.lock = threading.Lock()
 
+    def _flow_index(self, N, batch_size=32, shuffle=False, seed=None):
+        b = 0
+        total_b = 0
+        while 1:
+            if b == 0:
+                if seed is not None:
+                    np.random.seed(seed + total_b)
 
-    def flow(self, X, y, batch_size=32, shuffle=False, seed=None, save_to_dir=None, save_prefix="", save_format="jpeg"):
-        if seed:
-            random.seed(seed)
+                if shuffle:
+                    index_array = np.random.permutation(N)
+                else:
+                    index_array = np.arange(N)
 
-        if shuffle:
-            seed = random.randint(1, 10e6)
-            np.random.seed(seed)
-            np.random.shuffle(X)
-            np.random.seed(seed)
-            np.random.shuffle(y)
-
-        nb_batch = int(math.ceil(float(X.shape[0])/batch_size))
-        for b in range(nb_batch):
-            batch_end = (b+1)*batch_size
-            if batch_end > X.shape[0]:
-                nb_samples = X.shape[0] - b*batch_size
+            current_index = (b * batch_size) % N
+            if N >= current_index + batch_size:
+                current_batch_size = batch_size
             else:
-                nb_samples = batch_size
+                current_batch_size = N - current_index
 
-            bX = np.zeros(tuple([nb_samples]+list(X.shape)[1:]))
-            for i in range(nb_samples):
-                x = X[b*batch_size+i]
-                x = self.random_transform(x.astype("float32"))
-                x = self.standardize(x)
-                bX[i] = x
+            if current_batch_size == batch_size:
+                b += 1
+            else:
+                b = 0
+            total_b += 1
+            yield (index_array[current_index: current_index + current_batch_size],
+                   current_index, current_batch_size)
 
-            if save_to_dir:
-                for i in range(nb_samples):
-                    img = array_to_img(bX[i], scale=True)
-                    img.save(save_to_dir + "/" + save_prefix + "_" + str(i) + "." + save_format)
+    def flow(self, X, y, batch_size=32, shuffle=False, seed=None,
+             save_to_dir=None, save_prefix='', save_format='jpeg'):
+        assert len(X) == len(y)
+        self.X = X
+        self.y = y
+        self.save_to_dir = save_to_dir
+        self.save_prefix = save_prefix
+        self.save_format = save_format
+        self.flow_generator = self._flow_index(X.shape[0], batch_size,
+                                               shuffle, seed)
+        return self
 
-            yield bX, y[b*batch_size:b*batch_size+nb_samples]
+    def __iter__(self):
+        # needed if we want to do something like:
+        # for x, y in data_gen.flow(...):
+        return self
 
+    def next(self):
+        # for python 2.x.
+        # Keeps under lock only the mechanism which advances
+        # the indexing of each batch
+        # see # http://anandology.com/blog/using-iterators-and-generators/
+        with self.lock:
+            index_array, current_index, current_batch_size = next(self.flow_generator)
+        # The transformation of images is not under thread lock so it can be done in parallel
+        bX = np.zeros(tuple([current_batch_size] + list(self.X.shape)[1:]))
+        for i, j in enumerate(index_array):
+            x = self.X[j]
+            x = self.random_transform(x.astype('float32'))
+            x = self.standardize(x)
+            bX[i] = x
+        if self.save_to_dir:
+            for i in range(current_batch_size):
+                img = array_to_img(bX[i], scale=True)
+                img.save(self.save_to_dir + '/' + self.save_prefix + '_' + str(current_index + i) + '.' + self.save_format)
+        bY = self.y[index_array]
+        return bX, bY
+
+    def __next__(self):
+        # for python 3.x.
+        return self.next()
 
     def standardize(self, x):
+        if self.samplewise_center:
+            x -= np.mean(x, axis=1, keepdims=True)
+        if self.samplewise_std_normalization:
+            x /= (np.std(x, axis=1, keepdims=True) + 1e-7)
+
         if self.featurewise_center:
             x -= self.mean
         if self.featurewise_std_normalization:
-            x /= self.std
+            x /= (self.std + 1e-7)
 
         if self.zca_whitening:
-            flatx = np.reshape(x, (x.shape[0]*x.shape[1]*x.shape[2]))
+            flatx = np.reshape(x, (x.shape[0] * x.shape[1] * x.shape[2]))
             whitex = np.dot(flatx, self.principal_components)
             x = np.reshape(whitex, (x.shape[0], x.shape[1], x.shape[2]))
 
-        if self.samplewise_center:
-            x -= np.mean(x)
-        if self.samplewise_std_normalization:
-            x /= np.std(x)
-
         return x
 
-
     def random_transform(self, x):
         if self.rotation_range:
             x = random_rotation(x, self.rotation_range)
         if self.width_shift_range or self.height_shift_range:
             x = random_shift(x, self.width_shift_range, self.height_shift_range)
         if self.horizontal_flip:
-            if random.random() < 0.5:
+            if np.random.random() < 0.5:
                 x = horizontal_flip(x)
         if self.vertical_flip:
-            if random.random() < 0.5:
+            if np.random.random() < 0.5:
                 x = vertical_flip(x)
-
+        if self.shear_range:
+            x = random_shear(x, self.shear_range)
         # TODO:
         # zoom
         # barrel/fisheye
-        # shearing
         # channel shifting
         return x
 
     def fit(self, X,
-            augment=False,  # fit on randomly augmented samples
-            rounds=1,  # if augment, how many augmentation passes over the data do we use
+            augment=False,
+            rounds=1,
             seed=None):
-        '''
-            Required for featurewise_center, featurewise_std_normalization and zca_whitening.
+        '''Required for featurewise_center, featurewise_std_normalization
+        and zca_whitening.
+
+        # Arguments
+            X: Numpy array, the data to fit on.
+            augment: whether to fit on randomly augmented samples
+            rounds: if `augment`,
+                how many augmentation passes to do over the data
+            seed: random seed.
         '''
         X = np.copy(X)
         if augment:
-            aX = np.zeros(tuple([rounds*X.shape[0]]+list(X.shape)[1:]))
+            aX = np.zeros(tuple([rounds * X.shape[0]] + list(X.shape)[1:]))
             for r in range(rounds):
                 for i in range(X.shape[0]):
                     img = array_to_img(X[i])
                     img = self.random_transform(img)
-                    aX[i+r*X.shape[0]] = img_to_array(img)
+                    aX[i + r * X.shape[0]] = img_to_array(img)
             X = aX
 
         if self.featurewise_center:
@@ -236,11 +294,19 @@ class ImageDataGenerator(object):
             X -= self.mean
         if self.featurewise_std_normalization:
             self.std = np.std(X, axis=0)
-            X /= self.std
+            X /= (self.std + 1e-7)
 
         if self.zca_whitening:
-            flatX = np.reshape(X, (X.shape[0], X.shape[1]*X.shape[2]*X.shape[3]))
-            fudge = 10e-6
+            flatX = np.reshape(X, (X.shape[0], X.shape[1] * X.shape[2] * X.shape[3]))
             sigma = np.dot(flatX.T, flatX) / flatX.shape[1]
             U, S, V = linalg.svd(sigma)
-            self.principal_components = np.dot(np.dot(U, np.diag(1. / np.sqrt(S + fudge))), U.T)
+            self.principal_components = np.dot(np.dot(U, np.diag(1. / np.sqrt(S + 10e-7))), U.T)
+
+
+class GraphImageDataGenerator(ImageDataGenerator):
+    '''Example of how to build a generator for a Graph model
+    '''
+
+    def next(self):
+        bX, bY = super(GraphImageDataGenerator, self).next()
+        return {'input': bX, 'output': bY}

keras/preprocessing/sequence.py

@@ -4,19 +4,20 @@ import numpy as np
 import random
 from six.moves import range
 
-def pad_sequences(sequences, maxlen=None, dtype='int32', padding='pre', truncating='pre', value=0.):
-    """
-        Pad each sequence to the same length:
-        the length of the longest sequence.
 
-        If maxlen is provided, any sequence longer
-        than maxlen is truncated to maxlen. Truncation happens off either the beginning (default) or
-        the end of the sequence.
+def pad_sequences(sequences, maxlen=None, dtype='int32',
+                  padding='pre', truncating='pre', value=0.):
+    '''Pads each sequence to the same length:
+    the length of the longest sequence.
 
-        Supports post-padding and pre-padding (default).
+    If maxlen is provided, any sequence longer
+    than maxlen is truncated to maxlen.
+    Truncation happens off either the beginning (default) or
+    the end of the sequence.
 
-        Parameters:
-        -----------
+    Supports post-padding and pre-padding (default).
+
+    # Arguments
         sequences: list of lists where each element is a sequence
         maxlen: int, maximum length
         dtype: type to cast the resulting sequence.
@@ -25,53 +26,64 @@ def pad_sequences(sequences, maxlen=None, dtype='int32', padding='pre', truncati
             maxlen either in the beginning or in the end of the sequence
         value: float, value to pad the sequences to the desired value.
 
-        Returns:
+    # Returns
         x: numpy array with dimensions (number_of_sequences, maxlen)
-
-    """
+    '''
     lengths = [len(s) for s in sequences]
 
     nb_samples = len(sequences)
     if maxlen is None:
         maxlen = np.max(lengths)
 
-    x = (np.ones((nb_samples, maxlen)) * value).astype(dtype)
+    # take the sample shape from the first non empty sequence
+    # checking for consistency in the main loop below.
+    sample_shape = tuple()
+    for s in sequences:
+        if len(s) > 0:
+            sample_shape = np.asarray(s).shape[1:]
+            break
+
+    x = (np.ones((nb_samples, maxlen) + sample_shape) * value).astype(dtype)
     for idx, s in enumerate(sequences):
         if len(s) == 0:
-            continue # empty list was found
+            continue  # empty list was found
         if truncating == 'pre':
             trunc = s[-maxlen:]
         elif truncating == 'post':
             trunc = s[:maxlen]
         else:
-            raise ValueError("Truncating type '%s' not understood" % padding)
+            raise ValueError('Truncating type "%s" not understood' % truncating)
+
+        # check `trunc` has expected shape
+        trunc = np.asarray(trunc, dtype=dtype)
+        if trunc.shape[1:] != sample_shape:
+            raise ValueError('Shape of sample %s of sequence at position %s is different from expected shape %s' %
+                             (trunc.shape[1:], idx, sample_shape))
 
         if padding == 'post':
             x[idx, :len(trunc)] = trunc
         elif padding == 'pre':
             x[idx, -len(trunc):] = trunc
         else:
-            raise ValueError("Padding type '%s' not understood" % padding)
+            raise ValueError('Padding type "%s" not understood' % padding)
     return x
 
 
 def make_sampling_table(size, sampling_factor=1e-5):
-    '''
-        This generates an array where the ith element
-        is the probability that a word of rank i would be sampled,
-        according to the sampling distribution used in word2vec.
+    '''This generates an array where the ith element
+    is the probability that a word of rank i would be sampled,
+    according to the sampling distribution used in word2vec.
 
-        The word2vec formula is:
-            p(word) = min(1, sqrt(word.frequency/sampling_factor) / (word.frequency/sampling_factor))
+    The word2vec formula is:
+        p(word) = min(1, sqrt(word.frequency/sampling_factor) / (word.frequency/sampling_factor))
 
-        We assume that the word frequencies follow Zipf's law (s=1) to derive
-        a numerical approximation of frequency(rank):
-           frequency(rank) ~ 1/(rank * (log(rank) + gamma) + 1/2 - 1/(12*rank))
+    We assume that the word frequencies follow Zipf's law (s=1) to derive
+    a numerical approximation of frequency(rank):
+       frequency(rank) ~ 1/(rank * (log(rank) + gamma) + 1/2 - 1/(12*rank))
         where gamma is the Euler-Mascheroni constant.
 
-        Parameters:
-        -----------
-        size: int, number of possible words to sample. 
+    # Arguments
+        size: int, number of possible words to sample.
     '''
     gamma = 0.577
     rank = np.array(list(range(size)))
@@ -85,28 +97,28 @@ def make_sampling_table(size, sampling_factor=1e-5):
 def skipgrams(sequence, vocabulary_size,
               window_size=4, negative_samples=1., shuffle=True,
               categorical=False, sampling_table=None):
-    '''
-        Take a sequence (list of indexes of words),
-        returns couples of [word_index, other_word index] and labels (1s or 0s),
-        where label = 1 if 'other_word' belongs to the context of 'word',
-        and label=0 if 'other_word' is ramdomly sampled
+    '''Take a sequence (list of indexes of words),
+    returns couples of [word_index, other_word index] and labels (1s or 0s),
+    where label = 1 if 'other_word' belongs to the context of 'word',
+    and label=0 if 'other_word' is ramdomly sampled
 
-        Paramaters:
-        -----------
+    # Arguments
         vocabulary_size: int. maximum possible word index + 1
-        window_size: int. actually half-window. The window of a word wi will be [i-window_size, i+window_size+1]
-        negative_samples: float >= 0. 0 for no negative (=random) samples. 1 for same number as positive samples. etc.
-        categorical: bool. if False, labels will be integers (eg. [0, 1, 1 .. ]),
+        window_size: int. actually half-window.
+            The window of a word wi will be [i-window_size, i+window_size+1]
+        negative_samples: float >= 0. 0 for no negative (=random) samples.
+            1 for same number as positive samples. etc.
+        categorical: bool. if False, labels will be
+            integers (eg. [0, 1, 1 .. ]),
             if True labels will be categorical eg. [[1,0],[0,1],[0,1] .. ]
 
-        Returns:
-        --------
+    # Returns
         couples, lables: where `couples` are int pairs and
             `labels` are either 0 or 1.
 
-        Notes:
-        ------
-        By convention, index 0 in the vocabulary is a non-word and will be skipped.
+    # Notes
+        By convention, index 0 in the vocabulary is
+        a non-word and will be skipped.
     '''
     couples = []
     labels = []
@@ -135,7 +147,7 @@ def skipgrams(sequence, vocabulary_size,
         words = [c[0] for c in couples]
         random.shuffle(words)
 
-        couples += [[words[i%len(words)], random.randint(1, vocabulary_size-1)] for i in range(nb_negative_samples)]
+        couples += [[words[i %len(words)], random.randint(1, vocabulary_size-1)] for i in range(nb_negative_samples)]
         if categorical:
             labels += [[1,0]]*nb_negative_samples
         else:

keras/preprocessing/text.py

@@ -1,11 +1,11 @@
 # -*- coding: utf-8 -*-
-'''
-    These preprocessing utils would greatly benefit
-    from a fast Cython rewrite.
+'''These preprocessing utilities would greatly benefit
+from a fast Cython rewrite.
 '''
 from __future__ import absolute_import
 
-import string, sys
+import string
+import sys
 import numpy as np
 from six.moves import range
 from six.moves import zip
@@ -40,7 +40,7 @@ def one_hot(text, n, filters=base_filter(), lower=True, split=" "):
 
 class Tokenizer(object):
     def __init__(self, nb_words=None, filters=base_filter(),
-                 lower=True, split=' '):
+                 lower=True, split=' ', char_level=False):
         '''The class allows to vectorize a text corpus, by turning each
         text into either a sequence of integers (each integer being the index
         of a token in a dictionary) or into a vector where the coefficient
@@ -55,6 +55,7 @@ class Tokenizer(object):
                 tabs and line breaks, minus the `'` character.
             lower: boolean. Whether to convert the texts to lowercase.
             split: character or string to use for token splitting.
+            char_level: if True, every character will be treated as a word.
 
         By default, all punctuation is removed, turning the texts into
         space-separated sequences of words
@@ -70,10 +71,10 @@ class Tokenizer(object):
         self.lower = lower
         self.nb_words = nb_words
         self.document_count = 0
+        self.char_level = char_level
 
     def fit_on_texts(self, texts):
-        '''
-            required before using texts_to_sequences or texts_to_matrix
+        '''Required before using texts_to_sequences or texts_to_matrix
 
         # Arguments
             texts: can be a list of strings,
@@ -82,7 +83,7 @@ class Tokenizer(object):
         self.document_count = 0
         for text in texts:
             self.document_count += 1
-            seq = text_to_word_sequence(text, self.filters, self.lower, self.split)
+            seq = text if self.char_level else text_to_word_sequence(text, self.filters, self.lower, self.split)
             for w in seq:
                 if w in self.word_counts:
                     self.word_counts[w] += 1
@@ -104,9 +105,8 @@ class Tokenizer(object):
             self.index_docs[self.word_index[w]] = c
 
     def fit_on_sequences(self, sequences):
-        '''
-            required before using sequences_to_matrix
-            (if fit_on_texts was never called)
+        '''Required before using sequences_to_matrix
+        (if fit_on_texts was never called)
         '''
         self.document_count = len(sequences)
         self.index_docs = {}
@@ -119,12 +119,11 @@ class Tokenizer(object):
                     self.index_docs[i] += 1
 
     def texts_to_sequences(self, texts):
-        '''
-            Transform each text in texts in a sequence of integers.
-            Only top "nb_words" most frequent words will be taken into account.
-            Only words known by the tokenizer will be taken into account.
+        '''Transforms each text in texts in a sequence of integers.
+        Only top "nb_words" most frequent words will be taken into account.
+        Only words known by the tokenizer will be taken into account.
 
-            Returns a list of sequences.
+        Returns a list of sequences.
         '''
         res = []
         for vect in self.texts_to_sequences_generator(texts):
@@ -132,71 +131,84 @@ class Tokenizer(object):
         return res
 
     def texts_to_sequences_generator(self, texts):
-        '''
-            Transform each text in texts in a sequence of integers.
-            Only top "nb_words" most frequent words will be taken into account.
-            Only words known by the tokenizer will be taken into account.
+        '''Transforms each text in texts in a sequence of integers.
+        Only top "nb_words" most frequent words will be taken into account.
+        Only words known by the tokenizer will be taken into account.
 
-            Yields individual sequences.
+        Yields individual sequences.
+
+        # Arguments:
+            texts: list of strings.
         '''
         nb_words = self.nb_words
         for text in texts:
-            seq = text_to_word_sequence(text, self.filters, self.lower, self.split)
+            seq = text if self.char_level else text_to_word_sequence(text, self.filters, self.lower, self.split)
             vect = []
             for w in seq:
                 i = self.word_index.get(w)
                 if i is not None:
                     if nb_words and i >= nb_words:
-                        pass
+                        continue
                     else:
                         vect.append(i)
             yield vect
 
-    def texts_to_matrix(self, texts, mode="binary"):
-        '''
-            modes: binary, count, tfidf, freq
+    def texts_to_matrix(self, texts, mode='binary'):
+        '''Convert a list of texts to a Numpy matrix,
+        according to some vectorization mode.
+
+        # Arguments:
+            texts: list of strings.
+            modes: one of "binary", "count", "tfidf", "freq"
         '''
         sequences = self.texts_to_sequences(texts)
         return self.sequences_to_matrix(sequences, mode=mode)
 
-    def sequences_to_matrix(self, sequences, mode="binary"):
-        '''
-            modes: binary, count, tfidf, freq
+    def sequences_to_matrix(self, sequences, mode='binary'):
+        '''Converts a list of sequences into a Numpy matrix,
+        according to some vectorization mode.
+
+        # Arguments:
+            sequences: list of sequences
+                (a sequence is a list of integer word indices).
+            modes: one of "binary", "count", "tfidf", "freq"
         '''
         if not self.nb_words:
             if self.word_index:
                 nb_words = len(self.word_index) + 1
             else:
-                raise Exception("Specify a dimension (nb_words argument), or fit on some text data first.")
+                raise Exception('Specify a dimension (nb_words argument), '
+                                'or fit on some text data first.')
         else:
             nb_words = self.nb_words
 
-        if mode == "tfidf" and not self.document_count:
-            raise Exception("Fit the Tokenizer on some data before using tfidf mode.")
+        if mode == 'tfidf' and not self.document_count:
+            raise Exception('Fit the Tokenizer on some data '
+                            'before using tfidf mode.')
 
         X = np.zeros((len(sequences), nb_words))
         for i, seq in enumerate(sequences):
             if not seq:
-                pass
+                continue
             counts = {}
             for j in seq:
                 if j >= nb_words:
-                    pass
+                    continue
                 if j not in counts:
                     counts[j] = 1.
                 else:
                     counts[j] += 1
             for j, c in list(counts.items()):
-                if mode == "count":
+                if mode == 'count':
                     X[i][j] = c
-                elif mode == "freq":
+                elif mode == 'freq':
                     X[i][j] = c / len(seq)
-                elif mode == "binary":
+                elif mode == 'binary':
                     X[i][j] = 1
-                elif mode == "tfidf":
+                elif mode == 'tfidf':
                     tf = np.log(c / len(seq))
                     df = (1 + np.log(1 + self.index_docs.get(j, 0) / (1 + self.document_count)))
                     X[i][j] = tf / df
                 else:
-                    raise Exception("Unknown vectorization mode: " + str(mode))
+                    raise Exception('Unknown vectorization mode: ' + str(mode))
         return X

keras/utils/data_utils.py

@@ -0,0 +1,95 @@
+from __future__ import absolute_import
+from __future__ import print_function
+
+import tarfile
+import os
+import sys
+import shutil
+from six.moves.urllib.request import urlopen
+from six.moves.urllib.error import URLError, HTTPError
+
+from ..utils.generic_utils import Progbar
+
+
+# Under Python 2, 'urlretrieve' relies on FancyURLopener from legacy
+# urllib module, known to have issues with proxy management
+if sys.version_info[0] == 2:
+    def urlretrieve(url, filename, reporthook=None, data=None):
+        def chunk_read(response, chunk_size=8192, reporthook=None):
+            total_size = response.info().get('Content-Length').strip()
+            total_size = int(total_size)
+            count = 0
+            while 1:
+                chunk = response.read(chunk_size)
+                if not chunk:
+                    break
+                count += 1
+                if reporthook:
+                    reporthook(count, chunk_size, total_size)
+                yield chunk
+
+        response = urlopen(url, data)
+        with open(filename, 'wb') as fd:
+            for chunk in chunk_read(response, reporthook=reporthook):
+                fd.write(chunk)
+else:
+    from six.moves.urllib.request import urlretrieve
+
+
+def get_file(fname, origin, untar=False):
+    datadir_base = os.path.expanduser(os.path.join('~', '.keras'))
+    if not os.access(datadir_base, os.W_OK):
+        datadir_base = os.path.join('/tmp', '.keras')
+    datadir = os.path.join(datadir_base, 'datasets')
+    if not os.path.exists(datadir):
+        os.makedirs(datadir)
+
+    if untar:
+        untar_fpath = os.path.join(datadir, fname)
+        fpath = untar_fpath + '.tar.gz'
+    else:
+        fpath = os.path.join(datadir, fname)
+
+    if not os.path.exists(fpath):
+        print('Downloading data from',  origin)
+        global progbar
+        progbar = None
+
+        def dl_progress(count, block_size, total_size):
+            global progbar
+            if progbar is None:
+                progbar = Progbar(total_size)
+            else:
+                progbar.update(count*block_size)
+
+        error_msg = 'URL fetch failure on {}: {} -- {}'
+        try:
+            try:
+                urlretrieve(origin, fpath, dl_progress)
+            except URLError as e:
+                raise Exception(error_msg.format(origin, e.errno, e.reason))
+            except HTTPError as e:
+                raise Exception(error_msg.format(origin, e.code, e.msg))
+        except (Exception, KeyboardInterrupt) as e:
+            if os.path.exists(fpath):
+                os.remove(fpath)
+            raise e
+        progbar = None
+
+    if untar:
+        if not os.path.exists(untar_fpath):
+            print('Untaring file...')
+            tfile = tarfile.open(fpath, 'r:gz')
+            try:
+                tfile.extractall(path=datadir)
+            except (Exception, KeyboardInterrupt) as e:
+                if os.path.exists(untar_fpath):
+                    if os.path.isfile(untar_fpath):
+                        os.remove(untar_fpath)
+                    else:
+                        shutil.rmtree(untar_fpath)
+                raise e
+            tfile.close()
+        return untar_fpath
+
+    return fpath

keras/utils/generic_utils.py

@@ -63,15 +63,15 @@ class Progbar(object):
             numdigits = int(np.floor(np.log10(self.target))) + 1
             barstr = '%%%dd/%%%dd [' % (numdigits, numdigits)
             bar = barstr % (current, self.target)
-            prog = float(current)/self.target
-            prog_width = int(self.width*prog)
+            prog = float(current) / self.target
+            prog_width = int(self.width * prog)
             if prog_width > 0:
-                bar += ('='*(prog_width-1))
+                bar += ('=' * (prog_width-1))
                 if current < self.target:
                     bar += '>'
                 else:
                     bar += '='
-            bar += ('.'*(self.width-prog_width))
+            bar += ('.' * (self.width - prog_width))
             bar += ']'
             sys.stdout.write(bar)
             self.total_width = len(bar)
@@ -80,7 +80,7 @@ class Progbar(object):
                 time_per_unit = (now - self.start) / current
             else:
                 time_per_unit = 0
-            eta = time_per_unit*(self.target - current)
+            eta = time_per_unit * (self.target - current)
             info = ''
             if current < self.target:
                 info += ' - ETA: %ds' % eta
@@ -99,7 +99,7 @@ class Progbar(object):
 
             self.total_width += len(info)
             if prev_total_width > self.total_width:
-                info += ((prev_total_width-self.total_width) * " ")
+                info += ((prev_total_width - self.total_width) * " ")
 
             sys.stdout.write(info)
             sys.stdout.flush()
@@ -120,4 +120,4 @@ class Progbar(object):
                 sys.stdout.write(info + "\n")
 
     def add(self, n, values=[]):
-        self.update(self.seen_so_far+n, values)
+        self.update(self.seen_so_far + n, values)

keras/utils/layer_utils.py

@@ -10,6 +10,7 @@ from ..layers.embeddings import *
 from ..layers.noise import *
 from ..layers.normalization import *
 from ..layers.recurrent import *
+from ..layers.wrappers import *
 from ..layers import containers
 from .. import regularizers
 from .. import constraints
@@ -56,6 +57,7 @@ def container_from_config(original_layer_dict, custom_objects={}):
         for node in nodes:
             layer = container_from_config(layer_dict['nodes'].get(node['name']))
             node['layer'] = layer
+            node['create_output'] = False  # outputs will be added below
             graph_layer.add_node(**node)
 
         outputs = layer_dict.get('output_config')
@@ -71,10 +73,18 @@ def container_from_config(original_layer_dict, custom_objects={}):
                 kwargs[kwarg] = layer_dict[kwarg]
         return AutoEncoder(**kwargs)
 
-    else:
+    elif name == 'TimeDistributed':
+        child_layer = container_from_config(layer_dict.pop('layer'))
+        # the "name" keyword argument of layers is saved as "custom_name"
+        if 'custom_name' in layer_dict:
+            layer_dict['name'] = layer_dict.pop('custom_name')
+        return TimeDistributed(child_layer, **layer_dict)
+
+    else:  # this is a non-topological layer (e.g. Dense, etc.)
         layer_dict.pop('name')
 
         for k, v in layer_dict.items():
+            # a dictionary argument may be a regularizer or constraint
             if isinstance(v, dict):
                 vname = v.pop('name')
                 if vname in [x for x, y in inspect.getmembers(constraints, predicate=inspect.isclass)]:
@@ -85,6 +95,10 @@ def container_from_config(original_layer_dict, custom_objects={}):
                     # not a regularizer of constraint, don't touch it
                     v['name'] = vname
 
+        # the "name" keyword argument of layers is saved as "custom_name"
+        if 'custom_name' in layer_dict:
+            layer_dict['name'] = layer_dict.pop('custom_name')
+
         base_layer = get_layer(name, layer_dict)
         return base_layer
 

keras/utils/test_utils.py

@@ -12,7 +12,7 @@ def get_test_data(nb_train=1000, nb_test=500, input_shape=(10,), output_shape=(2
     '''
     nb_sample = nb_train + nb_test
     if classification:
-        y = np.random.randint(0, nb_class, size=(nb_sample, 1))
+        y = np.random.randint(0, nb_class, size=(nb_sample,))
         X = np.zeros((nb_sample,) + input_shape)
         for i in range(nb_sample):
             X[i] = np.random.normal(loc=y[i], scale=0.7, size=input_shape)

keras/utils/visualize_util.py

@@ -149,6 +149,6 @@ def to_graph(model, **kwargs):
     return ModelToDot()(model, **kwargs)
 
 
-def plot(model, to_file='model.png'):
-    graph = to_graph(model)
+def plot(model, to_file='model.png', **kwargs):
+    graph = to_graph(model, **kwargs)
     graph.write_png(to_file)

keras/wrappers/scikit_learn.py

@@ -1,266 +1,268 @@
 from __future__ import absolute_import
-import abc
 import copy
+import inspect
+import types
 import numpy as np
 
 from ..utils.np_utils import to_categorical
+from ..models import Sequential
 
 
 class BaseWrapper(object):
-    """
-    Base class for the Keras scikit-learn wrapper.
+    '''Base class for the Keras scikit-learn wrapper.
 
-    Warning: This class should not be used directly. Use derived classes instead.
+    Warning: This class should not be used directly.
+    Use descendant classes instead.
 
-    Parameters
-    ----------
-    train_batch_size : int, optional
-        Number of training samples evaluated at a time.
-    test_batch_size : int, optional
-        Number of test samples evaluated at a time.
-    nb_epochs : int, optional
-        Number of training epochs.
-    shuffle : boolean, optional
-        Whether to shuffle the samples at each epoch.
-    show_accuracy : boolean, optional
-        Whether to display class accuracy in the logs at each epoch.
-    validation_split : float [0, 1], optional
-        Fraction of the data to use as held-out validation data.
-    validation_data : tuple (X, y), optional
-        Data to be used as held-out validation data. Will override validation_split.
-    callbacks : list, optional
-        List of callbacks to apply during training.
-    verbose : int, optional
-        Verbosity level.
-    """
-    __metaclass__ = abc.ABCMeta
+    # Arguments
+        build_fn: callable function or class instance
+        sk_params: model parameters & fitting parameters
 
-    @abc.abstractmethod
-    def __init__(self, model, optimizer, loss,
-                 train_batch_size=128, test_batch_size=128,
-                 nb_epoch=100, shuffle=True, show_accuracy=False,
-                 validation_split=0, validation_data=None, callbacks=None,
-                 verbose=0,):
-        self.model = model
-        self.optimizer = optimizer
-        self.loss = loss
-        self.compiled_model_ = None
-        self.classes_ = []
-        self.config_ = []
-        self.weights_ = []
+    The build_fn should construct, compile and return a Keras model, which
+    will then be used to fit/predict. One of the following
+    three values could be passed to build_fn:
+    1. A function
+    2. An instance of a class that implements the __call__ method
+    3. None. This means you implement a class that inherits from either
+    `KerasClassifier` or `KerasRegressor`. The __call__ method of the
+    present class will then be treated as the default build_fn.
 
-        self.train_batch_size = train_batch_size
-        self.test_batch_size = test_batch_size
-        self.nb_epoch = nb_epoch
-        self.shuffle = shuffle
-        self.show_accuracy = show_accuracy
-        self.validation_split = validation_split
-        self.validation_data = validation_data
-        self.callbacks = [] if callbacks is None else callbacks
+    `sk_params` takes both model parameters and fitting parameters. Legal model
+    parameters are the arguments of `build_fn`. Note that like all other
+    estimators in scikit-learn, 'build_fn' should provide defalult values for
+    its arguments, so that you could create the estimator without passing any
+    values to `sk_params`.
 
-        self.verbose = verbose
+    `sk_params` could also accept parameters for calling `fit`, `predict`,
+    `predict_proba`, and `score` methods (e.g., `nb_epoch`, `batch_size`).
+    fitting (predicting) parameters are selected in the following order:
+
+    1. Values passed to the dictionary arguments of
+    `fit`, `predict`, `predict_proba`, and `score` methods
+    2. Values passed to `sk_params`
+    3. The default values of the `keras.models.Sequential`
+    `fit`, `predict`, `predict_proba` and `score` methods
+
+    When using scikit-learn's `grid_search` API, legal tunable parameters are
+    those you could pass to `sk_params`, including fitting parameters.
+    In other words, you could use `grid_search` to search for the best
+    `batch_size` or `nb_epoch` as well as the model parameters.
+    '''
+
+    def __init__(self, build_fn=None, **sk_params):
+        self.build_fn = build_fn
+        self.sk_params = sk_params
+        self.check_params(sk_params)
+
+    def check_params(self, params):
+        '''Check for user typos in "params" keys to avoid
+        unwanted usage of default values
+
+        # Arguments
+            params: dictionary
+                The parameters to be checked
+        '''
+        legal_params_fns = [Sequential.fit, Sequential.predict,
+                            Sequential.predict_classes, Sequential.evaluate]
+        if self.build_fn is None:
+            legal_params_fns.append(self.__call__)
+        elif not isinstance(self.build_fn, types.FunctionType):
+            legal_params_fns.append(self.build_fn.__call__)
+        else:
+            legal_params_fns.append(self.build_fn)
+
+        legal_params = []
+        for fn in legal_params_fns:
+            legal_params += inspect.getargspec(fn)[0]
+        legal_params = set(legal_params)
+
+        for params_name in params:
+            if params_name not in legal_params:
+                assert False, '{} is not a legal parameter'.format(params_name)
 
     def get_params(self, deep=True):
-        """
-        Get parameters for this estimator.
+        '''Get parameters for this estimator.
 
-        Parameters
-        ----------
-        deep: boolean, optional
-            If True, will return the parameters for this estimator and
-            contained subobjects that are estimators.
+        # Arguments
+            deep: boolean, optional
+                If True, will return the parameters for this estimator and
+                contained sub-objects that are estimators.
 
-        Returns
-        -------
-        params : dict
-            Dictionary of parameter names mapped to their values.
-        """
-        return {'model': self.model, 'optimizer': self.optimizer, 'loss': self.loss}
+        # Returns
+            params : dict
+                Dictionary of parameter names mapped to their values.
+        '''
+        res = copy.deepcopy(self.sk_params)
+        res.update({'build_fn': self.build_fn})
+        return res
 
     def set_params(self, **params):
-        """
-        Set the parameters of this estimator.
+        '''Set the parameters of this estimator.
 
-        Parameters
-        ----------
+        # Arguments
         params: dict
             Dictionary of parameter names mapped to their values.
 
-        Returns
-        -------
-        self
-        """
-        for parameter, value in params.items():
-            setattr(self, parameter, value)
+        # Returns
+            self
+        '''
+        self.check_params(params)
+        self.sk_params.update(params)
         return self
 
-    def fit(self, X, y):
-        """
-        Fit the model according to the given training data.
+    def fit(self, X, y, **kwargs):
+        '''Construct a new model with build_fn and fit the model according
+        to the given training data.
 
-        Makes a copy of the un-compiled model definition to use for
-        compilation and fitting, leaving the original definition
-        intact.
+        # Arguments
+            X : array-like, shape `(n_samples, n_features)`
+                Training samples where n_samples in the number of samples
+                and n_features is the number of features.
+            y : array-like, shape `(n_samples,)` or `(n_samples, n_outputs)`
+                True labels for X.
+            kwargs: dictionary arguments
+                Legal arguments are the arguments of `Sequential.fit`
 
-        Parameters
-        ----------
-        X : array-like, shape = (n_samples, n_features)
-            Training samples where n_samples in the number of samples
-            and n_features is the number of features.
-        y : array-like, shape = (n_samples) or (n_samples, n_outputs)
-            True labels for X.
+        # Returns
+            history : object
+                details about the training history at each epoch.
+        '''
 
-        Returns
-        -------
-        history : object
-            Returns details about the training history at each epoch.
-        """
-        if len(y.shape) == 1:
-            self.classes_ = list(np.unique(y))
-            if self.loss == 'categorical_crossentropy':
-                y = to_categorical(y)
+        if self.build_fn is None:
+            self.model = self.__call__(**self.filter_sk_params(self.__call__))
+        elif not isinstance(self.build_fn, types.FunctionType):
+            self.model = self.build_fn(
+                **self.filter_sk_params(self.build_fn.__call__))
         else:
-            self.classes_ = np.arange(0, y.shape[1])
+            self.model = self.build_fn(**self.filter_sk_params(self.build_fn))
 
-        self.compiled_model_ = copy.deepcopy(self.model)
-        self.compiled_model_.compile(optimizer=self.optimizer, loss=self.loss)
-        history = self.compiled_model_.fit(
-            X, y, batch_size=self.train_batch_size, nb_epoch=self.nb_epoch, verbose=self.verbose,
-            shuffle=self.shuffle, show_accuracy=self.show_accuracy,
-            validation_split=self.validation_split, validation_data=self.validation_data,
-            callbacks=self.callbacks)
+        if self.model.loss.__name__ == 'categorical_crossentropy' and len(y.shape) != 2:
+            y = to_categorical(y)
 
-        self.config_ = self.model.get_config()
-        self.weights_ = self.model.get_weights()
+        fit_args = copy.deepcopy(self.filter_sk_params(Sequential.fit))
+        fit_args.update(kwargs)
+
+        history = self.model.fit(X, y, **fit_args)
 
         return history
 
+    def filter_sk_params(self, fn, override={}):
+        '''Filter sk_params and return those in fn's arguments
+
+        # Arguments
+            fn : arbitrary function
+            override: dictionary, values to overrid sk_params
+
+        # Returns
+            res : dictionary dictionary containing variabls
+                in both sk_params and fn's arguments.
+        '''
+        res = {}
+        fn_args = inspect.getargspec(fn)[0]
+        for name, value in self.sk_params.items():
+            if name in fn_args:
+                res.update({name: value})
+        res.update(override)
+        return res
+
 
 class KerasClassifier(BaseWrapper):
-    """
-    Implementation of the scikit-learn classifier API for Keras.
+    '''Implementation of the scikit-learn classifier API for Keras.
+    '''
 
-    Parameters
-    ----------
-    model : object
-        An un-compiled Keras model object is required to use the scikit-learn wrapper.
-    optimizer : string
-        Optimization method used by the model during compilation/training.
-    loss : string
-        Loss function used by the model during compilation/training.
-    """
-    def __init__(self, model, optimizer='adam', loss='categorical_crossentropy', **kwargs):
-        super(KerasClassifier, self).__init__(model, optimizer, loss, **kwargs)
+    def predict(self, X, **kwargs):
+        '''Returns the class predictions for the given test data.
 
-    def predict(self, X):
-        """
-        Returns the class predictions for the given test data.
+        # Arguments
+            X: array-like, shape `(n_samples, n_features)`
+                Test samples where n_samples in the number of samples
+                and n_features is the number of features.
+            kwargs: dictionary arguments
+                Legal arguments are the arguments of `Sequential.predict_classes`.
 
-        Parameters
-        ----------
-        X : array-like, shape = (n_samples, n_features)
-            Test samples where n_samples in the number of samples
-            and n_features is the number of features.
+        # Returns
+            preds: array-like, shape `(n_samples,)`
+                Class predictions.
+        '''
+        kwargs = self.filter_sk_params(Sequential.predict_classes, kwargs)
+        return self.model.predict_classes(X, **kwargs)
 
-        Returns
-        -------
-        preds : array-like, shape = (n_samples)
-            Class predictions.
-        """
-        return self.compiled_model_.predict_classes(
-            X, batch_size=self.test_batch_size, verbose=self.verbose)
+    def predict_proba(self, X, **kwargs):
+        '''Returns class probability estimates for the given test data.
 
-    def predict_proba(self, X):
-        """
-        Returns class probability estimates for the given test data.
+        # Arguments
+            X: array-like, shape `(n_samples, n_features)`
+                Test samples where n_samples in the number of samples
+                and n_features is the number of features.
+            kwargs: dictionary arguments
+                Legal arguments are the arguments of `Sequential.predict_classes`.
 
-        Parameters
-        ----------
-        X : array-like, shape = (n_samples, n_features)
-            Test samples where n_samples in the number of samples
-            and n_features is the number of features.
+        # Returns
+            proba: array-like, shape `(n_samples, n_outputs)`
+                Class probability estimates.
+        '''
+        kwargs = self.filter_sk_params(Sequential.predict_proba, kwargs)
+        return self.model.predict_proba(X, **kwargs)
 
-        Returns
-        -------
-        proba : array-like, shape = (n_samples, n_outputs)
-            Class probability estimates.
-        """
-        return self.compiled_model_.predict_proba(
-            X, batch_size=self.test_batch_size, verbose=self.verbose)
+    def score(self, X, y, **kwargs):
+        '''Returns the mean accuracy on the given test data and labels.
 
-    def score(self, X, y):
-        """
-        Returns the mean accuracy on the given test data and labels.
+        # Arguments
+            X: array-like, shape `(n_samples, n_features)`
+                Test samples where n_samples in the number of samples
+                and n_features is the number of features.
+            y: array-like, shape `(n_samples,)` or `(n_samples, n_outputs)`
+                True labels for X.
+            kwargs: dictionary arguments
+                Legal arguments are the arguments of `Sequential.evaluate`.
 
-        Parameters
-        ----------
-        X : array-like, shape = (n_samples, n_features)
-            Test samples where n_samples in the number of samples
-            and n_features is the number of features.
-        y : array-like, shape = (n_samples) or (n_samples, n_outputs)
-            True labels for X.
-
-        Returns
-        -------
-        score : float
-            Mean accuracy of predictions on X wrt. y.
-        """
-        loss, accuracy = self.compiled_model_.evaluate(
-            X, y, batch_size=self.test_batch_size, show_accuracy=True, verbose=self.verbose)
+        # Returns
+            score: float
+                Mean accuracy of predictions on X wrt. y.
+        '''
+        kwargs = self.filter_sk_params(Sequential.evaluate, kwargs)
+        kwargs.update({'show_accuracy': True})
+        loss, accuracy = self.model.evaluate(X, y, **kwargs)
         return accuracy
 
 
 class KerasRegressor(BaseWrapper):
-    """
-    Implementation of the scikit-learn regressor API for Keras.
+    '''Implementation of the scikit-learn regressor API for Keras.
+    '''
 
-    Parameters
-    ----------
-    model : object
-        An un-compiled Keras model object is required to use the scikit-learn wrapper.
-    optimizer : string
-        Optimization method used by the model during compilation/training.
-    loss : string
-        Loss function used by the model during compilation/training.
-    """
-    def __init__(self, model, optimizer='adam', loss='mean_squared_error', **kwargs):
-        super(KerasRegressor, self).__init__(model, optimizer, loss, **kwargs)
+    def predict(self, X, **kwargs):
+        '''Returns predictions for the given test data.
 
-    def predict(self, X):
-        """
-        Returns predictions for the given test data.
+        # Arguments
+            X: array-like, shape `(n_samples, n_features)`
+                Test samples where n_samples in the number of samples
+                and n_features is the number of features.
+            kwargs: dictionary arguments
+                Legal arguments are the arguments of `Sequential.predict`.
+        # Returns
+            preds: array-like, shape `(n_samples,)`
+                Predictions.
+        '''
+        kwargs = self.filter_sk_params(Sequential.predict, kwargs)
+        return self.model.predict(X, **kwargs)
 
-        Parameters
-        ----------
-        X : array-like, shape = (n_samples, n_features)
-            Test samples where n_samples in the number of samples
-            and n_features is the number of features.
+    def score(self, X, y, **kwargs):
+        '''Returns the mean accuracy on the given test data and labels.
 
-        Returns
-        -------
-        preds : array-like, shape = (n_samples)
-            Predictions.
-        """
-        return self.compiled_model_.predict(
-            X, batch_size=self.test_batch_size, verbose=self.verbose).ravel()
+        # Arguments
+            X: array-like, shape `(n_samples, n_features)`
+                Test samples where n_samples in the number of samples
+                and n_features is the number of features.
+            y: array-like, shape `(n_samples,)`
+                True labels for X.
+            kwargs: dictionary arguments
+                Legal arguments are the arguments of `Sequential.evaluate`.
 
-    def score(self, X, y):
-        """
-        Returns the mean accuracy on the given test data and labels.
-
-        Parameters
-        ----------
-        X : array-like, shape = (n_samples, n_features)
-            Test samples where n_samples in the number of samples
-            and n_features is the number of features.
-        y : array-like, shape = (n_samples)
-            True labels for X.
-
-        Returns
-        -------
-        score : float
-            Loss from predictions on X wrt. y.
-        """
-        loss = self.compiled_model_.evaluate(
-            X, y, batch_size=self.test_batch_size, show_accuracy=False, verbose=self.verbose)
+        # Returns
+            score: float
+                Mean accuracy of predictions on X wrt. y.
+        '''
+        kwargs = self.filter_sk_params(Sequential.evaluate, kwargs)
+        kwargs.update({'show_accuracy': False})
+        loss = self.model.evaluate(X, y, **kwargs)
         return loss

pytest.ini

@@ -5,3 +5,34 @@ addopts=-v
         --durations=10
         --cov-report term-missing
         --cov=keras
+
+# Do not run tests in the build folder
+norecursedirs= build
+
+# PEP-8 The following are ignored:
+# E251 unexpected spaces around keyword / parameter equals
+# E225 missing whitespace around operator
+# E226 missing whitespace around arithmetic operator
+# W291 trailing whitespace
+# W293 blank line contains whitespace
+# E501 line too long (82 > 79 characters)
+# E402 module level import not at top of file - temporary measure to coninue adding ros python packaged in sys.path
+# E731 do not assign a lambda expression, use a def
+# E302 two blank lines between the functions
+# E231 missing whitespace after ,
+# E241 multiple spaces after ','
+# E261 at least two spaces before inline comment
+
+
+pep8ignore=* E251 \
+           * E225 \
+           * E226 \
+           * W291 \
+           * W293 \
+           * E501 \
+           * E402 \
+           * E731 \
+           * E302 \
+           * E231 \
+           * E241 \
+           * E261

setup.py

@@ -3,12 +3,12 @@ from setuptools import find_packages
 
 
 setup(name='Keras',
-      version='0.3.0',
-      description='Theano-based Deep Learning library',
+      version='0.3.3',
+      description='Deep Learning for Python',
       author='Francois Chollet',
       author_email='francois.chollet@gmail.com',
       url='https://github.com/fchollet/keras',
-      download_url='https://github.com/fchollet/keras/tarball/0.3.0',
+      download_url='https://github.com/fchollet/keras/tarball/0.3.3',
       license='MIT',
       install_requires=['theano', 'pyyaml', 'six'],
       extras_require={

tests/integration_tests/test_image_data_tasks.py

@@ -35,7 +35,7 @@ def test_image_classification():
         Activation('relu'),
         Dense(y_test.shape[-1], activation='softmax')
     ])
-    model.compile(loss='categorical_crossentropy', optimizer='sgd')
+    model.compile(loss='categorical_crossentropy', optimizer='rmsprop')
     history = model.fit(X_train, y_train, nb_epoch=10, batch_size=16,
                         validation_data=(X_test, y_test),
                         show_accuracy=True, verbose=0)

tests/integration_tests/test_temporal_data_tasks.py

@@ -5,8 +5,9 @@ import string
 
 from keras.utils.test_utils import get_test_data
 from keras.models import Sequential
-from keras.layers.core import TimeDistributedDense, Dropout, Dense
+from keras.layers.core import TimeDistributedDense, Dropout, Dense, Activation
 from keras.layers.recurrent import GRU, LSTM
+from keras.layers.embeddings import Embedding
 from keras.utils.np_utils import to_categorical
 
 
@@ -37,8 +38,8 @@ def test_temporal_classification():
 
 def test_temporal_regression():
     '''
-    Predict float numbers (regression) based on sequences of float numbers of length 3 using
-    single layer of GRU units
+    Predict float numbers (regression) based on sequences
+    of float numbers of length 3 using a single layer of GRU units
     '''
     np.random.seed(1337)
     (X_train, y_train), (X_test, y_test) = get_test_data(nb_train=500,
@@ -48,7 +49,7 @@ def test_temporal_regression():
                                                          classification=False)
     model = Sequential()
     model.add(GRU(y_train.shape[-1],
-              input_shape=(X_train.shape[1], X_train.shape[2])))
+                  input_shape=(X_train.shape[1], X_train.shape[2])))
     model.compile(loss='hinge', optimizer='adam')
     history = model.fit(X_train, y_train, nb_epoch=5, batch_size=16,
                         validation_data=(X_test, y_test), verbose=0)
@@ -71,7 +72,7 @@ def test_sequence_to_sequence():
 
     model = Sequential()
     model.add(TimeDistributedDense(y_train.shape[-1],
-              input_shape=(X_train.shape[1], X_train.shape[2])))
+                                   input_shape=(X_train.shape[1], X_train.shape[2])))
     model.compile(loss='hinge', optimizer='rmsprop')
     history = model.fit(X_train, y_train, nb_epoch=20, batch_size=16,
                         validation_data=(X_test, y_test), verbose=0)
@@ -127,5 +128,51 @@ def test_stacked_lstm_char_prediction():
     assert(generated == alphabet)
 
 
+def test_masked_temporal():
+    '''
+    Confirm that even with masking on both inputs and outputs, cross-entropies are
+    of the expected scale.
+
+    In this task, there are variable length inputs of integers from 1-9, and a random
+    subset of unmasked outputs. Each of these outputs has a 50% probability of being
+    the input number unchanged, and a 50% probability of being 2*input%10.
+
+    The ground-truth best cross-entropy loss should, then be -log(0.5) = 0.69
+
+    '''
+    np.random.seed(55318)
+    model = Sequential()
+    model.add(Embedding(10, 20, mask_zero=True))
+    model.add(TimeDistributedDense(10))
+    model.add(Activation('softmax'))
+    model.compile(loss='categorical_crossentropy',
+                  optimizer='adam',
+                  sample_weight_mode="temporal")
+
+    X = np.random.random_integers(1, 9, (50000, 20))
+    for rowi in range(X.shape[0]):
+        padding = np.random.random_integers(X.shape[1] / 2)
+        X[rowi, :padding] = 0
+
+    # 50% of the time the correct output is the input.
+    # The other 50% of the time it's 2 * input % 10
+    y = (X * np.random.random_integers(1, 2, X.shape)) % 10
+    Y = np.zeros((y.size, 10), dtype='int32')
+    for i, target in enumerate(y.flat):
+        Y[i, target] = 1
+    Y = Y.reshape(y.shape + (10,))
+
+    # Mask 50% of the outputs via sample weights
+    sample_weight = np.random.random_integers(0, 1, y.shape)
+    print("X shape: ", X.shape)
+    print("Y shape: ", Y.shape)
+    print("sample_weight shape: ", Y.shape)
+
+    history = model.fit(X, Y, validation_split=0.05,
+                        sample_weight=sample_weight,
+                        verbose=1, nb_epoch=2)
+    ground_truth = -np.log(0.5)
+    assert(np.abs(history.history['val_loss'][-1] - ground_truth) < 0.05)
+
 if __name__ == '__main__':
     pytest.main([__file__])

tests/keras/backend/test_backends.py

@@ -40,6 +40,8 @@ class TestBackend(object):
 
     def test_linear_operations(self):
         check_two_tensor_operation('dot', (4, 2), (2, 4))
+        check_two_tensor_operation('batch_dot', (4, 2, 3), (4, 5, 3),
+                                   axes=((2,), (2,)))
         check_single_tensor_operation('transpose', (4, 2))
 
     def test_shape_operations(self):
@@ -114,16 +116,20 @@ class TestBackend(object):
 
         check_single_tensor_operation('min', (4, 2))
         check_single_tensor_operation('min', (4, 2), axis=1, keepdims=True)
+        check_single_tensor_operation('min', (4, 2, 3), axis=[1, -1])
 
         check_single_tensor_operation('mean', (4, 2))
         check_single_tensor_operation('mean', (4, 2), axis=1, keepdims=True)
         check_single_tensor_operation('mean', (4, 2, 3), axis=-1, keepdims=True)
+        check_single_tensor_operation('mean', (4, 2, 3), axis=[1, -1])
 
         check_single_tensor_operation('std', (4, 2))
         check_single_tensor_operation('std', (4, 2), axis=1, keepdims=True)
+        check_single_tensor_operation('std', (4, 2, 3), axis=[1, -1])
 
         check_single_tensor_operation('prod', (4, 2))
         check_single_tensor_operation('prod', (4, 2), axis=1, keepdims=True)
+        check_single_tensor_operation('prod', (4, 2, 3), axis=[1, -1])
 
         # does not work yet, wait for bool <-> int casting in TF (coming soon)
         # check_single_tensor_operation('any', (4, 2))
@@ -141,6 +147,7 @@ class TestBackend(object):
         check_single_tensor_operation('exp', (4, 2))
         check_single_tensor_operation('log', (4, 2))
         check_single_tensor_operation('round', (4, 2))
+        check_single_tensor_operation('sign', (4, 2))
         check_single_tensor_operation('pow', (4, 2), a=3)
         check_single_tensor_operation('clip', (4, 2), min_value=0.4,
                                       max_value=0.6)
@@ -223,7 +230,7 @@ class TestBackend(object):
         last_output, outputs, new_states = KTH.rnn(th_rnn_step_fn, inputs,
                                                    initial_states,
                                                    go_backwards=False,
-                                                   masking=False)
+                                                   mask=None)
         th_last_output = KTH.eval(last_output)
         th_outputs = KTH.eval(outputs)
         assert len(new_states) == 1
@@ -235,7 +242,7 @@ class TestBackend(object):
         last_output, outputs, new_states = KTF.rnn(tf_rnn_step_fn, inputs,
                                                    initial_states,
                                                    go_backwards=False,
-                                                   masking=False)
+                                                   mask=None)
         tf_last_output = KTF.eval(last_output)
         tf_outputs = KTF.eval(outputs)
         assert len(new_states) == 1
@@ -269,7 +276,7 @@ class TestBackend(object):
         check_single_tensor_operation('tanh', (4, 2))
 
         # dropout
-        val = np.random.random((20, 20))
+        val = np.random.random((100, 100))
         xth = KTH.variable(val)
         xtf = KTF.variable(val)
         zth = KTH.eval(KTH.dropout(xth, level=0.2))
@@ -281,9 +288,11 @@ class TestBackend(object):
         check_two_tensor_operation('binary_crossentropy', (4, 2), (4, 2), from_logits=True)
         check_two_tensor_operation('categorical_crossentropy', (4, 2), (4, 2), from_logits=True)
         check_two_tensor_operation('binary_crossentropy', (4, 2), (4, 2), from_logits=False)
-
         check_two_tensor_operation('categorical_crossentropy', (4, 2), (4, 2), from_logits=False)
 
+        check_single_tensor_operation('l2_normalize', (4, 3), axis=-1)
+        check_single_tensor_operation('l2_normalize', (4, 3), axis=1)
+
     # def test_conv2d(self):
     #     '''conv2d works "properly" with Theano and TF but outputs different
     #     values in each case. Cause unclear (input / kernel shape format?)
@@ -324,28 +333,44 @@ class TestBackend(object):
     def test_random_normal(self):
         mean = 0.
         std = 1.
-        rand = KTF.get_value(KTF.random_normal((1000, 1000), mean=mean, std=std))
+        rand = KTF.eval(KTF.random_normal((1000, 1000), mean=mean, std=std))
         assert(rand.shape == (1000, 1000))
         assert(np.abs(np.mean(rand) - mean) < 0.01)
         assert(np.abs(np.std(rand) - std) < 0.01)
 
-        rand = KTF.get_value(KTF.random_normal((1000, 1000), mean=mean, std=std))
+        rand = KTH.eval(KTH.random_normal((1000, 1000), mean=mean, std=std))
         assert(rand.shape == (1000, 1000))
         assert(np.abs(np.mean(rand) - mean) < 0.01)
         assert(np.abs(np.std(rand) - std) < 0.01)
 
     def test_random_uniform(self):
-        mean = 0.
-        std = 1.
-        rand = KTF.get_value(KTF.random_normal((1000, 1000), mean=mean, std=std))
+        min = -1.
+        max = 1.
+        rand = KTF.eval(KTF.random_uniform((1000, 1000), min, max))
         assert(rand.shape == (1000, 1000))
-        assert(np.abs(np.mean(rand) - mean) < 0.01)
-        assert(np.abs(np.std(rand) - std) < 0.01)
+        assert(np.abs(np.mean(rand)) < 0.01)
+        assert(np.max(rand) <= max)
+        assert(np.min(rand) >= min)
 
-        rand = KTF.get_value(KTF.random_normal((1000, 1000), mean=mean, std=std))
+        rand = KTH.eval(KTH.random_uniform((1000, 1000), min, max))
         assert(rand.shape == (1000, 1000))
-        assert(np.abs(np.mean(rand) - mean) < 0.01)
-        assert(np.abs(np.std(rand) - std) < 0.01)
+        assert(np.abs(np.mean(rand)) < 0.01)
+        assert(np.max(rand) <= max)
+        assert(np.min(rand) >= min)
+
+    def test_random_binomial(self):
+        p = 0.5
+        rand = KTF.eval(KTF.random_binomial((1000, 1000), p))
+        assert(rand.shape == (1000, 1000))
+        assert(np.abs(np.mean(rand) - p) < 0.01)
+        assert(np.max(rand) == 1)
+        assert(np.min(rand) == 0)
+
+        rand = KTH.eval(KTH.random_binomial((1000, 1000), p))
+        assert(rand.shape == (1000, 1000))
+        assert(np.abs(np.mean(rand) - p) < 0.01)
+        assert(np.max(rand) == 1)
+        assert(np.min(rand) == 0)
 
 
 if __name__ == '__main__':

tests/keras/layers/test_advanced_activations.py

@@ -29,7 +29,7 @@ def test_leaky_relu():
         layer.input = K.variable(-inp)
         for train in [True, False]:
             outp = K.eval(layer.get_output(train))
-            assert_allclose(outp, -inp*alpha)
+            assert_allclose(outp, -inp * alpha)
 
         config = layer.get_config()
         assert config['alpha'] == alpha
@@ -53,7 +53,7 @@ def test_prelu():
 
         layer.input = K.variable(-inp)
         outp = K.eval(layer.get_output(train))
-        assert_allclose(-alphas*inp, outp)
+        assert_allclose(-alphas * inp, outp)
 
         # test with default weights
         layer = PReLU(input_shape=inp.flatten().shape)
@@ -65,7 +65,7 @@ def test_prelu():
         layer.input = K.variable(-inp)
         outp = K.eval(layer.get_output(train))
 
-        assert_allclose(0., alphas*outp)
+        assert_allclose(0., alphas * outp)
 
         layer.get_config()
 
@@ -84,7 +84,7 @@ def test_elu():
         layer.input = K.variable(-inp)
         for train in [True, False]:
             outp = K.eval(layer.get_output(train))
-            assert_allclose(outp, alpha*(np.exp(-inp)-1.), rtol=1e-3)
+            assert_allclose(outp, alpha * (np.exp(-inp) - 1.), rtol=1e-3)
 
         config = layer.get_config()
         assert config['alpha'] == alpha
@@ -107,7 +107,7 @@ def test_parametric_softplus():
             layer.build()
             for train in [True, False]:
                 outp = K.eval(layer.get_output(train))
-                assert_allclose(outp, alpha*np.log(1.+np.exp(beta*inp)),
+                assert_allclose(outp, alpha * np.log(1. + np.exp(beta * inp)),
                                 atol=1e-3)
 
             config = layer.get_config()
@@ -126,12 +126,12 @@ def test_thresholded_linear():
         layer.input = K.variable(inp)
         for train in [True, False]:
             outp = K.eval(layer.get_output(train))
-            assert_allclose(outp, inp*(np.abs(inp) >= theta))
+            assert_allclose(outp, inp * (np.abs(inp) >= theta))
 
         layer.input = K.variable(-inp)
         for train in [True, False]:
             outp = K.eval(layer.get_output(train))
-            assert_allclose(outp, -inp*(np.abs(inp) >= theta))
+            assert_allclose(outp, -inp * (np.abs(inp) >= theta))
 
         config = layer.get_config()
         assert config['theta'] == theta
@@ -148,16 +148,34 @@ def test_thresholded_relu():
         layer.input = K.variable(inp)
         for train in [True, False]:
             outp = K.eval(layer.get_output(train))
-            assert_allclose(outp, inp*(inp > theta))
+            assert_allclose(outp, inp * (inp > theta))
 
         layer.input = K.variable(-inp)
         for train in [True, False]:
             outp = K.eval(layer.get_output(train))
-            assert_allclose(outp, -inp*(-inp > theta))
+            assert_allclose(outp, -inp * (-inp > theta))
 
         config = layer.get_config()
         assert config['theta'] == theta
 
 
+def test_srelu():
+    from keras.layers.advanced_activations import SReLU
+    np.random.seed(1337)
+    inp = np.array([-2, -1., -0.5, 0., 0.5, 1., 2.])
+    out = np.array([-1.5, -1., -0.5, 0., 0.5, 1., 3.])
+    input_size = len(inp)
+    for train in [True, False]:
+        layer = SReLU(input_shape=inp.flatten().shape)
+        ones_proto = np.ones(input_size)
+        layer.set_weights([ones_proto * -1., ones_proto * 0.5,
+                           ones_proto * 2., ones_proto * 2.])
+        layer.input = K.variable(inp)
+        outp = K.eval(layer.get_output(train))
+        assert_allclose(out, outp)
+
+        layer.get_config()
+
+
 if __name__ == '__main__':
     pytest.main([__file__])

tests/keras/layers/test_call.py

@@ -6,7 +6,7 @@ from numpy.testing import assert_allclose
 
 from keras import backend as K
 from keras.layers.core import Dense
-from keras.models import Sequential
+from keras.models import Sequential, Graph
 
 
 def test_layer_call():
@@ -56,5 +56,157 @@ def test_sequential_call():
     assert_allclose(y1, y2)
 
 
+def test_graph_call():
+    """Test keras.models.Graph.__call__"""
+    nb_samples, input_dim, output_dim = 3, 10, 5
+    model = Graph()
+    model.add_input('input', input_shape=(input_dim, ))
+    model.add_node(Dense(output_dim=output_dim, input_dim=input_dim),
+                   input='input', name='output', create_output=True)
+
+    model.compile('sgd', {'output': 'mse'})
+
+    # test flat model
+    X = K.placeholder(ndim=2)
+    Y = model(X)
+    f = K.function([X], [Y])
+
+    x = np.ones((nb_samples, input_dim)).astype(K.floatx())
+    y1 = f([x])[0].astype(K.floatx())
+    y2 = model.predict({'input': x})['output']
+    # results of __call__ should match model.predict
+    assert_allclose(y1, y2)
+
+    # test nested Graph models
+    model2 = Graph()
+    model2.add_input('input', input_shape=(input_dim, ))
+    model2.add_node(model, input='input', name='output', create_output=True)
+    # need to turn off cache because we're reusing model
+    model2.cache_enabled = False
+    model2.compile('sgd', {'output': 'mse'})
+
+    Y2 = model2(X)
+    f = K.function([X], [Y2])
+
+    y1 = f([x])[0].astype(K.floatx())
+    y2 = model2.predict({'input': x})['output']
+    # results of __call__ should match model.predict
+    assert_allclose(y1, y2)
+
+
+def test_graph_multiple_in_out_call():
+    """Test keras.models.Graph.__call__ with multiple inputs"""
+    nb_samples, input_dim, output_dim = 3, 10, 5
+    model = Graph()
+    model.add_input('input1', input_shape=(input_dim, ))
+    model.add_input('input2', input_shape=(input_dim, ))
+    model.add_node(Dense(output_dim=output_dim, input_dim=input_dim),
+                   inputs=['input1', 'input2'], merge_mode='sum', name='output', create_output=True)
+
+    model.compile('sgd', {'output': 'mse'})
+
+    # test flat model
+    X1 = K.placeholder(ndim=2)
+    X2 = K.placeholder(ndim=2)
+    Y = model({'input1': X1, 'input2': X2})['output']
+    f = K.function([X1, X2], [Y])
+
+    x1 = np.ones((nb_samples, input_dim)).astype(K.floatx())
+    x2 = np.ones((nb_samples, input_dim)).astype(K.floatx()) * -2
+    y1 = f([x1, x2])[0].astype(K.floatx())
+    y2 = model.predict({'input1': x1, 'input2': x2})['output']
+    # results of __call__ should match model.predict
+    assert_allclose(y1, y2)
+
+    # test with single input, multiple outputs
+    model2 = Graph()
+    model2.add_input('input', input_shape=(input_dim, ))
+    model2.add_node(Dense(output_dim=output_dim, input_dim=input_dim),
+                    input='input', name='output1', create_output=True)
+    model2.add_node(Dense(output_dim=output_dim, input_dim=input_dim),
+                    input='input', name='output2', create_output=True)
+
+    model2.compile('sgd', {'output1': 'mse', 'output2': 'mse'})
+
+    # test flat model
+    X = K.placeholder(ndim=2)
+    Y = model2(X)
+    f = K.function([X], [Y['output1'], Y['output2']])
+
+    x = np.ones((nb_samples, input_dim)).astype(K.floatx())
+    out = f([x])
+    y1a = out[0].astype(K.floatx())
+    y1b = out[1].astype(K.floatx())
+    y2 = model2.predict({'input': x})
+    # results of __call__ should match model.predict
+    assert_allclose(y1a, y2['output1'])
+    assert_allclose(y1b, y2['output2'])
+
+    # test with multiple inputs, multiple outputs
+    model3 = Graph()
+    model3.add_input('input1', input_shape=(input_dim, ))
+    model3.add_input('input2', input_shape=(input_dim, ))
+    model3.add_shared_node(Dense(output_dim=output_dim, input_dim=input_dim),
+                           inputs=['input1', 'input2'], name='output',
+                           outputs=['output1', 'output2'], create_output=True)
+    model3.compile('sgd', {'output1': 'mse', 'output2': 'mse'})
+
+    # test flat model
+    Y = model3({'input1': X1, 'input2': X2})
+    f = K.function([X1, X2], [Y['output1'], Y['output2']])
+
+    x1 = np.ones((nb_samples, input_dim)).astype(K.floatx())
+    x2 = np.ones((nb_samples, input_dim)).astype(K.floatx()) * -2
+    out = f([x1, x2])
+    y1a = out[0].astype(K.floatx())
+    y1b = out[1].astype(K.floatx())
+    y2 = model3.predict({'input1': x1, 'input2': x2})
+    # results of __call__ should match model.predict
+    assert_allclose(y1a, y2['output1'])
+    assert_allclose(y1b, y2['output2'])
+
+
+def test_nested_call():
+    """Test nested Sequential and Graph models"""
+    nb_samples, input_dim, output_dim = 3, 10, 5
+    X = K.placeholder(ndim=2)
+    x = np.ones((nb_samples, input_dim)).astype(K.floatx())
+
+    # test Graph model nested inside Sequential model
+    model = Graph()
+    model.add_input('input', input_shape=(input_dim, ))
+    model.add_node(Dense(output_dim=output_dim, input_dim=input_dim),
+                   input='input', name='output', create_output=True)
+
+    model2 = Sequential()
+    model2.add(model)
+    model2.compile('sgd', 'mse')
+
+    Y2 = model2(X)
+    f = K.function([X], [Y2])
+
+    y1 = f([x])[0].astype(K.floatx())
+    y2 = model2.predict(x)
+    # results of __call__ should match model.predict
+    assert_allclose(y1, y2)
+
+    # test Sequential model inside Graph model
+    model3 = Sequential()
+    model3.add(Dense(output_dim=output_dim, input_dim=input_dim))
+
+    model4 = Graph()
+    model4.add_input('input', input_shape=(input_dim, ))
+    model4.add_node(model3, input='input', name='output', create_output=True)
+    model4.compile('sgd', {'output': 'mse'})
+
+    Y2 = model4(X)
+    f = K.function([X], [Y2])
+
+    y1 = f([x])[0].astype(K.floatx())
+    y2 = model4.predict({'input': x})['output']
+    # results of __call__ should match model.predict
+    assert_allclose(y1, y2)
+
+
 if __name__ == '__main__':
     pytest.main([__file__])

tests/keras/layers/test_convolutional.py

@@ -113,6 +113,38 @@ def test_convolution_2d():
                             layer.get_config()
 
 
+def test_convolution_2d_dim_ordering():
+    nb_filter = 4
+    nb_row = 3
+    nb_col = 2
+    stack_size = 3
+
+    np.random.seed(1337)
+    weights = [np.random.random((nb_filter, stack_size, nb_row, nb_col)),
+               np.random.random(nb_filter)]
+    input = np.random.random((1, stack_size, 10, 10))
+
+    layer = convolutional.Convolution2D(
+        nb_filter, nb_row, nb_col,
+        weights=weights,
+        input_shape=input.shape[1:],
+        dim_ordering='th')
+    layer.input = K.variable(input)
+    out_th = K.eval(layer.get_output(False))
+
+    input = np.transpose(input, (0, 2, 3, 1))
+    weights[0] = np.transpose(weights[0], (2, 3, 1, 0))
+    layer = convolutional.Convolution2D(
+        nb_filter, nb_row, nb_col,
+        weights=weights,
+        input_shape=input.shape[1:],
+        dim_ordering='tf')
+    layer.input = K.variable(input)
+    out_tf = K.eval(layer.get_output(False))
+
+    assert_allclose(out_tf, np.transpose(out_th, (0, 2, 3, 1)), atol=1e-05)
+
+
 def test_maxpooling_2d():
     nb_samples = 9
     stack_size = 7
@@ -131,16 +163,128 @@ def test_maxpooling_2d():
         layer.get_config()
 
 
+def test_maxpooling_2d_dim_ordering():
+    stack_size = 3
+
+    input = np.random.random((1, stack_size, 10, 10))
+
+    layer = convolutional.MaxPooling2D(
+        (2, 2),
+        input_shape=input.shape[1:],
+        dim_ordering='th')
+    layer.input = K.variable(input)
+    out_th = K.eval(layer.get_output(False))
+
+    input = np.transpose(input, (0, 2, 3, 1))
+    layer = convolutional.MaxPooling2D(
+        (2, 2),
+        input_shape=input.shape[1:],
+        dim_ordering='tf')
+    layer.input = K.variable(input)
+    out_tf = K.eval(layer.get_output(False))
+
+    assert_allclose(out_tf, np.transpose(out_th, (0, 2, 3, 1)), atol=1e-05)
+
+
 def test_averagepooling_2d():
     nb_samples = 9
     stack_size = 7
     input_nb_row = 11
     input_nb_col = 12
-    pool_size = (3, 3)
 
     input = np.ones((nb_samples, stack_size, input_nb_row, input_nb_col))
-    for strides in [(1, 1), (2, 2)]:
-        layer = convolutional.AveragePooling2D(strides=strides,
+    for border_mode in ['valid', 'same']:
+        for pool_size in [(2, 2), (3, 3), (4, 4), (5, 5)]:
+            for strides in [(1, 1), (2, 2)]:
+                layer = convolutional.AveragePooling2D(strides=strides,
+                                                       border_mode=border_mode,
+                                                       pool_size=pool_size)
+                layer.input = K.variable(input)
+                for train in [True, False]:
+                    out = K.eval(layer.get_output(train))
+                    if border_mode == 'same' and strides == (1, 1):
+                        assert input.shape == out.shape
+                layer.get_config()
+
+
+@pytest.mark.skipif(K._BACKEND != 'theano', reason="Requires Theano backend")
+def test_convolution_3d():
+    nb_samples = 8
+    nb_filter = 9
+    stack_size = 7
+    len_conv_dim1 = 2
+    len_conv_dim2 = 10
+    len_conv_dim3 = 6
+
+    input_len_dim1 = 10
+    input_len_dim2 = 11
+    input_len_dim3 = 12
+
+    weights_in = [np.ones((nb_filter, stack_size, len_conv_dim1, len_conv_dim2, len_conv_dim3)),
+                  np.ones(nb_filter)]
+
+    input = np.ones((nb_samples, stack_size, input_len_dim1,
+                     input_len_dim2, input_len_dim3))
+    for weight in [None, weights_in]:
+        for border_mode in ['same', 'valid']:
+            for subsample in [(1, 1, 1), (2, 2, 2)]:
+                if border_mode == 'same' and subsample != (1, 1, 1):
+                    continue
+                for W_regularizer in [None, 'l2']:
+                    for b_regularizer in [None, 'l2']:
+                        for act_regularizer in [None, 'l2']:
+                            layer = convolutional.Convolution3D(
+                                nb_filter, len_conv_dim1, len_conv_dim2, len_conv_dim3,
+                                weights=weight,
+                                border_mode=border_mode,
+                                W_regularizer=W_regularizer,
+                                b_regularizer=b_regularizer,
+                                activity_regularizer=act_regularizer,
+                                subsample=subsample,
+                                input_shape=(stack_size, None, None, None))
+
+                            layer.input = K.variable(input)
+                            for train in [True, False]:
+                                out = K.eval(layer.get_output(train))
+                                if border_mode == 'same' and subsample == (1, 1, 1):
+                                    assert out.shape[2:] == input.shape[2:]
+                            layer.get_config()
+
+
+@pytest.mark.skipif(K._BACKEND != 'theano', reason="Requires Theano backend")
+def test_maxpooling_3d():
+    nb_samples = 9
+    stack_size = 7
+    input_len_dim1 = 10
+    input_len_dim2 = 11
+    input_len_dim3 = 12
+    pool_size = (3, 3, 3)
+
+    input = np.ones((nb_samples, stack_size, input_len_dim1,
+                     input_len_dim2, input_len_dim3))
+    for strides in [(1, 1, 1), (2, 2, 2)]:
+        layer = convolutional.MaxPooling3D(strides=strides,
+                                           border_mode='valid',
+                                           pool_size=pool_size)
+        layer.input = K.variable(input)
+        for train in [True, False]:
+            K.eval(layer.get_output(train))
+        layer.get_config()
+
+
+@pytest.mark.skipif(K._BACKEND != 'theano', reason="Requires Theano backend")
+def test_averagepooling_3d():
+    nb_samples = 9
+    stack_size = 7
+    input_len_dim1 = 10
+    input_len_dim2 = 11
+    input_len_dim3 = 12
+    pool_size = (3, 3, 3)
+
+    input = np.ones((nb_samples, stack_size, input_len_dim1,
+                     input_len_dim2, input_len_dim3))
+    for strides in [(1, 1, 1), (2, 2, 2)]:
+        layer = convolutional.AveragePooling3D(strides=strides,
                                                border_mode='valid',
                                                pool_size=pool_size)
         layer.input = K.variable(input)
@@ -167,6 +311,28 @@ def test_zero_padding_2d():
     layer.get_config()
 
 
+@pytest.mark.skipif(K._BACKEND != 'theano', reason="Requires Theano backend")
+def test_zero_padding_3d():
+    nb_samples = 9
+    stack_size = 7
+    input_len_dim1 = 10
+    input_len_dim2 = 11
+    input_len_dim3 = 12
+
+    input = np.ones((nb_samples, stack_size, input_len_dim1,
+                     input_len_dim2, input_len_dim3))
+    layer = convolutional.ZeroPadding3D(padding=(2, 2, 2))
+    layer.input = K.variable(input)
+    for train in [True, False]:
+        out = K.eval(layer.get_output(train))
+        for offset in [0, 1, -1, -2]:
+            assert_allclose(out[:, :, offset, :, :], 0.)
+            assert_allclose(out[:, :, :, offset, :], 0.)
+            assert_allclose(out[:, :, :, :, offset], 0.)
+        assert_allclose(out[:, :, 2:-2, 2:-2, 2:-2], 1.)
+    layer.get_config()
+
+
 def test_upsampling_1d():
     nb_samples = 9
     nb_steps = 7
@@ -188,17 +354,90 @@ def test_upsampling_2d():
     input_nb_row = 11
     input_nb_col = 12
 
-    input = np.ones((nb_samples, stack_size, input_nb_row, input_nb_col))
+    for dim_ordering in ['th', 'tf']:
+        if dim_ordering == 'th':
+            input = np.random.rand(nb_samples, stack_size, input_nb_row,
+                                   input_nb_col)
+        else:  # tf
+            input = np.random.rand(nb_samples, input_nb_row, input_nb_col,
+                                   stack_size)
 
-    for length_row in [2, 3, 9]:
-        for length_col in [2, 3, 9]:
-            layer = convolutional.UpSampling2D(size=(length_row, length_col))
-            layer.input = K.variable(input)
-            for train in [True, False]:
-                out = K.eval(layer.get_output(train))
-                assert out.shape[2] == length_row * input_nb_row
-                assert out.shape[3] == length_col * input_nb_col
-        layer.get_config()
+        for length_row in [2, 3, 9]:
+            for length_col in [2, 3, 9]:
+                layer = convolutional.UpSampling2D(
+                    size=(length_row, length_col),
+                    input_shape=input.shape[1:],
+                    dim_ordering=dim_ordering)
+                layer.input = K.variable(input)
+                for train in [True, False]:
+                    out = K.eval(layer.get_output(train))
+                    if dim_ordering == 'th':
+                        assert out.shape[2] == length_row * input_nb_row
+                        assert out.shape[3] == length_col * input_nb_col
+                    else:  # tf
+                        assert out.shape[1] == length_row * input_nb_row
+                        assert out.shape[2] == length_col * input_nb_col
+
+                    # compare with numpy
+                    if dim_ordering == 'th':
+                        expected_out = np.repeat(input, length_row, axis=2)
+                        expected_out = np.repeat(expected_out, length_col, axis=3)
+                    else:  # tf
+                        expected_out = np.repeat(input, length_row, axis=1)
+                        expected_out = np.repeat(expected_out, length_col, axis=2)
+
+                    assert_allclose(out, expected_out)
+
+                layer.get_config()
+
+
+@pytest.mark.skipif(K._BACKEND != 'theano', reason="Requires Theano backend")
+def test_upsampling_3d():
+    nb_samples = 9
+    stack_size = 7
+    input_len_dim1 = 10
+    input_len_dim2 = 11
+    input_len_dim3 = 12
+
+    for dim_ordering in ['th', 'tf']:
+        if dim_ordering == 'th':
+            input = np.random.rand(nb_samples, stack_size, input_len_dim1, input_len_dim2,
+                                   input_len_dim3)
+        else:  # tf
+            input = np.random.rand(nb_samples, input_len_dim1, input_len_dim2, input_len_dim3,
+                                   stack_size)
+        for length_dim1 in [2, 3, 9]:
+            for length_dim2 in [2, 3, 9]:
+                for length_dim3 in [2, 3, 9]:
+                    layer = convolutional.UpSampling3D(
+                        size=(length_dim1, length_dim2, length_dim3),
+                        input_shape=input.shape[1:],
+                        dim_ordering=dim_ordering)
+                    layer.input = K.variable(input)
+                    for train in [True, False]:
+                        out = K.eval(layer.get_output(train))
+                        if dim_ordering == 'th':
+                            assert out.shape[2] == length_dim1 * input_len_dim1
+                            assert out.shape[3] == length_dim2 * input_len_dim2
+                            assert out.shape[4] == length_dim3 * input_len_dim3
+                        else:  # tf
+                            assert out.shape[1] == length_dim1 * input_len_dim1
+                            assert out.shape[2] == length_dim2 * input_len_dim2
+                            assert out.shape[3] == length_dim3 * input_len_dim3
+
+                        # compare with numpy
+                        if dim_ordering == 'th':
+                            expected_out = np.repeat(input, length_dim1, axis=2)
+                            expected_out = np.repeat(expected_out, length_dim2, axis=3)
+                            expected_out = np.repeat(expected_out, length_dim3, axis=4)
+                        else:  # tf
+                            expected_out = np.repeat(input, length_dim1, axis=1)
+                            expected_out = np.repeat(expected_out, length_dim2, axis=2)
+                            expected_out = np.repeat(expected_out, length_dim3, axis=3)
+
+                        assert_allclose(out, expected_out)
+
+                    layer.get_config()
 
 
 if __name__ == '__main__':

tests/keras/layers/test_core.py

@@ -5,6 +5,7 @@ from numpy.testing import assert_allclose
 
 from keras import backend as K
 from keras.layers import core
+from keras.layers import containers
 
 
 def test_input_output():
@@ -114,15 +115,41 @@ def test_autoencoder():
     _runner(layer)
 
 
-def test_autoencoder_second_layer():
-    # regression test for issue #1275
-    encoder = core.Dense(input_dim=10, output_dim=2)
-    decoder = core.Dense(input_dim=2, output_dim=10)
+def test_autoencoder_advanced():
+    encoder = containers.Sequential([core.Dense(5, input_shape=(10,))])
+    decoder = containers.Sequential([core.Dense(10, input_shape=(5,))])
+    X_train = np.random.random((100, 10))
+    X_test = np.random.random((100, 10))
+
     model = Sequential()
-    model.add(core.Dense(input_dim=20, output_dim=10))
-    model.add(core.AutoEncoder(encoder=encoder, decoder=decoder,
-                               output_reconstruction=False))
-    model.compile(loss='mse', optimizer='sgd')
+    model.add(core.Dense(output_dim=10, input_dim=10))
+    autoencoder = core.AutoEncoder(encoder=encoder, decoder=decoder,
+                                   output_reconstruction=True)
+    model.add(autoencoder)
+
+    # training the autoencoder:
+    model.compile(optimizer='sgd', loss='mse')
+    assert autoencoder.output_reconstruction
+
+    model.fit(X_train, X_train, nb_epoch=1, batch_size=32)
+
+    # predicting compressed representations of inputs:
+    autoencoder.output_reconstruction = False  # the autoencoder has to be recompiled after modifying this property
+    assert not autoencoder.output_reconstruction
+    model.compile(optimizer='sgd', loss='mse')
+    representations = model.predict(X_test)
+    assert representations.shape == (100, 5)
+
+    # the model is still trainable, although it now expects compressed representations as targets:
+    model.fit(X_test, representations, nb_epoch=1, batch_size=32)
+
+    # to keep training against the original inputs, just switch back output_reconstruction to True:
+    autoencoder.output_reconstruction = True
+    model.compile(optimizer='sgd', loss='mse')
+    model.fit(X_train, X_train, nb_epoch=1)
+
+    reconstructions = model.predict(X_test)
+    assert reconstructions.shape == (100, 10)
 
 
 def test_maxout_dense():
@@ -130,15 +157,28 @@ def test_maxout_dense():
     _runner(layer)
 
 
-@pytest.mark.skipif(K._BACKEND == 'tensorflow',
-                    reason='currently not working with TensorFlow')
+def test_naming():
+    layer = core.Dense(2, input_dim=2)
+    assert layer.name == 'dense'
+
+    model = Sequential()
+    model.add(core.Dense(2, input_dim=2, name='my_dense'))
+    model.add(core.Dense(2, name='my_dense'))
+
+    assert model.layers[0].name == 'my_dense'
+    assert model.layers[1].name == 'my_dense'
+
+    model.compile(optimizer='rmsprop', loss='mse')
+    model.train_on_batch(np.random.random((2, 2)), np.random.random((2, 2)))
+
+
 def test_sequences():
     '''Test masking sequences with zeroes as padding'''
     # integer inputs, one per timestep, like embeddings
-    layer = core.Masking()
-    func = K.function([layer.input], [layer.get_output_mask()])
+    layer = core.Masking(input_shape=(4, 1))
+    func = K.function([layer.get_input(True)], [layer.get_output_mask()])
     input_data = np.array([[[1], [2], [3], [0]],
-                          [[0], [4], [5], [0]]], dtype=np.int32)
+                           [[0], [4], [5], [0]]], dtype=np.int32)
 
     # This is the expected output mask, one dimension less
     expected = np.array([[1, 1, 1, 0], [0, 1, 1, 0]])
@@ -148,33 +188,29 @@ def test_sequences():
     assert np.all(output == expected), 'Output not as expected'
 
 
-@pytest.mark.skipif(K._BACKEND == 'tensorflow',
-                    reason='currently not working with TensorFlow')
 def test_non_zero():
     '''Test masking with non-zero mask value'''
-    layer = core.Masking(5)
+    layer = core.Masking(5, input_shape=(4, 2))
     func = K.function([layer.input], [layer.get_output_mask()])
     input_data = np.array([[[1, 1], [2, 1], [3, 1], [5, 5]],
-                          [[1, 5], [5, 0], [0, 0], [0, 0]]],
+                           [[1, 5], [5, 0], [0, 0], [0, 0]]],
                           dtype=np.int32)
     output = func([input_data])[0]
     expected = np.array([[1, 1, 1, 0], [1, 1, 1, 1]])
     assert np.all(output == expected), 'Output not as expected'
 
 
-@pytest.mark.skipif(K._BACKEND == 'tensorflow',
-                    reason='currently not working with TensorFlow')
 def test_non_zero_output():
     '''Test output of masking layer with non-zero mask value'''
-    layer = core.Masking(5)
+    layer = core.Masking(5, input_shape=(4, 2))
     func = K.function([layer.input], [layer.get_output()])
 
     input_data = np.array([[[1, 1], [2, 1], [3, 1], [5, 5]],
-                          [[1, 5], [5, 0], [0, 0], [0, 0]]],
+                           [[1, 5], [5, 0], [0, 0], [0, 0]]],
                           dtype=np.int32)
     output = func([input_data])[0]
     expected = np.array([[[1, 1], [2, 1], [3, 1], [0, 0]],
-                        [[1, 5], [5, 0], [0, 0], [0, 0]]])
+                         [[1, 5], [5, 0], [0, 0], [0, 0]]])
     assert np.all(output == expected), 'Output not as expected'
 
 
@@ -192,6 +228,7 @@ def _runner(layer):
     layer.trainable = True
     layer.trainable = False
 
+
 def test_siamese_all():
     right_input_layer = core.Dense(7, input_dim=3)
     left_input_layer = core.Dense(7, input_dim=3)
@@ -202,6 +239,7 @@ def test_siamese_all():
         siamese_layer.output_shape
         siamese_layer.get_output()
 
+
 @pytest.mark.skipif(K._BACKEND == 'tensorflow',
                     reason='currently not working with TensorFlow')
 def test_siamese_theano_only():

tests/keras/layers/test_embeddings.py

@@ -23,7 +23,7 @@ def test_unitnorm_constraint():
     lookup.compile(loss='binary_crossentropy', optimizer='sgd',
                    class_mode='binary')
     lookup.train_on_batch(X1, np.array([[1], [0]], dtype='int32'))
-    norm = np.linalg.norm(K.get_value(lookup.params[0]), axis=1)
+    norm = np.linalg.norm(K.get_value(lookup.trainable_weights[0]), axis=0)
     assert_allclose(norm, np.ones_like(norm).astype('float32'), rtol=1e-05)
 
 

tests/keras/layers/test_normalization.py

@@ -17,7 +17,7 @@ input_shapes = [np.ones((10, 10)), np.ones((10, 10, 10))]
 def test_batchnorm_mode_0():
     np.random.seed(1337)
     model = Sequential()
-    norm_m0 = normalization.BatchNormalization(input_shape=(10,))
+    norm_m0 = normalization.BatchNormalization(mode=0, input_shape=(10,))
     model.add(norm_m0)
     model.compile(loss='mse', optimizer='sgd')
 
@@ -31,6 +31,22 @@ def test_batchnorm_mode_0():
     assert_allclose(K.eval(K.std(out)), 1.0, atol=1e-1)
 
 
+def test_batchnorm_mode_0_convnet():
+    model = Sequential()
+    norm_m0 = normalization.BatchNormalization(mode=0, axis=1, input_shape=(3, 4, 4))
+    model.add(norm_m0)
+    model.compile(loss='mse', optimizer='sgd')
+
+    # centered on 5.0, variance 10.0
+    X = np.random.normal(loc=5.0, scale=10.0, size=(1000, 3, 4, 4))
+    model.fit(X, X, nb_epoch=5, verbose=0)
+    norm_m0.input = K.variable(X)
+    out = (norm_m0.get_output(train=True) - K.reshape(norm_m0.beta, (1, 3, 1, 1))) / K.reshape(norm_m0.gamma, (1, 3, 1, 1))
+
+    assert_allclose(K.eval(K.mean(out, axis=(0, 2, 3))), 0.0, atol=1e-1)
+    assert_allclose(K.eval(K.std(out, axis=(0, 2, 3))), 1.0, atol=1e-1)
+
+
 def test_batchnorm_mode_1():
     np.random.seed(1337)
     norm_m1 = normalization.BatchNormalization(input_shape=(10,), mode=1)
@@ -50,13 +66,15 @@ def test_batchnorm_shapes():
     Test batch normalization with various input shapes
     """
     for inp in input_shapes:
-        norm_m0 = normalization.BatchNormalization(input_shape=inp.shape, mode=0)
+        norm_m0 = normalization.BatchNormalization(batch_input_shape=inp.shape, mode=0)
         norm_m0.input = K.variable(inp)
-        out = (norm_m0.get_output(train=True) - norm_m0.beta) / norm_m0.gamma
+        out = norm_m0.get_output(train=True)
+        K.eval(out)
 
-        norm_m1 = normalization.BatchNormalization(input_shape=inp.shape, mode=1)
+        norm_m1 = normalization.BatchNormalization(batch_input_shape=inp.shape, mode=1)
         norm_m1.input = K.variable(inp)
-        out = (norm_m1.get_output(train=True) - norm_m1.beta) / norm_m1.gamma
+        out = norm_m1.get_output(train=True)
+        K.eval(out)
 
 
 def test_batchnorm_weight_init():
@@ -83,12 +101,7 @@ def test_batchnorm_weight_init():
 def test_batchnorm_config():
     norm = normalization.BatchNormalization(input_shape=(10, 10), mode=1,
                                             epsilon=0.1, momentum=0.9)
-    conf = norm.get_config()
-    del conf['cache_enabled']
-    conf_target = {"input_shape": (10, 10),
-                   "name": normalization.BatchNormalization.__name__,
-                   "epsilon": 0.1, "mode": 1, "momentum": 0.9}
-    assert(conf == conf_target)
+    norm.get_config()
 
 
 def test_batchnorm_save_weights():

tests/keras/layers/test_recurrent.py

@@ -2,11 +2,16 @@ import pytest
 import numpy as np
 from numpy.testing import assert_allclose
 
-from keras.layers import recurrent
-from keras import backend as K
+from keras.layers import recurrent, embeddings
 from keras.models import Sequential
+from keras.layers.core import Masking
+from keras import regularizers
 
-nb_samples, timesteps, input_dim, output_dim = 3, 3, 10, 5
+from keras import backend as K
+from keras.models import Sequential, model_from_json
+
+nb_samples, timesteps, embedding_dim, output_dim = 3, 5, 10, 5
+embedding_num = 12
 
 
 def _runner(layer_class):
@@ -16,8 +21,26 @@ def _runner(layer_class):
     """
     for ret_seq in [True, False]:
         layer = layer_class(output_dim, return_sequences=ret_seq,
-                            weights=None, input_shape=(timesteps, input_dim))
-        layer.input = K.variable(np.ones((nb_samples, timesteps, input_dim)))
+                            weights=None, input_shape=(timesteps, embedding_dim))
+        layer.input = K.variable(np.ones((nb_samples, timesteps, embedding_dim)))
+        layer.get_config()
+
+        for train in [True, False]:
+            out = K.eval(layer.get_output(train))
+            # Make sure the output has the desired shape
+            if ret_seq:
+                assert(out.shape == (nb_samples, timesteps, output_dim))
+            else:
+                assert(out.shape == (nb_samples, output_dim))
+
+            mask = layer.get_output_mask(train)
+
+    # check dropout
+    for ret_seq in [True, False]:
+        layer = layer_class(output_dim, return_sequences=ret_seq, weights=None, 
+                            batch_input_shape=(nb_samples, timesteps, embedding_dim),
+                            dropout_W=0.5, dropout_U=0.5)
+        layer.input = K.variable(np.ones((nb_samples, timesteps, embedding_dim)))
         layer.get_config()
 
         for train in [True, False]:
@@ -31,20 +54,23 @@ def _runner(layer_class):
             mask = layer.get_output_mask(train)
 
     # check statefulness
+    model = Sequential()
+    model.add(embeddings.Embedding(embedding_num, embedding_dim,
+                                   mask_zero=True,
+                                   input_length=timesteps,
+                                   batch_input_shape=(nb_samples, timesteps)))
     layer = layer_class(output_dim, return_sequences=False,
                         stateful=True,
-                        weights=None,
-                        batch_input_shape=(nb_samples, timesteps, input_dim))
-    model = Sequential()
+                        weights=None)
     model.add(layer)
     model.compile(optimizer='sgd', loss='mse')
-    out1 = model.predict(np.ones((nb_samples, timesteps, input_dim)))
+    out1 = model.predict(np.ones((nb_samples, timesteps)))
     assert(out1.shape == (nb_samples, output_dim))
 
     # train once so that the states change
-    model.train_on_batch(np.ones((nb_samples, timesteps, input_dim)),
+    model.train_on_batch(np.ones((nb_samples, timesteps)),
                          np.ones((nb_samples, output_dim)))
-    out2 = model.predict(np.ones((nb_samples, timesteps, input_dim)))
+    out2 = model.predict(np.ones((nb_samples, timesteps)))
 
     # if the state is not reset, output should be different
     assert(out1.max() != out2.max())
@@ -52,18 +78,45 @@ def _runner(layer_class):
     # check that output changes after states are reset
     # (even though the model itself didn't change)
     layer.reset_states()
-    out3 = model.predict(np.ones((nb_samples, timesteps, input_dim)))
+    out3 = model.predict(np.ones((nb_samples, timesteps)))
     assert(out2.max() != out3.max())
 
     # check that container-level reset_states() works
     model.reset_states()
-    out4 = model.predict(np.ones((nb_samples, timesteps, input_dim)))
+    out4 = model.predict(np.ones((nb_samples, timesteps)))
     assert_allclose(out3, out4, atol=1e-5)
 
     # check that the call to `predict` updated the states
-    out5 = model.predict(np.ones((nb_samples, timesteps, input_dim)))
+    out5 = model.predict(np.ones((nb_samples, timesteps)))
     assert(out4.max() != out5.max())
 
+    # Check masking
+    layer.reset_states()
+
+    left_padded_input = np.ones((nb_samples, timesteps))
+    left_padded_input[0, :1] = 0
+    left_padded_input[1, :2] = 0
+    left_padded_input[2, :3] = 0
+    out6 = model.predict(left_padded_input)
+
+    layer.reset_states()
+
+    right_padded_input = np.ones((nb_samples, timesteps))
+    right_padded_input[0, -1:] = 0
+    right_padded_input[1, -2:] = 0
+    right_padded_input[2, -3:] = 0
+    out7 = model.predict(right_padded_input)
+
+    assert_allclose(out7, out6, atol=1e-5)
+
+    # check regularizers
+    layer = layer_class(output_dim, return_sequences=ret_seq, weights=None,
+                        batch_input_shape=(nb_samples, timesteps, embedding_dim),
+                        W_regularizer=regularizers.WeightRegularizer(l1=0.01),
+                        U_regularizer=regularizers.WeightRegularizer(l1=0.01),
+                        b_regularizer='l2')
+    layer.input = K.variable(np.ones((nb_samples, timesteps, embedding_dim)))
+    out = K.eval(layer.get_output(train=True))
 
 
 def test_SimpleRNN():
@@ -78,5 +131,31 @@ def test_LSTM():
     _runner(recurrent.LSTM)
 
 
+def test_batch_input_shape_serialization():
+    model = Sequential()
+    model.add(embeddings.Embedding(2, 2,
+                                   mask_zero=True,
+                                   input_length=2,
+                                   batch_input_shape=(2, 2)))
+    json_data = model.to_json()
+    reconstructed_model = model_from_json(json_data)
+    assert(reconstructed_model.input_shape == (2, 2))
+
+
+def test_masking_layer():
+    ''' This test based on a previously failing issue here:
+    https://github.com/fchollet/keras/issues/1567
+
+    '''
+    model = Sequential()
+    model.add(Masking(input_shape=(3, 4)))
+    model.add(recurrent.LSTM(output_dim=5, return_sequences=True))
+    model.compile(loss='categorical_crossentropy', optimizer='adam')
+    I = np.random.random((6, 3, 4))
+    V = np.abs(np.random.random((6, 3, 5)))
+    V /= V.sum(axis=-1, keepdims=True)
+    model.fit(I, V, nb_epoch=1, batch_size=100, verbose=1)
+
+
 if __name__ == '__main__':
     pytest.main([__file__])

tests/keras/layers/test_wrappers.py

@@ -0,0 +1,64 @@
+import pytest
+import numpy as np
+from numpy.testing import assert_allclose
+
+from keras.layers import wrappers
+from keras.layers import core, convolutional
+from keras.models import Sequential, model_from_json
+
+
+def test_TimeDistributed():
+    # first, test with Dense layer
+    model = Sequential()
+    model.add(wrappers.TimeDistributed(core.Dense(2), input_shape=(3, 4)))
+    model.add(core.Activation('relu'))
+    model.compile(optimizer='rmsprop', loss='mse')
+    model.fit(np.random.random((10, 3, 4)), np.random.random((10, 3, 2)), nb_epoch=1, batch_size=10)
+
+    # test config
+    model.get_config()
+
+    # compare to TimeDistributedDense
+    test_input = np.random.random((1, 3, 4))
+    test_output = model.predict(test_input)
+    weights = model.layers[0].get_weights()
+
+    reference = Sequential()
+    reference.add(core.TimeDistributedDense(2, input_shape=(3, 4), weights=weights))
+    reference.add(core.Activation('relu'))
+    reference.compile(optimizer='rmsprop', loss='mse')
+
+    reference_output = reference.predict(test_input)
+    assert_allclose(test_output, reference_output, atol=1e-05)
+
+    # test when specifying a batch_input_shape
+    reference = Sequential()
+    reference.add(core.TimeDistributedDense(2, batch_input_shape=(1, 3, 4), weights=weights))
+    reference.add(core.Activation('relu'))
+    reference.compile(optimizer='rmsprop', loss='mse')
+
+    reference_output = reference.predict(test_input)
+    assert_allclose(test_output, reference_output, atol=1e-05)
+
+    # test with Convolution2D
+    model = Sequential()
+    model.add(wrappers.TimeDistributed(convolutional.Convolution2D(5, 2, 2, border_mode='same'), input_shape=(2, 3, 4, 4)))
+    model.add(core.Activation('relu'))
+    model.compile(optimizer='rmsprop', loss='mse')
+    model.train_on_batch(np.random.random((1, 2, 3, 4, 4)), np.random.random((1, 2, 5, 4, 4)))
+
+    model = model_from_json(model.to_json())
+    model.summary()
+
+    # test stacked layers
+    model = Sequential()
+    model.add(wrappers.TimeDistributed(core.Dense(2), input_shape=(3, 4)))
+    model.add(wrappers.TimeDistributed(core.Dense(3)))
+    model.add(core.Activation('relu'))
+    model.compile(optimizer='rmsprop', loss='mse')
+
+    model.fit(np.random.random((10, 3, 4)), np.random.random((10, 3, 3)), nb_epoch=1, batch_size=10)
+
+
+if __name__ == '__main__':
+    pytest.main([__file__])

tests/keras/preprocessing/test_image.py

@@ -5,36 +5,35 @@ import numpy as np
 import os
 import shutil
 
-def setup_function(func):
-    np.random.seed(1337)
 
+def setup_function(func):
     os.mkdir('test_images')
     os.mkdir('test_images/rgb')
     os.mkdir('test_images/gsc')
 
     img_w = img_h = 20
     for n in range(8):
-        bias = np.random.rand(img_w,img_h,1)*64
-        variance = np.random.rand(img_w,img_h,1)*(255-64)
-        imarray = np.random.rand(img_w,img_h,3) * variance + bias
+        bias = np.random.rand(img_w, img_h, 1) * 64
+        variance = np.random.rand(img_w, img_h, 1) * (255-64)
+        imarray = np.random.rand(img_w, img_h, 3) * variance + bias
         im = Image.fromarray(imarray.astype('uint8')).convert('RGBA')
         im.save('test_images/rgb/rgb_test_image_'+str(n)+'.png')
 
-        imarray = np.random.rand(img_w,img_h,1) * variance + bias
+        imarray = np.random.rand(img_w, img_h, 1) * variance + bias
         im = Image.fromarray(imarray.astype('uint8').squeeze()).convert('L')
         im.save('test_images/gsc/gsc_test_image_'+str(n)+'.png')
 
+
 def teardown_function(func):
     shutil.rmtree('test_images')
 
-def test_image_data_generator():
-    np.random.seed(1337)
 
-    for color_mode in ['gsc','rgb']:
-        file_list = list_pictures('test_images/'+color_mode)
+def test_image_data_generator():
+    for color_mode in ['gsc', 'rgb']:
+        file_list = list_pictures('test_images/' + color_mode)
         img_list = []
         for f in file_list:
-            img_list.append(img_to_array(load_img(f))[None,...])
+            img_list.append(img_to_array(load_img(f))[None, ...])
 
         images = np.vstack(img_list)
         generator = ImageDataGenerator(
@@ -42,21 +41,19 @@ def test_image_data_generator():
             samplewise_center=True,
             featurewise_std_normalization=True,
             samplewise_std_normalization=True,
-
             zca_whitening=True,
             rotation_range=90.,
             width_shift_range=10.,
             height_shift_range=10.,
+            shear_range=0.5,
             horizontal_flip=True,
-            vertical_flip=True
-        )
+            vertical_flip=True)
+        generator.fit(images, augment=True)
 
-        generator.fit(images,augment=True)
-
-        for x,y in generator.flow(images,np.arange(images.shape[0]), shuffle=True, save_to_dir='test_images'):
+        for x, y in generator.flow(images, np.arange(images.shape[0]),
+                                   shuffle=True, save_to_dir='test_images'):
             assert x.shape[1:] == images.shape[1:]
-            #TODO: make sure the normalization is working as inteded
-
+            break
 
 if __name__ == '__main__':
     pytest.main([__file__])

tests/keras/preprocessing/test_sequence.py

@@ -28,6 +28,39 @@ def test_pad_sequences():
     assert_allclose(b, [[1, 1, 1], [1, 1, 2], [1, 2, 3]])
 
 
+def test_pad_sequences_vector():
+    a = [[[1, 1]],
+         [[2, 1], [2, 2]],
+         [[3, 1], [3, 2], [3, 3]]]
+
+    # test padding
+    b = pad_sequences(a, maxlen=3, padding='pre')
+    assert_allclose(b, [[[0, 0], [0, 0], [1, 1]],
+                        [[0, 0], [2, 1], [2, 2]],
+                        [[3, 1], [3, 2], [3, 3]]])
+    b = pad_sequences(a, maxlen=3, padding='post')
+    assert_allclose(b, [[[1, 1], [0, 0], [0, 0]],
+                        [[2, 1], [2, 2], [0, 0]],
+                        [[3, 1], [3, 2], [3, 3]]])
+
+    # test truncating
+    b = pad_sequences(a, maxlen=2, truncating='pre')
+    assert_allclose(b, [[[0, 0], [1, 1]],
+                        [[2, 1], [2, 2]],
+                        [[3, 2], [3, 3]]])
+
+    b = pad_sequences(a, maxlen=2, truncating='post')
+    assert_allclose(b, [[[0, 0], [1, 1]],
+                        [[2, 1], [2, 2]],
+                        [[3, 1], [3, 2]]])
+
+    # test value
+    b = pad_sequences(a, maxlen=3, value=1)
+    assert_allclose(b, [[[1, 1], [1, 1], [1, 1]],
+                        [[1, 1], [2, 1], [2, 2]],
+                        [[3, 1], [3, 2], [3, 3]]])
+
+
 def test_make_sampling_table():
     a = make_sampling_table(3)
     assert_allclose(a, np.asarray([0.00315225,  0.00315225,  0.00547597]),

tests/keras/test_activations.py

@@ -32,6 +32,14 @@ def test_softmax():
     assert_allclose(result, expected, rtol=1e-05)
 
 
+def test_time_distributed_softmax():
+    x = K.placeholder(shape=(1, 1, 5))
+    f = K.function([x], [activations.softmax(x)])
+    test_values = get_standard_values()
+    test_values = np.reshape(test_values, (1, 1, np.size(test_values)))
+    f([test_values])[0]
+
+
 def test_softplus():
     '''
     Test using a reference softplus implementation

tests/keras/test_callbacks.py

@@ -127,7 +127,7 @@ def test_TensorBoard():
     import shutil
     import tensorflow as tf
     import keras.backend.tensorflow_backend as KTF
-    old_session = KTF._get_session()
+    old_session = KTF.get_session()
     filepath = './logs'
     (X_train, y_train), (X_test, y_test) = get_test_data(nb_train=train_samples,
                                                          nb_test=test_samples,
@@ -136,10 +136,33 @@ def test_TensorBoard():
                                                          nb_class=nb_class)
     y_test = np_utils.to_categorical(y_test)
     y_train = np_utils.to_categorical(y_train)
-    # case 1 Sequential wo accuracy
+
+    def data_generator(train):
+        if train:
+            max_batch_index = len(X_train) // batch_size
+        else:
+            max_batch_index = len(X_test) // batch_size
+        i = 0
+        while 1:
+            if train:
+                yield (X_train[i * batch_size: (i + 1) * batch_size], y_train[i * batch_size: (i + 1) * batch_size])
+            else:
+                yield (X_test[i * batch_size: (i + 1) * batch_size], y_test[i * batch_size: (i + 1) * batch_size])
+            i += 1
+            i = i % max_batch_index
+
+    def data_generator_graph(train):
+        while 1:
+            if train:
+                yield {'X_vars': X_train, 'output': y_train}
+            else:
+                yield {'X_vars': X_test, 'output': y_test}
+
+    # case 1 Sequential
+
     with tf.Graph().as_default():
         session = tf.Session('')
-        KTF._set_session(session)
+        KTF.set_session(session)
         model = Sequential()
         model.add(Dense(nb_hidden, input_dim=input_dim, activation='relu'))
         model.add(Dense(nb_class, activation='softmax'))
@@ -147,31 +170,45 @@ def test_TensorBoard():
 
         tsb = callbacks.TensorBoard(log_dir=filepath, histogram_freq=1)
         cbks = [tsb]
+
+        # fit with validation data
+        model.fit(X_train, y_train, batch_size=batch_size, show_accuracy=False,
+                  validation_data=(X_test, y_test), callbacks=cbks, nb_epoch=2)
+
+        # fit with validation data and accuracy
         model.fit(X_train, y_train, batch_size=batch_size, show_accuracy=True,
                   validation_data=(X_test, y_test), callbacks=cbks, nb_epoch=2)
+
+        # fit generator with validation data
+        model.fit_generator(data_generator(True), len(X_train), nb_epoch=2,
+                            show_accuracy=False,
+                            validation_data=(X_test, y_test),
+                            callbacks=cbks)
+
+        # fit generator without validation data
+        model.fit_generator(data_generator(True), len(X_train), nb_epoch=2,
+                            show_accuracy=False,
+                            callbacks=cbks)
+
+        # fit generator with validation data and accuracy
+        model.fit_generator(data_generator(True), len(X_train), nb_epoch=2,
+                            show_accuracy=True,
+                            validation_data=(X_test, y_test),
+                            callbacks=cbks)
+
+        # fit generator without validation data and accuracy
+        model.fit_generator(data_generator(True), len(X_train), nb_epoch=2,
+                            show_accuracy=True,
+                            callbacks=cbks)
+
         assert os.path.exists(filepath)
         shutil.rmtree(filepath)
 
-    # case 2 Sequential w accuracy
+    # case 2 Graph
+
     with tf.Graph().as_default():
         session = tf.Session('')
-        KTF._set_session(session)
-        model = Sequential()
-        model.add(Dense(nb_hidden, input_dim=input_dim, activation='relu'))
-        model.add(Dense(nb_class, activation='softmax'))
-        model.compile(loss='categorical_crossentropy', optimizer='sgd')
-
-        tsb = callbacks.TensorBoard(log_dir=filepath, histogram_freq=1)
-        cbks = [tsb]
-        model.fit(X_train, y_train, batch_size=batch_size, show_accuracy=True,
-                  validation_data=(X_test, y_test), callbacks=cbks, nb_epoch=2)
-        assert os.path.exists(filepath)
-        shutil.rmtree(filepath)
-
-    # case 3 Graph
-    with tf.Graph().as_default():
-        session = tf.Session('')
-        KTF._set_session(session)
+        KTF.set_session(session)
         model = Graph()
         model.add_input(name='X_vars', input_shape=(input_dim, ))
 
@@ -185,14 +222,31 @@ def test_TensorBoard():
 
         tsb = callbacks.TensorBoard(log_dir=filepath, histogram_freq=1)
         cbks = [tsb]
+
+        # fit with validation
         model.fit({'X_vars': X_train, 'output': y_train},
                   batch_size=batch_size,
                   validation_data={'X_vars': X_test, 'output': y_test},
                   callbacks=cbks, nb_epoch=2)
+
+        # fit wo validation
+        model.fit({'X_vars': X_train, 'output': y_train},
+                  batch_size=batch_size,
+                  callbacks=cbks, nb_epoch=2)
+
+        # fit generator with validation
+        model.fit_generator(data_generator_graph(True), 1000, nb_epoch=2,
+                            validation_data={'X_vars': X_test, 'output': y_test},
+                            callbacks=cbks)
+
+        # fit generator wo validation
+        model.fit_generator(data_generator_graph(True), 1000, nb_epoch=2,
+                            callbacks=cbks)
+
         assert os.path.exists(filepath)
         shutil.rmtree(filepath)
 
-    KTF._set_session(old_session)
+    KTF.set_session(old_session)
 
 if __name__ == '__main__':
     pytest.main([__file__])

tests/keras/test_constraints.py

@@ -54,7 +54,7 @@ def test_identity_oddballs():
 def test_unitnorm():
     unitnorm_instance = constraints.unitnorm()
     normalized = unitnorm_instance(K.variable(example_array))
-    norm_of_normalized = np.sqrt(np.sum(K.eval(normalized)**2, axis=1))
+    norm_of_normalized = np.sqrt(np.sum(K.eval(normalized)**2, axis=0))
     # in the unit norm constraint, it should be equal to 1.
     difference = norm_of_normalized - 1.
     largest_difference = np.max(np.abs(difference))

tests/keras/test_graph_model.py

@@ -0,0 +1,425 @@
+from __future__ import absolute_import
+from __future__ import print_function
+import pytest
+import os
+import numpy as np
+np.random.seed(1337)
+
+from keras import backend as K
+from keras.models import Graph, Sequential, model_from_json, model_from_yaml
+from keras.layers.core import Dense, Activation, Merge, Lambda, LambdaMerge, Siamese, add_shared_layer
+from keras.layers import containers
+from keras.utils.test_utils import get_test_data
+
+
+batch_size = 32
+
+(X_train_graph, y_train_graph), (X_test_graph, y_test_graph) = get_test_data(nb_train=1000,
+                                                                             nb_test=200,
+                                                                             input_shape=(32,),
+                                                                             classification=False,
+                                                                             output_shape=(4,))
+(X2_train_graph, y2_train_graph), (X2_test_graph, y2_test_graph) = get_test_data(nb_train=1000,
+                                                                                 nb_test=200,
+                                                                                 input_shape=(32,),
+                                                                                 classification=False,
+                                                                                 output_shape=(1,))
+
+
+def test_graph_fit_generator():
+    def data_generator_graph(train):
+        while 1:
+            if train:
+                yield {'input1': X_train_graph, 'output1': y_train_graph}
+            else:
+                yield {'input1': X_test_graph, 'output1': y_test_graph}
+
+    graph = Graph()
+    graph.add_input(name='input1', input_shape=(32,))
+
+    graph.add_node(Dense(16), name='dense1', input='input1')
+    graph.add_node(Dense(4), name='dense2', input='input1')
+    graph.add_node(Dense(4), name='dense3', input='dense1')
+
+    graph.add_output(name='output1',
+                     inputs=['dense2', 'dense3'],
+                     merge_mode='sum')
+    graph.compile('rmsprop', {'output1': 'mse'})
+
+    graph.fit_generator(data_generator_graph(True), 1000, nb_epoch=4)
+    graph.fit_generator(data_generator_graph(True), 1000, nb_epoch=4,
+                        validation_data={'input1': X_test_graph, 'output1': y_test_graph})
+    graph.fit_generator(data_generator_graph(True), 1000, nb_epoch=4,
+                        validation_data=data_generator_graph(False), nb_val_samples=batch_size * 3)
+    graph.fit_generator(data_generator_graph(True), 1000, nb_epoch=4,
+                        validation_data=data_generator_graph(False), nb_val_samples=batch_size * 3)
+    gen_loss = graph.evaluate_generator(data_generator_graph(True), 128, verbose=0)
+    assert(gen_loss < 3.)
+
+    loss = graph.evaluate({'input1': X_test_graph, 'output1': y_test_graph}, verbose=0)
+    assert(loss < 3.)
+
+    # test show_accuracy
+    graph.fit_generator(data_generator_graph(True), 1000, nb_epoch=4, show_accuracy=True)
+    graph.fit_generator(data_generator_graph(True), 1000, nb_epoch=4,
+                        validation_data={'input1': X_test_graph, 'output1': y_test_graph}, show_accuracy=True)
+    graph.fit_generator(data_generator_graph(True), 1000, nb_epoch=4,
+                        validation_data=data_generator_graph(False), nb_val_samples=batch_size * 3, show_accuracy=True)
+    graph.fit_generator(data_generator_graph(True), 1000, nb_epoch=4,
+                        validation_data=data_generator_graph(False), nb_val_samples=batch_size * 3, show_accuracy=True)
+    gen_loss = graph.evaluate_generator(data_generator_graph(True), 128, verbose=0, show_accuracy=True)
+
+
+def test_1o_1i():
+    # test a non-sequential graph with 1 input and 1 output
+    np.random.seed(1337)
+
+    graph = Graph()
+    graph.add_input(name='input1', input_shape=(32,))
+
+    graph.add_node(Dense(16), name='dense1', input='input1')
+    graph.add_node(Dense(4), name='dense2', input='input1')
+    graph.add_node(Dense(4), name='dense3', input='dense1')
+
+    graph.add_output(name='output1',
+                     inputs=['dense2', 'dense3'],
+                     merge_mode='sum')
+    graph.compile('rmsprop', {'output1': 'mse'})
+
+    graph.fit({'input1': X_train_graph, 'output1': y_train_graph},
+              nb_epoch=10)
+    out = graph.predict({'input1': X_test_graph})
+    assert(type(out == dict))
+    assert(len(out) == 1)
+    loss = graph.test_on_batch({'input1': X_test_graph, 'output1': y_test_graph})
+    loss = graph.train_on_batch({'input1': X_test_graph, 'output1': y_test_graph})
+    loss = graph.evaluate({'input1': X_test_graph, 'output1': y_test_graph}, verbose=0)
+    assert(loss < 2.5)
+
+    # test show_accuracy:
+    graph.fit({'input1': X_train_graph, 'output1': y_train_graph},
+              nb_epoch=1, show_accuracy=True)
+    loss, acc = graph.test_on_batch({'input1': X_test_graph, 'output1': y_test_graph}, accuracy=True)
+    loss, acc = graph.train_on_batch({'input1': X_test_graph, 'output1': y_test_graph}, accuracy=True)
+    loss, acc = graph.evaluate({'input1': X_test_graph, 'output1': y_test_graph}, verbose=0, show_accuracy=True)
+
+    # test validation split
+    graph.fit({'input1': X_train_graph, 'output1': y_train_graph},
+              validation_split=0.2, nb_epoch=1)
+    # test validation data
+    graph.fit({'input1': X_train_graph, 'output1': y_train_graph},
+              validation_data={'input1': X_train_graph, 'output1': y_train_graph},
+              nb_epoch=1)
+
+
+def test_1o_1i_2():
+    # test a more complex non-sequential graph with 1 input and 1 output
+    graph = Graph()
+    graph.add_input(name='input1', input_shape=(32,))
+
+    graph.add_node(Dense(16), name='dense1', input='input1')
+    graph.add_node(Dense(4), name='dense2-0', input='input1')
+    graph.add_node(Activation('relu'), name='dense2', input='dense2-0')
+
+    graph.add_node(Dense(16), name='dense3', input='dense2')
+    graph.add_node(Dense(4), name='dense4', inputs=['dense1', 'dense3'],
+                   merge_mode='sum')
+
+    graph.add_output(name='output1', inputs=['dense2', 'dense4'],
+                     merge_mode='sum')
+    graph.compile('rmsprop', {'output1': 'mse'})
+
+    graph.fit({'input1': X_train_graph, 'output1': y_train_graph},
+              nb_epoch=10)
+    out = graph.predict({'input1': X_train_graph})
+    assert(type(out == dict))
+    assert(len(out) == 1)
+
+    loss = graph.test_on_batch({'input1': X_test_graph, 'output1': y_test_graph})
+    loss = graph.train_on_batch({'input1': X_test_graph, 'output1': y_test_graph})
+    loss = graph.evaluate({'input1': X_test_graph, 'output1': y_test_graph})
+    assert(loss < 2.5)
+
+    graph.get_config(verbose=1)
+    graph.summary()
+
+
+def test_1o_2i():
+    # test a non-sequential graph with 2 inputs and 1 output
+    graph = Graph()
+    graph.add_input(name='input1', input_shape=(32,))
+    graph.add_input(name='input2', input_shape=(32,))
+
+    graph.add_node(Dense(16), name='dense1', input='input1')
+    graph.add_node(Dense(4), name='dense2', input='input2')
+    graph.add_node(Dense(4), name='dense3', input='dense1')
+
+    graph.add_output(name='output1', inputs=['dense2', 'dense3'],
+                     merge_mode='sum')
+    graph.compile('rmsprop', {'output1': 'mse'})
+
+    graph.fit({'input1': X_train_graph, 'input2': X2_train_graph, 'output1': y_train_graph},
+              nb_epoch=10)
+    out = graph.predict({'input1': X_test_graph, 'input2': X2_test_graph})
+    assert(type(out == dict))
+    assert(len(out) == 1)
+
+    loss = graph.test_on_batch({'input1': X_test_graph, 'input2': X2_test_graph, 'output1': y_test_graph})
+    loss = graph.train_on_batch({'input1': X_test_graph, 'input2': X2_test_graph, 'output1': y_test_graph})
+    loss = graph.evaluate({'input1': X_test_graph, 'input2': X2_test_graph, 'output1': y_test_graph})
+    assert(loss < 3.0)
+
+    graph.get_config(verbose=1)
+
+
+def test_siamese_3():
+    graph = Graph()
+    graph.add_input(name='input1', input_shape=(32,))
+    graph.add_input(name='input2', input_shape=(32,))
+
+    graph.add_shared_node(Dense(16), name='shared', inputs=['input1', 'input2'], merge_mode='sum')
+    graph.add_node(Dense(4), name='dense1', input='shared')
+    graph.add_node(Dense(4), name='dense2', input='dense1')
+
+    graph.add_output(name='output1', input='dense2')
+    graph.compile('rmsprop', {'output1': 'mse'})
+
+    graph.fit({'input1': X_train_graph, 'input2': X2_train_graph, 'output1': y_train_graph},
+              nb_epoch=10)
+    out = graph.predict({'input1': X_test_graph, 'input2': X2_test_graph})
+    assert(type(out == dict))
+    assert(len(out) == 1)
+
+    loss = graph.test_on_batch({'input1': X_test_graph, 'input2': X2_test_graph, 'output1': y_test_graph})
+    loss = graph.train_on_batch({'input1': X_test_graph, 'input2': X2_test_graph, 'output1': y_test_graph})
+    loss = graph.evaluate({'input1': X_test_graph, 'input2': X2_test_graph, 'output1': y_test_graph})
+    assert(loss < 3.0)
+
+    graph.get_config(verbose=1)
+
+
+def test_siamese_4():
+    graph = Graph()
+    graph.add_input(name='input1', input_shape=(32,))
+    graph.add_input(name='input2', input_shape=(32,))
+
+    graph.add_shared_node(Dense(16), name='shared1', inputs=['input1', 'input2'])
+    graph.add_shared_node(Dense(4), name='shared2', inputs=['shared1'])
+    graph.add_shared_node(Dense(4), name='shared3', inputs=['shared2'], merge_mode='sum')
+    graph.add_node(Dense(4), name='dense', input='shared3')
+
+    graph.add_output(name='output1', input='dense',
+                     merge_mode='sum')
+    graph.compile('rmsprop', {'output1': 'mse'})
+
+    graph.fit({'input1': X_train_graph, 'input2': X2_train_graph, 'output1': y_train_graph},
+              nb_epoch=10)
+    out = graph.predict({'input1': X_test_graph, 'input2': X2_test_graph})
+    assert(type(out == dict))
+    assert(len(out) == 1)
+
+    loss = graph.test_on_batch({'input1': X_test_graph, 'input2': X2_test_graph, 'output1': y_test_graph})
+    loss = graph.train_on_batch({'input1': X_test_graph, 'input2': X2_test_graph, 'output1': y_test_graph})
+    loss = graph.evaluate({'input1': X_test_graph, 'input2': X2_test_graph, 'output1': y_test_graph})
+    assert(loss < 3.0)
+
+    graph.get_config(verbose=1)
+
+
+def test_siamese_5():
+    graph = Graph()
+    graph.add_input(name='input1', input_shape=(32,))
+    graph.add_input(name='input2', input_shape=(32,))
+
+    graph.add_shared_node(Dense(16), name='shared1', inputs=['input1', 'input2'])
+    graph.add_shared_node(Dense(4), name='shared2', inputs=['shared1'])
+    graph.add_shared_node(Dense(4), name='shared3', inputs=['shared2'], outputs=['shared_output1','shared_output2'])
+    graph.add_node(Dense(4), name='dense1',  input='shared_output1')
+    graph.add_node(Dense(4), name='dense2',  input='shared_output2')
+
+    graph.add_output(name='output1', inputs=['dense1', 'dense2'],
+                     merge_mode='sum')
+    graph.compile('rmsprop', {'output1': 'mse'})
+
+    graph.fit({'input1': X_train_graph, 'input2': X2_train_graph, 'output1': y_train_graph},
+              nb_epoch=10)
+    out = graph.predict({'input1': X_test_graph, 'input2': X2_test_graph})
+    assert(type(out == dict))
+    assert(len(out) == 1)
+
+    loss = graph.test_on_batch({'input1': X_test_graph, 'input2': X2_test_graph, 'output1': y_test_graph})
+    loss = graph.train_on_batch({'input1': X_test_graph, 'input2': X2_test_graph, 'output1': y_test_graph})
+    loss = graph.evaluate({'input1': X_test_graph, 'input2': X2_test_graph, 'output1': y_test_graph})
+    assert(loss < 3.0)
+
+    graph.get_config(verbose=1)
+
+
+def test_2o_1i_weights():
+    # test a non-sequential graph with 1 input and 2 outputs
+    graph = Graph()
+    graph.add_input(name='input1', input_shape=(32,))
+
+    graph.add_node(Dense(16), name='dense1', input='input1')
+    graph.add_node(Dense(4), name='dense2', input='input1')
+    graph.add_node(Dense(1), name='dense3', input='dense1')
+
+    graph.add_output(name='output1', input='dense2')
+    graph.add_output(name='output2', input='dense3')
+    graph.compile('rmsprop', {'output1': 'mse', 'output2': 'mse'})
+
+    graph.fit({'input1': X_train_graph, 'output1': y_train_graph, 'output2': y2_train_graph},
+              nb_epoch=10)
+    out = graph.predict({'input1': X_test_graph})
+    assert(type(out == dict))
+    assert(len(out) == 2)
+    loss = graph.test_on_batch({'input1': X_test_graph, 'output1': y_test_graph, 'output2': y2_test_graph})
+    loss = graph.train_on_batch({'input1': X_test_graph, 'output1': y_test_graph, 'output2': y2_test_graph})
+    loss = graph.evaluate({'input1': X_test_graph, 'output1': y_test_graph, 'output2': y2_test_graph})
+    assert(loss < 4.)
+
+    # test weight saving
+    fname = 'test_2o_1i_weights_temp.h5'
+    graph.save_weights(fname, overwrite=True)
+
+    graph = Graph()
+    graph.add_input(name='input1', input_shape=(32,))
+    graph.add_node(Dense(16), name='dense1', input='input1')
+    graph.add_node(Dense(4), name='dense2', input='input1')
+    graph.add_node(Dense(1), name='dense3', input='dense1')
+    graph.add_output(name='output1', input='dense2')
+    graph.add_output(name='output2', input='dense3')
+    graph.compile('rmsprop', {'output1': 'mse', 'output2': 'mse'})
+    graph.load_weights('test_2o_1i_weights_temp.h5')
+    os.remove(fname)
+
+    nloss = graph.evaluate({'input1': X_test_graph, 'output1': y_test_graph, 'output2': y2_test_graph})
+    assert(loss == nloss)
+
+    # test loss weights
+    graph.compile('rmsprop', {'output1': 'mse', 'output2': 'mse'},
+                  loss_weights={'output1': 1., 'output2': 2.})
+    graph.fit({'input1': X_train_graph, 'output1': y_train_graph, 'output2': y2_train_graph},
+              nb_epoch=1)
+
+
+def test_2o_1i_sample_weights():
+    # test a non-sequential graph with 1 input and 2 outputs with sample weights
+    graph = Graph()
+    graph.add_input(name='input1', input_shape=(32,))
+
+    graph.add_node(Dense(16), name='dense1', input='input1')
+    graph.add_node(Dense(4), name='dense2', input='input1')
+    graph.add_node(Dense(1), name='dense3', input='dense1')
+
+    graph.add_output(name='output1', input='dense2')
+    graph.add_output(name='output2', input='dense3')
+
+    weights1 = np.random.uniform(size=y_train_graph.shape[0])
+    weights2 = np.random.uniform(size=y2_train_graph.shape[0])
+    weights1_test = np.random.uniform(size=y_test_graph.shape[0])
+    weights2_test = np.random.uniform(size=y2_test_graph.shape[0])
+
+    graph.compile('rmsprop', {'output1': 'mse', 'output2': 'mse'})
+
+    graph.fit({'input1': X_train_graph, 'output1': y_train_graph, 'output2': y2_train_graph},
+              nb_epoch=10,
+              sample_weight={'output1': weights1, 'output2': weights2})
+    out = graph.predict({'input1': X_test_graph})
+    assert(type(out == dict))
+    assert(len(out) == 2)
+    loss = graph.test_on_batch({'input1': X_test_graph, 'output1': y_test_graph, 'output2': y2_test_graph},
+                               sample_weight={'output1': weights1_test, 'output2': weights2_test})
+    loss = graph.train_on_batch({'input1': X_train_graph, 'output1': y_train_graph, 'output2': y2_train_graph},
+                                sample_weight={'output1': weights1, 'output2': weights2})
+    loss = graph.evaluate({'input1': X_train_graph, 'output1': y_train_graph, 'output2': y2_train_graph},
+                          sample_weight={'output1': weights1, 'output2': weights2})
+
+
+def test_recursive():
+    # test layer-like API
+
+    graph = containers.Graph()
+    graph.add_input(name='input1', input_shape=(32,))
+    graph.add_node(Dense(16), name='dense1', input='input1')
+    graph.add_node(Dense(4), name='dense2', input='input1')
+    graph.add_node(Dense(4), name='dense3', input='dense1')
+    graph.add_output(name='output1', inputs=['dense2', 'dense3'],
+                     merge_mode='sum')
+
+    seq = Sequential()
+    seq.add(Dense(32, input_shape=(32,)))
+    seq.add(graph)
+    seq.add(Dense(4))
+
+    seq.compile('rmsprop', 'mse')
+
+    seq.fit(X_train_graph, y_train_graph, batch_size=10, nb_epoch=10)
+    loss = seq.evaluate(X_test_graph, y_test_graph)
+    assert(loss < 2.5)
+
+    loss = seq.evaluate(X_test_graph, y_test_graph, show_accuracy=True)
+    seq.predict(X_test_graph)
+    seq.get_config(verbose=1)
+
+
+def test_create_output():
+    # test create_output argument
+    graph = Graph()
+    graph.add_input(name='input1', input_shape=(32,))
+
+    graph.add_node(Dense(16), name='dense1', input='input1')
+    graph.add_node(Dense(4), name='dense2', input='input1')
+    graph.add_node(Dense(4), name='dense3', input='dense1')
+    graph.add_node(Dense(4), name='output1', inputs=['dense2', 'dense3'],
+                   merge_mode='sum', create_output=True)
+    graph.compile('rmsprop', {'output1': 'mse'})
+
+    history = graph.fit({'input1': X_train_graph, 'output1': y_train_graph},
+                        nb_epoch=10)
+    out = graph.predict({'input1': X_test_graph})
+    assert(type(out == dict))
+    assert(len(out) == 1)
+
+    loss = graph.test_on_batch({'input1': X_test_graph, 'output1': y_test_graph})
+    loss = graph.train_on_batch({'input1': X_test_graph, 'output1': y_test_graph})
+    loss = graph.evaluate({'input1': X_test_graph, 'output1': y_test_graph})
+    assert(loss < 2.5)
+
+    # test serialization
+    config = graph.to_json()
+    del graph
+    graph = model_from_json(config)
+
+
+def test_count_params():
+    # test count params
+
+    nb_units = 100
+    nb_classes = 2
+
+    graph = Graph()
+    graph.add_input(name='input1', input_shape=(32,))
+    graph.add_input(name='input2', input_shape=(32,))
+    graph.add_node(Dense(nb_units),
+                   name='dense1', input='input1')
+    graph.add_node(Dense(nb_classes),
+                   name='dense2', input='input2')
+    graph.add_node(Dense(nb_classes),
+                   name='dense3', input='dense1')
+    graph.add_output(name='output', inputs=['dense2', 'dense3'],
+                     merge_mode='sum')
+
+    n = 32 * nb_units + nb_units
+    n += 32 * nb_classes + nb_classes
+    n += nb_units * nb_classes + nb_classes
+
+    assert(n == graph.count_params())
+
+    graph.compile('rmsprop', {'output': 'binary_crossentropy'})
+
+    assert(n == graph.count_params())
+
+
+if __name__ == '__main__':
+    pytest.main([__file__])

tests/keras/test_objectives.py

@@ -0,0 +1,36 @@
+import pytest
+import numpy as np
+
+from keras import objectives
+from keras import backend as K
+
+
+allobj = [objectives.mean_squared_error,
+          objectives.mean_absolute_error,
+          objectives.mean_absolute_percentage_error,
+          objectives.mean_squared_logarithmic_error,
+          objectives.squared_hinge,
+          objectives.hinge, objectives.categorical_crossentropy,
+          objectives.binary_crossentropy,
+          objectives.poisson,
+          objectives.cosine_proximity]
+
+
+def test_objective_shapes_3d():
+    y_a = K.variable(np.random.random((5, 6, 7)))
+    y_b = K.variable(np.random.random((5, 6, 7)))
+    for obj in allobj:
+        objective_output = obj(y_a, y_b)
+        assert K.eval(objective_output).shape == (5, 6)
+
+
+def test_objective_shapes_2d():
+    y_a = K.variable(np.random.random((6, 7)))
+    y_b = K.variable(np.random.random((6, 7)))
+    for obj in allobj:
+        objective_output = obj(y_a, y_b)
+        assert K.eval(objective_output).shape == (6,)
+
+
+if __name__ == "__main__":
+    pytest.main([__file__])

tests/keras/test_sequential_model.py

@@ -1,6 +1,7 @@
 from __future__ import absolute_import
 from __future__ import print_function
 import pytest
+import os
 import numpy as np
 np.random.seed(1337)
 
@@ -11,8 +12,6 @@ from keras.layers import containers
 from keras.utils import np_utils
 from keras.utils.test_utils import get_test_data
 
-import os
-
 
 input_dim = 32
 nb_hidden = 16
@@ -37,10 +36,6 @@ def _get_test_data():
     return (X_train, y_train), (X_test, y_test)
 
 
-####################
-# SEQUENTIAL TEST  #
-####################
-
 def test_sequential_fit_generator():
     (X_train, y_train), (X_test, y_test) = _get_test_data()
 
@@ -69,6 +64,10 @@ def test_sequential_fit_generator():
     model.fit_generator(data_generator(True), len(X_train), nb_epoch, show_accuracy=True)
     model.fit_generator(data_generator(True), len(X_train), nb_epoch, show_accuracy=False, validation_data=(X_test, y_test))
     model.fit_generator(data_generator(True), len(X_train), nb_epoch, show_accuracy=True, validation_data=(X_test, y_test))
+    model.fit_generator(data_generator(True), len(X_train), nb_epoch, show_accuracy=False,
+                        validation_data=data_generator(False), nb_val_samples=batch_size * 3)
+    model.fit_generator(data_generator(True), len(X_train), nb_epoch, show_accuracy=True,
+                        validation_data=data_generator(False), nb_val_samples=batch_size * 3)
 
     loss = model.evaluate(X_train, y_train, verbose=0)
     assert(loss < 0.9)
@@ -77,6 +76,21 @@ def test_sequential_fit_generator():
 def test_sequential():
     (X_train, y_train), (X_test, y_test) = _get_test_data()
 
+    # TODO: factor out
+    def data_generator(train):
+        if train:
+            max_batch_index = len(X_train) // batch_size
+        else:
+            max_batch_index = len(X_test) // batch_size
+        i = 0
+        while 1:
+            if train:
+                yield (X_train[i * batch_size: (i + 1) * batch_size], y_train[i * batch_size: (i + 1) * batch_size])
+            else:
+                yield (X_test[i * batch_size: (i + 1) * batch_size], y_test[i * batch_size: (i + 1) * batch_size])
+            i += 1
+            i = i % max_batch_index
+
     model = Sequential()
     model.add(Dense(nb_hidden, input_shape=(input_dim,)))
     model.add(Activation('relu'))
@@ -94,6 +108,66 @@ def test_sequential():
 
     model.train_on_batch(X_train[:32], y_train[:32])
 
+    gen_loss = model.evaluate_generator(data_generator(True), 256, verbose=0)
+    assert(gen_loss < 0.8)
+
+    loss = model.evaluate(X_test, y_test, verbose=0)
+    assert(loss < 0.8)
+
+    model.predict(X_test, verbose=0)
+    model.predict_classes(X_test, verbose=0)
+    model.predict_proba(X_test, verbose=0)
+    model.get_config(verbose=0)
+
+    fname = 'test_sequential_temp.h5'
+    model.save_weights(fname, overwrite=True)
+    model = Sequential()
+    model.add(Dense(nb_hidden, input_shape=(input_dim,)))
+    model.add(Activation('relu'))
+    model.add(Dense(nb_class))
+    model.add(Activation('softmax'))
+    model.compile(loss='categorical_crossentropy', optimizer='rmsprop')
+    model.load_weights(fname)
+    os.remove(fname)
+
+    nloss = model.evaluate(X_test, y_test, verbose=0)
+    assert(loss == nloss)
+
+    # test json serialization
+    json_data = model.to_json()
+    model = model_from_json(json_data)
+
+    # test yaml serialization
+    yaml_data = model.to_yaml()
+    model = model_from_yaml(yaml_data)
+
+
+def test_nested_sequential():
+    (X_train, y_train), (X_test, y_test) = _get_test_data()
+
+    inner = Sequential()
+    inner.add(Dense(nb_hidden, input_shape=(input_dim,)))
+    inner.add(Activation('relu'))
+    inner.add(Dense(nb_class))
+
+    middle = Sequential()
+    middle.add(inner)
+
+    model = Sequential()
+    model.add(middle)
+    model.add(Activation('softmax'))
+    model.compile(loss='categorical_crossentropy', optimizer='rmsprop')
+    model.summary()
+
+    model.fit(X_train, y_train, batch_size=batch_size, nb_epoch=nb_epoch, show_accuracy=True, verbose=1, validation_data=(X_test, y_test))
+    model.fit(X_train, y_train, batch_size=batch_size, nb_epoch=nb_epoch, show_accuracy=False, verbose=2, validation_data=(X_test, y_test))
+    model.fit(X_train, y_train, batch_size=batch_size, nb_epoch=nb_epoch, show_accuracy=True, verbose=2, validation_split=0.1)
+    model.fit(X_train, y_train, batch_size=batch_size, nb_epoch=nb_epoch, show_accuracy=False, verbose=1, validation_split=0.1)
+    model.fit(X_train, y_train, batch_size=batch_size, nb_epoch=nb_epoch, verbose=0)
+    model.fit(X_train, y_train, batch_size=batch_size, nb_epoch=nb_epoch, verbose=1, shuffle=False)
+
+    model.train_on_batch(X_train[:32], y_train[:32])
+
     loss = model.evaluate(X_test, y_test, verbose=0)
     assert(loss < 0.8)
 
@@ -102,12 +176,19 @@ def test_sequential():
     model.predict_proba(X_test, verbose=0)
     model.get_config(verbose=0)
 
-    fname = 'test_sequential_temp.h5'
+    fname = 'test_nested_sequential_temp.h5'
     model.save_weights(fname, overwrite=True)
+
+    inner = Sequential()
+    inner.add(Dense(nb_hidden, input_shape=(input_dim,)))
+    inner.add(Activation('relu'))
+    inner.add(Dense(nb_class))
+
+    middle = Sequential()
+    middle.add(inner)
+
     model = Sequential()
-    model.add(Dense(nb_hidden, input_shape=(input_dim,)))
-    model.add(Activation('relu'))
-    model.add(Dense(nb_class))
+    model.add(middle)
     model.add(Activation('softmax'))
     model.compile(loss='categorical_crossentropy', optimizer='rmsprop')
     model.load_weights(fname)
@@ -357,6 +438,7 @@ def test_merge_overlap():
 
 def test_lambda():
     (X_train, y_train), (X_test, y_test) = _get_test_data()
+
     def func(X):
         s = X[0]
         for i in range(1, len(X)):
@@ -420,6 +502,22 @@ def test_lambda():
     nloss = model.evaluate([X_test, X_test], y_test, verbose=0)
     assert(loss == nloss)
 
+    # test "join" mode in Lambda
+    def difference(input_dict):
+        assert(len(input_dict) == 2)
+        keys = list(input_dict.keys())
+        return input_dict[keys[0]] - input_dict[keys[1]]
+
+    g = Graph()
+    g.add_input(name='input_a', input_shape=(2,))
+    g.add_input(name='input_b', input_shape=(2,))
+    g.add_node(Lambda(difference, output_shape=(2,)),
+               inputs=['input_a', 'input_b'],
+               merge_mode='join',
+               name='d')
+    g.add_output(name='output', input='d')
+    g.compile(loss={'output': 'categorical_crossentropy'}, optimizer='rmsprop')
+
 
 def test_sequential_count_params():
     input_dim = 20
@@ -565,383 +663,5 @@ def test_siamese_2():
     assert(loss == nloss)
 
 
-###############
-# GRAPH TEST  #
-###############
-
-(X_train_graph, y_train_graph), (X_test_graph, y_test_graph) = get_test_data(nb_train=1000,
-                                                                             nb_test=200,
-                                                                             input_shape=(32,),
-                                                                             classification=False,
-                                                                             output_shape=(4,))
-(X2_train_graph, y2_train_graph), (X2_test_graph, y2_test_graph) = get_test_data(nb_train=1000,
-                                                                                 nb_test=200,
-                                                                                 input_shape=(32,),
-                                                                                 classification=False,
-                                                                                 output_shape=(1,))
-
-
-def test_graph_fit_generator():
-    def data_generator_graph(train):
-        while 1:
-            if train:
-                yield {'input1': X_train_graph, 'output1': y_train_graph}
-            else:
-                yield {'input1': X_test_graph, 'output1': y_test_graph}
-
-    graph = Graph()
-    graph.add_input(name='input1', input_shape=(32,))
-
-    graph.add_node(Dense(16), name='dense1', input='input1')
-    graph.add_node(Dense(4), name='dense2', input='input1')
-    graph.add_node(Dense(4), name='dense3', input='dense1')
-
-    graph.add_output(name='output1',
-                     inputs=['dense2', 'dense3'],
-                     merge_mode='sum')
-    graph.compile('rmsprop', {'output1': 'mse'})
-
-    graph.fit_generator(data_generator_graph(True), 1000, nb_epoch=4)
-    graph.fit_generator(data_generator_graph(True), 1000, nb_epoch=4)
-    graph.fit_generator(data_generator_graph(True), 1000, nb_epoch=4, validation_data={'input1': X_test_graph, 'output1': y_test_graph})
-    graph.fit_generator(data_generator_graph(True), 1000, nb_epoch=4, validation_data={'input1': X_test_graph, 'output1': y_test_graph})
-
-    loss = graph.evaluate({'input1': X_test_graph, 'output1': y_test_graph}, verbose=0)
-    assert(loss < 3.)
-
-
-def test_1o_1i():
-    # test a non-sequential graph with 1 input and 1 output
-    np.random.seed(1337)
-
-    graph = Graph()
-    graph.add_input(name='input1', input_shape=(32,))
-
-    graph.add_node(Dense(16), name='dense1', input='input1')
-    graph.add_node(Dense(4), name='dense2', input='input1')
-    graph.add_node(Dense(4), name='dense3', input='dense1')
-
-    graph.add_output(name='output1',
-                     inputs=['dense2', 'dense3'],
-                     merge_mode='sum')
-    graph.compile('rmsprop', {'output1': 'mse'})
-
-    graph.fit({'input1': X_train_graph, 'output1': y_train_graph},
-              nb_epoch=10)
-    out = graph.predict({'input1': X_test_graph})
-    assert(type(out == dict))
-    assert(len(out) == 1)
-    loss = graph.test_on_batch({'input1': X_test_graph, 'output1': y_test_graph})
-    loss = graph.train_on_batch({'input1': X_test_graph, 'output1': y_test_graph})
-    loss = graph.evaluate({'input1': X_test_graph, 'output1': y_test_graph}, verbose=0)
-    assert(loss < 2.5)
-
-    # test validation split
-    graph.fit({'input1': X_train_graph, 'output1': y_train_graph},
-              validation_split=0.2, nb_epoch=1)
-    # test validation data
-    graph.fit({'input1': X_train_graph, 'output1': y_train_graph},
-              validation_data={'input1': X_train_graph, 'output1': y_train_graph},
-              nb_epoch=1)
-
-
-def test_1o_1i_2():
-    # test a more complex non-sequential graph with 1 input and 1 output
-    graph = Graph()
-    graph.add_input(name='input1', input_shape=(32,))
-
-    graph.add_node(Dense(16), name='dense1', input='input1')
-    graph.add_node(Dense(4), name='dense2-0', input='input1')
-    graph.add_node(Activation('relu'), name='dense2', input='dense2-0')
-
-    graph.add_node(Dense(16), name='dense3', input='dense2')
-    graph.add_node(Dense(4), name='dense4', inputs=['dense1', 'dense3'],
-                   merge_mode='sum')
-
-    graph.add_output(name='output1', inputs=['dense2', 'dense4'],
-                     merge_mode='sum')
-    graph.compile('rmsprop', {'output1': 'mse'})
-
-    graph.fit({'input1': X_train_graph, 'output1': y_train_graph},
-              nb_epoch=10)
-    out = graph.predict({'input1': X_train_graph})
-    assert(type(out == dict))
-    assert(len(out) == 1)
-
-    loss = graph.test_on_batch({'input1': X_test_graph, 'output1': y_test_graph})
-    loss = graph.train_on_batch({'input1': X_test_graph, 'output1': y_test_graph})
-    loss = graph.evaluate({'input1': X_test_graph, 'output1': y_test_graph})
-    assert(loss < 2.5)
-
-    graph.get_config(verbose=1)
-    graph.summary()
-
-
-def test_1o_2i():
-    # test a non-sequential graph with 2 inputs and 1 output
-    graph = Graph()
-    graph.add_input(name='input1', input_shape=(32,))
-    graph.add_input(name='input2', input_shape=(32,))
-
-    graph.add_node(Dense(16), name='dense1', input='input1')
-    graph.add_node(Dense(4), name='dense2', input='input2')
-    graph.add_node(Dense(4), name='dense3', input='dense1')
-
-    graph.add_output(name='output1', inputs=['dense2', 'dense3'],
-                     merge_mode='sum')
-    graph.compile('rmsprop', {'output1': 'mse'})
-
-    graph.fit({'input1': X_train_graph, 'input2': X2_train_graph, 'output1': y_train_graph},
-              nb_epoch=10)
-    out = graph.predict({'input1': X_test_graph, 'input2': X2_test_graph})
-    assert(type(out == dict))
-    assert(len(out) == 1)
-
-    loss = graph.test_on_batch({'input1': X_test_graph, 'input2': X2_test_graph, 'output1': y_test_graph})
-    loss = graph.train_on_batch({'input1': X_test_graph, 'input2': X2_test_graph, 'output1': y_test_graph})
-    loss = graph.evaluate({'input1': X_test_graph, 'input2': X2_test_graph, 'output1': y_test_graph})
-    assert(loss < 3.0)
-
-    graph.get_config(verbose=1)
-
-
-def test_siamese_3():
-    graph = Graph()
-    graph.add_input(name='input1', input_shape=(32,))
-    graph.add_input(name='input2', input_shape=(32,))
-
-    graph.add_shared_node(Dense(16), name='shared', inputs=['input1', 'input2'], merge_mode='sum')
-    graph.add_node(Dense(4), name='dense1', input='shared')
-    graph.add_node(Dense(4), name='dense2', input='dense1')
-
-    graph.add_output(name='output1', input='dense2')
-    graph.compile('rmsprop', {'output1': 'mse'})
-
-    graph.fit({'input1': X_train_graph, 'input2': X2_train_graph, 'output1': y_train_graph},
-              nb_epoch=10)
-    out = graph.predict({'input1': X_test_graph, 'input2': X2_test_graph})
-    assert(type(out == dict))
-    assert(len(out) == 1)
-
-    loss = graph.test_on_batch({'input1': X_test_graph, 'input2': X2_test_graph, 'output1': y_test_graph})
-    loss = graph.train_on_batch({'input1': X_test_graph, 'input2': X2_test_graph, 'output1': y_test_graph})
-    loss = graph.evaluate({'input1': X_test_graph, 'input2': X2_test_graph, 'output1': y_test_graph})
-    assert(loss < 3.0)
-
-    graph.get_config(verbose=1)
-
-
-def test_siamese_4():
-    graph = Graph()
-    graph.add_input(name='input1', input_shape=(32,))
-    graph.add_input(name='input2', input_shape=(32,))
-
-    graph.add_shared_node(Dense(16), name='shared1', inputs=['input1', 'input2'])
-    graph.add_shared_node(Dense(4), name='shared2', inputs=['shared1'])
-    graph.add_shared_node(Dense(4), name='shared3', inputs=['shared2'], merge_mode='sum')
-    graph.add_node(Dense(4), name='dense', input='shared3')
-
-    graph.add_output(name='output1', input='dense',
-                     merge_mode='sum')
-    graph.compile('rmsprop', {'output1': 'mse'})
-
-    graph.fit({'input1': X_train_graph, 'input2': X2_train_graph, 'output1': y_train_graph},
-              nb_epoch=10)
-    out = graph.predict({'input1': X_test_graph, 'input2': X2_test_graph})
-    assert(type(out == dict))
-    assert(len(out) == 1)
-
-    loss = graph.test_on_batch({'input1': X_test_graph, 'input2': X2_test_graph, 'output1': y_test_graph})
-    loss = graph.train_on_batch({'input1': X_test_graph, 'input2': X2_test_graph, 'output1': y_test_graph})
-    loss = graph.evaluate({'input1': X_test_graph, 'input2': X2_test_graph, 'output1': y_test_graph})
-    assert(loss < 3.0)
-
-    graph.get_config(verbose=1)
-
-
-def test_siamese_5():
-    graph = Graph()
-    graph.add_input(name='input1', input_shape=(32,))
-    graph.add_input(name='input2', input_shape=(32,))
-
-    graph.add_shared_node(Dense(16), name='shared1', inputs=['input1', 'input2'])
-    graph.add_shared_node(Dense(4), name='shared2', inputs=['shared1'])
-    graph.add_shared_node(Dense(4), name='shared3', inputs=['shared2'], outputs=['shared_output1','shared_output2'])
-    graph.add_node(Dense(4), name='dense1',  input='shared_output1')
-    graph.add_node(Dense(4), name='dense2',  input='shared_output2')
-
-    graph.add_output(name='output1', inputs=['dense1', 'dense2'],
-                     merge_mode='sum')
-    graph.compile('rmsprop', {'output1': 'mse'})
-
-    graph.fit({'input1': X_train_graph, 'input2': X2_train_graph, 'output1': y_train_graph},
-              nb_epoch=10)
-    out = graph.predict({'input1': X_test_graph, 'input2': X2_test_graph})
-    assert(type(out == dict))
-    assert(len(out) == 1)
-
-    loss = graph.test_on_batch({'input1': X_test_graph, 'input2': X2_test_graph, 'output1': y_test_graph})
-    loss = graph.train_on_batch({'input1': X_test_graph, 'input2': X2_test_graph, 'output1': y_test_graph})
-    loss = graph.evaluate({'input1': X_test_graph, 'input2': X2_test_graph, 'output1': y_test_graph})
-    assert(loss < 3.0)
-
-    graph.get_config(verbose=1)
-
-
-def test_2o_1i_weights():
-    # test a non-sequential graph with 1 input and 2 outputs
-    graph = Graph()
-    graph.add_input(name='input1', input_shape=(32,))
-
-    graph.add_node(Dense(16), name='dense1', input='input1')
-    graph.add_node(Dense(4), name='dense2', input='input1')
-    graph.add_node(Dense(1), name='dense3', input='dense1')
-
-    graph.add_output(name='output1', input='dense2')
-    graph.add_output(name='output2', input='dense3')
-    graph.compile('rmsprop', {'output1': 'mse', 'output2': 'mse'})
-
-    graph.fit({'input1': X_train_graph, 'output1': y_train_graph, 'output2': y2_train_graph},
-              nb_epoch=10)
-    out = graph.predict({'input1': X_test_graph})
-    assert(type(out == dict))
-    assert(len(out) == 2)
-    loss = graph.test_on_batch({'input1': X_test_graph, 'output1': y_test_graph, 'output2': y2_test_graph})
-    loss = graph.train_on_batch({'input1': X_test_graph, 'output1': y_test_graph, 'output2': y2_test_graph})
-    loss = graph.evaluate({'input1': X_test_graph, 'output1': y_test_graph, 'output2': y2_test_graph})
-    assert(loss < 4.)
-
-    # test weight saving
-    fname = 'test_2o_1i_weights_temp.h5'
-    graph.save_weights(fname, overwrite=True)
-
-    graph = Graph()
-    graph.add_input(name='input1', input_shape=(32,))
-    graph.add_node(Dense(16), name='dense1', input='input1')
-    graph.add_node(Dense(4), name='dense2', input='input1')
-    graph.add_node(Dense(1), name='dense3', input='dense1')
-    graph.add_output(name='output1', input='dense2')
-    graph.add_output(name='output2', input='dense3')
-    graph.compile('rmsprop', {'output1': 'mse', 'output2': 'mse'})
-    graph.load_weights('test_2o_1i_weights_temp.h5')
-    os.remove(fname)
-
-    nloss = graph.evaluate({'input1': X_test_graph, 'output1': y_test_graph, 'output2': y2_test_graph})
-    assert(loss == nloss)
-
-
-def test_2o_1i_sample_weights():
-    # test a non-sequential graph with 1 input and 2 outputs with sample weights
-    graph = Graph()
-    graph.add_input(name='input1', input_shape=(32,))
-
-    graph.add_node(Dense(16), name='dense1', input='input1')
-    graph.add_node(Dense(4), name='dense2', input='input1')
-    graph.add_node(Dense(1), name='dense3', input='dense1')
-
-    graph.add_output(name='output1', input='dense2')
-    graph.add_output(name='output2', input='dense3')
-
-    weights1 = np.random.uniform(size=y_train_graph.shape[0])
-    weights2 = np.random.uniform(size=y2_train_graph.shape[0])
-    weights1_test = np.random.uniform(size=y_test_graph.shape[0])
-    weights2_test = np.random.uniform(size=y2_test_graph.shape[0])
-
-    graph.compile('rmsprop', {'output1': 'mse', 'output2': 'mse'})
-
-    graph.fit({'input1': X_train_graph, 'output1': y_train_graph, 'output2': y2_train_graph},
-              nb_epoch=10,
-              sample_weight={'output1': weights1, 'output2': weights2})
-    out = graph.predict({'input1': X_test_graph})
-    assert(type(out == dict))
-    assert(len(out) == 2)
-    loss = graph.test_on_batch({'input1': X_test_graph, 'output1': y_test_graph, 'output2': y2_test_graph},
-                               sample_weight={'output1': weights1_test, 'output2': weights2_test})
-    loss = graph.train_on_batch({'input1': X_train_graph, 'output1': y_train_graph, 'output2': y2_train_graph},
-                                sample_weight={'output1': weights1, 'output2': weights2})
-    loss = graph.evaluate({'input1': X_train_graph, 'output1': y_train_graph, 'output2': y2_train_graph},
-                          sample_weight={'output1': weights1, 'output2': weights2})
-
-
-def test_recursive():
-    # test layer-like API
-
-    graph = containers.Graph()
-    graph.add_input(name='input1', input_shape=(32,))
-    graph.add_node(Dense(16), name='dense1', input='input1')
-    graph.add_node(Dense(4), name='dense2', input='input1')
-    graph.add_node(Dense(4), name='dense3', input='dense1')
-    graph.add_output(name='output1', inputs=['dense2', 'dense3'],
-                     merge_mode='sum')
-
-    seq = Sequential()
-    seq.add(Dense(32, input_shape=(32,)))
-    seq.add(graph)
-    seq.add(Dense(4))
-
-    seq.compile('rmsprop', 'mse')
-
-    seq.fit(X_train_graph, y_train_graph, batch_size=10, nb_epoch=10)
-    loss = seq.evaluate(X_test_graph, y_test_graph)
-    assert(loss < 2.5)
-
-    loss = seq.evaluate(X_test_graph, y_test_graph, show_accuracy=True)
-    seq.predict(X_test_graph)
-    seq.get_config(verbose=1)
-
-
-def test_create_output():
-    # test create_output argument
-    graph = Graph()
-    graph.add_input(name='input1', input_shape=(32,))
-
-    graph.add_node(Dense(16), name='dense1', input='input1')
-    graph.add_node(Dense(4), name='dense2', input='input1')
-    graph.add_node(Dense(4), name='dense3', input='dense1')
-    graph.add_node(Dense(4), name='output1', inputs=['dense2', 'dense3'],
-                   merge_mode='sum', create_output=True)
-    graph.compile('rmsprop', {'output1': 'mse'})
-
-    history = graph.fit({'input1': X_train_graph, 'output1': y_train_graph},
-                        nb_epoch=10)
-    out = graph.predict({'input1': X_test_graph})
-    assert(type(out == dict))
-    assert(len(out) == 1)
-
-    loss = graph.test_on_batch({'input1': X_test_graph, 'output1': y_test_graph})
-    loss = graph.train_on_batch({'input1': X_test_graph, 'output1': y_test_graph})
-    loss = graph.evaluate({'input1': X_test_graph, 'output1': y_test_graph})
-    assert(loss < 2.5)
-
-
-def test_count_params():
-    # test count params
-
-    nb_units = 100
-    nb_classes = 2
-
-    graph = Graph()
-    graph.add_input(name='input1', input_shape=(32,))
-    graph.add_input(name='input2', input_shape=(32,))
-    graph.add_node(Dense(nb_units),
-                   name='dense1', input='input1')
-    graph.add_node(Dense(nb_classes),
-                   name='dense2', input='input2')
-    graph.add_node(Dense(nb_classes),
-                   name='dense3', input='dense1')
-    graph.add_output(name='output', inputs=['dense2', 'dense3'],
-                     merge_mode='sum')
-
-    n = 32 * nb_units + nb_units
-    n += 32 * nb_classes + nb_classes
-    n += nb_units * nb_classes + nb_classes
-
-    assert(n == graph.count_params())
-
-    graph.compile('rmsprop', {'output': 'binary_crossentropy'})
-
-    assert(n == graph.count_params())
-
-
 if __name__ == '__main__':
     pytest.main([__file__])

tests/keras/wrappers/test_scikit_learn.py

@@ -37,33 +37,85 @@ y_test = np_utils.to_categorical(y_test, nb_classes=nb_class)
                                                                      output_shape=(1,))
 
 
-@pytest.mark.skipif(K._BACKEND=='tensorflow', reason="currently not working with TensorFlow")
-def test_keras_classifier():
+def build_fn_clf(hidden_dims=50):
     model = Sequential()
     model.add(Dense(input_dim, input_shape=(input_dim,)))
     model.add(Activation('relu'))
+    model.add(Dense(hidden_dims))
+    model.add(Activation('relu'))
     model.add(Dense(nb_class))
     model.add(Activation('softmax'))
-
-    sklearn_clf = KerasClassifier(model, optimizer=optim, loss=loss,
-                                  train_batch_size=batch_size,
-                                  test_batch_size=batch_size,
-                                  nb_epoch=nb_epoch)
-    sklearn_clf.fit(X_train, y_train)
-    sklearn_clf.score(X_test, y_test)
+    model.compile(optimizer='sgd', loss='categorical_crossentropy',
+                  class_mode='binary')
+    return model
 
 
-@pytest.mark.skipif(K._BACKEND=='tensorflow', reason="currently not working with TensorFlow")
-def test_keras_regressor():
+class Class_build_fn_clf(object):
+    def __call__(self, hidden_dims):
+        return build_fn_clf(hidden_dims)
+
+
+class Inherit_class_build_fn_clf(KerasClassifier):
+    def __call__(self, hidden_dims):
+        return build_fn_clf(hidden_dims)
+
+
+def build_fn_reg(hidden_dims=50):
     model = Sequential()
     model.add(Dense(input_dim, input_shape=(input_dim,)))
     model.add(Activation('relu'))
+    model.add(Dense(hidden_dims))
+    model.add(Activation('relu'))
     model.add(Dense(1))
-    model.add(Activation('softmax'))
+    model.add(Activation('linear'))
+    model.compile(optimizer='sgd', loss='mean_absolute_error')
+    return model
 
-    sklearn_regressor = KerasRegressor(model, optimizer=optim, loss=loss,
-                                       train_batch_size=batch_size,
-                                       test_batch_size=batch_size,
-                                       nb_epoch=nb_epoch)
-    sklearn_regressor.fit(X_train_reg, y_train_reg)
-    sklearn_regressor.score(X_test_reg, y_test_reg)
+
+class Class_build_fn_reg(object):
+    def __call__(self, hidden_dims):
+        return build_fn_reg(hidden_dims)
+
+
+class Inherit_class_build_fn_reg(KerasRegressor):
+    def __call__(self, hidden_dims):
+        return build_fn_reg(hidden_dims)
+
+for fn in [build_fn_clf, Class_build_fn_clf(), Inherit_class_build_fn_clf]:
+    if fn is Inherit_class_build_fn_clf:
+        classifier = Inherit_class_build_fn_clf(
+            build_fn=None, hidden_dims=50, batch_size=batch_size, nb_epoch=nb_epoch)
+    else:
+        classifier = KerasClassifier(
+            build_fn=fn, hidden_dims=50, batch_size=batch_size, nb_epoch=nb_epoch)
+
+    classifier.fit(X_train, y_train, batch_size=batch_size, nb_epoch=nb_epoch)
+    score = classifier.score(X_train, y_train, batch_size=batch_size)
+    preds = classifier.predict(X_test, batch_size=batch_size)
+    proba = classifier.predict_proba(X_test, batch_size=batch_size)
+
+
+for fn in [build_fn_reg, Class_build_fn_reg(), Inherit_class_build_fn_reg]:
+    if fn is Inherit_class_build_fn_reg:
+        regressor = Inherit_class_build_fn_reg(
+            build_fn=None, hidden_dims=50, batch_size=batch_size, nb_epoch=nb_epoch)
+    else:
+        regressor = KerasRegressor(
+            build_fn=fn, hidden_dims=50, batch_size=batch_size, nb_epoch=nb_epoch)
+
+    regressor.fit(X_train_reg, y_train_reg,
+                  batch_size=batch_size, nb_epoch=nb_epoch)
+    score = regressor.score(X_train_reg, y_train_reg, batch_size=batch_size)
+    preds = regressor.predict(X_test, batch_size=batch_size)
+
+
+# Usage of sklearn's grid_search
+# from sklearn import grid_search
+# parameters = dict(hidden_dims = [20, 30], batch_size=[64, 128], nb_epoch=[2], verbose=[0])
+# classifier = Inherit_class_build_fn_clf()
+# clf = grid_search.GridSearchCV(classifier, parameters)
+# clf.fit(X_train, y_train)
+# parameters = dict(hidden_dims = [20, 30], batch_size=[64, 128], nb_epoch=[2], verbose=[0])
+# regressor = Inherit_class_build_fn_reg()
+# reg = grid_search.GridSearchCV(regressor, parameters, scoring='mean_squared_error', n_jobs=1, cv=2, verbose=2)
+# reg.fit(X_train_reg, y_train_reg)

tests/test_loss_masking.py

@@ -7,15 +7,13 @@ from keras import objectives
 from keras import backend as K
 
 
-@pytest.mark.skipif(K._BACKEND == 'tensorflow',
-                    reason='currently not working with TensorFlow')
 def test_masking():
     np.random.seed(1337)
     X = np.array(
         [[[1, 1], [2, 1], [3, 1], [5, 5]],
          [[1, 5], [5, 0], [0, 0], [0, 0]]], dtype=np.int32)
     model = Sequential()
-    model.add(Masking(mask_value=0, input_shape=(None, 2)))
+    model.add(Masking(mask_value=0, input_shape=(4, 2)))
     model.add(TimeDistributedDense(1, init='one'))
     model.compile(loss='mse', optimizer='sgd')
     y = model.predict(X)

tests/test_loss_weighting.py

@@ -4,30 +4,30 @@ import pytest
 import numpy as np
 np.random.seed(1337)
 
-from keras.datasets import mnist
+from keras.utils.test_utils import get_test_data
 from keras.models import Sequential, Graph
-from keras.layers.core import Dense, Activation
+from keras.layers import Dense, Activation, RepeatVector, TimeDistributedDense, GRU
 from keras.utils import np_utils
 
 nb_classes = 10
 batch_size = 128
 nb_epoch = 15
-weighted_class = 9
+weighted_class = 5
 standard_weight = 1
-high_weight = 5
-max_train_samples = 5000
-max_test_samples = 1000
+high_weight = 10
+train_samples = 5000
+test_samples = 1000
+timesteps = 3
+input_dim = 10
+loss = 'mse'
 
-# the data, shuffled and split between tran and test sets
-(X_train, y_train), (X_test, y_test) = mnist.load_data()
-X_train = X_train.reshape(60000, 784)[:max_train_samples]
-X_test = X_test.reshape(10000, 784)[:max_test_samples]
-X_train = X_train.astype("float32") / 255
-X_test = X_test.astype("float32") / 255
+(X_train, y_train), (X_test, y_test) = get_test_data(nb_train=train_samples,
+                                                     nb_test=test_samples,
+                                                     input_shape=(input_dim,),
+                                                     classification=True,
+                                                     nb_class=nb_classes)
 
 # convert class vectors to binary class matrices
-y_train = y_train[:max_train_samples]
-y_test = y_test[:max_test_samples]
 Y_train = np_utils.to_categorical(y_train, nb_classes)
 Y_test = np_utils.to_categorical(y_test, nb_classes)
 test_ids = np.where(y_test == np.array(weighted_class))[0]
@@ -38,26 +38,60 @@ class_weight[weighted_class] = high_weight
 sample_weight = np.ones((y_train.shape[0])) * standard_weight
 sample_weight[y_train == weighted_class] = high_weight
 
+temporal_X_train = np.reshape(X_train, (len(X_train), 1, X_train.shape[1]))
+temporal_X_train = np.repeat(temporal_X_train, timesteps, axis=1)
+temporal_X_test = np.reshape(X_test, (len(X_test), 1, X_test.shape[1]))
+temporal_X_test = np.repeat(temporal_X_test, timesteps, axis=1)
+
+temporal_Y_train = np.reshape(Y_train, (len(Y_train), 1, Y_train.shape[1]))
+temporal_Y_train = np.repeat(temporal_Y_train, timesteps, axis=1)
+temporal_Y_test = np.reshape(Y_test, (len(Y_test), 1, Y_test.shape[1]))
+temporal_Y_test = np.repeat(temporal_Y_test, timesteps, axis=1)
+
+temporal_sample_weight = np.reshape(sample_weight, (len(sample_weight), 1))
+temporal_sample_weight = np.repeat(temporal_sample_weight, timesteps, axis=1)
+
 
 def create_sequential_model():
     model = Sequential()
-    model.add(Dense(50, input_shape=(784,)))
+    model.add(Dense(32, input_shape=(input_dim,)))
     model.add(Activation('relu'))
-    model.add(Dense(10))
+    model.add(Dense(nb_classes))
     model.add(Activation('softmax'))
     return model
 
 
 def create_graph_model():
     model = Graph()
-    model.add_input(name='input', input_shape=(784,))
-    model.add_node(Dense(50, activation='relu'), name='d1', input='input')
-    model.add_node(Dense(10, activation='softmax'), name='d2', input='d1')
+    model.add_input(name='input', input_shape=(input_dim,))
+    model.add_node(Dense(32, activation='relu'), name='d1', input='input')
+    model.add_node(Dense(nb_classes, activation='softmax'), name='d2', input='d1')
     model.add_output(name='output', input='d2')
     return model
 
 
-def _test_weights_sequential(model, class_weight=None, sample_weight=None):
+def create_temporal_sequential_model():
+    model = Sequential()
+    model.add(GRU(32, input_shape=(timesteps, input_dim), return_sequences=True))
+    model.add(TimeDistributedDense(nb_classes))
+    model.add(Activation('softmax'))
+    return model
+
+
+def create_temporal_graph_model():
+    model = Graph()
+    model.add_input(name='input', input_shape=(timesteps, input_dim))
+    model.add_node(GRU(32, return_sequences=True),
+                   name='d1', input='input')
+    model.add_node(TimeDistributedDense(nb_classes, activation='softmax'),
+                   name='d2', input='d1')
+    model.add_output(name='output', input='d2')
+    return model
+
+
+def _test_weights_sequential(model, class_weight=None, sample_weight=None,
+                             X_train=X_train, Y_train=Y_train,
+                             X_test=X_test, Y_test=Y_test):
     if sample_weight is not None:
         model.fit(X_train, Y_train, batch_size=batch_size,
                   nb_epoch=nb_epoch // 3, verbose=0,
@@ -88,7 +122,9 @@ def _test_weights_sequential(model, class_weight=None, sample_weight=None):
     return score
 
 
-def _test_weights_graph(model, class_weight=None, sample_weight=None):
+def _test_weights_graph(model, class_weight=None, sample_weight=None,
+                        X_train=X_train, Y_train=Y_train,
+                        X_test=X_test, Y_test=Y_test):
     model.fit({'input': X_train, 'output': Y_train},
               batch_size=batch_size, nb_epoch=nb_epoch // 2, verbose=0,
               class_weight={'output': class_weight},
@@ -109,40 +145,80 @@ def _test_weights_graph(model, class_weight=None, sample_weight=None):
     return score
 
 
-def test_sequential():
-    for loss in ['mae', 'mse']:
-        # no weights: reference point
-        model = create_sequential_model()
-        model.compile(loss=loss, optimizer='rmsprop')
-        standard_score = _test_weights_sequential(model)
-        # test class_weight
-        model = create_sequential_model()
-        model.compile(loss=loss, optimizer='rmsprop')
-        score = _test_weights_sequential(model, class_weight=class_weight)
-        assert(score < standard_score)
-        # test sample_weight
-        model = create_sequential_model()
-        model.compile(loss=loss, optimizer='rmsprop')
-        score = _test_weights_sequential(model, sample_weight=sample_weight)
-        assert(score < standard_score)
+# no weights: reference point
+model = create_sequential_model()
+model.compile(loss=loss, optimizer='rmsprop')
+standard_score_sequential = _test_weights_sequential(model)
+
+model = create_graph_model()
+model.compile(loss={'output': loss}, optimizer='rmsprop')
+standard_score_graph = _test_weights_graph(model)
 
 
-def test_graph():
-    for loss in ['mae', 'mse']:
-        # no weights: reference point
-        model = create_graph_model()
-        model.compile(loss={'output': loss}, optimizer='rmsprop')
-        standard_score = _test_weights_graph(model)
-        # test class_weight
-        model = create_graph_model()
-        model.compile(loss={'output': loss}, optimizer='rmsprop')
-        score = _test_weights_graph(model, class_weight=class_weight)
-        assert(score < standard_score)
-        # test sample_weight
-        model = create_graph_model()
-        model.compile(loss={'output': loss}, optimizer='rmsprop')
-        score = _test_weights_graph(model, sample_weight=sample_weight)
-        assert(score < standard_score)
+def test_sequential_class_weights():
+    model = create_sequential_model()
+    model.compile(loss=loss, optimizer='rmsprop')
+    score = _test_weights_sequential(model, class_weight=class_weight)
+    assert(score < standard_score_sequential)
+
+
+def test_sequential_sample_weights():
+    model = create_sequential_model()
+    model.compile(loss=loss, optimizer='rmsprop')
+    score = _test_weights_sequential(model, sample_weight=sample_weight)
+    assert(score < standard_score_sequential)
+
+
+def test_sequential_temporal_sample_weights():
+    model = create_temporal_sequential_model()
+    model.compile(loss=loss, optimizer='rmsprop',
+                  sample_weight_mode='temporal')
+    score = _test_weights_sequential(model,
+                                     sample_weight=temporal_sample_weight,
+                                     X_train=temporal_X_train,
+                                     X_test=temporal_X_test,
+                                     Y_train=temporal_Y_train,
+                                     Y_test=temporal_Y_test)
+    assert(score < standard_score_sequential)
+
+    # a twist: sample-wise weights with temporal output
+    model = create_temporal_sequential_model()
+    model.compile(loss=loss, optimizer='rmsprop',
+                  sample_weight_mode=None)
+    score = _test_weights_sequential(model,
+                                     sample_weight=sample_weight,
+                                     X_train=temporal_X_train,
+                                     X_test=temporal_X_test,
+                                     Y_train=temporal_Y_train,
+                                     Y_test=temporal_Y_test)
+    assert(score < standard_score_sequential)
+
+
+def test_graph_class_weights():
+    model = create_graph_model()
+    model.compile(loss={'output': loss}, optimizer='rmsprop')
+    score = _test_weights_graph(model, class_weight=class_weight)
+    assert(score < standard_score_graph)
+
+
+def test_graph_sample_weights():
+    model = create_graph_model()
+    model.compile(loss={'output': loss}, optimizer='rmsprop')
+    score = _test_weights_graph(model, sample_weight=sample_weight)
+    assert(score < standard_score_graph)
+
+
+def test_graph_temporal_sample_weight():
+    model = create_temporal_graph_model()
+    model.compile(loss={'output': loss}, optimizer='rmsprop',
+                  sample_weight_modes={'output': 'temporal'})
+    score = _test_weights_graph(model,
+                                sample_weight=temporal_sample_weight,
+                                X_train=temporal_X_train,
+                                X_test=temporal_X_test,
+                                Y_train=temporal_Y_train,
+                                Y_test=temporal_Y_test)
+    assert(score < standard_score_graph)
 
 
 if __name__ == '__main__':

tests/test_shape_inference.py

@@ -15,6 +15,7 @@ def check_layer_output_shape(layer, input_data):
 
     function = K.function([layer.input], [layer.get_output()])
     output = function([input_data])[0]
+
     assert output.shape[1:] == expected_output_shape
 
 
@@ -36,6 +37,7 @@ def test_Reshape():
     layer = Reshape(dims=(2, -1))
     check_layer_output_shape(layer, input_data)
 
+
 def test_Permute():
     layer = Permute(dims=(1, 3, 2))
     input_data = np.random.random((2, 2, 4, 3))
@@ -86,11 +88,11 @@ def test_Convolution1D():
 
 def test_Convolution2D():
     for border_mode in ['same', 'valid']:
-        for nb_row, nb_col in [(2, 2), (3, 3)]:
-            for subsample in [(1, 1), (2, 2)]:
-                if (subsample[0] > 1 or subsample[1] > 1) and border_mode == 'same':
+        for nb_row, nb_col in [(3, 3), (4, 4), (3, 4)]:
+            for subsample in [(1, 1), (2, 2), (3, 3)]:
+                if (subsample[0] > nb_row or subsample[1] > nb_col) and border_mode == 'same':
                     continue
-                for input_data_shape in [(2, 1, 3, 3), (2, 1, 4, 4)]:
+                for input_data_shape in [(2, 1, 5, 5), (2, 1, 6, 6)]:
                     layer = Convolution2D(nb_filter=1, nb_row=nb_row,
                                           nb_col=nb_row,
                                           border_mode=border_mode,
@@ -99,7 +101,7 @@ def test_Convolution2D():
                     input_data = np.random.random(input_data_shape)
                     check_layer_output_shape(layer, input_data)
 
-                for input_data_shape in [(2, 3, 3, 1)]:
+                for input_data_shape in [(2, 5, 5, 1)]:
                     layer = Convolution2D(nb_filter=1, nb_row=nb_row,
                                           nb_col=nb_row,
                                           border_mode=border_mode,