Prepare new PyPI release.

* make clipnorm work with embeddings layer

* test for embedding + clipnorm

* update embedding_with_clipnorm test nb_epoch to epochs

.github/stale.yml

@@ -0,0 +1,19 @@
+# Configuration for probot-stale - https://github.com/probot/stale
+
+# Number of days of inactivity before an Issue or Pull Request becomes stale
+daysUntilStale: 90
+# Number of days of inactivity before a stale Issue or Pull Request is closed
+daysUntilClose: 30
+# Issues or Pull Requests with these labels will never be considered stale
+exemptLabels:
+  - bug
+  - Announcement
+  - help wanted
+  - To investigate
+# Label to use when marking as stale
+staleLabel: stale
+# Comment to post when marking as stale. Set to `false` to disable
+markComment: >
+  This issue has been automatically marked as stale because it has not had
+  recent activity. It will be closed after 30 days if no further activity 
+  occurs, but feel free to re-open a closed issue if needed.

.travis.yml

@@ -17,6 +17,10 @@ matrix:
           env: KERAS_BACKEND=theano
         - python: 3.5
           env: KERAS_BACKEND=theano
+        - python: 2.7
+          env: KERAS_BACKEND=cntk
+        - python: 3.5
+          env: KERAS_BACKEND=cntk
 install:
   # code below is taken from http://conda.pydata.org/docs/travis.html
   # We do this conditionally because it saves us some downloading if the
@@ -49,6 +53,22 @@ install:
 
   # install TensorFlow (CPU version).
   - pip install tensorflow
+  
+  # install cntk
+  - if [[ "$TRAVIS_PYTHON_VERSION" == "2.7" ]]; then
+      pip install https://cntk.ai/PythonWheel/CPU-Only/cntk-2.0-cp27-cp27mu-linux_x86_64.whl;
+    elif [[ "$TRAVIS_PYTHON_VERSION" == "3.5" ]]; then
+      pip install https://cntk.ai/PythonWheel/CPU-Only/cntk-2.0-cp35-cp35m-linux_x86_64.whl;
+    fi
+
+  #install open mpi
+  - rm -rf ~/mpi
+  - mkdir ~/mpi
+  - pushd ~/mpi
+  - wget http://cntk.ai/PythonWheel/ForKeras/depends/openmpi_1.10-3.zip
+  - unzip ./openmpi_1.10-3.zip
+  - sudo dpkg -i openmpi_1.10-3.deb
+  - popd
 
 # command to run tests
 script:

CONTRIBUTING.md

@@ -19,6 +19,7 @@ To easily update Theano: `pip install git+git://github.com/Theano/Theano.git --u
 
 The more information you provide, the easier it is for us to validate that there is a bug and the faster we'll be able to take action. If you want your issue to be resolved quickly, following the steps above is crucial.
 
+---
 
 ## Requesting a Feature
 
@@ -31,11 +32,15 @@ You can also use Github issues to request features you would like to see in Kera
 3. After discussing the feature you may choose to attempt a Pull Request. If you're at all able, start writing some code. We always have more work to do than time to do it. If you can write some code then that will speed the process along.
 
 
+---
+
 ## Requests for Contributions
 
 [This is the board](https://github.com/fchollet/keras/projects/1) where we list current outstanding issues and features to be added. If you want to start contributing to Keras, this is the place to start.
 
 
+---
+
 ## Pull Requests
 
 **Where should I submit my pull request?**
@@ -49,27 +54,29 @@ Here's a quick guide to submitting your improvements:
 
 2. Write the code. This is the hard part!
 
-3. Make sure any new function or class you introduce has proper docstrings. Make sure any code you touch still has up-to-date docstrings and documentation.
+3. Make sure any new function or class you introduce has proper docstrings. Make sure any code you touch still has up-to-date docstrings and documentation. **Docstring style should be respected.** In particular, they should be formatted in MarkDown, and there should be sections for `Arguments`, `Returns`, `Raises` (if applicable). Look at other docstrings in the codebase for examples.
 
 4. Write tests. Your code should have full unit test coverage. If you want to see your PR merged promptly, this is crucial.
 
 5. Run our test suite locally. It's easy: from the Keras folder, simply run: `py.test tests/`.
-  - You will need to install the test requirements as well: `pip install -e .[tests]`.
+    - You will need to install the test requirements as well: `pip install -e .[tests]`.
 
 6. Make sure all tests are passing:
-  - with the Theano backend, on Python 2.7 and Python 3.5
-  - with the TensorFlow backend, on Python 2.7
+    - with the Theano backend, on Python 2.7 and Python 3.5. Make sure you have the development version of Theano.
+    - with the TensorFlow backend, on Python 2.7 and Python 3.5. Make sure you have the development version of TensorFlow.
 
 7. We use PEP8 syntax conventions, but we aren't dogmatic when it comes to line length. Make sure your lines stay reasonably sized, though. To make your life easier, we recommend running a PEP8 linter:
-  - Install PEP8 packages: `pip install pep8 pytest-pep8 autopep8`
-  - Run a standalone PEP8 check: `py.test --pep8 -m pep8`
-  - You can automatically fix some PEP8 error by running: `autopep8 -i --select <errors> <FILENAME>` for example: `autopep8 -i --select E128 tests/keras/backend/test_backends.py`
+    - Install PEP8 packages: `pip install pep8 pytest-pep8 autopep8`
+    - Run a standalone PEP8 check: `py.test --pep8 -m pep8`
+    - You can automatically fix some PEP8 error by running: `autopep8 -i --select <errors> <FILENAME>` for example: `autopep8 -i --select E128 tests/keras/backend/test_backends.py`
 
-8. When committing, use appropriate, descriptive commit messages. Make sure that your branch history is not a string of "bug fix", "fix", "oops", etc. When submitting your PR, squash your commits into a single commit with an appropriate commit message, to make sure the project history stays clean and readable. See ['rebase and squash'](http://rebaseandsqua.sh/) for technical help on how to squash your commits.
+8. When committing, use appropriate, descriptive commit messages.
 
 9. Update the documentation. If introducing new functionality, make sure you include code snippets demonstrating the usage of your new feature.
 
-10. Submit your PR. If your changes have been approved in a previous discussion, and if you have complete (and passing) unit tests, your PR is likely to be merged promptly. Otherwise, well...
+10. Submit your PR. If your changes have been approved in a previous discussion, and if you have complete (and passing) unit tests as well as proper docstrings/documentation, your PR is likely to be merged promptly. Otherwise, well...
+
+---
 
 ## Adding new examples
 

LICENSE

@@ -8,8 +8,12 @@ All contributions by Google:
 Copyright (c) 2015, Google, Inc.
 All rights reserved.
 
+All contributions by Microsoft:
+Copyright (c) 2017, Microsoft, Inc.
+All rights reserved.
+
 All other contributions:
-Copyright (c) 2015, the respective contributors.
+Copyright (c) 2015 - 2017, the respective contributors.
 All rights reserved.
 
 Each contributor holds copyright over their respective contributions.

README.md

@@ -1,11 +1,11 @@
-# Keras: Deep Learning library for TensorFlow and Theano
+# Keras: Deep Learning for Python
 
 [![Build Status](https://travis-ci.org/fchollet/keras.svg?branch=master)](https://travis-ci.org/fchollet/keras)
 [![license](https://img.shields.io/github/license/mashape/apistatus.svg?maxAge=2592000)](https://github.com/fchollet/keras/blob/master/LICENSE)
 
 ## You have just found Keras.
 
-Keras is a high-level neural networks API, written in Python and capable of running on top of either [TensorFlow](https://github.com/tensorflow/tensorflow) or [Theano](https://github.com/Theano/Theano). It was developed with a focus on enabling fast experimentation. *Being able to go from idea to result with the least possible delay is key to doing good research.*
+Keras is a high-level neural networks API, written in Python and capable of running on top of either [TensorFlow](https://github.com/tensorflow/tensorflow), [CNTK](https://github.com/Microsoft/cntk) or [Theano](https://github.com/Theano/Theano). It was developed with a focus on enabling fast experimentation. *Being able to go from idea to result with the least possible delay is key to doing good research.*
 
 Use Keras if you need a deep learning library that:
 
@@ -125,6 +125,11 @@ Keras uses the following dependencies:
 - TensorFlow
     - [See installation instructions](https://www.tensorflow.org/install/).
 
+*When using the CNTK backend:*
+
+- CNTK
+    - [See installation instructions](https://docs.microsoft.com/en-us/cognitive-toolkit/setup-cntk-on-your-machine).
+
 *When using the Theano backend:*
 
 - Theano
@@ -143,7 +148,7 @@ sudo pip install keras
 ------------------
 
 
-## Switching from TensorFlow to Theano
+## Switching from TensorFlow to CNTK or Theano
 
 By default, Keras will use TensorFlow as its tensor manipulation library. [Follow these instructions](http://keras.io/backend/) to configure the Keras backend.
 

docs/autogen.py

@@ -120,14 +120,13 @@ PAGES = [
             models.Sequential.fit,
             models.Sequential.evaluate,
             models.Sequential.predict,
-            models.Sequential.predict_classes,
-            models.Sequential.predict_proba,
             models.Sequential.train_on_batch,
             models.Sequential.test_on_batch,
             models.Sequential.predict_on_batch,
             models.Sequential.fit_generator,
             models.Sequential.evaluate_generator,
             models.Sequential.predict_generator,
+            models.Sequential.get_layer,
         ],
     },
     {
@@ -388,7 +387,7 @@ def process_class_docstring(docstring):
                        r'\n    __\1__\n\n',
                        docstring)
 
-    docstring = re.sub(r'    ([^\s\\]+):(.*)\n',
+    docstring = re.sub(r'    ([^\s\\\(]+):(.*)\n',
                        r'    - __\1__:\2\n',
                        docstring)
 
@@ -406,7 +405,7 @@ def process_function_docstring(docstring):
                        r'\n        __\1__\n\n',
                        docstring)
 
-    docstring = re.sub(r'    ([^\s\\]+):(.*)\n',
+    docstring = re.sub(r'    ([^\s\\\(]+):(.*)\n',
                        r'    - __\1__:\2\n',
                        docstring)
 

docs/templates/applications.md

@@ -130,10 +130,10 @@ for i, layer in enumerate(base_model.layers):
    print(i, layer.name)
 
 # we chose to train the top 2 inception blocks, i.e. we will freeze
-# the first 172 layers and unfreeze the rest:
-for layer in model.layers[:172]:
+# the first 249 layers and unfreeze the rest:
+for layer in model.layers[:249]:
    layer.trainable = False
-for layer in model.layers[172:]:
+for layer in model.layers[249:]:
    layer.trainable = True
 
 # we need to recompile the model for these modifications to take effect
@@ -253,7 +253,7 @@ The default input size for this model is 224x224.
 - input_shape: optional shape tuple, only to be specified
     if `include_top` is False (otherwise the input shape
     has to be `(224, 224, 3)` (with `channels_last` data format)
-    or `(3, 224, 244)` (with `channels_first` data format).
+    or `(3, 224, 224)` (with `channels_first` data format).
     It should have exactly 3 inputs channels,
     and width and height should be no smaller than 48.
     E.g. `(200, 200, 3)` would be one valid value.
@@ -309,7 +309,7 @@ The default input size for this model is 224x224.
 - input_shape: optional shape tuple, only to be specified
     if `include_top` is False (otherwise the input shape
     has to be `(224, 224, 3)` (with `channels_last` data format)
-    or `(3, 224, 244)` (with `channels_first` data format).
+    or `(3, 224, 224)` (with `channels_first` data format).
     It should have exactly 3 inputs channels,
     and width and height should be no smaller than 48.
     E.g. `(200, 200, 3)` would be one valid value.
@@ -367,7 +367,7 @@ The default input size for this model is 224x224.
 - input_shape: optional shape tuple, only to be specified
     if `include_top` is False (otherwise the input shape
     has to be `(224, 224, 3)` (with `channels_last` data format)
-    or `(3, 224, 244)` (with `channels_first` data format).
+    or `(3, 224, 224)` (with `channels_first` data format).
     It should have exactly 3 inputs channels,
     and width and height should be no smaller than 197.
     E.g. `(200, 200, 3)` would be one valid value.

docs/templates/backend.md

@@ -4,12 +4,13 @@
 
 Keras is a model-level library, providing high-level building blocks for developing deep learning models. It does not handle itself low-level operations such as tensor products, convolutions and so on. Instead, it relies on a specialized, well-optimized tensor manipulation library to do so, serving as the "backend engine" of Keras. Rather than picking one single tensor library and making the implementation of Keras tied to that library, Keras handles the problem in a modular way, and several different backend engines can be plugged seamlessly into Keras.
 
-At this time, Keras has two backend implementations available: the **TensorFlow** backend and the **Theano** backend.
+At this time, Keras has three backend implementations available: the **TensorFlow** backend, the **Theano** backend, and the **CNTK** backend.
 
 - [TensorFlow](http://www.tensorflow.org/) is an open-source symbolic tensor manipulation framework developed by Google, Inc.
 - [Theano](http://deeplearning.net/software/theano/) is an open-source symbolic tensor manipulation framework developed by LISA/MILA Lab at Université de Montréal.
+- [CNTK](https://www.microsoft.com/en-us/cognitive-toolkit/) is an open-source, commercial-grade toolkit for deep learning developed by Microsoft.
 
-In the future, we are likely to add more backend options. Go ask Microsoft about how their CNTK backend project is doing.
+In the future, we are likely to add more backend options.
 
 ----
 
@@ -34,7 +35,7 @@ The default configuration file looks like this:
 }
 ```
 
-Simply change the field `backend` to either `"theano"` or `"tensorflow"`, and Keras will use the new configuration next time you run any Keras code.
+Simply change the field `backend` to `"theano"`, `"tensorflow"`, or `"cntk"`, and Keras will use the new configuration next time you run any Keras code.
 
 You can also define the environment variable ``KERAS_BACKEND`` and this will
 override what is defined in your config file :
@@ -65,7 +66,7 @@ You can change these settings by editing `$HOME/.keras/keras.json`.
   - For 3D data, `"channels_last"` assumes `(conv_dim1, conv_dim2, conv_dim3, channels)` while `"channels_first"` assumes `(channels, conv_dim1, conv_dim2, conv_dim3)`.
 * `epsilon`: float, a numeric fuzzing constant used to avoid dividing by zero in some operations.
 * `floatx`: string, `"float16"`, `"float32"`, or `"float64"`. Default float precision.
-* `backend`: string, `"tensorflow"` or `"theano"`.
+* `backend`: string, `"tensorflow"`, `"theano"`, or `"cntk"`.
 
 ----
 

docs/templates/callbacks.md

@@ -41,9 +41,9 @@ model.add(Activation('softmax'))
 model.compile(loss='categorical_crossentropy', optimizer='rmsprop')
 
 history = LossHistory()
-model.fit(X_train, Y_train, batch_size=128, epochs=20, verbose=0, callbacks=[history])
+model.fit(x_train, y_train, batch_size=128, epochs=20, verbose=0, callbacks=[history])
 
-print history.losses
+print(history.losses)
 # outputs
 '''
 [0.66047596406559383, 0.3547245744908703, ..., 0.25953155204159617, 0.25901699725311789]
@@ -65,8 +65,6 @@ model.compile(loss='categorical_crossentropy', optimizer='rmsprop')
 '''
 saves the model weights after each epoch if the validation loss decreased
 '''
-checkpointer = ModelCheckpoint(filepath="/tmp/weights.hdf5", verbose=1, save_best_only=True)
-model.fit(X_train, Y_train, batch_size=128, epochs=20, verbose=0, validation_data=(X_test, Y_test), callbacks=[checkpointer])
-
+checkpointer = ModelCheckpoint(filepath='/tmp/weights.hdf5', verbose=1, save_best_only=True)
+model.fit(x_train, y_train, batch_size=128, epochs=20, verbose=0, validation_data=(X_test, Y_test), callbacks=[checkpointer])
 ```
-

docs/templates/getting-started/faq.md

@@ -38,7 +38,7 @@ Please cite Keras in your publications if it helps your research. Here is an exa
 
 ### How can I run Keras on GPU?
 
-If you are running on the TensorFlow backend, your code will automatically run on GPU if any available GPU is detected.
+If you are running on the TensorFlow or CNTK backends, your code will automatically run on GPU if any available GPU is detected.
 
 If you are running on the Theano backend, you can use one of the following methods:
 
@@ -153,16 +153,16 @@ For example:
 """
 Assume original model looks like this:
     model = Sequential()
-    model.add(Dense(2, input_dim=3, name="dense_1"))
-    model.add(Dense(3, name="dense_2"))
+    model.add(Dense(2, input_dim=3, name='dense_1'))
+    model.add(Dense(3, name='dense_2'))
     ...
     model.save_weights(fname)
 """
 
 # new model
 model = Sequential()
-model.add(Dense(2, input_dim=3, name="dense_1"))  # will be loaded
-model.add(Dense(10, name="new_dense"))  # will not be loaded
+model.add(Dense(2, input_dim=3, name='dense_1'))  # will be loaded
+model.add(Dense(10, name='new_dense'))  # will not be loaded
 
 # load weights from first model; will only affect the first layer, dense_1.
 model.load_weights(fname, by_name=True)
@@ -201,7 +201,7 @@ from keras import backend as K
 # with a Sequential model
 get_3rd_layer_output = K.function([model.layers[0].input],
                                   [model.layers[3].output])
-layer_output = get_3rd_layer_output([X])[0]
+layer_output = get_3rd_layer_output([x])[0]
 ```
 
 Similarly, you could build a Theano and TensorFlow function directly.
@@ -214,17 +214,17 @@ get_3rd_layer_output = K.function([model.layers[0].input, K.learning_phase()],
                                   [model.layers[3].output])
 
 # output in test mode = 0
-layer_output = get_3rd_layer_output([X, 0])[0]
+layer_output = get_3rd_layer_output([x, 0])[0]
 
 # output in train mode = 1
-layer_output = get_3rd_layer_output([X, 1])[0]
+layer_output = get_3rd_layer_output([x, 1])[0]
 ```
 
 ---
 
 ### How can I use Keras with datasets that don't fit in memory?
 
-You can do batch training using `model.train_on_batch(X, y)` and `model.test_on_batch(X, y)`. See the [models documentation](/models/sequential).
+You can do batch training using `model.train_on_batch(x, y)` and `model.test_on_batch(x, y)`. See the [models documentation](/models/sequential).
 
 Alternatively, you can write a generator that yields batches of training data and use the method `model.fit_generator(data_generator, steps_per_epoch, epochs)`.
 
@@ -239,7 +239,7 @@ You can use an `EarlyStopping` callback:
 ```python
 from keras.callbacks import EarlyStopping
 early_stopping = EarlyStopping(monitor='val_loss', patience=2)
-model.fit(X, y, validation_split=0.2, callbacks=[early_stopping])
+model.fit(x, y, validation_split=0.2, callbacks=[early_stopping])
 ```
 
 Find out more in the [callbacks documentation](/callbacks).
@@ -268,7 +268,7 @@ Validation data is never shuffled.
 The `model.fit` method returns an `History` callback, which has a `history` attribute containing the lists of successive losses and other metrics.
 
 ```python
-hist = model.fit(X, y, validation_split=0.2)
+hist = model.fit(x, y, validation_split=0.2)
 print(hist.history)
 ```
 
@@ -315,7 +315,7 @@ Making a RNN stateful means that the states for the samples of each batch will b
 When using stateful RNNs, it is therefore assumed that:
 
 - all batches have the same number of samples
-- If `X1` and `X2` are successive batches of samples, then `X2[i]` is the follow-up sequence to `X1[i]`, for every `i`.
+- If `x1` and `x2` are successive batches of samples, then `x2[i]` is the follow-up sequence to `x1[i]`, for every `i`.
 
 To use statefulness in RNNs, you need to:
 
@@ -332,7 +332,7 @@ Example:
 
 ```python
 
-X  # this is our input data, of shape (32, 21, 16)
+x  # this is our input data, of shape (32, 21, 16)
 # we will feed it to our model in sequences of length 10
 
 model = Sequential()
@@ -342,10 +342,10 @@ model.add(Dense(16, activation='softmax'))
 model.compile(optimizer='rmsprop', loss='categorical_crossentropy')
 
 # we train the network to predict the 11th timestep given the first 10:
-model.train_on_batch(X[:, :10, :], np.reshape(X[:, 10, :], (32, 16)))
+model.train_on_batch(x[:, :10, :], np.reshape(x[:, 10, :], (32, 16)))
 
 # the state of the network has changed. We can feed the follow-up sequences:
-model.train_on_batch(X[:, 10:20, :], np.reshape(X[:, 20, :], (32, 16)))
+model.train_on_batch(x[:, 10:20, :], np.reshape(x[:, 20, :], (32, 16)))
 
 # let's reset the states of the LSTM layer:
 model.reset_states()
@@ -418,8 +418,8 @@ You can also directly use a HDF5 dataset:
 ```python
 import h5py
 with h5py.File('input/file.hdf5', 'r') as f:
-    X_data = f['X_data']
-    model.predict(X_data)
+    x_data = f['x_data']
+    model.predict(x_data)
 ```
 
 ---

docs/templates/getting-started/functional-api-guide.md

@@ -361,7 +361,7 @@ from keras.models import Model, Sequential
 # First, let's define a vision model using a Sequential model.
 # This model will encode an image into a vector.
 vision_model = Sequential()
-vision_model.add(Conv2D(64, (3, 3) activation='relu', padding='same', input_shape=(3, 224, 224)))
+vision_model.add(Conv2D(64, (3, 3), activation='relu', padding='same', input_shape=(3, 224, 224)))
 vision_model.add(Conv2D(64, (3, 3), activation='relu'))
 vision_model.add(MaxPooling2D((2, 2)))
 vision_model.add(Conv2D(128, (3, 3), activation='relu', padding='same'))

docs/templates/index.md

@@ -1,3 +1,3 @@
-# Keras: Deep Learning library for Theano and TensorFlow
+# Keras: The Python Deep Learning library
 
 {{autogenerated}}

docs/templates/layers/writing-your-own-keras-layers.md

@@ -21,7 +21,8 @@ class MyLayer(Layer):
 
     def build(self, input_shape):
         # Create a trainable weight variable for this layer.
-        self.kernel = self.add_weight(shape=(input_shape[1], self.output_dim),
+        self.kernel = self.add_weight(name='kernel', 
+                                      shape=(input_shape[1], self.output_dim),
                                       initializer='uniform',
                                       trainable=True)
         super(MyLayer, self).build(input_shape)  # Be sure to call this somewhere!

docs/templates/preprocessing/image.md

@@ -7,6 +7,7 @@ keras.preprocessing.image.ImageDataGenerator(featurewise_center=False,
     featurewise_std_normalization=False,
     samplewise_std_normalization=False,
     zca_whitening=False,
+    zca_epsilon=1e-6,
     rotation_range=0.,
     width_shift_range=0.,
     height_shift_range=0.,
@@ -29,6 +30,7 @@ Generate batches of tensor image data with real-time data augmentation. The data
     - __samplewise_center__: Boolean. Set each sample mean to 0.
     - __featurewise_std_normalization__: Boolean. Divide inputs by std of the dataset, feature-wise.
     - __samplewise_std_normalization__: Boolean. Divide each input by its std.
+    - __zca_epsilon__: epsilon for ZCA whitening. Default is 1e-6.
     - __zca_whitening__: Boolean. Apply ZCA whitening.
     - __rotation_range__: Int. Degree range for random rotations.
     - __width_shift_range__: Float (fraction of total width). Range for random horizontal shifts.
@@ -66,7 +68,7 @@ Generate batches of tensor image data with real-time data augmentation. The data
             - __augment__: Boolean (default: False). Whether to fit on randomly augmented samples.
             - __rounds__: int (default: 1). If augment, how many augmentation passes over the data to use.
             - __seed__: int (default: None). Random seed.
-    - __flow(X, y)__: Takes numpy data & label arrays, and generates batches of augmented/normalized data. Yields batches indefinitely, in an infinite loop.
+    - __flow(x, y)__: Takes numpy data & label arrays, and generates batches of augmented/normalized data. Yields batches indefinitely, in an infinite loop.
         - __Arguments__:
             - __x__: data. Should have rank 4.
                 In case of grayscale data,
@@ -78,7 +80,7 @@ Generate batches of tensor image data with real-time data augmentation. The data
             - __seed__: int (default: None).
             - __save_to_dir__: None or str (default: None). This allows you to optimally specify a directory to which to save the augmented pictures being generated (useful for visualizing what you are doing).
             - __save_prefix__: str (default: `''`). Prefix to use for filenames of saved pictures (only relevant if `save_to_dir` is set).
-            - __save_format__: one of "png", "jpeg" (only relevant if `save_to_dir` is set). Default: "jpeg".
+            - __save_format__: one of "png", "jpeg" (only relevant if `save_to_dir` is set). Default: "png".
         - __yields__: Tuples of `(x, y)` where `x` is a numpy array of image data and `y` is a numpy array of corresponding labels.
             The generator loops indefinitely.
     - __flow_from_directory(directory)__: Takes the path to a directory, and generates batches of augmented/normalized data. Yields batches indefinitely, in an infinite loop.
@@ -95,7 +97,7 @@ Generate batches of tensor image data with real-time data augmentation. The data
             - __seed__: optional random seed for shuffling and transformations.
             - __save_to_dir__: None or str (default: None). This allows you to optimally specify a directory to which to save the augmented pictures being generated (useful for visualizing what you are doing).
             - __save_prefix__: str. Prefix to use for filenames of saved pictures (only relevant if `save_to_dir` is set).
-            - __save_format__: one of "png", "jpeg" (only relevant if `save_to_dir` is set). Default: "jpeg".
+            - __save_format__: one of "png", "jpeg" (only relevant if `save_to_dir` is set). Default: "png".
             - __follow_links__: whether to follow symlinks inside class subdirectories (default: False).
 
 

docs/templates/preprocessing/text.md

@@ -3,7 +3,7 @@
 
 ```python
 keras.preprocessing.text.text_to_word_sequence(text, 
-    filters=base_filter(), lower=True, split=" ")
+    filters='!"#$%&()*+,-./:;<=>?@[\\]^_`{|}~\t\n', lower=True, split=" ")
 ```
 
 Split a sentence into a list of words.
@@ -12,7 +12,7 @@ Split a sentence into a list of words.
 
 - __Arguments__:
     - __text__: str.
-    - __filters__: list (or concatenation) of characters to filter out, such as punctuation. Default: base_filter(), includes basic punctuation, tabs, and newlines.
+    - __filters__: list (or concatenation) of characters to filter out, such as punctuation. Default: '!"#$%&()*+,-./:;<=>?@[\\]^_`{|}~\t\n' , includes basic punctuation, tabs, and newlines.
     - __lower__: boolean. Whether to set the text to lowercase.
     - __split__: str. Separator for word splitting.
 
@@ -20,7 +20,7 @@ Split a sentence into a list of words.
 
 ```python
 keras.preprocessing.text.one_hot(text, n,
-    filters=base_filter(), lower=True, split=" ")
+    filters='!"#$%&()*+,-./:;<=>?@[\\]^_`{|}~\t\n', lower=True, split=" ")
 ```
 
 One-hot encode a text into a list of word indexes in a vocabulary of size n.
@@ -33,14 +33,15 @@ One-hot encode a text into a list of word indexes in a vocabulary of size n.
 ## Tokenizer
 
 ```python
-keras.preprocessing.text.Tokenizer(num_words=None, filters=base_filter(), 
-    lower=True, split=" ")
+keras.preprocessing.text.Tokenizer(num_words=None, filters='!"#$%&()*+,-./:;<=>?@[\\]^_`{|}~\t\n', 
+    lower=True, split=" ", char_level=False)
 ```
 
 Class for vectorizing texts, or/and turning texts into sequences (=list of word indexes, where the word of rank i in the dataset (starting at 1) has index i).
 
 - __Arguments__: Same as `text_to_word_sequence` above.
     - __num_words__: None or int. Maximum number of words to work with (if set, tokenization will be restricted to the top num_words most common words in the dataset).
+    - __char_level__: if True, every character will be treated as a token.
 
 - __Methods__:
 

examples/imdb_bidirectional_lstm.py

@@ -24,7 +24,7 @@ print('Loading data...')
 print(len(x_train), 'train sequences')
 print(len(x_test), 'test sequences')
 
-print("Pad sequences (samples x time)")
+print('Pad sequences (samples x time)')
 x_train = sequence.pad_sequences(x_train, maxlen=maxlen)
 x_test = sequence.pad_sequences(x_test, maxlen=maxlen)
 print('x_train shape:', x_train.shape)

examples/neural_doodle.py

@@ -196,8 +196,8 @@ x = mask_input
 for layer in image_model.layers[1:]:
     name = 'mask_%s' % layer.name
     if 'conv' in layer.name:
-        x = AveragePooling2D((3, 3), strides=(
-            1, 1), name=name, border_mode='same')(x)
+        x = AveragePooling2D((3, 3), padding='same', strides=(
+            1, 1), name=name)(x)
     elif 'pool' in layer.name:
         x = AveragePooling2D((2, 2), name=name)(x)
 mask_model = Model(mask_input, x)
@@ -238,6 +238,7 @@ def region_style_loss(style_image, target_image, style_mask, target_mask):
         masked_target = K.permute_dimensions(
             target_image, (2, 0, 1)) * target_mask
         num_channels = K.shape(style_image)[-1]
+    num_channels = K.cast(num_channels, dtype='float32')
     s = gram_matrix(masked_style) / K.mean(style_mask) / num_channels
     c = gram_matrix(masked_target) / K.mean(target_mask) / num_channels
     return K.mean(K.square(s - c))

examples/neural_style_transfer.py

@@ -57,7 +57,7 @@ from scipy.optimize import fmin_l_bfgs_b
 import time
 import argparse
 
-from keras.applications import vgg16
+from keras.applications import vgg19
 from keras import backend as K
 
 parser = argparse.ArgumentParser(description='Neural style transfer with Keras.')
@@ -99,7 +99,7 @@ def preprocess_image(image_path):
     img = load_img(image_path, target_size=(img_nrows, img_ncols))
     img = img_to_array(img)
     img = np.expand_dims(img, axis=0)
-    img = vgg16.preprocess_input(img)
+    img = vgg19.preprocess_input(img)
     return img
 
 # util function to convert a tensor into a valid image
@@ -137,7 +137,7 @@ input_tensor = K.concatenate([base_image,
 
 # build the VGG16 network with our 3 images as input
 # the model will be loaded with pre-trained ImageNet weights
-model = vgg16.VGG16(input_tensor=input_tensor,
+model = vgg19.VGG19(input_tensor=input_tensor,
                     weights='imagenet', include_top=False)
 print('Model loaded.')
 
@@ -199,7 +199,7 @@ def total_variation_loss(x):
 
 # combine these loss functions into a single scalar
 loss = K.variable(0.)
-layer_features = outputs_dict['block4_conv2']
+layer_features = outputs_dict['block5_conv2']
 base_image_features = layer_features[0, :, :, :]
 combination_features = layer_features[2, :, :, :]
 loss += content_weight * content_loss(base_image_features,
@@ -273,10 +273,7 @@ evaluator = Evaluator()
 
 # run scipy-based optimization (L-BFGS) over the pixels of the generated image
 # so as to minimize the neural style loss
-if K.image_data_format() == 'channels_first':
-    x = np.random.uniform(0, 255, (1, 3, img_nrows, img_ncols)) - 128.
-else:
-    x = np.random.uniform(0, 255, (1, img_nrows, img_ncols, 3)) - 128.
+x = preprocess_image(base_image_path)
 
 for i in range(iterations):
     print('Start of iteration', i)

keras/__init__.py

@@ -20,4 +20,4 @@ from . import regularizers
 # Importable from root because it's technically not a layer
 from .layers import Input
 
-__version__ = '2.0.4'
+__version__ = '2.0.5'

keras/applications/inception_v3.py

@@ -27,7 +27,6 @@ from ..layers import AveragePooling2D
 from ..layers import GlobalAveragePooling2D
 from ..layers import GlobalMaxPooling2D
 from ..engine.topology import get_source_inputs
-from ..utils.layer_utils import convert_all_kernels_in_model
 from ..utils.data_utils import get_file
 from .. import backend as K
 from .imagenet_utils import decode_predictions
@@ -384,8 +383,6 @@ def InceptionV3(include_top=True,
                 cache_subdir='models',
                 md5_hash='bcbd6486424b2319ff4ef7d526e38f63')
         model.load_weights(weights_path)
-        if K.backend() == 'theano':
-            convert_all_kernels_in_model(model)
     return model
 
 

keras/applications/resnet50.py

@@ -43,7 +43,7 @@ def identity_block(input_tensor, kernel_size, filters, stage, block):
 
     # Arguments
         input_tensor: input tensor
-        kernel_size: defualt 3, the kernel size of middle conv layer at main path
+        kernel_size: default 3, the kernel size of middle conv layer at main path
         filters: list of integers, the filterss of 3 conv layer at main path
         stage: integer, current stage label, used for generating layer names
         block: 'a','b'..., current block label, used for generating layer names
@@ -81,7 +81,7 @@ def conv_block(input_tensor, kernel_size, filters, stage, block, strides=(2, 2))
 
     # Arguments
         input_tensor: input tensor
-        kernel_size: defualt 3, the kernel size of middle conv layer at main path
+        kernel_size: default 3, the kernel size of middle conv layer at main path
         filters: list of integers, the filterss of 3 conv layer at main path
         stage: integer, current stage label, used for generating layer names
         block: 'a','b'..., current block label, used for generating layer names
@@ -264,21 +264,19 @@ def ResNet50(include_top=True, weights='imagenet',
         model.load_weights(weights_path)
         if K.backend() == 'theano':
             layer_utils.convert_all_kernels_in_model(model)
-
-        if K.image_data_format() == 'channels_first':
             if include_top:
                 maxpool = model.get_layer(name='avg_pool')
                 shape = maxpool.output_shape[1:]
                 dense = model.get_layer(name='fc1000')
                 layer_utils.convert_dense_weights_data_format(dense, shape, 'channels_first')
 
-            if K.backend() == 'tensorflow':
-                warnings.warn('You are using the TensorFlow backend, yet you '
-                              'are using the Theano '
-                              'image data format convention '
-                              '(`image_data_format="channels_first"`). '
-                              'For best performance, set '
-                              '`image_data_format="channels_last"` in '
-                              'your Keras config '
-                              'at ~/.keras/keras.json.')
+        if K.image_data_format() == 'channels_first' and K.backend() == 'tensorflow':
+            warnings.warn('You are using the TensorFlow backend, yet you '
+                          'are using the Theano '
+                          'image data format convention '
+                          '(`image_data_format="channels_first"`). '
+                          'For best performance, set '
+                          '`image_data_format="channels_last"` in '
+                          'your Keras config '
+                          'at ~/.keras/keras.json.')
     return model

keras/applications/vgg16.py

@@ -59,7 +59,7 @@ def VGG16(include_top=True, weights='imagenet',
         input_shape: optional shape tuple, only to be specified
             if `include_top` is False (otherwise the input shape
             has to be `(224, 224, 3)` (with `channels_last` data format)
-            or `(3, 224, 244)` (with `channels_first` data format).
+            or `(3, 224, 224)` (with `channels_first` data format).
             It should have exactly 3 inputs channels,
             and width and height should be no smaller than 48.
             E.g. `(200, 200, 3)` would be one valid value.

keras/applications/vgg19.py

@@ -59,7 +59,7 @@ def VGG19(include_top=True, weights='imagenet',
         input_shape: optional shape tuple, only to be specified
             if `include_top` is False (otherwise the input shape
             has to be `(224, 224, 3)` (with `channels_last` data format)
-            or `(3, 224, 244)` (with `channels_first` data format).
+            or `(3, 224, 224)` (with `channels_first` data format).
             It should have exactly 3 inputs channels,
             and width and height should be no smaller than 48.
             E.g. `(200, 200, 3)` would be one valid value.

keras/backend/__init__.py

@@ -10,7 +10,6 @@ from .common import set_floatx
 from .common import cast_to_floatx
 from .common import image_data_format
 from .common import set_image_data_format
-from .common import is_keras_tensor
 
 # Obtain Keras base dir path: either ~/.keras or /tmp.
 _keras_base_dir = os.path.expanduser('~')
@@ -33,7 +32,7 @@ if os.path.exists(_config_path):
     _epsilon = _config.get('epsilon', epsilon())
     assert isinstance(_epsilon, float)
     _backend = _config.get('backend', _BACKEND)
-    assert _backend in {'theano', 'tensorflow'}
+    assert _backend in {'theano', 'tensorflow', 'cntk'}
     _image_data_format = _config.get('image_data_format',
                                      image_data_format())
     assert _image_data_format in {'channels_last', 'channels_first'}
@@ -44,28 +43,39 @@ if os.path.exists(_config_path):
     _BACKEND = _backend
 
 # Save config file, if possible.
-if os.access(_keras_base_dir, os.W_OK):
-    if not os.path.exists(_keras_dir):
-        try:
-            os.makedirs(_keras_dir)
-        except OSError:
-            pass
-    if not os.path.exists(_config_path):
-        _config = {'floatx': floatx(),
-                   'epsilon': epsilon(),
-                   'backend': _BACKEND,
-                   'image_data_format': image_data_format()}
+if not os.path.exists(_keras_dir):
+    try:
+        os.makedirs(_keras_dir)
+    except OSError:
+        # Except permission denied and potential race conditions
+        # in multi-threaded environments.
+        pass
+
+if not os.path.exists(_config_path):
+    _config = {
+        'floatx': floatx(),
+        'epsilon': epsilon(),
+        'backend': _BACKEND,
+        'image_data_format': image_data_format()
+    }
+    try:
         with open(_config_path, 'w') as f:
             f.write(json.dumps(_config, indent=4))
+    except IOError:
+        # Except permission denied.
+        pass
 
 # Set backend based on KERAS_BACKEND flag, if applicable.
 if 'KERAS_BACKEND' in os.environ:
     _backend = os.environ['KERAS_BACKEND']
-    assert _backend in {'theano', 'tensorflow'}
+    assert _backend in {'theano', 'tensorflow', 'cntk'}
     _BACKEND = _backend
 
 # Import backend functions.
-if _BACKEND == 'theano':
+if _BACKEND == 'cntk':
+    sys.stderr.write('Using CNTK backend\n')
+    from .cntk_backend import *
+elif _BACKEND == 'theano':
     sys.stderr.write('Using Theano backend.\n')
     from .theano_backend import *
 elif _BACKEND == 'tensorflow':

keras/backend/common.py

@@ -145,35 +145,6 @@ def set_image_data_format(data_format):
     _IMAGE_DATA_FORMAT = str(data_format)
 
 
-def is_keras_tensor(x):
-    """Returns whether `x` is a Keras tensor.
-
-    # Arguments
-        x: a potential tensor.
-
-    # Returns
-        A boolean: whether the argument is a Keras tensor.
-
-    # Examples
-    ```python
-        >>> from keras import backend as K
-        >>> np_var = numpy.array([1, 2])
-        >>> K.is_keras_tensor(np_var)
-        False
-        >>> keras_var = K.variable(np_var)
-        >>> K.is_keras_tensor(keras_var)  # A variable is not a Tensor.
-        False
-        >>> keras_placeholder = K.placeholder(shape=(2, 4, 5))
-        >>> K.is_keras_tensor(keras_placeholder)  # A placeholder is a Tensor.
-        True
-    ```
-    """
-    if hasattr(x, '_keras_shape'):
-        return True
-    else:
-        return False
-
-
 # Legacy methods
 
 def set_image_dim_ordering(dim_ordering):
@@ -193,7 +164,7 @@ def set_image_dim_ordering(dim_ordering):
     ```
 
     # Raises
-        ValueError if invalid `dim_ordering`
+        ValueError: if `dim_ordering` is invalid.
     """
     global _IMAGE_DATA_FORMAT
     if dim_ordering not in {'tf', 'th'}:

keras/backend/tensorflow_backend.py

@@ -4,17 +4,20 @@ from tensorflow.python.ops import tensor_array_ops
 from tensorflow.python.ops import control_flow_ops
 from tensorflow.python.ops import functional_ops
 from tensorflow.python.ops import ctc_ops as ctc
+from tensorflow.python.ops import variables as tf_variables
 
 from collections import defaultdict
+import inspect
 import numpy as np
 import os
-import warnings
 
 from .common import floatx
 from .common import _EPSILON
 from .common import image_data_format
+
 # Legacy functions
-from .common import set_image_dim_ordering, image_dim_ordering
+from .common import set_image_dim_ordering
+from .common import image_dim_ordering
 
 py_all = all
 py_sum = sum
@@ -183,11 +186,11 @@ def _convert_string_dtype(dtype):
     # Arguments
         dtype: A string representation of a type.
 
-    # Returns:
+    # Returns
         The type requested.
 
     # Raises
-        ValueError if `dtype` is not supported
+        ValueError: if `dtype` is not supported.
     """
     if dtype == 'float16':
         return tf.float16
@@ -354,6 +357,43 @@ def constant(value, dtype=None, shape=None, name=None):
     return tf.constant(value, dtype=dtype, shape=shape, name=name)
 
 
+def is_keras_tensor(x):
+    """Returns whether `x` is a Keras tensor.
+
+    # Arguments
+        x: a potential tensor.
+
+    # Returns
+        A boolean: whether the argument is a Keras tensor.
+
+    # Raises
+        ValueError: in case `x` is not a symbolic tensor.
+
+    # Examples
+    ```python
+        >>> from keras import backend as K
+        >>> np_var = numpy.array([1, 2])
+        >>> K.is_keras_tensor(np_var) # A numpy array is not a symbolic yensor.
+        ValueError
+        >>> k_var = tf.placeholder('float32', shape=(1,1))
+        >>> K.is_keras_tensor(k_var) # A variable created directly from tensorflow/theano is not a Keras tensor.
+        False
+        >>> keras_var = K.variable(np_var)
+        >>> K.is_keras_tensor(keras_var)  # A variable created with the keras backend is a Keras tensor.
+        True
+        >>> keras_placeholder = K.placeholder(shape=(2, 4, 5))
+        >>> K.is_keras_tensor(keras_placeholder)  # A placeholder is a Keras tensor.
+        True
+    ```
+    """
+    if not isinstance(x, (tf.Tensor,
+                          tf_variables.Variable,
+                          tf.SparseTensor)):
+        raise ValueError('Unexpectedly found an instance of type `' + str(type(x)) + '`. '
+                         'Expected a symbolic tensor instance.')
+    return hasattr(x, '_keras_history')
+
+
 def placeholder(shape=None, ndim=None, dtype=None, sparse=False, name=None):
     """Instantiates a placeholder tensor and returns it.
 
@@ -828,26 +868,26 @@ def update(x, new_x):
 def update_add(x, increment):
     """Update the value of `x` by adding `increment`.
 
-        # Arguments
-            x: A Variable.
-            increment: A tensor of same shape as `x`.
+    # Arguments
+        x: A Variable.
+        increment: A tensor of same shape as `x`.
 
-        # Returns
-            The variable `x` updated.
-        """
+    # Returns
+        The variable `x` updated.
+    """
     return tf.assign_add(x, increment)
 
 
 def update_sub(x, decrement):
     """Update the value of `x` by subtracting `decrement`.
 
-        # Arguments
-            x: A Variable.
-            decrement: A tensor of same shape as `x`.
+    # Arguments
+        x: A Variable.
+        decrement: A tensor of same shape as `x`.
 
-        # Returns
-            The variable `x` updated.
-        """
+    # Returns
+        The variable `x` updated.
+    """
     return tf.assign_sub(x, decrement)
 
 
@@ -1399,7 +1439,7 @@ def logsumexp(x, axis=None, keepdims=False):
         The reduced tensor.
     """
     axis = _normalize_axis(axis, ndim(x))
-    return tf.reduce_logsumexp(x, reduction_indices=axis, keep_dims=keepdims)
+    return tf.reduce_logsumexp(x, axis=axis, keep_dims=keepdims)
 
 
 def round(x):
@@ -1715,14 +1755,13 @@ def resize_images(x, height_factor, width_factor, data_format):
         x: Tensor or variable to resize.
         height_factor: Positive integer.
         width_factor: Positive integer.
-        data_format: One of `"channels_first"`, `"channels_last"`.
+        data_format: string, `"channels_last"` or `"channels_first"`.
 
     # Returns
         A tensor.
 
     # Raises
-        ValueError: if `data_format` is neither
-            `channels_last` or `channels_first`.
+        ValueError: if `data_format` is neither `"channels_last"` or `"channels_first"`.
     """
     if data_format == 'channels_first':
         original_shape = int_shape(x)
@@ -1754,14 +1793,13 @@ def resize_volumes(x, depth_factor, height_factor, width_factor, data_format):
         depth_factor: Positive integer.
         height_factor: Positive integer.
         width_factor: Positive integer.
-        data_format: One of `"channels_first"`, `"channels_last"`.
+        data_format: string, `"channels_last"` or `"channels_first"`.
 
     # Returns
         A tensor.
 
     # Raises
-        ValueError: if `data_format` is neither
-            `channels_last` or `channels_first`.
+        ValueError: if `data_format` is neither `"channels_last"` or `"channels_first"`.
     """
     if data_format == 'channels_first':
         output = repeat_elements(x, depth_factor, axis=2)
@@ -1900,14 +1938,14 @@ def batch_flatten(x):
 
 
 def expand_dims(x, axis=-1):
-    """Adds a 1-sized dimension at index "dim".
+    """Adds a 1-sized dimension at index "axis".
 
     # Arguments
         x: A tensor or variable.
         axis: Position where to add a new axis.
 
     # Returns
-        A tensor with expended dimensions.
+        A tensor with expanded dimensions.
     """
     return tf.expand_dims(x, axis)
 
@@ -1947,14 +1985,13 @@ def spatial_2d_padding(x, padding=((1, 1), (1, 1)), data_format=None):
     # Arguments
         x: Tensor or variable.
         padding: Tuple of 2 tuples, padding pattern.
-        data_format: One of `channels_last` or `channels_first`.
+        data_format: string, `"channels_last"` or `"channels_first"`.
 
     # Returns
         A padded 4D tensor.
 
     # Raises
-        ValueError: if `data_format` is neither
-            `channels_last` or `channels_first`.
+        ValueError: if `data_format` is neither `"channels_last"` or `"channels_first"`.
     """
     assert len(padding) == 2
     assert len(padding[0]) == 2
@@ -1990,14 +2027,13 @@ def spatial_3d_padding(x, padding=((1, 1), (1, 1), (1, 1)), data_format=None):
     # Arguments
         x: Tensor or variable.
         padding: Tuple of 3 tuples, padding pattern.
-        data_format: One of `channels_last` or `channels_first`.
+        data_format: string, `"channels_last"` or `"channels_first"`.
 
     # Returns
         A padded 5D tensor.
 
     # Raises
-        ValueError: if `data_format` is neither
-            `channels_last` or `channels_first`.
+        ValueError: if `data_format` is neither `"channels_last"` or `"channels_first"`.
 
     """
     assert len(padding) == 3
@@ -2187,9 +2223,10 @@ class Function(object):
         inputs: Feed placeholders to the computation graph.
         outputs: Output tensors to fetch.
         updates: Additional update ops to be run at function call.
+        name: a name to help users identify what this function does.
     """
 
-    def __init__(self, inputs, outputs, updates=None):
+    def __init__(self, inputs, outputs, updates=None, name=None, **session_kwargs):
         updates = updates or []
         if not isinstance(inputs, (list, tuple)):
             raise TypeError('`inputs` to a TensorFlow backend function '
@@ -2212,6 +2249,8 @@ class Function(object):
                     # assumed already an op
                     updates_ops.append(update)
             self.updates_op = tf.group(*updates_ops)
+        self.name = name
+        self.session_kwargs = session_kwargs
 
     def __call__(self, inputs):
         if not isinstance(inputs, (list, tuple)):
@@ -2226,7 +2265,8 @@ class Function(object):
             feed_dict[tensor] = value
         session = get_session()
         updated = session.run(self.outputs + [self.updates_op],
-                              feed_dict=feed_dict)
+                              feed_dict=feed_dict,
+                              **self.session_kwargs)
         return updated[:len(self.outputs)]
 
 
@@ -2237,18 +2277,21 @@ def function(inputs, outputs, updates=None, **kwargs):
         inputs: List of placeholder tensors.
         outputs: List of output tensors.
         updates: List of update ops.
-        **kwargs: Not used with TensorFlow.
+        **kwargs: Passed to `tf.Session.run`.
 
     # Returns
         Output values as Numpy arrays.
+
+    # Raises
+        ValueError: if invalid kwargs are passed in.
     """
     if kwargs:
-        msg = [
-            'Expected no kwargs, you passed %s' % len(kwargs),
-            'kwargs passed to function are ignored with Tensorflow backend'
-        ]
-        warnings.warn('\n'.join(msg))
-    return Function(inputs, outputs, updates=updates)
+        for key in kwargs:
+            if (key not in inspect.getargspec(tf.Session.run)[0] and
+                    key not in inspect.getargspec(Function.__init__)[0]):
+                msg = 'Invalid argument "%s" passed to K.function with Tensorflow backend' % key
+                raise ValueError(msg)
+    return Function(inputs, outputs, updates=updates, **kwargs)
 
 
 def gradients(loss, variables):
@@ -2726,7 +2769,7 @@ def sparse_categorical_crossentropy(output, target, from_logits=False):
     # Returns
         Output tensor.
     """
-    # Note: tf.nn.softmax_cross_entropy_with_logits
+    # Note: tf.nn.sparse_softmax_cross_entropy_with_logits
     # expects logits, Keras expects probabilities.
     if not from_logits:
         epsilon = _to_tensor(_EPSILON, output.dtype.base_dtype)
@@ -2759,7 +2802,7 @@ def binary_crossentropy(output, target, from_logits=False):
     # Returns
         A tensor.
     """
-    # Note: tf.nn.softmax_cross_entropy_with_logits
+    # Note: tf.nn.sigmoid_cross_entropy_with_logits
     # expects logits, Keras expects probabilities.
     if not from_logits:
         # transform back to logits
@@ -2833,7 +2876,7 @@ def dropout(x, level, noise_shape=None, seed=None):
     if seed is None:
         seed = np.random.randint(10e6)
     # the dummy 1. works around a TF bug
-    # (float32_ref vs. float32 incomptability)
+    # (float32_ref vs. float32 incompatibility)
     return tf.nn.dropout(x * 1., retain_prob, noise_shape, seed=seed)
 
 
@@ -2876,7 +2919,7 @@ def _preprocess_deconv_output_shape(x, shape, data_format):
     # Arguments
         x: input tensor.
         shape: output shape.
-        data_format: string, one of 'channels_last', 'channels_first'.
+        data_format: string, `"channels_last"` or `"channels_first"`.
 
     # Returns
         The output shape.
@@ -2895,7 +2938,7 @@ def _preprocess_conv2d_input(x, data_format):
 
     # Arguments
         x: input tensor.
-        data_format: string, one of 'channels_last', 'channels_first'.
+        data_format: string, `"channels_last"` or `"channels_first"`.
 
     # Returns
         A tensor.
@@ -2916,7 +2959,7 @@ def _preprocess_conv3d_input(x, data_format):
 
     # Arguments
         x: input tensor.
-        data_format: string, one of 'channels_last', 'channels_first'.
+        data_format: string, `"channels_last"` or `"channels_first"`.
 
     # Returns
         A tensor.
@@ -2933,7 +2976,7 @@ def _preprocess_conv2d_kernel(kernel, data_format):
 
     # Arguments
         kernel: kernel tensor.
-        data_format: string, one of 'channels_last', 'channels_first'.
+        data_format: string, `"channels_last"` or `"channels_first"`.
 
     # Returns
         A tensor.
@@ -2950,7 +2993,7 @@ def _preprocess_conv3d_kernel(kernel, data_format):
 
     # Arguments
         kernel: kernel tensor.
-        data_format: string, one of 'channels_last', 'channels_first'.
+        data_format: string, `"channels_last"` or `"channels_first"`.
 
     # Returns
         A tensor.
@@ -2966,13 +3009,13 @@ def _preprocess_padding(padding):
     """Convert keras' padding to tensorflow's padding.
 
     # Arguments
-        padding: string, one of 'same' , 'valid'
+        padding: string, `"same"` or `"valid"`.
 
     # Returns
-        a string, one of 'SAME', 'VALID'.
+        a string, `"SAME"` or `"VALID"`.
 
     # Raises
-        ValueError if invalid `padding'`
+        ValueError: if `padding` is invalid.
     """
     if padding == 'same':
         padding = 'SAME'
@@ -2988,7 +3031,7 @@ def _postprocess_conv2d_output(x, data_format):
 
     # Arguments
         x: A tensor.
-        data_format: string, one of "channels_last", "channels_first".
+        data_format: string, `"channels_last"` or `"channels_first"`.
 
     # Returns
         A tensor.
@@ -3005,13 +3048,13 @@ def _postprocess_conv2d_output(x, data_format):
 def _postprocess_conv3d_output(x, data_format):
     """Transpose and cast the output from conv3d if needed.
 
-        # Arguments
-            x: A tensor.
-            data_format: string, one of "channels_last", "channels_first".
+    # Arguments
+        x: A tensor.
+        data_format: string, `"channels_last"` or `"channels_first"`.
 
-        # Returns
-            A tensor.
-        """
+    # Returns
+        A tensor.
+    """
     if data_format == 'channels_first':
         x = tf.transpose(x, (0, 4, 1, 2, 3))
 
@@ -3029,7 +3072,7 @@ def conv1d(x, kernel, strides=1, padding='valid',
         kernel: kernel tensor.
         strides: stride integer.
         padding: string, `"same"`, `"causal"` or `"valid"`.
-        data_format: string, one of "channels_last", "channels_first".
+        data_format: string, `"channels_last"` or `"channels_first"`.
         dilation_rate: integer dilate rate.
 
     # Returns
@@ -3065,9 +3108,9 @@ def conv2d(x, kernel, strides=(1, 1), padding='valid',
         kernel: kernel tensor.
         strides: strides tuple.
         padding: string, `"same"` or `"valid"`.
-        data_format: `"channels_last"` or `"channels_first"`.
+        data_format: string, `"channels_last"` or `"channels_first"`.
             Whether to use Theano or TensorFlow data format
-            for inputs/kernels/ouputs.
+            for inputs/kernels/outputs.
         dilation_rate: tuple of 2 integers.
 
     # Returns
@@ -3106,9 +3149,9 @@ def conv2d_transpose(x, kernel, output_shape, strides=(1, 1),
         output_shape: 1D int tensor for the output shape.
         strides: strides tuple.
         padding: string, `"same"` or `"valid"`.
-        data_format: `"channels_last"` or `"channels_first"`.
+        data_format: string, `"channels_last"` or `"channels_first"`.
             Whether to use Theano or TensorFlow data format
-            for inputs/kernels/ouputs.
+            for inputs/kernels/outputs.
 
     # Returns
         A tensor, result of transposed 2D convolution.
@@ -3143,8 +3186,8 @@ def separable_conv2d(x, depthwise_kernel, pointwise_kernel, strides=(1, 1),
         depthwise_kernel: convolution kernel for the depthwise convolution.
         pointwise_kernel: kernel for the 1x1 convolution.
         strides: strides tuple (length 2).
-        padding: padding mode, "valid" or "same".
-        data_format: data format, "channels_first" or "channels_last".
+        padding: string, `"same"` or `"valid"`.
+        data_format: string, `"channels_last"` or `"channels_first"`.
         dilation_rate: tuple of integers,
             dilation rates for the separable convolution.
 
@@ -3179,9 +3222,9 @@ def conv3d(x, kernel, strides=(1, 1, 1), padding='valid',
         kernel: kernel tensor.
         strides: strides tuple.
         padding: string, `"same"` or `"valid"`.
-        data_format: `"channels_last"` or `"channels_first"`.
+        data_format: string, `"channels_last"` or `"channels_first"`.
             Whether to use Theano or TensorFlow data format
-            for inputs/kernels/ouputs.
+            for inputs/kernels/outputs.
         dilation_rate: tuple of 3 integers.
 
     # Returns
@@ -3219,16 +3262,16 @@ def pool2d(x, pool_size, strides=(1, 1),
         x: Tensor or variable.
         pool_size: tuple of 2 integers.
         strides: tuple of 2 integers.
-        padding: one of `"valid"`, `"same"`.
-        data_format: one of `"channels_first"`, `"channels_last"`.
-        pool_mode: one of `"max"`, `"avg"`.
+        padding: string, `"same"` or `"valid"`.
+        data_format: string, `"channels_last"` or `"channels_first"`.
+        pool_mode: string, `"max"` or `"avg"`.
 
     # Returns
         A tensor, result of 2D pooling.
 
     # Raises
-        ValueError: if `data_format` is neither `channels_last` or `channels_first`.
-        ValueError: if `pool_mode` is neither `max` or `avg`.
+        ValueError: if `data_format` is neither `"channels_last"` or `"channels_first"`.
+        ValueError: if `pool_mode` is neither `"max"` or `"avg"`.
     """
     if data_format is None:
         data_format = image_data_format()
@@ -3259,17 +3302,16 @@ def pool3d(x, pool_size, strides=(1, 1, 1), padding='valid',
         x: Tensor or variable.
         pool_size: tuple of 3 integers.
         strides: tuple of 3 integers.
-        padding: one of `"valid"`, `"same"`.
-        data_format: one of `"channels_first"`, `"channels_last"`.
-        pool_mode: one of `"max"`, `"avg"`.
+        padding: string, `"same"` or `"valid"`.
+        data_format: string, `"channels_last"` or `"channels_first"`.
+        pool_mode: string, `"max"` or `"avg"`.
 
     # Returns
         A tensor, result of 3D pooling.
 
     # Raises
-        ValueError: if `data_format` is neither
-            `channels_last` or `channels_first`.
-        ValueError: if `pool_mode` is neither `max` or `avg`.
+        ValueError: if `data_format` is neither `"channels_last"` or `"channels_first"`.
+        ValueError: if `pool_mode` is neither `"max"` or `"avg"`.
     """
     if data_format is None:
         data_format = image_data_format()
@@ -3298,35 +3340,60 @@ def bias_add(x, bias, data_format=None):
     # Arguments
         x: Tensor or variable.
         bias: Bias tensor to add.
-        data_format: Data format for 3D, 4D or 5D tensors:
-            one of "channels_first", "channels_last".
+        data_format: string, `"channels_last"` or `"channels_first"`.
 
     # Returns
         Output tensor.
 
     # Raises
-        ValueError: In case of invalid `data_format` argument.
+        ValueError: In one of the two cases below:
+                    1. invalid `data_format` argument.
+                    2. invalid bias shape.
+                       the bias should be either a vector or
+                       a tensor with ndim(x) - 1 dimension
     """
     if data_format is None:
         data_format = image_data_format()
     if data_format not in {'channels_first', 'channels_last'}:
         raise ValueError('Unknown data_format ' + str(data_format))
+    bias_shape = int_shape(bias)
+    if len(bias_shape) != 1 and len(bias_shape) != ndim(x) - 1:
+        raise ValueError('Unexpected bias dimensions %d, expect to be 1 or %d dimensions'
+                         % (len(bias_shape), ndim(x)))
     if ndim(x) == 5:
         if data_format == 'channels_first':
-            x += reshape(bias, (1, int_shape(bias)[0], 1, 1, 1))
+            if len(bias_shape) == 1:
+                x += reshape(bias, (1, bias_shape[0], 1, 1, 1))
+            else:
+                x += reshape(bias, (1, bias_shape[3]) + bias_shape[:3])
         elif data_format == 'channels_last':
-            x += reshape(bias, (1, 1, 1, 1, int_shape(bias)[0]))
+            if len(bias_shape) == 1:
+                x += reshape(bias, (1, 1, 1, bias_shape[0]))
+            else:
+                x += reshape(bias, (1,) + bias_shape)
     elif ndim(x) == 4:
         if data_format == 'channels_first':
-            x += reshape(bias, (1, int_shape(bias)[0], 1, 1))
+            if len(bias_shape) == 1:
+                x += reshape(bias, (1, bias_shape[0], 1, 1))
+            else:
+                x += reshape(bias, (1, bias_shape[2]) + bias_shape[:2])
         elif data_format == 'channels_last':
-            x = tf.nn.bias_add(x, bias,
-                               data_format='NHWC')
+            if len(bias_shape) == 1:
+                x = tf.nn.bias_add(x, bias,
+                                   data_format='NHWC')
+            else:
+                x += reshape(bias, (1,) + bias_shape)
     elif ndim(x) == 3:
         if data_format == 'channels_first':
-            x += reshape(bias, (1, int_shape(bias)[0], 1))
+            if len(bias_shape) == 1:
+                x += reshape(bias, (1, bias_shape[0], 1))
+            else:
+                x += reshape(bias, (1, bias_shape[1], bias_shape[0]))
         elif data_format == 'channels_last':
-            x += reshape(bias, (1, 1, int_shape(bias)[0]))
+            if len(bias_shape) == 1:
+                x += reshape(bias, (1, 1, bias_shape[0]))
+            else:
+                x += reshape(bias, (1, ) + bias_shape)
     else:
         x = tf.nn.bias_add(x, bias)
     return x
@@ -3591,3 +3658,110 @@ def foldr(fn, elems, initializer=None, name=None):
         Same type and shape as initializer
     """
     return tf.foldr(fn, elems, initializer=initializer, name=name)
+
+
+def local_conv1d(inputs, kernel, kernel_size, strides, data_format=None):
+    """Apply 1D conv with un-shared weights.
+
+    # Arguments
+        inputs: 3D tensor with shape: (batch_size, steps, input_dim)
+        kernel: the unshared weight for convolution,
+                with shape (output_length, feature_dim, filters)
+        kernel_size: a tuple of a single integer,
+                     specifying the length of the 1D convolution window
+        strides: a tuple of a single integer,
+                 specifying the stride length of the convolution
+        data_format: the data format, channels_first or channels_last
+
+    # Returns
+        the tensor after 1d conv with un-shared weights, with shape (batch_size, output_lenght, filters)
+
+    # Raises
+        ValueError: if `data_format` is neither `channels_last` or `channels_first`.
+    """
+    if data_format is None:
+        data_format = image_data_format()
+    if data_format not in {'channels_first', 'channels_last'}:
+        raise ValueError('Unknown data_format ' + str(data_format))
+
+    stride = strides[0]
+    kernel_shape = int_shape(kernel)
+    output_length, feature_dim, filters = kernel_shape
+
+    xs = []
+    for i in range(output_length):
+        slice_length = slice(i * stride,
+                             i * stride + kernel_size[0])
+        xs.append(reshape(inputs[:, slice_length, :],
+                          (1, -1, feature_dim)))
+    x_aggregate = concatenate(xs, axis=0)
+    # Shape: `(output_length, batch_size, filters)`.
+    output = batch_dot(x_aggregate, kernel)
+    return permute_dimensions(output, (1, 0, 2))
+
+
+def local_conv2d(inputs, kernel, kernel_size, strides, output_shape, data_format=None):
+    """Apply 2D conv with un-shared weights.
+
+    # Arguments
+        inputs: 4D tensor with shape:
+                (batch_size, filters, new_rows, new_cols)
+                if data_format='channels_first'
+                or 4D tensor with shape:
+                (batch_size, new_rows, new_cols, filters)
+                if data_format='channels_last'.
+        kernel: the unshared weight for convolution,
+                with shape (output_items, feature_dim, filters)
+        kernel_size: a tuple of 2 integers, specifying the
+                     width and height of the 2D convolution window.
+        strides: a tuple of 2 integers, specifying the strides
+                 of the convolution along the width and height.
+        output_shape: a tuple with (output_row, output_col)
+        data_format: the data format, channels_first or channels_last
+
+    # Returns
+        A 4d tensor with shape:
+        (batch_size, filters, new_rows, new_cols)
+        if data_format='channels_first'
+        or 4D tensor with shape:
+        (batch_size, new_rows, new_cols, filters)
+        if data_format='channels_last'.
+
+    # Raises
+        ValueError: if `data_format` is neither
+                    `channels_last` or `channels_first`.
+    """
+    if data_format is None:
+        data_format = image_data_format()
+    if data_format not in {'channels_first', 'channels_last'}:
+        raise ValueError('Unknown data_format ' + str(data_format))
+
+    stride_row, stride_col = strides
+    output_row, output_col = output_shape
+    kernel_shape = int_shape(kernel)
+    _, feature_dim, filters = kernel_shape
+
+    xs = []
+    for i in range(output_row):
+        for j in range(output_col):
+            slice_row = slice(i * stride_row,
+                              i * stride_row + kernel_size[0])
+            slice_col = slice(j * stride_col,
+                              j * stride_col + kernel_size[1])
+            if data_format == 'channels_first':
+                xs.append(reshape(inputs[:, :, slice_row, slice_col],
+                                  (1, -1, feature_dim)))
+            else:
+                xs.append(reshape(inputs[:, slice_row, slice_col, :],
+                                  (1, -1, feature_dim)))
+
+    x_aggregate = concatenate(xs, axis=0)
+    output = batch_dot(x_aggregate, kernel)
+    output = reshape(output,
+                     (output_row, output_col, -1, filters))
+
+    if data_format == 'channels_first':
+        output = permute_dimensions(output, (2, 3, 0, 1))
+    else:
+        output = permute_dimensions(output, (2, 0, 1, 3))
+    return output

keras/backend/theano_backend.py

@@ -164,6 +164,42 @@ def constant(value, dtype=None, shape=None, name=None):
     return const
 
 
+def is_keras_tensor(x):
+    """Returns whether `x` is a Keras tensor.
+
+    # Arguments
+        x: a potential tensor.
+
+    # Returns
+        A boolean: whether the argument is a Keras tensor.
+
+    # Raises
+        ValueError: in case `x` is not a symbolic tensor.
+
+    # Examples
+    ```python
+        >>> from keras import backend as K
+        >>> np_var = numpy.array([1, 2])
+        >>> K.is_keras_tensor(np_var) # A numpy array is not a symbolic tensor.
+        ValueError
+        >>> k_var = theano.shared(value=np.array([1,2,3]))
+        >>> K.is_keras_tensor(k_var) # A variable created directly from tensorflow/theano is not a Keras tensor.
+        False
+        >>> keras_var = K.variable(np_var)
+        >>> K.is_keras_tensor(keras_var) # A variable created with the keras backend is a Keras tensor.
+        True
+        >>> keras_placeholder = K.placeholder(shape=(2, 4, 5))
+        >>> K.is_keras_tensor(keras_placeholder)  # A placeholder is a Keras tensor.
+        True
+    ```
+    """
+    if not isinstance(x, (T.TensorVariable,
+                          T.sharedvar.TensorSharedVariable)):
+        raise ValueError('Unexpectedly found an instance of type `' + str(type(x)) + '`. '
+                         'Expected a symbolic tensor instance.')
+    return hasattr(x, '_keras_history')
+
+
 def placeholder(shape=None, ndim=None, dtype=None, sparse=False, name=None):
     """Instantiate an input data placeholder variable.
     """
@@ -1142,7 +1178,7 @@ def print_tensor(x, message=''):
 
 class Function(object):
 
-    def __init__(self, inputs, outputs, updates=[], **kwargs):
+    def __init__(self, inputs, outputs, updates=[], name=None, **kwargs):
         unique_variables_to_update = {}
         for v, nv in updates:
             if v not in unique_variables_to_update:
@@ -1151,7 +1187,9 @@ class Function(object):
         self.function = theano.function(inputs, outputs, updates=updates,
                                         allow_input_downcast=True,
                                         on_unused_input='ignore',
+                                        name=name,
                                         **kwargs)
+        self.name = name
 
     def __call__(self, inputs):
         assert isinstance(inputs, (list, tuple))
@@ -1163,7 +1201,7 @@ def function(inputs, outputs, updates=[], **kwargs):
         function_args = inspect.getargspec(theano.function)[0]
         for key in kwargs.keys():
             if key not in function_args:
-                msg = 'Invalid argument "%s" passed to K.function' % key
+                msg = 'Invalid argument "%s" passed to K.function with Theano backend' % key
                 raise ValueError(msg)
     return Function(inputs, outputs, updates=updates, **kwargs)
 
@@ -1524,8 +1562,9 @@ def dropout(x, level, noise_shape=None, seed=None):
     return x
 
 
-def l2_normalize(x, axis):
-    norm = T.sqrt(T.sum(T.square(x), axis=axis, keepdims=True))
+def l2_normalize(x, axis, epsilon=1e-12):
+    square_sum = T.sum(T.square(x), axis=axis, keepdims=True)
+    norm = T.sqrt(T.maximum(square_sum, epsilon))
     return x / norm
 
 
@@ -1775,7 +1814,7 @@ def conv2d(x, kernel, strides=(1, 1), padding='valid',
         padding: string, "same" or "valid".
         data_format: "channels_last" or "channels_first".
             Whether to use Theano or TensorFlow data format
-        in inputs/kernels/ouputs.
+        in inputs/kernels/outputs.
     """
     if data_format is None:
         data_format = image_data_format()
@@ -1819,7 +1858,10 @@ def conv2d_transpose(x, kernel, output_shape, strides=(1, 1),
         padding: string, "same" or "valid".
         data_format: "channels_last" or "channels_first".
             Whether to use Theano or TensorFlow data format
-        in inputs/kernels/ouputs.
+        in inputs/kernels/outputs.
+
+    # Raises
+        ValueError: if using an even kernel size with padding 'same'.
     """
     flip_filters = False
     if data_format is None:
@@ -1838,6 +1880,12 @@ def conv2d_transpose(x, kernel, output_shape, strides=(1, 1),
     else:
         # Will only work if `kernel` is a shared variable.
         kernel_shape = kernel.eval().shape
+
+    if padding == 'same' and kernel_shape[0] % 2 == 0:
+        raise ValueError('In `Conv2DTranspose`, with padding mode `same`, '
+                         'even kernel sizes are only supported with Tensorflow. '
+                         'With Theano, set `kernel_size` to an odd number.')
+
     kernel_shape = _preprocess_conv2d_filter_shape(kernel_shape, data_format)
 
     x = _preprocess_conv2d_input(x, data_format)
@@ -1871,7 +1919,7 @@ def conv3d(x, kernel, strides=(1, 1, 1),
         padding: string, "same" or "valid".
         data_format: "channels_last" or "channels_first".
             Whether to use Theano or TensorFlow data format
-        in inputs/kernels/ouputs.
+        in inputs/kernels/outputs.
     """
     if data_format is None:
         data_format = image_data_format()
@@ -2011,21 +2059,44 @@ def bias_add(x, bias, data_format=None):
         data_format = image_data_format()
     if data_format not in {'channels_first', 'channels_last'}:
         raise ValueError('Unknown data_format ' + str(data_format))
+    if ndim(bias) != 1 and ndim(bias) != ndim(x) - 1:
+        raise ValueError('Unexpected bias dimensions %d, '
+                         'expect to be 1 or %d dimensions'
+                         % (ndim(bias), ndim(x) - 1))
+    bias_shape = tuple(bias.shape)
     if ndim(x) == 5:
         if data_format == 'channels_first':
-            x += reshape(bias, (1, bias.shape[0], 1, 1, 1))
+            if ndim(bias) == 1:
+                x += reshape(bias, (1, bias_shape[0], 1, 1, 1))
+            else:
+                x += reshape(bias, (1, bias_shape[3]) + bias_shape[:3])
         elif data_format == 'channels_last':
-            x += reshape(bias, (1, 1, 1, 1, bias.shape[0]))
+            if ndim(bias) == 1:
+                x += reshape(bias, (1, 1, 1, 1, bias_shape[0]))
+            else:
+                x += reshape(bias, (1,) + bias_shape)
     elif ndim(x) == 4:
         if data_format == 'channels_first':
-            x += reshape(bias, (1, bias.shape[0], 1, 1))
+            if ndim(bias) == 1:
+                x += reshape(bias, (1, bias_shape[0], 1, 1))
+            else:
+                x += reshape(bias, (1, bias_shape[2]) + bias_shape[:2])
         elif data_format == 'channels_last':
-            x += reshape(bias, (1, 1, 1, bias.shape[0]))
+            if ndim(bias) == 1:
+                x += reshape(bias, (1, 1, 1, bias_shape[0]))
+            else:
+                x += reshape(bias, (1,) + bias_shape)
     elif ndim(x) == 3:
         if data_format == 'channels_first':
-            x += reshape(bias, (1, bias.shape[0], 1))
+            if ndim(bias) == 1:
+                x += reshape(bias, (1, bias_shape[0], 1))
+            else:
+                x += reshape(bias, (1, bias_shape[1], bias_shape[0]))
         elif data_format == 'channels_last':
-            x += reshape(bias, (1, 1, bias.shape[0]))
+            if ndim(bias) == 1:
+                x += reshape(bias, (1, 1, bias_shape[0]))
+            else:
+                x += reshape(bias, (1,) + bias_shape)
     else:
         x += bias
     return x
@@ -2243,3 +2314,72 @@ def foldr(fn, elems, initializer=None, name=None):
     fn2 = lambda x, acc: fn(acc, x)
 
     return theano.foldr(fn2, elems, initializer, name=name)[0]
+
+
+def local_conv1d(inputs, kernel, kernel_size, strides, data_format=None):
+    if data_format is None:
+        data_format = image_data_format()
+    if data_format not in {'channels_first', 'channels_last'}:
+        raise ValueError('Unknown data_format ' + str(data_format))
+
+    stride = strides[0]
+    kernel_shape = int_shape(kernel)
+    output_length, feature_dim, filters = kernel_shape
+
+    xs = []
+    for i in range(output_length):
+        slice_length = slice(i * stride,
+                             i * stride + kernel_size[0])
+        xs.append(reshape(inputs[:, slice_length, :],
+                          (1, -1, feature_dim)))
+    x_aggregate = concatenate(xs, axis=0)
+    # Shape: `(output_length, batch_size, filters)`.
+    output = batch_dot(x_aggregate, kernel)
+    return permute_dimensions(output, (1, 0, 2))
+
+
+def local_conv2d(inputs, kernel, kernel_size, strides, output_shape, data_format=None):
+    if data_format is None:
+        data_format = image_data_format()
+    if data_format not in {'channels_first', 'channels_last'}:
+        raise ValueError('Unknown data_format ' + str(data_format))
+
+    stride_row, stride_col = strides
+    output_row, output_col = output_shape
+    kernel_shape = int_shape(kernel)
+    _, feature_dim, filters = kernel_shape
+
+    if data_format == 'channels_first':
+        output = []
+        for i in range(output_row):
+            for j in range(output_col):
+                slice_row = slice(i * stride_row,
+                                  i * stride_row + kernel_size[0])
+                slice_col = slice(j * stride_col,
+                                  j * stride_col + kernel_size[1])
+                x_flatten = reshape(inputs[:, :, slice_row, slice_col],
+                                    (1, -1, feature_dim))
+                output.append(dot(x_flatten,
+                                  kernel[i * output_col + j, :, :]))
+        output = concatenate(output, axis=0)
+        output = reshape(output,
+                         (output_row, output_col, -1, filters))
+        output = permute_dimensions(output, (2, 3, 0, 1))
+    else:
+        xs = []
+        for i in range(output_row):
+            for j in range(output_col):
+                slice_row = slice(i * stride_row,
+                                  i * stride_row + kernel_size[0])
+                slice_col = slice(j * stride_col,
+                                  j * stride_col + kernel_size[1])
+                xs.append(reshape(inputs[:, slice_row, slice_col, :],
+                                  (1, -1, feature_dim)))
+
+        x_aggregate = concatenate(xs, axis=0)
+        output = batch_dot(x_aggregate, kernel)
+        output = reshape(output,
+                         (output_row, output_col, -1, filters))
+        output = permute_dimensions(output, (2, 0, 1, 3))
+
+    return output

keras/callbacks.py

@@ -227,6 +227,21 @@ class BaseLogger(Callback):
                     logs[k] = self.totals[k] / self.seen
 
 
+class TerminateOnNaN(Callback):
+    """Callback that terminates training when a NaN loss is encountered."""
+
+    def __init__(self):
+        super(TerminateOnNaN, self).__init__()
+
+    def on_batch_end(self, batch, logs=None):
+        logs = logs or {}
+        loss = logs.get('loss')
+        if loss is not None:
+            if np.isnan(loss) or np.isinf(loss):
+                print('Batch %d: Invalid loss, terminating training' % (batch))
+                self.model.stop_training = True
+
+
 class ProgbarLogger(Callback):
     """Callback that prints metrics to stdout.
 
@@ -467,7 +482,9 @@ class EarlyStopping(Callback):
             self.min_delta *= -1
 
     def on_train_begin(self, logs=None):
-        self.wait = 0  # Allow instances to be re-used
+        # Allow instances to be re-used
+        self.wait = 0
+        self.stopped_epoch = 0
         self.best = np.Inf if self.monitor_op == np.less else -np.Inf
 
     def on_epoch_end(self, epoch, logs=None):
@@ -582,15 +599,20 @@ class TensorBoard(Callback):
 
     # Arguments
         log_dir: the path of the directory where to save the log
-            files to be parsed by Tensorboard.
+            files to be parsed by TensorBoard.
         histogram_freq: frequency (in epochs) at which to compute activation
-            histograms for the layers of the model. If set to 0,
-            histograms won't be computed.
-        write_graph: whether to visualize the graph in Tensorboard.
+            and weight histograms for the layers of the model. If set to 0,
+            histograms won't be computed. Validation data (or split) must be
+            specified for histogram visualizations.
+        write_graph: whether to visualize the graph in TensorBoard.
             The log file can become quite large when
             write_graph is set to True.
+        write_grads: whether to visualize gradient histograms in TensorBoard.
+            `histogram_freq` must be greater than 0.
+        batch_size: size of batch of inputs to feed to the network
+            for histograms computation.
         write_images: whether to write model weights to visualize as
-            image in Tensorboard.
+            image in TensorBoard.
         embeddings_freq: frequency (in epochs) at which selected embedding
             layers will be saved.
         embeddings_layer_names: a list of names of layers to keep eye on. If
@@ -604,7 +626,9 @@ class TensorBoard(Callback):
 
     def __init__(self, log_dir='./logs',
                  histogram_freq=0,
+                 batch_size=32,
                  write_graph=True,
+                 write_grads=False,
                  write_images=False,
                  embeddings_freq=0,
                  embeddings_layer_names=None,
@@ -617,10 +641,12 @@ class TensorBoard(Callback):
         self.histogram_freq = histogram_freq
         self.merged = None
         self.write_graph = write_graph
+        self.write_grads = write_grads
         self.write_images = write_images
         self.embeddings_freq = embeddings_freq
         self.embeddings_layer_names = embeddings_layer_names
         self.embeddings_metadata = embeddings_metadata or {}
+        self.batch_size = batch_size
 
     def set_model(self, model):
         self.model = model
@@ -630,14 +656,42 @@ class TensorBoard(Callback):
 
                 for weight in layer.weights:
                     tf.summary.histogram(weight.name, weight)
+                    if self.write_grads:
+                        grads = model.optimizer.get_gradients(model.total_loss,
+                                                              weight)
+                        tf.summary.histogram('{}_grad'.format(weight.name), grads)
                     if self.write_images:
                         w_img = tf.squeeze(weight)
-                        shape = w_img.get_shape()
-                        if len(shape) > 1 and shape[0] > shape[1]:
-                            w_img = tf.transpose(w_img)
-                        if len(shape) == 1:
-                            w_img = tf.expand_dims(w_img, 0)
-                        w_img = tf.expand_dims(tf.expand_dims(w_img, 0), -1)
+                        shape = K.int_shape(w_img)
+                        if len(shape) == 2:  # dense layer kernel case
+                            if shape[0] > shape[1]:
+                                w_img = tf.transpose(w_img)
+                                shape = K.int_shape(w_img)
+                            w_img = tf.reshape(w_img, [1,
+                                                       shape[0],
+                                                       shape[1],
+                                                       1])
+                        elif len(shape) == 3:  # convnet case
+                            if K.image_data_format() == 'channels_last':
+                                # switch to channels_first to display
+                                # every kernel as a separate image
+                                w_img = tf.transpose(w_img, perm=[2, 0, 1])
+                                shape = K.int_shape(w_img)
+                            w_img = tf.reshape(w_img, [shape[0],
+                                                       shape[1],
+                                                       shape[2],
+                                                       1])
+                        elif len(shape) == 1:  # bias case
+                            w_img = tf.reshape(w_img, [1,
+                                                       shape[0],
+                                                       1,
+                                                       1])
+                        else:
+                            # not possible to handle 3D convnets etc.
+                            continue
+
+                        shape = K.int_shape(w_img)
+                        assert len(shape) == 4 and shape[-1] in [1, 3, 4]
                         tf.summary.image(weight.name, w_img)
 
                 if hasattr(layer, 'output'):
@@ -652,8 +706,6 @@ class TensorBoard(Callback):
             self.writer = tf.summary.FileWriter(self.log_dir)
 
         if self.embeddings_freq:
-            self.saver = tf.train.Saver()
-
             embeddings_layer_names = self.embeddings_layer_names
 
             if not embeddings_layer_names:
@@ -664,6 +716,8 @@ class TensorBoard(Callback):
                           for layer in self.model.layers
                           if layer.name in embeddings_layer_names}
 
+            self.saver = tf.train.Saver(list(embeddings.values()))
+
             embeddings_metadata = {}
 
             if not isinstance(self.embeddings_metadata, str):
@@ -673,15 +727,13 @@ class TensorBoard(Callback):
                                        for layer_name in embeddings.keys()}
 
             config = projector.ProjectorConfig()
-            self.embeddings_logs = []
+            self.embeddings_ckpt_path = os.path.join(self.log_dir,
+                                                     'keras_embedding.ckpt')
 
             for layer_name, tensor in embeddings.items():
                 embedding = config.embeddings.add()
                 embedding.tensor_name = tensor.name
 
-                self.embeddings_logs.append(os.path.join(self.log_dir,
-                                                         layer_name + '.ckpt'))
-
                 if layer_name in embeddings_metadata:
                     embedding.metadata_path = embeddings_metadata[layer_name]
 
@@ -692,24 +744,37 @@ class TensorBoard(Callback):
 
         if self.validation_data and self.histogram_freq:
             if epoch % self.histogram_freq == 0:
-                # TODO: implement batched calls to sess.run
-                # (current call will likely go OOM on GPU)
-                if self.model.uses_learning_phase:
-                    cut_v_data = len(self.model.inputs)
-                    val_data = self.validation_data[:cut_v_data] + [0]
-                    tensors = self.model.inputs + [K.learning_phase()]
-                else:
-                    val_data = self.validation_data
-                    tensors = self.model.inputs
-                feed_dict = dict(zip(tensors, val_data))
-                result = self.sess.run([self.merged], feed_dict=feed_dict)
-                summary_str = result[0]
-                self.writer.add_summary(summary_str, epoch)
 
-        if self.embeddings_freq and self.embeddings_logs:
+                val_data = self.validation_data
+                tensors = (self.model.inputs +
+                           self.model.targets +
+                           self.model.sample_weights)
+
+                if self.model.uses_learning_phase:
+                    tensors += [K.learning_phase()]
+
+                assert len(val_data) == len(tensors)
+                val_size = val_data[0].shape[0]
+                i = 0
+                while i < val_size:
+                    step = min(self.batch_size, val_size - i)
+                    batch_val = []
+                    batch_val.append(val_data[0][i:i + step])
+                    batch_val.append(val_data[1][i:i + step])
+                    batch_val.append(val_data[2][i:i + step])
+                    if self.model.uses_learning_phase:
+                        batch_val.append(val_data[3])
+                    feed_dict = dict(zip(tensors, batch_val))
+                    result = self.sess.run([self.merged], feed_dict=feed_dict)
+                    summary_str = result[0]
+                    self.writer.add_summary(summary_str, epoch)
+                    i += self.batch_size
+
+        if self.embeddings_freq and self.embeddings_ckpt_path:
             if epoch % self.embeddings_freq == 0:
-                for log in self.embeddings_logs:
-                    self.saver.save(self.sess, log, epoch)
+                self.saver.save(self.sess,
+                                self.embeddings_ckpt_path,
+                                epoch)
 
         for name, value in logs.items():
             if name in ['batch', 'size']:
@@ -879,7 +944,9 @@ class CSVLogger(Callback):
 
         def handle_value(k):
             is_zero_dim_ndarray = isinstance(k, np.ndarray) and k.ndim == 0
-            if isinstance(k, Iterable) and not is_zero_dim_ndarray:
+            if isinstance(k, six.string_types):
+                return k
+            elif isinstance(k, Iterable) and not is_zero_dim_ndarray:
                 return '"[%s]"' % (', '.join(map(str, k)))
             else:
                 return k

keras/constraints.py

@@ -58,7 +58,7 @@ class NonNeg(Constraint):
     """
 
     def __call__(self, w):
-        w *= K.cast(w >= 0., K.floatx())
+        w *= K.cast(K.greater_equal(w, 0.), K.floatx())
         return w
 
 

keras/datasets/boston_housing.py

@@ -16,7 +16,9 @@ def load_data(path='boston_housing.npz', seed=113, test_split=0.2):
         Tuple of Numpy arrays: `(x_train, y_train), (x_test, y_test)`.
     """
     assert 0 <= test_split < 1
-    path = get_file(path, origin='https://s3.amazonaws.com/keras-datasets/boston_housing.npz')
+    path = get_file(path,
+                    origin='https://s3.amazonaws.com/keras-datasets/boston_housing.npz',
+                    file_hash='f553886a1f8d56431e820c5b82552d9d95cfcb96d1e678153f8839538947dff5')
     f = np.load(path)
     x = f['x']
     y = f['y']

keras/engine/topology.py

@@ -413,13 +413,24 @@ class Layer(object):
             ValueError: in case of mismatch between
                 the provided inputs and the expectations of the layer.
         """
+        inputs = _to_list(inputs)
+        for x in inputs:
+            try:
+                K.is_keras_tensor(x)
+            except ValueError:
+                raise ValueError('Layer ' + self.name + ' was called with '
+                                 'an input that isn\'t a symbolic tensor. '
+                                 'Received type: ' +
+                                 str(type(x)) + '. Full input: ' +
+                                 str(inputs) + '. All inputs to the layer '
+                                 'should be tensors.')
+
         if not self.input_spec:
             return
         if not isinstance(self.input_spec, (list, tuple)):
             input_spec = _to_list(self.input_spec)
         else:
             input_spec = self.input_spec
-        inputs = _to_list(inputs)
         if len(inputs) != len(input_spec):
             raise ValueError('Layer ' + self.name + ' expects ' +
                              str(len(input_spec)) + ' inputs, '
@@ -1077,7 +1088,7 @@ class Layer(object):
         if hasattr(self, '_losses'):
             self._losses += losses
         # Update self._per_input_updates
-        if inputs == []:
+        if isinstance(input, list) and inputs == []:
             inputs = None
         if inputs is not None:
             inputs_hash = _object_list_uid(inputs)
@@ -1109,7 +1120,7 @@ class Layer(object):
         if hasattr(self, '_updates'):
             self._updates += updates
         # Update self._per_input_updates
-        if inputs == []:
+        if isinstance(inputs, list) and inputs == []:
             inputs = None
         if inputs is not None:
             inputs_hash = _object_list_uid(inputs)
@@ -1287,7 +1298,8 @@ class InputLayer(Layer):
             raise ValueError('Only provide the input_shape OR '
                              'batch_input_shape argument to '
                              'InputLayer, not both at the same time.')
-        if input_tensor is not None:
+        if input_tensor is not None and batch_input_shape is None:
+            # If input_tensor is set, and batch_input_shape is not set:
             # Attempt automatic input shape inference.
             try:
                 batch_input_shape = K.int_shape(input_tensor)
@@ -1359,7 +1371,7 @@ def Input(shape=None, batch_shape=None,
     attributes that allow us to build a Keras model
     just by knowing the inputs and outputs of the model.
 
-    For instance, if a, b and c and Keras tensors,
+    For instance, if a, b and c are Keras tensors,
     it becomes possible to do:
     `model = Model(input=[a, b], output=c)`
 
@@ -1456,6 +1468,9 @@ class Container(Layer):
 
     # Class Methods
         from_config
+
+    # Raises
+        TypeError: if input tensors are not Keras tensors from InputLayer objects
     """
 
     @interfaces.legacy_model_constructor_support
@@ -1482,13 +1497,19 @@ class Container(Layer):
             self.outputs = [outputs]
 
         # Check for redundancy in inputs.
-        inputs_set = set(self.inputs)
-        if len(inputs_set) != len(self.inputs):
+        if len(set(self.inputs)) != len(self.inputs):
             raise ValueError('The list of inputs passed to the model '
                              'is redundant. '
                              'All inputs should only appear once.'
                              ' Found: ' + str(self.inputs))
 
+        # Check for redundancy in outputs.
+        if len(set(self.outputs)) != len(self.outputs):
+            warnings.warn('The list of outputs passed to the model '
+                          'is redundant. '
+                          'All outputs should only appear once.'
+                          ' Found: ' + str(self.outputs))
+
         # List of initial layers (1 to 1 mapping with self.inputs,
         # hence the same layer might appear twice)
         self.input_layers = []
@@ -1590,6 +1611,15 @@ class Container(Layer):
         self._feed_inputs = []
         self._feed_input_shapes = []
         for i, layer in enumerate(self.input_layers):
+            # Check that layer is an InputLayer.
+            if not isinstance(layer, InputLayer):
+                raise TypeError(
+                    'Input layers to a `Model` must be `InputLayer` objects. '
+                    'Received inputs: {}. '
+                    'Input {} (0-based) originates '
+                    'from layer type `{}`.'.format(inputs,
+                                                   i,
+                                                   layer.__class__.__name__))
             self.input_names.append(layer.name)
             if layer.is_placeholder:
                 self._feed_input_names.append(layer.name)
@@ -1661,7 +1691,6 @@ class Container(Layer):
                 layer = node.inbound_layers[i]
                 node_index = node.node_indices[i]
                 tensor_index = node.tensor_indices[i]
-                next_node = layer.inbound_nodes[node_index]
                 build_map_of_graph(x, finished_nodes, nodes_in_progress,
                                    layer, node_index, tensor_index)
 
@@ -1679,6 +1708,15 @@ class Container(Layer):
             # If the depth is not set, the node has no outbound nodes (depth 0).
             depth = nodes_depths.setdefault(node, 0)
 
+            # Update the depth of the corresponding layer
+            previous_depth = layers_depths.get(node.outbound_layer, 0)
+            # If we've seen this layer before at a higher depth, we should use that depth instead
+            # of the node depth.  This is necessary for shared layers that have inputs at different
+            # depth levels in the graph.
+            depth = max(depth, previous_depth)
+            layers_depths[node.outbound_layer] = depth
+            nodes_depths[node] = depth
+
             # Update the depth of inbound nodes.
             for i in range(len(node.inbound_layers)):
                 inbound_layer = node.inbound_layers[i]
@@ -1687,10 +1725,6 @@ class Container(Layer):
                 previous_depth = nodes_depths.get(inbound_node, 0)
                 nodes_depths[inbound_node] = max(depth + 1, previous_depth)
 
-            # Update the depth of the corresponding layer
-            previous_depth = layers_depths.get(node.outbound_layer, 0)
-            layers_depths[node.outbound_layer] = max(depth, previous_depth)
-
         # Build a dict {depth: list of nodes with this depth}
         nodes_by_depth = {}
         for node, depth in nodes_depths.items():
@@ -2889,16 +2923,20 @@ def preprocess_weights_for_loading(layer, weights,
                                                     (2, 3, 1, 0))
                 weights = [kernel, recurrent_kernel, bias]
 
-    if original_backend and K.backend() != original_backend:
-        conv_layers = ['Conv1D',
-                       'Conv2D',
-                       'Conv3D',
-                       'Conv2DTranspose']
-        if layer.__class__.__name__ in conv_layers:
+    conv_layers = ['Conv1D',
+                   'Conv2D',
+                   'Conv3D',
+                   'Conv2DTranspose',
+                   'ConvLSTM2D']
+    if layer.__class__.__name__ in conv_layers:
+        if original_backend and K.backend() != original_backend:
             weights[0] = conv_utils.convert_kernel(weights[0])
-        if layer.__class__.__name__ == 'ConvLSTM2D':
-            weights[0] = conv_utils.convert_kernel(weights[0])
-            weights[1] = conv_utils.convert_kernel(weights[1])
+            if layer.__class__.__name__ == 'ConvLSTM2D':
+                weights[1] = conv_utils.convert_kernel(weights[1])
+        if K.int_shape(layer.weights[0]) != weights[0].shape:
+            weights[0] = np.transpose(weights[0], (3, 2, 0, 1))
+            if layer.__class__.__name__ == 'ConvLSTM2D':
+                weights[1] = np.transpose(weights[1], (3, 2, 0, 1))
     return weights
 
 

keras/engine/training.py

@@ -241,7 +241,7 @@ def _check_array_lengths(inputs, targets, weights):
 
 
 def _check_loss_and_target_compatibility(targets, loss_fns, output_shapes):
-    """Does validation on the compatiblity of targets and loss functions.
+    """Does validation on the compatibility of targets and loss functions.
 
     This helps prevent users from using loss functions incorrectly.
 
@@ -709,7 +709,8 @@ class Model(Container):
                 `sample_weight_mode` on each output by passing a
                 dictionary or a list of modes.
             **kwargs: when using the Theano backend, these arguments
-                are passed into K.function. Ignored for Tensorflow backend.
+                are passed into K.function. When using the Tensorflow backend,
+                these arguments are passed into `tf.Session.run`.
 
         # Raises
             ValueError: In case of invalid arguments for
@@ -1016,6 +1017,7 @@ class Model(Container):
             self.train_function = K.function(inputs,
                                              [self.total_loss] + self.metrics_tensors,
                                              updates=updates,
+                                             name='train_function',
                                              **self._function_kwargs)
 
     def _make_test_function(self):
@@ -1030,6 +1032,7 @@ class Model(Container):
             self.test_function = K.function(inputs,
                                             [self.total_loss] + self.metrics_tensors,
                                             updates=self.state_updates,
+                                            name='test_function',
                                             **self._function_kwargs)
 
     def _make_predict_function(self):
@@ -1046,6 +1049,7 @@ class Model(Container):
             self.predict_function = K.function(inputs,
                                                self.outputs,
                                                updates=self.state_updates,
+                                               name='predict_function',
                                                **kwargs)
 
     def _fit_loop(self, f, ins, out_labels=None, batch_size=32,
@@ -1156,6 +1160,8 @@ class Model(Container):
                     batch_logs[l] = o
 
                 callbacks.on_batch_end(batch_index, batch_logs)
+                if callback_model.stop_training:
+                    break
 
                 if batch_index == len(batches) - 1:  # Last batch.
                     if do_validation:
@@ -1456,7 +1462,10 @@ class Model(Container):
 
         elif validation_split and 0. < validation_split < 1.:
             do_validation = True
-            split_at = int(len(x[0]) * (1. - validation_split))
+            if hasattr(x[0], 'shape'):
+                split_at = int(x[0].shape[0] * (1. - validation_split))
+            else:
+                split_at = int(len(x[0]) * (1. - validation_split))
             x, val_x = (_slice_arrays(x, 0, split_at), _slice_arrays(x, split_at))
             y, val_y = (_slice_arrays(y, 0, split_at), _slice_arrays(y, split_at))
             sample_weights, val_sample_weights = (
@@ -1836,8 +1845,11 @@ class Model(Container):
                                  str(validation_data))
             val_x, val_y, val_sample_weights = self._standardize_user_data(
                 val_x, val_y, val_sample_weight)
+            val_data = val_x + val_y + val_sample_weights
+            if self.uses_learning_phase and not isinstance(K.learning_phase(), int):
+                val_data += [0.]
             for cbk in callbacks:
-                cbk.validation_data = val_x + [val_y, val_sample_weights]
+                cbk.validation_data = val_data
         enqueuer = None
 
         try:

keras/layers/advanced_activations.py

@@ -202,7 +202,7 @@ class ThresholdedReLU(Layer):
         self.theta = K.cast_to_floatx(theta)
 
     def call(self, inputs, mask=None):
-        return inputs * K.cast(inputs > self.theta, K.floatx())
+        return inputs * K.cast(K.greater(inputs, self.theta), K.floatx())
 
     def get_config(self):
         config = {'theta': float(self.theta)}

keras/layers/convolutional.py

@@ -256,6 +256,9 @@ class Conv1D(_Conv):
             Specifying any stride value != 1 is incompatible with specifying
             any `dilation_rate` value != 1.
         padding: One of `"valid"`, `"causal"` or `"same"` (case-insensitive).
+            `"valid"` means "no padding".
+            `"same"` results in padding the input such that
+            the output has the same length as the original input.
             `"causal"` results in causal (dilated) convolutions, e.g. output[t]
             does not depend on input[t+1:]. Useful when modeling temporal data
             where the model should not violate the temporal order.
@@ -1252,7 +1255,7 @@ class ZeroPadding1D(Layer):
 class ZeroPadding2D(Layer):
     """Zero-padding layer for 2D input (e.g. picture).
 
-    This layer can add rows and columns or zeros
+    This layer can add rows and columns of zeros
     at the top, bottom, left and right side of an image tensor.
 
     # Arguments
@@ -1430,15 +1433,15 @@ class ZeroPadding3D(Layer):
     def compute_output_shape(self, input_shape):
         if self.data_format == 'channels_first':
             if input_shape[2] is not None:
-                dim1 = input_shape[2] + 2 * self.padding[0][0]
+                dim1 = input_shape[2] + self.padding[0][0] + self.padding[0][1]
             else:
                 dim1 = None
             if input_shape[3] is not None:
-                dim2 = input_shape[3] + 2 * self.padding[1][0]
+                dim2 = input_shape[3] + self.padding[1][0] + self.padding[1][1]
             else:
                 dim2 = None
             if input_shape[4] is not None:
-                dim3 = input_shape[4] + 2 * self.padding[2][0]
+                dim3 = input_shape[4] + self.padding[2][0] + self.padding[2][1]
             else:
                 dim3 = None
             return (input_shape[0],
@@ -1448,15 +1451,15 @@ class ZeroPadding3D(Layer):
                     dim3)
         elif self.data_format == 'channels_last':
             if input_shape[1] is not None:
-                dim1 = input_shape[1] + 2 * self.padding[0][1]
+                dim1 = input_shape[1] + self.padding[0][0] + self.padding[0][1]
             else:
                 dim1 = None
             if input_shape[2] is not None:
-                dim2 = input_shape[2] + 2 * self.padding[1][1]
+                dim2 = input_shape[2] + self.padding[1][0] + self.padding[1][1]
             else:
                 dim2 = None
             if input_shape[3] is not None:
-                dim3 = input_shape[3] + 2 * self.padding[2][1]
+                dim3 = input_shape[3] + self.padding[2][0] + self.padding[2][1]
             else:
                 dim3 = None
             return (input_shape[0],

keras/layers/convolutional_recurrent.py

@@ -105,9 +105,12 @@ class ConvRecurrent2D(Recurrent):
         self.return_sequences = return_sequences
         self.go_backwards = go_backwards
         self.stateful = stateful
-        self.input_spec = InputSpec(ndim=5)
+        self.input_spec = [InputSpec(ndim=5)]
+        self.state_spec = None
 
     def compute_output_shape(self, input_shape):
+        if isinstance(input_shape, list):
+            input_shape = input_shape[0]
         if self.data_format == 'channels_first':
             rows = input_shape[3]
             cols = input_shape[4]
@@ -328,11 +331,13 @@ class ConvLSTM2D(ConvRecurrent2D):
 
         self.dropout = min(1., max(0., dropout))
         self.recurrent_dropout = min(1., max(0., recurrent_dropout))
+        self.state_spec = [InputSpec(ndim=4), InputSpec(ndim=4)]
 
     def build(self, input_shape):
-        # TODO: better handling of input spec
-        self.input_spec = InputSpec(shape=input_shape)
-
+        if isinstance(input_shape, list):
+            input_shape = input_shape[0]
+        batch_size = input_shape[0] if self.stateful else None
+        self.input_spec[0] = InputSpec(shape=(batch_size, None) + input_shape[2:])
         if self.stateful:
             self.reset_states()
         else:
@@ -347,6 +352,10 @@ class ConvLSTM2D(ConvRecurrent2D):
             raise ValueError('The channel dimension of the inputs '
                              'should be defined. Found `None`.')
         input_dim = input_shape[channel_axis]
+        state_shape = [None] * 4
+        state_shape[channel_axis] = input_dim
+        state_shape = tuple(state_shape)
+        self.state_spec = [InputSpec(shape=state_shape), InputSpec(shape=state_shape)]
         kernel_shape = self.kernel_size + (input_dim, self.filters * 4)
         self.kernel_shape = kernel_shape
         recurrent_kernel_shape = self.kernel_size + (self.filters, self.filters * 4)
@@ -413,7 +422,7 @@ class ConvLSTM2D(ConvRecurrent2D):
     def reset_states(self):
         if not self.stateful:
             raise RuntimeError('Layer must be stateful.')
-        input_shape = self.input_spec.shape
+        input_shape = self.input_spec[0].shape
         output_shape = self.compute_output_shape(input_shape)
         if not input_shape[0]:
             raise ValueError('If a RNN is stateful, a complete '

keras/layers/core.py

@@ -299,13 +299,13 @@ class Reshape(Layer):
     """Reshapes an output to a certain shape.
 
     # Arguments
-        target_shape: target shape. Tuple of integers,
-            does not include the samples dimension (batch size).
+        target_shape: target shape. Tuple of integers.
+            Does not include the batch axis.
 
     # Input shape
         Arbitrary, although all dimensions in the input shaped must be fixed.
         Use the keyword argument `input_shape`
-        (tuple of integers, does not include the samples axis)
+        (tuple of integers, does not include the batch axis)
         when using this layer as the first layer in a model.
 
     # Output shape
@@ -335,27 +335,22 @@ class Reshape(Layer):
         self.target_shape = tuple(target_shape)
 
     def _fix_unknown_dimension(self, input_shape, output_shape):
-        """Find and replace a missing dimension in an output shape.
+        """Finds and replaces a missing dimension in an output shape.
 
         This is a near direct port of the internal Numpy function
         `_fix_unknown_dimension` in `numpy/core/src/multiarray/shape.c`
 
         # Arguments
-            input_shape: shape of array being reshaped
-            output_shape: desired shape of the array with at most
+            input_shape: original shape of array being reshaped
+            output_shape: target shape of the array, with at most
                 a single -1 which indicates a dimension that should be
                 derived from the input shape.
 
         # Returns
-            The new output shape with a -1 replaced with its computed value.
-
-            Raises a ValueError if the total array size of the output_shape is
-            different then the input_shape, or more then one unknown dimension
-            is specified.
+            The new output shape with a `-1` replaced with its computed value.
 
         # Raises
-            ValueError: in case of invalid values
-                for `input_shape` or `input_shape`.
+            ValueError: if `input_shape` and `output_shape` do not match.
         """
         output_shape = list(output_shape)
         msg = 'total size of new array must be unchanged'
@@ -386,13 +381,11 @@ class Reshape(Layer):
 
     def call(self, inputs):
         # In case the target shape is not fully defined,
-        # we need access to the shape of x.
-        # solution:
-        # 1) rely on x._keras_shape
-        # 2) fallback: K.int_shape
+        # we need access to the shape of `inputs`.
+        # solution: rely on `K.int_shape`.
         target_shape = self.target_shape
         if -1 in target_shape:
-            # target shape not fully defined
+            # Target shape not fully defined.
             input_shape = None
             try:
                 input_shape = K.int_shape(inputs)
@@ -720,6 +713,16 @@ class Lambda(Layer):
         else:
             output_shape = config['output_shape']
 
+        # If arguments were numpy array, they have been saved as
+        # list. We need to recover the ndarray
+        if 'arguments' in config:
+            for key in config['arguments']:
+                if isinstance(config['arguments'][key], dict):
+                    arg_dict = config['arguments'][key]
+                    if 'type' in arg_dict and arg_dict['type'] == 'ndarray':
+                        # Overwrite the argument with its numpy translation
+                        config['arguments'][key] = np.array(arg_dict['value'])
+
         config['function'] = function
         config['output_shape'] = output_shape
         return cls(**config)

keras/layers/embeddings.py

@@ -108,11 +108,25 @@ class Embedding(Layer):
             return K.not_equal(inputs, 0)
 
     def compute_output_shape(self, input_shape):
-        if not self.input_length:
-            input_length = input_shape[1]
+        if self.input_length is None:
+            return input_shape + (self.output_dim,)
         else:
-            input_length = self.input_length
-        return (input_shape[0], input_length, self.output_dim)
+            # input_length can be tuple if input is 3D or higher
+            if isinstance(self.input_length, (list, tuple)):
+                in_lens = list(self.input_length)
+            else:
+                in_lens = [self.input_length]
+            if len(in_lens) != len(input_shape) - 1:
+                ValueError('"input_length" is %s, but received input has shape %s' %
+                           (str(self.input_length), str(input_shape)))
+            else:
+                for i, (s1, s2) in enumerate(zip(in_lens, input_shape[1:])):
+                    if s1 is not None and s2 is not None and s1 != s2:
+                        ValueError('"input_length" is %s, but received input has shape %s' %
+                                   (str(self.input_length), str(input_shape)))
+                    elif s1 is None:
+                        in_lens[i] = s2
+            return (input_shape[0],) + tuple(in_lens) + (self.output_dim,)
 
     def call(self, inputs):
         if K.dtype(inputs) != 'int32':

keras/layers/local.py

@@ -147,22 +147,11 @@ class LocallyConnected1D(Layer):
         return (input_shape[0], length, self.filters)
 
     def call(self, inputs):
-        stride = self.strides[0]
-        output_length, feature_dim, filters = self.kernel_shape
-
-        xs = []
-        for i in range(output_length):
-            slice_length = slice(i * stride,
-                                 i * stride + self.kernel_size[0])
-            xs.append(K.reshape(inputs[:, slice_length, :],
-                                (1, -1, feature_dim)))
-        x_aggregate = K.concatenate(xs, axis=0)
-        # Shape: `(output_length, batch_size, filters)`.
-        output = K.batch_dot(x_aggregate, self.kernel)
-        output = K.permute_dimensions(output, (1, 0, 2))
+        output_length, _, filters = self.kernel_shape
 
+        output = K.local_conv1d(inputs, self.kernel, self.kernel_size, self.strides)
         if self.use_bias:
-            output += K.reshape(self.bias, (1, output_length, filters))
+            output = K.bias_add(output, self.bias)
         if self.activation is not None:
             output = self.activation(output)
         return output
@@ -363,62 +352,19 @@ class LocallyConnected2D(Layer):
             return (input_shape[0], rows, cols, self.filters)
 
     def call(self, inputs):
-        stride_row, stride_col = self.strides
-        _, feature_dim, filters = self.kernel_shape
+        _, _, filters = self.kernel_shape
 
-        if self.data_format == 'channels_first':
-            if K.backend() == 'theano':
-                output = []
-                for i in range(self.output_row):
-                    for j in range(self.output_col):
-                        slice_row = slice(i * stride_row,
-                                          i * stride_row + self.kernel_size[0])
-                        slice_col = slice(j * stride_col,
-                                          j * stride_col + self.kernel_size[1])
-                        x_flatten = K.reshape(inputs[:, :, slice_row, slice_col],
-                                              (1, -1, feature_dim))
-                        output.append(K.dot(x_flatten,
-                                      self.kernel[i * self.output_col + j, :, :]))
-                output = K.concatenate(output, axis=0)
-            else:
-                xs = []
-                for i in range(self.output_row):
-                    for j in range(self.output_col):
-                        slice_row = slice(i * stride_row,
-                                          i * stride_row + self.kernel_size[0])
-                        slice_col = slice(j * stride_col,
-                                          j * stride_col + self.kernel_size[1])
-                        xs.append(K.reshape(inputs[:, :, slice_row, slice_col],
-                                            (1, -1, feature_dim)))
-                x_aggregate = K.concatenate(xs, axis=0)
-                output = K.batch_dot(x_aggregate, self.kernel)
-            output = K.reshape(output,
-                               (self.output_row, self.output_col, -1, filters))
-            output = K.permute_dimensions(output, (2, 3, 0, 1))
-
-        elif self.data_format == 'channels_last':
-            xs = []
-            for i in range(self.output_row):
-                for j in range(self.output_col):
-                    slice_row = slice(i * stride_row,
-                                      i * stride_row + self.kernel_size[0])
-                    slice_col = slice(j * stride_col,
-                                      j * stride_col + self.kernel_size[1])
-                    xs.append(K.reshape(inputs[:, slice_row, slice_col, :],
-                                        (1, -1, feature_dim)))
-            x_aggregate = K.concatenate(xs, axis=0)
-            output = K.batch_dot(x_aggregate, self.kernel)
-            output = K.reshape(output,
-                               (self.output_row, self.output_col, -1, filters))
-            output = K.permute_dimensions(output, (2, 0, 1, 3))
+        output = K.local_conv2d(inputs,
+                                self.kernel,
+                                self.kernel_size,
+                                self.strides,
+                                (self.output_row, self.output_col),
+                                self.data_format)
 
         if self.use_bias:
-            if self.data_format == 'channels_first':
-                output += K.reshape(self.bias,
-                                    (1, filters, self.output_row, self.output_col))
-            elif self.data_format == 'channels_last':
-                output += K.reshape(self.bias,
-                                    (1, self.output_row, self.output_col, filters))
+            if self.data_format == 'channels_first' or self.data_format == 'channels_last':
+                output = K.bias_add(output, self.bias, data_format=self.data_format)
+
         output = self.activation(output)
         return output
 

keras/layers/recurrent.py

@@ -96,6 +96,8 @@ class Recurrent(Layer):
             `[(input_dim, output_dim), (output_dim, output_dim), (output_dim,)]`.
         return_sequences: Boolean. Whether to return the last output
             in the output sequence, or the full sequence.
+        return_state: Boolean. Whether to return the last state
+            in addition to the output.
         go_backwards: Boolean (default False).
             If True, process the input sequence backwards and return the
             reversed sequence.
@@ -139,6 +141,9 @@ class Recurrent(Layer):
         (Optional) 2D tensors with shape `(batch_size, output_dim)`.
 
     # Output shape
+        - if `return_state`: a list of tensors. The first tensor is
+            the output. The remaining tensors are the last states,
+            each with shape `(batch_size, units)`.
         - if `return_sequences`: 3D tensor with shape
             `(batch_size, timesteps, units)`.
         - else, 2D tensor with shape `(batch_size, units)`.
@@ -183,6 +188,7 @@ class Recurrent(Layer):
     """
 
     def __init__(self, return_sequences=False,
+                 return_state=False,
                  go_backwards=False,
                  stateful=False,
                  unroll=False,
@@ -190,7 +196,11 @@ class Recurrent(Layer):
                  **kwargs):
         super(Recurrent, self).__init__(**kwargs)
         self.return_sequences = return_sequences
+        self.return_state = return_state
         self.go_backwards = go_backwards
+        if K.backend() == 'cntk' and stateful:
+            raise ValueError('Stateful RNN is not currently supported with CNTK.')
+
         self.stateful = stateful
         self.unroll = unroll
         self.implementation = implementation
@@ -203,18 +213,27 @@ class Recurrent(Layer):
     def compute_output_shape(self, input_shape):
         if isinstance(input_shape, list):
             input_shape = input_shape[0]
+
         if self.return_sequences:
-            return (input_shape[0], input_shape[1], self.units)
+            output_shape = (input_shape[0], input_shape[1], self.units)
         else:
-            return (input_shape[0], self.units)
+            output_shape = (input_shape[0], self.units)
+
+        if self.return_state:
+            state_shape = [(input_shape[0], self.units) for _ in self.states]
+            return [output_shape] + state_shape
+        else:
+            return output_shape
 
     def compute_mask(self, inputs, mask):
-        if self.return_sequences:
-            if isinstance(mask, list):
-                return mask[0]
-            return mask
+        if isinstance(mask, list):
+            mask = mask[0]
+        output_mask = mask if self.return_sequences else None
+        if self.return_state:
+            state_mask = [None for _ in self.states]
+            return [output_mask] + state_mask
         else:
-            return None
+            return output_mask
 
     def step(self, inputs, states):
         raise NotImplementedError
@@ -256,6 +275,8 @@ class Recurrent(Layer):
             # Compute the full input spec, including state
             input_spec = self.input_spec
             state_spec = self.state_spec
+            if not isinstance(input_spec, list):
+                input_spec = [input_spec]
             if not isinstance(state_spec, list):
                 state_spec = [state_spec]
             self.input_spec = input_spec + state_spec
@@ -330,9 +351,18 @@ class Recurrent(Layer):
             outputs._uses_learning_phase = True
 
         if self.return_sequences:
-            return outputs
+            output = outputs
         else:
-            return last_output
+            output = last_output
+
+        if self.return_state:
+            if not isinstance(states, (list, tuple)):
+                states = [states]
+            else:
+                states = list(states)
+            return [output] + states
+        else:
+            return output
 
     def reset_states(self, states=None):
         if not self.stateful:
@@ -376,6 +406,7 @@ class Recurrent(Layer):
 
     def get_config(self):
         config = {'return_sequences': self.return_sequences,
+                  'return_state': self.return_state,
                   'go_backwards': self.go_backwards,
                   'stateful': self.stateful,
                   'unroll': self.unroll,

keras/legacy/interfaces.py

@@ -161,7 +161,7 @@ def recurrent_args_preprocessor(args, kwargs):
             kwargs.pop('forget_bias_init')
             warnings.warn('The `forget_bias_init` argument '
                           'has been ignored. Use `unit_forget_bias=True` '
-                          'instead to intialize with ones.', stacklevel=3)
+                          'instead to initialize with ones.', stacklevel=3)
     if 'input_dim' in kwargs:
         input_length = kwargs.pop('input_length', None)
         input_dim = kwargs.pop('input_dim')
@@ -461,7 +461,7 @@ def convlstm2d_args_preprocessor(args, kwargs):
         else:
             warnings.warn('The `forget_bias_init` argument '
                           'has been ignored. Use `unit_forget_bias=True` '
-                          'instead to intialize with ones.', stacklevel=3)
+                          'instead to initialize with ones.', stacklevel=3)
     args, kwargs, _converted = conv2d_args_preprocessor(args, kwargs)
     return args, kwargs, converted + _converted
 

keras/losses.py

@@ -33,6 +33,12 @@ def hinge(y_true, y_pred):
     return K.mean(K.maximum(1. - y_true * y_pred, 0.), axis=-1)
 
 
+def categorical_hinge(y_true, y_pred):
+    pos = K.sum(y_true * y_pred, axis=-1)
+    neg = K.max((1. - y_true) * y_pred, axis=-1)
+    return K.maximum(0., neg - pos + 1.)
+
+
 def logcosh(y_true, y_pred):
     def cosh(x):
         return (K.exp(x) + K.exp(-x)) / 2

keras/metrics.py

@@ -36,6 +36,11 @@ def sparse_categorical_accuracy(y_true, y_pred):
 def top_k_categorical_accuracy(y_true, y_pred, k=5):
     return K.mean(K.in_top_k(y_pred, K.argmax(y_true, axis=-1), k), axis=-1)
 
+
+def sparse_top_k_categorical_accuracy(y_true, y_pred, k=5):
+    return K.mean(K.in_top_k(y_pred, K.cast(K.max(y_true, axis=-1), 'int32'), k), axis=-1)
+
+
 # Aliases
 
 mse = MSE = mean_squared_error

keras/models.py

@@ -74,7 +74,11 @@ def save_model(model, filepath, overwrite=True, include_optimizer=True):
 
         # if obj is any numpy type
         if type(obj).__module__ == np.__name__:
-            return obj.item()
+            if isinstance(obj, np.ndarray):
+                return {'type': type(obj),
+                        'value': obj.tolist()}
+            else:
+                return obj.item()
 
         # misc functions (e.g. loss function)
         if callable(obj):
@@ -140,8 +144,8 @@ def save_model(model, filepath, overwrite=True, include_optimizer=True):
                 weight_values = K.batch_get_value(symbolic_weights)
                 weight_names = []
                 for i, (w, val) in enumerate(zip(symbolic_weights, weight_values)):
-                    # Default values of symbolic_weights is /variable for theano
-                    if K.backend() == 'theano':
+                    # Default values of symbolic_weights is /variable for theano and cntk
+                    if K.backend() == 'theano' or K.backend() == 'cntk':
                         if hasattr(w, 'name') and w.name != "/variable":
                             name = str(w.name)
                         else:
@@ -167,7 +171,7 @@ def save_model(model, filepath, overwrite=True, include_optimizer=True):
     f.close()
 
 
-def load_model(filepath, custom_objects=None):
+def load_model(filepath, custom_objects=None, compile=True):
     """Loads a model saved via `save_model`.
 
     # Arguments
@@ -175,12 +179,16 @@ def load_model(filepath, custom_objects=None):
         custom_objects: Optional dictionary mapping names
             (strings) to custom classes or functions to be
             considered during deserialization.
+        compile: Boolean, whether to compile the model
+            after loading.
 
     # Returns
         A Keras model instance. If an optimizer was found
         as part of the saved model, the model is already
         compiled. Otherwise, the model is uncompiled and
-        a warning will be displayed.
+        a warning will be displayed. When `compile` is set
+        to False, the compilation is omitted without any
+        warning.
 
     # Raises
         ImportError: if h5py is not available.
@@ -229,56 +237,58 @@ def load_model(filepath, custom_objects=None):
         if obj in custom_objects:
             return custom_objects[obj]
         return obj
+    with h5py.File(filepath, mode='r') as f:
+        # instantiate model
+        model_config = f.attrs.get('model_config')
+        if model_config is None:
+            raise ValueError('No model found in config file.')
+        model_config = json.loads(model_config.decode('utf-8'))
+        model = model_from_config(model_config, custom_objects=custom_objects)
 
-    f = h5py.File(filepath, mode='r')
+        # set weights
+        topology.load_weights_from_hdf5_group(f['model_weights'], model.layers)
 
-    # instantiate model
-    model_config = f.attrs.get('model_config')
-    if model_config is None:
-        raise ValueError('No model found in config file.')
-    model_config = json.loads(model_config.decode('utf-8'))
-    model = model_from_config(model_config, custom_objects=custom_objects)
+        # Early return if compilation is not required.
+        if not compile:
+            return model
 
-    # set weights
-    topology.load_weights_from_hdf5_group(f['model_weights'], model.layers)
+        # instantiate optimizer
+        training_config = f.attrs.get('training_config')
+        if training_config is None:
+            warnings.warn('No training configuration found in save file: '
+                          'the model was *not* compiled. Compile it manually.')
+            return model
+        training_config = json.loads(training_config.decode('utf-8'))
+        optimizer_config = training_config['optimizer_config']
+        optimizer = optimizers.deserialize(optimizer_config,
+                                           custom_objects=custom_objects)
 
-    # instantiate optimizer
-    training_config = f.attrs.get('training_config')
-    if training_config is None:
-        warnings.warn('No training configuration found in save file: '
-                      'the model was *not* compiled. Compile it manually.')
-        f.close()
-        return model
-    training_config = json.loads(training_config.decode('utf-8'))
-    optimizer_config = training_config['optimizer_config']
-    optimizer = optimizers.deserialize(optimizer_config,
-                                       custom_objects=custom_objects)
+        # Recover loss functions and metrics.
+        loss = convert_custom_objects(training_config['loss'])
+        metrics = convert_custom_objects(training_config['metrics'])
+        sample_weight_mode = training_config['sample_weight_mode']
+        loss_weights = training_config['loss_weights']
 
-    # Recover loss functions and metrics.
-    loss = convert_custom_objects(training_config['loss'])
-    metrics = convert_custom_objects(training_config['metrics'])
-    sample_weight_mode = training_config['sample_weight_mode']
-    loss_weights = training_config['loss_weights']
+        # Compile model.
+        model.compile(optimizer=optimizer,
+                      loss=loss,
+                      metrics=metrics,
+                      loss_weights=loss_weights,
+                      sample_weight_mode=sample_weight_mode)
 
-    # Compile model.
-    model.compile(optimizer=optimizer,
-                  loss=loss,
-                  metrics=metrics,
-                  loss_weights=loss_weights,
-                  sample_weight_mode=sample_weight_mode)
-
-    # Set optimizer weights.
-    if 'optimizer_weights' in f:
-        # Build train function (to get weight updates).
-        if isinstance(model, Sequential):
-            model.model._make_train_function()
-        else:
-            model._make_train_function()
-        optimizer_weights_group = f['optimizer_weights']
-        optimizer_weight_names = [n.decode('utf8') for n in optimizer_weights_group.attrs['weight_names']]
-        optimizer_weight_values = [optimizer_weights_group[n] for n in optimizer_weight_names]
-        model.optimizer.set_weights(optimizer_weight_values)
-    f.close()
+        # Set optimizer weights.
+        if 'optimizer_weights' in f:
+            # Build train function (to get weight updates).
+            if isinstance(model, Sequential):
+                model.model._make_train_function()
+            else:
+                model._make_train_function()
+            optimizer_weights_group = f['optimizer_weights']
+            optimizer_weight_names = [n.decode('utf8') for n in
+                                      optimizer_weights_group.attrs['weight_names']]
+            optimizer_weight_values = [optimizer_weights_group[n] for n in
+                                       optimizer_weight_names]
+            model.optimizer.set_weights(optimizer_weight_values)
     return model
 
 
@@ -295,7 +305,7 @@ def model_from_config(config, custom_objects=None):
         A Keras model instance (uncompiled).
 
     # Raises
-        TypeError if `config` is not a dictionary
+        TypeError: if `config` is not a dictionary.
     """
     if isinstance(config, list):
         raise TypeError('`model_from_config` expects a dictionary, not a list. '
@@ -756,7 +766,8 @@ class Sequential(Model):
                 sample weighting (2D weights), set this to "temporal".
                 "None" defaults to sample-wise weights (1D).
             **kwargs: for Theano backend, these are passed into K.function.
-                Ignored for Tensorflow backend.
+                When using the Tensorflow backend, these are passed into
+                `tf.Session.run`.
 
         # Example
             ```python
@@ -777,11 +788,14 @@ class Sequential(Model):
                            **kwargs)
         self.optimizer = self.model.optimizer
         self.loss = self.model.loss
+        self.total_loss = self.model.total_loss
         self.loss_weights = self.model.loss_weights
         self.metrics = self.model.metrics
         self.metrics_tensors = self.model.metrics_tensors
         self.metrics_names = self.model.metrics_names
         self.sample_weight_mode = self.model.sample_weight_mode
+        self.sample_weights = self.model.sample_weights
+        self.targets = self.model.targets
 
     def fit(self, x, y, batch_size=32, epochs=10, verbose=1, callbacks=None,
             validation_split=0., validation_data=None, shuffle=True,

keras/optimizers.py

@@ -1,5 +1,6 @@
 from __future__ import absolute_import
 import six
+import copy
 from six.moves import zip
 
 from . import backend as K
@@ -11,8 +12,31 @@ if K.backend() == 'tensorflow':
 
 
 def clip_norm(g, c, n):
-    if c > 0:
-        g = K.switch(n >= c, g * c / n, g)
+    if c <= 0:  # if clipnorm == 0 no need to add ops to the graph
+        return g
+
+    # tf require using a special op to multiply IndexedSliced by scalar
+    if K.backend() == 'tensorflow':
+        condition = n >= c
+        then_expression = tf.scalar_mul(c / n, g)
+        else_expression = g
+
+        # saving the shape to avoid converting sparse tensor to dense
+        if isinstance(then_expression, tf.Tensor):
+            g_shape = copy.copy(then_expression.get_shape())
+        elif isinstance(then_expression, tf.IndexedSlices):
+            g_shape = copy.copy(then_expression.dense_shape)
+        if condition.dtype != tf.bool:
+            condition = tf.cast(condition, 'bool')
+        g = tf.cond(condition,
+                    lambda: then_expression,
+                    lambda: else_expression)
+        if isinstance(then_expression, tf.Tensor):
+            g.set_shape(g_shape)
+        elif isinstance(then_expression, tf.IndexedSlices):
+            g._dense_shape = g_shape
+    else:
+        g = K.switch(K.greater_equal(n, c), g * c / n, g)
     return g
 
 

keras/preprocessing/image.py

@@ -13,6 +13,8 @@ from six.moves import range
 import os
 import threading
 import warnings
+import multiprocessing.pool
+from functools import partial
 
 from .. import backend as K
 
@@ -346,6 +348,7 @@ class ImageDataGenerator(object):
         featurewise_std_normalization: divide inputs by std of the dataset.
         samplewise_std_normalization: divide each input by its std.
         zca_whitening: apply ZCA whitening.
+        zca_epsilon: epsilon for ZCA whitening. Default is 1e-6.
         rotation_range: degrees (0 to 180).
         width_shift_range: fraction of total width.
         height_shift_range: fraction of total height.
@@ -382,6 +385,7 @@ class ImageDataGenerator(object):
                  featurewise_std_normalization=False,
                  samplewise_std_normalization=False,
                  zca_whitening=False,
+                 zca_epsilon=1e-6,
                  rotation_range=0.,
                  width_shift_range=0.,
                  height_shift_range=0.,
@@ -402,6 +406,7 @@ class ImageDataGenerator(object):
         self.featurewise_std_normalization = featurewise_std_normalization
         self.samplewise_std_normalization = samplewise_std_normalization
         self.zca_whitening = zca_whitening
+        self.zca_epsilon = zca_epsilon
         self.rotation_range = rotation_range
         self.width_shift_range = width_shift_range
         self.height_shift_range = height_shift_range
@@ -443,7 +448,7 @@ class ImageDataGenerator(object):
                              'Received arg: ', zoom_range)
 
     def flow(self, x, y=None, batch_size=32, shuffle=True, seed=None,
-             save_to_dir=None, save_prefix='', save_format='jpeg'):
+             save_to_dir=None, save_prefix='', save_format='png'):
         return NumpyArrayIterator(
             x, y, self,
             batch_size=batch_size,
@@ -460,7 +465,7 @@ class ImageDataGenerator(object):
                             batch_size=32, shuffle=True, seed=None,
                             save_to_dir=None,
                             save_prefix='',
-                            save_format='jpeg',
+                            save_format='png',
                             follow_links=False):
         return DirectoryIterator(
             directory, self,
@@ -633,8 +638,8 @@ class ImageDataGenerator(object):
         if x.ndim != 4:
             raise ValueError('Input to `.fit()` should have rank 4. '
                              'Got array with shape: ' + str(x.shape))
-        if x.shape[self.channel_axis] not in {1, 3, 4}:
-            raise ValueError(
+        if x.shape[self.channel_axis] not in {3, 4}:
+            warnings.warn(
                 'Expected input to be images (as Numpy array) '
                 'following the data format convention "' + self.data_format + '" '
                 '(channels on axis ' + str(self.channel_axis) + '), i.e. expected '
@@ -671,7 +676,7 @@ class ImageDataGenerator(object):
             flat_x = np.reshape(x, (x.shape[0], x.shape[1] * x.shape[2] * x.shape[3]))
             sigma = np.dot(flat_x.T, flat_x) / flat_x.shape[0]
             u, s, _ = linalg.svd(sigma)
-            self.principal_components = np.dot(np.dot(u, np.diag(1. / np.sqrt(s + 10e-7))), u.T)
+            self.principal_components = np.dot(np.dot(u, np.diag(1. / np.sqrt(s + self.zca_epsilon))), u.T)
 
 
 class Iterator(object):
@@ -752,7 +757,7 @@ class NumpyArrayIterator(Iterator):
     def __init__(self, x, y, image_data_generator,
                  batch_size=32, shuffle=False, seed=None,
                  data_format=None,
-                 save_to_dir=None, save_prefix='', save_format='jpeg'):
+                 save_to_dir=None, save_prefix='', save_format='png'):
         if y is not None and len(x) != len(y):
             raise ValueError('X (images tensor) and y (labels) '
                              'should have the same length. '
@@ -818,6 +823,73 @@ class NumpyArrayIterator(Iterator):
         return batch_x, batch_y
 
 
+def _count_valid_files_in_directory(directory, white_list_formats, follow_links):
+    """Count files with extension in `white_list_formats` contained in a directory.
+
+    # Arguments
+        directory: absolute path to the directory containing files to be counted
+        white_list_formats: set of strings containing allowed extensions for
+            the files to be counted.
+
+    # Returns
+        the count of files with extension in `white_list_formats` contained in
+        the directory.
+    """
+    def _recursive_list(subpath):
+        return sorted(os.walk(subpath, followlinks=follow_links), key=lambda tpl: tpl[0])
+
+    samples = 0
+    for root, _, files in _recursive_list(directory):
+        for fname in files:
+            is_valid = False
+            for extension in white_list_formats:
+                if fname.lower().endswith('.' + extension):
+                    is_valid = True
+                    break
+            if is_valid:
+                samples += 1
+    return samples
+
+
+def _list_valid_filenames_in_directory(directory, white_list_formats,
+                                       class_indices, follow_links):
+    """List paths of files in `subdir` relative from `directory` whose extensions are in `white_list_formats`.
+
+    # Arguments
+        directory: absolute path to a directory containing the files to list.
+            The directory name is used as class label and must be a key of `class_indices`.
+        white_list_formats: set of strings containing allowed extensions for
+            the files to be counted.
+        class_indices: dictionary mapping a class name to its index.
+
+    # Returns
+        classes: a list of class indices
+        filenames: the path of valid files in `directory`, relative from
+            `directory`'s parent (e.g., if `directory` is "dataset/class1",
+            the filenames will be ["class1/file1.jpg", "class1/file2.jpg", ...]).
+    """
+    def _recursive_list(subpath):
+        return sorted(os.walk(subpath, followlinks=follow_links), key=lambda tpl: tpl[0])
+
+    classes = []
+    filenames = []
+    subdir = os.path.basename(directory)
+    basedir = os.path.dirname(directory)
+    for root, _, files in _recursive_list(directory):
+        for fname in files:
+            is_valid = False
+            for extension in white_list_formats:
+                if fname.lower().endswith('.' + extension):
+                    is_valid = True
+                    break
+            if is_valid:
+                classes.append(class_indices[subdir])
+                # add filename relative to directory
+                absolute_path = os.path.join(root, fname)
+                filenames.append(os.path.relpath(absolute_path, basedir))
+    return classes, filenames
+
+
 class DirectoryIterator(Iterator):
     """Iterator capable of reading images from a directory on disk.
 
@@ -838,6 +910,8 @@ class DirectoryIterator(Iterator):
             `"binary"`: binary targets (if there are only two classes),
             `"categorical"`: categorical targets,
             `"sparse"`: integer targets,
+            `"input"`: targets are images identical to input images (mainly
+                used to work with autoencoders),
             `None`: no targets get yielded (only input images are yielded).
         batch_size: Integer, size of a batch.
         shuffle: Boolean, whether to shuffle the data between epochs.
@@ -858,7 +932,7 @@ class DirectoryIterator(Iterator):
                  classes=None, class_mode='categorical',
                  batch_size=32, shuffle=True, seed=None,
                  data_format=None,
-                 save_to_dir=None, save_prefix='', save_format='jpeg',
+                 save_to_dir=None, save_prefix='', save_format='png',
                  follow_links=False):
         if data_format is None:
             data_format = K.image_data_format()
@@ -881,10 +955,12 @@ class DirectoryIterator(Iterator):
             else:
                 self.image_shape = (1,) + self.target_size
         self.classes = classes
-        if class_mode not in {'categorical', 'binary', 'sparse', None}:
+        if class_mode not in {'categorical', 'binary', 'sparse',
+                              'input', None}:
             raise ValueError('Invalid class_mode:', class_mode,
                              '; expected one of "categorical", '
-                             '"binary", "sparse", or None.')
+                             '"binary", "sparse", "input"'
+                             ' or None.')
         self.class_mode = class_mode
         self.save_to_dir = save_to_dir
         self.save_prefix = save_prefix
@@ -906,38 +982,33 @@ class DirectoryIterator(Iterator):
         def _recursive_list(subpath):
             return sorted(os.walk(subpath, followlinks=follow_links), key=lambda tpl: tpl[0])
 
-        for subdir in classes:
-            subpath = os.path.join(directory, subdir)
-            for root, _, files in _recursive_list(subpath):
-                for fname in files:
-                    is_valid = False
-                    for extension in white_list_formats:
-                        if fname.lower().endswith('.' + extension):
-                            is_valid = True
-                            break
-                    if is_valid:
-                        self.samples += 1
+        pool = multiprocessing.pool.ThreadPool()
+        function_partial = partial(_count_valid_files_in_directory,
+                                   white_list_formats=white_list_formats,
+                                   follow_links=follow_links)
+        self.samples = sum(pool.map(function_partial,
+                                    (os.path.join(directory, subdir)
+                                     for subdir in classes)))
+
         print('Found %d images belonging to %d classes.' % (self.samples, self.num_class))
 
         # second, build an index of the images in the different class subfolders
+        results = []
+
         self.filenames = []
         self.classes = np.zeros((self.samples,), dtype='int32')
         i = 0
-        for subdir in classes:
-            subpath = os.path.join(directory, subdir)
-            for root, _, files in _recursive_list(subpath):
-                for fname in files:
-                    is_valid = False
-                    for extension in white_list_formats:
-                        if fname.lower().endswith('.' + extension):
-                            is_valid = True
-                            break
-                    if is_valid:
-                        self.classes[i] = self.class_indices[subdir]
-                        i += 1
-                        # add filename relative to directory
-                        absolute_path = os.path.join(root, fname)
-                        self.filenames.append(os.path.relpath(absolute_path, directory))
+        for dirpath in (os.path.join(directory, subdir) for subdir in classes):
+            results.append(pool.apply_async(_list_valid_filenames_in_directory,
+                                            (dirpath, white_list_formats,
+                                             self.class_indices, follow_links)))
+        for res in results:
+            classes, filenames = res.get()
+            self.classes[i:i + len(classes)] = classes
+            self.filenames += filenames
+            i += len(classes)
+        pool.close()
+        pool.join()
         super(DirectoryIterator, self).__init__(self.samples, batch_size, shuffle, seed)
 
     def next(self):
@@ -972,7 +1043,9 @@ class DirectoryIterator(Iterator):
                                                                   format=self.save_format)
                 img.save(os.path.join(self.save_to_dir, fname))
         # build batch of labels
-        if self.class_mode == 'sparse':
+        if self.class_mode == 'input':
+            batch_y = batch_x.copy()
+        elif self.class_mode == 'sparse':
             batch_y = self.classes[index_array]
         elif self.class_mode == 'binary':
             batch_y = self.classes[index_array].astype(K.floatx())

keras/preprocessing/text.py

@@ -11,6 +11,7 @@ import sys
 import numpy as np
 from six.moves import range
 from six.moves import zip
+from collections import OrderedDict
 import warnings
 
 if sys.version_info < (3,):
@@ -22,7 +23,7 @@ else:
 def text_to_word_sequence(text,
                           filters='!"#$%&()*+,-./:;<=>?@[\\]^_`{|}~\t\n',
                           lower=True, split=" "):
-    """Converts a text to a sequence of word indices.
+    """Converts a text to a sequence of words (or tokens).
 
     # Arguments
         text: Input text (string).
@@ -31,7 +32,7 @@ def text_to_word_sequence(text,
         split: Sentence split marker (string).
 
     # Returns
-        A list of integer word indices.
+        A list of words (or tokens).
     """
     if lower:
         text = text.lower()
@@ -68,7 +69,7 @@ class Tokenizer(object):
             tabs and line breaks, minus the `'` character.
         lower: boolean. Whether to convert the texts to lowercase.
         split: character or string to use for token splitting.
-        char_level: if True, every character will be treated as a word.
+        char_level: if True, every character will be treated as a token.
 
     By default, all punctuation is removed, turning the texts into
     space-separated sequences of words
@@ -92,7 +93,7 @@ class Tokenizer(object):
         if kwargs:
             raise TypeError('Unrecognized keyword arguments: ' + str(kwargs))
 
-        self.word_counts = {}
+        self.word_counts = OrderedDict()
         self.word_docs = {}
         self.filters = filters
         self.split = split

keras/utils/conv_utils.py

@@ -70,7 +70,7 @@ def convert_kernel(kernel):
     Also works reciprocally, since the transformation is its own inverse.
 
     # Arguments
-        kernel: Numpy array (4D or 5D).
+        kernel: Numpy array (3D, 4D or 5D).
 
     # Returns
         The converted kernel.
@@ -79,7 +79,7 @@ def convert_kernel(kernel):
         ValueError: in case of invalid kernel shape or invalid data_format.
     """
     kernel = np.asarray(kernel)
-    if not 4 <= kernel.ndim <= 5:
+    if not 3 <= kernel.ndim <= 5:
         raise ValueError('Invalid kernel shape:', kernel.shape)
     slices = [slice(None, None, -1) for _ in range(kernel.ndim)]
     no_flip = (slice(None, None), slice(None, None))

keras/utils/data_utils.py

@@ -2,7 +2,6 @@
 from __future__ import absolute_import
 from __future__ import print_function
 
-import functools
 import tarfile
 import zipfile
 import os
@@ -36,8 +35,10 @@ if sys.version_info[0] == 2:
             data: `data` argument passed to `urlopen`.
         """
         def chunk_read(response, chunk_size=8192, reporthook=None):
-            total_size = response.info().get('Content-Length').strip()
-            total_size = int(total_size)
+            content_type = response.info().get('Content-Length')
+            total_size = -1
+            if content_type is not None:
+                total_size = int(content_type.strip())
             count = 0
             while 1:
                 chunk = response.read(chunk_size)
@@ -186,19 +187,24 @@ def get_file(fname,
 
     if download:
         print('Downloading data from', origin)
-        progbar = None
 
-        def dl_progress(count, block_size, total_size, progbar=None):
-            if progbar is None:
-                progbar = Progbar(total_size)
+        class ProgressTracker(object):
+            # Maintain progbar for the lifetime of download.
+            # This design was chosen for Python 2.7 compatibility.
+            progbar = None
+
+        def dl_progress(count, block_size, total_size):
+            if ProgressTracker.progbar is None:
+                if total_size is -1:
+                    total_size = None
+                ProgressTracker.progbar = Progbar(total_size)
             else:
-                progbar.update(count * block_size)
+                ProgressTracker.progbar.update(count * block_size)
 
         error_msg = 'URL fetch failure on {}: {} -- {}'
         try:
             try:
-                urlretrieve(origin, fpath,
-                            functools.partial(dl_progress, progbar=progbar))
+                urlretrieve(origin, fpath, dl_progress)
             except URLError as e:
                 raise Exception(error_msg.format(origin, e.errno, e.reason))
             except HTTPError as e:
@@ -207,7 +213,7 @@ def get_file(fname,
             if os.path.exists(fpath):
                 os.remove(fpath)
             raise
-        progbar = None
+        ProgressTracker.progbar = None
 
     if untar:
         if not os.path.exists(untar_fpath):

keras/utils/generic_utils.py

@@ -133,17 +133,20 @@ def deserialize_keras_object(identifier, module_objects=None,
                                  ': ' + class_name)
         if hasattr(cls, 'from_config'):
             arg_spec = inspect.getargspec(cls.from_config)
+            custom_objects = custom_objects or {}
             if 'custom_objects' in arg_spec.args:
-                custom_objects = custom_objects or {}
                 return cls.from_config(config['config'],
                                        custom_objects=dict(list(_GLOBAL_CUSTOM_OBJECTS.items()) +
                                                            list(custom_objects.items())))
-            return cls.from_config(config['config'])
+            with CustomObjectScope(custom_objects):
+                return cls.from_config(config['config'])
         else:
             # Then `cls` may be a function returning a class.
             # in this case by convention `config` holds
             # the kwargs of the function.
-            return cls(**config['config'])
+            custom_objects = custom_objects or {}
+            with CustomObjectScope(custom_objects):
+                return cls(**config['config'])
     elif isinstance(identifier, six.string_types):
         function_name = identifier
         if custom_objects and function_name in custom_objects:
@@ -153,7 +156,7 @@ def deserialize_keras_object(identifier, module_objects=None,
         else:
             fn = module_objects.get(function_name)
             if fn is None:
-                raise ValueError('Unknown ' + printable_module_name,
+                raise ValueError('Unknown ' + printable_module_name +
                                  ':' + function_name)
         return fn
     else:
@@ -208,12 +211,14 @@ class Progbar(object):
     """Displays a progress bar.
 
     # Arguments
-        target: Total number of steps expected.
+        target: Total number of steps expected, None if unknown.
         interval: Minimum visual progress update interval (in seconds).
     """
 
     def __init__(self, target, width=30, verbose=1, interval=0.05):
         self.width = width
+        if target is None:
+            target = -1
         self.target = target
         self.sum_values = {}
         self.unique_values = []
@@ -253,21 +258,22 @@ class Progbar(object):
             sys.stdout.write('\b' * prev_total_width)
             sys.stdout.write('\r')
 
-            numdigits = int(np.floor(np.log10(self.target))) + 1
-            barstr = '%%%dd/%%%dd [' % (numdigits, numdigits)
-            bar = barstr % (current, self.target)
-            prog = float(current) / self.target
-            prog_width = int(self.width * prog)
-            if prog_width > 0:
-                bar += ('=' * (prog_width - 1))
-                if current < self.target:
-                    bar += '>'
-                else:
-                    bar += '='
-            bar += ('.' * (self.width - prog_width))
-            bar += ']'
-            sys.stdout.write(bar)
-            self.total_width = len(bar)
+            if self.target is not -1:
+                numdigits = int(np.floor(np.log10(self.target))) + 1
+                barstr = '%%%dd/%%%dd [' % (numdigits, numdigits)
+                bar = barstr % (current, self.target)
+                prog = float(current) / self.target
+                prog_width = int(self.width * prog)
+                if prog_width > 0:
+                    bar += ('=' * (prog_width - 1))
+                    if current < self.target:
+                        bar += '>'
+                    else:
+                        bar += '='
+                bar += ('.' * (self.width - prog_width))
+                bar += ']'
+                sys.stdout.write(bar)
+                self.total_width = len(bar)
 
             if current:
                 time_per_unit = (now - self.start) / current
@@ -275,7 +281,7 @@ class Progbar(object):
                 time_per_unit = 0
             eta = time_per_unit * (self.target - current)
             info = ''
-            if current < self.target:
+            if current < self.target and self.target is not -1:
                 info += ' - ETA: %ds' % eta
             else:
                 info += ' - %ds' % (now - self.start)

keras/utils/io_utils.py

@@ -62,8 +62,8 @@ class HDF5Matrix(object):
         return self.end - self.start
 
     def __getitem__(self, key):
-        start, stop = key.start, key.stop
         if isinstance(key, slice):
+            start, stop = key.start, key.stop
             if start is None:
                 start = 0
             if stop is None:

keras/utils/layer_utils.py

@@ -167,7 +167,7 @@ def convert_dense_weights_data_format(dense,
             came before the target `Dense` layer.
         target_data_format: One of "channels_last", "channels_first".
             Set it "channels_last"
-            if converting a "chnnels_first" model to "channels_last",
+            if converting a "channels_first" model to "channels_last",
             or reciprocally.
     """
     assert target_data_format in {'channels_last', 'channels_first'}

keras/utils/vis_utils.py

@@ -5,29 +5,43 @@ try:
     # pydot-ng is a fork of pydot that is better maintained.
     import pydot_ng as pydot
 except ImportError:
-    # Fall back on pydot if necessary.
+    # pydotplus is an improved version of pydot
     try:
-        import pydot
+        import pydotplus as pydot
     except ImportError:
-        pydot = None
+        # Fall back on pydot if necessary.
+        try:
+            import pydot
+        except ImportError:
+            pydot = None
 
 
 def _check_pydot():
     try:
-        # Attempt to create an image of a blank graph to check the pydot/graphviz installation.
+        # Attempt to create an image of a blank graph
+        # to check the pydot/graphviz installation.
         pydot.Dot.create(pydot.Dot())
-    except Exception:  # pydot raises a generic Exception here, so no specific class can be caught.
+    except Exception:
+        # pydot raises a generic Exception here,
+        # so no specific class can be caught.
         raise ImportError('Failed to import pydot. You must install pydot'
                           ' and graphviz for `pydotprint` to work.')
 
 
-def model_to_dot(model, show_shapes=False, show_layer_names=True):
-    """Converts a Keras model to dot format.
+def model_to_dot(model,
+                 show_shapes=False,
+                 show_layer_names=True,
+                 rankdir='TB'):
+    """Convert a Keras model to dot format.
 
     # Arguments
         model: A Keras model instance.
         show_shapes: whether to display shape information.
         show_layer_names: whether to display layer names.
+        rankdir: `rankdir` argument passed to PyDot,
+            a string specifying the format of the plot:
+            'TB' creates a vertical plot;
+            'LR' creates a horizontal plot.
 
     # Returns
         A `pydot.Dot` instance representing the Keras model.
@@ -37,7 +51,7 @@ def model_to_dot(model, show_shapes=False, show_layer_names=True):
 
     _check_pydot()
     dot = pydot.Dot()
-    dot.set('rankdir', 'TB')
+    dot.set('rankdir', rankdir)
     dot.set('concentrate', True)
     dot.set_node_defaults(shape='record')
 
@@ -78,8 +92,9 @@ def model_to_dot(model, show_shapes=False, show_layer_names=True):
                     [str(ishape) for ishape in layer.input_shapes])
             else:
                 inputlabels = 'multiple'
-            label = '%s\n|{input:|output:}|{{%s}|{%s}}' % (label, inputlabels, outputlabels)
-
+            label = '%s\n|{input:|output:}|{{%s}|{%s}}' % (label,
+                                                           inputlabels,
+                                                           outputlabels)
         node = pydot.Node(layer_id, label=label)
         dot.add_node(node)
 
@@ -99,8 +114,21 @@ def model_to_dot(model, show_shapes=False, show_layer_names=True):
 def plot_model(model,
                to_file='model.png',
                show_shapes=False,
-               show_layer_names=True):
-    dot = model_to_dot(model, show_shapes, show_layer_names)
+               show_layer_names=True,
+               rankdir='TB'):
+    """Converts a Keras model to dot format and save to a file.
+
+    # Arguments
+        model: A Keras model instance
+        to_file: File name of the plot image.
+        show_shapes: whether to display shape information.
+        show_layer_names: whether to display layer names.
+        rankdir: `rankdir` argument passed to PyDot,
+            a string specifying the format of the plot:
+            'TB' creates a vertical plot;
+            'LR' creates a horizontal plot.
+    """
+    dot = model_to_dot(model, show_shapes, show_layer_names, rankdir)
     _, extension = os.path.splitext(to_file)
     if not extension:
         extension = 'png'

keras/wrappers/scikit_learn.py

@@ -90,7 +90,7 @@ class BaseWrapper(object):
         """Gets parameters for this estimator.
 
         # Arguments
-            **params: ignored (exists for API compatiblity).
+            **params: ignored (exists for API compatibility).
 
         # Returns
             Dictionary of parameter names mapped to their values.

setup.py

@@ -3,12 +3,12 @@ from setuptools import find_packages
 
 
 setup(name='Keras',
-      version='2.0.4',
+      version='2.0.5',
       description='Deep Learning for Python',
       author='Francois Chollet',
       author_email='francois.chollet@gmail.com',
       url='https://github.com/fchollet/keras',
-      download_url='https://github.com/fchollet/keras/tarball/2.0.4',
+      download_url='https://github.com/fchollet/keras/tarball/2.0.5',
       license='MIT',
       install_requires=['theano', 'pyyaml', 'six'],
       extras_require={

tests/integration_tests/test_temporal_data_tasks.py

@@ -6,7 +6,8 @@ import string
 from keras.utils.test_utils import get_test_data, keras_test
 from keras.utils.np_utils import to_categorical
 from keras.models import Sequential
-from keras import layers
+from keras import layers, optimizers
+import keras.backend as K
 import keras
 
 
@@ -204,5 +205,14 @@ def test_masked_temporal():
     ground_truth = -np.log(0.5)
     assert(np.abs(history.history['loss'][-1] - ground_truth) < 0.06)
 
+
+@pytest.mark.skipif(K.backend() != 'tensorflow', reason='Requires TF backend')
+@keras_test
+def test_embedding_with_clipnorm():
+    model = Sequential()
+    model.add(layers.Embedding(input_dim=1, output_dim=1))
+    model.compile(optimizer=optimizers.SGD(clipnorm=0.1), loss='mse')
+    model.fit(np.array([[0]]), np.array([[[0.5]]]), epochs=1)
+
 if __name__ == '__main__':
     pytest.main([__file__])

tests/keras/activations_test.py

@@ -136,6 +136,10 @@ def test_elu():
     assert_allclose(result, test_values, rtol=1e-05)
 
     negative_values = np.array([[-1, -2]], dtype=K.floatx())
+    # cntk can't rebind the input shape, so create the model again to test different batch size
+    if (K.backend() == 'cntk'):
+        x2 = K.placeholder(ndim=2)
+        f = K.function([x2], [activations.elu(x2, 0.5)])
     result = f([negative_values])[0]
     true_result = (np.exp(negative_values) - 1) / 2
 

tests/keras/applications/applications_test.py

@@ -11,12 +11,24 @@ def test_resnet50():
 
 
 @keras_test
+@pytest.mark.skipif((K.backend() == 'cntk'),
+                    reason="cntk does not support padding with non-concrete dimension")
 def test_resnet50_notop():
     model = applications.ResNet50(weights=None, include_top=False)
     assert model.output_shape == (None, None, None, 2048)
 
 
 @keras_test
+def test_resnet50_notop_specified_input_shape():
+    input_shape = (3, 300, 300) if K.image_data_format() == 'channels_first' else (300, 300, 3)
+    model = applications.ResNet50(weights=None, include_top=False, input_shape=input_shape)
+    output_shape = (None, 2048, 1, 1) if K.image_data_format() == 'channels_first' else (None, 1, 1, 2048)
+    assert model.output_shape == output_shape
+
+
+@keras_test
+@pytest.mark.skipif((K.backend() == 'cntk'),
+                    reason="cntk does not support padding with non-concrete dimension")
 def test_resnet50_pooling():
     model = applications.ResNet50(weights=None,
                                   include_top=False,
@@ -24,6 +36,16 @@ def test_resnet50_pooling():
     assert model.output_shape == (None, 2048)
 
 
+@keras_test
+def test_resnet50_pooling_specified_input_shape():
+    input_shape = (3, 300, 300) if K.image_data_format() == 'channels_first' else (300, 300, 3)
+    model = applications.ResNet50(weights=None,
+                                  include_top=False,
+                                  pooling='avg',
+                                  input_shape=input_shape)
+    assert model.output_shape == (None, 2048)
+
+
 @keras_test
 def test_vgg16():
     model = applications.VGG16(weights=None)
@@ -31,17 +53,36 @@ def test_vgg16():
 
 
 @keras_test
+@pytest.mark.skipif((K.backend() == 'cntk'),
+                    reason="cntk does not support padding with non-concrete dimension")
 def test_vgg16_notop():
     model = applications.VGG16(weights=None, include_top=False)
     assert model.output_shape == (None, None, None, 512)
 
 
 @keras_test
+def test_vgg16_notop_specified_input_shape():
+    input_shape = (3, 300, 300) if K.image_data_format() == 'channels_first' else (300, 300, 3)
+    model = applications.VGG16(weights=None, include_top=False, input_shape=input_shape)
+    output_shape = (None, 512, 9, 9) if K.image_data_format() == 'channels_first' else (None, 9, 9, 512)
+    assert model.output_shape == output_shape
+
+
+@keras_test
+@pytest.mark.skipif((K.backend() == 'cntk'),
+                    reason="cntk does not support padding with non-concrete dimension")
 def test_vgg16_pooling():
     model = applications.VGG16(weights=None, include_top=False, pooling='avg')
     assert model.output_shape == (None, 512)
 
 
+@keras_test
+def test_vgg16_pooling_specified_input_shape():
+    input_shape = (3, 300, 300) if K.image_data_format() == 'channels_first' else (300, 300, 3)
+    model = applications.VGG16(weights=None, include_top=False, pooling='avg', input_shape=input_shape)
+    assert model.output_shape == (None, 512)
+
+
 @keras_test
 def test_vgg19():
     model = applications.VGG19(weights=None)
@@ -49,17 +90,36 @@ def test_vgg19():
 
 
 @keras_test
+@pytest.mark.skipif((K.backend() == 'cntk'),
+                    reason="cntk does not support padding with non-concrete dimension")
 def test_vgg19_notop():
-    model = applications.VGG16(weights=None, include_top=False)
+    model = applications.VGG19(weights=None, include_top=False)
     assert model.output_shape == (None, None, None, 512)
 
 
 @keras_test
+def test_vgg19_notop_specified_input_shape():
+    input_shape = (3, 300, 300) if K.image_data_format() == 'channels_first' else (300, 300, 3)
+    model = applications.VGG19(weights=None, include_top=False, input_shape=input_shape)
+    output_shape = (None, 512, 9, 9) if K.image_data_format() == 'channels_first' else (None, 9, 9, 512)
+    assert model.output_shape == output_shape
+
+
+@keras_test
+@pytest.mark.skipif((K.backend() == 'cntk'),
+                    reason="cntk does not support padding with non-concrete dimension")
 def test_vgg19_pooling():
     model = applications.VGG16(weights=None, include_top=False, pooling='avg')
     assert model.output_shape == (None, 512)
 
 
+@keras_test
+def test_vgg19_pooling_specified_input_shape():
+    input_shape = (3, 300, 300) if K.image_data_format() == 'channels_first' else (300, 300, 3)
+    model = applications.VGG16(weights=None, include_top=False, pooling='avg', input_shape=input_shape)
+    assert model.output_shape == (None, 512)
+
+
 @keras_test
 @pytest.mark.skipif((K.backend() != 'tensorflow'),
                     reason='Requires tensorflow backend')
@@ -91,12 +151,16 @@ def test_inceptionv3():
 
 
 @keras_test
+@pytest.mark.skipif((K.backend() == 'cntk'),
+                    reason="cntk does not support padding with non-concrete dimension")
 def test_inceptionv3_notop():
     model = applications.InceptionV3(weights=None, include_top=False)
     assert model.output_shape == (None, None, None, 2048)
 
 
 @keras_test
+@pytest.mark.skipif((K.backend() == 'cntk'),
+                    reason="cntk does not support padding with non-concrete dimension")
 def test_inceptionv3_pooling():
     model = applications.InceptionV3(weights=None, include_top=False, pooling='avg')
     assert model.output_shape == (None, 2048)

tests/keras/engine/test_topology.py

@@ -77,6 +77,8 @@ def test_trainable_weights():
 
 
 @keras_test
+@pytest.mark.skipif((K.backend() == 'cntk'),
+                    reason="cntk does not support add learning_phase() as input")
 def test_learning_phase():
     a = Input(shape=(32,), name='input_a')
     b = Input(shape=(32,), name='input_b')
@@ -439,12 +441,11 @@ def test_recursion():
     with pytest.raises(Exception) as e:
         Model([j], [m, n])
 
-    # redudant outputs
+    # redundant outputs
     j = Input(shape=(32,), name='input_j')
     k = Input(shape=(32,), name='input_k')
     m, n = model([j, k])
-    # this should work lol
-    # TODO: raise a warning
+    # this should work with a warning
     Model([j, k], [m, n, n])
 
     # redundant inputs
@@ -495,7 +496,7 @@ def test_load_layers():
     from keras.models import Model
     from keras.engine.topology import preprocess_weights_for_loading
 
-    if K.backend() == 'tensorflow':
+    if K.backend() == 'tensorflow' or K.backend() == 'cntk':
         inputs = Input(shape=(10, 20, 20, 1))
     else:
         inputs = Input(shape=(10, 1, 20, 20))
@@ -551,6 +552,7 @@ def test_load_layers():
     assert np.all(K.eval(model.layers[2].weights[5]) == weight_tensor_bi_convlstm_new[5])
 
 
+@keras_test
 def test_recursion_with_bn_and_loss():
     model1 = Sequential([
         layers.Dense(5, input_dim=5, activity_regularizer='l1'),
@@ -577,5 +579,32 @@ def test_recursion_with_bn_and_loss():
     model2.fit(x, y, verbose=0, epochs=1)
 
 
+def test_shared_layer_depth_is_correct():
+    # Basic outline here: we have a shared embedding layer, and two inputs that go through
+    # different depths of computation in the graph before the final output.  We need the computed
+    # depth of the input layers to be the same, because they both pass through the embedding layer
+    # before anything else happens.  That's what we're testing.
+    from keras.layers import Embedding, Input, Dense, Concatenate
+    from keras.models import Model
+    input1 = Input(shape=(10,), name="input1")
+    input2 = Input(shape=(10,), name="input2")
+    embedding_layer = Embedding(name="embedding", input_dim=5, output_dim=10)
+    embedded_input1 = embedding_layer(input1)
+    embedded_input2 = embedding_layer(input2)
+    transformed_input2 = Dense(6)(Dense(5)(Dense(3)(embedded_input2)))
+    final_output = Dense(2)(Concatenate()([embedded_input1, transformed_input2]))
+    model = Model(inputs=[input1, input2], outputs=final_output)
+    input1_depth = -1
+    input2_depth = -1
+    for depth, layers in model.layers_by_depth.items():
+        for layer in layers:
+            if layer.name == 'input1':
+                input1_depth = depth
+            if layer.name == 'input2':
+                input2_depth = depth
+    assert input1_depth != -1
+    assert input1_depth == input2_depth
+
+
 if __name__ == '__main__':
     pytest.main([__file__])

tests/keras/engine/test_training.py

@@ -1,6 +1,7 @@
 import pytest
 import numpy as np
 from numpy.testing import assert_allclose
+import scipy.sparse as sparse
 
 from keras.layers import Dense, Dropout
 from keras.engine.topology import Input
@@ -198,6 +199,18 @@ def test_model_methods():
     out = model.predict([input_a_np, input_b_np], batch_size=4)
 
 
+@pytest.mark.skipif(K.backend() != 'tensorflow', reason='sparse operations supported only by TF')
+@keras_test
+def test_sparse_input_validation_split():
+    test_input = sparse.random(6, 3, density=0.25).tocsr()
+    in1 = Input(shape=(3,), sparse=True)
+    out1 = Dense(4)(in1)
+    test_output = np.random.random((6, 4))
+    model = Model(in1, out1)
+    model.compile('rmsprop', 'mse')
+    model.fit(test_input, test_output, epochs=1, batch_size=2, validation_split=0.2)
+
+
 @keras_test
 def test_trainable_argument():
     x = np.random.random((5, 3))
@@ -433,6 +446,8 @@ def test_model_with_partial_loss():
 
 
 @keras_test
+@pytest.mark.skipif((K.backend() == 'cntk'),
+                    reason="cntk does not support external loss yet")
 def test_model_with_external_loss():
     # None loss, only regularization loss.
     a = Input(shape=(3,), name='input_a')

tests/keras/layers/convolutional_recurrent_test.py

@@ -3,8 +3,8 @@ import numpy as np
 from numpy.testing import assert_allclose
 
 from keras import backend as K
-from keras.models import Sequential
-from keras.layers import convolutional_recurrent
+from keras.models import Sequential, Model
+from keras.layers import convolutional_recurrent, Input
 from keras.utils.test_utils import layer_test
 from keras import regularizers
 
@@ -43,67 +43,69 @@ def test_convolutional_recurrent():
             if data_format == 'channels_first' or return_sequences:
                 continue
 
-            # Tests for statefulness
-            model = Sequential()
-            kwargs = {'data_format': data_format,
-                      'return_sequences': return_sequences,
-                      'filters': filters,
-                      'kernel_size': (num_row, num_col),
-                      'stateful': True,
-                      'batch_input_shape': inputs.shape,
-                      'padding': 'same'}
-            layer = convolutional_recurrent.ConvLSTM2D(**kwargs)
+            # cntk doesn't support statefulness on LSTM yet, will enable it on cntk later
+            if K.backend() != 'cntk':
+                # Tests for statefulness
+                model = Sequential()
+                kwargs = {'data_format': data_format,
+                          'return_sequences': return_sequences,
+                          'filters': filters,
+                          'kernel_size': (num_row, num_col),
+                          'stateful': True,
+                          'batch_input_shape': inputs.shape,
+                          'padding': 'same'}
+                layer = convolutional_recurrent.ConvLSTM2D(**kwargs)
 
-            model.add(layer)
-            model.compile(optimizer='sgd', loss='mse')
-            out1 = model.predict(np.ones_like(inputs))
+                model.add(layer)
+                model.compile(optimizer='sgd', loss='mse')
+                out1 = model.predict(np.ones_like(inputs))
 
-            # train once so that the states change
-            model.train_on_batch(np.ones_like(inputs),
-                                 np.random.random(out1.shape))
-            out2 = model.predict(np.ones_like(inputs))
+                # train once so that the states change
+                model.train_on_batch(np.ones_like(inputs),
+                                     np.random.random(out1.shape))
+                out2 = model.predict(np.ones_like(inputs))
 
-            # if the state is not reset, output should be different
-            assert(out1.max() != out2.max())
+                # if the state is not reset, output should be different
+                assert(out1.max() != out2.max())
 
-            # check that output changes after states are reset
-            # (even though the model itself didn't change)
-            layer.reset_states()
-            out3 = model.predict(np.ones_like(inputs))
-            assert(out2.max() != out3.max())
+                # check that output changes after states are reset
+                # (even though the model itself didn't change)
+                layer.reset_states()
+                out3 = model.predict(np.ones_like(inputs))
+                assert(out2.max() != out3.max())
 
-            # check that container-level reset_states() works
-            model.reset_states()
-            out4 = model.predict(np.ones_like(inputs))
-            assert_allclose(out3, out4, atol=1e-5)
+                # check that container-level reset_states() works
+                model.reset_states()
+                out4 = model.predict(np.ones_like(inputs))
+                assert_allclose(out3, out4, atol=1e-5)
 
-            # check that the call to `predict` updated the states
-            out5 = model.predict(np.ones_like(inputs))
-            assert(out4.max() != out5.max())
+                # check that the call to `predict` updated the states
+                out5 = model.predict(np.ones_like(inputs))
+                assert(out4.max() != out5.max())
 
-            # check regularizers
-            kwargs = {'data_format': data_format,
-                      'return_sequences': return_sequences,
-                      'kernel_size': (num_row, num_col),
-                      'stateful': True,
-                      'filters': filters,
-                      'batch_input_shape': inputs.shape,
-                      'kernel_regularizer': regularizers.L1L2(l1=0.01),
-                      'recurrent_regularizer': regularizers.L1L2(l1=0.01),
-                      'bias_regularizer': 'l2',
-                      'activity_regularizer': 'l2',
-                      'kernel_constraint': 'max_norm',
-                      'recurrent_constraint': 'max_norm',
-                      'bias_constraint': 'max_norm',
-                      'padding': 'same'}
+                # check regularizers
+                kwargs = {'data_format': data_format,
+                          'return_sequences': return_sequences,
+                          'kernel_size': (num_row, num_col),
+                          'stateful': True,
+                          'filters': filters,
+                          'batch_input_shape': inputs.shape,
+                          'kernel_regularizer': regularizers.L1L2(l1=0.01),
+                          'recurrent_regularizer': regularizers.L1L2(l1=0.01),
+                          'bias_regularizer': 'l2',
+                          'activity_regularizer': 'l2',
+                          'kernel_constraint': 'max_norm',
+                          'recurrent_constraint': 'max_norm',
+                          'bias_constraint': 'max_norm',
+                          'padding': 'same'}
 
-            layer = convolutional_recurrent.ConvLSTM2D(**kwargs)
-            layer.build(inputs.shape)
-            assert len(layer.losses) == 3
-            assert layer.activity_regularizer
-            output = layer(K.variable(np.ones(inputs.shape)))
-            assert len(layer.losses) == 4
-            K.eval(output)
+                layer = convolutional_recurrent.ConvLSTM2D(**kwargs)
+                layer.build(inputs.shape)
+                assert len(layer.losses) == 3
+                assert layer.activity_regularizer
+                output = layer(K.variable(np.ones(inputs.shape)))
+                assert len(layer.losses) == 4
+                K.eval(output)
 
             # check dropout
             layer_test(convolutional_recurrent.ConvLSTM2D,
@@ -116,5 +118,18 @@ def test_convolutional_recurrent():
                                'recurrent_dropout': 0.1},
                        input_shape=inputs.shape)
 
+            # check state initialization
+            layer = convolutional_recurrent.ConvLSTM2D(filters=filters,
+                                                       kernel_size=(num_row, num_col),
+                                                       data_format=data_format,
+                                                       return_sequences=return_sequences)
+            layer.build(inputs.shape)
+            x = Input(batch_shape=inputs.shape)
+            initial_state = layer.get_initial_state(x)
+            y = layer(x, initial_state=initial_state)
+            model = Model(x, y)
+            assert model.predict(inputs).shape == layer.compute_output_shape(inputs.shape)
+
+
 if __name__ == '__main__':
     pytest.main([__file__])

tests/keras/layers/convolutional_test.py

@@ -17,6 +17,8 @@ else:
 
 
 @keras_test
+@pytest.mark.skipif((K.backend() == 'cntk'),
+                    reason="cntk does not support dilated conv")
 def test_causal_dilated_conv():
     # Causal:
     layer_test(convolutional.Conv1D,
@@ -122,6 +124,8 @@ def test_averagepooling_1d():
 
 
 @keras_test
+@pytest.mark.skipif((K.backend() == 'cntk'),
+                    reason="cntk does not support dilated conv")
 def test_convolution_2d():
     num_samples = 2
     filters = 2
@@ -494,20 +498,54 @@ def test_zero_padding_3d():
                      stack_size))
 
     # basic test
-    layer_test(convolutional.ZeroPadding3D,
-               kwargs={'padding': (2, 2, 2)},
-               input_shape=inputs.shape)
+    for data_format in ['channels_first', 'channels_last']:
+        layer_test(convolutional.ZeroPadding3D,
+                   kwargs={'padding': (2, 2, 2), 'data_format': data_format},
+                   input_shape=inputs.shape)
+        layer_test(convolutional.ZeroPadding3D,
+                   kwargs={'padding': ((1, 2), (3, 4), (0, 2)), 'data_format': data_format},
+                   input_shape=inputs.shape)
 
-    # correctness test
-    layer = convolutional.ZeroPadding3D(padding=(2, 2, 2))
-    layer.build(inputs.shape)
-    output = layer(K.variable(inputs))
-    np_output = K.eval(output)
-    for offset in [0, 1, -1, -2]:
-        assert_allclose(np_output[:, offset, :, :, :], 0.)
-        assert_allclose(np_output[:, :, offset, :, :], 0.)
-        assert_allclose(np_output[:, :, :, offset, :], 0.)
-    assert_allclose(np_output[:, 2:-2, 2:-2, 2:-2, :], 1.)
+        # correctness test
+        layer = convolutional.ZeroPadding3D(padding=(2, 2, 2),
+                                            data_format=data_format)
+        layer.build(inputs.shape)
+        output = layer(K.variable(inputs))
+        np_output = K.eval(output)
+        if data_format == 'channels_last':
+            for offset in [0, 1, -1, -2]:
+                assert_allclose(np_output[:, offset, :, :, :], 0.)
+                assert_allclose(np_output[:, :, offset, :, :], 0.)
+                assert_allclose(np_output[:, :, :, offset, :], 0.)
+            assert_allclose(np_output[:, 2:-2, 2:-2, 2:-2, :], 1.)
+        elif data_format == 'channels_first':
+            for offset in [0, 1, -1, -2]:
+                assert_allclose(np_output[:, :, offset, :, :], 0.)
+                assert_allclose(np_output[:, :, :, offset, :], 0.)
+                assert_allclose(np_output[:, :, :, :, offset], 0.)
+            assert_allclose(np_output[:, :, 2:-2, 2:-2, 2:-2], 1.)
+
+        layer = convolutional.ZeroPadding3D(padding=((1, 2), (3, 4), (0, 2)),
+                                            data_format=data_format)
+        layer.build(inputs.shape)
+        output = layer(K.variable(inputs))
+        np_output = K.eval(output)
+        if data_format == 'channels_last':
+            for dim1_offset in [0, -1, -2]:
+                assert_allclose(np_output[:, dim1_offset, :, :, :], 0.)
+            for dim2_offset in [0, 1, 2, -1, -2, -3, -4]:
+                assert_allclose(np_output[:, :, dim2_offset, :, :], 0.)
+            for dim3_offset in [-1, -2]:
+                assert_allclose(np_output[:, :, :, dim3_offset, :], 0.)
+            assert_allclose(np_output[:, 1:-2, 3:-4, 0:-2, :], 1.)
+        elif data_format == 'channels_first':
+            for dim1_offset in [0, -1, -2]:
+                assert_allclose(np_output[:, :, dim1_offset, :, :], 0.)
+            for dim2_offset in [0, 1, 2, -1, -2, -3, -4]:
+                assert_allclose(np_output[:, :, :, dim2_offset, :], 0.)
+            for dim3_offset in [-1, -2]:
+                assert_allclose(np_output[:, :, :, :, dim3_offset], 0.)
+            assert_allclose(np_output[:, :, 1:-2, 3:-4, 0:-2], 1.)
 
 
 @keras_test
@@ -563,6 +601,8 @@ def test_upsampling_2d():
                 assert_allclose(np_output, expected_out)
 
 
+@pytest.mark.skipif((K.backend() == 'cntk'),
+                    reason="cntk does not support it yet")
 def test_upsampling_3d():
     num_samples = 2
     stack_size = 2
@@ -617,6 +657,8 @@ def test_upsampling_3d():
 
 
 @keras_test
+@pytest.mark.skipif((K.backend() == 'cntk'),
+                    reason="cntk does not support slice to 0 dimension")
 def test_cropping_1d():
     num_samples = 2
     time_length = 4

tests/keras/layers/embeddings_test.py

@@ -16,6 +16,16 @@ def test_embedding():
                input_shape=(3, 2),
                input_dtype='int32',
                expected_output_dtype=K.floatx())
+    layer_test(Embedding,
+               kwargs={'output_dim': 4, 'input_dim': 10, 'mask_zero': True},
+               input_shape=(3, 2, 5),
+               input_dtype='int32',
+               expected_output_dtype=K.floatx())
+    layer_test(Embedding,
+               kwargs={'output_dim': 4, 'input_dim': 10, 'mask_zero': True, 'input_length': (None, 5)},
+               input_shape=(3, 2, 5),
+               input_dtype='int32',
+               expected_output_dtype=K.floatx())
 
 
 if __name__ == '__main__':

tests/keras/layers/noise_test.py

@@ -2,9 +2,12 @@ import pytest
 from keras.utils.test_utils import layer_test
 from keras.utils.test_utils import keras_test
 from keras.layers import noise
+from keras import backend as K
 
 
 @keras_test
+@pytest.mark.skipif((K.backend() == 'cntk'),
+                    reason="cntk does not support it yet")
 def test_GaussianNoise():
     layer_test(noise.GaussianNoise,
                kwargs={'stddev': 1.},
@@ -12,6 +15,8 @@ def test_GaussianNoise():
 
 
 @keras_test
+@pytest.mark.skipif((K.backend() == 'cntk'),
+                    reason="cntk does not support it yet")
 def test_GaussianDropout():
     layer_test(noise.GaussianDropout,
                kwargs={'rate': 0.5},

tests/keras/layers/normalization_test.py

@@ -2,7 +2,7 @@ import pytest
 import numpy as np
 from numpy.testing import assert_allclose
 
-from keras.layers import Dense, Activation, Input
+from keras.layers import Input
 from keras.utils.test_utils import layer_test, keras_test
 from keras.layers import normalization
 from keras.models import Sequential, Model

tests/keras/layers/recurrent_test.py

@@ -77,6 +77,8 @@ def test_implementation_mode(layer_class):
 
 
 @rnn_test
+@pytest.mark.skipif((K.backend() == 'cntk'),
+                    reason="cntk does not support stateful RNN yet")
 def test_statefulness(layer_class):
     model = Sequential()
     model.add(embeddings.Embedding(embedding_num, embedding_dim,
@@ -147,6 +149,8 @@ def test_regularizer(layer_class):
 
 
 @keras_test
+@pytest.mark.skipif((K.backend() == 'cntk'),
+                    reason="cntk does not support mask on RNN yet")
 def test_masking_layer():
     ''' This test based on a previously failing issue here:
     https://github.com/fchollet/keras/issues/1567
@@ -170,7 +174,9 @@ def test_masking_layer():
 
 @rnn_test
 def test_from_config(layer_class):
-    for stateful in (False, True):
+    # cntk does not support stateful yet.
+    stateful_flags = (False, True) if K.backend() != 'cntk' else (False,)
+    for stateful in stateful_flags:
         l1 = layer_class(units=1, stateful=stateful)
         l2 = layer_class.from_config(l1.get_config())
         assert l1.get_config() == l2.get_config()
@@ -220,6 +226,8 @@ def test_specify_initial_state_non_keras_tensor(layer_class):
 
 
 @rnn_test
+@pytest.mark.skipif((K.backend() == 'cntk'),
+                    reason="cntk does not support stateful RNN yet")
 def test_reset_states_with_values(layer_class):
     num_states = 2 if layer_class is recurrent.LSTM else 1
 
@@ -253,7 +261,7 @@ def test_specify_state_with_masking(layer_class):
     num_states = 2 if layer_class is recurrent.LSTM else 1
 
     inputs = Input((timesteps, embedding_dim))
-    masked_inputs = Masking()(inputs)
+    _ = Masking()(inputs)
     initial_state = [Input((units,)) for _ in range(num_states)]
     output = layer_class(units)(inputs, initial_state=initial_state)
 
@@ -266,5 +274,37 @@ def test_specify_state_with_masking(layer_class):
     targets = np.random.random((num_samples, units))
     model.fit([inputs] + initial_state, targets)
 
+
+@rnn_test
+@pytest.mark.skipif((K.backend() == 'cntk'),
+                    reason="cntk does not support stateful RNN yet")
+def test_return_state(layer_class):
+    num_states = 2 if layer_class is recurrent.LSTM else 1
+
+    inputs = Input(batch_shape=(num_samples, timesteps, embedding_dim))
+    layer = layer_class(units, return_state=True, stateful=True)
+    outputs = layer(inputs)
+    output, state = outputs[0], outputs[1:]
+    assert len(state) == num_states
+    model = Model(inputs, state[0])
+
+    inputs = np.random.random((num_samples, timesteps, embedding_dim))
+    state = model.predict(inputs)
+    np.testing.assert_allclose(K.eval(layer.states[0]), state, atol=1e-4)
+
+
+@rnn_test
+def test_state_reuse(layer_class):
+    inputs = Input(batch_shape=(num_samples, timesteps, embedding_dim))
+    layer = layer_class(units, return_state=True, return_sequences=True)
+    outputs = layer(inputs)
+    output, state = outputs[0], outputs[1:]
+    output = layer_class(units)(output, initial_state=state)
+    model = Model(inputs, output)
+
+    inputs = np.random.random((num_samples, timesteps, embedding_dim))
+    outputs = model.predict(inputs)
+
+
 if __name__ == '__main__':
     pytest.main([__file__])

tests/keras/layers/wrappers_test.py

@@ -5,6 +5,7 @@ from keras.utils.test_utils import keras_test
 from keras.layers import wrappers, Input
 from keras.layers import core, convolutional, recurrent, embeddings
 from keras.models import Sequential, Model, model_from_json
+from keras import backend as K
 
 
 @keras_test
@@ -108,6 +109,8 @@ def test_regularizers():
 
 
 @keras_test
+@pytest.mark.skipif((K.backend() == 'cntk'),
+                    reason="cntk does not support reverse yet")
 def test_Bidirectional():
     rnn = recurrent.SimpleRNN
     samples = 2

tests/keras/legacy/layers_test.py

@@ -184,6 +184,8 @@ def test_merge():
 
 
 @keras_test
+@pytest.mark.skipif((K.backend() == 'cntk'),
+                    reason="cntk does not support stateful RNN yet")
 def test_merge_mask_2d():
     rand = lambda *shape: np.asarray(np.random.random(shape) > 0.5, dtype='int32')
 
@@ -217,6 +219,8 @@ def test_merge_mask_2d():
 
 
 @keras_test
+@pytest.mark.skipif((K.backend() == 'cntk'),
+                    reason="cntk does not support stateful RNN yet")
 def test_merge_mask_3d():
     rand = lambda *shape: np.asarray(np.random.random(shape) > 0.5, dtype='int32')
 

tests/keras/losses_test.py

@@ -10,12 +10,14 @@ allobj = [losses.mean_squared_error,
           losses.mean_absolute_percentage_error,
           losses.mean_squared_logarithmic_error,
           losses.squared_hinge,
-          losses.hinge, losses.categorical_crossentropy,
+          losses.hinge,
+          losses.categorical_crossentropy,
           losses.binary_crossentropy,
           losses.kullback_leibler_divergence,
           losses.poisson,
           losses.cosine_proximity,
-          losses.logcosh]
+          losses.logcosh,
+          losses.categorical_hinge]
 
 
 def test_objective_shapes_3d():
@@ -45,5 +47,15 @@ def test_cce_one_hot():
     assert K.eval(losses.sparse_categorical_crossentropy(y_a, y_b)).shape == (6,)
 
 
+def test_categorical_hinge():
+    y_pred = K.variable(np.array([[0.3, 0.2, 0.1],
+                                  [0.1, 0.2, 0.7]]))
+    y_true = K.variable(np.array([[0, 1, 0],
+                                  [1, 0, 0]]))
+    expected_loss = ((0.3 - 0.2 + 1) + (0.7 - 0.1 + 1)) / 2.0
+    loss = K.eval(losses.categorical_hinge(y_true, y_pred))
+    assert np.isclose(expected_loss, np.mean(loss))
+
+
 if __name__ == '__main__':
     pytest.main([__file__])

tests/keras/metrics_test.py

@@ -42,6 +42,8 @@ def test_sparse_metrics():
         assert K.eval(metric(y_a, y_b)).shape == (6,)
 
 
+@pytest.mark.skipif((K.backend() == 'cntk'),
+                    reason="keras cntk backend does not support top_k yet")
 def test_top_k_categorical_accuracy():
     y_pred = K.variable(np.array([[0.3, 0.2, 0.1], [0.1, 0.2, 0.7]]))
     y_true = K.variable(np.array([[0, 1, 0], [1, 0, 0]]))
@@ -56,5 +58,21 @@ def test_top_k_categorical_accuracy():
     assert failure_result == 0
 
 
+@pytest.mark.skipif((K.backend() == 'cntk'),
+                    reason="keras cntk backend does not support top_k yet")
+def test_sparse_top_k_categorical_accuracy():
+    y_pred = K.variable(np.array([[0.3, 0.2, 0.1], [0.1, 0.2, 0.7]]))
+    y_true = K.variable(np.array([[1], [0]]))
+    success_result = K.eval(metrics.sparse_top_k_categorical_accuracy(y_true, y_pred,
+                                                                      k=3))
+    assert success_result == 1
+    partial_result = K.eval(metrics.sparse_top_k_categorical_accuracy(y_true, y_pred,
+                                                                      k=2))
+    assert partial_result == 0.5
+    failure_result = K.eval(metrics.sparse_top_k_categorical_accuracy(y_true, y_pred,
+                                                                      k=1))
+    assert failure_result == 0
+
+
 if __name__ == '__main__':
     pytest.main([__file__])

tests/keras/preprocessing/image_test.py

@@ -73,22 +73,11 @@ class TestImage:
         with pytest.raises(ValueError):
             x = np.random.random((3, 10, 10))
             generator.fit(x)
-        with pytest.raises(ValueError):
-            x = np.random.random((32, 3, 10, 10))
-            generator.fit(x)
-        with pytest.raises(ValueError):
-            x = np.random.random((32, 10, 10, 5))
-            generator.fit(x)
+
         # Test flow with invalid data
-        with pytest.raises(ValueError):
-            x = np.random.random((32, 10, 10, 5))
-            generator.flow(np.arange(x.shape[0]))
         with pytest.raises(ValueError):
             x = np.random.random((32, 10, 10))
             generator.flow(np.arange(x.shape[0]))
-        with pytest.raises(ValueError):
-            x = np.random.random((32, 3, 10, 10))
-            generator.flow(np.arange(x.shape[0]))
 
     def test_image_data_generator_fit(self):
         generator = image.ImageDataGenerator(
@@ -160,6 +149,32 @@ class TestImage:
         assert(sorted(dir_iterator.filenames) == sorted(filenames))
         shutil.rmtree(tmp_folder)
 
+    def test_directory_iterator_class_mode_input(self):
+        tmp_folder = tempfile.mkdtemp(prefix='test_images')
+        os.mkdir(os.path.join(tmp_folder, 'class-1'))
+
+        # save the images in the paths
+        count = 0
+        for test_images in self.all_test_images:
+            for im in test_images:
+                filename = os.path.join(tmp_folder, 'class-1', 'image-{}.jpg'.format(count))
+                im.save(os.path.join(tmp_folder, filename))
+                count += 1
+
+        # create iterator
+        generator = image.ImageDataGenerator()
+        dir_iterator = generator.flow_from_directory(tmp_folder, class_mode='input')
+        batch = next(dir_iterator)
+
+        # check if input and output have the same shape
+        assert(batch[0].shape == batch[1].shape)
+        # check if the input and output images are not the same numpy array
+        input_img = batch[0][0]
+        output_img = batch[1][0]
+        output_img[0][0][0] += 1
+        assert(input_img[0][0][0] != output_img[0][0][0])
+        shutil.rmtree(tmp_folder)
+
     def test_img_utils(self):
         height, width = 10, 8
 

tests/keras/test_callbacks.py

@@ -6,9 +6,12 @@ import pytest
 from csv import Sniffer
 import shutil
 from keras import optimizers
+from keras import initializers
 from keras import callbacks
 from keras.models import Sequential
-from keras.layers.core import Dense
+from keras.layers.core import Dense, Dropout
+from keras.layers.convolutional import Conv2D
+from keras.layers.pooling import MaxPooling2D, GlobalAveragePooling2D
 from keras.utils.test_utils import get_test_data
 from keras.utils.test_utils import keras_test
 from keras import backend as K
@@ -22,6 +25,34 @@ train_samples = 20
 test_samples = 20
 
 
+@keras_test
+def test_TerminateOnNaN():
+    np.random.seed(1337)
+    (X_train, y_train), (X_test, y_test) = get_test_data(num_train=train_samples,
+                                                         num_test=test_samples,
+                                                         input_shape=(input_dim,),
+                                                         classification=True,
+                                                         num_classes=num_class)
+
+    y_test = np_utils.to_categorical(y_test)
+    y_train = np_utils.to_categorical(y_train)
+    cbks = [callbacks.TerminateOnNaN()]
+    model = Sequential()
+    initializer = initializers.Constant(value=1e5)
+    for _ in range(5):
+        model.add(Dense(num_hidden, input_dim=input_dim, activation='relu',
+                        kernel_initializer=initializer))
+    model.add(Dense(num_class, activation='linear'))
+    model.compile(loss='mean_squared_error',
+                  optimizer='rmsprop')
+
+    history = model.fit(X_train, y_train, batch_size=batch_size,
+                        validation_data=(X_test, y_test), callbacks=cbks, epochs=20)
+    loss = history.history['loss']
+    assert len(loss) == 1
+    assert loss[0] == np.inf
+
+
 @keras_test
 def test_ModelCheckpoint():
     np.random.seed(1337)
@@ -271,9 +302,9 @@ def test_CSVLogger():
 @pytest.mark.skipif((K.backend() != 'tensorflow'),
                     reason='Requires tensorflow backend')
 def test_TensorBoard():
-    np.random.seed(1337)
+    np.random.seed(np.random.randint(1, 1e7))
+    filepath = './logs_' + str(np.random.randint(1, 1e4))
 
-    filepath = './logs'
     (X_train, y_train), (X_test, y_test) = get_test_data(
         num_train=train_samples,
         num_test=test_samples,
@@ -309,13 +340,17 @@ def test_TensorBoard():
     # case 1 Sequential
     model = Sequential()
     model.add(Dense(num_hidden, input_dim=input_dim, activation='relu'))
+    model.add(Dropout(0.1))
     model.add(Dense(num_class, activation='softmax'))
     model.compile(loss='categorical_crossentropy',
                   optimizer='sgd',
                   metrics=['accuracy'])
 
     tsb = callbacks.TensorBoard(log_dir=filepath, histogram_freq=1,
-                                write_images=True)
+                                write_images=True, write_grads=True,
+                                embeddings_freq=1,
+                                embeddings_layer_names=['dense_1'],
+                                batch_size=5)
     cbks = [tsb]
 
     # fit with validation data
@@ -348,6 +383,97 @@ def test_TensorBoard():
     shutil.rmtree(filepath)
 
 
+@keras_test
+@pytest.mark.skipif((K.backend() != 'tensorflow'),
+                    reason='Requires tensorflow backend')
+def test_TensorBoard_convnet():
+    np.random.seed(np.random.randint(1, 1e7))
+    filepath = './logs_' + str(np.random.randint(1, 1e4))
+
+    input_shape = (16, 16, 3)
+    (x_train, y_train), (x_test, y_test) = get_test_data(num_train=500,
+                                                         num_test=200,
+                                                         input_shape=input_shape,
+                                                         classification=True,
+                                                         num_classes=4)
+    y_train = np_utils.to_categorical(y_train)
+    y_test = np_utils.to_categorical(y_test)
+
+    model = Sequential([
+        Conv2D(filters=8, kernel_size=3,
+               activation='relu',
+               input_shape=input_shape),
+        MaxPooling2D(pool_size=2),
+        Conv2D(filters=4, kernel_size=(3, 3),
+               activation='relu', padding='same'),
+        GlobalAveragePooling2D(),
+        Dense(y_test.shape[-1], activation='softmax')
+    ])
+    model.compile(loss='categorical_crossentropy',
+                  optimizer='rmsprop',
+                  metrics=['accuracy'])
+    tsb = callbacks.TensorBoard(log_dir=filepath, histogram_freq=1,
+                                write_images=True, write_grads=True,
+                                batch_size=16)
+    cbks = [tsb]
+    model.summary()
+    history = model.fit(x_train, y_train, epochs=2, batch_size=16,
+                        validation_data=(x_test, y_test),
+                        callbacks=cbks,
+                        verbose=0)
+    assert os.path.exists(filepath)
+    shutil.rmtree(filepath)
+
+
+@keras_test
+def test_CallbackValData():
+    np.random.seed(1337)
+    (X_train, y_train), (X_test, y_test) = get_test_data(num_train=train_samples,
+                                                         num_test=test_samples,
+                                                         input_shape=(input_dim,),
+                                                         classification=True,
+                                                         num_classes=num_class)
+    y_test = np_utils.to_categorical(y_test)
+    y_train = np_utils.to_categorical(y_train)
+    model = Sequential()
+    model.add(Dense(num_hidden, input_dim=input_dim, activation='relu'))
+    model.add(Dense(num_class, activation='softmax'))
+    model.compile(loss='categorical_crossentropy',
+                  optimizer='sgd',
+                  metrics=['accuracy'])
+
+    cbk = callbacks.LambdaCallback(on_train_end=lambda x: 1)
+    model.fit(X_train, y_train, batch_size=batch_size,
+              validation_data=(X_test, y_test), callbacks=[cbk], epochs=1)
+
+    def data_generator(train):
+        if train:
+            max_batch_index = len(X_train) // batch_size
+        else:
+            max_batch_index = len(X_test) // batch_size
+        i = 0
+        while 1:
+            if train:
+                yield (X_train[i * batch_size: (i + 1) * batch_size],
+                       y_train[i * batch_size: (i + 1) * batch_size])
+            else:
+                yield (X_test[i * batch_size: (i + 1) * batch_size],
+                       y_test[i * batch_size: (i + 1) * batch_size])
+            i += 1
+            i = i % max_batch_index
+
+    cbk2 = callbacks.LambdaCallback(on_train_end=lambda x: 1)
+    model.fit_generator(data_generator(True), len(X_train), epochs=1,
+                        validation_data=(X_test, y_test),
+                        callbacks=[cbk2])
+
+    # callback validation data should always have x, y, and sample weights
+    assert len(cbk.validation_data) == len(cbk2.validation_data) == 3
+    assert cbk.validation_data[0] is cbk2.validation_data[0]
+    assert cbk.validation_data[1] is cbk2.validation_data[1]
+    assert cbk.validation_data[2].shape == cbk2.validation_data[2].shape
+
+
 @keras_test
 def test_LambdaCallback():
     np.random.seed(1337)
@@ -386,7 +512,9 @@ def test_LambdaCallback():
                     reason="Requires tensorflow backend")
 def test_TensorBoard_with_ReduceLROnPlateau():
     import shutil
-    filepath = './logs'
+    np.random.seed(np.random.randint(1, 1e7))
+    filepath = './logs_' + str(np.random.randint(1, 1e4))
+
     (X_train, y_train), (X_test, y_test) = get_test_data(num_train=train_samples,
                                                          num_test=test_samples,
                                                          input_shape=(input_dim,),

tests/keras/utils/data_utils_test.py

@@ -9,8 +9,6 @@ from six.moves.urllib.parse import urljoin
 from keras.utils.data_utils import get_file
 from keras.utils.data_utils import validate_file
 from keras.utils.data_utils import _hash_file
-from keras import activations
-from keras import regularizers
 
 
 def test_data_utils():

tests/keras/utils/io_utils_test.py

@@ -0,0 +1,65 @@
+'''Tests for functions in io_utils.py.
+'''
+import os
+import pytest
+from keras.models import Sequential
+from keras.layers import Dense
+from keras.utils.io_utils import HDF5Matrix
+import numpy as np
+import warnings
+import h5py
+
+
+def create_dataset(h5_path='test.h5'):
+    X = np.random.randn(200, 10).astype('float32')
+    y = np.random.randint(0, 2, size=(200, 1))
+    f = h5py.File(h5_path, 'w')
+    # Creating dataset to store features
+    X_dset = f.create_dataset('my_data', (200, 10), dtype='f')
+    X_dset[:] = X
+    # Creating dataset to store labels
+    y_dset = f.create_dataset('my_labels', (200, 1), dtype='i')
+    y_dset[:] = y
+    f.close()
+
+
+def test_io_utils():
+    '''Tests the HDF5Matrix code using the sample from @jfsantos at
+    https://gist.github.com/jfsantos/e2ef822c744357a4ed16ec0c885100a3
+    '''
+    h5_path = 'test.h5'
+    create_dataset(h5_path)
+
+    # Instantiating HDF5Matrix for the training set, which is a slice of the first 150 elements
+    X_train = HDF5Matrix(h5_path, 'my_data', start=0, end=150)
+    y_train = HDF5Matrix(h5_path, 'my_labels', start=0, end=150)
+
+    # Likewise for the test set
+    X_test = HDF5Matrix(h5_path, 'my_data', start=150, end=200)
+    y_test = HDF5Matrix(h5_path, 'my_labels', start=150, end=200)
+
+    # HDF5Matrix behave more or less like Numpy matrices with regards to indexing
+    assert y_train.shape == (150, 1), 'HDF5Matrix shape should match input array'
+    # But they do not support negative indices, so don't try print(X_train[-1])
+
+    model = Sequential()
+    model.add(Dense(64, input_shape=(10,), activation='relu'))
+    model.add(Dense(1, activation='sigmoid'))
+
+    model.compile(loss='binary_crossentropy', optimizer='sgd')
+
+    # Note: you have to use shuffle='batch' or False with HDF5Matrix
+    model.fit(X_train, y_train, batch_size=32, shuffle='batch', verbose=False)
+    # test that evalutation and prediction don't crash and return reasonable results
+    out_pred = model.predict(X_test, batch_size=32, verbose=False)
+    out_eval = model.evaluate(X_test, y_test, batch_size=32, verbose=False)
+
+    assert out_pred.shape == (50, 1), 'Prediction shape does not match'
+    assert out_eval.shape == (), 'Shape of evaluation does not match'
+    assert out_eval > 0, 'Evaluation value does not meet criteria: {}'.format(out_eval)
+
+    os.remove(h5_path)
+
+
+if __name__ == '__main__':
+    pytest.main([__file__])

tests/keras/wrappers/scikit_learn_test.py

@@ -2,8 +2,6 @@ import pytest
 import numpy as np
 
 from keras.utils.test_utils import get_test_data
-from keras.utils import np_utils
-from keras import backend as K
 
 from keras.models import Sequential
 from keras.layers.core import Dense, Activation

tests/test_model_saving.py

@@ -4,6 +4,7 @@ import tempfile
 import numpy as np
 from numpy.testing import assert_allclose
 
+from keras import backend as K
 from keras.models import Model, Sequential
 from keras.layers import Dense, Lambda, RepeatVector, TimeDistributed
 from keras.layers import Input
@@ -76,7 +77,7 @@ def test_sequential_model_saving_2():
 
 
 @keras_test
-def test_fuctional_model_saving():
+def test_functional_model_saving():
     input = Input(shape=(3,))
     x = Dense(2)(input)
     output = Dense(3)(x)
@@ -131,10 +132,11 @@ def test_saving_multiple_metrics_outputs():
 
 @keras_test
 def test_saving_without_compilation():
+    """Test saving model without compiling.
+    """
     model = Sequential()
     model.add(Dense(2, input_shape=(3,)))
     model.add(Dense(3))
-    model.compile(loss='mse', optimizer='sgd', metrics=['acc'])
 
     _, fname = tempfile.mkstemp('.h5')
     save_model(model, fname)
@@ -291,5 +293,49 @@ def test_saving_lambda_custom_objects():
     assert_allclose(out, out2, atol=1e-05)
 
 
+@keras_test
+def test_saving_lambda_numpy_array_arguments():
+    mean = np.random.random((4, 2, 3))
+    std = np.abs(np.random.random((4, 2, 3))) + 1e-5
+    input = Input(shape=(4, 2, 3))
+    output = Lambda(lambda image, mu, std: (image - mu) / std,
+                    arguments={'mu': mean, 'std': std})(input)
+    model = Model(input, output)
+    model.compile(loss='mse', optimizer='sgd', metrics=['acc'])
+
+    _, fname = tempfile.mkstemp('.h5')
+    save_model(model, fname)
+
+    model = load_model(fname)
+    os.remove(fname)
+
+    assert_allclose(mean, model.layers[1].arguments['mu'])
+    assert_allclose(std, model.layers[1].arguments['std'])
+
+
+@keras_test
+def test_saving_custom_activation_function():
+    x = Input(shape=(3,))
+    output = Dense(3, activation=K.cos)(x)
+
+    model = Model(x, output)
+    model.compile(loss=losses.MSE,
+                  optimizer=optimizers.RMSprop(lr=0.0001),
+                  metrics=[metrics.categorical_accuracy])
+    x = np.random.random((1, 3))
+    y = np.random.random((1, 3))
+    model.train_on_batch(x, y)
+
+    out = model.predict(x)
+    _, fname = tempfile.mkstemp('.h5')
+    save_model(model, fname)
+
+    model = load_model(fname, custom_objects={'cos': K.cos})
+    os.remove(fname)
+
+    out2 = model.predict(x)
+    assert_allclose(out, out2, atol=1e-05)
+
+
 if __name__ == '__main__':
     pytest.main([__file__])