Update backend docs.

* conv3d_tranpose in tf and th

* fix _preprocess_deconv_output_shape error

* cntk conv3d_tranpose

* conv3d_tranpose test

* formatting

* cleanup tests

* fix incorrect axis ordering and docs

* fix incorrect axis ordering and docs

* deconv3d_output_shape to fix errors

* remove conv2d_transpose reference in theano backend

* remove kernel_size loop from test

* put depth first in test and add dim to invalid use case input

* formatting - removed extra line

* fix pep8

* remove extraneous args from tf conv3d_transpose function

* default val for data_format=None

.github/stale.yml

@@ -0,0 +1,19 @@
+# Configuration for probot-stale - https://github.com/probot/stale
+
+# Number of days of inactivity before an Issue or Pull Request becomes stale
+daysUntilStale: 90
+# Number of days of inactivity before a stale Issue or Pull Request is closed
+daysUntilClose: 30
+# Issues or Pull Requests with these labels will never be considered stale
+exemptLabels:
+  - bug
+  - Announcement
+  - help wanted
+  - To investigate
+# Label to use when marking as stale
+staleLabel: stale
+# Comment to post when marking as stale. Set to `false` to disable
+markComment: >
+  This issue has been automatically marked as stale because it has not had
+  recent activity. It will be closed after 30 days if no further activity 
+  occurs, but feel free to re-open a closed issue if needed.

.gitignore

@@ -19,3 +19,4 @@ examples/img/*
 
 # developer environments
 .idea
+.vscode

.travis.yml

@@ -7,14 +7,20 @@ matrix:
           env: KERAS_BACKEND=tensorflow TEST_MODE=PEP8
         - python: 2.7
           env: KERAS_BACKEND=tensorflow TEST_MODE=INTEGRATION_TESTS
+        - python: 3.5
+          env: KERAS_BACKEND=tensorflow TEST_MODE=DOC
         - python: 2.7
           env: KERAS_BACKEND=tensorflow
         - python: 3.5
           env: KERAS_BACKEND=tensorflow
         - python: 2.7
-          env: KERAS_BACKEND=theano
+          env: KERAS_BACKEND=theano THEANO_FLAGS=optimizer=fast_compile
         - python: 3.5
-          env: KERAS_BACKEND=theano
+          env: KERAS_BACKEND=theano THEANO_FLAGS=optimizer=fast_compile
+        - python: 2.7
+          env: KERAS_BACKEND=cntk
+        - python: 3.5
+          env: KERAS_BACKEND=cntk
 install:
   # code below is taken from http://conda.pydata.org/docs/travis.html
   # We do this conditionally because it saves us some downloading if the
@@ -34,7 +40,7 @@ install:
 
   - conda create -q -n test-environment python=$TRAVIS_PYTHON_VERSION numpy scipy matplotlib pandas pytest h5py
   - source activate test-environment
-  - pip install git+git://github.com/Theano/Theano.git
+  - pip install theano
 
   # install PIL for preprocessing tests
   - if [[ "$TRAVIS_PYTHON_VERSION" == "2.7" ]]; then
@@ -45,8 +51,24 @@ install:
 
   - pip install -e .[tests]
 
-  # install TensorFlow
+  # install TensorFlow (CPU version).
   - pip install tensorflow
+  
+  # install cntk
+  - if [[ "$TRAVIS_PYTHON_VERSION" == "2.7" ]]; then
+      pip install https://cntk.ai/PythonWheel/CPU-Only/cntk-2.0-cp27-cp27mu-linux_x86_64.whl;
+    elif [[ "$TRAVIS_PYTHON_VERSION" == "3.5" ]]; then
+      pip install https://cntk.ai/PythonWheel/CPU-Only/cntk-2.0-cp35-cp35m-linux_x86_64.whl;
+    fi
+
+  #install open mpi
+  - rm -rf ~/mpi
+  - mkdir ~/mpi
+  - pushd ~/mpi
+  - wget http://cntk.ai/PythonWheel/ForKeras/depends/openmpi_1.10-3.zip
+  - unzip ./openmpi_1.10-3.zip
+  - sudo dpkg -i openmpi_1.10-3.deb
+  - popd
 
 # command to run tests
 script:
@@ -61,8 +83,8 @@ script:
        PYTHONPATH=$PWD:$PYTHONPATH py.test tests/integration_tests;
     elif [[ "$TEST_MODE" == "PEP8" ]]; then
        PYTHONPATH=$PWD:$PYTHONPATH py.test --pep8 -m pep8 -n0;
+    elif [[ "$TEST_MODE" == "DOC" ]]; then
+        PYTHONPATH=$PWD:$PYTHONPATH py.test tests/test_documentation.py;
     else
-       PYTHONPATH=$PWD:$PYTHONPATH py.test tests/ --ignore=tests/integration_tests;
+       PYTHONPATH=$PWD:$PYTHONPATH py.test tests/ --ignore=tests/integration_tests --ignore=tests/test_documentation.py --cov=keras tests/ --cov-fail-under 80 --cov-report term-missing;
     fi
-after_success:
-  - coveralls

CONTRIBUTING.md

@@ -19,6 +19,7 @@ To easily update Theano: `pip install git+git://github.com/Theano/Theano.git --u
 
 The more information you provide, the easier it is for us to validate that there is a bug and the faster we'll be able to take action. If you want your issue to be resolved quickly, following the steps above is crucial.
 
+---
 
 ## Requesting a Feature
 
@@ -31,45 +32,58 @@ You can also use Github issues to request features you would like to see in Kera
 3. After discussing the feature you may choose to attempt a Pull Request. If you're at all able, start writing some code. We always have more work to do than time to do it. If you can write some code then that will speed the process along.
 
 
+---
+
 ## Requests for Contributions
 
 [This is the board](https://github.com/fchollet/keras/projects/1) where we list current outstanding issues and features to be added. If you want to start contributing to Keras, this is the place to start.
 
 
+---
+
 ## Pull Requests
 
 **Where should I submit my pull request?**
 
 1. **Keras improvements and bugfixes** go to the [Keras `master` branch](https://github.com/fchollet/keras/tree/master).
-2. **New features** such as layers and datasets go to [keras-contrib](https://github.com/farizrahman4u/keras-contrib). Unless it is a new feature listed in [Requests for Contributions](https://github.com/fchollet/keras/projects/1), in which case it belongs in core Keras.
+2. **Experimental new features** such as layers and datasets go to [keras-contrib](https://github.com/farizrahman4u/keras-contrib). Unless it is a new feature listed in [Requests for Contributions](https://github.com/fchollet/keras/projects/1), in which case it belongs in core Keras. If you think your feature belongs in core Keras, you can submit a design doc to explain your feature and argue for it (see explainations below).
 
 Here's a quick guide to submitting your improvements:
 
-1. If your PR introduces a change in functionality, make sure you start by opening an issue to discuss whether the change should be made, and how to handle it. This will save you from having your PR closed down the road! Of course, if your PR is a simple bug fix, you don't need to do that.
+1. If your PR introduces a change in functionality, make sure you start by writing a design doc and sending it to the Keras mailing list to discuss whether the change should be made, and how to handle it. This will save you from having your PR closed down the road! Of course, if your PR is a simple bug fix, you don't need to do that. The process for writing and submitting design docs is as follow:
+    - Start from [this Google Doc template](https://docs.google.com/document/d/1ZXNfce77LDW9tFAj6U5ctaJmI5mT7CQXOFMEAZo-mAA/edit#), and copy it to new Google doc.
+    - Fill in the content. Note that you will need to insert code examples. To insert code, use a Google Doc extension such as [CodePretty](https://chrome.google.com/webstore/detail/code-pretty/igjbncgfgnfpbnifnnlcmjfbnidkndnh?hl=en) (there are several such extensions available).
+    - Set sharing settings to "everyone with the link is allowed to comment"
+    - Send the document to `keras-users@googlegroups.com` with a subject that starts with `[API DESIGN REVIEW]` (all caps) so that we notice it.
+    - Wait for comments, and answer them as they come. Edit the proposal as necessary.
+    - The proposal will finally be approved or rejected. Once approved, you can send out Pull Requests or ask others to write Pull Requests.
 
-2. Write the code. This is the hard part!
 
-3. Make sure any new function or class you introduce has proper docstrings. Make sure any code you touch still has up-to-date docstrings and documentation.
+2. Write the code (or get others to write it). This is the hard part!
+
+3. Make sure any new function or class you introduce has proper docstrings. Make sure any code you touch still has up-to-date docstrings and documentation. **Docstring style should be respected.** In particular, they should be formatted in MarkDown, and there should be sections for `Arguments`, `Returns`, `Raises` (if applicable). Look at other docstrings in the codebase for examples.
 
 4. Write tests. Your code should have full unit test coverage. If you want to see your PR merged promptly, this is crucial.
 
 5. Run our test suite locally. It's easy: from the Keras folder, simply run: `py.test tests/`.
-  - You will need to install the test requirements as well: `pip install -e .[tests]`.
+    - You will need to install the test requirements as well: `pip install -e .[tests]`.
 
 6. Make sure all tests are passing:
-  - with the Theano backend, on Python 2.7 and Python 3.5
-  - with the TensorFlow backend, on Python 2.7
+    - with the Theano backend, on Python 2.7 and Python 3.5. Make sure you have the development version of Theano.
+    - with the TensorFlow backend, on Python 2.7 and Python 3.5. Make sure you have the development version of TensorFlow.
 
 7. We use PEP8 syntax conventions, but we aren't dogmatic when it comes to line length. Make sure your lines stay reasonably sized, though. To make your life easier, we recommend running a PEP8 linter:
-  - Install PEP8 packages: `pip install pep8 pytest-pep8 autopep8`
-  - Run a standalone PEP8 check: `py.test --pep8 -m pep8`
-  - You can automatically fix some PEP8 error by running: `autopep8 -i --select <errors> <FILENAME>` for example: `autopep8 -i --select E128 tests/keras/backend/test_backends.py`
+    - Install PEP8 packages: `pip install pep8 pytest-pep8 autopep8`
+    - Run a standalone PEP8 check: `py.test --pep8 -m pep8`
+    - You can automatically fix some PEP8 error by running: `autopep8 -i --select <errors> <FILENAME>` for example: `autopep8 -i --select E128 tests/keras/backend/test_backends.py`
 
-8. When committing, use appropriate, descriptive commit messages. Make sure that your branch history is not a string of "bug fix", "fix", "oops", etc. When submitting your PR, squash your commits into a single commit with an appropriate commit message, to make sure the project history stays clean and readable. See ['rebase and squash'](http://rebaseandsqua.sh/) for technical help on how to squash your commits.
+8. When committing, use appropriate, descriptive commit messages.
 
 9. Update the documentation. If introducing new functionality, make sure you include code snippets demonstrating the usage of your new feature.
 
-10. Submit your PR. If your changes have been approved in a previous discussion, and if you have complete (and passing) unit tests, your PR is likely to be merged promptly. Otherwise, well...
+10. Submit your PR. If your changes have been approved in a previous discussion, and if you have complete (and passing) unit tests as well as proper docstrings/documentation, your PR is likely to be merged promptly. Otherwise, well...
+
+---
 
 ## Adding new examples
 

LICENSE

@@ -8,8 +8,12 @@ All contributions by Google:
 Copyright (c) 2015, Google, Inc.
 All rights reserved.
 
+All contributions by Microsoft:
+Copyright (c) 2017, Microsoft, Inc.
+All rights reserved.
+
 All other contributions:
-Copyright (c) 2015, the respective contributors.
+Copyright (c) 2015 - 2017, the respective contributors.
 All rights reserved.
 
 Each contributor holds copyright over their respective contributions.

README.md

@@ -1,11 +1,11 @@
-# Keras: Deep Learning library for TensorFlow and Theano
+# Keras: Deep Learning for Python
 
 [![Build Status](https://travis-ci.org/fchollet/keras.svg?branch=master)](https://travis-ci.org/fchollet/keras)
 [![license](https://img.shields.io/github/license/mashape/apistatus.svg?maxAge=2592000)](https://github.com/fchollet/keras/blob/master/LICENSE)
 
 ## You have just found Keras.
 
-Keras is a high-level neural networks API, written in Python and capable of running on top of either [TensorFlow](https://github.com/tensorflow/tensorflow) or [Theano](https://github.com/Theano/Theano). It was developed with a focus on enabling fast experimentation. *Being able to go from idea to result with the least possible delay is key to doing good research.*
+Keras is a high-level neural networks API, written in Python and capable of running on top of [TensorFlow](https://github.com/tensorflow/tensorflow), [CNTK](https://github.com/Microsoft/cntk), or [Theano](https://github.com/Theano/Theano). It was developed with a focus on enabling fast experimentation. *Being able to go from idea to result with the least possible delay is key to doing good research.*
 
 Use Keras if you need a deep learning library that:
 
@@ -125,6 +125,11 @@ Keras uses the following dependencies:
 - TensorFlow
     - [See installation instructions](https://www.tensorflow.org/install/).
 
+*When using the CNTK backend:*
+
+- CNTK
+    - [See installation instructions](https://docs.microsoft.com/en-us/cognitive-toolkit/setup-cntk-on-your-machine).
+
 *When using the Theano backend:*
 
 - Theano
@@ -143,7 +148,7 @@ sudo pip install keras
 ------------------
 
 
-## Switching from TensorFlow to Theano
+## Switching from TensorFlow to CNTK or Theano
 
 By default, Keras will use TensorFlow as its tensor manipulation library. [Follow these instructions](http://keras.io/backend/) to configure the Keras backend.
 

docker/Dockerfile

@@ -7,10 +7,10 @@ RUN mkdir -p $CONDA_DIR && \
     echo export PATH=$CONDA_DIR/bin:'$PATH' > /etc/profile.d/conda.sh && \
     apt-get update && \
     apt-get install -y wget git libhdf5-dev g++ graphviz && \
-    wget --quiet https://repo.continuum.io/miniconda/Miniconda3-3.9.1-Linux-x86_64.sh && \
-    echo "6c6b44acdd0bc4229377ee10d52c8ac6160c336d9cdd669db7371aa9344e1ac3 *Miniconda3-3.9.1-Linux-x86_64.sh" | sha256sum -c - && \
-    /bin/bash /Miniconda3-3.9.1-Linux-x86_64.sh -f -b -p $CONDA_DIR && \
-    rm Miniconda3-3.9.1-Linux-x86_64.sh
+    wget --quiet https://repo.continuum.io/miniconda/Miniconda3-4.2.12-Linux-x86_64.sh && \
+    echo "c59b3dd3cad550ac7596e0d599b91e75d88826db132e4146030ef471bb434e9a *Miniconda3-4.2.12-Linux-x86_64.sh" | sha256sum -c - && \
+    /bin/bash /Miniconda3-4.2.12-Linux-x86_64.sh -f -b -p $CONDA_DIR && \
+    rm Miniconda3-4.2.12-Linux-x86_64.sh
 
 ENV NB_USER keras
 ENV NB_UID 1000
@@ -24,13 +24,14 @@ RUN useradd -m -s /bin/bash -N -u $NB_UID $NB_USER && \
 USER keras
 
 # Python
-ARG python_version=3.5.2
-ARG tensorflow_version=0.12.0rc0-cp35-cp35m
+ARG python_version=3.5
+
 RUN conda install -y python=${python_version} && \
-    pip install https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow_gpu-${tensorflow_version}-linux_x86_64.whl && \
-    pip install git+git://github.com/Theano/Theano.git && \
-    pip install ipdb pytest pytest-cov python-coveralls coverage==3.7.1 pytest-xdist pep8 pytest-pep8 pydot_ng && \
-    conda install Pillow scikit-learn notebook pandas matplotlib nose pyyaml six h5py && \
+    pip install --upgrade pip && \
+    pip install tensorflow-gpu && \
+    conda install Pillow scikit-learn notebook pandas matplotlib mkl nose pyyaml six h5py && \
+    conda install theano pygpu && \
+    git clone git://github.com/fchollet/keras.git /src && pip install -e /src[tests] && \
     pip install git+git://github.com/fchollet/keras.git && \
     conda clean -yt
 

docker/theanorc

@@ -1,5 +1,5 @@
 [global]
 floatX = float32
 optimizer=None
-device = gpu
+device = cuda
 

docs/autogen.py

@@ -8,9 +8,7 @@ Index
 - Getting started
     Getting started with the sequential model
     Getting started with the functional api
-    Examples
     FAQ
-    Installation guide
 
 - Models
     About Keras models
@@ -26,18 +24,23 @@ Index
         explain common layer functions: get_weights, set_weights, get_config
         explain input_shape
         explain usage on non-Keras tensors
-    Core layers
-    Convolutional
-    Recurrent
-    Embeddings
-    Normalization
-    Advanced activations
-    Noise
+    Core Layers
+    Convolutional Layers
+    Pooling Layers
+    Locally-connected Layers
+    Recurrent Layers
+    Embedding Layers
+    Merge Layers
+    Advanced Activations Layers
+    Normalization Layers
+    Noise Layers
+    Layer Wrappers
+    Writing your own Keras layers
 
 - Preprocessing
-    Image preprocessing
-    Text preprocessing
-    Sequence preprocessing
+    Sequence Preprocessing
+    Text Preprocessing
+    Image Preprocessing
 
 Losses
 Metrics
@@ -45,12 +48,15 @@ Optimizers
 Activations
 Callbacks
 Datasets
+Applications
 Backend
-Initializations
+Initializers
 Regularizers
 Constraints
 Visualization
 Scikit-learn API
+Utils
+Contributing
 
 '''
 from __future__ import print_function
@@ -114,14 +120,13 @@ PAGES = [
             models.Sequential.fit,
             models.Sequential.evaluate,
             models.Sequential.predict,
-            models.Sequential.predict_classes,
-            models.Sequential.predict_proba,
             models.Sequential.train_on_batch,
             models.Sequential.test_on_batch,
             models.Sequential.predict_on_batch,
             models.Sequential.fit_generator,
             models.Sequential.evaluate_generator,
             models.Sequential.predict_generator,
+            models.Sequential.get_layer,
         ],
     },
     {
@@ -281,7 +286,8 @@ PAGES = [
         'page': 'utils.md',
         'all_module_functions': [utils],
         'classes': [utils.CustomObjectScope,
-                    utils.HDF5Matrix]
+                    utils.HDF5Matrix,
+                    utils.Sequence]
     },
 ]
 
@@ -315,9 +321,11 @@ def get_classes_ancestors(classes):
 
 
 def get_function_signature(function, method=True):
-    signature = getattr(function, '_legacy_support_signature', None)
-    if signature is None:
+    wrapped = getattr(function, '_original_function', None)
+    if wrapped is None:
         signature = inspect.getargspec(function)
+    else:
+        signature = inspect.getargspec(wrapped)
     defaults = signature.defaults
     if method:
         args = signature.args[1:]
@@ -382,7 +390,7 @@ def process_class_docstring(docstring):
                        r'\n    __\1__\n\n',
                        docstring)
 
-    docstring = re.sub(r'    ([^\s\\]+):(.*)\n',
+    docstring = re.sub(r'    ([^\s\\\(]+):(.*)\n',
                        r'    - __\1__:\2\n',
                        docstring)
 
@@ -400,7 +408,7 @@ def process_function_docstring(docstring):
                        r'\n        __\1__\n\n',
                        docstring)
 
-    docstring = re.sub(r'    ([^\s\\]+):(.*)\n',
+    docstring = re.sub(r'    ([^\s\\\(]+):(.*)\n',
                        r'    - __\1__:\2\n',
                        docstring)
 
@@ -509,3 +517,5 @@ for page_data in PAGES:
     if not os.path.exists(subdir):
         os.makedirs(subdir)
     open(path, 'w').write(mkdown)
+
+shutil.copyfile('../CONTRIBUTING.md', 'sources/contributing.md')

docs/mkdocs.yml

@@ -51,3 +51,4 @@ pages:
 - Visualization: visualization.md
 - Scikit-learn API: scikit-learn-api.md
 - Utils: utils.md
+- Contributing: contributing.md

docs/templates/applications.md

@@ -15,7 +15,7 @@ Weights are downloaded automatically when instantiating a model. They are stored
 - [ResNet50](#resnet50)
 - [InceptionV3](#inceptionv3)
 
-All of these architectures (except Xception) are compatible with both TensorFlow and Theano, and upon instantiation the models will be built according to the image data format set in your Keras configuration file at `~/.keras/keras.json`. For instance, if you have set `image_data_format=tf`, then any model loaded from this repository will get built according to the TensorFlow data format convention, "Width-Height-Depth".
+All of these architectures (except Xception) are compatible with both TensorFlow and Theano, and upon instantiation the models will be built according to the image data format set in your Keras configuration file at `~/.keras/keras.json`. For instance, if you have set `image_data_format=channels_last`, then any model loaded from this repository will get built according to the TensorFlow data format convention, "Width-Height-Depth".
 
 The Xception model is only available for TensorFlow, due to its reliance on `SeparableConvolution` layers.
 
@@ -75,7 +75,7 @@ from keras.models import Model
 import numpy as np
 
 base_model = VGG19(weights='imagenet')
-model = Model(input=base_model.input, output=base_model.get_layer('block4_pool').output)
+model = Model(inputs=base_model.input, outputs=base_model.get_layer('block4_pool').output)
 
 img_path = 'elephant.jpg'
 img = image.load_img(img_path, target_size=(224, 224))
@@ -107,7 +107,7 @@ x = Dense(1024, activation='relu')(x)
 predictions = Dense(200, activation='softmax')(x)
 
 # this is the model we will train
-model = Model(input=base_model.input, output=predictions)
+model = Model(inputs=base_model.input, outputs=predictions)
 
 # first: train only the top layers (which were randomly initialized)
 # i.e. freeze all convolutional InceptionV3 layers
@@ -130,10 +130,10 @@ for i, layer in enumerate(base_model.layers):
    print(i, layer.name)
 
 # we chose to train the top 2 inception blocks, i.e. we will freeze
-# the first 172 layers and unfreeze the rest:
-for layer in model.layers[:172]:
+# the first 249 layers and unfreeze the rest:
+for layer in model.layers[:249]:
    layer.trainable = False
-for layer in model.layers[172:]:
+for layer in model.layers[249:]:
    layer.trainable = True
 
 # we need to recompile the model for these modifications to take effect
@@ -253,7 +253,7 @@ The default input size for this model is 224x224.
 - input_shape: optional shape tuple, only to be specified
     if `include_top` is False (otherwise the input shape
     has to be `(224, 224, 3)` (with `channels_last` data format)
-    or `(3, 224, 244)` (with `channels_first` data format).
+    or `(3, 224, 224)` (with `channels_first` data format).
     It should have exactly 3 inputs channels,
     and width and height should be no smaller than 48.
     E.g. `(200, 200, 3)` would be one valid value.
@@ -309,7 +309,7 @@ The default input size for this model is 224x224.
 - input_shape: optional shape tuple, only to be specified
     if `include_top` is False (otherwise the input shape
     has to be `(224, 224, 3)` (with `channels_last` data format)
-    or `(3, 224, 244)` (with `channels_first` data format).
+    or `(3, 224, 224)` (with `channels_first` data format).
     It should have exactly 3 inputs channels,
     and width and height should be no smaller than 48.
     E.g. `(200, 200, 3)` would be one valid value.
@@ -367,7 +367,7 @@ The default input size for this model is 224x224.
 - input_shape: optional shape tuple, only to be specified
     if `include_top` is False (otherwise the input shape
     has to be `(224, 224, 3)` (with `channels_last` data format)
-    or `(3, 224, 244)` (with `channels_first` data format).
+    or `(3, 224, 224)` (with `channels_first` data format).
     It should have exactly 3 inputs channels,
     and width and height should be no smaller than 197.
     E.g. `(200, 200, 3)` would be one valid value.

docs/templates/backend.md

@@ -4,12 +4,13 @@
 
 Keras is a model-level library, providing high-level building blocks for developing deep learning models. It does not handle itself low-level operations such as tensor products, convolutions and so on. Instead, it relies on a specialized, well-optimized tensor manipulation library to do so, serving as the "backend engine" of Keras. Rather than picking one single tensor library and making the implementation of Keras tied to that library, Keras handles the problem in a modular way, and several different backend engines can be plugged seamlessly into Keras.
 
-At this time, Keras has two backend implementations available: the **TensorFlow** backend and the **Theano** backend.
+At this time, Keras has three backend implementations available: the **TensorFlow** backend, the **Theano** backend, and the **CNTK** backend.
 
 - [TensorFlow](http://www.tensorflow.org/) is an open-source symbolic tensor manipulation framework developed by Google, Inc.
 - [Theano](http://deeplearning.net/software/theano/) is an open-source symbolic tensor manipulation framework developed by LISA/MILA Lab at Université de Montréal.
+- [CNTK](https://www.microsoft.com/en-us/cognitive-toolkit/) is an open-source, commercial-grade toolkit for deep learning developed by Microsoft.
 
-In the future, we are likely to add more backend options. Go ask Microsoft about how their CNTK backend project is doing.
+In the future, we are likely to add more backend options.
 
 ----
 
@@ -34,7 +35,7 @@ The default configuration file looks like this:
 }
 ```
 
-Simply change the field `backend` to either `"theano"` or `"tensorflow"`, and Keras will use the new configuration next time you run any Keras code.
+Simply change the field `backend` to `"theano"`, `"tensorflow"`, or `"cntk"`, and Keras will use the new configuration next time you run any Keras code.
 
 You can also define the environment variable ``KERAS_BACKEND`` and this will
 override what is defined in your config file :
@@ -65,7 +66,7 @@ You can change these settings by editing `$HOME/.keras/keras.json`.
   - For 3D data, `"channels_last"` assumes `(conv_dim1, conv_dim2, conv_dim3, channels)` while `"channels_first"` assumes `(channels, conv_dim1, conv_dim2, conv_dim3)`.
 * `epsilon`: float, a numeric fuzzing constant used to avoid dividing by zero in some operations.
 * `floatx`: string, `"float16"`, `"float32"`, or `"float64"`. Default float precision.
-* `backend`: string, `"tensorflow"` or `"theano"`.
+* `backend`: string, `"tensorflow"`, `"theano"`, or `"cntk"`.
 
 ----
 
@@ -75,7 +76,7 @@ If you want the Keras modules you write to be compatible with both Theano (`th`)
 
 You can import the backend module via:
 ```python
-*from keras import backend as K*
+from keras import backend as K
 ```
 
 The code below instantiates an input placeholder. It's equivalent to `tf.placeholder()` or `th.tensor.matrix()`, `th.tensor.tensor3()`, etc.

docs/templates/callbacks.md

@@ -36,14 +36,14 @@ class LossHistory(keras.callbacks.Callback):
         self.losses.append(logs.get('loss'))
 
 model = Sequential()
-model.add(Dense(10, input_dim=784, init='uniform'))
+model.add(Dense(10, input_dim=784, kernel_initializer='uniform'))
 model.add(Activation('softmax'))
 model.compile(loss='categorical_crossentropy', optimizer='rmsprop')
 
 history = LossHistory()
-model.fit(X_train, Y_train, batch_size=128, epochs=20, verbose=0, callbacks=[history])
+model.fit(x_train, y_train, batch_size=128, epochs=20, verbose=0, callbacks=[history])
 
-print history.losses
+print(history.losses)
 # outputs
 '''
 [0.66047596406559383, 0.3547245744908703, ..., 0.25953155204159617, 0.25901699725311789]
@@ -58,15 +58,13 @@ print history.losses
 from keras.callbacks import ModelCheckpoint
 
 model = Sequential()
-model.add(Dense(10, input_dim=784, init='uniform'))
+model.add(Dense(10, input_dim=784, kernel_initializer='uniform'))
 model.add(Activation('softmax'))
 model.compile(loss='categorical_crossentropy', optimizer='rmsprop')
 
 '''
 saves the model weights after each epoch if the validation loss decreased
 '''
-checkpointer = ModelCheckpoint(filepath="/tmp/weights.hdf5", verbose=1, save_best_only=True)
-model.fit(X_train, Y_train, batch_size=128, epochs=20, verbose=0, validation_data=(X_test, Y_test), callbacks=[checkpointer])
-
+checkpointer = ModelCheckpoint(filepath='/tmp/weights.hdf5', verbose=1, save_best_only=True)
+model.fit(x_train, y_train, batch_size=128, epochs=20, verbose=0, validation_data=(X_test, Y_test), callbacks=[checkpointer])
 ```
-

docs/templates/constraints.md

@@ -2,7 +2,7 @@
 
 Functions from the `constraints` module allow setting constraints (eg. non-negativity) on network parameters during optimization.
 
-The penalties are applied on a per-layer basis. The exact API will depend on the layer, but the layers `Dense`, `Convolution1D`, `Convolution2D` and `Convolution3D` have a unified API.
+The penalties are applied on a per-layer basis. The exact API will depend on the layer, but the layers `Dense`, `Conv1D`, `Conv2D` and `Conv3D` have a unified API.
 
 These layers expose 2 keyword arguments:
 
@@ -17,6 +17,7 @@ model.add(Dense(64, kernel_constraint=max_norm(2.)))
 
 ## Available constraints
 
-- __max_norm__(max_value=2, axis=0): maximum-norm constraint
-- __non_neg__(): non-negativity constraint
-- __unit_norm__(): unit-norm constraint, enforces the matrix to have unit norm along the last axis
+- __max_norm(max_value=2, axis=0)__: maximum-norm constraint
+- __non_neg()__: non-negativity constraint
+- __unit_norm(axis=0)__: unit-norm constraint
+- __min_max_norm(min_value=0.0, max_value=1.0, rate=1.0, axis=0)__:  minimum/maximum-norm constraint

docs/templates/datasets.md

@@ -55,7 +55,7 @@ As a convention, "0" does not stand for a specific word, but instead is used to
 ```python
 from keras.datasets import imdb
 
-(x_train, y_train), (x_test, y_test) = imdb.load_data(path="imdb_full.pkl",
+(x_train, y_train), (x_test, y_test) = imdb.load_data(path="imdb.npz",
                                                       num_words=None,
                                                       skip_top=0,
                                                       maxlen=None,
@@ -72,13 +72,13 @@ from keras.datasets import imdb
 - __Arguments:__
 
     - __path__: if you do not have the data locally (at `'~/.keras/datasets/' + path`), it will be downloaded to this location.
-    - __num_words__: integer or None. Top most frequent words to consider. Any less frequent word will appear as 0 in the sequence data.
-    - __skip_top__: integer. Top most frequent words to ignore (they will appear as 0s in the sequence data).
+    - __num_words__: integer or None. Top most frequent words to consider. Any less frequent word will appear as `oov_char` value in the sequence data.
+    - __skip_top__: integer. Top most frequent words to ignore (they will appear as `oov_char` value in the sequence data).
     - __maxlen__: int. Maximum sequence length. Any longer sequence will be truncated.
     - __seed__: int. Seed for reproducible data shuffling.
-    - __start_char__: char. The start of a sequence will be marked with this character.
+    - __start_char__: int. The start of a sequence will be marked with this character.
         Set to 1 because 0 is usually the padding character.
-    - __oov_char__: char. words that were cut out because of the `num_words`
+    - __oov_char__: int. words that were cut out because of the `num_words`
         or `skip_top` limit will be replaced with this character.
     - __index_from__: int. Index actual words with this index and higher.
 
@@ -94,7 +94,7 @@ Dataset of 11,228 newswires from Reuters, labeled over 46 topics. As with the IM
 ```python
 from keras.datasets import reuters
 
-(x_train, y_train), (x_test, y_test) = reuters.load_data(path="reuters.pkl",
+(x_train, y_train), (x_test, y_test) = reuters.load_data(path="reuters.npz",
                                                          num_words=None,
                                                          skip_top=0,
                                                          maxlen=None,
@@ -107,12 +107,12 @@ from keras.datasets import reuters
 
 The specifications are the same as that of the IMDB dataset, with the addition of:
 
-    - __test_split__: float. Fraction of the dataset to be used as test data.
+- __test_split__: float. Fraction of the dataset to be used as test data.
 
 This dataset also makes available the word index used for encoding the sequences:
 
 ```python
-word_index = reuters.get_word_index(path="reuters_word_index.pkl")
+word_index = reuters.get_word_index(path="reuters_word_index.json")
 ```
 
 - __Returns:__ A dictionary where key are words (str) and values are indexes (integer). eg. `word_index["giraffe"]` might return `1234`. 

docs/templates/getting-started/faq.md

@@ -16,6 +16,7 @@
 - [How can I remove a layer from a Sequential model?](#how-can-i-remove-a-layer-from-a-sequential-model)
 - [How can I use pre-trained models in Keras?](#how-can-i-use-pre-trained-models-in-keras)
 - [How can I use HDF5 inputs with Keras?](#how-can-i-use-hdf5-inputs-with-keras)
+- [Where is the Keras configuration file stored?](#where-is-the-keras-configuration-file-stored)
 
 ---
 
@@ -26,7 +27,7 @@ Please cite Keras in your publications if it helps your research. Here is an exa
 ```
 @misc{chollet2015keras,
   title={Keras},
-  author={Chollet, Fran\c{c}ois},
+  author={Chollet, Fran\c{c}ois and others},
   year={2015},
   publisher={GitHub},
   howpublished={\url{https://github.com/fchollet/keras}},
@@ -37,7 +38,7 @@ Please cite Keras in your publications if it helps your research. Here is an exa
 
 ### How can I run Keras on GPU?
 
-If you are running on the TensorFlow backend, your code will automatically run on GPU if any available GPU is detected.
+If you are running on the TensorFlow or CNTK backends, your code will automatically run on GPU if any available GPU is detected.
 
 If you are running on the Theano backend, you can use one of the following methods:
 
@@ -152,16 +153,16 @@ For example:
 """
 Assume original model looks like this:
     model = Sequential()
-    model.add(Dense(2, input_dim=3, name="dense_1"))
-    model.add(Dense(3, name="dense_2"))
+    model.add(Dense(2, input_dim=3, name='dense_1'))
+    model.add(Dense(3, name='dense_2'))
     ...
     model.save_weights(fname)
 """
 
 # new model
 model = Sequential()
-model.add(Dense(2, input_dim=3, name="dense_1"))  # will be loaded
-model.add(Dense(10, name="new_dense"))  # will not be loaded
+model.add(Dense(2, input_dim=3, name='dense_1'))  # will be loaded
+model.add(Dense(10, name='new_dense'))  # will not be loaded
 
 # load weights from first model; will only affect the first layer, dense_1.
 model.load_weights(fname, by_name=True)
@@ -200,7 +201,7 @@ from keras import backend as K
 # with a Sequential model
 get_3rd_layer_output = K.function([model.layers[0].input],
                                   [model.layers[3].output])
-layer_output = get_3rd_layer_output([X])[0]
+layer_output = get_3rd_layer_output([x])[0]
 ```
 
 Similarly, you could build a Theano and TensorFlow function directly.
@@ -213,17 +214,17 @@ get_3rd_layer_output = K.function([model.layers[0].input, K.learning_phase()],
                                   [model.layers[3].output])
 
 # output in test mode = 0
-layer_output = get_3rd_layer_output([X, 0])[0]
+layer_output = get_3rd_layer_output([x, 0])[0]
 
 # output in train mode = 1
-layer_output = get_3rd_layer_output([X, 1])[0]
+layer_output = get_3rd_layer_output([x, 1])[0]
 ```
 
 ---
 
 ### How can I use Keras with datasets that don't fit in memory?
 
-You can do batch training using `model.train_on_batch(X, y)` and `model.test_on_batch(X, y)`. See the [models documentation](/models/sequential).
+You can do batch training using `model.train_on_batch(x, y)` and `model.test_on_batch(x, y)`. See the [models documentation](/models/sequential).
 
 Alternatively, you can write a generator that yields batches of training data and use the method `model.fit_generator(data_generator, steps_per_epoch, epochs)`.
 
@@ -238,7 +239,7 @@ You can use an `EarlyStopping` callback:
 ```python
 from keras.callbacks import EarlyStopping
 early_stopping = EarlyStopping(monitor='val_loss', patience=2)
-model.fit(X, y, validation_split=0.2, callbacks=[early_stopping])
+model.fit(x, y, validation_split=0.2, callbacks=[early_stopping])
 ```
 
 Find out more in the [callbacks documentation](/callbacks).
@@ -267,7 +268,7 @@ Validation data is never shuffled.
 The `model.fit` method returns an `History` callback, which has a `history` attribute containing the lists of successive losses and other metrics.
 
 ```python
-hist = model.fit(X, y, validation_split=0.2)
+hist = model.fit(x, y, validation_split=0.2)
 print(hist.history)
 ```
 
@@ -314,7 +315,7 @@ Making a RNN stateful means that the states for the samples of each batch will b
 When using stateful RNNs, it is therefore assumed that:
 
 - all batches have the same number of samples
-- If `X1` and `X2` are successive batches of samples, then `X2[i]` is the follow-up sequence to `X1[i]`, for every `i`.
+- If `x1` and `x2` are successive batches of samples, then `x2[i]` is the follow-up sequence to `x1[i]`, for every `i`.
 
 To use statefulness in RNNs, you need to:
 
@@ -331,7 +332,7 @@ Example:
 
 ```python
 
-X  # this is our input data, of shape (32, 21, 16)
+x  # this is our input data, of shape (32, 21, 16)
 # we will feed it to our model in sequences of length 10
 
 model = Sequential()
@@ -341,10 +342,10 @@ model.add(Dense(16, activation='softmax'))
 model.compile(optimizer='rmsprop', loss='categorical_crossentropy')
 
 # we train the network to predict the 11th timestep given the first 10:
-model.train_on_batch(X[:, :10, :], np.reshape(X[:, 10, :], (32, 16)))
+model.train_on_batch(x[:, :10, :], np.reshape(x[:, 10, :], (32, 16)))
 
 # the state of the network has changed. We can feed the follow-up sequences:
-model.train_on_batch(X[:, 10:20, :], np.reshape(X[:, 20, :], (32, 16)))
+model.train_on_batch(x[:, 10:20, :], np.reshape(x[:, 20, :], (32, 16)))
 
 # let's reset the states of the LSTM layer:
 model.reset_states()
@@ -410,13 +411,46 @@ The VGG16 model is also the basis for several Keras example scripts:
 
 ### How can I use HDF5 inputs with Keras?
 
-You can use the `HDF5Matrix` class from `keras.utils.io_utils`. See [the documentation](/io_utils/#HDF5Matrix) for details.
+You can use the `HDF5Matrix` class from `keras.utils.io_utils`. See [the HDF5Matrix documentation](/utils/#hdf5matrix) for details.
 
 You can also directly use a HDF5 dataset:
 
 ```python
 import h5py
 with h5py.File('input/file.hdf5', 'r') as f:
-    X_data = f['X_data']
-    model.predict(X_data)
+    x_data = f['x_data']
+    model.predict(x_data)
 ```
+
+---
+
+### Where is the Keras configuration file stored?
+
+The default directory where all Keras data is stored is:
+
+```bash
+$HOME/.keras/
+```
+
+Note that Windows users should replace `$HOME` with `%USERPROFILE%`.
+In case Keras cannot create the above directory (e.g. due to permission issues), `/tmp/.keras/` is used as a backup.
+
+The Keras configuration file is a JSON file stored at `$HOME/.keras/keras.json`. The default configuration file looks like this:
+
+```
+{
+    "image_data_format": "channels_last",
+    "epsilon": 1e-07,
+    "floatx": "float32",
+    "backend": "tensorflow"
+}
+```
+
+It contains the following fields:
+
+- The image data format to be used as default by image processing layers and utilities (either `channels_last` or `channels_first`).
+- The `epsilon` numerical fuzz factor to be used to prevent division by zero in some operations.
+- The default float data type.
+- The default backend. See the [backend documentation](/backend).
+
+Likewise, cached dataset files, such as those downloaded with [`get_file()`](/utils/#get_file), are stored by default in `$HOME/.keras/datasets/`.

docs/templates/getting-started/functional-api-guide.md

@@ -361,7 +361,7 @@ from keras.models import Model, Sequential
 # First, let's define a vision model using a Sequential model.
 # This model will encode an image into a vector.
 vision_model = Sequential()
-vision_model.add(Conv2D(64, (3, 3) activation='relu', padding='same', input_shape=(3, 224, 224)))
+vision_model.add(Conv2D(64, (3, 3), activation='relu', padding='same', input_shape=(3, 224, 224)))
 vision_model.add(Conv2D(64, (3, 3), activation='relu'))
 vision_model.add(MaxPooling2D((2, 2)))
 vision_model.add(Conv2D(128, (3, 3), activation='relu', padding='same'))

docs/templates/getting-started/sequential-model-guide.md

@@ -9,7 +9,7 @@ from keras.models import Sequential
 from keras.layers import Dense, Activation
 
 model = Sequential([
-    Dense(32, input_dim=784),
+    Dense(32, input_shape=(784,)),
     Activation('relu'),
     Dense(10),
     Activation('softmax'),
@@ -121,10 +121,10 @@ data = np.random.random((1000, 100))
 labels = np.random.randint(10, size=(1000, 1))
 
 # Convert labels to categorical one-hot encoding
-binary_labels = keras.utils.to_categorical(labels, num_classes=10)
+one_hot_labels = keras.utils.to_categorical(labels, num_classes=10)
 
 # Train the model, iterating on the data in batches of 32 samples
-model.fit(data, binary_labels, epochs=10, batch_size=32)
+model.fit(data, one_hot_labels, epochs=10, batch_size=32)
 ```
 
 ----
@@ -152,6 +152,13 @@ from keras.models import Sequential
 from keras.layers import Dense, Dropout, Activation
 from keras.optimizers import SGD
 
+# Generate dummy data
+import numpy as np
+x_train = np.random.random((1000, 20))
+y_train = keras.utils.to_categorical(np.random.randint(10, size=(1000, 1)), num_classes=10)
+x_test = np.random.random((100, 20))
+y_test = keras.utils.to_categorical(np.random.randint(10, size=(100, 1)), num_classes=10)
+
 model = Sequential()
 # Dense(64) is a fully-connected layer with 64 hidden units.
 # in the first layer, you must specify the expected input data shape:
@@ -177,6 +184,16 @@ score = model.evaluate(x_test, y_test, batch_size=128)
 ### MLP for binary classification:
 
 ```python
+import numpy as np
+from keras.models import Sequential
+from keras.layers import Dense, Dropout
+
+# Generate dummy data
+x_train = np.random.random((1000, 20))
+y_train = np.random.randint(2, size=(1000, 1))
+x_test = np.random.random((100, 20))
+y_test = np.random.randint(2, size=(100, 1))
+
 model = Sequential()
 model.add(Dense(64, input_dim=20, activation='relu'))
 model.add(Dropout(0.5))
@@ -187,17 +204,30 @@ model.add(Dense(1, activation='sigmoid'))
 model.compile(loss='binary_crossentropy',
               optimizer='rmsprop',
               metrics=['accuracy'])
+
+model.fit(x_train, y_train,
+          epochs=20,
+          batch_size=128)
+score = model.evaluate(x_test, y_test, batch_size=128)
 ```
 
 
 ### VGG-like convnet:
 
 ```python
+import numpy as np
+import keras
 from keras.models import Sequential
 from keras.layers import Dense, Dropout, Flatten
 from keras.layers import Conv2D, MaxPooling2D
 from keras.optimizers import SGD
 
+# Generate dummy data
+x_train = np.random.random((100, 100, 100, 3))
+y_train = keras.utils.to_categorical(np.random.randint(10, size=(100, 1)), num_classes=10)
+x_test = np.random.random((20, 100, 100, 3))
+y_test = keras.utils.to_categorical(np.random.randint(10, size=(20, 1)), num_classes=10)
+
 model = Sequential()
 # input: 100x100 images with 3 channels -> (100, 100, 3) tensors.
 # this applies 32 convolution filters of size 3x3 each.
@@ -220,6 +250,7 @@ sgd = SGD(lr=0.01, decay=1e-6, momentum=0.9, nesterov=True)
 model.compile(loss='categorical_crossentropy', optimizer=sgd)
 
 model.fit(x_train, y_train, batch_size=32, epochs=10)
+score = model.evaluate(x_test, y_test, batch_size=32)
 ```
 
 
@@ -323,7 +354,7 @@ A stateful recurrent model is one for which the internal states (memories) obtai
 of samples are reused as initial states for the samples of the next batch. This allows to process longer sequences
 while keeping computational complexity manageable.
 
-[You can read more about stateful RNNs in the FAQ.](/faq/#how-can-i-use-stateful-rnns)
+[You can read more about stateful RNNs in the FAQ.](/getting-started/faq/#how-can-i-use-stateful-rnns)
 
 ```python
 from keras.models import Sequential

docs/templates/index.md

@@ -1,3 +1,3 @@
-# Keras: Deep Learning library for Theano and TensorFlow
+# Keras: The Python Deep Learning library
 
 {{autogenerated}}

docs/templates/initializers.md

@@ -39,5 +39,5 @@ from keras import backend as K
 def my_init(shape, dtype=None):
     return K.random_normal(shape, dtype=dtype)
 
-model.add(Dense(64, init=my_init))
+model.add(Dense(64, kernel_initializer=my_init))
 ```

docs/templates/layers/writing-your-own-keras-layers.md

@@ -21,7 +21,8 @@ class MyLayer(Layer):
 
     def build(self, input_shape):
         # Create a trainable weight variable for this layer.
-        self.kernel = self.add_weight(shape=(input_shape[1], self.output_dim),
+        self.kernel = self.add_weight(name='kernel', 
+                                      shape=(input_shape[1], self.output_dim),
                                       initializer='uniform',
                                       trainable=True)
         super(MyLayer, self).build(input_shape)  # Be sure to call this somewhere!

docs/templates/preprocessing/image.md

@@ -7,6 +7,7 @@ keras.preprocessing.image.ImageDataGenerator(featurewise_center=False,
     featurewise_std_normalization=False,
     samplewise_std_normalization=False,
     zca_whitening=False,
+    zca_epsilon=1e-6,
     rotation_range=0.,
     width_shift_range=0.,
     height_shift_range=0.,
@@ -18,6 +19,7 @@ keras.preprocessing.image.ImageDataGenerator(featurewise_center=False,
     horizontal_flip=False,
     vertical_flip=False,
     rescale=None,
+    preprocessing_function=None,
     data_format=K.image_data_format())
 ```
 
@@ -28,6 +30,7 @@ Generate batches of tensor image data with real-time data augmentation. The data
     - __samplewise_center__: Boolean. Set each sample mean to 0.
     - __featurewise_std_normalization__: Boolean. Divide inputs by std of the dataset, feature-wise.
     - __samplewise_std_normalization__: Boolean. Divide each input by its std.
+    - __zca_epsilon__: epsilon for ZCA whitening. Default is 1e-6.
     - __zca_whitening__: Boolean. Apply ZCA whitening.
     - __rotation_range__: Int. Degree range for random rotations.
     - __width_shift_range__: Float (fraction of total width). Range for random horizontal shifts.
@@ -42,6 +45,11 @@ Generate batches of tensor image data with real-time data augmentation. The data
     - __rescale__: rescaling factor. Defaults to None. If None or 0, no rescaling is applied,
             otherwise we multiply the data by the value provided (before applying
             any other transformation).
+    - __preprocessing_function__: function that will be implied on each input.
+            The function will run before any other modification on it.
+            The function should take one argument:
+            one image (Numpy tensor with rank 3),
+            and should output a Numpy tensor with the same shape.
     - _data_format_: One of {"channels_first", "channels_last"}.
         "channels_last" mode means that the images should have shape `(samples, height, width, channels)`,
         "channels_first" mode means that the images should have shape `(samples, channels, height, width)`.
@@ -50,19 +58,19 @@ Generate batches of tensor image data with real-time data augmentation. The data
         If you never set it, then it will be "channels_last".
 
 - __Methods__:
-    - __fit(X)__: Compute the internal data stats related to the data-dependent transformations, based on an array of sample data.
+    - __fit(x)__: Compute the internal data stats related to the data-dependent transformations, based on an array of sample data.
         Only required if featurewise_center or featurewise_std_normalization or zca_whitening.
         - __Arguments__:
-            - __X__: sample data. Should have rank 4.
+            - __x__: sample data. Should have rank 4.
                 In case of grayscale data,
                 the channels axis should have value 1, and in case
                 of RGB data, it should have value 3.
             - __augment__: Boolean (default: False). Whether to fit on randomly augmented samples.
             - __rounds__: int (default: 1). If augment, how many augmentation passes over the data to use.
             - __seed__: int (default: None). Random seed.
-    - __flow(X, y)__: Takes numpy data & label arrays, and generates batches of augmented/normalized data. Yields batches indefinitely, in an infinite loop.
+    - __flow(x, y)__: Takes numpy data & label arrays, and generates batches of augmented/normalized data. Yields batches indefinitely, in an infinite loop.
         - __Arguments__:
-            - __X__: data. Should have rank 4.
+            - __x__: data. Should have rank 4.
                 In case of grayscale data,
                 the channels axis should have value 1, and in case
                 of RGB data, it should have value 3.
@@ -72,7 +80,7 @@ Generate batches of tensor image data with real-time data augmentation. The data
             - __seed__: int (default: None).
             - __save_to_dir__: None or str (default: None). This allows you to optimally specify a directory to which to save the augmented pictures being generated (useful for visualizing what you are doing).
             - __save_prefix__: str (default: `''`). Prefix to use for filenames of saved pictures (only relevant if `save_to_dir` is set).
-            - __save_format__: one of "png", "jpeg" (only relevant if `save_to_dir` is set). Default: "jpeg".
+            - __save_format__: one of "png", "jpeg" (only relevant if `save_to_dir` is set). Default: "png".
         - __yields__: Tuples of `(x, y)` where `x` is a numpy array of image data and `y` is a numpy array of corresponding labels.
             The generator loops indefinitely.
     - __flow_from_directory(directory)__: Takes the path to a directory, and generates batches of augmented/normalized data. Yields batches indefinitely, in an infinite loop.
@@ -82,25 +90,25 @@ Generate batches of tensor image data with real-time data augmentation. The data
                 See [this script](https://gist.github.com/fchollet/0830affa1f7f19fd47b06d4cf89ed44d) for more details.
             - __target_size__: tuple of integers, default: `(256, 256)`. The dimensions to which all images found will be resized.
             - __color_mode__: one of "grayscale", "rbg". Default: "rgb". Whether the images will be converted to have 1 or 3 color channels.
-            - __classes__: optional list of class subdirectories (e.g. `['dogs', 'cats']`). Default: None. If not provided, the list of classes will be automatically inferred (and the order of the classes, which will map to the label indices, will be alphanumeric).
-            - __class_mode__: one of "categorical", "binary", "sparse" or None. Default: "categorical". Determines the type of label arrays that are returned: "categorical" will be 2D one-hot encoded labels, "binary" will be 1D binary labels, "sparse" will be 1D integer labels. If None, no labels are returned (the generator will only yield batches of image data, which is useful to use `model.predict_generator()`, `model.evaluate_generator()`, etc.).
+            - __classes__: optional list of class subdirectories (e.g. `['dogs', 'cats']`). Default: None. If not provided, the list of classes will be automatically inferred from the subdirectory names/structure under `directory`, where each subdirectory will be treated as a different class (and the order of the classes, which will map to the label indices, will be alphanumeric). The dictionary containing the mapping from class names to class indices can be obtained via the attribute `class_indices`.
+            - __class_mode__: one of "categorical", "binary", "sparse" or None. Default: "categorical". Determines the type of label arrays that are returned: "categorical" will be 2D one-hot encoded labels, "binary" will be 1D binary labels, "sparse" will be 1D integer labels. If None, no labels are returned (the generator will only yield batches of image data, which is useful to use `model.predict_generator()`, `model.evaluate_generator()`, etc.). Please note that in case of class_mode None, the data still needs to reside in a subdirectory of `directory` for it to work correctly.
             - __batch_size__: size of the batches of data (default: 32).
             - __shuffle__: whether to shuffle the data (default: True)
             - __seed__: optional random seed for shuffling and transformations.
             - __save_to_dir__: None or str (default: None). This allows you to optimally specify a directory to which to save the augmented pictures being generated (useful for visualizing what you are doing).
             - __save_prefix__: str. Prefix to use for filenames of saved pictures (only relevant if `save_to_dir` is set).
-            - __save_format__: one of "png", "jpeg" (only relevant if `save_to_dir` is set). Default: "jpeg".
+            - __save_format__: one of "png", "jpeg" (only relevant if `save_to_dir` is set). Default: "png".
             - __follow_links__: whether to follow symlinks inside class subdirectories (default: False).
 
 
 - __Examples__:
 
-Example of using `.flow(X, y)`:
+Example of using `.flow(x, y)`:
 
 ```python
-(X_train, y_train), (X_test, y_test) = cifar10.load_data()
-Y_train = np_utils.to_categorical(y_train, num_classes)
-Y_test = np_utils.to_categorical(y_test, num_classes)
+(x_train, y_train), (x_test, y_test) = cifar10.load_data()
+y_train = np_utils.to_categorical(y_train, num_classes)
+y_test = np_utils.to_categorical(y_test, num_classes)
 
 datagen = ImageDataGenerator(
     featurewise_center=True,
@@ -112,20 +120,20 @@ datagen = ImageDataGenerator(
 
 # compute quantities required for featurewise normalization
 # (std, mean, and principal components if ZCA whitening is applied)
-datagen.fit(X_train)
+datagen.fit(x_train)
 
 # fits the model on batches with real-time data augmentation:
-model.fit_generator(datagen.flow(X_train, Y_train, batch_size=32),
-                    steps_per_epoch=len(X_train), epochs=epochs)
+model.fit_generator(datagen.flow(x_train, y_train, batch_size=32),
+                    steps_per_epoch=len(x_train) / 32, epochs=epochs)
 
 # here's a more "manual" example
 for e in range(epochs):
-    print 'Epoch', e
+    print('Epoch', e)
     batches = 0
-    for X_batch, Y_batch in datagen.flow(X_train, Y_train, batch_size=32):
-        loss = model.train(X_batch, Y_batch)
+    for x_batch, y_batch in datagen.flow(x_train, y_train, batch_size=32):
+        model.fit(x_batch, y_batch)
         batches += 1
-        if batches >= len(X_train) / 32:
+        if batches >= len(x_train) / 32:
             # we need to break the loop by hand because
             # the generator loops indefinitely
             break

docs/templates/preprocessing/text.md

@@ -2,8 +2,10 @@
 ## text_to_word_sequence
 
 ```python
-keras.preprocessing.text.text_to_word_sequence(text, 
-    filters=base_filter(), lower=True, split=" ")
+keras.preprocessing.text.text_to_word_sequence(text,
+                                               filters='!"#$%&()*+,-./:;<=>?@[\\]^_`{|}~\t\n',
+                                               lower=True,
+                                               split=" ")
 ```
 
 Split a sentence into a list of words.
@@ -12,35 +14,81 @@ Split a sentence into a list of words.
 
 - __Arguments__:
     - __text__: str.
-    - __filters__: list (or concatenation) of characters to filter out, such as punctuation. Default: base_filter(), includes basic punctuation, tabs, and newlines.
+    - __filters__: list (or concatenation) of characters to filter out, such as
+         punctuation. Default: '!"#$%&()*+,-./:;<=>?@[\\]^_`{|}~\t\n' , includes
+         basic punctuation, tabs, and newlines.
     - __lower__: boolean. Whether to set the text to lowercase.
     - __split__: str. Separator for word splitting.
 
 ## one_hot
 
 ```python
-keras.preprocessing.text.one_hot(text, n,
-    filters=base_filter(), lower=True, split=" ")
+keras.preprocessing.text.one_hot(text,
+                                 n,
+                                 filters='!"#$%&()*+,-./:;<=>?@[\\]^_`{|}~\t\n',
+                                 lower=True,
+                                 split=" ")
 ```
 
-One-hot encode a text into a list of word indexes in a vocabulary of size n.
+One-hot encodes a text into a list of word indexes in a vocabulary of size n.
+
+This is a wrapper to the `hashing_trick` function using `hash` as the hashing function.
 
 - __Return__: List of integers in [1, n]. Each integer encodes a word (unicity non-guaranteed).
 
-- __Arguments__: Same as `text_to_word_sequence` above.
+- __Arguments__:
+    - __text__: str.
     - __n__: int. Size of vocabulary.
+    - __filters__: list (or concatenation) of characters to filter out, such as
+         punctuation. Default: '!"#$%&()*+,-./:;<=>?@[\\]^_`{|}~\t\n' , includes
+         basic punctuation, tabs, and newlines.
+    - __lower__: boolean. Whether to set the text to lowercase.
+    - __split__: str. Separator for word splitting.
+    
+## hashing_trick
+
+```python
+keras.preprocessing.text.hashing_trick(text, 
+                                       n,
+                                       hash_function=None,
+                                       filters='!"#$%&()*+,-./:;<=>?@[\\]^_`{|}~\t\n',
+                                       lower=True,
+                                       split=' ')
+```
+
+Converts a text to a sequence of indices in a fixed-size hashing space
+
+- __Return__:
+        A list of integer word indices (unicity non-guaranteed).
+- __Arguments__:
+    - __text__: str.
+    - __n__: Dimension of the hashing space.
+    - __hash_function__: defaults to python `hash` function, can be 'md5' or
+            any function that takes in input a string and returns a int.
+            Note that 'hash' is not a stable hashing function, so
+            it is not consistent across different runs, while 'md5'
+            is a stable hashing function.
+    - __filters__: list (or concatenation) of characters to filter out, such as
+         punctuation. Default: '!"#$%&()*+,-./:;<=>?@[\\]^_`{|}~\t\n' , includes
+         basic punctuation, tabs, and newlines.
+    - __lower__: boolean. Whether to set the text to lowercase.
+    - __split__: str. Separator for word splitting.
 
 ## Tokenizer
 
 ```python
-keras.preprocessing.text.Tokenizer(num_words=None, filters=base_filter(), 
-    lower=True, split=" ")
+keras.preprocessing.text.Tokenizer(num_words=None,
+                                   filters='!"#$%&()*+,-./:;<=>?@[\\]^_`{|}~\t\n',
+                                   lower=True,
+                                   split=" ",
+                                   char_level=False)
 ```
 
 Class for vectorizing texts, or/and turning texts into sequences (=list of word indexes, where the word of rank i in the dataset (starting at 1) has index i).
 
 - __Arguments__: Same as `text_to_word_sequence` above.
     - __num_words__: None or int. Maximum number of words to work with (if set, tokenization will be restricted to the top num_words most common words in the dataset).
+    - __char_level__: if True, every character will be treated as a token.
 
 - __Methods__:
 

docs/templates/visualization.md

@@ -19,7 +19,7 @@ You can also directly obtain the `pydot.Graph` object and render it yourself,
 for example to show it in an ipython notebook :
 ```python
 from IPython.display import SVG
-from keras.utils.visualize_util import model_to_dot
+from keras.utils.vis_utils import model_to_dot
 
 SVG(model_to_dot(model).create(prog='dot', format='svg'))
 ```

examples/addition_rnn.py

@@ -78,7 +78,7 @@ INVERT = True
 
 # Maximum length of input is 'int + int' (e.g., '345+678'). Maximum length of
 # int is DIGITS.
-MAxLEN = DIGITS + 1 + DIGITS
+MAXLEN = DIGITS + 1 + DIGITS
 
 # All the numbers, plus sign and space for padding.
 chars = '0123456789+ '
@@ -98,9 +98,9 @@ while len(questions) < TRAINING_SIZE:
     if key in seen:
         continue
     seen.add(key)
-    # Pad the data with spaces such that it is always MAxLEN.
+    # Pad the data with spaces such that it is always MAXLEN.
     q = '{}+{}'.format(a, b)
-    query = q + ' ' * (MAxLEN - len(q))
+    query = q + ' ' * (MAXLEN - len(q))
     ans = str(a + b)
     # Answers can be of maximum size DIGITS + 1.
     ans += ' ' * (DIGITS + 1 - len(ans))
@@ -113,10 +113,10 @@ while len(questions) < TRAINING_SIZE:
 print('Total addition questions:', len(questions))
 
 print('Vectorization...')
-x = np.zeros((len(questions), MAxLEN, len(chars)), dtype=np.bool)
+x = np.zeros((len(questions), MAXLEN, len(chars)), dtype=np.bool)
 y = np.zeros((len(questions), DIGITS + 1, len(chars)), dtype=np.bool)
 for i, sentence in enumerate(questions):
-    x[i] = ctable.encode(sentence, MAxLEN)
+    x[i] = ctable.encode(sentence, MAXLEN)
 for i, sentence in enumerate(expected):
     y[i] = ctable.encode(sentence, DIGITS + 1)
 
@@ -151,7 +151,7 @@ model = Sequential()
 # "Encode" the input sequence using an RNN, producing an output of HIDDEN_SIZE.
 # Note: In a situation where your input sequences have a variable length,
 # use input_shape=(None, num_feature).
-model.add(RNN(HIDDEN_SIZE, input_shape=(MAxLEN, len(chars))))
+model.add(RNN(HIDDEN_SIZE, input_shape=(MAXLEN, len(chars))))
 # As the decoder RNN's input, repeatedly provide with the last hidden state of
 # RNN for each time step. Repeat 'DIGITS + 1' times as that's the maximum
 # length of output, e.g., when DIGITS=3, max output is 999+999=1998.
@@ -179,7 +179,9 @@ for iteration in range(1, 200):
     print()
     print('-' * 50)
     print('Iteration', iteration)
-    model.fit(x_train, y_train, batch_size=BATCH_SIZE, epochs=1,
+    model.fit(x_train, y_train,
+              batch_size=BATCH_SIZE,
+              epochs=1,
               validation_data=(x_val, y_val))
     # Select 10 samples from the validation set at random so we can visualize
     # errors.

examples/antirectifier.py

@@ -98,8 +98,10 @@ model.compile(loss='categorical_crossentropy',
 
 # train the model
 model.fit(x_train, y_train,
-          batch_size=batch_size, epochs=epochs,
-          verbose=1, validation_data=(x_test, y_test))
+          batch_size=batch_size,
+          epochs=epochs,
+          verbose=1,
+          validation_data=(x_test, y_test))
 
 # next, compare with an equivalent network
 # with2x bigger Dense layers and ReLU

examples/cifar10_cnn.py

@@ -20,11 +20,6 @@ num_classes = 10
 epochs = 200
 data_augmentation = True
 
-# input image dimensions
-img_rows, img_cols = 32, 32
-# The CIFAR10 images are RGB.
-img_channels = 3
-
 # The data, shuffled and split between train and test sets:
 (x_train, y_train), (x_test, y_test) = cifar10.load_data()
 print('x_train shape:', x_train.shape)
@@ -59,9 +54,12 @@ model.add(Dropout(0.5))
 model.add(Dense(num_classes))
 model.add(Activation('softmax'))
 
+# initiate RMSprop optimizer
+opt = keras.optimizers.rmsprop(lr=0.0001, decay=1e-6)
+
 # Let's train the model using RMSprop
 model.compile(loss='categorical_crossentropy',
-              optimizer='rmsprop',
+              optimizer=opt,
               metrics=['accuracy'])
 
 x_train = x_train.astype('float32')
@@ -91,7 +89,7 @@ else:
         horizontal_flip=True,  # randomly flip images
         vertical_flip=False)  # randomly flip images
 
-    # Compute quantities required for featurewise normalization
+    # Compute quantities required for feature-wise normalization
     # (std, mean, and principal components if ZCA whitening is applied).
     datagen.fit(x_train)
 

examples/deep_dream.py

@@ -8,24 +8,16 @@ e.g.:
 ```
 python deep_dream.py img/mypic.jpg results/dream
 ```
-
-It is preferable to run this script on GPU, for speed.
-If running on CPU, prefer the TensorFlow backend (much faster).
-
-Example results: http://i.imgur.com/FX6ROg9.jpg
 '''
 from __future__ import print_function
 
 from keras.preprocessing.image import load_img, img_to_array
 import numpy as np
-from scipy.misc import imsave
-from scipy.optimize import fmin_l_bfgs_b
-import time
+import scipy
 import argparse
 
-from keras.applications import vgg16
+from keras.applications import inception_v3
 from keras import backend as K
-from keras.layers import Input
 
 parser = argparse.ArgumentParser(description='Deep Dreams with Keras.')
 parser.add_argument('base_image_path', metavar='base', type=str,
@@ -37,183 +29,167 @@ args = parser.parse_args()
 base_image_path = args.base_image_path
 result_prefix = args.result_prefix
 
-# dimensions of the generated picture.
-img_height = 600
-img_width = 600
-
-# some settings we found interesting
-saved_settings = {
-    'bad_trip': {'features': {'block4_conv1': 0.05,
-                              'block4_conv2': 0.01,
-                              'block4_conv3': 0.01},
-                 'continuity': 0.1,
-                 'dream_l2': 0.8,
-                 'jitter': 5},
-    'dreamy': {'features': {'block5_conv1': 0.05,
-                            'block5_conv2': 0.02},
-               'continuity': 0.1,
-               'dream_l2': 0.02,
-               'jitter': 0},
+# These are the names of the layers
+# for which we try to maximize activation,
+# as well as their weight in the final loss
+# we try to maximize.
+# You can tweak these setting to obtain new visual effects.
+settings = {
+    'features': {
+        'mixed2': 0.2,
+        'mixed3': 0.5,
+        'mixed4': 2.,
+        'mixed5': 1.5,
+    },
 }
-# the settings we will use in this experiment
-settings = saved_settings['dreamy']
 
 
 def preprocess_image(image_path):
-    # util function to open, resize and format pictures
-    # into appropriate tensors
-    img = load_img(image_path, target_size=(img_height, img_width))
+    # Util function to open, resize and format pictures
+    # into appropriate tensors.
+    img = load_img(image_path)
     img = img_to_array(img)
     img = np.expand_dims(img, axis=0)
-    img = vgg16.preprocess_input(img)
+    img = inception_v3.preprocess_input(img)
     return img
 
 
 def deprocess_image(x):
-    # util function to convert a tensor into a valid image
+    # Util function to convert a tensor into a valid image.
     if K.image_data_format() == 'channels_first':
-        x = x.reshape((3, img_height, img_width))
+        x = x.reshape((3, x.shape[2], x.shape[3]))
         x = x.transpose((1, 2, 0))
     else:
-        x = x.reshape((img_height, img_width, 3))
-    # Remove zero-center by mean pixel
-    x[:, :, 0] += 103.939
-    x[:, :, 1] += 116.779
-    x[:, :, 2] += 123.68
-    # 'BGR'->'RGB'
-    x = x[:, :, ::-1]
+        x = x.reshape((x.shape[1], x.shape[2], 3))
+    x /= 2.
+    x += 0.5
+    x *= 255.
     x = np.clip(x, 0, 255).astype('uint8')
     return x
 
-if K.image_data_format() == 'channels_first':
-    img_size = (3, img_height, img_width)
-else:
-    img_size = (img_height, img_width, 3)
-# this will contain our generated image
-dream = Input(batch_shape=(1,) + img_size)
+K.set_learning_phase(0)
 
-# build the VGG16 network with our placeholder
-# the model will be loaded with pre-trained ImageNet weights
-model = vgg16.VGG16(input_tensor=dream,
-                    weights='imagenet', include_top=False)
+# Build the InceptionV3 network with our placeholder.
+# The model will be loaded with pre-trained ImageNet weights.
+model = inception_v3.InceptionV3(weights='imagenet',
+                                 include_top=False)
+dream = model.input
 print('Model loaded.')
 
-# get the symbolic outputs of each "key" layer (we gave them unique names).
+# Get the symbolic outputs of each "key" layer (we gave them unique names).
 layer_dict = dict([(layer.name, layer) for layer in model.layers])
 
-
-def continuity_loss(x):
-    # continuity loss util function
-    assert K.ndim(x) == 4
-    if K.image_data_format() == 'channels_first':
-        a = K.square(x[:, :, :img_height - 1, :img_width - 1] -
-                     x[:, :, 1:, :img_width - 1])
-        b = K.square(x[:, :, :img_height - 1, :img_width - 1] -
-                     x[:, :, :img_height - 1, 1:])
-    else:
-        a = K.square(x[:, :img_height - 1, :img_width - 1, :] -
-                     x[:, 1:, :img_width - 1, :])
-        b = K.square(x[:, :img_height - 1, :img_width - 1, :] -
-                     x[:, :img_height - 1, 1:, :])
-    return K.sum(K.pow(a + b, 1.25))
-
-# define the loss
+# Define the loss.
 loss = K.variable(0.)
 for layer_name in settings['features']:
-    # add the L2 norm of the features of a layer to the loss
+    # Add the L2 norm of the features of a layer to the loss.
     assert layer_name in layer_dict.keys(), 'Layer ' + layer_name + ' not found in model.'
     coeff = settings['features'][layer_name]
     x = layer_dict[layer_name].output
-    shape = layer_dict[layer_name].output_shape
-    # we avoid border artifacts by only involving non-border pixels in the loss
+    # We avoid border artifacts by only involving non-border pixels in the loss.
+    scaling = K.prod(K.cast(K.shape(x), 'float32'))
     if K.image_data_format() == 'channels_first':
-        loss -= coeff * K.sum(K.square(x[:, :, 2: shape[2] - 2, 2: shape[3] - 2])) / np.prod(shape[1:])
+        loss += coeff * K.sum(K.square(x[:, :, 2: -2, 2: -2])) / scaling
     else:
-        loss -= coeff * K.sum(K.square(x[:, 2: shape[1] - 2, 2: shape[2] - 2, :])) / np.prod(shape[1:])
+        loss += coeff * K.sum(K.square(x[:, 2: -2, 2: -2, :])) / scaling
 
-# add continuity loss (gives image local coherence, can result in an artful blur)
-loss += settings['continuity'] * continuity_loss(dream) / np.prod(img_size)
-# add image L2 norm to loss (prevents pixels from taking very high values, makes image darker)
-loss += settings['dream_l2'] * K.sum(K.square(dream)) / np.prod(img_size)
+# Compute the gradients of the dream wrt the loss.
+grads = K.gradients(loss, dream)[0]
+# Normalize gradients.
+grads /= K.maximum(K.mean(K.abs(grads)), 1e-7)
 
-# feel free to further modify the loss as you see fit, to achieve new effects...
-
-# compute the gradients of the dream wrt the loss
-grads = K.gradients(loss, dream)
-
-outputs = [loss]
-if isinstance(grads, (list, tuple)):
-    outputs += grads
-else:
-    outputs.append(grads)
-
-f_outputs = K.function([dream], outputs)
+# Set up function to retrieve the value
+# of the loss and gradients given an input image.
+outputs = [loss, grads]
+fetch_loss_and_grads = K.function([dream], outputs)
 
 
 def eval_loss_and_grads(x):
-    x = x.reshape((1,) + img_size)
-    outs = f_outputs([x])
+    outs = fetch_loss_and_grads([x])
     loss_value = outs[0]
-    if len(outs[1:]) == 1:
-        grad_values = outs[1].flatten().astype('float64')
-    else:
-        grad_values = np.array(outs[1:]).flatten().astype('float64')
+    grad_values = outs[1]
     return loss_value, grad_values
 
 
-class Evaluator(object):
-    """Loss and gradients evaluator.
+def resize_img(img, size):
+    img = np.copy(img)
+    if K.image_data_format() == 'channels_first':
+        factors = (1, 1,
+                   float(size[0]) / img.shape[2],
+                   float(size[1]) / img.shape[3])
+    else:
+        factors = (1,
+                   float(size[0]) / img.shape[1],
+                   float(size[1]) / img.shape[2],
+                   1)
+    return scipy.ndimage.zoom(img, factors, order=1)
 
-    This Evaluator class makes it possible
-    to compute loss and gradients in one pass
-    while retrieving them via two separate functions,
-    "loss" and "grads". This is done because scipy.optimize
-    requires separate functions for loss and gradients,
-    but computing them separately would be inefficient.
-    """
 
-    def __init__(self):
-        self.loss_value = None
-        self.grad_values = None
-
-    def loss(self, x):
-        assert self.loss_value is None
+def gradient_ascent(x, iterations, step, max_loss=None):
+    for i in range(iterations):
         loss_value, grad_values = eval_loss_and_grads(x)
-        self.loss_value = loss_value
-        self.grad_values = grad_values
-        return self.loss_value
+        if max_loss is not None and loss_value > max_loss:
+            break
+        print('..Loss value at', i, ':', loss_value)
+        x += step * grad_values
+    return x
 
-    def grads(self, x):
-        assert self.loss_value is not None
-        grad_values = np.copy(self.grad_values)
-        self.loss_value = None
-        self.grad_values = None
-        return grad_values
 
-evaluator = Evaluator()
+def save_img(img, fname):
+    pil_img = deprocess_image(np.copy(img))
+    scipy.misc.imsave(fname, pil_img)
 
-# Run scipy-based optimization (L-BFGS) over the pixels of the generated image
-# so as to minimize the loss
-x = preprocess_image(base_image_path)
-for i in range(5):
-    print('Start of iteration', i)
-    start_time = time.time()
 
-    # Add a random jitter to the initial image.
-    # This will be reverted at decoding time
-    random_jitter = (settings['jitter'] * 2) * (np.random.random(img_size) - 0.5)
-    x += random_jitter
+"""Process:
 
-    # Run L-BFGS for 7 steps
-    x, min_val, info = fmin_l_bfgs_b(evaluator.loss, x.flatten(),
-                                     fprime=evaluator.grads, maxfun=7)
-    print('Current loss value:', min_val)
-    # Decode the dream and save it
-    x = x.reshape(img_size)
-    x -= random_jitter
-    img = deprocess_image(np.copy(x))
-    fname = result_prefix + '_at_iteration_%d.png' % i
-    imsave(fname, img)
-    end_time = time.time()
-    print('Image saved as', fname)
-    print('Iteration %d completed in %ds' % (i, end_time - start_time))
+- Load the original image.
+- Define a number of processing scales (i.e. image shapes),
+    from smallest to largest.
+- Resize the original image to the smallest scale.
+- For every scale, starting with the smallest (i.e. current one):
+    - Run gradient ascent
+    - Upscale image to the next scale
+    - Reinject the detail that was lost at upscaling time
+- Stop when we are back to the original size.
+
+To obtain the detail lost during upscaling, we simply
+take the original image, shrink it down, upscale it,
+and compare the result to the (resized) original image.
+"""
+
+
+# Playing with these hyperparameters will also allow you to achieve new effects
+step = 0.01  # Gradient ascent step size
+num_octave = 3  # Number of scales at which to run gradient ascent
+octave_scale = 1.4  # Size ratio between scales
+iterations = 20  # Number of ascent steps per scale
+max_loss = 10.
+
+img = preprocess_image(base_image_path)
+if K.image_data_format() == 'channels_first':
+    original_shape = img.shape[2:]
+else:
+    original_shape = img.shape[1:3]
+successive_shapes = [original_shape]
+for i in range(1, num_octave):
+    shape = tuple([int(dim / (octave_scale ** i)) for dim in original_shape])
+    successive_shapes.append(shape)
+successive_shapes = successive_shapes[::-1]
+original_img = np.copy(img)
+shrunk_original_img = resize_img(img, successive_shapes[0])
+
+for shape in successive_shapes:
+    print('Processing image shape', shape)
+    img = resize_img(img, shape)
+    img = gradient_ascent(img,
+                          iterations=iterations,
+                          step=step,
+                          max_loss=max_loss)
+    upscaled_shrunk_original_img = resize_img(shrunk_original_img, shape)
+    same_size_original = resize_img(original_img, shape)
+    lost_detail = same_size_original - upscaled_shrunk_original_img
+
+    img += lost_detail
+    shrunk_original_img = resize_img(original_img, shape)
+
+save_img(img, fname=result_prefix + '.png')

examples/image_ocr.py

@@ -127,17 +127,17 @@ def shuffle_mats_or_lists(matrix_list, stop_ind=None):
         stop_ind = len_val
     assert stop_ind <= len_val
 
-    a = range(stop_ind)
+    a = list(range(stop_ind))
     np.random.shuffle(a)
-    a += range(stop_ind, len_val)
+    a += list(range(stop_ind, len_val))
     for mat in matrix_list:
         if isinstance(mat, np.ndarray):
             ret.append(mat[a])
         elif isinstance(mat, list):
             ret.append([mat[i] for i in a])
         else:
-            raise TypeError('shuffle_mats_or_lists only supports '
-                            'numpy.array and list objects')
+            raise TypeError('`shuffle_mats_or_lists` only supports '
+                            'numpy.array and list objects.')
     return ret
 
 
@@ -416,7 +416,7 @@ def train(run_name, start_epoch, stop_epoch, img_w):
         input_shape = (img_w, img_h, 1)
 
     fdir = os.path.dirname(get_file('wordlists.tgz',
-                                    origin='http://www.isosemi.com/datasets/wordlists.tgz', untar=True))
+                                    origin='http://www.mythic-ai.com/datasets/wordlists.tgz', untar=True))
 
     img_gen = TextImageGenerator(monogram_file=os.path.join(fdir, 'wordlist_mono_clean.txt'),
                                  bigram_file=os.path.join(fdir, 'wordlist_bi_clean.txt'),

examples/imdb_bidirectional_lstm.py

@@ -24,7 +24,7 @@ print('Loading data...')
 print(len(x_train), 'train sequences')
 print(len(x_test), 'test sequences')
 
-print("Pad sequences (samples x time)")
+print('Pad sequences (samples x time)')
 x_train = sequence.pad_sequences(x_train, maxlen=maxlen)
 x_test = sequence.pad_sequences(x_test, maxlen=maxlen)
 print('x_train shape:', x_train.shape)

examples/imdb_cnn_lstm.py

@@ -67,7 +67,9 @@ model.compile(loss='binary_crossentropy',
               metrics=['accuracy'])
 
 print('Train...')
-model.fit(x_train, y_train, batch_size=batch_size, epochs=epochs,
+model.fit(x_train, y_train,
+          batch_size=batch_size,
+          epochs=epochs,
           validation_data=(x_test, y_test))
 score, acc = model.evaluate(x_test, y_test, batch_size=batch_size)
 print('Test score:', score)

examples/imdb_lstm.py

@@ -45,7 +45,9 @@ model.compile(loss='binary_crossentropy',
               metrics=['accuracy'])
 
 print('Train...')
-model.fit(x_train, y_train, batch_size=batch_size, epochs=15,
+model.fit(x_train, y_train,
+          batch_size=batch_size,
+          epochs=15,
           validation_data=(x_test, y_test))
 score, acc = model.evaluate(x_test, y_test,
                             batch_size=batch_size)

examples/lstm_benchmark.py

@@ -12,8 +12,8 @@ into one, large matrix, resulting in faster computation time as the GPU can
 utilize more cores, at the expense of reduced regularization because the same
 dropout is shared across the gates.
 
-Note that the relative performance of the different `consume_less` modes
-can vary depending on your device, your model and the size of your data.
+Note that the relative performance of the different implementations can
+vary depending on your device, your model and the size of your data.
 '''
 
 import time
@@ -37,7 +37,7 @@ print('Loading data...')
 X_train = sequence.pad_sequences(X_train, max_length)
 X_test = sequence.pad_sequences(X_test, max_length)
 
-# Compile and train different models while meauring performance.
+# Compile and train different models while measuring performance.
 results = []
 for mode in modes:
     print('Testing mode: implementation={}'.format(mode))

examples/lstm_text_generation.py

@@ -73,7 +73,9 @@ for iteration in range(1, 60):
     print()
     print('-' * 50)
     print('Iteration', iteration)
-    model.fit(X, y, batch_size=128, epochs=1)
+    model.fit(X, y,
+              batch_size=128,
+              epochs=1)
 
     start_index = random.randint(0, len(text) - maxlen - 1)
 

examples/mnist_acgan.py

@@ -101,7 +101,7 @@ def build_discriminator():
     cnn.add(LeakyReLU())
     cnn.add(Dropout(0.3))
 
-    cnn.add(Conv2D(64, 3, padding='same', strides=2))
+    cnn.add(Conv2D(64, 3, padding='same', strides=1))
     cnn.add(LeakyReLU())
     cnn.add(Dropout(0.3))
 
@@ -222,7 +222,7 @@ if __name__ == '__main__':
             noise = np.random.uniform(-1, 1, (2 * batch_size, latent_size))
             sampled_labels = np.random.randint(0, 10, 2 * batch_size)
 
-            # we want to train the genrator to trick the discriminator
+            # we want to train the generator to trick the discriminator
             # For the generator, we want all the {fake, not-fake} labels to say
             # not-fake
             trick = np.ones(2 * batch_size)

examples/mnist_cnn.py

@@ -60,8 +60,11 @@ model.compile(loss=keras.losses.categorical_crossentropy,
               optimizer=keras.optimizers.Adadelta(),
               metrics=['accuracy'])
 
-model.fit(x_train, y_train, batch_size=batch_size, epochs=epochs,
-          verbose=1, validation_data=(x_test, y_test))
+model.fit(x_train, y_train,
+          batch_size=batch_size,
+          epochs=epochs,
+          verbose=1,
+          validation_data=(x_test, y_test))
 score = model.evaluate(x_test, y_test, verbose=0)
 print('Test loss:', score[0])
 print('Test accuracy:', score[1])

examples/mnist_hierarchical_rnn.py

@@ -79,8 +79,10 @@ model.compile(loss='categorical_crossentropy',
 
 # Training.
 model.fit(x_train, y_train,
-          batch_size=batch_size, epochs=epochs,
-          verbose=1, validation_data=(x_test, y_test))
+          batch_size=batch_size,
+          epochs=epochs,
+          verbose=1,
+          validation_data=(x_test, y_test))
 
 # Evaluation.
 scores = model.evaluate(x_test, y_test, verbose=0)

examples/mnist_irnn.py

@@ -62,8 +62,11 @@ model.compile(loss='categorical_crossentropy',
               optimizer=rmsprop,
               metrics=['accuracy'])
 
-model.fit(x_train, y_train, batch_size=batch_size, epochs=epochs,
-          verbose=1, validation_data=(x_test, y_test))
+model.fit(x_train, y_train,
+          batch_size=batch_size,
+          epochs=epochs,
+          verbose=1,
+          validation_data=(x_test, y_test))
 
 scores = model.evaluate(x_test, y_test, verbose=0)
 print('IRNN test score:', scores[0])

examples/mnist_mlp.py

@@ -48,8 +48,10 @@ model.compile(loss='categorical_crossentropy',
               metrics=['accuracy'])
 
 history = model.fit(x_train, y_train,
-                    batch_size=batch_size, epochs=epochs,
-                    verbose=1, validation_data=(x_test, y_test))
+                    batch_size=batch_size,
+                    epochs=epochs,
+                    verbose=1,
+                    validation_data=(x_test, y_test))
 score = model.evaluate(x_test, y_test, verbose=0)
 print('Test loss:', score[0])
 print('Test accuracy:', score[1])

examples/mnist_net2net.py

@@ -26,7 +26,7 @@ Notes
 
 Experiments
 - Teacher model: a basic CNN model trained on MNIST for 3 epochs.
-- Net2WiderNet exepriment:
+- Net2WiderNet experiment:
   + Student model has a wider Conv2D layer and a wider FC layer.
   + Comparison of 'random-padding' vs 'net2wider' weight initialization.
   + With both methods, student model should immediately perform as well as
@@ -231,7 +231,8 @@ def make_teacher_model(train_data, validation_data, epochs=3):
                   metrics=['accuracy'])
 
     train_x, train_y = train_data
-    history = model.fit(train_x, train_y, epochs=epochs,
+    history = model.fit(train_x, train_y,
+                        epochs=epochs,
                         validation_data=validation_data)
     return model, history
 
@@ -280,7 +281,8 @@ def make_wider_student_model(teacher_model, train_data,
                   metrics=['accuracy'])
 
     train_x, train_y = train_data
-    history = model.fit(train_x, train_y, epochs=epochs,
+    history = model.fit(train_x, train_y,
+                        epochs=epochs,
                         validation_data=validation_data)
     return model, history
 
@@ -328,7 +330,8 @@ def make_deeper_student_model(teacher_model, train_data,
                   metrics=['accuracy'])
 
     train_x, train_y = train_data
-    history = model.fit(train_x, train_y, epochs=epochs,
+    history = model.fit(train_x, train_y,
+                        epochs=epochs,
                         validation_data=validation_data)
     return model, history
 

examples/mnist_siamese_graph.py

@@ -24,7 +24,7 @@ from keras import backend as K
 
 def euclidean_distance(vects):
     x, y = vects
-    return K.sqrt(K.sum(K.square(x - y), axis=1, keepdims=True))
+    return K.sqrt(K.maximum(K.sum(K.square(x - y), axis=1, keepdims=True), K.epsilon()))
 
 
 def eucl_dist_output_shape(shapes):
@@ -117,9 +117,9 @@ model = Model([input_a, input_b], distance)
 rms = RMSprop()
 model.compile(loss=contrastive_loss, optimizer=rms)
 model.fit([tr_pairs[:, 0], tr_pairs[:, 1]], tr_y,
-          validation_data=([te_pairs[:, 0], te_pairs[:, 1]], te_y),
           batch_size=128,
-          epochs=epochs)
+          epochs=epochs,
+          validation_data=([te_pairs[:, 0], te_pairs[:, 1]], te_y))
 
 # compute final accuracy on training and test sets
 pred = model.predict([tr_pairs[:, 0], tr_pairs[:, 1]])

examples/mnist_swwae.py

@@ -35,12 +35,12 @@ applied as a bias because we know the MNIST digits are mapped to [0,1].
 References:
 [3]
 'Deep Residual Learning for Image Recognition'
-Kaiming He, xiangyu Zhang, Shaoqing Ren, Jian Sun
+Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun
 https://arxiv.org/abs/1512.03385v1
 
 [4]
 'Identity Mappings in Deep Residual Networks'
-Kaiming He, xiangyu Zhang, Shaoqing Ren, Jian Sun
+Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun
 https://arxiv.org/abs/1603.05027v3
 
 '''
@@ -186,8 +186,10 @@ model = Model(img_input, y)
 model.compile('adam', 'mse')
 
 # Fit the model
-model.fit(x_train, x_train, validation_data=(x_test, x_test),
-          batch_size=batch_size, epochs=epochs)
+model.fit(x_train, x_train,
+          batch_size=batch_size,
+          epochs=epochs,
+          validation_data=(x_test, x_test))
 
 # Plot
 x_recon = model.predict(x_test[:25])

examples/mnist_transfer_cnn.py

@@ -63,7 +63,8 @@ def train_model(model, train, test, num_classes):
 
     t = now()
     model.fit(x_train, y_train,
-              batch_size=batch_size, epochs=epochs,
+              batch_size=batch_size,
+              epochs=epochs,
               verbose=1,
               validation_data=(x_test, y_test))
     print('Training time: %s' % (now() - t))

examples/neural_doodle.py

@@ -196,8 +196,8 @@ x = mask_input
 for layer in image_model.layers[1:]:
     name = 'mask_%s' % layer.name
     if 'conv' in layer.name:
-        x = AveragePooling2D((3, 3), strides=(
-            1, 1), name=name, border_mode='same')(x)
+        x = AveragePooling2D((3, 3), padding='same', strides=(
+            1, 1), name=name)(x)
     elif 'pool' in layer.name:
         x = AveragePooling2D((2, 2), name=name)(x)
 mask_model = Model(mask_input, x)
@@ -238,6 +238,7 @@ def region_style_loss(style_image, target_image, style_mask, target_mask):
         masked_target = K.permute_dimensions(
             target_image, (2, 0, 1)) * target_mask
         num_channels = K.shape(style_image)[-1]
+    num_channels = K.cast(num_channels, dtype='float32')
     s = gram_matrix(masked_style) / K.mean(style_mask) / num_channels
     c = gram_matrix(masked_target) / K.mean(target_mask) / num_channels
     return K.mean(K.square(s - c))

examples/neural_style_transfer.py

@@ -57,7 +57,7 @@ from scipy.optimize import fmin_l_bfgs_b
 import time
 import argparse
 
-from keras.applications import vgg16
+from keras.applications import vgg19
 from keras import backend as K
 
 parser = argparse.ArgumentParser(description='Neural style transfer with Keras.')
@@ -99,7 +99,7 @@ def preprocess_image(image_path):
     img = load_img(image_path, target_size=(img_nrows, img_ncols))
     img = img_to_array(img)
     img = np.expand_dims(img, axis=0)
-    img = vgg16.preprocess_input(img)
+    img = vgg19.preprocess_input(img)
     return img
 
 # util function to convert a tensor into a valid image
@@ -137,7 +137,7 @@ input_tensor = K.concatenate([base_image,
 
 # build the VGG16 network with our 3 images as input
 # the model will be loaded with pre-trained ImageNet weights
-model = vgg16.VGG16(input_tensor=input_tensor,
+model = vgg19.VGG19(input_tensor=input_tensor,
                     weights='imagenet', include_top=False)
 print('Model loaded.')
 
@@ -199,7 +199,7 @@ def total_variation_loss(x):
 
 # combine these loss functions into a single scalar
 loss = K.variable(0.)
-layer_features = outputs_dict['block4_conv2']
+layer_features = outputs_dict['block5_conv2']
 base_image_features = layer_features[0, :, :, :]
 combination_features = layer_features[2, :, :, :]
 loss += content_weight * content_loss(base_image_features,
@@ -273,10 +273,7 @@ evaluator = Evaluator()
 
 # run scipy-based optimization (L-BFGS) over the pixels of the generated image
 # so as to minimize the neural style loss
-if K.image_data_format() == 'channels_first':
-    x = np.random.uniform(0, 255, (1, 3, img_nrows, img_ncols)) - 128.
-else:
-    x = np.random.uniform(0, 255, (1, img_nrows, img_ncols, 3)) - 128.
+x = preprocess_image(base_image_path)
 
 for i in range(iterations):
     print('Start of iteration', i)

examples/pretrained_word_embeddings.py

@@ -143,6 +143,7 @@ model.compile(loss='categorical_crossentropy',
               optimizer='rmsprop',
               metrics=['acc'])
 
-# happy learning!
-model.fit(x_train, y_train, validation_data=(x_val, y_val),
-          epochs=10, batch_size=128)
+model.fit(x_train, y_train,
+          batch_size=128,
+          epochs=10,
+          validation_data=(x_val, y_val))

examples/reuters_mlp.py

@@ -50,8 +50,10 @@ model.compile(loss='categorical_crossentropy',
               metrics=['accuracy'])
 
 history = model.fit(x_train, y_train,
-                    epochs=epochs, batch_size=batch_size,
-                    verbose=1, validation_split=0.1)
+                    batch_size=batch_size,
+                    epochs=epochs,
+                    verbose=1,
+                    validation_split=0.1)
 score = model.evaluate(x_test, y_test,
                        batch_size=batch_size, verbose=1)
 print('Test score:', score[0])

examples/reuters_mlp_relu_vs_selu.py

@@ -0,0 +1,174 @@
+'''Compares self-normalizing MLPs with regular MLPs.
+
+Compares the performance of a simple MLP using two
+different activation functions: RELU and SELU
+on the Reuters newswire topic classification task.
+
+# Reference:
+  Klambauer, G., Unterthiner, T., Mayr, A., & Hochreiter, S. (2017).
+  Self-Normalizing Neural Networks. arXiv preprint arXiv:1706.02515.
+  https://arxiv.org/abs/1706.02515
+'''
+from __future__ import print_function
+
+import numpy as np
+import matplotlib.pyplot as plt
+import keras
+from keras.datasets import reuters
+from keras.models import Sequential
+from keras.layers import Dense, Activation, Dropout
+from keras.layers.noise import AlphaDropout
+from keras.preprocessing.text import Tokenizer
+
+max_words = 1000
+batch_size = 16
+epochs = 40
+plot = True
+
+
+def create_network(n_dense=6,
+                   dense_units=16,
+                   activation='selu',
+                   dropout=AlphaDropout,
+                   dropout_rate=0.1,
+                   kernel_initializer='lecun_normal',
+                   optimizer='adam',
+                   num_classes=1,
+                   max_words=max_words):
+    """Generic function to create a fully-connected neural network.
+
+    # Arguments
+        n_dense: int > 0. Number of dense layers.
+        dense_units: int > 0. Number of dense units per layer.
+        dropout: keras.layers.Layer. A dropout layer to apply.
+        dropout_rate: 0 <= float <= 1. The rate of dropout.
+        kernel_initializer: str. The initializer for the weights.
+        optimizer: str/keras.optimizers.Optimizer. The optimizer to use.
+        num_classes: int > 0. The number of classes to predict.
+        max_words: int > 0. The maximum number of words per data point.
+
+    # Returns
+        A Keras model instance (compiled).
+    """
+    model = Sequential()
+    model.add(Dense(dense_units, input_shape=(max_words,),
+                    kernel_initializer=kernel_initializer))
+    model.add(Activation(activation))
+    model.add(dropout(dropout_rate))
+
+    for i in range(n_dense - 1):
+        model.add(Dense(dense_units, kernel_initializer=kernel_initializer))
+        model.add(Activation(activation))
+        model.add(dropout(dropout_rate))
+
+    model.add(Dense(num_classes))
+    model.add(Activation('softmax'))
+    model.compile(loss='categorical_crossentropy',
+                  optimizer=optimizer,
+                  metrics=['accuracy'])
+    return model
+
+
+network1 = {
+    'n_dense': 6,
+    'dense_units': 16,
+    'activation': 'relu',
+    'dropout': Dropout,
+    'dropout_rate': 0.5,
+    'kernel_initializer': 'glorot_uniform',
+    'optimizer': 'sgd'
+}
+
+network2 = {
+    'n_dense': 6,
+    'dense_units': 16,
+    'activation': 'selu',
+    'dropout': AlphaDropout,
+    'dropout_rate': 0.1,
+    'kernel_initializer': 'lecun_normal',
+    'optimizer': 'sgd'
+}
+
+print('Loading data...')
+(x_train, y_train), (x_test, y_test) = reuters.load_data(num_words=max_words,
+                                                         test_split=0.2)
+print(len(x_train), 'train sequences')
+print(len(x_test), 'test sequences')
+
+num_classes = np.max(y_train) + 1
+print(num_classes, 'classes')
+
+print('Vectorizing sequence data...')
+tokenizer = Tokenizer(num_words=max_words)
+x_train = tokenizer.sequences_to_matrix(x_train, mode='binary')
+x_test = tokenizer.sequences_to_matrix(x_test, mode='binary')
+print('x_train shape:', x_train.shape)
+print('x_test shape:', x_test.shape)
+
+print('Convert class vector to binary class matrix '
+      '(for use with categorical_crossentropy)')
+y_train = keras.utils.to_categorical(y_train, num_classes)
+y_test = keras.utils.to_categorical(y_test, num_classes)
+print('y_train shape:', y_train.shape)
+print('y_test shape:', y_test.shape)
+
+print('\nBuilding network 1...')
+
+model1 = create_network(num_classes=num_classes, **network1)
+history_model1 = model1.fit(x_train,
+                            y_train,
+                            batch_size=batch_size,
+                            epochs=epochs,
+                            verbose=1,
+                            validation_split=0.1)
+
+score_model1 = model1.evaluate(x_test,
+                               y_test,
+                               batch_size=batch_size,
+                               verbose=1)
+
+
+print('\nBuilding network 2...')
+model2 = create_network(num_classes=num_classes, **network2)
+
+history_model2 = model2.fit(x_train,
+                            y_train,
+                            batch_size=batch_size,
+                            epochs=epochs,
+                            verbose=1,
+                            validation_split=0.1)
+
+score_model2 = model2.evaluate(x_test,
+                               y_test,
+                               batch_size=batch_size,
+                               verbose=1)
+
+print('\nNetwork 1 results')
+print('Hyperparameters:', network1)
+print('Test score:', score_model1[0])
+print('Test accuracy:', score_model1[1])
+print('Network 2 results')
+print('Hyperparameters:', network2)
+print('Test score:', score_model2[0])
+print('Test accuracy:', score_model2[1])
+
+plt.plot(range(epochs),
+         history_model1.history['val_loss'],
+         'g-',
+         label='Network 1 Val Loss')
+plt.plot(range(epochs),
+         history_model2.history['val_loss'],
+         'r-',
+         label='Network 2 Val Loss')
+plt.plot(range(epochs),
+         history_model1.history['loss'],
+         'g--',
+         label='Network 1 Loss')
+plt.plot(range(epochs),
+         history_model2.history['loss'],
+         'r--',
+         label='Network 2 Loss')
+plt.xlabel('Epochs')
+plt.ylabel('Loss')
+plt.legend()
+plt.savefig('comparison_of_networks.png')

examples/stateful_lstm.py

@@ -70,11 +70,10 @@ for i in range(epochs):
     # Each of these series are offset by one step and can be
     # extracted with cos[i::batch_size].
 
-    model.fit(cos,
-              expected_output,
+    model.fit(cos, expected_output,
               batch_size=batch_size,
-              verbose=1,
               epochs=1,
+              verbose=1,
               shuffle=False)
     model.reset_states()
 

examples/variational_autoencoder.py

@@ -6,7 +6,7 @@ import numpy as np
 import matplotlib.pyplot as plt
 from scipy.stats import norm
 
-from keras.layers import Input, Dense, Lambda
+from keras.layers import Input, Dense, Lambda, Layer
 from keras.models import Model
 from keras import backend as K
 from keras import metrics
@@ -19,6 +19,7 @@ intermediate_dim = 256
 epochs = 50
 epsilon_std = 1.0
 
+
 x = Input(batch_shape=(batch_size, original_dim))
 h = Dense(intermediate_dim, activation='relu')(x)
 z_mean = Dense(latent_dim)(h)
@@ -41,13 +42,29 @@ h_decoded = decoder_h(z)
 x_decoded_mean = decoder_mean(h_decoded)
 
 
-def vae_loss(x, x_decoded_mean):
-    xent_loss = original_dim * metrics.binary_crossentropy(x, x_decoded_mean)
-    kl_loss = - 0.5 * K.sum(1 + z_log_var - K.square(z_mean) - K.exp(z_log_var), axis=-1)
-    return xent_loss + kl_loss
+# Custom loss layer
+class CustomVariationalLayer(Layer):
+    def __init__(self, **kwargs):
+        self.is_placeholder = True
+        super(CustomVariationalLayer, self).__init__(**kwargs)
+
+    def vae_loss(self, x, x_decoded_mean):
+        xent_loss = original_dim * metrics.binary_crossentropy(x, x_decoded_mean)
+        kl_loss = - 0.5 * K.sum(1 + z_log_var - K.square(z_mean) - K.exp(z_log_var), axis=-1)
+        return K.mean(xent_loss + kl_loss)
+
+    def call(self, inputs):
+        x = inputs[0]
+        x_decoded_mean = inputs[1]
+        loss = self.vae_loss(x, x_decoded_mean)
+        self.add_loss(loss, inputs=inputs)
+        # We won't actually use the output.
+        return x
+
+y = CustomVariationalLayer()([x, x_decoded_mean])
+vae = Model(x, y)
+vae.compile(optimizer='rmsprop', loss=None)
 
-vae = Model(x, x_decoded_mean)
-vae.compile(optimizer='rmsprop', loss=vae_loss)
 
 # train the VAE on MNIST digits
 (x_train, y_train), (x_test, y_test) = mnist.load_data()
@@ -57,7 +74,7 @@ x_test = x_test.astype('float32') / 255.
 x_train = x_train.reshape((len(x_train), np.prod(x_train.shape[1:])))
 x_test = x_test.reshape((len(x_test), np.prod(x_test.shape[1:])))
 
-vae.fit(x_train, x_train,
+vae.fit(x_train,
         shuffle=True,
         epochs=epochs,
         batch_size=batch_size,

examples/variational_autoencoder_deconv.py

@@ -7,7 +7,7 @@ import numpy as np
 import matplotlib.pyplot as plt
 from scipy.stats import norm
 
-from keras.layers import Input, Dense, Lambda, Flatten, Reshape
+from keras.layers import Input, Dense, Lambda, Flatten, Reshape, Layer
 from keras.layers import Conv2D, Conv2DTranspose
 from keras.models import Model
 from keras import backend as K
@@ -79,7 +79,8 @@ decoder_deconv_1 = Conv2DTranspose(filters,
                                    padding='same',
                                    strides=1,
                                    activation='relu')
-decoder_deconv_2 = Conv2DTranspose(filters, num_conv,
+decoder_deconv_2 = Conv2DTranspose(filters,
+                                   kernel_size=num_conv,
                                    padding='same',
                                    strides=1,
                                    activation='relu')
@@ -106,17 +107,31 @@ x_decoded_relu = decoder_deconv_3_upsamp(deconv_2_decoded)
 x_decoded_mean_squash = decoder_mean_squash(x_decoded_relu)
 
 
-def vae_loss(x, x_decoded_mean):
-    # NOTE: binary_crossentropy expects a batch_size by dim
-    # for x and x_decoded_mean, so we MUST flatten these!
-    x = K.flatten(x)
-    x_decoded_mean = K.flatten(x_decoded_mean)
-    xent_loss = img_rows * img_cols * metrics.binary_crossentropy(x, x_decoded_mean)
-    kl_loss = - 0.5 * K.mean(1 + z_log_var - K.square(z_mean) - K.exp(z_log_var), axis=-1)
-    return xent_loss + kl_loss
+# Custom loss layer
+class CustomVariationalLayer(Layer):
+    def __init__(self, **kwargs):
+        self.is_placeholder = True
+        super(CustomVariationalLayer, self).__init__(**kwargs)
 
-vae = Model(x, x_decoded_mean_squash)
-vae.compile(optimizer='rmsprop', loss=vae_loss)
+    def vae_loss(self, x, x_decoded_mean_squash):
+        x = K.flatten(x)
+        x_decoded_mean_squash = K.flatten(x_decoded_mean_squash)
+        xent_loss = img_rows * img_cols * metrics.binary_crossentropy(x, x_decoded_mean_squash)
+        kl_loss = - 0.5 * K.mean(1 + z_log_var - K.square(z_mean) - K.exp(z_log_var), axis=-1)
+        return K.mean(xent_loss + kl_loss)
+
+    def call(self, inputs):
+        x = inputs[0]
+        x_decoded_mean_squash = inputs[1]
+        loss = self.vae_loss(x, x_decoded_mean_squash)
+        self.add_loss(loss, inputs=inputs)
+        # We don't use this output.
+        return x
+
+
+y = CustomVariationalLayer()([x, x_decoded_mean_squash])
+vae = Model(x, y)
+vae.compile(optimizer='rmsprop', loss=None)
 vae.summary()
 
 # train the VAE on MNIST digits
@@ -129,7 +144,7 @@ x_test = x_test.reshape((x_test.shape[0],) + original_img_size)
 
 print('x_train.shape:', x_train.shape)
 
-vae.fit(x_train, x_train,
+vae.fit(x_train,
         shuffle=True,
         epochs=epochs,
         batch_size=batch_size,

keras/__init__.py

@@ -1,5 +1,6 @@
 from __future__ import absolute_import
 
+from . import utils
 from . import activations
 from . import applications
 from . import backend
@@ -7,7 +8,6 @@ from . import datasets
 from . import engine
 from . import layers
 from . import preprocessing
-from . import utils
 from . import wrappers
 from . import callbacks
 from . import constraints
@@ -17,5 +17,7 @@ from . import models
 from . import losses
 from . import optimizers
 from . import regularizers
+# Importable from root because it's technically not a layer
+from .layers import Input
 
-__version__ = '2.0.2'
+__version__ = '2.0.5'

keras/activations.py

@@ -1,7 +1,9 @@
 from __future__ import absolute_import
 import six
+import warnings
 from . import backend as K
 from .utils.generic_utils import deserialize_keras_object
+from .engine import Layer
 
 
 def softmax(x, axis=-1):
@@ -32,6 +34,20 @@ def elu(x, alpha=1.0):
     return K.elu(x, alpha)
 
 
+def selu(x):
+    """Scaled Exponential Linear Unit. (Klambauer et al., 2017)
+
+    # Arguments
+        x: A tensor or variable to compute the activation function for.
+
+    # References
+        - [Self-Normalizing Neural Networks](https://arxiv.org/abs/1706.02515)
+    """
+    alpha = 1.6732632423543772848170429916717
+    scale = 1.0507009873554804934193349852946
+    return scale * K.elu(x, alpha)
+
+
 def softplus(x):
     return K.softplus(x)
 
@@ -78,6 +94,13 @@ def get(identifier):
         identifier = str(identifier)
         return deserialize(identifier)
     elif callable(identifier):
+        if isinstance(identifier, Layer):
+            warnings.warn((
+                'Do not pass a layer instance (such as {identifier}) as the '
+                'activation argument of another layer. Instead, advanced '
+                'activation layers should be used just like any other '
+                'layer in a model.'
+            ).format(identifier=identifier.__class__.__name__))
         return identifier
     else:
         raise ValueError('Could not interpret '

keras/applications/__init__.py

@@ -3,3 +3,4 @@ from .vgg19 import VGG19
 from .resnet50 import ResNet50
 from .inception_v3 import InceptionV3
 from .xception import Xception
+from .mobilenet import MobileNet

keras/applications/inception_v3.py

@@ -27,7 +27,6 @@ from ..layers import AveragePooling2D
 from ..layers import GlobalAveragePooling2D
 from ..layers import GlobalMaxPooling2D
 from ..engine.topology import get_source_inputs
-from ..utils.layer_utils import convert_all_kernels_in_model
 from ..utils.data_utils import get_file
 from .. import backend as K
 from .imagenet_utils import decode_predictions
@@ -157,7 +156,10 @@ def InceptionV3(include_top=True,
     if input_tensor is None:
         img_input = Input(shape=input_shape)
     else:
-        img_input = Input(tensor=input_tensor, shape=input_shape)
+        if not K.is_keras_tensor(input_tensor):
+            img_input = Input(tensor=input_tensor, shape=input_shape)
+        else:
+            img_input = input_tensor
 
     if K.image_data_format() == 'channels_first':
         channel_axis = 1
@@ -381,8 +383,6 @@ def InceptionV3(include_top=True,
                 cache_subdir='models',
                 md5_hash='bcbd6486424b2319ff4ef7d526e38f63')
         model.load_weights(weights_path)
-        if K.backend() == 'theano':
-            convert_all_kernels_in_model(model)
     return model
 
 

keras/applications/mobilenet.py

@@ -0,0 +1,641 @@
+"""MobileNet v1 models for Keras.
+
+MobileNet is a general architecture and can be used for multiple use cases.
+Depending on the use case, it can use different input layer size and
+different width factors. This allows different width models to reduce
+the number of multiply-adds and thereby
+reduce inference cost on mobile devices.
+
+MobileNets support any input size greater than 32 x 32, with larger image sizes
+offering better performance.
+The number of parameters and number of multiply-adds
+can be modified by using the `alpha` parameter,
+which increases/decreases the number of filters in each layer.
+By altering the image size and `alpha` parameter,
+all 16 models from the paper can be built, with ImageNet weights provided.
+
+The paper demonstrates the performance of MobileNets using `alpha` values of
+1.0 (also called 100 % MobileNet), 0.75, 0.5 and 0.25.
+For each of these `alpha` values, weights for 4 different input image sizes
+are provided (224, 192, 160, 128).
+
+The following table describes the size and accuracy of the 100% MobileNet
+on size 224 x 224:
+----------------------------------------------------------------------------
+Width Multiplier (alpha) | ImageNet Acc |  Multiply-Adds (M) |  Params (M)
+----------------------------------------------------------------------------
+|   1.0 MobileNet-224    |    70.6 %     |        529        |     4.2     |
+|   0.75 MobileNet-224   |    68.4 %     |        325        |     2.6     |
+|   0.50 MobileNet-224   |    63.7 %     |        149        |     1.3     |
+|   0.25 MobileNet-224   |    50.6 %     |        41         |     0.5     |
+----------------------------------------------------------------------------
+
+The following table describes the performance of
+the 100 % MobileNet on various input sizes:
+------------------------------------------------------------------------
+      Resolution      | ImageNet Acc | Multiply-Adds (M) | Params (M)
+------------------------------------------------------------------------
+|  1.0 MobileNet-224  |    70.6 %    |        529        |     4.2     |
+|  1.0 MobileNet-192  |    69.1 %    |        529        |     4.2     |
+|  1.0 MobileNet-160  |    67.2 %    |        529        |     4.2     |
+|  1.0 MobileNet-128  |    64.4 %    |        529        |     4.2     |
+------------------------------------------------------------------------
+
+The weights for all 16 models are obtained and translated
+from Tensorflow checkpoints found at
+https://github.com/tensorflow/models/blob/master/slim/nets/mobilenet_v1.md
+
+# Reference
+- [MobileNets: Efficient Convolutional Neural Networks for
+   Mobile Vision Applications](https://arxiv.org/pdf/1704.04861.pdf))
+"""
+from __future__ import print_function
+from __future__ import absolute_import
+from __future__ import division
+
+import warnings
+
+from ..models import Model
+from ..layers import Input
+from ..layers import Activation
+from ..layers import Dropout
+from ..layers import Reshape
+from ..layers import BatchNormalization
+from ..layers import GlobalAveragePooling2D
+from ..layers import GlobalMaxPooling2D
+from ..layers import Conv2D
+from .. import initializers
+from .. import regularizers
+from .. import constraints
+from ..utils import conv_utils
+from ..utils.data_utils import get_file
+from ..engine.topology import get_source_inputs
+from ..engine import InputSpec
+from ..applications.imagenet_utils import _obtain_input_shape
+from ..applications.imagenet_utils import decode_predictions
+from .. import backend as K
+
+
+BASE_WEIGHT_PATH = 'https://github.com/fchollet/deep-learning-models/releases/download/v0.6/'
+
+
+def relu6(x):
+    return K.relu(x, max_value=6)
+
+
+def preprocess_input(x):
+    x /= 255.
+    x -= 0.5
+    x *= 2.
+    return x
+
+
+class DepthwiseConv2D(Conv2D):
+    """Depthwise separable 2D convolution.
+
+    Depthwise Separable convolutions consists in performing
+    just the first step in a depthwise spatial convolution
+    (which acts on each input channel separately).
+    The `depth_multiplier` argument controls how many
+    output channels are generated per input channel in the depthwise step.
+
+    # Arguments
+        kernel_size: An integer or tuple/list of 2 integers, specifying the
+            width and height of the 2D convolution window.
+            Can be a single integer to specify the same value for
+            all spatial dimensions.
+        strides: An integer or tuple/list of 2 integers,
+            specifying the strides of the convolution along the width and height.
+            Can be a single integer to specify the same value for
+            all spatial dimensions.
+            Specifying any stride value != 1 is incompatible with specifying
+            any `dilation_rate` value != 1.
+        padding: one of `"valid"` or `"same"` (case-insensitive).
+        depth_multiplier: The number of depthwise convolution output channels
+            for each input channel.
+            The total number of depthwise convolution output
+            channels will be equal to `filters_in * depth_multiplier`.
+        data_format: A string,
+            one of `channels_last` (default) or `channels_first`.
+            The ordering of the dimensions in the inputs.
+            `channels_last` corresponds to inputs with shape
+            `(batch, height, width, channels)` while `channels_first`
+            corresponds to inputs with shape
+            `(batch, channels, height, width)`.
+            It defaults to the `image_data_format` value found in your
+            Keras config file at `~/.keras/keras.json`.
+            If you never set it, then it will be "channels_last".
+        activation: Activation function to use
+            (see [activations](../activations.md)).
+            If you don't specify anything, no activation is applied
+            (ie. "linear" activation: `a(x) = x`).
+        use_bias: Boolean, whether the layer uses a bias vector.
+        depthwise_initializer: Initializer for the depthwise kernel matrix
+            (see [initializers](../initializers.md)).
+        bias_initializer: Initializer for the bias vector
+            (see [initializers](../initializers.md)).
+        depthwise_regularizer: Regularizer function applied to
+            the depthwise kernel matrix
+            (see [regularizer](../regularizers.md)).
+        bias_regularizer: Regularizer function applied to the bias vector
+            (see [regularizer](../regularizers.md)).
+        activity_regularizer: Regularizer function applied to
+            the output of the layer (its "activation").
+            (see [regularizer](../regularizers.md)).
+        depthwise_constraint: Constraint function applied to
+            the depthwise kernel matrix
+            (see [constraints](../constraints.md)).
+        bias_constraint: Constraint function applied to the bias vector
+            (see [constraints](../constraints.md)).
+
+    # Input shape
+        4D tensor with shape:
+        `[batch, channels, rows, cols]` if data_format='channels_first'
+        or 4D tensor with shape:
+        `[batch, rows, cols, channels]` if data_format='channels_last'.
+
+    # Output shape
+        4D tensor with shape:
+        `[batch, filters, new_rows, new_cols]` if data_format='channels_first'
+        or 4D tensor with shape:
+        `[batch, new_rows, new_cols, filters]` if data_format='channels_last'.
+        `rows` and `cols` values might have changed due to padding.
+    """
+
+    def __init__(self,
+                 kernel_size,
+                 strides=(1, 1),
+                 padding='valid',
+                 depth_multiplier=1,
+                 data_format=None,
+                 activation=None,
+                 use_bias=True,
+                 depthwise_initializer='glorot_uniform',
+                 bias_initializer='zeros',
+                 depthwise_regularizer=None,
+                 bias_regularizer=None,
+                 activity_regularizer=None,
+                 depthwise_constraint=None,
+                 bias_constraint=None,
+                 **kwargs):
+        super(DepthwiseConv2D, self).__init__(
+            filters=None,
+            kernel_size=kernel_size,
+            strides=strides,
+            padding=padding,
+            data_format=data_format,
+            activation=activation,
+            use_bias=use_bias,
+            bias_regularizer=bias_regularizer,
+            activity_regularizer=activity_regularizer,
+            bias_constraint=bias_constraint,
+            **kwargs)
+        self.depth_multiplier = depth_multiplier
+        self.depthwise_initializer = initializers.get(depthwise_initializer)
+        self.depthwise_regularizer = regularizers.get(depthwise_regularizer)
+        self.depthwise_constraint = constraints.get(depthwise_constraint)
+        self.bias_initializer = initializers.get(bias_initializer)
+
+    def build(self, input_shape):
+        if len(input_shape) < 4:
+            raise ValueError('Inputs to `DepthwiseConv2D` should have rank 4. '
+                             'Received input shape:', str(input_shape))
+        if self.data_format == 'channels_first':
+            channel_axis = 1
+        else:
+            channel_axis = 3
+        if input_shape[channel_axis] is None:
+            raise ValueError('The channel dimension of the inputs to '
+                             '`DepthwiseConv2D` '
+                             'should be defined. Found `None`.')
+        input_dim = int(input_shape[channel_axis])
+        depthwise_kernel_shape = (self.kernel_size[0],
+                                  self.kernel_size[1],
+                                  input_dim,
+                                  self.depth_multiplier)
+
+        self.depthwise_kernel = self.add_weight(
+            shape=depthwise_kernel_shape,
+            initializer=self.depthwise_initializer,
+            name='depthwise_kernel',
+            regularizer=self.depthwise_regularizer,
+            constraint=self.depthwise_constraint)
+
+        if self.use_bias:
+            self.bias = self.add_weight(shape=(input_dim * self.depth_multiplier,),
+                                        initializer=self.bias_initializer,
+                                        name='bias',
+                                        regularizer=self.bias_regularizer,
+                                        constraint=self.bias_constraint)
+        else:
+            self.bias = None
+        # Set input spec.
+        self.input_spec = InputSpec(ndim=4, axes={channel_axis: input_dim})
+        self.built = True
+
+    def call(self, inputs, training=None):
+        outputs = K.depthwise_conv2d(
+            inputs,
+            self.depthwise_kernel,
+            strides=self.strides,
+            padding=self.padding,
+            dilation_rate=self.dilation_rate,
+            data_format=self.data_format)
+
+        if self.bias:
+            outputs = K.bias_add(
+                outputs,
+                self.bias,
+                data_format=self.data_format)
+
+        if self.activation is not None:
+            return self.activation(outputs)
+
+        return outputs
+
+    def compute_output_shape(self, input_shape):
+        if self.data_format == 'channels_first':
+            rows = input_shape[2]
+            cols = input_shape[3]
+            out_filters = input_shape[1] * self.depth_multiplier
+        elif self.data_format == 'channels_last':
+            rows = input_shape[1]
+            cols = input_shape[2]
+            out_filters = input_shape[3] * self.depth_multiplier
+
+        rows = conv_utils.conv_output_length(rows, self.kernel_size[0],
+                                             self.padding,
+                                             self.strides[0])
+        cols = conv_utils.conv_output_length(cols, self.kernel_size[1],
+                                             self.padding,
+                                             self.strides[1])
+
+        if self.data_format == 'channels_first':
+            return (input_shape[0], out_filters, rows, cols)
+        elif self.data_format == 'channels_last':
+            return (input_shape[0], rows, cols, out_filters)
+
+    def get_config(self):
+        config = super(DepthwiseConv2D, self).get_config()
+        config.pop('filters')
+        config.pop('kernel_initializer')
+        config.pop('kernel_regularizer')
+        config.pop('kernel_constraint')
+        config['depth_multiplier'] = self.depth_multiplier
+        config['depthwise_initializer'] = initializers.serialize(self.depthwise_initializer)
+        config['depthwise_regularizer'] = regularizers.serialize(self.depthwise_regularizer)
+        config['depthwise_constraint'] = constraints.serialize(self.depthwise_constraint)
+        return config
+
+
+def MobileNet(input_shape=None,
+              alpha=1.0,
+              depth_multiplier=1,
+              dropout=1e-3,
+              include_top=True,
+              weights='imagenet',
+              input_tensor=None,
+              pooling=None,
+              classes=1000):
+    """Instantiates the MobileNet architecture.
+
+    Note that only TensorFlow is supported for now,
+    therefore it only works with the data format
+    `image_data_format='channels_last'` in your Keras config
+    at `~/.keras/keras.json`.
+
+    To load a MobileNet model via `load_model`, import the custom
+    objects `relu6` and `DepthwiseConv2D` and pass them to the
+    `custom_objects` parameter.
+    E.g.
+    model = load_model('mobilenet.h5', custom_objects={
+                       'relu6': mobilenet.relu6,
+                       'DepthwiseConv2D': mobilenet.DepthwiseConv2D})
+
+    # Arguments
+        input_shape: optional shape tuple, only to be specified
+            if `include_top` is False (otherwise the input shape
+            has to be `(224, 224, 3)` (with `channels_last` data format)
+            or (3, 224, 224) (with `channels_first` data format).
+            It should have exactly 3 inputs channels,
+            and width and height should be no smaller than 32.
+            E.g. `(200, 200, 3)` would be one valid value.
+        alpha: controls the width of the network.
+            - If `alpha` < 1.0, proportionally decreases the number
+                of filters in each layer.
+            - If `alpha` > 1.0, proportionally increases the number
+                of filters in each layer.
+            - If `alpha` = 1, default number of filters from the paper
+                 are used at each layer.
+        depth_multiplier: depth multiplier for depthwise convolution
+            (also called the resolution multiplier)
+        dropout: dropout rate
+        include_top: whether to include the fully-connected
+            layer at the top of the network.
+        weights: `None` (random initialization) or
+            `imagenet` (ImageNet weights)
+        input_tensor: optional Keras tensor (i.e. output of
+            `layers.Input()`)
+            to use as image input for the model.
+        pooling: Optional pooling mode for feature extraction
+            when `include_top` is `False`.
+            - `None` means that the output of the model
+                will be the 4D tensor output of the
+                last convolutional layer.
+            - `avg` means that global average pooling
+                will be applied to the output of the
+                last convolutional layer, and thus
+                the output of the model will be a
+                2D tensor.
+            - `max` means that global max pooling will
+                be applied.
+        classes: optional number of classes to classify images
+            into, only to be specified if `include_top` is True, and
+            if no `weights` argument is specified.
+
+    # Returns
+        A Keras model instance.
+
+    # Raises
+        ValueError: in case of invalid argument for `weights`,
+            or invalid input shape.
+        RuntimeError: If attempting to run this model with a
+            backend that does not support separable convolutions.
+    """
+
+    if K.backend() != 'tensorflow':
+        raise RuntimeError('Only Tensorflow backend is currently supported, '
+                           'as other backends do not support '
+                           'depthwise convolution.')
+
+    if weights not in {'imagenet', None}:
+        raise ValueError('The `weights` argument should be either '
+                         '`None` (random initialization) or `imagenet` '
+                         '(pre-training on ImageNet).')
+
+    if weights == 'imagenet' and include_top and classes != 1000:
+        raise ValueError('If using `weights` as ImageNet with `include_top` '
+                         'as true, `classes` should be 1000')
+
+    # Determine proper input shape.
+    input_shape = _obtain_input_shape(input_shape,
+                                      default_size=224,
+                                      min_size=32,
+                                      data_format=K.image_data_format(),
+                                      include_top=include_top or weights)
+    if K.image_data_format() == 'channels_last':
+        row_axis, col_axis = (0, 1)
+    else:
+        row_axis, col_axis = (1, 2)
+    rows = input_shape[row_axis]
+    cols = input_shape[col_axis]
+
+    if weights == 'imagenet':
+        if depth_multiplier != 1:
+            raise ValueError('If imagenet weights are being loaded, '
+                             'depth multiplier must be 1')
+
+        if alpha not in [0.25, 0.50, 0.75, 1.0]:
+            raise ValueError('If imagenet weights are being loaded, '
+                             'alpha can be one of'
+                             '`0.25`, `0.50`, `0.75` or `1.0` only.')
+
+        if rows != cols or rows not in [128, 160, 192, 224]:
+            raise ValueError('If imagenet weights are being loaded, '
+                             'input must have a static square shape (one of '
+                             '(128,128), (160,160), (192,192), or (224, 224)).'
+                             ' Input shape provided = %s' % (input_shape,))
+
+    if K.image_data_format() != 'channels_last':
+        warnings.warn('The MobileNet family of models is only available '
+                      'for the input data format "channels_last" '
+                      '(width, height, channels). '
+                      'However your settings specify the default '
+                      'data format "channels_first" (channels, width, height).'
+                      ' You should set `image_data_format="channels_last"` '
+                      'in your Keras config located at ~/.keras/keras.json. '
+                      'The model being returned right now will expect inputs '
+                      'to follow the "channels_last" data format.')
+        K.set_image_data_format('channels_last')
+        old_data_format = 'channels_first'
+    else:
+        old_data_format = None
+
+    if input_tensor is None:
+        img_input = Input(shape=input_shape)
+    else:
+        if not K.is_keras_tensor(input_tensor):
+            img_input = Input(tensor=input_tensor, shape=input_shape)
+        else:
+            img_input = input_tensor
+
+    x = _conv_block(img_input, 32, alpha, strides=(2, 2))
+    x = _depthwise_conv_block(x, 64, alpha, depth_multiplier, block_id=1)
+
+    x = _depthwise_conv_block(x, 128, alpha, depth_multiplier,
+                              strides=(2, 2), block_id=2)
+    x = _depthwise_conv_block(x, 128, alpha, depth_multiplier, block_id=3)
+
+    x = _depthwise_conv_block(x, 256, alpha, depth_multiplier,
+                              strides=(2, 2), block_id=4)
+    x = _depthwise_conv_block(x, 256, alpha, depth_multiplier, block_id=5)
+
+    x = _depthwise_conv_block(x, 512, alpha, depth_multiplier,
+                              strides=(2, 2), block_id=6)
+    x = _depthwise_conv_block(x, 512, alpha, depth_multiplier, block_id=7)
+    x = _depthwise_conv_block(x, 512, alpha, depth_multiplier, block_id=8)
+    x = _depthwise_conv_block(x, 512, alpha, depth_multiplier, block_id=9)
+    x = _depthwise_conv_block(x, 512, alpha, depth_multiplier, block_id=10)
+    x = _depthwise_conv_block(x, 512, alpha, depth_multiplier, block_id=11)
+
+    x = _depthwise_conv_block(x, 1024, alpha, depth_multiplier,
+                              strides=(2, 2), block_id=12)
+    x = _depthwise_conv_block(x, 1024, alpha, depth_multiplier, block_id=13)
+
+    if include_top:
+        if K.image_data_format() == 'channels_first':
+            shape = (int(1024 * alpha), 1, 1)
+        else:
+            shape = (1, 1, int(1024 * alpha))
+
+        x = GlobalAveragePooling2D()(x)
+        x = Reshape(shape, name='reshape_1')(x)
+        x = Dropout(dropout, name='dropout')(x)
+        x = Conv2D(classes, (1, 1),
+                   padding='same', name='conv_preds')(x)
+        x = Activation('softmax', name='act_softmax')(x)
+        x = Reshape((classes,), name='reshape_2')(x)
+    else:
+        if pooling == 'avg':
+            x = GlobalAveragePooling2D()(x)
+        elif pooling == 'max':
+            x = GlobalMaxPooling2D()(x)
+
+    # Ensure that the model takes into account
+    # any potential predecessors of `input_tensor`.
+    if input_tensor is not None:
+        inputs = get_source_inputs(input_tensor)
+    else:
+        inputs = img_input
+
+    # Create model.
+    model = Model(inputs, x, name='mobilenet_%0.2f_%s' % (alpha, rows))
+
+    # load weights
+    if weights == 'imagenet':
+        if K.image_data_format() == 'channels_first':
+            raise ValueError('Weights for "channels_last" format '
+                             'are not available.')
+        if alpha == 1.0:
+            alpha_text = '1_0'
+        elif alpha == 0.75:
+            alpha_text = '7_5'
+        elif alpha == 0.50:
+            alpha_text = '5_0'
+        else:
+            alpha_text = '2_5'
+
+        if include_top:
+            model_name = 'mobilenet_%s_%d_tf.h5' % (alpha_text, rows)
+            weigh_path = BASE_WEIGHT_PATH + model_name
+            weights_path = get_file(model_name,
+                                    weigh_path,
+                                    cache_subdir='models')
+        else:
+            model_name = 'mobilenet_%s_%d_tf_no_top.h5' % (alpha_text, rows)
+            weigh_path = BASE_WEIGHT_PATH + model_name
+            weights_path = get_file(model_name,
+                                    weigh_path,
+                                    cache_subdir='models')
+        model.load_weights(weights_path)
+
+    if old_data_format:
+        K.set_image_data_format(old_data_format)
+    return model
+
+
+def _conv_block(inputs, filters, alpha, kernel=(3, 3), strides=(1, 1)):
+    """Adds an initial convolution layer (with batch normalization and relu6).
+
+    # Arguments
+        inputs: Input tensor of shape `(rows, cols, 3)`
+            (with `channels_last` data format) or
+            (3, rows, cols) (with `channels_first` data format).
+            It should have exactly 3 inputs channels,
+            and width and height should be no smaller than 32.
+            E.g. `(224, 224, 3)` would be one valid value.
+        filters: Integer, the dimensionality of the output space
+            (i.e. the number output of filters in the convolution).
+        alpha: controls the width of the network.
+            - If `alpha` < 1.0, proportionally decreases the number
+                of filters in each layer.
+            - If `alpha` > 1.0, proportionally increases the number
+                of filters in each layer.
+            - If `alpha` = 1, default number of filters from the paper
+                 are used at each layer.
+        kernel: An integer or tuple/list of 2 integers, specifying the
+            width and height of the 2D convolution window.
+            Can be a single integer to specify the same value for
+            all spatial dimensions.
+        strides: An integer or tuple/list of 2 integers,
+            specifying the strides of the convolution along the width and height.
+            Can be a single integer to specify the same value for
+            all spatial dimensions.
+            Specifying any stride value != 1 is incompatible with specifying
+            any `dilation_rate` value != 1.
+
+    # Input shape
+        4D tensor with shape:
+        `(samples, channels, rows, cols)` if data_format='channels_first'
+        or 4D tensor with shape:
+        `(samples, rows, cols, channels)` if data_format='channels_last'.
+
+    # Output shape
+        4D tensor with shape:
+        `(samples, filters, new_rows, new_cols)` if data_format='channels_first'
+        or 4D tensor with shape:
+        `(samples, new_rows, new_cols, filters)` if data_format='channels_last'.
+        `rows` and `cols` values might have changed due to stride.
+
+    # Returns
+        Output tensor of block.
+    """
+    channel_axis = 1 if K.image_data_format() == 'channels_first' else -1
+    filters = int(filters * alpha)
+    x = Conv2D(filters, kernel,
+               padding='same',
+               use_bias=False,
+               strides=strides,
+               name='conv1')(inputs)
+    x = BatchNormalization(axis=channel_axis, name='conv1_bn')(x)
+    return Activation(relu6, name='conv1_relu')(x)
+
+
+def _depthwise_conv_block(inputs, pointwise_conv_filters, alpha,
+                          depth_multiplier=1, strides=(1, 1), block_id=1):
+    """Adds a depthwise convolution block.
+
+    A depthwise convolution block consists of a depthwise conv,
+    batch normalization, relu6, pointwise convolution,
+    batch normalization and relu6 activation.
+
+    # Arguments
+        inputs: Input tensor of shape `(rows, cols, channels)`
+            (with `channels_last` data format) or
+            (channels, rows, cols) (with `channels_first` data format).
+        pointwise_conv_filters: Integer, the dimensionality of the output space
+            (i.e. the number output of filters in the pointwise convolution).
+        alpha: controls the width of the network.
+            - If `alpha` < 1.0, proportionally decreases the number
+                of filters in each layer.
+            - If `alpha` > 1.0, proportionally increases the number
+                of filters in each layer.
+            - If `alpha` = 1, default number of filters from the paper
+                 are used at each layer.
+        depth_multiplier: The number of depthwise convolution output channels
+            for each input channel.
+            The total number of depthwise convolution output
+            channels will be equal to `filters_in * depth_multiplier`.
+        strides: An integer or tuple/list of 2 integers,
+            specifying the strides of the convolution along the width and height.
+            Can be a single integer to specify the same value for
+            all spatial dimensions.
+            Specifying any stride value != 1 is incompatible with specifying
+            any `dilation_rate` value != 1.
+        block_id: Integer, a unique identification designating the block number.
+
+    # Input shape
+        4D tensor with shape:
+        `(batch, channels, rows, cols)` if data_format='channels_first'
+        or 4D tensor with shape:
+        `(batch, rows, cols, channels)` if data_format='channels_last'.
+
+    # Output shape
+        4D tensor with shape:
+        `(batch, filters, new_rows, new_cols)` if data_format='channels_first'
+        or 4D tensor with shape:
+        `(batch, new_rows, new_cols, filters)` if data_format='channels_last'.
+        `rows` and `cols` values might have changed due to stride.
+
+    # Returns
+        Output tensor of block.
+    """
+    channel_axis = 1 if K.image_data_format() == 'channels_first' else -1
+    pointwise_conv_filters = int(pointwise_conv_filters * alpha)
+
+    x = DepthwiseConv2D((3, 3),
+                        padding='same',
+                        depth_multiplier=depth_multiplier,
+                        strides=strides,
+                        use_bias=False,
+                        name='conv_dw_%d' % block_id)(inputs)
+    x = BatchNormalization(axis=channel_axis, name='conv_dw_%d_bn' % block_id)(x)
+    x = Activation(relu6, name='conv_dw_%d_relu' % block_id)(x)
+
+    x = Conv2D(pointwise_conv_filters, (1, 1),
+               padding='same',
+               use_bias=False,
+               strides=(1, 1),
+               name='conv_pw_%d' % block_id)(x)
+    x = BatchNormalization(axis=channel_axis, name='conv_pw_%d_bn' % block_id)(x)
+    return Activation(relu6, name='conv_pw_%d_relu' % block_id)(x)

keras/applications/resnet50.py

@@ -43,7 +43,7 @@ def identity_block(input_tensor, kernel_size, filters, stage, block):
 
     # Arguments
         input_tensor: input tensor
-        kernel_size: defualt 3, the kernel size of middle conv layer at main path
+        kernel_size: default 3, the kernel size of middle conv layer at main path
         filters: list of integers, the filterss of 3 conv layer at main path
         stage: integer, current stage label, used for generating layer names
         block: 'a','b'..., current block label, used for generating layer names
@@ -77,11 +77,11 @@ def identity_block(input_tensor, kernel_size, filters, stage, block):
 
 
 def conv_block(input_tensor, kernel_size, filters, stage, block, strides=(2, 2)):
-    """conv_block is the block that has a conv layer at shortcut
+    """A block that has a conv layer at shortcut.
 
     # Arguments
         input_tensor: input tensor
-        kernel_size: defualt 3, the kernel size of middle conv layer at main path
+        kernel_size: default 3, the kernel size of middle conv layer at main path
         filters: list of integers, the filterss of 3 conv layer at main path
         stage: integer, current stage label, used for generating layer names
         block: 'a','b'..., current block label, used for generating layer names
@@ -149,7 +149,7 @@ def ResNet50(include_top=True, weights='imagenet',
         input_shape: optional shape tuple, only to be specified
             if `include_top` is False (otherwise the input shape
             has to be `(224, 224, 3)` (with `channels_last` data format)
-            or `(3, 224, 244)` (with `channels_first` data format).
+            or `(3, 224, 224)` (with `channels_first` data format).
             It should have exactly 3 inputs channels,
             and width and height should be no smaller than 197.
             E.g. `(200, 200, 3)` would be one valid value.
@@ -264,21 +264,19 @@ def ResNet50(include_top=True, weights='imagenet',
         model.load_weights(weights_path)
         if K.backend() == 'theano':
             layer_utils.convert_all_kernels_in_model(model)
-
-        if K.image_data_format() == 'channels_first':
             if include_top:
                 maxpool = model.get_layer(name='avg_pool')
                 shape = maxpool.output_shape[1:]
                 dense = model.get_layer(name='fc1000')
                 layer_utils.convert_dense_weights_data_format(dense, shape, 'channels_first')
 
-            if K.backend() == 'tensorflow':
-                warnings.warn('You are using the TensorFlow backend, yet you '
-                              'are using the Theano '
-                              'image data format convention '
-                              '(`image_data_format="channels_first"`). '
-                              'For best performance, set '
-                              '`image_data_format="channels_last"` in '
-                              'your Keras config '
-                              'at ~/.keras/keras.json.')
+        if K.image_data_format() == 'channels_first' and K.backend() == 'tensorflow':
+            warnings.warn('You are using the TensorFlow backend, yet you '
+                          'are using the Theano '
+                          'image data format convention '
+                          '(`image_data_format="channels_first"`). '
+                          'For best performance, set '
+                          '`image_data_format="channels_last"` in '
+                          'your Keras config '
+                          'at ~/.keras/keras.json.')
     return model

keras/applications/vgg16.py

@@ -59,7 +59,7 @@ def VGG16(include_top=True, weights='imagenet',
         input_shape: optional shape tuple, only to be specified
             if `include_top` is False (otherwise the input shape
             has to be `(224, 224, 3)` (with `channels_last` data format)
-            or `(3, 224, 244)` (with `channels_first` data format).
+            or `(3, 224, 224)` (with `channels_first` data format).
             It should have exactly 3 inputs channels,
             and width and height should be no smaller than 48.
             E.g. `(200, 200, 3)` would be one valid value.

keras/applications/vgg19.py

@@ -59,7 +59,7 @@ def VGG19(include_top=True, weights='imagenet',
         input_shape: optional shape tuple, only to be specified
             if `include_top` is False (otherwise the input shape
             has to be `(224, 224, 3)` (with `channels_last` data format)
-            or `(3, 224, 244)` (with `channels_first` data format).
+            or `(3, 224, 224)` (with `channels_first` data format).
             It should have exactly 3 inputs channels,
             and width and height should be no smaller than 48.
             E.g. `(200, 200, 3)` would be one valid value.

keras/backend/__init__.py

@@ -10,28 +10,29 @@ from .common import set_floatx
 from .common import cast_to_floatx
 from .common import image_data_format
 from .common import set_image_data_format
-from .common import is_keras_tensor
 
+# Obtain Keras base dir path: either ~/.keras or /tmp.
 _keras_base_dir = os.path.expanduser('~')
 if not os.access(_keras_base_dir, os.W_OK):
     _keras_base_dir = '/tmp'
-
 _keras_dir = os.path.join(_keras_base_dir, '.keras')
-if not os.path.exists(_keras_dir):
-    os.makedirs(_keras_dir)
 
 # Default backend: TensorFlow.
 _BACKEND = 'tensorflow'
 
+# Attempt to read Keras config file.
 _config_path = os.path.expanduser(os.path.join(_keras_dir, 'keras.json'))
 if os.path.exists(_config_path):
-    _config = json.load(open(_config_path))
+    try:
+        _config = json.load(open(_config_path))
+    except ValueError:
+        _config = {}
     _floatx = _config.get('floatx', floatx())
     assert _floatx in {'float16', 'float32', 'float64'}
     _epsilon = _config.get('epsilon', epsilon())
     assert isinstance(_epsilon, float)
     _backend = _config.get('backend', _BACKEND)
-    assert _backend in {'theano', 'tensorflow'}
+    assert _backend in {'theano', 'tensorflow', 'cntk'}
     _image_data_format = _config.get('image_data_format',
                                      image_data_format())
     assert _image_data_format in {'channels_last', 'channels_first'}
@@ -41,22 +42,40 @@ if os.path.exists(_config_path):
     set_image_data_format(_image_data_format)
     _BACKEND = _backend
 
-# save config file
-if not os.path.exists(_config_path):
-    _config = {'floatx': floatx(),
-               'epsilon': epsilon(),
-               'backend': _BACKEND,
-               'image_data_format': image_data_format()}
-    with open(_config_path, 'w') as f:
-        f.write(json.dumps(_config, indent=4))
+# Save config file, if possible.
+if not os.path.exists(_keras_dir):
+    try:
+        os.makedirs(_keras_dir)
+    except OSError:
+        # Except permission denied and potential race conditions
+        # in multi-threaded environments.
+        pass
 
+if not os.path.exists(_config_path):
+    _config = {
+        'floatx': floatx(),
+        'epsilon': epsilon(),
+        'backend': _BACKEND,
+        'image_data_format': image_data_format()
+    }
+    try:
+        with open(_config_path, 'w') as f:
+            f.write(json.dumps(_config, indent=4))
+    except IOError:
+        # Except permission denied.
+        pass
+
+# Set backend based on KERAS_BACKEND flag, if applicable.
 if 'KERAS_BACKEND' in os.environ:
     _backend = os.environ['KERAS_BACKEND']
-    assert _backend in {'theano', 'tensorflow'}
+    assert _backend in {'theano', 'tensorflow', 'cntk'}
     _BACKEND = _backend
 
-# import backend
-if _BACKEND == 'theano':
+# Import backend functions.
+if _BACKEND == 'cntk':
+    sys.stderr.write('Using CNTK backend\n')
+    from .cntk_backend import *
+elif _BACKEND == 'theano':
     sys.stderr.write('Using Theano backend.\n')
     from .theano_backend import *
 elif _BACKEND == 'tensorflow':

keras/backend/common.py

@@ -44,7 +44,7 @@ def set_epsilon(e):
 
 
 def floatx():
-    """Returns the default float type, as a string
+    """Returns the default float type, as a string.
     (e.g. 'float16', 'float32', 'float64').
 
     # Returns
@@ -63,7 +63,7 @@ def set_floatx(floatx):
     """Sets the default float type.
 
     # Arguments
-        String: 'float16', 'float32', or 'float64'.
+        floatx: String, 'float16', 'float32', or 'float64'.
 
     # Example
     ```python
@@ -109,8 +109,7 @@ def cast_to_floatx(x):
 
 
 def image_data_format():
-    """Returns the default image data format
-    convention ('channels_first' or 'channels_last').
+    """Returns the default image data format convention ('channels_first' or 'channels_last').
 
     # Returns
         A string, either `'channels_first'` or `'channels_last'`
@@ -146,42 +145,13 @@ def set_image_data_format(data_format):
     _IMAGE_DATA_FORMAT = str(data_format)
 
 
-def is_keras_tensor(x):
-    """Returns whether `x` is a Keras tensor.
-
-    # Arguments
-        x: a potential tensor.
-
-    # Returns
-        A boolean: whether the argument is a Keras tensor.
-
-    # Examples
-    ```python
-        >>> from keras import backend as K
-        >>> np_var = numpy.array([1, 2])
-        >>> K.is_keras_tensor(np_var)
-        False
-        >>> keras_var = K.variable(np_var)
-        >>> K.is_keras_tensor(keras_var)  # A variable is not a Tensor.
-        False
-        >>> keras_placeholder = K.placeholder(shape=(2, 4, 5))
-        >>> K.is_keras_tensor(keras_placeholder)  # A placeholder is a Tensor.
-        True
-    ```
-    """
-    if hasattr(x, '_keras_shape'):
-        return True
-    else:
-        return False
-
-
 # Legacy methods
 
 def set_image_dim_ordering(dim_ordering):
     """Legacy setter for `image_data_format`.
 
     # Arguments
-        dim_ordering: string. `'tf'` or `'th'`.
+        dim_ordering: string. `tf` or `th`.
 
     # Example
     ```python
@@ -192,6 +162,9 @@ def set_image_dim_ordering(dim_ordering):
         >>> K.image_data_format()
         'channels_last'
     ```
+
+    # Raises
+        ValueError: if `dim_ordering` is invalid.
     """
     global _IMAGE_DATA_FORMAT
     if dim_ordering not in {'tf', 'th'}:
@@ -205,6 +178,9 @@ def set_image_dim_ordering(dim_ordering):
 
 def image_dim_ordering():
     """Legacy getter for `image_data_format`.
+
+    # Returns
+        string, one of `'th'`, `'tf'`
     """
     if _IMAGE_DATA_FORMAT == 'channels_first':
         return 'th'

keras/backend/theano_backend.py

@@ -13,9 +13,10 @@ try:
     from theano.tensor.nnet.nnet import softsign as T_softsign
 except ImportError:
     from theano.sandbox.softsign import softsign as T_softsign
-import inspect
+
 import numpy as np
 from .common import _FLOATX, floatx, _EPSILON, image_data_format
+from ..utils.generic_utils import has_arg
 # Legacy functions
 from .common import set_image_dim_ordering, image_dim_ordering
 
@@ -163,6 +164,42 @@ def constant(value, dtype=None, shape=None, name=None):
     return const
 
 
+def is_keras_tensor(x):
+    """Returns whether `x` is a Keras tensor.
+
+    # Arguments
+        x: a potential tensor.
+
+    # Returns
+        A boolean: whether the argument is a Keras tensor.
+
+    # Raises
+        ValueError: in case `x` is not a symbolic tensor.
+
+    # Examples
+    ```python
+        >>> from keras import backend as K
+        >>> np_var = numpy.array([1, 2])
+        >>> K.is_keras_tensor(np_var) # A numpy array is not a symbolic tensor.
+        ValueError
+        >>> k_var = theano.shared(value=np.array([1,2,3]))
+        >>> K.is_keras_tensor(k_var) # A variable created directly from tensorflow/theano is not a Keras tensor.
+        False
+        >>> keras_var = K.variable(np_var)
+        >>> K.is_keras_tensor(keras_var) # A variable created with the keras backend is a Keras tensor.
+        True
+        >>> keras_placeholder = K.placeholder(shape=(2, 4, 5))
+        >>> K.is_keras_tensor(keras_placeholder)  # A placeholder is a Keras tensor.
+        True
+    ```
+    """
+    if not isinstance(x, (T.TensorVariable,
+                          T.sharedvar.TensorSharedVariable)):
+        raise ValueError('Unexpectedly found an instance of type `' + str(type(x)) + '`. '
+                         'Expected a symbolic tensor instance.')
+    return hasattr(x, '_keras_history')
+
+
 def placeholder(shape=None, ndim=None, dtype=None, sparse=False, name=None):
     """Instantiate an input data placeholder variable.
     """
@@ -258,6 +295,18 @@ def zeros_like(x, dtype=None, name=None):
     return T.zeros_like(x, dtype=dtype)
 
 
+def identity(x):
+    """Returns a tensor with the same content as the input tensor.
+
+    # Arguments
+        x: The input tensor.
+
+    # Returns
+        A tensor of the same shape, type and content.
+    """
+    return x.copy()
+
+
 def random_uniform_variable(shape, low, high, dtype=None, name=None):
     return variable(np.random.uniform(low=low, high=high, size=shape),
                     dtype=dtype, name=name)
@@ -396,10 +445,14 @@ def transpose(x):
 
 
 def gather(reference, indices):
-    """reference: a tensor.
-    indices: an int tensor of indices.
+    """Retrieves the elements of indices `indices` in the tensor `reference`.
 
-    Return: a tensor of same type as reference.
+    # Arguments
+        reference: A tensor.
+        indices: An integer tensor of indices.
+
+    # Returns
+        A tensor of same type as `reference`.
     """
     y = reference[indices]
     if hasattr(reference, '_keras_shape') and hasattr(indices, '_keras_shape'):
@@ -430,6 +483,32 @@ def prod(x, axis=None, keepdims=False):
     return T.prod(x, axis=axis, keepdims=keepdims)
 
 
+def cumsum(x, axis=0):
+    """Cumulative sum of the values in a tensor, alongside the specified axis.
+
+    # Arguments
+        x: A tensor or variable.
+        axis: An integer, the axis to compute the sum.
+
+    # Returns
+        A tensor of the cumulative sum of values of `x` along `axis`.
+    """
+    return T.extra_ops.cumsum(x, axis=axis)
+
+
+def cumprod(x, axis=0):
+    """Cumulative product of the values in a tensor, alongside the specified axis.
+
+    # Arguments
+        x: A tensor or variable.
+        axis: An integer, the axis to compute the product.
+
+    # Returns
+        A tensor of the cumulative product of values of `x` along `axis`.
+    """
+    return T.extra_ops.cumprod(x, axis=axis)
+
+
 def mean(x, axis=None, keepdims=False):
     """Mean of a tensor, alongside the specified axis.
     """
@@ -489,6 +568,29 @@ def log(x):
     return T.log(x)
 
 
+def logsumexp(x, axis=None, keepdims=False):
+    """Computes log(sum(exp(elements across dimensions of a tensor))).
+
+    This function is more numerically stable than log(sum(exp(x))).
+    It avoids overflows caused by taking the exp of large inputs and
+    underflows caused by taking the log of small inputs.
+
+    # Arguments
+        x: A tensor or variable.
+        axis: An integer, the axis to reduce over.
+        keepdims: A boolean, whether to keep the dimensions or not.
+            If `keepdims` is `False`, the rank of the tensor is reduced
+            by 1. If `keepdims` is `True`, the reduced dimension is
+            retained with length 1.
+
+    # Returns
+        The reduced tensor.
+    """
+    # Theano has a built-in optimization for logsumexp (see https://github.com/Theano/Theano/pull/4736)
+    # so we can just write the expression directly:
+    return T.log(T.sum(T.exp(x), axis=axis, keepdims=keepdims))
+
+
 def round(x):
     return T.round(x, mode='half_to_even')
 
@@ -589,7 +691,7 @@ def batch_normalization(x, mean, var, beta, gamma, epsilon=1e-3):
 
     if mean.ndim == 1:
         # based on TensorFlow's default: normalize along rightmost dimension
-        reduction_axes = range(x.ndim - 1)
+        reduction_axes = list(range(x.ndim - 1))
     else:
         reduction_axes = [i for i in range(x.ndim) if mean.broadcastable[i]]
 
@@ -712,6 +814,8 @@ def concatenate(tensors, axis=-1):
 def reshape(x, shape):
     y = T.reshape(x, shape)
     if _is_explicit_shape(shape):
+        if -1 in shape:
+            shape = tuple(x if x != -1 else None for x in shape)
         y._keras_shape = shape
         if hasattr(x, '_uses_learning_phase'):
             y._uses_learning_phase = x._uses_learning_phase
@@ -829,7 +933,7 @@ def tile(x, n):
                     output_shape += (None,)
                 else:
                     output_shape += (i * j,)
-        elif type(n) is int:
+        elif isinstance(n, int):
             output_shape = x._keras_shape[:-1]
             if x._keras_shape[-1] is None:
                 output_shape += (None,)
@@ -951,7 +1055,7 @@ def spatial_2d_padding(x, padding=((1, 1), (1, 1)), data_format=None):
                    slice(top_pad, input_shape[2] + top_pad),
                    slice(left_pad, input_shape[3] + left_pad))
 
-    elif data_format == 'channels_last':
+    else:
         output_shape = (input_shape[0],
                         input_shape[1] + top_pad + bottom_pad,
                         input_shape[2] + left_pad + right_pad,
@@ -961,8 +1065,6 @@ def spatial_2d_padding(x, padding=((1, 1), (1, 1)), data_format=None):
                    slice(top_pad, input_shape[1] + top_pad),
                    slice(left_pad, input_shape[2] + left_pad),
                    slice(None))
-    else:
-        raise ValueError('Invalid data_format:', data_format)
     y = T.set_subtensor(output[indices], x)
     y._keras_shape = output_shape
     return y
@@ -991,7 +1093,7 @@ def spatial_3d_padding(x, padding=((1, 1), (1, 1), (1, 1)), data_format=None):
                    slice(padding[1][0], input_shape[3] + padding[1][0]),
                    slice(padding[2][0], input_shape[4] + padding[2][0]))
 
-    elif data_format == 'channels_last':
+    else:
         output_shape = (input_shape[0],
                         input_shape[1] + padding[0][0] + padding[0][1],
                         input_shape[2] + padding[1][0] + padding[1][1],
@@ -1003,8 +1105,6 @@ def spatial_3d_padding(x, padding=((1, 1), (1, 1), (1, 1)), data_format=None):
                    slice(padding[1][0], input_shape[2] + padding[1][0]),
                    slice(padding[2][0], input_shape[3] + padding[2][0]),
                    slice(None))
-    else:
-        raise ValueError('Invalid data_format:', data_format)
     return T.set_subtensor(output[indices], x)
 
 
@@ -1041,8 +1141,8 @@ def pattern_broadcast(x, broatcastable):
 
 def get_value(x):
     if not hasattr(x, 'get_value'):
-        raise TypeError('get_value() can only be called on a variable. '
-                        'If you have an expression instead, use eval().')
+        raise TypeError('`get_value` can only be called on a variable. '
+                        'If you have an expression instead, use `eval()`.')
     return x.get_value()
 
 
@@ -1078,7 +1178,7 @@ def print_tensor(x, message=''):
 
 class Function(object):
 
-    def __init__(self, inputs, outputs, updates=[], **kwargs):
+    def __init__(self, inputs, outputs, updates=[], name=None, **kwargs):
         unique_variables_to_update = {}
         for v, nv in updates:
             if v not in unique_variables_to_update:
@@ -1087,7 +1187,9 @@ class Function(object):
         self.function = theano.function(inputs, outputs, updates=updates,
                                         allow_input_downcast=True,
                                         on_unused_input='ignore',
+                                        name=name,
                                         **kwargs)
+        self.name = name
 
     def __call__(self, inputs):
         assert isinstance(inputs, (list, tuple))
@@ -1096,10 +1198,9 @@ class Function(object):
 
 def function(inputs, outputs, updates=[], **kwargs):
     if len(kwargs) > 0:
-        function_args = inspect.getargspec(theano.function)[0]
         for key in kwargs.keys():
-            if key not in function_args:
-                msg = 'Invalid argument "%s" passed to K.function' % key
+            if not has_arg(theano.function, key, True):
+                msg = 'Invalid argument "%s" passed to K.function with Theano backend' % key
                 raise ValueError(msg)
     return Function(inputs, outputs, updates=updates, **kwargs)
 
@@ -1127,19 +1228,19 @@ def rnn(step_function, inputs, initial_states,
             (at least 3D).
         step_function:
             Parameters:
-                input: tensor with shape (samples, ...) (no time dimension),
+                inputs: tensor with shape (samples, ...) (no time dimension),
                     representing input for the batch of samples at a certain
                     time step.
                 states: list of tensors.
             Returns:
-                output: tensor with shape (samples, ...) (no time dimension),
+                outputs: tensor with shape (samples, ...) (no time dimension),
                 new_states: list of tensors, same length and shapes
                     as 'states'.
         initial_states: tensor with shape (samples, ...) (no time dimension),
             containing the initial values for the states used in
             the step function.
-        go_backwards: boolean. If True, do the iteration over
-            the time dimension in reverse order.
+        go_backwards: boolean. If True, do the iteration over the time
+            dimension in reverse order and return the reversed sequence.
         mask: binary tensor with shape (samples, time),
             with a zero for every element that is masked.
         constants: a list of constant values passed at each step.
@@ -1213,14 +1314,14 @@ def rnn(step_function, inputs, initial_states,
             if len(initial_states) > 0:
                 initial_states[0] = T.unbroadcast(initial_states[0], 0, 1)
 
-            def _step(input, mask, output_tm1, *states):
-                output, new_states = step_function(input, states)
+            def _step(inputs, mask, output_tm1, *states):
+                outputs, new_states = step_function(inputs, states)
                 # output previous output if masked.
-                output = T.switch(mask, output, output_tm1)
+                outputs = T.switch(mask, outputs, output_tm1)
                 return_states = []
                 for state, new_state in zip(states, new_states):
                     return_states.append(T.switch(mask, new_state, state))
-                return [output] + return_states
+                return [outputs] + return_states
 
             results, _ = theano.scan(
                 _step,
@@ -1246,8 +1347,8 @@ def rnn(step_function, inputs, initial_states,
             successive_states = []
             states = initial_states
             for i in indices:
-                output, states = step_function(inputs[i], states + constants)
-                successive_outputs.append(output)
+                outputs, states = step_function(inputs[i], states + constants)
+                successive_outputs.append(outputs)
                 successive_states.append(states)
             outputs = T.stack(*successive_outputs)
             states = []
@@ -1255,9 +1356,9 @@ def rnn(step_function, inputs, initial_states,
                 states.append(T.stack(*[states_at_step[i] for states_at_step in successive_states]))
 
         else:
-            def _step(input, *states):
-                output, new_states = step_function(input, states)
-                return [output] + new_states
+            def _step(inputs, *states):
+                outputs, new_states = step_function(inputs, states)
+                return [outputs] + new_states
 
             # Theano likes to make shape==1 dimensions in the initial states (outputs_info) broadcastable
             if len(initial_states) > 0:
@@ -1288,7 +1389,18 @@ def rnn(step_function, inputs, initial_states,
 
 
 def switch(condition, then_expression, else_expression):
-    """condition: scalar tensor.
+    """Switches between two operations depending on a scalar value.
+
+    Note that both `then_expression` and `else_expression`
+    should be symbolic tensors of the *same shape*.
+
+    # Arguments
+        condition: scalar tensor (`int` or `bool`).
+        then_expression: either a tensor, or a callable that returns a tensor.
+        else_expression: either a tensor, or a callable that returns a tensor.
+
+    # Returns
+        The selected tensor.
     """
     if callable(then_expression):
         then_expression = then_expression()
@@ -1389,7 +1501,7 @@ def softsign(x):
     return T_softsign(x)
 
 
-def categorical_crossentropy(output, target, from_logits=False):
+def categorical_crossentropy(target, output, from_logits=False):
     if from_logits:
         output = T.nnet.softmax(output)
     else:
@@ -1400,14 +1512,14 @@ def categorical_crossentropy(output, target, from_logits=False):
     return T.nnet.categorical_crossentropy(output, target)
 
 
-def sparse_categorical_crossentropy(output, target, from_logits=False):
+def sparse_categorical_crossentropy(target, output, from_logits=False):
     target = T.cast(T.flatten(target), 'int32')
     target = T.extra_ops.to_one_hot(target, nb_class=output.shape[-1])
     target = reshape(target, shape(output))
     return categorical_crossentropy(output, target, from_logits)
 
 
-def binary_crossentropy(output, target, from_logits=False):
+def binary_crossentropy(target, output, from_logits=False):
     if from_logits:
         output = T.nnet.sigmoid(output)
     # avoid numerical instability with _EPSILON clipping
@@ -1460,26 +1572,42 @@ def dropout(x, level, noise_shape=None, seed=None):
     return x
 
 
-def l2_normalize(x, axis):
-    norm = T.sqrt(T.sum(T.square(x), axis=axis, keepdims=True))
+def l2_normalize(x, axis, epsilon=1e-12):
+    square_sum = T.sum(T.square(x), axis=axis, keepdims=True)
+    norm = T.sqrt(T.maximum(square_sum, epsilon))
     return x / norm
 
 
 def in_top_k(predictions, targets, k):
-    """Returns whether the `targets` are in the top `k` `predictions`
+    """Returns whether the `targets` are in the top `k` `predictions`.
 
     # Arguments
-        predictions: A tensor of shape batch_size x classess and type float32.
-        targets: A tensor of shape batch_size and type int32 or int64.
-        k: An int, number of top elements to consider.
+        predictions: A tensor of shape `(batch_size, classes)` and type `float32`.
+        targets: A 1D tensor of length `batch_size` and type `int32` or `int64`.
+        k: An `int`, number of top elements to consider.
 
     # Returns
-        A tensor of shape batch_size and type int. output_i is 1 if
-        targets_i is within top-k values of predictions_i
+        A 1D tensor of length `batch_size` and type `bool`.
+        `output[i]` is `True` if `predictions[i, targets[i]]` is within top-`k`
+        values of `predictions[i]`.
     """
-    predictions_top_k = T.argsort(predictions)[:, -k:]
-    result, _ = theano.map(lambda prediction, target: any(equal(prediction, target)), sequences=[predictions_top_k, targets])
-    return result
+    # handle k < 1 and k >= predictions.shape[1] cases to match TF behavior
+    if k < 1:
+        # dtype='bool' is only available since Theano 0.9.0
+        try:
+            return T.zeros_like(targets, dtype='bool')
+        except TypeError:
+            return T.zeros_like(targets, dtype='int8')
+
+    if k >= int_shape(predictions)[1]:
+        try:
+            return T.ones_like(targets, dtype='bool')
+        except TypeError:
+            return T.ones_like(targets, dtype='int8')
+
+    predictions_k = T.sort(predictions)[:, -k]
+    targets_values = predictions[T.arange(targets.shape[0]), targets]
+    return T.ge(targets_values, predictions_k)
 
 
 # CONVOLUTIONS
@@ -1696,7 +1824,7 @@ def conv2d(x, kernel, strides=(1, 1), padding='valid',
         padding: string, "same" or "valid".
         data_format: "channels_last" or "channels_first".
             Whether to use Theano or TensorFlow data format
-        in inputs/kernels/ouputs.
+        in inputs/kernels/outputs.
     """
     if data_format is None:
         data_format = image_data_format()
@@ -1740,7 +1868,10 @@ def conv2d_transpose(x, kernel, output_shape, strides=(1, 1),
         padding: string, "same" or "valid".
         data_format: "channels_last" or "channels_first".
             Whether to use Theano or TensorFlow data format
-        in inputs/kernels/ouputs.
+        in inputs/kernels/outputs.
+
+    # Raises
+        ValueError: if using an even kernel size with padding 'same'.
     """
     flip_filters = False
     if data_format is None:
@@ -1759,6 +1890,12 @@ def conv2d_transpose(x, kernel, output_shape, strides=(1, 1),
     else:
         # Will only work if `kernel` is a shared variable.
         kernel_shape = kernel.eval().shape
+
+    if padding == 'same' and kernel_shape[0] % 2 == 0:
+        raise ValueError('In `Conv2DTranspose`, with padding mode `same`, '
+                         'even kernel sizes are only supported with Tensorflow. '
+                         'With Theano, set `kernel_size` to an odd number.')
+
     kernel_shape = _preprocess_conv2d_filter_shape(kernel_shape, data_format)
 
     x = _preprocess_conv2d_input(x, data_format)
@@ -1781,6 +1918,11 @@ def separable_conv2d(x, depthwise_kernel, pointwise_kernel, strides=(1, 1),
     raise NotImplementedError
 
 
+def depthwise_conv2d(x, depthwise_kernel, strides=(1, 1), padding='valid',
+                     data_format=None, dilation_rate=(1, 1)):
+    raise NotImplementedError
+
+
 def conv3d(x, kernel, strides=(1, 1, 1),
            padding='valid', data_format=None,
            dilation_rate=(1, 1, 1)):
@@ -1792,7 +1934,7 @@ def conv3d(x, kernel, strides=(1, 1, 1),
         padding: string, "same" or "valid".
         data_format: "channels_last" or "channels_first".
             Whether to use Theano or TensorFlow data format
-        in inputs/kernels/ouputs.
+        in inputs/kernels/outputs.
     """
     if data_format is None:
         data_format = image_data_format()
@@ -1825,6 +1967,63 @@ def conv3d(x, kernel, strides=(1, 1, 1),
     return conv_out
 
 
+def conv3d_transpose(x, kernel, output_shape, strides=(1, 1, 1),
+                     padding='valid', data_format=None):
+    """3D deconvolution (transposed convolution).
+
+    # Arguments
+        kernel: kernel tensor.
+        output_shape: desired dimensions of output.
+        strides: strides tuple.
+        padding: string, "same" or "valid".
+        data_format: "channels_last" or "channels_first".
+            Whether to use Theano or TensorFlow data format
+        in inputs/kernels/outputs.
+
+    # Raises
+        ValueError: if using an even kernel size with padding 'same'.
+    """
+    flip_filters = False
+    if data_format is None:
+        data_format = image_data_format()
+    if data_format not in {'channels_first', 'channels_last'}:
+        raise ValueError('Unknown data_format ' + data_format)
+
+    if data_format == 'channels_last':
+        output_shape = (output_shape[0],
+                        output_shape[4],
+                        output_shape[1],
+                        output_shape[2],
+                        output_shape[3])
+
+    if hasattr(kernel, '_keras_shape'):
+        kernel_shape = kernel._keras_shape
+    else:
+        # Will only work if `kernel` is a shared variable.
+        kernel_shape = kernel.eval().shape
+
+    if padding == 'same' and kernel_shape[0] % 2 == 0:
+        raise ValueError('In `Conv3DTranspose`, with padding mode `same`, '
+                         'even kernel sizes are only supported with Tensorflow. '
+                         'With Theano, set `kernel_size` to an odd number.')
+
+    kernel_shape = _preprocess_conv3d_filter_shape(kernel_shape, data_format)
+
+    x = _preprocess_conv3d_input(x, data_format)
+    kernel = _preprocess_conv3d_kernel(kernel, data_format)
+
+    th_padding = _preprocess_padding(padding)
+    op = T.nnet.abstract_conv.AbstractConv3d_gradInputs(imshp=None,
+                                                        kshp=kernel_shape,
+                                                        subsample=strides,
+                                                        border_mode=th_padding,
+                                                        filter_flip=not flip_filters)
+    conv_out = op(kernel, x, output_shape[2:])
+    conv_out = _postprocess_conv3d_output(conv_out, x, padding,
+                                          kernel_shape, strides, data_format)
+    return conv_out
+
+
 def pool2d(x, pool_size, strides=(1, 1), padding='valid',
            data_format=None, pool_mode='max'):
     if data_format is None:
@@ -1843,9 +2042,6 @@ def pool2d(x, pool_size, strides=(1, 1), padding='valid',
     else:
         raise ValueError('Invalid border mode:', padding)
 
-    if data_format not in {'channels_first', 'channels_last'}:
-        raise ValueError('Unknown data_format:', data_format)
-
     if data_format == 'channels_last':
         x = x.dimshuffle((0, 3, 1, 2))
 
@@ -1855,10 +2051,14 @@ def pool2d(x, pool_size, strides=(1, 1), padding='valid',
                                 pad=pad,
                                 mode='max')
     elif pool_mode == 'avg':
+        if padding == 'same':
+            th_avg_pool_mode = 'average_inc_pad'
+        elif padding == 'valid':
+            th_avg_pool_mode = 'average_exc_pad'
         pool_out = pool.pool_2d(x, ws=pool_size, stride=strides,
                                 ignore_border=True,
                                 pad=pad,
-                                mode='average_exc_pad')
+                                mode=th_avg_pool_mode)
     else:
         raise ValueError('Invalid pooling mode:', pool_mode)
     if padding == 'same':
@@ -1889,8 +2089,6 @@ def pool3d(x, pool_size, strides=(1, 1, 1), padding='valid',
         padding = (0, 0, 0)
     else:
         raise ValueError('Invalid padding:', padding)
-    if data_format not in {'channels_first', 'channels_last'}:
-        raise ValueError('Unknown data_format:', data_format)
 
     if data_format == 'channels_last':
         x = x.dimshuffle((0, 4, 1, 2, 3))
@@ -1928,21 +2126,44 @@ def bias_add(x, bias, data_format=None):
         data_format = image_data_format()
     if data_format not in {'channels_first', 'channels_last'}:
         raise ValueError('Unknown data_format ' + str(data_format))
+    if ndim(bias) != 1 and ndim(bias) != ndim(x) - 1:
+        raise ValueError('Unexpected bias dimensions %d, '
+                         'expect to be 1 or %d dimensions'
+                         % (ndim(bias), ndim(x) - 1))
+    bias_shape = tuple(bias.shape)
     if ndim(x) == 5:
         if data_format == 'channels_first':
-            x += reshape(bias, (1, bias.shape[0], 1, 1, 1))
+            if ndim(bias) == 1:
+                x += reshape(bias, (1, bias_shape[0], 1, 1, 1))
+            else:
+                x += reshape(bias, (1, bias_shape[3]) + bias_shape[:3])
         elif data_format == 'channels_last':
-            x += reshape(bias, (1, 1, 1, 1, bias.shape[0]))
+            if ndim(bias) == 1:
+                x += reshape(bias, (1, 1, 1, 1, bias_shape[0]))
+            else:
+                x += reshape(bias, (1,) + bias_shape)
     elif ndim(x) == 4:
         if data_format == 'channels_first':
-            x += reshape(bias, (1, bias.shape[0], 1, 1))
+            if ndim(bias) == 1:
+                x += reshape(bias, (1, bias_shape[0], 1, 1))
+            else:
+                x += reshape(bias, (1, bias_shape[2]) + bias_shape[:2])
         elif data_format == 'channels_last':
-            x += reshape(bias, (1, 1, 1, bias.shape[0]))
+            if ndim(bias) == 1:
+                x += reshape(bias, (1, 1, 1, bias_shape[0]))
+            else:
+                x += reshape(bias, (1,) + bias_shape)
     elif ndim(x) == 3:
         if data_format == 'channels_first':
-            x += reshape(bias, (1, bias.shape[0], 1))
+            if ndim(bias) == 1:
+                x += reshape(bias, (1, bias_shape[0], 1))
+            else:
+                x += reshape(bias, (1, bias_shape[1], bias_shape[0]))
         elif data_format == 'channels_last':
-            x += reshape(bias, (1, 1, bias.shape[0]))
+            if ndim(bias) == 1:
+                x += reshape(bias, (1, 1, bias_shape[0]))
+            else:
+                x += reshape(bias, (1,) + bias_shape)
     else:
         x += bias
     return x
@@ -2099,7 +2320,7 @@ def ctc_batch_cost(y_true, y_pred, input_length, label_length):
 
 # HIGH ORDER FUNCTIONS
 
-def map_fn(fn, elems, name=None):
+def map_fn(fn, elems, name=None, dtype=None):
     """Map the function fn over the elements elems and return the outputs.
 
     # Arguments
@@ -2133,9 +2354,8 @@ def foldl(fn, elems, initializer=None, name=None):
 
     # We need to change the order of the arguments because theano accepts x as
     # first parameter and accumulator as second
-    fn2 = lambda x, acc: fn(acc, x)
-
-    return theano.foldl(fn2, elems, initializer, name=name)[0]
+    return theano.foldl(lambda x, acc: fn(acc, x),
+                        elems, initializer, name=name)[0]
 
 
 def foldr(fn, elems, initializer=None, name=None):
@@ -2157,6 +2377,73 @@ def foldr(fn, elems, initializer=None, name=None):
 
     # We need to change the order of the arguments because theano accepts x as
     # first parameter and accumulator as second
-    fn2 = lambda x, acc: fn(acc, x)
+    return theano.foldr(lambda x, acc: fn(acc, x),
+                        elems, initializer, name=name)[0]
 
-    return theano.foldr(fn2, elems, initializer, name=name)[0]
+
+def local_conv1d(inputs, kernel, kernel_size, strides, data_format=None):
+    if data_format is None:
+        data_format = image_data_format()
+    if data_format not in {'channels_first', 'channels_last'}:
+        raise ValueError('Unknown data_format ' + str(data_format))
+
+    stride = strides[0]
+    kernel_shape = int_shape(kernel)
+    output_length, feature_dim, filters = kernel_shape
+
+    xs = []
+    for i in range(output_length):
+        slice_length = slice(i * stride,
+                             i * stride + kernel_size[0])
+        xs.append(reshape(inputs[:, slice_length, :],
+                          (1, -1, feature_dim)))
+    x_aggregate = concatenate(xs, axis=0)
+    # Shape: `(output_length, batch_size, filters)`.
+    output = batch_dot(x_aggregate, kernel)
+    return permute_dimensions(output, (1, 0, 2))
+
+
+def local_conv2d(inputs, kernel, kernel_size, strides, output_shape, data_format=None):
+    if data_format is None:
+        data_format = image_data_format()
+    if data_format not in {'channels_first', 'channels_last'}:
+        raise ValueError('Unknown data_format ' + str(data_format))
+
+    stride_row, stride_col = strides
+    output_row, output_col = output_shape
+    kernel_shape = int_shape(kernel)
+    _, feature_dim, filters = kernel_shape
+
+    if data_format == 'channels_first':
+        output = []
+        for i in range(output_row):
+            for j in range(output_col):
+                slice_row = slice(i * stride_row,
+                                  i * stride_row + kernel_size[0])
+                slice_col = slice(j * stride_col,
+                                  j * stride_col + kernel_size[1])
+                x_flatten = reshape(inputs[:, :, slice_row, slice_col],
+                                    (1, -1, feature_dim))
+                output.append(dot(x_flatten,
+                                  kernel[i * output_col + j, :, :]))
+        output = concatenate(output, axis=0)
+        output = reshape(output,
+                         (output_row, output_col, -1, filters))
+        output = permute_dimensions(output, (2, 3, 0, 1))
+    else:
+        xs = []
+        for i in range(output_row):
+            for j in range(output_col):
+                slice_row = slice(i * stride_row,
+                                  i * stride_row + kernel_size[0])
+                slice_col = slice(j * stride_col,
+                                  j * stride_col + kernel_size[1])
+                xs.append(reshape(inputs[:, slice_row, slice_col, :],
+                                  (1, -1, feature_dim)))
+
+        x_aggregate = concatenate(xs, axis=0)
+        output = batch_dot(x_aggregate, kernel)
+        output = reshape(output,
+                         (output_row, output_col, -1, filters))
+        output = permute_dimensions(output, (2, 0, 1, 3))
+    return output

keras/callbacks.py

@@ -3,6 +3,7 @@ from __future__ import print_function
 
 import os
 import csv
+import six
 
 import numpy as np
 import time
@@ -22,6 +23,7 @@ except ImportError:
 
 if K.backend() == 'tensorflow':
     import tensorflow as tf
+    from tensorflow.contrib.tensorboard.plugins import projector
 
 
 class CallbackList(object):
@@ -225,6 +227,21 @@ class BaseLogger(Callback):
                     logs[k] = self.totals[k] / self.seen
 
 
+class TerminateOnNaN(Callback):
+    """Callback that terminates training when a NaN loss is encountered."""
+
+    def __init__(self):
+        super(TerminateOnNaN, self).__init__()
+
+    def on_batch_end(self, batch, logs=None):
+        logs = logs or {}
+        loss = logs.get('loss')
+        if loss is not None:
+            if np.isnan(loss) or np.isinf(loss):
+                print('Batch %d: Invalid loss, terminating training' % (batch))
+                self.model.stop_training = True
+
+
 class ProgbarLogger(Callback):
     """Callback that prints metrics to stdout.
 
@@ -465,14 +482,19 @@ class EarlyStopping(Callback):
             self.min_delta *= -1
 
     def on_train_begin(self, logs=None):
-        self.wait = 0  # Allow instances to be re-used
+        # Allow instances to be re-used
+        self.wait = 0
+        self.stopped_epoch = 0
         self.best = np.Inf if self.monitor_op == np.less else -np.Inf
 
     def on_epoch_end(self, epoch, logs=None):
         current = logs.get(self.monitor)
         if current is None:
-            warnings.warn('Early stopping requires %s available!' %
-                          (self.monitor), RuntimeWarning)
+            warnings.warn(
+                'Early stopping conditioned on metric `%s` '
+                'which is not available. Available metrics are: %s' %
+                (self.monitor, ','.join(list(logs.keys()))), RuntimeWarning
+            )
 
         if self.monitor_op(current - self.min_delta, self.best):
             self.best = current
@@ -501,9 +523,6 @@ class RemoteMonitor(Callback):
         path: String; path relative to `root` to which the events will be sent.
         field: String; JSON field under which the data will be stored.
         headers: Dictionary; optional custom HTTP headers.
-            Defaults to:
-            `{'Accept': 'application/json',
-              'Content-Type': 'application/json'}`
     """
 
     def __init__(self,
@@ -512,9 +531,7 @@ class RemoteMonitor(Callback):
                  field='data',
                  headers=None):
         super(RemoteMonitor, self).__init__()
-        if headers is None:
-            headers = {'Accept': 'application/json',
-                       'Content-Type': 'application/json'}
+
         self.root = root
         self.path = path
         self.field = field
@@ -564,38 +581,56 @@ class LearningRateScheduler(Callback):
 class TensorBoard(Callback):
     """Tensorboard basic visualizations.
 
+    [TensorBoard](https://www.tensorflow.org/get_started/summaries_and_tensorboard)
+    is a visualization tool provided with TensorFlow.
+
     This callback writes a log for TensorBoard, which allows
     you to visualize dynamic graphs of your training and test
     metrics, as well as activation histograms for the different
     layers in your model.
 
-    TensorBoard is a visualization tool provided with TensorFlow.
-
     If you have installed TensorFlow with pip, you should be able
     to launch TensorBoard from the command line:
     ```
     tensorboard --logdir=/full_path_to_your_logs
     ```
-    You can find more information about TensorBoard
-    [here](https://www.tensorflow.org/versions/master/how_tos/summaries_and_tensorboard/index.html).
 
     # Arguments
         log_dir: the path of the directory where to save the log
-            files to be parsed by Tensorboard.
+            files to be parsed by TensorBoard.
         histogram_freq: frequency (in epochs) at which to compute activation
-            histograms for the layers of the model. If set to 0,
-            histograms won't be computed.
-        write_graph: whether to visualize the graph in Tensorboard.
+            and weight histograms for the layers of the model. If set to 0,
+            histograms won't be computed. Validation data (or split) must be
+            specified for histogram visualizations.
+        write_graph: whether to visualize the graph in TensorBoard.
             The log file can become quite large when
             write_graph is set to True.
+        write_grads: whether to visualize gradient histograms in TensorBoard.
+            `histogram_freq` must be greater than 0.
+        batch_size: size of batch of inputs to feed to the network
+            for histograms computation.
         write_images: whether to write model weights to visualize as
-            image in Tensorboard.
+            image in TensorBoard.
+        embeddings_freq: frequency (in epochs) at which selected embedding
+            layers will be saved.
+        embeddings_layer_names: a list of names of layers to keep eye on. If
+            None or empty list all the embedding layer will be watched.
+        embeddings_metadata: a dictionary which maps layer name to a file name
+            in which metadata for this embedding layer is saved. See the
+            [details](https://www.tensorflow.org/how_tos/embedding_viz/#metadata_optional)
+            about metadata files format. In case if the same metadata file is
+            used for all embedding layers, string can be passed.
     """
 
     def __init__(self, log_dir='./logs',
                  histogram_freq=0,
+                 batch_size=32,
                  write_graph=True,
-                 write_images=False):
+                 write_grads=False,
+                 write_images=False,
+                 embeddings_freq=0,
+                 embeddings_layer_names=None,
+                 embeddings_metadata=None):
         super(TensorBoard, self).__init__()
         if K.backend() != 'tensorflow':
             raise RuntimeError('TensorBoard callback only works '
@@ -604,7 +639,12 @@ class TensorBoard(Callback):
         self.histogram_freq = histogram_freq
         self.merged = None
         self.write_graph = write_graph
+        self.write_grads = write_grads
         self.write_images = write_images
+        self.embeddings_freq = embeddings_freq
+        self.embeddings_layer_names = embeddings_layer_names
+        self.embeddings_metadata = embeddings_metadata or {}
+        self.batch_size = batch_size
 
     def set_model(self, model):
         self.model = model
@@ -613,16 +653,45 @@ class TensorBoard(Callback):
             for layer in self.model.layers:
 
                 for weight in layer.weights:
-                    tf.summary.histogram(weight.name, weight)
+                    mapped_weight_name = weight.name.replace(':', '_')
+                    tf.summary.histogram(mapped_weight_name, weight)
+                    if self.write_grads:
+                        grads = model.optimizer.get_gradients(model.total_loss,
+                                                              weight)
+                        tf.summary.histogram('{}_grad'.format(mapped_weight_name), grads)
                     if self.write_images:
                         w_img = tf.squeeze(weight)
-                        shape = w_img.get_shape()
-                        if len(shape) > 1 and shape[0] > shape[1]:
-                            w_img = tf.transpose(w_img)
-                        if len(shape) == 1:
-                            w_img = tf.expand_dims(w_img, 0)
-                        w_img = tf.expand_dims(tf.expand_dims(w_img, 0), -1)
-                        tf.summary.image(weight.name, w_img)
+                        shape = K.int_shape(w_img)
+                        if len(shape) == 2:  # dense layer kernel case
+                            if shape[0] > shape[1]:
+                                w_img = tf.transpose(w_img)
+                                shape = K.int_shape(w_img)
+                            w_img = tf.reshape(w_img, [1,
+                                                       shape[0],
+                                                       shape[1],
+                                                       1])
+                        elif len(shape) == 3:  # convnet case
+                            if K.image_data_format() == 'channels_last':
+                                # switch to channels_first to display
+                                # every kernel as a separate image
+                                w_img = tf.transpose(w_img, perm=[2, 0, 1])
+                                shape = K.int_shape(w_img)
+                            w_img = tf.reshape(w_img, [shape[0],
+                                                       shape[1],
+                                                       shape[2],
+                                                       1])
+                        elif len(shape) == 1:  # bias case
+                            w_img = tf.reshape(w_img, [1,
+                                                       shape[0],
+                                                       1,
+                                                       1])
+                        else:
+                            # not possible to handle 3D convnets etc.
+                            continue
+
+                        shape = K.int_shape(w_img)
+                        assert len(shape) == 4 and shape[-1] in [1, 3, 4]
+                        tf.summary.image(mapped_weight_name, w_img)
 
                 if hasattr(layer, 'output'):
                     tf.summary.histogram('{}_out'.format(layer.name),
@@ -635,24 +704,76 @@ class TensorBoard(Callback):
         else:
             self.writer = tf.summary.FileWriter(self.log_dir)
 
+        if self.embeddings_freq:
+            embeddings_layer_names = self.embeddings_layer_names
+
+            if not embeddings_layer_names:
+                embeddings_layer_names = [layer.name for layer in self.model.layers
+                                          if type(layer).__name__ == 'Embedding']
+
+            embeddings = {layer.name: layer.weights[0]
+                          for layer in self.model.layers
+                          if layer.name in embeddings_layer_names}
+
+            self.saver = tf.train.Saver(list(embeddings.values()))
+
+            embeddings_metadata = {}
+
+            if not isinstance(self.embeddings_metadata, str):
+                embeddings_metadata = self.embeddings_metadata
+            else:
+                embeddings_metadata = {layer_name: self.embeddings_metadata
+                                       for layer_name in embeddings.keys()}
+
+            config = projector.ProjectorConfig()
+            self.embeddings_ckpt_path = os.path.join(self.log_dir,
+                                                     'keras_embedding.ckpt')
+
+            for layer_name, tensor in embeddings.items():
+                embedding = config.embeddings.add()
+                embedding.tensor_name = tensor.name
+
+                if layer_name in embeddings_metadata:
+                    embedding.metadata_path = embeddings_metadata[layer_name]
+
+            projector.visualize_embeddings(self.writer, config)
+
     def on_epoch_end(self, epoch, logs=None):
         logs = logs or {}
 
         if self.validation_data and self.histogram_freq:
             if epoch % self.histogram_freq == 0:
-                # TODO: implement batched calls to sess.run
-                # (current call will likely go OOM on GPU)
+
+                val_data = self.validation_data
+                tensors = (self.model.inputs +
+                           self.model.targets +
+                           self.model.sample_weights)
+
                 if self.model.uses_learning_phase:
-                    cut_v_data = len(self.model.inputs)
-                    val_data = self.validation_data[:cut_v_data] + [0]
-                    tensors = self.model.inputs + [K.learning_phase()]
-                else:
-                    val_data = self.validation_data
-                    tensors = self.model.inputs
-                feed_dict = dict(zip(tensors, val_data))
-                result = self.sess.run([self.merged], feed_dict=feed_dict)
-                summary_str = result[0]
-                self.writer.add_summary(summary_str, epoch)
+                    tensors += [K.learning_phase()]
+
+                assert len(val_data) == len(tensors)
+                val_size = val_data[0].shape[0]
+                i = 0
+                while i < val_size:
+                    step = min(self.batch_size, val_size - i)
+                    batch_val = []
+                    batch_val.append(val_data[0][i:i + step])
+                    batch_val.append(val_data[1][i:i + step])
+                    batch_val.append(val_data[2][i:i + step])
+                    if self.model.uses_learning_phase:
+                        batch_val.append(val_data[3])
+                    feed_dict = dict(zip(tensors, batch_val))
+                    result = self.sess.run([self.merged], feed_dict=feed_dict)
+                    summary_str = result[0]
+                    self.writer.add_summary(summary_str, epoch)
+                    i += self.batch_size
+
+        if self.embeddings_freq and self.embeddings_ckpt_path:
+            if epoch % self.embeddings_freq == 0:
+                self.saver.save(self.sess,
+                                self.embeddings_ckpt_path,
+                                epoch)
 
         for name, value in logs.items():
             if name in ['batch', 'size']:
@@ -678,9 +799,9 @@ class ReduceLROnPlateau(Callback):
 
     # Example
         ```python
-            reduce_lr = ReduceLROnPlateau(monitor='val_loss', factor=0.2,
-                                          patience=5, min_lr=0.001)
-            model.fit(X_train, Y_train, callbacks=[reduce_lr])
+        reduce_lr = ReduceLROnPlateau(monitor='val_loss', factor=0.2,
+                                      patience=5, min_lr=0.001)
+        model.fit(X_train, Y_train, callbacks=[reduce_lr])
         ```
 
     # Arguments
@@ -752,8 +873,12 @@ class ReduceLROnPlateau(Callback):
         logs['lr'] = K.get_value(self.model.optimizer.lr)
         current = logs.get(self.monitor)
         if current is None:
-            warnings.warn('Learning Rate Plateau Reducing requires %s available!' %
-                          self.monitor, RuntimeWarning)
+            warnings.warn(
+                'Reduce LR on plateau conditioned on metric `%s` '
+                'which is not available. Available metrics are: %s' %
+                (self.monitor, ','.join(list(logs.keys()))), RuntimeWarning
+            )
+
         else:
             if self.in_cooldown():
                 self.cooldown_counter -= 1
@@ -787,8 +912,8 @@ class CSVLogger(Callback):
 
     # Example
         ```python
-            csv_logger = CSVLogger('training.log')
-            model.fit(X_train, Y_train, callbacks=[csv_logger])
+        csv_logger = CSVLogger('training.log')
+        model.fit(X_train, Y_train, callbacks=[csv_logger])
         ```
 
     # Arguments
@@ -805,23 +930,26 @@ class CSVLogger(Callback):
         self.writer = None
         self.keys = None
         self.append_header = True
+        self.file_flags = 'b' if six.PY2 and os.name == 'nt' else ''
         super(CSVLogger, self).__init__()
 
     def on_train_begin(self, logs=None):
         if self.append:
             if os.path.exists(self.filename):
-                with open(self.filename) as f:
+                with open(self.filename, 'r' + self.file_flags) as f:
                     self.append_header = not bool(len(f.readline()))
-            self.csv_file = open(self.filename, 'a')
+            self.csv_file = open(self.filename, 'a' + self.file_flags)
         else:
-            self.csv_file = open(self.filename, 'w')
+            self.csv_file = open(self.filename, 'w' + self.file_flags)
 
     def on_epoch_end(self, epoch, logs=None):
         logs = logs or {}
 
         def handle_value(k):
             is_zero_dim_ndarray = isinstance(k, np.ndarray) and k.ndim == 0
-            if isinstance(k, Iterable) and not is_zero_dim_ndarray:
+            if isinstance(k, six.string_types):
+                return k
+            elif isinstance(k, Iterable) and not is_zero_dim_ndarray:
                 return '"[%s]"' % (', '.join(map(str, k)))
             else:
                 return k
@@ -848,11 +976,12 @@ class CSVLogger(Callback):
 
 
 class LambdaCallback(Callback):
-    """Callback for creating simple, custom callbacks on-the-fly.
+    r"""Callback for creating simple, custom callbacks on-the-fly.
 
     This callback is constructed with anonymous functions that will be called
     at the appropriate time. Note that the callbacks expects positional
     arguments, as:
+
      - `on_epoch_begin` and `on_epoch_end` expect two positional arguments:
         `epoch`, `logs`
      - `on_batch_begin` and `on_batch_end` expect two positional arguments:
@@ -874,12 +1003,15 @@ class LambdaCallback(Callback):
         batch_print_callback = LambdaCallback(
             on_batch_begin=lambda batch,logs: print(batch))
 
-        # Plot the loss after every epoch.
-        import numpy as np
-        import matplotlib.pyplot as plt
-        plot_loss_callback = LambdaCallback(
-            on_epoch_end=lambda epoch, logs: plt.plot(np.arange(epoch),
-                                                      logs['loss']))
+        # Stream the epoch loss to a file in JSON format. The file content
+        # is not well-formed JSON but rather has a JSON object per line.
+        import json
+        json_log = open('loss_log.json', mode='wt', buffering=1)
+        json_logging_callback = LambdaCallback(
+            on_epoch_end=lambda epoch, logs: json_log.write(
+                json.dumps({'epoch': epoch, 'loss': logs['loss']}) + '\n'),
+            on_train_end=lambda logs: json_log.close()
+        )
 
         # Terminate some processes after having finished model training.
         processes = ...
@@ -889,7 +1021,7 @@ class LambdaCallback(Callback):
 
         model.fit(...,
                   callbacks=[batch_print_callback,
-                             plot_loss_callback,
+                             json_logging_callback,
                              cleanup_callback])
         ```
     """

keras/constraints.py

@@ -27,7 +27,7 @@ class MaxNorm(Constraint):
             has shape `(input_dim, output_dim)`,
             set `axis` to `0` to constrain each weight vector
             of length `(input_dim,)`.
-            In a `Convolution2D` layer with `data_format="channels_last"`,
+            In a `Conv2D` layer with `data_format="channels_last"`,
             the weight tensor has shape
             `(rows, cols, input_depth, output_depth)`,
             set `axis` to `[0, 1, 2]`
@@ -58,7 +58,7 @@ class NonNeg(Constraint):
     """
 
     def __call__(self, w):
-        w *= K.cast(w >= 0., K.floatx())
+        w *= K.cast(K.greater_equal(w, 0.), K.floatx())
         return w
 
 
@@ -71,7 +71,7 @@ class UnitNorm(Constraint):
             has shape `(input_dim, output_dim)`,
             set `axis` to `0` to constrain each weight vector
             of length `(input_dim,)`.
-            In a `Convolution2D` layer with `data_format="channels_last"`,
+            In a `Conv2D` layer with `data_format="channels_last"`,
             the weight tensor has shape
             `(rows, cols, input_depth, output_depth)`,
             set `axis` to `[0, 1, 2]`
@@ -112,7 +112,7 @@ class MinMaxNorm(Constraint):
             has shape `(input_dim, output_dim)`,
             set `axis` to `0` to constrain each weight vector
             of length `(input_dim,)`.
-            In a `Convolution2D` layer with `dim_ordering="tf"`,
+            In a `Conv2D` layer with `data_format="channels_last"`,
             the weight tensor has shape
             `(rows, cols, input_depth, output_depth)`,
             set `axis` to `[0, 1, 2]`

keras/datasets/boston_housing.py

@@ -16,7 +16,9 @@ def load_data(path='boston_housing.npz', seed=113, test_split=0.2):
         Tuple of Numpy arrays: `(x_train, y_train), (x_test, y_test)`.
     """
     assert 0 <= test_split < 1
-    path = get_file(path, origin='https://s3.amazonaws.com/keras-datasets/boston_housing.npz')
+    path = get_file(path,
+                    origin='https://s3.amazonaws.com/keras-datasets/boston_housing.npz',
+                    file_hash='f553886a1f8d56431e820c5b82552d9d95cfcb96d1e678153f8839538947dff5')
     f = np.load(path)
     x = f['x']
     y = f['y']

keras/datasets/cifar100.py

@@ -19,7 +19,7 @@ def load_data(label_mode='fine'):
         ValueError: in case of invalid `label_mode`.
     """
     if label_mode not in ['fine', 'coarse']:
-        raise ValueError('label_mode must be one of "fine" "coarse".')
+        raise ValueError('`label_mode` must be one of `"fine"`, `"coarse"`.')
 
     dirname = 'cifar-100-python'
     origin = 'http://www.cs.toronto.edu/~kriz/cifar-100-python.tar.gz'

keras/datasets/imdb.py

@@ -1,5 +1,6 @@
 from __future__ import absolute_import
 from ..utils.data_utils import get_file
+from ..preprocessing.sequence import _remove_long_seq
 from six.moves import zip
 import numpy as np
 import json
@@ -16,7 +17,7 @@ def load_data(path='imdb.npz', num_words=None, skip_top=0,
         num_words: max number of words to include. Words are ranked
             by how often they occur (in the training set) and only
             the most frequent words are kept
-        skip_top: skip the top N most frequently occuring words
+        skip_top: skip the top N most frequently occurring words
             (which may not be informative).
         maxlen: truncate sequences after this length.
         seed: random seed for sample shuffling.
@@ -47,14 +48,10 @@ def load_data(path='imdb.npz', num_words=None, skip_top=0,
     if kwargs:
         raise TypeError('Unrecognized keyword arguments: ' + str(kwargs))
 
-    path = get_file(path,
-                    origin='https://s3.amazonaws.com/text-datasets/imdb.npz')
-    f = np.load(path)
-    x_train = f['x_train']
-    labels_train = f['y_train']
-    x_test = f['x_test']
-    labels_test = f['y_test']
-    f.close()
+    path = get_file(path, origin='https://s3.amazonaws.com/text-datasets/imdb.npz')
+    with np.load(path) as f:
+        x_train, labels_train = f['x_train'], f['y_train']
+        x_test, labels_test = f['x_test'], f['y_test']
 
     np.random.seed(seed)
     np.random.shuffle(x_train)
@@ -75,18 +72,11 @@ def load_data(path='imdb.npz', num_words=None, skip_top=0,
         xs = [[w + index_from for w in x] for x in xs]
 
     if maxlen:
-        new_xs = []
-        new_labels = []
-        for x, y in zip(xs, labels):
-            if len(x) < maxlen:
-                new_xs.append(x)
-                new_labels.append(y)
-        xs = new_xs
-        labels = new_labels
-    if not xs:
-        raise ValueError('After filtering for sequences shorter than maxlen=' +
-                         str(maxlen) + ', no sequence was kept. '
-                         'Increase maxlen.')
+        xs, labels = _remove_long_seq(maxlen, xs, labels)
+        if not xs:
+            raise ValueError('After filtering for sequences shorter than maxlen=' +
+                             str(maxlen) + ', no sequence was kept. '
+                             'Increase maxlen.')
     if not num_words:
         num_words = max([max(x) for x in xs])
 
@@ -94,22 +84,13 @@ def load_data(path='imdb.npz', num_words=None, skip_top=0,
     # reserve 'index_from' (=3 by default) characters:
     # 0 (padding), 1 (start), 2 (OOV)
     if oov_char is not None:
-        xs = [[oov_char if (w >= num_words or w < skip_top) else w for w in x] for x in xs]
+        xs = [[w if (skip_top <= w < num_words) else oov_char for w in x] for x in xs]
     else:
-        new_xs = []
-        for x in xs:
-            nx = []
-            for w in x:
-                if w >= num_words or w < skip_top:
-                    nx.append(w)
-            new_xs.append(nx)
-        xs = new_xs
+        xs = [[w for w in x if (skip_top <= w < num_words)] for x in xs]
 
-    x_train = np.array(xs[:len(x_train)])
-    y_train = np.array(labels[:len(x_train)])
-
-    x_test = np.array(xs[len(x_train):])
-    y_test = np.array(labels[len(x_train):])
+    idx = len(x_train)
+    x_train, y_train = np.array(xs[:idx]), np.array(labels[:idx])
+    x_test, y_test = np.array(xs[idx:]), np.array(labels[idx:])
 
     return (x_train, y_train), (x_test, y_test)
 

keras/datasets/mnist.py

@@ -14,9 +14,7 @@ def load_data(path='mnist.npz'):
     """
     path = get_file(path, origin='https://s3.amazonaws.com/img-datasets/mnist.npz')
     f = np.load(path)
-    x_train = f['x_train']
-    y_train = f['y_train']
-    x_test = f['x_test']
-    y_test = f['y_test']
+    x_train, y_train = f['x_train'], f['y_train']
+    x_test, y_test = f['x_test'], f['y_test']
     f.close()
     return (x_train, y_train), (x_test, y_test)

keras/datasets/reuters.py

@@ -1,6 +1,7 @@
 # -*- coding: utf-8 -*-
 from __future__ import absolute_import
 from ..utils.data_utils import get_file
+from ..preprocessing.sequence import _remove_long_seq
 from six.moves import zip
 import numpy as np
 import json
@@ -17,7 +18,7 @@ def load_data(path='reuters.npz', num_words=None, skip_top=0,
         num_words: max number of words to include. Words are ranked
             by how often they occur (in the training set) and only
             the most frequent words are kept
-        skip_top: skip the top N most frequently occuring words
+        skip_top: skip the top N most frequently occurring words
             (which may not be informative).
         maxlen: truncate sequences after this length.
         test_split: Fraction of the dataset to be used as test data.
@@ -46,10 +47,8 @@ def load_data(path='reuters.npz', num_words=None, skip_top=0,
         raise TypeError('Unrecognized keyword arguments: ' + str(kwargs))
 
     path = get_file(path, origin='https://s3.amazonaws.com/text-datasets/reuters.npz')
-    npzfile = np.load(path)
-    xs = npzfile['x']
-    labels = npzfile['y']
-    npzfile.close()
+    with np.load(path) as f:
+        xs, labels = f['x'], f['y']
 
     np.random.seed(seed)
     np.random.shuffle(xs)
@@ -62,14 +61,7 @@ def load_data(path='reuters.npz', num_words=None, skip_top=0,
         xs = [[w + index_from for w in x] for x in xs]
 
     if maxlen:
-        new_xs = []
-        new_labels = []
-        for x, y in zip(xs, labels):
-            if len(x) < maxlen:
-                new_xs.append(x)
-                new_labels.append(y)
-        xs = new_xs
-        labels = new_labels
+        xs, labels = _remove_long_seq(maxlen, xs, labels)
 
     if not num_words:
         num_words = max([max(x) for x in xs])
@@ -78,22 +70,13 @@ def load_data(path='reuters.npz', num_words=None, skip_top=0,
     # reserve 'index_from' (=3 by default) characters:
     # 0 (padding), 1 (start), 2 (OOV)
     if oov_char is not None:
-        xs = [[oov_char if (w >= num_words or w < skip_top) else w for w in x] for x in xs]
+        xs = [[w if (skip_top <= w < num_words) else oov_char for w in x] for x in xs]
     else:
-        new_xs = []
-        for x in xs:
-            nx = []
-            for w in x:
-                if w >= num_words or w < skip_top:
-                    nx.append(w)
-            new_xs.append(nx)
-        xs = new_xs
+        xs = [[w for w in x if (skip_top <= w < num_words)] for x in xs]
 
-    x_train = np.array(xs[:int(len(xs) * (1 - test_split))])
-    y_train = np.array(labels[:int(len(xs) * (1 - test_split))])
-
-    x_test = np.array(xs[int(len(xs) * (1 - test_split)):])
-    y_test = np.array(labels[int(len(xs) * (1 - test_split)):])
+    idx = int(len(xs) * (1 - test_split))
+    x_train, y_train = np.array(xs[:idx]), np.array(labels[:idx])
+    x_test, y_test = np.array(xs[idx:]), np.array(labels[idx:])
 
     return (x_train, y_train), (x_test, y_test)
 

keras/engine/topology.py

@@ -10,13 +10,13 @@ import warnings
 import copy
 import os
 import re
-import inspect
 from six.moves import zip
 
 from .. import backend as K
 from .. import initializers
 from ..utils.io_utils import ask_to_proceed_with_overwrite
 from ..utils.layer_utils import print_summary as print_layer_summary
+from ..utils.generic_utils import has_arg
 from ..utils import conv_utils
 from ..legacy import interfaces
 
@@ -252,7 +252,11 @@ class Layer(object):
         self._trainable_weights = []
         self._non_trainable_weights = []
         self._constraints = {}  # dict {tensor: constraint instance}
-        self.built = False
+        self._losses = []
+        self._updates = []
+        self._per_input_losses = {}
+        self._per_input_updates = {}
+        self._built = False
 
         # These lists will be filled via successive calls
         # to self._add_inbound_node().
@@ -308,6 +312,22 @@ class Layer(object):
         else:
             self._initial_weights = None
 
+    @property
+    def losses(self):
+        return self._losses
+
+    @property
+    def updates(self):
+        return self._updates
+
+    @property
+    def built(self):
+        return self._built
+
+    @built.setter
+    def built(self, value):
+        self._built = value
+
     @property
     def constraints(self):
         return self._constraints
@@ -340,28 +360,35 @@ class Layer(object):
     def non_trainable_weights(self, weights):
         self._non_trainable_weights = weights
 
-    def add_weight(self, shape, initializer,
-                   name=None,
-                   trainable=True,
+    @interfaces.legacy_add_weight_support
+    def add_weight(self,
+                   name,
+                   shape,
+                   dtype=None,
+                   initializer=None,
                    regularizer=None,
+                   trainable=True,
                    constraint=None):
         """Adds a weight variable to the layer.
 
         # Arguments
-            shape: The shape tuple of the weight.
-            initializer: An Initializer instance (callable).
             name: String, the name for the weight variable.
+            shape: The shape tuple of the weight.
+            dtype: The dtype of the weight.
+            initializer: An Initializer instance (callable).
+            regularizer: An optional Regularizer instance.
             trainable: A boolean, whether the weight should
                 be trained via backprop or not (assuming
                 that the layer itself is also trainable).
-            regularizer: An optional Regularizer instance.
             constraint: An optional Constraint instance.
 
         # Returns
             The created weight variable.
         """
         initializer = initializers.get(initializer)
-        weight = K.variable(initializer(shape), dtype=K.floatx(), name=name)
+        if dtype is None:
+            dtype = K.floatx()
+        weight = K.variable(initializer(shape), dtype=dtype, name=name)
         if regularizer is not None:
             self.add_loss(regularizer(weight))
         if constraint is not None:
@@ -386,19 +413,30 @@ class Layer(object):
             ValueError: in case of mismatch between
                 the provided inputs and the expectations of the layer.
         """
+        inputs = _to_list(inputs)
+        for x in inputs:
+            try:
+                K.is_keras_tensor(x)
+            except ValueError:
+                raise ValueError('Layer ' + self.name + ' was called with '
+                                 'an input that isn\'t a symbolic tensor. '
+                                 'Received type: ' +
+                                 str(type(x)) + '. Full input: ' +
+                                 str(inputs) + '. All inputs to the layer '
+                                 'should be tensors.')
+
         if not self.input_spec:
             return
         if not isinstance(self.input_spec, (list, tuple)):
             input_spec = _to_list(self.input_spec)
         else:
             input_spec = self.input_spec
-        inputs = _to_list(inputs)
         if len(inputs) != len(input_spec):
             raise ValueError('Layer ' + self.name + ' expects ' +
                              str(len(input_spec)) + ' inputs, '
                              'but it received ' + str(len(inputs)) +
                              ' input tensors. Input received: ' +
-                             str(input))
+                             str(inputs))
         for input_index, (x, spec) in enumerate(zip(inputs, input_spec)):
             if spec is None:
                 continue
@@ -467,11 +505,12 @@ class Layer(object):
                                     str(spec.shape) + ', found shape=' +
                                     str(x_shape))
 
-    def call(self, inputs):
+    def call(self, inputs, **kwargs):
         """This is where the layer's logic lives.
 
         # Arguments
-            inputs: input tensor, or list/tuple of input tensors.
+            inputs: Input tensor, or list/tuple of input tensors.
+            **kwargs: Additional keyword arguments.
 
         # Returns
             A tensor or list/tuple of tensors.
@@ -503,6 +542,8 @@ class Layer(object):
             ValueError: in case the layer is missing shape information
                 for its `build` call.
         """
+        if isinstance(inputs, list):
+            inputs = inputs[:]
         with K.name_scope(self.name):
             # Handle laying building (weight creating, input spec locking).
             if not self.built:
@@ -540,9 +581,10 @@ class Layer(object):
 
             # Handle mask propagation.
             previous_mask = _collect_previous_mask(inputs)
+            user_kwargs = copy.copy(kwargs)
             if not _is_all_none(previous_mask):
                 # The previous layer generated a mask.
-                if 'mask' in inspect.getargspec(self.call).args:
+                if has_arg(self.call, 'mask'):
                     if 'mask' not in kwargs:
                         # If mask is explicitly passed to __call__,
                         # we should override the default mask.
@@ -554,6 +596,20 @@ class Layer(object):
             output = self.call(inputs, **kwargs)
             output_mask = self.compute_mask(inputs, previous_mask)
 
+            # If the layer returns tensors from its inputs, unmodified,
+            # we copy them to avoid loss of tensor metadata.
+            output_ls = _to_list(output)
+            inputs_ls = _to_list(inputs)
+            output_ls_copy = []
+            for x in output_ls:
+                if x in inputs_ls:
+                    x = K.identity(x)
+                output_ls_copy.append(x)
+            if len(output_ls_copy) == 1:
+                output = output_ls_copy[0]
+            else:
+                output = output_ls_copy
+
             # Infering the output shape is only relevant for Theano.
             if all([s is not None for s in _to_list(input_shape)]):
                 output_shape = self.compute_output_shape(input_shape)
@@ -571,7 +627,7 @@ class Layer(object):
             self._add_inbound_node(input_tensors=inputs, output_tensors=output,
                                    input_masks=previous_mask, output_masks=output_mask,
                                    input_shapes=input_shape, output_shapes=output_shape,
-                                   arguments=kwargs)
+                                   arguments=user_kwargs)
 
             # Apply activity regularizer if any:
             if hasattr(self, 'activity_regularizer') and self.activity_regularizer is not None:
@@ -688,7 +744,7 @@ class Layer(object):
                                     str(mask))
             # masking not explicitly supported: return None as mask
             return None
-        # if masking is explictly supported, by default
+        # if masking is explicitly supported, by default
         # carry over the input mask
         return mask
 
@@ -1025,23 +1081,15 @@ class Layer(object):
                 (e.g. L2 weight regularization, which only depends
                 on the layer's weights variables, not on any inputs tensors).
         """
-        if losses is None:
+        if losses is None or losses == []:
             return
         # Update self.losses
         losses = _to_list(losses)
-        if not hasattr(self, 'losses'):
-            self.losses = []
-        try:
-            self.losses += losses
-        except AttributeError:
-            # In case self.losses isn't settable
-            # (i.e. it's a getter method).
-            # In that case the `losses` property is
-            # auto-computed and shouldn't be set.
-            pass
+        if hasattr(self, '_losses'):
+            self._losses += losses
         # Update self._per_input_updates
-        if not hasattr(self, '_per_input_losses'):
-            self._per_input_losses = {}
+        if isinstance(input, list) and inputs == []:
+            inputs = None
         if inputs is not None:
             inputs_hash = _object_list_uid(inputs)
         else:
@@ -1065,23 +1113,15 @@ class Layer(object):
                 the updates as conditional on these inputs.
                 If None is passed, the updates are assumed unconditional.
         """
-        if updates is None:
+        if updates is None or updates == []:
             return
         # Update self.updates
         updates = _to_list(updates)
-        if not hasattr(self, 'updates'):
-            self.updates = []
-        try:
-            self.updates += updates
-        except AttributeError:
-            # In case self.updates isn't settable
-            # (i.e. it's a getter method).
-            # In that case the `updates` property is
-            # auto-computed and shouldn't be set.
-            pass
+        if hasattr(self, '_updates'):
+            self._updates += updates
         # Update self._per_input_updates
-        if not hasattr(self, '_per_input_updates'):
-            self._per_input_updates = {}
+        if isinstance(inputs, list) and inputs == []:
+            inputs = None
         if inputs is not None:
             inputs_hash = _object_list_uid(inputs)
         else:
@@ -1093,8 +1133,6 @@ class Layer(object):
         self._per_input_updates[inputs_hash] += updates
 
     def get_updates_for(self, inputs):
-        if not hasattr(self, '_per_input_updates'):
-            return []
         if inputs is not None:
             inputs_hash = _object_list_uid(inputs)
         else:
@@ -1104,8 +1142,6 @@ class Layer(object):
         return []
 
     def get_losses_for(self, inputs):
-        if not hasattr(self, '_per_input_losses'):
-            return []
         if inputs is not None:
             inputs_hash = _object_list_uid(inputs)
         else:
@@ -1245,6 +1281,7 @@ class InputLayer(Layer):
         name: Name of the layer (string).
     """
 
+    @interfaces.legacy_input_support
     def __init__(self, input_shape=None, batch_size=None,
                  batch_input_shape=None,
                  dtype=None, input_tensor=None, sparse=False, name=None):
@@ -1261,7 +1298,8 @@ class InputLayer(Layer):
             raise ValueError('Only provide the input_shape OR '
                              'batch_input_shape argument to '
                              'InputLayer, not both at the same time.')
-        if input_tensor is not None:
+        if input_tensor is not None and batch_input_shape is None:
+            # If input_tensor is set, and batch_input_shape is not set:
             # Attempt automatic input shape inference.
             try:
                 batch_input_shape = K.int_shape(input_tensor)
@@ -1333,7 +1371,7 @@ def Input(shape=None, batch_shape=None,
     attributes that allow us to build a Keras model
     just by knowing the inputs and outputs of the model.
 
-    For instance, if a, b and c and Keras tensors,
+    For instance, if a, b and c are Keras tensors,
     it becomes possible to do:
     `model = Model(input=[a, b], output=c)`
 
@@ -1430,6 +1468,9 @@ class Container(Layer):
 
     # Class Methods
         from_config
+
+    # Raises
+        TypeError: if input tensors are not Keras tensors from InputLayer objects
     """
 
     @interfaces.legacy_model_constructor_support
@@ -1442,6 +1483,8 @@ class Container(Layer):
 
         self.supports_masking = False
         self.trainable = True
+        self._per_input_losses = {}
+        self._per_input_updates = {}
 
         # Container-specific properties.
         if isinstance(inputs, (list, tuple)):
@@ -1454,13 +1497,19 @@ class Container(Layer):
             self.outputs = [outputs]
 
         # Check for redundancy in inputs.
-        inputs_set = set(self.inputs)
-        if len(inputs_set) != len(self.inputs):
+        if len(set(self.inputs)) != len(self.inputs):
             raise ValueError('The list of inputs passed to the model '
                              'is redundant. '
                              'All inputs should only appear once.'
                              ' Found: ' + str(self.inputs))
 
+        # Check for redundancy in outputs.
+        if len(set(self.outputs)) != len(self.outputs):
+            warnings.warn('The list of outputs passed to the model '
+                          'is redundant. '
+                          'All outputs should only appear once.'
+                          ' Found: ' + str(self.outputs))
+
         # List of initial layers (1 to 1 mapping with self.inputs,
         # hence the same layer might appear twice)
         self.input_layers = []
@@ -1480,7 +1529,7 @@ class Container(Layer):
         # every time the Container is called on a set on input tensors,
         # we compute the output tensors,
         # output masks and output shapes in one pass,
-        # then cache them here. When of of these output is queried later,
+        # then cache them here. When one of these output is queried later,
         # we retrieve it from there instead of recomputing it.
         self._output_mask_cache = {}
         self._output_tensor_cache = {}
@@ -1562,6 +1611,15 @@ class Container(Layer):
         self._feed_inputs = []
         self._feed_input_shapes = []
         for i, layer in enumerate(self.input_layers):
+            # Check that layer is an InputLayer.
+            if not isinstance(layer, InputLayer):
+                raise TypeError(
+                    'Input layers to a `Model` must be `InputLayer` objects. '
+                    'Received inputs: {}. '
+                    'Input {} (0-based) originates '
+                    'from layer type `{}`.'.format(inputs,
+                                                   i,
+                                                   layer.__class__.__name__))
             self.input_names.append(layer.name)
             if layer.is_placeholder:
                 self._feed_input_names.append(layer.name)
@@ -1580,72 +1638,92 @@ class Container(Layer):
         nodes_depths = {}  # dict {node: depth value}
         layers_depths = {}  # dict {layer: depth value}
         layer_indices = {}  # dict {layer: index in traversal}
+        nodes_in_decreasing_depth = []
 
-        def make_node_marker(node, depth):
-            return str(id(node)) + '-' + str(depth)
-
-        def build_map_of_graph(tensor, seen_nodes=None, depth=0,
+        def build_map_of_graph(tensor, finished_nodes, nodes_in_progress,
                                layer=None, node_index=None, tensor_index=None):
             """Builds a map of the graph of layers.
 
-            This recursively updates the maps `nodes_depths`,
-            `layers_depths` and the set `container_nodes`.
-
-            Does not try to detect cycles in the graph.
+            This recursively updates the map `layer_indices`,
+            the list `nodes_in_decreasing_depth` and the set `container_nodes`.
 
             # Arguments
                 tensor: Some tensor in a graph.
-                seen_nodes: Set of node ids ("{layer.name}_ib-{node_index}")
-                    of nodes seen so far. Useful to prevent infinite loops.
-                depth: Current depth in the graph (0 = last output).
+                finished_nodes: Set of nodes whose subgraphs have been traversed
+                    completely. Useful to prevent duplicated work.
+                nodes_in_progress: Set of nodes that are currently active on the
+                    recursion stack. Useful to detect cycles.
                 layer: Layer from which `tensor` comes from. If not provided,
                     will be obtained from `tensor._keras_history`.
                 node_index: Node index from which `tensor` comes from.
                 tensor_index: Tensor_index from which `tensor` comes from.
+
+            # Raises
+                RuntimeError: if a cycle is detected.
             """
-            seen_nodes = seen_nodes or set()
             if not layer or node_index is None or tensor_index is None:
                 layer, node_index, tensor_index = tensor._keras_history
             node = layer.inbound_nodes[node_index]
 
             # Prevent cycles.
-            seen_nodes.add(make_node_marker(node, depth))
+            if node in nodes_in_progress:
+                raise RuntimeError(
+                    'The tensor ' + str(tensor) + ' at layer "' +
+                    layer.name + '" is part of a cycle.')
+
+            # Don't repeat work for shared subgraphs
+            if node in finished_nodes:
+                return
 
             node_key = layer.name + '_ib-' + str(node_index)
             # Update container_nodes.
             container_nodes.add(node_key)
-            # Update nodes_depths.
-            node_depth = nodes_depths.get(node)
-            if node_depth is None:
-                nodes_depths[node] = depth
-            else:
-                nodes_depths[node] = max(depth, node_depth)
-            # Update layers_depths.
-            previously_seen_depth = layers_depths.get(layer)
-            if previously_seen_depth is None:
-                current_depth = depth
-            else:
-                current_depth = max(depth, previously_seen_depth)
-            layers_depths[layer] = current_depth
+
+            # Store the traversal order for layer sorting.
             if layer not in layer_indices:
                 layer_indices[layer] = len(layer_indices)
 
+            nodes_in_progress.add(node)
+
             # Propagate to all previous tensors connected to this node.
             for i in range(len(node.inbound_layers)):
                 x = node.input_tensors[i]
                 layer = node.inbound_layers[i]
                 node_index = node.node_indices[i]
                 tensor_index = node.tensor_indices[i]
-                next_node = layer.inbound_nodes[node_index]
-                # use node_marker to prevent cycles
-                node_marker = make_node_marker(next_node, current_depth + 1)
-                if node_marker not in seen_nodes:
-                    build_map_of_graph(x, seen_nodes, current_depth + 1,
-                                       layer, node_index, tensor_index)
+                build_map_of_graph(x, finished_nodes, nodes_in_progress,
+                                   layer, node_index, tensor_index)
 
+            finished_nodes.add(node)
+            nodes_in_progress.remove(node)
+
+            nodes_in_decreasing_depth.append(node)
+
+        finished_nodes = set()
+        nodes_in_progress = set()
         for x in self.outputs:
-            seen_nodes = set()
-            build_map_of_graph(x, seen_nodes, depth=0)
+            build_map_of_graph(x, finished_nodes, nodes_in_progress)
+
+        for node in reversed(nodes_in_decreasing_depth):
+            # If the depth is not set, the node has no outbound nodes (depth 0).
+            depth = nodes_depths.setdefault(node, 0)
+
+            # Update the depth of the corresponding layer
+            previous_depth = layers_depths.get(node.outbound_layer, 0)
+            # If we've seen this layer before at a higher depth, we should use that depth instead
+            # of the node depth.  This is necessary for shared layers that have inputs at different
+            # depth levels in the graph.
+            depth = max(depth, previous_depth)
+            layers_depths[node.outbound_layer] = depth
+            nodes_depths[node] = depth
+
+            # Update the depth of inbound nodes.
+            for i in range(len(node.inbound_layers)):
+                inbound_layer = node.inbound_layers[i]
+                node_index = node.node_indices[i]
+                inbound_node = inbound_layer.inbound_nodes[node_index]
+                previous_depth = nodes_depths.get(inbound_node, 0)
+                nodes_depths[inbound_node] = max(depth + 1, previous_depth)
 
         # Build a dict {depth: list of nodes with this depth}
         nodes_by_depth = {}
@@ -1796,19 +1874,16 @@ class Container(Layer):
         updates = []
         for layer in self.layers:
             if hasattr(layer, 'updates'):
-                if len(layer.inbound_nodes) == 1:
-                    updates += layer.updates
-                else:
-                    # Collect updates that are dependent on inputs
-                    # that are part of the model.
-                    for node_index, node in enumerate(layer.inbound_nodes):
-                        node_key = layer.name + '_ib-' + str(node_index)
-                        if node_key in self.container_nodes:
-                            # The model owns this layer node.
-                            inputs = node.input_tensors
-                            updates += layer.get_updates_for(inputs)
-                    # Collect unconditional updates.
-                    updates += layer.get_updates_for(None)
+                # Collect updates that are dependent on inputs
+                # that are part of the model.
+                for node_index, node in enumerate(layer.inbound_nodes):
+                    node_key = layer.name + '_ib-' + str(node_index)
+                    if node_key in self.container_nodes:
+                        # The model owns this layer node.
+                        inputs = node.input_tensors
+                        updates += layer.get_updates_for(inputs)
+                # Collect unconditional updates.
+                updates += layer.get_updates_for(None)
         return updates
 
     @property
@@ -1827,22 +1902,18 @@ class Container(Layer):
         # Retrieve losses for all internal layers.
         for layer in self.layers:
             if hasattr(layer, 'losses'):
-                if len(layer.inbound_nodes) == 1:
-                    losses += layer.losses
-                else:
-                    # Collect losses that are dependent on inputs
-                    # that are part of the model.
-                    for node_index, node in enumerate(layer.inbound_nodes):
-                        node_key = layer.name + '_ib-' + str(node_index)
-                        if node_key in self.container_nodes:
-                            # The model owns this layer node.
-                            inputs = node.input_tensors
-                            losses += layer.get_losses_for(inputs)
-                    # Collect unconditional losses.
-                    losses += layer.get_losses_for(None)
+                # Collect losses that are dependent on inputs
+                # that are part of the model.
+                for node_index, node in enumerate(layer.inbound_nodes):
+                    node_key = layer.name + '_ib-' + str(node_index)
+                    if node_key in self.container_nodes:
+                        # The model owns this layer node.
+                        inputs = node.input_tensors
+                        losses += layer.get_losses_for(inputs)
+                # Collect unconditional losses.
+                losses += layer.get_losses_for(None)
         # Add any potential unconditional model-level loss.
-        if hasattr(self, '_per_input_losses'):
-            losses += self._per_input_losses.get(None, [])
+        losses += self.get_losses_for(None)
         return losses
 
     @property
@@ -2125,6 +2196,7 @@ class Container(Layer):
                 for x in reference_input_tensors:
                     if str(id(x)) in tensor_map:
                         computed_data.append(tensor_map[str(id(x))])
+
                 if len(computed_data) == len(reference_input_tensors):
                     # call layer
                     with K.name_scope(layer.name):
@@ -2134,7 +2206,7 @@ class Container(Layer):
                             kwargs = {}
                         if len(computed_data) == 1:
                             computed_tensor, computed_mask = computed_data[0]
-                            if 'mask' in inspect.getargspec(layer.call).args:
+                            if has_arg(layer.call, 'mask'):
                                 if 'mask' not in kwargs:
                                     kwargs['mask'] = computed_mask
                             output_tensors = _to_list(layer.call(computed_tensor, **kwargs))
@@ -2145,23 +2217,27 @@ class Container(Layer):
                         else:
                             computed_tensors = [x[0] for x in computed_data]
                             computed_masks = [x[1] for x in computed_data]
-                            if 'mask' in inspect.getargspec(layer.call).args:
+                            if has_arg(layer.call, 'mask'):
                                 if 'mask' not in kwargs:
                                     kwargs['mask'] = computed_masks
                             output_tensors = _to_list(layer.call(computed_tensors, **kwargs))
                             output_masks = _to_list(layer.compute_mask(computed_tensors,
                                                                        computed_masks))
 
+                        # Apply activity regularizer if any:
+                        if hasattr(layer, 'activity_regularizer') and layer.activity_regularizer is not None:
+                            regularization_losses = [layer.activity_regularizer(x) for x in computed_tensors]
+                            layer.add_loss(regularization_losses, computed_tensors)
+
                     # Update model updates and losses:
-                    layer_inputs = [x[0] for x in computed_data]
                     # Keep track of updates that depend on the inputs
                     # (e.g. BN updates).
-                    self.add_update(layer.get_updates_for(layer_inputs), inputs)
+                    self.add_update(layer.get_updates_for(computed_tensors), inputs)
                     # Keep track of unconditional updates (e.g. a counter).
                     self.add_update(layer.get_updates_for(None), None)
                     # Keep track of losses that depend on the inputs
                     # (e.g. activity regularizers).
-                    self.add_loss(layer.get_losses_for(layer_inputs), inputs)
+                    self.add_loss(layer.get_losses_for(computed_tensors), inputs)
                     # Keep track of unconditional losses
                     # (e.g. weight regularizers).
                     self.add_loss(layer.get_losses_for(None), None)
@@ -2390,7 +2466,7 @@ class Container(Layer):
             output_tensors.append(layer_output_tensors[tensor_index])
         return cls(inputs=input_tensors, outputs=output_tensors, name=name)
 
-    def save(self, filepath, overwrite=True):
+    def save(self, filepath, overwrite=True, include_optimizer=True):
         """Save the model to a single HDF5 file.
 
         The savefile includes:
@@ -2411,6 +2487,7 @@ class Container(Layer):
             filepath: String, path to the file to save the weights to.
             overwrite: Whether to silently overwrite any existing file at the
                 target location, or provide the user with a manual prompt.
+            include_optimizer: If True, save optimizer's state together.
 
         # Example
 
@@ -2426,7 +2503,7 @@ class Container(Layer):
         ```
         """
         from ..models import save_model
-        save_model(self, filepath, overwrite)
+        save_model(self, filepath, overwrite, include_optimizer)
 
     def save_weights(self, filepath, overwrite=True):
         """Dumps all layer weights to a HDF5 file.
@@ -2560,10 +2637,25 @@ class Container(Layer):
         """
         return yaml.dump(self._updated_config(), **kwargs)
 
-    def summary(self, line_length=None, positions=None):
-        print_layer_summary(self,
-                            line_length=line_length,
-                            positions=positions)
+    def summary(self, line_length=None, positions=None, print_fn=print):
+        """Prints a string summary of the network.
+
+        # Arguments
+            line_length: Total length of printed lines
+                (e.g. set this to adapt the display to different
+                terminal window sizes).
+            positions: Relative or absolute positions of log elements
+                in each line. If not provided,
+                defaults to `[.33, .55, .67, 1.]`.
+            print_fn: Print function to use.
+                It will be called on each line of the summary.
+                You can set it to a custom function
+                in order to capture the string summary.
+        """
+        return print_layer_summary(self,
+                                   line_length=line_length,
+                                   positions=positions,
+                                   print_fn=print_fn)
 
 
 def get_source_inputs(tensor, layer=None, node_index=None):
@@ -2742,6 +2834,25 @@ def preprocess_weights_for_loading(layer, weights,
         A list of weights values (Numpy arrays).
     """
     if original_keras_version == '1':
+        if layer.__class__.__name__ == 'Bidirectional':
+            num_weights_per_layer = len(weights) // 2
+
+            forward_weights = preprocess_weights_for_loading(layer.forward_layer,
+                                                             weights[:num_weights_per_layer],
+                                                             original_keras_version,
+                                                             original_backend)
+            backward_weights = preprocess_weights_for_loading(layer.backward_layer,
+                                                              weights[num_weights_per_layer:],
+                                                              original_keras_version,
+                                                              original_backend)
+            weights = forward_weights + backward_weights
+
+        if layer.__class__.__name__ == 'TimeDistributed':
+            weights = preprocess_weights_for_loading(layer.layer,
+                                                     weights,
+                                                     original_keras_version,
+                                                     original_backend)
+
         if layer.__class__.__name__ == 'Conv1D':
             shape = weights[0].shape
             # Handle Keras 1.1 format
@@ -2827,16 +2938,45 @@ def preprocess_weights_for_loading(layer, weights,
                                                     (2, 3, 1, 0))
                 weights = [kernel, recurrent_kernel, bias]
 
-    if original_backend and K.backend() != original_backend:
-        conv_layers = ['Conv1D',
-                       'Conv2D',
-                       'Conv3D',
-                       'Conv2DTranspose']
-        if layer.__class__.__name__ in conv_layers:
+        if layer.__class__.__name__ in ['Model', 'Sequential']:
+            new_weights = []
+            # trainable weights
+            for sublayer in layer.layers:
+                num_weights = len(sublayer.trainable_weights)
+                if num_weights > 0:
+                    new_weights.extend(preprocess_weights_for_loading(
+                        layer=sublayer,
+                        weights=weights[:num_weights],
+                        original_keras_version=original_keras_version,
+                        original_backend=original_backend))
+                    weights = weights[num_weights:]
+
+            # non-trainable weights
+            for sublayer in layer.layers:
+                num_weights = len([l for l in sublayer.weights if l not in sublayer.trainable_weights])
+                if num_weights > 0:
+                    new_weights.extend(preprocess_weights_for_loading(
+                        layer=sublayer,
+                        weights=weights[:num_weights],
+                        original_keras_version=original_keras_version,
+                        original_backend=original_backend))
+                    weights = weights[num_weights:]
+            weights = new_weights
+
+    conv_layers = ['Conv1D',
+                   'Conv2D',
+                   'Conv3D',
+                   'Conv2DTranspose',
+                   'ConvLSTM2D']
+    if layer.__class__.__name__ in conv_layers:
+        if original_backend and K.backend() != original_backend:
             weights[0] = conv_utils.convert_kernel(weights[0])
-        if layer.__class__.__name__ == 'ConvLSTM2D':
-            weights[0] = conv_utils.convert_kernel(weights[0])
-            weights[1] = conv_utils.convert_kernel(weights[1])
+            if layer.__class__.__name__ == 'ConvLSTM2D':
+                weights[1] = conv_utils.convert_kernel(weights[1])
+        if K.int_shape(layer.weights[0]) != weights[0].shape:
+            weights[0] = np.transpose(weights[0], (3, 2, 0, 1))
+            if layer.__class__.__name__ == 'ConvLSTM2D':
+                weights[1] = np.transpose(weights[1], (3, 2, 0, 1))
     return weights
 
 

keras/engine/training.py

@@ -4,12 +4,13 @@ from __future__ import absolute_import
 
 import warnings
 import copy
-import time
 import numpy as np
-import multiprocessing
-import threading
 import six
 
+from keras.utils import Sequence
+from keras.utils import GeneratorEnqueuer
+from keras.utils import OrderedEnqueuer
+
 try:
     import queue
 except ImportError:
@@ -50,6 +51,8 @@ def _standardize_input_data(data, names, shapes=None,
     # Raises
         ValueError: in case of improperly formatted user-provided data.
     """
+    if not names:
+        return []
     if data is None:
         return [None for _ in range(len(names))]
     if isinstance(data, dict):
@@ -63,7 +66,8 @@ def _standardize_input_data(data, names, shapes=None,
     elif isinstance(data, list):
         if len(data) != len(names):
             if data and hasattr(data[0], 'shape'):
-                raise ValueError('Error when checking ' + exception_prefix +
+                raise ValueError('Error when checking model ' +
+                                 exception_prefix +
                                  ': the list of Numpy arrays '
                                  'that you are passing to your model '
                                  'is not the size the model expected. '
@@ -77,7 +81,8 @@ def _standardize_input_data(data, names, shapes=None,
                     data = [np.asarray(data)]
                 else:
                     raise ValueError(
-                        'Error when checking ' + exception_prefix +
+                        'Error when checking model ' +
+                        exception_prefix +
                         ': you are passing a list as '
                         'input to your model, '
                         'but the model expects '
@@ -88,15 +93,17 @@ def _standardize_input_data(data, names, shapes=None,
         arrays = data
     else:
         if not hasattr(data, 'shape'):
-            raise TypeError('Error when checking ' + exception_prefix +
+            raise TypeError('Error when checking model ' +
+                            exception_prefix +
                             ': data should be a Numpy array, '
                             'or list/dict of Numpy arrays. '
                             'Found: ' + str(data)[:200] + '...')
-        if len(names) != 1:
+        if len(names) > 1:
             # Case: model expects multiple inputs but only received
             # a single Numpy array.
-            raise ValueError('The model expects ' + str(len(names)) +
-                             ' input arrays, but only received one array. '
+            raise ValueError('The model expects ' + str(len(names)) + ' ' +
+                             exception_prefix +
+                             ' arrays, but only received one array. '
                              'Found: array with shape ' + str(data.shape))
         arrays = [data]
 
@@ -193,7 +200,7 @@ def _standardize_sample_weights(sample_weight, output_names):
                                                 'sample_weight')
 
 
-def _check_array_lengths(inputs, targets, weights):
+def _check_array_lengths(inputs, targets, weights=None):
     """Does user input validation for numpy arrays.
 
     # Arguments
@@ -204,29 +211,34 @@ def _check_array_lengths(inputs, targets, weights):
     # Raises
         ValueError: in case of incorrectly formatted data.
     """
-    x_lengths = [x.shape[0] for x in inputs]
-    y_lengths = [y.shape[0] for y in targets]
-    w_lengths = [w.shape[0] for w in weights]
-    set_x = set(x_lengths)
+    def set_of_lengths(x):
+        # return a set with the variation between
+        # different shapes, with None => 0
+        if x is None:
+            return {0}
+        else:
+            return set([0 if y is None else y.shape[0] for y in x])
+
+    set_x = set_of_lengths(inputs)
+    set_y = set_of_lengths(targets)
+    set_w = set_of_lengths(weights)
     if len(set_x) > 1:
         raise ValueError('All input arrays (x) should have '
                          'the same number of samples. Got array shapes: ' +
                          str([x.shape for x in inputs]))
-    set_y = set(y_lengths)
     if len(set_y) > 1:
         raise ValueError('All target arrays (y) should have '
                          'the same number of samples. Got array shapes: ' +
                          str([y.shape for y in targets]))
-    set_w = set(w_lengths)
-    if len(set_w) > 1:
-        raise ValueError('All sample_weight arrays should have '
-                         'the same number of samples. Got array shapes: ' +
-                         str([w.shape for w in weights]))
     if set_x and set_y and list(set_x)[0] != list(set_y)[0]:
         raise ValueError('Input arrays should have '
                          'the same number of samples as target arrays. '
                          'Found ' + str(list(set_x)[0]) + ' input samples '
                          'and ' + str(list(set_y)[0]) + ' target samples.')
+    if len(set_w) > 1:
+        raise ValueError('All sample_weight arrays should have '
+                         'the same number of samples. Got array shapes: ' +
+                         str([w.shape for w in weights]))
     if set_y and set_w and list(set_y)[0] != list(set_w)[0]:
         raise ValueError('Sample_weight arrays should have '
                          'the same number of samples as target arrays. Got ' +
@@ -235,7 +247,7 @@ def _check_array_lengths(inputs, targets, weights):
 
 
 def _check_loss_and_target_compatibility(targets, loss_fns, output_shapes):
-    """Does validation on the compatiblity of targets and loss functions.
+    """Does validation on the compatibility of targets and loss functions.
 
     This helps prevent users from using loss functions incorrectly.
 
@@ -248,7 +260,7 @@ def _check_loss_and_target_compatibility(targets, loss_fns, output_shapes):
         ValueError: if a loss function or target array
             is incompatible with an output.
     """
-    key_losses = {'mean_square_error',
+    key_losses = {'mean_squared_error',
                   'binary_crossentropy',
                   'categorical_crossentropy'}
     for y, loss, shape in zip(targets, loss_fns, output_shapes):
@@ -380,21 +392,25 @@ def _slice_arrays(arrays, start=None, stop=None):
     # Returns
         A slice of the array(s).
     """
-    if isinstance(arrays, list):
+    if arrays is None:
+        return [None]
+    elif isinstance(arrays, list):
         if hasattr(start, '__len__'):
             # hdf5 datasets only support list objects as indices
             if hasattr(start, 'shape'):
                 start = start.tolist()
-            return [x[start] for x in arrays]
+            return [None if x is None else x[start] for x in arrays]
         else:
-            return [x[start:stop] for x in arrays]
+            return [None if x is None else x[start:stop] for x in arrays]
     else:
         if hasattr(start, '__len__'):
             if hasattr(start, 'shape'):
                 start = start.tolist()
             return arrays[start]
-        else:
+        elif hasattr(start, '__getitem__'):
             return arrays[start:stop]
+        else:
+            return [None]
 
 
 def _weighted_masked_objective(fn):
@@ -437,13 +453,12 @@ def _weighted_masked_objective(fn):
             #  to the number of unmasked samples.
             score_array /= K.mean(mask)
 
-        # reduce score_array to same ndim as weight array
-        ndim = K.ndim(score_array)
-        weight_ndim = K.ndim(weights)
-        score_array = K.mean(score_array, axis=list(range(weight_ndim, ndim)))
-
         # apply sample weighting
         if weights is not None:
+            # reduce score_array to same ndim as weight array
+            ndim = K.ndim(score_array)
+            weight_ndim = K.ndim(weights)
+            score_array = K.mean(score_array, axis=list(range(weight_ndim, ndim)))
             score_array *= weights
             score_array /= K.mean(K.cast(K.not_equal(weights, 0), K.floatx()))
         return K.mean(score_array)
@@ -556,7 +571,7 @@ def _standardize_weights(y, sample_weight=None, class_weight=None,
         return sample_weight
     elif isinstance(class_weight, dict):
         if len(y.shape) > 2:
-            raise ValueError('class_weight not supported for '
+            raise ValueError('`class_weight` not supported for '
                              '3+ dimensional targets.')
         if y.shape[1] > 1:
             y_classes = y.argmax(axis=1)
@@ -573,97 +588,6 @@ def _standardize_weights(y, sample_weight=None, class_weight=None,
             return np.ones((y.shape[0], y.shape[1]), dtype=K.floatx())
 
 
-class GeneratorEnqueuer(object):
-    """Builds a queue out of a data generator.
-
-    Used in `fit_generator`, `evaluate_generator`, `predict_generator`.
-
-    # Arguments
-        generator: a generator function which endlessly yields data
-        pickle_safe: use multiprocessing if True, otherwise threading
-    """
-
-    def __init__(self, generator, pickle_safe=False):
-        self._generator = generator
-        self._pickle_safe = pickle_safe
-        self._threads = []
-        self._stop_event = None
-        self.queue = None
-
-    def start(self, workers=1, max_q_size=10, wait_time=0.05):
-        """Kicks off threads which add data from the generator into the queue.
-
-        # Arguments
-            workers: number of worker threads
-            max_q_size: queue size (when full, threads could block on put())
-            wait_time: time to sleep in-between calls to put()
-        """
-
-        def data_generator_task():
-            while not self._stop_event.is_set():
-                try:
-                    if self._pickle_safe or self.queue.qsize() < max_q_size:
-                        generator_output = next(self._generator)
-                        self.queue.put(generator_output)
-                    else:
-                        time.sleep(wait_time)
-                except Exception:
-                    self._stop_event.set()
-                    raise
-
-        try:
-            if self._pickle_safe:
-                self.queue = multiprocessing.Queue(maxsize=max_q_size)
-                self._stop_event = multiprocessing.Event()
-            else:
-                self.queue = queue.Queue()
-                self._stop_event = threading.Event()
-
-            for _ in range(workers):
-                if self._pickle_safe:
-                    # Reset random seed else all children processes
-                    # share the same seed
-                    np.random.seed()
-                    thread = multiprocessing.Process(target=data_generator_task)
-                    thread.daemon = True
-                else:
-                    thread = threading.Thread(target=data_generator_task)
-                self._threads.append(thread)
-                thread.start()
-        except:
-            self.stop()
-            raise
-
-    def is_running(self):
-        return self._stop_event is not None and not self._stop_event.is_set()
-
-    def stop(self, timeout=None):
-        """Stop running threads and wait for them to exit, if necessary.
-
-        Should be called by the same thread which called start().
-
-        # Arguments
-            timeout: maximum time to wait on thread.join()
-        """
-        if self.is_running():
-            self._stop_event.set()
-
-        for thread in self._threads:
-            if thread.is_alive():
-                if self._pickle_safe:
-                    thread.terminate()
-                else:
-                    thread.join(timeout)
-
-        if self._pickle_safe:
-            if self.queue is not None:
-                self.queue.close()
-
-        self._threads = []
-        self._stop_event = None
-        self.queue = None
-
-
 class Model(Container):
     """The `Model` class adds training & evaluation routines to a `Container`.
     """
@@ -679,6 +603,8 @@ class Model(Container):
                 See [losses](/losses).
                 If the model has multiple outputs, you can use a different loss
                 on each output by passing a dictionary or a list of losses.
+                The loss value that will be minimized by the model
+                will then be the sum of all individual losses.
             metrics: list of metrics to be evaluated by the model
                 during training and testing.
                 Typically you will use `metrics=['accuracy']`.
@@ -688,6 +614,9 @@ class Model(Container):
             loss_weights: Optional list or dictionary specifying scalar
                 coefficients (Python floats) to weight the loss contributions
                 of different model outputs.
+                The loss value that will be minimized by the model
+                will then be the *weighted sum* of all individual losses,
+                weighted by the `loss_weights` coefficients.
                 If a list, it is expected to have a 1:1 mapping
                 to the model's outputs. If a tensor, it is expected to map
                 output names (strings) to scalar coefficients.
@@ -698,7 +627,8 @@ class Model(Container):
                 `sample_weight_mode` on each output by passing a
                 dictionary or a list of modes.
             **kwargs: when using the Theano backend, these arguments
-                are passed into K.function. Ignored for Tensorflow backend.
+                are passed into K.function. When using the Tensorflow backend,
+                these arguments are passed into `tf.Session.run`.
 
         # Raises
             ValueError: In case of invalid arguments for
@@ -939,7 +869,8 @@ class Model(Container):
                     # (because of class mode duality)
                     output_shape = self.internal_output_shapes[i]
                     acc_fn = None
-                    if output_shape[-1] == 1 or self.loss_functions[i] == losses.binary_crossentropy:
+                    if (output_shape[-1] == 1 or
+                       self.loss_functions[i] == losses.binary_crossentropy):
                         # case: binary accuracy
                         acc_fn = metrics_module.binary_accuracy
                     elif self.loss_functions[i] == losses.sparse_categorical_crossentropy:
@@ -977,14 +908,8 @@ class Model(Container):
         self.test_function = None
         self.predict_function = None
 
-        # Collected trainable weights and sort them deterministically.
+        # Collected trainable weights, sorted in topological order.
         trainable_weights = self.trainable_weights
-        # Sort weights by name.
-        if trainable_weights:
-            if K.backend() == 'theano':
-                trainable_weights.sort(key=lambda x: x.name if x.name else x.auto_name)
-            else:
-                trainable_weights.sort(key=lambda x: x.name)
         self._collected_trainable_weights = trainable_weights
 
     def _make_train_function(self):
@@ -1004,6 +929,7 @@ class Model(Container):
             self.train_function = K.function(inputs,
                                              [self.total_loss] + self.metrics_tensors,
                                              updates=updates,
+                                             name='train_function',
                                              **self._function_kwargs)
 
     def _make_test_function(self):
@@ -1018,6 +944,7 @@ class Model(Container):
             self.test_function = K.function(inputs,
                                             [self.total_loss] + self.metrics_tensors,
                                             updates=self.state_updates,
+                                            name='test_function',
                                             **self._function_kwargs)
 
     def _make_predict_function(self):
@@ -1034,6 +961,7 @@ class Model(Container):
             self.predict_function = K.function(inputs,
                                                self.outputs,
                                                updates=self.state_updates,
+                                               name='predict_function',
                                                **kwargs)
 
     def _fit_loop(self, f, ins, out_labels=None, batch_size=32,
@@ -1125,7 +1053,7 @@ class Model(Container):
                 batch_ids = index_array[batch_start:batch_end]
                 try:
                     if isinstance(ins[-1], float):
-                        # do not slice the training phase flag
+                        # Do not slice the training phase flag.
                         ins_batch = _slice_arrays(ins[:-1], batch_ids) + [ins[-1]]
                     else:
                         ins_batch = _slice_arrays(ins, batch_ids)
@@ -1144,17 +1072,17 @@ class Model(Container):
                     batch_logs[l] = o
 
                 callbacks.on_batch_end(batch_index, batch_logs)
+                if callback_model.stop_training:
+                    break
 
-                if batch_index == len(batches) - 1:  # last batch
-                    # validation
+                if batch_index == len(batches) - 1:  # Last batch.
                     if do_validation:
-                        # replace with self._evaluate
                         val_outs = self._test_loop(val_f, val_ins,
                                                    batch_size=batch_size,
                                                    verbose=0)
                         if not isinstance(val_outs, list):
                             val_outs = [val_outs]
-                        # same labels assumed
+                        # Same labels assumed.
                         for l, o in zip(out_labels, val_outs):
                             epoch_logs['val_' + l] = o
             callbacks.on_epoch_end(epoch, epoch_logs)
@@ -1194,7 +1122,7 @@ class Model(Container):
         for batch_index, (batch_start, batch_end) in enumerate(batches):
             batch_ids = index_array[batch_start:batch_end]
             if ins and isinstance(ins[-1], float):
-                # do not slice the training phase flag
+                # Do not slice the training phase flag.
                 ins_batch = _slice_arrays(ins[:-1], batch_ids) + [ins[-1]]
             else:
                 ins_batch = _slice_arrays(ins, batch_ids)
@@ -1205,7 +1133,7 @@ class Model(Container):
             if batch_index == 0:
                 for batch_out in batch_outs:
                     shape = (samples,) + batch_out.shape[1:]
-                    outs.append(np.zeros(shape, dtype=K.floatx()))
+                    outs.append(np.zeros(shape, dtype=batch_out.dtype))
 
             for i, batch_out in enumerate(batch_outs):
                 outs[i][batch_start:batch_end] = batch_out
@@ -1248,7 +1176,7 @@ class Model(Container):
         for batch_index, (batch_start, batch_end) in enumerate(batches):
             batch_ids = index_array[batch_start:batch_end]
             if isinstance(ins[-1], float):
-                # do not slice the training phase flag
+                # Do not slice the training phase flag.
                 ins_batch = _slice_arrays(ins[:-1], batch_ids) + [ins[-1]]
             else:
                 ins_batch = _slice_arrays(ins, batch_ids)
@@ -1292,11 +1220,11 @@ class Model(Container):
         x = _standardize_input_data(x, self._feed_input_names,
                                     self._feed_input_shapes,
                                     check_batch_axis=False,
-                                    exception_prefix='model input')
+                                    exception_prefix='input')
         y = _standardize_input_data(y, self._feed_output_names,
                                     output_shapes,
                                     check_batch_axis=False,
-                                    exception_prefix='model target')
+                                    exception_prefix='target')
         sample_weights = _standardize_sample_weights(sample_weight,
                                                      self._feed_output_names)
         class_weights = _standardize_class_weights(class_weight,
@@ -1317,6 +1245,20 @@ class Model(Container):
                                  str(x[0].shape[0]) + ' samples')
         return x, y, sample_weights
 
+    def _get_deduped_metrics_names(self):
+        out_labels = self.metrics_names
+
+        # Rename duplicated metrics name
+        # (can happen with an output layer shared among multiple dataflows).
+        deduped_out_labels = []
+        for i, label in enumerate(out_labels):
+            new_label = label
+            if out_labels.count(label) > 1:
+                dup_idx = out_labels[:i].count(label)
+                new_label += '_' + str(dup_idx + 1)
+            deduped_out_labels.append(new_label)
+        return deduped_out_labels
+
     def fit(self, x=None,
             y=None,
             batch_size=32,
@@ -1346,7 +1288,7 @@ class Model(Container):
             batch_size: integer. Number of samples per gradient update.
             epochs: integer, the number of times to iterate
                 over the training data arrays.
-                verbose: 0, 1, or 2. Verbosity mode.
+            verbose: 0, 1, or 2. Verbosity mode.
                 0 = silent, 1 = verbose, 2 = one log line per epoch.
             callbacks: list of callbacks to be called during training.
                 See [callbacks](/callbacks).
@@ -1396,14 +1338,14 @@ class Model(Container):
         if kwargs:
             raise TypeError('Unrecognized keyword arguments: ' + str(kwargs))
 
-        # validate user data
+        # Validate user data.
         x, y, sample_weights = self._standardize_user_data(
             x, y,
             sample_weight=sample_weight,
             class_weight=class_weight,
             check_batch_axis=False,
             batch_size=batch_size)
-        # prepare validation data
+        # Prepare validation data.
         if validation_data:
             do_validation = True
             if len(validation_data) == 2:
@@ -1432,7 +1374,10 @@ class Model(Container):
 
         elif validation_split and 0. < validation_split < 1.:
             do_validation = True
-            split_at = int(len(x[0]) * (1. - validation_split))
+            if hasattr(x[0], 'shape'):
+                split_at = int(x[0].shape[0] * (1. - validation_split))
+            else:
+                split_at = int(len(x[0]) * (1. - validation_split))
             x, val_x = (_slice_arrays(x, 0, split_at), _slice_arrays(x, split_at))
             y, val_y = (_slice_arrays(y, 0, split_at), _slice_arrays(y, split_at))
             sample_weights, val_sample_weights = (
@@ -1449,7 +1394,7 @@ class Model(Container):
             val_f = None
             val_ins = None
 
-        # prepare input arrays and training function
+        # Prepare input arrays and training function.
         if self.uses_learning_phase and not isinstance(K.learning_phase(), int):
             ins = x + y + sample_weights + [1.]
         else:
@@ -1457,26 +1402,15 @@ class Model(Container):
         self._make_train_function()
         f = self.train_function
 
-        # prepare display labels
-        out_labels = self.metrics_names
-
-        # rename duplicated metrics name
-        # (can happen with an output layer shared among multiple dataflows)
-        deduped_out_labels = []
-        for i, label in enumerate(out_labels):
-            new_label = label
-            if out_labels.count(label) > 1:
-                dup_idx = out_labels[:i].count(label)
-                new_label += '_' + str(dup_idx + 1)
-            deduped_out_labels.append(new_label)
-        out_labels = deduped_out_labels
+        # Prepare display labels.
+        out_labels = self._get_deduped_metrics_names()
 
         if do_validation:
             callback_metrics = copy.copy(out_labels) + ['val_' + n for n in out_labels]
         else:
             callback_metrics = copy.copy(out_labels)
 
-        # delegate logic to _fit_loop
+        # Delegate logic to `_fit_loop`.
         return self._fit_loop(f, ins, out_labels=out_labels,
                               batch_size=batch_size, epochs=epochs,
                               verbose=verbose, callbacks=callbacks,
@@ -1511,13 +1445,13 @@ class Model(Container):
             and/or metrics). The attribute `model.metrics_names` will give you
             the display labels for the scalar outputs.
         """
-        # validate user data
+        # Validate user data.
         x, y, sample_weights = self._standardize_user_data(
             x, y,
             sample_weight=sample_weight,
             check_batch_axis=False,
             batch_size=batch_size)
-        # prepare inputs, delegate logic to _test_loop
+        # Prepare inputs, delegate logic to `_test_loop`.
         if self.uses_learning_phase and not isinstance(K.learning_phase(), int):
             ins = x + y + sample_weights + [0.]
         else:
@@ -1548,7 +1482,7 @@ class Model(Container):
                 or in case a stateful model receives a number of samples
                 that is not a multiple of the batch size.
         """
-        # validate user data
+        # Validate user data.
         x = _standardize_input_data(x, self._feed_input_names,
                                     self._feed_input_shapes,
                                     check_batch_axis=False)
@@ -1561,7 +1495,7 @@ class Model(Container):
                                  str(x[0].shape[0]) + ' samples. '
                                  'Batch size: ' + str(batch_size) + '.')
 
-        # prepare inputs, delegate logic to _predict_loop
+        # Prepare inputs, delegate logic to `_predict_loop`.
         if self.uses_learning_phase and not isinstance(K.learning_phase(), int):
             ins = x + [0.]
         else:
@@ -1694,9 +1628,9 @@ class Model(Container):
                       validation_data=None,
                       validation_steps=None,
                       class_weight=None,
-                      max_q_size=10,
+                      max_queue_size=10,
                       workers=1,
-                      pickle_safe=False,
+                      use_multiprocessing=False,
                       initial_epoch=0):
         """Fits the model on data yielded batch-by-batch by a Python generator.
 
@@ -1704,15 +1638,21 @@ class Model(Container):
         For instance, this allows you to do real-time data augmentation
         on images on CPU in parallel to training your model on GPU.
 
+        The use of `keras.utils.Sequence` guarantees the ordering
+        and guarantees the single use of every input per epoch when
+        using `use_multiprocessing=True`.
+
         # Arguments
-            generator: a generator.
+            generator: a generator or an instance of Sequence (keras.utils.Sequence)
+                    object in order to avoid duplicate data
+                    when using multiprocessing.
                 The output of the generator must be either
                 - a tuple (inputs, targets)
                 - a tuple (inputs, targets, sample_weights).
                 All arrays should contain the same number of samples.
                 The generator is expected to loop over its data
                 indefinitely. An epoch finishes when `steps_per_epoch`
-                samples have been seen by the model.
+                batches have been seen by the model.
             steps_per_epoch: Total number of steps (batches of samples)
                 to yield from `generator` before declaring one epoch
                 finished and starting the next epoch. It should typically
@@ -1730,10 +1670,10 @@ class Model(Container):
                 to yield from `generator` before stopping.
             class_weight: dictionary mapping class indices to a weight
                 for the class.
-            max_q_size: maximum size for the generator queue
+            max_queue_size: maximum size for the generator queue
             workers: maximum number of processes to spin up
                 when using process based threading
-            pickle_safe: if True, use process based threading.
+            use_multiprocessing: if True, use process based threading.
                 Note that because
                 this implementation relies on multiprocessing,
                 you should not pass
@@ -1778,13 +1718,15 @@ class Model(Container):
         # python 2 has 'next', 3 has '__next__'
         # avoid any explicit version checks
         val_gen = (hasattr(validation_data, 'next') or
-                   hasattr(validation_data, '__next__'))
+                   hasattr(validation_data, '__next__') or
+                   isinstance(validation_data, Sequence))
         if val_gen and not validation_steps:
             raise ValueError('When using a generator for validation data, '
                              'you must specify a value for '
                              '`validation_steps`.')
 
-        out_labels = self.metrics_names
+        # Prepare display labels.
+        out_labels = self._get_deduped_metrics_names()
         callback_metrics = out_labels + ['val_' + n for n in out_labels]
 
         # prepare callbacks
@@ -1816,19 +1758,36 @@ class Model(Container):
             elif len(validation_data) == 3:
                 val_x, val_y, val_sample_weight = validation_data
             else:
-                raise ValueError('validation_data should be a tuple '
+                raise ValueError('`validation_data` should be a tuple '
                                  '`(val_x, val_y, val_sample_weight)` '
                                  'or `(val_x, val_y)`. Found: ' +
                                  str(validation_data))
             val_x, val_y, val_sample_weights = self._standardize_user_data(
                 val_x, val_y, val_sample_weight)
+            val_data = val_x + val_y + val_sample_weights
+            if self.uses_learning_phase and not isinstance(K.learning_phase(), int):
+                val_data += [0.]
             for cbk in callbacks:
-                cbk.validation_data = val_x + [val_y, val_sample_weights]
+                cbk.validation_data = val_data
+        is_sequence = isinstance(generator, Sequence)
+        if not is_sequence and use_multiprocessing and workers > 1:
+            warnings.warn(
+                UserWarning('Using a generator with `use_multiprocessing=True`'
+                            ' and multiple workers may duplicate your data.'
+                            ' Please consider using the`keras.utils.Sequence'
+                            ' class.'))
         enqueuer = None
 
         try:
-            enqueuer = GeneratorEnqueuer(generator, pickle_safe=pickle_safe)
-            enqueuer.start(max_q_size=max_q_size, workers=workers)
+            if is_sequence:
+                enqueuer = OrderedEnqueuer(generator,
+                                           use_multiprocessing=use_multiprocessing)
+            else:
+                enqueuer = GeneratorEnqueuer(generator,
+                                             use_multiprocessing=use_multiprocessing,
+                                             wait_time=wait_time)
+            enqueuer.start(workers=workers, max_queue_size=max_queue_size)
+            output_generator = enqueuer.get()
 
             callback_model.stop_training = False
             while epoch < epochs:
@@ -1836,16 +1795,10 @@ class Model(Container):
                 steps_done = 0
                 batch_index = 0
                 while steps_done < steps_per_epoch:
-                    generator_output = None
-                    while enqueuer.is_running():
-                        if not enqueuer.queue.empty():
-                            generator_output = enqueuer.queue.get()
-                            break
-                        else:
-                            time.sleep(wait_time)
+                    generator_output = next(output_generator)
 
                     if not hasattr(generator_output, '__len__'):
-                        raise ValueError('output of generator should be '
+                        raise ValueError('Output of generator should be '
                                          'a tuple `(x, y, sample_weight)` '
                                          'or `(x, y)`. Found: ' +
                                          str(generator_output))
@@ -1855,7 +1808,7 @@ class Model(Container):
                     elif len(generator_output) == 3:
                         x, y, sample_weight = generator_output
                     else:
-                        raise ValueError('output of generator should be '
+                        raise ValueError('Output of generator should be '
                                          'a tuple `(x, y, sample_weight)` '
                                          'or `(x, y)`. Found: ' +
                                          str(generator_output))
@@ -1893,9 +1846,9 @@ class Model(Container):
                             val_outs = self.evaluate_generator(
                                 validation_data,
                                 validation_steps,
-                                max_q_size=max_q_size,
+                                max_queue_size=max_queue_size,
                                 workers=workers,
-                                pickle_safe=pickle_safe)
+                                use_multiprocessing=use_multiprocessing)
                         else:
                             # No need for try/except because
                             # data has already been validated.
@@ -1924,21 +1877,26 @@ class Model(Container):
 
     @interfaces.legacy_generator_methods_support
     def evaluate_generator(self, generator, steps,
-                           max_q_size=10, workers=1, pickle_safe=False):
+                           max_queue_size=10,
+                           workers=1,
+                           use_multiprocessing=False):
         """Evaluates the model on a data generator.
 
         The generator should return the same kind of data
         as accepted by `test_on_batch`.
 
-        Arguments:
+        # Arguments
             generator: Generator yielding tuples (inputs, targets)
                 or (inputs, targets, sample_weights)
+                or an instance of Sequence (keras.utils.Sequence)
+                    object in order to avoid duplicate data
+                    when using multiprocessing.
             steps: Total number of steps (batches of samples)
                 to yield from `generator` before stopping.
-            max_q_size: maximum size for the generator queue
+            max_queue_size: maximum size for the generator queue
             workers: maximum number of processes to spin up
                 when using process based threading
-            pickle_safe: if True, use process based threading.
+            use_multiprocessing: if True, use process based threading.
                 Note that because
                 this implementation relies on multiprocessing,
                 you should not pass
@@ -1962,23 +1920,30 @@ class Model(Container):
         wait_time = 0.01
         all_outs = []
         batch_sizes = []
+        is_sequence = isinstance(generator, Sequence)
+        if not is_sequence and use_multiprocessing and workers > 1:
+            warnings.warn(
+                UserWarning('Using a generator with `use_multiprocessing=True`'
+                            ' and multiple workers may duplicate your data.'
+                            ' Please consider using the`keras.utils.Sequence'
+                            ' class.'))
         enqueuer = None
 
         try:
-            enqueuer = GeneratorEnqueuer(generator, pickle_safe=pickle_safe)
-            enqueuer.start(workers=workers, max_q_size=max_q_size)
+            if is_sequence:
+                enqueuer = OrderedEnqueuer(generator,
+                                           use_multiprocessing=use_multiprocessing)
+            else:
+                enqueuer = GeneratorEnqueuer(generator,
+                                             use_multiprocessing=use_multiprocessing,
+                                             wait_time=wait_time)
+            enqueuer.start(workers=workers, max_queue_size=max_queue_size)
+            output_generator = enqueuer.get()
 
             while steps_done < steps:
-                generator_output = None
-                while enqueuer.is_running():
-                    if not enqueuer.queue.empty():
-                        generator_output = enqueuer.queue.get()
-                        break
-                    else:
-                        time.sleep(wait_time)
-
+                generator_output = next(output_generator)
                 if not hasattr(generator_output, '__len__'):
-                    raise ValueError('output of generator should be a tuple '
+                    raise ValueError('Output of generator should be a tuple '
                                      '(x, y, sample_weight) '
                                      'or (x, y). Found: ' +
                                      str(generator_output))
@@ -1988,7 +1953,7 @@ class Model(Container):
                 elif len(generator_output) == 3:
                     x, y, sample_weight = generator_output
                 else:
-                    raise ValueError('output of generator should be a tuple '
+                    raise ValueError('Output of generator should be a tuple '
                                      '(x, y, sample_weight) '
                                      'or (x, y). Found: ' +
                                      str(generator_output))
@@ -2000,6 +1965,9 @@ class Model(Container):
                     batch_size = len(list(x.values())[0])
                 else:
                     batch_size = len(x)
+                if batch_size == 0:
+                    raise ValueError('Received an empty batch. '
+                                     'Batches should at least contain one item.')
                 all_outs.append(outs)
 
                 steps_done += 1
@@ -2021,21 +1989,26 @@ class Model(Container):
 
     @interfaces.legacy_generator_methods_support
     def predict_generator(self, generator, steps,
-                          max_q_size=10, workers=1,
-                          pickle_safe=False, verbose=0):
+                          max_queue_size=10,
+                          workers=1,
+                          use_multiprocessing=False,
+                          verbose=0):
         """Generates predictions for the input samples from a data generator.
 
         The generator should return the same kind of data as accepted by
         `predict_on_batch`.
 
         # Arguments
-            generator: Generator yielding batches of input samples.
+            generator: Generator yielding batches of input samples
+                    or an instance of Sequence (keras.utils.Sequence)
+                    object in order to avoid duplicate data
+                    when using multiprocessing.
             steps: Total number of steps (batches of samples)
                 to yield from `generator` before stopping.
-            max_q_size: Maximum size for the generator queue.
+            max_queue_size: Maximum size for the generator queue.
             workers: Maximum number of processes to spin up
                 when using process based threading
-            pickle_safe: If `True`, use process based threading.
+            use_multiprocessing: If `True`, use process based threading.
                 Note that because
                 this implementation relies on multiprocessing,
                 you should not pass
@@ -2056,24 +2029,31 @@ class Model(Container):
         steps_done = 0
         wait_time = 0.01
         all_outs = []
+        is_sequence = isinstance(generator, Sequence)
+        if not is_sequence and use_multiprocessing and workers > 1:
+            warnings.warn(
+                UserWarning('Using a generator with `use_multiprocessing=True`'
+                            ' and multiple workers may duplicate your data.'
+                            ' Please consider using the`keras.utils.Sequence'
+                            ' class.'))
         enqueuer = None
 
         try:
-            enqueuer = GeneratorEnqueuer(generator, pickle_safe=pickle_safe)
-            enqueuer.start(workers=workers, max_q_size=max_q_size)
+            if is_sequence:
+                enqueuer = OrderedEnqueuer(generator,
+                                           use_multiprocessing=use_multiprocessing)
+            else:
+                enqueuer = GeneratorEnqueuer(generator,
+                                             use_multiprocessing=use_multiprocessing,
+                                             wait_time=wait_time)
+            enqueuer.start(workers=workers, max_queue_size=max_queue_size)
+            output_generator = enqueuer.get()
 
             if verbose == 1:
                 progbar = Progbar(target=steps)
 
             while steps_done < steps:
-                generator_output = None
-                while enqueuer.is_running():
-                    if not enqueuer.queue.empty():
-                        generator_output = enqueuer.queue.get()
-                        break
-                    else:
-                        time.sleep(wait_time)
-
+                generator_output = next(output_generator)
                 if isinstance(generator_output, tuple):
                     # Compatibility with the generators
                     # used for training.
@@ -2082,7 +2062,7 @@ class Model(Container):
                     elif len(generator_output) == 3:
                         x, _, _ = generator_output
                     else:
-                        raise ValueError('output of generator should be '
+                        raise ValueError('Output of generator should be '
                                          'a tuple `(x, y, sample_weight)` '
                                          'or `(x, y)`. Found: ' +
                                          str(generator_output))

keras/initializers.py

@@ -22,14 +22,16 @@ class Initializer(object):
 
 
 class Zeros(Initializer):
-    """Initializer that generates tensors initialized to 0."""
+    """Initializer that generates tensors initialized to 0.
+    """
 
     def __call__(self, shape, dtype=None):
         return K.constant(0, shape=shape, dtype=dtype)
 
 
 class Ones(Initializer):
-    """Initializer that generates tensors initialized to 1."""
+    """Initializer that generates tensors initialized to 1.
+    """
 
     def __call__(self, shape, dtype=None):
         return K.constant(1, shape=shape, dtype=dtype)
@@ -111,7 +113,7 @@ class RandomUniform(Initializer):
 class TruncatedNormal(Initializer):
     """Initializer that generates a truncated normal distribution.
 
-    These values are similar to values from a `random_normal_initializer`
+    These values are similar to values from a `RandomNormal`
     except that values more than two standard deviations from the mean
     are discarded and re-drawn. This is the recommended initializer for
     neural network weights and filters.
@@ -146,6 +148,7 @@ class VarianceScaling(Initializer):
 
     With `distribution="normal"`, samples are drawn from a truncated normal
     distribution centered on zero, with `stddev = sqrt(scale / n)` where n is:
+
         - number of input units in the weight tensor, if mode = "fan_in"
         - number of output units, if mode = "fan_out"
         - average of the numbers of input and output units, if mode = "fan_avg"
@@ -368,6 +371,29 @@ def he_normal(seed=None):
                            seed=seed)
 
 
+def lecun_normal(seed=None):
+    """LeCun normal initializer.
+
+    It draws samples from a truncated normal distribution centered on 0
+    with `stddev = sqrt(1 / fan_in)`
+    where `fan_in` is the number of input units in the weight tensor.
+
+    # Arguments
+        seed: A Python integer. Used to seed the random generator.
+
+    # Returns
+        An initializer.
+
+    # References
+        - [Self-Normalizing Neural Networks](https://arxiv.org/abs/1706.02515)
+        - [Efficient Backprop](http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf)
+    """
+    return VarianceScaling(scale=1.,
+                           mode='fan_in',
+                           distribution='normal',
+                           seed=seed)
+
+
 def he_uniform(seed=None):
     """He uniform variance scaling initializer.
 

keras/layers/__init__.py

@@ -21,6 +21,14 @@ from ..legacy.layers import *
 
 
 def serialize(layer):
+    """Serialize a layer.
+
+    # Arguments
+        layer: a Layer object.
+
+    # Returns
+        dictionary with config.
+    """
     return {'class_name': layer.__class__.__name__,
             'config': layer.get_config()}
 

keras/layers/advanced_activations.py

@@ -41,7 +41,7 @@ class LeakyReLU(Layer):
         return K.relu(inputs, alpha=self.alpha)
 
     def get_config(self):
-        config = {'alpha': self.alpha}
+        config = {'alpha': float(self.alpha)}
         base_config = super(LeakyReLU, self).get_config()
         return dict(list(base_config.items()) + list(config.items()))
 
@@ -104,7 +104,7 @@ class PReLU(Layer):
             for i in self.shared_axes:
                 param_shape[i - 1] = 1
                 self.param_broadcast[i - 1] = True
-        self.alpha = self.add_weight(param_shape,
+        self.alpha = self.add_weight(shape=param_shape,
                                      name='alpha',
                                      initializer=self.alpha_initializer,
                                      regularizer=self.alpha_regularizer,
@@ -202,7 +202,7 @@ class ThresholdedReLU(Layer):
         self.theta = K.cast_to_floatx(theta)
 
     def call(self, inputs, mask=None):
-        return inputs * K.cast(inputs > self.theta, K.floatx())
+        return inputs * K.cast(K.greater(inputs, self.theta), K.floatx())
 
     def get_config(self):
         config = {'theta': float(self.theta)}

keras/layers/convolutional.py

@@ -127,13 +127,13 @@ class _Conv(Layer):
         input_dim = input_shape[channel_axis]
         kernel_shape = self.kernel_size + (input_dim, self.filters)
 
-        self.kernel = self.add_weight(kernel_shape,
+        self.kernel = self.add_weight(shape=kernel_shape,
                                       initializer=self.kernel_initializer,
                                       name='kernel',
                                       regularizer=self.kernel_regularizer,
                                       constraint=self.kernel_constraint)
         if self.use_bias:
-            self.bias = self.add_weight((self.filters,),
+            self.bias = self.add_weight(shape=(self.filters,),
                                         initializer=self.bias_initializer,
                                         name='bias',
                                         regularizer=self.bias_regularizer,
@@ -256,8 +256,11 @@ class Conv1D(_Conv):
             Specifying any stride value != 1 is incompatible with specifying
             any `dilation_rate` value != 1.
         padding: One of `"valid"`, `"causal"` or `"same"` (case-insensitive).
+            `"valid"` means "no padding".
+            `"same"` results in padding the input such that
+            the output has the same length as the original input.
             `"causal"` results in causal (dilated) convolutions, e.g. output[t]
-            depends solely on input[:t-1]. Useful when modeling temporal data
+            does not depend on input[t+1:]. Useful when modeling temporal data
             where the model should not violate the temporal order.
             See [WaveNet: A Generative Model for Raw Audio, section 2.1](https://arxiv.org/abs/1609.03499).
         dilation_rate: an integer or tuple/list of a single integer, specifying
@@ -473,7 +476,7 @@ class Conv3D(_Conv):
     When using this layer as the first layer in a model,
     provide the keyword argument `input_shape`
     (tuple of integers, does not include the sample axis),
-    e.g. `input_shape=(128, 128, 128, 3)` for 128x128x128 volumes
+    e.g. `input_shape=(128, 128, 128, 1)` for 128x128x128 volumes
     with a single channel,
     in `data_format="channels_last"`.
 
@@ -481,7 +484,7 @@ class Conv3D(_Conv):
         filters: Integer, the dimensionality of the output space
             (i.e. the number output of filters in the convolution).
         kernel_size: An integer or tuple/list of 3 integers, specifying the
-            width and height of the 3D convolution window.
+            depth, height and width of the 3D convolution window.
             Can be a single integer to specify the same value for
             all spatial dimensions.
         strides: An integer or tuple/list of 3 integers,
@@ -604,7 +607,7 @@ class Conv2DTranspose(Conv2D):
 
     # Arguments
         filters: Integer, the dimensionality of the output space
-            (i.e. the number output of filters in the convolution).
+            (i.e. the number of output filters in the convolution).
         kernel_size: An integer or tuple/list of 2 integers, specifying the
             width and height of the 2D convolution window.
             Can be a single integer to specify the same value for
@@ -677,7 +680,7 @@ class Conv2DTranspose(Conv2D):
                  kernel_size,
                  strides=(1, 1),
                  padding='valid',
-                 data_format='channels_last',
+                 data_format=None,
                  activation=None,
                  use_bias=True,
                  kernel_initializer='glorot_uniform',
@@ -721,13 +724,13 @@ class Conv2DTranspose(Conv2D):
         input_dim = input_shape[channel_axis]
         kernel_shape = self.kernel_size + (self.filters, input_dim)
 
-        self.kernel = self.add_weight(kernel_shape,
+        self.kernel = self.add_weight(shape=kernel_shape,
                                       initializer=self.kernel_initializer,
                                       name='kernel',
                                       regularizer=self.kernel_regularizer,
                                       constraint=self.kernel_constraint)
         if self.use_bias:
-            self.bias = self.add_weight((self.filters,),
+            self.bias = self.add_weight(shape=(self.filters,),
                                         initializer=self.bias_initializer,
                                         name='bias',
                                         regularizer=self.bias_regularizer,
@@ -803,6 +806,237 @@ class Conv2DTranspose(Conv2D):
         return config
 
 
+class Conv3DTranspose(Conv3D):
+    """Transposed convolution layer (sometimes called Deconvolution).
+
+    The need for transposed convolutions generally arises
+    from the desire to use a transformation going in the opposite direction
+    of a normal convolution, i.e., from something that has the shape of the
+    output of some convolution to something that has the shape of its input
+    while maintaining a connectivity pattern that is compatible with
+    said convolution.
+
+    When using this layer as the first layer in a model,
+    provide the keyword argument `input_shape`
+    (tuple of integers, does not include the sample axis),
+    e.g. `input_shape=(128, 128, 128, 3)` for a 128x128x128 volume with 3 channels
+    if `data_format="channels_last"`.
+
+    # Arguments
+        filters: Integer, the dimensionality of the output space
+            (i.e. the number of output filters in the convolution).
+        kernel_size: An integer or tuple/list of 3 integers, specifying the
+            width and height of the 3D convolution window.
+            Can be a single integer to specify the same value for
+            all spatial dimensions.
+        strides: An integer or tuple/list of 3 integers,
+            specifying the strides of the convolution along the width and height.
+            Can be a single integer to specify the same value for
+            all spatial dimensions.
+            Specifying any stride value != 1 is incompatible with specifying
+            any `dilation_rate` value != 1.
+        padding: one of `"valid"` or `"same"` (case-insensitive).
+        data_format: A string,
+            one of `channels_last` (default) or `channels_first`.
+            The ordering of the dimensions in the inputs.
+            `channels_last` corresponds to inputs with shape
+            `(batch, depth, height, width, channels)` while `channels_first`
+            corresponds to inputs with shape
+            `(batch, channels, depth, height, width)`.
+            It defaults to the `image_data_format` value found in your
+            Keras config file at `~/.keras/keras.json`.
+            If you never set it, then it will be "channels_last".
+        dilation_rate: an integer or tuple/list of 3 integers, specifying
+            the dilation rate to use for dilated convolution.
+            Can be a single integer to specify the same value for
+            all spatial dimensions.
+            Currently, specifying any `dilation_rate` value != 1 is
+            incompatible with specifying any stride value != 1.
+        activation: Activation function to use
+            (see [activations](../activations.md)).
+            If you don't specify anything, no activation is applied
+            (ie. "linear" activation: `a(x) = x`).
+        use_bias: Boolean, whether the layer uses a bias vector.
+        kernel_initializer: Initializer for the `kernel` weights matrix
+            (see [initializers](../initializers.md)).
+        bias_initializer: Initializer for the bias vector
+            (see [initializers](../initializers.md)).
+        kernel_regularizer: Regularizer function applied to
+            the `kernel` weights matrix
+            (see [regularizer](../regularizers.md)).
+        bias_regularizer: Regularizer function applied to the bias vector
+            (see [regularizer](../regularizers.md)).
+        activity_regularizer: Regularizer function applied to
+            the output of the layer (its "activation").
+            (see [regularizer](../regularizers.md)).
+        kernel_constraint: Constraint function applied to the kernel matrix
+            (see [constraints](../constraints.md)).
+        bias_constraint: Constraint function applied to the bias vector
+            (see [constraints](../constraints.md)).
+
+    # Input shape
+        5D tensor with shape:
+        `(batch, channels, depth, rows, cols)` if data_format='channels_first'
+        or 5D tensor with shape:
+        `(batch, depth, rows, cols, channels)` if data_format='channels_last'.
+
+    # Output shape
+        5D tensor with shape:
+        `(batch, filters, new_depth, new_rows, new_cols)` if data_format='channels_first'
+        or 5D tensor with shape:
+        `(batch, new_depth, new_rows, new_cols, filters)` if data_format='channels_last'.
+        `depth` and `rows` and `cols` values might have changed due to padding.
+
+    # References
+        - [A guide to convolution arithmetic for deep learning](https://arxiv.org/abs/1603.07285v1)
+        - [Deconvolutional Networks](http://www.matthewzeiler.com/pubs/cvpr2010/cvpr2010.pdf)
+    """
+
+    def __init__(self, filters,
+                 kernel_size,
+                 strides=(1, 1, 1),
+                 padding='valid',
+                 data_format=None,
+                 activation=None,
+                 use_bias=True,
+                 kernel_initializer='glorot_uniform',
+                 bias_initializer='zeros',
+                 kernel_regularizer=None,
+                 bias_regularizer=None,
+                 activity_regularizer=None,
+                 kernel_constraint=None,
+                 bias_constraint=None,
+                 **kwargs):
+        super(Conv3DTranspose, self).__init__(
+            filters,
+            kernel_size,
+            strides=strides,
+            padding=padding,
+            data_format=data_format,
+            activation=activation,
+            use_bias=use_bias,
+            kernel_initializer=kernel_initializer,
+            bias_initializer=bias_initializer,
+            kernel_regularizer=kernel_regularizer,
+            bias_regularizer=bias_regularizer,
+            activity_regularizer=activity_regularizer,
+            kernel_constraint=kernel_constraint,
+            bias_constraint=bias_constraint,
+            **kwargs)
+        self.input_spec = InputSpec(ndim=5)
+
+    def build(self, input_shape):
+        if len(input_shape) != 5:
+            raise ValueError('Inputs should have rank ' +
+                             str(5) +
+                             '; Received input shape:', str(input_shape))
+        if self.data_format == 'channels_first':
+            channel_axis = 1
+        else:
+            channel_axis = -1
+        if input_shape[channel_axis] is None:
+            raise ValueError('The channel dimension of the inputs '
+                             'should be defined. Found `None`.')
+        input_dim = input_shape[channel_axis]
+        kernel_shape = self.kernel_size + (self.filters, input_dim)
+
+        self.kernel = self.add_weight(shape=kernel_shape,
+                                      initializer=self.kernel_initializer,
+                                      name='kernel',
+                                      regularizer=self.kernel_regularizer,
+                                      constraint=self.kernel_constraint)
+        if self.use_bias:
+            self.bias = self.add_weight(shape=(self.filters,),
+                                        initializer=self.bias_initializer,
+                                        name='bias',
+                                        regularizer=self.bias_regularizer,
+                                        constraint=self.bias_constraint)
+        else:
+            self.bias = None
+        # Set input spec.
+        self.input_spec = InputSpec(ndim=5, axes={channel_axis: input_dim})
+        self.built = True
+
+    def call(self, inputs):
+        input_shape = K.shape(inputs)
+        batch_size = input_shape[0]
+        if self.data_format == 'channels_first':
+            d_axis, h_axis, w_axis = 2, 3, 4
+        else:
+            d_axis, h_axis, w_axis = 1, 2, 3
+
+        depth = input_shape[d_axis]
+        height = input_shape[h_axis]
+        width = input_shape[w_axis]
+
+        kernel_d, kernel_h, kernel_w = self.kernel_size
+        stride_d, stride_h, stride_w = self.strides
+
+        # Infer the dynamic output shape:
+        out_depth = conv_utils.deconv_length(depth,
+                                             stride_d, kernel_d,
+                                             self.padding)
+        out_height = conv_utils.deconv_length(height,
+                                              stride_h, kernel_h,
+                                              self.padding)
+        out_width = conv_utils.deconv_length(width,
+                                             stride_w, kernel_w,
+                                             self.padding)
+
+        if self.data_format == 'channels_first':
+            output_shape = (batch_size, self.filters, out_depth, out_height, out_width)
+        else:
+            output_shape = (batch_size, out_depth, out_height, out_width, self.filters)
+
+        outputs = K.conv3d_transpose(inputs,
+                                     self.kernel,
+                                     output_shape,
+                                     self.strides,
+                                     padding=self.padding,
+                                     data_format=self.data_format)
+
+        if self.bias:
+            outputs = K.bias_add(
+                outputs,
+                self.bias,
+                data_format=self.data_format)
+
+        if self.activation is not None:
+            return self.activation(outputs)
+        return outputs
+
+    def compute_output_shape(self, input_shape):
+        output_shape = list(input_shape)
+        if self.data_format == 'channels_first':
+            c_axis, d_axis, h_axis, w_axis = 1, 2, 3, 4
+        else:
+            c_axis, d_axis, h_axis, w_axis = 4, 1, 2, 3
+
+        kernel_d, kernel_h, kernel_w = self.kernel_size
+        stride_d, stride_h, stride_w = self.strides
+
+        output_shape[c_axis] = self.filters
+        output_shape[d_axis] = conv_utils.deconv_length(output_shape[d_axis],
+                                                        stride_d,
+                                                        kernel_d,
+                                                        self.padding)
+        output_shape[h_axis] = conv_utils.deconv_length(output_shape[h_axis],
+                                                        stride_h,
+                                                        kernel_h,
+                                                        self.padding)
+        output_shape[w_axis] = conv_utils.deconv_length(output_shape[w_axis],
+                                                        stride_w,
+                                                        kernel_w,
+                                                        self.padding)
+
+        return tuple(output_shape)
+
+    def get_config(self):
+        config = super(Conv3DTranspose, self).get_config()
+        config.pop('dilation_rate')
+        return config
+
+
 class SeparableConv2D(Conv2D):
     """Depthwise separable 2D convolution.
 
@@ -952,20 +1186,20 @@ class SeparableConv2D(Conv2D):
                                   self.filters)
 
         self.depthwise_kernel = self.add_weight(
-            depthwise_kernel_shape,
+            shape=depthwise_kernel_shape,
             initializer=self.depthwise_initializer,
             name='depthwise_kernel',
             regularizer=self.depthwise_regularizer,
             constraint=self.depthwise_constraint)
         self.pointwise_kernel = self.add_weight(
-            pointwise_kernel_shape,
+            shape=pointwise_kernel_shape,
             initializer=self.pointwise_initializer,
             name='pointwise_kernel',
             regularizer=self.pointwise_regularizer,
             constraint=self.pointwise_constraint)
 
         if self.use_bias:
-            self.bias = self.add_weight((self.filters,),
+            self.bias = self.add_weight(shape=(self.filters,),
                                         initializer=self.bias_initializer,
                                         name='bias',
                                         regularizer=self.bias_regularizer,
@@ -1232,7 +1466,10 @@ class ZeroPadding1D(Layer):
         self.input_spec = InputSpec(ndim=3)
 
     def compute_output_shape(self, input_shape):
-        length = input_shape[1] + self.padding[0] + self.padding[1] if input_shape[1] is not None else None
+        if input_shape[1] is not None:
+            length = input_shape[1] + self.padding[0] + self.padding[1]
+        else:
+            length = None
         return (input_shape[0],
                 length,
                 input_shape[2])
@@ -1249,7 +1486,7 @@ class ZeroPadding1D(Layer):
 class ZeroPadding2D(Layer):
     """Zero-padding layer for 2D input (e.g. picture).
 
-    This layer can add rows and columns or zeros
+    This layer can add rows and columns of zeros
     at the top, bottom, left and right side of an image tensor.
 
     # Arguments
@@ -1259,7 +1496,7 @@ class ZeroPadding2D(Layer):
             - If tuple of 2 ints:
                 interpreted as two different
                 symmetric padding values for height and width:
-                `(symmetric_height_pad, symmetrc_width_pad)`.
+                `(symmetric_height_pad, symmetric_width_pad)`.
             - If tuple of 2 tuples of 2 ints:
                 interpreted as
                 `((top_pad, bottom_pad), (left_pad, right_pad))`
@@ -1318,15 +1555,27 @@ class ZeroPadding2D(Layer):
 
     def compute_output_shape(self, input_shape):
         if self.data_format == 'channels_first':
-            rows = input_shape[2] + self.padding[0][0] + self.padding[0][1] if input_shape[2] is not None else None
-            cols = input_shape[3] + self.padding[1][0] + self.padding[1][1] if input_shape[3] is not None else None
+            if input_shape[2] is not None:
+                rows = input_shape[2] + self.padding[0][0] + self.padding[0][1]
+            else:
+                rows = None
+            if input_shape[3] is not None:
+                cols = input_shape[3] + self.padding[1][0] + self.padding[1][1]
+            else:
+                cols = None
             return (input_shape[0],
                     input_shape[1],
                     rows,
                     cols)
         elif self.data_format == 'channels_last':
-            rows = input_shape[1] + self.padding[0][0] + self.padding[0][1] if input_shape[1] is not None else None
-            cols = input_shape[2] + self.padding[1][0] + self.padding[1][1] if input_shape[2] is not None else None
+            if input_shape[1] is not None:
+                rows = input_shape[1] + self.padding[0][0] + self.padding[0][1]
+            else:
+                rows = None
+            if input_shape[2] is not None:
+                cols = input_shape[2] + self.padding[1][0] + self.padding[1][1]
+            else:
+                cols = None
             return (input_shape[0],
                     rows,
                     cols,
@@ -1414,18 +1663,36 @@ class ZeroPadding3D(Layer):
 
     def compute_output_shape(self, input_shape):
         if self.data_format == 'channels_first':
-            dim1 = input_shape[2] + 2 * self.padding[0][0] if input_shape[2] is not None else None
-            dim2 = input_shape[3] + 2 * self.padding[1][0] if input_shape[3] is not None else None
-            dim3 = input_shape[4] + 2 * self.padding[2][0] if input_shape[4] is not None else None
+            if input_shape[2] is not None:
+                dim1 = input_shape[2] + self.padding[0][0] + self.padding[0][1]
+            else:
+                dim1 = None
+            if input_shape[3] is not None:
+                dim2 = input_shape[3] + self.padding[1][0] + self.padding[1][1]
+            else:
+                dim2 = None
+            if input_shape[4] is not None:
+                dim3 = input_shape[4] + self.padding[2][0] + self.padding[2][1]
+            else:
+                dim3 = None
             return (input_shape[0],
                     input_shape[1],
                     dim1,
                     dim2,
                     dim3)
         elif self.data_format == 'channels_last':
-            dim1 = input_shape[1] + 2 * self.padding[0][1] if input_shape[1] is not None else None
-            dim2 = input_shape[2] + 2 * self.padding[1][1] if input_shape[2] is not None else None
-            dim3 = input_shape[3] + 2 * self.padding[2][1] if input_shape[3] is not None else None
+            if input_shape[1] is not None:
+                dim1 = input_shape[1] + self.padding[0][0] + self.padding[0][1]
+            else:
+                dim1 = None
+            if input_shape[2] is not None:
+                dim2 = input_shape[2] + self.padding[1][0] + self.padding[1][1]
+            else:
+                dim2 = None
+            if input_shape[3] is not None:
+                dim3 = input_shape[3] + self.padding[2][0] + self.padding[2][1]
+            else:
+                dim3 = None
             return (input_shape[0],
                     dim1,
                     dim2,
@@ -1501,7 +1768,7 @@ class Cropping2D(Layer):
             - If tuple of 2 ints:
                 interpreted as two different
                 symmetric cropping values for height and width:
-                `(symmetric_height_crop, symmetrc_width_crop)`.
+                `(symmetric_height_crop, symmetric_width_crop)`.
             - If tuple of 2 tuples of 2 ints:
                 interpreted as
                 `((top_crop, bottom_crop), (left_crop, right_crop))`
@@ -1538,7 +1805,7 @@ class Cropping2D(Layer):
         model.add(Cropping2D(cropping=((2, 2), (4, 4)),
                              input_shape=(28, 28, 3)))
         # now model.output_shape == (None, 24, 20, 3)
-        model.add(Conv2D(64, (3, 3), padding='same))
+        model.add(Conv2D(64, (3, 3), padding='same'))
         model.add(Cropping2D(cropping=((2, 2), (2, 2))))
         # now model.output_shape == (None, 20, 16. 64)
     ```
@@ -1705,18 +1972,36 @@ class Cropping3D(Layer):
 
     def compute_output_shape(self, input_shape):
         if self.data_format == 'channels_first':
-            dim1 = input_shape[2] - self.cropping[0][0] - self.cropping[0][1] if input_shape[2] is not None else None
-            dim2 = input_shape[3] - self.cropping[1][0] - self.cropping[1][1] if input_shape[3] is not None else None
-            dim3 = input_shape[4] - self.cropping[2][0] - self.cropping[2][1] if input_shape[4] is not None else None
+            if input_shape[2] is not None:
+                dim1 = input_shape[2] - self.cropping[0][0] - self.cropping[0][1]
+            else:
+                dim1 = None
+            if input_shape[3] is not None:
+                dim2 = input_shape[3] - self.cropping[1][0] - self.cropping[1][1]
+            else:
+                dim2 = None
+            if input_shape[4] is not None:
+                dim3 = input_shape[4] - self.cropping[2][0] - self.cropping[2][1]
+            else:
+                dim3 = None
             return (input_shape[0],
                     input_shape[1],
                     dim1,
                     dim2,
                     dim3)
         elif self.data_format == 'channels_last':
-            dim1 = input_shape[1] - self.cropping[0][0] - self.cropping[0][1] if input_shape[1] is not None else None
-            dim2 = input_shape[2] - self.cropping[1][0] - self.cropping[1][1] if input_shape[2] is not None else None
-            dim3 = input_shape[3] - self.cropping[2][0] - self.cropping[2][1] if input_shape[3] is not None else None
+            if input_shape[1] is not None:
+                dim1 = input_shape[1] - self.cropping[0][0] - self.cropping[0][1]
+            else:
+                dim1 = None
+            if input_shape[2] is not None:
+                dim2 = input_shape[2] - self.cropping[1][0] - self.cropping[1][1]
+            else:
+                dim2 = None
+            if input_shape[3] is not None:
+                dim3 = input_shape[3] - self.cropping[2][0] - self.cropping[2][1]
+            else:
+                dim3 = None
             return (input_shape[0],
                     dim1,
                     dim2,
@@ -1837,6 +2122,7 @@ Convolution3D = Conv3D
 SeparableConvolution2D = SeparableConv2D
 Convolution2DTranspose = Conv2DTranspose
 Deconvolution2D = Deconv2D = Conv2DTranspose
+Deconvolution3D = Deconv3D = Conv3DTranspose
 
 # Legacy aliases
 AtrousConv1D = AtrousConvolution1D

keras/layers/convolutional_recurrent.py

@@ -105,9 +105,12 @@ class ConvRecurrent2D(Recurrent):
         self.return_sequences = return_sequences
         self.go_backwards = go_backwards
         self.stateful = stateful
-        self.input_spec = InputSpec(ndim=5)
+        self.input_spec = [InputSpec(ndim=5)]
+        self.state_spec = None
 
     def compute_output_shape(self, input_shape):
+        if isinstance(input_shape, list):
+            input_shape = input_shape[0]
         if self.data_format == 'channels_first':
             rows = input_shape[3]
             cols = input_shape[4]
@@ -328,11 +331,13 @@ class ConvLSTM2D(ConvRecurrent2D):
 
         self.dropout = min(1., max(0., dropout))
         self.recurrent_dropout = min(1., max(0., recurrent_dropout))
+        self.state_spec = [InputSpec(ndim=4), InputSpec(ndim=4)]
 
     def build(self, input_shape):
-        # TODO: better handling of input spec
-        self.input_spec = InputSpec(shape=input_shape)
-
+        if isinstance(input_shape, list):
+            input_shape = input_shape[0]
+        batch_size = input_shape[0] if self.stateful else None
+        self.input_spec[0] = InputSpec(shape=(batch_size, None) + input_shape[2:])
         if self.stateful:
             self.reset_states()
         else:
@@ -340,30 +345,34 @@ class ConvLSTM2D(ConvRecurrent2D):
             self.states = [None, None]
 
         if self.data_format == 'channels_first':
-            channel_axis = 1
+            channel_axis = 2
         else:
             channel_axis = -1
         if input_shape[channel_axis] is None:
             raise ValueError('The channel dimension of the inputs '
                              'should be defined. Found `None`.')
         input_dim = input_shape[channel_axis]
+        state_shape = [None] * 4
+        state_shape[channel_axis] = input_dim
+        state_shape = tuple(state_shape)
+        self.state_spec = [InputSpec(shape=state_shape), InputSpec(shape=state_shape)]
         kernel_shape = self.kernel_size + (input_dim, self.filters * 4)
         self.kernel_shape = kernel_shape
         recurrent_kernel_shape = self.kernel_size + (self.filters, self.filters * 4)
 
-        self.kernel = self.add_weight(kernel_shape,
+        self.kernel = self.add_weight(shape=kernel_shape,
                                       initializer=self.kernel_initializer,
                                       name='kernel',
                                       regularizer=self.kernel_regularizer,
                                       constraint=self.kernel_constraint)
         self.recurrent_kernel = self.add_weight(
-            recurrent_kernel_shape,
+            shape=recurrent_kernel_shape,
             initializer=self.recurrent_initializer,
             name='recurrent_kernel',
             regularizer=self.recurrent_regularizer,
             constraint=self.recurrent_constraint)
         if self.use_bias:
-            self.bias = self.add_weight((self.filters * 4,),
+            self.bias = self.add_weight(shape=(self.filters * 4,),
                                         initializer=self.bias_initializer,
                                         name='bias',
                                         regularizer=self.bias_regularizer,
@@ -396,7 +405,7 @@ class ConvLSTM2D(ConvRecurrent2D):
             self.bias_o = None
         self.built = True
 
-    def get_initial_states(self, inputs):
+    def get_initial_state(self, inputs):
         # (samples, timesteps, rows, cols, filters)
         initial_state = K.zeros_like(inputs)
         # (samples, rows, cols, filters)
@@ -413,7 +422,7 @@ class ConvLSTM2D(ConvRecurrent2D):
     def reset_states(self):
         if not self.stateful:
             raise RuntimeError('Layer must be stateful.')
-        input_shape = self.input_spec.shape
+        input_shape = self.input_spec[0].shape
         output_shape = self.compute_output_shape(input_shape)
         if not input_shape[0]:
             raise ValueError('If a RNN is stateful, a complete '

keras/layers/core.py

@@ -5,7 +5,6 @@ from __future__ import division
 import numpy as np
 
 import copy
-import inspect
 import types as python_types
 import warnings
 
@@ -19,6 +18,7 @@ from ..engine import Layer
 from ..utils.generic_utils import func_dump
 from ..utils.generic_utils import func_load
 from ..utils.generic_utils import deserialize_keras_object
+from ..utils.generic_utils import has_arg
 from ..legacy import interfaces
 
 
@@ -73,7 +73,7 @@ class Dropout(Layer):
     """Applies Dropout to the input.
 
     Dropout consists in randomly setting
-    a fraction `p` of input units to 0 at each update during training time,
+    a fraction `rate` of input units to 0 at each update during training time,
     which helps prevent overfitting.
 
     # Arguments
@@ -129,7 +129,7 @@ class SpatialDropout1D(Dropout):
     between feature maps and should be used instead.
 
     # Arguments
-        p: float between 0 and 1. Fraction of the input units to drop.
+        rate: float between 0 and 1. Fraction of the input units to drop.
 
     # Input shape
         3D tensor with shape:
@@ -193,8 +193,8 @@ class SpatialDropout2D(Dropout):
         if data_format is None:
             data_format = K.image_data_format()
         if data_format not in {'channels_last', 'channels_first'}:
-            raise ValueError('data_format must be in '
-                             '{"channels_last", "channels_first"}')
+            raise ValueError('`data_format` must be in '
+                             '{`"channels_last"`, `"channels_first"`}')
         self.data_format = data_format
         self.input_spec = InputSpec(ndim=4)
 
@@ -202,10 +202,8 @@ class SpatialDropout2D(Dropout):
         input_shape = K.shape(inputs)
         if self.data_format == 'channels_first':
             noise_shape = (input_shape[0], input_shape[1], 1, 1)
-        elif self.data_format == 'channels_last':
-            noise_shape = (input_shape[0], 1, 1, input_shape[3])
         else:
-            raise ValueError('Invalid data_format:', self.data_format)
+            noise_shape = (input_shape[0], 1, 1, input_shape[3])
         return noise_shape
 
 
@@ -248,8 +246,8 @@ class SpatialDropout3D(Dropout):
         if data_format is None:
             data_format = K.image_data_format()
         if data_format not in {'channels_last', 'channels_first'}:
-            raise ValueError('data_format must be in '
-                             '{"channels_last", "channels_first"}')
+            raise ValueError('`data_format` must be in '
+                             '{`"channels_last"`, `"channels_first"`}')
         self.data_format = data_format
         self.input_spec = InputSpec(ndim=5)
 
@@ -257,10 +255,8 @@ class SpatialDropout3D(Dropout):
         input_shape = K.shape(inputs)
         if self.data_format == 'channels_first':
             noise_shape = (input_shape[0], input_shape[1], 1, 1, 1)
-        elif self.data_format == 'channels_last':
-            noise_shape = (input_shape[0], 1, 1, 1, input_shape[4])
         else:
-            raise ValueError('Invalid data_format:', self.data_format)
+            noise_shape = (input_shape[0], 1, 1, 1, input_shape[4])
         return noise_shape
 
 
@@ -299,13 +295,13 @@ class Reshape(Layer):
     """Reshapes an output to a certain shape.
 
     # Arguments
-        target_shape: target shape. Tuple of integers,
-            does not include the samples dimension (batch size).
+        target_shape: target shape. Tuple of integers.
+            Does not include the batch axis.
 
     # Input shape
         Arbitrary, although all dimensions in the input shaped must be fixed.
         Use the keyword argument `input_shape`
-        (tuple of integers, does not include the samples axis)
+        (tuple of integers, does not include the batch axis)
         when using this layer as the first layer in a model.
 
     # Output shape
@@ -335,27 +331,22 @@ class Reshape(Layer):
         self.target_shape = tuple(target_shape)
 
     def _fix_unknown_dimension(self, input_shape, output_shape):
-        """Find and replace a missing dimension in an output shape.
+        """Finds and replaces a missing dimension in an output shape.
 
         This is a near direct port of the internal Numpy function
         `_fix_unknown_dimension` in `numpy/core/src/multiarray/shape.c`
 
         # Arguments
-            input_shape: shape of array being reshaped
-            output_shape: desired shape of the array with at most
+            input_shape: original shape of array being reshaped
+            output_shape: target shape of the array, with at most
                 a single -1 which indicates a dimension that should be
                 derived from the input shape.
 
         # Returns
-            The new output shape with a -1 replaced with its computed value.
-
-            Raises a ValueError if the total array size of the output_shape is
-            different then the input_shape, or more then one unknown dimension
-            is specified.
+            The new output shape with a `-1` replaced with its computed value.
 
         # Raises
-            ValueError: in case of invalid values
-                for `input_shape` or `input_shape`.
+            ValueError: if `input_shape` and `output_shape` do not match.
         """
         output_shape = list(output_shape)
         msg = 'total size of new array must be unchanged'
@@ -386,13 +377,11 @@ class Reshape(Layer):
 
     def call(self, inputs):
         # In case the target shape is not fully defined,
-        # we need access to the shape of x.
-        # solution:
-        # 1) rely on x._keras_shape
-        # 2) fallback: K.int_shape
+        # we need access to the shape of `inputs`.
+        # solution: rely on `K.int_shape`.
         target_shape = self.target_shape
         if -1 in target_shape:
-            # target shape not fully defined
+            # Target shape not fully defined.
             input_shape = None
             try:
                 input_shape = K.int_shape(inputs)
@@ -467,7 +456,7 @@ class Flatten(Layer):
 
     ```python
         model = Sequential()
-        model.add(Convolution2D(64, 3, 3,
+        model.add(Conv2D(64, 3, 3,
                                 border_mode='same',
                                 input_shape=(3, 32, 32)))
         # now: model.output_shape == (None, 64, 32, 32)
@@ -648,13 +637,12 @@ class Lambda(Layer):
         else:
             shape = self._output_shape(input_shape)
             if not isinstance(shape, (list, tuple)):
-                raise ValueError('output_shape function must return a tuple')
+                raise ValueError('`output_shape` function must return a tuple.')
             return tuple(shape)
 
     def call(self, inputs, mask=None):
         arguments = self.arguments
-        arg_spec = inspect.getargspec(self.function)
-        if 'mask' in arg_spec.args:
+        if has_arg(self.function, 'mask'):
             arguments['mask'] = mask
         return self.function(inputs, **arguments)
 
@@ -720,6 +708,16 @@ class Lambda(Layer):
         else:
             output_shape = config['output_shape']
 
+        # If arguments were numpy array, they have been saved as
+        # list. We need to recover the ndarray
+        if 'arguments' in config:
+            for key in config['arguments']:
+                if isinstance(config['arguments'][key], dict):
+                    arg_dict = config['arguments'][key]
+                    if 'type' in arg_dict and arg_dict['type'] == 'ndarray':
+                        # Overwrite the argument with its numpy translation
+                        config['arguments'][key] = np.array(arg_dict['value'])
+
         config['function'] = function
         config['output_shape'] = output_shape
         return cls(**config)
@@ -820,13 +818,13 @@ class Dense(Layer):
         assert len(input_shape) >= 2
         input_dim = input_shape[-1]
 
-        self.kernel = self.add_weight((input_dim, self.units),
+        self.kernel = self.add_weight(shape=(input_dim, self.units),
                                       initializer=self.kernel_initializer,
                                       name='kernel',
                                       regularizer=self.kernel_regularizer,
                                       constraint=self.kernel_constraint)
         if self.use_bias:
-            self.bias = self.add_weight((self.units,),
+            self.bias = self.add_weight(shape=(self.units,),
                                         initializer=self.bias_initializer,
                                         name='bias',
                                         regularizer=self.bias_regularizer,

keras/layers/embeddings.py

@@ -31,17 +31,17 @@ class Embedding(Layer):
     ```
 
     # Arguments
-      input_dim: int > 0. Size of the vocabulary, ie.
-          1 + maximum integer index occurring in the input data.
+      input_dim: int > 0. Size of the vocabulary,
+          i.e. maximum integer index + 1.
       output_dim: int >= 0. Dimension of the dense embedding.
       embeddings_initializer: Initializer for the `embeddings` matrix
-            (see [initializers](../initializers.md)).
+          (see [initializers](../initializers.md)).
       embeddings_regularizer: Regularizer function applied to
-            the `embeddings` matrix
-            (see [regularizer](../regularizers.md)).
+          the `embeddings` matrix
+          (see [regularizer](../regularizers.md)).
       embeddings_constraint: Constraint function applied to
-            the `embeddings` matrix
-            (see [constraints](../constraints.md)).
+          the `embeddings` matrix
+          (see [constraints](../constraints.md)).
       mask_zero: Whether or not the input value 0 is a special "padding"
           value that should be masked out.
           This is useful when using [recurrent layers](recurrent.md)
@@ -49,7 +49,8 @@ class Embedding(Layer):
           If this is `True` then all subsequent layers
           in the model need to support masking or an exception will be raised.
           If mask_zero is set to True, as a consequence, index 0 cannot be
-          used in the vocabulary (input_dim should equal `|vocabulary| + 2`).
+          used in the vocabulary (input_dim should equal size of
+          vocabulary + 1).
       input_length: Length of input sequences, when it is constant.
           This argument is required if you are going to connect
           `Flatten` then `Dense` layers upstream
@@ -74,7 +75,6 @@ class Embedding(Layer):
                  mask_zero=False,
                  input_length=None,
                  **kwargs):
-        kwargs['dtype'] = 'int32'
         if 'input_shape' not in kwargs:
             if input_length:
                 kwargs['input_shape'] = (input_length,)
@@ -93,11 +93,12 @@ class Embedding(Layer):
 
     def build(self, input_shape):
         self.embeddings = self.add_weight(
-            (self.input_dim, self.output_dim),
+            shape=(self.input_dim, self.output_dim),
             initializer=self.embeddings_initializer,
             name='embeddings',
             regularizer=self.embeddings_regularizer,
-            constraint=self.embeddings_constraint)
+            constraint=self.embeddings_constraint,
+            dtype=self.dtype)
         self.built = True
 
     def compute_mask(self, inputs, mask=None):
@@ -107,11 +108,25 @@ class Embedding(Layer):
             return K.not_equal(inputs, 0)
 
     def compute_output_shape(self, input_shape):
-        if not self.input_length:
-            input_length = input_shape[1]
+        if self.input_length is None:
+            return input_shape + (self.output_dim,)
         else:
-            input_length = self.input_length
-        return (input_shape[0], input_length, self.output_dim)
+            # input_length can be tuple if input is 3D or higher
+            if isinstance(self.input_length, (list, tuple)):
+                in_lens = list(self.input_length)
+            else:
+                in_lens = [self.input_length]
+            if len(in_lens) != len(input_shape) - 1:
+                ValueError('"input_length" is %s, but received input has shape %s' %
+                           (str(self.input_length), str(input_shape)))
+            else:
+                for i, (s1, s2) in enumerate(zip(in_lens, input_shape[1:])):
+                    if s1 is not None and s2 is not None and s1 != s2:
+                        ValueError('"input_length" is %s, but received input has shape %s' %
+                                   (str(self.input_length), str(input_shape)))
+                    elif s1 is None:
+                        in_lens[i] = s2
+            return (input_shape[0],) + tuple(in_lens) + (self.output_dim,)
 
     def call(self, inputs):
         if K.dtype(inputs) != 'int32':

keras/layers/local.py

@@ -41,7 +41,8 @@ class LocallyConnected1D(Layer):
             specifying the stride length of the convolution.
             Specifying any stride value != 1 is incompatible with specifying
             any `dilation_rate` value != 1.
-        padding: One of `"valid"` or `"same"` (case-insensitive).
+        padding: Currently only supports `"valid"` (case-insensitive).
+            `"same"` may be supported in the future.
         activation: Activation function to use
             (see [activations](../activations.md)).
             If you don't specify anything, no activation is applied
@@ -121,14 +122,14 @@ class LocallyConnected1D(Layer):
                              self.kernel_size[0] * input_dim,
                              self.filters)
         self.kernel = self.add_weight(
-            self.kernel_shape,
+            shape=self.kernel_shape,
             initializer=self.kernel_initializer,
             name='kernel',
             regularizer=self.kernel_regularizer,
             constraint=self.kernel_constraint)
         if self.use_bias:
             self.bias = self.add_weight(
-                (output_length, self.filters),
+                shape=(output_length, self.filters),
                 initializer=self.bias_initializer,
                 name='bias',
                 regularizer=self.bias_regularizer,
@@ -146,22 +147,11 @@ class LocallyConnected1D(Layer):
         return (input_shape[0], length, self.filters)
 
     def call(self, inputs):
-        stride = self.strides[0]
-        output_length, feature_dim, filters = self.kernel_shape
-
-        xs = []
-        for i in range(output_length):
-            slice_length = slice(i * stride,
-                                 i * stride + self.kernel_size[0])
-            xs.append(K.reshape(inputs[:, slice_length, :],
-                                (1, -1, feature_dim)))
-        x_aggregate = K.concatenate(xs, axis=0)
-        # Shape: `(output_length, batch_size, filters)`.
-        output = K.batch_dot(x_aggregate, self.kernel)
-        output = K.permute_dimensions(output, (1, 0, 2))
+        output_length, _, filters = self.kernel_shape
 
+        output = K.local_conv1d(inputs, self.kernel, self.kernel_size, self.strides)
         if self.use_bias:
-            output += K.reshape(self.bias, (1, output_length, filters))
+            output = K.bias_add(output, self.bias)
         if self.activation is not None:
             output = self.activation(output)
         return output
@@ -219,9 +209,8 @@ class LocallyConnected2D(Layer):
             specifying the strides of the convolution along the width and height.
             Can be a single integer to specify the same value for
             all spatial dimensions.
-            Specifying any stride value != 1 is incompatible with specifying
-            any `dilation_rate` value != 1.
-        padding: one of `"valid"` or `"same"` (case-insensitive).
+        padding: Currently only support `"valid"` (case-insensitive).
+            `"same"` will be supported in future.
         data_format: A string,
             one of `channels_last` (default) or `channels_first`.
             The ordering of the dimensions in the inputs.
@@ -325,13 +314,13 @@ class LocallyConnected2D(Layer):
         self.kernel_shape = (output_row * output_col,
                              self.kernel_size[0] * self.kernel_size[1] * input_filter,
                              self.filters)
-        self.kernel = self.add_weight(self.kernel_shape,
+        self.kernel = self.add_weight(shape=self.kernel_shape,
                                       initializer=self.kernel_initializer,
                                       name='kernel',
                                       regularizer=self.kernel_regularizer,
                                       constraint=self.kernel_constraint)
         if self.use_bias:
-            self.bias = self.add_weight((output_row, output_col, self.filters),
+            self.bias = self.add_weight(shape=(output_row, output_col, self.filters),
                                         initializer=self.bias_initializer,
                                         name='bias',
                                         regularizer=self.bias_regularizer,
@@ -363,62 +352,19 @@ class LocallyConnected2D(Layer):
             return (input_shape[0], rows, cols, self.filters)
 
     def call(self, inputs):
-        stride_row, stride_col = self.strides
-        _, feature_dim, filters = self.kernel_shape
+        _, _, filters = self.kernel_shape
 
-        if self.data_format == 'channels_first':
-            if K.backend() == 'theano':
-                output = []
-                for i in range(self.output_row):
-                    for j in range(self.output_col):
-                        slice_row = slice(i * stride_row,
-                                          i * stride_row + self.kernel_size[0])
-                        slice_col = slice(j * stride_col,
-                                          j * stride_col + self.kernel_size[1])
-                        x_flatten = K.reshape(inputs[:, :, slice_row, slice_col],
-                                              (1, -1, feature_dim))
-                        output.append(K.dot(x_flatten,
-                                      self.kernel[i * self.output_col + j, :, :]))
-                output = K.concatenate(output, axis=0)
-            else:
-                xs = []
-                for i in range(self.output_row):
-                    for j in range(self.output_col):
-                        slice_row = slice(i * stride_row,
-                                          i * stride_row + self.kernel_size[0])
-                        slice_col = slice(j * stride_col,
-                                          j * stride_col + self.kernel_size[1])
-                        xs.append(K.reshape(inputs[:, :, slice_row, slice_col],
-                                            (1, -1, feature_dim)))
-                x_aggregate = K.concatenate(xs, axis=0)
-                output = K.batch_dot(x_aggregate, self.kernel)
-            output = K.reshape(output,
-                               (self.output_row, self.output_col, -1, filters))
-            output = K.permute_dimensions(output, (2, 3, 0, 1))
-
-        elif self.data_format == 'channels_last':
-            xs = []
-            for i in range(self.output_row):
-                for j in range(self.output_col):
-                    slice_row = slice(i * stride_row,
-                                      i * stride_row + self.kernel_size[0])
-                    slice_col = slice(j * stride_col,
-                                      j * stride_col + self.kernel_size[1])
-                    xs.append(K.reshape(inputs[:, slice_row, slice_col, :],
-                                        (1, -1, feature_dim)))
-            x_aggregate = K.concatenate(xs, axis=0)
-            output = K.batch_dot(x_aggregate, self.kernel)
-            output = K.reshape(output,
-                               (self.output_row, self.output_col, -1, filters))
-            output = K.permute_dimensions(output, (2, 0, 1, 3))
+        output = K.local_conv2d(inputs,
+                                self.kernel,
+                                self.kernel_size,
+                                self.strides,
+                                (self.output_row, self.output_col),
+                                self.data_format)
 
         if self.use_bias:
-            if self.data_format == 'channels_first':
-                output += K.reshape(self.bias,
-                                    (1, filters, self.output_row, self.output_col))
-            elif self.data_format == 'channels_last':
-                output += K.reshape(self.bias,
-                                    (1, self.output_row, self.output_col, filters))
+            if self.data_format == 'channels_first' or self.data_format == 'channels_last':
+                output = K.bias_add(output, self.bias, data_format=self.data_format)
+
         output = self.activation(output)
         return output
 

keras/layers/merge.py

@@ -18,6 +18,44 @@ class _Merge(Layer):
     def _merge_function(self, inputs):
         raise NotImplementedError
 
+    def _compute_elemwise_op_output_shape(self, shape1, shape2):
+        """Computes the shape of the resultant of an elementwise operation.
+
+        # Arguments
+            shape1: tuple or None. Shape of the first tensor
+            shape2: tuple or None. Shape of the second tensor
+
+        # Returns
+            expected output shape when an element-wise operation is
+            carried out on 2 tensors with shapes shape1 and shape2.
+            tuple or None.
+
+        # Raises
+            ValueError: if shape1 and shape2 are not compatible for
+                element-wise operations.
+        """
+        if None in [shape1, shape2]:
+            return None
+        elif len(shape1) < len(shape2):
+            return self._compute_elemwise_op_output_shape(shape2, shape1)
+        elif len(shape2) == 0:
+            return shape1
+        output_shape = list(shape1[:-len(shape2)])
+        for i, j in zip(shape1[-len(shape2):], shape2):
+            if i is None or j is None:
+                output_shape.append(None)
+            elif i == 1:
+                output_shape.append(j)
+            elif j == 1:
+                output_shape.append(i)
+            else:
+                if i != j:
+                    raise ValueError('Operands could not be broadcast '
+                                     'together with shapes ' +
+                                     str(shape1) + ' ' + str(shape2))
+                output_shape.append(i)
+        return tuple(output_shape)
+
     def build(self, input_shape):
         # Used purely for shape validation.
         if not isinstance(input_shape, list):
@@ -27,26 +65,105 @@ class _Merge(Layer):
             raise ValueError('A merge layer should be called '
                              'on a list of at least 2 inputs. '
                              'Got ' + str(len(input_shape)) + ' inputs.')
-        if all([shape is None for shape in input_shape]):
-            return
-        # TODO: handle shapes with None entries.
-        input_shapes_set = set(input_shape)
-        if None in input_shapes_set:
-            input_shapes_set.remove(None)
-        if len(input_shapes_set) > 1:
-            raise ValueError('Only tensors of same shape can '
-                             'be merged by layer' + self.name +
-                             ' Got input shapes: %s' % input_shape)
+        batch_sizes = [s[0] for s in input_shape if s is not None]
+        batch_sizes = set(batch_sizes)
+        batch_sizes -= set([None])
+        if len(batch_sizes) > 1:
+            raise ValueError('Can not merge tensors with different '
+                             'batch sizes. Got tensors with shapes : ' +
+                             str(input_shape))
+        if input_shape[0] is None:
+            output_shape = None
+        else:
+            output_shape = input_shape[0][1:]
+        for i in range(1, len(input_shape)):
+            if input_shape[i] is None:
+                shape = None
+            else:
+                shape = input_shape[i][1:]
+            output_shape = self._compute_elemwise_op_output_shape(output_shape, shape)
+        # If the inputs have different ranks, we have to reshape them
+        # to make them broadcastable.
+        if None not in input_shape and len(set(map(len, input_shape))) == 1:
+            self._reshape_required = False
+        else:
+            self._reshape_required = True
 
     def call(self, inputs):
-        return self._merge_function(inputs)
+        if self._reshape_required:
+            reshaped_inputs = []
+            input_ndims = list(map(K.ndim, inputs))
+            if None not in input_ndims:
+                # If ranks of all inputs are available,
+                # we simply expand each of them at axis=1
+                # until all of them have the same rank.
+                max_ndim = max(input_ndims)
+                for x in inputs:
+                    x_ndim = K.ndim(x)
+                    for _ in range(max_ndim - x_ndim):
+                        x = K.expand_dims(x, 1)
+                    reshaped_inputs.append(x)
+                return self._merge_function(reshaped_inputs)
+            else:
+                # Transpose all inputs so that batch size is the last dimension.
+                # (batch_size, dim1, dim2, ... ) -> (dim1, dim2, ... , batch_size)
+                transposed = False
+                for x in inputs:
+                    x_ndim = K.ndim(x)
+                    if x_ndim is None:
+                        x_shape = K.shape(x)
+                        batch_size = x_shape[0]
+                        new_shape = K.concatenate([x_shape[1:], K.expand_dims(batch_size)])
+                        x_transposed = K.reshape(x, K.stack([batch_size, K.prod(x_shape[1:])]))
+                        x_transposed = K.permute_dimensions(x_transposed, (1, 0))
+                        x_transposed = K.reshape(x_transposed, new_shape)
+                        reshaped_inputs.append(x_transposed)
+                        transposed = True
+                    elif x_ndim > 1:
+                        dims = list(range(1, x_ndim)) + [0]
+                        reshaped_inputs.append(K.permute_dimensions(x, dims))
+                        transposed = True
+                    else:
+                        # We don't transpose inputs if they are 1D vectors or scalars.
+                        reshaped_inputs.append(x)
+                y = self._merge_function(reshaped_inputs)
+                y_ndim = K.ndim(y)
+                if transposed:
+                    # If inputs have been transposed, we have to transpose the output too.
+                    if y_ndim is None:
+                        y_shape = K.shape(y)
+                        y_ndim = K.shape(y_shape)[0]
+                        batch_size = y_shape[y_ndim - 1]
+                        new_shape = K.concatenate([K.expand_dims(batch_size), y_shape[:y_ndim - 1]])
+                        y = K.reshape(y, (-1, batch_size))
+                        y = K.permute_dimensions(y, (1, 0))
+                        y = K.reshape(y, new_shape)
+                    elif y_ndim > 1:
+                        dims = [y_ndim - 1] + list(range(y_ndim - 1))
+                        y = K.permute_dimensions(y, dims)
+                return y
+        else:
+            return self._merge_function(inputs)
 
     def compute_output_shape(self, input_shape):
-        # Layers that change the shape should already implement
-        # compute_output_shape anyway
-        # TODO: If the merge layer in the future accepts broadcastable inputs
-        #       then both this function and build should be changed
-        return input_shape[0]
+        if input_shape[0] is None:
+            output_shape = None
+        else:
+            output_shape = input_shape[0][1:]
+        for i in range(1, len(input_shape)):
+            if input_shape[i] is None:
+                shape = None
+            else:
+                shape = input_shape[i][1:]
+            output_shape = self._compute_elemwise_op_output_shape(output_shape, shape)
+        batch_sizes = [s[0] for s in input_shape if s is not None]
+        batch_sizes = set(batch_sizes)
+        batch_sizes -= set([None])
+        if len(batch_sizes) == 1:
+            output_shape = (list(batch_sizes)[0],) + output_shape
+        else:
+            output_shape = (None,) + output_shape
+        return output_shape
 
     def compute_mask(self, inputs, mask=None):
         if mask is None:

keras/layers/noise.py

@@ -5,6 +5,7 @@ from ..engine import Layer
 from .. import backend as K
 import numpy as np
 from ..legacy import interfaces
+from ..engine import InputSpec
 
 
 class GaussianNoise(Layer):
@@ -90,3 +91,70 @@ class GaussianDropout(Layer):
         config = {'rate': self.rate}
         base_config = super(GaussianDropout, self).get_config()
         return dict(list(base_config.items()) + list(config.items()))
+
+
+class AlphaDropout(Layer):
+    """Applies Alpha Dropout to the input.
+
+    Alpha Dropout is a `Dropout` that keeps mean and variance of inputs
+    to their original values, in order to ensure the self-normalizing property
+    even after this dropout.
+    Alpha Dropout fits well to Scaled Exponential Linear Units
+    by randomly setting activations to the negative saturation value.
+
+    # Arguments
+        rate: float, drop probability (as with `Dropout`).
+            The multiplicative noise will have
+            standard deviation `sqrt(rate / (1 - rate))`.
+        seed: A Python integer to use as random seed.
+
+    # Input shape
+        Arbitrary. Use the keyword argument `input_shape`
+        (tuple of integers, does not include the samples axis)
+        when using this layer as the first layer in a model.
+
+    # Output shape
+        Same shape as input.
+
+    # References
+        - [Self-Normalizing Neural Networks](https://arxiv.org/abs/1706.02515)
+    """
+    def __init__(self, rate, noise_shape=None, seed=None, **kwargs):
+        super(AlphaDropout, self).__init__(**kwargs)
+        self.rate = rate
+        self.noise_shape = noise_shape
+        self.seed = seed
+        self.supports_masking = True
+
+    def _get_noise_shape(self, inputs):
+        return self.noise_shape if self.noise_shape else K.shape(inputs)
+
+    def call(self, inputs, training=None):
+        if 0. < self.rate < 1.:
+            noise_shape = self._get_noise_shape(inputs)
+
+            def dropped_inputs(inputs=inputs, rate=self.rate, seed=self.seed):
+                alpha = 1.6732632423543772848170429916717
+                scale = 1.0507009873554804934193349852946
+                alpha_p = -alpha * scale
+
+                kept_idx = K.greater_equal(K.random_uniform(noise_shape, seed=seed), rate)
+                kept_idx = K.cast(kept_idx, K.floatx())
+
+                # Get affine transformation params
+                a = ((1 - rate) * (1 + rate * alpha_p ** 2)) ** -0.5
+                b = -a * alpha_p * rate
+
+                # Apply mask
+                x = inputs * kept_idx + alpha_p * (1 - kept_idx)
+
+                # Do affine transformation
+                return a * x + b
+
+            return K.in_train_phase(dropped_inputs, inputs, training=training)
+        return inputs
+
+    def get_config(self):
+        config = {'rate': self.rate}
+        base_config = super(AlphaDropout, self).get_config()
+        return dict(list(base_config.items()) + list(config.items()))

keras/layers/normalization.py

@@ -96,7 +96,7 @@ class BatchNormalization(Layer):
         shape = (dim,)
 
         if self.scale:
-            self.gamma = self.add_weight(shape,
+            self.gamma = self.add_weight(shape=shape,
                                          name='gamma',
                                          initializer=self.gamma_initializer,
                                          regularizer=self.gamma_regularizer,
@@ -104,7 +104,7 @@ class BatchNormalization(Layer):
         else:
             self.gamma = None
         if self.center:
-            self.beta = self.add_weight(shape,
+            self.beta = self.add_weight(shape=shape,
                                         name='beta',
                                         initializer=self.beta_initializer,
                                         regularizer=self.beta_regularizer,
@@ -112,12 +112,12 @@ class BatchNormalization(Layer):
         else:
             self.beta = None
         self.moving_mean = self.add_weight(
-            shape,
+            shape=shape,
             name='moving_mean',
             initializer=self.moving_mean_initializer,
             trainable=False)
         self.moving_variance = self.add_weight(
-            shape,
+            shape=shape,
             name='moving_variance',
             initializer=self.moving_variance_initializer,
             trainable=False)
@@ -133,57 +133,59 @@ class BatchNormalization(Layer):
         broadcast_shape[self.axis] = input_shape[self.axis]
 
         # Determines whether broadcasting is needed.
-        needs_broadcasting = (sorted(reduction_axes) != range(ndim)[:-1])
+        needs_broadcasting = (sorted(reduction_axes) != list(range(ndim))[:-1])
 
-        normed, mean, variance = K.normalize_batch_in_training(
+        def normalize_inference():
+            if needs_broadcasting:
+                # In this case we must explicitly broadcast all parameters.
+                broadcast_moving_mean = K.reshape(self.moving_mean,
+                                                  broadcast_shape)
+                broadcast_moving_variance = K.reshape(self.moving_variance,
+                                                      broadcast_shape)
+                if self.center:
+                    broadcast_beta = K.reshape(self.beta, broadcast_shape)
+                else:
+                    broadcast_beta = None
+                if self.scale:
+                    broadcast_gamma = K.reshape(self.gamma,
+                                                broadcast_shape)
+                else:
+                    broadcast_gamma = None
+                return K.batch_normalization(
+                    inputs,
+                    broadcast_moving_mean,
+                    broadcast_moving_variance,
+                    broadcast_beta,
+                    broadcast_gamma,
+                    epsilon=self.epsilon)
+            else:
+                return K.batch_normalization(
+                    inputs,
+                    self.moving_mean,
+                    self.moving_variance,
+                    self.beta,
+                    self.gamma,
+                    epsilon=self.epsilon)
+
+        # If the learning phase is *static* and set to inference:
+        if training in {0, False}:
+            return normalize_inference()
+
+        # If the learning is either dynamic, or set to training:
+        normed_training, mean, variance = K.normalize_batch_in_training(
             inputs, self.gamma, self.beta, reduction_axes,
             epsilon=self.epsilon)
 
-        if training in {0, False}:
-            return normed
-        else:
-            self.add_update([K.moving_average_update(self.moving_mean,
-                                                     mean,
-                                                     self.momentum),
-                             K.moving_average_update(self.moving_variance,
-                                                     variance,
-                                                     self.momentum)],
-                            inputs)
-
-            def normalize_inference():
-                if needs_broadcasting:
-                    # In this case we must explictly broadcast all parameters.
-                    broadcast_moving_mean = K.reshape(self.moving_mean,
-                                                      broadcast_shape)
-                    broadcast_moving_variance = K.reshape(self.moving_variance,
-                                                          broadcast_shape)
-                    if self.center:
-                        broadcast_beta = K.reshape(self.beta, broadcast_shape)
-                    else:
-                        broadcast_beta = None
-                    if self.scale:
-                        broadcast_gamma = K.reshape(self.gamma,
-                                                    broadcast_shape)
-                    else:
-                        broadcast_gamma = None
-                    return K.batch_normalization(
-                        inputs,
-                        broadcast_moving_mean,
-                        broadcast_moving_variance,
-                        broadcast_beta,
-                        broadcast_gamma,
-                        epsilon=self.epsilon)
-                else:
-                    return K.batch_normalization(
-                        inputs,
-                        self.moving_mean,
-                        self.moving_variance,
-                        self.beta,
-                        self.gamma,
-                        epsilon=self.epsilon)
+        self.add_update([K.moving_average_update(self.moving_mean,
+                                                 mean,
+                                                 self.momentum),
+                         K.moving_average_update(self.moving_variance,
+                                                 variance,
+                                                 self.momentum)],
+                        inputs)
 
         # Pick the normalized form corresponding to the training phase.
-        return K.in_train_phase(normed,
+        return K.in_train_phase(normed_training,
                                 normalize_inference,
                                 training=training)
 

keras/layers/recurrent.py

@@ -78,12 +78,16 @@ class Recurrent(Layer):
         # now model.output_shape == (None, 32)
         # note: `None` is the batch dimension.
 
-        # the following is identical:
-        model = Sequential()
-        model.add(LSTM(32, input_dim=64, input_length=10))
-
-        # for subsequent layers, not need to specify the input size:
+        # for subsequent layers, no need to specify the input size:
         model.add(LSTM(16))
+
+        # to stack recurrent layers, you must use return_sequences=True
+        # on any recurrent layer that feeds into another recurrent layer.
+        # note that you only need to specify the input size on the first layer.
+        model = Sequential()
+        model.add(LSTM(64, input_dim=64, input_length=10, return_sequences=True))
+        model.add(LSTM(32, return_sequences=True))
+        model.add(LSTM(10))
     ```
 
     # Arguments
@@ -92,8 +96,11 @@ class Recurrent(Layer):
             `[(input_dim, output_dim), (output_dim, output_dim), (output_dim,)]`.
         return_sequences: Boolean. Whether to return the last output
             in the output sequence, or the full sequence.
+        return_state: Boolean. Whether to return the last state
+            in addition to the output.
         go_backwards: Boolean (default False).
-            If True, process the input sequence backwards.
+            If True, process the input sequence backwards and return the
+            reversed sequence.
         stateful: Boolean (default False). If True, the last state
             for each sample at index i in a batch will be used as initial
             state for the sample of index i in the following batch.
@@ -134,6 +141,9 @@ class Recurrent(Layer):
         (Optional) 2D tensors with shape `(batch_size, output_dim)`.
 
     # Output shape
+        - if `return_state`: a list of tensors. The first tensor is
+            the output. The remaining tensors are the last states,
+            each with shape `(batch_size, units)`.
         - if `return_sequences`: 3D tensor with shape
             `(batch_size, timesteps, units)`.
         - else, 2D tensor with shape `(batch_size, units)`.
@@ -165,14 +175,20 @@ class Recurrent(Layer):
         To reset the states of your model, call `.reset_states()` on either
         a specific layer, or on your entire model.
 
-    # Note on specifying initial states in RNNs
-        You can specify the initial state of RNN layers by calling them with
-        the keyword argument `initial_state`. The value of `initial_state`
-        should be a tensor or list of tensors representing the initial state
-        of the RNN layer.
+    # Note on specifying the initial state of RNNs
+        You can specify the initial state of RNN layers symbolically by
+        calling them with the keyword argument `initial_state`. The value of
+        `initial_state` should be a tensor or list of tensors representing
+        the initial state of the RNN layer.
+
+        You can specify the initial state of RNN layers numerically by
+        calling `reset_states` with the keyword argument `states`. The value of
+        `states` should be a numpy array or list of numpy arrays representing
+        the initial state of the RNN layer.
     """
 
     def __init__(self, return_sequences=False,
+                 return_state=False,
                  go_backwards=False,
                  stateful=False,
                  unroll=False,
@@ -180,12 +196,16 @@ class Recurrent(Layer):
                  **kwargs):
         super(Recurrent, self).__init__(**kwargs)
         self.return_sequences = return_sequences
+        self.return_state = return_state
         self.go_backwards = go_backwards
+        if K.backend() == 'cntk' and stateful:
+            raise ValueError('Stateful RNN is not currently supported with CNTK.')
+
         self.stateful = stateful
         self.unroll = unroll
         self.implementation = implementation
         self.supports_masking = True
-        self.input_spec = InputSpec(ndim=3)
+        self.input_spec = [InputSpec(ndim=3)]
         self.state_spec = None
         self.dropout = 0
         self.recurrent_dropout = 0
@@ -193,16 +213,27 @@ class Recurrent(Layer):
     def compute_output_shape(self, input_shape):
         if isinstance(input_shape, list):
             input_shape = input_shape[0]
+
         if self.return_sequences:
-            return (input_shape[0], input_shape[1], self.units)
+            output_shape = (input_shape[0], input_shape[1], self.units)
         else:
-            return (input_shape[0], self.units)
+            output_shape = (input_shape[0], self.units)
+
+        if self.return_state:
+            state_shape = [(input_shape[0], self.units) for _ in self.states]
+            return [output_shape] + state_shape
+        else:
+            return output_shape
 
     def compute_mask(self, inputs, mask):
-        if self.return_sequences:
-            return mask
+        if isinstance(mask, list):
+            mask = mask[0]
+        output_mask = mask if self.return_sequences else None
+        if self.return_state:
+            state_mask = [None for _ in self.states]
+            return [output_mask] + state_mask
         else:
-            return None
+            return output_mask
 
     def step(self, inputs, states):
         raise NotImplementedError
@@ -210,14 +241,14 @@ class Recurrent(Layer):
     def get_constants(self, inputs, training=None):
         return []
 
-    def get_initial_states(self, inputs):
+    def get_initial_state(self, inputs):
         # build an all-zero tensor of shape (samples, output_dim)
         initial_state = K.zeros_like(inputs)  # (samples, timesteps, input_dim)
         initial_state = K.sum(initial_state, axis=(1, 2))  # (samples,)
         initial_state = K.expand_dims(initial_state)  # (samples, 1)
         initial_state = K.tile(initial_state, [1, self.units])  # (samples, output_dim)
-        initial_states = [initial_state for _ in range(len(self.states))]
-        return initial_states
+        initial_state = [initial_state for _ in range(len(self.states))]
+        return initial_state
 
     def preprocess_input(self, inputs, training=None):
         return inputs
@@ -227,51 +258,63 @@ class Recurrent(Layer):
         # and if it a Keras tensor,
         # then add it to the inputs and temporarily
         # modify the input spec to include the state.
-        if initial_state is not None:
-            if hasattr(initial_state, '_keras_history'):
-                # Compute the full input spec, including state
-                input_spec = self.input_spec
-                state_spec = self.state_spec
-                if not isinstance(state_spec, list):
-                    state_spec = [state_spec]
-                self.input_spec = [input_spec] + state_spec
+        if initial_state is None:
+            return super(Recurrent, self).__call__(inputs, **kwargs)
 
-                # Compute the full inputs, including state
-                if not isinstance(initial_state, (list, tuple)):
-                    initial_state = [initial_state]
-                inputs = [inputs] + list(initial_state)
+        if not isinstance(initial_state, (list, tuple)):
+            initial_state = [initial_state]
 
-                # Perform the call
-                output = super(Recurrent, self).__call__(inputs, **kwargs)
+        is_keras_tensor = hasattr(initial_state[0], '_keras_history')
+        for tensor in initial_state:
+            if hasattr(tensor, '_keras_history') != is_keras_tensor:
+                raise ValueError('The initial state of an RNN layer cannot be'
+                                 ' specified with a mix of Keras tensors and'
+                                 ' non-Keras tensors')
 
-                # Restore original input spec
-                self.input_spec = input_spec
-                return output
-            else:
-                kwargs['initial_state'] = initial_state
-        return super(Recurrent, self).__call__(inputs, **kwargs)
+        if is_keras_tensor:
+            # Compute the full input spec, including state
+            input_spec = self.input_spec
+            state_spec = self.state_spec
+            if not isinstance(input_spec, list):
+                input_spec = [input_spec]
+            if not isinstance(state_spec, list):
+                state_spec = [state_spec]
+            self.input_spec = input_spec + state_spec
 
-    def call(self, inputs, mask=None, initial_state=None, training=None):
+            # Compute the full inputs, including state
+            inputs = [inputs] + list(initial_state)
+
+            # Perform the call
+            output = super(Recurrent, self).__call__(inputs, **kwargs)
+
+            # Restore original input spec
+            self.input_spec = input_spec
+            return output
+        else:
+            kwargs['initial_state'] = initial_state
+            return super(Recurrent, self).__call__(inputs, **kwargs)
+
+    def call(self, inputs, mask=None, training=None, initial_state=None):
         # input shape: `(samples, time (padded with zeros), input_dim)`
         # note that the .build() method of subclasses MUST define
         # self.input_spec and self.state_spec with complete input shapes.
-        if initial_state is not None:
-            if not isinstance(initial_state, (list, tuple)):
-                initial_states = [initial_state]
-            else:
-                initial_states = list(initial_state)
         if isinstance(inputs, list):
-            initial_states = inputs[1:]
+            initial_state = inputs[1:]
             inputs = inputs[0]
+        elif initial_state is not None:
+            pass
         elif self.stateful:
-            initial_states = self.states
+            initial_state = self.states
         else:
-            initial_states = self.get_initial_states(inputs)
+            initial_state = self.get_initial_state(inputs)
 
-        if len(initial_states) != len(self.states):
+        if isinstance(mask, list):
+            mask = mask[0]
+
+        if len(initial_state) != len(self.states):
             raise ValueError('Layer has ' + str(len(self.states)) +
                              ' states but was passed ' +
-                             str(len(initial_states)) +
+                             str(len(initial_state)) +
                              ' initial states.')
         input_shape = K.int_shape(inputs)
         if self.unroll and input_shape[1] is None:
@@ -290,7 +333,7 @@ class Recurrent(Layer):
         preprocessed_input = self.preprocess_input(inputs, training=None)
         last_output, outputs, states = K.rnn(self.step,
                                              preprocessed_input,
-                                             initial_states,
+                                             initial_state,
                                              go_backwards=self.go_backwards,
                                              mask=mask,
                                              constants=constants,
@@ -308,17 +351,23 @@ class Recurrent(Layer):
             outputs._uses_learning_phase = True
 
         if self.return_sequences:
-            return outputs
+            output = outputs
         else:
-            return last_output
+            output = last_output
 
-    def reset_states(self, states_value=None):
+        if self.return_state:
+            if not isinstance(states, (list, tuple)):
+                states = [states]
+            else:
+                states = list(states)
+            return [output] + states
+        else:
+            return output
+
+    def reset_states(self, states=None):
         if not self.stateful:
             raise AttributeError('Layer must be stateful.')
-        if not self.input_spec:
-            raise RuntimeError('Layer has never been called '
-                               'and thus has no states.')
-        batch_size = self.input_spec.shape[0]
+        batch_size = self.input_spec[0].shape[0]
         if not batch_size:
             raise ValueError('If a RNN is stateful, it needs to know '
                              'its batch size. Specify the batch size '
@@ -330,34 +379,34 @@ class Recurrent(Layer):
                              '- If using the functional API, specify '
                              'the time dimension by passing a '
                              '`batch_shape` argument to your Input layer.')
-        if states_value is not None:
-            if not isinstance(states_value, (list, tuple)):
-                states_value = [states_value]
-            if len(states_value) != len(self.states):
-                raise ValueError('The layer has ' + str(len(self.states)) +
-                                 ' states, but the `states_value` '
-                                 'argument passed '
-                                 'only has ' + str(len(states_value)) +
-                                 ' entries')
+        # initialize state if None
         if self.states[0] is None:
             self.states = [K.zeros((batch_size, self.units))
                            for _ in self.states]
-            if not states_value:
-                return
-        for i, state in enumerate(self.states):
-            if states_value:
-                value = states_value[i]
+        elif states is None:
+            for state in self.states:
+                K.set_value(state, np.zeros((batch_size, self.units)))
+        else:
+            if not isinstance(states, (list, tuple)):
+                states = [states]
+            if len(states) != len(self.states):
+                raise ValueError('Layer ' + self.name + ' expects ' +
+                                 str(len(self.states)) + ' states, '
+                                 'but it received ' + str(len(states)) +
+                                 ' state values. Input received: ' +
+                                 str(states))
+            for index, (value, state) in enumerate(zip(states, self.states)):
                 if value.shape != (batch_size, self.units):
-                    raise ValueError(
-                        'Expected state #' + str(i) +
-                        ' to have shape ' + str((batch_size, self.units)) +
-                        ' but got array with shape ' + str(value.shape))
-            else:
-                value = np.zeros((batch_size, self.units))
-            K.set_value(state, value)
+                    raise ValueError('State ' + str(index) +
+                                     ' is incompatible with layer ' +
+                                     self.name + ': expected shape=' +
+                                     str((batch_size, self.units)) +
+                                     ', found shape=' + str(value.shape))
+                K.set_value(state, value)
 
     def get_config(self):
         config = {'return_sequences': self.return_sequences,
+                  'return_state': self.return_state,
                   'go_backwards': self.go_backwards,
                   'stateful': self.stateful,
                   'unroll': self.unroll,
@@ -373,7 +422,7 @@ class SimpleRNN(Recurrent):
         units: Positive integer, dimensionality of the output space.
         activation: Activation function to use
             (see [activations](../activations.md)).
-            If you don't specify anything, no activation is applied
+            If you pass None, no activation is applied
             (ie. "linear" activation: `a(x) = x`).
         use_bias: Boolean, whether the layer uses a bias vector.
         kernel_initializer: Initializer for the `kernel` weights matrix,
@@ -452,6 +501,7 @@ class SimpleRNN(Recurrent):
 
         self.dropout = min(1., max(0., dropout))
         self.recurrent_dropout = min(1., max(0., recurrent_dropout))
+        self.state_spec = InputSpec(shape=(None, self.units))
 
     def build(self, input_shape):
         if isinstance(input_shape, list):
@@ -459,26 +509,25 @@ class SimpleRNN(Recurrent):
 
         batch_size = input_shape[0] if self.stateful else None
         self.input_dim = input_shape[2]
-        self.input_spec = InputSpec(shape=(batch_size, None, self.input_dim))
-        self.state_spec = InputSpec(shape=(batch_size, self.units))
+        self.input_spec[0] = InputSpec(shape=(batch_size, None, self.input_dim))
 
         self.states = [None]
         if self.stateful:
             self.reset_states()
 
-        self.kernel = self.add_weight((self.input_dim, self.units),
+        self.kernel = self.add_weight(shape=(self.input_dim, self.units),
                                       name='kernel',
                                       initializer=self.kernel_initializer,
                                       regularizer=self.kernel_regularizer,
                                       constraint=self.kernel_constraint)
         self.recurrent_kernel = self.add_weight(
-            (self.units, self.units),
+            shape=(self.units, self.units),
             name='recurrent_kernel',
             initializer=self.recurrent_initializer,
             regularizer=self.recurrent_regularizer,
             constraint=self.recurrent_constraint)
         if self.use_bias:
-            self.bias = self.add_weight((self.units,),
+            self.bias = self.add_weight(shape=(self.units,),
                                         name='bias',
                                         initializer=self.bias_initializer,
                                         regularizer=self.bias_regularizer,
@@ -528,7 +577,7 @@ class SimpleRNN(Recurrent):
 
     def get_constants(self, inputs, training=None):
         constants = []
-        if self.implementation == 0 and 0 < self.dropout < 1:
+        if self.implementation != 0 and 0 < self.dropout < 1:
             input_shape = K.int_shape(inputs)
             input_dim = input_shape[-1]
             ones = K.ones_like(K.reshape(inputs[:, 0, 0], (-1, 1)))
@@ -585,7 +634,7 @@ class GRU(Recurrent):
         units: Positive integer, dimensionality of the output space.
         activation: Activation function to use
             (see [activations](../activations.md)).
-            If you don't specify anything, no activation is applied
+            If you pass None, no activation is applied
             (ie. "linear" activation: `a(x) = x`).
         recurrent_activation: Activation function to use
             for the recurrent step
@@ -671,6 +720,7 @@ class GRU(Recurrent):
 
         self.dropout = min(1., max(0., dropout))
         self.recurrent_dropout = min(1., max(0., recurrent_dropout))
+        self.state_spec = InputSpec(shape=(None, self.units))
 
     def build(self, input_shape):
         if isinstance(input_shape, list):
@@ -678,29 +728,28 @@ class GRU(Recurrent):
 
         batch_size = input_shape[0] if self.stateful else None
         self.input_dim = input_shape[2]
-        self.input_spec = InputSpec(shape=(batch_size, None, self.input_dim))
-        self.state_spec = InputSpec(shape=(batch_size, self.units))
+        self.input_spec[0] = InputSpec(shape=(batch_size, None, self.input_dim))
 
         self.states = [None]
         if self.stateful:
             self.reset_states()
 
-        self.kernel = self.add_weight((self.input_dim, self.units * 3),
+        self.kernel = self.add_weight(shape=(self.input_dim, self.units * 3),
                                       name='kernel',
                                       initializer=self.kernel_initializer,
                                       regularizer=self.kernel_regularizer,
                                       constraint=self.kernel_constraint)
         self.recurrent_kernel = self.add_weight(
-            (self.units, self.units * 3),
+            shape=(self.units, self.units * 3),
             name='recurrent_kernel',
             initializer=self.recurrent_initializer,
             regularizer=self.recurrent_regularizer,
             constraint=self.recurrent_constraint)
 
         if self.use_bias:
-            self.bias = self.add_weight((self.units * 3,),
+            self.bias = self.add_weight(shape=(self.units * 3,),
                                         name='bias',
-                                        initializer='zero',
+                                        initializer=self.bias_initializer,
                                         regularizer=self.bias_regularizer,
                                         constraint=self.bias_constraint)
         else:
@@ -746,7 +795,7 @@ class GRU(Recurrent):
 
     def get_constants(self, inputs, training=None):
         constants = []
-        if self.implementation == 0 and 0 < self.dropout < 1:
+        if self.implementation != 0 and 0 < self.dropout < 1:
             input_shape = K.int_shape(inputs)
             input_dim = input_shape[-1]
             ones = K.ones_like(K.reshape(inputs[:, 0, 0], (-1, 1)))
@@ -858,7 +907,7 @@ class LSTM(Recurrent):
         units: Positive integer, dimensionality of the output space.
         activation: Activation function to use
             (see [activations](../activations.md)).
-            If you don't specify anything, no activation is applied
+            If you pass None, no activation is applied
             (ie. "linear" activation: `a(x) = x`).
         recurrent_activation: Activation function to use
             for the recurrent step
@@ -904,7 +953,7 @@ class LSTM(Recurrent):
             the linear transformation of the recurrent state.
 
     # References
-        - [Long short-term memory](http://deeplearning.cs.cmu.edu/pdfs/Hochreiter97_lstm.pdf) (original 1997 paper)
+        - [Long short-term memory](http://www.bioinf.jku.at/publications/older/2604.pdf) (original 1997 paper)
         - [Learning to forget: Continual prediction with LSTM](http://www.mitpressjournals.org/doi/pdf/10.1162/089976600300015015)
         - [Supervised sequence labeling with recurrent neural networks](http://www.cs.toronto.edu/~graves/preprint.pdf)
         - [A Theoretically Grounded Application of Dropout in Recurrent Neural Networks](http://arxiv.org/abs/1512.05287)
@@ -950,6 +999,8 @@ class LSTM(Recurrent):
 
         self.dropout = min(1., max(0., dropout))
         self.recurrent_dropout = min(1., max(0., recurrent_dropout))
+        self.state_spec = [InputSpec(shape=(None, self.units)),
+                           InputSpec(shape=(None, self.units))]
 
     def build(self, input_shape):
         if isinstance(input_shape, list):
@@ -957,36 +1008,39 @@ class LSTM(Recurrent):
 
         batch_size = input_shape[0] if self.stateful else None
         self.input_dim = input_shape[2]
-        self.input_spec = InputSpec(shape=(batch_size, None, self.input_dim))
-        self.state_spec = [InputSpec(shape=(batch_size, self.units)),
-                           InputSpec(shape=(batch_size, self.units))]
+        self.input_spec[0] = InputSpec(shape=(batch_size, None, self.input_dim))
 
         self.states = [None, None]
         if self.stateful:
             self.reset_states()
 
-        self.kernel = self.add_weight((self.input_dim, self.units * 4),
+        self.kernel = self.add_weight(shape=(self.input_dim, self.units * 4),
                                       name='kernel',
                                       initializer=self.kernel_initializer,
                                       regularizer=self.kernel_regularizer,
                                       constraint=self.kernel_constraint)
         self.recurrent_kernel = self.add_weight(
-            (self.units, self.units * 4),
+            shape=(self.units, self.units * 4),
             name='recurrent_kernel',
             initializer=self.recurrent_initializer,
             regularizer=self.recurrent_regularizer,
             constraint=self.recurrent_constraint)
 
         if self.use_bias:
-            self.bias = self.add_weight((self.units * 4,),
+            if self.unit_forget_bias:
+                def bias_initializer(shape, *args, **kwargs):
+                    return K.concatenate([
+                        self.bias_initializer((self.units,), *args, **kwargs),
+                        initializers.Ones()((self.units,), *args, **kwargs),
+                        self.bias_initializer((self.units * 2,), *args, **kwargs),
+                    ])
+            else:
+                bias_initializer = self.bias_initializer
+            self.bias = self.add_weight(shape=(self.units * 4,),
                                         name='bias',
-                                        initializer=self.bias_initializer,
+                                        initializer=bias_initializer,
                                         regularizer=self.bias_regularizer,
                                         constraint=self.bias_constraint)
-            if self.unit_forget_bias:
-                bias_value = np.zeros((self.units * 4,))
-                bias_value[self.units: self.units * 2] = 1.
-                K.set_value(self.bias, bias_value)
         else:
             self.bias = None
 
@@ -1036,7 +1090,7 @@ class LSTM(Recurrent):
 
     def get_constants(self, inputs, training=None):
         constants = []
-        if self.implementation == 0 and 0 < self.dropout < 1:
+        if self.implementation != 0 and 0 < self.dropout < 1:
             input_shape = K.int_shape(inputs)
             input_dim = input_shape[-1]
             ones = K.ones_like(K.reshape(inputs[:, 0, 0], (-1, 1)))

keras/layers/wrappers.py

@@ -4,6 +4,7 @@ from __future__ import absolute_import
 import copy
 from ..engine import Layer
 from ..engine import InputSpec
+from ..utils.generic_utils import has_arg
 from .. import backend as K
 
 
@@ -23,18 +24,53 @@ class Wrapper(Layer):
         super(Wrapper, self).__init__(**kwargs)
 
     def build(self, input_shape=None):
-        # Assumes that self.layer is already set.
-        # Should be called at the end of .build() in the children classes.
-        self.trainable_weights = getattr(self.layer, 'trainable_weights', [])
-        self.non_trainable_weights = getattr(self.layer, 'non_trainable_weights', [])
-        self.updates = getattr(self.layer, 'updates', [])
-        self.losses = getattr(self.layer, 'losses', [])
-        self.constraints = getattr(self.layer, 'constraints', {})
         self.built = True
 
+    @property
+    def activity_regularizer(self):
+        if hasattr(self.layer, 'activity_regularizer'):
+            return self.layer.activity_regularizer
+        else:
+            return None
+
+    @property
+    def trainable_weights(self):
+        return self.layer.trainable_weights
+
+    @property
+    def non_trainable_weights(self):
+        return self.layer.non_trainable_weights
+
+    @property
+    def updates(self):
+        if hasattr(self.layer, 'updates'):
+            return self.layer.updates
+        return []
+
+    def get_updates_for(self, inputs=None):
+        if inputs is None:
+            updates = self.layer.get_updates_for(None)
+            return updates + super(Wrapper, self).get_updates_for(None)
+        return super(Wrapper, self).get_updates_for(inputs)
+
+    @property
+    def losses(self):
+        if hasattr(self.layer, 'losses'):
+            return self.layer.losses
+        return []
+
+    def get_losses_for(self, inputs=None):
+        if inputs is None:
+            losses = self.layer.get_losses_for(None)
+            return losses + super(Wrapper, self).get_losses_for(None)
+        return super(Wrapper, self).get_losses_for(inputs)
+
+    @property
+    def constraints(self):
+        return self.layer.constraints
+
     def get_weights(self):
-        weights = self.layer.get_weights()
-        return weights
+        return self.layer.get_weights()
 
     def set_weights(self, weights):
         self.layer.set_weights(weights)
@@ -48,12 +84,13 @@ class Wrapper(Layer):
     @classmethod
     def from_config(cls, config, custom_objects=None):
         from . import deserialize as deserialize_layer
-        layer = deserialize_layer(config.pop('layer'), custom_objects=custom_objects)
+        layer = deserialize_layer(config.pop('layer'),
+                                  custom_objects=custom_objects)
         return cls(layer, **config)
 
 
 class TimeDistributed(Wrapper):
-    """This wrapper allows to apply a layer to every temporal slice of an input.
+    """This wrapper applies a layer to every temporal slice of an input.
 
     The input should be at least 3D, and the dimension of index one
     will be considered to be the temporal dimension.
@@ -71,13 +108,18 @@ class TimeDistributed(Wrapper):
         model = Sequential()
         model.add(TimeDistributed(Dense(8), input_shape=(10, 16)))
         # now model.output_shape == (None, 10, 8)
+    ```
 
-        # subsequent layers: no need for input_shape
+    The output will then have shape `(32, 10, 8)`.
+
+    In subsequent layers, there is no need for the `input_shape`:
+
+    ```python
         model.add(TimeDistributed(Dense(32)))
         # now model.output_shape == (None, 10, 32)
     ```
 
-    The output will then have shape `(32, 10, 8)`.
+    The output will then have shape `(32, 10, 32)`.
 
     `TimeDistributed` can be used with arbitrary layers, not just `Dense`,
     for instance with a `Conv2D` layer:
@@ -111,12 +153,21 @@ class TimeDistributed(Wrapper):
         timesteps = input_shape[1]
         return (child_output_shape[0], timesteps) + child_output_shape[1:]
 
-    def call(self, inputs, mask=None):
+    def call(self, inputs, training=None, mask=None):
+        kwargs = {}
+        if has_arg(self.layer.call, 'training'):
+            kwargs['training'] = training
+        uses_learning_phase = False
+
         input_shape = K.int_shape(inputs)
         if input_shape[0]:
             # batch size matters, use rnn-based implementation
             def step(x, _):
-                output = self.layer.call(x)
+                global uses_learning_phase
+                output = self.layer.call(x, **kwargs)
+                if hasattr(output, '_uses_learning_phase'):
+                    uses_learning_phase = (output._uses_learning_phase or
+                                           uses_learning_phase)
                 return output, []
 
             _, outputs, _ = K.rnn(step, inputs,
@@ -133,7 +184,10 @@ class TimeDistributed(Wrapper):
                 input_length = K.shape(inputs)[1]
             # Shape: (num_samples * timesteps, ...)
             inputs = K.reshape(inputs, (-1,) + input_shape[2:])
-            y = self.layer.call(inputs)  # (num_samples * timesteps, ...)
+            # (num_samples * timesteps, ...)
+            y = self.layer.call(inputs, **kwargs)
+            if hasattr(y, '_uses_learning_phase'):
+                uses_learning_phase = y._uses_learning_phase
             # Shape: (num_samples, timesteps, ...)
             output_shape = self.compute_output_shape(input_shape)
             y = K.reshape(y, (-1, input_length) + output_shape[2:])
@@ -143,6 +197,9 @@ class TimeDistributed(Wrapper):
            self.layer.activity_regularizer is not None):
             regularization_loss = self.layer.activity_regularizer(y)
             self.add_loss(regularization_loss, inputs)
+
+        if uses_learning_phase:
+            y._uses_learning_phase = True
         return y
 
 
@@ -157,11 +214,15 @@ class Bidirectional(Wrapper):
             If None, the outputs will not be combined,
             they will be returned as a list.
 
+    # Raises
+        ValueError: In case of invalid `merge_mode` argument.
+
     # Examples
 
     ```python
         model = Sequential()
-        model.add(Bidirectional(LSTM(10, return_sequences=True), input_shape=(5, 10)))
+        model.add(Bidirectional(LSTM(10, return_sequences=True),
+                                input_shape=(5, 10)))
         model.add(Bidirectional(LSTM(10)))
         model.add(Dense(5))
         model.add(Activation('softmax'))
@@ -208,29 +269,46 @@ class Bidirectional(Wrapper):
         elif self.merge_mode is None:
             return [self.forward_layer.compute_output_shape(input_shape)] * 2
 
-    def call(self, inputs, mask=None):
-        y = self.forward_layer.call(inputs, mask)
-        y_rev = self.backward_layer.call(inputs, mask)
+    def call(self, inputs, training=None, mask=None):
+        kwargs = {}
+        if has_arg(self.layer.call, 'training'):
+            kwargs['training'] = training
+        if has_arg(self.layer.call, 'mask'):
+            kwargs['mask'] = mask
+
+        y = self.forward_layer.call(inputs, **kwargs)
+        y_rev = self.backward_layer.call(inputs, **kwargs)
         if self.return_sequences:
             y_rev = K.reverse(y_rev, 1)
         if self.merge_mode == 'concat':
-            return K.concatenate([y, y_rev])
+            output = K.concatenate([y, y_rev])
         elif self.merge_mode == 'sum':
-            return y + y_rev
+            output = y + y_rev
         elif self.merge_mode == 'ave':
-            return (y + y_rev) / 2
+            output = (y + y_rev) / 2
         elif self.merge_mode == 'mul':
-            return y * y_rev
+            output = y * y_rev
         elif self.merge_mode is None:
-            return [y, y_rev]
+            output = [y, y_rev]
+
+        # Properly set learning phase
+        if 0 < self.layer.dropout + self.layer.recurrent_dropout:
+            if self.merge_mode is None:
+                for out in output:
+                    out._uses_learning_phase = True
+            else:
+                output._uses_learning_phase = True
+        return output
 
     def reset_states(self):
         self.forward_layer.reset_states()
         self.backward_layer.reset_states()
 
     def build(self, input_shape):
-        self.forward_layer.build(input_shape)
-        self.backward_layer.build(input_shape)
+        with K.name_scope(self.forward_layer.name):
+            self.forward_layer.build(input_shape)
+        with K.name_scope(self.backward_layer.name):
+            self.backward_layer.build(input_shape)
         self.built = True
 
     def compute_mask(self, inputs, mask):

keras/legacy/interfaces.py

@@ -3,7 +3,6 @@
 import six
 import warnings
 import functools
-import inspect
 import numpy as np
 
 
@@ -86,7 +85,7 @@ def generate_legacy_interface(allowed_positional_args=None,
                 warnings.warn('Update your `' + object_name +
                               '` call to the Keras 2 API: ' + signature, stacklevel=2)
             return func(*args, **kwargs)
-        wrapper._legacy_support_signature = inspect.getargspec(func)
+        wrapper._original_function = func
         return wrapper
     return legacy_support
 
@@ -161,7 +160,7 @@ def recurrent_args_preprocessor(args, kwargs):
             kwargs.pop('forget_bias_init')
             warnings.warn('The `forget_bias_init` argument '
                           'has been ignored. Use `unit_forget_bias=True` '
-                          'instead to intialize with ones.', stacklevel=3)
+                          'instead to initialize with ones.', stacklevel=3)
     if 'input_dim' in kwargs:
         input_length = kwargs.pop('input_length', None)
         input_dim = kwargs.pop('input_dim')
@@ -461,7 +460,7 @@ def convlstm2d_args_preprocessor(args, kwargs):
         else:
             warnings.warn('The `forget_bias_init` argument '
                           'has been ignored. Use `unit_forget_bias=True` '
-                          'instead to intialize with ones.', stacklevel=3)
+                          'instead to initialize with ones.', stacklevel=3)
     args, kwargs, _converted = conv2d_args_preprocessor(args, kwargs)
     return args, kwargs, converted + _converted
 
@@ -577,14 +576,21 @@ def generator_methods_args_preprocessor(args, kwargs):
             if hasattr(generator, 'batch_size'):
                 kwargs['steps_per_epoch'] = samples_per_epoch // generator.batch_size
             else:
-                warnings.warn('The semantics of the Keras 2 argument '
-                              ' `steps_per_epoch` is not the same as the '
-                              'Keras 1 argument `samples_per_epoch`. '
-                              '`steps_per_epoch` is the number of batches '
-                              'to draw from the generator at each epoch. '
-                              'Update your method calls accordingly.', stacklevel=3)
                 kwargs['steps_per_epoch'] = samples_per_epoch
             converted.append(('samples_per_epoch', 'steps_per_epoch'))
+
+    keras1_args = {'samples_per_epoch', 'val_samples', 'nb_epoch', 'nb_val_samples', 'nb_worker'}
+    if keras1_args.intersection(kwargs.keys()):
+        warnings.warn('The semantics of the Keras 2 argument '
+                      '`steps_per_epoch` is not the same as the '
+                      'Keras 1 argument `samples_per_epoch`. '
+                      '`steps_per_epoch` is the number of batches '
+                      'to draw from the generator at each epoch. '
+                      'Basically steps_per_epoch = samples_per_epoch/batch_size. '
+                      'Similarly `nb_val_samples`->`validation_steps` and '
+                      '`val_samples`->`steps` arguments have changed. '
+                      'Update your method calls accordingly.', stacklevel=3)
+
     return args, kwargs, converted
 
 
@@ -594,7 +600,9 @@ legacy_generator_methods_support = generate_legacy_method_interface(
                  ('val_samples', 'steps'),
                  ('nb_epoch', 'epochs'),
                  ('nb_val_samples', 'validation_steps'),
-                 ('nb_worker', 'workers')],
+                 ('nb_worker', 'workers'),
+                 ('pickle_safe', 'use_multiprocessing'),
+                 ('max_q_size', 'max_queue_size')],
     preprocessor=generator_methods_args_preprocessor)
 
 
@@ -602,3 +610,24 @@ legacy_model_constructor_support = generate_legacy_interface(
     allowed_positional_args=None,
     conversions=[('input', 'inputs'),
                  ('output', 'outputs')])
+
+legacy_input_support = generate_legacy_interface(
+    allowed_positional_args=None,
+    conversions=[('input_dtype', 'dtype')])
+
+
+def add_weight_args_preprocessing(args, kwargs):
+    if len(args) > 1:
+        if isinstance(args[1], (tuple, list)):
+            kwargs['shape'] = args[1]
+            args = (args[0],) + args[2:]
+            if len(args) > 1:
+                if isinstance(args[1], six.string_types):
+                    kwargs['name'] = args[1]
+                    args = (args[0],) + args[2:]
+    return args, kwargs, []
+
+
+legacy_add_weight_support = generate_legacy_interface(
+    allowed_positional_args=['name', 'shape'],
+    preprocessor=add_weight_args_preprocessing)

keras/legacy/layers.py

@@ -1,10 +1,9 @@
-import inspect
 import types as python_types
 import warnings
 
 from ..engine.topology import Layer, InputSpec
 from .. import backend as K
-from ..utils.generic_utils import func_dump, func_load
+from ..utils.generic_utils import func_dump, func_load, has_arg
 from .. import regularizers
 from .. import constraints
 from .. import activations
@@ -76,6 +75,10 @@ class Merge(Layer):
         self._output_mask = output_mask
         self.arguments = arguments if arguments else {}
         self._initial_weights = None
+        self._updates = []
+        self._losses = []
+        self._per_input_updates = {}
+        self._per_input_losses = {}
 
         # Layer parameters.
         self.inbound_nodes = []
@@ -193,8 +196,7 @@ class Merge(Layer):
         # Case: "mode" is a lambda or function.
         if callable(self.mode):
             arguments = self.arguments
-            arg_spec = inspect.getargspec(self.mode)
-            if 'mask' in arg_spec.args:
+            if has_arg(self.mode, 'mask'):
                 arguments['mask'] = mask
             return self.mode(inputs, **arguments)
 

keras/losses.py

@@ -33,16 +33,28 @@ def hinge(y_true, y_pred):
     return K.mean(K.maximum(1. - y_true * y_pred, 0.), axis=-1)
 
 
+def categorical_hinge(y_true, y_pred):
+    pos = K.sum(y_true * y_pred, axis=-1)
+    neg = K.max((1. - y_true) * y_pred, axis=-1)
+    return K.maximum(0., neg - pos + 1.)
+
+
+def logcosh(y_true, y_pred):
+    def cosh(x):
+        return (K.exp(x) + K.exp(-x)) / 2
+    return K.mean(K.log(cosh(y_pred - y_true)), axis=-1)
+
+
 def categorical_crossentropy(y_true, y_pred):
-    return K.categorical_crossentropy(y_pred, y_true)
+    return K.categorical_crossentropy(y_true, y_pred)
 
 
 def sparse_categorical_crossentropy(y_true, y_pred):
-    return K.sparse_categorical_crossentropy(y_pred, y_true)
+    return K.sparse_categorical_crossentropy(y_true, y_pred)
 
 
 def binary_crossentropy(y_true, y_pred):
-    return K.mean(K.binary_crossentropy(y_pred, y_true), axis=-1)
+    return K.mean(K.binary_crossentropy(y_true, y_pred), axis=-1)
 
 
 def kullback_leibler_divergence(y_true, y_pred):

keras/metrics.py

@@ -6,6 +6,7 @@ from .losses import mean_absolute_error
 from .losses import mean_absolute_percentage_error
 from .losses import mean_squared_logarithmic_error
 from .losses import hinge
+from .losses import logcosh
 from .losses import squared_hinge
 from .losses import categorical_crossentropy
 from .losses import sparse_categorical_crossentropy
@@ -35,6 +36,11 @@ def sparse_categorical_accuracy(y_true, y_pred):
 def top_k_categorical_accuracy(y_true, y_pred, k=5):
     return K.mean(K.in_top_k(y_pred, K.argmax(y_true, axis=-1), k), axis=-1)
 
+
+def sparse_top_k_categorical_accuracy(y_true, y_pred, k=5):
+    return K.mean(K.in_top_k(y_pred, K.cast(K.max(y_true, axis=-1), 'int32'), k), axis=-1)
+
+
 # Aliases
 
 mse = MSE = mean_squared_error

keras/models.py

@@ -27,7 +27,7 @@ except ImportError:
     h5py = None
 
 
-def save_model(model, filepath, overwrite=True):
+def save_model(model, filepath, overwrite=True, include_optimizer=True):
     """Save a model to a HDF5 file.
 
     The saved model contains:
@@ -45,6 +45,7 @@ def save_model(model, filepath, overwrite=True):
         overwrite: Whether we should overwrite any existing
             model at the target location, or instead
             ask the user with a manual prompt.
+        include_optimizer: If True, save optimizer's state together.
 
     # Raises
         ImportError: if h5py is not available.
@@ -73,7 +74,11 @@ def save_model(model, filepath, overwrite=True):
 
         # if obj is any numpy type
         if type(obj).__module__ == np.__name__:
-            return obj.item()
+            if isinstance(obj, np.ndarray):
+                return {'type': type(obj),
+                        'value': obj.tolist()}
+            else:
+                return obj.item()
 
         # misc functions (e.g. loss function)
         if callable(obj):
@@ -108,7 +113,7 @@ def save_model(model, filepath, overwrite=True):
         model_layers = model.layers
     topology.save_weights_to_hdf5_group(model_weights_group, model_layers)
 
-    if hasattr(model, 'optimizer'):
+    if include_optimizer and hasattr(model, 'optimizer'):
         if isinstance(model.optimizer, optimizers.TFOptimizer):
             warnings.warn(
                 'TensorFlow optimizers do not '
@@ -139,8 +144,8 @@ def save_model(model, filepath, overwrite=True):
                 weight_values = K.batch_get_value(symbolic_weights)
                 weight_names = []
                 for i, (w, val) in enumerate(zip(symbolic_weights, weight_values)):
-                    # Default values of symbolic_weights is /variable for theano
-                    if K.backend() == 'theano':
+                    # Default values of symbolic_weights is /variable for theano and cntk
+                    if K.backend() == 'theano' or K.backend() == 'cntk':
                         if hasattr(w, 'name') and w.name != "/variable":
                             name = str(w.name)
                         else:
@@ -166,7 +171,7 @@ def save_model(model, filepath, overwrite=True):
     f.close()
 
 
-def load_model(filepath, custom_objects=None):
+def load_model(filepath, custom_objects=None, compile=True):
     """Loads a model saved via `save_model`.
 
     # Arguments
@@ -174,19 +179,23 @@ def load_model(filepath, custom_objects=None):
         custom_objects: Optional dictionary mapping names
             (strings) to custom classes or functions to be
             considered during deserialization.
+        compile: Boolean, whether to compile the model
+            after loading.
 
     # Returns
         A Keras model instance. If an optimizer was found
         as part of the saved model, the model is already
         compiled. Otherwise, the model is uncompiled and
-        a warning will be displayed.
+        a warning will be displayed. When `compile` is set
+        to False, the compilation is omitted without any
+        warning.
 
     # Raises
         ImportError: if h5py is not available.
         ValueError: In case of an invalid savefile.
     """
     if h5py is None:
-        raise ImportError('`save_model` requires h5py.')
+        raise ImportError('`load_model` requires h5py.')
 
     if not custom_objects:
         custom_objects = {}
@@ -198,79 +207,81 @@ def load_model(filepath, custom_objects=None):
             obj: object, dict, or list.
 
         # Returns
-            The same structure, where occurences
+            The same structure, where occurrences
                 of a custom object name have been replaced
                 with the custom object.
         """
         if isinstance(obj, list):
             deserialized = []
             for value in obj:
-                if value in custom_objects:
-                    deserialized.append(custom_objects[value])
-                else:
-                    deserialized.append(value)
+                deserialized.append(convert_custom_objects(value))
             return deserialized
         if isinstance(obj, dict):
             deserialized = {}
             for key, value in obj.items():
-                if value in custom_objects:
-                    deserialized[key] = custom_objects[value]
-                else:
-                    deserialized[key] = value
+                deserialized[key] = convert_custom_objects(value)
             return deserialized
         if obj in custom_objects:
             return custom_objects[obj]
         return obj
+    with h5py.File(filepath, mode='r') as f:
+        # instantiate model
+        model_config = f.attrs.get('model_config')
+        if model_config is None:
+            raise ValueError('No model found in config file.')
+        model_config = json.loads(model_config.decode('utf-8'))
+        model = model_from_config(model_config, custom_objects=custom_objects)
 
-    f = h5py.File(filepath, mode='r')
+        # set weights
+        topology.load_weights_from_hdf5_group(f['model_weights'], model.layers)
 
-    # instantiate model
-    model_config = f.attrs.get('model_config')
-    if model_config is None:
-        raise ValueError('No model found in config file.')
-    model_config = json.loads(model_config.decode('utf-8'))
-    model = model_from_config(model_config, custom_objects=custom_objects)
+        # Early return if compilation is not required.
+        if not compile:
+            return model
 
-    # set weights
-    topology.load_weights_from_hdf5_group(f['model_weights'], model.layers)
+        # instantiate optimizer
+        training_config = f.attrs.get('training_config')
+        if training_config is None:
+            warnings.warn('No training configuration found in save file: '
+                          'the model was *not* compiled. Compile it manually.')
+            return model
+        training_config = json.loads(training_config.decode('utf-8'))
+        optimizer_config = training_config['optimizer_config']
+        optimizer = optimizers.deserialize(optimizer_config,
+                                           custom_objects=custom_objects)
 
-    # instantiate optimizer
-    training_config = f.attrs.get('training_config')
-    if training_config is None:
-        warnings.warn('No training configuration found in save file: '
-                      'the model was *not* compiled. Compile it manually.')
-        f.close()
-        return model
-    training_config = json.loads(training_config.decode('utf-8'))
-    optimizer_config = training_config['optimizer_config']
-    optimizer = optimizers.deserialize(optimizer_config,
-                                       custom_objects=custom_objects)
+        # Recover loss functions and metrics.
+        loss = convert_custom_objects(training_config['loss'])
+        metrics = convert_custom_objects(training_config['metrics'])
+        sample_weight_mode = training_config['sample_weight_mode']
+        loss_weights = training_config['loss_weights']
 
-    # Recover loss functions and metrics.
-    loss = convert_custom_objects(training_config['loss'])
-    metrics = convert_custom_objects(training_config['metrics'])
-    sample_weight_mode = training_config['sample_weight_mode']
-    loss_weights = training_config['loss_weights']
+        # Compile model.
+        model.compile(optimizer=optimizer,
+                      loss=loss,
+                      metrics=metrics,
+                      loss_weights=loss_weights,
+                      sample_weight_mode=sample_weight_mode)
 
-    # Compile model.
-    model.compile(optimizer=optimizer,
-                  loss=loss,
-                  metrics=metrics,
-                  loss_weights=loss_weights,
-                  sample_weight_mode=sample_weight_mode)
-
-    # Set optimizer weights.
-    if 'optimizer_weights' in f:
-        # Build train function (to get weight updates).
-        if isinstance(model, Sequential):
-            model.model._make_train_function()
-        else:
-            model._make_train_function()
-        optimizer_weights_group = f['optimizer_weights']
-        optimizer_weight_names = [n.decode('utf8') for n in optimizer_weights_group.attrs['weight_names']]
-        optimizer_weight_values = [optimizer_weights_group[n] for n in optimizer_weight_names]
-        model.optimizer.set_weights(optimizer_weight_values)
-    f.close()
+        # Set optimizer weights.
+        if 'optimizer_weights' in f:
+            # Build train function (to get weight updates).
+            if isinstance(model, Sequential):
+                model.model._make_train_function()
+            else:
+                model._make_train_function()
+            optimizer_weights_group = f['optimizer_weights']
+            optimizer_weight_names = [n.decode('utf8') for n in
+                                      optimizer_weights_group.attrs['weight_names']]
+            optimizer_weight_values = [optimizer_weights_group[n] for n in
+                                       optimizer_weight_names]
+            try:
+                model.optimizer.set_weights(optimizer_weight_values)
+            except ValueError:
+                warnings.warn('Error in loading the saved optimizer '
+                              'state. As a result, your model is '
+                              'starting with a freshly initialized '
+                              'optimizer.')
     return model
 
 
@@ -285,9 +296,12 @@ def model_from_config(config, custom_objects=None):
 
     # Returns
         A Keras model instance (uncompiled).
+
+    # Raises
+        TypeError: if `config` is not a dictionary.
     """
     if isinstance(config, list):
-        raise TypeError('`model_fom_config` expects a dictionary, not a list. '
+        raise TypeError('`model_from_config` expects a dictionary, not a list. '
                         'Maybe you meant to use '
                         '`Sequential.from_config(config)`?')
     return layer_module.deserialize(config, custom_objects=custom_objects)
@@ -736,7 +750,7 @@ class Sequential(Model):
             optimizer: str (name of optimizer) or optimizer object.
                 See [optimizers](/optimizers).
             loss: str (name of objective function) or objective function.
-                See [objectives](/objectives).
+                See [losses](/losses).
             metrics: list of metrics to be evaluated by the model
                 during training and testing.
                 Typically you will use `metrics=['accuracy']`.
@@ -745,7 +759,8 @@ class Sequential(Model):
                 sample weighting (2D weights), set this to "temporal".
                 "None" defaults to sample-wise weights (1D).
             **kwargs: for Theano backend, these are passed into K.function.
-                Ignored for Tensorflow backend.
+                When using the Tensorflow backend, these are passed into
+                `tf.Session.run`.
 
         # Example
             ```python
@@ -766,11 +781,14 @@ class Sequential(Model):
                            **kwargs)
         self.optimizer = self.model.optimizer
         self.loss = self.model.loss
+        self.total_loss = self.model.total_loss
         self.loss_weights = self.model.loss_weights
         self.metrics = self.model.metrics
         self.metrics_tensors = self.model.metrics_tensors
         self.metrics_names = self.model.metrics_names
         self.sample_weight_mode = self.model.sample_weight_mode
+        self.sample_weights = self.model.sample_weights
+        self.targets = self.model.targets
 
     def fit(self, x, y, batch_size=32, epochs=10, verbose=1, callbacks=None,
             validation_split=0., validation_data=None, shuffle=True,
@@ -1008,9 +1026,9 @@ class Sequential(Model):
                       validation_data=None,
                       validation_steps=None,
                       class_weight=None,
-                      max_q_size=10,
+                      max_queue_size=10,
                       workers=1,
-                      pickle_safe=False,
+                      use_multiprocessing=False,
                       initial_epoch=0):
         """Fits the model on data generated batch-by-batch by a Python generator.
 
@@ -1025,8 +1043,8 @@ class Sequential(Model):
                 - a tuple (inputs, targets, sample_weights).
                 All arrays should contain the same number of samples.
                 The generator is expected to loop over its data
-                indefinitely. An epoch finishes when `samples_per_epoch`
-                samples have been seen by the model.
+                indefinitely. An epoch finishes when `steps_per_epoch`
+                batches have been seen by the model.
             steps_per_epoch: Total number of steps (batches of samples)
                 to yield from `generator` before declaring one epoch
                 finished and starting the next epoch. It should typically
@@ -1041,13 +1059,15 @@ class Sequential(Model):
                 - A tuple (inputs, targets, sample_weights).
             validation_steps: Only relevant if `validation_data`
                 is a generator.
-                Number of samples to use from validation generator
-                at the end of every epoch.
+                Number of steps to yield from validation generator
+                at the end of every epoch. It should typically
+                be equal to the number of unique samples of your
+                validation dataset divided by the batch size.
             class_weight: Dictionary mapping class indices to a weight
                 for the class.
-            max_q_size: Maximum size for the generator queue
+            max_queue_size: Maximum size for the generator queue
             workers: Maximum number of processes to spin up
-            pickle_safe: Ff True, use process based threading.
+            use_multiprocessing: if True, use process based threading.
                 Note that because
                 this implementation relies on multiprocessing,
                 you should not pass
@@ -1074,10 +1094,10 @@ class Sequential(Model):
                         # and labels, from each line in the file
                         x, y = process_line(line)
                         yield (x, y)
-                    f.close()
+                        f.close()
 
             model.fit_generator(generate_arrays_from_file('/my_file.txt'),
-                                samples_per_epoch=10000, epochs=10)
+                                steps_per_epoch=1000, epochs=10)
         ```
         """
         if self.model is None:
@@ -1091,15 +1111,15 @@ class Sequential(Model):
                                         validation_data=validation_data,
                                         validation_steps=validation_steps,
                                         class_weight=class_weight,
-                                        max_q_size=max_q_size,
+                                        max_queue_size=max_queue_size,
                                         workers=workers,
-                                        pickle_safe=pickle_safe,
+                                        use_multiprocessing=use_multiprocessing,
                                         initial_epoch=initial_epoch)
 
     @interfaces.legacy_generator_methods_support
     def evaluate_generator(self, generator, steps,
-                           max_q_size=10, workers=1,
-                           pickle_safe=False):
+                           max_queue_size=10, workers=1,
+                           use_multiprocessing=False):
         """Evaluates the model on a data generator.
 
         The generator should return the same kind of data
@@ -1110,9 +1130,9 @@ class Sequential(Model):
                 or (inputs, targets, sample_weights)
             steps: Total number of steps (batches of samples)
                 to yield from `generator` before stopping.
-            max_q_size: maximum size for the generator queue
+            max_queue_size: maximum size for the generator queue
             workers: maximum number of processes to spin up
-            pickle_safe: if True, use process based threading.
+            use_multiprocessing: if True, use process based threading.
                 Note that because this implementation
                 relies on multiprocessing, you should not pass
                 non picklable arguments to the generator
@@ -1132,14 +1152,14 @@ class Sequential(Model):
                                'before being used.')
         return self.model.evaluate_generator(generator,
                                              steps,
-                                             max_q_size=max_q_size,
+                                             max_queue_size=max_queue_size,
                                              workers=workers,
-                                             pickle_safe=pickle_safe)
+                                             use_multiprocessing=use_multiprocessing)
 
     @interfaces.legacy_generator_methods_support
     def predict_generator(self, generator, steps,
-                          max_q_size=10, workers=1,
-                          pickle_safe=False, verbose=0):
+                          max_queue_size=10, workers=1,
+                          use_multiprocessing=False, verbose=0):
         """Generates predictions for the input samples from a data generator.
 
         The generator should return the same kind of data as accepted by
@@ -1149,9 +1169,9 @@ class Sequential(Model):
             generator: generator yielding batches of input samples.
             steps: Total number of steps (batches of samples)
                 to yield from `generator` before stopping.
-            max_q_size: maximum size for the generator queue
+            max_queue_size: maximum size for the generator queue
             workers: maximum number of processes to spin up
-            pickle_safe: if True, use process based threading.
+            use_multiprocessing: if True, use process based threading.
                 Note that because this implementation
                 relies on multiprocessing, you should not pass
                 non picklable arguments to the generator
@@ -1164,9 +1184,9 @@ class Sequential(Model):
         if self.model is None:
             self.build()
         return self.model.predict_generator(generator, steps,
-                                            max_q_size=max_q_size,
+                                            max_queue_size=max_queue_size,
                                             workers=workers,
-                                            pickle_safe=pickle_safe,
+                                            use_multiprocessing=use_multiprocessing,
                                             verbose=verbose)
 
     def get_config(self):
@@ -1180,13 +1200,13 @@ class Sequential(Model):
         return copy.deepcopy(config)
 
     @classmethod
-    def from_config(cls, config):
+    def from_config(cls, config, custom_objects=None):
         if 'class_name' not in config[0] or config[0]['class_name'] == 'Merge':
             return cls.legacy_from_config(config)
 
         model = cls()
         for conf in config:
-            layer = layer_module.deserialize(conf)
+            layer = layer_module.deserialize(conf, custom_objects=custom_objects)
             model.add(layer)
         return model
 
@@ -1217,6 +1237,15 @@ class Sequential(Model):
 
     @classmethod
     def legacy_from_config(cls, config, layer_cache=None):
+        """Load a model from a legacy configuration.
+
+        # Arguments
+            config: dictionary with configuration.
+            layer_cache: cache to draw pre-existing layer.
+
+        # Returns
+            The loaded Model.
+        """
         if not layer_cache:
             layer_cache = {}
 

keras/optimizers.py

@@ -1,5 +1,6 @@
 from __future__ import absolute_import
 import six
+import copy
 from six.moves import zip
 
 from . import backend as K
@@ -11,8 +12,31 @@ if K.backend() == 'tensorflow':
 
 
 def clip_norm(g, c, n):
-    if c > 0:
-        g = K.switch(n >= c, g * c / n, g)
+    if c <= 0:  # if clipnorm == 0 no need to add ops to the graph
+        return g
+
+    # tf require using a special op to multiply IndexedSliced by scalar
+    if K.backend() == 'tensorflow':
+        condition = n >= c
+        then_expression = tf.scalar_mul(c / n, g)
+        else_expression = g
+
+        # saving the shape to avoid converting sparse tensor to dense
+        if isinstance(then_expression, tf.Tensor):
+            g_shape = copy.copy(then_expression.get_shape())
+        elif isinstance(then_expression, tf.IndexedSlices):
+            g_shape = copy.copy(then_expression.dense_shape)
+        if condition.dtype != tf.bool:
+            condition = tf.cast(condition, 'bool')
+        g = tf.cond(condition,
+                    lambda: then_expression,
+                    lambda: else_expression)
+        if isinstance(then_expression, tf.Tensor):
+            g.set_shape(g_shape)
+        elif isinstance(then_expression, tf.IndexedSlices):
+            g._dense_shape = g_shape
+    else:
+        g = K.switch(K.greater_equal(n, c), g * c / n, g)
     return g
 
 
@@ -195,8 +219,7 @@ class RMSprop(Optimizer):
 
     def get_updates(self, params, constraints, loss):
         grads = self.get_gradients(loss, params)
-        shapes = [K.get_variable_shape(p) for p in params]
-        accumulators = [K.zeros(shape) for shape in shapes]
+        accumulators = [K.zeros(K.get_variable_shape(p), dtype=K.dtype(p)) for p in params]
         self.weights = accumulators
         self.updates = []
 
@@ -389,9 +412,8 @@ class Adam(Optimizer):
         lr_t = lr * (K.sqrt(1. - K.pow(self.beta_2, t)) /
                      (1. - K.pow(self.beta_1, t)))
 
-        shapes = [K.get_variable_shape(p) for p in params]
-        ms = [K.zeros(shape) for shape in shapes]
-        vs = [K.zeros(shape) for shape in shapes]
+        ms = [K.zeros(K.get_variable_shape(p), dtype=K.dtype(p)) for p in params]
+        vs = [K.zeros(K.get_variable_shape(p), dtype=K.dtype(p)) for p in params]
         self.weights = [self.iterations] + ms + vs
 
         for p, g, m, v in zip(params, grads, ms, vs):

keras/preprocessing/image.py

@@ -13,6 +13,8 @@ from six.moves import range
 import os
 import threading
 import warnings
+import multiprocessing.pool
+from functools import partial
 
 from .. import backend as K
 
@@ -137,7 +139,7 @@ def random_zoom(x, zoom_range, row_axis=1, col_axis=2, channel_axis=0,
         ValueError: if `zoom_range` isn't a tuple.
     """
     if len(zoom_range) != 2:
-        raise ValueError('zoom_range should be a tuple or list of two floats. '
+        raise ValueError('`zoom_range` should be a tuple or list of two floats. '
                          'Received arg: ', zoom_range)
 
     if zoom_range[0] == 1 and zoom_range[1] == 1:
@@ -325,9 +327,9 @@ def load_img(path, grayscale=False, target_size=None):
         if img.mode != 'RGB':
             img = img.convert('RGB')
     if target_size:
-        wh_tuple = (target_size[1], target_size[0])
-        if img.size != wh_tuple:
-            img = img.resize(wh_tuple)
+        hw_tuple = (target_size[1], target_size[0])
+        if img.size != hw_tuple:
+            img = img.resize(hw_tuple)
     return img
 
 
@@ -346,6 +348,7 @@ class ImageDataGenerator(object):
         featurewise_std_normalization: divide inputs by std of the dataset.
         samplewise_std_normalization: divide each input by its std.
         zca_whitening: apply ZCA whitening.
+        zca_epsilon: epsilon for ZCA whitening. Default is 1e-6.
         rotation_range: degrees (0 to 180).
         width_shift_range: fraction of total width.
         height_shift_range: fraction of total height.
@@ -382,6 +385,7 @@ class ImageDataGenerator(object):
                  featurewise_std_normalization=False,
                  samplewise_std_normalization=False,
                  zca_whitening=False,
+                 zca_epsilon=1e-6,
                  rotation_range=0.,
                  width_shift_range=0.,
                  height_shift_range=0.,
@@ -402,6 +406,7 @@ class ImageDataGenerator(object):
         self.featurewise_std_normalization = featurewise_std_normalization
         self.samplewise_std_normalization = samplewise_std_normalization
         self.zca_whitening = zca_whitening
+        self.zca_epsilon = zca_epsilon
         self.rotation_range = rotation_range
         self.width_shift_range = width_shift_range
         self.height_shift_range = height_shift_range
@@ -416,8 +421,8 @@ class ImageDataGenerator(object):
         self.preprocessing_function = preprocessing_function
 
         if data_format not in {'channels_last', 'channels_first'}:
-            raise ValueError('data_format should be "channels_last" (channel after row and '
-                             'column) or "channels_first" (channel before row and column). '
+            raise ValueError('`data_format` should be `"channels_last"` (channel after row and '
+                             'column) or `"channels_first"` (channel before row and column). '
                              'Received arg: ', data_format)
         self.data_format = data_format
         if data_format == 'channels_first':
@@ -438,12 +443,12 @@ class ImageDataGenerator(object):
         elif len(zoom_range) == 2:
             self.zoom_range = [zoom_range[0], zoom_range[1]]
         else:
-            raise ValueError('zoom_range should be a float or '
+            raise ValueError('`zoom_range` should be a float or '
                              'a tuple or list of two floats. '
                              'Received arg: ', zoom_range)
 
     def flow(self, x, y=None, batch_size=32, shuffle=True, seed=None,
-             save_to_dir=None, save_prefix='', save_format='jpeg'):
+             save_to_dir=None, save_prefix='', save_format='png'):
         return NumpyArrayIterator(
             x, y, self,
             batch_size=batch_size,
@@ -460,7 +465,7 @@ class ImageDataGenerator(object):
                             batch_size=32, shuffle=True, seed=None,
                             save_to_dir=None,
                             save_prefix='',
-                            save_format='jpeg',
+                            save_format='png',
                             follow_links=False):
         return DirectoryIterator(
             directory, self,
@@ -633,8 +638,8 @@ class ImageDataGenerator(object):
         if x.ndim != 4:
             raise ValueError('Input to `.fit()` should have rank 4. '
                              'Got array with shape: ' + str(x.shape))
-        if x.shape[self.channel_axis] not in {1, 3, 4}:
-            raise ValueError(
+        if x.shape[self.channel_axis] not in {3, 4}:
+            warnings.warn(
                 'Expected input to be images (as Numpy array) '
                 'following the data format convention "' + self.data_format + '" '
                 '(channels on axis ' + str(self.channel_axis) + '), i.e. expected '
@@ -671,7 +676,7 @@ class ImageDataGenerator(object):
             flat_x = np.reshape(x, (x.shape[0], x.shape[1] * x.shape[2] * x.shape[3]))
             sigma = np.dot(flat_x.T, flat_x) / flat_x.shape[0]
             u, s, _ = linalg.svd(sigma)
-            self.principal_components = np.dot(np.dot(u, np.diag(1. / np.sqrt(s + 10e-7))), u.T)
+            self.principal_components = np.dot(np.dot(u, np.diag(1. / np.sqrt(s + self.zca_epsilon))), u.T)
 
 
 class Iterator(object):
@@ -752,7 +757,7 @@ class NumpyArrayIterator(Iterator):
     def __init__(self, x, y, image_data_generator,
                  batch_size=32, shuffle=False, seed=None,
                  data_format=None,
-                 save_to_dir=None, save_prefix='', save_format='jpeg'):
+                 save_to_dir=None, save_prefix='', save_format='png'):
         if y is not None and len(x) != len(y):
             raise ValueError('X (images tensor) and y (labels) '
                              'should have the same length. '
@@ -818,6 +823,73 @@ class NumpyArrayIterator(Iterator):
         return batch_x, batch_y
 
 
+def _count_valid_files_in_directory(directory, white_list_formats, follow_links):
+    """Count files with extension in `white_list_formats` contained in a directory.
+
+    # Arguments
+        directory: absolute path to the directory containing files to be counted
+        white_list_formats: set of strings containing allowed extensions for
+            the files to be counted.
+
+    # Returns
+        the count of files with extension in `white_list_formats` contained in
+        the directory.
+    """
+    def _recursive_list(subpath):
+        return sorted(os.walk(subpath, followlinks=follow_links), key=lambda tpl: tpl[0])
+
+    samples = 0
+    for root, _, files in _recursive_list(directory):
+        for fname in files:
+            is_valid = False
+            for extension in white_list_formats:
+                if fname.lower().endswith('.' + extension):
+                    is_valid = True
+                    break
+            if is_valid:
+                samples += 1
+    return samples
+
+
+def _list_valid_filenames_in_directory(directory, white_list_formats,
+                                       class_indices, follow_links):
+    """List paths of files in `subdir` relative from `directory` whose extensions are in `white_list_formats`.
+
+    # Arguments
+        directory: absolute path to a directory containing the files to list.
+            The directory name is used as class label and must be a key of `class_indices`.
+        white_list_formats: set of strings containing allowed extensions for
+            the files to be counted.
+        class_indices: dictionary mapping a class name to its index.
+
+    # Returns
+        classes: a list of class indices
+        filenames: the path of valid files in `directory`, relative from
+            `directory`'s parent (e.g., if `directory` is "dataset/class1",
+            the filenames will be ["class1/file1.jpg", "class1/file2.jpg", ...]).
+    """
+    def _recursive_list(subpath):
+        return sorted(os.walk(subpath, followlinks=follow_links), key=lambda tpl: tpl[0])
+
+    classes = []
+    filenames = []
+    subdir = os.path.basename(directory)
+    basedir = os.path.dirname(directory)
+    for root, _, files in _recursive_list(directory):
+        for fname in files:
+            is_valid = False
+            for extension in white_list_formats:
+                if fname.lower().endswith('.' + extension):
+                    is_valid = True
+                    break
+            if is_valid:
+                classes.append(class_indices[subdir])
+                # add filename relative to directory
+                absolute_path = os.path.join(root, fname)
+                filenames.append(os.path.relpath(absolute_path, basedir))
+    return classes, filenames
+
+
 class DirectoryIterator(Iterator):
     """Iterator capable of reading images from a directory on disk.
 
@@ -838,6 +910,8 @@ class DirectoryIterator(Iterator):
             `"binary"`: binary targets (if there are only two classes),
             `"categorical"`: categorical targets,
             `"sparse"`: integer targets,
+            `"input"`: targets are images identical to input images (mainly
+                used to work with autoencoders),
             `None`: no targets get yielded (only input images are yielded).
         batch_size: Integer, size of a batch.
         shuffle: Boolean, whether to shuffle the data between epochs.
@@ -858,7 +932,7 @@ class DirectoryIterator(Iterator):
                  classes=None, class_mode='categorical',
                  batch_size=32, shuffle=True, seed=None,
                  data_format=None,
-                 save_to_dir=None, save_prefix='', save_format='jpeg',
+                 save_to_dir=None, save_prefix='', save_format='png',
                  follow_links=False):
         if data_format is None:
             data_format = K.image_data_format()
@@ -881,10 +955,12 @@ class DirectoryIterator(Iterator):
             else:
                 self.image_shape = (1,) + self.target_size
         self.classes = classes
-        if class_mode not in {'categorical', 'binary', 'sparse', None}:
+        if class_mode not in {'categorical', 'binary', 'sparse',
+                              'input', None}:
             raise ValueError('Invalid class_mode:', class_mode,
                              '; expected one of "categorical", '
-                             '"binary", "sparse", or None.')
+                             '"binary", "sparse", "input"'
+                             ' or None.')
         self.class_mode = class_mode
         self.save_to_dir = save_to_dir
         self.save_prefix = save_prefix
@@ -906,38 +982,33 @@ class DirectoryIterator(Iterator):
         def _recursive_list(subpath):
             return sorted(os.walk(subpath, followlinks=follow_links), key=lambda tpl: tpl[0])
 
-        for subdir in classes:
-            subpath = os.path.join(directory, subdir)
-            for root, _, files in _recursive_list(subpath):
-                for fname in files:
-                    is_valid = False
-                    for extension in white_list_formats:
-                        if fname.lower().endswith('.' + extension):
-                            is_valid = True
-                            break
-                    if is_valid:
-                        self.samples += 1
+        pool = multiprocessing.pool.ThreadPool()
+        function_partial = partial(_count_valid_files_in_directory,
+                                   white_list_formats=white_list_formats,
+                                   follow_links=follow_links)
+        self.samples = sum(pool.map(function_partial,
+                                    (os.path.join(directory, subdir)
+                                     for subdir in classes)))
+
         print('Found %d images belonging to %d classes.' % (self.samples, self.num_class))
 
         # second, build an index of the images in the different class subfolders
+        results = []
+
         self.filenames = []
         self.classes = np.zeros((self.samples,), dtype='int32')
         i = 0
-        for subdir in classes:
-            subpath = os.path.join(directory, subdir)
-            for root, _, files in _recursive_list(subpath):
-                for fname in files:
-                    is_valid = False
-                    for extension in white_list_formats:
-                        if fname.lower().endswith('.' + extension):
-                            is_valid = True
-                            break
-                    if is_valid:
-                        self.classes[i] = self.class_indices[subdir]
-                        i += 1
-                        # add filename relative to directory
-                        absolute_path = os.path.join(root, fname)
-                        self.filenames.append(os.path.relpath(absolute_path, directory))
+        for dirpath in (os.path.join(directory, subdir) for subdir in classes):
+            results.append(pool.apply_async(_list_valid_filenames_in_directory,
+                                            (dirpath, white_list_formats,
+                                             self.class_indices, follow_links)))
+        for res in results:
+            classes, filenames = res.get()
+            self.classes[i:i + len(classes)] = classes
+            self.filenames += filenames
+            i += len(classes)
+        pool.close()
+        pool.join()
         super(DirectoryIterator, self).__init__(self.samples, batch_size, shuffle, seed)
 
     def next(self):
@@ -972,7 +1043,9 @@ class DirectoryIterator(Iterator):
                                                                   format=self.save_format)
                 img.save(os.path.join(self.save_to_dir, fname))
         # build batch of labels
-        if self.class_mode == 'sparse':
+        if self.class_mode == 'input':
+            batch_y = batch_x.copy()
+        elif self.class_mode == 'sparse':
             batch_y = self.classes[index_array]
         elif self.class_mode == 'binary':
             batch_y = self.classes[index_array].astype(K.floatx())

keras/preprocessing/sequence.py

@@ -104,7 +104,7 @@ def make_sampling_table(size, sampling_factor=1e-5):
         is the probability that a word of rank i should be sampled.
     """
     gamma = 0.577
-    rank = np.array(list(range(size)))
+    rank = np.arange(size)
     rank[0] = 1
     inv_fq = rank * (np.log(rank) + gamma) + 0.5 - 1. / (12. * rank)
     f = sampling_factor * inv_fq
@@ -127,7 +127,7 @@ def skipgrams(sequence, vocabulary_size,
             of word indices (integers). If using a `sampling_table`,
             word indices are expected to match the rank
             of the words in a reference dataset (e.g. 10 would encode
-            the 10-th most frequently occuring token).
+            the 10-th most frequently occurring token).
             Note that index 0 is expected to be a non-word and will be skipped.
         vocabulary_size: int. maximum possible word index + 1
         window_size: int. actually half-window.
@@ -191,3 +191,22 @@ def skipgrams(sequence, vocabulary_size,
         random.shuffle(labels)
 
     return couples, labels
+
+
+def _remove_long_seq(maxlen, seq, label):
+    """Removes sequences that exceed the maximum length.
+
+    # Arguments
+        maxlen: int, maximum length
+        seq: list of lists where each sublist is a sequence
+        label: list where each element is an integer
+
+    # Returns
+        new_seq, new_label: shortened lists for `seq` and `label`.
+    """
+    new_seq, new_label = [], []
+    for x, y in zip(seq, label):
+        if len(x) < maxlen:
+            new_seq.append(x)
+            new_label.append(y)
+    return new_seq, new_label

keras/preprocessing/text.py

@@ -8,10 +8,13 @@ from __future__ import division
 
 import string
 import sys
+import warnings
+from collections import OrderedDict
+from hashlib import md5
+
 import numpy as np
 from six.moves import range
 from six.moves import zip
-import warnings
 
 if sys.version_info < (3,):
     maketrans = string.maketrans
@@ -22,7 +25,7 @@ else:
 def text_to_word_sequence(text,
                           filters='!"#$%&()*+,-./:;<=>?@[\\]^_`{|}~\t\n',
                           lower=True, split=" "):
-    """Converts a text to a sequence of word indices.
+    """Converts a text to a sequence of words (or tokens).
 
     # Arguments
         text: Input text (string).
@@ -31,7 +34,7 @@ def text_to_word_sequence(text,
         split: Sentence split marker (string).
 
     # Returns
-        A list of integer word indices.
+        A list of words (or tokens).
     """
     if lower:
         text = text.lower()
@@ -44,11 +47,58 @@ def one_hot(text, n,
             filters='!"#$%&()*+,-./:;<=>?@[\\]^_`{|}~\t\n',
             lower=True,
             split=' '):
+    """One-hot encodes a text into a list of word indexes of size n.
+
+    This is a wrapper to the `hashing_trick` function using `hash` as the
+    hashing function, unicity of word to index mapping non-guaranteed.
+    """
+    return hashing_trick(text, n,
+                         hash_function=hash,
+                         filters=filters,
+                         lower=lower,
+                         split=split)
+
+
+def hashing_trick(text, n,
+                  hash_function=None,
+                  filters='!"#$%&()*+,-./:;<=>?@[\\]^_`{|}~\t\n',
+                  lower=True,
+                  split=' '):
+    """Converts a text to a sequence of indexes in a fixed-size hashing space.
+
+    # Arguments
+        text: Input text (string).
+        n: Dimension of the hashing space.
+        hash_function: if `None` uses python `hash` function, can be 'md5' or
+            any function that takes in input a string and returns a int.
+            Note that `hash` is not a stable hashing function, so
+            it is not consistent across different runs, while 'md5'
+            is a stable hashing function.
+        filters: Sequence of characters to filter out.
+        lower: Whether to convert the input to lowercase.
+        split: Sentence split marker (string).
+
+    # Returns
+        A list of integer word indices (unicity non-guaranteed).
+
+    `0` is a reserved index that won't be assigned to any word.
+
+    Two or more words may be assigned to the same index, due to possible
+    collisions by the hashing function.
+    The [probability](https://en.wikipedia.org/wiki/Birthday_problem#Probability_table)
+    of a collision is in relation to the dimension of the hashing space and
+    the number of distinct objects.
+    """
+    if hash_function is None:
+        hash_function = hash
+    elif hash_function == 'md5':
+        hash_function = lambda w: int(md5(w.encode()).hexdigest(), 16)
+
     seq = text_to_word_sequence(text,
                                 filters=filters,
                                 lower=lower,
                                 split=split)
-    return [(abs(hash(w)) % (n - 1) + 1) for w in seq]
+    return [(hash_function(w) % (n - 1) + 1) for w in seq]
 
 
 class Tokenizer(object):
@@ -68,7 +118,7 @@ class Tokenizer(object):
             tabs and line breaks, minus the `'` character.
         lower: boolean. Whether to convert the texts to lowercase.
         split: character or string to use for token splitting.
-        char_level: if True, every character will be treated as a word.
+        char_level: if True, every character will be treated as a token.
 
     By default, all punctuation is removed, turning the texts into
     space-separated sequences of words
@@ -92,7 +142,7 @@ class Tokenizer(object):
         if kwargs:
             raise TypeError('Unrecognized keyword arguments: ' + str(kwargs))
 
-        self.word_counts = {}
+        self.word_counts = OrderedDict()
         self.word_docs = {}
         self.filters = filters
         self.split = split

keras/utils/__init__.py

@@ -1,13 +1,16 @@
 from __future__ import absolute_import
 from . import np_utils
-from . import conv_utils
-from . import data_utils
 from . import generic_utils
+from . import data_utils
 from . import io_utils
+from . import conv_utils
 
 # Globally-importable utils.
 from .io_utils import HDF5Matrix
 from .data_utils import get_file
+from .data_utils import Sequence
+from .data_utils import GeneratorEnqueuer
+from .data_utils import OrderedEnqueuer
 from .generic_utils import CustomObjectScope
 from .generic_utils import custom_object_scope
 from .generic_utils import get_custom_objects

keras/utils/conv_utils.py

@@ -70,7 +70,7 @@ def convert_kernel(kernel):
     Also works reciprocally, since the transformation is its own inverse.
 
     # Arguments
-        kernel: Numpy array (4D or 5D).
+        kernel: Numpy array (3D, 4D or 5D).
 
     # Returns
         The converted kernel.
@@ -78,7 +78,8 @@ def convert_kernel(kernel):
     # Raises
         ValueError: in case of invalid kernel shape or invalid data_format.
     """
-    if not 4 <= kernel.ndim <= 5:
+    kernel = np.asarray(kernel)
+    if not 3 <= kernel.ndim <= 5:
         raise ValueError('Invalid kernel shape:', kernel.shape)
     slices = [slice(None, None, -1) for _ in range(kernel.ndim)]
     no_flip = (slice(None, None), slice(None, None))

keras/utils/data_utils.py

@@ -2,19 +2,32 @@
 from __future__ import absolute_import
 from __future__ import print_function
 
-import functools
-import tarfile
-import os
-import sys
-import shutil
 import hashlib
-from six.moves.urllib.request import urlopen
-from six.moves.urllib.error import URLError
+import multiprocessing
+import os
+import random
+import shutil
+import sys
+import tarfile
+import threading
+import time
+import zipfile
+from abc import abstractmethod
+from multiprocessing.pool import ThreadPool
+
+import numpy as np
+import six
 from six.moves.urllib.error import HTTPError
+from six.moves.urllib.error import URLError
+from six.moves.urllib.request import urlopen
+
+try:
+    import queue
+except ImportError:
+    import Queue as queue
 
 from ..utils.generic_utils import Progbar
 
-
 if sys.version_info[0] == 2:
     def urlretrieve(url, filename, reporthook=None, data=None):
         """Replacement for `urlretrive` for Python 2.
@@ -33,9 +46,12 @@ if sys.version_info[0] == 2:
                 a block size in bytes, and the total size of the file.
             data: `data` argument passed to `urlopen`.
         """
+
         def chunk_read(response, chunk_size=8192, reporthook=None):
-            total_size = response.info().get('Content-Length').strip()
-            total_size = int(total_size)
+            content_type = response.info().get('Content-Length')
+            total_size = -1
+            if content_type is not None:
+                total_size = int(content_type.strip())
             count = 0
             while 1:
                 chunk = response.read(chunk_size)
@@ -55,24 +71,108 @@ else:
     from six.moves.urllib.request import urlretrieve
 
 
-def get_file(fname, origin, untar=False,
-             md5_hash=None, cache_subdir='datasets'):
-    """Downloads a file from a URL if it not already in the cache.
-
-    Passing the MD5 hash will verify the file after download
-    as well as if it is already present in the cache.
+def _extract_archive(file_path, path='.', archive_format='auto'):
+    """Extracts an archive if it matches tar, tar.gz, tar.bz, or zip formats.
 
     # Arguments
-        fname: name of the file
-        origin: original URL of the file
-        untar: boolean, whether the file should be decompressed
-        md5_hash: MD5 hash of the file for verification
-        cache_subdir: directory being used as the cache
+        file_path: path to the archive file
+        path: path to extract the archive file
+        archive_format: Archive format to try for extracting the file.
+            Options are 'auto', 'tar', 'zip', and None.
+            'tar' includes tar, tar.gz, and tar.bz files.
+            The default 'auto' is ['tar', 'zip'].
+            None or an empty list will return no matches found.
+
+    # Returns
+        True if a match was found and an archive extraction was completed,
+        False otherwise.
+    """
+    if archive_format is None:
+        return False
+    if archive_format is 'auto':
+        archive_format = ['tar', 'zip']
+    if isinstance(archive_format, six.string_types):
+        archive_format = [archive_format]
+
+    for archive_type in archive_format:
+        if archive_type is 'tar':
+            open_fn = tarfile.open
+            is_match_fn = tarfile.is_tarfile
+        if archive_type is 'zip':
+            open_fn = zipfile.ZipFile
+            is_match_fn = zipfile.is_zipfile
+
+        if is_match_fn(file_path):
+            with open_fn(file_path) as archive:
+                try:
+                    archive.extractall(path)
+                except (tarfile.TarError, RuntimeError,
+                        KeyboardInterrupt):
+                    if os.path.exists(path):
+                        if os.path.isfile(path):
+                            os.remove(path)
+                        else:
+                            shutil.rmtree(path)
+                    raise
+            return True
+    return False
+
+
+def get_file(fname,
+             origin,
+             untar=False,
+             md5_hash=None,
+             file_hash=None,
+             cache_subdir='datasets',
+             hash_algorithm='auto',
+             extract=False,
+             archive_format='auto',
+             cache_dir=None):
+    """Downloads a file from a URL if it not already in the cache.
+
+    By default the file at the url `origin` is downloaded to the
+    cache_dir `~/.keras`, placed in the cache_subdir `datasets`,
+    and given the filename `fname`. The final location of a file
+    `example.txt` would therefore be `~/.keras/datasets/example.txt`.
+
+    Files in tar, tar.gz, tar.bz, and zip formats can also be extracted.
+    Passing a hash will verify the file after download. The command line
+    programs `shasum` and `sha256sum` can compute the hash.
+
+    # Arguments
+        fname: Name of the file. If an absolute path `/path/to/file.txt` is
+            specified the file will be saved at that location.
+        origin: Original URL of the file.
+        untar: Deprecated in favor of 'extract'.
+            boolean, whether the file should be decompressed
+        md5_hash: Deprecated in favor of 'file_hash'.
+            md5 hash of the file for verification
+        file_hash: The expected hash string of the file after download.
+            The sha256 and md5 hash algorithms are both supported.
+        cache_subdir: Subdirectory under the Keras cache dir where the file is
+            saved. If an absolute path `/path/to/folder` is
+            specified the file will be saved at that location.
+        hash_algorithm: Select the hash algorithm to verify the file.
+            options are 'md5', 'sha256', and 'auto'.
+            The default 'auto' detects the hash algorithm in use.
+        extract: True tries extracting the file as an Archive, like tar or zip.
+        archive_format: Archive format to try for extracting the file.
+            Options are 'auto', 'tar', 'zip', and None.
+            'tar' includes tar, tar.gz, and tar.bz files.
+            The default 'auto' is ['tar', 'zip'].
+            None or an empty list will return no matches found.
+        cache_dir: Location to store cached files, when None it
+            defaults to the [Keras Directory](/faq/#where-is-the-keras-configuration-filed-stored).
 
     # Returns
         Path to the downloaded file
     """
-    datadir_base = os.path.expanduser(os.path.join('~', '.keras'))
+    if cache_dir is None:
+        cache_dir = os.path.expanduser(os.path.join('~', '.keras'))
+    if md5_hash is not None and file_hash is None:
+        file_hash = md5_hash
+        hash_algorithm = 'md5'
+    datadir_base = os.path.expanduser(cache_dir)
     if not os.access(datadir_base, os.W_OK):
         datadir_base = os.path.join('/tmp', '.keras')
     datadir = os.path.join(datadir_base, cache_subdir)
@@ -88,29 +188,36 @@ def get_file(fname, origin, untar=False,
     download = False
     if os.path.exists(fpath):
         # File found; verify integrity if a hash was provided.
-        if md5_hash is not None:
-            if not validate_file(fpath, md5_hash):
+        if file_hash is not None:
+            if not validate_file(fpath, file_hash, algorithm=hash_algorithm):
                 print('A local file was found, but it seems to be '
-                      'incomplete or outdated.')
+                      'incomplete or outdated because the ' + hash_algorithm +
+                      ' file hash does not match the original value of ' +
+                      file_hash + ' so we will re-download the data.')
                 download = True
     else:
         download = True
 
     if download:
         print('Downloading data from', origin)
-        progbar = None
 
-        def dl_progress(count, block_size, total_size, progbar=None):
-            if progbar is None:
-                progbar = Progbar(total_size)
+        class ProgressTracker(object):
+            # Maintain progbar for the lifetime of download.
+            # This design was chosen for Python 2.7 compatibility.
+            progbar = None
+
+        def dl_progress(count, block_size, total_size):
+            if ProgressTracker.progbar is None:
+                if total_size is -1:
+                    total_size = None
+                ProgressTracker.progbar = Progbar(total_size)
             else:
-                progbar.update(count * block_size)
+                ProgressTracker.progbar.update(count * block_size)
 
         error_msg = 'URL fetch failure on {}: {} -- {}'
         try:
             try:
-                urlretrieve(origin, fpath,
-                            functools.partial(dl_progress, progbar=progbar))
+                urlretrieve(origin, fpath, dl_progress)
             except URLError as e:
                 raise Exception(error_msg.format(origin, e.errno, e.reason))
             except HTTPError as e:
@@ -119,42 +226,411 @@ def get_file(fname, origin, untar=False,
             if os.path.exists(fpath):
                 os.remove(fpath)
             raise
-        progbar = None
+        ProgressTracker.progbar = None
 
     if untar:
         if not os.path.exists(untar_fpath):
-            print('Untaring file...')
-            tfile = tarfile.open(fpath, 'r:gz')
-            try:
-                tfile.extractall(path=datadir)
-            except (Exception, KeyboardInterrupt) as e:
-                if os.path.exists(untar_fpath):
-                    if os.path.isfile(untar_fpath):
-                        os.remove(untar_fpath)
-                    else:
-                        shutil.rmtree(untar_fpath)
-                raise
-            tfile.close()
+            _extract_archive(fpath, datadir, archive_format='tar')
         return untar_fpath
 
+    if extract:
+        _extract_archive(fpath, datadir, archive_format)
+
     return fpath
 
 
-def validate_file(fpath, md5_hash):
-    """Validates a file against a MD5 hash.
+def _hash_file(fpath, algorithm='sha256', chunk_size=65535):
+    """Calculates a file sha256 or md5 hash.
+
+    # Example
+
+    ```python
+       >>> from keras.data_utils import _hash_file
+       >>> _hash_file('/path/to/file.zip')
+       'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'
+    ```
 
     # Arguments
         fpath: path to the file being validated
-        md5_hash: the MD5 hash being validated against
+        algorithm: hash algorithm, one of 'auto', 'sha256', or 'md5'.
+            The default 'auto' detects the hash algorithm in use.
+        chunk_size: Bytes to read at a time, important for large files.
+
+    # Returns
+        The file hash
+    """
+    if (algorithm is 'sha256') or (algorithm is 'auto' and len(hash) is 64):
+        hasher = hashlib.sha256()
+    else:
+        hasher = hashlib.md5()
+
+    with open(fpath, 'rb') as fpath_file:
+        for chunk in iter(lambda: fpath_file.read(chunk_size), b''):
+            hasher.update(chunk)
+
+    return hasher.hexdigest()
+
+
+def validate_file(fpath, file_hash, algorithm='auto', chunk_size=65535):
+    """Validates a file against a sha256 or md5 hash.
+
+    # Arguments
+        fpath: path to the file being validated
+        file_hash:  The expected hash string of the file.
+            The sha256 and md5 hash algorithms are both supported.
+        algorithm: Hash algorithm, one of 'auto', 'sha256', or 'md5'.
+            The default 'auto' detects the hash algorithm in use.
+        chunk_size: Bytes to read at a time, important for large files.
 
     # Returns
         Whether the file is valid
     """
-    hasher = hashlib.md5()
-    with open(fpath, 'rb') as f:
-        buf = f.read()
-        hasher.update(buf)
-    if str(hasher.hexdigest()) == str(md5_hash):
+    if ((algorithm is 'sha256') or
+            (algorithm is 'auto' and len(file_hash) is 64)):
+        hasher = 'sha256'
+    else:
+        hasher = 'md5'
+
+    if str(_hash_file(fpath, hasher, chunk_size)) == str(file_hash):
         return True
     else:
         return False
+
+
+class Sequence(object):
+    """Base object for fitting to a sequence of data, such as a dataset.
+
+    Every `Sequence` must implements the `__getitem__` and the `__len__` methods.
+
+    # Examples
+
+    ```python
+    from skimage.io import imread
+    from skimage.transform import resize
+    import numpy as np
+
+    # Here, `x_set` is list of path to the images
+    # and `y_set` are the associated classes.
+
+    class CIFAR10Sequence(Sequence):
+        def __init__(self, x_set, y_set, batch_size):
+            self.X,self.y = x_set,y_set
+            self.batch_size = batch_size
+
+        def __len__(self):
+            return len(self.X) // self.batch_size
+
+        def __getitem__(self,idx):
+            batch_x = self.X[idx*self.batch_size:(idx+1)*self.batch_size]
+            batch_y = self.y[idx*self.batch_size:(idx+1)*self.batch_size]
+
+            return np.array([
+                resize(imread(file_name), (200,200))
+                   for file_name in batch_x]), np.array(batch_y)
+    ```
+    """
+
+    @abstractmethod
+    def __getitem__(self, index):
+        """Gets batch at position `index`.
+
+        # Arguments
+            index: position of the batch in the Sequence.
+
+        # Returns
+            A batch
+        """
+        raise NotImplementedError
+
+    @abstractmethod
+    def __len__(self):
+        """Number of batch in the Sequence.
+
+        # Returns
+            The number of batches in the Sequence.
+        """
+        raise NotImplementedError
+
+
+def get_index(ds, i):
+    """Quick fix for Python2, otherwise, it cannot be pickled.
+
+    # Arguments
+        ds: a Sequence object
+        i: index
+
+    # Returns
+        The value at index `i`.
+    """
+    return ds[i]
+
+
+class SequenceEnqueuer(object):
+    """Base class to enqueue inputs.
+
+    The task of an Enqueuer is to use parallelism to speed up preprocessing.
+    This is done with processes or threads.
+
+    # Examples
+
+    ```python
+    enqueuer = SequenceEnqueuer(...)
+    enqueuer.start()
+    datas = enqueuer.get()
+    for data in datas:
+        # Use the inputs; training, evaluating, predicting.
+        # ... stop sometime.
+    enqueuer.close()
+    ```
+
+    The `enqueuer.get()` should be an infinite stream of datas.
+
+    """
+
+    @abstractmethod
+    def is_running(self):
+        raise NotImplementedError
+
+    @abstractmethod
+    def start(self, workers=1, max_queue_size=10):
+        """Starts the handler's workers.
+
+        # Arguments
+            workers: number of worker threads
+            max_queue_size: queue size
+                (when full, threads could block on `put()`).
+        """
+        raise NotImplementedError
+
+    @abstractmethod
+    def stop(self, timeout=None):
+        """Stop running threads and wait for them to exit, if necessary.
+
+        Should be called by the same thread which called start().
+
+        # Arguments
+            timeout: maximum time to wait on thread.join()
+        """
+        raise NotImplementedError
+
+    @abstractmethod
+    def get(self):
+        """Creates a generator to extract data from the queue.
+
+        Skip the data if it is `None`.
+
+        # Returns
+            Generator yielding tuples `(inputs, targets)`
+                or `(inputs, targets, sample_weights)`.
+        """
+        raise NotImplementedError
+
+
+class OrderedEnqueuer(SequenceEnqueuer):
+    """Builds a Enqueuer from a Sequence.
+
+    Used in `fit_generator`, `evaluate_generator`, `predict_generator`.
+
+    # Arguments
+        sequence: A `keras.utils.data_utils.Sequence` object.
+        use_multiprocessing: use multiprocessing if True, otherwise threading
+        scheduling: Sequential querying of datas if 'sequential', random otherwise.
+    """
+
+    def __init__(self, sequence,
+                 use_multiprocessing=False,
+                 scheduling='sequential'):
+        self.sequence = sequence
+        self.use_multiprocessing = use_multiprocessing
+        self.scheduling = scheduling
+        self.workers = 0
+        self.executor = None
+        self.queue = None
+        self.run_thread = None
+        self.stop_signal = None
+
+    def is_running(self):
+        return self.stop_signal is not None and not self.stop_signal.is_set()
+
+    def start(self, workers=1, max_queue_size=10):
+        """Start the handler's workers.
+
+        # Arguments
+            workers: number of worker threads
+            max_queue_size: queue size
+                (when full, workers could block on `put()`)
+        """
+        if self.use_multiprocessing:
+            self.executor = multiprocessing.Pool(workers)
+        else:
+            self.executor = ThreadPool(workers)
+        self.queue = queue.Queue(max_queue_size)
+        self.stop_signal = threading.Event()
+        self.run_thread = threading.Thread(target=self._run)
+        self.run_thread.daemon = True
+        self.run_thread.start()
+
+    def _run(self):
+        """Function to submit request to the executor and queue the `Future` objects."""
+        sequence = list(range(len(self.sequence)))
+        while True:
+            if self.scheduling is not 'sequential':
+                random.shuffle(sequence)
+            for i in sequence:
+                if self.stop_signal.is_set():
+                    return
+                self.queue.put(
+                    self.executor.apply_async(get_index,
+                                              (self.sequence, i)), block=True)
+
+    def get(self):
+        """Creates a generator to extract data from the queue.
+
+        Skip the data if it is `None`.
+
+        # Returns
+            Generator yielding tuples (inputs, targets)
+                or (inputs, targets, sample_weights)
+        """
+        try:
+            while self.is_running():
+                inputs = self.queue.get(block=True).get()
+                if inputs is not None:
+                    yield inputs
+        except Exception as e:
+            self.stop()
+            raise StopIteration(e)
+
+    def stop(self, timeout=None):
+        """Stops running threads and wait for them to exit, if necessary.
+
+        Should be called by the same thread which called `start()`.
+
+        # Arguments
+            timeout: maximum time to wait on `thread.join()`
+        """
+        self.stop_signal.set()
+        with self.queue.mutex:
+            self.queue.queue.clear()
+            self.queue.unfinished_tasks = 0
+            self.queue.not_full.notify()
+        self.executor.close()
+        self.executor.join()
+        self.run_thread.join(timeout)
+
+
+class GeneratorEnqueuer(SequenceEnqueuer):
+    """Builds a queue out of a data generator.
+
+    Used in `fit_generator`, `evaluate_generator`, `predict_generator`.
+
+    # Arguments
+        generator: a generator function which endlessly yields data
+        use_multiprocessing: use multiprocessing if True, otherwise threading
+        wait_time: time to sleep in-between calls to `put()`
+        random_seed: Initial seed for workers,
+            will be incremented by one for each workers.
+    """
+
+    def __init__(self, generator,
+                 use_multiprocessing=False,
+                 wait_time=0.05,
+                 random_seed=None):
+        self.wait_time = wait_time
+        self._generator = generator
+        self._use_multiprocessing = use_multiprocessing
+        self._threads = []
+        self._stop_event = None
+        self.queue = None
+        self.random_seed = random_seed
+
+    def start(self, workers=1, max_queue_size=10):
+        """Kicks off threads which add data from the generator into the queue.
+
+        # Arguments
+            workers: number of worker threads
+            max_queue_size: queue size
+                (when full, threads could block on `put()`)
+        """
+
+        def data_generator_task():
+            while not self._stop_event.is_set():
+                try:
+                    if self._use_multiprocessing or self.queue.qsize() < max_queue_size:
+                        generator_output = next(self._generator)
+                        self.queue.put(generator_output)
+                    else:
+                        time.sleep(self.wait_time)
+                except Exception:
+                    self._stop_event.set()
+                    raise
+
+        try:
+            if self._use_multiprocessing:
+                self.queue = multiprocessing.Queue(maxsize=max_queue_size)
+                self._stop_event = multiprocessing.Event()
+            else:
+                self.queue = queue.Queue()
+                self._stop_event = threading.Event()
+
+            for _ in range(workers):
+                if self._use_multiprocessing:
+                    # Reset random seed else all children processes
+                    # share the same seed
+                    np.random.seed(self.random_seed)
+                    thread = multiprocessing.Process(target=data_generator_task)
+                    thread.daemon = True
+                    if self.random_seed is not None:
+                        self.random_seed += 1
+                else:
+                    thread = threading.Thread(target=data_generator_task)
+                self._threads.append(thread)
+                thread.start()
+        except:
+            self.stop()
+            raise
+
+    def is_running(self):
+        return self._stop_event is not None and not self._stop_event.is_set()
+
+    def stop(self, timeout=None):
+        """Stops running threads and wait for them to exit, if necessary.
+
+        Should be called by the same thread which called `start()`.
+
+        # Arguments
+            timeout: maximum time to wait on `thread.join()`.
+        """
+        if self.is_running():
+            self._stop_event.set()
+
+        for thread in self._threads:
+            if thread.is_alive():
+                if self._use_multiprocessing:
+                    thread.terminate()
+                else:
+                    thread.join(timeout)
+
+        if self._use_multiprocessing:
+            if self.queue is not None:
+                self.queue.close()
+
+        self._threads = []
+        self._stop_event = None
+        self.queue = None
+
+    def get(self):
+        """Creates a generator to extract data from the queue.
+
+        Skip the data if it is `None`.
+
+        # Returns
+            A generator
+        """
+        while self.is_running():
+            if not self.queue.empty():
+                inputs = self.queue.get()
+                if inputs is not None:
+                    yield inputs
+            else:
+                time.sleep(self.wait_time)

keras/utils/generic_utils.py

@@ -27,8 +27,8 @@ class CustomObjectScope(object):
     Consider a custom object `MyObject`
 
     ```python
-        with CustomObjectScope({"MyObject":MyObject}):
-            layer = Dense(..., W_regularizer="MyObject")
+        with CustomObjectScope({'MyObject':MyObject}):
+            layer = Dense(..., kernel_regularizer='MyObject')
             # save, load, etc. will recognize custom object by name
     ```
     """
@@ -63,8 +63,8 @@ def custom_object_scope(*args):
     Consider a custom object `MyObject`
 
     ```python
-        with custom_object_scope({"MyObject":MyObject}):
-            layer = Dense(..., W_regularizer="MyObject")
+        with custom_object_scope({'MyObject':MyObject}):
+            layer = Dense(..., kernel_regularizer='MyObject')
             # save, load, etc. will recognize custom object by name
     ```
 
@@ -89,7 +89,7 @@ def get_custom_objects():
 
     ```python
         get_custom_objects().clear()
-        get_custom_objects()["MyObject"] = MyObject
+        get_custom_objects()['MyObject'] = MyObject
     ```
 
     # Returns
@@ -132,18 +132,20 @@ def deserialize_keras_object(identifier, module_objects=None,
                 raise ValueError('Unknown ' + printable_module_name +
                                  ': ' + class_name)
         if hasattr(cls, 'from_config'):
-            arg_spec = inspect.getargspec(cls.from_config)
-            if 'custom_objects' in arg_spec.args:
-                custom_objects = custom_objects or {}
+            custom_objects = custom_objects or {}
+            if has_arg(cls.from_config, 'custom_objects'):
                 return cls.from_config(config['config'],
                                        custom_objects=dict(list(_GLOBAL_CUSTOM_OBJECTS.items()) +
                                                            list(custom_objects.items())))
-            return cls.from_config(config['config'])
+            with CustomObjectScope(custom_objects):
+                return cls.from_config(config['config'])
         else:
             # Then `cls` may be a function returning a class.
             # in this case by convention `config` holds
             # the kwargs of the function.
-            return cls(**config['config'])
+            custom_objects = custom_objects or {}
+            with CustomObjectScope(custom_objects):
+                return cls(**config['config'])
     elif isinstance(identifier, six.string_types):
         function_name = identifier
         if custom_objects and function_name in custom_objects:
@@ -153,7 +155,7 @@ def deserialize_keras_object(identifier, module_objects=None,
         else:
             fn = module_objects.get(function_name)
             if fn is None:
-                raise ValueError('Unknown ' + printable_module_name,
+                raise ValueError('Unknown ' + printable_module_name +
                                  ':' + function_name)
         return fn
     else:
@@ -161,10 +163,6 @@ def deserialize_keras_object(identifier, module_objects=None,
                          printable_module_name + ': ' + identifier)
 
 
-def make_tuple(*args):
-    return args
-
-
 def func_dump(func):
     """Serializes a user defined function.
 
@@ -208,16 +206,60 @@ def func_load(code, defaults=None, closure=None, globs=None):
                                      closure=closure)
 
 
+def has_arg(fn, name, accept_all=False):
+    """Checks if a callable accepts a given keyword argument.
+
+    For Python 2, checks if there is an argument with the given name.
+
+    For Python 3, checks if there is an argument with the given name, and
+    also whether this argument can be called with a keyword (i.e. if it is
+    not a positional-only argument).
+
+    # Arguments
+        fn: Callable to inspect.
+        name: Check if `fn` can be called with `name` as a keyword argument.
+        accept_all: What to return if there is no parameter called `name`
+                    but the function accepts a `**kwargs` argument.
+
+    # Returns
+        bool, whether `fn` accepts a `name` keyword argument.
+    """
+    if sys.version_info < (3,):
+        arg_spec = inspect.getargspec(fn)
+        if accept_all and arg_spec.keywords is not None:
+            return True
+        return (name in arg_spec.args)
+    elif sys.version_info < (3, 3):
+        arg_spec = inspect.getfullargspec(fn)
+        if accept_all and arg_spec.varkw is not None:
+            return True
+        return (name in arg_spec.args or
+                name in arg_spec.kwonlyargs)
+    else:
+        signature = inspect.signature(fn)
+        parameter = signature.parameters.get(name)
+        if parameter is None:
+            if accept_all:
+                for param in signature.parameters.values():
+                    if param.kind == inspect.Parameter.VAR_KEYWORD:
+                        return True
+            return False
+        return (parameter.kind in (inspect.Parameter.POSITIONAL_OR_KEYWORD,
+                                   inspect.Parameter.KEYWORD_ONLY))
+
+
 class Progbar(object):
     """Displays a progress bar.
 
     # Arguments
-        target: Total number of steps expected.
+        target: Total number of steps expected, None if unknown.
         interval: Minimum visual progress update interval (in seconds).
     """
 
     def __init__(self, target, width=30, verbose=1, interval=0.05):
         self.width = width
+        if target is None:
+            target = -1
         self.target = target
         self.sum_values = {}
         self.unique_values = []
@@ -257,21 +299,22 @@ class Progbar(object):
             sys.stdout.write('\b' * prev_total_width)
             sys.stdout.write('\r')
 
-            numdigits = int(np.floor(np.log10(self.target))) + 1
-            barstr = '%%%dd/%%%dd [' % (numdigits, numdigits)
-            bar = barstr % (current, self.target)
-            prog = float(current) / self.target
-            prog_width = int(self.width * prog)
-            if prog_width > 0:
-                bar += ('=' * (prog_width - 1))
-                if current < self.target:
-                    bar += '>'
-                else:
-                    bar += '='
-            bar += ('.' * (self.width - prog_width))
-            bar += ']'
-            sys.stdout.write(bar)
-            self.total_width = len(bar)
+            if self.target is not -1:
+                numdigits = int(np.floor(np.log10(self.target))) + 1
+                barstr = '%%%dd/%%%dd [' % (numdigits, numdigits)
+                bar = barstr % (current, self.target)
+                prog = float(current) / self.target
+                prog_width = int(self.width * prog)
+                if prog_width > 0:
+                    bar += ('=' * (prog_width - 1))
+                    if current < self.target:
+                        bar += '>'
+                    else:
+                        bar += '='
+                bar += ('.' * (self.width - prog_width))
+                bar += ']'
+                sys.stdout.write(bar)
+                self.total_width = len(bar)
 
             if current:
                 time_per_unit = (now - self.start) / current
@@ -279,7 +322,7 @@ class Progbar(object):
                 time_per_unit = 0
             eta = time_per_unit * (self.target - current)
             info = ''
-            if current < self.target:
+            if current < self.target and self.target is not -1:
                 info += ' - ETA: %ds' % eta
             else:
                 info += ' - %ds' % (now - self.start)

keras/utils/io_utils.py

@@ -63,8 +63,13 @@ class HDF5Matrix(object):
 
     def __getitem__(self, key):
         if isinstance(key, slice):
-            if key.stop + self.start <= self.end:
-                idx = slice(key.start + self.start, key.stop + self.start)
+            start, stop = key.start, key.stop
+            if start is None:
+                start = 0
+            if stop is None:
+                stop = self.data.shape[0]
+            if stop + self.start <= self.end:
+                idx = slice(start + self.start, stop + self.start)
             else:
                 raise IndexError
         elif isinstance(key, int):
@@ -91,8 +96,40 @@ class HDF5Matrix(object):
 
     @property
     def shape(self):
+        """Gets a numpy-style shape tuple giving the dataset dimensions.
+
+        # Returns
+            A numpy-style shape tuple.
+        """
         return (self.end - self.start,) + self.data.shape[1:]
 
+    @property
+    def dtype(self):
+        """Gets the datatype of the dataset.
+
+        # Returns
+            A numpy dtype string.
+        """
+        return self.data.dtype
+
+    @property
+    def ndim(self):
+        """Gets the number of dimensions (rank) of the dataset.
+
+        # Returns
+            An integer denoting the number of dimensions (rank) of the dataset.
+        """
+        return self.data.ndim
+
+    @property
+    def size(self):
+        """Gets the total dataset size (number of elements).
+
+        # Returns
+            An integer denoting the number of elements in the dataset.
+        """
+        return np.prod(self.shape)
+
 
 def ask_to_proceed_with_overwrite(filepath):
     """Produces a prompt asking about overwriting a file.