merge

* Improve documents

* Fix style

.github/stale.yml

@@ -0,0 +1,19 @@
+# Configuration for probot-stale - https://github.com/probot/stale
+
+# Number of days of inactivity before an Issue or Pull Request becomes stale
+daysUntilStale: 90
+# Number of days of inactivity before a stale Issue or Pull Request is closed
+daysUntilClose: 30
+# Issues or Pull Requests with these labels will never be considered stale
+exemptLabels:
+  - bug
+  - Announcement
+  - help wanted
+  - To investigate
+# Label to use when marking as stale
+staleLabel: stale
+# Comment to post when marking as stale. Set to `false` to disable
+markComment: >
+  This issue has been automatically marked as stale because it has not had
+  recent activity. It will be closed after 30 days if no further activity 
+  occurs, but feel free to re-open a closed issue if needed.

.gitignore

@@ -11,6 +11,7 @@ docs/theme/*
 docs/sources/*
 tags
 Keras.egg-info
+examples/img/*
 
 # test-related
 .coverage

.travis.yml

@@ -7,6 +7,8 @@ matrix:
           env: KERAS_BACKEND=tensorflow TEST_MODE=PEP8
         - python: 2.7
           env: KERAS_BACKEND=tensorflow TEST_MODE=INTEGRATION_TESTS
+        - python: 3.5
+          env: KERAS_BACKEND=tensorflow TEST_MODE=DOC
         - python: 2.7
           env: KERAS_BACKEND=tensorflow
         - python: 3.5
@@ -34,7 +36,7 @@ install:
 
   - conda create -q -n test-environment python=$TRAVIS_PYTHON_VERSION numpy scipy matplotlib pandas pytest h5py
   - source activate test-environment
-  - pip install git+git://github.com/Theano/Theano.git
+  - pip install theano
 
   # install PIL for preprocessing tests
   - if [[ "$TRAVIS_PYTHON_VERSION" == "2.7" ]]; then
@@ -45,7 +47,7 @@ install:
 
   - pip install -e .[tests]
 
-  # install TensorFlow
+  # install TensorFlow (CPU version).
   - pip install tensorflow
 
 # command to run tests
@@ -61,8 +63,8 @@ script:
        PYTHONPATH=$PWD:$PYTHONPATH py.test tests/integration_tests;
     elif [[ "$TEST_MODE" == "PEP8" ]]; then
        PYTHONPATH=$PWD:$PYTHONPATH py.test --pep8 -m pep8 -n0;
+    elif [[ "$TEST_MODE" == "DOC" ]]; then
+        PYTHONPATH=$PWD:$PYTHONPATH py.test tests/test_documentation.py;
     else
-       PYTHONPATH=$PWD:$PYTHONPATH py.test tests/ --ignore=tests/integration_tests;
+       PYTHONPATH=$PWD:$PYTHONPATH py.test tests/ --ignore=tests/integration_tests --ignore=tests/test_documentation.py --cov=keras tests/ --cov-fail-under 78 --cov-report term-missing;
     fi
-after_success:
-  - coveralls

CONTRIBUTING.md

@@ -30,15 +30,26 @@ You can also use Github issues to request features you would like to see in Kera
 
 3. After discussing the feature you may choose to attempt a Pull Request. If you're at all able, start writing some code. We always have more work to do than time to do it. If you can write some code then that will speed the process along.
 
+
+## Requests for Contributions
+
+[This is the board](https://github.com/fchollet/keras/projects/1) where we list current outstanding issues and features to be added. If you want to start contributing to Keras, this is the place to start.
+
+
 ## Pull Requests
 
-We love pull requests. Here's a quick guide:
+**Where should I submit my pull request?**
+
+1. **Keras improvements and bugfixes** go to the [Keras `master` branch](https://github.com/fchollet/keras/tree/master).
+2. **New features** such as layers and datasets go to [keras-contrib](https://github.com/farizrahman4u/keras-contrib). Unless it is a new feature listed in [Requests for Contributions](https://github.com/fchollet/keras/projects/1), in which case it belongs in core Keras.
+
+Here's a quick guide to submitting your improvements:
 
 1. If your PR introduces a change in functionality, make sure you start by opening an issue to discuss whether the change should be made, and how to handle it. This will save you from having your PR closed down the road! Of course, if your PR is a simple bug fix, you don't need to do that.
 
 2. Write the code. This is the hard part!
 
-3. Make sure any new function or class you introduce has proper docstrings. Make sure any code you touch still has up-to-date docstrings and documentation.
+3. Make sure any new function or class you introduce has proper docstrings. Make sure any code you touch still has up-to-date docstrings and documentation. **Docstring style should be respected.** In particular, they should be formatted in MarkDown, and there should be sections for `Arguments`, `Returns`, `Raises` (if applicable). Look at other docstrings in the codebase for examples.
 
 4. Write tests. Your code should have full unit test coverage. If you want to see your PR merged promptly, this is crucial.
 
@@ -46,19 +57,20 @@ We love pull requests. Here's a quick guide:
   - You will need to install the test requirements as well: `pip install -e .[tests]`.
 
 6. Make sure all tests are passing:
-  - with the Theano backend, on Python 2.7 and Python 3.5
-  - with the TensorFlow backend, on Python 2.7
+  - with the Theano backend, on Python 2.7 and Python 3.5. Make sure you have the development version of Theano.
+  - with the TensorFlow backend, on Python 2.7 and Python 3.5. Make sure you have the development version of TensorFlow.
 
 7. We use PEP8 syntax conventions, but we aren't dogmatic when it comes to line length. Make sure your lines stay reasonably sized, though. To make your life easier, we recommend running a PEP8 linter:
   - Install PEP8 packages: `pip install pep8 pytest-pep8 autopep8`
   - Run a standalone PEP8 check: `py.test --pep8 -m pep8`
   - You can automatically fix some PEP8 error by running: `autopep8 -i --select <errors> <FILENAME>` for example: `autopep8 -i --select E128 tests/keras/backend/test_backends.py`
 
-8. When committing, use appropriate, descriptive commit messages. Make sure that your branch history is not a string of "bug fix", "fix", "oops", etc. When submitting your PR, squash your commits into a single commit with an appropriate commit message, to make sure the project history stays clean and readable. See ['rebase and squash'](http://rebaseandsqua.sh/) for technical help on how to squash your commits.
+8. When committing, use appropriate, descriptive commit messages.
 
 9. Update the documentation. If introducing new functionality, make sure you include code snippets demonstrating the usage of your new feature.
 
-10. Submit your PR. If your changes have been approved in a previous discussion, and if you have complete (and passing) unit tests, your PR is likely to be merged promptly. Otherwise, well...
+10. Submit your PR. If your changes have been approved in a previous discussion, and if you have complete (and passing) unit tests as well as proper docstrings/documentation, your PR is likely to be merged promptly. Otherwise, well...
+
 
 ## Adding new examples
 

docker/Dockerfile

@@ -7,10 +7,10 @@ RUN mkdir -p $CONDA_DIR && \
     echo export PATH=$CONDA_DIR/bin:'$PATH' > /etc/profile.d/conda.sh && \
     apt-get update && \
     apt-get install -y wget git libhdf5-dev g++ graphviz && \
-    wget --quiet https://repo.continuum.io/miniconda/Miniconda3-3.9.1-Linux-x86_64.sh && \
-    echo "6c6b44acdd0bc4229377ee10d52c8ac6160c336d9cdd669db7371aa9344e1ac3 *Miniconda3-3.9.1-Linux-x86_64.sh" | sha256sum -c - && \
-    /bin/bash /Miniconda3-3.9.1-Linux-x86_64.sh -f -b -p $CONDA_DIR && \
-    rm Miniconda3-3.9.1-Linux-x86_64.sh
+    wget --quiet https://repo.continuum.io/miniconda/Miniconda3-4.2.12-Linux-x86_64.sh && \
+    echo "c59b3dd3cad550ac7596e0d599b91e75d88826db132e4146030ef471bb434e9a *Miniconda3-4.2.12-Linux-x86_64.sh" | sha256sum -c - && \
+    /bin/bash /Miniconda3-4.2.12-Linux-x86_64.sh -f -b -p $CONDA_DIR && \
+    rm Miniconda3-4.2.12-Linux-x86_64.sh
 
 ENV NB_USER keras
 ENV NB_UID 1000
@@ -24,13 +24,14 @@ RUN useradd -m -s /bin/bash -N -u $NB_UID $NB_USER && \
 USER keras
 
 # Python
-ARG python_version=3.5.2
-ARG tensorflow_version=0.12.0rc0-cp35-cp35m
+ARG python_version=3.5
+
 RUN conda install -y python=${python_version} && \
-    pip install https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow_gpu-${tensorflow_version}-linux_x86_64.whl && \
-    pip install git+git://github.com/Theano/Theano.git && \
-    pip install ipdb pytest pytest-cov python-coveralls coverage==3.7.1 pytest-xdist pep8 pytest-pep8 pydot_ng && \
-    conda install Pillow scikit-learn notebook pandas matplotlib nose pyyaml six h5py && \
+    pip install --upgrade pip && \
+    pip install tensorflow-gpu && \
+    conda install Pillow scikit-learn notebook pandas matplotlib mkl nose pyyaml six h5py && \
+    conda install theano pygpu && \
+    git clone git://github.com/fchollet/keras.git /src && pip install -e /src[tests] && \
     pip install git+git://github.com/fchollet/keras.git && \
     conda clean -yt
 

docker/theanorc

@@ -1,5 +1,5 @@
 [global]
 floatX = float32
 optimizer=None
-device = gpu
+device = cuda
 

docs/autogen.py

@@ -8,9 +8,7 @@ Index
 - Getting started
     Getting started with the sequential model
     Getting started with the functional api
-    Examples
     FAQ
-    Installation guide
 
 - Models
     About Keras models
@@ -26,18 +24,23 @@ Index
         explain common layer functions: get_weights, set_weights, get_config
         explain input_shape
         explain usage on non-Keras tensors
-    Core layers
-    Convolutional
-    Recurrent
-    Embeddings
-    Normalization
-    Advanced activations
-    Noise
+    Core Layers
+    Convolutional Layers
+    Pooling Layers
+    Locally-connected Layers
+    Recurrent Layers
+    Embedding Layers
+    Merge Layers
+    Advanced Activations Layers
+    Normalization Layers
+    Noise Layers
+    Layer Wrappers
+    Writing your own Keras layers
 
 - Preprocessing
-    Image preprocessing
-    Text preprocessing
-    Sequence preprocessing
+    Sequence Preprocessing
+    Text Preprocessing
+    Image Preprocessing
 
 Losses
 Metrics
@@ -45,12 +48,15 @@ Optimizers
 Activations
 Callbacks
 Datasets
+Applications
 Backend
-Initializations
+Initializers
 Regularizers
 Constraints
 Visualization
 Scikit-learn API
+Utils
+Contributing
 
 '''
 from __future__ import print_function
@@ -114,14 +120,13 @@ PAGES = [
             models.Sequential.fit,
             models.Sequential.evaluate,
             models.Sequential.predict,
-            models.Sequential.predict_classes,
-            models.Sequential.predict_proba,
             models.Sequential.train_on_batch,
             models.Sequential.test_on_batch,
             models.Sequential.predict_on_batch,
             models.Sequential.fit_generator,
             models.Sequential.evaluate_generator,
             models.Sequential.predict_generator,
+            models.Sequential.get_layer,
         ],
     },
     {
@@ -315,7 +320,9 @@ def get_classes_ancestors(classes):
 
 
 def get_function_signature(function, method=True):
-    signature = inspect.getargspec(function)
+    signature = getattr(function, '_legacy_support_signature', None)
+    if signature is None:
+        signature = inspect.getargspec(function)
     defaults = signature.defaults
     if method:
         args = signature.args[1:]
@@ -380,7 +387,7 @@ def process_class_docstring(docstring):
                        r'\n    __\1__\n\n',
                        docstring)
 
-    docstring = re.sub(r'    ([^\s\\]+):(.*)\n',
+    docstring = re.sub(r'    ([^\s\\\(]+):(.*)\n',
                        r'    - __\1__:\2\n',
                        docstring)
 
@@ -398,7 +405,7 @@ def process_function_docstring(docstring):
                        r'\n        __\1__\n\n',
                        docstring)
 
-    docstring = re.sub(r'    ([^\s\\]+):(.*)\n',
+    docstring = re.sub(r'    ([^\s\\\(]+):(.*)\n',
                        r'    - __\1__:\2\n',
                        docstring)
 
@@ -507,3 +514,5 @@ for page_data in PAGES:
     if not os.path.exists(subdir):
         os.makedirs(subdir)
     open(path, 'w').write(mkdown)
+
+shutil.copyfile('../CONTRIBUTING.md', 'sources/contributing.md')

docs/mkdocs.yml

@@ -1,6 +1,6 @@
 site_name: Keras Documentation
 theme: readthedocs
-theme_dir: theme
+#theme_dir: theme
 docs_dir: sources
 repo_url: http://github.com/fchollet/keras
 site_url: http://keras.io/
@@ -51,3 +51,4 @@ pages:
 - Visualization: visualization.md
 - Scikit-learn API: scikit-learn-api.md
 - Utils: utils.md
+- Contributing: contributing.md

docs/templates/applications.md

@@ -15,7 +15,7 @@ Weights are downloaded automatically when instantiating a model. They are stored
 - [ResNet50](#resnet50)
 - [InceptionV3](#inceptionv3)
 
-All of these architectures (except Xception) are compatible with both TensorFlow and Theano, and upon instantiation the models will be built according to the image data format set in your Keras configuration file at `~/.keras/keras.json`. For instance, if you have set `image_data_format=tf`, then any model loaded from this repository will get built according to the TensorFlow data format convention, "Width-Height-Depth".
+All of these architectures (except Xception) are compatible with both TensorFlow and Theano, and upon instantiation the models will be built according to the image data format set in your Keras configuration file at `~/.keras/keras.json`. For instance, if you have set `image_data_format=channels_last`, then any model loaded from this repository will get built according to the TensorFlow data format convention, "Width-Height-Depth".
 
 The Xception model is only available for TensorFlow, due to its reliance on `SeparableConvolution` layers.
 
@@ -75,7 +75,7 @@ from keras.models import Model
 import numpy as np
 
 base_model = VGG19(weights='imagenet')
-model = Model(input=base_model.input, output=base_model.get_layer('block4_pool').output)
+model = Model(inputs=base_model.input, outputs=base_model.get_layer('block4_pool').output)
 
 img_path = 'elephant.jpg'
 img = image.load_img(img_path, target_size=(224, 224))
@@ -107,7 +107,7 @@ x = Dense(1024, activation='relu')(x)
 predictions = Dense(200, activation='softmax')(x)
 
 # this is the model we will train
-model = Model(input=base_model.input, output=predictions)
+model = Model(inputs=base_model.input, outputs=predictions)
 
 # first: train only the top layers (which were randomly initialized)
 # i.e. freeze all convolutional InceptionV3 layers
@@ -253,7 +253,7 @@ The default input size for this model is 224x224.
 - input_shape: optional shape tuple, only to be specified
     if `include_top` is False (otherwise the input shape
     has to be `(224, 224, 3)` (with `channels_last` data format)
-    or `(3, 224, 244)` (with `channels_first` data format).
+    or `(3, 224, 224)` (with `channels_first` data format).
     It should have exactly 3 inputs channels,
     and width and height should be no smaller than 48.
     E.g. `(200, 200, 3)` would be one valid value.
@@ -309,7 +309,7 @@ The default input size for this model is 224x224.
 - input_shape: optional shape tuple, only to be specified
     if `include_top` is False (otherwise the input shape
     has to be `(224, 224, 3)` (with `channels_last` data format)
-    or `(3, 224, 244)` (with `channels_first` data format).
+    or `(3, 224, 224)` (with `channels_first` data format).
     It should have exactly 3 inputs channels,
     and width and height should be no smaller than 48.
     E.g. `(200, 200, 3)` would be one valid value.
@@ -367,7 +367,7 @@ The default input size for this model is 224x224.
 - input_shape: optional shape tuple, only to be specified
     if `include_top` is False (otherwise the input shape
     has to be `(224, 224, 3)` (with `channels_last` data format)
-    or `(3, 224, 244)` (with `channels_first` data format).
+    or `(3, 224, 224)` (with `channels_first` data format).
     It should have exactly 3 inputs channels,
     and width and height should be no smaller than 197.
     E.g. `(200, 200, 3)` would be one valid value.

docs/templates/backend.md

@@ -17,10 +17,12 @@ In the future, we are likely to add more backend options. Go ask Microsoft about
 
 If you have run Keras at least once, you will find the Keras configuration file at:
 
-`~/.keras/keras.json`
+`$HOME/.keras/keras.json`
 
 If it isn't there, you can create it.
 
+**NOTE for Windows Users:** Please change `$HOME` with `%USERPROFILE%`.
+
 The default configuration file looks like this:
 
 ```
@@ -56,7 +58,7 @@ Using TensorFlow backend.
 }
 ```
 
-You can change these settings by editing `~/.keras/keras.json`. 
+You can change these settings by editing `$HOME/.keras/keras.json`. 
 
 * `image_data_format`: string, either `"channels_last"` or `"channels_first"`. It specifies which data format convention Keras will follow. (`keras.backend.image_data_format()` returns it.)
   - For 2D data (e.g. image), `"channels_last"` assumes `(rows, cols, channels)` while `"channels_first"` assumes `(channels, rows, cols)`. 
@@ -69,14 +71,14 @@ You can change these settings by editing `~/.keras/keras.json`.
 
 ## Using the abstract Keras backend to write new code
 
-If you want the Keras modules you write to be compatible with both Theano and TensorFlow, you have to write them via the abstract Keras backend API. Here's an intro.
+If you want the Keras modules you write to be compatible with both Theano (`th`) and TensorFlow (`tf`), you have to write them via the abstract Keras backend API. Here's an intro.
 
 You can import the backend module via:
 ```python
 from keras import backend as K
 ```
 
-The code below instantiates an input placeholder. It's equivalent to `tf.placeholder()` or `T.matrix()`, `T.tensor3()`, etc.
+The code below instantiates an input placeholder. It's equivalent to `tf.placeholder()` or `th.tensor.matrix()`, `th.tensor.tensor3()`, etc.
 
 ```python
 input = K.placeholder(shape=(2, 4, 5))
@@ -86,9 +88,10 @@ input = K.placeholder(shape=(None, 4, 5))
 input = K.placeholder(ndim=3)
 ```
 
-The code below instantiates a shared variable. It's equivalent to `tf.variable()` or `theano.shared()`.
+The code below instantiates a shared variable. It's equivalent to `tf.Variable()` or `th.shared()`.
 
 ```python
+import numpy as np
 val = np.random.random((3, 4, 5))
 var = K.variable(value=val)
 
@@ -101,11 +104,16 @@ var = K.ones(shape=(3, 4, 5))
 Most tensor operations you will need can be done as you would in TensorFlow or Theano:
 
 ```python
+# Initializing Tensors with Random Numbers
+b = K.random_uniform_variable(shape=(3, 4)). # Uniform distribution
+c = K.random_normal_variable(shape=(3, 4)). # Gaussian distribution
+d = K.random_normal_variable(shape=(3, 4)).
+# Tensor Arithmetics
 a = b + c * K.abs(d)
 c = K.dot(a, K.transpose(b))
-a = K.sum(b, axis=2)
+a = K.sum(b, axis=1)
 a = K.softmax(b)
-a = concatenate([b, c], axis=-1)
+a = K.concatenate([b, c], axis=-1)
 # etc...
 ```
 

docs/templates/callbacks.md

@@ -36,14 +36,14 @@ class LossHistory(keras.callbacks.Callback):
         self.losses.append(logs.get('loss'))
 
 model = Sequential()
-model.add(Dense(10, input_dim=784, init='uniform'))
+model.add(Dense(10, input_dim=784, kernel_initializer='uniform'))
 model.add(Activation('softmax'))
 model.compile(loss='categorical_crossentropy', optimizer='rmsprop')
 
 history = LossHistory()
-model.fit(X_train, Y_train, batch_size=128, epochs=20, verbose=0, callbacks=[history])
+model.fit(x_train, y_train, batch_size=128, epochs=20, verbose=0, callbacks=[history])
 
-print history.losses
+print(history.losses)
 # outputs
 '''
 [0.66047596406559383, 0.3547245744908703, ..., 0.25953155204159617, 0.25901699725311789]
@@ -58,15 +58,13 @@ print history.losses
 from keras.callbacks import ModelCheckpoint
 
 model = Sequential()
-model.add(Dense(10, input_dim=784, init='uniform'))
+model.add(Dense(10, input_dim=784, kernel_initializer='uniform'))
 model.add(Activation('softmax'))
 model.compile(loss='categorical_crossentropy', optimizer='rmsprop')
 
 '''
 saves the model weights after each epoch if the validation loss decreased
 '''
-checkpointer = ModelCheckpoint(filepath="/tmp/weights.hdf5", verbose=1, save_best_only=True)
-model.fit(X_train, Y_train, batch_size=128, epochs=20, verbose=0, validation_data=(X_test, Y_test), callbacks=[checkpointer])
-
+checkpointer = ModelCheckpoint(filepath='/tmp/weights.hdf5', verbose=1, save_best_only=True)
+model.fit(x_train, y_train, batch_size=128, epochs=20, verbose=0, validation_data=(X_test, Y_test), callbacks=[checkpointer])
 ```
-

docs/templates/constraints.md

@@ -2,7 +2,7 @@
 
 Functions from the `constraints` module allow setting constraints (eg. non-negativity) on network parameters during optimization.
 
-The penalties are applied on a per-layer basis. The exact API will depend on the layer, but the layers `Dense`, `Convolution1D`, `Convolution2D` and `Convolution3D` have a unified API.
+The penalties are applied on a per-layer basis. The exact API will depend on the layer, but the layers `Dense`, `Conv1D`, `Conv2D` and `Conv3D` have a unified API.
 
 These layers expose 2 keyword arguments:
 
@@ -17,6 +17,6 @@ model.add(Dense(64, kernel_constraint=max_norm(2.)))
 
 ## Available constraints
 
-- __max_norm__(m=2): maximum-norm constraint
+- __max_norm__(max_value=2, axis=0): maximum-norm constraint
 - __non_neg__(): non-negativity constraint
 - __unit_norm__(): unit-norm constraint, enforces the matrix to have unit norm along the last axis

docs/templates/datasets.md

@@ -55,7 +55,7 @@ As a convention, "0" does not stand for a specific word, but instead is used to
 ```python
 from keras.datasets import imdb
 
-(x_train, y_train), (x_test, y_test) = imdb.load_data(path="imdb_full.pkl",
+(x_train, y_train), (x_test, y_test) = imdb.load_data(path="imdb.npz",
                                                       num_words=None,
                                                       skip_top=0,
                                                       maxlen=None,
@@ -72,13 +72,13 @@ from keras.datasets import imdb
 - __Arguments:__
 
     - __path__: if you do not have the data locally (at `'~/.keras/datasets/' + path`), it will be downloaded to this location.
-    - __num_words__: integer or None. Top most frequent words to consider. Any less frequent word will appear as 0 in the sequence data.
-    - __skip_top__: integer. Top most frequent words to ignore (they will appear as 0s in the sequence data).
+    - __num_words__: integer or None. Top most frequent words to consider. Any less frequent word will appear as `oov_char` value in the sequence data.
+    - __skip_top__: integer. Top most frequent words to ignore (they will appear as `oov_char` value in the sequence data).
     - __maxlen__: int. Maximum sequence length. Any longer sequence will be truncated.
     - __seed__: int. Seed for reproducible data shuffling.
-    - __start_char__: char. The start of a sequence will be marked with this character.
+    - __start_char__: int. The start of a sequence will be marked with this character.
         Set to 1 because 0 is usually the padding character.
-    - __oov_char__: char. words that were cut out because of the `num_words`
+    - __oov_char__: int. words that were cut out because of the `num_words`
         or `skip_top` limit will be replaced with this character.
     - __index_from__: int. Index actual words with this index and higher.
 
@@ -94,7 +94,7 @@ Dataset of 11,228 newswires from Reuters, labeled over 46 topics. As with the IM
 ```python
 from keras.datasets import reuters
 
-(x_train, y_train), (x_test, y_test) = reuters.load_data(path="reuters.pkl",
+(x_train, y_train), (x_test, y_test) = reuters.load_data(path="reuters.npz",
                                                          num_words=None,
                                                          skip_top=0,
                                                          maxlen=None,
@@ -107,12 +107,12 @@ from keras.datasets import reuters
 
 The specifications are the same as that of the IMDB dataset, with the addition of:
 
-    - __test_split__: float. Fraction of the dataset to be used as test data.
+- __test_split__: float. Fraction of the dataset to be used as test data.
 
 This dataset also makes available the word index used for encoding the sequences:
 
 ```python
-word_index = reuters.get_word_index(path="reuters_word_index.pkl")
+word_index = reuters.get_word_index(path="reuters_word_index.json")
 ```
 
 - __Returns:__ A dictionary where key are words (str) and values are indexes (integer). eg. `word_index["giraffe"]` might return `1234`. 

docs/templates/getting-started/faq.md

@@ -2,7 +2,7 @@
 
 - [How should I cite Keras?](#how-should-i-cite-keras)
 - [How can I run Keras on GPU?](#how-can-i-run-keras-on-gpu)
-- [What does \["sample", "batch", "epoch"\] mean?](#what-does-sample-batch-epoch-mean)
+- [What does "sample", "batch", "epoch" mean?](#what-does-sample-batch-epoch-mean)
 - [How can I save a Keras model?](#how-can-i-save-a-keras-model)
 - [Why is the training loss much higher than the testing loss?](#why-is-the-training-loss-much-higher-than-the-testing-loss)
 - [How can I obtain the output of an intermediate layer?](#how-can-i-obtain-the-output-of-an-intermediate-layer)
@@ -15,6 +15,8 @@
 - [How can I use stateful RNNs?](#how-can-i-use-stateful-rnns)
 - [How can I remove a layer from a Sequential model?](#how-can-i-remove-a-layer-from-a-sequential-model)
 - [How can I use pre-trained models in Keras?](#how-can-i-use-pre-trained-models-in-keras)
+- [How can I use HDF5 inputs with Keras?](#how-can-i-use-hdf5-inputs-with-keras)
+- [Where is the Keras configuration filed stored?](#where-is-the-keras-configuration-filed-stored)
 
 ---
 
@@ -25,7 +27,7 @@ Please cite Keras in your publications if it helps your research. Here is an exa
 ```
 @misc{chollet2015keras,
   title={Keras},
-  author={Chollet, Fran\c{c}ois},
+  author={Chollet, Fran\c{c}ois and others},
   year={2015},
   publisher={GitHub},
   howpublished={\url{https://github.com/fchollet/keras}},
@@ -58,7 +60,7 @@ theano.config.floatX = 'float32'
 
 ---
 
-### What does \["sample", "batch", "epoch"\] mean?
+### What does "sample", "batch", "epoch" mean?
 
 Below are some common definitions that are necessary to know and understand to correctly utilize Keras:
 
@@ -151,16 +153,16 @@ For example:
 """
 Assume original model looks like this:
     model = Sequential()
-    model.add(Dense(2, input_dim=3, name="dense_1"))
-    model.add(Dense(3, name="dense_2"))
+    model.add(Dense(2, input_dim=3, name='dense_1'))
+    model.add(Dense(3, name='dense_2'))
     ...
     model.save_weights(fname)
 """
 
 # new model
 model = Sequential()
-model.add(Dense(2, input_dim=3, name="dense_1"))  # will be loaded
-model.add(Dense(10, name="new_dense"))  # will not be loaded
+model.add(Dense(2, input_dim=3, name='dense_1'))  # will be loaded
+model.add(Dense(10, name='new_dense'))  # will not be loaded
 
 # load weights from first model; will only affect the first layer, dense_1.
 model.load_weights(fname, by_name=True)
@@ -199,7 +201,7 @@ from keras import backend as K
 # with a Sequential model
 get_3rd_layer_output = K.function([model.layers[0].input],
                                   [model.layers[3].output])
-layer_output = get_3rd_layer_output([X])[0]
+layer_output = get_3rd_layer_output([x])[0]
 ```
 
 Similarly, you could build a Theano and TensorFlow function directly.
@@ -212,19 +214,19 @@ get_3rd_layer_output = K.function([model.layers[0].input, K.learning_phase()],
                                   [model.layers[3].output])
 
 # output in test mode = 0
-layer_output = get_3rd_layer_output([X, 0])[0]
+layer_output = get_3rd_layer_output([x, 0])[0]
 
 # output in train mode = 1
-layer_output = get_3rd_layer_output([X, 1])[0]
+layer_output = get_3rd_layer_output([x, 1])[0]
 ```
 
 ---
 
 ### How can I use Keras with datasets that don't fit in memory?
 
-You can do batch training using `model.train_on_batch(X, y)` and `model.test_on_batch(X, y)`. See the [models documentation](/models/sequential).
+You can do batch training using `model.train_on_batch(x, y)` and `model.test_on_batch(x, y)`. See the [models documentation](/models/sequential).
 
-Alternatively, you can write a generator that yields batches of training data and use the method `model.fit_generator(data_generator, samples_per_epoch, epochs)`.
+Alternatively, you can write a generator that yields batches of training data and use the method `model.fit_generator(data_generator, steps_per_epoch, epochs)`.
 
 You can see batch training in action in our [CIFAR10 example](https://github.com/fchollet/keras/blob/master/examples/cifar10_cnn.py).
 
@@ -237,7 +239,7 @@ You can use an `EarlyStopping` callback:
 ```python
 from keras.callbacks import EarlyStopping
 early_stopping = EarlyStopping(monitor='val_loss', patience=2)
-model.fit(X, y, validation_split=0.2, callbacks=[early_stopping])
+model.fit(x, y, validation_split=0.2, callbacks=[early_stopping])
 ```
 
 Find out more in the [callbacks documentation](/callbacks).
@@ -266,7 +268,7 @@ Validation data is never shuffled.
 The `model.fit` method returns an `History` callback, which has a `history` attribute containing the lists of successive losses and other metrics.
 
 ```python
-hist = model.fit(X, y, validation_split=0.2)
+hist = model.fit(x, y, validation_split=0.2)
 print(hist.history)
 ```
 
@@ -313,12 +315,13 @@ Making a RNN stateful means that the states for the samples of each batch will b
 When using stateful RNNs, it is therefore assumed that:
 
 - all batches have the same number of samples
-- If `X1` and `X2` are successive batches of samples, then `X2[i]` is the follow-up sequence to `X1[i]`, for every `i`.
+- If `x1` and `x2` are successive batches of samples, then `x2[i]` is the follow-up sequence to `x1[i]`, for every `i`.
 
 To use statefulness in RNNs, you need to:
 
 - explicitly specify the batch size you are using, by passing a `batch_size` argument to the first layer in your model. E.g. `batch_size=32` for a 32-samples batch of sequences of 10 timesteps with 16 features per timestep.
 - set `stateful=True` in your RNN layer(s).
+- specify `shuffle=False` when calling fit().
 
 To reset the states accumulated:
 
@@ -329,7 +332,7 @@ Example:
 
 ```python
 
-X  # this is our input data, of shape (32, 21, 16)
+x  # this is our input data, of shape (32, 21, 16)
 # we will feed it to our model in sequences of length 10
 
 model = Sequential()
@@ -339,10 +342,10 @@ model.add(Dense(16, activation='softmax'))
 model.compile(optimizer='rmsprop', loss='categorical_crossentropy')
 
 # we train the network to predict the 11th timestep given the first 10:
-model.train_on_batch(X[:, :10, :], np.reshape(X[:, 10, :], (32, 16)))
+model.train_on_batch(x[:, :10, :], np.reshape(x[:, 10, :], (32, 16)))
 
 # the state of the network has changed. We can feed the follow-up sequences:
-model.train_on_batch(X[:, 10:20, :], np.reshape(X[:, 20, :], (32, 16)))
+model.train_on_batch(x[:, 10:20, :], np.reshape(x[:, 20, :], (32, 16)))
 
 # let's reset the states of the LSTM layer:
 model.reset_states()
@@ -403,3 +406,51 @@ The VGG16 model is also the basis for several Keras example scripts:
 - [Style transfer](https://github.com/fchollet/keras/blob/master/examples/neural_style_transfer.py)
 - [Feature visualization](https://github.com/fchollet/keras/blob/master/examples/conv_filter_visualization.py)
 - [Deep dream](https://github.com/fchollet/keras/blob/master/examples/deep_dream.py)
+
+---
+
+### How can I use HDF5 inputs with Keras?
+
+You can use the `HDF5Matrix` class from `keras.utils.io_utils`. See [the HDF5Matrix documentation](/utils/#hdf5matrix) for details.
+
+You can also directly use a HDF5 dataset:
+
+```python
+import h5py
+with h5py.File('input/file.hdf5', 'r') as f:
+    x_data = f['x_data']
+    model.predict(x_data)
+```
+
+---
+
+### Where is the Keras configuration filed stored?
+
+The default directory where all Keras data is stored is:
+
+```bash
+$HOME/.keras/
+```
+
+Note that Windows users should replace `$HOME` with `%USERPROFILE%`.
+In case Keras cannot create the above directory (e.g. due to permission issues), `/tmp/.keras/` is used as a backup.
+
+The Keras configuration file is a JSON file stored at `$HOME/.keras/keras.json`. The default configuration file looks like this:
+
+```
+{
+    "image_data_format": "channels_last",
+    "epsilon": 1e-07,
+    "floatx": "float32",
+    "backend": "tensorflow"
+}
+```
+
+It contains the following fields:
+
+- The image data format to be used as default by image processing layers and utilities (either `channels_last` or `channels_first`).
+- The `epsilon` numerical fuzz factor to be used to prevent division by zero in some operations.
+- The default float data type.
+- The default backend. See the [backend documentation](/backend).
+
+Likewise, cached dataset files, such as those downloaded with [`get_file()`](/utils/#get_file), are stored by default in `$HOME/.keras/datasets/`.

docs/templates/getting-started/sequential-model-guide.md

@@ -9,7 +9,7 @@ from keras.models import Sequential
 from keras.layers import Dense, Activation
 
 model = Sequential([
-    Dense(32, input_dim=784),
+    Dense(32, input_shape=(784,)),
     Activation('relu'),
     Dense(10),
     Activation('softmax'),
@@ -98,7 +98,7 @@ model.compile(optimizer='rmsprop',
 
 # Generate dummy data
 import numpy as np
-data = np.random.random((1000, 784))
+data = np.random.random((1000, 100))
 labels = np.random.randint(2, size=(1000, 1))
 
 # Train the model, iterating on the data in batches of 32 samples
@@ -117,14 +117,14 @@ model.compile(optimizer='rmsprop',
 
 # Generate dummy data
 import numpy as np
-data = np.random.random((1000, 784))
-labels = np.random.randint(10, size=(1000, 10))
+data = np.random.random((1000, 100))
+labels = np.random.randint(10, size=(1000, 1))
 
 # Convert labels to categorical one-hot encoding
-binary_labels = keras.utils.to_categorical(labels, num_classes=10)
+one_hot_labels = keras.utils.to_categorical(labels, num_classes=10)
 
 # Train the model, iterating on the data in batches of 32 samples
-model.fit(data, binary_labels, epochs=10, batch_size=32)
+model.fit(data, one_hot_labels, epochs=10, batch_size=32)
 ```
 
 ----
@@ -152,6 +152,13 @@ from keras.models import Sequential
 from keras.layers import Dense, Dropout, Activation
 from keras.optimizers import SGD
 
+# Generate dummy data
+import numpy as np
+x_train = np.random.random((1000, 20))
+y_train = keras.utils.to_categorical(np.random.randint(10, size=(1000, 1)), num_classes=10)
+x_test = np.random.random((100, 20))
+y_test = keras.utils.to_categorical(np.random.randint(10, size=(100, 1)), num_classes=10)
+
 model = Sequential()
 # Dense(64) is a fully-connected layer with 64 hidden units.
 # in the first layer, you must specify the expected input data shape:
@@ -177,6 +184,16 @@ score = model.evaluate(x_test, y_test, batch_size=128)
 ### MLP for binary classification:
 
 ```python
+import numpy as np
+from keras.models import Sequential
+from keras.layers import Dense, Dropout
+
+# Generate dummy data
+x_train = np.random.random((1000, 20))
+y_train = np.random.randint(2, size=(1000, 1))
+x_test = np.random.random((100, 20))
+y_test = np.random.randint(2, size=(100, 1))
+
 model = Sequential()
 model.add(Dense(64, input_dim=20, activation='relu'))
 model.add(Dropout(0.5))
@@ -187,21 +204,34 @@ model.add(Dense(1, activation='sigmoid'))
 model.compile(loss='binary_crossentropy',
               optimizer='rmsprop',
               metrics=['accuracy'])
+
+model.fit(x_train, y_train,
+          epochs=20,
+          batch_size=128)
+score = model.evaluate(x_test, y_test, batch_size=128)
 ```
 
 
 ### VGG-like convnet:
 
 ```python
+import numpy as np
+import keras
 from keras.models import Sequential
 from keras.layers import Dense, Dropout, Flatten
 from keras.layers import Conv2D, MaxPooling2D
 from keras.optimizers import SGD
 
+# Generate dummy data
+x_train = np.random.random((100, 100, 100, 3))
+y_train = keras.utils.to_categorical(np.random.randint(10, size=(100, 1)), num_classes=10)
+x_test = np.random.random((20, 100, 100, 3))
+y_test = keras.utils.to_categorical(np.random.randint(10, size=(20, 1)), num_classes=10)
+
 model = Sequential()
-# input: 100x100 images with 3 channels -> (3, 100, 100) tensors.
+# input: 100x100 images with 3 channels -> (100, 100, 3) tensors.
 # this applies 32 convolution filters of size 3x3 each.
-model.add(Conv2D(32, (3, 3), activation='relu', input_shape=(3, 100, 100)))
+model.add(Conv2D(32, (3, 3), activation='relu', input_shape=(100, 100, 3)))
 model.add(Conv2D(32, (3, 3), activation='relu'))
 model.add(MaxPooling2D(pool_size=(2, 2)))
 model.add(Dropout(0.25))
@@ -220,6 +250,7 @@ sgd = SGD(lr=0.01, decay=1e-6, momentum=0.9, nesterov=True)
 model.compile(loss='categorical_crossentropy', optimizer=sgd)
 
 model.fit(x_train, y_train, batch_size=32, epochs=10)
+score = model.evaluate(x_test, y_test, batch_size=32)
 ```
 
 
@@ -251,12 +282,12 @@ score = model.evaluate(x_test, y_test, batch_size=16)
 from keras.models import Sequential
 from keras.layers import Dense, Dropout
 from keras.layers import Embedding
-from keras.layers import Conv1D, GlobalAveragePooling1D
+from keras.layers import Conv1D, GlobalAveragePooling1D, MaxPooling1D
 
 model = Sequential()
 model.add(Conv1D(64, 3, activation='relu', input_shape=(seq_length, 100)))
 model.add(Conv1D(64, 3, activation='relu'))
-model.add(MaxPooling1D((3, 3)))
+model.add(MaxPooling1D(3))
 model.add(Conv1D(128, 3, activation='relu'))
 model.add(Conv1D(128, 3, activation='relu'))
 model.add(GlobalAveragePooling1D())
@@ -323,7 +354,7 @@ A stateful recurrent model is one for which the internal states (memories) obtai
 of samples are reused as initial states for the samples of the next batch. This allows to process longer sequences
 while keeping computational complexity manageable.
 
-[You can read more about stateful RNNs in the FAQ.](/faq/#how-can-i-use-stateful-rnns)
+[You can read more about stateful RNNs in the FAQ.](/getting-started/faq/#how-can-i-use-stateful-rnns)
 
 ```python
 from keras.models import Sequential
@@ -358,6 +389,6 @@ x_val = np.random.random((batch_size * 3, timesteps, data_dim))
 y_val = np.random.random((batch_size * 3, num_classes))
 
 model.fit(x_train, y_train,
-          batch_size=batch_size, epochs=5,
+          batch_size=batch_size, epochs=5, shuffle=False,
           validation_data=(x_val, y_val))
 ```

docs/templates/initializers.md

@@ -39,5 +39,5 @@ from keras import backend as K
 def my_init(shape, dtype=None):
     return K.random_normal(shape, dtype=dtype)
 
-model.add(Dense(64, init=my_init))
+model.add(Dense(64, kernel_initializer=my_init))
 ```

docs/templates/metrics.md

@@ -17,7 +17,7 @@ model.compile(loss='mean_squared_error',
               metrics=[metrics.mae, metrics.categorical_accuracy])
 ```
 
-A metric function is similar to an [objective function](/objectives), except that the results from evaluating a metric are not used when training the model.
+A metric function is similar to an [loss function](/losses), except that the results from evaluating a metric are not used when training the model.
 
 You can either pass the name of an existing metric, or pass a Theano/TensorFlow symbolic function (see [Custom metrics](#custom-metrics)).
 

docs/templates/preprocessing/image.md

@@ -7,6 +7,7 @@ keras.preprocessing.image.ImageDataGenerator(featurewise_center=False,
     featurewise_std_normalization=False,
     samplewise_std_normalization=False,
     zca_whitening=False,
+    zca_epsilon=1e-6,
     rotation_range=0.,
     width_shift_range=0.,
     height_shift_range=0.,
@@ -18,6 +19,7 @@ keras.preprocessing.image.ImageDataGenerator(featurewise_center=False,
     horizontal_flip=False,
     vertical_flip=False,
     rescale=None,
+    preprocessing_function=None,
     data_format=K.image_data_format())
 ```
 
@@ -28,6 +30,7 @@ Generate batches of tensor image data with real-time data augmentation. The data
     - __samplewise_center__: Boolean. Set each sample mean to 0.
     - __featurewise_std_normalization__: Boolean. Divide inputs by std of the dataset, feature-wise.
     - __samplewise_std_normalization__: Boolean. Divide each input by its std.
+    - __zca_epsilon__: epsilon for ZCA whitening. Default is 1e-6.
     - __zca_whitening__: Boolean. Apply ZCA whitening.
     - __rotation_range__: Int. Degree range for random rotations.
     - __width_shift_range__: Float (fraction of total width). Range for random horizontal shifts.
@@ -42,6 +45,11 @@ Generate batches of tensor image data with real-time data augmentation. The data
     - __rescale__: rescaling factor. Defaults to None. If None or 0, no rescaling is applied,
             otherwise we multiply the data by the value provided (before applying
             any other transformation).
+    - __preprocessing_function__: function that will be implied on each input.
+            The function will run before any other modification on it.
+            The function should take one argument:
+            one image (Numpy tensor with rank 3),
+            and should output a Numpy tensor with the same shape.
     - _data_format_: One of {"channels_first", "channels_last"}.
         "channels_last" mode means that the images should have shape `(samples, height, width, channels)`,
         "channels_first" mode means that the images should have shape `(samples, channels, height, width)`.
@@ -50,19 +58,19 @@ Generate batches of tensor image data with real-time data augmentation. The data
         If you never set it, then it will be "channels_last".
 
 - __Methods__:
-    - __fit(X)__: Compute the internal data stats related to the data-dependent transformations, based on an array of sample data.
+    - __fit(x)__: Compute the internal data stats related to the data-dependent transformations, based on an array of sample data.
         Only required if featurewise_center or featurewise_std_normalization or zca_whitening.
         - __Arguments__:
-            - __X__: sample data. Should have rank 4.
+            - __x__: sample data. Should have rank 4.
                 In case of grayscale data,
                 the channels axis should have value 1, and in case
                 of RGB data, it should have value 3.
             - __augment__: Boolean (default: False). Whether to fit on randomly augmented samples.
             - __rounds__: int (default: 1). If augment, how many augmentation passes over the data to use.
             - __seed__: int (default: None). Random seed.
-    - __flow(X, y)__: Takes numpy data & label arrays, and generates batches of augmented/normalized data. Yields batches indefinitely, in an infinite loop.
+    - __flow(x, y)__: Takes numpy data & label arrays, and generates batches of augmented/normalized data. Yields batches indefinitely, in an infinite loop.
         - __Arguments__:
-            - __X__: data. Should have rank 4.
+            - __x__: data. Should have rank 4.
                 In case of grayscale data,
                 the channels axis should have value 1, and in case
                 of RGB data, it should have value 3.
@@ -72,7 +80,7 @@ Generate batches of tensor image data with real-time data augmentation. The data
             - __seed__: int (default: None).
             - __save_to_dir__: None or str (default: None). This allows you to optimally specify a directory to which to save the augmented pictures being generated (useful for visualizing what you are doing).
             - __save_prefix__: str (default: `''`). Prefix to use for filenames of saved pictures (only relevant if `save_to_dir` is set).
-            - __save_format__: one of "png", "jpeg" (only relevant if `save_to_dir` is set). Default: "jpeg".
+            - __save_format__: one of "png", "jpeg" (only relevant if `save_to_dir` is set). Default: "png".
         - __yields__: Tuples of `(x, y)` where `x` is a numpy array of image data and `y` is a numpy array of corresponding labels.
             The generator loops indefinitely.
     - __flow_from_directory(directory)__: Takes the path to a directory, and generates batches of augmented/normalized data. Yields batches indefinitely, in an infinite loop.
@@ -82,25 +90,25 @@ Generate batches of tensor image data with real-time data augmentation. The data
                 See [this script](https://gist.github.com/fchollet/0830affa1f7f19fd47b06d4cf89ed44d) for more details.
             - __target_size__: tuple of integers, default: `(256, 256)`. The dimensions to which all images found will be resized.
             - __color_mode__: one of "grayscale", "rbg". Default: "rgb". Whether the images will be converted to have 1 or 3 color channels.
-            - __classes__: optional list of class subdirectories (e.g. `['dogs', 'cats']`). Default: None. If not provided, the list of classes will be automatically inferred (and the order of the classes, which will map to the label indices, will be alphanumeric).
-            - __class_mode__: one of "categorical", "binary", "sparse" or None. Default: "categorical". Determines the type of label arrays that are returned: "categorical" will be 2D one-hot encoded labels, "binary" will be 1D binary labels, "sparse" will be 1D integer labels. If None, no labels are returned (the generator will only yield batches of image data, which is useful to use `model.predict_generator()`, `model.evaluate_generator()`, etc.).
+            - __classes__: optional list of class subdirectories (e.g. `['dogs', 'cats']`). Default: None. If not provided, the list of classes will be automatically inferred from the subdirectory names/structure under `directory`, where each subdirectory will be treated as a different class (and the order of the classes, which will map to the label indices, will be alphanumeric). The dictionary containing the mapping from class names to class indices can be obtained via the attribute `class_indices`.
+            - __class_mode__: one of "categorical", "binary", "sparse" or None. Default: "categorical". Determines the type of label arrays that are returned: "categorical" will be 2D one-hot encoded labels, "binary" will be 1D binary labels, "sparse" will be 1D integer labels. If None, no labels are returned (the generator will only yield batches of image data, which is useful to use `model.predict_generator()`, `model.evaluate_generator()`, etc.). Please note that in case of class_mode None, the data still needs to reside in a subdirectory of `directory` for it to work correctly.
             - __batch_size__: size of the batches of data (default: 32).
             - __shuffle__: whether to shuffle the data (default: True)
             - __seed__: optional random seed for shuffling and transformations.
             - __save_to_dir__: None or str (default: None). This allows you to optimally specify a directory to which to save the augmented pictures being generated (useful for visualizing what you are doing).
             - __save_prefix__: str. Prefix to use for filenames of saved pictures (only relevant if `save_to_dir` is set).
-            - __save_format__: one of "png", "jpeg" (only relevant if `save_to_dir` is set). Default: "jpeg".
+            - __save_format__: one of "png", "jpeg" (only relevant if `save_to_dir` is set). Default: "png".
             - __follow_links__: whether to follow symlinks inside class subdirectories (default: False).
 
 
 - __Examples__:
 
-Example of using `.flow(X, y)`:
+Example of using `.flow(x, y)`:
 
 ```python
-(X_train, y_train), (X_test, y_test) = cifar10.load_data()
-Y_train = np_utils.to_categorical(y_train, num_classes)
-Y_test = np_utils.to_categorical(y_test, num_classes)
+(x_train, y_train), (x_test, y_test) = cifar10.load_data()
+y_train = np_utils.to_categorical(y_train, num_classes)
+y_test = np_utils.to_categorical(y_test, num_classes)
 
 datagen = ImageDataGenerator(
     featurewise_center=True,
@@ -112,20 +120,20 @@ datagen = ImageDataGenerator(
 
 # compute quantities required for featurewise normalization
 # (std, mean, and principal components if ZCA whitening is applied)
-datagen.fit(X_train)
+datagen.fit(x_train)
 
 # fits the model on batches with real-time data augmentation:
-model.fit_generator(datagen.flow(X_train, Y_train, batch_size=32),
-                    samples_per_epoch=len(X_train), epochs=epochs)
+model.fit_generator(datagen.flow(x_train, y_train, batch_size=32),
+                    steps_per_epoch=len(x_train) / 32, epochs=epochs)
 
 # here's a more "manual" example
 for e in range(epochs):
-    print 'Epoch', e
+    print('Epoch', e)
     batches = 0
-    for X_batch, Y_batch in datagen.flow(X_train, Y_train, batch_size=32):
-        loss = model.train(X_batch, Y_batch)
+    for x_batch, y_batch in datagen.flow(x_train, y_train, batch_size=32):
+        model.fit(x_batch, y_batch)
         batches += 1
-        if batches >= len(X_train) / 32:
+        if batches >= len(x_train) / 32:
             # we need to break the loop by hand because
             # the generator loops indefinitely
             break
@@ -156,10 +164,10 @@ validation_generator = test_datagen.flow_from_directory(
 
 model.fit_generator(
         train_generator,
-        samples_per_epoch=2000,
+        steps_per_epoch=2000,
         epochs=50,
         validation_data=validation_generator,
-        num_val_samples=800)
+        validation_steps=800)
 ```
 
 Example of transforming images and masks together.
@@ -195,6 +203,6 @@ train_generator = zip(image_generator, mask_generator)
 
 model.fit_generator(
     train_generator,
-    samples_per_epoch=2000,
+    steps_per_epoch=2000,
     epochs=50)
 ```

docs/templates/preprocessing/text.md

@@ -3,7 +3,7 @@
 
 ```python
 keras.preprocessing.text.text_to_word_sequence(text, 
-    filters=base_filter(), lower=True, split=" ")
+    filters='!"#$%&()*+,-./:;<=>?@[\\]^_`{|}~\t\n', lower=True, split=" ")
 ```
 
 Split a sentence into a list of words.
@@ -12,7 +12,7 @@ Split a sentence into a list of words.
 
 - __Arguments__:
     - __text__: str.
-    - __filters__: list (or concatenation) of characters to filter out, such as punctuation. Default: base_filter(), includes basic punctuation, tabs, and newlines.
+    - __filters__: list (or concatenation) of characters to filter out, such as punctuation. Default: '!"#$%&()*+,-./:;<=>?@[\\]^_`{|}~\t\n' , includes basic punctuation, tabs, and newlines.
     - __lower__: boolean. Whether to set the text to lowercase.
     - __split__: str. Separator for word splitting.
 
@@ -20,7 +20,7 @@ Split a sentence into a list of words.
 
 ```python
 keras.preprocessing.text.one_hot(text, n,
-    filters=base_filter(), lower=True, split=" ")
+    filters='!"#$%&()*+,-./:;<=>?@[\\]^_`{|}~\t\n', lower=True, split=" ")
 ```
 
 One-hot encode a text into a list of word indexes in a vocabulary of size n.
@@ -33,14 +33,15 @@ One-hot encode a text into a list of word indexes in a vocabulary of size n.
 ## Tokenizer
 
 ```python
-keras.preprocessing.text.Tokenizer(num_words=None, filters=base_filter(), 
-    lower=True, split=" ")
+keras.preprocessing.text.Tokenizer(num_words=None, filters='!"#$%&()*+,-./:;<=>?@[\\]^_`{|}~\t\n', 
+    lower=True, split=" ", char_level=False)
 ```
 
 Class for vectorizing texts, or/and turning texts into sequences (=list of word indexes, where the word of rank i in the dataset (starting at 1) has index i).
 
 - __Arguments__: Same as `text_to_word_sequence` above.
     - __num_words__: None or int. Maximum number of words to work with (if set, tokenization will be restricted to the top num_words most common words in the dataset).
+    - __char_level__: if True, every character will be treated as a token.
 
 - __Methods__:
 

docs/templates/regularizers.md

@@ -14,10 +14,10 @@ These layers expose 3 keyword arguments:
 ## Example
 
 ```python
-from keras.regularizers import l2, activity_l2
+from keras import regularizers
 model.add(Dense(64, input_dim=64,
-                kernel_regularizer=l2(0.01),
-                activity_regularizer=activity_l2(0.01)))
+                kernel_regularizer=regularizers.l2(0.01),
+                activity_regularizer=regularizers.l1(0.01)))
 ```
 
 ## Available penalties

docs/templates/visualization.md

@@ -19,7 +19,7 @@ You can also directly obtain the `pydot.Graph` object and render it yourself,
 for example to show it in an ipython notebook :
 ```python
 from IPython.display import SVG
-from keras.utils.visualize_util import model_to_dot
+from keras.utils.vis_utils import model_to_dot
 
 SVG(model_to_dot(model).create(prog='dot', format='svg'))
 ```

examples/addition_rnn.py

@@ -78,7 +78,7 @@ INVERT = True
 
 # Maximum length of input is 'int + int' (e.g., '345+678'). Maximum length of
 # int is DIGITS.
-MAxLEN = DIGITS + 1 + DIGITS
+MAXLEN = DIGITS + 1 + DIGITS
 
 # All the numbers, plus sign and space for padding.
 chars = '0123456789+ '
@@ -98,9 +98,9 @@ while len(questions) < TRAINING_SIZE:
     if key in seen:
         continue
     seen.add(key)
-    # Pad the data with spaces such that it is always MAxLEN.
+    # Pad the data with spaces such that it is always MAXLEN.
     q = '{}+{}'.format(a, b)
-    query = q + ' ' * (MAxLEN - len(q))
+    query = q + ' ' * (MAXLEN - len(q))
     ans = str(a + b)
     # Answers can be of maximum size DIGITS + 1.
     ans += ' ' * (DIGITS + 1 - len(ans))
@@ -113,10 +113,10 @@ while len(questions) < TRAINING_SIZE:
 print('Total addition questions:', len(questions))
 
 print('Vectorization...')
-x = np.zeros((len(questions), MAxLEN, len(chars)), dtype=np.bool)
+x = np.zeros((len(questions), MAXLEN, len(chars)), dtype=np.bool)
 y = np.zeros((len(questions), DIGITS + 1, len(chars)), dtype=np.bool)
 for i, sentence in enumerate(questions):
-    x[i] = ctable.encode(sentence, MAxLEN)
+    x[i] = ctable.encode(sentence, MAXLEN)
 for i, sentence in enumerate(expected):
     y[i] = ctable.encode(sentence, DIGITS + 1)
 
@@ -151,7 +151,7 @@ model = Sequential()
 # "Encode" the input sequence using an RNN, producing an output of HIDDEN_SIZE.
 # Note: In a situation where your input sequences have a variable length,
 # use input_shape=(None, num_feature).
-model.add(RNN(HIDDEN_SIZE, input_shape=(MAxLEN, len(chars))))
+model.add(RNN(HIDDEN_SIZE, input_shape=(MAXLEN, len(chars))))
 # As the decoder RNN's input, repeatedly provide with the last hidden state of
 # RNN for each time step. Repeat 'DIGITS + 1' times as that's the maximum
 # length of output, e.g., when DIGITS=3, max output is 999+999=1998.
@@ -179,7 +179,9 @@ for iteration in range(1, 200):
     print()
     print('-' * 50)
     print('Iteration', iteration)
-    model.fit(x_train, y_train, batch_size=BATCH_SIZE, epochs=1,
+    model.fit(x_train, y_train,
+              batch_size=BATCH_SIZE,
+              epochs=1,
               validation_data=(x_val, y_val))
     # Select 10 samples from the validation set at random so we can visualize
     # errors.

examples/antirectifier.py

@@ -2,7 +2,7 @@
 
 We build a custom activation layer called 'Antirectifier',
 which modifies the shape of the tensor that passes through it.
-We need to specify two methods: `get_output_shape_for` and `call`.
+We need to specify two methods: `compute_output_shape` and `call`.
 
 Note that the same result can also be achieved via a Lambda layer.
 
@@ -98,8 +98,10 @@ model.compile(loss='categorical_crossentropy',
 
 # train the model
 model.fit(x_train, y_train,
-          batch_size=batch_size, epochs=epochs,
-          verbose=1, validation_data=(x_test, y_test))
+          batch_size=batch_size,
+          epochs=epochs,
+          verbose=1,
+          validation_data=(x_test, y_test))
 
 # next, compare with an equivalent network
 # with2x bigger Dense layers and ReLU

examples/babi_memnn.py

@@ -14,9 +14,9 @@ Time per epoch: 3s on CPU (core i7).
 '''
 from __future__ import print_function
 
-from keras.models import Sequential
+from keras.models import Sequential, Model
 from keras.layers.embeddings import Embedding
-from keras.layers import Activation, Dense, Merge, Permute, Dropout
+from keras.layers import Input, Activation, Dense, Permute, Dropout, add, dot, concatenate
 from keras.layers import LSTM
 from keras.utils.data_utils import get_file
 from keras.preprocessing.sequence import pad_sequences
@@ -38,7 +38,8 @@ def tokenize(sent):
 def parse_stories(lines, only_supporting=False):
     '''Parse stories provided in the bAbi tasks format
 
-    If only_supporting is true, only the sentences that support the answer are kept.
+    If only_supporting is true, only the sentences
+    that support the answer are kept.
     '''
     data = []
     story = []
@@ -68,9 +69,12 @@ def parse_stories(lines, only_supporting=False):
 
 
 def get_stories(f, only_supporting=False, max_length=None):
-    '''Given a file name, read the file, retrieve the stories, and then convert the sentences into a single story.
+    '''Given a file name, read the file,
+    retrieve the stories,
+    and then convert the sentences into a single story.
 
-    If max_length is supplied, any stories longer than max_length tokens will be discarded.
+    If max_length is supplied,
+    any stories longer than max_length tokens will be discarded.
     '''
     data = parse_stories(f.readlines(), only_supporting=only_supporting)
     flatten = lambda data: reduce(lambda x, y: x + y, data)
@@ -85,7 +89,8 @@ def vectorize_stories(data, word_idx, story_maxlen, query_maxlen):
     for story, query, answer in data:
         x = [word_idx[w] for w in story]
         xq = [word_idx[w] for w in query]
-        y = np.zeros(len(word_idx) + 1)  # let's not forget that index 0 is reserved
+        # let's not forget that index 0 is reserved
+        y = np.zeros(len(word_idx) + 1)
         y[word_idx[answer]] = 1
         X.append(x)
         Xq.append(xq)
@@ -93,7 +98,6 @@ def vectorize_stories(data, word_idx, story_maxlen, query_maxlen):
     return (pad_sequences(X, maxlen=story_maxlen),
             pad_sequences(Xq, maxlen=query_maxlen), np.array(Y))
 
-
 try:
     path = get_file('babi-tasks-v1-2.tar.gz', origin='https://s3.amazonaws.com/text-datasets/babi_tasks_1-20_v1-2.tar.gz')
 except:
@@ -116,7 +120,11 @@ print('Extracting stories for the challenge:', challenge_type)
 train_stories = get_stories(tar.extractfile(challenge.format('train')))
 test_stories = get_stories(tar.extractfile(challenge.format('test')))
 
-vocab = sorted(reduce(lambda x, y: x | y, (set(story + q + [answer]) for story, q, answer in train_stories + test_stories)))
+vocab = set()
+for story, q, answer in train_stories + test_stories:
+    vocab |= set(story + q + [answer])
+vocab = sorted(vocab)
+
 # Reserve 0 for masking via pad_sequences
 vocab_size = len(vocab) + 1
 story_maxlen = max(map(len, (x for x, _, _ in train_stories + test_stories)))
@@ -135,8 +143,14 @@ print('-')
 print('Vectorizing the word sequences...')
 
 word_idx = dict((c, i + 1) for i, c in enumerate(vocab))
-inputs_train, queries_train, answers_train = vectorize_stories(train_stories, word_idx, story_maxlen, query_maxlen)
-inputs_test, queries_test, answers_test = vectorize_stories(test_stories, word_idx, story_maxlen, query_maxlen)
+inputs_train, queries_train, answers_train = vectorize_stories(train_stories,
+                                                               word_idx,
+                                                               story_maxlen,
+                                                               query_maxlen)
+inputs_test, queries_test, answers_test = vectorize_stories(test_stories,
+                                                            word_idx,
+                                                            story_maxlen,
+                                                            query_maxlen)
 
 print('-')
 print('inputs: integer tensor of shape (samples, max_length)')
@@ -153,13 +167,25 @@ print('answers_test shape:', answers_test.shape)
 print('-')
 print('Compiling...')
 
+# placeholders
+input_sequence = Input((story_maxlen,))
+question = Input((query_maxlen,))
+
+# encoders
 # embed the input sequence into a sequence of vectors
 input_encoder_m = Sequential()
 input_encoder_m.add(Embedding(input_dim=vocab_size,
-                              output_dim=64,
-                              input_length=story_maxlen))
+                              output_dim=64))
 input_encoder_m.add(Dropout(0.3))
 # output: (samples, story_maxlen, embedding_dim)
+
+# embed the input into a sequence of vectors of size query_maxlen
+input_encoder_c = Sequential()
+input_encoder_c.add(Embedding(input_dim=vocab_size,
+                              output_dim=query_maxlen))
+input_encoder_c.add(Dropout(0.3))
+# output: (samples, story_maxlen, query_maxlen)
+
 # embed the question into a sequence of vectors
 question_encoder = Sequential()
 question_encoder.add(Embedding(input_dim=vocab_size,
@@ -167,44 +193,43 @@ question_encoder.add(Embedding(input_dim=vocab_size,
                                input_length=query_maxlen))
 question_encoder.add(Dropout(0.3))
 # output: (samples, query_maxlen, embedding_dim)
-# compute a 'match' between input sequence elements (which are vectors)
-# and the question vector sequence
-match = Sequential()
-match.add(Merge([input_encoder_m, question_encoder],
-                mode='dot',
-                dot_axes=[2, 2]))
-match.add(Activation('softmax'))
-# output: (samples, story_maxlen, query_maxlen)
-# embed the input into a single vector with size = story_maxlen:
-input_encoder_c = Sequential()
-input_encoder_c.add(Embedding(input_dim=vocab_size,
-                              output_dim=query_maxlen,
-                              input_length=story_maxlen))
-input_encoder_c.add(Dropout(0.3))
-# output: (samples, story_maxlen, query_maxlen)
-# sum the match vector with the input vector:
-response = Sequential()
-response.add(Merge([match, input_encoder_c], mode='sum'))
-# output: (samples, story_maxlen, query_maxlen)
-response.add(Permute((2, 1)))  # output: (samples, query_maxlen, story_maxlen)
 
-# concatenate the match vector with the question vector,
-# and do logistic regression on top
-answer = Sequential()
-answer.add(Merge([response, question_encoder], mode='concat', concat_axis=-1))
+# encode input sequence and questions (which are indices)
+# to sequences of dense vectors
+input_encoded_m = input_encoder_m(input_sequence)
+input_encoded_c = input_encoder_c(input_sequence)
+question_encoded = question_encoder(question)
+
+# compute a 'match' between the first input vector sequence
+# and the question vector sequence
+# shape: `(samples, story_maxlen, query_maxlen)`
+match = dot([input_encoded_m, question_encoded], axes=(2, 2))
+match = Activation('softmax')(match)
+
+# add the match matrix with the second input vector sequence
+response = add([match, input_encoded_c])  # (samples, story_maxlen, query_maxlen)
+response = Permute((2, 1))(response)  # (samples, query_maxlen, story_maxlen)
+
+# concatenate the match matrix with the question vector sequence
+answer = concatenate([response, question_encoded])
+
 # the original paper uses a matrix multiplication for this reduction step.
 # we choose to use a RNN instead.
-answer.add(LSTM(32))
-# one regularization layer -- more would probably be needed.
-answer.add(Dropout(0.3))
-answer.add(Dense(vocab_size))
-# we output a probability distribution over the vocabulary
-answer.add(Activation('softmax'))
+answer = LSTM(32)(answer)  # (samples, 32)
 
-answer.compile(optimizer='rmsprop', loss='categorical_crossentropy',
-               metrics=['accuracy'])
-# Note: you could use a Graph model to avoid repeat the input twice
-answer.fit([inputs_train, queries_train, inputs_train], answers_train,
-           batch_size=32,
-           epochs=120,
-           validation_data=([inputs_test, queries_test, inputs_test], answers_test))
+# one regularization layer -- more would probably be needed.
+answer = Dropout(0.3)(answer)
+answer = Dense(vocab_size)(answer)  # (samples, vocab_size)
+# we output a probability distribution over the vocabulary
+answer = Activation('softmax')(answer)
+
+# build the final model
+model = Model([input_sequence, question], answer)
+model.compile(optimizer='rmsprop', loss='categorical_crossentropy',
+              metrics=['accuracy'])
+
+# train
+model.fit([inputs_train, queries_train], answers_train,
+          batch_size=32,
+          epochs=120,
+          validation_data=([inputs_test, queries_test], answers_test))

examples/babi_rnn.py

@@ -83,7 +83,8 @@ def tokenize(sent):
 def parse_stories(lines, only_supporting=False):
     '''Parse stories provided in the bAbi tasks format
 
-    If only_supporting is true, only the sentences that support the answer are kept.
+    If only_supporting is true,
+    only the sentences that support the answer are kept.
     '''
     data = []
     story = []
@@ -113,9 +114,11 @@ def parse_stories(lines, only_supporting=False):
 
 
 def get_stories(f, only_supporting=False, max_length=None):
-    '''Given a file name, read the file, retrieve the stories, and then convert the sentences into a single story.
+    '''Given a file name, read the file, retrieve the stories,
+    and then convert the sentences into a single story.
 
-    If max_length is supplied, any stories longer than max_length tokens will be discarded.
+    If max_length is supplied,
+    any stories longer than max_length tokens will be discarded.
     '''
     data = parse_stories(f.readlines(), only_supporting=only_supporting)
     flatten = lambda data: reduce(lambda x, y: x + y, data)
@@ -130,7 +133,8 @@ def vectorize_stories(data, word_idx, story_maxlen, query_maxlen):
     for story, query, answer in data:
         x = [word_idx[w] for w in story]
         xq = [word_idx[w] for w in query]
-        y = np.zeros(len(word_idx) + 1)  # let's not forget that index 0 is reserved
+        # let's not forget that index 0 is reserved
+        y = np.zeros(len(word_idx) + 1)
         y[word_idx[answer]] = 1
         xs.append(x)
         xqs.append(xq)
@@ -140,10 +144,13 @@ def vectorize_stories(data, word_idx, story_maxlen, query_maxlen):
 RNN = recurrent.LSTM
 EMBED_HIDDEN_SIZE = 50
 SENT_HIDDEN_SIZE = 100
-QUERy_HIDDEN_SIZE = 100
+QUERY_HIDDEN_SIZE = 100
 BATCH_SIZE = 32
 EPOCHS = 40
-print('RNN / Embed / Sent / Query = {}, {}, {}, {}'.format(RNN, EMBED_HIDDEN_SIZE, SENT_HIDDEN_SIZE, QUERy_HIDDEN_SIZE))
+print('RNN / Embed / Sent / Query = {}, {}, {}, {}'.format(RNN,
+                                                           EMBED_HIDDEN_SIZE,
+                                                           SENT_HIDDEN_SIZE,
+                                                           QUERY_HIDDEN_SIZE))
 
 try:
     path = get_file('babi-tasks-v1-2.tar.gz', origin='https://s3.amazonaws.com/text-datasets/babi_tasks_1-20_v1-2.tar.gz')
@@ -164,7 +171,11 @@ challenge = 'tasks_1-20_v1-2/en/qa2_two-supporting-facts_{}.txt'
 train = get_stories(tar.extractfile(challenge.format('train')))
 test = get_stories(tar.extractfile(challenge.format('test')))
 
-vocab = sorted(reduce(lambda x, y: x | y, (set(story + q + [answer]) for story, q, answer in train + test)))
+vocab = set()
+for story, q, answer in train + test:
+    vocab |= set(story + q + [answer])
+vocab = sorted(vocab)
+
 # Reserve 0 for masking via pad_sequences
 vocab_size = len(vocab) + 1
 word_idx = dict((c, i + 1) for i, c in enumerate(vocab))
@@ -203,6 +214,10 @@ model.compile(optimizer='adam',
               metrics=['accuracy'])
 
 print('Training')
-model.fit([x, xq], y, batch_size=BATCH_SIZE, epochs=EPOCHS, validation_split=0.05)
-loss, acc = model.evaluate([tx, txq], ty, batch_size=BATCH_SIZE)
+model.fit([x, xq], y,
+          batch_size=BATCH_SIZE,
+          epochs=EPOCHS,
+          validation_split=0.05)
+loss, acc = model.evaluate([tx, txq], ty,
+                           batch_size=BATCH_SIZE)
 print('Test loss / test accuracy = {:.4f} / {:.4f}'.format(loss, acc))

examples/cifar10_cnn.py

@@ -20,11 +20,6 @@ num_classes = 10
 epochs = 200
 data_augmentation = True
 
-# input image dimensions
-img_rows, img_cols = 32, 32
-# The CIFAR10 images are RGB.
-img_channels = 3
-
 # The data, shuffled and split between train and test sets:
 (x_train, y_train), (x_test, y_test) = cifar10.load_data()
 print('x_train shape:', x_train.shape)
@@ -59,9 +54,12 @@ model.add(Dropout(0.5))
 model.add(Dense(num_classes))
 model.add(Activation('softmax'))
 
+# initiate RMSprop optimizer
+opt = keras.optimizers.rmsprop(lr=0.0001, decay=1e-6)
+
 # Let's train the model using RMSprop
 model.compile(loss='categorical_crossentropy',
-              optimizer='rmsprop',
+              optimizer=opt,
               metrics=['accuracy'])
 
 x_train = x_train.astype('float32')
@@ -91,7 +89,7 @@ else:
         horizontal_flip=True,  # randomly flip images
         vertical_flip=False)  # randomly flip images
 
-    # Compute quantities required for featurewise normalization
+    # Compute quantities required for feature-wise normalization
     # (std, mean, and principal components if ZCA whitening is applied).
     datagen.fit(x_train)
 

examples/deep_dream.py

@@ -8,24 +8,16 @@ e.g.:
 ```
 python deep_dream.py img/mypic.jpg results/dream
 ```
-
-It is preferable to run this script on GPU, for speed.
-If running on CPU, prefer the TensorFlow backend (much faster).
-
-Example results: http://i.imgur.com/FX6ROg9.jpg
 '''
 from __future__ import print_function
 
 from keras.preprocessing.image import load_img, img_to_array
 import numpy as np
-from scipy.misc import imsave
-from scipy.optimize import fmin_l_bfgs_b
-import time
+import scipy
 import argparse
 
-from keras.applications import vgg16
+from keras.applications import inception_v3
 from keras import backend as K
-from keras.layers import Input
 
 parser = argparse.ArgumentParser(description='Deep Dreams with Keras.')
 parser.add_argument('base_image_path', metavar='base', type=str,
@@ -37,183 +29,167 @@ args = parser.parse_args()
 base_image_path = args.base_image_path
 result_prefix = args.result_prefix
 
-# dimensions of the generated picture.
-img_height = 600
-img_width = 600
-
-# some settings we found interesting
-saved_settings = {
-    'bad_trip': {'features': {'block4_conv1': 0.05,
-                              'block4_conv2': 0.01,
-                              'block4_conv3': 0.01},
-                 'continuity': 0.1,
-                 'dream_l2': 0.8,
-                 'jitter': 5},
-    'dreamy': {'features': {'block5_conv1': 0.05,
-                            'block5_conv2': 0.02},
-               'continuity': 0.1,
-               'dream_l2': 0.02,
-               'jitter': 0},
+# These are the names of the layers
+# for which we try to maximize activation,
+# as well as their weight in the final loss
+# we try to maximize.
+# You can tweak these setting to obtain new visual effects.
+settings = {
+    'features': {
+        'mixed2': 0.2,
+        'mixed3': 0.5,
+        'mixed4': 2.,
+        'mixed5': 1.5,
+    },
 }
-# the settings we will use in this experiment
-settings = saved_settings['dreamy']
 
 
 def preprocess_image(image_path):
-    # util function to open, resize and format pictures
-    # into appropriate tensors
-    img = load_img(image_path, target_size=(img_height, img_width))
+    # Util function to open, resize and format pictures
+    # into appropriate tensors.
+    img = load_img(image_path)
     img = img_to_array(img)
     img = np.expand_dims(img, axis=0)
-    img = vgg16.preprocess_input(img)
+    img = inception_v3.preprocess_input(img)
     return img
 
 
 def deprocess_image(x):
-    # util function to convert a tensor into a valid image
+    # Util function to convert a tensor into a valid image.
     if K.image_data_format() == 'channels_first':
-        x = x.reshape((3, img_height, img_width))
+        x = x.reshape((3, x.shape[2], x.shape[3]))
         x = x.transpose((1, 2, 0))
     else:
-        x = x.reshape((img_height, img_width, 3))
-    # Remove zero-center by mean pixel
-    x[:, :, 0] += 103.939
-    x[:, :, 1] += 116.779
-    x[:, :, 2] += 123.68
-    # 'BGR'->'RGB'
-    x = x[:, :, ::-1]
+        x = x.reshape((x.shape[1], x.shape[2], 3))
+    x /= 2.
+    x += 0.5
+    x *= 255.
     x = np.clip(x, 0, 255).astype('uint8')
     return x
 
-if K.image_data_format() == 'channels_first':
-    img_size = (3, img_height, img_width)
-else:
-    img_size = (img_height, img_width, 3)
-# this will contain our generated image
-dream = Input(batch_shape=(1,) + img_size)
+K.set_learning_phase(0)
 
-# build the VGG16 network with our placeholder
-# the model will be loaded with pre-trained ImageNet weights
-model = vgg16.VGG16(input_tensor=dream,
-                    weights='imagenet', include_top=False)
+# Build the InceptionV3 network with our placeholder.
+# The model will be loaded with pre-trained ImageNet weights.
+model = inception_v3.InceptionV3(weights='imagenet',
+                                 include_top=False)
+dream = model.input
 print('Model loaded.')
 
-# get the symbolic outputs of each "key" layer (we gave them unique names).
+# Get the symbolic outputs of each "key" layer (we gave them unique names).
 layer_dict = dict([(layer.name, layer) for layer in model.layers])
 
-
-def continuity_loss(x):
-    # continuity loss util function
-    assert K.ndim(x) == 4
-    if K.image_data_format() == 'channels_first':
-        a = K.square(x[:, :, :img_height - 1, :img_width - 1] -
-                     x[:, :, 1:, :img_width - 1])
-        b = K.square(x[:, :, :img_height - 1, :img_width - 1] -
-                     x[:, :, :img_height - 1, 1:])
-    else:
-        a = K.square(x[:, :img_height - 1, :img_width - 1, :] -
-                     x[:, 1:, :img_width - 1, :])
-        b = K.square(x[:, :img_height - 1, :img_width - 1, :] -
-                     x[:, :img_height - 1, 1:, :])
-    return K.sum(K.pow(a + b, 1.25))
-
-# define the loss
+# Define the loss.
 loss = K.variable(0.)
 for layer_name in settings['features']:
-    # add the L2 norm of the features of a layer to the loss
+    # Add the L2 norm of the features of a layer to the loss.
     assert layer_name in layer_dict.keys(), 'Layer ' + layer_name + ' not found in model.'
     coeff = settings['features'][layer_name]
     x = layer_dict[layer_name].output
-    shape = layer_dict[layer_name].output_shape
-    # we avoid border artifacts by only involving non-border pixels in the loss
+    # We avoid border artifacts by only involving non-border pixels in the loss.
+    scaling = K.prod(K.cast(K.shape(x), 'float32'))
     if K.image_data_format() == 'channels_first':
-        loss -= coeff * K.sum(K.square(x[:, :, 2: shape[2] - 2, 2: shape[3] - 2])) / np.prod(shape[1:])
+        loss += coeff * K.sum(K.square(x[:, :, 2: -2, 2: -2])) / scaling
     else:
-        loss -= coeff * K.sum(K.square(x[:, 2: shape[1] - 2, 2: shape[2] - 2, :])) / np.prod(shape[1:])
+        loss += coeff * K.sum(K.square(x[:, 2: -2, 2: -2, :])) / scaling
 
-# add continuity loss (gives image local coherence, can result in an artful blur)
-loss += settings['continuity'] * continuity_loss(dream) / np.prod(img_size)
-# add image L2 norm to loss (prevents pixels from taking very high values, makes image darker)
-loss += settings['dream_l2'] * K.sum(K.square(dream)) / np.prod(img_size)
+# Compute the gradients of the dream wrt the loss.
+grads = K.gradients(loss, dream)[0]
+# Normalize gradients.
+grads /= K.maximum(K.mean(K.abs(grads)), 1e-7)
 
-# feel free to further modify the loss as you see fit, to achieve new effects...
-
-# compute the gradients of the dream wrt the loss
-grads = K.gradients(loss, dream)
-
-outputs = [loss]
-if isinstance(grads, (list, tuple)):
-    outputs += grads
-else:
-    outputs.append(grads)
-
-f_outputs = K.function([dream], outputs)
+# Set up function to retrieve the value
+# of the loss and gradients given an input image.
+outputs = [loss, grads]
+fetch_loss_and_grads = K.function([dream], outputs)
 
 
 def eval_loss_and_grads(x):
-    x = x.reshape((1,) + img_size)
-    outs = f_outputs([x])
+    outs = fetch_loss_and_grads([x])
     loss_value = outs[0]
-    if len(outs[1:]) == 1:
-        grad_values = outs[1].flatten().astype('float64')
-    else:
-        grad_values = np.array(outs[1:]).flatten().astype('float64')
+    grad_values = outs[1]
     return loss_value, grad_values
 
 
-class Evaluator(object):
-    """Loss and gradients evaluator.
+def resize_img(img, size):
+    img = np.copy(img)
+    if K.image_data_format() == 'channels_first':
+        factors = (1, 1,
+                   float(size[0]) / img.shape[2],
+                   float(size[1]) / img.shape[3])
+    else:
+        factors = (1,
+                   float(size[0]) / img.shape[1],
+                   float(size[1]) / img.shape[2],
+                   1)
+    return scipy.ndimage.zoom(img, factors, order=1)
 
-    This Evaluator class makes it possible
-    to compute loss and gradients in one pass
-    while retrieving them via two separate functions,
-    "loss" and "grads". This is done because scipy.optimize
-    requires separate functions for loss and gradients,
-    but computing them separately would be inefficient.
-    """
 
-    def __init__(self):
-        self.loss_value = None
-        self.grad_values = None
-
-    def loss(self, x):
-        assert self.loss_value is None
+def gradient_ascent(x, iterations, step, max_loss=None):
+    for i in range(iterations):
         loss_value, grad_values = eval_loss_and_grads(x)
-        self.loss_value = loss_value
-        self.grad_values = grad_values
-        return self.loss_value
+        if max_loss is not None and loss_value > max_loss:
+            break
+        print('..Loss value at', i, ':', loss_value)
+        x += step * grad_values
+    return x
 
-    def grads(self, x):
-        assert self.loss_value is not None
-        grad_values = np.copy(self.grad_values)
-        self.loss_value = None
-        self.grad_values = None
-        return grad_values
 
-evaluator = Evaluator()
+def save_img(img, fname):
+    pil_img = deprocess_image(np.copy(img))
+    scipy.misc.imsave(fname, pil_img)
 
-# Run scipy-based optimization (L-BFGS) over the pixels of the generated image
-# so as to minimize the loss
-x = preprocess_image(base_image_path)
-for i in range(5):
-    print('Start of iteration', i)
-    start_time = time.time()
 
-    # Add a random jitter to the initial image.
-    # This will be reverted at decoding time
-    random_jitter = (settings['jitter'] * 2) * (np.random.random(img_size) - 0.5)
-    x += random_jitter
+"""Process:
 
-    # Run L-BFGS for 7 steps
-    x, min_val, info = fmin_l_bfgs_b(evaluator.loss, x.flatten(),
-                                     fprime=evaluator.grads, maxfun=7)
-    print('Current loss value:', min_val)
-    # Decode the dream and save it
-    x = x.reshape(img_size)
-    x -= random_jitter
-    img = deprocess_image(np.copy(x))
-    fname = result_prefix + '_at_iteration_%d.png' % i
-    imsave(fname, img)
-    end_time = time.time()
-    print('Image saved as', fname)
-    print('Iteration %d completed in %ds' % (i, end_time - start_time))
+- Load the original image.
+- Define a number of processing scales (i.e. image shapes),
+    from smallest to largest.
+- Resize the original image to the smallest scale.
+- For every scale, starting with the smallest (i.e. current one):
+    - Run gradient ascent
+    - Upscale image to the next scale
+    - Reinject the detail that was lost at upscaling time
+- Stop when we are back to the original size.
+
+To obtain the detail lost during upscaling, we simply
+take the original image, shrink it down, upscale it,
+and compare the result to the (resized) original image.
+"""
+
+
+# Playing with these hyperparameters will also allow you to achieve new effects
+step = 0.01  # Gradient ascent step size
+num_octave = 3  # Number of scales at which to run gradient ascent
+octave_scale = 1.4  # Size ratio between scales
+iterations = 20  # Number of ascent steps per scale
+max_loss = 10.
+
+img = preprocess_image(base_image_path)
+if K.image_data_format() == 'channels_first':
+    original_shape = img.shape[2:]
+else:
+    original_shape = img.shape[1:3]
+successive_shapes = [original_shape]
+for i in range(1, num_octave):
+    shape = tuple([int(dim / (octave_scale ** i)) for dim in original_shape])
+    successive_shapes.append(shape)
+successive_shapes = successive_shapes[::-1]
+original_img = np.copy(img)
+shrunk_original_img = resize_img(img, successive_shapes[0])
+
+for shape in successive_shapes:
+    print('Processing image shape', shape)
+    img = resize_img(img, shape)
+    img = gradient_ascent(img,
+                          iterations=iterations,
+                          step=step,
+                          max_loss=max_loss)
+    upscaled_shrunk_original_img = resize_img(shrunk_original_img, shape)
+    same_size_original = resize_img(original_img, shape)
+    lost_detail = same_size_original - upscaled_shrunk_original_img
+
+    img += lost_detail
+    shrunk_original_img = resize_img(original_img, shape)
+
+save_img(img, fname=result_prefix + '.png')

examples/image_ocr.py

@@ -41,9 +41,10 @@ import numpy as np
 from scipy import ndimage
 import pylab
 from keras import backend as K
-from keras.layers.convolutional import Convolution2D, MaxPooling2D
+from keras.layers.convolutional import Conv2D, MaxPooling2D
 from keras.layers import Input, Dense, Activation
-from keras.layers import Reshape, Lambda, merge
+from keras.layers import Reshape, Lambda
+from keras.layers.merge import add, concatenate
 from keras.models import Model
 from keras.layers.recurrent import GRU
 from keras.optimizers import SGD
@@ -126,9 +127,9 @@ def shuffle_mats_or_lists(matrix_list, stop_ind=None):
         stop_ind = len_val
     assert stop_ind <= len_val
 
-    a = range(stop_ind)
+    a = list(range(stop_ind))
     np.random.shuffle(a)
-    a += range(stop_ind, len_val)
+    a += list(range(stop_ind, len_val))
     for mat in matrix_list:
         if isinstance(mat, np.ndarray):
             ret.append(mat[a])
@@ -403,8 +404,8 @@ def train(run_name, start_epoch, stop_epoch, img_w):
     val_words = int(words_per_epoch * (val_split))
 
     # Network parameters
-    conv_filterss = 16
-    filter_size = 3
+    conv_filters = 16
+    kernel_size = (3, 3)
     pool_size = 2
     time_dense_size = 32
     rnn_size = 512
@@ -415,7 +416,7 @@ def train(run_name, start_epoch, stop_epoch, img_w):
         input_shape = (img_w, img_h, 1)
 
     fdir = os.path.dirname(get_file('wordlists.tgz',
-                                    origin='http://www.isosemi.com/datasets/wordlists.tgz', untar=True))
+                                    origin='http://www.mythic-ai.com/datasets/wordlists.tgz', untar=True))
 
     img_gen = TextImageGenerator(monogram_file=os.path.join(fdir, 'wordlist_mono_clean.txt'),
                                  bigram_file=os.path.join(fdir, 'wordlist_bi_clean.txt'),
@@ -427,14 +428,16 @@ def train(run_name, start_epoch, stop_epoch, img_w):
                                  )
     act = 'relu'
     input_data = Input(name='the_input', shape=input_shape, dtype='float32')
-    inner = Convolution2D(conv_filterss, filter_size, filter_size, border_mode='same',
-                          activation=act, init='he_normal', name='conv1')(input_data)
+    inner = Conv2D(conv_filters, kernel_size, padding='same',
+                   activation=act, kernel_initializer='he_normal',
+                   name='conv1')(input_data)
     inner = MaxPooling2D(pool_size=(pool_size, pool_size), name='max1')(inner)
-    inner = Convolution2D(conv_filterss, filter_size, filter_size, border_mode='same',
-                          activation=act, init='he_normal', name='conv2')(inner)
+    inner = Conv2D(conv_filters, kernel_size, padding='same',
+                   activation=act, kernel_initializer='he_normal',
+                   name='conv2')(inner)
     inner = MaxPooling2D(pool_size=(pool_size, pool_size), name='max2')(inner)
 
-    conv_to_rnn_dims = (img_w // (pool_size ** 2), (img_h // (pool_size ** 2)) * conv_filterss)
+    conv_to_rnn_dims = (img_w // (pool_size ** 2), (img_h // (pool_size ** 2)) * conv_filters)
     inner = Reshape(target_shape=conv_to_rnn_dims, name='reshape')(inner)
 
     # cuts down input size going into RNN:
@@ -442,17 +445,17 @@ def train(run_name, start_epoch, stop_epoch, img_w):
 
     # Two layers of bidirecitonal GRUs
     # GRU seems to work as well, if not better than LSTM:
-    gru_1 = GRU(rnn_size, return_sequences=True, init='he_normal', name='gru1')(inner)
-    gru_1b = GRU(rnn_size, return_sequences=True, go_backwards=True, init='he_normal', name='gru1_b')(inner)
-    gru1_merged = merge([gru_1, gru_1b], mode='sum')
-    gru_2 = GRU(rnn_size, return_sequences=True, init='he_normal', name='gru2')(gru1_merged)
-    gru_2b = GRU(rnn_size, return_sequences=True, go_backwards=True, init='he_normal', name='gru2_b')(gru1_merged)
+    gru_1 = GRU(rnn_size, return_sequences=True, kernel_initializer='he_normal', name='gru1')(inner)
+    gru_1b = GRU(rnn_size, return_sequences=True, go_backwards=True, kernel_initializer='he_normal', name='gru1_b')(inner)
+    gru1_merged = add([gru_1, gru_1b])
+    gru_2 = GRU(rnn_size, return_sequences=True, kernel_initializer='he_normal', name='gru2')(gru1_merged)
+    gru_2b = GRU(rnn_size, return_sequences=True, go_backwards=True, kernel_initializer='he_normal', name='gru2_b')(gru1_merged)
 
     # transforms RNN output to character activations:
-    inner = Dense(img_gen.get_output_size(), init='he_normal',
-                  name='dense2')(merge([gru_2, gru_2b], mode='concat'))
+    inner = Dense(img_gen.get_output_size(), kernel_initializer='he_normal',
+                  name='dense2')(concatenate([gru_2, gru_2b]))
     y_pred = Activation('softmax', name='softmax')(inner)
-    Model(input=[input_data], output=y_pred).summary()
+    Model(inputs=input_data, outputs=y_pred).summary()
 
     labels = Input(name='the_labels', shape=[img_gen.absolute_max_string_len], dtype='float32')
     input_length = Input(name='input_length', shape=[1], dtype='int64')
@@ -464,7 +467,7 @@ def train(run_name, start_epoch, stop_epoch, img_w):
     # clipnorm seems to speeds up convergence
     sgd = SGD(lr=0.02, decay=1e-6, momentum=0.9, nesterov=True, clipnorm=5)
 
-    model = Model(input=[input_data, labels, input_length, label_length], output=[loss_out])
+    model = Model(inputs=[input_data, labels, input_length, label_length], outputs=loss_out)
 
     # the loss calc occurs elsewhere, so use a dummy lambda func for the loss
     model.compile(loss={'ctc': lambda y_true, y_pred: y_pred}, optimizer=sgd)
@@ -476,8 +479,8 @@ def train(run_name, start_epoch, stop_epoch, img_w):
 
     viz_cb = VizCallback(run_name, test_func, img_gen.next_val())
 
-    model.fit_generator(generator=img_gen.next_train(), samples_per_epoch=(words_per_epoch - val_words),
-                        epochs=stop_epoch, validation_data=img_gen.next_val(), num_val_samples=val_words,
+    model.fit_generator(generator=img_gen.next_train(), steps_per_epoch=(words_per_epoch - val_words),
+                        epochs=stop_epoch, validation_data=img_gen.next_val(), validation_steps=val_words,
                         callbacks=[viz_cb, img_gen], initial_epoch=start_epoch)
 
 

examples/imdb_cnn_lstm.py

@@ -67,7 +67,9 @@ model.compile(loss='binary_crossentropy',
               metrics=['accuracy'])
 
 print('Train...')
-model.fit(x_train, y_train, batch_size=batch_size, epochs=epochs,
+model.fit(x_train, y_train,
+          batch_size=batch_size,
+          epochs=epochs,
           validation_data=(x_test, y_test))
 score, acc = model.evaluate(x_test, y_test, batch_size=batch_size)
 print('Test score:', score)

examples/imdb_lstm.py

@@ -45,7 +45,9 @@ model.compile(loss='binary_crossentropy',
               metrics=['accuracy'])
 
 print('Train...')
-model.fit(x_train, y_train, batch_size=batch_size, epochs=15,
+model.fit(x_train, y_train,
+          batch_size=batch_size,
+          epochs=15,
           validation_data=(x_test, y_test))
 score, acc = model.evaluate(x_test, y_test,
                             batch_size=batch_size)

examples/lstm_benchmark.py

@@ -12,8 +12,8 @@ into one, large matrix, resulting in faster computation time as the GPU can
 utilize more cores, at the expense of reduced regularization because the same
 dropout is shared across the gates.
 
-Note that the relative performance of the different `consume_less` modes
-can vary depending on your device, your model and the size of your data.
+Note that the relative performance of the different implementations can
+vary depending on your device, your model and the size of your data.
 '''
 
 import time

examples/lstm_text_generation.py

@@ -73,7 +73,9 @@ for iteration in range(1, 60):
     print()
     print('-' * 50)
     print('Iteration', iteration)
-    model.fit(X, y, batch_size=128, epochs=1)
+    model.fit(X, y,
+              batch_size=128,
+              epochs=1)
 
     start_index = random.randint(0, len(text) - maxlen - 1)
 

examples/mnist_acgan.py

@@ -101,7 +101,7 @@ def build_discriminator():
     cnn.add(LeakyReLU())
     cnn.add(Dropout(0.3))
 
-    cnn.add(Conv2D(64, 3, padding='same', strides=2))
+    cnn.add(Conv2D(64, 3, padding='same', strides=1))
     cnn.add(LeakyReLU())
     cnn.add(Dropout(0.3))
 
@@ -222,7 +222,7 @@ if __name__ == '__main__':
             noise = np.random.uniform(-1, 1, (2 * batch_size, latent_size))
             sampled_labels = np.random.randint(0, 10, 2 * batch_size)
 
-            # we want to train the genrator to trick the discriminator
+            # we want to train the generator to trick the discriminator
             # For the generator, we want all the {fake, not-fake} labels to say
             # not-fake
             trick = np.ones(2 * batch_size)

examples/mnist_cnn.py

@@ -60,8 +60,11 @@ model.compile(loss=keras.losses.categorical_crossentropy,
               optimizer=keras.optimizers.Adadelta(),
               metrics=['accuracy'])
 
-model.fit(x_train, y_train, batch_size=batch_size, epochs=epochs,
-          verbose=1, validation_data=(x_test, y_test))
+model.fit(x_train, y_train,
+          batch_size=batch_size,
+          epochs=epochs,
+          verbose=1,
+          validation_data=(x_test, y_test))
 score = model.evaluate(x_test, y_test, verbose=0)
 print('Test loss:', score[0])
 print('Test accuracy:', score[1])

examples/mnist_hierarchical_rnn.py

@@ -79,8 +79,10 @@ model.compile(loss='categorical_crossentropy',
 
 # Training.
 model.fit(x_train, y_train,
-          batch_size=batch_size, epochs=epochs,
-          verbose=1, validation_data=(x_test, y_test))
+          batch_size=batch_size,
+          epochs=epochs,
+          verbose=1,
+          validation_data=(x_test, y_test))
 
 # Evaluation.
 scores = model.evaluate(x_test, y_test, verbose=0)

examples/mnist_irnn.py

@@ -62,8 +62,11 @@ model.compile(loss='categorical_crossentropy',
               optimizer=rmsprop,
               metrics=['accuracy'])
 
-model.fit(x_train, y_train, batch_size=batch_size, epochs=epochs,
-          verbose=1, validation_data=(x_test, y_test))
+model.fit(x_train, y_train,
+          batch_size=batch_size,
+          epochs=epochs,
+          verbose=1,
+          validation_data=(x_test, y_test))
 
 scores = model.evaluate(x_test, y_test, verbose=0)
 print('IRNN test score:', scores[0])

examples/mnist_mlp.py

@@ -48,8 +48,10 @@ model.compile(loss='categorical_crossentropy',
               metrics=['accuracy'])
 
 history = model.fit(x_train, y_train,
-                    batch_size=batch_size, epochs=epochs,
-                    verbose=1, validation_data=(x_test, y_test))
+                    batch_size=batch_size,
+                    epochs=epochs,
+                    verbose=1,
+                    validation_data=(x_test, y_test))
 score = model.evaluate(x_test, y_test, verbose=0)
 print('Test loss:', score[0])
 print('Test accuracy:', score[1])

examples/mnist_net2net.py

@@ -26,7 +26,7 @@ Notes
 
 Experiments
 - Teacher model: a basic CNN model trained on MNIST for 3 epochs.
-- Net2WiderNet exepriment:
+- Net2WiderNet experiment:
   + Student model has a wider Conv2D layer and a wider FC layer.
   + Comparison of 'random-padding' vs 'net2wider' weight initialization.
   + With both methods, student model should immediately perform as well as
@@ -231,7 +231,8 @@ def make_teacher_model(train_data, validation_data, epochs=3):
                   metrics=['accuracy'])
 
     train_x, train_y = train_data
-    history = model.fit(train_x, train_y, epochs=epochs,
+    history = model.fit(train_x, train_y,
+                        epochs=epochs,
                         validation_data=validation_data)
     return model, history
 
@@ -280,7 +281,8 @@ def make_wider_student_model(teacher_model, train_data,
                   metrics=['accuracy'])
 
     train_x, train_y = train_data
-    history = model.fit(train_x, train_y, epochs=epochs,
+    history = model.fit(train_x, train_y,
+                        epochs=epochs,
                         validation_data=validation_data)
     return model, history
 
@@ -328,7 +330,8 @@ def make_deeper_student_model(teacher_model, train_data,
                   metrics=['accuracy'])
 
     train_x, train_y = train_data
-    history = model.fit(train_x, train_y, epochs=epochs,
+    history = model.fit(train_x, train_y,
+                        epochs=epochs,
                         validation_data=validation_data)
     return model, history
 

examples/mnist_siamese_graph.py

@@ -24,7 +24,7 @@ from keras import backend as K
 
 def euclidean_distance(vects):
     x, y = vects
-    return K.sqrt(K.sum(K.square(x - y), axis=1, keepdims=True))
+    return K.sqrt(K.maximum(K.sum(K.square(x - y), axis=1, keepdims=True), K.epsilon()))
 
 
 def eucl_dist_output_shape(shapes):
@@ -117,9 +117,9 @@ model = Model([input_a, input_b], distance)
 rms = RMSprop()
 model.compile(loss=contrastive_loss, optimizer=rms)
 model.fit([tr_pairs[:, 0], tr_pairs[:, 1]], tr_y,
-          validation_data=([te_pairs[:, 0], te_pairs[:, 1]], te_y),
           batch_size=128,
-          epochs=epochs)
+          epochs=epochs,
+          validation_data=([te_pairs[:, 0], te_pairs[:, 1]], te_y))
 
 # compute final accuracy on training and test sets
 pred = model.predict([tr_pairs[:, 0], tr_pairs[:, 1]])

examples/mnist_swwae.py

@@ -35,12 +35,12 @@ applied as a bias because we know the MNIST digits are mapped to [0,1].
 References:
 [3]
 'Deep Residual Learning for Image Recognition'
-Kaiming He, xiangyu Zhang, Shaoqing Ren, Jian Sun
+Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun
 https://arxiv.org/abs/1512.03385v1
 
 [4]
 'Identity Mappings in Deep Residual Networks'
-Kaiming He, xiangyu Zhang, Shaoqing Ren, Jian Sun
+Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun
 https://arxiv.org/abs/1603.05027v3
 
 '''
@@ -51,20 +51,45 @@ from keras.datasets import mnist
 from keras.models import Model
 from keras.layers import Activation
 from keras.layers import UpSampling2D, Conv2D, MaxPooling2D
-from keras.layers import Input, BatchNormalization
+from keras.layers import Input, BatchNormalization, ELU
 import matplotlib.pyplot as plt
 import keras.backend as K
 from keras import layers
 
 
-def convresblock(x, nfeats=8, ksize=3, nskipped=2):
-    ''' The proposed residual block from [4]'''
+def convresblock(x, nfeats=8, ksize=3, nskipped=2, elu=True):
+    """The proposed residual block from [4].
+
+    Running with elu=True will use ELU nonlinearity and running with
+    elu=False will use BatchNorm + RELU nonlinearity.  While ELU's are fast
+    due to the fact they do not suffer from BatchNorm overhead, they may
+    overfit because they do not offer the stochastic element of the batch
+    formation process of BatchNorm, which acts as a good regularizer.
+
+    # Arguments
+        x: 4D tensor, the tensor to feed through the block
+        nfeats: Integer, number of feature maps for conv layers.
+        ksize: Integer, width and height of conv kernels in first convolution.
+        nskipped: Integer, number of conv layers for the residual function.
+        elu: Boolean, whether to use ELU or BN+RELU.
+
+    # Input shape
+        4D tensor with shape:
+        `(batch, channels, rows, cols)`
+
+    # Output shape
+        4D tensor with shape:
+        `(batch, filters, rows, cols)`
+    """
     y0 = Conv2D(nfeats, ksize, padding='same')(x)
     y = y0
     for i in range(nskipped):
-        y = BatchNormalization(axis=1)(y)
-        y = Activation('relu')(y)
-        y = Conv2D(nfeats, ksize, padding='same')(y)
+        if elu:
+            y = ELU()(y)
+        else:
+            y = BatchNormalization(axis=1)(y)
+            y = Activation('relu')(y)
+        y = Conv2D(nfeats, 1, padding='same')(y)
     return layers.add([y0, y])
 
 
@@ -143,7 +168,8 @@ y = img_input
 for i in range(nlayers):
     y_prepool = convresblock(y, nfeats=nfeats_all[i + 1], ksize=ksize)
     y = MaxPooling2D(pool_size=(pool_sizes[i], pool_sizes[i]))(y_prepool)
-    wheres[i] = layers.Lambda(getwhere, output_shape=lambda x: x[0])([y_prepool, y])
+    wheres[i] = layers.Lambda(
+        getwhere, output_shape=lambda x: x[0])([y_prepool, y])
 
 # Now build the decoder, and use the stored 'where' masks to place the features
 for i in range(nlayers):
@@ -160,8 +186,10 @@ model = Model(img_input, y)
 model.compile('adam', 'mse')
 
 # Fit the model
-model.fit(x_train, x_train, validation_data=(x_test, x_test),
-          batch_size=batch_size, epochs=epochs)
+model.fit(x_train, x_train,
+          batch_size=batch_size,
+          epochs=epochs,
+          validation_data=(x_test, x_test))
 
 # Plot
 x_recon = model.predict(x_test[:25])

examples/mnist_transfer_cnn.py

@@ -63,7 +63,8 @@ def train_model(model, train, test, num_classes):
 
     t = now()
     model.fit(x_train, y_train,
-              batch_size=batch_size, epochs=epochs,
+              batch_size=batch_size,
+              epochs=epochs,
               verbose=1,
               validation_data=(x_test, y_test))
     print('Training time: %s' % (now() - t))

examples/neural_doodle.py

@@ -196,8 +196,8 @@ x = mask_input
 for layer in image_model.layers[1:]:
     name = 'mask_%s' % layer.name
     if 'conv' in layer.name:
-        x = AveragePooling2D((3, 3), strides=(
-            1, 1), name=name, border_mode='same')(x)
+        x = AveragePooling2D((3, 3), padding='same', strides=(
+            1, 1), name=name)(x)
     elif 'pool' in layer.name:
         x = AveragePooling2D((2, 2), name=name)(x)
 mask_model = Model(mask_input, x)
@@ -238,6 +238,7 @@ def region_style_loss(style_image, target_image, style_mask, target_mask):
         masked_target = K.permute_dimensions(
             target_image, (2, 0, 1)) * target_mask
         num_channels = K.shape(style_image)[-1]
+    num_channels = K.cast(num_channels, dtype='float32')
     s = gram_matrix(masked_style) / K.mean(style_mask) / num_channels
     c = gram_matrix(masked_target) / K.mean(target_mask) / num_channels
     return K.mean(K.square(s - c))

examples/pretrained_word_embeddings.py

@@ -143,6 +143,7 @@ model.compile(loss='categorical_crossentropy',
               optimizer='rmsprop',
               metrics=['acc'])
 
-# happy learning!
-model.fit(x_train, y_train, validation_data=(x_val, y_val),
-          epochs=10, batch_size=128)
+model.fit(x_train, y_train,
+          batch_size=128,
+          epochs=10,
+          validation_data=(x_val, y_val))

examples/reuters_mlp.py

@@ -50,8 +50,10 @@ model.compile(loss='categorical_crossentropy',
               metrics=['accuracy'])
 
 history = model.fit(x_train, y_train,
-                    epochs=epochs, batch_size=batch_size,
-                    verbose=1, validation_split=0.1)
+                    batch_size=batch_size,
+                    epochs=epochs,
+                    verbose=1,
+                    validation_split=0.1)
 score = model.evaluate(x_test, y_test,
                        batch_size=batch_size, verbose=1)
 print('Test score:', score[0])

examples/stateful_lstm.py

@@ -62,11 +62,18 @@ model.compile(loss='mse', optimizer='rmsprop')
 print('Training')
 for i in range(epochs):
     print('Epoch', i, '/', epochs)
-    model.fit(cos,
-              expected_output,
+
+    # Note that the last state for sample i in a batch will
+    # be used as initial state for sample i in the next batch.
+    # Thus we are simultaneously training on batch_size series with
+    # lower resolution than the original series contained in cos.
+    # Each of these series are offset by one step and can be
+    # extracted with cos[i::batch_size].
+
+    model.fit(cos, expected_output,
               batch_size=batch_size,
-              verbose=1,
               epochs=1,
+              verbose=1,
               shuffle=False)
     model.reset_states()
 

examples/variational_autoencoder.py

@@ -6,7 +6,7 @@ import numpy as np
 import matplotlib.pyplot as plt
 from scipy.stats import norm
 
-from keras.layers import Input, Dense, Lambda
+from keras.layers import Input, Dense, Lambda, Layer
 from keras.models import Model
 from keras import backend as K
 from keras import metrics
@@ -19,6 +19,7 @@ intermediate_dim = 256
 epochs = 50
 epsilon_std = 1.0
 
+
 x = Input(batch_shape=(batch_size, original_dim))
 h = Dense(intermediate_dim, activation='relu')(x)
 z_mean = Dense(latent_dim)(h)
@@ -41,13 +42,29 @@ h_decoded = decoder_h(z)
 x_decoded_mean = decoder_mean(h_decoded)
 
 
-def vae_loss(x, x_decoded_mean):
-    xent_loss = original_dim * metrics.binary_crossentropy(x, x_decoded_mean)
-    kl_loss = - 0.5 * K.sum(1 + z_log_var - K.square(z_mean) - K.exp(z_log_var), axis=-1)
-    return xent_loss + kl_loss
+# Custom loss layer
+class CustomVariationalLayer(Layer):
+    def __init__(self, **kwargs):
+        self.is_placeholder = True
+        super(CustomVariationalLayer, self).__init__(**kwargs)
+
+    def vae_loss(self, x, x_decoded_mean):
+        xent_loss = original_dim * metrics.binary_crossentropy(x, x_decoded_mean)
+        kl_loss = - 0.5 * K.sum(1 + z_log_var - K.square(z_mean) - K.exp(z_log_var), axis=-1)
+        return K.mean(xent_loss + kl_loss)
+
+    def call(self, inputs):
+        x = inputs[0]
+        x_decoded_mean = inputs[1]
+        loss = self.vae_loss(x, x_decoded_mean)
+        self.add_loss(loss, inputs=inputs)
+        # We won't actually use the output.
+        return x
+
+y = CustomVariationalLayer()([x, x_decoded_mean])
+vae = Model(x, y)
+vae.compile(optimizer='rmsprop', loss=None)
 
-vae = Model(x, x_decoded_mean)
-vae.compile(optimizer='rmsprop', loss=vae_loss)
 
 # train the VAE on MNIST digits
 (x_train, y_train), (x_test, y_test) = mnist.load_data()
@@ -57,7 +74,7 @@ x_test = x_test.astype('float32') / 255.
 x_train = x_train.reshape((len(x_train), np.prod(x_train.shape[1:])))
 x_test = x_test.reshape((len(x_test), np.prod(x_test.shape[1:])))
 
-vae.fit(x_train, x_train,
+vae.fit(x_train,
         shuffle=True,
         epochs=epochs,
         batch_size=batch_size,

examples/variational_autoencoder_deconv.py

@@ -7,7 +7,7 @@ import numpy as np
 import matplotlib.pyplot as plt
 from scipy.stats import norm
 
-from keras.layers import Input, Dense, Lambda, Flatten, Reshape
+from keras.layers import Input, Dense, Lambda, Flatten, Reshape, Layer
 from keras.layers import Conv2D, Conv2DTranspose
 from keras.models import Model
 from keras import backend as K
@@ -106,17 +106,31 @@ x_decoded_relu = decoder_deconv_3_upsamp(deconv_2_decoded)
 x_decoded_mean_squash = decoder_mean_squash(x_decoded_relu)
 
 
-def vae_loss(x, x_decoded_mean):
-    # NOTE: binary_crossentropy expects a batch_size by dim
-    # for x and x_decoded_mean, so we MUST flatten these!
-    x = K.flatten(x)
-    x_decoded_mean = K.flatten(x_decoded_mean)
-    xent_loss = img_rows * img_cols * metrics.binary_crossentropy(x, x_decoded_mean)
-    kl_loss = - 0.5 * K.mean(1 + z_log_var - K.square(z_mean) - K.exp(z_log_var), axis=-1)
-    return xent_loss + kl_loss
+# Custom loss layer
+class CustomVariationalLayer(Layer):
+    def __init__(self, **kwargs):
+        self.is_placeholder = True
+        super(CustomVariationalLayer, self).__init__(**kwargs)
 
-vae = Model(x, x_decoded_mean_squash)
-vae.compile(optimizer='rmsprop', loss=vae_loss)
+    def vae_loss(self, x, x_decoded_mean_squash):
+        x = K.flatten(x)
+        x_decoded_mean_squash = K.flatten(x_decoded_mean_squash)
+        xent_loss = img_rows * img_cols * metrics.binary_crossentropy(x, x_decoded_mean_squash)
+        kl_loss = - 0.5 * K.mean(1 + z_log_var - K.square(z_mean) - K.exp(z_log_var), axis=-1)
+        return K.mean(xent_loss + kl_loss)
+
+    def call(self, inputs):
+        x = inputs[0]
+        x_decoded_mean_squash = inputs[1]
+        loss = self.vae_loss(x, x_decoded_mean_squash)
+        self.add_loss(loss, inputs=inputs)
+        # We don't use this output.
+        return x
+
+
+y = CustomVariationalLayer()([x, x_decoded_mean_squash])
+vae = Model(x, y)
+vae.compile(optimizer='rmsprop', loss=None)
 vae.summary()
 
 # train the VAE on MNIST digits
@@ -129,7 +143,7 @@ x_test = x_test.reshape((x_test.shape[0],) + original_img_size)
 
 print('x_train.shape:', x_train.shape)
 
-vae.fit(x_train, x_train,
+vae.fit(x_train,
         shuffle=True,
         epochs=epochs,
         batch_size=batch_size,

keras/__init__.py

@@ -1,21 +1,28 @@
+"""The Keras API.
+"""
 from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
 
 from . import activations
 from . import applications
 from . import backend
-from . import datasets
-from . import engine
-from . import layers
-from . import preprocessing
-from . import utils
-from . import wrappers
 from . import callbacks
 from . import constraints
+from . import datasets
+from . import engine
 from . import initializers
+from . import layers
+from . import losses
 from . import metrics
 from . import models
-from . import losses
 from . import optimizers
+from . import preprocessing
 from . import regularizers
+from . import utils
+from . import wrappers
 
-__version__ = '2.0.0'
+# Importable from root because it's technically not a layer
+from .layers import Input
+
+__version__ = '2.0.4-tf'

keras/activations.py

@@ -1,21 +1,38 @@
+"""Keras built-in activation functions.
+"""
 from __future__ import absolute_import
-import six
+from __future__ import division
+from __future__ import print_function
+
+import warnings
 from . import backend as K
+import six
 from .utils.generic_utils import deserialize_keras_object
+from .engine import Layer
 
 
-def softmax(x):
+def softmax(x, axis=-1):
+    """Softmax activation function.
+
+    # Arguments
+        x : Tensor.
+        axis: Integer, axis along which the softmax normalization is applied.
+
+    # Returns
+        Tensor, output of softmax transformation.
+
+    # Raises
+        ValueError: In case `dim(x) == 1`.
+    """
     ndim = K.ndim(x)
     if ndim == 2:
         return K.softmax(x)
-    elif ndim == 3:
-        e = K.exp(x - K.max(x, axis=-1, keepdims=True))
-        s = K.sum(e, axis=-1, keepdims=True)
+    elif ndim > 2:
+        e = K.exp(x - K.max(x, axis=axis, keepdims=True))
+        s = K.sum(e, axis=axis, keepdims=True)
         return e / s
     else:
-        raise ValueError('Cannot apply softmax to a tensor '
-                         'that is not 2D or 3D. '
-                         'Here, ndim=' + str(ndim))
+        raise ValueError('Cannot apply softmax to a tensor that is 1D')
 
 
 def elu(x, alpha=1.0):
@@ -68,6 +85,13 @@ def get(identifier):
         identifier = str(identifier)
         return deserialize(identifier)
     elif callable(identifier):
+        if isinstance(identifier, Layer):
+            warnings.warn((
+                'Do not pass a layer instance (such as {identifier}) as the '
+                'activation argument of another layer. Instead, advanced '
+                'activation layers should be used just like any other '
+                'layer in a model.'
+            ).format(identifier=identifier.__class__.__name__))
         return identifier
     else:
         raise ValueError('Could not interpret '

keras/applications/__init__.py

@@ -1,5 +1,11 @@
+"""Keras Applications: models with automatic loading of pre-trained weights.
+"""
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+from .inception_v3 import InceptionV3
+from .resnet50 import ResNet50
 from .vgg16 import VGG16
 from .vgg19 import VGG19
-from .resnet50 import ResNet50
-from .inception_v3 import InceptionV3
 from .xception import Xception

keras/applications/imagenet_utils.py

@@ -1,7 +1,13 @@
+"""Utilities used by models pre-trained on ImageNet.
+"""
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
 import json
 
-from ..utils.data_utils import get_file
 from .. import backend as K
+from ..utils.data_utils import get_file
 
 CLASS_INDEX = None
 CLASS_INDEX_PATH = 'https://s3.amazonaws.com/deep-learning-models/image-models/imagenet_class_index.json'

keras/applications/inception_v3.py

@@ -10,28 +10,29 @@ and that the input preprocessing function is also different (same as Xception).
 - [Rethinking the Inception Architecture for Computer Vision](http://arxiv.org/abs/1512.00567)
 
 """
-from __future__ import print_function
 from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
 
 import warnings
 
-from ..models import Model
+from .. import backend as K
 from .. import layers
+from ..engine.topology import get_source_inputs
+from .imagenet_utils import _obtain_input_shape
+from .imagenet_utils import decode_predictions  # pylint: disable=unused-import
+from ..layers import AveragePooling2D
 from ..layers import Activation
-from ..layers import Dense
-from ..layers import Input
 from ..layers import BatchNormalization
 from ..layers import Conv2D
-from ..layers import MaxPooling2D
-from ..layers import AveragePooling2D
+from ..layers import Dense
 from ..layers import GlobalAveragePooling2D
 from ..layers import GlobalMaxPooling2D
-from ..engine.topology import get_source_inputs
-from ..utils.layer_utils import convert_all_kernels_in_model
+from ..layers import Input
+from ..layers import MaxPooling2D
+from ..models import Model
 from ..utils.data_utils import get_file
-from .. import backend as K
-from .imagenet_utils import decode_predictions
-from .imagenet_utils import _obtain_input_shape
+from ..utils.layer_utils import convert_all_kernels_in_model
 
 
 WEIGHTS_PATH = 'https://github.com/fchollet/deep-learning-models/releases/download/v0.5/inception_v3_weights_tf_dim_ordering_tf_kernels.h5'
@@ -381,8 +382,6 @@ def InceptionV3(include_top=True,
                 cache_subdir='models',
                 md5_hash='bcbd6486424b2319ff4ef7d526e38f63')
         model.load_weights(weights_path)
-        if K.backend() == 'theano':
-            convert_all_kernels_in_model(model)
     return model
 
 

keras/applications/resnet50.py

@@ -7,31 +7,32 @@
 
 Adapted from code contributed by BigMoyan.
 """
-from __future__ import print_function
 from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
 
 import warnings
 
-from ..layers import Input
+from .. import backend as K
 from .. import layers
-from ..layers import Dense
+from ..engine.topology import get_source_inputs
+from .imagenet_utils import _obtain_input_shape
+from .imagenet_utils import decode_predictions  # pylint: disable=unused-import
+from .imagenet_utils import preprocess_input  # pylint: disable=unused-import
 from ..layers import Activation
-from ..layers import Flatten
-from ..layers import Conv2D
-from ..layers import MaxPooling2D
-from ..layers import ZeroPadding2D
 from ..layers import AveragePooling2D
+from ..layers import BatchNormalization
+from ..layers import Conv2D
+from ..layers import Dense
+from ..layers import Flatten
 from ..layers import GlobalAveragePooling2D
 from ..layers import GlobalMaxPooling2D
-from ..layers import BatchNormalization
+from ..layers import Input
+from ..layers import MaxPooling2D
+from ..layers import ZeroPadding2D
 from ..models import Model
-from .. import backend as K
-from ..engine.topology import get_source_inputs
 from ..utils import layer_utils
 from ..utils.data_utils import get_file
-from .imagenet_utils import decode_predictions
-from .imagenet_utils import preprocess_input
-from .imagenet_utils import _obtain_input_shape
 
 
 WEIGHTS_PATH = 'https://github.com/fchollet/deep-learning-models/releases/download/v0.2/resnet50_weights_tf_dim_ordering_tf_kernels.h5'
@@ -77,7 +78,7 @@ def identity_block(input_tensor, kernel_size, filters, stage, block):
 
 
 def conv_block(input_tensor, kernel_size, filters, stage, block, strides=(2, 2)):
-    """conv_block is the block that has a conv layer at shortcut
+    """conv_block is the block that has a conv layer at shortcut.
 
     # Arguments
         input_tensor: input tensor
@@ -85,6 +86,7 @@ def conv_block(input_tensor, kernel_size, filters, stage, block, strides=(2, 2))
         filters: list of integers, the filterss of 3 conv layer at main path
         stage: integer, current stage label, used for generating layer names
         block: 'a','b'..., current block label, used for generating layer names
+        strides: Tuple of integers.
 
     # Returns
         Output tensor for the block.
@@ -140,8 +142,8 @@ def ResNet50(include_top=True, weights='imagenet',
     specified in your Keras config file.
 
     # Arguments
-        include_top: whether to include the 3 fully-connected
-            layers at the top of the network.
+        include_top: whether to include the fully-connected
+            layer at the top of the network.
         weights: one of `None` (random initialization)
             or "imagenet" (pre-training on ImageNet).
         input_tensor: optional Keras tensor (i.e. output of `layers.Input()`)
@@ -149,7 +151,7 @@ def ResNet50(include_top=True, weights='imagenet',
         input_shape: optional shape tuple, only to be specified
             if `include_top` is False (otherwise the input shape
             has to be `(224, 224, 3)` (with `channels_last` data format)
-            or `(3, 224, 244)` (with `channels_first` data format).
+            or `(3, 224, 224)` (with `channels_first` data format).
             It should have exactly 3 inputs channels,
             and width and height should be no smaller than 197.
             E.g. `(200, 200, 3)` would be one valid value.
@@ -194,10 +196,8 @@ def ResNet50(include_top=True, weights='imagenet',
     if input_tensor is None:
         img_input = Input(shape=input_shape)
     else:
-        if not K.is_keras_tensor(input_tensor):
-            img_input = Input(tensor=input_tensor, shape=input_shape)
-        else:
-            img_input = input_tensor
+        img_input = Input(tensor=input_tensor, shape=input_shape)
+
     if K.image_data_format() == 'channels_last':
         bn_axis = 3
     else:
@@ -264,21 +264,19 @@ def ResNet50(include_top=True, weights='imagenet',
         model.load_weights(weights_path)
         if K.backend() == 'theano':
             layer_utils.convert_all_kernels_in_model(model)
-
-        if K.image_data_format() == 'channels_first':
             if include_top:
                 maxpool = model.get_layer(name='avg_pool')
                 shape = maxpool.output_shape[1:]
                 dense = model.get_layer(name='fc1000')
                 layer_utils.convert_dense_weights_data_format(dense, shape, 'channels_first')
 
-            if K.backend() == 'tensorflow':
-                warnings.warn('You are using the TensorFlow backend, yet you '
-                              'are using the Theano '
-                              'image data format convention '
-                              '(`image_data_format="channels_first"`). '
-                              'For best performance, set '
-                              '`image_data_format="channels_last"` in '
-                              'your Keras config '
-                              'at ~/.keras/keras.json.')
+        if K.image_data_format() == 'channels_first' and K.backend() == 'tensorflow':
+            warnings.warn('You are using the TensorFlow backend, yet you '
+                          'are using the Theano '
+                          'image data format convention '
+                          '(`image_data_format="channels_first"`). '
+                          'For best performance, set '
+                          '`image_data_format="channels_last"` in '
+                          'your Keras config '
+                          'at ~/.keras/keras.json.')
     return model

keras/applications/vgg16.py

@@ -6,26 +6,27 @@
 - [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://arxiv.org/abs/1409.1556)
 
 """
-from __future__ import print_function
 from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
 
 import warnings
 
-from ..models import Model
-from ..layers import Flatten
-from ..layers import Dense
-from ..layers import Input
+from .. import backend as K
+from ..engine.topology import get_source_inputs
+from .imagenet_utils import _obtain_input_shape
+from .imagenet_utils import decode_predictions  # pylint: disable=unused-import
+from .imagenet_utils import preprocess_input  # pylint: disable=unused-import
 from ..layers import Conv2D
-from ..layers import MaxPooling2D
+from ..layers import Dense
+from ..layers import Flatten
 from ..layers import GlobalAveragePooling2D
 from ..layers import GlobalMaxPooling2D
-from ..engine.topology import get_source_inputs
+from ..layers import Input
+from ..layers import MaxPooling2D
+from ..models import Model
 from ..utils import layer_utils
 from ..utils.data_utils import get_file
-from .. import backend as K
-from .imagenet_utils import decode_predictions
-from .imagenet_utils import preprocess_input
-from .imagenet_utils import _obtain_input_shape
 
 
 WEIGHTS_PATH = 'https://github.com/fchollet/deep-learning-models/releases/download/v0.1/vgg16_weights_tf_dim_ordering_tf_kernels.h5'
@@ -59,7 +60,7 @@ def VGG16(include_top=True, weights='imagenet',
         input_shape: optional shape tuple, only to be specified
             if `include_top` is False (otherwise the input shape
             has to be `(224, 224, 3)` (with `channels_last` data format)
-            or `(3, 224, 244)` (with `channels_first` data format).
+            or `(3, 224, 224)` (with `channels_first` data format).
             It should have exactly 3 inputs channels,
             and width and height should be no smaller than 48.
             E.g. `(200, 200, 3)` would be one valid value.
@@ -103,10 +104,8 @@ def VGG16(include_top=True, weights='imagenet',
     if input_tensor is None:
         img_input = Input(shape=input_shape)
     else:
-        if not K.is_keras_tensor(input_tensor):
-            img_input = Input(tensor=input_tensor, shape=input_shape)
-        else:
-            img_input = input_tensor
+        img_input = Input(tensor=input_tensor, shape=input_shape)
+
     # Block 1
     x = Conv2D(64, (3, 3), activation='relu', padding='same', name='block1_conv1')(img_input)
     x = Conv2D(64, (3, 3), activation='relu', padding='same', name='block1_conv2')(x)

keras/applications/vgg19.py

@@ -6,26 +6,27 @@
 - [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://arxiv.org/abs/1409.1556)
 
 """
-from __future__ import print_function
 from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
 
 import warnings
 
-from ..models import Model
-from ..layers import Flatten
-from ..layers import Dense
-from ..layers import Input
+from .. import backend as K
+from ..engine.topology import get_source_inputs
+from .imagenet_utils import _obtain_input_shape
+from .imagenet_utils import decode_predictions  # pylint: disable=unused-import
+from .imagenet_utils import preprocess_input  # pylint: disable=unused-import
 from ..layers import Conv2D
-from ..layers import MaxPooling2D
+from ..layers import Dense
+from ..layers import Flatten
 from ..layers import GlobalAveragePooling2D
 from ..layers import GlobalMaxPooling2D
-from ..engine.topology import get_source_inputs
+from ..layers import Input
+from ..layers import MaxPooling2D
+from ..models import Model
 from ..utils import layer_utils
 from ..utils.data_utils import get_file
-from .. import backend as K
-from .imagenet_utils import decode_predictions
-from .imagenet_utils import preprocess_input
-from .imagenet_utils import _obtain_input_shape
 
 
 WEIGHTS_PATH = 'https://github.com/fchollet/deep-learning-models/releases/download/v0.1/vgg19_weights_tf_dim_ordering_tf_kernels.h5'
@@ -59,7 +60,7 @@ def VGG19(include_top=True, weights='imagenet',
         input_shape: optional shape tuple, only to be specified
             if `include_top` is False (otherwise the input shape
             has to be `(224, 224, 3)` (with `channels_last` data format)
-            or `(3, 224, 244)` (with `channels_first` data format).
+            or `(3, 224, 224)` (with `channels_first` data format).
             It should have exactly 3 inputs channels,
             and width and height should be no smaller than 48.
             E.g. `(200, 200, 3)` would be one valid value.
@@ -103,10 +104,8 @@ def VGG19(include_top=True, weights='imagenet',
     if input_tensor is None:
         img_input = Input(shape=input_shape)
     else:
-        if not K.is_keras_tensor(input_tensor):
-            img_input = Input(tensor=input_tensor, shape=input_shape)
-        else:
-            img_input = input_tensor
+        img_input = Input(tensor=input_tensor, shape=input_shape)
+
     # Block 1
     x = Conv2D(64, (3, 3), activation='relu', padding='same', name='block1_conv1')(img_input)
     x = Conv2D(64, (3, 3), activation='relu', padding='same', name='block1_conv2')(x)

keras/applications/xception.py

@@ -17,27 +17,28 @@ due to its reliance on `SeparableConvolution` layers.
 - [Xception: Deep Learning with Depthwise Separable Convolutions](https://arxiv.org/abs/1610.02357)
 
 """
-from __future__ import print_function
 from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
 
 import warnings
 
-from ..models import Model
+from .. import backend as K
 from .. import layers
-from ..layers import Dense
-from ..layers import Input
-from ..layers import BatchNormalization
+from ..engine.topology import get_source_inputs
+from .imagenet_utils import _obtain_input_shape
+from .imagenet_utils import decode_predictions  # pylint: disable=unused-import
 from ..layers import Activation
+from ..layers import BatchNormalization
 from ..layers import Conv2D
-from ..layers import SeparableConv2D
-from ..layers import MaxPooling2D
+from ..layers import Dense
 from ..layers import GlobalAveragePooling2D
 from ..layers import GlobalMaxPooling2D
-from ..engine.topology import get_source_inputs
+from ..layers import Input
+from ..layers import MaxPooling2D
+from ..layers import SeparableConv2D
+from ..models import Model
 from ..utils.data_utils import get_file
-from .. import backend as K
-from .imagenet_utils import decode_predictions
-from .imagenet_utils import _obtain_input_shape
 
 
 TF_WEIGHTS_PATH = 'https://github.com/fchollet/deep-learning-models/releases/download/v0.4/xception_weights_tf_dim_ordering_tf_kernels.h5'
@@ -133,10 +134,7 @@ def Xception(include_top=True, weights='imagenet',
     if input_tensor is None:
         img_input = Input(shape=input_shape)
     else:
-        if not K.is_keras_tensor(input_tensor):
-            img_input = Input(tensor=input_tensor, shape=input_shape)
-        else:
-            img_input = input_tensor
+        img_input = Input(tensor=input_tensor, shape=input_shape)
 
     x = Conv2D(32, (3, 3), strides=(2, 2), use_bias=False, name='block1_conv1')(img_input)
     x = BatchNormalization(name='block1_conv1_bn')(x)

keras/backend/__init__.py

@@ -1,73 +0,0 @@
-from __future__ import absolute_import
-from __future__ import print_function
-import os
-import json
-import sys
-from .common import epsilon
-from .common import floatx
-from .common import set_epsilon
-from .common import set_floatx
-from .common import cast_to_floatx
-from .common import image_data_format
-from .common import set_image_data_format
-from .common import is_keras_tensor
-
-_keras_base_dir = os.path.expanduser('~')
-if not os.access(_keras_base_dir, os.W_OK):
-    _keras_base_dir = '/tmp'
-
-_keras_dir = os.path.join(_keras_base_dir, '.keras')
-if not os.path.exists(_keras_dir):
-    os.makedirs(_keras_dir)
-
-# Default backend: TensorFlow.
-_BACKEND = 'tensorflow'
-
-_config_path = os.path.expanduser(os.path.join(_keras_dir, 'keras.json'))
-if os.path.exists(_config_path):
-    _config = json.load(open(_config_path))
-    _floatx = _config.get('floatx', floatx())
-    assert _floatx in {'float16', 'float32', 'float64'}
-    _epsilon = _config.get('epsilon', epsilon())
-    assert isinstance(_epsilon, float)
-    _backend = _config.get('backend', _BACKEND)
-    assert _backend in {'theano', 'tensorflow'}
-    _image_data_format = _config.get('image_data_format',
-                                     image_data_format())
-    assert _image_data_format in {'channels_last', 'channels_first'}
-
-    set_floatx(_floatx)
-    set_epsilon(_epsilon)
-    set_image_data_format(_image_data_format)
-    _BACKEND = _backend
-
-# save config file
-if not os.path.exists(_config_path):
-    _config = {'floatx': floatx(),
-               'epsilon': epsilon(),
-               'backend': _BACKEND,
-               'image_data_format': image_data_format()}
-    with open(_config_path, 'w') as f:
-        f.write(json.dumps(_config, indent=4))
-
-if 'KERAS_BACKEND' in os.environ:
-    _backend = os.environ['KERAS_BACKEND']
-    assert _backend in {'theano', 'tensorflow'}
-    _BACKEND = _backend
-
-# import backend
-if _BACKEND == 'theano':
-    sys.stderr.write('Using Theano backend.\n')
-    from .theano_backend import *
-elif _BACKEND == 'tensorflow':
-    sys.stderr.write('Using TensorFlow backend.\n')
-    from .tensorflow_backend import *
-else:
-    raise ValueError('Unknown backend: ' + str(_BACKEND))
-
-
-def backend():
-    """Publicly accessible method
-    for determining the current backend.
-    """
-    return _BACKEND

keras/backend/common.py

@@ -1,212 +0,0 @@
-import numpy as np
-
-# the type of float to use throughout the session.
-_FLOATX = 'float32'
-_EPSILON = 10e-8
-_IMAGE_DATA_FORMAT = 'channels_last'
-
-
-def epsilon():
-    """Returns the value of the fuzz
-    factor used in numeric expressions.
-
-    # Returns
-        A float.
-
-    # Example
-    ```python
-        >>> keras.backend.epsilon()
-        1e-08
-    ```
-    """
-    return _EPSILON
-
-
-def set_epsilon(e):
-    """Sets the value of the fuzz
-    factor used in numeric expressions.
-
-    # Arguments
-        e: float. New value of epsilon.
-
-    # Example
-    ```python
-        >>> from keras import backend as K
-        >>> K.epsilon()
-        1e-08
-        >>> K.set_epsilon(1e-05)
-        >>> K.epsilon()
-        1e-05
-    ```
-    """
-    global _EPSILON
-    _EPSILON = e
-
-
-def floatx():
-    """Returns the default float type, as a string
-    (e.g. 'float16', 'float32', 'float64').
-
-    # Returns
-        String, the current default float type.
-
-    # Example
-    ```python
-        >>> keras.backend.floatx()
-        'float32'
-    ```
-    """
-    return _FLOATX
-
-
-def set_floatx(floatx):
-    """Sets the default float type.
-
-    # Arguments
-        String: 'float16', 'float32', or 'float64'.
-
-    # Example
-    ```python
-        >>> from keras import backend as K
-        >>> K.floatx()
-        'float32'
-        >>> K.set_floatx('float16')
-        >>> K.floatx()
-        'float16'
-    ```
-    """
-    global _FLOATX
-    if floatx not in {'float16', 'float32', 'float64'}:
-        raise ValueError('Unknown floatx type: ' + str(floatx))
-    _FLOATX = str(floatx)
-
-
-def cast_to_floatx(x):
-    """Cast a Numpy array to the default Keras float type.
-
-    # Arguments
-        x: Numpy array.
-
-    # Returns
-        The same Numpy array, cast to its new type.
-
-    # Example
-    ```python
-        >>> from keras import backend as K
-        >>> K.floatx()
-        'float32'
-        >>> arr = numpy.array([1.0, 2.0], dtype='float64')
-        >>> arr.dtype
-        dtype('float64')
-        >>> new_arr = K.cast_to_floatx(arr)
-        >>> new_arr
-        array([ 1.,  2.], dtype=float32)
-        >>> new_arr.dtype
-        dtype('float32')
-    ```
-    """
-    return np.asarray(x, dtype=_FLOATX)
-
-
-def image_data_format():
-    """Returns the default image data format
-    convention ('channels_first' or 'channels_last').
-
-    # Returns
-        A string, either `'channels_first'` or `'channels_last'`
-
-    # Example
-    ```python
-        >>> keras.backend.image_data_format()
-        'channels_first'
-    ```
-    """
-    return _IMAGE_DATA_FORMAT
-
-
-def set_image_data_format(data_format):
-    """Sets the value of the data format convention.
-
-    # Arguments
-        data_format: string. `'channels_first'` or `'channels_last'`.
-
-    # Example
-    ```python
-        >>> from keras import backend as K
-        >>> K.image_data_format()
-        'channels_first'
-        >>> K.set_image_data_format('channels_last')
-        >>> K.image_data_format()
-        'channels_last'
-    ```
-    """
-    global _IMAGE_DATA_FORMAT
-    if data_format not in {'channels_last', 'channels_first'}:
-        raise ValueError('Unknown data_format:', data_format)
-    _IMAGE_DATA_FORMAT = str(data_format)
-
-
-def is_keras_tensor(x):
-    """Returns whether `x` is a Keras tensor.
-
-    # Arguments
-        x: a potential tensor.
-
-    # Returns
-        A boolean: whether the argument is a Keras tensor.
-
-    # Examples
-    ```python
-        >>> from keras import backend as K
-        >>> np_var = numpy.array([1, 2])
-        >>> K.is_keras_tensor(np_var)
-        False
-        >>> keras_var = K.variable(np_var)
-        >>> K.is_keras_tensor(keras_var)  # A variable is not a Tensor.
-        False
-        >>> keras_placeholder = K.placeholder(shape=(2, 4, 5))
-        >>> K.is_keras_tensor(keras_placeholder)  # A placeholder is a Tensor.
-        True
-    ```
-    """
-    if hasattr(x, '_keras_shape'):
-        return True
-    else:
-        return False
-
-
-# Legacy methods
-
-def set_image_dim_ordering(dim_ordering):
-    """Sets the value of the image data format.
-
-    # Arguments
-        data_format: string. `'channels_first'` or `'channels_last'`.
-
-    # Example
-    ```python
-        >>> from keras import backend as K
-        >>> K.image_data_format()
-        'channels_first'
-        >>> K.set_image_data_format('channels_last')
-        >>> K.image_data_format()
-        'channels_last'
-    ```
-    """
-    global _IMAGE_DATA_FORMAT
-    if dim_ordering not in {'tf', 'th'}:
-        raise ValueError('Unknown dim_ordering:', dim_ordering)
-    if dim_ordering == 'th':
-        data_format = 'channels_first'
-    else:
-        data_format = 'channels_last'
-    _IMAGE_DATA_FORMAT = data_format
-
-
-def image_dim_ordering():
-    """Legacy getter for data format.
-    """
-    if _IMAGE_DATA_FORMAT == 'channels_first':
-        return 'th'
-    else:
-        return 'tf'

keras/callbacks.py

@@ -1,27 +1,33 @@
+"""Keras callbacks: utilities called at certain points during model training.
+"""
 from __future__ import absolute_import
+from __future__ import division
 from __future__ import print_function
 
-import os
+from collections import deque
+from collections import Iterable
+from collections import OrderedDict
 import csv
-
-import numpy as np
-import time
+import six
 import json
+import os
+import time
 import warnings
 
-from collections import deque
-from collections import OrderedDict
-from collections import Iterable
-from .utils.generic_utils import Progbar
 from . import backend as K
+import numpy as np
+from tensorflow.python.summary import summary as tf_summary
+from tensorflow.contrib.tensorboard.plugins import projector
+from tensorflow.python.training import saver as saver_lib
+from tensorflow.python.ops import array_ops
+from .utils.generic_utils import Progbar
 
+# pylint: disable=g-import-not-at-top
 try:
     import requests
 except ImportError:
     requests = None
-
-if K.backend() == 'tensorflow':
-    import tensorflow as tf
+# pylint: enable=g-import-not-at-top
 
 
 class CallbackList(object):
@@ -225,6 +231,21 @@ class BaseLogger(Callback):
                     logs[k] = self.totals[k] / self.seen
 
 
+class TerminateOnNaN(Callback):
+    """Callback that terminates training when a NaN loss is encountered."""
+
+    def __init__(self):
+        super(TerminateOnNaN, self).__init__()
+
+    def on_batch_end(self, batch, logs=None):
+        logs = logs or {}
+        loss = logs.get('loss')
+        if loss is not None:
+            if np.isnan(loss) or np.isinf(loss):
+                print('Batch %d: Invalid loss, terminating training' % (batch))
+                self.model.stop_training = True
+
+
 class ProgbarLogger(Callback):
     """Callback that prints metrics to stdout.
 
@@ -465,7 +486,9 @@ class EarlyStopping(Callback):
             self.min_delta *= -1
 
     def on_train_begin(self, logs=None):
-        self.wait = 0  # Allow instances to be re-used
+        # Allow instances to be re-used
+        self.wait = 0
+        self.stopped_epoch = 0
         self.best = np.Inf if self.monitor_op == np.less else -np.Inf
 
     def on_epoch_end(self, epoch, logs=None):
@@ -502,8 +525,7 @@ class RemoteMonitor(Callback):
         field: String; JSON field under which the data will be stored.
         headers: Dictionary; optional custom HTTP headers.
             Defaults to:
-            `{'Accept': 'application/json',
-              'Content-Type': 'application/json'}`
+            `{'Accept': 'application/json', 'Content-Type': 'application/json'}`
     """
 
     def __init__(self,
@@ -562,6 +584,7 @@ class LearningRateScheduler(Callback):
 
 
 class TensorBoard(Callback):
+    # pylint: disable=line-too-long
     """Tensorboard basic visualizations.
 
     This callback writes a log for TensorBoard, which allows
@@ -577,85 +600,184 @@ class TensorBoard(Callback):
     tensorboard --logdir=/full_path_to_your_logs
     ```
     You can find more information about TensorBoard
-    [here](https://www.tensorflow.org/versions/master/how_tos/summaries_and_tensorboard/index.html).
+    [here](https://www.tensorflow.org/get_started/summaries_and_tensorboard).
 
     # Arguments
         log_dir: the path of the directory where to save the log
-            files to be parsed by Tensorboard
+            files to be parsed by TensorBoard.
         histogram_freq: frequency (in epochs) at which to compute activation
-            histograms for the layers of the model. If set to 0,
-            histograms won't be computed.
-        write_graph: whether to visualize the graph in Tensorboard.
+            and weight histograms for the layers of the model. If set to 0,
+            histograms won't be computed. Validation data (or split) must be
+            specified for histogram visualizations.
+        write_graph: whether to visualize the graph in TensorBoard.
             The log file can become quite large when
             write_graph is set to True.
+        write_grads: whether to visualize gradient histograms in TensorBoard.
+            `histogram_freq` must be greater than 0.
+        batch_size: size of batch of inputs to feed to the network
+            for histograms computation.
+        write_images: whether to write model weights to visualize as
+            image in TensorBoard.
+        embeddings_freq: frequency (in epochs) at which selected embedding
+            layers will be saved.
+        embeddings_layer_names: a list of names of layers to keep eye on. If
+            None or empty list all the embedding layer will be watched.
+        embeddings_metadata: a dictionary which maps layer name to a file name
+            in which metadata for this embedding layer is saved. See the
+            [details](https://www.tensorflow.org/how_tos/embedding_viz/#metadata_optional)
+            about metadata files format. In case if the same metadata file is
+            used for all embedding layers, string can be passed.
     """
+    # pylint: enable=line-too-long
 
     def __init__(self, log_dir='./logs',
                  histogram_freq=0,
+                 batch_size=32,
                  write_graph=True,
-                 write_images=False):
+                 write_grads=False,
+                 write_images=False,
+                 embeddings_freq=0,
+                 embeddings_layer_names=None,
+                 embeddings_metadata=None):
         super(TensorBoard, self).__init__()
-        if K.backend() != 'tensorflow':
-            raise RuntimeError('TensorBoard callback only works '
-                               'with the TensorFlow backend.')
         self.log_dir = log_dir
         self.histogram_freq = histogram_freq
         self.merged = None
         self.write_graph = write_graph
+        self.write_grads = write_grads
         self.write_images = write_images
+        self.embeddings_freq = embeddings_freq
+        self.embeddings_layer_names = embeddings_layer_names
+        self.embeddings_metadata = embeddings_metadata or {}
+        self.batch_size = batch_size
 
     def set_model(self, model):
         self.model = model
         self.sess = K.get_session()
         if self.histogram_freq and self.merged is None:
             for layer in self.model.layers:
-
                 for weight in layer.weights:
-                    tf.summary.histogram(weight.name, weight)
+                    tf_summary.histogram(weight.name, weight)
+                    if self.write_grads:
+                        grads = model.optimizer.get_gradients(model.total_loss,
+                                                              weight)
+                        tf_summary.histogram('{}_grad'.format(weight.name), grads)
                     if self.write_images:
-                        w_img = tf.squeeze(weight)
-                        shape = w_img.get_shape()
-                        if len(shape) > 1 and shape[0] > shape[1]:
-                            w_img = tf.transpose(w_img)
-                        if len(shape) == 1:
-                            w_img = tf.expand_dims(w_img, 0)
-                        w_img = tf.expand_dims(tf.expand_dims(w_img, 0), -1)
-                        tf.summary.image(weight.name, w_img)
+                        w_img = array_ops.squeeze(weight)
+                        shape = K.int_shape(w_img)
+                        if len(shape) == 2:  # dense layer kernel case
+                            if shape[0] > shape[1]:
+                                w_img = array_ops.transpose(w_img)
+                                shape = K.int_shape(w_img)
+                            w_img = array_ops.reshape(w_img, [1,
+                                                      shape[0],
+                                                      shape[1],
+                                                      1])
+                        elif len(shape) == 3:  # convnet case
+                            if K.image_data_format() == 'channels_last':
+                                # switch to channels_first to display
+                                # every kernel as a separate image
+                                w_img = array_ops.transpose(w_img, perm=[2, 0, 1])
+                                shape = K.int_shape(w_img)
+                            w_img = array_ops.reshape(
+                                w_img, [shape[0], shape[1], shape[2], 1])
+                        elif len(shape) == 1:  # bias case
+                            w_img = array_ops.reshape(
+                                w_img, [1, shape[0], 1, 1])
+                        else:
+                            # not possible to handle 3D convnets etc.
+                            continue
+
+                        shape = K.int_shape(w_img)
+                        assert len(shape) == 4 and shape[-1] in [1, 3, 4]
+                        tf_summary.image(weight.name, w_img)
 
                 if hasattr(layer, 'output'):
-                    tf.summary.histogram('{}_out'.format(layer.name),
+                    tf_summary.histogram('{}_out'.format(layer.name),
                                          layer.output)
-        self.merged = tf.summary.merge_all()
+        self.merged = tf_summary.merge_all()
 
         if self.write_graph:
-            self.writer = tf.summary.FileWriter(self.log_dir,
+            self.writer = tf_summary.FileWriter(self.log_dir,
                                                 self.sess.graph)
         else:
-            self.writer = tf.summary.FileWriter(self.log_dir)
+            self.writer = tf_summary.FileWriter(self.log_dir)
+
+        if self.embeddings_freq:
+            embeddings_layer_names = self.embeddings_layer_names
+
+            if not embeddings_layer_names:
+                embeddings_layer_names = [layer.name for layer in self.model.layers
+                                          if type(layer).__name__ == 'Embedding']
+
+            embeddings = {layer.name: layer.weights[0]
+                          for layer in self.model.layers
+                          if layer.name in embeddings_layer_names}
+
+            self.saver = saver_lib.Saver(list(embeddings.values()))
+
+            embeddings_metadata = {}
+
+            if not isinstance(self.embeddings_metadata, str):
+                embeddings_metadata = self.embeddings_metadata
+            else:
+                embeddings_metadata = {layer_name: self.embeddings_metadata
+                                       for layer_name in embeddings.keys()}
+
+            config = projector.ProjectorConfig()
+            self.embeddings_ckpt_path = os.path.join(self.log_dir,
+                                                     'keras_embedding.ckpt')
+
+            for layer_name, tensor in embeddings.items():
+                embedding = config.embeddings.add()
+                embedding.tensor_name = tensor.name
+
+                if layer_name in embeddings_metadata:
+                    embedding.metadata_path = embeddings_metadata[layer_name]
+
+            projector.visualize_embeddings(self.writer, config)
 
     def on_epoch_end(self, epoch, logs=None):
         logs = logs or {}
 
         if self.validation_data and self.histogram_freq:
             if epoch % self.histogram_freq == 0:
-                # TODO: implement batched calls to sess.run
-                # (current call will likely go OOM on GPU)
+
+                val_data = self.validation_data
+                tensors = (self.model.inputs +
+                           self.model.targets +
+                           self.model.sample_weights)
+
                 if self.model.uses_learning_phase:
-                    cut_v_data = len(self.model.inputs)
-                    val_data = self.validation_data[:cut_v_data] + [0]
-                    tensors = self.model.inputs + [K.learning_phase()]
-                else:
-                    val_data = self.validation_data
-                    tensors = self.model.inputs
-                feed_dict = dict(zip(tensors, val_data))
-                result = self.sess.run([self.merged], feed_dict=feed_dict)
-                summary_str = result[0]
-                self.writer.add_summary(summary_str, epoch)
+                    tensors += [K.learning_phase()]
+
+                assert len(val_data) == len(tensors)
+                val_size = val_data[0].shape[0]
+                i = 0
+                while i < val_size:
+                    step = min(self.batch_size, val_size - i)
+                    batch_val = []
+                    batch_val.append(val_data[0][i:i + step])
+                    batch_val.append(val_data[1][i:i + step])
+                    batch_val.append(val_data[2][i:i + step])
+                    if self.model.uses_learning_phase:
+                        batch_val.append(val_data[3])
+                    feed_dict = dict(zip(tensors, batch_val))
+                    result = self.sess.run([self.merged], feed_dict=feed_dict)
+                    summary_str = result[0]
+                    self.writer.add_summary(summary_str, epoch)
+                    i += self.batch_size
+
+        if self.embeddings_freq and self.embeddings_ckpt_path:
+            if epoch % self.embeddings_freq == 0:
+                self.saver.save(self.sess,
+                                self.embeddings_ckpt_path,
+                                epoch)
 
         for name, value in logs.items():
             if name in ['batch', 'size']:
                 continue
-            summary = tf.Summary()
+            summary = tf_summary.Summary()
             summary_value = summary.value.add()
             summary_value.simple_value = value.item()
             summary_value.tag = name
@@ -676,9 +798,9 @@ class ReduceLROnPlateau(Callback):
 
     # Example
         ```python
-            reduce_lr = ReduceLROnPlateau(monitor='val_loss', factor=0.2,
-                                          patience=5, min_lr=0.001)
-            model.fit(X_train, Y_train, callbacks=[reduce_lr])
+        reduce_lr = ReduceLROnPlateau(monitor='val_loss', factor=0.2,
+                                      patience=5, min_lr=0.001)
+        model.fit(X_train, Y_train, callbacks=[reduce_lr])
         ```
 
     # Arguments
@@ -785,8 +907,8 @@ class CSVLogger(Callback):
 
     # Example
         ```python
-            csv_logger = CSVLogger('training.log')
-            model.fit(X_train, Y_train, callbacks=[csv_logger])
+        csv_logger = CSVLogger('training.log')
+        model.fit(X_train, Y_train, callbacks=[csv_logger])
         ```
 
     # Arguments
@@ -803,23 +925,26 @@ class CSVLogger(Callback):
         self.writer = None
         self.keys = None
         self.append_header = True
+        self.file_flags = 'b' if six.PY2 and os.name == 'nt' else ''
         super(CSVLogger, self).__init__()
 
     def on_train_begin(self, logs=None):
         if self.append:
             if os.path.exists(self.filename):
-                with open(self.filename) as f:
+                with open(self.filename, 'r' + self.file_flags) as f:
                     self.append_header = not bool(len(f.readline()))
-            self.csv_file = open(self.filename, 'a')
+            self.csv_file = open(self.filename, 'a' + self.file_flags)
         else:
-            self.csv_file = open(self.filename, 'w')
+            self.csv_file = open(self.filename, 'w' + self.file_flags)
 
     def on_epoch_end(self, epoch, logs=None):
         logs = logs or {}
 
         def handle_value(k):
             is_zero_dim_ndarray = isinstance(k, np.ndarray) and k.ndim == 0
-            if isinstance(k, Iterable) and not is_zero_dim_ndarray:
+            if isinstance(k, six.string_types):
+                return k
+            elif isinstance(k, Iterable) and not is_zero_dim_ndarray:
                 return '"[%s]"' % (', '.join(map(str, k)))
             else:
                 return k
@@ -851,6 +976,7 @@ class LambdaCallback(Callback):
     This callback is constructed with anonymous functions that will be called
     at the appropriate time. Note that the callbacks expects positional
     arguments, as:
+
      - `on_epoch_begin` and `on_epoch_end` expect two positional arguments:
         `epoch`, `logs`
      - `on_batch_begin` and `on_batch_end` expect two positional arguments:

keras/constraints.py

@@ -1,8 +1,13 @@
+"""Constraints: functions that impose constraints on weights values.
+"""
 from __future__ import absolute_import
-import six
+from __future__ import division
+from __future__ import print_function
+
 from . import backend as K
-from .utils.generic_utils import serialize_keras_object
+import six
 from .utils.generic_utils import deserialize_keras_object
+from .utils.generic_utils import serialize_keras_object
 
 
 class Constraint(object):
@@ -142,16 +147,12 @@ class MinMaxNorm(Constraint):
 
 # Aliases.
 
+# pylint: disable=invalid-name
 max_norm = MaxNorm
 non_neg = NonNeg
 unit_norm = UnitNorm
 min_max_norm = MinMaxNorm
-
-
-# Legacy aliases.
-maxnorm = max_norm
-nonneg = non_neg
-unitnorm = unit_norm
+# pylint: enable=invalid-name
 
 
 def serialize(constraint):

keras/datasets/__init__.py

@@ -1,8 +1,13 @@
+"""Keras datasets: utilities for downloading and pre-processing common datasets.
+"""
 from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
 
-from . import mnist
-from . import imdb
-from . import reuters
+from . import boston_housing
 from . import cifar10
 from . import cifar100
-from . import boston_housing
+from . import imdb
+from . import mnist
+from . import reuters
+

keras/datasets/boston_housing.py

@@ -1,5 +1,11 @@
-from ..utils.data_utils import get_file
+"""Boston housing price regression dataset.
+"""
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
 import numpy as np
+from ..utils.data_utils import get_file
 
 
 def load_data(path='boston_housing.npz', seed=113, test_split=0.2):

keras/datasets/cifar.py

@@ -1,5 +1,10 @@
+"""Utilities used by the CIFAR10 and CIFAR100 datasets.
+"""
 # -*- coding: utf-8 -*-
 from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
 import sys
 from six.moves import cPickle
 

keras/datasets/cifar10.py

@@ -1,10 +1,16 @@
+"""CIFAR10 small image classification dataset.
+"""
 from __future__ import absolute_import
-from .cifar import load_batch
-from ..utils.data_utils import get_file
-from .. import backend as K
-import numpy as np
+from __future__ import division
+from __future__ import print_function
+
 import os
 
+from .. import backend as K
+from .cifar import load_batch
+import numpy as np
+from ..utils.data_utils import get_file
+
 
 def load_data():
     """Loads CIFAR10 dataset.

keras/datasets/cifar100.py

@@ -1,10 +1,16 @@
+"""CIFAR100 small image classification dataset.
+"""
 from __future__ import absolute_import
-from .cifar import load_batch
-from ..utils.data_utils import get_file
-from .. import backend as K
-import numpy as np
+from __future__ import division
+from __future__ import print_function
+
 import os
 
+from .. import backend as K
+from .cifar import load_batch
+import numpy as np
+from ..utils.data_utils import get_file
+
 
 def load_data(label_mode='fine'):
     """Loads CIFAR100 dataset.

keras/datasets/imdb.py

@@ -1,14 +1,19 @@
+"""IMDB movie review sentiment classification dataset.
+"""
 from __future__ import absolute_import
-from ..utils.data_utils import get_file
-from six.moves import zip
-import numpy as np
+from __future__ import division
+from __future__ import print_function
+
 import json
-import warnings
+
+import numpy as np
+from six.moves import zip
+from ..utils.data_utils import get_file
 
 
 def load_data(path='imdb.npz', num_words=None, skip_top=0,
               maxlen=None, seed=113,
-              start_char=1, oov_char=2, index_from=3, **kwargs):
+              start_char=1, oov_char=2, index_from=3):
     """Loads the IMDB dataset.
 
     # Arguments
@@ -39,14 +44,6 @@ def load_data(path='imdb.npz', num_words=None, skip_top=0,
     Words that were not seen in the training set but are in the test set
     have simply been skipped.
     """
-    # Legacy support
-    if 'nb_words' in kwargs:
-        warnings.warn('The `nb_words` argument in `load_data` '
-                      'has been renamed `num_words`.')
-        num_words = kwargs.pop('nb_words')
-    if kwargs:
-        raise TypeError('Unrecognized keyword arguments: ' + str(kwargs))
-
     path = get_file(path,
                     origin='https://s3.amazonaws.com/text-datasets/imdb.npz')
     f = np.load(path)
@@ -83,10 +80,10 @@ def load_data(path='imdb.npz', num_words=None, skip_top=0,
                 new_labels.append(y)
         xs = new_xs
         labels = new_labels
-    if not xs:
-        raise ValueError('After filtering for sequences shorter than maxlen=' +
-                         str(maxlen) + ', no sequence was kept. '
-                         'Increase maxlen.')
+        if not xs:
+            raise ValueError('After filtering for sequences shorter than maxlen=' +
+                             str(maxlen) + ', no sequence was kept. '
+                             'Increase maxlen.')
     if not num_words:
         num_words = max([max(x) for x in xs])
 
@@ -100,7 +97,7 @@ def load_data(path='imdb.npz', num_words=None, skip_top=0,
         for x in xs:
             nx = []
             for w in x:
-                if w >= num_words or w < skip_top:
+                if skip_top <= w < num_words:
                     nx.append(w)
             new_xs.append(nx)
         xs = new_xs

keras/datasets/mnist.py

@@ -1,5 +1,11 @@
-from ..utils.data_utils import get_file
+"""MNIST handwritten digits classification dataset.
+"""
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
 import numpy as np
+from ..utils.data_utils import get_file
 
 
 def load_data(path='mnist.npz'):

keras/datasets/reuters.py

@@ -1,15 +1,20 @@
+"""Reuters newswire topic classification dataset.
+"""
 # -*- coding: utf-8 -*-
 from __future__ import absolute_import
-from ..utils.data_utils import get_file
-from six.moves import zip
-import numpy as np
+from __future__ import division
+from __future__ import print_function
+
 import json
-import warnings
+
+import numpy as np
+from six.moves import zip
+from ..utils.data_utils import get_file
 
 
 def load_data(path='reuters.npz', num_words=None, skip_top=0,
               maxlen=None, test_split=0.2, seed=113,
-              start_char=1, oov_char=2, index_from=3, **kwargs):
+              start_char=1, oov_char=2, index_from=3):
     """Loads the Reuters newswire classification dataset.
 
     # Arguments
@@ -37,14 +42,6 @@ def load_data(path='reuters.npz', num_words=None, skip_top=0,
     Words that were not seen in the training set but are in the test set
     have simply been skipped.
     """
-    # Legacy support
-    if 'nb_words' in kwargs:
-        warnings.warn('The `nb_words` argument in `load_data` '
-                      'has been renamed `num_words`.')
-        num_words = kwargs.pop('nb_words')
-    if kwargs:
-        raise TypeError('Unrecognized keyword arguments: ' + str(kwargs))
-
     path = get_file(path, origin='https://s3.amazonaws.com/text-datasets/reuters.npz')
     npzfile = np.load(path)
     xs = npzfile['x']
@@ -84,7 +81,7 @@ def load_data(path='reuters.npz', num_words=None, skip_top=0,
         for x in xs:
             nx = []
             for w in x:
-                if w >= num_words or w < skip_top:
+                if skip_top <= w < num_words:
                     nx.append(w)
             new_xs.append(nx)
         xs = new_xs

keras/engine/__init__.py

@@ -1,8 +1,14 @@
-# note: topology.Node is an internal class,
+"""The Keras Engine: graph topology and training loop functionality.
+"""
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+# Note: topology.Node is an internal class,
 # it isn't meant to be used by Keras users.
-from .topology import InputSpec
+from .topology import get_source_inputs
 from .topology import Input
 from .topology import InputLayer
+from .topology import InputSpec
 from .topology import Layer
-from .topology import get_source_inputs
 from .training import Model

keras/engine/training.py

@@ -1,28 +1,32 @@
+"""Keras training and evaluation routines.
+"""
 # -*- coding: utf-8 -*-
-from __future__ import print_function
 from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
 
-import warnings
 import copy
-import time
-import numpy as np
 import multiprocessing
 import threading
-import six
+import time
+import warnings
 
+from .. import backend as K
+from .. import callbacks as cbks
+from .. import losses
+from .. import metrics as metrics_module
+from .. import optimizers
+import numpy as np
+import six
+from .topology import Container
+from ..utils.generic_utils import Progbar
+
+# pylint: disable=g-import-not-at-top
 try:
     import queue
 except ImportError:
     import Queue as queue
-
-from .topology import Container
-from .. import backend as K
-from .. import optimizers
-from .. import losses
-from .. import metrics as metrics_module
-from ..utils.generic_utils import Progbar
-from .. import callbacks as cbks
-from ..legacy import interfaces
+# pylint: enable=g-import-not-at-top
 
 
 def _standardize_input_data(data, names, shapes=None,
@@ -50,6 +54,8 @@ def _standardize_input_data(data, names, shapes=None,
     # Raises
         ValueError: in case of improperly formatted user-provided data.
     """
+    if not names:
+        return []
     if data is None:
         return [None for _ in range(len(names))]
     if isinstance(data, dict):
@@ -63,7 +69,8 @@ def _standardize_input_data(data, names, shapes=None,
     elif isinstance(data, list):
         if len(data) != len(names):
             if data and hasattr(data[0], 'shape'):
-                raise ValueError('Error when checking ' + exception_prefix +
+                raise ValueError('Error when checking model ' +
+                                 exception_prefix +
                                  ': the list of Numpy arrays '
                                  'that you are passing to your model '
                                  'is not the size the model expected. '
@@ -77,7 +84,8 @@ def _standardize_input_data(data, names, shapes=None,
                     data = [np.asarray(data)]
                 else:
                     raise ValueError(
-                        'Error when checking ' + exception_prefix +
+                        'Error when checking model ' +
+                        exception_prefix +
                         ': you are passing a list as '
                         'input to your model, '
                         'but the model expects '
@@ -88,15 +96,17 @@ def _standardize_input_data(data, names, shapes=None,
         arrays = data
     else:
         if not hasattr(data, 'shape'):
-            raise TypeError('Error when checking ' + exception_prefix +
+            raise TypeError('Error when checking model ' +
+                            exception_prefix +
                             ': data should be a Numpy array, '
                             'or list/dict of Numpy arrays. '
                             'Found: ' + str(data)[:200] + '...')
-        if len(names) != 1:
+        if len(names) > 1:
             # Case: model expects multiple inputs but only received
             # a single Numpy array.
             raise ValueError('The model expects ' + str(len(names)) +
-                             ' input arrays, but only received one array. '
+                             exception_prefix +
+                             ' arrays, but only received one array. '
                              'Found: array with shape ' + str(data.shape))
         arrays = [data]
 
@@ -149,7 +159,7 @@ def _standardize_sample_or_class_weights(x_weight, output_names, weight_type):
     # Raises
         ValueError: In case of invalid user-provided argument.
     """
-    if x_weight is None or len(x_weight) == 0:
+    if x_weight is None or len(x_weight) == 0:  # pylint: disable=g-explicit-length-test
         return [None for _ in output_names]
     if len(output_names) == 1:
         if isinstance(x_weight, list) and len(x_weight) == 1:
@@ -429,7 +439,6 @@ def _weighted_masked_objective(fn):
         # score_array has ndim >= 2
         score_array = fn(y_true, y_pred)
         if mask is not None:
-            # Cast the mask to floatX to avoid float64 upcasting in theano
             mask = K.cast(mask, K.floatx())
             # mask should have the same shape as score_array
             score_array *= mask
@@ -478,7 +487,6 @@ def _masked_objective(fn):
         # score_array has ndim >= 2
         score_array = fn(y_true, y_pred)
         if mask is not None:
-            # Cast the mask to floatX to avoid float64 upcasting in theano
             mask = K.cast(mask, K.floatx())
             # mask should have the same shape as score_array
             score_array *= mask
@@ -679,6 +687,8 @@ class Model(Container):
                 See [losses](/losses).
                 If the model has multiple outputs, you can use a different loss
                 on each output by passing a dictionary or a list of losses.
+                The loss value that will be minimized by the model
+                will then be the sum of all individual losses.
             metrics: list of metrics to be evaluated by the model
                 during training and testing.
                 Typically you will use `metrics=['accuracy']`.
@@ -688,6 +698,9 @@ class Model(Container):
             loss_weights: Optional list or dictionary specifying scalar
                 coefficients (Python floats) to weight the loss contributions
                 of different model outputs.
+                The loss value that will be minimized by the model
+                will then be the *weighted sum* of all individual losses,
+                weighted by the `loss_weights` coefficients.
                 If a list, it is expected to have a 1:1 mapping
                 to the model's outputs. If a tensor, it is expected to map
                 output names (strings) to scalar coefficients.
@@ -697,12 +710,12 @@ class Model(Container):
                 If the model has multiple outputs, you can use a different
                 `sample_weight_mode` on each output by passing a
                 dictionary or a list of modes.
-            **kwargs: when using the Theano backend, these arguments
-                are passed into K.function. Ignored for Tensorflow backend.
+            **kwargs: Additional arguments passed to `tf.Session.run`.
 
         # Raises
             ValueError: In case of invalid arguments for
                 `optimizer`, `loss`, `metrics` or `sample_weight_mode`.
+            RuntimeError: If the model has no loss to optimize.
         """
         loss = loss or {}
         self.optimizer = optimizers.get(optimizer)
@@ -726,7 +739,7 @@ class Model(Container):
                                   'We assume this was done on purpose, '
                                   'and we will not be expecting '
                                   'any data to be passed to "' + name +
-                                  '" during training.')
+                                  '" during training.', stacklevel=2)
                 loss_functions.append(losses.get(loss.get(name)))
         elif isinstance(loss, list):
             if len(loss) != len(self.outputs):
@@ -939,7 +952,8 @@ class Model(Container):
                     # (because of class mode duality)
                     output_shape = self.internal_output_shapes[i]
                     acc_fn = None
-                    if output_shape[-1] == 1 or self.loss_functions[i] == losses.binary_crossentropy:
+                    if (output_shape[-1] == 1 or
+                       self.loss_functions[i] == losses.binary_crossentropy):
                         # case: binary accuracy
                         acc_fn = metrics_module.binary_accuracy
                     elif self.loss_functions[i] == losses.sparse_categorical_crossentropy:
@@ -971,27 +985,25 @@ class Model(Container):
         # Functions for train, test and predict will
         # be compiled lazily when required.
         # This saves time when the user is not using all functions.
-        self._function_kwargs = kwargs
-
         self.train_function = None
         self.test_function = None
         self.predict_function = None
+        self._function_kwargs = kwargs
 
         # Collected trainable weights and sort them deterministically.
         trainable_weights = self.trainable_weights
         # Sort weights by name.
         if trainable_weights:
-            if K.backend() == 'theano':
-                trainable_weights.sort(key=lambda x: x.name if x.name else x.auto_name)
-            else:
-                trainable_weights.sort(key=lambda x: x.name)
+            trainable_weights.sort(key=lambda x: x.name)
         self._collected_trainable_weights = trainable_weights
 
     def _make_train_function(self):
         if not hasattr(self, 'train_function'):
             raise RuntimeError('You must compile your model before using it.')
         if self.train_function is None:
-            inputs = self._feed_inputs + self._feed_targets + self._feed_sample_weights
+            inputs = (self._feed_inputs +
+                      self._feed_targets +
+                      self._feed_sample_weights)
             if self.uses_learning_phase and not isinstance(K.learning_phase(), int):
                 inputs += [K.learning_phase()]
 
@@ -1004,13 +1016,16 @@ class Model(Container):
             self.train_function = K.function(inputs,
                                              [self.total_loss] + self.metrics_tensors,
                                              updates=updates,
+                                             name='train_function',
                                              **self._function_kwargs)
 
     def _make_test_function(self):
         if not hasattr(self, 'test_function'):
             raise RuntimeError('You must compile your model before using it.')
         if self.test_function is None:
-            inputs = self._feed_inputs + self._feed_targets + self._feed_sample_weights
+            inputs = (self._feed_inputs +
+                      self._feed_targets +
+                      self._feed_sample_weights)
             if self.uses_learning_phase and not isinstance(K.learning_phase(), int):
                 inputs += [K.learning_phase()]
             # Return loss and metrics, no gradient updates.
@@ -1018,11 +1033,13 @@ class Model(Container):
             self.test_function = K.function(inputs,
                                             [self.total_loss] + self.metrics_tensors,
                                             updates=self.state_updates,
+                                            name='test_function',
                                             **self._function_kwargs)
 
     def _make_predict_function(self):
         if not hasattr(self, 'predict_function'):
             self.predict_function = None
+            self._function_kwargs = {}
         if self.predict_function is None:
             if self.uses_learning_phase and not isinstance(K.learning_phase(), int):
                 inputs = self._feed_inputs + [K.learning_phase()]
@@ -1030,11 +1047,11 @@ class Model(Container):
                 inputs = self._feed_inputs
             # Gets network outputs. Does not update weights.
             # Does update the network states.
-            kwargs = getattr(self, '_function_kwargs', {})
             self.predict_function = K.function(inputs,
                                                self.outputs,
                                                updates=self.state_updates,
-                                               **kwargs)
+                                               name='predict_function',
+                                               **self._function_kwargs)
 
     def _fit_loop(self, f, ins, out_labels=None, batch_size=32,
                   epochs=100, verbose=1, callbacks=None,
@@ -1125,7 +1142,7 @@ class Model(Container):
                 batch_ids = index_array[batch_start:batch_end]
                 try:
                     if isinstance(ins[-1], float):
-                        # do not slice the training phase flag
+                        # Do not slice the training phase flag.
                         ins_batch = _slice_arrays(ins[:-1], batch_ids) + [ins[-1]]
                     else:
                         ins_batch = _slice_arrays(ins, batch_ids)
@@ -1144,17 +1161,17 @@ class Model(Container):
                     batch_logs[l] = o
 
                 callbacks.on_batch_end(batch_index, batch_logs)
+                if callback_model.stop_training:
+                    break
 
-                if batch_index == len(batches) - 1:  # last batch
-                    # validation
+                if batch_index == len(batches) - 1:  # Last batch.
                     if do_validation:
-                        # replace with self._evaluate
                         val_outs = self._test_loop(val_f, val_ins,
                                                    batch_size=batch_size,
                                                    verbose=0)
                         if not isinstance(val_outs, list):
                             val_outs = [val_outs]
-                        # same labels assumed
+                        # Same labels assumed.
                         for l, o in zip(out_labels, val_outs):
                             epoch_logs['val_' + l] = o
             callbacks.on_epoch_end(epoch, epoch_logs)
@@ -1194,7 +1211,7 @@ class Model(Container):
         for batch_index, (batch_start, batch_end) in enumerate(batches):
             batch_ids = index_array[batch_start:batch_end]
             if ins and isinstance(ins[-1], float):
-                # do not slice the training phase flag
+                # Do not slice the training phase flag.
                 ins_batch = _slice_arrays(ins[:-1], batch_ids) + [ins[-1]]
             else:
                 ins_batch = _slice_arrays(ins, batch_ids)
@@ -1205,7 +1222,7 @@ class Model(Container):
             if batch_index == 0:
                 for batch_out in batch_outs:
                     shape = (samples,) + batch_out.shape[1:]
-                    outs.append(np.zeros(shape, dtype=K.floatx()))
+                    outs.append(np.zeros(shape, dtype=batch_out.dtype))
 
             for i, batch_out in enumerate(batch_outs):
                 outs[i][batch_start:batch_end] = batch_out
@@ -1248,7 +1265,7 @@ class Model(Container):
         for batch_index, (batch_start, batch_end) in enumerate(batches):
             batch_ids = index_array[batch_start:batch_end]
             if isinstance(ins[-1], float):
-                # do not slice the training phase flag
+                # Do not slice the training phase flag.
                 ins_batch = _slice_arrays(ins[:-1], batch_ids) + [ins[-1]]
             else:
                 ins_batch = _slice_arrays(ins, batch_ids)
@@ -1292,11 +1309,11 @@ class Model(Container):
         x = _standardize_input_data(x, self._feed_input_names,
                                     self._feed_input_shapes,
                                     check_batch_axis=False,
-                                    exception_prefix='model input')
+                                    exception_prefix='input')
         y = _standardize_input_data(y, self._feed_output_names,
                                     output_shapes,
                                     check_batch_axis=False,
-                                    exception_prefix='model target')
+                                    exception_prefix='target')
         sample_weights = _standardize_sample_weights(sample_weight,
                                                      self._feed_output_names)
         class_weights = _standardize_class_weights(class_weight,
@@ -1317,6 +1334,20 @@ class Model(Container):
                                  str(x[0].shape[0]) + ' samples')
         return x, y, sample_weights
 
+    def _get_deduped_metrics_names(self):
+        out_labels = self.metrics_names
+
+        # Rename duplicated metrics name
+        # (can happen with an output layer shared among multiple dataflows).
+        deduped_out_labels = []
+        for i, label in enumerate(out_labels):
+            new_label = label
+            if out_labels.count(label) > 1:
+                dup_idx = out_labels[:i].count(label)
+                new_label += '_' + str(dup_idx + 1)
+            deduped_out_labels.append(new_label)
+        return deduped_out_labels
+
     def fit(self, x=None,
             y=None,
             batch_size=32,
@@ -1328,8 +1359,7 @@ class Model(Container):
             shuffle=True,
             class_weight=None,
             sample_weight=None,
-            initial_epoch=0,
-            **kwargs):
+            initial_epoch=0):
         """Trains the model for a fixed number of epochs (iterations on a dataset).
 
         # Arguments
@@ -1346,7 +1376,7 @@ class Model(Container):
             batch_size: integer. Number of samples per gradient update.
             epochs: integer, the number of times to iterate
                 over the training data arrays.
-                verbose: 0, 1, or 2. Verbosity mode.
+            verbose: 0, 1, or 2. Verbosity mode.
                 0 = silent, 1 = verbose, 2 = one log line per epoch.
             callbacks: list of callbacks to be called during training.
                 See [callbacks](/callbacks).
@@ -1388,29 +1418,21 @@ class Model(Container):
             ValueError: In case of mismatch between the provided input data
                 and what the model expects.
         """
-        # Legacy support
-        if 'nb_epoch' in kwargs:
-            warnings.warn('The `nb_epoch` argument in `fit` '
-                          'has been renamed `epochs`.')
-            epochs = kwargs.pop('nb_epoch')
-        if kwargs:
-            raise TypeError('Unrecognized keyword arguments: ' + str(kwargs))
-
-        # validate user data
+        # Validate user data.
         x, y, sample_weights = self._standardize_user_data(
             x, y,
             sample_weight=sample_weight,
             class_weight=class_weight,
             check_batch_axis=False,
             batch_size=batch_size)
-        # prepare validation data
+        # Prepare validation data.
         if validation_data:
             do_validation = True
             if len(validation_data) == 2:
-                val_x, val_y = validation_data
+                val_x, val_y = validation_data  # pylint: disable=unpacking-non-sequence
                 val_sample_weight = None
             elif len(validation_data) == 3:
-                val_x, val_y, val_sample_weight = validation_data
+                val_x, val_y, val_sample_weight = validation_data  # pylint: disable=unpacking-non-sequence
             else:
                 raise ValueError('When passing validation_data, '
                                  'it must contain 2 (x_val, y_val) '
@@ -1449,7 +1471,7 @@ class Model(Container):
             val_f = None
             val_ins = None
 
-        # prepare input arrays and training function
+        # Prepare input arrays and training function.
         if self.uses_learning_phase and not isinstance(K.learning_phase(), int):
             ins = x + y + sample_weights + [1.]
         else:
@@ -1457,26 +1479,15 @@ class Model(Container):
         self._make_train_function()
         f = self.train_function
 
-        # prepare display labels
-        out_labels = self.metrics_names
-
-        # rename duplicated metrics name
-        # (can happen with an output layer shared among multiple dataflows)
-        deduped_out_labels = []
-        for i, label in enumerate(out_labels):
-            new_label = label
-            if out_labels.count(label) > 1:
-                dup_idx = out_labels[:i].count(label)
-                new_label += '_' + str(dup_idx + 1)
-            deduped_out_labels.append(new_label)
-        out_labels = deduped_out_labels
+        # Prepare display labels.
+        out_labels = self._get_deduped_metrics_names()
 
         if do_validation:
             callback_metrics = copy.copy(out_labels) + ['val_' + n for n in out_labels]
         else:
             callback_metrics = copy.copy(out_labels)
 
-        # delegate logic to _fit_loop
+        # Delegate logic to `_fit_loop`.
         return self._fit_loop(f, ins, out_labels=out_labels,
                               batch_size=batch_size, epochs=epochs,
                               verbose=verbose, callbacks=callbacks,
@@ -1511,13 +1522,13 @@ class Model(Container):
             and/or metrics). The attribute `model.metrics_names` will give you
             the display labels for the scalar outputs.
         """
-        # validate user data
+        # Validate user data.
         x, y, sample_weights = self._standardize_user_data(
             x, y,
             sample_weight=sample_weight,
             check_batch_axis=False,
             batch_size=batch_size)
-        # prepare inputs, delegate logic to _test_loop
+        # Prepare inputs, delegate logic to `_test_loop`.
         if self.uses_learning_phase and not isinstance(K.learning_phase(), int):
             ins = x + y + sample_weights + [0.]
         else:
@@ -1548,7 +1559,7 @@ class Model(Container):
                 or in case a stateful model receives a number of samples
                 that is not a multiple of the batch size.
         """
-        # validate user data
+        # Validate user data.
         x = _standardize_input_data(x, self._feed_input_names,
                                     self._feed_input_shapes,
                                     check_batch_axis=False)
@@ -1561,7 +1572,7 @@ class Model(Container):
                                  str(x[0].shape[0]) + ' samples. '
                                  'Batch size: ' + str(batch_size) + '.')
 
-        # prepare inputs, delegate logic to _predict_loop
+        # Prepare inputs, delegate logic to `_predict_loop`.
         if self.uses_learning_phase and not isinstance(K.learning_phase(), int):
             ins = x + [0.]
         else:
@@ -1594,7 +1605,7 @@ class Model(Container):
                 In this case you should make sure to specify
                 sample_weight_mode="temporal" in compile().
             class_weight: optional dictionary mapping
-                lass indices (integers) to
+                class indices (integers) to
                 a weight (float) to apply to the model's loss for the samples
                 from this class during training.
                 This can be useful to tell the model to "pay more attention" to
@@ -1685,7 +1696,6 @@ class Model(Container):
             return outputs[0]
         return outputs
 
-    @interfaces.legacy_generator_methods_support
     def fit_generator(self, generator,
                       steps_per_epoch,
                       epochs=1,
@@ -1711,8 +1721,8 @@ class Model(Container):
                 - a tuple (inputs, targets, sample_weights).
                 All arrays should contain the same number of samples.
                 The generator is expected to loop over its data
-                indefinitely. An epoch finishes when `samples_per_epoch`
-                samples have been seen by the model.
+                indefinitely. An epoch finishes when `steps_per_epoch`
+                batches have been seen by the model.
             steps_per_epoch: Total number of steps (batches of samples)
                 to yield from `generator` before declaring one epoch
                 finished and starting the next epoch. It should typically
@@ -1760,7 +1770,7 @@ class Model(Container):
                     f.close()
 
             model.fit_generator(generate_arrays_from_file('/my_file.txt'),
-                                samples_per_epoch=10000, epochs=10)
+                                steps_per_epoch=10000, epochs=10)
         ```
 
         # Raises
@@ -1784,7 +1794,8 @@ class Model(Container):
                              'you must specify a value for '
                              '`validation_steps`.')
 
-        out_labels = self.metrics_names
+        # Prepare display labels.
+        out_labels = self._get_deduped_metrics_names()
         callback_metrics = out_labels + ['val_' + n for n in out_labels]
 
         # prepare callbacks
@@ -1811,10 +1822,10 @@ class Model(Container):
 
         if do_validation and not val_gen:
             if len(validation_data) == 2:
-                val_x, val_y = validation_data
+                val_x, val_y = validation_data  # pylint: disable=unpacking-non-sequence
                 val_sample_weight = None
             elif len(validation_data) == 3:
-                val_x, val_y, val_sample_weight = validation_data
+                val_x, val_y, val_sample_weight = validation_data  # pylint: disable=unpacking-non-sequence
             else:
                 raise ValueError('validation_data should be a tuple '
                                  '`(val_x, val_y, val_sample_weight)` '
@@ -1822,8 +1833,11 @@ class Model(Container):
                                  str(validation_data))
             val_x, val_y, val_sample_weights = self._standardize_user_data(
                 val_x, val_y, val_sample_weight)
+            val_data = val_x + val_y + val_sample_weights
+            if self.uses_learning_phase and not isinstance(K.learning_phase(), int):
+                val_data += [0.]
             for cbk in callbacks:
-                cbk.validation_data = val_x + [val_y, val_sample_weights]
+                cbk.validation_data = val_data
         enqueuer = None
 
         try:
@@ -1850,10 +1864,10 @@ class Model(Container):
                                          'or `(x, y)`. Found: ' +
                                          str(generator_output))
                     if len(generator_output) == 2:
-                        x, y = generator_output
+                        x, y = generator_output  # pylint: disable=unpacking-non-sequence
                         sample_weight = None
                     elif len(generator_output) == 3:
-                        x, y, sample_weight = generator_output
+                        x, y, sample_weight = generator_output  # pylint: disable=unpacking-non-sequence
                     else:
                         raise ValueError('output of generator should be '
                                          'a tuple `(x, y, sample_weight)` '
@@ -1922,7 +1936,6 @@ class Model(Container):
         callbacks.on_train_end()
         return self.history
 
-    @interfaces.legacy_generator_methods_support
     def evaluate_generator(self, generator, steps,
                            max_q_size=10, workers=1, pickle_safe=False):
         """Evaluates the model on a data generator.
@@ -1930,7 +1943,7 @@ class Model(Container):
         The generator should return the same kind of data
         as accepted by `test_on_batch`.
 
-        Arguments:
+        # Arguments
             generator: Generator yielding tuples (inputs, targets)
                 or (inputs, targets, sample_weights)
             steps: Total number of steps (batches of samples)
@@ -1983,10 +1996,10 @@ class Model(Container):
                                      'or (x, y). Found: ' +
                                      str(generator_output))
                 if len(generator_output) == 2:
-                    x, y = generator_output
+                    x, y = generator_output  # pylint: disable=unpacking-non-sequence
                     sample_weight = None
                 elif len(generator_output) == 3:
-                    x, y, sample_weight = generator_output
+                    x, y, sample_weight = generator_output  # pylint: disable=unpacking-non-sequence
                 else:
                     raise ValueError('output of generator should be a tuple '
                                      '(x, y, sample_weight) '
@@ -2019,9 +2032,9 @@ class Model(Container):
                                            weights=batch_sizes))
             return averages
 
-    @interfaces.legacy_generator_methods_support
     def predict_generator(self, generator, steps,
-                          max_q_size=10, workers=1, pickle_safe=False):
+                          max_q_size=10, workers=1,
+                          pickle_safe=False, verbose=0):
         """Generates predictions for the input samples from a data generator.
 
         The generator should return the same kind of data as accepted by
@@ -2041,6 +2054,7 @@ class Model(Container):
                 non picklable arguments to the generator
                 as they can't be passed
                 easily to children processes.
+            verbose: verbosity mode, 0 or 1.
 
         # Returns
             Numpy array(s) of predictions.
@@ -2060,6 +2074,9 @@ class Model(Container):
             enqueuer = GeneratorEnqueuer(generator, pickle_safe=pickle_safe)
             enqueuer.start(workers=workers, max_q_size=max_q_size)
 
+            if verbose == 1:
+                progbar = Progbar(target=steps)
+
             while steps_done < steps:
                 generator_output = None
                 while enqueuer.is_running():
@@ -2073,9 +2090,9 @@ class Model(Container):
                     # Compatibility with the generators
                     # used for training.
                     if len(generator_output) == 2:
-                        x, _ = generator_output
+                        x, _ = generator_output  # pylint: disable=unpacking-non-sequence
                     elif len(generator_output) == 3:
-                        x, _, _ = generator_output
+                        x, _, _ = generator_output  # pylint: disable=unpacking-non-sequence
                     else:
                         raise ValueError('output of generator should be '
                                          'a tuple `(x, y, sample_weight)` '
@@ -2097,6 +2114,8 @@ class Model(Container):
                 for i, out in enumerate(outs):
                     all_outs[i].append(out)
                 steps_done += 1
+                if verbose == 1:
+                    progbar.update(steps_done)
 
         finally:
             if enqueuer is not None:

keras/initializers.py

@@ -1,9 +1,17 @@
+"""Keras initializer classes (soon to be replaced with core TF initializers).
+"""
 from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import math
+
+from . import backend as K
 import numpy as np
 import six
-from . import backend as K
-from .utils.generic_utils import serialize_keras_object
+from tensorflow.python.framework import tensor_shape
 from .utils.generic_utils import deserialize_keras_object
+from .utils.generic_utils import serialize_keras_object
 
 
 class Initializer(object):
@@ -22,14 +30,16 @@ class Initializer(object):
 
 
 class Zeros(Initializer):
-    """Initializer that generates tensors initialized to 0."""
+    """Initializer that generates tensors initialized to 0.
+    """
 
     def __call__(self, shape, dtype=None):
         return K.constant(0, shape=shape, dtype=dtype)
 
 
 class Ones(Initializer):
-    """Initializer that generates tensors initialized to 1."""
+    """Initializer that generates tensors initialized to 1.
+    """
 
     def __call__(self, shape, dtype=None):
         return K.constant(1, shape=shape, dtype=dtype)
@@ -111,7 +121,7 @@ class RandomUniform(Initializer):
 class TruncatedNormal(Initializer):
     """Initializer that generates a truncated normal distribution.
 
-    These values are similar to values from a `random_normal_initializer`
+    These values are similar to values from a `RandomNormal`
     except that values more than two standard deviations from the mean
     are discarded and re-drawn. This is the recommended initializer for
     neural network weights and filters.
@@ -146,6 +156,7 @@ class VarianceScaling(Initializer):
 
     With `distribution="normal"`, samples are drawn from a truncated normal
     distribution centered on zero, with `stddev = sqrt(scale / n)` where n is:
+
         - number of input units in the weight tensor, if mode = "fan_in"
         - number of output units, if mode = "fan_out"
         - average of the numbers of input and output units, if mode = "fan_avg"
@@ -196,11 +207,11 @@ class VarianceScaling(Initializer):
         else:
             scale /= max(1., float(fan_in + fan_out) / 2)
         if self.distribution == 'normal':
-            stddev = np.sqrt(scale)
+            stddev = math.sqrt(scale)
             return K.truncated_normal(shape, 0., stddev,
                                       dtype=dtype, seed=self.seed)
         else:
-            limit = np.sqrt(3. * scale)
+            limit = math.sqrt(3. * scale)
             return K.random_uniform(shape, -limit, limit,
                                     dtype=dtype, seed=self.seed)
 
@@ -392,6 +403,7 @@ def he_uniform(seed=None):
 
 # Compatibility aliases
 
+# pylint: disable=invalid-name
 zero = zeros = Zeros
 one = ones = Ones
 constant = Constant
@@ -400,6 +412,7 @@ normal = random_normal = RandomNormal
 truncated_normal = TruncatedNormal
 identity = Identity
 orthogonal = Orthogonal
+# pylint: enable=invalid-name
 
 # Utility functions
 
@@ -420,6 +433,7 @@ def _compute_fans(shape, data_format='channels_last'):
     # Raises
         ValueError: in case of invalid `data_format` argument.
     """
+    shape = tensor_shape.TensorShape(shape).as_list()
     if len(shape) == 2:
         fan_in = shape[0]
         fan_out = shape[1]
@@ -439,8 +453,8 @@ def _compute_fans(shape, data_format='channels_last'):
             raise ValueError('Invalid data_format: ' + data_format)
     else:
         # No specific assumptions.
-        fan_in = np.sqrt(np.prod(shape))
-        fan_out = np.sqrt(np.prod(shape))
+        fan_in = math.sqrt(np.prod(shape))
+        fan_out = math.sqrt(np.prod(shape))
     return fan_in, fan_out
 
 

keras/layers/__init__.py

@@ -1,46 +1,26 @@
+"""Keras layers module.
+"""
+# pylint: disable=wildcard-import
 from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
 
-from ..utils.generic_utils import deserialize_keras_object
-from ..engine import Layer
+from .advanced_activations import *
+from .convolutional import *
+from .convolutional_recurrent import *
+from .core import *
+from .embeddings import *
 from ..engine import Input
 from ..engine import InputLayer
 from ..engine import InputSpec
-from .merge import *
-from .core import *
-from .convolutional import *
-from .pooling import *
+from ..engine import Layer
 from .local import *
-from .recurrent import *
-from .normalization import *
-from .embeddings import *
+from .merge import *
 from .noise import *
-from .advanced_activations import *
+from .normalization import *
+from .pooling import *
+from .recurrent import *
+from .serialization import deserialize
+from .serialization import serialize
 from .wrappers import *
 from .convolutional_recurrent import *
-from ..legacy.layers import *
-
-
-def serialize(layer):
-    return {'class_name': layer.__class__.__name__,
-            'config': layer.get_config()}
-
-
-def deserialize(config, custom_objects=None):
-    """Instantiate a layer from a config dictionary.
-
-    # Arguments
-        config: dict of the form {'class_name': str, 'config': dict}
-        custom_objects: dict mapping class names (or function names)
-            of custom (non-Keras) objects to class/functions
-
-    # Returns
-        Layer instance (may be Model, Sequential, Layer...)
-    """
-    from .. import models
-    globs = globals()  # All layers.
-    globs['Model'] = models.Model
-    globs['Sequential'] = models.Sequential
-    return deserialize_keras_object(config,
-                                    module_objects=globs,
-                                    custom_objects=custom_objects,
-                                    printable_module_name='layer')

keras/layers/advanced_activations.py

@@ -1,13 +1,17 @@
+"""Layers that act as activation functions.
+"""
 # -*- coding: utf-8 -*-
 from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
 
+from .. import backend as K
+from .. import constraints
 from .. import initializers
 from .. import regularizers
-from .. import constraints
-from ..engine import Layer
 from ..engine import InputSpec
-from .. import backend as K
-from ..legacy import interfaces
+from ..engine import Layer
+from tensorflow.python.framework import tensor_shape
 
 
 class LeakyReLU(Layer):
@@ -28,8 +32,6 @@ class LeakyReLU(Layer):
     # Arguments
         alpha: float >= 0. Negative slope coefficient.
 
-    # References
-        - [Rectifier Nonlinearities Improve Neural Network Acoustic Models](https://web.stanford.edu/~awni/papers/relu_hybrid_icml2013_final.pdf)
     """
 
     def __init__(self, alpha=0.3, **kwargs):
@@ -75,11 +77,8 @@ class PReLU(Layer):
             so that each filter only has one set of parameters,
             set `shared_axes=[1, 2]`.
 
-    # References
-        - [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://arxiv.org/abs/1502.01852)
     """
 
-    @interfaces.legacy_prelu_support
     def __init__(self, alpha_initializer='zeros',
                  alpha_regularizer=None,
                  alpha_constraint=None,
@@ -98,13 +97,14 @@ class PReLU(Layer):
             self.shared_axes = list(shared_axes)
 
     def build(self, input_shape):
-        param_shape = list(input_shape[1:])
+        input_shape = tensor_shape.TensorShape(input_shape).as_list()
+        param_shape = input_shape[1:]
         self.param_broadcast = [False] * len(param_shape)
         if self.shared_axes is not None:
             for i in self.shared_axes:
                 param_shape[i - 1] = 1
                 self.param_broadcast[i - 1] = True
-        self.alpha = self.add_weight(param_shape,
+        self.alpha = self.add_weight(shape=param_shape,
                                      name='alpha',
                                      initializer=self.alpha_initializer,
                                      regularizer=self.alpha_regularizer,
@@ -156,8 +156,6 @@ class ELU(Layer):
     # Arguments
         alpha: scale for the negative factor.
 
-    # References
-        - [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)](https://arxiv.org/abs/1511.07289v1)
     """
 
     def __init__(self, alpha=1.0, **kwargs):
@@ -192,8 +190,6 @@ class ThresholdedReLU(Layer):
     # Arguments
         theta: float >= 0. Threshold location of activation.
 
-    # References
-        - [Zero-Bias Autoencoders and the Benefits of Co-Adapting Features](http://arxiv.org/abs/1402.3337)
     """
 
     def __init__(self, theta=1.0, **kwargs):

keras/layers/convolutional_recurrent.py

@@ -1,17 +1,20 @@
 # -*- coding: utf-8 -*-
+"""Convolutional-recurrent layers.
+"""
 from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
 
-from .. import backend as K
 from .. import activations
+from .. import backend as K
+from .. import constraints
 from .. import initializers
 from .. import regularizers
-from .. import constraints
-from .recurrent import Recurrent
-
-import numpy as np
 from ..engine import InputSpec
+import numpy as np
+from .recurrent import Recurrent
+from tensorflow.python.framework import tensor_shape
 from ..utils import conv_utils
-from ..legacy import interfaces
 
 
 class ConvRecurrent2D(Recurrent):
@@ -62,7 +65,7 @@ class ConvRecurrent2D(Recurrent):
     # Masking
         This layer supports masking for input data with a variable number
         of timesteps. To introduce masks to your data,
-        use an [Embedding](embeddings.md) layer with the `mask_zero` parameter
+        use an `Embedding` layer with the `mask_zero` parameter
         set to `True`.
         **Note:** for the time being, masking is only supported with Theano.
 
@@ -105,9 +108,13 @@ class ConvRecurrent2D(Recurrent):
         self.return_sequences = return_sequences
         self.go_backwards = go_backwards
         self.stateful = stateful
-        self.input_spec = InputSpec(ndim=5)
+        self.input_spec = [InputSpec(ndim=5)]
+        self.state_spec = None
 
-    def compute_output_shape(self, input_shape):
+    def _compute_output_shape(self, input_shape):
+        if type(input_shape) is list:
+            input_shape = input_shape[0]
+        input_shape = tensor_shape.TensorShape(input_shape).as_list()
         if self.data_format == 'channels_first':
             rows = input_shape[3]
             cols = input_shape[4]
@@ -126,16 +133,28 @@ class ConvRecurrent2D(Recurrent):
                                              dilation=self.dilation_rate[1])
         if self.return_sequences:
             if self.data_format == 'channels_first':
-                return (input_shape[0], input_shape[1],
-                        self.filters, rows, cols)
+                return tensor_shape.TensorShape([input_shape[0],
+                                                input_shape[1],
+                                                self.filters,
+                                                rows,
+                                                cols])
             elif self.data_format == 'channels_last':
-                return (input_shape[0], input_shape[1],
-                        rows, cols, self.filters)
+                return tensor_shape.TensorShape([input_shape[0],
+                                                input_shape[1],
+                                                rows,
+                                                cols,
+                                                self.filters])
         else:
             if self.data_format == 'channels_first':
-                return (input_shape[0], self.filters, rows, cols)
+                return tensor_shape.TensorShape([input_shape[0],
+                                                self.filters,
+                                                rows,
+                                                cols])
             elif self.data_format == 'channels_last':
-                return (input_shape[0], rows, cols, self.filters)
+                return tensor_shape.TensorShape([input_shape[0],
+                                                rows,
+                                                cols,
+                                                self.filters])
 
     def get_config(self):
         config = {'filters': self.filters,
@@ -181,46 +200,34 @@ class ConvLSTM2D(ConvRecurrent2D):
             the dilation rate to use for dilated convolution.
             Currently, specifying any `dilation_rate` value != 1 is
             incompatible with specifying any `strides` value != 1.
-        activation: Activation function to use
-            (see [activations](../activations.md)).
+        activation: Activation function to use.
             If you don't specify anything, no activation is applied
             (ie. "linear" activation: `a(x) = x`).
         recurrent_activation: Activation function to use
-            for the recurrent step
-            (see [activations](../activations.md)).
+            for the recurrent step.
         use_bias: Boolean, whether the layer uses a bias vector.
         kernel_initializer: Initializer for the `kernel` weights matrix,
-            used for the linear transformation of the inputs.
-            (see [initializers](../initializers.md)).
+            used for the linear transformation of the inputs..
         recurrent_initializer: Initializer for the `recurrent_kernel`
             weights matrix,
-            used for the linear transformation of the recurrent state.
-            (see [initializers](../initializers.md)).
-        bias_initializer: Initializer for the bias vector
-            (see [initializers](../initializers.md)).
+            used for the linear transformation of the recurrent state..
+        bias_initializer: Initializer for the bias vector.
         unit_forget_bias: Boolean.
             If True, add 1 to the bias of the forget gate at initialization.
             Use in combination with `bias_initializer="zeros"`.
             This is recommended in [Jozefowicz et al.](http://www.jmlr.org/proceedings/papers/v37/jozefowicz15.pdf)
         kernel_regularizer: Regularizer function applied to
-            the `kernel` weights matrix
-            (see [regularizer](../regularizers.md)).
+            the `kernel` weights matrix.
         recurrent_regularizer: Regularizer function applied to
-            the `recurrent_kernel` weights matrix
-            (see [regularizer](../regularizers.md)).
-        bias_regularizer: Regularizer function applied to the bias vector
-            (see [regularizer](../regularizers.md)).
+            the `recurrent_kernel` weights matrix.
+        bias_regularizer: Regularizer function applied to the bias vector.
         activity_regularizer: Regularizer function applied to
-            the output of the layer (its "activation").
-            (see [regularizer](../regularizers.md)).
+            the output of the layer (its "activation")..
         kernel_constraint: Constraint function applied to
-            the `kernel` weights matrix
-            (see [constraints](../constraints.md)).
+            the `kernel` weights matrix.
         recurrent_constraint: Constraint function applied to
-            the `recurrent_kernel` weights matrix
-            (see [constraints](../constraints.md)).
-        bias_constraint: Constraint function applied to the bias vector
-            (see [constraints](../constraints.md)).
+            the `recurrent_kernel` weights matrix.
+        bias_constraint: Constraint function applied to the bias vector.
         return_sequences: Boolean. Whether to return the last output
             in the output sequence, or the full sequence.
         go_backwards: Boolean (default False).
@@ -271,7 +278,6 @@ class ConvLSTM2D(ConvRecurrent2D):
         cells output
     """
 
-    @interfaces.legacy_convlstm2d_support
     def __init__(self, filters,
                  kernel_size,
                  strides=(1, 1),
@@ -328,10 +334,14 @@ class ConvLSTM2D(ConvRecurrent2D):
 
         self.dropout = min(1., max(0., dropout))
         self.recurrent_dropout = min(1., max(0., recurrent_dropout))
+        self.state_spec = [InputSpec(ndim=4), InputSpec(ndim=4)]
 
     def build(self, input_shape):
-        # TODO: better handling of input spec
-        self.input_spec = InputSpec(shape=input_shape)
+        if isinstance(input_shape, list):
+            input_shape = input_shape[0]
+        input_shape = tuple(tensor_shape.TensorShape(input_shape).as_list())
+        batch_size = input_shape[0] if self.stateful else None
+        self.input_spec[0] = InputSpec(shape=(batch_size, None) + input_shape[2:])
 
         if self.stateful:
             self.reset_states()
@@ -340,30 +350,34 @@ class ConvLSTM2D(ConvRecurrent2D):
             self.states = [None, None]
 
         if self.data_format == 'channels_first':
-            channel_axis = 1
+            channel_axis = 2
         else:
             channel_axis = -1
         if input_shape[channel_axis] is None:
             raise ValueError('The channel dimension of the inputs '
                              'should be defined. Found `None`.')
         input_dim = input_shape[channel_axis]
+        state_shape = [None] * 4
+        state_shape[channel_axis] = input_dim
+        state_shape = tuple(state_shape)
+        self.state_spec = [InputSpec(shape=state_shape), InputSpec(shape=state_shape)]
         kernel_shape = self.kernel_size + (input_dim, self.filters * 4)
         self.kernel_shape = kernel_shape
         recurrent_kernel_shape = self.kernel_size + (self.filters, self.filters * 4)
 
-        self.kernel = self.add_weight(kernel_shape,
+        self.kernel = self.add_weight(shape=kernel_shape,
                                       initializer=self.kernel_initializer,
                                       name='kernel',
                                       regularizer=self.kernel_regularizer,
                                       constraint=self.kernel_constraint)
         self.recurrent_kernel = self.add_weight(
-            recurrent_kernel_shape,
+            shape=recurrent_kernel_shape,
             initializer=self.recurrent_initializer,
             name='recurrent_kernel',
             regularizer=self.recurrent_regularizer,
             constraint=self.recurrent_constraint)
         if self.use_bias:
-            self.bias = self.add_weight((self.filters * 4,),
+            self.bias = self.add_weight(shape=(self.filters * 4,),
                                         initializer=self.bias_initializer,
                                         name='bias',
                                         regularizer=self.bias_regularizer,
@@ -396,7 +410,7 @@ class ConvLSTM2D(ConvRecurrent2D):
             self.bias_o = None
         self.built = True
 
-    def get_initial_states(self, inputs):
+    def get_initial_state(self, inputs):
         # (samples, timesteps, rows, cols, filters)
         initial_state = K.zeros_like(inputs)
         # (samples, rows, cols, filters)
@@ -413,8 +427,9 @@ class ConvLSTM2D(ConvRecurrent2D):
     def reset_states(self):
         if not self.stateful:
             raise RuntimeError('Layer must be stateful.')
-        input_shape = self.input_spec.shape
-        output_shape = self.compute_output_shape(input_shape)
+        input_shape = self.input_spec[0].shape
+        output_shape = self._compute_output_shape(input_shape)
+
         if not input_shape[0]:
             raise ValueError('If a RNN is stateful, a complete '
                              'input_shape must be provided '
@@ -465,7 +480,7 @@ class ConvLSTM2D(ConvRecurrent2D):
                                    padding=self.padding)
             ones += 1.
 
-            def dropped_inputs():
+            def dropped_inputs():  # pylint: disable=function-redefined
                 return K.dropout(ones, self.recurrent_dropout)
             rec_dp_mask = [K.in_train_phase(dropped_inputs,
                                             ones,

keras/layers/core.py

@@ -1,25 +1,26 @@
 # -*- coding: utf-8 -*-
+"""Core Keras layers.
+"""
 from __future__ import absolute_import
 from __future__ import division
-
-import numpy as np
+from __future__ import print_function
 
 import copy
 import inspect
 import types as python_types
-import warnings
 
-from .. import backend as K
 from .. import activations
+from .. import backend as K
+from .. import constraints
 from .. import initializers
 from .. import regularizers
-from .. import constraints
 from ..engine import InputSpec
 from ..engine import Layer
+import numpy as np
+from tensorflow.python.framework import tensor_shape
+from ..utils.generic_utils import deserialize_keras_object
 from ..utils.generic_utils import func_dump
 from ..utils.generic_utils import func_load
-from ..utils.generic_utils import deserialize_keras_object
-from ..legacy import interfaces
 
 
 class Masking(Layer):
@@ -27,7 +28,7 @@ class Masking(Layer):
 
     For each timestep in the input tensor (dimension #1 in the tensor),
     if all values in the input tensor at that timestep
-    are equal to `mask_value`, then the timestep will masked (skipped)
+    are equal to `mask_value`, then the timestep will be masked (skipped)
     in all downstream layers (as long as they support masking).
 
     If any downstream layer does not support masking yet receives such
@@ -73,7 +74,7 @@ class Dropout(Layer):
     """Applies Dropout to the input.
 
     Dropout consists in randomly setting
-    a fraction `p` of input units to 0 at each update during training time,
+    a fraction `rate` of input units to 0 at each update during training time,
     which helps prevent overfitting.
 
     # Arguments
@@ -85,11 +86,8 @@ class Dropout(Layer):
             you want the dropout mask to be the same for all timesteps,
             you can use `noise_shape=(batch_size, 1, features)`.
         seed: A Python integer to use as random seed.
-
-    # References
-        - [Dropout: A Simple Way to Prevent Neural Networks from Overfitting](http://www.cs.toronto.edu/~rsalakhu/papers/srivastava14a.pdf)
     """
-    @interfaces.legacy_dropout_support
+
     def __init__(self, rate, noise_shape=None, seed=None, **kwargs):
         super(Dropout, self).__init__(**kwargs)
         self.rate = min(1., max(0., rate))
@@ -107,9 +105,9 @@ class Dropout(Layer):
             def dropped_inputs():
                 return K.dropout(inputs, self.rate, noise_shape,
                                  seed=self.seed)
-            output = K.in_train_phase(dropped_inputs, inputs,
-                                      training=training)
-        return output
+            return K.in_train_phase(dropped_inputs, inputs,
+                                    training=training)
+        return inputs
 
     def get_config(self):
         config = {'rate': self.rate}
@@ -129,7 +127,7 @@ class SpatialDropout1D(Dropout):
     between feature maps and should be used instead.
 
     # Arguments
-        p: float between 0 and 1. Fraction of the input units to drop.
+        rate: float between 0 and 1. Fraction of the input units to drop.
 
     # Input shape
         3D tensor with shape:
@@ -142,7 +140,6 @@ class SpatialDropout1D(Dropout):
         - [Efficient Object Localization Using Convolutional Networks](https://arxiv.org/abs/1411.4280)
     """
 
-    @interfaces.legacy_spatialdropout1d_support
     def __init__(self, rate, **kwargs):
         super(SpatialDropout1D, self).__init__(rate, **kwargs)
         self.input_spec = InputSpec(ndim=3)
@@ -187,7 +184,6 @@ class SpatialDropout2D(Dropout):
         - [Efficient Object Localization Using Convolutional Networks](https://arxiv.org/abs/1411.4280)
     """
 
-    @interfaces.legacy_spatialdropoutNd_support
     def __init__(self, rate, data_format=None, **kwargs):
         super(SpatialDropout2D, self).__init__(rate, **kwargs)
         if data_format is None:
@@ -242,7 +238,6 @@ class SpatialDropout3D(Dropout):
         - [Efficient Object Localization Using Convolutional Networks](https://arxiv.org/abs/1411.4280)
     """
 
-    @interfaces.legacy_spatialdropoutNd_support
     def __init__(self, rate, data_format=None, **kwargs):
         super(SpatialDropout3D, self).__init__(rate, **kwargs)
         if data_format is None:
@@ -269,7 +264,6 @@ class Activation(Layer):
 
     # Arguments
         activation: name of activation function to use
-            (see: [activations](../activations.md)),
             or alternatively, a Theano or TensorFlow operation.
 
     # Input shape
@@ -377,30 +371,24 @@ class Reshape(Layer):
             output_shape[unknown] = original // known
         elif original != known:
             raise ValueError(msg)
+        return output_shape
 
-        return tuple(output_shape)
-
-    def compute_output_shape(self, input_shape):
-        return (input_shape[0],) + self._fix_unknown_dimension(
-            input_shape[1:], self.target_shape)
+    def _compute_output_shape(self, input_shape):
+        input_shape = tensor_shape.TensorShape(input_shape).as_list()
+        output_shape = [input_shape[0]]
+        output_shape += self._fix_unknown_dimension(input_shape[1:],
+                                                    self.target_shape)
+        return tensor_shape.TensorShape(output_shape)
 
     def call(self, inputs):
         # In case the target shape is not fully defined,
         # we need access to the shape of x.
-        # solution:
-        # 1) rely on x._keras_shape
-        # 2) fallback: K.int_shape
         target_shape = self.target_shape
         if -1 in target_shape:
             # target shape not fully defined
-            input_shape = None
-            try:
-                input_shape = K.int_shape(inputs)
-            except TypeError:
-                pass
-            if input_shape is not None:
-                target_shape = self.compute_output_shape(input_shape)[1:]
-        return K.reshape(inputs, (-1,) + target_shape)
+            target_shape = self._compute_output_shape(inputs.get_shape())
+            target_shape = target_shape.as_list()[1:]
+        return K.reshape(inputs, (-1,) + tuple(target_shape))
 
     def get_config(self):
         config = {'target_shape': self.target_shape}
@@ -443,13 +431,13 @@ class Permute(Layer):
         self.dims = tuple(dims)
         self.input_spec = InputSpec(ndim=len(self.dims) + 1)
 
-    def compute_output_shape(self, input_shape):
-        input_shape = list(input_shape)
+    def _compute_output_shape(self, input_shape):
+        input_shape = tensor_shape.TensorShape(input_shape).as_list()
         output_shape = copy.copy(input_shape)
         for i, dim in enumerate(self.dims):
             target_dim = input_shape[dim]
             output_shape[i + 1] = target_dim
-        return tuple(output_shape)
+        return tensor_shape.TensorShape(output_shape)
 
     def call(self, inputs):
         return K.permute_dimensions(inputs, (0,) + self.dims)
@@ -481,7 +469,8 @@ class Flatten(Layer):
         super(Flatten, self).__init__(**kwargs)
         self.input_spec = InputSpec(min_ndim=3)
 
-    def compute_output_shape(self, input_shape):
+    def _compute_output_shape(self, input_shape):
+        input_shape = tensor_shape.TensorShape(input_shape).as_list()
         if not all(input_shape[1:]):
             raise ValueError('The shape of the input to "Flatten" '
                              'is not fully defined '
@@ -489,10 +478,12 @@ class Flatten(Layer):
                              'Make sure to pass a complete "input_shape" '
                              'or "batch_input_shape" argument to the first '
                              'layer in your model.')
-        return (input_shape[0], np.prod(input_shape[1:]))
+        return tensor_shape.TensorShape([input_shape[0], np.prod(input_shape[1:])])
 
     def call(self, inputs):
-        return K.batch_flatten(inputs)
+        outputs = K.batch_flatten(inputs)
+        outputs.set_shape(self._compute_output_shape(inputs.get_shape()))
+        return outputs
 
 
 class RepeatVector(Layer):
@@ -525,8 +516,9 @@ class RepeatVector(Layer):
         self.n = n
         self.input_spec = InputSpec(ndim=2)
 
-    def compute_output_shape(self, input_shape):
-        return (input_shape[0], self.n, input_shape[1])
+    def _compute_output_shape(self, input_shape):
+        input_shape = tensor_shape.TensorShape(input_shape).as_list()
+        return tensor_shape.TensorShape([input_shape[0], self.n, input_shape[1]])
 
     def call(self, inputs):
         return K.repeat(inputs, self.n)
@@ -558,30 +550,12 @@ class Lambda(Layer):
             neg = K.relu(-x)
             return K.concatenate([pos, neg], axis=1)
 
-        def antirectifier_output_shape(input_shape):
-            shape = list(input_shape)
-            assert len(shape) == 2  # only valid for 2D tensors
-            shape[-1] *= 2
-            return tuple(shape)
-
-        model.add(Lambda(antirectifier,
-                         output_shape=antirectifier_output_shape))
+        model.add(Lambda(antirectifier))
     ```
 
     # Arguments
         function: The function to be evaluated.
             Takes input tensor as first argument.
-        output_shape: Expected output shape from function.
-            Only relevant when using Theano.
-            Can be a tuple or function.
-            If a tuple, it only specifies the first dimension onward;
-                 sample dimension is assumed either the same as the input:
-                 `output_shape = (input_shape[0], ) + output_shape`
-                 or, the input is `None` and
-                 the sample dimension is also `None`:
-                 `output_shape = (None, ) + output_shape`
-            If a function, it specifies the entire shape as a function of the
-            input shape: `output_shape = f(input_shape)`
         arguments: optional dictionary of keyword arguments to be passed
             to the function.
 
@@ -595,9 +569,10 @@ class Lambda(Layer):
         (or auto-inferred when using TensorFlow).
     """
 
-    @interfaces.legacy_lambda_support
-    def __init__(self, function, output_shape=None,
-                 mask=None, arguments=None, **kwargs):
+    def __init__(self, function,
+                 mask=None,
+                 arguments=None,
+                 **kwargs):
         super(Lambda, self).__init__(**kwargs)
         self.function = function
         self.arguments = arguments if arguments else {}
@@ -605,52 +580,6 @@ class Lambda(Layer):
             self.supports_masking = True
         self.mask = mask
 
-        if output_shape is None:
-            self._output_shape = None
-        elif isinstance(output_shape, (tuple, list)):
-            self._output_shape = tuple(output_shape)
-        else:
-            if not callable(output_shape):
-                raise TypeError('In Lambda, `output_shape` '
-                                'must be a list, a tuple, or a function.')
-            self._output_shape = output_shape
-
-    def compute_output_shape(self, input_shape):
-        if self._output_shape is None:
-            # With TensorFlow, we can infer the output shape directly:
-            if K.backend() == 'tensorflow':
-                if isinstance(input_shape, list):
-                    xs = [K.placeholder(shape=shape) for shape in input_shape]
-                    x = self.call(xs)
-                else:
-                    x = K.placeholder(shape=input_shape)
-                    x = self.call(x)
-                if isinstance(x, list):
-                    return [K.int_shape(x_elem) for x_elem in x]
-                else:
-                    return K.int_shape(x)
-            # Otherwise, we default to the input shape.
-            warnings.warn('`output_shape` argument not specified for layer {} '
-                          'and cannot be automatically inferred '
-                          'with the Theano backend. '
-                          'Defaulting to output shape `{}` '
-                          '(same as input shape). '
-                          'If the expected output shape is different, '
-                          'specify it via the `output_shape` argument.'
-                          .format(self.name, input_shape))
-            return input_shape
-        elif isinstance(self._output_shape, (tuple, list)):
-            if isinstance(input_shape, list):
-                num_samples = input_shape[0][0]
-            else:
-                num_samples = input_shape[0] if input_shape else None
-            return (num_samples,) + tuple(self._output_shape)
-        else:
-            shape = self._output_shape(input_shape)
-            if not isinstance(shape, (list, tuple)):
-                raise ValueError('output_shape function must return a tuple')
-            return tuple(shape)
-
     def call(self, inputs, mask=None):
         arguments = self.arguments
         arg_spec = inspect.getargspec(self.function)
@@ -671,20 +600,8 @@ class Lambda(Layer):
             function = self.function.__name__
             function_type = 'function'
 
-        if isinstance(self._output_shape, python_types.LambdaType):
-            output_shape = func_dump(self._output_shape)
-            output_shape_type = 'lambda'
-        elif callable(self._output_shape):
-            output_shape = self._output_shape.__name__
-            output_shape_type = 'function'
-        else:
-            output_shape = self._output_shape
-            output_shape_type = 'raw'
-
         config = {'function': function,
                   'function_type': function_type,
-                  'output_shape': output_shape,
-                  'output_shape_type': output_shape_type,
                   'arguments': self.arguments}
         base_config = super(Lambda, self).get_config()
         return dict(list(base_config.items()) + list(config.items()))
@@ -707,21 +624,7 @@ class Lambda(Layer):
         else:
             raise TypeError('Unknown function type:', function_type)
 
-        output_shape_type = config.pop('output_shape_type')
-        if output_shape_type == 'function':
-            # Simple lookup in custom objects
-            output_shape = deserialize_keras_object(
-                config['output_shape'],
-                custom_objects=custom_objects,
-                printable_module_name='output_shape function in Lambda layer')
-        elif output_shape_type == 'lambda':
-            # Unsafe deserialization from bytecode
-            output_shape = func_load(config['output_shape'], globs=globs)
-        else:
-            output_shape = config['output_shape']
-
         config['function'] = function
-        config['output_shape'] = output_shape
         return cls(**config)
 
 
@@ -754,28 +657,20 @@ class Dense(Layer):
 
     # Arguments
         units: Positive integer, dimensionality of the output space.
-        activation: Activation function to use
-            (see [activations](../activations.md)).
+        activation: Activation function to use.
             If you don't specify anything, no activation is applied
             (ie. "linear" activation: `a(x) = x`).
         use_bias: Boolean, whether the layer uses a bias vector.
-        kernel_initializer: Initializer for the `kernel` weights matrix
-            (see [initializers](../initializers.md)).
-        bias_initializer: Initializer for the bias vector
-            (see [initializers](../initializers.md)).
+        kernel_initializer: Initializer for the `kernel` weights matrix.
+        bias_initializer: Initializer for the bias vector.
         kernel_regularizer: Regularizer function applied to
-            the `kernel` weights matrix
-            (see [regularizer](../regularizers.md)).
-        bias_regularizer: Regularizer function applied to the bias vector
-            (see [regularizer](../regularizers.md)).
+            the `kernel` weights matrix.
+        bias_regularizer: Regularizer function applied to the bias vector.
         activity_regularizer: Regularizer function applied to
-            the output of the layer (its "activation").
-            (see [regularizer](../regularizers.md)).
+            the output of the layer (its "activation")..
         kernel_constraint: Constraint function applied to
-            the `kernel` weights matrix
-            (see [constraints](../constraints.md)).
-        bias_constraint: Constraint function applied to the bias vector
-            (see [constraints](../constraints.md)).
+            the `kernel` weights matrix.
+        bias_constraint: Constraint function applied to the bias vector.
 
     # Input shape
         nD tensor with shape: `(batch_size, ..., input_dim)`.
@@ -788,7 +683,6 @@ class Dense(Layer):
         the output would have shape `(batch_size, units)`.
     """
 
-    @interfaces.legacy_dense_support
     def __init__(self, units,
                  activation=None,
                  use_bias=True,
@@ -820,13 +714,13 @@ class Dense(Layer):
         assert len(input_shape) >= 2
         input_dim = input_shape[-1]
 
-        self.kernel = self.add_weight((input_dim, self.units),
+        self.kernel = self.add_weight(shape=(input_dim, self.units),
                                       initializer=self.kernel_initializer,
                                       name='kernel',
                                       regularizer=self.kernel_regularizer,
                                       constraint=self.kernel_constraint)
         if self.use_bias:
-            self.bias = self.add_weight((self.units,),
+            self.bias = self.add_weight(shape=(self.units,),
                                         initializer=self.bias_initializer,
                                         name='bias',
                                         regularizer=self.bias_regularizer,
@@ -844,12 +738,13 @@ class Dense(Layer):
             output = self.activation(output)
         return output
 
-    def compute_output_shape(self, input_shape):
+    def _compute_output_shape(self, input_shape):
+        input_shape = tensor_shape.TensorShape(input_shape).as_list()
         assert input_shape and len(input_shape) >= 2
         assert input_shape[-1]
         output_shape = list(input_shape)
         output_shape[-1] = self.units
-        return tuple(output_shape)
+        return tensor_shape.TensorShape(output_shape)
 
     def get_config(self):
         config = {
@@ -857,7 +752,7 @@ class Dense(Layer):
             'activation': activations.serialize(self.activation),
             'use_bias': self.use_bias,
             'kernel_initializer': initializers.serialize(self.kernel_initializer),
-            'bias_initializer': initializers.serialize(self.kernel_initializer),
+            'bias_initializer': initializers.serialize(self.bias_initializer),
             'kernel_regularizer': regularizers.serialize(self.kernel_regularizer),
             'bias_regularizer': regularizers.serialize(self.bias_regularizer),
             'activity_regularizer': regularizers.serialize(self.activity_regularizer),

keras/layers/embeddings.py

@@ -1,11 +1,15 @@
+"""Embedding layer.
+"""
 from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
 
 from .. import backend as K
+from .. import constraints
 from .. import initializers
 from .. import regularizers
-from .. import constraints
 from ..engine import Layer
-from ..legacy import interfaces
+from tensorflow.python.framework import tensor_shape
 
 
 class Embedding(Layer):
@@ -31,25 +35,23 @@ class Embedding(Layer):
     ```
 
     # Arguments
-      input_dim: int > 0. Size of the vocabulary, ie.
-          1 + maximum integer index occurring in the input data.
+      input_dim: int > 0. Size of the vocabulary,
+          i.e. maximum integer index + 1.
       output_dim: int >= 0. Dimension of the dense embedding.
-      embeddings_initializer: Initializer for the `embeddings` matrix
-            (see [initializers](../initializers.md)).
+      embeddings_initializer: Initializer for the `embeddings` matrix.
       embeddings_regularizer: Regularizer function applied to
-            the `embeddings` matrix
-            (see [regularizer](../regularizers.md)).
+            the `embeddings` matrix.
       embeddings_constraint: Constraint function applied to
-            the `embeddings` matrix
-            (see [constraints](../constraints.md)).
+            the `embeddings` matrix.
       mask_zero: Whether or not the input value 0 is a special "padding"
           value that should be masked out.
-          This is useful when using [recurrent layers](recurrent.md)
-          which may take variable length input.
+          This is useful when using recurrent layers,
+          which may take variable length inputs.
           If this is `True` then all subsequent layers
           in the model need to support masking or an exception will be raised.
           If mask_zero is set to True, as a consequence, index 0 cannot be
-          used in the vocabulary (input_dim should equal `|vocabulary| + 2`).
+          used in the vocabulary (input_dim should equal size of
+          vocabulary + 1).
       input_length: Length of input sequences, when it is constant.
           This argument is required if you are going to connect
           `Flatten` then `Dense` layers upstream
@@ -65,7 +67,6 @@ class Embedding(Layer):
         - [A Theoretically Grounded Application of Dropout in Recurrent Neural Networks](http://arxiv.org/abs/1512.05287)
     """
 
-    @interfaces.legacy_embedding_support
     def __init__(self, input_dim, output_dim,
                  embeddings_initializer='uniform',
                  embeddings_regularizer=None,
@@ -92,8 +93,9 @@ class Embedding(Layer):
         self.input_length = input_length
 
     def build(self, input_shape):
+        input_shape = tensor_shape.TensorShape(input_shape).as_list()
         self.embeddings = self.add_weight(
-            (self.input_dim, self.output_dim),
+            shape=(self.input_dim, self.output_dim),
             initializer=self.embeddings_initializer,
             name='embeddings',
             regularizer=self.embeddings_regularizer,
@@ -106,12 +108,15 @@ class Embedding(Layer):
         else:
             return K.not_equal(inputs, 0)
 
-    def compute_output_shape(self, input_shape):
+    def _compute_output_shape(self, input_shape):
+        input_shape = tensor_shape.TensorShape(input_shape).as_list()
         if not self.input_length:
             input_length = input_shape[1]
         else:
             input_length = self.input_length
-        return (input_shape[0], input_length, self.output_dim)
+        return tensor_shape.TensorShape([input_shape[0],
+                                        input_length,
+                                        self.output_dim])
 
     def call(self, inputs):
         if K.dtype(inputs) != 'int32':

keras/layers/local.py

@@ -1,15 +1,19 @@
+"""Locally-connected layers.
+"""
 # -*- coding: utf-8 -*-
 from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
 
-from .. import backend as K
 from .. import activations
+from .. import backend as K
+from .. import constraints
 from .. import initializers
 from .. import regularizers
-from .. import constraints
-from ..engine import Layer
 from ..engine import InputSpec
+from ..engine import Layer
+from tensorflow.python.framework import tensor_shape
 from ..utils import conv_utils
-from ..legacy import interfaces
 
 
 class LocallyConnected1D(Layer):
@@ -41,28 +45,21 @@ class LocallyConnected1D(Layer):
             specifying the stride length of the convolution.
             Specifying any stride value != 1 is incompatible with specifying
             any `dilation_rate` value != 1.
-        padding: One of `"valid"` or `"same"` (case-insensitive).
-        activation: Activation function to use
-            (see [activations](../activations.md)).
+        padding: Currently only supports `"valid"` (case-insensitive).
+            `"same"` may be supported in the future.
+        activation: Activation function to use.
             If you don't specify anything, no activation is applied
             (ie. "linear" activation: `a(x) = x`).
         use_bias: Boolean, whether the layer uses a bias vector.
-        kernel_initializer: Initializer for the `kernel` weights matrix
-            (see [initializers](../initializers.md)).
-        bias_initializer: Initializer for the bias vector
-            (see [initializers](../initializers.md)).
+        kernel_initializer: Initializer for the `kernel` weights matrix.
+        bias_initializer: Initializer for the bias vector.
         kernel_regularizer: Regularizer function applied to
-            the `kernel` weights matrix
-            (see [regularizer](../regularizers.md)).
-        bias_regularizer: Regularizer function applied to the bias vector
-            (see [regularizer](../regularizers.md)).
+            the `kernel` weights matrix.
+        bias_regularizer: Regularizer function applied to the bias vector.
         activity_regularizer: Regularizer function applied to
-            the output of the layer (its "activation").
-            (see [regularizer](../regularizers.md)).
-        kernel_constraint: Constraint function applied to the kernel matrix
-            (see [constraints](../constraints.md)).
-        bias_constraint: Constraint function applied to the bias vector
-            (see [constraints](../constraints.md)).
+            the output of the layer (its "activation")..
+        kernel_constraint: Constraint function applied to the kernel matrix.
+        bias_constraint: Constraint function applied to the bias vector.
 
     # Input shape
         3D tensor with shape: `(batch_size, steps, input_dim)`
@@ -72,7 +69,6 @@ class LocallyConnected1D(Layer):
         `steps` value might have changed due to padding or strides.
     """
 
-    @interfaces.legacy_conv1d_support
     def __init__(self, filters,
                  kernel_size,
                  strides=1,
@@ -109,6 +105,7 @@ class LocallyConnected1D(Layer):
         self.input_spec = InputSpec(ndim=3)
 
     def build(self, input_shape):
+        input_shape = tensor_shape.TensorShape(input_shape).as_list()
         input_dim = input_shape[2]
         if input_dim is None:
             raise ValueError('Axis 2 of input should be fully-defined. '
@@ -121,14 +118,14 @@ class LocallyConnected1D(Layer):
                              self.kernel_size[0] * input_dim,
                              self.filters)
         self.kernel = self.add_weight(
-            self.kernel_shape,
+            shape=self.kernel_shape,
             initializer=self.kernel_initializer,
             name='kernel',
             regularizer=self.kernel_regularizer,
             constraint=self.kernel_constraint)
         if self.use_bias:
             self.bias = self.add_weight(
-                (output_length, self.filters),
+                shape=(output_length, self.filters),
                 initializer=self.bias_initializer,
                 name='bias',
                 regularizer=self.bias_regularizer,
@@ -138,12 +135,13 @@ class LocallyConnected1D(Layer):
         self.input_spec = InputSpec(ndim=3, axes={2: input_dim})
         self.built = True
 
-    def compute_output_shape(self, input_shape):
+    def _compute_output_shape(self, input_shape):
+        input_shape = tensor_shape.TensorShape(input_shape).as_list()
         length = conv_utils.conv_output_length(input_shape[1],
                                                self.kernel_size[0],
                                                self.padding,
                                                self.strides[0])
-        return (input_shape[0], length, self.filters)
+        return tensor_shape.TensorShape([input_shape[0], length, self.filters])
 
     def call(self, inputs):
         stride = self.strides[0]
@@ -175,7 +173,7 @@ class LocallyConnected1D(Layer):
             'activation': activations.serialize(self.activation),
             'use_bias': self.use_bias,
             'kernel_initializer': initializers.serialize(self.kernel_initializer),
-            'bias_initializer': initializers.serialize(self.kernel_initializer),
+            'bias_initializer': initializers.serialize(self.bias_initializer),
             'kernel_regularizer': regularizers.serialize(self.kernel_regularizer),
             'bias_regularizer': regularizers.serialize(self.bias_regularizer),
             'activity_regularizer': regularizers.serialize(self.activity_regularizer),
@@ -219,40 +217,31 @@ class LocallyConnected2D(Layer):
             specifying the strides of the convolution along the width and height.
             Can be a single integer to specify the same value for
             all spatial dimensions.
-            Specifying any stride value != 1 is incompatible with specifying
-            any `dilation_rate` value != 1.
-        padding: one of `"valid"` or `"same"` (case-insensitive).
+        padding: Currently only support `"valid"` (case-insensitive).
+            `"same"` will be supported in future.
         data_format: A string,
             one of `channels_last` (default) or `channels_first`.
             The ordering of the dimensions in the inputs.
             `channels_last` corresponds to inputs with shape
-            `(batch, width, height, channels)` while `channels_first`
+            `(batch, height, width, channels)` while `channels_first`
             corresponds to inputs with shape
-            `(batch, channels, width, height)`.
+            `(batch, channels, height, width)`.
             It defaults to the `image_data_format` value found in your
             Keras config file at `~/.keras/keras.json`.
             If you never set it, then it will be "channels_last".
-        activation: Activation function to use
-            (see [activations](../activations.md)).
+        activation: Activation function to use.
             If you don't specify anything, no activation is applied
             (ie. "linear" activation: `a(x) = x`).
         use_bias: Boolean, whether the layer uses a bias vector.
-        kernel_initializer: Initializer for the `kernel` weights matrix
-            (see [initializers](../initializers.md)).
-        bias_initializer: Initializer for the bias vector
-            (see [initializers](../initializers.md)).
+        kernel_initializer: Initializer for the `kernel` weights matrix.
+        bias_initializer: Initializer for the bias vector.
         kernel_regularizer: Regularizer function applied to
-            the `kernel` weights matrix
-            (see [regularizer](../regularizers.md)).
-        bias_regularizer: Regularizer function applied to the bias vector
-            (see [regularizer](../regularizers.md)).
+            the `kernel` weights matrix.
+        bias_regularizer: Regularizer function applied to the bias vector.
         activity_regularizer: Regularizer function applied to
-            the output of the layer (its "activation").
-            (see [regularizer](../regularizers.md)).
-        kernel_constraint: Constraint function applied to the kernel matrix
-            (see [constraints](../constraints.md)).
-        bias_constraint: Constraint function applied to the bias vector
-            (see [constraints](../constraints.md)).
+            the output of the layer (its "activation")..
+        kernel_constraint: Constraint function applied to the kernel matrix.
+        bias_constraint: Constraint function applied to the bias vector.
 
     # Input shape
         4D tensor with shape:
@@ -268,7 +257,6 @@ class LocallyConnected2D(Layer):
         `rows` and `cols` values might have changed due to padding.
     """
 
-    @interfaces.legacy_conv2d_support
     def __init__(self, filters,
                  kernel_size,
                  strides=(1, 1),
@@ -305,6 +293,7 @@ class LocallyConnected2D(Layer):
         self.input_spec = InputSpec(ndim=4)
 
     def build(self, input_shape):
+        input_shape = tensor_shape.TensorShape(input_shape).as_list()
         if self.data_format == 'channels_last':
             input_row, input_col = input_shape[1:-1]
             input_filter = input_shape[3]
@@ -316,6 +305,7 @@ class LocallyConnected2D(Layer):
                              ' a LocallyConnected2D layer '
                              'should be fully-defined, but layer received '
                              'the inputs shape ' + str(input_shape))
+
         output_row = conv_utils.conv_output_length(input_row, self.kernel_size[0],
                                                    self.padding, self.strides[0])
         output_col = conv_utils.conv_output_length(input_col, self.kernel_size[1],
@@ -325,13 +315,13 @@ class LocallyConnected2D(Layer):
         self.kernel_shape = (output_row * output_col,
                              self.kernel_size[0] * self.kernel_size[1] * input_filter,
                              self.filters)
-        self.kernel = self.add_weight(self.kernel_shape,
+        self.kernel = self.add_weight(shape=self.kernel_shape,
                                       initializer=self.kernel_initializer,
                                       name='kernel',
                                       regularizer=self.kernel_regularizer,
                                       constraint=self.kernel_constraint)
         if self.use_bias:
-            self.bias = self.add_weight((output_row, output_col, self.filters),
+            self.bias = self.add_weight(shape=(output_row, output_col, self.filters),
                                         initializer=self.bias_initializer,
                                         name='bias',
                                         regularizer=self.bias_regularizer,
@@ -344,23 +334,23 @@ class LocallyConnected2D(Layer):
             self.input_spec = InputSpec(ndim=4, axes={-1: input_filter})
         self.built = True
 
-    def compute_output_shape(self, input_shape):
+    def _compute_output_shape(self, input_shape):
+        input_shape = tensor_shape.TensorShape(input_shape).as_list()
         if self.data_format == 'channels_first':
             rows = input_shape[2]
             cols = input_shape[3]
         elif self.data_format == 'channels_last':
             rows = input_shape[1]
             cols = input_shape[2]
-
         rows = conv_utils.conv_output_length(rows, self.kernel_size[0],
                                              self.padding, self.strides[0])
         cols = conv_utils.conv_output_length(cols, self.kernel_size[1],
                                              self.padding, self.strides[1])
 
         if self.data_format == 'channels_first':
-            return (input_shape[0], self.filters, rows, cols)
+            return tensor_shape.TensorShape([input_shape[0], self.filters, rows, cols])
         elif self.data_format == 'channels_last':
-            return (input_shape[0], rows, cols, self.filters)
+            return tensor_shape.TensorShape([input_shape[0], rows, cols, self.filters])
 
     def call(self, inputs):
         stride_row, stride_col = self.strides
@@ -432,7 +422,7 @@ class LocallyConnected2D(Layer):
             'activation': activations.serialize(self.activation),
             'use_bias': self.use_bias,
             'kernel_initializer': initializers.serialize(self.kernel_initializer),
-            'bias_initializer': initializers.serialize(self.kernel_initializer),
+            'bias_initializer': initializers.serialize(self.bias_initializer),
             'kernel_regularizer': regularizers.serialize(self.kernel_regularizer),
             'bias_regularizer': regularizers.serialize(self.bias_regularizer),
             'activity_regularizer': regularizers.serialize(self.activity_regularizer),

keras/layers/merge.py

@@ -1,5 +1,12 @@
-from ..engine.topology import Layer
+"""Layers can merge several input tensors into a single output tensor.
+"""
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
 from .. import backend as K
+from ..engine.topology import Layer
+from tensorflow.python.framework import tensor_shape
 
 
 class _Merge(Layer):
@@ -18,6 +25,44 @@ class _Merge(Layer):
     def _merge_function(self, inputs):
         raise NotImplementedError
 
+    def _compute_elemwise_op_output_shape(self, shape1, shape2):
+        """Computes the shape of the resultant of an elementwise operation.
+
+        # Arguments
+            shape1: tuple or None. Shape of the first tensor
+            shape2: tuple or None. Shape of the second tensor
+
+        # Returns
+            expected output shape when an element-wise operation is
+            carried out on 2 tensors with shapes shape1 and shape2.
+            tuple or None.
+
+        # Raises
+            ValueError: if shape1 and shape2 are not compatible for
+                element-wise operations.
+        """
+        if None in [shape1, shape2]:
+            return None
+        elif len(shape1) < len(shape2):
+            return self._compute_elemwise_op_output_shape(shape2, shape1)
+        elif not shape2:
+            return shape1
+        output_shape = list(shape1[:-len(shape2)])
+        for i, j in zip(shape1[-len(shape2):], shape2):
+            if i is None or j is None:
+                output_shape.append(None)
+            elif i == 1:
+                output_shape.append(j)
+            elif j == 1:
+                output_shape.append(i)
+            else:
+                if i != j:
+                    raise ValueError('Operands could not be broadcast '
+                                     'together with shapes ' +
+                                     str(shape1) + ' ' + str(shape2))
+                output_shape.append(i)
+        return tuple(output_shape)
+
     def build(self, input_shape):
         # Used purely for shape validation.
         if not isinstance(input_shape, list):
@@ -27,19 +72,105 @@ class _Merge(Layer):
             raise ValueError('A merge layer should be called '
                              'on a list of at least 2 inputs. '
                              'Got ' + str(len(input_shape)) + ' inputs.')
-        if all([shape is None for shape in input_shape]):
-            return
-        # TODO: handle shapes with None entries.
-        input_shapes_set = set(input_shape)
-        if None in input_shapes_set:
-            input_shapes_set.remove(None)
-        if len(input_shapes_set) > 1:
-            raise ValueError('Only tensors of same shape can '
-                             'be merged by layer' + self.name +
-                             ' Got input shapes: %s' % input_shape)
+        batch_sizes = [s[0] for s in input_shape if s is not None]
+        batch_sizes = set(batch_sizes)
+        batch_sizes -= set([None])
+        if len(batch_sizes) > 1:
+            raise ValueError('Can not merge tensors with different '
+                             'batch sizes. Got tensors with shapes : ' +
+                             str(input_shape))
+        if input_shape[0] is None:
+            output_shape = None
+        else:
+            output_shape = input_shape[0][1:]
+        for i in range(1, len(input_shape)):
+            if input_shape[i] is None:
+                shape = None
+            else:
+                shape = input_shape[i][1:]
+            output_shape = self._compute_elemwise_op_output_shape(output_shape, shape)
+        # If the inputs have different ranks, we have to reshape them
+        # to make them broadcastable.
+        if None not in input_shape and len(set(map(len, input_shape))) == 1:
+            self._reshape_required = False
+        else:
+            self._reshape_required = True
 
     def call(self, inputs):
-        return self._merge_function(inputs)
+        if self._reshape_required:
+            reshaped_inputs = []
+            input_ndims = list(map(K.ndim, inputs))
+            if None not in input_ndims:
+                # If ranks of all inputs are available,
+                # we simply expand each of them at axis=1
+                # until all of them have the same rank.
+                max_ndim = max(input_ndims)
+                for x in inputs:
+                    x_ndim = K.ndim(x)
+                    for _ in range(max_ndim - x_ndim):
+                        x = K.expand_dims(x, 1)
+                    reshaped_inputs.append(x)
+                return self._merge_function(reshaped_inputs)
+            else:
+                # Transpose all inputs so that batch size is the last dimension.
+                # (batch_size, dim1, dim2, ... ) -> (dim1, dim2, ... , batch_size)
+                transposed = False
+                for x in inputs:
+                    x_ndim = K.ndim(x)
+                    if x_ndim is None:
+                        x_shape = K.shape(x)
+                        batch_size = x_shape[0]
+                        new_shape = K.concatenate([x_shape[1:], K.expand_dims(batch_size)])
+                        x_transposed = K.reshape(x, K.stack([batch_size, K.prod(x_shape[1:])]))
+                        x_transposed = K.permute_dimensions(x_transposed, (1, 0))
+                        x_transposed = K.reshape(x_transposed, new_shape)
+                        reshaped_inputs.append(x_transposed)
+                        transposed = True
+                    elif x_ndim > 1:
+                        dims = list(range(1, x_ndim)) + [0]
+                        reshaped_inputs.append(K.permute_dimensions(x, dims))
+                        transposed = True
+                    else:
+                        # We don't transpose inputs if they are 1D vectors or scalars.
+                        reshaped_inputs.append(x)
+                y = self._merge_function(reshaped_inputs)
+                y_ndim = K.ndim(y)
+                if transposed:
+                    # If inputs have been transposed, we have to transpose the output too.
+                    if y_ndim is None:
+                        y_shape = K.shape(y)
+                        y_ndim = K.shape(y_shape)[0]
+                        batch_size = y_shape[y_ndim - 1]
+                        new_shape = K.concatenate([K.expand_dims(batch_size), y_shape[:y_ndim - 1]])
+                        y = K.reshape(y, (-1, batch_size))
+                        y = K.permute_dimensions(y, (1, 0))
+                        y = K.reshape(y, new_shape)
+                    elif y_ndim > 1:
+                        dims = [y_ndim - 1] + list(range(y_ndim - 1))
+                        y = K.permute_dimensions(y, dims)
+                return y
+        else:
+            return self._merge_function(inputs)
+
+    def compute_output_shape(self, input_shape):
+        if input_shape[0] is None:
+            output_shape = None
+        else:
+            output_shape = input_shape[0][1:]
+        for i in range(1, len(input_shape)):
+            if input_shape[i] is None:
+                shape = None
+            else:
+                shape = input_shape[i][1:]
+            output_shape = self._compute_elemwise_op_output_shape(output_shape, shape)
+        batch_sizes = [s[0] for s in input_shape if s is not None]
+        batch_sizes = set(batch_sizes)
+        batch_sizes -= set([None])
+        if len(batch_sizes) == 1:
+            output_shape = (list(batch_sizes)[0],) + output_shape
+        else:
+            output_shape = (None,) + output_shape
+        return output_shape
 
     def compute_mask(self, inputs, mask=None):
         if mask is None:
@@ -141,7 +272,7 @@ class Concatenate(_Merge):
                              'on a list of inputs')
         if all([shape is None for shape in input_shape]):
             return
-        reduced_inputs_shapes = [list(shape) for shape in input_shape]
+        reduced_inputs_shapes = [tensor_shape.TensorShape(shape).as_list() for shape in input_shape]
         shape_set = set()
         for i in range(len(reduced_inputs_shapes)):
             del reduced_inputs_shapes[i][self.axis]
@@ -158,18 +289,19 @@ class Concatenate(_Merge):
                              'on a list of inputs.')
         return K.concatenate(inputs, axis=self.axis)
 
-    def compute_output_shape(self, input_shape):
+    def _compute_output_shape(self, input_shape):
         if not isinstance(input_shape, list):
             raise ValueError('A `Concatenate` layer should be called '
                              'on a list of inputs.')
         input_shapes = input_shape
-        output_shape = list(input_shapes[0])
+        output_shape = tensor_shape.TensorShape(input_shapes[0]).as_list()
         for shape in input_shapes[1:]:
+            shape = tensor_shape.TensorShape(shape).as_list()
             if output_shape[self.axis] is None or shape[self.axis] is None:
                 output_shape[self.axis] = None
                 break
             output_shape[self.axis] += shape[self.axis]
-        return tuple(output_shape)
+        return tensor_shape.TensorShape(output_shape)
 
     def compute_mask(self, inputs, mask=None):
         if mask is None:
@@ -190,8 +322,8 @@ class Concatenate(_Merge):
         for input_i, mask_i in zip(inputs, mask):
             if mask_i is None:
                 # Input is unmasked. Append all 1s to masks,
-                # but cast it to uint8 first
-                masks.append(K.cast(K.ones_like(input_i), 'uint8'))
+                # but cast it to bool first
+                masks.append(K.cast(K.ones_like(input_i), 'bool'))
             elif K.ndim(mask_i) < K.ndim(input_i):
                 # Mask is smaller than the input, expand it
                 masks.append(K.expand_dims(mask_i))
@@ -247,8 +379,8 @@ class Dot(_Merge):
         if not isinstance(input_shape, list) or len(input_shape) != 2:
             raise ValueError('A `Dot` layer should be called '
                              'on a list of 2 inputs.')
-        shape1 = input_shape[0]
-        shape2 = input_shape[1]
+        shape1 = tensor_shape.TensorShape(input_shape[0]).as_list()
+        shape2 = tensor_shape.TensorShape(input_shape[1]).as_list()
         if shape1 is None or shape2 is None:
             return
         if isinstance(self.axes, int):
@@ -285,12 +417,12 @@ class Dot(_Merge):
         output = K.batch_dot(x1, x2, axes)
         return output
 
-    def compute_output_shape(self, input_shape):
+    def _compute_output_shape(self, input_shape):
         if not isinstance(input_shape, list) or len(input_shape) != 2:
             raise ValueError('A `Dot` layer should be called '
                              'on a list of 2 inputs.')
-        shape1 = list(input_shape[0])
-        shape2 = list(input_shape[1])
+        shape1 = tensor_shape.TensorShape(input_shape[0]).as_list()
+        shape2 = tensor_shape.TensorShape(input_shape[1]).as_list()
         if isinstance(self.axes, int):
             if self.axes < 0:
                 axes = [self.axes % len(shape1), self.axes % len(shape2)]
@@ -304,7 +436,7 @@ class Dot(_Merge):
         output_shape = shape1 + shape2
         if len(output_shape) == 1:
             output_shape += [1]
-        return tuple(output_shape)
+        return tensor_shape.TensorShape(output_shape)
 
     def compute_mask(self, inputs, mask=None):
         return None

keras/layers/noise.py

@@ -1,10 +1,13 @@
 # -*- coding: utf-8 -*-
+"""Layers for regularization models via the addition of noise.
+"""
 from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
 
-from ..engine import Layer
 from .. import backend as K
+from ..engine import Layer
 import numpy as np
-from ..legacy import interfaces
 
 
 class GaussianNoise(Layer):
@@ -29,7 +32,6 @@ class GaussianNoise(Layer):
         Same shape as input.
     """
 
-    @interfaces.legacy_gaussiannoise_support
     def __init__(self, stddev, **kwargs):
         super(GaussianNoise, self).__init__(**kwargs)
         self.supports_masking = True
@@ -70,7 +72,6 @@ class GaussianDropout(Layer):
         - [Dropout: A Simple Way to Prevent Neural Networks from Overfitting Srivastava, Hinton, et al. 2014](http://www.cs.toronto.edu/~rsalakhu/papers/srivastava14a.pdf)
     """
 
-    @interfaces.legacy_gaussiandropout_support
     def __init__(self, rate, **kwargs):
         super(GaussianDropout, self).__init__(**kwargs)
         self.supports_masking = True

keras/layers/normalization.py

@@ -1,12 +1,17 @@
 # -*- coding: utf-8 -*-
+"""Normalization layers.
+"""
 from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
 
-from ..engine import Layer, InputSpec
+from .. import backend as K
+from .. import constraints
 from .. import initializers
 from .. import regularizers
-from .. import constraints
-from .. import backend as K
-from ..legacy import interfaces
+from ..engine import InputSpec
+from ..engine import Layer
+from tensorflow.python.framework import tensor_shape
 
 
 class BatchNormalization(Layer):
@@ -52,7 +57,6 @@ class BatchNormalization(Layer):
         - [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://arxiv.org/abs/1502.03167)
     """
 
-    @interfaces.legacy_batchnorm_support
     def __init__(self,
                  axis=-1,
                  momentum=0.99,
@@ -85,6 +89,7 @@ class BatchNormalization(Layer):
         self.gamma_constraint = constraints.get(gamma_constraint)
 
     def build(self, input_shape):
+        input_shape = tensor_shape.TensorShape(input_shape).as_list()
         dim = input_shape[self.axis]
         if dim is None:
             raise ValueError('Axis ' + str(self.axis) + ' of '
@@ -96,7 +101,7 @@ class BatchNormalization(Layer):
         shape = (dim,)
 
         if self.scale:
-            self.gamma = self.add_weight(shape,
+            self.gamma = self.add_weight(shape=shape,
                                          name='gamma',
                                          initializer=self.gamma_initializer,
                                          regularizer=self.gamma_regularizer,
@@ -104,7 +109,7 @@ class BatchNormalization(Layer):
         else:
             self.gamma = None
         if self.center:
-            self.beta = self.add_weight(shape,
+            self.beta = self.add_weight(shape=shape,
                                         name='beta',
                                         initializer=self.beta_initializer,
                                         regularizer=self.beta_regularizer,
@@ -112,19 +117,19 @@ class BatchNormalization(Layer):
         else:
             self.beta = None
         self.moving_mean = self.add_weight(
-            shape,
+            shape=shape,
             name='moving_mean',
             initializer=self.moving_mean_initializer,
             trainable=False)
         self.moving_variance = self.add_weight(
-            shape,
+            shape=shape,
             name='moving_variance',
             initializer=self.moving_variance_initializer,
             trainable=False)
         self.built = True
 
     def call(self, inputs, training=None):
-        input_shape = K.int_shape(inputs)
+        input_shape = inputs.get_shape().as_list()
         # Prepare broadcasting shape.
         ndim = len(input_shape)
         reduction_axes = list(range(len(input_shape)))
@@ -133,57 +138,59 @@ class BatchNormalization(Layer):
         broadcast_shape[self.axis] = input_shape[self.axis]
 
         # Determines whether broadcasting is needed.
-        needs_broadcasting = (sorted(reduction_axes) != range(ndim)[:-1])
+        needs_broadcasting = (sorted(reduction_axes) != list(range(ndim))[:-1])
 
-        normed, mean, variance = K.normalize_batch_in_training(
+        def normalize_inference():
+            if needs_broadcasting:
+                # In this case we must explictly broadcast all parameters.
+                broadcast_moving_mean = K.reshape(self.moving_mean,
+                                                  broadcast_shape)
+                broadcast_moving_variance = K.reshape(self.moving_variance,
+                                                      broadcast_shape)
+                if self.center:
+                    broadcast_beta = K.reshape(self.beta, broadcast_shape)
+                else:
+                    broadcast_beta = None
+                if self.scale:
+                    broadcast_gamma = K.reshape(self.gamma,
+                                                broadcast_shape)
+                else:
+                    broadcast_gamma = None
+                return K.batch_normalization(
+                    inputs,
+                    broadcast_moving_mean,
+                    broadcast_moving_variance,
+                    broadcast_beta,
+                    broadcast_gamma,
+                    epsilon=self.epsilon)
+            else:
+                return K.batch_normalization(
+                    inputs,
+                    self.moving_mean,
+                    self.moving_variance,
+                    self.beta,
+                    self.gamma,
+                    epsilon=self.epsilon)
+
+        # If the learning phase is *static* and set to inference:
+        if training in {0, False}:
+            return normalize_inference()
+
+        # If the learning is either dynamic, or set to training:
+        normed_training, mean, variance = K.normalize_batch_in_training(
             inputs, self.gamma, self.beta, reduction_axes,
             epsilon=self.epsilon)
 
-        if training in {0, False}:
-            return normed
-        else:
-            self.add_update([K.moving_average_update(self.moving_mean,
-                                                     mean,
-                                                     self.momentum),
-                             K.moving_average_update(self.moving_variance,
-                                                     variance,
-                                                     self.momentum)],
-                            inputs)
-
-            def normalize_inference():
-                if needs_broadcasting:
-                    # In this case we must explictly broadcast all parameters.
-                    broadcast_moving_mean = K.reshape(self.moving_mean,
-                                                      broadcast_shape)
-                    broadcast_moving_variance = K.reshape(self.moving_variance,
-                                                          broadcast_shape)
-                    if self.center:
-                        broadcast_beta = K.reshape(self.beta, broadcast_shape)
-                    else:
-                        broadcast_beta = None
-                    if self.scale:
-                        broadcast_gamma = K.reshape(self.gamma,
-                                                    broadcast_shape)
-                    else:
-                        broadcast_gamma = None
-                    return K.batch_normalization(
-                        inputs,
-                        broadcast_moving_mean,
-                        broadcast_moving_variance,
-                        broadcast_beta,
-                        broadcast_gamma,
-                        epsilon=self.epsilon)
-                else:
-                    return K.batch_normalization(
-                        inputs,
-                        self.moving_mean,
-                        self.moving_variance,
-                        self.beta,
-                        self.gamma,
-                        epsilon=self.epsilon)
+        self.add_update([K.moving_average_update(self.moving_mean,
+                                                 mean,
+                                                 self.momentum),
+                         K.moving_average_update(self.moving_variance,
+                                                 variance,
+                                                 self.momentum)],
+                        inputs)
 
         # Pick the normalized form corresponding to the training phase.
-        return K.in_train_phase(normed,
+        return K.in_train_phase(normed_training,
                                 normalize_inference,
                                 training=training)
 

keras/layers/pooling.py

@@ -1,11 +1,15 @@
 # -*- coding: utf-8 -*-
+"""Pooling layers.
+"""
 from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
 
 from .. import backend as K
-from ..engine import Layer
 from ..engine import InputSpec
+from ..engine import Layer
+from tensorflow.python.framework import tensor_shape
 from ..utils import conv_utils
-from ..legacy import interfaces
 
 
 class _Pooling1D(Layer):
@@ -22,12 +26,13 @@ class _Pooling1D(Layer):
         self.padding = conv_utils.normalize_padding(padding)
         self.input_spec = InputSpec(ndim=3)
 
-    def compute_output_shape(self, input_shape):
+    def _compute_output_shape(self, input_shape):
+        input_shape = tensor_shape.TensorShape(input_shape).as_list()
         length = conv_utils.conv_output_length(input_shape[1],
                                                self.pool_size[0],
                                                self.padding,
                                                self.strides[0])
-        return (input_shape[0], length, input_shape[2])
+        return tensor_shape.TensorShape([input_shape[0], length, input_shape[2]])
 
     def _pooling_function(self, inputs, pool_size, strides,
                           padding, data_format):
@@ -67,7 +72,6 @@ class MaxPooling1D(_Pooling1D):
         3D tensor with shape: `(batch_size, downsampled_steps, features)`.
     """
 
-    @interfaces.legacy_pooling1d_support
     def __init__(self, pool_size=2, strides=None,
                  padding='valid', **kwargs):
         super(MaxPooling1D, self).__init__(pool_size, strides,
@@ -97,7 +101,6 @@ class AveragePooling1D(_Pooling1D):
         3D tensor with shape: `(batch_size, downsampled_steps, features)`.
     """
 
-    @interfaces.legacy_pooling1d_support
     def __init__(self, pool_size=2, strides=None,
                  padding='valid', **kwargs):
         super(AveragePooling1D, self).__init__(pool_size, strides,
@@ -126,11 +129,12 @@ class _Pooling2D(Layer):
         self.data_format = conv_utils.normalize_data_format(data_format)
         self.input_spec = InputSpec(ndim=4)
 
-    def compute_output_shape(self, input_shape):
+    def _compute_output_shape(self, input_shape):
+        input_shape = tensor_shape.TensorShape(input_shape).as_list()
         if self.data_format == 'channels_first':
             rows = input_shape[2]
             cols = input_shape[3]
-        elif self.data_format == 'channels_last':
+        else:
             rows = input_shape[1]
             cols = input_shape[2]
         rows = conv_utils.conv_output_length(rows, self.pool_size[0],
@@ -138,9 +142,9 @@ class _Pooling2D(Layer):
         cols = conv_utils.conv_output_length(cols, self.pool_size[1],
                                              self.padding, self.strides[1])
         if self.data_format == 'channels_first':
-            return (input_shape[0], input_shape[1], rows, cols)
-        elif self.data_format == 'channels_last':
-            return (input_shape[0], rows, cols, input_shape[3])
+            return tensor_shape.TensorShape([input_shape[0], input_shape[1], rows, cols])
+        else:
+            return tensor_shape.TensorShape([input_shape[0], rows, cols, input_shape[3]])
 
     def _pooling_function(self, inputs, pool_size, strides,
                           padding, data_format):
@@ -180,9 +184,9 @@ class MaxPooling2D(_Pooling2D):
             one of `channels_last` (default) or `channels_first`.
             The ordering of the dimensions in the inputs.
             `channels_last` corresponds to inputs with shape
-            `(batch, width, height, channels)` while `channels_first`
+            `(batch, height, width, channels)` while `channels_first`
             corresponds to inputs with shape
-            `(batch, channels, width, height)`.
+            `(batch, channels, height, width)`.
             It defaults to the `image_data_format` value found in your
             Keras config file at `~/.keras/keras.json`.
             If you never set it, then it will be "channels_last".
@@ -204,7 +208,6 @@ class MaxPooling2D(_Pooling2D):
             `(batch_size, channels, pooled_rows, pooled_cols)`
     """
 
-    @interfaces.legacy_pooling2d_support
     def __init__(self, pool_size=(2, 2), strides=None, padding='valid',
                  data_format=None, **kwargs):
         super(MaxPooling2D, self).__init__(pool_size, strides, padding,
@@ -235,9 +238,9 @@ class AveragePooling2D(_Pooling2D):
             one of `channels_last` (default) or `channels_first`.
             The ordering of the dimensions in the inputs.
             `channels_last` corresponds to inputs with shape
-            `(batch, width, height, channels)` while `channels_first`
+            `(batch, height, width, channels)` while `channels_first`
             corresponds to inputs with shape
-            `(batch, channels, width, height)`.
+            `(batch, channels, height, width)`.
             It defaults to the `image_data_format` value found in your
             Keras config file at `~/.keras/keras.json`.
             If you never set it, then it will be "channels_last".
@@ -259,7 +262,6 @@ class AveragePooling2D(_Pooling2D):
             `(batch_size, channels, pooled_rows, pooled_cols)`
     """
 
-    @interfaces.legacy_pooling2d_support
     def __init__(self, pool_size=(2, 2), strides=None, padding='valid',
                  data_format=None, **kwargs):
         super(AveragePooling2D, self).__init__(pool_size, strides, padding,
@@ -287,12 +289,13 @@ class _Pooling3D(Layer):
         self.data_format = conv_utils.normalize_data_format(data_format)
         self.input_spec = InputSpec(ndim=5)
 
-    def compute_output_shape(self, input_shape):
+    def _compute_output_shape(self, input_shape):
+        input_shape = tensor_shape.TensorShape(input_shape).as_list()
         if self.data_format == 'channels_first':
             len_dim1 = input_shape[2]
             len_dim2 = input_shape[3]
             len_dim3 = input_shape[4]
-        elif self.data_format == 'channels_last':
+        else:
             len_dim1 = input_shape[1]
             len_dim2 = input_shape[2]
             len_dim3 = input_shape[3]
@@ -303,13 +306,17 @@ class _Pooling3D(Layer):
         len_dim3 = conv_utils.conv_output_length(len_dim3, self.pool_size[2],
                                                  self.padding, self.strides[2])
         if self.data_format == 'channels_first':
-            return (input_shape[0],
-                    input_shape[1],
-                    len_dim1, len_dim2, len_dim3)
-        elif self.data_format == 'channels_last':
-            return (input_shape[0],
-                    len_dim1, len_dim2, len_dim3,
-                    input_shape[4])
+            return tensor_shape.TensorShape([input_shape[0],
+                                   input_shape[1],
+                                   len_dim1,
+                                   len_dim2,
+                                   len_dim3])
+        else:
+            return tensor_shape.TensorShape([input_shape[0],
+                                   len_dim1,
+                                   len_dim2,
+                                   len_dim3,
+                                   input_shape[4]])
 
     def _pooling_function(self, inputs, pool_size, strides,
                           padding, data_format):
@@ -369,7 +376,6 @@ class MaxPooling3D(_Pooling3D):
             `(batch_size, channels, pooled_dim1, pooled_dim2, pooled_dim3)`
     """
 
-    @interfaces.legacy_pooling3d_support
     def __init__(self, pool_size=(2, 2, 2), strides=None, padding='valid',
                  data_format=None, **kwargs):
         super(MaxPooling3D, self).__init__(pool_size, strides, padding,
@@ -419,7 +425,6 @@ class AveragePooling3D(_Pooling3D):
             `(batch_size, channels, pooled_dim1, pooled_dim2, pooled_dim3)`
     """
 
-    @interfaces.legacy_pooling3d_support
     def __init__(self, pool_size=(2, 2, 2), strides=None, padding='valid',
                  data_format=None, **kwargs):
         super(AveragePooling3D, self).__init__(pool_size, strides, padding,
@@ -441,8 +446,9 @@ class _GlobalPooling1D(Layer):
         super(_GlobalPooling1D, self).__init__(**kwargs)
         self.input_spec = InputSpec(ndim=3)
 
-    def compute_output_shape(self, input_shape):
-        return (input_shape[0], input_shape[2])
+    def _compute_output_shape(self, input_shape):
+        input_shape = tensor_shape.TensorShape(input_shape).as_list()
+        return tensor_shape.TensorShape([input_shape[0], input_shape[2]])
 
     def call(self, inputs):
         raise NotImplementedError
@@ -482,17 +488,17 @@ class _GlobalPooling2D(Layer):
     """Abstract class for different global pooling 2D layers.
     """
 
-    @interfaces.legacy_global_pooling_support
     def __init__(self, data_format=None, **kwargs):
         super(_GlobalPooling2D, self).__init__(**kwargs)
         self.data_format = conv_utils.normalize_data_format(data_format)
         self.input_spec = InputSpec(ndim=4)
 
-    def compute_output_shape(self, input_shape):
+    def _compute_output_shape(self, input_shape):
+        input_shape = tensor_shape.TensorShape(input_shape).as_list()
         if self.data_format == 'channels_last':
-            return (input_shape[0], input_shape[3])
+            return tensor_shape.TensorShape([input_shape[0], input_shape[3]])
         else:
-            return (input_shape[0], input_shape[1])
+            return tensor_shape.TensorShape([input_shape[0], input_shape[1]])
 
     def call(self, inputs):
         raise NotImplementedError
@@ -511,9 +517,9 @@ class GlobalAveragePooling2D(_GlobalPooling2D):
             one of `channels_last` (default) or `channels_first`.
             The ordering of the dimensions in the inputs.
             `channels_last` corresponds to inputs with shape
-            `(batch, width, height, channels)` while `channels_first`
+            `(batch, height, width, channels)` while `channels_first`
             corresponds to inputs with shape
-            `(batch, channels, width, height)`.
+            `(batch, channels, height, width)`.
             It defaults to the `image_data_format` value found in your
             Keras config file at `~/.keras/keras.json`.
             If you never set it, then it will be "channels_last".
@@ -546,9 +552,9 @@ class GlobalMaxPooling2D(_GlobalPooling2D):
             one of `channels_last` (default) or `channels_first`.
             The ordering of the dimensions in the inputs.
             `channels_last` corresponds to inputs with shape
-            `(batch, width, height, channels)` while `channels_first`
+            `(batch, height, width, channels)` while `channels_first`
             corresponds to inputs with shape
-            `(batch, channels, width, height)`.
+            `(batch, channels, height, width)`.
             It defaults to the `image_data_format` value found in your
             Keras config file at `~/.keras/keras.json`.
             If you never set it, then it will be "channels_last".
@@ -577,17 +583,17 @@ class _GlobalPooling3D(Layer):
     """Abstract class for different global pooling 3D layers.
     """
 
-    @interfaces.legacy_global_pooling_support
     def __init__(self, data_format=None, **kwargs):
         super(_GlobalPooling3D, self).__init__(**kwargs)
         self.data_format = conv_utils.normalize_data_format(data_format)
         self.input_spec = InputSpec(ndim=5)
 
-    def compute_output_shape(self, input_shape):
+    def _compute_output_shape(self, input_shape):
+        input_shape = tensor_shape.TensorShape(input_shape).as_list()
         if self.data_format == 'channels_last':
-            return (input_shape[0], input_shape[4])
+            return tensor_shape.TensorShape([input_shape[0], input_shape[4]])
         else:
-            return (input_shape[0], input_shape[1])
+            return tensor_shape.TensorShape([input_shape[0], input_shape[1]])
 
     def call(self, inputs):
         raise NotImplementedError

keras/layers/recurrent.py

@@ -1,16 +1,20 @@
 # -*- coding: utf-8 -*-
+"""Recurrent layers.
+"""
 from __future__ import absolute_import
-import numpy as np
+from __future__ import division
+from __future__ import print_function
 
-from .. import backend as K
 from .. import activations
+from .. import backend as K
+from .. import constraints
 from .. import initializers
 from .. import regularizers
-from .. import constraints
-from ..engine import Layer
 from ..engine import InputSpec
-from ..legacy import interfaces
-
+from ..engine import Layer
+import numpy as np
+from tensorflow.python.framework import tensor_shape
+# pylint: disable=access-member-before-definition
 
 def _time_distributed_dense(x, w, b=None, dropout=None,
                             input_dim=None, output_dim=None,
@@ -78,12 +82,16 @@ class Recurrent(Layer):
         # now model.output_shape == (None, 32)
         # note: `None` is the batch dimension.
 
-        # the following is identical:
-        model = Sequential()
-        model.add(LSTM(32, input_dim=64, input_length=10))
-
-        # for subsequent layers, not need to specify the input size:
+        # for subsequent layers, no need to specify the input size:
         model.add(LSTM(16))
+
+        # to stack recurrent layers, you must use return_sequences=True
+        # on any recurrent layer that feeds into another recurrent layer.
+        # note that you only need to specify the input size on the first layer.
+        model = Sequential()
+        model.add(LSTM(64, input_dim=64, input_length=10, return_sequences=True))
+        model.add(LSTM(32, return_sequences=True))
+        model.add(LSTM(10))
     ```
 
     # Arguments
@@ -93,7 +101,8 @@ class Recurrent(Layer):
         return_sequences: Boolean. Whether to return the last output
             in the output sequence, or the full sequence.
         go_backwards: Boolean (default False).
-            If True, process the input sequence backwards.
+            If True, process the input sequence backwards and return the
+            reversed sequence.
         stateful: Boolean (default False). If True, the last state
             for each sample at index i in a batch will be used as initial
             state for the sample of index i in the following batch.
@@ -141,7 +150,7 @@ class Recurrent(Layer):
     # Masking
         This layer supports masking for input data with a variable number
         of timesteps. To introduce masks to your data,
-        use an [Embedding](embeddings.md) layer with the `mask_zero` parameter
+        use an `Embedding` layer with the `mask_zero` parameter
         set to `True`.
 
     # Note on using statefulness in RNNs
@@ -165,11 +174,16 @@ class Recurrent(Layer):
         To reset the states of your model, call `.reset_states()` on either
         a specific layer, or on your entire model.
 
-    # Note on specifying initial states in RNNs
-        You can specify the initial state of RNN layers by calling them with
-        the keyword argument `initial_state`. The value of `initial_state`
-        should be a tensor or list of tensors representing the initial state
-        of the RNN layer.
+    # Note on specifying the initial state of RNNs
+        You can specify the initial state of RNN layers symbolically by
+        calling them with the keyword argument `initial_state`. The value of
+        `initial_state` should be a tensor or list of tensors representing
+        the initial state of the RNN layer.
+
+        You can specify the initial state of RNN layers numerically by
+        calling `reset_states` with the keyword argument `states`. The value of
+        `states` should be a numpy array or list of numpy arrays representing
+        the initial state of the RNN layer.
     """
 
     def __init__(self, return_sequences=False,
@@ -185,21 +199,24 @@ class Recurrent(Layer):
         self.unroll = unroll
         self.implementation = implementation
         self.supports_masking = True
-        self.input_spec = InputSpec(ndim=3)
+        self.input_spec = [InputSpec(ndim=3)]
         self.state_spec = None
         self.dropout = 0
         self.recurrent_dropout = 0
 
-    def compute_output_shape(self, input_shape):
+    def _compute_output_shape(self, input_shape):
         if isinstance(input_shape, list):
             input_shape = input_shape[0]
+        input_shape = tensor_shape.TensorShape(input_shape).as_list()
         if self.return_sequences:
-            return (input_shape[0], input_shape[1], self.units)
+            return tensor_shape.TensorShape([input_shape[0], input_shape[1], self.units])
         else:
-            return (input_shape[0], self.units)
+            return tensor_shape.TensorShape([input_shape[0], self.units])
 
     def compute_mask(self, inputs, mask):
         if self.return_sequences:
+            if isinstance(mask, list):
+                return mask[0]
             return mask
         else:
             return None
@@ -210,14 +227,14 @@ class Recurrent(Layer):
     def get_constants(self, inputs, training=None):
         return []
 
-    def get_initial_states(self, inputs):
+    def get_initial_state(self, inputs):
         # build an all-zero tensor of shape (samples, output_dim)
         initial_state = K.zeros_like(inputs)  # (samples, timesteps, input_dim)
         initial_state = K.sum(initial_state, axis=(1, 2))  # (samples,)
         initial_state = K.expand_dims(initial_state)  # (samples, 1)
         initial_state = K.tile(initial_state, [1, self.units])  # (samples, output_dim)
-        initial_states = [initial_state for _ in range(len(self.states))]
-        return initial_states
+        initial_state = [initial_state for _ in range(len(self.states))]
+        return initial_state
 
     def preprocess_input(self, inputs, training=None):
         return inputs
@@ -227,51 +244,63 @@ class Recurrent(Layer):
         # and if it a Keras tensor,
         # then add it to the inputs and temporarily
         # modify the input spec to include the state.
-        if initial_state is not None:
-            if hasattr(initial_state, '_keras_history'):
-                # Compute the full input spec, including state
-                input_spec = self.input_spec
-                state_spec = self.state_spec
-                if not isinstance(state_spec, list):
-                    state_spec = [state_spec]
-                self.input_spec = [input_spec] + state_spec
+        if initial_state is None:
+            return super(Recurrent, self).__call__(inputs, **kwargs)
 
-                # Compute the full inputs, including state
-                if not isinstance(initial_state, (list, tuple)):
-                    initial_state = [initial_state]
-                inputs = [inputs] + list(initial_state)
+        if not isinstance(initial_state, (list, tuple)):
+            initial_state = [initial_state]
 
-                # Perform the call
-                output = super(Recurrent, self).__call__(inputs, **kwargs)
+        is_keras_tensor = hasattr(initial_state[0], '_keras_history')
+        for tensor in initial_state:
+            if hasattr(tensor, '_keras_history') != is_keras_tensor:
+                raise ValueError('The initial state of an RNN layer cannot be'
+                                 ' specified with a mix of Keras tensors and'
+                                 ' non-Keras tensors')
 
-                # Restore original input spec
-                self.input_spec = input_spec
-                return output
-            else:
-                kwargs['initial_state'] = initial_state
-        return super(Recurrent, self).__call__(inputs, **kwargs)
+        if is_keras_tensor:
+            # Compute the full input spec, including state
+            input_spec = self.input_spec
+            state_spec = self.state_spec
+            if not isinstance(input_spec, list):
+                input_spec = [input_spec]
+            if not isinstance(state_spec, list):
+                state_spec = [state_spec]
+            self.input_spec = input_spec + state_spec
 
-    def call(self, inputs, mask=None, initial_state=None, training=None):
+            # Compute the full inputs, including state
+            inputs = [inputs] + list(initial_state)
+
+            # Perform the call
+            output = super(Recurrent, self).__call__(inputs, **kwargs)
+
+            # Restore original input spec
+            self.input_spec = input_spec
+            return output
+        else:
+            kwargs['initial_state'] = initial_state
+            return super(Recurrent, self).__call__(inputs, **kwargs)
+
+    def call(self, inputs, mask=None, training=None, initial_state=None):
         # input shape: `(samples, time (padded with zeros), input_dim)`
         # note that the .build() method of subclasses MUST define
         # self.input_spec and self.state_spec with complete input shapes.
-        if initial_state is not None:
-            if not isinstance(initial_state, (list, tuple)):
-                initial_states = [initial_state]
-            else:
-                initial_states = list(initial_state)
         if isinstance(inputs, list):
-            initial_states = inputs[1:]
+            initial_state = inputs[1:]
             inputs = inputs[0]
+        elif initial_state is not None:
+            pass
         elif self.stateful:
-            initial_states = self.states
+            initial_state = self.states
         else:
-            initial_states = self.get_initial_states(inputs)
+            initial_state = self.get_initial_state(inputs)
 
-        if len(initial_states) != len(self.states):
+        if isinstance(mask, list):
+            mask = mask[0]
+
+        if len(initial_state) != len(self.states):
             raise ValueError('Layer has ' + str(len(self.states)) +
                              ' states but was passed ' +
-                             str(len(initial_states)) +
+                             str(len(initial_state)) +
                              ' initial states.')
         input_shape = K.int_shape(inputs)
         if self.unroll and input_shape[1] is None:
@@ -290,12 +319,11 @@ class Recurrent(Layer):
         preprocessed_input = self.preprocess_input(inputs, training=None)
         last_output, outputs, states = K.rnn(self.step,
                                              preprocessed_input,
-                                             initial_states,
+                                             initial_state,
                                              go_backwards=self.go_backwards,
                                              mask=mask,
                                              constants=constants,
-                                             unroll=self.unroll,
-                                             input_length=input_shape[1])
+                                             unroll=self.unroll)
         if self.stateful:
             updates = []
             for i in range(len(states)):
@@ -312,13 +340,10 @@ class Recurrent(Layer):
         else:
             return last_output
 
-    def reset_states(self, states_value=None):
+    def reset_states(self, states=None):
         if not self.stateful:
             raise AttributeError('Layer must be stateful.')
-        if not self.input_spec:
-            raise RuntimeError('Layer has never been called '
-                               'and thus has no states.')
-        batch_size = self.input_spec.shape[0]
+        batch_size = self.input_spec[0].shape[0]
         if not batch_size:
             raise ValueError('If a RNN is stateful, it needs to know '
                              'its batch size. Specify the batch size '
@@ -330,31 +355,30 @@ class Recurrent(Layer):
                              '- If using the functional API, specify '
                              'the time dimension by passing a '
                              '`batch_shape` argument to your Input layer.')
-        if states_value is not None:
-            if not isinstance(states_value, (list, tuple)):
-                states_value = [states_value]
-            if len(states_value) != len(self.states):
-                raise ValueError('The layer has ' + str(len(self.states)) +
-                                 ' states, but the `states_value` '
-                                 'argument passed '
-                                 'only has ' + str(len(states_value)) +
-                                 ' entries')
+        # initialize state if None
         if self.states[0] is None:
             self.states = [K.zeros((batch_size, self.units))
                            for _ in self.states]
-            if not states_value:
-                return
-        for i, state in enumerate(self.states):
-            if states_value:
-                value = states_value[i]
+        elif states is None:
+            for state in self.states:
+                K.set_value(state, np.zeros((batch_size, self.units)))
+        else:
+            if not isinstance(states, (list, tuple)):
+                states = [states]
+            if len(states) != len(self.states):
+                raise ValueError('Layer ' + self.name + ' expects ' +
+                                 str(len(self.states)) + ' states, '
+                                 'but it received ' + str(len(states)) +
+                                 ' state values. Input received: ' +
+                                 str(states))
+            for index, (value, state) in enumerate(zip(states, self.states)):
                 if value.shape != (batch_size, self.units):
-                    raise ValueError(
-                        'Expected state #' + str(i) +
-                        ' to have shape ' + str((batch_size, self.units)) +
-                        ' but got array with shape ' + str(value.shape))
-            else:
-                value = np.zeros((batch_size, self.units))
-            K.set_value(state, value)
+                    raise ValueError('State ' + str(index) +
+                                     ' is incompatible with layer ' +
+                                     self.name + ': expected shape=' +
+                                     str((batch_size, self.units)) +
+                                     ', found shape=' + str(value.shape))
+                K.set_value(state, value)
 
     def get_config(self):
         config = {'return_sequences': self.return_sequences,
@@ -371,39 +395,29 @@ class SimpleRNN(Recurrent):
 
     # Arguments
         units: Positive integer, dimensionality of the output space.
-        activation: Activation function to use
-            (see [activations](../activations.md)).
+        activation: Activation function to use.
             If you don't specify anything, no activation is applied
+            If you pass None, no activation is applied
             (ie. "linear" activation: `a(x) = x`).
         use_bias: Boolean, whether the layer uses a bias vector.
         kernel_initializer: Initializer for the `kernel` weights matrix,
-            used for the linear transformation of the inputs.
-            (see [initializers](../initializers.md)).
+            used for the linear transformation of the inputs..
         recurrent_initializer: Initializer for the `recurrent_kernel`
             weights matrix,
-            used for the linear transformation of the recurrent state.
-            (see [initializers](../initializers.md)).
-        bias_initializer: Initializer for the bias vector
-            (see [initializers](../initializers.md)).
+            used for the linear transformation of the recurrent state..
+        bias_initializer: Initializer for the bias vector.
         kernel_regularizer: Regularizer function applied to
-            the `kernel` weights matrix
-            (see [regularizer](../regularizers.md)).
+            the `kernel` weights matrix.
         recurrent_regularizer: Regularizer function applied to
-            the `recurrent_kernel` weights matrix
-            (see [regularizer](../regularizers.md)).
-        bias_regularizer: Regularizer function applied to the bias vector
-            (see [regularizer](../regularizers.md)).
+            the `recurrent_kernel` weights matrix.
+        bias_regularizer: Regularizer function applied to the bias vector.
         activity_regularizer: Regularizer function applied to
-            the output of the layer (its "activation").
-            (see [regularizer](../regularizers.md)).
+            the output of the layer (its "activation")..
         kernel_constraint: Constraint function applied to
-            the `kernel` weights matrix
-            (see [constraints](../constraints.md)).
+            the `kernel` weights matrix.
         recurrent_constraint: Constraint function applied to
-            the `recurrent_kernel` weights matrix
-            (see [constraints](../constraints.md)).
-        bias_constraint: Constraint function applied to the bias vector
-            (see [constraints](../constraints.md)).
+            the `recurrent_kernel` weights matrix.
+        bias_constraint: Constraint function applied to the bias vector.
         dropout: Float between 0 and 1.
             Fraction of the units to drop for
             the linear transformation of the inputs.
@@ -415,7 +429,6 @@ class SimpleRNN(Recurrent):
         - [A Theoretically Grounded Application of Dropout in Recurrent Neural Networks](http://arxiv.org/abs/1512.05287)
     """
 
-    @interfaces.legacy_recurrent_support
     def __init__(self, units,
                  activation='tanh',
                  use_bias=True,
@@ -452,33 +465,34 @@ class SimpleRNN(Recurrent):
 
         self.dropout = min(1., max(0., dropout))
         self.recurrent_dropout = min(1., max(0., recurrent_dropout))
+        self.state_spec = InputSpec(shape=(None, self.units))
 
     def build(self, input_shape):
         if isinstance(input_shape, list):
             input_shape = input_shape[0]
+        input_shape = tensor_shape.TensorShape(input_shape).as_list()
 
         batch_size = input_shape[0] if self.stateful else None
         self.input_dim = input_shape[2]
-        self.input_spec = InputSpec(shape=(batch_size, None, self.input_dim))
-        self.state_spec = InputSpec(shape=(batch_size, self.units))
+        self.input_spec[0] = InputSpec(shape=(batch_size, None, self.input_dim))
 
         self.states = [None]
         if self.stateful:
             self.reset_states()
 
-        self.kernel = self.add_weight((self.input_dim, self.units),
+        self.kernel = self.add_weight(shape=(self.input_dim, self.units),
                                       name='kernel',
                                       initializer=self.kernel_initializer,
                                       regularizer=self.kernel_regularizer,
                                       constraint=self.kernel_constraint)
         self.recurrent_kernel = self.add_weight(
-            (self.units, self.units),
+            shape=(self.units, self.units),
             name='recurrent_kernel',
             initializer=self.recurrent_initializer,
             regularizer=self.recurrent_regularizer,
             constraint=self.recurrent_constraint)
         if self.use_bias:
-            self.bias = self.add_weight((self.units,),
+            self.bias = self.add_weight(shape=(self.units,),
                                         name='bias',
                                         initializer=self.bias_initializer,
                                         regularizer=self.bias_regularizer,
@@ -491,7 +505,7 @@ class SimpleRNN(Recurrent):
         if self.implementation > 0:
             return inputs
         else:
-            input_shape = K.int_shape(inputs)
+            input_shape = inputs.get_shape().as_list()
             input_dim = input_shape[2]
             timesteps = input_shape[1]
             return _time_distributed_dense(inputs,
@@ -528,7 +542,7 @@ class SimpleRNN(Recurrent):
 
     def get_constants(self, inputs, training=None):
         constants = []
-        if self.implementation == 0 and 0 < self.dropout < 1:
+        if self.implementation != 0 and 0 < self.dropout < 1:
             input_shape = K.int_shape(inputs)
             input_dim = input_shape[-1]
             ones = K.ones_like(K.reshape(inputs[:, 0, 0], (-1, 1)))
@@ -548,7 +562,7 @@ class SimpleRNN(Recurrent):
             ones = K.ones_like(K.reshape(inputs[:, 0, 0], (-1, 1)))
             ones = K.tile(ones, (1, self.units))
 
-            def dropped_inputs():
+            def dropped_inputs():  # pylint: disable=function-redefined
                 return K.dropout(ones, self.recurrent_dropout)
             rec_dp_mask = K.in_train_phase(dropped_inputs,
                                            ones,
@@ -583,42 +597,30 @@ class GRU(Recurrent):
 
     # Arguments
         units: Positive integer, dimensionality of the output space.
-        activation: Activation function to use
-            (see [activations](../activations.md)).
-            If you don't specify anything, no activation is applied
+        activation: Activation function to use.
+            If you pass None, no activation is applied
             (ie. "linear" activation: `a(x) = x`).
         recurrent_activation: Activation function to use
-            for the recurrent step
-            (see [activations](../activations.md)).
+            for the recurrent step.
         use_bias: Boolean, whether the layer uses a bias vector.
         kernel_initializer: Initializer for the `kernel` weights matrix,
-            used for the linear transformation of the inputs.
-            (see [initializers](../initializers.md)).
+            used for the linear transformation of the inputs..
         recurrent_initializer: Initializer for the `recurrent_kernel`
             weights matrix,
-            used for the linear transformation of the recurrent state.
-            (see [initializers](../initializers.md)).
-        bias_initializer: Initializer for the bias vector
-            (see [initializers](../initializers.md)).
+            used for the linear transformation of the recurrent state..
+        bias_initializer: Initializer for the bias vector.
         kernel_regularizer: Regularizer function applied to
-            the `kernel` weights matrix
-            (see [regularizer](../regularizers.md)).
+            the `kernel` weights matrix.
         recurrent_regularizer: Regularizer function applied to
-            the `recurrent_kernel` weights matrix
-            (see [regularizer](../regularizers.md)).
-        bias_regularizer: Regularizer function applied to the bias vector
-            (see [regularizer](../regularizers.md)).
+            the `recurrent_kernel` weights matrix.
+        bias_regularizer: Regularizer function applied to the bias vector.
         activity_regularizer: Regularizer function applied to
-            the output of the layer (its "activation").
-            (see [regularizer](../regularizers.md)).
+            the output of the layer (its "activation")..
         kernel_constraint: Constraint function applied to
-            the `kernel` weights matrix
-            (see [constraints](../constraints.md)).
+            the `kernel` weights matrix.
         recurrent_constraint: Constraint function applied to
-            the `recurrent_kernel` weights matrix
-            (see [constraints](../constraints.md)).
-        bias_constraint: Constraint function applied to the bias vector
-            (see [constraints](../constraints.md)).
+            the `recurrent_kernel` weights matrix.
+        bias_constraint: Constraint function applied to the bias vector.
         dropout: Float between 0 and 1.
             Fraction of the units to drop for
             the linear transformation of the inputs.
@@ -632,7 +634,6 @@ class GRU(Recurrent):
         - [A Theoretically Grounded Application of Dropout in Recurrent Neural Networks](http://arxiv.org/abs/1512.05287)
     """
 
-    @interfaces.legacy_recurrent_support
     def __init__(self, units,
                  activation='tanh',
                  recurrent_activation='hard_sigmoid',
@@ -671,36 +672,36 @@ class GRU(Recurrent):
 
         self.dropout = min(1., max(0., dropout))
         self.recurrent_dropout = min(1., max(0., recurrent_dropout))
+        self.state_spec = InputSpec(shape=(None, self.units))
 
     def build(self, input_shape):
         if isinstance(input_shape, list):
             input_shape = input_shape[0]
-
+        input_shape = tensor_shape.TensorShape(input_shape).as_list()
         batch_size = input_shape[0] if self.stateful else None
         self.input_dim = input_shape[2]
-        self.input_spec = InputSpec(shape=(batch_size, None, self.input_dim))
-        self.state_spec = InputSpec(shape=(batch_size, self.units))
+        self.input_spec[0] = InputSpec(shape=(batch_size, None, self.input_dim))
 
         self.states = [None]
         if self.stateful:
             self.reset_states()
 
-        self.kernel = self.add_weight((self.input_dim, self.units * 3),
+        self.kernel = self.add_weight(shape=(self.input_dim, self.units * 3),
                                       name='kernel',
                                       initializer=self.kernel_initializer,
                                       regularizer=self.kernel_regularizer,
                                       constraint=self.kernel_constraint)
         self.recurrent_kernel = self.add_weight(
-            (self.units, self.units * 3),
+            shape=(self.units, self.units * 3),
             name='recurrent_kernel',
             initializer=self.recurrent_initializer,
             regularizer=self.recurrent_regularizer,
             constraint=self.recurrent_constraint)
 
         if self.use_bias:
-            self.bias = self.add_weight((self.units * 3,),
+            self.bias = self.add_weight(shape=(self.units * 3,),
                                         name='bias',
-                                        initializer='zero',
+                                        initializer=self.bias_initializer,
                                         regularizer=self.bias_regularizer,
                                         constraint=self.bias_constraint)
         else:
@@ -727,7 +728,7 @@ class GRU(Recurrent):
 
     def preprocess_input(self, inputs, training=None):
         if self.implementation == 0:
-            input_shape = K.int_shape(inputs)
+            input_shape = inputs.get_shape().as_list()
             input_dim = input_shape[2]
             timesteps = input_shape[1]
 
@@ -746,7 +747,7 @@ class GRU(Recurrent):
 
     def get_constants(self, inputs, training=None):
         constants = []
-        if self.implementation == 0 and 0 < self.dropout < 1:
+        if self.implementation != 0 and 0 < self.dropout < 1:
             input_shape = K.int_shape(inputs)
             input_dim = input_shape[-1]
             ones = K.ones_like(K.reshape(inputs[:, 0, 0], (-1, 1)))
@@ -766,7 +767,7 @@ class GRU(Recurrent):
             ones = K.ones_like(K.reshape(inputs[:, 0, 0], (-1, 1)))
             ones = K.tile(ones, (1, self.units))
 
-            def dropped_inputs():
+            def dropped_inputs():  # pylint: disable=function-redefined
                 return K.dropout(ones, self.recurrent_dropout)
             rec_dp_mask = [K.in_train_phase(dropped_inputs,
                                             ones,
@@ -812,7 +813,7 @@ class GRU(Recurrent):
                 if self.use_bias:
                     x_z = K.bias_add(x_z, self.bias_z)
                     x_r = K.bias_add(x_r, self.bias_r)
-                    x_h = K.bias_add(x_r, self.bias_h)
+                    x_h = K.bias_add(x_h, self.bias_h)
             else:
                 raise ValueError('Unknown `implementation` mode.')
             z = self.recurrent_activation(x_z + K.dot(h_tm1 * rec_dp_mask[0],
@@ -856,46 +857,34 @@ class LSTM(Recurrent):
 
     # Arguments
         units: Positive integer, dimensionality of the output space.
-        activation: Activation function to use
-            (see [activations](../activations.md)).
-            If you don't specify anything, no activation is applied
+        activation: Activation function to use.
+            If you pass None, no activation is applied
             (ie. "linear" activation: `a(x) = x`).
         recurrent_activation: Activation function to use
-            for the recurrent step
-            (see [activations](../activations.md)).
+            for the recurrent step.
         use_bias: Boolean, whether the layer uses a bias vector.
         kernel_initializer: Initializer for the `kernel` weights matrix,
-            used for the linear transformation of the inputs.
-            (see [initializers](../initializers.md)).
+            used for the linear transformation of the inputs..
         recurrent_initializer: Initializer for the `recurrent_kernel`
             weights matrix,
-            used for the linear transformation of the recurrent state.
-            (see [initializers](../initializers.md)).
-        bias_initializer: Initializer for the bias vector
-            (see [initializers](../initializers.md)).
+            used for the linear transformation of the recurrent state..
+        bias_initializer: Initializer for the bias vector.
         unit_forget_bias: Boolean.
             If True, add 1 to the bias of the forget gate at initialization.
             Setting it to true will also force `bias_initializer="zeros"`.
             This is recommended in [Jozefowicz et al.](http://www.jmlr.org/proceedings/papers/v37/jozefowicz15.pdf)
         kernel_regularizer: Regularizer function applied to
-            the `kernel` weights matrix
-            (see [regularizer](../regularizers.md)).
+            the `kernel` weights matrix.
         recurrent_regularizer: Regularizer function applied to
-            the `recurrent_kernel` weights matrix
-            (see [regularizer](../regularizers.md)).
-        bias_regularizer: Regularizer function applied to the bias vector
-            (see [regularizer](../regularizers.md)).
+            the `recurrent_kernel` weights matrix.
+        bias_regularizer: Regularizer function applied to the bias vector.
         activity_regularizer: Regularizer function applied to
-            the output of the layer (its "activation").
-            (see [regularizer](../regularizers.md)).
+            the output of the layer (its "activation")..
         kernel_constraint: Constraint function applied to
-            the `kernel` weights matrix
-            (see [constraints](../constraints.md)).
+            the `kernel` weights matrix.
         recurrent_constraint: Constraint function applied to
-            the `recurrent_kernel` weights matrix
-            (see [constraints](../constraints.md)).
-        bias_constraint: Constraint function applied to the bias vector
-            (see [constraints](../constraints.md)).
+            the `recurrent_kernel` weights matrix.
+        bias_constraint: Constraint function applied to the bias vector.
         dropout: Float between 0 and 1.
             Fraction of the units to drop for
             the linear transformation of the inputs.
@@ -905,11 +894,10 @@ class LSTM(Recurrent):
 
     # References
         - [Long short-term memory](http://deeplearning.cs.cmu.edu/pdfs/Hochreiter97_lstm.pdf) (original 1997 paper)
-        - [Learning to forget: Continual prediction with LSTM](http://www.mitpressjournals.org/doi/pdf/10.1162/089976600300015015)
         - [Supervised sequence labeling with recurrent neural networks](http://www.cs.toronto.edu/~graves/preprint.pdf)
         - [A Theoretically Grounded Application of Dropout in Recurrent Neural Networks](http://arxiv.org/abs/1512.05287)
     """
-    @interfaces.legacy_recurrent_support
+
     def __init__(self, units,
                  activation='tanh',
                  recurrent_activation='hard_sigmoid',
@@ -950,43 +938,48 @@ class LSTM(Recurrent):
 
         self.dropout = min(1., max(0., dropout))
         self.recurrent_dropout = min(1., max(0., recurrent_dropout))
+        self.state_spec = [InputSpec(shape=(None, self.units)),
+                           InputSpec(shape=(None, self.units))]
 
     def build(self, input_shape):
         if isinstance(input_shape, list):
             input_shape = input_shape[0]
-
+        input_shape = tensor_shape.TensorShape(input_shape).as_list()
         batch_size = input_shape[0] if self.stateful else None
         self.input_dim = input_shape[2]
-        self.input_spec = InputSpec(shape=(batch_size, None, self.input_dim))
-        self.state_spec = [InputSpec(shape=(batch_size, self.units)),
-                           InputSpec(shape=(batch_size, self.units))]
+        self.input_spec[0] = InputSpec(shape=(batch_size, None, self.input_dim))
 
         self.states = [None, None]
         if self.stateful:
             self.reset_states()
 
-        self.kernel = self.add_weight((self.input_dim, self.units * 4),
+        self.kernel = self.add_weight(shape=(self.input_dim, self.units * 4),
                                       name='kernel',
                                       initializer=self.kernel_initializer,
                                       regularizer=self.kernel_regularizer,
                                       constraint=self.kernel_constraint)
         self.recurrent_kernel = self.add_weight(
-            (self.units, self.units * 4),
+            shape=(self.units, self.units * 4),
             name='recurrent_kernel',
             initializer=self.recurrent_initializer,
             regularizer=self.recurrent_regularizer,
             constraint=self.recurrent_constraint)
 
         if self.use_bias:
-            self.bias = self.add_weight((self.units * 4,),
+            if self.unit_forget_bias:
+                def bias_initializer(shape, *args, **kwargs):
+                    return K.concatenate([
+                        self.bias_initializer((self.units,), *args, **kwargs),
+                        initializers.Ones()((self.units,), *args, **kwargs),
+                        self.bias_initializer((self.units * 2,), *args, **kwargs),
+                    ])
+            else:
+                bias_initializer = self.bias_initializer
+            self.bias = self.add_weight(shape=(self.units * 4,),
                                         name='bias',
-                                        initializer=self.bias_initializer,
+                                        initializer=bias_initializer,
                                         regularizer=self.bias_regularizer,
                                         constraint=self.bias_constraint)
-            if self.unit_forget_bias:
-                bias_value = np.zeros((self.units * 4,))
-                bias_value[self.units: self.units * 2] = 1.
-                K.set_value(self.bias, bias_value)
         else:
             self.bias = None
 
@@ -1014,7 +1007,7 @@ class LSTM(Recurrent):
 
     def preprocess_input(self, inputs, training=None):
         if self.implementation == 0:
-            input_shape = K.int_shape(inputs)
+            input_shape = inputs.get_shape().as_list()
             input_dim = input_shape[2]
             timesteps = input_shape[1]
 
@@ -1036,7 +1029,7 @@ class LSTM(Recurrent):
 
     def get_constants(self, inputs, training=None):
         constants = []
-        if self.implementation == 0 and 0 < self.dropout < 1:
+        if self.implementation != 0 and 0 < self.dropout < 1:
             input_shape = K.int_shape(inputs)
             input_dim = input_shape[-1]
             ones = K.ones_like(K.reshape(inputs[:, 0, 0], (-1, 1)))
@@ -1056,7 +1049,7 @@ class LSTM(Recurrent):
             ones = K.ones_like(K.reshape(inputs[:, 0, 0], (-1, 1)))
             ones = K.tile(ones, (1, self.units))
 
-            def dropped_inputs():
+            def dropped_inputs():  # pylint: disable=function-redefined
                 return K.dropout(ones, self.recurrent_dropout)
             rec_dp_mask = [K.in_train_phase(dropped_inputs,
                                             ones,

keras/layers/serialization.py

@@ -0,0 +1,49 @@
+"""Layer serialization/deserialization functions.
+"""
+# pylint: disable=wildcard-import
+# pylint: disable=unused-import
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+from .advanced_activations import *
+from .convolutional import *
+from .convolutional_recurrent import *
+from .core import *
+from .embeddings import *
+from ..engine import Input
+from ..engine import InputLayer
+from .local import *
+from .merge import *
+from .noise import *
+from .normalization import *
+from .pooling import *
+from .recurrent import *
+from ..utils.generic_utils import deserialize_keras_object
+from .wrappers import *
+
+
+def serialize(layer):
+    return {'class_name': layer.__class__.__name__,
+            'config': layer.get_config()}
+
+
+def deserialize(config, custom_objects=None):
+    """Instantiates a layer from a config dictionary.
+
+    # Arguments
+        config: dict of the form {'class_name': str, 'config': dict}
+        custom_objects: dict mapping class names (or function names)
+            of custom (non-Keras) objects to class/functions
+
+    # Returns
+        Layer instance (may be Model, Sequential, Layer...)
+    """
+    from .. import models  # pylint: disable=g-import-not-at-top
+    globs = globals()  # All layers.
+    globs['Model'] = models.Model
+    globs['Sequential'] = models.Sequential
+    return deserialize_keras_object(config,
+                                    module_objects=globs,
+                                    custom_objects=custom_objects,
+                                    printable_module_name='layer')

keras/layers/wrappers.py

@@ -1,10 +1,16 @@
 # -*- coding: utf-8 -*-
+"""Wrapper layers: layers that augment the functionality of another layer.
+"""
 from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
 
 import copy
+import inspect
 from ..engine import Layer
 from ..engine import InputSpec
 from .. import backend as K
+from tensorflow.python.framework import tensor_shape
 
 
 class Wrapper(Layer):
@@ -23,18 +29,53 @@ class Wrapper(Layer):
         super(Wrapper, self).__init__(**kwargs)
 
     def build(self, input_shape=None):
-        # Assumes that self.layer is already set.
-        # Should be called at the end of .build() in the children classes.
-        self.trainable_weights = getattr(self.layer, 'trainable_weights', [])
-        self.non_trainable_weights = getattr(self.layer, 'non_trainable_weights', [])
-        self.updates = getattr(self.layer, 'updates', [])
-        self.losses = getattr(self.layer, 'losses', [])
-        self.constraints = getattr(self.layer, 'constraints', {})
         self.built = True
 
+    @property
+    def activity_regularizer(self):
+        if hasattr(self.layer, 'activity_regularizer'):
+            return self.layer.activity_regularizer
+        else:
+            return None
+
+    @property
+    def trainable_weights(self):
+        return self.layer.trainable_weights
+
+    @property
+    def non_trainable_weights(self):
+        return self.layer.non_trainable_weights
+
+    @property
+    def updates(self):
+        if hasattr(self.layer, 'updates'):
+            return self.layer.updates
+        return []
+
+    def get_updates_for(self, inputs=None):
+        if inputs is None:
+            updates = self.layer.get_updates_for(None)
+            return updates + super(Wrapper, self).get_updates_for(None)
+        return super(Wrapper, self).get_updates_for(inputs)
+
+    @property
+    def losses(self):
+        if hasattr(self.layer, 'losses'):
+            return self.layer.losses
+        return []
+
+    def get_losses_for(self, inputs=None):
+        if inputs is None:
+            losses = self.layer.get_losses_for(None)
+            return losses + super(Wrapper, self).get_losses_for(None)
+        return super(Wrapper, self).get_losses_for(inputs)
+
+    @property
+    def constraints(self):
+        return self.layer.constraints
+
     def get_weights(self):
-        weights = self.layer.get_weights()
-        return weights
+        return self.layer.get_weights()
 
     def set_weights(self, weights):
         self.layer.set_weights(weights)
@@ -46,9 +87,10 @@ class Wrapper(Layer):
         return dict(list(base_config.items()) + list(config.items()))
 
     @classmethod
-    def from_config(cls, config):
-        from . import deserialize as deserialize_layer
-        layer = deserialize_layer(config.pop('layer'))
+    def from_config(cls, config, custom_objects=None):
+        from . import deserialize as deserialize_layer  # pylint: disable=g-import-not-at-top
+        layer = deserialize_layer(config.pop('layer'),
+                                  custom_objects=custom_objects)
         return cls(layer, **config)
 
 
@@ -71,13 +113,18 @@ class TimeDistributed(Wrapper):
         model = Sequential()
         model.add(TimeDistributed(Dense(8), input_shape=(10, 16)))
         # now model.output_shape == (None, 10, 8)
+    ```
 
-        # subsequent layers: no need for input_shape
+    The output will then have shape `(32, 10, 8)`.
+
+    In subsequent layers, there is no need for the `input_shape`:
+
+    ```python
         model.add(TimeDistributed(Dense(32)))
         # now model.output_shape == (None, 10, 32)
     ```
 
-    The output will then have shape `(32, 10, 8)`.
+    The output will then have shape `(32, 10, 32)`.
 
     `TimeDistributed` can be used with arbitrary layers, not just `Dense`,
     for instance with a `Conv2D` layer:
@@ -97,19 +144,24 @@ class TimeDistributed(Wrapper):
         self.supports_masking = True
 
     def build(self, input_shape):
+        input_shape = tensor_shape.TensorShape(input_shape).as_list()
         assert len(input_shape) >= 3
         self.input_spec = InputSpec(shape=input_shape)
-        child_input_shape = (input_shape[0],) + input_shape[2:]
+        child_input_shape = [input_shape[0]] + input_shape[2:]
         if not self.layer.built:
             self.layer.build(child_input_shape)
             self.layer.built = True
         super(TimeDistributed, self).build()
 
-    def compute_output_shape(self, input_shape):
-        child_input_shape = (input_shape[0],) + input_shape[2:]
-        child_output_shape = self.layer.compute_output_shape(child_input_shape)
+    def _compute_output_shape(self, input_shape):
+        input_shape = tensor_shape.TensorShape(input_shape).as_list()
+        child_input_shape = tensor_shape.TensorShape(
+            [input_shape[0]] + input_shape[2:])
+        child_output_shape = self.layer._compute_output_shape(  # pylint: disable=protected-access
+            child_input_shape).as_list()
         timesteps = input_shape[1]
-        return (child_output_shape[0], timesteps) + child_output_shape[1:]
+        return tensor_shape.TensorShape(
+            [child_output_shape[0], timesteps] + child_output_shape[1:])
 
     def call(self, inputs, mask=None):
         input_shape = K.int_shape(inputs)
@@ -121,7 +173,6 @@ class TimeDistributed(Wrapper):
 
             _, outputs, _ = K.rnn(step, inputs,
                                   initial_states=[],
-                                  input_length=input_shape[1],
                                   unroll=False)
             y = outputs
         else:
@@ -135,8 +186,8 @@ class TimeDistributed(Wrapper):
             inputs = K.reshape(inputs, (-1,) + input_shape[2:])
             y = self.layer.call(inputs)  # (num_samples * timesteps, ...)
             # Shape: (num_samples, timesteps, ...)
-            output_shape = self.compute_output_shape(input_shape)
-            y = K.reshape(y, (-1, input_length) + output_shape[2:])
+            output_shape = self._compute_output_shape(input_shape).as_list()  # pylint: disable=protected-access
+            y = K.reshape(y, [-1, input_length] + output_shape[2:])
 
         # Apply activity regularizer if any:
         if (hasattr(self.layer, 'activity_regularizer') and
@@ -157,6 +208,9 @@ class Bidirectional(Wrapper):
             If None, the outputs will not be combined,
             they will be returned as a list.
 
+    # Raises
+        ValueError: In case of invalid `merge_mode` argument.
+
     # Examples
 
     ```python
@@ -198,39 +252,59 @@ class Bidirectional(Wrapper):
         self.forward_layer.set_weights(weights[:nw // 2])
         self.backward_layer.set_weights(weights[nw // 2:])
 
-    def compute_output_shape(self, input_shape):
+    def _compute_output_shape(self, input_shape):
+        input_shape = tensor_shape.TensorShape(input_shape).as_list()
         if self.merge_mode in ['sum', 'ave', 'mul']:
-            return self.forward_layer.compute_output_shape(input_shape)
+            return self.forward_layer._compute_output_shape(input_shape)  # pylint: disable=protected-access
         elif self.merge_mode == 'concat':
-            shape = list(self.forward_layer.compute_output_shape(input_shape))
+            shape = self.forward_layer._compute_output_shape(input_shape).as_list()  # pylint: disable=protected-access
             shape[-1] *= 2
-            return tuple(shape)
+            return tensor_shape.TensorShape(shape)
         elif self.merge_mode is None:
-            return [self.forward_layer.compute_output_shape(input_shape)] * 2
+            shape = self.forward_layer._compute_output_shape(input_shape)  # pylint: disable=protected-access
+            return [shape, copy.copy(shape)]
 
-    def call(self, inputs, mask=None):
-        y = self.forward_layer.call(inputs, mask)
-        y_rev = self.backward_layer.call(inputs, mask)
+    def call(self, inputs, training=None, mask=None):
+        kwargs = {}
+        func_args = inspect.getargspec(self.layer.call).args
+        if 'training' in func_args:
+            kwargs['training'] = training
+        if 'mask' in func_args:
+            kwargs['mask'] = mask
+
+        y = self.forward_layer.call(inputs, **kwargs)
+        y_rev = self.backward_layer.call(inputs, **kwargs)
         if self.return_sequences:
             y_rev = K.reverse(y_rev, 1)
         if self.merge_mode == 'concat':
-            return K.concatenate([y, y_rev])
+            output = K.concatenate([y, y_rev])
         elif self.merge_mode == 'sum':
-            return y + y_rev
+            output = y + y_rev
         elif self.merge_mode == 'ave':
-            return (y + y_rev) / 2
+            output = (y + y_rev) / 2
         elif self.merge_mode == 'mul':
-            return y * y_rev
+            output = y * y_rev
         elif self.merge_mode is None:
-            return [y, y_rev]
+            output = [y, y_rev]
+
+        # Properly set learning phase
+        if 0 < self.layer.dropout + self.layer.recurrent_dropout:
+            if self.merge_mode is None:
+                for out in output:
+                    out._uses_learning_phase = True
+            else:
+                output._uses_learning_phase = True
+        return output
 
     def reset_states(self):
         self.forward_layer.reset_states()
         self.backward_layer.reset_states()
 
     def build(self, input_shape):
-        self.forward_layer.build(input_shape)
-        self.backward_layer.build(input_shape)
+        with K.name_scope(self.forward_layer.name):
+            self.forward_layer.build(input_shape)
+        with K.name_scope(self.backward_layer.name):
+            self.backward_layer.build(input_shape)
         self.built = True
 
     def compute_mask(self, inputs, mask):

keras/legacy/interfaces.py

@@ -3,6 +3,7 @@
 import six
 import warnings
 import functools
+import inspect
 import numpy as np
 
 
@@ -83,8 +84,9 @@ def generate_legacy_interface(allowed_positional_args=None,
                         signature += ', '
                 signature += ')`'
                 warnings.warn('Update your `' + object_name +
-                              '` call to the Keras 2 API: ' + signature)
+                              '` call to the Keras 2 API: ' + signature, stacklevel=2)
             return func(*args, **kwargs)
+        wrapper._legacy_support_signature = inspect.getargspec(func)
         return wrapper
     return legacy_support
 
@@ -122,7 +124,7 @@ def embedding_kwargs_preprocessor(args, kwargs):
         kwargs.pop('dropout')
         warnings.warn('The `dropout` argument is no longer support in `Embedding`. '
                       'You can apply a `keras.layers.SpatialDropout1D` layer '
-                      'right after the `Embedding` layer to get the same behavior.')
+                      'right after the `Embedding` layer to get the same behavior.', stacklevel=3)
     return args, kwargs, converted
 
 legacy_embedding_support = generate_legacy_interface(
@@ -148,7 +150,7 @@ legacy_gaussiannoise_support = generate_legacy_interface(
     conversions=[('sigma', 'stddev')])
 
 
-def lstm_args_preprocessor(args, kwargs):
+def recurrent_args_preprocessor(args, kwargs):
     converted = []
     if 'forget_bias_init' in kwargs:
         if kwargs['forget_bias_init'] == 'one':
@@ -159,7 +161,16 @@ def lstm_args_preprocessor(args, kwargs):
             kwargs.pop('forget_bias_init')
             warnings.warn('The `forget_bias_init` argument '
                           'has been ignored. Use `unit_forget_bias=True` '
-                          'instead to intialize with ones.')
+                          'instead to intialize with ones.', stacklevel=3)
+    if 'input_dim' in kwargs:
+        input_length = kwargs.pop('input_length', None)
+        input_dim = kwargs.pop('input_dim')
+        input_shape = (input_length, input_dim)
+        kwargs['input_shape'] = input_shape
+        converted.append(('input_dim', 'input_shape'))
+        warnings.warn('The `input_dim` and `input_length` arguments '
+                      'in recurrent layers are deprecated. '
+                      'Use `input_shape` instead.', stacklevel=3)
     return args, kwargs, converted
 
 legacy_recurrent_support = generate_legacy_interface(
@@ -177,7 +188,7 @@ legacy_recurrent_support = generate_legacy_interface(
     value_conversions={'consume_less': {'cpu': 0,
                                         'mem': 1,
                                         'gpu': 2}},
-    preprocessor=lstm_args_preprocessor)
+    preprocessor=recurrent_args_preprocessor)
 
 legacy_gaussiandropout_support = generate_legacy_interface(
     allowed_positional_args=['rate'],
@@ -450,7 +461,7 @@ def convlstm2d_args_preprocessor(args, kwargs):
         else:
             warnings.warn('The `forget_bias_init` argument '
                           'has been ignored. Use `unit_forget_bias=True` '
-                          'instead to intialize with ones.')
+                          'instead to intialize with ones.', stacklevel=3)
     args, kwargs, _converted = conv2d_args_preprocessor(args, kwargs)
     return args, kwargs, converted + _converted
 
@@ -493,7 +504,7 @@ def zeropadding2d_args_preprocessor(args, kwargs):
             kwargs['padding'] = ((top_pad, bottom_pad), (left_pad, right_pad))
             warnings.warn('The `padding` argument in the Keras 2 API no longer'
                           'accepts dict types. You can now input argument as: '
-                          '`padding=(top_pad, bottom_pad, left_pad, right_pad)`.')
+                          '`padding=(top_pad, bottom_pad, left_pad, right_pad)`.', stacklevel=3)
     elif len(args) == 2 and isinstance(args[1], dict):
         if set(args[1].keys()) <= {'top_pad', 'bottom_pad',
                                    'left_pad', 'right_pad'}:
@@ -504,7 +515,7 @@ def zeropadding2d_args_preprocessor(args, kwargs):
             args = (args[0], ((top_pad, bottom_pad), (left_pad, right_pad)))
             warnings.warn('The `padding` argument in the Keras 2 API no longer'
                           'accepts dict types. You can now input argument as: '
-                          '`padding=((top_pad, bottom_pad), (left_pad, right_pad))`')
+                          '`padding=((top_pad, bottom_pad), (left_pad, right_pad))`', stacklevel=3)
     return args, kwargs, converted
 
 legacy_zeropadding2d_support = generate_legacy_interface(
@@ -571,7 +582,7 @@ def generator_methods_args_preprocessor(args, kwargs):
                               'Keras 1 argument `samples_per_epoch`. '
                               '`steps_per_epoch` is the number of batches '
                               'to draw from the generator at each epoch. '
-                              'Update your method calls accordingly.')
+                              'Update your method calls accordingly.', stacklevel=3)
                 kwargs['steps_per_epoch'] = samples_per_epoch
             converted.append(('samples_per_epoch', 'steps_per_epoch'))
     return args, kwargs, converted
@@ -591,3 +602,24 @@ legacy_model_constructor_support = generate_legacy_interface(
     allowed_positional_args=None,
     conversions=[('input', 'inputs'),
                  ('output', 'outputs')])
+
+legacy_input_support = generate_legacy_interface(
+    allowed_positional_args=None,
+    conversions=[('input_dtype', 'dtype')])
+
+
+def add_weight_args_preprocessing(args, kwargs):
+    if len(args) > 1:
+        if isinstance(args[1], (tuple, list)):
+            kwargs['shape'] = args[1]
+            args = (args[0],) + args[2:]
+            if len(args) > 1:
+                if isinstance(args[1], six.string_types):
+                    kwargs['name'] = args[1]
+                    args = (args[0],) + args[2:]
+    return args, kwargs, []
+
+
+legacy_add_weight_support = generate_legacy_interface(
+    allowed_positional_args=['name', 'shape'],
+    preprocessor=add_weight_args_preprocessing)

keras/legacy/layers.py

@@ -46,7 +46,7 @@ class Merge(Layer):
             (1:1 mapping to input tensors)
             and return a single shape tuple, including the
             batch size (same convention as the
-            `get_output_shape_for` method of layers).
+            `compute_output_shape` method of layers).
         node_indices: Optional list of integers containing
             the output node index for each input layer
             (in case some input layers have multiple output nodes).
@@ -66,7 +66,7 @@ class Merge(Layer):
         warnings.warn('The `Merge` layer is deprecated '
                       'and will be removed after 08/2017. '
                       'Use instead layers from `keras.layers.merge`, '
-                      'e.g. `add`, `concatenate`, etc.')
+                      'e.g. `add`, `concatenate`, etc.', stacklevel=2)
         self.layers = layers
         self.mode = mode
         self.concat_axis = concat_axis
@@ -76,6 +76,10 @@ class Merge(Layer):
         self._output_mask = output_mask
         self.arguments = arguments if arguments else {}
         self._initial_weights = None
+        self._updates = []
+        self._losses = []
+        self._per_input_updates = {}
+        self._per_input_losses = {}
 
         # Layer parameters.
         self.inbound_nodes = []
@@ -286,7 +290,7 @@ class Merge(Layer):
 
         assert hasattr(mask, '__len__') and len(mask) == len(inputs)
 
-        if self.mode in ['sum', 'mul', 'ave']:
+        if self.mode in ['sum', 'mul', 'ave', 'max']:
             masks = [K.expand_dims(m, 0) for m in mask if m is not None]
             return K.all(K.concatenate(masks, axis=0), axis=0, keepdims=False)
         elif self.mode == 'concat':
@@ -297,8 +301,8 @@ class Merge(Layer):
             for input_i, mask_i in zip(inputs, mask):
                 if mask_i is None:
                     # Input is unmasked. Append all 1s to masks,
-                    # but cast it to uint8 first
-                    masks.append(K.cast(K.ones_like(input_i), 'uint8'))
+                    # but cast it to bool first
+                    masks.append(K.cast(K.ones_like(input_i), 'bool'))
                 elif K.ndim(mask_i) < K.ndim(input_i):
                     # Mask is smaller than the input, expand it
                     masks.append(K.expand_dims(mask_i))
@@ -361,6 +365,7 @@ class Merge(Layer):
 
     @classmethod
     def from_config(cls, config):
+        config = config.copy()
         mode_type = config.pop('mode_type')
         if mode_type == 'function':
             mode = globals()[config['mode']]
@@ -406,7 +411,7 @@ def merge(inputs, mode='sum', concat_axis=-1,
     ```
     # Arguments
         mode: String or lambda/function. If string, must be one
-            of: 'sum', 'mul', 'concat', 'ave', 'cos', 'dot'.
+            of: 'sum', 'mul', 'concat', 'ave', 'cos', 'dot', 'max'.
             If lambda/function, it should take as input a list of tensors
             and return a single tensor.
         concat_axis: Integer, axis to use in mode `concat`.
@@ -417,7 +422,7 @@ def merge(inputs, mode='sum', concat_axis=-1,
             If the latter case, it should take as input a list of shape tuples
             (1:1 mapping to input tensors) and return a single shape tuple,
             including the batch size
-            (same convention as the `get_output_shape_for` method of layers).
+            (same convention as the `compute_output_shape` method of layers).
         node_indices: Optional list of integers containing
             the output node index for each input layer
             (in case some input layers have multiple output nodes).
@@ -429,7 +434,7 @@ def merge(inputs, mode='sum', concat_axis=-1,
     warnings.warn('The `merge` function is deprecated '
                   'and will be removed after 08/2017. '
                   'Use instead layers from `keras.layers.merge`, '
-                  'e.g. `sum`, `concatenate`, etc.')
+                  'e.g. `add`, `concatenate`, etc.', stacklevel=2)
     all_keras_tensors = True
     for x in inputs:
         if not hasattr(x, '_keras_history'):

keras/legacy/models.py

@@ -1,27 +0,0 @@
-from .layers import Merge
-
-
-def needs_legacy_support(model):
-    return isinstance(model.layers[0], Merge)
-
-
-def legacy_sequential_layers(model):
-    layers = []
-    if model.layers:
-        if isinstance(model.layers[0], Merge):
-            merge = model.layers[0]
-            for layer in merge.layers:
-                if hasattr(layer, 'layers'):
-                    for sublayer in layer.layers:
-                        if sublayer not in layers:
-                            layers.append(sublayer)
-                else:
-                    if layer not in layers:
-                        layers.append(layer)
-        else:
-            if model.layers[0] not in layers:
-                layers.append(model.layers[0])
-        for layer in model.layers[1:]:
-            if layer not in layers:
-                layers.append(layer)
-    return layers

keras/losses.py

@@ -1,6 +1,11 @@
+"""Built-in Keras loss functions.
+"""
 from __future__ import absolute_import
-import six
+from __future__ import division
+from __future__ import print_function
+
 from . import backend as K
+import six
 from .utils.generic_utils import deserialize_keras_object
 
 
@@ -13,6 +18,7 @@ def mean_absolute_error(y_true, y_pred):
 
 
 def mean_absolute_percentage_error(y_true, y_pred):
+    # Equivalent to MAE, but sometimes easier to interpret.
     diff = K.abs((y_true - y_pred) / K.clip(K.abs(y_true),
                                             K.epsilon(),
                                             None))
@@ -33,6 +39,18 @@ def hinge(y_true, y_pred):
     return K.mean(K.maximum(1. - y_true * y_pred, 0.), axis=-1)
 
 
+def categorical_hinge(y_true, y_pred):
+    pos = K.sum(y_true * y_pred, axis=-1)
+    neg = K.max((1.0 - y_true) * y_pred, axis=-1)
+    return K.mean(K.maximum(0.0, neg - pos + 1), axis=-1)
+
+
+def logcosh(y_true, y_pred):
+    def cosh(x):
+        return (K.exp(x) + K.exp(-x)) / 2
+    return K.mean(K.log(cosh(y_pred - y_true)), axis=-1)
+
+
 def categorical_crossentropy(y_true, y_pred):
     return K.categorical_crossentropy(y_pred, y_true)
 

keras/metrics.py

@@ -1,18 +1,26 @@
+"""Built-in Keras metrics functions.
+"""
 from __future__ import absolute_import
-import six
+from __future__ import division
+from __future__ import print_function
+
 from . import backend as K
-from .losses import mean_squared_error
+# pylint: disable=unused-import
+from .losses import binary_crossentropy
+from .losses import categorical_crossentropy
+from .losses import cosine_proximity
+from .losses import hinge
+from .losses import kullback_leibler_divergence
 from .losses import mean_absolute_error
 from .losses import mean_absolute_percentage_error
+from .losses import mean_squared_error
 from .losses import mean_squared_logarithmic_error
-from .losses import hinge
-from .losses import squared_hinge
-from .losses import categorical_crossentropy
-from .losses import sparse_categorical_crossentropy
-from .losses import binary_crossentropy
-from .losses import kullback_leibler_divergence
+from .losses import logcosh
 from .losses import poisson
-from .losses import cosine_proximity
+from .losses import sparse_categorical_crossentropy
+from .losses import squared_hinge
+# pylint: disable=unused-import
+import six
 from .utils.generic_utils import deserialize_keras_object
 
 
@@ -21,13 +29,15 @@ def binary_accuracy(y_true, y_pred):
 
 
 def categorical_accuracy(y_true, y_pred):
-    return K.equal(K.argmax(y_true, axis=-1),
-                   K.argmax(y_pred, axis=-1))
+    return K.cast(K.equal(K.argmax(y_true, axis=-1),
+                          K.argmax(y_pred, axis=-1)),
+                  K.floatx())
 
 
 def sparse_categorical_accuracy(y_true, y_pred):
-    return K.equal(K.max(y_true, axis=-1),
-                   K.cast(K.argmax(y_pred, axis=-1), K.floatx()))
+    return K.cast(K.equal(K.max(y_true, axis=-1),
+                          K.cast(K.argmax(y_pred, axis=-1), K.floatx())),
+                  K.floatx())
 
 
 def top_k_categorical_accuracy(y_true, y_pred, k=5):

keras/models.py

@@ -1,33 +1,39 @@
+"""Home of the Sequential model, and the `save_model`/`load_model` functions.
+"""
 # -*- coding: utf-8 -*-
 from __future__ import absolute_import
+from __future__ import division
 from __future__ import print_function
 
-import warnings
 import copy
 import json
 import os
-import yaml
-import numpy as np
+import warnings
 
 from . import backend as K
-from . import optimizers
 from . import layers as layer_module
-from .utils.io_utils import ask_to_proceed_with_overwrite
-from .engine.training import Model
+from . import optimizers
 from .engine import topology
-from .engine.topology import Layer
 from .engine.topology import Input
-from .legacy import layers as legacy_layers
-from .legacy import models as legacy_models
-from .legacy import interfaces
+from .engine.topology import Layer
+from .engine.training import Model
+import numpy as np
+from .utils.io_utils import ask_to_proceed_with_overwrite
 
+# pylint: disable=g-import-not-at-top
 try:
     import h5py
 except ImportError:
     h5py = None
 
+try:
+    import yaml
+except ImportError:
+    yaml = None
+# pylint: enable=g-import-not-at-top
 
-def save_model(model, filepath, overwrite=True):
+
+def save_model(model, filepath, overwrite=True, include_optimizer=True):
     """Save a model to a HDF5 file.
 
     The saved model contains:
@@ -45,6 +51,7 @@ def save_model(model, filepath, overwrite=True):
         overwrite: Whether we should overwrite any existing
             model at the target location, or instead
             ask the user with a manual prompt.
+        include_optimizer: If True, save optimizer's state together.
 
     # Raises
         ImportError: if h5py is not available.
@@ -85,7 +92,7 @@ def save_model(model, filepath, overwrite=True):
 
         raise TypeError('Not JSON Serializable:', obj)
 
-    from . import __version__ as keras_version
+    from . import __version__ as keras_version  # pylint: disable=g-import-not-at-top
 
     # If file exists and should not be overwritten.
     if not overwrite and os.path.isfile(filepath):
@@ -102,13 +109,10 @@ def save_model(model, filepath, overwrite=True):
     }, default=get_json_type).encode('utf8')
 
     model_weights_group = f.create_group('model_weights')
-    if legacy_models.needs_legacy_support(model):
-        model_layers = legacy_models.legacy_sequential_layers(model)
-    else:
-        model_layers = model.layers
+    model_layers = model.layers
     topology.save_weights_to_hdf5_group(model_weights_group, model_layers)
 
-    if hasattr(model, 'optimizer'):
+    if include_optimizer and hasattr(model, 'optimizer'):
         if isinstance(model.optimizer, optimizers.TFOptimizer):
             warnings.warn(
                 'TensorFlow optimizers do not '
@@ -166,7 +170,7 @@ def save_model(model, filepath, overwrite=True):
     f.close()
 
 
-def load_model(filepath, custom_objects=None):
+def load_model(filepath, custom_objects=None, compile=True):
     """Loads a model saved via `save_model`.
 
     # Arguments
@@ -174,19 +178,23 @@ def load_model(filepath, custom_objects=None):
         custom_objects: Optional dictionary mapping names
             (strings) to custom classes or functions to be
             considered during deserialization.
+        compile: Boolean, whether to compile the model
+            after loading.
 
     # Returns
         A Keras model instance. If an optimizer was found
         as part of the saved model, the model is already
         compiled. Otherwise, the model is uncompiled and
-        a warning will be displayed.
+        a warning will be displayed. When `compile` is set
+        to False, the compilation is omitted without any
+        warning.
 
     # Raises
         ImportError: if h5py is not available.
         ValueError: In case of an invalid savefile.
     """
     if h5py is None:
-        raise ImportError('`save_model` requires h5py.')
+        raise ImportError('`load_model` requires h5py.')
 
     if not custom_objects:
         custom_objects = {}
@@ -213,7 +221,14 @@ def load_model(filepath, custom_objects=None):
         if isinstance(obj, dict):
             deserialized = {}
             for key, value in obj.items():
-                if value in custom_objects:
+                deserialized[key] = []
+                if isinstance(value, list):
+                    for element in value:
+                        if element in custom_objects:
+                            deserialized[key].append(custom_objects[element])
+                        else:
+                            deserialized[key].append(element)
+                elif value in custom_objects:
                     deserialized[key] = custom_objects[value]
                 else:
                     deserialized[key] = value
@@ -234,6 +249,11 @@ def load_model(filepath, custom_objects=None):
     # set weights
     topology.load_weights_from_hdf5_group(f['model_weights'], model.layers)
 
+    # Early return if compilation is not required.
+    if not compile:
+        f.close()
+        return model
+
     # instantiate optimizer
     training_config = f.attrs.get('training_config')
     if training_config is None:
@@ -285,9 +305,12 @@ def model_from_config(config, custom_objects=None):
 
     # Returns
         A Keras model instance (uncompiled).
+
+    # Raises
+        TypeError if `config` is not a dictionary
     """
     if isinstance(config, list):
-        raise TypeError('`model_fom_config` expects a dictionary, not a list. '
+        raise TypeError('`model_from_config` expects a dictionary, not a list. '
                         'Maybe you meant to use '
                         '`Sequential.from_config(config)`?')
     return layer_module.deserialize(config, custom_objects=custom_objects)
@@ -304,7 +327,12 @@ def model_from_yaml(yaml_string, custom_objects=None):
 
     # Returns
         A Keras model instance (uncompiled).
+
+    # Raises
+        ImportError: if yaml module is not found.
     """
+    if yaml is None:
+        raise ImportError('Requires yaml module installed.')
     config = yaml.load(yaml_string)
     return layer_module.deserialize(config, custom_objects=custom_objects)
 
@@ -448,9 +476,7 @@ class Sequential(Model):
                           output_tensors=self.outputs,
                           # no model-level masking for now
                           input_masks=[None for _ in self.inputs],
-                          output_masks=[None],
-                          input_shapes=[x._keras_shape for x in self.inputs],
-                          output_shapes=[self.outputs[0]._keras_shape])
+                          output_masks=[None])
         else:
             output_tensor = layer(self.outputs[0])
             if isinstance(output_tensor, list):
@@ -461,7 +487,7 @@ class Sequential(Model):
             self.outputs = [output_tensor]
             # update self.inbound_nodes
             self.inbound_nodes[0].output_tensors = self.outputs
-            self.inbound_nodes[0].output_shapes = [self.outputs[0]._keras_shape]
+            self.inbound_nodes[0].output_shapes = [K.int_shape(self.outputs[0])]
 
         self.layers.append(layer)
         self.built = False
@@ -485,7 +511,7 @@ class Sequential(Model):
             self.outputs = [self.layers[-1].output]
             # update self.inbound_nodes
             self.inbound_nodes[0].output_tensors = self.outputs
-            self.inbound_nodes[0].output_shapes = [self.outputs[0]._keras_shape]
+            self.inbound_nodes[0].output_shapes = [K.int_shape(self.outputs[0])]
         self.built = False
 
     def get_layer(self, name=None, index=None):
@@ -550,36 +576,9 @@ class Sequential(Model):
             self.build()
         return self.model.uses_learning_phase
 
-    @property
-    def _flattened_layers(self):
-        layers = []
-        if self.layers:
-            # Support for legacy models
-            if isinstance(self.layers[0], legacy_layers.Merge):
-                merge = self.layers[0]
-                for layer in merge.layers:
-                    if hasattr(layer, '_flattened_layers'):
-                        for sublayer in layer._flattened_layers:
-                            if sublayer not in layers:
-                                layers.append(sublayer)
-                    elif hasattr(layer, 'layers'):
-                        for sublayer in layer.layers:
-                            if sublayer not in layers:
-                                layers.append(sublayer)
-                    else:
-                        if layer not in layers:
-                            layers.append(layer)
-            else:
-                if self.layers[0] not in layers:
-                    layers.append(self.layers[0])
-            for layer in self.layers[1:]:
-                if layer not in layers:
-                    layers.append(layer)
-        return layers
-
     def _gather_list_attr(self, attr):
         all_attrs = []
-        for layer in self._flattened_layers:
+        for layer in self.layers:
             all_attrs += getattr(layer, attr, [])
         return all_attrs
 
@@ -597,12 +596,10 @@ class Sequential(Model):
     def trainable_weights(self):
         if not self.trainable:
             return []
-        # Support for legacy behavior
         return self._gather_list_attr('trainable_weights')
 
     @property
     def non_trainable_weights(self):
-        # Support for legacy behavior
         weights = self._gather_list_attr('non_trainable_weights')
         if not self.trainable:
             trainable_weights = self._gather_list_attr('trainable_weights')
@@ -656,14 +653,6 @@ class Sequential(Model):
             A flat list of Numpy arrays
             (one array per model weight).
         """
-        # Legacy support
-        if legacy_models.needs_legacy_support(self):
-            layers = legacy_models.legacy_sequential_layers(self)
-            weights = []
-            for layer in layers:
-                weights.append(layer.get_weights())
-            return weights
-
         if self.model is None:
             self.build()
         return self.model.get_weights()
@@ -676,14 +665,6 @@ class Sequential(Model):
                 of Numpy arrays with shapes and types matching
                 the output of `model.get_weights()`.
         """
-        # Legacy support
-        if legacy_models.needs_legacy_support(self):
-            layers = legacy_models.legacy_sequential_layers(self)
-            for layer in layers:
-                nb_param = len(layer.weights)
-                layer.set_weights(weights[:nb_param])
-                weights = weights[nb_param:]
-
         if self.model is None:
             self.build()
         self.model.set_weights(weights)
@@ -694,12 +675,7 @@ class Sequential(Model):
         f = h5py.File(filepath, mode='r')
         if 'layer_names' not in f.attrs and 'model_weights' in f:
             f = f['model_weights']
-
-        # Legacy support
-        if legacy_models.needs_legacy_support(self):
-            layers = legacy_models.legacy_sequential_layers(self)
-        else:
-            layers = self.layers
+        layers = self.layers
         if by_name:
             topology.load_weights_from_hdf5_group_by_name(f, layers)
         else:
@@ -715,12 +691,7 @@ class Sequential(Model):
             proceed = ask_to_proceed_with_overwrite(filepath)
             if not proceed:
                 return
-        # Legacy support
-        if legacy_models.needs_legacy_support(self):
-            layers = legacy_models.legacy_sequential_layers(self)
-        else:
-            layers = self.layers
-
+        layers = self.layers
         f = h5py.File(filepath, 'w')
         topology.save_weights_to_hdf5_group(f, layers)
         f.flush()
@@ -736,7 +707,7 @@ class Sequential(Model):
             optimizer: str (name of optimizer) or optimizer object.
                 See [optimizers](/optimizers).
             loss: str (name of objective function) or objective function.
-                See [objectives](/objectives).
+                See [losses](/losses).
             metrics: list of metrics to be evaluated by the model
                 during training and testing.
                 Typically you will use `metrics=['accuracy']`.
@@ -745,7 +716,8 @@ class Sequential(Model):
                 sample weighting (2D weights), set this to "temporal".
                 "None" defaults to sample-wise weights (1D).
             **kwargs: for Theano backend, these are passed into K.function.
-                Ignored for Tensorflow backend.
+                When using the Tensorflow backend, these are passed into
+                `tf.Session.run`.
 
         # Example
             ```python
@@ -766,15 +738,18 @@ class Sequential(Model):
                            **kwargs)
         self.optimizer = self.model.optimizer
         self.loss = self.model.loss
+        self.total_loss = self.model.total_loss
         self.loss_weights = self.model.loss_weights
         self.metrics = self.model.metrics
         self.metrics_tensors = self.model.metrics_tensors
         self.metrics_names = self.model.metrics_names
         self.sample_weight_mode = self.model.sample_weight_mode
+        self.sample_weights = self.model.sample_weights
+        self.targets = self.model.targets
 
     def fit(self, x, y, batch_size=32, epochs=10, verbose=1, callbacks=None,
             validation_split=0., validation_data=None, shuffle=True,
-            class_weight=None, sample_weight=None, initial_epoch=0, **kwargs):
+            class_weight=None, sample_weight=None, initial_epoch=0):
         """Trains the model for a fixed number of epochs.
 
         # Arguments
@@ -821,14 +796,6 @@ class Sequential(Model):
         # Raises
             RuntimeError: if the model was never compiled.
         """
-        # Legacy support
-        if 'nb_epoch' in kwargs:
-            warnings.warn('The `nb_epoch` argument in `fit` '
-                          'has been renamed `epochs`.')
-            epochs = kwargs.pop('nb_epoch')
-        if kwargs:
-            raise TypeError('Unrecognized keyword arguments: ' + str(kwargs))
-
         if self.model is None:
             raise RuntimeError('The model needs to be compiled '
                                'before being used.')
@@ -999,7 +966,6 @@ class Sequential(Model):
         else:
             return (proba > 0.5).astype('int32')
 
-    @interfaces.legacy_generator_methods_support
     def fit_generator(self, generator,
                       steps_per_epoch,
                       epochs=1,
@@ -1025,12 +991,12 @@ class Sequential(Model):
                 - a tuple (inputs, targets, sample_weights).
                 All arrays should contain the same number of samples.
                 The generator is expected to loop over its data
-                indefinitely. An epoch finishes when `samples_per_epoch`
-                samples have been seen by the model.
+                indefinitely. An epoch finishes when `steps_per_epoch`
+                batches have been seen by the model.
             steps_per_epoch: Total number of steps (batches of samples)
                 to yield from `generator` before declaring one epoch
                 finished and starting the next epoch. It should typically
-                be equal to the number of unique samples if your dataset
+                be equal to the number of unique samples of your dataset
                 divided by the batch size.
             epochs: Integer, total number of iterations on the data.
             verbose: Verbosity mode, 0, 1, or 2.
@@ -1041,8 +1007,10 @@ class Sequential(Model):
                 - A tuple (inputs, targets, sample_weights).
             validation_steps: Only relevant if `validation_data`
                 is a generator.
-                Number of samples to use from validation generator
-                at the end of every epoch.
+                Number of steps to yield from validation generator
+                at the end of every epoch. It should typically
+                be equal to the number of unique samples of your
+                validation dataset divided by the batch size.
             class_weight: Dictionary mapping class indices to a weight
                 for the class.
             max_q_size: Maximum size for the generator queue
@@ -1074,10 +1042,10 @@ class Sequential(Model):
                         # and labels, from each line in the file
                         x, y = process_line(line)
                         yield (x, y)
-                    f.close()
+                        f.close()
 
             model.fit_generator(generate_arrays_from_file('/my_file.txt'),
-                                samples_per_epoch=10000, epochs=10)
+                                steps_per_epoch=1000, epochs=10)
         ```
         """
         if self.model is None:
@@ -1096,7 +1064,6 @@ class Sequential(Model):
                                         pickle_safe=pickle_safe,
                                         initial_epoch=initial_epoch)
 
-    @interfaces.legacy_generator_methods_support
     def evaluate_generator(self, generator, steps,
                            max_q_size=10, workers=1,
                            pickle_safe=False):
@@ -1136,9 +1103,9 @@ class Sequential(Model):
                                              workers=workers,
                                              pickle_safe=pickle_safe)
 
-    @interfaces.legacy_generator_methods_support
     def predict_generator(self, generator, steps,
-                          max_q_size=10, workers=1, pickle_safe=False):
+                          max_q_size=10, workers=1,
+                          pickle_safe=False, verbose=0):
         """Generates predictions for the input samples from a data generator.
 
         The generator should return the same kind of data as accepted by
@@ -1155,6 +1122,7 @@ class Sequential(Model):
                 relies on multiprocessing, you should not pass
                 non picklable arguments to the generator
                 as they can't be passed easily to children processes.
+            verbose: verbosity mode, 0 or 1.
 
         # Returns
             A Numpy array of predictions.
@@ -1164,12 +1132,10 @@ class Sequential(Model):
         return self.model.predict_generator(generator, steps,
                                             max_q_size=max_q_size,
                                             workers=workers,
-                                            pickle_safe=pickle_safe)
+                                            pickle_safe=pickle_safe,
+                                            verbose=verbose)
 
     def get_config(self):
-        if isinstance(self.layers[0], legacy_layers.Merge):
-            return self.legacy_get_config()
-
         config = []
         for layer in self.layers:
             config.append({'class_name': layer.__class__.__name__,
@@ -1177,83 +1143,9 @@ class Sequential(Model):
         return copy.deepcopy(config)
 
     @classmethod
-    def from_config(cls, config):
-        if 'class_name' not in config[0] or config[0]['class_name'] == 'Merge':
-            return cls.legacy_from_config(config)
-
+    def from_config(cls, config, custom_objects=None):
         model = cls()
         for conf in config:
-            layer = layer_module.deserialize(conf)
-            model.add(layer)
-        return model
-
-    def legacy_get_config(self):
-        """Retrieves the model configuration as a Python list.
-
-        # Returns
-            A list of dicts (each dict is a layer config).
-        """
-        config = []
-        if isinstance(self.layers[0], legacy_layers.Merge):
-            assert hasattr(self.layers[0], 'layers')
-            layers = []
-            for layer in self.layers[0].layers:
-                layer_config = {'class_name': layer.__class__.__name__,
-                                'config': layer.get_config()}
-                layers.append(layer_config)
-            merge_config = self.layers[0].get_config()
-            merge_config['layers'] = layers
-            config.append({'class_name': 'Merge', 'config': merge_config})
-        else:
-            config.append({'class_name': self.layers[0].__class__.__name__,
-                           'config': self.layers[0].get_config()})
-        for layer in self.layers[1:]:
-            config.append({'class_name': layer.__class__.__name__,
-                           'config': layer.get_config()})
-        return copy.deepcopy(config)
-
-    @classmethod
-    def legacy_from_config(cls, config, layer_cache=None):
-        if not layer_cache:
-            layer_cache = {}
-
-        def normalize_legacy_config(conf):
-            if 'class_name' not in conf:
-                class_name = conf['name']
-                name = conf.get('custom_name')
-                conf['name'] = name
-                return {'class_name': class_name,
-                        'config': conf}
-            return conf
-
-        # the model we will return
-        model = cls()
-
-        def get_or_create_layer(layer_data):
-            name = layer_data['config'].get('name')
-            if name in layer_cache:
-                return layer_cache[name]
-            layer = layer_module.deserialize(layer_data)
-            layer_cache[name] = layer
-            return layer
-
-        first_layer = config[0]
-        first_layer = normalize_legacy_config(first_layer)
-        if first_layer['class_name'] == 'Merge':
-            merge_inputs = []
-            first_layer_config = first_layer['config']
-            for merge_input_config in first_layer_config.pop('layers'):
-                merge_input = layer_module.deserialize(merge_input_config)
-                merge_inputs.append(merge_input)
-            first_layer_config['layers'] = merge_inputs
-            merge = legacy_layers.Merge.from_config(first_layer_config)
-            model.add(merge)
-        else:
-            layer = get_or_create_layer(first_layer)
-            model.add(layer)
-
-        for conf in config[1:]:
-            conf = normalize_legacy_config(conf)
-            layer = get_or_create_layer(conf)
+            layer = layer_module.deserialize(conf, custom_objects=custom_objects)
             model.add(layer)
         return model

keras/objectives.py

@@ -1,6 +0,0 @@
-"""Legacy objectives module.
-
-Only kept for backwards API compatibility.
-"""
-from __future__ import absolute_import
-from .losses import *

keras/optimizers.py

@@ -1,13 +1,15 @@
+"""Keras optimizer classes (will eventually be replaced with core optimizers).
+"""
 from __future__ import absolute_import
-import six
-from six.moves import zip
+from __future__ import division
+from __future__ import print_function
 
 from . import backend as K
-from .utils.generic_utils import serialize_keras_object
+import six
+from six.moves import zip
+from tensorflow.python.training import optimizer as tf_optimizer_module
 from .utils.generic_utils import deserialize_keras_object
-
-if K.backend() == 'tensorflow':
-    import tensorflow as tf
+from .utils.generic_utils import serialize_keras_object
 
 
 def clip_norm(g, c, n):
@@ -134,7 +136,7 @@ class SGD(Optimizer):
             self.updates .append(K.update_add(self.iterations, 1))
 
         # momentum
-        shapes = [K.get_variable_shape(p) for p in params]
+        shapes = [K.int_shape(p) for p in params]
         moments = [K.zeros(shape) for shape in shapes]
         self.weights = [self.iterations] + moments
         for p, g, m in zip(params, grads, moments):
@@ -164,6 +166,7 @@ class SGD(Optimizer):
 
 
 class RMSprop(Optimizer):
+    # pylint: disable=line-too-long
     """RMSProp optimizer.
 
     It is recommended to leave the parameters of this optimizer
@@ -182,6 +185,7 @@ class RMSprop(Optimizer):
     # References
         - [rmsprop: Divide the gradient by a running average of its recent magnitude](http://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf)
     """
+    # pylint: enable=line-too-long
 
     def __init__(self, lr=0.001, rho=0.9, epsilon=1e-8, decay=0.,
                  **kwargs):
@@ -195,7 +199,7 @@ class RMSprop(Optimizer):
 
     def get_updates(self, params, constraints, loss):
         grads = self.get_gradients(loss, params)
-        shapes = [K.get_variable_shape(p) for p in params]
+        shapes = [K.int_shape(p) for p in params]
         accumulators = [K.zeros(shape) for shape in shapes]
         self.weights = accumulators
         self.updates = []
@@ -228,6 +232,7 @@ class RMSprop(Optimizer):
 
 
 class Adagrad(Optimizer):
+    # pylint: disable=line-too-long
     """Adagrad optimizer.
 
     It is recommended to leave the parameters of this optimizer
@@ -241,6 +246,7 @@ class Adagrad(Optimizer):
     # References
         - [Adaptive Subgradient Methods for Online Learning and Stochastic Optimization](http://www.jmlr.org/papers/volume12/duchi11a/duchi11a.pdf)
     """
+    # pylint: enable=line-too-long
 
     def __init__(self, lr=0.01, epsilon=1e-8, decay=0., **kwargs):
         super(Adagrad, self).__init__(**kwargs)
@@ -252,7 +258,7 @@ class Adagrad(Optimizer):
 
     def get_updates(self, params, constraints, loss):
         grads = self.get_gradients(loss, params)
-        shapes = [K.get_variable_shape(p) for p in params]
+        shapes = [K.int_shape(p) for p in params]
         accumulators = [K.zeros(shape) for shape in shapes]
         self.weights = accumulators
         self.updates = []
@@ -282,6 +288,7 @@ class Adagrad(Optimizer):
 
 
 class Adadelta(Optimizer):
+    # pylint: disable=line-too-long
     """Adadelta optimizer.
 
     It is recommended to leave the parameters of this optimizer
@@ -297,6 +304,7 @@ class Adadelta(Optimizer):
     # References
         - [Adadelta - an adaptive learning rate method](http://arxiv.org/abs/1212.5701)
     """
+    # pylint: enable=line-too-long
 
     def __init__(self, lr=1.0, rho=0.95, epsilon=1e-8, decay=0.,
                  **kwargs):
@@ -310,7 +318,7 @@ class Adadelta(Optimizer):
 
     def get_updates(self, params, constraints, loss):
         grads = self.get_gradients(loss, params)
-        shapes = [K.get_variable_shape(p) for p in params]
+        shapes = [K.int_shape(p) for p in params]
         accumulators = [K.zeros(shape) for shape in shapes]
         delta_accumulators = [K.zeros(shape) for shape in shapes]
         self.weights = accumulators + delta_accumulators
@@ -351,6 +359,7 @@ class Adadelta(Optimizer):
 
 
 class Adam(Optimizer):
+    # pylint: disable=line-too-long
     """Adam optimizer.
 
     Default parameters follow those provided in the original paper.
@@ -365,6 +374,7 @@ class Adam(Optimizer):
     # References
         - [Adam - A Method for Stochastic Optimization](http://arxiv.org/abs/1412.6980v8)
     """
+    # pylint: enable=line-too-long
 
     def __init__(self, lr=0.001, beta_1=0.9, beta_2=0.999,
                  epsilon=1e-8, decay=0., **kwargs):
@@ -389,7 +399,7 @@ class Adam(Optimizer):
         lr_t = lr * (K.sqrt(1. - K.pow(self.beta_2, t)) /
                      (1. - K.pow(self.beta_1, t)))
 
-        shapes = [K.get_variable_shape(p) for p in params]
+        shapes = [K.int_shape(p) for p in params]
         ms = [K.zeros(shape) for shape in shapes]
         vs = [K.zeros(shape) for shape in shapes]
         self.weights = [self.iterations] + ms + vs
@@ -421,6 +431,7 @@ class Adam(Optimizer):
 
 
 class Adamax(Optimizer):
+    # pylint: disable=line-too-long
     """Adamax optimizer from Adam paper's Section 7.
 
     It is a variant of Adam based on the infinity norm.
@@ -435,6 +446,7 @@ class Adamax(Optimizer):
     # References
         - [Adam - A Method for Stochastic Optimization](http://arxiv.org/abs/1412.6980v8)
     """
+    # pylint: enable=line-too-long
 
     def __init__(self, lr=0.002, beta_1=0.9, beta_2=0.999,
                  epsilon=1e-8, decay=0., **kwargs):
@@ -458,7 +470,7 @@ class Adamax(Optimizer):
         t = self.iterations + 1
         lr_t = lr / (1. - K.pow(self.beta_1, t))
 
-        shapes = [K.get_variable_shape(p) for p in params]
+        shapes = [K.int_shape(p) for p in params]
         # zero init of 1st moment
         ms = [K.zeros(shape) for shape in shapes]
         # zero init of exponentially weighted infinity norm
@@ -493,6 +505,7 @@ class Adamax(Optimizer):
 
 
 class Nadam(Optimizer):
+    # pylint: disable=line-too-long
     """Nesterov Adam optimizer.
 
     Much like Adam is essentially RMSprop with momentum,
@@ -511,6 +524,7 @@ class Nadam(Optimizer):
         - [Nadam report](http://cs229.stanford.edu/proj2015/054_report.pdf)
         - [On the importance of initialization and momentum in deep learning](http://www.cs.toronto.edu/~fritz/absps/momentum.pdf)
     """
+    # pylint: enable=line-too-long
 
     def __init__(self, lr=0.002, beta_1=0.9, beta_2=0.999,
                  epsilon=1e-8, schedule_decay=0.004, **kwargs):
@@ -536,7 +550,7 @@ class Nadam(Optimizer):
         m_schedule_next = self.m_schedule * momentum_cache_t * momentum_cache_t_1
         self.updates.append((self.m_schedule, m_schedule_new))
 
-        shapes = [K.get_variable_shape(p) for p in params]
+        shapes = [K.int_shape(p) for p in params]
         ms = [K.zeros(shape) for shape in shapes]
         vs = [K.zeros(shape) for shape in shapes]
 
@@ -578,7 +592,7 @@ class TFOptimizer(Optimizer):
     """Wrapper class for native TensorFlow optimizers.
     """
 
-    def __init__(self, optimizer):
+    def __init__(self, optimizer):  # pylint: disable=super-init-not-called
         self.optimizer = optimizer
         self.iterations = K.variable(0., name='iterations')
         self.updates = []
@@ -608,6 +622,7 @@ class TFOptimizer(Optimizer):
 
 # Aliases.
 
+# pylint: disable=invalid-name
 sgd = SGD
 rmsprop = RMSprop
 adagrad = Adagrad
@@ -615,6 +630,7 @@ adadelta = Adadelta
 adam = Adam
 adamax = Adamax
 nadam = Nadam
+# pylint: enable=invalid-name
 
 
 def serialize(optimizer):
@@ -670,10 +686,9 @@ def get(identifier):
     # Raises
         ValueError: If `identifier` cannot be interpreted.
     """
-    if K.backend() == 'tensorflow':
-        # Wrap TF optimizer instances
-        if isinstance(identifier, tf.train.Optimizer):
-            return TFOptimizer(identifier)
+    # Wrap TF optimizer instances
+    if isinstance(identifier, tf_optimizer_module.Optimizer):
+        return TFOptimizer(identifier)
     if isinstance(identifier, dict):
         return deserialize(identifier)
     elif isinstance(identifier, six.string_types):

keras/preprocessing/__init__.py

@@ -0,0 +1,9 @@
+"""Data preprocessing module.
+"""
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+from . import image
+from . import sequence
+from . import text

keras/preprocessing/image.py

@@ -1,25 +1,33 @@
 """Fairly basic set of tools for real-time data augmentation on image data.
+
 Can easily be extended to include new transformations,
 new preprocessing methods, etc...
 """
 from __future__ import absolute_import
+from __future__ import division
 from __future__ import print_function
 
-import numpy as np
-import re
-from scipy import linalg
-import scipy.ndimage as ndi
-from six.moves import range
 import os
+import re
 import threading
 import warnings
 
 from .. import backend as K
+import numpy as np
+from six.moves import range
 
+# pylint: disable=g-import-not-at-top
 try:
     from PIL import Image as pil_image
 except ImportError:
     pil_image = None
+try:
+    from scipy import linalg
+    import scipy.ndimage as ndi
+except ImportError:
+    linalg = None
+    ndi = None
+# pylint: enable=g-import-not-at-top
 
 
 def random_rotation(x, rg, row_axis=1, col_axis=2, channel_axis=0,
@@ -250,7 +258,7 @@ def array_to_img(x, data_format=None, scale=True):
     if data_format == 'channels_first':
         x = x.transpose(1, 2, 0)
     if scale:
-        x = x + max(-np.min(x), 0)
+        x = x + max(-np.min(x), 0)  # pylint: disable=g-no-augmented-assignment
         x_max = np.max(x)
         if x_max != 0:
             x /= x_max
@@ -325,9 +333,9 @@ def load_img(path, grayscale=False, target_size=None):
         if img.mode != 'RGB':
             img = img.convert('RGB')
     if target_size:
-        wh_tuple = (target_size[1], target_size[0])
-        if img.size != wh_tuple:
-            img = img.resize(wh_tuple)
+        hw_tuple = (target_size[1], target_size[0])
+        if img.size != hw_tuple:
+            img = img.resize(hw_tuple)
     return img
 
 
@@ -346,6 +354,7 @@ class ImageDataGenerator(object):
         featurewise_std_normalization: divide inputs by std of the dataset.
         samplewise_std_normalization: divide each input by its std.
         zca_whitening: apply ZCA whitening.
+        zca_epsilon: epsilon for ZCA whitening. Default is 1e-6.
         rotation_range: degrees (0 to 180).
         width_shift_range: fraction of total width.
         height_shift_range: fraction of total height.
@@ -382,6 +391,7 @@ class ImageDataGenerator(object):
                  featurewise_std_normalization=False,
                  samplewise_std_normalization=False,
                  zca_whitening=False,
+                 zca_epsilon=1e-6,
                  rotation_range=0.,
                  width_shift_range=0.,
                  height_shift_range=0.,
@@ -402,6 +412,7 @@ class ImageDataGenerator(object):
         self.featurewise_std_normalization = featurewise_std_normalization
         self.samplewise_std_normalization = samplewise_std_normalization
         self.zca_whitening = zca_whitening
+        self.zca_epsilon = zca_epsilon
         self.rotation_range = rotation_range
         self.width_shift_range = width_shift_range
         self.height_shift_range = height_shift_range
@@ -443,7 +454,7 @@ class ImageDataGenerator(object):
                              'Received arg: ', zoom_range)
 
     def flow(self, x, y=None, batch_size=32, shuffle=True, seed=None,
-             save_to_dir=None, save_prefix='', save_format='jpeg'):
+             save_to_dir=None, save_prefix='', save_format='png'):
         return NumpyArrayIterator(
             x, y, self,
             batch_size=batch_size,
@@ -460,7 +471,7 @@ class ImageDataGenerator(object):
                             batch_size=32, shuffle=True, seed=None,
                             save_to_dir=None,
                             save_prefix='',
-                            save_format='jpeg',
+                            save_format='png',
                             follow_links=False):
         return DirectoryIterator(
             directory, self,
@@ -529,7 +540,13 @@ class ImageDataGenerator(object):
 
         # Returns
             A randomly transformed version of the input (same shape).
+
+        # Raises
+            ImportError: if Scipy is not available.
         """
+        if ndi is None:
+            raise ImportError('Scipy is required for image transformations.')
+
         # x is a single image, so it doesn't have image number at index 0
         img_row_axis = self.row_axis - 1
         img_col_axis = self.col_axis - 1
@@ -628,6 +645,7 @@ class ImageDataGenerator(object):
 
         # Raises
             ValueError: in case of invalid input `x`.
+            ImportError: if Scipy is not available.
         """
         x = np.asarray(x, dtype=K.floatx())
         if x.ndim != 4:
@@ -668,10 +686,13 @@ class ImageDataGenerator(object):
             x /= (self.std + K.epsilon())
 
         if self.zca_whitening:
+            if linalg is None:
+                raise ImportError('Scipy is required for zca_whitening.')
+
             flat_x = np.reshape(x, (x.shape[0], x.shape[1] * x.shape[2] * x.shape[3]))
             sigma = np.dot(flat_x.T, flat_x) / flat_x.shape[0]
             u, s, _ = linalg.svd(sigma)
-            self.principal_components = np.dot(np.dot(u, np.diag(1. / np.sqrt(s + 10e-7))), u.T)
+            self.principal_components = np.dot(np.dot(u, np.diag(1. / np.sqrt(s + self.zca_epsilon))), u.T)
 
 
 class Iterator(object):
@@ -708,7 +729,7 @@ class Iterator(object):
                     index_array = np.random.permutation(n)
 
             current_index = (self.batch_index * batch_size) % n
-            if n >= current_index + batch_size:
+            if n > current_index + batch_size:
                 current_batch_size = batch_size
                 self.batch_index += 1
             else:
@@ -718,7 +739,7 @@ class Iterator(object):
             yield (index_array[current_index: current_index + current_batch_size],
                    current_index, current_batch_size)
 
-    def __iter__(self):
+    def __iter__(self):  # pylint: disable=non-iterator-returned
         # Needed if we want to do something like:
         # for x, y in data_gen.flow(...):
         return self
@@ -752,7 +773,7 @@ class NumpyArrayIterator(Iterator):
     def __init__(self, x, y, image_data_generator,
                  batch_size=32, shuffle=False, seed=None,
                  data_format=None,
-                 save_to_dir=None, save_prefix='', save_format='jpeg'):
+                 save_to_dir=None, save_prefix='', save_format='png'):
         if y is not None and len(x) != len(y):
             raise ValueError('X (images tensor) and y (labels) '
                              'should have the same length. '
@@ -838,6 +859,8 @@ class DirectoryIterator(Iterator):
             `"binary"`: binary targets (if there are only two classes),
             `"categorical"`: categorical targets,
             `"sparse"`: integer targets,
+            `"input"`: targets are images identical to input images (mainly
+                used to work with autoencoders),
             `None`: no targets get yielded (only input images are yielded).
         batch_size: Integer, size of a batch.
         shuffle: Boolean, whether to shuffle the data between epochs.
@@ -858,7 +881,7 @@ class DirectoryIterator(Iterator):
                  classes=None, class_mode='categorical',
                  batch_size=32, shuffle=True, seed=None,
                  data_format=None,
-                 save_to_dir=None, save_prefix='', save_format='jpeg',
+                 save_to_dir=None, save_prefix='', save_format='png',
                  follow_links=False):
         if data_format is None:
             data_format = K.image_data_format()
@@ -881,10 +904,12 @@ class DirectoryIterator(Iterator):
             else:
                 self.image_shape = (1,) + self.target_size
         self.classes = classes
-        if class_mode not in {'categorical', 'binary', 'sparse', None}:
+        if class_mode not in {'categorical', 'binary', 'sparse',
+                              'input', None}:
             raise ValueError('Invalid class_mode:', class_mode,
                              '; expected one of "categorical", '
-                             '"binary", "sparse", or None.')
+                             '"binary", "sparse", "input"'
+                             ' or None.')
         self.class_mode = class_mode
         self.save_to_dir = save_to_dir
         self.save_prefix = save_prefix
@@ -972,7 +997,9 @@ class DirectoryIterator(Iterator):
                                                                   format=self.save_format)
                 img.save(os.path.join(self.save_to_dir, fname))
         # build batch of labels
-        if self.class_mode == 'sparse':
+        if self.class_mode == 'input':
+            batch_y = batch_x.copy()
+        elif self.class_mode == 'sparse':
             batch_y = self.classes[index_array]
         elif self.class_mode == 'binary':
             batch_y = self.classes[index_array].astype(K.floatx())

keras/preprocessing/sequence.py

@@ -1,8 +1,13 @@
 # -*- coding: utf-8 -*-
+"""Preprocessing utilities for sequence data.
+"""
 from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import random
 
 import numpy as np
-import random
 from six.moves import range
 
 
@@ -50,16 +55,16 @@ def pad_sequences(sequences, maxlen=None, dtype='int32',
     # checking for consistency in the main loop below.
     sample_shape = tuple()
     for s in sequences:
-        if len(s) > 0:
+        if len(s) > 0:  # pylint: disable=g-explicit-length-test
             sample_shape = np.asarray(s).shape[1:]
             break
 
     x = (np.ones((num_samples, maxlen) + sample_shape) * value).astype(dtype)
     for idx, s in enumerate(sequences):
-        if not len(s):
+        if not len(s):  # pylint: disable=g-explicit-length-test
             continue  # empty list/array was found
         if truncating == 'pre':
-            trunc = s[-maxlen:]
+            trunc = s[-maxlen:]  # pylint: disable=invalid-unary-operand-type
         elif truncating == 'post':
             trunc = s[:maxlen]
         else: