Update backend docs.

* conv3d_tranpose in tf and th

* fix _preprocess_deconv_output_shape error

* cntk conv3d_tranpose

* conv3d_tranpose test

* formatting

* cleanup tests

* fix incorrect axis ordering and docs

* fix incorrect axis ordering and docs

* deconv3d_output_shape to fix errors

* remove conv2d_transpose reference in theano backend

* remove kernel_size loop from test

* put depth first in test and add dim to invalid use case input

* formatting - removed extra line

* fix pep8

* remove extraneous args from tf conv3d_transpose function

* default val for data_format=None

.gitignore

@@ -19,3 +19,4 @@ examples/img/*
 
 # developer environments
 .idea
+.vscode

.travis.yml

@@ -14,9 +14,9 @@ matrix:
         - python: 3.5
           env: KERAS_BACKEND=tensorflow
         - python: 2.7
-          env: KERAS_BACKEND=theano
+          env: KERAS_BACKEND=theano THEANO_FLAGS=optimizer=fast_compile
         - python: 3.5
-          env: KERAS_BACKEND=theano
+          env: KERAS_BACKEND=theano THEANO_FLAGS=optimizer=fast_compile
         - python: 2.7
           env: KERAS_BACKEND=cntk
         - python: 3.5
@@ -86,5 +86,5 @@ script:
     elif [[ "$TEST_MODE" == "DOC" ]]; then
         PYTHONPATH=$PWD:$PYTHONPATH py.test tests/test_documentation.py;
     else
-       PYTHONPATH=$PWD:$PYTHONPATH py.test tests/ --ignore=tests/integration_tests --ignore=tests/test_documentation.py --cov=keras tests/ --cov-fail-under 78 --cov-report term-missing;
+       PYTHONPATH=$PWD:$PYTHONPATH py.test tests/ --ignore=tests/integration_tests --ignore=tests/test_documentation.py --cov=keras tests/ --cov-fail-under 80 --cov-report term-missing;
     fi

CONTRIBUTING.md

@@ -46,13 +46,20 @@ You can also use Github issues to request features you would like to see in Kera
 **Where should I submit my pull request?**
 
 1. **Keras improvements and bugfixes** go to the [Keras `master` branch](https://github.com/fchollet/keras/tree/master).
-2. **New features** such as layers and datasets go to [keras-contrib](https://github.com/farizrahman4u/keras-contrib). Unless it is a new feature listed in [Requests for Contributions](https://github.com/fchollet/keras/projects/1), in which case it belongs in core Keras.
+2. **Experimental new features** such as layers and datasets go to [keras-contrib](https://github.com/farizrahman4u/keras-contrib). Unless it is a new feature listed in [Requests for Contributions](https://github.com/fchollet/keras/projects/1), in which case it belongs in core Keras. If you think your feature belongs in core Keras, you can submit a design doc to explain your feature and argue for it (see explainations below).
 
 Here's a quick guide to submitting your improvements:
 
-1. If your PR introduces a change in functionality, make sure you start by opening an issue to discuss whether the change should be made, and how to handle it. This will save you from having your PR closed down the road! Of course, if your PR is a simple bug fix, you don't need to do that.
+1. If your PR introduces a change in functionality, make sure you start by writing a design doc and sending it to the Keras mailing list to discuss whether the change should be made, and how to handle it. This will save you from having your PR closed down the road! Of course, if your PR is a simple bug fix, you don't need to do that. The process for writing and submitting design docs is as follow:
+    - Start from [this Google Doc template](https://docs.google.com/document/d/1ZXNfce77LDW9tFAj6U5ctaJmI5mT7CQXOFMEAZo-mAA/edit#), and copy it to new Google doc.
+    - Fill in the content. Note that you will need to insert code examples. To insert code, use a Google Doc extension such as [CodePretty](https://chrome.google.com/webstore/detail/code-pretty/igjbncgfgnfpbnifnnlcmjfbnidkndnh?hl=en) (there are several such extensions available).
+    - Set sharing settings to "everyone with the link is allowed to comment"
+    - Send the document to `keras-users@googlegroups.com` with a subject that starts with `[API DESIGN REVIEW]` (all caps) so that we notice it.
+    - Wait for comments, and answer them as they come. Edit the proposal as necessary.
+    - The proposal will finally be approved or rejected. Once approved, you can send out Pull Requests or ask others to write Pull Requests.
 
-2. Write the code. This is the hard part!
+
+2. Write the code (or get others to write it). This is the hard part!
 
 3. Make sure any new function or class you introduce has proper docstrings. Make sure any code you touch still has up-to-date docstrings and documentation. **Docstring style should be respected.** In particular, they should be formatted in MarkDown, and there should be sections for `Arguments`, `Returns`, `Raises` (if applicable). Look at other docstrings in the codebase for examples.
 

README.md

@@ -5,7 +5,7 @@
 
 ## You have just found Keras.
 
-Keras is a high-level neural networks API, written in Python and capable of running on top of either [TensorFlow](https://github.com/tensorflow/tensorflow), [CNTK](https://github.com/Microsoft/cntk) or [Theano](https://github.com/Theano/Theano). It was developed with a focus on enabling fast experimentation. *Being able to go from idea to result with the least possible delay is key to doing good research.*
+Keras is a high-level neural networks API, written in Python and capable of running on top of [TensorFlow](https://github.com/tensorflow/tensorflow), [CNTK](https://github.com/Microsoft/cntk), or [Theano](https://github.com/Theano/Theano). It was developed with a focus on enabling fast experimentation. *Being able to go from idea to result with the least possible delay is key to doing good research.*
 
 Use Keras if you need a deep learning library that:
 

docs/autogen.py

@@ -286,7 +286,8 @@ PAGES = [
         'page': 'utils.md',
         'all_module_functions': [utils],
         'classes': [utils.CustomObjectScope,
-                    utils.HDF5Matrix]
+                    utils.HDF5Matrix,
+                    utils.Sequence]
     },
 ]
 
@@ -320,9 +321,11 @@ def get_classes_ancestors(classes):
 
 
 def get_function_signature(function, method=True):
-    signature = getattr(function, '_legacy_support_signature', None)
-    if signature is None:
+    wrapped = getattr(function, '_original_function', None)
+    if wrapped is None:
         signature = inspect.getargspec(function)
+    else:
+        signature = inspect.getargspec(wrapped)
     defaults = signature.defaults
     if method:
         args = signature.args[1:]

docs/templates/constraints.md

@@ -17,6 +17,7 @@ model.add(Dense(64, kernel_constraint=max_norm(2.)))
 
 ## Available constraints
 
-- __max_norm__(max_value=2, axis=0): maximum-norm constraint
-- __non_neg__(): non-negativity constraint
-- __unit_norm__(): unit-norm constraint, enforces the matrix to have unit norm along the last axis
+- __max_norm(max_value=2, axis=0)__: maximum-norm constraint
+- __non_neg()__: non-negativity constraint
+- __unit_norm(axis=0)__: unit-norm constraint
+- __min_max_norm(min_value=0.0, max_value=1.0, rate=1.0, axis=0)__:  minimum/maximum-norm constraint

docs/templates/getting-started/faq.md

@@ -16,7 +16,7 @@
 - [How can I remove a layer from a Sequential model?](#how-can-i-remove-a-layer-from-a-sequential-model)
 - [How can I use pre-trained models in Keras?](#how-can-i-use-pre-trained-models-in-keras)
 - [How can I use HDF5 inputs with Keras?](#how-can-i-use-hdf5-inputs-with-keras)
-- [Where is the Keras configuration filed stored?](#where-is-the-keras-configuration-filed-stored)
+- [Where is the Keras configuration file stored?](#where-is-the-keras-configuration-file-stored)
 
 ---
 
@@ -424,7 +424,7 @@ with h5py.File('input/file.hdf5', 'r') as f:
 
 ---
 
-### Where is the Keras configuration filed stored?
+### Where is the Keras configuration file stored?
 
 The default directory where all Keras data is stored is:
 

docs/templates/preprocessing/text.md

@@ -2,8 +2,10 @@
 ## text_to_word_sequence
 
 ```python
-keras.preprocessing.text.text_to_word_sequence(text, 
-    filters='!"#$%&()*+,-./:;<=>?@[\\]^_`{|}~\t\n', lower=True, split=" ")
+keras.preprocessing.text.text_to_word_sequence(text,
+                                               filters='!"#$%&()*+,-./:;<=>?@[\\]^_`{|}~\t\n',
+                                               lower=True,
+                                               split=" ")
 ```
 
 Split a sentence into a list of words.
@@ -12,29 +14,74 @@ Split a sentence into a list of words.
 
 - __Arguments__:
     - __text__: str.
-    - __filters__: list (or concatenation) of characters to filter out, such as punctuation. Default: '!"#$%&()*+,-./:;<=>?@[\\]^_`{|}~\t\n' , includes basic punctuation, tabs, and newlines.
+    - __filters__: list (or concatenation) of characters to filter out, such as
+         punctuation. Default: '!"#$%&()*+,-./:;<=>?@[\\]^_`{|}~\t\n' , includes
+         basic punctuation, tabs, and newlines.
     - __lower__: boolean. Whether to set the text to lowercase.
     - __split__: str. Separator for word splitting.
 
 ## one_hot
 
 ```python
-keras.preprocessing.text.one_hot(text, n,
-    filters='!"#$%&()*+,-./:;<=>?@[\\]^_`{|}~\t\n', lower=True, split=" ")
+keras.preprocessing.text.one_hot(text,
+                                 n,
+                                 filters='!"#$%&()*+,-./:;<=>?@[\\]^_`{|}~\t\n',
+                                 lower=True,
+                                 split=" ")
 ```
 
-One-hot encode a text into a list of word indexes in a vocabulary of size n.
+One-hot encodes a text into a list of word indexes in a vocabulary of size n.
+
+This is a wrapper to the `hashing_trick` function using `hash` as the hashing function.
 
 - __Return__: List of integers in [1, n]. Each integer encodes a word (unicity non-guaranteed).
 
-- __Arguments__: Same as `text_to_word_sequence` above.
+- __Arguments__:
+    - __text__: str.
     - __n__: int. Size of vocabulary.
+    - __filters__: list (or concatenation) of characters to filter out, such as
+         punctuation. Default: '!"#$%&()*+,-./:;<=>?@[\\]^_`{|}~\t\n' , includes
+         basic punctuation, tabs, and newlines.
+    - __lower__: boolean. Whether to set the text to lowercase.
+    - __split__: str. Separator for word splitting.
+    
+## hashing_trick
+
+```python
+keras.preprocessing.text.hashing_trick(text, 
+                                       n,
+                                       hash_function=None,
+                                       filters='!"#$%&()*+,-./:;<=>?@[\\]^_`{|}~\t\n',
+                                       lower=True,
+                                       split=' ')
+```
+
+Converts a text to a sequence of indices in a fixed-size hashing space
+
+- __Return__:
+        A list of integer word indices (unicity non-guaranteed).
+- __Arguments__:
+    - __text__: str.
+    - __n__: Dimension of the hashing space.
+    - __hash_function__: defaults to python `hash` function, can be 'md5' or
+            any function that takes in input a string and returns a int.
+            Note that 'hash' is not a stable hashing function, so
+            it is not consistent across different runs, while 'md5'
+            is a stable hashing function.
+    - __filters__: list (or concatenation) of characters to filter out, such as
+         punctuation. Default: '!"#$%&()*+,-./:;<=>?@[\\]^_`{|}~\t\n' , includes
+         basic punctuation, tabs, and newlines.
+    - __lower__: boolean. Whether to set the text to lowercase.
+    - __split__: str. Separator for word splitting.
 
 ## Tokenizer
 
 ```python
-keras.preprocessing.text.Tokenizer(num_words=None, filters='!"#$%&()*+,-./:;<=>?@[\\]^_`{|}~\t\n', 
-    lower=True, split=" ", char_level=False)
+keras.preprocessing.text.Tokenizer(num_words=None,
+                                   filters='!"#$%&()*+,-./:;<=>?@[\\]^_`{|}~\t\n',
+                                   lower=True,
+                                   split=" ",
+                                   char_level=False)
 ```
 
 Class for vectorizing texts, or/and turning texts into sequences (=list of word indexes, where the word of rank i in the dataset (starting at 1) has index i).

examples/image_ocr.py

@@ -136,8 +136,8 @@ def shuffle_mats_or_lists(matrix_list, stop_ind=None):
         elif isinstance(mat, list):
             ret.append([mat[i] for i in a])
         else:
-            raise TypeError('shuffle_mats_or_lists only supports '
-                            'numpy.array and list objects')
+            raise TypeError('`shuffle_mats_or_lists` only supports '
+                            'numpy.array and list objects.')
     return ret
 
 

examples/lstm_benchmark.py

@@ -37,7 +37,7 @@ print('Loading data...')
 X_train = sequence.pad_sequences(X_train, max_length)
 X_test = sequence.pad_sequences(X_test, max_length)
 
-# Compile and train different models while meauring performance.
+# Compile and train different models while measuring performance.
 results = []
 for mode in modes:
     print('Testing mode: implementation={}'.format(mode))

examples/reuters_mlp_relu_vs_selu.py

@@ -0,0 +1,174 @@
+'''Compares self-normalizing MLPs with regular MLPs.
+
+Compares the performance of a simple MLP using two
+different activation functions: RELU and SELU
+on the Reuters newswire topic classification task.
+
+# Reference:
+  Klambauer, G., Unterthiner, T., Mayr, A., & Hochreiter, S. (2017).
+  Self-Normalizing Neural Networks. arXiv preprint arXiv:1706.02515.
+  https://arxiv.org/abs/1706.02515
+'''
+from __future__ import print_function
+
+import numpy as np
+import matplotlib.pyplot as plt
+import keras
+from keras.datasets import reuters
+from keras.models import Sequential
+from keras.layers import Dense, Activation, Dropout
+from keras.layers.noise import AlphaDropout
+from keras.preprocessing.text import Tokenizer
+
+max_words = 1000
+batch_size = 16
+epochs = 40
+plot = True
+
+
+def create_network(n_dense=6,
+                   dense_units=16,
+                   activation='selu',
+                   dropout=AlphaDropout,
+                   dropout_rate=0.1,
+                   kernel_initializer='lecun_normal',
+                   optimizer='adam',
+                   num_classes=1,
+                   max_words=max_words):
+    """Generic function to create a fully-connected neural network.
+
+    # Arguments
+        n_dense: int > 0. Number of dense layers.
+        dense_units: int > 0. Number of dense units per layer.
+        dropout: keras.layers.Layer. A dropout layer to apply.
+        dropout_rate: 0 <= float <= 1. The rate of dropout.
+        kernel_initializer: str. The initializer for the weights.
+        optimizer: str/keras.optimizers.Optimizer. The optimizer to use.
+        num_classes: int > 0. The number of classes to predict.
+        max_words: int > 0. The maximum number of words per data point.
+
+    # Returns
+        A Keras model instance (compiled).
+    """
+    model = Sequential()
+    model.add(Dense(dense_units, input_shape=(max_words,),
+                    kernel_initializer=kernel_initializer))
+    model.add(Activation(activation))
+    model.add(dropout(dropout_rate))
+
+    for i in range(n_dense - 1):
+        model.add(Dense(dense_units, kernel_initializer=kernel_initializer))
+        model.add(Activation(activation))
+        model.add(dropout(dropout_rate))
+
+    model.add(Dense(num_classes))
+    model.add(Activation('softmax'))
+    model.compile(loss='categorical_crossentropy',
+                  optimizer=optimizer,
+                  metrics=['accuracy'])
+    return model
+
+
+network1 = {
+    'n_dense': 6,
+    'dense_units': 16,
+    'activation': 'relu',
+    'dropout': Dropout,
+    'dropout_rate': 0.5,
+    'kernel_initializer': 'glorot_uniform',
+    'optimizer': 'sgd'
+}
+
+network2 = {
+    'n_dense': 6,
+    'dense_units': 16,
+    'activation': 'selu',
+    'dropout': AlphaDropout,
+    'dropout_rate': 0.1,
+    'kernel_initializer': 'lecun_normal',
+    'optimizer': 'sgd'
+}
+
+print('Loading data...')
+(x_train, y_train), (x_test, y_test) = reuters.load_data(num_words=max_words,
+                                                         test_split=0.2)
+print(len(x_train), 'train sequences')
+print(len(x_test), 'test sequences')
+
+num_classes = np.max(y_train) + 1
+print(num_classes, 'classes')
+
+print('Vectorizing sequence data...')
+tokenizer = Tokenizer(num_words=max_words)
+x_train = tokenizer.sequences_to_matrix(x_train, mode='binary')
+x_test = tokenizer.sequences_to_matrix(x_test, mode='binary')
+print('x_train shape:', x_train.shape)
+print('x_test shape:', x_test.shape)
+
+print('Convert class vector to binary class matrix '
+      '(for use with categorical_crossentropy)')
+y_train = keras.utils.to_categorical(y_train, num_classes)
+y_test = keras.utils.to_categorical(y_test, num_classes)
+print('y_train shape:', y_train.shape)
+print('y_test shape:', y_test.shape)
+
+print('\nBuilding network 1...')
+
+model1 = create_network(num_classes=num_classes, **network1)
+history_model1 = model1.fit(x_train,
+                            y_train,
+                            batch_size=batch_size,
+                            epochs=epochs,
+                            verbose=1,
+                            validation_split=0.1)
+
+score_model1 = model1.evaluate(x_test,
+                               y_test,
+                               batch_size=batch_size,
+                               verbose=1)
+
+
+print('\nBuilding network 2...')
+model2 = create_network(num_classes=num_classes, **network2)
+
+history_model2 = model2.fit(x_train,
+                            y_train,
+                            batch_size=batch_size,
+                            epochs=epochs,
+                            verbose=1,
+                            validation_split=0.1)
+
+score_model2 = model2.evaluate(x_test,
+                               y_test,
+                               batch_size=batch_size,
+                               verbose=1)
+
+print('\nNetwork 1 results')
+print('Hyperparameters:', network1)
+print('Test score:', score_model1[0])
+print('Test accuracy:', score_model1[1])
+print('Network 2 results')
+print('Hyperparameters:', network2)
+print('Test score:', score_model2[0])
+print('Test accuracy:', score_model2[1])
+
+plt.plot(range(epochs),
+         history_model1.history['val_loss'],
+         'g-',
+         label='Network 1 Val Loss')
+plt.plot(range(epochs),
+         history_model2.history['val_loss'],
+         'r-',
+         label='Network 2 Val Loss')
+plt.plot(range(epochs),
+         history_model1.history['loss'],
+         'g--',
+         label='Network 1 Loss')
+plt.plot(range(epochs),
+         history_model2.history['loss'],
+         'r--',
+         label='Network 2 Loss')
+plt.xlabel('Epochs')
+plt.ylabel('Loss')
+plt.legend()
+plt.savefig('comparison_of_networks.png')

examples/variational_autoencoder_deconv.py

@@ -79,7 +79,8 @@ decoder_deconv_1 = Conv2DTranspose(filters,
                                    padding='same',
                                    strides=1,
                                    activation='relu')
-decoder_deconv_2 = Conv2DTranspose(filters, num_conv,
+decoder_deconv_2 = Conv2DTranspose(filters,
+                                   kernel_size=num_conv,
                                    padding='same',
                                    strides=1,
                                    activation='relu')

keras/__init__.py

@@ -1,5 +1,6 @@
 from __future__ import absolute_import
 
+from . import utils
 from . import activations
 from . import applications
 from . import backend
@@ -7,7 +8,6 @@ from . import datasets
 from . import engine
 from . import layers
 from . import preprocessing
-from . import utils
 from . import wrappers
 from . import callbacks
 from . import constraints

keras/activations.py

@@ -34,6 +34,20 @@ def elu(x, alpha=1.0):
     return K.elu(x, alpha)
 
 
+def selu(x):
+    """Scaled Exponential Linear Unit. (Klambauer et al., 2017)
+
+    # Arguments
+        x: A tensor or variable to compute the activation function for.
+
+    # References
+        - [Self-Normalizing Neural Networks](https://arxiv.org/abs/1706.02515)
+    """
+    alpha = 1.6732632423543772848170429916717
+    scale = 1.0507009873554804934193349852946
+    return scale * K.elu(x, alpha)
+
+
 def softplus(x):
     return K.softplus(x)
 

keras/applications/__init__.py

@@ -3,3 +3,4 @@ from .vgg19 import VGG19
 from .resnet50 import ResNet50
 from .inception_v3 import InceptionV3
 from .xception import Xception
+from .mobilenet import MobileNet

keras/applications/mobilenet.py

@@ -0,0 +1,641 @@
+"""MobileNet v1 models for Keras.
+
+MobileNet is a general architecture and can be used for multiple use cases.
+Depending on the use case, it can use different input layer size and
+different width factors. This allows different width models to reduce
+the number of multiply-adds and thereby
+reduce inference cost on mobile devices.
+
+MobileNets support any input size greater than 32 x 32, with larger image sizes
+offering better performance.
+The number of parameters and number of multiply-adds
+can be modified by using the `alpha` parameter,
+which increases/decreases the number of filters in each layer.
+By altering the image size and `alpha` parameter,
+all 16 models from the paper can be built, with ImageNet weights provided.
+
+The paper demonstrates the performance of MobileNets using `alpha` values of
+1.0 (also called 100 % MobileNet), 0.75, 0.5 and 0.25.
+For each of these `alpha` values, weights for 4 different input image sizes
+are provided (224, 192, 160, 128).
+
+The following table describes the size and accuracy of the 100% MobileNet
+on size 224 x 224:
+----------------------------------------------------------------------------
+Width Multiplier (alpha) | ImageNet Acc |  Multiply-Adds (M) |  Params (M)
+----------------------------------------------------------------------------
+|   1.0 MobileNet-224    |    70.6 %     |        529        |     4.2     |
+|   0.75 MobileNet-224   |    68.4 %     |        325        |     2.6     |
+|   0.50 MobileNet-224   |    63.7 %     |        149        |     1.3     |
+|   0.25 MobileNet-224   |    50.6 %     |        41         |     0.5     |
+----------------------------------------------------------------------------
+
+The following table describes the performance of
+the 100 % MobileNet on various input sizes:
+------------------------------------------------------------------------
+      Resolution      | ImageNet Acc | Multiply-Adds (M) | Params (M)
+------------------------------------------------------------------------
+|  1.0 MobileNet-224  |    70.6 %    |        529        |     4.2     |
+|  1.0 MobileNet-192  |    69.1 %    |        529        |     4.2     |
+|  1.0 MobileNet-160  |    67.2 %    |        529        |     4.2     |
+|  1.0 MobileNet-128  |    64.4 %    |        529        |     4.2     |
+------------------------------------------------------------------------
+
+The weights for all 16 models are obtained and translated
+from Tensorflow checkpoints found at
+https://github.com/tensorflow/models/blob/master/slim/nets/mobilenet_v1.md
+
+# Reference
+- [MobileNets: Efficient Convolutional Neural Networks for
+   Mobile Vision Applications](https://arxiv.org/pdf/1704.04861.pdf))
+"""
+from __future__ import print_function
+from __future__ import absolute_import
+from __future__ import division
+
+import warnings
+
+from ..models import Model
+from ..layers import Input
+from ..layers import Activation
+from ..layers import Dropout
+from ..layers import Reshape
+from ..layers import BatchNormalization
+from ..layers import GlobalAveragePooling2D
+from ..layers import GlobalMaxPooling2D
+from ..layers import Conv2D
+from .. import initializers
+from .. import regularizers
+from .. import constraints
+from ..utils import conv_utils
+from ..utils.data_utils import get_file
+from ..engine.topology import get_source_inputs
+from ..engine import InputSpec
+from ..applications.imagenet_utils import _obtain_input_shape
+from ..applications.imagenet_utils import decode_predictions
+from .. import backend as K
+
+
+BASE_WEIGHT_PATH = 'https://github.com/fchollet/deep-learning-models/releases/download/v0.6/'
+
+
+def relu6(x):
+    return K.relu(x, max_value=6)
+
+
+def preprocess_input(x):
+    x /= 255.
+    x -= 0.5
+    x *= 2.
+    return x
+
+
+class DepthwiseConv2D(Conv2D):
+    """Depthwise separable 2D convolution.
+
+    Depthwise Separable convolutions consists in performing
+    just the first step in a depthwise spatial convolution
+    (which acts on each input channel separately).
+    The `depth_multiplier` argument controls how many
+    output channels are generated per input channel in the depthwise step.
+
+    # Arguments
+        kernel_size: An integer or tuple/list of 2 integers, specifying the
+            width and height of the 2D convolution window.
+            Can be a single integer to specify the same value for
+            all spatial dimensions.
+        strides: An integer or tuple/list of 2 integers,
+            specifying the strides of the convolution along the width and height.
+            Can be a single integer to specify the same value for
+            all spatial dimensions.
+            Specifying any stride value != 1 is incompatible with specifying
+            any `dilation_rate` value != 1.
+        padding: one of `"valid"` or `"same"` (case-insensitive).
+        depth_multiplier: The number of depthwise convolution output channels
+            for each input channel.
+            The total number of depthwise convolution output
+            channels will be equal to `filters_in * depth_multiplier`.
+        data_format: A string,
+            one of `channels_last` (default) or `channels_first`.
+            The ordering of the dimensions in the inputs.
+            `channels_last` corresponds to inputs with shape
+            `(batch, height, width, channels)` while `channels_first`
+            corresponds to inputs with shape
+            `(batch, channels, height, width)`.
+            It defaults to the `image_data_format` value found in your
+            Keras config file at `~/.keras/keras.json`.
+            If you never set it, then it will be "channels_last".
+        activation: Activation function to use
+            (see [activations](../activations.md)).
+            If you don't specify anything, no activation is applied
+            (ie. "linear" activation: `a(x) = x`).
+        use_bias: Boolean, whether the layer uses a bias vector.
+        depthwise_initializer: Initializer for the depthwise kernel matrix
+            (see [initializers](../initializers.md)).
+        bias_initializer: Initializer for the bias vector
+            (see [initializers](../initializers.md)).
+        depthwise_regularizer: Regularizer function applied to
+            the depthwise kernel matrix
+            (see [regularizer](../regularizers.md)).
+        bias_regularizer: Regularizer function applied to the bias vector
+            (see [regularizer](../regularizers.md)).
+        activity_regularizer: Regularizer function applied to
+            the output of the layer (its "activation").
+            (see [regularizer](../regularizers.md)).
+        depthwise_constraint: Constraint function applied to
+            the depthwise kernel matrix
+            (see [constraints](../constraints.md)).
+        bias_constraint: Constraint function applied to the bias vector
+            (see [constraints](../constraints.md)).
+
+    # Input shape
+        4D tensor with shape:
+        `[batch, channels, rows, cols]` if data_format='channels_first'
+        or 4D tensor with shape:
+        `[batch, rows, cols, channels]` if data_format='channels_last'.
+
+    # Output shape
+        4D tensor with shape:
+        `[batch, filters, new_rows, new_cols]` if data_format='channels_first'
+        or 4D tensor with shape:
+        `[batch, new_rows, new_cols, filters]` if data_format='channels_last'.
+        `rows` and `cols` values might have changed due to padding.
+    """
+
+    def __init__(self,
+                 kernel_size,
+                 strides=(1, 1),
+                 padding='valid',
+                 depth_multiplier=1,
+                 data_format=None,
+                 activation=None,
+                 use_bias=True,
+                 depthwise_initializer='glorot_uniform',
+                 bias_initializer='zeros',
+                 depthwise_regularizer=None,
+                 bias_regularizer=None,
+                 activity_regularizer=None,
+                 depthwise_constraint=None,
+                 bias_constraint=None,
+                 **kwargs):
+        super(DepthwiseConv2D, self).__init__(
+            filters=None,
+            kernel_size=kernel_size,
+            strides=strides,
+            padding=padding,
+            data_format=data_format,
+            activation=activation,
+            use_bias=use_bias,
+            bias_regularizer=bias_regularizer,
+            activity_regularizer=activity_regularizer,
+            bias_constraint=bias_constraint,
+            **kwargs)
+        self.depth_multiplier = depth_multiplier
+        self.depthwise_initializer = initializers.get(depthwise_initializer)
+        self.depthwise_regularizer = regularizers.get(depthwise_regularizer)
+        self.depthwise_constraint = constraints.get(depthwise_constraint)
+        self.bias_initializer = initializers.get(bias_initializer)
+
+    def build(self, input_shape):
+        if len(input_shape) < 4:
+            raise ValueError('Inputs to `DepthwiseConv2D` should have rank 4. '
+                             'Received input shape:', str(input_shape))
+        if self.data_format == 'channels_first':
+            channel_axis = 1
+        else:
+            channel_axis = 3
+        if input_shape[channel_axis] is None:
+            raise ValueError('The channel dimension of the inputs to '
+                             '`DepthwiseConv2D` '
+                             'should be defined. Found `None`.')
+        input_dim = int(input_shape[channel_axis])
+        depthwise_kernel_shape = (self.kernel_size[0],
+                                  self.kernel_size[1],
+                                  input_dim,
+                                  self.depth_multiplier)
+
+        self.depthwise_kernel = self.add_weight(
+            shape=depthwise_kernel_shape,
+            initializer=self.depthwise_initializer,
+            name='depthwise_kernel',
+            regularizer=self.depthwise_regularizer,
+            constraint=self.depthwise_constraint)
+
+        if self.use_bias:
+            self.bias = self.add_weight(shape=(input_dim * self.depth_multiplier,),
+                                        initializer=self.bias_initializer,
+                                        name='bias',
+                                        regularizer=self.bias_regularizer,
+                                        constraint=self.bias_constraint)
+        else:
+            self.bias = None
+        # Set input spec.
+        self.input_spec = InputSpec(ndim=4, axes={channel_axis: input_dim})
+        self.built = True
+
+    def call(self, inputs, training=None):
+        outputs = K.depthwise_conv2d(
+            inputs,
+            self.depthwise_kernel,
+            strides=self.strides,
+            padding=self.padding,
+            dilation_rate=self.dilation_rate,
+            data_format=self.data_format)
+
+        if self.bias:
+            outputs = K.bias_add(
+                outputs,
+                self.bias,
+                data_format=self.data_format)
+
+        if self.activation is not None:
+            return self.activation(outputs)
+
+        return outputs
+
+    def compute_output_shape(self, input_shape):
+        if self.data_format == 'channels_first':
+            rows = input_shape[2]
+            cols = input_shape[3]
+            out_filters = input_shape[1] * self.depth_multiplier
+        elif self.data_format == 'channels_last':
+            rows = input_shape[1]
+            cols = input_shape[2]
+            out_filters = input_shape[3] * self.depth_multiplier
+
+        rows = conv_utils.conv_output_length(rows, self.kernel_size[0],
+                                             self.padding,
+                                             self.strides[0])
+        cols = conv_utils.conv_output_length(cols, self.kernel_size[1],
+                                             self.padding,
+                                             self.strides[1])
+
+        if self.data_format == 'channels_first':
+            return (input_shape[0], out_filters, rows, cols)
+        elif self.data_format == 'channels_last':
+            return (input_shape[0], rows, cols, out_filters)
+
+    def get_config(self):
+        config = super(DepthwiseConv2D, self).get_config()
+        config.pop('filters')
+        config.pop('kernel_initializer')
+        config.pop('kernel_regularizer')
+        config.pop('kernel_constraint')
+        config['depth_multiplier'] = self.depth_multiplier
+        config['depthwise_initializer'] = initializers.serialize(self.depthwise_initializer)
+        config['depthwise_regularizer'] = regularizers.serialize(self.depthwise_regularizer)
+        config['depthwise_constraint'] = constraints.serialize(self.depthwise_constraint)
+        return config
+
+
+def MobileNet(input_shape=None,
+              alpha=1.0,
+              depth_multiplier=1,
+              dropout=1e-3,
+              include_top=True,
+              weights='imagenet',
+              input_tensor=None,
+              pooling=None,
+              classes=1000):
+    """Instantiates the MobileNet architecture.
+
+    Note that only TensorFlow is supported for now,
+    therefore it only works with the data format
+    `image_data_format='channels_last'` in your Keras config
+    at `~/.keras/keras.json`.
+
+    To load a MobileNet model via `load_model`, import the custom
+    objects `relu6` and `DepthwiseConv2D` and pass them to the
+    `custom_objects` parameter.
+    E.g.
+    model = load_model('mobilenet.h5', custom_objects={
+                       'relu6': mobilenet.relu6,
+                       'DepthwiseConv2D': mobilenet.DepthwiseConv2D})
+
+    # Arguments
+        input_shape: optional shape tuple, only to be specified
+            if `include_top` is False (otherwise the input shape
+            has to be `(224, 224, 3)` (with `channels_last` data format)
+            or (3, 224, 224) (with `channels_first` data format).
+            It should have exactly 3 inputs channels,
+            and width and height should be no smaller than 32.
+            E.g. `(200, 200, 3)` would be one valid value.
+        alpha: controls the width of the network.
+            - If `alpha` < 1.0, proportionally decreases the number
+                of filters in each layer.
+            - If `alpha` > 1.0, proportionally increases the number
+                of filters in each layer.
+            - If `alpha` = 1, default number of filters from the paper
+                 are used at each layer.
+        depth_multiplier: depth multiplier for depthwise convolution
+            (also called the resolution multiplier)
+        dropout: dropout rate
+        include_top: whether to include the fully-connected
+            layer at the top of the network.
+        weights: `None` (random initialization) or
+            `imagenet` (ImageNet weights)
+        input_tensor: optional Keras tensor (i.e. output of
+            `layers.Input()`)
+            to use as image input for the model.
+        pooling: Optional pooling mode for feature extraction
+            when `include_top` is `False`.
+            - `None` means that the output of the model
+                will be the 4D tensor output of the
+                last convolutional layer.
+            - `avg` means that global average pooling
+                will be applied to the output of the
+                last convolutional layer, and thus
+                the output of the model will be a
+                2D tensor.
+            - `max` means that global max pooling will
+                be applied.
+        classes: optional number of classes to classify images
+            into, only to be specified if `include_top` is True, and
+            if no `weights` argument is specified.
+
+    # Returns
+        A Keras model instance.
+
+    # Raises
+        ValueError: in case of invalid argument for `weights`,
+            or invalid input shape.
+        RuntimeError: If attempting to run this model with a
+            backend that does not support separable convolutions.
+    """
+
+    if K.backend() != 'tensorflow':
+        raise RuntimeError('Only Tensorflow backend is currently supported, '
+                           'as other backends do not support '
+                           'depthwise convolution.')
+
+    if weights not in {'imagenet', None}:
+        raise ValueError('The `weights` argument should be either '
+                         '`None` (random initialization) or `imagenet` '
+                         '(pre-training on ImageNet).')
+
+    if weights == 'imagenet' and include_top and classes != 1000:
+        raise ValueError('If using `weights` as ImageNet with `include_top` '
+                         'as true, `classes` should be 1000')
+
+    # Determine proper input shape.
+    input_shape = _obtain_input_shape(input_shape,
+                                      default_size=224,
+                                      min_size=32,
+                                      data_format=K.image_data_format(),
+                                      include_top=include_top or weights)
+    if K.image_data_format() == 'channels_last':
+        row_axis, col_axis = (0, 1)
+    else:
+        row_axis, col_axis = (1, 2)
+    rows = input_shape[row_axis]
+    cols = input_shape[col_axis]
+
+    if weights == 'imagenet':
+        if depth_multiplier != 1:
+            raise ValueError('If imagenet weights are being loaded, '
+                             'depth multiplier must be 1')
+
+        if alpha not in [0.25, 0.50, 0.75, 1.0]:
+            raise ValueError('If imagenet weights are being loaded, '
+                             'alpha can be one of'
+                             '`0.25`, `0.50`, `0.75` or `1.0` only.')
+
+        if rows != cols or rows not in [128, 160, 192, 224]:
+            raise ValueError('If imagenet weights are being loaded, '
+                             'input must have a static square shape (one of '
+                             '(128,128), (160,160), (192,192), or (224, 224)).'
+                             ' Input shape provided = %s' % (input_shape,))
+
+    if K.image_data_format() != 'channels_last':
+        warnings.warn('The MobileNet family of models is only available '
+                      'for the input data format "channels_last" '
+                      '(width, height, channels). '
+                      'However your settings specify the default '
+                      'data format "channels_first" (channels, width, height).'
+                      ' You should set `image_data_format="channels_last"` '
+                      'in your Keras config located at ~/.keras/keras.json. '
+                      'The model being returned right now will expect inputs '
+                      'to follow the "channels_last" data format.')
+        K.set_image_data_format('channels_last')
+        old_data_format = 'channels_first'
+    else:
+        old_data_format = None
+
+    if input_tensor is None:
+        img_input = Input(shape=input_shape)
+    else:
+        if not K.is_keras_tensor(input_tensor):
+            img_input = Input(tensor=input_tensor, shape=input_shape)
+        else:
+            img_input = input_tensor
+
+    x = _conv_block(img_input, 32, alpha, strides=(2, 2))
+    x = _depthwise_conv_block(x, 64, alpha, depth_multiplier, block_id=1)
+
+    x = _depthwise_conv_block(x, 128, alpha, depth_multiplier,
+                              strides=(2, 2), block_id=2)
+    x = _depthwise_conv_block(x, 128, alpha, depth_multiplier, block_id=3)
+
+    x = _depthwise_conv_block(x, 256, alpha, depth_multiplier,
+                              strides=(2, 2), block_id=4)
+    x = _depthwise_conv_block(x, 256, alpha, depth_multiplier, block_id=5)
+
+    x = _depthwise_conv_block(x, 512, alpha, depth_multiplier,
+                              strides=(2, 2), block_id=6)
+    x = _depthwise_conv_block(x, 512, alpha, depth_multiplier, block_id=7)
+    x = _depthwise_conv_block(x, 512, alpha, depth_multiplier, block_id=8)
+    x = _depthwise_conv_block(x, 512, alpha, depth_multiplier, block_id=9)
+    x = _depthwise_conv_block(x, 512, alpha, depth_multiplier, block_id=10)
+    x = _depthwise_conv_block(x, 512, alpha, depth_multiplier, block_id=11)
+
+    x = _depthwise_conv_block(x, 1024, alpha, depth_multiplier,
+                              strides=(2, 2), block_id=12)
+    x = _depthwise_conv_block(x, 1024, alpha, depth_multiplier, block_id=13)
+
+    if include_top:
+        if K.image_data_format() == 'channels_first':
+            shape = (int(1024 * alpha), 1, 1)
+        else:
+            shape = (1, 1, int(1024 * alpha))
+
+        x = GlobalAveragePooling2D()(x)
+        x = Reshape(shape, name='reshape_1')(x)
+        x = Dropout(dropout, name='dropout')(x)
+        x = Conv2D(classes, (1, 1),
+                   padding='same', name='conv_preds')(x)
+        x = Activation('softmax', name='act_softmax')(x)
+        x = Reshape((classes,), name='reshape_2')(x)
+    else:
+        if pooling == 'avg':
+            x = GlobalAveragePooling2D()(x)
+        elif pooling == 'max':
+            x = GlobalMaxPooling2D()(x)
+
+    # Ensure that the model takes into account
+    # any potential predecessors of `input_tensor`.
+    if input_tensor is not None:
+        inputs = get_source_inputs(input_tensor)
+    else:
+        inputs = img_input
+
+    # Create model.
+    model = Model(inputs, x, name='mobilenet_%0.2f_%s' % (alpha, rows))
+
+    # load weights
+    if weights == 'imagenet':
+        if K.image_data_format() == 'channels_first':
+            raise ValueError('Weights for "channels_last" format '
+                             'are not available.')
+        if alpha == 1.0:
+            alpha_text = '1_0'
+        elif alpha == 0.75:
+            alpha_text = '7_5'
+        elif alpha == 0.50:
+            alpha_text = '5_0'
+        else:
+            alpha_text = '2_5'
+
+        if include_top:
+            model_name = 'mobilenet_%s_%d_tf.h5' % (alpha_text, rows)
+            weigh_path = BASE_WEIGHT_PATH + model_name
+            weights_path = get_file(model_name,
+                                    weigh_path,
+                                    cache_subdir='models')
+        else:
+            model_name = 'mobilenet_%s_%d_tf_no_top.h5' % (alpha_text, rows)
+            weigh_path = BASE_WEIGHT_PATH + model_name
+            weights_path = get_file(model_name,
+                                    weigh_path,
+                                    cache_subdir='models')
+        model.load_weights(weights_path)
+
+    if old_data_format:
+        K.set_image_data_format(old_data_format)
+    return model
+
+
+def _conv_block(inputs, filters, alpha, kernel=(3, 3), strides=(1, 1)):
+    """Adds an initial convolution layer (with batch normalization and relu6).
+
+    # Arguments
+        inputs: Input tensor of shape `(rows, cols, 3)`
+            (with `channels_last` data format) or
+            (3, rows, cols) (with `channels_first` data format).
+            It should have exactly 3 inputs channels,
+            and width and height should be no smaller than 32.
+            E.g. `(224, 224, 3)` would be one valid value.
+        filters: Integer, the dimensionality of the output space
+            (i.e. the number output of filters in the convolution).
+        alpha: controls the width of the network.
+            - If `alpha` < 1.0, proportionally decreases the number
+                of filters in each layer.
+            - If `alpha` > 1.0, proportionally increases the number
+                of filters in each layer.
+            - If `alpha` = 1, default number of filters from the paper
+                 are used at each layer.
+        kernel: An integer or tuple/list of 2 integers, specifying the
+            width and height of the 2D convolution window.
+            Can be a single integer to specify the same value for
+            all spatial dimensions.
+        strides: An integer or tuple/list of 2 integers,
+            specifying the strides of the convolution along the width and height.
+            Can be a single integer to specify the same value for
+            all spatial dimensions.
+            Specifying any stride value != 1 is incompatible with specifying
+            any `dilation_rate` value != 1.
+
+    # Input shape
+        4D tensor with shape:
+        `(samples, channels, rows, cols)` if data_format='channels_first'
+        or 4D tensor with shape:
+        `(samples, rows, cols, channels)` if data_format='channels_last'.
+
+    # Output shape
+        4D tensor with shape:
+        `(samples, filters, new_rows, new_cols)` if data_format='channels_first'
+        or 4D tensor with shape:
+        `(samples, new_rows, new_cols, filters)` if data_format='channels_last'.
+        `rows` and `cols` values might have changed due to stride.
+
+    # Returns
+        Output tensor of block.
+    """
+    channel_axis = 1 if K.image_data_format() == 'channels_first' else -1
+    filters = int(filters * alpha)
+    x = Conv2D(filters, kernel,
+               padding='same',
+               use_bias=False,
+               strides=strides,
+               name='conv1')(inputs)
+    x = BatchNormalization(axis=channel_axis, name='conv1_bn')(x)
+    return Activation(relu6, name='conv1_relu')(x)
+
+
+def _depthwise_conv_block(inputs, pointwise_conv_filters, alpha,
+                          depth_multiplier=1, strides=(1, 1), block_id=1):
+    """Adds a depthwise convolution block.
+
+    A depthwise convolution block consists of a depthwise conv,
+    batch normalization, relu6, pointwise convolution,
+    batch normalization and relu6 activation.
+
+    # Arguments
+        inputs: Input tensor of shape `(rows, cols, channels)`
+            (with `channels_last` data format) or
+            (channels, rows, cols) (with `channels_first` data format).
+        pointwise_conv_filters: Integer, the dimensionality of the output space
+            (i.e. the number output of filters in the pointwise convolution).
+        alpha: controls the width of the network.
+            - If `alpha` < 1.0, proportionally decreases the number
+                of filters in each layer.
+            - If `alpha` > 1.0, proportionally increases the number
+                of filters in each layer.
+            - If `alpha` = 1, default number of filters from the paper
+                 are used at each layer.
+        depth_multiplier: The number of depthwise convolution output channels
+            for each input channel.
+            The total number of depthwise convolution output
+            channels will be equal to `filters_in * depth_multiplier`.
+        strides: An integer or tuple/list of 2 integers,
+            specifying the strides of the convolution along the width and height.
+            Can be a single integer to specify the same value for
+            all spatial dimensions.
+            Specifying any stride value != 1 is incompatible with specifying
+            any `dilation_rate` value != 1.
+        block_id: Integer, a unique identification designating the block number.
+
+    # Input shape
+        4D tensor with shape:
+        `(batch, channels, rows, cols)` if data_format='channels_first'
+        or 4D tensor with shape:
+        `(batch, rows, cols, channels)` if data_format='channels_last'.
+
+    # Output shape
+        4D tensor with shape:
+        `(batch, filters, new_rows, new_cols)` if data_format='channels_first'
+        or 4D tensor with shape:
+        `(batch, new_rows, new_cols, filters)` if data_format='channels_last'.
+        `rows` and `cols` values might have changed due to stride.
+
+    # Returns
+        Output tensor of block.
+    """
+    channel_axis = 1 if K.image_data_format() == 'channels_first' else -1
+    pointwise_conv_filters = int(pointwise_conv_filters * alpha)
+
+    x = DepthwiseConv2D((3, 3),
+                        padding='same',
+                        depth_multiplier=depth_multiplier,
+                        strides=strides,
+                        use_bias=False,
+                        name='conv_dw_%d' % block_id)(inputs)
+    x = BatchNormalization(axis=channel_axis, name='conv_dw_%d_bn' % block_id)(x)
+    x = Activation(relu6, name='conv_dw_%d_relu' % block_id)(x)
+
+    x = Conv2D(pointwise_conv_filters, (1, 1),
+               padding='same',
+               use_bias=False,
+               strides=(1, 1),
+               name='conv_pw_%d' % block_id)(x)
+    x = BatchNormalization(axis=channel_axis, name='conv_pw_%d_bn' % block_id)(x)
+    return Activation(relu6, name='conv_pw_%d_relu' % block_id)(x)

keras/applications/resnet50.py

@@ -77,7 +77,7 @@ def identity_block(input_tensor, kernel_size, filters, stage, block):
 
 
 def conv_block(input_tensor, kernel_size, filters, stage, block, strides=(2, 2)):
-    """conv_block is the block that has a conv layer at shortcut
+    """A block that has a conv layer at shortcut.
 
     # Arguments
         input_tensor: input tensor

keras/backend/cntk_backend.py

@@ -1,7 +1,7 @@
 from __future__ import print_function
 import cntk as C
 import numpy as np
-from .common import _FLOATX, _EPSILON, image_dim_ordering, image_data_format
+from .common import floatx, epsilon, image_dim_ordering, image_data_format
 from collections import defaultdict
 from contextlib import contextmanager
 import warnings
@@ -130,7 +130,10 @@ def _convert_dtype_string(dtype):
                          'float64.' % dtype)
 
 
-def variable(value, dtype=_FLOATX, name=None):
+def variable(value, dtype=None, name=None):
+    if dtype is None:
+        dtype = floatx()
+
     if name is None:
         name = ''
 
@@ -227,10 +230,12 @@ def eval(x):
 def placeholder(
         shape=None,
         ndim=None,
-        dtype=_FLOATX,
+        dtype=None,
         sparse=False,
         name=None,
         dynamic_axis_num=1):
+    if dtype is None:
+        dtype = floatx()
     if not shape:
         if ndim:
             shape = tuple([None for _ in range(ndim)])
@@ -304,7 +309,7 @@ def _prepare_name(name, default):
 
 def constant(value, dtype=None, shape=None, name=None):
     if dtype is None:
-        dtype = _FLOATX
+        dtype = floatx()
     if shape is None:
         shape = ()
     np_value = value * np.ones(shape)
@@ -351,8 +356,10 @@ def random_uniform(shape, minval=0.0, maxval=1.0, dtype=None, seed=None):
     return random_uniform_variable(shape, minval, maxval, dtype, seed)
 
 
-def random_uniform_variable(shape, low, high, dtype=_FLOATX,
-                            name=None, seed=None):
+def random_uniform_variable(shape, low, high,
+                            dtype=None, name=None, seed=None):
+    if dtype is None:
+        dtype = floatx()
     if seed is None:
         # ensure that randomness is conditioned by the Numpy RNG
         seed = np.random.randint(10e3)
@@ -380,9 +387,11 @@ def random_normal_variable(
         shape,
         mean,
         scale,
-        dtype=_FLOATX,
+        dtype=None,
         name=None,
         seed=None):
+    if dtype is None:
+        dtype = floatx()
     if seed is None:
         # ensure that randomness is conditioned by the Numpy RNG
         seed = np.random.randint(10e7)
@@ -403,7 +412,9 @@ def random_normal_variable(
         name=name)
 
 
-def random_normal(shape, mean=0.0, stddev=1.0, dtype=_FLOATX, seed=None):
+def random_normal(shape, mean=0.0, stddev=1.0, dtype=None, seed=None):
+    if dtype is None:
+        dtype = floatx()
     for _ in shape:
         if _ is None:
             raise ValueError('CNTK Backend: randomness op with '
@@ -435,17 +446,23 @@ def dtype(x):
     return _convert_dtype_string(x.dtype)
 
 
-def zeros(shape, dtype=_FLOATX, name=None):
+def zeros(shape, dtype=None, name=None):
+    if dtype is None:
+        dtype = floatx()
     ctype = _convert_string_dtype(dtype)
     return variable(value=np.zeros(shape, ctype), dtype=dtype, name=name)
 
 
-def ones(shape, dtype=_FLOATX, name=None):
+def ones(shape, dtype=None, name=None):
+    if dtype is None:
+        dtype = floatx()
     ctype = _convert_string_dtype(dtype)
     return variable(value=np.ones(shape, ctype), dtype=dtype, name=name)
 
 
-def eye(size, dtype=_FLOATX, name=None):
+def eye(size, dtype=None, name=None):
+    if dtype is None:
+        dtype = floatx()
     return variable(np.eye(size), dtype, name)
 
 
@@ -652,16 +669,16 @@ def _normalize_axis(axis, x):
 
     nones = _get_dynamic_axis_num(x)
 
-    if type(axis) is tuple:
+    if isinstance(axis, tuple):
         _axis = list(axis)
-    elif type(axis) is int:
+    elif isinstance(axis, int):
         _axis = [axis]
-    elif type(axis) is list:
+    elif isinstance(axis, list):
         _axis = list(axis)
     else:
         _axis = axis
 
-    if type(_axis) is list:
+    if isinstance(_axis, list):
         for i, a in enumerate(_axis):
             if a is not None and a < 0:
                 _axis[i] = (a % ndim)
@@ -729,7 +746,7 @@ def all(x, axis=None, keepdims=False):
         return all_matrix
 
 
-def classification_error(output, target, axis=-1):
+def classification_error(target, output, axis=-1):
     return C.ops.reduce_mean(
         C.equal(
             argmax(
@@ -801,10 +818,10 @@ def clip(x, min_value, max_value):
     return C.clip(x, min_value, max_value)
 
 
-def binary_crossentropy(output, target, from_logits=False):
+def binary_crossentropy(target, output, from_logits=False):
     if from_logits:
         output = C.sigmoid(output)
-    output = C.clip(output, _EPSILON, 1.0 - _EPSILON)
+    output = C.clip(output, epsilon(), 1.0 - epsilon())
     output = -target * C.log(output) - (1.0 - target) * C.log(1.0 - output)
     return output
 
@@ -1352,6 +1369,16 @@ def conv2d(x, kernel, strides=(1, 1), padding='valid',
     return _postprocess_conv2d_output(x, data_format)
 
 
+def separable_conv2d(x, depthwise_kernel, pointwise_kernel, strides=(1, 1),
+                     padding='valid', data_format=None, dilation_rate=(1, 1)):
+    raise NotImplementedError
+
+
+def depthwise_conv2d(x, depthwise_kernel, strides=(1, 1), padding='valid',
+                     data_format=None, dilation_rate=(1, 1)):
+    raise NotImplementedError
+
+
 def conv3d(x, kernel, strides=(1, 1, 1), padding='valid',
            data_format=None, dilation_rate=(1, 1, 1)):
     if data_format is None:
@@ -1376,6 +1403,41 @@ def conv3d(x, kernel, strides=(1, 1, 1), padding='valid',
     return _postprocess_conv3d_output(x, data_format)
 
 
+def conv3d_transpose(x, kernel, output_shape, strides=(1, 1, 1),
+                     padding='valid', data_format=None):
+    if data_format is None:
+        data_format = image_data_format()
+    if data_format not in {'channels_first', 'channels_last'}:
+        raise ValueError('Unknown data_format ' + str(data_format))
+
+    x = _preprocess_conv3d_input(x, data_format)
+    kernel = _preprocess_conv3d_kernel(kernel, data_format)
+    padding = _preprocess_border_mode(padding)
+    strides = (1,) + strides
+    # cntk output_shape does not include batch axis
+    output_shape = output_shape[1:]
+    # in keras2, need handle output shape in different format
+    if data_format == 'channels_last':
+        shape = list(output_shape)
+        shape[0] = output_shape[3]
+        shape[1] = output_shape[0]
+        shape[2] = output_shape[1]
+        shape[3] = output_shape[2]
+        output_shape = tuple(shape)
+
+    x = C.convolution_transpose(
+        kernel,
+        x,
+        strides,
+        auto_padding=[
+            False,
+            padding,
+            padding,
+            padding],
+        output_shape=output_shape)
+    return _postprocess_conv3d_output(x, data_format)
+
+
 def pool2d(x, pool_size, strides=(1, 1),
            padding='valid', data_format=None,
            pool_mode='max'):
@@ -1477,7 +1539,7 @@ def softsign(x):
     return x / (1 + C.abs(x))
 
 
-def categorical_crossentropy(output, target, from_logits=False):
+def categorical_crossentropy(target, output, from_logits=False):
     if from_logits:
         result = C.cross_entropy_with_softmax(output, target)
         # cntk's result shape is (batch, 1), while keras expect (batch, )
@@ -1485,18 +1547,19 @@ def categorical_crossentropy(output, target, from_logits=False):
     else:
         # scale preds so that the class probas of each sample sum to 1
         output /= C.reduce_sum(output, axis=-1)
-        # avoid numerical instability with _EPSILON clipping
-        output = C.clip(output, _EPSILON, 1.0 - _EPSILON)
+        # avoid numerical instability with epsilon clipping
+        output = C.clip(output, epsilon(), 1.0 - epsilon())
         return -sum(target * C.log(output), axis=-1)
 
 
-def sparse_categorical_crossentropy(output, target, from_logits=False):
+def sparse_categorical_crossentropy(target, output, from_logits=False):
     target = C.one_hot(target, output.shape[-1])
     target = C.reshape(target, output.shape)
     return categorical_crossentropy(output, target, from_logits)
 
 
 class Function(object):
+
     def __init__(self, inputs, outputs, updates=[], **kwargs):
         self.placeholders = inputs
         self.trainer = None
@@ -1533,11 +1596,12 @@ class Function(object):
                     p_list.append(grad_parameter_dict[g])
                     u_list.append(g)
                 else:
-                    raise ValueError('CNTK backend: when constructing trainer, '
-                                     'found gradient node `%s` which is not '
-                                     'related to any parameters in the model. '
-                                     'Please double check how the gradient node '
-                                     'is constructed.' % g)
+                    raise ValueError(
+                        'CNTK backend: when constructing trainer, '
+                        'found gradient node `%s` which is not '
+                        'related to any parameters in the model. '
+                        'Please double check how the gradient node '
+                        'is constructed.' % g)
 
             if len(u_list) > 0:
                 learner = C.cntk_py.universal_learner(p_list, u_list, update_func)
@@ -1567,6 +1631,17 @@ class Function(object):
         else:
             self.metrics_func = None
 
+    @staticmethod
+    def _is_input_shape_compatible(input, placeholder):
+        if hasattr(input, 'shape') and hasattr(placeholder, 'shape'):
+            num_dynamic = get_num_dynamic_axis(placeholder)
+            input_shape = input.shape[num_dynamic:]
+            placeholder_shape = placeholder.shape
+            for i, p in zip(input_shape, placeholder_shape):
+                if i != p and p != C.InferredDimension:
+                    return False
+        return True
+
     def __call__(self, inputs):
         assert type(inputs) in {list, tuple}
         feed_dict = {}
@@ -1576,6 +1651,15 @@ class Function(object):
                value.dtype != np.float32 and
                value.dtype != np.float64):
                 value = value.astype(np.float32)
+            # in current version cntk can't support input with variable
+            # length. Will support it in next release.
+            if not self._is_input_shape_compatible(value, tensor):
+                raise ValueError(
+                    'CNTK backend: The placeholder has been resolved '
+                    'to shape `%s`, but input shape is `%s`. Currently '
+                    'CNTK can not take variable length inputs. Please '
+                    'pass inputs that have a static shape.'
+                    % (tensor.shape, value.shape))
             feed_dict[tensor] = value
 
         updated = []
@@ -1585,9 +1669,10 @@ class Function(object):
                 if argument in feed_dict:
                     input_dict[argument] = feed_dict[argument]
                 else:
-                    raise ValueError('CNTK backend: argument %s is not found in inputs. '
-                                     'Please double check the model and inputs in '
-                                     '`train_function`.' % argument.name)
+                    raise ValueError(
+                        'CNTK backend: argument %s is not found in inputs. '
+                        'Please double check the model and inputs in '
+                        '`train_function`.' % argument.name)
 
             result = self.trainer.train_minibatch(
                 input_dict, self.trainer_output)
@@ -1603,9 +1688,10 @@ class Function(object):
                 if argument in feed_dict:
                     input_dict[argument] = feed_dict[argument]
                 else:
-                    raise ValueError('CNTK backend: metrics argument %s '
-                                     'is not found in inputs. Please double '
-                                     'check the model and inputs.' % argument.name)
+                    raise ValueError(
+                        'CNTK backend: metrics argument %s '
+                        'is not found in inputs. Please double '
+                        'check the model and inputs.' % argument.name)
             output_values = self.metrics_func.eval(input_dict, as_numpy=False)
             if isinstance(output_values, dict):
                 for o in self.metrics_outputs:
@@ -1623,9 +1709,10 @@ class Function(object):
                 if argument in feed_dict:
                     input_dict[argument] = feed_dict[argument]
                 else:
-                    raise ValueError('CNTK backend: assign ops argument %s '
-                                     'is not found in inputs. Please double '
-                                     'check the model and inputs.' % argument.name)
+                    raise ValueError(
+                        'CNTK backend: assign ops argument %s '
+                        'is not found in inputs. Please double '
+                        'check the model and inputs.' % argument.name)
             self.unrelated_updates.eval(input_dict, as_numpy=False)
         return updated
 

keras/backend/common.py

@@ -63,7 +63,7 @@ def set_floatx(floatx):
     """Sets the default float type.
 
     # Arguments
-        String: 'float16', 'float32', or 'float64'.
+        floatx: String, 'float16', 'float32', or 'float64'.
 
     # Example
     ```python

keras/backend/tensorflow_backend.py

@@ -7,13 +7,14 @@ from tensorflow.python.ops import ctc_ops as ctc
 from tensorflow.python.ops import variables as tf_variables
 
 from collections import defaultdict
-import inspect
+
 import numpy as np
 import os
 
 from .common import floatx
 from .common import _EPSILON
 from .common import image_data_format
+from ..utils.generic_utils import has_arg
 
 # Legacy functions
 from .common import set_image_dim_ordering
@@ -450,15 +451,15 @@ def shape(x):
         >>> tf_session = K.get_session()
         >>> val = np.array([[1, 2], [3, 4]])
         >>> kvar = K.variable(value=val)
-        >>> input = keras.backend.placeholder(shape=(2, 4, 5))
+        >>> inputs = keras.backend.placeholder(shape=(2, 4, 5))
         >>> K.shape(kvar)
         <tf.Tensor 'Shape_8:0' shape=(2,) dtype=int32>
-        >>> K.shape(input)
+        >>> K.shape(inputs)
         <tf.Tensor 'Shape_9:0' shape=(3,) dtype=int32>
         # To get integer shape (Instead, you can use K.int_shape(x))
         >>> K.shape(kvar).eval(session=tf_session)
         array([2, 2], dtype=int32)
-        >>> K.shape(input).eval(session=tf_session)
+        >>> K.shape(inputs).eval(session=tf_session)
         array([2, 4, 5], dtype=int32)
     ```
     """
@@ -477,8 +478,8 @@ def int_shape(x):
     # Examples
     ```python
         >>> from keras import backend as K
-        >>> input = K.placeholder(shape=(2, 4, 5))
-        >>> K.int_shape(input)
+        >>> inputs = K.placeholder(shape=(2, 4, 5))
+        >>> K.int_shape(inputs)
         (2, 4, 5)
         >>> val = np.array([[1, 2], [3, 4]])
         >>> kvar = K.variable(value=val)
@@ -507,10 +508,10 @@ def ndim(x):
     # Examples
     ```python
         >>> from keras import backend as K
-        >>> input = K.placeholder(shape=(2, 4, 5))
+        >>> inputs = K.placeholder(shape=(2, 4, 5))
         >>> val = np.array([[1, 2], [3, 4]])
         >>> kvar = K.variable(value=val)
-        >>> K.ndim(input)
+        >>> K.ndim(inputs)
         3
         >>> K.ndim(kvar)
         2
@@ -549,7 +550,7 @@ def dtype(x):
         'float32_ref'
     ```
     """
-    return x.dtype.name
+    return x.dtype.base_dtype.name
 
 
 def eval(x):
@@ -856,7 +857,7 @@ def update(x, new_x):
     """Update the value of `x` to `new_x`.
 
     # Arguments
-        x: A Variable.
+        x: A `Variable`.
         new_x: A tensor of same shape as `x`.
 
     # Returns
@@ -869,7 +870,7 @@ def update_add(x, increment):
     """Update the value of `x` by adding `increment`.
 
     # Arguments
-        x: A Variable.
+        x: A `Variable`.
         increment: A tensor of same shape as `x`.
 
     # Returns
@@ -882,7 +883,7 @@ def update_sub(x, decrement):
     """Update the value of `x` by subtracting `decrement`.
 
     # Arguments
-        x: A Variable.
+        x: A `Variable`.
         decrement: A tensor of same shape as `x`.
 
     # Returns
@@ -895,12 +896,13 @@ def moving_average_update(x, value, momentum):
     """Compute the moving average of a variable.
 
     # Arguments
-        x: A Variable.
-        value: A tensor with the same shape as `variable`.
+        x: A `Variable`.
+        value: A tensor with the same shape as `x`.
         momentum: The moving average momentum.
 
     # Returns
-        An Operation to update the variable."""
+        An operation to update the variable.
+    """
     return moving_averages.assign_moving_average(
         x, value, momentum, zero_debias=False)
 
@@ -1087,10 +1089,10 @@ def transpose(x):
     ```
 
     ```python
-        >>> input = K.placeholder((2, 3))
-        >>> input
+        >>> inputs = K.placeholder((2, 3))
+        >>> inputs
         <tf.Tensor 'Placeholder_11:0' shape=(2, 3) dtype=float32>
-        >>> input_transposed = K.transpose(input)
+        >>> input_transposed = K.transpose(inputs)
         >>> input_transposed
         <tf.Tensor 'transpose_4:0' shape=(3, 2) dtype=float32>
 
@@ -1151,7 +1153,7 @@ def max(x, axis=None, keepdims=False):
         A tensor with maximum values of `x`.
     """
     axis = _normalize_axis(axis, ndim(x))
-    return tf.reduce_max(x, reduction_indices=axis, keep_dims=keepdims)
+    return tf.reduce_max(x, axis=axis, keep_dims=keepdims)
 
 
 def min(x, axis=None, keepdims=False):
@@ -1169,7 +1171,7 @@ def min(x, axis=None, keepdims=False):
         A tensor with miminum values of `x`.
     """
     axis = _normalize_axis(axis, ndim(x))
-    return tf.reduce_min(x, reduction_indices=axis, keep_dims=keepdims)
+    return tf.reduce_min(x, axis=axis, keep_dims=keepdims)
 
 
 def sum(x, axis=None, keepdims=False):
@@ -1187,7 +1189,7 @@ def sum(x, axis=None, keepdims=False):
         A tensor with sum of `x`.
     """
     axis = _normalize_axis(axis, ndim(x))
-    return tf.reduce_sum(x, reduction_indices=axis, keep_dims=keepdims)
+    return tf.reduce_sum(x, axis=axis, keep_dims=keepdims)
 
 
 def prod(x, axis=None, keepdims=False):
@@ -1205,7 +1207,7 @@ def prod(x, axis=None, keepdims=False):
         A tensor with the product of elements of `x`.
     """
     axis = _normalize_axis(axis, ndim(x))
-    return tf.reduce_prod(x, reduction_indices=axis, keep_dims=keepdims)
+    return tf.reduce_prod(x, axis=axis, keep_dims=keepdims)
 
 
 def cumsum(x, axis=0):
@@ -1253,10 +1255,10 @@ def var(x, axis=None, keepdims=False):
     axis = _normalize_axis(axis, ndim(x))
     if x.dtype.base_dtype == tf.bool:
         x = tf.cast(x, floatx())
-    m = tf.reduce_mean(x, reduction_indices=axis, keep_dims=True)
+    m = tf.reduce_mean(x, axis=axis, keep_dims=True)
     devs_squared = tf.square(x - m)
     return tf.reduce_mean(devs_squared,
-                          reduction_indices=axis,
+                          axis=axis,
                           keep_dims=keepdims)
 
 
@@ -1294,7 +1296,7 @@ def mean(x, axis=None, keepdims=False):
     axis = _normalize_axis(axis, ndim(x))
     if x.dtype.base_dtype == tf.bool:
         x = tf.cast(x, floatx())
-    return tf.reduce_mean(x, reduction_indices=axis, keep_dims=keepdims)
+    return tf.reduce_mean(x, axis=axis, keep_dims=keepdims)
 
 
 def any(x, axis=None, keepdims=False):
@@ -1310,7 +1312,7 @@ def any(x, axis=None, keepdims=False):
     """
     axis = _normalize_axis(axis, ndim(x))
     x = tf.cast(x, tf.bool)
-    return tf.reduce_any(x, reduction_indices=axis, keep_dims=keepdims)
+    return tf.reduce_any(x, axis=axis, keep_dims=keepdims)
 
 
 def all(x, axis=None, keepdims=False):
@@ -1326,7 +1328,7 @@ def all(x, axis=None, keepdims=False):
     """
     axis = _normalize_axis(axis, ndim(x))
     x = tf.cast(x, tf.bool)
-    return tf.reduce_all(x, reduction_indices=axis, keep_dims=keepdims)
+    return tf.reduce_all(x, axis=axis, keep_dims=keepdims)
 
 
 def argmax(x, axis=-1):
@@ -2088,9 +2090,6 @@ def one_hot(indices, num_classes):
     # Returns
         (n + 1)D one hot representation of the input
         with shape `(batch_size, dim1, dim2, ... dim(n-1), num_classes)`
-
-    # Returns
-        The one-hot tensor.
     """
     return tf.one_hot(indices, depth=num_classes, axis=-1)
 
@@ -2287,8 +2286,7 @@ def function(inputs, outputs, updates=None, **kwargs):
     """
     if kwargs:
         for key in kwargs:
-            if (key not in inspect.getargspec(tf.Session.run)[0] and
-                    key not in inspect.getargspec(Function.__init__)[0]):
+            if not (has_arg(tf.Session.run, key, True) or has_arg(Function.__init__, key, True)):
                 msg = 'Invalid argument "%s" passed to K.function with Tensorflow backend' % key
                 raise ValueError(msg)
     return Function(inputs, outputs, updates=updates, **kwargs)
@@ -2329,12 +2327,12 @@ def rnn(step_function, inputs, initial_states,
     # Arguments
         step_function: RNN step function.
             Parameters:
-                input: tensor with shape `(samples, ...)` (no time dimension),
+                inputs: tensor with shape `(samples, ...)` (no time dimension),
                     representing input for the batch of samples at a certain
                     time step.
                 states: list of tensors.
             Returns:
-                output: tensor with shape `(samples, output_dim)`
+                outputs: tensor with shape `(samples, output_dim)`
                     (no time dimension).
                 new_states: list of tensors, same length and shapes
                     as 'states'. The first state in the list must be the
@@ -2724,14 +2722,14 @@ def softsign(x):
     return tf.nn.softsign(x)
 
 
-def categorical_crossentropy(output, target, from_logits=False):
+def categorical_crossentropy(target, output, from_logits=False):
     """Categorical crossentropy between an output tensor and a target tensor.
 
     # Arguments
+        target: A tensor of the same shape as `output`.
         output: A tensor resulting from a softmax
             (unless `from_logits` is True, in which
             case `output` is expected to be the logits).
-        target: A tensor of the same shape as `output`.
         from_logits: Boolean, whether `output` is the
             result of a softmax, or is a tensor of logits.
 
@@ -2743,26 +2741,26 @@ def categorical_crossentropy(output, target, from_logits=False):
     if not from_logits:
         # scale preds so that the class probas of each sample sum to 1
         output /= tf.reduce_sum(output,
-                                reduction_indices=len(output.get_shape()) - 1,
+                                axis=len(output.get_shape()) - 1,
                                 keep_dims=True)
         # manual computation of crossentropy
         epsilon = _to_tensor(_EPSILON, output.dtype.base_dtype)
         output = tf.clip_by_value(output, epsilon, 1. - epsilon)
         return - tf.reduce_sum(target * tf.log(output),
-                               reduction_indices=len(output.get_shape()) - 1)
+                               axis=len(output.get_shape()) - 1)
     else:
         return tf.nn.softmax_cross_entropy_with_logits(labels=target,
                                                        logits=output)
 
 
-def sparse_categorical_crossentropy(output, target, from_logits=False):
+def sparse_categorical_crossentropy(target, output, from_logits=False):
     """Categorical crossentropy with integer targets.
 
     # Arguments
+        target: An integer tensor.
         output: A tensor resulting from a softmax
             (unless `from_logits` is True, in which
             case `output` is expected to be the logits).
-        target: An integer tensor.
         from_logits: Boolean, whether `output` is the
             result of a softmax, or is a tensor of logits.
 
@@ -2789,12 +2787,12 @@ def sparse_categorical_crossentropy(output, target, from_logits=False):
         return res
 
 
-def binary_crossentropy(output, target, from_logits=False):
+def binary_crossentropy(target, output, from_logits=False):
     """Binary crossentropy between an output tensor and a target tensor.
 
     # Arguments
-        output: A tensor.
         target: A tensor with the same shape as `output`.
+        output: A tensor.
         from_logits: Whether `output` is expected to be a logits tensor.
             By default, we consider that `output`
             encodes a probability distribution.
@@ -2913,6 +2911,26 @@ def in_top_k(predictions, targets, k):
 
 # CONVOLUTIONS
 
+def _preprocess_deconv3d_output_shape(x, shape, data_format):
+    """Get the output_shape for the 3D deconvolution.
+
+    # Arguments
+        x: input tensor.
+        shape: output shape.
+        data_format: string, `"channels_last"` or `"channels_first"`.
+
+    # Returns
+        The output shape.
+    """
+    if data_format == 'channels_first':
+        shape = (shape[0], shape[2], shape[3], shape[4], shape[1])
+
+    if shape[0] is None:
+        shape = (tf.shape(x)[0], ) + tuple(shape[1:])
+        shape = tf.stack(list(shape))
+    return shape
+
+
 def _preprocess_deconv_output_shape(x, shape, data_format):
     """Get the output_shape for the deconvolution.
 
@@ -3213,6 +3231,41 @@ def separable_conv2d(x, depthwise_kernel, pointwise_kernel, strides=(1, 1),
     return _postprocess_conv2d_output(x, data_format)
 
 
+def depthwise_conv2d(x, depthwise_kernel, strides=(1, 1), padding='valid',
+                     data_format=None, dilation_rate=(1, 1)):
+    """2D convolution with separable filters.
+
+    # Arguments
+        x: input tensor
+        depthwise_kernel: convolution kernel for the depthwise convolution.
+        strides: strides tuple (length 2).
+        padding: string, `"same"` or `"valid"`.
+        data_format: string, `"channels_last"` or `"channels_first"`.
+        dilation_rate: tuple of integers,
+            dilation rates for the separable convolution.
+
+    # Returns
+        Output tensor.
+
+    # Raises
+        ValueError: if `data_format` is neither `channels_last` or `channels_first`.
+    """
+    if data_format is None:
+        data_format = image_data_format()
+    if data_format not in {'channels_first', 'channels_last'}:
+        raise ValueError('Unknown data_format ' + str(data_format))
+
+    x = _preprocess_conv2d_input(x, data_format)
+    padding = _preprocess_padding(padding)
+    strides = (1,) + strides + (1,)
+
+    x = tf.nn.depthwise_conv2d(x, depthwise_kernel,
+                               strides=strides,
+                               padding=padding,
+                               rate=dilation_rate)
+    return _postprocess_conv2d_output(x, data_format)
+
+
 def conv3d(x, kernel, strides=(1, 1, 1), padding='valid',
            data_format=None, dilation_rate=(1, 1, 1)):
     """3D convolution.
@@ -3253,6 +3306,43 @@ def conv3d(x, kernel, strides=(1, 1, 1), padding='valid',
     return _postprocess_conv3d_output(x, data_format)
 
 
+def conv3d_transpose(x, kernel, output_shape, strides=(1, 1, 1),
+                     padding='valid', data_format=None):
+    """3D deconvolution (i.e. transposed convolution).
+
+    # Arguments
+        x: input tensor.
+        kernel: kernel tensor.
+        output_shape: 1D int tensor for the output shape.
+        strides: strides tuple.
+        padding: string, "same" or "valid".
+        data_format: string, `"channels_last"` or `"channels_first"`.
+            Whether to use Theano or TensorFlow data format
+            for inputs/kernels/outputs.
+
+    # Returns
+        A tensor, result of transposed 3D convolution.
+
+    # Raises
+        ValueError: if `data_format` is neither `channels_last` or `channels_first`.
+    """
+    if data_format is None:
+        data_format = image_data_format()
+    if data_format not in {'channels_first', 'channels_last'}:
+        raise ValueError('Unknown data_format ' + str(data_format))
+    if isinstance(output_shape, (tuple, list)):
+        output_shape = tf.stack(output_shape)
+
+    x = _preprocess_conv3d_input(x, data_format)
+    output_shape = _preprocess_deconv3d_output_shape(x, output_shape, data_format)
+    padding = _preprocess_padding(padding)
+    strides = (1,) + strides + (1,)
+
+    x = tf.nn.conv3d_transpose(x, kernel, output_shape, strides,
+                               padding=padding)
+    return _postprocess_conv3d_output(x, data_format)
+
+
 def pool2d(x, pool_size, strides=(1, 1),
            padding='valid', data_format=None,
            pool_mode='max'):
@@ -3655,7 +3745,7 @@ def foldr(fn, elems, initializer=None, name=None):
         name: A string name for the foldr node in the graph
 
     # Returns
-        Same type and shape as initializer
+        Tensor with same type and shape as `initializer`.
     """
     return tf.foldr(fn, elems, initializer=initializer, name=name)
 

keras/backend/theano_backend.py

@@ -1,7 +1,6 @@
 from collections import defaultdict
 from contextlib import contextmanager
 import theano
-from theano import ifelse
 from theano import tensor as T
 from theano.sandbox.rng_mrg import MRG_RandomStreams as RandomStreams
 from theano.tensor.signal import pool
@@ -14,9 +13,10 @@ try:
     from theano.tensor.nnet.nnet import softsign as T_softsign
 except ImportError:
     from theano.sandbox.softsign import softsign as T_softsign
-import inspect
+
 import numpy as np
 from .common import _FLOATX, floatx, _EPSILON, image_data_format
+from ..utils.generic_utils import has_arg
 # Legacy functions
 from .common import set_image_dim_ordering, image_dim_ordering
 
@@ -445,10 +445,14 @@ def transpose(x):
 
 
 def gather(reference, indices):
-    """reference: a tensor.
-    indices: an int tensor of indices.
+    """Retrieves the elements of indices `indices` in the tensor `reference`.
 
-    Return: a tensor of same type as reference.
+    # Arguments
+        reference: A tensor.
+        indices: An integer tensor of indices.
+
+    # Returns
+        A tensor of same type as `reference`.
     """
     y = reference[indices]
     if hasattr(reference, '_keras_shape') and hasattr(indices, '_keras_shape'):
@@ -929,7 +933,7 @@ def tile(x, n):
                     output_shape += (None,)
                 else:
                     output_shape += (i * j,)
-        elif type(n) is int:
+        elif isinstance(n, int):
             output_shape = x._keras_shape[:-1]
             if x._keras_shape[-1] is None:
                 output_shape += (None,)
@@ -1051,7 +1055,7 @@ def spatial_2d_padding(x, padding=((1, 1), (1, 1)), data_format=None):
                    slice(top_pad, input_shape[2] + top_pad),
                    slice(left_pad, input_shape[3] + left_pad))
 
-    elif data_format == 'channels_last':
+    else:
         output_shape = (input_shape[0],
                         input_shape[1] + top_pad + bottom_pad,
                         input_shape[2] + left_pad + right_pad,
@@ -1061,8 +1065,6 @@ def spatial_2d_padding(x, padding=((1, 1), (1, 1)), data_format=None):
                    slice(top_pad, input_shape[1] + top_pad),
                    slice(left_pad, input_shape[2] + left_pad),
                    slice(None))
-    else:
-        raise ValueError('Invalid data_format:', data_format)
     y = T.set_subtensor(output[indices], x)
     y._keras_shape = output_shape
     return y
@@ -1091,7 +1093,7 @@ def spatial_3d_padding(x, padding=((1, 1), (1, 1), (1, 1)), data_format=None):
                    slice(padding[1][0], input_shape[3] + padding[1][0]),
                    slice(padding[2][0], input_shape[4] + padding[2][0]))
 
-    elif data_format == 'channels_last':
+    else:
         output_shape = (input_shape[0],
                         input_shape[1] + padding[0][0] + padding[0][1],
                         input_shape[2] + padding[1][0] + padding[1][1],
@@ -1103,8 +1105,6 @@ def spatial_3d_padding(x, padding=((1, 1), (1, 1), (1, 1)), data_format=None):
                    slice(padding[1][0], input_shape[2] + padding[1][0]),
                    slice(padding[2][0], input_shape[3] + padding[2][0]),
                    slice(None))
-    else:
-        raise ValueError('Invalid data_format:', data_format)
     return T.set_subtensor(output[indices], x)
 
 
@@ -1141,8 +1141,8 @@ def pattern_broadcast(x, broatcastable):
 
 def get_value(x):
     if not hasattr(x, 'get_value'):
-        raise TypeError('get_value() can only be called on a variable. '
-                        'If you have an expression instead, use eval().')
+        raise TypeError('`get_value` can only be called on a variable. '
+                        'If you have an expression instead, use `eval()`.')
     return x.get_value()
 
 
@@ -1198,9 +1198,8 @@ class Function(object):
 
 def function(inputs, outputs, updates=[], **kwargs):
     if len(kwargs) > 0:
-        function_args = inspect.getargspec(theano.function)[0]
         for key in kwargs.keys():
-            if key not in function_args:
+            if not has_arg(theano.function, key, True):
                 msg = 'Invalid argument "%s" passed to K.function with Theano backend' % key
                 raise ValueError(msg)
     return Function(inputs, outputs, updates=updates, **kwargs)
@@ -1229,12 +1228,12 @@ def rnn(step_function, inputs, initial_states,
             (at least 3D).
         step_function:
             Parameters:
-                input: tensor with shape (samples, ...) (no time dimension),
+                inputs: tensor with shape (samples, ...) (no time dimension),
                     representing input for the batch of samples at a certain
                     time step.
                 states: list of tensors.
             Returns:
-                output: tensor with shape (samples, ...) (no time dimension),
+                outputs: tensor with shape (samples, ...) (no time dimension),
                 new_states: list of tensors, same length and shapes
                     as 'states'.
         initial_states: tensor with shape (samples, ...) (no time dimension),
@@ -1315,14 +1314,14 @@ def rnn(step_function, inputs, initial_states,
             if len(initial_states) > 0:
                 initial_states[0] = T.unbroadcast(initial_states[0], 0, 1)
 
-            def _step(input, mask, output_tm1, *states):
-                output, new_states = step_function(input, states)
+            def _step(inputs, mask, output_tm1, *states):
+                outputs, new_states = step_function(inputs, states)
                 # output previous output if masked.
-                output = T.switch(mask, output, output_tm1)
+                outputs = T.switch(mask, outputs, output_tm1)
                 return_states = []
                 for state, new_state in zip(states, new_states):
                     return_states.append(T.switch(mask, new_state, state))
-                return [output] + return_states
+                return [outputs] + return_states
 
             results, _ = theano.scan(
                 _step,
@@ -1348,8 +1347,8 @@ def rnn(step_function, inputs, initial_states,
             successive_states = []
             states = initial_states
             for i in indices:
-                output, states = step_function(inputs[i], states + constants)
-                successive_outputs.append(output)
+                outputs, states = step_function(inputs[i], states + constants)
+                successive_outputs.append(outputs)
                 successive_states.append(states)
             outputs = T.stack(*successive_outputs)
             states = []
@@ -1357,9 +1356,9 @@ def rnn(step_function, inputs, initial_states,
                 states.append(T.stack(*[states_at_step[i] for states_at_step in successive_states]))
 
         else:
-            def _step(input, *states):
-                output, new_states = step_function(input, states)
-                return [output] + new_states
+            def _step(inputs, *states):
+                outputs, new_states = step_function(inputs, states)
+                return [outputs] + new_states
 
             # Theano likes to make shape==1 dimensions in the initial states (outputs_info) broadcastable
             if len(initial_states) > 0:
@@ -1390,7 +1389,18 @@ def rnn(step_function, inputs, initial_states,
 
 
 def switch(condition, then_expression, else_expression):
-    """condition: scalar tensor.
+    """Switches between two operations depending on a scalar value.
+
+    Note that both `then_expression` and `else_expression`
+    should be symbolic tensors of the *same shape*.
+
+    # Arguments
+        condition: scalar tensor (`int` or `bool`).
+        then_expression: either a tensor, or a callable that returns a tensor.
+        else_expression: either a tensor, or a callable that returns a tensor.
+
+    # Returns
+        The selected tensor.
     """
     if callable(then_expression):
         then_expression = then_expression()
@@ -1491,7 +1501,7 @@ def softsign(x):
     return T_softsign(x)
 
 
-def categorical_crossentropy(output, target, from_logits=False):
+def categorical_crossentropy(target, output, from_logits=False):
     if from_logits:
         output = T.nnet.softmax(output)
     else:
@@ -1502,14 +1512,14 @@ def categorical_crossentropy(output, target, from_logits=False):
     return T.nnet.categorical_crossentropy(output, target)
 
 
-def sparse_categorical_crossentropy(output, target, from_logits=False):
+def sparse_categorical_crossentropy(target, output, from_logits=False):
     target = T.cast(T.flatten(target), 'int32')
     target = T.extra_ops.to_one_hot(target, nb_class=output.shape[-1])
     target = reshape(target, shape(output))
     return categorical_crossentropy(output, target, from_logits)
 
 
-def binary_crossentropy(output, target, from_logits=False):
+def binary_crossentropy(target, output, from_logits=False):
     if from_logits:
         output = T.nnet.sigmoid(output)
     # avoid numerical instability with _EPSILON clipping
@@ -1908,6 +1918,11 @@ def separable_conv2d(x, depthwise_kernel, pointwise_kernel, strides=(1, 1),
     raise NotImplementedError
 
 
+def depthwise_conv2d(x, depthwise_kernel, strides=(1, 1), padding='valid',
+                     data_format=None, dilation_rate=(1, 1)):
+    raise NotImplementedError
+
+
 def conv3d(x, kernel, strides=(1, 1, 1),
            padding='valid', data_format=None,
            dilation_rate=(1, 1, 1)):
@@ -1952,6 +1967,63 @@ def conv3d(x, kernel, strides=(1, 1, 1),
     return conv_out
 
 
+def conv3d_transpose(x, kernel, output_shape, strides=(1, 1, 1),
+                     padding='valid', data_format=None):
+    """3D deconvolution (transposed convolution).
+
+    # Arguments
+        kernel: kernel tensor.
+        output_shape: desired dimensions of output.
+        strides: strides tuple.
+        padding: string, "same" or "valid".
+        data_format: "channels_last" or "channels_first".
+            Whether to use Theano or TensorFlow data format
+        in inputs/kernels/outputs.
+
+    # Raises
+        ValueError: if using an even kernel size with padding 'same'.
+    """
+    flip_filters = False
+    if data_format is None:
+        data_format = image_data_format()
+    if data_format not in {'channels_first', 'channels_last'}:
+        raise ValueError('Unknown data_format ' + data_format)
+
+    if data_format == 'channels_last':
+        output_shape = (output_shape[0],
+                        output_shape[4],
+                        output_shape[1],
+                        output_shape[2],
+                        output_shape[3])
+
+    if hasattr(kernel, '_keras_shape'):
+        kernel_shape = kernel._keras_shape
+    else:
+        # Will only work if `kernel` is a shared variable.
+        kernel_shape = kernel.eval().shape
+
+    if padding == 'same' and kernel_shape[0] % 2 == 0:
+        raise ValueError('In `Conv3DTranspose`, with padding mode `same`, '
+                         'even kernel sizes are only supported with Tensorflow. '
+                         'With Theano, set `kernel_size` to an odd number.')
+
+    kernel_shape = _preprocess_conv3d_filter_shape(kernel_shape, data_format)
+
+    x = _preprocess_conv3d_input(x, data_format)
+    kernel = _preprocess_conv3d_kernel(kernel, data_format)
+
+    th_padding = _preprocess_padding(padding)
+    op = T.nnet.abstract_conv.AbstractConv3d_gradInputs(imshp=None,
+                                                        kshp=kernel_shape,
+                                                        subsample=strides,
+                                                        border_mode=th_padding,
+                                                        filter_flip=not flip_filters)
+    conv_out = op(kernel, x, output_shape[2:])
+    conv_out = _postprocess_conv3d_output(conv_out, x, padding,
+                                          kernel_shape, strides, data_format)
+    return conv_out
+
+
 def pool2d(x, pool_size, strides=(1, 1), padding='valid',
            data_format=None, pool_mode='max'):
     if data_format is None:
@@ -1970,9 +2042,6 @@ def pool2d(x, pool_size, strides=(1, 1), padding='valid',
     else:
         raise ValueError('Invalid border mode:', padding)
 
-    if data_format not in {'channels_first', 'channels_last'}:
-        raise ValueError('Unknown data_format:', data_format)
-
     if data_format == 'channels_last':
         x = x.dimshuffle((0, 3, 1, 2))
 
@@ -2020,8 +2089,6 @@ def pool3d(x, pool_size, strides=(1, 1, 1), padding='valid',
         padding = (0, 0, 0)
     else:
         raise ValueError('Invalid padding:', padding)
-    if data_format not in {'channels_first', 'channels_last'}:
-        raise ValueError('Unknown data_format:', data_format)
 
     if data_format == 'channels_last':
         x = x.dimshuffle((0, 4, 1, 2, 3))
@@ -2287,9 +2354,8 @@ def foldl(fn, elems, initializer=None, name=None):
 
     # We need to change the order of the arguments because theano accepts x as
     # first parameter and accumulator as second
-    fn2 = lambda x, acc: fn(acc, x)
-
-    return theano.foldl(fn2, elems, initializer, name=name)[0]
+    return theano.foldl(lambda x, acc: fn(acc, x),
+                        elems, initializer, name=name)[0]
 
 
 def foldr(fn, elems, initializer=None, name=None):
@@ -2311,9 +2377,8 @@ def foldr(fn, elems, initializer=None, name=None):
 
     # We need to change the order of the arguments because theano accepts x as
     # first parameter and accumulator as second
-    fn2 = lambda x, acc: fn(acc, x)
-
-    return theano.foldr(fn2, elems, initializer, name=name)[0]
+    return theano.foldr(lambda x, acc: fn(acc, x),
+                        elems, initializer, name=name)[0]
 
 
 def local_conv1d(inputs, kernel, kernel_size, strides, data_format=None):
@@ -2381,5 +2446,4 @@ def local_conv2d(inputs, kernel, kernel_size, strides, output_shape, data_format
         output = reshape(output,
                          (output_row, output_col, -1, filters))
         output = permute_dimensions(output, (2, 0, 1, 3))
-
     return output

keras/callbacks.py

@@ -490,8 +490,11 @@ class EarlyStopping(Callback):
     def on_epoch_end(self, epoch, logs=None):
         current = logs.get(self.monitor)
         if current is None:
-            warnings.warn('Early stopping requires %s available!' %
-                          (self.monitor), RuntimeWarning)
+            warnings.warn(
+                'Early stopping conditioned on metric `%s` '
+                'which is not available. Available metrics are: %s' %
+                (self.monitor, ','.join(list(logs.keys()))), RuntimeWarning
+            )
 
         if self.monitor_op(current - self.min_delta, self.best):
             self.best = current
@@ -520,8 +523,6 @@ class RemoteMonitor(Callback):
         path: String; path relative to `root` to which the events will be sent.
         field: String; JSON field under which the data will be stored.
         headers: Dictionary; optional custom HTTP headers.
-            Defaults to:
-            `{'Accept': 'application/json', 'Content-Type': 'application/json'}`
     """
 
     def __init__(self,
@@ -530,9 +531,7 @@ class RemoteMonitor(Callback):
                  field='data',
                  headers=None):
         super(RemoteMonitor, self).__init__()
-        if headers is None:
-            headers = {'Accept': 'application/json',
-                       'Content-Type': 'application/json'}
+
         self.root = root
         self.path = path
         self.field = field
@@ -582,20 +581,19 @@ class LearningRateScheduler(Callback):
 class TensorBoard(Callback):
     """Tensorboard basic visualizations.
 
+    [TensorBoard](https://www.tensorflow.org/get_started/summaries_and_tensorboard)
+    is a visualization tool provided with TensorFlow.
+
     This callback writes a log for TensorBoard, which allows
     you to visualize dynamic graphs of your training and test
     metrics, as well as activation histograms for the different
     layers in your model.
 
-    TensorBoard is a visualization tool provided with TensorFlow.
-
     If you have installed TensorFlow with pip, you should be able
     to launch TensorBoard from the command line:
     ```
     tensorboard --logdir=/full_path_to_your_logs
     ```
-    You can find more information about TensorBoard
-    [here](https://www.tensorflow.org/get_started/summaries_and_tensorboard).
 
     # Arguments
         log_dir: the path of the directory where to save the log
@@ -655,11 +653,12 @@ class TensorBoard(Callback):
             for layer in self.model.layers:
 
                 for weight in layer.weights:
-                    tf.summary.histogram(weight.name, weight)
+                    mapped_weight_name = weight.name.replace(':', '_')
+                    tf.summary.histogram(mapped_weight_name, weight)
                     if self.write_grads:
                         grads = model.optimizer.get_gradients(model.total_loss,
                                                               weight)
-                        tf.summary.histogram('{}_grad'.format(weight.name), grads)
+                        tf.summary.histogram('{}_grad'.format(mapped_weight_name), grads)
                     if self.write_images:
                         w_img = tf.squeeze(weight)
                         shape = K.int_shape(w_img)
@@ -692,7 +691,7 @@ class TensorBoard(Callback):
 
                         shape = K.int_shape(w_img)
                         assert len(shape) == 4 and shape[-1] in [1, 3, 4]
-                        tf.summary.image(weight.name, w_img)
+                        tf.summary.image(mapped_weight_name, w_img)
 
                 if hasattr(layer, 'output'):
                     tf.summary.histogram('{}_out'.format(layer.name),
@@ -874,8 +873,12 @@ class ReduceLROnPlateau(Callback):
         logs['lr'] = K.get_value(self.model.optimizer.lr)
         current = logs.get(self.monitor)
         if current is None:
-            warnings.warn('Learning Rate Plateau Reducing requires %s available!' %
-                          self.monitor, RuntimeWarning)
+            warnings.warn(
+                'Reduce LR on plateau conditioned on metric `%s` '
+                'which is not available. Available metrics are: %s' %
+                (self.monitor, ','.join(list(logs.keys()))), RuntimeWarning
+            )
+
         else:
             if self.in_cooldown():
                 self.cooldown_counter -= 1
@@ -973,7 +976,7 @@ class CSVLogger(Callback):
 
 
 class LambdaCallback(Callback):
-    """Callback for creating simple, custom callbacks on-the-fly.
+    r"""Callback for creating simple, custom callbacks on-the-fly.
 
     This callback is constructed with anonymous functions that will be called
     at the appropriate time. Note that the callbacks expects positional
@@ -1000,12 +1003,15 @@ class LambdaCallback(Callback):
         batch_print_callback = LambdaCallback(
             on_batch_begin=lambda batch,logs: print(batch))
 
-        # Plot the loss after every epoch.
-        import numpy as np
-        import matplotlib.pyplot as plt
-        plot_loss_callback = LambdaCallback(
-            on_epoch_end=lambda epoch, logs: plt.plot(np.arange(epoch),
-                                                      logs['loss']))
+        # Stream the epoch loss to a file in JSON format. The file content
+        # is not well-formed JSON but rather has a JSON object per line.
+        import json
+        json_log = open('loss_log.json', mode='wt', buffering=1)
+        json_logging_callback = LambdaCallback(
+            on_epoch_end=lambda epoch, logs: json_log.write(
+                json.dumps({'epoch': epoch, 'loss': logs['loss']}) + '\n'),
+            on_train_end=lambda logs: json_log.close()
+        )
 
         # Terminate some processes after having finished model training.
         processes = ...
@@ -1015,7 +1021,7 @@ class LambdaCallback(Callback):
 
         model.fit(...,
                   callbacks=[batch_print_callback,
-                             plot_loss_callback,
+                             json_logging_callback,
                              cleanup_callback])
         ```
     """

keras/constraints.py

@@ -27,7 +27,7 @@ class MaxNorm(Constraint):
             has shape `(input_dim, output_dim)`,
             set `axis` to `0` to constrain each weight vector
             of length `(input_dim,)`.
-            In a `Convolution2D` layer with `data_format="channels_last"`,
+            In a `Conv2D` layer with `data_format="channels_last"`,
             the weight tensor has shape
             `(rows, cols, input_depth, output_depth)`,
             set `axis` to `[0, 1, 2]`
@@ -71,7 +71,7 @@ class UnitNorm(Constraint):
             has shape `(input_dim, output_dim)`,
             set `axis` to `0` to constrain each weight vector
             of length `(input_dim,)`.
-            In a `Convolution2D` layer with `data_format="channels_last"`,
+            In a `Conv2D` layer with `data_format="channels_last"`,
             the weight tensor has shape
             `(rows, cols, input_depth, output_depth)`,
             set `axis` to `[0, 1, 2]`
@@ -112,7 +112,7 @@ class MinMaxNorm(Constraint):
             has shape `(input_dim, output_dim)`,
             set `axis` to `0` to constrain each weight vector
             of length `(input_dim,)`.
-            In a `Convolution2D` layer with `dim_ordering="tf"`,
+            In a `Conv2D` layer with `data_format="channels_last"`,
             the weight tensor has shape
             `(rows, cols, input_depth, output_depth)`,
             set `axis` to `[0, 1, 2]`

keras/datasets/cifar100.py

@@ -19,7 +19,7 @@ def load_data(label_mode='fine'):
         ValueError: in case of invalid `label_mode`.
     """
     if label_mode not in ['fine', 'coarse']:
-        raise ValueError('label_mode must be one of "fine" "coarse".')
+        raise ValueError('`label_mode` must be one of `"fine"`, `"coarse"`.')
 
     dirname = 'cifar-100-python'
     origin = 'http://www.cs.toronto.edu/~kriz/cifar-100-python.tar.gz'

keras/datasets/imdb.py

@@ -1,5 +1,6 @@
 from __future__ import absolute_import
 from ..utils.data_utils import get_file
+from ..preprocessing.sequence import _remove_long_seq
 from six.moves import zip
 import numpy as np
 import json
@@ -16,7 +17,7 @@ def load_data(path='imdb.npz', num_words=None, skip_top=0,
         num_words: max number of words to include. Words are ranked
             by how often they occur (in the training set) and only
             the most frequent words are kept
-        skip_top: skip the top N most frequently occuring words
+        skip_top: skip the top N most frequently occurring words
             (which may not be informative).
         maxlen: truncate sequences after this length.
         seed: random seed for sample shuffling.
@@ -47,14 +48,10 @@ def load_data(path='imdb.npz', num_words=None, skip_top=0,
     if kwargs:
         raise TypeError('Unrecognized keyword arguments: ' + str(kwargs))
 
-    path = get_file(path,
-                    origin='https://s3.amazonaws.com/text-datasets/imdb.npz')
-    f = np.load(path)
-    x_train = f['x_train']
-    labels_train = f['y_train']
-    x_test = f['x_test']
-    labels_test = f['y_test']
-    f.close()
+    path = get_file(path, origin='https://s3.amazonaws.com/text-datasets/imdb.npz')
+    with np.load(path) as f:
+        x_train, labels_train = f['x_train'], f['y_train']
+        x_test, labels_test = f['x_test'], f['y_test']
 
     np.random.seed(seed)
     np.random.shuffle(x_train)
@@ -75,14 +72,7 @@ def load_data(path='imdb.npz', num_words=None, skip_top=0,
         xs = [[w + index_from for w in x] for x in xs]
 
     if maxlen:
-        new_xs = []
-        new_labels = []
-        for x, y in zip(xs, labels):
-            if len(x) < maxlen:
-                new_xs.append(x)
-                new_labels.append(y)
-        xs = new_xs
-        labels = new_labels
+        xs, labels = _remove_long_seq(maxlen, xs, labels)
         if not xs:
             raise ValueError('After filtering for sequences shorter than maxlen=' +
                              str(maxlen) + ', no sequence was kept. '
@@ -94,22 +84,13 @@ def load_data(path='imdb.npz', num_words=None, skip_top=0,
     # reserve 'index_from' (=3 by default) characters:
     # 0 (padding), 1 (start), 2 (OOV)
     if oov_char is not None:
-        xs = [[oov_char if (w >= num_words or w < skip_top) else w for w in x] for x in xs]
+        xs = [[w if (skip_top <= w < num_words) else oov_char for w in x] for x in xs]
     else:
-        new_xs = []
-        for x in xs:
-            nx = []
-            for w in x:
-                if skip_top <= w < num_words:
-                    nx.append(w)
-            new_xs.append(nx)
-        xs = new_xs
+        xs = [[w for w in x if (skip_top <= w < num_words)] for x in xs]
 
-    x_train = np.array(xs[:len(x_train)])
-    y_train = np.array(labels[:len(x_train)])
-
-    x_test = np.array(xs[len(x_train):])
-    y_test = np.array(labels[len(x_train):])
+    idx = len(x_train)
+    x_train, y_train = np.array(xs[:idx]), np.array(labels[:idx])
+    x_test, y_test = np.array(xs[idx:]), np.array(labels[idx:])
 
     return (x_train, y_train), (x_test, y_test)
 

keras/datasets/mnist.py

@@ -14,9 +14,7 @@ def load_data(path='mnist.npz'):
     """
     path = get_file(path, origin='https://s3.amazonaws.com/img-datasets/mnist.npz')
     f = np.load(path)
-    x_train = f['x_train']
-    y_train = f['y_train']
-    x_test = f['x_test']
-    y_test = f['y_test']
+    x_train, y_train = f['x_train'], f['y_train']
+    x_test, y_test = f['x_test'], f['y_test']
     f.close()
     return (x_train, y_train), (x_test, y_test)

keras/datasets/reuters.py

@@ -1,6 +1,7 @@
 # -*- coding: utf-8 -*-
 from __future__ import absolute_import
 from ..utils.data_utils import get_file
+from ..preprocessing.sequence import _remove_long_seq
 from six.moves import zip
 import numpy as np
 import json
@@ -17,7 +18,7 @@ def load_data(path='reuters.npz', num_words=None, skip_top=0,
         num_words: max number of words to include. Words are ranked
             by how often they occur (in the training set) and only
             the most frequent words are kept
-        skip_top: skip the top N most frequently occuring words
+        skip_top: skip the top N most frequently occurring words
             (which may not be informative).
         maxlen: truncate sequences after this length.
         test_split: Fraction of the dataset to be used as test data.
@@ -46,10 +47,8 @@ def load_data(path='reuters.npz', num_words=None, skip_top=0,
         raise TypeError('Unrecognized keyword arguments: ' + str(kwargs))
 
     path = get_file(path, origin='https://s3.amazonaws.com/text-datasets/reuters.npz')
-    npzfile = np.load(path)
-    xs = npzfile['x']
-    labels = npzfile['y']
-    npzfile.close()
+    with np.load(path) as f:
+        xs, labels = f['x'], f['y']
 
     np.random.seed(seed)
     np.random.shuffle(xs)
@@ -62,14 +61,7 @@ def load_data(path='reuters.npz', num_words=None, skip_top=0,
         xs = [[w + index_from for w in x] for x in xs]
 
     if maxlen:
-        new_xs = []
-        new_labels = []
-        for x, y in zip(xs, labels):
-            if len(x) < maxlen:
-                new_xs.append(x)
-                new_labels.append(y)
-        xs = new_xs
-        labels = new_labels
+        xs, labels = _remove_long_seq(maxlen, xs, labels)
 
     if not num_words:
         num_words = max([max(x) for x in xs])
@@ -78,22 +70,13 @@ def load_data(path='reuters.npz', num_words=None, skip_top=0,
     # reserve 'index_from' (=3 by default) characters:
     # 0 (padding), 1 (start), 2 (OOV)
     if oov_char is not None:
-        xs = [[oov_char if (w >= num_words or w < skip_top) else w for w in x] for x in xs]
+        xs = [[w if (skip_top <= w < num_words) else oov_char for w in x] for x in xs]
     else:
-        new_xs = []
-        for x in xs:
-            nx = []
-            for w in x:
-                if skip_top <= w < num_words:
-                    nx.append(w)
-            new_xs.append(nx)
-        xs = new_xs
+        xs = [[w for w in x if (skip_top <= w < num_words)] for x in xs]
 
-    x_train = np.array(xs[:int(len(xs) * (1 - test_split))])
-    y_train = np.array(labels[:int(len(xs) * (1 - test_split))])
-
-    x_test = np.array(xs[int(len(xs) * (1 - test_split)):])
-    y_test = np.array(labels[int(len(xs) * (1 - test_split)):])
+    idx = int(len(xs) * (1 - test_split))
+    x_train, y_train = np.array(xs[:idx]), np.array(labels[:idx])
+    x_test, y_test = np.array(xs[idx:]), np.array(labels[idx:])
 
     return (x_train, y_train), (x_test, y_test)
 

keras/engine/topology.py

@@ -10,13 +10,13 @@ import warnings
 import copy
 import os
 import re
-import inspect
 from six.moves import zip
 
 from .. import backend as K
 from .. import initializers
 from ..utils.io_utils import ask_to_proceed_with_overwrite
 from ..utils.layer_utils import print_summary as print_layer_summary
+from ..utils.generic_utils import has_arg
 from ..utils import conv_utils
 from ..legacy import interfaces
 
@@ -584,7 +584,7 @@ class Layer(object):
             user_kwargs = copy.copy(kwargs)
             if not _is_all_none(previous_mask):
                 # The previous layer generated a mask.
-                if 'mask' in inspect.getargspec(self.call).args:
+                if has_arg(self.call, 'mask'):
                     if 'mask' not in kwargs:
                         # If mask is explicitly passed to __call__,
                         # we should override the default mask.
@@ -744,7 +744,7 @@ class Layer(object):
                                     str(mask))
             # masking not explicitly supported: return None as mask
             return None
-        # if masking is explictly supported, by default
+        # if masking is explicitly supported, by default
         # carry over the input mask
         return mask
 
@@ -1529,7 +1529,7 @@ class Container(Layer):
         # every time the Container is called on a set on input tensors,
         # we compute the output tensors,
         # output masks and output shapes in one pass,
-        # then cache them here. When of of these output is queried later,
+        # then cache them here. When one of these output is queried later,
         # we retrieve it from there instead of recomputing it.
         self._output_mask_cache = {}
         self._output_tensor_cache = {}
@@ -2206,7 +2206,7 @@ class Container(Layer):
                             kwargs = {}
                         if len(computed_data) == 1:
                             computed_tensor, computed_mask = computed_data[0]
-                            if 'mask' in inspect.getargspec(layer.call).args:
+                            if has_arg(layer.call, 'mask'):
                                 if 'mask' not in kwargs:
                                     kwargs['mask'] = computed_mask
                             output_tensors = _to_list(layer.call(computed_tensor, **kwargs))
@@ -2217,7 +2217,7 @@ class Container(Layer):
                         else:
                             computed_tensors = [x[0] for x in computed_data]
                             computed_masks = [x[1] for x in computed_data]
-                            if 'mask' in inspect.getargspec(layer.call).args:
+                            if has_arg(layer.call, 'mask'):
                                 if 'mask' not in kwargs:
                                     kwargs['mask'] = computed_masks
                             output_tensors = _to_list(layer.call(computed_tensors, **kwargs))
@@ -2637,10 +2637,25 @@ class Container(Layer):
         """
         return yaml.dump(self._updated_config(), **kwargs)
 
-    def summary(self, line_length=None, positions=None):
-        print_layer_summary(self,
-                            line_length=line_length,
-                            positions=positions)
+    def summary(self, line_length=None, positions=None, print_fn=print):
+        """Prints a string summary of the network.
+
+        # Arguments
+            line_length: Total length of printed lines
+                (e.g. set this to adapt the display to different
+                terminal window sizes).
+            positions: Relative or absolute positions of log elements
+                in each line. If not provided,
+                defaults to `[.33, .55, .67, 1.]`.
+            print_fn: Print function to use.
+                It will be called on each line of the summary.
+                You can set it to a custom function
+                in order to capture the string summary.
+        """
+        return print_layer_summary(self,
+                                   line_length=line_length,
+                                   positions=positions,
+                                   print_fn=print_fn)
 
 
 def get_source_inputs(tensor, layer=None, node_index=None):
@@ -2923,6 +2938,31 @@ def preprocess_weights_for_loading(layer, weights,
                                                     (2, 3, 1, 0))
                 weights = [kernel, recurrent_kernel, bias]
 
+        if layer.__class__.__name__ in ['Model', 'Sequential']:
+            new_weights = []
+            # trainable weights
+            for sublayer in layer.layers:
+                num_weights = len(sublayer.trainable_weights)
+                if num_weights > 0:
+                    new_weights.extend(preprocess_weights_for_loading(
+                        layer=sublayer,
+                        weights=weights[:num_weights],
+                        original_keras_version=original_keras_version,
+                        original_backend=original_backend))
+                    weights = weights[num_weights:]
+
+            # non-trainable weights
+            for sublayer in layer.layers:
+                num_weights = len([l for l in sublayer.weights if l not in sublayer.trainable_weights])
+                if num_weights > 0:
+                    new_weights.extend(preprocess_weights_for_loading(
+                        layer=sublayer,
+                        weights=weights[:num_weights],
+                        original_keras_version=original_keras_version,
+                        original_backend=original_backend))
+                    weights = weights[num_weights:]
+            weights = new_weights
+
     conv_layers = ['Conv1D',
                    'Conv2D',
                    'Conv3D',

keras/engine/training.py

@@ -4,12 +4,13 @@ from __future__ import absolute_import
 
 import warnings
 import copy
-import time
 import numpy as np
-import multiprocessing
-import threading
 import six
 
+from keras.utils import Sequence
+from keras.utils import GeneratorEnqueuer
+from keras.utils import OrderedEnqueuer
+
 try:
     import queue
 except ImportError:
@@ -100,7 +101,7 @@ def _standardize_input_data(data, names, shapes=None,
         if len(names) > 1:
             # Case: model expects multiple inputs but only received
             # a single Numpy array.
-            raise ValueError('The model expects ' + str(len(names)) +
+            raise ValueError('The model expects ' + str(len(names)) + ' ' +
                              exception_prefix +
                              ' arrays, but only received one array. '
                              'Found: array with shape ' + str(data.shape))
@@ -199,7 +200,7 @@ def _standardize_sample_weights(sample_weight, output_names):
                                                 'sample_weight')
 
 
-def _check_array_lengths(inputs, targets, weights):
+def _check_array_lengths(inputs, targets, weights=None):
     """Does user input validation for numpy arrays.
 
     # Arguments
@@ -210,29 +211,34 @@ def _check_array_lengths(inputs, targets, weights):
     # Raises
         ValueError: in case of incorrectly formatted data.
     """
-    x_lengths = [x.shape[0] for x in inputs]
-    y_lengths = [y.shape[0] for y in targets]
-    w_lengths = [w.shape[0] for w in weights]
-    set_x = set(x_lengths)
+    def set_of_lengths(x):
+        # return a set with the variation between
+        # different shapes, with None => 0
+        if x is None:
+            return {0}
+        else:
+            return set([0 if y is None else y.shape[0] for y in x])
+
+    set_x = set_of_lengths(inputs)
+    set_y = set_of_lengths(targets)
+    set_w = set_of_lengths(weights)
     if len(set_x) > 1:
         raise ValueError('All input arrays (x) should have '
                          'the same number of samples. Got array shapes: ' +
                          str([x.shape for x in inputs]))
-    set_y = set(y_lengths)
     if len(set_y) > 1:
         raise ValueError('All target arrays (y) should have '
                          'the same number of samples. Got array shapes: ' +
                          str([y.shape for y in targets]))
-    set_w = set(w_lengths)
-    if len(set_w) > 1:
-        raise ValueError('All sample_weight arrays should have '
-                         'the same number of samples. Got array shapes: ' +
-                         str([w.shape for w in weights]))
     if set_x and set_y and list(set_x)[0] != list(set_y)[0]:
         raise ValueError('Input arrays should have '
                          'the same number of samples as target arrays. '
                          'Found ' + str(list(set_x)[0]) + ' input samples '
                          'and ' + str(list(set_y)[0]) + ' target samples.')
+    if len(set_w) > 1:
+        raise ValueError('All sample_weight arrays should have '
+                         'the same number of samples. Got array shapes: ' +
+                         str([w.shape for w in weights]))
     if set_y and set_w and list(set_y)[0] != list(set_w)[0]:
         raise ValueError('Sample_weight arrays should have '
                          'the same number of samples as target arrays. Got ' +
@@ -254,7 +260,7 @@ def _check_loss_and_target_compatibility(targets, loss_fns, output_shapes):
         ValueError: if a loss function or target array
             is incompatible with an output.
     """
-    key_losses = {'mean_square_error',
+    key_losses = {'mean_squared_error',
                   'binary_crossentropy',
                   'categorical_crossentropy'}
     for y, loss, shape in zip(targets, loss_fns, output_shapes):
@@ -386,21 +392,25 @@ def _slice_arrays(arrays, start=None, stop=None):
     # Returns
         A slice of the array(s).
     """
-    if isinstance(arrays, list):
+    if arrays is None:
+        return [None]
+    elif isinstance(arrays, list):
         if hasattr(start, '__len__'):
             # hdf5 datasets only support list objects as indices
             if hasattr(start, 'shape'):
                 start = start.tolist()
-            return [x[start] for x in arrays]
+            return [None if x is None else x[start] for x in arrays]
         else:
-            return [x[start:stop] for x in arrays]
+            return [None if x is None else x[start:stop] for x in arrays]
     else:
         if hasattr(start, '__len__'):
             if hasattr(start, 'shape'):
                 start = start.tolist()
             return arrays[start]
-        else:
+        elif hasattr(start, '__getitem__'):
             return arrays[start:stop]
+        else:
+            return [None]
 
 
 def _weighted_masked_objective(fn):
@@ -443,13 +453,12 @@ def _weighted_masked_objective(fn):
             #  to the number of unmasked samples.
             score_array /= K.mean(mask)
 
-        # reduce score_array to same ndim as weight array
-        ndim = K.ndim(score_array)
-        weight_ndim = K.ndim(weights)
-        score_array = K.mean(score_array, axis=list(range(weight_ndim, ndim)))
-
         # apply sample weighting
         if weights is not None:
+            # reduce score_array to same ndim as weight array
+            ndim = K.ndim(score_array)
+            weight_ndim = K.ndim(weights)
+            score_array = K.mean(score_array, axis=list(range(weight_ndim, ndim)))
             score_array *= weights
             score_array /= K.mean(K.cast(K.not_equal(weights, 0), K.floatx()))
         return K.mean(score_array)
@@ -562,7 +571,7 @@ def _standardize_weights(y, sample_weight=None, class_weight=None,
         return sample_weight
     elif isinstance(class_weight, dict):
         if len(y.shape) > 2:
-            raise ValueError('class_weight not supported for '
+            raise ValueError('`class_weight` not supported for '
                              '3+ dimensional targets.')
         if y.shape[1] > 1:
             y_classes = y.argmax(axis=1)
@@ -579,97 +588,6 @@ def _standardize_weights(y, sample_weight=None, class_weight=None,
             return np.ones((y.shape[0], y.shape[1]), dtype=K.floatx())
 
 
-class GeneratorEnqueuer(object):
-    """Builds a queue out of a data generator.
-
-    Used in `fit_generator`, `evaluate_generator`, `predict_generator`.
-
-    # Arguments
-        generator: a generator function which endlessly yields data
-        pickle_safe: use multiprocessing if True, otherwise threading
-    """
-
-    def __init__(self, generator, pickle_safe=False):
-        self._generator = generator
-        self._pickle_safe = pickle_safe
-        self._threads = []
-        self._stop_event = None
-        self.queue = None
-
-    def start(self, workers=1, max_q_size=10, wait_time=0.05):
-        """Kicks off threads which add data from the generator into the queue.
-
-        # Arguments
-            workers: number of worker threads
-            max_q_size: queue size (when full, threads could block on put())
-            wait_time: time to sleep in-between calls to put()
-        """
-
-        def data_generator_task():
-            while not self._stop_event.is_set():
-                try:
-                    if self._pickle_safe or self.queue.qsize() < max_q_size:
-                        generator_output = next(self._generator)
-                        self.queue.put(generator_output)
-                    else:
-                        time.sleep(wait_time)
-                except Exception:
-                    self._stop_event.set()
-                    raise
-
-        try:
-            if self._pickle_safe:
-                self.queue = multiprocessing.Queue(maxsize=max_q_size)
-                self._stop_event = multiprocessing.Event()
-            else:
-                self.queue = queue.Queue()
-                self._stop_event = threading.Event()
-
-            for _ in range(workers):
-                if self._pickle_safe:
-                    # Reset random seed else all children processes
-                    # share the same seed
-                    np.random.seed()
-                    thread = multiprocessing.Process(target=data_generator_task)
-                    thread.daemon = True
-                else:
-                    thread = threading.Thread(target=data_generator_task)
-                self._threads.append(thread)
-                thread.start()
-        except:
-            self.stop()
-            raise
-
-    def is_running(self):
-        return self._stop_event is not None and not self._stop_event.is_set()
-
-    def stop(self, timeout=None):
-        """Stop running threads and wait for them to exit, if necessary.
-
-        Should be called by the same thread which called start().
-
-        # Arguments
-            timeout: maximum time to wait on thread.join()
-        """
-        if self.is_running():
-            self._stop_event.set()
-
-        for thread in self._threads:
-            if thread.is_alive():
-                if self._pickle_safe:
-                    thread.terminate()
-                else:
-                    thread.join(timeout)
-
-        if self._pickle_safe:
-            if self.queue is not None:
-                self.queue.close()
-
-        self._threads = []
-        self._stop_event = None
-        self.queue = None
-
-
 class Model(Container):
     """The `Model` class adds training & evaluation routines to a `Container`.
     """
@@ -990,14 +908,8 @@ class Model(Container):
         self.test_function = None
         self.predict_function = None
 
-        # Collected trainable weights and sort them deterministically.
+        # Collected trainable weights, sorted in topological order.
         trainable_weights = self.trainable_weights
-        # Sort weights by name.
-        if trainable_weights:
-            if K.backend() == 'theano':
-                trainable_weights.sort(key=lambda x: x.name if x.name else x.auto_name)
-            else:
-                trainable_weights.sort(key=lambda x: x.name)
         self._collected_trainable_weights = trainable_weights
 
     def _make_train_function(self):
@@ -1716,9 +1628,9 @@ class Model(Container):
                       validation_data=None,
                       validation_steps=None,
                       class_weight=None,
-                      max_q_size=10,
+                      max_queue_size=10,
                       workers=1,
-                      pickle_safe=False,
+                      use_multiprocessing=False,
                       initial_epoch=0):
         """Fits the model on data yielded batch-by-batch by a Python generator.
 
@@ -1726,8 +1638,14 @@ class Model(Container):
         For instance, this allows you to do real-time data augmentation
         on images on CPU in parallel to training your model on GPU.
 
+        The use of `keras.utils.Sequence` guarantees the ordering
+        and guarantees the single use of every input per epoch when
+        using `use_multiprocessing=True`.
+
         # Arguments
-            generator: a generator.
+            generator: a generator or an instance of Sequence (keras.utils.Sequence)
+                    object in order to avoid duplicate data
+                    when using multiprocessing.
                 The output of the generator must be either
                 - a tuple (inputs, targets)
                 - a tuple (inputs, targets, sample_weights).
@@ -1752,10 +1670,10 @@ class Model(Container):
                 to yield from `generator` before stopping.
             class_weight: dictionary mapping class indices to a weight
                 for the class.
-            max_q_size: maximum size for the generator queue
+            max_queue_size: maximum size for the generator queue
             workers: maximum number of processes to spin up
                 when using process based threading
-            pickle_safe: if True, use process based threading.
+            use_multiprocessing: if True, use process based threading.
                 Note that because
                 this implementation relies on multiprocessing,
                 you should not pass
@@ -1800,7 +1718,8 @@ class Model(Container):
         # python 2 has 'next', 3 has '__next__'
         # avoid any explicit version checks
         val_gen = (hasattr(validation_data, 'next') or
-                   hasattr(validation_data, '__next__'))
+                   hasattr(validation_data, '__next__') or
+                   isinstance(validation_data, Sequence))
         if val_gen and not validation_steps:
             raise ValueError('When using a generator for validation data, '
                              'you must specify a value for '
@@ -1839,7 +1758,7 @@ class Model(Container):
             elif len(validation_data) == 3:
                 val_x, val_y, val_sample_weight = validation_data
             else:
-                raise ValueError('validation_data should be a tuple '
+                raise ValueError('`validation_data` should be a tuple '
                                  '`(val_x, val_y, val_sample_weight)` '
                                  'or `(val_x, val_y)`. Found: ' +
                                  str(validation_data))
@@ -1850,11 +1769,25 @@ class Model(Container):
                 val_data += [0.]
             for cbk in callbacks:
                 cbk.validation_data = val_data
+        is_sequence = isinstance(generator, Sequence)
+        if not is_sequence and use_multiprocessing and workers > 1:
+            warnings.warn(
+                UserWarning('Using a generator with `use_multiprocessing=True`'
+                            ' and multiple workers may duplicate your data.'
+                            ' Please consider using the`keras.utils.Sequence'
+                            ' class.'))
         enqueuer = None
 
         try:
-            enqueuer = GeneratorEnqueuer(generator, pickle_safe=pickle_safe)
-            enqueuer.start(max_q_size=max_q_size, workers=workers)
+            if is_sequence:
+                enqueuer = OrderedEnqueuer(generator,
+                                           use_multiprocessing=use_multiprocessing)
+            else:
+                enqueuer = GeneratorEnqueuer(generator,
+                                             use_multiprocessing=use_multiprocessing,
+                                             wait_time=wait_time)
+            enqueuer.start(workers=workers, max_queue_size=max_queue_size)
+            output_generator = enqueuer.get()
 
             callback_model.stop_training = False
             while epoch < epochs:
@@ -1862,16 +1795,10 @@ class Model(Container):
                 steps_done = 0
                 batch_index = 0
                 while steps_done < steps_per_epoch:
-                    generator_output = None
-                    while enqueuer.is_running():
-                        if not enqueuer.queue.empty():
-                            generator_output = enqueuer.queue.get()
-                            break
-                        else:
-                            time.sleep(wait_time)
+                    generator_output = next(output_generator)
 
                     if not hasattr(generator_output, '__len__'):
-                        raise ValueError('output of generator should be '
+                        raise ValueError('Output of generator should be '
                                          'a tuple `(x, y, sample_weight)` '
                                          'or `(x, y)`. Found: ' +
                                          str(generator_output))
@@ -1881,7 +1808,7 @@ class Model(Container):
                     elif len(generator_output) == 3:
                         x, y, sample_weight = generator_output
                     else:
-                        raise ValueError('output of generator should be '
+                        raise ValueError('Output of generator should be '
                                          'a tuple `(x, y, sample_weight)` '
                                          'or `(x, y)`. Found: ' +
                                          str(generator_output))
@@ -1919,9 +1846,9 @@ class Model(Container):
                             val_outs = self.evaluate_generator(
                                 validation_data,
                                 validation_steps,
-                                max_q_size=max_q_size,
+                                max_queue_size=max_queue_size,
                                 workers=workers,
-                                pickle_safe=pickle_safe)
+                                use_multiprocessing=use_multiprocessing)
                         else:
                             # No need for try/except because
                             # data has already been validated.
@@ -1950,7 +1877,9 @@ class Model(Container):
 
     @interfaces.legacy_generator_methods_support
     def evaluate_generator(self, generator, steps,
-                           max_q_size=10, workers=1, pickle_safe=False):
+                           max_queue_size=10,
+                           workers=1,
+                           use_multiprocessing=False):
         """Evaluates the model on a data generator.
 
         The generator should return the same kind of data
@@ -1959,12 +1888,15 @@ class Model(Container):
         # Arguments
             generator: Generator yielding tuples (inputs, targets)
                 or (inputs, targets, sample_weights)
+                or an instance of Sequence (keras.utils.Sequence)
+                    object in order to avoid duplicate data
+                    when using multiprocessing.
             steps: Total number of steps (batches of samples)
                 to yield from `generator` before stopping.
-            max_q_size: maximum size for the generator queue
+            max_queue_size: maximum size for the generator queue
             workers: maximum number of processes to spin up
                 when using process based threading
-            pickle_safe: if True, use process based threading.
+            use_multiprocessing: if True, use process based threading.
                 Note that because
                 this implementation relies on multiprocessing,
                 you should not pass
@@ -1988,23 +1920,30 @@ class Model(Container):
         wait_time = 0.01
         all_outs = []
         batch_sizes = []
+        is_sequence = isinstance(generator, Sequence)
+        if not is_sequence and use_multiprocessing and workers > 1:
+            warnings.warn(
+                UserWarning('Using a generator with `use_multiprocessing=True`'
+                            ' and multiple workers may duplicate your data.'
+                            ' Please consider using the`keras.utils.Sequence'
+                            ' class.'))
         enqueuer = None
 
         try:
-            enqueuer = GeneratorEnqueuer(generator, pickle_safe=pickle_safe)
-            enqueuer.start(workers=workers, max_q_size=max_q_size)
+            if is_sequence:
+                enqueuer = OrderedEnqueuer(generator,
+                                           use_multiprocessing=use_multiprocessing)
+            else:
+                enqueuer = GeneratorEnqueuer(generator,
+                                             use_multiprocessing=use_multiprocessing,
+                                             wait_time=wait_time)
+            enqueuer.start(workers=workers, max_queue_size=max_queue_size)
+            output_generator = enqueuer.get()
 
             while steps_done < steps:
-                generator_output = None
-                while enqueuer.is_running():
-                    if not enqueuer.queue.empty():
-                        generator_output = enqueuer.queue.get()
-                        break
-                    else:
-                        time.sleep(wait_time)
-
+                generator_output = next(output_generator)
                 if not hasattr(generator_output, '__len__'):
-                    raise ValueError('output of generator should be a tuple '
+                    raise ValueError('Output of generator should be a tuple '
                                      '(x, y, sample_weight) '
                                      'or (x, y). Found: ' +
                                      str(generator_output))
@@ -2014,7 +1953,7 @@ class Model(Container):
                 elif len(generator_output) == 3:
                     x, y, sample_weight = generator_output
                 else:
-                    raise ValueError('output of generator should be a tuple '
+                    raise ValueError('Output of generator should be a tuple '
                                      '(x, y, sample_weight) '
                                      'or (x, y). Found: ' +
                                      str(generator_output))
@@ -2026,6 +1965,9 @@ class Model(Container):
                     batch_size = len(list(x.values())[0])
                 else:
                     batch_size = len(x)
+                if batch_size == 0:
+                    raise ValueError('Received an empty batch. '
+                                     'Batches should at least contain one item.')
                 all_outs.append(outs)
 
                 steps_done += 1
@@ -2047,21 +1989,26 @@ class Model(Container):
 
     @interfaces.legacy_generator_methods_support
     def predict_generator(self, generator, steps,
-                          max_q_size=10, workers=1,
-                          pickle_safe=False, verbose=0):
+                          max_queue_size=10,
+                          workers=1,
+                          use_multiprocessing=False,
+                          verbose=0):
         """Generates predictions for the input samples from a data generator.
 
         The generator should return the same kind of data as accepted by
         `predict_on_batch`.
 
         # Arguments
-            generator: Generator yielding batches of input samples.
+            generator: Generator yielding batches of input samples
+                    or an instance of Sequence (keras.utils.Sequence)
+                    object in order to avoid duplicate data
+                    when using multiprocessing.
             steps: Total number of steps (batches of samples)
                 to yield from `generator` before stopping.
-            max_q_size: Maximum size for the generator queue.
+            max_queue_size: Maximum size for the generator queue.
             workers: Maximum number of processes to spin up
                 when using process based threading
-            pickle_safe: If `True`, use process based threading.
+            use_multiprocessing: If `True`, use process based threading.
                 Note that because
                 this implementation relies on multiprocessing,
                 you should not pass
@@ -2082,24 +2029,31 @@ class Model(Container):
         steps_done = 0
         wait_time = 0.01
         all_outs = []
+        is_sequence = isinstance(generator, Sequence)
+        if not is_sequence and use_multiprocessing and workers > 1:
+            warnings.warn(
+                UserWarning('Using a generator with `use_multiprocessing=True`'
+                            ' and multiple workers may duplicate your data.'
+                            ' Please consider using the`keras.utils.Sequence'
+                            ' class.'))
         enqueuer = None
 
         try:
-            enqueuer = GeneratorEnqueuer(generator, pickle_safe=pickle_safe)
-            enqueuer.start(workers=workers, max_q_size=max_q_size)
+            if is_sequence:
+                enqueuer = OrderedEnqueuer(generator,
+                                           use_multiprocessing=use_multiprocessing)
+            else:
+                enqueuer = GeneratorEnqueuer(generator,
+                                             use_multiprocessing=use_multiprocessing,
+                                             wait_time=wait_time)
+            enqueuer.start(workers=workers, max_queue_size=max_queue_size)
+            output_generator = enqueuer.get()
 
             if verbose == 1:
                 progbar = Progbar(target=steps)
 
             while steps_done < steps:
-                generator_output = None
-                while enqueuer.is_running():
-                    if not enqueuer.queue.empty():
-                        generator_output = enqueuer.queue.get()
-                        break
-                    else:
-                        time.sleep(wait_time)
-
+                generator_output = next(output_generator)
                 if isinstance(generator_output, tuple):
                     # Compatibility with the generators
                     # used for training.
@@ -2108,7 +2062,7 @@ class Model(Container):
                     elif len(generator_output) == 3:
                         x, _, _ = generator_output
                     else:
-                        raise ValueError('output of generator should be '
+                        raise ValueError('Output of generator should be '
                                          'a tuple `(x, y, sample_weight)` '
                                          'or `(x, y)`. Found: ' +
                                          str(generator_output))

keras/initializers.py

@@ -371,6 +371,29 @@ def he_normal(seed=None):
                            seed=seed)
 
 
+def lecun_normal(seed=None):
+    """LeCun normal initializer.
+
+    It draws samples from a truncated normal distribution centered on 0
+    with `stddev = sqrt(1 / fan_in)`
+    where `fan_in` is the number of input units in the weight tensor.
+
+    # Arguments
+        seed: A Python integer. Used to seed the random generator.
+
+    # Returns
+        An initializer.
+
+    # References
+        - [Self-Normalizing Neural Networks](https://arxiv.org/abs/1706.02515)
+        - [Efficient Backprop](http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf)
+    """
+    return VarianceScaling(scale=1.,
+                           mode='fan_in',
+                           distribution='normal',
+                           seed=seed)
+
+
 def he_uniform(seed=None):
     """He uniform variance scaling initializer.
 

keras/layers/advanced_activations.py

@@ -41,7 +41,7 @@ class LeakyReLU(Layer):
         return K.relu(inputs, alpha=self.alpha)
 
     def get_config(self):
-        config = {'alpha': self.alpha}
+        config = {'alpha': float(self.alpha)}
         base_config = super(LeakyReLU, self).get_config()
         return dict(list(base_config.items()) + list(config.items()))
 

keras/layers/convolutional.py

@@ -476,7 +476,7 @@ class Conv3D(_Conv):
     When using this layer as the first layer in a model,
     provide the keyword argument `input_shape`
     (tuple of integers, does not include the sample axis),
-    e.g. `input_shape=(128, 128, 128, 3)` for 128x128x128 volumes
+    e.g. `input_shape=(128, 128, 128, 1)` for 128x128x128 volumes
     with a single channel,
     in `data_format="channels_last"`.
 
@@ -484,7 +484,7 @@ class Conv3D(_Conv):
         filters: Integer, the dimensionality of the output space
             (i.e. the number output of filters in the convolution).
         kernel_size: An integer or tuple/list of 3 integers, specifying the
-            width and height of the 3D convolution window.
+            depth, height and width of the 3D convolution window.
             Can be a single integer to specify the same value for
             all spatial dimensions.
         strides: An integer or tuple/list of 3 integers,
@@ -806,6 +806,237 @@ class Conv2DTranspose(Conv2D):
         return config
 
 
+class Conv3DTranspose(Conv3D):
+    """Transposed convolution layer (sometimes called Deconvolution).
+
+    The need for transposed convolutions generally arises
+    from the desire to use a transformation going in the opposite direction
+    of a normal convolution, i.e., from something that has the shape of the
+    output of some convolution to something that has the shape of its input
+    while maintaining a connectivity pattern that is compatible with
+    said convolution.
+
+    When using this layer as the first layer in a model,
+    provide the keyword argument `input_shape`
+    (tuple of integers, does not include the sample axis),
+    e.g. `input_shape=(128, 128, 128, 3)` for a 128x128x128 volume with 3 channels
+    if `data_format="channels_last"`.
+
+    # Arguments
+        filters: Integer, the dimensionality of the output space
+            (i.e. the number of output filters in the convolution).
+        kernel_size: An integer or tuple/list of 3 integers, specifying the
+            width and height of the 3D convolution window.
+            Can be a single integer to specify the same value for
+            all spatial dimensions.
+        strides: An integer or tuple/list of 3 integers,
+            specifying the strides of the convolution along the width and height.
+            Can be a single integer to specify the same value for
+            all spatial dimensions.
+            Specifying any stride value != 1 is incompatible with specifying
+            any `dilation_rate` value != 1.
+        padding: one of `"valid"` or `"same"` (case-insensitive).
+        data_format: A string,
+            one of `channels_last` (default) or `channels_first`.
+            The ordering of the dimensions in the inputs.
+            `channels_last` corresponds to inputs with shape
+            `(batch, depth, height, width, channels)` while `channels_first`
+            corresponds to inputs with shape
+            `(batch, channels, depth, height, width)`.
+            It defaults to the `image_data_format` value found in your
+            Keras config file at `~/.keras/keras.json`.
+            If you never set it, then it will be "channels_last".
+        dilation_rate: an integer or tuple/list of 3 integers, specifying
+            the dilation rate to use for dilated convolution.
+            Can be a single integer to specify the same value for
+            all spatial dimensions.
+            Currently, specifying any `dilation_rate` value != 1 is
+            incompatible with specifying any stride value != 1.
+        activation: Activation function to use
+            (see [activations](../activations.md)).
+            If you don't specify anything, no activation is applied
+            (ie. "linear" activation: `a(x) = x`).
+        use_bias: Boolean, whether the layer uses a bias vector.
+        kernel_initializer: Initializer for the `kernel` weights matrix
+            (see [initializers](../initializers.md)).
+        bias_initializer: Initializer for the bias vector
+            (see [initializers](../initializers.md)).
+        kernel_regularizer: Regularizer function applied to
+            the `kernel` weights matrix
+            (see [regularizer](../regularizers.md)).
+        bias_regularizer: Regularizer function applied to the bias vector
+            (see [regularizer](../regularizers.md)).
+        activity_regularizer: Regularizer function applied to
+            the output of the layer (its "activation").
+            (see [regularizer](../regularizers.md)).
+        kernel_constraint: Constraint function applied to the kernel matrix
+            (see [constraints](../constraints.md)).
+        bias_constraint: Constraint function applied to the bias vector
+            (see [constraints](../constraints.md)).
+
+    # Input shape
+        5D tensor with shape:
+        `(batch, channels, depth, rows, cols)` if data_format='channels_first'
+        or 5D tensor with shape:
+        `(batch, depth, rows, cols, channels)` if data_format='channels_last'.
+
+    # Output shape
+        5D tensor with shape:
+        `(batch, filters, new_depth, new_rows, new_cols)` if data_format='channels_first'
+        or 5D tensor with shape:
+        `(batch, new_depth, new_rows, new_cols, filters)` if data_format='channels_last'.
+        `depth` and `rows` and `cols` values might have changed due to padding.
+
+    # References
+        - [A guide to convolution arithmetic for deep learning](https://arxiv.org/abs/1603.07285v1)
+        - [Deconvolutional Networks](http://www.matthewzeiler.com/pubs/cvpr2010/cvpr2010.pdf)
+    """
+
+    def __init__(self, filters,
+                 kernel_size,
+                 strides=(1, 1, 1),
+                 padding='valid',
+                 data_format=None,
+                 activation=None,
+                 use_bias=True,
+                 kernel_initializer='glorot_uniform',
+                 bias_initializer='zeros',
+                 kernel_regularizer=None,
+                 bias_regularizer=None,
+                 activity_regularizer=None,
+                 kernel_constraint=None,
+                 bias_constraint=None,
+                 **kwargs):
+        super(Conv3DTranspose, self).__init__(
+            filters,
+            kernel_size,
+            strides=strides,
+            padding=padding,
+            data_format=data_format,
+            activation=activation,
+            use_bias=use_bias,
+            kernel_initializer=kernel_initializer,
+            bias_initializer=bias_initializer,
+            kernel_regularizer=kernel_regularizer,
+            bias_regularizer=bias_regularizer,
+            activity_regularizer=activity_regularizer,
+            kernel_constraint=kernel_constraint,
+            bias_constraint=bias_constraint,
+            **kwargs)
+        self.input_spec = InputSpec(ndim=5)
+
+    def build(self, input_shape):
+        if len(input_shape) != 5:
+            raise ValueError('Inputs should have rank ' +
+                             str(5) +
+                             '; Received input shape:', str(input_shape))
+        if self.data_format == 'channels_first':
+            channel_axis = 1
+        else:
+            channel_axis = -1
+        if input_shape[channel_axis] is None:
+            raise ValueError('The channel dimension of the inputs '
+                             'should be defined. Found `None`.')
+        input_dim = input_shape[channel_axis]
+        kernel_shape = self.kernel_size + (self.filters, input_dim)
+
+        self.kernel = self.add_weight(shape=kernel_shape,
+                                      initializer=self.kernel_initializer,
+                                      name='kernel',
+                                      regularizer=self.kernel_regularizer,
+                                      constraint=self.kernel_constraint)
+        if self.use_bias:
+            self.bias = self.add_weight(shape=(self.filters,),
+                                        initializer=self.bias_initializer,
+                                        name='bias',
+                                        regularizer=self.bias_regularizer,
+                                        constraint=self.bias_constraint)
+        else:
+            self.bias = None
+        # Set input spec.
+        self.input_spec = InputSpec(ndim=5, axes={channel_axis: input_dim})
+        self.built = True
+
+    def call(self, inputs):
+        input_shape = K.shape(inputs)
+        batch_size = input_shape[0]
+        if self.data_format == 'channels_first':
+            d_axis, h_axis, w_axis = 2, 3, 4
+        else:
+            d_axis, h_axis, w_axis = 1, 2, 3
+
+        depth = input_shape[d_axis]
+        height = input_shape[h_axis]
+        width = input_shape[w_axis]
+
+        kernel_d, kernel_h, kernel_w = self.kernel_size
+        stride_d, stride_h, stride_w = self.strides
+
+        # Infer the dynamic output shape:
+        out_depth = conv_utils.deconv_length(depth,
+                                             stride_d, kernel_d,
+                                             self.padding)
+        out_height = conv_utils.deconv_length(height,
+                                              stride_h, kernel_h,
+                                              self.padding)
+        out_width = conv_utils.deconv_length(width,
+                                             stride_w, kernel_w,
+                                             self.padding)
+
+        if self.data_format == 'channels_first':
+            output_shape = (batch_size, self.filters, out_depth, out_height, out_width)
+        else:
+            output_shape = (batch_size, out_depth, out_height, out_width, self.filters)
+
+        outputs = K.conv3d_transpose(inputs,
+                                     self.kernel,
+                                     output_shape,
+                                     self.strides,
+                                     padding=self.padding,
+                                     data_format=self.data_format)
+
+        if self.bias:
+            outputs = K.bias_add(
+                outputs,
+                self.bias,
+                data_format=self.data_format)
+
+        if self.activation is not None:
+            return self.activation(outputs)
+        return outputs
+
+    def compute_output_shape(self, input_shape):
+        output_shape = list(input_shape)
+        if self.data_format == 'channels_first':
+            c_axis, d_axis, h_axis, w_axis = 1, 2, 3, 4
+        else:
+            c_axis, d_axis, h_axis, w_axis = 4, 1, 2, 3
+
+        kernel_d, kernel_h, kernel_w = self.kernel_size
+        stride_d, stride_h, stride_w = self.strides
+
+        output_shape[c_axis] = self.filters
+        output_shape[d_axis] = conv_utils.deconv_length(output_shape[d_axis],
+                                                        stride_d,
+                                                        kernel_d,
+                                                        self.padding)
+        output_shape[h_axis] = conv_utils.deconv_length(output_shape[h_axis],
+                                                        stride_h,
+                                                        kernel_h,
+                                                        self.padding)
+        output_shape[w_axis] = conv_utils.deconv_length(output_shape[w_axis],
+                                                        stride_w,
+                                                        kernel_w,
+                                                        self.padding)
+
+        return tuple(output_shape)
+
+    def get_config(self):
+        config = super(Conv3DTranspose, self).get_config()
+        config.pop('dilation_rate')
+        return config
+
+
 class SeparableConv2D(Conv2D):
     """Depthwise separable 2D convolution.
 
@@ -1891,6 +2122,7 @@ Convolution3D = Conv3D
 SeparableConvolution2D = SeparableConv2D
 Convolution2DTranspose = Conv2DTranspose
 Deconvolution2D = Deconv2D = Conv2DTranspose
+Deconvolution3D = Deconv3D = Conv3DTranspose
 
 # Legacy aliases
 AtrousConv1D = AtrousConvolution1D

keras/layers/core.py

@@ -5,7 +5,6 @@ from __future__ import division
 import numpy as np
 
 import copy
-import inspect
 import types as python_types
 import warnings
 
@@ -19,6 +18,7 @@ from ..engine import Layer
 from ..utils.generic_utils import func_dump
 from ..utils.generic_utils import func_load
 from ..utils.generic_utils import deserialize_keras_object
+from ..utils.generic_utils import has_arg
 from ..legacy import interfaces
 
 
@@ -193,8 +193,8 @@ class SpatialDropout2D(Dropout):
         if data_format is None:
             data_format = K.image_data_format()
         if data_format not in {'channels_last', 'channels_first'}:
-            raise ValueError('data_format must be in '
-                             '{"channels_last", "channels_first"}')
+            raise ValueError('`data_format` must be in '
+                             '{`"channels_last"`, `"channels_first"`}')
         self.data_format = data_format
         self.input_spec = InputSpec(ndim=4)
 
@@ -202,10 +202,8 @@ class SpatialDropout2D(Dropout):
         input_shape = K.shape(inputs)
         if self.data_format == 'channels_first':
             noise_shape = (input_shape[0], input_shape[1], 1, 1)
-        elif self.data_format == 'channels_last':
-            noise_shape = (input_shape[0], 1, 1, input_shape[3])
         else:
-            raise ValueError('Invalid data_format:', self.data_format)
+            noise_shape = (input_shape[0], 1, 1, input_shape[3])
         return noise_shape
 
 
@@ -248,8 +246,8 @@ class SpatialDropout3D(Dropout):
         if data_format is None:
             data_format = K.image_data_format()
         if data_format not in {'channels_last', 'channels_first'}:
-            raise ValueError('data_format must be in '
-                             '{"channels_last", "channels_first"}')
+            raise ValueError('`data_format` must be in '
+                             '{`"channels_last"`, `"channels_first"`}')
         self.data_format = data_format
         self.input_spec = InputSpec(ndim=5)
 
@@ -257,10 +255,8 @@ class SpatialDropout3D(Dropout):
         input_shape = K.shape(inputs)
         if self.data_format == 'channels_first':
             noise_shape = (input_shape[0], input_shape[1], 1, 1, 1)
-        elif self.data_format == 'channels_last':
-            noise_shape = (input_shape[0], 1, 1, 1, input_shape[4])
         else:
-            raise ValueError('Invalid data_format:', self.data_format)
+            noise_shape = (input_shape[0], 1, 1, 1, input_shape[4])
         return noise_shape
 
 
@@ -460,7 +456,7 @@ class Flatten(Layer):
 
     ```python
         model = Sequential()
-        model.add(Convolution2D(64, 3, 3,
+        model.add(Conv2D(64, 3, 3,
                                 border_mode='same',
                                 input_shape=(3, 32, 32)))
         # now: model.output_shape == (None, 64, 32, 32)
@@ -641,13 +637,12 @@ class Lambda(Layer):
         else:
             shape = self._output_shape(input_shape)
             if not isinstance(shape, (list, tuple)):
-                raise ValueError('output_shape function must return a tuple')
+                raise ValueError('`output_shape` function must return a tuple.')
             return tuple(shape)
 
     def call(self, inputs, mask=None):
         arguments = self.arguments
-        arg_spec = inspect.getargspec(self.function)
-        if 'mask' in arg_spec.args:
+        if has_arg(self.function, 'mask'):
             arguments['mask'] = mask
         return self.function(inputs, **arguments)
 

keras/layers/embeddings.py

@@ -75,7 +75,6 @@ class Embedding(Layer):
                  mask_zero=False,
                  input_length=None,
                  **kwargs):
-        kwargs['dtype'] = 'int32'
         if 'input_shape' not in kwargs:
             if input_length:
                 kwargs['input_shape'] = (input_length,)
@@ -98,7 +97,8 @@ class Embedding(Layer):
             initializer=self.embeddings_initializer,
             name='embeddings',
             regularizer=self.embeddings_regularizer,
-            constraint=self.embeddings_constraint)
+            constraint=self.embeddings_constraint,
+            dtype=self.dtype)
         self.built = True
 
     def compute_mask(self, inputs, mask=None):

keras/layers/noise.py

@@ -5,6 +5,7 @@ from ..engine import Layer
 from .. import backend as K
 import numpy as np
 from ..legacy import interfaces
+from ..engine import InputSpec
 
 
 class GaussianNoise(Layer):
@@ -90,3 +91,70 @@ class GaussianDropout(Layer):
         config = {'rate': self.rate}
         base_config = super(GaussianDropout, self).get_config()
         return dict(list(base_config.items()) + list(config.items()))
+
+
+class AlphaDropout(Layer):
+    """Applies Alpha Dropout to the input.
+
+    Alpha Dropout is a `Dropout` that keeps mean and variance of inputs
+    to their original values, in order to ensure the self-normalizing property
+    even after this dropout.
+    Alpha Dropout fits well to Scaled Exponential Linear Units
+    by randomly setting activations to the negative saturation value.
+
+    # Arguments
+        rate: float, drop probability (as with `Dropout`).
+            The multiplicative noise will have
+            standard deviation `sqrt(rate / (1 - rate))`.
+        seed: A Python integer to use as random seed.
+
+    # Input shape
+        Arbitrary. Use the keyword argument `input_shape`
+        (tuple of integers, does not include the samples axis)
+        when using this layer as the first layer in a model.
+
+    # Output shape
+        Same shape as input.
+
+    # References
+        - [Self-Normalizing Neural Networks](https://arxiv.org/abs/1706.02515)
+    """
+    def __init__(self, rate, noise_shape=None, seed=None, **kwargs):
+        super(AlphaDropout, self).__init__(**kwargs)
+        self.rate = rate
+        self.noise_shape = noise_shape
+        self.seed = seed
+        self.supports_masking = True
+
+    def _get_noise_shape(self, inputs):
+        return self.noise_shape if self.noise_shape else K.shape(inputs)
+
+    def call(self, inputs, training=None):
+        if 0. < self.rate < 1.:
+            noise_shape = self._get_noise_shape(inputs)
+
+            def dropped_inputs(inputs=inputs, rate=self.rate, seed=self.seed):
+                alpha = 1.6732632423543772848170429916717
+                scale = 1.0507009873554804934193349852946
+                alpha_p = -alpha * scale
+
+                kept_idx = K.greater_equal(K.random_uniform(noise_shape, seed=seed), rate)
+                kept_idx = K.cast(kept_idx, K.floatx())
+
+                # Get affine transformation params
+                a = ((1 - rate) * (1 + rate * alpha_p ** 2)) ** -0.5
+                b = -a * alpha_p * rate
+
+                # Apply mask
+                x = inputs * kept_idx + alpha_p * (1 - kept_idx)
+
+                # Do affine transformation
+                return a * x + b
+
+            return K.in_train_phase(dropped_inputs, inputs, training=training)
+        return inputs
+
+    def get_config(self):
+        config = {'rate': self.rate}
+        base_config = super(AlphaDropout, self).get_config()
+        return dict(list(base_config.items()) + list(config.items()))

keras/layers/normalization.py

@@ -137,7 +137,7 @@ class BatchNormalization(Layer):
 
         def normalize_inference():
             if needs_broadcasting:
-                # In this case we must explictly broadcast all parameters.
+                # In this case we must explicitly broadcast all parameters.
                 broadcast_moving_mean = K.reshape(self.moving_mean,
                                                   broadcast_shape)
                 broadcast_moving_variance = K.reshape(self.moving_variance,

keras/layers/recurrent.py

@@ -953,7 +953,7 @@ class LSTM(Recurrent):
             the linear transformation of the recurrent state.
 
     # References
-        - [Long short-term memory](http://deeplearning.cs.cmu.edu/pdfs/Hochreiter97_lstm.pdf) (original 1997 paper)
+        - [Long short-term memory](http://www.bioinf.jku.at/publications/older/2604.pdf) (original 1997 paper)
         - [Learning to forget: Continual prediction with LSTM](http://www.mitpressjournals.org/doi/pdf/10.1162/089976600300015015)
         - [Supervised sequence labeling with recurrent neural networks](http://www.cs.toronto.edu/~graves/preprint.pdf)
         - [A Theoretically Grounded Application of Dropout in Recurrent Neural Networks](http://arxiv.org/abs/1512.05287)

keras/layers/wrappers.py

@@ -2,9 +2,9 @@
 from __future__ import absolute_import
 
 import copy
-import inspect
 from ..engine import Layer
 from ..engine import InputSpec
+from ..utils.generic_utils import has_arg
 from .. import backend as K
 
 
@@ -84,12 +84,13 @@ class Wrapper(Layer):
     @classmethod
     def from_config(cls, config, custom_objects=None):
         from . import deserialize as deserialize_layer
-        layer = deserialize_layer(config.pop('layer'), custom_objects=custom_objects)
+        layer = deserialize_layer(config.pop('layer'),
+                                  custom_objects=custom_objects)
         return cls(layer, **config)
 
 
 class TimeDistributed(Wrapper):
-    """This wrapper allows to apply a layer to every temporal slice of an input.
+    """This wrapper applies a layer to every temporal slice of an input.
 
     The input should be at least 3D, and the dimension of index one
     will be considered to be the temporal dimension.
@@ -152,12 +153,21 @@ class TimeDistributed(Wrapper):
         timesteps = input_shape[1]
         return (child_output_shape[0], timesteps) + child_output_shape[1:]
 
-    def call(self, inputs, mask=None):
+    def call(self, inputs, training=None, mask=None):
+        kwargs = {}
+        if has_arg(self.layer.call, 'training'):
+            kwargs['training'] = training
+        uses_learning_phase = False
+
         input_shape = K.int_shape(inputs)
         if input_shape[0]:
             # batch size matters, use rnn-based implementation
             def step(x, _):
-                output = self.layer.call(x)
+                global uses_learning_phase
+                output = self.layer.call(x, **kwargs)
+                if hasattr(output, '_uses_learning_phase'):
+                    uses_learning_phase = (output._uses_learning_phase or
+                                           uses_learning_phase)
                 return output, []
 
             _, outputs, _ = K.rnn(step, inputs,
@@ -174,7 +184,10 @@ class TimeDistributed(Wrapper):
                 input_length = K.shape(inputs)[1]
             # Shape: (num_samples * timesteps, ...)
             inputs = K.reshape(inputs, (-1,) + input_shape[2:])
-            y = self.layer.call(inputs)  # (num_samples * timesteps, ...)
+            # (num_samples * timesteps, ...)
+            y = self.layer.call(inputs, **kwargs)
+            if hasattr(y, '_uses_learning_phase'):
+                uses_learning_phase = y._uses_learning_phase
             # Shape: (num_samples, timesteps, ...)
             output_shape = self.compute_output_shape(input_shape)
             y = K.reshape(y, (-1, input_length) + output_shape[2:])
@@ -184,6 +197,9 @@ class TimeDistributed(Wrapper):
            self.layer.activity_regularizer is not None):
             regularization_loss = self.layer.activity_regularizer(y)
             self.add_loss(regularization_loss, inputs)
+
+        if uses_learning_phase:
+            y._uses_learning_phase = True
         return y
 
 
@@ -205,7 +221,8 @@ class Bidirectional(Wrapper):
 
     ```python
         model = Sequential()
-        model.add(Bidirectional(LSTM(10, return_sequences=True), input_shape=(5, 10)))
+        model.add(Bidirectional(LSTM(10, return_sequences=True),
+                                input_shape=(5, 10)))
         model.add(Bidirectional(LSTM(10)))
         model.add(Dense(5))
         model.add(Activation('softmax'))
@@ -254,10 +271,9 @@ class Bidirectional(Wrapper):
 
     def call(self, inputs, training=None, mask=None):
         kwargs = {}
-        func_args = inspect.getargspec(self.layer.call).args
-        if 'training' in func_args:
+        if has_arg(self.layer.call, 'training'):
             kwargs['training'] = training
-        if 'mask' in func_args:
+        if has_arg(self.layer.call, 'mask'):
             kwargs['mask'] = mask
 
         y = self.forward_layer.call(inputs, **kwargs)

keras/legacy/interfaces.py

@@ -3,7 +3,6 @@
 import six
 import warnings
 import functools
-import inspect
 import numpy as np
 
 
@@ -86,7 +85,7 @@ def generate_legacy_interface(allowed_positional_args=None,
                 warnings.warn('Update your `' + object_name +
                               '` call to the Keras 2 API: ' + signature, stacklevel=2)
             return func(*args, **kwargs)
-        wrapper._legacy_support_signature = inspect.getargspec(func)
+        wrapper._original_function = func
         return wrapper
     return legacy_support
 
@@ -577,14 +576,21 @@ def generator_methods_args_preprocessor(args, kwargs):
             if hasattr(generator, 'batch_size'):
                 kwargs['steps_per_epoch'] = samples_per_epoch // generator.batch_size
             else:
-                warnings.warn('The semantics of the Keras 2 argument '
-                              ' `steps_per_epoch` is not the same as the '
-                              'Keras 1 argument `samples_per_epoch`. '
-                              '`steps_per_epoch` is the number of batches '
-                              'to draw from the generator at each epoch. '
-                              'Update your method calls accordingly.', stacklevel=3)
                 kwargs['steps_per_epoch'] = samples_per_epoch
             converted.append(('samples_per_epoch', 'steps_per_epoch'))
+
+    keras1_args = {'samples_per_epoch', 'val_samples', 'nb_epoch', 'nb_val_samples', 'nb_worker'}
+    if keras1_args.intersection(kwargs.keys()):
+        warnings.warn('The semantics of the Keras 2 argument '
+                      '`steps_per_epoch` is not the same as the '
+                      'Keras 1 argument `samples_per_epoch`. '
+                      '`steps_per_epoch` is the number of batches '
+                      'to draw from the generator at each epoch. '
+                      'Basically steps_per_epoch = samples_per_epoch/batch_size. '
+                      'Similarly `nb_val_samples`->`validation_steps` and '
+                      '`val_samples`->`steps` arguments have changed. '
+                      'Update your method calls accordingly.', stacklevel=3)
+
     return args, kwargs, converted
 
 
@@ -594,7 +600,9 @@ legacy_generator_methods_support = generate_legacy_method_interface(
                  ('val_samples', 'steps'),
                  ('nb_epoch', 'epochs'),
                  ('nb_val_samples', 'validation_steps'),
-                 ('nb_worker', 'workers')],
+                 ('nb_worker', 'workers'),
+                 ('pickle_safe', 'use_multiprocessing'),
+                 ('max_q_size', 'max_queue_size')],
     preprocessor=generator_methods_args_preprocessor)
 
 

keras/legacy/layers.py

@@ -1,10 +1,9 @@
-import inspect
 import types as python_types
 import warnings
 
 from ..engine.topology import Layer, InputSpec
 from .. import backend as K
-from ..utils.generic_utils import func_dump, func_load
+from ..utils.generic_utils import func_dump, func_load, has_arg
 from .. import regularizers
 from .. import constraints
 from .. import activations
@@ -197,8 +196,7 @@ class Merge(Layer):
         # Case: "mode" is a lambda or function.
         if callable(self.mode):
             arguments = self.arguments
-            arg_spec = inspect.getargspec(self.mode)
-            if 'mask' in arg_spec.args:
+            if has_arg(self.mode, 'mask'):
                 arguments['mask'] = mask
             return self.mode(inputs, **arguments)
 

keras/losses.py

@@ -46,15 +46,15 @@ def logcosh(y_true, y_pred):
 
 
 def categorical_crossentropy(y_true, y_pred):
-    return K.categorical_crossentropy(y_pred, y_true)
+    return K.categorical_crossentropy(y_true, y_pred)
 
 
 def sparse_categorical_crossentropy(y_true, y_pred):
-    return K.sparse_categorical_crossentropy(y_pred, y_true)
+    return K.sparse_categorical_crossentropy(y_true, y_pred)
 
 
 def binary_crossentropy(y_true, y_pred):
-    return K.mean(K.binary_crossentropy(y_pred, y_true), axis=-1)
+    return K.mean(K.binary_crossentropy(y_true, y_pred), axis=-1)
 
 
 def kullback_leibler_divergence(y_true, y_pred):

keras/models.py

@@ -207,32 +207,19 @@ def load_model(filepath, custom_objects=None, compile=True):
             obj: object, dict, or list.
 
         # Returns
-            The same structure, where occurences
+            The same structure, where occurrences
                 of a custom object name have been replaced
                 with the custom object.
         """
         if isinstance(obj, list):
             deserialized = []
             for value in obj:
-                if value in custom_objects:
-                    deserialized.append(custom_objects[value])
-                else:
-                    deserialized.append(value)
+                deserialized.append(convert_custom_objects(value))
             return deserialized
         if isinstance(obj, dict):
             deserialized = {}
             for key, value in obj.items():
-                deserialized[key] = []
-                if isinstance(value, list):
-                    for element in value:
-                        if element in custom_objects:
-                            deserialized[key].append(custom_objects[element])
-                        else:
-                            deserialized[key].append(element)
-                elif value in custom_objects:
-                    deserialized[key] = custom_objects[value]
-                else:
-                    deserialized[key] = value
+                deserialized[key] = convert_custom_objects(value)
             return deserialized
         if obj in custom_objects:
             return custom_objects[obj]
@@ -288,7 +275,13 @@ def load_model(filepath, custom_objects=None, compile=True):
                                       optimizer_weights_group.attrs['weight_names']]
             optimizer_weight_values = [optimizer_weights_group[n] for n in
                                        optimizer_weight_names]
-            model.optimizer.set_weights(optimizer_weight_values)
+            try:
+                model.optimizer.set_weights(optimizer_weight_values)
+            except ValueError:
+                warnings.warn('Error in loading the saved optimizer '
+                              'state. As a result, your model is '
+                              'starting with a freshly initialized '
+                              'optimizer.')
     return model
 
 
@@ -1033,9 +1026,9 @@ class Sequential(Model):
                       validation_data=None,
                       validation_steps=None,
                       class_weight=None,
-                      max_q_size=10,
+                      max_queue_size=10,
                       workers=1,
-                      pickle_safe=False,
+                      use_multiprocessing=False,
                       initial_epoch=0):
         """Fits the model on data generated batch-by-batch by a Python generator.
 
@@ -1072,9 +1065,9 @@ class Sequential(Model):
                 validation dataset divided by the batch size.
             class_weight: Dictionary mapping class indices to a weight
                 for the class.
-            max_q_size: Maximum size for the generator queue
+            max_queue_size: Maximum size for the generator queue
             workers: Maximum number of processes to spin up
-            pickle_safe: Ff True, use process based threading.
+            use_multiprocessing: if True, use process based threading.
                 Note that because
                 this implementation relies on multiprocessing,
                 you should not pass
@@ -1118,15 +1111,15 @@ class Sequential(Model):
                                         validation_data=validation_data,
                                         validation_steps=validation_steps,
                                         class_weight=class_weight,
-                                        max_q_size=max_q_size,
+                                        max_queue_size=max_queue_size,
                                         workers=workers,
-                                        pickle_safe=pickle_safe,
+                                        use_multiprocessing=use_multiprocessing,
                                         initial_epoch=initial_epoch)
 
     @interfaces.legacy_generator_methods_support
     def evaluate_generator(self, generator, steps,
-                           max_q_size=10, workers=1,
-                           pickle_safe=False):
+                           max_queue_size=10, workers=1,
+                           use_multiprocessing=False):
         """Evaluates the model on a data generator.
 
         The generator should return the same kind of data
@@ -1137,9 +1130,9 @@ class Sequential(Model):
                 or (inputs, targets, sample_weights)
             steps: Total number of steps (batches of samples)
                 to yield from `generator` before stopping.
-            max_q_size: maximum size for the generator queue
+            max_queue_size: maximum size for the generator queue
             workers: maximum number of processes to spin up
-            pickle_safe: if True, use process based threading.
+            use_multiprocessing: if True, use process based threading.
                 Note that because this implementation
                 relies on multiprocessing, you should not pass
                 non picklable arguments to the generator
@@ -1159,14 +1152,14 @@ class Sequential(Model):
                                'before being used.')
         return self.model.evaluate_generator(generator,
                                              steps,
-                                             max_q_size=max_q_size,
+                                             max_queue_size=max_queue_size,
                                              workers=workers,
-                                             pickle_safe=pickle_safe)
+                                             use_multiprocessing=use_multiprocessing)
 
     @interfaces.legacy_generator_methods_support
     def predict_generator(self, generator, steps,
-                          max_q_size=10, workers=1,
-                          pickle_safe=False, verbose=0):
+                          max_queue_size=10, workers=1,
+                          use_multiprocessing=False, verbose=0):
         """Generates predictions for the input samples from a data generator.
 
         The generator should return the same kind of data as accepted by
@@ -1176,9 +1169,9 @@ class Sequential(Model):
             generator: generator yielding batches of input samples.
             steps: Total number of steps (batches of samples)
                 to yield from `generator` before stopping.
-            max_q_size: maximum size for the generator queue
+            max_queue_size: maximum size for the generator queue
             workers: maximum number of processes to spin up
-            pickle_safe: if True, use process based threading.
+            use_multiprocessing: if True, use process based threading.
                 Note that because this implementation
                 relies on multiprocessing, you should not pass
                 non picklable arguments to the generator
@@ -1191,9 +1184,9 @@ class Sequential(Model):
         if self.model is None:
             self.build()
         return self.model.predict_generator(generator, steps,
-                                            max_q_size=max_q_size,
+                                            max_queue_size=max_queue_size,
                                             workers=workers,
-                                            pickle_safe=pickle_safe,
+                                            use_multiprocessing=use_multiprocessing,
                                             verbose=verbose)
 
     def get_config(self):

keras/optimizers.py

@@ -219,8 +219,7 @@ class RMSprop(Optimizer):
 
     def get_updates(self, params, constraints, loss):
         grads = self.get_gradients(loss, params)
-        shapes = [K.get_variable_shape(p) for p in params]
-        accumulators = [K.zeros(shape) for shape in shapes]
+        accumulators = [K.zeros(K.get_variable_shape(p), dtype=K.dtype(p)) for p in params]
         self.weights = accumulators
         self.updates = []
 
@@ -413,9 +412,8 @@ class Adam(Optimizer):
         lr_t = lr * (K.sqrt(1. - K.pow(self.beta_2, t)) /
                      (1. - K.pow(self.beta_1, t)))
 
-        shapes = [K.get_variable_shape(p) for p in params]
-        ms = [K.zeros(shape) for shape in shapes]
-        vs = [K.zeros(shape) for shape in shapes]
+        ms = [K.zeros(K.get_variable_shape(p), dtype=K.dtype(p)) for p in params]
+        vs = [K.zeros(K.get_variable_shape(p), dtype=K.dtype(p)) for p in params]
         self.weights = [self.iterations] + ms + vs
 
         for p, g, m, v in zip(params, grads, ms, vs):

keras/preprocessing/image.py

@@ -139,7 +139,7 @@ def random_zoom(x, zoom_range, row_axis=1, col_axis=2, channel_axis=0,
         ValueError: if `zoom_range` isn't a tuple.
     """
     if len(zoom_range) != 2:
-        raise ValueError('zoom_range should be a tuple or list of two floats. '
+        raise ValueError('`zoom_range` should be a tuple or list of two floats. '
                          'Received arg: ', zoom_range)
 
     if zoom_range[0] == 1 and zoom_range[1] == 1:
@@ -421,8 +421,8 @@ class ImageDataGenerator(object):
         self.preprocessing_function = preprocessing_function
 
         if data_format not in {'channels_last', 'channels_first'}:
-            raise ValueError('data_format should be "channels_last" (channel after row and '
-                             'column) or "channels_first" (channel before row and column). '
+            raise ValueError('`data_format` should be `"channels_last"` (channel after row and '
+                             'column) or `"channels_first"` (channel before row and column). '
                              'Received arg: ', data_format)
         self.data_format = data_format
         if data_format == 'channels_first':
@@ -443,7 +443,7 @@ class ImageDataGenerator(object):
         elif len(zoom_range) == 2:
             self.zoom_range = [zoom_range[0], zoom_range[1]]
         else:
-            raise ValueError('zoom_range should be a float or '
+            raise ValueError('`zoom_range` should be a float or '
                              'a tuple or list of two floats. '
                              'Received arg: ', zoom_range)
 

keras/preprocessing/sequence.py

@@ -104,7 +104,7 @@ def make_sampling_table(size, sampling_factor=1e-5):
         is the probability that a word of rank i should be sampled.
     """
     gamma = 0.577
-    rank = np.array(list(range(size)))
+    rank = np.arange(size)
     rank[0] = 1
     inv_fq = rank * (np.log(rank) + gamma) + 0.5 - 1. / (12. * rank)
     f = sampling_factor * inv_fq
@@ -127,7 +127,7 @@ def skipgrams(sequence, vocabulary_size,
             of word indices (integers). If using a `sampling_table`,
             word indices are expected to match the rank
             of the words in a reference dataset (e.g. 10 would encode
-            the 10-th most frequently occuring token).
+            the 10-th most frequently occurring token).
             Note that index 0 is expected to be a non-word and will be skipped.
         vocabulary_size: int. maximum possible word index + 1
         window_size: int. actually half-window.
@@ -191,3 +191,22 @@ def skipgrams(sequence, vocabulary_size,
         random.shuffle(labels)
 
     return couples, labels
+
+
+def _remove_long_seq(maxlen, seq, label):
+    """Removes sequences that exceed the maximum length.
+
+    # Arguments
+        maxlen: int, maximum length
+        seq: list of lists where each sublist is a sequence
+        label: list where each element is an integer
+
+    # Returns
+        new_seq, new_label: shortened lists for `seq` and `label`.
+    """
+    new_seq, new_label = [], []
+    for x, y in zip(seq, label):
+        if len(x) < maxlen:
+            new_seq.append(x)
+            new_label.append(y)
+    return new_seq, new_label

keras/preprocessing/text.py

@@ -8,11 +8,13 @@ from __future__ import division
 
 import string
 import sys
+import warnings
+from collections import OrderedDict
+from hashlib import md5
+
 import numpy as np
 from six.moves import range
 from six.moves import zip
-from collections import OrderedDict
-import warnings
 
 if sys.version_info < (3,):
     maketrans = string.maketrans
@@ -45,11 +47,58 @@ def one_hot(text, n,
             filters='!"#$%&()*+,-./:;<=>?@[\\]^_`{|}~\t\n',
             lower=True,
             split=' '):
+    """One-hot encodes a text into a list of word indexes of size n.
+
+    This is a wrapper to the `hashing_trick` function using `hash` as the
+    hashing function, unicity of word to index mapping non-guaranteed.
+    """
+    return hashing_trick(text, n,
+                         hash_function=hash,
+                         filters=filters,
+                         lower=lower,
+                         split=split)
+
+
+def hashing_trick(text, n,
+                  hash_function=None,
+                  filters='!"#$%&()*+,-./:;<=>?@[\\]^_`{|}~\t\n',
+                  lower=True,
+                  split=' '):
+    """Converts a text to a sequence of indexes in a fixed-size hashing space.
+
+    # Arguments
+        text: Input text (string).
+        n: Dimension of the hashing space.
+        hash_function: if `None` uses python `hash` function, can be 'md5' or
+            any function that takes in input a string and returns a int.
+            Note that `hash` is not a stable hashing function, so
+            it is not consistent across different runs, while 'md5'
+            is a stable hashing function.
+        filters: Sequence of characters to filter out.
+        lower: Whether to convert the input to lowercase.
+        split: Sentence split marker (string).
+
+    # Returns
+        A list of integer word indices (unicity non-guaranteed).
+
+    `0` is a reserved index that won't be assigned to any word.
+
+    Two or more words may be assigned to the same index, due to possible
+    collisions by the hashing function.
+    The [probability](https://en.wikipedia.org/wiki/Birthday_problem#Probability_table)
+    of a collision is in relation to the dimension of the hashing space and
+    the number of distinct objects.
+    """
+    if hash_function is None:
+        hash_function = hash
+    elif hash_function == 'md5':
+        hash_function = lambda w: int(md5(w.encode()).hexdigest(), 16)
+
     seq = text_to_word_sequence(text,
                                 filters=filters,
                                 lower=lower,
                                 split=split)
-    return [(abs(hash(w)) % (n - 1) + 1) for w in seq]
+    return [(hash_function(w) % (n - 1) + 1) for w in seq]
 
 
 class Tokenizer(object):

keras/utils/__init__.py

@@ -1,13 +1,16 @@
 from __future__ import absolute_import
 from . import np_utils
-from . import conv_utils
-from . import data_utils
 from . import generic_utils
+from . import data_utils
 from . import io_utils
+from . import conv_utils
 
 # Globally-importable utils.
 from .io_utils import HDF5Matrix
 from .data_utils import get_file
+from .data_utils import Sequence
+from .data_utils import GeneratorEnqueuer
+from .data_utils import OrderedEnqueuer
 from .generic_utils import CustomObjectScope
 from .generic_utils import custom_object_scope
 from .generic_utils import get_custom_objects

keras/utils/data_utils.py

@@ -2,20 +2,32 @@
 from __future__ import absolute_import
 from __future__ import print_function
 
-import tarfile
-import zipfile
-import os
-import sys
-import shutil
 import hashlib
+import multiprocessing
+import os
+import random
+import shutil
+import sys
+import tarfile
+import threading
+import time
+import zipfile
+from abc import abstractmethod
+from multiprocessing.pool import ThreadPool
+
+import numpy as np
 import six
-from six.moves.urllib.request import urlopen
-from six.moves.urllib.error import URLError
 from six.moves.urllib.error import HTTPError
+from six.moves.urllib.error import URLError
+from six.moves.urllib.request import urlopen
+
+try:
+    import queue
+except ImportError:
+    import Queue as queue
 
 from ..utils.generic_utils import Progbar
 
-
 if sys.version_info[0] == 2:
     def urlretrieve(url, filename, reporthook=None, data=None):
         """Replacement for `urlretrive` for Python 2.
@@ -34,6 +46,7 @@ if sys.version_info[0] == 2:
                 a block size in bytes, and the total size of the file.
             data: `data` argument passed to `urlopen`.
         """
+
         def chunk_read(response, chunk_size=8192, reporthook=None):
             content_type = response.info().get('Content-Length')
             total_size = -1
@@ -282,3 +295,342 @@ def validate_file(fpath, file_hash, algorithm='auto', chunk_size=65535):
         return True
     else:
         return False
+
+
+class Sequence(object):
+    """Base object for fitting to a sequence of data, such as a dataset.
+
+    Every `Sequence` must implements the `__getitem__` and the `__len__` methods.
+
+    # Examples
+
+    ```python
+    from skimage.io import imread
+    from skimage.transform import resize
+    import numpy as np
+
+    # Here, `x_set` is list of path to the images
+    # and `y_set` are the associated classes.
+
+    class CIFAR10Sequence(Sequence):
+        def __init__(self, x_set, y_set, batch_size):
+            self.X,self.y = x_set,y_set
+            self.batch_size = batch_size
+
+        def __len__(self):
+            return len(self.X) // self.batch_size
+
+        def __getitem__(self,idx):
+            batch_x = self.X[idx*self.batch_size:(idx+1)*self.batch_size]
+            batch_y = self.y[idx*self.batch_size:(idx+1)*self.batch_size]
+
+            return np.array([
+                resize(imread(file_name), (200,200))
+                   for file_name in batch_x]), np.array(batch_y)
+    ```
+    """
+
+    @abstractmethod
+    def __getitem__(self, index):
+        """Gets batch at position `index`.
+
+        # Arguments
+            index: position of the batch in the Sequence.
+
+        # Returns
+            A batch
+        """
+        raise NotImplementedError
+
+    @abstractmethod
+    def __len__(self):
+        """Number of batch in the Sequence.
+
+        # Returns
+            The number of batches in the Sequence.
+        """
+        raise NotImplementedError
+
+
+def get_index(ds, i):
+    """Quick fix for Python2, otherwise, it cannot be pickled.
+
+    # Arguments
+        ds: a Sequence object
+        i: index
+
+    # Returns
+        The value at index `i`.
+    """
+    return ds[i]
+
+
+class SequenceEnqueuer(object):
+    """Base class to enqueue inputs.
+
+    The task of an Enqueuer is to use parallelism to speed up preprocessing.
+    This is done with processes or threads.
+
+    # Examples
+
+    ```python
+    enqueuer = SequenceEnqueuer(...)
+    enqueuer.start()
+    datas = enqueuer.get()
+    for data in datas:
+        # Use the inputs; training, evaluating, predicting.
+        # ... stop sometime.
+    enqueuer.close()
+    ```
+
+    The `enqueuer.get()` should be an infinite stream of datas.
+
+    """
+
+    @abstractmethod
+    def is_running(self):
+        raise NotImplementedError
+
+    @abstractmethod
+    def start(self, workers=1, max_queue_size=10):
+        """Starts the handler's workers.
+
+        # Arguments
+            workers: number of worker threads
+            max_queue_size: queue size
+                (when full, threads could block on `put()`).
+        """
+        raise NotImplementedError
+
+    @abstractmethod
+    def stop(self, timeout=None):
+        """Stop running threads and wait for them to exit, if necessary.
+
+        Should be called by the same thread which called start().
+
+        # Arguments
+            timeout: maximum time to wait on thread.join()
+        """
+        raise NotImplementedError
+
+    @abstractmethod
+    def get(self):
+        """Creates a generator to extract data from the queue.
+
+        Skip the data if it is `None`.
+
+        # Returns
+            Generator yielding tuples `(inputs, targets)`
+                or `(inputs, targets, sample_weights)`.
+        """
+        raise NotImplementedError
+
+
+class OrderedEnqueuer(SequenceEnqueuer):
+    """Builds a Enqueuer from a Sequence.
+
+    Used in `fit_generator`, `evaluate_generator`, `predict_generator`.
+
+    # Arguments
+        sequence: A `keras.utils.data_utils.Sequence` object.
+        use_multiprocessing: use multiprocessing if True, otherwise threading
+        scheduling: Sequential querying of datas if 'sequential', random otherwise.
+    """
+
+    def __init__(self, sequence,
+                 use_multiprocessing=False,
+                 scheduling='sequential'):
+        self.sequence = sequence
+        self.use_multiprocessing = use_multiprocessing
+        self.scheduling = scheduling
+        self.workers = 0
+        self.executor = None
+        self.queue = None
+        self.run_thread = None
+        self.stop_signal = None
+
+    def is_running(self):
+        return self.stop_signal is not None and not self.stop_signal.is_set()
+
+    def start(self, workers=1, max_queue_size=10):
+        """Start the handler's workers.
+
+        # Arguments
+            workers: number of worker threads
+            max_queue_size: queue size
+                (when full, workers could block on `put()`)
+        """
+        if self.use_multiprocessing:
+            self.executor = multiprocessing.Pool(workers)
+        else:
+            self.executor = ThreadPool(workers)
+        self.queue = queue.Queue(max_queue_size)
+        self.stop_signal = threading.Event()
+        self.run_thread = threading.Thread(target=self._run)
+        self.run_thread.daemon = True
+        self.run_thread.start()
+
+    def _run(self):
+        """Function to submit request to the executor and queue the `Future` objects."""
+        sequence = list(range(len(self.sequence)))
+        while True:
+            if self.scheduling is not 'sequential':
+                random.shuffle(sequence)
+            for i in sequence:
+                if self.stop_signal.is_set():
+                    return
+                self.queue.put(
+                    self.executor.apply_async(get_index,
+                                              (self.sequence, i)), block=True)
+
+    def get(self):
+        """Creates a generator to extract data from the queue.
+
+        Skip the data if it is `None`.
+
+        # Returns
+            Generator yielding tuples (inputs, targets)
+                or (inputs, targets, sample_weights)
+        """
+        try:
+            while self.is_running():
+                inputs = self.queue.get(block=True).get()
+                if inputs is not None:
+                    yield inputs
+        except Exception as e:
+            self.stop()
+            raise StopIteration(e)
+
+    def stop(self, timeout=None):
+        """Stops running threads and wait for them to exit, if necessary.
+
+        Should be called by the same thread which called `start()`.
+
+        # Arguments
+            timeout: maximum time to wait on `thread.join()`
+        """
+        self.stop_signal.set()
+        with self.queue.mutex:
+            self.queue.queue.clear()
+            self.queue.unfinished_tasks = 0
+            self.queue.not_full.notify()
+        self.executor.close()
+        self.executor.join()
+        self.run_thread.join(timeout)
+
+
+class GeneratorEnqueuer(SequenceEnqueuer):
+    """Builds a queue out of a data generator.
+
+    Used in `fit_generator`, `evaluate_generator`, `predict_generator`.
+
+    # Arguments
+        generator: a generator function which endlessly yields data
+        use_multiprocessing: use multiprocessing if True, otherwise threading
+        wait_time: time to sleep in-between calls to `put()`
+        random_seed: Initial seed for workers,
+            will be incremented by one for each workers.
+    """
+
+    def __init__(self, generator,
+                 use_multiprocessing=False,
+                 wait_time=0.05,
+                 random_seed=None):
+        self.wait_time = wait_time
+        self._generator = generator
+        self._use_multiprocessing = use_multiprocessing
+        self._threads = []
+        self._stop_event = None
+        self.queue = None
+        self.random_seed = random_seed
+
+    def start(self, workers=1, max_queue_size=10):
+        """Kicks off threads which add data from the generator into the queue.
+
+        # Arguments
+            workers: number of worker threads
+            max_queue_size: queue size
+                (when full, threads could block on `put()`)
+        """
+
+        def data_generator_task():
+            while not self._stop_event.is_set():
+                try:
+                    if self._use_multiprocessing or self.queue.qsize() < max_queue_size:
+                        generator_output = next(self._generator)
+                        self.queue.put(generator_output)
+                    else:
+                        time.sleep(self.wait_time)
+                except Exception:
+                    self._stop_event.set()
+                    raise
+
+        try:
+            if self._use_multiprocessing:
+                self.queue = multiprocessing.Queue(maxsize=max_queue_size)
+                self._stop_event = multiprocessing.Event()
+            else:
+                self.queue = queue.Queue()
+                self._stop_event = threading.Event()
+
+            for _ in range(workers):
+                if self._use_multiprocessing:
+                    # Reset random seed else all children processes
+                    # share the same seed
+                    np.random.seed(self.random_seed)
+                    thread = multiprocessing.Process(target=data_generator_task)
+                    thread.daemon = True
+                    if self.random_seed is not None:
+                        self.random_seed += 1
+                else:
+                    thread = threading.Thread(target=data_generator_task)
+                self._threads.append(thread)
+                thread.start()
+        except:
+            self.stop()
+            raise
+
+    def is_running(self):
+        return self._stop_event is not None and not self._stop_event.is_set()
+
+    def stop(self, timeout=None):
+        """Stops running threads and wait for them to exit, if necessary.
+
+        Should be called by the same thread which called `start()`.
+
+        # Arguments
+            timeout: maximum time to wait on `thread.join()`.
+        """
+        if self.is_running():
+            self._stop_event.set()
+
+        for thread in self._threads:
+            if thread.is_alive():
+                if self._use_multiprocessing:
+                    thread.terminate()
+                else:
+                    thread.join(timeout)
+
+        if self._use_multiprocessing:
+            if self.queue is not None:
+                self.queue.close()
+
+        self._threads = []
+        self._stop_event = None
+        self.queue = None
+
+    def get(self):
+        """Creates a generator to extract data from the queue.
+
+        Skip the data if it is `None`.
+
+        # Returns
+            A generator
+        """
+        while self.is_running():
+            if not self.queue.empty():
+                inputs = self.queue.get()
+                if inputs is not None:
+                    yield inputs
+            else:
+                time.sleep(self.wait_time)

keras/utils/generic_utils.py

@@ -132,9 +132,8 @@ def deserialize_keras_object(identifier, module_objects=None,
                 raise ValueError('Unknown ' + printable_module_name +
                                  ': ' + class_name)
         if hasattr(cls, 'from_config'):
-            arg_spec = inspect.getargspec(cls.from_config)
             custom_objects = custom_objects or {}
-            if 'custom_objects' in arg_spec.args:
+            if has_arg(cls.from_config, 'custom_objects'):
                 return cls.from_config(config['config'],
                                        custom_objects=dict(list(_GLOBAL_CUSTOM_OBJECTS.items()) +
                                                            list(custom_objects.items())))
@@ -207,6 +206,48 @@ def func_load(code, defaults=None, closure=None, globs=None):
                                      closure=closure)
 
 
+def has_arg(fn, name, accept_all=False):
+    """Checks if a callable accepts a given keyword argument.
+
+    For Python 2, checks if there is an argument with the given name.
+
+    For Python 3, checks if there is an argument with the given name, and
+    also whether this argument can be called with a keyword (i.e. if it is
+    not a positional-only argument).
+
+    # Arguments
+        fn: Callable to inspect.
+        name: Check if `fn` can be called with `name` as a keyword argument.
+        accept_all: What to return if there is no parameter called `name`
+                    but the function accepts a `**kwargs` argument.
+
+    # Returns
+        bool, whether `fn` accepts a `name` keyword argument.
+    """
+    if sys.version_info < (3,):
+        arg_spec = inspect.getargspec(fn)
+        if accept_all and arg_spec.keywords is not None:
+            return True
+        return (name in arg_spec.args)
+    elif sys.version_info < (3, 3):
+        arg_spec = inspect.getfullargspec(fn)
+        if accept_all and arg_spec.varkw is not None:
+            return True
+        return (name in arg_spec.args or
+                name in arg_spec.kwonlyargs)
+    else:
+        signature = inspect.signature(fn)
+        parameter = signature.parameters.get(name)
+        if parameter is None:
+            if accept_all:
+                for param in signature.parameters.values():
+                    if param.kind == inspect.Parameter.VAR_KEYWORD:
+                        return True
+            return False
+        return (parameter.kind in (inspect.Parameter.POSITIONAL_OR_KEYWORD,
+                                   inspect.Parameter.KEYWORD_ONLY))
+
+
 class Progbar(object):
     """Displays a progress bar.
 

keras/utils/io_utils.py

@@ -96,8 +96,40 @@ class HDF5Matrix(object):
 
     @property
     def shape(self):
+        """Gets a numpy-style shape tuple giving the dataset dimensions.
+
+        # Returns
+            A numpy-style shape tuple.
+        """
         return (self.end - self.start,) + self.data.shape[1:]
 
+    @property
+    def dtype(self):
+        """Gets the datatype of the dataset.
+
+        # Returns
+            A numpy dtype string.
+        """
+        return self.data.dtype
+
+    @property
+    def ndim(self):
+        """Gets the number of dimensions (rank) of the dataset.
+
+        # Returns
+            An integer denoting the number of dimensions (rank) of the dataset.
+        """
+        return self.data.ndim
+
+    @property
+    def size(self):
+        """Gets the total dataset size (number of elements).
+
+        # Returns
+            An integer denoting the number of elements in the dataset.
+        """
+        return np.prod(self.shape)
+
 
 def ask_to_proceed_with_overwrite(filepath):
     """Produces a prompt asking about overwriting a file.

keras/utils/layer_utils.py

@@ -5,14 +5,20 @@ from .. import backend as K
 import numpy as np
 
 
-def print_summary(model, line_length=None, positions=None):
+def print_summary(model, line_length=None, positions=None, print_fn=print):
     """Prints a summary of a model.
 
     # Arguments
         model: Keras model instance.
-        line_length: total length of printed lines
-        positions: relative or absolute positions of log elements in each line.
+        line_length: Total length of printed lines
+            (e.g. set this to adapt the display to different
+            terminal window sizes).
+        positions: Relative or absolute positions of log elements in each line.
             If not provided, defaults to `[.33, .55, .67, 1.]`.
+        print_fn: Print function to use.
+            It will be called on each line of the summary.
+            You can set it to a custom function
+            in order to capture the string summary.
     """
     if model.__class__.__name__ == 'Sequential':
         sequential_like = True
@@ -51,11 +57,11 @@ def print_summary(model, line_length=None, positions=None):
             line += str(fields[i])
             line = line[:positions[i]]
             line += ' ' * (positions[i] - len(line))
-        print(line)
+        print_fn(line)
 
-    print('_' * line_length)
+    print_fn('_' * line_length)
     print_row(to_display, positions)
-    print('=' * line_length)
+    print_fn('=' * line_length)
 
     def print_layer_summary(layer):
         try:
@@ -108,19 +114,19 @@ def print_summary(model, line_length=None, positions=None):
         else:
             print_layer_summary_with_connections(layers[i])
         if i == len(layers) - 1:
-            print('=' * line_length)
+            print_fn('=' * line_length)
         else:
-            print('_' * line_length)
+            print_fn('_' * line_length)
 
     trainable_count = int(
         np.sum([K.count_params(p) for p in set(model.trainable_weights)]))
     non_trainable_count = int(
         np.sum([K.count_params(p) for p in set(model.non_trainable_weights)]))
 
-    print('Total params: {:,}'.format(trainable_count + non_trainable_count))
-    print('Trainable params: {:,}'.format(trainable_count))
-    print('Non-trainable params: {:,}'.format(non_trainable_count))
-    print('_' * line_length)
+    print_fn('Total params: {:,}'.format(trainable_count + non_trainable_count))
+    print_fn('Trainable params: {:,}'.format(trainable_count))
+    print_fn('Non-trainable params: {:,}'.format(non_trainable_count))
+    print_fn('_' * line_length)
 
 
 def convert_all_kernels_in_model(model):

keras/utils/test_utils.py

@@ -1,9 +1,9 @@
 """Utilities related to Keras unit tests."""
 import numpy as np
 from numpy.testing import assert_allclose
-import inspect
 import six
 
+from .generic_utils import has_arg
 from ..engine import Model, Input
 from ..models import Sequential
 from ..models import model_from_json
@@ -71,7 +71,9 @@ def layer_test(layer_cls, kwargs={}, input_shape=None, input_dtype=None,
     layer.set_weights(weights)
 
     # test and instantiation from weights
-    if 'weights' in inspect.getargspec(layer_cls.__init__):
+    # Checking for empty weights array to avoid a problem where some
+    # legacy layers return bad values from get_weights()
+    if has_arg(layer_cls.__init__, 'weights') and len(weights):
         kwargs['weights'] = weights
         layer = layer_cls(**kwargs)
 

keras/wrappers/scikit_learn.py

@@ -1,12 +1,12 @@
 from __future__ import absolute_import
 
 import copy
-import inspect
 import types
 
 import numpy as np
 
 from ..utils.np_utils import to_categorical
+from ..utils.generic_utils import has_arg
 from ..models import Sequential
 
 
@@ -75,13 +75,11 @@ class BaseWrapper(object):
         else:
             legal_params_fns.append(self.build_fn)
 
-        legal_params = []
-        for fn in legal_params_fns:
-            legal_params += inspect.getargspec(fn)[0]
-        legal_params = set(legal_params)
-
         for params_name in params:
-            if params_name not in legal_params:
+            for fn in legal_params_fns:
+                if has_arg(fn, params_name):
+                    break
+            else:
                 if params_name != 'nb_epoch':
                     raise ValueError(
                         '{} is not a legal parameter'.format(params_name))
@@ -163,9 +161,8 @@ class BaseWrapper(object):
         """
         override = override or {}
         res = {}
-        fn_args = inspect.getargspec(fn)[0]
         for name, value in self.sk_params.items():
-            if name in fn_args:
+            if has_arg(fn, name):
                 res.update({name: value})
         res.update(override)
         return res

tests/keras/activations_test.py

@@ -15,7 +15,7 @@ def get_standard_values():
 def test_serialization():
     all_activations = ['softmax', 'relu', 'elu', 'tanh',
                        'sigmoid', 'hard_sigmoid', 'linear',
-                       'softplus', 'softsign']
+                       'softplus', 'softsign', 'selu']
     for name in all_activations:
         fn = activations.get(name)
         ref_fn = getattr(activations, name)
@@ -146,6 +146,29 @@ def test_elu():
     assert_allclose(result, true_result)
 
 
+def test_selu():
+    x = K.placeholder(ndim=2)
+    f = K.function([x], [activations.selu(x)])
+    alpha = 1.6732632423543772848170429916717
+    scale = 1.0507009873554804934193349852946
+
+    positive_values = get_standard_values()
+    result = f([positive_values])[0]
+    assert_allclose(result, positive_values * scale, rtol=1e-05)
+
+    negative_values = np.array([[-1, -2]], dtype=K.floatx())
+
+    # cntk can't rebind the input shape, so create the model again to test different batch size
+    if (K.backend() == 'cntk'):
+        x2 = K.placeholder(ndim=2)
+        f = K.function([x2], [activations.selu(x2)])
+
+    result = f([negative_values])[0]
+    true_result = (np.exp(negative_values) - 1) * scale * alpha
+
+    assert_allclose(result, true_result)
+
+
 def test_tanh():
     test_values = get_standard_values()
 

tests/keras/applications/applications_test.py

@@ -1,5 +1,8 @@
 import pytest
 from keras.utils.test_utils import keras_test
+from keras.utils.test_utils import layer_test
+from keras.utils.generic_utils import CustomObjectScope
+from keras.models import Sequential
 from keras import applications
 from keras import backend as K
 
@@ -166,5 +169,73 @@ def test_inceptionv3_pooling():
     assert model.output_shape == (None, 2048)
 
 
+@keras_test
+@pytest.mark.skipif((K.backend() != 'tensorflow'),
+                    reason="MobileNets are supported only on Tensorflow")
+def test_mobilenet():
+    model = applications.MobileNet(weights=None)
+    assert model.output_shape == (None, 1000)
+
+
+@keras_test
+@pytest.mark.skipif((K.backend() != 'tensorflow'),
+                    reason="MobileNets are supported only on Tensorflow")
+def test_mobilenet_no_top():
+    model = applications.MobileNet(weights=None, include_top=False)
+    assert model.output_shape == (None, None, None, 1024)
+
+
+@keras_test
+@pytest.mark.skipif((K.backend() != 'tensorflow'),
+                    reason="MobileNets are supported only on Tensorflow")
+def test_mobilenet_pooling():
+    model = applications.MobileNet(weights=None, include_top=False, pooling='avg')
+    assert model.output_shape == (None, 1024)
+
+
+@pytest.mark.skipif(K.backend() != 'tensorflow', reason='Requires TF backend')
+@keras_test
+def test_depthwise_conv_2d():
+    _convolution_paddings = ['valid', 'same']
+    num_samples = 2
+    stack_size = 3
+    num_row = 7
+    num_col = 6
+
+    with CustomObjectScope({'relu6': applications.mobilenet.relu6,
+                            'DepthwiseConv2D': applications.mobilenet.DepthwiseConv2D}):
+        for padding in _convolution_paddings:
+            for strides in [(1, 1), (2, 2)]:
+                for multiplier in [1, 2]:
+                    if padding == 'same' and strides != (1, 1):
+                        continue
+
+                    layer_test(applications.mobilenet.DepthwiseConv2D,
+                               kwargs={'kernel_size': (3, 3),
+                                       'padding': padding,
+                                       'strides': strides,
+                                       'depth_multiplier': multiplier},
+                               input_shape=(num_samples, num_row, num_col, stack_size))
+
+        layer_test(applications.mobilenet.DepthwiseConv2D,
+                   kwargs={'kernel_size': 3,
+                           'padding': padding,
+                           'data_format': 'channels_first',
+                           'activation': None,
+                           'depthwise_regularizer': 'l2',
+                           'bias_regularizer': 'l2',
+                           'activity_regularizer': 'l2',
+                           'depthwise_constraint': 'unit_norm',
+                           'strides': strides,
+                           'depth_multiplier': multiplier},
+                   input_shape=(num_samples, stack_size, num_row, num_col))
+
+        # Test invalid use case
+        with pytest.raises(ValueError):
+            model = Sequential([applications.mobilenet.DepthwiseConv2D(kernel_size=3,
+                                                                       padding=padding,
+                                                                       batch_input_shape=(None, None, 5, None))])
+
+
 if __name__ == '__main__':
     pytest.main([__file__])

tests/keras/applications/imagenet_utils_test.py

@@ -0,0 +1,112 @@
+import pytest
+import numpy as np
+from numpy.testing import assert_allclose
+
+from keras.applications import imagenet_utils as utils
+
+
+def test_preprocess_input():
+    x = np.random.uniform(0, 255, (2, 3, 2, 3))
+    assert utils.preprocess_input(x).shape == x.shape
+
+    out1 = utils.preprocess_input(x, 'channels_last')
+    out2 = utils.preprocess_input(np.transpose(x, (0, 3, 1, 2)), 'channels_first')
+    assert_allclose(out1, out2.transpose(0, 2, 3, 1))
+
+
+def test_decode_predictions():
+    x = np.zeros((2, 1000))
+    x[0, 372] = 1.0
+    x[1, 549] = 1.0
+    outs = utils.decode_predictions(x, top=1)
+    scores = [out[0][2] for out in outs]
+    assert scores[0] == scores[1]
+
+    # the numbers of columns and ImageNet classes are not identical.
+    with pytest.raises(ValueError):
+        utils.decode_predictions(np.ones((2, 100)))
+
+
+def test_obtain_input_shape():
+    # input_shape and default_size are not identical.
+    with pytest.raises(ValueError):
+        utils._obtain_input_shape(
+            input_shape=(224, 224, 3),
+            default_size=299,
+            min_size=139,
+            data_format='channels_last',
+            include_top=True)
+
+    # Test invalid use cases
+    for data_format in ['channels_last', 'channels_first']:
+        # input_shape is smaller than min_size.
+        shape = (100, 100)
+        input_shape = shape + (3,) if data_format == 'channels_last' else (3,) + shape
+        with pytest.raises(ValueError):
+            utils._obtain_input_shape(
+                input_shape=input_shape,
+                default_size=None,
+                min_size=139,
+                data_format=data_format,
+                include_top=False)
+
+        # shape is 1D.
+        shape = (100,)
+        input_shape = shape + (3,) if data_format == 'channels_last' else (3,) + shape
+        with pytest.raises(ValueError):
+            utils._obtain_input_shape(
+                input_shape=input_shape,
+                default_size=None,
+                min_size=139,
+                data_format=data_format,
+                include_top=False)
+
+        # the number of channels is 5 not 3.
+        shape = (100, 100)
+        input_shape = shape + (5,) if data_format == 'channels_last' else (5,) + shape
+        with pytest.raises(ValueError):
+            utils._obtain_input_shape(
+                input_shape=input_shape,
+                default_size=None,
+                min_size=139,
+                data_format=data_format,
+                include_top=False)
+
+    assert utils._obtain_input_shape(
+        input_shape=None,
+        default_size=None,
+        min_size=139,
+        data_format='channels_last',
+        include_top=False) == (None, None, 3)
+
+    assert utils._obtain_input_shape(
+        input_shape=None,
+        default_size=None,
+        min_size=139,
+        data_format='channels_first',
+        include_top=False) == (3, None, None)
+
+    assert utils._obtain_input_shape(
+        input_shape=None,
+        default_size=None,
+        min_size=139,
+        data_format='channels_last',
+        include_top=False) == (None, None, 3)
+
+    assert utils._obtain_input_shape(
+        input_shape=(150, 150, 3),
+        default_size=None,
+        min_size=139,
+        data_format='channels_last',
+        include_top=False) == (150, 150, 3)
+
+    assert utils._obtain_input_shape(
+        input_shape=(3, None, None),
+        default_size=None,
+        min_size=139,
+        data_format='channels_first',
+        include_top=False) == (3, None, None)
+
+
+if __name__ == '__main__':
+    pytest.main([__file__])

tests/keras/backend/backend_test.py

@@ -131,11 +131,11 @@ def check_cross_entropy_with_valid_probability_distribution():
     yth = KTH.variable(yval)
     yc = KC.placeholder((4, 2))
 
-    ztf = KTF.eval(KTF.categorical_crossentropy(xtf, ytf, from_logits=True))
-    zth = KTH.eval(KTH.categorical_crossentropy(xth, yth, from_logits=True))
+    ztf = KTF.eval(KTF.categorical_crossentropy(ytf, xtf, from_logits=True))
+    zth = KTH.eval(KTH.categorical_crossentropy(yth, xth, from_logits=True))
 
     func_cntk = KC.function([xc, yc],
-                            [KC.categorical_crossentropy(xc, yc, from_logits=True), ])
+                            [KC.categorical_crossentropy(yc, xc, from_logits=True)])
     zc = func_cntk([xval, yval])
     # Keras function return a list, take the first output
     assert len(zc) == 1
@@ -183,6 +183,18 @@ class TestBackend(object):
             keras_placeholder = K.placeholder(shape=(2, 4, 5))
             assert K.is_keras_tensor(keras_placeholder) is True
 
+    def test_set_learning_phase(self):
+        # not supported learning_phase
+        for backend in (KTF, KTH):
+            with pytest.raises(ValueError):
+                backend.set_learning_phase(2)
+
+    def test_eye(self):
+        zth = KTH.eval(KTH.eye(3))
+        ztf = KTF.eval(KTF.eye(3))
+        assert zth.shape == ztf.shape
+        assert_allclose(zth, ztf, atol=1e-05)
+
     def test_linear_operations(self):
         check_two_tensor_operation('dot', (4, 2), (2, 4), BACKENDS)
         check_two_tensor_operation('dot', (4, 2), (5, 2, 3), BACKENDS)
@@ -244,6 +256,20 @@ class TestBackend(object):
         check_single_tensor_operation('reverse', (4, 3, 2), [KTH, KTF], axes=1)
         check_single_tensor_operation('reverse', (4, 3, 2), [KTH, KTF], axes=(1, 2))
 
+    def test_random_variables(self):
+        zth = KTH.eval(KTH.random_uniform_variable((2, 3), 0, 1))
+        ztf = KTF.eval(KTF.random_uniform_variable((2, 3), 0, 1))
+        assert zth.shape == ztf.shape
+
+        zth = KTH.eval(KTH.random_normal_variable((2, 3), 0, 1))
+        ztf = KTF.eval(KTF.random_normal_variable((2, 3), 0, 1))
+        assert zth.shape == ztf.shape
+
+        # not supported dtype
+        for dtype in ['int16', 'int32', 'int64', 'uint8', 'uint16', 'double']:
+            with pytest.raises(ValueError):
+                ztf = KTF.random_normal_variable((2, 3), 0, 1, dtype=dtype)
+
     def test_batch_dot_shape(self):
         x_batch = KTF.ones(shape=(32, 20))
         y_batch = KTF.ones(shape=(32, 20))
@@ -361,6 +387,11 @@ class TestBackend(object):
                     y = K.repeat_elements(x, reps, axis=rep_axis)
                     assert y._keras_shape == tuple(shape)
 
+        # Test invalid use cases
+        with pytest.raises(ValueError):
+            ztf = KTF.placeholder(shape=(None, 2, 3))
+            KTF.repeat_elements(ztf, 5, axis=0)
+
     def test_tile(self):
         shape = (3, 4)
         arr = np.arange(np.prod(shape)).reshape(shape)
@@ -376,6 +407,8 @@ class TestBackend(object):
             if hasattr(z_list[i], '_keras_shape'):
                 assert z_list[i]._keras_shape == z_list[i].shape
 
+        check_single_tensor_operation('tile', (2, 5), BACKENDS, n=[5, 2])
+
         # test theano shape inference when
         # input shape has None entries
         if K.backend() == 'theano':
@@ -806,7 +839,7 @@ class TestBackend(object):
                         np.log(np.sum(np.exp(x_np), axis=axis, keepdims=keepdims)),
                         rtol=1e-5)
 
-    @pytest.mark.parametrize('K', [KTH, KTF], ids=["KTH", "KTF"])
+    @pytest.mark.parametrize('K', [KTF], ids=["KTF"])
     def test_logsumexp_optim(self, K):
         '''
         Check if optimization works.
@@ -853,7 +886,7 @@ class TestBackend(object):
         check_two_tensor_operation('binary_crossentropy', (4, 2), (4, 2), BACKENDS, from_logits=True)
         # cross_entropy call require the label is a valid probability distribution,
         # otherwise it is garbage in garbage out...
-        # due to the algo difference, we can't guranteen CNTK has the same result on the garbage input.
+        # due to the algo difference, we can't guarantee CNTK has the same result on the garbage input.
         # so create a seperate test case for valid lable input
         check_two_tensor_operation('categorical_crossentropy', (4, 2), (4, 2), [KTH, KTF], from_logits=True)
         check_cross_entropy_with_valid_probability_distribution()
@@ -863,6 +896,11 @@ class TestBackend(object):
         check_single_tensor_operation('l2_normalize', (4, 3), BACKENDS, axis=-1)
         check_single_tensor_operation('l2_normalize', (4, 3), BACKENDS, axis=1)
 
+        # Test invalid use cases
+        for x, k in zip(x_list, [KTH, KTF]):
+            with pytest.raises(ValueError):
+                z = k.dropout(x, level=-0.5)
+
     def test_in_top_k(self):
         batch_size = 20
         num_classes = 10
@@ -876,7 +914,8 @@ class TestBackend(object):
         predictions_tf = KTF.variable(predictions, dtype='float32')
         targets_tf = KTF.variable(targets, dtype='int32')
 
-        for k in range(1, num_classes + 1):
+        # (k == 0 or k > num_classes) does not raise an error but just return an unmeaningful tensor.
+        for k in range(0, num_classes + 1):
             res_th = KTH.eval(KTH.in_top_k(predictions_th, targets_th, k))
             res_tf = KTF.eval(KTF.in_top_k(predictions_tf, targets_tf, k))
 
@@ -992,6 +1031,12 @@ class TestBackend(object):
         kernel_tf = KTF.variable(kernel_val)
         kernel_c = KC.variable(kernel_val)
 
+        # Test invalid use cases
+        with pytest.raises(ValueError):
+            KTH.conv2d(xth, kernel_th, data_format='channels_middle')
+        with pytest.raises(ValueError):
+            KTF.conv2d(xtf, kernel_tf, data_format='channels_middle')
+
         zth = KTH.eval(KTH.conv2d(xth, kernel_th, data_format='channels_last'))
         ztf = KTF.eval(KTF.conv2d(xtf, kernel_tf, data_format='channels_last'))
 
@@ -1061,6 +1106,12 @@ class TestBackend(object):
         kernel_tf = KTF.variable(kernel_val)
         kernel_c = KC.variable(kernel_val)
 
+        # Test invalid use cases
+        with pytest.raises(ValueError):
+            KTH.conv3d(xth, kernel_th, data_format='channels_middle')
+        with pytest.raises(ValueError):
+            KTF.conv3d(xtf, kernel_tf, data_format='channels_middle')
+
         zth = KTH.eval(KTH.conv3d(xth, kernel_th, data_format='channels_last'))
         ztf = KTF.eval(KTF.conv3d(xtf, kernel_tf, data_format='channels_last'))
 
@@ -1129,6 +1180,9 @@ class TestBackend(object):
         cntk_check_single_tensor_operation('pool3d', (5, 9, 11, 5, 3), pool_size=(2, 3, 2),
                                            strides=(1, 1, 1), pool_mode='avg')
 
+        check_single_tensor_operation('pool3d', (2, 6, 6, 6, 3), [KTH, KTF], pool_size=(3, 3, 3),
+                                      strides=(1, 1, 1), padding='same', pool_mode='avg')
+
     def test_random_normal(self):
         mean = 0.
         std = 1.
@@ -1159,6 +1213,183 @@ class TestBackend(object):
 
     '''need special handle for different backend'''
 
+    def test_internal_conv_utils(self):
+        xshape = (5, 4, 3, 2)
+        xval = np.random.random(xshape)
+        xtf = KTF.variable(xval)
+        ztf = KTF._preprocess_deconv_output_shape(xtf, xshape, 'channels_first')
+        assert ztf == (5, 3, 2, 4)
+
+        for dtype in [None, 'float64']:
+            xval = np.random.random((5, 4, 3, 2))
+            xtf = KTF.variable(xval, dtype=dtype)
+            ztf = KTF.eval(KTF._preprocess_conv2d_input(xtf, 'channels_first'))
+            assert ztf.shape == (5, 3, 2, 4)
+
+            xval = np.random.random((6, 5, 4, 3, 2))
+            xtf = KTF.variable(xval, dtype=dtype)
+            ztf = KTF.eval(KTF._preprocess_conv3d_input(xtf, 'channels_first'))
+            assert ztf.shape == (6, 4, 3, 2, 5)
+
+            xval = np.random.random((5, 4, 3, 2))
+            xtf = KTF.variable(xval, dtype=dtype)
+            ztf = KTF.eval(KTF._preprocess_conv2d_kernel(xtf, 'channels_first'))
+            assert ztf.shape == (3, 2, 4, 5)
+
+            xval = np.random.random((6, 5, 4, 3, 2))
+            xtf = KTF.variable(xval, dtype=dtype)
+            ztf = KTF.eval(KTF._preprocess_conv3d_kernel(xtf, 'channels_first'))
+            assert ztf.shape == (4, 3, 2, 5, 6)
+
+        xval = np.random.random((5, 4, 3, 2))
+        xtf = KTF.variable(xval)
+        ztf = KTF.eval(KTF._postprocess_conv2d_output(xtf, 'channels_first'))
+        assert ztf.shape == (5, 2, 4, 3)
+
+        xval = np.random.random((6, 5, 4, 3, 2))
+        xtf = KTF.variable(xval)
+        ztf = KTF.eval(KTF._postprocess_conv3d_output(xtf, 'channels_first'))
+        assert ztf.shape == (6, 2, 5, 4, 3)
+
+    def test_pooling_invalid_use(self):
+        for (input_shape, pool_size) in ([(5, 10, 12, 3), (5, 10, 12, 5, 3)], [(2, 2), (2, 2, 2)]):
+            for backend in (KTH, KTF):
+                x = backend.variable(np.random.random(input_shape))
+                if len(pool_size) == 2:
+                    with pytest.raises(ValueError):
+                        backend.pool2d(x, pool_size=pool_size, data_format='channels_middle')
+                    with pytest.raises(ValueError):
+                        backend.pool2d(x, pool_size=pool_size, padding='twice')
+                    with pytest.raises(ValueError):
+                        backend.pool2d(x, pool_size=pool_size, pool_mode='median')
+                else:
+                    with pytest.raises(ValueError):
+                        backend.pool3d(x, pool_size=pool_size, data_format='channels_middle')
+                    with pytest.raises(ValueError):
+                        backend.pool3d(x, pool_size=pool_size, padding='twice')
+                    with pytest.raises(ValueError):
+                        backend.pool3d(x, pool_size=pool_size, pool_mode='median')
+
+    def test_resize_images(self):
+        for data_format in ['channels_first', 'channels_last']:
+            shape = (5, 5)
+            if data_format == 'channels_first':
+                x_shape = (2, 3) + shape
+            elif data_format == 'channels_last':
+                x_shape = (2,) + shape + (3,)
+            check_single_tensor_operation('resize_images', x_shape,
+                                          BACKENDS,
+                                          height_factor=2,
+                                          width_factor=2,
+                                          data_format=data_format)
+
+        # Test invalid use cases
+        for backend in (KTH, KTF):
+            x = backend.variable(np.random.random(x_shape))
+            with pytest.raises(ValueError):
+                backend.resize_images(x, 2, 2, data_format='channels_middle')
+
+    def test_resize_volumes(self):
+        for data_format in ['channels_first', 'channels_last']:
+            shape = (5, 5, 5)
+            if data_format == 'channels_first':
+                x_shape = (2, 3) + shape
+            elif data_format == 'channels_last':
+                x_shape = (2,) + shape + (3,)
+            check_single_tensor_operation('resize_volumes', x_shape,
+                                          [KTH, KTF],
+                                          depth_factor=2,
+                                          height_factor=2,
+                                          width_factor=2,
+                                          data_format=data_format)
+
+        # Test invalid use cases
+        for backend in (KTH, KTF):
+            x = backend.variable(np.random.random(x_shape))
+            with pytest.raises(ValueError):
+                backend.resize_volumes(x, 2, 2, 2, data_format='channels_middle')
+
+    def test_temporal_padding(self):
+        check_single_tensor_operation('temporal_padding', (2, 3, 4),
+                                      BACKENDS, padding=(2, 2))
+
+    def test_spatial_2d_padding(self):
+        for data_format in ['channels_first', 'channels_last']:
+            shape = (5, 5)
+            padding = ((1, 2), (2, 1))
+            if data_format == 'channels_first':
+                x_shape = (1, 3) + shape
+            else:
+                x_shape = (1,) + shape + (3,)
+            x = np.random.random(x_shape)
+            xth = KTH.variable(x)
+            xtf = KTF.variable(x)
+            zth = KTH.eval(KTH.spatial_2d_padding(xth, padding=padding, data_format=data_format))
+            ztf = KTF.eval(KTF.spatial_2d_padding(xtf, padding=padding, data_format=data_format))
+            assert zth.shape == ztf.shape
+            assert_allclose(zth, ztf, atol=1e-05)
+
+        # Test invalid use cases
+        for backend in (KTH, KTF):
+            x = backend.variable(np.random.random(x_shape))
+            with pytest.raises(ValueError):
+                backend.spatial_2d_padding(x, padding=padding, data_format='channels_middle')
+
+    def test_spatial_3d_padding(self):
+        for data_format in ['channels_first', 'channels_last']:
+            shape = (5, 5, 5)
+            padding = ((1, 2), (2, 1), (1, 2))
+            if data_format == 'channels_first':
+                x_shape = (1, 3) + shape
+            else:
+                x_shape = (1,) + shape + (3,)
+            check_single_tensor_operation('spatial_3d_padding', x_shape,
+                                          BACKENDS,
+                                          padding=padding,
+                                          data_format=data_format)
+
+        # Test invalid use cases
+        for backend in (KTH, KTF):
+            x = backend.variable(np.random.random(x_shape))
+            with pytest.raises(ValueError):
+                backend.spatial_3d_padding(x, padding=padding, data_format='channels_middle')
+
+    def test_bias_add(self):
+        for data_format in ['channels_first', 'channels_last']:
+            for shape in [(3,), (2, 3), (5, 3, 2)]:
+                if data_format == 'channels_first':
+                    x_shape = (1, 4) + shape
+                else:
+                    x_shape = (1,) + shape + (4,)
+                bias_shape = (4,)
+                check_two_tensor_operation('bias_add', x_shape, bias_shape,
+                                           [KTH, KTF],
+                                           data_format=data_format)
+
+        # Test invalid use casess
+        for backend in (KTH, KTF):
+            x = backend.variable(np.random.random(x_shape))
+            b = backend.variable(np.random.random(bias_shape))
+            with pytest.raises(ValueError):
+                KTF.bias_add(x, b, data_format='channels_middle')
+
+    def test_batchnorm(self):
+        shape = (2, 3)
+        for data_format in ['channels_first', 'channels_last']:
+            if data_format == 'channels_first':
+                x_shape = (1, 4) + shape
+            else:
+                x_shape = (1,) + shape + (4,)
+            xth = KTH.variable(np.random.random(x_shape))
+            xtf = KTF.variable(np.random.random(x_shape))
+            zth, _, _ = KTH.normalize_batch_in_training(xth, None, None,
+                                                        reduction_axes='per-activation')
+            ztf, _, _ = KTF.normalize_batch_in_training(xtf, None, None,
+                                                        reduction_axes=[0, 1, 2, 3])
+            zth = KTH.eval(zth)
+            ztf = KTF.eval(ztf)
+            assert zth.shape == ztf.shape
+
     def test_ctc(self):
         # simplified version of TensorFlow's test
 
@@ -1450,13 +1681,29 @@ class TestBackend(object):
                 t = backend.arange(10, dtype=dtype)
                 assert backend.dtype(t) == dtype
 
+    def test_in_train_phase(self):
+        xval = np.random.random((3, 3))
+        xth = KTH.variable(xval)
+        xtf = KTF.variable(xval)
+        yval = np.random.random((2, 2))
+        yth = KTH.variable(yval)
+        ytf = KTF.variable(yval)
+
+        for training in [True, False]:
+            zth = KTH.eval(KTH.in_train_phase(xth, yth, training=training))
+            ztf = KTF.eval(KTF.in_train_phase(xtf, ytf, training=training))
+            assert zth.shape == ztf.shape
+            zth = KTH.eval(KTH.in_train_phase(lambda: xth, lambda: yth, training=training))
+            ztf = KTF.eval(KTF.in_train_phase(lambda: xtf, lambda: ytf, training=training))
+            assert zth.shape == ztf.shape
+
     def test_setfloatx_incorrect_values(self):
         # Keep track of the old value
         old_floatx = floatx()
         # Try some incorrect values
         initial = floatx()
         for value in ['', 'beerfloat', 123]:
-            with pytest.raises(Exception):
+            with pytest.raises(ValueError):
                 set_floatx(value)
         assert floatx() == initial
         # Restore old value

tests/keras/engine/test_topology.py

@@ -195,13 +195,13 @@ def test_node_construction():
     assert test_layer.input_shape == (None, 32)
     assert test_layer.output_shape == (None, 16)
 
-    with pytest.raises(Exception):
+    with pytest.raises(AttributeError):
         dense.input
-    with pytest.raises(Exception):
+    with pytest.raises(AttributeError):
         dense.output
-    with pytest.raises(Exception):
+    with pytest.raises(AttributeError):
         dense.input_mask
-    with pytest.raises(Exception):
+    with pytest.raises(AttributeError):
         dense.output_mask
 
     assert dense.get_input_at(0) == a
@@ -431,14 +431,14 @@ def test_recursion():
     k = Input(shape=(32,), name='input_k')
     m, n = model([j, k])
 
-    with pytest.raises(Exception):
+    with pytest.raises(TypeError):
         Model([j, k], [m, n])
 
     # disconnected graph
     j = Input(shape=(32,), name='input_j')
     k = Input(shape=(32,), name='input_k')
     m, n = model([j, k])
-    with pytest.raises(Exception) as e:
+    with pytest.raises(RuntimeError):
         Model([j], [m, n])
 
     # redundant outputs
@@ -452,14 +452,14 @@ def test_recursion():
     j = Input(shape=(32,), name='input_j')
     k = Input(shape=(32,), name='input_k')
     m, n = model([j, k])
-    with pytest.raises(Exception):
+    with pytest.raises(ValueError):
         Model([j, k, j], [m, n])
 
     # i have not idea what I'm doing: garbage as inputs/outputs
     j = Input(shape=(32,), name='input_j')
     k = Input(shape=(32,), name='input_k')
     m, n = model([j, k])
-    with pytest.raises(Exception):
+    with pytest.raises(TypeError):
         Model([j, k], [m, n, 0])
 
     ####################################################

tests/keras/engine/test_training.py

@@ -1,17 +1,91 @@
 import pytest
 import numpy as np
 from numpy.testing import assert_allclose
+import sys
 import scipy.sparse as sparse
 
 from keras.layers import Dense, Dropout
 from keras.engine.topology import Input
-from keras.engine.training import Model, _check_loss_and_target_compatibility
+from keras.engine.training import Model
+from keras.engine.training import Model
+from keras.engine.training import _check_loss_and_target_compatibility
+from keras.engine.training import _weighted_masked_objective
+from keras.engine.training import _check_array_lengths
+from keras.engine.training import _slice_arrays
 from keras.models import Sequential
 from keras import backend as K
+from keras.utils import Sequence
 from keras.utils.test_utils import keras_test
 from keras.callbacks import LambdaCallback
 
 
+class RandomSequence(Sequence):
+    def __init__(self, batch_size):
+        self.batch_size = batch_size
+
+    def __len__(self):
+        return 12
+
+    def __getitem__(self, idx):
+        return [np.random.random((self.batch_size, 3)), np.random.random((self.batch_size, 3))], [
+            np.random.random((self.batch_size, 4)),
+            np.random.random((self.batch_size, 3))]
+
+
+@keras_test
+def test_check_array_lengths():
+    _check_array_lengths(None, None, None)
+    a_np = np.random.random((4, 3, 3))
+    _check_array_lengths(a_np, a_np, a_np)
+    _check_array_lengths([a_np, a_np], [a_np, a_np], [a_np, a_np])
+    _check_array_lengths([None], [None], [None])
+
+    b_np = np.random.random((3, 4))
+    with pytest.raises(ValueError):
+        _check_array_lengths(a_np, None, None)
+    with pytest.raises(ValueError):
+        _check_array_lengths(a_np, a_np, None)
+    with pytest.raises(ValueError):
+        _check_array_lengths([a_np], [None], None)
+    with pytest.raises(ValueError):
+        _check_array_lengths([a_np], [b_np], None)
+    with pytest.raises(ValueError):
+        _check_array_lengths([a_np], None, [b_np])
+
+
+@keras_test
+def test_slice_arrays():
+    input_a = np.random.random((10, 3))
+    _slice_arrays(None)
+    _slice_arrays(input_a, 0)
+    _slice_arrays(input_a, 0, 1)
+    _slice_arrays(input_a, stop=2)
+    input_a = [None, [1, 1], None, [1, 1]]
+    _slice_arrays(input_a, 0)
+    _slice_arrays(input_a, 0, 1)
+    _slice_arrays(input_a, stop=2)
+    input_a = [None]
+    _slice_arrays(input_a, 0)
+    _slice_arrays(input_a, 0, 1)
+    _slice_arrays(input_a, stop=2)
+    input_a = None
+    _slice_arrays(input_a, 0)
+    _slice_arrays(input_a, 0, 1)
+    _slice_arrays(input_a, stop=2)
+
+
+@keras_test
+def test_weighted_masked_objective():
+    a = Input(shape=(3,), name='input_a')
+
+    # weighted_masked_objective
+    def mask_dummy(y_true=None, y_pred=None, weight=None):
+        return K.placeholder(y_true.shape)
+
+    weighted_function = _weighted_masked_objective(K.categorical_crossentropy)
+    weighted_function(a, a, None)
+
+
 @keras_test
 def test_model_methods():
     a = Input(shape=(3,), name='input_a')
@@ -26,8 +100,6 @@ def test_model_methods():
     optimizer = 'rmsprop'
     loss = 'mse'
     loss_weights = [1., 0.5]
-    model.compile(optimizer, loss, metrics=[], loss_weights=loss_weights,
-                  sample_weight_mode=None)
 
     input_a_np = np.random.random((10, 3))
     input_b_np = np.random.random((10, 3))
@@ -35,6 +107,13 @@ def test_model_methods():
     output_a_np = np.random.random((10, 4))
     output_b_np = np.random.random((10, 3))
 
+    # training/testing doesn't work before compiling.
+    with pytest.raises(RuntimeError):
+        model.train_on_batch([input_a_np, input_b_np], [output_a_np, output_b_np])
+
+    model.compile(optimizer, loss, metrics=[], loss_weights=loss_weights,
+                  sample_weight_mode=None)
+
     # test train_on_batch
     out = model.train_on_batch([input_a_np, input_b_np],
                                [output_a_np, output_b_np])
@@ -75,7 +154,9 @@ def test_model_methods():
     out = model.fit({'input_a': input_a_np, 'input_b': input_b_np},
                     {'dense_1': output_a_np, 'dropout': output_b_np},
                     epochs=1, batch_size=4, validation_split=0.5,
-                    validation_data=({'input_a': input_a_np, 'input_b': input_b_np}, {'dense_1': output_a_np, 'dropout': output_b_np}))
+                    validation_data=(
+                        {'input_a': input_a_np, 'input_b': input_b_np},
+                        {'dense_1': output_a_np, 'dropout': output_b_np}))
 
     # test_on_batch
     out = model.test_on_batch([input_a_np, input_b_np],
@@ -169,6 +250,7 @@ def test_model_methods():
         while True:
             yield ([np.random.random((batch_sz, 3)), np.random.random((batch_sz, 3))],
                    [np.random.random((batch_sz, 4)), np.random.random((batch_sz, 3))])
+
     out = model.fit_generator(gen_data(4), steps_per_epoch=3, epochs=5,
                               initial_epoch=2, callbacks=[tracker_cb])
     assert trained_epochs == [2, 3, 4]
@@ -198,6 +280,142 @@ def test_model_methods():
     out = model.evaluate([input_a_np, input_b_np], [output_a_np, output_b_np], batch_size=4)
     out = model.predict([input_a_np, input_b_np], batch_size=4)
 
+    # empty batch
+    with pytest.raises(ValueError):
+        def gen_data():
+            while True:
+                yield (np.asarray([]), np.asarray([]))
+        out = model.evaluate_generator(gen_data(), steps=1)
+
+    # x is not a list of numpy arrays.
+    with pytest.raises(ValueError):
+        out = model.predict([None])
+
+    # x does not match _feed_input_names.
+    with pytest.raises(ValueError):
+        out = model.predict([input_a_np, None, input_b_np])
+    with pytest.raises(ValueError):
+        out = model.predict([None, input_a_np, input_b_np])
+
+    # all input/output/weight arrays should have the same number of samples.
+    with pytest.raises(ValueError):
+        out = model.train_on_batch([input_a_np, input_b_np[:2]],
+                                   [output_a_np, output_b_np],
+                                   sample_weight=sample_weight)
+    with pytest.raises(ValueError):
+        out = model.train_on_batch([input_a_np, input_b_np],
+                                   [output_a_np, output_b_np[:2]],
+                                   sample_weight=sample_weight)
+    with pytest.raises(ValueError):
+        out = model.train_on_batch([input_a_np, input_b_np],
+                                   [output_a_np, output_b_np],
+                                   sample_weight=[sample_weight[1], sample_weight[1][:2]])
+
+    # `sample_weight` is neither a dict nor a list.
+    with pytest.raises(TypeError):
+        out = model.train_on_batch([input_a_np, input_b_np],
+                                   [output_a_np, output_b_np],
+                                   sample_weight=tuple(sample_weight))
+
+    # `validation_data` is neither a tuple nor a triple.
+    with pytest.raises(ValueError):
+        out = model.fit([input_a_np, input_b_np],
+                        [output_a_np, output_b_np],
+                        epochs=1, batch_size=4,
+                        validation_data=([input_a_np, input_b_np],))
+
+    # `loss` does not match outputs.
+    with pytest.raises(ValueError):
+        model.compile(optimizer, loss=['mse', 'mae', 'mape'])
+
+    # `loss_weights` does not match output_names.
+    with pytest.raises(ValueError):
+        model.compile(optimizer, loss='mse', loss_weights={'lstm': 0.5})
+
+    # `loss_weights` does not match outputs.
+    with pytest.raises(ValueError):
+        model.compile(optimizer, loss='mse', loss_weights=[0.5])
+
+    # `loss_weights` is invalid type.
+    with pytest.raises(TypeError):
+        model.compile(optimizer, loss='mse', loss_weights=(0.5, 0.5))
+
+    # `sample_weight_mode` does not match output_names.
+    with pytest.raises(ValueError):
+        model.compile(optimizer, loss='mse', sample_weight_mode={'lstm': 'temporal'})
+
+    # `sample_weight_mode` does not match output_names.
+    with pytest.raises(ValueError):
+        model.compile(optimizer, loss='mse', sample_weight_mode=['temporal'])
+
+    # `sample_weight_mode` matches output_names partially.
+    with pytest.raises(ValueError):
+        model.compile(optimizer, loss='mse', sample_weight_mode={'dense_1': 'temporal'})
+
+    # `loss` does not exist.
+    with pytest.raises(RuntimeError):
+        model.compile(optimizer, loss=[])
+
+    model.compile(optimizer, loss=['mse', 'mae'])
+    model.compile(optimizer, loss='mse', loss_weights={'dense_1': 0.2, 'dropout': 0.8})
+    model.compile(optimizer, loss='mse', loss_weights=[0.2, 0.8])
+
+    # the rank of weight arrays should be 1.
+    with pytest.raises(ValueError):
+        out = model.train_on_batch([input_a_np, input_b_np],
+                                   [output_a_np, output_b_np],
+                                   sample_weight=[None, np.random.random((10, 20, 30))])
+
+    model.compile(optimizer, loss='mse', sample_weight_mode={'dense_1': None, 'dropout': 'temporal'})
+    model.compile(optimizer, loss='mse', sample_weight_mode=[None, 'temporal'])
+
+    # the rank of output arrays should be at least 3D.
+    with pytest.raises(ValueError):
+        out = model.train_on_batch([input_a_np, input_b_np],
+                                   [output_a_np, output_b_np],
+                                   sample_weight=sample_weight)
+
+    model.compile(optimizer, loss, metrics=[], loss_weights=loss_weights,
+                  sample_weight_mode=None)
+    trained_epochs = []
+    out = model.fit_generator(generator=RandomSequence(3), steps_per_epoch=4, epochs=5,
+                              initial_epoch=0, validation_data=RandomSequence(4),
+                              validation_steps=3, callbacks=[tracker_cb])
+    assert trained_epochs == [0, 1, 2, 3, 4]
+
+
+@pytest.mark.skipif(sys.version_info < (3,), reason='Cannot catch warnings in python 2')
+@keras_test
+def test_warnings():
+    a = Input(shape=(3,), name='input_a')
+    b = Input(shape=(3,), name='input_b')
+
+    a_2 = Dense(4, name='dense_1')(a)
+    dp = Dropout(0.5, name='dropout')
+    b_2 = dp(b)
+
+    model = Model([a, b], [a_2, b_2])
+
+    optimizer = 'rmsprop'
+    loss = 'mse'
+    loss_weights = [1., 0.5]
+    model.compile(optimizer, loss, metrics=[], loss_weights=loss_weights,
+                  sample_weight_mode=None)
+
+    def gen_data(batch_sz):
+        while True:
+            yield ([np.random.random((batch_sz, 3)), np.random.random((batch_sz, 3))],
+                   [np.random.random((batch_sz, 4)), np.random.random((batch_sz, 3))])
+
+    with pytest.warns(Warning) as w:
+        out = model.fit_generator(gen_data(4), steps_per_epoch=10, use_multiprocessing=True, workers=2)
+    warning_raised = any(['Sequence' in str(w_.message) for w_ in w])
+    assert warning_raised, 'No warning raised when using generator with processes.'
+
+    with pytest.warns(None) as w:
+        out = model.fit_generator(RandomSequence(3), steps_per_epoch=4, use_multiprocessing=True, workers=2)
+    assert all(['Sequence' not in str(w_.message) for w_ in w]), 'A warning was raised for Sequence.'
+
 
 @pytest.mark.skipif(K.backend() != 'tensorflow', reason='sparse operations supported only by TF')
 @keras_test
@@ -225,9 +443,9 @@ def test_trainable_argument():
     assert_allclose(out, out_2)
 
     # test with nesting
-    input = Input(shape=(3,))
-    output = model(input)
-    model = Model(input, output)
+    inputs = Input(shape=(3,))
+    outputs = model(inputs)
+    model = Model(inputs, outputs)
     model.compile('rmsprop', 'mse')
     out = model.predict(x)
     model.train_on_batch(x, y)
@@ -245,7 +463,7 @@ def test_check_not_failing():
 @keras_test
 def test_check_last_is_one():
     a = np.random.random((2, 3, 1))
-    with pytest.raises(Exception) as exc:
+    with pytest.raises(ValueError) as exc:
         _check_loss_and_target_compatibility([a], [K.categorical_crossentropy], [a.shape])
 
     assert 'You are passing a target array' in str(exc)
@@ -254,7 +472,7 @@ def test_check_last_is_one():
 @keras_test
 def test_check_bad_shape():
     a = np.random.random((2, 3, 5))
-    with pytest.raises(Exception) as exc:
+    with pytest.raises(ValueError) as exc:
         _check_loss_and_target_compatibility([a], [K.categorical_crossentropy], [(2, 3, 6)])
 
     assert 'targets to have the same shape' in str(exc)

tests/keras/initializers_test.py

@@ -90,6 +90,14 @@ def test_he_normal(tensor_shape):
             target_mean=0., target_std=None, target_max=2 * scale)
 
 
+@pytest.mark.parametrize('tensor_shape', [FC_SHAPE, CONV_SHAPE], ids=['FC', 'CONV'])
+def test_lecun_normal(tensor_shape):
+    fan_in, _ = initializers._compute_fans(tensor_shape)
+    scale = np.sqrt(1. / fan_in)
+    _runner(initializers.lecun_normal(), tensor_shape,
+            target_mean=0., target_std=scale)
+
+
 @pytest.mark.parametrize('tensor_shape', [FC_SHAPE, CONV_SHAPE], ids=['FC', 'CONV'])
 def test_orthogonal(tensor_shape):
     _runner(initializers.orthogonal(), tensor_shape,
@@ -99,7 +107,7 @@ def test_orthogonal(tensor_shape):
 @pytest.mark.parametrize('tensor_shape', [(100, 100), (1, 2, 3, 4)], ids=['FC', 'CONV'])
 def test_identity(tensor_shape):
     if len(tensor_shape) > 2:
-        with pytest.raises(Exception):
+        with pytest.raises(ValueError):
             _runner(initializers.identity(), tensor_shape,
                     target_mean=1. / tensor_shape[0], target_max=1.)
     else:

tests/keras/layers/convolutional_test.py

@@ -5,8 +5,10 @@ from numpy.testing import assert_allclose
 from keras.utils.test_utils import layer_test
 from keras.utils.test_utils import keras_test
 from keras import backend as K
+from keras.engine.topology import InputLayer
 from keras.layers import convolutional
 from keras.layers import pooling
+from keras.models import Sequential
 
 
 # TensorFlow does not support full convolution.
@@ -99,9 +101,15 @@ def test_conv_1d():
                kwargs={'filters': filters,
                        'kernel_size': kernel_size,
                        'padding': padding,
-                       'dilation_rate': 2},
+                       'dilation_rate': 2,
+                       'activation': None},
                input_shape=(batch_size, steps, input_dim))
 
+    convolutional.Conv1D(filters=filters,
+                         kernel_size=kernel_size,
+                         padding=padding,
+                         input_shape=(input_dim,))
+
 
 @keras_test
 def test_maxpooling_1d():
@@ -151,6 +159,8 @@ def test_convolution_2d():
                kwargs={'filters': filters,
                        'kernel_size': 3,
                        'padding': padding,
+                       'data_format': 'channels_last',
+                       'activation': None,
                        'kernel_regularizer': 'l2',
                        'bias_regularizer': 'l2',
                        'activity_regularizer': 'l2',
@@ -167,6 +177,13 @@ def test_convolution_2d():
                        'dilation_rate': (2, 2)},
                input_shape=(num_samples, num_row, num_col, stack_size))
 
+    # Test invalid use case
+    with pytest.raises(ValueError):
+        model = Sequential([convolutional.Conv2D(filters=filters,
+                                                 kernel_size=kernel_size,
+                                                 padding=padding,
+                                                 batch_input_shape=(None, None, 5, None))])
+
 
 @keras_test
 def test_conv2d_transpose():
@@ -194,6 +211,7 @@ def test_conv2d_transpose():
                        'kernel_size': 3,
                        'padding': padding,
                        'data_format': 'channels_first',
+                       'activation': None,
                        'kernel_regularizer': 'l2',
                        'bias_regularizer': 'l2',
                        'activity_regularizer': 'l2',
@@ -203,6 +221,13 @@ def test_conv2d_transpose():
                input_shape=(num_samples, stack_size, num_row, num_col),
                fixed_batch_size=True)
 
+    # Test invalid use case
+    with pytest.raises(ValueError):
+        model = Sequential([convolutional.Conv2DTranspose(filters=filters,
+                                                          kernel_size=3,
+                                                          padding=padding,
+                                                          batch_input_shape=(None, None, 5, None))])
+
 
 @pytest.mark.skipif(K.backend() != 'tensorflow', reason='Requires TF backend')
 @keras_test
@@ -231,6 +256,8 @@ def test_separable_conv_2d():
                kwargs={'filters': filters,
                        'kernel_size': 3,
                        'padding': padding,
+                       'data_format': 'channels_first',
+                       'activation': None,
                        'depthwise_regularizer': 'l2',
                        'pointwise_regularizer': 'l2',
                        'bias_regularizer': 'l2',
@@ -239,7 +266,14 @@ def test_separable_conv_2d():
                        'depthwise_constraint': 'unit_norm',
                        'strides': strides,
                        'depth_multiplier': multiplier},
-               input_shape=(num_samples, num_row, num_col, stack_size))
+               input_shape=(num_samples, stack_size, num_row, num_col))
+
+    # Test invalid use case
+    with pytest.raises(ValueError):
+        model = Sequential([convolutional.SeparableConv2D(filters=filters,
+                                                          kernel_size=3,
+                                                          padding=padding,
+                                                          batch_input_shape=(None, None, 5, None))])
 
 
 @keras_test
@@ -342,6 +376,7 @@ def test_convolution_3d():
                kwargs={'filters': filters,
                        'kernel_size': (1, 2, 3),
                        'padding': padding,
+                       'activation': None,
                        'kernel_regularizer': 'l2',
                        'bias_regularizer': 'l2',
                        'activity_regularizer': 'l2',
@@ -353,6 +388,51 @@ def test_convolution_3d():
                             stack_size))
 
 
+@keras_test
+def test_conv3d_transpose():
+    filters = 2
+    stack_size = 3
+    num_depth = 7
+    num_row = 5
+    num_col = 6
+
+    for padding in _convolution_paddings:
+        for strides in [(1, 1, 1), (2, 2, 2)]:
+            for data_format in ['channels_first', 'channels_last']:
+                if padding == 'same' and strides != (1, 1, 1):
+                    continue
+                layer_test(convolutional.Conv3DTranspose,
+                           kwargs={'filters': filters,
+                                   'kernel_size': 3,
+                                   'padding': padding,
+                                   'strides': strides,
+                                   'data_format': data_format},
+                           input_shape=(None, num_depth, num_row, num_col, stack_size),
+                           fixed_batch_size=True)
+
+    layer_test(convolutional.Conv3DTranspose,
+               kwargs={'filters': filters,
+                       'kernel_size': 3,
+                       'padding': padding,
+                       'data_format': 'channels_first',
+                       'activation': None,
+                       'kernel_regularizer': 'l2',
+                       'bias_regularizer': 'l2',
+                       'activity_regularizer': 'l2',
+                       'kernel_constraint': 'max_norm',
+                       'bias_constraint': 'max_norm',
+                       'strides': strides},
+               input_shape=(None, stack_size, num_depth, num_row, num_col),
+               fixed_batch_size=True)
+
+    # Test invalid use case
+    with pytest.raises(ValueError):
+        model = Sequential([convolutional.Conv3DTranspose(filters=filters,
+                                                          kernel_size=3,
+                                                          padding=padding,
+                                                          batch_input_shape=(None, None, 5, None, None))])
+
+
 @keras_test
 def test_maxpooling_3d():
     pool_size = (3, 3, 3)
@@ -393,29 +473,29 @@ def test_zero_padding_1d():
     input_dim = 2
     num_steps = 5
     shape = (num_samples, num_steps, input_dim)
-    input = np.ones(shape)
+    inputs = np.ones(shape)
 
     # basic test
     layer_test(convolutional.ZeroPadding1D,
                kwargs={'padding': 2},
-               input_shape=input.shape)
+               input_shape=inputs.shape)
     layer_test(convolutional.ZeroPadding1D,
                kwargs={'padding': (1, 2)},
-               input_shape=input.shape)
+               input_shape=inputs.shape)
 
     # correctness test
     layer = convolutional.ZeroPadding1D(padding=2)
     layer.build(shape)
-    output = layer(K.variable(input))
-    np_output = K.eval(output)
+    outputs = layer(K.variable(inputs))
+    np_output = K.eval(outputs)
     for offset in [0, 1, -1, -2]:
         assert_allclose(np_output[:, offset, :], 0.)
     assert_allclose(np_output[:, 2:-2, :], 1.)
 
     layer = convolutional.ZeroPadding1D(padding=(1, 2))
     layer.build(shape)
-    output = layer(K.variable(input))
-    np_output = K.eval(output)
+    outputs = layer(K.variable(inputs))
+    np_output = K.eval(outputs)
     for left_offset in [0]:
         assert_allclose(np_output[:, left_offset, :], 0.)
     for right_offset in [-1, -2]:
@@ -446,8 +526,8 @@ def test_zero_padding_2d():
         layer = convolutional.ZeroPadding2D(padding=(2, 2),
                                             data_format=data_format)
         layer.build(inputs.shape)
-        output = layer(K.variable(inputs))
-        np_output = K.eval(output)
+        outputs = layer(K.variable(inputs))
+        np_output = K.eval(outputs)
         if data_format == 'channels_last':
             for offset in [0, 1, -1, -2]:
                 assert_allclose(np_output[:, offset, :, :], 0.)
@@ -462,8 +542,8 @@ def test_zero_padding_2d():
         layer = convolutional.ZeroPadding2D(padding=((1, 2), (3, 4)),
                                             data_format=data_format)
         layer.build(inputs.shape)
-        output = layer(K.variable(inputs))
-        np_output = K.eval(output)
+        outputs = layer(K.variable(inputs))
+        np_output = K.eval(outputs)
         if data_format == 'channels_last':
             for top_offset in [0]:
                 assert_allclose(np_output[:, top_offset, :, :], 0.)
@@ -510,8 +590,8 @@ def test_zero_padding_3d():
         layer = convolutional.ZeroPadding3D(padding=(2, 2, 2),
                                             data_format=data_format)
         layer.build(inputs.shape)
-        output = layer(K.variable(inputs))
-        np_output = K.eval(output)
+        outputs = layer(K.variable(inputs))
+        np_output = K.eval(outputs)
         if data_format == 'channels_last':
             for offset in [0, 1, -1, -2]:
                 assert_allclose(np_output[:, offset, :, :, :], 0.)
@@ -528,8 +608,8 @@ def test_zero_padding_3d():
         layer = convolutional.ZeroPadding3D(padding=((1, 2), (3, 4), (0, 2)),
                                             data_format=data_format)
         layer.build(inputs.shape)
-        output = layer(K.variable(inputs))
-        np_output = K.eval(output)
+        outputs = layer(K.variable(inputs))
+        np_output = K.eval(outputs)
         if data_format == 'channels_last':
             for dim1_offset in [0, -1, -2]:
                 assert_allclose(np_output[:, dim1_offset, :, :, :], 0.)
@@ -581,8 +661,8 @@ def test_upsampling_2d():
                     size=(length_row, length_col),
                     data_format=data_format)
                 layer.build(inputs.shape)
-                output = layer(K.variable(inputs))
-                np_output = K.eval(output)
+                outputs = layer(K.variable(inputs))
+                np_output = K.eval(outputs)
                 if data_format == 'channels_first':
                     assert np_output.shape[2] == length_row * input_num_row
                     assert np_output.shape[3] == length_col * input_num_col
@@ -632,8 +712,8 @@ def test_upsampling_3d():
                         size=(length_dim1, length_dim2, length_dim3),
                         data_format=data_format)
                     layer.build(inputs.shape)
-                    output = layer(K.variable(inputs))
-                    np_output = K.eval(output)
+                    outputs = layer(K.variable(inputs))
+                    np_output = K.eval(outputs)
                     if data_format == 'channels_first':
                         assert np_output.shape[2] == length_dim1 * input_len_dim1
                         assert np_output.shape[3] == length_dim2 * input_len_dim2
@@ -694,8 +774,8 @@ def test_cropping_2d():
         layer = convolutional.Cropping2D(cropping=cropping,
                                          data_format=data_format)
         layer.build(inputs.shape)
-        output = layer(K.variable(inputs))
-        np_output = K.eval(output)
+        outputs = layer(K.variable(inputs))
+        np_output = K.eval(outputs)
         # compare with numpy
         if data_format == 'channels_first':
             expected_out = inputs[:,
@@ -722,11 +802,17 @@ def test_cropping_2d():
         layer = convolutional.Cropping2D(cropping=cropping,
                                          data_format=data_format)
         layer.build(inputs.shape)
-        output = layer(K.variable(inputs))
-        np_output = K.eval(output)
+        outputs = layer(K.variable(inputs))
+        np_output = K.eval(outputs)
         # compare with input
         assert_allclose(np_output, inputs)
 
+    # Test invalid use cases
+    with pytest.raises(ValueError):
+        layer = convolutional.Cropping2D(cropping=((1, 1),))
+    with pytest.raises(ValueError):
+        layer = convolutional.Cropping2D(cropping=lambda x: x)
+
 
 def test_cropping_3d():
     num_samples = 2
@@ -753,8 +839,8 @@ def test_cropping_3d():
         layer = convolutional.Cropping3D(cropping=cropping,
                                          data_format=data_format)
         layer.build(inputs.shape)
-        output = layer(K.variable(inputs))
-        np_output = K.eval(output)
+        outputs = layer(K.variable(inputs))
+        np_output = K.eval(outputs)
         # compare with numpy
         if data_format == 'channels_first':
             expected_out = inputs[:,
@@ -783,10 +869,16 @@ def test_cropping_3d():
         layer = convolutional.Cropping3D(cropping=cropping,
                                          data_format=data_format)
         layer.build(inputs.shape)
-        output = layer(K.variable(inputs))
-        np_output = K.eval(output)
+        outputs = layer(K.variable(inputs))
+        np_output = K.eval(outputs)
         # compare with input
         assert_allclose(np_output, inputs)
 
+    # Test invalid use cases
+    with pytest.raises(ValueError):
+        layer = convolutional.Cropping3D(cropping=((1, 1),))
+    with pytest.raises(ValueError):
+        layer = convolutional.Cropping3D(cropping=lambda x: x)
+
 if __name__ == '__main__':
     pytest.main([__file__])

tests/keras/layers/core_test.py

@@ -32,9 +32,23 @@ def test_dropout():
                kwargs={'rate': 0.5},
                input_shape=(2, 3, 4))
 
-    layer_test(layers.SpatialDropout2D,
-               kwargs={'rate': 0.5},
-               input_shape=(2, 3, 4, 5))
+    for data_format in ['channels_last', 'channels_first']:
+        for shape in [(4, 5), (4, 5, 6)]:
+            if data_format == 'channels_last':
+                input_shape = (2,) + shape + (3,)
+            else:
+                input_shape = (2, 3) + shape
+            layer_test(layers.SpatialDropout2D if len(shape) == 2 else layers.SpatialDropout3D,
+                       kwargs={'rate': 0.5,
+                               'data_format': data_format},
+                       input_shape=input_shape)
+
+            # Test invalid use cases
+            with pytest.raises(ValueError):
+                layer_test(layers.SpatialDropout2D if len(shape) == 2 else layers.SpatialDropout3D,
+                           kwargs={'rate': 0.5,
+                                   'data_format': 'channels_middle'},
+                           input_shape=input_shape)
 
 
 @keras_test
@@ -97,6 +111,24 @@ def test_lambda():
                        'arguments': {'a': 0.6, 'b': 0.4}},
                input_shape=(3, 2))
 
+    def antirectifier(x):
+        x -= K.mean(x, axis=1, keepdims=True)
+        x = K.l2_normalize(x, axis=1)
+        pos = K.relu(x)
+        neg = K.relu(-x)
+        return K.concatenate([pos, neg], axis=1)
+
+    def antirectifier_output_shape(input_shape):
+        shape = list(input_shape)
+        assert len(shape) == 2  # only valid for 2D tensors
+        shape[-1] *= 2
+        return tuple(shape)
+
+    layer_test(layers.Lambda,
+               kwargs={'function': antirectifier,
+                       'output_shape': antirectifier_output_shape},
+               input_shape=(3, 2))
+
     # test serialization with function
     def f(x):
         return x + 1

tests/keras/layers/noise_test.py

@@ -23,5 +23,14 @@ def test_GaussianDropout():
                input_shape=(3, 2, 3))
 
 
+@keras_test
+@pytest.mark.skipif((K.backend() == 'cntk'),
+                    reason="cntk does not support it yet")
+def test_AlphaDropout():
+    layer_test(noise.AlphaDropout,
+               kwargs={'rate': 0.1},
+               input_shape=(3, 2, 3))
+
+
 if __name__ == '__main__':
     pytest.main([__file__])

tests/keras/layers/normalization_test.py

@@ -15,7 +15,7 @@ input_shapes = [np.ones((10, 10)), np.ones((10, 10, 10))]
 
 
 @keras_test
-def basic_batchnorm_test():
+def test_basic_batchnorm():
     from keras import regularizers
     layer_test(normalization.BatchNormalization,
                kwargs={'momentum': 0.9,
@@ -29,9 +29,6 @@ def basic_batchnorm_test():
                        'moving_mean_initializer': 'zeros',
                        'moving_variance_initializer': 'ones'},
                input_shape=(3, 4, 2))
-    layer_test(normalization.BatchNormalization,
-               kwargs={'scale': False, 'center': False},
-               input_shape=(3, 3))
 
 
 @keras_test

tests/keras/layers/recurrent_test.py

@@ -50,6 +50,23 @@ def test_dynamic_behavior(layer_class):
     model.train_on_batch(x, y)
 
 
+@rnn_test
+@pytest.mark.skipif(K.backend() == 'cntk', reason='Stateful is not supported with CNTK')
+def test_stateful_invalid_use(layer_class):
+    layer = layer_class(units,
+                        stateful=True,
+                        batch_input_shape=(num_samples, timesteps, embedding_dim))
+    model = Sequential()
+    model.add(layer)
+    model.compile('sgd', 'mse')
+    x = np.random.random((num_samples * 2, timesteps, embedding_dim))
+    y = np.random.random((num_samples * 2, units))
+    with pytest.raises(ValueError):
+        model.fit(x, y)
+    with pytest.raises(ValueError):
+        model.predict(x, batch_size=num_samples + 1)
+
+
 @rnn_test
 def test_dropout(layer_class):
     layer_test(layer_class,
@@ -74,6 +91,12 @@ def test_implementation_mode(layer_class):
                    kwargs={'units': units,
                            'implementation': mode},
                    input_shape=(num_samples, timesteps, embedding_dim))
+    layer_test(layer_class,
+               kwargs={'units': units,
+                       'implementation': mode,
+                       'dropout': 0.1,
+                       'recurrent_dropout': 0.1},
+               input_shape=(num_samples, timesteps, embedding_dim))
 
 
 @rnn_test

tests/keras/layers/wrappers_test.py

@@ -88,6 +88,19 @@ def test_TimeDistributed():
     outer_model.fit(np.random.random((10, 3, 2)), np.random.random((10, 3, 3)), epochs=1, batch_size=10)
 
 
+@keras_test
+@pytest.mark.skipif((K.backend() == 'cntk'),
+                    reason='cntk does not support dropout yet')
+def test_TimeDistributed_learning_phase():
+    # test layers that need learning_phase to be set
+    np.random.seed(1234)
+    x = Input(shape=(3, 2))
+    y = wrappers.TimeDistributed(core.Dropout(.999))(x, training=True)
+    model = Model(x, y)
+    y = model.predict(np.random.random((10, 3, 2)))
+    assert_allclose(np.mean(y), 0., atol=1e-1, rtol=1e-1)
+
+
 @keras_test
 def test_regularizers():
     model = Sequential()
@@ -110,7 +123,7 @@ def test_regularizers():
 
 @keras_test
 @pytest.mark.skipif((K.backend() == 'cntk'),
-                    reason="cntk does not support reverse yet")
+                    reason='cntk does not support reverse yet')
 def test_Bidirectional():
     rnn = recurrent.SimpleRNN
     samples = 2
@@ -145,18 +158,18 @@ def test_Bidirectional():
         model.fit(x, y, epochs=1, batch_size=1)
 
         # test with functional API
-        input = Input((timesteps, dim))
-        output = wrappers.Bidirectional(rnn(output_dim, dropout=dropout_rate,
-                                            recurrent_dropout=dropout_rate),
-                                        merge_mode=mode)(input)
-        model = Model(input, output)
+        inputs = Input((timesteps, dim))
+        outputs = wrappers.Bidirectional(rnn(output_dim, dropout=dropout_rate,
+                                             recurrent_dropout=dropout_rate),
+                                         merge_mode=mode)(inputs)
+        model = Model(inputs, outputs)
         model.compile(loss='mse', optimizer='sgd')
         model.fit(x, y, epochs=1, batch_size=1)
 
         # Bidirectional and stateful
-        input = Input(batch_shape=(1, timesteps, dim))
-        output = wrappers.Bidirectional(rnn(output_dim, stateful=True), merge_mode=mode)(input)
-        model = Model(input, output)
+        inputs = Input(batch_shape=(1, timesteps, dim))
+        outputs = wrappers.Bidirectional(rnn(output_dim, stateful=True), merge_mode=mode)(inputs)
+        model = Model(inputs, outputs)
         model.compile(loss='mse', optimizer='sgd')
         model.fit(x, y, epochs=1, batch_size=1)
 

tests/keras/legacy/interface_test.py

@@ -1,6 +1,7 @@
 import pytest
 import json
 from keras.utils.test_utils import keras_test
+from keras.engine.topology import preprocess_weights_for_loading
 import keras
 import numpy as np
 
@@ -109,6 +110,10 @@ def test_lstm_legacy_interface():
     new_layer = keras.layers.LSTM(2, input_shape=[3, 5], name='d')
     assert json.dumps(old_layer.get_config()) == json.dumps(new_layer.get_config())
 
+    preprocess_weights_for_loading(new_layer,
+                                   [np.random.random(x) for x in [(5, 2), (2, 2), (2,)] * 4],
+                                   original_keras_version='1')
+
     old_layer = keras.layers.LSTM(input_shape=[3, 5], output_dim=2, name='d', consume_less='mem')
     new_layer = keras.layers.LSTM(2, input_shape=[3, 5], name='d', implementation=1)
     assert json.dumps(old_layer.get_config()) == json.dumps(new_layer.get_config())
@@ -207,6 +212,10 @@ def test_gru_legacy_interface():
     new_layer = keras.layers.GRU(2, input_shape=[3, 5], name='d')
     assert json.dumps(old_layer.get_config()) == json.dumps(new_layer.get_config())
 
+    preprocess_weights_for_loading(new_layer,
+                                   [np.random.random(x) for x in [(5, 2), (2, 2), (2,)] * 3],
+                                   original_keras_version='1')
+
     old_layer = keras.layers.GRU(2, init='normal',
                                  inner_init='glorot_uniform',
                                  inner_activation='hard_sigmoid',
@@ -798,13 +807,14 @@ def test_generator_methods_interface():
                         samples_per_epoch=1,
                         validation_data=val_generator(),
                         nb_val_samples=1,
-                        nb_worker=1)
+                        nb_worker=1, pickle_safe=True, max_q_size=3)
+
     model.evaluate_generator(generator=train_generator(),
                              val_samples=2,
-                             nb_worker=1)
+                             nb_worker=1, pickle_safe=False, max_q_size=3)
     model.predict_generator(generator=pred_generator(),
                             val_samples=2,
-                            nb_worker=1)
+                            nb_worker=1, pickle_safe=False, max_q_size=3)
 
 
 def test_spatialdropout1d_legacy_interface():

tests/keras/legacy/models_test.py

@@ -19,6 +19,17 @@ batch_size = 32
 epochs = 1
 
 
+@pytest.fixture
+def in_tmpdir(tmpdir):
+    """Runs a function in a temporary directory.
+
+    Checks that the directory is empty afterwards.
+    """
+    with tmpdir.as_cwd():
+        yield None
+    assert not tmpdir.listdir()
+
+
 def _get_test_data():
     np.random.seed(1234)
 
@@ -36,7 +47,7 @@ def _get_test_data():
 
 
 @keras_test
-def test_merge_sum():
+def test_merge_sum(in_tmpdir):
     (x_train, y_train), (x_test, y_test) = _get_test_data()
     left = Sequential()
     left.add(Dense(num_hidden, input_shape=(input_dim,)))
@@ -131,7 +142,7 @@ def test_merge_dot():
 
 
 @keras_test
-def test_merge_concat():
+def test_merge_concat(in_tmpdir):
     (x_train, y_train), (x_test, y_test) = _get_test_data()
 
     left = Sequential(name='branch_1')
@@ -171,7 +182,7 @@ def test_merge_concat():
 
 
 @keras_test
-def test_merge_recursivity():
+def test_merge_recursivity(in_tmpdir):
     (x_train, y_train), (x_test, y_test) = _get_test_data()
     left = Sequential()
     left.add(Dense(num_hidden, input_shape=(input_dim,)))
@@ -228,7 +239,7 @@ def test_merge_recursivity():
 
 
 @keras_test
-def test_merge_overlap():
+def test_merge_overlap(in_tmpdir):
     (x_train, y_train), (x_test, y_test) = _get_test_data()
     left = Sequential()
     left.add(Dense(num_hidden, input_shape=(input_dim,)))

tests/keras/optimizers_test.py

@@ -88,5 +88,24 @@ def test_clipvalue():
     _test_optimizer(sgd)
 
 
+def test_tfoptimizer():
+    from keras import constraints
+    from tensorflow import train
+    optimizer = optimizers.TFOptimizer(train.AdamOptimizer)
+    model = Sequential()
+    model.add(Dense(2, input_shape=(3,), kernel_constraint=constraints.MaxNorm(1)))
+    model.compile(loss='mean_squared_error', optimizer=optimizer)
+    # TF optimizers do not support weights constraints
+    with pytest.raises(ValueError):
+        model.fit(np.random.random((5, 3)), np.random.random((5, 2)), epochs=1, batch_size=5, verbose=0)
+    # not supported
+    with pytest.raises(NotImplementedError):
+        optimizer.weights
+    with pytest.raises(NotImplementedError):
+        optimizer.get_config()
+    with pytest.raises(NotImplementedError):
+        optimizer.from_config(None)
+
+
 if __name__ == '__main__':
     pytest.main([__file__])

tests/keras/preprocessing/image_test.py

@@ -3,8 +3,6 @@ from keras.preprocessing import image
 from PIL import Image
 import numpy as np
 import os
-import shutil
-import tempfile
 
 
 class TestImage:
@@ -29,7 +27,7 @@ class TestImage:
     def teardown_class(cls):
         del cls.all_test_images
 
-    def test_image_data_generator(self):
+    def test_image_data_generator(self, tmpdir):
         for test_images in self.all_test_images:
             img_list = []
             for im in test_images:
@@ -54,12 +52,10 @@ class TestImage:
                 vertical_flip=True)
             generator.fit(images, augment=True)
 
-            tmp_folder = tempfile.mkdtemp(prefix='test_images')
             for x, y in generator.flow(images, np.arange(images.shape[0]),
-                                       shuffle=True, save_to_dir=tmp_folder):
+                                       shuffle=True, save_to_dir=str(tmpdir)):
                 assert x.shape[1:] == images.shape[1:]
                 break
-            shutil.rmtree(tmp_folder)
 
     def test_image_data_generator_invalid_data(self):
         generator = image.ImageDataGenerator(
@@ -86,6 +82,7 @@ class TestImage:
             featurewise_std_normalization=True,
             samplewise_std_normalization=True,
             zca_whitening=True,
+            zoom_range=(0.2, 0.2),
             data_format='channels_last')
         # Test grayscale
         x = np.random.random((32, 10, 10, 1))
@@ -107,9 +104,8 @@ class TestImage:
         x = np.random.random((32, 3, 10, 10))
         generator.fit(x)
 
-    def test_directory_iterator(self):
+    def test_directory_iterator(self, tmpdir):
         num_classes = 2
-        tmp_folder = tempfile.mkdtemp(prefix='test_images')
 
         # create folders and subfolders
         paths = []
@@ -122,7 +118,7 @@ class TestImage:
                 os.path.join(class_directory, 'subfolder-1', 'sub-subfolder')
             ]
             for path in classpaths:
-                os.mkdir(os.path.join(tmp_folder, path))
+                tmpdir.join(path).mkdir()
             paths.append(classpaths)
 
         # save the images in the paths
@@ -136,34 +132,38 @@ class TestImage:
                 classpaths = paths[im_class]
                 filename = os.path.join(classpaths[count % len(classpaths)], 'image-{}.jpg'.format(count))
                 filenames.append(filename)
-                im.save(os.path.join(tmp_folder, filename))
+                im.save(str(tmpdir / filename))
                 count += 1
 
         # create iterator
         generator = image.ImageDataGenerator()
-        dir_iterator = generator.flow_from_directory(tmp_folder)
+        dir_iterator = generator.flow_from_directory(str(tmpdir))
 
         # check number of classes and images
         assert(len(dir_iterator.class_indices) == num_classes)
         assert(len(dir_iterator.classes) == count)
         assert(sorted(dir_iterator.filenames) == sorted(filenames))
-        shutil.rmtree(tmp_folder)
 
-    def test_directory_iterator_class_mode_input(self):
-        tmp_folder = tempfile.mkdtemp(prefix='test_images')
-        os.mkdir(os.path.join(tmp_folder, 'class-1'))
+        # Test invalid use cases
+        with pytest.raises(ValueError):
+            generator.flow_from_directory(str(tmpdir), color_mode='cmyk')
+        with pytest.raises(ValueError):
+            generator.flow_from_directory(str(tmpdir), class_mode='output')
+
+    def test_directory_iterator_class_mode_input(self, tmpdir):
+        tmpdir.join('class-1').mkdir()
 
         # save the images in the paths
         count = 0
         for test_images in self.all_test_images:
             for im in test_images:
-                filename = os.path.join(tmp_folder, 'class-1', 'image-{}.jpg'.format(count))
-                im.save(os.path.join(tmp_folder, filename))
+                filename = str(tmpdir / 'class-1' / 'image-{}.jpg'.format(count))
+                im.save(filename)
                 count += 1
 
         # create iterator
         generator = image.ImageDataGenerator()
-        dir_iterator = generator.flow_from_directory(tmp_folder, class_mode='input')
+        dir_iterator = generator.flow_from_directory(str(tmpdir), class_mode='input')
         batch = next(dir_iterator)
 
         # check if input and output have the same shape
@@ -173,7 +173,6 @@ class TestImage:
         output_img = batch[1][0]
         output_img[0][0][0] += 1
         assert(input_img[0][0][0] != output_img[0][0][0])
-        shutil.rmtree(tmp_folder)
 
     def test_img_utils(self):
         height, width = 10, 8
@@ -204,6 +203,31 @@ class TestImage:
         x = image.img_to_array(img, data_format='channels_last')
         assert x.shape == (height, width, 1)
 
+        # Test invalid use case
+        with pytest.raises(ValueError):
+            x = np.random.random((height, width))  # not 3D
+            img = image.array_to_img(x, data_format='channels_first')
+        with pytest.raises(ValueError):
+            x = np.random.random((height, width, 3))
+            img = image.array_to_img(x, data_format='channels')  # unknown data_format
+        with pytest.raises(ValueError):
+            x = np.random.random((height, width, 5))  # neither RGB nor gray-scale
+            img = image.array_to_img(x, data_format='channels_last')
+        with pytest.raises(ValueError):
+            x = np.random.random((height, width, 3))
+            img = image.img_to_array(x, data_format='channels')  # unknown data_format
+        with pytest.raises(ValueError):
+            x = np.random.random((height, width, 5, 3))  # neither RGB nor gray-scale
+            img = image.img_to_array(x, data_format='channels_last')
+
+    def test_random_transforms(self):
+        x = np.random.random((2, 28, 28))
+        assert image.random_rotation(x, 45).shape == (2, 28, 28)
+        assert image.random_shift(x, 1, 1).shape == (2, 28, 28)
+        assert image.random_shear(x, 20).shape == (2, 28, 28)
+        assert image.random_zoom(x, (5, 5)).shape == (2, 28, 28)
+        assert image.random_channel_shift(x, 20).shape == (2, 28, 28)
+
 
 if __name__ == '__main__':
     pytest.main([__file__])

tests/keras/preprocessing/text_test.py

@@ -1,6 +1,7 @@
-from keras.preprocessing.text import Tokenizer, one_hot
-import pytest
 import numpy as np
+import pytest
+
+from keras.preprocessing.text import Tokenizer, one_hot, hashing_trick
 
 
 def test_one_hot():
@@ -11,6 +12,22 @@ def test_one_hot():
     assert np.min(encoded) >= 0
 
 
+def test_hashing_trick_hash():
+    text = 'The cat sat on the mat.'
+    encoded = hashing_trick(text, 5)
+    assert len(encoded) == 6
+    assert np.max(encoded) <= 4
+    assert np.min(encoded) >= 1
+
+
+def test_hashing_trick_md5():
+    text = 'The cat sat on the mat.'
+    encoded = hashing_trick(text, 5, hash_function='md5')
+    assert len(encoded) == 6
+    assert np.max(encoded) <= 4
+    assert np.min(encoded) >= 1
+
+
 def test_tokenizer():
     texts = ['The cat sat on the mat.',
              'The dog sat on the log.',

tests/keras/test_callbacks.py

@@ -54,9 +54,9 @@ def test_TerminateOnNaN():
 
 
 @keras_test
-def test_ModelCheckpoint():
+def test_ModelCheckpoint(tmpdir):
     np.random.seed(1337)
-    filepath = 'checkpoint.h5'
+    filepath = str(tmpdir / 'checkpoint.h5')
     (X_train, y_train), (X_test, y_test) = get_test_data(num_train=train_samples,
                                                          num_test=test_samples,
                                                          input_shape=(input_dim,),
@@ -80,7 +80,7 @@ def test_ModelCheckpoint():
                                       save_best_only=save_best_only, mode=mode)]
     model.fit(X_train, y_train, batch_size=batch_size,
               validation_data=(X_test, y_test), callbacks=cbks, epochs=1)
-    assert os.path.exists(filepath)
+    assert os.path.isfile(filepath)
     os.remove(filepath)
 
     # case 2
@@ -89,7 +89,7 @@ def test_ModelCheckpoint():
                                       save_best_only=save_best_only, mode=mode)]
     model.fit(X_train, y_train, batch_size=batch_size,
               validation_data=(X_test, y_test), callbacks=cbks, epochs=1)
-    assert os.path.exists(filepath)
+    assert os.path.isfile(filepath)
     os.remove(filepath)
 
     # case 3
@@ -99,7 +99,7 @@ def test_ModelCheckpoint():
                                       save_best_only=save_best_only, mode=mode)]
     model.fit(X_train, y_train, batch_size=batch_size,
               validation_data=(X_test, y_test), callbacks=cbks, epochs=1)
-    assert os.path.exists(filepath)
+    assert os.path.isfile(filepath)
     os.remove(filepath)
 
     # case 4
@@ -108,7 +108,7 @@ def test_ModelCheckpoint():
                                       save_best_only=save_best_only, mode=mode)]
     model.fit(X_train, y_train, batch_size=batch_size,
               validation_data=(X_test, y_test), callbacks=cbks, epochs=1)
-    assert os.path.exists(filepath)
+    assert os.path.isfile(filepath)
     os.remove(filepath)
 
     # case 5
@@ -121,12 +121,13 @@ def test_ModelCheckpoint():
                                       period=period)]
     model.fit(X_train, y_train, batch_size=batch_size,
               validation_data=(X_test, y_test), callbacks=cbks, epochs=4)
-    assert os.path.exists(filepath.format(epoch=1))
-    assert os.path.exists(filepath.format(epoch=3))
+    assert os.path.isfile(filepath.format(epoch=1))
+    assert os.path.isfile(filepath.format(epoch=3))
     assert not os.path.exists(filepath.format(epoch=0))
     assert not os.path.exists(filepath.format(epoch=2))
     os.remove(filepath.format(epoch=1))
     os.remove(filepath.format(epoch=3))
+    assert not tmpdir.listdir()
 
 
 @keras_test
@@ -244,9 +245,9 @@ def test_ReduceLROnPlateau():
 
 
 @keras_test
-def test_CSVLogger():
+def test_CSVLogger(tmpdir):
     np.random.seed(1337)
-    filepath = 'log.tsv'
+    filepath = str(tmpdir / 'log.tsv')
     sep = '\t'
     (X_train, y_train), (X_test, y_test) = get_test_data(num_train=train_samples,
                                                          num_test=test_samples,
@@ -273,7 +274,7 @@ def test_CSVLogger():
     model.fit(X_train, y_train, batch_size=batch_size,
               validation_data=(X_test, y_test), callbacks=cbks, epochs=1)
 
-    assert os.path.exists(filepath)
+    assert os.path.isfile(filepath)
     with open(filepath) as csvfile:
         dialect = Sniffer().sniff(csvfile.read())
     assert dialect.delimiter == sep
@@ -296,14 +297,15 @@ def test_CSVLogger():
         assert len(re.findall('epoch', output)) == 1
 
     os.remove(filepath)
+    assert not tmpdir.listdir()
 
 
 @keras_test
 @pytest.mark.skipif((K.backend() != 'tensorflow'),
                     reason='Requires tensorflow backend')
-def test_TensorBoard():
+def test_TensorBoard(tmpdir):
     np.random.seed(np.random.randint(1, 1e7))
-    filepath = './logs_' + str(np.random.randint(1, 1e4))
+    filepath = str(tmpdir / 'logs')
 
     (X_train, y_train), (X_test, y_test) = get_test_data(
         num_train=train_samples,
@@ -379,16 +381,17 @@ def test_TensorBoard():
     model.fit_generator(data_generator(True), len(X_train), epochs=2,
                         callbacks=cbks)
 
-    assert os.path.exists(filepath)
+    assert os.path.isdir(filepath)
     shutil.rmtree(filepath)
+    assert not tmpdir.listdir()
 
 
 @keras_test
 @pytest.mark.skipif((K.backend() != 'tensorflow'),
                     reason='Requires tensorflow backend')
-def test_TensorBoard_convnet():
+def test_TensorBoard_convnet(tmpdir):
     np.random.seed(np.random.randint(1, 1e7))
-    filepath = './logs_' + str(np.random.randint(1, 1e4))
+    filepath = str(tmpdir / 'logs')
 
     input_shape = (16, 16, 3)
     (x_train, y_train), (x_test, y_test) = get_test_data(num_train=500,
@@ -421,8 +424,9 @@ def test_TensorBoard_convnet():
                         validation_data=(x_test, y_test),
                         callbacks=cbks,
                         verbose=0)
-    assert os.path.exists(filepath)
+    assert os.path.isdir(filepath)
     shutil.rmtree(filepath)
+    assert not tmpdir.listdir()
 
 
 @keras_test
@@ -510,10 +514,10 @@ def test_LambdaCallback():
 @keras_test
 @pytest.mark.skipif((K.backend() != 'tensorflow'),
                     reason="Requires tensorflow backend")
-def test_TensorBoard_with_ReduceLROnPlateau():
+def test_TensorBoard_with_ReduceLROnPlateau(tmpdir):
     import shutil
     np.random.seed(np.random.randint(1, 1e7))
-    filepath = './logs_' + str(np.random.randint(1, 1e4))
+    filepath = str(tmpdir / 'logs')
 
     (X_train, y_train), (X_test, y_test) = get_test_data(num_train=train_samples,
                                                          num_test=test_samples,
@@ -542,8 +546,9 @@ def test_TensorBoard_with_ReduceLROnPlateau():
     model.fit(X_train, y_train, batch_size=batch_size,
               validation_data=(X_test, y_test), callbacks=cbks, epochs=2)
 
-    assert os.path.exists(filepath)
+    assert os.path.isdir(filepath)
     shutil.rmtree(filepath)
+    assert not tmpdir.listdir()
 
 
 if __name__ == '__main__':

tests/keras/test_sequential_model.py

@@ -21,6 +21,17 @@ batch_size = 32
 epochs = 1
 
 
+@pytest.fixture
+def in_tmpdir(tmpdir):
+    """Runs a function in a temporary directory.
+
+    Checks that the directory is empty afterwards.
+    """
+    with tmpdir.as_cwd():
+        yield None
+    assert not tmpdir.listdir()
+
+
 @keras_test
 def test_sequential_pop():
     model = Sequential()
@@ -87,12 +98,12 @@ def test_sequential_fit_generator():
     model.fit_generator(data_generator(True), 5, epochs,
                         validation_data=data_generator(False),
                         validation_steps=3)
-    model.fit_generator(data_generator(True), 5, epochs, max_q_size=2)
+    model.fit_generator(data_generator(True), 5, epochs, max_queue_size=2)
     model.evaluate(x_train, y_train)
 
 
 @keras_test
-def test_sequential():
+def test_sequential(in_tmpdir):
     (x_train, y_train), (x_test, y_test) = _get_test_data()
 
     # TODO: factor out
@@ -122,8 +133,8 @@ def test_sequential():
 
     loss = model.evaluate(x_test, y_test)
 
-    prediction = model.predict_generator(data_generator(x_test, y_test), 1, max_q_size=2, verbose=1)
-    gen_loss = model.evaluate_generator(data_generator(x_test, y_test, 50), 1, max_q_size=2)
+    prediction = model.predict_generator(data_generator(x_test, y_test), 1, max_queue_size=2, verbose=1)
+    gen_loss = model.evaluate_generator(data_generator(x_test, y_test, 50), 1, max_queue_size=2)
     pred_loss = K.eval(K.mean(losses.get(model.loss)(K.variable(y_test), K.variable(prediction))))
 
     assert(np.isclose(pred_loss, loss))
@@ -160,7 +171,7 @@ def test_sequential():
 
 
 @keras_test
-def test_nested_sequential():
+def test_nested_sequential(in_tmpdir):
     (x_train, y_train), (x_test, y_test) = _get_test_data()
 
     inner = Sequential()

tests/keras/utils/data_utils_test.py

@@ -1,17 +1,41 @@
 """Tests for functions in data_utils.py.
 """
 import os
-import pytest
+import sys
 import tarfile
+import threading
 import zipfile
-from six.moves.urllib.request import pathname2url
+from itertools import cycle
+
+import numpy as np
+import pytest
 from six.moves.urllib.parse import urljoin
+from six.moves.urllib.request import pathname2url
+
+from keras.utils import Sequence
+from keras.utils import GeneratorEnqueuer
+from keras.utils import OrderedEnqueuer
+from keras.utils.data_utils import _hash_file
 from keras.utils.data_utils import get_file
 from keras.utils.data_utils import validate_file
-from keras.utils.data_utils import _hash_file
+
+if sys.version_info < (3,):
+    def next(x):
+        return x.next()
 
 
-def test_data_utils():
+@pytest.fixture
+def in_tmpdir(tmpdir):
+    """Runs a function in a temporary directory.
+
+    Checks that the directory is empty afterwards.
+    """
+    with tmpdir.as_cwd():
+        yield None
+    assert not tmpdir.listdir()
+
+
+def test_data_utils(in_tmpdir):
     """Tests get_file from a url, plus extraction and validation.
     """
     dirname = 'data_utils'
@@ -53,5 +77,154 @@ def test_data_utils():
     os.remove('test.txt')
     os.remove('test.zip')
 
+
+"""Enqueuers Tests"""
+
+
+class threadsafe_iter:
+    """Takes an iterator/generator and makes it thread-safe by
+    serializing call to the `next` method of given iterator/generator.
+    """
+
+    def __init__(self, it):
+        self.it = it
+        self.lock = threading.Lock()
+
+    def __iter__(self):
+        return self
+
+    def __next__(self):
+        return self.next()
+
+    def next(self):
+        with self.lock:
+            return next(self.it)
+
+
+def threadsafe_generator(f):
+    """A decorator that takes a generator function and makes it thread-safe.
+    """
+
+    def g(*a, **kw):
+        return threadsafe_iter(f(*a, **kw))
+
+    return g
+
+
+class TestSequence(Sequence):
+    def __init__(self, shape):
+        self.shape = shape
+
+    def __getitem__(self, item):
+        return np.ones(self.shape, dtype=np.uint8) * item
+
+    def __len__(self):
+        return 100
+
+
+class FaultSequence(Sequence):
+    def __getitem__(self, item):
+        raise IndexError(item, 'is not present')
+
+    def __len__(self):
+        return 100
+
+
+@threadsafe_generator
+def create_generator_from_sequence_threads(ds):
+    for i in cycle(range(len(ds))):
+        yield ds[i]
+
+
+def create_generator_from_sequence_pcs(ds):
+    for i in cycle(range(len(ds))):
+        yield ds[i]
+
+
+def test_generator_enqueuer_threads():
+    enqueuer = GeneratorEnqueuer(create_generator_from_sequence_threads(
+        TestSequence([3, 200, 200, 3])), use_multiprocessing=False)
+    enqueuer.start(3, 10)
+    gen_output = enqueuer.get()
+    acc = []
+    for i in range(100):
+        acc.append(int(next(gen_output)[0, 0, 0, 0]))
+
+    """
+     Not comparing the order since it is not guarantee.
+     It may get ordered, but not a lot, one thread can take the GIL before he was supposed to.
+    """
+    assert len(set(acc) - set(range(100))) == 0, "Output is not the same"
+    enqueuer.stop()
+
+
+def test_generator_enqueuer_processes():
+    enqueuer = GeneratorEnqueuer(create_generator_from_sequence_pcs(
+        TestSequence([3, 200, 200, 3])), use_multiprocessing=True)
+    enqueuer.start(3, 10)
+    gen_output = enqueuer.get()
+    acc = []
+    for i in range(100):
+        acc.append(int(next(gen_output)[0, 0, 0, 0]))
+    assert acc != list(range(100)), "Order was keep in GeneratorEnqueuer with processes"
+    enqueuer.stop()
+
+
+def test_generator_enqueuer_fail_threads():
+    enqueuer = GeneratorEnqueuer(create_generator_from_sequence_threads(
+        FaultSequence()), use_multiprocessing=False)
+    enqueuer.start(3, 10)
+    gen_output = enqueuer.get()
+    with pytest.raises(StopIteration):
+        next(gen_output)
+
+
+def test_generator_enqueuer_fail_processes():
+    enqueuer = GeneratorEnqueuer(create_generator_from_sequence_pcs(
+        FaultSequence()), use_multiprocessing=True)
+    enqueuer.start(3, 10)
+    gen_output = enqueuer.get()
+    with pytest.raises(StopIteration):
+        next(gen_output)
+
+
+def test_ordered_enqueuer_threads():
+    enqueuer = OrderedEnqueuer(TestSequence([3, 200, 200, 3]), use_multiprocessing=False)
+    enqueuer.start(3, 10)
+    gen_output = enqueuer.get()
+    acc = []
+    for i in range(100):
+        acc.append(next(gen_output)[0, 0, 0, 0])
+    assert acc == list(range(100)), "Order was not keep in GeneratorEnqueuer with threads"
+    enqueuer.stop()
+
+
+def test_ordered_enqueuer_processes():
+    enqueuer = OrderedEnqueuer(TestSequence([3, 200, 200, 3]), use_multiprocessing=True)
+    enqueuer.start(3, 10)
+    gen_output = enqueuer.get()
+    acc = []
+    for i in range(100):
+        acc.append(next(gen_output)[0, 0, 0, 0])
+    assert acc == list(range(100)), "Order was not keep in GeneratorEnqueuer with processes"
+    enqueuer.stop()
+
+
+def test_ordered_enqueuer_fail_threads():
+    enqueuer = OrderedEnqueuer(FaultSequence(), use_multiprocessing=False)
+    enqueuer.start(3, 10)
+    gen_output = enqueuer.get()
+    with pytest.raises(StopIteration):
+        next(gen_output)
+
+
+def test_ordered_enqueuer_fail_processes():
+    enqueuer = OrderedEnqueuer(FaultSequence(), use_multiprocessing=True)
+    enqueuer.start(3, 10)
+    gen_output = enqueuer.get()
+    with pytest.raises(StopIteration):
+        next(gen_output)
+
+
 if __name__ == '__main__':
     pytest.main([__file__])

tests/keras/utils/generic_utils_test.py

@@ -1,5 +1,6 @@
+import sys
 import pytest
-from keras.utils.generic_utils import custom_object_scope
+from keras.utils.generic_utils import custom_object_scope, has_arg
 from keras import activations
 from keras import regularizers
 
@@ -20,5 +21,46 @@ def test_custom_objects_scope():
         assert cl.__class__ == CustomClass
 
 
+@pytest.mark.parametrize('fn, name, accept_all, expected', [
+    ('f(x)', 'x', False, True),
+    ('f(x)', 'y', False, False),
+    ('f(x)', 'y', True, False),
+    ('f(x, y)', 'y', False, True),
+    ('f(x, y=1)', 'y', False, True),
+    ('f(x, **kwargs)', 'x', False, True),
+    ('f(x, **kwargs)', 'y', False, False),
+    ('f(x, **kwargs)', 'y', True, True),
+    ('f(x, y=1, **kwargs)', 'y', False, True),
+    # Keyword-only arguments (Python 3 only)
+    ('f(x, *args, y=1)', 'y', False, True),
+    ('f(x, *args, y=1)', 'z', True, False),
+    ('f(x, *, y=1)', 'x', False, True),
+    ('f(x, *, y=1)', 'y', False, True),
+    # lambda
+    (lambda x: x, 'x', False, True),
+    (lambda x: x, 'y', False, False),
+    (lambda x: x, 'y', True, False),
+])
+def test_has_arg(fn, name, accept_all, expected):
+    if isinstance(fn, str):
+        context = dict()
+        try:
+            exec('def {}: pass'.format(fn), context)
+        except SyntaxError:
+            if sys.version_info >= (3,):
+                raise
+            pytest.skip('Function is not compatible with Python 2')
+        context.pop('__builtins__', None)  # Sometimes exec adds builtins to the context
+        fn, = context.values()
+
+    assert has_arg(fn, name, accept_all) is expected
+
+
+@pytest.mark.xfail(sys.version_info < (3, 3),
+                   reason='inspect API does not reveal positional-only arguments')
+def test_has_arg_positional_only():
+    assert has_arg(pow, 'x') is False
+
+
 if __name__ == '__main__':
     pytest.main([__file__])

tests/keras/utils/io_utils_test.py

@@ -10,6 +10,17 @@ import warnings
 import h5py
 
 
+@pytest.fixture
+def in_tmpdir(tmpdir):
+    """Runs a function in a temporary directory.
+
+    Checks that the directory is empty afterwards.
+    """
+    with tmpdir.as_cwd():
+        yield None
+    assert not tmpdir.listdir()
+
+
 def create_dataset(h5_path='test.h5'):
     X = np.random.randn(200, 10).astype('float32')
     y = np.random.randint(0, 2, size=(200, 1))
@@ -23,7 +34,7 @@ def create_dataset(h5_path='test.h5'):
     f.close()
 
 
-def test_io_utils():
+def test_io_utils(in_tmpdir):
     '''Tests the HDF5Matrix code using the sample from @jfsantos at
     https://gist.github.com/jfsantos/e2ef822c744357a4ed16ec0c885100a3
     '''
@@ -42,6 +53,10 @@ def test_io_utils():
     assert y_train.shape == (150, 1), 'HDF5Matrix shape should match input array'
     # But they do not support negative indices, so don't try print(X_train[-1])
 
+    assert y_train.dtype == np.dtype('i'), 'HDF5Matrix dtype should match input array'
+    assert y_train.ndim == 2, 'HDF5Matrix ndim should match input array'
+    assert y_train.size == 150, 'HDF5Matrix ndim should match input array'
+
     model = Sequential()
     model.add(Dense(64, input_shape=(10,), activation='relu'))
     model.add(Dense(1, activation='sigmoid'))

tests/keras/utils/layer_utils_test.py

@@ -0,0 +1,65 @@
+import pytest
+import numpy as np
+from numpy.testing import assert_allclose
+from keras import backend as K
+from keras.layers import Conv2D
+from keras.layers import Dense
+from keras.layers import Flatten
+from keras.models import Sequential
+from keras.utils import layer_utils
+from keras.utils.test_utils import keras_test
+
+
+@keras_test
+def test_convert_weights():
+    def get_model(shape, data_format):
+        model = Sequential()
+        model.add(Conv2D(filters=2,
+                         kernel_size=(4, 3),
+                         input_shape=shape,
+                         data_format=data_format))
+        model.add(Flatten())
+        model.add(Dense(5))
+        return model
+
+    for data_format in ['channels_first', 'channels_last']:
+        if data_format == 'channels_first':
+            shape = (3, 5, 5)
+            target_shape = (5, 5, 3)
+            prev_shape = (2, 3, 2)
+            flip = lambda x: np.flip(np.flip(x, axis=2), axis=3)
+            transpose = lambda x: np.transpose(x, (0, 2, 3, 1))
+            target_data_format = 'channels_last'
+        elif data_format == 'channels_last':
+            shape = (5, 5, 3)
+            target_shape = (3, 5, 5)
+            prev_shape = (2, 2, 3)
+            flip = lambda x: np.flip(np.flip(x, axis=1), axis=2)
+            transpose = lambda x: np.transpose(x, (0, 3, 1, 2))
+            target_data_format = 'channels_first'
+
+        model1 = get_model(shape, data_format)
+        model2 = get_model(target_shape, target_data_format)
+        conv = K.function([model1.input], [model1.layers[0].output])
+
+        x = np.random.random((1,) + shape)
+
+        # Test equivalence of convert_all_kernels_in_model
+        convout1 = conv([x])[0]
+        layer_utils.convert_all_kernels_in_model(model1)
+        convout2 = flip(conv([flip(x)])[0])
+
+        assert_allclose(convout1, convout2, atol=1e-5)
+
+        # Test equivalence of convert_dense_weights_data_format
+        out1 = model1.predict(x)
+        layer_utils.convert_dense_weights_data_format(model1.layers[2], prev_shape, target_data_format)
+        for (src, dst) in zip(model1.layers, model2.layers):
+            dst.set_weights(src.get_weights())
+        out2 = model2.predict(transpose(x))
+
+        assert_allclose(out1, out2, atol=1e-5)
+
+
+if __name__ == '__main__':
+    pytest.main([__file__])

tests/test_model_saving.py

@@ -78,11 +78,11 @@ def test_sequential_model_saving_2():
 
 @keras_test
 def test_functional_model_saving():
-    input = Input(shape=(3,))
-    x = Dense(2)(input)
-    output = Dense(3)(x)
+    inputs = Input(shape=(3,))
+    x = Dense(2)(inputs)
+    outputs = Dense(3)(x)
 
-    model = Model(input, output)
+    model = Model(inputs, outputs)
     model.compile(loss=losses.MSE,
                   optimizer=optimizers.RMSprop(lr=0.0001),
                   metrics=[metrics.categorical_accuracy])
@@ -103,12 +103,12 @@ def test_functional_model_saving():
 
 @keras_test
 def test_saving_multiple_metrics_outputs():
-    input = Input(shape=(5,))
-    x = Dense(5)(input)
+    inputs = Input(shape=(5,))
+    x = Dense(5)(inputs)
     output1 = Dense(1, name='output1')(x)
     output2 = Dense(1, name='output2')(x)
 
-    model = Model(inputs=input, outputs=[output1, output2])
+    model = Model(inputs=inputs, outputs=[output1, output2])
 
     metrics = {'output1': ['mse', 'binary_accuracy'],
                'output2': ['mse', 'binary_accuracy']
@@ -270,11 +270,11 @@ def square_fn(x):
 
 @keras_test
 def test_saving_lambda_custom_objects():
-    input = Input(shape=(3,))
-    x = Lambda(lambda x: square_fn(x), output_shape=(3,))(input)
-    output = Dense(3)(x)
+    inputs = Input(shape=(3,))
+    x = Lambda(lambda x: square_fn(x), output_shape=(3,))(inputs)
+    outputs = Dense(3)(x)
 
-    model = Model(input, output)
+    model = Model(inputs, outputs)
     model.compile(loss=losses.MSE,
                   optimizer=optimizers.RMSprop(lr=0.0001),
                   metrics=[metrics.categorical_accuracy])
@@ -297,10 +297,10 @@ def test_saving_lambda_custom_objects():
 def test_saving_lambda_numpy_array_arguments():
     mean = np.random.random((4, 2, 3))
     std = np.abs(np.random.random((4, 2, 3))) + 1e-5
-    input = Input(shape=(4, 2, 3))
-    output = Lambda(lambda image, mu, std: (image - mu) / std,
-                    arguments={'mu': mean, 'std': std})(input)
-    model = Model(input, output)
+    inputs = Input(shape=(4, 2, 3))
+    outputs = Lambda(lambda image, mu, std: (image - mu) / std,
+                     arguments={'mu': mean, 'std': std})(inputs)
+    model = Model(inputs, outputs)
     model.compile(loss='mse', optimizer='sgd', metrics=['acc'])
 
     _, fname = tempfile.mkstemp('.h5')

tests/test_multiprocessing.py

@@ -7,12 +7,24 @@ from keras.layers.core import Dense
 from keras.utils.test_utils import keras_test
 
 
+@pytest.fixture
+def in_tmpdir(tmpdir):
+    """Runs a function in a temporary directory.
+
+    Checks that the directory is empty afterwards.
+    """
+    with tmpdir.as_cwd():
+        yield None
+    assert not tmpdir.listdir()
+
+
 @keras_test
 def test_multiprocessing_training():
     arr_data = np.random.randint(0, 256, (50, 2))
     arr_labels = np.random.randint(0, 2, 50)
+    arr_weights = np.random.random(50)
 
-    def custom_generator():
+    def custom_generator(use_weights=False):
         batch_size = 10
         n_samples = 50
 
@@ -22,7 +34,11 @@ def test_multiprocessing_training():
             end = start + batch_size
             X = arr_data[start: end]
             y = arr_labels[start: end]
-            yield X, y
+            if use_weights:
+                w = arr_weights[start: end]
+                yield X, y, w
+            else:
+                yield X, y
 
     # Build a NN
     model = Sequential()
@@ -33,20 +49,60 @@ def test_multiprocessing_training():
                         steps_per_epoch=5,
                         epochs=1,
                         verbose=1,
-                        max_q_size=10,
+                        max_queue_size=10,
                         workers=4,
-                        pickle_safe=True)
+                        use_multiprocessing=True)
 
     model.fit_generator(custom_generator(),
                         steps_per_epoch=5,
                         epochs=1,
                         verbose=1,
-                        max_q_size=10,
-                        pickle_safe=False)
+                        max_queue_size=10,
+                        use_multiprocessing=False)
+
+    model.fit_generator(custom_generator(True),
+                        steps_per_epoch=5,
+                        validation_data=(arr_data[:10],
+                                         arr_labels[:10],
+                                         arr_weights[:10]),
+                        validation_steps=1)
+
+    model.fit_generator(custom_generator(True),
+                        steps_per_epoch=5,
+                        validation_data=custom_generator(True),
+                        validation_steps=1)
+
+    # Test invalid use cases
+    def invalid_generator():
+        while True:
+            yield arr_data[:10], arr_data[:10], arr_labels[:10], arr_labels[:10]
+
+    # not specified `validation_steps`
+    with pytest.raises(ValueError):
+        model.fit_generator(custom_generator(),
+                            steps_per_epoch=5,
+                            validation_data=custom_generator())
+
+    # validation data is neither a tuple nor a triple.
+    with pytest.raises(ValueError):
+        model.fit_generator(custom_generator(),
+                            steps_per_epoch=5,
+                            validation_data=(arr_data[:10],
+                                             arr_data[:10],
+                                             arr_labels[:10],
+                                             arr_weights[:10]),
+                            validation_steps=1)
+
+    # validation generator is neither a tuple nor a triple.
+    with pytest.raises(ValueError):
+        model.fit_generator(custom_generator(),
+                            steps_per_epoch=5,
+                            validation_data=invalid_generator(),
+                            validation_steps=1)
 
 
 @keras_test
-def test_multiprocessing_training_fromfile():
+def test_multiprocessing_training_fromfile(in_tmpdir):
     arr_data = np.random.randint(0, 256, (50, 2))
     arr_labels = np.random.randint(0, 2, 50)
     np.savez('data.npz', **{'data': arr_data, 'labels': arr_labels})
@@ -75,16 +131,16 @@ def test_multiprocessing_training_fromfile():
                         steps_per_epoch=5,
                         epochs=1,
                         verbose=1,
-                        max_q_size=10,
+                        max_queue_size=10,
                         workers=2,
-                        pickle_safe=True)
+                        use_multiprocessing=True)
 
     model.fit_generator(custom_generator(),
                         steps_per_epoch=5,
                         epochs=1,
                         verbose=1,
-                        max_q_size=10,
-                        pickle_safe=False)
+                        max_queue_size=10,
+                        use_multiprocessing=False)
 
     os.remove('data.npz')
 
@@ -110,13 +166,13 @@ def test_multiprocessing_predicting():
     model.compile(loss='mse', optimizer='adadelta')
     model.predict_generator(custom_generator(),
                             steps=5,
-                            max_q_size=10,
+                            max_queue_size=10,
                             workers=2,
-                            pickle_safe=True)
+                            use_multiprocessing=True)
     model.predict_generator(custom_generator(),
                             steps=5,
-                            max_q_size=10,
-                            pickle_safe=False)
+                            max_queue_size=10,
+                            use_multiprocessing=False)
 
 
 @keras_test
@@ -143,13 +199,13 @@ def test_multiprocessing_evaluating():
 
     model.evaluate_generator(custom_generator(),
                              steps=5,
-                             max_q_size=10,
+                             max_queue_size=10,
                              workers=2,
-                             pickle_safe=True)
+                             use_multiprocessing=True)
     model.evaluate_generator(custom_generator(),
                              steps=5,
-                             max_q_size=10,
-                             pickle_safe=False)
+                             max_queue_size=10,
+                             use_multiprocessing=False)
 
 
 @keras_test
@@ -170,16 +226,16 @@ def test_multiprocessing_fit_error():
 
     samples = batch_size * (good_batches + 1)
 
-    with pytest.raises(Exception):
+    with pytest.raises(StopIteration):
         model.fit_generator(
             custom_generator(), samples, 1,
-            workers=4, pickle_safe=True,
+            workers=4, use_multiprocessing=True,
         )
 
-    with pytest.raises(Exception):
+    with pytest.raises(StopIteration):
         model.fit_generator(
             custom_generator(), samples, 1,
-            pickle_safe=False,
+            use_multiprocessing=False,
         )
 
 
@@ -199,45 +255,45 @@ def test_multiprocessing_evaluate_error():
     model.add(Dense(1, input_shape=(2, )))
     model.compile(loss='mse', optimizer='adadelta')
 
-    with pytest.raises(Exception):
+    with pytest.raises(StopIteration):
         model.evaluate_generator(
             custom_generator(), good_batches + 1, 1,
-            workers=4, pickle_safe=True,
+            workers=4, use_multiprocessing=True,
         )
 
-    with pytest.raises(Exception):
+    with pytest.raises(StopIteration):
         model.evaluate_generator(
             custom_generator(), good_batches + 1, 1,
-            pickle_safe=False,
+            use_multiprocessing=False,
         )
 
 
 @keras_test
 def test_multiprocessing_predict_error():
-    batch_size = 10
     good_batches = 3
+    workers = 4
 
     def custom_generator():
         """Raises an exception after a few good batches"""
         for i in range(good_batches):
-            yield (np.random.randint(batch_size, 256, (50, 2)),
-                   np.random.randint(batch_size, 2, 50))
+            yield (np.random.randint(1, 256, size=(2, 5)),
+                   np.random.randint(1, 256, size=(2, 5)))
         raise RuntimeError
 
     model = Sequential()
-    model.add(Dense(1, input_shape=(2, )))
+    model.add(Dense(1, input_shape=(5,)))
     model.compile(loss='mse', optimizer='adadelta')
 
-    with pytest.raises(Exception):
+    with pytest.raises(StopIteration):
         model.predict_generator(
-            custom_generator(), good_batches + 1, 1,
-            workers=4, pickle_safe=True,
+            custom_generator(), good_batches * workers + 1, 1,
+            workers=workers, use_multiprocessing=True,
         )
 
-    with pytest.raises(Exception):
+    with pytest.raises(StopIteration):
         model.predict_generator(
             custom_generator(), good_batches + 1, 1,
-            pickle_safe=False,
+            use_multiprocessing=False,
         )