Merge branch 'master' of https://github.com/fchollet/keras

batch_input_shape fix

.gitignore

@@ -9,3 +9,11 @@ keras/datasets/temp/*
 docs/site/*
 docs/theme/*
 tags
+Keras.egg-info
+
+# test-related
+.coverage
+.cache
+
+# developer environments
+.idea

.travis.yml

@@ -1,9 +1,18 @@
 sudo: required
 dist: trusty
 language: python
-python:
-  - "2.7"
-  - "3.4"
+matrix:
+    include:
+        - python: 3.4
+          env: KERAS_BACKEND=theano
+        - python: 3.4
+          env: KERAS_BACKEND=tensorflow
+        - python: 2.7
+          env: KERAS_BACKEND=theano
+        - python: 2.7
+          env: KERAS_BACKEND=tensorflow
+        - python: 2.7
+          env: KERAS_BACKEND=theano INTEGRATION_TESTS=true
 install:
   # code below is taken from http://conda.pydata.org/docs/travis.html
   # We do this conditionally because it saves us some downloading if the
@@ -23,20 +32,35 @@ install:
 
   - conda create -q -n test-environment python=$TRAVIS_PYTHON_VERSION numpy scipy matplotlib pandas pytest h5py
   - source activate test-environment
-  - pip install pytest-cov python-coveralls
+  - pip install pytest-cov python-coveralls pytest-xdist coverage==3.7.1 #we need this version of coverage for coveralls.io to work
   - pip install git+git://github.com/Theano/Theano.git
+
+  # install PIL for preprocessing tests
+  - if [[ "$TRAVIS_PYTHON_VERSION" == "2.7" ]]; then
+      conda install pil;
+    elif [[ "$TRAVIS_PYTHON_VERSION" == "3.4" ]]; then
+      conda install Pillow;
+    fi
+
   - python setup.py install
 
   # install TensorFlow
   - if [[ "$TRAVIS_PYTHON_VERSION" == "2.7" ]]; then
-      pip install https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-0.5.0-cp27-none-linux_x86_64.whl;
+      pip install https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-0.6.0-cp27-none-linux_x86_64.whl;
+    elif [[ "$TRAVIS_PYTHON_VERSION" == "3.4" ]]; then
+      pip install https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-0.6.0-cp34-none-linux_x86_64.whl;
     fi
 # command to run tests
 script:
-  - PYTHONPATH=$PWD:$PYTHONPATH py.test -v --cov-report term-missing --cov keras tests/
-  - if [[ "$TRAVIS_PYTHON_VERSION" == "2.7" ]]; then
-      sed -i -e 's/theano/tensorflow/g' ~/.keras/keras.json;
-      PYTHONPATH=$PWD:$PYTHONPATH py.test -v --cov-report term-missing --cov keras tests/;
+  # run keras backend init to initialize backend config
+  - python -c "import keras.backend"
+  # set up keras backend
+  - sed -i -e 's/"backend":[[:space:]]*"[^"]*/"backend":\ "'$KERAS_BACKEND'/g' ~/.keras/keras.json;
+  - echo -e "Running tests with the following config:\n$(cat ~/.keras/keras.json)"
+  - if [[ "$INTEGRATION_TESTS" == "true" ]]; then
+       PYTHONPATH=$PWD:$PYTHONPATH py.test tests/integration_tests;
+    else
+       PYTHONPATH=$PWD:$PYTHONPATH py.test tests/ --ignore=tests/integration_tests;
     fi
 after_success:
   - coveralls

CONTRIBUTING.md

@@ -0,0 +1,60 @@
+# On Github Issues and Pull Requests
+
+Found a bug? Have a new feature to suggest? Want to contribute changes to the codebase? Make sure to read this first.
+
+## Bug reporting
+
+Your code doesn't work, and you have determined that the issue lies with Keras? Follow these steps to report a bug.
+
+1. Your bug may already be fixed. Make sure to update to the current Keras master branch, as well as the latest Theano/TensorFlow master branch.
+To easily update Theano: `pip install git+git://github.com/Theano/Theano.git --upgrade`
+
+2. Search for similar issues. Make sure to delete `is:open` on the issue search to find solved tickets as well. It's possible somebody has encountered this bug already. Also remember to check out Keras' [FAQ](http://keras.io/faq/). Still having a problem? Open an issue on Github to let us know.
+
+3. Make sure you provide us with useful information about your configuration: what OS are you using? What Keras backend are you using? Are you running on GPU? If so, what is your version of Cuda, of cuDNN? What is your GPU?
+
+4. Provide us with a script to reproduce the issue. This script should be runnable as-is and should not require external data download (use randomly generated data if you need to run a model on some test data). We recommend that you use Github Gists to post your code. Any issue that cannot be reproduced is likely to be closed.
+
+5. If possible, take a stab at fixing the bug yourself --if you can!
+
+The more information you provide, the easier it is for us to validate that there is a bug and the faster we'll be able to take action. If you want your issue to be resolved quickly, following the steps above is crucial.
+
+
+## Requesting a Feature
+
+You can also use Github issues to request features you would like to see in Keras, or changes in the Keras API.
+
+1. Provide a clear and detailed explanation of the feature you want and why it's important to add. Keep in mind that we want features that will be useful to the majority of our users and not just a small subset. If you're just targeting a minority of users, consider writing an add-on library for Keras. It is crucial for Keras to avoid bloating the API and codebase.
+
+2. Provide code snippets demonstrating the API you have in mind and illustrating the use cases of your feature. Of course, you don't need to write any real code at this point!
+
+3. After discussing the feature you may choose to attempt a Pull Request. If you're at all able, start writing some code. We always have more work to do than time to do it. If you can write some code then that will speed the process along.
+
+## Pull Requests
+
+We love pull requests. Here's a quick guide:
+
+1. If your PR introduces a change in functionality, make sure you start by opening an issue to discuss whether the change should be made, and how to handle it. This will save you from having your PR closed down the road! Of course, if your PR is a simple bug fix, you don't need to do that.
+
+2. Write the code. This is the hard part! We use PEP8 syntax conventions, but we aren't dogmatic when it comes to line length. Make sure your lines stay reasonably sized, though. To make your life easier, we recommend installing a PEP8 linter.
+
+3. Make sure any new function or class you introduce has proper docstrings. Make sure any code you touch still has up-to-date docstrings and documentation.
+
+4. Write tests. Your code should have full unit test coverage. If you want to see your PR merged promptly, this is crucial.
+
+5. Run our test suite locally. It's easy: from the Keras folder, simply run: `py.test tests/`.
+  - You will need to install `pytest`, `coveralls`, `pytest-cov`, `pytest-xdist`: `pip install pytest pytest-cov python-coveralls pytest-xdist`
+
+6. Make sure all tests are passing:
+  - with the Theano backend, on Python 2.7 and Python 3.5
+  - with the TensorFlow backend, on Python 2.7
+
+7. When committing, use appropriate, descriptive commit messages. Make sure that your branch history is not a string of "bug fix", "fix", "oops", etc. When submitting your PR, squash your commits into a single commit with an appropriate commit message, to make sure the project history stays clean and readable. See ['rebase and squash'](http://rebaseandsqua.sh/) for technical help on how to squash your commits.
+
+8. Update the documentation. If introducing new functionality, make sure you include code snippets demonstrating the usage of your new feature.
+
+9. Submit your PR. If your changes have been approved in a previous discussion, and if you have have complete (and passing) unit tests, your PR is likely to be merged promptly. Otherwise, well...
+
+## Adding new examples
+
+Even if you don't contribute to the Keras source code, if you have an application of Keras that is concise and powerful, please consider adding it to our collection of examples. Existing examples show idiomatic Keras code: make sure to keep your own script in the same spirit.

README.md

@@ -1,10 +1,13 @@
 # Keras: Deep Learning library for Theano and TensorFlow
 
+![Build status](https://api.travis-ci.org/fchollet/keras.svg)
+
 ## You have just found Keras.
 
-Keras is a minimalist, highly modular neural networks library, written in Python and capable of running either on top of either [TensorFlow](https://github.com/tensorflow/tensorflow) or [Theano](https://github.com/Theano/Theano). It was developed with a focus on enabling fast experimentation. Being able to go from idea to result with the least possible delay is key to doing good research.
+Keras is a minimalist, highly modular neural networks library, written in Python and capable of running on top of either [TensorFlow](https://github.com/tensorflow/tensorflow) or [Theano](https://github.com/Theano/Theano). It was developed with a focus on enabling fast experimentation. Being able to go from idea to result with the least possible delay is key to doing good research.
 
 Use Keras if you need a deep learning library that:
+
 - allows for easy and fast prototyping (through total modularity, minimalism, and extensibility).
 - supports both convolutional networks and recurrent networks, as well as combinations of the two.
 - supports arbitrary connectivity schemes (including multi-input and multi-output training).
@@ -12,9 +15,7 @@ Use Keras if you need a deep learning library that:
 
 Read the documentation at [Keras.io](http://keras.io).
 
-Keras is compatible with:
-    - __Python 2.7-3.5__ with the Theano backend
-    - __Python 2.7__ with the TensorFlow backend
+Keras is compatible with: __Python 2.7-3.5__.
 
 
 ------------------
@@ -36,7 +37,7 @@ Keras is compatible with:
 
 ## Getting started: 30 seconds to Keras
 
-The core datastructure of Keras is a __model__, a way to organize layers. There are two types of models: [`Sequential`](/models/#sequential) and [`Graph`](/models/#graph).
+The core data structure of Keras is a __model__, a way to organize layers. There are two types of models: [`Sequential`](http://keras.io/models/#sequential) and [`Graph`](http://keras.io/models/#graph).
 
 Here's the `Sequential` model (a linear pile of layers):
 
@@ -108,7 +109,8 @@ Keras uses the following dependencies:
 - HDF5 and h5py (optional, required if you use model saving/loading functions)
 - Optional but recommended if you use CNNs: cuDNN.
 
-When using the Theano backend:
+*When using the Theano backend:*
+
 - Theano
     - [See installation instructions](http://deeplearning.net/software/theano/install.html#install).
 
@@ -117,11 +119,12 @@ When using the Theano backend:
 sudo pip install git+git://github.com/Theano/Theano.git
 ```
 
-When using the TensorFlow backend:
+*When using the TensorFlow backend:*
+
 - TensorFlow
     - [See installation instructions](https://github.com/tensorflow/tensorflow#download-and-setup).
 
-To install, `cd` to the Keras folder and run the install command:
+To install Keras, `cd` to the Keras folder and run the install command:
 ```
 sudo python setup.py install
 ```
@@ -145,20 +148,7 @@ By default, Keras will use Theano as its tensor manipulation library. [Follow th
 
 You can ask questions and join the development discussion on the [Keras Google group](https://groups.google.com/forum/#!forum/keras-users).
 
-------------------
-
-
-## Contribution Guidelines
-
-Keras welcomes all contributions from the community. 
-
-- Keep a pragmatic mindset and avoid bloat. Only add to the source if that is the only path forward.
-- New features should be documented. Make sure you update the documentation along with your Pull Request.
-- Any new function or class should have a proper docstring.
-- The documentation for every new feature should include a usage example in the form of a code snippet. 
-- All changes should be tested. Make sure any new feature you add has a corresponding unit test.
-- Please no Pull Requests about coding style.
-- Even if you don't contribute to the Keras source code, if you have an application of Keras that is concise and powerful, please consider adding it to our collection of [examples](https://github.com/fchollet/keras/tree/master/examples).
+You can also post bug reports and feature requests in [Github issues](https://github.com/fchollet/keras/issues). Make sure to read [our guidelines](https://github.com/fchollet/keras/blob/master/CONTRIBUTING.md) first.
 
 
 ------------------
@@ -172,4 +162,4 @@ Keras was initially developed as part of the research effort of project ONEIROS
 
 >_"Oneiroi are beyond our unravelling --who can be sure what tale they tell? Not all that men look for comes to pass. Two gates there are that give passage to fleeting Oneiroi; one is made of horn, one of ivory. The Oneiroi that pass through sawn ivory are deceitful, bearing a message that will not be fulfilled; those that come out through polished horn have truth behind them, to be accomplished for men who see them."_ Homer, Odyssey 19. 562 ff (Shewring translation).
 
-------------------
+------------------

docs/README.md

@@ -5,5 +5,7 @@ Our documentation uses extended Markdown, as implemented by [MkDocs](http://mkdo
 
 ## Building the documentation
 
-- install MkDocs: `sudo pip install mkdocs`
-- `cd` to the `docs/` folder and run: `mkdocs serve`
+- install MkDocs: `pip install mkdocs`
+- `cd` to the `docs/` folder and run:
+    - `python autogen.py`
+    - `mkdocs serve`

docs/autogen.py

@@ -0,0 +1,256 @@
+# -*- coding: utf-8 -*-
+from __future__ import print_function
+import re
+import inspect
+import os
+import shutil
+
+from keras.layers import convolutional
+from keras.layers import recurrent
+from keras.layers import core
+from keras.layers import noise
+from keras.layers import normalization
+from keras.layers import advanced_activations
+from keras.layers import containers
+from keras.layers import embeddings
+from keras import optimizers
+from keras import callbacks
+from keras import models
+
+MODULES = [(convolutional, 'keras.layers.convolutional'),
+           (recurrent, 'keras.layers.recurrent'),
+           (noise, 'keras.layers.noise'),
+           (normalization, 'keras.layers.normalization'),
+           (advanced_activations, 'keras.layers.advanced_activations'),
+           (containers, 'keras.layers.containers'),
+           (core, 'keras.layers.core'),
+           (embeddings, 'keras.layers.embeddings'),
+           (optimizers, 'keras.optimizers'),
+           (callbacks, 'keras.callbacks'),
+           (models, 'keras.models')]
+
+SKIP = ['build', 'get_params', 'MaskedLayer',
+        'SiameseHead', 'MaskedLambda',
+        'CallbackList']
+ROOT = 'http://keras.io/'
+INCLUDE_METHODS_FOR = [
+    'Layer',
+    'Graph',
+    'Sequential',
+    'Callback',
+]
+
+
+def get_earliest_class_that_defined_member(member, cls):
+    ancestors = get_classes_ancestors([cls])
+    result = None
+    for ancestor in ancestors:
+        if member in dir(ancestor):
+            result = ancestor
+    if not result:
+        return cls
+    return result
+
+
+def get_classes_ancestors(classes):
+    ancestors = []
+    for cls in classes:
+        ancestors += cls.__bases__
+    filtered_ancestors = []
+    for ancestor in ancestors:
+        if ancestor.__name__ in ['object']:
+            continue
+        filtered_ancestors.append(ancestor)
+    if filtered_ancestors:
+        return filtered_ancestors + get_classes_ancestors(filtered_ancestors)
+    else:
+        return filtered_ancestors
+
+
+def get_method_signature(method):
+    signature = inspect.getargspec(method)
+    defaults = signature.defaults
+    args = signature.args[1:]
+    if defaults:
+        kwargs = zip(args[-len(defaults):], defaults)
+        args = args[:-len(defaults)]
+    else:
+        kwargs = []
+    st = '%s.%s(' % (method.__module__, method.__name__)
+    for a in args:
+        st += str(a) + ', '
+    for a, v in kwargs:
+        if  type(v) == str:
+            v = '\'' + v + '\''
+        elif type(v) == unicode:
+            v = 'u\'' + v + '\''
+        st += str(a) + '=' + str(v) + ', '
+    if kwargs or args:
+        return st[:-2] + ')'
+    else:
+        return st + ')'
+
+
+def class_to_docs_link(cls):
+    module_name = cls.__module__
+    assert module_name[:6] == 'keras.'
+    module_name = module_name[6:]
+    link = ROOT + module_name.replace('.', '/') + '#' + cls.__name__.lower()
+    return link
+
+
+def class_to_source_link(cls):
+    module_name = cls.__module__
+    assert module_name[:6] == 'keras.'
+    path = module_name.replace('.', '/')
+    path += '.py'
+    line = inspect.getsourcelines(cls)[-1]
+    link = 'https://github.com/fchollet/keras/blob/master/' + path + '#L' + str(line)
+    return '[[source]](' + link + ')'
+
+
+def code_snippet(snippet):
+    result = '```python\n'
+    result += snippet + '\n'
+    result += '```\n'
+    return result
+
+
+def process_class_docstring(docstring):
+    docstring = re.sub(r'    # (.*)\n',
+                       r'    __\1__\n\n',
+                       docstring)
+
+    docstring = re.sub(r'    ([^\s\\]+):(.*)\n',
+                       r'    - __\1__:\2\n',
+                       docstring)
+    docstring = docstring.replace('    ' * 3, '\t')
+    docstring = docstring.replace('    ', '')
+    return docstring
+
+
+def process_method_docstring(docstring):
+    docstring = re.sub(r'    # (.*)\n',
+                       r'    __\1__\n\n',
+                       docstring)
+
+    docstring = re.sub(r'    ([^\s\\]+):(.*)\n',
+                       r'    - __\1__:\2\n',
+                       docstring)
+    docstring = docstring.replace('    ' * 4, '\t')
+    docstring = docstring.replace('    ', '')
+    return docstring
+
+print('Cleaning up existing sources directory.')
+if os.path.exists('sources'):
+    shutil.rmtree('sources')
+print('Populating sources directory with templates.')
+for subdir, dirs, fnames in os.walk('templates'):
+    for fname in fnames:
+        new_subdir = subdir.replace('templates', 'sources')
+        if not os.path.exists(new_subdir):
+            os.makedirs(new_subdir)
+        if fname[-3:] == '.md':
+            fpath = os.path.join(subdir, fname)
+            new_fpath = fpath.replace('templates', 'sources')
+            shutil.copy(fpath, new_fpath)
+
+print('Starting autogeneration.')
+covered_so_far = set()
+for module, module_name in MODULES:
+    class_pages = []
+    for name in dir(module):
+        if name in SKIP:
+            continue
+        if name[0] == '_':
+            continue
+        module_member = getattr(module, name)
+        if module_member in covered_so_far:
+            continue
+        if inspect.isclass(module_member):
+            cls = module_member
+            if cls.__module__ == module_name:
+
+                try:
+                    class_signature = get_method_signature(cls.__init__)
+                    class_signature = class_signature.replace('__init__', cls.__name__)
+                except:
+                    # in case the class inherits from object and does not
+                    # define __init__
+                    class_signature = module_name + '.' + cls.__name__ + '()'
+
+                methods = []
+                methods_not_defined_here = []
+                for name in dir(cls):
+                    if name in SKIP:
+                        continue
+                    if name[0] == '_':
+                        continue
+                    cls_member = getattr(cls, name)
+                    if inspect.ismethod(cls_member):
+                        method = cls_member
+                        signature = inspect.getargspec(method)
+                        defaults = signature.defaults
+                        args = signature.args[1:]
+                        if defaults:
+                            kwargs = zip(args[-len(defaults):], defaults)
+                            args = args[:-len(defaults)]
+                        else:
+                            kwargs = []
+
+                        defined_by = get_earliest_class_that_defined_member(method.__name__, cls)
+                        if cls == defined_by:
+                            methods.append(method)
+                        else:
+                            methods_not_defined_here.append((method, defined_by))
+
+                blocks = []
+                blocks.append('<span style="float:right;">' + class_to_source_link(cls) + '</span>')
+                blocks.append('# ' + cls.__name__ + '\n')
+                blocks.append(code_snippet(class_signature))
+                docstring = cls.__doc__
+                if docstring:
+                    blocks.append(process_class_docstring(docstring))
+
+                if cls.__name__ in INCLUDE_METHODS_FOR:
+                    if methods or methods_not_defined_here:
+                        blocks.append('### Methods\n')
+                        for method in methods:
+                            signature = get_method_signature(method)
+                            signature = signature.replace(module_name + '.', '')
+                            blocks.append(code_snippet(signature))
+                            docstring = method.__doc__
+                            if docstring:
+                                blocks.append(process_method_docstring(docstring))
+                        for method, defined_by in methods_not_defined_here:
+                            signature = get_method_signature(method)
+                            method_module_name = method.__module__
+                            signature = signature.replace(method_module_name + '.', '')
+                            link = '[' + defined_by.__name__ + '](' + class_to_docs_link(defined_by) + ')'
+                            blocks.append(code_snippet(signature))
+                            blocks.append('Defined by ' + link + '.\n')
+
+                mkdown = '\n'.join(blocks)
+                class_pages.append((id(cls), mkdown))
+                covered_so_far.add(module_member)
+
+    class_pages.sort(key=lambda x: x[0])
+    class_pages = [x[1] for x in class_pages]
+    module_page = '\n----\n\n'.join(class_pages)
+
+    # save module page.
+    # Either insert content into existing page,
+    # or create page otherwise
+    path = 'sources/' + module_name.replace('.', '/')[6:] + '.md'
+    if os.path.exists(path):
+        template = open(path).read()
+        assert '{{autogenerated}}' in template, ('Template found for ' + path +
+                                                 ' but missing {{autogenerated}} tag.')
+        module_page = template.replace('{{autogenerated}}', module_page)
+        print('...inserting autogenerated content into template:', path)
+    else:
+        print('...creating new page with autogenerated content:', path)
+    subdir = os.path.dirname(path)
+    if not os.path.exists(subdir):
+        os.makedirs(subdir)
+    open(path, 'w').write(module_page)

docs/mkdocs.yml

@@ -15,6 +15,7 @@ pages:
 - Index: documentation.md
 - Examples: examples.md
 - FAQ: faq.md
+- Backends: backend.md
 - Optimizers: optimizers.md
 - Objectives: objectives.md
 - Models: models.md

docs/sources/layers/advanced_activations.md

@@ -1,106 +0,0 @@
-
-## LeakyReLU
-
-```python
-keras.layers.advanced_activations.LeakyReLU(alpha=0.3)
-```
-
-Special version of a Rectified Linear Unit that allows a small gradient when the unit is not active (`f(x) = alpha*x for x < 0`).
-
-
-- __Input shape__: Arbitrary. Use the keyword argument `input_shape` (tuple of integers, does not include the samples axis) when using this layer as the first layer in a model.
-
-- __Output shape__: Same as input.
-
-- __Arguments__:
-    - __alpha__: float >= 0. Negative slope coefficient.
-
----
-
-## PReLU
-
-```python
-keras.layers.advanced_activations.PReLU()
-```
-
-Parametrized linear unit. Similar to a LeakyReLU, where each input unit has its alpha coefficient, and where these coefficients are learned during training.
-
-
-- __Input shape__: Arbitrary. Use the keyword argument `input_shape` (tuple of integers, does not include the samples axis) when using this layer as the first layer in a model.
-
-- __Output shape__: Same as input.
-
-- __References__:
-    - [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](http://arxiv.org/pdf/1502.01852v1.pdf)
-
----
-
-## ELU
-
-```python
-keras.layers.advanced_activations.ELU()
-```
-
-Exponential linear unit. Negative values pushes mean unit activations closer to zero, with the advantage of having a noise-robust deactivation state.
-
-
-- __Input shape__: Arbitrary. Use the keyword argument `input_shape` (tuple of integers, does not include the samples axis) when using this layer as the first layer in a model.
-
-- __Output shape__: Same as input.
-
-- __References__:
-    - [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)](http://arxiv.org/pdf/1511.07289v1.pdf)
-
----
-
-## ParametricSoftplus
-
-```python
-keras.layers.advanced_activations.ParametricSoftplus()
-```
-
-Parametric Softplus of the form: (`f(x) = alpha * (1 + exp(beta * x))`). This is essentially a smooth version of ReLU where the parameters control the sharpness of the rectification. The parameters are initialized to more closely approximate a ReLU than the standard `softplus`: `alpha` initialized to `0.2` and `beta`  initialized to `5.0`. The parameters are fit separately for each hidden unit.
-
-- __Input shape__: Arbitrary. Use the keyword argument `input_shape=...` when using this layer as the first layer in a model.
-
-- __Output shape__: Same as input.
-
-- __References__:
-    - [Inferring Nonlinear Neuronal Computation Based on Physiologically Plausible Inputs](http://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1003143)
-
-## Thresholded Linear
-
-```python
-keras.layers.advanced_activations.ThresholdedLinear(theta)
-```
-
-Parametrized linear unit. provides a threshold near zero where values are zeroed.
-
-
-- __Input shape__: Arbitrary. Use the keyword argument `input_shape` (tuple of integers, does not include the samples axis) when using this layer as the first layer in a model.
-
-- __Output shape__: Same as input.
-
-- __Arguments__:
-    - __theta__: float >= 0. Threshold location of activation
-
-- __References__:
-    - [Zero-Bias Autoencoders and the Benefits of Co-Adapting Features](http://arxiv.org/pdf/1402.3337.pdf)
-
-## Thresholded ReLu
-
-```python
-keras.layers.advanced_activations.ThresholdedReLu(theta)
-```
-
-Parametrized rectified linear unit. provides a threshold near zero where values are zeroed.
-
-- __Input shape__: Arbitrary. Use the keyword argument `input_shape=...` when using this layer as the first layer in a model.
-
-- __Output shape__: Same as input.
-
-- __Arguments__:
-    - __theta__: float >= 0. Threshold location of activation
-
-- __References__:
-    - [Zero-Bias Autoencoders and the Benefits of Co-Adapting Features](http://arxiv.org/pdf/1402.3337.pdf)

docs/sources/layers/containers.md

@@ -1,21 +0,0 @@
-Containers are ensembles of layers that can be interacted with through the same API as `Layer` objects.
-
-## Sequential
-
-```python
-keras.layers.containers.Sequential(layers=[])
-```
-
-The Sequential container is a linear stack of layers. Apart from the `add` methods and the `layers` constructor argument, the API is identical to that of the `Layer` class.
-
-This class is also the basis for the `keras.models.Sequential` architecture.
-
-The `layers` constructor argument is a list of Layer instances.
-
-__Methods__:
-
-```python
-add(layer)
-```
-
-Add a new layer to the stack.

docs/sources/layers/convolutional.md

@@ -1,202 +0,0 @@
-
-## Convolution1D
-
-```python
-keras.layers.convolutional.Convolution1D(nb_filter, filter_length, 
-        init='uniform',
-        activation='linear',
-        weights=None, 
-        border_mode='valid',
-        subsample_length=1, 
-        W_regularizer=None, b_regularizer=None,
-        W_constraint=None, b_constraint=None,
-        input_dim=None, input_length=None)
-```
-
-Convolution operator for filtering neighborhoods of one-dimensional inputs. When using this layer as the first layer in a model, either provide the keyword argument `input_dim` (int, e.g. 128 for sequences of 128-dimensional vectors), or `input_shape` (tuple of integers, e.g. (10, 128) for sequences of 10 vectors of 128-dimensional vectors).
-
-- __Input shape__: 3D tensor with shape: `(samples, steps, input_dim)`.
-
-- __Output shape__: 3D tensor with shape: `(samples, new_steps, nb_filter)`. `steps` value might have changed due to padding.
-
-- __Arguments__:
-    - __nb_filter__: Number of convolution kernels to use (dimensionality of the output).
-    - __filter_length__: The extension (spatial or temporal) of each filter.
-    - __init__: name of initialization function for the weights of the layer (see: [initializations](../initializations.md)), or alternatively, Theano function to use for weights initialization. This parameter is only relevant if you don't pass a `weights` argument.
-    - __activation__: name of activation function to use (see: [activations](../activations.md)), or alternatively, elementwise Theano function. If you don't specify anything, no activation is applied (ie. "linear" activation: a(x) = x).
-    - __weights__: list of numpy arrays to set as initial weights.
-    - __border_mode__: 'valid' or 'same'.
-    - __subsample_length__: factor by which to subsample output.
-    - __W_regularizer__: instance of [WeightRegularizer](../regularizers.md) (eg. L1 or L2 regularization), applied to the main weights matrix.
-    - __b_regularizer__: instance of [WeightRegularizer](../regularizers.md), applied to the bias.
-    - __activity_regularizer__: instance of [ActivityRegularizer](../regularizers.md), applied to the network output.
-    - __W_constraint__: instance of the [constraints](../constraints.md) module (eg. maxnorm, nonneg), applied to the main weights matrix.
-    - __b_constraint__: instance of the [constraints](../constraints.md) module, applied to the bias.
-    - __input_dim__: Number of channels/dimensions in the input. Either this argument or the keyword argument `input_shape` must be provided when using this layer as the first layer in a model.
-    - __input_length__: Length of input sequences, when it is constant. This argument is required if you are going to connect `Flatten` then `Dense` layers upstream (without it, the shape of the dense outputs cannot be computed).
-
----
-
-## Convolution2D
-
-```python
-keras.layers.convolutional.Convolution2D(nb_filter, nb_row, nb_col, 
-        init='glorot_uniform',
-        activation='linear',
-        weights=None, 
-        border_mode='valid',
-        subsample=(1, 1),
-        W_regularizer=None, b_regularizer=None,
-        W_constraint=None,
-        dim_ordering='th')
-```
-
-Convolution operator for filtering windows of two-dimensional inputs. When using this layer as the first layer in a model, provide the keyword argument `input_shape` (tuple of integers, does not include the sample axis), e.g. `input_shape=(3, 128, 128)` for 128x128 RGB pictures.
-
-- __Input shape__: 4D tensor with shape: `(samples, channels, rows, cols)` if dim_ordering='th'
-or 4D tensor with shape: `(samples, rows, cols, channels)` if dim_ordering='tf'.
-
-- __Output shape__: 4D tensor with shape: `(samples, nb_filter, nb_row, nb_col)` if dim_ordering='th'
-or 4D tensor with shape: `(samples, nb_row, nb_col, nb_filter)` if dim_ordering='tf'.
-
-
-- __Arguments__:
-
-    - __nb_filter__: Number of convolution filters to use.
-    - __nb_row__: Number of rows in the convolution kernel.
-    - __nb_col__: Number of columns in the convolution kernel.
-    - __init__: name of initialization function for the weights of the layer (see: [initializations](../initializations.md)), or alternatively, Theano function to use for weights initialization. This parameter is only relevant if you don't pass a `weights` argument.
-    - __activation__: name of activation function to use (see: [activations](../activations.md)), or alternatively, elementwise Theano function. If you don't specify anything, no activation is applied (ie. "linear" activation: a(x) = x).
-    - __weights__: list of numpy arrays to set as initial weights.
-    - __border_mode__: 'valid' or 'same'.
-    - __subsample__: tuple of length 2. Factor by which to subsample output. Also called strides elsewhere.
-    - __W_regularizer__: instance of [WeightRegularizer](../regularizers.md) (eg. L1 or L2 regularization), applied to the main weights matrix.
-    - __b_regularizer__: instance of [WeightRegularizer](../regularizers.md), applied to the bias.
-    - __activity_regularizer__: instance of [ActivityRegularizer](../regularizers.md), applied to the network output.
-    - __W_constraint__: instance of the [constraints](../constraints.md) module (eg. maxnorm, nonneg), applied to the main weights matrix.
-    - __b_constraint__: instance of the [constraints](../constraints.md) module, applied to the bias.
-    - __dim_ordering__: 'th' or 'tf'. In 'th' mode, the channels dimension (the depth) is at index 1, in 'tf' mode is it at index 3.
-
-
----
-
-## MaxPooling1D
-
-```python
-keras.layers.convolutional.MaxPooling1D(pool_length=2, stride=None, border_mode='valid')
-```
-
-Max pooling operation for temporal data.
-
-- __Input shape__: 3D tensor with shape: `(samples, steps, features)`.
-
-- __Output shape__: 3D tensor with shape: `(samples, downsampled_steps, features)`.
-
-- __Arguments__:
-
-    - __pool_length__: factor by which to downscale. 2 will halve the input.
-    - __stride__: integer or None. Stride value.
-    - __border_mode__: 'valid' or 'same'. **Note:** 'same' will only work with TensorFlow for the time being.
-
----
-
-## MaxPooling2D
-
-```python
-keras.layers.convolutional.MaxPooling2D(pool_size=(2, 2), border_mode='valid', dim_ordering='th')
-```
-
-Max pooling operation for spatial data.
-
-- __Input shape__: 4D tensor with shape: `(samples, channels, rows, cols)` if dim_ordering='th'
-or 4D tensor with shape: `(samples, rows, cols, channels)` if dim_ordering='tf'.
-
-- __Output shape__: 4D tensor with shape: `(nb_samples, channels, pooled_rows, pooled_cols)` if dim_ordering='th'
-or 4D tensor with shape: `(samples, pooled_rows, pooled_cols, channels)` if dim_ordering='tf'.
-
-- __Arguments__:
-
-    - __pool_size__: tuple of 2 integers, factors by which to downscale (vertical, horizontal). (2, 2) will halve the image in each dimension.
-    - __strides__: tuple of 2 integers, or None. Strides values.
-    - __border_mode__: 'valid' or 'same'. **Note:** 'same' will only work with TensorFlow for the time being.
-    - __dim_ordering__: 'th' or 'tf'. In 'th' mode, the channels dimension (the depth) is at index 1, in 'tf' mode is it at index 3.
-
-
----
-
-## UpSampling1D
-
-```python
-keras.layers.convolutional.UpSampling1D(length=2)
-```
-
-Repeats each temporal step `length` times along the time axis.
-
-- __Input shape__: 3D tensor with shape: `(samples, steps, features)`.
-
-- __Output shape__: 3D tensor with shape: `(samples, upsampled_steps, features)`.
-
-- __Arguments__:
-    - __length__: integer. Upsampling factor.
-
----
-
-
-## UpSampling2D
-
-```python
-keras.layers.convolutional.UpSampling2D(size=(2, 2), dim_ordering='th')
-```
-
-Repeats the rows and columns of the data by size[0] and size[1] respectively.
-
-- __Input shape__: 4D tensor with shape: `(samples, channels, rows, cols)` if dim_ordering='th'
-or 4D tensor with shape: `(samples, rows, cols, channels)` if dim_ordering='tf'.
-
-- __Output shape__: 4D tensor with shape: `(samples, channels, upsampled_rows, upsampled_cols)` if dim_ordering='th'
-or 4D tensor with shape: `(samples, upsampled_rows, upsampled_cols, channels)` if dim_ordering='tf'.
-
-- __Arguments__:
-    - __size__: tuple of 2 integers. The upsampling factors for rows and columns.
-    - __dim_ordering__: 'th' or 'tf'. In 'th' mode, the channels dimension (the depth) is at index 1, in 'tf' mode is it at index 3.
-
----
-
-
-## ZeroPadding1D
-
-```python
-keras.layers.convolutional.ZeroPaddding1D(padding=1)
-```
-
-Pads the input with zeros left and right along the time axis.
-
-- __Input shape__: 3D tensor with shape: `(nb_samples, steps, dim)`.
-
-- __Output shape__: 3D tensor with shape: `(nb_samples, padded_steps, dim)`.
-
-- __Arguments__:
-    - __padding__: integer, the size of the padding.
-
----
-
-
-## ZeroPadding2D
-
-```python
-keras.layers.convolutional.ZeroPaddding2D(padding=(1, 1), dim_ordering='th')
-```
-
-Pads the rows and columns of the input with zeros, left and right.
-
-- __Input shape__: 4D tensor with shape: `(samples, channels, rows, cols)` if dim_ordering='th'
-or 4D tensor with shape: `(samples, rows, cols, channels)` if dim_ordering='tf'.
-
-- __Output shape__: 4D tensor with shape: `(samples, channels, padded_rows, padded_cols)` if dim_ordering='th'
-or 4D tensor with shape: `(samples, padded_rows, padded_cols, channels)` if dim_ordering='tf'.
-
-- __Arguments__:
-    - __padding__: tuple of 2 integers, the size of the padding for rows and columns respectively.
-    - __dim_ordering__: 'th' or 'tf'. In 'th' mode, the channels dimension (the depth) is at index 1, in 'tf' mode is it at index 3.
-
----

docs/sources/layers/core.md

@@ -1,477 +0,0 @@
-## Base class
-
-```python
-keras.layers.core.Layer()
-```
-
-__Methods__:
-
-```python
-set_previous(previous_layer)
-```
-
-Connect the input of the current layer to the output of the argument layer.
-
-- __Return__: None.
-
-- __Arguments__: 
-    - __previous_layer__: Layer object.
-
-
-
-```python
-get_output(train)
-```
-
-Get the output of the layer.
-
-- __Return__: Theano tensor.
-
-- __Arguments__: 
-    - __train__: Boolean. Specifies whether output is computed in training mode or in testing mode, which can change the logic, for instance in there are any `Dropout` layers in the network. 
-
-
-
-```python
-get_input(train)
-```
-
-Get the input of the layer.
-
-- __Return__: Theano tensor.
-
-- __Arguments__: 
-    - __train__: Boolean. Specifies whether output is computed in training mode or in testing mode, which can change the logic, for instance in there are any `Dropout` layers in the network. 
-
-
-
-```python
-get_weights()
-```
-
-Get the weights of the parameters of the layer.
-
-- __Return__: List of numpy arrays (one per layer parameter). 
-
-
-
-```python
-set_weights(weights)
-```
-
-Set the weights of the parameters of the layer.
-
-- __Arguments__: 
-    - __weights__: List of numpy arrays (one per layer parameter). Should be in the same order as what `get_weights(self)` returns.
-
-
-```python
-get_config()
-```
-
-- __Return__: Configuration dictionary describing the layer.
-
-
----
-
-## Dense
-```python
-keras.layers.core.Dense(output_dim,
-                        init='glorot_uniform',
-                        activation='linear',
-                        weights=None,
-                        W_regularizer=None, b_regularizer=None, activity_regularizer=None,
-                        W_constraint=None, b_constraint=None,
-                        input_dim=None)
-```
-
-Standard 1D fully-connect layer. 
-
-- __Input shape__: 2D tensor with shape: `(nb_samples, input_dim)`.
-
-- __Output shape__: 2D tensor with shape: `(nb_samples, output_dim)`.
-
-- __Arguments__:
-
-    - __output_dim__: int >= 0. 
-    - __init__: name of initialization function for the weights of the layer (see: [initializations](../initializations.md)), or alternatively, Theano function to use for weights initialization. This parameter is only relevant if you don't pass a `weights` argument.
-    - __activation__: name of activation function to use (see: [activations](../activations.md)), or alternatively, elementwise Theano function. If you don't specify anything, no activation is applied (ie. "linear" activation: a(x) = x).
-    - __weights__: list of numpy arrays to set as initial weights. The list should have 1 element, of shape `(input_dim, output_dim)`.
-    - __W_regularizer__: instance of [WeightRegularizer](../regularizers.md) (eg. L1 or L2 regularization), applied to the main weights matrix.
-    - __b_regularizer__: instance of [WeightRegularizer](../regularizers.md), applied to the bias.
-    - __activity_regularizer__: instance of [ActivityRegularizer](../regularizers.md), applied to the network output.
-    - __W_constraint__: instance of the [constraints](../constraints.md) module (eg. maxnorm, nonneg), applied to the main weights matrix.
-    - __b_constraint__: instance of the [constraints](../constraints.md) module, applied to the bias.
-    - __input_dim__: dimensionality of the input (integer). This argument (or alternatively, the keyword argument `input_shape`) is required when using this layer as the first layer in a model. 
-
----
-
-## TimeDistributedDense
-```python
-keras.layers.core.TimeDistributedDense(output_dim,
-                                       init='glorot_uniform',
-                                       activation='linear',
-                                       weights=None
-                                       W_regularizer=None, b_regularizer=None, activity_regularizer=None,
-                                       W_constraint=None, b_constraint=None,
-                                       input_dim=None, input_length=None)
-```
-
-Fully-connected layer distributed over the time dimension. Useful after a recurrent network set to `return_sequences=True`.
-
-- __Input shape__: 3D tensor with shape: `(nb_samples, timesteps, input_dim)`.
-
-- __Arguments__:
-    - __output_dim__: int >= 0. 
-    - __init__: name of initialization function for the weights of the layer (see: [initializations](../initializations.md)), or alternatively, Theano function to use for weights initialization. This parameter is only relevant if you don't pass a `weights` argument.
-    - __activation__: name of activation function to use (see: [activations](../activations.md)), or alternatively, elementwise Theano function. If you don't specify anything, no activation is applied (ie. "linear" activation: a(x) = x).
-    - __weights__: list of numpy arrays to set as initial weights. The list should have 1 element, of shape `(input_dim, output_dim)`.
-    - __W_regularizer__: instance of [WeightRegularizer](../regularizers.md) (eg. L1 or L2 regularization), applied to the main weights matrix.
-    - __b_regularizer__: instance of [WeightRegularizer](../regularizers.md), applied to the bias.
-    - __activity_regularizer__: instance of [ActivityRegularizer](../regularizers.md), applied to the network output.
-    - __W_constraint__: instance of the [constraints](../constraints.md) module (eg. maxnorm, nonneg), applied to the main weights matrix.
-    - __b_constraint__: instance of the [constraints](../constraints.md) module, applied to the bias.
-    - __input_dim__: dimensionality of the input (integer). This argument (or alternatively, the keyword argument `input_shape`) is required when using this layer as the first layer in a model.
-    - __input_length__: Length of input sequences, when it is constant. This argument is required if you are going to connect `Flatten` then `Dense` layers upstream (without it, the shape of the dense outputs cannot be computed).
-
-- __Example__:
-```python
-# input shape: (nb_samples, timesteps, 10)
-model.add(LSTM(5, return_sequences=True, input_dim=10)) # output shape: (nb_samples, timesteps, 5)
-model.add(TimeDistributedDense(15)) # output shape: (nb_samples, timesteps, 15)
-```
-
-
----
-
-## AutoEncoder
-```python
-keras.layers.core.AutoEncoder(encoder, decoder, output_reconstruction=True, weights=None):
-```
-
-A customizable autoencoder model. If `output_reconstruction = True` then dim(input) = dim(output) else dim(output) = dim(hidden)
-
-
-- __Input shape__: The layer shape is defined by the encoder definitions
-
-- __Output shape__: The layer shape is defined by the decoder definitions
-
-- __Arguments__:
-
-    - __encoder__: A [layer](./) or [layer container](./containers.md).
-
-    - __decoder__: A [layer](./) or [layer container](./containers.md).
-    
-    - __output_reconstruction__: If this is False, then when .predict() is called, the output is the deepest hidden layer's activation. Otherwise, the output of the final decoder layer is presented. Be sure your validation data conforms to this logic if you decide to use any.
-    
-    - __weights__: list of numpy arrays to set as initial weights. The list should have 1 element, of shape `(input_dim, output_dim)`.
-
-- __Example__:
-```python
-from keras.layers import containers
-
-# input shape: (nb_samples, 32)
-encoder = containers.Sequential([Dense(16, input_dim=32), Dense(8)])
-decoder = containers.Sequential([Dense(16, input_dim=8), Dense(32)])
-
-autoencoder = Sequential()
-autoencoder.add(AutoEncoder(encoder=encoder, decoder=decoder, output_reconstruction=False))
-```
-
-
----
-
-## Activation
-```python
-keras.layers.core.Activation(activation)
-```
-Apply an activation function to the input. 
-
-
-- __Input shape__: Arbitrary. Use the keyword argument `input_shape` (tuple of integers, does not include the samples axis) when using this layer as the first layer in a model.
-
-- __Output shape__: Same as input.
-
-- __Arguments__:
-
-    - __activation__: name of activation function to use (see: [activations](../activations.md)), or alternatively, elementwise Theano function.
-
-
----
-
-## Dropout
-```python
-keras.layers.core.Dropout(p)
-```
-Apply dropout to the input. Dropout consists in randomly setting a fraction `p` of input units to 0 at each update during training time, which helps prevent overfitting. Reference: [Dropout: A Simple Way to Prevent Neural Networks from Overfitting](http://www.cs.toronto.edu/~rsalakhu/papers/srivastava14a.pdf)
-
-
-- __Input shape__: Arbitrary. Use the keyword argument `input_shape` (tuple of integers, does not include the samples axis) when using this layer as the first layer in a model.
-
-- __Output shape__: Same as input.
-
-- __Arguments__:
-
-    - __p__: float (0 <= p < 1). Fraction of the input that gets dropped out at training time. 
-
----
-
-
-## Reshape
-```python
-keras.layers.core.Reshape(dims)
-```
-
-Reshape the input to a new shape containing the same number of units. 
-
-
-- __Input shape__: Arbitrary. Use the keyword argument `input_shape` (tuple of integers, does not include the samples axis) when using this layer as the first layer in a model.
-
-- __Output shape__: `(nb_samples, dims)`.
-
-- __Arguments__:
-
-    - dims: tuple of integers. Dimensions of the new shape.
-
-- __Example__:
-```python
-# input shape: (nb_samples, 10)
-model.add(Dense(100, input_dim=10)) # output shape: (nb_samples, 100)
-model.add(Reshape(dims=(10, 10)))  # output shape: (nb_samples, 10, 10)
-```
-
----
-
-## Flatten
-```python
-keras.layers.core.Flatten()
-```
-
-Convert a nD input to 1D. 
-
-- __Input shape__: Arbitrary. Use the keyword argument `input_shape` (tuple of integers, does not include the samples axis) when using this layer as the first layer in a model.
-
-- __Output shape__: `(nb_samples, nb_input_units)`.
-
----
-
-## RepeatVector
-```python
-keras.layers.core.RepeatVector(n)
-```
-
-Repeat the 1D input n times. Dimensions of input are assumed to be `(nb_samples, dim)`. Output will have the shape `(nb_samples, n, dim)`.
-
-Note that the output is still a single tensor; `RepeatVector` does not split the data flow.
-
-- __Input shape__: Arbitrary. Use the keyword argument `input_shape` (tuple of integers, does not include the samples axis) when using this layer as the first layer in a model.
-
-- __Output shape__: `(nb_samples, n, input_dims)`.
-
-- __Arguments__:
-    - __n__: int. 
-
----
-
-## Permute
-```python
-keras.layers.core.Permute(dims)
-```
-Permute the dimensions of the input data according to the given tuple. Sometimes useful for connecting RNNs and convnets together.
-
-- __Input shape__: Arbitrary. Use the keyword argument `input_shape` (tuple of integers, does not include the samples axis) when using this layer as the first layer in a model.
-
-- __Output shape__: Same as the input shape, but with the dimensions re-ordered according to the ordering specified by the tuple.
-
-- __Argument__: tuple specifying the permutation scheme (e.g. `(2, 1)` permutes the first and second dimension of the input).
-
-- __Example__:
-```python
-# input shape: (nb_samples, 10)
-model.add(Dense(50, input_dim=10)) # output shape: (nb_samples, 50)
-model.add(Reshape(dims=(10, 5))) # output shape: (nb_samples, 10, 5)
-model.add(Permute(dims=(2, 1))) #output shape: (nb_samples, 5, 10)
-```
-
----
-
-## ActivityRegularization
-```python
-keras.layers.core.ActivityRegularization(l1=0., l2=0.)
-```
-
-Leaves the input unchanged, but adds a term to the loss function based on the input activity. L1 and L2 regularization supported.
-
-This layer can be used, for instance, to induce activation sparsity in the previous layer.
-
----
-
-## MaxoutDense
-```python
-keras.layers.core.MaxoutDense(output_dim, nb_feature=4,
-        init='glorot_uniform',
-        weights=None,
-        W_regularizer=None, b_regularizer=None, activity_regularizer=None,
-        W_constraint=None, b_constraint=None,
-        input_dim=None)
-```
-
-A dense maxout layer. A `MaxoutDense` layer takes the element-wise maximum of `nb_feature` `Dense(input_dim, output_dim)` linear layers. This allows the layer to learn a convex, piecewise linear activation function over the inputs. See [this paper](http://arxiv.org/pdf/1302.4389.pdf) for more details. Note that this is a *linear* layer -- if you wish to apply activation function (you shouldn't need to -- they are universal function approximators), an `Activation` layer must be added after.
-
-- __Input shape__: 2D tensor with shape: `(nb_samples, input_dim)`.
-
-- __Output shape__: 2D tensor with shape: `(nb_samples, output_dim)`.
-
-- __Arguments__:
-
-    - __output_dim__: int >= 0. 
-    - __nb_feature__: int >= 0. the number of features to create for the maxout. This is equivalent to the number of piecewise elements to be allowed for the activation function. 
-    - __init__: name of initialization function for the weights of the layer (see: [initializations](../initializations.md)), or alternatively, Theano function to use for weights initialization. This parameter is only relevant if you don't pass a `weights` argument.
-    - __weights__: list of numpy arrays to set as initial weights. The list should have 1 element, of shape `(input_dim, output_dim)`.
-    - __W_regularizer__: instance of [WeightRegularizer](../regularizers.md) (eg. L1 or L2 regularization), applied to the main weights matrix.
-    - __b_regularizer__: instance of [WeightRegularizer](../regularizers.md), applied to the bias.
-    - __activity_regularizer__: instance of [ActivityRegularizer](../regularizers.md), applied to the network output.
-    - __W_constraint__: instance of the [constraints](../constraints.md) module (eg. maxnorm, nonneg), applied to the main weights matrix.
-    - __b_constraint__: instance of the [constraints](../constraints.md) module, applied to the bias.
-    - __input_dim__: dimensionality of the input (integer). This argument (or alternatively, the keyword argument `input_shape`) is required when using this layer as the first layer in a model.
-
-```python
-# input shape: (nb_samples, 10)
-model.add(Dense(100, input_dim=10)) # output shape: (nb_samples, 100)
-model.add(MaxoutDense(50, nb_feature=10)) # output shape: (nb_samples, 50)
-```
-
-## Merge
-```python
-keras.layers.core.Merge(layers, mode='sum', concat_axis=-1, dot_axes=-1)
-```
-
-Merge the output of a list of layers (or containers) into a single tensor.
-
-- __Arguments__:
-    - __layers__: List of layers or [containers](/layers/containers/).
-    - __mode__: String, one of `{'sum', 'mul', 'concat', 'ave', 'dot'}`. `sum`, `mul` and `ave` will simply sum/multiply/average the outputs of the layers (therefore all layers should have an output with the same shape). `concat` will concatenate the outputs along the dimension specified by `concate_axis` (therefore all layers should have an output that only differ along this dimension). `dot` will dot tensor contraction on the axes specified by `dot_axes` (see [the Numpy documentation](http://docs.scipy.org/doc/numpy-1.10.1/reference/generated/numpy.tensordot.html) for more details).
-    - __concat_axis__: axis to use in `concat` mode.
-    - __dot_axes__: axis or axes to use in `dot` mode (see [the Numpy documentation](http://docs.scipy.org/doc/numpy-1.10.1/reference/generated/numpy.tensordot.html) for more details).
-
-
-- __Notes__:
-    - `dot` mode only works with Theano for the time being.
-
-- __Example__:
-
-```python
-left = Sequential()
-left.add(Dense(50, input_shape=(784,)))
-left.add(Activation('relu'))
-
-right = Sequential()
-right.add(Dense(50, input_shape=(784,)))
-right.add(Activation('relu'))
-
-model = Sequential()
-model.add(Merge([left, right], mode='sum'))
-
-model.add(Dense(10))
-model.add(Activation('softmax'))
-
-model.compile(loss='categorical_crossentropy', optimizer='rmsprop')
-
-model.fit([X_train, X_train], Y_train, batch_size=128, nb_epoch=20, validation_data=([X_test, X_test], Y_test))
-```
-
-## Masking
-```python
-keras.layers.core.Masking(mask_value=0.)
-```
-
-Create a mask for the input data by using `mask_value` as the sentinel value which should be masked out.
-Given an input of dimensions `(nb_samples, timesteps, input_dim)`, return the input untouched as output, and supply a mask of shape `(nb_samples, timesteps)` where all timesteps which had *all* their values equal to `mask_value` are masked out.
-
-- __Input shape__: 3D tensor with shape: `(nb_samples, timesteps, features)`.
-
-- __Output shape__: 3D tensor with shape: `(nb_samples, timesteps, features)`.
-
-- __Notes__: Masking only works in Theano for the time being.
-
-## Lambda
-```python
-keras.layers.core.Lambda(function, output_shape=None)
-```
-
-Used for evaluating an arbitrary Theano expression on the output of the previous layer.
-
-- __Input shape__: Arbitrary. Use the keyword argument input_shape (tuple of integers, does not include the samples axis) when using this layer as the first layer in a model.
-
-- __Output shape__: Specified by the `output_shape` argument.
-
-- __Arguments__:
-
-    - __function__: The expression to be evaluated. Takes one argument: the output of the previous layer.
-    - __output_shape__: Shape of the tensor returned by `function`. Should be a shape tuple (not including the samples dimension) or a function of the full input shape tuple (including samples dimension).
-
-- __Example__:
-
-```python
-# custom softmax function
-def sharp_softmax(X, beta=1.5):
-    return theano.tensor.nnet.softmax(X * beta)
-
-def output_shape(input_shape):
-    # here input_shape includes the samples dimension
-    return input_shape  # shape is unchanged
-
-model = Sequential()
-model.add(Dense(input_dim=10, output_dim=10))
-model.add(Lambda(sharp_softmax, output_shape))
-model.add(Dense(1))
-model.add(Activation('sigmoid'))
-```
-
-
-## LambdaMerge
-```python
-keras.layers.core.LambdaMerge(layers, function, output_shape=None)
-```
-
-Merge the output of a list of layers (or containers) into a single tensor, using an arbitrary Theano expression.
-
-- __Arguments__:
-    - __layers__: List of layers or [containers](/layers/containers/).
-    - __function__: The expression to be evaluated. Takes one argument: the list of input tensors.
-    - __output_shape__: Shape of the tensor returned by `function`. Should be a shape tuple (not including samples dimension) or a function of the list of input shape tuples (including samples dimension).
-
-- __Example__:
-
-```python
-# root mean square function
-def rms(inputs):
-    # inputs is a list of tensors
-    s = inputs[0] ** 2
-    for i in range(1, len(inputs)):
-        s += inputs[i] ** 2
-    s /= len(inputs)
-    s = theano.tensor.sqrt(s)
-    # return a single tensor
-    return s
-
-def output_shape(input_shapes):
-    # return the shape of the first tensor
-    return input_shapes[0]
-
-left = Sequential()
-left.add(Dense(input_dim=10, output_dim=10))
-left.add(Activation('sigmoid'))
-
-right = Sequential()
-right.add(Dense(input_dim=10, output_dim=10))
-right.add(Activation('sigmoid'))
-
-model = Sequential()
-model.add(LambdaMerge([left, right], rms, output_shape))
-
-model.add(Dense(1))
-model.add(Activation('sigmoid'))
-```
-
----

docs/sources/layers/embeddings.md

@@ -1,31 +0,0 @@
-
-## Embedding
-
-```python
-keras.layers.embeddings.Embedding(input_dim, output_dim,
-                                  init='uniform',
-                                  weights=None,
-                                  W_regularizer=None, W_constraint=None,
-                                  mask_zero=False,
-                                  input_length=None)
-```
-
-Turn positive integers (indexes) into denses vectors of fixed size,
-eg. `[[4], [20]] -> [[0.25, 0.1], [0.6, -0.2]]`
-
-- __Input shape__: 2D tensor with shape: `(nb_samples, sequence_length)`.
-
-- __Output shape__: 3D tensor with shape: `(nb_samples, sequence_length, output_dim)`.
-
-- __Arguments__:
-
-    - __input_dim__: int >= 0. Size of the vocabulary, ie. 1+maximum integer index occurring in the input data.
-    - __output_dim__: int >= 0. Dimension of the dense embedding.
-    - __init__: name of initialization function for the weights of the layer (see: [initializations](../initializations.md)), or alternatively, Theano function to use for weights initialization. This parameter is only relevant if you don't pass a `weights` argument.
-    - __weights__: list of numpy arrays to set as initial weights. The list should have 1 element, of shape `(input_dim, output_dim)`.
-    - __W_regularizer__: instance of the [regularizers](../regularizers.md) module (eg. L1 or L2 regularization), applied to the embedding matrix.
-    - __W_constraint__: instance of the [constraints](../constraints.md) module (eg. maxnorm, nonneg), applied to the embedding matrix.
-	- __mask_zero__: Whether or not the input value 0 is a special "padding" value that should be masked out. This is useful for [recurrent layers](recurrent.md) which may take variable length input. If this is `True` then all subsequent layers in the model need to support masking or an exception will be raised.
-    - __input_length__: Length of input sequences, when it is constant. This argument is required if you are going to connect `Flatten` then `Dense` layers upstream (without it, the shape of the dense outputs cannot be computed).
-
----

docs/sources/layers/noise.md

@@ -1,37 +0,0 @@
-
-
-## GaussianNoise
-```python
-keras.layers.noise.GaussianNoise(sigma)
-```
-Apply to the input an additive zero-centred gaussian noise with standard deviation `sigma`. This is useful to mitigate overfitting (you could see it as a kind of random data augmentation). Gaussian Noise (GS) is a natural choice as corruption process for real valued inputs.
-
-Only active at training time.
-
-- __Input shape__: Arbitrary. Use the keyword argument `input_shape` (tuple of integers, does not include the samples axis) when using this layer as the first layer in a model.
-
-- __Output shape__: Same as input.
-
-- __Arguments__:
-
-    - __sigma__: float, standard deviation of the noise distribution.
-
----
-
-## GaussianDropout
-```python
-keras.layers.noise.GaussianDropout(p)
-```
-Apply to the input an multiplicative one-centred gaussian noise with standard deviation `sqrt(p/(1-p))`. p refers to drop probability to match Dropout layer syntax. 
-
-Only active at training time.
-
-- __Input shape__: Arbitrary. Use the keyword argument `input_shape` (tuple of integers, does not include the samples axis) when using this layer as the first layer in a model.
-
-- __Output shape__: Same as input.
-
-- __Arguments__:
-
-    - __p__: float, drop probability as with Dropout.
-
-

docs/sources/layers/normalization.md

@@ -1,19 +0,0 @@
-
-## BatchNormalization
-
-```python
-keras.layers.normalization.BatchNormalization(epsilon=1e-6, weights=None)
-```
-
-Normalize the activations of the previous layer at each batch.
-
-- __Input shape__: Arbitrary. Use the keyword argument `input_shape` (tuple of integers, does not include the samples axis) when using this layer as the first layer in a model.
-
-- __Output shape__: Same as input.
-
-- __Arguments__: 
-    - __epsilon__: small float > 0. Fuzz parameter.
-    - __weights__: Initialization weights. List of 2 numpy arrays, with shapes: `[(input_shape,), (input_shape,)]`
-
-- __References__:
-    - [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](http://arxiv.org/pdf/1502.03167v3.pdf)

docs/sources/layers/recurrent.md

@@ -1,130 +0,0 @@
-
-## SimpleRNN
-
-```python
-keras.layers.recurrent.SimpleRNN(output_dim,
-        init='glorot_uniform', inner_init='orthogonal',
-        activation='sigmoid',
-        weights=None,
-        return_sequences=False,
-        go_backwards=False,
-        stateful=False,
-        input_dim=None, input_length=None)
-```
-Fully connected RNN where output is to fed back to input. 
-
-- __Input shape__: 3D tensor with shape: `(nb_samples, timesteps, input_dim)`.
-
-- __Output shape__: 
-    - if `return_sequences`: 3D tensor with shape: `(nb_samples, timesteps, output_dim)`.
-    - else: 2D tensor with shape: `(nb_samples, output_dim)`.
-
-- __Masking__: This layer supports masking for input data with a variable number of timesteps To introduce masks to your data, use an [Embedding](embeddings.md) layer with the `mask_zero` parameter set to `True`. **Note:** for the time being, masking in only supported with Theano.
-
-- __Notes__: When using the TensorFlow backend, the number of timesteps used must be fixed. Make sure to pass an `input_length` int argument or a complete `input_shape` tuple argument.
-
-
-- __Arguments__:
-    - __output_dim__: dimension of the internal projections and the final output.
-    - __init__: weight initialization function. Can be the name of an existing function (str), or a Theano function (see: [initializations](../initializations.md)).
-    - __activation__: activation function. Can be the name of an existing function (str), or a Theano function (see: [activations](../activations.md)).
-    - __weights__: list of numpy arrays to set as initial weights. The list should have 3 elements, of shapes: `[(input_dim, output_dim), (output_dim, output_dim), (output_dim,)]`.
-    - __return_sequences__: Boolean. Whether to return the last output in the output sequence, or the full sequence.
-    - __go_backwards__: Boolean (default False). If True, rocess the input sequence backwards.
-    - __stateful__: Boolean (default False). If True, the last state for each sample at index i in a batch will be used as initial state for the sample of index i in the following batch.
-    - __input_dim__: dimensionality of the input (integer). This argument (or alternatively, the keyword argument `input_shape`) is required when using this layer as the first layer in a model.
-    - __input_length__: Length of input sequences, when it is constant. This argument is required if you are going to connect `Flatten` then `Dense` layers upstream (without it, the shape of the dense outputs cannot be computed).
-
-
----
-
-## GRU
-
-```python
-keras.layers.recurrent.GRU(output_dim,
-        init='glorot_uniform', inner_init='orthogonal',
-        activation='sigmoid', inner_activation='hard_sigmoid',
-        return_sequences=False,
-        go_backwards=False,
-        stateful=False,
-        input_dim=None, input_length=None)
-```
-
-Gated Recurrent Unit - Cho et al. 2014.
-
-- __Input shape__: 3D tensor with shape: `(nb_samples, timesteps, input_dim)`.
-
-- __Output shape__:
-    - if `return_sequences`: 3D tensor with shape: `(nb_samples, timesteps, output_dim)`.
-    - else: 2D tensor with shape: `(nb_samples, output_dim)`.
-
-- __Masking__: This layer supports masking for input data with a variable number of timesteps To introduce masks to your data, use an [Embedding](embeddings.md) layer with the `mask_zero` parameter set to true. **Note:** for the time being, masking in only supported with Theano.
-
-- __Notes__: When using the TensorFlow backend, the number of timesteps used must be fixed. Make sure to pass an `input_length` int argument or a complete `input_shape` tuple argument.
-
-- __Arguments__:
-    - __output_dim__: dimension of the internal projections and the final output.
-    - __init__: weight initialization function for the output cell. Can be the name of an existing function (str), or a Theano function (see: [initializations](../initializations.md)).
-    - __inner_init__: weight initialization function for the inner cells.
-    - __activation__: activation function for the output. Can be the name of an existing function (str), or a Theano function (see: [activations](../activations.md)).
-    - __inner_activation__: activation function for the inner cells.
-    - __weights__: list of numpy arrays to set as initial weights. The list should have 9 elements.
-    - __return_sequences__: Boolean. Whether to return the last output in the output sequence, or the full sequence.
-    - __go_backwards__: Boolean (default False). If True, rocess the input sequence backwards.
-    - __stateful__: Boolean (default False). If True, the last state for each sample at index i in a batch will be used as initial state for the sample of index i in the following batch.
-    - __input_dim__: dimensionality of the input (integer). This argument (or alternatively, the keyword argument `input_shape`) is required when using this layer as the first layer in a model.
-    - __input_length__: Length of input sequences, when it is constant. This argument is required if you are going to connect `Flatten` then `Dense` layers upstream (without it, the shape of the dense outputs cannot be computed).
-
-
-- __References__: 
-    - [On the Properties of Neural Machine Translation: Encoder–Decoder Approaches](http://www.aclweb.org/anthology/W14-4012)
-    - [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](http://arxiv.org/pdf/1412.3555v1.pdf)
-
----
-
-## LSTM
-
-```python
-keras.layers.recurrent.LSTM(output_dim,
-        init='glorot_uniform', inner_init='orthogonal', forget_bias_init='one',
-        activation='tanh', inner_activation='hard_sigmoid',
-        weights=None,
-        return_sequences=False,
-        go_backwards=False,
-        stateful=False,
-        input_dim=None, input_length=None)
-```
-
-Long-Short Term Memory unit - Hochreiter 1997.
-
-- __Input shape__: 3D tensor with shape: `(nb_samples, timesteps, input_dim)`.
-
-- __Output shape__:
-    - if `return_sequences`: 3D tensor with shape: `(nb_samples, timesteps, output_dim)`.
-    - else: 2D tensor with shape: `(nb_samples, output_dim)`.
-
-- __Masking__: This layer supports masking for input data with a variable number of timesteps To introduce masks to your data, use an [Embedding](embeddings.md) layer with the `mask_zero` parameter set to true. **Note:** for the time being, masking in only supported with Theano.
-
-- __Notes__: When using the TensorFlow backend, the number of timesteps used must be fixed. Make sure to pass an `input_length` int argument or a complete `input_shape` tuple argument.
-
-- __Arguments__:
-    - __output_dim__: dimension of the internal projections and the final output.
-    - __init__: weight initialization function for the output cell. Can be the name of an existing function (str), or a Theano function (see: [initializations](../initializations.md)).
-    - __inner_init__: weight initialization function for the inner cells.
-    - __forget_bias_init__: initialization function for the bias of the forget gate. [Jozefowicz et al.](http://www.jmlr.org/proceedings/papers/v37/jozefowicz15.pdf) recommend initializing with ones.
-    - __activation__: activation function for the output. Can be the name of an existing function (str), or a Theano function (see: [activations](../activations.md)).
-    - __inner_activation__: activation function for the inner cells.
-    - __weights__: list of numpy arrays to set as initial weights. The list should have 12 elements.
-    - __return_sequences__: Boolean. Whether to return the last output in the output sequence, or the full sequence.
-    - __go_backwards__: Boolean (default False). If True, rocess the input sequence backwards.
-    - __stateful__: Boolean (default False). If True, the last state for each sample at index i in a batch will be used as initial state for the sample of index i in the following batch.
-    - __input_dim__: dimensionality of the input (integer). This argument (or alternatively, the keyword argument `input_shape`) is required when using this layer as the first layer in a model.
-    - __input_length__: Length of input sequences, when it is constant. This argument is required if you are going to connect `Flatten` then `Dense` layers upstream (without it, the shape of the dense outputs cannot be computed).
-
-
-- __References__: 
-    - [Long short-term memory](http://deeplearning.cs.cmu.edu/pdfs/Hochreiter97_lstm.pdf) (original 1997 paper)
-    - [Learning to forget: Continual prediction with LSTM](http://www.mitpressjournals.org/doi/pdf/10.1162/089976600300015015)
-    - [Supervised sequence labelling with recurrent neural networks](http://www.cs.toronto.edu/~graves/preprint.pdf)
-
----

docs/sources/models.md

@@ -1,211 +0,0 @@
-## Sequential
-
-Linear stack of layers.
-
-```python
-model = keras.models.Sequential()
-```
-- __Methods__:
-    - __add__(layer): Add a layer to the model.
-    - __compile__(optimizer, loss, class_mode="categorical"):
-        - __Arguments__:
-            - __optimizer__: str (name of optimizer) or optimizer object. See [optimizers](optimizers.md).
-            - __loss__: str (name of objective function) or objective function. See [objectives](objectives.md).
-            - __class_mode__: one of "categorical", "binary". This is only used for computing classification accuracy or using the predict_classes method.
-    - __fit__(X, y, batch_size=128, nb_epoch=100, verbose=1, validation_split=0., validation_data=None, shuffle=True, show_accuracy=False, callbacks=[], class_weight=None, sample_weight=None): Train a model for a fixed number of epochs.
-        - __Return__: a history object. It `history` attribute is a record of training loss values at successive epochs, as well as validation loss values (if applicable).
-        - __Arguments__:
-            - __X__: data.
-            - __y__: labels.
-            - __batch_size__: int. Number of samples per gradient update.
-            - __nb_epoch__: int.
-            - __verbose__: 0 for no logging to stdout, 1 for progress bar logging, 2 for one log line per epoch.
-            - __callbacks__: `keras.callbacks.Callback` list. List of callbacks to apply during training. See [callbacks](callbacks.md).
-            - __validation_split__: float (0. < x < 1). Fraction of the data to use as held-out validation data.
-            - __validation_data__: tuple (X, y) to be used as held-out validation data. Will override validation_split.
-            - __shuffle__: boolean or str (for 'batch'). Whether to shuffle the samples at each epoch. 'batch' is a special option for dealing with the limitations of HDF5 data; it shuffles in batch-sized chunks.
-            - __show_accuracy__: boolean. Whether to display class accuracy in the logs to stdout at each epoch.
-            - __class_weight__: dictionary mapping classes to a weight value, used for scaling the loss function (during training only).
-            - __sample_weight__: list or numpy array with 1:1 mapping to the training samples, used for scaling the loss function (during training only). For time-distributed data, there is one weight per sample *per timestep*, i.e. if your output data is shaped `(nb_samples, timesteps, output_dim)`, your mask should be of shape `(nb_samples, timesteps, 1)`. This allows you to mask out or reweight individual output timesteps, which is useful in sequence to sequence learning.
-    - __evaluate__(X, y, batch_size=128, show_accuracy=False, verbose=1, sample_weight=None): Show performance of the model over some validation data.
-        - __Return__: The loss score over the data, or a `(loss, accuracy)` tuple if `show_accuracy=True`.
-        - __Arguments__: Same meaning as fit method above. verbose is used as a binary flag (progress bar or nothing).
-    - __predict__(X, batch_size=128, verbose=1):
-        - __Return__: An array of predictions for some test data.
-        - __Arguments__: Same meaning as fit method above.
-    - __predict_classes__(X, batch_size=128, verbose=1): Return an array of class predictions for some test data.
-        - __Return__: An array of labels for some test data.
-        - __Arguments__: Same meaning as fit method above. verbose is used as a binary flag (progress bar or nothing).
-    - __train_on_batch__(X, y, accuracy=False, class_weight=None, sample_weight=None): Single gradient update on one batch.
-        - __Return__: loss over the data, or tuple `(loss, accuracy)` if `accuracy=True`.
-    - __test_on_batch__(X, y, accuracy=False, sample_weight=None): Single performance evaluation on one batch.
-        - __Return__: loss over the data, or tuple `(loss, accuracy)` if `accuracy=True`.
-    - __save_weights__(fname, overwrite=False): Store the weights of all layers to a HDF5 file. If overwrite==False and the file already exists, an exception will be thrown.
-    - __load_weights__(fname): Sets the weights of a model, based to weights stored by __save_weights__. You can only __load_weights__ on a savefile from a model with an identical architecture. __load_weights__ can be called either before or after the __compile__ step.
-
-__Examples__:
-
-```python
-from keras.models import Sequential
-from keras.layers.core import Dense, Dropout, Activation
-from keras.optimizers import SGD
-
-model = Sequential()
-model.add(Dense(2, init='uniform', input_dim=64))
-model.add(Activation('softmax'))
-
-model.compile(loss='mse', optimizer='sgd')
-
-'''
-Demonstration of verbose modes 1 and 2
-'''
-model.fit(X_train, y_train, nb_epoch=3, batch_size=16, verbose=1)
-# outputs
-'''
-Train on 37800 samples, validate on 4200 samples
-Epoch 0
-37800/37800 [==============================] - 7s - loss: 0.0385
-Epoch 1
-37800/37800 [==============================] - 8s - loss: 0.0140
-Epoch 2
-10960/37800 [=======>......................] - ETA: 4s - loss: 0.0109
-'''
-
-model.fit(X_train, y_train, nb_epoch=3, batch_size=16, verbose=2)
-# outputs
-'''
-Train on 37800 samples, validate on 4200 samples
-Epoch 0
-loss: 0.0190
-Epoch 1
-loss: 0.0146
-Epoch 2
-loss: 0.0049
-'''
-
-'''
-Demonstration of show_accuracy
-'''
-model.fit(X_train, y_train, nb_epoch=3, batch_size=16, verbose=2, show_accuracy=True)
-# outputs
-'''
-Train on 37800 samples, validate on 4200 samples
-Epoch 0
-loss: 0.0190 - acc.: 0.8750
-Epoch 1
-loss: 0.0146 - acc.: 0.8750
-Epoch 2
-loss: 0.0049 - acc.: 1.0000
-'''
-
-'''
-Demonstration of validation_split
-'''
-model.fit(X_train, y_train, nb_epoch=3, batch_size=16, validation_split=0.1, show_accuracy=True, verbose=1)
-# outputs
-'''
-Train on 37800 samples, validate on 4200 samples
-Epoch 0
-37800/37800 [==============================] - 7s - loss: 0.0385 - acc.: 0.7258 - val. loss: 0.0160 - val. acc.: 0.9136
-Epoch 1
-37800/37800 [==============================] - 8s - loss: 0.0140 - acc.: 0.9265 - val. loss: 0.0109 - val. acc.: 0.9383
-Epoch 2
-10960/37800 [=======>......................] - ETA: 4s - loss: 0.0109 - acc.: 0.9420
-'''
-```
-
----
-
-## Graph
-
-Arbitrary connection graph. It can have any number of inputs and outputs, with each output trained with its own loss function. The quantity being optimized by a Graph model is the sum of all loss functions over the different outputs.
-
-```python
-model = keras.models.Graph()
-```
-- __Methods__:
-    - __add_input__(name, input_shape, dtype='float'): Add an input with shape dimensionality `ndim`. 
-        - __Arguments__:
-            - __input_shape__: Integer tuple, shape of the expected input (not including the samples axis). E.g. (10,) for 10-dimensional vectors, (None, 128) for sequences (of variable length) of 128-dimensional vectors, (3, 32, 32) for 32x32 images with RGB channels.
-            - __dtype__: `float` or `int`. Type of the expected input data.
-    - __add_output__(name, input=None, inputs=[], merge_mode='concat'): Add an output connect to `input` or `inputs`.
-        - __Arguments__:
-            - __name__: str. unique identifier of the output.
-            - __input__: str name of the node that the output is connected to. Only specify *one* of either `input` or `inputs`.
-            - __inputs__: list of str names of the node that the output is connected to.
-            - __merge_mode__: "sum" or "concat". Only applicable if `inputs` list is specified. Merge mode for the different inputs.
-    - __add_node__(layer, name, input=None, inputs=[], merge_mode='concat'): Add an output connect to `input` or `inputs`.
-        - __Arguments__:
-            - __layer__: Layer instance.
-            - __name__: str. unique identifier of the node.
-            - __input__: str name of the node/input that the node is connected to. Only specify *one* of either `input` or `inputs`.
-            - __inputs__: list of str names of the node that the node is connected to.
-            - __merge_mode__: "sum" or "concat". Only applicable if `inputs` list is specified. Merge mode for the different inputs.
-    - __add_shared_node__(layer, name, inputs=[], merge_mode=None, outputs=[]): Add a shared node connected to `inputs`. A shared node is a layer that will be applied separately to every incoming input, and that uses only one set of weights. The merging operation occurs on the outputs of the layer. 
-        - __Arguments__:
-            - __layer__: Layer instance.
-            - __name__: str. unique identifier of the node.
-            - __inputs__: list of str names of the node that the node is connected to.
-            - __merge_mode__: Merge mode for the different inputs.
-            - __outputs__: Optional. List of names for outputs, when merge_mode = None.
-    - __compile__(optimizer, loss):
-        - __Arguments__:
-            - __optimizer__: str (name of optimizer) or optimizer object. See [optimizers](optimizers.md).
-            - __loss__: dictionary mapping the name(s) of the output(s) to a loss function (string name of objective function or objective function. See [objectives](objectives.md)).
-    - __fit__(data, batch_size=128, nb_epoch=100, verbose=1, validation_split=0., validation_data=None, shuffle=True, callbacks=[]): Train a model for a fixed number of epochs.
-        - __Return__: a history object. It `history` attribute is a record of training loss values at successive epochs, as well as validation loss values (if applicable).
-        - __Arguments__:
-            - __data__:dictionary mapping input names out outputs names to appropriate numpy arrays. All arrays should contain the same number of samples.
-            - __batch_size__: int. Number of samples per gradient update.
-            - __nb_epoch__: int.
-            - __verbose__: 0 for no logging to stdout, 1 for progress bar logging, 2 for one log line per epoch.
-            - __callbacks__: `keras.callbacks.Callback` list. List of callbacks to apply during training. See [callbacks](callbacks.md).
-            - __validation_split__: float (0. < x < 1). Fraction of the data to use as held-out validation data.
-            - __validation_data__: tuple (X, y) to be used as held-out validation data. Will override validation_split.
-            - __shuffle__: boolean. Whether to shuffle the samples at each epoch.
-    - __evaluate__(data, batch_size=128, verbose=1): Show performance of the model over some validation data.
-        - __Return__: The loss score over the data.
-        - __Arguments__: Same meaning as fit method above. verbose is used as a binary flag (progress bar or nothing).
-    - __predict__(data, batch_size=128, verbose=1):
-        - __Return__: A dictionary mapping output names to arrays of predictions over the data.
-        - __Arguments__: Same meaning as fit method above. Only inputs need to be specified in `data`.
-    - __train_on_batch__(data): Single gradient update on one batch.
-        - __Return__: loss over the data.
-    - __test_on_batch__(data): Single performance evaluation on one batch.
-        - __Return__: loss over the data.
-    - __save_weights__(fname, overwrite=False): Store the weights of all layers to a HDF5 file. If `overwrite==False` and the file already exists, an exception will be thrown.
-    - __load_weights__(fname): Sets the weights of a model, based to weights stored by __save_weights__. You can only __load_weights__ on a savefile from a model with an identical architecture. __load_weights__ can be called either before or after the __compile__ step.
-
-
-__Examples__:
-
-```python
-# graph model with one input and two outputs
-graph = Graph()
-graph.add_input(name='input', input_shape=(32,))
-graph.add_node(Dense(16), name='dense1', input='input')
-graph.add_node(Dense(4), name='dense2', input='input')
-graph.add_node(Dense(4), name='dense3', input='dense1')
-graph.add_output(name='output1', input='dense2')
-graph.add_output(name='output2', input='dense3')
-
-graph.compile('rmsprop', {'output1':'mse', 'output2':'mse'})
-history = graph.fit({'input':X_train, 'output1':y_train, 'output2':y2_train}, nb_epoch=10)
-
-```
-
-```python
-# graph model with two inputs and one output
-graph = Graph()
-graph.add_input(name='input1', input_shape=(32,))
-graph.add_input(name='input2', input_shape=(32,))
-graph.add_node(Dense(16), name='dense1', input='input1')
-graph.add_node(Dense(4), name='dense2', input='input2')
-graph.add_node(Dense(4), name='dense3', input='dense1')
-graph.add_output(name='output', inputs=['dense2', 'dense3'], merge_mode='sum')
-graph.compile('rmsprop', {'output':'mse'})
-
-history = graph.fit({'input1':X_train, 'input2':X2_train, 'output':y_train}, nb_epoch=10)
-predictions = graph.predict({'input1':X_test, 'input2':X2_test}) # {'output':...}
-
-```

docs/sources/optimizers.md

@@ -1,117 +0,0 @@
-
-## Usage of optimizers
-
-An optimizer is one of the two arguments required for compiling a Keras model:
-
-```python
-model = Sequential()
-model.add(Dense(64, init='uniform', input_dim=10))
-model.add(Activation('tanh'))
-model.add(Activation('softmax'))
-
-sgd = SGD(lr=0.1, decay=1e-6, momentum=0.9, nesterov=True)
-model.compile(loss='mean_squared_error', optimizer=sgd)
-```
-
-You can either instantiate an optimizer before passing it to `model.compile()` , as in the above example, or you can call it by its name. In the latter case, the default parameters for the optimizer will be used.
-
-```python
-# pass optimizer by name: default parameters will be used
-model.compile(loss='mean_squared_error', optimizer='sgd')
-```
-
----
-
-## Base class
-
-```python
-keras.optimizers.Optimizer(**kwargs)
-```
-
-All optimizers descended from this class support the following keyword argument:
-
-- __clipnorm__: float >= 0.
-
-Note: this is base class for building optimizers, not an actual optimizer that can be used for training models.
-
----
-
-##  SGD
-
-```python
-keras.optimizers.SGD(lr=0.01, momentum=0., decay=0., nesterov=False)
-``` 
-
-__Arguments__:
-
-- __lr__: float >= 0. Learning rate.
-- __momentum__: float >= 0. Parameter updates momentum.
-- __decay__: float >= 0. Learning rate decay over each update.
-- __nesterov__: boolean. Whether to apply Nesterov momentum.
-
----
-
-##  Adagrad
-
-```python
-keras.optimizers.Adagrad(lr=0.01, epsilon=1e-6)
-```
-
-It is recommended to leave the parameters of this optimizer at their default values.
-
-__Arguments__:
-
-- __lr__: float >= 0. Learning rate. 
-- __epsilon__: float >= 0. 
-
----
-
-##  Adadelta
-
-```python
-keras.optimizers.Adadelta(lr=1.0, rho=0.95, epsilon=1e-6)
-```
-
-It is recommended to leave the parameters of this optimizer at their default values.
-
-__Arguments__:
-
-- __lr__: float >= 0. Learning rate. It is recommended to leave it at the default value.
-- __rho__: float >= 0. 
-- __epsilon__: float >= 0. Fuzz factor.
-
-For more info, see *"Adadelta: an adaptive learning rate method"* by Matthew Zeiler.
-
----
-
-##  RMSprop 
-
-```python
-keras.optimizers.RMSprop(lr=0.001, rho=0.9, epsilon=1e-6)
-```
-
-It is recommended to leave the parameters of this optimizer at their default values.
-
-__Arguments__:
-
-- __lr__: float >= 0. Learning rate. 
-- __rho__: float >= 0.
-- __epsilon__: float >= 0. Fuzz factor.
-
----
-
-## Adam
-
-```python
-keras.optimizers.Adam(lr=0.001, beta_1=0.9, beta_2=0.999, epsilon=1e-8)
-```
-
-Adam optimizer, proposed by Kingma and Lei Ba in [Adam: A Method For Stochastic Optimization](http://arxiv.org/pdf/1412.6980v8.pdf). Default parameters are those suggested in the paper.
-
-__Arguments__:
-
-- __lr__: float >= 0. Learning rate.
-- __beta_1__, __beta_2__: floats, 0 < beta < 1. Generally close to 1.
-- __epsilon__: float >= 0. Fuzz factor.
-
----

docs/templates/backend.md

@@ -23,6 +23,15 @@ It probably looks like this:
 
 Simply change the field `backend` to either `"theano"` or `"tensorflow"`, and Keras will use the new configuration next time you run any Keras code.
 
+You can also define the environment variable ``KERAS_BACKEND`` and this will
+override what is defined in your config file :
+
+```bash
+KERAS_BACKEND=tensorflow python -c "from keras import backend; print backend._BACKEND"
+Using TensorFlow backend.
+tensorflow
+```
+
 ## Using the abstract Keras backend to write new code
 
 If you want the Keras modules you write to be compatible with both Theano and TensorFlow, you have to write them via the abstract Keras backend API. Here's an intro.
@@ -32,7 +41,7 @@ You can import the backend module via:
 from keras import backend as K
 ```
 
-This instantiates an input placeholder. It's equivalent to `tf.placeholder()` or `T.matrix()`, `T.tensor3()`, etc.
+The code below instantiates an input placeholder. It's equivalent to `tf.placeholder()` or `T.matrix()`, `T.tensor3()`, etc.
 
 ```python
 input = K.placeholder(shape=(2, 4, 5))
@@ -42,16 +51,16 @@ input = K.placeholder(shape=(None, 4, 5))
 input = K.placeholder(ndim=3)
 ```
 
-This instantiates a shared variable. It's equivalent to `tf.variable()` or `theano.shared()`.
+The code below instantiates a shared variable. It's equivalent to `tf.variable()` or `theano.shared()`.
 
 ```python
 val = np.random.random((3, 4, 5))
 var = K.variable(value=val)
 
 # all-zeros variable:
-var = K.ones(shape=(3, 4, 5))
-# all-ones:
 var = K.zeros(shape=(3, 4, 5))
+# all-ones:
+var = K.ones(shape=(3, 4, 5))
 ```
 
 Most tensor operations you will need can be done as you would in TensorFlow or Theano:

docs/templates/callbacks.md

@@ -4,51 +4,12 @@ A callback is a set of functions to be applied at given stages of the training p
 
 ---
 
-## Base class
-
-```python
-keras.callbacks.Callback()
-```
-- __Properties__:
-    - __params__: dict. Training parameters (eg. verbosity, batch size, number of epochs...).
-    - __model__: `keras.models.Model`. Reference of the model being trained.
-- __Methods__:
-    - __on_train_begin__(logs={}): Method called at the beginning of training.
-    - __on_train_end__(logs={}): Method called at the end of training.
-    - __on_epoch_begin__(epoch, logs={}): Method called at the beginning of epoch `epoch`.
-    - __on_epoch_end__(epoch, logs={}): Method called at the end of epoch `epoch`.
-    - __on_batch_begin__(batch, logs={}): Method called at the beginning of batch `batch`.
-    - __on_batch_end__(batch, logs={}): Method called at the end of batch `batch`.
-
-The `logs` dictionary will contain keys for quantities relevant to the current batch or epoch. Currently, the `.fit()` method of the `Sequential` model class will include the following quantities in the `logs` that it passes to its callbacks:
-- __on_epoch_end__: logs optionally include `val_loss` (if validation is enabled in `fit`), and `val_accuracy` (if validation and accuracy monitoring are enabled).
-- __on_batch_begin__: logs include `size`, the number of samples in the current batch.
-- __on_batch_end__: logs include `loss`, and optionally `accuracy` (if accuracy monitoring is enabled).
-
----
-
-## Available callbacks
-
-```python
-keras.callbacks.ModelCheckpoint(filepath, verbose=0, save_best_only=False)
-```
-
-Save the model after every epoch. If `save_best_only=True`, the latest best model according to the validation loss will not be overwritten.
-`filepath` can contain named formatting options, which will be filled the value of `epoch` and keys in `logs` (passed in `on_epoch_end`).
-
-For example: if `filepath` is `weights.{epoch:02d}-{val_loss:.2f}.hdf5`, then multiple files will be save with the epoch number and the validation loss.
-
-
-```python
-keras.callbacks.EarlyStopping(monitor='val_loss', patience=0, verbose=0)
-```
-
-Stop training after no improvement of the metric `monitor` is seen for `patience` epochs.
+{{autogenerated}}
 
 ---
 
 
-## Create a callback
+# Create a callback
 
 You can create a custom callback by extending the base class `keras.callbacks.Callback`. A callback has access to its associated model through the class property `self.model`.
 

docs/templates/examples.md

@@ -167,6 +167,7 @@ model.fit([images, partial_captions], next_words, batch_size=16, nb_epoch=100)
 ```
 
 In the examples folder, you will find example models for real datasets:
+
 - CIFAR10 small images classification: Convolutional Neural Network (CNN) with realtime data augmentation
 - IMDB movie review sentiment classification: LSTM over sequences of words
 - Reuters newswires topic classification: Multilayer Perceptron (MLP)

docs/templates/faq.md

@@ -20,6 +20,8 @@
 
 [How can I record the training / validation loss / accuracy at each epoch?](#how-can-i-record-the-training-validation-loss-accuracy-at-each-epoch)
 
+[How can I use stateful RNNs?](#how-can-i-use-stateful-rnns)
+
 ---
 
 ### How can I run Keras on GPU?
@@ -105,22 +107,22 @@ You can build a Theano function that will return the output of a certain layer g
 
 ```python
 # with a Sequential model
-get_3rd_layer_output = theano.function([model.layers[0].input], 
+get_3rd_layer_output = theano.function([model.layers[0].input],
                                        model.layers[3].get_output(train=False))
 layer_output = get_3rd_layer_output(X)
 
 # with a Graph model
 get_conv_layer_output = theano.function([model.inputs[i].input for i in model.input_order],
-                                        model.outputs['conv'].get_output(train=False),
+                                        model.nodes['conv'].get_output(train=False),
                                         on_unused_input='ignore')
-conv_output = get_conv_output(input_data_dict)
+conv_output = get_conv_layer_output([input_data_dict[i] for i in model.input_order])
 ```
 
 ---
 
 ### Isn't there a bug with Merge or Graph related to input concatenation?
 
-Yes, there was a known bug with tensor concatenation in Thenao that was fixed early 2015. 
+Yes, there was a known bug with tensor concatenation in Theano that was fixed early 2015.
 Please upgrade to the latest version of Theano:
 
 ```bash
@@ -153,7 +155,7 @@ Find out more in the [callbacks documentation](callbacks.md).
 
 ### How is the validation split computed?
 
-If you set the `validation_split` arugment in `model.fit` to e.g. 0.1, then the validation data used will be the *last 10%* of the data. If you set it to 0.25, it will be the last 25% of the data, etc.
+If you set the `validation_split` argument in `model.fit` to e.g. 0.1, then the validation data used will be the *last 10%* of the data. If you set it to 0.25, it will be the last 25% of the data, etc.
 
 
 ---
@@ -176,4 +178,52 @@ hist = model.fit(X, y, validation_split=0.2)
 print(hist.history)
 ```
 
----
+---
+
+### How can I use stateful RNNs?
+
+Making a RNN stateful means that the states for the samples of each batch will be reused as initial states for the samples in the next batch.
+
+When using stateful RNNs, it is therefore assumed that:
+
+- all batches have the same number of samples
+- If `X1` and `X2` are successive batches of samples, then `X2[i]` is the follow-up sequence to `X1[i]`, for every `i`.
+
+To use statefulness in RNNs, you need to:
+
+- explicitly specify the batch size you are using, by passing a `batch_input_shape` argument to the first layer in your model. It should be a tuple of integers, e.g. `(32, 10, 16)` for a 32-samples batch of sequences of 10 timesteps with 16 features per timestep.
+- set `stateful=True` in your RNN layer(s).
+
+To reset the states accumulated:
+
+- use `model.reset_states()` to reset the states of all layers in the model
+- use `layer.reset_states()` to reset the states of a specific stateful RNN layer
+
+Example:
+
+```python
+
+X  # this is our input data, of shape (32, 21, 16)
+# we will feed it to our model in sequences of length 10
+
+model = Sequential()
+model.add(LSTM(32, batch_input_shape=(32, 10, 16), stateful=True))
+model.add(Dense(16, activation='softmax'))
+
+model.compile(optimizer='rmsprop', loss='categorical_crossentropy')
+
+# we train the network to predict the 11th timestep given the first 10:
+model.train_on_batch(X[:, :10, :], np.reshape(X[:, 10, :], (32, 16)))
+
+# the state of the network has changed. We can feed the follow-up sequences:
+model.train_on_batch(X[:, 10:20, :], np.reshape(X[:, 20, :], (32, 16)))
+
+# let's reset the states of the LSTM layer:
+model.reset_states()
+
+# another way to do it in this case:
+model.layers[0].reset_states()
+```
+
+Notes that the methods `predict`, `fit`, `train_on_batch`, `predict_classes`, etc. will *all* update the states of the stateful layers in a model. This allows you to do not only stateful training, but also stateful prediction.
+

docs/templates/index.md

@@ -2,9 +2,10 @@
 
 ## You have just found Keras.
 
-Keras is a minimalist, highly modular neural networks library, written in Python and capable of running either on top of either [TensorFlow](https://github.com/tensorflow/tensorflow) or [Theano](https://github.com/Theano/Theano). It was developed with a focus on enabling fast experimentation. Being able to go from idea to result with the least possible delay is key to doing good research.
+Keras is a minimalist, highly modular neural networks library, written in Python and capable of running on top of either [TensorFlow](https://github.com/tensorflow/tensorflow) or [Theano](https://github.com/Theano/Theano). It was developed with a focus on enabling fast experimentation. Being able to go from idea to result with the least possible delay is key to doing good research.
 
 Use Keras if you need a deep learning library that:
+
 - allows for easy and fast prototyping (through total modularity, minimalism, and extensibility).
 - supports both convolutional networks and recurrent networks, as well as combinations of the two.
 - supports arbitrary connectivity schemes (including multi-input and multi-output training).
@@ -12,9 +13,7 @@ Use Keras if you need a deep learning library that:
 
 Read the documentation at [Keras.io](http://keras.io).
 
-Keras is compatible with:
-    - __Python 2.7-3.5__ with the Theano backend
-    - __Python 2.7__ with the TensorFlow backend
+Keras is compatible with: __Python 2.7-3.5__.
 
 
 ------------------
@@ -36,7 +35,7 @@ Keras is compatible with:
 
 ## Getting started: 30 seconds to Keras
 
-The core datastructure of Keras is a __model__, a way to organize layers. There are two types of models: [`Sequential`](/models/#sequential) and [`Graph`](/models/#graph).
+The core datastructure of Keras is a __model__, a way to organize layers. There are two types of models: [`Sequential`](http://keras.io/models/#sequential) and [`Graph`](http://keras.io/models/#graph).
 
 Here's the `Sequential` model (a linear pile of layers):
 
@@ -108,7 +107,8 @@ Keras uses the following dependencies:
 - HDF5 and h5py (optional, required if you use model saving/loading functions)
 - Optional but recommended if you use CNNs: cuDNN.
 
-When using the Theano backend:
+*When using the Theano backend:*
+
 - Theano
     - [See installation instructions](http://deeplearning.net/software/theano/install.html#install).
 
@@ -117,11 +117,12 @@ When using the Theano backend:
 sudo pip install git+git://github.com/Theano/Theano.git
 ```
 
-When using the TensorFlow backend:
+*When using the TensorFlow backend:*
+
 - TensorFlow
     - [See installation instructions](https://github.com/tensorflow/tensorflow#download-and-setup).
 
-To install, `cd` to the Keras folder and run the install command:
+To install Keras, `cd` to the Keras folder and run the install command:
 ```
 sudo python setup.py install
 ```
@@ -145,20 +146,7 @@ By default, Keras will use Theano as its tensor manipulation library. [Follow th
 
 You can ask questions and join the development discussion on the [Keras Google group](https://groups.google.com/forum/#!forum/keras-users).
 
-------------------
-
-
-## Contribution Guidelines
-
-Keras welcomes all contributions from the community. 
-
-- Keep a pragmatic mindset and avoid bloat. Only add to the source if that is the only path forward.
-- New features should be documented. Make sure you update the documentation along with your Pull Request.
-- Any new function or class should have a proper docstring.
-- The documentation for every new feature should include a usage example in the form of a code snippet. 
-- All changes should be tested. Make sure any new feature you add has a corresponding unit test.
-- Please no Pull Requests about coding style.
-- Even if you don't contribute to the Keras source code, if you have an application of Keras that is concise and powerful, please consider adding it to our collection of [examples](https://github.com/fchollet/keras/tree/master/examples).
+You can also post bug reports and feature requests in [Github issues](https://github.com/fchollet/keras/issues). Make sure to read [our guidelines](https://github.com/fchollet/keras/blob/master/CONTRIBUTING.md) first.
 
 
 ------------------
@@ -172,4 +160,4 @@ Keras was initially developed as part of the research effort of project ONEIROS
 
 >_"Oneiroi are beyond our unravelling --who can be sure what tale they tell? Not all that men look for comes to pass. Two gates there are that give passage to fleeting Oneiroi; one is made of horn, one of ivory. The Oneiroi that pass through sawn ivory are deceitful, bearing a message that will not be fulfilled; those that come out through polished horn have truth behind them, to be accomplished for men who see them."_ Homer, Odyssey 19. 562 ff (Shewring translation).
 
-------------------
+------------------

docs/templates/models.md

@@ -0,0 +1,114 @@
+Keras has two models: __Sequential__, a linear stack of layers, and __Graph__, a directed acyclic graph of layers.
+
+# Using the Sequential model
+
+```python
+from keras.models import Sequential
+from keras.layers.core import Dense, Dropout, Activation
+from keras.optimizers import SGD
+
+model = Sequential()
+model.add(Dense(2, init='uniform', input_dim=64))
+model.add(Activation('softmax'))
+
+model.compile(optimizer='sgd', loss='mse')
+
+'''
+Train the model for 3 epochs, in batches of 16 samples,
+on data stored in the Numpy array X_train,
+and labels stored in the Numpy array y_train:
+'''
+model.fit(X_train, y_train, nb_epoch=3, batch_size=16, verbose=1)
+'''
+What you will see with mode verbose=1:
+Train on 37800 samples, validate on 4200 samples
+Epoch 0
+37800/37800 [==============================] - 7s - loss: 0.0385
+Epoch 1
+37800/37800 [==============================] - 8s - loss: 0.0140
+Epoch 2
+10960/37800 [=======>......................] - ETA: 4s - loss: 0.0109
+'''
+
+model.fit(X_train, y_train, nb_epoch=3, batch_size=16, verbose=2)
+'''
+What you will see with mode verbose=2:
+Train on 37800 samples, validate on 4200 samples
+Epoch 0
+loss: 0.0190
+Epoch 1
+loss: 0.0146
+Epoch 2
+loss: 0.0049
+'''
+
+'''
+Demonstration of the show_accuracy argument
+'''
+model.fit(X_train, y_train, nb_epoch=3, batch_size=16, verbose=2, show_accuracy=True)
+'''
+Train on 37800 samples, validate on 4200 samples
+Epoch 0
+loss: 0.0190 - acc.: 0.8750
+Epoch 1
+loss: 0.0146 - acc.: 0.8750
+Epoch 2
+loss: 0.0049 - acc.: 1.0000
+'''
+
+'''
+Demonstration of the validation_split argument
+'''
+model.fit(X_train, y_train, nb_epoch=3, batch_size=16,
+          validation_split=0.1, show_accuracy=True, verbose=1)
+'''
+Train on 37800 samples, validate on 4200 samples
+Epoch 0
+37800/37800 [==============================] - 7s - loss: 0.0385 - acc.: 0.7258 - val. loss: 0.0160 - val. acc.: 0.9136
+Epoch 1
+37800/37800 [==============================] - 8s - loss: 0.0140 - acc.: 0.9265 - val. loss: 0.0109 - val. acc.: 0.9383
+Epoch 2
+10960/37800 [=======>......................] - ETA: 4s - loss: 0.0109 - acc.: 0.9420
+'''
+```
+
+# Using the Graph model
+
+```python
+# graph model with one input and two outputs
+graph = Graph()
+graph.add_input(name='input', input_shape=(32,))
+graph.add_node(Dense(16), name='dense1', input='input')
+graph.add_node(Dense(4), name='dense2', input='input')
+graph.add_node(Dense(4), name='dense3', input='dense1')
+graph.add_output(name='output1', input='dense2')
+graph.add_output(name='output2', input='dense3')
+
+graph.compile(optimizer='rmsprop', loss={'output1':'mse', 'output2':'mse'})
+history = graph.fit({'input':X_train, 'output1':y_train, 'output2':y2_train}, nb_epoch=10)
+
+```
+
+```python
+# graph model with two inputs and one output
+graph = Graph()
+graph.add_input(name='input1', input_shape=(32,))
+graph.add_input(name='input2', input_shape=(32,))
+graph.add_node(Dense(16), name='dense1', input='input1')
+graph.add_node(Dense(4), name='dense2', input='input2')
+graph.add_node(Dense(4), name='dense3', input='dense1')
+graph.add_output(name='output', inputs=['dense2', 'dense3'], merge_mode='sum')
+graph.compile(optimizer='rmsprop', loss={'output':'mse'})
+
+history = graph.fit({'input1':X_train, 'input2':X2_train, 'output':y_train}, nb_epoch=10)
+predictions = graph.predict({'input1':X_test, 'input2':X2_test}) # {'output':...}
+
+```
+
+----
+
+# Model API documentation
+
+
+
+{{autogenerated}}

docs/templates/optimizers.md

@@ -0,0 +1,25 @@
+
+## Usage of optimizers
+
+An optimizer is one of the two arguments required for compiling a Keras model:
+
+```python
+model = Sequential()
+model.add(Dense(64, init='uniform', input_dim=10))
+model.add(Activation('tanh'))
+model.add(Activation('softmax'))
+
+sgd = SGD(lr=0.1, decay=1e-6, momentum=0.9, nesterov=True)
+model.compile(loss='mean_squared_error', optimizer=sgd)
+```
+
+You can either instantiate an optimizer before passing it to `model.compile()` , as in the above example, or you can call it by its name. In the latter case, the default parameters for the optimizer will be used.
+
+```python
+# pass optimizer by name: default parameters will be used
+model.compile(loss='mean_squared_error', optimizer='sgd')
+```
+
+---
+
+{{autogenerated}}

examples/addition_rnn.py

@@ -1,13 +1,5 @@
 # -*- coding: utf-8 -*-
-from __future__ import print_function
-from keras.models import Sequential, slice_X
-from keras.layers.core import Activation, TimeDistributedDense, RepeatVector
-from keras.layers import recurrent
-import numpy as np
-from six.moves import range
-
-"""
-An implementation of sequence to sequence learning for performing addition
+'''An implementation of sequence to sequence learning for performing addition
 Input: "535+61"
 Output: "596"
 Padding is handled by using a repeated sentinel character (space)
@@ -32,16 +24,23 @@ Four digits inverted:
 Five digits inverted:
 + One layer LSTM (128 HN), 550k training examples = 99% train/test accuracy in 30 epochs
 
-"""
+'''
+
+from __future__ import print_function
+from keras.models import Sequential, slice_X
+from keras.layers.core import Activation, TimeDistributedDense, RepeatVector
+from keras.layers import recurrent
+import numpy as np
+from six.moves import range
 
 
 class CharacterTable(object):
-    """
+    '''
     Given a set of characters:
     + Encode them to a one hot integer representation
     + Decode the one hot integer representation to their character output
     + Decode a vector of probabilties to their character output
-    """
+    '''
     def __init__(self, chars, maxlen):
         self.chars = sorted(set(chars))
         self.char_indices = dict((c, i) for i, c in enumerate(self.chars))
@@ -150,7 +149,8 @@ for iteration in range(1, 200):
     print()
     print('-' * 50)
     print('Iteration', iteration)
-    model.fit(X_train, y_train, batch_size=BATCH_SIZE, nb_epoch=1, validation_data=(X_val, y_val), show_accuracy=True)
+    model.fit(X_train, y_train, batch_size=BATCH_SIZE, nb_epoch=1,
+              validation_data=(X_val, y_val), show_accuracy=True)
     ###
     # Select 10 samples from the validation set at random so we can visualize errors
     for i in range(10):

examples/babi_memnn.py

@@ -1,3 +1,18 @@
+'''Train a memory network on the bAbI dataset.
+
+References:
+- Jason Weston, Antoine Bordes, Sumit Chopra, Tomas Mikolov, Alexander M. Rush,
+  "Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks",
+  http://arxiv.org/abs/1502.05698
+
+- Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, Rob Fergus,
+  "End-To-End Memory Networks",
+  http://arxiv.org/abs/1503.08895
+
+Reaches 98.6% accuracy on task 'single_supporting_fact_10k' after 120 epochs.
+Time per epoch: 3s on CPU (core i7).
+'''
+
 from __future__ import print_function
 from keras.models import Sequential
 from keras.layers.embeddings import Embedding
@@ -10,22 +25,6 @@ import tarfile
 import numpy as np
 import re
 
-"""
-Train a memory network on the bAbI dataset.
-
-References:
-- Jason Weston, Antoine Bordes, Sumit Chopra, Tomas Mikolov, Alexander M. Rush,
-  "Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks",
-  http://arxiv.org/abs/1503.08895
-
-- Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, Rob Fergus,
-  "End-To-End Memory Networks",
-  http://arxiv.org/abs/1503.08895
-
-Reaches 93% accuracy on task 'single_supporting_fact_10k' after 70 epochs.
-Time per epoch: 3s on CPU (core i7).
-"""
-
 
 def tokenize(sent):
     '''Return the tokens of a sentence including punctuation.
@@ -154,12 +153,14 @@ input_encoder_m = Sequential()
 input_encoder_m.add(Embedding(input_dim=vocab_size,
                               output_dim=64,
                               input_length=story_maxlen))
+input_encoder_m.add(Dropout(0.3))
 # output: (samples, story_maxlen, embedding_dim)
 # embed the question into a sequence of vectors
 question_encoder = Sequential()
 question_encoder.add(Embedding(input_dim=vocab_size,
                                output_dim=64,
                                input_length=query_maxlen))
+question_encoder.add(Dropout(0.3))
 # output: (samples, query_maxlen, embedding_dim)
 # compute a 'match' between input sequence elements (which are vectors)
 # and the question vector sequence
@@ -173,6 +174,7 @@ input_encoder_c = Sequential()
 input_encoder_c.add(Embedding(input_dim=vocab_size,
                               output_dim=query_maxlen,
                               input_length=story_maxlen))
+input_encoder_c.add(Dropout(0.3))
 # output: (samples, story_maxlen, query_maxlen)
 # sum the match vector with the input vector:
 response = Sequential()
@@ -186,9 +188,9 @@ answer = Sequential()
 answer.add(Merge([response, question_encoder], mode='concat', concat_axis=-1))
 # the original paper uses a matrix multiplication for this reduction step.
 # we choose to use a RNN instead.
-answer.add(LSTM(64))
+answer.add(LSTM(32))
 # one regularization layer -- more would probably be needed.
-answer.add(Dropout(0.25))
+answer.add(Dropout(0.3))
 answer.add(Dense(vocab_size))
 # we output a probability distribution over the vocabulary
 answer.add(Activation('softmax'))
@@ -197,7 +199,6 @@ answer.compile(optimizer='rmsprop', loss='categorical_crossentropy')
 # Note: you could use a Graph model to avoid repeat the input twice
 answer.fit([inputs_train, queries_train, inputs_train], answers_train,
            batch_size=32,
-           nb_epoch=70,
+           nb_epoch=120,
            show_accuracy=True,
            validation_data=([inputs_test, queries_test, inputs_test], answers_test))
-

examples/babi_rnn.py

@@ -1,21 +1,4 @@
-from __future__ import absolute_import
-from __future__ import print_function
-from functools import reduce
-import re
-import tarfile
-
-import numpy as np
-np.random.seed(1337)  # for reproducibility
-
-from keras.datasets.data_utils import get_file
-from keras.layers.embeddings import Embedding
-from keras.layers.core import Dense, Merge
-from keras.layers import recurrent
-from keras.models import Sequential
-from keras.preprocessing.sequence import pad_sequences
-
-'''
-Trains two recurrent neural networks based upon a story and a question.
+'''Trains two recurrent neural networks based upon a story and a question.
 The resulting merged vector is then queried to answer a range of bAbI tasks.
 
 The results are comparable to those for an LSTM model provided in Weston et al.:
@@ -73,6 +56,21 @@ noise to find the relevant statements, improving performance substantially.
 This becomes especially obvious on QA2 and QA3, both far longer than QA1.
 '''
 
+from __future__ import print_function
+from functools import reduce
+import re
+import tarfile
+
+import numpy as np
+np.random.seed(1337)  # for reproducibility
+
+from keras.datasets.data_utils import get_file
+from keras.layers.embeddings import Embedding
+from keras.layers.core import Dense, Merge
+from keras.layers import recurrent
+from keras.models import Sequential
+from keras.preprocessing.sequence import pad_sequences
+
 
 def tokenize(sent):
     '''Return the tokens of a sentence including punctuation.

examples/cifar10_cnn.py

@@ -1,4 +1,16 @@
-from __future__ import absolute_import
+'''Train a simple deep CNN on the CIFAR10 small images dataset.
+
+GPU run command:
+    THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 python cifar10_cnn.py
+
+It gets down to 0.65 test logloss in 25 epochs, and down to 0.55 after 50 epochs.
+(it's still underfitting at that point, though).
+
+Note: the data was pickled with Python 2, and some encoding issues might prevent you
+from loading it in Python 3. You might have to load it in Python 2,
+save it in a different format, load it in Python 3 and repickle it.
+'''
+
 from __future__ import print_function
 from keras.datasets import cifar10
 from keras.preprocessing.image import ImageDataGenerator
@@ -9,20 +21,6 @@ from keras.optimizers import SGD, Adadelta, Adagrad
 from keras.utils import np_utils, generic_utils
 from six.moves import range
 
-'''
-    Train a (fairly simple) deep CNN on the CIFAR10 small images dataset.
-
-    GPU run command:
-        THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 python cifar10_cnn.py
-
-    It gets down to 0.65 test logloss in 25 epochs, and down to 0.55 after 50 epochs.
-    (it's still underfitting at that point, though).
-
-    Note: the data was pickled with Python 2, and some encoding issues might prevent you
-    from loading it in Python 3. You might have to load it in Python 2,
-    save it in a different format, load it in Python 3 and repickle it.
-'''
-
 batch_size = 32
 nb_classes = 10
 nb_epoch = 200
@@ -71,19 +69,19 @@ model.add(Activation('softmax'))
 sgd = SGD(lr=0.01, decay=1e-6, momentum=0.9, nesterov=True)
 model.compile(loss='categorical_crossentropy', optimizer=sgd)
 
-X_train = X_train.astype("float32")
-X_test = X_test.astype("float32")
+X_train = X_train.astype('float32')
+X_test = X_test.astype('float32')
 X_train /= 255
 X_test /= 255
 
 if not data_augmentation:
-    print("Not using data augmentation or normalization")
+    print('Not using data augmentation or normalization')
     model.fit(X_train, Y_train, batch_size=batch_size, nb_epoch=nb_epoch)
     score = model.evaluate(X_test, Y_test, batch_size=batch_size)
     print('Test score:', score)
 
 else:
-    print("Using real time data augmentation")
+    print('Using real time data augmentation')
 
     # this will do preprocessing and realtime data augmentation
     datagen = ImageDataGenerator(
@@ -106,16 +104,16 @@ else:
         print('-'*40)
         print('Epoch', e)
         print('-'*40)
-        print("Training...")
+        print('Training...')
         # batch train with realtime data augmentation
         progbar = generic_utils.Progbar(X_train.shape[0])
         for X_batch, Y_batch in datagen.flow(X_train, Y_train):
             loss = model.train_on_batch(X_batch, Y_batch)
-            progbar.add(X_batch.shape[0], values=[("train loss", loss)])
+            progbar.add(X_batch.shape[0], values=[('train loss', loss[0])])
 
-        print("Testing...")
+        print('Testing...')
         # test time!
         progbar = generic_utils.Progbar(X_test.shape[0])
         for X_batch, Y_batch in datagen.flow(X_test, Y_test):
             score = model.test_on_batch(X_batch, Y_batch)
-            progbar.add(X_batch.shape[0], values=[("test loss", score)])
+            progbar.add(X_batch.shape[0], values=[('test loss', score[0])])

examples/imdb_bidirectional_lstm.py

@@ -1,4 +1,12 @@
-from __future__ import absolute_import
+'''Train a Bidirectional LSTM on the IMDB sentiment classification task.
+
+GPU command:
+    THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 python imdb_bidirectional_lstm.py
+
+Output after 4 epochs on CPU: ~0.8146
+Time per epoch on CPU (Core i7): ~150s.
+'''
+
 from __future__ import print_function
 import numpy as np
 np.random.seed(1337)  # for reproducibility
@@ -11,21 +19,12 @@ from keras.layers.embeddings import Embedding
 from keras.layers.recurrent import LSTM
 from keras.datasets import imdb
 
-'''
-    Train a Bidirectional LSTM on the IMDB sentiment classification task.
-
-    GPU command:
-        THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 python imdb_bidirectional_lstm.py
-
-    Output after 4 epochs on CPU: ~0.8146
-    Time per epoch on CPU (Core i7): ~150s.
-'''
 
 max_features = 20000
 maxlen = 100  # cut texts after this number of words (among top max_features most common words)
 batch_size = 32
 
-print("Loading data...")
+print('Loading data...')
 (X_train, y_train), (X_test, y_test) = imdb.load_data(nb_words=max_features,
                                                       test_split=0.2)
 print(len(X_train), 'train sequences')
@@ -53,7 +52,7 @@ model.add_output(name='output', input='sigmoid')
 # try using different optimizers and different optimizer configs
 model.compile('adam', {'output': 'binary_crossentropy'})
 
-print("Train...")
+print('Train...')
 model.fit({'input': X_train, 'output': y_train},
           batch_size=batch_size,
           nb_epoch=4)

examples/imdb_cnn.py

@@ -1,4 +1,10 @@
-from __future__ import absolute_import
+'''This example demonstrates the use of Convolution1D for text classification.
+
+Run on GPU: THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 python imdb_cnn.py
+
+Get to 0.835 test accuracy after 2 epochs. 100s/epoch on K520 GPU.
+'''
+
 from __future__ import print_function
 import numpy as np
 np.random.seed(1337)  # for reproducibility
@@ -10,14 +16,6 @@ from keras.layers.embeddings import Embedding
 from keras.layers.convolutional import Convolution1D, MaxPooling1D
 from keras.datasets import imdb
 
-'''
-    This example demonstrates the use of Convolution1D
-    for text classification.
-
-    Run on GPU: THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 python imdb_cnn.py
-
-    Get to 0.835 test accuracy after 2 epochs. 100s/epoch on K520 GPU.
-'''
 
 # set parameters:
 max_features = 5000
@@ -29,13 +27,13 @@ filter_length = 3
 hidden_dims = 250
 nb_epoch = 2
 
-print("Loading data...")
+print('Loading data...')
 (X_train, y_train), (X_test, y_test) = imdb.load_data(nb_words=max_features,
                                                       test_split=0.2)
 print(len(X_train), 'train sequences')
 print(len(X_test), 'test sequences')
 
-print("Pad sequences (samples x time)")
+print('Pad sequences (samples x time)')
 X_train = sequence.pad_sequences(X_train, maxlen=maxlen)
 X_test = sequence.pad_sequences(X_test, maxlen=maxlen)
 print('X_train shape:', X_train.shape)
@@ -53,8 +51,8 @@ model.add(Dropout(0.25))
 # word group filters of size filter_length:
 model.add(Convolution1D(nb_filter=nb_filter,
                         filter_length=filter_length,
-                        border_mode="valid",
-                        activation="relu",
+                        border_mode='valid',
+                        activation='relu',
                         subsample_length=1))
 # we use standard max pooling (halving the output of the previous layer):
 model.add(MaxPooling1D(pool_length=2))
@@ -74,7 +72,7 @@ model.add(Activation('sigmoid'))
 
 model.compile(loss='binary_crossentropy',
               optimizer='rmsprop',
-              class_mode="binary")
+              class_mode='binary')
 model.fit(X_train, y_train, batch_size=batch_size,
           nb_epoch=nb_epoch, show_accuracy=True,
           validation_data=(X_test, y_test))

examples/imdb_cnn_lstm.py

@@ -1,11 +1,17 @@
-from __future__ import absolute_import
+'''Train a recurrent convolutional network on the IMDB sentiment
+classification task.
+
+GPU command:
+    THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 python imdb_lstm.py
+
+Get to 0.8498 test accuracy after 2 epochs. 41s/epoch on K520 GPU.
+'''
+
 from __future__ import print_function
 import numpy as np
 np.random.seed(1337)  # for reproducibility
 
 from keras.preprocessing import sequence
-from keras.optimizers import SGD, RMSprop, Adagrad
-from keras.utils import np_utils
 from keras.models import Sequential
 from keras.layers.core import Dense, Dropout, Activation
 from keras.layers.embeddings import Embedding
@@ -13,14 +19,6 @@ from keras.layers.recurrent import LSTM, GRU, SimpleRNN
 from keras.layers.convolutional import Convolution1D, MaxPooling1D
 from keras.datasets import imdb
 
-'''
-    Train a recurrent convolutional network on the IMDB sentiment classification task.
-
-    GPU command:
-        THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 python imdb_lstm.py
-
-    Get to 0.8498 test accuracy after 2 epochs. 41s/epoch on K520 GPU.
-'''
 
 # Embedding
 max_features = 20000
@@ -45,12 +43,12 @@ batch_size is highly sensitive.
 Only 2 epochs are needed as the dataset is very small.
 '''
 
-print("Loading data...")
+print('Loading data...')
 (X_train, y_train), (X_test, y_test) = imdb.load_data(nb_words=max_features, test_split=0.2)
 print(len(X_train), 'train sequences')
 print(len(X_test), 'test sequences')
 
-print("Pad sequences (samples x time)")
+print('Pad sequences (samples x time)')
 X_train = sequence.pad_sequences(X_train, maxlen=maxlen)
 X_test = sequence.pad_sequences(X_test, maxlen=maxlen)
 print('X_train shape:', X_train.shape)
@@ -63,8 +61,8 @@ model.add(Embedding(max_features, embedding_size, input_length=maxlen))
 model.add(Dropout(0.25))
 model.add(Convolution1D(nb_filter=nb_filter,
                         filter_length=filter_length,
-                        border_mode="valid",
-                        activation="relu",
+                        border_mode='valid',
+                        activation='relu',
                         subsample_length=1))
 model.add(MaxPooling1D(pool_length=pool_length))
 model.add(LSTM(lstm_output_size))
@@ -73,9 +71,9 @@ model.add(Activation('sigmoid'))
 
 model.compile(loss='binary_crossentropy',
               optimizer='adam',
-              class_mode="binary")
+              class_mode='binary')
 
-print("Train...")
+print('Train...')
 model.fit(X_train, y_train, batch_size=batch_size, nb_epoch=nb_epoch,
           validation_data=(X_test, y_test), show_accuracy=True)
 score, acc = model.evaluate(X_test, y_test, batch_size=batch_size,

examples/imdb_lstm.py

@@ -1,4 +1,21 @@
-from __future__ import absolute_import
+'''Train a LSTM on the IMDB sentiment classification task.
+
+The dataset is actually too small for LSTM to be of any advantage
+compared to simpler, much faster methods such as TF-IDF+LogReg.
+
+Notes:
+
+- RNNs are tricky. Choice of batch size is important,
+choice of loss and optimizer is critical, etc.
+Some configurations won't converge.
+
+- LSTM loss decrease patterns during training can be quite different
+from what you see with CNNs/MLPs/etc.
+
+GPU command:
+    THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 python imdb_lstm.py
+'''
+
 from __future__ import print_function
 import numpy as np
 np.random.seed(1337)  # for reproducibility
@@ -11,30 +28,11 @@ from keras.layers.embeddings import Embedding
 from keras.layers.recurrent import LSTM
 from keras.datasets import imdb
 
-'''
-    Train a LSTM on the IMDB sentiment classification task.
-
-    The dataset is actually too small for LSTM to be of any advantage
-    compared to simpler, much faster methods such as TF-IDF+LogReg.
-
-    Notes:
-
-    - RNNs are tricky. Choice of batch size is important,
-    choice of loss and optimizer is critical, etc.
-    Some configurations won't converge.
-
-    - LSTM loss decrease patterns during training can be quite different
-    from what you see with CNNs/MLPs/etc.
-
-    GPU command:
-        THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 python imdb_lstm.py
-'''
-
 max_features = 20000
 maxlen = 100  # cut texts after this number of words (among top max_features most common words)
 batch_size = 32
 
-print("Loading data...")
+print('Loading data...')
 (X_train, y_train), (X_test, y_test) = imdb.load_data(nb_words=max_features,
                                                       test_split=0.2)
 print(len(X_train), 'train sequences')

examples/kaggle_otto_nn.py

@@ -1,6 +1,26 @@
-from __future__ import absolute_import
-from __future__ import print_function
+'''This demonstrates how to reach a score of 0.4890 (local validation)
+on the Kaggle Otto challenge, with a deep net using Keras.
 
+Requires Scikit-Learn and Pandas.
+
+Recommended to run on GPU:
+    Command: THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 python kaggle_otto_nn.py
+    On EC2 g2.2xlarge instance: 19s/epoch. 6-7 minutes total training time.
+
+Best validation score at epoch 21: 0.4881
+
+Try it at home:
+    - with/without BatchNormalization (BatchNormalization helps!)
+    - with ReLU or with PReLU (PReLU helps!)
+    - with smaller layers, largers layers
+    - with more layers, less layers
+    - with different optimizers (SGD+momentum+decay is probably better than Adam!)
+
+Get the data from Kaggle:
+https://www.kaggle.com/c/otto-group-product-classification-challenge/data
+'''
+
+from __future__ import print_function
 import numpy as np
 import pandas as pd
 np.random.seed(1337)  # for reproducibility
@@ -14,28 +34,6 @@ from keras.utils import np_utils, generic_utils
 from sklearn.preprocessing import LabelEncoder
 from sklearn.preprocessing import StandardScaler
 
-'''
-    This demonstrates how to reach a score of 0.4890 (local validation)
-    on the Kaggle Otto challenge, with a deep net using Keras.
-
-    Compatible Python 2.7-3.4. Requires Scikit-Learn and Pandas.
-
-    Recommended to run on GPU:
-        Command: THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 python kaggle_otto_nn.py
-        On EC2 g2.2xlarge instance: 19s/epoch. 6-7 minutes total training time.
-
-    Best validation score at epoch 21: 0.4881
-
-    Try it at home:
-        - with/without BatchNormalization (BatchNormalization helps!)
-        - with ReLU or with PReLU (PReLU helps!)
-        - with smaller layers, largers layers
-        - with more layers, less layers
-        - with different optimizers (SGD+momentum+decay is probably better than Adam!)
-
-    Get the data from Kaggle: https://www.kaggle.com/c/otto-group-product-classification-challenge/data
-'''
-
 
 def load_data(path, train=True):
     df = pd.read_csv(path)
@@ -76,9 +74,9 @@ def make_submission(y_prob, ids, encoder, fname):
             probas = ','.join([i] + [str(p) for p in probs.tolist()])
             f.write(probas)
             f.write('\n')
-    print("Wrote submission to file {}.".format(fname))
+    print('Wrote submission to file {}.'.format(fname))
 
-print("Loading data...")
+print('Loading data...')
 X, labels = load_data('train.csv', train=True)
 X, scaler = preprocess_data(X)
 y, encoder = preprocess_labels(labels)
@@ -92,7 +90,7 @@ print(nb_classes, 'classes')
 dims = X.shape[1]
 print(dims, 'dims')
 
-print("Building model...")
+print('Building model...')
 
 model = Sequential()
 model.add(Dense(512, input_shape=(dims,)))
@@ -113,11 +111,11 @@ model.add(Dropout(0.5))
 model.add(Dense(nb_classes))
 model.add(Activation('softmax'))
 
-model.compile(loss='categorical_crossentropy', optimizer="adam")
+model.compile(loss='categorical_crossentropy', optimizer='adam')
 
-print("Training model...")
+print('Training model...')
 model.fit(X, y, nb_epoch=20, batch_size=128, validation_split=0.15)
 
-print("Generating submission...")
+print('Generating submission...')
 proba = model.predict_proba(X_test)
 make_submission(proba, ids, encoder, fname='keras-otto.csv')

examples/lstm_text_generation.py

@@ -1,3 +1,15 @@
+'''Example script to generate text from Nietzsche's writings.
+
+At least 20 epochs are required before the generated text
+starts sounding coherent.
+
+It is recommended to run this script on GPU, as recurrent
+networks are quite computationally intensive.
+
+If you try this script on new data, make sure your corpus
+has at least ~100k characters. ~1M is better.
+'''
+
 from __future__ import print_function
 from keras.models import Sequential
 from keras.layers.core import Dense, Activation, Dropout
@@ -7,19 +19,6 @@ import numpy as np
 import random
 import sys
 
-'''
-    Example script to generate text from Nietzsche's writings.
-
-    At least 20 epochs are required before the generated text
-    starts sounding coherent.
-
-    It is recommended to run this script on GPU, as recurrent
-    networks are quite computationally intensive.
-
-    If you try this script on new data, make sure your corpus
-    has at least ~100k characters. ~1M is better.
-'''
-
 path = get_file('nietzsche.txt', origin="https://s3.amazonaws.com/text-datasets/nietzsche.txt")
 text = open(path).read().lower()
 print('corpus length:', len(text))
@@ -86,7 +85,7 @@ for iteration in range(1, 60):
         print('----- Generating with seed: "' + sentence + '"')
         sys.stdout.write(generated)
 
-        for iteration in range(400):
+        for i in range(400):
             x = np.zeros((1, maxlen, len(chars)))
             for t, char in enumerate(sentence):
                 x[0, t, char_indices[char]] = 1.

examples/mnist_cnn.py

@@ -1,4 +1,11 @@
-from __future__ import absolute_import
+'''Train a simple convnet on the MNIST dataset.
+
+Run on GPU: THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 python mnist_cnn.py
+
+Get to 99.25% test accuracy after 12 epochs (there is still a lot of margin for parameter tuning).
+16 seconds per epoch on a GRID K520 GPU.
+'''
+
 from __future__ import print_function
 import numpy as np
 np.random.seed(1337)  # for reproducibility
@@ -9,15 +16,6 @@ from keras.layers.core import Dense, Dropout, Activation, Flatten
 from keras.layers.convolutional import Convolution2D, MaxPooling2D
 from keras.utils import np_utils
 
-'''
-    Train a simple convnet on the MNIST dataset.
-
-    Run on GPU: THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 python mnist_cnn.py
-
-    Get to 99.25% test accuracy after 12 epochs (there is still a lot of margin for parameter tuning).
-    16 seconds per epoch on a GRID K520 GPU.
-'''
-
 batch_size = 128
 nb_classes = 10
 nb_epoch = 12
@@ -36,8 +34,8 @@ nb_conv = 3
 
 X_train = X_train.reshape(X_train.shape[0], 1, img_rows, img_cols)
 X_test = X_test.reshape(X_test.shape[0], 1, img_rows, img_cols)
-X_train = X_train.astype("float32")
-X_test = X_test.astype("float32")
+X_train = X_train.astype('float32')
+X_test = X_test.astype('float32')
 X_train /= 255
 X_test /= 255
 print('X_train shape:', X_train.shape)
@@ -51,7 +49,7 @@ Y_test = np_utils.to_categorical(y_test, nb_classes)
 model = Sequential()
 
 model.add(Convolution2D(nb_filters, nb_conv, nb_conv,
-                        border_mode='same',
+                        border_mode='valid',
                         input_shape=(1, img_rows, img_cols)))
 model.add(Activation('relu'))
 model.add(Convolution2D(nb_filters, nb_conv, nb_conv))
@@ -68,7 +66,8 @@ model.add(Activation('softmax'))
 
 model.compile(loss='categorical_crossentropy', optimizer='adadelta')
 
-model.fit(X_train, Y_train, batch_size=batch_size, nb_epoch=nb_epoch, show_accuracy=True, verbose=1, validation_data=(X_test, Y_test))
+model.fit(X_train, Y_train, batch_size=batch_size, nb_epoch=nb_epoch,
+          show_accuracy=True, verbose=1, validation_data=(X_test, Y_test))
 score = model.evaluate(X_test, Y_test, show_accuracy=True, verbose=0)
 print('Test score:', score[0])
 print('Test accuracy:', score[1])

examples/mnist_irnn.py

@@ -1,4 +1,18 @@
-from __future__ import absolute_import
+'''This is a reproduction of the IRNN experiment
+with pixel-by-pixel sequential MNIST in
+"A Simple Way to Initialize Recurrent Networks of Rectified Linear Units"
+by Quoc V. Le, Navdeep Jaitly, Geoffrey E. Hinton
+
+arXiv:1504.00941v2 [cs.NE] 7 Apr 201
+http://arxiv.org/pdf/1504.00941v2.pdf
+
+Optimizer is replaced with RMSprop which yields more stable and steady
+improvement.
+
+Reaches 0.93 train/test accuracy after 900 epochs
+(which roughly corresponds to 1687500 steps in the original paper.)
+'''
+
 from __future__ import print_function
 import numpy as np
 np.random.seed(1337)  # for reproducibility
@@ -11,21 +25,6 @@ from keras.layers.recurrent import SimpleRNN, LSTM
 from keras.optimizers import RMSprop
 from keras.utils import np_utils
 
-'''
-    This is a reproduction of the IRNN experiment
-    with pixel-by-pixel sequential MNIST in
-    "A Simple Way to Initialize Recurrent Networks of Rectified Linear Units "
-    by Quoc V. Le, Navdeep Jaitly, Geoffrey E. Hinton
-
-    arXiv:1504.00941v2 [cs.NE] 7 Apr 201
-    http://arxiv.org/pdf/1504.00941v2.pdf
-
-    Optimizer is replaced with RMSprop which yields more stable and steady
-    improvement.
-
-    Reaches 0.93 train/test accuracy after 900 epochs
-    (which roughly corresponds to 1687500 steps in the original paper.)
-'''
 
 batch_size = 32
 nb_classes = 10
@@ -40,8 +39,8 @@ clip_norm = 1.0
 
 X_train = X_train.reshape(X_train.shape[0], -1, 1)
 X_test = X_test.reshape(X_test.shape[0], -1, 1)
-X_train = X_train.astype("float32")
-X_test = X_test.astype("float32")
+X_train = X_train.astype('float32')
+X_test = X_test.astype('float32')
 X_train /= 255
 X_test /= 255
 print('X_train shape:', X_train.shape)

examples/mnist_mlp.py

@@ -1,4 +1,10 @@
-from __future__ import absolute_import
+'''Train a simple deep NN on the MNIST dataset.
+
+Get to 98.40% test accuracy after 20 epochs
+(there is *a lot* of margin for parameter tuning).
+2 seconds per epoch on a K520 GPU.
+'''
+
 from __future__ import print_function
 import numpy as np
 np.random.seed(1337)  # for reproducibility
@@ -9,13 +15,6 @@ from keras.layers.core import Dense, Dropout, Activation
 from keras.optimizers import SGD, Adam, RMSprop
 from keras.utils import np_utils
 
-'''
-    Train a simple deep NN on the MNIST dataset.
-
-    Get to 98.40% test accuracy after 20 epochs
-    (there is *a lot* of margin for parameter tuning).
-    2 seconds per epoch on a K520 GPU.
-'''
 
 batch_size = 128
 nb_classes = 10
@@ -26,8 +25,8 @@ nb_epoch = 20
 
 X_train = X_train.reshape(60000, 784)
 X_test = X_test.reshape(10000, 784)
-X_train = X_train.astype("float32")
-X_test = X_test.astype("float32")
+X_train = X_train.astype('float32')
+X_test = X_test.astype('float32')
 X_train /= 255
 X_test /= 255
 print(X_train.shape[0], 'train samples')

examples/mnist_transfer_cnn.py

@@ -1,4 +1,16 @@
-from __future__ import absolute_import
+'''Transfer learning toy example:
+
+1- Train a simple convnet on the MNIST dataset the first 5 digits [0..4].
+2- Freeze convolutional layers and fine-tune dense layers
+   for the classification of digits [5..9].
+
+Run on GPU: THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 python mnist_cnn.py
+
+Get to 99.8% test accuracy after 5 epochs
+for the first five digits classifier
+and 99.2% for the last five digits after transfer + fine-tuning.
+'''
+
 from __future__ import print_function
 import numpy as np
 import datetime
@@ -11,18 +23,6 @@ from keras.layers.core import Dense, Dropout, Activation, Flatten
 from keras.layers.convolutional import Convolution2D, MaxPooling2D
 from keras.utils import np_utils
 
-'''
-    Transfer learning toy example:
-        1- Train a simple convnet on the MNIST dataset the first 5 digits [0..4].
-        2- Freeze convolutional layers and fine-tune dense layers
-           for the classification of digits [5..9].
-
-    Run on GPU: THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 python mnist_cnn.py
-
-    Get to 99.8% test accuracy after 5 epochs
-    for the first five digits classifier
-    and 99.2% for the last five digits after transfer + fine-tuning.
-'''
 
 now = datetime.datetime.now
 
@@ -43,8 +43,8 @@ nb_conv = 3
 def train_model(model, train, test, nb_classes):
     X_train = train[0].reshape(train[0].shape[0], 1, img_rows, img_cols)
     X_test = test[0].reshape(test[0].shape[0], 1, img_rows, img_cols)
-    X_train = X_train.astype("float32")
-    X_test = X_test.astype("float32")
+    X_train = X_train.astype('float32')
+    X_test = X_test.astype('float32')
     X_train /= 255
     X_test /= 255
     print('X_train shape:', X_train.shape)

examples/neural_turing_machine_copy.py

@@ -1,168 +0,0 @@
-from __future__ import absolute_import
-from __future__ import print_function
-import numpy as np
-np.random.seed(123)
-import matplotlib.pyplot as plt
-
-from theano import function
-
-from keras.models import Sequential
-from keras.layers.core import TimeDistributedDense, Activation
-from keras.layers.recurrent import LSTM
-from keras.optimizers import Adam
-from keras.utils import generic_utils
-
-from keras.layers.ntm import NeuralTuringMachine as NTM
-
-"""
-Copy Problem defined in Graves et. al [0]
-
-Training data is made of sequences with length 1 to 20. 
-Test data are sequences of length 100.
-The model is tested every 500 weight updates.
-After about 3500 updates, the accuracy jumps from around 50% to >90%.
-
-Estimated compile time: 12 min
-Estimated time to train Neural Turing Machine and 3 layer LSTM on an NVidia GTX 680: 2h
-
-[0]: http://arxiv.org/pdf/1410.5401v2.pdf
-"""
-
-batch_size = 100
-
-h_dim = 128
-n_slots = 128
-m_length = 20
-input_dim = 8
-lr = 1e-3
-clipvalue = 10
-
-##### Neural Turing Machine ######
-
-ntm = NTM(h_dim, n_slots=n_slots, m_length=m_length, shift_range=3,
-          inner_rnn='lstm', return_sequences=True, input_dim=input_dim)
-model = Sequential()
-model.add(ntm)
-model.add(TimeDistributedDense(input_dim))
-model.add(Activation('sigmoid'))
-
-sgd = Adam(lr=lr, clipvalue=clipvalue)
-model.compile(loss='binary_crossentropy', optimizer=sgd)
-
-# LSTM - Run this for comparison
-
-sgd2 = Adam(lr=lr, clipvalue=clipvalue)
-lstm = Sequential()
-lstm.add(LSTM(input_dim=input_dim, output_dim=h_dim*2, return_sequences=True))
-lstm.add(LSTM(output_dim=h_dim*2, return_sequences=True))
-lstm.add(LSTM(output_dim=h_dim*2, return_sequences=True))
-lstm.add(TimeDistributedDense(input_dim))
-lstm.add(Activation('sigmoid'))
-
-lstm.compile(loss='binary_crossentropy', optimizer=sgd)
-
-###### DATASET ########
-
-def get_sample(batch_size=128, n_bits=8, max_size=20, min_size=1):
-    # generate samples with random length
-    inp = np.zeros((batch_size, 2*max_size-1, n_bits))
-    out = np.zeros((batch_size, 2*max_size-1, n_bits))
-    sw = np.zeros((batch_size, 2*max_size-1, 1))
-    for i in range(batch_size):
-        t = np.random.randint(low=min_size, high=max_size)
-        x = np.random.uniform(size=(t, n_bits)) > .5
-        for j,f in enumerate(x.sum(axis=-1)): # remove fake flags
-            if f>=n_bits:
-                x[j, :] = 0.
-        del_flag = np.ones((1, n_bits))
-        inp[i, :t+1] = np.concatenate([x, del_flag], axis=0)
-        out[i, t+1:(2*t+1)] = x
-        sw[i, t+1:(2*t+1)] = 1
-    return inp, out, sw
-
-def show_pattern(inp, out, sw, file_name='ntm_output.png'):
-    ''' Helper function to visualize results '''
-    plt.figure(figsize=(10, 10))
-    plt.subplot(131)
-    plt.imshow(inp>.5)
-    plt.subplot(132)
-    plt.imshow(out>.5)
-    plt.subplot(133)
-    plt.imshow(sw>.5)
-    plt.savefig(file_name)
-    plt.close()
-
-# Show data example:
-inp, out, sw = get_sample(1, 8, 20)
-
-plt.subplot(131)
-plt.title('input')
-plt.imshow(inp[0], cmap='gray')
-plt.subplot(132)
-plt.title('desired')
-plt.imshow(out[0], cmap='gray')
-plt.subplot(133)
-plt.title('sample_weight')
-plt.imshow(sw[0], cmap='gray')
-
-# training uses sequences of length 1 to 20. Test uses series of length 100.
-def test_model(model, file_name, min_size=100):
-    I, V, sw = get_sample(batch_size=500, n_bits=input_dim, max_size=min_size+1, min_size=min_size)
-    Y = np.asarray(model.predict(I, batch_size=100) > .5).astype('float64')
-    acc = (V[:, -min_size:, :] == Y[:, -min_size:, :]).mean() * 100
-    show_pattern(Y[0], V[0], sw[0], file_name)
-    return acc
-
-##### TRAIN ######
-nb_epoch = 4000
-progbar = generic_utils.Progbar(nb_epoch)
-for e in range(nb_epoch):
-    I, V, sw = get_sample(n_bits=input_dim, max_size=20, min_size=1, batch_size=100)
-
-    loss1 = model.train_on_batch(I, V, sample_weight=sw)
-    loss2 = lstm.train_on_batch(I, V, sample_weight=sw)
-
-    progbar.add(1, values=[("NTM", loss1), ("LSTM", loss2)])
-
-    if e % 500 == 0:
-        print("")
-        acc1 = test_model(model, 'ntm.png')
-        acc2 = test_model(lstm, 'lstm.png')
-        print("NTM  test acc: {}".format(acc1))
-        print("LSTM test acc: {}".format(acc2))
-
-##### VISUALIZATION #####
-X = model.get_input()
-Y = ntm.get_full_output()[0:3]  # (memory over time, read_vectors, write_vectors)
-F = function([X], Y, allow_input_downcast=True)
-
-inp, out, sw = get_sample(1, 8, 21, 20)
-mem, read, write = F(inp.astype('float32'))
-Y = model.predict(inp)
-
-plt.figure(figsize=(15, 12))
-
-plt.subplot(221)
-plt.imshow(write[0])
-plt.xlabel('memory location')
-plt.ylabel('time')
-plt.title('write')
-
-plt.subplot(222)
-plt.imshow(read[0])
-plt.title('read')
-
-plt.subplot(223)
-plt.title('desired')
-plt.imshow(out[0])
-
-plt.subplot(224)
-plt.imshow(Y[0]>.5)
-plt.title('output')
-
-plt.figure(figsize=(15, 10))
-plt.subplot(325)
-plt.ylabel('time')
-plt.xlabel('location')
-plt.title('memory evolving in time (avg value per location)')
-plt.imshow(mem[0].mean(axis=-1))

examples/reuters_mlp.py

@@ -1,4 +1,10 @@
-from __future__ import absolute_import
+'''Train and evaluate a simple MLP on the Reuters newswire topic classification task.
+GPU run command:
+    THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 python examples/reuters_mlp.py
+CPU run command:
+    python examples/reuters_mlp.py
+'''
+
 from __future__ import print_function
 import numpy as np
 np.random.seed(1337)  # for reproducibility
@@ -10,19 +16,11 @@ from keras.layers.normalization import BatchNormalization
 from keras.utils import np_utils
 from keras.preprocessing.text import Tokenizer
 
-'''
-    Train and evaluate a simple MLP on the Reuters newswire topic classification task.
-    GPU run command:
-        THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 python examples/reuters_mlp.py
-    CPU run command:
-        python examples/reuters_mlp.py
-'''
-
 max_words = 1000
 batch_size = 32
 nb_epoch = 5
 
-print("Loading data...")
+print('Loading data...')
 (X_train, y_train), (X_test, y_test) = reuters.load_data(nb_words=max_words, test_split=0.2)
 print(len(X_train), 'train sequences')
 print(len(X_test), 'test sequences')
@@ -30,20 +28,20 @@ print(len(X_test), 'test sequences')
 nb_classes = np.max(y_train)+1
 print(nb_classes, 'classes')
 
-print("Vectorizing sequence data...")
+print('Vectorizing sequence data...')
 tokenizer = Tokenizer(nb_words=max_words)
-X_train = tokenizer.sequences_to_matrix(X_train, mode="binary")
-X_test = tokenizer.sequences_to_matrix(X_test, mode="binary")
+X_train = tokenizer.sequences_to_matrix(X_train, mode='binary')
+X_test = tokenizer.sequences_to_matrix(X_test, mode='binary')
 print('X_train shape:', X_train.shape)
 print('X_test shape:', X_test.shape)
 
-print("Convert class vector to binary class matrix (for use with categorical_crossentropy)")
+print('Convert class vector to binary class matrix (for use with categorical_crossentropy)')
 Y_train = np_utils.to_categorical(y_train, nb_classes)
 Y_test = np_utils.to_categorical(y_test, nb_classes)
 print('Y_train shape:', Y_train.shape)
 print('Y_test shape:', Y_test.shape)
 
-print("Building model...")
+print('Building model...')
 model = Sequential()
 model.add(Dense(512, input_shape=(max_words,)))
 model.add(Activation('relu'))

keras/__init__.py

@@ -1,15 +1 @@
-
-"""
-Keras: Theano-based Deep Learning library
-==================================
-Keras is a minimalist, highly modular neural network library in 
-the spirit of Torch, written in Python / Theano so as not to have 
-to deal with the dearth of ecosystem in Lua. It was developed with 
-a focus on enabling fast experimentation. Being able to go from 
-idea to result with the least possible delay is key to doing 
-good research.
-
-See http://keras.io/
-"""
-
-__version__ = '0.2.0'
+__version__ = '0.3.0'

keras/activations.py

@@ -39,7 +39,7 @@ def hard_sigmoid(x):
 
 def linear(x):
     '''
-    The function returns the variable that is passed in, so all types work
+    The function returns the variable that is passed in, so all types work.
     '''
     return x
 

keras/backend/__init__.py

@@ -4,12 +4,16 @@ import os
 import json
 from .common import epsilon, floatx, set_epsilon, set_floatx
 
-_keras_dir = os.path.expanduser(os.path.join('~', '.keras'))
+_keras_base_dir = os.path.expanduser('~')
+if not os.access(_keras_base_dir, os.W_OK):
+    _keras_base_dir = '/tmp'
+
+_keras_dir = os.path.join(_keras_base_dir, '.keras')
 if not os.path.exists(_keras_dir):
     os.makedirs(_keras_dir)
 
 _BACKEND = 'theano'
-_config_path = os.path.expanduser(os.path.join('~', '.keras', 'keras.json'))
+_config_path = os.path.expanduser(os.path.join(_keras_dir, 'keras.json'))
 if os.path.exists(_config_path):
     _config = json.load(open(_config_path))
     _floatx = _config.get('floatx', floatx())
@@ -27,7 +31,14 @@ else:
     _config = {'floatx': floatx(),
                'epsilon': epsilon(),
                'backend': _BACKEND}
-    json.dump(_config, open(_config_path, 'w'))
+    with open(_config_path, 'w') as f:
+        # add new line in order for bash 'cat' display the content correctly
+        f.write(json.dumps(_config) + '\n')
+
+if 'KERAS_BACKEND' in os.environ:
+    _backend = os.environ['KERAS_BACKEND']
+    assert _backend in {'theano', 'tensorflow'}
+    _BACKEND = _backend
 
 if _BACKEND == 'theano':
     print('Using Theano backend.')
@@ -36,4 +47,4 @@ elif _BACKEND == 'tensorflow':
     print('Using TensorFlow backend.')
     from .tensorflow_backend import *
 else:
-    raise Exception('Unknown backend: ' + str(backend))
+    raise Exception('Unknown backend: ' + str(_BACKEND))

keras/backend/common.py

@@ -22,6 +22,7 @@ def set_floatx(floatx):
     global _FLOATX
     if floatx not in {'float32', 'float64'}:
         raise Exception('Unknown floatx type: ' + str(floatx))
+    floatx = str(floatx)
     _FLOATX = floatx
 
 

keras/backend/tensorflow_backend.py

@@ -236,6 +236,20 @@ def permute_dimensions(x, pattern):
     return tf.transpose(x, perm=pattern)
 
 
+def repeat_elements(x, rep, axis):
+    '''Repeats the elements of a tensor along an axis, like np.repeat
+
+    If x has shape (s1, s2, s3) and axis=1, the output
+    will have shape (s1, s2 * rep, s3)
+    '''
+    x_shape = x.get_shape().as_list()
+    # slices along the repeat axis
+    splits = tf.split(axis, x_shape[axis], x)
+    # repeat each slice the given number of reps
+    x_rep = [s for s in splits for i in range(rep)]
+    return tf.concat(axis, x_rep)
+
+
 def repeat(x, n):
     '''Repeat a 2D tensor:
 
@@ -274,9 +288,6 @@ def squeeze(x, axis):
 def temporal_padding(x, padding=1):
     '''Pad the middle dimension of a 3D tensor
     with "padding" zeros left and right.
-
-    Appologies for the inane API, but Theano makes this
-    really hard.
     '''
     pattern = [[0, 0], [padding, padding], [0, 0]]
     return tf.pad(x, pattern)
@@ -391,14 +402,14 @@ def rnn(step_function, inputs, initial_states,
             # if all-zero input timestep, return
             # all-zero output and unchanged states
             switch = tf.reduce_any(input)
-            output = tf.control_flow_ops.cond(switch,
-                                              lambda: output,
-                                              lambda: 0. * output)
+            output = tf.python.control_flow_ops.cond(switch,
+                                                     lambda: output,
+                                                     lambda: 0. * output)
             return_states = []
             for state, new_state in zip(states, new_states):
-                return_states.append(tf.control_flow_ops.cond(switch,
-                                                              lambda: new_state,
-                                                              lambda: state))
+                return_states.append(tf.python.control_flow_ops.cond(switch,
+                                                                     lambda: new_state,
+                                                                     lambda: state))
             states = return_states
         else:
             states = new_states
@@ -416,9 +427,9 @@ def rnn(step_function, inputs, initial_states,
 def switch(condition, then_expression, else_expression):
     '''condition: scalar tensor.
     '''
-    return tf.control_flow_ops.cond(condition,
-                                    lambda: then_expression,
-                                    lambda: else_expression)
+    return tf.python.control_flow_ops.cond(condition,
+                                           lambda: then_expression,
+                                           lambda: else_expression)
 
 
 # NN OPERATIONS
@@ -502,7 +513,8 @@ def dropout(x, level, seed=None):
 # CONVOLUTIONS
 
 
-def conv2d(x, kernel, strides=(1, 1), border_mode='valid', dim_ordering='th'):
+def conv2d(x, kernel, strides=(1, 1), border_mode='valid', dim_ordering='th',
+           image_shape=None, filter_shape=None):
     '''
     Run on cuDNN if available.
     border_mode: string, "same" or "valid".
@@ -544,8 +556,8 @@ def conv2d(x, kernel, strides=(1, 1), border_mode='valid', dim_ordering='th'):
     return x
 
 
-def maxpool2d(x, pool_size, strides=(1, 1),
-              border_mode='valid', dim_ordering='th'):
+def pool2d(x, pool_size, strides=(1, 1),
+           border_mode='valid', dim_ordering='th', pool_mode='max'):
     '''
     pool_size: tuple of 2 integers.
     strides: tuple of 2 integers.
@@ -566,18 +578,23 @@ def maxpool2d(x, pool_size, strides=(1, 1),
         # tf max_pool only supports float32
         x = tf.cast(x, 'float32')
 
-    if dim_ordering == 'th':
-        # TF uses the last dimension as channel dimension,
-        # instead of the 2nd one.
-        # TH input shape: (samples, input_depth, rows, cols)
-        # TF input shape: (samples, rows, cols, input_depth)
-        # TH kernel shape: (depth, input_depth, rows, cols)
-        # TF kernel shape: (rows, cols, input_depth, depth)
-        x = tf.transpose(x, (0, 2, 3, 1))
-        x = tf.nn.max_pool(x, pool_size, strides, padding=padding)
-        x = tf.transpose(x, (0, 3, 1, 2))
-    elif dim_ordering == 'tf':
-        x = tf.nn.max_pool(x, pool_size, strides, padding=padding)
+    if dim_ordering in {'tf', 'th'}:
+        if dim_ordering == 'th':
+            # TF uses the last dimension as channel dimension,
+            # instead of the 2nd one.
+            # TH input shape: (samples, input_depth, rows, cols)
+            # TF input shape: (samples, rows, cols, input_depth)
+            # TH kernel shape: (depth, input_depth, rows, cols)
+            # TF kernel shape: (rows, cols, input_depth, depth)
+            x = tf.transpose(x, (0, 2, 3, 1))
+        if pool_mode == 'max':
+            x = tf.nn.max_pool(x, pool_size, strides, padding=padding)
+        elif pool_mode == 'avg':
+            x = tf.nn.avg_pool(x, pool_size, strides, padding=padding)
+        else:
+            raise Exception('Invalid pooling mode: ' + str(pool_mode))
+        if dim_ordering == 'th':
+            x = tf.transpose(x, (0, 3, 1, 2))
     else:
         raise Exception('Unknown dim_ordering: ' + str(dim_ordering))
 

keras/backend/theano_backend.py

@@ -11,7 +11,7 @@ theano.config.floatX = _FLOATX
 
 
 def _on_gpu():
-    '''Returns whether the session is set to
+    '''Return whether the session is set to
     run on GPU or not (i.e. on CPU).
     '''
     return theano.config.device[:3] == 'gpu'
@@ -19,7 +19,7 @@ def _on_gpu():
 
 if _on_gpu():
     '''Import cuDNN only if running on GPU:
-    not having Cuda install should not
+    not having Cuda installed should not
     prevent from running the present code.
     '''
     from theano.sandbox.cuda import dnn
@@ -243,11 +243,19 @@ def permute_dimensions(x, pattern):
     return x.dimshuffle(pattern)
 
 
-def repeat(x, n):
-    '''Repeat a 2D tensor:
+def repeat_elements(x, rep, axis):
+    '''Repeat the elements of a tensor along an axis, like np.repeat.
 
-    if x has shape (samples, dim) and n=2,
-    the output will have shape (samples, 2, dim)
+    If x has shape (s1, s2, s3) and axis=1, the output
+    will have shape (s1, s2 * rep, s3).
+    '''
+    return T.repeat(x, rep, axis=axis)
+
+def repeat(x, n):
+    '''Repeat a 2D tensor.
+
+    If x has shape (samples, dim) and n=2,
+    the output will have shape (samples, 2, dim).
     '''
     tensors = [x] * n
     stacked = T.stack(*tensors)
@@ -369,7 +377,7 @@ def gradients(loss, variables):
 
 def rnn(step_function, inputs, initial_states,
         go_backwards=False, masking=True):
-    '''Iterates over the time dimension of a tensor.
+    '''Iterate over the time dimension of a tensor.
 
     Parameters
     ----------
@@ -407,15 +415,12 @@ def rnn(step_function, inputs, initial_states,
     '''
     inputs = inputs.dimshuffle((1, 0, 2))
 
-    def _step(*args):
-        global single_result
-        input = args[0]
-        states = args[1:]
+    def _step(input, *states):
         output, new_states = step_function(input, states)
         if masking:
             # if all-zero input timestep, return
             # all-zero output and unchanged states
-            switch = T.any(input)
+            switch = T.any(input, axis=-1, keepdims=True)
             output = T.switch(switch, output, 0. * output)
             return_states = []
             for state, new_state in zip(states, new_states):
@@ -515,7 +520,8 @@ def dropout(x, level, seed=None):
 # CONVOLUTIONS
 
 
-def conv2d(x, kernel, strides=(1, 1), border_mode='valid', dim_ordering='th'):
+def conv2d(x, kernel, strides=(1, 1), border_mode='valid', dim_ordering='th',
+           image_shape=None, filter_shape=None):
     '''
     Run on cuDNN if available.
     border_mode: string, "same" or "valid".
@@ -532,15 +538,24 @@ def conv2d(x, kernel, strides=(1, 1), border_mode='valid', dim_ordering='th'):
         # TF kernel shape: (rows, cols, input_depth, depth)
         x = x.dimshuffle((0, 3, 1, 2))
         kernel = kernel.dimshuffle((3, 2, 0, 1))
+        if image_shape:
+            image_shape = (image_shape[0], image_shape[3],
+                           image_shape[1], image_shape[2])
+        if filter_shape:
+            filter_shape = (filter_shape[3], filter_shape[2],
+                            filter_shape[0], filter_shape[1])
 
     if _on_gpu() and dnn.dnn_available():
         if border_mode == 'same':
             assert(strides == (1, 1))
-            pad_x = (kernel.shape[2] - strides[0]) // 2
-            pad_y = (kernel.shape[3] - strides[1]) // 2
             conv_out = dnn.dnn_conv(img=x,
                                     kerns=kernel,
-                                    border_mode=(pad_x, pad_y))
+                                    border_mode='full')
+            shift_x = (kernel.shape[2] - 1) // 2
+            shift_y = (kernel.shape[3] - 1) // 2
+            conv_out = conv_out[:, :,
+                                shift_x:x.shape[2] + shift_x,
+                                shift_y:x.shape[3] + shift_y]
         else:
             conv_out = dnn.dnn_conv(img=x,
                                     kerns=kernel,
@@ -557,7 +572,9 @@ def conv2d(x, kernel, strides=(1, 1), border_mode='valid', dim_ordering='th'):
 
         conv_out = T.nnet.conv.conv2d(x, kernel,
                                       border_mode=th_border_mode,
-                                      subsample=strides)
+                                      subsample=strides,
+                                      image_shape=image_shape,
+                                      filter_shape=filter_shape)
         if border_mode == 'same':
             shift_x = (kernel.shape[2] - 1) // 2
             shift_y = (kernel.shape[3] - 1) // 2
@@ -569,8 +586,8 @@ def conv2d(x, kernel, strides=(1, 1), border_mode='valid', dim_ordering='th'):
     return conv_out
 
 
-def maxpool2d(x, pool_size, strides=(1, 1), border_mode='valid',
-              dim_ordering='th'):
+def pool2d(x, pool_size, strides=(1, 1), border_mode='valid',
+           dim_ordering='th', pool_mode='max'):
     if border_mode == 'same':
         # TODO: add implementation for border_mode="same"
         raise Exception('border_mode="same" not supported with Theano.')
@@ -586,19 +603,26 @@ def maxpool2d(x, pool_size, strides=(1, 1), border_mode='valid',
     if dim_ordering == 'tf':
         x = x.dimshuffle((0, 3, 1, 2))
 
-    pool_out = downsample.max_pool_2d(x,
-                                      ds=pool_size,
-                                      st=strides,
-                                      ignore_border=ignore_border,
-                                      padding=padding,
-                                      mode='average_exc_pad')
+    if pool_mode == 'max':
+        pool_out = downsample.max_pool_2d(x, ds=pool_size, st=strides,
+                                          ignore_border=ignore_border,
+                                          padding=padding,
+                                          mode='max')
+    elif pool_mode == 'avg':
+        pool_out = downsample.max_pool_2d(x, ds=pool_size, st=strides,
+                                          ignore_border=ignore_border,
+                                          padding=padding,
+                                          mode='average_exc_pad')
+    else:
+        raise Exception('Invalid pooling mode: ' + str(pool_mode))
+
     if dim_ordering == 'tf':
         pool_out = pool_out.dimshuffle((0, 2, 3, 1))
     return pool_out
 
-
 # RANDOMNESS
 
+
 def random_normal(shape, mean=0.0, std=1.0, dtype=_FLOATX, seed=None):
     if seed is None:
         seed = np.random.randint(10e6)
@@ -612,8 +636,6 @@ def random_uniform(shape, low=0.0, high=1.0, dtype=_FLOATX, seed=None):
     rng = RandomStreams(seed=seed)
     return rng.uniform(shape, low=low, high=high, dtype=dtype)
 
-
-
 '''
 more TODO:
 

keras/callbacks.py

@@ -8,6 +8,7 @@ import warnings
 
 from collections import deque
 from .utils.generic_utils import Progbar
+from keras import backend as K
 
 
 class CallbackList(object):
@@ -43,21 +44,27 @@ class CallbackList(object):
             callback.on_batch_begin(batch, logs)
         self._delta_ts_batch_begin.append(time.time() - t_before_callbacks)
         delta_t_median = np.median(self._delta_ts_batch_begin)
-        if self._delta_t_batch > 0. and delta_t_median > 0.95 * self._delta_t_batch and delta_t_median > 0.1:
+        if self._delta_t_batch > 0. and delta_t_median > 0.95 * \
+           self._delta_t_batch and delta_t_median > 0.1:
             warnings.warn('Method on_batch_begin() is slow compared '
-                          'to the batch update (%f). Check your callbacks.' % delta_t_median)
+                          'to the batch update (%f). Check your callbacks.'
+                          % delta_t_median)
         self._t_enter_batch = time.time()
 
     def on_batch_end(self, batch, logs={}):
+        if not hasattr(self, '_t_enter_batch'):
+            self._t_enter_batch = time.time()
         self._delta_t_batch = time.time() - self._t_enter_batch
         t_before_callbacks = time.time()
         for callback in self.callbacks:
             callback.on_batch_end(batch, logs)
         self._delta_ts_batch_end.append(time.time() - t_before_callbacks)
         delta_t_median = np.median(self._delta_ts_batch_end)
-        if self._delta_t_batch > 0. and delta_t_median > 0.95 * self._delta_t_batch and delta_t_median > 0.1:
+        if self._delta_t_batch > 0. and delta_t_median > 0.95 * \
+           self._delta_t_batch and delta_t_median > 0.1:
             warnings.warn('Method on_batch_end() is slow compared '
-                          'to the batch update (%f). Check your callbacks.' % delta_t_median)
+                          'to the batch update (%f). Check your callbacks.'
+                          % delta_t_median)
 
     def on_train_begin(self, logs={}):
         for callback in self.callbacks:
@@ -69,7 +76,30 @@ class CallbackList(object):
 
 
 class Callback(object):
+    '''Abstract base class used to build new callbacks.
 
+    # Properties
+        params: dict. Training parameters
+            (eg. verbosity, batch size, number of epochs...).
+        model: instance of `keras.models.Model`.
+            Reference of the model being trained.
+
+    The `logs` dictionary that callback methods
+    take as argument will contain keys for quantities relevant to
+    the current batch or epoch.
+
+    Currently, the `.fit()` method of the `Sequential` model class
+    will include the following quantities in the `logs` that
+    it passes to its callbacks:
+
+        on_epoch_end: logs optionally include `val_loss`
+            (if validation is enabled in `fit`), and `val_acc`
+            (if validation and accuracy monitoring are enabled).
+        on_batch_begin: logs include `size`,
+            the number of samples in the current batch.
+        on_batch_end: logs include `loss`, and optionally `acc`
+            (if accuracy monitoring is enabled).
+    '''
     def __init__(self):
         pass
 
@@ -99,6 +129,12 @@ class Callback(object):
 
 
 class BaseLogger(Callback):
+    '''Callback that prints events to the standard output.
+
+    This callback is automatically applied to
+    every Keras model (it is the basis of the verbosity modes
+    in models).
+    '''
     def on_train_begin(self, logs={}):
         self.verbose = self.params['verbose']
         self.nb_epoch = self.params['nb_epoch']
@@ -128,7 +164,8 @@ class BaseLogger(Callback):
             if k in logs:
                 self.log_values.append((k, logs[k]))
 
-        # skip progbar update for the last batch; will be handled by on_epoch_end
+        # skip progbar update for the last batch;
+        # will be handled by on_epoch_end
         if self.verbose and self.seen < self.params['nb_sample']:
             self.progbar.update(self.seen, self.log_values)
 
@@ -143,7 +180,13 @@ class BaseLogger(Callback):
 
 
 class History(Callback):
+    '''Callback that records events
+    into a `History` object.
 
+    This callback is automatically applied to
+    every Keras model. The `History` object
+    gets returned by the `fit` method of models.
+    '''
     def on_train_begin(self, logs={}):
         self.epoch = []
         self.history = {}
@@ -175,26 +218,56 @@ class History(Callback):
 
 
 class ModelCheckpoint(Callback):
-    def __init__(self, filepath, monitor='val_loss', verbose=0, save_best_only=False, mode='auto'):
+    '''Save the model after every epoch.
+
+    `filepath` can contain named formatting options,
+    which will be filled the value of `epoch` and
+    keys in `logs` (passed in `on_epoch_end`).
+
+    For example: if `filepath` is `weights.{epoch:02d}-{val_loss:.2f}.hdf5`,
+    then multiple files will be save with the epoch number and
+    the validation loss.
+
+    # Arguments
+        filepath: string, path to save the model file.
+        monitor: quantity to monitor.
+        verbose: verbosity mode, 0 or 1.
+        save_best_only: if `save_best_only=True`,
+            the latest best model according to
+            the validation loss will not be overwritten.
+        mode: one of {auto, min, max}.
+            If `save_best_only=True`, the decision
+            to overwrite the current save file is made
+            based on either the maximization or the
+            minization of the monitored. For `val_acc`,
+            this should be `max`, for `val_loss` this should
+            be `min`, etc. In `auto` mode, the direction is
+            automatically inferred from the name of the monitored quantity.
+
+    '''
+    def __init__(self, filepath, monitor='val_loss', verbose=0,
+                 save_best_only=False, mode='auto'):
 
         super(Callback, self).__init__()
         self.monitor = monitor
         self.verbose = verbose
         self.filepath = filepath
         self.save_best_only = save_best_only
-        
+
         if mode not in ['auto', 'min', 'max']:
-            warnings.warn("ModelCheckpoint mode %s is unknown, fallback to auto mode" % (self.mode), RuntimeWarning)
+            warnings.warn('ModelCheckpoint mode %s is unknown, '
+                          'fallback to auto mode.' % (self.mode),
+                          RuntimeWarning)
             mode = 'auto'
-            
-        if mode == "min":
+
+        if mode == 'min':
             self.monitor_op = np.less
             self.best = np.Inf
-        elif mode == "max":
+        elif mode == 'max':
             self.monitor_op = np.greater
             self.best = -np.Inf
         else:
-            if "acc" in self.monitor:
+            if 'acc' in self.monitor:
                 self.monitor_op = np.greater
                 self.best = -np.Inf
             else:
@@ -206,50 +279,97 @@ class ModelCheckpoint(Callback):
         if self.save_best_only:
             current = logs.get(self.monitor)
             if current is None:
-                warnings.warn("Can save best model only with %s available, skipping." % (self.monitor), RuntimeWarning)
+                warnings.warn('Can save best model only with %s available, '
+                              'skipping.' % (self.monitor), RuntimeWarning)
             else:
                 if self.monitor_op(current, self.best):
                     if self.verbose > 0:
-                        print("Epoch %05d: %s improved from %0.5f to %0.5f, saving model to %s"
-                              % (epoch, self.monitor, self.best, current, filepath))
+                        print('Epoch %05d: %s improved from %0.5f to %0.5f,'
+                              ' saving model to %s'
+                              % (epoch, self.monitor, self.best,
+                                 current, filepath))
                     self.best = current
                     self.model.save_weights(filepath, overwrite=True)
                 else:
                     if self.verbose > 0:
-                        print("Epoch %05d: %s did not improve" % (epoch, self.monitor))
+                        print('Epoch %05d: %s did not improve' %
+                              (epoch, self.monitor))
         else:
             if self.verbose > 0:
-                print("Epoch %05d: saving model to %s" % (epoch, filepath))
+                print('Epoch %05d: saving model to %s' % (epoch, filepath))
             self.model.save_weights(filepath, overwrite=True)
 
 
 class EarlyStopping(Callback):
-    def __init__(self, monitor='val_loss', patience=0, verbose=0):
+    '''Stop training when a monitored quantity has stopped improving.
+
+    # Arguments
+        monitor: quantity to be monitored.
+        patience: number of epochs with no improvement
+            after which training will be stopped.
+        verbose: verbosity mode.
+        mode: one of {auto, min, max}. In 'min' mode,
+            training will stop when the quantity
+            monitored has stopped decreasing; in 'max'
+            mode it will stop when the quantity
+            monitored has stopped increasing.
+    '''
+    def __init__(self, monitor='val_loss', patience=0, verbose=0, mode='auto'):
         super(Callback, self).__init__()
 
         self.monitor = monitor
         self.patience = patience
         self.verbose = verbose
-        self.best = np.Inf
         self.wait = 0
 
+        if mode not in ['auto', 'min', 'max']:
+            warnings.warn('EarlyStopping mode %s is unknown, '
+                          'fallback to auto mode.' % (self.mode), RuntimeWarning)
+            mode = 'auto'
+
+        if mode == 'min':
+            self.monitor_op = np.less
+            self.best = np.Inf
+        elif mode == 'max':
+            self.monitor_op = np.greater
+            self.best = -np.Inf
+        else:
+            if 'acc' in self.monitor:
+                self.monitor_op = np.greater
+                self.best = -np.Inf
+            else:
+                self.monitor_op = np.less
+                self.best = np.Inf
+
     def on_epoch_end(self, epoch, logs={}):
         current = logs.get(self.monitor)
         if current is None:
-            warnings.warn("Early stopping requires %s available!" % (self.monitor), RuntimeWarning)
+            warnings.warn('Early stopping requires %s available!' %
+                          (self.monitor), RuntimeWarning)
 
-        if current < self.best:
+        if self.monitor_op(current, self.best):
             self.best = current
             self.wait = 0
         else:
             if self.wait >= self.patience:
                 if self.verbose > 0:
-                    print("Epoch %05d: early stopping" % (epoch))
+                    print('Epoch %05d: early stopping' % (epoch))
                 self.model.stop_training = True
             self.wait += 1
 
 
 class RemoteMonitor(Callback):
+    '''Callback used to stream events to a server.
+
+    Requires the `requests` library.
+
+    # Arguments
+        root: root url to which the events will be sent (at the end
+            of every epoch). Events are sent to
+            `root + '/publish/epoch/end/'`. Calls are HTTP POST,
+            with a `data` argument which is a JSON-encoded dictionary
+            of event data.
+    '''
     def __init__(self, root='http://localhost:9000'):
         self.root = root
 
@@ -277,19 +397,128 @@ class RemoteMonitor(Callback):
             send[k] = v
 
         try:
-            r = requests.post(self.root + '/publish/epoch/end/', {'data': json.dumps(send)})
+            requests.post(self.root + '/publish/epoch/end/',
+                          {'data': json.dumps(send)})
         except:
-            print('Warning: could not reach RemoteMonitor root server at ' + str(self.root))
+            print('Warning: could not reach RemoteMonitor '
+                  'root server at ' + str(self.root))
 
 
 class LearningRateScheduler(Callback):
-    '''LearningRateScheduler
-    schedule is a function that gets an epoch number as input and returns a new
-    learning rate as output.
+    '''Learning rate scheduler.
+
+    # Arguments
+        schedule: a function that takes an epoch index as input
+            (integer, indexed from 0) and returns a new
+            learning rate as output (float).
     '''
     def __init__(self, schedule):
         super(LearningRateScheduler, self).__init__()
         self.schedule = schedule
 
     def on_epoch_begin(self, epoch, logs={}):
-        self.model.optimizer.lr.set_value(self.schedule(epoch))
+        assert hasattr(self.model.optimizer, 'lr'), \
+            'Optimizer must have a "lr" attribute.'
+        lr = self.schedule(epoch)
+        assert type(lr) == float, 'The output of the "schedule" function should be float.'
+        K.set_value(self.model.optimizer.lr, lr)
+
+
+class TensorBoard(Callback):
+    ''' Tensorboard basic visualizations.
+
+    This callback writes a log for TensorBoard, which allows
+    you to visualize dynamic graphs of your training and test
+    metrics, as well as activation histograms for the different
+    layers in your model.
+
+    TensorBoard is a visualization tool provided with TensorFlow.
+
+    If you have installed TensorFlow with pip, you should be able
+    to launch TensorBoard from the command line:
+    ```
+    tensorboard --logdir=/full_path_to_your_logs
+    ```
+    You can find more information about TensorBoard
+    [here](https://www.tensorflow.org/versions/master/how_tos/summaries_and_tensorboard/index.html).
+
+    # Arguments
+        log_dir: the path of the directory where to save the log
+            files to be parsed by tensorboard
+        histogram_freq: frequency (in epochs) at which to compute activation
+            histograms for the layers of the model. If set to 0,
+            histograms won't be computed.
+    '''
+    def __init__(self, log_dir='./logs', histogram_freq=0):
+        super(Callback, self).__init__()
+        if K._BACKEND != 'tensorflow':
+            raise Exception('TensorBoard callback only works '
+                            'with the TensorFlow backend.')
+        self.log_dir = log_dir
+        self.histogram_freq = histogram_freq
+
+    def _set_model(self, model):
+        import tensorflow as tf
+        import keras.backend.tensorflow_backend as KTF
+
+        self.model = model
+        self.sess = KTF._get_session()
+        if self.histogram_freq:
+            mod_type = self.model.get_config()['name']
+            if mod_type == 'Sequential':
+                layers = {l.get_config()['name']: l for l in self.model.layers}
+            elif mod_type == 'Graph':
+                layers = self.model.nodes
+            else:
+                raise Exception('Unrecognized model:',
+                                self.model.get_config()['name'])
+            for l in layers:
+                cur_layer = layers[l]
+                if hasattr(cur_layer, 'W'):
+                    tf.histogram_summary('{}_W'.format(l), cur_layer.W)
+                if hasattr(cur_layer, 'b'):
+                    tf.histogram_summary('{}_b'.format(l), cur_layer.b)
+                if hasattr(cur_layer, 'get_output'):
+                    tf.histogram_summary('{}_out'.format(l),
+                                         cur_layer.get_output())
+        self.merged = tf.merge_all_summaries()
+        self.writer = tf.train.SummaryWriter(self.log_dir,
+                                             self.sess.graph_def)
+
+    def on_epoch_begin(self, epoch, logs={}):
+        self.seen = 0
+        self.totals = {}
+
+    def on_batch_end(self, batch, logs={}):
+        batch_size = logs.get('size', 0)
+        self.seen += batch_size
+        for k, v in logs.items():
+            if k in self.totals:
+                self.totals[k] += v * batch_size
+            else:
+                self.totals[k] = v * batch_size
+
+    def on_epoch_end(self, epoch, logs={}):
+        import tensorflow as tf
+
+        if self.model.validation_data and self.histogram_freq:
+            if epoch % self.histogram_freq == 0:
+                if self.params.get('show_accuracy'):
+                    test_function = self.model._test_with_acc
+                else:
+                    test_function = self.model._test
+                names = [v.name for v in test_function.inputs]
+                feed_dict = dict(zip(names, self.model.validation_data))
+                result = self.sess.run([self.merged], feed_dict=feed_dict)
+                summary_str = result[0]
+                self.writer.add_summary(summary_str, epoch)
+
+        for name, value in self.totals.items() + logs.items():
+            if name in ['batch', 'size']:
+                continue
+            summary = tf.Summary()
+            summary_value = summary.value.add()
+            summary_value.simple_value = value
+            summary_value.tag = name
+            self.writer.add_summary(summary, epoch)
+        self.writer.flush()

keras/datasets/cifar10.py

@@ -10,7 +10,6 @@ def load_data():
     origin = "http://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz"
     path = get_file(dirname, origin=origin, untar=True)
 
-    nb_test_samples = 10000
     nb_train_samples = 50000
 
     X_train = np.zeros((nb_train_samples, 3, 32, 32), dtype="uint8")

keras/datasets/data_utils.py

@@ -14,7 +14,10 @@ class ParanoidURLopener(FancyURLopener):
 
 
 def get_file(fname, origin, untar=False):
-    datadir = os.path.expanduser(os.path.join('~', '.keras', 'datasets'))
+    datadir_base = os.path.expanduser(os.path.join('~', '.keras'))
+    if not os.access(datadir_base, os.W_OK):
+        datadir_base = os.path.join('/tmp', '.keras')
+    datadir = os.path.join(datadir_base, 'datasets')
     if not os.path.exists(datadir):
         os.makedirs(datadir)
 

keras/datasets/imdb.py

@@ -2,12 +2,12 @@ from __future__ import absolute_import
 from six.moves import cPickle
 import gzip
 from .data_utils import get_file
-import random
 from six.moves import zip
 import numpy as np
 
 
-def load_data(path="imdb.pkl", nb_words=None, skip_top=0, maxlen=None, test_split=0.2, seed=113,
+def load_data(path="imdb.pkl", nb_words=None, skip_top=0,
+              maxlen=None, test_split=0.2, seed=113,
               start_char=1, oov_char=2, index_from=3):
 
     path = get_file(path, origin="https://s3.amazonaws.com/text-datasets/imdb.pkl")
@@ -39,7 +39,10 @@ def load_data(path="imdb.pkl", nb_words=None, skip_top=0, maxlen=None, test_spli
                 new_labels.append(y)
         X = new_X
         labels = new_labels
-
+    if not X:
+        raise Exception('After filtering for sequences shorter than maxlen=' +
+                        str(maxlen) + ', no sequence was kept. '
+                        'Increase maxlen.')
     if not nb_words:
         nb_words = max([max(x) for x in X])
 
@@ -57,10 +60,10 @@ def load_data(path="imdb.pkl", nb_words=None, skip_top=0, maxlen=None, test_spli
             nX.append(nx)
         X = nX
 
-    X_train = X[:int(len(X)*(1-test_split))]
-    y_train = labels[:int(len(X)*(1-test_split))]
+    X_train = X[:int(len(X) * (1 - test_split))]
+    y_train = labels[:int(len(X) * (1 - test_split))]
 
-    X_test = X[int(len(X)*(1-test_split)):]
-    y_test = labels[int(len(X)*(1-test_split)):]
+    X_test = X[int(len(X) * (1 - test_split)):]
+    y_test = labels[int(len(X) * (1 - test_split)):]
 
     return (X_train, y_train), (X_test, y_test)

keras/datasets/mnist.py

@@ -19,5 +19,4 @@ def load_data(path="mnist.pkl.gz"):
         data = cPickle.load(f, encoding="bytes")
 
     f.close()
-
     return data  # (X_train, y_train), (X_test, y_test)

keras/datasets/reuters.py

@@ -1,93 +1,17 @@
 # -*- coding: utf-8 -*-
 from __future__ import absolute_import
-from __future__ import print_function
 from .data_utils import get_file
-import string
-import random
-import os
 from six.moves import cPickle
 from six.moves import zip
 import numpy as np
 
 
-def make_reuters_dataset(path=os.path.join('datasets', 'temp', 'reuters21578'), min_samples_per_topic=15):
-    import re
-    from ..preprocessing.text import Tokenizer
-
-    wire_topics = []
-    topic_counts = {}
-    wire_bodies = []
-
-    for fname in os.listdir(path):
-        if 'sgm' in fname:
-            s = open(os.path.join(path, fname)).read()
-            tag = '<TOPICS>'
-            while tag in s:
-                s = s[s.find(tag)+len(tag):]
-                topics = s[:s.find('</')]
-                if topics and '</D><D>' not in topics:
-                    topic = topics.replace('<D>', '').replace('</D>', '')
-                    wire_topics.append(topic)
-                    topic_counts[topic] = topic_counts.get(topic, 0) + 1
-                else:
-                    continue
-
-                bodytag = '<BODY>'
-                body = s[s.find(bodytag)+len(bodytag):]
-                body = body[:body.find('</')]
-                wire_bodies.append(body)
-
-    # only keep most common topics
-    items = list(topic_counts.items())
-    items.sort(key=lambda x: x[1])
-    kept_topics = set()
-    for x in items:
-        print(x[0] + ': ' + str(x[1]))
-        if x[1] >= min_samples_per_topic:
-            kept_topics.add(x[0])
-    print('-')
-    print('Kept topics:', len(kept_topics))
-
-    # filter wires with rare topics
-    kept_wires = []
-    labels = []
-    topic_indexes = {}
-    for t, b in zip(wire_topics, wire_bodies):
-        if t in kept_topics:
-            if t not in topic_indexes:
-                topic_index = len(topic_indexes)
-                topic_indexes[t] = topic_index
-            else:
-                topic_index = topic_indexes[t]
-
-            labels.append(topic_index)
-            kept_wires.append(b)
-
-    # vectorize wires
-    tokenizer = Tokenizer()
-    tokenizer.fit_on_texts(kept_wires)
-    X = tokenizer.texts_to_sequences(kept_wires)
-
-    print('Sanity check:')
-    for w in ["banana", "oil", "chocolate", "the", "dsft"]:
-        print('...index of', w, ':', tokenizer.word_index.get(w))
-    print('text reconstruction:')
-    reverse_word_index = dict([(v, k) for k, v in tokenizer.word_index.items()])
-    print(' '.join(reverse_word_index[i] for i in X[10]))
-
-    dataset = (X, labels)
-    print('-')
-    print('Saving...')
-    cPickle.dump(dataset, open(os.path.join('datasets', 'data', 'reuters.pkl'), 'w'))
-    cPickle.dump(tokenizer.word_index, open(os.path.join('datasets', 'data', 'reuters_word_index.pkl'), 'w'))
-
-
-def load_data(path="reuters.pkl", nb_words=None, skip_top=0, maxlen=None, test_split=0.2, seed=113,
+def load_data(path="reuters.pkl", nb_words=None, skip_top=0,
+              maxlen=None, test_split=0.2, seed=113,
               start_char=1, oov_char=2, index_from=3):
 
     path = get_file(path, origin="https://s3.amazonaws.com/text-datasets/reuters.pkl")
     f = open(path, 'rb')
-
     X, labels = cPickle.load(f)
     f.close()
 
@@ -128,11 +52,11 @@ def load_data(path="reuters.pkl", nb_words=None, skip_top=0, maxlen=None, test_s
             nX.append(nx)
         X = nX
 
-    X_train = X[:int(len(X)*(1-test_split))]
-    y_train = labels[:int(len(X)*(1-test_split))]
+    X_train = X[:int(len(X) * (1 - test_split))]
+    y_train = labels[:int(len(X) * (1 - test_split))]
 
-    X_test = X[int(len(X)*(1-test_split)):]
-    y_test = labels[int(len(X)*(1-test_split)):]
+    X_test = X[int(len(X) * (1 - test_split)):]
+    y_test = labels[int(len(X) * (1 - test_split)):]
 
     return (X_train, y_train), (X_test, y_test)
 
@@ -141,8 +65,3 @@ def get_word_index(path="reuters_word_index.pkl"):
     path = get_file(path, origin="https://s3.amazonaws.com/text-datasets/reuters_word_index.pkl")
     f = open(path, 'rb')
     return cPickle.load(f)
-
-
-if __name__ == "__main__":
-    make_reuters_dataset()
-    (X_train, y_train), (X_test, y_test) = load_data()

keras/initializations.py

@@ -14,7 +14,7 @@ def uniform(shape, scale=0.05):
 
 
 def normal(shape, scale=0.05):
-    return K.variable(np.random.randn(*shape) * scale)
+    return K.variable(np.random.normal(loc=0.0, scale=scale, size=shape))
 
 
 def lecun_uniform(shape):
@@ -68,7 +68,8 @@ def orthogonal(shape, scale=1.1):
 
 def identity(shape, scale=1):
     if len(shape) != 2 or shape[0] != shape[1]:
-        raise Exception("Identity matrix initialization can only be used for 2D square matrices")
+        raise Exception('Identity matrix initialization can only be used '
+                        'for 2D square matrices.')
     else:
         return K.variable(scale * np.identity(shape[0]))
 

keras/layers/advanced_activations.py

@@ -1,10 +1,25 @@
 from .. import initializations
-from ..layers.core import Layer, MaskedLayer
+from ..layers.core import MaskedLayer
 from .. import backend as K
 import numpy as np
 
 
 class LeakyReLU(MaskedLayer):
+    '''Special version of a Rectified Linear Unit
+    that allows a small gradient when the unit is not active
+    (`f(x) = alpha*x for x < 0`).
+
+    # Input shape
+        Arbitrary. Use the keyword argument `input_shape`
+        (tuple of integers, does not include the samples axis)
+        when using this layer as the first layer in a model.
+
+    # Output shape
+        Same shape as the input.
+
+    # Arguments
+        alpha: float >= 0. Negative slope coefficient.
+    '''
     def __init__(self, alpha=0.3, **kwargs):
         super(LeakyReLU, self).__init__(**kwargs)
         self.alpha = alpha
@@ -22,10 +37,20 @@ class LeakyReLU(MaskedLayer):
 
 class PReLU(MaskedLayer):
     '''
-        Reference:
-            Delving Deep into Rectifiers: Surpassing Human-Level
-            Performance on ImageNet Classification
-                http://arxiv.org/pdf/1502.01852v1.pdf
+    # Input shape
+        Arbitrary. Use the keyword argument `input_shape`
+        (tuple of integers, does not include the samples axis)
+        when using this layer as the first layer in a model.
+
+    # Output shape
+        Same shape as the input.
+
+    # Arguments:
+        init: initialization function for the weights.
+        weights: initial weights, as a list of a single numpy array.
+
+    # References:
+        - [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](http://arxiv.org/pdf/1502.01852v1.pdf)
     '''
     def __init__(self, init='zero', weights=None, **kwargs):
         self.init = initializations.get(init)
@@ -55,6 +80,21 @@ class PReLU(MaskedLayer):
 
 
 class ELU(MaskedLayer):
+    '''
+    # Input shape
+        Arbitrary. Use the keyword argument `input_shape`
+        (tuple of integers, does not include the samples axis)
+        when using this layer as the first layer in a model.
+
+    # Output shape
+        Same shape as the input.
+
+    # Arguments
+        alpha: scale for the negative factor.
+
+    # References
+        - [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)](http://arxiv.org/pdf/1511.07289v1.pdf)
+    '''
     def __init__(self, alpha=1.0, **kwargs):
         super(ELU, self).__init__(**kwargs)
         self.alpha = alpha
@@ -73,13 +113,23 @@ class ELU(MaskedLayer):
 
 
 class ParametricSoftplus(MaskedLayer):
-    '''
-        Parametric Softplus of the form: alpha * log(1 + exp(beta * X))
+    '''Parametric Softplus of the form: alpha * log(1 + exp(beta * X))
 
-        Reference:
-            Inferring Nonlinear Neuronal Computation
-            Based on Physiologically Plausible Inputs
-            http://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1003143
+    # Input shape
+        Arbitrary. Use the keyword argument `input_shape`
+        (tuple of integers, does not include the samples axis)
+        when using this layer as the first layer in a model.
+
+    # Output shape
+        Same shape as the input.
+
+    # Arguments
+        alpha_init: float. Initial value of the alpha weights.
+        beta_init: float. Initial values of the beta weights.
+        weights: initial weights, as a list of 2 numpy arrays.
+
+    # References:
+        - [Inferring Nonlinear Neuronal Computation Based on Physiologically Plausible Inputs](http://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1003143)
     '''
     def __init__(self, alpha_init=0.2, beta_init=5.0,
                  weights=None, **kwargs):
@@ -111,12 +161,21 @@ class ParametricSoftplus(MaskedLayer):
 
 
 class ThresholdedLinear(MaskedLayer):
-    '''
-        Thresholded Linear Activation
+    '''Thresholded Linear Activation.
 
-        Reference:
-            Zero-Bias Autoencoders and the Benefits of Co-Adapting Features
-            http://arxiv.org/pdf/1402.3337.pdf
+    # Input shape
+        Arbitrary. Use the keyword argument `input_shape`
+        (tuple of integers, does not include the samples axis)
+        when using this layer as the first layer in a model.
+
+    # Output shape
+        Same shape as the input.
+
+    # Arguments
+        theta: float >= 0. Threshold location of activation.
+
+    # References
+        [Zero-Bias Autoencoders and the Benefits of Co-Adapting Features](http://arxiv.org/pdf/1402.3337.pdf)
     '''
     def __init__(self, theta=1.0, **kwargs):
         super(ThresholdedLinear, self).__init__(**kwargs)
@@ -134,12 +193,21 @@ class ThresholdedLinear(MaskedLayer):
 
 
 class ThresholdedReLU(MaskedLayer):
-    '''
-        Thresholded Rectified Activation
+    '''Thresholded Rectified Activation.
 
-        Reference:
-            Zero-Bias Autoencoders and the Benefits of Co-Adapting Features
-            http://arxiv.org/pdf/1402.3337.pdf
+    # Input shape
+        Arbitrary. Use the keyword argument `input_shape`
+        (tuple of integers, does not include the samples axis)
+        when using this layer as the first layer in a model.
+
+    # Output shape
+        Same shape as the input.
+
+    # Arguments
+        theta: float >= 0. Threshold location of activation.
+
+    # References
+        [Zero-Bias Autoencoders and the Benefits of Co-Adapting Features](http://arxiv.org/pdf/1402.3337.pdf)
     '''
     def __init__(self, theta=1.0, **kwargs):
         super(ThresholdedReLU, self).__init__(**kwargs)

keras/layers/containers.py

@@ -9,24 +9,60 @@ from six.moves import range
 
 
 class Sequential(Layer):
-    '''
-        Simple linear stack of layers.
+    '''The Sequential container is a linear stack of layers.
+    Apart from the `add` methods and the `layers` constructor argument,
+    the API is identical to that of the `Layer` class.
 
-        inherited from Layer:
-        - get_params
-        - get_output_mask
-        - supports_masked_input
-    '''
+    This class is also the basis for the `keras.models.Sequential` model.
 
+    # Arguments
+        layers: list of layers to be added to the container.
+    '''
     def __init__(self, layers=[]):
         self.layers = []
+        self.layer_cache = {}
         for layer in layers:
             self.add(layer)
+        self._cache_enabled = True
+
+    def __call__(self, X, mask=None, train=False):
+        # turn off layer cache temporarily
+        tmp_cache_enabled = self.cache_enabled
+        self.cache_enabled = False
+        # recursively search for a layer which is not a Sequential model
+        layer = self
+        while issubclass(layer.__class__, Sequential):
+            layer = layer.layers[0]
+        # set temporary input to first layer
+        tmp_input = layer.get_input
+        tmp_mask = None
+        layer.get_input = lambda _: X
+        if hasattr(layer, 'get_input_mask'):
+            tmp_mask = layer.get_input_mask
+            layer.get_input_mask = lambda _: mask
+        Y = self.get_output(train=train)
+        # return input from first layer to what it was
+        layer.get_input = tmp_input
+        if hasattr(layer, 'get_input_mask'):
+            layer.get_input_mask = tmp_mask
+        self.cache_enabled = tmp_cache_enabled
+        return Y
+
+    @property
+    def cache_enabled(self):
+        return self._cache_enabled
+
+    @cache_enabled.setter
+    def cache_enabled(self, value):
+        self._cache_enabled = value
+        for l in self.layers:
+            l.cache_enabled = value
 
     def set_previous(self, layer):
         self.layers[0].previous = layer
 
     def add(self, layer):
+        layer.layer_cache = self.layer_cache
         self.layers.append(layer)
         if len(self.layers) > 1:
             self.layers[-1].set_previous(self.layers[-2])
@@ -65,6 +101,25 @@ class Sequential(Layer):
                 updates += l.get_params()[3]
         return updates
 
+    @property
+    def state_updates(self):
+        """
+        Return the `updates` from all layers in the sequence that are
+        stateful.  This is useful for separating _training_ updates and
+        _prediction_ updates for when we need to update a layers internal state
+        during a stateful prediction.
+        """
+        state_updates = []
+        for l in self.layers:
+            if getattr(l, 'stateful', False):
+                state_updates += l.get_params()[3]
+        return state_updates
+
+    def reset_states(self):
+        for l in self.layers:
+            if hasattr(l, 'reset_states') and getattr(l, 'stateful', False):
+                l.reset_states()
+
     @property
     def output_shape(self):
         return self.layers[-1].output_shape
@@ -75,7 +130,7 @@ class Sequential(Layer):
     def set_input(self):
         for l in self.layers:
             if hasattr(l, 'input'):
-                ndim = len(K.get_shape(l.input))
+                ndim = K.ndim(l.input)
                 self.layers[0].input = K.placeholder(ndim=ndim)
                 break
 
@@ -105,27 +160,22 @@ class Sequential(Layer):
             weights = weights[nb_param:]
 
     def get_config(self):
-        return {"name": self.__class__.__name__,
-                "layers": [layer.get_config() for layer in self.layers]}
+        return {'name': self.__class__.__name__,
+                'layers': [layer.get_config() for layer in self.layers]}
 
     def count_params(self):
         return sum([layer.count_params() for layer in self.layers])
 
 
 class Graph(Layer):
-    '''
-        Implement a NN graph with arbitrary layer connections,
-        arbitrary number of inputs and arbitrary number of outputs.
+    '''Implement a NN graph with arbitrary layer connections,
+    arbitrary number of inputs and arbitrary number of outputs.
 
-        Note: Graph can only be used as a layer
-        (connect, input, get_input, get_output)
-        when it has exactly one input and one output.
+    This class is also the basis for the `keras.models.Graph` model.
 
-        inherited from Layer:
-            - get_output_mask
-            - supports_masked_input
-            - get_weights
-            - set_weights
+    Note: `Graph` can only be used as a layer
+    (connect, input, get_input, get_output)
+    when it has exactly one input and one output.
     '''
     def __init__(self):
         self.namespace = set()  # strings
@@ -137,6 +187,7 @@ class Graph(Layer):
         self.input_config = []  # dicts
         self.output_config = []  # dicts
         self.node_config = []  # dicts
+        self.layer_cache = {}
 
     @property
     def nb_input(self):
@@ -178,14 +229,35 @@ class Graph(Layer):
                 updates += l.get_params()[3]
         return updates
 
+    @property
+    def state_updates(self):
+        """
+        Return the `updates` from all nodes in that graph for nodes that are
+        stateful.  This is useful for separating _training_ updates and
+        _prediction_ updates for when we need to update a layers internal state
+        during a stateful prediction.
+        """
+        state_updates = []
+        for l in self.nodes.values():
+            if getattr(l, 'stateful', False):
+                state_updates += l.get_params()[3]
+        return state_updates
+
+    def reset_states(self):
+        for l in self.nodes.values():
+            if hasattr(l, 'reset_states') and getattr(l, 'stateful', False):
+                l.reset_states()
+
     def set_previous(self, layer, connection_map={}):
         if self.nb_input != layer.nb_output:
-            raise Exception('Cannot connect layers: input count does not match output count.')
+            raise Exception('Cannot connect layers: '
+                            'input count does not match output count.')
         if self.nb_input == 1:
             self.inputs[self.input_order[0]].set_previous(layer)
         else:
             if not connection_map:
-                raise Exception('Cannot attach multi-input layer: no connection_map provided.')
+                raise Exception('Cannot attach multi-input layer: '
+                                'no connection_map provided.')
             for k, v in connection_map.items():
                 if k in self.inputs and v in layer.outputs:
                     self.inputs[k].set_previous(layer.outputs[v])
@@ -217,17 +289,31 @@ class Graph(Layer):
         else:
             return dict([(k, v.get_output(train)) for k, v in self.outputs.items()])
 
-    def add_input(self, name, input_shape, dtype='float'):
+    def add_input(self, name, input_shape=None,
+                  batch_input_shape=None, dtype='float'):
+        '''Add an input to the graph.
+
+        # Arguments:
+            name: string. The name of the new input. Must be unique in the graph.
+            input_shape: a tuple of integers, the expected shape of the input samples.
+                Does not include the batch size.
+            batch_input_shape: a tuple of integers, the expected shape of the
+                whole input batch, including the batch size.
+            dtype: 'float' or 'int'.
+        '''
         if name in self.namespace:
             raise Exception('Duplicate node identifier: ' + name)
         self.namespace.add(name)
         self.input_order.append(name)
         layer = Layer()  # empty layer
-        layer.set_input_shape(input_shape)
+        if input_shape:
+            layer.set_input_shape((None,) + tuple(input_shape))
+        elif batch_input_shape:
+            layer.set_input_shape(batch_input_shape)
         if dtype == 'float':
             layer.input = K.placeholder(shape=layer.input_shape, name=name)
         else:
-            if len(input_shape) == 1:
+            if (input_shape and len(input_shape) == 1) or (batch_input_shape and len(batch_input_shape) == 2):
                 layer.input = K.placeholder(shape=layer.input_shape,
                                             dtype='int32',
                                             name=name)
@@ -241,6 +327,25 @@ class Graph(Layer):
     def add_node(self, layer, name, input=None, inputs=[],
                  merge_mode='concat', concat_axis=-1, dot_axes=-1,
                  create_output=False):
+        '''Add a node in the graph. It can be connected to multiple
+        inputs, which will first be merged into one tensor
+        according to the mode specified.
+
+        # Arguments
+            layer: the layer at the node.
+            name: name for the node.
+            input: when connecting the layer to a single input,
+                this is the name of the incoming node.
+            inputs: when connecting the layer to multiple inputs,
+                this is a list of names of incoming nodes.
+            merge_mode: one of {concat, sum, dot, ave, mul}
+            concat_axis: when `merge_mode=='concat'`, this is the
+                input concatenation axis.
+            dot_axes: when `merge_mode='dot'`, this is the contraction axes
+                specification; see the `Merge layer for details.
+            create_output: boolean. Set this to `True` if you want the output
+                of your node to be an output of the graph.
+        '''
         if name in self.namespace:
             raise Exception('Duplicate node identifier: ' + name)
         if input:
@@ -264,6 +369,7 @@ class Graph(Layer):
             layer.set_previous(merge)
 
         self.namespace.add(name)
+        layer.layer_cache = self.layer_cache
         self.nodes[name] = layer
         self.node_config.append({'name': name,
                                  'input': input,
@@ -279,20 +385,21 @@ class Graph(Layer):
     def add_shared_node(self, layer, name, inputs=[], merge_mode=None,
                         concat_axis=-1, dot_axes=-1, outputs=[],
                         create_output=False):
-        '''
-        Used to shared / multi input-multi output node
+        '''Used to share a same layer across multiple nodes.
 
-        Arguments
-        ------------
-        layer - The layer to be shared across multiple inputs
-        name - Name of the shared layer
-        inputs - List of names of input nodes
-        merge_mode - Similar to merge_mode argument of add_node()
-        concat_axis - Similar to concat_axis argument of add_node()
-        dot_axes - Similar to dot_axes argument of add_node()
-        outputs - Names for output nodes. Used when merge_mode = None
-        create_output -  Similar to create_output argument of add_node().
-            Output will be created only if merge_mode is given
+        Supposed, for instance, that you want to apply one same `Dense`
+        layer after to the output of two different nodes.
+        You can then add the `Dense` layer as a shared node.
+
+        # Arguments
+            layer: The layer to be shared across multiple inputs
+            name: Name of the shared node
+            inputs: List of names of input nodes
+            merge_mode: Same meaning as `merge_mode` argument of `add_node()`
+            concat_axis: Same meaning as `concat_axis` argument of `add_node()`
+            dot_axes: Same meaning as `dot_axes` argument of `add_node()`
+            outputs: Used when `merge_mode=None`. Names for the output nodes.
+            create_output: Same meaning as `create_output` argument of `add_node()`.
         '''
         if name in self.namespace:
             raise Exception('Duplicate node identifier: ' + name)
@@ -301,7 +408,7 @@ class Graph(Layer):
                 raise Exception('Duplicate node identifier: ' + o)
         if merge_mode:
             if merge_mode not in {'sum', 'ave', 'mul', 'dot', 'cos', 'concat', 'join'}:
-                raise Eception("Invalid merge mode")
+                raise Exception('Invalid merge mode')
         layers = []
         for i in range(len(inputs)):
             input = inputs[i]
@@ -322,8 +429,10 @@ class Graph(Layer):
                 layers.append(n)
             else:
                 raise Exception('Unknown identifier: ' + input)
-        s = Siamese(layer, layers, merge_mode, concat_axis=concat_axis, dot_axes=dot_axes)
-        s.set_name(name)
+        s = Siamese(layer, layers, merge_mode,
+                    concat_axis=concat_axis,
+                    dot_axes=dot_axes,
+                    is_graph=True)
         self.namespace.add(name)
         self.nodes[name] = s
         self.node_config.append({'name': name,
@@ -337,22 +446,37 @@ class Graph(Layer):
                 sh = SiameseHead(i)
                 sh.previous = s
                 sh_name = outputs[i]
-                sh.set_name(sh_name)
                 self.namespace.add(sh_name)
                 self.nodes[sh_name] = sh
                 self.node_config.append({'name': sh_name,
-                                         'inputs': [s],
+                                         'inputs': [name],
                                          'create_output': create_output})
                 if create_output:
                     self.add_output(sh_name, input=sh_name)
 
         if create_output and merge_mode:
             if merge_mode == 'join':
-                raise Exception("Output can not be of type OrderedDict")
+                raise Exception('Output can not be of type OrderedDict')
             self.add_output(name, input=name)
 
     def add_output(self, name, input=None, inputs=[],
                    merge_mode='concat', concat_axis=-1, dot_axes=-1):
+        '''Add an output to the graph.
+
+        This output can merge several node outputs into a single output.
+
+        # Arguments
+            name: name of the output.
+            input: when connecting the layer to a single input,
+                this is the name of the incoming node.
+            inputs: when connecting the layer to multiple inputs,
+                this is a list of names of incoming nodes.
+            merge_mode: one of {concat, sum, dot, ave, mul}
+            concat_axis: when `merge_mode=='concat'`, this is the
+                input concatenation axis.
+            dot_axes: when `merge_mode='dot'`, this is the contraction axes
+                specification; see the `Merge layer for details.
+        '''
         if name in self.output_order:
             raise Exception('Duplicate output identifier: ' + name)
         if input:
@@ -381,13 +505,13 @@ class Graph(Layer):
                                    'dot_axes': dot_axes})
 
     def get_config(self):
-        return {"name": self.__class__.__name__,
-                "input_config": self.input_config,
-                "node_config": self.node_config,
-                "output_config": self.output_config,
-                "input_order": self.input_order,
-                "output_order": self.output_order,
-                "nodes": dict([(c["name"], self.nodes[c["name"]].get_config()) for c in self.node_config])}
+        return {'name': self.__class__.__name__,
+                'input_config': self.input_config,
+                'node_config': self.node_config,
+                'output_config': self.output_config,
+                'input_order': self.input_order,
+                'output_order': self.output_order,
+                'nodes': dict([(c['name'], self.nodes[c['name']].get_config()) for c in self.node_config])}
 
     def count_params(self):
         return sum([layer.count_params() for layer in self.nodes.values()])

keras/layers/convolutional.py

@@ -18,6 +18,54 @@ def conv_output_length(input_length, filter_size, border_mode, stride):
 
 
 class Convolution1D(Layer):
+    '''Convolution operator for filtering neighborhoods of one-dimensional inputs.
+    When using this layer as the first layer in a model,
+    either provide the keyword argument `input_dim`
+    (int, e.g. 128 for sequences of 128-dimensional vectors),
+    or `input_shape` (tuple of integers, e.g. (10, 128) for sequences
+    of 10 vectors of 128-dimensional vectors).
+
+    # Input shape
+        3D tensor with shape: `(samples, steps, input_dim)`.
+
+    # Output shape
+        3D tensor with shape: `(samples, new_steps, nb_filter)`.
+        `steps` value might have changed due to padding.
+
+    # Arguments
+        nb_filter: Number of convolution kernels to use
+            (dimensionality of the output).
+        filter_length: The extension (spatial or temporal) of each filter.
+        init: name of initialization function for the weights of the layer
+            (see [initializations](../initializations.md)),
+            or alternatively, Theano function to use for weights initialization.
+            This parameter is only relevant if you don't pass a `weights` argument.
+        activation: name of activation function to use
+            (see [activations](../activations.md)),
+            or alternatively, elementwise Theano function.
+            If you don't specify anything, no activation is applied
+            (ie. "linear" activation: a(x) = x).
+        weights: list of numpy arrays to set as initial weights.
+        border_mode: 'valid' or 'same'.
+        subsample_length: factor by which to subsample output.
+        W_regularizer: instance of [WeightRegularizer](../regularizers.md)
+            (eg. L1 or L2 regularization), applied to the main weights matrix.
+        b_regularizer: instance of [WeightRegularizer](../regularizers.md),
+            applied to the bias.
+        activity_regularizer: instance of [ActivityRegularizer](../regularizers.md),
+            applied to the network output.
+        W_constraint: instance of the [constraints](../constraints.md) module
+            (eg. maxnorm, nonneg), applied to the main weights matrix.
+        b_constraint: instance of the [constraints](../constraints.md) module,
+            applied to the bias.
+        input_dim: Number of channels/dimensions in the input.
+            Either this argument or the keyword argument `input_shape`must be
+            provided when using this layer as the first layer in a model.
+        input_length: Length of input sequences, when it is constant.
+            This argument is required if you are going to connect
+            `Flatten` then `Dense` layers upstream
+            (without it, the shape of the dense outputs cannot be computed).
+    '''
     input_ndim = 3
 
     def __init__(self, nb_filter, filter_length,
@@ -93,7 +141,8 @@ class Convolution1D(Layer):
         X = K.expand_dims(X, -1)  # add a dimension of the right
         X = K.permute_dimensions(X, (0, 2, 1, 3))
         conv_out = K.conv2d(X, self.W, strides=self.subsample,
-                            border_mode=self.border_mode, dim_ordering='th')
+                            border_mode=self.border_mode,
+                            dim_ordering='th')
 
         output = conv_out + K.reshape(self.b, (1, self.nb_filter, 1, 1))
         output = self.activation(output)
@@ -102,25 +151,75 @@ class Convolution1D(Layer):
         return output
 
     def get_config(self):
-        config = {"name": self.__class__.__name__,
-                  "nb_filter": self.nb_filter,
-                  "filter_length": self.filter_length,
-                  "init": self.init.__name__,
-                  "activation": self.activation.__name__,
-                  "border_mode": self.border_mode,
-                  "subsample_length": self.subsample_length,
-                  "W_regularizer": self.W_regularizer.get_config() if self.W_regularizer else None,
-                  "b_regularizer": self.b_regularizer.get_config() if self.b_regularizer else None,
-                  "activity_regularizer": self.activity_regularizer.get_config() if self.activity_regularizer else None,
-                  "W_constraint": self.W_constraint.get_config() if self.W_constraint else None,
-                  "b_constraint": self.b_constraint.get_config() if self.b_constraint else None,
-                  "input_dim": self.input_dim,
-                  "input_length": self.input_length}
+        config = {'name': self.__class__.__name__,
+                  'nb_filter': self.nb_filter,
+                  'filter_length': self.filter_length,
+                  'init': self.init.__name__,
+                  'activation': self.activation.__name__,
+                  'border_mode': self.border_mode,
+                  'subsample_length': self.subsample_length,
+                  'W_regularizer': self.W_regularizer.get_config() if self.W_regularizer else None,
+                  'b_regularizer': self.b_regularizer.get_config() if self.b_regularizer else None,
+                  'activity_regularizer': self.activity_regularizer.get_config() if self.activity_regularizer else None,
+                  'W_constraint': self.W_constraint.get_config() if self.W_constraint else None,
+                  'b_constraint': self.b_constraint.get_config() if self.b_constraint else None,
+                  'input_dim': self.input_dim,
+                  'input_length': self.input_length}
         base_config = super(Convolution1D, self).get_config()
         return dict(list(base_config.items()) + list(config.items()))
 
 
 class Convolution2D(Layer):
+    '''Convolution operator for filtering windows of two-dimensional inputs.
+    When using this layer as the first layer in a model,
+    provide the keyword argument `input_shape`
+    (tuple of integers, does not include the sample axis),
+    e.g. `input_shape=(3, 128, 128)` for 128x128 RGB pictures.
+
+    # Input shape
+        4D tensor with shape:
+        `(samples, channels, rows, cols)` if dim_ordering='th'
+        or 4D tensor with shape:
+        `(samples, rows, cols, channels)` if dim_ordering='tf'.
+
+    # Output shape
+        4D tensor with shape:
+        `(samples, nb_filter, nb_row, nb_col)` if dim_ordering='th'
+        or 4D tensor with shape:
+        `(samples, nb_row, nb_col, nb_filter)` if dim_ordering='tf'.
+
+
+    # Arguments
+        nb_filter: Number of convolution filters to use.
+        nb_row: Number of rows in the convolution kernel.
+        nb_col: Number of columns in the convolution kernel.
+        init: name of initialization function for the weights of the layer
+            (see [initializations](../initializations.md)), or alternatively,
+            Theano function to use for weights initialization.
+            This parameter is only relevant if you don't pass
+            a `weights` argument.
+        activation: name of activation function to use
+            (see [activations](../activations.md)),
+            or alternatively, elementwise Theano function.
+            If you don't specify anything, no activation is applied
+            (ie. "linear" activation: a(x) = x).
+        weights: list of numpy arrays to set as initial weights.
+        border_mode: 'valid' or 'same'.
+        subsample: tuple of length 2. Factor by which to subsample output.
+            Also called strides elsewhere.
+        W_regularizer: instance of [WeightRegularizer](../regularizers.md)
+            (eg. L1 or L2 regularization), applied to the main weights matrix.
+        b_regularizer: instance of [WeightRegularizer](../regularizers.md),
+            applied to the bias.
+        activity_regularizer: instance of [ActivityRegularizer](../regularizers.md),
+            applied to the network output.
+        W_constraint: instance of the [constraints](../constraints.md) module
+            (eg. maxnorm, nonneg), applied to the main weights matrix.
+        b_constraint: instance of the [constraints](../constraints.md) module,
+            applied to the bias.
+        dim_ordering: 'th' or 'tf'. In 'th' mode, the channels dimension
+            (the depth) is at index 1, in 'tf' mode is it at index 3.
+    '''
     input_ndim = 4
 
     def __init__(self, nb_filter, nb_row, nb_col,
@@ -212,37 +311,45 @@ class Convolution2D(Layer):
         X = self.get_input(train)
         conv_out = K.conv2d(X, self.W, strides=self.subsample,
                             border_mode=self.border_mode,
-                            dim_ordering=self.dim_ordering)
-
-        output = conv_out + K.reshape(self.b, (1, self.nb_filter, 1, 1))
+                            dim_ordering=self.dim_ordering,
+                            image_shape=self.input_shape,
+                            filter_shape=self.W_shape)
+        if self.dim_ordering == 'th':
+            output = conv_out + K.reshape(self.b, (1, self.nb_filter, 1, 1))
+        elif self.dim_ordering == 'tf':
+            output = conv_out + K.reshape(self.b, (1, 1, 1, self.nb_filter))
+        else:
+            raise Exception('Invalid dim_ordering: ' + self.dim_ordering)
         output = self.activation(output)
         return output
 
     def get_config(self):
-        config = {"name": self.__class__.__name__,
-                  "nb_filter": self.nb_filter,
-                  "nb_row": self.nb_row,
-                  "nb_col": self.nb_col,
-                  "init": self.init.__name__,
-                  "activation": self.activation.__name__,
-                  "border_mode": self.border_mode,
-                  "subsample": self.subsample,
-                  "dim_ordering": self.dim_ordering,
-                  "W_regularizer": self.W_regularizer.get_config() if self.W_regularizer else None,
-                  "b_regularizer": self.b_regularizer.get_config() if self.b_regularizer else None,
-                  "activity_regularizer": self.activity_regularizer.get_config() if self.activity_regularizer else None,
-                  "W_constraint": self.W_constraint.get_config() if self.W_constraint else None,
-                  "b_constraint": self.b_constraint.get_config() if self.b_constraint else None}
+        config = {'name': self.__class__.__name__,
+                  'nb_filter': self.nb_filter,
+                  'nb_row': self.nb_row,
+                  'nb_col': self.nb_col,
+                  'init': self.init.__name__,
+                  'activation': self.activation.__name__,
+                  'border_mode': self.border_mode,
+                  'subsample': self.subsample,
+                  'dim_ordering': self.dim_ordering,
+                  'W_regularizer': self.W_regularizer.get_config() if self.W_regularizer else None,
+                  'b_regularizer': self.b_regularizer.get_config() if self.b_regularizer else None,
+                  'activity_regularizer': self.activity_regularizer.get_config() if self.activity_regularizer else None,
+                  'W_constraint': self.W_constraint.get_config() if self.W_constraint else None,
+                  'b_constraint': self.b_constraint.get_config() if self.b_constraint else None}
         base_config = super(Convolution2D, self).get_config()
         return dict(list(base_config.items()) + list(config.items()))
 
 
-class MaxPooling1D(Layer):
-    input_ndim = 3
+class _Pooling1D(Layer):
+    '''Abstract class for different pooling 1D layers.
+    '''
+    input_dim = 3
 
     def __init__(self, pool_length=2, stride=None,
                  border_mode='valid', **kwargs):
-        super(MaxPooling1D, self).__init__(**kwargs)
+        super(_Pooling1D, self).__init__(**kwargs)
         if stride is None:
             stride = pool_length
         self.pool_length = pool_length
@@ -260,31 +367,92 @@ class MaxPooling1D(Layer):
                                     self.border_mode, self.stride)
         return (input_shape[0], length, input_shape[2])
 
+    def _pooling_function(self, back_end, inputs, pool_size, strides,
+                          border_mode, dim_ordering):
+        raise NotImplementedError
+
     def get_output(self, train=False):
         X = self.get_input(train)
         X = K.expand_dims(X, -1)   # add dummy last dimension
         X = K.permute_dimensions(X, (0, 2, 1, 3))
-        output = K.maxpool2d(X, pool_size=self.pool_size, strides=self.st,
-                             border_mode=self.border_mode,
-                             dim_ordering='th')
+        output = self._pooling_function(inputs=X, pool_size=self.pool_size,
+                                        strides=self.st,
+                                        border_mode=self.border_mode,
+                                        dim_ordering='th')
         output = K.permute_dimensions(output, (0, 2, 1, 3))
         return K.squeeze(output, 3)  # remove dummy last dimension
 
     def get_config(self):
-        config = {"name": self.__class__.__name__,
-                  "stride": self.stride,
-                  "pool_length": self.pool_length,
-                  "border_mode": self.border_mode}
-        base_config = super(MaxPooling1D, self).get_config()
+        config = {'name': self.__class__.__name__,
+                  'stride': self.stride,
+                  'pool_length': self.pool_length,
+                  'border_mode': self.border_mode}
+        base_config = super(_Pooling1D, self).get_config()
         return dict(list(base_config.items()) + list(config.items()))
 
 
-class MaxPooling2D(Layer):
+class MaxPooling1D(_Pooling1D):
+    '''Max pooling operation for temporal data.
+
+    # Input shape
+        3D tensor with shape: `(samples, steps, features)`.
+
+    # Output shape
+        3D tensor with shape: `(samples, downsampled_steps, features)`.
+
+    # Arguments
+        pool_length: factor by which to downscale. 2 will halve the input.
+        stride: integer or None. Stride value.
+        border_mode: 'valid' or 'same'.
+            Note: 'same' will only work with TensorFlow for the time being.
+    '''
+    def __init__(self, pool_length=2, stride=None,
+                 border_mode='valid', **kwargs):
+        super(MaxPooling1D, self).__init__(pool_length, stride,
+                                           border_mode, **kwargs)
+
+    def _pooling_function(self, inputs, pool_size, strides,
+                          border_mode, dim_ordering):
+        output = K.pool2d(inputs, pool_size, strides,
+                          border_mode, dim_ordering, pool_mode='max')
+        return output
+
+
+class AveragePooling1D(_Pooling1D):
+    '''Average pooling for temporal data.
+
+        # Input shape
+        3D tensor with shape: `(samples, steps, features)`.
+
+    # Output shape
+        3D tensor with shape: `(samples, downsampled_steps, features)`.
+
+    # Arguments
+        pool_length: factor by which to downscale. 2 will halve the input.
+        stride: integer or None. Stride value.
+        border_mode: 'valid' or 'same'.
+            Note: 'same' will only work with TensorFlow for the time being.
+    '''
+    def __init__(self, pool_length=2, stride=None,
+                 border_mode='valid', **kwargs):
+        super(AveragePooling1D, self).__init__(pool_length, stride,
+                                               border_mode, **kwargs)
+
+    def _pooling_function(self, inputs, pool_size, strides,
+                          border_mode, dim_ordering):
+        output = K.pool2d(inputs, pool_size, strides,
+                          border_mode, dim_ordering, pool_mode='avg')
+        return output
+
+
+class _Pooling2D(Layer):
+    '''Abstract class for different pooling 2D layers.
+    '''
     input_ndim = 4
 
     def __init__(self, pool_size=(2, 2), strides=None, border_mode='valid',
                  dim_ordering='th', **kwargs):
-        super(MaxPooling2D, self).__init__(**kwargs)
+        super(_Pooling2D, self).__init__(**kwargs)
         self.input = K.placeholder(ndim=4)
         self.pool_size = tuple(pool_size)
         if strides is None:
@@ -319,25 +487,114 @@ class MaxPooling2D(Layer):
         else:
             raise Exception('Invalid dim_ordering: ' + self.dim_ordering)
 
+    def _pooling_function(self, inputs, pool_size, strides,
+                          border_mode, dim_ordering):
+        raise NotImplementedError
+
     def get_output(self, train=False):
         X = self.get_input(train)
-        output = K.maxpool2d(X, pool_size=self.pool_size,
-                             strides=self.strides,
-                             border_mode=self.border_mode,
-                             dim_ordering=self.dim_ordering)
+        output = self._pooling_function(inputs=X, pool_size=self.pool_size,
+                                        strides=self.strides,
+                                        border_mode=self.border_mode,
+                                        dim_ordering=self.dim_ordering)
         return output
 
     def get_config(self):
-        config = {"name": self.__class__.__name__,
-                  "pool_size": self.pool_size,
-                  "border_mode": self.border_mode,
-                  "strides": self.strides,
-                  "dim_ordering": self.dim_ordering}
-        base_config = super(MaxPooling2D, self).get_config()
+        config = {'name': self.__class__.__name__,
+                  'pool_size': self.pool_size,
+                  'border_mode': self.border_mode,
+                  'strides': self.strides,
+                  'dim_ordering': self.dim_ordering}
+        base_config = super(_Pooling2D, self).get_config()
         return dict(list(base_config.items()) + list(config.items()))
 
 
+class MaxPooling2D(_Pooling2D):
+    '''Max pooling operation for spatial data.
+
+    # Input shape
+        4D tensor with shape:
+        `(samples, channels, rows, cols)` if dim_ordering='th'
+        or 4D tensor with shape:
+        `(samples, rows, cols, channels)` if dim_ordering='tf'.
+
+    # Output shape
+        4D tensor with shape:
+        `(nb_samples, channels, pooled_rows, pooled_cols)` if dim_ordering='th'
+        or 4D tensor with shape:
+        `(samples, pooled_rows, pooled_cols, channels)` if dim_ordering='tf'.
+
+    # Arguments
+        pool_size: tuple of 2 integers,
+            factors by which to downscale (vertical, horizontal).
+            (2, 2) will halve the image in each dimension.
+        strides: tuple of 2 integers, or None. Strides values.
+        border_mode: 'valid' or 'same'.
+            Note: 'same' will only work with TensorFlow for the time being.
+        dim_ordering: 'th' or 'tf'. In 'th' mode, the channels dimension
+            (the depth) is at index 1, in 'tf' mode is it at index 3.
+    '''
+    def __init__(self, pool_size=(2, 2), strides=None, border_mode='valid',
+                 dim_ordering='th', **kwargs):
+        super(MaxPooling2D, self).__init__(pool_size, strides, border_mode,
+                                           dim_ordering, **kwargs)
+
+    def _pooling_function(self, inputs, pool_size, strides,
+                          border_mode, dim_ordering):
+        output = K.pool2d(inputs, pool_size, strides,
+                          border_mode, dim_ordering, pool_mode='max')
+        return output
+
+
+class AveragePooling2D(_Pooling2D):
+    '''Average pooling operation for spatial data.
+
+    # Input shape
+        4D tensor with shape:
+        `(samples, channels, rows, cols)` if dim_ordering='th'
+        or 4D tensor with shape:
+        `(samples, rows, cols, channels)` if dim_ordering='tf'.
+
+    # Output shape
+        4D tensor with shape:
+        `(nb_samples, channels, pooled_rows, pooled_cols)` if dim_ordering='th'
+        or 4D tensor with shape:
+        `(samples, pooled_rows, pooled_cols, channels)` if dim_ordering='tf'.
+
+    # Arguments
+        pool_size: tuple of 2 integers,
+            factors by which to downscale (vertical, horizontal).
+            (2, 2) will halve the image in each dimension.
+        strides: tuple of 2 integers, or None. Strides values.
+        border_mode: 'valid' or 'same'.
+            Note: 'same' will only work with TensorFlow for the time being.
+        dim_ordering: 'th' or 'tf'. In 'th' mode, the channels dimension
+            (the depth) is at index 1, in 'tf' mode is it at index 3.
+    '''
+    def __init__(self, pool_size=(2, 2), strides=None, border_mode='valid',
+                 dim_ordering='th', **kwargs):
+        super(AveragePooling2D, self).__init__(pool_size, strides, border_mode,
+                                               dim_ordering, **kwargs)
+
+    def _pooling_function(self, inputs, pool_size, strides,
+                          border_mode, dim_ordering):
+        output = K.pool2d(inputs, pool_size, strides,
+                          border_mode, dim_ordering, pool_mode='avg')
+        return output
+
+
 class UpSampling1D(Layer):
+    '''Repeat each temporal step `length` times along the time axis.
+
+    # Input shape
+        3D tensor with shape: `(samples, steps, features)`.
+
+    # Output shape
+        3D tensor with shape: `(samples, upsampled_steps, features)`.
+
+    # Arguments:
+        length: integer. Upsampling factor.
+    '''
     input_ndim = 3
 
     def __init__(self, length=2, **kwargs):
@@ -352,17 +609,38 @@ class UpSampling1D(Layer):
 
     def get_output(self, train=False):
         X = self.get_input(train)
-        output = K.concatenate([X] * self.length, axis=1)
+        output = K.repeat_elements(X, self.length, axis=1)
         return output
 
     def get_config(self):
-        config = {"name": self.__class__.__name__,
-                  "length": self.length}
+        config = {'name': self.__class__.__name__,
+                  'length': self.length}
         base_config = super(UpSampling1D, self).get_config()
         return dict(list(base_config.items()) + list(config.items()))
 
 
 class UpSampling2D(Layer):
+    '''Repeat the rows and columns of the data
+    by size[0] and size[1] respectively.
+
+    # Input shape
+        4D tensor with shape:
+        `(samples, channels, rows, cols)` if dim_ordering='th'
+        or 4D tensor with shape:
+        `(samples, rows, cols, channels)` if dim_ordering='tf'.
+
+    # Output shape
+        4D tensor with shape:
+        `(samples, channels, upsampled_rows, upsampled_cols)` if dim_ordering='th'
+        or 4D tensor with shape:
+        `(samples, upsampled_rows, upsampled_cols, channels)` if dim_ordering='tf'.
+
+    # Arguments
+        size: tuple of 2 integers. The upsampling factors for rows and columns.
+        dim_ordering: 'th' or 'tf'.
+            In 'th' mode, the channels dimension (the depth)
+            is at index 1, in 'tf' mode is it at index 3.
+    '''
     input_ndim = 4
 
     def __init__(self, size=(2, 2), dim_ordering='th', **kwargs):
@@ -391,39 +669,36 @@ class UpSampling2D(Layer):
     def get_output(self, train=False):
         X = self.get_input(train)
         if self.dim_ordering == 'th':
-            output = K.concatenate([X] * self.size[0], axis=2)
-            output = K.concatenate([output] * self.size[1], axis=3)
+            output = K.repeat_elements(X, self.size[0], axis=2)
+            output = K.repeat_elements(output, self.size[1], axis=3)
         elif self.dim_ordering == 'tf':
-            output = K.concatenate([X] * self.size[0], axis=1)
-            output = K.concatenate([output] * self.size[1], axis=2)
+            output = K.repeat_elements(X, self.size[0], axis=1)
+            output = K.repeat_elements(output, self.size[1], axis=2)
         else:
             raise Exception('Invalid dim_ordering: ' + self.dim_ordering)
         return output
 
     def get_config(self):
-        config = {"name": self.__class__.__name__,
-                  "size": self.size}
+        config = {'name': self.__class__.__name__,
+                  'size': self.size}
         base_config = super(UpSampling2D, self).get_config()
         return dict(list(base_config.items()) + list(config.items()))
 
 
 class ZeroPadding1D(Layer):
-    """Zero-padding layer for 1D input (e.g. temporal sequence).
+    '''Zero-padding layer for 1D input (e.g. temporal sequence).
 
-    Input shape
-    -----------
-    3D tensor with shape (samples, axis_to_pad, features)
+    # Input shape
+        3D tensor with shape (samples, axis_to_pad, features)
 
-    Output shape
-    ------------
-    3D tensor with shape (samples, padded_axis, features)
+    # Output shape
+        3D tensor with shape (samples, padded_axis, features)
 
-    Arguments
-    ---------
-    padding: int
-        How many zeros to add at the beginning and end of
-        the padding dimension (axis 1).
-    """
+    # Arguments
+        padding: int
+            How many zeros to add at the beginning and end of
+            the padding dimension (axis 1).
+    '''
     input_ndim = 3
 
     def __init__(self, padding=1, **kwargs):
@@ -443,31 +718,28 @@ class ZeroPadding1D(Layer):
         return K.temporal_padding(X, padding=self.padding)
 
     def get_config(self):
-        config = {"name": self.__class__.__name__,
-                  "padding": self.padding}
+        config = {'name': self.__class__.__name__,
+                  'padding': self.padding}
         base_config = super(ZeroPadding1D, self).get_config()
         return dict(list(base_config.items()) + list(config.items()))
 
 
 class ZeroPadding2D(Layer):
-    """Zero-padding layer for 2D input (e.g. picture).
+    '''Zero-padding layer for 2D input (e.g. picture).
 
-    Input shape
-    -----------
-    4D tensor with shape:
+    # Input shape
+        4D tensor with shape:
         (samples, depth, first_axis_to_pad, second_axis_to_pad)
 
-    Output shape
-    ------------
-    4D tensor with shape:
+    # Output shape
+        4D tensor with shape:
         (samples, depth, first_padded_axis, second_padded_axis)
 
-    Arguments
-    ---------
-    padding: tuple of int (length 2)
-        How many zeros to add at the beginning and end of
-        the 2 padding dimensions (axis 3 and 4).
-    """
+    # Arguments
+        padding: tuple of int (length 2)
+            How many zeros to add at the beginning and end of
+            the 2 padding dimensions (axis 3 and 4).
+    '''
     input_ndim = 4
 
     def __init__(self, padding=(1, 1), dim_ordering='th', **kwargs):
@@ -499,7 +771,7 @@ class ZeroPadding2D(Layer):
                                     dim_ordering=self.dim_ordering)
 
     def get_config(self):
-        config = {"name": self.__class__.__name__,
-                  "padding": self.padding}
+        config = {'name': self.__class__.__name__,
+                  'padding': self.padding}
         base_config = super(ZeroPadding2D, self).get_config()
         return dict(list(base_config.items()) + list(config.items()))

keras/layers/embeddings.py

@@ -8,12 +8,40 @@ from ..constraints import unitnorm
 
 
 class Embedding(Layer):
-    '''
-        Turn positive integers (indexes) into denses vectors of fixed size.
-        eg. [[4], [20]] -> [[0.25, 0.1], [0.6, -0.2]]
+    '''Turn positive integers (indexes) into dense vectors of fixed size.
+    eg. [[4], [20]] -> [[0.25, 0.1], [0.6, -0.2]]
 
-        @input_dim: size of vocabulary (highest input integer + 1)
-        @out_dim: size of dense representation
+    This layer can only be used as the first layer in a model.
+
+    # Input shape
+        2D tensor with shape: `(nb_samples, sequence_length)`.
+
+    # Output shape
+        3D tensor with shape: `(nb_samples, sequence_length, output_dim)`.
+
+    # Arguments
+      input_dim: int >= 0. Size of the vocabulary, ie.
+          1 + maximum integer index occurring in the input data.
+      output_dim: int >= 0. Dimension of the dense embedding.
+      init: name of initialization function for the weights
+          of the layer (see: [initializations](../initializations.md)),
+          or alternatively, Theano function to use for weights initialization.
+          This parameter is only relevant if you don't pass a `weights` argument.
+      weights: list of numpy arrays to set as initial weights.
+          The list should have 1 element, of shape `(input_dim, output_dim)`.
+      W_regularizer: instance of the [regularizers](../regularizers.md) module
+        (eg. L1 or L2 regularization), applied to the embedding matrix.
+      W_constraint: instance of the [constraints](../constraints.md) module
+          (eg. maxnorm, nonneg), applied to the embedding matrix.
+      mask_zero: Whether or not the input value 0 is a special "padding"
+          value that should be masked out.
+          This is useful for [recurrent layers](recurrent.md) which may take
+          variable length input. If this is `True` then all subsequent layers
+          in the model need to support masking or an exception will be raised.
+      input_length: Length of input sequences, when it is constant.
+          This argument is required if you are going to connect
+          `Flatten` then `Dense` layers upstream
+          (without it, the shape of the dense outputs cannot be computed).
     '''
     input_ndim = 2
 
@@ -40,7 +68,7 @@ class Embedding(Layer):
         super(Embedding, self).__init__(**kwargs)
 
     def build(self):
-        self.input = K.placeholder(shape=(None, self.input_length),
+        self.input = K.placeholder(shape=(self.input_shape[0], self.input_length),
                                    dtype='int32')
         self.W = self.init((self.input_dim, self.output_dim))
         self.params = [self.W]

keras/layers/noise.py

@@ -4,8 +4,24 @@ from .. import backend as K
 
 
 class GaussianNoise(MaskedLayer):
-    '''
-        Corruption process with GaussianNoise
+    '''Apply to the input an additive zero-centred gaussian noise with
+    standard deviation `sigma`. This is useful to mitigate overfitting
+    (you could see it as a kind of random data augmentation).
+    Gaussian Noise (GS) is a natural choice as corruption process
+    for real valued inputs.
+
+    As it is a regularization layer, it is only active at training time.
+
+    # Input shape
+        Arbitrary. Use the keyword argument `input_shape`
+        (tuple of integers, does not include the samples axis)
+        when using this layer as the first layer in a model.
+
+    # Output shape
+        Same shape as input.
+
+    # Arguments
+        sigma: float, standard deviation of the noise distribution.
     '''
     def __init__(self, sigma, **kwargs):
         super(GaussianNoise, self).__init__(**kwargs)
@@ -28,12 +44,16 @@ class GaussianNoise(MaskedLayer):
 
 
 class GaussianDropout(MaskedLayer):
-    '''
-        Multiplicative Gaussian Noise
-        Reference:
-            Dropout: A Simple Way to Prevent Neural Networks from Overfitting
-            Srivastava, Hinton, et al. 2014
-            http://www.cs.toronto.edu/~rsalakhu/papers/srivastava14a.pdf
+    '''Apply to the input an multiplicative one-centred gaussian noise
+    with standard deviation `sqrt(p/(1-p))`.
+
+    As it is a regularization layer, it is only active at training time.
+
+    # Arguments
+        p: float, drop probability (as with `Dropout`).
+
+    # References:
+        [Dropout: A Simple Way to Prevent Neural Networks from Overfitting Srivastava, Hinton, et al. 2014](http://www.cs.toronto.edu/~rsalakhu/papers/srivastava14a.pdf)
     '''
     def __init__(self, p, **kwargs):
         super(GaussianDropout, self).__init__(**kwargs)

keras/layers/normalization.py

@@ -4,18 +4,38 @@ from .. import backend as K
 
 
 class BatchNormalization(Layer):
-    '''
-        Reference:
-            Batch Normalization: Accelerating Deep Network Training
-            by Reducing Internal Covariate Shift
-                http://arxiv.org/pdf/1502.03167v3.pdf
+    '''Normalize the activations of the previous layer at each batch,
+    i.e. applies a transformation that maintains the mean activation
+    close to 0 and the activation standard deviation close to 1.
 
-            mode: 0 -> featurewise normalization
-                  1 -> samplewise normalization
-                       (may sometimes outperform featurewise mode)
+    # Input shape
+        Arbitrary. Use the keyword argument `input_shape`
+        (tuple of integers, does not include the samples axis)
+        when using this layer as the first layer in a model.
 
-            momentum: momentum term in the computation
-            of a running estimate of the mean and std of the data
+    # Output shape
+        Same shape as input.
+
+    # Arguments
+        epsilon: small float > 0. Fuzz parameter.
+        mode: integer, 0 or 1.
+            - 0: feature-wise normalization.
+                If the input has multiple feature dimensions,
+                each will be normalized separately
+                (e.g. for an image input with shape
+                `(channels, rows, cols)`,
+                each combination of a channel, row and column
+                will be normalized separately).
+            - 1: sample-wise normalization. This mode assumes a 2D input.
+        momentum: momentum in the computation of the
+            exponential average of the mean and standard deviation
+            of the data, for feature-wise normalization.
+        weights: Initialization weights.
+            List of 2 numpy arrays, with shapes:
+            `[(input_shape,), (input_shape,)]`
+
+    # References
+        - [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](http://arxiv.org/pdf/1502.03167v3.pdf)
     '''
     def __init__(self, epsilon=1e-6, mode=0, momentum=0.9,
                  weights=None, **kwargs):
@@ -30,22 +50,12 @@ class BatchNormalization(Layer):
         input_shape = self.input_shape  # starts with samples axis
         input_shape = input_shape[1:]
 
-        self.gamma = self.init((input_shape))
+        self.gamma = self.init(input_shape)
         self.beta = K.zeros(input_shape)
 
         self.params = [self.gamma, self.beta]
         self.running_mean = K.zeros(input_shape)
-        self.running_std = K.ones((input_shape))
-
-        # initialize self.updates: batch mean/std computation
-        X = self.get_input(train=True)
-        m = K.mean(X, axis=0)
-        std = K.mean(K.square(X - m) + self.epsilon, axis=0)
-        std = K.sqrt(std)
-        mean_update = self.momentum * self.running_mean + (1-self.momentum) * m
-        std_update = self.momentum * self.running_std + (1-self.momentum) * std
-        self.updates = [(self.running_mean, mean_update),
-                        (self.running_std, std_update)]
+        self.running_std = K.ones(input_shape)
 
         if self.initial_weights is not None:
             self.set_weights(self.initial_weights)
@@ -64,6 +74,13 @@ class BatchNormalization(Layer):
     def get_output(self, train):
         X = self.get_input(train)
         if self.mode == 0:
+            m = K.mean(X, axis=0)
+            std = K.mean(K.square(X - m) + self.epsilon, axis=0)
+            std = K.sqrt(std)
+            mean_update = self.momentum * self.running_mean + (1-self.momentum) * m
+            std_update = self.momentum * self.running_std + (1-self.momentum) * std
+            self.updates = [(self.running_mean, mean_update),
+                            (self.running_std, std_update)]
             X_normed = ((X - self.running_mean) /
                         (self.running_std + self.epsilon))
         elif self.mode == 1:
@@ -80,44 +97,3 @@ class BatchNormalization(Layer):
                   "momentum": self.momentum}
         base_config = super(BatchNormalization, self).get_config()
         return dict(list(base_config.items()) + list(config.items()))
-
-
-class LRN2D(Layer):
-    """
-    This code is adapted from pylearn2.
-    License at: https://github.com/lisa-lab/pylearn2/blob/master/LICENSE.txt
-    """
-
-    def __init__(self, alpha=1e-4, k=2, beta=0.75, n=5, **kwargs):
-        if n % 2 == 0:
-            raise NotImplementedError("LRN2D only works with odd n. n provided: " + str(n))
-        super(LRN2D, self).__init__(**kwargs)
-        self.alpha = alpha
-        self.k = k
-        self.beta = beta
-        self.n = n
-
-    def get_output(self, train):
-        X = self.get_input(train)
-        b, ch, r, c = K.shape(X)
-        half_n = self.n // 2
-        input_sqr = K.square(X)
-        extra_channels = K.zeros((b, ch + 2 * half_n, r, c))
-        input_sqr = K.concatenate([extra_channels[:, :half_n, :, :],
-                                   input_sqr,
-                                   extra_channels[:, half_n + ch:, :, :]],
-                                  axis=1)
-        scale = self.k
-        for i in range(self.n):
-            scale += self.alpha * input_sqr[:, i:i + ch, :, :]
-        scale = scale ** self.beta
-        return X / scale
-
-    def get_config(self):
-        config = {"name": self.__class__.__name__,
-                  "alpha": self.alpha,
-                  "k": self.k,
-                  "beta": self.beta,
-                  "n": self.n}
-        base_config = super(LRN2D, self).get_config()
-        return dict(list(base_config.items()) + list(config.items()))

keras/layers/ntm.py

@@ -1,326 +0,0 @@
-import numpy as np
-from scipy.linalg import circulant
-
-from .. import backend as K
-import theano
-import theano.tensor as T
-floatX = theano.config.floatX
-
-from keras.layers.recurrent import Recurrent, GRU, LSTM
-from keras.utils.theano_utils import shared_zeros, alloc_zeros_matrix, shared_scalar
-tol = 1e-4
-
-
-def _update_controller(self, inp, h_tm1, M, mask):
-    """ Update inner RNN controler
-    We have to update the inner RNN inside the Neural Turing Machine, this
-    is an almost literal copy of keras.layers.recurrent.GRU and 
-    keras.layers.recurrent.LSTM see these clases for further details.
-    """
-    x = T.concatenate([inp, M], axis=-1)
-    # get inputs
-    if self.inner_rnn == 'gru':
-        x_z = T.dot(x, self.rnn.W_z) + self.rnn.b_z
-        x_r = T.dot(x, self.rnn.W_r) + self.rnn.b_r
-        x_h = T.dot(x, self.rnn.W_h) + self.rnn.b_h
-
-    elif self.inner_rnn == 'lstm':
-        xi = T.dot(x, self.rnn.W_i) + self.rnn.b_i
-        xf = T.dot(x, self.rnn.W_f) + self.rnn.b_f
-        xc = T.dot(x, self.rnn.W_c) + self.rnn.b_c
-        xo = T.dot(x, self.rnn.W_o) + self.rnn.b_o
-
-    elif self.inner_rnn == 'simple':
-        x = T.dot(x, self.rnn.W) + self.rnn.b
-
-    # update state
-    if self.inner_rnn == 'gru':
-        h = self.rnn._step(x_z, x_r, x_h, 1., h_tm1[0],
-                           self.rnn.U_z,
-                           self.rnn.U_r,
-                           self.rnn.U_h)
-        h = mask[:, None] * h + (1-mask[:, None])*h_tm1[0]
-        h = (h, )
-
-    elif self.inner_rnn == 'lstm':
-        h = self.rnn._step(xi, xf, xo, xc, 1.,
-                           h_tm1[1], h_tm1[0],
-                           self.rnn.U_i, self.rnn.U_f,
-                           self.rnn.U_o, self.rnn.U_c)
-        h = h[::-1]
-        h = tuple([mask[:, None]*h[i] +
-                   (1-mask[:, None])*h_tm1[i] for i in range(len(h))])
-
-    elif self.inner_rnn == 'simple':
-        h = self.rnn._step(x, 1, h_tm1[0], self.rnn.U)
-        h = mask[:, None] * h + (1-mask[:, None])*h_tm1[0]
-        h = (h, )
-
-    return h
-
-
-def _circulant(leng, n_shifts):
-    """ Generate circulant copies of a vector.
-    This will generate a tensor with `n_shifts` of rotated versions the
-    identity matrix. When this tensor is multiplied by a vector
-    the result are `n_shifts` shifted versions of that vector. Since
-    everything is done with inner products, this operation is differentiable.
-
-    Paramters:
-    ----------
-    leng: int > 0, number of memory locations
-    n_shifts: int > 0, number of allowed shifts (if 1, no shift)
-
-    Returns:
-    --------
-    shift operation, a tensor with dimensions (n_shifts, leng, leng)
-    """
-    eye = np.eye(leng)
-    shifts = range(n_shifts//2, -n_shifts//2, -1)
-    C = np.asarray([np.roll(eye, s, axis=1) for s in shifts])
-    return theano.shared(C.astype(theano.config.floatX))
-
-
-def _renorm(x):
-    return x / (x.sum(axis=1, keepdims=True))
-
-
-def _softmax(x):
-    wt = x.flatten(ndim=2)
-    w = T.nnet.softmax(wt)
-    return w.reshape(x.shape)  # T.clip(s, 0, 1)
-
-
-def _cosine_distance(M, k):
-    dot = (M * k[:, None, :]).sum(axis=-1)
-    nM = T.sqrt((M**2).sum(axis=-1))
-    nk = T.sqrt((k**2).sum(axis=-1, keepdims=True))
-    return dot / (nM * nk)
-
-
-class NeuralTuringMachine(Recurrent):
-    """ Neural Turing Machines
-    
-    Parameters:
-    -----------
-    shift_range: int, number of available shifts, ex. if 3, avilable shifts are
-                 (-1, 0, 1)
-    n_slots: number of memory locations
-    m_length: memory length at each location
-    inner_rnn: str, supported values are 'gru' and 'lstm'
-    output_dim: hidden state size (RNN controller output_dim)
-
-    Known issues and TODO:
-    ----------------------
-    Theano may complain when n_slots == 1.
-    Add multiple reading and writing heads.
-
-    """
-    def __init__(self, output_dim, n_slots, m_length, shift_range=3,
-                 inner_rnn='gru', truncate_gradient=-1, return_sequences=False,
-                 init='glorot_uniform', inner_init='orthogonal',
-                 input_dim=None, input_length=None, **kwargs):
-        if K._BACKEND != 'theano':
-            raise Exception('NeuralTuringMachine is only available for Theano for the time being. ' +
-                            'It will be adapted to TensorFlow soon.')
-        self.output_dim = output_dim
-        self.n_slots = n_slots
-        self.m_length = m_length
-        self.shift_range = shift_range
-        self.init = init
-        self.inner_init = inner_init
-        self.inner_rnn = inner_rnn
-        self.return_sequences = return_sequences
-        self.truncate_gradient = truncate_gradient
-
-        self.input_dim = input_dim
-        self.input_length = input_length
-        if self.input_dim:
-            kwargs['input_shape'] = (self.input_length, self.input_dim)
-        super(NeuralTuringMachine, self).__init__(**kwargs)
-
-    def build(self):
-        input_leng, input_dim = self.input_shape[1:]
-        self.input = T.tensor3()
-
-        if self.inner_rnn == 'gru':
-            self.rnn = GRU(
-                input_dim=input_dim+self.m_length,
-                input_length=input_leng,
-                output_dim=self.output_dim, init=self.init,
-                inner_init=self.inner_init)
-        elif self.inner_rnn == 'lstm':
-            self.rnn = LSTM(
-                input_dim=input_dim+self.m_length,
-                input_length=input_leng,
-                output_dim=self.output_dim, init=self.init,
-                inner_init=self.inner_init)
-        else:
-            raise ValueError('this inner_rnn is not implemented yet.')
-
-        self.rnn.build()
-
-        # initial memory, state, read and write vecotrs
-        self.M = theano.shared((.001*np.ones((1,)).astype(floatX)))
-        self.init_h = shared_zeros((self.output_dim))
-        self.init_wr = self.rnn.init((self.n_slots,))
-        self.init_ww = self.rnn.init((self.n_slots,))
-
-        # write
-        self.W_e = self.rnn.init((self.output_dim, self.m_length))  # erase
-        self.b_e = shared_zeros((self.m_length))
-        self.W_a = self.rnn.init((self.output_dim, self.m_length))  # add
-        self.b_a = shared_zeros((self.m_length))
-
-        # get_w  parameters for reading operation
-        self.W_k_read = self.rnn.init((self.output_dim, self.m_length))
-        self.b_k_read = self.rnn.init((self.m_length, ))
-        self.W_c_read = self.rnn.init((self.output_dim, 3))  # 3 = beta, g, gamma see eq. 5, 7, 9 in Graves et. al 2014
-        self.b_c_read = shared_zeros((3))
-        self.W_s_read = self.rnn.init((self.output_dim, self.shift_range))
-        self.b_s_read = shared_zeros((self.shift_range)) 
-
-        # get_w  parameters for writing operation
-        self.W_k_write = self.rnn.init((self.output_dim, self.m_length))
-        self.b_k_write = self.rnn.init((self.m_length, ))
-        self.W_c_write = self.rnn.init((self.output_dim, 3))  # 3 = beta, g, gamma see eq. 5, 7, 9
-        self.b_c_write = shared_zeros((3))
-        self.W_s_write = self.rnn.init((self.output_dim, self.shift_range))
-        self.b_s_write = shared_zeros((self.shift_range))
-
-        self.C = _circulant(self.n_slots, self.shift_range)
-
-        self.params = self.rnn.params + [
-            self.W_e, self.b_e,
-            self.W_a, self.b_a,
-            self.W_k_read, self.b_k_read,
-            self.W_c_read, self.b_c_read,
-            self.W_s_read, self.b_s_read,
-            self.W_k_write, self.b_k_write,
-            self.W_s_write, self.b_s_write,
-            self.W_c_write, self.b_c_write,
-            self.M,
-            self.init_h, self.init_wr, self.init_ww]
-
-        if self.inner_rnn == 'lstm':
-            self.init_c = shared_zeros((self.output_dim))
-            self.params = self.params + [self.init_c, ]
-
-    def _read(self, w, M):
-        return (w[:, :, None]*M).sum(axis=1)
-
-    def _write(self, w, e, a, M, mask):
-        Mtilda = M * (1 - w[:, :, None]*e[:, None, :])
-        Mout = Mtilda + w[:, :, None]*a[:, None, :]
-        return mask[:, None, None]*Mout + (1-mask[:, None, None])*M
-
-    def _get_content_w(self, beta, k, M):
-        num = beta[:, None] * _cosine_distance(M, k)
-        return _softmax(num)
-
-    def _get_location_w(self, g, s, C, gamma, wc, w_tm1, mask):
-        wg = g[:, None] * wc + (1-g[:, None])*w_tm1
-        Cs = (C[None, :, :, :] * wg[:, None, None, :]).sum(axis=3)
-        wtilda = (Cs * s[:, :, None]).sum(axis=1)
-        wout = _renorm(wtilda ** gamma[:, None])
-        return mask[:, None] * wout + (1-mask[:, None])*w_tm1
-
-    def _get_controller_output(self, h, W_k, b_k, W_c, b_c, W_s, b_s):
-        k = T.tanh(T.dot(h, W_k) + b_k)  # + 1e-6
-        c = T.dot(h, W_c) + b_c
-        beta = T.nnet.relu(c[:, 0]) + 1e-6
-        g = T.nnet.sigmoid(c[:, 1])
-        gamma = T.nnet.relu(c[:, 2]) + 1
-        s = T.nnet.softmax(T.dot(h, W_s) + b_s)
-        return k, beta, g, gamma, s
-
-    def _get_initial_states(self, batch_size):
-        init_M = self.M.dimshuffle(0, 'x', 'x').repeat(
-           batch_size, axis=0).repeat(self.n_slots, axis=1).repeat(
-               self.m_length, axis=2)
-
-        init_h = self.init_h.dimshuffle(('x', 0)).repeat(batch_size, axis=0)
-        init_wr = self.init_wr.dimshuffle(('x', 0)).repeat(batch_size, axis=0)
-        init_ww = self.init_ww.dimshuffle(('x', 0)).repeat(batch_size, axis=0)
-        if self.inner_rnn == 'lstm':
-            init_c = self.init_c.dimshuffle(('x', 0)).repeat(batch_size, axis=0)
-            return init_M, T.nnet.softmax(init_wr), T.nnet.softmax(init_ww), init_h, init_c
-        else:
-            return init_M, T.nnet.softmax(init_wr), T.nnet.softmax(init_ww), init_h
-
-    def _step(self, x, mask, M_tm1, wr_tm1, ww_tm1, *args):
-        # read
-        if self.inner_rnn == 'lstm':
-            h_tm1 = args[0:2][::-1]  # (cell_tm1, h_tm1)
-        else:
-            h_tm1 = args[0:1]  # (h_tm1, )
-        k_read, beta_read, g_read, gamma_read, s_read = self._get_controller_output(
-            h_tm1[-1], self.W_k_read, self.b_k_read, self.W_c_read, self.b_c_read,
-            self.W_s_read, self.b_s_read)
-        wc_read = self._get_content_w(beta_read, k_read, M_tm1)
-        wr_t = self._get_location_w(g_read, s_read, self.C, gamma_read,
-                                    wc_read, wr_tm1, mask)
-        M_read = self._read(wr_t, M_tm1)
-
-        # update controller
-        h_t = _update_controller(self, x, h_tm1, M_read, mask)
-
-        # write
-        k_write, beta_write, g_write, gamma_write, s_write = self._get_controller_output(
-            h_t[-1], self.W_k_write, self.b_k_write, self.W_c_write,
-            self.b_c_write, self.W_s_write, self.b_s_write)
-        wc_write = self._get_content_w(beta_write, k_write, M_tm1)
-        ww_t = self._get_location_w(g_write, s_write, self.C, gamma_write,
-                                    wc_write, ww_tm1, mask)
-        e = T.nnet.sigmoid(T.dot(h_t[-1], self.W_e) + self.b_e)
-        a = T.tanh(T.dot(h_t[-1], self.W_a) + self.b_a)
-        M_t = self._write(ww_t, e, a, M_tm1, mask)
-
-        return (M_t, wr_t, ww_t) + h_t
-
-    def get_output(self, train=False):
-        outputs = self.get_full_output(train)
-
-        if self.return_sequences:
-            return outputs[-1]
-        else:
-            return outputs[-1][:, -1]
-
-    @property
-    def output_shape(self):
-        input_shape = self.input_shape
-        if self.return_sequences:
-            return input_shape[0], input_shape[1], self.output_dim
-        else:
-            return input_shape[0], self.output_dim
-
-    def get_full_output(self, train=False):
-        """
-        This method is for research and visualization purposes. Use it as:
-        X = model.get_input()  # full model
-        Y = ntm.get_output()    # this layer
-        F = theano.function([X], Y, allow_input_downcast=True)
-        [memory, read_address, write_address, rnn_state] = F(x)
-
-        if inner_rnn == "lstm" use it as:
-        [memory, read_address, write_address, rnn_cell, rnn_state] = F(x)
-
-        """
-        X = self.get_input(train)
-        padded_mask = self.get_padded_shuffled_mask(train, X, pad=1)[:, :, 0]
-        X = X.dimshuffle((1, 0, 2))
-
-        init_states = self._get_initial_states(X.shape[1])
-        outputs, updates = theano.scan(self._step,
-                                       sequences=[X, padded_mask],
-                                       outputs_info=init_states,
-                                       non_sequences=self.params,
-                                       truncate_gradient=self.truncate_gradient)
-
-        out = [outputs[0].dimshuffle((1, 0, 2, 3)),
-               outputs[1].dimshuffle(1, 0, 2),
-               outputs[2].dimshuffle((1, 0, 2)),
-               outputs[3].dimshuffle((1, 0, 2))]
-        if self.inner_rnn == 'lstm':
-            out + [outputs[4].dimshuffle((1, 0, 2))]
-        return out

keras/layers/recurrent.py

@@ -8,6 +8,78 @@ from ..layers.core import MaskedLayer
 
 
 class Recurrent(MaskedLayer):
+    '''Abstract base class for recurrent layers.
+    Do not use in a model -- it's not a functional layer!
+
+    All recurrent layers (GRU, LSTM, SimpleRNN) also
+    follow the specifications of this class and accept
+    the keyword arguments listed below.
+
+    # Input shape
+        3D tensor with shape `(nb_samples, timesteps, input_dim)`.
+
+    # Output shape
+        - if `return_sequences`: 3D tensor with shape
+            `(nb_samples, timesteps, output_dim)`.
+        - else, 2D tensor with shape `(nb_samples, output_dim)`.
+
+    # Arguments
+        weights: list of numpy arrays to set as initial weights.
+            The list should have 3 elements, of shapes:
+            `[(input_dim, output_dim), (output_dim, output_dim), (output_dim,)]`.
+        return_sequences: Boolean. Whether to return the last output
+            in the output sequence, or the full sequence.
+        go_backwards: Boolean (default False).
+            If True, process the input sequence backwards.
+        stateful: Boolean (default False). If True, the last state
+            for each sample at index i in a batch will be used as initial
+            state for the sample of index i in the following batch.
+        input_dim: dimensionality of the input (integer).
+            This argument (or alternatively, the keyword argument `input_shape`)
+            is required when using this layer as the first layer in a model.
+        input_length: Length of input sequences, to be specified
+            when it is constant.
+            This argument is required if you are going to connect
+            `Flatten` then `Dense` layers upstream
+            (without it, the shape of the dense outputs cannot be computed).
+            Note that if the recurrent layer is not the first layer
+            in your model, you would need to specify the input length
+            at the level of the first layer
+            (e.g. via the `input_shape` argument)
+
+    # Masking
+        This layer supports masking for input data with a variable number
+        of timesteps. To introduce masks to your data,
+        use an [Embedding](embeddings.md) layer with the `mask_zero` parameter
+        set to `True`.
+        **Note:** for the time being, masking is only supported with Theano.
+
+    # TensorFlow warning
+        For the time being, when using the TensorFlow backend,
+        the number of timesteps used must be specified in your model.
+        Make sure to pass an `input_length` int argument to your
+        recurrent layer (if it comes first in your model),
+        or to pass a complete `input_shape` argument to the first layer
+        in your model otherwise.
+
+
+    # Note on using statefulness in RNNs
+        You can set RNN layers to be 'stateful', which means that the states
+        computed for the samples in one batch will be reused as initial states
+        for the samples in the next batch.
+        This assumes a one-to-one mapping between
+        samples in different successive batches.
+
+        To enable statefulness:
+            - specify `stateful=True` in the layer constructor.
+            - specify a fixed batch size for your model, by passing
+                a `batch_input_shape=(...)` to the first layer in your model.
+                This is the expected shape of your inputs *including the batch size*.
+                It should be a tuple of integers, e.g. `(32, 10, 100)`.
+
+        To reset the states of your model, call `.reset_states()` on either
+        a specific layer, or on your entire model.
+    '''
     input_ndim = 3
 
     def __init__(self, weights=None,
@@ -41,6 +113,15 @@ class Recurrent(MaskedLayer):
     def step(self, x, states):
         raise NotImplementedError
 
+    def get_initial_states(self, X):
+        # build an all-zero tensor of shape (samples, output_dim)
+        initial_state = K.zeros_like(X)  # (samples, timesteps, input_dim)
+        initial_state = K.sum(initial_state, axis=1)  # (samples, input_dim)
+        reducer = K.zeros((self.input_dim, self.output_dim))
+        initial_state = K.dot(initial_state, reducer)  # (samples, output_dim)
+        initial_states = [initial_state for _ in range(len(self.states))]
+        return initial_states
+
     def get_output(self, train=False):
         # input shape: (nb_samples, time (padded with zeros), input_dim)
         X = self.get_input(train)
@@ -48,15 +129,15 @@ class Recurrent(MaskedLayer):
         if K._BACKEND == 'tensorflow':
             if not self.input_shape[1]:
                 raise Exception('When using TensorFlow, you should define ' +
-                                'explicitely the number of timesteps of ' +
+                                'explicitly the number of timesteps of ' +
                                 'your sequences. Make sure the first layer ' +
-                                'has an "input_shape" argument with a defined ' +
-                                'first dimension.')
+                                'has a "batch_input_shape" argument ' +
+                                'including the samples axis.')
 
         mask = self.get_output_mask(train)
         if mask:
             # apply mask
-            X *= K.expand_dims(mask)
+            X *= K.cast(K.expand_dims(mask), X.dtype)
             masking = True
         else:
             masking = False
@@ -64,19 +145,15 @@ class Recurrent(MaskedLayer):
         if self.stateful:
             initial_states = self.states
         else:
-            # build an all-zero tensor of shape (samples, output_dim)
-            initial_state = K.zeros_like(X)  # (samples, timesteps, input_dim)
-            initial_state = K.sum(initial_state, axis=1)  # (samples, input_dim)
-            reducer = K.zeros((self.input_dim, self.output_dim))
-            initial_state = K.dot(initial_state, reducer)  # (samples, output_dim)
-            initial_states = [initial_state for _ in range(len(self.states))]
+            initial_states = self.get_initial_states(X)
 
         last_output, outputs, states = K.rnn(self.step, X, initial_states,
                                              go_backwards=self.go_backwards,
                                              masking=masking)
         if self.stateful:
+            self.updates = []
             for i in range(len(states)):
-                K.set_value(self.states[i], states[i])
+                self.updates.append((self.states[i], states[i]))
 
         if self.return_sequences:
             return outputs
@@ -95,17 +172,17 @@ class Recurrent(MaskedLayer):
 
 
 class SimpleRNN(Recurrent):
-    '''
-        Fully-connected RNN where the output is to fed back to input.
-        Takes inputs with shape:
-        (nb_samples, max_sample_length, input_dim)
-        (samples shorter than `max_sample_length`
-         are padded with zeros at the end)
-        and returns outputs with shape:
-        if not return_sequences:
-            (nb_samples, output_dim)
-        if return_sequences:
-            (nb_samples, max_sample_length, output_dim)
+    '''Fully-connected RNN where the output is to fed back to input.
+
+    # Arguments
+        output_dim: dimension of the internal projections and the final output.
+        init: weight initialization function.
+            Can be the name of an existing function (str),
+            or a Theano function (see: [initializations](../initializations.md)).
+        inner_init: initialization function of the inner cells.
+        activation: activation function.
+            Can be the name of an existing function (str),
+            or a Theano function (see: [activations](../activations.md)).
     '''
     def __init__(self, output_dim,
                  init='glorot_uniform', inner_init='orthogonal',
@@ -119,11 +196,7 @@ class SimpleRNN(Recurrent):
     def build(self):
         input_shape = self.input_shape
         if self.stateful:
-            if not input_shape[0]:
-                raise Exception('If a RNN is stateful, a complete ' +
-                                'input_shape must be provided ' +
-                                '(including batch size).')
-            self.states = [K.zeros(input_shape[0], self.output_dim)]
+            self.reset_states()
         else:
             # initial states: all-zero tensor of shape (output_dim)
             self.states = [None]
@@ -132,19 +205,32 @@ class SimpleRNN(Recurrent):
 
         self.W = self.init((input_dim, self.output_dim))
         self.U = self.inner_init((self.output_dim, self.output_dim))
-        self.b = K.zeros((self.output_dim))
+        self.b = K.zeros((self.output_dim,))
         self.params = [self.W, self.U, self.b]
 
         if self.initial_weights is not None:
             self.set_weights(self.initial_weights)
             del self.initial_weights
 
+    def reset_states(self):
+        assert self.stateful, 'Layer must be stateful.'
+        input_shape = self.input_shape
+        if not input_shape[0]:
+            raise Exception('If a RNN is stateful, a complete ' +
+                            'input_shape must be provided ' +
+                            '(including batch size).')
+        if hasattr(self, 'states'):
+            K.set_value(self.states[0],
+                        np.zeros((input_shape[0], self.output_dim)))
+        else:
+            self.states = [K.zeros((input_shape[0], self.output_dim))]
+
     def step(self, x, states):
         # states only contains the previous output.
         assert len(states) == 1
         prev_output = states[0]
         h = K.dot(x, self.W) + self.b
-        output = self.activation(h * K.dot(prev_output, self.U))
+        output = self.activation(h + K.dot(prev_output, self.U))
         return output, [output]
 
     def get_config(self):
@@ -157,26 +243,22 @@ class SimpleRNN(Recurrent):
 
 
 class GRU(Recurrent):
-    '''
-        Gated Recurrent Unit - Cho et al. 2014
-        Acts as a spatiotemporal projection,
-        turning a sequence of vectors into a single vector.
-        Takes inputs with shape:
-        (nb_samples, max_sample_length, input_dim)
-        (samples shorter than `max_sample_length`
-         are padded with zeros at the end)
-        and returns outputs with shape:
-        if not return_sequences:
-            (nb_samples, output_dim)
-        if return_sequences:
-            (nb_samples, max_sample_length, output_dim)
-        References:
-            On the Properties of Neural Machine Translation:
-            Encoder–Decoder Approaches
-                http://www.aclweb.org/anthology/W14-4012
-            Empirical Evaluation of Gated Recurrent Neural Networks
-            on Sequence Modeling
-                http://arxiv.org/pdf/1412.3555v1.pdf
+    '''Gated Recurrent Unit - Cho et al. 2014.
+
+    # Arguments
+        output_dim: dimension of the internal projections and the final output.
+        init: weight initialization function.
+            Can be the name of an existing function (str),
+            or a Theano function (see: [initializations](../initializations.md)).
+        inner_init: initialization function of the inner cells.
+        activation: activation function.
+            Can be the name of an existing function (str),
+            or a Theano function (see: [activations](../activations.md)).
+        inner_activation: activation function for the inner cells.
+
+    # References
+        - [On the Properties of Neural Machine Translation: Encoder–Decoder Approaches](http://www.aclweb.org/anthology/W14-4012)
+        - [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](http://arxiv.org/pdf/1412.3555v1.pdf)
     '''
     def __init__(self, output_dim,
                  init='glorot_uniform', inner_init='orthogonal',
@@ -212,11 +294,7 @@ class GRU(Recurrent):
                        self.W_h, self.U_h, self.b_h]
 
         if self.stateful:
-            if not input_shape[0]:
-                raise Exception('If a RNN is stateful, a complete ' +
-                                'input_shape must be provided ' +
-                                '(including batch size).')
-            self.states = [K.zeros(input_shape[0], self.output_dim)]
+            self.reset_states()
         else:
             # initial states: all-zero tensor of shape (output_dim)
             self.states = [None]
@@ -225,6 +303,19 @@ class GRU(Recurrent):
             self.set_weights(self.initial_weights)
             del self.initial_weights
 
+    def reset_states(self):
+        assert self.stateful, 'Layer must be stateful.'
+        input_shape = self.input_shape
+        if not input_shape[0]:
+            raise Exception('If a RNN is stateful, a complete ' +
+                            'input_shape must be provided ' +
+                            '(including batch size).')
+        if hasattr(self, 'states'):
+            K.set_value(self.states[0],
+                        np.zeros((input_shape[0], self.output_dim)))
+        else:
+            self.states = [K.zeros((input_shape[0], self.output_dim))]
+
     def step(self, x, states):
         assert len(states) == 1
         x_z = K.dot(x, self.W_z) + self.b_z
@@ -235,7 +326,7 @@ class GRU(Recurrent):
         z = self.inner_activation(x_z + K.dot(h_tm1, self.U_z))
         r = self.inner_activation(x_r + K.dot(h_tm1, self.U_r))
 
-        hh = self.inner_activation(x_h + K.dot(r * h_tm1, self.U_h))
+        hh = self.activation(x_h + K.dot(r * h_tm1, self.U_h))
         h = z * h_tm1 + (1 - z) * hh
         return h, [h]
 
@@ -250,27 +341,29 @@ class GRU(Recurrent):
 
 
 class LSTM(Recurrent):
-    '''
-        Acts as a spatiotemporal projection,
-        turning a sequence of vectors into a single vector.
-        Takes inputs with shape:
-        (nb_samples, max_sample_length, input_dim)
-        (samples shorter than `max_sample_length`
-         are padded with zeros at the end)
-        and returns outputs with shape:
-        if not return_sequences:
-            (nb_samples, output_dim)
-        if return_sequences:
-            (nb_samples, max_sample_length, output_dim)
-        For a step-by-step description of the algorithm, see:
-        http://deeplearning.net/tutorial/lstm.html
-        References:
-            Long short-term memory (original 97 paper)
-                http://deeplearning.cs.cmu.edu/pdfs/Hochreiter97_lstm.pdf
-            Learning to forget: Continual prediction with LSTM
-                http://www.mitpressjournals.org/doi/pdf/10.1162/089976600300015015
-            Supervised sequence labelling with recurrent neural networks
-                http://www.cs.toronto.edu/~graves/preprint.pdf
+    '''Long-Short Term Memory unit - Hochreiter 1997.
+
+    For a step-by-step description of the algorithm, see
+    [this tutorial](http://deeplearning.net/tutorial/lstm.html).
+
+    # Arguments
+        output_dim: dimension of the internal projections and the final output.
+        init: weight initialization function.
+            Can be the name of an existing function (str),
+            or a Theano function (see: [initializations](../initializations.md)).
+        inner_init: initialization function of the inner cells.
+        forget_bias_init: initialization function for the bias of the forget gate.
+            [Jozefowicz et al.](http://www.jmlr.org/proceedings/papers/v37/jozefowicz15.pdf)
+            recommend initializing with ones.
+        activation: activation function.
+            Can be the name of an existing function (str),
+            or a Theano function (see: [activations](../activations.md)).
+        inner_activation: activation function for the inner cells.
+
+    # References
+        - [Long short-term memory](http://deeplearning.cs.cmu.edu/pdfs/Hochreiter97_lstm.pdf) (original 1997 paper)
+        - [Learning to forget: Continual prediction with LSTM](http://www.mitpressjournals.org/doi/pdf/10.1162/089976600300015015)
+        - [Supervised sequence labelling with recurrent neural networks](http://www.cs.toronto.edu/~graves/preprint.pdf)
     '''
     def __init__(self, output_dim,
                  init='glorot_uniform', inner_init='orthogonal',
@@ -291,31 +384,26 @@ class LSTM(Recurrent):
         self.input = K.placeholder(input_shape)
 
         if self.stateful:
-            if not input_shape[0]:
-                raise Exception('If a RNN is stateful, a complete ' +
-                                'input_shape must be provided ' +
-                                '(including batch size).')
-            self.states = [K.zeros(input_shape[0], self.output_dim),
-                           K.zeros(input_shape[0], self.output_dim)]
+            self.reset_states()
         else:
             # initial states: 2 all-zero tensor of shape (output_dim)
             self.states = [None, None]
 
         self.W_i = self.init((input_dim, self.output_dim))
         self.U_i = self.inner_init((self.output_dim, self.output_dim))
-        self.b_i = K.zeros((self.output_dim))
+        self.b_i = K.zeros((self.output_dim,))
 
         self.W_f = self.init((input_dim, self.output_dim))
         self.U_f = self.inner_init((self.output_dim, self.output_dim))
-        self.b_f = self.forget_bias_init((self.output_dim))
+        self.b_f = self.forget_bias_init((self.output_dim,))
 
         self.W_c = self.init((input_dim, self.output_dim))
         self.U_c = self.inner_init((self.output_dim, self.output_dim))
-        self.b_c = K.zeros((self.output_dim))
+        self.b_c = K.zeros((self.output_dim,))
 
         self.W_o = self.init((input_dim, self.output_dim))
         self.U_o = self.inner_init((self.output_dim, self.output_dim))
-        self.b_o = K.zeros((self.output_dim))
+        self.b_o = K.zeros((self.output_dim,))
 
         self.params = [self.W_i, self.U_i, self.b_i,
                        self.W_c, self.U_c, self.b_c,
@@ -326,6 +414,22 @@ class LSTM(Recurrent):
             self.set_weights(self.initial_weights)
             del self.initial_weights
 
+    def reset_states(self):
+        assert self.stateful, 'Layer must be stateful.'
+        input_shape = self.input_shape
+        if not input_shape[0]:
+            raise Exception('If a RNN is stateful, a complete ' +
+                            'input_shape must be provided ' +
+                            '(including batch size).')
+        if hasattr(self, 'states'):
+            K.set_value(self.states[0],
+                        np.zeros((input_shape[0], self.output_dim)))
+            K.set_value(self.states[1],
+                        np.zeros((input_shape[0], self.output_dim)))
+        else:
+            self.states = [K.zeros((input_shape[0], self.output_dim)),
+                           K.zeros((input_shape[0], self.output_dim))]
+
     def step(self, x, states):
         assert len(states) == 2
         h_tm1 = states[0]

keras/objectives.py

@@ -16,13 +16,13 @@ def mean_absolute_error(y_true, y_pred):
 
 
 def mean_absolute_percentage_error(y_true, y_pred):
-    diff = K.abs((y_true - y_pred) / K.clip(K.abs(y_true), K._EPSILON, np.inf))
+    diff = K.abs((y_true - y_pred) / K.clip(K.abs(y_true), K.epsilon(), np.inf))
     return 100. * K.mean(diff, axis=-1)
 
 
 def mean_squared_logarithmic_error(y_true, y_pred):
-    first_log = K.log(K.clip(y_pred, K._EPSILON, np.inf) + 1.)
-    second_log = K.log(K.clip(y_true, K._EPSILON, np.inf) + 1.)
+    first_log = K.log(K.clip(y_pred, K.epsilon(), np.inf) + 1.)
+    second_log = K.log(K.clip(y_true, K.epsilon(), np.inf) + 1.)
     return K.mean(K.square(first_log - second_log), axis=-1)
 
 
@@ -35,7 +35,7 @@ def hinge(y_true, y_pred):
 
 
 def categorical_crossentropy(y_true, y_pred):
-    '''Expects a binary class matrix instead of a vector of scalar classes
+    '''Expects a binary class matrix instead of a vector of scalar classes.
     '''
     return K.mean(K.categorical_crossentropy(y_pred, y_true), axis=-1)
 
@@ -45,7 +45,7 @@ def binary_crossentropy(y_true, y_pred):
 
 
 def poisson_loss(y_true, y_pred):
-    return K.mean(y_pred - y_true * K.log(y_pred + K._EPSILON), axis=-1)
+    return K.mean(y_pred - y_true * K.log(y_pred + K.epsilon()), axis=-1)
 
 # aliases
 mse = MSE = mean_squared_error

keras/optimizers.py

@@ -16,6 +16,18 @@ def kl_divergence(p, p_hat):
 
 
 class Optimizer(object):
+    '''Abstract optimizer base class.
+
+    Note: this is the parent class of all optimizers, not an actual optimizer
+    that can be used for training models.
+
+    All Keras optimizers support the following keyword arguments:
+
+        clipnorm: float >= 0. Gradients will be clipped
+            when their L2 norm exceeds this value.
+        clipvalue: float >= 0. Gradients will be clipped
+            when their absolute value exceeds this value.
+    '''
     def __init__(self, **kwargs):
         self.__dict__.update(kwargs)
         self.updates = []
@@ -45,7 +57,15 @@ class Optimizer(object):
 
 
 class SGD(Optimizer):
+    '''Stochastic gradient descent, with support for momentum,
+    decay, and Nesterov momentum.
 
+    # Arguments
+        lr: float >= 0. Learning rate.
+        momentum: float >= 0. Parameter updates momentum.
+        decay: float >= 0. Learning rate decay over each update.
+        nesterov: boolean. Whether to apply Nesterov momentum.
+    '''
     def __init__(self, lr=0.01, momentum=0., decay=0., nesterov=False,
                  *args, **kwargs):
         super(SGD, self).__init__(**kwargs)
@@ -82,6 +102,19 @@ class SGD(Optimizer):
 
 
 class RMSprop(Optimizer):
+    '''RMSProp optimizer.
+
+    It is recommended to leave the parameters of this optimizer
+    at their default values.
+
+    This optimizer is usually a good choice for recurrent
+    neural networks.
+
+    # Arguments
+        lr: float >= 0. Learning rate.
+        rho: float >= 0.
+        epsilon: float >= 0. Fuzz factor.
+    '''
     def __init__(self, lr=0.001, rho=0.9, epsilon=1e-6, *args, **kwargs):
         super(RMSprop, self).__init__(**kwargs)
         self.__dict__.update(locals())
@@ -110,6 +143,15 @@ class RMSprop(Optimizer):
 
 
 class Adagrad(Optimizer):
+    '''Adagrad optimizer.
+
+    It is recommended to leave the parameters of this optimizer
+    at their default values.
+
+    # Arguments
+        lr: float >= 0. Learning rate.
+        epsilon: float >= 0.
+    '''
     def __init__(self, lr=0.01, epsilon=1e-6, *args, **kwargs):
         super(Adagrad, self).__init__(**kwargs)
         self.__dict__.update(locals())
@@ -134,8 +176,18 @@ class Adagrad(Optimizer):
 
 
 class Adadelta(Optimizer):
-    '''
-        Reference: http://arxiv.org/abs/1212.5701
+    '''Adadelta optimizer.
+
+    It is recommended to leave the parameters of this optimizer
+    at their default values.
+
+    # Arguments
+        lr: float >= 0. Learning rate. It is recommended to leave it at the default value.
+        rho: float >= 0.
+        epsilon: float >= 0. Fuzz factor.
+
+    # References
+        - [Adadelta - an adaptive learning rate method](http://arxiv.org/abs/1212.5701)
     '''
     def __init__(self, lr=1.0, rho=0.95, epsilon=1e-6, *args, **kwargs):
         super(Adadelta, self).__init__(**kwargs)
@@ -168,15 +220,22 @@ class Adadelta(Optimizer):
     def get_config(self):
         return {"name": self.__class__.__name__,
                 "lr": float(K.get_value(self.lr)),
-                "rho": float(K.get_value(self.rho)),
+                "rho": self.rho,
                 "epsilon": self.epsilon}
 
 
 class Adam(Optimizer):
-    '''
-        Reference: http://arxiv.org/abs/1412.6980v8
+    '''Adam optimizer.
 
-        Default parameters follow those provided in the original paper.
+    Default parameters follow those provided in the original paper.
+
+    # Arguments
+        lr: float >= 0. Learning rate.
+        beta_1/beta_2: floats, 0 < beta < 1. Generally close to 1.
+        epsilon: float >= 0. Fuzz factor.
+
+    # References
+        - [Adam - A Method for Stochastic Optimization](http://arxiv.org/abs/1412.6980v8)
     '''
     def __init__(self, lr=0.001, beta_1=0.9, beta_2=0.999, epsilon=1e-8,
                  *args, **kwargs):
@@ -216,12 +275,66 @@ class Adam(Optimizer):
                 "beta_2": float(K.get_value(self.beta_2)),
                 "epsilon": self.epsilon}
 
+class Adamax(Optimizer):
+    '''Adamax optimizer from Adam paper's Section 7. It is a variant
+     of Adam based on the infinity norm.
+
+    Default parameters follow those provided in the paper.
+
+    # Arguments
+        lr: float >= 0. Learning rate.
+        beta_1/beta_2: floats, 0 < beta < 1. Generally close to 1.
+        epsilon: float >= 0. Fuzz factor.
+
+    # References
+        - [Adam - A Method for Stochastic Optimization](http://arxiv.org/abs/1412.6980v8)
+    '''
+    def __init__(self, lr=0.002, beta_1=0.9, beta_2=0.999, epsilon=1e-8,
+                 *args, **kwargs):
+        super(Adamax, self).__init__(**kwargs)
+        self.__dict__.update(locals())
+        self.iterations = K.variable(0)
+        self.lr = K.variable(lr)
+        self.beta_1 = K.variable(beta_1)
+        self.beta_2 = K.variable(beta_2)
+
+    def get_updates(self, params, constraints, loss):
+        grads = self.get_gradients(loss, params)
+        self.updates = [(self.iterations, self.iterations+1.)]
+
+        t = self.iterations + 1
+        lr_t = self.lr / (1 - K.pow(self.beta_1, t))
+
+        for p, g, c in zip(params, grads, constraints):
+            # zero init of 1st moment
+            m = K.variable(np.zeros(K.get_value(p).shape))
+            # zero init of exponentially weighted infinity norm
+            u = K.variable(np.zeros(K.get_value(p).shape))
+
+            m_t = (self.beta_1 * m) + (1 - self.beta_1) * g
+            u_t = K.maximum(self.beta_2 * u, K.abs(g))
+            p_t = p - lr_t * m_t / (u_t + self.epsilon)
+
+            self.updates.append((m, m_t))
+            self.updates.append((u, u_t))
+            self.updates.append((p, c(p_t)))  # apply constraints
+        return self.updates
+
+    def get_config(self):
+        return {"name": self.__class__.__name__,
+                "lr": float(K.get_value(self.lr)),
+                "beta_1": float(K.get_value(self.beta_1)),
+                "beta_2": float(K.get_value(self.beta_2)),
+                "epsilon": self.epsilon}
+
+
 # aliases
 sgd = SGD
 rmsprop = RMSprop
 adagrad = Adagrad
 adadelta = Adadelta
 adam = Adam
+adamax = Adamax
 
 
 def get(identifier, kwargs=None):

keras/preprocessing/sequence.py

@@ -6,7 +6,7 @@ from six.moves import range
 
 def pad_sequences(sequences, maxlen=None, dtype='int32', padding='pre', truncating='pre', value=0.):
     """
-        Pad each sequence to the same length: 
+        Pad each sequence to the same length:
         the length of the longest sequence.
 
         If maxlen is provided, any sequence longer
@@ -15,6 +15,19 @@ def pad_sequences(sequences, maxlen=None, dtype='int32', padding='pre', truncati
 
         Supports post-padding and pre-padding (default).
 
+        Parameters:
+        -----------
+        sequences: list of lists where each element is a sequence
+        maxlen: int, maximum length
+        dtype: type to cast the resulting sequence.
+        padding: 'pre' or 'post', pad either before or after each sequence.
+        truncating: 'pre' or 'post', remove values from sequences larger than
+            maxlen either in the beginning or in the end of the sequence
+        value: float, value to pad the sequences to the desired value.
+
+        Returns:
+        x: numpy array with dimensions (number_of_sequences, maxlen)
+
     """
     lengths = [len(s) for s in sequences]
 
@@ -47,39 +60,53 @@ def make_sampling_table(size, sampling_factor=1e-5):
         This generates an array where the ith element
         is the probability that a word of rank i would be sampled,
         according to the sampling distribution used in word2vec.
-        
+
         The word2vec formula is:
             p(word) = min(1, sqrt(word.frequency/sampling_factor) / (word.frequency/sampling_factor))
 
-        We assume that the word frequencies follow Zipf's law (s=1) to derive 
+        We assume that the word frequencies follow Zipf's law (s=1) to derive
         a numerical approximation of frequency(rank):
            frequency(rank) ~ 1/(rank * (log(rank) + gamma) + 1/2 - 1/(12*rank))
         where gamma is the Euler-Mascheroni constant.
+
+        Parameters:
+        -----------
+        size: int, number of possible words to sample. 
     '''
     gamma = 0.577
     rank = np.array(list(range(size)))
     rank[0] = 1
     inv_fq = rank * (np.log(rank) + gamma) + 0.5 - 1./(12.*rank)
     f = sampling_factor * inv_fq
+
     return np.minimum(1., f / np.sqrt(f))
 
 
-def skipgrams(sequence, vocabulary_size, 
-    window_size=4, negative_samples=1., shuffle=True, 
-    categorical=False, sampling_table=None):
-    ''' 
-        Take a sequence (list of indexes of words), 
+def skipgrams(sequence, vocabulary_size,
+              window_size=4, negative_samples=1., shuffle=True,
+              categorical=False, sampling_table=None):
+    '''
+        Take a sequence (list of indexes of words),
         returns couples of [word_index, other_word index] and labels (1s or 0s),
         where label = 1 if 'other_word' belongs to the context of 'word',
         and label=0 if 'other_word' is ramdomly sampled
 
-        @param vocabulary_size: int. maximum possible word index + 1
-        @param window_size: int. actually half-window. The window of a word wi will be [i-window_size, i+window_size+1]
-        @param negative_samples: float >= 0. 0 for no negative (=random) samples. 1 for same number as positive samples. etc.
-        @param categorical: bool. if False, labels will be integers (eg. [0, 1, 1 .. ]), 
+        Paramaters:
+        -----------
+        vocabulary_size: int. maximum possible word index + 1
+        window_size: int. actually half-window. The window of a word wi will be [i-window_size, i+window_size+1]
+        negative_samples: float >= 0. 0 for no negative (=random) samples. 1 for same number as positive samples. etc.
+        categorical: bool. if False, labels will be integers (eg. [0, 1, 1 .. ]),
             if True labels will be categorical eg. [[1,0],[0,1],[0,1] .. ]
 
-        Note: by convention, index 0 in the vocabulary is a non-word and will be skipped.
+        Returns:
+        --------
+        couples, lables: where `couples` are int pairs and
+            `labels` are either 0 or 1.
+
+        Notes:
+        ------
+        By convention, index 0 in the vocabulary is a non-word and will be skipped.
     '''
     couples = []
     labels = []

keras/preprocessing/text.py

@@ -39,7 +39,30 @@ def one_hot(text, n, filters=base_filter(), lower=True, split=" "):
 
 
 class Tokenizer(object):
-    def __init__(self, nb_words=None, filters=base_filter(), lower=True, split=" "):
+    def __init__(self, nb_words=None, filters=base_filter(),
+                 lower=True, split=' '):
+        '''The class allows to vectorize a text corpus, by turning each
+        text into either a sequence of integers (each integer being the index
+        of a token in a dictionary) or into a vector where the coefficient
+        for each token could be binary, based on word count, based on tf-idf...
+
+        # Arguments
+            nb_words: the maximum number of words to keep, based
+                on word frequency. Only the most common `nb_words` words will
+                be kept.
+            filters: a string where each element is a character that will be
+                filtered from the texts. The default is all punctuation, plus
+                tabs and line breaks, minus the `'` character.
+            lower: boolean. Whether to convert the texts to lowercase.
+            split: character or string to use for token splitting.
+
+        By default, all punctuation is removed, turning the texts into
+        space-separated sequences of words
+        (words maybe include the `'` character). These sequences are then
+        split into lists of tokens. They will then be indexed or vectorized.
+
+        `0` is a reserved index that won't be assigned to any word.
+        '''
         self.word_counts = {}
         self.word_docs = {}
         self.filters = filters
@@ -51,7 +74,10 @@ class Tokenizer(object):
     def fit_on_texts(self, texts):
         '''
             required before using texts_to_sequences or texts_to_matrix
-            @param texts: can be a list or a generator (for memory-efficiency)
+
+        # Arguments
+            texts: can be a list of strings,
+                or a generator of strings (for memory-efficiency)
         '''
         self.document_count = 0
         for text in texts:
@@ -141,12 +167,12 @@ class Tokenizer(object):
             if self.word_index:
                 nb_words = len(self.word_index) + 1
             else:
-                raise Exception("Specify a dimension (nb_words argument), or fit on some text data first")
+                raise Exception("Specify a dimension (nb_words argument), or fit on some text data first.")
         else:
             nb_words = self.nb_words
 
         if mode == "tfidf" and not self.document_count:
-            raise Exception("Fit the Tokenizer on some data before using tfidf mode")
+            raise Exception("Fit the Tokenizer on some data before using tfidf mode.")
 
         X = np.zeros((len(sequences), nb_words))
         for i, seq in enumerate(sequences):

keras/utils/generic_utils.py

@@ -5,11 +5,13 @@ import sys
 import six
 
 
-def get_from_module(identifier, module_params, module_name, instantiate=False, kwargs=None):
+def get_from_module(identifier, module_params, module_name,
+                    instantiate=False, kwargs=None):
     if isinstance(identifier, six.string_types):
         res = module_params.get(identifier)
         if not res:
-            raise Exception('Invalid ' + str(module_name) + ': ' + str(identifier))
+            raise Exception('Invalid ' + str(module_name) + ': ' +
+                            str(identifier))
         if instantiate and not kwargs:
             return res()
         elif instantiate and kwargs:
@@ -23,28 +25,6 @@ def make_tuple(*args):
     return args
 
 
-def printv(v, prefix=''):
-    if type(v) == dict:
-        if 'name' in v:
-            print(prefix + '#' + v['name'])
-            del v['name']
-        prefix += '...'
-        for nk, nv in v.items():
-            if type(nv) in [dict, list]:
-                print(prefix + nk + ':')
-                printv(nv, prefix)
-            else:
-                print(prefix + nk + ':' + str(nv))
-    elif type(v) == list:
-        prefix += '...'
-        for i, nv in enumerate(v):
-            print(prefix + '#' + str(i))
-            printv(nv, prefix)
-    else:
-        prefix += '...'
-        print(prefix + str(v))
-
-
 class Progbar(object):
     def __init__(self, target, width=30, verbose=1):
         '''
@@ -107,10 +87,15 @@ class Progbar(object):
             else:
                 info += ' - %ds' % (now - self.start)
             for k in self.unique_values:
+                info += ' - %s:' % k
                 if type(self.sum_values[k]) is list:
-                    info += ' - %s: %.4f' % (k, self.sum_values[k][0] / max(1, self.sum_values[k][1]))
+                    avg = self.sum_values[k][0] / max(1, self.sum_values[k][1])
+                    if abs(avg) > 1e-3:
+                        info += ' %.4f' % avg
+                    else:
+                        info += ' %.4e' % avg
                 else:
-                    info += ' - %s: %s' % (k, self.sum_values[k])
+                    info += ' %s' % self.sum_values[k]
 
             self.total_width += len(info)
             if prev_total_width > self.total_width:
@@ -126,7 +111,12 @@ class Progbar(object):
             if current >= self.target:
                 info = '%ds' % (now - self.start)
                 for k in self.unique_values:
-                    info += ' - %s: %.4f' % (k, self.sum_values[k][0] / max(1, self.sum_values[k][1]))
+                    info += ' - %s:' % k
+                    avg = self.sum_values[k][0] / max(1, self.sum_values[k][1])
+                    if avg > 1e-3:
+                        info += ' %.4f' % avg
+                    else:
+                        info += ' %.4e' % avg
                 sys.stdout.write(info + "\n")
 
     def add(self, n, values=[]):

keras/utils/layer_utils.py

@@ -26,12 +26,14 @@ def container_from_config(original_layer_dict, custom_objects={}):
 
     if name == 'Merge':
         mode = layer_dict.get('mode')
+        concat_axis = layer_dict.get('concat_axis')
+        dot_axes = layer_dict.get('dot_axes')
         layers = layer_dict.get('layers')
         layer_list = []
         for layer in layers:
             init_layer = container_from_config(layer)
             layer_list.append(init_layer)
-        merge_layer = Merge(layer_list, mode)
+        merge_layer = Merge(layer_list, mode, concat_axis, dot_axes)
         return merge_layer
 
     elif name == 'Sequential':
@@ -87,6 +89,65 @@ def container_from_config(original_layer_dict, custom_objects={}):
         return base_layer
 
 
+def model_summary(model):
+    param_count = 0  # param count in the model
+
+    def display(objects, positions):
+        line = ''
+        for i in range(len(objects)):
+            line += str(objects[i])
+            line = line[:positions[i]]
+            line += ' ' * (positions[i] - len(line))
+        print(line)
+
+    def display_layer_info(layer, name, positions):
+        layer_type = layer.__class__.__name__
+        output_shape = layer.output_shape
+        params = layer.count_params()
+        to_display = ['%s (%s)' % (layer_type, name), output_shape, params]
+        display(to_display, positions)
+
+    line_length = 80  # total length of printed lines
+    positions = [30, 60, 80]  # absolute positions of log elements in each line
+    # header names for the different log elements
+    to_display = ['Layer (name)', 'Output Shape', 'Param #']
+
+    # for sequential models, we start by printing
+    # the expect input shape
+    if model.__class__.__name__ == 'Sequential':
+        print('-' * line_length)
+        print('Initial input shape: ' + str(model.input_shape))
+
+    # print header
+    print('-' * line_length)
+    display(to_display, positions)
+    print('-' * line_length)
+
+    if model.__class__.__name__ == 'Sequential':
+        for layer in model.layers:
+            name = getattr(layer, 'name', 'Unnamed')
+            display_layer_info(layer, name, positions)
+            param_count += layer.count_params()
+
+    elif model.__class__.__name__ == 'Graph':
+        for name in model.input_order:
+            layer = model.inputs[name]
+            display_layer_info(layer, name, positions)
+
+        for name in model.nodes:
+            layer = model.nodes[name]
+            display_layer_info(layer, name, positions)
+            param_count += layer.count_params()
+
+        for name in model.output_order:
+            layer = model.outputs[name]
+            display_layer_info(layer, name, positions)
+
+    print('-' * line_length)
+    print('Total params: %s' % param_count)
+    print('-' * line_length)
+
+
 from .generic_utils import get_from_module
 def get_layer(identifier, kwargs=None):
     return get_from_module(identifier, globals(), 'layer',

keras/utils/np_utils.py

@@ -7,7 +7,7 @@ from six.moves import zip
 
 def to_categorical(y, nb_classes=None):
     '''Convert class vector (integers from 0 to nb_classes)
-    to binary class matrix, for use with categorical_crossentropy
+    to binary class matrix, for use with categorical_crossentropy.
     '''
     y = np.asarray(y, dtype='int32')
     if not nb_classes:

keras/utils/test_utils.py

@@ -15,13 +15,13 @@ def get_test_data(nb_train=1000, nb_test=500, input_shape=(10,), output_shape=(2
         y = np.random.randint(0, nb_class, size=(nb_sample, 1))
         X = np.zeros((nb_sample,) + input_shape)
         for i in range(nb_sample):
-            X[i] = np.random.normal(loc=y[i], scale=1.0, size=input_shape)
+            X[i] = np.random.normal(loc=y[i], scale=0.7, size=input_shape)
     else:
         y_loc = np.random.random((nb_sample,))
         X = np.zeros((nb_sample,) + input_shape)
         y = np.zeros((nb_sample,) + output_shape)
         for i in range(nb_sample):
-            X[i] = np.random.normal(loc=y_loc[i], scale=1.0, size=input_shape)
-            y[i] = np.random.normal(loc=y_loc[i], scale=1.0, size=output_shape)
+            X[i] = np.random.normal(loc=y_loc[i], scale=0.7, size=input_shape)
+            y[i] = np.random.normal(loc=y_loc[i], scale=0.7, size=output_shape)
 
     return (X[:nb_train], y[:nb_train]), (X[nb_train:], y[nb_train:])

keras/utils/visualize_util.py

@@ -1,40 +1,153 @@
-import pydot
-# old pydot will not work with python3, must use one
-# that works with python3 such as pydot2 or pydot
-from keras.models import Sequential, Graph
+import itertools
+from keras.layers.containers import Graph, Sequential
+from keras.layers.core import Merge
 
-def to_graph(model):
-    graph = pydot.Dot(graph_type='digraph')
-    if type(model) == Sequential:
-        previous_node = None
-        written_nodes = []
-        n = 1
-        for node in model.get_config()['layers']:
-            # append number in case layers have same name to differentiate
-            if (node['name'] + str(n)) in written_nodes:
-                n += 1
-            current_node = pydot.Node(node['name'] + str(n))
-            written_nodes.append(node['name'] + str(n))
-            graph.add_node(current_node)
-            if previous_node:
-                graph.add_edge(pydot.Edge(previous_node, current_node))
-            previous_node = current_node
-    elif type(model) == Graph:
-        # don't need to append number for names since all nodes labeled
-        for input_node in model.input_config:
-            graph.add_node(pydot.Node(input_node['name']))
+try:
+    # pydot-ng is a fork of pydot that is better maintained
+    import pydot_ng as pydot
+except ImportError:
+    # fall back on pydot if necessary
+    import pydot
+if not pydot.find_graphviz():
+    raise RuntimeError("Failed to import pydot. You must install pydot"
+                       " and graphviz for `pydotprint` to work.")
 
-        # intermediate and output nodes have input defined
-        for layer_config in [model.node_config, model.output_config]:
-            for node in layer_config:
-                graph.add_node(pydot.Node(node['name']))
-                # possible to have multiple 'inputs' vs 1 'input'
-                if node['inputs']:
-                    for e in node['inputs']:
-                        graph.add_edge(pydot.Edge(e, node['name']))
+
+def layer_typename(layer):
+    return type(layer).__module__ + "." + type(layer).__name__
+
+
+def get_layer_to_name(model):
+    """Returns a dict mapping layer to their name in the model"""
+    if not isinstance(model, Graph):
+        return {}
+    else:
+        node_to_name = itertools.chain(
+            model.nodes.items(), model.inputs.items(), model.outputs.items()
+        )
+        return {v: k for k, v in node_to_name}
+
+
+class ModelToDot(object):
+    """
+    This is a helper class which visits a keras model (Sequential or Graph) and
+    returns a pydot.Graph representation.
+
+    This is implemented as a class because we need to maintain various states.
+
+    Use it as ```ModelToDot()(model)```
+
+    Keras models can have an arbitrary number of inputs and outputs. A given
+    layer can have multiple inputs but has a single output. We therefore
+    explore the model by starting at its output and crawling "up" the tree.
+    """
+    def _pydot_node_for_layer(self, layer, label):
+        """
+        Returns the pydot.Node corresponding to the given layer.
+        `label` specify the name of the layer (only used if the layer isn't yet
+            associated with a pydot.Node)
+        """
+        # Check if this already exists (will be the case for nodes that
+        # serve as input to more than one layer)
+        if layer in self.layer_to_pydotnode:
+            node = self.layer_to_pydotnode[layer]
+        else:
+            layer_id = 'layer%d' % self.idgen
+            self.idgen += 1
+
+            label = label + " (" + layer_typename(layer) + ")"
+
+            if self.show_shape:
+                # Build the label that will actually contain a table with the
+                # input/output
+                outputlabels = str(layer.output_shape)
+                if hasattr(layer, 'input_shape'):
+                    inputlabels = str(layer.input_shape)
+                elif hasattr(layer, 'input_shapes'):
+                    inputlabels = ', '.join(
+                        [str(ishape) for ishape in layer.input_shapes])
                 else:
-                    graph.add_edge(pydot.Edge(node['input'], node['name']))
-    return graph
+                    inputlabels = ''
+                label = "%s\n|{input:|output:}|{{%s}|{%s}}" % (
+                        label, inputlabels, outputlabels)
+
+            node = pydot.Node(layer_id, label=label)
+            self.g.add_node(node)
+            self.layer_to_pydotnode[layer] = node
+        return node
+
+    def _process_layer(self, layer, layer_to_name=None, connect_to=None):
+        """
+        Process a layer, adding its node to the graph and creating edges to its
+        outputs.
+
+        `connect_to` specify where the output of the current layer will be
+            connected
+        `layer_to_name` is a dict mapping layer to their name in the Graph
+            model. Should be {} when processing a Sequential model
+        """
+        # The layer can be a container layer, in which case we can recurse
+        is_graph = isinstance(layer, Graph)
+        is_seq = isinstance(layer, Sequential)
+        if self.recursive and (is_graph or is_seq):
+            # We got a container layer, recursively transform it
+            if is_graph:
+                child_layers = layer.outputs.values()
+            else:
+                child_layers = [layer.layers[-1]]
+            for l in child_layers:
+                self._process_layer(l, layer_to_name=get_layer_to_name(layer),
+                                    connect_to=connect_to)
+        else:
+            # This is a simple layer.
+            label = layer_to_name.get(layer, '')
+            layer_node = self._pydot_node_for_layer(layer, label=label)
+
+            if connect_to is not None:
+                self.g.add_edge(pydot.Edge(layer_node, connect_to))
+
+            # Proceed upwards to the parent(s). Only Merge layers have more
+            # than one parent
+            if isinstance(layer, Merge):  # Merge layer
+                for l in layer.layers:
+                    self._process_layer(l, layer_to_name,
+                                        connect_to=layer_node)
+            elif hasattr(layer, 'previous') and layer.previous is not None:
+                self._process_layer(layer.previous, layer_to_name,
+                                    connect_to=layer_node)
+
+    def __call__(self, model, recursive=True, show_shape=False,
+                 connect_to=None):
+        self.idgen = 0
+        # Maps keras layer to the pydot.Node representing them
+        self.layer_to_pydotnode = {}
+        self.recursive = recursive
+        self.show_shape = show_shape
+
+        self.g = pydot.Dot()
+        self.g.set('rankdir', 'TB')
+        self.g.set('concentrate', True)
+        self.g.set_node_defaults(shape='record')
+
+        if hasattr(model, 'outputs'):
+            # Graph
+            for name, l in model.outputs.items():
+                self._process_layer(l, get_layer_to_name(model),
+                                    connect_to=connect_to)
+        else:
+            # Sequential container
+            self._process_layer(model.layers[-1], {}, connect_to=connect_to)
+
+        return self.g
+
+
+def to_graph(model, **kwargs):
+    """
+    `recursive` controls whether we recursively explore container layers
+    `show_shape` controls whether the shape is shown in the graph
+    """
+    return ModelToDot()(model, **kwargs)
+
 
 def plot(model, to_file='model.png'):
     graph = to_graph(model)

pytest.ini

@@ -0,0 +1,7 @@
+# Configuration of py.test
+[pytest]
+addopts=-v
+        -n 2
+        --durations=10
+        --cov-report term-missing
+        --cov=keras

setup.py

@@ -3,12 +3,12 @@ from setuptools import find_packages
 
 
 setup(name='Keras',
-      version='0.2.0',
+      version='0.3.0',
       description='Theano-based Deep Learning library',
       author='Francois Chollet',
       author_email='francois.chollet@gmail.com',
       url='https://github.com/fchollet/keras',
-      download_url='https://github.com/fchollet/keras/tarball/0.2.0',
+      download_url='https://github.com/fchollet/keras/tarball/0.3.0',
       license='MIT',
       install_requires=['theano', 'pyyaml', 'six'],
       extras_require={

tests/integration_tests/test_image_data_tasks.py

@@ -0,0 +1,46 @@
+from __future__ import print_function
+import numpy as np
+import pytest
+
+from keras.utils.test_utils import get_test_data
+from keras.models import Sequential
+from keras.layers.core import Dense, Flatten, Activation
+from keras.layers.convolutional import Convolution2D, MaxPooling2D
+from keras.utils.np_utils import to_categorical
+
+
+def test_image_classification():
+    '''
+    Classify random 16x16 color images into several classes using logistic regression
+    with convolutional hidden layer.
+    '''
+    np.random.seed(1337)
+    input_shape = (3, 16, 16)
+    (X_train, y_train), (X_test, y_test) = get_test_data(nb_train=500,
+                                                         nb_test=200,
+                                                         input_shape=input_shape,
+                                                         classification=True,
+                                                         nb_class=4)
+    y_train = to_categorical(y_train)
+    y_test = to_categorical(y_test)
+    # convolution kernel size
+    nb_conv = 3
+    # size of pooling area for max pooling
+    nb_pool = 2
+
+    model = Sequential([
+        Convolution2D(nb_filter=8, nb_row=nb_conv, nb_col=nb_conv, input_shape=input_shape),
+        MaxPooling2D(pool_size=(nb_pool, nb_pool)),
+        Flatten(),
+        Activation('relu'),
+        Dense(y_test.shape[-1], activation='softmax')
+    ])
+    model.compile(loss='categorical_crossentropy', optimizer='sgd')
+    history = model.fit(X_train, y_train, nb_epoch=10, batch_size=16,
+                        validation_data=(X_test, y_test),
+                        show_accuracy=True, verbose=0)
+    assert(history.history['val_acc'][-1] > 0.85)
+
+
+if __name__ == '__main__':
+    pytest.main([__file__])

tests/integration_tests/test_temporal_data_tasks.py

@@ -0,0 +1,131 @@
+from __future__ import print_function
+import numpy as np
+import pytest
+import string
+
+from keras.utils.test_utils import get_test_data
+from keras.models import Sequential
+from keras.layers.core import TimeDistributedDense, Dropout, Dense
+from keras.layers.recurrent import GRU, LSTM
+from keras.utils.np_utils import to_categorical
+
+
+def test_temporal_classification():
+    '''
+    Classify temporal sequences of float numbers of length 3 into 2 classes using
+    single layer of GRU units and softmax applied to the last activations of the units
+    '''
+    np.random.seed(1337)
+    (X_train, y_train), (X_test, y_test) = get_test_data(nb_train=500,
+                                                         nb_test=200,
+                                                         input_shape=(3, 5),
+                                                         classification=True,
+                                                         nb_class=2)
+    y_train = to_categorical(y_train)
+    y_test = to_categorical(y_test)
+
+    model = Sequential()
+    model.add(GRU(y_train.shape[-1],
+                  input_shape=(X_train.shape[1], X_train.shape[2]),
+                  activation='softmax'))
+    model.compile(loss='categorical_crossentropy', optimizer='adadelta')
+    history = model.fit(X_train, y_train, nb_epoch=5, batch_size=16,
+                        validation_data=(X_test, y_test),
+                        show_accuracy=True, verbose=0)
+    assert(history.history['val_acc'][-1] > 0.9)
+
+
+def test_temporal_regression():
+    '''
+    Predict float numbers (regression) based on sequences of float numbers of length 3 using
+    single layer of GRU units
+    '''
+    np.random.seed(1337)
+    (X_train, y_train), (X_test, y_test) = get_test_data(nb_train=500,
+                                                         nb_test=200,
+                                                         input_shape=(3, 5),
+                                                         output_shape=(2,),
+                                                         classification=False)
+    model = Sequential()
+    model.add(GRU(y_train.shape[-1],
+              input_shape=(X_train.shape[1], X_train.shape[2])))
+    model.compile(loss='hinge', optimizer='adam')
+    history = model.fit(X_train, y_train, nb_epoch=5, batch_size=16,
+                        validation_data=(X_test, y_test), verbose=0)
+    assert(history.history['val_loss'][-1] < 0.75)
+
+
+def test_sequence_to_sequence():
+    '''
+    Apply a same Dense layer for each element of time dimension of the input
+    and make predictions of the output sequence elements.
+    This does not make use of the temporal structure of the sequence
+    (see TimeDistributedDense for more details)
+    '''
+    np.random.seed(1337)
+    (X_train, y_train), (X_test, y_test) = get_test_data(nb_train=500,
+                                                         nb_test=200,
+                                                         input_shape=(3, 5),
+                                                         output_shape=(3, 5),
+                                                         classification=False)
+
+    model = Sequential()
+    model.add(TimeDistributedDense(y_train.shape[-1],
+              input_shape=(X_train.shape[1], X_train.shape[2])))
+    model.compile(loss='hinge', optimizer='rmsprop')
+    history = model.fit(X_train, y_train, nb_epoch=20, batch_size=16,
+                        validation_data=(X_test, y_test), verbose=0)
+    assert(history.history['val_loss'][-1] < 0.8)
+
+
+def test_stacked_lstm_char_prediction():
+    '''
+    Learn alphabetical char sequence with stacked LSTM.
+    Predict the whole alphabet based on the first two letters ('ab' -> 'ab...z')
+    See non-toy example in examples/lstm_text_generation.py
+    '''
+    np.random.seed(1336)
+    # generate alphabet: http://stackoverflow.com/questions/16060899/alphabet-range-python
+    alphabet = string.ascii_lowercase
+    number_of_chars = len(alphabet)
+
+    # generate char sequences of length 'sequence_length' out of alphabet and store the next char as label (e.g. 'ab'->'c')
+    sequence_length = 2
+    sentences = [alphabet[i: i + sequence_length] for i in range(len(alphabet) - sequence_length)]
+    next_chars = [alphabet[i + sequence_length] for i in range(len(alphabet) - sequence_length)]
+
+    # Transform sequences and labels into 'one-hot' encoding
+    X = np.zeros((len(sentences), sequence_length, number_of_chars), dtype=np.bool)
+    y = np.zeros((len(sentences), number_of_chars), dtype=np.bool)
+    for i, sentence in enumerate(sentences):
+        for t, char in enumerate(sentence):
+            X[i, t, ord(char)-ord('a')] = 1
+        y[i, ord(next_chars[i])-ord('a')] = 1
+
+    # learn the alphabet with stacked LSTM
+    model = Sequential([
+        LSTM(16, return_sequences=True, input_shape=(sequence_length, number_of_chars)),
+        LSTM(16, return_sequences=False),
+        Dense(number_of_chars, activation='softmax')
+    ])
+    model.compile(loss='categorical_crossentropy', optimizer='adam')
+    model.fit(X, y, batch_size=1, nb_epoch=60, verbose=1)
+
+    # prime the model with 'ab' sequence and let it generate the learned alphabet
+    sentence = alphabet[:sequence_length]
+    generated = sentence
+    for iteration in range(number_of_chars-sequence_length):
+        x = np.zeros((1, sequence_length, number_of_chars))
+        for t, char in enumerate(sentence):
+            x[0, t, ord(char) - ord('a')] = 1.
+        preds = model.predict(x, verbose=0)[0]
+        next_char = chr(np.argmax(preds) + ord('a'))
+        generated += next_char
+        sentence = sentence[1:] + next_char
+
+    # check that it did generate the alphabet correctly
+    assert(generated == alphabet)
+
+
+if __name__ == '__main__':
+    pytest.main([__file__])

tests/integration_tests/test_vector_data_tasks.py

@@ -0,0 +1,63 @@
+from __future__ import print_function
+import numpy as np
+import pytest
+
+from keras.utils.test_utils import get_test_data
+from keras.models import Sequential
+from keras.layers.core import Dense
+from keras.utils.np_utils import to_categorical
+
+
+def test_vector_classification():
+    '''
+    Classify random float vectors into 2 classes with logistic regression
+    using 2 layer neural network with ReLU hidden units.
+    '''
+    np.random.seed(1337)
+    nb_hidden = 10
+
+    (X_train, y_train), (X_test, y_test) = get_test_data(nb_train=500,
+                                                         nb_test=200,
+                                                         input_shape=(20,),
+                                                         classification=True,
+                                                         nb_class=2)
+    y_train = to_categorical(y_train)
+    y_test = to_categorical(y_test)
+
+    model = Sequential([
+        Dense(nb_hidden, input_shape=(X_train.shape[-1],), activation='relu'),
+        Dense(y_train.shape[-1], activation='softmax')
+    ])
+    model.compile(loss='categorical_crossentropy', optimizer='rmsprop')
+    history = model.fit(X_train, y_train, nb_epoch=15, batch_size=16,
+                        validation_data=(X_test, y_test),
+                        show_accuracy=True, verbose=0)
+    assert(history.history['val_acc'][-1] > 0.8)
+
+
+def test_vector_regression():
+    '''
+    Perform float data prediction (regression) using 2 layer MLP
+    with tanh and sigmoid activations.
+    '''
+    np.random.seed(1337)
+    nb_hidden = 10
+    (X_train, y_train), (X_test, y_test) = get_test_data(nb_train=500,
+                                                         nb_test=200,
+                                                         input_shape=(20,),
+                                                         output_shape=(2,),
+                                                         classification=False)
+
+    model = Sequential([
+        Dense(nb_hidden, input_shape=(X_train.shape[-1],), activation='tanh'),
+        Dense(y_train.shape[-1])
+    ])
+
+    model.compile(loss='hinge', optimizer='adagrad')
+    history = model.fit(X_train, y_train, nb_epoch=20, batch_size=16,
+                        validation_data=(X_test, y_test), verbose=0)
+    assert (history.history['val_loss'][-1] < 0.9)
+
+
+if __name__ == '__main__':
+    pytest.main([__file__])

tests/keras/backend/test_backends.py

@@ -1,12 +1,10 @@
 import sys
-import unittest
+import pytest
 from numpy.testing import assert_allclose
 import numpy as np
-import pytest
 
-if sys.version_info.major == 2:
-    from keras.backend import theano_backend as KTH
-    from keras.backend import tensorflow_backend as KTF
+from keras.backend import theano_backend as KTH
+from keras.backend import tensorflow_backend as KTF
 
 
 def check_single_tensor_operation(function_name, input_shape, **kwargs):
@@ -18,7 +16,7 @@ def check_single_tensor_operation(function_name, input_shape, **kwargs):
     ztf = KTF.eval(getattr(KTF, function_name)(xtf, **kwargs))
 
     assert zth.shape == ztf.shape
-    assert_allclose(zth, ztf, atol=1e-06)
+    assert_allclose(zth, ztf, atol=1e-05)
 
 
 def check_two_tensor_operation(function_name, x_input_shape,
@@ -35,11 +33,10 @@ def check_two_tensor_operation(function_name, x_input_shape,
     ztf = KTF.eval(getattr(KTF, function_name)(xtf, ytf, **kwargs))
 
     assert zth.shape == ztf.shape
-    assert_allclose(zth, ztf, atol=1e-06)
+    assert_allclose(zth, ztf, atol=1e-05)
 
 
-@pytest.mark.skipif(sys.version_info.major != 2, reason="Requires Python 2.7")
-class TestBackend(unittest.TestCase):
+class TestBackend(object):
 
     def test_linear_operations(self):
         check_two_tensor_operation('dot', (4, 2), (2, 4))
@@ -56,7 +53,7 @@ class TestBackend(unittest.TestCase):
         zth = KTH.eval(KTH.concatenate([xth, yth], axis=-1))
         ztf = KTF.eval(KTF.concatenate([xtf, ytf], axis=-1))
         assert zth.shape == ztf.shape
-        assert_allclose(zth, ztf, atol=1e-06)
+        assert_allclose(zth, ztf, atol=1e-05)
 
         check_single_tensor_operation('reshape', (4, 2), shape=(8, 1))
         check_single_tensor_operation('permute_dimensions', (4, 2, 3),
@@ -67,6 +64,26 @@ class TestBackend(unittest.TestCase):
         check_single_tensor_operation('expand_dims', (4, 3, 2), dim=1)
         check_single_tensor_operation('squeeze', (4, 3, 1), axis=2)
 
+    def test_repeat_elements(self):
+        reps = 3
+        for ndims in [1, 2, 3]:
+            shape = np.arange(2, 2+ndims)
+            arr = np.arange(np.prod(shape)).reshape(shape)
+            arr_th = KTH.variable(arr)
+            arr_tf = KTF.variable(arr)
+
+            for rep_axis in range(ndims):
+                np_rep = np.repeat(arr, reps, axis=rep_axis)
+                th_rep = KTH.eval(
+                    KTH.repeat_elements(arr_th, reps, axis=rep_axis))
+                tf_rep = KTF.eval(
+                    KTF.repeat_elements(arr_tf, reps, axis=rep_axis))
+
+                assert th_rep.shape == np_rep.shape
+                assert tf_rep.shape == np_rep.shape
+                assert_allclose(np_rep, th_rep, atol=1e-05)
+                assert_allclose(np_rep, tf_rep, atol=1e-05)
+
     def test_value_manipulation(self):
         val = np.random.random((4, 2))
         xth = KTH.variable(val)
@@ -76,7 +93,7 @@ class TestBackend(unittest.TestCase):
         valth = KTH.get_value(xth)
         valtf = KTF.get_value(xtf)
         assert valtf.shape == valth.shape
-        assert_allclose(valth, valtf, atol=1e-06)
+        assert_allclose(valth, valtf, atol=1e-05)
 
         # set_value
         val = np.random.random((4, 2))
@@ -86,7 +103,7 @@ class TestBackend(unittest.TestCase):
         valth = KTH.get_value(xth)
         valtf = KTF.get_value(xtf)
         assert valtf.shape == valth.shape
-        assert_allclose(valth, valtf, atol=1e-06)
+        assert_allclose(valth, valtf, atol=1e-05)
 
         # count_params
         assert KTH.count_params(xth) == KTF.count_params(xtf)
@@ -149,7 +166,7 @@ class TestBackend(unittest.TestCase):
         zth = KTH.eval(gradth[0])
         ztf = KTF.eval(gradtf[0])
         assert zth.shape == ztf.shape
-        assert_allclose(zth, ztf, atol=1e-06)
+        assert_allclose(zth, ztf, atol=1e-05)
 
     def test_function(self):
         val = np.random.random((4, 2))
@@ -171,12 +188,12 @@ class TestBackend(unittest.TestCase):
         function_outputs_th = fth([input_val])[0]
         function_outputs_tf = ftf([input_val])[0]
         assert function_outputs_th.shape == function_outputs_tf.shape
-        assert_allclose(function_outputs_th, function_outputs_tf, atol=1e-06)
+        assert_allclose(function_outputs_th, function_outputs_tf, atol=1e-05)
 
         new_val_th = KTH.get_value(xth)
         new_val_tf = KTF.get_value(xtf)
         assert new_val_th.shape == new_val_tf.shape
-        assert_allclose(new_val_th, new_val_tf, atol=1e-06)
+        assert_allclose(new_val_th, new_val_tf, atol=1e-05)
 
     def test_rnn(self):
         # implement a simple RNN
@@ -224,9 +241,9 @@ class TestBackend(unittest.TestCase):
         assert len(new_states) == 1
         tf_state = KTF.eval(new_states[0])
 
-        assert_allclose(tf_last_output, th_last_output, atol=1e-06)
-        assert_allclose(tf_outputs, th_outputs, atol=1e-06)
-        assert_allclose(tf_state, th_state, atol=1e-06)
+        assert_allclose(tf_last_output, th_last_output, atol=1e-04)
+        assert_allclose(tf_outputs, th_outputs, atol=1e-04)
+        assert_allclose(tf_state, th_state, atol=1e-04)
 
     def test_switch(self):
         val = np.random.random()
@@ -240,7 +257,7 @@ class TestBackend(unittest.TestCase):
         ztf = KTF.eval(xtf)
 
         assert zth.shape == ztf.shape
-        assert_allclose(zth, ztf, atol=1e-06)
+        assert_allclose(zth, ztf, atol=1e-05)
 
     def test_nn_operations(self):
         check_single_tensor_operation('relu', (4, 2), alpha=0.1, max_value=0.5)
@@ -291,17 +308,17 @@ class TestBackend(unittest.TestCase):
     #     check_two_tensor_operation('conv2d', (5, 3, 10, 12), (4, 3, 3, 3),
     #                                strides=(2, 2), border_mode='valid')
 
-    # def test_maxpool2d(self):
-    #     '''maxpool2d works "properly" with Theano and TF but outputs different
+    # def test_pool2d(self):
+    #     '''pool2d works "properly" with Theano and TF but outputs different
     #     values in each case. Cause unclear (input shape format?)
     #     '''
-    #     check_single_tensor_operation('maxpool2d', (5, 3, 10, 12), pool_size=(2, 2),
+    #     check_single_tensor_operation('pool2d', (5, 3, 10, 12), pool_size=(2, 2),
     #                                   strides=(1, 1), border_mode='valid')
 
-    #     check_single_tensor_operation('maxpool2d', (5, 3, 9, 11), pool_size=(2, 2),
+    #     check_single_tensor_operation('pool2d', (5, 3, 9, 11), pool_size=(2, 2),
     #                                   strides=(1, 1), border_mode='valid')
 
-    #     check_single_tensor_operation('maxpool2d', (5, 3, 9, 11), pool_size=(2, 3),
+    #     check_single_tensor_operation('pool2d', (5, 3, 9, 11), pool_size=(2, 3),
     #                                   strides=(1, 1), border_mode='valid')
 
     def test_random_normal(self):
@@ -332,4 +349,4 @@ class TestBackend(unittest.TestCase):
 
 
 if __name__ == '__main__':
-    unittest.main()
+    pytest.main([__file__])

tests/keras/datasets/test_datasets.py

@@ -0,0 +1,27 @@
+from __future__ import print_function
+import pytest
+from keras.datasets import cifar10, cifar100, reuters, imdb, mnist
+
+
+def test_cifar():
+    (X_train, y_train), (X_test, y_test) = cifar10.load_data()
+    (X_train, y_train), (X_test, y_test) = cifar100.load_data('fine')
+    (X_train, y_train), (X_test, y_test) = cifar100.load_data('coarse')
+
+
+def test_reuters():
+    (X_train, y_train), (X_test, y_test) = reuters.load_data()
+    (X_train, y_train), (X_test, y_test) = reuters.load_data(maxlen=10)
+
+
+def test_mnist():
+    (X_train, y_train), (X_test, y_test) = mnist.load_data()
+
+
+def test_imdb():
+    (X_train, y_train), (X_test, y_test) = imdb.load_data()
+    (X_train, y_train), (X_test, y_test) = imdb.load_data(maxlen=40)
+
+
+if __name__ == '__main__':
+    pytest.main([__file__])

tests/keras/layers/test_advanced_activations.py

@@ -0,0 +1,163 @@
+import pytest
+from numpy.testing import assert_allclose
+import numpy as np
+from keras import backend as K
+
+
+def get_standard_values():
+    '''
+    These are just a set of floats used for testing the activation
+    functions, and are useful in multiple tests.
+
+    The values should all be non-negative because they and their negatives
+    are used to test the ReLU derivates in this file.
+    '''
+    return np.array([[0, 0.1, 0.5, 0.9, 1.0, 10, 1e2, 0.01]], dtype=K.floatx())
+
+
+def test_leaky_relu():
+    np.random.seed(1337)
+    from keras.layers.advanced_activations import LeakyReLU
+    inp = get_standard_values()
+    for alpha in [0., .5, -1.]:
+        layer = LeakyReLU(alpha=alpha)
+        layer.input = K.variable(inp)
+        for train in [True, False]:
+            outp = K.eval(layer.get_output(train))
+            assert_allclose(outp, inp)
+
+        layer.input = K.variable(-inp)
+        for train in [True, False]:
+            outp = K.eval(layer.get_output(train))
+            assert_allclose(outp, -inp*alpha)
+
+        config = layer.get_config()
+        assert config['alpha'] == alpha
+
+
+def test_prelu():
+    from keras.layers.advanced_activations import PReLU
+    np.random.seed(1337)
+    inp = get_standard_values()
+
+    for train in [True, False]:
+        # test with custom weights
+        alphas = np.random.random(inp.shape)
+        layer = PReLU(weights=alphas, input_shape=inp.flatten().shape)
+        # calling build here causes an error, unclear if this is a bug
+        # layer.build()
+
+        layer.input = K.variable(inp)
+        outp = K.eval(layer.get_output(train))
+        assert_allclose(inp, outp)
+
+        layer.input = K.variable(-inp)
+        outp = K.eval(layer.get_output(train))
+        assert_allclose(-alphas*inp, outp)
+
+        # test with default weights
+        layer = PReLU(input_shape=inp.flatten().shape)
+        # layer.build()
+        layer.input = K.variable(inp)
+        outp = K.eval(layer.get_output(train))
+        assert_allclose(inp, outp)
+
+        layer.input = K.variable(-inp)
+        outp = K.eval(layer.get_output(train))
+
+        assert_allclose(0., alphas*outp)
+
+        layer.get_config()
+
+
+def test_elu():
+    from keras.layers.advanced_activations import ELU
+    np.random.seed(1337)
+    inp = get_standard_values()
+    for alpha in [0.1, .5, -1., 1.]:
+        layer = ELU(alpha=alpha)
+        layer.input = K.variable(inp)
+        for train in [True, False]:
+            outp = K.eval(layer.get_output(train))
+            assert_allclose(outp, inp, rtol=1e-3)
+
+        layer.input = K.variable(-inp)
+        for train in [True, False]:
+            outp = K.eval(layer.get_output(train))
+            assert_allclose(outp, alpha*(np.exp(-inp)-1.), rtol=1e-3)
+
+        config = layer.get_config()
+        assert config['alpha'] == alpha
+
+
+@pytest.mark.skipif(K._BACKEND == 'tensorflow',
+                    reason='currently not working with TensorFlow')
+def test_parametric_softplus():
+    from keras.layers.advanced_activations import ParametricSoftplus
+    np.random.seed(1337)
+    inp = np.vstack((get_standard_values(), -get_standard_values()))
+    # large values cause overflow in exp
+    inp = inp[:-2]
+    for alpha in [.5, -1., 1., 5]:
+        for beta in [.5, -1., 1., 2]:
+            layer = ParametricSoftplus(alpha_init=alpha,
+                                       beta_init=beta,
+                                       input_shape=inp.shape)
+            layer.input = K.variable(inp)
+            layer.build()
+            for train in [True, False]:
+                outp = K.eval(layer.get_output(train))
+                assert_allclose(outp, alpha*np.log(1.+np.exp(beta*inp)),
+                                atol=1e-3)
+
+            config = layer.get_config()
+            assert config['alpha_init'] == alpha
+            assert config['beta_init'] == beta
+
+
+@pytest.mark.skipif(K._BACKEND == 'tensorflow',
+                    reason='currently not working with TensorFlow')
+def test_thresholded_linear():
+    from keras.layers.advanced_activations import ThresholdedLinear
+    np.random.seed(1337)
+    inp = get_standard_values()
+    for theta in [0., .5, 1.]:
+        layer = ThresholdedLinear(theta=theta)
+        layer.input = K.variable(inp)
+        for train in [True, False]:
+            outp = K.eval(layer.get_output(train))
+            assert_allclose(outp, inp*(np.abs(inp) >= theta))
+
+        layer.input = K.variable(-inp)
+        for train in [True, False]:
+            outp = K.eval(layer.get_output(train))
+            assert_allclose(outp, -inp*(np.abs(inp) >= theta))
+
+        config = layer.get_config()
+        assert config['theta'] == theta
+
+
+@pytest.mark.skipif(K._BACKEND == 'tensorflow',
+                    reason='currently not working with TensorFlow')
+def test_thresholded_relu():
+    from keras.layers.advanced_activations import ThresholdedReLU
+    np.random.seed(1337)
+    inp = get_standard_values()
+    for theta in [-1, 0., .5, 1.]:
+        layer = ThresholdedReLU(theta=theta)
+        layer.input = K.variable(inp)
+        for train in [True, False]:
+            outp = K.eval(layer.get_output(train))
+            assert_allclose(outp, inp*(inp > theta))
+
+        layer.input = K.variable(-inp)
+        for train in [True, False]:
+            outp = K.eval(layer.get_output(train))
+            assert_allclose(outp, -inp*(-inp > theta))
+
+        config = layer.get_config()
+        assert config['theta'] == theta
+
+
+if __name__ == '__main__':
+    pytest.main([__file__])

tests/keras/layers/test_call.py

@@ -0,0 +1,60 @@
+"""Test keras.layers.core.Layer.__call__"""
+
+import pytest
+import numpy as np
+from numpy.testing import assert_allclose
+
+from keras import backend as K
+from keras.layers.core import Dense
+from keras.models import Sequential
+
+
+def test_layer_call():
+    """Test keras.layers.core.Layer.__call__"""
+    nb_samples, input_dim, output_dim = 3, 10, 5
+    layer = Dense(output_dim, input_dim=input_dim)
+    W = np.asarray(K.eval(layer.W)).astype(K.floatx())
+    X = K.placeholder(ndim=2)
+    Y = layer(X)
+    f = K.function([X], [Y])
+
+    x = np.ones((nb_samples, input_dim)).astype(K.floatx())
+    y = f([x])[0].astype(K.floatx())
+    t = np.dot(x, W).astype(K.floatx())
+    assert_allclose(t, y, rtol=.2)
+
+
+def test_sequential_call():
+    """Test keras.models.Sequential.__call__"""
+    nb_samples, input_dim, output_dim = 3, 10, 5
+    model = Sequential()
+    model.add(Dense(output_dim=output_dim, input_dim=input_dim))
+    model.compile('sgd', 'mse')
+
+    # test flat model
+    X = K.placeholder(ndim=2)
+    Y = model(X)
+    f = K.function([X], [Y])
+
+    x = np.ones((nb_samples, input_dim)).astype(K.floatx())
+    y1 = f([x])[0].astype(K.floatx())
+    y2 = model.predict(x)
+    # results of __call__ should match model.predict
+    assert_allclose(y1, y2)
+
+    # test nested model
+    model2 = Sequential()
+    model2.add(model)
+    model2.compile('sgd', 'mse')
+
+    Y2 = model2(X)
+    f = K.function([X], [Y2])
+
+    y1 = f([x])[0].astype(K.floatx())
+    y2 = model2.predict(x)
+    # results of __call__ should match model.predict
+    assert_allclose(y1, y2)
+
+
+if __name__ == '__main__':
+    pytest.main([__file__])

tests/keras/layers/test_convolutional.py

@@ -1,4 +1,4 @@
-import unittest
+import pytest
 import numpy as np
 from numpy.testing import assert_allclose
 
@@ -6,161 +6,200 @@ from keras import backend as K
 from keras.layers import convolutional
 
 
-class TestConvolutions(unittest.TestCase):
-    def test_convolution_1d(self):
-        nb_samples = 9
-        nb_steps = 7
-        input_dim = 10
-        filter_length = 6
-        nb_filter = 5
+def test_convolution_1d():
+    nb_samples = 9
+    nb_steps = 7
+    input_dim = 10
+    filter_length = 6
+    nb_filter = 5
 
-        weights_in = [np.ones((nb_filter, input_dim, filter_length, 1)),
-                      np.ones(nb_filter)]
+    weights_in = [np.ones((nb_filter, input_dim, filter_length, 1)),
+                  np.ones(nb_filter)]
 
-        input = np.ones((nb_samples, nb_steps, input_dim))
-        for weight in [None, weights_in]:
-            for border_mode in ['valid', 'same']:
-                for subsample_length in [1]:
-                    if border_mode == 'same' and subsample_length != 1:
-                        continue
-                    for W_regularizer in [None, 'l2']:
-                        for b_regularizer in [None, 'l2']:
-                            for act_regularizer in [None, 'l2']:
-                                layer = convolutional.Convolution1D(
-                                    nb_filter, filter_length,
-                                    weights=weight,
-                                    border_mode=border_mode,
-                                    W_regularizer=W_regularizer,
-                                    b_regularizer=b_regularizer,
-                                    activity_regularizer=act_regularizer,
-                                    subsample_length=subsample_length,
-                                    input_shape=(None, input_dim))
+    input = np.ones((nb_samples, nb_steps, input_dim))
+    for weight in [None, weights_in]:
+        for border_mode in ['valid', 'same']:
+            for subsample_length in [1]:
+                if border_mode == 'same' and subsample_length != 1:
+                    continue
+                for W_regularizer in [None, 'l2']:
+                    for b_regularizer in [None, 'l2']:
+                        for act_regularizer in [None, 'l2']:
+                            layer = convolutional.Convolution1D(
+                                nb_filter, filter_length,
+                                weights=weight,
+                                border_mode=border_mode,
+                                W_regularizer=W_regularizer,
+                                b_regularizer=b_regularizer,
+                                activity_regularizer=act_regularizer,
+                                subsample_length=subsample_length,
+                                input_shape=(None, input_dim))
+
+                        layer.input = K.variable(input)
+                        for train in [True, False]:
+                            out = K.eval(layer.get_output(train))
+                            assert input.shape[0] == out.shape[0]
+                            if border_mode == 'same' and subsample_length == 1:
+                                assert input.shape[1] == out.shape[1]
+                        layer.get_config()
+
+
+def test_maxpooling_1d():
+    nb_samples = 9
+    nb_steps = 7
+    input_dim = 10
+
+    input = np.ones((nb_samples, nb_steps, input_dim))
+    for stride in [1, 2]:
+        layer = convolutional.MaxPooling1D(stride=stride,
+                                           border_mode='valid')
+        layer.input = K.variable(input)
+        for train in [True, False]:
+            K.eval(layer.get_output(train))
+        layer.get_config()
+
+
+def test_averagepooling_1d():
+    nb_samples = 9
+    nb_steps = 7
+    input_dim = 10
+
+    input = np.ones((nb_samples, nb_steps, input_dim))
+    for stride in [1, 2]:
+        layer = convolutional.AveragePooling1D(stride=stride,
+                                               border_mode='valid')
+        layer.input = K.variable(input)
+        for train in [True, False]:
+            K.eval(layer.get_output(train))
+        layer.get_config()
+
+
+def test_convolution_2d():
+    nb_samples = 8
+    nb_filter = 9
+    stack_size = 7
+    nb_row = 10
+    nb_col = 6
+
+    input_nb_row = 11
+    input_nb_col = 12
+
+    weights_in = [np.ones((nb_filter, stack_size, nb_row, nb_col)), np.ones(nb_filter)]
+
+    input = np.ones((nb_samples, stack_size, input_nb_row, input_nb_col))
+    for weight in [None, weights_in]:
+        for border_mode in ['valid', 'same']:
+            for subsample in [(1, 1), (2, 2)]:
+                if border_mode == 'same' and subsample != (1, 1):
+                    continue
+                for W_regularizer in [None, 'l2']:
+                    for b_regularizer in [None, 'l2']:
+                        for act_regularizer in [None, 'l2']:
+                            layer = convolutional.Convolution2D(
+                                nb_filter, nb_row, nb_col,
+                                weights=weight,
+                                border_mode=border_mode,
+                                W_regularizer=W_regularizer,
+                                b_regularizer=b_regularizer,
+                                activity_regularizer=act_regularizer,
+                                subsample=subsample,
+                                input_shape=(stack_size, None, None))
 
                             layer.input = K.variable(input)
                             for train in [True, False]:
                                 out = K.eval(layer.get_output(train))
-                                assert input.shape[0] == out.shape[0]
-                                if border_mode == 'same' and subsample_length == 1:
-                                    assert input.shape[1] == out.shape[1]
+                                if border_mode == 'same' and subsample == (1, 1):
+                                    assert out.shape[2:] == input.shape[2:]
                             layer.get_config()
 
-    def test_maxpooling_1d(self):
-        nb_samples = 9
-        nb_steps = 7
-        input_dim = 10
 
-        input = np.ones((nb_samples, nb_steps, input_dim))
-        for stride in [1, 2]:
-            layer = convolutional.MaxPooling1D(stride=stride,
-                                               border_mode='valid')
-            layer.input = K.variable(input)
-            for train in [True, False]:
-                K.eval(layer.get_output(train))
-            layer.get_config()
+def test_maxpooling_2d():
+    nb_samples = 9
+    stack_size = 7
+    input_nb_row = 11
+    input_nb_col = 12
+    pool_size = (3, 3)
 
-    def test_convolution_2d(self):
-        nb_samples = 8
-        nb_filter = 9
-        stack_size = 7
-        nb_row = 10
-        nb_col = 6
+    input = np.ones((nb_samples, stack_size, input_nb_row, input_nb_col))
+    for strides in [(1, 1), (2, 2)]:
+        layer = convolutional.MaxPooling2D(strides=strides,
+                                           border_mode='valid',
+                                           pool_size=pool_size)
+        layer.input = K.variable(input)
+        for train in [True, False]:
+            K.eval(layer.get_output(train))
+        layer.get_config()
 
-        input_nb_row = 11
-        input_nb_col = 12
 
-        weights_in = [np.ones((nb_filter, stack_size, nb_row, nb_col)), np.ones(nb_filter)]
+def test_averagepooling_2d():
+    nb_samples = 9
+    stack_size = 7
+    input_nb_row = 11
+    input_nb_col = 12
+    pool_size = (3, 3)
 
-        input = np.ones((nb_samples, stack_size, input_nb_row, input_nb_col))
-        for weight in [None, weights_in]:
-            for border_mode in ['valid', 'same']:
-                for subsample in [(1, 1), (2, 2)]:
-                    if border_mode == 'same' and subsample != (1, 1):
-                        continue
-                    for W_regularizer in [None, 'l2']:
-                        for b_regularizer in [None, 'l2']:
-                            for act_regularizer in [None, 'l2']:
-                                layer = convolutional.Convolution2D(
-                                    nb_filter, nb_row, nb_col,
-                                    weights=weight,
-                                    border_mode=border_mode,
-                                    W_regularizer=W_regularizer,
-                                    b_regularizer=b_regularizer,
-                                    activity_regularizer=act_regularizer,
-                                    subsample=subsample,
-                                    input_shape=(stack_size, None, None))
-
-                                layer.input = K.variable(input)
-                                for train in [True, False]:
-                                    out = K.eval(layer.get_output(train))
-                                    if border_mode == 'same' and subsample == (1, 1):
-                                        assert out.shape[2:] == input.shape[2:]
-                                layer.get_config()
-
-    def test_maxpooling_2d(self):
-        nb_samples = 9
-        stack_size = 7
-        input_nb_row = 11
-        input_nb_col = 12
-        pool_size = (3, 3)
-
-        input = np.ones((nb_samples, stack_size, input_nb_row, input_nb_col))
-        for strides in [(1, 1), (2, 2)]:
-            layer = convolutional.MaxPooling2D(strides=strides,
+    input = np.ones((nb_samples, stack_size, input_nb_row, input_nb_col))
+    for strides in [(1, 1), (2, 2)]:
+        layer = convolutional.AveragePooling2D(strides=strides,
                                                border_mode='valid',
                                                pool_size=pool_size)
-            layer.input = K.variable(input)
-            for train in [True, False]:
-                K.eval(layer.get_output(train))
-            layer.get_config()
+        layer.input = K.variable(input)
+        for train in [True, False]:
+            K.eval(layer.get_output(train))
+        layer.get_config()
 
-    def test_zero_padding_2d(self):
-        nb_samples = 9
-        stack_size = 7
-        input_nb_row = 11
-        input_nb_col = 12
 
-        input = np.ones((nb_samples, stack_size, input_nb_row, input_nb_col))
-        layer = convolutional.ZeroPadding2D(padding=(2, 2))
+def test_zero_padding_2d():
+    nb_samples = 9
+    stack_size = 7
+    input_nb_row = 11
+    input_nb_col = 12
+
+    input = np.ones((nb_samples, stack_size, input_nb_row, input_nb_col))
+    layer = convolutional.ZeroPadding2D(padding=(2, 2))
+    layer.input = K.variable(input)
+    for train in [True, False]:
+        out = K.eval(layer.get_output(train))
+        for offset in [0, 1, -1, -2]:
+            assert_allclose(out[:, :, offset, :], 0.)
+            assert_allclose(out[:, :, :, offset], 0.)
+        assert_allclose(out[:, :, 2:-2, 2:-2], 1.)
+    layer.get_config()
+
+
+def test_upsampling_1d():
+    nb_samples = 9
+    nb_steps = 7
+    input_dim = 10
+
+    input = np.ones((nb_samples, nb_steps, input_dim))
+    for length in [2, 3, 9]:
+        layer = convolutional.UpSampling1D(length=length)
         layer.input = K.variable(input)
         for train in [True, False]:
             out = K.eval(layer.get_output(train))
-            for offset in [0, 1, -1, -2]:
-                assert_allclose(out[:, :, offset, :], 0.)
-                assert_allclose(out[:, :, :, offset], 0.)
-            assert_allclose(out[:, :, 2:-2, 2:-2], 1.)
+            assert out.shape[1] == length * nb_steps
         layer.get_config()
 
-    def test_upsampling_1d(self):
-        nb_samples = 9
-        nb_steps = 7
-        input_dim = 10
 
-        input = np.ones((nb_samples, nb_steps, input_dim))
-        for length in [2, 3, 9]:
-            layer = convolutional.UpSampling1D(length=length)
+def test_upsampling_2d():
+    nb_samples = 9
+    stack_size = 7
+    input_nb_row = 11
+    input_nb_col = 12
+
+    input = np.ones((nb_samples, stack_size, input_nb_row, input_nb_col))
+
+    for length_row in [2, 3, 9]:
+        for length_col in [2, 3, 9]:
+            layer = convolutional.UpSampling2D(size=(length_row, length_col))
             layer.input = K.variable(input)
             for train in [True, False]:
                 out = K.eval(layer.get_output(train))
-                assert out.shape[1] == length * nb_steps
-            layer.get_config()
+                assert out.shape[2] == length_row * input_nb_row
+                assert out.shape[3] == length_col * input_nb_col
+        layer.get_config()
 
-    def test_upsampling_2d(self):
-        nb_samples = 9
-        stack_size = 7
-        input_nb_row = 11
-        input_nb_col = 12
-
-        input = np.ones((nb_samples, stack_size, input_nb_row, input_nb_col))
-
-        for length_row in [2, 3, 9]:
-            for length_col in [2, 3, 9]:
-                layer = convolutional.UpSampling2D(size=(length_row, length_col))
-                layer.input = K.variable(input)
-                for train in [True, False]:
-                    out = K.eval(layer.get_output(train))
-                    assert out.shape[2] == length_row * input_nb_row
-                    assert out.shape[3] == length_col * input_nb_col
-            layer.get_config()
 
 if __name__ == '__main__':
-    unittest.main()
+    pytest.main([__file__])

tests/keras/layers/test_core.py

@@ -1,166 +1,220 @@
-import unittest
+import pytest
 import numpy as np
+from keras.models import Sequential
 from numpy.testing import assert_allclose
 
 from keras import backend as K
 from keras.layers import core
 
 
-class TestLayerBase(unittest.TestCase):
-    def test_input_output(self):
-        nb_samples = 10
-        input_dim = 5
-        layer = core.Layer()
+def test_input_output():
+    nb_samples = 10
+    input_dim = 5
+    layer = core.Layer()
 
-        # Once an input is provided, it should be reachable through the
-        # appropriate getters
-        input = np.ones((nb_samples, input_dim))
-        layer.input = K.variable(input)
-        for train in [True, False]:
-            assert_allclose(K.eval(layer.get_input(train)), input)
-            assert_allclose(K.eval(layer.get_output(train)), input)
-
-    def test_connections(self):
-        nb_samples = 10
-        input_dim = 5
-        layer1 = core.Layer()
-        layer2 = core.Layer()
-
-        input = np.ones((nb_samples, input_dim))
-        layer1.input = K.variable(input)
-
-        # After connecting, input of layer1 should be passed through
-        layer2.set_previous(layer1)
-        for train in [True, False]:
-            assert_allclose(K.eval(layer2.get_input(train)), input)
-            assert_allclose(K.eval(layer2.get_output(train)), input)
+    # Once an input is provided, it should be reachable through the
+    # appropriate getters
+    input = np.ones((nb_samples, input_dim))
+    layer.input = K.variable(input)
+    for train in [True, False]:
+        assert_allclose(K.eval(layer.get_input(train)), input)
+        assert_allclose(K.eval(layer.get_output(train)), input)
 
 
-class TestConfigParams(unittest.TestCase):
-    """
-    Test the constructor, config and params functions of all layers in core.
-    """
+def test_connections():
+    nb_samples = 10
+    input_dim = 5
+    layer1 = core.Layer()
+    layer2 = core.Layer()
 
-    def _runner(self, layer):
-        conf = layer.get_config()
-        assert (type(conf) == dict)
+    input = np.ones((nb_samples, input_dim))
+    layer1.input = K.variable(input)
 
-        param = layer.get_params()
-        # Typically a list or a tuple, but may be any iterable
-        assert hasattr(param, '__iter__')
-
-    def test_base(self):
-        layer = core.Layer()
-        self._runner(layer)
-
-    def test_masked(self):
-        layer = core.MaskedLayer()
-        self._runner(layer)
-
-    def test_merge(self):
-        layer_1 = core.Layer()
-        layer_2 = core.Layer()
-        layer_1.set_input_shape((None,))
-        layer_2.set_input_shape((None,))
-        layer = core.Merge([layer_1, layer_2])
-        self._runner(layer)
-
-    def test_dropout(self):
-        layer = core.Dropout(0.5)
-        self._runner(layer)
-
-    def test_activation(self):
-        layer = core.Activation('linear')
-        self._runner(layer)
-
-    def test_reshape(self):
-        layer = core.Reshape(dims=(10, 10))
-        self._runner(layer)
-
-    def test_flatten(self):
-        layer = core.Flatten()
-        self._runner(layer)
-
-    def test_repeat_vector(self):
-        layer = core.RepeatVector(10)
-        self._runner(layer)
-
-    def test_dense(self):
-        layer = core.Dense(10, input_shape=(10,))
-        self._runner(layer)
-
-    def test_act_reg(self):
-        layer = core.ActivityRegularization(0.5, 0.5)
-        self._runner(layer)
-
-    def test_time_dist_dense(self):
-        layer = core.TimeDistributedDense(10, input_shape=(None, 10))
-        self._runner(layer)
-
-    def test_time_dist_merge(self):
-        layer = core.TimeDistributedMerge()
-        self._runner(layer)
-
-    def test_autoencoder(self):
-        layer_1 = core.Layer()
-        layer_2 = core.Layer()
-
-        layer = core.AutoEncoder(layer_1, layer_2)
-        self._runner(layer)
-
-    def test_maxout_dense(self):
-        layer = core.MaxoutDense(10, 10)
-        self._runner(layer)
+    # After connecting, input of layer1 should be passed through
+    layer2.set_previous(layer1)
+    for train in [True, False]:
+        assert_allclose(K.eval(layer2.get_input(train)), input)
+        assert_allclose(K.eval(layer2.get_output(train)), input)
 
 
-class TestMasking(unittest.TestCase):
-    """Test the Masking class"""
+def test_base():
+    layer = core.Layer()
+    _runner(layer)
 
-    def test_sequences(self):
-        """Test masking sequences with zeroes as padding"""
-        if K._BACKEND == "tensorflow":
-            return
-        # integer inputs, one per timestep, like embeddings
-        layer = core.Masking()
-        func = K.function([layer.input], [layer.get_output_mask()])
-        input_data = np.array([[[1], [2], [3], [0]],
-                              [[0], [4], [5], [0]]], dtype=np.int32)
 
-        # This is the expected output mask, one dimension less
-        expected = np.array([[1, 1, 1, 0], [0, 1, 1, 0]])
+def test_masked():
+    layer = core.MaskedLayer()
+    _runner(layer)
 
-        # get mask for this input
-        output = func([input_data])[0]
-        self.assertTrue(np.all(output == expected))
 
-    def test_non_zero(self):
-        """Test masking with non-zero mask value"""
-        if K._BACKEND == "tensorflow":
-            return
-        layer = core.Masking(5)
-        func = K.function([layer.input], [layer.get_output_mask()])
-        input_data = np.array([[[1, 1], [2, 1], [3, 1], [5, 5]],
-                              [[1, 5], [5, 0], [0, 0], [0, 0]]],
-                              dtype=np.int32)
-        output = func([input_data])[0]
-        expected = np.array([[1, 1, 1, 0], [1, 1, 1, 1]])
-        self.assertTrue(np.all(output == expected))
+def test_merge():
+    layer_1 = core.Layer()
+    layer_2 = core.Layer()
+    layer_1.set_input_shape((None,))
+    layer_2.set_input_shape((None,))
+    layer = core.Merge([layer_1, layer_2])
+    _runner(layer)
 
-    def test_non_zero_output(self):
-        """Test output of masking layer with non-zero mask value"""
-        if K._BACKEND == "tensorflow":
-            return
-        layer = core.Masking(5)
-        func = K.function([layer.input], [layer.get_output()])
 
-        input_data = np.array([[[1, 1], [2, 1], [3, 1], [5, 5]],
-                              [[1, 5], [5, 0], [0, 0], [0, 0]]],
-                              dtype=np.int32)
-        output = func([input_data])[0]
-        expected = np.array([[[1, 1], [2, 1], [3, 1], [0, 0]],
-                            [[1, 5], [5, 0], [0, 0], [0, 0]]])
-        self.assertTrue(np.all(output == expected))
+def test_dropout():
+    layer = core.Dropout(0.5)
+    _runner(layer)
 
 
+def test_activation():
+    layer = core.Activation('linear')
+    _runner(layer)
+
+
+def test_reshape():
+    layer = core.Reshape(dims=(10, 10))
+    _runner(layer)
+
+
+def test_flatten():
+    layer = core.Flatten()
+    _runner(layer)
+
+
+def test_repeat_vector():
+    layer = core.RepeatVector(10)
+    _runner(layer)
+
+
+def test_dense():
+    layer = core.Dense(10, input_shape=(10,))
+    _runner(layer)
+
+
+def test_act_reg():
+    layer = core.ActivityRegularization(0.5, 0.5)
+    _runner(layer)
+
+
+def test_time_dist_dense():
+    layer = core.TimeDistributedDense(10, input_shape=(None, 10))
+    _runner(layer)
+
+
+def test_time_dist_merge():
+    layer = core.TimeDistributedMerge()
+    _runner(layer)
+
+
+def test_highway():
+    layer = core.Highway(input_shape=(10,))
+    _runner(layer)
+
+
+def test_autoencoder():
+    layer_1 = core.Layer()
+    layer_2 = core.Layer()
+
+    layer = core.AutoEncoder(layer_1, layer_2)
+    _runner(layer)
+
+
+def test_autoencoder_second_layer():
+    # regression test for issue #1275
+    encoder = core.Dense(input_dim=10, output_dim=2)
+    decoder = core.Dense(input_dim=2, output_dim=10)
+    model = Sequential()
+    model.add(core.Dense(input_dim=20, output_dim=10))
+    model.add(core.AutoEncoder(encoder=encoder, decoder=decoder,
+                               output_reconstruction=False))
+    model.compile(loss='mse', optimizer='sgd')
+
+
+def test_maxout_dense():
+    layer = core.MaxoutDense(10, 10, input_shape=(20,))
+    _runner(layer)
+
+
+@pytest.mark.skipif(K._BACKEND == 'tensorflow',
+                    reason='currently not working with TensorFlow')
+def test_sequences():
+    '''Test masking sequences with zeroes as padding'''
+    # integer inputs, one per timestep, like embeddings
+    layer = core.Masking()
+    func = K.function([layer.input], [layer.get_output_mask()])
+    input_data = np.array([[[1], [2], [3], [0]],
+                          [[0], [4], [5], [0]]], dtype=np.int32)
+
+    # This is the expected output mask, one dimension less
+    expected = np.array([[1, 1, 1, 0], [0, 1, 1, 0]])
+
+    # get mask for this input
+    output = func([input_data])[0]
+    assert np.all(output == expected), 'Output not as expected'
+
+
+@pytest.mark.skipif(K._BACKEND == 'tensorflow',
+                    reason='currently not working with TensorFlow')
+def test_non_zero():
+    '''Test masking with non-zero mask value'''
+    layer = core.Masking(5)
+    func = K.function([layer.input], [layer.get_output_mask()])
+    input_data = np.array([[[1, 1], [2, 1], [3, 1], [5, 5]],
+                          [[1, 5], [5, 0], [0, 0], [0, 0]]],
+                          dtype=np.int32)
+    output = func([input_data])[0]
+    expected = np.array([[1, 1, 1, 0], [1, 1, 1, 1]])
+    assert np.all(output == expected), 'Output not as expected'
+
+
+@pytest.mark.skipif(K._BACKEND == 'tensorflow',
+                    reason='currently not working with TensorFlow')
+def test_non_zero_output():
+    '''Test output of masking layer with non-zero mask value'''
+    layer = core.Masking(5)
+    func = K.function([layer.input], [layer.get_output()])
+
+    input_data = np.array([[[1, 1], [2, 1], [3, 1], [5, 5]],
+                          [[1, 5], [5, 0], [0, 0], [0, 0]]],
+                          dtype=np.int32)
+    output = func([input_data])[0]
+    expected = np.array([[[1, 1], [2, 1], [3, 1], [0, 0]],
+                        [[1, 5], [5, 0], [0, 0], [0, 0]]])
+    assert np.all(output == expected), 'Output not as expected'
+
+
+def _runner(layer):
+    assert isinstance(layer, core.Layer)
+    layer.build()
+    conf = layer.get_config()
+    assert (type(conf) == dict)
+
+    param = layer.get_params()
+    # Typically a list or a tuple, but may be any iterable
+    assert hasattr(param, '__iter__')
+
+    # Test the setter for the trainable attribute
+    layer.trainable = True
+    layer.trainable = False
+
+def test_siamese_all():
+    right_input_layer = core.Dense(7, input_dim=3)
+    left_input_layer = core.Dense(7, input_dim=3)
+
+    shared_layer = core.Dense(5,input_dim=7)
+    for mode in ['sum', 'mul', 'ave', 'concat']:
+        siamese_layer = core.Siamese(shared_layer, [left_input_layer, right_input_layer], merge_mode=mode)
+        siamese_layer.output_shape
+        siamese_layer.get_output()
+
+@pytest.mark.skipif(K._BACKEND == 'tensorflow',
+                    reason='currently not working with TensorFlow')
+def test_siamese_theano_only():
+    right_input_layer = core.Dense(7, input_dim=3)
+    left_input_layer = core.Dense(7, input_dim=3)
+
+    shared_layer = core.Dense(5,input_dim=7)
+
+    for mode in ['dot', 'cos']:
+        siamese_layer = core.Siamese(shared_layer, [left_input_layer, right_input_layer], merge_mode=mode,
+                                     dot_axes=([1], [1]))
+        siamese_layer.output_shape
+        siamese_layer.get_output()
+
 if __name__ == '__main__':
-    unittest.main()
+    pytest.main([__file__])

tests/keras/layers/test_embeddings.py

@@ -0,0 +1,31 @@
+import pytest
+import numpy as np
+from numpy.testing import assert_allclose
+from keras.models import Sequential
+from keras.layers.core import Dense, Activation, Flatten
+from keras.layers.embeddings import Embedding
+from keras.constraints import unitnorm
+from keras import backend as K
+
+
+X1 = np.array([[1], [2]], dtype='int32')
+W1 = np.array([[0.1, 0.2], [0.3, 0.4], [0.5, 0.6]], dtype='float32')
+
+
+def test_unitnorm_constraint():
+    lookup = Sequential()
+    lookup.add(Embedding(3, 2, weights=[W1],
+                         W_constraint=unitnorm(),
+                         input_length=1))
+    lookup.add(Flatten())
+    lookup.add(Dense(1))
+    lookup.add(Activation('sigmoid'))
+    lookup.compile(loss='binary_crossentropy', optimizer='sgd',
+                   class_mode='binary')
+    lookup.train_on_batch(X1, np.array([[1], [0]], dtype='int32'))
+    norm = np.linalg.norm(K.get_value(lookup.params[0]), axis=1)
+    assert_allclose(norm, np.ones_like(norm).astype('float32'), rtol=1e-05)
+
+
+if __name__ == '__main__':
+    pytest.main([__file__])

tests/keras/layers/test_noise.py

@@ -0,0 +1,41 @@
+import pytest
+import numpy as np
+from keras import backend as K
+from keras.layers import core
+from keras.layers import noise
+
+input_shape = (10, 10)
+batch_input_shape = (10, 10, 10)
+
+
+def test_GaussianNoise():
+    layer = noise.GaussianNoise(sigma=1., input_shape=input_shape)
+    _runner(layer)
+
+
+def test_GaussianDropout():
+    layer = noise.GaussianDropout(p=0.2, input_shape=input_shape)
+    _runner(layer)
+
+
+def _runner(layer):
+    assert isinstance(layer, core.Layer)
+    layer.build()
+    conf = layer.get_config()
+    assert (type(conf) == dict)
+
+    param = layer.get_params()
+    # Typically a list or a tuple, but may be any iterable
+    assert hasattr(param, '__iter__')
+    layer.input = K.variable(np.random.random(batch_input_shape))
+    output = layer.get_output(train=False)
+    output_np = K.eval(output)
+    assert output_np.shape == batch_input_shape
+
+    output = layer.get_output(train=True)
+    output_np = K.eval(output)
+    assert output_np.shape == batch_input_shape
+
+
+if __name__ == '__main__':
+    pytest.main([__file__])

tests/keras/layers/test_normalization.py

@@ -0,0 +1,125 @@
+import pytest
+import numpy as np
+from keras.layers.core import Dense, Activation
+from numpy.testing import assert_allclose
+
+from keras.layers import normalization
+from keras.models import Sequential, Graph
+from keras import backend as K
+
+
+input_1 = np.arange(10)
+input_2 = np.zeros(10)
+input_3 = np.ones((10))
+input_shapes = [np.ones((10, 10)), np.ones((10, 10, 10))]
+
+
+def test_batchnorm_mode_0():
+    np.random.seed(1337)
+    model = Sequential()
+    norm_m0 = normalization.BatchNormalization(input_shape=(10,))
+    model.add(norm_m0)
+    model.compile(loss='mse', optimizer='sgd')
+
+    # centered on 5.0, variance 10.0
+    X = np.random.normal(loc=5.0, scale=10.0, size=(1000, 10))
+    model.fit(X, X, nb_epoch=5, verbose=0)
+    norm_m0.input = K.variable(X)
+    out = (norm_m0.get_output(train=True) - norm_m0.beta) / norm_m0.gamma
+
+    assert_allclose(K.eval(K.mean(out)), 0.0, atol=1e-1)
+    assert_allclose(K.eval(K.std(out)), 1.0, atol=1e-1)
+
+
+def test_batchnorm_mode_1():
+    np.random.seed(1337)
+    norm_m1 = normalization.BatchNormalization(input_shape=(10,), mode=1)
+
+    for inp in [input_1, input_2, input_3]:
+        norm_m1.input = K.variable(inp)
+        out = (norm_m1.get_output(train=True) - norm_m1.beta) / norm_m1.gamma
+        assert_allclose(K.eval(K.mean(out)), 0.0, atol=1e-1)
+        if inp.std() > 0.:
+            assert_allclose(K.eval(K.std(out)), 1.0, atol=1e-1)
+        else:
+            assert_allclose(K.eval(K.std(out)), 0.0, atol=1e-1)
+
+
+def test_batchnorm_shapes():
+    """
+    Test batch normalization with various input shapes
+    """
+    for inp in input_shapes:
+        norm_m0 = normalization.BatchNormalization(input_shape=inp.shape, mode=0)
+        norm_m0.input = K.variable(inp)
+        out = (norm_m0.get_output(train=True) - norm_m0.beta) / norm_m0.gamma
+
+        norm_m1 = normalization.BatchNormalization(input_shape=inp.shape, mode=1)
+        norm_m1.input = K.variable(inp)
+        out = (norm_m1.get_output(train=True) - norm_m1.beta) / norm_m1.gamma
+
+
+def test_batchnorm_weight_init():
+    """
+    Test weight initialization
+    """
+    np.random.seed(1337)
+    norm_m1 = normalization.BatchNormalization(input_shape=(10,), mode=1,
+                                               weights=[np.ones(10), np.ones(10), np.zeros(10), np.zeros(10)])
+
+    for inp in [input_1, input_2, input_3]:
+        norm_m1.input = K.variable(inp)
+        out = (norm_m1.get_output(train=True) - np.ones(10)) / 1.
+        assert_allclose(K.eval(K.mean(out)), 0.0, atol=1e-1)
+        if inp.std() > 0.:
+            assert_allclose(K.eval(K.std(out)), 1.0, atol=1e-1)
+        else:
+            assert_allclose(K.eval(K.std(out)), 0.0, atol=1e-1)
+
+    assert_allclose(K.eval(norm_m1.gamma), np.ones(10), atol=1e-1)
+    assert_allclose(K.eval(norm_m1.beta), np.ones(10), atol=1e-1)
+
+
+def test_batchnorm_config():
+    norm = normalization.BatchNormalization(input_shape=(10, 10), mode=1,
+                                            epsilon=0.1, momentum=0.9)
+    conf = norm.get_config()
+    del conf['cache_enabled']
+    conf_target = {"input_shape": (10, 10),
+                   "name": normalization.BatchNormalization.__name__,
+                   "epsilon": 0.1, "mode": 1, "momentum": 0.9}
+    assert(conf == conf_target)
+
+
+def test_batchnorm_save_weights():
+    norm = normalization.BatchNormalization(input_shape=(10, 10), mode=1,
+                                            epsilon=0.1)
+    weights = norm.get_weights()
+    assert(len(weights) == 4)
+    norm.set_weights(weights)
+
+
+def test_batchnorm_nested():
+    # regression test for issue #1386
+    g = Graph()
+    g.add_input("input", input_shape=[20])
+    g.add_node(Dense(10), "dense", "input")
+    g.add_node(normalization.BatchNormalization(), "bn", "dense")
+    g.add_node(Activation('relu'), "activ", "bn")
+    g.add_output("output", "activ")
+
+    g2 = Graph()
+    g2.add_input("input", input_shape=[10])
+    g2.add_node(Dense(15), "dense", "input")
+    g2.add_node(normalization.BatchNormalization(), "bn", "dense")
+    g2.add_node(Activation('relu'), "activ", "bn")
+    g2.add_output("output", "activ")
+
+    model = Sequential()
+    model.add(g)
+    model.add(g2)
+    model.compile(loss="mse", optimizer="adadelta")
+
+
+if __name__ == '__main__':
+    pytest.main([__file__])

tests/keras/layers/test_recurrent.py

@@ -1,8 +1,10 @@
-import unittest
+import pytest
 import numpy as np
+from numpy.testing import assert_allclose
 
 from keras.layers import recurrent
 from keras import backend as K
+from keras.models import Sequential
 
 nb_samples, timesteps, input_dim, output_dim = 3, 3, 10, 5
 
@@ -28,20 +30,53 @@ def _runner(layer_class):
 
             mask = layer.get_output_mask(train)
 
+    # check statefulness
+    layer = layer_class(output_dim, return_sequences=False,
+                        stateful=True,
+                        weights=None,
+                        batch_input_shape=(nb_samples, timesteps, input_dim))
+    model = Sequential()
+    model.add(layer)
+    model.compile(optimizer='sgd', loss='mse')
+    out1 = model.predict(np.ones((nb_samples, timesteps, input_dim)))
+    assert(out1.shape == (nb_samples, output_dim))
 
-class TestRNNS(unittest.TestCase):
-    """
-    Test all the RNNs using a generic test runner function defined above.
-    """
-    def test_simple(self):
-        _runner(recurrent.SimpleRNN)
+    # train once so that the states change
+    model.train_on_batch(np.ones((nb_samples, timesteps, input_dim)),
+                         np.ones((nb_samples, output_dim)))
+    out2 = model.predict(np.ones((nb_samples, timesteps, input_dim)))
 
-    def test_gru(self):
-        _runner(recurrent.GRU)
+    # if the state is not reset, output should be different
+    assert(out1.max() != out2.max())
 
-    def test_lstm(self):
-        _runner(recurrent.LSTM)
+    # check that output changes after states are reset
+    # (even though the model itself didn't change)
+    layer.reset_states()
+    out3 = model.predict(np.ones((nb_samples, timesteps, input_dim)))
+    assert(out2.max() != out3.max())
+
+    # check that container-level reset_states() works
+    model.reset_states()
+    out4 = model.predict(np.ones((nb_samples, timesteps, input_dim)))
+    assert_allclose(out3, out4, atol=1e-5)
+
+    # check that the call to `predict` updated the states
+    out5 = model.predict(np.ones((nb_samples, timesteps, input_dim)))
+    assert(out4.max() != out5.max())
+
+
+
+def test_SimpleRNN():
+    _runner(recurrent.SimpleRNN)
+
+
+def test_GRU():
+    _runner(recurrent.GRU)
+
+
+def test_LSTM():
+    _runner(recurrent.LSTM)
 
 
 if __name__ == '__main__':
-    unittest.main()
+    pytest.main([__file__])

tests/keras/preprocessing/test_image.py

@@ -0,0 +1,62 @@
+import pytest
+from keras.preprocessing.image import *
+from PIL import Image
+import numpy as np
+import os
+import shutil
+
+def setup_function(func):
+    np.random.seed(1337)
+
+    os.mkdir('test_images')
+    os.mkdir('test_images/rgb')
+    os.mkdir('test_images/gsc')
+
+    img_w = img_h = 20
+    for n in range(8):
+        bias = np.random.rand(img_w,img_h,1)*64
+        variance = np.random.rand(img_w,img_h,1)*(255-64)
+        imarray = np.random.rand(img_w,img_h,3) * variance + bias
+        im = Image.fromarray(imarray.astype('uint8')).convert('RGBA')
+        im.save('test_images/rgb/rgb_test_image_'+str(n)+'.png')
+
+        imarray = np.random.rand(img_w,img_h,1) * variance + bias
+        im = Image.fromarray(imarray.astype('uint8').squeeze()).convert('L')
+        im.save('test_images/gsc/gsc_test_image_'+str(n)+'.png')
+
+def teardown_function(func):
+    shutil.rmtree('test_images')
+
+def test_image_data_generator():
+    np.random.seed(1337)
+
+    for color_mode in ['gsc','rgb']:
+        file_list = list_pictures('test_images/'+color_mode)
+        img_list = []
+        for f in file_list:
+            img_list.append(img_to_array(load_img(f))[None,...])
+
+        images = np.vstack(img_list)
+        generator = ImageDataGenerator(
+            featurewise_center=True,
+            samplewise_center=True,
+            featurewise_std_normalization=True,
+            samplewise_std_normalization=True,
+
+            zca_whitening=True,
+            rotation_range=90.,
+            width_shift_range=10.,
+            height_shift_range=10.,
+            horizontal_flip=True,
+            vertical_flip=True
+        )
+
+        generator.fit(images,augment=True)
+
+        for x,y in generator.flow(images,np.arange(images.shape[0]), shuffle=True, save_to_dir='test_images'):
+            assert x.shape[1:] == images.shape[1:]
+            #TODO: make sure the normalization is working as inteded
+
+
+if __name__ == '__main__':
+    pytest.main([__file__])

tests/keras/preprocessing/test_sequence.py

@@ -0,0 +1,53 @@
+import numpy as np
+from numpy.testing import assert_allclose
+
+import pytest
+
+from keras.preprocessing.sequence import pad_sequences
+from keras.preprocessing.sequence import make_sampling_table
+from keras.preprocessing.sequence import skipgrams
+
+
+def test_pad_sequences():
+    a = [[1], [1, 2], [1, 2, 3]]
+
+    # test padding
+    b = pad_sequences(a, maxlen=3, padding='pre')
+    assert_allclose(b, [[0, 0, 1], [0, 1, 2], [1, 2, 3]])
+    b = pad_sequences(a, maxlen=3, padding='post')
+    assert_allclose(b, [[1, 0, 0], [1, 2, 0], [1, 2, 3]])
+
+    # test truncating
+    b = pad_sequences(a, maxlen=2, truncating='pre')
+    assert_allclose(b, [[0, 1], [1, 2], [2, 3]])
+    b = pad_sequences(a, maxlen=2, truncating='post')
+    assert_allclose(b, [[0, 1], [1, 2], [1, 2]])
+
+    # test value
+    b = pad_sequences(a, maxlen=3, value=1)
+    assert_allclose(b, [[1, 1, 1], [1, 1, 2], [1, 2, 3]])
+
+
+def test_make_sampling_table():
+    a = make_sampling_table(3)
+    assert_allclose(a, np.asarray([0.00315225,  0.00315225,  0.00547597]),
+                    rtol=.1)
+
+
+def test_skipgrams():
+    # test with no window size and binary labels
+    couples, labels = skipgrams(np.arange(3), vocabulary_size=3)
+    for couple in couples:
+        assert couple[0] in [0, 1, 2] and couple[1] in [0, 1, 2]
+
+    # test window size and categorical labels
+    couples, labels = skipgrams(np.arange(5), vocabulary_size=5, window_size=1,
+                                categorical=True)
+    for couple in couples:
+        assert couple[0] - couple[1] <= 3
+    for l in labels:
+        assert len(l) == 2
+
+
+if __name__ == '__main__':
+    pytest.main([__file__])