Merge branch 'master' of https://github.com/fchollet/keras

* Added Convolution1D instead of Conv1D, which is depreceated

* updated rest of the example using Conv1D

* Python3 fails to decode utf-8 data, thus using encoding='latin-1'

* added condition for Encoding line 65-67

* Conv1D reverted back to the way it was

.travis.yml

@@ -12,7 +12,9 @@ matrix:
         - python: 2.7
           env: KERAS_BACKEND=tensorflow
         - python: 2.7
-          env: KERAS_BACKEND=theano INTEGRATION_TESTS=true
+          env: KERAS_BACKEND=theano TEST_MODE=INTEGRATION_TESTS
+        - python: 2.7
+          env: KERAS_BACKEND=theano TEST_MODE=PEP8
 install:
   # code below is taken from http://conda.pydata.org/docs/travis.html
   # We do this conditionally because it saves us some downloading if the
@@ -33,6 +35,7 @@ install:
   - conda create -q -n test-environment python=$TRAVIS_PYTHON_VERSION numpy scipy matplotlib pandas pytest h5py
   - source activate test-environment
   - pip install pytest-cov python-coveralls pytest-xdist coverage==3.7.1 #we need this version of coverage for coveralls.io to work
+  - pip install pep8 pytest-pep8
   - pip install git+git://github.com/Theano/Theano.git
 
   # install PIL for preprocessing tests
@@ -46,19 +49,23 @@ install:
 
   # install TensorFlow
   - if [[ "$TRAVIS_PYTHON_VERSION" == "2.7" ]]; then
-      pip install https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-0.6.0-cp27-none-linux_x86_64.whl;
+      pip install https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-0.9.0-cp27-none-linux_x86_64.whl;
     elif [[ "$TRAVIS_PYTHON_VERSION" == "3.4" ]]; then
-      pip install https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-0.6.0-cp34-none-linux_x86_64.whl;
+      pip install https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-0.9.0-cp34-cp34m-linux_x86_64.whl;
     fi
 # command to run tests
 script:
   # run keras backend init to initialize backend config
   - python -c "import keras.backend"
+  # create dataset directory to avoid concurrent directory creation at runtime
+  - mkdir ~/.keras/datasets
   # set up keras backend
   - sed -i -e 's/"backend":[[:space:]]*"[^"]*/"backend":\ "'$KERAS_BACKEND'/g' ~/.keras/keras.json;
   - echo -e "Running tests with the following config:\n$(cat ~/.keras/keras.json)"
-  - if [[ "$INTEGRATION_TESTS" == "true" ]]; then
+  - if [[ "$TEST_MODE" == "INTEGRATION_TESTS" ]]; then
        PYTHONPATH=$PWD:$PYTHONPATH py.test tests/integration_tests;
+    elif [[ "$TEST_MODE" == "PEP8" ]]; then
+       PYTHONPATH=$PWD:$PYTHONPATH py.test --pep8 -m pep8 -n0;
     else
        PYTHONPATH=$PWD:$PYTHONPATH py.test tests/ --ignore=tests/integration_tests;
     fi

CONTRIBUTING.md

@@ -36,25 +36,30 @@ We love pull requests. Here's a quick guide:
 
 1. If your PR introduces a change in functionality, make sure you start by opening an issue to discuss whether the change should be made, and how to handle it. This will save you from having your PR closed down the road! Of course, if your PR is a simple bug fix, you don't need to do that.
 
-2. Write the code. This is the hard part! We use PEP8 syntax conventions, but we aren't dogmatic when it comes to line length. Make sure your lines stay reasonably sized, though. To make your life easier, we recommend installing a PEP8 linter.
+2. Write the code. This is the hard part!
 
 3. Make sure any new function or class you introduce has proper docstrings. Make sure any code you touch still has up-to-date docstrings and documentation.
 
 4. Write tests. Your code should have full unit test coverage. If you want to see your PR merged promptly, this is crucial.
 
 5. Run our test suite locally. It's easy: from the Keras folder, simply run: `py.test tests/`.
-  - You will need to install `pytest`, `coveralls`, `pytest-cov`, `pytest-xdist`: `pip install pytest pytest-cov python-coveralls pytest-xdist`
+  - You will need to install `pytest`, `coveralls`, `pytest-cov`, `pytest-xdist`: `pip install pytest pytest-cov python-coveralls pytest-xdist pep8 pytest-pep8`
 
 6. Make sure all tests are passing:
   - with the Theano backend, on Python 2.7 and Python 3.5
   - with the TensorFlow backend, on Python 2.7
 
-7. When committing, use appropriate, descriptive commit messages. Make sure that your branch history is not a string of "bug fix", "fix", "oops", etc. When submitting your PR, squash your commits into a single commit with an appropriate commit message, to make sure the project history stays clean and readable. See ['rebase and squash'](http://rebaseandsqua.sh/) for technical help on how to squash your commits.
+7. We use PEP8 syntax conventions, but we aren't dogmatic when it comes to line length. Make sure your lines stay reasonably sized, though. To make your life easier, we recommend running a PEP8 linter:
+  - Install PEP8 packages: `pip install pep8 pytest-pep8 autopep8`
+  - Run a standalone PEP8 check: `py.test --pep8 -m pep8`
+  - You can automatically fix some PEP8 error by running: `autopep8 -i --select <errors> <FILENAME>` for example: `autopep8 -i --select E128 tests/keras/backend/test_backends.py`
 
-8. Update the documentation. If introducing new functionality, make sure you include code snippets demonstrating the usage of your new feature.
+8. When committing, use appropriate, descriptive commit messages. Make sure that your branch history is not a string of "bug fix", "fix", "oops", etc. When submitting your PR, squash your commits into a single commit with an appropriate commit message, to make sure the project history stays clean and readable. See ['rebase and squash'](http://rebaseandsqua.sh/) for technical help on how to squash your commits.
 
-9. Submit your PR. If your changes have been approved in a previous discussion, and if you have have complete (and passing) unit tests, your PR is likely to be merged promptly. Otherwise, well...
+9. Update the documentation. If introducing new functionality, make sure you include code snippets demonstrating the usage of your new feature.
+
+10. Submit your PR. If your changes have been approved in a previous discussion, and if you have complete (and passing) unit tests, your PR is likely to be merged promptly. Otherwise, well...
 
 ## Adding new examples
 
-Even if you don't contribute to the Keras source code, if you have an application of Keras that is concise and powerful, please consider adding it to our collection of examples. Existing examples show idiomatic Keras code: make sure to keep your own script in the same spirit.
+Even if you don't contribute to the Keras source code, if you have an application of Keras that is concise and powerful, please consider adding it to our collection of examples. [Existing examples](https://github.com/fchollet/keras/tree/master/examples) show idiomatic Keras code: make sure to keep your own script in the same spirit.

ISSUE_TEMPLATE.md

@@ -0,0 +1,9 @@
+Please make sure that the boxes below are checked before you submit your issue. Thank you!
+
+- [ ] Check that you are up-to-date with the master branch of Keras. You can update with:
+pip install git+git://github.com/fchollet/keras.git --upgrade --no-deps
+
+- [ ] If running on Theano, check that you are up-to-date with the master branch of Theano. You can update with:
+pip install git+git://github.com/Theano/Theano.git --upgrade --no-deps
+
+- [ ] Provide a link to a GitHub Gist of a Python script that can reproduce your issue (or just copy the script here if it is short).

README.md

@@ -1,6 +1,8 @@
 # Keras: Deep Learning library for Theano and TensorFlow
 
-![Build status](https://api.travis-ci.org/fchollet/keras.svg)
+[![Build Status](https://travis-ci.org/fchollet/keras.svg?branch=master)](https://travis-ci.org/fchollet/keras)
+[![PyPI version](https://badge.fury.io/py/keras.svg)](https://badge.fury.io/py/keras)
+[![license](https://img.shields.io/github/license/mashape/apistatus.svg?maxAge=2592000)](https://github.com/fchollet/keras/blob/master/LICENSE)
 
 ## You have just found Keras.
 
@@ -37,9 +39,9 @@ Keras is compatible with: __Python 2.7-3.5__.
 
 ## Getting started: 30 seconds to Keras
 
-The core data structure of Keras is a __model__, a way to organize layers. There are two types of models: [`Sequential`](http://keras.io/models/#sequential) and [`Graph`](http://keras.io/models/#graph).
+The core data structure of Keras is a __model__, a way to organize layers. The main type of model is the [`Sequential`](http://keras.io/getting-started/sequential-model-guide) model, a linear stack of layers. For more complex architectures, you should use the [Keras functional API](http://keras.io/getting-started/functional-api-guide).
 
-Here's the `Sequential` model (a linear pile of layers):
+Here's the `Sequential` model:
 
 ```python
 from keras.models import Sequential
@@ -50,17 +52,17 @@ model = Sequential()
 Stacking layers is as easy as `.add()`:
 
 ```python
-from keras.layers.core import Dense, Activation
+from keras.layers import Dense, Activation
 
-model.add(Dense(output_dim=64, input_dim=100, init="glorot_uniform"))
+model.add(Dense(output_dim=64, input_dim=100))
 model.add(Activation("relu"))
-model.add(Dense(output_dim=10, init="glorot_uniform"))
+model.add(Dense(output_dim=10))
 model.add(Activation("softmax"))
 ```
 
 Once your model looks good, configure its learning process with `.compile()`:
 ```python
-model.compile(loss='categorical_crossentropy', optimizer='sgd')
+model.compile(loss='categorical_crossentropy', optimizer='sgd', metrics=['accuracy'])
 ```
 
 If you need to, you can further configure your optimizer. A core principle of Keras is to make things reasonably simple, while allowing the user to be fully in control when they need to (the ultimate control being the easy extensibility of the source code).
@@ -81,7 +83,7 @@ model.train_on_batch(X_batch, Y_batch)
 
 Evaluate your performance in one line:
 ```python
-objective_score = model.evaluate(X_test, Y_test, batch_size=32)
+loss_and_metrics = model.evaluate(X_test, Y_test, batch_size=32)
 ```
 
 Or generate predictions on new data:
@@ -90,11 +92,14 @@ classes = model.predict_classes(X_test, batch_size=32)
 proba = model.predict_proba(X_test, batch_size=32)
 ```
 
-Building a network of LSTMs, a deep CNN, a Neural Turing Machine, a word2vec embedder or any other model is just as fast. The ideas behind deep learning are simple, so why should their implementation be painful?
+Building a question answering system, an image classification model, a Neural Turing Machine, a word2vec embedder or any other model is just as fast. The ideas behind deep learning are simple, so why should their implementation be painful?
 
-Have a look at these [starter examples](http://keras.io/examples/).
+For a more in-depth tutorial about Keras, you can check out:
 
-In the [examples folder](https://github.com/fchollet/keras/tree/master/examples) of the repo, you will find more advanced models: question-answering with memory networks, text generation with stacked LSTMs, neural turing machines, etc.
+- [Getting started with the Sequential model](http://keras.io/getting-started/sequential-model-guide)
+- [Getting started with the functional API](http://keras.io/getting-started/functional-api-guide)
+
+In the [examples folder](https://github.com/fchollet/keras/tree/master/examples) of the repository, you will find more advanced models: question-answering with memory networks, text generation with stacked LSTMs, etc.
 
 
 ------------------
@@ -114,11 +119,6 @@ Keras uses the following dependencies:
 - Theano
     - [See installation instructions](http://deeplearning.net/software/theano/install.html#install).
 
-**Note**: You should use the latest version of Theano, not the PyPI version. Install it with:
-```
-sudo pip install git+git://github.com/Theano/Theano.git
-```
-
 *When using the TensorFlow backend:*
 
 - TensorFlow

docker/Dockerfile

@@ -0,0 +1,46 @@
+FROM nvidia/cuda:7.5-cudnn5-devel
+
+ENV CONDA_DIR /opt/conda
+ENV PATH $CONDA_DIR/bin:$PATH
+
+RUN mkdir -p $CONDA_DIR && \
+    echo export PATH=$CONDA_DIR/bin:'$PATH' > /etc/profile.d/conda.sh && \
+    apt-get update && \
+    apt-get install -y wget git libhdf5-dev g++ graphviz && \
+    wget --quiet https://repo.continuum.io/miniconda/Miniconda3-3.9.1-Linux-x86_64.sh && \
+    echo "6c6b44acdd0bc4229377ee10d52c8ac6160c336d9cdd669db7371aa9344e1ac3 *Miniconda3-3.9.1-Linux-x86_64.sh" | sha256sum -c - && \
+    /bin/bash /Miniconda3-3.9.1-Linux-x86_64.sh -f -b -p $CONDA_DIR && \
+    rm Miniconda3-3.9.1-Linux-x86_64.sh
+
+ENV NB_USER keras
+ENV NB_UID 1000
+
+RUN useradd -m -s /bin/bash -N -u $NB_UID $NB_USER && \
+    mkdir -p $CONDA_DIR && \
+    chown keras $CONDA_DIR -R && \
+    mkdir -p /src && \
+    chown keras /src
+
+USER keras
+
+# Python
+ARG python_version=3.5.1
+ARG tensorflow_version=0.9.0rc0-cp35-cp35m
+RUN conda install -y python=${python_version} && \
+    pip install https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow-${tensorflow_version}-linux_x86_64.whl && \
+    pip install git+git://github.com/Theano/Theano.git && \
+    pip install ipdb pytest pytest-cov python-coveralls coverage==3.7.1 pytest-xdist pep8 pytest-pep8 pydot_ng && \
+    conda install Pillow scikit-learn notebook pandas matplotlib nose pyyaml six h5py && \
+    pip install git+git://github.com/fchollet/keras.git && \
+    conda clean -yt
+
+ADD theanorc /home/keras/.theanorc
+
+ENV PYTHONPATH='/src/:$PYTHONPATH'
+
+WORKDIR /src
+
+EXPOSE 8888
+
+CMD jupyter notebook --port=8888 --ip=0.0.0.0
+

docker/Makefile

@@ -0,0 +1,26 @@
+help:
+	@cat Makefile
+
+DATA?="${HOME}/Data"
+GPU?=0
+DOCKER_FILE=Dockerfile
+DOCKER=GPU=$(GPU) nvidia-docker
+BACKEND=tensorflow
+TEST=tests/
+SRC=$(shell dirname `pwd`)
+
+build:
+	docker build -t keras --build-arg python_version=3.5 -f $(DOCKER_FILE) .
+
+bash: build
+	$(DOCKER) run -it -v $(SRC):/src -v $(DATA):/data --env KERAS_BACKEND=$(BACKEND) keras bash
+
+ipython: build
+	$(DOCKER) run -it -v $(SRC):/src -v $(DATA):/data --env KERAS_BACKEND=$(BACKEND) keras ipython
+
+notebook: build
+	$(DOCKER) run -it -v $(SRC):/src -v $(DATA):/data --net=host --env KERAS_BACKEND=$(BACKEND) keras
+
+test: build
+	$(DOCKER) run -it -v $(SRC):/src -v $(DATA):/data --env KERAS_BACKEND=$(BACKEND) keras py.test $(TEST)
+

docker/README.md

@@ -0,0 +1,58 @@
+# Using Keras via Docker
+
+This directory contains `Dockerfile` to make it easy to get up and running with
+Keras via [Docker](http://www.docker.com/).
+
+## Installing Docker
+
+General installation instructions are
+[on the Docker site](https://docs.docker.com/installation/), but we give some
+quick links here:
+
+* [OSX](https://docs.docker.com/installation/mac/): [docker toolbox](https://www.docker.com/toolbox)
+* [ubuntu](https://docs.docker.com/installation/ubuntulinux/)
+
+## Running the container
+
+We are using `Makefile` to simplify docker commands within make commands.
+
+Build the container and start a jupyter notebook
+
+    $ make notebook
+
+Build the container and start an iPython shell
+
+    $ make ipython
+
+Build the container and start a bash
+
+    $ make bash
+
+For GPU support install NVidia drivers (ideally latest) and
+[nvidia-docker](https://github.com/NVIDIA/nvidia-docker). Run using
+
+    $ make notebook GPU=0 # or [ipython, bash]
+
+Switch between Theano and TensorFlow
+
+    $ make notebook BACKEND=theano
+    $ make notebook BACKEND=tensorflow
+
+Mount a volume for external data sets
+
+    $ make DATA=~/mydata
+
+Prints all make tasks
+
+    $ make help
+
+You can change Theano parameters by editing `/docker/theanorc`.
+
+
+Note: If you would have a problem running nvidia-docker you may try the old way
+we have used. But it is not recommended. If you find a bug in the nvidia-docker report
+it there please and try using the nvidia-docker as described above.
+
+    $ export CUDA_SO=$(\ls /usr/lib/x86_64-linux-gnu/libcuda.* | xargs -I{} echo '-v {}:{}')
+    $ export DEVICES=$(\ls /dev/nvidia* | xargs -I{} echo '--device {}:{}')
+    $ docker run -it -p 8888:8888 $CUDA_SO $DEVICES gcr.io/tensorflow/tensorflow:latest-gpu

docker/theanorc

@@ -0,0 +1,5 @@
+[global]
+floatX = float32
+optimizer=None
+device = gpu
+

docs/README.md

@@ -8,4 +8,5 @@ Our documentation uses extended Markdown, as implemented by [MkDocs](http://mkdo
 - install MkDocs: `pip install mkdocs`
 - `cd` to the `docs/` folder and run:
     - `python autogen.py`
-    - `mkdocs serve`
+    - `mkdocs serve`    # Starts a local webserver:  [localhost:8000](localhost:8000)
+    - `mkdocs build`    # Builds a static site in "site" directory

docs/autogen.py

@@ -1,9 +1,68 @@
 # -*- coding: utf-8 -*-
+'''
+General documentation architecture:
+
+Home
+Index
+
+- Getting started
+    Getting started with the sequential model
+    Getting started with the functional api
+    Examples
+    FAQ
+    Installation guide
+
+- Models
+    About Keras models
+        explain when one should use Sequential or functional API
+        explain compilation step
+        explain weight saving, weight loading
+        explain serialization, deserialization
+    Sequential
+    Model (functional API)
+
+- Layers
+    About Keras layers
+        explain common layer functions: get_weights, set_weights, get_config
+        explain input_shape
+        explain usage on non-Keras tensors
+    Core layers
+    Convolutional
+    Recurrent
+    Embeddings
+    Normalization
+    Advanced activations
+    Noise
+
+- Preprocessing
+    Image preprocessing
+    Text preprocessing
+    Sequence preprocessing
+
+Objectives
+Optimizers
+Activations
+Callbacks
+Datasets
+Backend
+Initializations
+Regularizers
+Constraints
+Visualization
+Scikit-learn API
+
+'''
 from __future__ import print_function
+from __future__ import unicode_literals
+
 import re
 import inspect
 import os
 import shutil
+import sys
+if sys.version[0] == '2':
+    reload(sys)
+    sys.setdefaultencoding('utf8')
 
 from keras.layers import convolutional
 from keras.layers import recurrent
@@ -11,35 +70,155 @@ from keras.layers import core
 from keras.layers import noise
 from keras.layers import normalization
 from keras.layers import advanced_activations
-from keras.layers import containers
 from keras.layers import embeddings
+from keras.layers import wrappers
 from keras import optimizers
 from keras import callbacks
 from keras import models
+from keras.engine import topology
+from keras import objectives
+from keras import backend
+from keras import constraints
+from keras import activations
+from keras import regularizers
 
-MODULES = [(convolutional, 'keras.layers.convolutional'),
-           (recurrent, 'keras.layers.recurrent'),
-           (noise, 'keras.layers.noise'),
-           (normalization, 'keras.layers.normalization'),
-           (advanced_activations, 'keras.layers.advanced_activations'),
-           (containers, 'keras.layers.containers'),
-           (core, 'keras.layers.core'),
-           (embeddings, 'keras.layers.embeddings'),
-           (optimizers, 'keras.optimizers'),
-           (callbacks, 'keras.callbacks'),
-           (models, 'keras.models')]
 
-SKIP = ['build', 'get_params', 'MaskedLayer',
-        'SiameseHead', 'MaskedLambda',
-        'CallbackList']
-ROOT = 'http://keras.io/'
-INCLUDE_METHODS_FOR = [
-    'Layer',
-    'Graph',
-    'Sequential',
-    'Callback',
+EXCLUDE = {
+    'Optimizer',
+    'Wrapper',
+    'get_session',
+    'set_session',
+    'CallbackList',
+}
+
+PAGES = [
+    {
+        'page': 'models/sequential.md',
+        'functions': [
+            models.Sequential.compile,
+            models.Sequential.fit,
+            models.Sequential.evaluate,
+            models.Sequential.predict,
+            models.Sequential.predict_classes,
+            models.Sequential.predict_proba,
+            models.Sequential.train_on_batch,
+            models.Sequential.test_on_batch,
+            models.Sequential.predict_on_batch,
+            models.Sequential.fit_generator,
+            models.Sequential.evaluate_generator,
+        ],
+    },
+    {
+        'page': 'models/model.md',
+        'functions': [
+            models.Model.compile,
+            models.Model.fit,
+            models.Model.evaluate,
+            models.Model.predict,
+            models.Model.train_on_batch,
+            models.Model.test_on_batch,
+            models.Model.predict_on_batch,
+            models.Model.fit_generator,
+            models.Model.evaluate_generator,
+            models.Model.get_layer,
+        ]
+    },
+    {
+        'page': 'layers/core.md',
+        'classes': [
+            core.Dense,
+            core.Activation,
+            core.Dropout,
+            core.Flatten,
+            core.Reshape,
+            core.Permute,
+            core.RepeatVector,
+            topology.Merge,
+            core.Lambda,
+            core.ActivityRegularization,
+            core.Masking,
+            core.Highway,
+            core.MaxoutDense,
+            core.TimeDistributedDense,
+        ],
+    },
+    {
+        'page': 'layers/convolutional.md',
+        'classes': [
+            convolutional.Convolution1D,
+            convolutional.Convolution2D,
+            convolutional.AtrousConv2D,
+            convolutional.Convolution3D,
+            convolutional.UpSampling1D,
+            convolutional.UpSampling2D,
+            convolutional.UpSampling3D,
+            convolutional.ZeroPadding1D,
+            convolutional.ZeroPadding2D,
+            convolutional.ZeroPadding3D,
+        ],
+    },
+    {
+        'page': 'layers/pooling.md',
+        'classes': [
+            convolutional.MaxPooling1D,
+            convolutional.MaxPooling2D,
+            convolutional.MaxPooling3D,
+            convolutional.AveragePooling1D,
+            convolutional.AveragePooling2D,
+            convolutional.AveragePooling3D,
+        ],
+    },
+    {
+        'page': 'layers/recurrent.md',
+        'classes': [
+            recurrent.Recurrent,
+            recurrent.SimpleRNN,
+            recurrent.GRU,
+            recurrent.LSTM,
+        ],
+    },
+    {
+        'page': 'layers/embeddings.md',
+        'classes': [
+            embeddings.Embedding,
+        ],
+    },
+    {
+        'page': 'layers/normalization.md',
+        'classes': [
+            normalization.BatchNormalization,
+        ],
+    },
+    {
+        'page': 'layers/advanced-activations.md',
+        'all_module_classes': [advanced_activations],
+    },
+    {
+        'page': 'layers/noise.md',
+        'all_module_classes': [noise],
+    },
+    {
+        'page': 'layers/wrappers.md',
+        'all_module_classes': [wrappers],
+    },
+
+
+    {
+        'page': 'optimizers.md',
+        'all_module_classes': [optimizers],
+    },
+    {
+        'page': 'callbacks.md',
+        'all_module_classes': [callbacks],
+    },
+    {
+        'page': 'backend.md',
+        'all_module_functions': [backend],
+    },
 ]
 
+ROOT = 'http://keras.io/'
+
 
 def get_earliest_class_that_defined_member(member, cls):
     ancestors = get_classes_ancestors([cls])
@@ -67,23 +246,24 @@ def get_classes_ancestors(classes):
         return filtered_ancestors
 
 
-def get_method_signature(method):
-    signature = inspect.getargspec(method)
+def get_function_signature(function, method=True):
+    signature = inspect.getargspec(function)
     defaults = signature.defaults
-    args = signature.args[1:]
+    if method:
+        args = signature.args[1:]
+    else:
+        args = signature.args
     if defaults:
         kwargs = zip(args[-len(defaults):], defaults)
         args = args[:-len(defaults)]
     else:
         kwargs = []
-    st = '%s.%s(' % (method.__module__, method.__name__)
+    st = '%s.%s(' % (function.__module__, function.__name__)
     for a in args:
         st += str(a) + ', '
     for a, v in kwargs:
-        if  type(v) == str:
+        if type(v) == str:
             v = '\'' + v + '\''
-        elif type(v) == unicode:
-            v = 'u\'' + v + '\''
         st += str(a) + '=' + str(v) + ', '
     if kwargs or args:
         return st[:-2] + ')'
@@ -91,6 +271,17 @@ def get_method_signature(method):
         return st + ')'
 
 
+def get_class_signature(cls):
+    try:
+        class_signature = get_function_signature(cls.__init__)
+        class_signature = class_signature.replace('__init__', cls.__name__)
+    except:
+        # in case the class inherits from object and does not
+        # define __init__
+        class_signature = cls.__module__ + '.' + cls.__name__ + '()'
+    return class_signature
+
+
 def class_to_docs_link(cls):
     module_name = cls.__module__
     assert module_name[:6] == 'keras.'
@@ -117,26 +308,33 @@ def code_snippet(snippet):
 
 
 def process_class_docstring(docstring):
-    docstring = re.sub(r'    # (.*)\n',
-                       r'    __\1__\n\n',
+    docstring = re.sub(r'\n    # (.*)\n',
+                       r'\n    __\1__\n\n',
                        docstring)
 
     docstring = re.sub(r'    ([^\s\\]+):(.*)\n',
                        r'    - __\1__:\2\n',
                        docstring)
+
+    docstring = docstring.replace('    ' * 5, '\t\t')
     docstring = docstring.replace('    ' * 3, '\t')
     docstring = docstring.replace('    ', '')
     return docstring
 
 
-def process_method_docstring(docstring):
-    docstring = re.sub(r'    # (.*)\n',
-                       r'    __\1__\n\n',
+def process_function_docstring(docstring):
+    docstring = re.sub(r'\n    # (.*)\n',
+                       r'\n    __\1__\n\n',
+                       docstring)
+    docstring = re.sub(r'\n        # (.*)\n',
+                       r'\n        __\1__\n\n',
                        docstring)
 
     docstring = re.sub(r'    ([^\s\\]+):(.*)\n',
                        r'    - __\1__:\2\n',
                        docstring)
+
+    docstring = docstring.replace('    ' * 6, '\t\t')
     docstring = docstring.replace('    ' * 4, '\t')
     docstring = docstring.replace('    ', '')
     return docstring
@@ -144,6 +342,7 @@ def process_method_docstring(docstring):
 print('Cleaning up existing sources directory.')
 if os.path.exists('sources'):
     shutil.rmtree('sources')
+
 print('Populating sources directory with templates.')
 for subdir, dirs, fnames in os.walk('templates'):
     for fname in fnames:
@@ -156,101 +355,75 @@ for subdir, dirs, fnames in os.walk('templates'):
             shutil.copy(fpath, new_fpath)
 
 print('Starting autogeneration.')
-covered_so_far = set()
-for module, module_name in MODULES:
-    class_pages = []
-    for name in dir(module):
-        if name in SKIP:
-            continue
-        if name[0] == '_':
-            continue
-        module_member = getattr(module, name)
-        if module_member in covered_so_far:
-            continue
-        if inspect.isclass(module_member):
-            cls = module_member
-            if cls.__module__ == module_name:
+for page_data in PAGES:
+    blocks = []
+    classes = page_data.get('classes', [])
+    for module in page_data.get('all_module_classes', []):
+        module_classes = []
+        for name in dir(module):
+            if name[0] == '_' or name in EXCLUDE:
+                continue
+            module_member = getattr(module, name)
+            if inspect.isclass(module_member):
+                cls = module_member
+                if cls.__module__ == module.__name__:
+                    if cls not in module_classes:
+                        module_classes.append(cls)
+        module_classes.sort(key=lambda x: id(x))
+        classes += module_classes
 
-                try:
-                    class_signature = get_method_signature(cls.__init__)
-                    class_signature = class_signature.replace('__init__', cls.__name__)
-                except:
-                    # in case the class inherits from object and does not
-                    # define __init__
-                    class_signature = module_name + '.' + cls.__name__ + '()'
+    for cls in classes:
+        subblocks = []
+        signature = get_class_signature(cls)
+        subblocks.append('<span style="float:right;">' + class_to_source_link(cls) + '</span>')
+        subblocks.append('### ' + cls.__name__ + '\n')
+        subblocks.append(code_snippet(signature))
+        docstring = cls.__doc__
+        if docstring:
+            subblocks.append(process_class_docstring(docstring))
+        blocks.append('\n'.join(subblocks))
 
-                methods = []
-                methods_not_defined_here = []
-                for name in dir(cls):
-                    if name in SKIP:
-                        continue
-                    if name[0] == '_':
-                        continue
-                    cls_member = getattr(cls, name)
-                    if inspect.ismethod(cls_member):
-                        method = cls_member
-                        signature = inspect.getargspec(method)
-                        defaults = signature.defaults
-                        args = signature.args[1:]
-                        if defaults:
-                            kwargs = zip(args[-len(defaults):], defaults)
-                            args = args[:-len(defaults)]
-                        else:
-                            kwargs = []
+    functions = page_data.get('functions', [])
+    for module in page_data.get('all_module_functions', []):
+        module_functions = []
+        for name in dir(module):
+            if name[0] == '_' or name in EXCLUDE:
+                continue
+            module_member = getattr(module, name)
+            if inspect.isfunction(module_member):
+                function = module_member
+                if module.__name__ in function.__module__:
+                    if function not in module_functions:
+                        module_functions.append(function)
+        module_functions.sort(key=lambda x: id(x))
+        functions += module_functions
 
-                        defined_by = get_earliest_class_that_defined_member(method.__name__, cls)
-                        if cls == defined_by:
-                            methods.append(method)
-                        else:
-                            methods_not_defined_here.append((method, defined_by))
-
-                blocks = []
-                blocks.append('<span style="float:right;">' + class_to_source_link(cls) + '</span>')
-                blocks.append('# ' + cls.__name__ + '\n')
-                blocks.append(code_snippet(class_signature))
-                docstring = cls.__doc__
-                if docstring:
-                    blocks.append(process_class_docstring(docstring))
-
-                if cls.__name__ in INCLUDE_METHODS_FOR:
-                    if methods or methods_not_defined_here:
-                        blocks.append('### Methods\n')
-                        for method in methods:
-                            signature = get_method_signature(method)
-                            signature = signature.replace(module_name + '.', '')
-                            blocks.append(code_snippet(signature))
-                            docstring = method.__doc__
-                            if docstring:
-                                blocks.append(process_method_docstring(docstring))
-                        for method, defined_by in methods_not_defined_here:
-                            signature = get_method_signature(method)
-                            method_module_name = method.__module__
-                            signature = signature.replace(method_module_name + '.', '')
-                            link = '[' + defined_by.__name__ + '](' + class_to_docs_link(defined_by) + ')'
-                            blocks.append(code_snippet(signature))
-                            blocks.append('Defined by ' + link + '.\n')
-
-                mkdown = '\n'.join(blocks)
-                class_pages.append((id(cls), mkdown))
-                covered_so_far.add(module_member)
-
-    class_pages.sort(key=lambda x: x[0])
-    class_pages = [x[1] for x in class_pages]
-    module_page = '\n----\n\n'.join(class_pages)
+    for function in functions:
+        subblocks = []
+        signature = get_function_signature(function, method=False)
+        signature = signature.replace(function.__module__ + '.', '')
+        subblocks.append('### ' + function.__name__ + '\n')
+        subblocks.append(code_snippet(signature))
+        docstring = function.__doc__
+        if docstring:
+            subblocks.append(process_function_docstring(docstring))
+            blocks.append('\n\n'.join(subblocks))
 
+    mkdown = '\n----\n\n'.join(blocks)
     # save module page.
     # Either insert content into existing page,
     # or create page otherwise
-    path = 'sources/' + module_name.replace('.', '/')[6:] + '.md'
+    page_name = page_data['page']
+    path = os.path.join('sources', page_name)
     if os.path.exists(path):
         template = open(path).read()
         assert '{{autogenerated}}' in template, ('Template found for ' + path +
                                                  ' but missing {{autogenerated}} tag.')
-        module_page = template.replace('{{autogenerated}}', module_page)
+        mkdown = template.replace('{{autogenerated}}', mkdown)
         print('...inserting autogenerated content into template:', path)
     else:
         print('...creating new page with autogenerated content:', path)
     subdir = os.path.dirname(path)
     if not os.path.exists(subdir):
         os.makedirs(subdir)
-    open(path, 'w').write(module_page)
+    open(path, 'w').write(mkdown)

docs/mkdocs.yml

@@ -3,8 +3,8 @@ theme: readthedocs
 docs_dir: sources
 repo_url: http://github.com/fchollet/keras
 site_url: http://keras.io/
-#theme_dir: theme
-site_description: Documentation for fast and lightweight Keras Deep Learning library.
+# theme_dir: theme
+site_description: 'Documentation for Keras, the Python Deep Learning library.'
 
 dev_addr: '0.0.0.0:8000'
 google_analytics: ['UA-61785484-1', 'keras.io']
@@ -12,31 +12,42 @@ google_analytics: ['UA-61785484-1', 'keras.io']
 
 pages:
 - Home: index.md
-- Index: documentation.md
-- Examples: examples.md
-- FAQ: faq.md
-- Backends: backend.md
-- Optimizers: optimizers.md
-- Objectives: objectives.md
-- Models: models.md
-- Activations: activations.md
-- Initializations: initializations.md
-- Regularizers: regularizers.md
-- Constraints: constraints.md
-- Callbacks: callbacks.md
-- Datasets: datasets.md
-- Visualization: visualization.md
+- Getting started:
+  - Guide to the Sequential model: getting-started/sequential-model-guide.md
+  - Guide to the Functional API: getting-started/functional-api-guide.md
+  - FAQ: getting-started/faq.md
+- Models:
+  - About Keras models: models/about-keras-models.md
+  - Sequential: models/sequential.md
+  - Model (functional API): models/model.md
 - Layers:
+  - About Keras layers: layers/about-keras-layers.md
   - Core Layers: layers/core.md
   - Convolutional Layers: layers/convolutional.md
+  - Pooling Layers: layers/pooling.md
   - Recurrent Layers: layers/recurrent.md
-  - Advanced Activations Layers: layers/advanced_activations.md
-  - Normalization Layers: layers/normalization.md
   - Embedding Layers: layers/embeddings.md
+  - Advanced Activations Layers: layers/advanced-activations.md
+  - Normalization Layers: layers/normalization.md
   - Noise layers: layers/noise.md
-  - Containers: layers/containers.md
+  - Layer wrappers: layers/wrappers.md
+  - Writing your own Keras layers: layers/writing-your-own-keras-layers.md
 - Preprocessing:
   - Sequence Preprocessing: preprocessing/sequence.md
   - Text Preprocessing: preprocessing/text.md
   - Image Preprocessing: preprocessing/image.md
+- Objectives: objectives.md
+- Optimizers: optimizers.md
+- Activations: activations.md
+- Callbacks: callbacks.md
+- Datasets: datasets.md
+- Backend: backend.md
+- Initializations: initializations.md
+- Regularizers: regularizers.md
+- Constraints: constraints.md
+- Visualization: visualization.md
+- Scikit-learn API: scikit-learn-api.md
+
+
+
 

docs/templates/activations.md

@@ -14,11 +14,13 @@ is equivalent to:
 model.add(Dense(64, activation='tanh'))
 ```
 
-You can also pass an element-wise Theano function as an activation:
+You can also pass an element-wise Theano/TensorFlow function as an activation:
 
 ```python
+from keras import backend as K
+
 def tanh(x):
-    return theano.tensor.tanh(x)
+    return K.tanh(x)
 
 model.add(Dense(64, activation=tanh))
 model.add(Activation(tanh))
@@ -28,6 +30,7 @@ model.add(Activation(tanh))
 
 - __softmax__: Softmax applied across inputs last dimension. Expects shape either `(nb_samples, nb_timesteps, nb_dims)` or `(nb_samples, nb_dims)`.
 - __softplus__
+- __softsign__
 - __relu__
 - __tanh__
 - __sigmoid__
@@ -36,4 +39,4 @@ model.add(Activation(tanh))
 
 ## On Advanced Activations
 
-Activations that are more complex than a simple Theano function (eg. learnable activations, configurable activations, etc.) are available as [Advanced Activation layers](layers/advanced_activations.md), and can be found in the module `keras.layers.advanced_activations`. These include PReLU and LeakyReLU.
+Activations that are more complex than a simple Theano/TensorFlow function (eg. learnable activations, configurable activations, etc.) are available as [Advanced Activation layers](layers/advanced-activations.md), and can be found in the module `keras.layers.advanced_activations`. These include PReLU and LeakyReLU.

docs/templates/backend.md

@@ -9,6 +9,8 @@ At this time, Keras has two backend implementations available: the **Theano** ba
 - [Theano](http://deeplearning.net/software/theano/) is an open-source symbolic tensor manipulation framework developed by LISA/MILA Lab at Université de Montréal.
 - [TensorFlow](http://www.tensorflow.org/) is an open-source symbolic tensor manipulation framework developed by Google, Inc.
 
+----
+
 ## Switching from one backend to another
 
 If you have run Keras at least once, you will find the Keras configuration file at:
@@ -27,11 +29,13 @@ You can also define the environment variable ``KERAS_BACKEND`` and this will
 override what is defined in your config file :
 
 ```bash
-KERAS_BACKEND=tensorflow python -c "from keras import backend; print backend._BACKEND"
+KERAS_BACKEND=tensorflow python -c "from keras import backend; print(backend._BACKEND)"
 Using TensorFlow backend.
 tensorflow
 ```
 
+----
+
 ## Using the abstract Keras backend to write new code
 
 If you want the Keras modules you write to be compatible with both Theano and TensorFlow, you have to write them via the abstract Keras backend API. Here's an intro.
@@ -74,8 +78,12 @@ a = concatenate([b, c], axis=-1)
 # etc...
 ```
 
-For more information, see the code at `keras/backend/theano_backend.py` and `keras/backend/tensorflow_backend.py`.
-
+----
+
+## Backend functions
+
+
+{{autogenerated}}
 
 
 

docs/templates/datasets.md

@@ -2,13 +2,13 @@
 
 ## CIFAR10 small image classification
 
-`keras.datasets.cifar10`
-
 Dataset of 50,000 32x32 color training images, labeled over 10 categories, and 10,000 test images.
 
 ### Usage:
 
 ```python
+from keras.datasets import cifar10
+
 (X_train, y_train), (X_test, y_test) = cifar10.load_data()
 ```
 
@@ -21,13 +21,13 @@ Dataset of 50,000 32x32 color training images, labeled over 10 categories, and 1
 
 ## CIFAR100 small image classification
 
-`keras.datasets.cifar100`
-
 Dataset of 50,000 32x32 color training images, labeled over 100 categories, and 10,000 test images.
 
 ### Usage:
 
 ```python
+from keras.datasets import cifar100
+
 (X_train, y_train), (X_test, y_test) = cifar100.load_data(label_mode='fine')
 ```
 
@@ -44,8 +44,6 @@ Dataset of 50,000 32x32 color training images, labeled over 100 categories, and
 
 ## IMDB Movie reviews sentiment classification
 
-`keras.datasets.imdb`
-
 Dataset of 25,000 movies reviews from IMDB, labeled by sentiment (positive/negative). Reviews have been preprocessed, and each review is encoded as a [sequence](preprocessing/sequence.md) of word indexes (integers). For convenience, words are indexed by overall frequency in the dataset, so that for instance the integer "3" encodes the 3rd most frequent word in the data. This allows for quick filtering operations such as: "only consider the top 10,000 most common words, but eliminate the top 20 most common words".
 
 As a convention, "0" does not stand for a specific word, but instead is used to encode any unknown word.
@@ -53,8 +51,13 @@ As a convention, "0" does not stand for a specific word, but instead is used to
 ### Usage:
 
 ```python
-(X_train, y_train), (X_test, y_test) = imdb.load_data(path="imdb.pkl", \
-nb_words=None, skip_top=0, maxlen=None, test_split=0.1, seed=113)
+from keras.datasets import imdb
+
+(X_train, y_train), (X_test, y_test) = imdb.load_data(path="imdb.pkl",
+                                                      nb_words=None,
+                                                      skip_top=0,
+                                                      maxlen=None,
+                                                      test_split=0.1)
 ```
 - __Return:__
     - 2 tuples:
@@ -74,15 +77,18 @@ nb_words=None, skip_top=0, maxlen=None, test_split=0.1, seed=113)
 
 ## Reuters newswire topics classification
 
-`keras.datasets.reuters`
-
 Dataset of 11,228 newswires from Reuters, labeled over 46 topics. As with the IMDB dataset, each wire is encoded as a sequence of word indexes (same conventions).
 
 ### Usage:
 
 ```python
-(X_train, y_train), (X_test, y_test) = reuters.load_data(path="reuters.pkl", \
-nb_words=None, skip_top=0, maxlen=None, test_split=0.1, seed=113)
+from keras.datasets import reuters
+
+(X_train, y_train), (X_test, y_test) = reuters.load_data(path="reuters.pkl",
+                                                         nb_words=None,
+                                                         skip_top=0,
+                                                         maxlen=None,
+                                                         test_split=0.1)
 ```
 
 The specifications are the same as that of the IMDB dataset.
@@ -101,13 +107,13 @@ word_index = reuters.get_word_index(path="reuters_word_index.pkl")
     
 ## MNIST database of handwritten digits
 
-`keras.datasets.mnist`
-
 Dataset of 60,000 28x28 grayscale images of the 10 digits, along with a test set of 10,000 images.
 
 ### Usage:
 
 ```python
+from keras.datasets import mnist
+
 (X_train, y_train), (X_test, y_test) = mnist.load_data()
 ```
 

docs/templates/documentation.md

@@ -1,40 +0,0 @@
-# Keras Documentation Index
-
-## Introduction
-
-- [Home](index.md)
-- [Index](documentation.md)
-- [Examples](examples.md)
-- [FAQ](faq.md)
-- [Backend](backend.md)
-
----
-
-## Base functionality
-
-- [Optimizers](optimizers.md)
-- [Objectives](objectives.md)
-- [Models](models.md)
-- [Activations](activations.md)
-- [Initializations](initializations.md)
-- [Regularizers](regularizers.md)
-- [Constraints](constraints.md)
-- [Callbacks](callbacks.md)
-- [Datasets](datasets.md)
-
----
-
-## Layers
-- [Core](layers/core.md)
-- [Convolutional](layers/convolutional.md)
-- [Recurrent](layers/recurrent.md)
-- [Advanced Activations](layers/advanced_activations.md)
-- [Normalization](layers/normalization.md)
-- [Embeddings](layers/embeddings.md)
-
----
-
-## Preprocessing
-- [Sequence](preprocessing/sequence.md)
-- [Text](preprocessing/text.md)
-- [Image](preprocessing/image.md)

docs/templates/examples.md

@@ -1,177 +0,0 @@
-
-Here are a few examples to get you started!
-
-### Multilayer Perceptron (MLP):
-
-```python
-from keras.models import Sequential
-from keras.layers.core import Dense, Dropout, Activation
-from keras.optimizers import SGD
-
-model = Sequential()
-# Dense(64) is a fully-connected layer with 64 hidden units.
-# in the first layer, you must specify the expected input data shape:
-# here, 20-dimensional vectors.
-model.add(Dense(64, input_dim=20, init='uniform'))
-model.add(Activation('tanh'))
-model.add(Dropout(0.5))
-model.add(Dense(64, init='uniform'))
-model.add(Activation('tanh'))
-model.add(Dropout(0.5))
-model.add(Dense(2, init='uniform'))
-model.add(Activation('softmax'))
-
-sgd = SGD(lr=0.1, decay=1e-6, momentum=0.9, nesterov=True)
-model.compile(loss='mean_squared_error', optimizer=sgd)
-
-model.fit(X_train, y_train, nb_epoch=20, batch_size=16)
-score = model.evaluate(X_test, y_test, batch_size=16)
-```
-
-### Alternative implementation of MLP:
-
-```python
-model = Sequential()
-model.add(Dense(64, input_dim=20, init='uniform', activation='tanh'))
-model.add(Dropout(0.5))
-model.add(Dense(64, init='uniform', activation='tanh'))
-model.add(Dropout(0.5))
-model.add(Dense(2, init='uniform', activation='softmax'))
-
-sgd = SGD(lr=0.1, decay=1e-6, momentum=0.9, nesterov=True)
-model.compile(loss='mean_squared_error', optimizer=sgd)
-```
-
-### VGG-like convnet:
-
-```python
-from keras.models import Sequential
-from keras.layers.core import Dense, Dropout, Activation, Flatten
-from keras.layers.convolutional import Convolution2D, MaxPooling2D
-from keras.optimizers import SGD
-
-model = Sequential()
-# input: 100x100 images with 3 channels -> (3, 100, 100) tensors.
-# this applies 32 convolution filters of size 3x3 each.
-model.add(Convolution2D(32, 3, 3, border_mode='full', input_shape=(3, 100, 100)))
-model.add(Activation('relu'))
-model.add(Convolution2D(32, 3, 3))
-model.add(Activation('relu'))
-model.add(MaxPooling2D(pool_size=(2, 2)))
-model.add(Dropout(0.25))
-
-model.add(Convolution2D(64, 3, 3, border_mode='valid'))
-model.add(Activation('relu'))
-model.add(Convolution2D(64, 3, 3))
-model.add(Activation('relu'))
-model.add(MaxPooling2D(pool_size=(2, 2)))
-model.add(Dropout(0.25))
-
-model.add(Flatten())
-# Note: Keras does automatic shape inference.
-model.add(Dense(256))
-model.add(Activation('relu'))
-model.add(Dropout(0.5))
-
-model.add(Dense(10))
-model.add(Activation('softmax'))
-
-sgd = SGD(lr=0.1, decay=1e-6, momentum=0.9, nesterov=True)
-model.compile(loss='categorical_crossentropy', optimizer=sgd)
-
-model.fit(X_train, Y_train, batch_size=32, nb_epoch=1)
-
-```
-
-### Sequence classification with LSTM:
-
-```python
-from keras.models import Sequential
-from keras.layers.core import Dense, Dropout, Activation
-from keras.layers.embeddings import Embedding
-from keras.layers.recurrent import LSTM
-
-model = Sequential()
-model.add(Embedding(max_features, 256, input_length=maxlen))
-model.add(LSTM(output_dim=128, activation='sigmoid', inner_activation='hard_sigmoid'))
-model.add(Dropout(0.5))
-model.add(Dense(1))
-model.add(Activation('sigmoid'))
-
-model.compile(loss='binary_crossentropy', optimizer='rmsprop')
-
-model.fit(X_train, Y_train, batch_size=16, nb_epoch=10)
-score = model.evaluate(X_test, Y_test, batch_size=16)
-```
-
-### Architecture for learning image captions with a convnet and a Gated Recurrent Unit:
-(word-level embedding, caption of maximum length 16 words).
-
-Note that getting this to work well will require using a bigger convnet, initialized with pre-trained weights.
-
-```python
-max_caption_len = 16
-vocab_size = 10000
-
-# first, let's define an image model that
-# will encode pictures into 128-dimensional vectors.
-# it should be initialized with pre-trained weights.
-image_model = Sequential()
-image_model.add(Convolution2D(32, 3, 3, border_mode='full', input_shape=(3, 100, 100)))
-image_model.add(Activation('relu'))
-image_model.add(Convolution2D(32, 3, 3))
-image_model.add(Activation('relu'))
-image_model.add(MaxPooling2D(pool_size=(2, 2)))
-
-image_model.add(Convolution2D(64, 3, 3, border_mode='full'))
-image_model.add(Activation('relu'))
-image_model.add(Convolution2D(64, 3, 3))
-image_model.add(Activation('relu'))
-image_model.add(MaxPooling2D(pool_size=(2, 2)))
-
-image_model.add(Flatten())
-image_model.add(Dense(128))
-
-# let's load the weights from a save file.
-image_model.load_weights('weight_file.h5')
-
-# next, let's define a RNN model that encodes sequences of words
-# into sequences of 128-dimensional word vectors.
-language_model = Sequential()
-language_model.add(Embedding(vocab_size, 256, input_length=max_caption_len))
-language_model.add(GRU(output_dim=128, return_sequences=True))
-language_model.add(TimeDistributedDense(128))
-
-# let's repeat the image vector to turn it into a sequence.
-image_model.add(RepeatVector(max_caption_len))
-
-# the output of both models will be tensors of shape (samples, max_caption_len, 128).
-# let's concatenate these 2 vector sequences.
-model = Merge([image_model, language_model], mode='concat', concat_axis=-1)
-# let's encode this vector sequence into a single vector
-model.add(GRU(256, 256, return_sequences=False))
-# which will be used to compute a probability
-# distribution over what the next word in the caption should be!
-model.add(Dense(vocab_size))
-model.add(Activation('softmax'))
-
-model.compile(loss='categorical_crossentropy', optimizer='rmsprop')
-
-# "images" is a numpy float array of shape (nb_samples, nb_channels=3, width, height).
-# "captions" is a numpy integer array of shape (nb_samples, max_caption_len)
-# containing word index sequences representing partial captions.
-# "next_words" is a numpy float array of shape (nb_samples, vocab_size)
-# containing a categorical encoding (0s and 1s) of the next word in the corresponding
-# partial caption.
-model.fit([images, partial_captions], next_words, batch_size=16, nb_epoch=100)
-```
-
-In the examples folder, you will find example models for real datasets:
-
-- CIFAR10 small images classification: Convolutional Neural Network (CNN) with realtime data augmentation
-- IMDB movie review sentiment classification: LSTM over sequences of words
-- Reuters newswires topic classification: Multilayer Perceptron (MLP)
-- MNIST handwritten digits classification: MLP & CNN
-- Character-level text generation with LSTM
-
-...and more.

docs/templates/faq.md

@@ -1,229 +0,0 @@
-# Keras FAQ: Frequently Asked Keras Questions
-
-[How can I run Keras on GPU?](#how-can-i-run-keras-on-gpu)
-
-[How can I save a Keras model?](#how-can-i-save-a-keras-model)
-
-[Why is the training loss much higher than the testing loss?](#why-is-the-training-loss-much-higher-than-the-testing-loss)
-
-[How can I visualize the output of an intermediate layer?](#how-can-i-visualize-the-output-of-an-intermediate-layer)
-
-[Isn't there a bug with Merge or Graph related to input concatenation?](#isnt-there-a-bug-with-merge-or-graph-related-to-input-concatenation)
-
-[How can I use Keras with datasets that don't fit in memory?](#how-can-i-use-keras-with-datasets-that-dont-fit-in-memory)
-
-[How can I interrupt training when the validation loss isn't decreasing anymore?](#how-can-i-interrupt-training-when-the-validation-loss-isnt-decreasing-anymore)
-
-[How is the validation split computed?](#how-is-the-validation-split-computed)
-
-[Is the data shuffled during training?](#is-the-data-shuffled-during-training)
-
-[How can I record the training / validation loss / accuracy at each epoch?](#how-can-i-record-the-training-validation-loss-accuracy-at-each-epoch)
-
-[How can I use stateful RNNs?](#how-can-i-use-stateful-rnns)
-
----
-
-### How can I run Keras on GPU?
-
-Method 1: use Theano flags.
-```bash
-THEANO_FLAGS=device=gpu,floatX=float32 python my_keras_script.py
-```
-
-The name 'gpu' might have to be changed depending on your device's identifier (e.g. `gpu0`, `gpu1`, etc).
-
-Method 2: set up your `.theanorc`: [Instructions](http://deeplearning.net/software/theano/library/config.html)
-
-Method 3: manually set `theano.config.device`, `theano.config.floatX` at the beginning of your code:
-```python
-import theano
-theano.config.device = 'gpu'
-theano.config.floatX = 'float32'
-```
-
----
-
-### How can I save a Keras model?
-
-*It is not recommended to use pickle or cPickle to save a Keras model.*
-
-If you only need to save the architecture of a model, and not its weights, you can do:
-
-```python
-# save as JSON
-json_string = model.to_json()
-
-# save as YAML
-yaml_string = model.to_yaml()
-```
-
-You can then build a fresh model from this data:
-
-```python
-# model reconstruction from JSON:
-from keras.models import model_from_json
-model = model_from_json(json_string)
-
-# model reconstruction from YAML
-model = model_from_yaml(yaml_string)
-```
-
-If you need to save the weights of a model, you can do so in HDF5:
-```python
-model.save_weights('my_model_weights.h5')
-```
-
-Assuming you have code for instantiating your model, you can then load the weights you saved into a model with the same architecture:
-
-```python
-model.load_weights('my_model_weights.h5')
-```
-
-This leads us to a way to save and reconstruct models from only serialized data:
-```python
-json_string = model.to_json()
-open('my_model_architecture.json', 'w').write(json_string)
-model.save_weights('my_model_weights.h5')
-
-# elsewhere...
-model = model_from_json(open('my_model_architecture.json').read())
-model.load_weights('my_model_weights.h5')
-```
-
----
-
-### Why is the training loss much higher than the testing loss?
-
-A Keras model has two modes: training and testing. Regularization mechanisms, such as Dropout and L1/L2 weight regularization, are turned off at testing time.
-
-Besides, the training loss is the average of the losses over each batch of training data. Because your model is changing over time, the loss over the first batches of an epoch is generally higher than over the last batches. On the other hand, the testing loss for an epoch is computed using the model as it is at the end of the epoch, resulting in a lower loss.
-
----
-
-### How can I visualize the output of an intermediate layer?
-
-You can build a Theano function that will return the output of a certain layer given a certain input, for example:
-
-```python
-# with a Sequential model
-get_3rd_layer_output = theano.function([model.layers[0].input],
-                                       model.layers[3].get_output(train=False))
-layer_output = get_3rd_layer_output(X)
-
-# with a Graph model
-get_conv_layer_output = theano.function([model.inputs[i].input for i in model.input_order],
-                                        model.nodes['conv'].get_output(train=False),
-                                        on_unused_input='ignore')
-conv_output = get_conv_layer_output([input_data_dict[i] for i in model.input_order])
-```
-
----
-
-### Isn't there a bug with Merge or Graph related to input concatenation?
-
-Yes, there was a known bug with tensor concatenation in Theano that was fixed early 2015.
-Please upgrade to the latest version of Theano:
-
-```bash
-sudo pip install git+git://github.com/Theano/Theano.git
-```
-
----
-
-### How can I use Keras with datasets that don't fit in memory?
-
-You can do batch training using `model.train_on_batch(X, y)` and `model.test_on_batch(X, y)`. See the [models documentation](models.md).
-
-You can also see batch training in action in our [CIFAR10 example](https://github.com/fchollet/keras/blob/master/examples/cifar10_cnn.py).
-
----
-
-### How can I interrupt training when the validation loss isn't decreasing anymore?
-
-You can use an `EarlyStopping` callback:
-
-```python
-from keras.callbacks import EarlyStopping
-early_stopping = EarlyStopping(monitor='val_loss', patience=2)
-model.fit(X, y, validation_split=0.2, callbacks=[early_stopping])
-```
-
-Find out more in the [callbacks documentation](callbacks.md).
-
----
-
-### How is the validation split computed?
-
-If you set the `validation_split` argument in `model.fit` to e.g. 0.1, then the validation data used will be the *last 10%* of the data. If you set it to 0.25, it will be the last 25% of the data, etc.
-
-
----
-
-### Is the data shuffled during training?
-
-Yes, if the `shuffle` argument in `model.fit` is set to `True` (which is the default), the training data will be randomly shuffled at each epoch.
-
-Validation data isn't shuffled.
-
----
-
-
-### How can I record the training / validation loss / accuracy at each epoch?
-
-The `model.fit` method returns an `History` callback, which has a `history` attribute containing the lists of successive losses / accuracies.
-
-```python
-hist = model.fit(X, y, validation_split=0.2)
-print(hist.history)
-```
-
----
-
-### How can I use stateful RNNs?
-
-Making a RNN stateful means that the states for the samples of each batch will be reused as initial states for the samples in the next batch.
-
-When using stateful RNNs, it is therefore assumed that:
-
-- all batches have the same number of samples
-- If `X1` and `X2` are successive batches of samples, then `X2[i]` is the follow-up sequence to `X1[i]`, for every `i`.
-
-To use statefulness in RNNs, you need to:
-
-- explicitly specify the batch size you are using, by passing a `batch_input_shape` argument to the first layer in your model. It should be a tuple of integers, e.g. `(32, 10, 16)` for a 32-samples batch of sequences of 10 timesteps with 16 features per timestep.
-- set `stateful=True` in your RNN layer(s).
-
-To reset the states accumulated:
-
-- use `model.reset_states()` to reset the states of all layers in the model
-- use `layer.reset_states()` to reset the states of a specific stateful RNN layer
-
-Example:
-
-```python
-
-X  # this is our input data, of shape (32, 21, 16)
-# we will feed it to our model in sequences of length 10
-
-model = Sequential()
-model.add(LSTM(32, batch_input_shape=(32, 10, 16), stateful=True))
-model.add(Dense(16, activation='softmax'))
-
-model.compile(optimizer='rmsprop', loss='categorical_crossentropy')
-
-# we train the network to predict the 11th timestep given the first 10:
-model.train_on_batch(X[:, :10, :], np.reshape(X[:, 10, :], (32, 16)))
-
-# the state of the network has changed. We can feed the follow-up sequences:
-model.train_on_batch(X[:, 10:20, :], np.reshape(X[:, 20, :], (32, 16)))
-
-# let's reset the states of the LSTM layer:
-model.reset_states()
-
-# another way to do it in this case:
-model.layers[0].reset_states()
-```
-
-Notes that the methods `predict`, `fit`, `train_on_batch`, `predict_classes`, etc. will *all* update the states of the stateful layers in a model. This allows you to do not only stateful training, but also stateful prediction.
-

docs/templates/getting-started/faq.md

@@ -0,0 +1,344 @@
+# Keras FAQ: Frequently Asked Keras Questions
+
+- [How should I cite Keras?](#how-should-i-cite-keras)
+- [How can I run Keras on GPU?](#how-can-i-run-keras-on-gpu)
+- [How can I save a Keras model?](#how-can-i-save-a-keras-model)
+- [Why is the training loss much higher than the testing loss?](#why-is-the-training-loss-much-higher-than-the-testing-loss)
+- [How can I visualize the output of an intermediate layer?](#how-can-i-visualize-the-output-of-an-intermediate-layer)
+- [How can I use Keras with datasets that don't fit in memory?](#how-can-i-use-keras-with-datasets-that-dont-fit-in-memory)
+- [How can I interrupt training when the validation loss isn't decreasing anymore?](#how-can-i-interrupt-training-when-the-validation-loss-isnt-decreasing-anymore)
+- [How is the validation split computed?](#how-is-the-validation-split-computed)
+- [Is the data shuffled during training?](#is-the-data-shuffled-during-training)
+- [How can I record the training / validation loss / accuracy at each epoch?](#how-can-i-record-the-training-validation-loss-accuracy-at-each-epoch)
+- [How can I "freeze" layers?](#how-can-i-freeze-keras-layers)
+- [How can I use stateful RNNs?](#how-can-i-use-stateful-rnns)
+- [How can I remove a layer from a Sequential model?](#how-can-i-remove-a-layer-from-a-sequential-model)
+- [How can I use pre-trained models in Keras?](#how-can-i-use-pre-trained-models-in-keras)
+
+---
+
+### How should I cite Keras?
+
+Please cite Keras in your publications if it helps your research. Here is an example BibTeX entry:
+
+```
+@misc{chollet2015keras,
+  title={Keras},
+  author={Chollet, Fran\c{c}ois},
+  year={2015},
+  publisher={GitHub},
+  howpublished={\url{https://github.com/fchollet/keras}},
+}
+```
+
+### How can I run Keras on GPU?
+
+If you are running on the TensorFlow backend, your code will automatically run on GPU if any available GPU is detected.
+If you are running on the Theano backend, you can use one of the following methods:
+
+Method 1: use Theano flags.
+```bash
+THEANO_FLAGS=device=gpu,floatX=float32 python my_keras_script.py
+```
+
+The name 'gpu' might have to be changed depending on your device's identifier (e.g. `gpu0`, `gpu1`, etc).
+
+Method 2: set up your `.theanorc`: [Instructions](http://deeplearning.net/software/theano/library/config.html)
+
+Method 3: manually set `theano.config.device`, `theano.config.floatX` at the beginning of your code:
+```python
+import theano
+theano.config.device = 'gpu'
+theano.config.floatX = 'float32'
+```
+
+---
+
+### How can I save a Keras model?
+
+*It is not recommended to use pickle or cPickle to save a Keras model.*
+
+You can use `model.save(filepath)` to save a Keras model into a single HDF5 file which will contain:
+
+- the architecture of the model, allowing to re-create the model
+- the weights of the model
+- the training configuration (loss, optimizer)
+- the state of the optimizer, allowing to resume training exactly where you left off.
+
+You can then use `keras.models.load_model(filepath)` to reinstantiate your model.
+`load_model` will also take care of compiling the model using the saved training configuration
+(unless the model was never compiled in the first place).
+
+Example:
+
+```python
+from keras.models import load_model
+
+model.save('my_model.h5')  # creates a HDF5 file 'my_model.h5'
+del model  # deletes the existing model
+
+# returns a compiled model
+# identical to the previous one
+model = load_model('my_model.h5')
+```
+
+If you only need to save the **architecture of a model**, and not its weights or its training configuration, you can do:
+
+```python
+# save as JSON
+json_string = model.to_json()
+
+# save as YAML
+yaml_string = model.to_yaml()
+```
+
+The generated JSON / YAML files are human-readable and can be manually edited if needed.
+
+You can then build a fresh model from this data:
+
+```python
+# model reconstruction from JSON:
+from keras.models import model_from_json
+model = model_from_json(json_string)
+
+# model reconstruction from YAML
+model = model_from_yaml(yaml_string)
+```
+
+If you need to save the **weights of a model**, you can do so in HDF5 with the code below.
+
+Note that you will first need to install HDF5 and the Python library h5py, which do not come bundled with Keras.
+
+```python
+model.save_weights('my_model_weights.h5')
+```
+
+Assuming you have code for instantiating your model, you can then load the weights you saved into a model with the same architecture:
+
+```python
+model.load_weights('my_model_weights.h5')
+```
+
+---
+
+### Why is the training loss much higher than the testing loss?
+
+A Keras model has two modes: training and testing. Regularization mechanisms, such as Dropout and L1/L2 weight regularization, are turned off at testing time.
+
+Besides, the training loss is the average of the losses over each batch of training data. Because your model is changing over time, the loss over the first batches of an epoch is generally higher than over the last batches. On the other hand, the testing loss for an epoch is computed using the model as it is at the end of the epoch, resulting in a lower loss.
+
+---
+
+### How can I visualize the output of an intermediate layer?
+
+You can build a Keras function that will return the output of a certain layer given a certain input, for example:
+
+```python
+from keras import backend as K
+
+# with a Sequential model
+get_3rd_layer_output = K.function([model.layers[0].input],
+                                  [model.layers[3].output])
+layer_output = get_3rd_layer_output([X])[0]
+```
+
+Similarly, you could build a Theano and TensorFlow function directly.
+
+Note that if your model has a different behavior in training and testing phase (e.g. if it uses `Dropout`, `BatchNormalization`, etc.), you will need
+to pass the learning phase flag to your function:
+
+```python
+get_3rd_layer_output = K.function([model.layers[0].input, K.learning_phase()],
+                                  [model.layers[3].output])
+
+# output in test mode = 0
+layer_output = get_3rd_layer_output([X, 0])[0]
+
+# output in train mode = 1
+layer_output = get_3rd_layer_output([X, 1])[0]
+```
+
+Another more flexible way of getting output from intermediate layers is to use the [functional API](/getting-started/functional-api-guide). For example, if you have created an autoencoder for MNIST:
+
+```python
+inputs = Input(shape=(784,))
+encoded = Dense(32, activation='relu')(inputs)
+decoded = Dense(784)(encoded)
+model = Model(input=inputs, output=decoded)
+```
+
+After compiling and training the model, you can get the output of the data from the encoder like this:
+
+```python
+encoder = Model(input=inputs, output=encoded)
+X_encoded = encoder.predict(X)
+```
+
+---
+
+### How can I use Keras with datasets that don't fit in memory?
+
+You can do batch training using `model.train_on_batch(X, y)` and `model.test_on_batch(X, y)`. See the [models documentation](/models/sequential).
+
+Alternatively, you can write a generator that yields batches of training data and use the method `model.fit_generator(data_generator, samples_per_epoch, nb_epoch)`.
+
+You can see batch training in action in our [CIFAR10 example](https://github.com/fchollet/keras/blob/master/examples/cifar10_cnn.py).
+
+---
+
+### How can I interrupt training when the validation loss isn't decreasing anymore?
+
+You can use an `EarlyStopping` callback:
+
+```python
+from keras.callbacks import EarlyStopping
+early_stopping = EarlyStopping(monitor='val_loss', patience=2)
+model.fit(X, y, validation_split=0.2, callbacks=[early_stopping])
+```
+
+Find out more in the [callbacks documentation](/callbacks).
+
+---
+
+### How is the validation split computed?
+
+If you set the `validation_split` argument in `model.fit` to e.g. 0.1, then the validation data used will be the *last 10%* of the data. If you set it to 0.25, it will be the last 25% of the data, etc.
+
+
+---
+
+### Is the data shuffled during training?
+
+Yes, if the `shuffle` argument in `model.fit` is set to `True` (which is the default), the training data will be randomly shuffled at each epoch.
+
+Validation data is never shuffled.
+
+---
+
+
+### How can I record the training / validation loss / accuracy at each epoch?
+
+The `model.fit` method returns an `History` callback, which has a `history` attribute containing the lists of successive losses and other metrics.
+
+```python
+hist = model.fit(X, y, validation_split=0.2)
+print(hist.history)
+```
+
+---
+
+### How can I "freeze" Keras layers?
+
+To "freeze" a layer means to exclude it from training, i.e. its weights will never be updated. This is useful in the context of fine-tuning a model, or using fixed embeddings for a text input.
+
+You can pass a `trainable` argument (boolean) to a layer constructor to set a layer to be non-trainable:
+
+```python
+frozen_layer = Dense(32, trainable=False)
+```
+
+Additionally, you can set the `trainable` property of a layer to `True` or `False` after instantiation. For this to take effect, you will need to call `compile()` on your model after modifying the `trainable` property. Here's an example:
+
+```python
+x = Input(shape=(32,))
+layer = Dense(32)
+layer.trainable = False
+y = layer(x)
+
+frozen_model = Model(x, y)
+# in the model below, the weights of `layer` will not be updated during training
+frozen_model.compile(optimizer='rmsprop', loss='mse')
+
+layer.trainable = True
+trainable_model = Model(x, y)
+# with this model the weights of the layer will be updated during training
+# (which will also affect the above model since it uses the same layer instance)
+trainable_model.compile(optimizer='rmsprop', loss='mse')
+
+frozen_model.fit(data, labels)  # this does NOT update the weights of `layer`
+trainable_model.fit(data, labels)  # this updates the weights of `layer`
+```
+
+---
+
+### How can I use stateful RNNs?
+
+Making a RNN stateful means that the states for the samples of each batch will be reused as initial states for the samples in the next batch.
+
+When using stateful RNNs, it is therefore assumed that:
+
+- all batches have the same number of samples
+- If `X1` and `X2` are successive batches of samples, then `X2[i]` is the follow-up sequence to `X1[i]`, for every `i`.
+
+To use statefulness in RNNs, you need to:
+
+- explicitly specify the batch size you are using, by passing a `batch_input_shape` argument to the first layer in your model. It should be a tuple of integers, e.g. `(32, 10, 16)` for a 32-samples batch of sequences of 10 timesteps with 16 features per timestep.
+- set `stateful=True` in your RNN layer(s).
+
+To reset the states accumulated:
+
+- use `model.reset_states()` to reset the states of all layers in the model
+- use `layer.reset_states()` to reset the states of a specific stateful RNN layer
+
+Example:
+
+```python
+
+X  # this is our input data, of shape (32, 21, 16)
+# we will feed it to our model in sequences of length 10
+
+model = Sequential()
+model.add(LSTM(32, batch_input_shape=(32, 10, 16), stateful=True))
+model.add(Dense(16, activation='softmax'))
+
+model.compile(optimizer='rmsprop', loss='categorical_crossentropy')
+
+# we train the network to predict the 11th timestep given the first 10:
+model.train_on_batch(X[:, :10, :], np.reshape(X[:, 10, :], (32, 16)))
+
+# the state of the network has changed. We can feed the follow-up sequences:
+model.train_on_batch(X[:, 10:20, :], np.reshape(X[:, 20, :], (32, 16)))
+
+# let's reset the states of the LSTM layer:
+model.reset_states()
+
+# another way to do it in this case:
+model.layers[0].reset_states()
+```
+
+Notes that the methods `predict`, `fit`, `train_on_batch`, `predict_classes`, etc. will *all* update the states of the stateful layers in a model. This allows you to do not only stateful training, but also stateful prediction.
+
+---
+
+### How can I remove a layer from a Sequential model?
+
+You can remove the last added layer in a Sequential model by calling `.pop()`:
+
+```python
+model = Sequential()
+model.add(Dense(32, activation='relu', input_dim=784))
+model.add(Dense(32, activation='relu'))
+
+print(len(model.layers))  # "2"
+
+model.pop()
+print(len(model.layers))  # "1"
+```
+
+---
+
+### How can I use pre-trained models in Keras?
+
+Code and pre-trained weights are available for the following image classification models:
+
+- [VGG-16](https://gist.github.com/baraldilorenzo/07d7802847aaad0a35d3)
+- [VGG-19](https://gist.github.com/baraldilorenzo/8d096f48a1be4a2d660d)
+- [AlexNet](https://github.com/heuritech/convnets-keras)
+
+For an example of how to use such a pre-trained model for feature extraction or for fine-tuning, see [this blog post](http://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html).
+
+The VGG-16 model is also the basis for several Keras example scripts:
+
+- [Style transfer](https://github.com/fchollet/keras/blob/master/examples/neural_style_transfer.py)
+- [Feature visualization](https://github.com/fchollet/keras/blob/master/examples/conv_filter_visualization.py)
+- [Deep dream](https://github.com/fchollet/keras/blob/master/examples/deep_dream.py)

docs/templates/getting-started/functional-api-guide.md

@@ -0,0 +1,421 @@
+# Getting started with the Keras functional API
+
+The Keras functional API is the way to go for defining complex models, such as multi-output models, directed acyclic graphs, or models with shared layers.
+
+This guide assumes that you are already familiar with the `Sequential` model.
+
+Let's start with something simple.
+
+-----
+
+## First example: fully connected network
+
+The `Sequential` model is probably a better choice to implement such a network, but it helps to start with something really simple.
+
+- A layer instance is callable (on a tensor), and it returns a tensor
+- Input tensor(s) and output tensor(s) can then be used to define a `Model`
+- Such a model can be trained just like Keras `Sequential` models.
+
+```python
+from keras.layers import Input, Dense
+from keras.models import Model
+
+# this returns a tensor
+inputs = Input(shape=(784,))
+
+# a layer instance is callable on a tensor, and returns a tensor
+x = Dense(64, activation='relu')(inputs)
+x = Dense(64, activation='relu')(x)
+predictions = Dense(10, activation='softmax')(x)
+
+# this creates a model that includes
+# the Input layer and three Dense layers
+model = Model(input=inputs, output=predictions)
+model.compile(optimizer='rmsprop',
+              loss='categorical_crossentropy',
+              metrics=['accuracy'])
+model.fit(data, labels)  # starts training
+```
+
+-----
+
+## All models are callable, just like layers
+
+With the functional API, it is easy to re-use trained models: you can treat any model as if it were a layer, by calling it on a tensor. Note that by calling a model you aren't just re-using the *architecture* of the model, you are also re-using its weights.
+
+```python
+x = Input(shape=(784,))
+# this works, and returns the 10-way softmax we defined above.
+y = model(x)
+```
+
+This can allow, for instance, to quickly create models that can process *sequences* of inputs. You could turn an image classification model into a video classification model, in just one line.
+
+```python
+from keras.layers import TimeDistributed
+
+# input tensor for sequences of 20 timesteps,
+# each containing a 784-dimensional vector
+input_sequences = Input(shape=(20, 784))
+
+# this applies our previous model to every timestep in the input sequences.
+# the output of the previous model was a 10-way softmax,
+# so the output of the layer below will be a sequence of 20 vectors of size 10.
+processed_sequences = TimeDistributed(model)(input_sequences)
+```
+
+-----
+
+## Multi-input and multi-output models
+
+Here's a good use case for the functional API: models with multiple inputs and outputs. The functional API makes it easy to manipulate a large number of intertwined datastreams.
+
+Let's consider the following model. We seek to predict how many retweets and likes a news headline will receive on Twitter. The main input to the model will be the headline itself, as a sequence of words, but to spice things up, our model will also have an auxiliary input, receiving extra data such as the time of day when the headline was posted, etc.
+The model will also be supervised via two loss functions. Using the main loss function earlier in a model is a good regularization mechanism for deep models.
+
+Here's what our model looks like:
+
+<img src="https://s3.amazonaws.com/keras.io/img/multi-input-multi-output-graph.png" alt="multi-input-multi-output-graph" style="width: 400px;"/>
+
+Let's implement it with the functional API.
+
+The main input will receive the headline, as a sequence of integers (each integer encodes a word).
+The integers will be between 1 and 10,000 (a vocabulary of 10,000 words) and the sequences will be 100 words long.
+
+```python
+from keras.layers import Input, Embedding, LSTM, Dense, merge
+from keras.models import Model
+
+# headline input: meant to receive sequences of 100 integers, between 1 and 10000.
+# note that we can name any layer by passing it a "name" argument.
+main_input = Input(shape=(100,), dtype='int32', name='main_input')
+
+# this embedding layer will encode the input sequence
+# into a sequence of dense 512-dimensional vectors.
+x = Embedding(output_dim=512, input_dim=10000, input_length=100)(main_input)
+
+# a LSTM will transform the vector sequence into a single vector,
+# containing information about the entire sequence
+lstm_out = LSTM(32)(x)
+```
+
+Here we insert the auxiliary loss, allowing the LSTM and Embedding layer to be trained smoothly even though the main loss will be much higher in the model.
+
+```python
+auxiliary_loss = Dense(1, activation='sigmoid', name='aux_output')(lstm_out)
+```
+
+At this point, we feed into the model our auxiliary input data by concatenating it with the LSTM output:
+
+```python
+auxiliary_input = Input(shape=(5,), name='aux_input')
+x = merge([lstm_out, auxiliary_input], mode='concat')
+
+# we stack a deep fully-connected network on top
+x = Dense(64, activation='relu')(x)
+x = Dense(64, activation='relu')(x)
+x = Dense(64, activation='relu')(x)
+
+# and finally we add the main logistic regression layer
+main_loss = Dense(1, activation='sigmoid', name='main_output')(x)
+```
+
+This defines a model with two inputs and two outputs:
+
+```python
+model = Model(input=[main_input, auxiliary_input], output=[main_loss, auxiliary_loss])
+```
+
+We compile the model and assign a weight of 0.2 to the auxiliary loss.
+To specify different `loss_weights` or `loss` for each different output, you can use a list or a dictionary.
+Here we pass a single loss as the `loss` argument, so the same loss will be used on all outputs.
+
+```python
+model.compile(optimizer='rmsprop', loss='binary_crossentropy',
+              loss_weights=[1., 0.2])
+```
+
+We can train the model by passing it lists of input arrays and target arrays:
+
+```python
+model.fit([headline_data, additional_data], [labels, labels],
+          nb_epoch=50, batch_size=32)
+```
+
+Since our inputs and outputs are named (we passed them a "name" argument),
+We could also have compiled the model via:
+
+```python
+model.compile(optimizer='rmsprop',
+              loss={'main_output': 'binary_crossentropy', 'aux_output': 'binary_crossentropy'},
+              loss_weights={'main_output': 1., 'aux_output': 0.2})
+
+# and trained it via:
+model.fit({'main_input': headline_data, 'aux_input': additional_data},
+          {'main_output': labels, 'aux_output': labels},
+          nb_epoch=50, batch_size=32)
+```
+
+-----
+
+## Shared layers
+
+Another good use for the functional API are models that use shared layers. Let's take a look at shared layers.
+
+Let's consider a dataset of tweets. We want to build a model that can tell whether two tweets are from the same person or not (this can allow us to compare users by the similarity of their tweets, for instance).
+
+One way to achieve this is to build a model that encodes two tweets into two vectors, concatenates the vectors and adds a logistic regression of top, outputting a probability that the two tweets share the same author. The model would then be trained on positive tweet pairs and negative tweet pairs.
+
+Because the problem is symmetric, the mechanism that encodes the first tweet should be reused (weights and all) to encode the second tweet. Here we use a shared LSTM layer to encode the tweets.
+
+Let's build this with the functional API. We will take as input for a tweet a binary matrix of shape `(140, 256)`, i.e. a sequence of 140 vectors of size 256, where each dimension in the 256-dimensional vector encodes the presence/absence of a character (out of an alphabet of 256 frequent characters).
+
+```python
+from keras.layers import Input, LSTM, Dense, merge
+from keras.models import Model
+
+tweet_a = Input(shape=(140, 256))
+tweet_b = Input(shape=(140, 256))
+```
+
+To share a layer across different inputs, simply instantiate the layer once, then call it on as many inputs as you want:
+
+```python
+# this layer can take as input a matrix
+# and will return a vector of size 64
+shared_lstm = LSTM(64)
+
+# when we reuse the same layer instance
+# multiple times, the weights of the layer
+# are also being reused
+# (it is effectively *the same* layer)
+encoded_a = shared_lstm(tweet_a)
+encoded_b = shared_lstm(tweet_b)
+
+# we can then concatenate the two vectors:
+merged_vector = merge([encoded_a, encoded_b], mode='concat', concat_axis=-1)
+
+# and add a logistic regression on top
+predictions = Dense(1, activation='sigmoid')(merged_vector)
+
+# we define a trainable model linking the
+# tweet inputs to the predictions
+model = Model(input=[tweet_a, tweet_b], output=predictions)
+
+model.compile(optimizer='rmsprop',
+              loss='binary_crossentropy',
+              metrics=['accuracy'])
+model.fit([data_a, data_b], labels, nb_epoch=10)
+```
+
+Let's pause to take a look at how to read the shared layer's output or output shape.
+
+-----
+
+## The concept of layer "node"
+
+Whenever you are calling a layer on some input, you are creating a new tensor (the output of the layer), and you are adding a "node" to the layer, linking the input tensor to the output tensor. When you are calling the same layer multiple times, that layer owns multiple nodes indexed as 0, 1, 2...
+
+In previous versions of Keras, you could obtain the output tensor of a layer instance via `layer.get_output()`, or its output shape via `layer.output_shape`. You still can (except `get_output()` has been replaced by the property `output`). But what if a layer is connected to multiple inputs?
+
+As long as a layer is only connected to one input, there is no confusion, and `.output` will return the one output of the layer:
+
+```python
+a = Input(shape=(140, 256))
+
+lstm = LSTM(32)
+encoded_a = lstm(a)
+
+assert lstm.output == encoded_a
+```
+
+Not so if the layer has multiple inputs:
+```python
+a = Input(shape=(140, 256))
+b = Input(shape=(140, 256))
+
+lstm = LSTM(32)
+encoded_a = lstm(a)
+encoded_b = lstm(b)
+
+lstm.output
+```
+```
+>> AssertionError: Layer lstm_1 has multiple inbound nodes,
+hence the notion of "layer output" is ill-defined.
+Use `get_output_at(node_index)` instead.
+```
+
+Okay then. The following works:
+
+```python
+assert lstm.get_output_at(0) == encoded_a
+assert lstm.get_output_at(1) == encoded_b
+```
+
+Simple enough, right?
+
+The same is true for the properties `input_shape` and `output_shape`: as long as the layer has only one node, or as long as all nodes have the same input/output shape, then the notion of "layer output/input shape" is well defined, and that one shape will be returned by `layer.output_shape`/`layer.input_shape`. But if, for instance, you apply a same `Convolution2D` layer to an input of shape `(3, 32, 32)`, and then to an input of shape `(3, 64, 64)`, the layer will have multiple input/output shapes, and you will have to fetch them by specifying the index of the node they belong to:
+
+```python
+a = Input(shape=(3, 32, 32))
+b = Input(shape=(3, 64, 64))
+
+conv = Convolution2D(16, 3, 3, border_mode='same')
+conved_a = conv(a)
+
+# only one input so far, the following will work:
+assert conv.input_shape == (None, 3, 32, 32)
+
+conved_b = conv(b)
+# now the `.input_shape` property wouldn't work, but this does:
+assert conv.get_input_shape_at(0) == (None, 3, 32, 32)
+assert conv.get_input_shape_at(1) == (None, 3, 64, 64)
+```
+
+-----
+
+## More examples
+
+Code examples are still the best way to get started, so here are a few more.
+
+### Inception module
+
+For more information about the Inception architecture, see [Going Deeper with Convolutions](http://arxiv.org/abs/1409.4842).
+
+```python
+from keras.layers import merge, Convolution2D, MaxPooling2D, Input
+
+input_img = Input(shape=(3, 256, 256))
+
+tower_1 = Convolution2D(64, 1, 1, border_mode='same', activation='relu')(input_img)
+tower_1 = Convolution2D(64, 3, 3, border_mode='same', activation='relu')(tower_1)
+
+tower_2 = Convolution2D(64, 1, 1, border_mode='same', activation='relu')(input_img)
+tower_2 = Convolution2D(64, 5, 5, border_mode='same', activation='relu')(tower_2)
+
+tower_3 = MaxPooling2D((3, 3), strides=(1, 1), border_mode='same')(input_img)
+tower_3 = Convolution2D(64, 1, 1, border_mode='same', activation='relu')(tower_3)
+
+output = merge([tower_1, tower_2, tower_3], mode='concat', concat_axis=1)
+```
+
+### Residual connection on a convolution layer
+
+For more information about residual networks, see [Deep Residual Learning for Image Recognition](http://arxiv.org/abs/1512.03385).
+
+```python
+from keras.layers import merge, Convolution2D, Input
+
+# input tensor for a 3-channel 256x256 image
+x = Input(shape=(3, 256, 256))
+# 3x3 conv with 3 output channels (same as input channels)
+y = Convolution2D(3, 3, 3, border_mode='same')(x)
+# this returns x + y.
+z = merge([x, y], mode='sum')
+```
+
+### Shared vision model
+
+This model re-uses the same image-processing module on two inputs, to classify whether two MNIST digits are the same digit or different digits.
+
+```python
+from keras.layers import merge, Convolution2D, MaxPooling2D, Input, Dense, Flatten
+from keras.models import Model
+
+# first, define the vision modules
+digit_input = Input(shape=(1, 27, 27))
+x = Convolution2D(64, 3, 3)(digit_input)
+x = Convolution2D(64, 3, 3)(x)
+x = MaxPooling2D((2, 2))(x)
+out = Flatten()(x)
+
+vision_model = Model(digit_input, out)
+
+# then define the tell-digits-apart model
+digit_a = Input(shape=(1, 27, 27))
+digit_b = Input(shape=(1, 27, 27))
+
+# the vision model will be shared, weights and all
+out_a = vision_model(digit_a)
+out_b = vision_model(digit_b)
+
+concatenated = merge([out_a, out_b], mode='concat')
+out = Dense(1, activation='sigmoid')(concatenated)
+
+classification_model = Model([digit_a, digit_b], out)
+```
+
+### Visual question answering model
+
+This model can select the correct one-word answer when asked a natural-language question about a picture.
+
+It works by encoding the question into a vector, encoding the image into a vector, concatenating the two, and training on top a logistic regression over some vocabulary of potential answers.
+
+```python
+from keras.layers import Convolution2D, MaxPooling2D, Flatten
+from keras.layers import Input, LSTM, Embedding, Dense, merge
+from keras.models import Model, Sequential
+
+# first, let's define a vision model using a Sequential model.
+# this model will encode an image into a vector.
+vision_model = Sequential()
+vision_model.add(Convolution2D(64, 3, 3, activation='relu', border_mode='same', input_shape=(3, 224, 224)))
+vision_model.add(Convolution2D(64, 3, 3, activation='relu'))
+vision_model.add(MaxPooling2D((2, 2)))
+vision_model.add(Convolution2D(128, 3, 3, activation='relu', border_mode='same'))
+vision_model.add(Convolution2D(128, 3, 3, activation='relu'))
+vision_model.add(MaxPooling2D((2, 2)))
+vision_model.add(Convolution2D(256, 3, 3, activation='relu', border_mode='same'))
+vision_model.add(Convolution2D(256, 3, 3, activation='relu'))
+vision_model.add(Convolution2D(256, 3, 3, activation='relu'))
+vision_model.add(MaxPooling2D((2, 2)))
+vision_model.add(Flatten())
+
+# now let's get a tensor with the output of our vision model:
+image_input = Input(shape=(3, 224, 224))
+encoded_image = vision_model(image_input)
+
+# next, let's define a language model to encode the question into a vector.
+# each question will be at most 100 word long,
+# and we will index words as integers from 1 to 9999.
+question_input = Input(shape=(100,), dtype='int32')
+embedded_question = Embedding(input_dim=10000, output_dim=256, input_length=100)(question_input)
+encoded_question = LSTM(256)(embedded_question)
+
+# let's concatenate the question vector and the image vector:
+merged = merge([encoded_question, encoded_image], mode='concat')
+
+# and let's train a logistic regression over 1000 words on top:
+output = Dense(1000, activation='softmax')(merged)
+
+# this is our final model:
+vqa_model = Model(input=[image_input, question_input], output=output)
+
+# the next stage would be training this model on actual data.
+```
+
+### Video question answering model
+
+Now that we have trained our image QA model, we can quickly turn it into a video QA model. With appropriate training, you will be able to show it a short video (e.g. 100-frame human action) and ask a natural language question about the video (e.g. "what sport is the boy playing?" -> "football").
+
+```python
+from keras.layers import TimeDistributed
+
+video_input = Input(shape=(100, 3, 224, 224))
+# this is our video encoded via the previously trained vision_model (weights are reused)
+encoded_frame_sequence = TimeDistributed(vision_model)(video_input)  # the output will be a sequence of vectors
+encoded_video = LSTM(256)(encoded_frame_sequence)  # the output will be a vector
+
+# this is a model-level representation of the question encoder, reusing the same weights as before:
+question_encoder = Model(input=question_input, output=encoded_question)
+
+# let's use it to encode the question:
+video_question_input = Input(shape=(100,), dtype='int32')
+encoded_video_question = question_encoder(video_question_input)
+
+# and this is our video question answering model:
+merged = merge([encoded_video, encoded_video_question], mode='concat')
+output = Dense(1000, activation='softmax')(merged)
+video_qa_model = Model(input=[video_input, video_question_input], output=output)
+```

docs/templates/getting-started/sequential-model-guide.md

@@ -0,0 +1,549 @@
+# Getting started with the Keras Sequential model
+
+The `Sequential` model is a linear stack of layers.
+
+You can create a `Sequential` model by passing a list of layer instances to the constructor:
+
+```python
+from keras.models import Sequential
+from keras.layers import Dense, Activation
+
+model = Sequential([
+    Dense(32, input_dim=784),
+    Activation('relu'),
+    Dense(10),
+    Activation('softmax'),
+])
+```
+
+You can also simply add layers via the `.add()` method:
+
+```python
+model = Sequential()
+model.add(Dense(32, input_dim=784))
+model.add(Activation('relu'))
+```
+
+----
+
+## Specifying the input shape
+
+The model needs to know what input shape it should expect. For this reason, the first layer in a `Sequential` model (and only the first, because following layers can do automatic shape inference) needs to receive information about its input shape. There are several possible ways to do this:
+
+- pass an `input_shape` argument to the first layer. This is a shape tuple (a tuple of integers or `None` entries, where `None` indicates that any positive integer may be expected). In `input_shape`, the batch dimension is not included.
+- pass instead a `batch_input_shape` argument, where the batch dimension is included. This is useful for specifying a fixed batch size (e.g. with stateful RNNs).
+- some 2D layers, such as `Dense`, support the specification of their input shape via the argument `input_dim`, and some 3D temporal layers support the arguments `input_dim` and `input_length`.
+
+As such, the following three snippets are strictly equivalent:
+```python
+model = Sequential()
+model.add(Dense(32, input_shape=(784,)))
+```
+```python
+model = Sequential()
+model.add(Dense(32, batch_input_shape=(None, 784)))
+# note that batch dimension is "None" here,
+# so the model will be able to process batches of any size.
+```
+```python
+model = Sequential()
+model.add(Dense(32, input_dim=784))
+```
+
+And so are the following three snippets:
+```python
+model = Sequential()
+model.add(LSTM(32, input_shape=(10, 64)))
+```
+```python
+model = Sequential()
+model.add(LSTM(32, batch_input_shape=(None, 10, 64)))
+```
+```python
+model = Sequential()
+model.add(LSTM(32, input_length=10, input_dim=64))
+```
+
+----
+
+## The Merge layer
+
+Multiple `Sequential` instances can be merged into a single output via a `Merge` layer. The output is a layer that can be added as first layer in a new `Sequential` model. For instance, here's a model with two separate input branches getting merged:
+
+```python
+from keras.layers import Merge
+
+left_branch = Sequential()
+left_branch.add(Dense(32, input_dim=784))
+
+right_branch = Sequential()
+right_branch.add(Dense(32, input_dim=784))
+
+merged = Merge([left_branch, right_branch], mode='concat')
+
+final_model = Sequential()
+final_model.add(merged)
+final_model.add(Dense(10, activation='softmax'))
+```
+
+<img src="https://s3.amazonaws.com/keras.io/img/two_branches_sequential_model.png" alt="two branch Sequential" style="width: 400px;"/>
+
+Such a two-branch model can then be trained via e.g.:
+
+```python
+final_model.compile(optimizer='rmsprop', loss='categorical_crossentropy')
+final_model.fit([input_data_1, input_data_2], targets)  # we pass one data array per model input
+```
+
+The `Merge` layer supports a number of pre-defined modes:
+
+- `sum` (default): element-wise sum
+- `concat`: tensor concatenation. You can specify the concatenation axis via the argument `concat_axis`.
+- `mul`: element-wise multiplication
+- `ave`: tensor average
+- `dot`: dot product. You can specify which axes to reduce along via the argument `dot_axes`.
+- `cos`: cosine proximity between vectors in 2D tensors.
+
+You can also pass a function as the `mode` argument, allowing for arbitrary transformations:
+
+```python
+merged = Merge([left_branch, right_branch], mode=lambda x, y: x - y)
+```
+
+Now you know enough to be able to define *almost* any model with Keras. For complex models that cannot be expressed via `Sequential` and `Merge`, you can use [the functional API](/getting-started/functional-api-guide).
+
+
+----
+
+## Compilation
+
+Before training a model, you need to configure the learning process, which is done via the `compile` method. It receives three arguments:
+
+- an optimizer. This could be the string identifier of an existing optimizer (such as `rmsprop` or `adagrad`), or an instance of the `Optimizer` class. See: [optimizers](/optimizers).
+- a loss function. This is the objective that the model will try to minimize. It can be the string identifier of an existing loss function (such as `categorical_crossentropy` or `mse`), or it can be an objective function. See: [objectives](/objectives).
+- a list of metrics. For any classification problem you will want to set this to `metrics=['accuracy']`. A metric could be the string identifier of an existing metric (only `accuracy` is supported at this point), or a custom metric function.
+
+```python
+# for a multi-class classification problem
+model.compile(optimizer='rmsprop',
+              loss='categorical_crossentropy',
+              metrics=['accuracy'])
+
+# for a binary classification problem
+model.compile(optimizer='rmsprop',
+              loss='binary_crossentropy',
+              metrics=['accuracy'])
+
+# for a mean squared error regression problem
+model.compile(optimizer='rmsprop',
+              loss='mse')
+```
+
+----
+
+## Training
+
+Keras models are trained on Numpy arrays of input data and labels. For training a model, you will typically use the `fit` function. [Read its documentation here](/models/sequential). 
+
+```python
+# for a single-input model with 2 classes (binary):
+
+model = Sequential()
+model.add(Dense(1, input_dim=784, activation='sigmoid'))
+model.compile(optimizer='rmsprop',
+              loss='binary_crossentropy',
+              metrics=['accuracy'])
+
+# generate dummy data
+import numpy as np
+data = np.random.random((1000, 784))
+labels = np.random.randint(2, size=(1000, 1))
+
+# train the model, iterating on the data in batches
+# of 32 samples
+model.fit(data, labels, nb_epoch=10, batch_size=32)
+```
+```python
+# for a multi-input model with 10 classes:
+
+left_branch = Sequential()
+left_branch.add(Dense(32, input_dim=784))
+
+right_branch = Sequential()
+right_branch.add(Dense(32, input_dim=784))
+
+merged = Merge([left_branch, right_branch], mode='concat')
+
+model = Sequential()
+model.add(merged)
+model.add(Dense(10, activation='softmax'))
+
+model.compile(optimizer='rmsprop',
+              loss='categorical_crossentropy',
+              metrics=['accuracy'])
+
+# generate dummy data
+import numpy as np
+from keras.utils.np_utils import to_categorical
+data_1 = np.random.random((1000, 784))
+data_2 = np.random.random((1000, 784))
+
+# these are integers between 0 and 9
+labels = np.random.randint(10, size=(1000, 1))
+# we convert the labels to a binary matrix of size (1000, 10)
+# for use with categorical_crossentropy
+labels = to_categorical(labels, 10)
+
+# train the model
+# note that we are passing a list of Numpy arrays as training data
+# since the model has 2 inputs
+model.fit([data_1, data_2], labels, nb_epoch=10, batch_size=32)
+```
+
+----
+
+
+## Examples
+
+Here are a few examples to get you started!
+
+In the examples folder, you will also find example models for real datasets:
+
+- CIFAR10 small images classification: Convolutional Neural Network (CNN) with realtime data augmentation
+- IMDB movie review sentiment classification: LSTM over sequences of words
+- Reuters newswires topic classification: Multilayer Perceptron (MLP)
+- MNIST handwritten digits classification: MLP & CNN
+- Character-level text generation with LSTM
+
+...and more.
+
+
+### Multilayer Perceptron (MLP) for multi-class softmax classification:
+
+```python
+from keras.models import Sequential
+from keras.layers import Dense, Dropout, Activation
+from keras.optimizers import SGD
+
+model = Sequential()
+# Dense(64) is a fully-connected layer with 64 hidden units.
+# in the first layer, you must specify the expected input data shape:
+# here, 20-dimensional vectors.
+model.add(Dense(64, input_dim=20, init='uniform'))
+model.add(Activation('tanh'))
+model.add(Dropout(0.5))
+model.add(Dense(64, init='uniform'))
+model.add(Activation('tanh'))
+model.add(Dropout(0.5))
+model.add(Dense(10, init='uniform'))
+model.add(Activation('softmax'))
+
+sgd = SGD(lr=0.1, decay=1e-6, momentum=0.9, nesterov=True)
+model.compile(loss='categorical_crossentropy',
+              optimizer=sgd,
+              metrics=['accuracy'])
+
+model.fit(X_train, y_train,
+          nb_epoch=20,
+          batch_size=16)
+score = model.evaluate(X_test, y_test, batch_size=16)
+```
+
+
+### Alternative implementation of a similar MLP:
+
+```python
+model = Sequential()
+model.add(Dense(64, input_dim=20, activation='relu'))
+model.add(Dropout(0.5))
+model.add(Dense(64, activation='relu'))
+model.add(Dropout(0.5))
+model.add(Dense(10, activation='softmax'))
+
+model.compile(loss='categorical_crossentropy',
+              optimizer='adadelta',
+              metrics=['accuracy'])
+```
+
+
+### MLP for binary classification:
+```python
+model = Sequential()
+model.add(Dense(64, input_dim=20, init='uniform', activation='relu'))
+model.add(Dropout(0.5))
+model.add(Dense(64, activation='relu'))
+model.add(Dropout(0.5))
+model.add(Dense(1, activation='sigmoid'))
+
+model.compile(loss='binary_crossentropy',
+              optimizer='rmsprop',
+              metrics=['accuracy'])
+```
+
+
+### VGG-like convnet:
+
+```python
+from keras.models import Sequential
+from keras.layers import Dense, Dropout, Activation, Flatten
+from keras.layers import Convolution2D, MaxPooling2D
+from keras.optimizers import SGD
+
+model = Sequential()
+# input: 100x100 images with 3 channels -> (3, 100, 100) tensors.
+# this applies 32 convolution filters of size 3x3 each.
+model.add(Convolution2D(32, 3, 3, border_mode='valid', input_shape=(3, 100, 100)))
+model.add(Activation('relu'))
+model.add(Convolution2D(32, 3, 3))
+model.add(Activation('relu'))
+model.add(MaxPooling2D(pool_size=(2, 2)))
+model.add(Dropout(0.25))
+
+model.add(Convolution2D(64, 3, 3, border_mode='valid'))
+model.add(Activation('relu'))
+model.add(Convolution2D(64, 3, 3))
+model.add(Activation('relu'))
+model.add(MaxPooling2D(pool_size=(2, 2)))
+model.add(Dropout(0.25))
+
+model.add(Flatten())
+# Note: Keras does automatic shape inference.
+model.add(Dense(256))
+model.add(Activation('relu'))
+model.add(Dropout(0.5))
+
+model.add(Dense(10))
+model.add(Activation('softmax'))
+
+sgd = SGD(lr=0.1, decay=1e-6, momentum=0.9, nesterov=True)
+model.compile(loss='categorical_crossentropy', optimizer=sgd)
+
+model.fit(X_train, Y_train, batch_size=32, nb_epoch=1)
+```
+
+
+### Sequence classification with LSTM:
+
+```python
+from keras.models import Sequential
+from keras.layers import Dense, Dropout, Activation
+from keras.layers import Embedding
+from keras.layers import LSTM
+
+model = Sequential()
+model.add(Embedding(max_features, 256, input_length=maxlen))
+model.add(LSTM(output_dim=128, activation='sigmoid', inner_activation='hard_sigmoid'))
+model.add(Dropout(0.5))
+model.add(Dense(1))
+model.add(Activation('sigmoid'))
+
+model.compile(loss='binary_crossentropy',
+              optimizer='rmsprop',
+              metrics=['accuracy'])
+
+model.fit(X_train, Y_train, batch_size=16, nb_epoch=10)
+score = model.evaluate(X_test, Y_test, batch_size=16)
+```
+
+### Architecture for learning image captions with a convnet and a Gated Recurrent Unit:
+(word-level embedding, caption of maximum length 16 words).
+
+Note that getting this to work well will require using a bigger convnet, initialized with pre-trained weights.
+
+```python
+max_caption_len = 16
+vocab_size = 10000
+
+# first, let's define an image model that
+# will encode pictures into 128-dimensional vectors.
+# it should be initialized with pre-trained weights.
+image_model = Sequential()
+image_model.add(Convolution2D(32, 3, 3, border_mode='valid', input_shape=(3, 100, 100)))
+image_model.add(Activation('relu'))
+image_model.add(Convolution2D(32, 3, 3))
+image_model.add(Activation('relu'))
+image_model.add(MaxPooling2D(pool_size=(2, 2)))
+
+image_model.add(Convolution2D(64, 3, 3, border_mode='valid'))
+image_model.add(Activation('relu'))
+image_model.add(Convolution2D(64, 3, 3))
+image_model.add(Activation('relu'))
+image_model.add(MaxPooling2D(pool_size=(2, 2)))
+
+image_model.add(Flatten())
+image_model.add(Dense(128))
+
+# let's load the weights from a save file.
+image_model.load_weights('weight_file.h5')
+
+# next, let's define a RNN model that encodes sequences of words
+# into sequences of 128-dimensional word vectors.
+language_model = Sequential()
+language_model.add(Embedding(vocab_size, 256, input_length=max_caption_len))
+language_model.add(GRU(output_dim=128, return_sequences=True))
+language_model.add(TimeDistributed(Dense(128)))
+
+# let's repeat the image vector to turn it into a sequence.
+image_model.add(RepeatVector(max_caption_len))
+
+# the output of both models will be tensors of shape (samples, max_caption_len, 128).
+# let's concatenate these 2 vector sequences.
+model = Sequential()
+model.add(Merge([image_model, language_model], mode='concat', concat_axis=-1))
+# let's encode this vector sequence into a single vector
+model.add(GRU(256, return_sequences=False))
+# which will be used to compute a probability
+# distribution over what the next word in the caption should be!
+model.add(Dense(vocab_size))
+model.add(Activation('softmax'))
+
+model.compile(loss='categorical_crossentropy', optimizer='rmsprop')
+
+# "images" is a numpy float array of shape (nb_samples, nb_channels=3, width, height).
+# "captions" is a numpy integer array of shape (nb_samples, max_caption_len)
+# containing word index sequences representing partial captions.
+# "next_words" is a numpy float array of shape (nb_samples, vocab_size)
+# containing a categorical encoding (0s and 1s) of the next word in the corresponding
+# partial caption.
+model.fit([images, partial_captions], next_words, batch_size=16, nb_epoch=100)
+```
+
+
+### Stacked LSTM for sequence classification
+
+In this model, we stack 3 LSTM layers on top of each other,
+making the model capable of learning higher-level temporal representations.
+
+The first two LSTMs return their full output sequences, but the last one only returns
+the last step in its output sequence, thus dropping the temporal dimension
+(i.e. converting the input sequence into a single vector).
+
+<img src="https://keras.io/img/regular_stacked_lstm.png" alt="stacked LSTM" style="width: 300px;"/>
+
+```python
+from keras.models import Sequential
+from keras.layers import LSTM, Dense
+import numpy as np
+
+data_dim = 16
+timesteps = 8
+nb_classes = 10
+
+# expected input data shape: (batch_size, timesteps, data_dim)
+model = Sequential()
+model.add(LSTM(32, return_sequences=True,
+               input_shape=(timesteps, data_dim)))  # returns a sequence of vectors of dimension 32
+model.add(LSTM(32, return_sequences=True))  # returns a sequence of vectors of dimension 32
+model.add(LSTM(32))  # return a single vector of dimension 32
+model.add(Dense(10, activation='softmax'))
+
+model.compile(loss='categorical_crossentropy',
+              optimizer='rmsprop',
+              metrics=['accuracy'])
+
+# generate dummy training data
+x_train = np.random.random((1000, timesteps, data_dim))
+y_train = np.random.random((1000, nb_classes))
+
+# generate dummy validation data
+x_val = np.random.random((100, timesteps, data_dim))
+y_val = np.random.random((100, nb_classes))
+
+model.fit(x_train, y_train,
+          batch_size=64, nb_epoch=5,
+          validation_data=(x_val, y_val))
+```
+
+
+### Same stacked LSTM model, rendered "stateful"
+
+A stateful recurrent model is one for which the internal states (memories) obtained after processing a batch
+of samples are reused as initial states for the samples of the next batch. This allows to process longer sequences
+while keeping computational complexity manageable.
+
+[You can read more about stateful RNNs in the FAQ.](/faq/#how-can-i-use-stateful-rnns)
+
+```python
+from keras.models import Sequential
+from keras.layers import LSTM, Dense
+import numpy as np
+
+data_dim = 16
+timesteps = 8
+nb_classes = 10
+batch_size = 32
+
+# expected input batch shape: (batch_size, timesteps, data_dim)
+# note that we have to provide the full batch_input_shape since the network is stateful.
+# the sample of index i in batch k is the follow-up for the sample i in batch k-1.
+model = Sequential()
+model.add(LSTM(32, return_sequences=True, stateful=True,
+               batch_input_shape=(batch_size, timesteps, data_dim)))
+model.add(LSTM(32, return_sequences=True, stateful=True))
+model.add(LSTM(32, stateful=True))
+model.add(Dense(10, activation='softmax'))
+
+model.compile(loss='categorical_crossentropy',
+              optimizer='rmsprop',
+              metrics=['accuracy'])
+
+# generate dummy training data
+x_train = np.random.random((batch_size * 10, timesteps, data_dim))
+y_train = np.random.random((batch_size * 10, nb_classes))
+
+# generate dummy validation data
+x_val = np.random.random((batch_size * 3, timesteps, data_dim))
+y_val = np.random.random((batch_size * 3, nb_classes))
+
+model.fit(x_train, y_train,
+          batch_size=batch_size, nb_epoch=5,
+          validation_data=(x_val, y_val))
+```
+
+
+### Two merged LSTM encoders for classification over two parallel sequences
+
+In this model, two input sequences are encoded into vectors by two separate LSTM modules.
+
+These two vectors are then concatenated, and a fully connected network is trained on top of the concatenated representations.
+
+<img src="https://keras.io/img/dual_lstm.png" alt="Dual LSTM" style="width: 600px;"/>
+
+```python
+from keras.models import Sequential
+from keras.layers import Merge, LSTM, Dense
+import numpy as np
+
+data_dim = 16
+timesteps = 8
+nb_classes = 10
+
+encoder_a = Sequential()
+encoder_a.add(LSTM(32, input_shape=(timesteps, data_dim)))
+
+encoder_b = Sequential()
+encoder_b.add(LSTM(32, input_shape=(timesteps, data_dim)))
+
+decoder = Sequential()
+decoder.add(Merge([encoder_a, encoder_b], mode='concat'))
+decoder.add(Dense(32, activation='relu'))
+decoder.add(Dense(nb_classes, activation='softmax'))
+
+decoder.compile(loss='categorical_crossentropy',
+                optimizer='rmsprop',
+                metrics=['accuracy'])
+
+# generate dummy training data
+x_train_a = np.random.random((1000, timesteps, data_dim))
+x_train_b = np.random.random((1000, timesteps, data_dim))
+y_train = np.random.random((1000, nb_classes))
+
+# generate dummy validation data
+x_val_a = np.random.random((100, timesteps, data_dim))
+x_val_b = np.random.random((100, timesteps, data_dim))
+y_val = np.random.random((100, nb_classes))
+
+decoder.fit([x_train_a, x_train_b], y_train,
+            batch_size=64, nb_epoch=5,
+            validation_data=([x_val_a, x_val_b], y_val))
+```

docs/templates/index.md

@@ -33,11 +33,12 @@ Keras is compatible with: __Python 2.7-3.5__.
 ------------------
 
 
+
 ## Getting started: 30 seconds to Keras
 
-The core datastructure of Keras is a __model__, a way to organize layers. There are two types of models: [`Sequential`](http://keras.io/models/#sequential) and [`Graph`](http://keras.io/models/#graph).
+The core data structure of Keras is a __model__, a way to organize layers. The main type of model is the [`Sequential`](http://keras.io/getting-started/sequential-model-guide) model, a linear stack of layers. For more complex architectures, you should use the [Keras functional API](http://keras.io/getting-started/functional-api-guide).
 
-Here's the `Sequential` model (a linear pile of layers):
+Here's the `Sequential` model:
 
 ```python
 from keras.models import Sequential
@@ -48,17 +49,17 @@ model = Sequential()
 Stacking layers is as easy as `.add()`:
 
 ```python
-from keras.layers.core import Dense, Activation
+from keras.layers import Dense, Activation
 
-model.add(Dense(output_dim=64, input_dim=100, init="glorot_uniform"))
+model.add(Dense(output_dim=64, input_dim=100))
 model.add(Activation("relu"))
-model.add(Dense(output_dim=10, init="glorot_uniform"))
+model.add(Dense(output_dim=10))
 model.add(Activation("softmax"))
 ```
 
 Once your model looks good, configure its learning process with `.compile()`:
 ```python
-model.compile(loss='categorical_crossentropy', optimizer='sgd')
+model.compile(loss='categorical_crossentropy', optimizer='sgd', metrics=['accuracy'])
 ```
 
 If you need to, you can further configure your optimizer. A core principle of Keras is to make things reasonably simple, while allowing the user to be fully in control when they need to (the ultimate control being the easy extensibility of the source code).
@@ -79,7 +80,7 @@ model.train_on_batch(X_batch, Y_batch)
 
 Evaluate your performance in one line:
 ```python
-objective_score = model.evaluate(X_test, Y_test, batch_size=32)
+loss_and_metrics = model.evaluate(X_test, Y_test, batch_size=32)
 ```
 
 Or generate predictions on new data:
@@ -88,12 +89,14 @@ classes = model.predict_classes(X_test, batch_size=32)
 proba = model.predict_proba(X_test, batch_size=32)
 ```
 
-Building a network of LSTMs, a deep CNN, a Neural Turing Machine, a word2vec embedder or any other model is just as fast. The ideas behind deep learning are simple, so why should their implementation be painful?
+Building a question answering system, an image classification model, a Neural Turing Machine, a word2vec embedder or any other model is just as fast. The ideas behind deep learning are simple, so why should their implementation be painful?
 
-Have a look at these [starter examples](http://keras.io/examples/).
+For a more in-depth tutorial about Keras, you can check out:
 
-In the [examples folder](https://github.com/fchollet/keras/tree/master/examples) of the repo, you will find more advanced models: question-answering with memory networks, text generation with stacked LSTMs, neural turing machines, etc.
+- [Getting started with the Sequential model](http://keras.io/getting-started/sequential-model-guide)
+- [Getting started with the functional API](http://keras.io/getting-started/functional-api-guide)
 
+In the [examples folder](https://github.com/fchollet/keras/tree/master/examples) of the repository, you will find more advanced models: question-answering with memory networks, text generation with stacked LSTMs, etc.
 
 ------------------
 
@@ -112,11 +115,6 @@ Keras uses the following dependencies:
 - Theano
     - [See installation instructions](http://deeplearning.net/software/theano/install.html#install).
 
-**Note**: You should use the latest version of Theano, not the PyPI version. Install it with:
-```
-sudo pip install git+git://github.com/Theano/Theano.git
-```
-
 *When using the TensorFlow backend:*
 
 - TensorFlow

docs/templates/initializations.md

@@ -1,7 +1,7 @@
 
 ## Usage of initializations
 
-Initializations define the probability distribution used to set the initial random weights of Keras layers.
+Initializations define the way to set the initial random weights of Keras layers.
 
 The keyword arguments used for passing initializations to layers will depend on the layer. Usually it is simply `init`:
 
@@ -21,3 +21,30 @@ model.add(Dense(64, init='uniform'))
 - __glorot_uniform__
 - __he_normal__: Gaussian initialization scaled by fan_in (He et al., 2014)
 - __he_uniform__
+
+
+An initialization may be passed as a string (must match one of the available initializations above), or as a callable.
+If a callable, then it must take two arguments: `shape` (shape of the variable to initialize) and `name` (name of the variable),
+and it must return a variable (e.g. output of `K.variable()`):
+
+```python
+from keras import backend as K
+import numpy as np
+
+def my_init(shape, name=None):
+    value = np.random.random(shape)
+    return K.variable(value, name=name)
+
+model.add(Dense(64, init=my_init))
+```
+
+You could also use functions from `keras.initializations` in this way:
+
+```python
+from keras import initializations
+
+def my_init(shape, name=None):
+    return initializations.normal(shape, scale=0.01, name=name)
+
+model.add(Dense(64, init=my_init))
+```

docs/templates/layers/about-keras-layers.md

@@ -0,0 +1,27 @@
+# About Keras layers
+
+All Keras layers have a number of methods in common:
+
+- `layer.get_weights()`: returns the weights of the layer as a list of Numpy arrays.
+- `layer.set_weights(weights)`: sets the weights of the layer from a list of Numpy arrays (with the same shapes as the output of `get_weights`).
+- `layer.get_config()`: returns a dictionary containing the configuration of the layer. The layer can be reinstantiated from its config via:
+```python
+from keras.utils.layer_utils import layer_from_config
+
+config = layer.get_config()
+layer = layer_from_config(config)
+```
+
+If a layer has a single node (i.e. if it isn't a shared layer), you can get its input tensor, output tensor, input shape and output shape via:
+
+- `layer.input`
+- `layer.output`
+- `layer.input_shape`
+- `layer.output_shape`
+
+If the layer has multiple nodes (see: [the concept of layer node and shared layers](/getting-started/functional-api-guide/#the-concept-of-layer-node)), you can use the following methods:
+
+- `layer.get_input_at(node_index)`
+- `layer.get_output_at(node_index)`
+- `layer.get_input_shape_at(node_index)`
+- `layer.get_output_shape_at(node_index)`

docs/templates/layers/writing-your-own-keras-layers.md

@@ -0,0 +1,34 @@
+# Writing your own Keras layers
+
+For simple, stateless custom operations, you are probably better off using `layers.core.Lambda` layers. But for any custom operation that has trainable weights, you should implement your own layer.
+
+Here is the skeleton of a Keras layer. There are only three methods you need to implement:
+
+- `build(input_shape)`: this is where you will define your weights. Trainable weights should be added to the list `self.trainable_weights`. Other attributes of note are: `self.non_trainable_weights` (list) and `self.updates` (list of update tuples (tensor, new_tensor)). For an example of how to use `non_trainable_weights` and `updates`, see the code for the `BatchNormalization` layer.
+- `call(x)`: this is where the layer's logic lives. Unless you want your layer to support masking, you only have to care about the first argument passed to `call`: the input tensor.
+- `get_output_shape_for(input_shape)`: in case your layer modifies the shape of its input, you should specify here the shape transformation logic. This allows Keras to do automatic shape inference.
+
+```python
+from keras import backend as K
+from keras.engine.topology import Layer
+import numpy as np
+
+class MyLayer(Layer):
+    def __init__(self, output_dim, **kwargs):
+        self.output_dim = output_dim
+        super(MyLayer, self).__init__(**kwargs)
+
+    def build(self, input_shape):
+        input_dim = input_shape[1]
+        initial_weight_value = np.random.random((input_dim, output_dim))
+        self.W = K.variable(initial_weight_value)
+        self.trainable_weights = [self.W]
+
+    def call(self, x, mask=None):
+        return K.dot(x, self.W)
+
+    def get_output_shape_for(self, input_shape):
+        return (input_shape[0], self.output_dim)
+```
+
+The existing Keras layers provide ample examples of how to implement almost anything. Never hesitate to read the source code!

docs/templates/models.md

@@ -1,114 +0,0 @@
-Keras has two models: __Sequential__, a linear stack of layers, and __Graph__, a directed acyclic graph of layers.
-
-# Using the Sequential model
-
-```python
-from keras.models import Sequential
-from keras.layers.core import Dense, Dropout, Activation
-from keras.optimizers import SGD
-
-model = Sequential()
-model.add(Dense(2, init='uniform', input_dim=64))
-model.add(Activation('softmax'))
-
-model.compile(optimizer='sgd', loss='mse')
-
-'''
-Train the model for 3 epochs, in batches of 16 samples,
-on data stored in the Numpy array X_train,
-and labels stored in the Numpy array y_train:
-'''
-model.fit(X_train, y_train, nb_epoch=3, batch_size=16, verbose=1)
-'''
-What you will see with mode verbose=1:
-Train on 37800 samples, validate on 4200 samples
-Epoch 0
-37800/37800 [==============================] - 7s - loss: 0.0385
-Epoch 1
-37800/37800 [==============================] - 8s - loss: 0.0140
-Epoch 2
-10960/37800 [=======>......................] - ETA: 4s - loss: 0.0109
-'''
-
-model.fit(X_train, y_train, nb_epoch=3, batch_size=16, verbose=2)
-'''
-What you will see with mode verbose=2:
-Train on 37800 samples, validate on 4200 samples
-Epoch 0
-loss: 0.0190
-Epoch 1
-loss: 0.0146
-Epoch 2
-loss: 0.0049
-'''
-
-'''
-Demonstration of the show_accuracy argument
-'''
-model.fit(X_train, y_train, nb_epoch=3, batch_size=16, verbose=2, show_accuracy=True)
-'''
-Train on 37800 samples, validate on 4200 samples
-Epoch 0
-loss: 0.0190 - acc.: 0.8750
-Epoch 1
-loss: 0.0146 - acc.: 0.8750
-Epoch 2
-loss: 0.0049 - acc.: 1.0000
-'''
-
-'''
-Demonstration of the validation_split argument
-'''
-model.fit(X_train, y_train, nb_epoch=3, batch_size=16,
-          validation_split=0.1, show_accuracy=True, verbose=1)
-'''
-Train on 37800 samples, validate on 4200 samples
-Epoch 0
-37800/37800 [==============================] - 7s - loss: 0.0385 - acc.: 0.7258 - val. loss: 0.0160 - val. acc.: 0.9136
-Epoch 1
-37800/37800 [==============================] - 8s - loss: 0.0140 - acc.: 0.9265 - val. loss: 0.0109 - val. acc.: 0.9383
-Epoch 2
-10960/37800 [=======>......................] - ETA: 4s - loss: 0.0109 - acc.: 0.9420
-'''
-```
-
-# Using the Graph model
-
-```python
-# graph model with one input and two outputs
-graph = Graph()
-graph.add_input(name='input', input_shape=(32,))
-graph.add_node(Dense(16), name='dense1', input='input')
-graph.add_node(Dense(4), name='dense2', input='input')
-graph.add_node(Dense(4), name='dense3', input='dense1')
-graph.add_output(name='output1', input='dense2')
-graph.add_output(name='output2', input='dense3')
-
-graph.compile(optimizer='rmsprop', loss={'output1':'mse', 'output2':'mse'})
-history = graph.fit({'input':X_train, 'output1':y_train, 'output2':y2_train}, nb_epoch=10)
-
-```
-
-```python
-# graph model with two inputs and one output
-graph = Graph()
-graph.add_input(name='input1', input_shape=(32,))
-graph.add_input(name='input2', input_shape=(32,))
-graph.add_node(Dense(16), name='dense1', input='input1')
-graph.add_node(Dense(4), name='dense2', input='input2')
-graph.add_node(Dense(4), name='dense3', input='dense1')
-graph.add_output(name='output', inputs=['dense2', 'dense3'], merge_mode='sum')
-graph.compile(optimizer='rmsprop', loss={'output':'mse'})
-
-history = graph.fit({'input1':X_train, 'input2':X2_train, 'output':y_train}, nb_epoch=10)
-predictions = graph.predict({'input1':X_test, 'input2':X2_test}) # {'output':...}
-
-```
-
-----
-
-# Model API documentation
-
-
-
-{{autogenerated}}

docs/templates/models/about-keras-models.md

@@ -0,0 +1,33 @@
+# About Keras models
+
+There are two types of models available in Keras: [the Sequential model](/models/sequential) and [the Model class used with functional API](/models/model).
+
+These models have a number of methods in common:
+
+- `model.summary()`: prints a summary representation of your model.
+- `model.get_config()`: returns a dictionary containing the configuration of the model. The model can be reinstantiated from its config via:
+```python
+config = model.get_config()
+model = Model.from_config(config)
+# or, for Sequential:
+model = Sequential.from_config(config)
+```
+
+- `model.get_weights()`: returns a list of all weight tensors in the model, as Numpy arrays.
+- `model.set_weights(weights)`: sets the values of the weights of the model, from a list of Numpy arrays. The arrays in the list should have the same shape as those returned by `get_weights()`.
+- `model.to_json()`: returns a representation of the model as a JSON string. Note that the representation does not include the weights, only the architecture. You can reinstantiate the same model (with reinitialized weights) from the JSON string via:
+```python
+from models import model_from_json
+
+json_string = model.to_json()
+model = model_from_json(json_string)
+```
+- `model.to_yaml()`: returns a representation of the model as a YAML string. Note that the representation does not include the weights, only the architecture. You can reinstantiate the same model (with reinitialized weights) from the YAML string via:
+```python
+from models import model_from_yaml
+
+yaml_string = model.to_yaml()
+model = model_from_yaml(yaml_string)
+```
+- `model.save_weights(filepath)`: saves the weights of the model as a HDF5 file.
+- `model.load_weights(filepath)`: loads the weights of the model from a HDF5 file (created by `save_weights`).

docs/templates/models/model.md

@@ -0,0 +1,32 @@
+# Model class API
+
+In the functional API, given an input tensor and output tensor, you can instantiate a `Model` via:
+
+```python
+from keras.models import Model
+from keras.layers import Input, Dense
+
+a = Input(shape=(32,))
+b = Dense(32)(a)
+model = Model(input=a, output=b)
+```
+
+This model will include all layers required in the computation of `b` given `a`.
+
+In the case of multi-input or multi-output models, you can use lists as well:
+
+```python
+model = Model(input=[a1, a2], output=[b1, b3, b3])
+```
+
+For a detailed introduction of what `Model` can do, read [this guide to the Keras functional API](/getting-started/functional-api-guide).
+
+## Useful attributes of Model
+
+- `model.layers` is a flattened list of the layers comprising the model graph.
+- `model.inputs` is the list of input tensors.
+- `model.outputs` is the list of output tensors.
+
+## Methods
+
+{{autogenerated}}

docs/templates/models/sequential.md

@@ -0,0 +1,14 @@
+# The Sequential model API
+
+To get started, read [this guide to the Keras Sequential model](/getting-started/sequential-model-guide).
+
+## Useful attributes of Model
+
+- `model.layers` is a list of the layers added to the model.
+
+
+----
+
+## Sequential model methods
+
+{{autogenerated}}

docs/templates/objectives.md

@@ -7,10 +7,10 @@ An objective function (or loss function, or optimization score function) is one
 model.compile(loss='mean_squared_error', optimizer='sgd')
 ```
 
-You can either pass the name of an existing objective, or pass a Theano symbolic function that returns a scalar for each data-point and takes the following two arguments:
+You can either pass the name of an existing objective, or pass a Theano/TensorFlow symbolic function that returns a scalar for each data-point and takes the following two arguments:
 
-- __y_true__: True labels. Theano tensor.
-- __y_pred__: Predictions. Theano tensor of the same shape as y_true.
+- __y_true__: True labels. Theano/TensorFlow tensor.
+- __y_pred__: Predictions. Theano/TensorFlow tensor of the same shape as y_true.
 
 The actual optimized objective is the mean of the output array across all datapoints.
 
@@ -19,7 +19,6 @@ For a few examples of such functions, check out the [objectives source](https://
 ## Available objectives
 
 - __mean_squared_error__ / __mse__
-- __root_mean_squared_error__ / __rmse__
 - __mean_absolute_error__ / __mae__
 - __mean_absolute_percentage_error__ / __mape__
 - __mean_squared_logarithmic_error__ / __msle__
@@ -27,3 +26,7 @@ For a few examples of such functions, check out the [objectives source](https://
 - __hinge__
 - __binary_crossentropy__: Also known as logloss. 
 - __categorical_crossentropy__: Also known as multiclass logloss. __Note__: using this objective requires that your labels are binary arrays of shape `(nb_samples, nb_classes)`.
+- __sparse_categorical_crossentropy__: As above but accepts sparse labels. __Note__: this objective still requires that your labels have the same number of dimensions as your outputs; you may need to add a length-1 dimension to the shape of your labels, e.g with `np.expand_dims(y, -1)`.
+- __kullback_leibler_divergence__ / __kld__: Information gain from a predicted probability distribution Q to a true probability distribution P. Gives a measure of difference between both distributions.
+- __poisson__: Mean of `(predictions - targets * log(predictions))`
+- __cosine_proximity__: The opposite (negative) of the mean cosine proximity between predictions and targets.

docs/templates/optimizers.md

@@ -9,7 +9,7 @@ model.add(Dense(64, init='uniform', input_dim=10))
 model.add(Activation('tanh'))
 model.add(Activation('softmax'))
 
-sgd = SGD(lr=0.1, decay=1e-6, momentum=0.9, nesterov=True)
+sgd = SGD(lr=0.01, decay=1e-6, momentum=0.9, nesterov=True)
 model.compile(loss='mean_squared_error', optimizer=sgd)
 ```
 
@@ -22,4 +22,23 @@ model.compile(loss='mean_squared_error', optimizer='sgd')
 
 ---
 
+## Parameters common to all Keras optimizers
+
+The parameters `clipnorm` and `clipvalue` can be used with all optimizers to control gradient clipping:
+
+```python
+# all parameter gradients will be clipped to
+# a maximum norm of 1.
+sgd = SGD(lr=0.01, clipnorm=1.)
+```
+
+```python
+# all parameter gradients will be clipped to
+# a maximum value of 0.5 and
+# a minimum value of -0.5.
+sgd = SGD(lr=0.01, clipvalue=0.5)
+```
+
+---
+
 {{autogenerated}}

docs/templates/preprocessing/image.md

@@ -2,19 +2,26 @@
 ## ImageDataGenerator
 
 ```python
-keras.preprocessing.image.ImageDataGenerator(featurewise_center=True,
+keras.preprocessing.image.ImageDataGenerator(featurewise_center=False,
     samplewise_center=False,
-    featurewise_std_normalization=True,
+    featurewise_std_normalization=False,
     samplewise_std_normalization=False,
     zca_whitening=False,
     rotation_range=0.,
     width_shift_range=0.,
     height_shift_range=0.,
+    shear_range=0.,
+    zoom_range=0.,
+    channel_shift_range=0.,
+    fill_mode='nearest',
+    cval=0.,
     horizontal_flip=False,
-    vertical_flip=False)
+    vertical_flip=False,
+    rescale=None,
+    dim_ordering=K.image_dim_ordering())
 ```
 
-Generate batches of tensor image data with real-time data augmentation.
+Generate batches of tensor image data with real-time data augmentation. The data will be looped over (in batches) indefinitely.
 
 - __Arguments__:
     - __featurewise_center__: Boolean. Set input mean to 0 over the dataset.
@@ -25,28 +32,63 @@ Generate batches of tensor image data with real-time data augmentation.
     - __rotation_range__: Int. Degree range for random rotations.
     - __width_shift_range__: Float (fraction of total width). Range for random horizontal shifts.
     - __height_shift_range__: Float (fraction of total height). Range for random vertical shifts.
+    - __shear_range__: Float. Shear Intensity (Shear angle in counter-clockwise direction as radians)
+    - __zoom_range__: Float or [lower, upper]. Range for random zoom. If a float, `[lower, upper] = [1-zoom_range, 1+zoom_range]`.
+    - __channel_shift_range__: Float. Range for random channel shifts.
+    - __fill_mode__: One of {"constant", "nearest", "reflect" or "wrap"}.  Points outside the boundaries of the input are filled according to the given mode.
+    - __cval__: Float or Int. Value used for points outside the boundaries when `fill_mode = "constant"`.
     - __horizontal_flip__: Boolean. Randomly flip inputs horizontally.
     - __vertical_flip__: Boolean. Randomly flip inputs vertically.
+    - __rescale__: rescaling factor. Defaults to None. If None or 0, no rescaling is applied,
+            otherwise we multiply the data by the value provided (before applying
+            any other transformation).
+    - __dim_ordering__: One of {"th", "tf"}.
+        "tf" mode means that the images should have shape `(samples, width, height, channels)`,
+        "th" mode means that the images should have shape `(samples, channels, width, height)`.
+        It defaults to the `image_dim_ordering` value found in your
+        Keras config file at `~/.keras/keras.json`.
+        If you never set it, then it will be "th".
 
 - __Methods__:
-    - __fit(X)__: Required if featurewise_center or featurewise_std_normalization or zca_whitening. Compute necessary quantities on some sample data.
-        - __Arguments__: 
+    - __fit(X)__: Compute the internal data stats related to the data-dependent transformations, based on an array of sample data.
+        Only required if featurewise_center or featurewise_std_normalization or zca_whitening.
+        - __Arguments__:
             - __X__: sample data.
             - __augment__: Boolean (default: False). Whether to fit on randomly augmented samples.
             - __rounds__: int (default: 1). If augment, how many augmentation passes over the data to use.
-    - __flow(X, y)__:
-        - __Arguments__: 
+    - __flow(X, y)__: Takes numpy data & label arrays, and generates batches of augmented/normalized data. Yields batches indefinitely, in an infinite loop.
+        - __Arguments__:
             - __X__: data.
             - __y__: labels.
             - __batch_size__: int (default: 32).
             - __shuffle__: boolean (defaut: False).
-            - __save_to_dir__: None or str. This allows you to optimally specify a directory to which to save the augmented pictures being generated (useful for visualizing what you are doing). 
-            - __save_prefix__: str. Prefix to use for filenames of saved pictures.
-            - __save_format__: one of "png", jpeg". 
+            - __save_to_dir__: None or str (default: None). This allows you to optimally specify a directory to which to save the augmented pictures being generated (useful for visualizing what you are doing).
+            - __save_prefix__: str (default: `''`). Prefix to use for filenames of saved pictures (only relevant if `save_to_dir` is set).
+            - __save_format__: one of "png", "jpeg" (only relevant if `save_to_dir` is set). Default: "jpeg".
+        - ___yields__: Tuples of `(x, y)` where `x` is a numpy array of image data and `y` is a numpy array of corresponding labels.
+            The generator loops indefinitely.
+    - __flow_from_directory(directory)__: Takes the path to a directory, and generates batches of augmented/normalized data. Yields batches indefinitely, in an infinite loop.
+        - __Arguments__:
+            - __directory: path to the target directory. It should contain one subdirectory per class,
+                and the subdirectories should contain PNG or JPG images. See [this script](https://gist.github.com/fchollet/0830affa1f7f19fd47b06d4cf89ed44d) for more details.
+            - __target_size__: tuple of integers, default: `(256, 256)`. The dimensions to which all images found will be resized.
+            - __color_mode__: one of "grayscale", "rbg". Default: "rgb". Whether the images will be converted to have 1 or 3 color channels.
+            - __classes__: optional list of class subdirectories (e.g. `['dogs', 'cats']`). Default: None. If not provided, the list of classes will be automatically inferred (and the order of the classes, which will map to the label indices, will be alphanumeric).
+            - __class_mode__: one of "categorical", "binary", "sparse" or None. Default: "categorical". Determines the type of label arrays that are returned: "categorical" will be 2D one-hot encoded labels, "binary" will be 1D binary labels, "sparse" will be 1D integer labels. If None, no labels are returned (the generator will only yield batches of image data, which is useful to use `model.predict_generator()`, `model.evaluate_generator()`, etc.).
+            - __batch_size__: size of the batches of data (default: 32).
+            - __shuffle__: whether to shuffle the data (default: True)
+            - __seed__: optional random seed for shuffling.
+            - __save_to_dir__: None or str (default: None). This allows you to optimally specify a directory to which to save the augmented pictures being generated (useful for visualizing what you are doing).
+            - __save_prefix__: str. Prefix to use for filenames of saved pictures (only relevant if `save_to_dir` is set).
+            - __save_format__: one of "png", "jpeg" (only relevant if `save_to_dir` is set). Default: "jpeg".
+
+
+- __Examples__:
+
+Example of using `.flow(X, y)`:
 
-- __Example__:
 ```python
-(X_train, y_train), (X_test, y_test) = cifar10.load_data(test_split=0.1)
+(X_train, y_train), (X_test, y_test) = cifar10.load_data()
 Y_train = np_utils.to_categorical(y_train, nb_classes)
 Y_test = np_utils.to_categorical(y_test, nb_classes)
 
@@ -58,13 +100,54 @@ datagen = ImageDataGenerator(
     height_shift_range=0.2,
     horizontal_flip=True)
 
-# compute quantities required for featurewise normalization 
+# compute quantities required for featurewise normalization
 # (std, mean, and principal components if ZCA whitening is applied)
 datagen.fit(X_train)
 
+# fits the model on batches with real-time data augmentation:
+model.fit_generator(datagen.flow(X_train, Y_train, batch_size=32),
+                    samples_per_epoch=len(X_train), nb_epoch=nb_epoch)
+
+# here's a more "manual" example
 for e in range(nb_epoch):
     print 'Epoch', e
-    # batch train with realtime data augmentation
-    for X_batch, Y_batch in datagen.flow(X_train, Y_train):
+    batches = 0
+    for X_batch, Y_batch in datagen.flow(X_train, Y_train, batch_size=32):
         loss = model.train(X_batch, Y_batch)
-```
+        batches += 1
+        if batches >= len(X_train) / 32:
+            # we need to break the loop by hand because
+            # the generator loops indefinitely
+            break
+```
+
+Example of using `.flow_from_directory(directory)`:
+
+```python
+train_datagen = ImageDataGenerator(
+        rescale=1./255,
+        shear_range=0.2,
+        zoom_range=0.2,
+        horizontal_flip=True)
+
+test_datagen = ImageDataGenerator(rescale=1./255)
+
+train_generator = train_datagen.flow_from_directory(
+        'data/train',
+        target_size=(150, 150),
+        batch_size=32,
+        class_mode='binary')
+
+validation_generator = test_datagen.flow_from_directory(
+        'data/validation',
+        target_size=(150, 150),
+        batch_size=32,
+        class_mode='binary')
+
+model.fit_generator(
+        train_generator,
+        samples_per_epoch=2000,
+        nb_epoch=50,
+        validation_data=validation_generator,
+        nb_val_samples=800)
+```

docs/templates/preprocessing/sequence.md

@@ -4,26 +4,29 @@
 keras.preprocessing.sequence.pad_sequences(sequences, maxlen=None, dtype='int32')
 ```
 
-Transform a list of `nb_samples sequences` (lists of scalars) into a 2D numpy array of shape `(nb_samples, nb_timesteps)`. `nb_timesteps` is either the `maxlen` argument if provided, or the length of the longest sequence otherwise. Sequences that are shorter than `nb_timesteps` are padded with zeros at the end.
+Transform a list of `nb_samples sequences` (lists of scalars) into a 2D Numpy array of shape `(nb_samples, nb_timesteps)`. `nb_timesteps` is either the `maxlen` argument if provided, or the length of the longest sequence otherwise. Sequences that are shorter than `nb_timesteps` are padded with zeros at the end.
 
-- __Return__: 2D numpy array of shape `(nb_samples, nb_timesteps)`.
+- __Return__: 2D Numpy array of shape `(nb_samples, nb_timesteps)`.
 
 - __Arguments__:
     - __sequences__: List of lists of int or float.
     - __maxlen__: None or int. Maximum sequence length, longer sequences are truncated and shorter sequences are padded with zeros at the end.
-    - __dtype__: datatype of the numpy array returned.
+    - __dtype__: datatype of the Numpy array returned.
+    - __padding__: 'pre' or 'post', pad either before or after each sequence.
+    - __truncating__: 'pre' or 'post', remove values from sequences larger than maxlen either in the beginning or in the end of the sequence
+    - __value__: float, value to pad the sequences to the desired value.
 
 ---
 
 ## skipgrams
 
 ```python
-keras.preprocessing.sequence.skipgrams(sequence, vocabulary_size, 
-    window_size=4, negative_samples=1., shuffle=True, 
+keras.preprocessing.sequence.skipgrams(sequence, vocabulary_size,
+    window_size=4, negative_samples=1., shuffle=True,
     categorical=False, sampling_table=None)
 ```
 
-Transforms a sequence of word indexes (list of int) into couples of the form: 
+Transforms a sequence of word indexes (list of int) into couples of the form:
 
 - (word, word in the same window), with label 1 (positive samples).
 - (word, random word from the vocabulary), with label 0 (negative samples).
@@ -31,8 +34,8 @@ Transforms a sequence of word indexes (list of int) into couples of the form:
 Read more about Skipgram in this gnomic paper by Mikolov et al.: [Efficient Estimation of Word Representations in
 Vector Space](http://arxiv.org/pdf/1301.3781v3.pdf)
 
-- __Return__: tuple `(couples, labels)`. 
-    - `couples` is a list of 2-elements lists of int: `[word_index, other_word_index]`. 
+- __Return__: tuple `(couples, labels)`.
+    - `couples` is a list of 2-elements lists of int: `[word_index, other_word_index]`.
     - `labels` is a list of 0 and 1, where 1 indicates that `other_word_index` was found in the same window as `word_index`, and 0 indicates that `other_word_index` was random.
     - if categorical is set to True, the labels are categorical, ie. 1 becomes [0,1], and 0 becomes [1, 0].
 
@@ -43,7 +46,7 @@ Vector Space](http://arxiv.org/pdf/1301.3781v3.pdf)
     - __negative_samples__: float >= 0. 0 for no negative (=random) samples. 1 for same number as positive samples. etc.
     - __shuffle__: boolean. Whether to shuffle the samples.
     - __categorical__: boolean. Whether to make the returned labels categorical.
-    - __sampling_table__: numpy array of shape `(vocabulary_size,)` where `sampling_table[i]` is the probability of sampling the word with index i (assumed to be i-th most common word in the dataset).
+    - __sampling_table__: Numpy array of shape `(vocabulary_size,)` where `sampling_table[i]` is the probability of sampling the word with index i (assumed to be i-th most common word in the dataset).
 
 
 ---
@@ -56,7 +59,7 @@ keras.preprocessing.sequence.make_sampling_table(size, sampling_factor=1e-5)
 
 Used for generating the `sampling_table` argument for `skipgrams`. `sampling_table[i]` is the probability of sampling the word i-th most common word in a dataset (more common words should be sampled less frequently, for balance).
 
-- __Return__: numpy array of shape `(size,)`.
+- __Return__: Numpy array of shape `(size,)`.
 
 - __Arguments__:
     - __size__: size of the vocabulary considered.

docs/templates/scikit-learn-api.md

@@ -0,0 +1,45 @@
+# Wrappers for the Scikit-Learn API
+
+You can use `Sequential` Keras models (single-input only) as part of your Scikit-Learn workflow via the wrappers found at `keras.wrappers.scikit_learn.py`.
+
+There are two wrappers available:
+
+`keras.wrappers.scikit_learn.KerasClassifier(build_fn=None, **sk_params)`, which implements the Scikit-Learn classifier interface,
+
+`keras.wrappers.scikit_learn.KerasRegressor(build_fn=None, **sk_params)`, which implements the Scikit-Learn regressor interface.
+
+### Arguments
+
+- __build_fn__: callable function or class instance
+- __sk_params__: model parameters & fitting parameters
+
+`build_fn` should construct, compile and return a Keras model, which
+will then be used to fit/predict. One of the following
+three values could be passed to build_fn:
+
+1. A function
+2. An instance of a class that implements the __call__ method
+3. None. This means you implement a class that inherits from either
+`KerasClassifier` or `KerasRegressor`. The __call__ method of the
+present class will then be treated as the default build_fn.
+
+`sk_params` takes both model parameters and fitting parameters. Legal model
+parameters are the arguments of `build_fn`. Note that like all other
+estimators in scikit-learn, 'build_fn' should provide default values for
+its arguments, so that you could create the estimator without passing any
+values to `sk_params`.
+
+`sk_params` could also accept parameters for calling `fit`, `predict`,
+`predict_proba`, and `score` methods (e.g., `nb_epoch`, `batch_size`).
+fitting (predicting) parameters are selected in the following order:
+
+1. Values passed to the dictionary arguments of
+`fit`, `predict`, `predict_proba`, and `score` methods
+2. Values passed to `sk_params`
+3. The default values of the `keras.models.Sequential`
+`fit`, `predict`, `predict_proba` and `score` methods
+
+When using scikit-learn's `grid_search` API, legal tunable parameters are
+those you could pass to `sk_params`, including fitting parameters.
+In other words, you could use `grid_search` to search for the best
+`batch_size` or `nb_epoch` as well as the model parameters.

docs/templates/visualization.md

@@ -10,11 +10,16 @@ from keras.utils.visualize_util import plot
 plot(model, to_file='model.png')
 ```
 
+`plot` takes two optional arguments:
+
+- `show_shapes` (defaults to False) controls whether output shapes are shown in the graph.
+- `show_layer_names` (defaults to True) controls whether layer names are shown in the graph.
+
 You can also directly obtain the `pydot.Graph` object and render it yourself,
 for example to show it in an ipython notebook :
 ```python
 from IPython.display import SVG
-from keras.utils.visualize_util import to_graph
+from keras.utils.visualize_util import model_to_dot
 
-SVG(to_graph(model).create(prog='dot', format='svg'))
+SVG(model_to_dot(model).create(prog='dot', format='svg'))
 ```

examples/addition_rnn.py

@@ -27,9 +27,9 @@ Five digits inverted:
 '''
 
 from __future__ import print_function
-from keras.models import Sequential, slice_X
-from keras.layers.core import Activation, TimeDistributedDense, RepeatVector
-from keras.layers import recurrent
+from keras.models import Sequential
+from keras.engine.training import slice_X
+from keras.layers import Activation, TimeDistributed, Dense, RepeatVector, recurrent
 import numpy as np
 from six.moves import range
 
@@ -39,7 +39,7 @@ class CharacterTable(object):
     Given a set of characters:
     + Encode them to a one hot integer representation
     + Decode the one hot integer representation to their character output
-    + Decode a vector of probabilties to their character output
+    + Decode a vector of probabilities to their character output
     '''
     def __init__(self, chars, maxlen):
         self.chars = sorted(set(chars))
@@ -139,10 +139,12 @@ for _ in range(LAYERS):
     model.add(RNN(HIDDEN_SIZE, return_sequences=True))
 
 # For each of step of the output sequence, decide which character should be chosen
-model.add(TimeDistributedDense(len(chars)))
+model.add(TimeDistributed(Dense(len(chars))))
 model.add(Activation('softmax'))
 
-model.compile(loss='categorical_crossentropy', optimizer='adam')
+model.compile(loss='categorical_crossentropy',
+              optimizer='adam',
+              metrics=['accuracy'])
 
 # Train the model each generation and show predictions against the validation dataset
 for iteration in range(1, 200):
@@ -150,7 +152,7 @@ for iteration in range(1, 200):
     print('-' * 50)
     print('Iteration', iteration)
     model.fit(X_train, y_train, batch_size=BATCH_SIZE, nb_epoch=1,
-              validation_data=(X_val, y_val), show_accuracy=True)
+              validation_data=(X_val, y_val))
     ###
     # Select 10 samples from the validation set at random so we can visualize errors
     for i in range(10):

examples/antirectifier.py

@@ -0,0 +1,104 @@
+'''The example demonstrates how to write custom layers for Keras.
+
+We build a custom activation layer called 'Antirectifier',
+which modifies the shape of the tensor that passes through it.
+We need to specify two methods: `get_output_shape_for` and `call`.
+
+Note that the same result can also be achieved via a Lambda layer.
+
+Because our custom layer is written with primitives from the Keras
+backend (`K`), our code can run both on TensorFlow and Theano.
+'''
+
+from __future__ import print_function
+from keras.models import Sequential
+from keras.layers import Dense, Dropout, Layer, Activation
+from keras.datasets import mnist
+from keras import backend as K
+from keras.utils import np_utils
+
+
+class Antirectifier(Layer):
+    '''This is the combination of a sample-wise
+    L2 normalization with the concatenation of the
+    positive part of the input with the negative part
+    of the input. The result is a tensor of samples that are
+    twice as large as the input samples.
+
+    It can be used in place of a ReLU.
+
+    # Input shape
+        2D tensor of shape (samples, n)
+
+    # Output shape
+        2D tensor of shape (samples, 2*n)
+
+    # Theoretical justification
+        When applying ReLU, assuming that the distribution
+        of the previous output is approximately centered around 0.,
+        you are discarding half of your input. This is inefficient.
+
+        Antirectifier allows to return all-positive outputs like ReLU,
+        without discarding any data.
+
+        Tests on MNIST show that Antirectifier allows to train networks
+        with twice less parameters yet with comparable
+        classification accuracy as an equivalent ReLU-based network.
+    '''
+    def get_output_shape_for(self, input_shape):
+        shape = list(input_shape)
+        assert len(shape) == 2  # only valid for 2D tensors
+        shape[-1] *= 2
+        return tuple(shape)
+
+    def call(self, x, mask=None):
+        x -= K.mean(x, axis=1, keepdims=True)
+        x = K.l2_normalize(x, axis=1)
+        pos = K.relu(x)
+        neg = K.relu(-x)
+        return K.concatenate([pos, neg], axis=1)
+
+# global parameters
+batch_size = 128
+nb_classes = 10
+nb_epoch = 40
+
+# the data, shuffled and split between train and test sets
+(X_train, y_train), (X_test, y_test) = mnist.load_data()
+
+X_train = X_train.reshape(60000, 784)
+X_test = X_test.reshape(10000, 784)
+X_train = X_train.astype('float32')
+X_test = X_test.astype('float32')
+X_train /= 255
+X_test /= 255
+print(X_train.shape[0], 'train samples')
+print(X_test.shape[0], 'test samples')
+
+# convert class vectors to binary class matrices
+Y_train = np_utils.to_categorical(y_train, nb_classes)
+Y_test = np_utils.to_categorical(y_test, nb_classes)
+
+# build the model
+model = Sequential()
+model.add(Dense(256, input_shape=(784,)))
+model.add(Antirectifier())
+model.add(Dropout(0.1))
+model.add(Dense(256))
+model.add(Antirectifier())
+model.add(Dropout(0.1))
+model.add(Dense(10))
+model.add(Activation('softmax'))
+
+# compile the model
+model.compile(loss='categorical_crossentropy',
+              optimizer='rmsprop',
+              metrics=['accuracy'])
+
+# train the model
+model.fit(X_train, Y_train,
+          batch_size=batch_size, nb_epoch=nb_epoch,
+          verbose=1, validation_data=(X_test, Y_test))
+
+# next, compare with an equivalent network
+# with2x bigger Dense layers and ReLU

examples/babi_memnn.py

@@ -1,4 +1,4 @@
-'''Train a memory network on the bAbI dataset.
+'''Trains a memory network on the bAbI dataset.
 
 References:
 - Jason Weston, Antoine Bordes, Sumit Chopra, Tomas Mikolov, Alexander M. Rush,
@@ -16,9 +16,9 @@ Time per epoch: 3s on CPU (core i7).
 from __future__ import print_function
 from keras.models import Sequential
 from keras.layers.embeddings import Embedding
-from keras.layers.core import Activation, Dense, Merge, Permute, Dropout
-from keras.layers.recurrent import LSTM
-from keras.datasets.data_utils import get_file
+from keras.layers import Activation, Dense, Merge, Permute, Dropout
+from keras.layers import LSTM
+from keras.utils.data_utils import get_file
 from keras.preprocessing.sequence import pad_sequences
 from functools import reduce
 import tarfile
@@ -94,8 +94,13 @@ def vectorize_stories(data, word_idx, story_maxlen, query_maxlen):
             pad_sequences(Xq, maxlen=query_maxlen), np.array(Y))
 
 
-path = get_file('babi-tasks-v1-2.tar.gz',
-                origin='http://www.thespermwhale.com/jaseweston/babi/tasks_1-20_v1-2.tar.gz')
+try:
+    path = get_file('babi-tasks-v1-2.tar.gz', origin='http://www.thespermwhale.com/jaseweston/babi/tasks_1-20_v1-2.tar.gz')
+except:
+    print('Error downloading dataset, please download it manually:\n'
+          '$ wget http://www.thespermwhale.com/jaseweston/babi/tasks_1-20_v1-2.tar.gz\n'
+          '$ mv tasks_1-20_v1-2.tar.gz ~/.keras/datasets/babi-tasks-v1-2.tar.gz')
+    raise
 tar = tarfile.open(path)
 
 challenges = {
@@ -167,7 +172,7 @@ question_encoder.add(Dropout(0.3))
 match = Sequential()
 match.add(Merge([input_encoder_m, question_encoder],
                 mode='dot',
-                dot_axes=[(2,), (2,)]))
+                dot_axes=[2, 2]))
 # output: (samples, story_maxlen, query_maxlen)
 # embed the input into a single vector with size = story_maxlen:
 input_encoder_c = Sequential()
@@ -195,10 +200,10 @@ answer.add(Dense(vocab_size))
 # we output a probability distribution over the vocabulary
 answer.add(Activation('softmax'))
 
-answer.compile(optimizer='rmsprop', loss='categorical_crossentropy')
+answer.compile(optimizer='rmsprop', loss='categorical_crossentropy',
+               metrics=['accuracy'])
 # Note: you could use a Graph model to avoid repeat the input twice
 answer.fit([inputs_train, queries_train, inputs_train], answers_train,
            batch_size=32,
            nb_epoch=120,
-           show_accuracy=True,
            validation_data=([inputs_test, queries_test, inputs_test], answers_test))

examples/babi_rnn.py

@@ -7,8 +7,8 @@ http://arxiv.org/abs/1502.05698
 
 Task Number                  | FB LSTM Baseline | Keras QA
 ---                          | ---              | ---
-QA1 - Single Supporting Fact | 50               | 52.1
-QA2 - Two Supporting Facts   | 20               | 37.0
+QA1 - Single Supporting Fact | 50               | 100.0
+QA2 - Two Supporting Facts   | 20               | 50.0
 QA3 - Three Supporting Facts | 20               | 20.5
 QA4 - Two Arg. Relations     | 61               | 62.9
 QA5 - Three Arg. Relations   | 70               | 61.9
@@ -34,8 +34,8 @@ https://research.facebook.com/researchers/1543934539189348
 Notes:
 
 - With default word, sentence, and query vector sizes, the GRU model achieves:
-  - 52.1% test accuracy on QA1 in 20 epochs (2 seconds per epoch on CPU)
-  - 37.0% test accuracy on QA2 in 20 epochs (16 seconds per epoch on CPU)
+  - 100% test accuracy on QA1 in 20 epochs (2 seconds per epoch on CPU)
+  - 50% test accuracy on QA2 in 20 epochs (16 seconds per epoch on CPU)
 In comparison, the Facebook paper achieves 50% and 20% for the LSTM baseline.
 
 - The task does not traditionally parse the question separately. This likely
@@ -64,9 +64,9 @@ import tarfile
 import numpy as np
 np.random.seed(1337)  # for reproducibility
 
-from keras.datasets.data_utils import get_file
+from keras.utils.data_utils import get_file
 from keras.layers.embeddings import Embedding
-from keras.layers.core import Dense, Merge
+from keras.layers import Dense, Merge, Dropout, RepeatVector
 from keras.layers import recurrent
 from keras.models import Sequential
 from keras.preprocessing.sequence import pad_sequences
@@ -138,15 +138,21 @@ def vectorize_stories(data, word_idx, story_maxlen, query_maxlen):
         Y.append(y)
     return pad_sequences(X, maxlen=story_maxlen), pad_sequences(Xq, maxlen=query_maxlen), np.array(Y)
 
-RNN = recurrent.GRU
+RNN = recurrent.LSTM
 EMBED_HIDDEN_SIZE = 50
 SENT_HIDDEN_SIZE = 100
 QUERY_HIDDEN_SIZE = 100
 BATCH_SIZE = 32
-EPOCHS = 20
+EPOCHS = 40
 print('RNN / Embed / Sent / Query = {}, {}, {}, {}'.format(RNN, EMBED_HIDDEN_SIZE, SENT_HIDDEN_SIZE, QUERY_HIDDEN_SIZE))
 
-path = get_file('babi-tasks-v1-2.tar.gz', origin='http://www.thespermwhale.com/jaseweston/babi/tasks_1-20_v1-2.tar.gz')
+try:
+    path = get_file('babi-tasks-v1-2.tar.gz', origin='http://www.thespermwhale.com/jaseweston/babi/tasks_1-20_v1-2.tar.gz')
+except:
+    print('Error downloading dataset, please download it manually:\n'
+          '$ wget http://www.thespermwhale.com/jaseweston/babi/tasks_1-20_v1-2.tar.gz\n'
+          '$ mv tasks_1-20_v1-2.tar.gz ~/.keras/datasets/babi-tasks-v1-2.tar.gz')
+    raise
 tar = tarfile.open(path)
 # Default QA1 with 1000 samples
 # challenge = 'tasks_1-20_v1-2/en/qa1_single-supporting-fact_{}.txt'
@@ -178,20 +184,28 @@ print('story_maxlen, query_maxlen = {}, {}'.format(story_maxlen, query_maxlen))
 print('Build model...')
 
 sentrnn = Sequential()
-sentrnn.add(Embedding(vocab_size, EMBED_HIDDEN_SIZE, mask_zero=True))
-sentrnn.add(RNN(SENT_HIDDEN_SIZE, return_sequences=False))
+sentrnn.add(Embedding(vocab_size, EMBED_HIDDEN_SIZE,
+                      input_length=story_maxlen))
+sentrnn.add(Dropout(0.3))
 
 qrnn = Sequential()
-qrnn.add(Embedding(vocab_size, EMBED_HIDDEN_SIZE))
-qrnn.add(RNN(QUERY_HIDDEN_SIZE, return_sequences=False))
+qrnn.add(Embedding(vocab_size, EMBED_HIDDEN_SIZE,
+                   input_length=query_maxlen))
+qrnn.add(Dropout(0.3))
+qrnn.add(RNN(EMBED_HIDDEN_SIZE, return_sequences=False))
+qrnn.add(RepeatVector(story_maxlen))
 
 model = Sequential()
-model.add(Merge([sentrnn, qrnn], mode='concat'))
+model.add(Merge([sentrnn, qrnn], mode='sum'))
+model.add(RNN(EMBED_HIDDEN_SIZE, return_sequences=False))
+model.add(Dropout(0.3))
 model.add(Dense(vocab_size, activation='softmax'))
 
-model.compile(optimizer='adam', loss='categorical_crossentropy', class_mode='categorical')
+model.compile(optimizer='adam',
+              loss='categorical_crossentropy',
+              metrics=['accuracy'])
 
 print('Training')
-model.fit([X, Xq], Y, batch_size=BATCH_SIZE, nb_epoch=EPOCHS, validation_split=0.05, show_accuracy=True)
-loss, acc = model.evaluate([tX, tXq], tY, batch_size=BATCH_SIZE, show_accuracy=True)
+model.fit([X, Xq], Y, batch_size=BATCH_SIZE, nb_epoch=EPOCHS, validation_split=0.05)
+loss, acc = model.evaluate([tX, tXq], tY, batch_size=BATCH_SIZE)
 print('Test loss / test accuracy = {:.4f} / {:.4f}'.format(loss, acc))

examples/cifar10_cnn.py

@@ -15,11 +15,10 @@ from __future__ import print_function
 from keras.datasets import cifar10
 from keras.preprocessing.image import ImageDataGenerator
 from keras.models import Sequential
-from keras.layers.core import Dense, Dropout, Activation, Flatten
-from keras.layers.convolutional import Convolution2D, MaxPooling2D
-from keras.optimizers import SGD, Adadelta, Adagrad
-from keras.utils import np_utils, generic_utils
-from six.moves import range
+from keras.layers import Dense, Dropout, Activation, Flatten
+from keras.layers import Convolution2D, MaxPooling2D
+from keras.optimizers import SGD
+from keras.utils import np_utils
 
 batch_size = 32
 nb_classes = 10
@@ -67,7 +66,9 @@ model.add(Activation('softmax'))
 
 # let's train the model using SGD + momentum (how original).
 sgd = SGD(lr=0.01, decay=1e-6, momentum=0.9, nesterov=True)
-model.compile(loss='categorical_crossentropy', optimizer=sgd)
+model.compile(loss='categorical_crossentropy',
+              optimizer=sgd,
+              metrics=['accuracy'])
 
 X_train = X_train.astype('float32')
 X_test = X_test.astype('float32')
@@ -75,24 +76,25 @@ X_train /= 255
 X_test /= 255
 
 if not data_augmentation:
-    print('Not using data augmentation or normalization')
-    model.fit(X_train, Y_train, batch_size=batch_size, nb_epoch=nb_epoch)
-    score = model.evaluate(X_test, Y_test, batch_size=batch_size)
-    print('Test score:', score)
-
+    print('Not using data augmentation.')
+    model.fit(X_train, Y_train,
+              batch_size=batch_size,
+              nb_epoch=nb_epoch,
+              validation_data=(X_test, Y_test),
+              shuffle=True)
 else:
-    print('Using real time data augmentation')
+    print('Using real-time data augmentation.')
 
     # this will do preprocessing and realtime data augmentation
     datagen = ImageDataGenerator(
-        featurewise_center=True,  # set input mean to 0 over the dataset
+        featurewise_center=False,  # set input mean to 0 over the dataset
         samplewise_center=False,  # set each sample mean to 0
-        featurewise_std_normalization=True,  # divide inputs by std of the dataset
+        featurewise_std_normalization=False,  # divide inputs by std of the dataset
         samplewise_std_normalization=False,  # divide each input by its std
         zca_whitening=False,  # apply ZCA whitening
-        rotation_range=20,  # randomly rotate images in the range (degrees, 0 to 180)
-        width_shift_range=0.2,  # randomly shift images horizontally (fraction of total width)
-        height_shift_range=0.2,  # randomly shift images vertically (fraction of total height)
+        rotation_range=0,  # randomly rotate images in the range (degrees, 0 to 180)
+        width_shift_range=0.1,  # randomly shift images horizontally (fraction of total width)
+        height_shift_range=0.1,  # randomly shift images vertically (fraction of total height)
         horizontal_flip=True,  # randomly flip images
         vertical_flip=False)  # randomly flip images
 
@@ -100,20 +102,9 @@ else:
     # (std, mean, and principal components if ZCA whitening is applied)
     datagen.fit(X_train)
 
-    for e in range(nb_epoch):
-        print('-'*40)
-        print('Epoch', e)
-        print('-'*40)
-        print('Training...')
-        # batch train with realtime data augmentation
-        progbar = generic_utils.Progbar(X_train.shape[0])
-        for X_batch, Y_batch in datagen.flow(X_train, Y_train):
-            loss = model.train_on_batch(X_batch, Y_batch)
-            progbar.add(X_batch.shape[0], values=[('train loss', loss[0])])
-
-        print('Testing...')
-        # test time!
-        progbar = generic_utils.Progbar(X_test.shape[0])
-        for X_batch, Y_batch in datagen.flow(X_test, Y_test):
-            score = model.test_on_batch(X_batch, Y_batch)
-            progbar.add(X_batch.shape[0], values=[('test loss', score[0])])
+    # fit the model on the batches generated by datagen.flow()
+    model.fit_generator(datagen.flow(X_train, Y_train,
+                        batch_size=batch_size),
+                        samples_per_epoch=X_train.shape[0],
+                        nb_epoch=nb_epoch,
+                        validation_data=(X_test, Y_test))

examples/conv_filter_visualization.py

@@ -0,0 +1,184 @@
+'''Visualization of the filters of VGG16, via gradient ascent in input space.
+
+This script can run on CPU in a few minutes (with the TensorFlow backend).
+
+Results example: http://i.imgur.com/4nj4KjN.jpg
+
+Before running this script, download the weights for the VGG16 model at:
+https://drive.google.com/file/d/0Bz7KyqmuGsilT0J5dmRCM0ROVHc/view?usp=sharing
+(source: https://gist.github.com/baraldilorenzo/07d7802847aaad0a35d3)
+and make sure the variable `weights_path` in this script matches the location of the file.
+'''
+from __future__ import print_function
+from scipy.misc import imsave
+import numpy as np
+import time
+import os
+import h5py
+
+from keras.models import Sequential
+from keras.layers import Convolution2D, ZeroPadding2D, MaxPooling2D
+from keras import backend as K
+
+# dimensions of the generated pictures for each filter.
+img_width = 128
+img_height = 128
+
+# path to the model weights file.
+weights_path = 'vgg16_weights.h5'
+
+# the name of the layer we want to visualize (see model definition below)
+layer_name = 'conv5_1'
+
+# util function to convert a tensor into a valid image
+def deprocess_image(x):
+    # normalize tensor: center on 0., ensure std is 0.1
+    x -= x.mean()
+    x /= (x.std() + 1e-5)
+    x *= 0.1
+
+    # clip to [0, 1]
+    x += 0.5
+    x = np.clip(x, 0, 1)
+
+    # convert to RGB array
+    x *= 255
+    x = x.transpose((1, 2, 0))
+    x = np.clip(x, 0, 255).astype('uint8')
+    return x
+
+# build the VGG16 network
+model = Sequential()
+model.add(ZeroPadding2D((1, 1), batch_input_shape=(1, 3, img_width, img_height)))
+first_layer = model.layers[-1]
+# this is a placeholder tensor that will contain our generated images
+input_img = first_layer.input
+
+model.add(Convolution2D(64, 3, 3, activation='relu', name='conv1_1'))
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(64, 3, 3, activation='relu', name='conv1_2'))
+model.add(MaxPooling2D((2, 2), strides=(2, 2)))
+
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(128, 3, 3, activation='relu', name='conv2_1'))
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(128, 3, 3, activation='relu', name='conv2_2'))
+model.add(MaxPooling2D((2, 2), strides=(2, 2)))
+
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(256, 3, 3, activation='relu', name='conv3_1'))
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(256, 3, 3, activation='relu', name='conv3_2'))
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(256, 3, 3, activation='relu', name='conv3_3'))
+model.add(MaxPooling2D((2, 2), strides=(2, 2)))
+
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(512, 3, 3, activation='relu', name='conv4_1'))
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(512, 3, 3, activation='relu', name='conv4_2'))
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(512, 3, 3, activation='relu', name='conv4_3'))
+model.add(MaxPooling2D((2, 2), strides=(2, 2)))
+
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(512, 3, 3, activation='relu', name='conv5_1'))
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(512, 3, 3, activation='relu', name='conv5_2'))
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(512, 3, 3, activation='relu', name='conv5_3'))
+model.add(MaxPooling2D((2, 2), strides=(2, 2)))
+
+# load the weights of the VGG16 networks
+# (trained on ImageNet, won the ILSVRC competition in 2014)
+# note: when there is a complete match between your model definition
+# and your weight savefile, you can simply call model.load_weights(filename)
+assert os.path.exists(weights_path), 'Model weights not found (see "weights_path" variable in script).'
+f = h5py.File(weights_path)
+for k in range(f.attrs['nb_layers']):
+    if k >= len(model.layers):
+        # we don't look at the last (fully-connected) layers in the savefile
+        break
+    g = f['layer_{}'.format(k)]
+    weights = [g['param_{}'.format(p)] for p in range(g.attrs['nb_params'])]
+    model.layers[k].set_weights(weights)
+f.close()
+print('Model loaded.')
+
+# get the symbolic outputs of each "key" layer (we gave them unique names).
+layer_dict = dict([(layer.name, layer) for layer in model.layers])
+
+
+def normalize(x):
+    # utility function to normalize a tensor by its L2 norm
+    return x / (K.sqrt(K.mean(K.square(x))) + 1e-5)
+
+
+kept_filters = []
+for filter_index in range(0, 200):
+    # we only scan through the first 200 filters,
+    # but there are actually 512 of them
+    print('Processing filter %d' % filter_index)
+    start_time = time.time()
+
+    # we build a loss function that maximizes the activation
+    # of the nth filter of the layer considered
+    layer_output = layer_dict[layer_name].output
+    loss = K.mean(layer_output[:, filter_index, :, :])
+
+    # we compute the gradient of the input picture wrt this loss
+    grads = K.gradients(loss, input_img)[0]
+
+    # normalization trick: we normalize the gradient
+    grads = normalize(grads)
+
+    # this function returns the loss and grads given the input picture
+    iterate = K.function([input_img], [loss, grads])
+
+    # step size for gradient ascent
+    step = 1.
+
+    # we start from a gray image with some random noise
+    input_img_data = np.random.random((1, 3, img_width, img_height)) * 20 + 128.
+
+    # we run gradient ascent for 20 steps
+    for i in range(20):
+        loss_value, grads_value = iterate([input_img_data])
+        input_img_data += grads_value * step
+
+        print('Current loss value:', loss_value)
+        if loss_value <= 0.:
+            # some filters get stuck to 0, we can skip them
+            break
+
+    # decode the resulting input image
+    if loss_value > 0:
+        img = deprocess_image(input_img_data[0])
+        kept_filters.append((img, loss_value))
+    end_time = time.time()
+    print('Filter %d processed in %ds' % (filter_index, end_time - start_time))
+
+# we will stich the best 64 filters on a 8 x 8 grid.
+n = 8
+
+# the filters that have the highest loss are assumed to be better-looking.
+# we will only keep the top 64 filters.
+kept_filters.sort(key=lambda x: x[1], reverse=True)
+kept_filters = kept_filters[:n * n]
+
+# build a black picture with enough space for
+# our 8 x 8 filters of size 128 x 128, with a 5px margin in between
+margin = 5
+width = n * img_width + (n - 1) * margin
+height = n * img_height + (n - 1) * margin
+stitched_filters = np.zeros((width, height, 3))
+
+# fill the picture with our saved filters
+for i in range(n):
+    for j in range(n):
+        img, loss = kept_filters[i * n + j]
+        stitched_filters[(img_width + margin) * i: (img_width + margin) * i + img_width,
+                         (img_height + margin) * j: (img_height + margin) * j + img_height, :] = img
+
+# save the result to disk
+imsave('stitched_filters_%dx%d.png' % (n, n), stitched_filters)

examples/deep_dream.py

@@ -0,0 +1,233 @@
+'''Deep Dreaming in Keras.
+
+Run the script with:
+```
+python deep_dream.py path_to_your_base_image.jpg prefix_for_results
+```
+e.g.:
+```
+python deep_dream.py img/mypic.jpg results/dream
+```
+
+It is preferable to run this script on GPU, for speed.
+If running on CPU, prefer the TensorFlow backend (much faster).
+
+Example results: http://i.imgur.com/FX6ROg9.jpg
+'''
+from __future__ import print_function
+from scipy.misc import imread, imresize, imsave
+import numpy as np
+from scipy.optimize import fmin_l_bfgs_b
+import time
+import argparse
+import h5py
+import os
+
+from keras.models import Sequential
+from keras.layers import Convolution2D, ZeroPadding2D, MaxPooling2D
+from keras import backend as K
+
+parser = argparse.ArgumentParser(description='Deep Dreams with Keras.')
+parser.add_argument('base_image_path', metavar='base', type=str,
+                    help='Path to the image to transform.')
+parser.add_argument('result_prefix', metavar='res_prefix', type=str,
+                    help='Prefix for the saved results.')
+
+args = parser.parse_args()
+base_image_path = args.base_image_path
+result_prefix = args.result_prefix
+
+# dimensions of the generated picture.
+img_width = 600
+img_height = 600
+
+# path to the model weights file.
+weights_path = 'vgg16_weights.h5'
+
+# some settings we found interesting
+saved_settings = {
+    'bad_trip': {'features': {'conv4_1': 0.05,
+                              'conv4_2': 0.01,
+                              'conv4_3': 0.01},
+                 'continuity': 0.1,
+                 'dream_l2': 0.8,
+                 'jitter': 5},
+    'dreamy': {'features': {'conv5_1': 0.05,
+                            'conv5_2': 0.02},
+               'continuity': 0.1,
+               'dream_l2': 0.02,
+               'jitter': 0},
+}
+# the settings we will use in this experiment
+settings = saved_settings['dreamy']
+
+# util function to open, resize and format pictures into appropriate tensors
+def preprocess_image(image_path):
+    img = imresize(imread(image_path), (img_width, img_height))
+    img = img.transpose((2, 0, 1)).astype('float64')
+    img = np.expand_dims(img, axis=0)
+    return img
+
+# util function to convert a tensor into a valid image
+def deprocess_image(x):
+    x = x.transpose((1, 2, 0))
+    x = np.clip(x, 0, 255).astype('uint8')
+    return x
+
+# build the VGG16 network
+model = Sequential()
+model.add(ZeroPadding2D((1, 1), batch_input_shape=(1, 3, img_width, img_height)))
+first_layer = model.layers[-1]
+# this is a placeholder tensor that will contain our generated images
+dream = first_layer.input
+
+model.add(Convolution2D(64, 3, 3, activation='relu', name='conv1_1'))
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(64, 3, 3, activation='relu', name='conv1_2'))
+model.add(MaxPooling2D((2, 2), strides=(2, 2)))
+
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(128, 3, 3, activation='relu', name='conv2_1'))
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(128, 3, 3, activation='relu', name='conv2_2'))
+model.add(MaxPooling2D((2, 2), strides=(2, 2)))
+
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(256, 3, 3, activation='relu', name='conv3_1'))
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(256, 3, 3, activation='relu', name='conv3_2'))
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(256, 3, 3, activation='relu', name='conv3_3'))
+model.add(MaxPooling2D((2, 2), strides=(2, 2)))
+
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(512, 3, 3, activation='relu', name='conv4_1'))
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(512, 3, 3, activation='relu', name='conv4_2'))
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(512, 3, 3, activation='relu', name='conv4_3'))
+model.add(MaxPooling2D((2, 2), strides=(2, 2)))
+
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(512, 3, 3, activation='relu', name='conv5_1'))
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(512, 3, 3, activation='relu', name='conv5_2'))
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(512, 3, 3, activation='relu', name='conv5_3'))
+model.add(MaxPooling2D((2, 2), strides=(2, 2)))
+
+# load the weights of the VGG16 networks
+# (trained on ImageNet, won the ILSVRC competition in 2014)
+# note: when there is a complete match between your model definition
+# and your weight savefile, you can simply call model.load_weights(filename)
+assert os.path.exists(weights_path), 'Model weights not found (see "weights_path" variable in script).'
+f = h5py.File(weights_path)
+for k in range(f.attrs['nb_layers']):
+    if k >= len(model.layers):
+        # we don't look at the last (fully-connected) layers in the savefile
+        break
+    g = f['layer_{}'.format(k)]
+    weights = [g['param_{}'.format(p)] for p in range(g.attrs['nb_params'])]
+    model.layers[k].set_weights(weights)
+f.close()
+print('Model loaded.')
+
+# get the symbolic outputs of each "key" layer (we gave them unique names).
+layer_dict = dict([(layer.name, layer) for layer in model.layers])
+
+# continuity loss util function
+def continuity_loss(x):
+    assert K.ndim(x) == 4
+    a = K.square(x[:, :, :img_width-1, :img_height-1] - x[:, :, 1:, :img_height-1])
+    b = K.square(x[:, :, :img_width-1, :img_height-1] - x[:, :, :img_width-1, 1:])
+    return K.sum(K.pow(a + b, 1.25))
+
+# define the loss
+loss = K.variable(0.)
+for layer_name in settings['features']:
+    # add the L2 norm of the features of a layer to the loss
+    assert layer_name in layer_dict.keys(), 'Layer ' + layer_name + ' not found in model.'
+    coeff = settings['features'][layer_name]
+    x = layer_dict[layer_name].output
+    shape = layer_dict[layer_name].output_shape
+    # we avoid border artifacts by only involving non-border pixels in the loss
+    loss -= coeff * K.sum(K.square(x[:, :, 2: shape[2]-2, 2: shape[3]-2])) / np.prod(shape[1:])
+
+# add continuity loss (gives image local coherence, can result in an artful blur)
+loss += settings['continuity'] * continuity_loss(dream) / (3 * img_width * img_height)
+# add image L2 norm to loss (prevents pixels from taking very high values, makes image darker)
+loss += settings['dream_l2'] * K.sum(K.square(dream)) / (3 * img_width * img_height)
+
+# feel free to further modify the loss as you see fit, to achieve new effects...
+
+# compute the gradients of the dream wrt the loss
+grads = K.gradients(loss, dream)
+
+outputs = [loss]
+if type(grads) in {list, tuple}:
+    outputs += grads
+else:
+    outputs.append(grads)
+
+f_outputs = K.function([dream], outputs)
+def eval_loss_and_grads(x):
+    x = x.reshape((1, 3, img_width, img_height))
+    outs = f_outputs([x])
+    loss_value = outs[0]
+    if len(outs[1:]) == 1:
+        grad_values = outs[1].flatten().astype('float64')
+    else:
+        grad_values = np.array(outs[1:]).flatten().astype('float64')
+    return loss_value, grad_values
+
+# this Evaluator class makes it possible
+# to compute loss and gradients in one pass
+# while retrieving them via two separate functions,
+# "loss" and "grads". This is done because scipy.optimize
+# requires separate functions for loss and gradients,
+# but computing them separately would be inefficient.
+class Evaluator(object):
+    def __init__(self):
+        self.loss_value = None
+        self.grad_values = None
+
+    def loss(self, x):
+        assert self.loss_value is None
+        loss_value, grad_values = eval_loss_and_grads(x)
+        self.loss_value = loss_value
+        self.grad_values = grad_values
+        return self.loss_value
+
+    def grads(self, x):
+        assert self.loss_value is not None
+        grad_values = np.copy(self.grad_values)
+        self.loss_value = None
+        self.grad_values = None
+        return grad_values
+
+evaluator = Evaluator()
+
+# run scipy-based optimization (L-BFGS) over the pixels of the generated image
+# so as to minimize the loss
+x = preprocess_image(base_image_path)
+for i in range(5):
+    print('Start of iteration', i)
+    start_time = time.time()
+
+    # add a random jitter to the initial image. This will be reverted at decoding time
+    random_jitter = (settings['jitter'] * 2) * (np.random.random((3, img_width, img_height)) - 0.5)
+    x += random_jitter
+
+    # run L-BFGS for 7 steps
+    x, min_val, info = fmin_l_bfgs_b(evaluator.loss, x.flatten(),
+                                     fprime=evaluator.grads, maxfun=7)
+    print('Current loss value:', min_val)
+    # decode the dream and save it
+    x = x.reshape((3, img_width, img_height))
+    x -= random_jitter
+    img = deprocess_image(x)
+    fname = result_prefix + '_at_iteration_%d.png' % i
+    imsave(fname, img)
+    end_time = time.time()
+    print('Image saved as', fname)
+    print('Iteration %d completed in %ds' % (i, end_time - start_time))

examples/imdb_bidirectional_lstm.py

@@ -1,8 +1,5 @@
 '''Train a Bidirectional LSTM on the IMDB sentiment classification task.
 
-GPU command:
-    THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 python imdb_bidirectional_lstm.py
-
 Output after 4 epochs on CPU: ~0.8146
 Time per epoch on CPU (Core i7): ~150s.
 '''
@@ -12,11 +9,8 @@ import numpy as np
 np.random.seed(1337)  # for reproducibility
 
 from keras.preprocessing import sequence
-from keras.utils.np_utils import accuracy
-from keras.models import Graph
-from keras.layers.core import Dense, Dropout
-from keras.layers.embeddings import Embedding
-from keras.layers.recurrent import LSTM
+from keras.models import Model
+from keras.layers import Dense, Dropout, Embedding, LSTM, Input, merge
 from keras.datasets import imdb
 
 
@@ -25,8 +19,7 @@ maxlen = 100  # cut texts after this number of words (among top max_features mos
 batch_size = 32
 
 print('Loading data...')
-(X_train, y_train), (X_test, y_test) = imdb.load_data(nb_words=max_features,
-                                                      test_split=0.2)
+(X_train, y_train), (X_test, y_test) = imdb.load_data(nb_words=max_features)
 print(len(X_train), 'train sequences')
 print(len(X_test), 'test sequences')
 
@@ -38,25 +31,30 @@ print('X_test shape:', X_test.shape)
 y_train = np.array(y_train)
 y_test = np.array(y_test)
 
-print('Build model...')
-model = Graph()
-model.add_input(name='input', input_shape=(maxlen,), dtype=int)
-model.add_node(Embedding(max_features, 128, input_length=maxlen),
-               name='embedding', input='input')
-model.add_node(LSTM(64), name='forward', input='embedding')
-model.add_node(LSTM(64, go_backwards=True), name='backward', input='embedding')
-model.add_node(Dropout(0.5), name='dropout', inputs=['forward', 'backward'])
-model.add_node(Dense(1, activation='sigmoid'), name='sigmoid', input='dropout')
-model.add_output(name='output', input='sigmoid')
+
+# this is the placeholder tensor for the input sequences
+sequence = Input(shape=(maxlen,), dtype='int32')
+# this embedding layer will transform the sequences of integers
+# into vectors of size 128
+embedded = Embedding(max_features, 128, input_length=maxlen)(sequence)
+
+# apply forwards LSTM
+forwards = LSTM(64)(embedded)
+# apply backwards LSTM
+backwards = LSTM(64, go_backwards=True)(embedded)
+
+# concatenate the outputs of the 2 LSTMs
+merged = merge([forwards, backwards], mode='concat', concat_axis=-1)
+after_dp = Dropout(0.5)(merged)
+output = Dense(1, activation='sigmoid')(after_dp)
+
+model = Model(input=sequence, output=output)
 
 # try using different optimizers and different optimizer configs
-model.compile('adam', {'output': 'binary_crossentropy'})
+model.compile('adam', 'binary_crossentropy', metrics=['accuracy'])
 
 print('Train...')
-model.fit({'input': X_train, 'output': y_train},
+model.fit(X_train, y_train,
           batch_size=batch_size,
-          nb_epoch=4)
-acc = accuracy(y_test,
-               np.round(np.array(model.predict({'input': X_test},
-                                               batch_size=batch_size)['output'])))
-print('Test accuracy:', acc)
+          nb_epoch=4,
+          validation_data=[X_test, y_test])

examples/imdb_cnn.py

@@ -1,8 +1,9 @@
 '''This example demonstrates the use of Convolution1D for text classification.
 
-Run on GPU: THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 python imdb_cnn.py
+Gets to 0.89 test accuracy after 2 epochs.
+90s/epoch on Intel i5 2.4Ghz CPU.
+10s/epoch on Tesla K40 GPU.
 
-Get to 0.835 test accuracy after 2 epochs. 100s/epoch on K520 GPU.
 '''
 
 from __future__ import print_function
@@ -11,25 +12,25 @@ np.random.seed(1337)  # for reproducibility
 
 from keras.preprocessing import sequence
 from keras.models import Sequential
-from keras.layers.core import Dense, Dropout, Activation, Flatten
-from keras.layers.embeddings import Embedding
-from keras.layers.convolutional import Convolution1D, MaxPooling1D
+from keras.layers import Dense, Dropout, Activation, Flatten
+from keras.layers import Embedding
+from keras.layers import Convolution1D, MaxPooling1D
 from keras.datasets import imdb
+from keras import backend as K
 
 
 # set parameters:
 max_features = 5000
-maxlen = 100
+maxlen = 400
 batch_size = 32
-embedding_dims = 100
+embedding_dims = 50
 nb_filter = 250
 filter_length = 3
 hidden_dims = 250
 nb_epoch = 2
 
 print('Loading data...')
-(X_train, y_train), (X_test, y_test) = imdb.load_data(nb_words=max_features,
-                                                      test_split=0.2)
+(X_train, y_train), (X_test, y_test) = imdb.load_data(nb_words=max_features)
 print(len(X_train), 'train sequences')
 print(len(X_test), 'test sequences')
 
@@ -44,8 +45,10 @@ model = Sequential()
 
 # we start off with an efficient embedding layer which maps
 # our vocab indices into embedding_dims dimensions
-model.add(Embedding(max_features, embedding_dims, input_length=maxlen))
-model.add(Dropout(0.25))
+model.add(Embedding(max_features,
+                    embedding_dims,
+                    input_length=maxlen,
+                    dropout=0.2))
 
 # we add a Convolution1D, which will learn nb_filter
 # word group filters of size filter_length:
@@ -54,8 +57,8 @@ model.add(Convolution1D(nb_filter=nb_filter,
                         border_mode='valid',
                         activation='relu',
                         subsample_length=1))
-# we use standard max pooling (halving the output of the previous layer):
-model.add(MaxPooling1D(pool_length=2))
+# we use max pooling:
+model.add(MaxPooling1D(pool_length=model.output_shape[1]))
 
 # We flatten the output of the conv layer,
 # so that we can add a vanilla dense layer:
@@ -63,7 +66,7 @@ model.add(Flatten())
 
 # We add a vanilla hidden layer:
 model.add(Dense(hidden_dims))
-model.add(Dropout(0.25))
+model.add(Dropout(0.2))
 model.add(Activation('relu'))
 
 # We project onto a single unit output layer, and squash it with a sigmoid:
@@ -71,8 +74,9 @@ model.add(Dense(1))
 model.add(Activation('sigmoid'))
 
 model.compile(loss='binary_crossentropy',
-              optimizer='rmsprop',
-              class_mode='binary')
-model.fit(X_train, y_train, batch_size=batch_size,
-          nb_epoch=nb_epoch, show_accuracy=True,
+              optimizer='adam',
+              metrics=['accuracy'])
+model.fit(X_train, y_train,
+          batch_size=batch_size,
+          nb_epoch=nb_epoch,
           validation_data=(X_test, y_test))

examples/imdb_cnn_lstm.py

@@ -1,22 +1,18 @@
 '''Train a recurrent convolutional network on the IMDB sentiment
 classification task.
 
-GPU command:
-    THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 python imdb_lstm.py
-
-Get to 0.8498 test accuracy after 2 epochs. 41s/epoch on K520 GPU.
+Gets to 0.8498 test accuracy after 2 epochs. 41s/epoch on K520 GPU.
 '''
-
 from __future__ import print_function
 import numpy as np
 np.random.seed(1337)  # for reproducibility
 
 from keras.preprocessing import sequence
 from keras.models import Sequential
-from keras.layers.core import Dense, Dropout, Activation
-from keras.layers.embeddings import Embedding
-from keras.layers.recurrent import LSTM, GRU, SimpleRNN
-from keras.layers.convolutional import Convolution1D, MaxPooling1D
+from keras.layers import Dense, Dropout, Activation
+from keras.layers import Embedding
+from keras.layers import LSTM, GRU, SimpleRNN
+from keras.layers import Convolution1D, MaxPooling1D
 from keras.datasets import imdb
 
 
@@ -26,9 +22,9 @@ maxlen = 100
 embedding_size = 128
 
 # Convolution
-filter_length = 3
+filter_length = 5
 nb_filter = 64
-pool_length = 2
+pool_length = 4
 
 # LSTM
 lstm_output_size = 70
@@ -44,7 +40,7 @@ Only 2 epochs are needed as the dataset is very small.
 '''
 
 print('Loading data...')
-(X_train, y_train), (X_test, y_test) = imdb.load_data(nb_words=max_features, test_split=0.2)
+(X_train, y_train), (X_test, y_test) = imdb.load_data(nb_words=max_features)
 print(len(X_train), 'train sequences')
 print(len(X_test), 'test sequences')
 
@@ -71,12 +67,11 @@ model.add(Activation('sigmoid'))
 
 model.compile(loss='binary_crossentropy',
               optimizer='adam',
-              class_mode='binary')
+              metrics=['accuracy'])
 
 print('Train...')
 model.fit(X_train, y_train, batch_size=batch_size, nb_epoch=nb_epoch,
-          validation_data=(X_test, y_test), show_accuracy=True)
-score, acc = model.evaluate(X_test, y_test, batch_size=batch_size,
-                            show_accuracy=True)
+          validation_data=(X_test, y_test))
+score, acc = model.evaluate(X_test, y_test, batch_size=batch_size)
 print('Test score:', score)
 print('Test accuracy:', acc)

examples/imdb_lstm.py

@@ -1,7 +1,7 @@
-'''Train a LSTM on the IMDB sentiment classification task.
+'''Trains a LSTM on the IMDB sentiment classification task.
 
 The dataset is actually too small for LSTM to be of any advantage
-compared to simpler, much faster methods such as TF-IDF+LogReg.
+compared to simpler, much faster methods such as TF-IDF + LogReg.
 
 Notes:
 
@@ -11,11 +11,7 @@ Some configurations won't converge.
 
 - LSTM loss decrease patterns during training can be quite different
 from what you see with CNNs/MLPs/etc.
-
-GPU command:
-    THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 python imdb_lstm.py
 '''
-
 from __future__ import print_function
 import numpy as np
 np.random.seed(1337)  # for reproducibility
@@ -23,22 +19,20 @@ np.random.seed(1337)  # for reproducibility
 from keras.preprocessing import sequence
 from keras.utils import np_utils
 from keras.models import Sequential
-from keras.layers.core import Dense, Dropout, Activation
-from keras.layers.embeddings import Embedding
-from keras.layers.recurrent import LSTM
+from keras.layers import Dense, Dropout, Activation, Embedding
+from keras.layers import LSTM, SimpleRNN, GRU
 from keras.datasets import imdb
 
 max_features = 20000
-maxlen = 100  # cut texts after this number of words (among top max_features most common words)
+maxlen = 80  # cut texts after this number of words (among top max_features most common words)
 batch_size = 32
 
 print('Loading data...')
-(X_train, y_train), (X_test, y_test) = imdb.load_data(nb_words=max_features,
-                                                      test_split=0.2)
+(X_train, y_train), (X_test, y_test) = imdb.load_data(nb_words=max_features)
 print(len(X_train), 'train sequences')
 print(len(X_test), 'test sequences')
 
-print("Pad sequences (samples x time)")
+print('Pad sequences (samples x time)')
 X_train = sequence.pad_sequences(X_train, maxlen=maxlen)
 X_test = sequence.pad_sequences(X_test, maxlen=maxlen)
 print('X_train shape:', X_train.shape)
@@ -46,22 +40,20 @@ print('X_test shape:', X_test.shape)
 
 print('Build model...')
 model = Sequential()
-model.add(Embedding(max_features, 128, input_length=maxlen))
-model.add(LSTM(128))  # try using a GRU instead, for fun
-model.add(Dropout(0.5))
+model.add(Embedding(max_features, 128, input_length=maxlen, dropout=0.2))
+model.add(LSTM(128, dropout_W=0.2, dropout_U=0.2))  # try using a GRU instead, for fun
 model.add(Dense(1))
 model.add(Activation('sigmoid'))
 
 # try using different optimizers and different optimizer configs
 model.compile(loss='binary_crossentropy',
               optimizer='adam',
-              class_mode="binary")
+              metrics=['accuracy'])
 
-print("Train...")
-model.fit(X_train, y_train, batch_size=batch_size, nb_epoch=3,
-          validation_data=(X_test, y_test), show_accuracy=True)
+print('Train...')
+model.fit(X_train, y_train, batch_size=batch_size, nb_epoch=15,
+          validation_data=(X_test, y_test))
 score, acc = model.evaluate(X_test, y_test,
-                            batch_size=batch_size,
-                            show_accuracy=True)
+                            batch_size=batch_size)
 print('Test score:', score)
 print('Test accuracy:', acc)

examples/inception_v3.py

@@ -0,0 +1,290 @@
+'''This script demonstrates how to build the Inception v3 architecture
+using the Keras functional API.
+We are not actually training it here, for lack of appropriate data.
+
+For more information about this architecture, see:
+
+"Rethinking the Inception Architecture for Computer Vision"
+Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, Zbigniew Wojna
+http://arxiv.org/abs/1512.00567
+'''
+from keras.layers import Convolution2D, MaxPooling2D, AveragePooling2D
+from keras.layers import BatchNormalization, Flatten, Dense, Dropout
+from keras.layers import Input, merge
+from keras.models import Model
+from keras import regularizers
+
+
+# global constants
+NB_CLASS = 1000  # number of classes
+DIM_ORDERING = 'th'  # 'th' (channels, width, height) or 'tf' (width, height, channels)
+WEIGHT_DECAY = 0.  # L2 regularization factor
+USE_BN = False  # whether to use batch normalization
+
+
+def conv2D_bn(x, nb_filter, nb_row, nb_col,
+              border_mode='same', subsample=(1, 1),
+              activation='relu', batch_norm=USE_BN,
+              weight_decay=WEIGHT_DECAY, dim_ordering=DIM_ORDERING):
+    '''Utility function to apply to a tensor a module conv + BN
+    with optional weight decay (L2 weight regularization).
+    '''
+    if weight_decay:
+        W_regularizer = regularizers.l2(weight_decay)
+        b_regularizer = regularizers.l2(weight_decay)
+    else:
+        W_regularizer = None
+        b_regularizer = None
+    x = Convolution2D(nb_filter, nb_row, nb_col,
+                      subsample=subsample,
+                      activation=activation,
+                      border_mode=border_mode,
+                      W_regularizer=W_regularizer,
+                      b_regularizer=b_regularizer,
+                      dim_ordering=dim_ordering)(x)
+    if batch_norm:
+        x = BatchNormalization()(x)
+    return x
+
+# Define image input layer
+
+if DIM_ORDERING == 'th':
+    img_input = Input(shape=(3, 299, 299))
+    CONCAT_AXIS = 1
+elif DIM_ORDERING == 'tf':
+    img_input = Input(shape=(299, 299, 3))
+    CONCAT_AXIS = 3
+else:
+    raise Exception('Invalid dim ordering: ' + str(DIM_ORDERING))
+
+# Entry module
+
+x = conv2D_bn(img_input, 32, 3, 3, subsample=(2, 2), border_mode='valid')
+x = conv2D_bn(x, 32, 3, 3, border_mode='valid')
+x = conv2D_bn(x, 64, 3, 3)
+x = MaxPooling2D((3, 3), strides=(2, 2), dim_ordering=DIM_ORDERING)(x)
+
+x = conv2D_bn(x, 80, 1, 1, border_mode='valid')
+x = conv2D_bn(x, 192, 3, 3, border_mode='valid')
+x = MaxPooling2D((3, 3), strides=(2, 2), dim_ordering=DIM_ORDERING)(x)
+
+# mixed: 35 x 35 x 256
+
+branch1x1 = conv2D_bn(x, 64, 1, 1)
+
+branch5x5 = conv2D_bn(x, 48, 1, 1)
+branch5x5 = conv2D_bn(branch5x5, 64, 5, 5)
+
+branch3x3dbl = conv2D_bn(x, 64, 1, 1)
+branch3x3dbl = conv2D_bn(branch3x3dbl, 96, 3, 3)
+branch3x3dbl = conv2D_bn(branch3x3dbl, 96, 3, 3)
+
+branch_pool = AveragePooling2D((3, 3), strides=(1, 1), border_mode='same', dim_ordering=DIM_ORDERING)(x)
+branch_pool = conv2D_bn(branch_pool, 32, 1, 1)
+x = merge([branch1x1, branch5x5, branch3x3dbl, branch_pool], mode='concat', concat_axis=CONCAT_AXIS)
+
+# mixed_1: 35 x 35 x 288
+
+branch1x1 = conv2D_bn(x, 64, 1, 1)
+
+branch5x5 = conv2D_bn(x, 48, 1, 1)
+branch5x5 = conv2D_bn(branch5x5, 64, 5, 5)
+
+branch3x3dbl = conv2D_bn(x, 64, 1, 1)
+branch3x3dbl = conv2D_bn(branch3x3dbl, 96, 3, 3)
+branch3x3dbl = conv2D_bn(branch3x3dbl, 96, 3, 3)
+
+branch_pool = AveragePooling2D((3, 3), strides=(1, 1), border_mode='same', dim_ordering=DIM_ORDERING)(x)
+branch_pool = conv2D_bn(branch_pool, 64, 1, 1)
+x = merge([branch1x1, branch5x5, branch3x3dbl, branch_pool], mode='concat', concat_axis=CONCAT_AXIS)
+
+# mixed2: 35 x 35 x 288
+
+branch1x1 = conv2D_bn(x, 64, 1, 1)
+
+branch5x5 = conv2D_bn(x, 48, 1, 1)
+branch5x5 = conv2D_bn(branch5x5, 64, 5, 5)
+
+branch3x3dbl = conv2D_bn(x, 64, 1, 1)
+branch3x3dbl = conv2D_bn(branch3x3dbl, 96, 3, 3)
+branch3x3dbl = conv2D_bn(branch3x3dbl, 96, 3, 3)
+
+branch_pool = AveragePooling2D((3, 3), strides=(1, 1), border_mode='same', dim_ordering=DIM_ORDERING)(x)
+branch_pool = conv2D_bn(branch_pool, 64, 1, 1)
+x = merge([branch1x1, branch5x5, branch3x3dbl, branch_pool], mode='concat', concat_axis=CONCAT_AXIS)
+
+# mixed3: 17 x 17 x 768
+
+branch3x3 = conv2D_bn(x, 384, 3, 3, subsample=(2, 2), border_mode='valid')
+
+branch3x3dbl = conv2D_bn(x, 64, 1, 1)
+branch3x3dbl = conv2D_bn(branch3x3dbl, 96, 3, 3)
+branch3x3dbl = conv2D_bn(branch3x3dbl, 96, 3, 3, subsample=(2, 2), border_mode='valid')
+
+branch_pool = MaxPooling2D((3, 3), strides=(2, 2), dim_ordering=DIM_ORDERING)(x)
+x = merge([branch3x3, branch3x3dbl, branch_pool], mode='concat', concat_axis=CONCAT_AXIS)
+
+# mixed4: 17 x 17 x 768
+
+branch1x1 = conv2D_bn(x, 192, 1, 1)
+
+branch7x7 = conv2D_bn(x, 128, 1, 1)
+branch7x7 = conv2D_bn(branch7x7, 128, 1, 7)
+branch7x7 = conv2D_bn(branch7x7, 192, 7, 1)
+
+branch7x7dbl = conv2D_bn(x, 128, 1, 1)
+branch7x7dbl = conv2D_bn(branch7x7dbl, 128, 7, 1)
+branch7x7dbl = conv2D_bn(branch7x7dbl, 128, 1, 7)
+branch7x7dbl = conv2D_bn(branch7x7dbl, 128, 7, 1)
+branch7x7dbl = conv2D_bn(branch7x7dbl, 192, 1, 7)
+
+branch_pool = AveragePooling2D((3, 3), strides=(1, 1), border_mode='same', dim_ordering=DIM_ORDERING)(x)
+branch_pool = conv2D_bn(branch_pool, 192, 1, 1)
+x = merge([branch1x1, branch7x7, branch7x7dbl, branch_pool], mode='concat', concat_axis=CONCAT_AXIS)
+
+# mixed5: 17 x 17 x 768
+
+branch1x1 = conv2D_bn(x, 192, 1, 1)
+
+branch7x7 = conv2D_bn(x, 160, 1, 1)
+branch7x7 = conv2D_bn(branch7x7, 160, 1, 7)
+branch7x7 = conv2D_bn(branch7x7, 192, 7, 1)
+
+branch7x7dbl = conv2D_bn(x, 160, 1, 1)
+branch7x7dbl = conv2D_bn(branch7x7dbl, 160, 7, 1)
+branch7x7dbl = conv2D_bn(branch7x7dbl, 160, 1, 7)
+branch7x7dbl = conv2D_bn(branch7x7dbl, 160, 7, 1)
+branch7x7dbl = conv2D_bn(branch7x7dbl, 192, 1, 7)
+
+branch_pool = AveragePooling2D((3, 3), strides=(1, 1), border_mode='same', dim_ordering=DIM_ORDERING)(x)
+branch_pool = conv2D_bn(branch_pool, 192, 1, 1)
+x = merge([branch1x1, branch7x7, branch7x7dbl, branch_pool], mode='concat', concat_axis=CONCAT_AXIS)
+
+# mixed5: 17 x 17 x 768
+
+branch1x1 = conv2D_bn(x, 192, 1, 1)
+
+branch7x7 = conv2D_bn(x, 160, 1, 1)
+branch7x7 = conv2D_bn(branch7x7, 160, 1, 7)
+branch7x7 = conv2D_bn(branch7x7, 192, 7, 1)
+
+branch7x7dbl = conv2D_bn(x, 160, 1, 1)
+branch7x7dbl = conv2D_bn(branch7x7dbl, 160, 7, 1)
+branch7x7dbl = conv2D_bn(branch7x7dbl, 160, 1, 7)
+branch7x7dbl = conv2D_bn(branch7x7dbl, 160, 7, 1)
+branch7x7dbl = conv2D_bn(branch7x7dbl, 192, 1, 7)
+
+branch_pool = AveragePooling2D((3, 3), strides=(1, 1), border_mode='same', dim_ordering=DIM_ORDERING)(x)
+branch_pool = conv2D_bn(branch_pool, 192, 1, 1)
+x = merge([branch1x1, branch7x7, branch7x7dbl, branch_pool], mode='concat', concat_axis=CONCAT_AXIS)
+
+# mixed6: 17 x 17 x 768
+
+branch1x1 = conv2D_bn(x, 192, 1, 1)
+
+branch7x7 = conv2D_bn(x, 160, 1, 1)
+branch7x7 = conv2D_bn(branch7x7, 160, 1, 7)
+branch7x7 = conv2D_bn(branch7x7, 192, 7, 1)
+
+branch7x7dbl = conv2D_bn(x, 160, 1, 1)
+branch7x7dbl = conv2D_bn(branch7x7dbl, 160, 7, 1)
+branch7x7dbl = conv2D_bn(branch7x7dbl, 192, 1, 7)
+branch7x7dbl = conv2D_bn(branch7x7dbl, 160, 7, 1)
+branch7x7dbl = conv2D_bn(branch7x7dbl, 192, 1, 7)
+
+branch_pool = AveragePooling2D((3, 3), strides=(1, 1), border_mode='same', dim_ordering=DIM_ORDERING)(x)
+branch_pool = conv2D_bn(branch_pool, 192, 1, 1)
+x = merge([branch1x1, branch7x7, branch7x7dbl, branch_pool], mode='concat', concat_axis=CONCAT_AXIS)
+
+# mixed7: 17 x 17 x 768
+
+branch1x1 = conv2D_bn(x, 192, 1, 1)
+
+branch7x7 = conv2D_bn(x, 192, 1, 1)
+branch7x7 = conv2D_bn(branch7x7, 192, 1, 7)
+branch7x7 = conv2D_bn(branch7x7, 192, 7, 1)
+
+branch7x7dbl = conv2D_bn(x, 160, 1, 1)
+branch7x7dbl = conv2D_bn(branch7x7dbl, 192, 7, 1)
+branch7x7dbl = conv2D_bn(branch7x7dbl, 192, 1, 7)
+branch7x7dbl = conv2D_bn(branch7x7dbl, 192, 7, 1)
+branch7x7dbl = conv2D_bn(branch7x7dbl, 192, 1, 7)
+
+branch_pool = AveragePooling2D((3, 3), strides=(1, 1), border_mode='same', dim_ordering=DIM_ORDERING)(x)
+branch_pool = conv2D_bn(branch_pool, 192, 1, 1)
+x = merge([branch1x1, branch7x7, branch7x7dbl, branch_pool], mode='concat', concat_axis=CONCAT_AXIS)
+
+# Auxiliary head
+
+aux_logits = AveragePooling2D((5, 5), strides=(3, 3), dim_ordering=DIM_ORDERING)(x)
+aux_logits = conv2D_bn(aux_logits, 128, 1, 1)
+aux_logits = conv2D_bn(aux_logits, 728, 5, 5, border_mode='valid')
+aux_logits = Flatten()(aux_logits)
+aux_preds = Dense(NB_CLASS, activation='softmax')(aux_logits)
+
+# mixed8: 8 x 8 x 1280
+
+branch3x3 = conv2D_bn(x, 192, 1, 1)
+branch3x3 = conv2D_bn(branch3x3, 320, 3, 3, subsample=(2, 2), border_mode='valid')
+
+branch7x7x3 = conv2D_bn(x, 192, 1, 1)
+branch7x7x3 = conv2D_bn(branch7x7x3, 192, 1, 7)
+branch7x7x3 = conv2D_bn(branch7x7x3, 192, 7, 1)
+branch7x7x3 = conv2D_bn(branch7x7x3, 192, 3, 3, subsample=(2, 2), border_mode='valid')
+
+branch_pool = AveragePooling2D((3, 3), strides=(2, 2), dim_ordering=DIM_ORDERING)(x)
+x = merge([branch3x3, branch7x7x3, branch_pool], mode='concat', concat_axis=CONCAT_AXIS)
+
+# mixed9: 8 x 8 x 2048
+
+branch1x1 = conv2D_bn(x, 320, 1, 1)
+
+branch3x3 = conv2D_bn(x, 384, 1, 1)
+branch3x3_1 = conv2D_bn(branch3x3, 384, 1, 3)
+branch3x3_2 = conv2D_bn(branch3x3, 384, 3, 1)
+branch3x3 = merge([branch3x3_1, branch3x3_2], mode='concat', concat_axis=CONCAT_AXIS)
+
+branch3x3dbl = conv2D_bn(x, 448, 1, 1)
+branch3x3dbl = conv2D_bn(branch3x3dbl, 384, 3, 3)
+branch3x3dbl_1 = conv2D_bn(branch3x3dbl, 384, 1, 3)
+branch3x3dbl_2 = conv2D_bn(branch3x3dbl, 384, 3, 1)
+branch3x3dbl = merge([branch3x3dbl_1, branch3x3dbl_2], mode='concat', concat_axis=CONCAT_AXIS)
+
+branch_pool = AveragePooling2D((3, 3), strides=(1, 1), border_mode='same', dim_ordering=DIM_ORDERING)(x)
+branch_pool = conv2D_bn(branch_pool, 192, 1, 1)
+x = merge([branch1x1, branch3x3, branch3x3dbl, branch_pool], mode='concat', concat_axis=CONCAT_AXIS)
+
+# mixed10: 8 x 8 x 2048
+
+branch1x1 = conv2D_bn(x, 320, 1, 1)
+
+branch3x3 = conv2D_bn(x, 384, 1, 1)
+branch3x3_1 = conv2D_bn(branch3x3, 384, 1, 3)
+branch3x3_2 = conv2D_bn(branch3x3, 384, 3, 1)
+branch3x3 = merge([branch3x3_1, branch3x3_2], mode='concat', concat_axis=CONCAT_AXIS)
+
+branch3x3dbl = conv2D_bn(x, 448, 1, 1)
+branch3x3dbl = conv2D_bn(branch3x3dbl, 384, 3, 3)
+branch3x3dbl_1 = conv2D_bn(branch3x3dbl, 384, 1, 3)
+branch3x3dbl_2 = conv2D_bn(branch3x3dbl, 384, 3, 1)
+branch3x3dbl = merge([branch3x3dbl_1, branch3x3dbl_2], mode='concat', concat_axis=CONCAT_AXIS)
+
+branch_pool = AveragePooling2D((3, 3), strides=(1, 1), border_mode='same', dim_ordering=DIM_ORDERING)(x)
+branch_pool = conv2D_bn(branch_pool, 192, 1, 1)
+x = merge([branch1x1, branch3x3, branch3x3dbl, branch_pool], mode='concat', concat_axis=CONCAT_AXIS)
+
+# Final pooling and prediction
+
+x = AveragePooling2D((8, 8), strides=(1, 1), dim_ordering=DIM_ORDERING)(x)
+x = Dropout(0.5)(x)
+x = Flatten()(x)
+preds = Dense(NB_CLASS, activation='softmax')(x)
+
+# Define model
+
+model = Model(input=img_input, output=[preds, aux_preds])
+model.compile('rmsprop', 'categorical_crossentropy')
+
+# train via e.g. `model.fit(x_train, [y_train] * 2, batch_size=32, nb_epoch=100)`
+# Note that for a large dataset it would be preferable
+# to train using `fit_generator` (see Keras docs).

examples/kaggle_otto_nn.py

@@ -1,121 +0,0 @@
-'''This demonstrates how to reach a score of 0.4890 (local validation)
-on the Kaggle Otto challenge, with a deep net using Keras.
-
-Requires Scikit-Learn and Pandas.
-
-Recommended to run on GPU:
-    Command: THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 python kaggle_otto_nn.py
-    On EC2 g2.2xlarge instance: 19s/epoch. 6-7 minutes total training time.
-
-Best validation score at epoch 21: 0.4881
-
-Try it at home:
-    - with/without BatchNormalization (BatchNormalization helps!)
-    - with ReLU or with PReLU (PReLU helps!)
-    - with smaller layers, largers layers
-    - with more layers, less layers
-    - with different optimizers (SGD+momentum+decay is probably better than Adam!)
-
-Get the data from Kaggle:
-https://www.kaggle.com/c/otto-group-product-classification-challenge/data
-'''
-
-from __future__ import print_function
-import numpy as np
-import pandas as pd
-np.random.seed(1337)  # for reproducibility
-
-from keras.models import Sequential
-from keras.layers.core import Dense, Dropout, Activation
-from keras.layers.normalization import BatchNormalization
-from keras.layers.advanced_activations import PReLU
-from keras.utils import np_utils, generic_utils
-
-from sklearn.preprocessing import LabelEncoder
-from sklearn.preprocessing import StandardScaler
-
-
-def load_data(path, train=True):
-    df = pd.read_csv(path)
-    X = df.values.copy()
-    if train:
-        np.random.shuffle(X)  # https://youtu.be/uyUXoap67N8
-        X, labels = X[:, 1:-1].astype(np.float32), X[:, -1]
-        return X, labels
-    else:
-        X, ids = X[:, 1:].astype(np.float32), X[:, 0].astype(str)
-        return X, ids
-
-
-def preprocess_data(X, scaler=None):
-    if not scaler:
-        scaler = StandardScaler()
-        scaler.fit(X)
-    X = scaler.transform(X)
-    return X, scaler
-
-
-def preprocess_labels(labels, encoder=None, categorical=True):
-    if not encoder:
-        encoder = LabelEncoder()
-        encoder.fit(labels)
-    y = encoder.transform(labels).astype(np.int32)
-    if categorical:
-        y = np_utils.to_categorical(y)
-    return y, encoder
-
-
-def make_submission(y_prob, ids, encoder, fname):
-    with open(fname, 'w') as f:
-        f.write('id,')
-        f.write(','.join([str(i) for i in encoder.classes_]))
-        f.write('\n')
-        for i, probs in zip(ids, y_prob):
-            probas = ','.join([i] + [str(p) for p in probs.tolist()])
-            f.write(probas)
-            f.write('\n')
-    print('Wrote submission to file {}.'.format(fname))
-
-print('Loading data...')
-X, labels = load_data('train.csv', train=True)
-X, scaler = preprocess_data(X)
-y, encoder = preprocess_labels(labels)
-
-X_test, ids = load_data('test.csv', train=False)
-X_test, _ = preprocess_data(X_test, scaler)
-
-nb_classes = y.shape[1]
-print(nb_classes, 'classes')
-
-dims = X.shape[1]
-print(dims, 'dims')
-
-print('Building model...')
-
-model = Sequential()
-model.add(Dense(512, input_shape=(dims,)))
-model.add(PReLU())
-model.add(BatchNormalization())
-model.add(Dropout(0.5))
-
-model.add(Dense(512))
-model.add(PReLU())
-model.add(BatchNormalization())
-model.add(Dropout(0.5))
-
-model.add(Dense(512))
-model.add(PReLU())
-model.add(BatchNormalization())
-model.add(Dropout(0.5))
-
-model.add(Dense(nb_classes))
-model.add(Activation('softmax'))
-
-model.compile(loss='categorical_crossentropy', optimizer='adam')
-
-print('Training model...')
-model.fit(X, y, nb_epoch=20, batch_size=128, validation_split=0.15)
-
-print('Generating submission...')
-proba = model.predict_proba(X_test)
-make_submission(proba, ids, encoder, fname='keras-otto.csv')

examples/lstm_benchmark.py

@@ -0,0 +1,83 @@
+'''Compare LSTM implementations on the IMDB sentiment classification task.
+
+consume_less='cpu' preprocesses input to the LSTM which typically results in
+faster computations at the expense of increased peak memory usage as the
+preprocessed input must be kept in memory.
+
+consume_less='mem' does away with the preprocessing, meaning that it might take
+a little longer, but should require less peak memory.
+
+consume_less='gpu' concatenates the input, output and forget gate's weights
+into one, large matrix, resulting in faster computation time as the GPU can
+utilize more cores, at the expense of reduced regularization because the same
+dropout is shared across the gates.
+
+Note that the relative performance of the different `consume_less` modes
+can vary depending on your device, your model and the size of your data.
+'''
+
+import time
+import numpy as np
+import matplotlib.pyplot as plt
+
+from keras.preprocessing import sequence
+from keras.models import Sequential
+from keras.layers import Embedding, Dense, LSTM
+from keras.datasets import imdb
+
+max_features = 20000
+max_length = 80
+embedding_dim = 256
+batch_size = 128
+epochs = 10
+modes = ['cpu', 'mem', 'gpu']
+
+print('Loading data...')
+(X_train, y_train), (X_test, y_test) = imdb.load_data(nb_words=max_features)
+X_train = sequence.pad_sequences(X_train, max_length)
+X_test = sequence.pad_sequences(X_test, max_length)
+
+# Compile and train different models while meauring performance.
+results = []
+for mode in modes:
+    print('Testing mode: consume_less="{}"'.format(mode))
+
+    model = Sequential()
+    model.add(Embedding(max_features, embedding_dim, input_length=max_length, dropout=0.2))
+    model.add(LSTM(embedding_dim, dropout_W=0.2, dropout_U=0.2, consume_less=mode))
+    model.add(Dense(1, activation='sigmoid'))
+    model.compile(loss='binary_crossentropy',
+                  optimizer='adam',
+                  metrics=['accuracy'])
+
+    start_time = time.time()
+    history = model.fit(X_train, y_train,
+                        batch_size=batch_size,
+                        nb_epoch=epochs,
+                        validation_data=(X_test, y_test))
+    average_time_per_epoch = (time.time() - start_time) / epochs
+
+    results.append((history, average_time_per_epoch))
+
+# Compare models' accuracy, loss and elapsed time per epoch.
+plt.style.use('ggplot')
+ax1 = plt.subplot2grid((2, 2), (0, 0))
+ax1.set_title('Accuracy')
+ax1.set_ylabel('Validation Accuracy')
+ax1.set_xlabel('Epochs')
+ax2 = plt.subplot2grid((2, 2), (1, 0))
+ax2.set_title('Loss')
+ax2.set_ylabel('Validation Loss')
+ax2.set_xlabel('Epochs')
+ax3 = plt.subplot2grid((2, 2), (0, 1), rowspan=2)
+ax3.set_title('Time')
+ax3.set_ylabel('Seconds')
+for mode, result in zip(modes, results):
+    ax1.plot(result[0].epoch, result[0].history['val_acc'], label=mode)
+    ax2.plot(result[0].epoch, result[0].history['val_loss'], label=mode)
+ax1.legend()
+ax2.legend()
+ax3.bar(np.arange(len(results)), [x[1] for x in results],
+        tick_label=modes, align='center')
+plt.tight_layout()
+plt.show()

examples/lstm_text_generation.py

@@ -12,9 +12,10 @@ has at least ~100k characters. ~1M is better.
 
 from __future__ import print_function
 from keras.models import Sequential
-from keras.layers.core import Dense, Activation, Dropout
-from keras.layers.recurrent import LSTM
-from keras.datasets.data_utils import get_file
+from keras.layers import Dense, Activation, Dropout
+from keras.layers import LSTM
+from keras.optimizers import RMSprop
+from keras.utils.data_utils import get_file
 import numpy as np
 import random
 import sys
@@ -23,13 +24,13 @@ path = get_file('nietzsche.txt', origin="https://s3.amazonaws.com/text-datasets/
 text = open(path).read().lower()
 print('corpus length:', len(text))
 
-chars = set(text)
+chars = sorted(list(set(text)))
 print('total chars:', len(chars))
 char_indices = dict((c, i) for i, c in enumerate(chars))
 indices_char = dict((i, c) for i, c in enumerate(chars))
 
 # cut the text in semi-redundant sequences of maxlen characters
-maxlen = 20
+maxlen = 40
 step = 3
 sentences = []
 next_chars = []
@@ -50,21 +51,22 @@ for i, sentence in enumerate(sentences):
 # build the model: 2 stacked LSTM
 print('Build model...')
 model = Sequential()
-model.add(LSTM(512, return_sequences=True, input_shape=(maxlen, len(chars))))
-model.add(Dropout(0.2))
-model.add(LSTM(512, return_sequences=False))
-model.add(Dropout(0.2))
+model.add(LSTM(128, input_shape=(maxlen, len(chars))))
 model.add(Dense(len(chars)))
 model.add(Activation('softmax'))
 
-model.compile(loss='categorical_crossentropy', optimizer='rmsprop')
+optimizer = RMSprop(lr=0.01)
+model.compile(loss='categorical_crossentropy', optimizer=optimizer)
 
 
-def sample(a, temperature=1.0):
+def sample(preds, temperature=1.0):
     # helper function to sample an index from a probability array
-    a = np.log(a) / temperature
-    a = np.exp(a) / np.sum(np.exp(a))
-    return np.argmax(np.random.multinomial(1, a, 1))
+    preds = np.asarray(preds).astype('float64')
+    preds = np.log(preds) / temperature
+    exp_preds = np.exp(preds)
+    preds = exp_preds / np.sum(exp_preds)
+    probas = np.random.multinomial(1, preds, 1)
+    return np.argmax(probas)
 
 # train the model, output generated text after each iteration
 for iteration in range(1, 60):

examples/mnist_cnn.py

@@ -1,8 +1,7 @@
-'''Train a simple convnet on the MNIST dataset.
+'''Trains a simple convnet on the MNIST dataset.
 
-Run on GPU: THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 python mnist_cnn.py
-
-Get to 99.25% test accuracy after 12 epochs (there is still a lot of margin for parameter tuning).
+Gets to 99.25% test accuracy after 12 epochs
+(there is still a lot of margin for parameter tuning).
 16 seconds per epoch on a GRID K520 GPU.
 '''
 
@@ -12,8 +11,8 @@ np.random.seed(1337)  # for reproducibility
 
 from keras.datasets import mnist
 from keras.models import Sequential
-from keras.layers.core import Dense, Dropout, Activation, Flatten
-from keras.layers.convolutional import Convolution2D, MaxPooling2D
+from keras.layers import Dense, Dropout, Activation, Flatten
+from keras.layers import Convolution2D, MaxPooling2D
 from keras.utils import np_utils
 
 batch_size = 128
@@ -29,7 +28,7 @@ nb_pool = 2
 # convolution kernel size
 nb_conv = 3
 
-# the data, shuffled and split between tran and test sets
+# the data, shuffled and split between train and test sets
 (X_train, y_train), (X_test, y_test) = mnist.load_data()
 
 X_train = X_train.reshape(X_train.shape[0], 1, img_rows, img_cols)
@@ -64,10 +63,12 @@ model.add(Dropout(0.5))
 model.add(Dense(nb_classes))
 model.add(Activation('softmax'))
 
-model.compile(loss='categorical_crossentropy', optimizer='adadelta')
+model.compile(loss='categorical_crossentropy',
+              optimizer='adadelta',
+              metrics=['accuracy'])
 
 model.fit(X_train, Y_train, batch_size=batch_size, nb_epoch=nb_epoch,
-          show_accuracy=True, verbose=1, validation_data=(X_test, Y_test))
-score = model.evaluate(X_test, Y_test, show_accuracy=True, verbose=0)
+          verbose=1, validation_data=(X_test, Y_test))
+score = model.evaluate(X_test, Y_test, verbose=0)
 print('Test score:', score[0])
 print('Test accuracy:', score[1])

examples/mnist_irnn.py

@@ -3,7 +3,7 @@ with pixel-by-pixel sequential MNIST in
 "A Simple Way to Initialize Recurrent Networks of Rectified Linear Units"
 by Quoc V. Le, Navdeep Jaitly, Geoffrey E. Hinton
 
-arXiv:1504.00941v2 [cs.NE] 7 Apr 201
+arXiv:1504.00941v2 [cs.NE] 7 Apr 2015
 http://arxiv.org/pdf/1504.00941v2.pdf
 
 Optimizer is replaced with RMSprop which yields more stable and steady
@@ -14,18 +14,15 @@ Reaches 0.93 train/test accuracy after 900 epochs
 '''
 
 from __future__ import print_function
-import numpy as np
-np.random.seed(1337)  # for reproducibility
 
 from keras.datasets import mnist
 from keras.models import Sequential
-from keras.layers.core import Dense, Activation
+from keras.layers import Dense, Activation
+from keras.layers import SimpleRNN
 from keras.initializations import normal, identity
-from keras.layers.recurrent import SimpleRNN, LSTM
 from keras.optimizers import RMSprop
 from keras.utils import np_utils
 
-
 batch_size = 32
 nb_classes = 10
 nb_epochs = 200
@@ -54,32 +51,20 @@ Y_test = np_utils.to_categorical(y_test, nb_classes)
 print('Evaluate IRNN...')
 model = Sequential()
 model.add(SimpleRNN(output_dim=hidden_units,
-                    init=lambda shape: normal(shape, scale=0.001),
-                    inner_init=lambda shape: identity(shape, scale=1.0),
-                    activation='relu', input_shape=X_train.shape[1:]))
+                    init=lambda shape, name: normal(shape, scale=0.001, name=name),
+                    inner_init=lambda shape, name: identity(shape, scale=1.0, name=name),
+                    activation='relu',
+                    input_shape=X_train.shape[1:]))
 model.add(Dense(nb_classes))
 model.add(Activation('softmax'))
 rmsprop = RMSprop(lr=learning_rate)
-model.compile(loss='categorical_crossentropy', optimizer=rmsprop)
+model.compile(loss='categorical_crossentropy',
+              optimizer=rmsprop,
+              metrics=['accuracy'])
 
 model.fit(X_train, Y_train, batch_size=batch_size, nb_epoch=nb_epochs,
-          show_accuracy=True, verbose=1, validation_data=(X_test, Y_test))
+          verbose=1, validation_data=(X_test, Y_test))
 
-scores = model.evaluate(X_test, Y_test, show_accuracy=True, verbose=0)
+scores = model.evaluate(X_test, Y_test, verbose=0)
 print('IRNN test score:', scores[0])
 print('IRNN test accuracy:', scores[1])
-
-print('Compare to LSTM...')
-model = Sequential()
-model.add(LSTM(hidden_units, input_shape=X_train.shape[1:]))
-model.add(Dense(nb_classes))
-model.add(Activation('softmax'))
-rmsprop = RMSprop(lr=learning_rate)
-model.compile(loss='categorical_crossentropy', optimizer=rmsprop)
-
-model.fit(X_train, Y_train, batch_size=batch_size, nb_epoch=nb_epochs,
-          show_accuracy=True, verbose=1, validation_data=(X_test, Y_test))
-
-scores = model.evaluate(X_test, Y_test, show_accuracy=True, verbose=0)
-print('LSTM test score:', scores[0])
-print('LSTM test accuracy:', scores[1])

examples/mnist_mlp.py

@@ -1,6 +1,6 @@
-'''Train a simple deep NN on the MNIST dataset.
+'''Trains a simple deep NN on the MNIST dataset.
 
-Get to 98.40% test accuracy after 20 epochs
+Gets to 98.40% test accuracy after 20 epochs
 (there is *a lot* of margin for parameter tuning).
 2 seconds per epoch on a K520 GPU.
 '''
@@ -20,7 +20,7 @@ batch_size = 128
 nb_classes = 10
 nb_epoch = 20
 
-# the data, shuffled and split between tran and test sets
+# the data, shuffled and split between train and test sets
 (X_train, y_train), (X_test, y_test) = mnist.load_data()
 
 X_train = X_train.reshape(60000, 784)
@@ -46,14 +46,15 @@ model.add(Dropout(0.2))
 model.add(Dense(10))
 model.add(Activation('softmax'))
 
-rms = RMSprop()
-model.compile(loss='categorical_crossentropy', optimizer=rms)
+model.summary()
 
-model.fit(X_train, Y_train,
-          batch_size=batch_size, nb_epoch=nb_epoch,
-          show_accuracy=True, verbose=2,
-          validation_data=(X_test, Y_test))
-score = model.evaluate(X_test, Y_test,
-                       show_accuracy=True, verbose=0)
+model.compile(loss='categorical_crossentropy',
+              optimizer=RMSprop(),
+              metrics=['accuracy'])
+
+history = model.fit(X_train, Y_train,
+                    batch_size=batch_size, nb_epoch=nb_epoch,
+                    verbose=1, validation_data=(X_test, Y_test))
+score = model.evaluate(X_test, Y_test, verbose=0)
 print('Test score:', score[0])
 print('Test accuracy:', score[1])

examples/mnist_siamese_graph.py

@@ -0,0 +1,130 @@
+'''Train a Siamese MLP on pairs of digits from the MNIST dataset.
+
+It follows Hadsell-et-al.'06 [1] by computing the Euclidean distance on the
+output of the shared network and by optimizing the contrastive loss (see paper
+for mode details).
+
+[1] "Dimensionality Reduction by Learning an Invariant Mapping"
+    http://yann.lecun.com/exdb/publis/pdf/hadsell-chopra-lecun-06.pdf
+
+Gets to 99.5% test accuracy after 20 epochs.
+3 seconds per epoch on a Titan X GPU
+'''
+from __future__ import absolute_import
+from __future__ import print_function
+import numpy as np
+np.random.seed(1337)  # for reproducibility
+
+import random
+from keras.datasets import mnist
+from keras.models import Sequential, Model
+from keras.layers import Dense, Dropout, Input, Lambda
+from keras.optimizers import SGD, RMSprop
+from keras import backend as K
+
+
+def euclidean_distance(vects):
+    x, y = vects
+    return K.sqrt(K.sum(K.square(x - y), axis=1, keepdims=True))
+
+
+def eucl_dist_output_shape(shapes):
+    shape1, shape2 = shapes
+    return (shape1[0], 1)
+
+
+def contrastive_loss(y_true, y_pred):
+    '''Contrastive loss from Hadsell-et-al.'06
+    http://yann.lecun.com/exdb/publis/pdf/hadsell-chopra-lecun-06.pdf
+    '''
+    margin = 1
+    return K.mean(y_true * K.square(y_pred) + (1 - y_true) * K.square(K.maximum(margin - y_pred, 0)))
+
+
+def create_pairs(x, digit_indices):
+    '''Positive and negative pair creation.
+    Alternates between positive and negative pairs.
+    '''
+    pairs = []
+    labels = []
+    n = min([len(digit_indices[d]) for d in range(10)]) - 1
+    for d in range(10):
+        for i in range(n):
+            z1, z2 = digit_indices[d][i], digit_indices[d][i+1]
+            pairs += [[x[z1], x[z2]]]
+            inc = random.randrange(1, 10)
+            dn = (d + inc) % 10
+            z1, z2 = digit_indices[d][i], digit_indices[dn][i]
+            pairs += [[x[z1], x[z2]]]
+            labels += [1, 0]
+    return np.array(pairs), np.array(labels)
+
+
+def create_base_network(input_dim):
+    '''Base network to be shared (eq. to feature extraction).
+    '''
+    seq = Sequential()
+    seq.add(Dense(128, input_shape=(input_dim,), activation='relu'))
+    seq.add(Dropout(0.1))
+    seq.add(Dense(128, activation='relu'))
+    seq.add(Dropout(0.1))
+    seq.add(Dense(128, activation='relu'))
+    return seq
+
+
+def compute_accuracy(predictions, labels):
+    '''Compute classification accuracy with a fixed threshold on distances.
+    '''
+    return labels[predictions.ravel() < 0.5].mean()
+
+
+# the data, shuffled and split between train and test sets
+(X_train, y_train), (X_test, y_test) = mnist.load_data()
+X_train = X_train.reshape(60000, 784)
+X_test = X_test.reshape(10000, 784)
+X_train = X_train.astype('float32')
+X_test = X_test.astype('float32')
+X_train /= 255
+X_test /= 255
+input_dim = 784
+nb_epoch = 20
+
+# create training+test positive and negative pairs
+digit_indices = [np.where(y_train == i)[0] for i in range(10)]
+tr_pairs, tr_y = create_pairs(X_train, digit_indices)
+
+digit_indices = [np.where(y_test == i)[0] for i in range(10)]
+te_pairs, te_y = create_pairs(X_test, digit_indices)
+
+# network definition
+base_network = create_base_network(input_dim)
+
+input_a = Input(shape=(input_dim,))
+input_b = Input(shape=(input_dim,))
+
+# because we re-use the same instance `base_network`,
+# the weights of the network
+# will be shared across the two branches
+processed_a = base_network(input_a)
+processed_b = base_network(input_b)
+
+distance = Lambda(euclidean_distance, output_shape=eucl_dist_output_shape)([processed_a, processed_b])
+
+model = Model(input=[input_a, input_b], output=distance)
+
+# train
+rms = RMSprop()
+model.compile(loss=contrastive_loss, optimizer=rms)
+model.fit([tr_pairs[:, 0], tr_pairs[:, 1]], tr_y,
+          validation_data=([te_pairs[:, 0], te_pairs[:, 1]], te_y),
+          batch_size=128,
+          nb_epoch=nb_epoch)
+
+# compute final accuracy on training and test sets
+pred = model.predict([tr_pairs[:, 0], tr_pairs[:, 1]])
+tr_acc = compute_accuracy(pred, tr_y)
+pred = model.predict([te_pairs[:, 0], te_pairs[:, 1]])
+te_acc = compute_accuracy(pred, te_y)
+
+print('* Accuracy on training set: %0.2f%%' % (100 * tr_acc))
+print('* Accuracy on test set: %0.2f%%' % (100 * te_acc))

examples/mnist_sklearn_wrapper.py

@@ -0,0 +1,94 @@
+'''Example of how to use sklearn wrapper
+
+Builds simple CNN models on MNIST and uses sklearn's GridSearchCV to find best model
+'''
+
+from __future__ import print_function
+import numpy as np
+np.random.seed(1337)  # for reproducibility
+
+from keras.datasets import mnist
+from keras.models import Sequential
+from keras.layers import Dense, Dropout, Activation, Flatten
+from keras.layers import Convolution2D, MaxPooling2D
+from keras.utils import np_utils
+from keras.wrappers.scikit_learn import KerasClassifier
+from sklearn.grid_search import GridSearchCV
+
+
+nb_classes = 10
+
+# input image dimensions
+img_rows, img_cols = 28, 28
+
+# load training data and do basic data normalization
+(X_train, y_train), (X_test, y_test) = mnist.load_data()
+X_train = X_train.reshape(X_train.shape[0], 1, img_rows, img_cols)
+X_test = X_test.reshape(X_test.shape[0], 1, img_rows, img_cols)
+X_train = X_train.astype('float32')
+X_test = X_test.astype('float32')
+X_train /= 255
+X_test /= 255
+
+# convert class vectors to binary class matrices
+y_train = np_utils.to_categorical(y_train, nb_classes)
+y_test = np_utils.to_categorical(y_test, nb_classes)
+
+def make_model(dense_layer_sizes, nb_filters, nb_conv, nb_pool):
+    '''Creates model comprised of 2 convolutional layers followed by dense layers
+
+    dense_layer_sizes: List of layer sizes. This list has one number for each layer
+    nb_filters: Number of convolutional filters in each convolutional layer
+    nb_conv: Convolutional kernel size
+    nb_pool: Size of pooling area for max pooling
+    '''
+
+    model = Sequential()
+
+    model.add(Convolution2D(nb_filters, nb_conv, nb_conv,
+                            border_mode='valid',
+                            input_shape=(1, img_rows, img_cols)))
+    model.add(Activation('relu'))
+    model.add(Convolution2D(nb_filters, nb_conv, nb_conv))
+    model.add(Activation('relu'))
+    model.add(MaxPooling2D(pool_size=(nb_pool, nb_pool)))
+    model.add(Dropout(0.25))
+
+    model.add(Flatten())
+    for layer_size in dense_layer_sizes:
+        model.add(Dense(layer_size))
+    model.add(Activation('relu'))
+    model.add(Dropout(0.5))
+    model.add(Dense(nb_classes))
+    model.add(Activation('softmax'))
+
+    model.compile(loss='categorical_crossentropy',
+                  optimizer='adadelta',
+                  metrics=['accuracy'])
+
+    return model
+
+dense_size_candidates = [[32], [64], [32, 32], [64, 64]]
+my_classifier = KerasClassifier(make_model, batch_size=32)
+validator = GridSearchCV(my_classifier,
+                         param_grid={'dense_layer_sizes': dense_size_candidates,
+                                     # nb_epoch is avail for tuning even when not
+                                     # an argument to model building function
+                                     'nb_epoch': [3, 6],
+                                     'nb_filters': [8],
+                                     'nb_conv': [3],
+                                     'nb_pool': [2]},
+                         scoring='log_loss',
+                         n_jobs=1)
+validator.fit(X_train, y_train)
+
+print('The parameters of the best model are: ')
+print(validator.best_params_)
+
+# validator.best_estimator_ returns sklearn-wrapped version of best model.
+# validator.best_estimator_.model returns the (unwrapped) keras model
+best_model = validator.best_estimator_.model
+metric_names = best_model.metrics_names
+metric_values = best_model.evaluate(X_test, y_test)
+for metric, value in zip(metric_names, metric_values):
+    print(metric, ': ', value)

examples/mnist_transfer_cnn.py

@@ -4,7 +4,7 @@
 2- Freeze convolutional layers and fine-tune dense layers
    for the classification of digits [5..9].
 
-Run on GPU: THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 python mnist_cnn.py
+Run on GPU: THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 python mnist_transfer_cnn.py
 
 Get to 99.8% test accuracy after 5 epochs
 for the first five digits classifier
@@ -19,8 +19,8 @@ np.random.seed(1337)  # for reproducibility
 
 from keras.datasets import mnist
 from keras.models import Sequential
-from keras.layers.core import Dense, Dropout, Activation, Flatten
-from keras.layers.convolutional import Convolution2D, MaxPooling2D
+from keras.layers import Dense, Dropout, Activation, Flatten
+from keras.layers import Convolution2D, MaxPooling2D
 from keras.utils import np_utils
 
 
@@ -55,15 +55,17 @@ def train_model(model, train, test, nb_classes):
     Y_train = np_utils.to_categorical(train[1], nb_classes)
     Y_test = np_utils.to_categorical(test[1], nb_classes)
 
-    model.compile(loss='categorical_crossentropy', optimizer='adadelta')
+    model.compile(loss='categorical_crossentropy',
+                  optimizer='adadelta',
+                  metrics=['accuracy'])
 
     t = now()
     model.fit(X_train, Y_train,
               batch_size=batch_size, nb_epoch=nb_epoch,
-              show_accuracy=True, verbose=1,
+              verbose=1,
               validation_data=(X_test, Y_test))
     print('Training time: %s' % (now() - t))
-    score = model.evaluate(X_test, Y_test, show_accuracy=True, verbose=0)
+    score = model.evaluate(X_test, Y_test, verbose=0)
     print('Test score:', score[0])
     print('Test accuracy:', score[1])
 

examples/neural_style_transfer.py

@@ -0,0 +1,302 @@
+'''Neural style transfer with Keras.
+
+Before running this script, download the weights for the VGG16 model at:
+https://drive.google.com/file/d/0Bz7KyqmuGsilT0J5dmRCM0ROVHc/view?usp=sharing
+(source: https://gist.github.com/baraldilorenzo/07d7802847aaad0a35d3)
+and make sure the variable `weights_path` in this script matches the location of the file.
+
+Run the script with:
+```
+python neural_style_transfer.py path_to_your_base_image.jpg path_to_your_reference.jpg prefix_for_results
+```
+e.g.:
+```
+python neural_style_transfer.py img/tuebingen.jpg img/starry_night.jpg results/my_result
+```
+
+It is preferable to run this script on GPU, for speed.
+If running on CPU, prefer the TensorFlow backend (much faster).
+
+Example result: https://twitter.com/fchollet/status/686631033085677568
+
+# Details
+
+Style transfer consists in generating an image
+with the same "content" as a base image, but with the
+"style" of a different picture (typically artistic).
+
+This is achieved through the optimization of a loss function
+that has 3 components: "style loss", "content loss",
+and "total variation loss":
+
+- The total variation loss imposes local spatial continuity between
+the pixels of the combination image, giving it visual coherence.
+
+- The style loss is where the deep learning keeps in --that one is defined
+using a deep convolutional neural network. Precisely, it consists in a sum of
+L2 distances between the Gram matrices of the representations of
+the base image and the style reference image, extracted from
+different layers of a convnet (trained on ImageNet). The general idea
+is to capture color/texture information at different spatial
+scales (fairly large scales --defined by the depth of the layer considered).
+
+ - The content loss is a L2 distance between the features of the base
+image (extracted from a deep layer) and the features of the combination image,
+keeping the generated image close enough to the original one.
+
+# References
+    - [A Neural Algorithm of Artistic Style](http://arxiv.org/abs/1508.06576)
+'''
+
+from __future__ import print_function
+from scipy.misc import imread, imresize, imsave
+import numpy as np
+from scipy.optimize import fmin_l_bfgs_b
+import time
+import os
+import argparse
+import h5py
+
+from keras.models import Sequential
+from keras.layers import Convolution2D, ZeroPadding2D, MaxPooling2D
+from keras import backend as K
+
+parser = argparse.ArgumentParser(description='Neural style transfer with Keras.')
+parser.add_argument('base_image_path', metavar='base', type=str,
+                    help='Path to the image to transform.')
+parser.add_argument('style_reference_image_path', metavar='ref', type=str,
+                    help='Path to the style reference image.')
+parser.add_argument('result_prefix', metavar='res_prefix', type=str,
+                    help='Prefix for the saved results.')
+
+args = parser.parse_args()
+base_image_path = args.base_image_path
+style_reference_image_path = args.style_reference_image_path
+result_prefix = args.result_prefix
+weights_path = 'vgg16_weights.h5'
+
+# these are the weights of the different loss components
+total_variation_weight = 1.
+style_weight = 1.
+content_weight = 0.025
+
+
+# dimensions of the generated picture.
+img_width = 400
+img_height = 400
+assert img_height == img_width, 'Due to the use of the Gram matrix, width and height must match.'
+
+# util function to open, resize and format pictures into appropriate tensors
+def preprocess_image(image_path):
+    img = imresize(imread(image_path), (img_width, img_height))
+    img = img[:, :, ::-1].astype('float64')
+    img[:, :, 0] -= 103.939
+    img[:, :, 1] -= 116.779
+    img[:, :, 2] -= 123.68
+    img = img.transpose((2, 0, 1))
+    img = np.expand_dims(img, axis=0)
+    return img
+
+# util function to convert a tensor into a valid image
+def deprocess_image(x):
+    x = x.transpose((1, 2, 0))
+    x[:, :, 0] += 103.939
+    x[:, :, 1] += 116.779
+    x[:, :, 2] += 123.68
+    x = x[:, :, ::-1]
+    x = np.clip(x, 0, 255).astype('uint8')
+    return x
+
+# get tensor representations of our images
+base_image = K.variable(preprocess_image(base_image_path))
+style_reference_image = K.variable(preprocess_image(style_reference_image_path))
+
+# this will contain our generated image
+combination_image = K.placeholder((1, 3, img_width, img_height))
+
+# combine the 3 images into a single Keras tensor
+input_tensor = K.concatenate([base_image,
+                              style_reference_image,
+                              combination_image], axis=0)
+
+# build the VGG16 network with our 3 images as input
+first_layer = ZeroPadding2D((1, 1))
+first_layer.set_input(input_tensor, shape=(3, 3, img_width, img_height))
+
+model = Sequential()
+model.add(first_layer)
+model.add(Convolution2D(64, 3, 3, activation='relu', name='conv1_1'))
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(64, 3, 3, activation='relu'))
+model.add(MaxPooling2D((2, 2), strides=(2, 2)))
+
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(128, 3, 3, activation='relu', name='conv2_1'))
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(128, 3, 3, activation='relu'))
+model.add(MaxPooling2D((2, 2), strides=(2, 2)))
+
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(256, 3, 3, activation='relu', name='conv3_1'))
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(256, 3, 3, activation='relu'))
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(256, 3, 3, activation='relu'))
+model.add(MaxPooling2D((2, 2), strides=(2, 2)))
+
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(512, 3, 3, activation='relu', name='conv4_1'))
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(512, 3, 3, activation='relu', name='conv4_2'))
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(512, 3, 3, activation='relu'))
+model.add(MaxPooling2D((2, 2), strides=(2, 2)))
+
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(512, 3, 3, activation='relu', name='conv5_1'))
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(512, 3, 3, activation='relu'))
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(512, 3, 3, activation='relu'))
+model.add(MaxPooling2D((2, 2), strides=(2, 2)))
+
+# load the weights of the VGG16 networks
+# (trained on ImageNet, won the ILSVRC competition in 2014)
+# note: when there is a complete match between your model definition
+# and your weight savefile, you can simply call model.load_weights(filename)
+assert os.path.exists(weights_path), 'Model weights not found (see "weights_path" variable in script).'
+f = h5py.File(weights_path)
+for k in range(f.attrs['nb_layers']):
+    if k >= len(model.layers):
+        # we don't look at the last (fully-connected) layers in the savefile
+        break
+    g = f['layer_{}'.format(k)]
+    weights = [g['param_{}'.format(p)] for p in range(g.attrs['nb_params'])]
+    model.layers[k].set_weights(weights)
+f.close()
+print('Model loaded.')
+
+# get the symbolic outputs of each "key" layer (we gave them unique names).
+outputs_dict = dict([(layer.name, layer.output) for layer in model.layers])
+
+# compute the neural style loss
+# first we need to define 4 util functions
+
+# the gram matrix of an image tensor (feature-wise outer product)
+def gram_matrix(x):
+    assert K.ndim(x) == 3
+    features = K.batch_flatten(x)
+    gram = K.dot(features, K.transpose(features))
+    return gram
+
+# the "style loss" is designed to maintain
+# the style of the reference image in the generated image.
+# It is based on the gram matrices (which capture style) of
+# feature maps from the style reference image
+# and from the generated image
+def style_loss(style, combination):
+    assert K.ndim(style) == 3
+    assert K.ndim(combination) == 3
+    S = gram_matrix(style)
+    C = gram_matrix(combination)
+    channels = 3
+    size = img_width * img_height
+    return K.sum(K.square(S - C)) / (4. * (channels ** 2) * (size ** 2))
+
+# an auxiliary loss function
+# designed to maintain the "content" of the
+# base image in the generated image
+def content_loss(base, combination):
+    return K.sum(K.square(combination - base))
+
+# the 3rd loss function, total variation loss,
+# designed to keep the generated image locally coherent
+def total_variation_loss(x):
+    assert K.ndim(x) == 4
+    a = K.square(x[:, :, :img_width-1, :img_height-1] - x[:, :, 1:, :img_height-1])
+    b = K.square(x[:, :, :img_width-1, :img_height-1] - x[:, :, :img_width-1, 1:])
+    return K.sum(K.pow(a + b, 1.25))
+
+# combine these loss functions into a single scalar
+loss = K.variable(0.)
+layer_features = outputs_dict['conv4_2']
+base_image_features = layer_features[0, :, :, :]
+combination_features = layer_features[2, :, :, :]
+loss += content_weight * content_loss(base_image_features,
+                                      combination_features)
+
+feature_layers = ['conv1_1', 'conv2_1', 'conv3_1', 'conv4_1', 'conv5_1']
+for layer_name in feature_layers:
+    layer_features = outputs_dict[layer_name]
+    style_reference_features = layer_features[1, :, :, :]
+    combination_features = layer_features[2, :, :, :]
+    sl = style_loss(style_reference_features, combination_features)
+    loss += (style_weight / len(feature_layers)) * sl
+loss += total_variation_weight * total_variation_loss(combination_image)
+
+# get the gradients of the generated image wrt the loss
+grads = K.gradients(loss, combination_image)
+
+outputs = [loss]
+if type(grads) in {list, tuple}:
+    outputs += grads
+else:
+    outputs.append(grads)
+
+f_outputs = K.function([combination_image], outputs)
+def eval_loss_and_grads(x):
+    x = x.reshape((1, 3, img_width, img_height))
+    outs = f_outputs([x])
+    loss_value = outs[0]
+    if len(outs[1:]) == 1:
+        grad_values = outs[1].flatten().astype('float64')
+    else:
+        grad_values = np.array(outs[1:]).flatten().astype('float64')
+    return loss_value, grad_values
+
+# this Evaluator class makes it possible
+# to compute loss and gradients in one pass
+# while retrieving them via two separate functions,
+# "loss" and "grads". This is done because scipy.optimize
+# requires separate functions for loss and gradients,
+# but computing them separately would be inefficient.
+class Evaluator(object):
+    def __init__(self):
+        self.loss_value = None
+        self.grads_values = None
+
+    def loss(self, x):
+        assert self.loss_value is None
+        loss_value, grad_values = eval_loss_and_grads(x)
+        self.loss_value = loss_value
+        self.grad_values = grad_values
+        return self.loss_value
+
+    def grads(self, x):
+        assert self.loss_value is not None
+        grad_values = np.copy(self.grad_values)
+        self.loss_value = None
+        self.grad_values = None
+        return grad_values
+
+evaluator = Evaluator()
+
+# run scipy-based optimization (L-BFGS) over the pixels of the generated image
+# so as to minimize the neural style loss
+x = np.random.uniform(0, 255, (1, 3, img_width, img_height))
+x[0, 0, :, :] -= 103.939
+x[0, 1, :, :] -= 116.779
+x[0, 2, :, :] -= 123.68
+for i in range(10):
+    print('Start of iteration', i)
+    start_time = time.time()
+    x, min_val, info = fmin_l_bfgs_b(evaluator.loss, x.flatten(),
+                                     fprime=evaluator.grads, maxfun=20)
+    print('Current loss value:', min_val)
+    # save current generated image
+    img = deprocess_image(x.copy().reshape((3, img_width, img_height)))
+    fname = result_prefix + '_at_iteration_%d.png' % i
+    imsave(fname, img)
+    end_time = time.time()
+    print('Image saved as', fname)
+    print('Iteration %d completed in %ds' % (i, end_time - start_time))

examples/pretrained_word_embeddings.py

@@ -0,0 +1,144 @@
+'''This script loads pre-trained word embeddings (GloVe embeddings)
+into a frozen Keras Embedding layer, and uses it to
+train a text classification model on the 20 Newsgroup dataset
+(classication of newsgroup messages into 20 different categories).
+
+GloVe embedding data can be found at:
+http://nlp.stanford.edu/data/glove.6B.zip
+(source page: http://nlp.stanford.edu/projects/glove/)
+
+20 Newsgroup data can be found at:
+http://www.cs.cmu.edu/afs/cs.cmu.edu/project/theo-20/www/data/news20.html
+'''
+
+from __future__ import print_function
+import os
+import numpy as np
+np.random.seed(1337)
+
+from keras.preprocessing.text import Tokenizer
+from keras.preprocessing.sequence import pad_sequences
+from keras.utils.np_utils import to_categorical
+from keras.layers import Dense, Input, Flatten
+from keras.layers import Conv1D, MaxPooling1D, Embedding
+from keras.models import Model
+import sys
+
+BASE_DIR = ''
+GLOVE_DIR = BASE_DIR + '/glove.6B/'
+TEXT_DATA_DIR = BASE_DIR + '/20_newsgroup/'
+MAX_SEQUENCE_LENGTH = 1000
+MAX_NB_WORDS = 20000
+EMBEDDING_DIM = 100
+VALIDATION_SPLIT = 0.2
+
+# first, build index mapping words in the embeddings set
+# to their embedding vector
+
+print('Indexing word vectors.')
+
+embeddings_index = {}
+f = open(os.path.join(GLOVE_DIR, 'glove.6B.100d.txt'))
+for line in f:
+    values = line.split()
+    word = values[0]
+    coefs = np.asarray(values[1:], dtype='float32')
+    embeddings_index[word] = coefs
+f.close()
+
+print('Found %s word vectors.' % len(embeddings_index))
+
+# second, prepare text samples and their labels
+print('Processing text dataset')
+
+texts = []  # list of text samples
+labels_index = {}  # dictionary mapping label name to numeric id
+labels = []  # list of label ids
+for name in sorted(os.listdir(TEXT_DATA_DIR)):
+    path = os.path.join(TEXT_DATA_DIR, name)
+    if os.path.isdir(path):
+        label_id = len(labels_index)
+        labels_index[name] = label_id
+        for fname in sorted(os.listdir(path)):
+            if fname.isdigit():
+                fpath = os.path.join(path, fname)
+                if sys.version_info < (3,):
+                    f = open(fpath)
+                else:
+                    f = open(fpath, encoding='latin-1')
+                texts.append(f.read())
+                f.close()
+                labels.append(label_id)
+
+print('Found %s texts.' % len(texts))
+
+# finally, vectorize the text samples into a 2D integer tensor
+tokenizer = Tokenizer(nb_words=MAX_NB_WORDS)
+tokenizer.fit_on_texts(texts)
+sequences = tokenizer.texts_to_sequences(texts)
+
+word_index = tokenizer.word_index
+print('Found %s unique tokens.' % len(word_index))
+
+data = pad_sequences(sequences, maxlen=MAX_SEQUENCE_LENGTH)
+
+labels = to_categorical(np.asarray(labels))
+print('Shape of data tensor:', data.shape)
+print('Shape of label tensor:', labels.shape)
+
+# split the data into a training set and a validation set
+indices = np.arange(data.shape[0])
+np.random.shuffle(indices)
+data = data[indices]
+labels = labels[indices]
+nb_validation_samples = int(VALIDATION_SPLIT * data.shape[0])
+
+x_train = data[:-nb_validation_samples]
+y_train = labels[:-nb_validation_samples]
+x_val = data[-nb_validation_samples:]
+y_val = labels[-nb_validation_samples:]
+
+print('Preparing embedding matrix.')
+
+# prepare embedding matrix
+nb_words = min(MAX_NB_WORDS, len(word_index))
+embedding_matrix = np.zeros((nb_words + 1, EMBEDDING_DIM))
+for word, i in word_index.items():
+    if i > MAX_NB_WORDS:
+        continue
+    embedding_vector = embeddings_index.get(word)
+    if embedding_vector is not None:
+        # words not found in embedding index will be all-zeros.
+        embedding_matrix[i] = embedding_vector
+
+# load pre-trained word embeddings into an Embedding layer
+# note that we set trainable = False so as to keep the embeddings fixed
+embedding_layer = Embedding(nb_words + 1,
+                            EMBEDDING_DIM,
+                            weights=[embedding_matrix],
+                            input_length=MAX_SEQUENCE_LENGTH,
+                            trainable=False)
+
+print('Training model.')
+
+# train a 1D convnet with global maxpooling
+sequence_input = Input(shape=(MAX_SEQUENCE_LENGTH,), dtype='int32')
+embedded_sequences = embedding_layer(sequence_input)
+x = Conv1D(128, 5, activation='relu')(embedded_sequences)
+x = MaxPooling1D(5)(x)
+x = Conv1D(128, 5, activation='relu')(x)
+x = MaxPooling1D(5)(x)
+x = Conv1D(128, 5, activation='relu')(x)
+x = MaxPooling1D(35)(x)
+x = Flatten()(x)
+x = Dense(128, activation='relu')(x)
+preds = Dense(len(labels_index), activation='softmax')(x)
+
+model = Model(sequence_input, preds)
+model.compile(loss='categorical_crossentropy',
+              optimizer='rmsprop',
+              metrics=['acc'])
+
+# happy learning!
+model.fit(x_train, y_train, validation_data=(x_val, y_val),
+          nb_epoch=2, batch_size=128)

examples/resnet_50.py

@@ -0,0 +1,220 @@
+'''This script demonstrates how to build a deep residual network
+using the Keras functional API.
+
+get_resnet50() returns the deep residual network model (50 layers)
+
+Please visit Kaiming He's GitHub homepage:
+https://github.com/KaimingHe
+for more information.
+
+The related paper is
+'Deep Residual Learning for Image Recognition'
+Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun
+http://arxiv.org/abs/1512.03385
+
+Pretrained weights were converted from Kaiming He's caffe model directly.
+
+For now we provide weights for the tensorflow backend only,
+thus use 'tf' dim_ordering (e.g. input_shape=(224, 224, 3) for 224*224 color image)
+would accelerate the computation, but we also provide weights for 'th' dim_ordering for compatibility.
+You can set your default dim ordering in your Keras config file at ~/.keras/keras.json
+
+please donwload them at:
+http://pan.baidu.com/s/1o8pO2q2 ('th' dim ordering, for China)
+http://pan.baidu.com/s/1pLanuTt ('tf' dim ordering, for China)
+
+https://drive.google.com/open?id=0B4ChsjFJvew3NVQ2U041Q0xHRHM ('th' dim ordering, for other countries)
+https://drive.google.com/open?id=0B4ChsjFJvew3NWN5THdxcTdSWmc ('tf' dim ordering, for other countries)
+
+@author: BigMoyan, University of Electronic Science and Technology of China
+'''
+from __future__ import print_function
+from keras.layers import merge
+from keras.layers.convolutional import Convolution2D, MaxPooling2D, ZeroPadding2D, AveragePooling2D
+from keras.layers.core import Dense, Activation, Flatten
+from keras.layers.normalization import BatchNormalization
+from keras.models import Model
+from keras.layers import Input
+from keras.preprocessing.image import load_img, img_to_array
+import keras.backend as K
+import numpy as np
+
+# The names of layers in resnet50 are generated with the following format
+# [type][stage][block]_branch[branch][layer]
+# type: 'res' for conv layer, 'bn' and 'scale' for BN layer
+# stage: from '2' to '5', current stage number
+# block: 'a','b','c'... for different blocks in a stage
+# branch: '1' for shortcut and '2' for main path
+# layer: 'a','b','c'... for different layers in a block
+
+
+def identity_block(input_tensor, kernel_size, filters, stage, block):
+    '''The identity_block is the block that has no conv layer at shortcut
+
+    # Arguments
+        input_tensor: input tensor
+        kernel_size: defualt 3, the kernel size of middle conv layer at main path
+        filters: list of integers, the nb_filters of 3 conv layer at main path
+        stage: integer, current stage label, used for generating layer names
+        block: 'a','b'..., current block label, used for generating layer names
+    '''
+    dim_ordering = K.image_dim_ordering()
+    nb_filter1, nb_filter2, nb_filter3 = filters
+    if dim_ordering == 'tf':
+        bn_axis = 3
+    else:
+        bn_axis = 1
+    conv_name_base = 'res' + str(stage) + block + '_branch'
+    bn_name_base = 'bn' + str(stage) + block + '_branch'
+
+    out = Convolution2D(nb_filter1, 1, 1, dim_ordering=dim_ordering, name=conv_name_base + '2a')(input_tensor)
+    out = BatchNormalization(axis=bn_axis, name=bn_name_base + '2a')(out)
+    out = Activation('relu')(out)
+
+    out = Convolution2D(nb_filter2, kernel_size, kernel_size, border_mode='same',
+                        dim_ordering=dim_ordering, name=conv_name_base + '2b')(out)
+    out = BatchNormalization(axis=bn_axis, name=bn_name_base + '2b')(out)
+    out = Activation('relu')(out)
+
+    out = Convolution2D(nb_filter3, 1, 1, dim_ordering=dim_ordering, name=conv_name_base + '2c')(out)
+    out = BatchNormalization(axis=bn_axis, name=bn_name_base + '2c')(out)
+
+    out = merge([out, input_tensor], mode='sum')
+    out = Activation('relu')(out)
+    return out
+
+
+def conv_block(input_tensor, kernel_size, filters, stage, block, strides=(2, 2)):
+    '''conv_block is the block that has a conv layer at shortcut
+
+    # Arguments
+        input_tensor: input tensor
+        kernel_size: defualt 3, the kernel size of middle conv layer at main path
+        filters: list of integers, the nb_filters of 3 conv layer at main path
+        stage: integer, current stage label, used for generating layer names
+        block: 'a','b'..., current block label, used for generating layer names
+
+    Note that from stage 3, the first conv layer at main path is with subsample=(2,2)
+    And the shortcut should has subsample=(2,2) as well
+    '''
+    nb_filter1, nb_filter2, nb_filter3 = filters
+    dim_ordering = K.image_dim_ordering()
+    if dim_ordering == 'tf':
+        bn_axis = 3
+    else:
+        bn_axis = 1
+    conv_name_base = 'res' + str(stage) + block + '_branch'
+    bn_name_base = 'bn' + str(stage) + block + '_branch'
+
+    out = Convolution2D(nb_filter1, 1, 1, subsample=strides,
+                        dim_ordering=dim_ordering, name=conv_name_base + '2a')(input_tensor)
+    out = BatchNormalization(axis=bn_axis, name=bn_name_base + '2a')(out)
+    out = Activation('relu')(out)
+
+    out = Convolution2D(nb_filter2, kernel_size, kernel_size, border_mode='same',
+                        dim_ordering=dim_ordering, name=conv_name_base + '2b')(out)
+    out = BatchNormalization(axis=bn_axis, name=bn_name_base + '2b')(out)
+    out = Activation('relu')(out)
+
+    out = Convolution2D(nb_filter3, 1, 1, dim_ordering=dim_ordering, name=conv_name_base + '2c')(out)
+    out = BatchNormalization(axis=bn_axis, name=bn_name_base + '2c')(out)
+
+    shortcut = Convolution2D(nb_filter3, 1, 1, subsample=strides,
+                             dim_ordering=dim_ordering, name=conv_name_base + '1')(input_tensor)
+    shortcut = BatchNormalization(axis=bn_axis, name=bn_name_base + '1')(shortcut)
+
+    out = merge([out, shortcut], mode='sum')
+    out = Activation('relu')(out)
+    return out
+
+
+def read_img(img_path):
+    '''This function returns a preprocessed image
+    '''
+    dim_ordering = K.image_dim_ordering()
+    mean = (103.939, 116.779, 123.68)
+    img = load_img(img_path, target_size=(224, 224))
+    img = img_to_array(img, dim_ordering=dim_ordering)
+
+    if dim_ordering == 'th':
+        img[0, :, :] -= mean[0]
+        img[1, :, :] -= mean[1]
+        img[2, :, :] -= mean[2]
+        # 'RGB'->'BGR'
+        img = img[::-1, :, :]
+    else:
+        img[:, :, 0] -= mean[0]
+        img[:, :, 1] -= mean[1]
+        img[:, :, 2] -= mean[2]
+        img = img[:, :, ::-1]
+
+    img = np.expand_dims(img, axis=0)
+    return img
+
+
+def get_resnet50():
+    '''This function returns the 50-layer residual network model
+    you should load pretrained weights if you want to use it directly.
+    Note that since the pretrained weights is converted from caffemodel
+    the order of channels for input image should be 'BGR' (the channel order of caffe)
+    '''
+    if K.image_dim_ordering() == 'tf':
+        inp = Input(shape=(224, 224, 3))
+        bn_axis = 3
+    else:
+        inp = Input(shape=(3, 224, 224))
+        bn_axis = 1
+
+    dim_ordering = K.image_dim_ordering()
+    out = ZeroPadding2D((3, 3), dim_ordering=dim_ordering)(inp)
+    out = Convolution2D(64, 7, 7, subsample=(2, 2), dim_ordering=dim_ordering, name='conv1')(out)
+    out = BatchNormalization(axis=bn_axis, name='bn_conv1')(out)
+    out = Activation('relu')(out)
+    out = MaxPooling2D((3, 3), strides=(2, 2), dim_ordering=dim_ordering)(out)
+
+    out = conv_block(out, 3, [64, 64, 256], stage=2, block='a', strides=(1, 1))
+    out = identity_block(out, 3, [64, 64, 256], stage=2, block='b')
+    out = identity_block(out, 3, [64, 64, 256], stage=2, block='c')
+
+    out = conv_block(out, 3, [128, 128, 512], stage=3, block='a')
+    out = identity_block(out, 3, [128, 128, 512], stage=3, block='b')
+    out = identity_block(out, 3, [128, 128, 512], stage=3, block='c')
+    out = identity_block(out, 3, [128, 128, 512], stage=3, block='d')
+
+    out = conv_block(out, 3, [256, 256, 1024], stage=4, block='a')
+    out = identity_block(out, 3, [256, 256, 1024], stage=4, block='b')
+    out = identity_block(out, 3, [256, 256, 1024], stage=4, block='c')
+    out = identity_block(out, 3, [256, 256, 1024], stage=4, block='d')
+    out = identity_block(out, 3, [256, 256, 1024], stage=4, block='e')
+    out = identity_block(out, 3, [256, 256, 1024], stage=4, block='f')
+
+    out = conv_block(out, 3, [512, 512, 2048], stage=5, block='a')
+    out = identity_block(out, 3, [512, 512, 2048], stage=5, block='b')
+    out = identity_block(out, 3, [512, 512, 2048], stage=5, block='c')
+
+    out = AveragePooling2D((7, 7), dim_ordering=dim_ordering)(out)
+    out = Flatten()(out)
+    out = Dense(1000, activation='softmax', name='fc1000')(out)
+
+    model = Model(inp, out)
+
+    return model
+
+
+if __name__ == '__main__':
+    weights_file = K.image_dim_ordering() + '_dim_ordering_resnet50.h5'
+    resnet_model = get_resnet50()
+    resnet_model.load_weights(weights_file)
+
+    # you may download synset_words from the address given at the begining of this file
+    class_table = open('synset_words.txt', 'r')
+    lines = class_table.readlines()
+
+    test_img1 = read_img('cat.jpg')
+    print('Result for test 1 is:')
+    print(lines[np.argmax(resnet_model.predict(test_img1)[0])])
+
+    test_img2 = read_img('elephant.jpg')
+    print('Result for test 2 is:')
+    print(lines[np.argmax(resnet_model.predict(test_img2)[0])])
+    class_table.close()

examples/reuters_mlp.py

@@ -1,8 +1,5 @@
-'''Train and evaluate a simple MLP on the Reuters newswire topic classification task.
-GPU run command:
-    THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 python examples/reuters_mlp.py
-CPU run command:
-    python examples/reuters_mlp.py
+'''Trains and evaluate a simple MLP
+on the Reuters newswire topic classification task.
 '''
 
 from __future__ import print_function
@@ -11,8 +8,7 @@ np.random.seed(1337)  # for reproducibility
 
 from keras.datasets import reuters
 from keras.models import Sequential
-from keras.layers.core import Dense, Dropout, Activation
-from keras.layers.normalization import BatchNormalization
+from keras.layers import Dense, Dropout, Activation
 from keras.utils import np_utils
 from keras.preprocessing.text import Tokenizer
 
@@ -49,9 +45,14 @@ model.add(Dropout(0.5))
 model.add(Dense(nb_classes))
 model.add(Activation('softmax'))
 
-model.compile(loss='categorical_crossentropy', optimizer='adam')
+model.compile(loss='categorical_crossentropy',
+              optimizer='adam',
+              metrics=['accuracy'])
 
-history = model.fit(X_train, Y_train, nb_epoch=nb_epoch, batch_size=batch_size, verbose=1, show_accuracy=True, validation_split=0.1)
-score = model.evaluate(X_test, Y_test, batch_size=batch_size, verbose=1, show_accuracy=True)
+history = model.fit(X_train, Y_train,
+                    nb_epoch=nb_epoch, batch_size=batch_size,
+                    verbose=1, validation_split=0.1)
+score = model.evaluate(X_test, Y_test,
+                       batch_size=batch_size, verbose=1)
 print('Test score:', score[0])
 print('Test accuracy:', score[1])

examples/stateful_lstm.py

@@ -0,0 +1,84 @@
+'''Example script showing how to use stateful RNNs
+to model long sequences efficiently.
+'''
+from __future__ import print_function
+import numpy as np
+import matplotlib.pyplot as plt
+from keras.models import Sequential
+from keras.layers import Dense, LSTM
+
+
+# since we are using stateful rnn tsteps can be set to 1
+tsteps = 1
+batch_size = 25
+epochs = 25
+# number of elements ahead that are used to make the prediction
+lahead = 1
+
+
+def gen_cosine_amp(amp=100, period=1000, x0=0, xn=50000, step=1, k=0.0001):
+    """Generates an absolute cosine time series with the amplitude
+    exponentially decreasing
+
+    Arguments:
+        amp: amplitude of the cosine function
+        period: period of the cosine function
+        x0: initial x of the time series
+        xn: final x of the time series
+        step: step of the time series discretization
+        k: exponential rate
+    """
+    cos = np.zeros(((xn - x0) * step, 1, 1))
+    for i in range(len(cos)):
+        idx = x0 + i * step
+        cos[i, 0, 0] = amp * np.cos(2 * np.pi * idx / period)
+        cos[i, 0, 0] = cos[i, 0, 0] * np.exp(-k * idx)
+    return cos
+
+
+print('Generating Data')
+cos = gen_cosine_amp()
+print('Input shape:', cos.shape)
+
+expected_output = np.zeros((len(cos), 1))
+for i in range(len(cos) - lahead):
+    expected_output[i, 0] = np.mean(cos[i + 1:i + lahead + 1])
+
+print('Output shape')
+print(expected_output.shape)
+
+print('Creating Model')
+model = Sequential()
+model.add(LSTM(50,
+               batch_input_shape=(batch_size, tsteps, 1),
+               return_sequences=True,
+               stateful=True))
+model.add(LSTM(50,
+               batch_input_shape=(batch_size, tsteps, 1),
+               return_sequences=False,
+               stateful=True))
+model.add(Dense(1))
+model.compile(loss='mse', optimizer='rmsprop')
+
+print('Training')
+for i in range(epochs):
+    print('Epoch', i, '/', epochs)
+    model.fit(cos,
+              expected_output,
+              batch_size=batch_size,
+              verbose=1,
+              nb_epoch=1,
+              shuffle=False)
+    model.reset_states()
+
+print('Predicting')
+predicted_output = model.predict(cos, batch_size=batch_size)
+
+print('Plotting Results')
+plt.subplot(2, 1, 1)
+plt.plot(expected_output)
+plt.title('Expected')
+plt.subplot(2, 1, 2)
+plt.plot(predicted_output)
+plt.title('Predicted')
+plt.show()

examples/variational_autoencoder.py

@@ -0,0 +1,97 @@
+'''This script demonstrates how to build a variational autoencoder with Keras.
+
+Reference: "Auto-Encoding Variational Bayes" https://arxiv.org/abs/1312.6114
+'''
+import numpy as np
+import matplotlib.pyplot as plt
+
+from keras.layers import Input, Dense, Lambda
+from keras.models import Model
+from keras import backend as K
+from keras import objectives
+from keras.datasets import mnist
+
+batch_size = 100
+original_dim = 784
+latent_dim = 2
+intermediate_dim = 256
+nb_epoch = 50
+
+x = Input(batch_shape=(batch_size, original_dim))
+h = Dense(intermediate_dim, activation='relu')(x)
+z_mean = Dense(latent_dim)(h)
+z_log_var = Dense(latent_dim)(h)
+
+
+def sampling(args):
+    z_mean, z_log_var = args
+    epsilon = K.random_normal(shape=(batch_size, latent_dim), mean=0.)
+    return z_mean + K.exp(z_log_var / 2) * epsilon
+
+# note that "output_shape" isn't necessary with the TensorFlow backend
+z = Lambda(sampling, output_shape=(latent_dim,))([z_mean, z_log_var])
+
+# we instantiate these layers separately so as to reuse them later
+decoder_h = Dense(intermediate_dim, activation='relu')
+decoder_mean = Dense(original_dim, activation='sigmoid')
+h_decoded = decoder_h(z)
+x_decoded_mean = decoder_mean(h_decoded)
+
+
+def vae_loss(x, x_decoded_mean):
+    xent_loss = original_dim * objectives.binary_crossentropy(x, x_decoded_mean)
+    kl_loss = - 0.5 * K.sum(1 + z_log_var - K.square(z_mean) - K.exp(z_log_var), axis=-1)
+    return xent_loss + kl_loss
+
+vae = Model(x, x_decoded_mean)
+vae.compile(optimizer='rmsprop', loss=vae_loss)
+
+# train the VAE on MNIST digits
+(x_train, y_train), (x_test, y_test) = mnist.load_data()
+
+x_train = x_train.astype('float32') / 255.
+x_test = x_test.astype('float32') / 255.
+x_train = x_train.reshape((len(x_train), np.prod(x_train.shape[1:])))
+x_test = x_test.reshape((len(x_test), np.prod(x_test.shape[1:])))
+
+vae.fit(x_train, x_train,
+        shuffle=True,
+        nb_epoch=nb_epoch,
+        batch_size=batch_size,
+        validation_data=(x_test, x_test))
+
+# build a model to project inputs on the latent space
+encoder = Model(x, z_mean)
+
+# display a 2D plot of the digit classes in the latent space
+x_test_encoded = encoder.predict(x_test, batch_size=batch_size)
+plt.figure(figsize=(6, 6))
+plt.scatter(x_test_encoded[:, 0], x_test_encoded[:, 1], c=y_test)
+plt.colorbar()
+plt.show()
+
+# build a digit generator that can sample from the learned distribution
+decoder_input = Input(shape=(latent_dim,))
+_h_decoded = decoder_h(decoder_input)
+_x_decoded_mean = decoder_mean(_h_decoded)
+generator = Model(decoder_input, _x_decoded_mean)
+
+# display a 2D manifold of the digits
+n = 15  # figure with 15x15 digits
+digit_size = 28
+figure = np.zeros((digit_size * n, digit_size * n))
+# we will sample n points within [-15, 15] standard deviations
+grid_x = np.linspace(-15, 15, n)
+grid_y = np.linspace(-15, 15, n)
+
+for i, yi in enumerate(grid_x):
+    for j, xi in enumerate(grid_y):
+        z_sample = np.array([[xi, yi]])
+        x_decoded = generator.predict(z_sample)
+        digit = x_decoded[0].reshape(digit_size, digit_size)
+        figure[i * digit_size: (i + 1) * digit_size,
+               j * digit_size: (j + 1) * digit_size] = digit
+
+plt.figure(figsize=(10, 10))
+plt.imshow(figure)
+plt.show()

examples/variational_autoencoder_deconv.py

@@ -0,0 +1,124 @@
+'''This script demonstrates how to build a variational autoencoder with Keras and deconvolution layers.
+
+Reference: "Auto-Encoding Variational Bayes" https://arxiv.org/abs/1312.6114
+'''
+import numpy as np
+import matplotlib.pyplot as plt
+
+from keras.layers import Input, Dense, Lambda, Flatten, Reshape
+from keras.layers import Convolution2D, Deconvolution2D, MaxPooling2D
+from keras.models import Model
+from keras import backend as K
+from keras import objectives
+from keras.datasets import mnist
+
+# input image dimensions
+img_rows, img_cols, img_chns = 28, 28, 1
+# number of convolutional filters to use
+nb_filters = 32
+# convolution kernel size
+nb_conv = 3
+
+batch_size = 16
+original_dim = (img_chns, img_rows, img_cols)
+latent_dim = 2
+intermediate_dim = 128
+epsilon_std = 0.01
+nb_epoch = 5
+
+
+x = Input(batch_shape=(batch_size,) + original_dim)
+c = Convolution2D(nb_filters, nb_conv, nb_conv, border_mode='same', activation='relu')(x)
+f = Flatten()(c)
+h = Dense(intermediate_dim, activation='relu')(f)
+
+z_mean = Dense(latent_dim)(h)
+z_log_var = Dense(latent_dim)(h)
+
+
+def sampling(args):
+    z_mean, z_log_var = args
+    epsilon = K.random_normal(shape=(batch_size, latent_dim),
+                              mean=0., std=epsilon_std)
+    return z_mean + K.exp(z_log_var) * epsilon
+
+# note that "output_shape" isn't necessary with the TensorFlow backend
+# so you could write `Lambda(sampling)([z_mean, z_log_var])`
+z = Lambda(sampling, output_shape=(latent_dim,))([z_mean, z_log_var])
+
+# we instantiate these layers separately so as to reuse them later
+decoder_h = Dense(intermediate_dim, activation='relu')
+decoder_f = Dense(nb_filters*img_rows*img_cols, activation='relu')
+decoder_c = Reshape((nb_filters, img_rows, img_cols))
+decoder_mean = Deconvolution2D(img_chns, nb_conv, nb_conv,
+                               (batch_size, img_chns, img_rows, img_cols),
+                               border_mode='same')
+
+h_decoded = decoder_h(z)
+f_decoded = decoder_f(h_decoded)
+c_decoded = decoder_c(f_decoded)
+x_decoded_mean = decoder_mean(c_decoded)
+
+
+def vae_loss(x, x_decoded_mean):
+    # NOTE: binary_crossentropy expects a batch_size by dim for x and x_decoded_mean, so we MUST flatten these!
+    x = K.flatten(x)
+    x_decoded_mean = K.flatten(x_decoded_mean)
+    xent_loss = objectives.binary_crossentropy(x, x_decoded_mean)
+    kl_loss = - 0.5 * K.mean(1 + z_log_var - K.square(z_mean) - K.exp(z_log_var), axis=-1)
+    return xent_loss + kl_loss
+
+vae = Model(x, x_decoded_mean)
+vae.compile(optimizer='rmsprop', loss=vae_loss)
+vae.summary()
+
+# train the VAE on MNIST digits
+(x_train, y_train), (x_test, y_test) = mnist.load_data()
+
+x_train = x_train.astype('float32')[:, None, :, :] / 255.
+x_test = x_test.astype('float32')[:, None, :, :] / 255.
+
+vae.fit(x_train, x_train,
+        shuffle=True,
+        nb_epoch=nb_epoch,
+        batch_size=batch_size,
+        validation_data=(x_test, x_test))
+
+
+# build a model to project inputs on the latent space
+encoder = Model(x, z_mean)
+
+# display a 2D plot of the digit classes in the latent space
+x_test_encoded = encoder.predict(x_test, batch_size=batch_size)
+plt.figure(figsize=(6, 6))
+plt.scatter(x_test_encoded[:, 0], x_test_encoded[:, 1], c=y_test)
+plt.colorbar()
+plt.show()
+
+# build a digit generator that can sample from the learned distribution
+decoder_input = Input(shape=(latent_dim,))
+_h_decoded = decoder_h(decoder_input)
+_f_decoded = decoder_f(_h_decoded)
+_c_decoded = decoder_c(_f_decoded)
+_x_decoded_mean = decoder_mean(_c_decoded)
+generator = Model(decoder_input, _x_decoded_mean)
+
+# display a 2D manifold of the digits
+n = 15  # figure with 15x15 digits
+digit_size = 28
+figure = np.zeros((digit_size * n, digit_size * n))
+# we will sample n points within [-15, 15] standard deviations
+grid_x = np.linspace(-15, 15, n)
+grid_y = np.linspace(-15, 15, n)
+
+for i, yi in enumerate(grid_x):
+    for j, xi in enumerate(grid_y):
+        z_sample = np.array([[xi, yi]])
+        x_decoded = generator.predict(z_sample)
+        digit = x_decoded[0].reshape(digit_size, digit_size)
+        figure[i * digit_size: (i + 1) * digit_size,
+               j * digit_size: (j + 1) * digit_size] = digit
+
+plt.figure(figsize=(10, 10))
+plt.imshow(figure)
+plt.show()

keras/__init__.py

@@ -1 +1,18 @@
-__version__ = '0.3.0'
+from __future__ import absolute_import
+from . import backend
+from . import datasets
+from . import engine
+from . import layers
+from . import preprocessing
+from . import utils
+from . import wrappers
+from . import callbacks
+from . import constraints
+from . import initializations
+from . import metrics
+from . import models
+from . import objectives
+from . import optimizers
+from . import regularizers
+
+__version__ = '1.0.7'

keras/activations.py

@@ -7,11 +7,9 @@ def softmax(x):
     if ndim == 2:
         return K.softmax(x)
     elif ndim == 3:
-        # apply softmax to each timestep
-        def step(x, states):
-            return K.softmax(x), []
-        last_output, outputs, states = K.rnn(step, x, [], masking=False)
-        return outputs
+        e = K.exp(x - K.max(x, axis=-1, keepdims=True))
+        s = K.sum(e, axis=-1, keepdims=True)
+        return e / s
     else:
         raise Exception('Cannot apply softmax to a tensor that is not 2D or 3D. ' +
                         'Here, ndim=' + str(ndim))
@@ -21,6 +19,10 @@ def softplus(x):
     return K.softplus(x)
 
 
+def softsign(x):
+    return K.softsign(x)
+
+
 def relu(x, alpha=0., max_value=None):
     return K.relu(x, alpha=alpha, max_value=max_value)
 
@@ -46,4 +48,6 @@ def linear(x):
 
 from .utils.generic_utils import get_from_module
 def get(identifier):
+    if identifier is None:
+        return linear
     return get_from_module(identifier, globals(), 'activation function')

keras/backend/__init__.py

@@ -2,7 +2,16 @@ from __future__ import absolute_import
 from __future__ import print_function
 import os
 import json
-from .common import epsilon, floatx, set_epsilon, set_floatx
+import sys
+from .common import epsilon
+from .common import floatx
+from .common import set_epsilon
+from .common import set_floatx
+from .common import get_uid
+from .common import cast_to_floatx
+from .common import image_dim_ordering
+from .common import set_image_dim_ordering
+from .common import is_keras_tensor
 
 _keras_base_dir = os.path.expanduser('~')
 if not os.access(_keras_base_dir, os.W_OK):
@@ -17,34 +26,45 @@ _config_path = os.path.expanduser(os.path.join(_keras_dir, 'keras.json'))
 if os.path.exists(_config_path):
     _config = json.load(open(_config_path))
     _floatx = _config.get('floatx', floatx())
-    assert _floatx in {'float32', 'float64'}
+    assert _floatx in {'float16', 'float32', 'float64'}
     _epsilon = _config.get('epsilon', epsilon())
     assert type(_epsilon) == float
     _backend = _config.get('backend', _BACKEND)
     assert _backend in {'theano', 'tensorflow'}
+    _image_dim_ordering = _config.get('image_dim_ordering', image_dim_ordering())
+    assert _image_dim_ordering in {'tf', 'th'}
 
     set_floatx(_floatx)
     set_epsilon(_epsilon)
+    set_image_dim_ordering(_image_dim_ordering)
     _BACKEND = _backend
-else:
-    # save config file, for easy edition
-    _config = {'floatx': floatx(),
-               'epsilon': epsilon(),
-               'backend': _BACKEND}
-    with open(_config_path, 'w') as f:
-        # add new line in order for bash 'cat' display the content correctly
-        f.write(json.dumps(_config) + '\n')
+
+# save config file
+_config = {'floatx': floatx(),
+           'epsilon': epsilon(),
+           'backend': _BACKEND,
+           'image_dim_ordering': image_dim_ordering()}
+with open(_config_path, 'w') as f:
+    f.write(json.dumps(_config, indent=4))
 
 if 'KERAS_BACKEND' in os.environ:
     _backend = os.environ['KERAS_BACKEND']
     assert _backend in {'theano', 'tensorflow'}
     _BACKEND = _backend
 
+# import backend
 if _BACKEND == 'theano':
-    print('Using Theano backend.')
+    sys.stderr.write('Using Theano backend.\n')
     from .theano_backend import *
 elif _BACKEND == 'tensorflow':
-    print('Using TensorFlow backend.')
+    sys.stderr.write('Using TensorFlow backend.\n')
     from .tensorflow_backend import *
 else:
     raise Exception('Unknown backend: ' + str(_BACKEND))
+
+
+def backend():
+    '''Publicly accessible method
+    for determining the current backend.
+    '''
+    return _BACKEND

keras/backend/common.py

@@ -1,32 +1,78 @@
 import numpy as np
 
+from collections import defaultdict
+
 # the type of float to use throughout the session.
 _FLOATX = 'float32'
 _EPSILON = 10e-8
+_UID_PREFIXES = defaultdict(int)
+_IMAGE_DIM_ORDERING = 'th'
 
 
 def epsilon():
+    '''Returns the value of the fuzz
+    factor used in numeric expressions.
+    '''
     return _EPSILON
 
 
 def set_epsilon(e):
+    '''Sets the value of the fuzz
+    factor used in numeric expressions.
+    '''
     global _EPSILON
     _EPSILON = e
 
 
 def floatx():
+    '''Returns the default float type, as a string
+    (e.g. 'float16', 'float32', 'float64').
+    '''
     return _FLOATX
 
 
 def set_floatx(floatx):
     global _FLOATX
-    if floatx not in {'float32', 'float64'}:
+    if floatx not in {'float16', 'float32', 'float64'}:
         raise Exception('Unknown floatx type: ' + str(floatx))
-    floatx = str(floatx)
-    _FLOATX = floatx
+    _FLOATX = str(floatx)
 
 
 def cast_to_floatx(x):
     '''Cast a Numpy array to floatx.
     '''
     return np.asarray(x, dtype=_FLOATX)
+
+
+def image_dim_ordering():
+    '''Returns the image dimension ordering
+    convention ('th' or 'tf').
+    '''
+    return _IMAGE_DIM_ORDERING
+
+
+def set_image_dim_ordering(dim_ordering):
+    '''Sets the value of the image dimension
+    ordering convention ('th' or 'tf').
+    '''
+    global _IMAGE_DIM_ORDERING
+    if dim_ordering not in {'tf', 'th'}:
+        raise Exception('Unknown dim_ordering:', dim_ordering)
+    _IMAGE_DIM_ORDERING = str(dim_ordering)
+
+
+def get_uid(prefix=''):
+    _UID_PREFIXES[prefix] += 1
+    return _UID_PREFIXES[prefix]
+
+
+def reset_uids():
+    global _UID_PREFIXES
+    _UID_PREFIXES = defaultdict(int)
+
+
+def is_keras_tensor(x):
+    if hasattr(x, '_keras_shape'):
+        return True
+    else:
+        return False

keras/callbacks.py

@@ -9,6 +9,7 @@ import warnings
 from collections import deque
 from .utils.generic_utils import Progbar
 from keras import backend as K
+from pkg_resources import parse_version
 
 
 class CallbackList(object):
@@ -60,8 +61,7 @@ class CallbackList(object):
             callback.on_batch_end(batch, logs)
         self._delta_ts_batch_end.append(time.time() - t_before_callbacks)
         delta_t_median = np.median(self._delta_ts_batch_end)
-        if self._delta_t_batch > 0. and delta_t_median > 0.95 * \
-           self._delta_t_batch and delta_t_median > 0.1:
+        if self._delta_t_batch > 0. and (delta_t_median > 0.95 * self._delta_t_batch and delta_t_median > 0.1):
             warnings.warn('Method on_batch_end() is slow compared '
                           'to the batch update (%f). Check your callbacks.'
                           % delta_t_median)
@@ -92,7 +92,8 @@ class Callback(object):
     will include the following quantities in the `logs` that
     it passes to its callbacks:
 
-        on_epoch_end: logs optionally include `val_loss`
+        on_epoch_end: logs include `acc` and `loss`, and
+            optionally include `val_loss`
             (if validation is enabled in `fit`), and `val_acc`
             (if validation and accuracy monitoring are enabled).
         on_batch_begin: logs include `size`,
@@ -129,11 +130,35 @@ class Callback(object):
 
 
 class BaseLogger(Callback):
-    '''Callback that prints events to the standard output.
+    '''Callback that accumulates epoch averages of
+    the metrics being monitored.
 
     This callback is automatically applied to
-    every Keras model (it is the basis of the verbosity modes
-    in models).
+    every Keras model.
+    '''
+    def on_epoch_begin(self, epoch, logs={}):
+        self.seen = 0
+        self.totals = {}
+
+    def on_batch_end(self, batch, logs={}):
+        batch_size = logs.get('size', 0)
+        self.seen += batch_size
+
+        for k, v in logs.items():
+            if k in self.totals:
+                self.totals[k] += v * batch_size
+            else:
+                self.totals[k] = v * batch_size
+
+    def on_epoch_end(self, epoch, logs={}):
+        for k in self.params['metrics']:
+            if k in self.totals:
+                # make value available to next callbacks
+                logs[k] = self.totals[k] / self.seen
+
+
+class ProgbarLogger(Callback):
+    '''Callback that prints metrics to stdout.
     '''
     def on_train_begin(self, logs={}):
         self.verbose = self.params['verbose']
@@ -145,7 +170,6 @@ class BaseLogger(Callback):
             self.progbar = Progbar(target=self.params['nb_sample'],
                                    verbose=self.verbose)
         self.seen = 0
-        self.totals = {}
 
     def on_batch_begin(self, batch, logs={}):
         if self.seen < self.params['nb_sample']:
@@ -155,11 +179,6 @@ class BaseLogger(Callback):
         batch_size = logs.get('size', 0)
         self.seen += batch_size
 
-        for k, v in logs.items():
-            if k in self.totals:
-                self.totals[k] += v * batch_size
-            else:
-                self.totals[k] = v * batch_size
         for k in self.params['metrics']:
             if k in logs:
                 self.log_values.append((k, logs[k]))
@@ -171,12 +190,10 @@ class BaseLogger(Callback):
 
     def on_epoch_end(self, epoch, logs={}):
         for k in self.params['metrics']:
-            if k in self.totals:
-                self.log_values.append((k, self.totals[k] / self.seen))
             if k in logs:
                 self.log_values.append((k, logs[k]))
         if self.verbose:
-            self.progbar.update(self.seen, self.log_values)
+            self.progbar.update(self.seen, self.log_values, force=True)
 
 
 class History(Callback):
@@ -191,30 +208,10 @@ class History(Callback):
         self.epoch = []
         self.history = {}
 
-    def on_epoch_begin(self, epoch, logs={}):
-        self.seen = 0
-        self.totals = {}
-
-    def on_batch_end(self, batch, logs={}):
-        batch_size = logs.get('size', 0)
-        self.seen += batch_size
-        for k, v in logs.items():
-            if k in self.totals:
-                self.totals[k] += v * batch_size
-            else:
-                self.totals[k] = v * batch_size
-
     def on_epoch_end(self, epoch, logs={}):
         self.epoch.append(epoch)
-        for k, v in self.totals.items():
-            if k not in self.history:
-                self.history[k] = []
-            self.history[k].append(v / self.seen)
-
         for k, v in logs.items():
-            if k not in self.history:
-                self.history[k] = []
-            self.history[k].append(v)
+            self.history.setdefault(k, []).append(v)
 
 
 class ModelCheckpoint(Callback):
@@ -234,29 +231,33 @@ class ModelCheckpoint(Callback):
         verbose: verbosity mode, 0 or 1.
         save_best_only: if `save_best_only=True`,
             the latest best model according to
-            the validation loss will not be overwritten.
+            the quantity monitored will not be overwritten.
         mode: one of {auto, min, max}.
             If `save_best_only=True`, the decision
             to overwrite the current save file is made
             based on either the maximization or the
-            minization of the monitored. For `val_acc`,
+            minimization of the monitored quantity. For `val_acc`,
             this should be `max`, for `val_loss` this should
             be `min`, etc. In `auto` mode, the direction is
             automatically inferred from the name of the monitored quantity.
+        save_weights_only: if True, then only the model's weights will be
+            saved (`model.save_weights(filepath)`), else the full model
+            is saved (`model.save(filepath)`).
 
     '''
     def __init__(self, filepath, monitor='val_loss', verbose=0,
-                 save_best_only=False, mode='auto'):
-
-        super(Callback, self).__init__()
+                 save_best_only=False, save_weights_only=False,
+                 mode='auto'):
+        super(ModelCheckpoint, self).__init__()
         self.monitor = monitor
         self.verbose = verbose
         self.filepath = filepath
         self.save_best_only = save_best_only
+        self.save_weights_only = save_weights_only
 
         if mode not in ['auto', 'min', 'max']:
             warnings.warn('ModelCheckpoint mode %s is unknown, '
-                          'fallback to auto mode.' % (self.mode),
+                          'fallback to auto mode.' % (mode),
                           RuntimeWarning)
             mode = 'auto'
 
@@ -289,7 +290,10 @@ class ModelCheckpoint(Callback):
                               % (epoch, self.monitor, self.best,
                                  current, filepath))
                     self.best = current
-                    self.model.save_weights(filepath, overwrite=True)
+                    if self.save_weights_only:
+                        self.model.save_weights(filepath, overwrite=True)
+                    else:
+                        self.model.save(filepath, overwrite=True)
                 else:
                     if self.verbose > 0:
                         print('Epoch %05d: %s did not improve' %
@@ -297,7 +301,10 @@ class ModelCheckpoint(Callback):
         else:
             if self.verbose > 0:
                 print('Epoch %05d: saving model to %s' % (epoch, filepath))
-            self.model.save_weights(filepath, overwrite=True)
+            if self.save_weights_only:
+                self.model.save_weights(filepath, overwrite=True)
+            else:
+                self.model.save(filepath, overwrite=True)
 
 
 class EarlyStopping(Callback):
@@ -315,7 +322,7 @@ class EarlyStopping(Callback):
             monitored has stopped increasing.
     '''
     def __init__(self, monitor='val_loss', patience=0, verbose=0, mode='auto'):
-        super(Callback, self).__init__()
+        super(EarlyStopping, self).__init__()
 
         self.monitor = monitor
         self.patience = patience
@@ -324,22 +331,23 @@ class EarlyStopping(Callback):
 
         if mode not in ['auto', 'min', 'max']:
             warnings.warn('EarlyStopping mode %s is unknown, '
-                          'fallback to auto mode.' % (self.mode), RuntimeWarning)
+                          'fallback to auto mode.' % (self.mode),
+                          RuntimeWarning)
             mode = 'auto'
 
         if mode == 'min':
             self.monitor_op = np.less
-            self.best = np.Inf
         elif mode == 'max':
             self.monitor_op = np.greater
-            self.best = -np.Inf
         else:
             if 'acc' in self.monitor:
                 self.monitor_op = np.greater
-                self.best = -np.Inf
             else:
                 self.monitor_op = np.less
-                self.best = np.Inf
+
+    def on_train_begin(self, logs={}):
+        self.wait = 0       # Allow instances to be re-used
+        self.best = np.Inf if self.monitor_op == np.less else -np.Inf
 
     def on_epoch_end(self, epoch, logs={}):
         current = logs.get(self.monitor)
@@ -366,39 +374,29 @@ class RemoteMonitor(Callback):
     # Arguments
         root: root url to which the events will be sent (at the end
             of every epoch). Events are sent to
-            `root + '/publish/epoch/end/'`. Calls are HTTP POST,
-            with a `data` argument which is a JSON-encoded dictionary
-            of event data.
+            `root + '/publish/epoch/end/'` by default. Calls are
+            HTTP POST, with a `data` argument which is a
+            JSON-encoded dictionary of event data.
     '''
-    def __init__(self, root='http://localhost:9000'):
+
+    def __init__(self,
+                 root='http://localhost:9000',
+                 path='/publish/epoch/end/',
+                 field='data'):
+        super(RemoteMonitor, self).__init__()
         self.root = root
-
-    def on_epoch_begin(self, epoch, logs={}):
-        self.seen = 0
-        self.totals = {}
-
-    def on_batch_end(self, batch, logs={}):
-        batch_size = logs.get('size', 0)
-        self.seen += batch_size
-        for k, v in logs.items():
-            if k in self.totals:
-                self.totals[k] += v * batch_size
-            else:
-                self.totals[k] = v * batch_size
+        self.path = path
+        self.field = field
 
     def on_epoch_end(self, epoch, logs={}):
         import requests
         send = {}
         send['epoch'] = epoch
-
-        for k, v in self.totals.items():
-            send[k] = v / self.seen
         for k, v in logs.items():
             send[k] = v
-
         try:
-            requests.post(self.root + '/publish/epoch/end/',
-                          {'data': json.dumps(send)})
+            requests.post(self.root + self.path,
+                          {self.field: json.dumps(send)})
         except:
             print('Warning: could not reach RemoteMonitor '
                   'root server at ' + str(self.root))
@@ -444,76 +442,72 @@ class TensorBoard(Callback):
 
     # Arguments
         log_dir: the path of the directory where to save the log
-            files to be parsed by tensorboard
+            files to be parsed by Tensorboard
         histogram_freq: frequency (in epochs) at which to compute activation
             histograms for the layers of the model. If set to 0,
             histograms won't be computed.
+        write_graph: whether to visualize the graph in Tensorboard.
+            The log file can become quite large when
+            write_graph is set to True.
     '''
-    def __init__(self, log_dir='./logs', histogram_freq=0):
-        super(Callback, self).__init__()
+
+    def __init__(self, log_dir='./logs', histogram_freq=0, write_graph=True):
+        super(TensorBoard, self).__init__()
         if K._BACKEND != 'tensorflow':
             raise Exception('TensorBoard callback only works '
                             'with the TensorFlow backend.')
         self.log_dir = log_dir
         self.histogram_freq = histogram_freq
+        self.merged = None
+        self.write_graph = write_graph
 
     def _set_model(self, model):
         import tensorflow as tf
         import keras.backend.tensorflow_backend as KTF
 
         self.model = model
-        self.sess = KTF._get_session()
-        if self.histogram_freq:
-            mod_type = self.model.get_config()['name']
-            if mod_type == 'Sequential':
-                layers = {l.get_config()['name']: l for l in self.model.layers}
-            elif mod_type == 'Graph':
-                layers = self.model.nodes
-            else:
-                raise Exception('Unrecognized model:',
-                                self.model.get_config()['name'])
-            for l in layers:
-                cur_layer = layers[l]
-                if hasattr(cur_layer, 'W'):
-                    tf.histogram_summary('{}_W'.format(l), cur_layer.W)
-                if hasattr(cur_layer, 'b'):
-                    tf.histogram_summary('{}_b'.format(l), cur_layer.b)
-                if hasattr(cur_layer, 'get_output'):
-                    tf.histogram_summary('{}_out'.format(l),
-                                         cur_layer.get_output())
+        self.sess = KTF.get_session()
+        if self.histogram_freq and self.merged is None:
+            layers = self.model.layers
+            for layer in layers:
+                if hasattr(layer, 'W'):
+                    tf.histogram_summary('{}_W'.format(layer), layer.W)
+                if hasattr(layer, 'b'):
+                    tf.histogram_summary('{}_b'.format(layer), layer.b)
+                if hasattr(layer, 'output'):
+                    tf.histogram_summary('{}_out'.format(layer),
+                                         layer.output)
         self.merged = tf.merge_all_summaries()
-        self.writer = tf.train.SummaryWriter(self.log_dir,
-                                             self.sess.graph_def)
-
-    def on_epoch_begin(self, epoch, logs={}):
-        self.seen = 0
-        self.totals = {}
-
-    def on_batch_end(self, batch, logs={}):
-        batch_size = logs.get('size', 0)
-        self.seen += batch_size
-        for k, v in logs.items():
-            if k in self.totals:
-                self.totals[k] += v * batch_size
+        if self.write_graph:
+            if parse_version(tf.__version__) >= parse_version('0.8.0'):
+                self.writer = tf.train.SummaryWriter(self.log_dir,
+                                                     self.sess.graph)
             else:
-                self.totals[k] = v * batch_size
+                self.writer = tf.train.SummaryWriter(self.log_dir,
+                                                     self.sess.graph_def)
+        else:
+            self.writer = tf.train.SummaryWriter(self.log_dir)
 
     def on_epoch_end(self, epoch, logs={}):
         import tensorflow as tf
 
         if self.model.validation_data and self.histogram_freq:
             if epoch % self.histogram_freq == 0:
-                if self.params.get('show_accuracy'):
-                    test_function = self.model._test_with_acc
+                # TODO: implement batched calls to sess.run
+                # (current call will likely go OOM on GPU)
+                if self.model.uses_learning_phase:
+                    cut_v_data = len(self.model.inputs)
+                    val_data = self.model.validation_data[:cut_v_data] + [0]
+                    tensors = self.model.inputs + [K.learning_phase()]
                 else:
-                    test_function = self.model._test
-                names = [v.name for v in test_function.inputs]
-                feed_dict = dict(zip(names, self.model.validation_data))
+                    val_data = self.model.validation_data
+                    tensors = self.model.inputs
+                feed_dict = dict(zip(tensors, val_data))
                 result = self.sess.run([self.merged], feed_dict=feed_dict)
                 summary_str = result[0]
                 self.writer.add_summary(summary_str, epoch)
 
-        for name, value in self.totals.items() + logs.items():
+        for name, value in logs.items():
             if name in ['batch', 'size']:
                 continue
             summary = tf.Summary()

keras/constraints.py

@@ -7,39 +7,87 @@ class Constraint(object):
         return p
 
     def get_config(self):
-        return {"name": self.__class__.__name__}
+        return {'name': self.__class__.__name__}
 
 
 class MaxNorm(Constraint):
-    def __init__(self, m=2):
+    '''Constrain the weights incident to each hidden unit to have a norm less than or equal to a desired value.
+
+    # Arguments
+        m: the maximum norm for the incoming weights.
+        axis: integer, axis along which to calculate weight norms. For instance,
+            in a `Dense` layer the weight matrix has shape (input_dim, output_dim),
+            set `axis` to `0` to constrain each weight vector of length (input_dim).
+            In a `MaxoutDense` layer the weight tensor has shape (nb_feature, input_dim, output_dim),
+            set `axis` to `1` to constrain each weight vector of length (input_dim),
+            i.e. constrain the filters incident to the `max` operation.
+            In a `Convolution2D` layer with the Theano backend, the weight tensor
+            has shape (nb_filter, stack_size, nb_row, nb_col), set `axis` to `[1,2,3]`
+            to constrain the weights of each filter tensor of size (stack_size, nb_row, nb_col).
+            In a `Convolution2D` layer with the TensorFlow backend, the weight tensor
+            has shape (nb_row, nb_col, stack_size, nb_filter), set `axis` to `[0,1,2]`
+            to constrain the weights of each filter tensor of size (nb_row, nb_col, stack_size).
+
+    # References
+        - [Dropout: A Simple Way to Prevent Neural Networks from Overfitting Srivastava, Hinton, et al. 2014](http://www.cs.toronto.edu/~rsalakhu/papers/srivastava14a.pdf)
+    '''
+    def __init__(self, m=2, axis=0):
         self.m = m
+        self.axis = axis
 
     def __call__(self, p):
-        norms = K.sqrt(K.sum(K.square(p), axis=0))
+        norms = K.sqrt(K.sum(K.square(p), axis=self.axis, keepdims=True))
         desired = K.clip(norms, 0, self.m)
-        p = p * (desired / (1e-7 + norms))
+        p = p * (desired / (K.epsilon() + norms))
         return p
 
     def get_config(self):
-        return {"name": self.__class__.__name__,
-                "m": self.m}
+        return {'name': self.__class__.__name__,
+                'm': self.m,
+                'axis': self.axis}
 
 
 class NonNeg(Constraint):
+    '''Constrain the weights to be non-negative.
+    '''
     def __call__(self, p):
         p *= K.cast(p >= 0., K.floatx())
         return p
 
 
 class UnitNorm(Constraint):
-    def __call__(self, p):
-        return p / K.sqrt(K.sum(K.square(p), axis=-1, keepdims=True))
+    '''Constrain the weights incident to each hidden unit to have unit norm.
+
+    # Arguments
+        axis: integer, axis along which to calculate weight norms. For instance,
+            in a `Dense` layer the weight matrix has shape (input_dim, output_dim),
+            set `axis` to `0` to constrain each weight vector of length (input_dim).
+            In a `MaxoutDense` layer the weight tensor has shape (nb_feature, input_dim, output_dim),
+            set `axis` to `1` to constrain each weight vector of length (input_dim),
+            i.e. constrain the filters incident to the `max` operation.
+            In a `Convolution2D` layer with the Theano backend, the weight tensor
+            has shape (nb_filter, stack_size, nb_row, nb_col), set `axis` to `[1,2,3]`
+            to constrain the weights of each filter tensor of size (stack_size, nb_row, nb_col).
+            In a `Convolution2D` layer with the TensorFlow backend, the weight tensor
+            has shape (nb_row, nb_col, stack_size, nb_filter), set `axis` to `[0,1,2]`
+            to constrain the weights of each filter tensor of size (nb_row, nb_col, stack_size).
+    '''
+    def __init__(self, axis=0):
+        self.axis = axis
+
+    def __call__(self, p):
+        return p / (K.epsilon() + K.sqrt(K.sum(K.square(p), axis=self.axis, keepdims=True)))
+
+    def get_config(self):
+        return {'name': self.__class__.__name__,
+                'axis': self.axis}
+
 
-identity = Constraint
 maxnorm = MaxNorm
 nonneg = NonNeg
 unitnorm = UnitNorm
 
 from .utils.generic_utils import get_from_module
 def get(identifier, kwargs=None):
-    return get_from_module(identifier, globals(), 'constraint', instantiate=True, kwargs=kwargs)
+    return get_from_module(identifier, globals(), 'constraint',
+                           instantiate=True, kwargs=kwargs)

keras/datasets/cifar.py

@@ -2,7 +2,7 @@
 from __future__ import absolute_import
 import sys
 from six.moves import cPickle
-from six.moves import range
+
 
 def load_batch(fpath, label_key='labels'):
     f = open(fpath, 'rb')

keras/datasets/cifar10.py

@@ -1,6 +1,6 @@
 from __future__ import absolute_import
 from .cifar import load_batch
-from .data_utils import get_file
+from ..utils.data_utils import get_file
 import numpy as np
 import os
 

keras/datasets/cifar100.py

@@ -1,6 +1,6 @@
 from __future__ import absolute_import
 from .cifar import load_batch
-from .data_utils import get_file
+from ..utils.data_utils import get_file
 import numpy as np
 import os
 

keras/datasets/data_utils.py

@@ -1,53 +1,4 @@
-from __future__ import absolute_import
-from __future__ import print_function
+from ..utils.data_utils import *
+import warnings
 
-import tarfile
-import os
-from six.moves.urllib.request import FancyURLopener
-
-from ..utils.generic_utils import Progbar
-
-
-class ParanoidURLopener(FancyURLopener):
-    def http_error_default(self, url, fp, errcode, errmsg, headers):
-        raise Exception('URL fetch failure on {}: {} -- {}'.format(url, errcode, errmsg))
-
-
-def get_file(fname, origin, untar=False):
-    datadir_base = os.path.expanduser(os.path.join('~', '.keras'))
-    if not os.access(datadir_base, os.W_OK):
-        datadir_base = os.path.join('/tmp', '.keras')
-    datadir = os.path.join(datadir_base, 'datasets')
-    if not os.path.exists(datadir):
-        os.makedirs(datadir)
-
-    if untar:
-        untar_fpath = os.path.join(datadir, fname)
-        fpath = untar_fpath + '.tar.gz'
-    else:
-        fpath = os.path.join(datadir, fname)
-
-    if not os.path.exists(fpath):
-        print('Downloading data from',  origin)
-        global progbar
-        progbar = None
-
-        def dl_progress(count, block_size, total_size):
-            global progbar
-            if progbar is None:
-                progbar = Progbar(total_size)
-            else:
-                progbar.update(count*block_size)
-
-        ParanoidURLopener().retrieve(origin, fpath, dl_progress)
-        progbar = None
-
-    if untar:
-        if not os.path.exists(untar_fpath):
-            print('Untaring file...')
-            tfile = tarfile.open(fpath, 'r:gz')
-            tfile.extractall(path=datadir)
-            tfile.close()
-        return untar_fpath
-
-    return fpath
+warnings.warn('data_utils has been moved to keras.utils.data_utils.')

keras/datasets/imdb.py

@@ -1,29 +1,61 @@
 from __future__ import absolute_import
 from six.moves import cPickle
 import gzip
-from .data_utils import get_file
+from ..utils.data_utils import get_file
 from six.moves import zip
 import numpy as np
+import sys
 
 
-def load_data(path="imdb.pkl", nb_words=None, skip_top=0,
-              maxlen=None, test_split=0.2, seed=113,
+def load_data(path='imdb_full.pkl', nb_words=None, skip_top=0,
+              maxlen=None, seed=113,
               start_char=1, oov_char=2, index_from=3):
+    '''
+    # Arguments
+        path: where to store the data (in `/.keras/dataset`)
+        nb_words: max number of words to include. Words are ranked
+            by how often they occur (in the training set) and only
+            the most frequent words are kept
+        skip_top: skip the top N most frequently occuring words
+            (which may not be informative).
+        maxlen: truncate sequences after this length.
+        seed: random seed for sample shuffling.
+        start_char: The start of a sequence will be marked with this character.
+            Set to 1 because 0 is usually the padding character.
+        oov_char: words that were cut out because of the `nb_words`
+            or `skip_top` limit will be replaced with this character.
+        index_from: index actual words with this index and higher.
 
-    path = get_file(path, origin="https://s3.amazonaws.com/text-datasets/imdb.pkl")
+    Note that the 'out of vocabulary' character is only used for
+    words that were present in the training set but are not included
+    because they're not making the `nb_words` cut here.
+    Words that were not seen in the trining set but are in the test set
+    have simply been skipped.
+    '''
+    path = get_file(path,
+                    origin='https://s3.amazonaws.com/text-datasets/imdb_full.pkl',
+                    md5_hash='d091312047c43cf9e4e38fef92437263')
 
-    if path.endswith(".gz"):
+    if path.endswith('.gz'):
         f = gzip.open(path, 'rb')
     else:
         f = open(path, 'rb')
 
-    X, labels = cPickle.load(f)
+    (x_train, labels_train), (x_test, labels_test) = cPickle.load(f)
     f.close()
 
     np.random.seed(seed)
-    np.random.shuffle(X)
+    np.random.shuffle(x_train)
     np.random.seed(seed)
-    np.random.shuffle(labels)
+    np.random.shuffle(labels_train)
+
+    np.random.seed(seed * 2)
+    np.random.shuffle(x_test)
+    np.random.seed(seed * 2)
+    np.random.shuffle(labels_test)
+
+    X = x_train + x_test
+    labels = labels_train + labels_test
 
     if start_char is not None:
         X = [[start_char] + [w + index_from for w in x] for x in X]
@@ -60,10 +92,25 @@ def load_data(path="imdb.pkl", nb_words=None, skip_top=0,
             nX.append(nx)
         X = nX
 
-    X_train = X[:int(len(X) * (1 - test_split))]
-    y_train = labels[:int(len(X) * (1 - test_split))]
+    X_train = np.array(X[:len(x_train)])
+    y_train = np.array(labels[:len(x_train)])
 
-    X_test = X[int(len(X) * (1 - test_split)):]
-    y_test = labels[int(len(X) * (1 - test_split)):]
+    X_test = np.array(X[len(x_train):])
+    y_test = np.array(labels[len(x_train):])
 
     return (X_train, y_train), (X_test, y_test)
+
+
+def get_word_index(path='imdb_word_index.pkl'):
+    path = get_file(path,
+                    origin='https://s3.amazonaws.com/text-datasets/imdb_word_index.pkl',
+                    md5_hash='72d94b01291be4ff843198d3b0e1e4d7')
+    f = open(path, 'rb')
+
+    if sys.version_info < (3,):
+        data = cPickle.load(f)
+    else:
+        data = cPickle.load(f, encoding='latin1')
+
+    f.close()
+    return data

keras/datasets/mnist.py

@@ -1,6 +1,6 @@
 # -*- coding: utf-8 -*-
 import gzip
-from .data_utils import get_file
+from ..utils.data_utils import get_file
 from six.moves import cPickle
 import sys
 

keras/datasets/reuters.py

@@ -1,16 +1,17 @@
 # -*- coding: utf-8 -*-
 from __future__ import absolute_import
-from .data_utils import get_file
+from ..utils.data_utils import get_file
 from six.moves import cPickle
 from six.moves import zip
 import numpy as np
+import sys
 
 
-def load_data(path="reuters.pkl", nb_words=None, skip_top=0,
+def load_data(path='reuters.pkl', nb_words=None, skip_top=0,
               maxlen=None, test_split=0.2, seed=113,
               start_char=1, oov_char=2, index_from=3):
 
-    path = get_file(path, origin="https://s3.amazonaws.com/text-datasets/reuters.pkl")
+    path = get_file(path, origin='https://s3.amazonaws.com/text-datasets/reuters.pkl')
     f = open(path, 'rb')
     X, labels = cPickle.load(f)
     f.close()
@@ -61,7 +62,14 @@ def load_data(path="reuters.pkl", nb_words=None, skip_top=0,
     return (X_train, y_train), (X_test, y_test)
 
 
-def get_word_index(path="reuters_word_index.pkl"):
-    path = get_file(path, origin="https://s3.amazonaws.com/text-datasets/reuters_word_index.pkl")
+def get_word_index(path='reuters_word_index.pkl'):
+    path = get_file(path, origin='https://s3.amazonaws.com/text-datasets/reuters_word_index.pkl')
     f = open(path, 'rb')
-    return cPickle.load(f)
+
+    if sys.version_info < (3,):
+        data = cPickle.load(f)
+    else:
+        data = cPickle.load(f, encoding='latin1')
+
+    f.close()
+    return data

keras/engine/__init__.py

@@ -0,0 +1,10 @@
+# note: topology.Node is an internal class,
+# it isn't meant to be used by Keras users.
+from .topology import InputSpec
+from .topology import Input
+from .topology import InputLayer
+from .topology import Layer
+from .topology import Merge
+from .topology import merge
+from .topology import get_source_inputs
+from .training import Model

keras/initializations.py

@@ -3,58 +3,77 @@ import numpy as np
 from . import backend as K
 
 
-def get_fans(shape):
-    fan_in = shape[0] if len(shape) == 2 else np.prod(shape[1:])
-    fan_out = shape[1] if len(shape) == 2 else shape[0]
+def get_fans(shape, dim_ordering='th'):
+    if len(shape) == 2:
+        fan_in = shape[0]
+        fan_out = shape[1]
+    elif len(shape) == 4 or len(shape) == 5:
+        # assuming convolution kernels (2D or 3D).
+        # TH kernel shape: (depth, input_depth, ...)
+        # TF kernel shape: (..., input_depth, depth)
+        if dim_ordering == 'th':
+            receptive_field_size = np.prod(shape[2:])
+            fan_in = shape[1] * receptive_field_size
+            fan_out = shape[0] * receptive_field_size
+        elif dim_ordering == 'tf':
+            receptive_field_size = np.prod(shape[:2])
+            fan_in = shape[-2] * receptive_field_size
+            fan_out = shape[-1] * receptive_field_size
+        else:
+            raise Exception('Invalid dim_ordering: ' + dim_ordering)
+    else:
+        # no specific assumptions
+        fan_in = np.sqrt(np.prod(shape))
+        fan_out = np.sqrt(np.prod(shape))
     return fan_in, fan_out
 
 
-def uniform(shape, scale=0.05):
-    return K.variable(np.random.uniform(low=-scale, high=scale, size=shape))
+def uniform(shape, scale=0.05, name=None):
+    return K.random_uniform_variable(shape, -scale, scale, name=name)
 
 
-def normal(shape, scale=0.05):
-    return K.variable(np.random.normal(loc=0.0, scale=scale, size=shape))
+def normal(shape, scale=0.05, name=None):
+    return K.random_normal_variable(shape, 0.0, scale, name=name)
 
 
-def lecun_uniform(shape):
+def lecun_uniform(shape, name=None, dim_ordering='th'):
     ''' Reference: LeCun 98, Efficient Backprop
         http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf
     '''
-    fan_in, fan_out = get_fans(shape)
+    fan_in, fan_out = get_fans(shape, dim_ordering=dim_ordering)
     scale = np.sqrt(3. / fan_in)
-    return uniform(shape, scale)
+    return uniform(shape, scale, name=name)
 
 
-def glorot_normal(shape):
+def glorot_normal(shape, name=None, dim_ordering='th'):
     ''' Reference: Glorot & Bengio, AISTATS 2010
     '''
-    fan_in, fan_out = get_fans(shape)
+    fan_in, fan_out = get_fans(shape, dim_ordering=dim_ordering)
     s = np.sqrt(2. / (fan_in + fan_out))
-    return normal(shape, s)
+    return normal(shape, s, name=name)
 
 
-def glorot_uniform(shape):
-    fan_in, fan_out = get_fans(shape)
+def glorot_uniform(shape, name=None, dim_ordering='th'):
+    fan_in, fan_out = get_fans(shape, dim_ordering=dim_ordering)
     s = np.sqrt(6. / (fan_in + fan_out))
-    return uniform(shape, s)
+    return uniform(shape, s, name=name)
 
 
-def he_normal(shape):
+def he_normal(shape, name=None, dim_ordering='th'):
     ''' Reference:  He et al., http://arxiv.org/abs/1502.01852
     '''
-    fan_in, fan_out = get_fans(shape)
+    fan_in, fan_out = get_fans(shape, dim_ordering=dim_ordering)
     s = np.sqrt(2. / fan_in)
-    return normal(shape, s)
+    return normal(shape, s, name=name)
 
 
-def he_uniform(shape):
-    fan_in, fan_out = get_fans(shape)
+def he_uniform(shape, name=None, dim_ordering='th'):
+    fan_in, fan_out = get_fans(shape, dim_ordering=dim_ordering)
     s = np.sqrt(6. / fan_in)
-    return uniform(shape, s)
+    return uniform(shape, s, name=name)
 
 
-def orthogonal(shape, scale=1.1):
+def orthogonal(shape, scale=1.1, name=None):
     ''' From Lasagne. Reference: Saxe et al., http://arxiv.org/abs/1312.6120
     '''
     flat_shape = (shape[0], np.prod(shape[1:]))
@@ -63,25 +82,26 @@ def orthogonal(shape, scale=1.1):
     # pick the one with the correct shape
     q = u if u.shape == flat_shape else v
     q = q.reshape(shape)
-    return K.variable(scale * q[:shape[0], :shape[1]])
+    return K.variable(scale * q[:shape[0], :shape[1]], name=name)
 
 
-def identity(shape, scale=1):
+def identity(shape, scale=1, name=None):
     if len(shape) != 2 or shape[0] != shape[1]:
         raise Exception('Identity matrix initialization can only be used '
                         'for 2D square matrices.')
     else:
-        return K.variable(scale * np.identity(shape[0]))
+        return K.variable(scale * np.identity(shape[0]), name=name)
 
 
-def zero(shape):
-    return K.zeros(shape)
+def zero(shape, name=None):
+    return K.zeros(shape, name=name)
 
 
-def one(shape):
-    return K.ones(shape)
+def one(shape, name=None):
+    return K.ones(shape, name=name)
 
 
 from .utils.generic_utils import get_from_module
-def get(identifier):
-    return get_from_module(identifier, globals(), 'initialization')
+def get(identifier, **kwargs):
+    return get_from_module(identifier, globals(),
+                           'initialization', kwargs=kwargs)

keras/layers/__init__.py

@@ -0,0 +1,12 @@
+from __future__ import absolute_import
+from ..engine import Layer, Input, InputLayer, Merge, merge, InputSpec
+from .core import *
+from .convolutional import *
+from .pooling import *
+from .local import *
+from .recurrent import *
+from .normalization import *
+from .embeddings import *
+from .noise import *
+from .advanced_activations import *
+from .wrappers import *

keras/layers/advanced_activations.py

@@ -1,13 +1,14 @@
 from .. import initializations
-from ..layers.core import MaskedLayer
+from ..engine import Layer
 from .. import backend as K
 import numpy as np
 
 
-class LeakyReLU(MaskedLayer):
+class LeakyReLU(Layer):
     '''Special version of a Rectified Linear Unit
-    that allows a small gradient when the unit is not active
-    (`f(x) = alpha*x for x < 0`).
+    that allows a small gradient when the unit is not active:
+    `f(x) = alpha * x for x < 0`,
+    `f(x) = x for x >= 0`.
 
     # Input shape
         Arbitrary. Use the keyword argument `input_shape`
@@ -21,22 +22,25 @@ class LeakyReLU(MaskedLayer):
         alpha: float >= 0. Negative slope coefficient.
     '''
     def __init__(self, alpha=0.3, **kwargs):
-        super(LeakyReLU, self).__init__(**kwargs)
+        self.supports_masking = True
         self.alpha = alpha
+        super(LeakyReLU, self).__init__(**kwargs)
 
-    def get_output(self, train):
-        X = self.get_input(train)
-        return K.relu(X, alpha=self.alpha)
+    def call(self, x, mask=None):
+        return K.relu(x, alpha=self.alpha)
 
     def get_config(self):
-        config = {"name": self.__class__.__name__,
-                  "alpha": self.alpha}
+        config = {'alpha': self.alpha}
         base_config = super(LeakyReLU, self).get_config()
         return dict(list(base_config.items()) + list(config.items()))
 
 
-class PReLU(MaskedLayer):
-    '''
+class PReLU(Layer):
+    '''Parametric Rectified Linear Unit:
+    `f(x) = alphas * x for x < 0`,
+    `f(x) = x for x >= 0`,
+    where `alphas` is a learned array with the same shape as x.
+
     # Input shape
         Arbitrary. Use the keyword argument `input_shape`
         (tuple of integers, does not include the samples axis)
@@ -45,42 +49,44 @@ class PReLU(MaskedLayer):
     # Output shape
         Same shape as the input.
 
-    # Arguments:
+    # Arguments
         init: initialization function for the weights.
-        weights: initial weights, as a list of a single numpy array.
+        weights: initial weights, as a list of a single Numpy array.
 
-    # References:
+    # References
         - [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](http://arxiv.org/pdf/1502.01852v1.pdf)
     '''
     def __init__(self, init='zero', weights=None, **kwargs):
+        self.supports_masking = True
         self.init = initializations.get(init)
         self.initial_weights = weights
         super(PReLU, self).__init__(**kwargs)
 
-    def build(self):
-        input_shape = self.input_shape[1:]
-        self.alphas = self.init(input_shape)
-        self.params = [self.alphas]
+    def build(self, input_shape):
+        self.alphas = self.init(input_shape[1:],
+                                name='{}_alphas'.format(self.name))
+        self.trainable_weights = [self.alphas]
 
         if self.initial_weights is not None:
             self.set_weights(self.initial_weights)
             del self.initial_weights
 
-    def get_output(self, train):
-        X = self.get_input(train)
-        pos = K.relu(X)
-        neg = self.alphas * (X - abs(X)) * 0.5
+    def call(self, x, mask=None):
+        pos = K.relu(x)
+        neg = self.alphas * (x - abs(x)) * 0.5
         return pos + neg
 
     def get_config(self):
-        config = {"name": self.__class__.__name__,
-                  "init": self.init.__name__}
+        config = {'init': self.init.__name__}
         base_config = super(PReLU, self).get_config()
         return dict(list(base_config.items()) + list(config.items()))
 
 
-class ELU(MaskedLayer):
-    '''
+class ELU(Layer):
+    '''Exponential Linear Unit:
+    `f(x) =  alpha * (exp(x) - 1.) for x < 0`,
+    `f(x) = x for x >= 0`.
+
     # Input shape
         Arbitrary. Use the keyword argument `input_shape`
         (tuple of integers, does not include the samples axis)
@@ -96,24 +102,24 @@ class ELU(MaskedLayer):
         - [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)](http://arxiv.org/pdf/1511.07289v1.pdf)
     '''
     def __init__(self, alpha=1.0, **kwargs):
+        self.supports_masking = True
+        self.alpha = K.cast_to_floatx(alpha)
         super(ELU, self).__init__(**kwargs)
-        self.alpha = alpha
 
-    def get_output(self, train):
-        X = self.get_input(train)
-        pos = K.relu(X)
-        neg = (X - abs(X)) * 0.5
+    def call(self, x, mask=None):
+        pos = K.relu(x)
+        neg = (x - abs(x)) * 0.5
         return pos + self.alpha * (K.exp(neg) - 1.)
 
     def get_config(self):
-        config = {"name": self.__class__.__name__,
-                  "alpha": self.alpha}
+        config = {'alpha': float(self.alpha)}
         base_config = super(ELU, self).get_config()
         return dict(list(base_config.items()) + list(config.items()))
 
 
-class ParametricSoftplus(MaskedLayer):
-    '''Parametric Softplus of the form: alpha * log(1 + exp(beta * X))
+class ParametricSoftplus(Layer):
+    '''Parametric Softplus:
+    `alpha * log(1 + exp(beta * x))`
 
     # Input shape
         Arbitrary. Use the keyword argument `input_shape`
@@ -128,40 +134,43 @@ class ParametricSoftplus(MaskedLayer):
         beta_init: float. Initial values of the beta weights.
         weights: initial weights, as a list of 2 numpy arrays.
 
-    # References:
+    # References
         - [Inferring Nonlinear Neuronal Computation Based on Physiologically Plausible Inputs](http://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1003143)
     '''
     def __init__(self, alpha_init=0.2, beta_init=5.0,
                  weights=None, **kwargs):
-        self.alpha_init = alpha_init
-        self.beta_init = beta_init
+        self.supports_masking = True
+        self.alpha_init = K.cast_to_floatx(alpha_init)
+        self.beta_init = K.cast_to_floatx(beta_init)
         self.initial_weights = weights
         super(ParametricSoftplus, self).__init__(**kwargs)
 
-    def build(self):
-        input_shape = self.input_shape[1:]
-        self.alphas = K.variable(self.alpha_init * np.ones(input_shape))
-        self.betas = K.variable(self.beta_init * np.ones(input_shape))
-        self.params = [self.alphas, self.betas]
+    def build(self, input_shape):
+        input_shape = input_shape[1:]
+        self.alphas = K.variable(self.alpha_init * np.ones(input_shape),
+                                 name='{}_alphas'.format(self.name))
+        self.betas = K.variable(self.beta_init * np.ones(input_shape),
+                                name='{}_betas'.format(self.name))
+        self.trainable_weights = [self.alphas, self.betas]
 
         if self.initial_weights is not None:
             self.set_weights(self.initial_weights)
             del self.initial_weights
 
-    def get_output(self, train):
-        X = self.get_input(train)
-        return K.softplus(self.betas * X) * self.alphas
+    def call(self, x, mask=None):
+        return K.softplus(self.betas * x) * self.alphas
 
     def get_config(self):
-        config = {"name": self.__class__.__name__,
-                  "alpha_init": self.alpha_init,
-                  "beta_init": self.beta_init}
+        config = {'alpha_init': float(self.alpha_init),
+                  'beta_init': float(self.beta_init)}
         base_config = super(ParametricSoftplus, self).get_config()
         return dict(list(base_config.items()) + list(config.items()))
 
 
-class ThresholdedLinear(MaskedLayer):
-    '''Thresholded Linear Activation.
+class ThresholdedReLU(Layer):
+    '''Thresholded Rectified Linear Unit:
+    `f(x) = x for x > theta`
+    `f(x) = 0 otherwise`.
 
     # Input shape
         Arbitrary. Use the keyword argument `input_shape`
@@ -175,50 +184,83 @@ class ThresholdedLinear(MaskedLayer):
         theta: float >= 0. Threshold location of activation.
 
     # References
-        [Zero-Bias Autoencoders and the Benefits of Co-Adapting Features](http://arxiv.org/pdf/1402.3337.pdf)
-    '''
-    def __init__(self, theta=1.0, **kwargs):
-        super(ThresholdedLinear, self).__init__(**kwargs)
-        self.theta = theta
-
-    def get_output(self, train):
-        X = self.get_input(train)
-        return K.switch(K.abs(X) < self.theta, 0, X)
-
-    def get_config(self):
-        config = {"name": self.__class__.__name__,
-                  "theta": self.theta}
-        base_config = super(ThresholdedLinear, self).get_config()
-        return dict(list(base_config.items()) + list(config.items()))
-
-
-class ThresholdedReLU(MaskedLayer):
-    '''Thresholded Rectified Activation.
-
-    # Input shape
-        Arbitrary. Use the keyword argument `input_shape`
-        (tuple of integers, does not include the samples axis)
-        when using this layer as the first layer in a model.
-
-    # Output shape
-        Same shape as the input.
-
-    # Arguments
-        theta: float >= 0. Threshold location of activation.
-
-    # References
-        [Zero-Bias Autoencoders and the Benefits of Co-Adapting Features](http://arxiv.org/pdf/1402.3337.pdf)
+        - [Zero-Bias Autoencoders and the Benefits of Co-Adapting Features](http://arxiv.org/pdf/1402.3337.pdf)
     '''
     def __init__(self, theta=1.0, **kwargs):
+        self.supports_masking = True
+        self.theta = K.cast_to_floatx(theta)
         super(ThresholdedReLU, self).__init__(**kwargs)
-        self.theta = theta
 
-    def get_output(self, train):
-        X = self.get_input(train)
-        return K.switch(X > self.theta, X, 0)
+    def call(self, x, mask=None):
+        return x * K.cast(x > self.theta, K.floatx())
 
     def get_config(self):
-        config = {"name": self.__class__.__name__,
-                  "theta": self.theta}
+        config = {'theta': float(self.theta)}
         base_config = super(ThresholdedReLU, self).get_config()
         return dict(list(base_config.items()) + list(config.items()))
+
+
+class SReLU(Layer):
+    '''S-shaped Rectified Linear Unit.
+
+    # Input shape
+        Arbitrary. Use the keyword argument `input_shape`
+        (tuple of integers, does not include the samples axis)
+        when using this layer as the first layer in a model.
+
+    # Output shape
+        Same shape as the input.
+
+    # Arguments
+        t_left_init: initialization function for the left part intercept
+        a_left_init: initialization function for the left part slope
+        t_right_init: initialization function for the right part intercept
+        a_right_init: initialization function for the right part slope
+
+    # References
+        - [Deep Learning with S-shaped Rectified Linear Activation Units](http://arxiv.org/abs/1512.07030)
+    '''
+    def __init__(self, t_left_init='zero', a_left_init='glorot_uniform',
+                 t_right_init='glorot_uniform', a_right_init='one', **kwargs):
+        self.supports_masking = True
+        self.t_left_init = t_left_init
+        self.a_left_init = a_left_init
+        self.t_right_init = t_right_init
+        self.a_right_init = a_right_init
+        super(SReLU, self).__init__(**kwargs)
+
+    def build(self, input_shape):
+        input_shape = input_shape[1:]
+
+        t_left_init = initializations.get(self.t_left_init)
+        a_left_init = initializations.get(self.a_left_init)
+        t_right_init = initializations.get(self.t_right_init)
+        a_right_init = initializations.get(self.a_right_init)
+
+        self.t_left = t_left_init(input_shape,
+                                  name='{}_t_left'.format(self.name))
+        self.a_left = a_left_init(input_shape,
+                                  name='{}_a_left'.format(self.name))
+        self.t_right = t_right_init(input_shape,
+                                    name='{}_t_right'.format(self.name))
+        self.a_right = a_right_init(input_shape,
+                                    name='{}_a_right'.format(self.name))
+        # ensure the the right part is always to the right of the left
+        self.t_right_actual = self.t_left + abs(self.t_right)
+        self.trainable_weights = [self.t_left, self.a_left,
+                                  self.t_right, self.a_right]
+
+    def call(self, x, mask=None):
+        Y_left_and_center = self.t_left + K.relu(x - self.t_left,
+                                                 self.a_left,
+                                                 self.t_right_actual - self.t_left)
+        Y_right = K.relu(x - self.t_right_actual) * self.a_right
+        return Y_left_and_center + Y_right
+
+    def get_config(self):
+        config = {'t_left_init': self.t_left_init,
+                  'a_left_init': self.a_left_init,
+                  't_right_init': self.t_right_init,
+                  'a_right_init': self.a_right_init}
+        base_config = super(SReLU, self).get_config()
+        return dict(list(base_config.items()) + list(config.items()))

keras/layers/containers.py

@@ -1,529 +0,0 @@
-# -*- coding: utf-8 -*-
-from __future__ import absolute_import
-from __future__ import print_function
-
-from collections import OrderedDict
-from .. import backend as K
-from ..layers.core import Layer, Merge, Siamese, SiameseHead
-from six.moves import range
-
-
-class Sequential(Layer):
-    '''The Sequential container is a linear stack of layers.
-    Apart from the `add` methods and the `layers` constructor argument,
-    the API is identical to that of the `Layer` class.
-
-    This class is also the basis for the `keras.models.Sequential` model.
-
-    # Arguments
-        layers: list of layers to be added to the container.
-    '''
-    def __init__(self, layers=[]):
-        self.layers = []
-        self.layer_cache = {}
-        for layer in layers:
-            self.add(layer)
-        self._cache_enabled = True
-
-    def __call__(self, X, mask=None, train=False):
-        # turn off layer cache temporarily
-        tmp_cache_enabled = self.cache_enabled
-        self.cache_enabled = False
-        # recursively search for a layer which is not a Sequential model
-        layer = self
-        while issubclass(layer.__class__, Sequential):
-            layer = layer.layers[0]
-        # set temporary input to first layer
-        tmp_input = layer.get_input
-        tmp_mask = None
-        layer.get_input = lambda _: X
-        if hasattr(layer, 'get_input_mask'):
-            tmp_mask = layer.get_input_mask
-            layer.get_input_mask = lambda _: mask
-        Y = self.get_output(train=train)
-        # return input from first layer to what it was
-        layer.get_input = tmp_input
-        if hasattr(layer, 'get_input_mask'):
-            layer.get_input_mask = tmp_mask
-        self.cache_enabled = tmp_cache_enabled
-        return Y
-
-    @property
-    def cache_enabled(self):
-        return self._cache_enabled
-
-    @cache_enabled.setter
-    def cache_enabled(self, value):
-        self._cache_enabled = value
-        for l in self.layers:
-            l.cache_enabled = value
-
-    def set_previous(self, layer):
-        self.layers[0].previous = layer
-
-    def add(self, layer):
-        layer.layer_cache = self.layer_cache
-        self.layers.append(layer)
-        if len(self.layers) > 1:
-            self.layers[-1].set_previous(self.layers[-2])
-            if not hasattr(self.layers[0], 'input'):
-                self.set_input()
-
-    @property
-    def params(self):
-        params = []
-        for l in self.layers:
-            if l.trainable:
-                params += l.get_params()[0]
-        return params
-
-    @property
-    def regularizers(self):
-        regularizers = []
-        for l in self.layers:
-            if l.trainable:
-                regularizers += l.get_params()[1]
-        return regularizers
-
-    @property
-    def constraints(self):
-        constraints = []
-        for l in self.layers:
-            if l.trainable:
-                constraints += l.get_params()[2]
-        return constraints
-
-    @property
-    def updates(self):
-        updates = []
-        for l in self.layers:
-            if l.trainable:
-                updates += l.get_params()[3]
-        return updates
-
-    @property
-    def state_updates(self):
-        """
-        Return the `updates` from all layers in the sequence that are
-        stateful.  This is useful for separating _training_ updates and
-        _prediction_ updates for when we need to update a layers internal state
-        during a stateful prediction.
-        """
-        state_updates = []
-        for l in self.layers:
-            if getattr(l, 'stateful', False):
-                state_updates += l.get_params()[3]
-        return state_updates
-
-    def reset_states(self):
-        for l in self.layers:
-            if hasattr(l, 'reset_states') and getattr(l, 'stateful', False):
-                l.reset_states()
-
-    @property
-    def output_shape(self):
-        return self.layers[-1].output_shape
-
-    def get_output(self, train=False):
-        return self.layers[-1].get_output(train)
-
-    def set_input(self):
-        for l in self.layers:
-            if hasattr(l, 'input'):
-                ndim = K.ndim(l.input)
-                self.layers[0].input = K.placeholder(ndim=ndim)
-                break
-
-    def get_input(self, train=False):
-        if not hasattr(self.layers[0], 'input'):
-            self.set_input()
-        return self.layers[0].get_input(train)
-
-    @property
-    def input_shape(self):
-        return self.layers[0].input_shape
-
-    @property
-    def input(self):
-        return self.get_input()
-
-    def get_weights(self):
-        weights = []
-        for layer in self.layers:
-            weights += layer.get_weights()
-        return weights
-
-    def set_weights(self, weights):
-        for i in range(len(self.layers)):
-            nb_param = len(self.layers[i].params)
-            self.layers[i].set_weights(weights[:nb_param])
-            weights = weights[nb_param:]
-
-    def get_config(self):
-        return {'name': self.__class__.__name__,
-                'layers': [layer.get_config() for layer in self.layers]}
-
-    def count_params(self):
-        return sum([layer.count_params() for layer in self.layers])
-
-
-class Graph(Layer):
-    '''Implement a NN graph with arbitrary layer connections,
-    arbitrary number of inputs and arbitrary number of outputs.
-
-    This class is also the basis for the `keras.models.Graph` model.
-
-    Note: `Graph` can only be used as a layer
-    (connect, input, get_input, get_output)
-    when it has exactly one input and one output.
-    '''
-    def __init__(self):
-        self.namespace = set()  # strings
-        self.nodes = OrderedDict()  # layer-like
-        self.inputs = {}  # layer-like
-        self.input_order = []  # strings
-        self.outputs = {}  # layer-like
-        self.output_order = []  # strings
-        self.input_config = []  # dicts
-        self.output_config = []  # dicts
-        self.node_config = []  # dicts
-        self.layer_cache = {}
-
-    @property
-    def nb_input(self):
-        return len(self.inputs)
-
-    @property
-    def nb_output(self):
-        return len(self.outputs)
-
-    @property
-    def params(self):
-        params = []
-        for l in self.nodes.values():
-            if l.trainable:
-                params += l.get_params()[0]
-        return params
-
-    @property
-    def regularizers(self):
-        regularizers = []
-        for l in self.nodes.values():
-            if l.trainable:
-                regularizers += l.get_params()[1]
-        return regularizers
-
-    @property
-    def constraints(self):
-        constraints = []
-        for l in self.nodes.values():
-            if l.trainable:
-                constraints += l.get_params()[2]
-        return constraints
-
-    @property
-    def updates(self):
-        updates = []
-        for l in self.nodes.values():
-            if l.trainable:
-                updates += l.get_params()[3]
-        return updates
-
-    @property
-    def state_updates(self):
-        """
-        Return the `updates` from all nodes in that graph for nodes that are
-        stateful.  This is useful for separating _training_ updates and
-        _prediction_ updates for when we need to update a layers internal state
-        during a stateful prediction.
-        """
-        state_updates = []
-        for l in self.nodes.values():
-            if getattr(l, 'stateful', False):
-                state_updates += l.get_params()[3]
-        return state_updates
-
-    def reset_states(self):
-        for l in self.nodes.values():
-            if hasattr(l, 'reset_states') and getattr(l, 'stateful', False):
-                l.reset_states()
-
-    def set_previous(self, layer, connection_map={}):
-        if self.nb_input != layer.nb_output:
-            raise Exception('Cannot connect layers: '
-                            'input count does not match output count.')
-        if self.nb_input == 1:
-            self.inputs[self.input_order[0]].set_previous(layer)
-        else:
-            if not connection_map:
-                raise Exception('Cannot attach multi-input layer: '
-                                'no connection_map provided.')
-            for k, v in connection_map.items():
-                if k in self.inputs and v in layer.outputs:
-                    self.inputs[k].set_previous(layer.outputs[v])
-                else:
-                    raise Exception('Invalid connection map.')
-
-    def get_input(self, train=False):
-        if len(self.inputs) == len(self.outputs) == 1:
-            return self.inputs[self.input_order[0]].get_input(train)
-        else:
-            return dict([(k, v.get_input(train)) for k, v in self.inputs.items()])
-
-    @property
-    def input(self):
-        return self.get_input()
-
-    @property
-    def output_shape(self):
-        if self.nb_output == 1:
-            # return tuple
-            return self.outputs[self.output_order[0]].output_shape
-        else:
-            # return dictionary mapping output names to shape tuples
-            return dict([(k, v.output_shape) for k, v in self.outputs.items()])
-
-    def get_output(self, train=False):
-        if len(self.inputs) == len(self.outputs) == 1:
-            return self.outputs[self.output_order[0]].get_output(train)
-        else:
-            return dict([(k, v.get_output(train)) for k, v in self.outputs.items()])
-
-    def add_input(self, name, input_shape=None,
-                  batch_input_shape=None, dtype='float'):
-        '''Add an input to the graph.
-
-        # Arguments:
-            name: string. The name of the new input. Must be unique in the graph.
-            input_shape: a tuple of integers, the expected shape of the input samples.
-                Does not include the batch size.
-            batch_input_shape: a tuple of integers, the expected shape of the
-                whole input batch, including the batch size.
-            dtype: 'float' or 'int'.
-        '''
-        if name in self.namespace:
-            raise Exception('Duplicate node identifier: ' + name)
-        self.namespace.add(name)
-        self.input_order.append(name)
-        layer = Layer()  # empty layer
-        if input_shape:
-            layer.set_input_shape((None,) + tuple(input_shape))
-        elif batch_input_shape:
-            layer.set_input_shape(batch_input_shape)
-        if dtype == 'float':
-            layer.input = K.placeholder(shape=layer.input_shape, name=name)
-        else:
-            if (input_shape and len(input_shape) == 1) or (batch_input_shape and len(batch_input_shape) == 2):
-                layer.input = K.placeholder(shape=layer.input_shape,
-                                            dtype='int32',
-                                            name=name)
-            else:
-                raise Exception('Type "int" can only be used with ndim==2 (Embedding).')
-        self.inputs[name] = layer
-        self.input_config.append({'name': name,
-                                  'input_shape': input_shape,
-                                  'dtype': dtype})
-
-    def add_node(self, layer, name, input=None, inputs=[],
-                 merge_mode='concat', concat_axis=-1, dot_axes=-1,
-                 create_output=False):
-        '''Add a node in the graph. It can be connected to multiple
-        inputs, which will first be merged into one tensor
-        according to the mode specified.
-
-        # Arguments
-            layer: the layer at the node.
-            name: name for the node.
-            input: when connecting the layer to a single input,
-                this is the name of the incoming node.
-            inputs: when connecting the layer to multiple inputs,
-                this is a list of names of incoming nodes.
-            merge_mode: one of {concat, sum, dot, ave, mul}
-            concat_axis: when `merge_mode=='concat'`, this is the
-                input concatenation axis.
-            dot_axes: when `merge_mode='dot'`, this is the contraction axes
-                specification; see the `Merge layer for details.
-            create_output: boolean. Set this to `True` if you want the output
-                of your node to be an output of the graph.
-        '''
-        if name in self.namespace:
-            raise Exception('Duplicate node identifier: ' + name)
-        if input:
-            if input not in self.namespace:
-                raise Exception('Unknown node/input identifier: ' + input)
-            if input in self.nodes:
-                layer.set_previous(self.nodes[input])
-            elif input in self.inputs:
-                layer.set_previous(self.inputs[input])
-        if inputs:
-            to_merge = []
-            for n in inputs:
-                if n in self.nodes:
-                    to_merge.append(self.nodes[n])
-                elif n in self.inputs:
-                    to_merge.append(self.inputs[n])
-                else:
-                    raise Exception('Unknown identifier: ' + n)
-            merge = Merge(to_merge, mode=merge_mode,
-                          concat_axis=concat_axis, dot_axes=dot_axes)
-            layer.set_previous(merge)
-
-        self.namespace.add(name)
-        layer.layer_cache = self.layer_cache
-        self.nodes[name] = layer
-        self.node_config.append({'name': name,
-                                 'input': input,
-                                 'inputs': inputs,
-                                 'merge_mode': merge_mode,
-                                 'concat_axis': concat_axis,
-                                 'dot_axes': dot_axes,
-                                 'create_output': create_output})
-
-        if create_output:
-            self.add_output(name, input=name)
-
-    def add_shared_node(self, layer, name, inputs=[], merge_mode=None,
-                        concat_axis=-1, dot_axes=-1, outputs=[],
-                        create_output=False):
-        '''Used to share a same layer across multiple nodes.
-
-        Supposed, for instance, that you want to apply one same `Dense`
-        layer after to the output of two different nodes.
-        You can then add the `Dense` layer as a shared node.
-
-        # Arguments
-            layer: The layer to be shared across multiple inputs
-            name: Name of the shared node
-            inputs: List of names of input nodes
-            merge_mode: Same meaning as `merge_mode` argument of `add_node()`
-            concat_axis: Same meaning as `concat_axis` argument of `add_node()`
-            dot_axes: Same meaning as `dot_axes` argument of `add_node()`
-            outputs: Used when `merge_mode=None`. Names for the output nodes.
-            create_output: Same meaning as `create_output` argument of `add_node()`.
-        '''
-        if name in self.namespace:
-            raise Exception('Duplicate node identifier: ' + name)
-        for o in outputs:
-            if o in self.namespace:
-                raise Exception('Duplicate node identifier: ' + o)
-        if merge_mode:
-            if merge_mode not in {'sum', 'ave', 'mul', 'dot', 'cos', 'concat', 'join'}:
-                raise Exception('Invalid merge mode')
-        layers = []
-        for i in range(len(inputs)):
-            input = inputs[i]
-            if input in self.nodes:
-                n = self.nodes[input]
-                if n.__class__.__name__ == 'Siamese':
-                    if n.merge_mode is None:
-                        for j in range(len(n.inputs)):
-                            sh = SiameseHead(j)
-                            sh.previous = n
-                            layers.append(sh)
-                    else:
-                        layers.append(n)
-                else:
-                    layers.append(n)
-            elif input in self.inputs:
-                n = self.inputs[input]
-                layers.append(n)
-            else:
-                raise Exception('Unknown identifier: ' + input)
-        s = Siamese(layer, layers, merge_mode,
-                    concat_axis=concat_axis,
-                    dot_axes=dot_axes,
-                    is_graph=True)
-        self.namespace.add(name)
-        self.nodes[name] = s
-        self.node_config.append({'name': name,
-                                 'inputs': inputs,
-                                 'merge_mode': merge_mode,
-                                 'concat_axis': concat_axis,
-                                 'dot_axes': dot_axes,
-                                 'create_output': create_output if merge_mode else False})
-        if not merge_mode:
-            for i in range(len(outputs)):
-                sh = SiameseHead(i)
-                sh.previous = s
-                sh_name = outputs[i]
-                self.namespace.add(sh_name)
-                self.nodes[sh_name] = sh
-                self.node_config.append({'name': sh_name,
-                                         'inputs': [name],
-                                         'create_output': create_output})
-                if create_output:
-                    self.add_output(sh_name, input=sh_name)
-
-        if create_output and merge_mode:
-            if merge_mode == 'join':
-                raise Exception('Output can not be of type OrderedDict')
-            self.add_output(name, input=name)
-
-    def add_output(self, name, input=None, inputs=[],
-                   merge_mode='concat', concat_axis=-1, dot_axes=-1):
-        '''Add an output to the graph.
-
-        This output can merge several node outputs into a single output.
-
-        # Arguments
-            name: name of the output.
-            input: when connecting the layer to a single input,
-                this is the name of the incoming node.
-            inputs: when connecting the layer to multiple inputs,
-                this is a list of names of incoming nodes.
-            merge_mode: one of {concat, sum, dot, ave, mul}
-            concat_axis: when `merge_mode=='concat'`, this is the
-                input concatenation axis.
-            dot_axes: when `merge_mode='dot'`, this is the contraction axes
-                specification; see the `Merge layer for details.
-        '''
-        if name in self.output_order:
-            raise Exception('Duplicate output identifier: ' + name)
-        if input:
-            if input not in self.namespace:
-                raise Exception('Unknown node/input identifier: ' + input)
-            if input in self.nodes:
-                self.outputs[name] = self.nodes[input]
-            elif input in self.inputs:
-                self.outputs[name] = self.inputs[input]
-        if inputs:
-            to_merge = []
-            for n in inputs:
-                if n not in self.nodes:
-                    raise Exception('Unknown identifier: ' + n)
-                to_merge.append(self.nodes[n])
-            merge = Merge(to_merge, mode=merge_mode,
-                          concat_axis=concat_axis, dot_axes=dot_axes)
-            self.outputs[name] = merge
-
-        self.output_order.append(name)
-        self.output_config.append({'name': name,
-                                   'input': input,
-                                   'inputs': inputs,
-                                   'merge_mode': merge_mode,
-                                   'concat_axis': concat_axis,
-                                   'dot_axes': dot_axes})
-
-    def get_config(self):
-        return {'name': self.__class__.__name__,
-                'input_config': self.input_config,
-                'node_config': self.node_config,
-                'output_config': self.output_config,
-                'input_order': self.input_order,
-                'output_order': self.output_order,
-                'nodes': dict([(c['name'], self.nodes[c['name']].get_config()) for c in self.node_config])}
-
-    def count_params(self):
-        return sum([layer.count_params() for layer in self.nodes.values()])
-
-    def get_weights(self):
-        weights = []
-        for layer in self.nodes.values():
-            weights += layer.get_weights()
-        return weights
-
-    def set_weights(self, weights):
-        for layer in self.nodes.values():
-            nb_param = len(layer.get_weights())
-            layer.set_weights(weights[:nb_param])
-            weights = weights[nb_param:]

keras/layers/embeddings.py

@@ -1,10 +1,8 @@
 from __future__ import absolute_import
+
 from .. import backend as K
-
-from .. import activations, initializations, regularizers, constraints
-from ..layers.core import Layer, MaskedLayer
-
-from ..constraints import unitnorm
+from .. import initializations, regularizers, constraints
+from ..engine import Layer
 
 
 class Embedding(Layer):
@@ -13,21 +11,31 @@ class Embedding(Layer):
 
     This layer can only be used as the first layer in a model.
 
-    # Input shape
-        2D tensor with shape: `(nb_samples, sequence_length)`.
+    # Example
 
-    # Output shape
-        3D tensor with shape: `(nb_samples, sequence_length, output_dim)`.
+    ```python
+      model = Sequential()
+      model.add(Embedding(1000, 64, input_length=10))
+      # the model will take as input an integer matrix of size (batch, input_length).
+      # the largest integer (i.e. word index) in the input should be no larger than 999 (vocabulary size).
+      # now model.output_shape == (None, 10, 64), where None is the batch dimension.
+
+      input_array = np.random.randint(1000, size=(32, 10))
+
+      model.compile('rmsprop', 'mse')
+      output_array = model.predict(input_array)
+      assert output_array.shape == (32, 10, 64)
+    ```
 
     # Arguments
-      input_dim: int >= 0. Size of the vocabulary, ie.
+      input_dim: int > 0. Size of the vocabulary, ie.
           1 + maximum integer index occurring in the input data.
       output_dim: int >= 0. Dimension of the dense embedding.
       init: name of initialization function for the weights
           of the layer (see: [initializations](../initializations.md)),
           or alternatively, Theano function to use for weights initialization.
           This parameter is only relevant if you don't pass a `weights` argument.
-      weights: list of numpy arrays to set as initial weights.
+      weights: list of Numpy arrays to set as initial weights.
           The list should have 1 element, of shape `(input_dim, output_dim)`.
       W_regularizer: instance of the [regularizers](../regularizers.md) module
         (eg. L1 or L2 regularization), applied to the embedding matrix.
@@ -38,10 +46,22 @@ class Embedding(Layer):
           This is useful for [recurrent layers](recurrent.md) which may take
           variable length input. If this is `True` then all subsequent layers
           in the model need to support masking or an exception will be raised.
+          If mask_zero is set to True, as a consequence, index 0 cannot be
+          used in the vocabulary (input_dim should equal |vocabulary| + 2).
       input_length: Length of input sequences, when it is constant.
           This argument is required if you are going to connect
           `Flatten` then `Dense` layers upstream
           (without it, the shape of the dense outputs cannot be computed).
+      dropout: float between 0 and 1. Fraction of the embeddings to drop.
+
+    # Input shape
+        2D tensor with shape: `(nb_samples, sequence_length)`.
+
+    # Output shape
+        3D tensor with shape: `(nb_samples, sequence_length, output_dim)`.
+
+    # References
+        - [A Theoretically Grounded Application of Dropout in Recurrent Neural Networks](http://arxiv.org/abs/1512.05287)
     '''
     input_ndim = 2
 
@@ -50,28 +70,35 @@ class Embedding(Layer):
                  W_regularizer=None, activity_regularizer=None,
                  W_constraint=None,
                  mask_zero=False,
-                 weights=None, **kwargs):
+                 weights=None, dropout=0., **kwargs):
         self.input_dim = input_dim
         self.output_dim = output_dim
         self.init = initializations.get(init)
         self.input_length = input_length
         self.mask_zero = mask_zero
+        self.dropout = dropout
 
         self.W_constraint = constraints.get(W_constraint)
-        self.constraints = [self.W_constraint]
 
         self.W_regularizer = regularizers.get(W_regularizer)
         self.activity_regularizer = regularizers.get(activity_regularizer)
 
+        if 0. < self.dropout < 1.:
+            self.uses_learning_phase = True
         self.initial_weights = weights
-        kwargs['input_shape'] = (self.input_dim,)
+        kwargs['input_shape'] = (self.input_length,)
+        kwargs['input_dtype'] = 'int32'
         super(Embedding, self).__init__(**kwargs)
 
-    def build(self):
-        self.input = K.placeholder(shape=(self.input_shape[0], self.input_length),
-                                   dtype='int32')
-        self.W = self.init((self.input_dim, self.output_dim))
-        self.params = [self.W]
+    def build(self, input_shape):
+        self.W = self.init((self.input_dim, self.output_dim),
+                           name='{}_W'.format(self.name))
+        self.trainable_weights = [self.W]
+
+        self.constraints = {}
+        if self.W_constraint:
+            self.constraints[self.W] = self.W_constraint
+
         self.regularizers = []
         if self.W_regularizer:
             self.W_regularizer.set_param(self.W)
@@ -84,33 +111,41 @@ class Embedding(Layer):
         if self.initial_weights is not None:
             self.set_weights(self.initial_weights)
 
-    def get_output_mask(self, train=None):
-        X = self.get_input(train)
+    def compute_mask(self, x, mask=None):
         if not self.mask_zero:
             return None
         else:
-            if K._BACKEND == "tensorflow":
-                raise Exception("Masking is Theano-only for the time being.")
-            return K.ones_like(X) * (1 - K.equal(X, 0))
+            return K.not_equal(x, 0)
 
-    @property
-    def output_shape(self):
-        return (self.input_shape[0], self.input_length, self.output_dim)
+    def get_output_shape_for(self, input_shape):
+        if not self.input_length:
+            input_length = input_shape[1]
+        else:
+            input_length = self.input_length
+        return (input_shape[0], input_length, self.output_dim)
 
-    def get_output(self, train=False):
-        X = self.get_input(train)
-        out = K.gather(self.W, X)
+    def call(self, x, mask=None):
+        if K.dtype(x) != 'int32':
+            x = K.cast(x, 'int32')
+        if 0. < self.dropout < 1.:
+            retain_p = 1. - self.dropout
+            B = K.random_binomial((self.input_dim,), p=retain_p) * (1. / retain_p)
+            B = K.expand_dims(B)
+            W = K.in_train_phase(self.W * B, self.W)
+        else:
+            W = self.W
+        out = K.gather(W, x)
         return out
 
     def get_config(self):
-        config = {"name": self.__class__.__name__,
-                  "input_dim": self.input_dim,
-                  "output_dim": self.output_dim,
-                  "init": self.init.__name__,
-                  "input_length": self.input_length,
-                  "mask_zero": self.mask_zero,
-                  "activity_regularizer": self.activity_regularizer.get_config() if self.activity_regularizer else None,
-                  "W_regularizer": self.W_regularizer.get_config() if self.W_regularizer else None,
-                  "W_constraint": self.W_constraint.get_config() if self.W_constraint else None}
+        config = {'input_dim': self.input_dim,
+                  'output_dim': self.output_dim,
+                  'init': self.init.__name__,
+                  'input_length': self.input_length,
+                  'mask_zero': self.mask_zero,
+                  'activity_regularizer': self.activity_regularizer.get_config() if self.activity_regularizer else None,
+                  'W_regularizer': self.W_regularizer.get_config() if self.W_regularizer else None,
+                  'W_constraint': self.W_constraint.get_config() if self.W_constraint else None,
+                  'dropout': self.dropout}
         base_config = super(Embedding, self).get_config()
         return dict(list(base_config.items()) + list(config.items()))

keras/layers/local.py

@@ -0,0 +1,423 @@
+# -*- coding: utf-8 -*-
+from __future__ import absolute_import
+
+from keras import backend as K
+from keras.layers import activations, initializations, regularizers, constraints
+from keras.engine import Layer, InputSpec
+from ..utils.np_utils import conv_output_length
+
+
+class LocallyConnected1D(Layer):
+    '''LocallyConnected1D layer works almost the same as Convolution1D layer,
+    except that weights are unshared, that is, a different set of filters is
+    applied at each different patch of the input. When using this layer as the
+    first layer in a model, either provide the keyword argument `input_dim`
+    (int, e.g. 128 for sequences of 128-dimensional vectors), or `input_shape`
+    (tuple of integers, e.g. (10, 128) for sequences of 10 vectors of
+    128-dimensional vectors). Also, you will need to fix shape of the previous
+    layer, since the weights can only be defined with determined output shape.
+
+    # Example
+    ```python
+        # apply a unshared weight convolution 1d of length 3 to a sequence with
+        # 10 timesteps, with 64 output filters
+        model = Sequential()
+        model.add(LocallyConnected1D(64, 3, input_shape=(10, 32)))
+        # now model.output_shape == (None, 8, 64)
+        # add a new conv1d on top
+        model.add(LocallyConnected1D(32, 3))
+        # now model.output_shape == (None, 6, 32)
+    ```
+    # Arguments
+        nb_filter: Dimensionality of the output.
+        filter_length: The extension (spatial or temporal) of each filter.
+        init: name of initialization function for the weights of the layer
+            (see [initializations](../initializations.md)),
+            or alternatively, Theano function to use for weights initialization.
+            This parameter is only relevant if you don't pass a `weights` argument.
+        activation: name of activation function to use
+            (see [activations](../activations.md)),
+            or alternatively, elementwise Theano function.
+            If you don't specify anything, no activation is applied
+            (ie. "linear" activation: a(x) = x).
+        weights: list of numpy arrays to set as initial weights.
+        border_mode: Only support 'valid'. Please make good use of
+            ZeroPadding1D to achieve same output length.
+        subsample_length: factor by which to subsample output.
+        W_regularizer: instance of [WeightRegularizer](../regularizers.md)
+            (eg. L1 or L2 regularization), applied to the main weights matrix.
+        b_regularizer: instance of [WeightRegularizer](../regularizers.md),
+            applied to the bias.
+        activity_regularizer: instance of [ActivityRegularizer](../regularizers.md),
+            applied to the network output.
+        W_constraint: instance of the [constraints](../constraints.md) module
+            (eg. maxnorm, nonneg), applied to the main weights matrix.
+        b_constraint: instance of the [constraints](../constraints.md) module,
+            applied to the bias.
+        bias: whether to include a bias (i.e. make the layer affine rather than linear).
+        input_dim: Number of channels/dimensions in the input.
+            Either this argument or the keyword argument `input_shape`must be
+            provided when using this layer as the first layer in a model.
+        input_length: Length of input sequences, when it is constant.
+            This argument is required if you are going to connect
+            `Flatten` then `Dense` layers upstream
+            (without it, the shape of the dense outputs cannot be computed).
+    # Input shape
+        3D tensor with shape: `(samples, steps, input_dim)`.
+    # Output shape
+        3D tensor with shape: `(samples, new_steps, nb_filter)`.
+        `steps` value might have changed due to padding.
+    '''
+    def __init__(self, nb_filter, filter_length,
+                 init='uniform', activation='linear', weights=None,
+                 border_mode='valid', subsample_length=1,
+                 W_regularizer=None, b_regularizer=None, activity_regularizer=None,
+                 W_constraint=None, b_constraint=None,
+                 bias=True, input_dim=None, input_length=None, **kwargs):
+        if border_mode != 'valid':
+            raise Exception('Invalid border mode for LocallyConnected1D '
+                            '(only "valid" is supported):', border_mode)
+        self.nb_filter = nb_filter
+        self.filter_length = filter_length
+        self.init = initializations.get(init, dim_ordering='th')
+        self.activation = activations.get(activation)
+
+        self.border_mode = border_mode
+        self.subsample_length = subsample_length
+
+        self.W_regularizer = regularizers.get(W_regularizer)
+        self.b_regularizer = regularizers.get(b_regularizer)
+        self.activity_regularizer = regularizers.get(activity_regularizer)
+
+        self.W_constraint = constraints.get(W_constraint)
+        self.b_constraint = constraints.get(b_constraint)
+
+        self.bias = bias
+        self.input_spec = [InputSpec(ndim=3)]
+        self.initial_weights = weights
+        self.input_dim = input_dim
+        self.input_length = input_length
+        if self.input_dim:
+            kwargs['input_shape'] = (self.input_length, self.input_dim)
+        super(LocallyConnected1D, self).__init__(**kwargs)
+
+    def build(self, input_shape):
+        input_dim = input_shape[2]
+        _, output_length, nb_filter = self.get_output_shape_for(input_shape)
+
+        self.W_shape = (output_length, self.filter_length * input_dim, nb_filter)
+        self.W = self.init(self.W_shape, name='{}_W'.format(self.name))
+        if self.bias:
+            self.b = K.zeros((output_length, self.nb_filter), name='{}_b'.format(self.name))
+            self.trainable_weights = [self.W, self.b]
+        else:
+            self.trainable_weights = [self.W]
+
+        self.regularizers = []
+        if self.W_regularizer:
+            self.W_regularizer.set_param(self.W)
+            self.regularizers.append(self.W_regularizer)
+        if self.b_regularizer:
+            self.b_regularizer.set_param(self.b)
+            self.regularizers.append(self.b_regularizer)
+        if self.activity_regularizer:
+            self.activity_regularizer.set_layer(self)
+            self.regularizers.append(self.activity_regularizer)
+
+        self.constraints = {}
+        if self.W_constraint:
+            self.constraints[self.W] = self.W_constraint
+        if self.b_constraint:
+            self.constraints[self.b] = self.b_constraint
+
+        if self.initial_weights is not None:
+            self.set_weights(self.initial_weights)
+            del self.initial_weights
+
+    def get_output_shape_for(self, input_shape):
+        length = conv_output_length(input_shape[1],
+                                    self.filter_length,
+                                    self.border_mode,
+                                    self.subsample_length)
+        return (input_shape[0], length, self.nb_filter)
+
+    def call(self, x, mask=None):
+        stride = self.subsample_length
+        output_length, feature_dim, nb_filter = self.W_shape
+
+        xs = []
+        for i in range(output_length):
+            slice_length = slice(i * stride, i * stride + self.filter_length)
+            xs.append(K.reshape(x[:, slice_length, :], (1, -1, feature_dim)))
+        x_aggregate = K.concatenate(xs, axis=0)
+        # (output_length, batch_size, nb_filter)
+        output = K.batch_dot(x_aggregate, self.W)
+        output = K.permute_dimensions(output, (1, 0, 2))
+
+        if self.bias:
+            output += K.reshape(self.b, (1, output_length, nb_filter))
+
+        output = self.activation(output)
+        return output
+
+    def get_config(self):
+        config = {'nb_filter': self.nb_filter,
+                  'filter_length': self.filter_length,
+                  'init': self.init.__name__,
+                  'activation': self.activation.__name__,
+                  'border_mode': self.border_mode,
+                  'subsample_length': self.subsample_length,
+                  'W_regularizer': self.W_regularizer.get_config() if self.W_regularizer else None,
+                  'b_regularizer': self.b_regularizer.get_config() if self.b_regularizer else None,
+                  'activity_regularizer': self.activity_regularizer.get_config() if self.activity_regularizer else None,
+                  'W_constraint': self.W_constraint.get_config() if self.W_constraint else None,
+                  'b_constraint': self.b_constraint.get_config() if self.b_constraint else None,
+                  'bias': self.bias,
+                  'input_dim': self.input_dim,
+                  'input_length': self.input_length}
+        base_config = super(LocallyConnected1D, self).get_config()
+        return dict(list(base_config.items()) + list(config.items()))
+
+
+class LocallyConnected2D(Layer):
+    '''LocallyConnected2D layer works almost the same as Convolution2D layer,
+    except that weights are unshared, that is, a different set of filters is
+    applied at each different patch of the input. When using this layer as the
+    first layer in a model, provide the keyword argument `input_shape` (tuple
+    of integers, does not include the sample axis), e.g.
+    `input_shape=(3, 128, 128)` for 128x128 RGB pictures. Also, you will need
+    to fix shape of the previous layer, since the weights can only be defined
+    with determined output shape.
+
+    # Examples
+    ```python
+        # apply a 3x3 unshared weights convolution with 64 output filters on a 32x32 image:
+        model = Sequential()
+        model.add(LocallyConnected2D(64, 3, 3, input_shape=(3, 32, 32)))
+        # now model.output_shape == (None, 64, 30, 30)
+        # notice that this layer will consume (30*30)*(3*3*3*64) + (30*30)*64 parameters
+
+        # add a 3x3 unshared weights convolution on top, with 32 output filters:
+        model.add(LocallyConnected2D(32, 3, 3))
+        # now model.output_shape == (None, 32, 28, 28)
+    ```
+
+    # Arguments
+        nb_filter: Number of convolution filters to use.
+        nb_row: Number of rows in the convolution kernel.
+        nb_col: Number of columns in the convolution kernel.
+        init: name of initialization function for the weights of the layer
+            (see [initializations](../initializations.md)), or alternatively,
+            Theano function to use for weights initialization.
+            This parameter is only relevant if you don't pass
+            a `weights` argument.
+        activation: name of activation function to use
+            (see [activations](../activations.md)),
+            or alternatively, elementwise Theano function.
+            If you don't specify anything, no activation is applied
+            (ie. "linear" activation: a(x) = x).
+        weights: list of numpy arrays to set as initial weights.
+        border_mode: Only support 'valid'. Please make good use of
+            ZeroPadding2D to achieve same output shape.
+        subsample: tuple of length 2. Factor by which to subsample output.
+            Also called strides elsewhere.
+        W_regularizer: instance of [WeightRegularizer](../regularizers.md)
+            (eg. L1 or L2 regularization), applied to the main weights matrix.
+        b_regularizer: instance of [WeightRegularizer](../regularizers.md),
+            applied to the bias.
+        activity_regularizer: instance of [ActivityRegularizer](../regularizers.md),
+            applied to the network output.
+        W_constraint: instance of the [constraints](../constraints.md) module
+            (eg. maxnorm, nonneg), applied to the main weights matrix.
+        b_constraint: instance of the [constraints](../constraints.md) module,
+            applied to the bias.
+        dim_ordering: 'th' or 'tf'. In 'th' mode, the channels dimension
+            (the depth) is at index 1, in 'tf' mode is it at index 3.
+        bias: whether to include a bias (i.e. make the layer affine rather than linear).
+
+    # Input shape
+        4D tensor with shape:
+        `(samples, channels, rows, cols)` if dim_ordering='th'
+        or 4D tensor with shape:
+        `(samples, rows, cols, channels)` if dim_ordering='tf'.
+
+    # Output shape
+        4D tensor with shape:
+        `(samples, nb_filter, new_rows, new_cols)` if dim_ordering='th'
+        or 4D tensor with shape:
+        `(samples, new_rows, new_cols, nb_filter)` if dim_ordering='tf'.
+        `rows` and `cols` values might have changed due to padding.
+    '''
+    def __init__(self, nb_filter, nb_row, nb_col,
+                 init='glorot_uniform', activation='linear', weights=None,
+                 border_mode='valid', subsample=(1, 1),
+                 dim_ordering='default',
+                 W_regularizer=None, b_regularizer=None, activity_regularizer=None,
+                 W_constraint=None, b_constraint=None,
+                 bias=True, **kwargs):
+        if dim_ordering == 'default':
+            dim_ordering = K.image_dim_ordering()
+        if border_mode != 'valid':
+            raise Exception('Invalid border mode for LocallyConnected2D '
+                            '(only "valid" is supported):', border_mode)
+        self.nb_filter = nb_filter
+        self.nb_row = nb_row
+        self.nb_col = nb_col
+        self.init = initializations.get(init, dim_ordering=dim_ordering)
+        self.activation = activations.get(activation)
+
+        self.border_mode = border_mode
+        self.subsample = tuple(subsample)
+        assert dim_ordering in {'tf', 'th'}, 'dim_ordering must be in {tf, th}'
+        self.dim_ordering = dim_ordering
+
+        self.W_regularizer = regularizers.get(W_regularizer)
+        self.b_regularizer = regularizers.get(b_regularizer)
+        self.activity_regularizer = regularizers.get(activity_regularizer)
+
+        self.W_constraint = constraints.get(W_constraint)
+        self.b_constraint = constraints.get(b_constraint)
+
+        self.bias = bias
+        self.input_spec = [InputSpec(ndim=4)]
+        self.initial_weights = weights
+        super(LocallyConnected2D, self).__init__(**kwargs)
+
+    def build(self, input_shape):
+        output_shape = self.get_output_shape_for(input_shape)
+        if self.dim_ordering == 'th':
+            _, nb_filter, output_row, output_col = output_shape
+            input_filter = input_shape[1]
+        elif self.dim_ordering == 'tf':
+            _, output_row, output_col, nb_filter = output_shape
+            input_filter = input_shape[3]
+        else:
+            raise Exception('Invalid dim_ordering: ' + self.dim_ordering)
+
+        self.output_row = output_row
+        self.output_col = output_col
+        self.W_shape = (output_row * output_col, self.nb_row * self.nb_col * input_filter, nb_filter)
+        self.W = self.init(self.W_shape, name='{}_W'.format(self.name))
+
+        if self.bias:
+            self.b = K.zeros((output_row, output_col, nb_filter), name='{}_b'.format(self.name))
+            self.trainable_weights = [self.W, self.b]
+        else:
+            self.trainable_weights = [self.W]
+
+        self.regularizers = []
+        if self.W_regularizer:
+            self.W_regularizer.set_param(self.W)
+            self.regularizers.append(self.W_regularizer)
+        if self.bias and self.b_regularizer:
+            self.b_regularizer.set_param(self.b)
+            self.regularizers.append(self.b_regularizer)
+        if self.activity_regularizer:
+            self.activity_regularizer.set_layer(self)
+            self.regularizers.append(self.activity_regularizer)
+
+        self.constraints = {}
+        if self.W_constraint:
+            self.constraints[self.W] = self.W_constraint
+        if self.bias and self.b_constraint:
+            self.constraints[self.b] = self.b_constraint
+
+        if self.initial_weights is not None:
+            self.set_weights(self.initial_weights)
+            del self.initial_weights
+
+    def get_output_shape_for(self, input_shape):
+        if self.dim_ordering == 'th':
+            rows = input_shape[2]
+            cols = input_shape[3]
+        elif self.dim_ordering == 'tf':
+            rows = input_shape[1]
+            cols = input_shape[2]
+        else:
+            raise Exception('Invalid dim_ordering: ' + self.dim_ordering)
+
+        rows = conv_output_length(rows, self.nb_row,
+                                  self.border_mode, self.subsample[0])
+        cols = conv_output_length(cols, self.nb_col,
+                                  self.border_mode, self.subsample[1])
+
+        if self.dim_ordering == 'th':
+            return (input_shape[0], self.nb_filter, rows, cols)
+        elif self.dim_ordering == 'tf':
+            return (input_shape[0], rows, cols, self.nb_filter)
+        else:
+            raise Exception('Invalid dim_ordering: ' + self.dim_ordering)
+
+    def call(self, x, mask=None):
+        stride_row, stride_col = self.subsample
+        _, feature_dim, nb_filter = self.W_shape
+
+        if self.dim_ordering == 'th':
+            if K._backend == 'theano':
+                output = []
+                for i in range(self.output_row):
+                    for j in range(self.output_col):
+                        slice_row = slice(i * stride_row,
+                                          i * stride_row + self.nb_row)
+                        slice_col = slice(j * stride_col,
+                                          j * stride_col + self.nb_col)
+                        x_flatten = K.reshape(x[:, :, slice_row, slice_col], (1, -1, feature_dim))
+                        output.append(K.dot(x_flatten, self.W[i * self.output_col + j, :, :]))
+                output = K.concatenate(output, axis=0)
+            else:
+                xs = []
+                for i in range(self.output_row):
+                    for j in range(self.output_col):
+                        slice_row = slice(i * stride_row,
+                                          i * stride_row + self.nb_row)
+                        slice_col = slice(j * stride_col,
+                                          j * stride_col + self.nb_col)
+                        xs.append(K.reshape(x[:, :, slice_row, slice_col], (1, -1, feature_dim)))
+                x_aggregate = K.concatenate(xs, axis=0)
+                output = K.batch_dot(x_aggregate, self.W)
+            output = K.reshape(output, (self.output_row, self.output_col, -1, nb_filter))
+            output = K.permute_dimensions(output, (2, 3, 0, 1))
+        elif self.dim_ordering == 'tf':
+            xs = []
+            for i in range(self.output_row):
+                for j in range(self.output_col):
+                    slice_row = slice(i * stride_row,
+                                      i * stride_row + self.nb_row)
+                    slice_col = slice(j * stride_col,
+                                      j * stride_col + self.nb_col)
+                    xs.append(K.reshape(x[:, slice_row, slice_col, :], (1, -1, feature_dim)))
+            x_aggregate = K.concatenate(xs, axis=0)
+            output = K.batch_dot(x_aggregate, self.W)
+            output = K.reshape(output, (self.output_row, self.output_col, -1, nb_filter))
+            output = K.permute_dimensions(output, (2, 0, 1, 3))
+        else:
+            raise Exception('Invalid dim_ordering: ' + self.dim_ordering)
+
+        if self.bias:
+            if self.dim_ordering == 'th':
+                output += K.reshape(self.b, (1, nb_filter, self.output_row, self.output_col))
+            elif self.dim_ordering == 'tf':
+                output += K.reshape(self.b, (1, self.output_row, self.output_col, nb_filter))
+            else:
+                raise Exception('Invalid dim_ordering: ' + self.dim_ordering)
+
+        output = self.activation(output)
+        return output
+
+    def get_config(self):
+        config = {'nb_filter': self.nb_filter,
+                  'nb_row': self.nb_row,
+                  'nb_col': self.nb_col,
+                  'init': self.init.__name__,
+                  'activation': self.activation.__name__,
+                  'border_mode': self.border_mode,
+                  'subsample': self.subsample,
+                  'dim_ordering': self.dim_ordering,
+                  'W_regularizer': self.W_regularizer.get_config() if self.W_regularizer else None,
+                  'b_regularizer': self.b_regularizer.get_config() if self.b_regularizer else None,
+                  'activity_regularizer': self.activity_regularizer.get_config() if self.activity_regularizer else None,
+                  'W_constraint': self.W_constraint.get_config() if self.W_constraint else None,
+                  'b_constraint': self.b_constraint.get_config() if self.b_constraint else None,
+                  'bias': self.bias}
+        base_config = super(LocallyConnected2D, self).get_config()
+        return dict(list(base_config.items()) + list(config.items()))

keras/layers/noise.py

@@ -1,10 +1,11 @@
 from __future__ import absolute_import
-from .core import MaskedLayer
+from ..engine import Layer
 from .. import backend as K
+import numpy as np
 
 
-class GaussianNoise(MaskedLayer):
-    '''Apply to the input an additive zero-centred gaussian noise with
+class GaussianNoise(Layer):
+    '''Apply to the input an additive zero-centered Gaussian noise with
     standard deviation `sigma`. This is useful to mitigate overfitting
     (you could see it as a kind of random data augmentation).
     Gaussian Noise (GS) is a natural choice as corruption process
@@ -12,6 +13,44 @@ class GaussianNoise(MaskedLayer):
 
     As it is a regularization layer, it is only active at training time.
 
+    # Arguments
+        sigma: float, standard deviation of the noise distribution.
+
+    # Input shape
+        Arbitrary. Use the keyword argument `input_shape`
+        (tuple of integers, does not include the samples axis)
+        when using this layer as the first layer in a model.
+
+    # Output shape
+        Same shape as input.
+    '''
+    def __init__(self, sigma, **kwargs):
+        self.supports_masking = True
+        self.sigma = sigma
+        self.uses_learning_phase = True
+        super(GaussianNoise, self).__init__(**kwargs)
+
+    def call(self, x, mask=None):
+        noise_x = x + K.random_normal(shape=K.shape(x),
+                                      mean=0.,
+                                      std=self.sigma)
+        return K.in_train_phase(noise_x, x)
+
+    def get_config(self):
+        config = {'sigma': self.sigma}
+        base_config = super(GaussianNoise, self).get_config()
+        return dict(list(base_config.items()) + list(config.items()))
+
+
+class GaussianDropout(Layer):
+    '''Apply to the input an multiplicative one-centered Gaussian noise
+    with standard deviation `sqrt(p/(1-p))`.
+
+    As it is a regularization layer, it is only active at training time.
+
+    # Arguments
+        p: float, drop probability (as with `Dropout`).
+
     # Input shape
         Arbitrary. Use the keyword argument `input_shape`
         (tuple of integers, does not include the samples axis)
@@ -20,56 +59,24 @@ class GaussianNoise(MaskedLayer):
     # Output shape
         Same shape as input.
 
-    # Arguments
-        sigma: float, standard deviation of the noise distribution.
-    '''
-    def __init__(self, sigma, **kwargs):
-        super(GaussianNoise, self).__init__(**kwargs)
-        self.sigma = sigma
-
-    def get_output(self, train=False):
-        X = self.get_input(train)
-        if not train or self.sigma == 0:
-            return X
-        else:
-            return X + K.random_normal(shape=K.shape(X),
-                                       mean=0.,
-                                       std=self.sigma)
-
-    def get_config(self):
-        config = {"name": self.__class__.__name__,
-                  "sigma": self.sigma}
-        base_config = super(GaussianNoise, self).get_config()
-        return dict(list(base_config.items()) + list(config.items()))
-
-
-class GaussianDropout(MaskedLayer):
-    '''Apply to the input an multiplicative one-centred gaussian noise
-    with standard deviation `sqrt(p/(1-p))`.
-
-    As it is a regularization layer, it is only active at training time.
-
-    # Arguments
-        p: float, drop probability (as with `Dropout`).
-
-    # References:
+    # References
         [Dropout: A Simple Way to Prevent Neural Networks from Overfitting Srivastava, Hinton, et al. 2014](http://www.cs.toronto.edu/~rsalakhu/papers/srivastava14a.pdf)
     '''
     def __init__(self, p, **kwargs):
-        super(GaussianDropout, self).__init__(**kwargs)
+        self.supports_masking = True
         self.p = p
+        if 0 < p < 1:
+            self.uses_learning_phase = True
+        super(GaussianDropout, self).__init__(**kwargs)
 
-    def get_output(self, train):
-        X = self.get_input(train)
-        if train:
-            # self.p refers to drop probability rather than
-            # retain probability (as in paper), for consistency
-            X *= K.random_normal(shape=K.shape(X), mean=1.0,
-                                 std=self.p / (1.0 - self.p))
-        return X
+    def call(self, x, mask=None):
+        if 0 < self.p < 1:
+            noise_x = x * K.random_normal(shape=K.shape(x), mean=1.0,
+                                          std=np.sqrt(self.p / (1.0 - self.p)))
+            return K.in_train_phase(noise_x, x)
+        return x
 
     def get_config(self):
-        config = {"name": self.__class__.__name__,
-                  "p": self.p}
+        config = {'p': self.p}
         base_config = super(GaussianDropout, self).get_config()
         return dict(list(base_config.items()) + list(config.items()))

keras/layers/normalization.py

@@ -1,4 +1,4 @@
-from ..layers.core import Layer
+from ..engine import Layer, InputSpec
 from .. import initializations
 from .. import backend as K
 
@@ -8,6 +8,43 @@ class BatchNormalization(Layer):
     i.e. applies a transformation that maintains the mean activation
     close to 0 and the activation standard deviation close to 1.
 
+    # Arguments
+        epsilon: small float > 0. Fuzz parameter.
+        mode: integer, 0, 1 or 2.
+            - 0: feature-wise normalization.
+                Each feature map in the input will
+                be normalized separately. The axis on which
+                to normalize is specified by the `axis` argument.
+                Note that if the input is a 4D image tensor
+                using Theano conventions (samples, channels, rows, cols)
+                then you should set `axis` to `1` to normalize along
+                the channels axis.
+                During training we use per-batch statistics to normalize
+                the data, and during testing we use running averages
+                computed during the training phase.
+            - 1: sample-wise normalization. This mode assumes a 2D input.
+            - 2: feature-wise normalization, like mode 0, but
+                using per-batch statistics to normalize the data during both
+                testing and training.
+        axis: integer, axis along which to normalize in mode 0. For instance,
+            if your input tensor has shape (samples, channels, rows, cols),
+            set axis to 1 to normalize per feature map (channels axis).
+        momentum: momentum in the computation of the
+            exponential average of the mean and standard deviation
+            of the data, for feature-wise normalization.
+        weights: Initialization weights.
+            List of 2 Numpy arrays, with shapes:
+            `[(input_shape,), (input_shape,)]`
+            Note that the order of this list is [gamma, beta, mean, std]
+        beta_init: name of initialization function for shift parameter
+            (see [initializations](../initializations.md)), or alternatively,
+            Theano/TensorFlow function to use for weights initialization.
+            This parameter is only relevant if you don't pass a `weights` argument.
+        gamma_init: name of initialization function for scale parameter (see
+            [initializations](../initializations.md)), or alternatively,
+            Theano/TensorFlow function to use for weights initialization.
+            This parameter is only relevant if you don't pass a `weights` argument.
+
     # Input shape
         Arbitrary. Use the keyword argument `input_shape`
         (tuple of integers, does not include the samples axis)
@@ -16,84 +53,108 @@ class BatchNormalization(Layer):
     # Output shape
         Same shape as input.
 
-    # Arguments
-        epsilon: small float > 0. Fuzz parameter.
-        mode: integer, 0 or 1.
-            - 0: feature-wise normalization.
-                If the input has multiple feature dimensions,
-                each will be normalized separately
-                (e.g. for an image input with shape
-                `(channels, rows, cols)`,
-                each combination of a channel, row and column
-                will be normalized separately).
-            - 1: sample-wise normalization. This mode assumes a 2D input.
-        momentum: momentum in the computation of the
-            exponential average of the mean and standard deviation
-            of the data, for feature-wise normalization.
-        weights: Initialization weights.
-            List of 2 numpy arrays, with shapes:
-            `[(input_shape,), (input_shape,)]`
-
     # References
-        - [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](http://arxiv.org/pdf/1502.03167v3.pdf)
+        - [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](http://jmlr.org/proceedings/papers/v37/ioffe15.html)
     '''
-    def __init__(self, epsilon=1e-6, mode=0, momentum=0.9,
-                 weights=None, **kwargs):
-        self.init = initializations.get("uniform")
+    def __init__(self, epsilon=1e-6, mode=0, axis=-1, momentum=0.99,
+                 weights=None, beta_init='zero', gamma_init='one', **kwargs):
+        self.supports_masking = True
+        self.beta_init = initializations.get(beta_init)
+        self.gamma_init = initializations.get(gamma_init)
         self.epsilon = epsilon
         self.mode = mode
+        self.axis = axis
         self.momentum = momentum
         self.initial_weights = weights
+        if self.mode == 0:
+            self.uses_learning_phase = True
         super(BatchNormalization, self).__init__(**kwargs)
 
-    def build(self):
-        input_shape = self.input_shape  # starts with samples axis
-        input_shape = input_shape[1:]
+    def build(self, input_shape):
+        self.input_spec = [InputSpec(shape=input_shape)]
+        shape = (input_shape[self.axis],)
 
-        self.gamma = self.init(input_shape)
-        self.beta = K.zeros(input_shape)
+        self.gamma = self.gamma_init(shape, name='{}_gamma'.format(self.name))
+        self.beta = self.beta_init(shape, name='{}_beta'.format(self.name))
+        self.trainable_weights = [self.gamma, self.beta]
 
-        self.params = [self.gamma, self.beta]
-        self.running_mean = K.zeros(input_shape)
-        self.running_std = K.ones(input_shape)
+        self.running_mean = K.zeros(shape,
+                                    name='{}_running_mean'.format(self.name))
+        self.running_std = K.ones(shape,
+                                  name='{}_running_std'.format(self.name))
+        self.non_trainable_weights = [self.running_mean, self.running_std]
 
         if self.initial_weights is not None:
             self.set_weights(self.initial_weights)
             del self.initial_weights
+        self.built = True
+        self.called_with = None
 
-    def get_weights(self):
-        super_weights = super(BatchNormalization, self).get_weights()
-        return super_weights + [K.get_value(self.running_mean),
-                                K.get_value(self.running_std)]
+    def call(self, x, mask=None):
+        if self.mode == 0 or self.mode == 2:
+            assert self.built, 'Layer must be built before being called'
+            input_shape = self.input_spec[0].shape
 
-    def set_weights(self, weights):
-        K.set_value(self.running_mean, weights[-2])
-        K.set_value(self.running_std, weights[-1])
-        super(BatchNormalization, self).set_weights(weights[:-2])
+            reduction_axes = list(range(len(input_shape)))
+            del reduction_axes[self.axis]
+            broadcast_shape = [1] * len(input_shape)
+            broadcast_shape[self.axis] = input_shape[self.axis]
+
+            if self.mode == 2:
+                x_normed, mean, std = K.normalize_batch_in_training(
+                    x, self.gamma, self.beta, reduction_axes,
+                    epsilon=self.epsilon)
+            else:
+                # mode 0
+                if self.called_with not in {None, x}:
+                    raise Exception('You are attempting to share a '
+                                    'same `BatchNormalization` layer across '
+                                    'different data flows. '
+                                    'This is not possible. '
+                                    'You should use `mode=2` in '
+                                    '`BatchNormalization`, which has '
+                                    'a similar behavior but is shareable '
+                                    '(see docs for a description of '
+                                    'the behavior).')
+                self.called_with = x
+                x_normed, mean, std = K.normalize_batch_in_training(
+                    x, self.gamma, self.beta, reduction_axes,
+                    epsilon=self.epsilon)
+
+                self.updates = [K.moving_average_update(self.running_mean, mean, self.momentum),
+                                K.moving_average_update(self.running_std, std, self.momentum)]
+
+                if sorted(reduction_axes) == range(K.ndim(x))[:-1]:
+                    x_normed_running = K.batch_normalization(
+                        x, self.running_mean, self.running_std,
+                        self.beta, self.gamma,
+                        epsilon=self.epsilon)
+                else:
+                    # need broadcasting
+                    broadcast_running_mean = K.reshape(self.running_mean, broadcast_shape)
+                    broadcast_running_std = K.reshape(self.running_std, broadcast_shape)
+                    broadcast_beta = K.reshape(self.beta, broadcast_shape)
+                    broadcast_gamma = K.reshape(self.gamma, broadcast_shape)
+                    x_normed_running = K.batch_normalization(
+                        x, broadcast_running_mean, broadcast_running_std,
+                        broadcast_beta, broadcast_gamma,
+                        epsilon=self.epsilon)
+
+                # pick the normalized form of x corresponding to the training phase
+                x_normed = K.in_train_phase(x_normed, x_normed_running)
 
-    def get_output(self, train):
-        X = self.get_input(train)
-        if self.mode == 0:
-            m = K.mean(X, axis=0)
-            std = K.mean(K.square(X - m) + self.epsilon, axis=0)
-            std = K.sqrt(std)
-            mean_update = self.momentum * self.running_mean + (1-self.momentum) * m
-            std_update = self.momentum * self.running_std + (1-self.momentum) * std
-            self.updates = [(self.running_mean, mean_update),
-                            (self.running_std, std_update)]
-            X_normed = ((X - self.running_mean) /
-                        (self.running_std + self.epsilon))
         elif self.mode == 1:
-            m = K.mean(X, axis=-1, keepdims=True)
-            std = K.std(X, axis=-1, keepdims=True)
-            X_normed = (X - m) / (std + self.epsilon)
-        out = self.gamma * X_normed + self.beta
-        return out
+            # sample-wise normalization
+            m = K.mean(x, axis=-1, keepdims=True)
+            std = K.sqrt(K.var(x, axis=-1, keepdims=True) + self.epsilon)
+            x_normed = (x - m) / (std + self.epsilon)
+            x_normed = self.gamma * x_normed + self.beta
+        return x_normed
 
     def get_config(self):
-        config = {"name": self.__class__.__name__,
-                  "epsilon": self.epsilon,
+        config = {"epsilon": self.epsilon,
                   "mode": self.mode,
+                  "axis": self.axis,
                   "momentum": self.momentum}
         base_config = super(BatchNormalization, self).get_config()
         return dict(list(base_config.items()) + list(config.items()))

keras/layers/pooling.py

@@ -0,0 +1,400 @@
+# -*- coding: utf-8 -*-
+from __future__ import absolute_import
+
+from .. import backend as K
+from ..engine import Layer, InputSpec
+from ..utils.np_utils import conv_output_length
+
+
+class _Pooling1D(Layer):
+    '''Abstract class for different pooling 1D layers.
+    '''
+    input_dim = 3
+
+    def __init__(self, pool_length=2, stride=None,
+                 border_mode='valid', **kwargs):
+        super(_Pooling1D, self).__init__(**kwargs)
+        if stride is None:
+            stride = pool_length
+        self.pool_length = pool_length
+        self.stride = stride
+        self.st = (self.stride, 1)
+        self.pool_size = (pool_length, 1)
+        assert border_mode in {'valid', 'same'}, 'border_mode must be in {valid, same}'
+        self.border_mode = border_mode
+        self.input_spec = [InputSpec(ndim=3)]
+
+    def get_output_shape_for(self, input_shape):
+        length = conv_output_length(input_shape[1], self.pool_length,
+                                    self.border_mode, self.stride)
+        return (input_shape[0], length, input_shape[2])
+
+    def _pooling_function(self, back_end, inputs, pool_size, strides,
+                          border_mode, dim_ordering):
+        raise NotImplementedError
+
+    def call(self, x, mask=None):
+        x = K.expand_dims(x, -1)   # add dummy last dimension
+        x = K.permute_dimensions(x, (0, 2, 1, 3))
+        output = self._pooling_function(inputs=x, pool_size=self.pool_size,
+                                        strides=self.st,
+                                        border_mode=self.border_mode,
+                                        dim_ordering='th')
+        output = K.permute_dimensions(output, (0, 2, 1, 3))
+        return K.squeeze(output, 3)  # remove dummy last dimension
+
+    def get_config(self):
+        config = {'stride': self.stride,
+                  'pool_length': self.pool_length,
+                  'border_mode': self.border_mode}
+        base_config = super(_Pooling1D, self).get_config()
+        return dict(list(base_config.items()) + list(config.items()))
+
+
+class MaxPooling1D(_Pooling1D):
+    '''Max pooling operation for temporal data.
+
+    # Input shape
+        3D tensor with shape: `(samples, steps, features)`.
+
+    # Output shape
+        3D tensor with shape: `(samples, downsampled_steps, features)`.
+
+    # Arguments
+        pool_length: size of the region to which max pooling is applied
+        stride: integer, or None. factor by which to downscale.
+            2 will halve the input.
+            If None, it will default to `pool_length`.
+        border_mode: 'valid' or 'same'.
+            Note: 'same' will only work with TensorFlow for the time being.
+    '''
+
+    def __init__(self, pool_length=2, stride=None,
+                 border_mode='valid', **kwargs):
+        super(MaxPooling1D, self).__init__(pool_length, stride,
+                                           border_mode, **kwargs)
+
+    def _pooling_function(self, inputs, pool_size, strides,
+                          border_mode, dim_ordering):
+        output = K.pool2d(inputs, pool_size, strides,
+                          border_mode, dim_ordering, pool_mode='max')
+        return output
+
+
+class AveragePooling1D(_Pooling1D):
+    '''Average pooling for temporal data.
+
+    # Arguments
+        pool_length: factor by which to downscale. 2 will halve the input.
+        stride: integer, or None. Stride value.
+            If None, it will default to `pool_length`.
+        border_mode: 'valid' or 'same'.
+            Note: 'same' will only work with TensorFlow for the time being.
+
+    # Input shape
+        3D tensor with shape: `(samples, steps, features)`.
+
+    # Output shape
+        3D tensor with shape: `(samples, downsampled_steps, features)`.
+    '''
+
+    def __init__(self, pool_length=2, stride=None,
+                 border_mode='valid', **kwargs):
+        super(AveragePooling1D, self).__init__(pool_length, stride,
+                                               border_mode, **kwargs)
+
+    def _pooling_function(self, inputs, pool_size, strides,
+                          border_mode, dim_ordering):
+        output = K.pool2d(inputs, pool_size, strides,
+                          border_mode, dim_ordering, pool_mode='avg')
+        return output
+
+
+class _Pooling2D(Layer):
+    '''Abstract class for different pooling 2D layers.
+    '''
+
+    def __init__(self, pool_size=(2, 2), strides=None, border_mode='valid',
+                 dim_ordering='default', **kwargs):
+        super(_Pooling2D, self).__init__(**kwargs)
+        if dim_ordering == 'default':
+            dim_ordering = K.image_dim_ordering()
+        self.pool_size = tuple(pool_size)
+        if strides is None:
+            strides = self.pool_size
+        self.strides = tuple(strides)
+        assert border_mode in {'valid', 'same'}, 'border_mode must be in {valid, same}'
+        self.border_mode = border_mode
+        assert dim_ordering in {'tf', 'th'}, 'dim_ordering must be in {tf, th}'
+        self.dim_ordering = dim_ordering
+        self.input_spec = [InputSpec(ndim=4)]
+
+    def get_output_shape_for(self, input_shape):
+        if self.dim_ordering == 'th':
+            rows = input_shape[2]
+            cols = input_shape[3]
+        elif self.dim_ordering == 'tf':
+            rows = input_shape[1]
+            cols = input_shape[2]
+        else:
+            raise Exception('Invalid dim_ordering: ' + self.dim_ordering)
+
+        rows = conv_output_length(rows, self.pool_size[0],
+                                  self.border_mode, self.strides[0])
+        cols = conv_output_length(cols, self.pool_size[1],
+                                  self.border_mode, self.strides[1])
+
+        if self.dim_ordering == 'th':
+            return (input_shape[0], input_shape[1], rows, cols)
+        elif self.dim_ordering == 'tf':
+            return (input_shape[0], rows, cols, input_shape[3])
+        else:
+            raise Exception('Invalid dim_ordering: ' + self.dim_ordering)
+
+    def _pooling_function(self, inputs, pool_size, strides,
+                          border_mode, dim_ordering):
+        raise NotImplementedError
+
+    def call(self, x, mask=None):
+        output = self._pooling_function(inputs=x, pool_size=self.pool_size,
+                                        strides=self.strides,
+                                        border_mode=self.border_mode,
+                                        dim_ordering=self.dim_ordering)
+        return output
+
+    def get_config(self):
+        config = {'pool_size': self.pool_size,
+                  'border_mode': self.border_mode,
+                  'strides': self.strides,
+                  'dim_ordering': self.dim_ordering}
+        base_config = super(_Pooling2D, self).get_config()
+        return dict(list(base_config.items()) + list(config.items()))
+
+
+class MaxPooling2D(_Pooling2D):
+    '''Max pooling operation for spatial data.
+
+    # Arguments
+        pool_size: tuple of 2 integers,
+            factors by which to downscale (vertical, horizontal).
+            (2, 2) will halve the image in each dimension.
+        strides: tuple of 2 integers, or None. Strides values.
+            If None, it will default to `pool_size`.
+        border_mode: 'valid' or 'same'.
+            Note: 'same' will only work with TensorFlow for the time being.
+        dim_ordering: 'th' or 'tf'. In 'th' mode, the channels dimension
+            (the depth) is at index 1, in 'tf' mode is it at index 3.
+            It defaults to the `image_dim_ordering` value found in your
+            Keras config file at `~/.keras/keras.json`.
+            If you never set it, then it will be "th".
+
+    # Input shape
+        4D tensor with shape:
+        `(samples, channels, rows, cols)` if dim_ordering='th'
+        or 4D tensor with shape:
+        `(samples, rows, cols, channels)` if dim_ordering='tf'.
+
+    # Output shape
+        4D tensor with shape:
+        `(nb_samples, channels, pooled_rows, pooled_cols)` if dim_ordering='th'
+        or 4D tensor with shape:
+        `(samples, pooled_rows, pooled_cols, channels)` if dim_ordering='tf'.
+    '''
+
+    def __init__(self, pool_size=(2, 2), strides=None, border_mode='valid',
+                 dim_ordering='default', **kwargs):
+        super(MaxPooling2D, self).__init__(pool_size, strides, border_mode,
+                                           dim_ordering, **kwargs)
+
+    def _pooling_function(self, inputs, pool_size, strides,
+                          border_mode, dim_ordering):
+        output = K.pool2d(inputs, pool_size, strides,
+                          border_mode, dim_ordering, pool_mode='max')
+        return output
+
+
+class AveragePooling2D(_Pooling2D):
+    '''Average pooling operation for spatial data.
+
+    # Arguments
+        pool_size: tuple of 2 integers,
+            factors by which to downscale (vertical, horizontal).
+            (2, 2) will halve the image in each dimension.
+        strides: tuple of 2 integers, or None. Strides values.
+            If None, it will default to `pool_size`.
+        border_mode: 'valid' or 'same'.
+            Note: 'same' will only work with TensorFlow for the time being.
+        dim_ordering: 'th' or 'tf'. In 'th' mode, the channels dimension
+            (the depth) is at index 1, in 'tf' mode is it at index 3.
+            It defaults to the `image_dim_ordering` value found in your
+            Keras config file at `~/.keras/keras.json`.
+            If you never set it, then it will be "th".
+
+    # Input shape
+        4D tensor with shape:
+        `(samples, channels, rows, cols)` if dim_ordering='th'
+        or 4D tensor with shape:
+        `(samples, rows, cols, channels)` if dim_ordering='tf'.
+
+    # Output shape
+        4D tensor with shape:
+        `(nb_samples, channels, pooled_rows, pooled_cols)` if dim_ordering='th'
+        or 4D tensor with shape:
+        `(samples, pooled_rows, pooled_cols, channels)` if dim_ordering='tf'.
+    '''
+
+    def __init__(self, pool_size=(2, 2), strides=None, border_mode='valid',
+                 dim_ordering='default', **kwargs):
+        super(AveragePooling2D, self).__init__(pool_size, strides, border_mode,
+                                               dim_ordering, **kwargs)
+
+    def _pooling_function(self, inputs, pool_size, strides,
+                          border_mode, dim_ordering):
+        output = K.pool2d(inputs, pool_size, strides,
+                          border_mode, dim_ordering, pool_mode='avg')
+        return output
+
+
+class _Pooling3D(Layer):
+    '''Abstract class for different pooling 3D layers.
+    '''
+
+    def __init__(self, pool_size=(2, 2, 2), strides=None, border_mode='valid',
+                 dim_ordering='default', **kwargs):
+        super(_Pooling3D, self).__init__(**kwargs)
+        if dim_ordering == 'default':
+            dim_ordering = K.image_dim_ordering()
+        self.pool_size = tuple(pool_size)
+        if strides is None:
+            strides = self.pool_size
+        self.strides = tuple(strides)
+        assert border_mode in {'valid', 'same'}, 'border_mode must be in {valid, same}'
+        self.border_mode = border_mode
+        assert dim_ordering in {'tf', 'th'}, 'dim_ordering must be in {tf, th}'
+        self.dim_ordering = dim_ordering
+        self.input_spec = [InputSpec(ndim=5)]
+
+    def get_output_shape_for(self, input_shape):
+        if self.dim_ordering == 'th':
+            len_dim1 = input_shape[2]
+            len_dim2 = input_shape[3]
+            len_dim3 = input_shape[4]
+        elif self.dim_ordering == 'tf':
+            len_dim1 = input_shape[1]
+            len_dim2 = input_shape[2]
+            len_dim3 = input_shape[3]
+        else:
+            raise Exception('Invalid dim_ordering: ' + self.dim_ordering)
+
+        len_dim1 = conv_output_length(len_dim1, self.pool_size[0],
+                                      self.border_mode, self.strides[0])
+        len_dim2 = conv_output_length(len_dim2, self.pool_size[1],
+                                      self.border_mode, self.strides[1])
+        len_dim3 = conv_output_length(len_dim3, self.pool_size[2],
+                                      self.border_mode, self.strides[2])
+
+        if self.dim_ordering == 'th':
+            return (input_shape[0], input_shape[1], len_dim1, len_dim2, len_dim3)
+        elif self.dim_ordering == 'tf':
+            return (input_shape[0], len_dim1, len_dim2, len_dim3, input_shape[4])
+        else:
+            raise Exception('Invalid dim_ordering: ' + self.dim_ordering)
+
+    def _pooling_function(self, inputs, pool_size, strides,
+                          border_mode, dim_ordering):
+        raise NotImplementedError
+
+    def call(self, x, mask=None):
+        output = self._pooling_function(inputs=x, pool_size=self.pool_size,
+                                        strides=self.strides,
+                                        border_mode=self.border_mode,
+                                        dim_ordering=self.dim_ordering)
+        return output
+
+    def get_config(self):
+        config = {'pool_size': self.pool_size,
+                  'border_mode': self.border_mode,
+                  'strides': self.strides,
+                  'dim_ordering': self.dim_ordering}
+        base_config = super(_Pooling3D, self).get_config()
+        return dict(list(base_config.items()) + list(config.items()))
+
+
+class MaxPooling3D(_Pooling3D):
+    '''Max pooling operation for 3D data (spatial or spatio-temporal).
+
+    # Arguments
+        pool_size: tuple of 3 integers,
+            factors by which to downscale (dim1, dim2, dim3).
+            (2, 2, 2) will halve the size of the 3D input in each dimension.
+        strides: tuple of 3 integers, or None. Strides values.
+        border_mode: 'valid' or 'same'.
+        dim_ordering: 'th' or 'tf'. In 'th' mode, the channels dimension
+            (the depth) is at index 1, in 'tf' mode is it at index 4.
+            It defaults to the `image_dim_ordering` value found in your
+            Keras config file at `~/.keras/keras.json`.
+            If you never set it, then it will be "th".
+
+    # Input shape
+        5D tensor with shape:
+        `(samples, channels, len_pool_dim1, len_pool_dim2, len_pool_dim3)` if dim_ordering='th'
+        or 5D tensor with shape:
+        `(samples, len_pool_dim1, len_pool_dim2, len_pool_dim3, channels)` if dim_ordering='tf'.
+
+    # Output shape
+        5D tensor with shape:
+        `(nb_samples, channels, pooled_dim1, pooled_dim2, pooled_dim3)` if dim_ordering='th'
+        or 5D tensor with shape:
+        `(samples, pooled_dim1, pooled_dim2, pooled_dim3, channels)` if dim_ordering='tf'.
+    '''
+
+    def __init__(self, pool_size=(2, 2, 2), strides=None, border_mode='valid',
+                 dim_ordering='default', **kwargs):
+        super(MaxPooling3D, self).__init__(pool_size, strides, border_mode,
+                                           dim_ordering, **kwargs)
+
+    def _pooling_function(self, inputs, pool_size, strides,
+                          border_mode, dim_ordering):
+        output = K.pool3d(inputs, pool_size, strides,
+                          border_mode, dim_ordering, pool_mode='max')
+        return output
+
+
+class AveragePooling3D(_Pooling3D):
+    '''Average pooling operation for 3D data (spatial or spatio-temporal).
+
+    # Arguments
+        pool_size: tuple of 3 integers,
+            factors by which to downscale (dim1, dim2, dim3).
+            (2, 2, 2) will halve the size of the 3D input in each dimension.
+        strides: tuple of 3 integers, or None. Strides values.
+        border_mode: 'valid' or 'same'.
+        dim_ordering: 'th' or 'tf'. In 'th' mode, the channels dimension
+            (the depth) is at index 1, in 'tf' mode is it at index 4.
+            It defaults to the `image_dim_ordering` value found in your
+            Keras config file at `~/.keras/keras.json`.
+            If you never set it, then it will be "th".
+
+    # Input shape
+        5D tensor with shape:
+        `(samples, channels, len_pool_dim1, len_pool_dim2, len_pool_dim3)` if dim_ordering='th'
+        or 5D tensor with shape:
+        `(samples, len_pool_dim1, len_pool_dim2, len_pool_dim3, channels)` if dim_ordering='tf'.
+
+    # Output shape
+        5D tensor with shape:
+        `(nb_samples, channels, pooled_dim1, pooled_dim2, pooled_dim3)` if dim_ordering='th'
+        or 5D tensor with shape:
+        `(samples, pooled_dim1, pooled_dim2, pooled_dim3, channels)` if dim_ordering='tf'.
+    '''
+
+    def __init__(self, pool_size=(2, 2, 2), strides=None, border_mode='valid',
+                 dim_ordering='default', **kwargs):
+        super(AveragePooling3D, self).__init__(pool_size, strides, border_mode,
+                                               dim_ordering, **kwargs)
+
+    def _pooling_function(self, inputs, pool_size, strides,
+                          border_mode, dim_ordering):
+        output = K.pool3d(inputs, pool_size, strides,
+                          border_mode, dim_ordering, pool_mode='avg')
+        return output

keras/layers/recurrent.py

@@ -3,28 +3,70 @@ from __future__ import absolute_import
 import numpy as np
 
 from .. import backend as K
-from .. import activations, initializations
-from ..layers.core import MaskedLayer
+from .. import activations, initializations, regularizers
+from ..engine import Layer, InputSpec
 
 
-class Recurrent(MaskedLayer):
+def time_distributed_dense(x, w, b=None, dropout=None,
+                           input_dim=None, output_dim=None, timesteps=None):
+    '''Apply y.w + b for every temporal slice y of x.
+    '''
+    if not input_dim:
+        # won't work with TensorFlow
+        input_dim = K.shape(x)[2]
+    if not timesteps:
+        # won't work with TensorFlow
+        timesteps = K.shape(x)[1]
+    if not output_dim:
+        # won't work with TensorFlow
+        output_dim = K.shape(w)[1]
+
+    if dropout is not None and 0. < dropout < 1.:
+        # apply the same dropout pattern at every timestep
+        ones = K.ones_like(K.reshape(x[:, 0, :], (-1, input_dim)))
+        dropout_matrix = K.dropout(ones, dropout)
+        expanded_dropout_matrix = K.repeat(dropout_matrix, timesteps)
+        x = K.in_train_phase(x * expanded_dropout_matrix, x)
+
+    # collapse time dimension and batch dimension together
+    x = K.reshape(x, (-1, input_dim))
+
+    x = K.dot(x, w)
+    if b:
+        x = x + b
+    # reshape to 3D tensor
+    x = K.reshape(x, (-1, timesteps, output_dim))
+    return x
+
+
+class Recurrent(Layer):
     '''Abstract base class for recurrent layers.
-    Do not use in a model -- it's not a functional layer!
+    Do not use in a model -- it's not a valid layer!
+    Use its children classes `LSTM`, `GRU` and `SimpleRNN` instead.
 
-    All recurrent layers (GRU, LSTM, SimpleRNN) also
+    All recurrent layers (`LSTM`, `GRU`, `SimpleRNN`) also
     follow the specifications of this class and accept
     the keyword arguments listed below.
 
-    # Input shape
-        3D tensor with shape `(nb_samples, timesteps, input_dim)`.
+    # Example
 
-    # Output shape
-        - if `return_sequences`: 3D tensor with shape
-            `(nb_samples, timesteps, output_dim)`.
-        - else, 2D tensor with shape `(nb_samples, output_dim)`.
+    ```python
+        # as the first layer in a Sequential model
+        model = Sequential()
+        model.add(LSTM(32, input_shape=(10, 64)))
+        # now model.output_shape == (None, 32)
+        # note: `None` is the batch dimension.
+
+        # the following is identical:
+        model = Sequential()
+        model.add(LSTM(32, input_dim=64, input_length=10))
+
+        # for subsequent layers, not need to specify the input size:
+        model.add(LSTM(16))
+    ```
 
     # Arguments
-        weights: list of numpy arrays to set as initial weights.
+        weights: list of Numpy arrays to set as initial weights.
             The list should have 3 elements, of shapes:
             `[(input_dim, output_dim), (output_dim, output_dim), (output_dim,)]`.
         return_sequences: Boolean. Whether to return the last output
@@ -34,6 +76,23 @@ class Recurrent(MaskedLayer):
         stateful: Boolean (default False). If True, the last state
             for each sample at index i in a batch will be used as initial
             state for the sample of index i in the following batch.
+        unroll: Boolean (default False). If True, the network will be unrolled,
+            else a symbolic loop will be used. When using TensorFlow, the network
+            is always unrolled, so this argument does not do anything.
+            Unrolling can speed-up a RNN, although it tends to be more memory-intensive.
+            Unrolling is only suitable for short sequences.
+        consume_less: one of "cpu", "mem", or "gpu" (LSTM/GRU only).
+            If set to "cpu", the RNN will use
+            an implementation that uses fewer, larger matrix products,
+            thus running faster on CPU but consuming more memory.
+            If set to "mem", the RNN will use more matrix products,
+            but smaller ones, thus running slower (may actually be faster on GPU)
+            while consuming less memory.
+            If set to "gpu" (LSTM/GRU only), the RNN will combine the input gate,
+            the forget gate and the output gate into a single matrix,
+            enabling more time-efficient parallelization on the GPU. Note: RNN
+            dropout must be shared for all gates, resulting in a slightly
+            reduced regularization.
         input_dim: dimensionality of the input (integer).
             This argument (or alternatively, the keyword argument `input_shape`)
             is required when using this layer as the first layer in a model.
@@ -47,12 +106,19 @@ class Recurrent(MaskedLayer):
             at the level of the first layer
             (e.g. via the `input_shape` argument)
 
+    # Input shape
+        3D tensor with shape `(nb_samples, timesteps, input_dim)`.
+
+    # Output shape
+        - if `return_sequences`: 3D tensor with shape
+            `(nb_samples, timesteps, output_dim)`.
+        - else, 2D tensor with shape `(nb_samples, output_dim)`.
+
     # Masking
         This layer supports masking for input data with a variable number
         of timesteps. To introduce masks to your data,
         use an [Embedding](embeddings.md) layer with the `mask_zero` parameter
         set to `True`.
-        **Note:** for the time being, masking is only supported with Theano.
 
     # TensorFlow warning
         For the time being, when using the TensorFlow backend,
@@ -73,83 +139,101 @@ class Recurrent(MaskedLayer):
         To enable statefulness:
             - specify `stateful=True` in the layer constructor.
             - specify a fixed batch size for your model, by passing
-                a `batch_input_shape=(...)` to the first layer in your model.
+                if sequential model:
+                  a `batch_input_shape=(...)` to the first layer in your model.
+                else for functional model with 1 or more Input layers:
+                  a `batch_shape=(...)` to all the first layers in your model.
                 This is the expected shape of your inputs *including the batch size*.
                 It should be a tuple of integers, e.g. `(32, 10, 100)`.
 
         To reset the states of your model, call `.reset_states()` on either
         a specific layer, or on your entire model.
-    '''
-    input_ndim = 3
 
+    # Note on using dropout with TensorFlow
+        When using the TensorFlow backend, specify a fixed batch size for your model
+        following the notes on statefulness RNNs.
+    '''
     def __init__(self, weights=None,
                  return_sequences=False, go_backwards=False, stateful=False,
+                 unroll=False, consume_less='cpu',
                  input_dim=None, input_length=None, **kwargs):
         self.return_sequences = return_sequences
         self.initial_weights = weights
         self.go_backwards = go_backwards
         self.stateful = stateful
+        self.unroll = unroll
+        self.consume_less = consume_less
 
+        self.supports_masking = True
+        self.input_spec = [InputSpec(ndim=3)]
         self.input_dim = input_dim
         self.input_length = input_length
         if self.input_dim:
             kwargs['input_shape'] = (self.input_length, self.input_dim)
         super(Recurrent, self).__init__(**kwargs)
 
-    def get_output_mask(self, train=False):
-        if self.return_sequences:
-            return super(Recurrent, self).get_output_mask(train)
-        else:
-            return None
-
-    @property
-    def output_shape(self):
-        input_shape = self.input_shape
+    def get_output_shape_for(self, input_shape):
         if self.return_sequences:
             return (input_shape[0], input_shape[1], self.output_dim)
         else:
             return (input_shape[0], self.output_dim)
 
+    def compute_mask(self, input, mask):
+        if self.return_sequences:
+            return mask
+        else:
+            return None
+
     def step(self, x, states):
         raise NotImplementedError
 
-    def get_initial_states(self, X):
+    def get_constants(self, x):
+        return []
+
+    def get_initial_states(self, x):
         # build an all-zero tensor of shape (samples, output_dim)
-        initial_state = K.zeros_like(X)  # (samples, timesteps, input_dim)
-        initial_state = K.sum(initial_state, axis=1)  # (samples, input_dim)
-        reducer = K.zeros((self.input_dim, self.output_dim))
-        initial_state = K.dot(initial_state, reducer)  # (samples, output_dim)
+        initial_state = K.zeros_like(x)  # (samples, timesteps, input_dim)
+        initial_state = K.sum(initial_state, axis=(1, 2))  # (samples,)
+        initial_state = K.expand_dims(initial_state)  # (samples, 1)
+        initial_state = K.tile(initial_state, [1, self.output_dim])  # (samples, output_dim)
         initial_states = [initial_state for _ in range(len(self.states))]
         return initial_states
 
-    def get_output(self, train=False):
+    def preprocess_input(self, x):
+        return x
+
+    def call(self, x, mask=None):
         # input shape: (nb_samples, time (padded with zeros), input_dim)
-        X = self.get_input(train)
-        assert K.ndim(X) == 3
+        # note that the .build() method of subclasses MUST define
+        # self.input_spec with a complete input shape.
+        input_shape = self.input_spec[0].shape
         if K._BACKEND == 'tensorflow':
-            if not self.input_shape[1]:
-                raise Exception('When using TensorFlow, you should define ' +
-                                'explicitly the number of timesteps of ' +
-                                'your sequences. Make sure the first layer ' +
-                                'has a "batch_input_shape" argument ' +
-                                'including the samples axis.')
-
-        mask = self.get_output_mask(train)
-        if mask:
-            # apply mask
-            X *= K.cast(K.expand_dims(mask), X.dtype)
-            masking = True
-        else:
-            masking = False
-
+            if not input_shape[1]:
+                raise Exception('When using TensorFlow, you should define '
+                                'explicitly the number of timesteps of '
+                                'your sequences.\n'
+                                'If your first layer is an Embedding, '
+                                'make sure to pass it an "input_length" '
+                                'argument. Otherwise, make sure '
+                                'the first layer has '
+                                'an "input_shape" or "batch_input_shape" '
+                                'argument, including the time axis. '
+                                'Found input shape at layer ' + self.name +
+                                ': ' + str(input_shape))
         if self.stateful:
             initial_states = self.states
         else:
-            initial_states = self.get_initial_states(X)
+            initial_states = self.get_initial_states(x)
+        constants = self.get_constants(x)
+        preprocessed_input = self.preprocess_input(x)
 
-        last_output, outputs, states = K.rnn(self.step, X, initial_states,
+        last_output, outputs, states = K.rnn(self.step, preprocessed_input,
+                                             initial_states,
                                              go_backwards=self.go_backwards,
-                                             masking=masking)
+                                             mask=mask,
+                                             constants=constants,
+                                             unroll=self.unroll,
+                                             input_length=input_shape[1])
         if self.stateful:
             self.updates = []
             for i in range(len(states)):
@@ -161,18 +245,23 @@ class Recurrent(MaskedLayer):
             return last_output
 
     def get_config(self):
-        config = {"name": self.__class__.__name__,
-                  "return_sequences": self.return_sequences,
-                  "input_dim": self.input_dim,
-                  "input_length": self.input_length,
-                  "go_backwards": self.go_backwards,
-                  "stateful": self.stateful}
+        config = {'return_sequences': self.return_sequences,
+                  'go_backwards': self.go_backwards,
+                  'stateful': self.stateful,
+                  'unroll': self.unroll,
+                  'consume_less': self.consume_less}
+        if self.stateful:
+            config['batch_input_shape'] = self.input_spec[0].shape
+        else:
+            config['input_dim'] = self.input_dim
+            config['input_length'] = self.input_length
+
         base_config = super(Recurrent, self).get_config()
         return dict(list(base_config.items()) + list(config.items()))
 
 
 class SimpleRNN(Recurrent):
-    '''Fully-connected RNN where the output is to fed back to input.
+    '''Fully-connected RNN where the output is to be fed back to input.
 
     # Arguments
         output_dim: dimension of the internal projections and the final output.
@@ -183,18 +272,38 @@ class SimpleRNN(Recurrent):
         activation: activation function.
             Can be the name of an existing function (str),
             or a Theano function (see: [activations](../activations.md)).
+        W_regularizer: instance of [WeightRegularizer](../regularizers.md)
+            (eg. L1 or L2 regularization), applied to the input weights matrices.
+        U_regularizer: instance of [WeightRegularizer](../regularizers.md)
+            (eg. L1 or L2 regularization), applied to the recurrent weights matrices.
+        b_regularizer: instance of [WeightRegularizer](../regularizers.md),
+            applied to the bias.
+        dropout_W: float between 0 and 1. Fraction of the input units to drop for input gates.
+        dropout_U: float between 0 and 1. Fraction of the input units to drop for recurrent connections.
+
+    # References
+        - [A Theoretically Grounded Application of Dropout in Recurrent Neural Networks](http://arxiv.org/abs/1512.05287)
     '''
     def __init__(self, output_dim,
                  init='glorot_uniform', inner_init='orthogonal',
-                 activation='sigmoid', **kwargs):
+                 activation='tanh',
+                 W_regularizer=None, U_regularizer=None, b_regularizer=None,
+                 dropout_W=0., dropout_U=0., **kwargs):
         self.output_dim = output_dim
         self.init = initializations.get(init)
         self.inner_init = initializations.get(inner_init)
         self.activation = activations.get(activation)
+        self.W_regularizer = regularizers.get(W_regularizer)
+        self.U_regularizer = regularizers.get(U_regularizer)
+        self.b_regularizer = regularizers.get(b_regularizer)
+        self.dropout_W, self.dropout_U = dropout_W, dropout_U
+
+        if self.dropout_W or self.dropout_U:
+            self.uses_learning_phase = True
         super(SimpleRNN, self).__init__(**kwargs)
 
-    def build(self):
-        input_shape = self.input_shape
+    def build(self, input_shape):
+        self.input_spec = [InputSpec(shape=input_shape)]
         if self.stateful:
             self.reset_states()
         else:
@@ -203,10 +312,24 @@ class SimpleRNN(Recurrent):
         input_dim = input_shape[2]
         self.input_dim = input_dim
 
-        self.W = self.init((input_dim, self.output_dim))
-        self.U = self.inner_init((self.output_dim, self.output_dim))
-        self.b = K.zeros((self.output_dim,))
-        self.params = [self.W, self.U, self.b]
+        self.W = self.init((input_dim, self.output_dim),
+                           name='{}_W'.format(self.name))
+        self.U = self.inner_init((self.output_dim, self.output_dim),
+                                 name='{}_U'.format(self.name))
+        self.b = K.zeros((self.output_dim,), name='{}_b'.format(self.name))
+
+        self.regularizers = []
+        if self.W_regularizer:
+            self.W_regularizer.set_param(self.W)
+            self.regularizers.append(self.W_regularizer)
+        if self.U_regularizer:
+            self.U_regularizer.set_param(self.U)
+            self.regularizers.append(self.U_regularizer)
+        if self.b_regularizer:
+            self.b_regularizer.set_param(self.b)
+            self.regularizers.append(self.b_regularizer)
+
+        self.trainable_weights = [self.W, self.U, self.b]
 
         if self.initial_weights is not None:
             self.set_weights(self.initial_weights)
@@ -214,30 +337,70 @@ class SimpleRNN(Recurrent):
 
     def reset_states(self):
         assert self.stateful, 'Layer must be stateful.'
-        input_shape = self.input_shape
+        input_shape = self.input_spec[0].shape
         if not input_shape[0]:
             raise Exception('If a RNN is stateful, a complete ' +
-                            'input_shape must be provided ' +
-                            '(including batch size).')
+                            'input_shape must be provided (including batch size).')
         if hasattr(self, 'states'):
             K.set_value(self.states[0],
                         np.zeros((input_shape[0], self.output_dim)))
         else:
             self.states = [K.zeros((input_shape[0], self.output_dim))]
 
+    def preprocess_input(self, x):
+        if self.consume_less == 'cpu':
+            input_shape = self.input_spec[0].shape
+            input_dim = input_shape[2]
+            timesteps = input_shape[1]
+            return time_distributed_dense(x, self.W, self.b, self.dropout_W,
+                                          input_dim, self.output_dim,
+                                          timesteps)
+        else:
+            return x
+
     def step(self, x, states):
-        # states only contains the previous output.
-        assert len(states) == 1
         prev_output = states[0]
-        h = K.dot(x, self.W) + self.b
-        output = self.activation(h + K.dot(prev_output, self.U))
+        B_U = states[1]
+        B_W = states[2]
+
+        if self.consume_less == 'cpu':
+            h = x
+        else:
+            h = K.dot(x * B_W, self.W) + self.b
+
+        output = self.activation(h + K.dot(prev_output * B_U, self.U))
         return output, [output]
 
+    def get_constants(self, x):
+        constants = []
+        if 0 < self.dropout_U < 1:
+            ones = K.ones_like(K.reshape(x[:, 0, 0], (-1, 1)))
+            ones = K.concatenate([ones] * self.output_dim, 1)
+            B_U = K.in_train_phase(K.dropout(ones, self.dropout_U), ones)
+            constants.append(B_U)
+        else:
+            constants.append(K.cast_to_floatx(1.))
+        if self.consume_less == 'cpu' and 0 < self.dropout_W < 1:
+            input_shape = self.input_spec[0].shape
+            input_dim = input_shape[-1]
+            ones = K.ones_like(K.reshape(x[:, 0, 0], (-1, 1)))
+            ones = K.concatenate([ones] * input_dim, 1)
+            B_W = K.in_train_phase(K.dropout(ones, self.dropout_W), ones)
+            constants.append(B_W)
+        else:
+            constants.append(K.cast_to_floatx(1.))
+        return constants
+
     def get_config(self):
-        config = {"output_dim": self.output_dim,
-                  "init": self.init.__name__,
-                  "inner_init": self.inner_init.__name__,
-                  "activation": self.activation.__name__}
+        config = {'output_dim': self.output_dim,
+                  'init': self.init.__name__,
+                  'inner_init': self.inner_init.__name__,
+                  'activation': self.activation.__name__,
+                  'W_regularizer': self.W_regularizer.get_config() if self.W_regularizer else None,
+                  'U_regularizer': self.U_regularizer.get_config() if self.U_regularizer else None,
+                  'b_regularizer': self.b_regularizer.get_config() if self.b_regularizer else None,
+                  'dropout_W': self.dropout_W,
+                  'dropout_U': self.dropout_U}
         base_config = super(SimpleRNN, self).get_config()
         return dict(list(base_config.items()) + list(config.items()))
 
@@ -255,43 +418,42 @@ class GRU(Recurrent):
             Can be the name of an existing function (str),
             or a Theano function (see: [activations](../activations.md)).
         inner_activation: activation function for the inner cells.
+        W_regularizer: instance of [WeightRegularizer](../regularizers.md)
+            (eg. L1 or L2 regularization), applied to the input weights matrices.
+        U_regularizer: instance of [WeightRegularizer](../regularizers.md)
+            (eg. L1 or L2 regularization), applied to the recurrent weights matrices.
+        b_regularizer: instance of [WeightRegularizer](../regularizers.md),
+            applied to the bias.
+        dropout_W: float between 0 and 1. Fraction of the input units to drop for input gates.
+        dropout_U: float between 0 and 1. Fraction of the input units to drop for recurrent connections.
 
     # References
         - [On the Properties of Neural Machine Translation: Encoder–Decoder Approaches](http://www.aclweb.org/anthology/W14-4012)
         - [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](http://arxiv.org/pdf/1412.3555v1.pdf)
+        - [A Theoretically Grounded Application of Dropout in Recurrent Neural Networks](http://arxiv.org/abs/1512.05287)
     '''
     def __init__(self, output_dim,
                  init='glorot_uniform', inner_init='orthogonal',
-                 activation='sigmoid', inner_activation='hard_sigmoid',
-                 **kwargs):
+                 activation='tanh', inner_activation='hard_sigmoid',
+                 W_regularizer=None, U_regularizer=None, b_regularizer=None,
+                 dropout_W=0., dropout_U=0., **kwargs):
         self.output_dim = output_dim
         self.init = initializations.get(init)
         self.inner_init = initializations.get(inner_init)
         self.activation = activations.get(activation)
         self.inner_activation = activations.get(inner_activation)
+        self.W_regularizer = regularizers.get(W_regularizer)
+        self.U_regularizer = regularizers.get(U_regularizer)
+        self.b_regularizer = regularizers.get(b_regularizer)
+        self.dropout_W, self.dropout_U = dropout_W, dropout_U
+
+        if self.dropout_W or self.dropout_U:
+            self.uses_learning_phase = True
         super(GRU, self).__init__(**kwargs)
 
-    def build(self):
-        input_shape = self.input_shape
-        input_dim = input_shape[2]
-        self.input_dim = input_dim
-        self.input = K.placeholder(input_shape)
-
-        self.W_z = self.init((input_dim, self.output_dim))
-        self.U_z = self.inner_init((self.output_dim, self.output_dim))
-        self.b_z = K.zeros((self.output_dim,))
-
-        self.W_r = self.init((input_dim, self.output_dim))
-        self.U_r = self.inner_init((self.output_dim, self.output_dim))
-        self.b_r = K.zeros((self.output_dim,))
-
-        self.W_h = self.init((input_dim, self.output_dim))
-        self.U_h = self.inner_init((self.output_dim, self.output_dim))
-        self.b_h = K.zeros((self.output_dim,))
-
-        self.params = [self.W_z, self.U_z, self.b_z,
-                       self.W_r, self.U_r, self.b_r,
-                       self.W_h, self.U_h, self.b_h]
+    def build(self, input_shape):
+        self.input_spec = [InputSpec(shape=input_shape)]
+        self.input_dim = input_shape[2]
 
         if self.stateful:
             self.reset_states()
@@ -299,43 +461,161 @@ class GRU(Recurrent):
             # initial states: all-zero tensor of shape (output_dim)
             self.states = [None]
 
+        if self.consume_less == 'gpu':
+
+            self.W = self.init((self.input_dim, 3 * self.output_dim),
+                               name='{}_W'.format(self.name))
+            self.U = self.inner_init((self.output_dim, 3 * self.output_dim),
+                                     name='{}_U'.format(self.name))
+
+            self.b = K.variable(np.hstack((np.zeros(self.output_dim),
+                                           np.zeros(self.output_dim),
+                                           np.zeros(self.output_dim))),
+                                name='{}_b'.format(self.name))
+
+            self.trainable_weights = [self.W, self.U, self.b]
+        else:
+
+            self.W_z = self.init((self.input_dim, self.output_dim),
+                                 name='{}_W_z'.format(self.name))
+            self.U_z = self.inner_init((self.output_dim, self.output_dim),
+                                       name='{}_U_z'.format(self.name))
+            self.b_z = K.zeros((self.output_dim,), name='{}_b_z'.format(self.name))
+
+            self.W_r = self.init((self.input_dim, self.output_dim),
+                                 name='{}_W_r'.format(self.name))
+            self.U_r = self.inner_init((self.output_dim, self.output_dim),
+                                       name='{}_U_r'.format(self.name))
+            self.b_r = K.zeros((self.output_dim,), name='{}_b_r'.format(self.name))
+
+            self.W_h = self.init((self.input_dim, self.output_dim),
+                                 name='{}_W_h'.format(self.name))
+            self.U_h = self.inner_init((self.output_dim, self.output_dim),
+                                       name='{}_U_h'.format(self.name))
+            self.b_h = K.zeros((self.output_dim,), name='{}_b_h'.format(self.name))
+
+            self.trainable_weights = [self.W_z, self.U_z, self.b_z,
+                                      self.W_r, self.U_r, self.b_r,
+                                      self.W_h, self.U_h, self.b_h]
+
+            self.W = K.concatenate([self.W_z, self.W_r, self.W_h])
+            self.U = K.concatenate([self.U_z, self.U_r, self.U_h])
+            self.b = K.concatenate([self.b_z, self.b_r, self.b_h])
+
+        self.regularizers = []
+        if self.W_regularizer:
+            self.W_regularizer.set_param(self.W)
+            self.regularizers.append(self.W_regularizer)
+        if self.U_regularizer:
+            self.U_regularizer.set_param(self.U)
+            self.regularizers.append(self.U_regularizer)
+        if self.b_regularizer:
+            self.b_regularizer.set_param(self.b)
+            self.regularizers.append(self.b_regularizer)
+
         if self.initial_weights is not None:
             self.set_weights(self.initial_weights)
             del self.initial_weights
 
     def reset_states(self):
         assert self.stateful, 'Layer must be stateful.'
-        input_shape = self.input_shape
+        input_shape = self.input_spec[0].shape
         if not input_shape[0]:
             raise Exception('If a RNN is stateful, a complete ' +
-                            'input_shape must be provided ' +
-                            '(including batch size).')
+                            'input_shape must be provided (including batch size).')
         if hasattr(self, 'states'):
             K.set_value(self.states[0],
                         np.zeros((input_shape[0], self.output_dim)))
         else:
             self.states = [K.zeros((input_shape[0], self.output_dim))]
 
+    def preprocess_input(self, x):
+        if self.consume_less == 'cpu':
+            input_shape = self.input_spec[0].shape
+            input_dim = input_shape[2]
+            timesteps = input_shape[1]
+
+            x_z = time_distributed_dense(x, self.W_z, self.b_z, self.dropout_W,
+                                         input_dim, self.output_dim, timesteps)
+            x_r = time_distributed_dense(x, self.W_r, self.b_r, self.dropout_W,
+                                         input_dim, self.output_dim, timesteps)
+            x_h = time_distributed_dense(x, self.W_h, self.b_h, self.dropout_W,
+                                         input_dim, self.output_dim, timesteps)
+            return K.concatenate([x_z, x_r, x_h], axis=2)
+        else:
+            return x
+
     def step(self, x, states):
-        assert len(states) == 1
-        x_z = K.dot(x, self.W_z) + self.b_z
-        x_r = K.dot(x, self.W_r) + self.b_r
-        x_h = K.dot(x, self.W_h) + self.b_h
+        h_tm1 = states[0]  # previous memory
+        B_U = states[1]  # dropout matrices for recurrent units
+        B_W = states[2]
 
-        h_tm1 = states[0]
-        z = self.inner_activation(x_z + K.dot(h_tm1, self.U_z))
-        r = self.inner_activation(x_r + K.dot(h_tm1, self.U_r))
+        if self.consume_less == 'gpu':
 
-        hh = self.activation(x_h + K.dot(r * h_tm1, self.U_h))
+            matrix_x = K.dot(x * B_W[0], self.W) + self.b
+            matrix_inner = K.dot(h_tm1 * B_U[0], self.U[:, :2 * self.output_dim])
+
+            x_z = matrix_x[:, :self.output_dim]
+            x_r = matrix_x[:, self.output_dim: 2 * self.output_dim]
+            inner_z = matrix_inner[:, :self.output_dim]
+            inner_r = matrix_inner[:, self.output_dim: 2 * self.output_dim]
+
+            z = self.inner_activation(x_z + inner_z)
+            r = self.inner_activation(x_r + inner_r)
+
+            x_h = matrix_x[:, 2 * self.output_dim:]
+            inner_h = K.dot(r * h_tm1 * B_U[0], self.U[:, 2 * self.output_dim:])
+            hh = self.activation(x_h + inner_h)
+        else:
+            if self.consume_less == 'cpu':
+                x_z = x[:, :self.output_dim]
+                x_r = x[:, self.output_dim: 2 * self.output_dim]
+                x_h = x[:, 2 * self.output_dim:]
+            elif self.consume_less == 'mem':
+                x_z = K.dot(x * B_W[0], self.W_z) + self.b_z
+                x_r = K.dot(x * B_W[1], self.W_r) + self.b_r
+                x_h = K.dot(x * B_W[2], self.W_h) + self.b_h
+            else:
+                raise Exception('Unknown `consume_less` mode.')
+            z = self.inner_activation(x_z + K.dot(h_tm1 * B_U[0], self.U_z))
+            r = self.inner_activation(x_r + K.dot(h_tm1 * B_U[1], self.U_r))
+
+            hh = self.activation(x_h + K.dot(r * h_tm1 * B_U[2], self.U_h))
         h = z * h_tm1 + (1 - z) * hh
         return h, [h]
 
+    def get_constants(self, x):
+        constants = []
+        if 0 < self.dropout_U < 1:
+            ones = K.ones_like(K.reshape(x[:, 0, 0], (-1, 1)))
+            ones = K.concatenate([ones] * self.output_dim, 1)
+            B_U = [K.in_train_phase(K.dropout(ones, self.dropout_U), ones) for _ in range(3)]
+            constants.append(B_U)
+        else:
+            constants.append([K.cast_to_floatx(1.) for _ in range(3)])
+
+        if 0 < self.dropout_W < 1:
+            input_shape = self.input_spec[0].shape
+            input_dim = input_shape[-1]
+            ones = K.ones_like(K.reshape(x[:, 0, 0], (-1, 1)))
+            ones = K.concatenate([ones] * input_dim, 1)
+            B_W = [K.in_train_phase(K.dropout(ones, self.dropout_W), ones) for _ in range(3)]
+            constants.append(B_W)
+        else:
+            constants.append([K.cast_to_floatx(1.) for _ in range(3)])
+        return constants
+
     def get_config(self):
-        config = {"output_dim": self.output_dim,
-                  "init": self.init.__name__,
-                  "inner_init": self.inner_init.__name__,
-                  "activation": self.activation.__name__,
-                  "inner_activation": self.inner_activation.__name__}
+        config = {'output_dim': self.output_dim,
+                  'init': self.init.__name__,
+                  'inner_init': self.inner_init.__name__,
+                  'activation': self.activation.__name__,
+                  'inner_activation': self.inner_activation.__name__,
+                  'W_regularizer': self.W_regularizer.get_config() if self.W_regularizer else None,
+                  'U_regularizer': self.U_regularizer.get_config() if self.U_regularizer else None,
+                  'b_regularizer': self.b_regularizer.get_config() if self.b_regularizer else None,
+                  'dropout_W': self.dropout_W,
+                  'dropout_U': self.dropout_U}
         base_config = super(GRU, self).get_config()
         return dict(list(base_config.items()) + list(config.items()))
 
@@ -359,56 +639,109 @@ class LSTM(Recurrent):
             Can be the name of an existing function (str),
             or a Theano function (see: [activations](../activations.md)).
         inner_activation: activation function for the inner cells.
+        W_regularizer: instance of [WeightRegularizer](../regularizers.md)
+            (eg. L1 or L2 regularization), applied to the input weights matrices.
+        U_regularizer: instance of [WeightRegularizer](../regularizers.md)
+            (eg. L1 or L2 regularization), applied to the recurrent weights matrices.
+        b_regularizer: instance of [WeightRegularizer](../regularizers.md),
+            applied to the bias.
+        dropout_W: float between 0 and 1. Fraction of the input units to drop for input gates.
+        dropout_U: float between 0 and 1. Fraction of the input units to drop for recurrent connections.
 
     # References
         - [Long short-term memory](http://deeplearning.cs.cmu.edu/pdfs/Hochreiter97_lstm.pdf) (original 1997 paper)
         - [Learning to forget: Continual prediction with LSTM](http://www.mitpressjournals.org/doi/pdf/10.1162/089976600300015015)
         - [Supervised sequence labelling with recurrent neural networks](http://www.cs.toronto.edu/~graves/preprint.pdf)
+        - [A Theoretically Grounded Application of Dropout in Recurrent Neural Networks](http://arxiv.org/abs/1512.05287)
     '''
     def __init__(self, output_dim,
                  init='glorot_uniform', inner_init='orthogonal',
                  forget_bias_init='one', activation='tanh',
-                 inner_activation='hard_sigmoid', **kwargs):
+                 inner_activation='hard_sigmoid',
+                 W_regularizer=None, U_regularizer=None, b_regularizer=None,
+                 dropout_W=0., dropout_U=0., **kwargs):
         self.output_dim = output_dim
         self.init = initializations.get(init)
         self.inner_init = initializations.get(inner_init)
         self.forget_bias_init = initializations.get(forget_bias_init)
         self.activation = activations.get(activation)
         self.inner_activation = activations.get(inner_activation)
+        self.W_regularizer = regularizers.get(W_regularizer)
+        self.U_regularizer = regularizers.get(U_regularizer)
+        self.b_regularizer = regularizers.get(b_regularizer)
+        self.dropout_W, self.dropout_U = dropout_W, dropout_U
+
+        if self.dropout_W or self.dropout_U:
+            self.uses_learning_phase = True
         super(LSTM, self).__init__(**kwargs)
 
-    def build(self):
-        input_shape = self.input_shape
-        input_dim = input_shape[2]
-        self.input_dim = input_dim
-        self.input = K.placeholder(input_shape)
+    def build(self, input_shape):
+        self.input_spec = [InputSpec(shape=input_shape)]
+        self.input_dim = input_shape[2]
 
         if self.stateful:
             self.reset_states()
         else:
-            # initial states: 2 all-zero tensor of shape (output_dim)
+            # initial states: 2 all-zero tensors of shape (output_dim)
             self.states = [None, None]
 
-        self.W_i = self.init((input_dim, self.output_dim))
-        self.U_i = self.inner_init((self.output_dim, self.output_dim))
-        self.b_i = K.zeros((self.output_dim,))
+        if self.consume_less == 'gpu':
+            self.W = self.init((self.input_dim, 4 * self.output_dim),
+                               name='{}_W'.format(self.name))
+            self.U = self.inner_init((self.output_dim, 4 * self.output_dim),
+                                     name='{}_U'.format(self.name))
 
-        self.W_f = self.init((input_dim, self.output_dim))
-        self.U_f = self.inner_init((self.output_dim, self.output_dim))
-        self.b_f = self.forget_bias_init((self.output_dim,))
+            self.b = K.variable(np.hstack((np.zeros(self.output_dim),
+                                           K.get_value(self.forget_bias_init((self.output_dim,))),
+                                           np.zeros(self.output_dim),
+                                           np.zeros(self.output_dim))),
+                                name='{}_b'.format(self.name))
+            self.trainable_weights = [self.W, self.U, self.b]
+        else:
+            self.W_i = self.init((self.input_dim, self.output_dim),
+                                 name='{}_W_i'.format(self.name))
+            self.U_i = self.inner_init((self.output_dim, self.output_dim),
+                                       name='{}_U_i'.format(self.name))
+            self.b_i = K.zeros((self.output_dim,), name='{}_b_i'.format(self.name))
 
-        self.W_c = self.init((input_dim, self.output_dim))
-        self.U_c = self.inner_init((self.output_dim, self.output_dim))
-        self.b_c = K.zeros((self.output_dim,))
+            self.W_f = self.init((self.input_dim, self.output_dim),
+                                 name='{}_W_f'.format(self.name))
+            self.U_f = self.inner_init((self.output_dim, self.output_dim),
+                                       name='{}_U_f'.format(self.name))
+            self.b_f = self.forget_bias_init((self.output_dim,),
+                                             name='{}_b_f'.format(self.name))
 
-        self.W_o = self.init((input_dim, self.output_dim))
-        self.U_o = self.inner_init((self.output_dim, self.output_dim))
-        self.b_o = K.zeros((self.output_dim,))
+            self.W_c = self.init((self.input_dim, self.output_dim),
+                                 name='{}_W_c'.format(self.name))
+            self.U_c = self.inner_init((self.output_dim, self.output_dim),
+                                       name='{}_U_c'.format(self.name))
+            self.b_c = K.zeros((self.output_dim,), name='{}_b_c'.format(self.name))
 
-        self.params = [self.W_i, self.U_i, self.b_i,
-                       self.W_c, self.U_c, self.b_c,
-                       self.W_f, self.U_f, self.b_f,
-                       self.W_o, self.U_o, self.b_o]
+            self.W_o = self.init((self.input_dim, self.output_dim),
+                                 name='{}_W_o'.format(self.name))
+            self.U_o = self.inner_init((self.output_dim, self.output_dim),
+                                       name='{}_U_o'.format(self.name))
+            self.b_o = K.zeros((self.output_dim,), name='{}_b_o'.format(self.name))
+
+            self.trainable_weights = [self.W_i, self.U_i, self.b_i,
+                                      self.W_c, self.U_c, self.b_c,
+                                      self.W_f, self.U_f, self.b_f,
+                                      self.W_o, self.U_o, self.b_o]
+
+            self.W = K.concatenate([self.W_i, self.W_f, self.W_c, self.W_o])
+            self.U = K.concatenate([self.U_i, self.U_f, self.U_c, self.U_o])
+            self.b = K.concatenate([self.b_i, self.b_f, self.b_c, self.b_o])
+
+        self.regularizers = []
+        if self.W_regularizer:
+            self.W_regularizer.set_param(self.W)
+            self.regularizers.append(self.W_regularizer)
+        if self.U_regularizer:
+            self.U_regularizer.set_param(self.U)
+            self.regularizers.append(self.U_regularizer)
+        if self.b_regularizer:
+            self.b_regularizer.set_param(self.b)
+            self.regularizers.append(self.b_regularizer)
 
         if self.initial_weights is not None:
             self.set_weights(self.initial_weights)
@@ -416,11 +749,10 @@ class LSTM(Recurrent):
 
     def reset_states(self):
         assert self.stateful, 'Layer must be stateful.'
-        input_shape = self.input_shape
+        input_shape = self.input_spec[0].shape
         if not input_shape[0]:
             raise Exception('If a RNN is stateful, a complete ' +
-                            'input_shape must be provided ' +
-                            '(including batch size).')
+                            'input_shape must be provided (including batch size).')
         if hasattr(self, 'states'):
             K.set_value(self.states[0],
                         np.zeros((input_shape[0], self.output_dim)))
@@ -430,29 +762,100 @@ class LSTM(Recurrent):
             self.states = [K.zeros((input_shape[0], self.output_dim)),
                            K.zeros((input_shape[0], self.output_dim))]
 
+    def preprocess_input(self, x):
+        if self.consume_less == 'cpu':
+            if 0 < self.dropout_W < 1:
+                dropout = self.dropout_W
+            else:
+                dropout = 0
+            input_shape = self.input_spec[0].shape
+            input_dim = input_shape[2]
+            timesteps = input_shape[1]
+
+            x_i = time_distributed_dense(x, self.W_i, self.b_i, dropout,
+                                         input_dim, self.output_dim, timesteps)
+            x_f = time_distributed_dense(x, self.W_f, self.b_f, dropout,
+                                         input_dim, self.output_dim, timesteps)
+            x_c = time_distributed_dense(x, self.W_c, self.b_c, dropout,
+                                         input_dim, self.output_dim, timesteps)
+            x_o = time_distributed_dense(x, self.W_o, self.b_o, dropout,
+                                         input_dim, self.output_dim, timesteps)
+            return K.concatenate([x_i, x_f, x_c, x_o], axis=2)
+        else:
+            return x
+
     def step(self, x, states):
-        assert len(states) == 2
         h_tm1 = states[0]
         c_tm1 = states[1]
+        B_U = states[2]
+        B_W = states[3]
 
-        x_i = K.dot(x, self.W_i) + self.b_i
-        x_f = K.dot(x, self.W_f) + self.b_f
-        x_c = K.dot(x, self.W_c) + self.b_c
-        x_o = K.dot(x, self.W_o) + self.b_o
+        if self.consume_less == 'gpu':
+            z = K.dot(x * B_W[0], self.W) + K.dot(h_tm1 * B_U[0], self.U) + self.b
+
+            z0 = z[:, :self.output_dim]
+            z1 = z[:, self.output_dim: 2 * self.output_dim]
+            z2 = z[:, 2 * self.output_dim: 3 * self.output_dim]
+            z3 = z[:, 3 * self.output_dim:]
+
+            i = self.inner_activation(z0)
+            f = self.inner_activation(z1)
+            c = f * c_tm1 + i * self.activation(z2)
+            o = self.inner_activation(z3)
+        else:
+            if self.consume_less == 'cpu':
+                x_i = x[:, :self.output_dim]
+                x_f = x[:, self.output_dim: 2 * self.output_dim]
+                x_c = x[:, 2 * self.output_dim: 3 * self.output_dim]
+                x_o = x[:, 3 * self.output_dim:]
+            elif self.consume_less == 'mem':
+                x_i = K.dot(x * B_W[0], self.W_i) + self.b_i
+                x_f = K.dot(x * B_W[1], self.W_f) + self.b_f
+                x_c = K.dot(x * B_W[2], self.W_c) + self.b_c
+                x_o = K.dot(x * B_W[3], self.W_o) + self.b_o
+            else:
+                raise Exception('Unknown `consume_less` mode.')
+
+            i = self.inner_activation(x_i + K.dot(h_tm1 * B_U[0], self.U_i))
+            f = self.inner_activation(x_f + K.dot(h_tm1 * B_U[1], self.U_f))
+            c = f * c_tm1 + i * self.activation(x_c + K.dot(h_tm1 * B_U[2], self.U_c))
+            o = self.inner_activation(x_o + K.dot(h_tm1 * B_U[3], self.U_o))
 
-        i = self.inner_activation(x_i + K.dot(h_tm1, self.U_i))
-        f = self.inner_activation(x_f + K.dot(h_tm1, self.U_f))
-        c = f * c_tm1 + i * self.activation(x_c + K.dot(h_tm1, self.U_c))
-        o = self.inner_activation(x_o + K.dot(h_tm1, self.U_o))
         h = o * self.activation(c)
         return h, [h, c]
 
+    def get_constants(self, x):
+        constants = []
+        if 0 < self.dropout_U < 1:
+            ones = K.ones_like(K.reshape(x[:, 0, 0], (-1, 1)))
+            ones = K.concatenate([ones] * self.output_dim, 1)
+            B_U = [K.in_train_phase(K.dropout(ones, self.dropout_U), ones) for _ in range(4)]
+            constants.append(B_U)
+        else:
+            constants.append([K.cast_to_floatx(1.) for _ in range(4)])
+
+        if 0 < self.dropout_W < 1:
+            input_shape = self.input_spec[0].shape
+            input_dim = input_shape[-1]
+            ones = K.ones_like(K.reshape(x[:, 0, 0], (-1, 1)))
+            ones = K.concatenate([ones] * input_dim, 1)
+            B_W = [K.in_train_phase(K.dropout(ones, self.dropout_W), ones) for _ in range(4)]
+            constants.append(B_W)
+        else:
+            constants.append([K.cast_to_floatx(1.) for _ in range(4)])
+        return constants
+
     def get_config(self):
-        config = {"output_dim": self.output_dim,
-                  "init": self.init.__name__,
-                  "inner_init": self.inner_init.__name__,
-                  "forget_bias_init": self.forget_bias_init.__name__,
-                  "activation": self.activation.__name__,
-                  "inner_activation": self.inner_activation.__name__}
+        config = {'output_dim': self.output_dim,
+                  'init': self.init.__name__,
+                  'inner_init': self.inner_init.__name__,
+                  'forget_bias_init': self.forget_bias_init.__name__,
+                  'activation': self.activation.__name__,
+                  'inner_activation': self.inner_activation.__name__,
+                  'W_regularizer': self.W_regularizer.get_config() if self.W_regularizer else None,
+                  'U_regularizer': self.U_regularizer.get_config() if self.U_regularizer else None,
+                  'b_regularizer': self.b_regularizer.get_config() if self.b_regularizer else None,
+                  'dropout_W': self.dropout_W,
+                  'dropout_U': self.dropout_U}
         base_config = super(LSTM, self).get_config()
         return dict(list(base_config.items()) + list(config.items()))

keras/layers/wrappers.py

@@ -0,0 +1,135 @@
+from ..engine import Layer, InputSpec
+from .. import backend as K
+
+
+class Wrapper(Layer):
+
+    def __init__(self, layer, **kwargs):
+        self.layer = layer
+        self.uses_learning_phase = layer.uses_learning_phase
+        super(Wrapper, self).__init__(**kwargs)
+
+    def build(self, input_shape=None):
+        '''Assumes that self.layer is already set.
+        Should be called at the end of .build() in the
+        children classes.
+        '''
+        self.trainable_weights = getattr(self.layer, 'trainable_weights', [])
+        self.non_trainable_weights = getattr(self.layer, 'non_trainable_weights', [])
+        self.updates = getattr(self.layer, 'updates', [])
+        self.regularizers = getattr(self.layer, 'regularizers', [])
+        self.constraints = getattr(self.layer, 'constraints', {})
+
+    def get_weights(self):
+        weights = self.layer.get_weights()
+        return weights
+
+    def set_weights(self, weights):
+        self.layer.set_weights(weights)
+
+    def get_config(self):
+        config = {'layer': {'class_name': self.layer.__class__.__name__,
+                            'config': self.layer.get_config()}}
+        base_config = super(Wrapper, self).get_config()
+        return dict(list(base_config.items()) + list(config.items()))
+
+    @classmethod
+    def from_config(cls, config):
+        from keras.utils.layer_utils import layer_from_config
+        layer = layer_from_config(config.pop('layer'))
+        return cls(layer, **config)
+
+
+class TimeDistributed(Wrapper):
+    """This wrapper allows to apply a layer to every
+    temporal slice of an input.
+
+    The input should be at least 3D,
+    and the dimension of index one will be considered to be
+    the temporal dimension.
+
+    Consider a batch of 32 samples, where each sample is a sequence of 10
+    vectors of 16 dimensions. The batch input shape of the layer is then `(32, 10, 16)`
+    (and the `input_shape`, not including the samples dimension, is `(10, 16)`).
+
+    You can then use `TimeDistributed` to apply a `Dense` layer to each of the 10 timesteps, independently:
+    ```python
+        # as the first layer in a model
+        model = Sequential()
+        model.add(TimeDistributed(Dense(8), input_shape=(10, 16)))
+        # now model.output_shape == (None, 10, 8)
+
+        # subsequent layers: no need for input_shape
+        model.add(TimeDistributed(Dense(32)))
+        # now model.output_shape == (None, 10, 32)
+    ```
+
+    The output will then have shape `(32, 10, 8)`.
+
+    Note this is strictly equivalent to using `layers.core.TimeDistributedDense`.
+    However what is different about `TimeDistributed`
+    is that it can be used with arbitrary layers, not just `Dense`,
+    for instance with a `Convolution2D` layer:
+
+    ```python
+        model = Sequential()
+        model.add(TimeDistributed(Convolution2D(64, 3, 3), input_shape=(10, 3, 299, 299)))
+    ```
+
+    # Arguments
+        layer: a layer instance.
+    """
+    def __init__(self, layer, **kwargs):
+        self.supports_masking = True
+        super(TimeDistributed, self).__init__(layer, **kwargs)
+
+    def build(self, input_shape):
+        assert len(input_shape) >= 3
+        self.input_spec = [InputSpec(shape=input_shape)]
+        if K._BACKEND == 'tensorflow':
+            if not input_shape[1]:
+                raise Exception('When using TensorFlow, you should define '
+                                'explicitly the number of timesteps of '
+                                'your sequences.\n'
+                                'If your first layer is an Embedding, '
+                                'make sure to pass it an "input_length" '
+                                'argument. Otherwise, make sure '
+                                'the first layer has '
+                                'an "input_shape" or "batch_input_shape" '
+                                'argument, including the time axis.')
+        child_input_shape = (input_shape[0],) + input_shape[2:]
+        if not self.layer.built:
+            self.layer.build(child_input_shape)
+            self.layer.built = True
+        super(TimeDistributed, self).build()
+
+    def get_output_shape_for(self, input_shape):
+        child_input_shape = (input_shape[0],) + input_shape[2:]
+        child_output_shape = self.layer.get_output_shape_for(child_input_shape)
+        timesteps = input_shape[1]
+        return (child_output_shape[0], timesteps) + child_output_shape[1:]
+
+    def call(self, X, mask=None):
+        input_shape = self.input_spec[0].shape
+        if input_shape[0]:
+            # batch size matters, use rnn-based implementation
+            def step(x, states):
+                output = self.layer.call(x)
+                return output, []
+
+            last_output, outputs, states = K.rnn(step, X,
+                                                 initial_states=[])
+            y = outputs
+        else:
+            # no batch size specified, therefore the layer will be able
+            # to process batches of any size
+            # we can go with reshape-based implementation for performance
+            input_length = input_shape[1]
+            if not input_length:
+                input_length = K.shape(X)[1]
+            X = K.reshape(X, (-1, ) + input_shape[2:])  # (nb_samples * timesteps, ...)
+            y = self.layer.call(X)  # (nb_samples * timesteps, ...)
+            # (nb_samples, timesteps, ...)
+            output_shape = self.get_output_shape_for(input_shape)
+            y = K.reshape(y, (-1, input_length) + output_shape[2:])
+        return y

keras/legacy/models.py

@@ -0,0 +1,775 @@
+from collections import OrderedDict
+import warnings
+import copy
+
+from .. import backend as K
+from ..layers import InputLayer, Layer, Merge
+from ..engine.training import Model
+
+
+class Graph(Model):
+    '''Arbitrary connection graph.
+
+    THIS IS A LEGACY MODEL AND SHOULD NOT BE USED
+    except for backwards compatibility support.
+
+    For multi-inputs/multi-outputs models, or
+    models using shared layers, use the functional API instead.
+    '''
+
+    def __init__(self, name=None):
+        # model attributes
+        self.inbound_nodes = []
+        self.outbound_nodes = []
+        self.built = False
+        self.supports_masking = False
+
+        # legacy attributes (we prefix them with _graph_)
+        self._graph_namespace = set()  # strings
+        self._graph_nodes = OrderedDict()  # layer-like
+        self._graph_inputs = OrderedDict()  # layer-like
+        self._graph_outputs = OrderedDict()  # layer-like
+        self._graph_input_config = []  # dicts
+        self._graph_output_config = []  # dicts
+        self._graph_node_config = []  # dicts
+        self._graph_shared_nodes_names = []
+
+        if not name:
+            prefix = 'graph_'
+            name = prefix + str(K.get_uid(prefix))
+        self.name = name
+
+    def __call__(self, x, mask=None):
+        self.build()
+        return super(Graph, self).__call__(x, mask)
+
+    def build(self, input_shape=None):
+        # this will crash if the input/output layers have multiple nodes
+        # no plans to support that case since Graph is deprecated
+        input_tensors = [layer.output for layer in self._graph_inputs.values()]
+        output_tensors = [layer.output for layer in self._graph_outputs.values()]
+        # actually create the model
+        super(Graph, self).__init__(input_tensors,
+                                    output_tensors,
+                                    name=self.name)
+        self.built = True
+
+    def compile(self, optimizer, loss,
+                metrics=[],
+                sample_weight_modes=None,
+                loss_weights=None,
+                **kwargs):
+        '''Configures the learning process.
+
+        # Arguments
+            optimizer: str (name of optimizer) or optimizer object.
+                See [optimizers](optimizers.md).
+            loss: dictionary mapping the name(s) of the output(s) to
+                a loss function (string name of objective function or
+                objective function. See [objectives](objectives.md)).
+            metrics: list of str (name of metrics) or
+                list of metrics functions. See [metrics](metrics.md).
+            sample_weight_modes: optional dictionary mapping certain
+                output names to a sample weight mode ("temporal" and None
+                are the only supported modes). If you need to do
+                timestep-wise loss weighting on one of your graph outputs,
+                you will need to set the sample weight mode for this output
+                to "temporal".
+            loss_weights: dictionary you can pass to specify a weight
+                coefficient for each loss function (in a multi-output model).
+                If no loss weight is specified for an output,
+                the weight for this output's loss will be considered to be 1.
+            kwargs: for Theano backend, these are passed into K.function.
+                Ignored for Tensorflow backend.
+        '''
+        # create the underlying Model
+        if not self.built:
+            self.build()
+        super(Graph, self).compile(optimizer, loss,
+                                   metrics=metrics,
+                                   sample_weight_mode=sample_weight_modes,
+                                   loss_weights=loss_weights,
+                                   **kwargs)
+
+    def add_input(self, name, input_shape=None,
+                  batch_input_shape=None, dtype='float'):
+        '''Adds an input to the graph.
+
+        # Arguments:
+            name: string. The name of the new input.
+                Must be unique in the graph.
+            input_shape: a tuple of integers,
+                the expected shape of the input samples.
+                Does not include the batch size.
+            batch_input_shape: a tuple of integers,
+                the expected shape of the whole input batch,
+                including the batch size.
+            dtype: 'float', or 'int'.
+        '''
+        if name in self._graph_namespace:
+            raise Exception('Duplicate node identifier: ' + name)
+        self._graph_namespace.add(name)
+        self.built = False
+
+        if dtype[:3] == 'int':
+            dtype = 'int32'
+        elif dtype[:5] == 'float':
+            dtype = K.floatx()
+        else:
+            raise Exception('Uknown dtype (should be "int" or "float"): ' +
+                            str(dtype))
+
+        # create input layer
+        input_layer = InputLayer(input_shape=input_shape,
+                                 batch_input_shape=batch_input_shape,
+                                 name=name, input_dtype=dtype)
+        self._graph_inputs[name] = input_layer
+
+        # append input config to self._graph_input_config
+        config = {'name': name, 'dtype': dtype}
+        if batch_input_shape:
+            config['batch_input_shape'] = batch_input_shape
+        else:
+            config['input_shape'] = input_shape
+        self._graph_input_config.append(config)
+
+    def add_node(self, layer, name, input=None, inputs=[],
+                 merge_mode='concat', concat_axis=-1, dot_axes=-1,
+                 create_output=False):
+        '''Adds a node in the graph. It can be connected to multiple
+        inputs, which will first be merged into one tensor
+        according to the mode specified.
+
+        # Arguments
+            layer: the layer at the node.
+            name: name for the node.
+            input: when connecting the layer to a single input,
+                this is the name of the incoming node.
+            inputs: when connecting the layer to multiple inputs,
+                this is a list of names of incoming nodes.
+            merge_mode: one of {concat, sum, dot, ave, mul}
+            concat_axis: when `merge_mode=='concat'`, this is the
+                input concatenation axis.
+            dot_axes: when `merge_mode='dot'`,
+                this is the contraction axes specification;
+                see the `Merge` layer for details.
+            create_output: boolean. Set this to `True` if you want the output
+                of your node to be an output of the graph.
+        '''
+        if name in self._graph_namespace:
+            raise Exception('Duplicate node identifier: ' + name)
+        self._graph_namespace.add(name)
+        layer.name = name
+        self.built = False
+
+        if input:
+            if input not in self._graph_namespace:
+                raise Exception('Unknown node/input identifier: ' + input)
+            if input in self._graph_nodes:
+                layer.add_inbound_node(self._graph_nodes[input])
+            elif input in self._graph_inputs:
+                layer.add_inbound_node(self._graph_inputs[input])
+        if inputs:
+            to_merge = []
+            for n in inputs:
+                if n in self._graph_nodes:
+                    to_merge.append(self._graph_nodes[n])
+                elif n in self._graph_inputs:
+                    to_merge.append(self._graph_inputs[n])
+                else:
+                    raise Exception('Unknown identifier: ' + n)
+            merge = Merge(to_merge, mode=merge_mode,
+                          concat_axis=concat_axis, dot_axes=dot_axes,
+                          name='merge_inputs_for_' + name)
+            layer.add_inbound_node(merge)
+        self._graph_nodes[name] = layer
+        self._graph_node_config.append({'name': name,
+                                        'input': input,
+                                        'inputs': inputs,
+                                        'merge_mode': merge_mode,
+                                        'concat_axis': concat_axis,
+                                        'dot_axes': dot_axes,
+                                        'create_output': create_output})
+        if create_output:
+            self.add_output(name, input=name)
+
+    def add_shared_node(self, layer, name, inputs=[], merge_mode=None,
+                        concat_axis=-1, dot_axes=-1, outputs=[],
+                        create_output=False):
+        '''Used to share a same layer across multiple nodes.
+
+        Supposed, for instance, that you want to apply one same `Dense` layer
+        after two different nodes ('node_a' and 'node_b').
+        You can then add the dense layer as a shared node by calling:
+
+        ```python
+        model.add_shared_node(my_dense, name='shared_dense', inputs=['node_a', 'node_b'], ...)
+        ```
+
+        If you want access to the output of dense(node_a) and dense(node_b) separately,
+        you can add these outputs to the Graph by passing an `outputs` argument:
+
+        ```python
+        model.add_shared_node(my_dense, name='shared_dense', inputs=['node_a', 'node_b'],
+                              outputs=['dense_output_a', 'dense_outputs_b'])
+        ```
+
+        Otherwise you can merge these different outputs via `merge_mode`.
+        In that case you can access the merged output
+        under the identifier `name`.
+
+        # Arguments
+            layer: The layer to be shared across multiple inputs
+            name: Name of the shared node
+            inputs: List of names of input nodes
+            merge_mode: Same meaning as `merge_mode` argument of `add_node()`
+            concat_axis: Same meaning as `concat_axis` argument of `add_node()`
+            dot_axes: Same meaning as `dot_axes` argument of `add_node()`
+            outputs: Used when `merge_mode=None`. Names for the output nodes.
+            create_output: Same meaning as `create_output` argument of `add_node()`.
+        '''
+        if name in self._graph_namespace:
+            raise Exception('Duplicate node identifier: ' + name)
+        self._graph_namespace.add(name)
+        self.built = False
+
+        for o in outputs:
+            if o in self._graph_namespace:
+                raise Exception('Duplicate node identifier: ' + o)
+        if merge_mode:
+            if merge_mode not in {'sum', 'ave', 'mul', 'dot', 'cos', 'concat'}:
+                raise Exception('Invalid merge mode:', merge_mode)
+        input_layers = []
+        for i in range(len(inputs)):
+            input = inputs[i]
+            if input in self._graph_nodes:
+                n = self._graph_nodes[input]
+                input_layers.append(n)
+            elif input in self._graph_inputs:
+                n = self._graph_inputs[input]
+                input_layers.append(n)
+            else:
+                raise Exception('Unknown identifier: ' + input)
+
+        created_node_indices = []
+        for input_layer in input_layers:
+            created_node_indices.append(len(layer.inbound_nodes))
+            layer.add_inbound_node(input_layer)
+
+        if merge_mode:
+            layer.name = 'input_for_' + name
+            # collect all output nodes of layer and merge them into a single output
+            merge = Merge([layer for _ in range(len(inputs))],
+                          mode=merge_mode,
+                          concat_axis=concat_axis, dot_axes=dot_axes,
+                          node_indices=created_node_indices,
+                          name=name)
+            self._graph_nodes[name] = merge
+            if create_output:
+                self.add_output(name, input=name)
+        else:
+            layer.name = name
+            # create one new layer per output node of layer,
+            # and add them to the Graph with their own identifiers
+            if len(outputs) != len(inputs):
+                raise Exception('When using merge_mode=None, '
+                                'you should provide a list of '
+                                'output names (`output` argument) '
+                                'the same size as `input`.')
+            for i in range(len(outputs)):
+                output_layer_name = outputs[i]
+                output_layer = Layer(name=output_layer_name)
+                output_layer.add_inbound_node(layer, created_node_indices[i])
+                self._graph_namespace.add(output_layer_name)
+                self._graph_nodes[output_layer_name] = output_layer
+                if create_output:
+                    self.add_output(output_layer_name, input=output_layer_name)
+
+        self._graph_node_config.append({'name': name,
+                                        'layer': {
+                                            'config': layer.get_config(),
+                                            'class_name': layer.__class__.__name__,
+                                        },
+                                        'inputs': inputs,
+                                        'merge_mode': merge_mode,
+                                        'concat_axis': concat_axis,
+                                        'dot_axes': dot_axes,
+                                        'outputs': outputs,
+                                        'create_output': create_output if merge_mode else False})
+        self._graph_shared_nodes_names.append(name)
+
+    def add_output(self, name, input=None, inputs=[],
+                   merge_mode='concat', concat_axis=-1, dot_axes=-1):
+        '''Adds an output to the graph.
+
+        This output can merge several node outputs into a single output.
+
+        # Arguments
+            name: name of the output.
+            input: when connecting the layer to a single input,
+                this is the name of the incoming node.
+            inputs: when connecting the layer to multiple inputs,
+                this is a list of names of incoming nodes.
+            merge_mode: one of {concat, sum, dot, ave, mul}
+            concat_axis: when `merge_mode=='concat'`, this is the
+                input concatenation axis.
+            dot_axes: when `merge_mode='dot'`,
+                this is the contraction axes specification;
+                see the `Merge layer for details.
+        '''
+        if name not in self._graph_namespace:
+            self._graph_namespace.add(name)
+        if name in self._graph_outputs:
+            raise Exception('Duplicate output identifier:', name)
+        self.built = False
+
+        if input:
+            if input in self._graph_nodes:
+                layer = self._graph_nodes[input]
+            elif input in self._graph_inputs:
+                layer = self._graph_inputs[input]
+            else:
+                raise Exception('Unknown node/input identifier: ' + input)
+            if layer.name == name:
+                self._graph_outputs[name] = layer
+            else:
+                layer.name = name
+                self._graph_outputs[name] = layer
+        if inputs:
+            to_merge = []
+            for n in inputs:
+                if n not in self._graph_nodes:
+                    raise Exception('Unknown identifier: ' + n)
+                to_merge.append(self._graph_nodes[n])
+            merge = Merge(to_merge, mode=merge_mode,
+                          concat_axis=concat_axis, dot_axes=dot_axes,
+                          name=name)
+            self._graph_outputs[name] = merge
+
+        self._graph_output_config.append({'name': name,
+                                          'input': input,
+                                          'inputs': inputs,
+                                          'merge_mode': merge_mode,
+                                          'concat_axis': concat_axis,
+                                          'dot_axes': dot_axes})
+
+    def _get_x(self, data):
+        x = []
+        for key in self._graph_inputs.keys():
+            if key not in data:
+                raise Exception('Expected to be provided an array '
+                                '(in dict argument `data`) for input "' +
+                                key + '".')
+            x.append(data[key])
+        return x
+
+    def _get_y(self, data):
+        y = []
+        for key in self._graph_outputs.keys():
+            if key not in data:
+                raise Exception('Expected to be provided an array '
+                                '(in dict argument `data`) for output "' +
+                                key + '".')
+            y.append(data[key])
+        return y
+
+    def fit(self, data, batch_size=32, nb_epoch=10, verbose=1, callbacks=[],
+            validation_split=0., validation_data=None, shuffle=True,
+            class_weight=None, sample_weight=None, **kwargs):
+        '''Trains the model for a fixed number of epochs.
+
+        Returns a history object. Its `history` attribute is a record of
+        training loss values at successive epochs,
+        as well as validation loss values (if applicable).
+
+        # Arguments
+            data: dictionary mapping input names and outputs names to
+                appropriate Numpy arrays. All arrays should contain
+                the same number of samples.
+            batch_size: int. Number of samples per gradient update.
+            nb_epoch: int.
+            verbose: 0 for no logging to stdout,
+                1 for progress bar logging, 2 for one log line per epoch.
+            callbacks: `keras.callbacks.Callback` list. List of callbacks
+                to apply during training. See [callbacks](callbacks.md).
+            validation_split: float (0. < x < 1). Fraction of the data to
+                use as held-out validation data.
+            validation_data: dictionary mapping input names and outputs names
+                to appropriate Numpy arrays to be used as
+                held-out validation data.
+                All arrays should contain the same number of samples.
+                Will override validation_split.
+            shuffle: boolean. Whether to shuffle the samples at each epoch.
+            class_weight: dictionary mapping output names to
+                class weight dictionaries.
+            sample_weight: dictionary mapping output names to
+                numpy arrays of sample weights.
+        '''
+        if 'show_accuracy' in kwargs:
+            kwargs.pop('show_accuracy')
+            warnings.warn('The "show_accuracy" argument is deprecated, '
+                          'instead you should pass the "accuracy" metric to '
+                          'the model at compile time:\n'
+                          '`model.compile(optimizer, loss, '
+                          'metrics=["accuracy"])`')
+        if kwargs:
+            raise Exception('Received unknown keyword arguments: ' +
+                            str(kwargs))
+        x = self._get_x(data)
+        y = self._get_y(data)
+
+        if type(validation_data) is tuple:
+            raise Exception('Cannot used sample_weight with '
+                            'validation data with legacy Graph model. '
+                            'validation_data should be a dictionary.')
+        if validation_data:
+            val_x = self._get_x(validation_data)
+            val_y = self._get_y(validation_data)
+            validation_data = (val_x, val_y)
+        return super(Graph, self).fit(x, y,
+                                      batch_size=batch_size,
+                                      nb_epoch=nb_epoch,
+                                      verbose=verbose,
+                                      callbacks=callbacks,
+                                      validation_split=validation_split,
+                                      validation_data=validation_data,
+                                      shuffle=shuffle,
+                                      class_weight=class_weight,
+                                      sample_weight=sample_weight)
+
+    def evaluate(self, data, batch_size=128,
+                 verbose=0, sample_weight={}, **kwargs):
+        '''Computes the loss on some input data, batch by batch.
+
+        Returns the scalar test loss over the data,
+        or a list of metrics values (starting with the test loss)
+        if applicable.
+
+        Arguments: see `fit` method.
+        '''
+        if 'show_accuracy' in kwargs:
+            kwargs.pop('show_accuracy')
+            warnings.warn('The "show_accuracy" argument is deprecated, '
+                          'instead you should pass the "accuracy" metric to '
+                          'the model at compile time:\n'
+                          '`model.compile(optimizer, loss, '
+                          'metrics=["accuracy"])`')
+        if kwargs:
+            raise Exception('Received unknown keyword arguments: ' +
+                            str(kwargs))
+        x = self._get_x(data)
+        y = self._get_y(data)
+        return super(Graph, self).evaluate(x, y,
+                                           batch_size=batch_size,
+                                           verbose=verbose,
+                                           sample_weight=sample_weight)
+
+    def predict(self, data, batch_size=128, verbose=0):
+        '''Generates output predictions for the input samples
+        batch by batch.
+
+        Arguments: see `fit` method.
+        '''
+        x = self._get_x(data)
+        output_list = super(Graph, self).predict(x, batch_size=batch_size,
+                                                 verbose=verbose)
+        if not isinstance(output_list, list):
+            output_list = [output_list]
+        return dict(zip(self._graph_outputs, output_list))
+
+    def train_on_batch(self, data,
+                       class_weight={},
+                       sample_weight={}, **kwargs):
+        '''Single gradient update on a batch of samples.
+
+        Returns the scalar train loss over the data,
+        or a list of metrics values (starting with the test loss)
+        if applicable.
+
+        Arguments: see `fit` method.
+        '''
+        if 'accuracy' in kwargs:
+            kwargs.pop('accuracy')
+            warnings.warn('The "accuracy" argument is deprecated, '
+                          'instead you should pass the "accuracy" metric to '
+                          'the model at compile time:\n'
+                          '`model.compile(optimizer, loss, '
+                          'metrics=["accuracy"])`')
+        if kwargs:
+            raise Exception('Received unknown keyword arguments: ' +
+                            str(kwargs))
+        x = self._get_x(data)
+        y = self._get_y(data)
+        return super(Graph, self).train_on_batch(x, y,
+                                                 sample_weight=sample_weight,
+                                                 class_weight=class_weight)
+
+    def test_on_batch(self, data, sample_weight={}, **kwargs):
+        '''Test the network on a single batch of samples.
+
+        Returns the scalar test loss over the data,
+        or a list of metrics values (starting with the test loss)
+        if applicable.
+
+        Arguments: see `fit` method.
+        '''
+        if 'accuracy' in kwargs:
+            kwargs.pop('accuracy')
+            warnings.warn('The "accuracy" argument is deprecated, '
+                          'instead you should pass the "accuracy" metric to '
+                          'the model at compile time:\n'
+                          '`model.compile(optimizer, loss, '
+                          'metrics=["accuracy"])`')
+        if kwargs:
+            raise Exception('Received unknown keyword arguments: ' +
+                            str(kwargs))
+        x = self._get_x(data)
+        y = self._get_y(data)
+        return super(Graph, self).test_on_batch(x, y,
+                                                sample_weight=sample_weight)
+
+    def predict_on_batch(self, data):
+        output_list = super(Graph, self).predict_on_batch(data)
+        if not isinstance(output_list, list):
+            output_list = [output_list]
+        return dict(zip(self._graph_outputs, output_list))
+
+    def fit_generator(self, generator, samples_per_epoch, nb_epoch,
+                      verbose=1, callbacks=[],
+                      validation_data=None, nb_val_samples=None,
+                      class_weight={},
+                      max_q_size=10, **kwargs):
+        '''Fits a model on data generated batch-by-batch by a Python generator.
+        The generator is run in parallel to the model, for efficiency.
+        For instance, this allows you to do real-time data augmentation
+        on images on CPU in parallel to training your model on GPU.
+
+        # Arguments
+            generator: a generator.
+                The output of the generator must be either a tuple
+                of dictionaries `(input_data, sample_weight)`
+                or a dictionary `input_data`
+                (mapping names of inputs and outputs to Numpy arrays).
+                All arrays should contain the same number of samples.
+                The generator is expected to loop over its data
+                indefinitely. An epoch finishes when `samples_per_epoch`
+                samples have been seen by the model.
+            samples_per_epoch: integer, number of samples to process before
+                going to the next epoch.
+            nb_epoch: integer, total number of iterations on the data.
+            verbose: verbosity mode, 0, 1, or 2.
+            callbacks: list of callbacks to be called during training.
+            validation_data: dictionary mapping input names and outputs names
+                to appropriate Numpy arrays to be used as
+                held-out validation data, or a generator yielding such
+                dictionaries. All arrays should contain the same number
+                of samples. If a generator, will be called until more than
+                `nb_val_samples` examples have been generated at the
+                end of every epoch. These examples will then be used
+                as the validation data.
+            nb_val_samples: number of samples to use from validation
+                generator at the end of every epoch.
+            class_weight: dictionary mapping class indices to a weight
+                for the class.
+
+        # Returns
+            A `History` object.
+
+        # Examples
+
+        ```python
+            def generate_arrays_from_file(path):
+                while 1:
+                    f = open(path)
+                    for line in f:
+                        # create Numpy arrays of input data
+                        # and labels, from each line in the file
+                        x1, x2, y = process_line(line)
+                        yield ({'input_1': x1, 'input_2': x2, 'output': y})
+                    f.close()
+
+            graph.fit_generator(generate_arrays_from_file('/my_file.txt'),
+                                samples_per_epoch=10000, nb_epoch=10)
+        ```
+        '''
+        if 'show_accuracy' in kwargs:
+            kwargs.pop('show_accuracy')
+            warnings.warn('The "show_accuracy" argument is deprecated, '
+                          'instead you should pass the "accuracy" metric to '
+                          'the model at compile time:\n'
+                          '`model.compile(optimizer, loss, '
+                          'metrics=["accuracy"])`')
+        if 'nb_worker' in kwargs:
+            kwargs.pop('nb_worker')
+            warnings.warn('The "nb_worker" argument is deprecated, '
+                          'please remove it from your code.')
+        if 'nb_val_worker' in kwargs:
+            kwargs.pop('nb_val_worker')
+            warnings.warn('The "nb_val_worker" argument is deprecated, '
+                          'please remove it from your code.')
+        if kwargs:
+            raise Exception('Received unknown keyword arguments: ' +
+                            str(kwargs))
+
+        self._train_on_batch = self.train_on_batch
+        self.train_on_batch = super(Graph, self).train_on_batch
+        self._evaluate = self.evaluate
+        self.evaluate = super(Graph, self).evaluate
+
+        if validation_data and type(validation_data) is tuple:
+            raise Exception('Cannot use sample_weight with '
+                            'validation_data in legacy Graph model.')
+        if validation_data and type(validation_data) is dict:
+            validation_data = (self._get_x(validation_data),
+                               self._get_y(validation_data))
+
+        original_generator = generator
+
+        def fixed_generator():
+            while 1:
+                data = next(original_generator)
+                if type(data) is tuple:
+                    data, sample_weight = data
+                    x = self._get_x(data)
+                    y = self._get_y(data)
+                    yield x, y, sample_weight
+                else:
+                    x = self._get_x(data)
+                    y = self._get_y(data)
+                    yield x, y
+
+        generator = fixed_generator()
+        history = super(Graph, self).fit_generator(generator,
+                                                   samples_per_epoch,
+                                                   nb_epoch,
+                                                   verbose=verbose,
+                                                   callbacks=callbacks,
+                                                   validation_data=validation_data,
+                                                   nb_val_samples=nb_val_samples,
+                                                   class_weight=class_weight,
+                                                   max_q_size=max_q_size)
+        self.train_on_batch = self._train_on_batch
+        self.evaluate = self._evaluate
+        return history
+
+    def evaluate_generator(self, generator, val_samples,
+                           verbose=1, max_q_size=10, **kwargs):
+        '''Evaluates the model on a generator. The generator should
+        return the same kind of data with every yield as accepted
+        by `evaluate`.
+
+        If `show_accuracy`, it returns a tuple `(loss, accuracy)`,
+        otherwise it returns the loss value.
+
+        Arguments:
+            generator:
+                generator yielding dictionaries of the kind accepted
+                by `evaluate`, or tuples of such dictionaries and
+                associated dictionaries of sample weights.
+            val_samples:
+                total number of samples to generate from `generator`
+                to use in validation.
+
+            Other arguments are the same as for `fit`.
+        '''
+        if 'show_accuracy' in kwargs:
+            kwargs.pop('show_accuracy')
+            warnings.warn('The "show_accuracy" argument is deprecated, '
+                          'instead you should pass the "accuracy" metric to '
+                          'the model at compile time:\n'
+                          '`model.compile(optimizer, loss, '
+                          'metrics=["accuracy"])`')
+        if 'verbose' in kwargs:
+            kwargs.pop('verbose')
+            warnings.warn('The "verbose" argument is deprecated.')
+        if kwargs:
+            raise Exception('Received unknown keyword arguments: ' +
+                            str(kwargs))
+
+        self._test_on_batch = self.test_on_batch
+        self.test_on_batch = super(Graph, self).test_on_batch
+
+        original_generator = generator
+
+        def fixed_generator():
+            while 1:
+                data = next(original_generator)
+                if type(data) is tuple:
+                    data, sample_weight = data
+                    x = self._get_x(data)
+                    y = self._get_y(data)
+                    yield x, y, sample_weight
+                else:
+                    x = self._get_x(data)
+                    y = self._get_y(data)
+                    yield x, y
+
+        generator = fixed_generator()
+        history = super(Graph, self).evaluate_generator(generator,
+                                                        val_samples,
+                                                        max_q_size=max_q_size)
+        self.test_on_batch = self._test_on_batch
+        return history
+
+    # get_weights, set_weights: inherited
+    def get_config(self):
+        config = {'input_config': self._graph_input_config,
+                  'node_config': self._graph_node_config,
+                  'output_config': self._graph_output_config}
+        nodes = {}
+        for name, node in self._graph_nodes.items():
+            nodes[name] = {'class_name': node.__class__.__name__,
+                           'config': node.get_config()}
+            if name in self._graph_shared_nodes_names:
+                nodes[name]['shared'] = True
+        config['nodes'] = nodes
+        return copy.deepcopy(config)
+
+    @classmethod
+    def from_config(cls, config):
+        # TODO: test legacy support
+        from keras.utils.layer_utils import layer_from_config
+
+        def normalize_legacy_config(conf):
+            if 'class_name' not in conf:
+                class_name = conf['name']
+                name = conf.get('custom_name')
+                conf['name'] = name
+                new_config = {
+                    'class_name': class_name,
+                    'config': conf,
+                }
+                return new_config
+            return conf
+
+        graph = cls()
+        inputs = config.get('input_config')
+        for input in inputs:
+            graph.add_input(**input)
+
+        nodes = config.get('node_config')
+        for node in nodes:
+            layer_config = config['nodes'][node['name']]
+            layer_config = normalize_legacy_config(layer_config)
+            if 'layer' in node:
+                # for add_shared_node
+                node['layer'] = layer_from_config(node['layer'])
+            else:
+                layer = layer_from_config(layer_config)
+                node['layer'] = layer
+
+            node['create_output'] = False  # outputs will be added below
+            if layer_config.get('shared'):
+                graph.add_shared_node(**node)
+            else:
+                graph.add_node(**node)
+
+        outputs = config.get('output_config')
+        for output in outputs:
+            graph.add_output(**output)
+        return graph
+
+    def load_weights(self, fname):
+        if not self.built:
+            self.build()
+        super(Graph, self).load_weights(fname)

keras/metrics.py

@@ -0,0 +1,84 @@
+import numpy as np
+from . import backend as K
+
+
+def binary_accuracy(y_true, y_pred):
+    return K.mean(K.equal(y_true, K.round(y_pred)))
+
+
+def categorical_accuracy(y_true, y_pred):
+    return K.mean(K.equal(K.argmax(y_true, axis=-1),
+                  K.argmax(y_pred, axis=-1)))
+
+
+def sparse_categorical_accuracy(y_true, y_pred):
+    return K.mean(K.equal(K.max(y_true, axis=-1),
+                          K.cast(K.argmax(y_pred, axis=-1), K.floatx())))
+
+
+def mean_squared_error(y_true, y_pred):
+    return K.mean(K.square(y_pred - y_true))
+
+
+def mean_absolute_error(y_true, y_pred):
+    return K.mean(K.abs(y_pred - y_true))
+
+
+def mean_absolute_percentage_error(y_true, y_pred):
+    diff = K.abs((y_true - y_pred) / K.clip(K.abs(y_true), K.epsilon(), np.inf))
+    return 100. * K.mean(diff)
+
+
+def mean_squared_logarithmic_error(y_true, y_pred):
+    first_log = K.log(K.clip(y_pred, K.epsilon(), np.inf) + 1.)
+    second_log = K.log(K.clip(y_true, K.epsilon(), np.inf) + 1.)
+    return K.mean(K.square(first_log - second_log))
+
+
+def squared_hinge(y_true, y_pred):
+    return K.mean(K.square(K.maximum(1. - y_true * y_pred, 0.)))
+
+
+def hinge(y_true, y_pred):
+    return K.mean(K.maximum(1. - y_true * y_pred, 0.))
+
+
+def categorical_crossentropy(y_true, y_pred):
+    '''Expects a binary class matrix instead of a vector of scalar classes.
+    '''
+    return K.mean(K.categorical_crossentropy(y_pred, y_true))
+
+
+def sparse_categorical_crossentropy(y_true, y_pred):
+    '''expects an array of integer classes.
+    Note: labels shape must have the same number of dimensions as output shape.
+    If you get a shape error, add a length-1 dimension to labels.
+    '''
+    return K.mean(K.sparse_categorical_crossentropy(y_pred, y_true))
+
+
+def binary_crossentropy(y_true, y_pred):
+    return K.mean(K.binary_crossentropy(y_pred, y_true))
+
+
+def poisson(y_true, y_pred):
+    return K.mean(y_pred - y_true * K.log(y_pred + K.epsilon()))
+
+
+def cosine_proximity(y_true, y_pred):
+    y_true = K.l2_normalize(y_true, axis=-1)
+    y_pred = K.l2_normalize(y_pred, axis=-1)
+    return -K.mean(y_true * y_pred)
+
+
+# aliases
+mse = MSE = mean_squared_error
+mae = MAE = mean_absolute_error
+mape = MAPE = mean_absolute_percentage_error
+msle = MSLE = mean_squared_logarithmic_error
+cosine = cosine_proximity
+
+
+from .utils.generic_utils import get_from_module
+def get(identifier):
+    return get_from_module(identifier, globals(), 'metric')

keras/objectives.py

@@ -7,10 +7,6 @@ def mean_squared_error(y_true, y_pred):
     return K.mean(K.square(y_pred - y_true), axis=-1)
 
 
-def root_mean_squared_error(y_true, y_pred):
-    return K.sqrt(K.mean(K.square(y_pred - y_true), axis=-1))
-
-
 def mean_absolute_error(y_true, y_pred):
     return K.mean(K.abs(y_pred - y_true), axis=-1)
 
@@ -37,22 +33,44 @@ def hinge(y_true, y_pred):
 def categorical_crossentropy(y_true, y_pred):
     '''Expects a binary class matrix instead of a vector of scalar classes.
     '''
-    return K.mean(K.categorical_crossentropy(y_pred, y_true), axis=-1)
+    return K.categorical_crossentropy(y_pred, y_true)
+
+
+def sparse_categorical_crossentropy(y_true, y_pred):
+    '''expects an array of integer classes.
+    Note: labels shape must have the same number of dimensions as output shape.
+    If you get a shape error, add a length-1 dimension to labels.
+    '''
+    return K.sparse_categorical_crossentropy(y_pred, y_true)
 
 
 def binary_crossentropy(y_true, y_pred):
     return K.mean(K.binary_crossentropy(y_pred, y_true), axis=-1)
 
 
-def poisson_loss(y_true, y_pred):
+def kullback_leibler_divergence(y_true, y_pred):
+    y_true = K.clip(y_true, K.epsilon(), 1)
+    y_pred = K.clip(y_pred, K.epsilon(), 1)
+    return K.sum(y_true * K.log(y_true / y_pred), axis=-1)
+
+
+def poisson(y_true, y_pred):
     return K.mean(y_pred - y_true * K.log(y_pred + K.epsilon()), axis=-1)
 
+
+def cosine_proximity(y_true, y_pred):
+    y_true = K.l2_normalize(y_true, axis=-1)
+    y_pred = K.l2_normalize(y_pred, axis=-1)
+    return -K.mean(y_true * y_pred, axis=-1)
+
+
 # aliases
 mse = MSE = mean_squared_error
-rmse = RMSE = root_mean_squared_error
 mae = MAE = mean_absolute_error
 mape = MAPE = mean_absolute_percentage_error
 msle = MSLE = mean_squared_logarithmic_error
+kld = KLD = kullback_leibler_divergence
+cosine = cosine_proximity
 
 from .utils.generic_utils import get_from_module
 def get(identifier):

keras/optimizers.py

@@ -1,6 +1,5 @@
 from __future__ import absolute_import
 from . import backend as K
-import numpy as np
 from .utils.generic_utils import get_from_module
 from six.moves import zip
 
@@ -11,8 +10,24 @@ def clip_norm(g, c, n):
     return g
 
 
-def kl_divergence(p, p_hat):
-    return p_hat - p + p * K.log(p / p_hat)
+def optimizer_from_config(config, custom_objects={}):
+    all_classes = {
+        'sgd': SGD,
+        'rmsprop': RMSprop,
+        'adagrad': Adagrad,
+        'adadelta': Adadelta,
+        'adam': Adam,
+        'adamax': Adamax,
+        'nadam': Nadam,
+    }
+    class_name = config['class_name']
+    if class_name in custom_objects:
+        cls = custom_objects[class_name]
+    else:
+        if class_name.lower() not in all_classes:
+            raise ValueError('Optimizer class not found:', class_name)
+        cls = all_classes[class_name.lower()]
+    return cls.from_config(config['config'])
 
 
 class Optimizer(object):
@@ -29,8 +44,14 @@ class Optimizer(object):
             when their absolute value exceeds this value.
     '''
     def __init__(self, **kwargs):
+        allowed_kwargs = {'clipnorm', 'clipvalue'}
+        for k in kwargs:
+            if k not in allowed_kwargs:
+                raise Exception('Unexpected keyword argument '
+                                'passed to optimizer: ' + str(k))
         self.__dict__.update(kwargs)
         self.updates = []
+        self.weights = []
 
     def get_state(self):
         return [K.get_value(u[0]) for u in self.updates]
@@ -52,13 +73,53 @@ class Optimizer(object):
             grads = [K.clip(g, -self.clipvalue, self.clipvalue) for g in grads]
         return grads
 
+    def set_weights(self, weights):
+        '''Sets the weights of the optimizer, from Numpy arrays.
+
+        Should only be called after computing the gradients
+        (otherwise the optimizer has no weights).
+
+        # Arguments
+            weights: a list of Numpy arrays. The number
+                of arrays and their shape must match
+                number of the dimensions of the weights
+                of the optimizer (i.e. it should match the
+                output of `get_weights`).
+        '''
+        params = self.weights
+        weight_value_tuples = []
+        param_values = K.batch_get_value(params)
+        for pv, p, w in zip(param_values, params, weights):
+            if pv.shape != w.shape:
+                raise Exception('Optimizer weight shape ' +
+                                str(pv.shape) +
+                                ' not compatible with '
+                                'provided weight shape ' + str(w.shape))
+            weight_value_tuples.append((p, w))
+        K.batch_set_value(weight_value_tuples)
+
+    def get_weights(self):
+        '''Returns the current weights of the optimizer,
+        as a list of numpy arrays.
+        '''
+        return K.batch_get_value(self.weights)
+
     def get_config(self):
-        return {"name": self.__class__.__name__}
+        config = {}
+        if hasattr(self, 'clipnorm'):
+            config['clipnorm'] = self.clipnorm
+        if hasattr(self, 'clipvalue'):
+            config['clipvalue'] = self.clipvalue
+        return config
+
+    @classmethod
+    def from_config(cls, config):
+        return cls(**config)
 
 
 class SGD(Optimizer):
     '''Stochastic gradient descent, with support for momentum,
-    decay, and Nesterov momentum.
+    learning rate decay, and Nesterov momentum.
 
     # Arguments
         lr: float >= 0. Learning rate.
@@ -66,8 +127,8 @@ class SGD(Optimizer):
         decay: float >= 0. Learning rate decay over each update.
         nesterov: boolean. Whether to apply Nesterov momentum.
     '''
-    def __init__(self, lr=0.01, momentum=0., decay=0., nesterov=False,
-                 *args, **kwargs):
+    def __init__(self, lr=0.01, momentum=0., decay=0.,
+                 nesterov=False, **kwargs):
         super(SGD, self).__init__(**kwargs)
         self.__dict__.update(locals())
         self.iterations = K.variable(0.)
@@ -77,35 +138,45 @@ class SGD(Optimizer):
 
     def get_updates(self, params, constraints, loss):
         grads = self.get_gradients(loss, params)
-        lr = self.lr * (1.0 / (1.0 + self.decay * self.iterations))
-        self.updates = [(self.iterations, self.iterations + 1.)]
+        lr = self.lr * (1. / (1. + self.decay * self.iterations))
+        self.updates = [K.update_add(self.iterations, 1)]
 
-        for p, g, c in zip(params, grads, constraints):
-            m = K.variable(np.zeros(K.get_value(p).shape))  # momentum
+        # momentum
+        shapes = [x.shape for x in K.batch_get_value(params)]
+        moments = [K.zeros(shape) for shape in shapes]
+        self.weights = [self.iterations] + moments
+        for p, g, m in zip(params, grads, moments):
             v = self.momentum * m - lr * g  # velocity
-            self.updates.append((m, v))
+            self.updates.append(K.update(m, v))
 
             if self.nesterov:
                 new_p = p + self.momentum * v - lr * g
             else:
                 new_p = p + v
 
-            self.updates.append((p, c(new_p)))  # apply constraints
+            # apply constraints
+            if p in constraints:
+                c = constraints[p]
+                new_p = c(new_p)
+
+            self.updates.append(K.update(p, new_p))
         return self.updates
 
     def get_config(self):
-        return {"name": self.__class__.__name__,
-                "lr": float(K.get_value(self.lr)),
-                "momentum": float(K.get_value(self.momentum)),
-                "decay": float(K.get_value(self.decay)),
-                "nesterov": self.nesterov}
+        config = {'lr': float(K.get_value(self.lr)),
+                  'momentum': float(K.get_value(self.momentum)),
+                  'decay': float(K.get_value(self.decay)),
+                  'nesterov': self.nesterov}
+        base_config = super(SGD, self).get_config()
+        return dict(list(base_config.items()) + list(config.items()))
 
 
 class RMSprop(Optimizer):
     '''RMSProp optimizer.
 
     It is recommended to leave the parameters of this optimizer
-    at their default values.
+    at their default values
+    (except the learning rate, which can be freely tuned).
 
     This optimizer is usually a good choice for recurrent
     neural networks.
@@ -115,7 +186,7 @@ class RMSprop(Optimizer):
         rho: float >= 0.
         epsilon: float >= 0. Fuzz factor.
     '''
-    def __init__(self, lr=0.001, rho=0.9, epsilon=1e-6, *args, **kwargs):
+    def __init__(self, lr=0.001, rho=0.9, epsilon=1e-8, **kwargs):
         super(RMSprop, self).__init__(**kwargs)
         self.__dict__.update(locals())
         self.lr = K.variable(lr)
@@ -123,23 +194,30 @@ class RMSprop(Optimizer):
 
     def get_updates(self, params, constraints, loss):
         grads = self.get_gradients(loss, params)
-        accumulators = [K.variable(np.zeros(K.get_value(p).shape)) for p in params]
+        shapes = [x.shape for x in K.batch_get_value(params)]
+        accumulators = [K.zeros(shape) for shape in shapes]
+        self.weights = accumulators
         self.updates = []
 
-        for p, g, a, c in zip(params, grads, accumulators, constraints):
+        for p, g, a in zip(params, grads, accumulators):
             # update accumulator
-            new_a = self.rho * a + (1 - self.rho) * K.square(g)
-            self.updates.append((a, new_a))
+            new_a = self.rho * a + (1. - self.rho) * K.square(g)
+            self.updates.append(K.update(a, new_a))
+            new_p = p - self.lr * g / (K.sqrt(new_a) + self.epsilon)
 
-            new_p = p - self.lr * g / K.sqrt(new_a + self.epsilon)
-            self.updates.append((p, c(new_p)))  # apply constraints
+            # apply constraints
+            if p in constraints:
+                c = constraints[p]
+                new_p = c(new_p)
+            self.updates.append(K.update(p, new_p))
         return self.updates
 
     def get_config(self):
-        return {"name": self.__class__.__name__,
-                "lr": float(K.get_value(self.lr)),
-                "rho": float(K.get_value(self.rho)),
-                "epsilon": self.epsilon}
+        config = {'lr': float(K.get_value(self.lr)),
+                  'rho': float(K.get_value(self.rho)),
+                  'epsilon': self.epsilon}
+        base_config = super(RMSprop, self).get_config()
+        return dict(list(base_config.items()) + list(config.items()))
 
 
 class Adagrad(Optimizer):
@@ -152,27 +230,34 @@ class Adagrad(Optimizer):
         lr: float >= 0. Learning rate.
         epsilon: float >= 0.
     '''
-    def __init__(self, lr=0.01, epsilon=1e-6, *args, **kwargs):
+    def __init__(self, lr=0.01, epsilon=1e-8, **kwargs):
         super(Adagrad, self).__init__(**kwargs)
         self.__dict__.update(locals())
         self.lr = K.variable(lr)
 
     def get_updates(self, params, constraints, loss):
         grads = self.get_gradients(loss, params)
-        accumulators = [K.variable(np.zeros(K.get_value(p).shape)) for p in params]
+        shapes = [x.shape for x in K.batch_get_value(params)]
+        accumulators = [K.zeros(shape) for shape in shapes]
+        self.weights = accumulators
         self.updates = []
 
-        for p, g, a, c in zip(params, grads, accumulators, constraints):
+        for p, g, a in zip(params, grads, accumulators):
             new_a = a + K.square(g)  # update accumulator
-            self.updates.append((a, new_a))
-            new_p = p - self.lr * g / K.sqrt(new_a + self.epsilon)
-            self.updates.append((p, c(new_p)))  # apply constraints
+            self.updates.append(K.update(a, new_a))
+            new_p = p - self.lr * g / (K.sqrt(new_a) + self.epsilon)
+            # apply constraints
+            if p in constraints:
+                c = constraints[p]
+                new_p = c(new_p)
+            self.updates.append(K.update(p, new_p))
         return self.updates
 
     def get_config(self):
-        return {"name": self.__class__.__name__,
-                "lr": float(K.get_value(self.lr)),
-                "epsilon": self.epsilon}
+        config = {'lr': float(K.get_value(self.lr)),
+                  'epsilon': self.epsilon}
+        base_config = super(Adagrad, self).get_config()
+        return dict(list(base_config.items()) + list(config.items()))
 
 
 class Adadelta(Optimizer):
@@ -182,46 +267,53 @@ class Adadelta(Optimizer):
     at their default values.
 
     # Arguments
-        lr: float >= 0. Learning rate. It is recommended to leave it at the default value.
+        lr: float >= 0. Learning rate.
+            It is recommended to leave it at the default value.
         rho: float >= 0.
         epsilon: float >= 0. Fuzz factor.
 
     # References
         - [Adadelta - an adaptive learning rate method](http://arxiv.org/abs/1212.5701)
     '''
-    def __init__(self, lr=1.0, rho=0.95, epsilon=1e-6, *args, **kwargs):
+    def __init__(self, lr=1.0, rho=0.95, epsilon=1e-8, **kwargs):
         super(Adadelta, self).__init__(**kwargs)
         self.__dict__.update(locals())
         self.lr = K.variable(lr)
 
     def get_updates(self, params, constraints, loss):
         grads = self.get_gradients(loss, params)
-        accumulators = [K.variable(np.zeros(K.get_value(p).shape)) for p in params]
-        delta_accumulators = [K.variable(np.zeros(K.get_value(p).shape)) for p in params]
+        shapes = [x.shape for x in K.batch_get_value(params)]
+        accumulators = [K.zeros(shape) for shape in shapes]
+        delta_accumulators = [K.zeros(shape) for shape in shapes]
+        self.weights = accumulators + delta_accumulators
         self.updates = []
 
-        for p, g, a, d_a, c in zip(params, grads, accumulators,
-                                   delta_accumulators, constraints):
+        for p, g, a, d_a in zip(params, grads, accumulators, delta_accumulators):
             # update accumulator
-            new_a = self.rho * a + (1 - self.rho) * K.square(g)
-            self.updates.append((a, new_a))
+            new_a = self.rho * a + (1. - self.rho) * K.square(g)
+            self.updates.append(K.update(a, new_a))
 
             # use the new accumulator and the *old* delta_accumulator
             update = g * K.sqrt(d_a + self.epsilon) / K.sqrt(new_a + self.epsilon)
 
             new_p = p - self.lr * update
-            self.updates.append((p, c(new_p)))  # apply constraints
+            # apply constraints
+            if p in constraints:
+                c = constraints[p]
+                new_p = c(new_p)
+            self.updates.append(K.update(p, new_p))
 
             # update delta_accumulator
             new_d_a = self.rho * d_a + (1 - self.rho) * K.square(update)
-            self.updates.append((d_a, new_d_a))
+            self.updates.append(K.update(d_a, new_d_a))
         return self.updates
 
     def get_config(self):
-        return {"name": self.__class__.__name__,
-                "lr": float(K.get_value(self.lr)),
-                "rho": self.rho,
-                "epsilon": self.epsilon}
+        config = {'lr': float(K.get_value(self.lr)),
+                  'rho': self.rho,
+                  'epsilon': self.epsilon}
+        base_config = super(Adadelta, self).get_config()
+        return dict(list(base_config.items()) + list(config.items()))
 
 
 class Adam(Optimizer):
@@ -237,8 +329,8 @@ class Adam(Optimizer):
     # References
         - [Adam - A Method for Stochastic Optimization](http://arxiv.org/abs/1412.6980v8)
     '''
-    def __init__(self, lr=0.001, beta_1=0.9, beta_2=0.999, epsilon=1e-8,
-                 *args, **kwargs):
+    def __init__(self, lr=0.001, beta_1=0.9, beta_2=0.999,
+                 epsilon=1e-8, **kwargs):
         super(Adam, self).__init__(**kwargs)
         self.__dict__.update(locals())
         self.iterations = K.variable(0)
@@ -248,32 +340,40 @@ class Adam(Optimizer):
 
     def get_updates(self, params, constraints, loss):
         grads = self.get_gradients(loss, params)
-        self.updates = [(self.iterations, self.iterations+1.)]
+        self.updates = [K.update_add(self.iterations, 1)]
 
         t = self.iterations + 1
-        lr_t = self.lr * K.sqrt(1 - K.pow(self.beta_2, t)) / (1 - K.pow(self.beta_1, t))
+        lr_t = self.lr * K.sqrt(1. - K.pow(self.beta_2, t)) / (1. - K.pow(self.beta_1, t))
 
-        for p, g, c in zip(params, grads, constraints):
-            # zero init of moment
-            m = K.variable(np.zeros(K.get_value(p).shape))
-            # zero init of velocity
-            v = K.variable(np.zeros(K.get_value(p).shape))
+        shapes = [x.shape for x in K.batch_get_value(params)]
+        ms = [K.zeros(shape) for shape in shapes]
+        vs = [K.zeros(shape) for shape in shapes]
+        self.weights = [self.iterations] + ms + vs
 
-            m_t = (self.beta_1 * m) + (1 - self.beta_1) * g
-            v_t = (self.beta_2 * v) + (1 - self.beta_2) * K.square(g)
+        for p, g, m, v in zip(params, grads, ms, vs):
+            m_t = (self.beta_1 * m) + (1. - self.beta_1) * g
+            v_t = (self.beta_2 * v) + (1. - self.beta_2) * K.square(g)
             p_t = p - lr_t * m_t / (K.sqrt(v_t) + self.epsilon)
 
-            self.updates.append((m, m_t))
-            self.updates.append((v, v_t))
-            self.updates.append((p, c(p_t)))  # apply constraints
+            self.updates.append(K.update(m, m_t))
+            self.updates.append(K.update(v, v_t))
+
+            new_p = p_t
+            # apply constraints
+            if p in constraints:
+                c = constraints[p]
+                new_p = c(new_p)
+            self.updates.append(K.update(p, new_p))
         return self.updates
 
     def get_config(self):
-        return {"name": self.__class__.__name__,
-                "lr": float(K.get_value(self.lr)),
-                "beta_1": float(K.get_value(self.beta_1)),
-                "beta_2": float(K.get_value(self.beta_2)),
-                "epsilon": self.epsilon}
+        config = {'lr': float(K.get_value(self.lr)),
+                  'beta_1': float(K.get_value(self.beta_1)),
+                  'beta_2': float(K.get_value(self.beta_2)),
+                  'epsilon': self.epsilon}
+        base_config = super(Adam, self).get_config()
+        return dict(list(base_config.items()) + list(config.items()))
+
 
 class Adamax(Optimizer):
     '''Adamax optimizer from Adam paper's Section 7. It is a variant
@@ -289,43 +389,133 @@ class Adamax(Optimizer):
     # References
         - [Adam - A Method for Stochastic Optimization](http://arxiv.org/abs/1412.6980v8)
     '''
-    def __init__(self, lr=0.002, beta_1=0.9, beta_2=0.999, epsilon=1e-8,
-                 *args, **kwargs):
+    def __init__(self, lr=0.002, beta_1=0.9, beta_2=0.999,
+                 epsilon=1e-8, **kwargs):
         super(Adamax, self).__init__(**kwargs)
         self.__dict__.update(locals())
-        self.iterations = K.variable(0)
+        self.iterations = K.variable(0.)
         self.lr = K.variable(lr)
         self.beta_1 = K.variable(beta_1)
         self.beta_2 = K.variable(beta_2)
 
     def get_updates(self, params, constraints, loss):
         grads = self.get_gradients(loss, params)
-        self.updates = [(self.iterations, self.iterations+1.)]
+        self.updates = [K.update_add(self.iterations, 1)]
 
         t = self.iterations + 1
-        lr_t = self.lr / (1 - K.pow(self.beta_1, t))
+        lr_t = self.lr / (1. - K.pow(self.beta_1, t))
 
-        for p, g, c in zip(params, grads, constraints):
-            # zero init of 1st moment
-            m = K.variable(np.zeros(K.get_value(p).shape))
-            # zero init of exponentially weighted infinity norm
-            u = K.variable(np.zeros(K.get_value(p).shape))
+        shapes = [x.shape for x in K.batch_get_value(params)]
+        # zero init of 1st moment
+        ms = [K.zeros(shape) for shape in shapes]
+        # zero init of exponentially weighted infinity norm
+        us = [K.zeros(shape) for shape in shapes]
+        self.weights = [self.iterations] + ms + us
 
-            m_t = (self.beta_1 * m) + (1 - self.beta_1) * g
+        for p, g, m, u in zip(params, grads, ms, us):
+
+            m_t = (self.beta_1 * m) + (1. - self.beta_1) * g
             u_t = K.maximum(self.beta_2 * u, K.abs(g))
             p_t = p - lr_t * m_t / (u_t + self.epsilon)
 
-            self.updates.append((m, m_t))
-            self.updates.append((u, u_t))
-            self.updates.append((p, c(p_t)))  # apply constraints
+            self.updates.append(K.update(m, m_t))
+            self.updates.append(K.update(u, u_t))
+
+            new_p = p_t
+            # apply constraints
+            if p in constraints:
+                c = constraints[p]
+                new_p = c(new_p)
+            self.updates.append(K.update(p, new_p))
         return self.updates
 
     def get_config(self):
-        return {"name": self.__class__.__name__,
-                "lr": float(K.get_value(self.lr)),
-                "beta_1": float(K.get_value(self.beta_1)),
-                "beta_2": float(K.get_value(self.beta_2)),
-                "epsilon": self.epsilon}
+        config = {'lr': float(K.get_value(self.lr)),
+                  'beta_1': float(K.get_value(self.beta_1)),
+                  'beta_2': float(K.get_value(self.beta_2)),
+                  'epsilon': self.epsilon}
+        base_config = super(Adamax, self).get_config()
+        return dict(list(base_config.items()) + list(config.items()))
+
+
+class Nadam(Optimizer):
+    '''
+    Nesterov Adam optimizer: Much like Adam is essentially RMSprop with momentum,
+    Nadam is Adam RMSprop with Nesterov momentum.
+
+    Default parameters follow those provided in the paper.
+    It is recommended to leave the parameters of this optimizer
+    at their default values.
+
+    # Arguments
+        lr: float >= 0. Learning rate.
+        beta_1/beta_2: floats, 0 < beta < 1. Generally close to 1.
+        epsilon: float >= 0. Fuzz factor.
+
+    # References
+        - [Nadam report](http://cs229.stanford.edu/proj2015/054_report.pdf)
+        - [On the importance of initialization and momentum in deep learning](http://www.cs.toronto.edu/~fritz/absps/momentum.pdf)
+    '''
+    def __init__(self, lr=0.002, beta_1=0.9, beta_2=0.999,
+                 epsilon=1e-8, schedule_decay=0.004, **kwargs):
+        super(Nadam, self).__init__(**kwargs)
+        self.__dict__.update(locals())
+        self.iterations = K.variable(0.)
+        self.m_schedule = K.variable(1.)
+        self.lr = K.variable(lr)
+        self.beta_1 = K.variable(beta_1)
+        self.beta_2 = K.variable(beta_2)
+        self.schedule_decay = schedule_decay
+
+    def get_updates(self, params, constraints, loss):
+        grads = self.get_gradients(loss, params)
+        self.updates = [K.update_add(self.iterations, 1)]
+
+        t = self.iterations + 1
+
+        # Due to the recommendations in [2], i.e. warming momentum schedule
+        momentum_cache_t = self.beta_1 * (1. - 0.5 * (K.pow(0.96, t * self.schedule_decay)))
+        momentum_cache_t_1 = self.beta_1 * (1. - 0.5 * (K.pow(0.96, (t + 1) * self.schedule_decay)))
+        m_schedule_new = self.m_schedule * momentum_cache_t
+        m_schedule_next = self.m_schedule * momentum_cache_t * momentum_cache_t_1
+        self.updates.append((self.m_schedule, m_schedule_new))
+
+        shapes = [x.shape for x in K.batch_get_value(params)]
+        ms = [K.zeros(shape) for shape in shapes]
+        vs = [K.zeros(shape) for shape in shapes]
+
+        self.weights = [self.iterations] + ms + vs
+
+        for p, g, m, v in zip(params, grads, ms, vs):
+            # the following equations given in [1]
+            g_prime = g / (1. - m_schedule_new)
+            m_t = self.beta_1 * m + (1. - self.beta_1) * g
+            m_t_prime = m_t / (1. - m_schedule_next)
+            v_t = self.beta_2 * v + (1. - self.beta_2) * K.square(g)
+            v_t_prime = v_t / (1. - K.pow(self.beta_2, t))
+            m_t_bar = (1. - momentum_cache_t) * g_prime + momentum_cache_t_1 * m_t_prime
+
+            self.updates.append(K.update(m, m_t))
+            self.updates.append(K.update(v, v_t))
+
+            p_t = p - self.lr * m_t_bar / (K.sqrt(v_t_prime) + self.epsilon)
+            new_p = p_t
+
+            # apply constraints
+            if p in constraints:
+                c = constraints[p]
+                new_p = c(new_p)
+            self.updates.append(K.update(p, new_p))
+        return self.updates
+
+    def get_config(self):
+        config = {'lr': float(K.get_value(self.lr)),
+                  'beta_1': float(K.get_value(self.beta_1)),
+                  'beta_2': float(K.get_value(self.beta_2)),
+                  'epsilon': self.epsilon,
+                  'schedule_decay': self.schedule_decay}
+        base_config = super(Nadam, self).get_config()
+        return dict(list(base_config.items()) + list(config.items()))
 
 
 # aliases
@@ -335,6 +525,7 @@ adagrad = Adagrad
 adadelta = Adadelta
 adam = Adam
 adamax = Adamax
+nadam = Nadam
 
 
 def get(identifier, kwargs=None):

keras/preprocessing/image.py

@@ -1,57 +1,78 @@
+'''Fairly basic set of tools for real-time data augmentation on image data.
+Can easily be extended to include new transformations,
+new preprocessing methods, etc...
+'''
 from __future__ import absolute_import
+from __future__ import print_function
 
 import numpy as np
 import re
-from scipy import ndimage
 from scipy import linalg
-
-from os import listdir
-from os.path import isfile, join
-import random, math
+import scipy.ndimage as ndi
 from six.moves import range
+import os
+import threading
 
-'''
-    Fairly basic set of tools for realtime data augmentation on image data.
-    Can easily be extended to include new transforms, new preprocessing methods, etc...
-'''
+from .. import backend as K
 
-def random_rotation(x, rg, fill_mode="nearest", cval=0.):
-    angle = random.uniform(-rg, rg)
-    x = ndimage.interpolation.rotate(x, angle, axes=(1,2), reshape=False, mode=fill_mode, cval=cval)
+
+def random_rotation(x, rg, row_index=1, col_index=2, channel_index=0,
+                    fill_mode='nearest', cval=0.):
+    theta = np.pi / 180 * np.random.uniform(-rg, rg)
+    rotation_matrix = np.array([[np.cos(theta), -np.sin(theta), 0],
+                                [np.sin(theta), np.cos(theta), 0],
+                                [0, 0, 1]])
+
+    h, w = x.shape[row_index], x.shape[col_index]
+    transform_matrix = transform_matrix_offset_center(rotation_matrix, h, w)
+    x = apply_transform(x, transform_matrix, channel_index, fill_mode, cval)
     return x
 
-def random_shift(x, wrg, hrg, fill_mode="nearest", cval=0.):
-    crop_left_pixels = 0
-    crop_right_pixels = 0
-    crop_top_pixels = 0
-    crop_bottom_pixels = 0
 
-    original_w = x.shape[1]
-    original_h = x.shape[2]
+def random_shift(x, wrg, hrg, row_index=1, col_index=2, channel_index=0,
+                 fill_mode='nearest', cval=0.):
+    h, w = x.shape[row_index], x.shape[col_index]
+    tx = np.random.uniform(-hrg, hrg) * h
+    ty = np.random.uniform(-wrg, wrg) * w
+    translation_matrix = np.array([[1, 0, tx],
+                                   [0, 1, ty],
+                                   [0, 0, 1]])
 
-    if wrg:
-        crop = random.uniform(0., wrg)
-        split = random.uniform(0, 1)
-        crop_left_pixels = int(split*crop*x.shape[1])
-        crop_right_pixels = int((1-split)*crop*x.shape[1])
-
-    if hrg:
-        crop = random.uniform(0., hrg)
-        split = random.uniform(0, 1)
-        crop_top_pixels = int(split*crop*x.shape[2])
-        crop_bottom_pixels = int((1-split)*crop*x.shape[2])
-
-    x = ndimage.interpolation.shift(x, (0, crop_left_pixels, crop_top_pixels), mode=fill_mode, cval=cval)
+    transform_matrix = translation_matrix  # no need to do offset
+    x = apply_transform(x, transform_matrix, channel_index, fill_mode, cval)
     return x
 
-def horizontal_flip(x):
-    for i in range(x.shape[0]):
-        x[i] = np.fliplr(x[i])
+
+def random_shear(x, intensity, row_index=1, col_index=2, channel_index=0,
+                 fill_mode='nearest', cval=0.):
+    shear = np.random.uniform(-intensity, intensity)
+    shear_matrix = np.array([[1, -np.sin(shear), 0],
+                             [0, np.cos(shear), 0],
+                             [0, 0, 1]])
+
+    h, w = x.shape[row_index], x.shape[col_index]
+    transform_matrix = transform_matrix_offset_center(shear_matrix, h, w)
+    x = apply_transform(x, transform_matrix, channel_index, fill_mode, cval)
     return x
 
-def vertical_flip(x):
-    for i in range(x.shape[0]):
-        x[i] = np.flipud(x[i])
+
+def random_zoom(x, zoom_range, row_index=1, col_index=2, channel_index=0,
+                fill_mode='nearest', cval=0.):
+    if len(zoom_range) != 2:
+        raise Exception('zoom_range should be a tuple or list of two floats. '
+                        'Received arg: ', zoom_range)
+
+    if zoom_range[0] == 1 and zoom_range[1] == 1:
+        zx, zy = 1, 1
+    else:
+        zx, zy = np.random.uniform(zoom_range[0], zoom_range[1], 2)
+    zoom_matrix = np.array([[zx, 0, 0],
+                            [0, zy, 0],
+                            [0, 0, 1]])
+
+    h, w = x.shape[row_index], x.shape[col_index]
+    transform_matrix = transform_matrix_offset_center(zoom_matrix, h, w)
+    x = apply_transform(x, transform_matrix, channel_index, fill_mode, cval)
     return x
 
 
@@ -59,188 +80,538 @@ def random_barrel_transform(x, intensity):
     # TODO
     pass
 
-def random_shear(x, intensity):
-    # TODO
-    pass
 
-def random_channel_shift(x, rg):
-    # TODO
-    pass
-
-def random_zoom(x, rg, fill_mode="nearest", cval=0.):
-    zoom_w = random.uniform(1.-rg, 1.)
-    zoom_h = random.uniform(1.-rg, 1.)
-    x = ndimage.interpolation.zoom(x, zoom=(1., zoom_w, zoom_h), mode=fill_mode, cval=cval)
-    return x # shape of result will be different from shape of input!
-
-
-
-
-def array_to_img(x, scale=True):
-    from PIL import Image
-    x = x.transpose(1, 2, 0) 
-    if scale:
-        x += max(-np.min(x), 0)
-        x /= np.max(x)
-        x *= 255
-    if x.shape[2] == 3:
-        # RGB
-        return Image.fromarray(x.astype("uint8"), "RGB")
-    else:
-        # grayscale
-        return Image.fromarray(x[:,:,0].astype("uint8"), "L")
-
-
-def img_to_array(img):
-    x = np.asarray(img, dtype='float32')
-    if len(x.shape)==3:
-        # RGB: height, width, channel -> channel, height, width
-        x = x.transpose(2, 0, 1)
-    else:
-        # grayscale: height, width -> channel, height, width
-        x = x.reshape((1, x.shape[0], x.shape[1]))
+def random_channel_shift(x, intensity, channel_index=0):
+    x = np.rollaxis(x, channel_index, 0)
+    min_x, max_x = np.min(x), np.max(x)
+    channel_images = [np.clip(x_channel + np.random.uniform(-intensity, intensity), min_x, max_x)
+                      for x_channel in x]
+    x = np.stack(channel_images, axis=0)
+    x = np.rollaxis(x, 0, channel_index+1)
     return x
 
 
-def load_img(path, grayscale=False):
+def transform_matrix_offset_center(matrix, x, y):
+    o_x = float(x) / 2 + 0.5
+    o_y = float(y) / 2 + 0.5
+    offset_matrix = np.array([[1, 0, o_x], [0, 1, o_y], [0, 0, 1]])
+    reset_matrix = np.array([[1, 0, -o_x], [0, 1, -o_y], [0, 0, 1]])
+    transform_matrix = np.dot(np.dot(offset_matrix, matrix), reset_matrix)
+    return transform_matrix
+
+
+def apply_transform(x, transform_matrix, channel_index=0, fill_mode='nearest', cval=0.):
+    x = np.rollaxis(x, channel_index, 0)
+    final_affine_matrix = transform_matrix[:2, :2]
+    final_offset = transform_matrix[:2, 2]
+    channel_images = [ndi.interpolation.affine_transform(x_channel, final_affine_matrix,
+                      final_offset, order=0, mode=fill_mode, cval=cval) for x_channel in x]
+    x = np.stack(channel_images, axis=0)
+    x = np.rollaxis(x, 0, channel_index+1)
+    return x
+
+
+def flip_axis(x, axis):
+    x = np.asarray(x).swapaxes(axis, 0)
+    x = x[::-1, ...]
+    x = x.swapaxes(0, axis)
+    return x
+
+
+def array_to_img(x, dim_ordering='default', scale=True):
+    from PIL import Image
+    if dim_ordering == 'default':
+        dim_ordering = K.image_dim_ordering()
+    if dim_ordering == 'th':
+        x = x.transpose(1, 2, 0)
+    if scale:
+        x += max(-np.min(x), 0)
+        x_max = np.max(x)
+        if x_max != 0:
+            x /= x_max
+        x *= 255
+    if x.shape[2] == 3:
+        # RGB
+        return Image.fromarray(x.astype('uint8'), 'RGB')
+    elif x.shape[2] == 1:
+        # grayscale
+        return Image.fromarray(x[:, :, 0].astype('uint8'), 'L')
+    else:
+        raise Exception('Unsupported channel number: ', x.shape[2])
+
+
+def img_to_array(img, dim_ordering='default'):
+    if dim_ordering == 'default':
+        dim_ordering = K.image_dim_ordering()
+    if dim_ordering not in ['th', 'tf']:
+        raise Exception('Unknown dim_ordering: ', dim_ordering)
+    # image has dim_ordering (height, width, channel)
+    x = np.asarray(img, dtype='float32')
+    if len(x.shape) == 3:
+        if dim_ordering == 'th':
+            x = x.transpose(2, 0, 1)
+    elif len(x.shape) == 2:
+        if dim_ordering == 'th':
+            x = x.reshape((1, x.shape[0], x.shape[1]))
+        else:
+            x = x.reshape((x.shape[0], x.shape[1], 1))
+    else:
+        raise Exception('Unsupported image shape: ', x.shape)
+    return x
+
+
+def load_img(path, grayscale=False, target_size=None):
     from PIL import Image
     img = Image.open(path)
     if grayscale:
         img = img.convert('L')
-    else: # Assure 3 channel even when loaded image is grayscale
+    else:  # Ensure 3 channel even when loaded image is grayscale
         img = img.convert('RGB')
+    if target_size:
+        img = img.resize((target_size[1], target_size[0]))
     return img
 
 
 def list_pictures(directory, ext='jpg|jpeg|bmp|png'):
-    return [join(directory,f) for f in listdir(directory) \
-        if isfile(join(directory,f)) and re.match('([\w]+\.(?:' + ext + '))', f)]
-
+    return [os.path.join(directory, f) for f in os.listdir(directory)
+            if os.path.isfile(os.path.join(directory, f)) and re.match('([\w]+\.(?:' + ext + '))', f)]
 
 
 class ImageDataGenerator(object):
-    '''
-        Generate minibatches with 
-        realtime data augmentation.
-    '''
-    def __init__(self, 
-            featurewise_center=True, # set input mean to 0 over the dataset
-            samplewise_center=False, # set each sample mean to 0
-            featurewise_std_normalization=True, # divide inputs by std of the dataset
-            samplewise_std_normalization=False, # divide each input by its std
+    '''Generate minibatches with
+    real-time data augmentation.
 
-            zca_whitening=False, # apply ZCA whitening
-            rotation_range=0., # degrees (0 to 180)
-            width_shift_range=0., # fraction of total width
-            height_shift_range=0., # fraction of total height
-            horizontal_flip=False,
-            vertical_flip=False,
-        ):
+    # Arguments
+        featurewise_center: set input mean to 0 over the dataset.
+        samplewise_center: set each sample mean to 0.
+        featurewise_std_normalization: divide inputs by std of the dataset.
+        samplewise_std_normalization: divide each input by its std.
+        zca_whitening: apply ZCA whitening.
+        rotation_range: degrees (0 to 180).
+        width_shift_range: fraction of total width.
+        height_shift_range: fraction of total height.
+        shear_range: shear intensity (shear angle in radians).
+        zoom_range: amount of zoom. if scalar z, zoom will be randomly picked
+            in the range [1-z, 1+z]. A sequence of two can be passed instead
+            to select this range.
+        channel_shift_range: shift range for each channels.
+        fill_mode: points outside the boundaries are filled according to the
+            given mode ('constant', 'nearest', 'reflect' or 'wrap'). Default
+            is 'nearest'.
+        cval: value used for points outside the boundaries when fill_mode is
+            'constant'. Default is 0.
+        horizontal_flip: whether to randomly flip images horizontally.
+        vertical_flip: whether to randomly flip images vertically.
+        rescale: rescaling factor. If None or 0, no rescaling is applied,
+            otherwise we multiply the data by the value provided (before applying
+            any other transformation).
+        dim_ordering: 'th' or 'tf'. In 'th' mode, the channels dimension
+            (the depth) is at index 1, in 'tf' mode it is at index 3.
+            It defaults to the `image_dim_ordering` value found in your
+            Keras config file at `~/.keras/keras.json`.
+            If you never set it, then it will be "th".
+    '''
+    def __init__(self,
+                 featurewise_center=False,
+                 samplewise_center=False,
+                 featurewise_std_normalization=False,
+                 samplewise_std_normalization=False,
+                 zca_whitening=False,
+                 rotation_range=0.,
+                 width_shift_range=0.,
+                 height_shift_range=0.,
+                 shear_range=0.,
+                 zoom_range=0.,
+                 channel_shift_range=0.,
+                 fill_mode='nearest',
+                 cval=0.,
+                 horizontal_flip=False,
+                 vertical_flip=False,
+                 rescale=None,
+                 dim_ordering='default'):
+        if dim_ordering == 'default':
+            dim_ordering = K.image_dim_ordering()
         self.__dict__.update(locals())
         self.mean = None
         self.std = None
         self.principal_components = None
+        self.rescale = rescale
 
+        if dim_ordering not in {'tf', 'th'}:
+            raise Exception('dim_ordering should be "tf" (channel after row and '
+                            'column) or "th" (channel before row and column). '
+                            'Received arg: ', dim_ordering)
+        self.dim_ordering = dim_ordering
+        if dim_ordering == 'th':
+            self.channel_index = 1
+            self.row_index = 2
+            self.col_index = 3
+        if dim_ordering == 'tf':
+            self.channel_index = 3
+            self.row_index = 1
+            self.col_index = 2
 
-    def flow(self, X, y, batch_size=32, shuffle=False, seed=None, save_to_dir=None, save_prefix="", save_format="jpeg"):
-        if seed:
-            random.seed(seed)
+        if np.isscalar(zoom_range):
+            self.zoom_range = [1 - zoom_range, 1 + zoom_range]
+        elif len(zoom_range) == 2:
+            self.zoom_range = [zoom_range[0], zoom_range[1]]
+        else:
+            raise Exception('zoom_range should be a float or '
+                            'a tuple or list of two floats. '
+                            'Received arg: ', zoom_range)
 
-        if shuffle:
-            seed = random.randint(1, 10e6)
-            np.random.seed(seed)
-            np.random.shuffle(X)
-            np.random.seed(seed)
-            np.random.shuffle(y)
-
-        nb_batch = int(math.ceil(float(X.shape[0])/batch_size))
-        for b in range(nb_batch):
-            batch_end = (b+1)*batch_size
-            if batch_end > X.shape[0]:
-                nb_samples = X.shape[0] - b*batch_size
-            else:
-                nb_samples = batch_size
-
-            bX = np.zeros(tuple([nb_samples]+list(X.shape)[1:]))
-            for i in range(nb_samples):
-                x = X[b*batch_size+i]
-                x = self.random_transform(x.astype("float32"))
-                x = self.standardize(x)
-                bX[i] = x
-
-            if save_to_dir:
-                for i in range(nb_samples):
-                    img = array_to_img(bX[i], scale=True)
-                    img.save(save_to_dir + "/" + save_prefix + "_" + str(i) + "." + save_format)
-
-            yield bX, y[b*batch_size:b*batch_size+nb_samples]
+    def flow(self, X, y=None, batch_size=32, shuffle=True, seed=None,
+             save_to_dir=None, save_prefix='', save_format='jpeg'):
+        return NumpyArrayIterator(
+            X, y, self,
+            batch_size=batch_size, shuffle=shuffle, seed=seed,
+            dim_ordering=self.dim_ordering,
+            save_to_dir=save_to_dir, save_prefix=save_prefix, save_format=save_format)
 
+    def flow_from_directory(self, directory,
+                            target_size=(256, 256), color_mode='rgb',
+                            classes=None, class_mode='categorical',
+                            batch_size=32, shuffle=True, seed=None,
+                            save_to_dir=None, save_prefix='', save_format='jpeg'):
+        return DirectoryIterator(
+            directory, self,
+            target_size=target_size, color_mode=color_mode,
+            classes=classes, class_mode=class_mode,
+            dim_ordering=self.dim_ordering,
+            batch_size=batch_size, shuffle=shuffle, seed=seed,
+            save_to_dir=save_to_dir, save_prefix=save_prefix, save_format=save_format)
 
     def standardize(self, x):
+        if self.rescale:
+            x *= self.rescale
+        # x is a single image, so it doesn't have image number at index 0
+        img_channel_index = self.channel_index - 1
+        if self.samplewise_center:
+            x -= np.mean(x, axis=img_channel_index, keepdims=True)
+        if self.samplewise_std_normalization:
+            x /= (np.std(x, axis=img_channel_index, keepdims=True) + 1e-7)
+
         if self.featurewise_center:
             x -= self.mean
         if self.featurewise_std_normalization:
-            x /= self.std
+            x /= (self.std + 1e-7)
 
         if self.zca_whitening:
-            flatx = np.reshape(x, (x.shape[0]*x.shape[1]*x.shape[2]))
+            flatx = np.reshape(x, (x.size))
             whitex = np.dot(flatx, self.principal_components)
             x = np.reshape(whitex, (x.shape[0], x.shape[1], x.shape[2]))
 
-        if self.samplewise_center:
-            x -= np.mean(x)
-        if self.samplewise_std_normalization:
-            x /= np.std(x)
-
         return x
 
-
     def random_transform(self, x):
+        # x is a single image, so it doesn't have image number at index 0
+        img_row_index = self.row_index - 1
+        img_col_index = self.col_index - 1
+        img_channel_index = self.channel_index - 1
+
+        # use composition of homographies to generate final transform that needs to be applied
         if self.rotation_range:
-            x = random_rotation(x, self.rotation_range)
-        if self.width_shift_range or self.height_shift_range:
-            x = random_shift(x, self.width_shift_range, self.height_shift_range)
+            theta = np.pi / 180 * np.random.uniform(-self.rotation_range, self.rotation_range)
+        else:
+            theta = 0
+        rotation_matrix = np.array([[np.cos(theta), -np.sin(theta), 0],
+                                    [np.sin(theta), np.cos(theta), 0],
+                                    [0, 0, 1]])
+        if self.height_shift_range:
+            tx = np.random.uniform(-self.height_shift_range, self.height_shift_range) * x.shape[img_row_index]
+        else:
+            tx = 0
+
+        if self.width_shift_range:
+            ty = np.random.uniform(-self.width_shift_range, self.width_shift_range) * x.shape[img_col_index]
+        else:
+            ty = 0
+
+        translation_matrix = np.array([[1, 0, tx],
+                                       [0, 1, ty],
+                                       [0, 0, 1]])
+        if self.shear_range:
+            shear = np.random.uniform(-self.shear_range, self.shear_range)
+        else:
+            shear = 0
+        shear_matrix = np.array([[1, -np.sin(shear), 0],
+                                 [0, np.cos(shear), 0],
+                                 [0, 0, 1]])
+
+        if self.zoom_range[0] == 1 and self.zoom_range[1] == 1:
+            zx, zy = 1, 1
+        else:
+            zx, zy = np.random.uniform(self.zoom_range[0], self.zoom_range[1], 2)
+        zoom_matrix = np.array([[zx, 0, 0],
+                                [0, zy, 0],
+                                [0, 0, 1]])
+
+        transform_matrix = np.dot(np.dot(np.dot(rotation_matrix, translation_matrix), shear_matrix), zoom_matrix)
+
+        h, w = x.shape[img_row_index], x.shape[img_col_index]
+        transform_matrix = transform_matrix_offset_center(transform_matrix, h, w)
+        x = apply_transform(x, transform_matrix, img_channel_index,
+                            fill_mode=self.fill_mode, cval=self.cval)
+        if self.channel_shift_range != 0:
+            x = random_channel_shift(x, self.channel_shift_range, img_channel_index)
+
         if self.horizontal_flip:
-            if random.random() < 0.5:
-                x = horizontal_flip(x)
+            if np.random.random() < 0.5:
+                x = flip_axis(x, img_col_index)
+
         if self.vertical_flip:
-            if random.random() < 0.5:
-                x = vertical_flip(x)
+            if np.random.random() < 0.5:
+                x = flip_axis(x, img_row_index)
 
         # TODO:
-        # zoom
+        # channel-wise normalization
         # barrel/fisheye
-        # shearing
-        # channel shifting
         return x
 
     def fit(self, X,
-            augment=False,  # fit on randomly augmented samples
-            rounds=1,  # if augment, how many augmentation passes over the data do we use
+            augment=False,
+            rounds=1,
             seed=None):
-        '''
-            Required for featurewise_center, featurewise_std_normalization and zca_whitening.
+        '''Required for featurewise_center, featurewise_std_normalization
+        and zca_whitening.
+
+        # Arguments
+            X: Numpy array, the data to fit on.
+            augment: whether to fit on randomly augmented samples
+            rounds: if `augment`,
+                how many augmentation passes to do over the data
+            seed: random seed.
         '''
         X = np.copy(X)
         if augment:
-            aX = np.zeros(tuple([rounds*X.shape[0]]+list(X.shape)[1:]))
+            aX = np.zeros(tuple([rounds * X.shape[0]] + list(X.shape)[1:]))
             for r in range(rounds):
                 for i in range(X.shape[0]):
-                    img = array_to_img(X[i])
-                    img = self.random_transform(img)
-                    aX[i+r*X.shape[0]] = img_to_array(img)
+                    aX[i + r * X.shape[0]] = self.random_transform(X[i])
             X = aX
 
         if self.featurewise_center:
             self.mean = np.mean(X, axis=0)
             X -= self.mean
+
         if self.featurewise_std_normalization:
             self.std = np.std(X, axis=0)
-            X /= self.std
+            X /= (self.std + 1e-7)
 
         if self.zca_whitening:
-            flatX = np.reshape(X, (X.shape[0], X.shape[1]*X.shape[2]*X.shape[3]))
-            fudge = 10e-6
+            flatX = np.reshape(X, (X.shape[0], X.shape[1] * X.shape[2] * X.shape[3]))
             sigma = np.dot(flatX.T, flatX) / flatX.shape[1]
             U, S, V = linalg.svd(sigma)
-            self.principal_components = np.dot(np.dot(U, np.diag(1. / np.sqrt(S + fudge))), U.T)
+            self.principal_components = np.dot(np.dot(U, np.diag(1. / np.sqrt(S + 10e-7))), U.T)
+
+
+class Iterator(object):
+
+    def __init__(self, N, batch_size, shuffle, seed):
+        self.N = N
+        self.batch_size = batch_size
+        self.shuffle = shuffle
+        self.batch_index = 0
+        self.total_batches_seen = 0
+        self.lock = threading.Lock()
+        self.index_generator = self._flow_index(N, batch_size, shuffle, seed)
+
+    def reset(self):
+        self.batch_index = 0
+
+    def _flow_index(self, N, batch_size=32, shuffle=False, seed=None):
+        # ensure self.batch_index is 0
+        self.reset()
+        while 1:
+            if self.batch_index == 0:
+                index_array = np.arange(N)
+                if shuffle:
+                    if seed is not None:
+                        np.random.seed(seed + self.total_batches_seen)
+                    index_array = np.random.permutation(N)
+
+            current_index = (self.batch_index * batch_size) % N
+            if N >= current_index + batch_size:
+                current_batch_size = batch_size
+                self.batch_index += 1
+            else:
+                current_batch_size = N - current_index
+                self.batch_index = 0
+            self.total_batches_seen += 1
+            yield (index_array[current_index: current_index + current_batch_size],
+                   current_index, current_batch_size)
+
+    def __iter__(self):
+        # needed if we want to do something like:
+        # for x, y in data_gen.flow(...):
+        return self
+
+    def __next__(self, *args, **kwargs):
+        return self.next(*args, **kwargs)
+
+
+class NumpyArrayIterator(Iterator):
+
+    def __init__(self, X, y, image_data_generator,
+                 batch_size=32, shuffle=False, seed=None,
+                 dim_ordering='default',
+                 save_to_dir=None, save_prefix='', save_format='jpeg'):
+        if y is not None and len(X) != len(y):
+            raise Exception('X (images tensor) and y (labels) '
+                            'should have the same length. '
+                            'Found: X.shape = %s, y.shape = %s' % (np.asarray(X).shape, np.asarray(y).shape))
+        if dim_ordering == 'default':
+            dim_ordering = K.image_dim_ordering()
+        self.X = X
+        self.y = y
+        self.image_data_generator = image_data_generator
+        self.dim_ordering = dim_ordering
+        self.save_to_dir = save_to_dir
+        self.save_prefix = save_prefix
+        self.save_format = save_format
+        super(NumpyArrayIterator, self).__init__(X.shape[0], batch_size, shuffle, seed)
+
+    def next(self):
+        # for python 2.x.
+        # Keeps under lock only the mechanism which advances
+        # the indexing of each batch
+        # see http://anandology.com/blog/using-iterators-and-generators/
+        with self.lock:
+            index_array, current_index, current_batch_size = next(self.index_generator)
+        # The transformation of images is not under thread lock so it can be done in parallel
+        batch_x = np.zeros(tuple([current_batch_size] + list(self.X.shape)[1:]))
+        for i, j in enumerate(index_array):
+            x = self.X[j]
+            x = self.image_data_generator.random_transform(x.astype('float32'))
+            x = self.image_data_generator.standardize(x)
+            batch_x[i] = x
+        if self.save_to_dir:
+            for i in range(current_batch_size):
+                img = array_to_img(batch_x[i], self.dim_ordering, scale=True)
+                fname = '{prefix}_{index}_{hash}.{format}'.format(prefix=self.save_prefix,
+                                                                  index=current_index + i,
+                                                                  hash=np.random.randint(1e4),
+                                                                  format=self.save_format)
+                img.save(os.path.join(self.save_to_dir, fname))
+        if self.y is None:
+            return batch_x
+        batch_y = self.y[index_array]
+        return batch_x, batch_y
+
+
+class DirectoryIterator(Iterator):
+
+    def __init__(self, directory, image_data_generator,
+                 target_size=(256, 256), color_mode='rgb',
+                 dim_ordering='default',
+                 classes=None, class_mode='categorical',
+                 batch_size=32, shuffle=True, seed=None,
+                 save_to_dir=None, save_prefix='', save_format='jpeg'):
+        if dim_ordering == 'default':
+            dim_ordering = K.image_dim_ordering()
+        self.directory = directory
+        self.image_data_generator = image_data_generator
+        self.target_size = tuple(target_size)
+        if color_mode not in {'rgb', 'grayscale'}:
+            raise ValueError('Invalid color mode:', color_mode,
+                             '; expected "rgb" or "grayscale".')
+        self.color_mode = color_mode
+        self.dim_ordering = dim_ordering
+        if self.color_mode == 'rgb':
+            if self.dim_ordering == 'tf':
+                self.image_shape = self.target_size + (3,)
+            else:
+                self.image_shape = (3,) + self.target_size
+        else:
+            if self.dim_ordering == 'tf':
+                self.image_shape = self.target_size + (1,)
+            else:
+                self.image_shape = (1,) + self.target_size
+        self.classes = classes
+        if class_mode not in {'categorical', 'binary', 'sparse', None}:
+            raise ValueError('Invalid class_mode:', class_mode,
+                             '; expected one of "categorical", '
+                             '"binary", "sparse", or None.')
+        self.class_mode = class_mode
+        self.save_to_dir = save_to_dir
+        self.save_prefix = save_prefix
+        self.save_format = save_format
+
+        white_list_formats = {'png', 'jpg', 'jpeg', 'bmp'}
+
+        # first, count the number of samples and classes
+        self.nb_sample = 0
+
+        if not classes:
+            classes = []
+            for subdir in sorted(os.listdir(directory)):
+                if os.path.isdir(os.path.join(directory, subdir)):
+                    classes.append(subdir)
+        self.nb_class = len(classes)
+        self.class_indices = dict(zip(classes, range(len(classes))))
+
+        for subdir in classes:
+            subpath = os.path.join(directory, subdir)
+            for fname in os.listdir(subpath):
+                is_valid = False
+                for extension in white_list_formats:
+                    if fname.lower().endswith('.' + extension):
+                        is_valid = True
+                        break
+                if is_valid:
+                    self.nb_sample += 1
+        print('Found %d images belonging to %d classes.' % (self.nb_sample, self.nb_class))
+
+        # second, build an index of the images in the different class subfolders
+        self.filenames = []
+        self.classes = np.zeros((self.nb_sample,), dtype='int32')
+        i = 0
+        for subdir in classes:
+            subpath = os.path.join(directory, subdir)
+            for fname in os.listdir(subpath):
+                is_valid = False
+                for extension in white_list_formats:
+                    if fname.lower().endswith('.' + extension):
+                        is_valid = True
+                        break
+                if is_valid:
+                    self.classes[i] = self.class_indices[subdir]
+                    self.filenames.append(os.path.join(subdir, fname))
+                    i += 1
+        super(DirectoryIterator, self).__init__(self.nb_sample, batch_size, shuffle, seed)
+
+    def next(self):
+        with self.lock:
+            index_array, current_index, current_batch_size = next(self.index_generator)
+        # The transformation of images is not under thread lock so it can be done in parallel
+        batch_x = np.zeros((current_batch_size,) + self.image_shape)
+        grayscale = self.color_mode == 'grayscale'
+        # build batch of image data
+        for i, j in enumerate(index_array):
+            fname = self.filenames[j]
+            img = load_img(os.path.join(self.directory, fname), grayscale=grayscale, target_size=self.target_size)
+            x = img_to_array(img, dim_ordering=self.dim_ordering)
+            x = self.image_data_generator.random_transform(x)
+            x = self.image_data_generator.standardize(x)
+            batch_x[i] = x
+        # optionally save augmented images to disk for debugging purposes
+        if self.save_to_dir:
+            for i in range(current_batch_size):
+                img = array_to_img(batch_x[i], self.dim_ordering, scale=True)
+                fname = '{prefix}_{index}_{hash}.{format}'.format(prefix=self.save_prefix,
+                                                                  index=current_index + i,
+                                                                  hash=np.random.randint(1e4),
+                                                                  format=self.save_format)
+                img.save(os.path.join(self.save_to_dir, fname))
+        # build batch of labels
+        if self.class_mode == 'sparse':
+            batch_y = self.classes[index_array]
+        elif self.class_mode == 'binary':
+            batch_y = self.classes[index_array].astype('float32')
+        elif self.class_mode == 'categorical':
+            batch_y = np.zeros((len(batch_x), self.nb_class), dtype='float32')
+            for i, label in enumerate(self.classes[index_array]):
+                batch_y[i, label] = 1.
+        else:
+            return batch_x
+        return batch_x, batch_y