Prepare 0.3.3 release (last release before 1.0).

add sign operation to backends

.gitignore

@@ -1,6 +1,19 @@
 *.DS_Store
 *.pyc
+*.swp
 temp/*
+dist/*
 build/*
 keras/datasets/data/*
-keras/datasets/temp/*
+keras/datasets/temp/*
+docs/site/*
+docs/theme/*
+tags
+Keras.egg-info
+
+# test-related
+.coverage
+.cache
+
+# developer environments
+.idea

.travis.yml

@@ -0,0 +1,71 @@
+sudo: required
+dist: trusty
+language: python
+matrix:
+    include:
+        - python: 3.4
+          env: KERAS_BACKEND=theano
+        - python: 3.4
+          env: KERAS_BACKEND=tensorflow
+        - python: 2.7
+          env: KERAS_BACKEND=theano
+        - python: 2.7
+          env: KERAS_BACKEND=tensorflow
+        - python: 2.7
+          env: KERAS_BACKEND=theano TEST_MODE=INTEGRATION_TESTS
+        - python: 2.7
+          env: KERAS_BACKEND=theano TEST_MODE=PEP8
+install:
+  # code below is taken from http://conda.pydata.org/docs/travis.html
+  # We do this conditionally because it saves us some downloading if the
+  # version is the same.
+  - if [[ "$TRAVIS_PYTHON_VERSION" == "2.7" ]]; then
+      wget https://repo.continuum.io/miniconda/Miniconda-latest-Linux-x86_64.sh -O miniconda.sh;
+    else
+      wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh;
+    fi
+  - bash miniconda.sh -b -p $HOME/miniconda
+  - export PATH="$HOME/miniconda/bin:$PATH"
+  - hash -r
+  - conda config --set always_yes yes --set changeps1 no
+  - conda update -q conda
+  # Useful for debugging any issues with conda
+  - conda info -a
+
+  - conda create -q -n test-environment python=$TRAVIS_PYTHON_VERSION numpy scipy matplotlib pandas pytest h5py
+  - source activate test-environment
+  - pip install pytest-cov python-coveralls pytest-xdist coverage==3.7.1 #we need this version of coverage for coveralls.io to work
+  - pip install pep8 pytest-pep8
+  - pip install git+git://github.com/Theano/Theano.git
+
+  # install PIL for preprocessing tests
+  - if [[ "$TRAVIS_PYTHON_VERSION" == "2.7" ]]; then
+      conda install pil;
+    elif [[ "$TRAVIS_PYTHON_VERSION" == "3.4" ]]; then
+      conda install Pillow;
+    fi
+
+  - python setup.py install
+
+  # install TensorFlow
+  - if [[ "$TRAVIS_PYTHON_VERSION" == "2.7" ]]; then
+      pip install https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-0.6.0-cp27-none-linux_x86_64.whl;
+    elif [[ "$TRAVIS_PYTHON_VERSION" == "3.4" ]]; then
+      pip install https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-0.6.0-cp34-none-linux_x86_64.whl;
+    fi
+# command to run tests
+script:
+  # run keras backend init to initialize backend config
+  - python -c "import keras.backend"
+  # set up keras backend
+  - sed -i -e 's/"backend":[[:space:]]*"[^"]*/"backend":\ "'$KERAS_BACKEND'/g' ~/.keras/keras.json;
+  - echo -e "Running tests with the following config:\n$(cat ~/.keras/keras.json)"
+  - if [[ "$TEST_MODE" == "INTEGRATION_TESTS" ]]; then
+       PYTHONPATH=$PWD:$PYTHONPATH py.test tests/integration_tests;
+    elif [[ "$TEST_MODE" == "PEP8" ]]; then
+       PYTHONPATH=$PWD:$PYTHONPATH py.test --pep8 -m pep8 -n0;
+    else
+       PYTHONPATH=$PWD:$PYTHONPATH py.test tests/ --ignore=tests/integration_tests;
+    fi
+after_success:
+  - coveralls

CONTRIBUTING.md

@@ -0,0 +1,65 @@
+# On Github Issues and Pull Requests
+
+Found a bug? Have a new feature to suggest? Want to contribute changes to the codebase? Make sure to read this first.
+
+## Bug reporting
+
+Your code doesn't work, and you have determined that the issue lies with Keras? Follow these steps to report a bug.
+
+1. Your bug may already be fixed. Make sure to update to the current Keras master branch, as well as the latest Theano/TensorFlow master branch.
+To easily update Theano: `pip install git+git://github.com/Theano/Theano.git --upgrade`
+
+2. Search for similar issues. Make sure to delete `is:open` on the issue search to find solved tickets as well. It's possible somebody has encountered this bug already. Also remember to check out Keras' [FAQ](http://keras.io/faq/). Still having a problem? Open an issue on Github to let us know.
+
+3. Make sure you provide us with useful information about your configuration: what OS are you using? What Keras backend are you using? Are you running on GPU? If so, what is your version of Cuda, of cuDNN? What is your GPU?
+
+4. Provide us with a script to reproduce the issue. This script should be runnable as-is and should not require external data download (use randomly generated data if you need to run a model on some test data). We recommend that you use Github Gists to post your code. Any issue that cannot be reproduced is likely to be closed.
+
+5. If possible, take a stab at fixing the bug yourself --if you can!
+
+The more information you provide, the easier it is for us to validate that there is a bug and the faster we'll be able to take action. If you want your issue to be resolved quickly, following the steps above is crucial.
+
+
+## Requesting a Feature
+
+You can also use Github issues to request features you would like to see in Keras, or changes in the Keras API.
+
+1. Provide a clear and detailed explanation of the feature you want and why it's important to add. Keep in mind that we want features that will be useful to the majority of our users and not just a small subset. If you're just targeting a minority of users, consider writing an add-on library for Keras. It is crucial for Keras to avoid bloating the API and codebase.
+
+2. Provide code snippets demonstrating the API you have in mind and illustrating the use cases of your feature. Of course, you don't need to write any real code at this point!
+
+3. After discussing the feature you may choose to attempt a Pull Request. If you're at all able, start writing some code. We always have more work to do than time to do it. If you can write some code then that will speed the process along.
+
+## Pull Requests
+
+We love pull requests. Here's a quick guide:
+
+1. If your PR introduces a change in functionality, make sure you start by opening an issue to discuss whether the change should be made, and how to handle it. This will save you from having your PR closed down the road! Of course, if your PR is a simple bug fix, you don't need to do that.
+
+2. Write the code. This is the hard part!
+
+3. Make sure any new function or class you introduce has proper docstrings. Make sure any code you touch still has up-to-date docstrings and documentation.
+
+4. Write tests. Your code should have full unit test coverage. If you want to see your PR merged promptly, this is crucial.
+
+5. Run our test suite locally. It's easy: from the Keras folder, simply run: `py.test tests/`.
+  - You will need to install `pytest`, `coveralls`, `pytest-cov`, `pytest-xdist`: `pip install pytest pytest-cov python-coveralls pytest-xdist pep8 pytest-pep8`
+
+6. Make sure all tests are passing:
+  - with the Theano backend, on Python 2.7 and Python 3.5
+  - with the TensorFlow backend, on Python 2.7
+
+7. We use PEP8 syntax conventions, but we aren't dogmatic when it comes to line length. Make sure your lines stay reasonably sized, though. To make your life easier, we recommend running a PEP8 linter:
+  - Install PEP8 packages: `pip install pep8 pytest-pep8 autopep8`
+  - Run a standalone PEP8 check: `py.test --pep8 -m pep8`
+  - You can automatically fix some PEP8 error by running: `autopep8 -i --select <errors> <FILENAME>` for example: `autopep8 -i --select E128 tests/keras/backend/test_backends.py`
+
+8. When committing, use appropriate, descriptive commit messages. Make sure that your branch history is not a string of "bug fix", "fix", "oops", etc. When submitting your PR, squash your commits into a single commit with an appropriate commit message, to make sure the project history stays clean and readable. See ['rebase and squash'](http://rebaseandsqua.sh/) for technical help on how to squash your commits.
+
+9. Update the documentation. If introducing new functionality, make sure you include code snippets demonstrating the usage of your new feature.
+
+10. Submit your PR. If your changes have been approved in a previous discussion, and if you have complete (and passing) unit tests, your PR is likely to be merged promptly. Otherwise, well...
+
+## Adding new examples
+
+Even if you don't contribute to the Keras source code, if you have an application of Keras that is concise and powerful, please consider adding it to our collection of examples. [Existing examples](https://github.com/fchollet/keras/tree/master/examples) show idiomatic Keras code: make sure to keep your own script in the same spirit.

ISSUE_TEMPLATE.md

@@ -0,0 +1,9 @@
+Please make sure that the boxes below are checked before you submit your issue. Thank you!
+
+- [ ] Check that you are up-to-date with the master branch of Keras. You can update with:
+pip install git+git://github.com/fchollet/keras.git --upgrade --no-deps
+
+- [ ] If running on Theano, check that you are up-to-date with the master branch of Theano. You can update with:
+pip install git+git://github.com/Theano/Theano.git --upgrade --no-deps
+
+- [ ] Provide a link to a GitHub Gist of a Python script that can reproduce your issue (or just copy the script here if it is short).

LICENSE

@@ -1,6 +1,23 @@
-The MIT License (MIT)
+COPYRIGHT
 
-Copyright (c) 2015 
+All contributions by François Chollet:
+Copyright (c) 2015, François Chollet.
+All rights reserved.
+
+All contributions by Google:
+Copyright (c) 2015, Google, Inc.
+All rights reserved.
+
+All other contributions:
+Copyright (c) 2015, the respective contributors.
+All rights reserved.
+
+Each contributor holds copyright over their respective contributions.
+The project versioning (Git) records all such contribution source information.
+
+LICENSE
+
+The MIT License (MIT)
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

README.md

@@ -1,209 +1,165 @@
-# Keras: Theano-based Deep Learning library
+# Keras: Deep Learning library for Theano and TensorFlow
+
+![Build status](https://api.travis-ci.org/fchollet/keras.svg)
 
 ## You have just found Keras.
 
-Keras is a minimalist, highly modular neural network library in the spirit of Torch, written in Python / Theano so as not to have to deal with the dearth of ecosystem in Lua. It was developed with a focus on enabling fast experimentation. Being able to go from idea to result with the least possible delay is key to doing good research.
+Keras is a minimalist, highly modular neural networks library, written in Python and capable of running on top of either [TensorFlow](https://github.com/tensorflow/tensorflow) or [Theano](https://github.com/Theano/Theano). It was developed with a focus on enabling fast experimentation. Being able to go from idea to result with the least possible delay is key to doing good research.
 
 Use Keras if you need a deep learning library that:
+
 - allows for easy and fast prototyping (through total modularity, minimalism, and extensibility).
-- supports both convolutional networks (for vision) and recurrent networks (for sequence data). As well as combinations of the two. 
-- runs seamlessly on the CPU and the GPU.
+- supports both convolutional networks and recurrent networks, as well as combinations of the two.
+- supports arbitrary connectivity schemes (including multi-input and multi-output training).
+- runs seamlessly on CPU and GPU.
 
 Read the documentation at [Keras.io](http://keras.io).
 
-Keras is compatible with __Python 2.7-3.4__.
+Keras is compatible with: __Python 2.7-3.5__.
+
+
+------------------
+
 
 ## Guiding principles
 
-- __Modularity.__ A model is understood as a sequence of standalone, fully-configurable modules that can be plugged together with as little restrictions as possible. In particular, neural layers, cost functions, optimizers, initialization schemes, activation functions and dropout are all standalone modules that you can combine to create new models. 
+- __Modularity.__ A model is understood as a sequence or a graph of standalone, fully-configurable modules that can be plugged together with as little restrictions as possible. In particular, neural layers, cost functions, optimizers, initialization schemes, activation functions, regularization schemes are all standalone modules that you can combine to create new models.
 
-- __Minimalism.__ Each module should be kept short and simple (<100 lines of code). Every piece of code should be transparent upon first reading. No black magic: it hurts iteration speed and ability to innovate. 
+- __Minimalism.__ Each module should be kept short and simple. Every piece of code should be transparent upon first reading. No black magic: it hurts iteration speed and ability to innovate.
 
-- __Easy extensibility.__ New features (a new module, per the above definition, or a new way to combine modules together) are dead simple to add (as new classes/functions), and existing modules provide ample examples.
+- __Easy extensibility.__ New modules are dead simple to add (as new classes and functions), and existing modules provide ample examples. To be able to easily create new modules allows for total expressiveness, making Keras suitable for advanced research.
 
-- __Work with Python__. No separate models configuration files in a declarative format (like in Caffe or PyLearn2). Models are described in Python code, which is compact, easier to debug, benefits from syntax highlighting, and most of all, allows for ease of extensibility. See for yourself with the examples below.
+- __Work with Python__. No separate models configuration files in a declarative format. Models are described in Python code, which is compact, easier to debug, and allows for ease of extensibility.
 
-## Examples
 
-### Multilayer Perceptron (MLP):
+------------------
+
+
+## Getting started: 30 seconds to Keras
+
+The core data structure of Keras is a __model__, a way to organize layers. There are two types of models: [`Sequential`](http://keras.io/models/#sequential) and [`Graph`](http://keras.io/models/#graph).
+
+Here's the `Sequential` model (a linear pile of layers):
 
 ```python
 from keras.models import Sequential
-from keras.layers.core import Dense, Dropout, Activation
+
+model = Sequential()
+```
+
+Stacking layers is as easy as `.add()`:
+
+```python
+from keras.layers.core import Dense, Activation
+
+model.add(Dense(output_dim=64, input_dim=100, init="glorot_uniform"))
+model.add(Activation("relu"))
+model.add(Dense(output_dim=10, init="glorot_uniform"))
+model.add(Activation("softmax"))
+```
+
+Once your model looks good, configure its learning process with `.compile()`:
+```python
+model.compile(loss='categorical_crossentropy', optimizer='sgd')
+```
+
+If you need to, you can further configure your optimizer. A core principle of Keras is to make things reasonably simple, while allowing the user to be fully in control when they need to (the ultimate control being the easy extensibility of the source code).
+```python
 from keras.optimizers import SGD
-
-model = Sequential()
-model.add(Dense(20, 64, init='uniform'))
-model.add(Activation('tanh'))
-model.add(Dropout(0.5))
-model.add(Dense(64, 64, init='uniform'))
-model.add(Activation('tanh'))
-model.add(Dropout(0.5))
-model.add(Dense(64, 2, init='uniform'))
-model.add(Activation('softmax'))
-
-sgd = SGD(lr=0.1, decay=1e-6, momentum=0.9, nesterov=True)
-model.compile(loss='mean_squared_error', optimizer=sgd)
-
-model.fit(X_train, y_train, nb_epoch=20, batch_size=16)
-score = model.evaluate(X_test, y_test, batch_size=16)
+model.compile(loss='categorical_crossentropy', optimizer=SGD(lr=0.01, momentum=0.9, nesterov=True))
 ```
 
-### Alternative implementation of MLP:
-
+You can now iterate on your training data in batches:
 ```python
-model = Sequential()
-model.add(Dense(20, 64, init='uniform', activation='tanh'))
-model.add(Dropout(0.5))
-model.add(Dense(64, 64, init='uniform', activation='tanh'))
-model.add(Dropout(0.5))
-model.add(Dense(64, 2, init='uniform', activation='softmax'))
-
-sgd = SGD(lr=0.1, decay=1e-6, momentum=0.9, nesterov=True)
-model.compile(loss='mean_squared_error', optimizer=sgd)
+model.fit(X_train, Y_train, nb_epoch=5, batch_size=32)
 ```
 
-### VGG-like convnet:
-
+Alternatively, you can feed batches to your model manually:
 ```python
-from keras.models import Sequential
-from keras.layers.core import Dense, Dropout, Activation, Flatten
-from keras.layers.convolutional import Convolution2D, MaxPooling2D
-from keras.optimizers import SGD
-
-model = Sequential()
-model.add(Convolution2D(32, 3, 3, 3, border_mode='full')) 
-model.add(Activation('relu'))
-model.add(Convolution2D(32, 32, 3, 3))
-model.add(Activation('relu'))
-model.add(MaxPooling2D(poolsize=(2, 2)))
-model.add(Dropout(0.25))
-
-model.add(Convolution2D(64, 32, 3, 3, border_mode='full')) 
-model.add(Activation('relu'))
-model.add(Convolution2D(64, 64, 3, 3)) 
-model.add(Activation('relu'))
-model.add(MaxPooling2D(poolsize=(2, 2)))
-model.add(Dropout(0.25))
-
-model.add(Flatten())
-model.add(Dense(64*8*8, 256))
-model.add(Activation('relu'))
-model.add(Dropout(0.5))
-
-model.add(Dense(256, 10))
-model.add(Activation('softmax'))
-
-sgd = SGD(lr=0.1, decay=1e-6, momentum=0.9, nesterov=True)
-model.compile(loss='categorical_crossentropy', optimizer=sgd)
-
-model.fit(X_train, Y_train, batch_size=32, nb_epoch=1)
-
+model.train_on_batch(X_batch, Y_batch)
 ```
 
-### Sequence classification with LSTM:
-
+Evaluate your performance in one line:
 ```python
-from keras.models import Sequential
-from keras.layers.core import Dense, Dropout, Activation
-from keras.layers.embeddings import Embedding
-from keras.layers.recurrent import LSTM
-
-model = Sequential()
-model.add(Embedding(max_features, 256))
-model.add(LSTM(256, 128, activation='sigmoid', inner_activation='hard_sigmoid'))
-model.add(Dropout(0.5))
-model.add(Dense(128, 1))
-model.add(Activation('sigmoid'))
-
-model.compile(loss='binary_crossentropy', optimizer='rmsprop')
-
-model.fit(X_train, Y_train, batch_size=16, nb_epoch=10)
-score = model.evaluate(X_test, Y_test, batch_size=16)
+objective_score = model.evaluate(X_test, Y_test, batch_size=32)
 ```
 
-### Architecture for learning image captions with a convnet and a Gated Recurrent Unit:
-(word-level embedding, caption of maximum length 16 words).
-
-Note that getting this to actually "work" will require using a bigger convnet, initialized with pre-trained weights.
-Displaying readable results will also require an embedding decoder.
-
+Or generate predictions on new data:
 ```python
-max_caption_len = 16
-
-model = Sequential()
-model.add(Convolution2D(32, 3, 3, 3, border_mode='full')) 
-model.add(Activation('relu'))
-model.add(Convolution2D(32, 32, 3, 3))
-model.add(Activation('relu'))
-model.add(MaxPooling2D(poolsize=(2, 2)))
-
-model.add(Convolution2D(64, 32, 3, 3, border_mode='full')) 
-model.add(Activation('relu'))
-model.add(Convolution2D(64, 64, 3, 3)) 
-model.add(Activation('relu'))
-model.add(MaxPooling2D(poolsize=(2, 2)))
-
-model.add(Convolution2D(128, 64, 3, 3, border_mode='full')) 
-model.add(Activation('relu'))
-model.add(Convolution2D(128, 128, 3, 3)) 
-model.add(Activation('relu'))
-model.add(MaxPooling2D(poolsize=(2, 2)))
-
-model.add(Flatten())
-model.add(Dense(128*4*4, 256))
-model.add(Activation('relu'))
-model.add(Dropout(0.5))
-
-model.add(Repeat(max_caption_len)) 
-# the GRU below returns sequences of max_caption_len vectors of size 256 (our word embedding size)
-model.add(GRU(256, 256, return_sequences=True))
-
-model.compile(loss='mean_squared_error', optimizer='rmsprop')
-
-# "images" is a numpy array of shape (nb_samples, nb_channels=3, width, height) 
-# "captions" is a numpy array of shape (nb_samples, max_caption_len=16, embedding_dim=256)
-# captions are supposed already embedded (dense vectors).
-model.fit(images, captions, batch_size=16, nb_epoch=100)
-    
+classes = model.predict_classes(X_test, batch_size=32)
+proba = model.predict_proba(X_test, batch_size=32)
 ```
 
-In the examples folder, you will find example models for real datasets:
-- CIFAR10 small images classification: Convnet with realtime data augmentation
-- IMDB movie review sentiment classification: LSTM over sequences of words
-- Reuters newswires topic classification: Multilayer Perceptron
-- MNIST handwritten digits classification: Multilayer Perceptron
+Building a network of LSTMs, a deep CNN, a Neural Turing Machine, a word2vec embedder or any other model is just as fast. The ideas behind deep learning are simple, so why should their implementation be painful?
+
+Have a look at these [starter examples](http://keras.io/examples/).
+
+In the [examples folder](https://github.com/fchollet/keras/tree/master/examples) of the repo, you will find more advanced models: question-answering with memory networks, text generation with stacked LSTMs, neural turing machines, etc.
 
 
-## Current capabilities
+------------------
 
-For complete coverage of the API, check out [the Keras documentation](http://keras.io).
-
-A few highlights: convnets, LSTM, GRU, word2vec-style embeddings, PReLU, batch normalization...
 
 ## Installation
 
 Keras uses the following dependencies:
 
 - numpy, scipy
-
-- Theano
-    - See installation instructions: http://deeplearning.net/software/theano/install.html#install
-
+- pyyaml
 - HDF5 and h5py (optional, required if you use model saving/loading functions)
-
 - Optional but recommended if you use CNNs: cuDNN.
 
-Once you have the dependencies installed, cd to the Keras folder and run the install command:
+*When using the Theano backend:*
+
+- Theano
+    - [See installation instructions](http://deeplearning.net/software/theano/install.html#install).
+
+**Note**: You should use the latest version of Theano, not the PyPI version. Install it with:
+```
+sudo pip install git+git://github.com/Theano/Theano.git
+```
+
+*When using the TensorFlow backend:*
+
+- TensorFlow
+    - [See installation instructions](https://github.com/tensorflow/tensorflow#download-and-setup).
+
+To install Keras, `cd` to the Keras folder and run the install command:
 ```
 sudo python setup.py install
 ```
 
+You can also install Keras from PyPI:
+```
+sudo pip install keras
+```
+
+------------------
+
+
+## Switching from Theano to TensorFlow
+
+By default, Keras will use Theano as its tensor manipulation library. [Follow these instructions](http://keras.io/backend/) to configure the Keras backend.
+
+------------------
+
+
+## Support
+
+You can ask questions and join the development discussion on the [Keras Google group](https://groups.google.com/forum/#!forum/keras-users).
+
+You can also post bug reports and feature requests in [Github issues](https://github.com/fchollet/keras/issues). Make sure to read [our guidelines](https://github.com/fchollet/keras/blob/master/CONTRIBUTING.md) first.
+
+
+------------------
+
+
 ## Why this name, Keras?
 
 Keras (κέρας) means _horn_ in Greek. It is a reference to a literary image from ancient Greek and Latin literature, first found in the _Odyssey_, where dream spirits (_Oneiroi_, singular _Oneiros_) are divided between those who deceive men with false visions, who arrive to Earth through a gate of ivory, and those who announce a future that will come to pass, who arrive through a gate of horn. It's a play on the words κέρας (horn) / κραίνω (fulfill), and ἐλέφας (ivory) / ἐλεφαίρομαι (deceive).
 
-Keras was developed as part of the research effort of project ONEIROS (Open-ended Neuro-Electronic Intelligent Robot Operating System).
+Keras was initially developed as part of the research effort of project ONEIROS (Open-ended Neuro-Electronic Intelligent Robot Operating System).
 
-_"Oneiroi are beyond our unravelling --who can be sure what tale they tell? Not all that men look for comes to pass. Two gates there are that give passage to fleeting Oneiroi; one is made of horn, one of ivory. The Oneiroi that pass through sawn ivory are deceitful, bearing a message that will not be fulfilled; those that come out through polished horn have truth behind them, to be accomplished for men who see them."_ Homer, Odyssey 19. 562 ff (Shewring translation).
+>_"Oneiroi are beyond our unravelling --who can be sure what tale they tell? Not all that men look for comes to pass. Two gates there are that give passage to fleeting Oneiroi; one is made of horn, one of ivory. The Oneiroi that pass through sawn ivory are deceitful, bearing a message that will not be fulfilled; those that come out through polished horn have truth behind them, to be accomplished for men who see them."_ Homer, Odyssey 19. 562 ff (Shewring translation).
 
+------------------

docs/README.md

@@ -5,5 +5,8 @@ Our documentation uses extended Markdown, as implemented by [MkDocs](http://mkdo
 
 ## Building the documentation
 
-- install MkDocs: `sudo pip install mkdocs`
-- `cd` to the `docs/` folder and run: `mkdocs serve`
+- install MkDocs: `pip install mkdocs`
+- `cd` to the `docs/` folder and run:
+    - `python autogen.py`
+    - `mkdocs serve`    # Starts a local webserver:  [localhost:8000](localhost:8000)
+    - `mkdocs build`    # Builds a static site in "site" directory

docs/autogen.py

@@ -0,0 +1,260 @@
+# -*- coding: utf-8 -*-
+from __future__ import print_function
+import re
+import inspect
+import os
+import shutil
+
+from keras.layers import convolutional
+from keras.layers import recurrent
+from keras.layers import core
+from keras.layers import noise
+from keras.layers import normalization
+from keras.layers import advanced_activations
+from keras.layers import containers
+from keras.layers import embeddings
+from keras import optimizers
+from keras import callbacks
+from keras import models
+
+MODULES = [(convolutional, 'keras.layers.convolutional'),
+           (recurrent, 'keras.layers.recurrent'),
+           (noise, 'keras.layers.noise'),
+           (normalization, 'keras.layers.normalization'),
+           (advanced_activations, 'keras.layers.advanced_activations'),
+           (containers, 'keras.layers.containers'),
+           (core, 'keras.layers.core'),
+           (embeddings, 'keras.layers.embeddings'),
+           (optimizers, 'keras.optimizers'),
+           (callbacks, 'keras.callbacks'),
+           (models, 'keras.models')]
+
+SKIP = ['build', 'get_params', 'MaskedLayer',
+        'SiameseHead', 'MaskedLambda',
+        'CallbackList']
+ROOT = 'http://keras.io/'
+INCLUDE_METHODS_FOR = [
+    'Layer',
+    'Graph',
+    'Sequential',
+    'Callback',
+]
+
+
+def get_earliest_class_that_defined_member(member, cls):
+    ancestors = get_classes_ancestors([cls])
+    result = None
+    for ancestor in ancestors:
+        if member in dir(ancestor):
+            result = ancestor
+    if not result:
+        return cls
+    return result
+
+
+def get_classes_ancestors(classes):
+    ancestors = []
+    for cls in classes:
+        ancestors += cls.__bases__
+    filtered_ancestors = []
+    for ancestor in ancestors:
+        if ancestor.__name__ in ['object']:
+            continue
+        filtered_ancestors.append(ancestor)
+    if filtered_ancestors:
+        return filtered_ancestors + get_classes_ancestors(filtered_ancestors)
+    else:
+        return filtered_ancestors
+
+
+def get_method_signature(method):
+    signature = inspect.getargspec(method)
+    defaults = signature.defaults
+    args = signature.args[1:]
+    if defaults:
+        kwargs = zip(args[-len(defaults):], defaults)
+        args = args[:-len(defaults)]
+    else:
+        kwargs = []
+    st = '%s.%s(' % (method.__module__, method.__name__)
+    for a in args:
+        st += str(a) + ', '
+    for a, v in kwargs:
+        if type(v) == str:
+            v = '\'' + v + '\''
+        elif type(v) == unicode:
+            v = 'u\'' + v + '\''
+        st += str(a) + '=' + str(v) + ', '
+    if kwargs or args:
+        return st[:-2] + ')'
+    else:
+        return st + ')'
+
+
+def class_to_docs_link(cls):
+    module_name = cls.__module__
+    assert module_name[:6] == 'keras.'
+    module_name = module_name[6:]
+    link = ROOT + module_name.replace('.', '/') + '#' + cls.__name__.lower()
+    return link
+
+
+def class_to_source_link(cls):
+    module_name = cls.__module__
+    assert module_name[:6] == 'keras.'
+    path = module_name.replace('.', '/')
+    path += '.py'
+    line = inspect.getsourcelines(cls)[-1]
+    link = 'https://github.com/fchollet/keras/blob/master/' + path + '#L' + str(line)
+    return '[[source]](' + link + ')'
+
+
+def code_snippet(snippet):
+    result = '```python\n'
+    result += snippet + '\n'
+    result += '```\n'
+    return result
+
+
+def process_class_docstring(docstring):
+    docstring = re.sub(r'\n    # (.*)\n',
+                       r'\n    __\1__\n\n',
+                       docstring)
+
+    docstring = re.sub(r'    ([^\s\\]+):(.*)\n',
+                       r'    - __\1__:\2\n',
+                       docstring)
+
+    docstring = docstring.replace('    ' * 5, '\t\t')
+    docstring = docstring.replace('    ' * 3, '\t')
+    docstring = docstring.replace('    ', '')
+    return docstring
+
+
+def process_method_docstring(docstring):
+    docstring = re.sub(r'\n        # (.*)\n',
+                       r'\n        __\1__\n\n',
+                       docstring)
+
+    docstring = re.sub(r'    ([^\s\\]+):(.*)\n',
+                       r'    - __\1__:\2\n',
+                       docstring)
+
+    docstring = docstring.replace('    ' * 6, '\t\t')
+    docstring = docstring.replace('    ' * 4, '\t')
+    docstring = docstring.replace('    ', '')
+    return docstring
+
+print('Cleaning up existing sources directory.')
+if os.path.exists('sources'):
+    shutil.rmtree('sources')
+print('Populating sources directory with templates.')
+for subdir, dirs, fnames in os.walk('templates'):
+    for fname in fnames:
+        new_subdir = subdir.replace('templates', 'sources')
+        if not os.path.exists(new_subdir):
+            os.makedirs(new_subdir)
+        if fname[-3:] == '.md':
+            fpath = os.path.join(subdir, fname)
+            new_fpath = fpath.replace('templates', 'sources')
+            shutil.copy(fpath, new_fpath)
+
+print('Starting autogeneration.')
+covered_so_far = set()
+for module, module_name in MODULES:
+    class_pages = []
+    for name in dir(module):
+        if name in SKIP:
+            continue
+        if name[0] == '_':
+            continue
+        module_member = getattr(module, name)
+        if module_member in covered_so_far:
+            continue
+        if inspect.isclass(module_member):
+            cls = module_member
+            if cls.__module__ == module_name:
+
+                try:
+                    class_signature = get_method_signature(cls.__init__)
+                    class_signature = class_signature.replace('__init__', cls.__name__)
+                except:
+                    # in case the class inherits from object and does not
+                    # define __init__
+                    class_signature = module_name + '.' + cls.__name__ + '()'
+
+                methods = []
+                methods_not_defined_here = []
+                for name in dir(cls):
+                    if name in SKIP:
+                        continue
+                    if name[0] == '_':
+                        continue
+                    cls_member = getattr(cls, name)
+                    if inspect.ismethod(cls_member):
+                        method = cls_member
+                        signature = inspect.getargspec(method)
+                        defaults = signature.defaults
+                        args = signature.args[1:]
+                        if defaults:
+                            kwargs = zip(args[-len(defaults):], defaults)
+                            args = args[:-len(defaults)]
+                        else:
+                            kwargs = []
+
+                        defined_by = get_earliest_class_that_defined_member(method.__name__, cls)
+                        if cls == defined_by:
+                            methods.append(method)
+                        else:
+                            methods_not_defined_here.append((method, defined_by))
+
+                blocks = []
+                blocks.append('<span style="float:right;">' + class_to_source_link(cls) + '</span>')
+                blocks.append('# ' + cls.__name__ + '\n')
+                blocks.append(code_snippet(class_signature))
+                docstring = cls.__doc__
+                if docstring:
+                    blocks.append(process_class_docstring(docstring))
+
+                if cls.__name__ in INCLUDE_METHODS_FOR:
+                    if methods or methods_not_defined_here:
+                        blocks.append('### Methods\n')
+                        for method in methods:
+                            signature = get_method_signature(method)
+                            signature = signature.replace(module_name + '.', '')
+                            blocks.append(code_snippet(signature))
+                            docstring = method.__doc__
+                            if docstring:
+                                blocks.append(process_method_docstring(docstring))
+                        for method, defined_by in methods_not_defined_here:
+                            signature = get_method_signature(method)
+                            method_module_name = method.__module__
+                            signature = signature.replace(method_module_name + '.', '')
+                            link = '[' + defined_by.__name__ + '](' + class_to_docs_link(defined_by) + ')'
+                            blocks.append(code_snippet(signature))
+                            blocks.append('Defined by ' + link + '.\n')
+
+                mkdown = '\n'.join(blocks)
+                class_pages.append((id(cls), mkdown))
+                covered_so_far.add(module_member)
+
+    class_pages.sort(key=lambda x: x[0])
+    class_pages = [x[1] for x in class_pages]
+    module_page = '\n----\n\n'.join(class_pages)
+
+    # save module page.
+    # Either insert content into existing page,
+    # or create page otherwise
+    path = 'sources/' + module_name.replace('.', '/')[6:] + '.md'
+    if os.path.exists(path):
+        template = open(path).read()
+        assert '{{autogenerated}}' in template, ('Template found for ' + path +
+                                                 ' but missing {{autogenerated}} tag.')
+        module_page = template.replace('{{autogenerated}}', module_page)
+        print('...inserting autogenerated content into template:', path)
+    else:
+        print('...creating new page with autogenerated content:', path)
+    subdir = os.path.dirname(path)
+    if not os.path.exists(subdir):
+        os.makedirs(subdir)
+    open(path, 'w').write(module_page)

docs/mkdocs.yml

@@ -2,42 +2,41 @@ site_name: Keras Documentation
 theme: readthedocs
 docs_dir: sources
 repo_url: http://github.com/fchollet/keras
-site_url: /
+site_url: http://keras.io/
 #theme_dir: theme
 site_description: Documentation for fast and lightweight Keras Deep Learning library.
-include_404: true
-include_search: true
 
 dev_addr: '0.0.0.0:8000'
 google_analytics: ['UA-61785484-1', 'keras.io']
 
 
 pages:
-- [index.md, Home]
-- [documentation.md, Index]
+- Home: index.md
+- Index: documentation.md
+- Examples: examples.md
+- FAQ: faq.md
+- Backends: backend.md
+- Optimizers: optimizers.md
+- Objectives: objectives.md
+- Models: models.md
+- Activations: activations.md
+- Initializations: initializations.md
+- Regularizers: regularizers.md
+- Constraints: constraints.md
+- Callbacks: callbacks.md
+- Datasets: datasets.md
+- Visualization: visualization.md
+- Layers:
+  - Core Layers: layers/core.md
+  - Convolutional Layers: layers/convolutional.md
+  - Recurrent Layers: layers/recurrent.md
+  - Advanced Activations Layers: layers/advanced_activations.md
+  - Normalization Layers: layers/normalization.md
+  - Embedding Layers: layers/embeddings.md
+  - Noise layers: layers/noise.md
+  - Containers: layers/containers.md
+- Preprocessing:
+  - Sequence Preprocessing: preprocessing/sequence.md
+  - Text Preprocessing: preprocessing/text.md
+  - Image Preprocessing: preprocessing/image.md
 
-- [examples.md, Examples]
-- [optimizers.md, Optimizers]
-- [objectives.md, Objectives]
-- [models.md, Models]
-- [activations.md, Activations]
-- [initializations.md, Initializations]
-- [regularizers.md, Regularizers]
-- [constraints.md, Constraints]
-- [callbacks.md, Callbacks]
-
-- [datasets.md, Datasets]
-
-- [layers/core.md, Layers, Core Layers]
-- [layers/convolutional.md, Layers, Convolutional Layers]
-- [layers/recurrent.md, Layers, Recurrent Layers]
-- [layers/advanced_activations.md, Layers, Advanced Activations Layers]
-- [layers/normalization.md, Layers, Normalization Layers]
-- [layers/embeddings.md, Layers, Embedding Layers]
-- [layers/containers.md, Layers, Containers]
-
-- [preprocessing/sequence.md, Preprocessing, Sequence Preprocessing]
-- [preprocessing/text.md, Preprocessing, Text Preprocessing]
-- [preprocessing/image.md, Preprocessing, Image Preprocessing]
-
-- [utils/visualization.md, Utils, Visualization Utilities]

docs/sources/activations.md

@@ -1,40 +0,0 @@
-
-## Usage of activations
-
-Activations can either be used through an `Activation` layer, or through the `activation` argument supported by all forward layers:
-
-```python
-from keras.layers.core import Activation, Dense
-
-model.add(Dense(64, 64, init='uniform'))
-model.add(Activation('tanh'))
-```
-is equivalent to:
-```python
-model.add(Dense(20, 64, init='uniform', activation='tanh'))
-```
-
-You can also pass an element-wise Theano function as an activation:
-
-```python
-def tanh(x):
-    return theano.tensor.tanh(x)
-
-model.add(Dense(20, 64, init='uniform', activation=tanh))
-model.add(Activation(tanh))
-```
-
-## Available activations
-
-- __softmax__: Should only be applied to 2D layers (expected shape: `(nb_samples, nb_dims)`).
-- __time_distributed_softmax__: Softmax applied to every sample at every timestep of a layer of shape `(nb_samples, nb_timesteps, nb_dims)`.
-- __softplus__
-- __relu__
-- __tanh__
-- __sigmoid__
-- __hard_sigmoid__
-- __linear__
-
-## On Advanced Activations
-
-Activations that are more complex than a simple Theano function (eg. learnable activations, configurable activations, etc.) are available as [Advanced Activation layers](layers/advanced_activations.md), and can be found in the module `keras.layers.advanced_activations`. These include PReLU and LeakyReLU.

docs/sources/callbacks.md

@@ -1,91 +0,0 @@
-## Usage of callbacks
-
-A callback is a set of functions to be applied at given stages of the training procedure. You can use callbacks to get a view on internal states and statistics of the model during training. You can pass a list of callback (as the keyword argument `callbacks`) to the `.fit()` method of the `Sequential` model. The relevant methods of the callbacks will then be called at each stage of the training. 
-
----
-
-## Base class
-
-```python
-keras.callbacks.Callback()
-```
-- __Properties__:
-    - __params__: dict. Training parameters (eg. verbosity, batch size, number of epochs...).
-    - __model__: `keras.models.Model`. Reference of the model being trained.
-- __Methods__:
-    - __on_train_begin__(logs={}): Method called at the beginning of training.
-    - __on_train_end__(logs={}): Method called at the end of training.
-    - __on_epoch_begin__(epoch, logs={}): Method called at the beginning of epoch `epoch`.
-    - __on_epoch_end__(epoch, logs={}): Method called at the end of epoch `epoch`.
-    - __on_batch_begin__(batch, logs={}): Method called at the beginning of batch `batch`.
-    - __on_batch_end__(batch, logs={}): Method called at the end of batch `batch`.
-
-The `logs` dictionary will contain keys for quantities relevant to the current batch or epoch. Currently, the `.fit()` method of the `Sequential` model class will include the following quantities in the `logs` that it passes to its callbacks:
-- __on_epoch_end__: logs optionally include `val_loss` (if validation is enabled in `fit`), and `val_accuracy` (if validation and accuracy monitoring are enabled).
-- __on_batch_begin__: logs include `size`, the number of samples in the current batch.
-- __on_batch_end__: logs include `loss`, and optionally `accuracy` (if accuracy monitoring is enabled).
-
----
-
-
-## Create a callback
-
-You can create a custom callback by extending the base class `keras.callbacks.Callback`. A callback has access to its associated model through the class property `self.model`.
-
-Here's a simple example saving a list of losses over each batch during training:
-```python
-class LossHistory(keras.callbacks.Callback):
-    def on_train_begin(self):
-        self.losses = []
-
-    def on_batch_end(self, batch, logs={}):
-        self.losses.append(logs.get('loss'))
-```
-
----
-
-### Example to record the loss history
-
-```python
-class LossHistory(keras.callbacks.Callback):
-    def on_train_begin(self):
-        self.losses = []
-
-    def on_batch_end(self, batch, logs={}):
-        self.losses.append(logs.get('loss'))
-
-model = Sequential()
-model.add(Dense(784, 10, init='uniform'))
-model.add(Activation('softmax'))
-model.compile(loss='categorical_crossentropy', optimizer='rmsprop')
-
-history = LossHistory()
-model.fit(X_train, Y_train, batch_size=128, nb_epoch=20, verbose=0, callbacks=[history])
-
-print history.losses
-# outputs
-'''
-[0.66047596406559383, 0.3547245744908703, ..., 0.25953155204159617, 0.25901699725311789]
-'''
-```
-
----
-
-### Example to checkpoint models
-
-```python
-from keras.callbacks import ModelCheckpoint
-
-model = Sequential()
-model.add(Dense(784, 10, init='uniform'))
-model.add(Activation('softmax'))
-model.compile(loss='categorical_crossentropy', optimizer='rmsprop')
-
-'''
-saves the model weights after each epoch if the validation loss decreased
-'''
-checkpointer = ModelCheckpoint(filepath="/tmp/weights.hdf5", verbose=1, save_best_only=True)
-model.fit(X_train, Y_train, batch_size=128, nb_epoch=20, verbose=0, validation_data=(X_test, Y_test), callbacks=[checkpointer])
-
-```
-

docs/sources/examples.md

@@ -1,161 +0,0 @@
-
-Here are a few examples to get you started!
-
-### Multilayer Perceptron (MLP)
-
-```python
-from keras.models import Sequential
-from keras.layers.core import Dense, Dropout, Activation
-from keras.optimizers import SGD
-
-model = Sequential()
-model.add(Dense(20, 64, init='uniform'))
-model.add(Activation('tanh'))
-model.add(Dropout(0.5))
-model.add(Dense(64, 64, init='uniform'))
-model.add(Activation('tanh'))
-model.add(Dropout(0.5))
-model.add(Dense(64, 2, init='uniform'))
-model.add(Activation('softmax'))
-
-sgd = SGD(lr=0.1, decay=1e-6, momentum=0.9, nesterov=True)
-model.compile(loss='mean_squared_error', optimizer=sgd)
-
-model.fit(X_train, y_train, nb_epoch=20, batch_size=16)
-score = model.evaluate(X_test, y_test, batch_size=16)
-```
-
----
-
-### Alternative implementation of MLP
-
-```python
-model = Sequential()
-model.add(Dense(20, 64, init='uniform', activation='tanh'))
-model.add(Dropout(0.5))
-model.add(Dense(64, 64, init='uniform', activation='tanh'))
-model.add(Dropout(0.5))
-model.add(Dense(64, 2, init='uniform', activation='softmax')
-
-sgd = SGD(lr=0.1, decay=1e-6, momentum=0.9, nesterov=True)
-model.compile(loss='mean_squared_error', optimizer=sgd)
-```
-
----
-
-### VGG-like convnet
-
-```python
-from keras.models import Sequential
-from keras.layers.core import Dense, Dropout, Activation, Flatten
-from keras.layers.convolutional import Convolution2D, MaxPooling2D
-from keras.optimizers import SGD
-
-model = Sequential()
-model.add(Convolution2D(32, 3, 3, 3, border_mode='full')) 
-model.add(Activation('relu'))
-model.add(Convolution2D(32, 32, 3, 3))
-model.add(Activation('relu'))
-model.add(MaxPooling2D(poolsize=(2, 2)))
-model.add(Dropout(0.25))
-
-model.add(Convolution2D(64, 32, 3, 3, border_mode='full')) 
-model.add(Activation('relu'))
-model.add(Convolution2D(64, 64, 3, 3)) 
-model.add(Activation('relu'))
-model.add(MaxPooling2D(poolsize=(2, 2)))
-model.add(Dropout(0.25))
-
-model.add(Flatten())
-model.add(Dense(64*8*8, 256))
-model.add(Activation('relu'))
-model.add(Dropout(0.5))
-
-model.add(Dense(256, 10))
-model.add(Activation('softmax'))
-
-sgd = SGD(lr=0.1, decay=1e-6, momentum=0.9, nesterov=True)
-model.compile(loss='categorical_crossentropy', optimizer=sgd)
-
-model.fit(X_train, Y_train, batch_size=32, nb_epoch=1)
-
-```
-
----
-
-### Sequence classification with LSTM
-
-```python
-from keras.models import Sequential
-from keras.layers.core import Dense, Dropout, Activation
-from keras.layers.embeddings import Embedding
-from keras.layers.recurrent import LSTM
-
-model = Sequential()
-model.add(Embedding(max_features, 256))
-model.add(LSTM(256, 128, activation='sigmoid', inner_activation='hard_sigmoid'))
-model.add(Dropout(0.5))
-model.add(Dense(128, 1))
-model.add(Activation('sigmoid'))
-
-model.compile(loss='binary_crossentropy', optimizer='rmsprop')
-
-model.fit(X_train, Y_train, batch_size=16, nb_epoch=10)
-score = model.evaluate(X_test, Y_test, batch_size=16)
-```
-
----
-
-### Architecture for learning image captions with a convnet and a Gated Recurrent Unit
-(word-level embedding, caption of maximum length 16 words).
-
-Note that getting this to actually "work" will require using a bigger convnet, initialized with pre-trained weights.
-Displaying readable results will also require an embedding decoder.
-
-```python
-max_caption_len = 16
-
-model = Sequential()
-model.add(Convolution2D(32, 3, 3, 3, border_mode='full')) 
-model.add(Activation('relu'))
-model.add(Convolution2D(32, 32, 3, 3))
-model.add(Activation('relu'))
-model.add(MaxPooling2D(poolsize=(2, 2)))
-
-model.add(Convolution2D(64, 32, 3, 3, border_mode='full')) 
-model.add(Activation('relu'))
-model.add(Convolution2D(64, 64, 3, 3)) 
-model.add(Activation('relu'))
-model.add(MaxPooling2D(poolsize=(2, 2)))
-
-model.add(Convolution2D(128, 64, 3, 3, border_mode='full')) 
-model.add(Activation('relu'))
-model.add(Convolution2D(128, 128, 3, 3)) 
-model.add(Activation('relu'))
-model.add(MaxPooling2D(poolsize=(2, 2)))
-
-model.add(Flatten())
-model.add(Dense(128*4*4, 256))
-model.add(Activation('relu'))
-model.add(Dropout(0.5))
-
-model.add(Repeat(max_caption_len)) 
-# the GRU below returns sequences of max_caption_len vectors of size 256 (our word embedding size)
-model.add(GRU(256, 256, return_sequences=True))
-
-model.compile(loss='mean_squared_error', optimizer='rmsprop')
-
-# "images" is a numpy array of shape (nb_samples, nb_channels=3, width, height) 
-# "captions" is a numpy array of shape (nb_samples, max_caption_len=16, embedding_dim=256)
-# captions are supposed already embedded (dense vectors).
-model.fit(images, captions, batch_size=16, nb_epoch=100)
-    
-```
-
----
-
-In the [examples folder](https://github.com/fchollet/keras/tree/master/examples), you will find example models for real datasets:
-
-- CIFAR10 small images classification: Convnet with realtime data augmentation
-- IMDB movie review sentiment classification: LSTM over sequences of words
-- Reuters newswires topic classification: Multilayer Perceptron

docs/sources/index.md

@@ -1,131 +0,0 @@
-# Keras: Theano-based Deep Learning library
-
-## Overview
-
-Keras is a minimalist, highly modular neural network library in the spirit of Torch, written in Python, that uses [Theano](http://deeplearning.net/software/theano/) under the hood for fast tensor manipulation on GPU and CPU. It was developed with a focus on enabling fast experimentation. 
-
-Use Keras if you need a deep learning library that:
-
-- allows for easy and fast prototyping (through total modularity, minimalism, and extensibility).
-- supports both __convolutional networks__ and __recurrent networks__ (LSTM, GRU, etc). As well as combinations of the two. 
-- runs seamlessly on the CPU and the GPU.
-
-## Guiding principles
-
-- __Modularity.__ A model is understood as a sequence of standalone, fully-configurable modules that can be plugged together with as little restrictions as possible. In particular, neural layers, cost functions, optimizers, initialization schemes, activation functions and dropout are all standalone modules that you can combine to create new models. 
-
-- __Minimalism.__ Each module should be kept short and simple (<100 lines of code). Every piece of code should be transparent upon first reading. No black magic: it hurts iteration speed and ability to innovate. 
-
-- __Easy extensibility.__ A new feature (a new module, per the above definition, or a new way to combine modules together) are dead simple to add (as new classes/functions), and existing modules provide ample examples.
-
-- __Work with Python__. No separate models configuration files in a declarative format (like in Caffe or PyLearn2). Models are described in Python code, which is compact, easier to debug, benefits from syntax highlighting, and most of all, allows for ease of extensibility.
-
-## Code
-
-Find the code on Github: [fchollet/keras](https://github.com/fchollet/keras).
-
-## License
-
-Keras is licensed under the [MIT license](http://opensource.org/licenses/MIT). 
-
-## Getting started: 30 seconds to Keras
-
-The core datastructure of Keras is a __model__, a way to organize layers. Here's a sequential model (a linear pile of layers).
-
-```python
-from keras.models import Sequential
-
-model = Sequential()
-```
-
-Stacking layers is as easy as `.add()`:
-
-```python
-from keras.layers.core import Dense, Activation
-
-model.add(Dense(input_dim=100, output_dim=64, init="uniform"))
-model.add(Activation("relu"))
-model.add(Dense(input_dim=64, output_dim=10, init="uniform"))
-model.add(Activation("softmax"))
-```
-
-Once your model looks good, configure its learning process with `.compile()`:
-```python
-model.compile(loss='categorical_crossentropy', optimizer='sgd')
-```
-
-If you need to, you can further configure your optimizer. A core principle of Keras is make things things reasonably simple, while allowing the user to be fully in control when they need to (the ultimate control being the easy extensibility of the source code).
-```python
-from keras.optimizers import SGD
-model.compile(loss='categorical_crossentropy', optimizer=SGD(lr=0.01, momentum=0.9, nesterov=True))
-```
-
-You can now iterate on your training data in batches:
-```python
-model.fit(X_train, Y_train, nb_epoch=5, batch_size=32)
-```
-
-Alternatively, you can feed batches to your model manually:
-```python
-model.train(X_batch, Y_batch)
-```
-
-Evaluate your performance in one line:
-```python
-objective_score = model.evaluate(X_test, Y_test, batch_size=32)
-```
-
-Or generate predictions on new data:
-```python
-classes = model.predict_classes(X_test, batch_size=32)
-proba = model.predict_proba(X_test, batch_size=32)
-```
-
-Building a network of LSTMs, a deep CNN, a word2vec embedder or any other model is just as fast. The ideas behind deep learning are simple, so why should their implementation be painful?
-
-Have a look at the [examples](examples.md).
-
-## Installation
-
-Keras uses the following dependencies:
-
-- numpy, scipy
-- Theano
-    - See [installation instructions](http://deeplearning.net/software/theano/install.html#install).
-- HDF5 and h5py (optional, required if you use model saving/loading functions)
-- Optional but recommended if you use CNNs: cuDNN.
-
-Once you have the dependencies installed, clone the repo:
-```bash
-git clone https://github.com/fchollet/keras.git
-```
-Go to the Keras folder and run the install command:
-```bash
-cd keras
-sudo python setup.py install
-```
-
-## Support
-
-You can ask questions and join the development discussion on the [Keras Google group](https://groups.google.com/forum/#!forum/keras-users).
-
-## Contribution Guidelines
-
-Keras welcomes all contributions from the community. 
-
-- Keep a pragmatic mindset and avoid bloat. Only add to the source if that is the only path forward.
-- New features should be documented. Make sure you update the documentation along with your Pull Request.
-- The documentation for every new feature should include a usage example in the form of a code snippet. 
-- All changes should be tested. A formal test process will be introduced very soon.
-- Even if you don't contribute to the Keras source code, if you have an application of Keras that is concise and powerful, please consider adding it to our collection of [examples](https://github.com/fchollet/keras/tree/master/examples).
-
-
-## Why this name, Keras?
-
-Keras (κέρας) means _horn_ in Greek. It is a reference to a literary image from ancient Greek and Latin literature, first found in the _Odyssey_, where dream spirits (_Oneiroi_, singular _Oneiros_) are divided between those who deceive men with false visions, who arrive to Earth through a gate of ivory, and those who announce a future that will come to pass, who arrive through a gate of horn. It's a play on the words κέρας (horn) / κραίνω (fulfill), and ἐλέφας (ivory) / ἐλεφαίρομαι (deceive).
-
-Keras was developed as part of the research effort of project ONEIROS (Open-ended Neuro-Electronic Intelligent Robot Operating System).
-
-> _"Oneiroi are beyond our unravelling --who can be sure what tale they tell? Not all that men look for comes to pass. Two gates there are that give passage to fleeting Oneiroi; one is made of horn, one of ivory. The Oneiroi that pass through sawn ivory are deceitful, bearing a message that will not be fulfilled; those that come out through polished horn have truth behind them, to be accomplished for men who see them."_ 
-
-> -- Homer, Odyssey 19. 562 ff (Shewring translation).

docs/sources/layers/advanced_activations.md

@@ -1,35 +0,0 @@
-
-## LeakyReLU
-
-```python
-keras.layers.advanced_activations.LeakyReLU(alpha=0.3)
-```
-
-Special version of a Rectified Linear Unit that allows a small gradient when the unit is not active (`f(x) = alpha*x for x < 0`).
-
-- __Input shape__: This layer does not assume a specific input shape. As a result, it cannot be used as the first layer in a model.
-
-- __Output shape__: Same as input.
-
-- __Arguments__:
-    - __alpha__: float >= 0. Negative slope coefficient.
-
----
-
-## PReLU
-
-```python
-keras.layers.advanced_activations.PReLU(input_shape)
-```
-
-Parametrized linear unit. Similar to a LeakyReLU, where each input unit has its alpha coefficient, and where these coefficients are learned during training.
-
-- __Input shape__: Same as `input_shape`. This layer cannot be used as first layer in a model.
-
-- __Output shape__: Same as input.
-
-- __Arguments__:
-    - __input_shape__: tuple.
-
-- __References__:
-    - [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](http://arxiv.org/pdf/1502.01852v1.pdf)

docs/sources/layers/containers.md

@@ -1,23 +0,0 @@
-# Containers
-
-Containers are ensembles of layers that can be interacted with through the same API as `Layer` objects.
-
-## Sequential
-
-```python
-keras.layers.containers.Sequential(layers=[])
-```
-
-The Sequential container is a linear stack of layers. Apart from the `add` methods and the `layers` constructor argument, the API is identical to that of the `Layer` class.
-
-This class is also the basis for the `keras.models.Sequential` architecture.
-
-The `layers` constructor argument is a list of Layer instances.
-
-__Methods__:
-
-```python
-add(layer)
-```
-
-Add a new layer to the stack.

docs/sources/layers/convolutional.md

@@ -1,20 +0,0 @@
-
-## Convolution2D
-
-```python
-keras.layers.convolutional.Convolution2D(nb_filter, stack_size, nb_row, nb_col, 
-        init='glorot_uniform', activation='linear', weights=None, 
-        image_shape=None, border_mode='valid', subsample=(1,1))
-```
-
-This is a wrapper for Theano's [conv2d](http://deeplearning.net/software/theano/library/tensor/nnet/conv.html#theano.tensor.nnet.conv.conv2d). 
-
----
-
-## MaxPooling2D
-
-```python
-keras.layers.convolutional.MaxPooling2D(poolsize=(2, 2), ignore_border=True)
-```
-
-This is a wrapper for Theano's [max_pool_2d](http://deeplearning.net/software/theano/library/tensor/signal/downsample.html).

docs/sources/layers/core.md

@@ -1,355 +0,0 @@
-## Base class
-
-```python
-keras.layers.core.Layer()
-```
-
-__Methods__:
-
-```python
-connect(previous_layer)
-```
-
-Connect the input of the current layer to the output of the argument layer.
-
-- __Return__: None.
-
-- __Arguments__: 
-    - __previous_layer__: Layer object.
-
-
-
-```python
-output(train)
-```
-
-Get the output of the layer.
-
-- __Return__: Theano tensor.
-
-- __Arguments__: 
-    - __train__: Boolean. Specifies whether output is computed in training mode or in testing mode, which can change the logic, for instance in there are any `Dropout` layers in the network. 
-
-
-
-```python
-get_input(train)
-```
-
-Get the input of the layer.
-
-- __Return__: Theano tensor.
-
-- __Arguments__: 
-    - __train__: Boolean. Specifies whether output is computed in training mode or in testing mode, which can change the logic, for instance in there are any `Dropout` layers in the network. 
-
-
-
-```python
-get_weights()
-```
-
-Get the weights of the parameters of the layer.
-
-- __Return__: List of numpy arrays (one per layer parameter). 
-
-
-
-```python
-set_weights(weights)
-```
-
-Set the weights of the parameters of the layer.
-
-- __Arguments__: 
-    - __weights__: List of numpy arrays (one per layer parameter). Should be in the same order as what `get_weights(self)` returns.
-
-
-
----
-
-## Dense
-```python
-keras.layers.core.Dense(input_dim, output_dim, init='glorot_uniform', activation='linear', weights=None \
-W_regularizer=None, b_regularizer=None, W_constraint=None, b_constraint=None)
-```
-
-Standard 1D fully-connect layer. 
-
-- __Input shape__: 2D tensor with shape: `(nb_samples, input_dim)`.
-
-- __Output shape__: 2D tensor with shape: `(nb_samples, output_dim)`.
-
-- __Arguments__:
-
-    - __input_dim__: int >= 0. 
-    - __output_dim__: int >= 0. 
-    - __init__: name of initialization function for the weights of the layer (see: [initializations](../initializations.md)), or alternatively, Theano function to use for weights initialization. This parameter is only relevant if you don't pass a `weights` argument.
-    - __activation__: name of activation function to use (see: [activations](../activations.md)), or alternatively, elementwise Theano function. If you don't specify anything, no activation is applied (ie. "linear" activation: a(x) = x).
-    - __weights__: list of numpy arrays to set as initial weights. The list should have 1 element, of shape `(input_dim, output_dim)`.
-    - __W_regularizer__: instance of the [regularizers](../regularizers.md) module (eg. L1 or L2 regularization), applied to the main weights matrix.
-    - __b_regularizer__: instance of the [regularizers](../regularizers.md) module, applied to the bias.
-    - __W_constraint__: instance of the [constraints](../constraints.md) module (eg. maxnorm, nonneg), applied to the main weights matrix.
-    - __b_constraint__: instance of the [constraints](../constraints.md) module, applied to the bias.
-
----
-
-## TimeDistributedDense
-```python
-keras.layers.core.TimeDistributedDense(input_dim, output_dim, init='glorot_uniform', activation='linear', weights=None \
-W_regularizer=None, b_regularizer=None, W_constraint=None, b_constraint=None)
-```
-
-Fully-connected layer distributed over the time dimension. Useful after a recurrent network set to `return_sequences=True`.
-
-- __Input shape__: 3D tensor with shape: `(nb_samples, nb_timesteps, input_dim)`.
-
-- __Arguments__:
-    - __input_dim__: int >= 0. 
-    - __output_dim__: int >= 0. 
-    - __init__: name of initialization function for the weights of the layer (see: [initializations](../initializations.md)), or alternatively, Theano function to use for weights initialization. This parameter is only relevant if you don't pass a `weights` argument.
-    - __activation__: name of activation function to use (see: [activations](../activations.md)), or alternatively, elementwise Theano function. If you don't specify anything, no activation is applied (ie. "linear" activation: a(x) = x).
-    - __weights__: list of numpy arrays to set as initial weights. The list should have 1 element, of shape `(input_dim, output_dim)`.
-    - __W_regularizer__: instance of the [regularizers](../regularizers.md) module (eg. L1 or L2 regularization), applied to the main weights matrix.
-    - __b_regularizer__: instance of the [regularizers](../regularizers.md) module, applied to the bias.
-    - __W_constraint__: instance of the [constraints](../constraints.md) module (eg. maxnorm, nonneg), applied to the main weights matrix.
-    - __b_constraint__: instance of the [constraints](../constraints.md) module, applied to the bias.
-
-- __Example__:
-```python
-# input shape: (nb_samples, nb_timesteps, 10)
-model.add(LSTM(10, 5, return_sequences=True)) # output shape: (nb_samples, nb_timesteps, 5)
-model.add(TimeDistributedDense(5, 10)) # output shape: (nb_samples, nb_timesteps, 10)
-```
-
-
----
-
-## AutoEncoder
-```python
-keras.layers.core.AutoEncoder(encoder, decoder, output_reconstruction=True, tie_weights=False, weights=None):
-```
-
-A customizable autoencoder model. If `output_reconstruction = True` then dim(input) = dim(output) else dim(output) = dim(hidden)
-
-
-- __Input shape__: The layer shape is defined by the encoder definitions
-
-- __Output shape__: The layer shape is defined by the decoder definitions
-
-- __Arguments__:
-
-    - __encoder__: A [layer](./) or [layer container](./containers.md).
-
-    - __decoder__: A [layer](./) or [layer container](./containers.md).
-    
-    - __output_reconstruction__: If this is False the when .predict() is called the output is the deepest hidden layer's activation. Otherwise the output of the final decoder layer is presented. Be sure your validation data confirms to this logic if you decide to use any.
-    
-    - __tie_weights__: If True then the encoder bias is tied to the decoder bias. **Note**: This required the encoder layer corresponding to this decoder layer to be of the same time, eg: Dense:Dense
-    
-    - __weights__: list of numpy arrays to set as initial weights. The list should have 1 element, of shape `(input_dim, output_dim)`.
-
-- __Example__:
-```python
-from keras.layers import containers
-
-# input shape: (nb_samples, 32)
-encoder = containers.Sequential([Dense(32, 16), Dense(16, 8)])
-decoder = containers.Sequential([Dense(8, 16), Dense(16, 32)])
-autoencoder.add(AutoEncoder(encoder=encoder, decoder=decoder, output_reconstruction=False, tie_weights=True))
-```
-
----
-
-## DenoisingAutoEncoder
-```python
-keras.layers.core.AutoEncoder(encoder, decoder, output_reconstruction=True, tie_weights=False, weights=None, corruption_level=0.3):
-```
-
-A denoising autoencoder model that inherits the base features from autoencoder.
-Since this layer uses similar logic to Dropout it cannot be the first layer in a pipeline.
-
-- __Input shape__: The layer shape is defined by the encoder definitions
-
-- __Output shape__: The layer shape is defined by the decoder definitions
-
-- __Arguments__:
-
-    - __encoder__: A [layer](./) or [layer container](./containers.md).
-
-    - __decoder__: A [layer](./) or [layer container](./containers.md).
-    
-    - __output_reconstruction__: If this is False the when .predict() is called the output is the deepest hidden layer's activation. Otherwise the output of the final decoder layer is presented. Be sure your validation data confirms to this logic if you decide to use any.
-    
-    - __tie_weights__: If True then the encoder bias is tied to the decoder bias. **Note**: This required the encoder layer corresponding to this decoder layer to be of the same time, eg: Dense:Dense
-    
-    - __weights__: list of numpy arrays to set as initial weights. The list should have 1 element, of shape `(input_dim, output_dim)`.
-    
-    - __corruption_level__: the amount of binomial noise added to the input layer of the model.
-
-- __Example__:
-```python
-# input shape: (nb_samples, 32)
-autoencoder.add(Dense(32, 32))
-autoencoder.add(DenoisingAutoEncoder(encoder=Dense(32, 16),
-                                     decoder=Dense(16, 32),
-                                     output_reconstruction=False, tie_weights=True,
-                                     corruption_level=0.3))
-```
-
-
----
-
-## Activation
-```python
-keras.layers.core.Activation(activation)
-```
-Apply an activation function to the input. 
-
-- __Input shape__: This layer does not assume a specific input shape. As a result, it cannot be used as the first layer in a model.
-
-- __Output shape__: Same as input.
-
-- __Arguments__:
-
-    - __activation__: name of activation function to use (see: [activations](../activations.md)), or alternatively, elementwise Theano function.
-
-
----
-
-## Dropout
-```python
-keras.layers.core.Dropout(p)
-```
-Apply dropout to the input. Dropout consists in randomly setting a fraction `p` of input units to 0 at each update during training time, which helps prevent overfitting. Reference: [Dropout: A Simple Way to Prevent Neural Networks from Overfitting](http://www.cs.toronto.edu/~rsalakhu/papers/srivastava14a.pdf)
-
-- __Input shape__: This layer does not assume a specific input shape. 
-
-- __Output shape__: Same as input.
-
-- __Arguments__:
-
-    - __p__: float (0 <= p < 1). Fraction of the input that gets dropped out at training time. 
-
-
----
-
-## Reshape
-```python
-keras.layers.core.Reshape(*dims)
-```
-
-Reshape the input to a new shape containing the same number of units. 
-
-- __Input shape__: This layer does not assume a specific input shape. 
-
-- __Output shape__: `(nb_samples, *dims)`.
-
-- __Arguments__:
-
-    - *dims: integers. Dimensions of the new shape.
-
-- __Example__:
-```python
-# input shape: (nb_samples, 10)
-model.add(Dense(10, 100)) # output shape: (nb_samples, 100)
-model.add(Reshape(10, 10))  # output shape: (nb_samples, 10, 10)
-```
-
----
-
-## Flatten
-```python
-keras.layers.core.Flatten()
-```
-
-Convert a nD input to 1D. 
-
-- __Input shape__: (nb_samples, *). This layer cannot be used as the first layer in a model.
-
-- __Output shape__: `(nb_samples, nb_input_units)`.
-
----
-
-## RepeatVector
-```python
-keras.layers.core.RepeatVector(n)
-```
-
-Repeat the 1D input n times. Dimensions of input are assumed to be (nb_samples, dim). Output will have the shape (nb_samples, n, dim).
-
-- __Input shape__: This layer does not assume a specific input shape. This layer cannot be used as the first layer in a model.
-
-- __Output shape__: `(nb_samples, n, input_dims)`.
-
-- __Arguments__:
-    - __n__: int. 
-
-- __Example__:
-
----
-
-## MaxoutDense
-```python
-keras.layers.core.MaxoutDense(input_dim, output_dim, nb_feature=4, init='glorot_uniform', weights=None, \
-        W_regularizer=None, b_regularizer=None, W_constraint=None, b_constraint=None)
-```
-
-A dense maxout layer. A `MaxoutDense` layer takes the element-wise maximum of `nb_feature` `Dense(input_dim, output_dim)` linear layers. This allows the layer to learn a convex, piecewise linear activation function over the inputs. See [this paper](http://arxiv.org/pdf/1302.4389.pdf) for more details. Note that this is a *linear* layer -- if you wish to apply activation function (you shouldn't need to -- they are universal function approximators), an `Activation` layer must be added after.
-
-- __Input shape__: 2D tensor with shape: `(nb_samples, input_dim)`.
-
-- __Output shape__: 2D tensor with shape: `(nb_samples, output_dim)`.
-
-- __Arguments__:
-
-    - __input_dim__: int >= 0. 
-    - __output_dim__: int >= 0. 
-    - __nb_feature__: int >= 0. the number of features to create for the maxout. This is equivalent to the number of piecewise elements to be allowed for the activation function. 
-    - __init__: name of initialization function for the weights of the layer (see: [initializations](../initializations.md)), or alternatively, Theano function to use for weights initialization. This parameter is only relevant if you don't pass a `weights` argument.
-    - __weights__: list of numpy arrays to set as initial weights. The list should have 1 element, of shape `(input_dim, output_dim)`.
-    - __W_regularizer__: instance of the [regularizers](../regularizers.md) module (eg. L1 or L2 regularization), applied to the main weights matrix.
-    - __b_regularizer__: instance of the [regularizers](../regularizers.md) module, applied to the bias.
-    - __W_constraint__: instance of the [constraints](../constraints.md) module (eg. maxnorm, nonneg), applied to the main weights matrix.
-    - __b_constraint__: instance of the [constraints](../constraints.md) module, applied to the bias.
-
-```python
-# input shape: (nb_samples, 10)
-model.add(Dense(10, 100)) # output shape: (nb_samples, 100)
-model.add(MaxoutDense(100, 100, nb_feature=10)) # output shape: (nb_samples, 100)
-model.add(RepeatVector(2))  # output shape: (nb_samples, 2, 10)
-```
-
-## Merge
-```python
-keras.layers.core.Merge(models, mode='sum')
-```
-
-Merge the output of a list of models into a single tensor, following one of two modes: `sum` or `concat`. 
-
-- __Arguments__:
-    - __models__: List of `Sequential` models.
-    - __mode__: String, one of `{'sum', 'concat'}`. `sum` will simply sum the outputs of the models (therefore all models should have an output with the same shape). `concat` will concatenate the outputs along the last dimension (therefore all models should have an output that only differ along the last dimension). 
-
-- __Example__:
-
-```python
-left = Sequential()
-left.add(Dense(784, 50))
-left.add(Activation('relu'))
-
-right = Sequential()
-right.add(Dense(784, 50))
-right.add(Activation('relu'))
-
-model = Sequential()
-model.add(Merge([left, right], mode='sum'))
-
-model.add(Dense(50, 10))
-model.add(Activation('softmax'))
-
-model.compile(loss='categorical_crossentropy', optimizer='rmsprop')
-
-model.fit([X_train, X_train], Y_train, batch_size=128, nb_epoch=20, validation_data=([X_test, X_test], Y_test))
-```
-

docs/sources/layers/embeddings.md

@@ -1,48 +0,0 @@
-
-## Embedding
-
-```python
-keras.layers.embeddings.Embedding(input_dim, output_dim, init='uniform', weights=None, W_regularizer=None, W_constraint=None)
-```
-
-Turn positive integers (indexes) into denses vectors of fixed size,
-eg. `[[4], [20]] -> [[0.25, 0.1], [0.6, -0.2]]`
-
-- __Input shape__: 2D tensor with shape: `(nb_samples, maxlen)`.
-
-- __Output shape__: 3D tensor with shape: `(nb_samples, maxlen, output_dim)`.
-
-- __Arguments__:
-
-    - __input_dim__: int >= 0. Size of the vocabulary, ie. 1+maximum integer index occuring in the input data.
-    - __output_dim__: int >= 0. Dimension of the dense embedding.
-    - __init__: name of initialization function for the weights of the layer (see: [initializations](../initializations.md)), or alternatively, Theano function to use for weights initialization. This parameter is only relevant if you don't pass a `weights` argument.
-    - __weights__: list of numpy arrays to set as initial weights. The list should have 1 element, of shape `(input_dim, output_dim)`.
-    - __W_regularizer__: instance of the [regularizers](../regularizers.md) module (eg. L1 or L2 regularization), applied to the embedding matrix.
-    - __W_constraint__: instance of the [constraints](../constraints.md) module (eg. maxnorm, nonneg), applied to the embedding matrix.
-
-
-## WordContextProduct
-
-```python
-keras.layers.embeddings.WordContextProduct(input_dim, proj_dim=128, 
-        init='uniform', activation='sigmoid', weights=None)
-```
-
-This layer turns a pair of words (a pivot word + a context word, ie. a word from the same context as a pivot, or a random, out-of-context word), indentified by their indices in a vocabulary, into two dense reprensentations (word representation and context representation).
-
-Then it returns `activation(dot(pivot_embedding, context_embedding))`, which can be trained to encode the probability of finding the context word in the context of the pivot word (or reciprocally depending on your training procedure).
-
-For more context, see Mikolov et al.: [Efficient Estimation of Word reprensentations in Vector Space](http://arxiv.org/pdf/1301.3781v3.pdf)
-
-- __Input shape__: 2D tensor with shape: `(nb_samples, 2)`.
-
-- __Output shape__: 2D tensor with shape: `(nb_samples, 1)`.
-
-- __Arguments__:
-
-    - __input_dim__: int >= 0. Size of the vocabulary, ie. 1+maximum integer index occuring in the input data.
-    - __proj_dim__: int >= 0. Dimension of the dense embedding used internally.
-    - __init__: name of initialization function for the embeddings (see: [initializations](../initializations.md)), or alternatively, Theano function to use for weights initialization. This parameter is only relevant if you don't pass a `weights` argument.
-    - __activation__: name of activation function to use (see: [activations](../activations.md)), or alternatively, elementwise Theano function.
-    - __weights__: list of numpy arrays to set as initial weights. The list should have 2 element, both of shape `(input_dim, proj_dim)`. The first element is the word embedding weights, the second one is the context embedding weights.

docs/sources/layers/normalization.md

@@ -1,20 +0,0 @@
-
-## BatchNormalization
-
-```python
-keras.layers.normalization.BatchNormalization(input_shape, epsilon=1e-6, weights=None)
-```
-
-Normalize the activations of the previous layer at each batch.
-
-- __Input shape__: Same as `input_shape`. This layer cannot be used as first layer in a model.
-
-- __Output shape__: Same as input.
-
-- __Arguments__:
-    - __input_shape__: tuple.
-    - __epsilon__: small float > 0. Fuzz parameter.
-    - __weights__: Initialization weights. List of 2 numpy arrays, with shapes: `[(input_shape,), (input_shape,)]`
-
-- __References__:
-    - [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](http://arxiv.org/pdf/1502.03167v3.pdf)

docs/sources/layers/recurrent.md

@@ -1,133 +0,0 @@
-
-## SimpleRNN
-
-```python
-keras.layers.recurrent.SimpleRNN(input_dim, output_dim, 
-        init='glorot_uniform', inner_init='orthogonal', activation='sigmoid', weights=None,
-        truncate_gradient=-1, return_sequences=False)
-```
-Fully connected RNN where output is to fed back to input. Not a particularly useful model, included for demonstration purposes.
-
-- __Input shape__: 3D tensor with shape: `(nb_samples, timesteps, input_dim)`.
-
-- __Output shape__: 
-    - if `return_sequences`: 3D tensor with shape: `(nb_samples, timesteps, ouput_dim)`.
-    - else: 2D tensor with shape: `(nb_samples, output_dim)`.
-
-- __Arguments__:
-    - __input_dim__: dimension of the input.
-    - __output_dim__: dimension of the internal projections and the final output.
-    - __init__: weight initialization function. Can be the name of an existing function (str), or a Theano function (see: [initializations](../initializations.md)).
-    - __activation__: activation function. Can be the name of an existing function (str), or a Theano function (see: [activations](../activations.md)).
-    - __weights__: list of numpy arrays to set as initial weights. The list should have 3 elements, of shapes: `[(input_dim, output_dim), (output_dim, output_dim), (output_dim,)]`.
-    - __truncate_gradient__: Number of steps to use in truncated BPTT. See: [Theano "scan"](http://deeplearning.net/software/theano/library/scan.html).
-    - __return_sequences__: Boolean. Whether to return the last output in the output sequence, or the full sequence.
-
----
-
-## SimpleDeepRNN
-
-```python
-keras.layers.recurrent.SimpleDeepRNN(input_dim, output_dim, depth=3,
-        init='glorot_uniform', inner_init='orthogonal', 
-        activation='sigmoid', inner_activation='hard_sigmoid',
-        weights=None, truncate_gradient=-1, return_sequences=False)
-```
-Fully connected RNN where the output of multiple timesteps (up to "depth" steps in the past) is fed back to the input: 
-
-```
-output = activation( W.x_t + b + inner_activation(U_1.h_tm1) + inner_activation(U_2.h_tm2) + ... )
-```
-
-Not a particularly useful model, included for demonstration purposes.
-
-- __Input shape__: 3D tensor with shape: `(nb_samples, timesteps, input_dim)`.
-
-- __Output shape__:
-    - if `return_sequences`: 3D tensor with shape: `(nb_samples, timesteps, ouput_dim)`.
-    - else: 2D tensor with shape: `(nb_samples, output_dim)`.
-
-- __Arguments__:
-    - __input_dim__: dimension of the input.
-    - __output_dim__: dimension of the internal projections and the final output.
-    - __depth__: int >= 1. Lookback depth (eg. depth=1 is equivalent to SimpleRNN).
-    - __init__: weight initialization function for the output cell. Can be the name of an existing function (str), or a Theano function (see: [initializations](../initializations.md)).
-    - __inner_init__: weight initialization function for the inner cells.
-    - __activation__: activation function for the output. Can be the name of an existing function (str), or a Theano function (see: [activations](../activations.md)).
-    - __inner_activation__: activation function for the inner cells.
-    - __weights__: list of numpy arrays to set as initial weights. The list should have depth+2 elements.
-    - __truncate_gradient__: Number of steps to use in truncated BPTT. See: [Theano "scan"](http://deeplearning.net/software/theano/library/scan.html).
-    - __return_sequences__: Boolean. Whether to return the last output in the output sequence, or the full sequence.
-
-
----
-
-## GRU
-
-```python
-keras.layers.recurrent.GRU(input_dim, output_dim=128, 
-        init='glorot_uniform', inner_init='orthogonal',
-        activation='sigmoid', inner_activation='hard_sigmoid',
-        weights=None, truncate_gradient=-1, return_sequences=False)
-```
-
-Gated Recurrent Unit - Cho et al. 2014.
-
-- __Input shape__: 3D tensor with shape: `(nb_samples, timesteps, input_dim)`.
-
-- __Output shape__:
-    - if `return_sequences`: 3D tensor with shape: `(nb_samples, timesteps, ouput_dim)`.
-    - else: 2D tensor with shape: `(nb_samples, output_dim)`.
-
-- __Arguments__:
-    - __input_dim__: dimension of the input.
-    - __output_dim__: dimension of the internal projections and the final output.
-    - __init__: weight initialization function for the output cell. Can be the name of an existing function (str), or a Theano function (see: [initializations](../initializations.md)).
-    - __inner_init__: weight initialization function for the inner cells.
-    - __activation__: activation function for the output. Can be the name of an existing function (str), or a Theano function (see: [activations](../activations.md)).
-    - __inner_activation__: activation function for the inner cells.
-    - __weights__: list of numpy arrays to set as initial weights. The list should have 9 elements.
-    - __truncate_gradient__: Number of steps to use in truncated BPTT. See: [Theano "scan"](http://deeplearning.net/software/theano/library/scan.html).
-    - __return_sequences__: Boolean. Whether to return the last output in the output sequence, or the full sequence.
-
-- __References__: 
-    - [On the Properties of Neural Machine Translation: Encoder–Decoder Approaches](http://www.aclweb.org/anthology/W14-4012)
-    - [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](http://arxiv.org/pdf/1412.3555v1.pdf)
-
----
-
-## LSTM
-
-```python
-keras.layers.recurrent.LSTM(input_dim, output_dim=128, 
-        init='glorot_uniform', inner_init='orthogonal', 
-        activation='tanh', inner_activation='hard_sigmoid',
-        weights=None, truncate_gradient=-1, return_sequences=False)
-```
-
-Long-Short Term Memory unit - Hochreiter 1997.
-
-- __Input shape__: 3D tensor with shape: `(nb_samples, timesteps, input_dim)`.
-
-- __Output shape__:
-    - if `return_sequences`: 3D tensor with shape: `(nb_samples, timesteps, ouput_dim)`.
-    - else: 2D tensor with shape: `(nb_samples, output_dim)`.
-
-- __Arguments__:
-- __input_dim__: dimension of the input.
-    - __output_dim__: dimension of the internal projections and the final output.
-    - __init__: weight initialization function for the output cell. Can be the name of an existing function (str), or a Theano function (see: [initializations](../initializations.md)).
-    - __inner_init__: weight initialization function for the inner cells.
-    - __activation__: activation function for the output. Can be the name of an existing function (str), or a Theano function (see: [activations](../activations.md)).
-    - __inner_activation__: activation function for the inner cells.
-    - __weights__: list of numpy arrays to set as initial weights. The list should have 12 elements.
-    - __truncate_gradient__: Number of steps to use in truncated BPTT. See: [Theano "scan"](http://deeplearning.net/software/theano/library/scan.html).
-    - __return_sequences__: Boolean. Whether to return the last output in the output sequence, or the full sequence.
-
-- __References__: 
-    - [Long short-term memory](http://deeplearning.cs.cmu.edu/pdfs/Hochreiter97_lstm.pdf) (original 1997 paper)
-    - [Learning to forget: Continual prediction with LSTM](http://www.mitpressjournals.org/doi/pdf/10.1162/089976600300015015)
-    - [Supervised sequence labelling with recurrent neural networks](http://www.cs.toronto.edu/~graves/preprint.pdf)
-            
-            
-                

docs/sources/models.md

@@ -1,114 +0,0 @@
-## Sequential
-
-Linear stack of layers.
-
-```python
-model = keras.models.Sequential()
-```
-- __Methods__:
-    - __add__(layer): Add a layer to the model.
-    - __compile__(optimizer, loss, class_mode="categorical"): 
-        - __Arguments__: 
-            - __optimizer__: str (name of optimizer) or optimizer object. See [optimizers](optimizers.md).
-            - __loss__: str (name of objective function) or objective function. See [objectives](objectives.md).
-            - __class_mode__: one of "categorical", "binary". This is only used for computing classification accuracy or using the predict_classes method. 
-            - __theano_mode__: A `theano.compile.mode.Mode` ([reference](http://deeplearning.net/software/theano/library/compile/mode.html)) instance controlling specifying compilation options.
-    - __fit__(X, y, batch_size=128, nb_epoch=100, verbose=1, validation_split=0., validation_data=None, shuffle=True, show_accuracy=False, callbacks=[]): Train a model for a fixed number of epochs.
-        - __Return__: a history dictionary with a record of training loss values at successive epochs, as well as validation loss values (if applicable), accuracy (if applicable), etc.
-        - __Arguments__: 
-            - __X__: data.
-            - __y__: labels.
-            - __batch_size__: int. Number of samples per gradient update.
-            - __nb_epoch__: int. 
-            - __verbose__: 0 for no logging to stdout, 1 for progress bar logging, 2 for one log line per epoch.
-            - __validation_split__: float (0. < x < 1). Fraction of the data to use as held-out validation data.
-            - __validation_data__: tuple (X, y) to be used as held-out validation data. Will override validation_split.
-            - __shuffle__: boolean. Whether to shuffle the samples at each epoch.
-            - __show_accuracy__: boolean. Whether to display class accuracy in the logs to stdout at each epoch.
-            - __callbacks__: `keras.callbacks.Callback` list. List of callbacks to apply during training. See [callbacks](callbacks.md).
-    - __evaluate__(X, y, batch_size=128, show_accuracy=False, verbose=1): Show performance of the model over some validation data.
-        - __Return__: The loss score over the data.
-        - __Arguments__: Same meaning as fit method above. verbose is used as a binary flag (progress bar or nothing).
-    - __predict__(X, batch_size=128, verbose=1): 
-        - __Return__: An array of predictions for some test data.
-        - __Arguments__: Same meaning as fit method above.
-    - __predict_classes__(X, batch_size=128, verbose=1): Return an array of class predictions for some test data.
-        - __Return__: An array of labels for some test data.
-        - __Arguments__: Same meaning as fit method above. verbose is used as a binary flag (progress bar or nothing).
-    - __train__(X, y, accuracy=False): Single gradient update on one batch. if accuracy==False, return tuple (loss_on_batch, accuracy_on_batch). Else, return loss_on_batch.
-        - __Return__: loss over the data, or tuple `(loss, accuracy)` if `accuracy=True`.
-    - __test__(X, y, accuracy=False): Single performance evaluation on one batch. if accuracy==False, return tuple (loss_on_batch, accuracy_on_batch). Else, return loss_on_batch.
-        - __Return__: loss over the data, or tuple `(loss, accuracy)` if `accuracy=True`.
-    - __save_weights__(fname, overwrite=False): Store the weights of all layers to a HDF5 file. If overwrite==False and the file already exists, an exception will be thrown.
-    - __load_weights__(fname): Sets the weights of a model, based to weights stored by __save__weights__. You can only __load__weights__ on a savefile from a model with an identical architecture. __load_weights__ can be called either before or after the __compile__ step.
-
-__Examples__:
-
-```python
-from keras.models import Sequential
-from keras.layers.core import Dense, Dropout, Activation
-from keras.optimizers import SGD
-
-model = Sequential()
-model.add(Dense(64, 2, init='uniform'))
-model.add(Activation('softmax'))
-
-model.compile(loss='mse', optimizer='sgd')
-
-'''
-Demonstration of verbose modes 1 and 2
-'''
-model.fit(X_train, y_train, nb_epoch=3, batch_size=16, verbose=1)
-# outputs
-'''
-Train on 37800 samples, validate on 4200 samples
-Epoch 0
-37800/37800 [==============================] - 7s - loss: 0.0385
-Epoch 1
-37800/37800 [==============================] - 8s - loss: 0.0140
-Epoch 2
-10960/37800 [=======>......................] - ETA: 4s - loss: 0.0109
-'''
-
-model.fit(X_train, y_train, nb_epoch=3, batch_size=16, verbose=2)
-# outputs
-'''
-Train on 37800 samples, validate on 4200 samples
-Epoch 0
-loss: 0.0190
-Epoch 1
-loss: 0.0146
-Epoch 2
-loss: 0.0049
-'''
-
-'''
-Demonstration of show_accuracy
-'''
-model.fit(X_train, y_train, nb_epoch=3, batch_size=16, verbose=2, show_accuracy=True)
-# outputs
-'''
-Train on 37800 samples, validate on 4200 samples
-Epoch 0
-loss: 0.0190 - acc.: 0.8750
-Epoch 1
-loss: 0.0146 - acc.: 0.8750
-Epoch 2
-loss: 0.0049 - acc.: 1.0000
-'''
-
-'''
-Demonstration of validation_split
-'''
-model.fit(X_train, y_train, nb_epoch=3, batch_size=16, validation_split=0.1, show_accuracy=True, verbose=1)
-# outputs
-'''
-Train on 37800 samples, validate on 4200 samples
-Epoch 0
-37800/37800 [==============================] - 7s - loss: 0.0385 - acc.: 0.7258 - val. loss: 0.0160 - val. acc.: 0.9136
-Epoch 1
-37800/37800 [==============================] - 8s - loss: 0.0140 - acc.: 0.9265 - val. loss: 0.0109 - val. acc.: 0.9383
-Epoch 2
-10960/37800 [=======>......................] - ETA: 4s - loss: 0.0109 - acc.: 0.9420
-'''
-```

docs/sources/optimizers.md

@@ -1,118 +0,0 @@
-
-## Usage of optimizers
-
-An optimizer is one of the two arguments required for compiling a Keras model:
-
-```python
-model = Sequential()
-model.add(Dense(20, 64, init='uniform'))
-model.add(Activation('tanh'))
-model.add(Activation('softmax'))
-
-sgd = SGD(lr=0.1, decay=1e-6, momentum=0.9, nesterov=True)
-model.compile(loss='mean_squared_error', optimizer=sgd)
-```
-
-You can either instantiate an optimizer before passing it to `model.compile()` , as in the above example, or you can call it by its name. In the latter case, the default parameters for the optimizer will be used.
-
-```python
-# pass optimizer by name: default parameters will be used
-model.compile(loss='mean_squared_error', optimizer='sgd')
-```
-
----
-
-## Base class
-
-```python
-keras.optimizers.Optimizer(**kwargs)
-```
-
-All optimizers descended from this class support the following keyword argument:
-
-- __clipnorm__: float >= 0.
-
-Note: this is base class for building optimizers, not an actual optimizer that can be used for training models.
-
----
-
-##  SGD
-
-```python
-keras.optimizers.SGD(lr=0.01, momentum=0., decay=0., nesterov=False)
-``` 
-
-__Arguments__:
-
-- __lr__: float >= 0. Learning rate.
-- __momentum__: float >= 0. Parameter updates momentum.
-- __decay__: float >= 0. Learning rate decay over each update.
-- __nesterov__: boolean. Whether to apply Nesterov momentum.
-
----
-
-##  Adagrad
-
-```python
-keras.optimizers.Adagrad(lr=0.01, epsilon=1e-6)
-```
-
-It is recommended to leave the parameters of this optimizer at their default values.
-
-__Arguments__:
-
-- __lr__: float >= 0. Learning rate. 
-- __epsilon__: float >= 0. 
-
----
-
-##  Adadelta
-
-```python
-keras.optimizers.Adadelta(lr=1.0, rho=0.95, epsilon=1e-6)
-```
-
-It is recommended to leave the parameters of this optimizer at their default values.
-
-__Arguments__:
-
-- __lr__: float >= 0. Learning rate. It is recommended to leave it at the default value.
-- __rho__: float >= 0. 
-- __epsilon__: float >= 0. Fuzz factor.
-
-For more info, see *"Adadelta: an adaptive learning rate method"* by Matthew Zeiler.
-
----
-
-##  RMSprop 
-
-```python
-keras.optimizers.RMSprop(lr=0.001, rho=0.9, epsilon=1e-6)
-```
-
-It is recommended to leave the parameters of this optimizer at their default values.
-
-__Arguments__:
-
-- __lr__: float >= 0. Learning rate. 
-- __rho__: float >= 0.
-- __epsilon__: float >= 0. Fuzz factor.
-
----
-
-## Adam
-
-```python
-keras.optimizers.Adam(lr=0.001, beta_1=0.9, beta_2=0.999, epsilon=1e-8, kappa=1-1e-8)
-```
-
-Adam optimizer, proposed by Kingma and Lei Ba in [Adam: A Method For Stochastic Optimization](http://arxiv.org/pdf/1412.6980v4.pdf). Default parameters are those suggested in the paper. The parameter "lambda" from the paper has been renamed kappa, for syntactic reasons.
-
-__Arguments__:
-
-- __lr__: float >= 0. Learning rate. 
-- __beta_1__, __beta_2__: floats, 0 < beta < 1. Generally close to 1.
-- __epsilon__: float >= 0. Fuzz factor.
-- __kappa__: float 0 < kappa < 1. Lambda parameter in the original paper.
-
----

docs/sources/regularizers.md

@@ -1,18 +0,0 @@
-## Usage of regularizers
-
-Regularizers allow to apply penalties on network parameters during optimization.
-
-The keyword arguments used for passing penalties to parameters in a layer will depend on the layer. 
-
-In the `Dense` layer it is simply `W_regularizer` for the main weights matrix, and `b_regularizer` for the bias.
-
-```python
-from keras.regularizers import l2
-model.add(Dense(64, 64, W_regularizer = l2(.01)))
-```
-
-## Available penalties
-
-- __l1__(l=0.01): L1 regularization penalty, also known as LASSO
-- __l2__(l=0.01): L2 regularization penalty, also known as weight decay, or Ridge
-- __l1l2__(l1=0.01, l2=0.01): L1-L2 regularization penalty, also known as ElasticNet

docs/sources/utils/visualization.md

@@ -1,28 +0,0 @@
-## Grapher
-
-Creates a visualization of the model structure using `pydot`.
-
-```python
-grapher = keras.utils.dot_utils.Grapher()
-```
-- __Methods__:
-    - __plot__(model, to_file): creates a graph visualizing the structure of `model` and writes it to `to_file`.
-      - __Arguments__:
-        - __model__: an instance of a Keras model (e.g. `Sequential`)
-        - __to_file__: the filename to save the visualization png to.
-
-__Examples__:
-
-```python
-from keras.models import Sequential
-from keras.layers.core import Dense, Activation
-from keras.utils.dot_utils import Grapher
-
-grapher = Grapher()
-
-model = Sequential()
-model.add(Dense(64, 2, init='uniform'))
-model.add(Activation('softmax'))
-grapher.plot(model, 'model.png')
-```
-

docs/templates/activations.md

@@ -0,0 +1,41 @@
+
+## Usage of activations
+
+Activations can either be used through an `Activation` layer, or through the `activation` argument supported by all forward layers:
+
+```python
+from keras.layers.core import Activation, Dense
+
+model.add(Dense(64))
+model.add(Activation('tanh'))
+```
+is equivalent to:
+```python
+model.add(Dense(64, activation='tanh'))
+```
+
+You can also pass an element-wise Theano/TensorFlow function as an activation:
+
+```python
+from keras import backend as K
+
+def tanh(x):
+    return K.tanh(x)
+
+model.add(Dense(64, activation=tanh))
+model.add(Activation(tanh))
+```
+
+## Available activations
+
+- __softmax__: Softmax applied across inputs last dimension. Expects shape either `(nb_samples, nb_timesteps, nb_dims)` or `(nb_samples, nb_dims)`.
+- __softplus__
+- __relu__
+- __tanh__
+- __sigmoid__
+- __hard_sigmoid__
+- __linear__
+
+## On Advanced Activations
+
+Activations that are more complex than a simple Theano/TensorFlow function (eg. learnable activations, configurable activations, etc.) are available as [Advanced Activation layers](layers/advanced_activations.md), and can be found in the module `keras.layers.advanced_activations`. These include PReLU and LeakyReLU.

docs/templates/backend.md

@@ -0,0 +1,83 @@
+# Keras backends
+
+## What is a "backend"?
+
+Keras is a model-level library, providing high-level building blocks for developing deep learning models. It does not handle itself low-level operations such as tensor products, convolutions and so on. Instead, it relies on a specialized, well-optimized tensor manipulation library to do so, serving as the "backend engine" of Keras. Rather than picking one single tensor library and making the implementation of Keras tied to that library, Keras handles the problem in a modular way, and several different backend engines can be plugged seamlessly into Keras.
+
+At this time, Keras has two backend implementations available: the **Theano** backend and the **TensorFlow** backend.
+
+- [Theano](http://deeplearning.net/software/theano/) is an open-source symbolic tensor manipulation framework developed by LISA/MILA Lab at Université de Montréal.
+- [TensorFlow](http://www.tensorflow.org/) is an open-source symbolic tensor manipulation framework developed by Google, Inc.
+
+## Switching from one backend to another
+
+If you have run Keras at least once, you will find the Keras configuration file at:
+
+`~/.keras/keras.json`
+
+If it isn't there, you can create it.
+
+It probably looks like this:
+
+`{"epsilon": 1e-07, "floatx": "float32", "backend": "theano"}`
+
+Simply change the field `backend` to either `"theano"` or `"tensorflow"`, and Keras will use the new configuration next time you run any Keras code.
+
+You can also define the environment variable ``KERAS_BACKEND`` and this will
+override what is defined in your config file :
+
+```bash
+KERAS_BACKEND=tensorflow python -c "from keras import backend; print backend._BACKEND"
+Using TensorFlow backend.
+tensorflow
+```
+
+## Using the abstract Keras backend to write new code
+
+If you want the Keras modules you write to be compatible with both Theano and TensorFlow, you have to write them via the abstract Keras backend API. Here's an intro.
+
+You can import the backend module via:
+```python
+from keras import backend as K
+```
+
+The code below instantiates an input placeholder. It's equivalent to `tf.placeholder()` or `T.matrix()`, `T.tensor3()`, etc.
+
+```python
+input = K.placeholder(shape=(2, 4, 5))
+# also works:
+input = K.placeholder(shape=(None, 4, 5))
+# also works:
+input = K.placeholder(ndim=3)
+```
+
+The code below instantiates a shared variable. It's equivalent to `tf.variable()` or `theano.shared()`.
+
+```python
+val = np.random.random((3, 4, 5))
+var = K.variable(value=val)
+
+# all-zeros variable:
+var = K.zeros(shape=(3, 4, 5))
+# all-ones:
+var = K.ones(shape=(3, 4, 5))
+```
+
+Most tensor operations you will need can be done as you would in TensorFlow or Theano:
+
+```python
+a = b + c * K.abs(d)
+c = K.dot(a, K.transpose(b))
+a = K.sum(b, axis=2)
+a = K.softmax(b)
+a = concatenate([b, c], axis=-1)
+# etc...
+```
+
+For more information, see the code at `keras/backend/theano_backend.py` and `keras/backend/tensorflow_backend.py`.
+
+
+
+
+
+

docs/templates/callbacks.md

@@ -0,0 +1,72 @@
+## Usage of callbacks
+
+A callback is a set of functions to be applied at given stages of the training procedure. You can use callbacks to get a view on internal states and statistics of the model during training. You can pass a list of callbacks (as the keyword argument `callbacks`) to the `.fit()` method of the `Sequential` model. The relevant methods of the callbacks will then be called at each stage of the training. 
+
+---
+
+{{autogenerated}}
+
+---
+
+
+# Create a callback
+
+You can create a custom callback by extending the base class `keras.callbacks.Callback`. A callback has access to its associated model through the class property `self.model`.
+
+Here's a simple example saving a list of losses over each batch during training:
+```python
+class LossHistory(keras.callbacks.Callback):
+    def on_train_begin(self, logs={}):
+        self.losses = []
+
+    def on_batch_end(self, batch, logs={}):
+        self.losses.append(logs.get('loss'))
+```
+
+---
+
+### Example: recording loss history
+
+```python
+class LossHistory(keras.callbacks.Callback):
+    def on_train_begin(self, logs={}):
+        self.losses = []
+
+    def on_batch_end(self, batch, logs={}):
+        self.losses.append(logs.get('loss'))
+
+model = Sequential()
+model.add(Dense(10, input_dim=784, init='uniform'))
+model.add(Activation('softmax'))
+model.compile(loss='categorical_crossentropy', optimizer='rmsprop')
+
+history = LossHistory()
+model.fit(X_train, Y_train, batch_size=128, nb_epoch=20, verbose=0, callbacks=[history])
+
+print history.losses
+# outputs
+'''
+[0.66047596406559383, 0.3547245744908703, ..., 0.25953155204159617, 0.25901699725311789]
+'''
+```
+
+---
+
+### Example: model checkpoints
+
+```python
+from keras.callbacks import ModelCheckpoint
+
+model = Sequential()
+model.add(Dense(10, input_dim=784, init='uniform'))
+model.add(Activation('softmax'))
+model.compile(loss='categorical_crossentropy', optimizer='rmsprop')
+
+'''
+saves the model weights after each epoch if the validation loss decreased
+'''
+checkpointer = ModelCheckpoint(filepath="/tmp/weights.hdf5", verbose=1, save_best_only=True)
+model.fit(X_train, Y_train, batch_size=128, nb_epoch=20, verbose=0, validation_data=(X_test, Y_test), callbacks=[checkpointer])
+
+```
+

docs/templates/constraints.md

@@ -2,14 +2,17 @@
 
 Functions from the `constraints` module allow setting constraints (eg. non-negativity) on network parameters during optimization.
 
-The keyword arguments used for passing constraints to parameters in a layer will depend on the layer. 
+The penalties are applied on a per-layer basis. The exact API will depend on the layer, but the layers `Dense`, `TimeDistributedDense`, `MaxoutDense`, `Convolution1D` and `Convolution2D` have a unified API.
 
-In the `Dense` layer it is simply `W_constraint` for the main weights matrix, and `b_constraint` for the bias.
+These layers expose 2 keyword arguments:
+
+- `W_constraint` for the main weights matrix
+- `b_constraint` for the bias.
 
 
 ```python
 from keras.constraints import maxnorm
-model.add(Dense(64, 64, W_constraint = maxnorm(2)))
+model.add(Dense(64, W_constraint = maxnorm(2)))
 ```
 
 ## Available constraints

docs/templates/documentation.md

@@ -5,6 +5,8 @@
 - [Home](index.md)
 - [Index](documentation.md)
 - [Examples](examples.md)
+- [FAQ](faq.md)
+- [Backend](backend.md)
 
 ---
 
@@ -15,6 +17,9 @@
 - [Models](models.md)
 - [Activations](activations.md)
 - [Initializations](initializations.md)
+- [Regularizers](regularizers.md)
+- [Constraints](constraints.md)
+- [Callbacks](callbacks.md)
 - [Datasets](datasets.md)
 
 ---

docs/templates/examples.md

@@ -0,0 +1,390 @@
+
+Here are a few examples to get you started!
+
+In the examples folder, you will also find example models for real datasets:
+
+- CIFAR10 small images classification: Convolutional Neural Network (CNN) with realtime data augmentation
+- IMDB movie review sentiment classification: LSTM over sequences of words
+- Reuters newswires topic classification: Multilayer Perceptron (MLP)
+- MNIST handwritten digits classification: MLP & CNN
+- Character-level text generation with LSTM
+
+...and more.
+
+------------------
+
+### Multilayer Perceptron (MLP) for multi-class softmax classification:
+
+```python
+from keras.models import Sequential
+from keras.layers import Dense, Dropout, Activation
+from keras.optimizers import SGD
+
+model = Sequential()
+# Dense(64) is a fully-connected layer with 64 hidden units.
+# in the first layer, you must specify the expected input data shape:
+# here, 20-dimensional vectors.
+model.add(Dense(64, input_dim=20, init='uniform'))
+model.add(Activation('tanh'))
+model.add(Dropout(0.5))
+model.add(Dense(64, init='uniform'))
+model.add(Activation('tanh'))
+model.add(Dropout(0.5))
+model.add(Dense(10, init='uniform'))
+model.add(Activation('softmax'))
+
+sgd = SGD(lr=0.1, decay=1e-6, momentum=0.9, nesterov=True)
+model.compile(loss='categorical_crossentropy',
+              optimizer=sgd)
+
+model.fit(X_train, y_train,
+          nb_epoch=20,
+          batch_size=16,
+          show_accuracy=True)
+score = model.evaluate(X_test, y_test, batch_size=16)
+```
+
+------------------
+
+### Alternative implementation of a similar MLP:
+
+```python
+model = Sequential()
+model.add(Dense(64, input_dim=20, activation='relu'))
+model.add(Dropout(0.5))
+model.add(Dense(64, activation='relu'))
+model.add(Dropout(0.5))
+model.add(Dense(10, activation='softmax'))
+
+model.compile(loss='categorical_crossentropy', optimizer='adadelta')
+```
+
+------------------
+
+### MLP for binary classification:
+```python
+model = Sequential()
+model.add(Dense(64, input_dim=20, init='uniform', activation='relu'))
+model.add(Dropout(0.5))
+model.add(Dense(64, activation='relu'))
+model.add(Dropout(0.5))
+model.add(Dense(1, activation='sigmoid'))
+
+model.compile(loss='binary_crossentropy',
+              optimizer='rmsprop')
+```
+
+------------------
+
+### VGG-like convnet:
+
+```python
+from keras.models import Sequential
+from keras.layers import Dense, Dropout, Activation, Flatten
+from keras.layers import Convolution2D, MaxPooling2D
+from keras.optimizers import SGD
+
+model = Sequential()
+# input: 100x100 images with 3 channels -> (3, 100, 100) tensors.
+# this applies 32 convolution filters of size 3x3 each.
+model.add(Convolution2D(32, 3, 3, border_mode='valid', input_shape=(3, 100, 100)))
+model.add(Activation('relu'))
+model.add(Convolution2D(32, 3, 3))
+model.add(Activation('relu'))
+model.add(MaxPooling2D(pool_size=(2, 2)))
+model.add(Dropout(0.25))
+
+model.add(Convolution2D(64, 3, 3, border_mode='valid'))
+model.add(Activation('relu'))
+model.add(Convolution2D(64, 3, 3))
+model.add(Activation('relu'))
+model.add(MaxPooling2D(pool_size=(2, 2)))
+model.add(Dropout(0.25))
+
+model.add(Flatten())
+# Note: Keras does automatic shape inference.
+model.add(Dense(256))
+model.add(Activation('relu'))
+model.add(Dropout(0.5))
+
+model.add(Dense(10))
+model.add(Activation('softmax'))
+
+sgd = SGD(lr=0.1, decay=1e-6, momentum=0.9, nesterov=True)
+model.compile(loss='categorical_crossentropy', optimizer=sgd)
+
+model.fit(X_train, Y_train, batch_size=32, nb_epoch=1)
+
+```
+
+------------------
+
+### Sequence classification with LSTM:
+
+```python
+from keras.models import Sequential
+from keras.layers import Dense, Dropout, Activation
+from keras.layers import Embedding
+from keras.layers import LSTM
+
+model = Sequential()
+model.add(Embedding(max_features, 256, input_length=maxlen))
+model.add(LSTM(output_dim=128, activation='sigmoid', inner_activation='hard_sigmoid'))
+model.add(Dropout(0.5))
+model.add(Dense(1))
+model.add(Activation('sigmoid'))
+
+model.compile(loss='binary_crossentropy', optimizer='rmsprop')
+
+model.fit(X_train, Y_train, batch_size=16, nb_epoch=10)
+score = model.evaluate(X_test, Y_test, batch_size=16)
+```
+
+### Architecture for learning image captions with a convnet and a Gated Recurrent Unit:
+(word-level embedding, caption of maximum length 16 words).
+
+Note that getting this to work well will require using a bigger convnet, initialized with pre-trained weights.
+
+```python
+max_caption_len = 16
+vocab_size = 10000
+
+# first, let's define an image model that
+# will encode pictures into 128-dimensional vectors.
+# it should be initialized with pre-trained weights.
+image_model = Sequential()
+image_model.add(Convolution2D(32, 3, 3, border_mode='valid', input_shape=(3, 100, 100)))
+image_model.add(Activation('relu'))
+image_model.add(Convolution2D(32, 3, 3))
+image_model.add(Activation('relu'))
+image_model.add(MaxPooling2D(pool_size=(2, 2)))
+
+image_model.add(Convolution2D(64, 3, 3, border_mode='valid'))
+image_model.add(Activation('relu'))
+image_model.add(Convolution2D(64, 3, 3))
+image_model.add(Activation('relu'))
+image_model.add(MaxPooling2D(pool_size=(2, 2)))
+
+image_model.add(Flatten())
+image_model.add(Dense(128))
+
+# let's load the weights from a save file.
+image_model.load_weights('weight_file.h5')
+
+# next, let's define a RNN model that encodes sequences of words
+# into sequences of 128-dimensional word vectors.
+language_model = Sequential()
+language_model.add(Embedding(vocab_size, 256, input_length=max_caption_len))
+language_model.add(GRU(output_dim=128, return_sequences=True))
+language_model.add(TimeDistributedDense(128))
+
+# let's repeat the image vector to turn it into a sequence.
+image_model.add(RepeatVector(max_caption_len))
+
+# the output of both models will be tensors of shape (samples, max_caption_len, 128).
+# let's concatenate these 2 vector sequences.
+model = Sequential()
+model.add(Merge([image_model, language_model], mode='concat', concat_axis=-1))
+# let's encode this vector sequence into a single vector
+model.add(GRU(256, return_sequences=False))
+# which will be used to compute a probability
+# distribution over what the next word in the caption should be!
+model.add(Dense(vocab_size))
+model.add(Activation('softmax'))
+
+model.compile(loss='categorical_crossentropy', optimizer='rmsprop')
+
+# "images" is a numpy float array of shape (nb_samples, nb_channels=3, width, height).
+# "captions" is a numpy integer array of shape (nb_samples, max_caption_len)
+# containing word index sequences representing partial captions.
+# "next_words" is a numpy float array of shape (nb_samples, vocab_size)
+# containing a categorical encoding (0s and 1s) of the next word in the corresponding
+# partial caption.
+model.fit([images, partial_captions], next_words, batch_size=16, nb_epoch=100)
+```
+
+------------------
+
+### Stacked LSTM for sequence classification
+
+In this model, we stack 3 LSTM layers on top of each other,
+making the model capable of learning higher-level temporal representations.
+
+The first two LSTMs return their full output sequences, but the last one only returns
+the last step in its output sequence, thus dropping the temporal dimension
+(i.e. converting the input sequence into a single vector).
+
+<img src="http://keras.io/img/regular_stacked_lstm.png" alt="stacked LSTM" style="width: 300px;"/>
+
+(N.B.: in Keras, "None" in an input shape indicates a variable dimension. In the graph above, the batch size is "None",
+meaning that any batch size is allowed for the input data).
+
+```python
+from keras.models import Sequential
+from keras.layers import LSTM, Dense
+import numpy as np
+
+data_dim = 16
+timesteps = 8
+nb_classes = 10
+
+# expected input data shape: (batch_size, timesteps, data_dim)
+model = Sequential()
+model.add(LSTM(32, return_sequences=True,
+               input_shape=(timesteps, data_dim)))  # returns a sequence of vectors of dimension 32
+model.add(LSTM(32, return_sequences=True))  # returns a sequence of vectors of dimension 32
+model.add(LSTM(32))  # return a single vector of dimension 32
+model.add(Dense(10, activation='softmax'))
+
+model.compile(loss='categorical_crossentropy', optimizer='rmsprop')
+
+# generate dummy training data
+x_train = np.random.random((1000, timesteps, data_dim))
+y_train = np.random.random((1000, nb_classes))
+
+# generate dummy validation data
+x_val = np.random.random((100, timesteps, data_dim))
+y_val = np.random.random((100, nb_classes))
+
+model.fit(x_train, y_train,
+          batch_size=64, nb_epoch=5, show_accuracy=True,
+          validation_data=(x_val, y_val))
+```
+
+------------------
+
+### Same stacked LSTM model, rendered "stateful"
+
+A stateful recurrent model is one for which the internal states (memories) obtained after processing a batch
+of samples are reused as initial states for the samples of the next batch. This allows to process longer sequences
+while keeping computational complexity manageable.
+
+[You can read more about stateful RNNs in the FAQ.](/faq/#how-can-i-use-stateful-rnns)
+
+```python
+from keras.models import Sequential
+from keras.layers import LSTM, Dense
+import numpy as np
+
+data_dim = 16
+timesteps = 8
+nb_classes = 10
+batch_size = 32
+
+# expected input batch shape: (batch_size, timesteps, data_dim)
+# note that we have to provide the full batch_input_shape since the network is stateful.
+# the sample of index i in batch k is the follow-up for the sample i in batch k-1.
+model = Sequential()
+model.add(LSTM(32, return_sequences=True, stateful=True,
+               batch_input_shape=(batch_size, timesteps, data_dim)))
+model.add(LSTM(32, return_sequences=True, stateful=True))
+model.add(LSTM(32, stateful=True))
+model.add(Dense(10, activation='softmax'))
+
+model.compile(loss='categorical_crossentropy', optimizer='rmsprop')
+
+# generate dummy training data
+x_train = np.random.random((batch_size * 10, timesteps, data_dim))
+y_train = np.random.random((batch_size * 10, nb_classes))
+
+# generate dummy validation data
+x_val = np.random.random((batch_size * 3, timesteps, data_dim))
+y_val = np.random.random((batch_size * 3, nb_classes))
+
+model.fit(x_train, y_train,
+          batch_size=batch_size, nb_epoch=5, show_accuracy=True,
+          validation_data=(x_val, y_val))
+```
+
+------------------
+
+### Two merged LSTM encoders for classification over two parallel sequences
+
+In this model, two input sequences are encoded into vectors by two separate LSTM modules.
+
+These two vectors are then concatenated, and a fully connected network is trained on top of the concatenated representations.
+
+![Dual LSTM](http://keras.io/img/dual_lstm.png)
+
+```python
+from keras.models import Sequential
+from keras.layers import Merge, LSTM, Dense
+import numpy as np
+
+data_dim = 16
+timesteps = 8
+nb_classes = 10
+
+encoder_a = Sequential()
+encoder_a.add(LSTM(32, input_shape=(timesteps, data_dim)))
+
+encoder_b = Sequential()
+encoder_b.add(LSTM(32, input_shape=(timesteps, data_dim)))
+
+decoder = Sequential()
+decoder.add(Merge([encoder_a, encoder_b], mode='concat'))
+decoder.add(Dense(32, activation='relu'))
+decoder.add(Dense(nb_classes, activation='softmax'))
+
+decoder.compile(loss='categorical_crossentropy', optimizer='rmsprop')
+
+# generate dummy training data
+x_train_a = np.random.random((1000, timesteps, data_dim))
+x_train_b = np.random.random((1000, timesteps, data_dim))
+y_train = np.random.random((1000, nb_classes))
+
+# generate dummy validation data
+x_val_a = np.random.random((100, timesteps, data_dim))
+x_val_b = np.random.random((100, timesteps, data_dim))
+y_val = np.random.random((100, nb_classes))
+
+decoder.fit([x_train_a, x_train_b], y_train,
+            batch_size=64, nb_epoch=5, show_accuracy=True,
+            validation_data=([x_val_a, x_val_b], y_val))
+```
+
+------------------
+
+### Single shared LSTM over two parallel sequences, for classification
+
+This is a similar setup as above, but now a single LSTM encoder is used for both input sequences.
+Such a setup makes sense if the two input sequences are the same type of object.
+
+<img src="http://keras.io/img/shared_lstm.png" alt="Shared LSTM" style="width: 500px;"/>
+
+```python
+from keras.models import Graph
+from keras.layers import LSTM, Dense
+import numpy as np
+
+data_dim = 16
+timesteps = 8
+nb_classes = 10
+
+encoder = Sequential()
+encoder.add(LSTM(32, input_shape=(timesteps, data_dim)))
+
+model = Graph()
+model.add_input(name='input_a', input_shape=(timesteps, data_dim))
+model.add_input(name='input_b', input_shape=(timesteps, data_dim))
+model.add_shared_node(encoder, name='shared_encoder', inputs=['input_a', 'input_b'],
+                      merge_mode='concat')
+model.add_node(Dense(64, activation='relu'), name='fc1', input='shared_encoder')
+model.add_node(Dense(3, activation='softmax'), name='output', input='fc1', create_output=True)
+
+model.compile(optimizer='adam', loss={'output': 'categorical_crossentropy'})
+
+# generate dummy training data
+x_train_a = np.random.random((1000, timesteps, data_dim))
+x_train_b = np.random.random((1000, timesteps, data_dim))
+y_train = np.random.random((1000, 3))
+
+# generate dummy validation data
+x_val_a = np.random.random((100, timesteps, data_dim))
+x_val_b = np.random.random((100, timesteps, data_dim))
+y_val = np.random.random((100, 3))
+
+model.fit({'input_a': x_train_a, 'input_b': x_train_b, 'output': y_train},
+          batch_size=64, nb_epoch=5,
+          validation_data={'input_a': x_val_a, 'input_b': x_val_b, 'output': y_val})
+```

docs/templates/faq.md

@@ -0,0 +1,244 @@
+# Keras FAQ: Frequently Asked Keras Questions
+
+[How should I cite Keras?](#how-should-i-cite-keras)
+
+[How can I run Keras on GPU?](#how-can-i-run-keras-on-gpu)
+
+[How can I save a Keras model?](#how-can-i-save-a-keras-model)
+
+[Why is the training loss much higher than the testing loss?](#why-is-the-training-loss-much-higher-than-the-testing-loss)
+
+[How can I visualize the output of an intermediate layer?](#how-can-i-visualize-the-output-of-an-intermediate-layer)
+
+[How can I use Keras with datasets that don't fit in memory?](#how-can-i-use-keras-with-datasets-that-dont-fit-in-memory)
+
+[How can I interrupt training when the validation loss isn't decreasing anymore?](#how-can-i-interrupt-training-when-the-validation-loss-isnt-decreasing-anymore)
+
+[How is the validation split computed?](#how-is-the-validation-split-computed)
+
+[Is the data shuffled during training?](#is-the-data-shuffled-during-training)
+
+[How can I record the training / validation loss / accuracy at each epoch?](#how-can-i-record-the-training-validation-loss-accuracy-at-each-epoch)
+
+[How can I use stateful RNNs?](#how-can-i-use-stateful-rnns)
+
+---
+
+### How should I cite Keras?
+
+Please cite Keras in your publications if it helps your research. Here is an example BibTeX entry:
+
+```
+@misc{chollet2015keras,
+  author = {Chollet, François},
+  title = {Keras},
+  year = {2015},
+  publisher = {GitHub},
+  journal = {GitHub repository},
+  howpublished = {\url{https://github.com/fchollet/keras}}
+}
+```
+
+### How can I run Keras on GPU?
+
+If you are running on the TensorFlow backend, your code will automatically run on GPU if any available GPU is detected.
+If you are running on the Theano backend, you can use one of the following methods:
+
+Method 1: use Theano flags.
+```bash
+THEANO_FLAGS=device=gpu,floatX=float32 python my_keras_script.py
+```
+
+The name 'gpu' might have to be changed depending on your device's identifier (e.g. `gpu0`, `gpu1`, etc).
+
+Method 2: set up your `.theanorc`: [Instructions](http://deeplearning.net/software/theano/library/config.html)
+
+Method 3: manually set `theano.config.device`, `theano.config.floatX` at the beginning of your code:
+```python
+import theano
+theano.config.device = 'gpu'
+theano.config.floatX = 'float32'
+```
+
+---
+
+### How can I save a Keras model?
+
+*It is not recommended to use pickle or cPickle to save a Keras model.*
+
+If you only need to save the architecture of a model, and not its weights, you can do:
+
+```python
+# save as JSON
+json_string = model.to_json()
+
+# save as YAML
+yaml_string = model.to_yaml()
+```
+
+You can then build a fresh model from this data:
+
+```python
+# model reconstruction from JSON:
+from keras.models import model_from_json
+model = model_from_json(json_string)
+
+# model reconstruction from YAML
+model = model_from_yaml(yaml_string)
+```
+
+If you need to save the weights of a model, you can do so in HDF5 with the code below.
+
+Note that you will first need to install HDF5 and the Python library h5py, which do not come bundled with Keras.
+
+```python
+model.save_weights('my_model_weights.h5')
+```
+
+Assuming you have code for instantiating your model, you can then load the weights you saved into a model with the same architecture:
+
+```python
+model.load_weights('my_model_weights.h5')
+```
+
+This leads us to a way to save and reconstruct models from only serialized data:
+```python
+json_string = model.to_json()
+open('my_model_architecture.json', 'w').write(json_string)
+model.save_weights('my_model_weights.h5')
+
+# elsewhere...
+model = model_from_json(open('my_model_architecture.json').read())
+model.load_weights('my_model_weights.h5')
+```
+
+---
+
+### Why is the training loss much higher than the testing loss?
+
+A Keras model has two modes: training and testing. Regularization mechanisms, such as Dropout and L1/L2 weight regularization, are turned off at testing time.
+
+Besides, the training loss is the average of the losses over each batch of training data. Because your model is changing over time, the loss over the first batches of an epoch is generally higher than over the last batches. On the other hand, the testing loss for an epoch is computed using the model as it is at the end of the epoch, resulting in a lower loss.
+
+---
+
+### How can I visualize the output of an intermediate layer?
+
+You can build a Keras function that will return the output of a certain layer given a certain input, for example:
+
+```python
+from keras import backend as K
+
+# with a Sequential model
+get_3rd_layer_output = K.function([model.layers[0].input],
+                                  [model.layers[3].get_output(train=False)])
+layer_output = get_3rd_layer_output([X])[0]
+
+# with a Graph model
+get_conv_layer_output = K.function([model.inputs[i].input for i in model.input_order],
+                                   [model.nodes['conv'].get_output(train=False)])
+conv_output = get_conv_layer_output([input_data_dict[i] for i in model.input_order])[0]
+```
+
+Similarly, you could build a Theano and TensorFlow function directly.
+
+---
+
+### How can I use Keras with datasets that don't fit in memory?
+
+You can do batch training using `model.train_on_batch(X, y)` and `model.test_on_batch(X, y)`. See the [models documentation](models.md).
+
+Alternatively, you can write a generator that yields batches of training data and use the method `model.fit_generator(data_generator, samples_per_epoch, nb_epoch)`.
+
+You can see batch training in action in our [CIFAR10 example](https://github.com/fchollet/keras/blob/master/examples/cifar10_cnn.py).
+
+---
+
+### How can I interrupt training when the validation loss isn't decreasing anymore?
+
+You can use an `EarlyStopping` callback:
+
+```python
+from keras.callbacks import EarlyStopping
+early_stopping = EarlyStopping(monitor='val_loss', patience=2)
+model.fit(X, y, validation_split=0.2, callbacks=[early_stopping])
+```
+
+Find out more in the [callbacks documentation](callbacks.md).
+
+---
+
+### How is the validation split computed?
+
+If you set the `validation_split` argument in `model.fit` to e.g. 0.1, then the validation data used will be the *last 10%* of the data. If you set it to 0.25, it will be the last 25% of the data, etc.
+
+
+---
+
+### Is the data shuffled during training?
+
+Yes, if the `shuffle` argument in `model.fit` is set to `True` (which is the default), the training data will be randomly shuffled at each epoch.
+
+Validation data isn't shuffled.
+
+---
+
+
+### How can I record the training / validation loss / accuracy at each epoch?
+
+The `model.fit` method returns an `History` callback, which has a `history` attribute containing the lists of successive losses / accuracies.
+
+```python
+hist = model.fit(X, y, validation_split=0.2)
+print(hist.history)
+```
+
+---
+
+### How can I use stateful RNNs?
+
+Making a RNN stateful means that the states for the samples of each batch will be reused as initial states for the samples in the next batch.
+
+When using stateful RNNs, it is therefore assumed that:
+
+- all batches have the same number of samples
+- If `X1` and `X2` are successive batches of samples, then `X2[i]` is the follow-up sequence to `X1[i]`, for every `i`.
+
+To use statefulness in RNNs, you need to:
+
+- explicitly specify the batch size you are using, by passing a `batch_input_shape` argument to the first layer in your model. It should be a tuple of integers, e.g. `(32, 10, 16)` for a 32-samples batch of sequences of 10 timesteps with 16 features per timestep.
+- set `stateful=True` in your RNN layer(s).
+
+To reset the states accumulated:
+
+- use `model.reset_states()` to reset the states of all layers in the model
+- use `layer.reset_states()` to reset the states of a specific stateful RNN layer
+
+Example:
+
+```python
+
+X  # this is our input data, of shape (32, 21, 16)
+# we will feed it to our model in sequences of length 10
+
+model = Sequential()
+model.add(LSTM(32, batch_input_shape=(32, 10, 16), stateful=True))
+model.add(Dense(16, activation='softmax'))
+
+model.compile(optimizer='rmsprop', loss='categorical_crossentropy')
+
+# we train the network to predict the 11th timestep given the first 10:
+model.train_on_batch(X[:, :10, :], np.reshape(X[:, 10, :], (32, 16)))
+
+# the state of the network has changed. We can feed the follow-up sequences:
+model.train_on_batch(X[:, 10:20, :], np.reshape(X[:, 20, :], (32, 16)))
+
+# let's reset the states of the LSTM layer:
+model.reset_states()
+
+# another way to do it in this case:
+model.layers[0].reset_states()
+```
+
+Notes that the methods `predict`, `fit`, `train_on_batch`, `predict_classes`, etc. will *all* update the states of the stateful layers in a model. This allows you to do not only stateful training, but also stateful prediction.
+

docs/templates/index.md

@@ -0,0 +1,163 @@
+# Keras: Deep Learning library for Theano and TensorFlow
+
+## You have just found Keras.
+
+Keras is a minimalist, highly modular neural networks library, written in Python and capable of running on top of either [TensorFlow](https://github.com/tensorflow/tensorflow) or [Theano](https://github.com/Theano/Theano). It was developed with a focus on enabling fast experimentation. Being able to go from idea to result with the least possible delay is key to doing good research.
+
+Use Keras if you need a deep learning library that:
+
+- allows for easy and fast prototyping (through total modularity, minimalism, and extensibility).
+- supports both convolutional networks and recurrent networks, as well as combinations of the two.
+- supports arbitrary connectivity schemes (including multi-input and multi-output training).
+- runs seamlessly on CPU and GPU.
+
+Read the documentation at [Keras.io](http://keras.io).
+
+Keras is compatible with: __Python 2.7-3.5__.
+
+
+------------------
+
+
+## Guiding principles
+
+- __Modularity.__ A model is understood as a sequence or a graph of standalone, fully-configurable modules that can be plugged together with as little restrictions as possible. In particular, neural layers, cost functions, optimizers, initialization schemes, activation functions, regularization schemes are all standalone modules that you can combine to create new models.
+
+- __Minimalism.__ Each module should be kept short and simple. Every piece of code should be transparent upon first reading. No black magic: it hurts iteration speed and ability to innovate.
+
+- __Easy extensibility.__ New modules are dead simple to add (as new classes and functions), and existing modules provide ample examples. To be able to easily create new modules allows for total expressiveness, making Keras suitable for advanced research.
+
+- __Work with Python__. No separate models configuration files in a declarative format. Models are described in Python code, which is compact, easier to debug, and allows for ease of extensibility.
+
+
+------------------
+
+
+## Getting started: 30 seconds to Keras
+
+The core datastructure of Keras is a __model__, a way to organize layers. There are two types of models: [`Sequential`](http://keras.io/models/#sequential) and [`Graph`](http://keras.io/models/#graph).
+
+Here's the `Sequential` model (a linear pile of layers):
+
+```python
+from keras.models import Sequential
+
+model = Sequential()
+```
+
+Stacking layers is as easy as `.add()`:
+
+```python
+from keras.layers.core import Dense, Activation
+
+model.add(Dense(output_dim=64, input_dim=100, init="glorot_uniform"))
+model.add(Activation("relu"))
+model.add(Dense(output_dim=10, init="glorot_uniform"))
+model.add(Activation("softmax"))
+```
+
+Once your model looks good, configure its learning process with `.compile()`:
+```python
+model.compile(loss='categorical_crossentropy', optimizer='sgd')
+```
+
+If you need to, you can further configure your optimizer. A core principle of Keras is to make things reasonably simple, while allowing the user to be fully in control when they need to (the ultimate control being the easy extensibility of the source code).
+```python
+from keras.optimizers import SGD
+model.compile(loss='categorical_crossentropy', optimizer=SGD(lr=0.01, momentum=0.9, nesterov=True))
+```
+
+You can now iterate on your training data in batches:
+```python
+model.fit(X_train, Y_train, nb_epoch=5, batch_size=32)
+```
+
+Alternatively, you can feed batches to your model manually:
+```python
+model.train_on_batch(X_batch, Y_batch)
+```
+
+Evaluate your performance in one line:
+```python
+objective_score = model.evaluate(X_test, Y_test, batch_size=32)
+```
+
+Or generate predictions on new data:
+```python
+classes = model.predict_classes(X_test, batch_size=32)
+proba = model.predict_proba(X_test, batch_size=32)
+```
+
+Building a network of LSTMs, a deep CNN, a Neural Turing Machine, a word2vec embedder or any other model is just as fast. The ideas behind deep learning are simple, so why should their implementation be painful?
+
+Have a look at these [starter examples](http://keras.io/examples/).
+
+In the [examples folder](https://github.com/fchollet/keras/tree/master/examples) of the repo, you will find more advanced models: question-answering with memory networks, text generation with stacked LSTMs, neural turing machines, etc.
+
+
+------------------
+
+
+## Installation
+
+Keras uses the following dependencies:
+
+- numpy, scipy
+- pyyaml
+- HDF5 and h5py (optional, required if you use model saving/loading functions)
+- Optional but recommended if you use CNNs: cuDNN.
+
+*When using the Theano backend:*
+
+- Theano
+    - [See installation instructions](http://deeplearning.net/software/theano/install.html#install).
+
+**Note**: You should use the latest version of Theano, not the PyPI version. Install it with:
+```
+sudo pip install git+git://github.com/Theano/Theano.git
+```
+
+*When using the TensorFlow backend:*
+
+- TensorFlow
+    - [See installation instructions](https://github.com/tensorflow/tensorflow#download-and-setup).
+
+To install Keras, `cd` to the Keras folder and run the install command:
+```
+sudo python setup.py install
+```
+
+You can also install Keras from PyPI:
+```
+sudo pip install keras
+```
+
+------------------
+
+
+## Switching from Theano to TensorFlow
+
+By default, Keras will use Theano as its tensor manipulation library. [Follow these instructions](http://keras.io/backend/) to configure the Keras backend.
+
+------------------
+
+
+## Support
+
+You can ask questions and join the development discussion on the [Keras Google group](https://groups.google.com/forum/#!forum/keras-users).
+
+You can also post bug reports and feature requests in [Github issues](https://github.com/fchollet/keras/issues). Make sure to read [our guidelines](https://github.com/fchollet/keras/blob/master/CONTRIBUTING.md) first.
+
+
+------------------
+
+
+## Why this name, Keras?
+
+Keras (κέρας) means _horn_ in Greek. It is a reference to a literary image from ancient Greek and Latin literature, first found in the _Odyssey_, where dream spirits (_Oneiroi_, singular _Oneiros_) are divided between those who deceive men with false visions, who arrive to Earth through a gate of ivory, and those who announce a future that will come to pass, who arrive through a gate of horn. It's a play on the words κέρας (horn) / κραίνω (fulfill), and ἐλέφας (ivory) / ἐλεφαίρομαι (deceive).
+
+Keras was initially developed as part of the research effort of project ONEIROS (Open-ended Neuro-Electronic Intelligent Robot Operating System).
+
+>_"Oneiroi are beyond our unravelling --who can be sure what tale they tell? Not all that men look for comes to pass. Two gates there are that give passage to fleeting Oneiroi; one is made of horn, one of ivory. The Oneiroi that pass through sawn ivory are deceitful, bearing a message that will not be fulfilled; those that come out through polished horn have truth behind them, to be accomplished for men who see them."_ Homer, Odyssey 19. 562 ff (Shewring translation).
+
+------------------

docs/templates/initializations.md

@@ -1,4 +1,3 @@
-# Initializations
 
 ## Usage of initializations
 
@@ -7,7 +6,7 @@ Initializations define the probability distribution used to set the initial rand
 The keyword arguments used for passing initializations to layers will depend on the layer. Usually it is simply `init`:
 
 ```python
-model.add(Dense(64, 64, init='uniform'))
+model.add(Dense(64, init='uniform'))
 ```
 
 ## Available initializations
@@ -15,6 +14,7 @@ model.add(Dense(64, 64, init='uniform'))
 - __uniform__
 - __lecun_uniform__: Uniform initialization scaled by the square root of the number of inputs (LeCun 98).
 - __normal__
+- __identity__: Use with square 2D layers (`shape[0] == shape[1]`).
 - __orthogonal__: Use with square 2D layers (`shape[0] == shape[1]`).
 - __zero__
 - __glorot_normal__: Gaussian initialization scaled by fan_in + fan_out (Glorot 2010)

docs/templates/models.md

@@ -0,0 +1,114 @@
+Keras has two models: __Sequential__, a linear stack of layers, and __Graph__, a directed acyclic graph of layers.
+
+# Using the Sequential model
+
+```python
+from keras.models import Sequential
+from keras.layers.core import Dense, Dropout, Activation
+from keras.optimizers import SGD
+
+model = Sequential()
+model.add(Dense(2, init='uniform', input_dim=64))
+model.add(Activation('softmax'))
+
+model.compile(optimizer='sgd', loss='mse')
+
+'''
+Train the model for 3 epochs, in batches of 16 samples,
+on data stored in the Numpy array X_train,
+and labels stored in the Numpy array y_train:
+'''
+model.fit(X_train, y_train, nb_epoch=3, batch_size=16, verbose=1)
+'''
+What you will see with mode verbose=1:
+Train on 37800 samples, validate on 4200 samples
+Epoch 0
+37800/37800 [==============================] - 7s - loss: 0.0385
+Epoch 1
+37800/37800 [==============================] - 8s - loss: 0.0140
+Epoch 2
+10960/37800 [=======>......................] - ETA: 4s - loss: 0.0109
+'''
+
+model.fit(X_train, y_train, nb_epoch=3, batch_size=16, verbose=2)
+'''
+What you will see with mode verbose=2:
+Train on 37800 samples, validate on 4200 samples
+Epoch 0
+loss: 0.0190
+Epoch 1
+loss: 0.0146
+Epoch 2
+loss: 0.0049
+'''
+
+'''
+Demonstration of the show_accuracy argument
+'''
+model.fit(X_train, y_train, nb_epoch=3, batch_size=16, verbose=2, show_accuracy=True)
+'''
+Train on 37800 samples, validate on 4200 samples
+Epoch 0
+loss: 0.0190 - acc.: 0.8750
+Epoch 1
+loss: 0.0146 - acc.: 0.8750
+Epoch 2
+loss: 0.0049 - acc.: 1.0000
+'''
+
+'''
+Demonstration of the validation_split argument
+'''
+model.fit(X_train, y_train, nb_epoch=3, batch_size=16,
+          validation_split=0.1, show_accuracy=True, verbose=1)
+'''
+Train on 37800 samples, validate on 4200 samples
+Epoch 0
+37800/37800 [==============================] - 7s - loss: 0.0385 - acc.: 0.7258 - val. loss: 0.0160 - val. acc.: 0.9136
+Epoch 1
+37800/37800 [==============================] - 8s - loss: 0.0140 - acc.: 0.9265 - val. loss: 0.0109 - val. acc.: 0.9383
+Epoch 2
+10960/37800 [=======>......................] - ETA: 4s - loss: 0.0109 - acc.: 0.9420
+'''
+```
+
+# Using the Graph model
+
+```python
+# graph model with one input and two outputs
+graph = Graph()
+graph.add_input(name='input', input_shape=(32,))
+graph.add_node(Dense(16), name='dense1', input='input')
+graph.add_node(Dense(4), name='dense2', input='input')
+graph.add_node(Dense(4), name='dense3', input='dense1')
+graph.add_output(name='output1', input='dense2')
+graph.add_output(name='output2', input='dense3')
+
+graph.compile(optimizer='rmsprop', loss={'output1':'mse', 'output2':'mse'})
+history = graph.fit({'input':X_train, 'output1':y_train, 'output2':y2_train}, nb_epoch=10)
+
+```
+
+```python
+# graph model with two inputs and one output
+graph = Graph()
+graph.add_input(name='input1', input_shape=(32,))
+graph.add_input(name='input2', input_shape=(32,))
+graph.add_node(Dense(16), name='dense1', input='input1')
+graph.add_node(Dense(4), name='dense2', input='input2')
+graph.add_node(Dense(4), name='dense3', input='dense1')
+graph.add_output(name='output', inputs=['dense2', 'dense3'], merge_mode='sum')
+graph.compile(optimizer='rmsprop', loss={'output':'mse'})
+
+history = graph.fit({'input1':X_train, 'input2':X2_train, 'output':y_train}, nb_epoch=10)
+predictions = graph.predict({'input1':X_test, 'input2':X2_test}) # {'output':...}
+
+```
+
+----
+
+# Model API documentation
+
+
+
+{{autogenerated}}

docs/templates/objectives.md

@@ -7,10 +7,12 @@ An objective function (or loss function, or optimization score function) is one
 model.compile(loss='mean_squared_error', optimizer='sgd')
 ```
 
-You can either pass the name of an existing objective, or pass a Theano symbolic function that returns a scalar and takes the following two arguments:
+You can either pass the name of an existing objective, or pass a Theano/TensorFlow symbolic function that returns a scalar for each data-point and takes the following two arguments:
 
-- __y_true__: True labels. Theano tensor.
-- __y_pred__: Predictions. Theano tensor of the same shape as y_true.
+- __y_true__: True labels. Theano/TensorFlow tensor.
+- __y_pred__: Predictions. Theano/TensorFlow tensor of the same shape as y_true.
+
+The actual optimized objective is the mean of the output array across all datapoints.
 
 For a few examples of such functions, check out the [objectives source](https://github.com/fchollet/keras/blob/master/keras/objectives.py).
 
@@ -18,7 +20,11 @@ For a few examples of such functions, check out the [objectives source](https://
 
 - __mean_squared_error__ / __mse__
 - __mean_absolute_error__ / __mae__
+- __mean_absolute_percentage_error__ / __mape__
+- __mean_squared_logarithmic_error__ / __msle__
 - __squared_hinge__
 - __hinge__
 - __binary_crossentropy__: Also known as logloss. 
-- __categorical_crossentropy__: Also known as multiclass logloss. __Note__: using this objective requires that your labels are binary arrays of shape `(nb_samples, nb_classes)`.
+- __categorical_crossentropy__: Also known as multiclass logloss. __Note__: using this objective requires that your labels are binary arrays of shape `(nb_samples, nb_classes)`.
+- __poisson__: mean of `(predictions - targets * log(predictions))`
+- __cosine_proximity__: the opposite (negative) of the mean cosine proximity between predictions and targets.

docs/templates/optimizers.md

@@ -0,0 +1,25 @@
+
+## Usage of optimizers
+
+An optimizer is one of the two arguments required for compiling a Keras model:
+
+```python
+model = Sequential()
+model.add(Dense(64, init='uniform', input_dim=10))
+model.add(Activation('tanh'))
+model.add(Activation('softmax'))
+
+sgd = SGD(lr=0.1, decay=1e-6, momentum=0.9, nesterov=True)
+model.compile(loss='mean_squared_error', optimizer=sgd)
+```
+
+You can either instantiate an optimizer before passing it to `model.compile()` , as in the above example, or you can call it by its name. In the latter case, the default parameters for the optimizer will be used.
+
+```python
+# pass optimizer by name: default parameters will be used
+model.compile(loss='mean_squared_error', optimizer='sgd')
+```
+
+---
+
+{{autogenerated}}

docs/templates/preprocessing/image.md

@@ -10,11 +10,12 @@ keras.preprocessing.image.ImageDataGenerator(featurewise_center=True,
     rotation_range=0.,
     width_shift_range=0.,
     height_shift_range=0.,
+    shear_range=0.,
     horizontal_flip=False,
     vertical_flip=False)
 ```
 
-Generate batches of tensor image data with real-time data augmentation.
+Generate batches of tensor image data with real-time data augmentation. The data will be looped over (in batches) indefinitely.
 
 - __Arguments__:
     - __featurewise_center__: Boolean. Set input mean to 0 over the dataset.
@@ -25,24 +26,25 @@ Generate batches of tensor image data with real-time data augmentation.
     - __rotation_range__: Int. Degree range for random rotations.
     - __width_shift_range__: Float (fraction of total width). Range for random horizontal shifts.
     - __height_shift_range__: Float (fraction of total height). Range for random vertical shifts.
+    - __shear_range__: Float. Shear Intensity (Shear angle in counter-clockwise direction as radians)
     - __horizontal_flip__: Boolean. Randomly flip inputs horizontally.
     - __vertical_flip__: Boolean. Randomly flip inputs vertically.
 
 - __Methods__:
     - __fit(X)__: Required if featurewise_center or featurewise_std_normalization or zca_whitening. Compute necessary quantities on some sample data.
-        - __Arguments__: 
+        - __Arguments__:
             - __X__: sample data.
             - __augment__: Boolean (default: False). Whether to fit on randomly augmented samples.
             - __rounds__: int (default: 1). If augment, how many augmentation passes over the data to use.
     - __flow(X, y)__:
-        - __Arguments__: 
+        - __Arguments__:
             - __X__: data.
             - __y__: labels.
             - __batch_size__: int (default: 32).
             - __shuffle__: boolean (defaut: False).
-            - __save_to_dir__: None or str. This allows you to optimally specify a directory to which to save the augmented pictures being generated (useful for visualizing what you are doing). 
+            - __save_to_dir__: None or str. This allows you to optimally specify a directory to which to save the augmented pictures being generated (useful for visualizing what you are doing).
             - __save_prefix__: str. Prefix to use for filenames of saved pictures.
-            - __save_format__: one of "png", jpeg". 
+            - __save_format__: one of "png", jpeg".
 
 - __Example__:
 ```python
@@ -58,13 +60,23 @@ datagen = ImageDataGenerator(
     height_shift_range=0.2,
     horizontal_flip=True)
 
-# compute quantities required for featurewise normalization 
+# compute quantities required for featurewise normalization
 # (std, mean, and principal components if ZCA whitening is applied)
 datagen.fit(X_train)
 
+# fits the model on batches with real-time data augmentation:
+model.fit_generator(datagen.flow(X_train, Y_train, batch_size=32),
+                    samples_per_epoch=len(X_train), nb_epoch=nb_epoch)
+
+# here's a more "manual" example
 for e in range(nb_epoch):
     print 'Epoch', e
-    # batch train with realtime data augmentation
-    for X_batch, Y_batch in datagen.flow(X_train, Y_train):
+    batches = 0
+    for X_batch, Y_batch in datagen.flow(X_train, Y_train, batch_size=32):
         loss = model.train(X_batch, Y_batch)
-```
+        batches += 1
+        if batches >= len(X_train) / 32:
+            # we need to break the loop by hand because
+            # the generator loops indefinitely
+            break
+```

docs/templates/preprocessing/sequence.md

@@ -12,6 +12,9 @@ Transform a list of `nb_samples sequences` (lists of scalars) into a 2D numpy ar
     - __sequences__: List of lists of int or float.
     - __maxlen__: None or int. Maximum sequence length, longer sequences are truncated and shorter sequences are padded with zeros at the end.
     - __dtype__: datatype of the numpy array returned.
+    - __padding__: 'pre' or 'post', pad either before or after each sequence.
+    - __truncating__: 'pre' or 'post', remove values from sequences larger than maxlen either in the beginning or in the end of the sequence
+    - __value__: float, value to pad the sequences to the desired value.
 
 ---
 

docs/templates/regularizers.md

@@ -0,0 +1,40 @@
+## Usage of regularizers
+
+Regularizers allow to apply penalties on layer parameters or layer activity during optimization. These penalties are incorporated in the loss function that the network optimizes.
+
+The penalties are applied on a per-layer basis. The exact API will depend on the layer, but the layers `Dense`, `TimeDistributedDense`, `MaxoutDense`, `Convolution1D` and `Convolution2D` have a unified API.
+
+These layers expose 3 keyword arguments:
+
+- `W_regularizer`: instance of `keras.regularizers.WeightRegularizer`
+- `b_regularizer`: instance of `keras.regularizers.WeightRegularizer`
+- `activity_regularizer`: instance of `keras.regularizers.ActivityRegularizer`
+
+
+## Example
+
+```python
+from keras.regularizers import l2, activity_l2
+model.add(Dense(64, input_dim=64, W_regularizer=l2(0.01), activity_regularizer=activity_l2(0.01)))
+```
+
+## Available penalties
+
+```python
+keras.regularizers.WeightRegularizer(l1=0., l2=0.)
+```
+
+```python
+keras.regularizers.ActivityRegularizer(l1=0., l2=0.)
+```
+
+## Shortcuts
+
+These are shortcut functions available in `keras.regularizers`.
+
+- __l1__(l=0.01): L1 weight regularization penalty, also known as LASSO
+- __l2__(l=0.01): L2 weight regularization penalty, also known as weight decay, or Ridge
+- __l1l2__(l1=0.01, l2=0.01): L1-L2 weight regularization penalty, also known as ElasticNet
+- __activity_l1__(l=0.01): L1 activity regularization
+- __activity_l2__(l=0.01): L2 activity regularization
+- __activity_l1l2__(l1=0.01, l2=0.01): L1+L2 activity regularization

docs/templates/visualization.md

@@ -0,0 +1,25 @@
+
+## Model visualization
+
+The `keras.utils.visualize_util` module provides utility functions to plot
+a Keras model (using graphviz).
+
+This will plot a graph of the model and save it to a file:
+```python
+from keras.utils.visualize_util import plot
+plot(model, to_file='model.png')
+```
+
+`plot` takes two optional arguments:
+
+- `recursive` (defaults to True) controls whether we recursively explore container layers.
+- `show_shape` (defaults to False) controls whether output shapes are shown in the graph.
+
+You can also directly obtain the `pydot.Graph` object and render it yourself,
+for example to show it in an ipython notebook :
+```python
+from IPython.display import SVG
+from keras.utils.visualize_util import to_graph
+
+SVG(to_graph(model).create(prog='dot', format='svg'))
+```

examples/addition_rnn.py

@@ -0,0 +1,166 @@
+# -*- coding: utf-8 -*-
+'''An implementation of sequence to sequence learning for performing addition
+Input: "535+61"
+Output: "596"
+Padding is handled by using a repeated sentinel character (space)
+
+Input may optionally be inverted, shown to increase performance in many tasks in:
+"Learning to Execute"
+http://arxiv.org/abs/1410.4615
+and
+"Sequence to Sequence Learning with Neural Networks"
+http://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks.pdf
+Theoretically it introduces shorter term dependencies between source and target.
+
+Two digits inverted:
++ One layer LSTM (128 HN), 5k training examples = 99% train/test accuracy in 55 epochs
+
+Three digits inverted:
++ One layer LSTM (128 HN), 50k training examples = 99% train/test accuracy in 100 epochs
+
+Four digits inverted:
++ One layer LSTM (128 HN), 400k training examples = 99% train/test accuracy in 20 epochs
+
+Five digits inverted:
++ One layer LSTM (128 HN), 550k training examples = 99% train/test accuracy in 30 epochs
+
+'''
+
+from __future__ import print_function
+from keras.models import Sequential, slice_X
+from keras.layers.core import Activation, TimeDistributedDense, RepeatVector
+from keras.layers import recurrent
+import numpy as np
+from six.moves import range
+
+
+class CharacterTable(object):
+    '''
+    Given a set of characters:
+    + Encode them to a one hot integer representation
+    + Decode the one hot integer representation to their character output
+    + Decode a vector of probabilties to their character output
+    '''
+    def __init__(self, chars, maxlen):
+        self.chars = sorted(set(chars))
+        self.char_indices = dict((c, i) for i, c in enumerate(self.chars))
+        self.indices_char = dict((i, c) for i, c in enumerate(self.chars))
+        self.maxlen = maxlen
+
+    def encode(self, C, maxlen=None):
+        maxlen = maxlen if maxlen else self.maxlen
+        X = np.zeros((maxlen, len(self.chars)))
+        for i, c in enumerate(C):
+            X[i, self.char_indices[c]] = 1
+        return X
+
+    def decode(self, X, calc_argmax=True):
+        if calc_argmax:
+            X = X.argmax(axis=-1)
+        return ''.join(self.indices_char[x] for x in X)
+
+
+class colors:
+    ok = '\033[92m'
+    fail = '\033[91m'
+    close = '\033[0m'
+
+# Parameters for the model and dataset
+TRAINING_SIZE = 50000
+DIGITS = 3
+INVERT = True
+# Try replacing GRU, or SimpleRNN
+RNN = recurrent.LSTM
+HIDDEN_SIZE = 128
+BATCH_SIZE = 128
+LAYERS = 1
+MAXLEN = DIGITS + 1 + DIGITS
+
+chars = '0123456789+ '
+ctable = CharacterTable(chars, MAXLEN)
+
+questions = []
+expected = []
+seen = set()
+print('Generating data...')
+while len(questions) < TRAINING_SIZE:
+    f = lambda: int(''.join(np.random.choice(list('0123456789')) for i in range(np.random.randint(1, DIGITS + 1))))
+    a, b = f(), f()
+    # Skip any addition questions we've already seen
+    # Also skip any such that X+Y == Y+X (hence the sorting)
+    key = tuple(sorted((a, b)))
+    if key in seen:
+        continue
+    seen.add(key)
+    # Pad the data with spaces such that it is always MAXLEN
+    q = '{}+{}'.format(a, b)
+    query = q + ' ' * (MAXLEN - len(q))
+    ans = str(a + b)
+    # Answers can be of maximum size DIGITS + 1
+    ans += ' ' * (DIGITS + 1 - len(ans))
+    if INVERT:
+        query = query[::-1]
+    questions.append(query)
+    expected.append(ans)
+print('Total addition questions:', len(questions))
+
+print('Vectorization...')
+X = np.zeros((len(questions), MAXLEN, len(chars)), dtype=np.bool)
+y = np.zeros((len(questions), DIGITS + 1, len(chars)), dtype=np.bool)
+for i, sentence in enumerate(questions):
+    X[i] = ctable.encode(sentence, maxlen=MAXLEN)
+for i, sentence in enumerate(expected):
+    y[i] = ctable.encode(sentence, maxlen=DIGITS + 1)
+
+# Shuffle (X, y) in unison as the later parts of X will almost all be larger digits
+indices = np.arange(len(y))
+np.random.shuffle(indices)
+X = X[indices]
+y = y[indices]
+
+# Explicitly set apart 10% for validation data that we never train over
+split_at = len(X) - len(X) / 10
+(X_train, X_val) = (slice_X(X, 0, split_at), slice_X(X, split_at))
+(y_train, y_val) = (y[:split_at], y[split_at:])
+
+print(X_train.shape)
+print(y_train.shape)
+
+print('Build model...')
+model = Sequential()
+# "Encode" the input sequence using an RNN, producing an output of HIDDEN_SIZE
+# note: in a situation where your input sequences have a variable length,
+# use input_shape=(None, nb_feature).
+model.add(RNN(HIDDEN_SIZE, input_shape=(MAXLEN, len(chars))))
+# For the decoder's input, we repeat the encoded input for each time step
+model.add(RepeatVector(DIGITS + 1))
+# The decoder RNN could be multiple layers stacked or a single layer
+for _ in range(LAYERS):
+    model.add(RNN(HIDDEN_SIZE, return_sequences=True))
+
+# For each of step of the output sequence, decide which character should be chosen
+model.add(TimeDistributedDense(len(chars)))
+model.add(Activation('softmax'))
+
+model.compile(loss='categorical_crossentropy', optimizer='adam')
+
+# Train the model each generation and show predictions against the validation dataset
+for iteration in range(1, 200):
+    print()
+    print('-' * 50)
+    print('Iteration', iteration)
+    model.fit(X_train, y_train, batch_size=BATCH_SIZE, nb_epoch=1,
+              validation_data=(X_val, y_val), show_accuracy=True)
+    ###
+    # Select 10 samples from the validation set at random so we can visualize errors
+    for i in range(10):
+        ind = np.random.randint(0, len(X_val))
+        rowX, rowy = X_val[np.array([ind])], y_val[np.array([ind])]
+        preds = model.predict_classes(rowX, verbose=0)
+        q = ctable.decode(rowX[0])
+        correct = ctable.decode(rowy[0])
+        guess = ctable.decode(preds[0], calc_argmax=False)
+        print('Q', q[::-1] if INVERT else q)
+        print('T', correct)
+        print(colors.ok + '☑' + colors.close if correct == guess else colors.fail + '☒' + colors.close, guess)
+        print('---')

examples/antirectifier.py

@@ -0,0 +1,106 @@
+'''The example demonstrates how to write custom layers for Keras.
+
+We build a custom activation layer called 'Antirectifier',
+which modifies the shape of the tensor that passes through it.
+We need to specify two methods: `output_shape` and `get_output`.
+
+Note that the same result can also be achieved via a Lambda layer.
+
+Because our custom layer is written with primitives from the Keras
+backend (`K`), our code can run both on TensorFlow and Theano.
+'''
+
+from __future__ import print_function
+import numpy as np
+from keras.models import Sequential
+from keras.layers.core import Dense, Dropout, Layer, Activation
+from keras.datasets import mnist
+from keras import backend as K
+from keras.utils import np_utils
+
+
+class Antirectifier(Layer):
+    '''This is the combination of a sample-wise
+    L2 normalization with the concatenation of the
+    positive part of the input with the negative part
+    of the input. The result is a tensor of samples that are
+    twice as large as the input samples.
+
+    It can be used in place of a ReLU.
+
+    # Input shape
+        2D tensor of shape (samples, n)
+
+    # Output shape
+        2D tensor of shape (samples, 2*n)
+
+    # Theoretical justification
+        When applying ReLU, assuming that the distribution
+        of the previous output is approximately centered around 0.,
+        you are discarding half of your input. This is inefficient.
+
+        Antirectifier allows to return all-positive outputs like ReLU,
+        without discarding any data.
+
+        Tests on MNIST show that Antirectifier allows to train networks
+        with twice less parameters yet with comparable
+        classification accuracy as an equivalent ReLU-based network.
+    '''
+    @property
+    def output_shape(self):
+        shape = list(self.input_shape)
+        assert len(shape) == 2  # only valid for 2D tensors
+        shape[-1] *= 2
+        return tuple(shape)
+
+    def get_output(self, train):
+        x = self.get_input(train)
+        x -= K.mean(x, axis=1, keepdims=True)
+        x = K.l2_normalize(x, axis=1)
+        pos = K.relu(x)
+        neg = K.relu(-x)
+        return K.concatenate([pos, neg], axis=1)
+
+# global parameters
+batch_size = 128
+nb_classes = 10
+nb_epoch = 40
+
+# the data, shuffled and split between train and test sets
+(X_train, y_train), (X_test, y_test) = mnist.load_data()
+
+X_train = X_train.reshape(60000, 784)
+X_test = X_test.reshape(10000, 784)
+X_train = X_train.astype('float32')
+X_test = X_test.astype('float32')
+X_train /= 255
+X_test /= 255
+print(X_train.shape[0], 'train samples')
+print(X_test.shape[0], 'test samples')
+
+# convert class vectors to binary class matrices
+Y_train = np_utils.to_categorical(y_train, nb_classes)
+Y_test = np_utils.to_categorical(y_test, nb_classes)
+
+# build the model
+model = Sequential()
+model.add(Dense(256, input_shape=(784,)))
+model.add(Antirectifier())
+model.add(Dropout(0.1))
+model.add(Dense(256))
+model.add(Antirectifier())
+model.add(Dropout(0.1))
+model.add(Dense(10))
+model.add(Activation('softmax'))
+
+# compile the model
+model.compile(loss='categorical_crossentropy', optimizer='rmsprop')
+
+# train the model
+model.fit(X_train, Y_train,
+          batch_size=batch_size, nb_epoch=nb_epoch,
+          show_accuracy=True, verbose=1,
+          validation_data=(X_test, Y_test))
+
+# next, compare with an equivalent network
+# with2x bigger Dense layers and ReLU

examples/babi_memnn.py

@@ -0,0 +1,204 @@
+'''Train a memory network on the bAbI dataset.
+
+References:
+- Jason Weston, Antoine Bordes, Sumit Chopra, Tomas Mikolov, Alexander M. Rush,
+  "Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks",
+  http://arxiv.org/abs/1502.05698
+
+- Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, Rob Fergus,
+  "End-To-End Memory Networks",
+  http://arxiv.org/abs/1503.08895
+
+Reaches 98.6% accuracy on task 'single_supporting_fact_10k' after 120 epochs.
+Time per epoch: 3s on CPU (core i7).
+'''
+
+from __future__ import print_function
+from keras.models import Sequential
+from keras.layers.embeddings import Embedding
+from keras.layers.core import Activation, Dense, Merge, Permute, Dropout
+from keras.layers.recurrent import LSTM
+from keras.utils.data_utils import get_file
+from keras.preprocessing.sequence import pad_sequences
+from functools import reduce
+import tarfile
+import numpy as np
+import re
+
+
+def tokenize(sent):
+    '''Return the tokens of a sentence including punctuation.
+
+    >>> tokenize('Bob dropped the apple. Where is the apple?')
+    ['Bob', 'dropped', 'the', 'apple', '.', 'Where', 'is', 'the', 'apple', '?']
+    '''
+    return [x.strip() for x in re.split('(\W+)?', sent) if x.strip()]
+
+
+def parse_stories(lines, only_supporting=False):
+    '''Parse stories provided in the bAbi tasks format
+
+    If only_supporting is true, only the sentences that support the answer are kept.
+    '''
+    data = []
+    story = []
+    for line in lines:
+        line = line.decode('utf-8').strip()
+        nid, line = line.split(' ', 1)
+        nid = int(nid)
+        if nid == 1:
+            story = []
+        if '\t' in line:
+            q, a, supporting = line.split('\t')
+            q = tokenize(q)
+            substory = None
+            if only_supporting:
+                # Only select the related substory
+                supporting = map(int, supporting.split())
+                substory = [story[i - 1] for i in supporting]
+            else:
+                # Provide all the substories
+                substory = [x for x in story if x]
+            data.append((substory, q, a))
+            story.append('')
+        else:
+            sent = tokenize(line)
+            story.append(sent)
+    return data
+
+
+def get_stories(f, only_supporting=False, max_length=None):
+    '''Given a file name, read the file, retrieve the stories, and then convert the sentences into a single story.
+
+    If max_length is supplied, any stories longer than max_length tokens will be discarded.
+    '''
+    data = parse_stories(f.readlines(), only_supporting=only_supporting)
+    flatten = lambda data: reduce(lambda x, y: x + y, data)
+    data = [(flatten(story), q, answer) for story, q, answer in data if not max_length or len(flatten(story)) < max_length]
+    return data
+
+
+def vectorize_stories(data, word_idx, story_maxlen, query_maxlen):
+    X = []
+    Xq = []
+    Y = []
+    for story, query, answer in data:
+        x = [word_idx[w] for w in story]
+        xq = [word_idx[w] for w in query]
+        y = np.zeros(len(word_idx) + 1)  # let's not forget that index 0 is reserved
+        y[word_idx[answer]] = 1
+        X.append(x)
+        Xq.append(xq)
+        Y.append(y)
+    return (pad_sequences(X, maxlen=story_maxlen),
+            pad_sequences(Xq, maxlen=query_maxlen), np.array(Y))
+
+
+path = get_file('babi-tasks-v1-2.tar.gz',
+                origin='http://www.thespermwhale.com/jaseweston/babi/tasks_1-20_v1-2.tar.gz')
+tar = tarfile.open(path)
+
+challenges = {
+    # QA1 with 10,000 samples
+    'single_supporting_fact_10k': 'tasks_1-20_v1-2/en-10k/qa1_single-supporting-fact_{}.txt',
+    # QA2 with 10,000 samples
+    'two_supporting_facts_10k': 'tasks_1-20_v1-2/en-10k/qa2_two-supporting-facts_{}.txt',
+}
+challenge_type = 'single_supporting_fact_10k'
+challenge = challenges[challenge_type]
+
+print('Extracting stories for the challenge:', challenge_type)
+train_stories = get_stories(tar.extractfile(challenge.format('train')))
+test_stories = get_stories(tar.extractfile(challenge.format('test')))
+
+vocab = sorted(reduce(lambda x, y: x | y, (set(story + q + [answer]) for story, q, answer in train_stories + test_stories)))
+# Reserve 0 for masking via pad_sequences
+vocab_size = len(vocab) + 1
+story_maxlen = max(map(len, (x for x, _, _ in train_stories + test_stories)))
+query_maxlen = max(map(len, (x for _, x, _ in train_stories + test_stories)))
+
+print('-')
+print('Vocab size:', vocab_size, 'unique words')
+print('Story max length:', story_maxlen, 'words')
+print('Query max length:', query_maxlen, 'words')
+print('Number of training stories:', len(train_stories))
+print('Number of test stories:', len(test_stories))
+print('-')
+print('Here\'s what a "story" tuple looks like (input, query, answer):')
+print(train_stories[0])
+print('-')
+print('Vectorizing the word sequences...')
+
+word_idx = dict((c, i + 1) for i, c in enumerate(vocab))
+inputs_train, queries_train, answers_train = vectorize_stories(train_stories, word_idx, story_maxlen, query_maxlen)
+inputs_test, queries_test, answers_test = vectorize_stories(test_stories, word_idx, story_maxlen, query_maxlen)
+
+print('-')
+print('inputs: integer tensor of shape (samples, max_length)')
+print('inputs_train shape:', inputs_train.shape)
+print('inputs_test shape:', inputs_test.shape)
+print('-')
+print('queries: integer tensor of shape (samples, max_length)')
+print('queries_train shape:', queries_train.shape)
+print('queries_test shape:', queries_test.shape)
+print('-')
+print('answers: binary (1 or 0) tensor of shape (samples, vocab_size)')
+print('answers_train shape:', answers_train.shape)
+print('answers_test shape:', answers_test.shape)
+print('-')
+print('Compiling...')
+
+# embed the input sequence into a sequence of vectors
+input_encoder_m = Sequential()
+input_encoder_m.add(Embedding(input_dim=vocab_size,
+                              output_dim=64,
+                              input_length=story_maxlen))
+input_encoder_m.add(Dropout(0.3))
+# output: (samples, story_maxlen, embedding_dim)
+# embed the question into a sequence of vectors
+question_encoder = Sequential()
+question_encoder.add(Embedding(input_dim=vocab_size,
+                               output_dim=64,
+                               input_length=query_maxlen))
+question_encoder.add(Dropout(0.3))
+# output: (samples, query_maxlen, embedding_dim)
+# compute a 'match' between input sequence elements (which are vectors)
+# and the question vector sequence
+match = Sequential()
+match.add(Merge([input_encoder_m, question_encoder],
+                mode='dot',
+                dot_axes=[(2,), (2,)]))
+# output: (samples, story_maxlen, query_maxlen)
+# embed the input into a single vector with size = story_maxlen:
+input_encoder_c = Sequential()
+input_encoder_c.add(Embedding(input_dim=vocab_size,
+                              output_dim=query_maxlen,
+                              input_length=story_maxlen))
+input_encoder_c.add(Dropout(0.3))
+# output: (samples, story_maxlen, query_maxlen)
+# sum the match vector with the input vector:
+response = Sequential()
+response.add(Merge([match, input_encoder_c], mode='sum'))
+# output: (samples, story_maxlen, query_maxlen)
+response.add(Permute((2, 1)))  # output: (samples, query_maxlen, story_maxlen)
+
+# concatenate the match vector with the question vector,
+# and do logistic regression on top
+answer = Sequential()
+answer.add(Merge([response, question_encoder], mode='concat', concat_axis=-1))
+# the original paper uses a matrix multiplication for this reduction step.
+# we choose to use a RNN instead.
+answer.add(LSTM(32))
+# one regularization layer -- more would probably be needed.
+answer.add(Dropout(0.3))
+answer.add(Dense(vocab_size))
+# we output a probability distribution over the vocabulary
+answer.add(Activation('softmax'))
+
+answer.compile(optimizer='rmsprop', loss='categorical_crossentropy')
+# Note: you could use a Graph model to avoid repeat the input twice
+answer.fit([inputs_train, queries_train, inputs_train], answers_train,
+           batch_size=32,
+           nb_epoch=120,
+           show_accuracy=True,
+           validation_data=([inputs_test, queries_test, inputs_test], answers_test))

examples/babi_rnn.py

@@ -0,0 +1,201 @@
+'''Trains two recurrent neural networks based upon a story and a question.
+The resulting merged vector is then queried to answer a range of bAbI tasks.
+
+The results are comparable to those for an LSTM model provided in Weston et al.:
+"Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks"
+http://arxiv.org/abs/1502.05698
+
+Task Number                  | FB LSTM Baseline | Keras QA
+---                          | ---              | ---
+QA1 - Single Supporting Fact | 50               | 100.0
+QA2 - Two Supporting Facts   | 20               | 50.0
+QA3 - Three Supporting Facts | 20               | 20.5
+QA4 - Two Arg. Relations     | 61               | 62.9
+QA5 - Three Arg. Relations   | 70               | 61.9
+QA6 - Yes/No Questions       | 48               | 50.7
+QA7 - Counting               | 49               | 78.9
+QA8 - Lists/Sets             | 45               | 77.2
+QA9 - Simple Negation        | 64               | 64.0
+QA10 - Indefinite Knowledge  | 44               | 47.7
+QA11 - Basic Coreference     | 72               | 74.9
+QA12 - Conjunction           | 74               | 76.4
+QA13 - Compound Coreference  | 94               | 94.4
+QA14 - Time Reasoning        | 27               | 34.8
+QA15 - Basic Deduction       | 21               | 32.4
+QA16 - Basic Induction       | 23               | 50.6
+QA17 - Positional Reasoning  | 51               | 49.1
+QA18 - Size Reasoning        | 52               | 90.8
+QA19 - Path Finding          | 8                | 9.0
+QA20 - Agent's Motivations   | 91               | 90.7
+
+For the resources related to the bAbI project, refer to:
+https://research.facebook.com/researchers/1543934539189348
+
+Notes:
+
+- With default word, sentence, and query vector sizes, the GRU model achieves:
+  - 100% test accuracy on QA1 in 20 epochs (2 seconds per epoch on CPU)
+  - 50% test accuracy on QA2 in 20 epochs (16 seconds per epoch on CPU)
+In comparison, the Facebook paper achieves 50% and 20% for the LSTM baseline.
+
+- The task does not traditionally parse the question separately. This likely
+improves accuracy and is a good example of merging two RNNs.
+
+- The word vector embeddings are not shared between the story and question RNNs.
+
+- See how the accuracy changes given 10,000 training samples (en-10k) instead
+of only 1000. 1000 was used in order to be comparable to the original paper.
+
+- Experiment with GRU, LSTM, and JZS1-3 as they give subtly different results.
+
+- The length and noise (i.e. 'useless' story components) impact the ability for
+LSTMs / GRUs to provide the correct answer. Given only the supporting facts,
+these RNNs can achieve 100% accuracy on many tasks. Memory networks and neural
+networks that use attentional processes can efficiently search through this
+noise to find the relevant statements, improving performance substantially.
+This becomes especially obvious on QA2 and QA3, both far longer than QA1.
+'''
+
+from __future__ import print_function
+from functools import reduce
+import re
+import tarfile
+
+import numpy as np
+np.random.seed(1337)  # for reproducibility
+
+from keras.utils.data_utils import get_file
+from keras.layers.embeddings import Embedding
+from keras.layers.core import Dense, Merge, Dropout, RepeatVector
+from keras.layers import recurrent
+from keras.models import Sequential
+from keras.preprocessing.sequence import pad_sequences
+
+
+def tokenize(sent):
+    '''Return the tokens of a sentence including punctuation.
+
+    >>> tokenize('Bob dropped the apple. Where is the apple?')
+    ['Bob', 'dropped', 'the', 'apple', '.', 'Where', 'is', 'the', 'apple', '?']
+    '''
+    return [x.strip() for x in re.split('(\W+)?', sent) if x.strip()]
+
+
+def parse_stories(lines, only_supporting=False):
+    '''Parse stories provided in the bAbi tasks format
+
+    If only_supporting is true, only the sentences that support the answer are kept.
+    '''
+    data = []
+    story = []
+    for line in lines:
+        line = line.decode('utf-8').strip()
+        nid, line = line.split(' ', 1)
+        nid = int(nid)
+        if nid == 1:
+            story = []
+        if '\t' in line:
+            q, a, supporting = line.split('\t')
+            q = tokenize(q)
+            substory = None
+            if only_supporting:
+                # Only select the related substory
+                supporting = map(int, supporting.split())
+                substory = [story[i - 1] for i in supporting]
+            else:
+                # Provide all the substories
+                substory = [x for x in story if x]
+            data.append((substory, q, a))
+            story.append('')
+        else:
+            sent = tokenize(line)
+            story.append(sent)
+    return data
+
+
+def get_stories(f, only_supporting=False, max_length=None):
+    '''Given a file name, read the file, retrieve the stories, and then convert the sentences into a single story.
+
+    If max_length is supplied, any stories longer than max_length tokens will be discarded.
+    '''
+    data = parse_stories(f.readlines(), only_supporting=only_supporting)
+    flatten = lambda data: reduce(lambda x, y: x + y, data)
+    data = [(flatten(story), q, answer) for story, q, answer in data if not max_length or len(flatten(story)) < max_length]
+    return data
+
+
+def vectorize_stories(data, word_idx, story_maxlen, query_maxlen):
+    X = []
+    Xq = []
+    Y = []
+    for story, query, answer in data:
+        x = [word_idx[w] for w in story]
+        xq = [word_idx[w] for w in query]
+        y = np.zeros(len(word_idx) + 1)  # let's not forget that index 0 is reserved
+        y[word_idx[answer]] = 1
+        X.append(x)
+        Xq.append(xq)
+        Y.append(y)
+    return pad_sequences(X, maxlen=story_maxlen), pad_sequences(Xq, maxlen=query_maxlen), np.array(Y)
+
+RNN = recurrent.LSTM
+EMBED_HIDDEN_SIZE = 50
+SENT_HIDDEN_SIZE = 100
+QUERY_HIDDEN_SIZE = 100
+BATCH_SIZE = 32
+EPOCHS = 40
+print('RNN / Embed / Sent / Query = {}, {}, {}, {}'.format(RNN, EMBED_HIDDEN_SIZE, SENT_HIDDEN_SIZE, QUERY_HIDDEN_SIZE))
+
+path = get_file('babi-tasks-v1-2.tar.gz', origin='http://www.thespermwhale.com/jaseweston/babi/tasks_1-20_v1-2.tar.gz')
+tar = tarfile.open(path)
+# Default QA1 with 1000 samples
+# challenge = 'tasks_1-20_v1-2/en/qa1_single-supporting-fact_{}.txt'
+# QA1 with 10,000 samples
+# challenge = 'tasks_1-20_v1-2/en-10k/qa1_single-supporting-fact_{}.txt'
+# QA2 with 1000 samples
+challenge = 'tasks_1-20_v1-2/en/qa2_two-supporting-facts_{}.txt'
+# QA2 with 10,000 samples
+# challenge = 'tasks_1-20_v1-2/en-10k/qa2_two-supporting-facts_{}.txt'
+train = get_stories(tar.extractfile(challenge.format('train')))
+test = get_stories(tar.extractfile(challenge.format('test')))
+
+vocab = sorted(reduce(lambda x, y: x | y, (set(story + q + [answer]) for story, q, answer in train + test)))
+# Reserve 0 for masking via pad_sequences
+vocab_size = len(vocab) + 1
+word_idx = dict((c, i + 1) for i, c in enumerate(vocab))
+story_maxlen = max(map(len, (x for x, _, _ in train + test)))
+query_maxlen = max(map(len, (x for _, x, _ in train + test)))
+
+X, Xq, Y = vectorize_stories(train, word_idx, story_maxlen, query_maxlen)
+tX, tXq, tY = vectorize_stories(test, word_idx, story_maxlen, query_maxlen)
+
+print('vocab = {}'.format(vocab))
+print('X.shape = {}'.format(X.shape))
+print('Xq.shape = {}'.format(Xq.shape))
+print('Y.shape = {}'.format(Y.shape))
+print('story_maxlen, query_maxlen = {}, {}'.format(story_maxlen, query_maxlen))
+
+print('Build model...')
+
+sentrnn = Sequential()
+sentrnn.add(Embedding(vocab_size, EMBED_HIDDEN_SIZE, input_length=story_maxlen, mask_zero=True))
+sentrnn.add(Dropout(0.3))
+
+qrnn = Sequential()
+qrnn.add(Embedding(vocab_size, EMBED_HIDDEN_SIZE, input_length=query_maxlen))
+qrnn.add(Dropout(0.3))
+qrnn.add(RNN(EMBED_HIDDEN_SIZE, return_sequences=False))
+qrnn.add(RepeatVector(story_maxlen))
+
+model = Sequential()
+model.add(Merge([sentrnn, qrnn], mode='sum'))
+model.add(RNN(EMBED_HIDDEN_SIZE, return_sequences=False))
+model.add(Dropout(0.3))
+model.add(Dense(vocab_size, activation='softmax'))
+
+model.compile(optimizer='adam', loss='categorical_crossentropy', class_mode='categorical')
+
+print('Training')
+model.fit([X, Xq], Y, batch_size=BATCH_SIZE, nb_epoch=EPOCHS, validation_split=0.05, show_accuracy=True)
+loss, acc = model.evaluate([tX, tXq], tY, batch_size=BATCH_SIZE, show_accuracy=True)
+print('Test loss / test accuracy = {:.4f} / {:.4f}'.format(loss, acc))

examples/cifar10_cnn.py

@@ -1,35 +1,38 @@
-from __future__ import absolute_import
+'''Train a simple deep CNN on the CIFAR10 small images dataset.
+
+GPU run command:
+    THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 python cifar10_cnn.py
+
+It gets down to 0.65 test logloss in 25 epochs, and down to 0.55 after 50 epochs.
+(it's still underfitting at that point, though).
+
+Note: the data was pickled with Python 2, and some encoding issues might prevent you
+from loading it in Python 3. You might have to load it in Python 2,
+save it in a different format, load it in Python 3 and repickle it.
+'''
+
 from __future__ import print_function
 from keras.datasets import cifar10
 from keras.preprocessing.image import ImageDataGenerator
 from keras.models import Sequential
 from keras.layers.core import Dense, Dropout, Activation, Flatten
 from keras.layers.convolutional import Convolution2D, MaxPooling2D
-from keras.optimizers import SGD, Adadelta, Adagrad
-from keras.utils import np_utils, generic_utils
-from six.moves import range
-
-'''
-    Train a (fairly simple) deep CNN on the CIFAR10 small images dataset.
-
-    GPU run command:
-        THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 python cifar10_cnn.py
-
-    It gets down to 0.65 test logloss in 25 epochs, and down to 0.55 after 50 epochs.
-    (it's still underfitting at that point, though).
-
-    Note: the data was pickled with Python 2, and some encoding issues might prevent you
-    from loading it in Python 3. You might have to load it in Python 2, 
-    save it in a different format, load it in Python 3 and repickle it.
-'''
+from keras.optimizers import SGD
+from keras.utils import np_utils
 
 batch_size = 32
 nb_classes = 10
 nb_epoch = 200
 data_augmentation = True
 
-# the data, shuffled and split between tran and test sets
+# input image dimensions
+img_rows, img_cols = 32, 32
+# the CIFAR10 images are RGB
+img_channels = 3
+
+# the data, shuffled and split between train and test sets
 (X_train, y_train), (X_test, y_test) = cifar10.load_data()
+print('X_train shape:', X_train.shape)
 print(X_train.shape[0], 'train samples')
 print(X_test.shape[0], 'test samples')
 
@@ -39,85 +42,65 @@ Y_test = np_utils.to_categorical(y_test, nb_classes)
 
 model = Sequential()
 
-model.add(Convolution2D(32, 3, 3, 3, border_mode='full')) 
+model.add(Convolution2D(32, 3, 3, border_mode='same',
+                        input_shape=(img_channels, img_rows, img_cols)))
 model.add(Activation('relu'))
-model.add(Convolution2D(32, 32, 3, 3))
+model.add(Convolution2D(32, 3, 3))
 model.add(Activation('relu'))
-model.add(MaxPooling2D(poolsize=(2, 2)))
+model.add(MaxPooling2D(pool_size=(2, 2)))
 model.add(Dropout(0.25))
 
-model.add(Convolution2D(64, 32, 3, 3, border_mode='full')) 
+model.add(Convolution2D(64, 3, 3, border_mode='same'))
 model.add(Activation('relu'))
-model.add(Convolution2D(64, 64, 3, 3)) 
+model.add(Convolution2D(64, 3, 3))
 model.add(Activation('relu'))
-model.add(MaxPooling2D(poolsize=(2, 2)))
+model.add(MaxPooling2D(pool_size=(2, 2)))
 model.add(Dropout(0.25))
 
 model.add(Flatten())
-model.add(Dense(64*8*8, 512, init='normal'))
+model.add(Dense(512))
 model.add(Activation('relu'))
 model.add(Dropout(0.5))
-
-model.add(Dense(512, nb_classes, init='normal'))
+model.add(Dense(nb_classes))
 model.add(Activation('softmax'))
 
 # let's train the model using SGD + momentum (how original).
 sgd = SGD(lr=0.01, decay=1e-6, momentum=0.9, nesterov=True)
 model.compile(loss='categorical_crossentropy', optimizer=sgd)
 
+X_train = X_train.astype('float32')
+X_test = X_test.astype('float32')
+X_train /= 255
+X_test /= 255
+
 if not data_augmentation:
-    print("Not using data augmentation or normalization")
-
-    X_train = X_train.astype("float32")
-    X_test = X_test.astype("float32")
-    X_train /= 255
-    X_test /= 255
-    model.fit(X_train, Y_train, batch_size=batch_size, nb_epoch=10)
-    score = model.evaluate(X_test, Y_test, batch_size=batch_size)
-    print('Test score:', score)
-
+    print('Not using data augmentation.')
+    model.fit(X_train, Y_train, batch_size=batch_size,
+              nb_epoch=nb_epoch, show_accuracy=True,
+              validation_data=(X_test, Y_test), shuffle=True)
 else:
-    print("Using real time data augmentation")
+    print('Using real-time data augmentation.')
 
     # this will do preprocessing and realtime data augmentation
     datagen = ImageDataGenerator(
-        featurewise_center=True, # set input mean to 0 over the dataset
-        samplewise_center=False, # set each sample mean to 0
-        featurewise_std_normalization=True, # divide inputs by std of the dataset
-        samplewise_std_normalization=False, # divide each input by its std
-        zca_whitening=False, # apply ZCA whitening
-        rotation_range=20, # randomly rotate images in the range (degrees, 0 to 180)
-        width_shift_range=0.2, # randomly shift images horizontally (fraction of total width)
-        height_shift_range=0.2, # randomly shift images vertically (fraction of total height)
-        horizontal_flip=True, # randomly flip images
-        vertical_flip=False) # randomly flip images
+        featurewise_center=False,  # set input mean to 0 over the dataset
+        samplewise_center=False,  # set each sample mean to 0
+        featurewise_std_normalization=False,  # divide inputs by std of the dataset
+        samplewise_std_normalization=False,  # divide each input by its std
+        zca_whitening=False,  # apply ZCA whitening
+        rotation_range=0,  # randomly rotate images in the range (degrees, 0 to 180)
+        width_shift_range=0.1,  # randomly shift images horizontally (fraction of total width)
+        height_shift_range=0.1,  # randomly shift images vertically (fraction of total height)
+        horizontal_flip=True,  # randomly flip images
+        vertical_flip=False)  # randomly flip images
 
-    # compute quantities required for featurewise normalization 
+    # compute quantities required for featurewise normalization
     # (std, mean, and principal components if ZCA whitening is applied)
     datagen.fit(X_train)
 
-    for e in range(nb_epoch):
-        print('-'*40)
-        print('Epoch', e)
-        print('-'*40)
-        print("Training...")
-        # batch train with realtime data augmentation
-        progbar = generic_utils.Progbar(X_train.shape[0])
-        for X_batch, Y_batch in datagen.flow(X_train, Y_train):
-            loss = model.train(X_batch, Y_batch)
-            progbar.add(X_batch.shape[0], values=[("train loss", loss)])
-
-        print("Testing...")
-        # test time!
-        progbar = generic_utils.Progbar(X_test.shape[0])
-        for X_batch, Y_batch in datagen.flow(X_test, Y_test):
-            score = model.test(X_batch, Y_batch)
-            progbar.add(X_batch.shape[0], values=[("test loss", score)])
-
-            
-
-
-
-
-
-
+    # fit the model on the batches generated by datagen.flow()
+    model.fit_generator(datagen.flow(X_train, Y_train, batch_size=batch_size),
+                        samples_per_epoch=X_train.shape[0],
+                        nb_epoch=nb_epoch, show_accuracy=True,
+                        validation_data=(X_test, Y_test),
+                        nb_worker=1)

examples/conv_filter_visualization.py

@@ -0,0 +1,186 @@
+'''Visualization of the filters of VGG16, via gradient ascent in input space.
+
+This script can run on CPU in a few minutes (with the TensorFlow backend).
+
+Results example: http://i.imgur.com/4nj4KjN.jpg
+
+Before running this script, download the weights for the VGG16 model at:
+https://drive.google.com/file/d/0Bz7KyqmuGsilT0J5dmRCM0ROVHc/view?usp=sharing
+(source: https://gist.github.com/baraldilorenzo/07d7802847aaad0a35d3)
+and make sure the variable `weights_path` in this script matches the location of the file.
+'''
+from __future__ import print_function
+from scipy.misc import imsave
+import numpy as np
+import time
+import os
+import h5py
+
+from keras.models import Sequential
+from keras.layers import Convolution2D, ZeroPadding2D, MaxPooling2D
+from keras import backend as K
+
+# dimensions of the generated pictures for each filter.
+img_width = 128
+img_height = 128
+
+# path to the model weights file.
+weights_path = 'vgg16_weights.h5'
+
+# the name of the layer we want to visualize (see model definition below)
+layer_name = 'conv5_1'
+
+# util function to convert a tensor into a valid image
+def deprocess_image(x):
+    # normalize tensor: center on 0., ensure std is 0.1
+    x -= x.mean()
+    x /= (x.std() + 1e-5)
+    x *= 0.1
+
+    # clip to [0, 1]
+    x += 0.5
+    x = np.clip(x, 0, 1)
+
+    # convert to RGB array
+    x *= 255
+    x = x.transpose((1, 2, 0))
+    x = np.clip(x, 0, 255).astype('uint8')
+    return x
+
+# this will contain our generated image
+input_img = K.placeholder((1, 3, img_width, img_height))
+
+# build the VGG16 network with our input_img as input
+first_layer = ZeroPadding2D((1, 1), input_shape=(3, img_width, img_height))
+first_layer.input = input_img
+
+model = Sequential()
+model.add(first_layer)
+model.add(Convolution2D(64, 3, 3, activation='relu', name='conv1_1'))
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(64, 3, 3, activation='relu', name='conv1_2'))
+model.add(MaxPooling2D((2, 2), strides=(2, 2)))
+
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(128, 3, 3, activation='relu', name='conv2_1'))
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(128, 3, 3, activation='relu', name='conv2_2'))
+model.add(MaxPooling2D((2, 2), strides=(2, 2)))
+
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(256, 3, 3, activation='relu', name='conv3_1'))
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(256, 3, 3, activation='relu', name='conv3_2'))
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(256, 3, 3, activation='relu', name='conv3_3'))
+model.add(MaxPooling2D((2, 2), strides=(2, 2)))
+
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(512, 3, 3, activation='relu', name='conv4_1'))
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(512, 3, 3, activation='relu', name='conv4_2'))
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(512, 3, 3, activation='relu', name='conv4_3'))
+model.add(MaxPooling2D((2, 2), strides=(2, 2)))
+
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(512, 3, 3, activation='relu', name='conv5_1'))
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(512, 3, 3, activation='relu', name='conv5_2'))
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(512, 3, 3, activation='relu', name='conv5_3'))
+model.add(MaxPooling2D((2, 2), strides=(2, 2)))
+
+# load the weights of the VGG16 networks
+# (trained on ImageNet, won the ILSVRC competition in 2014)
+# note: when there is a complete match between your model definition
+# and your weight savefile, you can simply call model.load_weights(filename)
+assert os.path.exists(weights_path), 'Model weights not found (see "weights_path" variable in script).'
+f = h5py.File(weights_path)
+for k in range(f.attrs['nb_layers']):
+    if k >= len(model.layers):
+        # we don't look at the last (fully-connected) layers in the savefile
+        break
+    g = f['layer_{}'.format(k)]
+    weights = [g['param_{}'.format(p)] for p in range(g.attrs['nb_params'])]
+    model.layers[k].set_weights(weights)
+f.close()
+print('Model loaded.')
+
+# get the symbolic outputs of each "key" layer (we gave them unique names).
+layer_dict = dict([(layer.name, layer) for layer in model.layers])
+
+
+def normalize(x):
+    # utility function to normalize a tensor by its L2 norm
+    return x / (K.sqrt(K.mean(K.square(x))) + 1e-5)
+
+
+kept_filters = []
+for filter_index in range(0, 200):
+    # we only scan through the first 200 filters,
+    # but there are actually 512 of them
+    print('Processing filter %d' % filter_index)
+    start_time = time.time()
+
+    # we build a loss function that maximizes the activation
+    # of the nth filter of the layer considered
+    layer_output = layer_dict[layer_name].get_output()
+    loss = K.mean(layer_output[:, filter_index, :, :])
+
+    # we compute the gradient of the input picture wrt this loss
+    grads = K.gradients(loss, input_img)[0]
+
+    # normalization trick: we normalize the gradient
+    grads = normalize(grads)
+
+    # this function returns the loss and grads given the input picture
+    iterate = K.function([input_img], [loss, grads])
+
+    # step size for gradient ascent
+    step = 1.
+
+    # we start from a gray image with some random noise
+    input_img_data = np.random.random((1, 3, img_width, img_height)) * 20 + 128.
+
+    # we run gradient ascent for 20 steps
+    for i in range(20):
+        loss_value, grads_value = iterate([input_img_data])
+        input_img_data += grads_value * step
+
+        print('Current loss value:', loss_value)
+        if loss_value <= 0.:
+            # some filters get stuck to 0, we can skip them
+            break
+
+    # decode the resulting input image
+    if loss_value > 0:
+        img = deprocess_image(input_img_data[0])
+        kept_filters.append((img, loss_value))
+    end_time = time.time()
+    print('Filter %d processed in %ds' % (filter_index, end_time - start_time))
+
+# we will stich the best 64 filters on a 8 x 8 grid.
+n = 8
+
+# the filters that have the highest loss are assumed to be better-looking.
+# we will only keep the top 64 filters.
+kept_filters.sort(key=lambda x: x[1], reverse=True)
+kept_filters = kept_filters[:n * n]
+
+# build a black picture with enough space for
+# our 8 x 8 filters of size 128 x 128, with a 5px margin in between
+margin = 5
+width = n * img_width + (n - 1) * margin
+height = n * img_height + (n - 1) * margin
+stitched_filters = np.zeros((width, height, 3))
+
+# fill the picture with our saved filters
+for i in range(n):
+    for j in range(n):
+        img, loss = kept_filters[i * n + j]
+        stitched_filters[(img_width + margin) * i: (img_width + margin) * i + img_width,
+                         (img_height + margin) * j: (img_height + margin) * j + img_height, :] = img
+
+# save the result to disk
+imsave('stitched_filters_%dx%d.png' % (n, n), stitched_filters)

examples/deep_dream.py

@@ -0,0 +1,235 @@
+'''Deep Dreaming in Keras.
+
+Run the script with:
+```
+python deep_dream.py path_to_your_base_image.jpg prefix_for_results
+```
+e.g.:
+```
+python deep_dream.py img/mypic.jpg results/dream
+```
+
+It is preferrable to run this script on GPU, for speed.
+If running on CPU, prefer the TensorFlow backend (much faster).
+
+Example results: http://i.imgur.com/FX6ROg9.jpg
+'''
+from __future__ import print_function
+from scipy.misc import imread, imresize, imsave
+import numpy as np
+from scipy.optimize import fmin_l_bfgs_b
+import time
+import argparse
+import h5py
+import os
+
+from keras.models import Sequential
+from keras.layers.convolutional import Convolution2D, ZeroPadding2D, MaxPooling2D
+from keras import backend as K
+
+parser = argparse.ArgumentParser(description='Deep Dreams with Keras.')
+parser.add_argument('base_image_path', metavar='base', type=str,
+                    help='Path to the image to transform.')
+parser.add_argument('result_prefix', metavar='res_prefix', type=str,
+                    help='Prefix for the saved results.')
+
+args = parser.parse_args()
+base_image_path = args.base_image_path
+result_prefix = args.result_prefix
+
+# dimensions of the generated picture.
+img_width = 600
+img_height = 600
+
+# path to the model weights file.
+weights_path = 'vgg16_weights.h5'
+
+# some settings we found interesting
+saved_settings = {
+    'bad_trip': {'features': {'conv4_1': 0.05,
+                              'conv4_2': 0.01,
+                              'conv4_3': 0.01},
+                 'continuity': 0.1,
+                 'dream_l2': 0.8,
+                 'jitter': 5},
+    'dreamy': {'features': {'conv5_1': 0.05,
+                            'conv5_2': 0.02},
+               'continuity': 0.1,
+               'dream_l2': 0.02,
+               'jitter': 0},
+}
+# the settings we will use in this experiment
+settings = saved_settings['dreamy']
+
+# util function to open, resize and format pictures into appropriate tensors
+def preprocess_image(image_path):
+    img = imresize(imread(image_path), (img_width, img_height))
+    img = img.transpose((2, 0, 1)).astype('float64')
+    img = np.expand_dims(img, axis=0)
+    return img
+
+# util function to convert a tensor into a valid image
+def deprocess_image(x):
+    x = x.transpose((1, 2, 0))
+    x = np.clip(x, 0, 255).astype('uint8')
+    return x
+
+# this will contain our generated image
+dream = K.placeholder((1, 3, img_width, img_height))
+
+# build the VGG16 network with our dream as input
+first_layer = ZeroPadding2D((1, 1), input_shape=(3, img_width, img_height))
+first_layer.input = dream
+
+model = Sequential()
+model.add(first_layer)
+model.add(Convolution2D(64, 3, 3, activation='relu', name='conv1_1'))
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(64, 3, 3, activation='relu', name='conv1_2'))
+model.add(MaxPooling2D((2, 2), strides=(2, 2)))
+
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(128, 3, 3, activation='relu', name='conv2_1'))
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(128, 3, 3, activation='relu', name='conv2_2'))
+model.add(MaxPooling2D((2, 2), strides=(2, 2)))
+
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(256, 3, 3, activation='relu', name='conv3_1'))
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(256, 3, 3, activation='relu', name='conv3_2'))
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(256, 3, 3, activation='relu', name='conv3_3'))
+model.add(MaxPooling2D((2, 2), strides=(2, 2)))
+
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(512, 3, 3, activation='relu', name='conv4_1'))
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(512, 3, 3, activation='relu', name='conv4_2'))
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(512, 3, 3, activation='relu', name='conv4_3'))
+model.add(MaxPooling2D((2, 2), strides=(2, 2)))
+
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(512, 3, 3, activation='relu', name='conv5_1'))
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(512, 3, 3, activation='relu', name='conv5_2'))
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(512, 3, 3, activation='relu', name='conv5_3'))
+model.add(MaxPooling2D((2, 2), strides=(2, 2)))
+
+# load the weights of the VGG16 networks
+# (trained on ImageNet, won the ILSVRC competition in 2014)
+# note: when there is a complete match between your model definition
+# and your weight savefile, you can simply call model.load_weights(filename)
+assert os.path.exists(weights_path), 'Model weights not found (see "weights_path" variable in script).'
+f = h5py.File(weights_path)
+for k in range(f.attrs['nb_layers']):
+    if k >= len(model.layers):
+        # we don't look at the last (fully-connected) layers in the savefile
+        break
+    g = f['layer_{}'.format(k)]
+    weights = [g['param_{}'.format(p)] for p in range(g.attrs['nb_params'])]
+    model.layers[k].set_weights(weights)
+f.close()
+print('Model loaded.')
+
+# get the symbolic outputs of each "key" layer (we gave them unique names).
+layer_dict = dict([(layer.name, layer) for layer in model.layers])
+
+# continuity loss util function
+def continuity_loss(x):
+    assert K.ndim(x) == 4
+    a = K.square(x[:, :, :img_width-1, :img_height-1] - x[:, :, 1:, :img_height-1])
+    b = K.square(x[:, :, :img_width-1, :img_height-1] - x[:, :, :img_width-1, 1:])
+    return K.sum(K.pow(a + b, 1.25))
+
+# define the loss
+loss = K.variable(0.)
+for layer_name in settings['features']:
+    # add the L2 norm of the features of a layer to the loss
+    assert layer_name in layer_dict.keys(), 'Layer ' + layer_name + ' not found in model.'
+    coeff = settings['features'][layer_name]
+    x = layer_dict[layer_name].get_output()
+    shape = layer_dict[layer_name].output_shape
+    # we avoid border artifacts by only involving non-border pixels in the loss
+    loss -= coeff * K.sum(K.square(x[:, :, 2: shape[2]-2, 2: shape[3]-2])) / np.prod(shape[1:])
+
+# add continuity loss (gives image local coherence, can result in an artful blur)
+loss += settings['continuity'] * continuity_loss(dream) / (3 * img_width * img_height)
+# add image L2 norm to loss (prevents pixels from taking very high values, makes image darker)
+loss += settings['dream_l2'] * K.sum(K.square(dream)) / (3 * img_width * img_height)
+
+# feel free to further modify the loss as you see fit, to achieve new effects...
+
+# compute the gradients of the dream wrt the loss
+grads = K.gradients(loss, dream)
+
+outputs = [loss]
+if type(grads) in {list, tuple}:
+    outputs += grads
+else:
+    outputs.append(grads)
+
+f_outputs = K.function([dream], outputs)
+def eval_loss_and_grads(x):
+    x = x.reshape((1, 3, img_width, img_height))
+    outs = f_outputs([x])
+    loss_value = outs[0]
+    if len(outs[1:]) == 1:
+        grad_values = outs[1].flatten().astype('float64')
+    else:
+        grad_values = np.array(outs[1:]).flatten().astype('float64')
+    return loss_value, grad_values
+
+# this Evaluator class makes it possible
+# to compute loss and gradients in one pass
+# while retrieving them via two separate functions,
+# "loss" and "grads". This is done because scipy.optimize
+# requires separate functions for loss and gradients,
+# but computing them separately would be inefficient.
+class Evaluator(object):
+    def __init__(self):
+        self.loss_value = None
+        self.grads_values = None
+
+    def loss(self, x):
+        assert self.loss_value is None
+        loss_value, grad_values = eval_loss_and_grads(x)
+        self.loss_value = loss_value
+        self.grad_values = grad_values
+        return self.loss_value
+
+    def grads(self, x):
+        assert self.loss_value is not None
+        grad_values = np.copy(self.grad_values)
+        self.loss_value = None
+        self.grad_values = None
+        return grad_values
+
+evaluator = Evaluator()
+
+# run scipy-based optimization (L-BFGS) over the pixels of the generated image
+# so as to minimize the loss
+x = preprocess_image(base_image_path)
+for i in range(5):
+    print('Start of iteration', i)
+    start_time = time.time()
+
+    # add a random offset jitter to the initial image. This will be reverted at decoding time
+    ox, oy = np.random.randint(-settings['jitter'], settings['jitter']+1, 2)
+    x = np.roll(np.roll(x, ox, -1), oy, -2)
+
+    # run L-BFGS for 7 steps
+    x, min_val, info = fmin_l_bfgs_b(evaluator.loss, x.flatten(),
+                                     fprime=evaluator.grads, maxfun=7)
+    print('Current loss value:', min_val)
+    # decode the dream and save it
+    x = x.reshape((3, img_width, img_height))
+    x = np.roll(np.roll(x, -ox, -1), -oy, -2) # unshift image
+    img = deprocess_image(x)
+    fname = result_prefix + '_at_iteration_%d.png' % i
+    imsave(fname, img)
+    end_time = time.time()
+    print('Image saved as', fname)
+    print('Iteration %d completed in %ds' % (i, end_time - start_time))

examples/imdb_bidirectional_lstm.py

@@ -0,0 +1,62 @@
+'''Train a Bidirectional LSTM on the IMDB sentiment classification task.
+
+GPU command:
+    THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 python imdb_bidirectional_lstm.py
+
+Output after 4 epochs on CPU: ~0.8146
+Time per epoch on CPU (Core i7): ~150s.
+'''
+
+from __future__ import print_function
+import numpy as np
+np.random.seed(1337)  # for reproducibility
+
+from keras.preprocessing import sequence
+from keras.utils.np_utils import accuracy
+from keras.models import Graph
+from keras.layers.core import Dense, Dropout
+from keras.layers.embeddings import Embedding
+from keras.layers.recurrent import LSTM
+from keras.datasets import imdb
+
+
+max_features = 20000
+maxlen = 100  # cut texts after this number of words (among top max_features most common words)
+batch_size = 32
+
+print('Loading data...')
+(X_train, y_train), (X_test, y_test) = imdb.load_data(nb_words=max_features,
+                                                      test_split=0.2)
+print(len(X_train), 'train sequences')
+print(len(X_test), 'test sequences')
+
+print("Pad sequences (samples x time)")
+X_train = sequence.pad_sequences(X_train, maxlen=maxlen)
+X_test = sequence.pad_sequences(X_test, maxlen=maxlen)
+print('X_train shape:', X_train.shape)
+print('X_test shape:', X_test.shape)
+y_train = np.array(y_train)
+y_test = np.array(y_test)
+
+print('Build model...')
+model = Graph()
+model.add_input(name='input', input_shape=(maxlen,), dtype=int)
+model.add_node(Embedding(max_features, 128, input_length=maxlen),
+               name='embedding', input='input')
+model.add_node(LSTM(64), name='forward', input='embedding')
+model.add_node(LSTM(64, go_backwards=True), name='backward', input='embedding')
+model.add_node(Dropout(0.5), name='dropout', inputs=['forward', 'backward'])
+model.add_node(Dense(1, activation='sigmoid'), name='sigmoid', input='dropout')
+model.add_output(name='output', input='sigmoid')
+
+# try using different optimizers and different optimizer configs
+model.compile('adam', {'output': 'binary_crossentropy'})
+
+print('Train...')
+model.fit({'input': X_train, 'output': y_train},
+          batch_size=batch_size,
+          nb_epoch=4, show_accuracy=True)
+acc = accuracy(y_test,
+               np.round(np.array(model.predict({'input': X_test},
+                                               batch_size=batch_size)['output'])))
+print('Test accuracy:', acc)

examples/imdb_cnn.py

@@ -0,0 +1,77 @@
+'''This example demonstrates the use of Convolution1D for text classification.
+
+Run on GPU: THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 python imdb_cnn.py
+
+Get to 0.835 test accuracy after 2 epochs. 100s/epoch on K520 GPU.
+'''
+
+from __future__ import print_function
+import numpy as np
+np.random.seed(1337)  # for reproducibility
+
+from keras.preprocessing import sequence
+from keras.models import Sequential
+from keras.layers.core import Dense, Dropout, Activation, Flatten
+from keras.layers.embeddings import Embedding
+from keras.layers.convolutional import Convolution1D, MaxPooling1D
+from keras.datasets import imdb
+
+
+# set parameters:
+max_features = 5000
+maxlen = 100
+batch_size = 32
+embedding_dims = 100
+nb_filter = 250
+filter_length = 3
+hidden_dims = 250
+nb_epoch = 2
+
+print('Loading data...')
+(X_train, y_train), (X_test, y_test) = imdb.load_data(nb_words=max_features,
+                                                      test_split=0.2)
+print(len(X_train), 'train sequences')
+print(len(X_test), 'test sequences')
+
+print('Pad sequences (samples x time)')
+X_train = sequence.pad_sequences(X_train, maxlen=maxlen)
+X_test = sequence.pad_sequences(X_test, maxlen=maxlen)
+print('X_train shape:', X_train.shape)
+print('X_test shape:', X_test.shape)
+
+print('Build model...')
+model = Sequential()
+
+# we start off with an efficient embedding layer which maps
+# our vocab indices into embedding_dims dimensions
+model.add(Embedding(max_features, embedding_dims, input_length=maxlen))
+model.add(Dropout(0.25))
+
+# we add a Convolution1D, which will learn nb_filter
+# word group filters of size filter_length:
+model.add(Convolution1D(nb_filter=nb_filter,
+                        filter_length=filter_length,
+                        border_mode='valid',
+                        activation='relu',
+                        subsample_length=1))
+# we use standard max pooling (halving the output of the previous layer):
+model.add(MaxPooling1D(pool_length=2))
+
+# We flatten the output of the conv layer,
+# so that we can add a vanilla dense layer:
+model.add(Flatten())
+
+# We add a vanilla hidden layer:
+model.add(Dense(hidden_dims))
+model.add(Dropout(0.25))
+model.add(Activation('relu'))
+
+# We project onto a single unit output layer, and squash it with a sigmoid:
+model.add(Dense(1))
+model.add(Activation('sigmoid'))
+
+model.compile(loss='binary_crossentropy',
+              optimizer='rmsprop')
+model.fit(X_train, y_train, batch_size=batch_size,
+          nb_epoch=nb_epoch, show_accuracy=True,
+          validation_data=(X_test, y_test))

examples/imdb_cnn_lstm.py

@@ -0,0 +1,82 @@
+'''Train a recurrent convolutional network on the IMDB sentiment
+classification task.
+
+GPU command:
+    THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 python imdb_cnn_lstm.py
+
+Get to 0.8498 test accuracy after 2 epochs. 41s/epoch on K520 GPU.
+'''
+
+from __future__ import print_function
+import numpy as np
+np.random.seed(1337)  # for reproducibility
+
+from keras.preprocessing import sequence
+from keras.models import Sequential
+from keras.layers.core import Dense, Dropout, Activation
+from keras.layers.embeddings import Embedding
+from keras.layers.recurrent import LSTM, GRU, SimpleRNN
+from keras.layers.convolutional import Convolution1D, MaxPooling1D
+from keras.datasets import imdb
+
+
+# Embedding
+max_features = 20000
+maxlen = 100
+embedding_size = 128
+
+# Convolution
+filter_length = 3
+nb_filter = 64
+pool_length = 2
+
+# LSTM
+lstm_output_size = 70
+
+# Training
+batch_size = 30
+nb_epoch = 2
+
+'''
+Note:
+batch_size is highly sensitive.
+Only 2 epochs are needed as the dataset is very small.
+'''
+
+print('Loading data...')
+(X_train, y_train), (X_test, y_test) = imdb.load_data(nb_words=max_features, test_split=0.2)
+print(len(X_train), 'train sequences')
+print(len(X_test), 'test sequences')
+
+print('Pad sequences (samples x time)')
+X_train = sequence.pad_sequences(X_train, maxlen=maxlen)
+X_test = sequence.pad_sequences(X_test, maxlen=maxlen)
+print('X_train shape:', X_train.shape)
+print('X_test shape:', X_test.shape)
+
+print('Build model...')
+
+model = Sequential()
+model.add(Embedding(max_features, embedding_size, input_length=maxlen))
+model.add(Dropout(0.25))
+model.add(Convolution1D(nb_filter=nb_filter,
+                        filter_length=filter_length,
+                        border_mode='valid',
+                        activation='relu',
+                        subsample_length=1))
+model.add(MaxPooling1D(pool_length=pool_length))
+model.add(LSTM(lstm_output_size))
+model.add(Dense(1))
+model.add(Activation('sigmoid'))
+
+model.compile(loss='binary_crossentropy',
+              optimizer='adam',
+              class_mode='binary')
+
+print('Train...')
+model.fit(X_train, y_train, batch_size=batch_size, nb_epoch=nb_epoch,
+          validation_data=(X_test, y_test), show_accuracy=True)
+score, acc = model.evaluate(X_test, y_test, batch_size=batch_size,
+                            show_accuracy=True)
+print('Test score:', score)
+print('Test accuracy:', acc)

examples/imdb_lstm.py

@@ -1,48 +1,44 @@
-from __future__ import absolute_import
+'''Train a LSTM on the IMDB sentiment classification task.
+
+The dataset is actually too small for LSTM to be of any advantage
+compared to simpler, much faster methods such as TF-IDF+LogReg.
+
+Notes:
+
+- RNNs are tricky. Choice of batch size is important,
+choice of loss and optimizer is critical, etc.
+Some configurations won't converge.
+
+- LSTM loss decrease patterns during training can be quite different
+from what you see with CNNs/MLPs/etc.
+
+GPU command:
+    THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 python imdb_lstm.py
+'''
+
 from __future__ import print_function
 import numpy as np
+np.random.seed(1337)  # for reproducibility
 
 from keras.preprocessing import sequence
-from keras.optimizers import SGD, RMSprop, Adagrad
 from keras.utils import np_utils
 from keras.models import Sequential
 from keras.layers.core import Dense, Dropout, Activation
 from keras.layers.embeddings import Embedding
-from keras.layers.recurrent import LSTM, GRU
+from keras.layers.recurrent import LSTM
 from keras.datasets import imdb
 
-'''
-    Train a LSTM on the IMDB sentiment classification task.
+max_features = 20000
+maxlen = 100  # cut texts after this number of words (among top max_features most common words)
+batch_size = 32
 
-    The dataset is actually too small for LSTM to be of any advantage 
-    compared to simpler, much faster methods such as TF-IDF+LogReg.
-
-    Notes: 
-
-    - RNNs are tricky. Choice of batch size is important, 
-    choice of loss and optimizer is critical, etc. 
-    Most configurations won't converge.
-
-    - LSTM loss decrease during training can be quite different 
-    from what you see with CNNs/MLPs/etc. It's more or less a sigmoid
-    instead of an inverse exponential.
-
-    GPU command:
-        THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 python imdb_lstm.py
-
-    250s/epoch on GPU (GT 650M), vs. 400s/epoch on CPU (2.4Ghz Core i7).
-'''
-
-max_features=20000
-maxlen = 100 # cut texts after this number of words (among top max_features most common words)
-batch_size = 16
-
-print("Loading data...")
-(X_train, y_train), (X_test, y_test) = imdb.load_data(nb_words=max_features, test_split=0.2)
+print('Loading data...')
+(X_train, y_train), (X_test, y_test) = imdb.load_data(nb_words=max_features,
+                                                      test_split=0.2)
 print(len(X_train), 'train sequences')
 print(len(X_test), 'test sequences')
 
-print("Pad sequences (samples x time)")
+print('Pad sequences (samples x time)')
 X_train = sequence.pad_sequences(X_train, maxlen=maxlen)
 X_test = sequence.pad_sequences(X_test, maxlen=maxlen)
 print('X_train shape:', X_train.shape)
@@ -50,21 +46,21 @@ print('X_test shape:', X_test.shape)
 
 print('Build model...')
 model = Sequential()
-model.add(Embedding(max_features, 256))
-model.add(LSTM(256, 128)) # try using a GRU instead, for fun
+model.add(Embedding(max_features, 128, input_length=maxlen, dropout=0.5))
+model.add(LSTM(128, dropout_W=0.5, dropout_U=0.1))  # try using a GRU instead, for fun
 model.add(Dropout(0.5))
-model.add(Dense(128, 1))
+model.add(Dense(1))
 model.add(Activation('sigmoid'))
 
 # try using different optimizers and different optimizer configs
-model.compile(loss='binary_crossentropy', optimizer='adam', class_mode="binary")
+model.compile(loss='binary_crossentropy',
+              optimizer='adam')
 
-print("Train...")
-model.fit(X_train, y_train, batch_size=batch_size, nb_epoch=5, validation_split=0.1, show_accuracy=True)
-score = model.evaluate(X_test, y_test, batch_size=batch_size)
+print('Train...')
+model.fit(X_train, y_train, batch_size=batch_size, nb_epoch=15,
+          validation_data=(X_test, y_test), show_accuracy=True)
+score, acc = model.evaluate(X_test, y_test,
+                            batch_size=batch_size,
+                            show_accuracy=True)
 print('Test score:', score)
-
-classes = model.predict_classes(X_test, batch_size=batch_size)
-acc = np_utils.accuracy(classes, y_test)
 print('Test accuracy:', acc)
-

examples/kaggle_otto_nn.py

@@ -1,8 +1,29 @@
-from __future__ import absolute_import
-from __future__ import print_function
+'''This demonstrates how to reach a score of 0.4890 (local validation)
+on the Kaggle Otto challenge, with a deep net using Keras.
 
+Requires Scikit-Learn and Pandas.
+
+Recommended to run on GPU:
+    Command: THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 python kaggle_otto_nn.py
+    On EC2 g2.2xlarge instance: 19s/epoch. 6-7 minutes total training time.
+
+Best validation score at epoch 21: 0.4881
+
+Try it at home:
+    - with/without BatchNormalization (BatchNormalization helps!)
+    - with ReLU or with PReLU (PReLU helps!)
+    - with smaller layers, largers layers
+    - with more layers, less layers
+    - with different optimizers (SGD+momentum+decay is probably better than Adam!)
+
+Get the data from Kaggle:
+https://www.kaggle.com/c/otto-group-product-classification-challenge/data
+'''
+
+from __future__ import print_function
 import numpy as np
 import pandas as pd
+np.random.seed(1337)  # for reproducibility
 
 from keras.models import Sequential
 from keras.layers.core import Dense, Dropout, Activation
@@ -13,41 +34,19 @@ from keras.utils import np_utils, generic_utils
 from sklearn.preprocessing import LabelEncoder
 from sklearn.preprocessing import StandardScaler
 
-'''
-    This demonstrates how to reach a score of 0.4890 (local validation)
-    on the Kaggle Otto challenge, with a deep net using Keras.
-
-    Compatible Python 2.7-3.4 
-
-    Recommended to run on GPU: 
-        Command: THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 python kaggle_otto_nn.py
-        On EC2 g2.2xlarge instance: 19s/epoch. 6-7 minutes total training time.
-
-    Best validation score at epoch 21: 0.4881 
-
-    Try it at home:
-        - with/without BatchNormalization (BatchNormalization helps!)
-        - with ReLU or with PReLU (PReLU helps!)
-        - with smaller layers, largers layers
-        - with more layers, less layers
-        - with different optimizers (SGD+momentum+decay is probably better than Adam!)
-
-    Get the data from Kaggle: https://www.kaggle.com/c/otto-group-product-classification-challenge/data
-'''
-
-np.random.seed(1337) # for reproducibility
 
 def load_data(path, train=True):
     df = pd.read_csv(path)
     X = df.values.copy()
     if train:
-        np.random.shuffle(X) # https://youtu.be/uyUXoap67N8
+        np.random.shuffle(X)  # https://youtu.be/uyUXoap67N8
         X, labels = X[:, 1:-1].astype(np.float32), X[:, -1]
         return X, labels
     else:
         X, ids = X[:, 1:].astype(np.float32), X[:, 0].astype(str)
         return X, ids
 
+
 def preprocess_data(X, scaler=None):
     if not scaler:
         scaler = StandardScaler()
@@ -55,6 +54,7 @@ def preprocess_data(X, scaler=None):
     X = scaler.transform(X)
     return X, scaler
 
+
 def preprocess_labels(labels, encoder=None, categorical=True):
     if not encoder:
         encoder = LabelEncoder()
@@ -64,6 +64,7 @@ def preprocess_labels(labels, encoder=None, categorical=True):
         y = np_utils.to_categorical(y)
     return y, encoder
 
+
 def make_submission(y_prob, ids, encoder, fname):
     with open(fname, 'w') as f:
         f.write('id,')
@@ -73,10 +74,9 @@ def make_submission(y_prob, ids, encoder, fname):
             probas = ','.join([i] + [str(p) for p in probs.tolist()])
             f.write(probas)
             f.write('\n')
-    print("Wrote submission to file {}.".format(fname))
+    print('Wrote submission to file {}.'.format(fname))
 
-
-print("Loading data...")
+print('Loading data...')
 X, labels = load_data('train.csv', train=True)
 X, scaler = preprocess_data(X)
 y, encoder = preprocess_labels(labels)
@@ -90,35 +90,32 @@ print(nb_classes, 'classes')
 dims = X.shape[1]
 print(dims, 'dims')
 
-print("Building model...")
+print('Building model...')
 
 model = Sequential()
-model.add(Dense(dims, 512, init='glorot_uniform'))
-model.add(PReLU((512,)))
-model.add(BatchNormalization((512,)))
+model.add(Dense(512, input_shape=(dims,)))
+model.add(PReLU())
+model.add(BatchNormalization())
 model.add(Dropout(0.5))
 
-model.add(Dense(512, 512, init='glorot_uniform'))
-model.add(PReLU((512,)))
-model.add(BatchNormalization((512,)))
+model.add(Dense(512))
+model.add(PReLU())
+model.add(BatchNormalization())
 model.add(Dropout(0.5))
 
-model.add(Dense(512, 512, init='glorot_uniform'))
-model.add(PReLU((512,)))
-model.add(BatchNormalization((512,)))
+model.add(Dense(512))
+model.add(PReLU())
+model.add(BatchNormalization())
 model.add(Dropout(0.5))
 
-model.add(Dense(512, nb_classes, init='glorot_uniform'))
+model.add(Dense(nb_classes))
 model.add(Activation('softmax'))
 
-model.compile(loss='categorical_crossentropy', optimizer="adam")
-
-print("Training model...")
+model.compile(loss='categorical_crossentropy', optimizer='adam')
 
+print('Training model...')
 model.fit(X, y, nb_epoch=20, batch_size=128, validation_split=0.15)
 
-print("Generating submission...")
-
+print('Generating submission...')
 proba = model.predict_proba(X_test)
 make_submission(proba, ids, encoder, fname='keras-otto.csv')
-

examples/lstm_text_generation.py

@@ -0,0 +1,108 @@
+'''Example script to generate text from Nietzsche's writings.
+
+At least 20 epochs are required before the generated text
+starts sounding coherent.
+
+It is recommended to run this script on GPU, as recurrent
+networks are quite computationally intensive.
+
+If you try this script on new data, make sure your corpus
+has at least ~100k characters. ~1M is better.
+'''
+
+from __future__ import print_function
+from keras.models import Sequential
+from keras.layers.core import Dense, Activation, Dropout
+from keras.layers.recurrent import LSTM
+from keras.utils.data_utils import get_file
+import numpy as np
+import random
+import sys
+
+path = get_file('nietzsche.txt', origin="https://s3.amazonaws.com/text-datasets/nietzsche.txt")
+
+try: 
+    text = open(path).read().lower()
+except UnicodeDecodeError:
+    import codecs
+    text = codecs.open(path, encoding='utf-8').read().lower()
+
+print('corpus length:', len(text))
+
+chars = set(text)
+print('total chars:', len(chars))
+char_indices = dict((c, i) for i, c in enumerate(chars))
+indices_char = dict((i, c) for i, c in enumerate(chars))
+
+# cut the text in semi-redundant sequences of maxlen characters
+maxlen = 20
+step = 3
+sentences = []
+next_chars = []
+for i in range(0, len(text) - maxlen, step):
+    sentences.append(text[i: i + maxlen])
+    next_chars.append(text[i + maxlen])
+print('nb sequences:', len(sentences))
+
+print('Vectorization...')
+X = np.zeros((len(sentences), maxlen, len(chars)), dtype=np.bool)
+y = np.zeros((len(sentences), len(chars)), dtype=np.bool)
+for i, sentence in enumerate(sentences):
+    for t, char in enumerate(sentence):
+        X[i, t, char_indices[char]] = 1
+    y[i, char_indices[next_chars[i]]] = 1
+
+
+# build the model: 2 stacked LSTM
+print('Build model...')
+model = Sequential()
+model.add(LSTM(512, return_sequences=True, input_shape=(maxlen, len(chars))))
+model.add(Dropout(0.2))
+model.add(LSTM(512, return_sequences=False))
+model.add(Dropout(0.2))
+model.add(Dense(len(chars)))
+model.add(Activation('softmax'))
+
+model.compile(loss='categorical_crossentropy', optimizer='rmsprop')
+
+
+def sample(a, temperature=1.0):
+    # helper function to sample an index from a probability array
+    a = np.log(a) / temperature
+    a = np.exp(a) / np.sum(np.exp(a))
+    return np.argmax(np.random.multinomial(1, a, 1))
+
+# train the model, output generated text after each iteration
+for iteration in range(1, 60):
+    print()
+    print('-' * 50)
+    print('Iteration', iteration)
+    model.fit(X, y, batch_size=128, nb_epoch=1)
+
+    start_index = random.randint(0, len(text) - maxlen - 1)
+
+    for diversity in [0.2, 0.5, 1.0, 1.2]:
+        print()
+        print('----- diversity:', diversity)
+
+        generated = ''
+        sentence = text[start_index: start_index + maxlen]
+        generated += sentence
+        print('----- Generating with seed: "' + sentence + '"')
+        sys.stdout.write(generated)
+
+        for i in range(400):
+            x = np.zeros((1, maxlen, len(chars)))
+            for t, char in enumerate(sentence):
+                x[0, t, char_indices[char]] = 1.
+
+            preds = model.predict(x, verbose=0)[0]
+            next_index = sample(preds, diversity)
+            next_char = indices_char[next_index]
+
+            generated += next_char
+            sentence = sentence[1:] + next_char
+
+            sys.stdout.write(next_char)
+            sys.stdout.flush()
+        print()

examples/mnist_cnn.py

@@ -0,0 +1,73 @@
+'''Train a simple convnet on the MNIST dataset.
+
+Run on GPU: THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 python mnist_cnn.py
+
+Get to 99.25% test accuracy after 12 epochs (there is still a lot of margin for parameter tuning).
+16 seconds per epoch on a GRID K520 GPU.
+'''
+
+from __future__ import print_function
+import numpy as np
+np.random.seed(1337)  # for reproducibility
+
+from keras.datasets import mnist
+from keras.models import Sequential
+from keras.layers.core import Dense, Dropout, Activation, Flatten
+from keras.layers.convolutional import Convolution2D, MaxPooling2D
+from keras.utils import np_utils
+
+batch_size = 128
+nb_classes = 10
+nb_epoch = 12
+
+# input image dimensions
+img_rows, img_cols = 28, 28
+# number of convolutional filters to use
+nb_filters = 32
+# size of pooling area for max pooling
+nb_pool = 2
+# convolution kernel size
+nb_conv = 3
+
+# the data, shuffled and split between train and test sets
+(X_train, y_train), (X_test, y_test) = mnist.load_data()
+
+X_train = X_train.reshape(X_train.shape[0], 1, img_rows, img_cols)
+X_test = X_test.reshape(X_test.shape[0], 1, img_rows, img_cols)
+X_train = X_train.astype('float32')
+X_test = X_test.astype('float32')
+X_train /= 255
+X_test /= 255
+print('X_train shape:', X_train.shape)
+print(X_train.shape[0], 'train samples')
+print(X_test.shape[0], 'test samples')
+
+# convert class vectors to binary class matrices
+Y_train = np_utils.to_categorical(y_train, nb_classes)
+Y_test = np_utils.to_categorical(y_test, nb_classes)
+
+model = Sequential()
+
+model.add(Convolution2D(nb_filters, nb_conv, nb_conv,
+                        border_mode='valid',
+                        input_shape=(1, img_rows, img_cols)))
+model.add(Activation('relu'))
+model.add(Convolution2D(nb_filters, nb_conv, nb_conv))
+model.add(Activation('relu'))
+model.add(MaxPooling2D(pool_size=(nb_pool, nb_pool)))
+model.add(Dropout(0.25))
+
+model.add(Flatten())
+model.add(Dense(128))
+model.add(Activation('relu'))
+model.add(Dropout(0.5))
+model.add(Dense(nb_classes))
+model.add(Activation('softmax'))
+
+model.compile(loss='categorical_crossentropy', optimizer='adadelta')
+
+model.fit(X_train, Y_train, batch_size=batch_size, nb_epoch=nb_epoch,
+          show_accuracy=True, verbose=1, validation_data=(X_test, Y_test))
+score = model.evaluate(X_test, Y_test, show_accuracy=True, verbose=0)
+print('Test score:', score[0])
+print('Test accuracy:', score[1])

examples/mnist_irnn.py

@@ -0,0 +1,85 @@
+'''This is a reproduction of the IRNN experiment
+with pixel-by-pixel sequential MNIST in
+"A Simple Way to Initialize Recurrent Networks of Rectified Linear Units"
+by Quoc V. Le, Navdeep Jaitly, Geoffrey E. Hinton
+
+arXiv:1504.00941v2 [cs.NE] 7 Apr 201
+http://arxiv.org/pdf/1504.00941v2.pdf
+
+Optimizer is replaced with RMSprop which yields more stable and steady
+improvement.
+
+Reaches 0.93 train/test accuracy after 900 epochs
+(which roughly corresponds to 1687500 steps in the original paper.)
+'''
+
+from __future__ import print_function
+import numpy as np
+np.random.seed(1337)  # for reproducibility
+
+from keras.datasets import mnist
+from keras.models import Sequential
+from keras.layers.core import Dense, Activation
+from keras.initializations import normal, identity
+from keras.layers.recurrent import SimpleRNN, LSTM
+from keras.optimizers import RMSprop
+from keras.utils import np_utils
+
+
+batch_size = 32
+nb_classes = 10
+nb_epochs = 200
+hidden_units = 100
+
+learning_rate = 1e-6
+clip_norm = 1.0
+
+# the data, shuffled and split between train and test sets
+(X_train, y_train), (X_test, y_test) = mnist.load_data()
+
+X_train = X_train.reshape(X_train.shape[0], -1, 1)
+X_test = X_test.reshape(X_test.shape[0], -1, 1)
+X_train = X_train.astype('float32')
+X_test = X_test.astype('float32')
+X_train /= 255
+X_test /= 255
+print('X_train shape:', X_train.shape)
+print(X_train.shape[0], 'train samples')
+print(X_test.shape[0], 'test samples')
+
+# convert class vectors to binary class matrices
+Y_train = np_utils.to_categorical(y_train, nb_classes)
+Y_test = np_utils.to_categorical(y_test, nb_classes)
+
+print('Evaluate IRNN...')
+model = Sequential()
+model.add(SimpleRNN(output_dim=hidden_units,
+                    init=lambda shape: normal(shape, scale=0.001),
+                    inner_init=lambda shape: identity(shape, scale=1.0),
+                    activation='relu', input_shape=X_train.shape[1:]))
+model.add(Dense(nb_classes))
+model.add(Activation('softmax'))
+rmsprop = RMSprop(lr=learning_rate)
+model.compile(loss='categorical_crossentropy', optimizer=rmsprop)
+
+model.fit(X_train, Y_train, batch_size=batch_size, nb_epoch=nb_epochs,
+          show_accuracy=True, verbose=1, validation_data=(X_test, Y_test))
+
+scores = model.evaluate(X_test, Y_test, show_accuracy=True, verbose=0)
+print('IRNN test score:', scores[0])
+print('IRNN test accuracy:', scores[1])
+
+print('Compare to LSTM...')
+model = Sequential()
+model.add(LSTM(hidden_units, input_shape=X_train.shape[1:]))
+model.add(Dense(nb_classes))
+model.add(Activation('softmax'))
+rmsprop = RMSprop(lr=learning_rate)
+model.compile(loss='categorical_crossentropy', optimizer=rmsprop)
+
+model.fit(X_train, Y_train, batch_size=batch_size, nb_epoch=nb_epochs,
+          show_accuracy=True, verbose=1, validation_data=(X_test, Y_test))
+
+scores = model.evaluate(X_test, Y_test, show_accuracy=True, verbose=0)
+print('LSTM test score:', scores[0])
+print('LSTM test accuracy:', scores[1])

examples/mnist_mlp.py

@@ -1,31 +1,32 @@
-from __future__ import absolute_import
+'''Train a simple deep NN on the MNIST dataset.
+
+Get to 98.40% test accuracy after 20 epochs
+(there is *a lot* of margin for parameter tuning).
+2 seconds per epoch on a K520 GPU.
+'''
+
 from __future__ import print_function
+import numpy as np
+np.random.seed(1337)  # for reproducibility
+
 from keras.datasets import mnist
 from keras.models import Sequential
 from keras.layers.core import Dense, Dropout, Activation
-from keras.regularizers import l2, l1
-from keras.constraints import maxnorm
 from keras.optimizers import SGD, Adam, RMSprop
 from keras.utils import np_utils
-import numpy as np
 
-'''
-    Train a simple deep NN on the MNIST dataset.
-'''
 
-batch_size = 64
+batch_size = 128
 nb_classes = 10
 nb_epoch = 20
 
-np.random.seed(1337) # for reproducibility
-
-# the data, shuffled and split between tran and test sets
+# the data, shuffled and split between train and test sets
 (X_train, y_train), (X_test, y_test) = mnist.load_data()
 
-X_train = X_train.reshape(60000,784)
-X_test = X_test.reshape(10000,784)
-X_train = X_train.astype("float32")
-X_test = X_test.astype("float32")
+X_train = X_train.reshape(60000, 784)
+X_test = X_test.reshape(10000, 784)
+X_train = X_train.astype('float32')
+X_test = X_test.astype('float32')
 X_train /= 255
 X_test /= 255
 print(X_train.shape[0], 'train samples')
@@ -36,19 +37,23 @@ Y_train = np_utils.to_categorical(y_train, nb_classes)
 Y_test = np_utils.to_categorical(y_test, nb_classes)
 
 model = Sequential()
-model.add(Dense(784, 128))
+model.add(Dense(512, input_shape=(784,)))
 model.add(Activation('relu'))
 model.add(Dropout(0.2))
-model.add(Dense(128, 128))
+model.add(Dense(512))
 model.add(Activation('relu'))
 model.add(Dropout(0.2))
-model.add(Dense(128, 10))
+model.add(Dense(10))
 model.add(Activation('softmax'))
 
 rms = RMSprop()
 model.compile(loss='categorical_crossentropy', optimizer=rms)
 
-model.fit(X_train, Y_train, batch_size=batch_size, nb_epoch=nb_epoch, show_accuracy=True, verbose=2, validation_data=(X_test, Y_test))
-score = model.evaluate(X_test, Y_test, show_accuracy=True, verbose=0)
+model.fit(X_train, Y_train,
+          batch_size=batch_size, nb_epoch=nb_epoch,
+          show_accuracy=True, verbose=2,
+          validation_data=(X_test, Y_test))
+score = model.evaluate(X_test, Y_test,
+                       show_accuracy=True, verbose=0)
 print('Test score:', score[0])
 print('Test accuracy:', score[1])

examples/mnist_siamese_graph.py

@@ -0,0 +1,124 @@
+'''Train a Siamese MLP on pairs of digits from the MNIST dataset.
+
+It follows Hadsell-et-al.'06 [1] by computing the Euclidean distance on the
+output of the shared network and by optimizing the contrastive loss (see paper
+for mode details).
+
+[1] "Dimensionality Reduction by Learning an Invariant Mapping"
+    http://yann.lecun.com/exdb/publis/pdf/hadsell-chopra-lecun-06.pdf
+
+Run on GPU: THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 python mnist_siamese_graph.py
+
+Gets to 99.5% test accuracy after 20 epochs.
+3 seconds per epoch on a Titan X GPU
+'''
+from __future__ import absolute_import
+from __future__ import print_function
+import numpy as np
+np.random.seed(1337)  # for reproducibility
+
+import random
+from keras.datasets import mnist
+from keras.models import Sequential, Graph
+from keras.layers.core import Dense, Dropout, Lambda
+from keras.optimizers import SGD, RMSprop
+from keras import backend as K
+
+
+def euclidean_distance(inputs):
+    assert len(inputs) == 2, ('Euclidean distance needs '
+                              '2 inputs, %d given' % len(inputs))
+    u, v = inputs.values()
+    return K.sqrt(K.sum(K.square(u - v), axis=1, keepdims=True))
+
+
+def contrastive_loss(y, d):
+    '''Contrastive loss from Hadsell-et-al.'06
+    http://yann.lecun.com/exdb/publis/pdf/hadsell-chopra-lecun-06.pdf
+    '''
+    margin = 1
+    return K.mean(y * K.square(d) + (1 - y) * K.square(K.maximum(margin - d, 0)))
+
+
+def create_pairs(x, digit_indices):
+    '''Positive and negative pair creation.
+    Alternates between positive and negative pairs.
+    '''
+    pairs = []
+    labels = []
+    n = min([len(digit_indices[d]) for d in range(10)]) - 1
+    for d in range(10):
+        for i in range(n):
+            z1, z2 = digit_indices[d][i], digit_indices[d][i+1]
+            pairs += [[x[z1], x[z2]]]
+            inc = random.randrange(1, 10)
+            dn = (d + inc) % 10
+            z1, z2 = digit_indices[d][i], digit_indices[dn][i]
+            pairs += [[x[z1], x[z2]]]
+            labels += [1, 0]
+    return np.array(pairs), np.array(labels)
+
+
+def create_base_network(input_dim):
+    '''Base network to be shared (eq. to feature extraction).
+    '''
+    seq = Sequential()
+    seq.add(Dense(128, input_shape=(input_dim,), activation='relu'))
+    seq.add(Dropout(0.1))
+    seq.add(Dense(128, activation='relu'))
+    seq.add(Dropout(0.1))
+    seq.add(Dense(128, activation='relu'))
+    return seq
+
+
+def compute_accuracy(predictions, labels):
+    '''Compute classification accuracy with a fixed threshold on distances.
+    '''
+    return labels[predictions.ravel() < 0.5].mean()
+
+
+# the data, shuffled and split between train and test sets
+(X_train, y_train), (X_test, y_test) = mnist.load_data()
+X_train = X_train.reshape(60000, 784)
+X_test = X_test.reshape(10000, 784)
+X_train = X_train.astype('float32')
+X_test = X_test.astype('float32')
+X_train /= 255
+X_test /= 255
+input_dim = 784
+nb_epoch = 20
+
+# create training+test positive and negative pairs
+digit_indices = [np.where(y_train == i)[0] for i in range(10)]
+tr_pairs, tr_y = create_pairs(X_train, digit_indices)
+
+digit_indices = [np.where(y_test == i)[0] for i in range(10)]
+te_pairs, te_y = create_pairs(X_test, digit_indices)
+
+# network definition
+base_network = create_base_network(input_dim)
+
+g = Graph()
+g.add_input(name='input_a', input_shape=(input_dim,))
+g.add_input(name='input_b', input_shape=(input_dim,))
+g.add_shared_node(base_network, name='shared', inputs=['input_a', 'input_b'],
+                  merge_mode='join')
+g.add_node(Lambda(euclidean_distance), name='d', input='shared')
+g.add_output(name='output', input='d')
+
+# train
+rms = RMSprop()
+g.compile(loss={'output': contrastive_loss}, optimizer=rms)
+g.fit({'input_a': tr_pairs[:, 0], 'input_b': tr_pairs[:, 1], 'output': tr_y},
+      validation_data={'input_a': te_pairs[:, 0], 'input_b': te_pairs[:, 1], 'output': te_y},
+      batch_size=128,
+      nb_epoch=nb_epoch)
+
+# compute final accuracy on training and test sets
+pred = g.predict({'input_a': tr_pairs[:, 0], 'input_b': tr_pairs[:, 1]})['output']
+tr_acc = compute_accuracy(pred, tr_y)
+pred = g.predict({'input_a': te_pairs[:, 0], 'input_b': te_pairs[:, 1]})['output']
+te_acc = compute_accuracy(pred, te_y)
+
+print('* Accuracy on training set: %0.2f%%' % (100 * tr_acc))
+print('* Accuracy on test set: %0.2f%%' % (100 * te_acc))

examples/mnist_transfer_cnn.py

@@ -0,0 +1,122 @@
+'''Transfer learning toy example:
+
+1- Train a simple convnet on the MNIST dataset the first 5 digits [0..4].
+2- Freeze convolutional layers and fine-tune dense layers
+   for the classification of digits [5..9].
+
+Run on GPU: THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 python mnist_transfer_cnn.py
+
+Get to 99.8% test accuracy after 5 epochs
+for the first five digits classifier
+and 99.2% for the last five digits after transfer + fine-tuning.
+'''
+
+from __future__ import print_function
+import numpy as np
+import datetime
+
+np.random.seed(1337)  # for reproducibility
+
+from keras.datasets import mnist
+from keras.models import Sequential
+from keras.layers.core import Dense, Dropout, Activation, Flatten
+from keras.layers.convolutional import Convolution2D, MaxPooling2D
+from keras.utils import np_utils
+
+
+now = datetime.datetime.now
+
+batch_size = 128
+nb_classes = 5
+nb_epoch = 5
+
+# input image dimensions
+img_rows, img_cols = 28, 28
+# number of convolutional filters to use
+nb_filters = 32
+# size of pooling area for max pooling
+nb_pool = 2
+# convolution kernel size
+nb_conv = 3
+
+
+def train_model(model, train, test, nb_classes):
+    X_train = train[0].reshape(train[0].shape[0], 1, img_rows, img_cols)
+    X_test = test[0].reshape(test[0].shape[0], 1, img_rows, img_cols)
+    X_train = X_train.astype('float32')
+    X_test = X_test.astype('float32')
+    X_train /= 255
+    X_test /= 255
+    print('X_train shape:', X_train.shape)
+    print(X_train.shape[0], 'train samples')
+    print(X_test.shape[0], 'test samples')
+
+    # convert class vectors to binary class matrices
+    Y_train = np_utils.to_categorical(train[1], nb_classes)
+    Y_test = np_utils.to_categorical(test[1], nb_classes)
+
+    model.compile(loss='categorical_crossentropy', optimizer='adadelta')
+
+    t = now()
+    model.fit(X_train, Y_train,
+              batch_size=batch_size, nb_epoch=nb_epoch,
+              show_accuracy=True, verbose=1,
+              validation_data=(X_test, Y_test))
+    print('Training time: %s' % (now() - t))
+    score = model.evaluate(X_test, Y_test, show_accuracy=True, verbose=0)
+    print('Test score:', score[0])
+    print('Test accuracy:', score[1])
+
+
+# the data, shuffled and split between train and test sets
+(X_train, y_train), (X_test, y_test) = mnist.load_data()
+
+# create two datasets one with digits below 5 and one with 5 and above
+X_train_lt5 = X_train[y_train < 5]
+y_train_lt5 = y_train[y_train < 5]
+X_test_lt5 = X_test[y_test < 5]
+y_test_lt5 = y_test[y_test < 5]
+
+X_train_gte5 = X_train[y_train >= 5]
+y_train_gte5 = y_train[y_train >= 5] - 5  # make classes start at 0 for
+X_test_gte5 = X_test[y_test >= 5]         # np_utils.to_categorical
+y_test_gte5 = y_test[y_test >= 5] - 5
+
+# define two groups of layers: feature (convolutions) and classification (dense)
+feature_layers = [
+    Convolution2D(nb_filters, nb_conv, nb_conv,
+                  border_mode='valid',
+                  input_shape=(1, img_rows, img_cols)),
+    Activation('relu'),
+    Convolution2D(nb_filters, nb_conv, nb_conv),
+    Activation('relu'),
+    MaxPooling2D(pool_size=(nb_pool, nb_pool)),
+    Dropout(0.25),
+    Flatten(),
+]
+classification_layers = [
+    Dense(128),
+    Activation('relu'),
+    Dropout(0.5),
+    Dense(nb_classes),
+    Activation('softmax')
+]
+
+# create complete model
+model = Sequential()
+for l in feature_layers + classification_layers:
+    model.add(l)
+
+# train model for 5-digit classification [0..4]
+train_model(model,
+            (X_train_lt5, y_train_lt5),
+            (X_test_lt5, y_test_lt5), nb_classes)
+
+# freeze feature layers and rebuild model
+for l in feature_layers:
+    l.trainable = False
+
+# transfer: train dense layers for new classification task [5..9]
+train_model(model,
+            (X_train_gte5, y_train_gte5),
+            (X_test_gte5, y_test_gte5), nb_classes)

examples/neural_style_transfer.py

@@ -0,0 +1,290 @@
+'''Neural style transfer with Keras.
+
+Before running this script, download the weights for the VGG16 model at:
+https://drive.google.com/file/d/0Bz7KyqmuGsilT0J5dmRCM0ROVHc/view?usp=sharing
+(source: https://gist.github.com/baraldilorenzo/07d7802847aaad0a35d3)
+and make sure the variable `weights_path` in this script matches the location of the file.
+
+Run the script with:
+```
+python neural_style_transfer.py path_to_your_base_image.jpg path_to_your_reference.jpg prefix_for_results
+```
+e.g.:
+```
+python neural_style_transfer.py img/tuebingen.jpg img/starry_night.jpg results/my_result
+```
+
+It is preferrable to run this script on GPU, for speed.
+If running on CPU, prefer the TensorFlow backend (much faster).
+
+Example result: https://twitter.com/fchollet/status/686631033085677568
+
+# Details
+
+Style transfer consists in generating an image
+with the same "content" as a base image, but with the
+"style" of a different picture (typically artistic).
+
+This is achieved through the optimization of a loss function
+that has 3 components: "style loss", "content loss",
+and "total variation loss":
+
+- The total variation loss imposes local spatial continuity between
+the pixels of the combination image, giving it visual coherence.
+
+- The style loss is where the deep learning keeps in --that one is defined
+using a deep convolutional neural network. Precisely, it consists in a sum of
+L2 distances betwen the Gram matrices of the representations of
+the base image and the style reference image, extracted from
+different layers of a convnet (trained on ImageNet). The general idea
+is to capture color/texture information at different spatial
+scales (fairly large scales --defined by the depth of the layer considered).
+
+ - The content loss is a L2 distance between the features of the base
+image (extracted from a deep layer) and the features of the combination image,
+keeping the generated image close enough to the original one.
+
+# References
+    - [A Neural Algorithm of Artistic Style](http://arxiv.org/abs/1508.06576)
+'''
+
+from __future__ import print_function
+from scipy.misc import imread, imresize, imsave
+import numpy as np
+from scipy.optimize import fmin_l_bfgs_b
+import time
+import os
+import argparse
+import h5py
+
+from keras.models import Sequential
+from keras.layers.convolutional import Convolution2D, ZeroPadding2D, MaxPooling2D
+from keras import backend as K
+
+parser = argparse.ArgumentParser(description='Neural style transfer with Keras.')
+parser.add_argument('base_image_path', metavar='base', type=str,
+                    help='Path to the image to transform.')
+parser.add_argument('style_reference_image_path', metavar='ref', type=str,
+                    help='Path to the style reference image.')
+parser.add_argument('result_prefix', metavar='res_prefix', type=str,
+                    help='Prefix for the saved results.')
+
+args = parser.parse_args()
+base_image_path = args.base_image_path
+style_reference_image_path = args.style_reference_image_path
+result_prefix = args.result_prefix
+weights_path = 'vgg16_weights.h5'
+
+# these are the weights of the different loss components
+total_variation_weight = 1.
+style_weight = 1.
+content_weight = 0.025
+
+# dimensions of the generated picture.
+img_width = 400
+img_height = 400
+assert img_height == img_width, 'Due to the use of the Gram matrix, width and height must match.'
+
+# util function to open, resize and format pictures into appropriate tensors
+def preprocess_image(image_path):
+    img = imresize(imread(image_path), (img_width, img_height))
+    img = img.transpose((2, 0, 1)).astype('float64')
+    img = np.expand_dims(img, axis=0)
+    return img
+
+# util function to convert a tensor into a valid image
+def deprocess_image(x):
+    x = x.transpose((1, 2, 0))
+    x = np.clip(x, 0, 255).astype('uint8')
+    return x
+
+# get tensor representations of our images
+base_image = K.variable(preprocess_image(base_image_path))
+style_reference_image = K.variable(preprocess_image(style_reference_image_path))
+
+# this will contain our generated image
+combination_image = K.placeholder((1, 3, img_width, img_height))
+
+# combine the 3 images into a single Keras tensor
+input_tensor = K.concatenate([base_image,
+                              style_reference_image,
+                              combination_image], axis=0)
+
+# build the VGG16 network with our 3 images as input
+first_layer = ZeroPadding2D((1, 1), input_shape=(3, img_width, img_height))
+first_layer.input = input_tensor
+
+model = Sequential()
+model.add(first_layer)
+model.add(Convolution2D(64, 3, 3, activation='relu', name='conv1_1'))
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(64, 3, 3, activation='relu'))
+model.add(MaxPooling2D((2, 2), strides=(2, 2)))
+
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(128, 3, 3, activation='relu', name='conv2_1'))
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(128, 3, 3, activation='relu'))
+model.add(MaxPooling2D((2, 2), strides=(2, 2)))
+
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(256, 3, 3, activation='relu', name='conv3_1'))
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(256, 3, 3, activation='relu'))
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(256, 3, 3, activation='relu'))
+model.add(MaxPooling2D((2, 2), strides=(2, 2)))
+
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(512, 3, 3, activation='relu', name='conv4_1'))
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(512, 3, 3, activation='relu', name='conv4_2'))
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(512, 3, 3, activation='relu'))
+model.add(MaxPooling2D((2, 2), strides=(2, 2)))
+
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(512, 3, 3, activation='relu', name='conv5_1'))
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(512, 3, 3, activation='relu'))
+model.add(ZeroPadding2D((1, 1)))
+model.add(Convolution2D(512, 3, 3, activation='relu'))
+model.add(MaxPooling2D((2, 2), strides=(2, 2)))
+
+# load the weights of the VGG16 networks
+# (trained on ImageNet, won the ILSVRC competition in 2014)
+# note: when there is a complete match between your model definition
+# and your weight savefile, you can simply call model.load_weights(filename)
+assert os.path.exists(weights_path), 'Model weights not found (see "weights_path" variable in script).'
+f = h5py.File(weights_path)
+for k in range(f.attrs['nb_layers']):
+    if k >= len(model.layers):
+        # we don't look at the last (fully-connected) layers in the savefile
+        break
+    g = f['layer_{}'.format(k)]
+    weights = [g['param_{}'.format(p)] for p in range(g.attrs['nb_params'])]
+    model.layers[k].set_weights(weights)
+f.close()
+print('Model loaded.')
+
+# get the symbolic outputs of each "key" layer (we gave them unique names).
+outputs_dict = dict([(layer.name, layer.get_output()) for layer in model.layers])
+
+# compute the neural style loss
+# first we need to define 4 util functions
+
+# the gram matrix of an image tensor (feature-wise outer product)
+def gram_matrix(x):
+    assert K.ndim(x) == 3
+    features = K.batch_flatten(x)
+    gram = K.dot(features, K.transpose(features))
+    return gram
+
+# the "style loss" is designed to maintain
+# the style of the reference image in the generated image.
+# It is based on the gram matrices (which capture style) of
+# feature maps from the style reference image
+# and from the generated image
+def style_loss(style, combination):
+    assert K.ndim(style) == 3
+    assert K.ndim(combination) == 3
+    S = gram_matrix(style)
+    C = gram_matrix(combination)
+    channels = 3
+    size = img_width * img_height
+    return K.sum(K.square(S - C)) / (4. * (channels ** 2) * (size ** 2))
+
+# an auxiliary loss function
+# designed to maintain the "content" of the
+# base image in the generated image
+def content_loss(base, combination):
+    return K.sum(K.square(combination - base))
+
+# the 3rd loss function, total variation loss,
+# designed to keep the generated image locally coherent
+def total_variation_loss(x):
+    assert K.ndim(x) == 4
+    a = K.square(x[:, :, :img_width-1, :img_height-1] - x[:, :, 1:, :img_height-1])
+    b = K.square(x[:, :, :img_width-1, :img_height-1] - x[:, :, :img_width-1, 1:])
+    return K.sum(K.pow(a + b, 1.25))
+
+# combine these loss functions into a single scalar
+loss = K.variable(0.)
+layer_features = outputs_dict['conv4_2']
+base_image_features = layer_features[0, :, :, :]
+combination_features = layer_features[2, :, :, :]
+loss += content_weight * content_loss(base_image_features,
+                                      combination_features)
+
+feature_layers = ['conv1_1', 'conv2_1', 'conv3_1', 'conv4_1', 'conv5_1']
+for layer_name in feature_layers:
+    layer_features = outputs_dict[layer_name]
+    style_reference_features = layer_features[1, :, :, :]
+    combination_features = layer_features[2, :, :, :]
+    sl = style_loss(style_reference_features, combination_features)
+    loss += (style_weight / len(feature_layers)) * sl
+loss += total_variation_weight * total_variation_loss(combination_image)
+
+# get the gradients of the generated image wrt the loss
+grads = K.gradients(loss, combination_image)
+
+outputs = [loss]
+if type(grads) in {list, tuple}:
+    outputs += grads
+else:
+    outputs.append(grads)
+
+f_outputs = K.function([combination_image], outputs)
+def eval_loss_and_grads(x):
+    x = x.reshape((1, 3, img_width, img_height))
+    outs = f_outputs([x])
+    loss_value = outs[0]
+    if len(outs[1:]) == 1:
+        grad_values = outs[1].flatten().astype('float64')
+    else:
+        grad_values = np.array(outs[1:]).flatten().astype('float64')
+    return loss_value, grad_values
+
+# this Evaluator class makes it possible
+# to compute loss and gradients in one pass
+# while retrieving them via two separate functions,
+# "loss" and "grads". This is done because scipy.optimize
+# requires separate functions for loss and gradients,
+# but computing them separately would be inefficient.
+class Evaluator(object):
+    def __init__(self):
+        self.loss_value = None
+        self.grads_values = None
+
+    def loss(self, x):
+        assert self.loss_value is None
+        loss_value, grad_values = eval_loss_and_grads(x)
+        self.loss_value = loss_value
+        self.grad_values = grad_values
+        return self.loss_value
+
+    def grads(self, x):
+        assert self.loss_value is not None
+        grad_values = np.copy(self.grad_values)
+        self.loss_value = None
+        self.grad_values = None
+        return grad_values
+
+evaluator = Evaluator()
+
+# run scipy-based optimization (L-BFGS) over the pixels of the generated image
+# so as to minimize the neural style loss
+x = np.random.uniform(0, 255, (1, 3, img_width, img_height))
+for i in range(10):
+    print('Start of iteration', i)
+    start_time = time.time()
+    x, min_val, info = fmin_l_bfgs_b(evaluator.loss, x.flatten(),
+                                     fprime=evaluator.grads, maxfun=20)
+    print('Current loss value:', min_val)
+    # save current generated image
+    img = deprocess_image(x.reshape((3, img_width, img_height)))
+    fname = result_prefix + '_at_iteration_%d.png' % i
+    imsave(fname, img)
+    end_time = time.time()
+    print('Image saved as', fname)
+    print('Iteration %d completed in %ds' % (i, end_time - start_time))

examples/reuters_mlp.py

@@ -1,6 +1,13 @@
-from __future__ import absolute_import
+'''Train and evaluate a simple MLP on the Reuters newswire topic classification task.
+GPU run command:
+    THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 python examples/reuters_mlp.py
+CPU run command:
+    python examples/reuters_mlp.py
+'''
+
 from __future__ import print_function
 import numpy as np
+np.random.seed(1337)  # for reproducibility
 
 from keras.datasets import reuters
 from keras.models import Sequential
@@ -9,18 +16,11 @@ from keras.layers.normalization import BatchNormalization
 from keras.utils import np_utils
 from keras.preprocessing.text import Tokenizer
 
-'''
-    Train and evaluate a simple MLP on the Reuters newswire topic classification task.
-    GPU run command:
-        THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 python examples/reuters_mlp.py
-    CPU run command:
-        python examples/reuters_mlp.py
-'''
-
 max_words = 1000
 batch_size = 32
+nb_epoch = 5
 
-print("Loading data...")
+print('Loading data...')
 (X_train, y_train), (X_test, y_test) = reuters.load_data(nb_words=max_words, test_split=0.2)
 print(len(X_train), 'train sequences')
 print(len(X_test), 'test sequences')
@@ -28,30 +28,30 @@ print(len(X_test), 'test sequences')
 nb_classes = np.max(y_train)+1
 print(nb_classes, 'classes')
 
-print("Vectorizing sequence data...")
+print('Vectorizing sequence data...')
 tokenizer = Tokenizer(nb_words=max_words)
-X_train = tokenizer.sequences_to_matrix(X_train, mode="binary")
-X_test = tokenizer.sequences_to_matrix(X_test, mode="binary")
+X_train = tokenizer.sequences_to_matrix(X_train, mode='binary')
+X_test = tokenizer.sequences_to_matrix(X_test, mode='binary')
 print('X_train shape:', X_train.shape)
 print('X_test shape:', X_test.shape)
 
-print("Convert class vector to binary class matrix (for use with categorical_crossentropy)")
+print('Convert class vector to binary class matrix (for use with categorical_crossentropy)')
 Y_train = np_utils.to_categorical(y_train, nb_classes)
 Y_test = np_utils.to_categorical(y_test, nb_classes)
 print('Y_train shape:', Y_train.shape)
 print('Y_test shape:', Y_test.shape)
 
-print("Building model...")
+print('Building model...')
 model = Sequential()
-model.add(Dense(max_words, 256, init='normal'))
+model.add(Dense(512, input_shape=(max_words,)))
 model.add(Activation('relu'))
 model.add(Dropout(0.5))
-model.add(Dense(256, nb_classes, init='normal'))
+model.add(Dense(nb_classes))
 model.add(Activation('softmax'))
 
 model.compile(loss='categorical_crossentropy', optimizer='adam')
 
-history = model.fit(X_train, Y_train, nb_epoch=4, batch_size=batch_size, verbose=1, show_accuracy=True, validation_split=0.1)
+history = model.fit(X_train, Y_train, nb_epoch=nb_epoch, batch_size=batch_size, verbose=1, show_accuracy=True, validation_split=0.1)
 score = model.evaluate(X_test, Y_test, batch_size=batch_size, verbose=1, show_accuracy=True)
 print('Test score:', score[0])
 print('Test accuracy:', score[1])

examples/skipgram_word_embeddings.py

@@ -1,215 +0,0 @@
-
-'''
-    We loop over words in a dataset, and for each word, we look at a context window around the word. 
-    We generate pairs of (pivot_word, other_word_from_same_context) with label 1,
-    and pairs of (pivot_word, random_word) with label 0 (skip-gram method).
-
-    We use the layer WordContextProduct to learn embeddings for the word couples,
-    and compute a proximity score between the embeddings (= p(context|word)),
-    trained with our positive and negative labels.
-
-    We then use the weights computed by WordContextProduct to encode words 
-    and demonstrate that the geometry of the embedding space 
-    captures certain useful semantic properties.
-
-    Read more about skip-gram in this particularly gnomic paper by Mikolov et al.: 
-        http://arxiv.org/pdf/1301.3781v3.pdf
-
-    Note: you should run this on GPU, otherwise training will be quite slow. 
-    On a EC2 GPU instance, expect 3 hours per 10e6 comments (~10e8 words) per epoch with dim_proj=256.
-    Should be much faster on a modern GPU.
-
-    GPU command:
-        THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 python skipgram_word_embeddings.py
-
-    Dataset: 5,845,908 Hacker News comments. 
-    Obtain the dataset at: 
-        https://mega.co.nz/#F!YohlwD7R!wec0yNO86SeaNGIYQBOR0A 
-        (HNCommentsAll.1perline.json.bz2)
-'''
-from __future__ import absolute_import
-from __future__ import print_function
-
-import numpy as np
-import theano
-import six.moves.cPickle
-import os, re, json
-
-from keras.preprocessing import sequence, text
-from keras.optimizers import SGD, RMSprop, Adagrad
-from keras.utils import np_utils, generic_utils
-from keras.models import Sequential
-from keras.layers.embeddings import WordContextProduct, Embedding
-from six.moves import range
-from six.moves import zip
-
-max_features = 50000 # vocabulary size: top 50,000 most common words in data
-skip_top = 100 # ignore top 100 most common words
-nb_epoch = 1
-dim_proj = 256 # embedding space dimension
-
-save = True
-load_model = False
-load_tokenizer = False
-train_model = True
-save_dir = os.path.expanduser("~/.keras/models")
-model_load_fname = "HN_skipgram_model.pkl"
-model_save_fname = "HN_skipgram_model.pkl"
-tokenizer_fname = "HN_tokenizer.pkl"
-
-data_path = os.path.expanduser("~/")+"HNCommentsAll.1perline.json"
-
-# text preprocessing utils
-html_tags = re.compile(r'<.*?>')
-to_replace = [('&#x27;', "'")]
-hex_tags = re.compile(r'&.*?;')
-
-def clean_comment(comment):
-    c = str(comment.encode("utf-8"))
-    c = html_tags.sub(' ', c)
-    for tag, char in to_replace:
-        c = c.replace(tag, char)
-    c = hex_tags.sub(' ', c)
-    return c
-
-def text_generator(path=data_path):
-    f = open(path)
-    for i, l in enumerate(f):
-        comment_data = json.loads(l)
-        comment_text = comment_data["comment_text"]
-        comment_text = clean_comment(comment_text)
-        if i % 10000 == 0:
-            print(i)
-        yield comment_text
-    f.close()
-
-# model management
-if load_tokenizer:
-    print('Load tokenizer...')
-    tokenizer = six.moves.cPickle.load(open(os.path.join(save_dir, tokenizer_fname), 'rb'))
-else:
-    print("Fit tokenizer...")
-    tokenizer = text.Tokenizer(nb_words=max_features)
-    tokenizer.fit_on_texts(text_generator())
-    if save:
-        print("Save tokenizer...")
-        if not os.path.exists(save_dir):
-            os.makedirs(save_dir)
-        six.moves.cPickle.dump(tokenizer, open(os.path.join(save_dir, tokenizer_fname), "wb"))
-
-# training process
-if train_model:
-    if load_model:
-        print('Load model...')
-        model = six.moves.cPickle.load(open(os.path.join(save_dir, model_load_fname), 'rb'))
-    else:
-        print('Build model...')
-        model = Sequential()
-        model.add(WordContextProduct(max_features, proj_dim=dim_proj, init="uniform"))
-        model.compile(loss='mse', optimizer='rmsprop')
-
-    sampling_table = sequence.make_sampling_table(max_features)
-
-    for e in range(nb_epoch):
-        print('-'*40)
-        print('Epoch', e)
-        print('-'*40)
-
-        progbar = generic_utils.Progbar(tokenizer.document_count)
-        samples_seen = 0
-        losses = []
-        
-        for i, seq in enumerate(tokenizer.texts_to_sequences_generator(text_generator())):
-            # get skipgram couples for one text in the dataset
-            couples, labels = sequence.skipgrams(seq, max_features, window_size=4, negative_samples=1., sampling_table=sampling_table)
-            if couples:
-                # one gradient update per sentence (one sentence = a few 1000s of word couples)
-                X = np.array(couples, dtype="int32")
-                loss = model.train(X, labels)
-                losses.append(loss)
-                if len(losses) % 100 == 0:
-                    progbar.update(i, values=[("loss", np.mean(losses))])
-                    losses = []
-                samples_seen += len(labels)
-        print('Samples seen:', samples_seen)
-    print("Training completed!")
-
-    if save:
-        print("Saving model...")
-        if not os.path.exists(save_dir):
-            os.makedirs(save_dir)
-        six.moves.cPickle.dump(model, open(os.path.join(save_dir, model_save_fname), "wb"))
-
-
-print("It's test time!")
-
-# recover the embedding weights trained with skipgram:
-weights = model.layers[0].get_weights()[0]
-
-# we no longer need this
-del model
-
-weights[:skip_top] = np.zeros((skip_top, dim_proj))
-norm_weights = np_utils.normalize(weights)
-
-word_index = tokenizer.word_index
-reverse_word_index = dict([(v, k) for k, v in list(word_index.items())])
-word_index = tokenizer.word_index
-
-def embed_word(w):
-    i = word_index.get(w)
-    if (not i) or (i<skip_top) or (i>=max_features):
-        return None
-    return norm_weights[i]
-
-def closest_to_point(point, nb_closest=10):
-    proximities = np.dot(norm_weights, point)
-    tups = list(zip(list(range(len(proximities))), proximities))
-    tups.sort(key=lambda x: x[1], reverse=True)
-    return [(reverse_word_index.get(t[0]), t[1]) for t in tups[:nb_closest]]  
-
-def closest_to_word(w, nb_closest=10):
-    i = word_index.get(w)
-    if (not i) or (i<skip_top) or (i>=max_features):
-        return []
-    return closest_to_point(norm_weights[i].T, nb_closest)
-
-
-''' the resuls in comments below were for: 
-    5.8M HN comments
-    dim_proj = 256
-    nb_epoch = 2
-    optimizer = rmsprop
-    loss = mse
-    max_features = 50000
-    skip_top = 100
-    negative_samples = 1.
-    window_size = 4
-    and frequency subsampling of factor 10e-5. 
-'''
-
-words = ["article", # post, story, hn, read, comments
-"3", # 6, 4, 5, 2
-"two", # three, few, several, each
-"great", # love, nice, working, looking
-"data", # information, memory, database
-"money", # company, pay, customers, spend
-"years", # ago, year, months, hours, week, days
-"android", # ios, release, os, mobile, beta
-"javascript", # js, css, compiler, library, jquery, ruby
-"look", # looks, looking
-"business", # industry, professional, customers
-"company", # companies, startup, founders, startups
-"after", # before, once, until
-"own", # personal, our, having
-"us", # united, country, american, tech, diversity, usa, china, sv
-"using", # javascript, js, tools (lol)
-"here", # hn, post, comments
-]
-
-for w in words:
-    res = closest_to_word(w)
-    print('====', w)
-    for r in res:
-        print(r)
-

examples/stateful_lstm.py

@@ -0,0 +1,85 @@
+'''Example script showing how to use stateful RNNs
+to model long sequences efficiently.
+'''
+from __future__ import print_function
+import numpy as np
+import matplotlib.pyplot as plt
+from keras.models import Sequential
+from keras.layers.core import Dense
+from keras.layers.recurrent import LSTM
+
+
+# since we are using stateful rnn tsteps can be set to 1
+tsteps = 1
+batch_size = 25
+epochs = 25
+# number of elements ahead that are used to make the prediction
+lahead = 1
+
+
+def gen_cosine_amp(amp=100, period=25, x0=0, xn=50000, step=1, k=0.0001):
+    """Generates an absolute cosine time series with the amplitude
+    exponentially decreasing
+
+    Arguments:
+        amp: amplitude of the cosine function
+        period: period of the cosine function
+        x0: initial x of the time series
+        xn: final x of the time series
+        step: step of the time series discretization
+        k: exponential rate
+    """
+    cos = np.zeros(((xn - x0) * step, 1, 1))
+    for i in range(len(cos)):
+        idx = x0 + i * step
+        cos[i, 0, 0] = amp * np.cos(idx / (2 * np.pi * period))
+        cos[i, 0, 0] = cos[i, 0, 0] * np.exp(-k * idx)
+    return cos
+
+
+print('Generating Data')
+cos = gen_cosine_amp()
+print('Input shape:', cos.shape)
+
+expected_output = np.zeros((len(cos), 1))
+for i in range(len(cos) - lahead):
+    expected_output[i, 0] = np.mean(cos[i + 1:i + lahead + 1])
+
+print('Output shape')
+print(expected_output.shape)
+
+print('Creating Model')
+model = Sequential()
+model.add(LSTM(50,
+               batch_input_shape=(batch_size, tsteps, 1),
+               return_sequences=True,
+               stateful=True))
+model.add(LSTM(50,
+               batch_input_shape=(batch_size, tsteps, 1),
+               return_sequences=False,
+               stateful=True))
+model.add(Dense(1))
+model.compile(loss='mse', optimizer='rmsprop')
+
+print('Training')
+for i in range(epochs):
+    print('Epoch', i, '/', epochs)
+    model.fit(cos,
+              expected_output,
+              batch_size=batch_size,
+              verbose=1,
+              nb_epoch=1,
+              shuffle=False)
+    model.reset_states()
+
+print('Predicting')
+predicted_output = model.predict(cos, batch_size=batch_size)
+
+print('Ploting Results')
+plt.subplot(2, 1, 1)
+plt.plot(expected_output)
+plt.title('Expected')
+plt.subplot(2, 1, 2)
+plt.plot(predicted_output)
+plt.title('Predicted')
+plt.show()

keras/__init__.py

@@ -0,0 +1 @@
+__version__ = '0.3.3'

keras/activations.py

@@ -1,34 +1,47 @@
 from __future__ import absolute_import
-import theano
-import theano.tensor as T
-import types
+from . import backend as K
+
 
 def softmax(x):
-    return T.nnet.softmax(x)
+    ndim = K.ndim(x)
+    if ndim == 2:
+        return K.softmax(x)
+    elif ndim == 3:
+        e = K.exp(x - K.max(x, axis=-1, keepdims=True))
+        s = K.sum(e, axis=-1, keepdims=True)
+        return e / s
+    else:
+        raise Exception('Cannot apply softmax to a tensor that is not 2D or 3D. ' +
+                        'Here, ndim=' + str(ndim))
 
-def time_distributed_softmax(x):
-    xshape = x.shape
-    X = x.reshape((xshape[0] * xshape[1], xshape[2]))
-    return T.nnet.softmax(X).reshape(xshape)
 
 def softplus(x):
-    return T.nnet.softplus(x)
+    return K.softplus(x)
+
+
+def relu(x, alpha=0., max_value=None):
+    return K.relu(x, alpha=alpha, max_value=max_value)
 
-def relu(x):
-    return (x + abs(x)) / 2.0
 
 def tanh(x):
-    return T.tanh(x)
+    return K.tanh(x)
+
 
 def sigmoid(x):
-    return T.nnet.sigmoid(x)
+    return K.sigmoid(x)
+
 
 def hard_sigmoid(x):
-    return T.nnet.hard_sigmoid(x)
+    return K.hard_sigmoid(x)
+
 
 def linear(x):
+    '''
+    The function returns the variable that is passed in, so all types work.
+    '''
     return x
 
+
 from .utils.generic_utils import get_from_module
 def get(identifier):
-    return get_from_module(identifier, globals(), 'activation function')
+    return get_from_module(identifier, globals(), 'activation function')

keras/backend/__init__.py

@@ -0,0 +1,51 @@
+from __future__ import absolute_import
+from __future__ import print_function
+import os
+import json
+import sys
+from .common import epsilon, floatx, set_epsilon, set_floatx
+
+_keras_base_dir = os.path.expanduser('~')
+if not os.access(_keras_base_dir, os.W_OK):
+    _keras_base_dir = '/tmp'
+
+_keras_dir = os.path.join(_keras_base_dir, '.keras')
+if not os.path.exists(_keras_dir):
+    os.makedirs(_keras_dir)
+
+_BACKEND = 'theano'
+_config_path = os.path.expanduser(os.path.join(_keras_dir, 'keras.json'))
+if os.path.exists(_config_path):
+    _config = json.load(open(_config_path))
+    _floatx = _config.get('floatx', floatx())
+    assert _floatx in {'float16', 'float32', 'float64'}
+    _epsilon = _config.get('epsilon', epsilon())
+    assert type(_epsilon) == float
+    _backend = _config.get('backend', _BACKEND)
+    assert _backend in {'theano', 'tensorflow'}
+
+    set_floatx(_floatx)
+    set_epsilon(_epsilon)
+    _BACKEND = _backend
+else:
+    # save config file, for easy edition
+    _config = {'floatx': floatx(),
+               'epsilon': epsilon(),
+               'backend': _BACKEND}
+    with open(_config_path, 'w') as f:
+        # add new line in order for bash 'cat' display the content correctly
+        f.write(json.dumps(_config) + '\n')
+
+if 'KERAS_BACKEND' in os.environ:
+    _backend = os.environ['KERAS_BACKEND']
+    assert _backend in {'theano', 'tensorflow'}
+    _BACKEND = _backend
+
+if _BACKEND == 'theano':
+    sys.stderr.write('Using Theano backend.\n')
+    from .theano_backend import *
+elif _BACKEND == 'tensorflow':
+    sys.stderr.write('Using TensorFlow backend.\n')
+    from .tensorflow_backend import *
+else:
+    raise Exception('Unknown backend: ' + str(_BACKEND))

keras/backend/common.py

@@ -0,0 +1,32 @@
+import numpy as np
+
+# the type of float to use throughout the session.
+_FLOATX = 'float32'
+_EPSILON = 10e-8
+
+
+def epsilon():
+    return _EPSILON
+
+
+def set_epsilon(e):
+    global _EPSILON
+    _EPSILON = e
+
+
+def floatx():
+    return _FLOATX
+
+
+def set_floatx(floatx):
+    global _FLOATX
+    if floatx not in {'float16', 'float32', 'float64'}:
+        raise Exception('Unknown floatx type: ' + str(floatx))
+    floatx = str(floatx)
+    _FLOATX = floatx
+
+
+def cast_to_floatx(x):
+    '''Cast a Numpy array to floatx.
+    '''
+    return np.asarray(x, dtype=_FLOATX)

keras/backend/tensorflow_backend.py

@@ -0,0 +1,754 @@
+import tensorflow as tf
+import numpy as np
+import os
+import warnings
+from .common import _FLOATX, _EPSILON
+
+# INTERNAL UTILS
+
+_SESSION = None
+
+
+def get_session():
+    global _SESSION
+    if _SESSION is None:
+        if not os.environ.get('OMP_NUM_THREADS'):
+            _SESSION = tf.Session(config=tf.ConfigProto(allow_soft_placement=True))
+        else:
+            nb_thread = int(os.environ.get('OMP_NUM_THREADS'))
+            _SESSION = tf.Session(config=tf.ConfigProto(intra_op_parallelism_threads=nb_thread, allow_soft_placement=True))
+    return _SESSION
+
+
+def set_session(session):
+    global _SESSION
+    _SESSION = session
+
+
+# VARIABLE MANIPULATION
+
+def variable(value, dtype=_FLOATX, name=None):
+    v = tf.Variable(np.asarray(value, dtype=dtype), name=name)
+    get_session().run(v.initializer)
+    return v
+
+
+def placeholder(shape=None, ndim=None, dtype=_FLOATX, name=None):
+    if not shape:
+        if ndim:
+            shape = [None for _ in range(ndim)]
+    return tf.placeholder(dtype, shape=shape, name=name)
+
+
+def shape(x):
+    # symbolic shape
+    return tf.shape(x)
+
+
+def int_shape(x):
+    shape = x.get_shape()
+    return tuple([i.__int__() for i in shape])
+
+
+def ndim(x):
+    return len(x.get_shape())
+
+
+def eval(x):
+    '''Run a graph.
+    '''
+    return x.eval(session=get_session())
+
+
+def zeros(shape, dtype=_FLOATX, name=None):
+    return variable(np.zeros(shape), dtype, name)
+
+
+def ones(shape, dtype=_FLOATX, name=None):
+    return variable(np.ones(shape), dtype, name)
+
+
+def ones_like(x, name=None):
+    return tf.ones_like(x, name=name)
+
+
+def zeros_like(x, name=None):
+    return tf.zeros_like(x, name=name)
+
+
+def count_params(x):
+    '''Return number of scalars in a tensor.
+    '''
+    shape = x.get_shape()
+    return np.prod([shape[i]._value for i in range(len(shape))])
+
+
+def cast(x, dtype):
+    return tf.cast(x, dtype)
+
+
+# LINEAR ALGEBRA
+
+def dot(x, y):
+    return tf.matmul(x, y)
+
+
+def batch_dot(x, y, axes=None):
+    if axes:
+        adj_x = None if axes[0][0] == ndim(x)-1 else True
+        adj_y = True if axes[1][0] == ndim(y)-1 else None
+    else:
+        adj_x = None
+        adj_y = None
+    return tf.batch_matmul(x, y, adj_x=adj_x, adj_y=adj_y)
+
+
+def transpose(x):
+    return tf.transpose(x)
+
+
+def gather(reference, indices):
+    '''
+    # Arguments
+        reference: a tensor.
+        indices: an int tensor of indices.
+
+    # Returns
+        a tensor of same type as `reference`.
+    '''
+    return tf.gather(reference, indices)
+
+
+# ELEMENT-WISE OPERATIONS
+
+def normalize_axis(axis, ndim):
+    if type(axis) is tuple:
+        axis = list(axis)
+    if type(axis) is list:
+        for i, a in enumerate(axis):
+            if a is not None and a < 0:
+                axis[i] = a % ndim
+    else:
+        if axis is not None and axis < 0:
+            axis = axis % ndim
+    return axis
+
+
+def max(x, axis=None, keepdims=False):
+    axis = normalize_axis(axis, ndim(x))
+    return tf.reduce_max(x, reduction_indices=axis, keep_dims=keepdims)
+
+
+def min(x, axis=None, keepdims=False):
+    axis = normalize_axis(axis, ndim(x))
+    return tf.reduce_min(x, reduction_indices=axis, keep_dims=keepdims)
+
+
+def sum(x, axis=None, keepdims=False):
+    '''Sum of the values in a tensor, alongside the specified axis.
+    '''
+    axis = normalize_axis(axis, ndim(x))
+    return tf.reduce_sum(x, reduction_indices=axis, keep_dims=keepdims)
+
+
+def prod(x, axis=None, keepdims=False):
+    '''Multiply the values in a tensor, alongside the specified axis.
+    '''
+    axis = normalize_axis(axis, ndim(x))
+    return tf.reduce_prod(x, reduction_indices=axis, keep_dims=keepdims)
+
+
+def std(x, axis=None, keepdims=False):
+    axis = normalize_axis(axis, ndim(x))
+    if x.dtype.base_dtype == tf.bool:
+        x = tf.cast(x, _FLOATX)
+    m = tf.reduce_mean(x, reduction_indices=axis, keep_dims=True)
+    devs_squared = tf.square(x - m)
+    return tf.sqrt(tf.reduce_mean(devs_squared,
+                                  reduction_indices=axis,
+                                  keep_dims=keepdims))
+
+
+def mean(x, axis=None, keepdims=False):
+    axis = normalize_axis(axis, ndim(x))
+    if x.dtype.base_dtype == tf.bool:
+        x = tf.cast(x, _FLOATX)
+    return tf.reduce_mean(x, reduction_indices=axis, keep_dims=keepdims)
+
+
+def any(x, axis=None, keepdims=False):
+    '''Bitwise reduction (logical OR).
+
+    Return array of uint8 (0s and 1s).
+    '''
+    axis = normalize_axis(axis, ndim(x))
+    x = tf.cast(x, tf.bool)
+    x = tf.reduce_any(x, reduction_indices=axis, keep_dims=keepdims)
+    return tf.cast(x, tf.uint8)
+
+
+def argmax(x, axis=-1):
+    if axis < 0:
+        axis = axis % len(x.get_shape())
+    return tf.argmax(x, axis)
+
+
+def argmin(x, axis=-1):
+    if axis < 0:
+        axis = axis % len(x.get_shape())
+    return tf.argmin(x, axis)
+
+
+def square(x):
+    return tf.square(x)
+
+
+def abs(x):
+    return tf.abs(x)
+
+
+def sqrt(x):
+    x = tf.clip_by_value(x, tf.cast(0., dtype=_FLOATX),
+                         tf.cast(np.inf, dtype=_FLOATX))
+    return tf.sqrt(x)
+
+
+def exp(x):
+    return tf.exp(x)
+
+
+def log(x):
+    return tf.log(x)
+
+
+def round(x):
+    return tf.round(x)
+
+
+def sign(x):
+    return tf.sign(x)
+
+
+def pow(x, a):
+    return tf.pow(x, a)
+
+
+def clip(x, min_value, max_value):
+    if max_value < min_value:
+        max_value = min_value
+    return tf.clip_by_value(x, tf.cast(min_value, dtype=_FLOATX),
+                            tf.cast(max_value, dtype=_FLOATX))
+
+
+def equal(x, y):
+    return tf.equal(x, y)
+
+
+def not_equal(x, y):
+    return tf.not_equal(x, y)
+
+
+def maximum(x, y):
+    return tf.maximum(x, y)
+
+
+def minimum(x, y):
+    return tf.minimum(x, y)
+
+
+# SHAPE OPERATIONS
+
+def concatenate(tensors, axis=-1):
+    if axis < 0:
+        if len(tensors[0].get_shape()):
+            axis = axis % len(tensors[0].get_shape())
+        else:
+            axis = 0
+    return tf.concat(axis, tensors)
+
+
+def reshape(x, shape):
+    return tf.reshape(x, shape)
+
+
+def permute_dimensions(x, pattern):
+    '''Transpose dimensions.
+
+    # Arguments
+        pattern: should be a tuple or list of
+            dimension indices, e.g. [0, 2, 1].
+    '''
+    return tf.transpose(x, perm=pattern)
+
+
+def resize_images(X, height_factor, width_factor, dim_ordering):
+    '''Resize the images contained in a 4D tensor of shape
+    - [batch, channels, height, width] (for 'th' dim_ordering)
+    - [batch, height, width, channels] (for 'tf' dim_ordering)
+    by a factor of (height_factor, width_factor). Both factors should be
+    positive integers.
+    '''
+    if dim_ordering == 'th':
+        new_shape = tf.shape(X)[2:]
+        new_shape *= tf.constant(np.array([height_factor, width_factor]).astype('int32'))
+        X = permute_dimensions(X, [0, 2, 3, 1])
+        X = tf.image.resize_nearest_neighbor(X, new_shape)
+        return permute_dimensions(X, [0, 3, 1, 2])
+    elif dim_ordering == 'tf':
+        new_shape = tf.shape(X)[1:3]
+        new_shape *= tf.constant(np.array([height_factor, width_factor]).astype('int32'))
+        return tf.image.resize_nearest_neighbor(X, new_shape)
+    else:
+        raise Exception('Invalid dim_ordering: ' + dim_ordering)
+
+
+def repeat_elements(x, rep, axis):
+    '''Repeats the elements of a tensor along an axis, like np.repeat
+
+    If x has shape (s1, s2, s3) and axis=1, the output
+    will have shape (s1, s2 * rep, s3)
+    '''
+    x_shape = x.get_shape().as_list()
+    # slices along the repeat axis
+    splits = tf.split(axis, x_shape[axis], x)
+    # repeat each slice the given number of reps
+    x_rep = [s for s in splits for i in range(rep)]
+    return tf.concat(axis, x_rep)
+
+
+def repeat(x, n):
+    '''Repeat a 2D tensor:
+
+    if x has shape (samples, dim) and n=2,
+    the output will have shape (samples, 2, dim)
+    '''
+    assert ndim(x) == 2
+    tensors = [x] * n
+    stacked = tf.pack(tensors)
+    return tf.transpose(stacked, (1, 0, 2))
+
+
+def tile(x, n):
+    return tf.tile(x, n)
+
+
+def flatten(x):
+    return tf.reshape(x, [-1])
+
+
+def batch_flatten(x):
+    '''Turn a n-D tensor into a 2D tensor where
+    the first dimension is conserved.
+    '''
+    x = tf.reshape(x, [-1, np.prod(x.get_shape()[1:].as_list())])
+    return x
+
+
+def expand_dims(x, dim=-1):
+    '''Add a 1-sized dimension at index "dim".
+    '''
+    return tf.expand_dims(x, dim)
+
+
+def squeeze(x, axis):
+    '''Remove a 1-dimension from the tensor at index "axis".
+    '''
+    return tf.squeeze(x, [axis])
+
+
+def temporal_padding(x, padding=1):
+    '''Pad the middle dimension of a 3D tensor
+    with "padding" zeros left and right.
+    '''
+    pattern = [[0, 0], [padding, padding], [0, 0]]
+    return tf.pad(x, pattern)
+
+
+def spatial_2d_padding(x, padding=(1, 1), dim_ordering='th'):
+    '''Pad the 2nd and 3rd dimensions of a 4D tensor
+    with "padding[0]" and "padding[1]" (resp.) zeros left and right.
+    '''
+    if dim_ordering == 'th':
+        pattern = [[0, 0], [0, 0],
+                   [padding[0], padding[0]], [padding[1], padding[1]]]
+    else:
+        pattern = [[0, 0],
+                   [padding[0], padding[0]], [padding[1], padding[1]],
+                   [0, 0]]
+    return tf.pad(x, pattern)
+
+
+def pack(x):
+    return tf.pack(x)
+
+
+# VALUE MANIPULATION
+
+
+def get_value(x):
+    '''Technically the same as eval() for TF.
+    '''
+    return x.eval(session=get_session())
+
+
+def set_value(x, value):
+    tf.assign(x, np.asarray(value)).op.run(session=get_session())
+
+
+# GRAPH MANIPULATION
+
+class Function(object):
+
+    def __init__(self, inputs, outputs, updates=[]):
+        assert type(inputs) in {list, tuple}, 'Input to a TensorFlow backend function should be a list or tuple.'
+        assert type(outputs) in {list, tuple}, 'Output to a TensorFlow backend function should be a list or tuple.'
+        assert type(updates) in {list, tuple}, 'Updates in a TensorFlow backend function should be a list or tuple.'
+        self.inputs = list(inputs)
+        self.outputs = list(outputs)
+        with tf.control_dependencies(self.outputs):
+            self.updates = [tf.assign(p, new_p) for (p, new_p) in updates]
+
+    def __call__(self, inputs):
+        assert type(inputs) in {list, tuple}
+        names = [v.name for v in self.inputs]
+        feed_dict = dict(zip(names, inputs))
+        session = get_session()
+        updated = session.run(self.outputs + self.updates, feed_dict=feed_dict)
+        return updated[:len(self.outputs)]
+
+
+def function(inputs, outputs, updates=[], **kwargs):
+    if len(kwargs) > 0:
+        msg = [
+            "Expected no kwargs, you passed %s" % len(kwargs),
+            "kwargs passed to function are ignored with Tensorflow backend"
+        ]
+        warnings.warn('\n'.join(msg))
+    return Function(inputs, outputs, updates=updates)
+
+
+def gradients(loss, variables):
+    return tf.gradients(loss, variables)
+
+
+# CONTROL FLOW
+
+def rnn(step_function, inputs, initial_states,
+        go_backwards=False, mask=None, constants=None):
+    '''Iterates over the time dimension of a tensor.
+
+    # Arguments
+        inputs: tensor of temporal data of shape (samples, time, ...)
+            (at least 3D).
+        step_function:
+            Parameters:
+                input: tensor with shape (samples, ...) (no time dimension),
+                    representing input for the batch of samples at a certain
+                    time step.
+                states: list of tensors.
+            Returns:
+                output: tensor with shape (samples, ...) (no time dimension),
+                new_states: list of tensors, same length and shapes
+                    as 'states'.
+        initial_states: tensor with shape (samples, ...) (no time dimension),
+            containing the initial values for the states used in
+            the step function.
+        go_backwards: boolean. If True, do the iteration over
+            the time dimension in reverse order.
+        mask: binary tensor with shape (samples, time, 1),
+            with a zero for every element that is masked.
+        constants: a list of constant values passed at each step.
+
+    # Returns
+        A tuple (last_output, outputs, new_states).
+            last_output: the latest output of the rnn, of shape (samples, ...)
+            outputs: tensor with shape (samples, time, ...) where each
+                entry outputs[s, t] is the output of the step function
+                at time t for sample s.
+            new_states: list of tensors, latest states returned by
+                the step function, of shape (samples, ...).
+    '''
+    ndim = len(inputs.get_shape())
+    assert ndim >= 3, "Input should be at least 3D."
+    axes = [1, 0] + list(range(2, ndim))
+    inputs = tf.transpose(inputs, (axes))
+    input_list = tf.unpack(inputs)
+    if constants is None:
+        constants = []
+
+    states = initial_states
+    successive_states = []
+    successive_outputs = []
+    if go_backwards:
+        input_list.reverse()
+
+    if mask is not None:
+        # Transpose not supported by bool tensor types, hence round-trip to uint8.
+        mask = tf.cast(mask, tf.uint8)
+        if len(mask.get_shape()) == ndim-1:
+            mask = expand_dims(mask)
+        mask = tf.cast(tf.transpose(mask, axes), tf.bool)
+        mask_list = tf.unpack(mask)
+
+        if go_backwards:
+            mask_list.reverse()
+
+        for input, mask_t in zip(input_list, mask_list):
+            output, new_states = step_function(input, states + constants)
+
+            # tf.select needs its condition tensor to be the same shape as its two
+            # result tensors, but in our case the condition (mask) tensor is
+            # (nsamples, 1), and A and B are (nsamples, ndimensions). So we need to
+            # broadcast the mask to match the shape of A and B. That's what the
+            # tile call does, is just repeat the mask along its second dimension
+            # ndimensions times.
+            tiled_mask_t = tf.tile(mask_t, tf.pack([1, tf.shape(output)[1]]))
+
+            if len(successive_outputs) == 0:
+                prev_output = zeros_like(output)
+            else:
+                prev_output = successive_outputs[-1]
+
+            output = tf.select(tiled_mask_t, output, prev_output)
+
+            return_states = []
+            for state, new_state in zip(states, new_states):
+                # (see earlier comment for tile explanation)
+                tiled_mask_t = tf.tile(mask_t, tf.pack([1, tf.shape(new_state)[1]]))
+                return_states.append(tf.select(tiled_mask_t, new_state, state))
+
+            states = return_states
+            successive_outputs.append(output)
+            successive_states.append(states)
+    else:
+        for input in input_list:
+            output, states = step_function(input, states + constants)
+            successive_outputs.append(output)
+            successive_states.append(states)
+
+    last_output = successive_outputs[-1]
+    outputs = tf.pack(successive_outputs)
+    new_states = successive_states[-1]
+
+    axes = [1, 0] + list(range(2, len(outputs.get_shape())))
+    outputs = tf.transpose(outputs, axes)
+    return last_output, outputs, new_states
+
+
+def switch(condition, then_expression, else_expression):
+    '''Switch between two operations depending on a scalar value.
+
+    # Arguments
+        condition: scalar tensor.
+        then_expression: TensorFlow operation.
+        else_expression: TensorFlow operation.
+    '''
+    return tf.python.control_flow_ops.cond(condition,
+                                           lambda: then_expression,
+                                           lambda: else_expression)
+
+
+# NN OPERATIONS
+
+def relu(x, alpha=0., max_value=None):
+    '''ReLU.
+
+    # Arguments
+        alpha: slope of negative section.
+        max_value: saturation threshold.
+    '''
+    negative_part = tf.nn.relu(-x)
+    x = tf.nn.relu(x)
+    if max_value is not None:
+        x = tf.clip_by_value(x, tf.cast(0., dtype=_FLOATX),
+                             tf.cast(max_value, dtype=_FLOATX))
+    if isinstance(alpha, (tuple, list, np.ndarray)) or np.isscalar(alpha):
+        alpha = tf.constant(alpha, dtype=_FLOATX)
+    x -= alpha * negative_part
+    return x
+
+
+def softmax(x):
+    return tf.nn.softmax(x)
+
+
+def softplus(x):
+    return tf.nn.softplus(x)
+
+
+def categorical_crossentropy(output, target, from_logits=False):
+    '''Note: tf.nn.softmax_cross_entropy_with_logits
+    expects logits, Keras expects probabilities.
+    '''
+    if not from_logits:
+        # scale preds so that the class probas of each sample sum to 1
+        output /= tf.reduce_sum(output,
+                                reduction_indices=len(output.get_shape()) - 1,
+                                keep_dims=True)
+        # manual computation of crossentropy
+        output = tf.clip_by_value(output, tf.cast(_EPSILON, dtype=_FLOATX),
+                                  tf.cast(1. - _EPSILON, dtype=_FLOATX))
+        return - tf.reduce_sum(target * tf.log(output),
+                               reduction_indices=len(output.get_shape()) - 1)
+    else:
+        return tf.nn.softmax_cross_entropy_with_logits(output, target)
+
+
+def binary_crossentropy(output, target, from_logits=False):
+    '''Note: tf.nn.sigmoid_cross_entropy_with_logits
+    expects logits, Keras expects probabilities.
+    '''
+    if not from_logits:
+        # transform back to logits
+        output = tf.clip_by_value(output, tf.cast(_EPSILON, dtype=_FLOATX),
+                                  tf.cast(1.-_EPSILON, dtype=_FLOATX))
+        output = tf.log(output / (1 - output))
+    return tf.nn.sigmoid_cross_entropy_with_logits(output, target)
+
+
+def sigmoid(x):
+    return tf.nn.sigmoid(x)
+
+
+def hard_sigmoid(x):
+    x = (0.2 * x) + 0.5
+    x = tf.clip_by_value(x, tf.cast(0., dtype=_FLOATX),
+                         tf.cast(1., dtype=_FLOATX))
+    return x
+
+
+def tanh(x):
+    return tf.nn.tanh(x)
+
+
+def dropout(x, level, seed=None):
+    retain_prob = 1. - level
+    if seed is None:
+        seed = np.random.randint(10e6)
+    # the dummy 1. works around a TF bug
+    # (float32_ref vs. float32 incomptability)
+    return tf.nn.dropout(x * 1., retain_prob, seed=seed)
+
+
+def l2_normalize(x, axis):
+    if axis < 0:
+        axis = axis % len(x.get_shape())
+    return tf.nn.l2_normalize(x, dim=axis)
+
+
+# CONVOLUTIONS
+
+
+def conv2d(x, kernel, strides=(1, 1), border_mode='valid', dim_ordering='th',
+           image_shape=None, filter_shape=None):
+    '''Runs on cuDNN if available.
+
+    # Arguments
+        border_mode: string, "same" or "valid".
+        dim_ordering: whether to use Theano or TensorFlow dimension ordering
+        in inputs/kernels/ouputs.
+    '''
+    if border_mode == 'same':
+        padding = 'SAME'
+    elif border_mode == 'valid':
+        padding = 'VALID'
+    else:
+        raise Exception('Invalid border mode: ' + str(border_mode))
+
+    strides = (1,) + strides + (1,)
+
+    if _FLOATX == 'float64':
+        # tf conv2d only supports float32
+        x = tf.cast(x, 'float32')
+        kernel = tf.cast(kernel, 'float32')
+
+    if dim_ordering == 'th':
+        # TF uses the last dimension as channel dimension,
+        # instead of the 2nd one.
+        # TH input shape: (samples, input_depth, rows, cols)
+        # TF input shape: (samples, rows, cols, input_depth)
+        # TH kernel shape: (depth, input_depth, rows, cols)
+        # TF kernel shape: (rows, cols, input_depth, depth)
+        x = tf.transpose(x, (0, 2, 3, 1))
+        kernel = tf.transpose(kernel, (2, 3, 1, 0))
+        x = tf.nn.conv2d(x, kernel, strides, padding=padding)
+        x = tf.transpose(x, (0, 3, 1, 2))
+    elif dim_ordering == 'tf':
+        x = tf.nn.conv2d(x, kernel, strides, padding=padding)
+    else:
+        raise Exception('Unknown dim_ordering: ' + str(dim_ordering))
+
+    if _FLOATX == 'float64':
+        x = tf.cast(x, 'float64')
+    return x
+
+
+def pool2d(x, pool_size, strides=(1, 1),
+           border_mode='valid', dim_ordering='th', pool_mode='max'):
+    '''
+    # Arguments
+        pool_size: tuple of 2 integers.
+        strides: tuple of 2 integers.
+        border_mode: one of "valid", "same".
+        dim_ordering: one of "th", "tf".
+    '''
+    if border_mode == 'same':
+        padding = 'SAME'
+    elif border_mode == 'valid':
+        padding = 'VALID'
+    else:
+        raise Exception('Invalid border mode: ' + str(border_mode))
+
+    strides = (1,) + strides + (1,)
+    pool_size = (1,) + pool_size + (1,)
+
+    if _FLOATX == 'float64':
+        # tf max_pool only supports float32
+        x = tf.cast(x, 'float32')
+
+    if dim_ordering in {'tf', 'th'}:
+        if dim_ordering == 'th':
+            # TF uses the last dimension as channel dimension,
+            # instead of the 2nd one.
+            # TH input shape: (samples, input_depth, rows, cols)
+            # TF input shape: (samples, rows, cols, input_depth)
+            # TH kernel shape: (depth, input_depth, rows, cols)
+            # TF kernel shape: (rows, cols, input_depth, depth)
+            x = tf.transpose(x, (0, 2, 3, 1))
+        if pool_mode == 'max':
+            x = tf.nn.max_pool(x, pool_size, strides, padding=padding)
+        elif pool_mode == 'avg':
+            x = tf.nn.avg_pool(x, pool_size, strides, padding=padding)
+        else:
+            raise Exception('Invalid pooling mode: ' + str(pool_mode))
+        if dim_ordering == 'th':
+            x = tf.transpose(x, (0, 3, 1, 2))
+    else:
+        raise Exception('Unknown dim_ordering: ' + str(dim_ordering))
+
+    if _FLOATX == 'float64':
+        x = tf.cast(x, 'float64')
+    return x
+
+
+# RANDOMNESS
+
+def random_normal(shape, mean=0.0, std=1.0, dtype=_FLOATX, seed=None):
+    if seed is None:
+        seed = np.random.randint(10e6)
+    return tf.random_normal(shape, mean=mean, stddev=std,
+                            dtype=dtype, seed=seed)
+
+
+def random_uniform(shape, low=0.0, high=1.0, dtype=_FLOATX, seed=None):
+    if seed is None:
+        seed = np.random.randint(10e6)
+    return tf.random_uniform(shape, minval=low, maxval=high,
+                             dtype=dtype, seed=seed)
+
+
+def random_binomial(shape, p=0.0, dtype=_FLOATX, seed=None):
+    if seed is None:
+        seed = np.random.randint(10e6)
+    return tf.select(tf.random_uniform(shape, dtype=dtype, seed=seed) <= p,
+                     tf.ones(shape), tf.zeros(shape))

keras/backend/theano_backend.py

@@ -0,0 +1,896 @@
+import theano
+from theano import tensor as T
+from theano.sandbox.rng_mrg import MRG_RandomStreams as RandomStreams
+from theano.tensor.signal import pool
+from theano.tensor.nnet import conv3d2d
+import inspect
+import numpy as np
+from .common import _FLOATX, _EPSILON
+
+
+# INTERNAL UTILS
+theano.config.floatX = _FLOATX
+
+
+# VARIABLE MANIPULATION
+
+def variable(value, dtype=_FLOATX, name=None):
+    '''Instantiate a tensor variable.
+    '''
+    value = np.asarray(value, dtype=dtype)
+    return theano.shared(value=value, name=name, strict=False)
+
+
+def placeholder(shape=None, ndim=None, dtype=_FLOATX, name=None):
+    '''Instantiate an input data placeholder variable.
+    '''
+    if shape is None and ndim is None:
+        raise Exception('Specify either a shape or ndim value.')
+    if shape is not None:
+        ndim = len(shape)
+
+    broadcast = (False,) * ndim
+    return T.TensorType(dtype, broadcast)(name)
+
+
+def shape(x):
+    '''Return the shape of a tensor.
+
+    Warning: type returned will be different for
+    Theano backend (Theano tensor type) and TF backend (TF TensorShape).
+    '''
+    return x.shape
+
+
+def ndim(x):
+    return x.ndim
+
+
+def eval(x):
+    '''Run a graph.
+    '''
+    return x.eval()
+
+
+def zeros(shape, dtype=_FLOATX, name=None):
+    '''Instantiate an all-zeros variable.
+    '''
+    return variable(np.zeros(shape), dtype, name)
+
+
+def ones(shape, dtype=_FLOATX, name=None):
+    '''Instantiate an all-ones variable.
+    '''
+    return variable(np.ones(shape), dtype, name)
+
+
+def ones_like(x):
+    return T.ones_like(x)
+
+
+def zeros_like(x):
+    return T.zeros_like(x)
+
+
+def count_params(x):
+    '''Return number of scalars in a tensor.
+
+    Return: numpy integer.
+    '''
+    return np.prod(x.shape.eval())
+
+
+def cast(x, dtype):
+    return T.cast(x, dtype)
+
+
+# LINEAR ALGEBRA
+
+'''
+Assumed overridden:
++, -, /, *, +=, -=, *=, /=
+'''
+
+
+def dot(x, y):
+    return T.dot(x, y)
+
+
+def batch_dot(x, y, axes=None):
+    if axes is None:
+        # behaves like tf.batch_matmul as default
+        axes = [(x.ndim-1,), (y.ndim-2,)]
+    return T.batched_tensordot(x, y, axes=axes)
+
+
+def transpose(x):
+    return T.transpose(x)
+
+
+def gather(reference, indices):
+    '''reference: a tensor.
+    indices: an int tensor of indices.
+
+    Return: a tensor of same type as reference.
+    '''
+    return reference[indices]
+
+
+# ELEMENT-WISE OPERATIONS
+
+
+def max(x, axis=None, keepdims=False):
+    return T.max(x, axis=axis, keepdims=keepdims)
+
+
+def min(x, axis=None, keepdims=False):
+    return T.min(x, axis=axis, keepdims=keepdims)
+
+
+def sum(x, axis=None, keepdims=False):
+    '''Sum of the values in a tensor, alongside the specified axis.
+    '''
+    return T.sum(x, axis=axis, keepdims=keepdims)
+
+
+def prod(x, axis=None, keepdims=False):
+    '''Multiply the values in a tensor, alongside the specified axis.
+    '''
+    return T.prod(x, axis=axis, keepdims=keepdims)
+
+
+def mean(x, axis=None, keepdims=False):
+    dtype = None
+    if 'int' in x.dtype:
+        dtype = _FLOATX
+    return T.mean(x, axis=axis, keepdims=keepdims, dtype=dtype)
+
+
+def std(x, axis=None, keepdims=False):
+    return T.std(x, axis=axis, keepdims=keepdims)
+
+
+def any(x, axis=None, keepdims=False):
+    '''Bitwise reduction (logical OR).
+    '''
+    return T.any(x, axis=axis, keepdims=keepdims)
+
+
+def argmax(x, axis=-1):
+    return T.argmax(x, axis=axis, keepdims=False)
+
+
+def argmin(x, axis=-1):
+    return T.argmin(x, axis=axis, keepdims=False)
+
+
+def square(x):
+    return T.sqr(x)
+
+
+def abs(x):
+    return T.abs_(x)
+
+
+def sqrt(x):
+    x = T.clip(x, 0., np.inf)
+    return T.sqrt(x)
+
+
+def exp(x):
+    return T.exp(x)
+
+
+def log(x):
+    return T.log(x)
+
+
+def round(x):
+    return T.round(x)
+
+
+def sign(x):
+    return T.sgn(x)
+
+
+def pow(x, a):
+    return T.pow(x, a)
+
+
+def clip(x, min_value, max_value):
+    if max_value < min_value:
+        max_value = min_value
+    return T.clip(x, min_value, max_value)
+
+
+def equal(x, y):
+    return T.eq(x, y)
+
+
+def not_equal(x, y):
+    return T.neq(x, y)
+
+
+def maximum(x, y):
+    return T.maximum(x, y)
+
+
+def minimum(x, y):
+    return T.minimum(x, y)
+
+
+# SHAPE OPERATIONS
+
+def concatenate(tensors, axis=-1):
+    return T.concatenate(tensors, axis=axis)
+
+
+def reshape(x, shape):
+    return T.reshape(x, shape)
+
+
+def permute_dimensions(x, pattern):
+    '''Transpose dimensions.
+
+    pattern should be a tuple or list of
+    dimension indices, e.g. [0, 2, 1].
+    '''
+    pattern = tuple(pattern)
+    return x.dimshuffle(pattern)
+
+
+def repeat_elements(x, rep, axis):
+    '''Repeat the elements of a tensor along an axis, like np.repeat.
+
+    If x has shape (s1, s2, s3) and axis=1, the output
+    will have shape (s1, s2 * rep, s3).
+    '''
+    return T.repeat(x, rep, axis=axis)
+
+
+def resize_images(X, height_factor, width_factor, dim_ordering):
+    '''Resize the images contained in a 4D tensor of shape
+    - [batch, channels, height, width] (for 'th' dim_ordering)
+    - [batch, height, width, channels] (for 'tf' dim_ordering)
+    by a factor of (height_factor, width_factor). Both factors should be
+    positive integers.
+    '''
+    if dim_ordering == 'th':
+        output = repeat_elements(X, height_factor, axis=2)
+        output = repeat_elements(output, width_factor, axis=3)
+        return output
+    elif dim_ordering == 'tf':
+        output = repeat_elements(X, height_factor, axis=1)
+        output = repeat_elements(output, width_factor, axis=2)
+        return output
+    else:
+        raise Exception('Invalid dim_ordering: ' + dim_ordering)
+
+
+def resize_volumes(X, depth_factor, height_factor, width_factor, dim_ordering):
+    '''Resize the volume contained in a 5D tensor of shape
+    - [batch, channels, depth, height, width] (for 'th' dim_ordering)
+    - [batch, depth, height, width, channels] (for 'tf' dim_ordering)
+    by a factor of (depth_factor, height_factor, width_factor).
+    Both factors should be positive integers.
+    '''
+    if dim_ordering == 'th':
+        output = repeat_elements(X, depth_factor, axis=2)
+        output = repeat_elements(output, height_factor, axis=3)
+        output = repeat_elements(output, width_factor, axis=4)
+        return output
+    elif dim_ordering == 'tf':
+        output = repeat_elements(X, depth_factor, axis=1)
+        output = repeat_elements(output, height_factor, axis=2)
+        output = repeat_elements(output, width_factor, axis=3)
+        return output
+    else:
+        raise Exception('Invalid dim_ordering: ' + dim_ordering)
+
+
+def repeat(x, n):
+    '''Repeat a 2D tensor.
+
+    If x has shape (samples, dim) and n=2,
+    the output will have shape (samples, 2, dim).
+    '''
+    assert x.ndim == 2
+    x = x.dimshuffle((0, 'x', 1))
+    return T.extra_ops.repeat(x, n, axis=1)
+
+
+def tile(x, n):
+    return T.tile(x, n)
+
+
+def flatten(x):
+    return T.flatten(x)
+
+
+def batch_flatten(x):
+    '''Turn a n-D tensor into a 2D tensor where
+    the first dimension is conserved.
+    '''
+    x = T.reshape(x, (x.shape[0], T.prod(x.shape) // x.shape[0]))
+    return x
+
+
+def expand_dims(x, dim=-1):
+    '''Add a 1-sized dimension at index "dim".
+    '''
+    pattern = [i for i in range(x.type.ndim)]
+    if dim < 0:
+        if x.type.ndim == 0:
+            dim = 0
+        else:
+            dim = dim % x.type.ndim + 1
+    pattern.insert(dim, 'x')
+    return x.dimshuffle(pattern)
+
+
+def squeeze(x, axis):
+    '''Remove a 1-dimension from the tensor at index "axis".
+    '''
+    x = T.addbroadcast(x, axis)
+    return T.squeeze(x)
+
+
+def temporal_padding(x, padding=1):
+    '''Pad the middle dimension of a 3D tensor
+    with "padding" zeros left and right.
+
+    Appologies for the inane API, but Theano makes this
+    really hard.
+    '''
+    input_shape = x.shape
+    output_shape = (input_shape[0],
+                    input_shape[1] + 2 * padding,
+                    input_shape[2])
+    output = T.zeros(output_shape)
+    return T.set_subtensor(output[:, padding:x.shape[1] + padding, :], x)
+
+
+def spatial_2d_padding(x, padding=(1, 1), dim_ordering='th'):
+    '''Pad the 2nd and 3rd dimensions of a 4D tensor
+    with "padding[0]" and "padding[1]" (resp.) zeros left and right.
+    '''
+    input_shape = x.shape
+    if dim_ordering == 'th':
+        output_shape = (input_shape[0],
+                        input_shape[1],
+                        input_shape[2] + 2 * padding[0],
+                        input_shape[3] + 2 * padding[1])
+        output = T.zeros(output_shape)
+        indices = (slice(None),
+                   slice(None),
+                   slice(padding[0], input_shape[2] + padding[0]),
+                   slice(padding[1], input_shape[3] + padding[1]))
+
+    elif dim_ordering == 'tf':
+        output_shape = (input_shape[0],
+                        input_shape[1] + 2 * padding[0],
+                        input_shape[2] + 2 * padding[1],
+                        input_shape[3])
+        output = T.zeros(output_shape)
+        indices = (slice(None),
+                   slice(padding[0], input_shape[1] + padding[0]),
+                   slice(padding[1], input_shape[2] + padding[1]),
+                   slice(None))
+    else:
+        raise Exception('Invalid dim_ordering: ' + dim_ordering)
+    return T.set_subtensor(output[indices], x)
+
+
+def spatial_3d_padding(x, padding=(1, 1, 1), dim_ordering='th'):
+    '''Pad the 2nd, 3rd and 4th dimensions of a 5D tensor
+    with "padding[0]", "padding[1]" and "padding[2]" (resp.) zeros left and right.
+    '''
+    input_shape = x.shape
+    if dim_ordering == 'th':
+        output_shape = (input_shape[0],
+                        input_shape[1],
+                        input_shape[2] + 2 * padding[0],
+                        input_shape[3] + 2 * padding[1],
+                        input_shape[4] + 2 * padding[2])
+        output = T.zeros(output_shape)
+        indices = (slice(None),
+                   slice(None),
+                   slice(padding[0], input_shape[2] + padding[0]),
+                   slice(padding[1], input_shape[3] + padding[1]),
+                   slice(padding[2], input_shape[4] + padding[2]))
+
+    elif dim_ordering == 'tf':
+        output_shape = (input_shape[0],
+                        input_shape[1] + 2 * padding[0],
+                        input_shape[2] + 2 * padding[1],
+                        input_shape[3] + 2 * padding[2],
+                        input_shape[4])
+        output = T.zeros(output_shape)
+        indices = (slice(None),
+                   slice(padding[0], input_shape[1] + padding[0]),
+                   slice(padding[1], input_shape[2] + padding[1]),
+                   slice(padding[2], input_shape[3] + padding[2]),
+                   slice(None))
+    else:
+        raise Exception('Invalid dim_ordering: ' + dim_ordering)
+    return T.set_subtensor(output[indices], x)
+
+
+def pack(x):
+    return T.stack(*x)
+
+# VALUE MANIPULATION
+
+
+def get_value(x):
+    if not hasattr(x, 'get_value'):
+        raise Exception("'get_value() can only be called on a variable. " +
+                        "If you have an expression instead, use eval().")
+    return x.get_value()
+
+
+def set_value(x, value):
+    x.set_value(np.asarray(value, dtype=x.dtype))
+
+
+# GRAPH MANIPULATION
+
+class Function(object):
+
+    def __init__(self, inputs, outputs, updates=[], **kwargs):
+        self.function = theano.function(inputs, outputs, updates=updates,
+                                        allow_input_downcast=True, **kwargs)
+
+    def __call__(self, inputs):
+        assert type(inputs) in {list, tuple}
+        return self.function(*inputs)
+
+
+def function(inputs, outputs, updates=[], **kwargs):
+    if len(kwargs) > 0:
+        function_args = inspect.getargspec(theano.function)[0]
+        for key in kwargs.keys():
+            if key not in function_args:
+                msg = "Invalid argument '%s' passed to K.function" % key
+                raise ValueError(msg)
+    return Function(inputs, outputs, updates=updates, **kwargs)
+
+
+def gradients(loss, variables):
+    return T.grad(loss, variables)
+
+
+# CONTROL FLOW
+
+def rnn(step_function, inputs, initial_states,
+        go_backwards=False, mask=None, constants=None):
+    '''Iterates over the time dimension of a tensor.
+
+    # Arguments
+        inputs: tensor of temporal data of shape (samples, time, ...)
+            (at least 3D).
+        step_function:
+            Parameters:
+                input: tensor with shape (samples, ...) (no time dimension),
+                    representing input for the batch of samples at a certain
+                    time step.
+                states: list of tensors.
+            Returns:
+                output: tensor with shape (samples, ...) (no time dimension),
+                new_states: list of tensors, same length and shapes
+                    as 'states'.
+        initial_states: tensor with shape (samples, ...) (no time dimension),
+            containing the initial values for the states used in
+            the step function.
+        go_backwards: boolean. If True, do the iteration over
+            the time dimension in reverse order.
+        mask: binary tensor with shape (samples, time),
+            with a zero for every element that is masked.
+        constants: a list of constant values passed at each step.
+
+
+    # Returns
+        A tuple (last_output, outputs, new_states).
+            last_output: the latest output of the rnn, of shape (samples, ...)
+            outputs: tensor with shape (samples, time, ...) where each
+                entry outputs[s, t] is the output of the step function
+                at time t for sample s.
+            new_states: list of tensors, latest states returned by
+                the step function, of shape (samples, ...).
+    '''
+    ndim = inputs.ndim
+    assert ndim >= 3, 'Input should be at least 3D.'
+
+    axes = [1, 0] + list(range(2, ndim))
+    inputs = inputs.dimshuffle(axes)
+
+    if mask is not None:
+        if mask.ndim == ndim-1:
+            mask = expand_dims(mask)
+        assert mask.ndim == ndim
+        mask = mask.dimshuffle(axes)
+
+        if constants is None:
+            constants = []
+        # build an all-zero tensor of shape (samples, output_dim)
+        initial_output = step_function(inputs[0], initial_states + constants)[0] * 0
+        # Theano gets confused by broadcasting patterns in the scan op
+        initial_output = T.unbroadcast(initial_output, 0, 1)
+
+        def _step(input, mask, output_tm1, *states):
+            output, new_states = step_function(input, states)
+            # output previous output if masked.
+            output = T.switch(mask, output, output_tm1)
+            return_states = []
+            for state, new_state in zip(states, new_states):
+                return_states.append(T.switch(mask, new_state, state))
+            return [output] + return_states
+
+        results, _ = theano.scan(
+            _step,
+            sequences=[inputs, mask],
+            outputs_info=[initial_output] + initial_states,
+            non_sequences=constants,
+            go_backwards=go_backwards)
+    else:
+        def _step(input, *states):
+            output, new_states = step_function(input, states)
+            return [output] + new_states
+
+        results, _ = theano.scan(
+            _step,
+            sequences=inputs,
+            outputs_info=[None] + initial_states,
+            non_sequences=constants,
+            go_backwards=go_backwards)
+
+    # deal with Theano API inconsistency
+    if type(results) is list:
+        outputs = results[0]
+        states = results[1:]
+    else:
+        outputs = results
+        states = []
+
+    outputs = T.squeeze(outputs)
+    last_output = outputs[-1]
+
+    axes = [1, 0] + list(range(2, outputs.ndim))
+    outputs = outputs.dimshuffle(axes)
+    states = [T.squeeze(state[-1]) for state in states]
+    return last_output, outputs, states
+
+
+def switch(condition, then_expression, else_expression):
+    '''condition: scalar tensor.
+    '''
+    return T.switch(condition, then_expression, else_expression)
+
+
+# NN OPERATIONS
+
+def relu(x, alpha=0., max_value=None):
+    assert hasattr(T.nnet, 'relu'), ('It looks like like your version of '
+                                     'Theano is out of date. '
+                                     'Install the latest version with:\n'
+                                     'pip install git+git://github.com/Theano/Theano.git --upgrade --no-deps')
+    x = T.nnet.relu(x, alpha)
+    if max_value is not None:
+        x = T.minimum(x, max_value)
+    return x
+
+
+def softmax(x):
+    return T.nnet.softmax(x)
+
+
+def softplus(x):
+    return T.nnet.softplus(x)
+
+
+def categorical_crossentropy(output, target, from_logits=False):
+    if from_logits:
+        output = T.nnet.softmax(output)
+    else:
+        # scale preds so that the class probas of each sample sum to 1
+        output /= output.sum(axis=-1, keepdims=True)
+    # avoid numerical instability with _EPSILON clipping
+    output = T.clip(output, _EPSILON, 1.0 - _EPSILON)
+    return T.nnet.categorical_crossentropy(output, target)
+
+
+def binary_crossentropy(output, target, from_logits=False):
+    if from_logits:
+        output = T.nnet.sigmoid(output)
+    # avoid numerical instability with _EPSILON clipping
+    output = T.clip(output, _EPSILON, 1.0 - _EPSILON)
+    return T.nnet.binary_crossentropy(output, target)
+
+
+def sigmoid(x):
+    return T.nnet.sigmoid(x)
+
+
+def hard_sigmoid(x):
+    return T.nnet.hard_sigmoid(x)
+
+
+def tanh(x):
+    return T.tanh(x)
+
+
+def dropout(x, level, seed=None):
+    if level < 0. or level >= 1:
+        raise Exception('Dropout level must be in interval [0, 1[.')
+    if seed is None:
+        seed = np.random.randint(10e6)
+    rng = RandomStreams(seed=seed)
+    retain_prob = 1. - level
+    x *= rng.binomial(x.shape, p=retain_prob, dtype=x.dtype)
+    x /= retain_prob
+    return x
+
+
+def l2_normalize(x, axis):
+    norm = T.sqrt(T.sum(T.square(x), axis=axis, keepdims=True))
+    return x / norm
+
+
+# CONVOLUTIONS
+
+def conv2d(x, kernel, strides=(1, 1), border_mode='valid', dim_ordering='th',
+           image_shape=None, filter_shape=None):
+    '''
+    border_mode: string, "same" or "valid".
+    '''
+    if dim_ordering not in {'th', 'tf'}:
+        raise Exception('Unknown dim_ordering ' + str(dim_ordering))
+
+    if dim_ordering == 'tf':
+        # TF uses the last dimension as channel dimension,
+        # instead of the 2nd one.
+        # TH input shape: (samples, input_depth, rows, cols)
+        # TF input shape: (samples, rows, cols, input_depth)
+        # TH kernel shape: (depth, input_depth, rows, cols)
+        # TF kernel shape: (rows, cols, input_depth, depth)
+        x = x.dimshuffle((0, 3, 1, 2))
+        kernel = kernel.dimshuffle((3, 2, 0, 1))
+        if image_shape:
+            image_shape = (image_shape[0], image_shape[3],
+                           image_shape[1], image_shape[2])
+        if filter_shape:
+            filter_shape = (filter_shape[3], filter_shape[2],
+                            filter_shape[0], filter_shape[1])
+
+    if border_mode == 'same':
+        th_border_mode = 'half'
+        np_kernel = kernel.eval()
+        assert strides[0] <= np_kernel.shape[2], 'strides should be smaller than the convolution window.'
+        assert strides[1] <= np_kernel.shape[3], 'strides should be smaller than the convolution window.'
+    elif border_mode == 'valid':
+        th_border_mode = 'valid'
+    else:
+        raise Exception('Border mode not supported: ' + str(border_mode))
+
+    # Theano might not accept like longs
+    def int_or_none(value):
+        try:
+            return int(value)
+        except TypeError:
+            return None
+
+    if image_shape is not None:
+        image_shape = tuple(int_or_none(v) for v in image_shape)
+
+    if filter_shape is not None:
+        filter_shape = tuple(int_or_none(v) for v in filter_shape)
+
+    conv_out = T.nnet.conv2d(x, kernel,
+                             border_mode=th_border_mode,
+                             subsample=strides,
+                             input_shape=image_shape,
+                             filter_shape=filter_shape)
+
+    if border_mode == 'same':
+        if np_kernel.shape[2] % 2 == 0:
+            conv_out = conv_out[:,:,:(x.shape[2]+strides[0]-1) // strides[0],:]
+        if np_kernel.shape[3] % 2 == 0:
+            conv_out = conv_out[:,:,:,:(x.shape[3]+strides[1]-1) // strides[1]]
+
+    if dim_ordering == 'tf':
+        conv_out = conv_out.dimshuffle((0, 2, 3, 1))
+    return conv_out
+
+
+def conv3d(x, kernel, strides=(1, 1, 1),
+           border_mode='valid', dim_ordering='th',
+           volume_shape=None, filter_shape=None):
+    '''
+    Run on cuDNN if available.
+    border_mode: string, "same" or "valid".
+    '''
+    if dim_ordering not in {'th', 'tf'}:
+        raise Exception('Unknown dim_ordering ' + str(dim_ordering))
+
+    if border_mode not in {'same', 'valid'}:
+        raise Exception('Invalid border mode: ' + str(border_mode))
+
+    if dim_ordering == 'tf':
+        # TF uses the last dimension as channel dimension,
+        # instead of the 2nd one.
+        # TH input shape: (samples, input_depth, conv_dim1, conv_dim2, conv_dim3)
+        # TF input shape: (samples, conv_dim1, conv_dim2, conv_dim3, input_depth)
+        # TH kernel shape: (out_depth, input_depth, kernel_dim1, kernel_dim2, kernel_dim3)
+        # TF kernel shape: (kernel_dim1, kernel_dim2, kernel_dim3, input_depth, out_depth)
+        x = x.dimshuffle((0, 4, 1, 2, 3))
+        kernel = kernel.dimshuffle((4, 3, 0, 1, 2))
+        if volume_shape:
+            volume_shape = (volume_shape[0], volume_shape[4],
+                            volume_shape[1], volume_shape[2], volume_shape[3])
+        if filter_shape:
+            filter_shape = (filter_shape[4], filter_shape[3],
+                            filter_shape[0], filter_shape[1], filter_shape[2])
+
+    if border_mode == 'same':
+        assert(strides == (1, 1, 1))
+        pad_dim1 = (kernel.shape[2] - 1)
+        pad_dim2 = (kernel.shape[3] - 1)
+        pad_dim3 = (kernel.shape[4] - 1)
+        output_shape = (x.shape[0], x.shape[1],
+                        x.shape[2] + pad_dim1,
+                        x.shape[3] + pad_dim2,
+                        x.shape[4] + pad_dim3)
+        output = T.zeros(output_shape)
+        indices = (slice(None), slice(None),
+                   slice(pad_dim1 // 2, x.shape[2] + pad_dim1 // 2),
+                   slice(pad_dim2 // 2, x.shape[3] + pad_dim2 // 2),
+                   slice(pad_dim3 // 2, x.shape[4] + pad_dim3 // 2))
+        x = T.set_subtensor(output[indices], x)
+        border_mode = 'valid'
+
+    border_mode_3d = (border_mode, border_mode, border_mode)
+    conv_out = conv3d2d.conv3d(signals=x.dimshuffle(0, 2, 1, 3, 4),
+                               filters=kernel.dimshuffle(0, 2, 1, 3, 4),
+                               border_mode=border_mode_3d)
+    conv_out = conv_out.dimshuffle(0, 2, 1, 3, 4)
+
+    # support strides by manually slicing the output
+    if strides != (1, 1, 1):
+        conv_out = conv_out[:, :, ::strides[0], ::strides[1], ::strides[2]]
+
+    if dim_ordering == 'tf':
+        conv_out = conv_out.dimshuffle((0, 2, 3, 4, 1))
+
+    return conv_out
+
+
+def pool2d(x, pool_size, strides=(1, 1), border_mode='valid',
+           dim_ordering='th', pool_mode='max'):
+    if border_mode == 'same':
+        w_pad = pool_size[0] - 2 if pool_size[0] % 2 == 1 else pool_size[0] - 1
+        h_pad = pool_size[1] - 2 if pool_size[1] % 2 == 1 else pool_size[1] - 1
+        padding = (w_pad, h_pad)
+    elif border_mode == 'valid':
+        padding = (0, 0)
+    else:
+        raise Exception('Invalid border mode: ' + str(border_mode))
+
+    if dim_ordering not in {'th', 'tf'}:
+        raise Exception('Unknown dim_ordering ' + str(dim_ordering))
+
+    if dim_ordering == 'tf':
+        x = x.dimshuffle((0, 3, 1, 2))
+
+    if pool_mode == 'max':
+        pool_out = pool.pool_2d(x, ds=pool_size, st=strides,
+                                ignore_border=True,
+                                padding=padding,
+                                mode='max')
+    elif pool_mode == 'avg':
+        pool_out = pool.pool_2d(x, ds=pool_size, st=strides,
+                                ignore_border=True,
+                                padding=padding,
+                                mode='average_exc_pad')
+    else:
+        raise Exception('Invalid pooling mode: ' + str(pool_mode))
+
+    if border_mode == 'same':
+        expected_width = (x.shape[2] + strides[0] - 1) // strides[0]
+        expected_height = (x.shape[3] + strides[1] - 1) // strides[1]
+
+        pool_out = pool_out[:, :,
+                            : expected_width,
+                            : expected_height]
+
+    if dim_ordering == 'tf':
+        pool_out = pool_out.dimshuffle((0, 2, 3, 1))
+    return pool_out
+
+
+def pool3d(x, pool_size, strides=(1, 1, 1), border_mode='valid',
+           dim_ordering='th', pool_mode='max'):
+    if border_mode == 'same':
+        # TODO: add implementation for border_mode="same"
+        raise Exception('border_mode="same" not supported with Theano.')
+    elif border_mode == 'valid':
+        ignore_border = True
+        padding = (0, 0)
+    else:
+        raise Exception('Invalid border mode: ' + str(border_mode))
+
+    if dim_ordering not in {'th', 'tf'}:
+        raise Exception('Unknown dim_ordering ' + str(dim_ordering))
+
+    if dim_ordering == 'tf':
+        x = x.dimshuffle((0, 4, 1, 2, 3))
+
+    if pool_mode == 'max':
+        # pooling over conv_dim2, conv_dim1 (last two channels)
+        output = pool.pool_2d(input=x.dimshuffle(0, 1, 4, 3, 2),
+                              ds=(pool_size[1], pool_size[0]),
+                              st=(strides[1], strides[0]),
+                              ignore_border=ignore_border,
+                              padding=padding,
+                              mode='max')
+
+        # pooling over conv_dim3
+        pool_out = pool.pool_2d(input=output.dimshuffle(0, 1, 4, 3, 2),
+                                ds=(1, pool_size[2]),
+                                st=(1, strides[2]),
+                                ignore_border=ignore_border,
+                                padding=padding,
+                                mode='max')
+
+    elif pool_mode == 'avg':
+        # pooling over conv_dim2, conv_dim1 (last two channels)
+        output = pool.pool_2d(input=x.dimshuffle(0, 1, 4, 3, 2),
+                              ds=(pool_size[1], pool_size[0]),
+                              st=(strides[1], strides[0]),
+                              ignore_border=ignore_border,
+                              padding=padding,
+                              mode='average_exc_pad')
+
+        # pooling over conv_dim3
+        pool_out = pool.pool_2d(input=output.dimshuffle(0, 1, 4, 3, 2),
+                                ds=(1, pool_size[2]),
+                                st=(1, strides[2]),
+                                ignore_border=ignore_border,
+                                padding=padding,
+                                mode='average_exc_pad')
+    else:
+        raise Exception('Invalid pooling mode: ' + str(pool_mode))
+
+    if dim_ordering == 'tf':
+        pool_out = pool_out.dimshuffle((0, 2, 3, 4, 1))
+    return pool_out
+
+
+# RANDOMNESS
+
+
+def random_normal(shape, mean=0.0, std=1.0, dtype=_FLOATX, seed=None):
+    if seed is None:
+        seed = np.random.randint(10e6)
+    rng = RandomStreams(seed=seed)
+    return rng.normal(size=shape, avg=mean, std=std, dtype=dtype)
+
+
+def random_uniform(shape, low=0.0, high=1.0, dtype=_FLOATX, seed=None):
+    if seed is None:
+        seed = np.random.randint(10e6)
+    rng = RandomStreams(seed=seed)
+    return rng.uniform(shape, low=low, high=high, dtype=dtype)
+
+
+def random_binomial(shape, p=0.0, dtype=_FLOATX, seed=None):
+    if seed is None:
+        seed = np.random.randint(10e6)
+    rng = RandomStreams(seed=seed)
+    return rng.binomial(shape, p=p, dtype=dtype)
+
+'''
+more TODO:
+
+tensordot -> soon to be introduced in TF
+batched_tensordot -> reimplement
+'''

keras/callbacks.py

@@ -1,16 +1,17 @@
 from __future__ import absolute_import
 from __future__ import print_function
-import theano
-import theano.tensor as T
-import numpy as np
-import warnings
-import time
-from collections import deque
 
+import numpy as np
+import time
+import json
+import warnings
+
+from collections import deque
 from .utils.generic_utils import Progbar
+from keras import backend as K
+
 
 class CallbackList(object):
-
     def __init__(self, callbacks=[], queue_length=10):
         self.callbacks = [c for c in callbacks]
         self.queue_length = queue_length
@@ -43,23 +44,27 @@ class CallbackList(object):
             callback.on_batch_begin(batch, logs)
         self._delta_ts_batch_begin.append(time.time() - t_before_callbacks)
         delta_t_median = np.median(self._delta_ts_batch_begin)
-        if self._delta_t_batch > 0. and delta_t_median > 0.95 * self._delta_t_batch \
-            and delta_t_median > 0.1:
+        if self._delta_t_batch > 0. and delta_t_median > 0.95 * \
+           self._delta_t_batch and delta_t_median > 0.1:
             warnings.warn('Method on_batch_begin() is slow compared '
-                'to the batch update (%f). Check your callbacks.' % delta_t_median)
+                          'to the batch update (%f). Check your callbacks.'
+                          % delta_t_median)
         self._t_enter_batch = time.time()
 
     def on_batch_end(self, batch, logs={}):
+        if not hasattr(self, '_t_enter_batch'):
+            self._t_enter_batch = time.time()
         self._delta_t_batch = time.time() - self._t_enter_batch
         t_before_callbacks = time.time()
         for callback in self.callbacks:
             callback.on_batch_end(batch, logs)
         self._delta_ts_batch_end.append(time.time() - t_before_callbacks)
         delta_t_median = np.median(self._delta_ts_batch_end)
-        if self._delta_t_batch > 0. and delta_t_median > 0.95 * self._delta_t_batch \
-            and delta_t_median > 0.1:
+        if self._delta_t_batch > 0. and delta_t_median > 0.95 * \
+           self._delta_t_batch and delta_t_median > 0.1:
             warnings.warn('Method on_batch_end() is slow compared '
-                'to the batch update (%f). Check your callbacks.' % delta_t_median)
+                          'to the batch update (%f). Check your callbacks.'
+                          % delta_t_median)
 
     def on_train_begin(self, logs={}):
         for callback in self.callbacks:
@@ -71,7 +76,31 @@ class CallbackList(object):
 
 
 class Callback(object):
+    '''Abstract base class used to build new callbacks.
 
+    # Properties
+        params: dict. Training parameters
+            (eg. verbosity, batch size, number of epochs...).
+        model: instance of `keras.models.Model`.
+            Reference of the model being trained.
+
+    The `logs` dictionary that callback methods
+    take as argument will contain keys for quantities relevant to
+    the current batch or epoch.
+
+    Currently, the `.fit()` method of the `Sequential` model class
+    will include the following quantities in the `logs` that
+    it passes to its callbacks:
+
+        on_epoch_end: logs include `acc` and `loss`, and
+            optionally include `val_loss`
+            (if validation is enabled in `fit`), and `val_acc`
+            (if validation and accuracy monitoring are enabled).
+        on_batch_begin: logs include `size`,
+            the number of samples in the current batch.
+        on_batch_end: logs include `loss`, and optionally `acc`
+            (if accuracy monitoring is enabled).
+    '''
     def __init__(self):
         pass
 
@@ -99,120 +128,370 @@ class Callback(object):
     def on_train_end(self, logs={}):
         pass
 
+
 class BaseLogger(Callback):
+    '''Callback that accumulates epoch averages of
+    the metrics being monitored.
 
-    def on_train_begin(self, logs={}):
-        self.verbose = self.params['verbose']
-
-    def on_epoch_begin(self, epoch, logs={}):
-        if self.verbose:
-            print('Epoch %d' % epoch)
-            self.progbar = Progbar(target=self.params['nb_sample'], \
-                verbose=self.verbose)
-        self.current = 0
-        self.tot_loss = 0.
-        self.tot_acc = 0.
-
-    def on_batch_begin(self, batch, logs={}):
-        if self.current < self.params['nb_sample']:
-            self.log_values = []
-
-    def on_batch_end(self, batch, logs={}):
-        batch_size = logs.get('size', 0)
-        self.current += batch_size
-
-        loss = logs.get('loss')
-        self.log_values.append(('loss', loss))
-        self.tot_loss += loss * batch_size
-        if self.params['show_accuracy']:
-            accuracy = logs.get('accuracy')
-            self.log_values.append(('acc.', accuracy))
-            self.tot_acc += accuracy * batch_size
-        # skip progbar update for the last batch; will be handled by on_epoch_end
-        if self.verbose and self.current < self.params['nb_sample']:
-            self.progbar.update(self.current, self.log_values)
-
-    def on_epoch_end(self, epoch, logs={}):
-        self.log_values.append(('loss', self.tot_loss / self.current))
-        if self.params['show_accuracy']:
-            self.log_values.append(('acc.', self.tot_acc / self.current))
-        if self.params['do_validation']:
-            val_loss = logs.get('val_loss')
-            self.log_values.append(('val. loss', val_loss))
-            if self.params['show_accuracy']:
-                val_acc = logs.get('val_accuracy')
-                self.log_values.append(('val. acc.', val_acc))
-        self.progbar.update(self.current, self.log_values)
-
-
-class History(Callback):
-
-    def on_train_begin(self, logs={}):
-        self.epoch = []
-        self.loss = []
-        if self.params['show_accuracy']:
-            self.accuracy = []
-        if self.params['do_validation']:
-            self.validation_loss = []
-            if self.params['show_accuracy']:
-                self.validation_accuracy = []
-
+    This callback is automatically applied to
+    every Keras model.
+    '''
     def on_epoch_begin(self, epoch, logs={}):
         self.seen = 0
-        self.tot_loss = 0.
-        self.tot_accuracy = 0.
+        self.totals = {}
 
     def on_batch_end(self, batch, logs={}):
         batch_size = logs.get('size', 0)
         self.seen += batch_size
-        self.tot_loss += logs.get('loss', 0.) * batch_size
-        if self.params['show_accuracy']:
-            self.tot_accuracy += logs.get('accuracy', 0.) * batch_size
+
+        for k, v in logs.items():
+            if k in self.totals:
+                self.totals[k] += v * batch_size
+            else:
+                self.totals[k] = v * batch_size
+
+    def on_epoch_end(self, epoch, logs={}):
+        for k in self.params['metrics']:
+            if k in self.totals:
+                # make value available to next callbacks
+                logs[k] = self.totals[k] / self.seen
+
+
+class ProgbarLogger(Callback):
+    '''Callback that prints metrics to stdout.
+    '''
+    def on_train_begin(self, logs={}):
+        self.verbose = self.params['verbose']
+        self.nb_epoch = self.params['nb_epoch']
+
+    def on_epoch_begin(self, epoch, logs={}):
+        if self.verbose:
+            print('Epoch %d/%d' % (epoch + 1, self.nb_epoch))
+            self.progbar = Progbar(target=self.params['nb_sample'],
+                                   verbose=self.verbose)
+        self.seen = 0
+
+    def on_batch_begin(self, batch, logs={}):
+        if self.seen < self.params['nb_sample']:
+            self.log_values = []
+
+    def on_batch_end(self, batch, logs={}):
+        batch_size = logs.get('size', 0)
+        self.seen += batch_size
+
+        for k in self.params['metrics']:
+            if k in logs:
+                self.log_values.append((k, logs[k]))
+
+        # skip progbar update for the last batch;
+        # will be handled by on_epoch_end
+        if self.verbose and self.seen < self.params['nb_sample']:
+            self.progbar.update(self.seen, self.log_values)
+
+    def on_epoch_end(self, epoch, logs={}):
+        for k in self.params['metrics']:
+            if k in logs:
+                self.log_values.append((k, logs[k]))
+        if self.verbose:
+            self.progbar.update(self.seen, self.log_values)
+
+
+class History(Callback):
+    '''Callback that records events
+    into a `History` object.
+
+    This callback is automatically applied to
+    every Keras model. The `History` object
+    gets returned by the `fit` method of models.
+    '''
+    def on_train_begin(self, logs={}):
+        self.epoch = []
+        self.history = {}
 
     def on_epoch_end(self, epoch, logs={}):
-        val_loss = logs.get('val_loss')
-        val_acc = logs.get('val_accuracy')
         self.epoch.append(epoch)
-        self.loss.append(self.tot_loss / self.seen)
-        if self.params['show_accuracy']:
-            self.accuracy.append(self.tot_accuracy / self.seen)
-        if self.params['do_validation']:
-            self.validation_loss.append(val_loss)
-            if self.params['show_accuracy']:
-                self.validation_accuracy.append(val_acc)
+        for k, v in logs.items():
+            if k not in self.history:
+                self.history[k] = []
+            self.history[k].append(v)
+
 
 class ModelCheckpoint(Callback):
-    def __init__(self, filepath, verbose=0, save_best_only=False):
+    '''Save the model after every epoch.
+
+    `filepath` can contain named formatting options,
+    which will be filled the value of `epoch` and
+    keys in `logs` (passed in `on_epoch_end`).
+
+    For example: if `filepath` is `weights.{epoch:02d}-{val_loss:.2f}.hdf5`,
+    then multiple files will be save with the epoch number and
+    the validation loss.
+
+    # Arguments
+        filepath: string, path to save the model file.
+        monitor: quantity to monitor.
+        verbose: verbosity mode, 0 or 1.
+        save_best_only: if `save_best_only=True`,
+            the latest best model according to
+            the validation loss will not be overwritten.
+        mode: one of {auto, min, max}.
+            If `save_best_only=True`, the decision
+            to overwrite the current save file is made
+            based on either the maximization or the
+            minization of the monitored. For `val_acc`,
+            this should be `max`, for `val_loss` this should
+            be `min`, etc. In `auto` mode, the direction is
+            automatically inferred from the name of the monitored quantity.
+
+    '''
+    def __init__(self, filepath, monitor='val_loss', verbose=0,
+                 save_best_only=False, mode='auto'):
+
         super(Callback, self).__init__()
-        
+        self.monitor = monitor
         self.verbose = verbose
         self.filepath = filepath
         self.save_best_only = save_best_only
-        self.loss = []
-        self.best_loss = np.Inf
-        self.val_loss = []
-        self.best_val_loss = np.Inf
+
+        if mode not in ['auto', 'min', 'max']:
+            warnings.warn('ModelCheckpoint mode %s is unknown, '
+                          'fallback to auto mode.' % (mode),
+                          RuntimeWarning)
+            mode = 'auto'
+
+        if mode == 'min':
+            self.monitor_op = np.less
+            self.best = np.Inf
+        elif mode == 'max':
+            self.monitor_op = np.greater
+            self.best = -np.Inf
+        else:
+            if 'acc' in self.monitor:
+                self.monitor_op = np.greater
+                self.best = -np.Inf
+            else:
+                self.monitor_op = np.less
+                self.best = np.Inf
 
     def on_epoch_end(self, epoch, logs={}):
-        '''currently, on_epoch_end receives epoch_logs from keras.models.Sequential.fit
-        which does only contain, if at all, the validation loss and validation accuracy'''
-        if self.save_best_only and self.params['do_validation']:
-            cur_val_loss = logs.get('val_loss')
-            self.val_loss.append(cur_val_loss)
-            if cur_val_loss < self.best_val_loss:
-                if self.verbose > 0:
-                    print("Epoch %05d: valdidation loss improved from %0.5f to %0.5f, saving model to %s"
-                        % (epoch, self.best_val_loss, cur_val_loss, self.filepath))
-                self.best_val_loss = cur_val_loss
-                self.model.save_weights(self.filepath, overwrite=True)
+        filepath = self.filepath.format(epoch=epoch, **logs)
+        if self.save_best_only:
+            current = logs.get(self.monitor)
+            if current is None:
+                warnings.warn('Can save best model only with %s available, '
+                              'skipping.' % (self.monitor), RuntimeWarning)
             else:
-                if self.verbose > 0:
-                    print("Epoch %05d: validation loss did not improve" % (epoch))
-        elif self.save_best_only and not self.params['do_validation']:
-            import warnings
-            warnings.warn("Can save best model only with validation data, skipping", RuntimeWarning)
-        elif not self.save_best_only:
+                if self.monitor_op(current, self.best):
+                    if self.verbose > 0:
+                        print('Epoch %05d: %s improved from %0.5f to %0.5f,'
+                              ' saving model to %s'
+                              % (epoch, self.monitor, self.best,
+                                 current, filepath))
+                    self.best = current
+                    self.model.save_weights(filepath, overwrite=True)
+                else:
+                    if self.verbose > 0:
+                        print('Epoch %05d: %s did not improve' %
+                              (epoch, self.monitor))
+        else:
             if self.verbose > 0:
-                print("Epoch %05d: saving model to %s" % (epoch, self.filepath))
-            self.model.save_weights(self.filepath, overwrite=True)
+                print('Epoch %05d: saving model to %s' % (epoch, filepath))
+            self.model.save_weights(filepath, overwrite=True)
 
+
+class EarlyStopping(Callback):
+    '''Stop training when a monitored quantity has stopped improving.
+
+    # Arguments
+        monitor: quantity to be monitored.
+        patience: number of epochs with no improvement
+            after which training will be stopped.
+        verbose: verbosity mode.
+        mode: one of {auto, min, max}. In 'min' mode,
+            training will stop when the quantity
+            monitored has stopped decreasing; in 'max'
+            mode it will stop when the quantity
+            monitored has stopped increasing.
+    '''
+    def __init__(self, monitor='val_loss', patience=0, verbose=0, mode='auto'):
+        super(Callback, self).__init__()
+
+        self.monitor = monitor
+        self.patience = patience
+        self.verbose = verbose
+        self.wait = 0
+
+        if mode not in ['auto', 'min', 'max']:
+            warnings.warn('EarlyStopping mode %s is unknown, '
+                          'fallback to auto mode.' % (self.mode), RuntimeWarning)
+            mode = 'auto'
+
+        if mode == 'min':
+            self.monitor_op = np.less
+            self.best = np.Inf
+        elif mode == 'max':
+            self.monitor_op = np.greater
+            self.best = -np.Inf
+        else:
+            if 'acc' in self.monitor:
+                self.monitor_op = np.greater
+                self.best = -np.Inf
+            else:
+                self.monitor_op = np.less
+                self.best = np.Inf
+
+    def on_epoch_end(self, epoch, logs={}):
+        current = logs.get(self.monitor)
+        if current is None:
+            warnings.warn('Early stopping requires %s available!' %
+                          (self.monitor), RuntimeWarning)
+
+        if self.monitor_op(current, self.best):
+            self.best = current
+            self.wait = 0
+        else:
+            if self.wait >= self.patience:
+                if self.verbose > 0:
+                    print('Epoch %05d: early stopping' % (epoch))
+                self.model.stop_training = True
+            self.wait += 1
+
+
+class RemoteMonitor(Callback):
+    '''Callback used to stream events to a server.
+
+    Requires the `requests` library.
+
+    # Arguments
+        root: root url to which the events will be sent (at the end
+            of every epoch). Events are sent to
+            `root + '/publish/epoch/end/'`. Calls are HTTP POST,
+            with a `data` argument which is a JSON-encoded dictionary
+            of event data.
+    '''
+    def __init__(self, root='http://localhost:9000'):
+        self.root = root
+
+    def on_epoch_end(self, epoch, logs={}):
+        import requests
+        send = {}
+        send['epoch'] = epoch
+        for k, v in logs.items():
+            send[k] = v
+
+        try:
+            requests.post(self.root + '/publish/epoch/end/',
+                          {'data': json.dumps(send)})
+        except:
+            print('Warning: could not reach RemoteMonitor '
+                  'root server at ' + str(self.root))
+
+
+class LearningRateScheduler(Callback):
+    '''Learning rate scheduler.
+
+    # Arguments
+        schedule: a function that takes an epoch index as input
+            (integer, indexed from 0) and returns a new
+            learning rate as output (float).
+    '''
+    def __init__(self, schedule):
+        super(LearningRateScheduler, self).__init__()
+        self.schedule = schedule
+
+    def on_epoch_begin(self, epoch, logs={}):
+        assert hasattr(self.model.optimizer, 'lr'), \
+            'Optimizer must have a "lr" attribute.'
+        lr = self.schedule(epoch)
+        assert type(lr) == float, 'The output of the "schedule" function should be float.'
+        K.set_value(self.model.optimizer.lr, lr)
+
+
+class TensorBoard(Callback):
+    ''' Tensorboard basic visualizations.
+
+    This callback writes a log for TensorBoard, which allows
+    you to visualize dynamic graphs of your training and test
+    metrics, as well as activation histograms for the different
+    layers in your model.
+
+    TensorBoard is a visualization tool provided with TensorFlow.
+
+    If you have installed TensorFlow with pip, you should be able
+    to launch TensorBoard from the command line:
+    ```
+    tensorboard --logdir=/full_path_to_your_logs
+    ```
+    You can find more information about TensorBoard
+    [here](https://www.tensorflow.org/versions/master/how_tos/summaries_and_tensorboard/index.html).
+
+    # Arguments
+        log_dir: the path of the directory where to save the log
+            files to be parsed by tensorboard
+        histogram_freq: frequency (in epochs) at which to compute activation
+            histograms for the layers of the model. If set to 0,
+            histograms won't be computed.
+    '''
+    def __init__(self, log_dir='./logs', histogram_freq=0):
+        super(Callback, self).__init__()
+        if K._BACKEND != 'tensorflow':
+            raise Exception('TensorBoard callback only works '
+                            'with the TensorFlow backend.')
+        self.log_dir = log_dir
+        self.histogram_freq = histogram_freq
+        self.merged = None
+
+    def _set_model(self, model):
+        import tensorflow as tf
+        import keras.backend.tensorflow_backend as KTF
+
+        self.model = model
+        self.sess = KTF.get_session()
+        if self.histogram_freq and not self.merged:
+            mod_type = self.model.get_config()['name']
+            if mod_type == 'Sequential':
+                layers = {l.get_config()['name']: l for l in self.model.layers}
+            elif mod_type == 'Graph':
+                layers = self.model.nodes
+            else:
+                raise Exception('Unrecognized model:',
+                                self.model.get_config()['name'])
+            for l in layers:
+                cur_layer = layers[l]
+                if hasattr(cur_layer, 'W'):
+                    tf.histogram_summary('{}_W'.format(l), cur_layer.W)
+                if hasattr(cur_layer, 'b'):
+                    tf.histogram_summary('{}_b'.format(l), cur_layer.b)
+                if hasattr(cur_layer, 'get_output'):
+                    tf.histogram_summary('{}_out'.format(l),
+                                         cur_layer.get_output())
+        self.merged = tf.merge_all_summaries()
+        self.writer = tf.train.SummaryWriter(self.log_dir,
+                                             self.sess.graph_def)
+
+    def on_epoch_end(self, epoch, logs={}):
+        import tensorflow as tf
+
+        if self.model.validation_data and self.histogram_freq:
+            if epoch % self.histogram_freq == 0:
+                if self.params.get('show_accuracy'):
+                    test_function = self.model._test_with_acc
+                else:
+                    test_function = self.model._test
+                names = [v.name for v in test_function.inputs]
+                # TODO: implement batched calls to sess.run
+                # (current call will likely go OOM on GPU)
+                feed_dict = dict(zip(names, self.model.validation_data))
+                result = self.sess.run([self.merged], feed_dict=feed_dict)
+                summary_str = result[0]
+                self.writer.add_summary(summary_str, epoch)
+
+        for name, value in logs.items():
+            if name in ['batch', 'size']:
+                continue
+            summary = tf.Summary()
+            summary_value = summary.value.add()
+            summary_value.simple_value = value
+            summary_value.tag = name
+            self.writer.add_summary(summary, epoch)
+        self.writer.flush()

keras/constraints.py

@@ -1,22 +1,95 @@
 from __future__ import absolute_import
-import theano
-import theano.tensor as T
-import numpy as np
+from . import backend as K
 
-def maxnorm(m=2):
-    def maxnorm_wrap(p):
-        norms = T.sqrt(T.sum(T.sqr(p), axis=0))
-        desired = T.clip(norms, 0, m)
-        p = p * (desired / (1e-7 + norms))
+
+class Constraint(object):
+    def __call__(self, p):
         return p
-    return maxnorm_wrap
 
-def nonneg(p):
-    p *= T.ge(p, 0)
-    return p
+    def get_config(self):
+        return {"name": self.__class__.__name__}
 
-def identity(g):
-    return g
 
-def unitnorm(e):
-    return e / T.sqrt(T.sum(e**2, axis=-1, keepdims=True))
+class MaxNorm(Constraint):
+    '''Constrain the weights incident to each hidden unit to have a norm less than or equal to a desired value.
+
+    # Arguments
+        m: the maximum norm for the incoming weights.
+        axis: integer, axis along which to calculate weight norms. For instance,
+            in a `Dense` layer the weight matrix has shape (input_dim, output_dim),
+            set `axis` to `0` to constrain each weight vector of length (input_dim).
+            In a `MaxoutDense` layer the weight tensor has shape (nb_feature, input_dim, output_dim),
+            set `axis` to `1` to constrain each weight vector of length (input_dim),
+            i.e. constrain the filters incident to the `max` operation.
+            In a `Convolution2D` layer with the Theano backend, the weight tensor
+            has shape (nb_filter, stack_size, nb_row, nb_col), set `axis` to `[1,2,3]`
+            to constrain the weights of each filter tensor of size (stack_size, nb_row, nb_col).
+            In a `Convolution2D` layer with the TensorFlow backend, the weight tensor
+            has shape (nb_row, nb_col, stack_size, nb_filter), set `axis` to `[0,1,2]`
+            to constrain the weights of each filter tensor of size (nb_row, nb_col, stack_size).
+
+    # References
+        - [Dropout: A Simple Way to Prevent Neural Networks from Overfitting Srivastava, Hinton, et al. 2014](http://www.cs.toronto.edu/~rsalakhu/papers/srivastava14a.pdf)
+    '''
+    def __init__(self, m=2, axis=0):
+        self.m = m
+        self.axis = axis
+
+    def __call__(self, p):
+        norms = K.sqrt(K.sum(K.square(p), axis=self.axis, keepdims=True))
+        desired = K.clip(norms, 0, self.m)
+        p = p * (desired / (K.epsilon() + norms))
+        return p
+
+    def get_config(self):
+        return {"name": self.__class__.__name__,
+                "m": self.m,
+                "axis": self.axis}
+
+
+class NonNeg(Constraint):
+    '''Constrain the weights to be non-negative.
+    '''
+    def __call__(self, p):
+        p *= K.cast(p >= 0., K.floatx())
+        return p
+
+
+class UnitNorm(Constraint):
+    '''Constrain the weights incident to each hidden unit to have unit norm.
+
+    # Arguments
+        axis: integer, axis along which to calculate weight norms. For instance,
+            in a `Dense` layer the weight matrix has shape (input_dim, output_dim),
+            set `axis` to `0` to constrain each weight vector of length (input_dim).
+            In a `MaxoutDense` layer the weight tensor has shape (nb_feature, input_dim, output_dim),
+            set `axis` to `1` to constrain each weight vector of length (input_dim),
+            i.e. constrain the filters incident to the `max` operation.
+            In a `Convolution2D` layer with the Theano backend, the weight tensor
+            has shape (nb_filter, stack_size, nb_row, nb_col), set `axis` to `[1,2,3]`
+            to constrain the weights of each filter tensor of size (stack_size, nb_row, nb_col).
+            In a `Convolution2D` layer with the TensorFlow backend, the weight tensor
+            has shape (nb_row, nb_col, stack_size, nb_filter), set `axis` to `[0,1,2]`
+            to constrain the weights of each filter tensor of size (nb_row, nb_col, stack_size).
+    '''
+    def __init__(self, axis=0):
+        self.axis = axis
+
+    def __call__(self, p):
+        return p / (K.epsilon() + K.sqrt(K.sum(K.square(p), axis=self.axis, keepdims=True)))
+
+    def get_config(self):
+        return {"name": self.__class__.__name__,
+                "axis": self.axis}
+
+
+identity = Constraint
+maxnorm = MaxNorm
+nonneg = NonNeg
+unitnorm = UnitNorm
+
+from .utils.generic_utils import get_from_module
+
+
+def get(identifier, kwargs=None):
+    return get_from_module(identifier, globals(), 'constraint', instantiate=True, kwargs=kwargs)

keras/datasets/cifar.py

@@ -1,15 +1,16 @@
 # -*- coding: utf-8 -*-
 from __future__ import absolute_import
 import sys
-import six.moves.cPickle
+from six.moves import cPickle
 from six.moves import range
 
+
 def load_batch(fpath, label_key='labels'):
     f = open(fpath, 'rb')
     if sys.version_info < (3,):
-        d = six.moves.cPickle.load(f)
+        d = cPickle.load(f)
     else:
-        d = six.moves.cPickle.load(f, encoding="bytes")
+        d = cPickle.load(f, encoding="bytes")
         # decode utf8
         for k, v in d.items():
             del(d[k])

keras/datasets/cifar10.py

@@ -1,15 +1,15 @@
 from __future__ import absolute_import
 from .cifar import load_batch
-from .data_utils import get_file
+from ..utils.data_utils import get_file
 import numpy as np
 import os
 
+
 def load_data():
     dirname = "cifar-10-batches-py"
     origin = "http://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz"
     path = get_file(dirname, origin=origin, untar=True)
 
-    nb_test_samples = 10000
     nb_train_samples = 50000
 
     X_train = np.zeros((nb_train_samples, 3, 32, 32), dtype="uint8")
@@ -20,11 +20,11 @@ def load_data():
         data, labels = load_batch(fpath)
         X_train[(i-1)*10000:i*10000, :, :, :] = data
         y_train[(i-1)*10000:i*10000] = labels
-    
+
     fpath = os.path.join(path, 'test_batch')
     X_test, y_test = load_batch(fpath)
 
     y_train = np.reshape(y_train, (len(y_train), 1))
     y_test = np.reshape(y_test, (len(y_test), 1))
 
-    return (X_train, y_train), (X_test, y_test) 
+    return (X_train, y_train), (X_test, y_test)

keras/datasets/cifar100.py

@@ -1,9 +1,10 @@
 from __future__ import absolute_import
 from .cifar import load_batch
-from .data_utils import get_file
+from ..utils.data_utils import get_file
 import numpy as np
 import os
 
+
 def load_data(label_mode='fine'):
     if label_mode not in ['fine', 'coarse']:
         raise Exception('label_mode must be one of "fine" "coarse".')
@@ -24,4 +25,4 @@ def load_data(label_mode='fine'):
     y_train = np.reshape(y_train, (len(y_train), 1))
     y_test = np.reshape(y_test, (len(y_test), 1))
 
-    return (X_train, y_train), (X_test, y_test) 
+    return (X_train, y_train), (X_test, y_test)

keras/datasets/data_utils.py

@@ -1,44 +1,4 @@
-from __future__ import absolute_import
-from __future__ import print_function
-import tarfile, inspect, os
-from six.moves.urllib.request import urlretrieve
-from ..utils.generic_utils import Progbar
-
-def get_file(fname, origin, untar=False):
-    datadir = os.path.expanduser(os.path.join('~', '.keras', 'datasets'))
-    if not os.path.exists(datadir):
-        os.makedirs(datadir)
-
-    if untar:
-        untar_fpath = os.path.join(datadir, fname)
-        fpath = untar_fpath + '.tar.gz'
-    else:
-        fpath = os.path.join(datadir, fname)
-
-    try:
-        f = open(fpath)
-    except:
-        print('Downloading data from',  origin)
-
-        global progbar
-        progbar = None
-        def dl_progress(count, block_size, total_size):
-            global progbar
-            if progbar is None:
-                progbar = Progbar(total_size)
-            else:
-                progbar.update(count*block_size)
-
-        urlretrieve(origin, fpath, dl_progress)
-        progbar = None
-
-    if untar:
-        if not os.path.exists(untar_fpath):
-            print('Untaring file...')
-            tfile = tarfile.open(fpath, 'r:gz')
-            tfile.extractall(path=datadir)
-            tfile.close()
-        return untar_fpath
-
-    return fpath
+from ..utils.data_utils import *
+import warnings
 
+warnings.warn('data_utils has been moved to keras.utils.data_utils.')

keras/datasets/imdb.py

@@ -1,11 +1,15 @@
 from __future__ import absolute_import
-import six.moves.cPickle
+from six.moves import cPickle
 import gzip
-from .data_utils import get_file
-import random
+from ..utils.data_utils import get_file
 from six.moves import zip
+import numpy as np
+
+
+def load_data(path="imdb.pkl", nb_words=None, skip_top=0,
+              maxlen=None, test_split=0.2, seed=113,
+              start_char=1, oov_char=2, index_from=3):
 
-def load_data(path="imdb.pkl", nb_words=None, skip_top=0, maxlen=None, test_split=0.2, seed=113):
     path = get_file(path, origin="https://s3.amazonaws.com/text-datasets/imdb.pkl")
 
     if path.endswith(".gz"):
@@ -13,13 +17,18 @@ def load_data(path="imdb.pkl", nb_words=None, skip_top=0, maxlen=None, test_spli
     else:
         f = open(path, 'rb')
 
-    X, labels = six.moves.cPickle.load(f)
+    X, labels = cPickle.load(f)
     f.close()
 
-    random.seed(seed)
-    random.shuffle(X)
-    random.seed(seed)
-    random.shuffle(labels)
+    np.random.seed(seed)
+    np.random.shuffle(X)
+    np.random.seed(seed)
+    np.random.shuffle(labels)
+
+    if start_char is not None:
+        X = [[start_char] + [w + index_from for w in x] for x in X]
+    elif index_from:
+        X = [[w + index_from for w in x] for x in X]
 
     if maxlen:
         new_X = []
@@ -30,16 +39,31 @@ def load_data(path="imdb.pkl", nb_words=None, skip_top=0, maxlen=None, test_spli
                 new_labels.append(y)
         X = new_X
         labels = new_labels
-
+    if not X:
+        raise Exception('After filtering for sequences shorter than maxlen=' +
+                        str(maxlen) + ', no sequence was kept. '
+                        'Increase maxlen.')
     if not nb_words:
         nb_words = max([max(x) for x in X])
 
-    X = [[0 if (w >= nb_words or w < skip_top) else w for w in x] for x in X]
-    X_train = X[:int(len(X)*(1-test_split))]
-    y_train = labels[:int(len(X)*(1-test_split))]
+    # by convention, use 2 as OOV word
+    # reserve 'index_from' (=3 by default) characters: 0 (padding), 1 (start), 2 (OOV)
+    if oov_char is not None:
+        X = [[oov_char if (w >= nb_words or w < skip_top) else w for w in x] for x in X]
+    else:
+        nX = []
+        for x in X:
+            nx = []
+            for w in x:
+                if (w >= nb_words or w < skip_top):
+                    nx.append(w)
+            nX.append(nx)
+        X = nX
 
-    X_test = X[int(len(X)*(1-test_split)):]
-    y_test = labels[int(len(X)*(1-test_split)):]
+    X_train = X[:int(len(X) * (1 - test_split))]
+    y_train = labels[:int(len(X) * (1 - test_split))]
+
+    X_test = X[int(len(X) * (1 - test_split)):]
+    y_test = labels[int(len(X) * (1 - test_split)):]
 
     return (X_train, y_train), (X_test, y_test)
-

keras/datasets/mnist.py

@@ -1,9 +1,10 @@
 # -*- coding: utf-8 -*-
 import gzip
-from .data_utils import get_file
-import six.moves.cPickle
+from ..utils.data_utils import get_file
+from six.moves import cPickle
 import sys
 
+
 def load_data(path="mnist.pkl.gz"):
     path = get_file(path, origin="https://s3.amazonaws.com/img-datasets/mnist.pkl.gz")
 
@@ -13,10 +14,9 @@ def load_data(path="mnist.pkl.gz"):
         f = open(path, 'rb')
 
     if sys.version_info < (3,):
-        data = six.moves.cPickle.load(f)
+        data = cPickle.load(f)
     else:
-        data = six.moves.cPickle.load(f, encoding="bytes")
+        data = cPickle.load(f, encoding="bytes")
 
     f.close()
-
-    return data # (X_train, y_train), (X_test, y_test)
+    return data  # (X_train, y_train), (X_test, y_test)

keras/datasets/reuters.py

@@ -1,94 +1,29 @@
 # -*- coding: utf-8 -*-
 from __future__ import absolute_import
-from __future__ import print_function
-from .data_utils import get_file
-import string
-import random
-import os
-import six.moves.cPickle
+from ..utils.data_utils import get_file
+from six.moves import cPickle
 from six.moves import zip
-
-def make_reuters_dataset(path=os.path.join('datasets', 'temp', 'reuters21578'), min_samples_per_topic=15):
-    import re
-    from ..preprocessing.text import Tokenizer
-
-    wire_topics = []
-    topic_counts = {}
-    wire_bodies = []
-
-    for fname in os.listdir(path):
-        if 'sgm' in fname:
-            s = open(path + fname).read()
-            tag = '<TOPICS>'
-            while tag in s:
-                s = s[s.find(tag)+len(tag):]
-                topics = s[:s.find('</')]
-                
-                if topics and not '</D><D>' in topics:
-                    topic = topics.replace('<D>', '').replace('</D>', '')
-                    wire_topics.append(topic)
-                    topic_counts[topic] = topic_counts.get(topic, 0) + 1
-                else:
-                    continue
-
-                bodytag = '<BODY>'
-                body = s[s.find(bodytag)+len(bodytag):]
-                body = body[:body.find('</')]
-                wire_bodies.append(body)
-
-    # only keep most common topics
-    items = list(topic_counts.items())
-    items.sort(key = lambda x: x[1])
-    kept_topics = set()
-    for x in items:
-        print(x[0] + ': ' + str(x[1]))
-        if x[1] >= min_samples_per_topic:
-            kept_topics.add(x[0])
-    print('-')
-    print('Kept topics:', len(kept_topics))
-
-    # filter wires with rare topics
-    kept_wires = []
-    labels = []
-    topic_indexes = {}
-    for t, b in zip(wire_topics, wire_bodies):
-        if t in kept_topics:
-            if t not in topic_indexes:
-                topic_index = len(topic_indexes)
-                topic_indexes[t] = topic_index
-            else:
-                topic_index = topic_indexes[t]
-
-            labels.append(topic_index)
-            kept_wires.append(b)
-
-    # vectorize wires
-    tokenizer = Tokenizer()
-    tokenizer.fit_on_texts(kept_wires)
-    X = tokenizer.texts_to_sequences(kept_wires)
-
-    print('Sanity check:')
-    for w in ["banana", "oil", "chocolate", "the", "dsft"]:
-        print('...index of', w, ':', tokenizer.word_index.get(w))
-
-    dataset = (X, labels) 
-    print('-')
-    print('Saving...')
-    six.moves.cPickle.dump(dataset, open(os.path.join('datasets', 'data', 'reuters.pkl'), 'w'))
-    six.moves.cPickle.dump(tokenizer.word_index, open(os.path.join('datasets','data', 'reuters_word_index.pkl'), 'w'))
+import numpy as np
 
 
+def load_data(path="reuters.pkl", nb_words=None, skip_top=0,
+              maxlen=None, test_split=0.2, seed=113,
+              start_char=1, oov_char=2, index_from=3):
 
-def load_data(path="reuters.pkl", nb_words=None, skip_top=0, maxlen=None, test_split=0.2, seed=113):
     path = get_file(path, origin="https://s3.amazonaws.com/text-datasets/reuters.pkl")
     f = open(path, 'rb')
-
-    X, labels = six.moves.cPickle.load(f)
+    X, labels = cPickle.load(f)
     f.close()
-    random.seed(seed)
-    random.shuffle(X)
-    random.seed(seed)
-    random.shuffle(labels)
+
+    np.random.seed(seed)
+    np.random.shuffle(X)
+    np.random.seed(seed)
+    np.random.shuffle(labels)
+
+    if start_char is not None:
+        X = [[start_char] + [w + index_from for w in x] for x in X]
+    elif index_from:
+        X = [[w + index_from for w in x] for x in X]
 
     if maxlen:
         new_X = []
@@ -103,12 +38,25 @@ def load_data(path="reuters.pkl", nb_words=None, skip_top=0, maxlen=None, test_s
     if not nb_words:
         nb_words = max([max(x) for x in X])
 
-    X = [[0 if (w >= nb_words or w < skip_top) else w for w in x] for x in X]
-    X_train = X[:int(len(X)*(1-test_split))]
-    y_train = labels[:int(len(X)*(1-test_split))]
+    # by convention, use 2 as OOV word
+    # reserve 'index_from' (=3 by default) characters: 0 (padding), 1 (start), 2 (OOV)
+    if oov_char is not None:
+        X = [[oov_char if (w >= nb_words or w < skip_top) else w for w in x] for x in X]
+    else:
+        nX = []
+        for x in X:
+            nx = []
+            for w in x:
+                if (w >= nb_words or w < skip_top):
+                    nx.append(w)
+            nX.append(nx)
+        X = nX
 
-    X_test = X[int(len(X)*(1-test_split)):]
-    y_test = labels[int(len(X)*(1-test_split)):]
+    X_train = X[:int(len(X) * (1 - test_split))]
+    y_train = labels[:int(len(X) * (1 - test_split))]
+
+    X_test = X[int(len(X) * (1 - test_split)):]
+    y_test = labels[int(len(X) * (1 - test_split)):]
 
     return (X_train, y_train), (X_test, y_test)
 
@@ -116,9 +64,4 @@ def load_data(path="reuters.pkl", nb_words=None, skip_top=0, maxlen=None, test_s
 def get_word_index(path="reuters_word_index.pkl"):
     path = get_file(path, origin="https://s3.amazonaws.com/text-datasets/reuters_word_index.pkl")
     f = open(path, 'rb')
-    return six.moves.cPickle.load(f)
-
-
-if __name__ == "__main__":
-    make_reuters_dataset()
-    (X_train, y_train), (X_test, y_test) = load_data()
+    return cPickle.load(f)

keras/initializations.py

@@ -1,68 +1,107 @@
 from __future__ import absolute_import
-import theano
-import theano.tensor as T
 import numpy as np
+from . import backend as K
 
-from .utils.theano_utils import sharedX, shared_zeros
 
-def get_fans(shape):
-    fan_in = shape[0] if len(shape) == 2 else np.prod(shape[1:])
-    fan_out = shape[1] if len(shape) == 2 else shape[0]
+def get_fans(shape, dim_ordering='th'):
+    if len(shape) == 2:
+        fan_in = shape[0]
+        fan_out = shape[1]
+    elif len(shape) == 4 or len(shape) == 5:
+        # assuming convolution kernels (2D or 3D).
+        # TH kernel shape: (depth, input_depth, ...)
+        # TF kernel shape: (..., input_depth, depth)
+        if dim_ordering == 'th':
+            fan_in = np.prod(shape[1:])
+            fan_out = shape[0]
+        elif dim_ordering == 'tf':
+            fan_in = np.prod(shape[:-1])
+            fan_out = shape[-1]
+        else:
+            raise Exception('Invalid dim_ordering: ' + dim_ordering)
+    else:
+        # no specific assumptions
+        fan_in = np.sqrt(np.prod(shape))
+        fan_out = np.sqrt(np.prod(shape))
     return fan_in, fan_out
 
 
-def uniform(shape, scale=0.05):
-    return sharedX(np.random.uniform(low=-scale, high=scale, size=shape))
+def uniform(shape, scale=0.05, name=None):
+    return K.variable(np.random.uniform(low=-scale, high=scale, size=shape),
+                      name=name)
 
-def normal(shape, scale=0.05):
-    return sharedX(np.random.randn(*shape) * scale)
 
-def lecun_uniform(shape):
+def normal(shape, scale=0.05, name=None):
+    return K.variable(np.random.normal(loc=0.0, scale=scale, size=shape),
+                      name=name)
+
+
+def lecun_uniform(shape, name=None, dim_ordering='th'):
     ''' Reference: LeCun 98, Efficient Backprop
         http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf
     '''
-    fan_in, fan_out = get_fans(shape)
-    scale = 1./np.sqrt(fan_in)
-    return uniform(shape, scale)
+    fan_in, fan_out = get_fans(shape, dim_ordering=dim_ordering)
+    scale = np.sqrt(3. / fan_in)
+    return uniform(shape, scale, name=name)
 
-def glorot_normal(shape):
+
+def glorot_normal(shape, name=None, dim_ordering='th'):
     ''' Reference: Glorot & Bengio, AISTATS 2010
     '''
-    fan_in, fan_out = get_fans(shape)
+    fan_in, fan_out = get_fans(shape, dim_ordering=dim_ordering)
     s = np.sqrt(2. / (fan_in + fan_out))
-    return normal(shape, s)
+    return normal(shape, s, name=name)
 
-def glorot_uniform(shape):
-    fan_in, fan_out = get_fans(shape)
-    s = np.sqrt(2. / (fan_in + fan_out))
-    return uniform(shape, s)
-    
-def he_normal(shape):
+
+def glorot_uniform(shape, name=None, dim_ordering='th'):
+    fan_in, fan_out = get_fans(shape, dim_ordering=dim_ordering)
+    s = np.sqrt(6. / (fan_in + fan_out))
+    return uniform(shape, s, name=name)
+
+
+def he_normal(shape, name=None, dim_ordering='th'):
     ''' Reference:  He et al., http://arxiv.org/abs/1502.01852
     '''
-    fan_in, fan_out = get_fans(shape)
+    fan_in, fan_out = get_fans(shape, dim_ordering=dim_ordering)
     s = np.sqrt(2. / fan_in)
-    return normal(shape, s)
+    return normal(shape, s, name=name)
 
-def he_uniform(shape):
-    fan_in, fan_out = get_fans(shape)
-    s = np.sqrt(2. / fan_in)
-    return uniform(shape, s)
 
-def orthogonal(shape, scale=1.1):
-    ''' From Lasagne
+def he_uniform(shape, name=None, dim_ordering='th'):
+    fan_in, fan_out = get_fans(shape, dim_ordering=dim_ordering)
+    s = np.sqrt(6. / fan_in)
+    return uniform(shape, s, name=name)
+
+
+def orthogonal(shape, scale=1.1, name=None):
+    ''' From Lasagne. Reference: Saxe et al., http://arxiv.org/abs/1312.6120
     '''
     flat_shape = (shape[0], np.prod(shape[1:]))
     a = np.random.normal(0.0, 1.0, flat_shape)
     u, _, v = np.linalg.svd(a, full_matrices=False)
-    q = u if u.shape == flat_shape else v # pick the one with the correct shape
+    # pick the one with the correct shape
+    q = u if u.shape == flat_shape else v
     q = q.reshape(shape)
-    return sharedX(scale * q[:shape[0], :shape[1]])
+    return K.variable(scale * q[:shape[0], :shape[1]], name=name)
 
-def zero(shape):
-    return shared_zeros(shape)
+
+def identity(shape, scale=1, name=None):
+    if len(shape) != 2 or shape[0] != shape[1]:
+        raise Exception('Identity matrix initialization can only be used '
+                        'for 2D square matrices.')
+    else:
+        return K.variable(scale * np.identity(shape[0]), name=name)
+
+
+def zero(shape, name=None):
+    return K.zeros(shape, name=name)
+
+
+def one(shape, name=None):
+    return K.ones(shape, name=name)
 
 
 from .utils.generic_utils import get_from_module
-def get(identifier):
-    return get_from_module(identifier, globals(), 'initialization')
+def get(identifier, **kwargs):
+    return get_from_module(identifier, globals(),
+                           'initialization', kwargs=kwargs)

keras/layers/__init__.py

@@ -0,0 +1,8 @@
+from __future__ import absolute_import
+from .core import *
+from .convolutional import *
+from .recurrent import *
+from .normalization import *
+from .embeddings import *
+from .noise import *
+from .advanced_activations import *

keras/layers/advanced_activations.py

@@ -1,38 +1,286 @@
-from ..layers.core import Layer
-from ..utils.theano_utils import shared_zeros
+from .. import initializations
+from ..layers.core import MaskedLayer
+from .. import backend as K
+import numpy as np
 
-class LeakyReLU(Layer):
-    def __init__(self, alpha=0.3):
-        super(LeakyReLU,self).__init__()
+
+class LeakyReLU(MaskedLayer):
+    '''Special version of a Rectified Linear Unit
+    that allows a small gradient when the unit is not active:
+    `f(x) = alpha*x for x < 0`.
+
+    # Input shape
+        Arbitrary. Use the keyword argument `input_shape`
+        (tuple of integers, does not include the samples axis)
+        when using this layer as the first layer in a model.
+
+    # Output shape
+        Same shape as the input.
+
+    # Arguments
+        alpha: float >= 0. Negative slope coefficient.
+    '''
+    def __init__(self, alpha=0.3, **kwargs):
+        super(LeakyReLU, self).__init__(**kwargs)
         self.alpha = alpha
 
     def get_output(self, train):
         X = self.get_input(train)
-        return ((X + abs(X)) / 2.0) + self.alpha * ((X - abs(X)) / 2.0)
+        return K.relu(X, alpha=self.alpha)
 
     def get_config(self):
-        return {"name":self.__class__.__name__,
-            "alpha":self.alpha}
+        config = {'name': self.__class__.__name__,
+                  'alpha': self.alpha}
+        base_config = super(LeakyReLU, self).get_config()
+        return dict(list(base_config.items()) + list(config.items()))
 
 
-class PReLU(Layer):
+class PReLU(MaskedLayer):
     '''
-        Reference: 
-            Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
-                http://arxiv.org/pdf/1502.01852v1.pdf
+    # Input shape
+        Arbitrary. Use the keyword argument `input_shape`
+        (tuple of integers, does not include the samples axis)
+        when using this layer as the first layer in a model.
+
+    # Output shape
+        Same shape as the input.
+
+    # Arguments:
+        init: initialization function for the weights.
+        weights: initial weights, as a list of a single numpy array.
+
+    # References:
+        - [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](http://arxiv.org/pdf/1502.01852v1.pdf)
     '''
-    def __init__(self, input_shape):
-        super(PReLU,self).__init__()
-        self.alphas = shared_zeros(input_shape)
-        self.params = [self.alphas]
-        self.input_shape = input_shape
+    def __init__(self, init='zero', weights=None, **kwargs):
+        self.init = initializations.get(init)
+        self.initial_weights = weights
+        super(PReLU, self).__init__(**kwargs)
+
+    def build(self):
+        input_shape = self.input_shape[1:]
+        self.alphas = self.init(input_shape,
+                                name='{}_alphas'.format(self.name))
+        self.trainable_weights = [self.alphas]
+
+        if self.initial_weights is not None:
+            self.set_weights(self.initial_weights)
+            del self.initial_weights
 
     def get_output(self, train):
         X = self.get_input(train)
-        pos = ((X + abs(X)) / 2.0)
-        neg = self.alphas * ((X - abs(X)) / 2.0)
+        pos = K.relu(X)
+        neg = self.alphas * (X - abs(X)) * 0.5
         return pos + neg
 
     def get_config(self):
-        return {"name":self.__class__.__name__,
-        "input_shape":self.input_shape}
+        config = {'name': self.__class__.__name__,
+                  'init': self.init.__name__}
+        base_config = super(PReLU, self).get_config()
+        return dict(list(base_config.items()) + list(config.items()))
+
+
+class ELU(MaskedLayer):
+    '''
+    # Input shape
+        Arbitrary. Use the keyword argument `input_shape`
+        (tuple of integers, does not include the samples axis)
+        when using this layer as the first layer in a model.
+
+    # Output shape
+        Same shape as the input.
+
+    # Arguments
+        alpha: scale for the negative factor.
+
+    # References
+        - [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)](http://arxiv.org/pdf/1511.07289v1.pdf)
+    '''
+    def __init__(self, alpha=1.0, **kwargs):
+        super(ELU, self).__init__(**kwargs)
+        self.alpha = alpha
+
+    def get_output(self, train):
+        X = self.get_input(train)
+        pos = K.relu(X)
+        neg = (X - abs(X)) * 0.5
+        return pos + self.alpha * (K.exp(neg) - 1.)
+
+    def get_config(self):
+        config = {'name': self.__class__.__name__,
+                  'alpha': self.alpha}
+        base_config = super(ELU, self).get_config()
+        return dict(list(base_config.items()) + list(config.items()))
+
+
+class ParametricSoftplus(MaskedLayer):
+    '''Parametric Softplus of the form: alpha * log(1 + exp(beta * X))
+
+    # Input shape
+        Arbitrary. Use the keyword argument `input_shape`
+        (tuple of integers, does not include the samples axis)
+        when using this layer as the first layer in a model.
+
+    # Output shape
+        Same shape as the input.
+
+    # Arguments
+        alpha_init: float. Initial value of the alpha weights.
+        beta_init: float. Initial values of the beta weights.
+        weights: initial weights, as a list of 2 numpy arrays.
+
+    # References:
+        - [Inferring Nonlinear Neuronal Computation Based on Physiologically Plausible Inputs](http://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1003143)
+    '''
+    def __init__(self, alpha_init=0.2, beta_init=5.0,
+                 weights=None, **kwargs):
+        self.alpha_init = alpha_init
+        self.beta_init = beta_init
+        self.initial_weights = weights
+        super(ParametricSoftplus, self).__init__(**kwargs)
+
+    def build(self):
+        input_shape = self.input_shape[1:]
+        self.alphas = K.variable(self.alpha_init * np.ones(input_shape),
+                                 name='{}_alphas'.format(self.name))
+        self.betas = K.variable(self.beta_init * np.ones(input_shape),
+                                name='{}_betas'.format(self.name))
+        self.trainable_weights = [self.alphas, self.betas]
+
+        if self.initial_weights is not None:
+            self.set_weights(self.initial_weights)
+            del self.initial_weights
+
+    def get_output(self, train):
+        X = self.get_input(train)
+        return K.softplus(self.betas * X) * self.alphas
+
+    def get_config(self):
+        config = {'name': self.__class__.__name__,
+                  'alpha_init': self.alpha_init,
+                  'beta_init': self.beta_init}
+        base_config = super(ParametricSoftplus, self).get_config()
+        return dict(list(base_config.items()) + list(config.items()))
+
+
+class ThresholdedLinear(MaskedLayer):
+    '''Thresholded Linear Activation.
+
+    # Input shape
+        Arbitrary. Use the keyword argument `input_shape`
+        (tuple of integers, does not include the samples axis)
+        when using this layer as the first layer in a model.
+
+    # Output shape
+        Same shape as the input.
+
+    # Arguments
+        theta: float >= 0. Threshold location of activation.
+
+    # References
+        [Zero-Bias Autoencoders and the Benefits of Co-Adapting Features](http://arxiv.org/pdf/1402.3337.pdf)
+    '''
+    def __init__(self, theta=1.0, **kwargs):
+        super(ThresholdedLinear, self).__init__(**kwargs)
+        self.theta = theta
+
+    def get_output(self, train):
+        X = self.get_input(train)
+        return K.switch(K.abs(X) < self.theta, 0, X)
+
+    def get_config(self):
+        config = {'name': self.__class__.__name__,
+                  'theta': self.theta}
+        base_config = super(ThresholdedLinear, self).get_config()
+        return dict(list(base_config.items()) + list(config.items()))
+
+
+class ThresholdedReLU(MaskedLayer):
+    '''Thresholded Rectified Activation.
+
+    # Input shape
+        Arbitrary. Use the keyword argument `input_shape`
+        (tuple of integers, does not include the samples axis)
+        when using this layer as the first layer in a model.
+
+    # Output shape
+        Same shape as the input.
+
+    # Arguments
+        theta: float >= 0. Threshold location of activation.
+
+    # References
+        [Zero-Bias Autoencoders and the Benefits of Co-Adapting Features](http://arxiv.org/pdf/1402.3337.pdf)
+    '''
+    def __init__(self, theta=1.0, **kwargs):
+        super(ThresholdedReLU, self).__init__(**kwargs)
+        self.theta = theta
+
+    def get_output(self, train):
+        X = self.get_input(train)
+        return K.switch(X > self.theta, X, 0)
+
+    def get_config(self):
+        config = {'name': self.__class__.__name__,
+                  'theta': self.theta}
+        base_config = super(ThresholdedReLU, self).get_config()
+        return dict(list(base_config.items()) + list(config.items()))
+
+
+class SReLU(MaskedLayer):
+    '''SReLU
+
+    # Input shape
+        Arbitrary. Use the keyword argument `input_shape`
+        (tuple of integers, does not include the samples axis)
+        when using this layer as the first layer in a model.
+
+    # Output shape
+        Same shape as the input.
+
+    # Arguments
+        t_left_init: initialization function for the left part intercept
+        a_left_init: initialization function for the left part slope
+        t_right_init: initialization function for the right part intercept
+        a_right_init: initialization function for the right part slope
+
+    # References
+        [Deep Learning with S-shaped Rectified Linear Activation Units](http://arxiv.org/abs/1512.07030)
+    '''
+    def __init__(self, t_left_init='zero', a_left_init='glorot_uniform',
+                 t_right_init='glorot_uniform', a_right_init='one', **kwargs):
+        self.t_left_init = initializations.get(t_left_init)
+        self.a_left_init = initializations.get(a_left_init)
+        self.t_right_init = initializations.get(t_right_init)
+        self.a_right_init = initializations.get(a_right_init)
+        super(SReLU, self).__init__(**kwargs)
+
+    def build(self):
+        input_shape = self.input_shape[1:]
+        self.t_left = self.t_left_init(input_shape,
+                                       name='{}_t_left'.format(self.name))
+        self.a_left = self.a_left_init(input_shape,
+                                       name='{}_a_left'.format(self.name))
+        self.t_right = self.t_right_init(input_shape,
+                                         name='{}_t_right'.format(self.name))
+        self.a_right = self.a_right_init(input_shape,
+                                         name='{}_a_right'.format(self.name))
+        # ensure the the right part is always to the right of the left
+        self.t_right_actual = self.t_left + abs(self.t_right)
+        self.trainable_weights = [self.t_left, self.a_left,
+                                  self.t_right, self.a_right]
+
+    def get_output(self, train=False):
+        X = self.get_input(train)
+        Y_left_and_center = self.t_left + K.relu(X - self.t_left,
+                                                 self.a_left,
+                                                 self.t_right_actual - self.t_left)
+        Y_right = K.relu(X - self.t_right_actual) * self.a_right
+        return Y_left_and_center + Y_right
+
+    def get_config(self):
+        return {'name': self.__class__.__name__,
+                't_left_init': self.t_left_init.__name__,
+                'a_left_init': self.a_left_init.__name__,
+                't_right_init': self.t_right_init.__name__,
+                'a_right_init': self.a_right_init.__name__}

keras/layers/containers.py

@@ -1,53 +1,150 @@
 # -*- coding: utf-8 -*-
 from __future__ import absolute_import
+from __future__ import print_function
 
-import theano.tensor as T
-from ..layers.core import Layer
+from collections import OrderedDict
+from .. import backend as K
+from ..layers.core import Layer, Merge, Siamese, SiameseHead
 from six.moves import range
 
-def ndim_tensor(ndim):
-    if ndim == 2:
-        return T.matrix()
-    elif ndim == 3:
-        return T.tensor3()
-    elif ndim == 4:
-        return T.tensor4()
-    return T.matrix()
 
 class Sequential(Layer):
+    '''The Sequential container is a linear stack of layers.
+    Apart from the `add` methods and the `layers` constructor argument,
+    the API is identical to that of the `Layer` class.
+
+    This class is also the basis for the `keras.models.Sequential` model.
+
+    # Arguments
+        layers: list of layers to be added to the container.
+    '''
     def __init__(self, layers=[]):
         self.layers = []
-        self.params = []
-        self.regularizers = []
-        self.constraints = []
+        self.layer_cache = {}
+        self.shape_cache = {}
         for layer in layers:
             self.add(layer)
+        self._cache_enabled = True
 
-    def connect(self, layer):
-        self.layers[0].previous = layer
+    @property
+    def cache_enabled(self):
+        return self._cache_enabled
+
+    @cache_enabled.setter
+    def cache_enabled(self, value):
+        self._cache_enabled = value
+        for l in self.layers:
+            l.cache_enabled = value
+
+    @property
+    def layer_cache(self):
+        return super(Sequential, self).layer_cache
+
+    @layer_cache.setter
+    def layer_cache(self, value):
+        self._layer_cache = value
+        for layer in self.layers:
+            layer.layer_cache = self._layer_cache
+
+    @property
+    def shape_cache(self):
+        return super(Sequential, self).shape_cache
+
+    @shape_cache.setter
+    def shape_cache(self, value):
+        self._shape_cache = value
+        for layer in self.layers:
+            layer.shape_cache = self._shape_cache
+
+    def set_previous(self, layer, reset_weights=True):
+        self.layers[0].set_previous(layer, reset_weights)
+
+    def clear_previous(self, reset_weights=True):
+        self.layers[0].clear_previous(reset_weights)
 
     def add(self, layer):
+        layer.layer_cache = self.layer_cache
+        layer.shape_cache = self.shape_cache
         self.layers.append(layer)
         if len(self.layers) > 1:
-            self.layers[-1].connect(self.layers[-2])
-        
-        params, regularizers, constraints = layer.get_params()
-        self.params += params
-        self.regularizers += regularizers
-        self.constraints += constraints
+            self.layers[-1].set_previous(self.layers[-2])
+            if not hasattr(self.layers[0], 'input'):
+                self.set_input()
+
+    @property
+    def trainable_weights(self):
+        weights = []
+        for l in self.layers:
+            if l.trainable:
+                weights += l.get_params()[0]
+        return weights
+
+    @property
+    def regularizers(self):
+        regularizers = []
+        for l in self.layers:
+            if l.trainable:
+                regularizers += l.get_params()[1]
+        return regularizers
+
+    @property
+    def constraints(self):
+        constraints = []
+        for l in self.layers:
+            if l.trainable:
+                constraints += l.get_params()[2]
+        return constraints
+
+    @property
+    def updates(self):
+        updates = []
+        for l in self.layers:
+            if l.trainable:
+                updates += l.get_params()[3]
+        return updates
+
+    @property
+    def state_updates(self):
+        """
+        Return the `updates` from all layers in the sequence that are
+        stateful.  This is useful for separating _training_ updates and
+        _prediction_ updates for when we need to update a layers internal state
+        during a stateful prediction.
+        """
+        state_updates = []
+        for l in self.layers:
+            if getattr(l, 'stateful', False):
+                state_updates += l.get_params()[3]
+        return state_updates
+
+    def reset_states(self):
+        for l in self.layers:
+            if hasattr(l, 'reset_states') and getattr(l, 'stateful', False):
+                l.reset_states()
+
+    @property
+    def output_shape(self):
+        return self.layers[-1].output_shape
 
     def get_output(self, train=False):
         return self.layers[-1].get_output(train)
 
+    def set_input(self):
+        for l in self.layers:
+            if hasattr(l, 'input'):
+                ndim = K.ndim(l.input)
+                self.layers[0].input = K.placeholder(ndim=ndim)
+                break
+
     def get_input(self, train=False):
         if not hasattr(self.layers[0], 'input'):
-            for l in self.layers:
-                if hasattr(l, 'input'):
-                    break
-            ndim = l.input.ndim 
-            self.layers[0].input = ndim_tensor(ndim)
+            self.set_input()
         return self.layers[0].get_input(train)
 
+    @property
+    def input_shape(self):
+        return self.layers[0].input_shape
+
     @property
     def input(self):
         return self.get_input()
@@ -59,11 +156,461 @@ class Sequential(Layer):
         return weights
 
     def set_weights(self, weights):
-        for i in range(len(self.layers)):
-            nb_param = len(self.layers[i].params)
-            self.layers[i].set_weights(weights[:nb_param])
+        for layer in self.layers:
+            nb_param = len(layer.get_weights())
+            layer.set_weights(weights[:nb_param])
             weights = weights[nb_param:]
 
     def get_config(self):
-        return {"name":self.__class__.__name__,
-            "layers":[layer.get_config() for layer in self.layers]}
+        return {'name': self.__class__.__name__,
+                'layers': [layer.get_config() for layer in self.layers]}
+
+    def count_params(self):
+        return sum([layer.count_params() for layer in self.layers])
+
+
+class Graph(Layer):
+    '''Implement a NN graph with arbitrary layer connections,
+    arbitrary number of inputs and arbitrary number of outputs.
+
+    This class is also the basis for the `keras.models.Graph` model.
+
+    Note: `Graph` can only be used as a layer
+    (connect, input, get_input, get_output)
+    when it has exactly one input and one output.
+    '''
+    def __init__(self):
+        self.namespace = set()  # strings
+        self.nodes = OrderedDict()  # layer-like
+        self.inputs = {}  # layer-like
+        self.input_order = []  # strings
+        self.outputs = {}  # layer-like
+        self.output_order = []  # strings
+        self.input_config = []  # dicts
+        self.output_config = []  # dicts
+        self.node_config = []  # dicts
+        self.layer_cache = {}
+        self.shape_cache = {}
+        self._cache_enabled = True
+
+    def __call__(self, X, mask=None, train=False):
+        if type(X) != dict:
+            return super(Graph, self).__call__(X, mask, train)
+        else:
+            # turn off layer cache temporarily
+            tmp_cache_enabled = self.cache_enabled
+            self.cache_enabled = False
+            # create a temporary layer for each input
+            tmp_previous = {}
+            for name, input in self.inputs.items():
+                layer = Layer(batch_input_shape=input.input_shape)
+                layer.input = X[name]
+                if hasattr(self, 'get_input_mask'):
+                    layer.get_input_mask = lambda _: mask[name]
+                # set temporary previous
+                if hasattr(input, 'previous'):
+                    tmp_previous[name] = input.previous
+                input.set_previous(layer, False)
+            Y = self.get_output(train=train)
+            # return previous to what it was
+            for name, input in self.inputs.items():
+                if name in tmp_previous:
+                    input.set_previous(tmp_previous[name], False)
+                else:
+                    input.clear_previous(False)
+            self.cache_enabled = tmp_cache_enabled
+            return Y
+
+    @property
+    def cache_enabled(self):
+        return self._cache_enabled
+
+    @cache_enabled.setter
+    def cache_enabled(self, value):
+        self._cache_enabled = value
+        for l in self.nodes.values():
+            l.cache_enabled = value
+        for l in self.inputs.values():
+            l.cache_enabled = value
+
+    @property
+    def layer_cache(self):
+        return super(Graph, self).layer_cache
+
+    @layer_cache.setter
+    def layer_cache(self, value):
+        self._layer_cache = value
+        for layer in self.nodes.values():
+            layer.layer_cache = self._layer_cache
+        for layer in self.inputs.values():
+            layer.layer_cache = self._layer_cache
+
+    @property
+    def shape_cache(self):
+        return super(Graph, self).shape_cache
+
+    @shape_cache.setter
+    def shape_cache(self, value):
+        self._shape_cache = value
+        for layer in self.nodes.values():
+            layer.shape_cache = self._shape_cache
+        for layer in self.inputs.values():
+            layer.shape_cache = self._shape_cache
+
+    @property
+    def nb_input(self):
+        return len(self.inputs)
+
+    @property
+    def nb_output(self):
+        return len(self.outputs)
+
+    @property
+    def trainable_weights(self):
+        weights = []
+        for l in self.nodes.values():
+            if l.trainable:
+                weights += l.get_params()[0]
+        return weights
+
+    @property
+    def regularizers(self):
+        regularizers = []
+        for l in self.nodes.values():
+            if l.trainable:
+                regularizers += l.get_params()[1]
+        return regularizers
+
+    @property
+    def constraints(self):
+        constraints = []
+        for l in self.nodes.values():
+            if l.trainable:
+                constraints += l.get_params()[2]
+        return constraints
+
+    @property
+    def updates(self):
+        updates = []
+        for l in self.nodes.values():
+            if l.trainable:
+                updates += l.get_params()[3]
+        return updates
+
+    @property
+    def state_updates(self):
+        """
+        Return the `updates` from all nodes in that graph for nodes that are
+        stateful.  This is useful for separating _training_ updates and
+        _prediction_ updates for when we need to update a layers internal state
+        during a stateful prediction.
+        """
+        state_updates = []
+        for l in self.nodes.values():
+            if getattr(l, 'stateful', False):
+                state_updates += l.get_params()[3]
+        return state_updates
+
+    def reset_states(self):
+        for l in self.nodes.values():
+            if hasattr(l, 'reset_states') and getattr(l, 'stateful', False):
+                l.reset_states()
+
+    def set_previous(self, layer, connection_map={}, reset_weights=True):
+        if self.nb_input != layer.nb_output:
+            raise Exception('Cannot connect layers: '
+                            'input count does not match output count.')
+        if self.nb_input == 1:
+            self.inputs[self.input_order[0]].set_previous(layer, reset_weights)
+        else:
+            if not connection_map:
+                raise Exception('Cannot attach multi-input layer: '
+                                'no connection_map provided.')
+            for k, v in connection_map.items():
+                if k in self.inputs and v in layer.outputs:
+                    self.inputs[k].set_previous(layer.outputs[v], reset_weights)
+                else:
+                    raise Exception('Invalid connection map.')
+
+    def clear_previous(self, reset_weights=True):
+        for k in self.inputs.values():
+            k.clear_previous(reset_weights)
+
+    @property
+    def input_shape(self):
+        if self.nb_input == 1:
+            # return tuple
+            return self.inputs[self.input_order[0]].input_shape
+        else:
+            # return dictionary mapping input names to shape tuples
+            return dict([(k, v.input_shape) for k, v in self.inputs.items()])
+
+    def get_input(self, train=False):
+        if len(self.inputs) == len(self.outputs) == 1:
+            return self.inputs[self.input_order[0]].get_input(train)
+        else:
+            return dict([(k, v.get_input(train)) for k, v in self.inputs.items()])
+
+    @property
+    def input(self):
+        return self.get_input()
+
+    @property
+    def output_shape(self):
+        if self.nb_output == 1:
+            # return tuple
+            return self.outputs[self.output_order[0]].output_shape
+        else:
+            # return dictionary mapping output names to shape tuples
+            return dict([(k, v.output_shape) for k, v in self.outputs.items()])
+
+    def get_output(self, train=False):
+        if len(self.inputs) == len(self.outputs) == 1:
+            return self.outputs[self.output_order[0]].get_output(train)
+        else:
+            return dict([(k, v.get_output(train)) for k, v in self.outputs.items()])
+
+    def add_input(self, name, input_shape=None,
+                  batch_input_shape=None, dtype='float'):
+        '''Add an input to the graph.
+
+        # Arguments:
+            name: string. The name of the new input. Must be unique in the graph.
+            input_shape: a tuple of integers, the expected shape of the input samples.
+                Does not include the batch size.
+            batch_input_shape: a tuple of integers, the expected shape of the
+                whole input batch, including the batch size.
+            dtype: 'float' or 'int'.
+        '''
+        if name in self.namespace:
+            raise Exception('Duplicate node identifier: ' + name)
+        self.namespace.add(name)
+        self.input_order.append(name)
+        layer = Layer(name=name)  # empty layer
+        if input_shape:
+            layer.set_input_shape((None,) + tuple(input_shape))
+        elif batch_input_shape:
+            layer.set_input_shape(batch_input_shape)
+        if dtype == 'float':
+            layer.input = K.placeholder(shape=layer.input_shape, name=name)
+        else:
+            if (input_shape and len(input_shape) == 1) or (batch_input_shape and len(batch_input_shape) == 2):
+                layer.input = K.placeholder(shape=layer.input_shape,
+                                            dtype='int32',
+                                            name=name)
+            else:
+                raise Exception('Type "int" can only be used with ndim==2 (Embedding).')
+        self.inputs[name] = layer
+        config = {'name': name, 'dtype': dtype}
+        if batch_input_shape:
+            config['batch_input_shape'] = batch_input_shape
+        else:
+            config['input_shape'] = input_shape
+        self.input_config.append(config)
+
+    def add_node(self, layer, name, input=None, inputs=[],
+                 merge_mode='concat', concat_axis=-1, dot_axes=-1,
+                 create_output=False):
+        '''Add a node in the graph. It can be connected to multiple
+        inputs, which will first be merged into one tensor
+        according to the mode specified.
+
+        # Arguments
+            layer: the layer at the node.
+            name: name for the node.
+            input: when connecting the layer to a single input,
+                this is the name of the incoming node.
+            inputs: when connecting the layer to multiple inputs,
+                this is a list of names of incoming nodes.
+            merge_mode: one of {concat, sum, dot, ave, mul}
+            concat_axis: when `merge_mode=='concat'`, this is the
+                input concatenation axis.
+            dot_axes: when `merge_mode='dot'`, this is the contraction axes
+                specification; see the `Merge layer for details.
+            create_output: boolean. Set this to `True` if you want the output
+                of your node to be an output of the graph.
+        '''
+        if name in self.namespace:
+            raise Exception('Duplicate node identifier: ' + name)
+        layer.name = name
+        if input:
+            if input not in self.namespace:
+                raise Exception('Unknown node/input identifier: ' + input)
+            if input in self.nodes:
+                layer.set_previous(self.nodes[input])
+            elif input in self.inputs:
+                layer.set_previous(self.inputs[input])
+        if inputs:
+            to_merge = []
+            for n in inputs:
+                if n in self.nodes:
+                    to_merge.append(self.nodes[n])
+                elif n in self.inputs:
+                    to_merge.append(self.inputs[n])
+                else:
+                    raise Exception('Unknown identifier: ' + n)
+            merge = Merge(to_merge, mode=merge_mode,
+                          concat_axis=concat_axis, dot_axes=dot_axes)
+            layer.set_previous(merge)
+
+        self.namespace.add(name)
+        layer.layer_cache = self.layer_cache
+        layer.shape_cache = self.shape_cache
+        self.nodes[name] = layer
+        self.node_config.append({'name': name,
+                                 'input': input,
+                                 'inputs': inputs,
+                                 'merge_mode': merge_mode,
+                                 'concat_axis': concat_axis,
+                                 'dot_axes': dot_axes,
+                                 'create_output': create_output})
+
+        if create_output:
+            self.add_output(name, input=name)
+
+    def add_shared_node(self, layer, name, inputs=[], merge_mode=None,
+                        concat_axis=-1, dot_axes=-1, outputs=[],
+                        create_output=False):
+        '''Used to share a same layer across multiple nodes.
+
+        Supposed, for instance, that you want to apply one same `Dense`
+        layer after to the output of two different nodes.
+        You can then add the `Dense` layer as a shared node.
+
+        # Arguments
+            layer: The layer to be shared across multiple inputs
+            name: Name of the shared node
+            inputs: List of names of input nodes
+            merge_mode: Same meaning as `merge_mode` argument of `add_node()`
+            concat_axis: Same meaning as `concat_axis` argument of `add_node()`
+            dot_axes: Same meaning as `dot_axes` argument of `add_node()`
+            outputs: Used when `merge_mode=None`. Names for the output nodes.
+            create_output: Same meaning as `create_output` argument of `add_node()`.
+        '''
+        if name in self.namespace:
+            raise Exception('Duplicate node identifier: ' + name)
+        for o in outputs:
+            if o in self.namespace:
+                raise Exception('Duplicate node identifier: ' + o)
+        if merge_mode:
+            if merge_mode not in {'sum', 'ave', 'mul', 'dot', 'cos', 'concat', 'join'}:
+                raise Exception('Invalid merge mode')
+        layers = []
+        for i in range(len(inputs)):
+            input = inputs[i]
+            if input in self.nodes:
+                n = self.nodes[input]
+                if n.__class__.__name__ == 'Siamese':
+                    if n.merge_mode is None:
+                        for j in range(len(n.inputs)):
+                            sh = SiameseHead(j)
+                            sh.previous = n
+                            layers.append(sh)
+                    else:
+                        layers.append(n)
+                else:
+                    layers.append(n)
+            elif input in self.inputs:
+                n = self.inputs[input]
+                layers.append(n)
+            else:
+                raise Exception('Unknown identifier: ' + input)
+        s = Siamese(layer, layers, merge_mode,
+                    concat_axis=concat_axis,
+                    dot_axes=dot_axes,
+                    is_graph=True)
+        self.namespace.add(name)
+        self.nodes[name] = s
+        self.node_config.append({'name': name,
+                                 'inputs': inputs,
+                                 'merge_mode': merge_mode,
+                                 'concat_axis': concat_axis,
+                                 'dot_axes': dot_axes,
+                                 'create_output': create_output if merge_mode else False})
+        if not merge_mode:
+            for i in range(len(outputs)):
+                sh = SiameseHead(i)
+                sh.previous = s
+                sh_name = outputs[i]
+                sh.name = sh_name
+                self.namespace.add(sh_name)
+                self.nodes[sh_name] = sh
+                self.node_config.append({'name': sh_name,
+                                         'inputs': [name],
+                                         'create_output': create_output})
+                if create_output:
+                    self.add_output(sh_name, input=sh_name)
+
+        if create_output and merge_mode:
+            if merge_mode == 'join':
+                raise Exception('Output can not be of type OrderedDict')
+            self.add_output(name, input=name)
+
+    def add_output(self, name, input=None, inputs=[],
+                   merge_mode='concat', concat_axis=-1, dot_axes=-1):
+        '''Add an output to the graph.
+
+        This output can merge several node outputs into a single output.
+
+        # Arguments
+            name: name of the output.
+            input: when connecting the layer to a single input,
+                this is the name of the incoming node.
+            inputs: when connecting the layer to multiple inputs,
+                this is a list of names of incoming nodes.
+            merge_mode: one of {concat, sum, dot, ave, mul}
+            concat_axis: when `merge_mode=='concat'`, this is the
+                input concatenation axis.
+            dot_axes: when `merge_mode='dot'`, this is the contraction axes
+                specification; see the `Merge layer for details.
+        '''
+        if name in self.output_order:
+            raise Exception('Duplicate output identifier: ' + name)
+        if input:
+            if input not in self.namespace:
+                raise Exception('Unknown node/input identifier: ' + input)
+            if input in self.nodes:
+                self.outputs[name] = self.nodes[input]
+            elif input in self.inputs:
+                self.outputs[name] = self.inputs[input]
+        if inputs:
+            to_merge = []
+            for n in inputs:
+                if n not in self.nodes:
+                    raise Exception('Unknown identifier: ' + n)
+                to_merge.append(self.nodes[n])
+            merge = Merge(to_merge, mode=merge_mode,
+                          concat_axis=concat_axis, dot_axes=dot_axes)
+            self.outputs[name] = merge
+
+        self.output_order.append(name)
+        self.output_config.append({'name': name,
+                                   'input': input,
+                                   'inputs': inputs,
+                                   'merge_mode': merge_mode,
+                                   'concat_axis': concat_axis,
+                                   'dot_axes': dot_axes})
+
+    def get_config(self):
+        return {'name': self.__class__.__name__,
+                'input_config': self.input_config,
+                'node_config': self.node_config,
+                'output_config': self.output_config,
+                'input_order': self.input_order,
+                'output_order': self.output_order,
+                'nodes': dict([(c['name'], self.nodes[c['name']].get_config()) for c in self.node_config])}
+
+    def count_params(self):
+        return sum([layer.count_params() for layer in self.nodes.values()])
+
+    def get_weights(self):
+        weights = []
+        for layer in self.nodes.values():
+            weights += layer.get_weights()
+        return weights
+
+    def set_weights(self, weights):
+        for layer in self.nodes.values():
+            nb_param = len(layer.get_weights())
+            layer.set_weights(weights[:nb_param])
+            weights = weights[nb_param:]

keras/layers/embeddings.py

@@ -1,104 +1,125 @@
 from __future__ import absolute_import
-import theano
-import theano.tensor as T
 
-from .. import activations, initializations
+from .. import backend as K
+from .. import initializations, regularizers, constraints
 from ..layers.core import Layer
-from ..constraints import unitnorm
 
 
 class Embedding(Layer):
-    '''
-        Turn positive integers (indexes) into denses vectors of fixed size. 
-        eg. [[4], [20]] -> [[0.25, 0.1], [0.6, -0.2]]
+    '''Turn positive integers (indexes) into dense vectors of fixed size.
+    eg. [[4], [20]] -> [[0.25, 0.1], [0.6, -0.2]]
 
-        @input_dim: size of vocabulary (highest input integer + 1)
-        @out_dim: size of dense representation
+    This layer can only be used as the first layer in a model.
+
+    # Input shape
+        2D tensor with shape: `(nb_samples, sequence_length)`.
+
+    # Output shape
+        3D tensor with shape: `(nb_samples, sequence_length, output_dim)`.
+
+    # Arguments
+      input_dim: int >= 0. Size of the vocabulary, ie.
+          1 + maximum integer index occurring in the input data.
+      output_dim: int >= 0. Dimension of the dense embedding.
+      init: name of initialization function for the weights
+          of the layer (see: [initializations](../initializations.md)),
+          or alternatively, Theano function to use for weights initialization.
+          This parameter is only relevant if you don't pass a `weights` argument.
+      weights: list of numpy arrays to set as initial weights.
+          The list should have 1 element, of shape `(input_dim, output_dim)`.
+      W_regularizer: instance of the [regularizers](../regularizers.md) module
+        (eg. L1 or L2 regularization), applied to the embedding matrix.
+      W_constraint: instance of the [constraints](../constraints.md) module
+          (eg. maxnorm, nonneg), applied to the embedding matrix.
+      mask_zero: Whether or not the input value 0 is a special "padding"
+          value that should be masked out.
+          This is useful for [recurrent layers](recurrent.md) which may take
+          variable length input. If this is `True` then all subsequent layers
+          in the model need to support masking or an exception will be raised.
+      input_length: Length of input sequences, when it is constant.
+          This argument is required if you are going to connect
+          `Flatten` then `Dense` layers upstream
+          (without it, the shape of the dense outputs cannot be computed).
+      dropout: float between 0 and 1. Fraction of the embeddings to drop.
+
+    # References
+        - [A Theoretically Grounded Application of Dropout in Recurrent Neural Networks](http://arxiv.org/abs/1512.05287)
     '''
-    def __init__(self, input_dim, output_dim, init='uniform', weights=None, W_regularizer=None, W_constraint=None):
-        super(Embedding,self).__init__()
-        self.init = initializations.get(init)
+    input_ndim = 2
+
+    def __init__(self, input_dim, output_dim,
+                 init='uniform', input_length=None,
+                 W_regularizer=None, activity_regularizer=None,
+                 W_constraint=None,
+                 mask_zero=False,
+                 weights=None, dropout=0., **kwargs):
         self.input_dim = input_dim
         self.output_dim = output_dim
+        self.init = initializations.get(init)
+        self.input_length = input_length
+        self.mask_zero = mask_zero
+        self.dropout = dropout
 
-        self.input = T.imatrix()
-        self.W = self.init((self.input_dim, self.output_dim))
-        self.params = [self.W]
-        self.constraints = [W_constraint]
-        self.regularizers = [W_regularizer]
+        self.W_constraint = constraints.get(W_constraint)
+        self.constraints = [self.W_constraint]
 
-        if weights is not None:
-            self.set_weights(weights)
+        self.W_regularizer = regularizers.get(W_regularizer)
+        self.activity_regularizer = regularizers.get(activity_regularizer)
+
+        self.initial_weights = weights
+        kwargs['input_shape'] = (self.input_dim,)
+        super(Embedding, self).__init__(**kwargs)
+
+    def build(self):
+        self.input = K.placeholder(shape=(self.input_shape[0], self.input_length),
+                                   dtype='int32')
+        self.W = self.init((self.input_dim, self.output_dim),
+                           name='{}_W'.format(self.name))
+        self.trainable_weights = [self.W]
+        self.regularizers = []
+        if self.W_regularizer:
+            self.W_regularizer.set_param(self.W)
+            self.regularizers.append(self.W_regularizer)
+
+        if self.activity_regularizer:
+            self.activity_regularizer.set_layer(self)
+            self.regularizers.append(self.activity_regularizer)
+
+        if self.initial_weights is not None:
+            self.set_weights(self.initial_weights)
+
+    def get_output_mask(self, train=None):
+        X = self.get_input(train)
+        if not self.mask_zero:
+            return None
+        else:
+            return K.not_equal(X, 0)
+
+    @property
+    def output_shape(self):
+        return (self.input_shape[0], self.input_length, self.output_dim)
 
     def get_output(self, train=False):
         X = self.get_input(train)
-        out = self.W[X]
+        retain_p = 1. - self.dropout
+        if train and self.dropout > 0:
+            B = K.random_binomial((self.input_dim,), p=retain_p)
+        else:
+            B = K.ones((self.input_dim)) * retain_p
+        # we zero-out rows of W at random
+        out = K.gather(self.W * K.expand_dims(B), X)
         return out
 
     def get_config(self):
-        return {"name":self.__class__.__name__,
-            "input_dim":self.input_dim,
-            "output_dim":self.output_dim,
-            "init":self.init.__name__}
-
-
-class WordContextProduct(Layer):
-    '''
-        This layer turns a pair of words (a pivot word + a context word, 
-        ie. a word from the same context, or a random, out-of-context word),
-        indentified by their index in a vocabulary, into two dense reprensentations
-        (word representation and context representation).
-
-        Then it returns activation(dot(pivot_embedding, context_embedding)),
-        which can be trained to encode the probability 
-        of finding the context word in the context of the pivot word
-        (or reciprocally depending on your training procedure).
-
-        The layer ingests integer tensors of shape:
-        (nb_samples, 2)
-        and outputs a float tensor of shape
-        (nb_samples, 1)
-
-        The 2nd dimension encodes (pivot, context).
-        input_dim is the size of the vocabulary.
-
-        For more context, see Mikolov et al.:
-            Efficient Estimation of Word reprensentations in Vector Space
-            http://arxiv.org/pdf/1301.3781v3.pdf
-    '''
-    def __init__(self, input_dim, proj_dim=128, 
-        init='uniform', activation='sigmoid', weights=None):
-        super(WordContextProduct,self).__init__()
-        self.input_dim = input_dim
-        self.proj_dim = proj_dim
-        self.init = initializations.get(init)
-        self.activation = activations.get(activation)
-
-        self.input = T.imatrix()
-        # two different embeddings for pivot word and its context
-        # because p(w|c) != p(c|w)
-        self.W_w = self.init((input_dim, proj_dim))
-        self.W_c = self.init((input_dim, proj_dim))
-
-        self.params = [self.W_w, self.W_c]
-
-        if weights is not None:
-            self.set_weights(weights)
-
-
-    def get_output(self, train=False):
-        X = self.get_input(train)
-        w = self.W_w[X[:, 0]] # nb_samples, proj_dim
-        c = self.W_c[X[:, 1]] # nb_samples, proj_dim
-
-        dot = T.sum(w * c, axis=1)
-        dot = theano.tensor.reshape(dot, (X.shape[0], 1))
-        return self.activation(dot)
-
-    def get_config(self):
-        return {"name":self.__class__.__name__,
-            "input_dim":self.input_dim,
-            "proj_dim":self.proj_dim,
-            "init":self.init.__name__,
-            "activation":self.activation.__name__}
-
+        config = {"name": self.__class__.__name__,
+                  "input_dim": self.input_dim,
+                  "output_dim": self.output_dim,
+                  "init": self.init.__name__,
+                  "input_length": self.input_length,
+                  "mask_zero": self.mask_zero,
+                  "activity_regularizer": self.activity_regularizer.get_config() if self.activity_regularizer else None,
+                  "W_regularizer": self.W_regularizer.get_config() if self.W_regularizer else None,
+                  "W_constraint": self.W_constraint.get_config() if self.W_constraint else None,
+                  "dropout": self.dropout}
+        base_config = super(Embedding, self).get_config()
+        return dict(list(base_config.items()) + list(config.items()))

keras/layers/noise.py

@@ -0,0 +1,75 @@
+from __future__ import absolute_import
+from .core import MaskedLayer
+from .. import backend as K
+
+
+class GaussianNoise(MaskedLayer):
+    '''Apply to the input an additive zero-centred gaussian noise with
+    standard deviation `sigma`. This is useful to mitigate overfitting
+    (you could see it as a kind of random data augmentation).
+    Gaussian Noise (GS) is a natural choice as corruption process
+    for real valued inputs.
+
+    As it is a regularization layer, it is only active at training time.
+
+    # Input shape
+        Arbitrary. Use the keyword argument `input_shape`
+        (tuple of integers, does not include the samples axis)
+        when using this layer as the first layer in a model.
+
+    # Output shape
+        Same shape as input.
+
+    # Arguments
+        sigma: float, standard deviation of the noise distribution.
+    '''
+    def __init__(self, sigma, **kwargs):
+        super(GaussianNoise, self).__init__(**kwargs)
+        self.sigma = sigma
+
+    def get_output(self, train=False):
+        X = self.get_input(train)
+        if not train or self.sigma == 0:
+            return X
+        else:
+            return X + K.random_normal(shape=K.shape(X),
+                                       mean=0.,
+                                       std=self.sigma)
+
+    def get_config(self):
+        config = {"name": self.__class__.__name__,
+                  "sigma": self.sigma}
+        base_config = super(GaussianNoise, self).get_config()
+        return dict(list(base_config.items()) + list(config.items()))
+
+
+class GaussianDropout(MaskedLayer):
+    '''Apply to the input an multiplicative one-centred gaussian noise
+    with standard deviation `sqrt(p/(1-p))`.
+
+    As it is a regularization layer, it is only active at training time.
+
+    # Arguments
+        p: float, drop probability (as with `Dropout`).
+
+    # References:
+        [Dropout: A Simple Way to Prevent Neural Networks from Overfitting Srivastava, Hinton, et al. 2014](http://www.cs.toronto.edu/~rsalakhu/papers/srivastava14a.pdf)
+    '''
+    def __init__(self, p, **kwargs):
+        super(GaussianDropout, self).__init__(**kwargs)
+        self.p = p
+
+    def get_output(self, train):
+        X = self.get_input(train)
+        if train:
+            # self.p refers to drop probability rather than
+            # retain probability (as in paper), for consistency
+            X *= K.random_normal(shape=K.shape(X), mean=1.0,
+                                 std=K.sqrt(self.p / (1.0 - self.p)))
+        return X
+
+    def get_config(self):
+        config = {"name": self.__class__.__name__,
+                  "p": self.p}
+        base_config = super(GaussianDropout, self).get_config()
+        return dict(list(base_config.items()) + list(config.items()))

keras/layers/normalization.py

@@ -1,69 +1,119 @@
 from ..layers.core import Layer
-from ..utils.theano_utils import shared_zeros
 from .. import initializations
+from .. import backend as K
 
-import theano.tensor as T
 
 class BatchNormalization(Layer):
-    '''
-        Reference: 
-            Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
-                http://arxiv.org/pdf/1502.03167v3.pdf
+    '''Normalize the activations of the previous layer at each batch,
+    i.e. applies a transformation that maintains the mean activation
+    close to 0 and the activation standard deviation close to 1.
 
-            mode: 0 -> featurewise normalization
-                  1 -> samplewise normalization (may sometimes outperform featurewise mode)
+    # Input shape
+        Arbitrary. Use the keyword argument `input_shape`
+        (tuple of integers, does not include the samples axis)
+        when using this layer as the first layer in a model.
 
-            momentum: momentum term in the computation of a running estimate of the mean and std of the data
+    # Output shape
+        Same shape as input.
+
+    # Arguments
+        epsilon: small float > 0. Fuzz parameter.
+        mode: integer, 0 or 1.
+            - 0: feature-wise normalization.
+                Each feature map in the input will
+                be normalized separately. The axis on which
+                to normalize is specified by the `axis` argument.
+                Note that if the input is a 4D image tensor
+                using Theano conventions (samples, channels, rows, cols)
+                then you should set `axis` to `1` to normalize along
+                the channels axis.
+            - 1: sample-wise normalization. This mode assumes a 2D input.
+        axis: integer, axis along which to normalize in mode 0. For instance,
+            if your input tensor has shape (samples, channels, rows, cols),
+            set axis to 1 to normalize per feature map (channels axis).
+        momentum: momentum in the computation of the
+            exponential average of the mean and standard deviation
+            of the data, for feature-wise normalization.
+        weights: Initialization weights.
+            List of 2 numpy arrays, with shapes:
+            `[(input_shape,), (input_shape,)]`
+        beta_init: name of initialization function for shift parameter
+            (see [initializations](../initializations.md)), or alternatively,
+            Theano/TensorFlow function to use for weights initialization.
+            This parameter is only relevant if you don't pass a `weights` argument.
+        gamma_init: name of initialization function for scale parameter (see
+            [initializations](../initializations.md)), or alternatively,
+            Theano/TensorFlow function to use for weights initialization.
+            This parameter is only relevant if you don't pass a `weights` argument.
+    # References
+        - [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](http://arxiv.org/pdf/1502.03167v3.pdf)
     '''
-    def __init__(self, input_shape, epsilon=1e-6, mode=0, momentum=0.9, weights=None):
-        super(BatchNormalization,self).__init__()
-        self.init = initializations.get("uniform")
-        self.input_shape = input_shape
+    def __init__(self, epsilon=1e-6, mode=0, axis=-1, momentum=0.9,
+                 weights=None, beta_init='zero', gamma_init='one', **kwargs):
+        self.beta_init = initializations.get(beta_init)
+        self.gamma_init = initializations.get(gamma_init)
         self.epsilon = epsilon
         self.mode = mode
+        self.axis = axis
         self.momentum = momentum
+        self.initial_weights = weights
+        super(BatchNormalization, self).__init__(**kwargs)
 
-        self.gamma = self.init((self.input_shape))
-        self.beta = shared_zeros(self.input_shape)
+    def build(self):
+        input_shape = self.input_shape  # starts with samples axis
+        shape = (input_shape[self.axis],)
 
-        self.running_mean = None
-        self.running_std = None
+        self.gamma = self.gamma_init(shape, name='{}_gamma'.format(self.name))
+        self.beta = self.beta_init(shape, name='{}_beta'.format(self.name))
+        self.trainable_weights = [self.gamma, self.beta]
 
-        self.params = [self.gamma, self.beta]
-        if weights is not None:
-            self.set_weights(weights)
+        self.running_mean = K.zeros(shape,
+                                    name='{}_running_mean'.format(self.name))
+        self.running_std = K.ones(shape,
+                                  name='{}_running_std'.format(self.name))
+        self.non_trainable_weights = [self.running_mean, self.running_std]
+
+        if self.initial_weights is not None:
+            self.set_weights(self.initial_weights)
+            del self.initial_weights
 
     def get_output(self, train):
         X = self.get_input(train)
-
         if self.mode == 0:
+            input_shape = self.input_shape
+            reduction_axes = list(range(len(input_shape)))
+            del reduction_axes[self.axis]
+            broadcast_shape = [1] * len(input_shape)
+            broadcast_shape[self.axis] = input_shape[self.axis]
             if train:
-                m = X.mean(axis=0)
-                # manual computation of std to prevent NaNs
-                std = T.mean((X-m)**2 + self.epsilon, axis=0) ** 0.5
-                X_normed = (X - m) / (std + self.epsilon)
-
-                if self.running_mean is None:
-                    self.running_mean = m
-                    self.running_std = std
-                else:
-                    self.running_mean *= self.momentum
-                    self.running_mean += (1-self.momentum) * m
-                    self.running_std *= self.momentum
-                    self.running_std += (1-self.momentum) * std
+                m = K.mean(X, axis=reduction_axes)
+                brodcast_m = K.reshape(m, broadcast_shape)
+                std = K.mean(K.square(X - brodcast_m) + self.epsilon, axis=reduction_axes)
+                std = K.sqrt(std)
+                brodcast_std = K.reshape(std, broadcast_shape)
+                mean_update = self.momentum * self.running_mean + (1-self.momentum) * m
+                std_update = self.momentum * self.running_std + (1-self.momentum) * std
+                self.updates = [(self.running_mean, mean_update),
+                                (self.running_std, std_update)]
+                X_normed = (X - brodcast_m) / (brodcast_std + self.epsilon)
             else:
-                X_normed = (X - self.running_mean) / (self.running_std + self.epsilon)
-
+                brodcast_m = K.reshape(self.running_mean, broadcast_shape)
+                brodcast_std = K.reshape(self.running_std, broadcast_shape)
+                X_normed = ((X - brodcast_m) /
+                            (brodcast_std + self.epsilon))
+            out = K.reshape(self.gamma, broadcast_shape) * X_normed + K.reshape(self.beta, broadcast_shape)
         elif self.mode == 1:
-            m = X.mean(axis=-1, keepdims=True)
-            std = X.std(axis=-1, keepdims=True)
+            m = K.mean(X, axis=-1, keepdims=True)
+            std = K.std(X, axis=-1, keepdims=True)
             X_normed = (X - m) / (std + self.epsilon)
-
-        out = self.gamma * X_normed + self.beta
+            out = self.gamma * X_normed + self.beta
         return out
 
     def get_config(self):
-        return {"name":self.__class__.__name__,
-            "input_shape":self.input_shape,
-            "epsilon":self.epsilon,
-            "mode":self.mode}
+        config = {"name": self.__class__.__name__,
+                  "epsilon": self.epsilon,
+                  "mode": self.mode,
+                  "axis": self.axis,
+                  "momentum": self.momentum}
+        base_config = super(BatchNormalization, self).get_config()
+        return dict(list(base_config.items()) + list(config.items()))

keras/layers/wrappers.py

@@ -0,0 +1,114 @@
+from .core import MaskedLayer
+from .. import backend as K
+
+
+class TimeDistributed(MaskedLayer):
+    """This wrapper allows to apply a layer to every
+    temporal slice of an input.
+
+    The input should be at least 3D,
+    and the dimension of index one will be considered to be
+    the temporal dimension.
+
+    Consider a batch of 32 samples, where each sample is a sequence of 10
+    vectors of 16 dimensions. The batch input shape of the layer is then `(32, 10, 16)`
+    (and the `input_shape`, not including the samples dimension, is `(10, 16)`).
+
+    You can then use `TimeDistributed` to apply a `Dense` layer to each of the 10 timesteps, independently:
+    ```python
+        model = Sequential()
+        model.add(TimeDistributed(Dense(8), input_shape=(10, 16)))
+    ```
+
+    The output will then have shape `(32, 10, 8)`.
+
+    Note this is strictly equivalent to using `layers.core.TimeDistributedDense`.
+    However what is different about `TimeDistributed`
+    is that it can be used with arbitrary layers, not just `Dense`,
+    for instance with a `Convolution2D` layer:
+
+    ```python
+        model = Sequential()
+        model.add(TimeDistributed(Convolution2D(64, 3, 3), input_shape=(10, 3, 299, 299)))
+    ```
+
+    # Arguments
+        layer: a layer instance.
+    """
+
+    def __init__(self, layer, **kwargs):
+        self.layer = layer
+        super(TimeDistributed, self).__init__(**kwargs)
+
+    def build(self):
+        input_shape = self.input_shape
+        assert len(input_shape) >= 3
+        child_input_shape = (input_shape[0],) + input_shape[2:]
+        self.layer.set_input_shape(child_input_shape)
+        self.layer.build()
+
+        trainable_weights, regularizers, constraints, updates = self.layer.get_params()
+        self.trainable_weights = trainable_weights
+        self.non_trainable_weights = self.layer.non_trainable_weights
+        self.regularizers = regularizers
+        self.constraints = constraints
+        self.updates = updates
+
+    @property
+    def output_shape(self):
+        child_output_shape = self.layer.output_shape
+        timesteps = self.input_shape[1]
+        return (child_output_shape[0], timesteps) + child_output_shape[1:]
+
+    def get_output(self, train=False):
+        X = self.get_input(train)
+        mask = self.get_input_mask(train)
+
+        if K._BACKEND == 'tensorflow':
+            if not self.input_shape[1]:
+                raise Exception('When using TensorFlow, you should define ' +
+                                'explicitly the number of timesteps of ' +
+                                'your sequences.\n' +
+                                'If your first layer is an Embedding, ' +
+                                'make sure to pass it an "input_length" ' +
+                                'argument. Otherwise, make sure ' +
+                                'the first layer has ' +
+                                'an "input_shape" or "batch_input_shape" ' +
+                                'argument, including the time axis.')
+
+        if self.input_shape[0]:
+            # batch size matters, use rnn-based implementation
+            def step(x, states):
+                output = self.layer(x, train=train)
+                return output, []
+
+            last_output, outputs, states = K.rnn(step, X,
+                                                 initial_states=[],
+                                                 mask=mask)
+            y = outputs
+        else:
+            # no batch size specified, therefore the layer will be able
+            # to process batches of any size
+            # we can go with reshape-based implementation for performance
+            input_shape = self.input_shape
+            x = K.reshape(X, (-1, ) + input_shape[2:])  # (nb_samples * timesteps, ...)
+            y = self.layer(x, train=False)  # (nb_samples * timesteps, ...)
+            input_length = input_shape[1]
+            if not input_length:
+                input_length = K.shape(X)[1]
+            # (nb_samples, timesteps, ...)
+            y = K.reshape(y, (-1, input_length) + self.layer.output_shape[1:])
+        return y
+
+    def get_weights(self):
+        weights = self.layer.get_weights()
+        return weights
+
+    def set_weights(self, weights):
+        self.layer.set_weights(weights)
+
+    def get_config(self):
+        config = {'name': self.__class__.__name__,
+                  'layer': self.layer.get_config()}
+        base_config = super(TimeDistributed, self).get_config()
+        return dict(list(base_config.items()) + list(config.items()))

keras/objectives.py

@@ -1,49 +1,62 @@
 from __future__ import absolute_import
-import theano
-import theano.tensor as T
 import numpy as np
-from six.moves import range
+from . import backend as K
 
-epsilon = 1.0e-15
 
 def mean_squared_error(y_true, y_pred):
-    return T.sqr(y_pred - y_true).mean()
+    return K.mean(K.square(y_pred - y_true), axis=-1)
+
 
 def mean_absolute_error(y_true, y_pred):
-    return T.abs_(y_pred - y_true).mean()
+    return K.mean(K.abs(y_pred - y_true), axis=-1)
+
+
+def mean_absolute_percentage_error(y_true, y_pred):
+    diff = K.abs((y_true - y_pred) / K.clip(K.abs(y_true), K.epsilon(), np.inf))
+    return 100. * K.mean(diff, axis=-1)
+
+
+def mean_squared_logarithmic_error(y_true, y_pred):
+    first_log = K.log(K.clip(y_pred, K.epsilon(), np.inf) + 1.)
+    second_log = K.log(K.clip(y_true, K.epsilon(), np.inf) + 1.)
+    return K.mean(K.square(first_log - second_log), axis=-1)
+
 
 def squared_hinge(y_true, y_pred):
-    return T.sqr(T.maximum(1. - y_true * y_pred, 0.)).mean()
+    return K.mean(K.square(K.maximum(1. - y_true * y_pred, 0.)), axis=-1)
+
 
 def hinge(y_true, y_pred):
-    return T.maximum(1. - y_true * y_pred, 0.).mean()
+    return K.mean(K.maximum(1. - y_true * y_pred, 0.), axis=-1)
+
 
 def categorical_crossentropy(y_true, y_pred):
-    '''Expects a binary class matrix instead of a vector of scalar classes
+    '''Expects a binary class matrix instead of a vector of scalar classes.
     '''
-    y_pred = T.clip(y_pred, epsilon, 1.0 - epsilon)
-    # scale preds so that the class probas of each sample sum to 1
-    y_pred /= y_pred.sum(axis=1, keepdims=True) 
-    return T.nnet.categorical_crossentropy(y_pred, y_true).mean()
+    return K.categorical_crossentropy(y_pred, y_true)
+
 
 def binary_crossentropy(y_true, y_pred):
-    y_pred = T.clip(y_pred, epsilon, 1.0 - epsilon)
-    return T.nnet.binary_crossentropy(y_pred, y_true).mean()
+    return K.mean(K.binary_crossentropy(y_pred, y_true), axis=-1)
+
+
+def poisson(y_true, y_pred):
+    return K.mean(y_pred - y_true * K.log(y_pred + K.epsilon()), axis=-1)
+
+
+def cosine_proximity(y_true, y_pred):
+    y_true = K.l2_normalize(y_true, axis=-1)
+    y_pred = K.l2_normalize(y_pred, axis=-1)
+    return -K.mean(y_true * y_pred, axis=-1)
+
 
 # aliases
 mse = MSE = mean_squared_error
 mae = MAE = mean_absolute_error
+mape = MAPE = mean_absolute_percentage_error
+msle = MSLE = mean_squared_logarithmic_error
+cosine = cosine_proximity
 
 from .utils.generic_utils import get_from_module
 def get(identifier):
     return get_from_module(identifier, globals(), 'objective')
-
-def to_categorical(y):
-    '''Convert class vector (integers from 0 to nb_classes)
-    to binary class matrix, for use with categorical_crossentropy
-    '''
-    nb_classes = np.max(y)+1
-    Y = np.zeros((len(y), nb_classes))
-    for i in range(len(y)):
-        Y[i, y[i]] = 1.
-    return Y

keras/optimizers.py

@@ -1,175 +1,333 @@
 from __future__ import absolute_import
-import theano
-import theano.tensor as T
+from . import backend as K
 import numpy as np
-
-from .utils.theano_utils import shared_zeros, shared_scalar
+from .utils.generic_utils import get_from_module
 from six.moves import zip
 
+
 def clip_norm(g, c, n):
     if c > 0:
-        g = T.switch(T.ge(n, c), g*c/n, g)
+        g = K.switch(n >= c, g * c / n, g)
     return g
 
+
 def kl_divergence(p, p_hat):
-    return p_hat - p + p*T.log(p/p_hat)
+    return p_hat - p + p * K.log(p / p_hat)
+
 
 class Optimizer(object):
-    
-    def get_updates(self, params, grads):
+    '''Abstract optimizer base class.
+
+    Note: this is the parent class of all optimizers, not an actual optimizer
+    that can be used for training models.
+
+    All Keras optimizers support the following keyword arguments:
+
+        clipnorm: float >= 0. Gradients will be clipped
+            when their L2 norm exceeds this value.
+        clipvalue: float >= 0. Gradients will be clipped
+            when their absolute value exceeds this value.
+    '''
+    def __init__(self, **kwargs):
+        self.__dict__.update(kwargs)
+        self.updates = []
+
+    def get_state(self):
+        return [K.get_value(u[0]) for u in self.updates]
+
+    def set_state(self, value_list):
+        assert len(self.updates) == len(value_list)
+        for u, v in zip(self.updates, value_list):
+            K.set_value(u[0], v)
+
+    def get_updates(self, params, constraints, loss):
         raise NotImplementedError
 
-    def get_gradients(self, cost, params, regularizers):
-        grads = T.grad(cost, params)
-
+    def get_gradients(self, loss, params):
+        grads = K.gradients(loss, params)
         if hasattr(self, 'clipnorm') and self.clipnorm > 0:
-            norm = T.sqrt(sum([T.sum(g**2) for g in grads]))
+            norm = K.sqrt(sum([K.sum(K.square(g)) for g in grads]))
             grads = [clip_norm(g, self.clipnorm, norm) for g in grads]
+        if hasattr(self, 'clipvalue') and self.clipvalue > 0:
+            grads = [K.clip(g, -self.clipvalue, self.clipvalue) for g in grads]
+        return grads
 
-        new_grads = []
-        for p, g, r in zip(params, grads, regularizers):
-            g = r(g, p)
-            new_grads.append(g)
-
-        return new_grads
+    def get_config(self):
+        return {"name": self.__class__.__name__}
 
 
 class SGD(Optimizer):
+    '''Stochastic gradient descent, with support for momentum,
+    decay, and Nesterov momentum.
 
-    def __init__(self, lr=0.01, momentum=0., decay=0., nesterov=False, *args, **kwargs):
-        self.__dict__.update(kwargs)
+    # Arguments
+        lr: float >= 0. Learning rate.
+        momentum: float >= 0. Parameter updates momentum.
+        decay: float >= 0. Learning rate decay over each update.
+        nesterov: boolean. Whether to apply Nesterov momentum.
+    '''
+    def __init__(self, lr=0.01, momentum=0., decay=0., nesterov=False,
+                 *args, **kwargs):
+        super(SGD, self).__init__(**kwargs)
         self.__dict__.update(locals())
-        self.iterations = shared_scalar(0)
+        self.iterations = K.variable(0.)
+        self.lr = K.variable(lr)
+        self.momentum = K.variable(momentum)
+        self.decay = K.variable(decay)
 
-    def get_updates(self, params, regularizers, constraints, cost):
-        grads = self.get_gradients(cost, params, regularizers)
+    def get_updates(self, params, constraints, loss):
+        grads = self.get_gradients(loss, params)
         lr = self.lr * (1.0 / (1.0 + self.decay * self.iterations))
-        updates = [(self.iterations, self.iterations+1.)]
+        self.updates = [(self.iterations, self.iterations + 1.)]
 
         for p, g, c in zip(params, grads, constraints):
-            m = shared_zeros(p.get_value().shape) # momentum
-            v = self.momentum * m - lr * g # velocity
-            updates.append((m, v)) 
+            m = K.variable(np.zeros(K.get_value(p).shape))  # momentum
+            v = self.momentum * m - lr * g  # velocity
+            self.updates.append((m, v))
 
             if self.nesterov:
                 new_p = p + self.momentum * v - lr * g
             else:
                 new_p = p + v
 
-            updates.append((p, c(new_p))) # apply constraints
-        return updates
+            self.updates.append((p, c(new_p)))  # apply constraints
+        return self.updates
+
+    def get_config(self):
+        return {"name": self.__class__.__name__,
+                "lr": float(K.get_value(self.lr)),
+                "momentum": float(K.get_value(self.momentum)),
+                "decay": float(K.get_value(self.decay)),
+                "nesterov": self.nesterov}
 
 
 class RMSprop(Optimizer):
+    '''RMSProp optimizer.
 
+    It is recommended to leave the parameters of this optimizer
+    at their default values.
+
+    This optimizer is usually a good choice for recurrent
+    neural networks.
+
+    # Arguments
+        lr: float >= 0. Learning rate.
+        rho: float >= 0.
+        epsilon: float >= 0. Fuzz factor.
+    '''
     def __init__(self, lr=0.001, rho=0.9, epsilon=1e-6, *args, **kwargs):
-        self.__dict__.update(kwargs)
+        super(RMSprop, self).__init__(**kwargs)
         self.__dict__.update(locals())
+        self.lr = K.variable(lr)
+        self.rho = K.variable(rho)
 
-    def get_updates(self, params, regularizers, constraints, cost):
-        grads = self.get_gradients(cost, params, regularizers)
-        accumulators = [shared_zeros(p.get_value().shape) for p in params]
-        updates = []
+    def get_updates(self, params, constraints, loss):
+        grads = self.get_gradients(loss, params)
+        accumulators = [K.variable(np.zeros(K.get_value(p).shape)) for p in params]
+        self.updates = []
 
         for p, g, a, c in zip(params, grads, accumulators, constraints):
-            new_a = self.rho * a + (1 - self.rho) * g ** 2 # update accumulator
-            updates.append((a, new_a))
+            # update accumulator
+            new_a = self.rho * a + (1 - self.rho) * K.square(g)
+            self.updates.append((a, new_a))
 
-            new_p = p - self.lr * g / T.sqrt(new_a + self.epsilon)
-            updates.append((p, c(new_p))) # apply constraints
-            
-        return updates
+            new_p = p - self.lr * g / K.sqrt(new_a + self.epsilon)
+            self.updates.append((p, c(new_p)))  # apply constraints
+        return self.updates
+
+    def get_config(self):
+        return {"name": self.__class__.__name__,
+                "lr": float(K.get_value(self.lr)),
+                "rho": float(K.get_value(self.rho)),
+                "epsilon": self.epsilon}
 
 
 class Adagrad(Optimizer):
+    '''Adagrad optimizer.
 
+    It is recommended to leave the parameters of this optimizer
+    at their default values.
+
+    # Arguments
+        lr: float >= 0. Learning rate.
+        epsilon: float >= 0.
+    '''
     def __init__(self, lr=0.01, epsilon=1e-6, *args, **kwargs):
-        self.__dict__.update(kwargs)
+        super(Adagrad, self).__init__(**kwargs)
         self.__dict__.update(locals())
+        self.lr = K.variable(lr)
 
-    def get_updates(self, params, regularizers, constraints, cost):
-        grads = self.get_gradients(cost, params, regularizers)
-        accumulators = [shared_zeros(p.get_value().shape) for p in params]
-        updates = []
+    def get_updates(self, params, constraints, loss):
+        grads = self.get_gradients(loss, params)
+        accumulators = [K.variable(np.zeros(K.get_value(p).shape)) for p in params]
+        self.updates = []
 
         for p, g, a, c in zip(params, grads, accumulators, constraints):
-            new_a = a + g ** 2 # update accumulator
-            updates.append((a, new_a))
+            new_a = a + K.square(g)  # update accumulator
+            self.updates.append((a, new_a))
+            new_p = p - self.lr * g / K.sqrt(new_a + self.epsilon)
+            self.updates.append((p, c(new_p)))  # apply constraints
+        return self.updates
 
-            new_p = p - self.lr * g / T.sqrt(new_a + self.epsilon)
-            updates.append((p, c(new_p))) # apply constraints
-        return updates
+    def get_config(self):
+        return {"name": self.__class__.__name__,
+                "lr": float(K.get_value(self.lr)),
+                "epsilon": self.epsilon}
 
 
 class Adadelta(Optimizer):
-    '''
-        Reference: http://arxiv.org/abs/1212.5701
+    '''Adadelta optimizer.
+
+    It is recommended to leave the parameters of this optimizer
+    at their default values.
+
+    # Arguments
+        lr: float >= 0. Learning rate. It is recommended to leave it at the default value.
+        rho: float >= 0.
+        epsilon: float >= 0. Fuzz factor.
+
+    # References
+        - [Adadelta - an adaptive learning rate method](http://arxiv.org/abs/1212.5701)
     '''
     def __init__(self, lr=1.0, rho=0.95, epsilon=1e-6, *args, **kwargs):
-        self.__dict__.update(kwargs)
+        super(Adadelta, self).__init__(**kwargs)
         self.__dict__.update(locals())
+        self.lr = K.variable(lr)
 
-    def get_updates(self, params, regularizers, constraints, cost):
-        grads = self.get_gradients(cost, params, regularizers)
-        accumulators = [shared_zeros(p.get_value().shape) for p in params]
-        delta_accumulators = [shared_zeros(p.get_value().shape) for p in params]
-        updates = []
+    def get_updates(self, params, constraints, loss):
+        grads = self.get_gradients(loss, params)
+        accumulators = [K.variable(np.zeros(K.get_value(p).shape)) for p in params]
+        delta_accumulators = [K.variable(np.zeros(K.get_value(p).shape)) for p in params]
+        self.updates = []
 
-        for p, g, a, d_a, c in zip(params, grads, accumulators, delta_accumulators, constraints):
-            new_a = self.rho * a + (1 - self.rho) * g ** 2 # update accumulator
-            updates.append((a, new_a))
+        for p, g, a, d_a, c in zip(params, grads, accumulators,
+                                   delta_accumulators, constraints):
+            # update accumulator
+            new_a = self.rho * a + (1 - self.rho) * K.square(g)
+            self.updates.append((a, new_a))
 
             # use the new accumulator and the *old* delta_accumulator
-            update = g * T.sqrt(d_a + self.epsilon) / T.sqrt(new_a + self.epsilon)
+            update = g * K.sqrt(d_a + self.epsilon) / K.sqrt(new_a + self.epsilon)
 
             new_p = p - self.lr * update
-            updates.append((p, c(new_p))) # apply constraints
+            self.updates.append((p, c(new_p)))  # apply constraints
 
             # update delta_accumulator
-            new_d_a = self.rho * d_a + (1 - self.rho) * update ** 2
-            updates.append((d_a, new_d_a))
-        return updates
+            new_d_a = self.rho * d_a + (1 - self.rho) * K.square(update)
+            self.updates.append((d_a, new_d_a))
+        return self.updates
+
+    def get_config(self):
+        return {"name": self.__class__.__name__,
+                "lr": float(K.get_value(self.lr)),
+                "rho": self.rho,
+                "epsilon": self.epsilon}
 
 
 class Adam(Optimizer):
-    '''
-        Reference: http://arxiv.org/abs/1412.6980
+    '''Adam optimizer.
 
-        Default parameters follow those provided in the original paper
+    Default parameters follow those provided in the original paper.
 
-        lambda is renamed kappa.
+    # Arguments
+        lr: float >= 0. Learning rate.
+        beta_1/beta_2: floats, 0 < beta < 1. Generally close to 1.
+        epsilon: float >= 0. Fuzz factor.
+
+    # References
+        - [Adam - A Method for Stochastic Optimization](http://arxiv.org/abs/1412.6980v8)
     '''
-    def __init__(self, lr=0.001, beta_1=0.9, beta_2=0.999, epsilon=1e-8, kappa=1-1e-8, *args, **kwargs):
-        self.__dict__.update(kwargs)
+    def __init__(self, lr=0.001, beta_1=0.9, beta_2=0.999, epsilon=1e-8,
+                 *args, **kwargs):
+        super(Adam, self).__init__(**kwargs)
         self.__dict__.update(locals())
-        self.iterations = shared_scalar(0)
+        self.iterations = K.variable(0)
+        self.lr = K.variable(lr)
+        self.beta_1 = K.variable(beta_1)
+        self.beta_2 = K.variable(beta_2)
 
-    def get_updates(self, params, regularizers, constraints, cost):
-        grads = self.get_gradients(cost, params, regularizers)
-        updates = [(self.iterations, self.iterations+1.)]
+    def get_updates(self, params, constraints, loss):
+        grads = self.get_gradients(loss, params)
+        self.updates = [(self.iterations, self.iterations+1.)]
 
-        i = self.iterations
-        beta_1_t = self.beta_1 * (self.kappa**i)
-
-        # the update below seems missing from the paper, but is obviously required
-        beta_2_t = self.beta_2 * (self.kappa**i) 
+        t = self.iterations + 1
+        lr_t = self.lr * K.sqrt(1 - K.pow(self.beta_2, t)) / (1 - K.pow(self.beta_1, t))
 
         for p, g, c in zip(params, grads, constraints):
-            m = theano.shared(p.get_value() * 0.) # zero init of moment
-            v = theano.shared(p.get_value() * 0.) # zero init of velocity
+            # zero init of moment
+            m = K.variable(np.zeros(K.get_value(p).shape))
+            # zero init of velocity
+            v = K.variable(np.zeros(K.get_value(p).shape))
 
-            m_t = (beta_1_t * m) + (1 - beta_1_t) * g
-            v_t = (beta_2_t * v) + (1 - beta_2_t) * (g**2)
+            m_t = (self.beta_1 * m) + (1 - self.beta_1) * g
+            v_t = (self.beta_2 * v) + (1 - self.beta_2) * K.square(g)
+            p_t = p - lr_t * m_t / (K.sqrt(v_t) + self.epsilon)
 
-            m_b_t = m_t / (1 - beta_1_t)
-            v_b_t = v_t / (1 - beta_2_t)
+            self.updates.append((m, m_t))
+            self.updates.append((v, v_t))
+            self.updates.append((p, c(p_t)))  # apply constraints
+        return self.updates
+
+    def get_config(self):
+        return {"name": self.__class__.__name__,
+                "lr": float(K.get_value(self.lr)),
+                "beta_1": float(K.get_value(self.beta_1)),
+                "beta_2": float(K.get_value(self.beta_2)),
+                "epsilon": self.epsilon}
+
+
+class Adamax(Optimizer):
+    '''Adamax optimizer from Adam paper's Section 7. It is a variant
+     of Adam based on the infinity norm.
+
+    Default parameters follow those provided in the paper.
+
+    # Arguments
+        lr: float >= 0. Learning rate.
+        beta_1/beta_2: floats, 0 < beta < 1. Generally close to 1.
+        epsilon: float >= 0. Fuzz factor.
+
+    # References
+        - [Adam - A Method for Stochastic Optimization](http://arxiv.org/abs/1412.6980v8)
+    '''
+    def __init__(self, lr=0.002, beta_1=0.9, beta_2=0.999, epsilon=1e-8,
+                 *args, **kwargs):
+        super(Adamax, self).__init__(**kwargs)
+        self.__dict__.update(locals())
+        self.iterations = K.variable(0)
+        self.lr = K.variable(lr)
+        self.beta_1 = K.variable(beta_1)
+        self.beta_2 = K.variable(beta_2)
+
+    def get_updates(self, params, constraints, loss):
+        grads = self.get_gradients(loss, params)
+        self.updates = [(self.iterations, self.iterations+1.)]
+
+        t = self.iterations + 1
+        lr_t = self.lr / (1 - K.pow(self.beta_1, t))
+
+        for p, g, c in zip(params, grads, constraints):
+            # zero init of 1st moment
+            m = K.variable(np.zeros(K.get_value(p).shape))
+            # zero init of exponentially weighted infinity norm
+            u = K.variable(np.zeros(K.get_value(p).shape))
+
+            m_t = (self.beta_1 * m) + (1 - self.beta_1) * g
+            u_t = K.maximum(self.beta_2 * u, K.abs(g))
+            p_t = p - lr_t * m_t / (u_t + self.epsilon)
+
+            self.updates.append((m, m_t))
+            self.updates.append((u, u_t))
+            self.updates.append((p, c(p_t)))  # apply constraints
+        return self.updates
+
+    def get_config(self):
+        return {"name": self.__class__.__name__,
+                "lr": float(K.get_value(self.lr)),
+                "beta_1": float(K.get_value(self.beta_1)),
+                "beta_2": float(K.get_value(self.beta_2)),
+                "epsilon": self.epsilon}
 
-            p_t = p - self.lr * m_b_t / (T.sqrt(v_b_t) + self.epsilon)
-            
-            updates.append((m, m_t))
-            updates.append((v, v_t))
-            updates.append((p, c(p_t))) # apply constraints
-        return updates
 
 # aliases
 sgd = SGD
@@ -177,7 +335,9 @@ rmsprop = RMSprop
 adagrad = Adagrad
 adadelta = Adadelta
 adam = Adam
+adamax = Adamax
 
-from .utils.generic_utils import get_from_module
-def get(identifier):
-    return get_from_module(identifier, globals(), 'optimizer', instantiate=True)
+
+def get(identifier, kwargs=None):
+    return get_from_module(identifier, globals(), 'optimizer',
+                           instantiate=True, kwargs=kwargs)

keras/preprocessing/image.py

@@ -1,53 +1,51 @@
+'''Fairly basic set of tools for realtime data augmentation on image data.
+Can easily be extended to include new transformations,
+new preprocessing methods, etc...
+'''
 from __future__ import absolute_import
 
 import numpy as np
+import re
 from scipy import ndimage
 from scipy import linalg
 
 from os import listdir
 from os.path import isfile, join
-import random, math
+import math
 from six.moves import range
+import threading
 
-'''
-    Fairly basic set of tools for realtime data augmentation on image data.
-    Can easily be extended to include new transforms, new preprocessing methods, etc...
-'''
 
-def random_rotation(x, rg, fill_mode="nearest", cval=0.):
-    angle = random.uniform(-rg, rg)
-    x = ndimage.interpolation.rotate(x, angle, axes=(1,2), reshape=False, mode=fill_mode, cval=cval)
+def random_rotation(x, rg, fill_mode='nearest', cval=0.):
+    angle = np.random.uniform(-rg, rg)
+    x = ndimage.interpolation.rotate(x, angle,
+                                     axes=(1, 2),
+                                     reshape=False,
+                                     mode=fill_mode,
+                                     cval=cval)
     return x
 
-def random_shift(x, wrg, hrg, fill_mode="nearest", cval=0.):
-    crop_left_pixels = 0
-    crop_right_pixels = 0
-    crop_top_pixels = 0
-    crop_bottom_pixels = 0
 
-    original_w = x.shape[1]
-    original_h = x.shape[2]
+def random_shift(x, wrg, hrg, fill_mode='nearest', cval=0.):
+    shift_x = shift_y = 0
 
     if wrg:
-        crop = random.uniform(0., wrg)
-        split = random.uniform(0, 1)
-        crop_left_pixels = int(split*crop*x.shape[1])
-        crop_right_pixels = int((1-split)*crop*x.shape[1])
-
+        shift_x = np.random.uniform(-wrg, wrg) * x.shape[2]
     if hrg:
-        crop = random.uniform(0., hrg)
-        split = random.uniform(0, 1)
-        crop_top_pixels = int(split*crop*x.shape[2])
-        crop_bottom_pixels = int((1-split)*crop*x.shape[2])
-
-    x = ndimage.interpolation.shift(x, (0, crop_left_pixels, crop_top_pixels), mode=fill_mode, cval=cval)
+        shift_y = np.random.uniform(-hrg, hrg) * x.shape[1]
+    x = ndimage.interpolation.shift(x, (0, shift_y, shift_x),
+                                    order=0,
+                                    mode=fill_mode,
+                                    cval=cval)
     return x
 
+
 def horizontal_flip(x):
     for i in range(x.shape[0]):
         x[i] = np.fliplr(x[i])
     return x
 
+
 def vertical_flip(x):
     for i in range(x.shape[0]):
         x[i] = np.flipud(x[i])
@@ -58,41 +56,51 @@ def random_barrel_transform(x, intensity):
     # TODO
     pass
 
-def random_shear(x, intensity):
-    # TODO
-    pass
+
+def random_shear(x, intensity, fill_mode='nearest', cval=0.):
+    shear = np.random.uniform(-intensity, intensity)
+    shear_matrix = np.array([[1.0, -math.sin(shear), 0.0],
+                            [0.0, math.cos(shear), 0.0],
+                            [0.0, 0.0, 1.0]])
+    x = ndimage.interpolation.affine_transform(x, shear_matrix,
+                                               mode=fill_mode,
+                                               order=3,
+                                               cval=cval)
+    return x
+
 
 def random_channel_shift(x, rg):
     # TODO
     pass
 
-def random_zoom(x, rg, fill_mode="nearest", cval=0.):
-    zoom_w = random.uniform(1.-rg, 1.)
-    zoom_h = random.uniform(1.-rg, 1.)
-    x = ndimage.interpolation.zoom(x, zoom=(1., zoom_w, zoom_h), mode=fill_mode, cval=cval)
-    return x # shape of result will be different from shape of input!
-
 
+def random_zoom(x, rg, fill_mode='nearest', cval=0.):
+    zoom_w = np.random.uniform(1.-rg, 1.)
+    zoom_h = np.random.uniform(1.-rg, 1.)
+    x = ndimage.interpolation.zoom(x, zoom=(1., zoom_w, zoom_h),
+                                   mode=fill_mode,
+                                   cval=cval)
+    return x  # shape of result will be different from shape of input!
 
 
 def array_to_img(x, scale=True):
     from PIL import Image
-    x = x.transpose(1, 2, 0) 
+    x = x.transpose(1, 2, 0)
     if scale:
         x += max(-np.min(x), 0)
         x /= np.max(x)
         x *= 255
     if x.shape[2] == 3:
         # RGB
-        return Image.fromarray(x.astype("uint8"), "RGB")
+        return Image.fromarray(x.astype('uint8'), 'RGB')
     else:
         # grayscale
-        return Image.fromarray(x[:,:,0].astype("uint8"), "L")
+        return Image.fromarray(x[:, :, 0].astype('uint8'), 'L')
 
 
 def img_to_array(img):
     x = np.asarray(img, dtype='float32')
-    if len(x.shape)==3:
+    if len(x.shape) == 3:
         # RGB: height, width, channel -> channel, height, width
         x = x.transpose(2, 0, 1)
     else:
@@ -103,134 +111,182 @@ def img_to_array(img):
 
 def load_img(path, grayscale=False):
     from PIL import Image
-    img = Image.open(open(path))
+    img = Image.open(path)
     if grayscale:
         img = img.convert('L')
-    else: # Assure 3 channel even when loaded image is grayscale
+    else:  # Ensure 3 channel even when loaded image is grayscale
         img = img.convert('RGB')
     return img
 
 
 def list_pictures(directory, ext='jpg|jpeg|bmp|png'):
-    return [join(directory,f) for f in listdir(directory) \
-        if isfile(join(directory,f)) and re.match('([\w]+\.(?:' + ext + '))', f)]
-
+    return [join(directory, f) for f in listdir(directory)
+            if isfile(join(directory, f)) and re.match('([\w]+\.(?:' + ext + '))', f)]
 
 
 class ImageDataGenerator(object):
-    '''
-        Generate minibatches with 
-        realtime data augmentation.
-    '''
-    def __init__(self, 
-            featurewise_center=True, # set input mean to 0 over the dataset
-            samplewise_center=False, # set each sample mean to 0
-            featurewise_std_normalization=True, # divide inputs by std of the dataset
-            samplewise_std_normalization=False, # divide each input by its std
+    '''Generate minibatches with
+    real-time data augmentation.
 
-            zca_whitening=False, # apply ZCA whitening
-            rotation_range=0., # degrees (0 to 180)
-            width_shift_range=0., # fraction of total width
-            height_shift_range=0., # fraction of total height
-            horizontal_flip=False,
-            vertical_flip=False,
-        ):
+    # Arguments
+        featurewise_center: set input mean to 0 over the dataset.
+        samplewise_center: set each sample mean to 0.
+        featurewise_std_normalization: divide inputs by std of the dataset.
+        samplewise_std_normalization: divide each input by its std.
+        zca_whitening: apply ZCA whitening.
+        rotation_range: degrees (0 to 180).
+        width_shift_range: fraction of total width.
+        height_shift_range: fraction of total height.
+        shear_range: shear intensity (shear angle in radians).
+        horizontal_flip: whether to randomly flip images horizontally.
+        vertical_flip: whether to randomly flip images vertically.
+    '''
+    def __init__(self,
+                 featurewise_center=True,
+                 samplewise_center=False,
+                 featurewise_std_normalization=True,
+                 samplewise_std_normalization=False,
+                 zca_whitening=False,
+                 rotation_range=0.,
+                 width_shift_range=0.,
+                 height_shift_range=0.,
+                 shear_range=0.,
+                 horizontal_flip=False,
+                 vertical_flip=False):
         self.__dict__.update(locals())
         self.mean = None
         self.std = None
         self.principal_components = None
+        self.lock = threading.Lock()
 
+    def _flow_index(self, N, batch_size=32, shuffle=False, seed=None):
+        b = 0
+        total_b = 0
+        while 1:
+            if b == 0:
+                if seed is not None:
+                    np.random.seed(seed + total_b)
 
-    def flow(self, X, y, batch_size=32, shuffle=False, seed=None, save_to_dir=None, save_prefix="", save_format="jpeg"):
-        if seed:
-            random.seed(seed)
+                if shuffle:
+                    index_array = np.random.permutation(N)
+                else:
+                    index_array = np.arange(N)
 
-        if shuffle:
-            seed = random.randint(1, 10e6)
-            np.random.seed(seed)
-            np.random.shuffle(X)
-            np.random.seed(seed)
-            np.random.shuffle(y)
-
-        nb_batch = int(math.ceil(float(X.shape[0])/batch_size))
-        for b in range(nb_batch):
-            batch_end = (b+1)*batch_size
-            if batch_end > X.shape[0]:
-                nb_samples = X.shape[0] - b*batch_size
+            current_index = (b * batch_size) % N
+            if N >= current_index + batch_size:
+                current_batch_size = batch_size
             else:
-                nb_samples = batch_size
+                current_batch_size = N - current_index
 
-            bX = np.zeros(tuple([nb_samples]+list(X.shape)[1:]))
-            for i in range(nb_samples):
-                x = X[b*batch_size+i]
-                x = self.random_transform(x.astype("float32"))
-                x = self.standardize(x)
-                bX[i] = x
+            if current_batch_size == batch_size:
+                b += 1
+            else:
+                b = 0
+            total_b += 1
+            yield (index_array[current_index: current_index + current_batch_size],
+                   current_index, current_batch_size)
 
-            if save_to_dir:
-                for i in range(nb_samples):
-                    img = array_to_img(bX[i], scale=True)
-                    img.save(save_to_dir + "/" + save_prefix + "_" + str(i) + "." + save_format)
+    def flow(self, X, y, batch_size=32, shuffle=False, seed=None,
+             save_to_dir=None, save_prefix='', save_format='jpeg'):
+        assert len(X) == len(y)
+        self.X = X
+        self.y = y
+        self.save_to_dir = save_to_dir
+        self.save_prefix = save_prefix
+        self.save_format = save_format
+        self.flow_generator = self._flow_index(X.shape[0], batch_size,
+                                               shuffle, seed)
+        return self
 
-            yield bX, y[b*batch_size:b*batch_size+nb_samples]
+    def __iter__(self):
+        # needed if we want to do something like:
+        # for x, y in data_gen.flow(...):
+        return self
 
+    def next(self):
+        # for python 2.x.
+        # Keeps under lock only the mechanism which advances
+        # the indexing of each batch
+        # see # http://anandology.com/blog/using-iterators-and-generators/
+        with self.lock:
+            index_array, current_index, current_batch_size = next(self.flow_generator)
+        # The transformation of images is not under thread lock so it can be done in parallel
+        bX = np.zeros(tuple([current_batch_size] + list(self.X.shape)[1:]))
+        for i, j in enumerate(index_array):
+            x = self.X[j]
+            x = self.random_transform(x.astype('float32'))
+            x = self.standardize(x)
+            bX[i] = x
+        if self.save_to_dir:
+            for i in range(current_batch_size):
+                img = array_to_img(bX[i], scale=True)
+                img.save(self.save_to_dir + '/' + self.save_prefix + '_' + str(current_index + i) + '.' + self.save_format)
+        bY = self.y[index_array]
+        return bX, bY
+
+    def __next__(self):
+        # for python 3.x.
+        return self.next()
 
     def standardize(self, x):
+        if self.samplewise_center:
+            x -= np.mean(x, axis=1, keepdims=True)
+        if self.samplewise_std_normalization:
+            x /= (np.std(x, axis=1, keepdims=True) + 1e-7)
+
         if self.featurewise_center:
             x -= self.mean
         if self.featurewise_std_normalization:
-            x /= self.std
+            x /= (self.std + 1e-7)
 
         if self.zca_whitening:
-            flatx = np.reshape(x, (x.shape[0]*x.shape[1]*x.shape[2]))
+            flatx = np.reshape(x, (x.shape[0] * x.shape[1] * x.shape[2]))
             whitex = np.dot(flatx, self.principal_components)
             x = np.reshape(whitex, (x.shape[0], x.shape[1], x.shape[2]))
 
-        if self.samplewise_center:
-            x -= np.mean(x)
-        if self.samplewise_std_normalization:
-            x /= np.std(x)
-
         return x
 
-
     def random_transform(self, x):
         if self.rotation_range:
             x = random_rotation(x, self.rotation_range)
         if self.width_shift_range or self.height_shift_range:
             x = random_shift(x, self.width_shift_range, self.height_shift_range)
         if self.horizontal_flip:
-            if random.random() < 0.5:
+            if np.random.random() < 0.5:
                 x = horizontal_flip(x)
         if self.vertical_flip:
-            if random.random() < 0.5:
+            if np.random.random() < 0.5:
                 x = vertical_flip(x)
-
+        if self.shear_range:
+            x = random_shear(x, self.shear_range)
         # TODO:
         # zoom
         # barrel/fisheye
-        # shearing
         # channel shifting
         return x
 
+    def fit(self, X,
+            augment=False,
+            rounds=1,
+            seed=None):
+        '''Required for featurewise_center, featurewise_std_normalization
+        and zca_whitening.
 
-    def fit(self, X, 
-            augment=False, # fit on randomly augmented samples
-            rounds=1, # if augment, how many augmentation passes over the data do we use
-            seed=None
-        ):
-        '''
-            Required for featurewise_center, featurewise_std_normalization and zca_whitening.
+        # Arguments
+            X: Numpy array, the data to fit on.
+            augment: whether to fit on randomly augmented samples
+            rounds: if `augment`,
+                how many augmentation passes to do over the data
+            seed: random seed.
         '''
         X = np.copy(X)
-        
         if augment:
-            aX = np.zeros(tuple([rounds*X.shape[0]]+list(X.shape)[1:]))
+            aX = np.zeros(tuple([rounds * X.shape[0]] + list(X.shape)[1:]))
             for r in range(rounds):
                 for i in range(X.shape[0]):
                     img = array_to_img(X[i])
                     img = self.random_transform(img)
-                    aX[i+r*X.shape[0]] = img_to_array(img)
+                    aX[i + r * X.shape[0]] = img_to_array(img)
             X = aX
 
         if self.featurewise_center:
@@ -238,13 +294,19 @@ class ImageDataGenerator(object):
             X -= self.mean
         if self.featurewise_std_normalization:
             self.std = np.std(X, axis=0)
-            X /= self.std
+            X /= (self.std + 1e-7)
 
         if self.zca_whitening:
-            flatX = np.reshape(X, (X.shape[0], X.shape[1]*X.shape[2]*X.shape[3]))
-            fudge = 10e-6
+            flatX = np.reshape(X, (X.shape[0], X.shape[1] * X.shape[2] * X.shape[3]))
             sigma = np.dot(flatX.T, flatX) / flatX.shape[1]
             U, S, V = linalg.svd(sigma)
-            self.principal_components = np.dot(np.dot(U, np.diag(1. / np.sqrt(S + fudge))), U.T)
+            self.principal_components = np.dot(np.dot(U, np.diag(1. / np.sqrt(S + 10e-7))), U.T)
 
 
+class GraphImageDataGenerator(ImageDataGenerator):
+    '''Example of how to build a generator for a Graph model
+    '''
+
+    def next(self):
+        bX, bY = super(GraphImageDataGenerator, self).next()
+        return {'input': bX, 'output': bY}

keras/preprocessing/sequence.py

@@ -4,64 +4,121 @@ import numpy as np
 import random
 from six.moves import range
 
-def pad_sequences(sequences, maxlen=None, dtype='int32'):
-    """
-        Pad each sequence to the same length: 
-        the length of the longuest sequence.
 
-        If maxlen is provided, any sequence longer
-        than maxlen is truncated to maxlen.
-    """
+def pad_sequences(sequences, maxlen=None, dtype='int32',
+                  padding='pre', truncating='pre', value=0.):
+    '''Pads each sequence to the same length:
+    the length of the longest sequence.
+
+    If maxlen is provided, any sequence longer
+    than maxlen is truncated to maxlen.
+    Truncation happens off either the beginning (default) or
+    the end of the sequence.
+
+    Supports post-padding and pre-padding (default).
+
+    # Arguments
+        sequences: list of lists where each element is a sequence
+        maxlen: int, maximum length
+        dtype: type to cast the resulting sequence.
+        padding: 'pre' or 'post', pad either before or after each sequence.
+        truncating: 'pre' or 'post', remove values from sequences larger than
+            maxlen either in the beginning or in the end of the sequence
+        value: float, value to pad the sequences to the desired value.
+
+    # Returns
+        x: numpy array with dimensions (number_of_sequences, maxlen)
+    '''
     lengths = [len(s) for s in sequences]
 
     nb_samples = len(sequences)
     if maxlen is None:
         maxlen = np.max(lengths)
 
-    x = np.zeros((nb_samples, maxlen)).astype(dtype)
+    # take the sample shape from the first non empty sequence
+    # checking for consistency in the main loop below.
+    sample_shape = tuple()
+    for s in sequences:
+        if len(s) > 0:
+            sample_shape = np.asarray(s).shape[1:]
+            break
+
+    x = (np.ones((nb_samples, maxlen) + sample_shape) * value).astype(dtype)
     for idx, s in enumerate(sequences):
-        x[idx, :lengths[idx]] = s[:maxlen]
+        if len(s) == 0:
+            continue  # empty list was found
+        if truncating == 'pre':
+            trunc = s[-maxlen:]
+        elif truncating == 'post':
+            trunc = s[:maxlen]
+        else:
+            raise ValueError('Truncating type "%s" not understood' % truncating)
+
+        # check `trunc` has expected shape
+        trunc = np.asarray(trunc, dtype=dtype)
+        if trunc.shape[1:] != sample_shape:
+            raise ValueError('Shape of sample %s of sequence at position %s is different from expected shape %s' %
+                             (trunc.shape[1:], idx, sample_shape))
+
+        if padding == 'post':
+            x[idx, :len(trunc)] = trunc
+        elif padding == 'pre':
+            x[idx, -len(trunc):] = trunc
+        else:
+            raise ValueError('Padding type "%s" not understood' % padding)
     return x
 
 
 def make_sampling_table(size, sampling_factor=1e-5):
-    '''
-        This generates an array where the ith element
-        is the probability that a word of rank i would be sampled,
-        according to the sampling distribution used in word2vec.
-        
-        The word2vec formula is:
-            p(word) = min(1, sqrt(word.frequency/sampling_factor) / (word.frequency/sampling_factor))
+    '''This generates an array where the ith element
+    is the probability that a word of rank i would be sampled,
+    according to the sampling distribution used in word2vec.
 
-        We assume that the word frequencies follow Zipf's law (s=1) to derive 
-        a numerical approximation of frequency(rank):
-           frequency(rank) ~ 1/(rank * (log(rank) + gamma) + 1/2 - 1/(12*rank))
+    The word2vec formula is:
+        p(word) = min(1, sqrt(word.frequency/sampling_factor) / (word.frequency/sampling_factor))
+
+    We assume that the word frequencies follow Zipf's law (s=1) to derive
+    a numerical approximation of frequency(rank):
+       frequency(rank) ~ 1/(rank * (log(rank) + gamma) + 1/2 - 1/(12*rank))
         where gamma is the Euler-Mascheroni constant.
+
+    # Arguments
+        size: int, number of possible words to sample.
     '''
     gamma = 0.577
     rank = np.array(list(range(size)))
     rank[0] = 1
     inv_fq = rank * (np.log(rank) + gamma) + 0.5 - 1./(12.*rank)
     f = sampling_factor * inv_fq
+
     return np.minimum(1., f / np.sqrt(f))
 
 
-def skipgrams(sequence, vocabulary_size, 
-    window_size=4, negative_samples=1., shuffle=True, 
-    categorical=False, sampling_table=None):
-    ''' 
-        Take a sequence (list of indexes of words), 
-        returns couples of [word_index, other_word index] and labels (1s or 0s),
-        where label = 1 if 'other_word' belongs to the context of 'word',
-        and label=0 if 'other_word' is ramdomly sampled
+def skipgrams(sequence, vocabulary_size,
+              window_size=4, negative_samples=1., shuffle=True,
+              categorical=False, sampling_table=None):
+    '''Take a sequence (list of indexes of words),
+    returns couples of [word_index, other_word index] and labels (1s or 0s),
+    where label = 1 if 'other_word' belongs to the context of 'word',
+    and label=0 if 'other_word' is ramdomly sampled
 
-        @param vocabulary_size: int. maximum possible word index + 1
-        @param window_size: int. actually half-window. The window of a word wi will be [i-window_size, i+window_size+1]
-        @param negative_samples: float >= 0. 0 for no negative (=random) samples. 1 for same number as positive samples. etc.
-        @param categorical: bool. if False, labels will be integers (eg. [0, 1, 1 .. ]), 
+    # Arguments
+        vocabulary_size: int. maximum possible word index + 1
+        window_size: int. actually half-window.
+            The window of a word wi will be [i-window_size, i+window_size+1]
+        negative_samples: float >= 0. 0 for no negative (=random) samples.
+            1 for same number as positive samples. etc.
+        categorical: bool. if False, labels will be
+            integers (eg. [0, 1, 1 .. ]),
             if True labels will be categorical eg. [[1,0],[0,1],[0,1] .. ]
 
-        Note: by convention, index 0 in the vocabulary is a non-word and will be skipped.
+    # Returns
+        couples, lables: where `couples` are int pairs and
+            `labels` are either 0 or 1.
+
+    # Notes
+        By convention, index 0 in the vocabulary is
+        a non-word and will be skipped.
     '''
     couples = []
     labels = []
@@ -90,7 +147,7 @@ def skipgrams(sequence, vocabulary_size,
         words = [c[0] for c in couples]
         random.shuffle(words)
 
-        couples += [[words[i%len(words)], random.randint(1, vocabulary_size-1)] for i in range(nb_negative_samples)]
+        couples += [[words[i %len(words)], random.randint(1, vocabulary_size-1)] for i in range(nb_negative_samples)]
         if categorical:
             labels += [[1,0]]*nb_negative_samples
         else:

keras/preprocessing/text.py

@@ -1,11 +1,11 @@
 # -*- coding: utf-8 -*-
-'''
-    These preprocessing utils would greatly benefit
-    from a fast Cython rewrite.
+'''These preprocessing utilities would greatly benefit
+from a fast Cython rewrite.
 '''
 from __future__ import absolute_import
 
-import string, sys
+import string
+import sys
 import numpy as np
 from six.moves import range
 from six.moves import zip
@@ -15,12 +15,14 @@ if sys.version_info < (3,):
 else:
     maketrans = str.maketrans
 
+
 def base_filter():
     f = string.punctuation
     f = f.replace("'", '')
     f += '\t\n'
     return f
 
+
 def text_to_word_sequence(text, filters=base_filter(), lower=True, split=" "):
     '''prune: sequence of characters to filter out
     '''
@@ -32,12 +34,36 @@ def text_to_word_sequence(text, filters=base_filter(), lower=True, split=" "):
 
 
 def one_hot(text, n, filters=base_filter(), lower=True, split=" "):
-    seq = text_to_word_sequence(text)
-    return [(abs(hash(w))%(n-1)+1) for w in seq]
+    seq = text_to_word_sequence(text, filters=filters, lower=lower, split=split)
+    return [(abs(hash(w)) % (n - 1) + 1) for w in seq]
 
 
 class Tokenizer(object):
-    def __init__(self, nb_words=None, filters=base_filter(), lower=True, split=" "):
+    def __init__(self, nb_words=None, filters=base_filter(),
+                 lower=True, split=' ', char_level=False):
+        '''The class allows to vectorize a text corpus, by turning each
+        text into either a sequence of integers (each integer being the index
+        of a token in a dictionary) or into a vector where the coefficient
+        for each token could be binary, based on word count, based on tf-idf...
+
+        # Arguments
+            nb_words: the maximum number of words to keep, based
+                on word frequency. Only the most common `nb_words` words will
+                be kept.
+            filters: a string where each element is a character that will be
+                filtered from the texts. The default is all punctuation, plus
+                tabs and line breaks, minus the `'` character.
+            lower: boolean. Whether to convert the texts to lowercase.
+            split: character or string to use for token splitting.
+            char_level: if True, every character will be treated as a word.
+
+        By default, all punctuation is removed, turning the texts into
+        space-separated sequences of words
+        (words maybe include the `'` character). These sequences are then
+        split into lists of tokens. They will then be indexed or vectorized.
+
+        `0` is a reserved index that won't be assigned to any word.
+        '''
         self.word_counts = {}
         self.word_docs = {}
         self.filters = filters
@@ -45,16 +71,19 @@ class Tokenizer(object):
         self.lower = lower
         self.nb_words = nb_words
         self.document_count = 0
+        self.char_level = char_level
 
     def fit_on_texts(self, texts):
-        '''
-            required before using texts_to_sequences or texts_to_matrix
-            @param texts: can be a list or a generator (for memory-efficiency)
+        '''Required before using texts_to_sequences or texts_to_matrix
+
+        # Arguments
+            texts: can be a list of strings,
+                or a generator of strings (for memory-efficiency)
         '''
         self.document_count = 0
         for text in texts:
             self.document_count += 1
-            seq = text_to_word_sequence(text, self.filters, self.lower, self.split)
+            seq = text if self.char_level else text_to_word_sequence(text, self.filters, self.lower, self.split)
             for w in seq:
                 if w in self.word_counts:
                     self.word_counts[w] += 1
@@ -67,19 +96,17 @@ class Tokenizer(object):
                     self.word_docs[w] = 1
 
         wcounts = list(self.word_counts.items())
-        wcounts.sort(key = lambda x: x[1], reverse=True)
+        wcounts.sort(key=lambda x: x[1], reverse=True)
         sorted_voc = [wc[0] for wc in wcounts]
-        self.word_index = dict(list(zip(sorted_voc, list(range(1, len(sorted_voc)+1)))))
+        self.word_index = dict(list(zip(sorted_voc, list(range(1, len(sorted_voc) + 1)))))
 
         self.index_docs = {}
         for w, c in list(self.word_docs.items()):
             self.index_docs[self.word_index[w]] = c
 
-
     def fit_on_sequences(self, sequences):
-        '''
-            required before using sequences_to_matrix 
-            (if fit_on_texts was never called)
+        '''Required before using sequences_to_matrix
+        (if fit_on_texts was never called)
         '''
         self.document_count = len(sequences)
         self.index_docs = {}
@@ -91,14 +118,12 @@ class Tokenizer(object):
                 else:
                     self.index_docs[i] += 1
 
-
     def texts_to_sequences(self, texts):
-        '''
-            Transform each text in texts in a sequence of integers.
-            Only top "nb_words" most frequent words will be taken into account.
-            Only words known by the tokenizer will be taken into account.
+        '''Transforms each text in texts in a sequence of integers.
+        Only top "nb_words" most frequent words will be taken into account.
+        Only words known by the tokenizer will be taken into account.
 
-            Returns a list of sequences.
+        Returns a list of sequences.
         '''
         res = []
         for vect in self.texts_to_sequences_generator(texts):
@@ -106,80 +131,84 @@ class Tokenizer(object):
         return res
 
     def texts_to_sequences_generator(self, texts):
-        '''
-            Transform each text in texts in a sequence of integers.
-            Only top "nb_words" most frequent words will be taken into account.
-            Only words known by the tokenizer will be taken into account.
+        '''Transforms each text in texts in a sequence of integers.
+        Only top "nb_words" most frequent words will be taken into account.
+        Only words known by the tokenizer will be taken into account.
 
-            Yields individual sequences.
+        Yields individual sequences.
+
+        # Arguments:
+            texts: list of strings.
         '''
         nb_words = self.nb_words
         for text in texts:
-            seq = text_to_word_sequence(text, self.filters, self.lower, self.split)
+            seq = text if self.char_level else text_to_word_sequence(text, self.filters, self.lower, self.split)
             vect = []
             for w in seq:
                 i = self.word_index.get(w)
                 if i is not None:
                     if nb_words and i >= nb_words:
-                        pass
+                        continue
                     else:
                         vect.append(i)
             yield vect
 
+    def texts_to_matrix(self, texts, mode='binary'):
+        '''Convert a list of texts to a Numpy matrix,
+        according to some vectorization mode.
 
-    def texts_to_matrix(self, texts, mode="binary"):
-        '''
-            modes: binary, count, tfidf, freq
+        # Arguments:
+            texts: list of strings.
+            modes: one of "binary", "count", "tfidf", "freq"
         '''
         sequences = self.texts_to_sequences(texts)
         return self.sequences_to_matrix(sequences, mode=mode)
 
-    def sequences_to_matrix(self, sequences, mode="binary"):
-        '''
-            modes: binary, count, tfidf, freq
+    def sequences_to_matrix(self, sequences, mode='binary'):
+        '''Converts a list of sequences into a Numpy matrix,
+        according to some vectorization mode.
+
+        # Arguments:
+            sequences: list of sequences
+                (a sequence is a list of integer word indices).
+            modes: one of "binary", "count", "tfidf", "freq"
         '''
         if not self.nb_words:
             if self.word_index:
-                nb_words = len(self.word_index)
+                nb_words = len(self.word_index) + 1
             else:
-                raise Exception("Specify a dimension (nb_words argument), or fit on some text data first")
+                raise Exception('Specify a dimension (nb_words argument), '
+                                'or fit on some text data first.')
         else:
             nb_words = self.nb_words
 
-        if mode == "tfidf" and not self.document_count:
-            raise Exception("Fit the Tokenizer on some data before using tfidf mode")
+        if mode == 'tfidf' and not self.document_count:
+            raise Exception('Fit the Tokenizer on some data '
+                            'before using tfidf mode.')
 
         X = np.zeros((len(sequences), nb_words))
         for i, seq in enumerate(sequences):
             if not seq:
-                pass
+                continue
             counts = {}
             for j in seq:
                 if j >= nb_words:
-                    pass
+                    continue
                 if j not in counts:
                     counts[j] = 1.
                 else:
                     counts[j] += 1
             for j, c in list(counts.items()):
-                if mode == "count":
+                if mode == 'count':
                     X[i][j] = c
-                elif mode == "freq":
-                    X[i][j] = c/len(seq)
-                elif mode == "binary":
+                elif mode == 'freq':
+                    X[i][j] = c / len(seq)
+                elif mode == 'binary':
                     X[i][j] = 1
-                elif mode == "tfidf":
-                    tf = np.log(c/len(seq))
-                    df = (1 + np.log(1 + self.index_docs.get(j, 0)/(1 + self.document_count)))
+                elif mode == 'tfidf':
+                    tf = np.log(c / len(seq))
+                    df = (1 + np.log(1 + self.index_docs.get(j, 0) / (1 + self.document_count)))
                     X[i][j] = tf / df
                 else:
-                    raise Exception("Unknown vectorization mode: " + str(mode))
+                    raise Exception('Unknown vectorization mode: ' + str(mode))
         return X
-
-
-
-                
-
-
-
-    

keras/regularizers.py

@@ -1,26 +1,86 @@
 from __future__ import absolute_import
-import theano
-import theano.tensor as T
-import numpy as np
+from . import backend as K
 
-def l1(l=.01):
-    def l1wrap(g, p):
-        g += T.sgn(p) * l
-        return g
-    return l1wrap
 
-def l2(l=.01):
-    def l2wrap(g, p):
-        g += p * l
-        return g
-    return l2wrap
+class Regularizer(object):
+    def set_param(self, p):
+        self.p = p
 
-def l1l2(l1=.01, l2=.01):
-    def l1l2wrap(g, p):
-        g += T.sgn(p) * l1
-        g += p * l2
-        return g
-    return l1l2wrap
+    def set_layer(self, layer):
+        self.layer = layer
 
-def identity(g, p):
-    return g
+    def __call__(self, loss):
+        return loss
+
+    def get_config(self):
+        return {"name": self.__class__.__name__}
+
+
+class WeightRegularizer(Regularizer):
+    def __init__(self, l1=0., l2=0.):
+        self.l1 = l1
+        self.l2 = l2
+
+    def set_param(self, p):
+        self.p = p
+
+    def __call__(self, loss):
+        loss += K.sum(K.abs(self.p)) * self.l1
+        loss += K.sum(K.square(self.p)) * self.l2
+        return loss
+
+    def get_config(self):
+        return {"name": self.__class__.__name__,
+                "l1": self.l1,
+                "l2": self.l2}
+
+
+class ActivityRegularizer(Regularizer):
+    def __init__(self, l1=0., l2=0.):
+        self.l1 = l1
+        self.l2 = l2
+
+    def set_layer(self, layer):
+        self.layer = layer
+
+    def __call__(self, loss):
+        output = self.layer.get_output(True)
+        loss += self.l1 * K.sum(K.mean(K.abs(output), axis=0))
+        loss += self.l2 * K.sum(K.mean(K.square(output), axis=0))
+        return loss
+
+    def get_config(self):
+        return {"name": self.__class__.__name__,
+                "l1": self.l1,
+                "l2": self.l2}
+
+
+def l1(l=0.01):
+    return WeightRegularizer(l1=l)
+
+
+def l2(l=0.01):
+    return WeightRegularizer(l2=l)
+
+
+def l1l2(l1=0.01, l2=0.01):
+    return WeightRegularizer(l1=l1, l2=l2)
+
+
+def activity_l1(l=0.01):
+    return ActivityRegularizer(l1=l)
+
+
+def activity_l2(l=0.01):
+    return ActivityRegularizer(l2=l)
+
+
+def activity_l1l2(l1=0.01, l2=0.01):
+    return ActivityRegularizer(l1=l1, l2=l2)
+
+identity = Regularizer
+
+from .utils.generic_utils import get_from_module
+def get(identifier, kwargs=None):
+    return get_from_module(identifier, globals(), 'regularizer',
+                           instantiate=True, kwargs=kwargs)

keras/utils/data_utils.py

@@ -0,0 +1,95 @@
+from __future__ import absolute_import
+from __future__ import print_function
+
+import tarfile
+import os
+import sys
+import shutil
+from six.moves.urllib.request import urlopen
+from six.moves.urllib.error import URLError, HTTPError
+
+from ..utils.generic_utils import Progbar
+
+
+# Under Python 2, 'urlretrieve' relies on FancyURLopener from legacy
+# urllib module, known to have issues with proxy management
+if sys.version_info[0] == 2:
+    def urlretrieve(url, filename, reporthook=None, data=None):
+        def chunk_read(response, chunk_size=8192, reporthook=None):
+            total_size = response.info().get('Content-Length').strip()
+            total_size = int(total_size)
+            count = 0
+            while 1:
+                chunk = response.read(chunk_size)
+                if not chunk:
+                    break
+                count += 1
+                if reporthook:
+                    reporthook(count, chunk_size, total_size)
+                yield chunk
+
+        response = urlopen(url, data)
+        with open(filename, 'wb') as fd:
+            for chunk in chunk_read(response, reporthook=reporthook):
+                fd.write(chunk)
+else:
+    from six.moves.urllib.request import urlretrieve
+
+
+def get_file(fname, origin, untar=False):
+    datadir_base = os.path.expanduser(os.path.join('~', '.keras'))
+    if not os.access(datadir_base, os.W_OK):
+        datadir_base = os.path.join('/tmp', '.keras')
+    datadir = os.path.join(datadir_base, 'datasets')
+    if not os.path.exists(datadir):
+        os.makedirs(datadir)
+
+    if untar:
+        untar_fpath = os.path.join(datadir, fname)
+        fpath = untar_fpath + '.tar.gz'
+    else:
+        fpath = os.path.join(datadir, fname)
+
+    if not os.path.exists(fpath):
+        print('Downloading data from',  origin)
+        global progbar
+        progbar = None
+
+        def dl_progress(count, block_size, total_size):
+            global progbar
+            if progbar is None:
+                progbar = Progbar(total_size)
+            else:
+                progbar.update(count*block_size)
+
+        error_msg = 'URL fetch failure on {}: {} -- {}'
+        try:
+            try:
+                urlretrieve(origin, fpath, dl_progress)
+            except URLError as e:
+                raise Exception(error_msg.format(origin, e.errno, e.reason))
+            except HTTPError as e:
+                raise Exception(error_msg.format(origin, e.code, e.msg))
+        except (Exception, KeyboardInterrupt) as e:
+            if os.path.exists(fpath):
+                os.remove(fpath)
+            raise e
+        progbar = None
+
+    if untar:
+        if not os.path.exists(untar_fpath):
+            print('Untaring file...')
+            tfile = tarfile.open(fpath, 'r:gz')
+            try:
+                tfile.extractall(path=datadir)
+            except (Exception, KeyboardInterrupt) as e:
+                if os.path.exists(untar_fpath):
+                    if os.path.isfile(untar_fpath):
+                        os.remove(untar_fpath)
+                    else:
+                        shutil.rmtree(untar_fpath)
+                raise e
+            tfile.close()
+        return untar_fpath
+
+    return fpath

keras/utils/dot_utils.py

@@ -1,51 +0,0 @@
-import pydot
-from keras.layers.core import Merge
-from keras.models import Model
-from collections import Counter
-
-class Grapher(object):
-
-    def __init__(self):
-        self.names = {}
-        self.class_counts = Counter()
-
-    def get_name(self, model):
-        """
-        returns the name of the model instance. If model does not have a `name` attribute, then it will be assigned
-        a generic (and unique) identifier based on its class
-        """
-        if hasattr(model, 'name'):
-            return model.name
-        clz = model.__class__.__name__
-        if model not in self.names:
-            self.class_counts[clz] += 1
-            self.names[model] = clz + str(self.class_counts[clz])
-        return self.names[model]
-
-    def add_edge(self, f, t, graph):
-        if f: graph.add_edge(pydot.Edge(f, t))
-        return t
-
-    def add_model(self, model, graph, parent=None):
-        """
-        Recursively adds `model` and its components to the pydot graph
-        """
-        this = self.get_name(model)
-        if isinstance(model, Model):
-            parent = self.add_edge(parent, this, graph)
-            for child in reversed(model.layers):
-                parent = self.add_model(child, graph, parent)
-        elif isinstance(model, Merge):
-            for child in model.models:
-                self.add_model(child, graph, this)
-            return self.add_edge(parent, this, graph)
-        else:
-            return self.add_edge(parent, this, graph)
-
-    def plot(self, model, to_file):
-        """
-        creates a graph visualizing the structure of `model` and writes it to `to_file`
-        """
-        graph = pydot.Dot(graph_type='graph')
-        self.add_model(model, graph)
-        graph.write_png(to_file)

keras/utils/generic_utils.py

@@ -2,41 +2,28 @@ from __future__ import absolute_import
 import numpy as np
 import time
 import sys
+import six
 
-def get_from_module(identifier, module_params, module_name, instantiate=False):
-    if type(identifier) is str:
+
+def get_from_module(identifier, module_params, module_name,
+                    instantiate=False, kwargs=None):
+    if isinstance(identifier, six.string_types):
         res = module_params.get(identifier)
         if not res:
-            raise Exception('Invalid ' + str(module_name) + ': ' + str(identifier))
-        if instantiate:
+            raise Exception('Invalid ' + str(module_name) + ': ' +
+                            str(identifier))
+        if instantiate and not kwargs:
             return res()
+        elif instantiate and kwargs:
+            return res(**kwargs)
         else:
             return res
     return identifier
 
+
 def make_tuple(*args):
     return args
 
-def printv(v, prefix=''):
-    if type(v) == dict:
-        if 'name' in v:
-            print(prefix + '#' + v['name'])
-            del v['name']
-        prefix += '...'
-        for nk, nv in v.items():
-            if type(nv) in [dict, list]:
-                print(prefix + nk + ':')
-                printv(nv, prefix)
-            else:
-                print(prefix + nk + ':' + str(nv))
-    elif type(v) == list:
-        prefix += '...'
-        for i, nv in enumerate(v):
-            print(prefix + '#' + str(i))
-            printv(nv, prefix) 
-    else:
-        prefix += '...'
-        print(prefix + str(v))
 
 class Progbar(object):
     def __init__(self, target, width=30, verbose=1):
@@ -60,11 +47,11 @@ class Progbar(object):
         '''
         for k, v in values:
             if k not in self.sum_values:
-                self.sum_values[k] = [v * (current-self.seen_so_far), current-self.seen_so_far]
+                self.sum_values[k] = [v * (current - self.seen_so_far), current - self.seen_so_far]
                 self.unique_values.append(k)
             else:
-                self.sum_values[k][0] += v * (current-self.seen_so_far)
-                self.sum_values[k][1] += (current-self.seen_so_far)
+                self.sum_values[k][0] += v * (current - self.seen_so_far)
+                self.sum_values[k][1] += (current - self.seen_so_far)
         self.seen_so_far = current
 
         now = time.time()
@@ -76,35 +63,43 @@ class Progbar(object):
             numdigits = int(np.floor(np.log10(self.target))) + 1
             barstr = '%%%dd/%%%dd [' % (numdigits, numdigits)
             bar = barstr % (current, self.target)
-            prog = float(current)/self.target
-            prog_width = int(self.width*prog)
+            prog = float(current) / self.target
+            prog_width = int(self.width * prog)
             if prog_width > 0:
-                bar += ('='*(prog_width-1))
+                bar += ('=' * (prog_width-1))
                 if current < self.target:
                     bar += '>'
                 else:
                     bar += '='
-            bar += ('.'*(self.width-prog_width))
+            bar += ('.' * (self.width - prog_width))
             bar += ']'
             sys.stdout.write(bar)
             self.total_width = len(bar)
-            
+
             if current:
                 time_per_unit = (now - self.start) / current
             else:
                 time_per_unit = 0
-            eta = time_per_unit*(self.target - current)
+            eta = time_per_unit * (self.target - current)
             info = ''
             if current < self.target:
                 info += ' - ETA: %ds' % eta
             else:
                 info += ' - %ds' % (now - self.start)
             for k in self.unique_values:
-                info += ' - %s: %.4f' % (k, self.sum_values[k][0]/ max(1, self.sum_values[k][1]))
+                info += ' - %s:' % k
+                if type(self.sum_values[k]) is list:
+                    avg = self.sum_values[k][0] / max(1, self.sum_values[k][1])
+                    if abs(avg) > 1e-3:
+                        info += ' %.4f' % avg
+                    else:
+                        info += ' %.4e' % avg
+                else:
+                    info += ' %s' % self.sum_values[k]
 
             self.total_width += len(info)
             if prev_total_width > self.total_width:
-                info += ((prev_total_width-self.total_width) * " ")
+                info += ((prev_total_width - self.total_width) * " ")
 
             sys.stdout.write(info)
             sys.stdout.flush()
@@ -116,9 +111,13 @@ class Progbar(object):
             if current >= self.target:
                 info = '%ds' % (now - self.start)
                 for k in self.unique_values:
-                    info += ' - %s: %.4f' % (k, self.sum_values[k][0]/ max(1, self.sum_values[k][1]))
+                    info += ' - %s:' % k
+                    avg = self.sum_values[k][0] / max(1, self.sum_values[k][1])
+                    if avg > 1e-3:
+                        info += ' %.4f' % avg
+                    else:
+                        info += ' %.4e' % avg
                 sys.stdout.write(info + "\n")
 
-
     def add(self, n, values=[]):
-        self.update(self.seen_so_far+n, values)
+        self.update(self.seen_so_far + n, values)