Update README.md

Added pascal voc support for tensorflow

README.md

@@ -1,2 +1,203 @@
-# offline-video-tagger
-An offline port of the video-tagging-tool for electron.
+# VoTT: Visual Object Tagging Tool
+
+This tool provides end to end support for generating datasets and validating object detection models from video and image assets.
+
+### End to End Object Detection Pipeline:
+![Pipeline: tag video, export tags to CNTK, train model, run model on a new video, validate model suggestions and fix errors, return to export tags](media/detectioninabox.jpg)
+
+The tool supports the following **features**:
+
+- The ability to tag and annotate Image Directories or Stand alone videos.
+- Computer-assisted tagging and tracking of objects in videos using the [Camshift tracking algorithm](http://opencv.jp/opencv-1.0.0_org/docs/papers/camshift.pdf).
+- Exporting tags and assets to CNTK , Tensorflow (PascalVOC) or YOLO format for training an object detection model.
+- Running and validating a trained CNTK object detection model on new videos to generate stronger models.
+
+## Table of Contents
+
+ - [Installation](#installation)
+ - [Tagging a Video](#tagging-a-video)
+ - [Tagging an Image Directory](#tagging-an-image-directory)
+ - [Reviewing and Improving an Object Detection Model](#reviewing-and-improving-an-object-detection-model)
+ - [Upcoming Features](#upcoming-features)
+ - [How to Contribute](#how-to-contribute)
+
+---
+## Installation
+
+### Installing the Visual Object Tagging Tool
+
+ 1. Download and extract the app [release package](https://github.com/CatalystCode/CNTK-Object-Detection-Video-Tagging-Tool/releases)
+
+ 2. Run the app by launching the "VOTT" executable which will be located inside the unzipped folder.
+
+ ### Installing CNTK with the FRCNN Prerequisites for Reviewing Model
+
+*Please note that installation of **CNTK and FAST-RCNN dependencies** are **optional for tagging** and are **only required for CNTK model review and training**.*
+
+1. Install [CNTK](https://github.com/Microsoft/CNTK/wiki/Setup-CNTK-on-your-machine) (*Note: currently the tool only supports the full installation method (non pip) of CNTK*).
+
+2. Follow the setup instructions of the [CNTK Fast-RCNN tutorial](https://github.com/Microsoft/CNTK/wiki/Object-Detection-using-Fast-R-CNN#setup) (*Note: Fast-RCNN currently only supports Linux python version 3.4 and not 3.5*).
+
+3. Configure `CNTK-Config.json` (which resides in the '\resources\app' directory of the tagging tool) with the following properties to enable the model review feature:
+
+```json
+{
+    "cntkPath" : "{CNTK Path default is c:/local/cntk}",
+}
+```
+## Tagging a Video
+
+ 1. Select the option to tag a video
+
+    ![](media/video-option.jpg)
+
+ 2. Load an MP4 video file either by dragging it into the app or clicking on and selecting it.
+
+    ![](media/2_load.jpg)
+
+ 3. Configure the tagging job and specify the settings in the screenshot below:
+
+    ![](media/3_Job_Configuration.jpg)
+
+    **Frame Extraction Rate**: number of frames to tag per second of video<br>
+
+    **Tagging Region Type**:  type of bounding box for tagging regions<br>
+      - *Rectangle*: tag bounding boxes of any dimension
+      - *Square*: tag bounding boxes of auto-fixed dimensions
+
+    **Suggested Region Method**: how to suggest regions for next frame<br>
+     - *Tracking*: Use camshift to track tagged regions in next frame
+     - *Copy Last Frame*: Copy all regions to the next frame.
+
+    **Enable Scene Change Detection**: Detect scene changes to prevent false positives when tracking. (Note this option is slightly slower)
+
+    **Labels**: labels of the tagged regions (e.g. `Cat`, `Dog`, `Horse`, `Person`)<br>
+
+ 4. Tag the video frame by frame
+ 
+    ![](media/4_Tagging_Job.jpg)
+
+    **Tagging**: click and drag a bounding box around the desired area, then move or resize the region until it fits the object
+     - Selected regions appear as red ![red](https://placehold.it/15/f03c15/000000?text=+) and unselected regions will appear as blue ![#1589F0](https://placehold.it/15/1589F0/000000?text=+).
+     - Assign a tag to a region by clicking on it and selecting the desired tag from the labeling toolbar at the bottom of the tagging control
+     - Click the ![cleartags](media/cleartags.png) button to clear all tags on a given frame
+
+    **Navigation**: users can navigate between video frames by using the ![prev-nxt](media/prev-next.png) buttons, the left/right arrow keys, or the video skip bar
+     - Tags are auto-saved each time a frame is changed
+
+    **Tracking**: new regions are tracked by default until a given scene changes.
+     - Since the [camshift algorithm](http://opencv.jp/opencv-1.0.0_org/docs/papers/camshift.pdf) has some known limitations, you can disable tracking for certain sets of frames. To toggle tracking *on* and *off* use the file menu setting, or the keyboard shortcut Ctrl/Cmd + T.
+
+
+ 5. Export video Tags using the Object Detection Menu or Ctrl/Cmd + E
+
+    ![]( media/5_Export.jpg)
+    
+    *Note on exporting: the tool reserves a random 20% sample of the tagged frames as a test set.*
+ 
+    Specify the following export configuration settings:
+    
+    ![]( media/5a_Export.jpg)
+    
+    - **Export Format**: What framework to export to defaults to *CNTK*<br>
+    - **Export Frames Until**: how far into the video the export operation will proceed<br>
+      - *Last Tagged Region*: exports frames up until the last frame containing tags
+      - *Last Visited Frame*: exports frames up until the last frame that the user explicitly visited
+      - *Last Frame*: exports all video frames<br>
+    - **Output directory**: directory path for exporting training data<br>
+    
+---
+
+## Tagging an Image Directory
+
+ 1. Select the option to tag an image directory
+
+    ![](media/image-option.jpg)
+
+ 2. Load an image directory by selecting it.
+
+    ![](media/2_load.jpg)
+
+ 3. Configure the tagging job and specify the settings in the screenshot below:
+
+    ![](media/3_image_Job_Configuration.jpg)
+
+    **Frame Extraction Rate**: number of frames to tag per second of video<br>
+
+    **Tagging Region Type**:  type of bounding box for tagging regions<br>
+      - *Rectangle*: tag bounding boxes of any dimension
+      - *Square*: tag bounding boxes of auto-fixed dimensions
+
+    **Labels**: labels of the tagged regions (e.g. `Cat`, `Dog`, `Horse`, `Person`)<br>
+
+ 4. Tag each Image
+ 
+    ![](media/4_image_Tagging_Job.jpg)
+
+    **Tagging**: click and drag a bounding box around the desired area, then move or resize the region until it fits the object
+     - Selected regions appear as red ![red](https://placehold.it/15/f03c15/000000?text=+) and unselected regions will appear as blue ![#1589F0](https://placehold.it/15/1589F0/000000?text=+).
+     - Assign a tag to a region by clicking on it and selecting the desired tag from the labeling toolbar at the bottom of the tagging control
+     - Click the ![cleartags](media/cleartags.png) button to clear all tags on a given frame
+
+    **Navigation**: users can navigate between video frames by using the ![prev-nxt](media/prev-next.png) buttons, the left/right arrow keys, or the video skip bar
+     - Tags are auto-saved each time a frame is changed
+
+ 5. Export Image directory tags Tags using the Object Detection Menu or Ctrl/Cmd + E
+
+    ![]( media/5_image_Export.jpg)
+    
+    *Note on exporting: the tool reserves a random 20% sample of the tagged frames as a test set.*
+ 
+    Specify the following export configuration settings:
+    
+    ![]( media/5a_Export.jpg)
+    
+    - **Export Format**: What framework to export to defaults to *CNTK*<br>
+    - **Export Frames Until**: how far into the video the export operation will proceed<br>
+      - *Last Tagged Region*: exports frames up until the last frame containing tags
+      - *Last Visited Frame*: exports frames up until the last frame that the user explicitly visited
+      - *Last Frame*: exports all video frames<br>
+    - **Output directory**: directory path for exporting training data<br>
+    
+---
+## Reviewing and Improving an Object Detection Model
+
+ 1. Train model with [Object Detection using FastRCNN](https://github.com/Microsoft/CNTK/wiki/Object-Detection-using-Fast-R-CNN#train-on-your-own-data)<br> *Note: the data is already in CNTK format, so you do not have to run `C1_DrawBboxesOnImages.py` or `C2_AssignLabelsToBboxes.py`*
+ 2. Since CNTK does not embed the names of the classes in the model, on default, the module returns non descriptive names for the classes, e.g. "class_1", "class_2".Place a JSON file named "model.json" in the same directory of the Fast-RCNN model file with the the correct tag labels. Format the json file as follows with your own class names:
+   ```json
+   {
+       "classes" : {
+           "background" : 0,
+           "human" : 1,
+     "cat" : 2,
+     "dog" : 3
+       }
+   }
+   ```
+
+ 3. Load a new asset that the model has not been trained on
+ 4. Configure a new or load a previous tagging job 
+ 4. Apply model to new asset using Ctrl/Cmd + R
+ 5. Specify a model path and temporary output directory<br> 
+    ![](media/6_Review.jpg)
+ 6. When the model finishes processing, validate tags, re-export and retrain it
+ 7. Repeat step 1 on new assets until the model performance is satisfactory
+ 
+ 
+## Supporting additonal object detection Export and Review formats.
+
+In the latest release we provide support for [Export and Review formats](https://github.com/CatalystCode/VOTT/tree/master/src/lib/detection_algorithms). To add a new object detection format, copy the interface folder and use the Yolo and CNTK implementations as reference. 
+
+## Upcoming Features 
+
+- Tagging project management
+
+-----------
+
+## How to Contribute
+
+You are welcome to send us any bugs you may find, suggestions, or any other comments.
+
+Before sending anything, please go over the repository issues list, just to make sure that it isn't already there.
+
+You are more than welcome to fork this repository and send us a pull request if you feel that what you've done should be included.

index.html

@@ -1,89 +0,0 @@
-<!--
-   Demo html page for an example of hosting the
-       video-tagging control.
--->
-
-<!doctype html>
-<html>
-  <head>
-    <meta charset="utf-8">
-    <meta name="viewport" content="width=device-width, minimum-scale=1.0, initial-scale=1.0, user-scalable=yes">
-    <title>Video Tagging Tool</title>
-    <!-- Insert this line above script imports for jquery -->
-    <script>if (typeof module === 'object') {window.module = module; module = undefined;}</script>
-    <script src="./bower_components/webcomponentsjs/webcomponents.min.js"></script>
-    <link rel="import" href="./bower_components/video-tagging/video-tagging.html">
-
-    <script src="./public/js/bootstrap.min.js"></script>
-    <script src="./public/js/bootstrap-tagsinput.min.js"></script>
-    <script src="./public/js/camshift.js"></script>
-    <script src="./public/js/scene-change-detector.js"></script>
-
-    <script src="./index.js"></script>
-
-    <link rel="stylesheet" href="./public/css/bootstrap-tagsinput.css" />
-    <link rel="stylesheet" href="./public/css/styles.css">
-  </head>
-  <body>
-    <div id="container">
-
-        <div id ='load-message'   onclick="fileSelected();">
-            <div id = "load-text"><h2>Please <b>Drag in</b> or <b>Click</b> to load a video for tagging.</h2></div>
-             <img id="vidImage" src ="./public/images/Wikiversity-Mooc-Icon-Video.svg.png" ></img>
-        </div>
-        <div id ='load-form-container' style ="display: none">
-          <h2>Tagging Job Configuration</h2>
-
-          <div class="form-group" id="framerateGroup">
-            <label for="exampleTextarea" title="(How many frames to extract per a second of video!)">Frame Extraction Rate (frames per a video second)</label>
-            <input id="framerate" type="number" min="1" max="60" value="1" maxlength="3" size="3" class="form-control" />
-          </div>
-          <div class="form-group">
-            <label for="exampleTextarea" title="(Type of region selector to tag frames)">Region Type</label>
-            <select id="regiontype" class="form-control" id="text" onchange="checkPointRegion();">
-                  <option selected="selected">Rectangle</option>
-                  <option>Point</option>
-                  <option>Square</option>
-            </select>
-          </div>
-
-          <div class="regionGroup">
-            <div class="form-group" id="regionPointGroup" style ="display: none">
-              <label for="exampleTextarea" title="(Region Size for point selector!)">Point Region Size</label>
-              <input class="form-control" type="text" value="25" id="regionsize"/>
-            </div>
-          </div>
-
-          <div class="form-group">
-           <label for="exampleTextarea" title="(Which frame to Export to.)">Export Until: </label>
-            <select id="exportTo" class="form-control" id="text" >
-                  <option value="tagged" selected="selected">Last Tagged Region</option>
-                  <option value="visited">Last Visited Frame</option>
-                  <option value="last">Last Frame</option>
-            </select>
-          </div>
-
-          <div class="form-group">
-            <label for="Out"> Output Path</label>
-            <input id="output" class="form-control" type="text" />
-          </div>
-
-          <div class="form-group">
-            <label for="Model"> Model Path</label>
-            <input id="model" class="form-control" type="text" />
-          </div>
-
-          <div class="form-group">
-            <label for="Tags"> Labels (Required)</label>
-            <input id="inputtags" class="form-control" type="text"  data-role="tagsinput" required/>
-          </div>
-
-          <div id="loadButton" class="btn btn-primary">Continue</div>
-        </div>
-        <div id='video-tagging-container' style="display: none">
-            <video-tagging id='video-tagging' ></video-tagging>
-        </div>
-    </div>
-
-  </body>
-</html>

index.js

@@ -1,617 +0,0 @@
-const remote = require('electron').remote;
-const basepath = remote.app.getAppPath();
-const dialog = remote.require('electron').dialog;
-const pathJS = require('path');
-const fs = require('fs');
-const rimraf = require('rimraf');
-const cntkModel= require('cntk-fastrcnn');
-const cntkDefaultPath = 'c:/local/cntk';
-const ipcRenderer = require('electron').ipcRenderer;
-var supertrackingFrameRate = 10;
-var trackingEnabled = true;
-var visitedFrames, //keep track of the visited frames
-    videotagging;
-
-//ipc rendering
-ipcRenderer.on('openVideo', (event, message) => {
-  fileSelected();
-});
-
-ipcRenderer.on('saveVideo', (event, message) => {
-  save();
-  let notification = new Notification('Offline Video Tagger', {
-   body: 'Successfully saved metadata in ' + `${videotagging.src}.json`
-  });
-});
-
-ipcRenderer.on('exportCNTK', (event, message) => {
-  exportCNTK();
-});
-
-ipcRenderer.on('reviewCNTK', (event, message) => {
-    if (fs.existsSync(cntkDefaultPath)) {
-        var modelLocation = $('#model').val();
-        if (fs.existsSync(modelLocation)) {
-          reviewCNTK();
-        } else {
-            alert(`No model found! Please make sure you put your model in the following directory: ${modelLocation}`)
-        }
-        
-    } else {
-      alert("This feature isn't supported by your system please check your CNTK configuration and try again later.");
-    }
-});
-
-ipcRenderer.on('toggleTracking', (event, message) => {
-  $('#video-tagging').off("stepFwdClicked-BeforeStep");
-  if (trackingEnabled) {
-      $('#video-tagging').off("stepFwdClicked-AfterStep");
-      //add event to prevent non tracked regions from giving suggestions
-      $('#video-tagging').on("stepFwdClicked-BeforeStep", () => {
-          save();
-          var regionCount = $('.regionCanvas').length;
-          if (regionCount) {         
-            var curFrame = videotagging.getCurrentFrame();
-            $.map($('.regionCanvas'), (regionCanvas, i) => { 
-              //get regionId
-              for (regInd = 0; regInd < videotagging.frames[curFrame].length; regInd++) {
-                if (videotagging.frames[curFrame][regInd].name = regionCanvas.id) {
-                  videotagging.frames[curFrame][regInd].blockSuggest = true;
-                }
-              }
-            });
-          }
-      });
-
-
-  } else {
-      videotagging.video.addEventListener("canplay",  initRegionTracking);
-  } 
-  trackingEnabled = !trackingEnabled;
-
-});
-
-//drag and drop support
-document.addEventListener('drop', (e) => {
-    e.preventDefault();
-    e.stopPropagation();
-    if (e.dataTransfer.files[0].type == "video/mp4") {
-      fileSelected(e.dataTransfer.files[0]);
-    }
-    return false;
-});
-
-document.addEventListener('dragover', (e) => {
-    e.preventDefault();
-    if (e.dataTransfer.files[0].type == "video/mp4") {
-      e.dataTransfer.dropEffect = "copy";
-    }
-    e.stopPropagation();
-});
-
-document.addEventListener('dragstart', (e) => {
-    e.preventDefault();
-    let file = e.dataTransfer.files[0];
-    if (file && file.type == "video/mp4") {
-        e.dataTransfer.effectAllowed = "copy";
-    }
-    e.stopPropagation();
-});
-
-// stop zooming
-document.addEventListener('mousewheel', (e) => {
-  if(e.ctrlKey) {
-    e.preventDefault();
-  }
-});
-
-//adds a loading animation to the tagger
-function addLoader() {
-  if(!$('.loader').length) {
-    $("<div class=\"loader\"></div>").appendTo($("#videoWrapper"));
-  }
-}
-
-//managed the visited frames
-function updateVisitedFrames(){
-  visitedFrames.add(videotagging.getCurrentFrame());
-}
-
-function checkPointRegion() {
-  if ($('#regiontype').val() != "Point") {
-    $('#regionPointGroup').hide();
-  } else {
-    $('#regionPointGroup').show();
-  }
-}
-
-//load logic
-function fileSelected(path) {
-   $('#load-message').hide();
-
-  if (path) {  //checking if a video is dropped
-    let pathName = path.path;
-    openPath(pathName);
-  } else { // showing system open dialog
-    dialog.showOpenDialog({
-      filters: [{ name: 'Videos', extensions: ['mp4']}],
-      properties: ['openFile']
-    },
-    function (pathName) {
-      if (pathName) openPath(pathName[0]);
-      else $('#load-message').show();
-    });
-  }
-
-  function openPath(pathName) {
-    var config;
-
-    // show configuration
-    $('#load-message').hide();
-    $('#video-tagging-container').hide()
-    $('#load-form-container').show();
-    $('#framerateGroup').show();
-    
-    //set title indicator
-    $('title').text(`Video Tagging Job Configuration: ${pathJS.basename(pathName, pathJS.extname(pathName))}`);
-    
-    $('#inputtags').tagsinput('removeAll');//remove all previous tag labels
-    $('#output').val(`${basepath}/cntk`);
-    $('#model').val(`${basepath}/cntk/Fast-RCNN.model`);
-
-    try {
-      config = require(`${pathName}.json`);
-      //restore tags
-      $('#inputtags').val(config.inputTags);
-      config.inputTags.split(",").forEach( tag => {
-          $("#inputtags").tagsinput('add',tag);
-      });
-    } catch (e) {
-      console.log(`Error loading save file ${e.message}`);
-    }
-
-    // REPLACE implementation, use jQuery to remove all event handlers on loadbutton
-    document.getElementById('loadButton').parentNode.replaceChild(document.getElementById('loadButton').cloneNode(true), document.getElementById('loadButton'));
-    document.getElementById('loadButton').addEventListener('click', loadTagger);
-    
-    function loadTagger (e) {
-      if(framerate.validity.valid && inputtags.validity.valid) {
-        $('.bootstrap-tagsinput').last().removeClass( "invalid" );
-       
-        $('title').text(`Video Tagging Job: ${pathJS.basename(pathName, pathJS.extname(pathName))}`); //set title indicator
-
-        videotagging = document.getElementById('video-tagging'); //find out why jquery doesn't play nice with polymer
-        videotagging.regiontype = $('#regiontype').val();
-        videotagging.multiregions = 1;
-        videotagging.regionsize = $('#regionsize').val();
-        videotagging.inputtagsarray = $('#inputtags').val().split(',');
-        videotagging.video.currentTime = 0;
-        videotagging.framerate = $('#framerate').val();
-
-        if (config) {
-          videotagging.inputframes = config.frames;
-          visitedFrames = new Set(config.visitedFrames);
-        } else {
-            videotagging.inputframes = {};
-            visitedFrames = new Set();
-        } 
-
-        videotagging.src = ''; // ensures reload if user opens same video 
-        videotagging.src = pathName;
-        
-        //set start time
-        videotagging.video.oncanplay = function (){
-            videotagging.videoStartTime = videotagging.video.currentTime;
-            videotagging.video.oncanplay = undefined;
-        }
-
-        //track visited frames
-        videotagging.video.removeEventListener("canplay", updateVisitedFrames); //remove old listener
-        videotagging.video.addEventListener("canplay",updateVisitedFrames);
-
-        //init region tracking
-        videotagging.video.addEventListener("canplay",  initRegionTracking);
-
-        $('#load-form-container').hide();
-        $('#video-tagging-container').show();
-
-        ipcRenderer.send('setFilePath', pathName);
-      } else {
-        $('.bootstrap-tagsinput').last().addClass( "invalid" );
-      }
-    }
-  }
-}
-
-//saves current video to config 
-function save() {
-    var saveObject = {
-      "frames" : videotagging.frames,
-      "inputTags": $('#inputtags').val(),
-      "exportTo": $('#exportTo').val(),
-      "visitedFrames": Array.from(visitedFrames),
-    };
-    
-    fs.writeFileSync(`${videotagging.src}.json`, JSON.stringify(saveObject));
-}
-
-//maps every frame in the video to an imageCanvas
-function mapVideo(exportUntil, frameHandler) {
-   return new Promise((resolve, reject) => {
-    //init canvas buffer
-    var frameCanvas = document.createElement("canvas");
-    frameCanvas.width = videotagging.video.videoWidth;
-    frameCanvas.height = videotagging.video.videoHeight;
-    var canvasContext = frameCanvas.getContext("2d");
-
-    // start exporting frames using the canplay eventListener
-    videotagging.video.removeEventListener("canplay", updateVisitedFrames); //stop recording frame movment
-    videotagging.video.addEventListener("canplay", iterateFrames);
-    videotagging.video.currentTime = 0;
-    videotagging.playingCallback();
-
-    function iterateFrames() {
-      var frameId = videotagging.getCurrentFrame();
-      var isLastFrame;
-
-      switch(exportUntil) {
-        case "tagged":
-          isLastFrame = (!Object.keys(videotagging.frames).length) || (frameId >= parseInt(Object.keys(videotagging.frames)[Object.keys(videotagging.frames).length-1]));
-          break;
-        case "visited":
-          var lastVisitedFrameId = Math.max.apply(Math, Array.from(visitedFrames));
-          isLastFrame = (frameId >= lastVisitedFrameId);        
-          break;
-        case "last":
-          isLastFrame = (videotagging.video.currentTime >= videotagging.video.duration);
-          break;
-      }
-
-      if (isLastFrame) {
-        videotagging.video.removeEventListener("canplay", iterateFrames);
-        videotagging.video.addEventListener("canplay", updateVisitedFrames);
-        resolve();
-      }
-      
-      frameHandler(frameId, frameCanvas, canvasContext);
-      if (!isLastFrame) {
-        videotagging.stepFwdClicked(false);
-      } 
-    }
-  });
-}
-
-//exports frames to cntk format for model training
-function exportCNTK() {
-  addLoader();
-
-  /* create constants for all the paths with meaningful names*/
-
-  //make sure paths exist
-  var exportPath = $('#output').val();
-  if (!fs.existsSync(`${exportPath}`)) fs.mkdirSync(`${exportPath}`);
-  var framesPath = `${exportPath}/${pathJS.basename(videotagging.src, pathJS.extname(videotagging.src))}_frames`;
-
-  //clear past directory 
-  rimraf(framesPath, () => { 
-    fs.mkdirSync(framesPath);
-    fs.mkdirSync(`${framesPath}/positive`);
-    fs.mkdirSync(`${framesPath}/negative`);
-
-    mapVideo($('#exportTo').val(), exportFrame).then(() => {
-      $(".loader").remove();
-      let notification = new Notification('Offline Video Tagger', {
-        body: 'Successfully exported CNTK files.'
-      });
-    })
-  });
-
-
-  function exportFrame(frameId, frameCanvas, canvasContext) {
-
-    //set default writepath to the negative folder
-    var writePath = `${framesPath}/negative/${pathJS.basename(videotagging.src, pathJS.extname(videotagging.src))}_frame_${frameId}.jpg`; //defaults to negative
-    var positiveWritePath = `${framesPath}/positive/${pathJS.basename(videotagging.src, pathJS.extname(videotagging.src))}_frame_${frameId}.jpg`;
-    //If frame contains tags generate the metadata and save it in the positive directory
-    var frameIsTagged = videotagging.frames.hasOwnProperty(frameId) && (videotagging.frames[frameId].length);
-    if (frameIsTagged && (videotagging.getUnlabeledRegionTags(frameId).length != videotagging.frames[frameId].length)) {
-        //genereate metadata from tags
-        var frameBBoxes = "",
-            frameLabels = "";
-        videotagging.frames[frameId].map( (tag) => {
-          if (!tag.tags[tag.tags.length-1]) {
-             return console.log(`frame ${frameId} region ${tag.name} has no label`);
-          }
-          var stanW = videotagging.video.videoWidth/tag.width;
-          var stanH = videotagging.video.videoHeight/tag.height;
-          frameBBoxes += `${tag.tags[tag.tags.length-1]}\n`;
-          frameLabels += `${parseInt(tag.x1 * stanW)}\t${parseInt(tag.y1 * stanH)}\t${parseInt(tag.x2 * stanW)}\t${parseInt(tag.y2 * stanH)}\n`;
-        });
-        if (frameBBoxes == "" || frameLabels == "") return;
-        fs.writeFileSync(positiveWritePath.replace('.jpg', '.bboxes.labels.tsv'), frameLabels, (err) => {console.error(err)});
-        fs.writeFileSync(positiveWritePath.replace('.jpg', '.bboxes.tsv'), frameBBoxes, (err) => {console.error(err)});
-        writePath = positiveWritePath; // set write path to positve write path
-    }
-
-    //draw the frame to the canvas
-    canvasContext.drawImage(videotagging.video, 0, 0);
-    var data = frameCanvas.toDataURL('image/jpeg').replace(/^data:image\/\w+;base64,/, ""); // strip off the data: url prefix to get just the base64-encoded bytes http://stackoverflow.com/questions/5867534/how-to-save-canvas-data-to-file
-    var buf = new Buffer(data, 'base64');
-
-    //write canvas to file and change frame
-    console.log('saving file', writePath);
-    if (writePath.includes("negative") && !visitedFrames.has(frameId)) return; //only write visited frames
-    fs.writeFileSync(writePath, buf);
-
-  }
-}
-
-//allows user to review cntk suggestions on a video
-function reviewCNTK() {
-  addLoader();
-  //check if an export directory for the current model exists
-  var exportPath = $('#output').val();
-  if (!fs.existsSync(`${exportPath}`)) fs.mkdirSync(`${exportPath}`);
-  var reviewPath = `${exportPath}/${pathJS.basename(videotagging.src, pathJS.extname(videotagging.src))}_review`;
-  //if the export directory does not exist create it and export all the frames then review
-  if (!fs.existsSync(reviewPath)) {
-    fs.mkdirSync(reviewPath);
-    mapVideo("last", saveFrame).then(review);
-  } else {
-    review();
-  }
-
-  function review() {
-    //run the model on the reviewPath directory
-    model = new cntkModel.CNTKFRCNNModel({cntkModelPath : $('#model').val(), verbose : true});
-    var modelTagsPromise = new Promise((resolve, reject) => { 
-      model.evaluateDirectory(reviewPath, (err, res) => {
-        if (err) {
-            console.info(err);
-            reject();
-        }
-        resolve(res);
-      });
-    });
-
-    modelTagsPromise.then( (modelTags) => {
-      videotagging.video.removeEventListener("canplay", initRegionTracking); //remove region tracking listener
-      $('#video-tagging').off("stepFwdClicked-BeforeStep");
-      $('#video-tagging').off("stepFwdClicked-AfterStep");
-      videotagging.frames=[];
-      videotagging.optionalTags.createTagControls(Object.keys(modelTags.classes));
-
-      //Create regions based on the provided modelTags
-      Object.keys(modelTags.frames).map( (pathId) => {
-        var frameImage = new Image();
-        frameImage.src = `${reviewPath}\\${pathId}`;
-        frameImage.onload = loadFrameRegions; 
-
-        function loadFrameRegions() {
-          var imageWidth = this.width;
-          var imageHeight = this.height;
-          frameId = pathId.replace(".jpg", "");//remove.jpg
-          videotagging.frames[frameId] = [];
-          modelTags.frames[pathId].regions.forEach( (region) => {
-            videotagging.frames[frameId].push({
-              x1:region.x1,
-              y1:region.y1,
-              x2:region.x2,
-              y2:region.y2,                          
-              id:videotagging.uniqueTagId++,
-              width:imageWidth,
-              height:imageHeight,
-              type:videotagging.regiontype,
-              tags:Object.keys(modelTags.classes).filter( (key) => {return modelTags.classes[key] === region.class }),
-              name:(videotagging.frames[frameId].length + 1)
-            }); 
-          });
-          }
-        });
-        videotagging.showAllRegions();
-
-      //cleanup and notify
-      $(".loader").remove();
-      videotagging.video.currentTime = 0;
-      videotagging.playingCallback();
-      let notification = new Notification('Offline Video Tagger', { body: 'Model Ready For Review.' });
-
-    });
-  }
-
-  function saveFrame(frameId, fCanvas, canvasContext){
-    canvasContext.drawImage(videotagging.video, 0, 0);
-    var writePath = reviewPath+ `/${frameId}.jpg`
-    var data = fCanvas.toDataURL('image/jpeg').replace(/^data:image\/\w+;base64,/, ""); // strip off the data: url prefix to get just the base64-encoded bytes http://stackoverflow.com/questions/5867534/how-to-save-canvas-data-to-file
-    var buf = new Buffer(data, 'base64');
-    //write canvas to file and change frame
-    console.log('saving file', writePath);
-    if (!fs.existsSync(writePath)) {
-      fs.writeFileSync(writePath, buf);
-    }  
-  }
-
-}
-
-//optomize superRegionTracking 
-function initRegionTracking () {
-    videotagging.video.removeEventListener("canplay", initRegionTracking); //remove old listener
-    $('#video-tagging').off("stepFwdClicked-BeforeStep");
-    $('#video-tagging').off("stepFwdClicked-AfterStep");
-    var regionsToTrack = [];
-    
-    $('#video-tagging').on("stepFwdClicked-BeforeStep", () => {
-      save();
-      videotagging.canMove = false;
-      var regionCount = $('.regionCanvas').length;
-      if (regionCount) {         
-        var curFrame = videotagging.getCurrentFrame();
-        //init store imagedata for scene detection
-        var stanW = videotagging.video.videoWidth/videotagging.video.offsetWidth,
-            stanH = videotagging.video.videoHeight/videotagging.video.offsetHeight;
-        $.map($('.regionCanvas'), (regionCanvas, i) => { 
-          //scale window to video 
-          var w = Math.round(parseInt(regionCanvas.style.width) * stanW);
-          var h = Math.round(parseInt(regionCanvas.style.height) * stanH);
-          var y = Math.round(parseInt(regionCanvas.style.top) * stanH);
-          var x = Math.round(parseInt(regionCanvas.style.left) * stanW);
-          //get regionId
-          var originalRegion = $.grep(videotagging.frames[curFrame], function(e){ return e.name == regionCanvas.id;})[0];
-          //if region is in blacklist don't track
-          if (originalRegion.blockSuggest) return;
-          //add region to be tracked 
-          regionsToTrack.push({x,y,w,h,originalRegion});
-        });
-      }
-    });
-
-    $('#video-tagging').on("stepFwdClicked-AfterStep", () => {
-        videotagging.video.addEventListener("canplay", afterStep);
-    });  
-
-    function afterStep() {
-      videotagging.video.removeEventListener("canplay", afterStep);
-      if (regionsToTrack.length) {
-          var curFrame = videotagging.getCurrentFrame(),
-              stanW = videotagging.video.offsetWidth/videotagging.video.videoWidth,
-              stanH = videotagging.video.offsetHeight/videotagging.video.videoHeight;
-          // pass regions to super tracking 
-          superTracking(regionsToTrack).then( (suggestions) => {
-            suggestions.forEach((suggestion) => {
-                var x1, y1, x2, y2;
-                if (suggestion.type == "copy") {
-                  x1 = suggestion.region.x * stanW;
-                  y1 = suggestion.region.y * stanH;
-                  x2 = x1 + suggestion.region.w * stanW;
-                  y2 = y1 + suggestion.region.h * stanH;
-                } else { //get bounding box of tracked object
-                  x1 = Math.max(Math.round(suggestion.region.x * stanW) - Math.round((suggestion.region.w * stanW)/2), 0),
-                  y1 = Math.max(Math.round(suggestion.region.y * stanH) - Math.round((suggestion.region.h * stanH)/2), 0),
-                  x2 = Math.min(Math.round(suggestion.region.w * stanW) + x1, videotagging.video.offsetWidth),
-                  y2 = Math.min(Math.round(suggestion.region.h * stanH) + y1, videotagging.video.offsetHeight);
-                }
-                // create new region
-                videotagging.createRegion(x1, y1, x2, y2);
-                videotagging.frames[curFrame][videotagging.frames[curFrame].length-1].tags = suggestion.originalRegion.tags;
-                videotagging.frames[curFrame][videotagging.frames[curFrame].length-1].suggestedBy = {frameId:curFrame-1, regionId:suggestion.originalRegion.id};        
-                // add suggested by to previous region to blacklist
-                videotagging.frames[curFrame-1][suggestion.originalRegion.name-1].blockSuggest = true;
-           });   
-            regionsToTrack = [];
-            videotagging.canMove = true;
-          }).catch( (e) => {
-              console.info(e);
-              regionsToTrack = [];
-              videotagging.canMove = true;
-          });
-      } else {
-        videotagging.canMove =true;
-      }
-      
-    }     
-
-    
-
-    function superTracking(regions) {
-      return new Promise((resolve, reject) => {
-
-        supertrackingFrameRate = Math.max(supertrackingFrameRate,videotagging.framerate + 1);
-        var video = document.createElement('video');
-        video.src = videotagging.video.src;
-        video.currentTime = videotagging.video.currentTime - (1/videotagging.framerate);
-        video.oncanplay = init;    
-        video.load();
-
-        var cstracker = new regiontrackr.camshift.Tracker({whitebalancing : false, calcAngles : false});
-        var tagging_duration, frameCanvas, canvasContext, scd, rcd;
-        var suggestions = [];
-        
-        function init() {
-          video.oncanplay = undefined;
-          frameCanvas = document.createElement("canvas");
-          frameCanvas.width = video.videoWidth;
-          frameCanvas.height = video.videoHeight;
-          canvasContext = frameCanvas.getContext("2d")
-          canvasContext.drawImage(video, 0, 0);
-
-          tagging_duration = Math.min(videotagging.video.currentTime , videotagging.video.duration);
-
-          //detect if scene change
-          scd = new SceneChangeDetector({ threshold:49, detectionRegion: { w:frameCanvas.width, h:frameCanvas.height } });
-          scd.detectSceneChange(video, frameCanvas, canvasContext, videotagging.framerate).then( (sceneChanged) => {          
-              if (!sceneChanged) {
-                video.oncanplay = trackFrames;    
-                video.currentTime += (1/supertrackingFrameRate);          
-              } else {
-                reject("scene changed");
-              }
-          }).catch((e) => {console.info(e)});
-        }
-
-        function trackFrames() { 
-          if (!regions.length) {
-            video.oncanplay = undefined;    
-            return resolve(suggestions);
-          }
-          var regionDetectionPromises = [];
-          regions.forEach((region, i) => {
-              // if first pass check whether the region changed
-              if (region.regionChanged === undefined) {
-                  rcd = new SceneChangeDetector({ threshold:1, detectionRegion: { w:region.w, h:region.h} });   
-                  regionDetectionPromises.push(rcd.detectRegionChange(video, frameCanvas, region, i, videotagging.framerate));
-              } else {
-                if (region.regionChanged) {
-                  trackRegion(region, i);
-                }
-              }                
-          });  
-
-          Promise.all(regionDetectionPromises).then((values) => {
-             values.sort((a,b) => { return b.index - a.index; });//sort so removal works correctly
-             values.forEach ( (rp) => { //rp is resolved promise
-                if (rp.regionChanged) {
-                  trackRegion(rp.region, rp.index); 
-                  regions[rp.index].regionChanged = rp.regionChanged; //make sure region change only executes once
-                } else {
-                  suggestions.push({type:"copy", region : rp.region, originalRegion: rp.region.originalRegion});
-                   regions.splice(rp.index,1);
-                }
-              });
-              if (video.currentTime >= tagging_duration) {
-                video.oncanplay = undefined;    
-                resolve(suggestions);
-              } else { 
-                video.currentTime += (1 / supertrackingFrameRate);//go to next frame segment
-              }
-          },(e)=>{console.info(e);});
-
-        }
-        function trackRegion(region, i) {   
-            //on exit condition push to suggestions
-            if (video.currentTime >= tagging_duration) {
-              if (region.trackedObject) {
-                suggestions.push( {type:"tracked", region:{x : region.trackedObject.x, y : region.trackedObject.y, w : region.w, h: region.h}, originalRegion: region.originalRegion});
-              } 
-            }
-            //init tracker for region  
-            if (!region.trackedObject){
-              cstracker.initTracker(frameCanvas, new regiontrackr.camshift.Rectangle(region.x, region.y, region.w, region.h));
-            } else {
-              if (region.trackedObject.width === 0 || region.trackedObject.height === 0 ) { //skip tracking if object disapeared
-                return;
-              } else {
-                cstracker.initTracker(frameCanvas, new regiontrackr.camshift.Rectangle(region.tx, region.ty, region.w, region.h));
-              }
-            }
-            //track region
-            canvasContext.drawImage(video, 0, 0);
-            cstracker.track(frameCanvas);
-            regions[i].trackedObject = cstracker.getTrackObj();  
-            //create new tracker 
-            regions[i].tx = Math.round(region.trackedObject.x - region.w/2);
-            regions[i].ty = Math.round(region.trackedObject.y - region.h/2);
-        }
-      });
-
-    } 
-}

main.js

@@ -27,7 +27,7 @@ function createWindow () {
       y: mainWindowState.y,
       minHeight: 480,
       minWidth: 480,
-      icon: __dirname + '/icon.png',
+      icon: __dirname + '/src/public/images/icon.png',
       show: false
   });
 
@@ -35,7 +35,7 @@ function createWindow () {
 
   // and load the index.html of the app.
   mainWindow.loadURL(url.format({
-    pathname: path.join(__dirname, 'index.html'),
+    pathname: path.join(__dirname, 'src/index.html'),
     protocol: 'file:',
     slashes: true
   }));
@@ -47,10 +47,59 @@ function createWindow () {
     let p = (process.platform === 'darwin') ? 1 : 0;
     menu.items[p].submenu.items[1].enabled = true;
     menu.items[p].submenu.items[2].enabled = true;
+    menu.items[p].submenu.items[3].enabled = true;
     menu.items[p+1].submenu.items[0].enabled = true;
     menu.items[p+1].submenu.items[1].enabled = true;
   });
 
+
+  // do this independently for each object
+  ipcMain.on('show-popup', function(event, arg) { 
+      let popup = new BrowserWindow({
+        parent: mainWindow, 
+        modal: true, 
+        show: false, 
+        frame: false,
+        autoHideMenuBar : true
+      });
+      
+      switch (arg.type) {
+        case "export":
+            popup.setSize(359, 300);
+            popup.loadURL(url.format({
+              pathname: path.join(__dirname, 'src/public/html/export-configuration.html'),
+              protocol: 'file:',
+              slashes: true
+            }));
+          break;
+
+        case "review":
+          popup.setSize(359, 310);
+          popup.loadURL(url.format({
+            pathname: path.join(__dirname, 'src/public/html/review-configuration.html'),
+            protocol: 'file:',
+            slashes: true
+          }));
+          break;
+          
+        default: return; 
+      }
+      
+      popup.once('ready-to-show', () => {
+        popup.send('configs', arg);
+        popup.show();
+      });
+
+  });
+
+  ipcMain.on('export-tags', (event, arg) => {
+    mainWindow.send('export-tags', arg);
+  });
+
+  ipcMain.on('review-model', (event, arg) => {
+    mainWindow.send('review-model', arg);
+  });
+
   mainWindow.on('ready-to-show', function() {
       mainWindow.show();
       mainWindow.focus();
@@ -74,6 +123,11 @@ function createWindow () {
           accelerator: 'CmdOrCtrl+O',
           click () { mainWindow.webContents.send('openVideo'); }
         },
+        {
+          label: 'Open Image Directory...',
+          accelerator: 'CmdOrCtrl+I',
+          click () { mainWindow.webContents.send('openImageDirectory'); }
+        },
         {
           label: 'Save',
           accelerator: 'CmdOrCtrl+S',
@@ -89,22 +143,34 @@ function createWindow () {
       ]
     },
     {
-      label: 'CNTK',
+      label: 'Object Detection',
       submenu: [
         {
-          label: 'Export Tags to CNTK',
+          label: 'Export Tags',
           accelerator: 'CmdOrCtrl+E',
           enabled: false,
-          click () { mainWindow.webContents.send('exportCNTK'); }
+          click () { mainWindow.webContents.send('export'); }
         },
         {
-          label: 'Review CNTK Model',
+          label: 'Review Detection Model',
           accelerator: 'CmdOrCtrl+R',
           enabled: false,
-          click () { mainWindow.webContents.send('reviewCNTK'); }
+          click () { mainWindow.webContents.send('review'); }
         }
       ]
     },
+    {
+    label: "Edit",
+      submenu: [
+          { label: "Undo", accelerator: "CmdOrCtrl+Z", selector: "undo:" },
+          { label: "Redo", accelerator: "Shift+CmdOrCtrl+Z", selector: "redo:" },
+          { type: "separator" },
+          { label: "Cut", accelerator: "CmdOrCtrl+X", selector: "cut:" },
+          { label: "Copy", accelerator: "CmdOrCtrl+C", selector: "copy:" },
+          { label: "Paste", accelerator: "CmdOrCtrl+V", selector: "paste:" },
+          { label: "Select All", accelerator: "CmdOrCtrl+A", selector: "selectAll:" }
+      ]
+    },
     {
       label: 'Debug',
       submenu: [
@@ -154,10 +220,12 @@ if (process.platform === 'darwin') {
     })
     template[1].submenu.push();
     template[2].submenu.push();
+    
   }
 
   const menu = Menu.buildFromTemplate(template);
   Menu.setApplicationMenu(menu);
+
 }
 
 // This method will be called when Electron has finished

package.json

@@ -1,29 +1,30 @@
 {
-  "name": "offline-video-tagger",
+  "name": "vott",
   "version": "1.0.0",
-  "description": "An offline port of the video-tagging-tool for electron.",
+  "description": "An electron app for building end to end Object Detection Models with CNTK from Sample Videos.",
   "main": "main.js",
   "scripts": {
     "start": "electron ."
   },
-  "repository": "https://github.com/CatalystCode/offline-video-tagger/",
+  "repository": "https://github.com/CatalystCode/VOTT",
   "keywords": [
-    "Electron",
     "Video-Tagging",
     "CNTK",
-    "tutorial",
-    "demo"
+    "Deep-Learning",
+    "Object-Detection"
   ],
-  "author": "GitHub",
+  "readme": "README.md",
+  "author": "aribornstein",
   "license": "MIT",
   "devDependencies": {
-    "electron": "^1.4.1",
     "bower": "^1.7.2"
   },
   "dependencies": {
-    "cntk-fastrcnn": "^0.1.5",
+    "async": "^2.1.5",
+    "cntk-fastrcnn": "^0.2.6",
+    "electron": "^1.4.1",
     "electron-window-state": "^4.0.2",
     "remote": "^0.2.6",
-    "rimraf": "^2.6.1"
+    "replace": "^0.3.0"
   }
 }