Trialwise Decoding¶
In this example, we will use a convolutional neural network on the Physiobank EEG Motor Movement/Imagery Dataset to decode two classes:
- Executed and imagined opening and closing of both hands
- Executed and imagined opening and closing of both feet
We use only one subject (with 90 trials) in this tutorial for
demonstration purposes. A more interesting decoding task with many more
trials would be to do cross-subject decoding on the same dataset.
Enable logging¶
In [2]:
import logging
import importlib
importlib.reload(logging) # see https://stackoverflow.com/a/21475297/1469195
log = logging.getLogger()
log.setLevel('INFO')
import sys
logging.basicConfig(format='%(asctime)s %(levelname)s : %(message)s',
level=logging.INFO, stream=sys.stdout)
Load data¶
You can load and preprocess your EEG dataset in any way, Braindecode
only expects a 3darray (trials, channels, timesteps) of input signals
X and a vector of labels y later (see below). In this tutorial,
we will use the MNE
library to load an EEG motor imagery/motor execution dataset. For a
tutorial from MNE using Common Spatial Patterns to decode this data, see
here.
For another library useful for loading EEG data, take a look at Neo
IO.
In [3]:
import mne
from mne.io import concatenate_raws
# 5,6,7,10,13,14 are codes for executed and imagined hands/feet
subject_id = 22 # carefully cherry-picked to give nice results on such limited data :)
event_codes = [5,6,9,10,13,14]
#event_codes = [3,4,5,6,7,8,9,10,11,12,13,14]
# This will download the files if you don't have them yet,
# and then return the paths to the files.
physionet_paths = mne.datasets.eegbci.load_data(subject_id, event_codes)
# Load each of the files
parts = [mne.io.read_raw_edf(path, preload=True,stim_channel='auto', verbose='WARNING')
for path in physionet_paths]
# Concatenate them
raw = concatenate_raws(parts)
# Find the events in this dataset
events = mne.find_events(raw, shortest_event=0, stim_channel='STI 014')
# Use only EEG channels
eeg_channel_inds = mne.pick_types(raw.info, meg=False, eeg=True, stim=False, eog=False,
exclude='bads')
# Extract trials, only using EEG channels
epoched = mne.Epochs(raw, events, dict(hands_or_left=2, feet_or_right=3), tmin=1, tmax=4.1, proj=False, picks=eeg_channel_inds,
baseline=None, preload=True)
Convert data to Braindecode format¶
Braindecode has a minimalistic SignalAndTarget class, with
attributes X for the signal and y for the labels. X should
have these dimensions: trials x channels x timesteps. y should have
one label per trial.
In [4]:
import numpy as np
# Convert data from volt to millivolt
# Pytorch expects float32 for input and int64 for labels.
X = (epoched.get_data() * 1e6).astype(np.float32)
y = (epoched.events[:,2] - 2).astype(np.int64) #2,3 -> 0,1
We use the first 40 trials for training and the next 30 trials for validation. The validation accuracies can be used to tune hyperparameters such as learning rate etc. The final 20 trials are split apart so we have a final hold-out evaluation set that is not part of any hyperparameter optimization. As mentioned before, this dataset is dangerously small to get any meaningful results and only used here for quick demonstration purposes.
In [5]:
from braindecode.datautil.signal_target import SignalAndTarget
train_set = SignalAndTarget(X[:40], y=y[:40])
valid_set = SignalAndTarget(X[40:70], y=y[40:70])
Create the model¶
Braindecode comes with some predefined convolutional neural network architectures for raw time-domain EEG. Here, we use the shallow ConvNet model from Deep learning with convolutional neural networks for EEG decoding and visualization.
In [6]:
from braindecode.models.shallow_fbcsp import ShallowFBCSPNet
from torch import nn
from braindecode.torch_ext.util import set_random_seeds
# Set if you want to use GPU
# You can also use torch.cuda.is_available() to determine if cuda is available on your machine.
cuda = False
set_random_seeds(seed=20170629, cuda=cuda)
n_classes = 2
in_chans = train_set.X.shape[1]
# final_conv_length = auto ensures we only get a single output in the time dimension
model = ShallowFBCSPNet(in_chans=in_chans, n_classes=n_classes,
input_time_length=train_set.X.shape[2],
final_conv_length='auto')
if cuda:
model.cuda()
We use AdamW to optimize the parameters of our network together with Cosine Annealing of the learning rate. We supply some default parameters that we have found to work well for motor decoding, however we strongly encourage you to perform your own hyperparameter optimization using cross validation on your training data.
We will now use the Braindecode model class directly to perform the
training in a few lines of code. If you instead want to use your own
training loop, have a look at the Trialwise Low-Level
Tutorial.
In [7]:
from braindecode.torch_ext.optimizers import AdamW
import torch.nn.functional as F
#optimizer = AdamW(model.parameters(), lr=1*0.01, weight_decay=0.5*0.001) # these are good values for the deep model
optimizer = AdamW(model.parameters(), lr=0.0625 * 0.01, weight_decay=0)
model.compile(loss=F.nll_loss, optimizer=optimizer, iterator_seed=1,)
Run the training¶
In [8]:
model.fit(train_set.X, train_set.y, epochs=30, batch_size=64, scheduler='cosine',
validation_data=(valid_set.X, valid_set.y),)
2018-08-08 11:49:59,336 INFO : Run until first stop...
2018-08-08 11:50:02,270 INFO : Time only for training updates: 2.93s
2018-08-08 11:50:04,211 INFO : Epoch 0
2018-08-08 11:50:04,216 INFO : train_loss 1.88531
2018-08-08 11:50:04,219 INFO : valid_loss 2.01343
2018-08-08 11:50:04,223 INFO : train_misclass 0.52500
2018-08-08 11:50:04,226 INFO : valid_misclass 0.53333
2018-08-08 11:50:04,230 INFO : runtime 0.00000
2018-08-08 11:50:04,234 INFO :
2018-08-08 11:50:06,959 INFO : Time only for training updates: 2.72s
2018-08-08 11:50:08,869 INFO : Epoch 1
2018-08-08 11:50:08,875 INFO : train_loss 0.66515
2018-08-08 11:50:08,879 INFO : valid_loss 0.78154
2018-08-08 11:50:08,883 INFO : train_misclass 0.35000
2018-08-08 11:50:08,887 INFO : valid_misclass 0.40000
2018-08-08 11:50:08,893 INFO : runtime 4.68826
2018-08-08 11:50:08,897 INFO :
2018-08-08 11:50:11,670 INFO : Time only for training updates: 2.77s
2018-08-08 11:50:13,611 INFO : Epoch 2
2018-08-08 11:50:13,613 INFO : train_loss 0.41463
2018-08-08 11:50:13,615 INFO : valid_loss 0.55128
2018-08-08 11:50:13,616 INFO : train_misclass 0.20000
2018-08-08 11:50:13,617 INFO : valid_misclass 0.23333
2018-08-08 11:50:13,619 INFO : runtime 4.71190
2018-08-08 11:50:13,620 INFO :
2018-08-08 11:50:16,332 INFO : Time only for training updates: 2.71s
2018-08-08 11:50:18,278 INFO : Epoch 3
2018-08-08 11:50:18,283 INFO : train_loss 0.32448
2018-08-08 11:50:18,287 INFO : valid_loss 0.49955
2018-08-08 11:50:18,291 INFO : train_misclass 0.17500
2018-08-08 11:50:18,295 INFO : valid_misclass 0.16667
2018-08-08 11:50:18,300 INFO : runtime 4.66119
2018-08-08 11:50:18,304 INFO :
2018-08-08 11:50:21,011 INFO : Time only for training updates: 2.71s
2018-08-08 11:50:22,936 INFO : Epoch 4
2018-08-08 11:50:22,940 INFO : train_loss 0.23701
2018-08-08 11:50:22,943 INFO : valid_loss 0.44736
2018-08-08 11:50:22,947 INFO : train_misclass 0.10000
2018-08-08 11:50:22,950 INFO : valid_misclass 0.20000
2018-08-08 11:50:22,953 INFO : runtime 4.67849
2018-08-08 11:50:22,957 INFO :
2018-08-08 11:50:25,645 INFO : Time only for training updates: 2.69s
2018-08-08 11:50:27,538 INFO : Epoch 5
2018-08-08 11:50:27,542 INFO : train_loss 0.21576
2018-08-08 11:50:27,546 INFO : valid_loss 0.45930
2018-08-08 11:50:27,550 INFO : train_misclass 0.07500
2018-08-08 11:50:27,555 INFO : valid_misclass 0.16667
2018-08-08 11:50:27,558 INFO : runtime 4.63209
2018-08-08 11:50:27,562 INFO :
2018-08-08 11:50:30,174 INFO : Time only for training updates: 2.61s
2018-08-08 11:50:32,077 INFO : Epoch 6
2018-08-08 11:50:32,082 INFO : train_loss 0.18904
2018-08-08 11:50:32,086 INFO : valid_loss 0.45700
2018-08-08 11:50:32,091 INFO : train_misclass 0.07500
2018-08-08 11:50:32,095 INFO : valid_misclass 0.13333
2018-08-08 11:50:32,099 INFO : runtime 4.53206
2018-08-08 11:50:32,103 INFO :
2018-08-08 11:50:34,804 INFO : Time only for training updates: 2.70s
2018-08-08 11:50:36,701 INFO : Epoch 7
2018-08-08 11:50:36,703 INFO : train_loss 0.15661
2018-08-08 11:50:36,704 INFO : valid_loss 0.44282
2018-08-08 11:50:36,705 INFO : train_misclass 0.05000
2018-08-08 11:50:36,706 INFO : valid_misclass 0.13333
2018-08-08 11:50:36,707 INFO : runtime 4.62976
2018-08-08 11:50:36,709 INFO :
2018-08-08 11:50:39,389 INFO : Time only for training updates: 2.68s
2018-08-08 11:50:41,334 INFO : Epoch 8
2018-08-08 11:50:41,339 INFO : train_loss 0.11771
2018-08-08 11:50:41,343 INFO : valid_loss 0.44731
2018-08-08 11:50:41,347 INFO : train_misclass 0.05000
2018-08-08 11:50:41,351 INFO : valid_misclass 0.16667
2018-08-08 11:50:41,354 INFO : runtime 4.58493
2018-08-08 11:50:41,358 INFO :
2018-08-08 11:50:44,141 INFO : Time only for training updates: 2.78s
2018-08-08 11:50:46,064 INFO : Epoch 9
2018-08-08 11:50:46,068 INFO : train_loss 0.09302
2018-08-08 11:50:46,071 INFO : valid_loss 0.45860
2018-08-08 11:50:46,075 INFO : train_misclass 0.05000
2018-08-08 11:50:46,078 INFO : valid_misclass 0.16667
2018-08-08 11:50:46,082 INFO : runtime 4.75208
2018-08-08 11:50:46,085 INFO :
2018-08-08 11:50:48,835 INFO : Time only for training updates: 2.75s
2018-08-08 11:50:50,793 INFO : Epoch 10
2018-08-08 11:50:50,797 INFO : train_loss 0.07532
2018-08-08 11:50:50,801 INFO : valid_loss 0.47792
2018-08-08 11:50:50,804 INFO : train_misclass 0.02500
2018-08-08 11:50:50,808 INFO : valid_misclass 0.23333
2018-08-08 11:50:50,811 INFO : runtime 4.69131
2018-08-08 11:50:50,814 INFO :
2018-08-08 11:50:53,557 INFO : Time only for training updates: 2.74s
2018-08-08 11:50:55,441 INFO : Epoch 11
2018-08-08 11:50:55,445 INFO : train_loss 0.06256
2018-08-08 11:50:55,448 INFO : valid_loss 0.49903
2018-08-08 11:50:55,452 INFO : train_misclass 0.00000
2018-08-08 11:50:55,456 INFO : valid_misclass 0.26667
2018-08-08 11:50:55,459 INFO : runtime 4.72475
2018-08-08 11:50:55,462 INFO :
2018-08-08 11:50:58,206 INFO : Time only for training updates: 2.74s
2018-08-08 11:51:00,290 INFO : Epoch 12
2018-08-08 11:51:00,295 INFO : train_loss 0.05512
2018-08-08 11:51:00,299 INFO : valid_loss 0.51851
2018-08-08 11:51:00,303 INFO : train_misclass 0.00000
2018-08-08 11:51:00,307 INFO : valid_misclass 0.26667
2018-08-08 11:51:00,311 INFO : runtime 4.64953
2018-08-08 11:51:00,314 INFO :
2018-08-08 11:51:03,079 INFO : Time only for training updates: 2.76s
2018-08-08 11:51:05,008 INFO : Epoch 13
2018-08-08 11:51:05,012 INFO : train_loss 0.05148
2018-08-08 11:51:05,015 INFO : valid_loss 0.53031
2018-08-08 11:51:05,019 INFO : train_misclass 0.00000
2018-08-08 11:51:05,022 INFO : valid_misclass 0.26667
2018-08-08 11:51:05,024 INFO : runtime 4.87176
2018-08-08 11:51:05,025 INFO :
2018-08-08 11:51:07,685 INFO : Time only for training updates: 2.65s
2018-08-08 11:51:09,548 INFO : Epoch 14
2018-08-08 11:51:09,554 INFO : train_loss 0.04834
2018-08-08 11:51:09,559 INFO : valid_loss 0.53809
2018-08-08 11:51:09,564 INFO : train_misclass 0.00000
2018-08-08 11:51:09,565 INFO : valid_misclass 0.30000
2018-08-08 11:51:09,570 INFO : runtime 4.60554
2018-08-08 11:51:09,572 INFO :
2018-08-08 11:51:12,250 INFO : Time only for training updates: 2.68s
2018-08-08 11:51:14,197 INFO : Epoch 15
2018-08-08 11:51:14,201 INFO : train_loss 0.04612
2018-08-08 11:51:14,205 INFO : valid_loss 0.54513
2018-08-08 11:51:14,209 INFO : train_misclass 0.00000
2018-08-08 11:51:14,212 INFO : valid_misclass 0.30000
2018-08-08 11:51:14,215 INFO : runtime 4.56585
2018-08-08 11:51:14,219 INFO :
2018-08-08 11:51:16,869 INFO : Time only for training updates: 2.65s
2018-08-08 11:51:18,730 INFO : Epoch 16
2018-08-08 11:51:18,735 INFO : train_loss 0.04390
2018-08-08 11:51:18,739 INFO : valid_loss 0.55042
2018-08-08 11:51:18,743 INFO : train_misclass 0.00000
2018-08-08 11:51:18,747 INFO : valid_misclass 0.30000
2018-08-08 11:51:18,751 INFO : runtime 4.61905
2018-08-08 11:51:18,755 INFO :
2018-08-08 11:51:21,534 INFO : Time only for training updates: 2.78s
2018-08-08 11:51:23,350 INFO : Epoch 17
2018-08-08 11:51:23,356 INFO : train_loss 0.04045
2018-08-08 11:51:23,361 INFO : valid_loss 0.55286
2018-08-08 11:51:23,366 INFO : train_misclass 0.00000
2018-08-08 11:51:23,370 INFO : valid_misclass 0.26667
2018-08-08 11:51:23,375 INFO : runtime 4.66496
2018-08-08 11:51:23,377 INFO :
2018-08-08 11:51:26,114 INFO : Time only for training updates: 2.74s
2018-08-08 11:51:28,027 INFO : Epoch 18
2018-08-08 11:51:28,032 INFO : train_loss 0.03734
2018-08-08 11:51:28,035 INFO : valid_loss 0.55090
2018-08-08 11:51:28,038 INFO : train_misclass 0.00000
2018-08-08 11:51:28,041 INFO : valid_misclass 0.26667
2018-08-08 11:51:28,045 INFO : runtime 4.57936
2018-08-08 11:51:28,049 INFO :
2018-08-08 11:51:30,782 INFO : Time only for training updates: 2.73s
2018-08-08 11:51:32,686 INFO : Epoch 19
2018-08-08 11:51:32,691 INFO : train_loss 0.03431
2018-08-08 11:51:32,695 INFO : valid_loss 0.54657
2018-08-08 11:51:32,699 INFO : train_misclass 0.00000
2018-08-08 11:51:32,703 INFO : valid_misclass 0.26667
2018-08-08 11:51:32,707 INFO : runtime 4.66918
2018-08-08 11:51:32,710 INFO :
2018-08-08 11:51:35,408 INFO : Time only for training updates: 2.69s
2018-08-08 11:51:37,324 INFO : Epoch 20
2018-08-08 11:51:37,329 INFO : train_loss 0.03115
2018-08-08 11:51:37,333 INFO : valid_loss 0.54284
2018-08-08 11:51:37,344 INFO : train_misclass 0.00000
2018-08-08 11:51:37,348 INFO : valid_misclass 0.26667
2018-08-08 11:51:37,353 INFO : runtime 4.62276
2018-08-08 11:51:37,356 INFO :
2018-08-08 11:51:39,948 INFO : Time only for training updates: 2.59s
2018-08-08 11:51:41,864 INFO : Epoch 21
2018-08-08 11:51:41,869 INFO : train_loss 0.02841
2018-08-08 11:51:41,873 INFO : valid_loss 0.54000
2018-08-08 11:51:41,877 INFO : train_misclass 0.00000
2018-08-08 11:51:41,881 INFO : valid_misclass 0.26667
2018-08-08 11:51:41,882 INFO : runtime 4.54261
2018-08-08 11:51:41,883 INFO :
2018-08-08 11:51:44,606 INFO : Time only for training updates: 2.72s
2018-08-08 11:51:46,509 INFO : Epoch 22
2018-08-08 11:51:46,514 INFO : train_loss 0.02606
2018-08-08 11:51:46,517 INFO : valid_loss 0.53694
2018-08-08 11:51:46,521 INFO : train_misclass 0.00000
2018-08-08 11:51:46,524 INFO : valid_misclass 0.26667
2018-08-08 11:51:46,527 INFO : runtime 4.65737
2018-08-08 11:51:46,531 INFO :
2018-08-08 11:51:49,151 INFO : Time only for training updates: 2.62s
2018-08-08 11:51:51,057 INFO : Epoch 23
2018-08-08 11:51:51,061 INFO : train_loss 0.02413
2018-08-08 11:51:51,065 INFO : valid_loss 0.53370
2018-08-08 11:51:51,068 INFO : train_misclass 0.00000
2018-08-08 11:51:51,072 INFO : valid_misclass 0.26667
2018-08-08 11:51:51,075 INFO : runtime 4.54569
2018-08-08 11:51:51,079 INFO :
2018-08-08 11:51:53,810 INFO : Time only for training updates: 2.73s
2018-08-08 11:51:55,771 INFO : Epoch 24
2018-08-08 11:51:55,776 INFO : train_loss 0.02262
2018-08-08 11:51:55,783 INFO : valid_loss 0.53038
2018-08-08 11:51:55,784 INFO : train_misclass 0.00000
2018-08-08 11:51:55,791 INFO : valid_misclass 0.26667
2018-08-08 11:51:55,792 INFO : runtime 4.65646
2018-08-08 11:51:55,799 INFO :
2018-08-08 11:51:58,476 INFO : Time only for training updates: 2.68s
2018-08-08 11:52:00,433 INFO : Epoch 25
2018-08-08 11:52:00,438 INFO : train_loss 0.02136
2018-08-08 11:52:00,441 INFO : valid_loss 0.52729
2018-08-08 11:52:00,445 INFO : train_misclass 0.00000
2018-08-08 11:52:00,449 INFO : valid_misclass 0.26667
2018-08-08 11:52:00,453 INFO : runtime 4.66856
2018-08-08 11:52:00,457 INFO :
2018-08-08 11:52:03,160 INFO : Time only for training updates: 2.70s
2018-08-08 11:52:05,055 INFO : Epoch 26
2018-08-08 11:52:05,059 INFO : train_loss 0.02039
2018-08-08 11:52:05,063 INFO : valid_loss 0.52436
2018-08-08 11:52:05,067 INFO : train_misclass 0.00000
2018-08-08 11:52:05,070 INFO : valid_misclass 0.26667
2018-08-08 11:52:05,073 INFO : runtime 4.68141
2018-08-08 11:52:05,076 INFO :
2018-08-08 11:52:07,806 INFO : Time only for training updates: 2.73s
2018-08-08 11:52:09,685 INFO : Epoch 27
2018-08-08 11:52:09,689 INFO : train_loss 0.01964
2018-08-08 11:52:09,693 INFO : valid_loss 0.52170
2018-08-08 11:52:09,696 INFO : train_misclass 0.00000
2018-08-08 11:52:09,700 INFO : valid_misclass 0.26667
2018-08-08 11:52:09,703 INFO : runtime 4.64933
2018-08-08 11:52:09,707 INFO :
2018-08-08 11:52:12,420 INFO : Time only for training updates: 2.71s
2018-08-08 11:52:14,408 INFO : Epoch 28
2018-08-08 11:52:14,413 INFO : train_loss 0.01904
2018-08-08 11:52:14,417 INFO : valid_loss 0.51934
2018-08-08 11:52:14,421 INFO : train_misclass 0.00000
2018-08-08 11:52:14,425 INFO : valid_misclass 0.26667
2018-08-08 11:52:14,429 INFO : runtime 4.61259
2018-08-08 11:52:14,433 INFO :
2018-08-08 11:52:17,092 INFO : Time only for training updates: 2.66s
2018-08-08 11:52:18,984 INFO : Epoch 29
2018-08-08 11:52:18,989 INFO : train_loss 0.01858
2018-08-08 11:52:18,994 INFO : valid_loss 0.51724
2018-08-08 11:52:18,999 INFO : train_misclass 0.00000
2018-08-08 11:52:19,003 INFO : valid_misclass 0.26667
2018-08-08 11:52:19,007 INFO : runtime 4.67249
2018-08-08 11:52:19,011 INFO :
Out[8]:
<braindecode.experiments.experiment.Experiment at 0x7ffae9f407f0>
The monitored values are also stored into a pandas dataframe:
In [9]:
model.epochs_df
Out[9]:
| train_loss | valid_loss | train_misclass | valid_misclass | runtime | |
|---|---|---|---|---|---|
| 0 | 1.885311 | 2.013432 | 0.525 | 0.533333 | 0.000000 |
| 1 | 0.665155 | 0.781536 | 0.350 | 0.400000 | 4.688265 |
| 2 | 0.414632 | 0.551279 | 0.200 | 0.233333 | 4.711897 |
| 3 | 0.324484 | 0.499546 | 0.175 | 0.166667 | 4.661192 |
| 4 | 0.237008 | 0.447357 | 0.100 | 0.200000 | 4.678494 |
| 5 | 0.215761 | 0.459302 | 0.075 | 0.166667 | 4.632092 |
| 6 | 0.189035 | 0.457000 | 0.075 | 0.133333 | 4.532057 |
| 7 | 0.156614 | 0.442818 | 0.050 | 0.133333 | 4.629760 |
| 8 | 0.117707 | 0.447310 | 0.050 | 0.166667 | 4.584930 |
| 9 | 0.093025 | 0.458601 | 0.050 | 0.166667 | 4.752083 |
| 10 | 0.075319 | 0.477922 | 0.025 | 0.233333 | 4.691312 |
| 11 | 0.062565 | 0.499030 | 0.000 | 0.266667 | 4.724754 |
| 12 | 0.055121 | 0.518506 | 0.000 | 0.266667 | 4.649527 |
| 13 | 0.051483 | 0.530313 | 0.000 | 0.266667 | 4.871762 |
| 14 | 0.048344 | 0.538089 | 0.000 | 0.300000 | 4.605540 |
| 15 | 0.046123 | 0.545129 | 0.000 | 0.300000 | 4.565845 |
| 16 | 0.043898 | 0.550419 | 0.000 | 0.300000 | 4.619048 |
| 17 | 0.040452 | 0.552857 | 0.000 | 0.266667 | 4.664961 |
| 18 | 0.037340 | 0.550898 | 0.000 | 0.266667 | 4.579360 |
| 19 | 0.034310 | 0.546568 | 0.000 | 0.266667 | 4.669176 |
| 20 | 0.031152 | 0.542845 | 0.000 | 0.266667 | 4.622759 |
| 21 | 0.028410 | 0.539997 | 0.000 | 0.266667 | 4.542608 |
| 22 | 0.026065 | 0.536938 | 0.000 | 0.266667 | 4.657368 |
| 23 | 0.024133 | 0.533703 | 0.000 | 0.266667 | 4.545693 |
| 24 | 0.022622 | 0.530376 | 0.000 | 0.266667 | 4.656460 |
| 25 | 0.021355 | 0.527293 | 0.000 | 0.266667 | 4.668564 |
| 26 | 0.020391 | 0.524360 | 0.000 | 0.266667 | 4.681408 |
| 27 | 0.019635 | 0.521704 | 0.000 | 0.266667 | 4.649333 |
| 28 | 0.019043 | 0.519344 | 0.000 | 0.266667 | 4.612593 |
| 29 | 0.018577 | 0.517245 | 0.000 | 0.266667 | 4.672487 |
Eventually, we arrive at 83.4% accuracy, so 25 from 30 trials are correctly predicted. In the Cropped Decoding Tutorial, we can learn how to achieve higher accuracies using cropped training.
Evaluation¶
Once we have all our hyperparameters and architectural choices done, we can evaluate the accuracies to report in our publication by evaluating on the test set:
In [10]:
test_set = SignalAndTarget(X[70:], y=y[70:])
model.evaluate(test_set.X, test_set.y)
Out[10]:
{'loss': 0.2964690923690796,
'misclass': 0.15000000000000002,
'runtime': 0.0007402896881103516}
We can also retrieve individual trial predictions as such:
In [11]:
model.predict(test_set.X)
Out[11]:
array([1, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 1, 0, 1, 0, 0, 1, 1, 1, 0])
If you want to try cross-subject decoding, changing the loading code to
the following will perform cross-subject decoding on imagined left vs
right hand closing, with 50 training and 5 validation subjects (Warning,
might be very slow if you are on CPU):
In [ ]:
import mne
import numpy as np
from mne.io import concatenate_raws
from braindecode.datautil.signal_target import SignalAndTarget
# First 50 subjects as train
physionet_paths = [ mne.datasets.eegbci.load_data(sub_id,[4,8,12,]) for sub_id in range(1,51)]
physionet_paths = np.concatenate(physionet_paths)
parts = [mne.io.read_raw_edf(path, preload=True,stim_channel='auto')
for path in physionet_paths]
raw = concatenate_raws(parts)
picks = mne.pick_types(raw.info, meg=False, eeg=True, stim=False, eog=False,
exclude='bads')
events = mne.find_events(raw, shortest_event=0, stim_channel='STI 014')
# Read epochs (train will be done only between 1 and 2s)
# Testing will be done with a running classifier
epoched = mne.Epochs(raw, events, dict(hands=2, feet=3), tmin=1, tmax=4.1, proj=False, picks=picks,
baseline=None, preload=True)
# 51-55 as validation subjects
physionet_paths_valid = [mne.datasets.eegbci.load_data(sub_id,[4,8,12,]) for sub_id in range(51,56)]
physionet_paths_valid = np.concatenate(physionet_paths_valid)
parts_valid = [mne.io.read_raw_edf(path, preload=True,stim_channel='auto')
for path in physionet_paths_valid]
raw_valid = concatenate_raws(parts_valid)
picks_valid = mne.pick_types(raw_valid.info, meg=False, eeg=True, stim=False, eog=False,
exclude='bads')
events_valid = mne.find_events(raw_valid, shortest_event=0, stim_channel='STI 014')
# Read epochs (train will be done only between 1 and 2s)
# Testing will be done with a running classifier
epoched_valid = mne.Epochs(raw_valid, events_valid, dict(hands=2, feet=3), tmin=1, tmax=4.1, proj=False, picks=picks_valid,
baseline=None, preload=True)
train_X = (epoched.get_data() * 1e6).astype(np.float32)
train_y = (epoched.events[:,2] - 2).astype(np.int64) #2,3 -> 0,1
valid_X = (epoched_valid.get_data() * 1e6).astype(np.float32)
valid_y = (epoched_valid.events[:,2] - 2).astype(np.int64) #2,3 -> 0,1
train_set = SignalAndTarget(train_X, y=train_y)
valid_set = SignalAndTarget(valid_X, y=valid_y)
Dataset references¶
This dataset was created and contributed to PhysioNet by the developers of the BCI2000 instrumentation system, which they used in making these recordings. The system is described in:
Schalk, G., McFarland, D.J., Hinterberger, T., Birbaumer, N., Wolpaw, J.R. (2004) BCI2000: A General-Purpose Brain-Computer Interface (BCI) System. IEEE TBME 51(6):1034-1043.
PhysioBank is a large and growing archive of well-characterized digital recordings of physiologic signals and related data for use by the biomedical research community and further described in:
Goldberger AL, Amaral LAN, Glass L, Hausdorff JM, Ivanov PCh, Mark RG, Mietus JE, Moody GB, Peng C-K, Stanley HE. (2000) PhysioBank, PhysioToolkit, and PhysioNet: Components of a New Research Resource for Complex Physiologic Signals. Circulation 101(23):e215-e220.