The final project follows the same format as the midterm project. You are expected to use the knowledge and skills that you acquired over the last few weeks to replicate (part of) an existing paper or to model your own data. Below we provide a list of papers and data sets that you may use for your project. You are of course free to select another paper or use a different data set, as long as your project implements a neural network. Projects may be implemented in small groups.
We hope the information provided on this page serves as inspiration for your projects. If you are still deciding on the contents of your project, we recommend choosing one of the following two options: (1) build a convolutional or recurrent neural network and test how targeted changes in input, architecture, cost function, or optimization algorithm change the network's performance, or (2) use a pretrained neural network to explain neural data. Please see below for more information on how to implement these options.
Below we list papers that you can use for your project. The first set of papers has a machine learning focus, i.e. the goal is to improve a network's performance on a certain task. Improvements are for example achieved by changing the network's architecture or optimization algorithm. We limit papers here to the domain of supervised learning, but you are certainly welcome to explore unsupervised or reinforcement learning in your project. If you would like to go beyond supervised learning, this blog may provide a good starting point. The second set of papers examines how well neural networks, usually trained on a particular task, can explain neural and behavioural data. These papers fall within the emerging field of cognitive computational neuroscience.
Machine learning Simonyan and Zisserman (2015) Very deep convolutional neural networks for large-scale image recognition. ICLR. He et al. (2015) Deep residual learning for image recognition. arXiv. Kingma and Ba (2014) Adam: A method for stochastic optimization. arXiv.
Cognitive computational neuroscience Khaligh-Razavi and Kriegeskorte (2014) Deep supervised, but not unsupervised, models may explain IT cortical representation. PLoS Comp Biol. Geirhos et al. (2017) Comparing deep neural networks agains humans: object recognition when the signal gets weaker. arXiv. Spoerer et al. (2017) Recurrent convolutional neural networks: A better model of biological object recognition. Front Psychol.
Option 1: Experiment with your own network There are many data sets available for training your neural network. For object recognition problems, data sets include MNIST, CIFAR-10, ImageNet, and MS COCO. Many of these datasets can be downloaded using built-in PyTorch functions. MNIST and CIFAR-10 are relatively small data sets that pose relatively simple categorization problems. This means that the input-output function can be learned with relatively shallow networks, e.g. 2-3 convolutional layers. MNIST and CIFAR-10 are therefore good choices, especially if you do not have huge amounts of computing power at your disposal. If you have a Google account, you may also consider using Google Colab, which provides access to GPU computing.
Option 2: Use a pretrained network to explain neural data You may also use pretrained neural networks to explain data from biological systems. The most commonly available pretrained networks are convolutional neural networks (CNNs). The pretrained CNNs available in PyTorch have usually been trained on ImageNet, and are thus well suited for explaining brain data acquired in animals or humans who were viewing images. We provide you here with one such data set.
The data set consists of functional magnetic resonance imaging (fMRI) data reported in Khaligh-Razavi and Kriegeskorte (2014). fMRI is a noninvasive method that provides an indirect measure of neural activity. The data were acquired from four human partipants while they were viewing a set of 96 object images. To replicate some of the findings reported in the paper, you need to feed these images into the pretrained networks, assess how they internally represent the images, and then compare the internal representations to the brain representations. For more information on how to compute the network's internal representations, see Nili et al. (2014). The fMRI data provided here has already been processed and converted into the representational format (so no need to change anything there). Below we provide some hints on how to load in the data and compute representations.
# load in the fMRI data (already in representational format) # import libraries import numpy as np import matplotlib.pyplot as plt from scipy.io import loadmat # load in data cd data_path # replace by local path data = loadmat('brainData.mat') # data from early visual cortex (EVC = V1 & V2) EVC = data['EVC'] EVC.shape # images x images x participants # data from inferior temporal cortex (IT) IT = data['IT'] IT.shape # images x images x participants # display the data fig, axs = plt.subplots(1, 5) axes = axs.ravel() mat = axs.imshow(np.average(IT,2), cmap=plt.winter()) axs.set_title('subject average', fontweight='bold') axs.set_xlabel('images') axs.set_ylabel('images') #fig.colorbar(mat, ax=axes) # uncomment if you want to see the colorbar for i in range(4): i += 1 axs[i].imshow(IT[:,:,i-1]) axs[i].set_title('subject ' + str(i)) axs[i].set_xticklabels() axs[i].set_yticklabels()
Once you have extracted the image activations from the pretrained network (see PyTorch documentation), you can compute RDMs for each layer of interest by using the following code:
# compute RDMs # import libraries import scipy.spatial.distance as dist from numpy import random import matplotlib.pyplot as plt # simulate image activations nimages = 10 nunits = 100 # nr of model units in the layer patterns = random.standard_normal( (nimages, nunits) ) # replace with the activations extracted from your network dissims = dist.pdist(patterns) rdm = dist.squareform(dissims) # show simulated RDM plt.figure(1) plt.imshow(rdm)
Many models are available via built-in PyTorch functions. If you are interested in modeling the visual system, the Brain-Score website is a good starting point for exploring available neural networks.
Project presentations will take place on Monday April 5th and Thursday April 8th, during normal class hours. Groups with even numbers are expected to present on the Monday, groups with odd numbers on the Thursday. You are expected to be present during both days.