Introduction to Neural Networks

Final project


The final project follows the same format as the midterm project. You are expected to use the knowledge and skills that you acquired over the last few weeks to replicate (part of) an existing paper or to model your own data. Below we provide a list of papers and data sets that you may use for your project. You are of course free to select another paper or use a different data set, as long as your project implements a neural network. Projects may be implemented in small groups.

We hope the information provided on this page serves as inspiration for your projects. If you are still deciding on the contents of your project, we recommend choosing one of the following two options: (1) build a convolutional or recurrent neural network and test how targeted changes in input, architecture, cost function, or optimization algorithm change the network's performance, or (2) use a pretrained neural network to explain neural data. Please see below for more information on how to implement these options.


Papers

Below we list papers that you can use for your project. The first set of papers has a machine learning focus, i.e. the goal is to improve a network's performance on a certain task. Improvements are for example achieved by changing the network's architecture or optimization algorithm. We limit papers here to the domain of supervised learning, but you are certainly welcome to explore unsupervised or reinforcement learning in your project. If you would like to go beyond supervised learning, this blog may provide a good starting point. The second set of papers examines how well neural networks, usually trained on a particular task, can explain neural and behavioural data. These papers fall within the emerging field of cognitive computational neuroscience.

Machine learning
Simonyan and Zisserman (2015) Very deep convolutional neural networks for large-scale image recognition. ICLR.
He et al. (2015) Deep residual learning for image recognition. arXiv.
Kingma and Ba (2014) Adam: A method for stochastic optimization. arXiv.

Cognitive computational neuroscience
Khaligh-Razavi and Kriegeskorte (2014) Deep supervised, but not unsupervised, models may explain IT cortical representation. PLoS Comp Biol.
Geirhos et al. (2017) Comparing deep neural networks agains humans: object recognition when the signal gets weaker. arXiv.
Spoerer et al. (2017) Recurrent convolutional neural networks: A better model of biological object recognition. Front Psychol.


Data sets

Option 1: Experiment with your own network
There are many data sets available for training your neural network. For object recognition problems, data sets include MNIST, CIFAR-10, ImageNet, and MS COCO. Many of these datasets can be downloaded using built-in PyTorch functions. MNIST and CIFAR-10 are relatively small data sets that pose relatively simple categorization problems. This means that the input-output function can be learned with relatively shallow networks, e.g. 2-3 convolutional layers. MNIST and CIFAR-10 are therefore good choices, especially if you do not have huge amounts of computing power at your disposal. If you have a Google account, you may also consider using Google Colab, which provides access to GPU computing.

Option 2: Use a pretrained network to explain neural data
You may also use pretrained neural networks to explain data from biological systems. The most commonly available pretrained networks are convolutional neural networks (CNNs). The pretrained CNNs available in PyTorch have usually been trained on ImageNet, and are thus well suited for explaining brain data acquired in animals or humans who were viewing images. We provide you here with one such data set.

The data set consists of functional magnetic resonance imaging (fMRI) data reported in Khaligh-Razavi and Kriegeskorte (2014). fMRI is a noninvasive method that provides an indirect measure of neural activity. The data were acquired from four human partipants while they were viewing a set of 96 object images. To replicate some of the findings reported in the paper, you need to feed these images into the pretrained networks, assess how they internally represent the images, and then compare the internal representations to the brain representations. For more information on how to compute the network's internal representations, see Nili et al. (2014). The fMRI data provided here has already been processed and converted into the representational format (so no need to change anything there). Below we provide some hints on how to load in the data and compute representations.

# load in the fMRI data (already in representational format)
# import libraries
import numpy as np
import matplotlib.pyplot as plt
from scipy.io import loadmat

# load in data
cd data_path # replace by local path
data = loadmat('brainData.mat') 

# data from early visual cortex (EVC = V1 & V2)
EVC = data['EVC']
EVC.shape # images x images x participants

# data from inferior temporal cortex (IT)
IT = data['IT']
IT.shape # images x images x participants

# display the data
fig, axs = plt.subplots(1, 5)
axes = axs.ravel()
mat = axs[0].imshow(np.average(IT,2), cmap=plt.winter())
axs[0].set_title('subject average', fontweight='bold')
axs[0].set_xlabel('images')
axs[0].set_ylabel('images')
#fig.colorbar(mat, ax=axes[0]) # uncomment if you want to see the colorbar
for i in range(4):
    i += 1
    axs[i].imshow(IT[:,:,i-1])
    axs[i].set_title('subject ' + str(i))
    axs[i].set_xticklabels([])
    axs[i].set_yticklabels([])

Expected output:

Fig 1 This figure shows the fMRI data for inferior temporal cortex (IT, top row; you should get this as output if you run the code above) and early visual cortex (EVC, bottom row). The data consist of representational dissimilarity matrices (RDMs). Each entry in the RDM reflects the dissimilarity between response patterns elicited by two images. The images are ordered on the axes of the RDM according to their filenames (e.g. the first image is image01_25210000.bmp). The dissimilarities are colour coded; blue means that two images elicited similar response patterns across a region of interest (IT or EVC), and green means that two images elicited different response patterns. The RDM characterizes how the region represents the images, i.e. it characterizes what image information is emphasized and what image information is de-emphasized. We can also compute such representations for each layer in a neural network and then compare the representations between brain and models. For more information on representational similarity analysis, see Kriegeskorte et al. (2008).

Once you have extracted the image activations from the pretrained network (see PyTorch documentation), you can compute RDMs for each layer of interest by using the following code:

# compute RDMs
# import libraries
import scipy.spatial.distance as dist
from numpy import random
import matplotlib.pyplot as plt

# simulate image activations
nimages = 10 
nunits = 100 # nr of model units in the layer
patterns = random.standard_normal( (nimages, nunits) ) # replace with the activations extracted from your network 
dissims = dist.pdist(patterns)
rdm = dist.squareform(dissims)

# show simulated RDM
plt.figure(1)
plt.imshow(rdm)

Models

Many models are available via built-in PyTorch functions. If you are interested in modeling the visual system, the Brain-Score website is a good starting point for exploring available neural networks.


Project presentation

Project presentations will take place on Monday April 5th and Thursday April 8th, during normal class hours. Groups with even numbers are expected to present on the Monday, groups with odd numbers on the Thursday. You are expected to be present during both days.