Introduction to Neural Networks

Problem Set 8


Problem 1: Visualize the HMAX model

Use the HMAX output saved during the programming lab to provide the missing information in the figure below. The supplementary material accompanying Serre et al. (2007) may also be helpful. The lines of code below give some hints to get you started.



Fig 1 Detailed schematic of the first four layers of HMAX. Missing information is indicated in red.

# load and examine the HMAX output
import pickle
import numpy as np

cd hmax_path # replace by local path 
output = pickle.load( open('output.pkl', 'rb'))

s1 = output['s1']
n_sscales = len(s1) # s = spatial
s1_sscale1 = s1[0]
s1_sscale1.shape

s2 = output['s2']
n_fscales = len(s2) # f = filter
s2_fscale1 = s2[0]
n_sscales = len(s2_fscale1)
s2_fscale1_sscale1 = s2_fscale1[0]
s2_fscale1_sscale1.shape

# save answers
# insert x, y, z, m, n, s (integers), and layer name, operation A, operation B (strings) ...
#    ... as values in the dictionary below and save the dictionary as a text file using the code below
dict = {'x':  , 'y':  , 'z':  , 'm':  , 'n':  , 's':  , 'layer name':  , 'operation A':  , 'operation B':  } 
f = open('ps8_prob1.txt', 'w')
f.writelines( [str(dict['x'])+'\n', str(dict['y'])+'\n', str(dict['z'])+'\n', str(dict['m'])+'\n', str(dict['n'])+'\n', str(dict['s'])+'\n', dict['layer name']+'\n', dict['operation A']+'\n', dict['operation B']] )
f.close()
# content of solution_code_ps8.py
def prob1_answers(i):
    f = open('ps8_prob1.txt', 'r')
    text = f.readlines()
    f.close() 
    
    return text[i]

Output: Upload the following two files to codepost: ps8_prob1.txt and solution_code_ps8.py (content as in code block above).


Problem 2: Visualize HMAX output

Visualize how the different HMAX layers respond to the image of the snake (image27_40820000_rs.jpg). Use the code provided under Problem 1 to load the pickle file and extract responses. Generate figures that show the response to the image in layers S1, C1, and S2. The response can be interpreted as the network's internal representation of the image at different stages of processing.

You only have to visualize the response for the following spatial scales and filter types. For S1, visualize the response to the image for filter type 2 at spatial scales 7 and 8. For C1, visualize the response to the image for filter type 2 at spatial scale 4. For S2, visualize the response to the image for filter type 111 at filter scale 4 and spatial scale 4.

Also show the filters ('filter types' in Fig 1) that were convolved with the image (S1) or with the responses of the previous layer (S2) to compute the responses you are showing. The code below gives some hints on how to access the filters.

# show filters
import matplotlib.pyplot as plt
from scipy.io import loadmat

## S1 ##
# first execute lines 52 - 153 of hmax.py (def gabor_filter and class S1)
# then select a filter size and build S1 filters, for example:
filters = S1(size=19, wavelength=3.7) # spatial scale 7
filters = S1(size=21, wavelength=3.65) # spatial scale 8

# show one of the four filters
filter = filters.gabor.weight.data[0,0,:,:]  
plt.figure(); plt.imshow(filter, cmap=plt.gray()); plt.title('filter')

## S2 ##
# load patches (= filters)
cd hmax_path # replace by local path
m = loadmat('universal_patch_set.mat') 
patches = [patch.reshape(shape[[2, 1, 0, 3]]).transpose(3, 0, 2, 1)
           for patch, shape in zip(m['patches'][0], m['patchSizes'].T)]
n_fscales = len(patches)
patches_fscale1 = patches[0]
patches_fscale1.shape

# show one of the 400 filters (filters are 3D instead of 2D)
patch = patches_fscale1[100,:,:,:]
fig, axs = plt.subplots(1, 4)
for dim3i in range(0,4):
    axs[dim3i].imshow(patch[dim3i,:,:], cmap=plt.gray())
    axs[dim3i].set_xlabel('component ' + str(dim3i+1))
    if dim3i == 0:
        axs[dim3i].set_title('s2 filter')

Output: Upload the following file to codepost: ps8_prob2.pdf. The file should contain figures showing the responses of the network to image27_40820000_rs.jpg. Show the responses in S1, C1, and S2 on the first page (4 figures) and the filters used in generating these responses on the second page (3 figures, one with subplots). You can optionally also upload the code that you used to solve both problems (ps8.py).