menu

Simulating Event-Driven Address-Event-Representation (AER) Systems With MegaSim

The Modular Event-driven Growing Asynchronous Simulator (MegaSim) is a tool designed for simulating Address-event-representation (AER) systems. In MegaSim the user may define a topology by utilizing modules that process input events, nodes to connect modules with each other and input stimuli sources for example data recorded from a DVS retina, or generated artificially by some algorithm. Modules can have an arbitrary number of input/output ports and can be either population of neurons or any algorithm described in C. Modules communicate through nodes by utilizing the AER events. In MegaSim it is assumed that events are communicated from an output port to an input port through asynchronous communication using Rqst and Ack signals, where there would be a delay between Rqst and Ack (a handshake-less link can be emulated by simply defining a zero delay for Ack).

In this workgroup we will demonstrate how to use and extend MegaSim.

Login to become a member send

Timetable

Day Time Location
Mon, 02.05.2016 19:00 - 20:00 Main lecture room

1. The MegaSim Simulator

Downloading and building MegaSim

You may download the source code from here: https://bitbucket.org/bernabelinares/megasim (~4.3 MB), copy it from my USB flash drive or from the local server Icon megasim_local (file missing!)

Additionally, you can download the jAER [1] software for playing the generated events.

Building the simulator

There are two makefiles to compile MegaSim, one for macs (Makefile_osx) and one for linux (Makefile_linux):

make –f Makefile_osx

make –f Makefile_linux

then source source.sh to add megasim and the python scripts to your path.

The python scripts folder includes a collection of scripts for generating STDP functions, plot the evolution of kernels interactively and convert MegaSim’s AER events to jAER data and vice versa. The prerequisites for the python scripts are:

Python version = 2.7

Numpy version >= 1.10.4

Matplotlib version >= 1.5.0

Simulator overview

The Modular Event-driven Growing Asynchronous Simulator (MegaSim) is a tool designed for simulating Address-event-representation (AER) systems written in C. In MegaSim the user may define a topology by utilizing modules that process input events, nodes to connect modules with each other and input stimuli sources, for example data recorded from a DVS retina, cochlea, or generated artificially by some algorithm. Modules may have an arbitrary number of input/output ports and can either be population of neurons, algorithms described in C, or a combination of both. Modules communicate through nodes by utilizing the AER events. In MegaSim it is assumed that events are communicated from an output port to an input port through asynchronous communication using request (Rqst) and acknowledgment (Ack) signals, where there would be a delay between Rqst and Ack. A handshake-less link can be emulated by simply defining a zero delay for Ack.

 

megafig.pngFigure 1. An example topology in MegaSim

 

It is possible to model clocked networks by defining a clock module that periodically sends update events or model hybrid systems that include clocked updates with asynchronous handshaking.

Upon the completion of a simulation there will be an event file for each node describing all the events that have been communicated through this node.  The following section describes the structure of AER events in MegaSim

AER representation in MegaSim

Table 1 shows how AER events are represented in MegaSim. For each event there are 3 timing values, and a number of event parameters typically in the form of X, Y and polarity. However these parameters can be any signed 32-bit integer and the user decides how a module interprets and processes them.

Table 1. The structure of AER events in MegaSim.
pre-Rqst Rqst Ack x y Polarity
0 20 40 4 5 -1
17 60 80 5 4 1
... ... ... ... ... ...
60 -1 -1 4 5 1

Where pre-Rqst represents the time an event is created inside a module, Rqst the time when this event is actually put on the node and Ack the time when acknowledgment signal is communicated. When an event has been queued in link, Rqst and Ack will be set to -1.

Handling and processing AER events

The simulator is responsible for looking at all nodes and selecting the earliest unprocessed events. Then calls the corresponding destination modules, and each module will process the incoming event, perform handshaking, perform computations at the destination module such as a) update all internal states since last time called, b) compute the effect of an incoming event, and c) if output events are generated they will be written on the output nodes with the present timestamp and then sorted according the their timestamps. This process is iterated until a maximum predefined simulation time, or until there are no more unprocessed events.  Upon the completion of a simulation there will be an event file for each node describing all the events that have been communicated through this node. 

MegaSim Modules

Currently MegaSim has the following modules:

  1. Module copy (module_copy): Which takes input events from a number of input ports and after some delay copies them on all output ports. Each event can be copied a number of times on each output port. This module can be used to simulate splitters, mergers or axonal delays.
  2. Module Convolution (module_conv): This module receives input events through a number of input ports, and for each input port applies a static convolution kernel. The generated output events can be sent to several output ports simultaneously. 
  3. Module Convolution Plastic (module_conv_plastic): This module is identical to module_conv however it includes an extra port for each plastic kernel to receive special AER DW events from the STDP modules and update the weights appropriately. 
  4. Module STDP (module_STDP): This module has two input ports one for the pre and one for the post events. It also has two buffers with user-defined size to store the pre and post events and uses them to compute the DWs based on a user defined STDP function. 
  5. Module order-based STDP (module_STDP_Orderbased): This module also includes two input ports for the pre and post events but has only one buffer that stores the pre events and computes STDP based on the order of events and not their precise timing. This module is based on the work of Roclin et al. 2013 [2].

Module Convolution

The module_conv utilises an event-driven LIF neuron with linear leakage, described in detail in [3]. The state of this neuron is updated based on the current and previous input event. It uses a signed neuron capable of generating spikes with positive and negative polarity by utilising a positive and negative threshold. By disabling the negative threshold, however, it becomes a standard leaky integrate and fire (LIF) neuron model with linear leakage, similarly to [2].

Table 2. Module_conv parameter file

.integers

n_in_ports N

n_out_ports O

delay_to_process 1

delay_to_ack 2

fifo_depth 1

n_repeat 1

delay_to_repeat 15

Nx_array 31

Ny_array 31

Xmin 0

Ymin 0

THplus 190454428

THplusInfo 1

THminus -2147483647

THminusInfo 0

Reset_to_reminder 0

MembReset 0

TLplus 2000

TLminus 2000

Tmin 0

T_Refract 10

Nx_kernel Nx

Ny_kernel Ny

Dx Ndx

Dy Ndy

<Kernel weights for input port N>

crop_xmin 0

crop_xmax 31

crop_ymin 0

crop_ymax 31

xshift_pre 0

yshift_pre 0

x_subsmp 1

y_subsmp 1

xshift_pos 0

yshift_pos 0

rectify 0

.floats

 

where n_in_ports and n_out_ports defines the number of input and output ports, delay_to_process refers to the time required to process an incoming event, delay_to_ack delay to acknowledge an incoming event, fifo_depth the FIFO size for incoming events, Nx_array and Ny_array define the width and height of the feature map, Xmin and Ymin define the starting index of the array. THplus is the value of the positive threshold, THplusInfo is the enable/disable flag for the positive threshold, THminus is the negative threshold, THminusInfo is the enable/disable flag for the negative threshold, MembReset is the membrane reset voltage, TLplus and TLminus define the positive and negative linear leakage rate, Tmin defines the minimum time between 2 consecutive spikes, T_Refract is the refractory period.

In addition, for every input port the user must define the following parameters: Nx_kernel and Ny_kernel, which is the width and height of the kernel, Dx and Dy which is the displacement in X and Y of the kernel, and finally the weights of the kernel themselves. The output of the convolution module can be cropped, shifted and subsampled by using the crop_xmincrop_xmaxcrop_ymincrop_ymaxxshift_preyshift_prex_subsmpy_subsmpxshift_posyshift_pos parameters. Finally, the rectify parameter will force the output events to have a positive polarity.

Module STDP

For this module two buffers are used, one for the pre and one for the post events. The user provides a precalculated STDP function that consists of ΔT and ΔW data and linear interpolation is used during the simulation to determine the intermediate values. Figure X shows two example STDP functions, one of which is a piecewise double exponential and a simplified rectangular one. 

figure1.pngFigure 2. Two example STDP functions inclue a piecewise double-exponential and a rectangular one, for the module_stdp.

 

Table 3. An example parameter file of the STDP module with a piecewise double exponential STDP function

.integers

n_in_ports 2

n_out_ports 1

delay_to_process 0

delay_to_ack 0

fifo_depth 0

n_repeat 1

delay_to_repeat 15

Nx_array 31

Ny_array 31

Xmin 0

Ymin 0

Nx_kernel 7

Ny_kernel 7

Dx -3

Dy -3

event_buffer_size 100

look_past 100

old_events_to_keep 50

stdp_function_data_points 20

-100 -88 -77 -66 -55 -44 -33 -22 -11 0 0 11 22 33 44 55 66 77 88

100 6 11 20 35 62 108 188 329 573 999 -800 -459 -263 -151 -86 -49 -28 -16 -9 -5.

.floats

 

where n_in_ports are the number of input ports, with the first port used for presynaptic events and the second port used for postsynaptic events, n_out_ports is set to 1 for the special DW events, Nx_array and Ny_array are the width and height of the plastic feature map, Nx_kernel  and Ny_kernel is the width and height of the plastic kernel, Dx and Dy are the displacement in X and Y of the kernel, event_buffer_size defines the pre and post event buffer sizes, look_past parameter describes how many past events in the buffer to consider when applying STDP, old_events_to_keep defines how many old events to keep when the buffers are full, stdp_function_data_points define the number of data points of the STDP function, and the following row defines the DT (pre-post) data points, and the next row the DW data points.

Module Order-based STDP

The module_stdp_orderbased is based on the work or Roclin et al. [2], with a slight modification to account for the shared weights of convolutional networks. In order-based STDP a single buffer is used to store presynaptic events and whenever there is a postsynaptic event only the weights relevant to the events in the buffer are potentiated, while the remaining ones are depressed. In this type of STDP what is important is not the timing of events but the order they appear in the buffer, as seen in Figure . Results have shown that the size of the buffer affects to be roughly the size of the object to be recognised [2].  A special AER event is generated with the X and Y coordinates of the weights of the kernel to be updated and the DW+/- value.

figure2.pngFigure 3. The STDP function of the module_stdp_orderbased.

 

Table 4. An example parameter file for the order-based STDP

.integers

n_in_ports 2

n_out_ports 1

delay_to_process 0

delay_to_ack 0

fifo_depth 0

n_repeat 1

delay_to_repeat 15

Nx_array 31

Ny_array 31

Xmin 0

Ymin 0

Nx_kernel 7

Ny_kernel 7

Dx -3

Dy -3

event_buffer_size 24

look_past 24

old_events_to_keep 23

num_potentiate 15

stdp_function_data_points 2

-200 -24

1000 -100

.floats

 

Setting up a Simulation

This section describes how to describe a neural network topology in MegaSim. In this particular example a spiking convolutional layer will be used with 4 precomputed gabor-like kernels as shown in Figure 4.

convKernels.pngFigure 4. Gabor kernels

In MegaSim there a netlist file describing the topology of a network, which includes the input sources, modules, nodes and parameters . The netlist file of this particular example can be seen in Table 2.

Table 5. An example netlist file
1. .netlist
2. . source {input} i.txt
3. module_conv {input} {l1_1} gabor_01.prm gab_001.stt
4. module_conv {input} {l1_2} gabor_02.prm gab_002.stt
5. module_conv {input} {l1_3} gabor_03.prm gab_003.stt
6. module_conv {input} {l1_4} gabor_04.prm gab_004.stt
7.  
8. .options
9. Tmax = 429290101

The netlist file begins with the “.netlist” keyword, and the following lines include the MegaSim modules, along with their input/output nodes (in brackets {}) and their configuration and initial states files. In this example, in line 2 we declare an input source, with a node name {input} and an input file with the AER events “i.txt”. Line 3 declares a convolutional module receiving input from the {input} node and has an output node {l1_1} followed by the parameter file “gabor_01.prm” (as described in the MegaSim modules section) and the initial state of the module “gab_001.stt”. In line 8, the options of the simulations are set “.options”, where we can set the maximum simulation time Tmax.

For this example the input data were recorded from DVS sensor and can be seen with jAER by playing the i.dat file. As a first step we are going to convert the DVS data to MegaSim AER data using the following command:

jaer_to_ascii.py i.dat

then execute the simulation:

megasim people_walking_gabor.sch

Convert the output of a convolutional module to jAER data:

convert_to_jaer.py node_l1_1.evs

the output can be played back with jAER, Figure 5, by selecting the es.cnm.imse.jaer.chip.retina.RetinaTeresa2 from the AEChip dropdown menu.

convpeoplewalkjaer.png5. jAER output of the first convolutional module.

2. Results

2.1 Development of new MegaSim modules

We developed a number of new modules, for example a module that receives inputs from a number of spiking convolutional modules and flattens them into a vector in order to be fully connected to a population of LIF neurons. The second module is a population of fully connected (FC) LIF neurons with STDP, in order to have a spike-based classifier (module_fully_connected), and finally a merger module that receives the output activity of several populations, merges them, and sends UDP packets of their events to jAER [1]. 

newmodules.pngFigure 6. The module_flatten, module_FC, and module_merger.

We tested the new modules on a simple task in which an input stimulus, of size 16x16, is a bar rotating randomly and is fully connected to a population of 4 LIF neurons with plastic synapses. In order to force each output neuron to respond to a particular orientation we used a population of teacher neurons, as seen in Figure 7.

 

rotatingbars.png7. Rotating bars benchmark with STDP and a population of teacher neurons.

Figure 8 shows the receptive fields of the neurons after learning, and the video below show the evolution of the weights during the learning process as well as the activity of the network.

receptiveFields.png8. The receptive fields of the output neurons after training.

Missing flash plugin. Download here.

2.2 Porting off-line trained deep nets on MegaSim 

For this task we used the methodology described by O' Connor et al. [4] to train a deep belief network (DBN) off-line, with 7 hidden layers, and then map it to a spiking neural network and run it on MegaSim. Results are presented as classification accuracies between Brian, SpiNNaker, and Megasim (Table 6). 

RBM_and_arch_updated.pngArchitecture of (a) an RBM and (b) a Deep Belief Network (DBN) with 7 hidden layers.

 

Table 6. Test for 1500 spikes per MNIST digit.
  Classification accuracy (%)
Brian 96.21
SpiNNaker 96.22
MegaSim  

 

The spiking activity of the DBN on MegaSim can be visualised through jAER:

Missing flash plugin. Download here.

The activity of the input layer is shown on the bottom left corner. Each pixel of an MNIST digit is converted into spike-trains using Poisson rates, with the rate proportional to the intesity of each pixel. Then the responses of the 7 hidden layers can be seen and on the top is the actual output activity (10 neurons, for digits 0 to 9) and the ground truth (label). The bottom right corner shows the activity of the reconstruction layer.

References

[1] [Online]. Available: http://jaer.wiki.sourceforge.net

[2] D. Roclin, O. Bichler, C. Gamrat, S. J. Thorpe, and J.-O. Klein, “Design study of efficient digital order-based STDP neuron implementations for extracting temporal features,” 2013 Int. Jt. Conf. Neural Networks, pp. 1–7, Aug. 2013.

[3] J. a Perez-Carrasco, B. Zhao, C. Serrano, B. Acha, T. Serrano-Gotarredona, S. Chen, and B. Linares-Barranco, “Mapping from Frame-Driven to Frame-Free Event-Driven Vision Systems by Low-Rate Rate-Coding. Application to Feedforward ConvNets,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 35, no. 11, pp. 2706–2719, 2013.

[4] P. O’Connor, D. Neil, S.-C. Liu, T. Delbruck, and M. Pfeiffer, “Real-time classification and sensor fusion with a spiking deep belief network.,” Front. Neurosci., vol. 7, no. October, p. 178, Jan. 2013.

Leader

Evangelos Stromatias

Members

Alessandro AImar
Enrico Calabrese
Tobi Delbruck
Gabriel Andres Fonseca Guerra
Paco Gómez-Rodríguez
Giacomo Indiveri
Alejandro Linares-Barranco
Melika Payvand
Gary Pineda-Garcia
Alexander Rast
Fabio Stefanini
Evangelos Stromatias
Ricardo Tapiador
Bernhard Vogginger
Qi Xu
bernabe linares-barranco