Coordinate regression for event-based vision

Why spiking neural networks?

Spiking neural networks (SNNs) and neuromorphic hardware are inherently parallel, asynchronous, low-energy devices that promise accelerated compute capacities for event-based machine learning systems. This work addresses coordinate regression using an inherently asynchronous and parallel spiking architecture that is both compatible with neuromorphic hardware, and, we hope, a step towards more useful neuromorphic algorithms.

The task: event-based dataset

We construted a dataset of geometric shapes (circles, triangles, squares) with center coordinate labels. To provide realistic, structured noise, we superimposed the sparse shapes on office-like scene renderings from the NeuroRobotics Platform (NRP). The shapes are sparsely sampled from a Bernouilli distribution (p=0.8) and are moving around with a brownian motion. The task is for any network to recognize the shapes and predict their center as accurately as possible.

We chose the geometric shapes as a means to control parameters such as shape velocity, event density, and shape complexity. For instance, the current shapes have a radius of 80 pixels, which is too large for any single kernel to learn. The convolutional layer is, therefore, forced to specialize on partial features to correctly solve the task.

In total, the dataset contains 2000 videos of 60 frames each (resembling 1ms of events) with a resolution of 640x480.

Neural network architecture

We use a classical convolutional model consisting of two convolutional layers, followed by an inverse convolution for a slight upsampling. We implemented a differentiable coordinate transform (DCT) layer to transform the pixel-wise activations into a 2-dimensional coordinate. Each convolution is interspersed with non-linear activations, batch normalization and dropout (p=0.1).

Translation-invariant receptive fields

Similar to conventional convolutional systems, we can define receptive field kernels for spiking neural networks. Importantly, we wish to retain translation-invariant properties to capture the moving shapes over time, as neatly illustrates in the work by [Lindeberg 2021].

Preconfiguring the receptive field kernels for the spiking neural networks significantly reduces training time, memory consumption, and ability for the network to generalize, as seen in the loss curves above.

Training and validation

We constructed and trained four independent networks over the same architecture

1. a non-spiking network where ReLU units constitute the nonlinearities (ANN)
2. a non-spiking network with custom receptive field kernels (ANN-RF)
3. a spiking version, with three leaky integrate-and-fire (LIF) nonlinearities feeding a final, non-spiking leaky integrator (SNN)
4. a spiking version, resembling (3), but where custom receptive field kernels promote translation-invariant feature recognition (SNN-RF)

The networks were trained with backpropagation-through-time using a regular l²-loss via the differentiable coordinate transform (DCT) method (presented futher below). The models are tested on unseen validation data (20% of the total training data).

Prediction errors and performance

The receptive field version of the SNN outperforms even the artificial neural network. If we further explore the prediction errors of the models (sampled over the entire validation dataset), the performance benefit becomes clearer: the predicted coordinates from the receptive field model is significantly closer to the actual, labelled coordinates.

Future work

This is still work in progress and more work is needed to generalize the results. However, there are already a few extensions that would immediately be interesting to explore

1. The gaussian structure of the output predictions can be exploited to further increase prediction accuracy.
2. The current shapes are quite dense (Bernouilli p=0.8), such that the artificial networks are able to converge to the shapes in the given frames. We wish to explore the sparseness of the shapes (lower the Bernouilli distribution of the shapes) while exploring the temporal process of recurrent artificial (non-spiking) networks.
3. We currently focus on translation-invariance. We wish to extend our method to more complex shapes that require both scale- and rotation-invariance.