Missions

The general objective of this research position is to design and implement computer vision attention models adapted to event data.

Activities

The biological retina has inspired the development of a new kind of camera: event- based sensors asynchronously measure per-pixel brightness changes and output a stream of events that encode the time, location, and sign of the brightness changes (positive or negative). In addition to eliminating redundancy, they benefit from several advantages over conventional frame cameras, from which they fundamentally differ. Event sensors are inspired from the human eye, that is primarily sensitive to changes in the luminance falling on its individual sensors. These changes are processed by layers of neurons in the retina through to the retinal ganglion cells that generate action potentials, or spikes, whenever a significant change is detected. Then these spikes propagate through the optic nerve to the brain. Cognitive attention mechanisms, inspired by the human brain's ability to selectively focus on relevant information, can offer significant benefits in embedded computer vision systems. The human eye has a small high-resolution region (the fovea) in the center of the field of vision, and a much larger peripheral vision, which has much lower resolution, combined with an increased sensitivity to movement. Therefore, limited resources are deployed to extract the most salient information from the scene without wasting energy capturing the entire scene at the highest resolution. This foveation mechanism has inspired the recent development of a variable-resolution event sensor. This sensor has an electronic control of the resolution in selected regions of interest, allowing to focus downstream computational resources on specific areas of the image that convey the most useful information. This sensor even goes beyond biology by allowing multiple regions of interest.

A first step in this project will consist in studying state-of-the-art attentional mechanisms in deep networks and their link with cognitive attention as implemented in the brain. Cognitive attention refers to the selective processing of sensory information by the brain based on its relevance and importance to the current task or goal. It involves the ability to focus one's attention on specific aspects of the environment while filtering out irrelevant or distracting information. In particular, the study will distinguish between both top-down and bottom-up attention. The second step will be the design an attention architecture that will allow selectively focusing on relevant regions while ignoring irrelevant part, which will depend on the target task (e.g., segmentation, object tracking, obstacle avoidance, etc.). The model will be based either on standard deep networks, or on spiking neural networks, based on previous work [GIT]. Spiking Neural Networks are a special class of artificial neural networks, where neurons communicate by sequences of asynchronous spikes. Therefore, they are a natural match for event-based cameras due to their asynchronous operation principle. This selection of regions will result in less data usage and smaller models (frugal system). In the third step, we will evaluate the impact of the attention mechanism on the general performance of the computer vision system. The target metrics will obviously depend on the selected task, and will include accuracy, MIOU, complexity, training time, inference time, etc. of the system.

The recruited researcher is also expected to participate in activities related to the NAMED project (active organisation of and participation in project meetings, participation in the drafting of deliverables), in collaboration with local and remote project members.

Skills

Programming skills in Python/C++ are expected, as well as an interest in research, machine learning, bio-inspiration, electronics, and neuroscience.

Work Context

This research position takes place in the context of an international collaborative project co-funded by the French ANR and the Swiss NSF. The project NAMED (Neuromorphic Attention Models for Event Data) that started on February 1st, 2024, in collaboration with SCALab in Lille and ETH Zürich. The field of embedded computer vision has become increasingly important in recent years as the demand for low-latency and energy-efficient vision systems has grown. One of the key challenges in this area is developing intelligent vision systems that can efficiently process large amounts of visual data while still maintaining high accuracy and reliability.

Constraints and risks

Desired experience: Less than 2 years.
The work will be carried out as part of an ANR project in partnership with the University of Lille and ETZ Zürich. This will involve working as part of a team and travelling to project meetings and conferences in France, Switzerland and internationally.