YuRo: Real-time 3D vision for robots

A novel hardware system was developed that integrates a pair of event-based vision sensors (DVS) and a pair of conventional RGB cameras to an FPGA. Simple sensor fusion is performed on the FPGA, streaming both event- and image-data for further processing. This system enables validation of algorithms combining fast event-based processing (e.g., real-time object tracking) with conventional image processing (e.g., image recognition and 3D vision). Conventional images benefit from larger datasets and pre-trained artificial neural networks, improving detection, recognition, and segmentation quality. However, detection speed remains a limitation for use in robot motion control loops.

Several algorithms were validated to enable real-time smart vision with 3D assessment for safe and efficient robot operation around humans:

a) Event-frames accumulated via the FPGA pipeline were made similar to grayscale images, allowing direct application of standard ANNs (YOLO, MediaPipe) trained on conventional images. Good results were obtained with high detection rates controlled through event-frame accumulation time.

b) Datasets were collected using calibrated and synchronized DVS-image sensor pairs, using conventional image detections as labels for DVS-data. These datasets enabled training of various neural networks (including spiking neural networks) with different event-based representations (e.g., time surfaces).

c) Different depth-perception algorithms were tested, including DVS-based stereo, RGB-based stereo, and time-of-flight sensors. Each method has distinct limitations (accuracy, range, surface properties, texture dependence). Safe robot operation requires complementary depth-perception streams.

Demonstration and Testing Multiple demonstrations showed system performance in closed motion control loops of robot arms, including obstacle avoidance and target acquisition. The technology was tested with elderly residents at a care facility (Haus Tabea) and at public events.