Real-time purchase behavior recognition system based on deep learning-based object detection and tracking for an unmanned product cabinet

摘要

We propose a system to recognize purchasing behavior by detecting and tracking products in real time using only camera sensors in an unmanned product cabinet. To detect and track products in real time, we focused on the simultaneous pre-processing of videos from multiple cameras for robust product detection. After synchronizing multiple videos, unnecessary frames with relatively little information are removed based on change detection. An object score is measured on a frame-by-frame basis to select the most significant frames. Next, the target products are detected and tracked in the selected frames. Finally, the purchasing behavior of the detected product is recognized based on the tracking information. These processes were used to design an end-to-end recognition framework. The contribution of this paper is significant in that by redesigning the existing deep neural networks a real-time integrated system for a practical application was successfully realized without any bottleneck from multi-camera inputs to final object recognition process. Furthermore, the proposed object detection network shows comparable performance with the state-of-the-art methods. We performed intensive experiments to evaluate pure object detection performance as well as to evaluate various purchase/return scenarios. For example, for a basic purchase/return scenario, the proposed system achieved about 92% or more accuracy, which can be the actual level of commercialization.