A real-time background subtraction algorithm: Multi-model background maintenance based on adaptive frame rate control

Introduction
In computer vision applications, such as video surveillance, human motion analysis, human-machine interaction, and object based video encoding, most applications attempt to detect and recognize events or moving foreground objects in live video. The increased applications in used of video cameras for surveillance has been a significant development of moving foreground detection in security industry. The detected foreground can then be made available to alert the operator in the event that an illegal or abnormal activity is happened. Background subtraction is the very first step required towards a solution to foreground detection. Development of real-time background subtraction algorithm is desirable.

Algorithm
Multi-model background maintenance (MBM) is a real-time background subtraction algorithm for video surveillance application. The main idea is to maintain and learn a time-varying background image using the statistical information of the multi-model Gaussian distributions by principle feature. The MBM can not only handle complex dynamic scenes, such as illumination changes and non-static background, but adapt to sleeping person, especially the people walking forward and backward problem.
The block diagram of base line MBM algorithm is depicted in Fig. 1. It consists of five blocks. The functionalities of each block are summarized in the following.

1. Multiple background maintenance: estimate the background models within a period, then update and maintenance these models based on the on-line clustering method.
2. Background buffer: store the estimated background image.
3. Background difference: differential the foreground pixels from current image.
4. Labeling: grouping the connected foreground pixels into foreground object.
5. Tracking: tracking the foreground object over time. 

Adaptive frame rate
The maintain rate of MBM can be adapted by control the Multiple background maintenance block processing rate. A maintenance interval is defined as the amount of multiple background maintenance process been done per second. A simulation is given in Fig. 2, where the pixel rate is depicted against the maintain interval by three different format. The simulation is running on the un-optimized C code under AMD 1.6GHz CPU. Such that, one can determine the maintain rate based on the different application or different format. Table 1 shows the throughput of the MBM under various maintain rate with three formats under. 

Experimental result
The throughput of the MBM is shown on table 1 under non-interlace and non-duplicated frame surveillance camera. For the surveillance camera based on BT.601, the performance can achieve 410 frames/per second, 96 frames/per second and 30 frames/per second respectively due to the interlace effect. The experimental results of three sequences are given in following. All results are recorded as compressed video data for easily evaluation.

        PETS 2001 sequence experiment result
Background maintain result 

        坪林隧道口
Background maintain result 

        信義路與基隆路交叉口
Background maintain result