Video segmentation

Made by Chung-Yuan Lin

Introduction

To achieve the content-based functionality, images and video sequences need to be segment as semantic objects by segmentation algorithm. Part of this evolution is due to the need to support a large number of new multimedia applications, such as MPEG-4 and MPEG-7.

Three major applications:

1. Compress

2. Description

3. Manipulating

Introduction algorithm

We introduce the mainly functionality of our algorithm in this section.

l Frame difference: generate consecutive frame difference by following criterion

l Multiscale feature extraction: constructed with mathematic morphological to partitioning frame into regions. Multiscale feature extraction consist of three major steps, reconstruction, top-hat transform and labeling, The resulting image contains all possible bright features in the image and excluded following conditions:

1. Not visually discernible

2. Noise

3. Small size features

l Higher order test: to extraction moving pixels by examining the statistical distribution of Gaussian, following by a motion regularity to correct wrongly condition, which may not obey Gaussian distribution in reality.

l Hierarchical decision: hierarchical decision constructed by three different level to merge meaningful features or removed unwanted features. The three different levels are:

1. Low level feature level: filtering unwanted features both in higher order test and multiscale feature extraction generated features.

2. Semantic level: construct multiscale feature extraction generated features to forming semantic region by calculating and comparing feature vector.

3. Object level: considered higher order test extracted motion feature and semantic level generated feature to partitioning VOP in video sequence.

Algorithm overview

There are two mainly concepts to design video segmentation, one is quality and the other is computation load. In our design, higher order test can success extract very small moving, thus we can avoid some refinement step as post-process, hierarchical decision examine feature by different criterion in different level, it remove the refer previous VOP process to save much computation load and memory space.

First, five consecutive frame are loaded to frame difference and current frame is loaded to multiscale feature extraction. The frame difference generated consecutive four frame difference and examine by higher order test to extract motion features, concurrently, multiscale feature extraction partition frame into different region, finally, hierarchical decision considered two generated features, region and motion, to construct VOP.

Experiment result

Claire sequence

Container sequence

Mother and daughter sequence

Weather sequence