
Extracting layers from video is very important for video
representation, analysis, compression, and synthesis. Assuming
that a scene can be approximately described by multiple planar
regions, this paper describes a robust and novel approach to
automatically extract a set of affine or projective
transformations induced by these regions, detect the occlusion
pixels over multiple consecutive frames, and segment the scene
into several motion layers. First, after determining a number of
seed regions using correspondences in two frames, we expand the
seed regions and reject the outliers employing the graph cuts
method integrated with level set representation. Next, these
initial regions are merged into several initial layers according
to the motion similarity. Third, an occlusion order constraint
on multiple frames is explored, which enforces that the
occlusion area increases with the temporal order in a short
period and effectively maintains segmentation consistency over
multiple consecutive frames. Then the correct layer segmentation
is obtained by using a graph cuts algorithm, and the occlusions
between the overlapping layers are explicitly determined.
Several experimental results are demonstrated to show that our
approach is effective and robust.
Related Publications:
Jiangjian Xiao and Mubarak Shah, “Motion
Layer Extraction in the Presence of Occlusion using Graph
Cut”, IEEE transactions on Pattern
Analysis and Machine Intelligence, Vol. 27, No. 10, pp.
16441659, 2005.
Jiangjian Xiao and Mubarak Shah,
“Motion Layer Extraction in the Presence of Occlusion
using Graph Cut", Oral presentation (6.5%),
IEEE Conference of Computer Vision and Pattern Recognition, June
27  July 2, Washington, DC. 2004.
Figure 1. Layer clustering by seed expanding
process. (a) An initial seed region in the first frame. (b) The
corresponding seed region in the second frame. (c) The warped
version of the second frame using the estimated affine
parameters. (d) The difference map between (a) and (c). (e) The
result after simple expansion and partitioning. (f) The result
after bipartitioning without the level set representation.
(gj) are the intermediate steps of bipartitioning with the
level set representation. (g) and (i) respectively are the
expansions of the seed region during the first and fourth
iterations using the level set representation. (h) and (j) are
the results obtained after the graph cuts partitioning, where
the new region can have an arbitrary compact contour. (k) and
(l) are 3D visualization of the level sets of (g) and (i). Note:
The red box is the initial seed region. The green contours are
obtained after using bipartitioning algorithm.
Video clip:
