SPatiotemporal REgularity Flow (SPREF)
Features
are important in image and video analysis. In this project, we propose a new
spatiotemporal feature that represents the directions in which a video or an
image is regular, i.e., the pixel appearances change the least. The directions
of regularity of a video are determined by its spatiotemporal properties. These
properties depend on the motion contents of the video, and the spatial
structure of the scene. We propose to model these directions with a 3D vector
field, which we refer to as the SPatiotemporal REgularity Flow (SPREF). The
SPREF can be estimated by minimizing an energy function formulated according to
its definition.
(Top)
A mini clip from the Alex sequence. (Bottom) The directions of regularity
obtained by (left) block motion and (right) SPREF. Notice that when the
sequence is sliced along the directions of regularity, the resulting surface is
smoother along the flow curves when SPREF is used.
Applications
Video Inpainting
When
an object is removed from a video, it leaves a spatiotemporal hole behind.
Video inpainting is filling this hole naturally, while preserving the video's
temporal regularity.
(Click the image to download video.)
The
spatiotemporal hole in the walking human sequence is inpainted by using a
method based on SPREF. Once the SPREF is computed, the missing parts of the video
can be easily recovered by interpolation. In addition, the basic unit of
inpaiting is subgof with unique SPREF, which facilitates the integrity of the
video.
Video Compression
According
to information theory, lower entropy results in higher compression ratio. Thus,
if a spatiotemporal region is filtered along the directions of regularity,
where entropy is lower, better compression can be obtained. Since SPREF
indicates the directions of regularity, it is a very suitable tool to increase
the efficiency of the compression. Moreover, its compactness due to the spline
representation has a low compression overhead.
We
show the results of the bandelet video compression on some standard video
sequences, i.e., Akiyo, Alex, Foreman and Mobile. All sequences are at QCIF resolution
except for Alex, whose resolution is CIF. In sequences with low motion content
(Alex and Akiyo), our results demonstrate the success of SPREF in improving the
wavelet compression. In other ones where the motion is dominant, we compare the
performance of SPREF-based compression with a motion-compensated wavelet
compression technique, namely the LIMAT framework of Secker and Taubman. The
improvement as a result of the directional decomposition and bandeletization in
SPREF-based compression can be clearly observed in this plot except for the
Mobile sequence. The reason is that the motion in the Mobile sequence consists
of many components such as global zooming out, rotating ball.
Associated publications:
Video
Compression Using Structural Flow, (Download PDF)
IEEE International Conference on Image Processing,
Genova, Italy, September 2005.
Video
Compression Using Spatiotemporal Regularity Flow,
IEEE Trans. Image Processing, Vol. 15, No. 12, pp. 3812-3823, December 2006. (Download PDF)
Spatiotemporal
Regularity Flow (SPREF): Its Estimation and Applications,
Submitted to IEEE Trans. CSVT IEEE Trans. Circuits and Systems
for Video Technology, vol. 17, No. 5, May 2007, pp. 584-589. (Download PDF)