Based
on the Lagrangian framework for fluid dynamics, a streakline
representation of flow is presented to solve computer vision
problems involving crowd and traffic flow. Streaklines are traced
in a fluid flow by injecting color material, such as smoke or dye,
which is transported with the flow and used for visualization. In the
context of computer vision, streaklines may be used in a
similar way to transport information about a scene, and they are
obtained by repeatedly initializing a fixed grid of particles at
each frame, then moving both current and past particles using
optical flow. Streaklines are the locus of points that connect
particles which originated from the same initial position. In this
paper, a streakline technique is developed to compute several
important aspects of a scene, such as flow and potential functions
using the Helmholtz decomposition theorem. This leads to a
representation of the flow that more accurately recognizes spatial
and temporal changes in the scene, compared with other commonly
used flow representations. Applications of the technique to
segmentation and behavior analysis provide comparison to
previously employed techniques, showing that the streakline method
outperforms the state-of-the-art in segmentation, and opening a
new domain of application for crowd analysis based on potentials.
I.
Introduction
We
introduce a new representation of flow to computer vision community.
The intuition comes from the resemblance of fluid flow to flow of
pedestrians. Based on Lagrangian Framework of fluid dynamics, there are
3 representations of flow:
Optical
Flow or Vector Field
Dense
Particle Trajectories or Pathlines
Streaklines
(a) Three representations of flow
1.
Challenges
In scenes with dynamic
motion, the flow changes continuously
Optical Flow is instantaneous, ignores the continuity of
motion
Trajectories represent the continuity of motion, not
imminent to capture behavior transitions
2. Solution
Streakline are instantaneous and capture the continuity of
motion
3. Advantages
The Streakline are more effective in dynamic scenes in
representing the changes in motion and behavior transitions.
With
regard to flow visualization, streaklines are defined as the traces of
a colored material in the low. To understand streaklines, consider a
fluid flow with an ink dye injected at a particular point. If the ink
is continuously injected, then a line will be traced out by the ink in
the direction of the flow, this is a streakline. If the direction of
flow changes, then the streaklines change accordingly.Streaklines
are new to computer vision research. In this context, streaklines may
be obtained by repeatedly initializing a grid of particles and moving
all particles according to the optical flow, in the spirit of a
Lagrangian fluid flow. In other words, place a particle at point p,
and move the particle one time step with the flow. In the next time
step, the point p is
initialized with a new particle, then both particles are moved with the
flow. Repeating this process on some time interval T
produces particle positions from which we obtain streaklines. Figure
(b) gives and example of streaklines in a video. For illustration
purposes, the streaklines are shown only on every 10th row and column.
(b) A
visualization of the strealines for an example video. The curves
represent the streaklines for every tenth row and column. Red and
yellow colors represent the different directions of motion.
2. Difference between
Streaklines, Pathlines, and Optical Flow
In
a dynamic scene, optical flow would be time varying. To illustrate the
effect of such an optical flow in different representations, we present
the following toy example. Consider a spatially consistent but time
varying vector field as:
If
we consider a certain point in this vector field it will generate a
curve as:
If
we initiate new particles at the same point at continuously we will be
able to generate a set of particle trajectories (pathlines)
as:
The
streakline is the curve that connects the position of all the particles
which are initiated at the certain point but at different times. Thus
the streakline in the toy example will be the red curve in the
following video:
III. Computation with
Streaklines
Streaklines enable us to
compute set of new features. In this section, we introduce two set of
new dense features: Streakflow and flow Potentials. Streaklines provide
a means to recognize spatial and temporal changes in the flow, that
neither streamlines nor pathlines could provide directly. This point is
made here using streak flow and potential functions. In essence, streak
flow is obtained by time integration of the velocity field (i.e.
optical flow), while potential functions are obtained from spatial
integration, and each provides useful information concerning the
dynamics in the scene.
1. Streakflow
(c)
Streak flow is computed as the weighted average of the imitating flow
vectors of each particle.
Streakflow is an
instantaneous vector field which represent the accumulative motion of
the scene. It resembles the temporal average of optical flow but it is
more imminent. This vector field has following properties:
Averages the velocities in direction of motion (Does not
average blindly) therefore it relates to the group motion
Imminently reactive to changes in the scene (Less lag than
temporal average optical flow)
Less noisy than optical flow (helpful in behavior
recognition)
A comparison of Streak
flow, Optical Flow, and average optical flow is give in Figure (d)
(d) The Streakflow stands in the middle of average optical flow and
instantaneous optical flow in capturing changes in the scene.
2. Potentials
Building
on the fluid dynamics approach
to crowd motion, we employ another concept from fluids providing a
different point of view. In simplified mathematical models of
fluids, it is often assumed that the fluid is imcompressible, and
irrotational. These assumptions imply several conservation
properties of the fluid, but most importantly, they lead to
potential functions, which are scalar functions that characterize
the flow in a unique way. For this discourse, potential functionsenable
accurate classification of behaviors in a scene, which is
not possible with streak flow alone.Since
the optical flow denotes a planar
vector field, the Helmholtz decomposition theorem states that, where and respectively denote the
incompressible and
irrotational parts of the vector field. To clarify, an
incompressible vector field is divergence free ,
and an irrotational vector field is curl free .
Thus, there are functions
and ,
known respectively as the stream function and the velocity
potential, satisfying and(see, for example [3]).
Following [4], we use Fourier transforms to
decompose incompressible and irrotational parts of the vector
field and estimate the potential functions using Helmholtz
decomposition theorem.
Figure
(e) illustrates the computed stream function and velocity potential for
a traffic scene where the cars are turning left.
(e) The
two types of potentials: The Velocity Potential and Stream Function.
IIV. Motion Segmentation
Streaklines are representations of motion and therefore, regions of
similar motion would have similar streakline. Given the video we first
compute the frame by frame optical flow and then we compute
streaklines. We find the pixel by pixel similarity map for the
neighboring streakline. Finally, a simple watershed segmentation
separates the regions of coherent motion.
(f) The segmentation
algorithm
The similarity of neighboring streaklines is computed as a weighted sum
of their similarity in shape and the similarity of the flow which they
are representing (equation 7 in the PDF version of the paper). The
shape similarity of streaklines is computed by normalize projection of
the vectors of streaklines as in Figure (g).
(g) The shape similarity
of streaklines
The flow similarity of the
neighboring pixels is computed based on the streakflow at their origin.
Lane
Detection
In
addition to segmenting a frame into regions of consistent motion, we
combine information from
potentials to detect lanes in each segment. As stated above, the area
between contours of corresponds
to the steady flow, and the rate of the incompressible flow
between a pair of contours is equal to the difference between the
values of
on those contours. Considering this, we detect lanes as parts of a
segmented region that fall between twocontours
of the stream function by a simple intersection operation.
(g) The lane detection
algorithm
V. Behavior Recognition
To detect abnormal behavior of crowds, it is necessary to have a global
picture of the behaviorin a scene, for which we use potential fields.
The surfaces and characterize
article positions and velocities in a global sense, and abnormal
behaviors are simply detected as large deviations from the expected.
Here, we present an algorithm to detect abnormal behavior in crowds
using potential functions for the flow (Figure h).
(h) The behavior recognition
algorithm
Divergent/
Convergent Regions
In
addition to detecting abnormal behaviors, we incorporate streaklines
and the velocity potential
to provide a description of the anomaly based on divergent/convergent
regions. The extrema on velocity potentials correspond to divergent or
convergent regions. To robustly detect these regions, we find the major
local extrema of ,
and then compute the average
divergence
factor , where Vi
is defined in Figure (i). Simple thresholding of this factor
distinguish divergent/convergent regions
(i) The computation of
divergence factor
(g) Sample results on
segmentation of Boston Sequence
VI. Results
(h)Qualitative Results
(j)
Quantitative Results
VII. Downloads
Please email me for passwords: ramin at cs dot ucf dot edu.
Helman, J., Hesselink, L.: Visualizing vector field
topology in fluid flows. IEEE Comput. Graph. Appl. 11 (1991) 36–46
Corpetti, T., Memin, E., Perez, P.: Extraction of singular
points from dense motion fields: An analytic approach. Journal of
Mathematical Imaging and Vision (2003)
IX. Bibtex Reference
@inproceedings{MehranMooreShah_ECCV10,
author="Ramin Mehran and Brian E. Moore and Mubarak Shah",
title="A Streakline Representation of Flow in Crowded Scenes",
booktitle="European Conference on Computer Vision(ECCV)",
year="2010",
}
denotes a planar
vector field, the Helmholtz decomposition theorem states that
, where 
and
respectively denote the
incompressible and
irrotational parts of the vector field. To clarify, an
incompressible vector field is divergence free 
,
and an irrotational vector field is curl free 
.
Thus, there are functions 
and 
,
known respectively as the stream function and the velocity
potential, satisfying 
and
(see, for example [3]).
Following [4], we use Fourier transforms to
decompose incompressible and irrotational parts of the vector
field and estimate the potential functions using Helmholtz
decomposition theorem.
, where Vi
is defined in Figure (i). Simple thresholding of this factor
distinguish divergent/convergent regions
