Final examination of Mr. Yaser Sheikh for the degree of PhD

Dissertation Title: Object Association across Multiple Cameras in Planar Scenes

Abstract:

Co-operative sensing presents an interesting paradigm: solving a difficult global sensing problem with a number of efficient but simpler local sensors. In this dissertation, this paradigm is introduced to the problem of scene understanding over wide planar dynamic scenes observed by multiple cameras. In particular, the problem of object association across multiple cameras, moving or stationary, in planar scenes is addressed. We present a unifying probabilistic framework that captures the underlying geometry of planar scenes, and propose algorithms to estimate geometric relationships between different cameras, which are subsequently used for global association of objects. We first present a local object detection scheme occurring at each individual camera that has fundamental innovations over existing approaches, allowing reliable detection in the presence of dynamic textures (like water or swaying foliage), nominal misalignment (due to noisy registration) and residual motion due to parallax.

After detection in each camera, algorithms for associating objects across multiple cameras in planar scenes are presented for two general problem configurations in terms of spatiotemporal overlap of the camera fields of view. We first address the case where limited spatial and temporal overlap can be assumed. Since the cameras are moving and often widely separated, direct appearance-based or proximity-based constraints cannot be used. Instead, we exploit geometric constraints on the relationship between the motion of each object across cameras, to test multiple correspondence hypotheses, without assuming any prior calibration information. We present a statistically and geometrically meaningful means of evaluating a hypothesized correspondence between two observations in different cameras. Since multiple cameras exist, ensuring coherency in association, i.e. transitive closure is maintained between more than two cameras, is an essential requirement. To ensure such coherency we pose the problem of object tracking across cameras as a k-dimensional matching and use an approximation to find the association. We show that, under appropriate conditions, re-entering objects can also be re-associated to their original labels. In addition, we show that as a result of tracking objects across cameras concurrent visualization of multiple aerial video streams is possible.

Finally, we propose a unifying framework for object association across multiple cameras and for estimating inter camera homographies between cameras with spatiotemporally overlapping and non-overlapping fields of view, whether they are moving or stationary. By making use of explicit models for the kinematics of objects, we present algorithms to estimate inter-frame homographies. Under an appropriate measurement noise model, an EM algorithm is applied to compute the maximum likelihood estimates of the inter-camera homographies and kinematic parameters. Rather than fit curves locally (in each camera) and match them across views, we propose an approach that simultaneously refines the estimates of inter-camera homographies and curve coefficients globally. We demonstrate the efficacy of the algorithms presented on a number of real sequences taken from multiple cameras, and report quantitative performance during numerical simulations.

Educational career:
B.S. Ghulam Ishaq Khan Institute of Engineering Sciences and Technology, 2001
M.S. University of Central Florida, 2005

Committee in charge:
Dr. Mubarak Shah, Chair
Dr. Charles Hughes
Dr. Annie Wu
Dr. Huaxin You

Approved for distribution by Mubarak Shah, Committee Chair, March 2006