Related Publication: Vladimir Reilly, Berkan Solmaz and Mubarak Shah, Geometric Constraints for Human Detection in Aerial Imagery, The 11th European Conference on Computer Vision (ECCV), 2010.

• 1. Introduction
• 2. Ground-Plane Normal and Shadow Constraints
• 3. Human Detection
• 4. Experimental Results
• 5. Downloads

Detecting humans in imagery taken from a UAV is a challenging problem due to small number of pixels on target, which makes it more difficult to distinguish people from background clutter, and results in much larger search space. We propose a method for human detection based on a number of geometric constraints obtained from the metadata. Specifically, we obtain the orientation of ground plane normal, the orientation of shadows cast by humans in the scene, and the relationship between human heights and the size of their corresponding shadows. In cases when metadata is not available we propose a method for automatically estimating shadow orientation from image data. We utilize the above information in a geometry based shadow, and human blob detector, which provides an initial estimation for locations of humans in the scene. These candidate locations are then classified as either human or clutter using a combination of wavelet features, and a Support Vector Machine. Our method works on a single frame, and unlike motion detection based methods, it bypasses the global motion compensation process, and allows for detection of stationary and slow moving humans, while avoiding the search across the entire image, which makes it more accurate and very fast. We show impressive results on sequences from the VIVID dataset and our own data, and provide comparative analysis.

Ground-Plane Normal and Shadow Constraints

The imagery obtained from the UAV has the following metadata associated with most of the frames. It has a set of aircraft parameters latitude, longitude, altitude, which define the position of the aircraft in the world, as well as pitch, yaw, roll which define the orientation of the aircraft within the world. Metadata also contains a set of camera parameters scan, elevation, twist which define the rotation of the camera with respect to the aircraft, as well as focal length, and time. We use this information to derive a set of world constraints, and then project them into the original image.

World Constraints

Figure 2. Left, the sensor model П₁ maps points in camera coordinates into world coordinates (since the transformation between image and camera coordinates is trivial we do not show it in the image). X corresponds to East direction, Y to North, Z to vertical direction. Vector S is pointing from an observer towards the sun along the ground. It is defined in terms of α azimuth angle between northern direction and the sun. Zenith angle Ɣ is between vertical direction and the sun. The height of a human is k, and the length of the shadow is l. We place the image plane into the world, and raytrace through it to find the world coordinates of the image points (we project from the image plane to the ground plane). We compute a homography H₁ between image points and their corresponding world coordinates on groundplane. Right, illustrates how we obtain the projection of the groundplane normal in the original image. Using a lowered sensor model П₂ we obtain another homography H₂, which maps points in camera coordinates to a plane above the ground plane. Mapping a world point p_c1 using H₁, and H₂, gives two image points p^'_c1, and p^'_c2. Vector from p^'_c1 to p^'_c2 is the projection of the normal vector.

Human Detection

In order to avoid the search over the entire frame, the first step in our human detection process is to constrain the search space of potential human candidates. We define the search space as a set of blobs oriented in direction of shadow, and direction of normal. To do so we utilize the image projection of the world constraints derived previously - the projected orientation of the normal to the ground plane, the projected orientation of the shadow, and the ratio between the projected person height, and projected shadow length. See Figure 3.

Figure 3. This figure illustrates the pipeline of applying image constraints to obtain an initial set of human candidates.

Our next step is to relate the shadow and human blob maps, and to remove shadow-human configurations that do not satisfy the image geometry which we derived from the metadata. We search every shadow blob, and try to pair it up with a potential object blob, if the shadow blob fails to match any object blobs, it is removed. If an object blob never gets assigned to a shadow blob it is also removed.

Figure 4. (a) shows shadow blob map (shown in red), and normal blob map (shown in green), overlaid on the original image. Notice there are false detections at the bottom of the image. Yellow arrow is the projected sun vector S', the projected normal vector z' is shown in green, and the ratio between the projected normal and shadow lengths is 2.284 (b) shows example candidates being refined. A valid configuration of human and shadow blobs (top) results in an intersection of the rays, and is kept as a human candidate. An invalid configuration of blobs (bottom) results in the divergence of the rays, and is removed from the set of human candidates. (c) shows refined blob maps after each normal blob was related to its corresponding shadow blob.

Wavelets have been shown to be useful in extracting distinguishing features from imagery. So in the final step of our method, we classify each object candidate as either a human or non-human using a combination of wavelet features and SVM (Figure 5). We chose wavelet features over HOG because we obtained higher classification rate on a validation set. We suspect that this is due to the fact that in the case of HOG, the small size of chips does not allow for the use of optimal overlapping grid parameters, giving too coarse sampling. We apply Daubechies 2 wavelet filter to each chip.

Figure 5. Object candidate classification pipeline. Four wavelet filters (LL, LH, HL, HH) produce scaled version of original image, as well as gradient like features in horizontal vertical and diagonal directions. The resulting outputs are vectorized, normalized, and concatenated to form a feature vector. These feature vectors are classified using SVM.

Experimental Results

1. Power Point Presentation (.pptx) (14.9 MB)