INRIA
INRIA

-----------------------
ESM Software Development Kits
-----------------------

Demonstrations

Visual Tracking

The ESM visual tracking has been tested on several sequences under disparate illumination conditions and it works with any camera with a single viewpoint. The algorithm is intrinsically robust to partial occlusion but it has been improved by using robust optimization techniques. The videos provided below shows several examples of the ESM visual tracking algorithm.

  • Visual tracking of planar objects with single viewpoint cameras

  • In the first video, a planar object is tracked using a perspective camera despite several illumination changes. Strong camera displacements can be handled in real-time by the ESM visual tracking. In the second video, a chip board is tracked at very high speed (100 Hz). The specular reflections do not affect the tracking and the area of interest can be partially out of the camera field of view. The chip board is not exactly planar and undergo severe projective distortion in the image. In the third video, the back of a car is tracked. The initial area of interest is only 40x40 pixel while the final one is 5 times bigger. Despite the very high change of scale the tracking is accurately performed. The ESM visual tracking can be used with any single viewpoint camera. For example, in the last two videos, a planar object is tracked in omnidirectional images acquired with a parabolic mirror. In the fifth video, the image on the top-left corner is the reprojection of the tracked area of interested in a reference frame.

    ESM visual tracking and servoing software ESM visual tracking and servoing software ESM visual tracking and servoing software ESM visual tracking and servoing software ESM visual tracking and servoing software

  • Visual tracking of rigid objects with unknow shape

  • In each video, an area of interest of a rigid object is tracked using a perspective camera. A reference template is selected manually in the first image of the sequence. The shape of the rigid object is unknown and it is thus recovered online during the visual tracking. In the videos, a red grid is superposed to the current area of interest in order to show the 3D structure of the object.

    ESM visual tracking and servoing software ESM visual tracking and servoing software ESM visual tracking and servoing software

  • Visual tracking of deformable surfaces

  • In each video, a part of a deformable object is tracked using a perspective camera. A reference template is selected manually in the first image of the sequence. In the top left corner we show the undeformed image (i.e. the reprojection of the current image in the reference frame). The undeformed image is almost unchanged during the entire sequence proving that the ESM visual tracking is able to estimate the deformation of each pixel of the reference template.

    ESM visual tracking and servoing software ESM visual tracking and servoing software ESM visual tracking and servoing software

  • Visual tracking robust to arbitray illumination changes

  • We propose a new approach to the direct image alignment of either Lambertian or non-Lambertian objects under shadows, inter-reflections, glints as well as ambient, diffuse and specular reflections which may vary in power, type, number and space. The method is based on a proposed model of illumination changes together with an appropriate geometric model of image motion. The parameters related to these models are simultaneously obtained through the ESM optimization technique which minimizes directly the intensity discrepancies. Comparison results with existing direct methods show significant improvements in the tracking performance.

    ESM visual tracking and servoing software ESM visual tracking and servoing software ESM visual tracking and servoing software ESM visual tracking and servoing software ESM visual tracking and servoing software

  • Outliers rejection for robust visual tracking

  • When the models used in the visual tracking are not accurate enough the optimization may fail. For example, if the target is partially occluded the overall motion of the area of interest will not be coherent. In the general case, outlier measures can be discarded by using robust cost functions. We have tested the use of M-estimators in the ESM visual tracking algorithm. The video below show an example of the robustness of the algorithm when tracking a planar object with severe illumination changes and specular reflections. In this case, illumination changes and specular reflections are not explicitly modeled (see section above) but treated as outliers. Although M-estimators allows to handle the case of partial occlusions that can be hardly modeled, the price to pay is a higher computation time and a lower convergence rate.

    ESM visual tracking and servoing software

  • Stereo visual tracking

  • The ESM technique has been successfully applied to the visual tracking using a stereo pair. The video below show an example of the visual tracking of a sphere. The user selects a region of interest in the left image (the blue rectangle in this case). After the corresponding region in the right image is found, the visual tracking starts.

    ESM visual tracking and servoing software

Visual Odometry

  • 3D Visual odometry with single viewpoint cameras

  • The ESM visual tracking can be used for estimating the displacement of a robot with respect to a reference frame (e.g, the initial position). The translation can be estimated only up to a scale factor. Additional information (e.g. a known distance) allow to recover the scale factor. When compared with the odometry of a well calibrated robot we obtain a very good precision. In the first video, the pose of the robot is directly estimated from the images (i.e. we impose a rigidity constraint on the planes in the scene) knowing the camera parameters and a model of the scene. In the second video, the pose of the robot and a piecewise-planar model of the scene are simultaneously estimated. This is a first step towards a visual SLAM approach for single viewpoint cameras.

    ESM visual tracking and servoing software ESM visual tracking and servoing software

  • 3D Visual odometry with a stereo pair

  • The stereo ESM visual tracking can be used for estimating the displacement of a car with respect to a reference frame (e.g, the initial position). The algorithm was tested on real full-scale sequences as can be seen in the videos below. Several test sequences from different streets in Versailles, France, were used to validate the results. The estimated 3D trajectory from a stereo-pair of cameras was superimposed on satellite images of the area. This first sequence is that of a relatively straight road. The distance traveled by the car has been measured using road markings in the images and satellite views with a precision of 2.9 cm/pixel for the Versailles region. The length of the path measured by Google earth was about 436m and the estimated length from the tracker is 420m giving an approximate drift of 4%. Throughout the sequence several moving vehicles pass in front of the cameras and a one stage a car is overtaken. The second sequence is particularly illustrative since a full loop of a round-about was performed. In particular this enables the drift to be measured at the crossing point in the trajectory. The drift at the crossing point was approximately 2m in the vertical direction to the road-plane. Considering that the trajectory around the round-about is approximately 200m long (measured using Google earth), this makes a drift of 1% on the measurable axis.

    ESM visual tracking and servoing software ESM visual tracking and servoing software ESM visual tracking and servoing software

Visual SLAM

The ESM visual tracking can be used to perform accurate Simultaneous Localisation And Mapping on large scale environments. In the computer vision community, the SLAM problem is also known as the Structure From Motion problem.
  • Monocular direct visual SLAM

  • We built an efficient method (named direct visual SLAM) that directly computes the 3D camera displacement and the scene structure. The method is robust to arbitrary illumination variations. Feature extraction is not needed given that the intensities of all possible pixels are directly used. The system is automatically initialized from the first image. Motion and structure parameters are simultaneously estimated with the ESM technique for faster processing and avoidance of irrelevant local minima. Thus, the method can cope with large inter-frame displacements. Rigidity and visibility constraints on the structure are enforced. All these factors significantly contribute to achieve accurate SLAM results. The video below shows the SLAM results on a real-world urban sequence captured in Versailles, France. The left frame shows the input images superposed with the tracked regions, while the right frame shows both the 3D pose and scene structure being incrementally recovered.

    ESM SLAM SDK ESM SLAM SDK


Last modification: 2010-02-03 09:58:12.000000000 +0100