INRIA
INRIA

-----------------------
ESM Software Development Kits
-----------------------

Applications

Mobile Robotics

The ESM visual tracking and servoing software can be used for several real-time applications. In particular, mobile robotic applications are very challenging since they need high accuracy and robustness.
  • Platooning with car-like vehicles

  • The ESM visual tracking and servoing software has been applied to autonomous vehicle navigation. The visual tracking directly estimates the homography between a selected reference template attached to the leading vehicle and the corresponding area in the current image. The relative position and orientation of the servoed car with regard to the leading one is computed by decomposing the homography. The control objective is stated in terms of path following task in order to cope with the non-holonomic constraints of the vehicles. The camera is mounted on a pan-tilt turret in order to keep the leader vehicle in the center of the image.

    ESM visual tracking and servoing software ESM visual tracking and servoing software



Visual Servoing

  • Positioning a 6 d.o.f. robot manipulator with respect to any rigid object using the direct visual servoing

  • We propose a new approach (named direct visual servoing) to vision-based control that does not require or estimate any 3D metric information, while also not depending on the object's shape or the camera motion. Thus, we do not rely on prior knowledge (leading to system flexibility), as well as we achieve robustness to camera calibration parameters. The sole requirement about the object is to be rigid. In addition, the visual tracking exploits all possible image information (pixel intensities are directly used without any feature extraction process) allowing us to attain high levels of accuracy while being computationally efficient. This latter is also important since real-time performance is always a major concern in robotic systems. Finally, the proposed control law possesses a very large domain of convergence due to a straightforward path planning step. This is highly desirable so that tasks can be performed despite large initial displacements.

    ESM visual tracking and servoing software ESM visual tracking and servoing software ESM visual tracking and servoing software ESM visual tracking and servoing software

  • Positioning a 6 d.o.f. robot manipulator with with respect to planar objects using the homography-based 2D visual servoing

  • The homography-based 2D visual servoing can be used to position a robot manipulator with respect to a planar object without having any measure (off-line or on-line) of the normal to the plane. Only the homography between the current and the reference image is needed for computing an error which is isomorph to the camera pose and to compute a stable control law. Thus, the same control law can be used with any single viewpoint camera. In the first video, the object is static and the robot is correctly positioned despite the initial displacement is very big. In the second video, the target is moving at 0.1 m/s and the robot is able to correctly track the target without any additional filter.

    ESM visual tracking and servoing software ESM visual tracking and servoing software

  • Positioning with the 2 1/2 D visual servoing

  • The ESM algorithm can be used to integrate visual tracking and a large class of vision-based control laws into a unified approach. The videos below show a positioning task using the 2 1/2 D visual servoing . The ESM algorithm is also used for matching the current image to the reference template with a coarse approximation of the homography matrix. An approximation of the normal to the plane is needed to have an unique solution for the homography decomposition.

    ESM visual tracking and servoing software ESM visual tracking and servoing software



Medical Robotics

  • Visual tracking of a deformable beating heart

  • Amongst all the many possible applications of real-time tracking of non-rigid objects we are currently studying in collaboration with LIRMM lab in Montepellier the robotic assistance for surgical intervention on a beating heart without a mechanical stabilizer. The goal is to substitute the mechanical stabilizer with a "virtual" stabilization, such that the heart is motionless in the surgical tools frame. The first video show that the basic version of the ESM visual tracking can be easily applied to locally planar deformable surfaces. In the second video below there is an example of the application of the ESM algorithm to the visual tracking of a larger part of a deformable beating heart. The undeformed image on the top left show that the tracking is correctly performed.

    ESM visual tracking and servoing software ESM visual trackin g and servoing software



Augmented Reality

  • Adding virtual objects with markerless ESM visual tracking

  • The ESM software can be used as a markerless visual tracking tools for computing the pose of the camera with respect to a a reference frame. Using the computed pose, we can add a virtual object in the scene. The video below show an example of a rotating sphere placed on the box.

    ESM visual tracking and servoing software

    The videos below were created using the ESM software during the practical course of Dr. Selim Benhimane at the Technischen Universitat of Munchen (Germany).

    ESM visual tracking and servoing software ESM visual tracking and servoing software


Last modification: 2010-02-02 13:24:25.000000000 +0100