We propose a new approach (named direct visual servoing) to vision-based control that does not require or estimate any 3D metric information, while also not depending on the object's shape or the camera motion. Thus, we do not rely on prior knowledge (leading to system flexibility), as well as we achieve robustness to camera calibration parameters. The sole requirement about the object is to be rigid. In addition, the visual tracking exploits all possible image information (pixel intensities are directly used without any feature extraction process) allowing us to attain high levels of accuracy while being computationally efficient. This latter is also important since real-time performance is always a major concern in robotic systems. Finally, the proposed control law possesses a very large domain of convergence due to a straightforward path planning step. This is highly desirable so that tasks can be performed despite large initial displacements.
The homography-based 2D visual servoing can be used to position a robot manipulator with respect to a planar object without having any measure (off-line or on-line) of the normal to the plane. Only the homography between the current and the reference image is needed for computing an error which is isomorph to the camera pose and to compute a stable control law. Thus, the same control law can be used with any single viewpoint camera. In the first video, the object is static and the robot is correctly positioned despite the initial displacement is very big. In the second video, the target is moving at 0.1 m/s and the robot is able to correctly track the target without any additional filter.
The ESM algorithm can be used to integrate visual tracking and a large class of vision-based control laws into a unified approach. The videos below show a positioning task using the 2 1/2 D visual servoing . The ESM algorithm is also used for matching the current image to the reference template with a coarse approximation of the homography matrix. An approximation of the normal to the plane is needed to have an unique solution for the homography decomposition.
Amongst all the many possible applications of real-time tracking of non-rigid objects we are currently studying in collaboration with LIRMM lab in Montepellier the robotic assistance for surgical intervention on a beating heart without a mechanical stabilizer. The goal is to substitute the mechanical stabilizer with a "virtual" stabilization, such that the heart is motionless in the surgical tools frame. The first video show that the basic version of the ESM visual tracking can be easily applied to locally planar deformable surfaces. In the second video below there is an example of the application of the ESM algorithm to the visual tracking of a larger part of a deformable beating heart. The undeformed image on the top left show that the tracking is correctly performed.
The ESM software can be used as a markerless visual tracking tools for computing the pose of the camera with respect to a a reference frame. Using the computed pose, we can add a virtual object in the scene. The video below show an example of a rotating sphere placed on the box.
The videos below were created using the ESM software during the practical course of Dr. Selim Benhimane at the Technischen Universitat of Munchen (Germany).