Force feedback pushing scheme for micromanipulation applications

  • Shahzad KhanEmail author
  • Asif Sabanovic
Open Access
Research Paper


Pushing micro-objects using point contact provides more flexibility and less complexity compared to pick and place operation. Due to the fact that in micro-world surface forces are much more dominant than inertial forces and these forces are distributed unevenly, pushing through the center of mass of the micro-object may not yield a pure translational motion. In order to translate a micro-object, the line of pushing should pass through the center of friction. In this paper, a semi-autonomous scheme based on hybrid vision/force feedback procedure is proposed to push micro-objects with human assistance using a custom built tele-micromanipulation setup to achieve translational motion. In the semi-autonomous pushing process, velocity controlled pushing with force feedback is realized along x-axis by the human operator while y-axis orientation is undertaken automatically using visual feedback. In this way the desired line of pushing for the micro-object is controlled to pass through the varying center of friction. Experimental results are shown to prove nano-Newton range force sensing, scaled bilateral teleoperation with force feedback and pushing operation.


Micromanipulation Pushing Bilateral teleoperation Center of friction Piezoresistive cantilever 

1 Introduction

As nature has provided us with things in dimensions ranging down till micro/nanometers, likewise humans also were able to fabricate components in the same scales. But the prominent challenge lies in the task of assembling components in a single and functionalized product. Use of techniques to produce complex micro/nano systems in a monolithic way is desirable, but is not always feasible. The current practice is to incorporate components into a single functional product and to assemble micro parts one by one [1, 2, 3]. One possible solution to this problem is to develop setup capable to assist humans to assembly micro-parts. The first and foremost requirement for the assembly process is to “precisely manipulate” objects. Manipulation includes cutting, pushing, pulling, indenting, or any type of interaction which changes the relative position and relation of entities. This paper concentrates on manipulation by pushing as it is a useful technique for manipulating delicate, small, or slippery parts, parts with uncertain location, or parts that are otherwise difficult to grasp and carry [4, 5, 6, 7]. The process of manipulation by pushing of micro-objects possesses many challenges due to the requirements of:
  • Actuators with high resolution (in nanometer range), high bandwidth (up to several kilo hertz), large force output (up to few newtons) and relatively large travel range (up to a few millimeters) [8].

  • Robust and transparent bilateral controllers for human intervention so that high fidelity position/force interaction between the operator and the remote micro environment can be achieved [9, 10].

  • Vision based algorithms to estimate the location of objects being manipulated and visual servoing to position manipulators so that these objects can be manipulated (pushed) along a desired trajectory [11, 12].

  • Controlled pushing force to generate the desired compensation surface forces arising between the object and the environment.

Pushing objects requires not only precise position control of actuators but also delicate control of forces involved in the manipulation process due to the fragility of the micro-object [7, 13]. It is also indispensable to sense and control the interaction forces involved in the manipulation process with nano-newton resolution which makes it necessary to utilize force feedback mechanism or bilateral control. Bilateral control, which is typically used for teleoperation, offers a solution to these tasks since it enables the operator to work remotely. That is, if actual presence of an operator is not possible, inclusion of a bilateral control system between the operator and the task would simply give a possibility to so called “telepresence” of the operator. Bilateral control is defined as the control of two systems working together on an actual or virtual task. Typically, it is used for teleoperation, in which one system is called the “master” side and the other is called the “slave” side of bilateral action. The slave subsystem tracks the positions of the master subsystem which provides the forces encountered by the slave side to the operator and hence, teleoperation is achieved [14, 15]. Nowadays, many researchers have come up with the notion of multilateral control [16, 17] consisting of more than two systems working with proper coordination to achieve a desired task. In order to perform tele-micromanipulation it is indispensable to achieve robust and transparent bilateral controllers for human intervention so that high fidelity position/force interaction between the operator and the remote micro/nano environment can be achieved [9, 10]. Bilateral control enables skilled teleoperation on several tasks, it offers better safety, low cost and high accuracy. On the other hand it also suffers from time delay problem [18, 19] which effects the transparency of the systems.

Apart from the above mentioned requirements, precise positioning of micro-objects lying on a substrate using a point contact pushing to track a desired trajectory posses lot of challenges. The pusher or probe needs to be controlled in such a way to reorient and transport the micro-object to its final location using a stable pushing1 operation. Using only a point contact with limited degrees of freedom, the task of pushing on a horizontal plane can be realized. In the micro-world, the inertial effect can be neglected compared to the frictional forces existing between the micro-object and the substrate which plays a very dominant role at this scale. The direction and the magnitude of the frictional forces will determine the direction and magnitude of the velocity thats needs to be applied at the contact point to move the object in a desired trajectory. The velocity at the contact point is the control variable for the object motion and it is necessary to understand the properties of frictional forces before planning the pushing operation.

In this work, we are interested in translation of a regular object from one location to another by orientating the line of action of the pushing force to the desired direction. The desired translational motion of the object cannot be achieved if the line of pushing at the contact point passes through the center of the mass of the micro-object. Due to the dominance of the frictional forces, the resultant line of pushing needs to be directed to eliminate any moment caused by the frictional force so that orientation effect of the pushed object is rejected to ensure only translational motion [20]. The frictional distribution in between the contact surface of the micro-object and supporting surface gives rise to centroid of the frictional distribution, center of friction2 where all the distribution of friction can be lumped into a single point. Mason [21] showed that when the resultant pushing force vector applied on an object passes through the center of friction, the motion of the object is a pure translation. The result is obtained by assuming that the coefficient of friction of object with the substrate is constant over the time. Unfortunately, this is not a feasible solution in micro-world, where due to uncertain topography of the surfaces the frequency distribution changes with respect to time, giving rise to the uncertain location of the center of friction which is not constant over time. Thus, the most important task lies in the online estimation of the center of friction using visual information and sensed force which is one of the focus of this work. The paper is organized as follows. Section 2 provides the problem definition and approach and Section 3 explains the custom built tele-micromanipulation setup. In Section 4, scaled bilateral teleoperation is demonstrated with experimental details concerning force/position tracking between the master and the slave. Finally, Section 5 provides the procedure for pushing micro-objects along with the experimental results in Section 6 and Section 7 concludes the paper and discusses future directions.

2 Problem definition and approach

The problem dealt within this work concerns utilizing semi-autonomous scheme for pushing of polygonal micro-object, by point contact to achieve translational motion with the aid of a human operator employing scaled bilateral teleoperation with force-feedback. In the past, researchers [21] have performed pushing operation to achieve translational motion by allowing the line of action to pass through the time-invariant center of friction which is not valid in case of micro world where the location of center of friction is time dependent. Thus, to achieve translation motion in the micro world, the proper line of action of the pushing force needs to be always pass through the varying center of friction of the polygonal micro-objects. Moreover, researchers [7] in the past have utilized automatic schemes to push a micro-object but in case of unpredictable environments, it is advisable to include the human operator in the loop due to its adaptability, understanding and decision making capabilities.

In order to prevent the sliding of the micro-object during the pushing operation, it is necessary that the the pusher falls within the friction cone3 as denoted in Fig. 1. Theoretical value of μ (frictional coefficient) between the silicon tip of the cantilever and silicon micro-object is 0.25 [22], thus the angle for friction cone can be calculated as 28.07°. Thus, while the pushing operation is in progress, it is inevitable that the probe stays within the friction cone.
Fig. 1

Friction cone between the pusher and the micro-object

Figure 2 represents the scenario of moving rectangular object using a point contact pushing to achieve pure translation motion. The rectangular micro-object has two points, namely COM (center of mass) and COF (center of friction). The contact point of the pusher is taken as the origin of the reference frame. The x-axis and y-axis of the frame is chosen to be parallel with the two sides of the rectangle. The velocity of the probe along x-axis (\(\vec{V}_{\!x}\)) and y-axis (\(\vec{V}_{\!y}\)) are controlled by visual feedback and human operator, respectively. The desired velocity vector \(\vec{V}_{\!des}\), resultant of \(\vec{V}_{\!x}\) and \(\vec{V}_{\!y}\) needs to pass through COF, hence have an angle θ d is required to achieve a pure translation motion. The value of \(\vec{V}_{\!y}\) cannot be controlled to achieve the desired velocity vector as it is administered by the human operator, rather it’s only a measurable quantity. The variable \(\vec{V}_{\!x}\) can be calculated by taking into consideration the value of \(\vec{V}_{\!y}\) to achieve the desired velocity vector \(\vec{V}_{\!des}\) which makes an angle θ d as denoted in the following equations.
Fig. 2

Calculation of velocity vector for known center of friction

The relationship between the \(\vec{V}_{\!x}\) and \(\vec{V}_{\!des}\) can be written as Eq. 1 by analyzing Fig. 2 and solving for \(\vec{V}_{\!des}\) yields Eq. 2.
$$ \vec{V}_{\!des}\cos\theta_{d}=\vec{V}_{\!x} \label{Eqn:Vx} $$
$$ \vec{V}_{\!des}=\frac{\vec{V}_{\!x}}{\cos\theta_{d}} \label{Eqn:Vx_solve} $$
Similarly, the relationship between the \(\vec{V}_{\!y}\) and \(\vec{V}_{\!des}\) can be written as Eq. 3 and inserting the Eq. 2 into Eq. 3 will yield Eq. 4
$$ \vec{V}_{\!des}\sin\theta_{d}=\vec{V}_{\!y} \label{Eqn:Vy} $$
$$ \vec{V}_{\!y}=\vec{V}_{\!x}\tan\theta_{d} \label{Eqn:Vy_solve} $$

The Eq. 4 indicates that by only controlling \(\vec{V}_{\!y}\), it is possible to achieve the resultant velocity vector \(\vec{V}_{\!des}\) to pass through COF.

The whole process of pushing a micro-object is divided into two concurrent process: in one process pushing is performed by the human operator which can react to unexpected situation by switching between force/position control and alters the velocity of the pusher while in contact with the micro-object. In other process, visual servoing process continuously changes the velocity of the probe in x-axis, so that the resultant line of pushing passes through the varying center of friction. The proposed structure have been shown in Fig. 3. The necessary subtasks utilized to perform the whole process are as follows:
  • A piezoresistive AFM microcantilever has been utilized to measure the interaction forces with the environment with nano-newton resolution.

  • Human operator interacts with the micro environment using scaled bilateral teleoperation. The desired position is commanded by the human operator and transferred to the micro environment after scaling and the resultant interaction forces are felt by the human operator after the force scaling.

  • Visual processing techniques are employed to detect position and orientation of the micro-object for the estimation of the desired line of pushing [23].

Fig. 3

Hybrid control structure for semi-automated pushing scheme

3 Micromanipulation setup

As mentioned in Section 2, the task of pushing of micro-object with human assistance has some requirements, thus a custom built tele-micromanipulation setup is developed which can perform fulfill the requirements. The developed systems is equipped with a master robot for human interface with force feedback, visual feedback for automatic aligning of the micro-cantilever and a slave robot containing high precision stages and force sensing with high resolution. The system is composed of three parts, namely human-machine interface, vision system and XYZ manipulator. The schematic of the overall system is shown in Fig. 4.
Fig. 4

Schematic of tele-micromanipulation system

The position data from the master side is scaled and transferred to the slave side, while simultaneously, the force measured at the slave side is scaled and transferred back to the master. Figure 5 shows the experimental setup. The modules which have been utilized for different functionality are as follows:
  • As a 1DOF master device, DC servo (Maxon Motor RE40) has been utilized along with the light rod connected with the shaft.

  • In order to move the cantilever with high precision, three axes Piezo stages P-611 by Physik Instrumente has been utilized as a fine motion comprising of closed loop stages which possess a travel range of \(100 \: \mu m\) per axis with one nanometer resolution.

  • An open loop piezoelectric micromotor linear drive (PiezoMike PI-854 from Physik Instrumente) have been utilized as the base stage, which is equipped with integrated high resolution piezo linear drives. Manually operable linear drives are capable of \(1 \: \mu m\) resolution and the automatic movement range with respect to the reference position can be set \(50 \: \mu m\) (\(25 \: \mu m\) in/out). The travel range of the manual drive is \(18\:mm\).

  • For nano-newton range force sensing, a piezoresistive AFM cantilever with an inbuilt Wheatstone bridge (from AppNano) have been utilized.

  • A real time control card dSPACE DS1103 is used as a control platform and the programming is done in C.

  • For visual feedback, a Nikon MM-40 Microscope along with two Unibrain Fire-i 400 FireWire camera is connected to the microscope to capture the visual data and transmit it to the computer. As an example, Fig. 6 shows the microscopic view of the pusher and the micro-object taken from the camera.

Fig. 5

Experimental setup for micromanipulation

Fig. 6

Microscopic view of the pusher and the micro-object

4 Scaled bilateral teleoperation

In this section implementation of scaled bilateral controller in a custom built tele-micromanipulation setup is presented. Force sensing with nN resolution using piezoresistive AFM (Atomic Force Microscope) micro-cantilever is demonstrated. Force/position tracking between the master and the slave is presented after necessary scaling.

4.1 Force sensing using piezoresistive AFM microcantilever

The piezoresistive AFM cantilever with inbuilt Wheatstone bridge from AppliedNanostructures is utilized as a force sensor as well as a probe for pushing operation as shown in Fig. 7. Piezoresistive sensors have been used for many other MEMS applications, including accelerometers, gyroscopes and AFM cantilevers. The primary advantage of this device is that the sensor impedance is relatively low (a few ), and it is possible to extract small signals without interference from noise with off-chip integrated circuits.
Fig. 7

Piezoresistive AFM Cantilever with inbuilt Wheatstone bridge

The working principle is based on the fact that as the force is applied at the free end of the cantilever with the glass slide, the change of resistance takes place depending on deflection of the cantilever. The amount of deflection is measured by the in-built Wheatstone bridge providing a corresponding voltage output, which is amplified by the custom built amplifier. The amplified voltage is send to the data acquisition dSpace1103 card and filtered with a first order low pass filter with a cutoff frequency of \(100\:Hz\) and the force is calculated using Hooke’s Law as Eq. 5.
$$ F=K_{c}z \label{eqn.hookeslaw} $$
Where K c is the known spring constant of 0.3603 N/m and z is the amount of cantilever deflection. The spring constant is calculated by considering a linear beam equation as Eq. 6.
$$ K_c=\frac{3EI}{L^3} \label{eqn.beamequation} $$
Where E represents the modulus of elasticity (\(190\:GPa\) for silicon), L represents the length of the cantilever (\(300\:\mu m\)) and I represent the moment of inertia calculated as Eq. 7.
$$ I=\frac{bh^3}{12} \label{eqn.inertiacalc} $$
Where b and h represents the width and height of the microcantilever which is \(50\:\mu m\) and \(1.6\:\mu m\) respectively and the calculated value of inertia is 17.067×10 − 24 m 4.

4.1.1 Experimental results for force sensing

The cantilever is mounted on the three axes closed loop stage and the interaction (contact and non-contact) forces between the tip and glass slide are measured. The position of the cantilever is aligned to be perpendicular to the plane of the optical axis in order to achieve better visibility of the distance between the cantilever and the glass slide. Since the displacement range of the x-axis of the closed loop stage is \(100\:\mu m\), the glass slide is brought within the range using open-loop manual PZT axes.

In order to verify force measurement, theoretical values of pull-off force (breaking load during the withdrawal of tip) between the silicon tip and the glass surface is compared with the experimental results. In case of the interaction between a spherical tip and a flat surface, the interaction force can be approximated by Dugdale model [24, 25] as Eq. 8.
$$ F_{\mathit{pull-off}} = \left( \frac{7}{4}- \frac{1}{4} \frac{4.04 \lambda^{\frac{1}{4}}-1 }{4.04 \lambda^{\frac{1}{4}}+1 } \right) \pi WR \label{Eqn:dugdale_proof} $$
Where, W is the work of adhesion between the two mediums, R is the radius of the sphere and λ is a coefficient, which can be used to choose the most appropriate contact model for a given case [26]. Using the interfacial energy the pull-off force can be calculated for λ = 0.54 according to the Dugdale model as \(39.43\:nN\) [27, 28]. Figure 8 demonstrates that experimentally determined pull-off force is close to \(40\:nN\), indicating a close match between the theoretically and experimentally determined values. The result clearly indicates that force sensing with the resolution of nN range is achieved and verified.
Fig. 8

Force curve for interaction between a silicon tip and a glass surface

4.2 Scaled bilateral control structure

Since the master and slave are working on macro and micro scales respectively, it is indispensable to use general bilateral controller to scale the position and forces between two sides for extensive capability. In other words, position information from the master is scaled down to slave and force information from the slave side in scaled up to master as shown in Fig. 9 comprising the master and the slave side. The piezo-stage on the the slave side is required to track the master’s position as dictated by the position controller. The 1D force of interaction with environment, generated by the piezoresistive cantilever, on the slave side is transferred to the master as a force opposing its motion, therefore causing a “feeling” of the environment by the operator. The conformity of this feeling with the real forces is called the “transparency”. Transparency is crucial for micro/nanomanipulation application for stability of the overall system.
Fig. 9

Scaled bilateral teleoperation control structure

In order to eliminate oscillations on the master side because of human hand tremor and on the slave side due to piezoresistive cantilever dynamics, position of master manipulator and force of slave manipulator are filtered by low pass filters before scaling.

4.2.1 Scaling of the position and force information

Since the master and slave side reside on macro and micro scales respectively, it is very vital to appropriately choose the scaling factor in order to attain the optimum performance and maintain stability [29]. In the ideal condition, the steady state condition of the bilateral controller should be Eq. 9.
$$ \begin{array}{lll} x_{s}&=&\alpha x_{m}\\ F_{m}&=&\beta F_{s} \label{Eq:6.1} \end{array} $$
Where α and β represent the position and force scaling respectively. x m , x s denotes the master and slave position respectively and F m , F s denotes the master and slave force respectively. To be able to meaningfully interact with the micro environment, positions and forces are scaled to match the operator requirements and to maintain stability by satisfying the condition αβ < 1 [30].

In the first and second experiments, scaling factors of \(\alpha = 0.027 \frac{\mu m}{deg}\) and \(\beta =0.00366 \frac {N}{nN}\) are used, an angular displacement of 1 deg on the master side corresponds to a linear displacement of 0.027 μm on the slave side and a force of 0.00366 nN on the slave side corresponds to a force of 1 N on the master side. The objective of these experiments is to provide very fine motion on the slave side for a relatively larger displacement on the master side using α as a scaling factor. Then the corresponding interaction forces are scaled by β to enable the human operator to feel the forces in the micro world.

4.2.2 Experimental validation for force/position tracking

In order to validate the position tracking between the master and the slave, the commanded position from the master is transferred after necessary scaling to be tracked by the slave side [31]. Figure 10 illustrates the experimental results for position tracking along with the tracking error of the bilateral controller. It can be clearly seen that the slave tracks the master position with high accuracy. The position tracking performance is acceptable for precise positioning of the micro cantilever.
Fig. 10

Position Tracking between the master and the slave

In order to validate the force tracking, the slave forces encountered from the environment is being transferred to the master side after necessary scaling. Figure 11 demonstrates the force tracking between the master and slave along with the tracking error. It can be clearly observed that the master tracks the slave force precisely. Position tracking and force tracking results explicitly shows that the maximum tracking error is 3.33% and 2.85% respectively and these results proves that sufficient transparency between the master and the slave side is achieved to perform the micromanipulation task.
Fig. 11

Force tracking between the master and the slave

5 Semi-autonomous pushing scheme

5.1 Point contact pushing for translational motion

Precise positioning of micro-objects lying on a substrate using a point contact pushing to track a desired trajectory possess lots of challenges. The pusher or probe needs to be controlled in such a way to reorient and transport the micro-object to its final location using a stable pushing operation. Using only a point contact with a limited number of freedom the task of pushing on a horizontal plane can be realized. Pushing in micro domain is dominated by the varying frictional distribution which can be lumped at a single point as center of friction. Thus, the resulting line of pushing needs to be directed through center of friction to achieve translational motion [21].

5.2 Center of friction

In this subsection, it is shown mathematically that in order to achieve translational motion using point contact pushing the resultant line of action needs to pass through the location of the center of friction of the micro-object lying on a support surface. The technique of point contact pushing proposed by Yoshikawa [32] has been applied in this subsection. The method does require to hold some of the assumptions which are mentioned as follows:
  1. 1)

    The micro-object is rigid.

  2. 2)

    The micro-object is in contact with the supporting surface with n points. In this case n is chosen to be 4 (virtual supporting points), as the four corners of the rectangle.

  3. 3)

    The position of the supporting points with respect to the object remains unchanged even when the micro-object is in motion.

  4. 4)

    Since the micro-object is pushed by point contact, the friction between the pusher and micro-object is assumed to be negligible due to the fact that contact area is very small.

  5. 5)

    The coefficient of friction between the object and the support surface may depend on the position of the supporting point, but is constant with respect to time.

  6. 6)

    The pushing force is applied in y-axis (horizontally) at the base of the micro-object to avoid minimize the force being exerted in z-axis.

  7. 7)

    The inertial force developed by the micro-object can be ignored in comparison with the frictional force as the motion of the moving object is very slow.

Figure 12 represents the micro-object lying on the supporting surface. A reference coordinate frame ∑  u (O u  − X u Y u Z u ) is attached to the supporting surface. An object coordinate frame ∑  o (O o  − X o Y o Z o ) is also fixed to the object with its X 0 Y 0 plane coinciding the base of the object. Some of the notations expressed in ∑ 0 are defined as follows:
  • p i : Position of i th supporting point.

  • v i : Velocity of object relative to support surface at p i .

  • a i : Magnitude of frictional force at p i .

  • f i : Frictional force at p i .

  • f : Frictional force vector.

  • m i : Frictional moment at p i .

  • F f : Total frictional force.

  • M f : Total frictional moment.

  • F c : The pushing force applied by the probe.

  • p c : The location of the contact point with the micro-object.

  • p g : The location of the center of friction.

Fig. 12

Reference frame and object frame

All moments are defined with respect to the origin of the frame Σ0.

The frictional force f i and the frictional moment m i at the i th supporting point are given by, Eqs. 10 and 11, respectively:
$$ f_{i}=-\frac{v_{i}}{\big\|v_{i}\big\|}a_{i} \label{Eqn:fi} $$
$$ m_{i}=p_{i}\times-\frac{v_{i}}{\big\|v_{i}\big\|}a_{i} \label{Eqn:mi} $$
Where ||.|| and × denote the Euclidean norm and the cross product respectively. Thus, the total frictional F f and total frictional moment M f can be represented in Eqs. 12 and 13, respectively:
$$ F_{i}=\sum_{i=1}^n f_{i}=-\sum_{i=1}^{n}\frac{v_{i}}{\big\|v_{i}\big\|}a_{i} \label{Eqn:Ff} $$
$$ M_{\!f}=\sum_{i=1}^{n}m_{i}=-\sum_{i=1}^{n}\left\{p_{i}\times\frac{v_{i}}{\big\|v_{i}\big\|}a_{i}\right\} \label{Eqn:Mf} $$
As the micro-object rotates, the position of instantaneous center of rotation of the motion \(p_{r}=[x_{r},y_{r},0]^T\) can be deduced by using visual data. In Fig. 13 the origin of the reference frame is placed at the lower left vertex of the rectangle. The edges PQ and PQ are the two edges of the rectangular micro-object before and after pushing by a probe using point contact. The midpoints of the line PP and QQ are deduced and a perpendicular line is formed from both the midpoints. The point where the lines intersect is the instantaneous center of rotation referred as p r and the location is denoted as \([x_{r},y_{r},0]^T\).
Fig. 13

Instantaneous center of rotation

The unit vector which is along the direction of relative velocity at each supporting point \(p_{i}=[x_{i},y_{i},0]^T\) is denoted in Eq. 14.
$$ \frac{v_{i}}{\big\|v_{i}\big\|}=k\times\frac{p_{i}-p_{r}}{\big\|p_{i}-p_{r}\big\|} \label{Eqn:unit_vector} $$
Where k is the unit vector that is along the direction of the rotation of the object. Let the rotational angle of frame ∑ 0 with respect to ∑  u be θ. The unit vector k can be calculated as \([0,0,sgn(\dot\theta)]^T\). The value of k = [0,0, − 1] T when the direction of rotation is counterclockwise and k = [0,0,1] T for clockwise. The pushing force F c at the contact point p c  = [x c ,y c ,0] can be decomposed as \(F_{c}=[F_{cx},F_{cy},0]^T\) and with assumption 7, one can state the following relations;
$$ F_{c}=-F_{\!f} \label{Eqn:equal_force} $$
$$ M_{c}=-M_{\!f}=p_{c}\times F_{c} \label{Eqn:equal_moment} $$
where \(M_{c}=[0,0,M_{cz}]^T\) denotes the moment due to F c . The total friction force F f can be decomposed as \(F_{\!f}=[F_{fx},F_{fy},0]^T\), \(M_{\!f}=[0,0,M_{fz}]^T\). From the Eqs. 1216, one can obtain Eqs. 1719.
$$ sgn(\dot\theta)F_{cx}=-\sum_{i=1}^{n} \frac{Y_{i}}{R_{i}}a_{i}\label{Eqn:Fcx} $$
$$ sgn(\dot\theta)F_{cy}=\sum_{i=1}^{n} \frac{X_{i}}{R_{i}}a_{i}\label{Eqn:Fcy} $$
$$ sgn(\dot\theta)M_{cz}=\sum_{i=1}^{n}\frac{x_{i}X_{i}+y_{i}Y_{i}}{R_{i}}a_{i} \label{Eqn:Mcz} $$
Then, the value of X i , Y i and R i can be expressed as
$$ X_{i}=x_{i}-x_{r}, Y_{i}=y_{i}-y_{r}, R_{i}=\sqrt{X_{i}^2+Y_{i}^2} \label{Eqn:XiandYi} $$
Since a i is the magnitude of the frictional force at the supporting point (x i ,y i ), a frictional force vector can be formed as \(f=[a_{1},a_{2}....,a_{n}]^T\) for n supporting points. If the object moves without any rotation, the direction of the relative velocity \(\frac{v_{i}}{\|v_{i}\|}\) of all the supporting point are the same and can be written as
$$ e_{v}=\frac{v_{i}}{\big\|v_{i}\big\|}, (i=1,2...,n) \label{Eqn:ev} $$
Rewriting the value of F f and M f , one can derive
$$ F_{\!f}=-e_{v}\sum_{i=1}^n a_{i} \label{Eqn:Ff_1} $$
$$ M_{\!f}=-\left\{\sum_{i=1}^n p_{i}a_{i}\right\}\times e_{v} \label{Eqn:Mf_1} $$
The location of center of friction p g can be written as
$$ p_{g}=\frac{\sum_{i=1}^n a_{i}p_{i}}{\sum_{i=1}^n a_{i}} \label{Eqn:Pg} $$
By the definition of p g , Eq. 23 can be written as
$$ M_{\!f}=p_{g}\times F_{\!f} \label{Eqn:Mf_2} $$
Equation 25 indicates the total frictional force F f to cause a frictional moment of M f which acts on the objects at point p g . From Eqs. 15 and 22, the pushing force applied to the micro-object can be written as
$$ F_{c}=e_{v}\sum_{i=1}^n a_{i} \label{Eqn:Fc_2} $$
$$ \begin{array}{lll} M_{c}+M_{\!f}=0 \\ \overrightarrow{p_{c}}\times \overrightarrow{F_{c}}+\overrightarrow{p_{g}}+\overrightarrow{F_{\!f}}=0 \\ \big(\overrightarrow{p_{c}}-\overrightarrow{p_{g} \big)}\overrightarrow{F_{c}}=0 \\ \overrightarrow{p_{g}}=\overrightarrow{p_{c}} \label{Eqn:Pg_Pc} \end{array} $$
By analyzing Eqs. 26 and 27, it can be concluded that for a translational motion the direction of F c needs to be the same as the motion of the object and the line of action of F c needs to pass through p g . In other words, if one applies a external force F c acting on the object such that the line of action F c passes through the point p g , then it is possible to push the object without any rotation.

5.3 Method for online estimation of the center of friction

The location of the COF would vary depending upon the varying frictional distribution, thus online estimation of varying location of COF needs to be performed and the probe needs to be aligned so that the line of action of the applied force F c passes through the estimated COF.

Visual information is utilized to determine the position and velocities of the four corners, the centroid of mass for the rectangular micro-object along with the position of the contact point p c  = (x c ,y c ) with the probe. The instantaneous center of rotation p r and orientation angle θ are calculated using image processing techniques [23]. The force F c measured by the probe can be decomposed into two dimension as F cx  = F c cosθ and F yx  = F c sinθ, where θ is the orientation angle. The moment M cz generated by the applied force can be written as
$$ M_{cz}=x_{c}F_{cy}-y_{c}F_{cx} \label{Eqn:Mc_3} $$
The relationship between the pushing force F c and frictional force vector f which can be written as
$$ F_{c}=Gf \label{Eqn:Fc_3} $$
Where F c is calculated for each two consecutive captured frames and can be written as Eq. 30.
$$\begin{array}{lll} F_{c} &=& \big[sgn\big(\dot\theta_{1}\big)F_{cx1}, sgn\big(\dot\theta_{1}\big)F_{\!cy1},\\ && sgn\big(\dot\theta_{1}\big)M_{\!cz1},sgn\big(\dot\theta_{1}\big)F_{\!cx2}, \\ & & sgn\big(\dot\theta_{2}\big)F_{\!cy2},sgn\big(\dot\theta_{2}\big)M_{\!cz2}\big]^T \label{Eqn:F_vector} \end{array} $$
where F cx1, F cy1, M cz1 represent pushing force in x-axis for the first captured frame, pushing force in y-axis for the first captured frame and moment in the z-direction for the first captured frame respectively. Similarly F cx2, F cy2, M cz2 represents the values for the second captured frame. The value of G 4×6 matrix is calculated using two sets of consecutive captured frame and four supporting points considering the vertices of the rectangle. The G 4×6 is written as
$${ G= \left[ \begin{array}{cccc} -\dfrac{Y_{11}}{R_{11}} & \quad -\dfrac{Y_{21}}{R_{21}} & \quad-\dfrac{Y_{31}}{R_{31}} & \quad-\dfrac{Y_{41}}{R_{41}}\\[6pt] \dfrac{X_{11}}{R_{11}} & \quad\dfrac{X_{21}}{R_{21}} & \quad\dfrac{X_{31}}{R_{31}} & \quad\dfrac{X_{41}}{R_{41}}\\[6pt] \dfrac{x_{1}X_{11}+y_{1}Y_{11}}{R_{11}} & \quad\dfrac{x_{2}X_{21}+y_{2}Y_{21}}{R_{21}} & \quad\dfrac{x_{3}X_{31}+y_{3}Y_{31}}{R_{31}} & \quad\dfrac{x_{4}X_{41}+y_{4}Y_{41}}{R_{41}}\\[6pt] -\dfrac{Y_{12}}{R_{12}} & \quad-\dfrac{Y_{22}}{R_{22}} & \quad-\dfrac{Y_{32}}{R_{32}} & \quad-\dfrac{Y_{42}}{R_{42}} \\[6pt] \dfrac{X_{12}}{R_{12}} & \quad\dfrac{X_{22}}{R_{22}} & \quad\dfrac{X_{32}}{R_{32}} & \quad\dfrac{X_{42}}{R_{42}} \\[6pt] \dfrac{x_{1}X_{12}+y_{1}Y_{12}}{R_{12}} & \quad\dfrac{x_{2}X_{22}+y_{2}Y_{22}}{R_{22}} & \quad\dfrac{x_{3}X_{32}+y_{3}Y_{32}}{R_{32}} & \quad\dfrac{x_{4}X_{42}+y_{4}Y_{42}}{R_{42}} \\ \end{array} \right]} $$
From Eq. 30, an estimate value of \(\hat{f}\) can be derived as
$$ \hat{f}=G^{+}F \label{Eqn:f_estimated} $$
where G  +  is the pseudo-inverse matrix of G matrix. From Eq. 24, the estimated location of the center of friction \(\hat{p_{g}}\) can be obtained as
$$ \hat{p_{g}}=\frac{X^{T}\hat{f}}{e_{n}^{T}\hat{f}}=\frac{X^{T}G^{+}F}{e_{n}^{T}G^{+}F} \label{Eqn:pg_estimated} $$
where X T represents the location of each vertices of the rectangle and can be written in matrix form as
$$ X=\left[ \begin{array}{cccc} x_{1} & x_{2} & x_{3} & x_{4} \\ y_{1} & y_{2} & y_{3} & y_{4} \\ 0 & 0 & 0 & 0 \\ \end{array} \right]^T $$
where e n represents unity vector with four elements as
$$ e_{n}=[1,1,1,1]^T $$

5.4 Pushing algorithm

The pushing operation is performed in several steps as follows:

  • Step 1: Aligning the micro-cantilever such that the probe is in contact with micro-object at the midpoint of the length.

  • Step 2: Human operator starts to push the object using bilateral teleoperation and monitors the behavior of the object using visual display. Concurrently, the visual processing algorithm generate the position and velocities of vertexes of the rectangle and contact point of the probe and the micro-object.

  • Step 3: The data from visual processing algorithm is utilized to calculate the center of rotation p r and concurrently the force exerted F c by the probe is utilized to calculate F cx , F cy and M cz .

  • Step 4: The matrix F c and G are formed using two successive visual and force data sets. The force data is downsampled and averaged to 30 Hz to match the sampling rate of visual frame capturing.

  • Step 5: The value of the center of friction p g is estimated using the values obtained in Step 4 and thereafter desired value of the velocity of the probe in x-direction \(\vec{V_{\!x}}\) is calculated so that the vector of the resultant can be orientated to ensure that the line of action passes through the estimated center of friction.

  • Step 6: \(\vec{V_{\!x}}\) is set to the calculated value mentioned in Step 5 and kept constant until the arrival of new visual data.

  • Step 7: The human operator continuously monitors any sliding of the micro-object at the contact point which may occur if the probe comes out of the friction cone. In case of sliding, the human operator reverts and changes the location of the contact point by orientating the rotational stage to a proper value.

  • Step 8: Step 3 is repeated for the next visual data and the first three rows of G matrix are updated each time new data sets becomes available. Step 3 to Step 6 are repeated in a recursive manner to track the location of the center of friction.

Human operator is responsible for generating desired force for pushing of the micro-object by visualizing the motion of the micro-object and can pull the probe back if undesirable behavior in the motion of the micro-object is observed during any of the above mentioned steps. Since humans are very good at adapting to unexpected situations, the force controlled pushing operation is administered by the human operator.

6 Experimental validation of pushing operation

In order to validate the above mentioned pushing algorithm, several experiments were conducted by pushing a micro-object of rectangular form made from silicon of size \(200\:\mu m\) lying on glass slide. As one of the experiment, the micro-object is pushed by the probe at the mid-point of the length of rectangle and the line of action is made to pass through the center of mass. Figure 14 demonstrates the snapshot of the pushing operation and it can be clearly observed that after several consecutive steps the micro-object starts to rotate. Thus, it is unmanageable to translate a micro-object by pushing through the center of mass.
Fig. 14

Snapshot of pushing rectangular object at the mid-point of the rectangle and line of action passes through the center of mass of the micro-object

The above results shows that in order to achieve translation motion it is necessary that the line of action passes through the varying location of center of friction to compensate the orientation angle. Figure 15 demonstrates the snapshot of pushing rectangular micro-object such that the line of action passes through the varying center of friction. It can be seen that the micro-object suffers from orientation error and it is compensated in the consecutive frames to attain translational motion. Moreover XY-trajectory of the probe is shown in Fig. 16 and coherence between the X-trajectory controlled by the visual feedback and the orientation error from the visual snapshot is quite vivid. The maximum error of the X-trajectory of the pusher for the travel range of \(90\:\mu m\) in Y-axes is 2.55% is clearly visible. Figure 17 shows independently the position of Y-axis controlled by the human operator, X-axis controlled by the visual feedback and forces during pushing operation. It can be clearly seen that the proposed procedures is able to compensate the orientation effect in order to attain translational motion to perform pushing operation using point contact method.
Fig. 15

Snapshot of pushing rectangular object such that the line of action passes through the center of friction

Fig. 16

XY trajectory of the pusher during pushing operation

Fig. 17

Top, middle and bottom figure represents the position of probe in Y-axes, X-axes and the pushing force respectively

7 Conclusions

In this paper, a semi-autonomous scheme based on hybrid vision/force feedback using a custom built tele-micromanipulation is proposed. The pushing operation is undertaken by the human operator using visual display which acts an impedance controller and can switch between velocity control to force control by adjusting the stiffness (muscle stiffness) depending upon the behavior of the motion of the micro-object. Visual module provides the information about the position and orientation of the micro-object to calculate the time-varying COF (center of friction) in recursive manner for each captured frame. The velocity at the contact point od the probe and the micro-object is altered using visual feedback procedures such that the resultant direction of velocity passes through the COF to achieve pure translational motion. Experimental results concerning nano-newton resolution force sensing, force/position tracking between the master and the slave is presented which is a requirement to fulfill the pushing operation. Experimental validation of pushing micro-object of \(200\:mu m\) size to attain translational motion is presented and the maximum tracking error is seen to be 2.55% for the travel range of \(90\:\mu m\). Some of the issues that needs to be tackled for further research are as follows:
  • Piezoresistive AFM cantilever is presently utilized to sense force with nanonewton range only in one dimension, for dexterous tele-micromanipulation (to move the object in more complicated trajectory) it’s worth to investigate force sensing in 2 and 3 dimensions.

  • In the present context, position and force scaling is discussed in the context of scaled bilateral teleoperation, it will be worth to consider the frequency scaling from slave to master as meaningful high frequency vibration occurs in the micro environment which is beyond the frequency range of the human operator.

  • The present discussions are limited to pushing of micro-objects but using a micro-gripper with force feedback is necessary for 3D manipulation of micro-objects.


  1. 1.

    The probe or pusher is always in contact with the micro-object during the pushing operation.

  2. 2.

    Center of friction is defined as single point where the frictional distribution between the interface of object/substrate can be lumped.

  3. 3.

    friction cone is defined as to be the set of all wrenches satisfying Coulomb’s law for an object at rest, i.e. all the wrenches satisfying: |f t | ≤ μ|f n |, in other words if the pusher falls out of the angle 2tan − 1 μ, it would slide with respect to the micro object



The authors gratefully acknowledge the financial contributions by TUBITAK (The Scientific and Technological Research Council of Turkey), Ankara and MicroNed, The Netherlands.

Open Access

This article is distributed under the terms of the Creative Commons Attribution Noncommercial License which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.

Supplementary material

(AVI 34.5 mb).


  1. 1.
    Dechev N, Cleghorn WL, Mills JK (2003) Construction of 3d mems microcoil using sequential robotic microassembly operations. In: ASME international mechanical engineering congressGoogle Scholar
  2. 2.
    Furuta K (1999) Experimental processing and assembling system (microfactory). In: International micromachine symposium, pp 173–177Google Scholar
  3. 3.
    Popa DO, Stephanou HE (2004) Micro and meso scale robotic assembly. In: WTEC workshop: review of U.S. research in roboticsGoogle Scholar
  4. 4.
    Lynch KM, Mason MT (1996) Stable pushing: mechanics, controllability, and planning. Int J Rob Res 15(6):533–556CrossRefGoogle Scholar
  5. 5.
    Lynch KM (1999) Locally controllable manipulation by stable pushing. IEEE Trans Robot Autom 15(2):318–327CrossRefGoogle Scholar
  6. 6.
    Sitti M (2003) Atomic force microscope probe based controlled pushing for nano-tribological characterization. IEEE/ASME Trans Mechatron 8(3)Google Scholar
  7. 7.
    Ferreira A, Cassier C, Hirai S (2004) Automatic microassembly system assisted by vision servoing and virtual reality. IEEE/ASME Trans Mechatron 9(2):321–333CrossRefGoogle Scholar
  8. 8.
    Khan S, Elitas M, Kunt ED, Sabanovic A (2006) Discrete sliding mode control of piezo actuator in nano-scale range. In: IIEEE/ICIT international conference on industrial technologyGoogle Scholar
  9. 9.
    Sitti M, Hashimoto H (2003) Teleoperated touch feedback from the surfaces at the nanoscale: modeling and experiments. IEEE/ASME Trans Mechatron 8(1):287–298CrossRefGoogle Scholar
  10. 10.
    Tsuji T, Natori K, Ohnishi K (2004) A controller design method of bilateral control system. In: European power electronics power electronics and motion control conference, vol 4, pp 123–128Google Scholar
  11. 11.
    Torsten S, Fatikow S (2006) Real-time object tracking for the robot-based nanohandling in a scanning electron microscope. Journal of Micromechatronics 3(3–4):267–284Google Scholar
  12. 12.
    Lynch NA, Onal CD, Schuster E, Sitti M (2008) Vision-based feedback strategy for controlled pushing of microparticles. Journal of Micro-Nano Mechatronics 4(1–2):73–83CrossRefGoogle Scholar
  13. 13.
    Zesch W, Fearing R (1998) Alignment of microparts using force controlled pushing. In: SPIE conf. on micro robotics and micromanipulation, BostonGoogle Scholar
  14. 14.
    Onal CD (2005) Bilateral control—a sliding mode control approach. MS thesis, Sabanci UniversityGoogle Scholar
  15. 15.
    Shimono T, Katsura S, Ohnishi K (2007) Abstraction and reproduction of force sensation from real environment by bilateral control. IEEE Trans Ind Electron 54(2):907–918CrossRefGoogle Scholar
  16. 16.
    Katsura S, Matsumoto Y, Ohnishi K (2005) Realization of “law of action and reaction” by multilateral control. IEEE Trans Ind Electron 52(5):1196–1205CrossRefGoogle Scholar
  17. 17.
    Katsura S, Ohnishi K (2006) A realization of haptic training system by multilateral control. IEEE Trans Ind Electron 53(6):1935–1942CrossRefGoogle Scholar
  18. 18.
    Huijun L, Aiguo S (2007) Virtual-environment modeling and correction for force-reflecting teleoperation with time delay. IEEE Trans Ind Electron 54(2):1227–1233CrossRefGoogle Scholar
  19. 19.
    Anderson RJ, Spong MW (1989) Bilateral control of teleoperators with time delay. IEEE Trans Ind Electron 34(5):494–501MathSciNetGoogle Scholar
  20. 20.
    Mason MT (2001) Mechanics of robotics manipulation. MIT, CambridgeGoogle Scholar
  21. 21.
    Mason MT (1986) Mechanics and planning of manipulator pushing operations. Int J Robot Res 5(3):53–71CrossRefGoogle Scholar
  22. 22.
    Deng K, Ko WH (1992) A study of static friction between silicon and silicon compounds. Journal of Microengineering (2, ed.), pp 14–20Google Scholar
  23. 23.
    Khan S, Nergiz AO, Elitas M, Patoglu V, Sabanovic A (2007) A hybrid force-position controller based man-machine interface for manipulation of micro objects. In: IEEE/MHS international conference on micro-nano mechatronicsGoogle Scholar
  24. 24.
    Johnson KL (1997) A continum mechanics model of adhesion and friction in a single asperity contact: in micro-nanotribology and its applications. Kluwer Academic, DordrechtGoogle Scholar
  25. 25.
    Derjaguin BV, Muller VM, Toporov YUP (1975) Effect of contact deformations on the adhesion of particles. J Colloid Interface Sci 53(2):314–326CrossRefGoogle Scholar
  26. 26.
    Maugis D (1992) Adhesion of spheres: the jkr-dmt transition model using a dugdale model. J Colloid Int Sci 150:243–269CrossRefGoogle Scholar
  27. 27.
    Sitti M, Fearing R (2003) Synthetic gecko foot-hair micro-nanostructures as dry adhesives. J Adhes Sci Technol 17(8):1055–1073CrossRefGoogle Scholar
  28. 28.
    Rougeot P, Regnier S, Chaillet N (2005) Force analysis for micromanipulation. In: IEEE CIRA, pp 105–110Google Scholar
  29. 29.
    Hashtrudi-Zaad K, Salcudean SE (2002) Transparency in time-delayed systems and the effect of local force feedback for transparent teleoperation. IEEE Trans Robot Autom 18(1):108–114CrossRefGoogle Scholar
  30. 30.
    Fite K, Speich JE, Goldfarb M (2001) Transparency and stability robustness in two-channel bilateral telemanipulation. ASME J Dyn Syst Meas Control 123:400–407CrossRefGoogle Scholar
  31. 31.
    Khan S, Sabanovic A (2009) Scaled bilateral teleoperation using discrete time sliding mode controller. IEEE Trans Ind Electron 56(9)Google Scholar
  32. 32.
    Yoshikawa T, Kurisu M (1991) Identification of the center of friction from pushing an object by a mobile robot. In: IEEE/RSJ international workshop on intelligent robots and systems — IROSGoogle Scholar

Copyright information

© The Author(s) 2009

Open AccessThis is an open access article distributed under the terms of the Creative Commons Attribution Noncommercial License (, which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.

Authors and Affiliations

  1. 1.Department of Precision and Microsystems EngineeringDelft University of TechnologyDelftThe Netherlands
  2. 2.Faculty of Engineering and Natural Sciences, Mechatronics ProgrammeSabanci UniversityIstanbulTurkey

Personalised recommendations