Journal of Micro-Nano Mechatronics

, Volume 4, Issue 1, pp 73–83

Vision-based feedback strategy for controlled pushing of microparticles

  • Nicholas A. Lynch
  • Cagdas D. Onal
  • Eugenio Schuster
  • Metin Sitti
Research Paper

DOI: 10.1007/s12213-008-0008-8

Cite this article as:
Lynch, N.A., Onal, C.D., Schuster, E. et al. J. Micro-Nano Mech. (2008) 4: 73. doi:10.1007/s12213-008-0008-8

Abstract

A strategy for controlled pushing is presented for microassembly of 4.5μm polystyrene particles on a flat glass substrate using an atomic force microscope probe tip. Real-time vision based feedback from a CCD camera mounted to a high resolution optical microscope is used to track particle positions relative to the tip and target position. Tip–particle system is modeled in 2D as a non-holonomic differential drive robot. Effectiveness of the controller is demonstrated through experiments performed using a single goal position as well as linking a series of target positions to form a single complex trajectory. Cell decomposition and wavefront expansion algorithms are implemented to autonomously locate a navigable path to a specified target position. Control strategy alleviates problem of slipping and spinning during pushing.

Keywords

Micro-particle manipulation Micro-assembly automation Vision-based feedback 

1 Introduction

Manipulation and assembly at the microscale and nanoscale has been an area of considerable research over the past several years. The overall goal is to form specific functional materials, devices, and systems through the positioning of individual micro/nanoscale materials. The results of this research have a wide range of applications stretching from MEMS and computer data storage to biotechnology and materials fabrication.

Critical to the advancement of microassembly is the development of autonomous capability at 2D and 3D scales. Basic modes of manipulation include pushing, pulling, and pick and place. Several positioning strategies have been proposed using contact as well as indirect noncontact methods. Direct manipulation has been achieved through pushing or pulling using a nanoprobe [1], [2], in addition to pick and place with a nanoscale gripper [3, 4]. Noncontact methods have utilized the relatively high attractive and repulsive forces at the micro/nanoscale. These techniques include optical tweezers [5] and dielectrophoresis using nanoelectrodes [6, 7, 8]. Though each method has successfully demonstrated micro/nanomanipulation capability, they lack a fully automated process.

One of the promising tools for manipulation is the atomic force microscope (AFM). The AFM is a subclass of the scanning probe microscope family featuring a sharp tip mounted to a microcantilever and is typically used as an imaging tool. During an AFM scan the deflections of the microcantilever due to changes in the surface topology are recorded and assembled to form an image of the sample. As a manipulation tool, the AFM probe tip can be used to directly contact and position individual particles.

Prior research has demonstrated the ability to position individual particles with an AFM through both pushing and pulling with sub-micrometer precision [9, 10, 11, 12, 13, 14]. In each of these experiments, nanoparticles or microparticles are deposited onto a glass slide mounted below the AFM tip. The dynamic mode, rolling, sliding or slip-stick, is a function of the size of the particle, the radius of the probe tip, the force applied to the particle, and the tip–particle contact angle [15, 16]. Particle positioning through pulling has been only achieved by selecting these parameters such that the lateral and vertical forces at the tip–particle interface exceed the maximum friction force and the adhesion force at the particle–substrate interface, respectively. However for most cases, adhesion forces at the particle–substrate interface are such that pulling cannot be achieved.

The general approach for past manipulation trials has been to define a formation for the particle, automatically or manually match each particle to a specific position, and then either push or pull particles one by one to their specified position. In moving each particle to their goal position the path consists of either a straight line or a series of connected straight lines regardless or whether pushing or pulling is used. Strong results have been achieved using pulling to position the particles [9]. However, as stated above, pulling is only a viable option for specific tip–particle combinations. Success has also been achieved through pushing, yet a common problem is that the particle will slip or spin about the tip [10, 13]. In these cases the program must be stopped and the tip repositioned behind the particle, significantly slowing down the manipulation procedure.

In this work an approach to controlled pushing at the microscale is presented. The primary motivation for this work is the need to eliminate sliding and spinning around the tip during manipulation. Pushing is the desired mode of manipulation as pulling can only be achieved for larger particles where larger adhesion forces are possible due to the large surface contact area at the tip–particle interface. As manipulations techniques are scaled down to the nanoscale only manipulation through pushing will be possible. It is with this in mind that the strategies presented in this paper focus upon controlled pushing.

Prior control strategies have been proposed that utilize force feedback to infer the position of the particle relative to the tip [17, 18]. AFM tip deflections are used to measure changes in the position of the particle as it is manipulated. The control strategy presented in this paper will rely on visual feedback to track the particle through time. Using this information, we are able to leverage control strategies developed for macroscale applications. Due to the limitations of existing AFMs, this strategy is only viable at the microscale. However, it is the goal that the feedback controller implemented in this work may form the foundation for future work towards a force feedback controller at the nanoscale.

The organization to this paper is as follows. In Section 2, the manipulation problem is presented along with a description of the experimental setup. This includes an overview of the model of the tip–particle interface, the critical assumptions, and a derivation of the kinematic model for the particle. In Section 3, the derivation of the controller is presented along with a discussion of its stability properties. In Section 4, the controller is implemented on the physical system and the results from a series of manipulation experiments are presented. Finally, the paper is concluded in Section 5 with a summary of results and a discussion of future work.

2 Problem definition

2.1 Experimental setup

The manipulation experiments presented in this paper were performed on the custom-built AFM setup in the NanoRobotics Laboratory at Carnegie Mellon University. This is the same setup used by [9, 10, 13]. The custom AFM is built around a high resolution optical microscope (Nikon Eclipse L200 with a 50× objective under darkfield lighting). Polystyrene (PS) particles of 4.5μm diameter are deposited onto the bottom of a transparent glass slide. This glass slide is mounted to a high precision XYZ piezoelectric stage. This stage has a range of 100μm along the x and y axes with ±10nm precision and a range of 15μm along the z axis with ±2nm precision. This stage receives position inputs from the host PC via D/A card and has its own feedback loop allowing quick response to control inputs. Mounted to a manual coarse positioning stage below the glass slide is a 20nm radius AFM probe tip (ULTRASHARP NC from Micromash Inc.). To close the feedback loop, a CCD color video camera is mounted on the optical microscope. Connected to the host PC via a framegrabber card, this camera provides updates at approximately 10–15Hz. The resolution of the captured image with respect to the fine-positioning stage is approximately 8pixels/μm. Figure 1 shows a simple diagram of the experimental setup.
Fig. 1

Simple diagram of the experimental setup [1]

The first step in the manipulation program is locating each of the particles as well as the AFM tip in the image plane. Images are processed individually within the manipulation program using the circle detection algorithm implemented in [9, 19]. Because of the spherical shape of the PS particles they appear as nearly perfect circles in the image frame. This algorithm uses two criteria to determine if an object in the image is a particle. First, each circle will possess pairs of opposing gradient vectors with opposite directions. Second, the slope of the line connecting the base points of these two vectors will closely match the slope of one of the vectors. This algorithm is able to reliably detect and track multiple particles in a given image without a significant amount of computation. One possible limitation, however, is that in extreme cases, where many of the gradient magnitudes and directions are disturbed, this algorithm might fail to detect a particle. The OpenCV library is used to handle all image processing tasks. Prior to manipulation, a calibration procedure is run to correlate measurements in the image frame to the motion of the piezoelectric stage. This procedure maps the image coordinates to the nanoscene using a sliding mode parameter estimator as defined in [9]. The details of the particle detection algorithm are given in Algorithm 1.

Typically, the Hough transform is utilized for the detection of circular shapes. However, the Hough transform is often slow and difficult to implement in real-time image processing. Given the controlled environment of the manipulation trials and the known geometry of the PS particles, the gradient detection algorithm is an adequate solution for particle detection. Additional methods have been proposed for robust detection, however improving upon these methods are not the focus of this work, but may be addressed in future research.

Algorithm 1: Particle detection
  1. 1.
    Calculate the gradient image.
    • Use the Sobel operator to find the x and y gradient images.

    • Calculate the magnitude and direction of the gradients at each pixel.

     
  2. 2.

    Threshold the image to find the maximum gradients.

     
  3. 3.

    Search for opposing gradient pairs directed along the same vector.

     
  4. 4.

    Implement a voting algorithm to avoid false positives.

     
  5. 5.

    Search for circles that most closely match the particle radius.

     
  6. 6.

    Ignore circles that are closer than a predefined threshold.

     

2.2 Tip–particle modeling

The first step of the design of the controller is modeling the tip–particle interaction and defining the particle kinematics. At the microscale and nanoscale, adhesion forces dominate. Therefore, gravitational and inertial forces are neglected. The forces at the particle–substrate interface are a function of the contact area (As) and the normal force (Fs) at the tip–particle interface. The contact area As can be found using the Johnson–Kendall–Roberts (JKR) contact mechanics model,
$$A_{{\text{s}}} = \pi {\left[ {\frac{{R_{{{\text{sp}}}} }}{{K_{{{\text{sp}}}} }}} \right]}{\left( {F_{{\text{s}}} + 3\pi R_{{{\text{sp}}}} \omega _{{{\text{sp}}}} + {\sqrt {6\pi R_{{{\text{sp}}}} \omega _{{{\text{sp}}}} F_{{\text{s}}} + {\left( {3\pi R_{{{\text{sp}}}} \omega _{{{\text{sp}}}} } \right)}^{2} } }} \right)}^{{\frac{2}{3}}} ,$$
(1)
where Ksp is the equivalent modulus of elasticity of the substrate and the particle, ωsp is the interaction free energy, and Rsp is the equivalent radius [20]. Using these parameters the condition for sliding with respect to the sliding force fs is defined
$$f_{\text{s}} > \mu F_{\text{s}} + \tau A_{\text{s}} .$$
(2)
Note that μ is the friction coefficient and τ is the shear strength of the surface in contact for sliding. Based upon the results presented in [15, 16] for the selected particle size and tip radius, the predicted mode of behavior for the particle is sliding.

Additionally, because these manipulation tasks are performed at relatively slow speeds, the system is assumed to be in quasi-static equilibrium and damping and inertial forces are neglected. Furthermore, because the tip used is relatively rigid and the tip radius is small relative to the radius of the particle, deflections of the tip during manipulation are neglected. Motion of the particle is restricted to the xy plane.

The interaction between tip and particle is depicted in Fig. 2. During manipulation, the spherical particle is pushed by a material point (tip) with velocity \(\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\rightharpoonup}$}} {v} _{\operatorname{tip} } \). In this case, the line defined by the contact point and the center of the spherical particle is tangent to the particle’s trajectory because the force applied to the particle is along this line, i.e.,
$$\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\rightharpoonup}$}} {v} _{part} = v_{\operatorname{part} } \hat x_\alpha = v_{\operatorname{part} } \cos \alpha \hat x + v_{\operatorname{part} } \sin \alpha \hat y = \dot x\hat x + \dot y\hat y.$$
(3)
The versor \(\hat x_\alpha \) lies in the direction of this line and defines an angle α with respect to the horizontal versor \(\hat x\). The velocity of the tip can be written as
$$\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\rightharpoonup}$}} {v} _{tip} = v_{\operatorname{tip} } \cos \beta \hat x_\alpha + v_{\operatorname{tip} } \sin \beta \hat y_\alpha = v_{\operatorname{tip} } \cos \left( {\alpha - \beta } \right)\hat x + v_{\operatorname{tip} } \sin \left( {\alpha - \beta } \right)\hat y = \dot x_{\operatorname{tip} } \hat x + \dot y_{\operatorname{tip} } \hat y,$$
(4)
where \(\beta \in \left[ { - \frac{\pi }{2},\frac{\pi }{2}} \right]\) is the angle between \(\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\rightharpoonup}$}} {v} _{\operatorname{tip} } \) and \(\hat x_\alpha \). We assume that the contact point between tip and particle has the same velocity of the tip pushing the particle. Therefore, we can write the velocity of the center of the particle as
$$\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\rightharpoonup}$}} {v} _{part} = \overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\rightharpoonup}$}} {v} _{\operatorname{tip} } + \bar \Omega \times \bar r_{\operatorname{tp} } .$$
(5)
Taking into account that \(\bar \Omega = \dot \alpha \hat z_\alpha = - \omega \hat z_\alpha \), where \(\hat z_\alpha = \hat x_\alpha \times \hat y_\alpha \), and \(\bar r_{tp} = R_p \hat x_\alpha \), we combine 3, 4 and 5 to obtain
$$v_{\operatorname{part} } = v_{\operatorname{tip} } \cos \beta ,$$
(6)
$$\omega = \frac{{v_{{\text{tip}}} \sin \beta }}{{R_{\text{p}} }}.$$
(7)
The input to the system is the tip velocity \(\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\rightharpoonup}$}} {v} _{\operatorname{tip} } \), which may be set by changing the magnitude of its components \(\dot x_{\operatorname{tip} } \) and \(\dot y_{\operatorname{tip} } \), or equivalently, its magnitude vtip and angle αβ, or just β for a given α. The model defined above for the particle kinematics can be summarized as
$$ {\left[ {\begin{array}{*{20}c} {{\ifmmode\expandafter\dot\else\expandafter\.\fi{x}}} \\ {{\ifmmode\expandafter\dot\else\expandafter\.\fi{y}}} \\ {{\ifmmode\expandafter\dot\else\expandafter\.\fi{\alpha }}} \\ \end{array} } \right]}_{{{\text{part}}}} = {\left[ {\begin{array}{*{20}c} {{\cos {\left( \alpha \right)}}} & {0} \\ {{\sin {\left( \alpha \right)}}} & {0} \\ {0} & {{ - 1}} \\ \end{array} } \right]}{\left[ {\begin{array}{*{20}c} {{v_{{{\text{part}}}} }} \\ {\omega } \\ \end{array} } \right]}. $$
(8)
Using the kinematic model defined above, a controller is defined in the next section. The controller regulates the translational velocity vpart and the rotational velocity ω of the particle by modifying the magnitude vtip and the orientation β of the tip velocity vector \(\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\rightharpoonup}$}} {v} _{\operatorname{tip} } \) as dictated by 6 and 7. It should be noted that though the controller model is derived assuming that the tip moves to position particles, the tip actually remains stationary. As noted in the description of the experimental setup, the piezoelectric stage, to which the slide containing the deposited particles is mounted, is moved relative to the tip. Despite the discrepancy, the relative motion between the tip and the microparticles remains the same and only requires a sign change for implementation.
Fig. 2

Schematic of tip–particle interaction model

3 Controller design

In Section 2 the kinematic Eq. 8 for the system were defined. In this section the controller to be used will be outlined along with a discussion of its stability properties. As discussed earlier, the goal is to design a controller that can drive a selected particle to a predefined target position with pushing as the only mode of motion. The model for the particle driven by the tip presented earlier mirrors a simple model for a non-holonomic differential drive vehicle. Feedback control and motion planning for non-holonomic vehicles has been an area of considerable study [21, 22, 23]. Therefore, given the similarity of the kinematics models, many of the control strategies and algorithms in the macro world can be scaled down for use in micromanipulation and nanomanipulation.

Consider the situation shown in Fig. 3. A PS particle is located in a given plane with initial position [x, y, α]T. Note that the orientation of the tip–particle system is defined by the position of the tip relative to the center of the particle. A goal position, [xg, yg]T, is predefined. The goal of the controller is to drive the particle to the goal position by controlling the velocity vpart and the rotation ω. The kinematic model in 8 can be transformed to polar coordinates by
$$\rho = \sqrt {\left( {x_{\text{g}} - x} \right)^2 + \left( {y_{\text{g}} - y} \right)^2 } ,$$
(9)
$$\gamma = - \alpha + \arctan \left( {\frac{{\left( {y_{\text{g}} - y} \right)}}{{\left( {x_g - x} \right)}}} \right).$$
(10)
Applying the above equations to 8 results in the polar system
$$\left[ {\matrix {\mathop \rho \limits^ \cdot } \\ {\mathop \gamma \limits^ \cdot } \ } \right]_{{\text{part}}} = \left[ {\matrix { - \cos \gamma } \\ {\frac{{\sin \gamma }}{\rho }} \ \matrix 0 \\ 1 \ } \right]\left[ {\matrix {v_{{\text{part}}} } \\ \omega \ } \right].$$
(11)
The next step is the design of the control signals vpart and ω. With this purpose we define the proportional control laws,
$$v_{part} = k_\rho \rho ,\quad \omega = - k_\gamma \gamma ,$$
(12)
where \(k_\rho >0\) and \(k_\gamma >0\) are positive constant gains.
Fig. 3

Schematic of controller model variables

Theorem

The control law 12, with\(k_\gamma >k_\rho \left( {{{1 + \bar \rho ^2 } \mathord{\left/ {\vphantom {{1 + \bar \rho ^2 } 2}} \right. \kern-\nulldelimiterspace} 2}} \right)\)makes the origin of system11asymptotically stable within the region of attraction defined by\(\rho \in \left[ {0,\bar \rho } \right]\)and\(\gamma \in \left[ { - \pi ,\pi } \right]\), with\(\bar \rho >0\)arbitrary.

Proof

Substituting the control law into the kinematic model 11 results in the closed-loop system described by
$$ {\left[ {\begin{array}{*{20}c} {{\ifmmode\expandafter\dot\else\expandafter\.\fi{\rho }}} \\ {{\ifmmode\expandafter\dot\else\expandafter\.\fi{\gamma }}} \\ \end{array} } \right]}_{{{\text{part}}}} = {\left[ {\begin{array}{*{20}c} {{ - k_{\rho } \rho \cos \gamma }} \\ {{k_{\rho } \sin \gamma - k_{\gamma } \gamma }} \\ \end{array} } \right]}. $$
(13)
We define the Lyapunov function candidate
$$V = \frac{1}{2}\rho ^2 + \frac{1}{2}\gamma ^2 .$$
(14)
The system is asymptotically stable [24] if it can be established that \(\dot V\left( {\rho ,\gamma } \right) <0\)\(\forall \left( {\rho ,\gamma } \right) \ne \left( {0,0} \right)\) within the region of attraction. Differentiating V with respect to time yields
$$\dot V = \rho \dot \rho + \gamma \dot \gamma .$$
(15)
Substituting the equations given in 13 into 15 yields
$$\dot V = - k_\rho \rho ^2 \cos \gamma + k_\rho \gamma \sin \gamma - k_\gamma \gamma ^2 .$$
(16)
Since \({{0 \leqslant \sin \gamma } \mathord{\left/ {\vphantom {{0 \leqslant \sin \gamma } {\gamma \leqslant 1}}} \right. \kern-\nulldelimiterspace} {\gamma \leqslant 1}} \Rightarrow \gamma \sin \gamma \leqslant \gamma ^2 \;\left( {\gamma \in \left[ { - \pi ,\pi } \right]} \right)\), we can write
$$\dot V \leqslant - k_\rho \rho ^2 \cos \gamma - \left( {k_\gamma - k_\rho } \right)\gamma ^2 $$
(17)
The derivative with respect to γ of the right hand side of 17 yields
$$- 2\left( {k_\gamma - k_\rho } \right)\gamma \left[ {1 - \frac{{k_\rho \rho ^2 }}{{2\left( {k_\gamma - k_\rho } \right)}}\frac{{\sin \gamma }}{\gamma }} \right].$$
(18)
We can note that this term will be zero only for γ = 0 since \({{0 \leqslant \sin \gamma } \mathord{\left/ {\vphantom {{0 \leqslant \sin \gamma } {\gamma \leqslant 1}}} \right. \kern-\nulldelimiterspace} {\gamma \leqslant 1}}\) and \(k_\gamma >k_\rho \left( {{{1 + \bar \rho ^2 } \mathord{\left/ {\vphantom {{1 + \bar \rho ^2 } 2}} \right. \kern-\nulldelimiterspace} 2}} \right)\) implies that \({{k_\rho \rho ^2 } \mathord{\left/ {\vphantom {{k_\rho \rho ^2 } {2\left( {k_\gamma - k_\rho } \right)}}} \right. \kern-\nulldelimiterspace} {2\left( {k_\gamma - k_\rho } \right)}} \leqslant {{k_\rho \bar \rho ^2 } \mathord{\left/ {\vphantom {{k_\rho \bar \rho ^2 } {2\left( {k_\gamma - k_\rho } \right)}}} \right. \kern-\nulldelimiterspace} {2\left( {k_\gamma - k_\rho } \right)}} <1\).

Therefore, the right hand side of 17 has a critical point (maximum) at γ = 0 for each ρ, equal to\( - k_\rho \rho ^2 \leqslant 0\), which implies that \(\dot V <0\,\;\forall \left( {\rho ,\gamma } \right) \ne \left( {0,0} \right)\) within the region of attraction. Therefore, the origin of system 13 is asymptotically stable.

The condition \(k_\gamma >k_\rho \left( {{{1 + \bar \rho ^2 } \mathord{\left/ {\vphantom {{1 + \bar \rho ^2 } 2}} \right. \kern-\nulldelimiterspace} 2}} \right)\) is sufficient but not necessary. Figure 4 shows \(\dot V\) as a function of \(\rho \in \left[ {0,\bar \rho = 10} \right]\) and \(\gamma \in \left[ { - \pi ,\pi } \right]\). Gains have been selected such that the surface \(\dot V\) is negative across our region of interest.
Fig. 4

Plot of Lyapunov surface (\(k_\rho = 0.1\) and \(k_\gamma = 10\))

4 Controller implementation

4.1 Single goal manipulation

In order to test the control strategy presented in Section 3, the algorithm was implemented on the custom AFM setup outlined in Section 2. As an initial test of the controller, the planned manipulation was simulated using MATLAB. Figure 5 shows a plot of the manipulation of a single particle to a target point located at the origin (0, 0). The initial position of the particle is given by the green triangle while the path of manipulation is indicated by the green x’s.
Fig. 5

Simulation of a single particle manipulation

As can been seen in simulated manipulation in Fig. 5, the action of the controller is to first reposition the tip behind the particle, correcting the error in γ. In the event that the particle begins to slip or slide about the tip, the orientation controller will correct this before the particle loses contact with the tip and manipulation must be restarted. The particle is then pushed along a straight line path to the goal position. The speed of manipulation is limited by the feedback rate of the camera and framegrabber. As noted earlier the update rate for the particle positions is only 10–15Hz.

For the first set of experiments, a single particle is moved in line with a second reference particle. The structure of the single point manipulation strategy is outlined in Algorithm 2. Figure 6a shows the configuration of the particle at the start of the algorithm for one of the experimental trials. The first step in the manipulation algorithm is to detect all of the particles within the image plane using the gradient detection algorithm. In the next step, the user selects the particle to be manipulated, locates the AFM probe tip, and then defines the target position for the particle. To minimize the amount of computation during the gradient detection algorithm a “region of interest” is formed around the particle. This region of interest is indicated by the box around the selected particle. Rather than search the entire image at each frame update, only this region is searched for the particle. After selecting the target position, particle is moved into contact with the tip. Once the particle comes into contact with the tip, the controller is turned on and run until the particle reaches the target position. The particle is released from the tip by lifting the slide in the z-direction and the tip is returned to its initial position. The final configuration of the particle is shown in Fig. 6b.
Fig. 6

a Single target point—before manipulation; b single target point—after manipulation

Algorithm 2: Single particle manipulation
  1. 1:

    Take a frame.

     
  2. 2:

    Detect all particles.

     
  3. 3:

    Select the particle to be manipulated/

     
  4. 4:

    Select the target position.

     
  5. 5:

    Select the tip location.

     
  6. 6:

    Move tip into contact with particle

     
  7. 7:

    while distance to goal > particle radius.

     
  8. 8:

     Update error measurements.

     
  9. 9:

     Apply control law.

     
  10. 10:

     Send output commands to positioning stage.

     
  11. 11:

    end

     
  12. 12:

    Lift tip to release particle.

     
  13. 13:

    Return stage to initial position.

     
This control strategy was repeated across several trials. Pushing to a single target point was achieved in a single manipulation without restarting and without loss of contact with the particle. To illustrate the corrective action of the controller when the particle begins to spin about the tip, refer to Fig. 7. Shown is the orientation control log for a trial where particle begins to spin on the tip. Spinning occurs when the tip and particle are not aligned along the same trajectory. On the output plot, it is evidenced by an increasing error in γ. As a response, control signal ω is increased until the tip and particle are realigned.
Fig. 7

Experimental control output log

As can be seen, the controller is able to correct the orientation of the particle when it begins to spin. It should also be noted that the noise that can be seen in the controller log is due to the uncertainty in the gradient based detection algorithm. As the particle is manipulated, changes in illumination and reflection result in a 1–2pixel (0.3–0.6μm) uncertainty in the position of the particle. This often leads to actions that are unnecessary or over corrective. This is also problematic as the particle approaches the target position. Near the target, small changes in the particle position can result in large spikes in the orientation error.

4.2 Multi-point manipulation

Above, it was shown that autonomous manipulation of a particle to a single unobstructed target point could be achieved. However a more complicated problem is to manipulate the particle to an obstructed point by defining a series of goal points that takes the particle around any obstacle. In this second experiment the user defines a series of goal points that can be linked together by unobstructed paths to reach a final target point. The algorithm used to manipulate a single particle along a multi-point path is outlined in Algorithm 3.

Figure 8a shows such a complex path to an obstructed target point. Each dot represents an intermediate target point to the final goal. Linking these points forms the manipulation path. For demonstration purposes this path is exaggerated such that it will not come into contact with another particle. It has already been established that convergence to a single point is guaranteed, therefore if a series of points are evaluated sequentially, convergence to the final point in the series is also guaranteed. The final configuration is shown in Fig. 8b.
Fig. 8

a Multi-target path—before manipulation; b multi-target path—after manipulation

Algorithm 3: Multi-point manipulation
  1. 1:

    Take a frame.

     
  2. 2:

    Detect all particles.

     
  3. 3:

    Select the particle to be manipulated.

     
  4. 4:

    Select the manipulation path (m points).

     
  5. 5:

    Select the tip location.

     
  6. 6:

    forn = 1→m points

     
  7. 7:

     while distance to point > threshold.

     
  8. 8:

     Update error measurements.

     
  9. 9:

     Apply control law.

     
  10. 10:

     Send output commands to positioning stage.

     
  11. 11:

     end

     
  12. 12:

    end

     
  13. 13:

    Lift tip to release particle.

     
  14. 14:

    Return stage to initial position.

     

4.3 Automatic path planning and manipulation

In the previous section it was established that a complex trajectory to a target position can be defined by linking together a series of operator selected points. The next step is to develop a strategy to automatically identify obstructions in the particle’s path and automatically define a safe path that will avoid collision with other particles. This is another scenario where techniques from the macro-world can be applied to micromanipulation and nanomanipulation. Obstacle avoidance has been studied extensively for robotic applications [21]. As discussed earlier the particle has been effectively modeled as a non-holonomic wheeled vehicle. For this application, cell decomposition and implementation of the wavefront expansion or grassfire algorithm can effectively locate a navigable path [25]. More sophisticated search algorithms such a Dijkstra’s algorithm, Depth-first, or A* have been implemented for graph based searches [26]. However, because the focus is on demonstration of the control algorithm, wavefront expansion has been selected for simplicity.

The first step is to decompose the image frame into a 10 × 10μm cell grid. Grid size has been selected based upon the size of the microparticle. Limiting the number of cell will improve the speed of locating a navigable path for manipulation. Each cell is then checked to see if it is occupied by any part of one of the particle. Each occupied cell is labeled with an ‘x.’ Having located the cells that are occupied by neighboring particles, a path is identified along free cells by implementing a wavefront expansion algorithm (Fig. 9).
Fig. 9

Shortcut found along initial path

The goal cell is located and labeled with the number ‘0.’ The algorithm then marks each adjacent, unoccupied cell with the number ‘1.’ Each cell that has not been marked and is unoccupied and adjacent to ‘1’ is marked with a ‘2’. This process continues until the cell containing the selected particle is found. The number assigned to each cell corresponds to the number of steps from the goal. If the starting cell is not found then it is concluded that there is no unobstructed path to the specified goal cell. If a path to the goal exists, it can be found by starting at the selected particle and counting down to 0 while minimizing the Manhattan distance to the goal cell.

Though a path is found it is clear that this is not the most efficient way to reach the goal. To minimize the time needed to navigate to the goal, the path planning algorithm searches the path starting from the goal, checking each point to see if there is a shortcut between points. In this example a shortcut is located from start to 3 and then from 3 to the goal. Thus, the ten point path is reduced to a single intermediate point. The revised path is indicated by the dashed line in Fig. 9. The overall algorithm is given in Algorithm 4.

Algorithm 4: Automatic path planning
  1. 1:

    Take a frame.

     
  2. 2:

    Detect all particles.

     
  3. 3:

    Select the particle to be manipulated.

     
  4. 4:

    Select the target point.

     
  5. 5:

    Select the tip location.

     
  6. 6:

    Generate occupancy grid.

     
  7. 7:

    Check each grid for particle obstruction.

     
  8. 8:

    Number each from cell until start is found.

     
  9. 9:

    Count back to 0 to find path to goal (m points).

     
  10. 10:

    forn = 1→m points

     
  11. 11:

     while distance point > threshold.

     
  12. 12:

     Update error measurements.

     
  13. 13:

     Apply control law.

     
  14. 14:

     Send output commands to positioning stage.

     
  15. 15:

      end

     
  16. 16:

    end

     
  17. 17:

    Lift tip to release particle.

     
  18. 18:

    Return stage to initial position.

     
Adding this path planning algorithm to the previous manipulation program significantly increases its utility. Essentially an operator can select any particle in the image frame and move it to any target position for which a navigable path exists. To demonstrate this program’s effectiveness, several trials were run in which a particle was required to be navigated through a field of obstacle particles to a desired target position (Fig. 10a and b).
Fig. 10

a Automatic path planning—before manipulation; b automatic path planning—after manipulation

4.4 Shape formation

As a final demonstration of the particle manipulation strategy, a random configuration of PS particles in a particular region of interest on the glass slide will be assembled to form the Lehigh University ‘LU’ logo. The target shape is composed of 12 individual target positions (Fig. 11). Each position in the shape is filled according to a predefined order. To ensure that each particle position is accessible, positions toward the center of the formation are filled first. Particles are matched to each target position by measuring the distance to the target and then checking for obstructions (Fig. 12). If a direct unobstructed path is available, the target position is filled using the single target manipulation approach. If the direct path is obstructed, the automatic planning algorithm identifies a multi-point path to the goal position. Screenshots from an experimental trial can be seen in Fig. 13.
Fig. 11

Given ‘LU’ shape target positions

Fig. 12

Particle assignment results

Fig. 13

‘LU’ step by step assembly snapshots

Mixed success was achieved in the assembly of the particles. Increasing static electric forces between the tip and particle as well as the glass slide and particle over the series of manipulation resulted in difficulty releasing the particle at the desired target position. Imprecise final position allowed particles to come into contact with each other and the difficulty in separating the particles prevented any further correction. Furthermore, after the particles came into contact with each other, the particle detection algorithm was unable to detect those particles as obstructions for subsequent manipulations. Over time, degradation of the overall shape of the assembly can be seen in the screenshots taken after each particle manipulation. For future experiments, final positioning precision may be significantly enhanced by improving the particle release strategy.

5 Conclusion

In this paper a strategy for controlled pushing of 4.5μm PS particles has been presented. This controller utilizes visual feedback at 10–15Hz to track particle positions and control the input velocity and direction of the tip. With this controller, particle positioning was achieved with a precision of less than 3pixels (0.87μm). Particle manipulation was demonstrated for a single target point as well as on a multi-point path. Furthermore, use of the algorithms presented to manipulate the PS particle to a recognizable formation was shown.

The primary focus of this paper has been manipulation at the microscale. However in order to extend the ideas of this paper to the nanoscale additional study is required to account for assumptions that have been neglected. These include tip deflections, particle deformation, and inertial forces. Lack of real-time visual feedback in the nanoscale is also a problem to be accounted for. Future development will incorporate piezoresistive probes to provide force feedback to the controller. Additional study is also needed to develop a reliable particle release strategy. In the current approach, positioning accuracy is hindered due to difficulty in releasing the particle from the tip when the target has been reached.

Acknowledgments

This work was supported in part by a grant from the Commonwealth of Pennsylvania, Department of Community and Economic Development, through the Pennsylvania Infrastructure Technology Alliance (PITA), and partially by the NSF CAREER award program (Sitti, IIS-0448042).

Copyright information

© Springer-Verlag 2008

Authors and Affiliations

  • Nicholas A. Lynch
    • 1
  • Cagdas D. Onal
    • 3
  • Eugenio Schuster
    • 2
  • Metin Sitti
    • 3
  1. 1.Department of Electrical EngineeringLehigh UniversityBethlehemUSA
  2. 2.Department of Mechanical Engineering and MechanicsLehigh UniversityBethlehemUSA
  3. 3.Department of Mechanical EngineeringCarnegie Mellon UniversityPittsburghUSA

Personalised recommendations