Keywords

1 Introduction

Wrinkles in nature can be observed in various forms, offering intriguing insights into the complexity and beauty of the natural world. They can manifest in different scales, from the microscopic to the macroscopic, and are often the result of intricate processes and interactions. In biological systems, wrinkles can be observed in various organisms and they play their own pivotal role. For example, in the human brain, wrinkles allow for a greater density of neurons and enhanced cognitive abilities; or wrinkles on leaves of plants help reduce water loss and increase light absorption. Unfortunately, wrinkles appearing on fabrics and textiles do not have the same beneficial effects and are actually unwanted consequence of manipulation. In fact, the sewing of these materials can be challenging, as they tend to wrinkle easily, affecting the overall appearance and durability of the final product. In order to overcome this issue, flattening wrinkles during the sewing process is crucial. Therefore, in recent years, the use of robotics technology in the sewing process has received increased attention as a potential solution to this problem. Conventionally, different methods are adopted to flatten wrinkles in flexible materials during the sewing process, including the use of steam or heat, mechanical pressure, and manual smoothing. However, these methods are often labour-intensive and time-consuming. Conversely, robotics technology offers a promising solution to this problem, as it can automate the process of flattening wrinkles in flexible materials [1, 2]. The efforts to employ robots in the production line of soft materials like clothes started decades ago [3]. As one of the earliest attempts, Gershon investigated a parallel decomposition of robotic sewing tasks involving interaction with a dynamic environment [4]. They decomposed the sewing task into four concurrent processes including a vision-based contour tracker, tension sensing to measure the fabric tension, a feeding mechanism to pass the fabric under the needle, and the sewing machine. Vision-based techniques are amongst the most popular approaches adopted to identify wrinkles appearing on the surface of fabrics. Initially, such features have been identified by applying Gabor filters that label the abrupt changes in intensity at different orientations as wrinkles [5]. Similarly, Sun et al. initially developed an identification method employing the wavelet filters [6]. A few years later, Sun et al. modified their method by processing 2.5D depth maps [7]. Consequently, the height and width of the wrinkles were identified by geometrical means. It has been further refined to compute the surface curvatures in order to finally detect and quantify the wrinkles [8, 9].

In fabric handling tasks performed by robots, the end-effector plays a crucial role as it directly interacts with the fabric. The design, structure, and material of the end-effector significantly impact the grasping performance. Hence, it was essential to develop suitable end-effectors for robots specifically tailored to fabric handling applications [10]. Various designs have been extensively studied, which can be grouped mainly into three classes according to the adopted grasping technology: physical adsorption, adhesion and mechanical clamping. The design of grippers belonging to the first category includes vacuum suction, or electrostatic adhesion, and they are widely employed to pick a single sheet of fabric from a stack [11, 12]. Conversely, the adhesion-based grippers are characterised by the addition of an adhesive media, such as polyurethane [13, 14]. Finally, mechanical clamping grippers heavily rely on the friction between the fingertip of the gripper and the fabric [15, 16]. This solution allows for a high variety of designs, mainly made of rigid or soft materials [17,18,19].

In this paper, a fully automated approach to flatten wrinkles on fabrics manipulated in industrial plants is proposed. It requires an RGB camera, a hoop to hold the fabric in position, and a robotic arm equipped with a parallel jaw gripper with soft fingers. Grippers with compliant fingers are designed to mimic the adaptability and dexterity of biological systems, making them an ideal candidate for biomimetic and biohybrid applications [20]. The use of soft materials in gripper design has the potential to enhance gripping performance while reducing the risk of damage to delicate objects. The soft fingers of the gripper can conform to the shape of the object being grasped, thereby improving grip stability. In addition, the compliance of the fingers can be controlled to optimize the performance of the gripper for a given task. These features make soft grippers an attractive option for applications such as robotic surgery, prosthetics, and industrial automation [21]. It is possible to create more natural and intuitive interactions between machines and humans, by integrating soft grippers into biohybrid systems. Therefore, the development of grippers with soft fingers has significant potential to advance the field of biomimetics and biohybrid systems [22]. The design of the soft fingers adopted in this work has been thoroughly studied to achieve high-quality results. The shape of fingers, the material they are made of and the way they are employed take inspiration from human fingers. The design of this device is tailored to the task to fulfil and the elasticity of the tissue, as the fingers have to be soft enough to be compliant with the fabric, yet not too stiff to break or damage it. Having a bio-inspired design enables the development of the wrinkles removal technique, which requires first the identification of the wrinkles on the fabric. The step is implemented by performing instance segmentation on the images of the fabric constrained by the hoop. This step is necessary to prevent the gripper from dragging the fabric out of the working area and hence having no effects on the wrinkles. Subsequently, similarly to the manual process, the fingers are driven over the fabric, spread open and outward, consequently stretching the fabric. This routine has to be iteratively applied in order to flatten the whole surface, therefore the best strategy has to be derived. Consequently, it was implemented as a hierarchy scheduling algorithm that defines the ordered sequence of points on the fabric at which to iteratively apply the gripper and stretch the fabric. After every iteration, the wrinkles have to be identified, quantifying the number of pixels on the image corresponding to the wrinkles. The process is iterated until the amount of these labelled pixels drops below a threshold set after evaluating the corresponding manual operation. Such a methodology results to be highly flexible, as it is not dependent on any specific hardware characteristics and it just requires a set of images of the manipulated fabric. The presented work was developed as part of SOFTMANBOT, a cross-sectorial project funded by the European Union Horizon 2020 research and innovation program. The project aims to design a robotic cell to automate the manipulation and assembly of two textile components (i.e. a foam pad and an elastic fabric) used in the production of cycling garments at Decathlon industrial plant. One of the main problems to tackle by the project relies on the difficulties of handling flexible objects, which results in changes of shape, requiring continuous adjustments of the applied forces. Moreover, the manufacturing process of cyclist garments is further hindered by the layered structure of the soft materials under manipulation, having their elastic behaviours mutually affected [23].

The paper is divided into four sections: in Sect. 2 the proposed methodology and the developed software architecture are presented, Sect. 3 illustrates the results of the implementation and test of the experimental setup; and finally, a brief discussion of the results and suggestions for future works are presented in Sect. 4.

2 Methodology

The present section describes the developed methodology to tackle the flattening of the wrinkles on soft materials, with a focus on the design of the fingers and the software implementation. In particular, the methodology was tailored on a fabric composed of polyamid (PA) and elastane (EA).

2.1 Soft Compliant Fingers

An essential part of this work concerns the design of a gripper with soft fingers capable of manipulating the fabric, by stretching it and consequently flattening the wrinkles. Conventionally, the operator has to hold a portion of fabric and stretch it before feeding it to the sewing needle. A schematic representation of the conventional methodology is reported in Fig. 1.

Fig. 1.
figure 1

Representation of the conventional wrinkle removal methodology. The operator grabs a small portion of the fabric, stretches it and feeds it under the sewing needle.

Full size image

However, such a methodology is highly complex to be replicated reliably by a robotic solution, as the operating space around the sewing needle is very limited. Hence, in order to develop a robotic solution, the wrinkles removal and the sewing processes had to be separated. The clamping mechanism holds firmly the fabric and the fingers are applied outside the hoop, preventing the creation of further wrinkles due to the spring-back effect of the fabric elasticity (Fig. 2).

Fig. 2.
figure 2

Representation of the wrinkle removal solution set at the base of the automated process. A hoop is employed to hold tightly the fabric and the fingers are applied outside the hoop to stretch the fabric and thus removing the wrinkles.

Full size image

Consequently, this design is highly suitable to be automated, having the robots flattening the wrinkles without any obstacle. The design of the fingers was inspired by the gestures usually performed by humans to stretch small surfaces. Two fingers of the same hands are applied across the wrinkle to flatten, and are then spread open (Fig. 3/0). Such a process was set at the base of the design of the soft fingers. Initially, multi-pegs fingers were evaluated, but unfortunately they required wide operational areas, and hence poorly replicated the human operation (Fig. 3/1–2). Consequently, the single-peg design was selected, and the performance of different models were evaluated, by varying the material in elasticity and softness (Fig. 3/3–6). The properties of the different designs are reported in Table 1.

Fig. 3.
figure 3

Gallery of the different designs of the soft fingers. On the left and labelled with 0, the motion performed by the hand inspiring the design of the fingers. The fingers are ordered from left to right according to increasing softness.

Full size image
Table 1. Technical properties of different designs of the soft fingers.
Full size table

The initial designs of the finger (Fig. 3.1 and Fig. 3.2) were comprised of four and two rigid fingers, which had to be installed in pairs on the parallel jaw of the gripper. The other models feature the same simplified design, consisting of a single prong, but with variable stiffness (Fig. 3.3, Fig. 3.4, Fig. 3.5, and Fig. 3.6). The first three prototypes were designed with a rigid structure to firmly pull the fabric, varying only the number of prongs and length, by controlling the force applied by the manipulator equipped with the gripper. Conversely, the last three finger designs were manufactured with the same material (i.e. Tango Black Plus), but with varying degrees of softness. This feature ensures deformation as soon as the fingers come into contact with rigid surfaces with the same applied force.

In order to test all the prototypes, the different designs were mounted on the wrist of the UR5 robot by using the compliance controller. This tool runs a ROS-controller that implements Forward Dynamics Compliance Control (FDCC) on a set of joints, i.e. a hybrid between force control and impedance control. This type of control is applied in the case the manipulator is in contact with rigid external surfaces, thus enabling the robot to follow surfaces, dampening the impacts and gaining collaborative behaviour. Such control technique was carefully selected as in the case of high normal force applied to the fabric, it may happen that a finger has to slide over the fabric. Conversely, an encompass coupling would grasp firmly the fabric and possibly damage the fabric in a similar scenario. Such a controller was kept constant and used to test the behaviour of the different designs of fingers, by evaluating their deformation, the grip on the fabric and the capability of removing wrinkles.

2.2 Wrinkles Removal

One of the direct consequences of handling soft materials is the creation of wrinkles. However, the presence of wrinkles hinders tissue production and lowers the quality of the final product. Consequently, one of the most challenging aspects of the automation of manufacturing processes concerns the ability to stretch fabrics, flatten wrinkles and prevent the creation of new ones. Therefore, a clamping mechanism consisting of two plastic circles housing magnets was designed and realised. The magnets ensure that the attractive force between the two hoops is strong enough to hold the fabric in place, without hindering its stretching. Moreover, the position of the hoops on the workbench is constrained on the workbench by a system of pull-up solenoids, which prevent any unwanted movements of the clamping mechanism and misalignments.

In this paper, we proposed an autonomous two-step iterative method to flatten the wrinkles, based on identification and stretching. The first stage is implemented through instance segmentation, a deep learning-based approach that can detect objects in images and demarcate their boundaries. The system employs Mask R-CNN [24] and has been trained to identify wrinkles and the centre of seams, i.e., the cross-section of pieces of fabric (Fig. 4).

Fig. 4.
figure 4

The instance segmentation technique was trained to detect two classes of features: the wrinkles on the fabric highlighted by the arrows, and the center of the seams, circled in red. (Color figure online)

Full size image

The developed approach was implemented via transfer learning, i.e., by using the pre-trained set of weights derived from the COCO dataset. The deep network was then trained for 200 epochs on a set of 350 different images, which were recorded by using a greyscale camera sensor installed on the robotic gripper. This method not only maps the pixels corresponding to the wrinkles, but also produces in output the coordinates, the area, the dimension and the orientation of the different wrinkles. The results of the identification system are then set in input to the wrinkle removal algorithm. Unfortunately, it may happen to detect wrinkles outside the region of interest (ROI) limited by the hoops of the clamping mechanism, and that hence do not affect the quality of the outcome. Therefore, a filtering process is first applied: the hoop clamping the fabric is identified in the raw image, its contour is extracted and used to filter out the wrinkles identified within the ROI (Fig. 5).

Fig. 5.
figure 5

Schematic representation of the filtering process applied to detect any possible wrinkle outside the ROI, delimited by the hoops of the clamping mechanism. The wrinkles identified out of the hoop are subsequently filtered out.

Full size image

After filtering the wrinkles, the ROI is split into eight radial sub-areas (Fig. 6), and the density of the wrinkles in each of them is computed. These values are fed in input to a dynamic hierarchy scheduling algorithm that assigns the highest priority to the areas containing the highest density value. Then, a specific point out of the clamping mechanism is assigned to each sub-area, thus defining the gripper application points.

Fig. 6.
figure 6

Schematic representation of the algorithm designed to derive the strategy to best flatten the wrinkles. The ROI limited by the hoop is split into eight radial sub-areas in order to compute the density of the wrinkles in each of them.

Full size image

The gripper equipped with the two soft fingers is then applied to the pre-defined grasping point, corresponding to the sub-area with the highest priority assigned by the dynamic hierarchy scheduling algorithm. Consequently, the gripper gently pulls and stretches the fabric outward, avoiding any damage to the fabric (Fig. 7).

The presence of the clamping mechanism guarantees that the fabric does not return to its original state after the manipulation is performed and the wrinkles are therefore removed correctly. The two-step methodology is iterated until the number of pixels labelling the wrinkles drops below a threshold, finely identified by analysing the conventional manually manufactured fabric. This value was selected not null, since small wrinkles are crucial for the high quality of the final product, as they prevent the garments from stretching too much and then breaking open when worn and deformed.

Fig. 7.
figure 7

Graphical representation of the wrinkle removal process. The gripper is first moved over the cloth, in order to take an image for the wrinkle identification (\(\hbox {t}=\hbox {t}_0\)). Once the wrinkles are identified, the density is estimated for the eight different regions, and the sub-area to apply the gripper is selected. The gripper is then lowered exerting pressure over the fabric in the pre-defined grasping point (\(\hbox {t}=\hbox {t}_1\)). Therefore, the gripper spreads the fingers open creating tension (\(\hbox {t}=\hbox {t}_2\)) and the end-effector pulls the fabric outward in order to flatten the wrinkles (\(\hbox {t}=\hbox {t}_3\)).

Full size image

3 Results

In this paper, we proposed an innovative design of soft fingers to be equipped on a robotic gripper, together with the methodology to automatically flatten wrinkles appearing on the surface of fabrics. This section describes an evaluation of the design process and the results obtained from the wrinkle removal technique.

3.1 Soft Compliant Fingers

Six different 3D-printed soft finger prototypes were designed and manufactured, differing in material, length, thickness and elasticity (Fig. 3). Various tissue manipulation routines were then performed to verify which of the prototypes was the most suitable for the application presented in this paper. The same robotic routine described in Fig. 7 was then performed with all the different finger types, to assess their behaviour. It became clear that prototypes with a high degree of deformation were more likely to hold the tissue, preventing it from slipping and escaping the grip. The deformation of the various prototypes responding to the same manipulator process is shown in Fig. 8.

Fig. 8.
figure 8

Gallery of the deformation experienced by the different designs of the soft fingers. Each prototype was mounted on the gripper and used to perform the same robotic routine. The softer the fingers, the higher the observed deformation.

Full size image

Regarding elasticity, the level of fabric damage was considered to be of primary importance. By increasing the force applied by the manipulator it was deduced that the softer the gripper fingers, the less damage the fabric suffered. In fact, testing the first three prototypes with greater stiffness showed that, especially after prolonged use, marks of manipulation and stretching remained evident on the fabric. Conversely, fingers that were too soft were not able to pull the fabric tight enough to remove creases cleanly. Consequently, the finger prototype realised with Tango Black Plus material with Shore A 60 was found to be the best candidate. Its unique features make it highly appropriate for fabric manipulation, as it offers a suitable level of flexibility without being overly rigid and hence damaging the fabric to stretch.

3.2 Wrinkles Removal

The identification of the best model of soft fingers enabled the tests of the wrinkle removal methodology. At the beginning of every iteration, the camera mounted on the wrist of the manipulator was positioned over the fabric to take an image and process it to detect the wrinkles and center of the stitches. The results of the identification process are depicted in Fig. 9.

Fig. 9.
figure 9

Schematic representation of the results of the detection. On the left, the wrinkles detected within the hoop delimiting the ROI are highlighted. On the right, the centre of the stitches is detected.

Full size image
Fig. 10.
figure 10

Schematic representation of the results of the wrinkles removal module. The top row depicts the original images, with a coloured mask highlighting the wrinkles. The bottom row highlights only the pixels assigned to the wrinkles. The different time points are obtained after applying the manipulation routine.

Full size image

The density of wrinkles identified in the ROI was thus computed for each sub-area. These values were then fed in input to the dynamic hierarchy scheduling algorithm, which prioritizes the selection of the sub-area to stretch based on the associated density values. Thus, the region with the highest density was selected for the first iteration, during which the soft fingers gripper stretches the fabric outward in order to mitigate the wrinkles in that area. After each iteration, the camera captured a new image to assess the density of wrinkles within the ROI once again, enabling the following step of the wrinkle removal process. This task is performed until the number of pixels labelling the wrinkles drops below a pre-determined threshold. In Fig. 10, it is possible to appreciate the results of five consecutive manipulation routines. Each pair of identification-manipulation tasks takes about 13 s, with the software run on a workstation with Intel Core i7 11700K at 3.60 GHz, 23 GB of RAM, and an NVidia QUADRO RTX A4000 of 16 GB as GPU.

It clearly emerges how the number of total wrinkles, and thus the quantity in each of the eight sub-regions, progressively decreases over the course of the iterations, without thereby affecting the quality of the final stitched product.

A video showing the results of here discussed automated tasks is available at https://youtu.be/c72TGHsyfOo.

4 Conclusions

Automating the industrial production of clothes is still a task far from being achieved, due to the complexity of robotic manipulation of soft objects, as fabric fold-over, wrinkles and uncertainty in localization are crucial issues, intrinsic to the material nonlinear mechanical behaviour. However, in this paper, we proposed a robotic system capable of removing wrinkles on the surface of fabrics. This system is based on the thorough design of soft compliant fingers equipped on a robotic gripper, and the wrinkle removal algorithm. The latter was conceived as an iterative methodology, alternating between the identification of wrinkles using a simple RGB camera and the robotic routine of stretching the fabric and then flattening the wrinkles.

A wide variety of soft fingers was evaluated, varying the design, the number of prongs, their size and stiffness. Rigid fingers were proven to have a high slip and to damage and scratch the fabric in the case of prolonged use. Conversely, highly flexible materials ensure a high adhesion between the fingers and the surface, allowing the fabric to be stretched tightly. Consequently, the designed soft fingers played a key role in the wrinkle removal methodology. The system thus developed is characterised by great flexibility, as it does not depend on any specific hardware requirements. Furthermore, the developed vision system can be applied to a wide variety of scenarios, as it has to be trained on a gallery of images of the tissue to be manipulated. The soft finger material was carefully adjusted to the elasticity of the fabric. However, different materials can be investigated to produce fingers that can best stretch other types of fabric. In addition, the fingers were 3D printed, a common and cost-effective manufacturing process that facilitates the design of the best finger implementation. Lastly, the developed pieces of software are computationally efficient, hence do not require powerful workstations to run.

The robotic solution presented in this paper was proven to be highly effective, flexible and easy to deploy. Therefore, it may be implemented in different scenarios, contributing to the automation of manufacturing processes that are labour-intensive and require a high level of accuracy. Consequently, the adoption of such autonomous systems can contribute to increased productivity, while improving the welfare of the workers.