1 Introduction

Bespoke shoes belong to the world of custom shoes and luxury footwear manufacturing that stand out for being a carefully crafted process. The so-called patina [1] is a personalization technique result of many steps for colouring and bleaching leather, through the generation of a process usually called in the industry as “patina growing” [2]. Due to the harsh effects that colour products have on leather, it is important to conclude the patina with a full leather care treatment to maintain its depth. The result is a soft sheen over the leather surface of the shoe, and it requires high skilfulness and dedication for achieving a great gleam. It could be fairly said that bespoke shoe production is more about art than industrial manufacturing [3, 4], as it can be seen in Fig. 1.

Fig. 1
figure 1

Shoes provided of a shiny handmade patina finishing (courtesy of Bespoke Factory Group)

Full size image

Massive shoe manufacturing is nowadays part of the factory industry. After discovering the potential of footwear mass customisation, traditional shoe producers have been integrating modern equipment and methodologies through information and communication technologies (ICT) to capture a much wider range of consumers, as described in preliminary studies [5]. However, in the fashion footwear industry, many operations are still handcrafted due to the high product variability and the need for ensuring the best product quality. A great number of sizes, colours, and leather qualities of footwear models are handled in complex manufacturing and assembly processes. In the particular case of bespoke shoemaking, which is grounded on the expertise of the artisan, the shining procedure consists of an iterative process of the addition of different proportions of water and wax, and specific drying times during several stages. In this context, the integration of automation-based solutions finds various limitations. Moreover, despite the evident benefit of modernizing the handmade approach to the customers using autonomous robotized systems, artisan businesses within the fashion footwear industry still offer some reluctancy to bringing automation to their production lines. The backbone of these small to medium artisan business is not always capable to invest large amounts of capital or willing to take risks that may interfere with the production flow [6].

The high added value shoe manufacturing is strongly characterized for giving the human touch to their products, to guarantee the excellent quality in the final result, which makes them very unique and elitist. This factor should be taken into account, and thus, the integration of a cooperative solution for the human–robot interaction becomes very meaningful when designing an automatic system for this kind of industry. This paper describes a practical approach for soft polishing in a collaborative environment with the craftsman and a robotic system.

1.1 Related works

Polishing processes can be encountered in many applications all across the industrial field, from car body polishing in the automotive industry [7] to mould manufacturing [8]. Several studies considering path planning [9], CAD systems, and force control [10, 11] have been developed with the aimed purpose of performing polishing tasks over complex geometry workpieces.

Footwear manufacturing is not exempt from specialized machinery or automated operations including robotic solutions [12][12] with a rising need for production monitoring [13]. In the last two decades, several projects for the footwear industry have emerged supported by the European Commission to transform a mass-produced product to a mass-customized one [14]. In 2010 arises the IDEA-Foot project aimed at the introduction of new methods for shoe standardization and the transfer of the geometrical information from the design to the production process in a digital standard data format [15].

Nevertheless, in the fashion footwear industry, most production is still mainly handcrafted, and short production runs are generally handled as required by customization. Due to the complexity of the manufacturing process and the importance of the final quality, few operations can be completely automatized. In this context, a group of robotic solution providers and research institutes, along with shoe manufacturers, formed in 2010 the ROBOFOOT consortium [16, 17]. It was conceived to promote the implementation of new manipulation strategies and devices for nonrigid parts and sensor-based robot programming and controlling tools through the introduction of smart robots. Within the scope of ROBOFOOT project, some of the initial results achieved [18] were the design and implementation of robotic cells that can combine roughing and gluing or inking and polishing processes. Concerning roughing processes or shoe surface treatments, several experimental results have been obtained through computer vision techniques [20, 21], cooperative robot control approaches [22. ], or specific control strategies [23].

Related to computer-aided design (CAD) information, previous studies have also provided robotic solutions for custom finishing operations [19] using CAD information for automatic generation, optimization, and validation of motion trajectories. Similar approaches in the scope of shoe manufacturing have been described in some studies in both senses: introducing cooperative robotics [24] and generating trajectories from CAD information and using a force-controlled approach [25].

In this work, the design of a collaborative environment using sensor-based robot force controlling for real-time adaptation of the trajectory and optimization with automatic tool orientation based on CAD/CAM systems have been addressed. This option has been chosen instead of other mechanical solutions like [71], more sophisticated but too expensive for some applications.

1.2 Objective of the research

The goal of this work is to give an automated functional and reliable solution to shoe patina application in the stage of the shining process. This task is performed by a 6DOF robot in collaboration with the craftsman, resulting in an improvement of the working conditions along with the productivity, and without disregarding the quality of the final product. The use of a collaborative robot allows an easy and intuitive way for manual path recording performed by the artisan while guiding the robot instead of generating the trajectory automatically using optimization algorithms like [72]. Following this purpose and based on [7] and [26], a novel and ergonomic cooperative tool design has been developed.

The polishing tool is intended to maintain the orthogonality with the shoe surfaces when performing the task. The use of CAD data is suggested to adjust the collaborative tool orientation to maintain this orthogonality.

The tool is covered with a soft rubber case to keep the application collaborative, and a leather-friendly material is used for the polishing task. Moreover, vision-based 3D shoe recognition and model matching is contemplated to ensure the correct location of the target shoe, avoiding potential error. The prototype demonstrates a productivity improvement in a real environment that enhances the artisan capabilities. The results demonstrate the feasibility of this approach since quality requirements are fulfilled without missing the valued “human-touch”. Nonetheless, the real novelty of the presented approach essentially lies in the mere fact that automation in the fashion footwear industry is still, to a certain extent, an unexplored field that needs innovative focuses to continue making technological progress. This work is focused on the shoe industry, but the same idea of industrializing patina generation could be extrapolated to other leather related industries (e.g. automotive industry, fashion, and complements).

The paper is structured as follows. Section 2 describes the design of the patina generation cell, including the process flow and the novel collaborative tool. Section 3 describes how the shoe is 3D located by the vision system. Section 4 describes how the trajectory is manually or semiautomatically generated. Section 5 shows the human–machine interface and how it works. Section 6 presents the experiential setup, its implementation in a real production line, and system validation tests. Section 6 shows usability tests performed based on NASA-TLX and SUS questionnaires. Finally, Sect. 7 summarizes the novelties presented in this work and its final conclusions.

2 Design of the system

The design and implementation of the systems is basically focused on two steps: the collaborative tool to apply patina on leather shoe surface and the robot setup where all elements required are integrated. Both of them are presented in the following subsections, and elements’ connections can be seen in Fig. 2, where physical devices (i.e. robot and controller, F/T sensor, cooperative tool, patina cream holder, 3D camera or depth sensor, HMI pad) and software developments (i.e. trajectory database, trajectory controller, force control similar to [70]) are depicted. Figure 2 shows that the robot has been inverted mounted to reach all required positions and orientations. Doing so, shoulder axis is just over the shoe, and access to the object is maximum.

Fig. 2
figure 2

Conceptual design of the robotic cell designed as a novel approach in the shoe industry

Full size image

In addition, the robot chosen in this project is not large to limit its acquisition cost, and therefore, it has a relatively small working area. On the other hand, features of the shoe require reaching certain positions with a fairly wide range of orientations. Therefore, the robot has been placed in an inverted configuration to make a better use of its working area, as the robot will be able to operate not only “around” it, but also in its “own vertical axis”. This is an advantage for the robot in inverted configuration over the traditional ground-attached configuration, which is a well-known fact [69]. However, although using the inverted configuration has the aforementioned advantage, it also has certain disadvantages, such as the greater load supported by the actuators or the need of a bulky and complex structure to attach the robot base, which has to support the weight of the robot, its forces, and accelerations.

2.1 The collaborative tool

Following the guideline described in [27] and previous experience of authors [7], the collaborative tool must gather several requirements: ergonomic, lightweight, collaborative, tool designed to avoid wrist robot singularities, easy installation and maintenance, appropriate for leather treatment, and not damped transmission (to avoid transient instabilities during the contact [73]).

The tool must be ergonomic and light for better handling and smoother path recording operations. For this purpose, a handlebar made in plastic is integrated into the tool with a pair of knobs where the buttons are placed at the height of the thumbs. Figure 3 shows an exploded view of tool elements meanwhile Fig. 4 shows a rendered view with elements description.

Fig. 3
figure 3

Exploded view of the collaborative tool showing: control buttons; stepper motor; soft cover/s; handlebar; and polyurethane sponge. a Perspective view and b front view of the tool

Full size image
Fig. 4
figure 4

Assembled and rendered collaborative tool with element description

Full size image

When considering a collaborative application, it is needed to ensure that any device is potentially harmful to the human being. This means that the tool should not contain any dangerous part exposed to the operators to guarantee that they work under safe conditions. Therefore, a rubber housing is designed to cover the metal pieces of the tool. This covering has been 3D printed using Filaflex material [28] with a 30% of density getting successful results.

The task of the robot and its position implies to reach some workpiece surface points close to wrist robot’s singularities. To avoid this singularity, the polishing tool has been designed turning its tool centre point (TCP) with 45 degrees inclination, following ideas presented in [29, 30]. The wrist singularity appears when the axis 4 and the axis 6 become aligned. The TCP would remain stationary, but the axis would move rapidly. With the proposed design, alignment of axis 4 and 6 is avoided and therefore the singularity also. In [31], the significance of singularities in the design and control of robot manipulators is described. This review tackles methods in robot kinematics that have been used in this applied research. This 45 degrees inclination design allows to tackle the application using an UR5e instead of a bigger and more expensive robot like an UR10e. Moreover, it increases ergonomics for the operator due to its relative position related to robot’s last link.

The collaborative tool contains a stepper motor that makes a soft sponge turn to generate the patina over the leather surface. The tool is provided with two push button as part of the path recording system. The input signal from the button is read by the robot to save the marked points by the user.

The first part of the tool is made in 3D printing with PLA plastic, which is in turn built in several pieces. The handlebar is connected directly to the core part as a single piece that counts with four holes to screw the whole tool to the end of the robotic arm. The knobs are bonded to the handlebar ends by a pair of screws and strong adhesive. The buttons are embedded in the knobs and fixed by a threaded ring (see Fig. 3 and Fig. 4). The wires pass through an internal hole of the handlebar up to the exterior wall of the core part and then are plugged into the robot connector.

Regarding the motorized system, the second part includes the stepper motor and the mechanical coupling devices, as well as the polishing tool. The casing is fixed through screws to a wedge-shaped piece also made of PLA plastic and screwed directly to the end of the robotic arm through the core part. The motor casing is built in aluminium, except for the critical parts that have to endure radial forces and mechanical stress, which are made of steel. For the same purpose, the motor coupling incorporates two bearings. The polishing tool consists of a polyurethane sponge attached by adhesive to a thin sheet of metal that works as a quick changeover (see Fig. 3). When the sponge service life has expired, it can be removed from the metal sheet, which can be cleaned with acetone.

Concerning motor selection, several aspects have been taken into account. For this application, the suitable working turning speed for the polishing tool ranges from 200 to 600 rpm. Attending to the motor torque in this particular case, the required torque is not very high. However, some considerations have to be made. On the one hand, the system needs a minimum of motor torque to overcome the friction forces to which the polishing tool is exposed. On the other hand, the motor weight increases considerably when looking at robustness with a higher torque motor. The tool weight hampers the smoothness of the path recording functionality. Therefore, there is a trade-off between torque and weight. The selected motor belongs to the line of Nema23, and the model reference is SY42STH47-1684. It has a maximum torque value of 18.9 kg·cm and an approximate weight of 1 kg. The overall weight of the tool is 2.90 kg.

The tool is mounted at the end of the robotic arm through four screws. The maintenance of the sponge has to be done after each shift or working day. The residual wax needs to be removed from the sponge if the robot has finished the task. The disassembly and reassembly of the sponge are fast and straightforward using a clamping bolt. When the sponge has reached its service life, it must be removed from the quick changeover.

The contact with the shoe surface materials considered was cotton lining and polyurethane foam. Cotton lining is currently used by the craftsman when shining the footwear. Polyurethane foams are usually provided to the customer along with the product for the same purpose. The use of polyurethane foam in the makeup market is widespread also. Different makeup foams or sponges were tested presenting excellent results. Moreover, the shape that many of these already manufactured sponges have is ideal for creamy product application and in special, for respectfully treating the leather. Also, because of the properties of the selected polishing material, the system can absorb vibrations and adapt better to the surfaces of complex geometry work pieces, while the fine control of the value of pressure between the tool and the leather becomes less critical.

Figure 5 shows the implementation of the collaborative tool attached to the end effector of a robot in an industrial environment. The system is currently installed in Bespoke Factory Group production plant in Almansa, Spain.

Fig. 5
figure 5

Views of the collaborative tool attached to the end effector of a robot in an industrial environment. Video available at: https://youtu.be/ghvTi7LfTFI

Full size image

2.2 The robotic cell

The collaborative robot that is used in this work is an UR5e from Universal Robots.

This 6-axis robotic arm incorporates a built-in constant contact force control, necessary for the tool surface adaptation and the correct polishing process accomplishment. This research is not implementing a low-level force control of the end effector but using robot’s manufacturer force control options. The robot has been selected due to strategic reasons of the costumer. Collaborators of the company are already users of universal robots, and the use of the same platforms makes easier maintenance, development, and modification tasks.

After analysing the application and considering robot’s reach, size of the shoe, and the human–robot interaction, it was decided to place the robot upside down, attached to a support structure. Subsequently, the location and use of the rest of the elements related to the process is decided. Figure 6 shows part of the system layout, where the mounted on ceiling robot, the collaborative tool, and the shoe stand are shown. The robot must spread the wax with a motor-driven sponge over the shoe surface while the work piece is at a fixed position on a stand. The shoe is placed on an inclined and height-adjustable structure that is accessible for the robot and the operator. This structure has been inclined around 30 degrees (value of α shown in Fig. 6) to allow the tool to reach the whole shoe upper. The inclination is required due to the surface features of the heel and the toe.

Fig. 6
figure 6

Partial system layout. Collaborative robot, polishing tool, and shoe stand

Full size image

The cell also includes an electric panel located next to the robot that hosts the logic programmable controller (PLC) in charge of motor-driven system actuation. The PLC uses two digital inputs to set the direction of the motor spin through a three-position switch. The motor speed can be regulated through an integrated analogue input of the PLC coming from a potentiometer. An output pulse signal is generated to control the stepper motor of the patina tool.

The shoe rests on a flat tray with a variable inclination and positioned on a vertical metal structure designed to hold the work piece. The stand can be adjusted in height and comes with a heated steel platter of easy positioning around the column of the structure, where the wax is placed. A temperature controller is mounted next to the wax platter with an on–off switch.

Figure 7a shows the CAD representation of the shoe holder, and Fig. 7b shows its implementation. The heated plate allows maintaining the wax at a stable temperature during colder seasons. The user can choose the set point temperature from the controller screen to work with the ideal wax texture. To achieve an adequate base is necessary to have certain control over the amount of wax to apply. Wax temperature is directly correlated with its texture, the applied force, and the sponge turning speed. The shoe stand also comes with a manual clamping system for comfortable mounting and gripping.

Fig. 7
figure 7

CAD representation of the shoe stand and implementation of the system

Full size image

The design of the robotic cell has been made considering the operations that must be performed by the robot and by the operator. These operations are described as follows:

  1. (1)

    Given a shoe model, the operator records the trajectories over the surfaces. To this effect, the operator moves the robot through the tool handlebar using a software function called FreeDrive that lets the robot move in “sensitive” motion, that is, with hardly any resistance. This FreeDrive function allows the robot arm to be manually pulled into desired positions and/or poses, resulting in a smoot robot motion that facilitates the user path recording. Path recording is done in three steps: frontal, left, and right. The flexibility for recording each path lets the operator decide the areas to be treated. It can be avoided, for instance, those parts with straps, buckles, or decorative elements. Then, saved paths are available for further optional modification and optimization.

  2. (2)

    Optimization functions for trajectories can be done off-line based on a CAD/CAM system. Loading the saved paths over the work piece surface, it is possible to readjust specific points to solve inaccuracies in the path specification or keep the desired orthogonality between the tool and the surface. Optimized points data set can be used for later program executions.

  3. (3)

    The GUI guides the operator in the definition of the process parameters, such as force values, speed, and other restrictions. Using computer vision, the system can recognize the presence of a shoe and its model. Once the shoe presence is verified, the robot can start the program execution.

  4. (4)

    The program starts with the robot approximation to the heated wax to get the sponge smeared. Consecutively, the robot proceeds to spread the wax addressing each part of the shoe, respectively. According to the process parameters that have been tuned by the operator, the wax application is performed several times on selected surfaces with defined force and speed. The axial force that the robot applies on the shoe surfaces is internally controlled to remain constant along the trajectories. The result is an appropriate layer on which the craftsman can work afterwards, saving him significant labour.

  5. (5)

    The development of a mirroring software function allows the robot to reproduce the same saved trajectories for the other side of the pair of shoes, resulting in time-saving. In other words, the system offers the possibility of repeating the polishing task in the symmetric pair of a shoe model, once the former is completed in the station. If the mirror function is selected, the robot will stop and wait until operator validation.

  6. (6)

    When the robot has finished the task, it follows the program flow defined by the operator and aims the next process or returns to home.

Once the robot path has been generated manually or semiautomatically, the robot performs the task autonomously. At any time, the operator can hold the collaborative tool to leave the preprogrammed trajectory and centre the patina pad on a specific area. This is done by pressing one of the buttons on the tool. Once the operator has finished the operation in the specific area, the robot can return to the path it was making at the point where it left off and continue until the job is finished. In this way, it is guaranteed that the entire surface of the shoe has been covered in the production process, avoiding leaving open pores in the material.

Due to the system configuration, the expertise of the craftsman is combined with automatic tasks that not only allow the worker to save time and physical effort but can also help unskilled operators with the robot movement programming. These are important working advantages that add value to the system.

Figures 8, 9, 10 and 11 shows the flowchart of the system and depicts the main behaviour of the robotized cell. Figure 12 shows a general view of the robot installation in the production line.

Fig. 8
figure 8

3D vision system: laboratory tests

Full size image
Fig. 9
figure 9

Detection and location in space of different shoe models with good lighting conditions, a and b, and with poor lighting conditions, c and d

Full size image
Fig. 10
figure 10

Methodology followed to design, implement, and validate the robotized patina finishing application

Full size image
Fig. 11
figure 11

shows the flowchart of the system and depicts the main behaviour of the robotized cell

Full size image
Fig. 12
figure 12

Prototype of functional robotized cell for patina application

Full size image

3 Computer vision

The appropriate task performance is strongly determined by adequate shoe position and orientation [24]. Following the previously defined paths over the surface like presented in [32] and [33], the robot moves with the polishing tool according to data provided in the demonstration step. The shoe is located in a stand equipped with a manual clamping system. Given the small and complex geometry of the workpiece, the high accuracy while addressing the predefined target points (teaching step) in the three-dimensional space is a critical aspect to be guaranteed. Therefore, the need for introducing a 3D computer vision technology for autonomous position and orientation verification is developed.

The proposed vision system consists of an Intel RealSense Depth D435 Camera [34, 35] installed 1300 mm higher and 700 mm in front of the shoe. The camera maps the environment to detect the presence of a specific shoe model and provide information about its position and orientation [36, 37]. The system can recognize the type of shoe and verify if it is placed in the right position or has slight displacements from the original location with a high level of accuracy [38]. The solution identifies the geometrical features of each type of shoe precisely, giving a model match according to the shoe model, size, and left/right foot [24]. The presented approach of model matching and shoe position recognition provides to the system a validation before the robot program execution.

3.1 Shoe recognition and identification

One of the main features in computer vision inspection is the effect that the environmental light can cause on the material surface of the objects. In this research, an appropriate selection external lighting source is required to work in an open environment shared by operators [39]. The fashion footwear shows significant gleams and work pieces are individually shaped and curved, so the incident light is susceptible to reflections and shades. Moreover, the shoe geometry features must contrast with the background to get an evaluable image by the vision system, so the tray where the shoe is placed must have a matte finishing. To design an effective and robust illumination solution, a dome illumination [40], featured by being free of reflections, has been used. Figure 8 and Fig. 9 show the 3D vision system and the whole set-up at CFZ Cobots lab. In this case, the camera, the robot, and the shoe can be seen. Figure 9 shows the detection of both shoes (left and right) with its visual features marked by colour. The computer vision package has been developed using ROS (robot operating system) [41]. The LiDAR camera connection is done through a node definition, where the footwear location data within the workspace is obtained after scanning the environment. A special library called ObjectNet3D [42] has been used to perform the object recognition. ObjectNet3D has been selected after comparing it with other libraries: Principal Axle Descriptor, G3DNet, and FusionNet. Based on accuracy comparation performed in [43], ObjectNet3D has been chosen.

The system allows the identification of relative position errors and identify the shoe model, size, and foot. The robot starts the program once it has received the verified data from the LiDAR camera. Otherwise, the robot raises a notification to the operator asking for further checks. The design of the computer vision solution is also used to improve the path definition and tool orientation to fulfil the application requirements. Through the introduction of CAD information, paths can be recalculated automatically while modifying the orientation of the points to have the polishing tool orthogonal to the shoe surface.

4 Operation description

Figure 10 summarizes the steps followed to design, implement, and evaluate the proposed patina growing system. First, a study about the application requirements has been conducted based on interviews with specialized workers. Secondly, a laboratory mock-up (see Fig. 8 for more information) has been designed and implemented, and finally, experts on robotics and production line workers have validated a factory version of the system (installation shown in Fig. 12).

Steps to operate the system are described in Fig. 11 flowchart. This flowchart shows the sequence to use the robot in normal operation, when data of the shoe to be polished is stored in the system. A mirror function can be selected to polish a left-shoe having only the trajectories of the same right-shoe model or vice versa. To grow patina over the leather shoe, the tool has to apply the wax in several layers on small concave and convex surfaces. When addressing complex geometries with the direct use of paths, the helpfulness of a contact force control that enables the sponge to adapt to the changing surfaces avoiding path inaccuracies becomes apparent. Furthermore, for the different sizes of the same model of shoe, the use of a controlled force allows the robot tool to realign with the new body contour and keep the same path plan saved for the model at issue. The force control is a robot’s built-in function that allows controlling the applied force in the direction of the tool Z-axis. The robot adjusts its position along the compliant axis to achieve the specified force.

4.1 Manual teaching

Generating trajectories by hand is an intuitive and easy way of saving paths over the object’s surface without the need for learning and using external complex software [45]. In some cases, it can produce greater results versus automatic path generation since they are based on the human experience [46]. The operator can selectively avoid areas that need any or special treat and plan a better trajectory for a specific model driven by its expertise. The robot function for path recording also lets the operator make quick and immediate corrections afterwards to improve the precision in the desired paths [47].

Although the advantages of using manual path recording over the workpiece are well known, the system would be susceptible to human errors since the suitability of the results is dependent on the operator level of experience [48]. It should be noted the importance of maintaining the tool closely perpendicular to the surface at every time during the path recording for an appropriate force control performance. The operator can eventually miss this consideration due to lack of skills or simply by mistake [49]. Additionally, an accurate completely automatic path generation based on machine learning technics could be by far much efficient. In this context, different approaches are described hereafter in which manual teaching is combined with automatic orientation and path generation.

4.2 Automatic orientation

The direct use of the polishing tool for saving the set of points over the surfaces gives an approximate visual notion of the spatial movements that the robot has to perform during the polishing task as it can be seen in previous works like [50]. Although tool position and orientation can be easily set up by hand, an optimization strategy is considered because of possible operator mistakes. Works like [51] show a deep review of automated industrial robot path planning for spray painting process, where the orientation of paint source is particularly relevant. The tool orientation can be readjusted using the data from the CAD model identified by the 3D camera to fit the desired angles for better force control. This orientation based on CAD is used in [52], where an automatic teaching for welding tasks is developed using a laser vision sensor. Likewise, some points can be modified or added off-line to avoid gaps between trajectories and improve the definition of the task. This approach is a simplification of [53], where a teaching-free robot system utilizing three-dimensional CAD product data is developed to perform welding operations.

4.3 Fully automated operation

Complete automatic path generation can be performed in simplified robots like in [54], where suboptimized trajectories can be obtained for 4 degrees of freedom mechanisms (excavators). An automatic collision-free trajectory generation for a 6 degrees of freedom robotic car painting is presented in [55] where the task is defined as collaborative but the implementation is proposed with industrial robots. The automatic orientation of the tool can be performed using the force/torque signal [56] or using the CAD model of the surface like in [57], where a CAD model is used to generate automatically the path of a laser cladding robot in additive manufacturing.

This work has been inspired in [58], where a robot-based flexible manufacturing with intelligent sensing is presented. The article presents a sensor-based concept to carry out short series production with robots. Intelligent sensing extends the previously proposed by the authors approach towards a flexible robotic production. In the approach proposed, trajectories can be drawn from a standard template with defined restrictions such as percentage of overlap, shape-related offsets, or turning radius. The possibility of introducing machine learning for estimated path proposals is also considered. Using data from a set of manual path recordings, the system could generate an appropriate trajectory projected over the object surface like [59].

5 Real implementation

The results of this project come from a contract signed between Bespoke Factory Group (BFG) enterprise and the authors of this article to design and implement the robot at its factory in Almansa (Spain). The company CFZ Cobots SL has collaborated in the development of the project providing software functions and development support. The system has been completely implemented and can be seen in Fig. 12. The robot has been positioned at the end of the production line, next to the manual patina section, which occupies an area of around 40 m2.

5.1 Human–machine interface

Given a shoe model, the operator places it manually and save the initial point for the first trajectory. The user interface (see Fig. 13a) shows how to save the starting point. Figure 13b shows the developed GUI, and Fig. 14 provides the operator with an example of the trajectory to be followed. The ending point must match with the starting point since the approximation position to address the surface is automatically calculated from the defined point. Then, the operator starts the path recording through an available software function that allows the user to start, stop, or cancel the path recording from the robot screen. The points over the surfaces are saved using a push button located in the knob of the tool. The operator will follow the same procedure for the separate three surfaces: front side, left side, and right side. Next, the operator can configure the parameters of the process, such as specific surface-related features like the tool translational speed, applied forces to surfaces, number of layers, or general functions like enabled surfaces to treat and mirror function activation. The user needs to verify the system before executing the program. The system verification consists of reproducing the resulting robot movements after automatic calculation from the user-defined points in slow motion, to prevent from possible collisions during the process. Once the program counts with the safety check, the operator can run the robot program. The graphical user interface (GUI) has been designed considering other industrial applications like automotive [60], furniture [61], and porcelain [62] to generate a general flowchart useful in other applications.

Fig. 13
figure 13

Development of the system GUI

Full size image
Fig. 14
figure 14

The GUI provides the user information of the starting point and an example of a trajectory to follow

Full size image

5.2 Validation tests

Current time process for handcrafted footwear varies a lot depending on the shoe model. Normally, the first shining layer takes the craftsman about 7 min on average. For achieving an appropriate result, one shoe needs between two and four layers of treatment, with approximately 1 h of drying time between layer application. Generally, the craftsman deals in first place with the initial layers of the entire amount of shoes for daily delivery. After the initial layer application, the former shoes have completed their drying time and the craftsman starts the next round. The production volume is around 7 or 10 pair of shoes each day. In particular seasons, the demand exceeds the capacity production, and the employees are required to work extra hours to handle the number of orders.

First layers are the briefest since the finishing stages need more dedication and finest performance. Typically, the first shining layer application takes one worker at least 3 h in total for the daily production volume. From this perspective, the robotic cell is formulated to undertake the initial layer development. Further layers are left for the craftsman that gives the last finishing.

The system approach for the prototyped robotic cell is semiautonomous since the operator has to replace the shoe manually at the stand. Nevertheless, the application can occasionally work to complete further layer treatment under operator supervision in a collaborative performance. The average times for cycle time are directly related to the selected system speed and the number of repetitions configured for each trajectory. The allowed system speed has been customized inside a range of values that safeguards the quality of the finishing and the safety of the operator. For the fastest allowed speed and two repetitions for each trajectory, the system fulfils the quality specifications and takes 10 min to complete one shoe. That means that the system is able to practically duplicate the productivity in the first stages. Also, positive assessments from workers highlight that the physical effort that implies eventual production loads diminishes by using the application. These results are analysed in Sect. 7.

Figure 15 shows the information provided by a real experiment in the production line. After the operators’ first contact with the robot, graph of linear and angular tool speeds are almost the same for experienced and novel users. The first block of Fig. 15 (six plots) correspond to these tool speed information (user A). Then after, force and torque made by user A and user B can be seen. User A is a highly trained in the use of the robot operator, and user B is a little trained in the use of the robot operator. The figure contains two red rectangles showing the smaller force values for the case of user A, while for user B, the applied force is of higher magnitude and more oscillatory. Coloured vertical lines represent different stages of the polishing process. Areas marked in yellow show short duration events related to wax application (C), removal of wax excess (E), and task finishing (H). Areas marked in violet represent the “teaching” activity where the operator shows the robot how to polish a whole shoe. The first patina application area corresponds to the front area of the shoe (F), the second area corresponds to the right side of the shoe (D), and the third area corresponds to the left side of the shoe (I).

Fig. 15
figure 15

Real experiment of two users teaching the robot how to polish a shoe

Full size image

Two demonstration videos showing the application have been created. They show how the task of patina creation is manually performed, how the tool is initially introduced testing an actuator, and it is also shown lab tests and factory implementation. The system is currently being used at Bespoke Factory Group in Almansa, Spain.

Short version of the video: https://www.youtube.com/watch?v=oG_IbJJKNug

Long version of the video: https://youtu.be/jVJqBTfO4s4

6 Usability tests

In a similar way to other works [24], several usability tests have been used to validate a system by using user interviews. In particular, two standard questionnaires have been used in this project: the NASA task load index (NASA-TLX) [63] and the system usability scale (SUS) [64]. On the one hand, the NASA-TLX questionnaire is considered, as it is commonly used to assess digital and physical experiences in work environments. On the other hand, the SUS questionnaire is considered to assess the usability of the proposed approach, as it is concise and is considered an industry standard.

The system has been installed in the factory, and it has been tested by a group of 6 people. Two of them are patina operators of the company (BFG). One of them is an expert patina operator (male) with more than 15 years of experience in performing the task. The other patina operator (female) is a 3 years of experience full competence operator. On the other hand, four experts on engineering (without previous experience on patina handwork) have tested the system. Three of them have advanced knowledge in robotics (one female and two males). The fourth tester has any knowledge on robotics, but he (male) is a specialist in mechanical engineering. With this group of people, the aim is to validate the functioning of the system in terms of usability and execution. Figure 16 and Table 1 show the main features of participants involved in the usability tests, gender, age, patina skills, and education level.

Fig. 16
figure 16

Representation of participant features involved in the usability tests

Full size image
Table 1 Users that tested the system during the development
Full size table

A particularly relevant aspect when using SUS or other usability tests is the minimum acceptable sample size [65]. Technically only two users are needed to have some measure of variability (standard deviation) and to generate confidence intervals [66]. Such a small sample size is not normally used to analyse usability tests; however, reliable data can be extracted using only five users [67].

Often (only) 5 or more users are analysed for early stage usability studies. The confidence intervals are wide enough, and the average SUS score is surprisingly stable. In the state of the art, several computer simulations have been performed showing that with a sample size starting at 5, the sample mean is within six points of the SUS score of a very large sample 50% of the time [65].

Nicholas Pappas summarized in 2010 [68] that if the actual SUS score was a 74, average SUS scores from five users will fall between 66 and 80 half of the time. Seventy-five percent of the time, the score differed by 10 points, and 95% of the time, by about 17 points.

Methodology to conduct the test:

  • The participant fills in a first form with relevant data: gender, age, operating robot skills, and education level.

  • The participant practices with the robot to be more confident with its use and manoeuvring.

  • The participant carries out the test performing the patina task.

  • The participant fills in the NASA-TLX and SUS questionnaires related to the experience of performing the test with the task.

  • The participant makes comments about his/her global perception and answers some additional general questions.

Additional considerations of the test: The first practice with the robot to be more confident with its use and manoeuvring took 10 min in all cases (experts and nonexperts on robotics). The test includes the whole patina process, and it takes between 10 and 15 min; Each user has tested the system three times, and information registered is the average punctuation given by the user.

Figure 17a shows that in all cases the effort of manual patina operators is greater when using the robot than when performing the task manually. The only two exceptions to it are the overall effort and frustration in performing the task. This is because path recording takes less time and requires less physical effort in both cases. Figure 17b shows the perception of effort that robotics users have when performing the task manually versus automatically. In this case, and as expected, the perception of effort perceived by the user is much higher when performing the task manually. Figure 17c shows that the perception of effort to perform the path recording is much higher in the case of craftsmen than in the case of robotics specialists.

Fig. 17
figure 17

Results of NASA-TLX [63] usability tests

Full size image

Figure 18a shows the SUS technique applied to perform the same task. Perception of effort in performing the task is higher when using the robot in almost all cases. This effect is also shown in Fig. 18b, because performing the task manually is faster and easier than performing it using the robot, the level of concentration and effort is higher in all cases.

Fig. 18
figure 18

Results of SUS [64] usability tests

Full size image

The robot has been used on BFG current production line for 3 months. During this time, it has been possible to validate its behaviour, modifying the robot programme when required to adapt it to the company’s needs. The robot is currently being used on BFG factory to support a patina operator who has increased his productivity by around 25%. Although, as previously mentioned, productivity in some stages practically doubles (100% productivity increment), the operator has to put on and take off the shoe manually from the stand and has to access the HMI menus among other tasks, its productivity is reduced to 25%. This reduction in productivity is significant and the team is currently working on automating some tasks to increase this figure.

7 Conclusions

In this paper, a novel collaborative robotic system for handcrafted shoe polishing has been designed. This application of cooperative co-creation is completely new in the footwear industry. The proposed solution focus on trajectory generation and its replication by the robotic arm to accomplish the polishing task. Regarding the type of trajectory generation, three approaches have been suggested according to the degree of autonomy of the final application. The first approach consists of manual path recording by a qualified operator through the straightforward use of the robot tool. The second approach combines the manual path recording with a CAD/CAM system for trajectory optimization. Finally, the last method is based fully on the CAD/CAM digital data for automatic trajectory generation.

The first approach has been considered for experimental validation of a working functional prototype. For this purpose, a complete robotic cell has been implemented in a collaborative environment, where the robot performs the first stage of the polishing task for a shoe located in a static position. Considering the complexity of the work piece geometry, the need for controlling the polishing tool contact force is assumed. With this aim, the collaborative robot UR5e from Universal Robots with a built-in force control function has been selected for this application. Force control is done in the direction of the surface normal, hence the importance of keeping the tool oriented orthogonal to the surface at any time. Taking these factors into account and because of the presence of small concave surfaces in a shoe, a novel cooperative tool focused on this specific application has been designed and implemented maximizing operator’s ergonomics and factory needs. An intuitive graphic user interface has been developed to enable the configuration of process parameters. Versatility in the definition of the tool trajectories and the possibility of customizing the process parameters allows the craftsmen to apply their knowledge, according to the special characteristics and requirements of each area of the shoe. Using computer vision, the 3D shoe recognition and model matching are also addressed. The system is equipped with a LiDAR camera that scans the environment and detects the presence of the footwear, its correct position, and the model variant.

The application has been tested by professional staff in the footwear industry giving successful results in terms of quality for the first stages of the process. The system can achieve a suitable base on which the craftsmen can work in subsequent stages and notably improving their productivity. Therefore, it can be claimed that the application has a positive impact on the initial requirements. Furthermore, the implementation of the described system introduces an innovative solution for this type of industry, both regarding the used materials for polishing and general process design. Still, among the system handicaps, it should be underlined the fact that given the great diversity of shapes and styles of footwear, it is not always possible to satisfactorily undertake the polishing task in all of them. Obtaining feasible and reliable solutions when dealing with the automation of complex polishing tasks requires deep research of the whole process. Even so, the proposed solution is potentially suitable to be used in other manufacturing industries due to the methodological procedure presented in this paper, where task flowcharts and HMI description are described in depth.

The next steps consist of using up to four shoe stands to get full cell capacity production. In future works, the implementation and validation of the other two presented approaches are contemplated, not only to increase the reliability and robustness of the system by optimization strategies but also the degree of automation of the robotic system. The integration of finer computer vision technology is meant to add more autonomy to the system while introducing an intermediate verification step for process safety conditions. These improvements in the design of the overall robotized solution are expected to enhance the efficiency of the process and contribute to reducing retouching operations.