Cross-industry standard test method developments: from manufacturing to wearable robots

Abstract

Manufacturing robotics is moving towards human-robot collaboration with light duty robots being used side by side with workers. Similarly, exoskeletons that are both passive (spring and counterbalance forces) and active (motor forces) are worn by humans and used to move body parts. Exoskeletons are also called ‘wearable robots’ when they are actively controlled using a computer and integrated sensing. Safety standards now allow, through risk assessment, both manufacturing and wearable robots to be used. However, performance standards for both systems are still lacking. Ongoing research to develop standard test methods to assess the performance of manufacturing robots and emergency response robots can inspire similar test methods for exoskeletons. This paper describes recent research on performance standards for manufacturing robots as well as search and rescue robots. It also discusses how the performance of wearable robots could benefit from using the same test methods.

1 Introduction

Wearable robots, such as exoskeletons, are part of a broad category that includes systems that guide humans to assist them in moving their bodies as well as human-guided systems that augment body motions and forces for added speed or strength. Wearable robots can be partial or full body systems and are currently being developed in many countries around the world (Wolff et al., 2014).

Wearable robots have current or potential applications in rehabilitation (Szondy, 2015), elderly care (Dale, 2014; Leber, 2014), military operations (Tucker, 2015), and manufacturing (Stinson, 2014). The International Organization of Standardization (ISO) 13482 personal care robot safety standard was developed to provide safeguards for elderly or other persons using wearable robots, such as exoskeletons, and provide some cross-industry (Bostelman, 2010) consideration to manufacturing, military, or other industries. Although ISO Standard 13482 has been published, it includes no normative references to directly assess risks or hazards, design, verification, installation, or validation. Also, Herr (2009) suggested that there are other types of exoskeletons as yet undeveloped which may offer, for example, a “significant decrease in the metabolic demands of walking or running”, for which some measures for the standard may be required.

A few studies have attempted to measure the performance of exoskeleton systems to increase the wearer’s speed, strength, and endurance. However, the proposed measurements are targeted towards specific situations and cannot be generalized. Repperger et al. (1990) conducted an experiment to evaluate the performance of a human being wearing an exoskeleton, using Fitt’s law to characterize the performance. Liu et al. (2004) proposed a method to control a lower extremity exoskeleton and described experiments investigating how to measure the human and exoskeleton zero-moment point (ZMP). A control mechanism based on a hydraulic pressure valve for a lower extremity exoskeleton was presented by Guo et al. (2012). The results of a performance test on the hydraulic pressure control system were used to adjust different parameters to improve the system. Asín-Prieto et al. (2015) suggested a regression-based method to reconstruct speed-dependent and angular trajectories, to provide a more natural gait when using wearable rehabilitation exoskeletons. Although this testing methodology seems efficient, it is still applicable only to the specific case under study. Schabowsky et al. (2010) proposed a new hand exoskeleton rehabilitation robot (HEXORR) together with appropriate sensors and enablers. They conducted a pilot study to investigate the performance of the device. However, the measures used were limited to the scope of the study. Maciejasz et al. (2014) surveyed robotic devices and exoskeletons for upper limb rehabilitation. Their review discussed various aspects of these devices (e.g., application field, target group, type of assistance, mechanical design, control strategy, and clinical evaluation), but no information was provided regarding a standard way to measure their performance.

Exoskeleton technology and collaborative industrial robots both require safe human-robot performance and capabilities. However, unlike for collaborative industrial robots, there are currently no standard test methods for measuring the safety and performance of wearable robots. Technological improvements to non-wearable (collaborative) robots, such as industrial robots, mobile robots, and mobile manipulators, have allowed robots and humans to work side by side or robots to work with other robots (Fryman and Matthias, 2012). Collaborative robot safety standards, i.e., ISO 10218-2 and ISO Technical Specification (TS) 15066, have been developed and continue to evolve.

Safety and performance test methods are being developed so that manufacturers and users can evaluate and compare the capabilities of emergency response robots (Jacoff et al., 2001) and industrial robots against the requirements of their applications and particular tasks. Test methods for these industries could provide valuable insights for wearable robot standards, including what metrics should be considered, what safety and performance test methods should be developed, and how generic test methods might demonstrate a measure of safety and/or performance.

This paper will begin with identifying the types of wearable robots used in manufacturing industries that require safety and performance testing, and considering metrics for testing these systems. Standard test methods that have been, or are currently being, developed for emergency response and industrial collaborative robots will be discussed. This will be followed by a brief discussion of the process of test method development. Lessons learned and basic concepts from response and industrial robot areas will then be considered in planning the development of test methods for wearable robots.

2 Types of wearables to be tested

Both passively and actively controlled exoskeletons can provide useful capabilities for the manufacturing industry. Passive exoskeletons, such as Fortis (Fig. 1a), are not robots although they have capabilities that can prolong a worker’s capabilities. Passive systems can be adapted to the wearer and to the task by making mechanical adjustments to the system. Similarly, actively controlled exoskeletons, considered wearable robots, provide capabilities that can potentially be programmed to adapt to the wearer and the task. An example of an actively controlled exoskeleton is shown in Fig. 1b, in which a worker demonstrates his increased lifting capability at a shipyard. Actively controlled exoskeletons use electronics, motors, computers, and intelligent software control to provide adaptability to the wearer and task.

Fig. 1

Examples of exoskeletons: (a) passive (Tucker, 2015); (b) active (McDonald, 2014)

Herr (2009) suggested that metabolic energy cost can be reduced when wearing some types of parallel-limb exoskeletons and other shoes. This is one measure of safety and performance that can be used to define exoskeleton usefulness. However, other metrics that are not currently in the literature go beyond metabolic cost. Herr (2009) also described exoskeletons that can provide increased lift capacity, although there is little supporting information available on the use of these systems by a variety of people (i.e., of various sizes, shapes, genders, ages, etc.).

Metrics for both passive and active exoskeletons, each considered a generic system-under-test (SUT), are similar, including:

• 1. Duration: maximum time that a task can be performed with the SUT compared to that when performing the task without the SUT.

• 2. Speed: velocities that can be achieved and sustained with the SUT compared to those achieved when performing the task without the SUT.

• 3. Pose: uncertainty accuracy/resolution (e.g., precision to move to a commanded location) and repeatability (e.g., move to the same commanded location more than once) for the SUT to position and orient the operator’s arm or leg as commanded. The positioning error of a tool or device when held by the controlled arm or leg is the measured component.

• 4. Back drivability or control force: force required to resist component reaction or move any or all components of the SUT when they are both driven or not driven.

• 5. Put-on/take-off complexity: difficulty in putting on or removing the SUT.

• 6. Ease of use: simplicity of initial training and ease of control of the SUT as it allows or improves task completion performance.

• 7. Vertical maneuvering: capability, speed to traverse inclines, steps, and undulating terrain.

• 8. Horizontal maneuvering: capability, speed to traverse forward, back, and side to side.

These metrics need to be refined and detailed rubrics should be provided to define the possible range of values and allow better quantification of these measures. Other metrics are listed by Wolff et al. (2014) for exoskeletons being considered or used for rehabilitation, including comfort, cost, portability, battery life, range of use, and several others related to maneuvering the body.

3 Test methods from non-wearable robots

The market for non-wearable or collaborative robots has been increasing recently, perhaps in part due to ISO 10218-2 and ISO/TS 15066 approvals, as well as research activities. The National Institute of Standards and Technology (NIST) has been performing research on collaborative robots within its performance of collaborative robot systems project (Marvel et al., 2014) as part of the robotics for smart manufacturing program.

Robots for flexible factory environments are limited by their inability to coordinate, communicate, and understand their actions, roles, and task statuses to collaborate effectively and efficiently with others. Limitations are driven by both the absence of tools and protocols needed for describing collaborative functions, and the complete lack of metrics for assessing how well robots can work together and with humans. The above project (Marvel et al., 2014) is in the process of providing the methods, protocols, and metrics necessary to evaluate the collaborative capabilities of robot systems.

Similarly, emergency response robotics is being researched at NIST within the robotics test facility (Jacoff, 2013), which is a laboratory for developing standard methods for measuring robot performance. The facility houses artifacts and equipment for measuring how well robots perform a variety of tasks that abstract real-world challenges. The application domains supported by this facility include urban search and rescue, bomb disposal, military ground operations, disaster response, and manufacturing. Artifacts are designed to be abstract representations of the environment and task challenges that a particular requirement addresses. Experiments are conducted by running a wide variety of robots through the prototype test methods to understand how best to capture data and to refine the physical artifacts and methodology.

The wearable robots community can leverage experience gained from performance test method development and applications from research on both manufacturing collaborative robotics and search and rescue robotics. The following sections describe industrial and response robot standards and test methods that have aspects that could be considered for the development of wearable robot standards.

3.1 Industrial robot standards and test methods

3.1.1 Standards

Current standards and working documents forming the foundation for eventual standards for industrial robots, service robots, mobile robots, and mobile manipulators that may be of interest to the wearable robots community are listed in Table 1. More details are provided below about some draft standard test methods that could have greater relevance to the exoskeleton community.

Table 1 Industrial robot standards and working documents

3.1.2 Navigation

Recent research on industrial robots in the area of measurement systems for navigation, docking, and the ‘ground truth system’ provides a measurement basis for test method development improved by an order of magnitude (Bostelman et al., 2015). Fig. 2 shows an example navigation concept currently being considered for the ASTM F45.02 navigation standard.

Fig. 2

Example of reconfigurable apparatus for navigation tests for automatic guided vehicles (AGVs) of various sizes (Bostelman et al., 2015)

The moveable barriers increase the path confinement per trial. An automatic guided vehicle (AGV) or mobile robot is to traverse the reconfigurable path without contacting the barriers. The vehicle performance is measured by how well it follows the path without touching the barriers as their width decreases.

3.1.3 Docking

Positioning, or docking, of the vehicle and onboard equipment after navigating allows the vehicle to access a pallet, tray station, or a table of parts for assembly. Measurements of how well the vehicle docking is performed are therefore critical for users to understand vehicle integration for assembly, material handling, etc. Docking is also being studied using collaborative robots and artifacts through the use of a mobile manipulator which includes a robot arm onboard an AGV. Fig. 3 shows the evaluation of mobile manipulator performance using a reconfigurable mobile manipulator artifact (RMMA).

Fig. 3

Docking performance measurement of a mobile manipulator with a reconfigurable mobile manipulator artifact (RMMA) (Bostelman et al., 2015)

Small spheres mounted on both the mobile manipulator and RMMA are used as fiducials for an optical ground truth system to measure mobile manipulator motion relative to the RMMA during test method development

The RMMA can be reconfigured to be horizontal (Fig. 3) or vertical, as well as positioned below or above the mobile manipulator. The RMMA allows for a non-contacting manipulator pose to align a laser retroreflector with reflector fiducials on the artifact to within a few millimeters, dependent upon required uncertainty measurement. Static base, indexed base (i.e., stop and measure the RMMA followed by moving to a new position, stopping and measuring at the second position), and dynamic base positioning can be tested using the RMMA.

Another test used for evaluating the performance of AGVs, mobile robots, and mobile manipulators is obstacle detection and avoidance (Bostelman et al., 2015).

3.1.4 Grasping

Current industrial grippers are typically two-fingered, pinch-types. Three- or more-fingered industrial grippers are being developed for more dexterous manufacturing applications, such as assembly (Bostelman and Falco, 2012). Some advanced grippers resemble human hands, although most do not have five digits. Fig. 4 shows an example of an advanced, highly dexterous robotic hand being developed, and an example of prehension of typical objects (Bostelman. 2010; Campbell. 2007).

Fig. 4

Examples of advanced highly dexterous robotic hands being developed (Campbell, 2007)

Grasping is another area in which performance test methods could be considered. A proposed roadmap for dexterous manipulation (Falco et al., 2014) includes impact areas focused on several aspects of dexterous arm and hand performance, including sensing, motion, control, and applications.

3.1.5 Test methods

Test methods are expected to address at least some level of the following capabilities:

• 1. Hand mechanics: position control, torque control of fingers/digits, grasp capacity (e.g., graspable object size and mass), grasp types supported, accuracy, and repeatability.

• 2. Tactile sensing: normal forces and pressure, force and impact sensitivity, location of touch, functional tasks, quasi-static and dynamic effects on grasp stability, in-hand manipulation of objects, and touch sensitivity (e.g., using touch to control finger position/force).

Draft test methods are being developed for robotic hands and advanced grippers under a Metrics Working Group for an Institute of Electrical and Electronic Engineers Technical Committee on robotic hand grasping and manipulation (Falco et al., 2013; 2015).

Hand exoskeletons that could benefit from industrial gripper test methods are being embedded in an astronaut’s glove (Favetto et al., 2012) and used as hand exercise devices (Sarakoglou et al., 2007).

The aforementioned roadmap (Falco et al., 2014) also includes dexterous robot arms, proposing fewer complex performance metrics than for dexterous grippers, e.g., reachable volume (i.e., the positions and orientations that an arm can achieve within the workspace), operational space (i.e., the positions and orientations in which the arm and/or hand can effectively perform the required operation), confined space access, and grasping objects while in motion.

3.2 Response robot test methods

Several performance standards have been created through the ASTM International Standards Development Organization under the E54 Committee for homeland security applications (Bostelman et al., 2015).

Specifically, the E54.08, subcommittee developed standard test method suite for evaluating emergency response robot capabilities, focuses on measuring capabilities of robots with respect to mobility, energy/power, radio communication, durability, logistics, safety, human-system interaction (HSI), sensors, and autonomy, although most response robots are tele operated. This suite of standards can provide cross-industry test methods that may apply to wearable robots and passive systems. Table 2 lists the potentially relevant standards (noted by ‘ASTM’), working documents under development (indicated by ‘WK’ prior to a number), and planned standards for future development.

Table 2 Potentially relevant standards (noted by ‘ASTM’), working documents under development (indicated by ‘WK’ prior to a number), and planned standards for wearable robot and passive system test methods

Maneuvering tasks are under the human-system interaction category because they are performed at a standoff distance by the operator, requiring high levels of situational awareness to perform successfully. Examples of some of the standard performance test method artifacts are shown in Fig. 5.

Fig. 5

Examples of inclined planes (a), stairs (b), varying terrain test apparatus and actual varying terrain (above the apparatus) (c), and artifacts of increasingly complex terrains (d) (References to color refer to the online version of this figure)

Current response robot test methods have been, or are being, developed to make it simple to measure, for example, how well a robot navigates around an obstacle on a level floor. Incrementally, more challenging conditions can also be tested, for example, to measure how well a robot navigates inclined planes, steps, undulating floors or complex terrains, and around obstacles (Fig. 5). Additionally, the navigation and obstacle avoidance tests can be combined with vision tests, since most response robots are tele operated. This combination also provides a human-in-the-loop test where a robot’s pitch and roll can skew the operator’s reference frame for the images provided by the onboard camera(s), thus hindering robot control. Each test generically simulates a particular capability that response robots must possess to be useful in critical situations. For example, undulating floors or complex terrains may appear in collapsed buildings where search and rescue robot missions are required.

4 Test method development

Industrial robot and search and rescue robot test methods are being developed in a similar manner. In the case of ASTM F45 performance standards development, the mobile robot and AGV industries were surveyed to establish their current and potential system capabilities to meet specific user application requirements. In the case of ASTM E54.08.01 response robot standards development, the process began with in-depth workshops with emergency responders to identify key performance metrics and deployment scenarios, particularly focusing on urban search and rescue operations. Over 100 requirements were identified over the course of three workshops, which were used to guide the test method development process (Messina et al., 2005). Over time, additional requirements were added from new constituencies, such as bomb squads (e.g., for counter-vehicle-borne improvised explosive devices).

Test method development begins with establishing metrics and, as with any experiment, the isolation of variables and hypothesized results follows. Test methods that allow simple, isolated measurements of capabilities, for example, navigation, can then be broken down into simple-through-complex tests. For example, open-area navigation of a straight line, followed by the addition of a curve, and then added obstacles in the path, and lastly, increasingly narrow path confinement, is one simple test method. In the response robots test suite, the configuration of a robot under test is to remain unchanged through all the test methods. In other words, if a heavier battery is used to extend the robot’s endurance in the power/energy tests, it must be in place during mobility tests, such as stair climbing or inclines, where a changed center of gravity may affect performance. This provides realistic information about configuration tradeoffs.

Ideally, the method should not require expensive, resource-intensive measurement systems and procedures, and thus minimalist test method apparatus design and use must be considered. Apparatus materials should resemble the actual robot application environment and be readily available, relatively inexpensive, and simple to construct as in the apparatuses shown in Figs. 2 and 5 for industrial and response robot test methods, respectively. Alternatively, the need for high precision measurement may require a different approach. The RMMA shown in Fig. 3 was designed and machined to be relatively precise compared to the positioning capability of a mobile manipulator. Even in this case, it is expected that the components can be fabricated through additive manufacturing (3D printing) to save cost and avoid machining, while maintaining the required precision.

The test method administration, procedures, and reporting methods are established. Periodic reviews of draft test methods with potential end users and robot developers, resulting in iterative improvement of the design and procedures, are desirable for ensuring that the standards are useful and usable.

5 Cross-industry test methods

This section discusses how industrial and response robot navigation, docking, combined navigation and docking, and grasping test methods could be applied to wearable robots. Methods developed for industrial and response robot performance tests can help minimize the development process or guide designs for wearable robots. For example, one type of navigation surface may be applicable to one manufacturer’s exoskeleton but not to another. Increasing complex terrain navigation may also show limitations throughout the robot development process. Similarly, an exoskeleton motor, spring, and/or counterbalance may be tuned for lifting or manipulating heavy loads but not for threading a needle. More specific applications of previously discussed concepts follow.

5.1 Navigation

Wearable robots for lower body movement can perform tests similar to manufacturing mobile robots and AGVs demonstrating navigation through confined areas. For example, barriers or a series of objects can be placed along a path that the human must follow while wearing the robot. The walls can be moved closer to the path and if the human collides with the barriers or objects, the metrics of stability, maneuverability, and velocity can be measured. An additional test could be to test avoidance or maneuverability when obstacles suddenly appear in the human’s path.

Similarly, wearable robot navigation tests can be performed using response robot artifacts and methods. For example, inclined planes, undulating floors, stairs, and various complex terrains such as sand, gravel, or wet floors can be navigated while avoiding obstacles in the path.

5.2 Docking

Wearable robots or passive exoskeletons that allow human arms to move and hold tools for longer periods of time at intended locations could be measured using the RMMA. The human can instead carry a laser retroreflector or insert pegs in holes on such an artifact using a variety of geometric patterns and RMMA configurations. Similar to the mobile manipulator (Fig. 3), fiducials detectable by an optical tracking system can measure the wearable robot motion if higher precision measurement data are required. This fine motion detection data can be used to further refine wearable robot motor tuning.

Fig. 6 depicts the RMMA, previously described for measuring performance of industrial robot arms and mobile manipulators, being used to measure the performance of an exoskeleton. The figure shows a human wearing arm exoskeletons and aligning a laser retroreflector to reflectors. The same RMMA could instead include holes in which the human could insert pegs or screws as potentially required for precision assembly applications.

Fig. 6

Graphics of a human wearing arm exoskeletons testing their performance using the RMMA for precision assembly applications when the RMMA is in horizontal (a), vertical-low (b), vertical-high (c), and overhead-angled (d) configurations

Both navigation and docking can be combined for full-body exoskeleton (i.e., legs and arms) access and dexterity tests. For example, the human in the exoskeleton would repeatedly move from a different location to the RMMA, similar to tests for the mobile manipulator. Once at the RMMA, the same docking test would be administered. The results of this test could show the time for a human, wearing leg and arm exoskeletons, to repeatedly approach and be positioned to reach the RMMA (using leg exoskeletons) followed by the time to transition from full-body motion to arm-only motion (using arm exoskeletons) when controlled by the exoskeleton.

Dynamic tests can also be administered with the RMMA moving relative to the human and the same alignment task performed as previously described. Additionally, both the humans with the exoskeleton and the RMMA can be moving while alignment or peg insertion tasks are performed.

5.3 Grasping

Grasping tests for hand exoskeletons are very similar to advanced robot gripper tests, where various objects are picked up and manipulated (e.g., rolled, yawed, pitched) in the hand using the fingertips and/or the palm, and placed (e.g., set on a surface, inserted into a mating hole). Four grasp tests described in Falco et al. (2013) and performed on the exoskeletons shown in Fig. 7 are: (1) power grip, (2) two-finger pinch, (3) three-finger pinch, and (4) lateral pinch. The following are examples of more specific hand exoskeleton tests:

Fig. 7

Examples of hand exoskeletons (Reprinted from Favetto et al. (2012), Copyright 2012, with permission from Scientific & Academic Publishing)

• 1. A key could be picked up, inserted into a keyhole, and rotated.

• 2. A ball is picked up, grasped using the fingers and palm, moved using only the fingers to the finger tips, and then rolled using only the fingertips.

• 3. Bars of variable diameters each attached to a spring, or thin to thick ropes each attached to a weight, are grasped and pulled and the force is measured.

• 4. A doorknob is grasped with the hand and rotated using the wrist and/or a hand-wheel is grasped with the hand and rotated using the wrist and arms.

• 5. A needle is threaded or a wrist watch-size gear is placed on a post and meshed with other similarly sized gears.

• 6. Repeated exercise of the fingers is followed by performing the above tests.

6 Conclusions

Much of the experience gained in the development of metrics and test methods for manufacturing and response applications can be applied to wearable robots. Manufacturing robotics is moving towards human-robot collaboration and response robotics is moving towards robot deployment in place of people. Test methods for both robot types are being developed to measure their performance and match it to the task at hand. Similarly, active exoskeletons are worn by humans to move body parts, and passive exoskeletons are already being used to allow humans to extend their productivity and endurance. Safety standards now allow both manufacturing and wearable robots to be used. However, performance standards for both systems are still lacking. Test methods developed for manufacturing and response robots can be directly applied to wearable robots, as described in this paper. Almost direct cross-over between these industries appears feasible and associated performance standards can also be developed for wearable robots. Future research should include the demonstration and testing of wearable robots using test methods similar to those developed for manufacturing and response robots.