Keywords

1 Introduction

Since the Brundtland Conference in 1987, the goal of sustainable development of society has gained importance. Sustainable development is defined by the sustainable development goals as a target of the United Nations. In addition to being anchored in politics, the public awareness of sustainable development is growing and reaching companies and industrial production processes [1]. Sustainability is often represented using the triple bottom line (TBL), which includes the social, economic and ecological dimensions [2]. In order to establish sustainability considerations in the product development (PD) level, quantification methods are required. They can be used to evaluate different development stages of a product or, more commonly, at the end of life to assess a product retroactively. However, the quantification of sustainability is not trivial, since many influencing variables have an effect on it [3]. The methods developed so far still have numerous weaknesses, so that the results of an evaluation are rarely comparable and the implementation is often very complex. In addition, the large number of methods and indicators creates an overload for product developers and customers [4]. Therefore, the aim of this research is to develop a concept to evaluate sustainability assessment methods and tools (SAMTs) and provide criteria for this purpose. The second chapter addresses sustainability in product development, SAMTs and the concept of life cycle assessment (LCA). Chapter three explains the main concept including the developed criteria which are used in an exemplary application in chapter four to illustrate a SAMT evaluation. Chapter five discusses limitations and emerging research needs before chapter six closes with a conclusion.

2 State of the Art

Sustainability in Product Development

To achieve a sustainable PD, SAMTs are intended to help product developers to include aspects of sustainability in their considerations. The early phases are the most important, as this is where the main influences on TBL are established [5]. In order to achieve an improvement in sustainability, a variety of methods and tools exist which use different approaches to show or influence the sustainability of a product. Many SAMTs are based on performance measurement systems that use indicators of sustainability which represent measurable key figures. Depending on the company's goals, vision and type of business, different indicator sets are used to develop methods for assessing sustainability. To represent applicability in the product life cycle (PLC) phases, SAMTs can be further divided into qualitative, semi-quantitative, and quantitative [6]. Qualitative SAMTs are suitable for easy and quick application at the beginning of the PLC. Quantitative assessments, on the other hand, achieve higher quality results but rely on accurate data and are used more towards the end of the PLC.

Life Cycle Assessment

In order to measure the sustainability of a product, it is useful for a company to consider the influences on the product's sustainability over the entire life cycle. LCA is a systematic procedure defined in ISO 14040/14044 that considers the inputs and outputs of processes during the PLC and assesses the impact of these on the environmental dimension. Essentially, a balance sheet is drawn up of the parameters relevant to the product during all phases of its life, and the environmental impact is then assessed with the aid of characterization models [7]. An LCA consists of four phases which are displayed in Fig. 1.

Fig. 1.
figure 1

Phases of a life cycle assessment according to ISO 14040/14044

Full size image

Life cycle-based methods form the basis for many SAMTs and are specialized for different use-cases. In order to successfully select a solution for a specific problem, there is a need for concepts and selection criteria. This research builds on existing approaches.

3 Concept

Ernzer and Birkhhofer [4] propose a three step approach to select the right LCA method based on company needs. Their first step is to analyze the wide variety of methods and structuring them into an organized pool. The main concept of this research builds on this first step and consists of criteria and categories for the structured evaluation of SAMTs. For this purpose, the system context is considered according to Pohl and Rupp [8] which places the SAMTs in context to the product development process (PDP). For a better understanding and applicability, the criteria are then aggregated into four categories which are displayed in Fig. 2. This chapter discusses the categories and highlights the development of the criteria and the system context analysis.

System Context Analysis

This section presents the analysis of the system context to derive criteria for the evaluation of a SAMT. The result is a series of requirements that are sorted and combined into criteria summarized in Fig. 2. The system context results in influences to a SAMT from two groups; external and internal influences. The first group includes laws, directives and regulations as well as documents and software. The first three can have an influence on the selection of a SAMT when it is used to achieve goals set by law. An example is the German law “Act on Corporate Due Diligence Obligations for the Prevention of Human Rights Violations in Supply Chains” [9], which sets requirements for responsible management of supply chains in order to improve the international human rights situation. Documents and software can dictate interfaces to company processes for a LCA tool. For example, the product data model might store the relevant information in different formats and structures which needs to be considered when choosing and implementing a SAMT.

The second group are internal influences by stakeholders. These can be persons within the system context using a SAMT and having different prerequisites on know-how, expectations or use cases. This research identifies several groups of stakeholders and analyzes them regarding their familiarity with a given computer platform, their familiarity with the application area and their expected level of analysis detail. The stakeholders considered in this research are project managers, product developers, product manager, business manager, quality assurance, process planning, sales, marketing, customers, logistics and production staff. All groups use different software tools regularly so the integration in those tools can be significant. The level of know-how and the level of assessment detail influence the choice of SAMTs and must be included in the criteria.

In addition to stakeholder requirements, there are other requirements for a SAMT, so categorization features and criteria from the literature are used to further consider requirements (see [6, 10, 11]). These sources and the criteria from the literature research performed by Lindahl and Ekermann [12] substantiate the criteria developed in this research.

Criteria and Aggregations

Lindahl [13] finds four underlying criteria that a SAMT must fulfill. These are the easy adoption and implementation, fulfillment of specified requirements, reduction of the risk to forget important elements in the PD and the reduction of time to solve the task. The here proposed criteria expand on these principles and ensure the evaluation and selection of a fitting solution. They cover the core functions of SAMTs and give users a methodology for a direct structured comparison. The developed criteria are then aggregated into four categories firstly without weighting, so that all criteria are considered equally. At this point the evaluation should be performed qualitatively due to the lack of standardization for SAMTs. The weighting for an individual application use-case can be added later based on the internal stakeholder requirements. Figure 2 shows the structure of the developed criteria. The first category is the scope of functions. It describes how many dimensions of sustainability the method covers, the extent to which the SAMT specifies how the assessment is to be carried out, and the extent to which it can be adapted to the needs of the company.

Fig. 2.
figure 2

Aggregated criteria for the structured evaluation of SAMTs

Full size image

The degree of specifications describes whether the SAMT defines each step of the assessment or only outlines the framework conditions. In the latter, users must make many methodological decisions to perform an assessment and accordingly have a deeper knowledge. Dimensions of sustainability are based on the TBL. This criterion describes whether the solution is focused on the ecological dimension or includes social and economic components as well. Adaptation to company needs is also an important criterion, as the idea of a sustainability assessment is often derived from the corporate strategy. For example, the choice of sustainability key figures or the integration of company specific product category rules. The second category is user friendliness. The first criterion is the effort and time required to perform the SAMT. The application complexity of a given method describes the user perspective and distinguishes whether there are user guides or documentations and the overall description and presentation. The SAMT complexity includes the structure of the calculation models and how many indicators are considered. The last criterion is the integration of the SAMT into the PDP which describes whether a SAMT is performed separately or integrated in the PDP and possibly even has interfaces to it. The third category is the quality of the results, which describes how accurate the results of a SAMT are. One factor influencing the quality of the results is the quality of the data, which describes whether quantitative or qualitative data are used. This is specific to the choice of SAMT and can be evaluated easily. At the same time, it is a key factor for analyzing the conflict between ease of use and the quality of results. Another aspect of this conflict is the scope of the results, describing the number of indicators to be analyzed. It is easier to assess for one company specific indicator than to cover a range of indicators used to prove the compliance with laws. The scientific substantiation is a measurement of how many case studies a method is supported by. The robustness and the handling of uncertainties of a SAMT describes whether a sensitivity analysis is provided and how uncertainties are included. The fourth category of criteria for a SAMT is defined by the usability of the results. This describes how the outputs from the SAMT can be used to incorporate and communicate aspects of sustainability into PD. It results from the way the SAMT communicates the results to the users and whether a decision support with possible suggestions takes place. For a high informative value of the evaluation results, transparency and comparability are important, so that they can be used for well-founded decisions. The criteria are intended as a foundation for a generalized method to evaluate SAMTs for a specific system context. The approach builds on the methodology of Ernzer and Birkhofer [4] and can be used by enterprises for an initial, qualitative analysis and selection of SAMTs.

4 Exemplary Application

In this section, the exemplary evaluation of SAMTs is performed using the developed criteria. Due to a paucity of data on many SAMTs and few use cases, a qualitative evaluation is conducted. The evaluation is based on case studies and the developer’s descriptions of the methods as well as the authors’ expert opinions after using the solutions. For the evaluation, each criterion is described by a catalogue of verbal components associated with quantitative values between 0 and 3 in order to compare the focuses of the different solutions. Figure 3 shows the evaluation of three methods and three tools. As a life cycle-based method, LCA according to ISO 14040 is evaluated and as a hybrid method the Partial Equilibrium Modeling (PEM) in combination with the LCA is analyzed. For the integrated method, the SEEbalance is used, which combines LCA with social LCA. These methods are widely used in practice and show an application of the proposed method. The criteria can be used to easily compare SAMTs with an LCA. The tool evaluation can be seen in the right graphic. OpenLCA is chosen for the full LCA tools, Solidworks Sustainability is used for the simplified LCA tools, and the Sustainable Balanced Scorecard (SBSC) is examined for the integrated tools. This chapter displays an excerpt of 22 total evaluated SAMTs over the course of this research.

Fig. 3.
figure 3

Evaluation results of the methods on the categories from the SAMT project. Left: Evaluation of the categories of the methods from the SAMT project based on examples. Right: Evaluation of the categories of the tools from the SAMT project based on examples.

Full size image

The analysis shows that the methods based on LCA have a high quality of results, which is due to the high quality of the data, the scope of the analyses and a good scientific substantiation. Likewise, they are characterized by a high level of functionality, with SEEbalance standing out by covering all three dimensions. The usability of the results is only in the middle range due to the fact that the methods do not offer support for interpretation. Likewise, the user-friendliness is rather low, since these methods are very scientific and many methodological decisions have to be made in order to perform an evaluation. This also leads to a low comparability of the results. The tool openLCA improves the usability of the results compared to the normal LCA. This is achieved via integrated evaluation tools and an operation with the aid of a graphical user interface. The quality of the results is maintained and all dimensions of sustainability can be considered. This contrasts with the simplified LCA tool Solidworks Sustainability, which is tightly integrated into a CAD software. Predefined evaluation criteria increase usability, but impose a low level of functionality. The method has a comparatively low quality of results due to the lack of case studies. The SBSC is a method that is supposed to measure the performance of sustainability by adding sustainability to a classic balanced scorecard. This is very much dependent on the user, as only a basic framework for the evaluation is given. Thus, it has a very high functional scope and can show a high quality of the data. However, due to the high degree of freedom and low level of integration, it is only moderately user-friendly. Thus, it can be stated that the analyzed methods and tools mostly have a high range of functions and a high quality of results, but the user-friendliness is not very high. This can be increased by the use of a tool. Based on this exemplary evaluation, a user would have a choice of methods that differ in the dimensions of sustainability and slightly differ in data quality. But most importantly based on the individual expertise, they could choose between a user-friendly simplified tool with less functions and lower quality and complex solutions requiring know-how but offering high results quality.

5 Discussion

Limitations

To ensure that all criteria are considered equally, a uniform weighting was used within the course of this research. Furthermore, the criteria provide indications for the evaluation, but should still be improved with the involvement of relevant stakeholders. In addition, some SAMTs are vaguely formulated and do not provide sufficient information. Experts or extensive transparency of the individual SAMTs are needed here. Ideally, this would involve a comparison of the different SAMTs in the context of a scientific study that looks at the same product for all SAMTs or a reflection of the methods by different experts and users. Therefore, the evaluation using the developed methodology still needs to be performed by an expert until more standards and comparable data exist.

Research Needs

Based on the analysis of the SAMTs using the developed concept, there are certain required aspects of research which can be categorized in a practical and a theoretical perspective. Above all, there is a need for standards and validation of existing SAMTs through case studies to enable comparability. For this purpose, the proposed criteria can be used for example, to create databases of evaluated SAMTs. A practical continuation could be to develop weightings of the criteria for different business segments. Likewise, most solutions lack consideration of all three sustainability dimensions and assistance systems for making decisions based on the assessment. Furthermore, there is a need for a deeper stakeholder analysis for the further development and validation of the here proposed criteria.

6 Conclusion

Sustainability is becoming increasingly important in society, politics and also in product development. To assess sustainability and make the results reliable and comparable, methods and tools are needed. However, there are many different solutions without standardization. In order to create an overview and to make a suitable selection of SAMTs, this paper presents criteria and categories via requirements based on a system context analysis and stakeholder analysis. The criteria are then put to use in an exemplary evaluation of LCA methods and tools. The qualitative results reveal strengths and weaknesses of the SAMTs which enable an effective selection for the specific company needs and use cases. The proposed criteria therefore consider different stakeholders and requirements resulting from a system context analysis in order to perform a structured analysis of SAMTs.