1 Introduction

Doctoral studies have a long tradition in higher education systems (Bogle 2018). Doctoral studies are highly relevant because they are considered as a key for technical development and industrial excellence in developed countries. Normally, a PhD diploma is compulsory for pursuing and it is highly valued for getting involved in research projects in companies. The goal of doctoral programs is to provide postgraduates with competences for the generation of knowledge in a given domain. The means to generate knowledge depends on the area, being research methods and techniques potentially different, and evolving in parallel with the development of the domain. In young domains such as Engineering Design, the discussion about which research procedures and paradigms should be employed is still open.

Simon (1996), in his book The Science of Design,  defined design as a search for an optimum in a space of alternatives that take into account the specifications and restrictions of a given problem. Hatchuel (2001) highlighted limitations of Simon’s position discussing that designing cannot be reduced to taking decisions among a bounded set because the number of concepts related to the problem and the possible number of decisions to be taken could be expandable and uncountable, not only due to human creativity but also to social interaction. (Subrahmanian et al. 2020) place Simon and Hatchuel’s approaches in a historical timeline that describes different models of how designing is understood, evidencing the challenges for research design as a discipline that defines a common language that includes the impact of context and users in designing, in addition to the problems.. Probably due to the youth of design as a research discipline, or due to its socio-technical nature, it does not yet have a consolidated research methods and techniques. Blessing and Chakrabarti (2009) proposed the DRM (Design Research Methodology) motivated by “frustration about the lack of a common terminology, benchmarked research method and a common research methodology in design”. Through the analysis of recent research papers, this work has the aim to confirm how these visions about research in engineering design are projected in current state-of-the-art publications.

Since the work of Blessing and Chakrabarti, there have been some relevant proposals that have shed light on different aspects of the global design research landscape. Koskinen et al. (2011) proposed the term ‘constructive design research’ and presented alternatives to integrate research within the practice of design. Joost et al. (2016) used the term ‘design as research’ in a volume that compiled discourses of experts about questions on design research and its relationship with other disciplines. Vaughan (2017) presented a survey that collected different points of view related to doctoral education in the opinions of design graduates about practice-based research design. Redström (2017) presented an essay about how to develop theory -knowledge- by practice, experimentation and making -design. These works are a multi-faceted compendium of practical experiences and visions of experts on how to perform activities related to research in the domain of design. Although many examples and discussions presented in the cited books focus on the topic of research through/by design, rather than on research in engineering design, all of them agree on the relevance of research into the design due to the increasing number of PhD programs that could benefit from background knowledge about this topic. In this paper, we present an alternative approach to shed light on the relations between research and design: instead of collecting the personal visions of experts, we summarise and classify research papers on research in Engineering Design in terms of aims and contributions, methods and approaches, data collection techniques, and research instruments used for the collection of data. To this end, we have carried out a systematic review of the literature on research in engineering design. The overarching research question (RQ) that drives the review is: What is the current landscape of research methods in engineering design?

Access to doctoral studies normally requires candidates to have a Master’s degree in which they have taken courses about research methodologies. Doctoral studies normally culminate with the defense of a PhD thesis in which postgraduates have to show their capabilities to generate knowledge in a specific field. Submitting a PhD thesis that includes activities previously reviewed in scientific journals is generally considered as a quality warranty of the research performed by the student. Although publishing journal papers is not the only way to assess the excellence of the research work performed in a PhD thesis, the quasi-exponential increase of scientific publications we are witnessing (Tenopir and King 2014) indicates that it is probably becoming a universal standard for rating the quality of research. Therefore, being aware of the kind of works published in scientific journals related to engineering design could be of outstanding importance for scholars who have to configure the contents of the courses related to research methodologies in this field, as well as for PhD supervisors and students to focalize efforts for being more productive in terms of publications. The analysis of scientific papers about research in engineering design performed presented in this paper aims to contribute to this aim.

There are many possible ways to analyse, categorise or classify research works because there are many dimensions of analysis. Creswell (2009) presents a classical distinction between (1) quantitative, (2) qualitative and (3) mixed-methods (combining qualitative and quantitative research methods). For quantitative methods experimental designs, non-experimental design are distinguished. For qualitative, narrative research, ethnographic research, phenomenological research, grounded theory and case study research are distinguished. For mixed-methods, sequential, concurrent and transformative methods are distinguished. Blessing and Chakrabarti (2009) identified the following ones: (1) paradigm, that includes empiricism (Randolph 2003; Solomon 2007) and ethno-methodology (Atkinson 1988), methodologies, theories, views and assumptions (Kothari 2004); (2) aim, research questions and hypotheses; (3) nature of the study, including observational vs interventional (Thiese 2014), comparative vs non-comparative; (4) units of analysis; (5) data collection methods including recordings, interview, questionnaires (De Leeuw 2008); (6) role of the researcher (Fink 2000); (7) time constraints, duration and continuation of the research process; (8) observed processes including layout drawing, prototype or product; (9) setting referring to laboratory or field research (Paluck and Cialdini 2014); (10) tasks including type and complexity and nature; (11) number of cases, case size and participants (Diggle et al. 2011); (12) object of analysis distinguishing objects, companies, projects, documents… (13) coding and analysis, analysis and (14) verification methods (Brewer and Crano 2014); or (15) findings, that is, statement models or conclusions resulting from the study. Reich and Subrahmanian (2021) use the PSI framework (Problem, Social and Institutional space) to analyse and categorise research design works focussing on dimensions related to the problem being addressed concerning (1) disciplinary, (2) structural complexity and (3) knowledge availability; dimensions related with who is included in designing concerning (4) the perspective required to formulate the problem, (5) the inclusion of participants in the design process and the (6) capabilities of the design team; and finally dimensions related with how designing is executed taking into account (7) the ties or connections between actors, (8) the accessibility to knowledge and (9) the institutional complexity (Reich and Subrahmanian 2020). The dimensions presented by Blessing and Chakrabari have the ambition to classify different aspects to be taken into account when research in engineering design works are tackled. The dimensions proposed by Reich and Subrahmian are complementary and arise when they analyse the factors influencing success in engineering design projects. When analysing papers, some of the details related to some of the listed dimensions could be missing in the descriptions (timing, success validation etc.) so that we had to devise alternative proposals.

Our analysis pivots around the division between empirical qualitative, quantitative research and mixed-methods proposed by (Creswell 2009). This classification was complemented with analytical research methods, as specified by (Adrion 1993), cited by (Glass 1995) (defined in Sect. 2.2). From this germinal division, data-collection methods, strategies, and contributions of the studies are reported in cross-analysis tables. We aim to identify the main goals and results pursued or obtained by researchers (dimensions 2 and 15 of Blessing and Chakrabarti 2009), the strategies of enquiry and methodologies they follow (dimensions 1, 3, 9, 10 of Blessing and Chakrabarti 2009), and which data sources and instruments are most (and least) commonly used (rest of dimensions of Blessing and Chakrabarti 2009) in the domain of engineering design.

The structure of the document is the following: First, we present the review method and the categories used to classify the papers. We then present the quantitative results of the number of papers in each of the categories and the cross relations of the different classes, shedding light on the relative weight of each of the qualitative and quantitative approaches and the most frequent data-collection methods used. Next, we discuss the usefulness of the obtained results for academics and professionals interested in research design and the paper ends with the conclusions. Complementary material is provided with a brief description of each of the analysed papers.

2 Method

We follow Kitchenham et al. (2009) as a guideline for performing the systematic review. The nature of the research question did not suit a usual search in the databases, as we were interested in analysing the approaches to research published in the field of engineering design. For this reason, we focused on identifying papers published in relevant journals in the field. The data sources are journal papers in the field of engineering design.

A simple search in the Journal of Citation Reports using the term “Design” as a key search title criterion, generates a list of 99 journals indexed in different categories. Only 80 are indexed in 2020, the rest of them in previous years. As we aimed to high-impact journals reporting research in engineering design, we focused on the journals indexed in SCIE (Science Citation Index Expanded) related to Science and Technology, discarding the 22 journals indexed in ESCI (Emerging Sources Citation Index), the 10 indexed in AHCI (Arts and Humanities Citation Index) and the 5 indexed in SSCI (Social Sciences Citation Index). Among the 43 remaining journals indexed in SCIE, 13 of them correspond to categories related to Chemistry and Biology (for example Anti-Cancer Drug Design or Molecular System Design & Engineering) 11 of them to Computer Science or Electrics (for example Design Codes and Cryptography or Computer Aid Design); 3 with Mathematics (for example Journal of Combinatorial Design) and 2 with Building (Architectural Engineering and Design Management or Structural Design of Tall and Special Building). Closer to engineering design are the 14 remaining journals: 4 indexed in Mechanics Journal of Mechanical Design, Mechanics Based Design of Structures and Machines, Journal of Advanced Mechanical Design Systems for Manufacturing and Journal of Strain Analysis for Engineering Design), 4 related to Materials (Materials & Design, Proceedings of the Institution of Mechanical Engineers, International Journal of Mechanics and Materials in Design and Road Materials and Pavement Design); and 2 related with vehicle design (Journal of Ship Production and Design, and International Journal of Vehicle Design). In spite of being closer to the topic of research in engineering design, we discarded these journals for being too specific. The remaining 4 journals were: (i) Design Studies (DS), (ii) the International Journal of Design (IJD), (iii) the Journal of Engineering Design (JED) and (iv) Research on Engineering Design (RED). Table 1 shows that these journals share the category denominated “Engineering Multidisciplinary”. In this category, there are 6 journals that have the term “Design” in the title, the four selected plus International Journal of Technology and Design Education (also indexed in SSCI), Artificial Intelligence for Engineering Design Analysis and Manufacturing (also indexed in Computer Science) that were discarded for being specialized in education and in artificial intelligence with applications in engineering design, respectively, and therefore, out of the focus of our research.

Table 1 Journals that are the focus of interest in the study. Eng Mult is the category named “Engineering, Multidisciplinary”, Eng Manu the one named “Engineering, Manufacturing” and Eng Ind the one named “Engineering, Industrial” of the SCIE JCR index. SS Inter is the category “Social Sciences, Interdisciplinary” of the SSCI JCR index. In the cells, A/B figures mean number of reviewed (A) papers versus total number of papers (B). Special issues are underlined.
Full size table

Each of the selected journals declare in their presentation their aims and audience: RED focuses on design theory and methodology, DS focuses on design processes, JED focuses on different aspects of the design of engineered products and systems, and IJD publishes research papers in all fields of design. The audience of DS, JEC and IJD is broader than the one of RED, which focuses on mechanical, civil, architectural, and manufacturing engineering. Overall, the four journals constitute a rich and representative sample that includes works of diverse nature, applying a variety of research methods and approaches to different problems in the context of research in engineering design.

Sample selection in systematic literature reviews must be structured, comprehensive, and transparent (Hiebl 2021). To comply with these three requirements, we established a recent and limited temporal window and applied random selection to select the sample. We collected 17 papers from each journal, as 17 is the number of papers available in one of the journals under analysis (IJD) and we chose to use the same number of papers per journal to avoid bias (i.e., giving more importance to one journal than another) in the study. For the journals with more than 17 papers in the period of analysis, random selection was applied. We focused on papers published between November 2018 and November 2019, which was the most recent available time window when this work was started.

This methodology led to a final total of 68 papers. We followed a collaborative team-coding approach (Saldaña 2021). Papers were selected and assigned randomly to a pair of reviewers. Each reviewer coded two papers every two weeks. Disagreements and new code proposals were resolved in periodic meetings involving the four researchers/authors. The first author of this paper played the role of “codebook editor” (MacQueen and Guest 2008), updating the code list after the meetings and he used the data from the analysis to build the final tables and present the resulting themes derived from the study.

With the aim of answering the general question of this review, RQ:, “What is the current landscape of research methods in engineering design?”, we focused on the following more specific sub-questions:

  1. RQ1:

    What are the research goals pursued by the analysed works?

  2. RQ2:

    What are the main experimental approaches found in the reviewed papers?

  3. RQ3:

    What data collection methods are employed in the reviewed works?

  4. RQ4:

    Which instruments are normally used to collect these data?

To answer these questions, we followed an anticipated data condensation approach (Miles et al. 2020). We defined four overarching topics corresponding to the research sub-questions: aims and contributions of the research; research approach; data collection techniques; and instruments for the collection of data. For each topic, we defined a set of categories, based on our revision of engineering design methods (see Sect. 2). During the iterative coding work, emerging categories were included when required. The new categories were used to re-codify all the works. This combination of deductive and inductive coding enabled us to derive new meanings from the data.

In the rest of this section, we present the categories that were identified in the analysis under each topic. Appendix shows complementary information with representative examples of the categories.

2.1 Aims and contributions

Concerning the aims/contributions of the research (RQ1), we started from an empty list of research targets which was enriched as the number of reviewed papers increased. Finally, the following research goals were identified through the coding process:

To study or propose a methodology, that focuses on papers whose main objective is to study an existing design methodology by analysing its validity in works that propose a new design methodology or that develop a part of it more deeply.

To delve into a given aspect of design, which includes papers that focus on exploring an aspect of a design (team communication, sketching, generation of ideas, materials...) or that explore one area of design that is recognised as challenging (social design, inclusive design, ecological design...).

To design, develop, or test a specific product, which includes those papers that set out the process of creation or development of a specific product or a group of them. Some of these works describe the overall process of creating a product, and others focus on a specific phase of its development (research, ideation, testing, and validation).

To make recommendations or propose guidelines, which include articles whose main aim is to systematize the results of their research to provide advice, either at a methodological level or in the design of new products.

Proposing a theory includes those articles that use logical reasoning or mental operations, such as imagination, intuition, abstraction, and deduction, with the aim of enunciating concepts or creating models, explanations, or theories about the phenomena under study.

Proposing a framework of analysis or a taxonomy that enables concepts or objects to be classified into categories.

More than one code could be assigned to each of the papers. This could be the case of a paper that aims to develop a specific product and ends by proposing guidelines.

2.2 Research approach

Concerning experimental approaches found in the reviewed papers (RQ2), as explained in the introduction, we propose the use of the distinction between quantitative, qualitative, mixed, and analytical research methods, defined as:

Quantitative empirical studies are those that aim at testing theories by examining relationships between variables, based on the collection of numerical data which is analysed using statistical procedures.

Qualitative empirical studies are those that aim at exploring and understanding in depth the meaning that individuals or groups give to a problem. They usually involve the collection of non-numerical data obtained in the participants’ settings and follow inductive analysis approaches in which the researchers interpret the meanings of the collected data.

Mixed-methods studies are those that combine both quantitative and qualitative approaches at diverse levels (data sources, analytical methods, etc.), so that the overall study is stronger than using each of the two approaches (i.e., quantitative, or qualitative) separately.

Analytical studies are those that focus on the formalization of a model and its demonstration. They start out by proposing a formal model with a mathematical formulation, derive results using deductive approaches, and, if possible, compare these results with empirical observations.

With respect to quantitative empirical studies, we subcategorize them into experiments, quasi-experiments and non-experiments, depending on the way the subjects of interest are assigned to an experimental group or to a control group:

Experiments: the assignment of subjects to the experimental or to the control group is random.

Quasi-experiments: there is not a random assignment of a subject to the groups.

Non-experiments: there is not control on the grouping of subjects.

When a known qualitative strategy of inquiry is used, it is also tagged. According to the definition proposed by Creswell (2009), strategies of inquiry are types of methods, designs or models that provide specific direction for procedures in a research design.

Ethnographic research documents the beliefs and practices of a particular cultural group or phenomenon in its natural environment from the perspective of insiders (Lapan et al. 2012). The researcher stays on site for a considerable amount of time to analyse practices and behaviours of groups, by observing, interviewing and (sometimes) participating in the process under analysis. Very popular in social sciences, it is also used in architecture (Cranz 2016).

In phenomenological research, the researcher identifies the essence of human experiences about a phenomenon as described by participants, while the researcher sets aside his or her own perspective (Wilson 2015).

Grounded theory is a strategy of inquiry in which the researcher derives a general theory grounded in the views of participants, involving the use of multiple stages of data collection (Jørgensen 2001).

Hermeneutics inquiry focuses on disclosing how participants’ interpretations of a phenomenon determine the way they live in the world (Stigliano 1989). This technique is popular in architecture (Pérez-Gómez 1999).

Case study research is an empirical strategy of inquiry that investigates a contemporary phenomenon within its real-life context (Yin 2009). It uses descriptions of programs, events, or other phenomena to construct a complete portrayal of a case for interpretation and possible action (Lapan et al. 2012).

Eikeland (2006) describes different approaches to action research that involve applied research, moving experimentation from laboratories to field, inviting the subjects of research to join the community of researchers and involving practitioners in research with the insistence of thinking through personal practices. Action research is a very popular approach in social sciences (Stringer 2008; Clark et al. 2020) and it is also proposed for architecture (Herr 2015) and for the practice of product design (Swann 2002). This method is related to the terms research-through-design, practice-based-design research or research-by-design (Redström 2017; Vaughan 2017), that has been discussed to be a kind of action research in works like (Kennedy-Clark 2013; Motta-Filho 2021).

Case study is generally used for exploratory research or for pre-testing some research hypotheses (Blessing and Chakrabarti 2009). Action research requires a high degree of flexibility and is usually qualitative, data-driven, participatory, and makes use of multiple data sources. Case study and action research also appear in the following criteria of classification, following the proposal of Blessing and Chakrabarti (2009) referring to data-collection techniques.

2.3 Data-collection techniques

In this subsection, we present the list of data-collection techniques we have tagged, to analyse what is proposed in RQ3. Following the list of data-collection methods presented in section A.4 of Blessing and Chakrabarti (2009), excluding experiments, case studies and action research we prefer to include in the list of inquiry research strategies presented in the previous subsection.

Observation is a technique in which the researcher records, in real time, what is happening, either by hand, recording it or using measuring equipment. As Blessing and Chakrabarti (2009) explain: ‘The quality of observational data is highly dependent on the skill, training and competency of the observer’ (Blessing and Chakrabarti 2009). Observations are the main source of data in ethnographic studies (see Sect. 2.1), but this strategy is also commonly used in social sciences (Creswell 2009) and in visual design (Goodwin 2000), architecture (Cuff 1992) and product design practice (Wasson 2000).

Simultaneous verbalization refers to the situation in which the participants speak aloud while using a system, with the aim of providing information about the cognitive behaviour of the participants, which may not be obtained through normal observation (Ohnemus and Biers 1993). Often used to analyse problem-solving behaviour, its most important feature is the real-time aspect. Simultaneous verbalization sessions usually last a few hours and never more than a day, due to the effort required by both the participants and the researchers in their corresponding analysis. Although audio recordings are sometimes used to record simultaneous verbalization, they are understood as inappropriate for a process such as design, which usually involves drawings and gestures, so video recordings are considered more appropriate.

Collecting technical documents consists of obtaining technical documents related to a particular project, topic or product, from various sources (Rapley and Rees 2018). Analysis of these documents is often used early in a research project to understand the organisation, the background of the project and the experience of the designers. It is commonly employed in most observational studies. However, if it is used as a single source of information, it can result in such limitations as the usual lack of data on the context in which the documents were created and the reason for their content. It is, therefore, convenient to complement them with other methods such as interviews.

Collecting physical objects involves mock-ups, prototypes and other physical models that may be relevant for developing a product or testing it. The model or prototype could refer to a part of the product or the whole product. For traditional engineering research, which focuses, for example, on the analysis of product behaviour, the products are the main source of data (Blessing and Chakrabarti 2009). In our review, we consider those works that start collecting different objects to carry out a study on their usefulness, or on the behaviour of users, for example. The object is a general term that can refer both to drawings and physical objects. Among the former, we find all those sketches, drawings and diagrams that have emerged throughout the conception of a product or its development, or throughout a research process, which could yield important information to organise ideas and draw conclusions.

Questionnaires are used to collect people´s thoughts or opinions about a certain product, process or method (Radhakrishna 2007). A priori, they seem easier to use than real-time methods, such as observation or simultaneous verbalization, and they are useful to obtain data from a greater number of cases. However, some of its disadvantages, such as the time required by the participants and the potential bias of the results, must also be taken into account.

Interviews have the same purpose as the questionnaires but are carried out face-to-face (King et al. 2019). Sometimes they are not carried out individually but using a group dynamic known as focus group: a group interview that mixes aspects of interviews and observations, as it provides information from the study of the interactions between participants. Focus groups can provide richer information than interviews, but they can have a negative effect on the contribution of specific participants.

2.4 Instruments for the collection of data

Data collection methods are supported by instrumentation. This section describes the categories we found to respond to RQ4, exposing the instruments that are normally used to collect these data. Independently of the strategy of inquiry applied, there are several instruments that are used to keep records of the observations. These recordings are important to keep evidence and to enable the reproducibility of the analysis. We tagged the papers depending on the use of classical audio, video and image recordings and the more recent technique of eye tracking (Bergstrom and Schall 2014).

In experiments and case studies, we are also interested in physical measurements that are used to objectify observations.

When questionnaires and/or interviews are the data-collection techniques, we tagged who is the attendee, distinguishing between stakeholders, users of products or participants (observed people) in the research and experts or designers. We also found it relevant to tag when the study uses workshops as a means to obtain information.

The last topic of interest that has been tagged is the fact that the research work uses simulation algorithms or tools as a source of information. We use this tag when the simulation tools are a fundamental part of the research, as it provides the information analysed in the paper (Behera et al. 2019), or because the tool or the algorithm itself is the main contribution (Mathias et al. 2019).

3 Results

3.1 Aims and contributions of the reviewed papers

Table 2 shows the codes assigned to each of the papers analysed. This section summarises the results related to RQ1 (research goals). As shown in Table 2, most of the works focus on methodologies or on the analysis of a specific aspect of the design processes. The presentation of a product and the building up of knowledge with taxonomies, guidelines, theories, or reviews, are exceptions.

Table 2 Coding of the papers analysed, according to the categories identified under each topic of the study
Full size table

Five papers propose a theory: (Comi et al. 2019) present the concept of shared professional vision; (Benavides and Lara-Rapp 2019) present the principle of weaker dependencies in axiomatic design; (Martinec et al. 2019) introduce the state-transition model (synthesis, analysis, evaluation) in conceptual design and Lloyd (2019) defends the theory of the social turn in design, Aktas and Mäkelä (2019) focus on the relation between craft, materials, makers.

Six works focus on the evaluation of a specific product: a software product in Takahashi et al. (2018) and Belkadi et al. (2019); or physical objects in the case of Roesler et al. (2019), Hyysalo et al. (2019b) and McKinnon and Sade (2019).

Concerning the works related to methodologies, we find papers that propose a method based on analytical methods or algorithmic solutions such as those related to axiomatic design (Chen et al. 2019a) and those related to such methods as research-through-design, where the importance of the method followed is prominent in the study (Tsai and Van Den Hoven 2018; Hyysalo et al. 2019b; McKinnon and Sade 2019; Hanrahan et al. 2019); or methodologies for product development such as Daalhuizen et al. (2019), with emphasis on different aspects such as work in groups (Gyory et al. 2019), sustainability (Santolaya et al. 2019) or democratised design (Hyysalo et al. 2019a).

A good number of papers present frameworks of analysis or classifications with different purposes. Bresciani (2019) for classifying visualization dimensions, McDonald and Michela (2019) to classify moral goods, Roy and Warren (2019) for card sets, Park-Lee and Person (2018) identify three practices on briefing, Vegt et al. (2019) deduce 3 types of invasiveness evoked by the rules in gamified brainstorming, Valverde et al. (2019) classify the type of feedback in automotive push buttons, Cooper (2019) presents the five waves in design research, Luck (2019) describes the framework to distinguish between design, design research, architectural design research and practice, Hobye and Ranten (2019) present five behavioural strategies for interactive products and Van Kuijk et al. (2019) presents a framework to analyse usability concepts of electronic products and Petreca et al. (2019) for analysing the relation between sensors and textile. We also include in this category the papers related to ontologies, that are used to represent knowledge.

Proposing recommendations is a common result in the analysed research papers, including a variety of themes such as recommendations on the use of guidelines by new designers (Reimlinger et al. 2019); the use of specific materials (Genç et al. 2018; Pedgley et al. 2018; Aktas and Mäkelä 2019; Petreca et al. 2019); how to orient future studies on the use of mobile technology by elderly people (Li and Luximon 2018), or about design and poverty (Jagtap 2019) or ethnographic studies in developing countries (Wood and Mattson 2019); appliance design (Selvefors et al. 2018); use of games in brainstorming (Vegt et al. 2019); or specifying requirements (Morkos et al. 2019). Cooper (2019) proposes interprets the history of design research through five waves.

The most frequent type of works delve into a particular aspect of product design such as sketching (Sung et al. 2019; Self 2019), prototyping (Menold et al. 2019; Mathias et al. 2019), material (Pedgley et al. 2018; Aktas and Mäkelä 2019; Barati et al. 2019; Petreca et al. 2019), interaction (Hobye and Ranten 2019; Valverde et al. 2019), briefing (Park-Lee and Person 2018), working in groups (Graeff et al. 2019), iterations and testing (Tahera et al. 2019; Piccolo et al. 2019); behavioural complexity (Hobye and Ranten 2019), manufacturing (Yang et al. 2019), or usability (Van Kuijk et al. 2019).

3.2 Strategies of inquiry and methodologies

This section summarises the results related to RQ2 (main experimental approaches founded): qualitative approaches are a majority, but the number of quantitative or mixed-methods studies is also relevant. Other approaches, such as the use of analytical methods, are less frequent. Table 3 shows that, when the goal of the paper is related to proposing or studying a methodology (first column in Table 3), the percentage of pure quantitative papers is lower than in the rest of the cases. Regarding whether there is a tendency towards any methodology depending on the journal; Table 2 shows that the Journal on Engineering Design seems to focus more than the other journals on non-qualitative strategies of inquiry.

Table 3 Number of works in each (sub)category per research approach
Full size table

When quantitative methods are used, experiments are more frequent than quasi-experiments and non-experiments (14 out of the 17 quantitative studies present an experiment). We found 26 experimental studies, with 5 quasi-experiments (Saliminamin et al. 2019; Vegt et al. 2019; Sung et al. 2019; Self 2019; Santolaya et al. 2019) and 4 non-experiments (Selvefors et al. 2018; Morkos et al. 2019; Roesler et al. 2019; Piccolo et al. 2019).

The use of case studies is pervasive in qualitative research (more than half the studies that classified as qualitative base the research on a case study). Furthermore, many quantitative studies support results from case studies; for example, some analytical studies in which case studies are used as proof of concept of the proposed models (Chen et al. 2019b; Zhang and Thomson 2019; Li et al. 2019a).

Nevertheless, other qualitative methods, such as ethnography, hermeneutics, action research and phenomenological studies, are also used. The use of specific methods related to design is scarce (the discussion about this concern is dealt with in detail below). Ethnography is used in three cases (Roesler et al. 2019; Van der Linden et al. 2019a; Comi et al. 2019)—also the annotation as observation in the tables—and one more paper uses ethnography as the study focus (Wood and Mattson 2019). Hermeneutics is used by (McDonald and Michela 2019; Cooper 2019; Lloyd 2019; Luck 2019).

Action research is used by Pakkanen et al. (2019) to investigate, in combination with case studies, modular systems in industrial environments. The work of Bresciani (2019) could be considered an action research study with the goal of building a grounded theory evaluation technique for visual thinking. McKinnon and Sade (2019) align their work in the field of research through design using a set of gadgets to obtain information about environmental home good practices. Research through design is also used by Genç et al. (2018) to explore new materials and Tsai and Van Den Hoven (2018) to explore user experience. Hyysalo et al. (2019b) and present the evaluation of a panel following the principles of research through design. Close to this method is that presented by Barati et al.(2019), who complement their study with workshops where a group of students explores their proposals.

3.3 Data-collection methods

Results regarding RQ3 (data collection methods) are summarised in this section. Table 3 shows which main methods and techniques for collecting data are used in the different studies. The analysis of the sources of information is completed with a revision of the instruments used to collect data and with a discussion about the role of human input presented in the following sections. None of the data-collection methods identified seem to be dominant in the papers studied.

Technical documents of diverse nature are the main source of information used (Table 2 reports 23 out of the 68 papers analysed using technical documents). Interviewing is also frequent (22 times reported in Table 2). Expert and user opinions are both used as sources of information, but neither is a majority (22 and 20 papers, respectively, reported in Table 2. Observation is mostly used in qualitative studies, where almost half use this technique. Concerning quantitative studies, apart from measurements, expert opinions appear as a frequent resource. This is because it is common to collect the opinions of experts in questionnaires or in evaluation templates that convert opinions into numeric values.

Verbalization is used in Martinec et al. (2019) and Gyory et al. (2019) for team work analysis and in (Khalaj and Pedgley 2019), where designers and users had to verbalize impressions.

Objects are collected as a data source in a relevant number of studies. Some are the results of students’ work as in Gralla et al. (2019); brainstorming outputs (Vegt et al. 2019); prototypes (Feijs and Toeters 2018; Barati et al. 2019), or commercial products (Roy and Warren 2019). Sketches are the type of object analysed in (Genç et al. 2018; Martinec et al. 2019; Gyory et al. 2019; Goucher-Lambert and Cagan 2019; Comi et al. 2019); while for (Li and Luximon 2018; Sung et al. 2019) sketches are the main concern of the research.

Questionnaires are less frequently used, and when this happens, they are designed ad-hoc for each study. Given the wide variety of topics and aims of the reviewed works, no standardised questionnaires have been found. Questionnaires, therefore, take different formats: Amazon Mechanical Turk is used once (Goucher-Lambert and Cagan 2019); a Likert scale tool evaluation (Graeff et al. 2019); binary and open questions (Pakkanen et al. 2019); ranking of preferences (Franceschini and Maisano 2019); or ad-hoc software tools (Li et al. 2019a).

Interviews are frequently used as a source of information in qualitative and mixed strategies of inquiry. Interviews are associated with phenomenological studies (Li and Luximon 2018; Park-Lee and Person 2018; Selvefors et al. 2018) and also in ethnographic studies (Roesler et al. 2019; Van der Linden et al. 2019a; Wood and Mattson 2019; Comi et al. 2019). The interviewed population can be a group of users of a given technology (Li and Luximon 2018) or a group of experts (Bresciani 2019).

Concerning the sample size used in the 24 papers whose research method has been classified as experimental, and taking into account that the sample may refer to studied objects or to participants/users, which, in turn, may be individuals or teams, the number of participants/users varies between 4, in Martinec et al. (2019), and 169, in Ozer and Cebeci (2019). The number of studied objects also varies from 6, in Mathias et al. (2019) to 256, in Li et al. (2019b). In Santolaya et al. (2019) a methodology is experimentally tested in 2 case studies.

3.4 Instruments

Results regarding RQ4 (instruments used to collect data) are summarised in this section. Measurements refer both to metrics obtained with a physical device and to qualitative ratings obtained from human-based scores. In the first group, we can mention the metrics of energetic consumption (Selvefors et al. 2018; Santolaya et al. 2019), mass material (Santolaya et al. 2019), volumes of objects (Mathias et al. 2019), displacement of buttons (Valverde et al. 2019), online shopping user interaction data (Ozer and Cebeci 2019), or the timing of tasks in (Mathias et al. 2019). In the second group, we can cite (Saliminamin et al. 2019; Gyory et al. 2019), which score the quality of design proposals, and (Franceschini and Maisano 2019), who use design preferences as the input for an analytical model.

Simulations and/or software developments of algorithms take on an important role in several papers. Belkadi et al. (2019) present a software tool; Chen et al. (2019a), Feijs and Toeters (2018), Mathias et al. (2019) and Takahashi et al. (2018) present or test software tools for different goals, such as analysing Lego buildings, and generating fashion patterns for projecting requirements into design parameters. Li et al. (2019a) focus on modelling knowledge; Piccolo et al. (2019) use analysis and visualization tools to present results; while Ozer and Cebeci (2019) and Saravanan and Jerald (2019) use machine learning techniques such as neural networks and clustering. De Lessio et al. (2019) present a software tool to support planning and Yang et al. (2019) to support manufacturing. Boussuge et al. (2019) propose using ontologies to capture high-level modelling and idealisation decisions, characterising the simulations of CAE models from CAD assemblies. Other papers related to ontologies use software to model them (Cheong and Butscher 2019; Hagedorn et al. 2019; Wang et al. 2019).

Workshops are frequently used for evaluating results and sharing experiences by a group of experts with discussions (Van der Linden et al. 2019a, b; McKinnon and Sade 2019; Self 2019; Wlazlak et al. 2019). In (Genç et al. 2018; Martinec et al. 2019), the workshops become designing activities in the research-through-design methodology. In Takahashi et al. (2018), workshops are used to observe users while they interact with a system and, in Pakkanen et al. (2019), to collect information from experts. In Garcia et al. (2019), workshops are meetings with stakeholders.

The opinions of stakeholders can be the core of the research study (Self 2019) or they can be used as part of usability tests (Takahashi et al. 2018). Most often, questionnaires and interviews are performed with users of a product (Selvefors et al. 2018; Roesler et al. 2019; Hanrahan et al. 2019; Ozer and Cebeci 2019); by active participants of the process under analysis, such as professionals in companies (Reimlinger et al. 2019; Wlazlak et al. 2019); or by students that are required to do a project (Vegt et al. 2019; Li et al. 2019a; Abi Akle et al. 2019; Graeff et al. 2019). The experts that participate in questionnaires or interviews are designers, architects, engineers (Li and Luximon 2018; Park-Lee and Person 2018; Pakkanen et al. 2019), or academic staff evaluating results (Morkos et al. 2019; Sung et al. 2019; McKinnon and Sade 2019). In interviews occurring in ethnographic studies, the subjects providing information could be considered the topic of analysis (Wood and Mattson 2019), but at the same time, they could be experts (Comi et al. 2019).

4 Discussion

4.1 Variety of aims and approaches

The principal finding of our research is that there is a very high diversity in the works we have analysed in the journals related to engineering design. This variety affects the aims and scopes of the research works, the methods, and the data sources. Table 4 shows that variety affects the papers in the four journals analysed with only minor differences among them. Thus, DS (Design Studies) and RED (Research in Engineering Design) seem to focus more on methodological aspects, while IJD (International Journal of Design) and JED (Journal of Engineering Design) focus more on delving into particular aspects of the design process or on products, but at most 7 papers out of the 17 falls into one of the categories. According to the results, DS and IJD journals attract more papers with a qualitative approach (only 2 papers in each journal are purely quantitative), while most of the papers from JED and RED follow a quantitative or analytical approach (only 3 and 7 papers, respectively, are purely qualitative). However, we have found papers with both approaches in all the journals. RED uses less self-reported data (interviews, questionnaires or workshops), while DS uses this source of data the most, but in both journals there are exceptions, such as the works of Mathias et al. (2019) in DS or Garcia et al. (2019) in RED.

Table 4 Number of works in the different journals
Full size table

Despite this broad spectrum of papers, we found a clear interest in methodologies and the in-depth analysis of a given aspect of the whole process of designing generally applied to a particular case study. The interest in both topics is justified by the nature of the design and the youth of the discipline. As a process of searching for optimum solutions, design is clearly related to methodological concerns. As a young discipline, the space for contributing to the different tasks of the whole design process is huge. The analysis of the process of engineering design has evolved from being considered from a purely technical perspective to being studied as a socio-technical process. From a technical point of view, (Beitz et al. 1996) distinguished between conceptual design and embodied design for identifying a list of tasks that contribute to facing problems of engineering design in an effective and systematic way. From a socio-technical perspective, different authors have pointed out that the design process is influenced by aspects related to teamwork capabilities (Dorst 2004), the inclusion of participants (Van der Bijl-Brouwer and Dorst 2017) or by the institutional complexity (Reich and Subrahmanian 2020). Our study shows that there is space for research works that focus on both perspectives of analysis, being found works that are closely related to tasks that affect conceptual design (Martinec et al. 2019; Benavides and Lara-Rapp 2019; Self 2019), embodied design (Petreca et al. 2019) and also to social aspects of the design process (Piccolo et al. 2019).

It has been observed that there are a relatively low number of papers proposing recommendations, guidelines, frameworks, and taxonomies. We understand how difficult it is generalizing and classifying a discipline with multiple tasks, agents, approaches and sub-domains. Nevertheless, generating these types of representations of knowledge could be a substrate for the growth of the discipline. Design is a context-specific endeavour, but trying to generalize results so that other authors could reuse the generated knowledge in other domains would be positive for the growth of the discipline. The selected papers include product development and engineering design, which are two different areas, albeit overlapping. Recommendations and guidelines are always useful for the practice of engineering design, but more importantly, classifying concepts and types of activities with frameworks and taxonomies is an essential process in the building of knowledge in any research area. The variety of aims and approaches is probably the reason for this deficit, but research in engineering design would benefit from works analysing the many methodologies proposed from a meta level that permits obtaining general concepts that are domain-independent and universally applicable.

Results presented in Table 2 and summarised in Table 3 could be used to derive patterns or preferred styles in research design. Papers using analytical approaches mainly use case studies to validate the proposed models and they use simulations to compare results with expectations. Here, the case studies are used as proof of concept of the proposed models. They do not consider human input as a main feature of analysis. The ones related to methodological concerns are the papers focusing on axiomatic design and the ones relating to specific aspects or to frameworks are the ones related to ontologies. Most papers with quantitative approaches use experimental setups in which they compare different configurations of a given problem. The means to collect numerical data highly depend on the type of work, with no outstanding method or instrument. This approach is mainly used when the goal is to study a given aspect of design, which is coherent with the fact that experiments are meant to measure variables that can be isolated, and therefore these studies need to focus on specific features of the design process. Like analytical papers, qualitative approaches are mainly based on case studies. The main difference is related to the nature of these case studies. In qualitative approaches, the case studies aim at gaining insight into the complexity of the studied design processes from the point of view of the participants. In consequence, the preferred data collection methods are observations and interviews and/or workshops, to collect data from users and experts. They use rich data sources (audio, photography, video or software tools) to make observations rigorously. Qualitative approaches are the most used methods, independently of the aim of the paper, but they are dominant for proposing frameworks of analysis or deriving guidelines and recommendations, probably because the active interpretation of experts is a must for these concerns. Papers using mixed methods triangulate the information obtained in quantitative experiments with information obtained with qualitative methods. Therefore, their pattern is closer to one of the papers using quantitative methods than to the ones using qualitative methods.

The application of one approach or another should respond to what Subrahmanian et al. (2020) call the different models of designing. When the artifact or the process is clear, analytical, and quantitative methods, closer to approaches followed in natural science can be applied. When people, culture, society, and politics must be taken into consideration, the use of analytical and quantitative methods is not appropriate. When individual designers play a role, and, especially, when social aspects and context must be taken into consideration, design processes become more complex and dynamic, involving aspects that are better studied by qualitative approaches that are able to capture the complexity of the object of study and the participants' perspectives.

4.2 Implications for the research in the engineering design community

As mentioned in the introduction, one of the objectives of this paper was to provide suggestions about the course contents that doctoral studies in the domain of engineering design must carry out. The first implication of our analysis relates to the type of research methodologies that students must be introduced to. According to the analysed papers, it seems essential that future researchers receive training in both qualitative and quantitative methods. The analysis shows that qualitative research is very common and that rich sources of data, such as observations or users and experts opinions collected through interviews are frequent. Furthermore, pure qualitative research approaches, like ethnography and phenomenology are commonly found. Nevertheless, experimental approaches should also have a relevant role in the student curricula because it is frequently used as well. We understand that this qualitative-quantitative duality responds to the nature of engineering design, a complex field that requires both technical background and the consideration of behavioural and social aspects related to design.

A second implication has to do with the instruments and data collection methods that researchers on engineering design must get familiar with. Research studies in this domain could require accessing real design scenarios that are authentic field studies rather than controlled lab studies. This is a relevant divergence with respect to other research domains that permit isolating variables and participants. There are implications for the instruments used for collecting data, with the need of considering techniques that permit collecting information in real settings and during longer periods of time. but also, that human fact is a relevant variable that affects both design teams managements, communication with users and social aspects. This fact justifies the use of technical reports, questionnaires, and observation as the main sources of information in these studies.

It must be noted that publishing in a journal should not be an end in itself, and the real value of a paper does not rely on the journal in which it is published but on its contribution to the growth of the discipline (Bladek 2014). However, there is a universal tendency to identify research quality and impact with these publications, and students that pursue a research career usually need to accomplish certain goals related to publishing. For this reason, we think that doctoral students in engineering design can find this work useful, as it provides an overview and pointers to different types of research work published in four top-quality journals in the field, and this may give them tips on the kind of knowledge they need to acquire to have their work published in these journals or similar ones.

4.3 Relation to other surveys

Probably due to the youth of engineering design as a research discipline, the number of papers devoted to literature reviews in these fields is still sparse. From the few reviews found, most refer to particular aspects of engineering design: such as inspiration and fixation (Crilly 2019); sustainability (Coskun et al. 2015); user value (Boztepe 2007); Alzheimer and play experience (Anderiesen et al. 2015); performance in industrial design(Candi and Gemser 2010); relation between creativity, functionality, and aesthetics (Han et al. 2021); fuzzy front-ends for product development (Park et al. 2021); surrogate models and computational complexity (Alizadeh et al. 2020); smart design (Pessôa and Becker 2020); design and poverty (Jagtap 2019); mass customization (Ferguson et al. 2014); product stigma (Schröppel et al. 2021); uncertainty (Han et al. 2020); decision-making methods (Renzi et al. 2017); modular product design (Bonvoisin et al. 2016); or product-service systems (Vasantha et al. 2012).

More interesting, for their similarity with respect to the present study, are the works presented by Tempczyk (1986) and Cantamessa (2003), both presenting reviews or surveys about research and studies on engineering design. These two works and the one presented in this paper differ in their sources of information. Tempczyk (1986) made a survey by sending questionnaires to academic staff concerning research subjects and methods; Cantamessa (2003) made a review of the proceedings of two editions of the conference on engineering design. There is a temporal distance of 17 years between the work of Tempczyk (1986) and the one of Cantamessa (2003) and 18 years between the work of Cantamessa (2003) and the present study, but we must highlight the fact that the three studies report methodologies as one of the main topics of research. Computer-aided products are reported by Tempczyk (1986) as a relevant topic, and Cantamessa (2003) also refers to software tools as a recurrent topic, while we also identified a category named simulation which included software tools and algorithms. The three works also report a high variety of approaches and themes. The main difference between these studies and the present one is that Tempczyk (1986) reports on training as an important concern for researchers and Cantamessa (2003) observes different streams of research, loosely coupled with an excess of referencing to previous works. As regards references to training concerns, we did not find any paper related to training, probably because, nowadays, there are journals specifically devoted to learning in the domain of engineering and design. As regards the criticism of Cantamessa (2003) concerning the notable amount of self-references in the analysed papers, we did not observe such a circumstance in the journal papers we have reviewed. On the contrary, our review has found that the papers reviewed contain complete state-of-the-art sections in which other research groups are referenced and other studies are discussed. This finding partially contradicts what Cantamessa (2003) found in his review. We think that the nature of the sources of data in his review, based on proceedings which are shorter could have influenced these divergent results. Our study may point to a more mature stage of research that builds on the knowledge already offered in the community. This finding may be based on the fact we are working on journal papers that offer more mature results.

4.4 Limitations

The systematic literature review presented in this paper covers a recent period of time spanning one year of publications. The sample is representative of recent research in engineering design, but it does not provide information about tendencies in the field. For example, we have observed a relevant number of quantitative studies in comparison to qualitative ones, but we cannot say if this is a tendency. Future work would be required to compare our results with those of a longitudinal study covering a larger period of years. We expect that our work can be considered as the first step in this longer-term study that could provide useful information about the evolution of research into the young discipline of engineering design.

By selecting Blessing and Chakrabarti (2009) as a framework to categorize research papers, we did not pay attention to the important concern of the success of the research which could be a critical point for connecting the study aim, with the approach, research method, etc. Reich and Subrahmanian (2021) show that it is possible to use the PSI framework (Problem, Social and Institutional space) to describe what researchers and designers did in case studies to analyse the matching of methods, aims and approaches with the success of the projects. In spite of our work being merely descriptive of the aims, methods and techniques used by authors, we offer a corpus of categorised research papers for analysing in future works on whether the research design is appropriate for its goals.

The analysis of the sample of journal papers selected has permitted us to build a consistent set of categories for classifying research works in engineering design. We consider this sample comprehensive, based on a saturation analysis carried out on the sample, that showed that all the categories used in the analysis could be identified with 69% of the papers that were actually used in the analysis. Nevertheless, while selecting 68 papers from only four journals, we could have discarded other works that could include other alternative approaches also valid for research in engineering design. Moreover, the choice of a single year-window is another limitation of this study, as it does not enable us to provide a full vision of the field and its evolution. Nevertheless, we think that the classification presented in this paper could be the basis for subsequent studies, which should consider a broader timeframe, and therefore, a larger selection of papers across several years. Other approaches for selecting the analysed papers like sampling at the same rate in all the journals could also have led to representative results.

5 Conclusions

In this paper, we have presented a systematic review of recent literature on research methods and instruments used in a one-year period of research papers in the field of engineering design. By taking this approach, we offer a "fixed image" of recent research in the area and point to some gaps and challenges in the field.

The review shows that there is no single methodological approach accepted as the standard in the field; and that there is a large variety of goals, approaches, data collection methods and instruments to collect them. In spite of this variety, we have observed a certain preference towards qualitative methods, which can be justified by the increasing consideration of engineering design as a complex process affecting humans and their contexts.

We think that this paper contributes to research in engineering design by providing initial evidence for researchers about the kind of work that are expected by high-impact scientific journals in this domain. Additionally, academics can find in this paper a list of topics (methodologies, data-collection procedures, instruments, etc.…) that must be part of the programme of courses on research in engineering design.

6 Appendix: Coding scheme: categories and examples

The tables included in this Appendix have aim to present the knowledge generated in this paper in the form of a coding scheme, that can be used as an instrument to describe the taxonomy of research aims (Table 5), approaches (Table 6), data collection techniques (Table 7), and instruments (Table 8) in engineering design.

Table 5 Categories and examples of works corresponding to the topic of aim of the research
Full size table
Table 6 Categories and examples of works corresponding to the topic of research approaches
Full size table
Table 7 Categories and examples of works corresponding to the topic of data collection techniques
Full size table
Table 8 Categories and examples of works corresponding to the topic of instruments
Full size table