Introduction

There is a broad consensus on the importance and advisability of testing laboratories adopting a Quality Management System (QMS) to support their work, no matter it is industrial oriented or a research oriented. However, laboratories involved in R&D testing have specific difficulties to implement a QMS due to the peculiar nature of their activity. Researchers and professionals have long discussed about the advisability of implementing a Quality Management System (QMS) in research testing laboratories. From the late 1990s (when ISO/IEC 17025 was first published [1]) to the present, the analysis on how these laboratories adopt quality management practices has gone through aspects such as the difficulties found, the critical success factors or the key indicators in the process of implementing a QMS. Also, there is a feeling that a QMS as stated in the existing standards does not offer a complete response to the needs of research testing laboratories in terms of scientific competence.

However, authors still agree on the benefits of a QMS on the research activity. Thus, the point is how to overcome the difficulties and how to incorporate scientific competence requirements to the traditional schemes for QMS in testing laboratories.

The first part of this paper includes a literature review focused on the hot topics regarding QMS in research testing laboratories: advantages and benefits of implementing a QMS; difficulties and limitations when implementing a QMS; and success factors for the implementation of a QMS. After the literature review, a questionnaire performed in collaboration with REDLAB (Red de Laboratorios de la Comunidad de Madrid) regarding QMS in research testing laboratories is presented. The results of this study support the findings in the literature review, and complete the picture of the difficulties and challenges found by research testing laboratories.

In the second part, two relevant standards are analyzed: ISO/IEC 17025 General requirements for the competence of testing and calibration laboratories [10] and UNE 166002 R&D&i management: R&D&i management system requirements [25]. The first is definitely the reference for any testing laboratory, no matter its scientific or industrial nature. The latter is used for this work as a basis to establish requirements for the scientific competence, which are not addressed by ISO/IEC 17025. The analysis of these standards results in a complete set of competence requirements for research testing laboratories.

At this point, the third part of the paper presents the proposal of an agile methodology that aims to fulfil the defined set of competence requirements trying to overcome the difficulties and limitations found.

Methodology

The objective of this work is giving a response to the three following research questions (RQs):


RQ1: Is there a real difference between industrial testing laboratories and research testing laboratories in terms of quality management?


RQ2: Is there an adequate normative context for the definition of a QMS is research testing laboratories?


RQ3: Is it possible to define a model for a QMS that overcomes the difficulties and limitations that research testing laboratories find when implementing and maintaining a quality management system?

RQ1 has been addressed by performing a systematic literature review based on Web of Science database. Also, a questionnaire regarding QMS aspects has been delivered to testing laboratories from REDLAB (Red de Laboratorios de la Comunidad de Madrid/ Testing Laboratories Network in Community of Madrid).

RQ2 has been addressed by reviewing the two relevant standards ISO/IE 17025 General requirements for the competence of testing and calibration laboratories and UNE 166002 R&D&i management: R&D&i management system requirements.

For RQ3, a model for a QMS based on agile principles has been defined. The model meets the technical and scientific requirements identified after the review of the relevant standards, which is a proof that may fit the purpose. Also, the agile focus has extensively proved to be a valid methodology for complex environments.

Literature review

Advantages and benefits of implementing a QMS

The convenience of implementing a QMS in research testing laboratories is widely recognized. The following are the main advantages and benefits found in the literature:

  • Need to count with quality management methods similar to those in the industry, in order to have the possibility of becoming supplier, subcontractor or partner ([2,3,4, 9, 21, 23, 24]);

  • Promotion of a mutual confidence among all parties with cooperation or funding purposes (customers, sponsors, scientists, authorities) ([3, 5, 9, 19,20,21,22, 24]);

  • Assurance of the technical and scientific competence ([2, 3, 5, 9, 18, 19, 22,23,24]);

  • Assurance of comparable research results, inside the laboratory during the phases of a project, or with other laboratories ([2, 3, 5, 19, 21, 24]);

  • More efficient management of the scientific and technical activities in the laboratory ([3, 20, 24]);

  • Improvement of the structural organization thorough a better definition of functions and responsibilities ([3, 20]);

  • Improvement of the equipment control [20, 23];

  • Improvement of existing working habits [19];

  • Promotion of the knowledge management and staff qualification ([3, 21, 22, 24]);

  • Improvement of staff commitment and satisfaction [24]

Difficulties and limitations

Once recognized the convenience of having a QMS in research testing laboratories, the point is that professionals find a number of difficulties in the implementation and maintenance. The main issues identified by authors are the following:

  • The excessive rigidity of a QMS limits the creative work which is strongly attached to research ([2, 3, 9, 23]);

  • The excessive rigidity of a QMS increases bureaucratic work and paperwork ([2, 7, 19, 23, 24]);

  • The complexity of the research activity (with changing requirements, multiple groups, technical uncertainty) is hardly compatible to a QMS ([3, 5, 9]);

  • Lack of specific standards for the definition of a QMS in research organizations ([2, 3, 5, 6, 7]);

  • Research results are not limited to a test results, but include scientific production [3];

  • Difficulty to measure the cost of “non-quality”, and so it is difficult to justify the investment of resources in quality management tasks ([9, 24]);

  • Lack of training in quality management among the researcher staff ([9, 24]);

  • Lack of commitment to quality management among the researcher staff and management staff ([5, 9, 24]);

  • Lack of human resources dedicated to support the QMS ([20,21,22]);

  • Short-term contracts and high turnover ([19, 21, 22]);

  • Resistance to change ([24])

Success factors for implementing a QMS in research testing laboratories

The existing difficulties and limitations have pushed authors to reflect on the factors to take into account to successfully implement a QMS in testing research laboratories. These success factors are the following:

  • “bottom-up” design of the system, in order to reinforce awareness and commitment of the staff [8];

  • Simple, flexible and well-adapted documentation system ([9, 19, 21]);

  • Modular and non-redundant system [9];

  • Self-sustainable system ([9, 21]);

  • The QMS must provide added value to the laboratory ([9, 21]);

  • The QMS must consider not only general quality management aspects, but also specific aspects such as scientific competence, creativity-flexibility balance [5];

  • Tailoring of the QMS to the peculiarities of the laboratory ([19, 21, 23]);

  • Promotion of a culture of quality ([19, 21]);

  • Management commitment [19];

Normative context

The reference standard for QMS in testing laboratories is ISO/IEC 17025. Numerous authors have analyzed the positive influence of having an implemented QMS according to ISO/IEC 17025 on laboratories performance [2]. However, many of them have called for the development of specific standards for research testing laboratories, which has not happened up to date. Today, two standards are used by testing laboratories as a reference for their QMS: ISO/IEC 17025 [10] and ISO 9001 [11]. Both ISO/IEC 17025 and ISO 9001 address aspects related to quality management. However, important differences exist between these two standards. While ISO/IEC 17025 defines general requirements for the competence of testing and calibration laboratories, ISO 9001 establishes requirements for a quality management system in any kind of organization, no matter the sector or the kind of activity being developed. In this line, ISO 17025 addresses technical and management requirements for the demonstration of the competence of testing laboratories, while ISO 9001 develops the requirements for the demonstration of the ability to provide products and services that meet the customer and regulatory requirements. ISO/IEC 17025 requirements contain the ones established by ISO 9001, and so the compliance to ISO/IEC 17025 principles implies the compliance to ISO 9001 principle (and not vice versa). As a last basic difference, it must be said that external recognition of a Quality Management System is subjected to a certification process in the case of ISO 9001, and to an accreditation process in the case of ISO/IEC 17025 to guarantee technical competence. [12,13,14,15] (among others) opt for ISO/IEC 17025 as a reference standard for research testing laboratories and recognize that the accreditation of a QMS against ISO/IEC 17025 adds value to the certification against ISO 9001. Cammann et al. [3] referred to Eurachem Guide [16], the guide for Quality Assurance for R&D and Non-Routine Analysis in the analytical chemistry field, based on the idea that laboratories performing non-routine measurements require a special approach in terms of quality management. Also, the British Department for Environment, Food & Rural Affairs published in May 2003 the “Joint Code of Practice for Research” [17], that applies to contractors funded by a number of British bodies, and addresses aspects related to the quality of research process and the quality of science, such as responsibilities, competence, project planning, quality control, health and safety, handling of samples and materials, facilities and equipment, documentation, records and field-based research.

The literature review suggests that these available standards do not consider the special difficulties, limitations and needs of research testing laboratories regarding quality management. In this work, the standard UNE 166002 [25] R&D&i management: R&D&i management system requirements is proposed as a basis to complement the scheme proposed by ISO/IEC 17025.

The purpose of UNE 166002 is to establish guidance and requirements for a management system based on the PDCA (plan-do-check-act) cycle, and suitable for any kind of organization involved in R&D&i. UNE 166002 addresses five general topics: context of the organization; leadership; planning; support to R&D&i; operational processes of R&D&i. There is a coincidence between ISO/IEC 17025 and UNE 166002 in the management of general aspects, and the latter includes a set of requirements that are not considered by ISO/IEC 17025. These requirements have to do with management of ideas, R&D&i vision and strategy, R&D&i policy and culture of innovation. Thus, the combination of ISO/IEC 17025 and UNE 166002 seems to be a good package as a standard framework for research testing laboratories.

Questionnaire

A study was carried out in collaboration with RedLab (Red de Laboratorios de la Comunidad de Madrid, Network of Laboratories of the Community of Madrid). The objective was to confirm the findings from the literature research in a working environment.

RedLab is an initiative of the General Directorate of Universities and Research founded in 2000 with the aim of bringing together the testing and calibration laboratories belonging to research centers and universities, disseminating their activity and supporting them in matters such as the quality and knowledge management. Currently 340 testing (300) and calibration (40) laboratories operating in Madrid (Spain) are members of this network. All the laboratories under the scope of this study are involved in R&D activities, since Redlab groups laboratories from universities and public research centers.

The questionnaire on which the study is based was distributed by RedLab to its members through the free access platform Typeform. The 40 questions in the questionnaire were grouped into seven blocks: (I) information about the respondent, (II and IV) information about the QMS implanted in the laboratory (maturity), (III) information about the tests carried out in the laboratory, (V) information on the critical points of the QMS, (VI) assessment of the QMS, (VII) benefits of the QMS. The questions were posed in different formats depending on the type of response expected: free text, form with a single answer, multiple answer form, numerical answer (0–10).

Participants description

378 people visited the questionnaire at Typeform. 115 valid and complete responses were received corresponding to testing and calibration laboratories. From these, responses from calibration laboratories were not considered for the purpose of this study, since the present work refers just to testing laboratories. After this filter, 88 responses corresponding to different laboratories were left, which is 29,33 % of the testing laboratories affiliated to RedLab. 2 out of the 88 laboratories declared not to have a QMS implanted. So, the analysis was done on 86 testing research laboratories.

The information obtained from the questionnaire was considered to be valid based on two aspects: the professional profile of the participants and their expertise in quality management systems.

Professionals who completed the questionnaire declared to be involved in the QMS implantation and maintenance. 76,74 % of the participants were laboratory managers and quality managers. The rest of them were technical managers, project managers and coordinators.

80,23% of the participant laboratories declared to have a QMS implanted before 2013, which means a system with an over four-year life. Four years were considered to be an adequate period to admit a relevant expertise in quality management for several reasons. In a four-year cycle, a laboratory has typically closed a quality assurance plan, one (at least) calibration plan, one (or several) management reviews and one (or several) internal audits. Thus, in this period, the laboratory has had the opportunity to identify its weakness and to adapt the system to the activity. Only 3,49% of the participants declared to have implanted a QMS in the last year. So, major part of the participants was considered to have a solid experience in QMS.

Analysis of the results

As a previous step to the analysis, the participants were classified according to two criteria:

  • The nature of the test methods (standard or non-standard, being non-standard those methods that are not recognized by standards, and thus require validation);

  • The routine nature of the activity (the laboratory performs repeatedly the same set of tests).

For the classification, participants were asked to provide information about the nature of the test methods used at their laboratories and the routine nature of the activity performed. On this basis, they were allocated in four groups: laboratories that perform tests according to standard methods on a routine basis (group 1); laboratories that perform tests according to standard methods on a non-routine basis (group 2); laboratories that perform tests according to non-standard methods on a repetitive basis (group 3); laboratories that perform tests according to non-standard methods on a non-repetitive basis (group 4).

Laboratories in group 1 do not perform a research activity itself even though they support research organizations, since their activity is based on pre-defined validated methods, and they always execute the same set of tests. On the contrary, the activity developed by laboratories in group 4 implies the validation of methods and a continuous adaptation to execute different kind of tests, and so these are considered to be real research testing laboratories.

For the purpose of this work, the two groups of interest are groups 1 (which has a clear industrial-oriented activity) and 4 (which has a clear research itself –oriented activity). Group 1 is labelled as “Industrial group”, and group 4 is labelled as “Research group”. Table 1 shows the most relevant results obtained through the questionnaire, referring to:

  • Number of laboratories that have a QMS implemented under a specific scheme (ISO 9001; ISO/IEC 17025; other scheme; no QMS implemented);

  • Number of laboratories with an external recognition of the implemented QMS (ENAC accreditation; certification; none; other);

  • Number of laboratories that have a specific difficulty in the implementation of the QMS. This question was designed as a multiple choice question: laboratories could mark several options,

  • Degree of compliance to QMS requirements. This question was designed as a numerical answer in a 0–10 scale, being 0 “no compliance at all to the requirement” and 10 “absolute compliance to the requirement”;

  • Valuation of the QMS by the managerial and technical staff. This question was designed as a numerical answer in a 0–10 scale, being 0 a very negative valuation and 10 a very positive valuation;

  • Benefits of the QMS. This question was designed as a numerical answer in a 0–10 scale, being 0 “no recognized benefit in this aspect” and 10 “absolutely recognized benefit in this aspect”.

Table 1 Results from the questionnaire
Full size table

For the 0–10 scale questions, the table shows the mean values of the recorded answers.

After the results, a set of interesting observations were made:

  • QMS is a widely use tool, no matter the industrial or research nature of the laboratory;

  • Most of the research testing tools base their QMS on ISO 9001 instead of ISO/IEC 17025;

  • Greatest difficulty found by the professionals in the implantation of a QMS is the control of documentation;

  • Laboratories from group 1 meet quality assurance requirements in a higher degree than laboratories from group 4;

  • Technical and managerial staff from group 1 appreciate the benefits of a QMS more than those from group 4;

  • The most important benefit from the implementation of a QMS is the assurance of quality in the case of participants from group 1; however, the most important one is the knowledge management for group 4;

These basic observations reinforce the findings in the literature, and support the idea that there are differences in the approach to the QMS in testing laboratories depending on the industrial or research nature, and that there are clear key points to be improved in the implementation of a QMS.

Proposal of a QMS for research testing laboratories

After the literature review, the questionnaire results and the normative context, the result of this work is the proposal of a model for a Quality Management System for research testing laboratories. This model has been designed under the following principles:

  • Compliance to general competence requirements for testing laboratories established by ISO/IEC 17025;

  • Compliance to specific competence requirements for R&D&i organizations established by UNE 166002;

  • Consideration of the difficulties and limitations reported by authors and professionals in a research context.

Requirements, objectives, resources and planning change and evolve in any research, making it difficult to normalize activities and define rigid procedures. Activity in a research testing laboratory has these characteristics (which are similar to the ones attributed to projects), and this is the reason why a QMS based on standard procedures is not suitable for a research testing laboratory. At this point, the agile approach for QMS raises. These methodologies have been successfully implemented in quality assurance for software development projects, due to the fact that the agile approach deals with changing requirements and uncertain environments, which is similar to the situation found at research testing laboratories.

Thus, an agile approach for the QMS in research testing laboratories is proposed, based on the agile principles [26]:

  • Need to adapt to changing environment, versus the strict observation of a closed planning;

  • Incremental and cooperative execution of activities;

  • Priority of individuals and interactions over processes and tools;

  • Tight communication with parties involved in the activities;

  • Focus on motivated individuals;

  • Constant focus on technical excellence;

  • Regular reflection on the own activity to adjust and improve habits and procedures.

To be consistent to these principles, the model is based on the celebration of several events integrated in the testing activity milestones that act as a trigger to quality management tasks.

The proposed model includes:

  • A set of competence requirements;

  • A set of events: test readiness review (TRR), test follow-up review (TFR), post-test review (PTR) and management review (MR).

Figure 1 summarizes the QMS model, including the inputs for the definition, the agile principles taken into account and the proposal itself.

Fig. 1
figure 1

Proposed quality management system model

Full size image

Competence requirements

After the analysis of the normative context, and taking into consideration the experts claims, a QMS exclusively based on ISO/IEC 17025 does not offer a complete response to research testing needs. In our proposal, requirements from ISO/IEC 17025 are completed with those from UNE 166002. As a result, four groups of requirements are set:

  • General requirements do not differ from those proposed by ISO/IEC 17025 (impartiality and confidentiality);

  • Resource requirements include those proposed by ISO/IEC 17025 and incorporate the need to create and maintain a R&D&i management unit and R&D&i units as defined by UNE 166002;

  • Process requirements include those proposed by ISO/IEC 17025 and incorporate the need to issue a test plan that must cover the following points: objectives and expected results, material and non-material resources, milestones, risk identification, support activities (technological surveillance, competitive intelligence);

  • Management requirements do not differ from those proposed by ISO/IEC 17025;

  • Research activity management requirements are incorporated as a new group, including the following: management of ideas, R&D&i vision and strategy, R&D&i policy and culture of innovation.

Table 2 shows the proposed set of requirements. Those coming from ISO/IEC 17025 are identified with the label in the standard. The new ones are identified with a sequential label with the format INV-n. Figure 2 is based on the schematic drawing according to ISO/IEC 17025 for the operational processes in the laboratory. Shaded elements refer to the resources and requirements in the proposed model (Table 3).

Table 2 QMS requirements
Full size table
Fig. 2
figure 2

Schematic drawing of the QMS requirements

Full size image
Table 3 TRR, TFR and PTR summary
Full size table

Events

As aforementioned, an agile approach for the QMS is proposed, in order to achieve two main goals:

  • Enabling the integration of the QMS in the day to day routine, adapting the system to the real needs of the laboratory, promoting the commitment of the key personnel and searching for the self-sustainability of the system;

  • Removing the unnecessary quality requirements, by putting the focus on the test as a trigger of the quality events.

Three events are suggested around the test: the Test Readiness Review (TRR), the Test Follow-Up Review (TFR), and the Post-Test Review (PTR). Necessary attendants to these meetings are the laboratory manager, the R&D&i unit manager, the test engineer and the quality assurance manager. Optionally, the customers and partners may attend.

Test Readiness Review (TRR) The main purpose of the TRR is to ensure that all the necessary conditions for starting the test are met. The TRR meeting addresses management aspects (review of customer request for test, laboratory quotation), technical aspects (assurance of the EUT Equipment Under Test readiness for the beginning of the tests, readiness of measurement equipment and facilities, review of the staff qualification, risks assessment), scientific aspects (research line, scientific objectives and context). The output from the TRR includes the declaration of the EUT, measurement equipment and facilities readiness; the testing method validated and the declaration of qualified staff.

Test Follow-Up Review (TFR) TFR purpose is to enable a meeting point for all the parties to follow-up the test progress and review the evolution of the technical and scientific relevant aspects, such as changes in the test requirements, evolution of the EUT, risks plan update, partial results to be transferred to activities for dissemination and exploitation of scientific results (publications, seminars, others). Any change in the management, technical or scientific aspects that were approved at TRR and are reviewed at TFR must be conveniently recorded as an output of TFR.

Depending on the complexity of the test, the celebration of several TFRs may be useful for a close and efficient tracking of the activities.

Post-Test Review (PTR) The main purpose of the PTR is that all the necessary conditions for the closure of the test are met, and to compile the knowledge generated during the test. PTR must address the identification of deviation and non-conformances, the presentation of the final test results, the review of the scientific objectives planned at TRR and TFR, the exploitation of the test results. Also, knowledge management actions must be undertaken: record of lessons learned, planning of dissemination activities and customer satisfaction evaluation.

Since TRR, FTR and PTR are events triggered by the test evolution, holding these reviews is a natural action that serves the key activity, which is the test itself. So, quality management becomes integrated in the day-to-day activity of the laboratory, which turns into an increase in the staff commitment, a better adaptation to the real needs and a self-sustainability of the system.

A fourth event which is not triggered by the test itself is considered in the model. This event is the Management Review (MR), which must be held on a regular basis (typically once per year) and is oriented to strategic and managerial aspects. The main purpose of the MR is the review of the QMS by the managerial board (MB). MR must address all the key points that require the managerial commitment, including (but not only): the policy and strategy review (including the update of general objectives and scientific objectives), the performance assessment (based on results of internal and/or external audits, performance indicators and feedback from customers), the evaluation of the scientific impact, the assurance of the quality of the tests results, the review of actions (preventive, corrective, actions for improvement), the knowledge management initiatives, and the evaluation of suppliers.

Conclusions

This work was triggered by three research questions regarding quality management in research testing laboratories:


RQ1: Is there a real difference between industrial testing laboratories and research testing laboratories in terms of quality management?


RQ2: Is there an adequate normative context for the definition of a QMS is research testing laboratories?


RQ3: Is it possible to define a model for a QMS that overcomes the difficulties and limitations that research testing laboratories find when implementing a maintaining a quality management system?

After applying the designed methodology, conclusions are:

RQ1: yes, there is a real difference between industrial testing laboratories and research testing laboratories in terms of quality management, as revealed by the literature review and supported by the results of the questionnaire. Research testing laboratories have specific difficulties, limitations and needs.

RQ2: no, there is not an adequate normative context, at least grouped on a single standard that addresses the dual nature of a research testing laboratory, as a testing laboratory and a R&D&i organization. The combination of two standards (ISO/IEC 17025 and UNE 166002) has been considered as a basis for this work.

RQ3: a model for QMS in research testing laboratories has been proposed. This model is the result of considering the difficulties and limitations reported by experts and professionals when implementing and maintaining a QMS in research testing laboratories, the success factors for the implementation and the agile principles. The model includes a set of competences requirements that follow the recommendations from ISO/IEC 17025 and incorporate the research and scientific approach from UNE 166002, and a set of reviews that enable the self-sustainability of the system and enable meeting points for the compliance to the aforementioned requirements.

The model has been built keeping in mind the following key aspects:

  • Observing the competence requirements for testing laboratories;

  • Simplifying the QMS and proposing a flexible approach;

  • Optimizing paperwork by reducing documentation and incorporating habits of continuous review;

  • Implanting following-up milestones to adequately manage the complexity of the research testing activities;

  • Promoting the innovation and communication culture;

  • Obtaining the maximum scientific return;

  • Adopting a self-sustainable model, in which the ordinary activity is a feedback for the maintenance of the QMS, thus reducing the resources dedicated to this task and improving the efficiency of the system.

Through this work, several references to the technical and managerial staff in relation to the implementation of a QMS have been done. The improvement of the staff qualification, commitment and satisfaction has been identified as one of the benefits from a QMS. On the other hand, the lack of training in quality management and the lack of commitment have been identified as difficulties for the implementation. Thus, this is a case of a vicious circle. The agile structure of the proposed model, built around the events (TRR, FTR, PTR and MR), aims to break this circle by involving the technical staff in the day-to-day maintenance and improvement of the system, and promoting the commitment of the managerial staff, which for sure is a success factor for the implementation of the QMS. Also, the specific needs (especially those related to scientific competence) and difficulties found by research testing laboratories have been taken into account.

There is a need that research organizations adopt QMS as an asset (and not as an obligation) to improve not also the management, but also the technical and scientific competence. This work, as a first step of our research, has tried to propose a tool to contribute to the success of a QMS in a kind of research organization, as research testing laboratories are.

Further research and limitations

The study based on the questionnaire refers to a reduced sample corresponding to testing laboratories from RedLab (Red de Laboratorios de la Comunidad de Madrid, Network of Laboratories of the Community of Madrid). Data have not been collected nor analyzed under strict sampling and statistical rules. They cannot be interpreted as concluding results, but only as a support to the findings in the literature review.

Further research will include the verification of the model with experts, and the subsequent iteration on the proposal.