1 Introduction

Project management (PM) is an established multi-disciplinary knowledge area with widely accepted methodologies and ample body of research, which, however, has two major points of debate: explanations of project performance and theoretical foundation (Padalkar and Gopinath 2016). Discussions on project performance start from project success and failure criteria (Atkinson 1999), and proliferate in the dimension of critical success factors (Cooke-Davies 2002; Müller and Jugdev 2012). In order to conceptualize empirical research findings and link notions of success criteria and success factors, general business excellence model was adapted to project management context (Westerveld 2003) and later augmented with organizational aspect of project quality to form a quality-based project excellence assessment model (Basu 2014). However, only a few studies have quantitatively demonstrated relationships between elements of project quality and other project performance dimensions (Harter et al. 2000). Therefore, there is a research gap between the conceptual holistic understanding of project excellence and empirical evidence to support it. In order to fill this research gap, current study adopts a systems perspective to quality (Kuhn 1974).

The research question is formulated as follows: how elements of a project management quality system are interrelated to achieve project excellence? To answer this question, a descriptive model of a project management quality system is created for the generic pharmaceutical research and development (R&D) setting, and the interconnections between its elements are studied in the light of capabilities relationships theories.

2 Theoretical Background

2.1 Theories

In the quality management literature, quality is mostly seen as a dimension of competitive advantage (Flynn et al. 1994). Trade-off theory (Skinner 1969) stipulates that competitive capabilities are contesting in nature and improvement in one dimension would imply a trade-off in another. This theory was partially supported by empirical evidence; however, accumulating counter-examples have led to the emergence of another research stream in capabilities relationship field—cumulative capabilities theory (De Meyer et al. 1989). This theory emphasizes a particular order in the development of competitive capabilities that is important to achieve sustainable improvements. It uses quality as the foundation for further capabilities acquisition, followed by dependability, speed and cost efficiency.

In the context of New Product Development (NPD) project management research, and supported by classical project management theory (Graves 1989), the existence of trade-offs between dimensions of NPD performance is largely assumed (Smith and Reinertsen 1997). However, empirical evidence regarding these trade-offs is mixed (Swink et al. 2006). It has been proposed that specific techniques can simultaneously improve NPD cost, time and quality by reducing waste in NPD project management activities via a higher level of processes maturity (Harter et al. 2000). This view is congruent with cumulative capabilities theory so that quality of project management processes form the foundation of achieving next levels of performance frontier. At the same time, due to diminishing returns from quality investments after a certain quality level is reached, there is global performance frontier that is established by state-of-the-art NPD project management (Swink et al. 2006).

Another similar approach is seen in excellence models (EFQM 2013; Westerveld 2003) that consist of Enablers (processes, behaviors, products and services, strategy, resources, and partnerships) and Results (for business, customers, people, and society). A causal relationship is assumed between Enablers and Results, and a positive feedback loop from Results to Enablers is achieved by learning, creativity, and innovation.

Mentioned theories provide a structure for studying the impact of project management processes quality on other project excellence dimensions. More specifically, they allow to conceptually describe functional-level quality management system in NPD project management in general and pharmaceutical R&D project and portfolio management in particular that can be used for systematic assessment of relationships between system components.

2.2 Pharmaceutical Project Management Quality System Model

In order to study the structure of the quality system, a Pharmaceutical Project Management Quality System (PPMQS) Model was created (Fig. 1).

Fig. 1
figure 1

Pharmaceutical project management quality system model. Coding: simple text—own definition by analogy; text with single asterisk (∗)—adaptation from (Friedli et al. 2017); text with double asterisk (∗∗)—own sub-categories and suppliers choice; underlined text—categories based on (Holmes and Walsh 2005); double underlined text—categories based on (ISO 2015, 2017) (“Information” is a compound category from several sources); grey background—own compilation of Enabling system

Full size image

This model is primarily based on the Pharmaceutical Production System Model (Friedli et al. 2017) and consists of Enabling (processes and behavior) and Result (outcomes and their metrics) systems.

Enabling system (Fig. 1, categories A, B, and C) was changed to suit project management context. It is based on IMSI Project Management Assessment Model (Holmes and Walsh 2005) and international standards for quality management in projects (ISO 2017) and portfolio management (ISO 2015). Individual enablers in category A (Fig. 1) represent project management knowledge areas. The system of category A enablers defines Project Management Quality Maturity (category B). Project governance (category C) represents portfolio management processes (ISO 2015) as viewed from a project management perspective.

Result system (Fig. 1, categories D, E, and F) was developed by analogy to Pharmaceutical Production System Model and consists of several aggregation levels of quality metrics that define Pharmaceutical Project Management Quality System (PPMQS) Excellence. At the lowest level of aggregation, individual quality metrics form compound scores of internal Operational Stability and external Supplier Reliability (categories D). Current study adopts product life-cycle approach for Effectiveness measurement of new product development (Suomala 2015). Therefore, Operational Stability incorporates post product launch metric of product quality, and Supplier Reliability category is derived from commercial supplier performance metrics. PPMQS Effectiveness (category E) is a compound score of Operational Stability and Supplier Reliability (categories D). PPMQS cost Efficiency metric (category E) is defined as a proportion of project management costs in overall product development costs. PPMQS Excellence (category F) comprises both PPMQS Effectiveness and Efficiency (categories E).

2.3 Hypotheses

Based on the quality systems theories, the following hypotheses were formulated for testing in the pharmaceutical generic R&D project management context:

H1

There is a positive link between PPMQS Effectiveness and Efficiency.

As shown in (Friedli et al. 2017), pharmaceutical manufacturing data has proven cumulative development of effectiveness and efficiency capabilities, however, for NPD only Efficiency (but not Effectiveness) metrics were demonstrated to be positively influenced by interdependencies capabilities (Brettel et al. 2011). In a more narrow sense, project management Effectiveness and Efficiency in the new product development process are expected to be positively related, at least at higher degrees of project management maturity.

H2

There are individual KPIs on the Operational Stability and Supplier Reliability level that can serve as surrogates for PPMQS Effectiveness.

The goal of identification of surrogate metrics that can reliably represent whole Pharmaceutical Project Management Quality System Effectiveness was substantiated by two considerations: facilitation of analyses with compound Effectiveness score sub-components and individual KPIs, and practical interest for system monitoring and system interventions.

H3

There is a positive link between Enabling and Result systems of PPMQS.

Positive feedback loop from “Results” to “Enablers” in excellence models (EFQM 2013) is expected to be translated into a gradual increase in project management maturity and project governance levels that are supposed to drive PPMQS Effectiveness and Efficiency.

3 Empirical Analysis

3.1 Methodology

Current work utilizes case study approach to test stated hypotheses on a sample of 30 R&D projects within a multinational generic pharmaceutical company (STADA) completed in 2 year period between 07/2015 and 07/2017.

3.1.1 Characterization of PPMQS Enabling and Result Systems

For the Enabling system, typical maturity model was developed as a five-step progression from an awareness of project management methodology needs (level 1) to best-in-class optimized process with the focus on continuous improvement (level 5) based on IMSI Project Management Assessment Model (Holmes and Walsh 2005) and industry standards (ISO 2015, 2017), which were adapted to generic pharmaceutical industry (Babler 2010; Bode-Greuel and Nickisch 2008). Data was collected via project-level questionnaires filled by responsible project managers. The questionnaire’s face validity was established by three independent experts. Subsequently, the questionnaire was pre-tested on one of the participants (five projects).

After data acquisition from the overall sample, the validity of the questionnaire was re-assessed based on the following parameters (statistical approach from (Abramowitz and Weinberg 2016)):

  • Reliability analysis per category: Cronbach’s Alpha (CA) > 0.6.

  • Confirmatory factor analysis (CFA) per category as per criteria specified in Table 1. Heywood cases were excluded from corresponding categories.

Table 1 Parameters and acceptance criteria of confirmatory factor analysis
Full size table

For categories that passed validation, factor scores (method—regression) were recorded, and exploratory factor analysis (EFA) was used to identify common factors underlying the complete survey as per parameters specified in Table 1, except the following: number of factors for extraction—based on Eigenvalues > 1; Varimax rotation if > 1 factor extracted; factor loadings for principal factor > 0.7. Factor scores for major factor were used as PM Quality Maturity score (Fig. 1, category B). The questionnaire was considered to be validated if most of the categories passed CA and CFA requirements, and if EFA for the whole questionnaire resulted in at least one valid factor.

Key performance indicators (KPIs) were chosen for assessment of the Result system of PPMQS, and an additional statistical analysis was performed to select KPIs with suitable reliability within the sample:

  • Reliability analysis per category: CA > 0.6.

  • CFA per category as per criteria specified in Table 1. Heywood cases were excluded from corresponding categories.

  • For categories that passed validation, factor scores (method—regression) were recorded for subsequent analysis (sub-categories within Operational Stability and category Supplier Reliability).

  • Individual KPIs that passed validation within categories were used for data aggregation (Operational Stability and PPMQS Effectiveness scores).

Validated quality metrics from both Enabling and Result systems were used in the final statistical analysis to explore interrelationships between PPMQS components in order to test the proposed hypotheses.

3.2 Testing of Hypotheses

Hypothesis 1: correlation between PPMQS Effectiveness and Efficiency. Bivariate Pearson’s Correlation on the overall sample and sub-sample of high performers (top 25%) for PM Quality Maturity score was calculated for PPMQS Effectiveness score and PPMQS Efficiency variable. Value, sign, and significance of correlation were used for interpretation of the existence of a positive link between these two parameters.

Hypothesis 2: identification of KPIs that can be used as surrogates for PPMQS Effectiveness. Bivariate correlation matrix of KPIs with PPMQS Effectiveness score was created to identify possible candidates to investigate similarity of distribution (KPIs with correlation significant at 0.05 level 2-tailed). Subsequently, linear relationship between PPMQS Effectiveness and candidate surrogates was checked via scatter plots and SPSS Test for linearity within means comparison analysis. Distribution assessment was done by splitting overall projects sample into categories of high-performers (top 25% of projects) and low performers (bottom 25% of projects) for the respective KPI. Afterwards, an independent samples t-test for equality of means was conducted for PPMQS Effectiveness score variable between these two groups with 95% confidence interval. KPIs that showed a mean difference significant at 0.05 level 2-tailed were considered to be good surrogates for PPMQS Effectiveness.

Hypothesis 3: identification of connections between Enabling and Result systems of PPMQS Model. Overall projects sample was split into high (top 50%) and low (bottom 50%) PM Quality Maturity projects and further subdivided into quadrants according to respective KPI performance (high—top 50% and low—bottom 50% of projects). An independent samples t-test for equality of means with 95% confidence interval was conducted for PPMQS Effectiveness surrogate variables between two groups of high and low KPI performers for low and high PM Quality Maturity groups. Afterwards, the effect of PM Quality Maturity on the mean difference and the significance level of the t-test was evaluated. Groups that showed a mean difference significant at 0.05 level 2-tailed were used to study mean differences on individual Enabler level with the same method.

4 Empirical Findings

4.1 Characterization of Enabling and Result Systems

Questionnaire validation resulted in 7 (out of initial 10) confirmed categories for the enablers of Scope, Time, Cost, Quality, Information, Procurement, and Interdependencies. Exploratory factor analysis based on confirmed enablers’ factor scores resulted in 1 common factor explaining 66.5% of the variance for the whole system. This result corresponds to the initial goal of measuring overall project management maturity. Project governance did not form 1 reliable category, and corresponding individual questions were subsequently analyzed separately.

KPIs validation resulted in 3 (out of 4) confirmed compound categories: Cost, Quality, and Supplier Reliability. Category Scope did not pass validation as an entity, and corresponding KPIs (regulatory and market scope parameters) were analyzed separately. Bivariate correlations between Result system KPIs were investigated for linearity, and the following KPIs were found to have a linear relationship:

  • Service rating and Risk increase—positive relationship;

  • Supplier overall rating and Significant deviations—negative relationship.

4.2 Testing of Hypotheses

No correlation was found between PPMQS Effectiveness and Efficiency. This result holds for the overall sample and high performers for PM Quality Maturity subgroup.

Based on bivariate correlation matrix between PPMQS Effectiveness and individual KPIs, the following KPIs were selected for linear relationship check: Risk increase, Cost vs. baseline, Cost vs. plan, Supplier overall rating, Supply rating, Service rating. All KPIs except “Service rating” showed required degree of linearity. Comparison of mean values for PPMQS Effectiveness between project groups which showed high or low performance for candidate surrogate KPIs via independent samples t-test showed mean difference significant at 0.05 level for Risk increase, Cost vs. baseline, Supplier overall rating and Supply rating. All these KPIs were considered to be good surrogates for PPMQS Effectiveness from a statistical viewpoint.

Out of all studied individual KPIs of the Result system, the following two KPIs have a different effect depending on the level of PM Quality Maturity:

  • Supplier overall performance. For low maturity projects, mean difference in the surrogate for PPMQS Effectiveness “Cost vs. baseline” is significant at 0.01 level. For surrogate “Risk increase” it is marginally not significant (p = 0.059). No such relation can be demonstrated for high maturity projects.

  • Significant deviations. For low maturity projects, there is a negative difference in the surrogate for PPMQS Effectiveness “Supplier overall performance”, which is significant at 0.01 level. For other surrogates, negative difference holds and is marginally not significant for “Cost vs. baseline” (p = 0.057) and not significant for “Risk increase”. No such relation can be demonstrated for high maturity projects. This result is in line with negative bivariate correlation between Supplier overall rating and Significant deviations.

These results demonstrate that there is a connection between Enabling and Result systems of PPMQS. In particular, low PM Quality Maturity level reveals a strong dependence of overall PPMQS Effectiveness on external suppliers. This effect is compensated by higher PM Quality Maturity levels. The second result shows that with less rigorous supplier selection process (at lower levels of PM maturity) higher project effectiveness is realized via quality drawbacks (lower post-launch production quality).

5 Discussion

Positive validation of both Enabling and Result systems components within studied sample demonstrates the suitability of applied PPMQS model to the pharmaceutical R&D project and portfolio management context. At the same time, despite evidence from pharmaceutical production regarding the positive correlation between quality system Effectiveness and Efficiency (Friedli et al. 2017), no such correlation could be found for the project management quality system. It should be noted that mean PM Quality Maturity level across all knowledge areas for high performers group (top 25% of the projects) was 2.32 (compared to 1.60 for low performers, on the scale from 1.0 to 5.0), therefore it could not be excluded that presence of correlation between PPMQS Effectiveness and Efficiency might be demonstrated for higher PM maturity levels.

In accordance with Excellence models (EFQM 2013), a connection was found between Enabling and Result systems of PPMQS. PM Maturity level does not have a direct influence on project Effectiveness; however, it acts as a modifier on the dependence of Effectiveness on external suppliers. This relationship partially proves Excellence models assumptions on the feedback loop from project Results to project Enablers as seen on the functional, but not individual project level. Therefore, Excellence model learning mechanism occurs across projects. In addition, study demonstrates that investments in project management processes are justified in settings of high supplier dependence (e.g., for pipelines with a substantial proportion of in-licensed projects).

Several KPIs were found to be representative of overall system: risk increase is a characteristic of high levels of uncertainty of pharmaceutical new product development; actual cost in comparison to baseline (decision point) is a measure of governance decision quality; supplier ratings emphasize the importance of partnerships quality and learning in accordance with the Excellence model (EFQM 2013).

An interesting result regarding the negative relationship between supplier rating and post-launch production quality was investigated further and demonstrated the existence of trade-offs assumed by classical project management theory (Graves 1989) between project performance dimensions during the development process and after product launch. In combination with the absence of correlation between Effectiveness and Efficiency, this result at first glance supports trade-off theory as opposed to cumulative capabilities theory. However, this conclusion holds only in conjunction with low PM quality maturity. Therefore, higher quality levels of project management processes help to eliminate trade-offs between project performance dimensions in accordance with performance frontiers theory (Swink et al. 2006) and decrease total cost of ownership of developed products.

Present study made two main contributions to the research domain. First, it supported quality-based understanding of project excellence by demonstrating that project management quality serves as the foundation for sustainable effectiveness of project management, also under conditions of external suppliers’ failure. Second, utilized systems approach to project quality allowed comprehensive analysis of interconnections between system components, which helped to reveal not only linear relations (as for identified surrogates of system performance), but also more complex non-linear modifying, feedback loop (system learning) relations, and system evolution (changing the level of performance frontier).

Identified patterns of the pharmaceutical project management quality system require further research. Notably, the generalizability of the findings should be investigated, e.g. via a multiple case study approach. Cross-sectional studies of generic pharmaceutical companies with significantly different levels of project management quality maturity would allow to increase sample size and examine causal relationships between quality system components.