Contemporary methods for evaluating complex project proposals

Abstract

The ability to evaluate project proposals, assessing future success, and organizational value is critical to overall business performance for most enterprises. Yet, predicting project success is difficult and often unreliable. A four-year field study shows that the effectiveness of available methods for evaluating and selecting large, complex project depends on the specific project type, organizational culture, and managerial skills. This paper examines the strength and limitations of various evaluation methods. It also shows that, especially in complex project situations, the decision-making process has to go beyond the application of just analytical methods, but has to incorporate both quantitative and qualitative measures into a combined rational judgmental evaluation process. Equally important, the evaluation process must be effectively linked among functional support groups and with senior management in order to strategically align the project proposal and to unify the evaluation team and stakeholder community behind the mission objectives. All of this requires leadership and managerial skills in planning, organizing, and communicating. The paper suggests specific leadership actions, organizational conditions, and managerial processes for evaluating complex project proposals toward future value and success.

Challenges of determining potential for future success

Predicting project success has never been easy. The long list of prominent project failures, ranging from product developments to government social programs, from computers to pharmaceutical, and from public transit to supersonic transport, reminds us of this reality (Cicmil et al. 2006; Gulla 2012; Lemon et al. 2002; Standish Group 2013). Many projects do not live up to their expectations or outright fail even before their technical completion in spite of careful feasibility analysis during their proposal or selection stages (El Emam and Koru 2008; Shore 2008). Obviously, the ability to evaluate project proposals and assess future success and organizational value is critical to overall business performance. In fact, few decisions have more impact on enterprise performance than the resource allocations for new projects (Shenhar et al. 2007). Virtually, every organization evaluates, selects, and implements projects. Whether these projects are product developments, organizational improvements, R&D undertakings or bid proposals, pursuing the ‘wrong’ project not only wastes company resources, but also causes the enterprise to (1) miss critical alternatives, (2) perform less agile in the market place, and (3) miss opportunities for leveraging core competencies. Yet, in spite of pressures to avoid these high-cost errors, predicting project success is difficult, and existing models are often unreliable. Especially in today’s complex and changing business environment (see ‘Today’s Complex Project Environments’ section), the process of evaluating and selecting the ‘best’ projects, most suitable and beneficial for the enterprise, has become both an art and a science, strongly influenced by human and organizational factors.

Objective and rationale for this study

While much has been published on specific methods for project evaluation, the literature lacks a summary and comparison of available tools and techniques. Moreover, few studies have examined the human side as a ‘cross-functional tool’ for collectively evaluating project proposals toward potential success, an important link that is missing in the management literature. This field study contributes to both areas. First, the paper summarizes the benefits, challenges, and limitations of popular project evaluation methods, tools, and techniques. Second, the paper reports the lessons learned from a five-year field study of organizational conditions, leadership style, and decision making with regard to impact on project selection processes.

In today’s increasingly complex and dynamic business environment (‘Today’s Complex Project Environments’ section), predicting success is multifaceted. Typically, it includes not only technical, but also financial, marketing, social, legal, ethical, and technological dimensions, many of them fuzzy or unknown at the time of the proposed project evaluation (Thamhain 2011ab). Hence, the DNA of success is highly complex, and outcomes are difficult to predict. Evaluating and selecting projects is not only an art and a science, but has to go beyond a simple cost-benefit analysis for most cases. To be effective, project opportunities must be analyzed relative to their potential value, strength, and importance to the enterprise. We have to understand and examine the whole spectrum of costs, risks, and benefits as part of the evaluation process, far beyond conventional project economics. This requires a comprehensive approach with sophisticated leadership, integrating organizational resources and facilitating a shared vision of risks across organizational borders, time, and space (Baker 2012; Bstieler 2005; Cicmil et al. 2006). Currently, we are quite efficient on the analytical side of project evaluation/selection. However, we are still weak in dealing with the hidden, less obvious dimensions of predicting success that involve a broader spectrum of project performance variables, connecting to the enterprise and its socio-economic environment (Thamhain and Skelton 2007). These are the issues explored in the study reported in this paper.

Today’s complex project environments

Our complex, fast-changing business environment requires sophisticated decision making for selecting the right project with the most desirable outcome assuring success and enterprise performance. Companies are under pressures for quicker, cheaper, and smarter solutions. They are globally networked and can leverage their resources and accelerate their schedules by forming alliances, consortia, and partnerships with other firms, universities, and government agencies, which range from simple cooperative agreements to open innovation, a concept of scouting for new product and service ideas, anywhere in the world. Projects are complex in nature and imbedded in lots of technology. All of these issues contribute to the complexities and uncertainties of project selection which is a great challenge for managers in virtually every segment of industry and government. They include computer, pharmaceutical, automotive, health care, transportation, and financial businesses, just to name a few. New technologies, especially computers and communications, have radically changed the workplace and transformed our global economy, with focus on effectiveness, value, and speed. These techniques offer more sophisticated capabilities for cross-functional integration, resources mobility, market responsiveness, and managerial decision support, but they also require more sophisticated skills both technically and socially, dealing effectively with higher levels of conflict, change, risks, and uncertainty. These challenges resulted in a shift in managerial focus from functional efficiency to effectiveness with attention to organizational interfaces, human factors, and overall enterprise performance.

What we know about project evaluation and selection

Managerial decision support for project selection has been known for a long time (Remer et al. 1993). Today, managers have available a large array of tools and techniques for project evaluation and selection (National Science Foundation NSF 2010) that can be grouped into three principle classes which are briefly summarized in Appendix 1 of this paper:

• 1. Primarily quantitative and rational approaches.

• 2. Primarily qualitative and intuitive approaches.

• 3. Mixed approaches, combining both quantitative and qualitative methods.

While in the past, decisions on project selection focused on quantitative approaches, such as ROI, payback periods, and net present value for determining desirability and potential success, today’s managers take a more balanced approach between quantitative and qualitative methods. They cast a much wider net for capturing a broad spectrum of variable that go far beyond the scope and limitations of traditional financial evaluations, such as ROI or payback (Kumar 2006).

Most managers are keenly aware of the intricate connections of success variables among organizational systems and processes, which often limit the effectiveness of analytical methods. Especially for complex projects and business processes, managers argue that no single person or group within an enterprise has all the smarts and insight for assessing these multi-variable influences and their cascading effects (Shakhsi-Niaei et al. 2011). Further, no analytical model seems sophisticated enough to represent the complexities and dynamics of all factors that might affect success or failure of a major project (Kavadias and Loch 2004; Zhang et al. 2009; Loch et al. 2001). These managers realize that while analytical methods provide a critically important toolset for project evaluation and selection, these methods also take the collective thinking and collaboration of all the stakeholders and key personnel of the enterprise and its partners to identify and deal with the complexity of risks in today’s business environment. As a result, an increasing number of organizations are complementing their analytical methods with managerial judgment and collective stakeholder experiences that include such broad measures as strategic desirability and projections from past performances (Hadad et al. 2012), hence moving beyond a narrow dependence on just analytical models. In addition, many companies have developed their own ‘systems,’ uniquely designed for dealing with uncertainties in their specific projects and enterprise environment (Henriksen and Traynor 2002; Kavadias and Loch 2004; Kumar 2006; Larson and Gray 2011). These systems emphasize the integration of various tools, often combining quantitative and qualitative methods to cast a wider net for capturing and assessing risk factors beyond the boundaries of conventional methods. Examples are well-known management tools, such as review meetings, Delphi processes, brainstorming, and focus groups, which have been skillfully integrated with analytical methods to leverage their effectiveness and improve their reliability. In addition, a broad spectrum of new and sophisticated tools and techniques, such as user-centered design (UCD), voice of the customer (VoC), and phase-gate processes, evolved which rely by and large on organizational collaboration and collective judgment processes to deal with the broad spectrum of risk variables that are dynamically distributed throughout the enterprise and its external environment.

The missing link

While project evaluation and selection methods have been studied extensively for several decades (Brenner 1994; Cook and Green 2000; Mantel et al. 2011), relatively little has been published on the role of collaboration across the total enterprise for evaluating potential project value (Oral et al. 1991). That is, we know little about management processes that involve the broader project community in a collective cross-functional way for dealing with project selection. The missing link is the people side as a cross-functional tool for collectively evaluating a project proposal toward potential success, one of the areas that are being examined in this paper.

Method

An exploratory field study format is used to investigate managerial approaches for evaluating and selecting project proposals. Specifically, this field study looks at both quantitative and qualitative models, and reports on their use and effectiveness for various project types and situations.

The work presented in this paper is a continuation of my ongoing research into project management alignment with enterprise strategy (Shenhar et al. 2007) and risk management in complex project situations (Thamhain and Skelton 2007; Thamhain 2011ab). The field study reported here focuses especially on data collected between 2008 and 2012.

Data

The unit of analysis used in this study is the project. Data were captured from 25 technology organizations between 2008 and 2012, as part of an ongoing research in the area of technology-oriented product development and team-based project management. The field study yielded data from 43 project teams with a total sample population of 425 professionals such as engineers, scientists, and technicians, plus their managers, including 10 supervisors, 53 project team leaders, 12 product managers, 5 directors of R&D, 4 directors of marketing, and 11 general management executives at the vice presidential level or higher. Together, the data covered more than 100 projects in 18 companies, as summarized in Table 1. The projects involved mostly high-technology product/service/process developments with budgets averaging US$28 million each. All project teams saw themselves working in a high-technology environment. The 18 host companies are large technology-based multinational companies of the FORTUNE 500 category. The data were obtained from three sources, questionnaires, participant observation, and in-depth retrospective interviewing, as discussed in the previous section. Specifically, in stage three, 138 interviews were held with team leaders, line managers, product managers, marketing directors, and general management executives. These discussions provided interesting and useful insight into the cross-functional issues and challenges involved in project evaluation and selection processes in complex business environment. Content analysis has been used in addition to standard statistical methods for evaluating the survey data.

Table 1 Summary of field sample statistics

Justification for the exploratory field study format

All components of this investigation, such as project management, product development, team work, decision making under uncertainty, technology, and business environment, involve highly complex sets of intricately related variables. Researchers have consistently pointed at the non-linear, often random nature of these processes that involve many facets of the organization, its members, and the environment (Bstieler 2005; Danneels and Kleinschmidt 2001 Nellore and Balachandra 2001; Thamhain 20082009; Verganti and Buganza 2005). Investigating these organizational processes simultaneously is not a simple task, making it unlikely to find simple models appropriate for researching these environments. Because of these complexities and the still evolving nature of these components, their theories and constructs, an exploratory field research format was chosen for the investigation which uses questionnaires in addition to two qualitative methods: participant observation and in-depth retrospective interviewing.

Defining project value, performance, and success

All of these measures are multifaceted. Typically, they include not only technical, but also financial, marketing, social, legal, and ethical dimensions. For most projects, the DNA of success is highly complex, and outcomes are difficult to predict, especially long term. Special attention must be given to the assessment of these variables, with their many facets and reference points throughout the project lifecycle and beyond. Especially, project success is highly judgmental and difficult to determine at the outset, but often only years after project completion, as we can see in aerospace, pharmaceutical, automobile, infrastructure or government service projects. Yet, in spite of these challenges, senior managers, collectively, seem to have a good sense of the potential success and value of a proposed project. We validated this fact in previous studies by testing the agreement among senior managers from various parts and different levels of the enterprise regarding their judgment of project performance (Kruglianskas and Thamhain 2000; Thamhain 20052006). These agreements were measured via Kruskal-Walles analysis of variance by ranks which provided evidence of the strong agreement among the managers on the degree of success expected for a given project under evaluation. Therefore, I used this finding as a basis for arguing the benefits of collective multifunctional judgment in selecting candidate projects, a discussion presented later in this paper. Yet, the topic of ‘measuring success’ and its underlying metrics is an area that needs additional study. Suggestions for future research include the relationship between short-term project performance and long-term project success, and underlying conditions and evaluation criteria that might be useful for predicting future project performance and success.

Results

Based on the findings from this field study, an overview is first given on how companies evaluate and select complex projects, followed by specific recommendations for effectively managing the process. The first segment includes a discussion on the effectiveness of some of the most popular quantitative and qualitative methods available to managers, and an assessment of the situational value of quantitative versus qualitative methods.

How do companies evaluate and select projects

Based on the findings from this field study, supported by additional observations on hundreds of projects during my action research, a framework for effective project evaluation and selection is summarized first and then followed by specific recommendations for effectively managing the process. While any decision model needs to be fine tuned toward specific application, the basic framework and management philosophy of evaluating and selecting project proposals seem to be appropriate for most projects, in spite of their differences in complexities, technologies, organizational structure, and culture among companies.

Evaluation dimensions and decision phases

Project opportunities must be analyzed relative to their potential value, strength, and importance to the enterprise. Four major dimensions should be considered for project evaluation and compared to available alternatives as graphically shown in Figure 1: (1) added value of the new project, consistent with the organizational objectives; (2) resource requests such as cost, personnel, and facilities needed to complete the new project; (3) readiness and ability of the enterprise to execute the project; and (4) managerial belief and desire. A well-organized project evaluation and selection process provides the framework for systematic data gathering and informed decision making toward resource allocation. Typically, the decision process can be broken into phases which are often overlapping and executed concurrently, such as shown in Figure 1 and described below:

Phase I. Deciding initial feasibility: screening and filtering, quick decision on the viability of an emerging project for further evaluation;

Phase II. Deciding strategic value to enterprise: identifying alternatives and options to the proposed project;

Phase III. Deciding detailed feasibility and value: determining the specific value and chances of success for a proposed project;

Phase IV. Deciding project go/no-go: committing resources for a project implementation.

Figure 1

Project selection decision model.

While the process seems simple, logical, and straightforward, developing meaningful support data is a complex undertaking. It is also expensive, time consuming, and often highly eclectic. Typically, decision making requires the following inputs: (1) specific resource requirements; (2) specific implementation risks; (3) specific benefits, i.e., economics, technology, markets, etc.; (4) benchmarking and comparative analysis of threats and opportunities; and (5) strategic perspective, including long- and short-term value assessment.

Estimating cost, schedules, risks, and benefits, such as those shown in Table 2, is always challenging. However, these estimates are relatively straightforward in comparison to predicting project success. The difficulty is in defining a meaningful aggregate indicator for project value and success. Methods for determining success range from purely intuitive to highly analytical. No method is seen as truly reliable in predicting success, especially for complex and technologically intensive projects. Yet, some companies have a better track record in selecting ‘winning’ projects than others. They seem to have the ability to create a more integrated picture of the potential benefits, costs, and risks for the proposed project relative to the company’s strength and strategic objectives.

Table 2 Typical criteria considered for project evaluation and selection

Methods of project evaluation and selection

Producing a comprehensive picture of future value that integrates the large array of potential costs and benefits is both a science and an art. During field interviews, managers confirm the changing paradigm of project evaluation that has been reported in the literature for some time (Baker 2012; Kavadias and Loch 2004; Kumar 2006; Shakhsi-Niaei et al. 2011). That is, not too long ago, in the 1990s and before, rational evaluation processes prevailed by and large for supporting project selections. However, in today’s more complex and dynamic business environment, managers point out that purely rational analytical processes apply only to a limited number of situations. Most of our technologically complex business scenarios require the integration of both analytical and judgmental techniques to capture the broad spectrum of variables affecting project success or failure, necessary for predicting success and making the best choice.

Yet, in spite of the shift toward including judgmental methods more specifically into the selection process, systematic information gathering and standardized methods are at the heart of any project evaluation process and provide the best assurance for reliably predicting project outcome and repeatability of the decision process. In the sample of 105 projects of my field study, managers report the following breakdown of approaches used for project evaluation and selection:

• Primarily quantitative and rational approaches: used for 18% of all projects.

• Primarily qualitative and intuitive approaches: used for 24% of all projects.

• Mixed approaches (quantitative/qualitative combination): used for 58% of all projects.

Given the fact that projects of the field sample consisted of complex, largely high-technology projects, it might look surprising that a relatively large number of project evaluations are based on either primarily quantitative or qualitative methods. That is, one might expect more mixed approaches. Follow-up interviews provide additional insight. Of the sample’s projects, 20% were very large undertakings of over US$100 million each, many of them defense contracts or alike. While these program evaluations included considerable qualitative inputs and judgment from focus groups, industry experts and management, the final go decision was strongly influenced by quantifiable data and highly rational methods of decision making. As a result, more than half of these ‘very large’ projects were judged as being selected by ‘primarily quantitative and rational approaches’. Similarly, we found special situations in the ‘primarily qualitative and intuitive’ category. Approximately one half of the projects in this category were of R&D or exploratory nature with very limited quantifiable data, especially in the area of future value (i.e., future sales, revenue, market share, etc.). While all of these projects went through detailed technical feasibility, budgeting, and project planning processes, the final ‘go decision’ was based on primarily qualitative and intuitive approaches for approximately 60% of these projects.

However, regardless of the primary nature (quantitative, qualitative, or combined focus), all project evaluations included some quantitative and qualitative assessments. Equally important, we observed highly interactive interdisciplinary effort among the various resource groups of the enterprise and its partners during the evaluation and selection processes for all projects, regardless of their type, size, and classification. That is, while management may judge the process leading to the final go/no-go decision as ‘primarily’ one of the three approaches, all decisions included both quantitative and qualitative inputs to some degree. Often, many meetings were needed even before the formal project evaluation process started, just to gain a basic picture of potential benefits, costs, and risks involved, and to determine the type of data needed. All of these activities occurred regardless of primary evaluation/selection focus.

Yet, in all situations, there is the risk of relying too heavily on quantitative data. This was pointed out especially by senior managers. Most concerned were those managers and project leaders who lived through a disappointing project rollout or delivery, or outright project failure, after careful feasibility study and selection procedures at the front end. These managers observed in retrospect that many of their front-end decision processes relied too much on analytical data and models with assumptions that did not hold or forecasts that proved unreliable, while not casting a wide enough net for scanning the business environment. They felt that more experiential information had to be captured from senior management and technology, market and user communities, to complement and validate the analytical models. The issues and potential remedies to these challenges will be explored in the next section.

Going beyond simple formulas and quantitative methods

While quantitative methods of project evaluation have the benefit of producing relatively quickly a measure of merit for simple comparison and ranking, they also have many limitations, as summarized in Table 3. Yet, in spite of the limitations inherent to quantitative evaluation and the increased use of qualitative approaches, virtually, every organization supports its project selections with some form of quantitative measures; the most popular are ROI, cost-benefit, and payback period (see Appendix 1 and 2). However, driven by the growing complexity of the business environment, quantitative decision models are becoming less capable of capturing the full spectrum of variables associated with project success or failure. As pointed out by one senior manager in our field study, “Numbers like favorable ROI and projected revenue are great selling points toward project approval. However, often I’m not comfortable with the data. The numbers don’t always seem to reflect market realities, customer reactions, changing technologies and dozens of other uncertainties and assumptions. Brainstorming the issues and defining critical success factors with experts from the areas critical, and getting inputs from user groups, often gives us a better sense of potential success or failure and how to plan for success that a cold number that is based on a complex financial model or projection that we don’t really trust”. These types of issues and inherent limitations of analytical methods concern managers across all industries, who started increasingly to augment quantitative methods and to explore alternatives. As observed in this field study and my own project work, and as a management consultant to hundreds of companies, there is an increasing trend for companies to supplement quantitative results with additional information for determining future cost-benefits and project success. Many of these contemporary decision making methods rely to a large degree on qualitative, judgmental decision making as summarized in Appendix 1. These data gathering methods cast a wider net and consider a broader spectrum of factors than those methods limited to quantitative measure. They can include fuzzy variables and decision parameters such as related to strategy and business ethics that may be difficult to describe or quantify, but are important in gaining overall perspective and a more comprehensive picture on potential benefits, risk, and challenges of the proposed project.

Table 3 Comparison of quantitative and qualitative approaches to project evaluation

Discussion and recommendations

Effective evaluation and selection of project opportunities involve many variables of the organizational and technological environment, reaching often far beyond cost and revenue measures. While economic models provide an important dimension of the project selection process, most situations are too complex to use simple quantitative methods as the sole basis for decision making. Many of today’s project evaluation procedures include a broad spectrum of variables and rely on a combination of rational and intuitive processes for defining the value of a new project venture to the enterprise. The better organizations understand their business processes, markets, customers, and technologies, the better they will be able to evaluate the value, risks, and challenges of a new project venture. Further, manageability of the evaluation process is critical to its results, especially in complex situations. The process must have a basic structure, discipline, and measurability to be conducive to the intricate multivariable analysis. One method of achieving structure and manageability calls for grouping the evaluation variables into four categories: (1) consistency and strength of the project with the business mission, strategy, and plan; (2) multifunctional ability to produce the project deliverables and objectives, including technical, cost, and time factors; (3) success in the customer environment; and (4) economics, including profitability. Modern phase management, such as Stage-Gate® processes provide managers with the tools for organizing and conducting project evaluations in a systematic way. The following section summarizes suggestions that can help managers in effectively evaluating and selecting projects toward successful implementation.

Seek out relevant information

Meaningful project evaluations require relevant quality information. The four sets of variables related to the strategy, results, customer, and economics, as identified above, can provide a framework for establishing the proper metrics and detailed data gathering.

Ensure competence and relevancy

Ensure that the right people become involved in the data collection and judgmental processes.

Take top-down look first, detail comes later

Detail is less important than information relevancy and evaluator expertise. Do not get hung-up on missing data during the early phases of the project evaluation. Evaluation processes should be iterative. It does not make sense to spend a lot of time and resources on gathering perfect data to justify a ‘no-go’ decision.

Select and match the right people

Whether the project evaluation consists of a simple economic analysis or a complex multifunctional assessment, competent people from functions critical to the overall success of the project should be involved.

Define success criteria

Whether deciding on a single project or choosing among alternatives, evaluation criteria must be defined. They can be quantitative, such as ROI, or qualitative, such as the chances of winning a contract. In either case, these evaluation criteria should cover the true spectrum of factors affecting success and failure of the project(s). The success criteria should be identified by seasoned enterprise personnel. In addition, people from outside of the company, such as vendors, subcontractors, and customers, are often included in this expert group and critical to the development of meaningful success criteria.

Strictly quantitative criteria can be misleading

Be aware of evaluation procedures based on quantitative criteria only (ROI, cost, market share, MARR, etc.). The input data used to calculate these criteria are likely based on rough estimates and are often unreliable. Furthermore, a reliance on strictly quantitative data considers only a narrow spectrum of factors affecting project success or failure, thus ignoring many other important factors, especially those that influence project success in a dynamic or non-linear way, typical for many complex technologically sophisticated undertakings. Evaluations based on predominately quantitative criteria should at least be augmented with some expert judgment as a ‘sanity check’.

Condense criteria list

Combine evaluation criteria, especially among the judgmental categories, to keep the list manageable. As a goal, try to stay within the 12 criteria for each category.

Gain broad perspective

The inputs to the project selection process should include the broadest possible spectrum of data from the business environment that affect success, failure, and limitations of the new project opportunity. Assumptions should be carefully examined.

Communicate across the enterprise

Facilitate communications among evaluators and functional support groups. Define the process for organizing the team and conducting the evaluation and selection process.

Ensure cross-functional representation and cooperation

People on the evaluation team must share a strategic vision across organizational lines. They also must have the desire to support the project if selected for implementation. The purpose, goals, objectives, and relationships of the project to the business mission should be clear to all parties involved in the evaluation/selection process.

Do not lose the big picture

As discussions go into detail during the evaluation, the team should maintain a broad perspective. Two global judgment factors can help focus on the big picture of project success: (1) overall cost-benefit perspective and (2) overall risk of failure assessment. These factors can be recorded on a ten-point scale, −5 to +5. This also leads to an effective two-dimensional graphic display for comparing competing project proposals.

Do your homework between iterations

Project evaluations are usually conducted progressively in iterative cycles. Therefore, the need for more information, clarification, and further analysis surfaces between each cycle. Necessary action items should be properly assigned and followed up to enhance the evaluation quality with each consecutive iteration.

Take a project-oriented approach

Plan, organize, and manage your project evaluation/selection process as a project. Proposal evaluation and selection processes require valuable resources that must be justified and carefully managed.

Resource availability and timing

Do not forget to include in your selection criteria the availability and timing of resources. Many otherwise successful projects fail because they cannot be completed within a required time period.

Use red-team reviews

Set up a special review team of senior personnel. This is especially useful for large and complex projects with major impact on overall business performance. This review team examines the decision parameters, qualitative measures, and assumption used in the evaluation process. Limitations, biases, and misinterpretations that may otherwise remain hidden can often be identified and dealt with.

Stimulate creativity and candor

Senior management should foster an innovative risk-shared ambience for the evaluation team. Especially, the evaluation of complex project situations involves intricate sets of variables. Criteria for success and failure are linked among many subsystems, such as organization, technology, and business, associated with a great deal of risks and uncertainty. Innovative approaches are required to evaluate the true potential of success for these projects. Risk sharing by senior management, recognition, visibility, and a favorable image in terms of high priority, interesting work, and importance of the project to the organization have been found strong drivers toward attracting and holding quality people on the evaluation team and toward gaining their active and innovative participation in the process.

Manage and lead

The evaluation team should be chaired by someone who has the trust, respect, and leadership credibility with the team members. Senior management can positively influence the work environment and the process by providing guidelines, charters, visibility, resources, and active support to the project evaluation team.

Conclusions

In summary, effective project evaluation and selection requires a broad scanning process across all segments of the enterprise and its environment to deal with the risks, uncertainties, ambiguities, and imperfections of data available for assessing the value of a new project venture relative to other opportunities. No single set of broad guidelines exists that guarantees the selection of successful projects. However, the process is not random! A better understanding of the organizational dynamics that affects project performance and the factors that drive cost, revenue, and other benefits can help in gaining a better, more meaningful insight into the future value of a prospective new project. Seeking out both quantitative and qualitative measures incorporated into a combined rational judgmental evaluation process often yields the most reliable predictor of future project value and desirability. Equally important, the process requires managerial leadership and skills in planning, organizing, and communicating. Above all, the leader of the project evaluation team must be a social architect who can unify the multifunctional process and its people. The leader must be able to foster an environment, professionally stimulating and conducive to risk sharing. It also must be effectively linked to the functional support groups needed for project implementation. Finally, organizational strategy must be aligned and integrated with the evaluation/selection process, early and throughout its evaluation cycle. Senior management has an important role in unifying the evaluation team behind the mission objectives and in facilitating the linkages to the stakeholders and ultimate user community. Senior management should further help in providing overall leadership and in building mutual trust, respect, and credibility among the members of the proposal evaluation team, all critical drivers toward a strong partnership of all team members and the basis for an effective enterprise-wide decision-making system. Taken together, this is the environment conducive for cross-functional communication, cooperation, and integration of the intricate variables needed for effective evaluation and selection of project proposals in complex business environments.

Appendices

Appendix 1

Summary of project evaluation and selection techniques

Some of the popular project evaluation and selection tools, techniques, and approaches are summarized in this Appendix, grouped into three classes:

• 1. Primarily quantitative and rational approaches.

• 2. Primarily qualitative and intuitive approaches.

• 3. Mixed approaches, combining both quantitative and qualitative methods.

Quantitative approaches to project evaluation and selection

Quantitative approaches are often favored to support project evaluation and selections if the decisions require economic justification. They are also commonly used to support judgment-based project selections. One of the features of quantitative approaches is the generation of numeric measures for simple and effective comparison, ranking, and selection. These approaches also help establish quantifiable norms and standards and lead to repeatable processes. Yet, the ultimate usefulness of these methods depends on the assumption that the decision parameters, such as cash flow, risks, and the underlying economic, social, political, and market factors, can actually be quantified and reliably estimated over the project life cycle. Therefore, quantitative techniques are effective and powerful decision support tools, if meaningful estimates of cost-benefits, such as capital expenditures and future revenues, can be obtained and converted into net present values for comparison. Because of their importance, quantitative methods have been discussed in the literature extensively, ranging from simple return on investment (ROI) calculations to elaborate simulations of project scenarios. Many companies eventually developed their own project evaluation/selection models, customized to their specific needs. However, the backbone for most of these customized models is a set of economic/financial measures which tries to determine the cost-benefit of the proposed venture, usually for some point in the future. Specifically, five measures are especially popular:

• 1. Net present value (NPV)

• 2. ROI

• 3. Cost-benefit (CB)

• 4. Payback period (PBP)

• 5. Pacifico and Sobelman project ratings

The calculation and application of these measures to project evaluation/selection will be illustrated by case examples. Specifically, four project proposals (described in Table 4) will be evaluated in this chapter, using the above measures. The results are summarized in Table 5.

Table 4 Description of four project proposals

Table 5 Cash flow and net value calculations of four project options or proposals

Net present value comparison

This method uses discounted cash flow as the basis for comparing the relative merit of alternative project opportunities. It assumes that all investment costs and revenues are known and that economic analysis is a valid basis for project selection. We can determine the NPV of a single revenue, or stream of future revenues, or costs expected in the future. Two types of presentations are common: (1) present worth (PW) and (2) net present value.

Present worth

This is the single revenue or cost (also called annuity A) which occurs at the end of a period n, subject to the prevailing interest rate i. Depending on the management philosophy and enterprise policies, this interest rate can be (1) the internal rate of return (IRR) realized by the company on similar investments or (2) the minimum attractive rate of return (MARR) acceptable to company management, or the prevailing discount rate. The present worth is calculated as

$$\mathrm{PW}\left(A|i,n\right)={\mathrm{PW}}_{\mathrm{n}}=A\frac{1}{{\left(1+i\right)}^n}.$$

For the examples used in this chapter, we consider the IRR (defined as the average return realized on similar investments) to be the prevailing interest rate.

Net present value

The net present value is defined as a series of revenues or costs, A _n , over N periods of time at a prevailing interest rate i:

$$\mathrm{NPV}\left({\displaystyle A_n}|i,N\right)={\displaystyle \sum _{n=1}^N}{\displaystyle A_n}\frac{1}{{(1+i)}^n}={\displaystyle \sum _{n=1}^N}{\displaystyle {\mathrm{PW}}_n}.$$

Three special cases exist for the net present value calculation: (1) for a uniform series of revenues or costs over N periods, NPV(A _n |i, N) = A[(1 + i)^{N − 1}]/i(1 + i)^N; (2) for an annuity or interest rate i approaching zero, NPV = A × N; and (3) for the revenue or cost series to continue forever, NPV = A/i. Table 5 applies these formulas to the four project alternatives described in Table 4, showing the most favorable 5-year net present value of US$3,192 for project option P3.

Return on investment comparison

Perhaps one of the most popular measures for project evaluation is the ROI:

$$\mathrm{ROI}=\frac{\mathrm{Revenue}\left(\mathrm{R}\right)-\mathrm{Cost}\left(\mathrm{C}\right)}{\mathrm{Investment}\left(\mathrm{I}\right)}.$$

ROI calculates the ratio of net revenue over investment. In its simplest form, the stream of cash flow is not discounted. One can look at the revenue on a year-by-year basis, relative to the initial investment. For example, project option 1 in Table 3 would produce a 20% ROI each year, while project option 2 would produce a 75% ROI during the first year, 50% during the second year, and so on. In a somewhat more sophisticated way, we can calculate the average ROI per year over a given revenue cycle as shown in Table 3:

$$R\overline{O}I\left({\displaystyle A_n},{\displaystyle I_n}|N\right)=\left[{\displaystyle \sum _{n=1}^N\frac{{\left(\mathrm{Revenue}R\right)}_n-{\left(\mathrm{Cost}C\right)}_n}{{\left(\mathrm{Investment}I\right)}_n}}\right]/\left[N\right].$$

We can then compare the average ROI to the MARR. Given a MARR of 10% for our project environment, all three project options P1, P2, and P3 compare favorable, with project P3 yielding the highest average return on investment of 54%. Although this is a popular measure, it does not permit a meaningful comparative analysis of alternative projects with fluctuating costs and revenues. Furthermore, it does not consider the time value of money.

Cost-benefit

Alternatively, we can calculate the net present value of the total ROI over the project lifecycle. This measure, known as cost-benefit (CB) is calculated as the present value stream of net revenues divided by the present value stream of investments. It is an effective measure for comparing project alternatives with fluctuating cash flows:

$${\displaystyle {\mathrm{CB}=\mathrm{ROI}}_{\mathrm{NPV}}}\left({\displaystyle A_n},{\displaystyle I_n}|i,N\right)=\frac{\left[{\displaystyle \sum _{n=1}^N}\mathit{NPV}\left({\displaystyle A_n}|i,N\right)\right]}{\left[{\displaystyle \sum _{n=1}^N}\mathit{NPV}\left({\displaystyle I_n}|i,N\right)\right]}.$$

In our example of four project options (Table 3), project proposal P4 produces the highest cost-benefit of 164% under the given assumption of i = MARR = 10%.

Payback period comparison

Another popular figure of merit for comparing project alternatives is the PBP. It indicates the time period of net revenues required to return the capital investment made on the project. For simplicity, undiscounted cash flows are often used to calculate a quick figure for comparison, which is quite meaningful if we deal with an initial investment and a steady stream of net revenue. However, for fluctuating revenue and/or cost steams, the net present value must be calculated for each period individually and cumulatively added up to the ‘break-even point’ in time, N_PBP, when the net present value of revenue equals the investment. Mathematically,

$$N_{\mathrm{PBP}}\mathrm{occurs}\mathrm{when}{\displaystyle \sum _{n=1}^N}\mathrm{NPV}\left({\displaystyle A_n}|i\right)\ge {\displaystyle \sum _{n=1}^N}\mathrm{NPV}\left({\displaystyle I_n}|i\right).$$

In our example of four project options (Table 3), project proposal P3 produces the shortest, most favorable payback period of 1.8 years under the given assumption of i = MARR = 10%.

Pacifico and Sobelman project ratings

The previously discussed methods of evaluating projects rely heavily on the assumption that technical and commercial success is assured, and all costs and revenues are predicable. Because these assumptions do not always hold, many companies have developed their own special procedures and formulas for comparing project alternatives. Two examples illustrate this special category of project evaluation metrics.

The project rating factor

This measure was originally developed by Carl Pacifico for assessing chemical products and predicting commercial success:

$$\mathrm{PR}=\frac{\mathit{pT}\times \mathit{pC}\times R}{\mathit{TC}}.$$

Pacifico’s formula is in essence an ROI calculation adjusted for risk. It includes probability of technical success (0.1 < pT < 1.0), probability of commercial success (0.1 < pC < 1.0), total net revenue over project lifecycle (R), and total capital investment for product development, manufacturing set-up, marketing, and related overheads (TC).

Product development figure of merit

The formula developed by Sobelman

$$z=\left(P\times {\displaystyle T_{\mathrm{LC}}}\right)-\left(C\times {\displaystyle T_{\mathrm{D}}}\right)$$

represents a modified cost-benefit measure which takes into account both the development time and commercial lifecycle of the product. It also includes average profit per year (P), estimated product lifecycle (T_LC), average development cost per year (C), and years of development (T_D).

Qualitative approaches to project evaluation and selection

While quantitative methods provide an important toolset for project proposal evaluation and selection, there is also a growing sense of frustration, especially among managers of complex and technologically advanced undertakings, that reliance on strictly quantitative methods does not always produce the most useful or reliable inputs for decision-making, nor are all methods equally suited for all situations. Therefore, it is not surprising that for project evaluations involving complex sets of business criteria, narrowly focused quantitative methods are often supplemented with broad scanning, intuitive processes, and collective, multifunctional decision making such as Delphi, nominal group technology, brainstorming, focus groups, sensitivity analysis, benchmarking, and UCD. Each of these techniques can either be used by itself to determine the best, most successful, or most valuable option, or these techniques are integrated into a comprehensive analytical framework for collective multifunctional decision making, which is being discussed next.

Collective, multifunctional evaluations

This process relies on subject experts from various functional areas for collectively defining and evaluating broad project success criteria, employing both quantitative and qualitative methods. The first step is to define the specific organizational areas critical to project success and to assign expert evaluators. For example, a product or service development project may typically include organizations such as R&D, engineering, testing, manufacturing, marketing, product assurance, and customer/field services. The function experts should be given the time and resources necessary for the evaluation. They also should have the commitment from senior management for full organizational support. Ideally, these evaluators should be members of the core team ultimately responsible for project implementation.

Evaluation factors

Early in the evaluation process, the team defines the factors which appear critical to the ultimate success of the projects under evaluation and arranges them into a list which includes both quantitative and qualitative factors. A mutually acceptable scale must be worked out for scoring the evaluation criteria. Studies of collective multi-functional assessment practices show that simple scales are most effective for leading to actionable team decisions. The four most popular and robust scales for judging situational outcomes are shown below:

• 1. Ten-point judgment scale: This scale ranges from +5 (most favorable) to − 5 (most unfavorable).

• 2. Three-point judgment scale: +1 (favorable), 0 (neutral or cannot judge), −1 (unfavorable).

• 3. Five-point judgment scale: A (highly favorable), B (favorable), C (marginally favorable), D (most likely unfavorable), F (definitely unfavorable).

• 4. Five-point Likert scale: 1 (strongly agree), 2 (agree), 3 (neutral), 4 (disagree), 5 (strongly disagree).

Weighing of criteria is not recommended for most applications as it complicates and often distorts the collective evaluation. Perspective and judgment are part of the strength and value of these qualitative methods, which can be lost by forcing too much of a quantitative framework on the qualitative evaluation.

The evaluation process

Evaluators first assess and then score all of the success factors they feel qualified to judge. Then, collective discussions follow. Initial discussions of project alternatives, their markets, business opportunities, and technologies involved are usually beneficial, but not necessary for the first round of the evaluation process. The objective of this first round of expert judgments is to get calibrated on the opportunities and challenges presented. Further, each evaluator has the opportunity to recommend (1) actions that could improve the quality and accuracy of the project evaluation, (2) additional data needed, and (3) suggestions for increasing project success. Before meeting at the next group session, agreed-on action items and activities for improving the decision process should be completed. The evaluation process is enhanced with each iteration by producing more accurate, refined, and comprehensive data. Typically, between 3 and 5 iterations are required before a go/no-go decision can be reached for a given project.

Mixed approaches, combining both quantitative and qualitative methods

Mixed approaches are the most common method of evaluating and selecting projects in today’s complex business environment. Virtually all evaluations of project proposals include some form of quantitative and qualitative methods. However, to qualify as a mixed approach, the project evaluation and selection process has to contain a fairly balanced array of both classes of analytical and judgmental tools and techniques of the various types discussed above.

Appendix 2

The following is a summary description of terms, variables, and abbreviations used in this chapter:

• Cross-functional involves actions which span organizational boundaries.

• Phase management involves breaking of projects into natural implementation phases, such as development, production, and marketing, as a basis for project planning, integration, and control. Phase management also provides the framework for concurrent engineering and Stage-Gate® processes.

• Project success is a comprehensive measure, defined in both quantitative and qualitative terms which includes economic, market, and strategic objectives.

• Stage-Gate® process is a framework originally developed by R Cooper and S Edgett for executing projects within predefined stages with measurable deliverables (at gates) at the end of each stage. These gates also provide the review metrics for ensuring successful transition and integration of the project into the next stage.

• Weighing of criteria is a a multiplier associated with specific evaluation criteria.

• Annuity (A) is the present worth of a revenue or cost at the end of a period n.

• Cost benefit (CB) is the net present value of all ROIs in dollars.

• Prevailing interest rate (i).

• Investment (I).

• Internal rate of return (IRR) is the average return on investment realized by a firm on its investment capital.

• Minimum attractive rate of return (MARR) on new investments acceptable to an organization.

• Net present value (NPV) of a stream of future revenues or costs.

• Payback period (PBP) is the time period needed to recover the original investment.

• Project rating factor (PR) is a measure developed by Carlo Pacifico for predicting project success.

• Present worth (PW, also called annuity) is the present value of a revenue or cost at the end of a period n.

• Return on investment (ROI)

• Project rating factor (z) is a measure developed by Sobelman for predicting project success.

Authors’ information

Dr. Hans Thamhain is a Professor of Management and Director of MOT and Project Management Programs at Bentley University, Boston/Waltham. Dr. Thamhain, held management positions with Verizon, General Electric and ITT, has written over seventy research papers and six professional reference books. He received the IEEE Engineering Manager Award in 2001, PMI’s Distinguished Contribution Award in 1998 and PMI’s Research Achievement Award in 2006. He is profiled in Marquis Who's Who in America and certified as NPDP and PMP.