Contemporary methods for evaluating complex project proposals
- First Online:
- 3.3k Downloads
The ability to evaluate project proposals, assessing future success, and organizational value is critical to overall business performance for most enterprises. Yet, predicting project success is difficult and often unreliable. A four-year field study shows that the effectiveness of available methods for evaluating and selecting large, complex project depends on the specific project type, organizational culture, and managerial skills. This paper examines the strength and limitations of various evaluation methods. It also shows that, especially in complex project situations, the decision-making process has to go beyond the application of just analytical methods, but has to incorporate both quantitative and qualitative measures into a combined rational judgmental evaluation process. Equally important, the evaluation process must be effectively linked among functional support groups and with senior management in order to strategically align the project proposal and to unify the evaluation team and stakeholder community behind the mission objectives. All of this requires leadership and managerial skills in planning, organizing, and communicating. The paper suggests specific leadership actions, organizational conditions, and managerial processes for evaluating complex project proposals toward future value and success.
KeywordsProject evaluation and selection Project management Team leadership Technology Decision making Rational Judgmental
Challenges of determining potential for future success
Predicting project success has never been easy. The long list of prominent project failures, ranging from product developments to government social programs, from computers to pharmaceutical, and from public transit to supersonic transport, reminds us of this reality (Cicmil et al. 2006; Gulla 2012; Lemon et al. 2002; Standish Group 2013). Many projects do not live up to their expectations or outright fail even before their technical completion in spite of careful feasibility analysis during their proposal or selection stages (El Emam and Koru 2008; Shore 2008). Obviously, the ability to evaluate project proposals and assess future success and organizational value is critical to overall business performance. In fact, few decisions have more impact on enterprise performance than the resource allocations for new projects (Shenhar et al. 2007). Virtually, every organization evaluates, selects, and implements projects. Whether these projects are product developments, organizational improvements, R&D undertakings or bid proposals, pursuing the ‘wrong’ project not only wastes company resources, but also causes the enterprise to (1) miss critical alternatives, (2) perform less agile in the market place, and (3) miss opportunities for leveraging core competencies. Yet, in spite of pressures to avoid these high-cost errors, predicting project success is difficult, and existing models are often unreliable. Especially in today’s complex and changing business environment (see ‘Today’s Complex Project Environments’ section), the process of evaluating and selecting the ‘best’ projects, most suitable and beneficial for the enterprise, has become both an art and a science, strongly influenced by human and organizational factors.
Objective and rationale for this study
While much has been published on specific methods for project evaluation, the literature lacks a summary and comparison of available tools and techniques. Moreover, few studies have examined the human side as a ‘cross-functional tool’ for collectively evaluating project proposals toward potential success, an important link that is missing in the management literature. This field study contributes to both areas. First, the paper summarizes the benefits, challenges, and limitations of popular project evaluation methods, tools, and techniques. Second, the paper reports the lessons learned from a five-year field study of organizational conditions, leadership style, and decision making with regard to impact on project selection processes.
In today’s increasingly complex and dynamic business environment (‘Today’s Complex Project Environments’ section), predicting success is multifaceted. Typically, it includes not only technical, but also financial, marketing, social, legal, ethical, and technological dimensions, many of them fuzzy or unknown at the time of the proposed project evaluation (Thamhain 2011ab). Hence, the DNA of success is highly complex, and outcomes are difficult to predict. Evaluating and selecting projects is not only an art and a science, but has to go beyond a simple cost-benefit analysis for most cases. To be effective, project opportunities must be analyzed relative to their potential value, strength, and importance to the enterprise. We have to understand and examine the whole spectrum of costs, risks, and benefits as part of the evaluation process, far beyond conventional project economics. This requires a comprehensive approach with sophisticated leadership, integrating organizational resources and facilitating a shared vision of risks across organizational borders, time, and space (Baker 2012; Bstieler 2005; Cicmil et al. 2006). Currently, we are quite efficient on the analytical side of project evaluation/selection. However, we are still weak in dealing with the hidden, less obvious dimensions of predicting success that involve a broader spectrum of project performance variables, connecting to the enterprise and its socio-economic environment (Thamhain and Skelton 2007). These are the issues explored in the study reported in this paper.
Today’s complex project environments
Our complex, fast-changing business environment requires sophisticated decision making for selecting the right project with the most desirable outcome assuring success and enterprise performance. Companies are under pressures for quicker, cheaper, and smarter solutions. They are globally networked and can leverage their resources and accelerate their schedules by forming alliances, consortia, and partnerships with other firms, universities, and government agencies, which range from simple cooperative agreements to open innovation, a concept of scouting for new product and service ideas, anywhere in the world. Projects are complex in nature and imbedded in lots of technology. All of these issues contribute to the complexities and uncertainties of project selection which is a great challenge for managers in virtually every segment of industry and government. They include computer, pharmaceutical, automotive, health care, transportation, and financial businesses, just to name a few. New technologies, especially computers and communications, have radically changed the workplace and transformed our global economy, with focus on effectiveness, value, and speed. These techniques offer more sophisticated capabilities for cross-functional integration, resources mobility, market responsiveness, and managerial decision support, but they also require more sophisticated skills both technically and socially, dealing effectively with higher levels of conflict, change, risks, and uncertainty. These challenges resulted in a shift in managerial focus from functional efficiency to effectiveness with attention to organizational interfaces, human factors, and overall enterprise performance.
What we know about project evaluation and selection
Primarily quantitative and rational approaches.
Primarily qualitative and intuitive approaches.
Mixed approaches, combining both quantitative and qualitative methods.
While in the past, decisions on project selection focused on quantitative approaches, such as ROI, payback periods, and net present value for determining desirability and potential success, today’s managers take a more balanced approach between quantitative and qualitative methods. They cast a much wider net for capturing a broad spectrum of variable that go far beyond the scope and limitations of traditional financial evaluations, such as ROI or payback (Kumar 2006).
Most managers are keenly aware of the intricate connections of success variables among organizational systems and processes, which often limit the effectiveness of analytical methods. Especially for complex projects and business processes, managers argue that no single person or group within an enterprise has all the smarts and insight for assessing these multi-variable influences and their cascading effects (Shakhsi-Niaei et al. 2011). Further, no analytical model seems sophisticated enough to represent the complexities and dynamics of all factors that might affect success or failure of a major project (Kavadias and Loch 2004; Zhang et al. 2009; Loch et al. 2001). These managers realize that while analytical methods provide a critically important toolset for project evaluation and selection, these methods also take the collective thinking and collaboration of all the stakeholders and key personnel of the enterprise and its partners to identify and deal with the complexity of risks in today’s business environment. As a result, an increasing number of organizations are complementing their analytical methods with managerial judgment and collective stakeholder experiences that include such broad measures as strategic desirability and projections from past performances (Hadad et al. 2012), hence moving beyond a narrow dependence on just analytical models. In addition, many companies have developed their own ‘systems,’ uniquely designed for dealing with uncertainties in their specific projects and enterprise environment (Henriksen and Traynor 2002; Kavadias and Loch 2004; Kumar 2006; Larson and Gray 2011). These systems emphasize the integration of various tools, often combining quantitative and qualitative methods to cast a wider net for capturing and assessing risk factors beyond the boundaries of conventional methods. Examples are well-known management tools, such as review meetings, Delphi processes, brainstorming, and focus groups, which have been skillfully integrated with analytical methods to leverage their effectiveness and improve their reliability. In addition, a broad spectrum of new and sophisticated tools and techniques, such as user-centered design (UCD), voice of the customer (VoC), and phase-gate processes, evolved which rely by and large on organizational collaboration and collective judgment processes to deal with the broad spectrum of risk variables that are dynamically distributed throughout the enterprise and its external environment.
The missing link
While project evaluation and selection methods have been studied extensively for several decades (Brenner 1994; Cook and Green 2000; Mantel et al. 2011), relatively little has been published on the role of collaboration across the total enterprise for evaluating potential project value (Oral et al. 1991). That is, we know little about management processes that involve the broader project community in a collective cross-functional way for dealing with project selection. The missing link is the people side as a cross-functional tool for collectively evaluating a project proposal toward potential success, one of the areas that are being examined in this paper.
An exploratory field study format is used to investigate managerial approaches for evaluating and selecting project proposals. Specifically, this field study looks at both quantitative and qualitative models, and reports on their use and effectiveness for various project types and situations.
The work presented in this paper is a continuation of my ongoing research into project management alignment with enterprise strategy (Shenhar et al. 2007) and risk management in complex project situations (Thamhain and Skelton 2007; Thamhain 2011ab). The field study reported here focuses especially on data collected between 2008 and 2012.
Summary of field sample statistics
Total sample population
Projects (product developments)
Senior managers and directors
Average project budget
Average project life cycle
Justification for the exploratory field study format
All components of this investigation, such as project management, product development, team work, decision making under uncertainty, technology, and business environment, involve highly complex sets of intricately related variables. Researchers have consistently pointed at the non-linear, often random nature of these processes that involve many facets of the organization, its members, and the environment (Bstieler 2005; Danneels and Kleinschmidt 2001 Nellore and Balachandra 2001; Thamhain 20082009; Verganti and Buganza 2005). Investigating these organizational processes simultaneously is not a simple task, making it unlikely to find simple models appropriate for researching these environments. Because of these complexities and the still evolving nature of these components, their theories and constructs, an exploratory field research format was chosen for the investigation which uses questionnaires in addition to two qualitative methods: participant observation and in-depth retrospective interviewing.
Defining project value, performance, and success
All of these measures are multifaceted. Typically, they include not only technical, but also financial, marketing, social, legal, and ethical dimensions. For most projects, the DNA of success is highly complex, and outcomes are difficult to predict, especially long term. Special attention must be given to the assessment of these variables, with their many facets and reference points throughout the project lifecycle and beyond. Especially, project success is highly judgmental and difficult to determine at the outset, but often only years after project completion, as we can see in aerospace, pharmaceutical, automobile, infrastructure or government service projects. Yet, in spite of these challenges, senior managers, collectively, seem to have a good sense of the potential success and value of a proposed project. We validated this fact in previous studies by testing the agreement among senior managers from various parts and different levels of the enterprise regarding their judgment of project performance (Kruglianskas and Thamhain 2000; Thamhain 20052006). These agreements were measured via Kruskal-Walles analysis of variance by ranks which provided evidence of the strong agreement among the managers on the degree of success expected for a given project under evaluation. Therefore, I used this finding as a basis for arguing the benefits of collective multifunctional judgment in selecting candidate projects, a discussion presented later in this paper. Yet, the topic of ‘measuring success’ and its underlying metrics is an area that needs additional study. Suggestions for future research include the relationship between short-term project performance and long-term project success, and underlying conditions and evaluation criteria that might be useful for predicting future project performance and success.
Based on the findings from this field study, an overview is first given on how companies evaluate and select complex projects, followed by specific recommendations for effectively managing the process. The first segment includes a discussion on the effectiveness of some of the most popular quantitative and qualitative methods available to managers, and an assessment of the situational value of quantitative versus qualitative methods.
How do companies evaluate and select projects
Based on the findings from this field study, supported by additional observations on hundreds of projects during my action research, a framework for effective project evaluation and selection is summarized first and then followed by specific recommendations for effectively managing the process. While any decision model needs to be fine tuned toward specific application, the basic framework and management philosophy of evaluating and selecting project proposals seem to be appropriate for most projects, in spite of their differences in complexities, technologies, organizational structure, and culture among companies.
Evaluation dimensions and decision phases
Project opportunities must be analyzed relative to their potential value, strength, and importance to the enterprise. Four major dimensions should be considered for project evaluation and compared to available alternatives as graphically shown in Figure 1: (1) added value of the new project, consistent with the organizational objectives; (2) resource requests such as cost, personnel, and facilities needed to complete the new project; (3) readiness and ability of the enterprise to execute the project; and (4) managerial belief and desire. A well-organized project evaluation and selection process provides the framework for systematic data gathering and informed decision making toward resource allocation. Typically, the decision process can be broken into phases which are often overlapping and executed concurrently, such as shown in Figure 1 and described below:
Phase I. Deciding initial feasibility: screening and filtering, quick decision on the viability of an emerging project for further evaluation;
Phase II. Deciding strategic value to enterprise: identifying alternatives and options to the proposed project;
Phase III. Deciding detailed feasibility and value: determining the specific value and chances of success for a proposed project;
While the process seems simple, logical, and straightforward, developing meaningful support data is a complex undertaking. It is also expensive, time consuming, and often highly eclectic. Typically, decision making requires the following inputs: (1) specific resource requirements; (2) specific implementation risks; (3) specific benefits, i.e., economics, technology, markets, etc.; (4) benchmarking and comparative analysis of threats and opportunities; and (5) strategic perspective, including long- and short-term value assessment.
Typical criteria considered for project evaluation and selection
The criteria relevant to the evaluation and selection of a particular project depend on the specific project type (i.e., product, service or process), business situation, industry, and market. Typically, evaluation procedures include the following criteria and measures:
• Cash flow, revenue, and profit
• Project cost
• Consistency with business plan
• Project duration (PLC)
• Resource availability
• Impact on other business activities
• Return on investment
• Market share
• Organizational readiness and strength
• Sales volume
• Product lifecycle (deliverables)
• Strategic value
• Project business follow on
• Technical complexity and feasibility
Methods of project evaluation and selection
Producing a comprehensive picture of future value that integrates the large array of potential costs and benefits is both a science and an art. During field interviews, managers confirm the changing paradigm of project evaluation that has been reported in the literature for some time (Baker 2012; Kavadias and Loch 2004; Kumar 2006; Shakhsi-Niaei et al. 2011). That is, not too long ago, in the 1990s and before, rational evaluation processes prevailed by and large for supporting project selections. However, in today’s more complex and dynamic business environment, managers point out that purely rational analytical processes apply only to a limited number of situations. Most of our technologically complex business scenarios require the integration of both analytical and judgmental techniques to capture the broad spectrum of variables affecting project success or failure, necessary for predicting success and making the best choice.
Yet, in spite of the shift toward including judgmental methods more specifically into the selection process, systematic information gathering and standardized methods are at the heart of any project evaluation process and provide the best assurance for reliably predicting project outcome and repeatability of the decision process. In the sample of 105 projects of my field study, managers report the following breakdown of approaches used for project evaluation and selection:
Primarily quantitative and rational approaches: used for 18% of all projects.
Primarily qualitative and intuitive approaches: used for 24% of all projects.
Mixed approaches (quantitative/qualitative combination): used for 58% of all projects.
Given the fact that projects of the field sample consisted of complex, largely high-technology projects, it might look surprising that a relatively large number of project evaluations are based on either primarily quantitative or qualitative methods. That is, one might expect more mixed approaches. Follow-up interviews provide additional insight. Of the sample’s projects, 20% were very large undertakings of over US$100 million each, many of them defense contracts or alike. While these program evaluations included considerable qualitative inputs and judgment from focus groups, industry experts and management, the final go decision was strongly influenced by quantifiable data and highly rational methods of decision making. As a result, more than half of these ‘very large’ projects were judged as being selected by ‘primarily quantitative and rational approaches’. Similarly, we found special situations in the ‘primarily qualitative and intuitive’ category. Approximately one half of the projects in this category were of R&D or exploratory nature with very limited quantifiable data, especially in the area of future value (i.e., future sales, revenue, market share, etc.). While all of these projects went through detailed technical feasibility, budgeting, and project planning processes, the final ‘go decision’ was based on primarily qualitative and intuitive approaches for approximately 60% of these projects.
However, regardless of the primary nature (quantitative, qualitative, or combined focus), all project evaluations included some quantitative and qualitative assessments. Equally important, we observed highly interactive interdisciplinary effort among the various resource groups of the enterprise and its partners during the evaluation and selection processes for all projects, regardless of their type, size, and classification. That is, while management may judge the process leading to the final go/no-go decision as ‘primarily’ one of the three approaches, all decisions included both quantitative and qualitative inputs to some degree. Often, many meetings were needed even before the formal project evaluation process started, just to gain a basic picture of potential benefits, costs, and risks involved, and to determine the type of data needed. All of these activities occurred regardless of primary evaluation/selection focus.
Yet, in all situations, there is the risk of relying too heavily on quantitative data. This was pointed out especially by senior managers. Most concerned were those managers and project leaders who lived through a disappointing project rollout or delivery, or outright project failure, after careful feasibility study and selection procedures at the front end. These managers observed in retrospect that many of their front-end decision processes relied too much on analytical data and models with assumptions that did not hold or forecasts that proved unreliable, while not casting a wide enough net for scanning the business environment. They felt that more experiential information had to be captured from senior management and technology, market and user communities, to complement and validate the analytical models. The issues and potential remedies to these challenges will be explored in the next section.
Going beyond simple formulas and quantitative methods
Comparison of quantitative and qualitative approaches to project evaluation
Clear and simple comparison, ranking, and selection
Search for meaningful evaluation metrics
Broad-based organizational involvement
Encourages data gathering and measurability
Understanding of problems, benefits, opportunities
Problem solving as part of selection process
Broadly distributed knowledge base
Useful input to sensitivity analysis and simulation
Multiple solutions and alternatives
Connectable to many analytical and statistical models
Multifunctional involvement leading to buy-in and risk sharing
Many success factors are not quantifiable
Complex, time-consuming process
Probabilities and weights may change
Biases introduced via organizational power and politics
True measures do not exist
Difficult to procedurize or repeat
Analyses and conclusions are often misleading
Conflict and disagreement over decision/outcome
Masking of hidden problems and opportunities
Does not fit conventional decision processes
Stifle innovative decision making
Intuition and emotion may obscure facts
Lack people involvement, buy-in, commitment
Used for justifying ‘wants’
Ineffective in dealing with multifunctional issues, non-linearities and dynamic situations
Lead to more fact finding than decision making
May mask hidden costs and benefits
Temptation for unnecessary expansion of fact finding
Temptation for acting too quickly and prematurely
Process requires effective managerial leadership
Discussion and recommendations
Effective evaluation and selection of project opportunities involve many variables of the organizational and technological environment, reaching often far beyond cost and revenue measures. While economic models provide an important dimension of the project selection process, most situations are too complex to use simple quantitative methods as the sole basis for decision making. Many of today’s project evaluation procedures include a broad spectrum of variables and rely on a combination of rational and intuitive processes for defining the value of a new project venture to the enterprise. The better organizations understand their business processes, markets, customers, and technologies, the better they will be able to evaluate the value, risks, and challenges of a new project venture. Further, manageability of the evaluation process is critical to its results, especially in complex situations. The process must have a basic structure, discipline, and measurability to be conducive to the intricate multivariable analysis. One method of achieving structure and manageability calls for grouping the evaluation variables into four categories: (1) consistency and strength of the project with the business mission, strategy, and plan; (2) multifunctional ability to produce the project deliverables and objectives, including technical, cost, and time factors; (3) success in the customer environment; and (4) economics, including profitability. Modern phase management, such as Stage-Gate® processes provide managers with the tools for organizing and conducting project evaluations in a systematic way. The following section summarizes suggestions that can help managers in effectively evaluating and selecting projects toward successful implementation.
Seek out relevant information
Meaningful project evaluations require relevant quality information. The four sets of variables related to the strategy, results, customer, and economics, as identified above, can provide a framework for establishing the proper metrics and detailed data gathering.
Ensure competence and relevancy
Ensure that the right people become involved in the data collection and judgmental processes.
Take top-down look first, detail comes later
Detail is less important than information relevancy and evaluator expertise. Do not get hung-up on missing data during the early phases of the project evaluation. Evaluation processes should be iterative. It does not make sense to spend a lot of time and resources on gathering perfect data to justify a ‘no-go’ decision.
Select and match the right people
Whether the project evaluation consists of a simple economic analysis or a complex multifunctional assessment, competent people from functions critical to the overall success of the project should be involved.
Define success criteria
Whether deciding on a single project or choosing among alternatives, evaluation criteria must be defined. They can be quantitative, such as ROI, or qualitative, such as the chances of winning a contract. In either case, these evaluation criteria should cover the true spectrum of factors affecting success and failure of the project(s). The success criteria should be identified by seasoned enterprise personnel. In addition, people from outside of the company, such as vendors, subcontractors, and customers, are often included in this expert group and critical to the development of meaningful success criteria.
Strictly quantitative criteria can be misleading
Be aware of evaluation procedures based on quantitative criteria only (ROI, cost, market share, MARR, etc.). The input data used to calculate these criteria are likely based on rough estimates and are often unreliable. Furthermore, a reliance on strictly quantitative data considers only a narrow spectrum of factors affecting project success or failure, thus ignoring many other important factors, especially those that influence project success in a dynamic or non-linear way, typical for many complex technologically sophisticated undertakings. Evaluations based on predominately quantitative criteria should at least be augmented with some expert judgment as a ‘sanity check’.
Condense criteria list
Combine evaluation criteria, especially among the judgmental categories, to keep the list manageable. As a goal, try to stay within the 12 criteria for each category.
Gain broad perspective
The inputs to the project selection process should include the broadest possible spectrum of data from the business environment that affect success, failure, and limitations of the new project opportunity. Assumptions should be carefully examined.
Communicate across the enterprise
Facilitate communications among evaluators and functional support groups. Define the process for organizing the team and conducting the evaluation and selection process.
Ensure cross-functional representation and cooperation
People on the evaluation team must share a strategic vision across organizational lines. They also must have the desire to support the project if selected for implementation. The purpose, goals, objectives, and relationships of the project to the business mission should be clear to all parties involved in the evaluation/selection process.
Do not lose the big picture
As discussions go into detail during the evaluation, the team should maintain a broad perspective. Two global judgment factors can help focus on the big picture of project success: (1) overall cost-benefit perspective and (2) overall risk of failure assessment. These factors can be recorded on a ten-point scale, −5 to +5. This also leads to an effective two-dimensional graphic display for comparing competing project proposals.
Do your homework between iterations
Project evaluations are usually conducted progressively in iterative cycles. Therefore, the need for more information, clarification, and further analysis surfaces between each cycle. Necessary action items should be properly assigned and followed up to enhance the evaluation quality with each consecutive iteration.
Take a project-oriented approach
Plan, organize, and manage your project evaluation/selection process as a project. Proposal evaluation and selection processes require valuable resources that must be justified and carefully managed.
Resource availability and timing
Do not forget to include in your selection criteria the availability and timing of resources. Many otherwise successful projects fail because they cannot be completed within a required time period.
Use red-team reviews
Set up a special review team of senior personnel. This is especially useful for large and complex projects with major impact on overall business performance. This review team examines the decision parameters, qualitative measures, and assumption used in the evaluation process. Limitations, biases, and misinterpretations that may otherwise remain hidden can often be identified and dealt with.
Stimulate creativity and candor
Senior management should foster an innovative risk-shared ambience for the evaluation team. Especially, the evaluation of complex project situations involves intricate sets of variables. Criteria for success and failure are linked among many subsystems, such as organization, technology, and business, associated with a great deal of risks and uncertainty. Innovative approaches are required to evaluate the true potential of success for these projects. Risk sharing by senior management, recognition, visibility, and a favorable image in terms of high priority, interesting work, and importance of the project to the organization have been found strong drivers toward attracting and holding quality people on the evaluation team and toward gaining their active and innovative participation in the process.
Manage and lead
The evaluation team should be chaired by someone who has the trust, respect, and leadership credibility with the team members. Senior management can positively influence the work environment and the process by providing guidelines, charters, visibility, resources, and active support to the project evaluation team.
In summary, effective project evaluation and selection requires a broad scanning process across all segments of the enterprise and its environment to deal with the risks, uncertainties, ambiguities, and imperfections of data available for assessing the value of a new project venture relative to other opportunities. No single set of broad guidelines exists that guarantees the selection of successful projects. However, the process is not random! A better understanding of the organizational dynamics that affects project performance and the factors that drive cost, revenue, and other benefits can help in gaining a better, more meaningful insight into the future value of a prospective new project. Seeking out both quantitative and qualitative measures incorporated into a combined rational judgmental evaluation process often yields the most reliable predictor of future project value and desirability. Equally important, the process requires managerial leadership and skills in planning, organizing, and communicating. Above all, the leader of the project evaluation team must be a social architect who can unify the multifunctional process and its people. The leader must be able to foster an environment, professionally stimulating and conducive to risk sharing. It also must be effectively linked to the functional support groups needed for project implementation. Finally, organizational strategy must be aligned and integrated with the evaluation/selection process, early and throughout its evaluation cycle. Senior management has an important role in unifying the evaluation team behind the mission objectives and in facilitating the linkages to the stakeholders and ultimate user community. Senior management should further help in providing overall leadership and in building mutual trust, respect, and credibility among the members of the proposal evaluation team, all critical drivers toward a strong partnership of all team members and the basis for an effective enterprise-wide decision-making system. Taken together, this is the environment conducive for cross-functional communication, cooperation, and integration of the intricate variables needed for effective evaluation and selection of project proposals in complex business environments.
Summary of project evaluation and selection techniques
Primarily quantitative and rational approaches.
Primarily qualitative and intuitive approaches.
Mixed approaches, combining both quantitative and qualitative methods.
Quantitative approaches to project evaluation and selection
Net present value (NPV)
Payback period (PBP)
Pacifico and Sobelman project ratings
Description of four project proposals
Project option P1
Management does not accept any new project proposal. Hence, neither investment capital is required nor is any revenue generated.
Project option P2
This opportunity requires a US$1,000 investment at the beginning of the first year and generates a US$200 revenue at the end of each of the following 5 years.
Project option P3
This opportunity requires a US$2,000 investment at the beginning of the first year and generates a variable stream of net revenues at the end of each of the next 5 years as follows: US$1,500, US$1,000, US$800, US$900, and US$1,200.
Project option P4
This opportunity requires a US$5,000 investment at the beginning of the first year and generates a variable stream of net revenues at the end of each of the next 5 years as follows: US$1,000, US$1,500, US$2,000, US$3,000, and US$4,000.
Cash flow and net value calculations of four project options or proposals
Given cash flow
Do-nothing option (P1)
Project option (P2)
Project option (P3)
Project option (P4)
End of year
Net cash flow (Σ · P)
Net present value at the end of year 5 (NPV|N = 5)
Net present value for revenue to continue ∞ (NPV|N = ∞)
Average annual return on investment (ROI|N = 5)
Cost Benefit (CB = ROINPV|N = 5)
Payback period for MARR = 10% (NPBP|i = 10)
Payback Period for MARR = 0% (NPBP|i = 0)
Net present value comparison
This method uses discounted cash flow as the basis for comparing the relative merit of alternative project opportunities. It assumes that all investment costs and revenues are known and that economic analysis is a valid basis for project selection. We can determine the NPV of a single revenue, or stream of future revenues, or costs expected in the future. Two types of presentations are common: (1) present worth (PW) and (2) net present value.
For the examples used in this chapter, we consider the IRR (defined as the average return realized on similar investments) to be the prevailing interest rate.
Net present value
Three special cases exist for the net present value calculation: (1) for a uniform series of revenues or costs over N periods, NPV(An|i, N) = A[(1 + i)N − 1]/i(1 + i)N; (2) for an annuity or interest rate i approaching zero, NPV = A × N; and (3) for the revenue or cost series to continue forever, NPV = A/i. Table 5 applies these formulas to the four project alternatives described in Table 4, showing the most favorable 5-year net present value of US$3,192 for project option P3.
Return on investment comparison
We can then compare the average ROI to the MARR. Given a MARR of 10% for our project environment, all three project options P1, P2, and P3 compare favorable, with project P3 yielding the highest average return on investment of 54%. Although this is a popular measure, it does not permit a meaningful comparative analysis of alternative projects with fluctuating costs and revenues. Furthermore, it does not consider the time value of money.
In our example of four project options (Table 3), project proposal P4 produces the highest cost-benefit of 164% under the given assumption of i = MARR = 10%.
Payback period comparison
In our example of four project options (Table 3), project proposal P3 produces the shortest, most favorable payback period of 1.8 years under the given assumption of i = MARR = 10%.
Pacifico and Sobelman project ratings
The previously discussed methods of evaluating projects rely heavily on the assumption that technical and commercial success is assured, and all costs and revenues are predicable. Because these assumptions do not always hold, many companies have developed their own special procedures and formulas for comparing project alternatives. Two examples illustrate this special category of project evaluation metrics.
The project rating factor
Pacifico’s formula is in essence an ROI calculation adjusted for risk. It includes probability of technical success (0.1 < pT < 1.0), probability of commercial success (0.1 < pC < 1.0), total net revenue over project lifecycle (R), and total capital investment for product development, manufacturing set-up, marketing, and related overheads (TC).
Product development figure of merit
represents a modified cost-benefit measure which takes into account both the development time and commercial lifecycle of the product. It also includes average profit per year (P), estimated product lifecycle (TLC), average development cost per year (C), and years of development (TD).
Qualitative approaches to project evaluation and selection
While quantitative methods provide an important toolset for project proposal evaluation and selection, there is also a growing sense of frustration, especially among managers of complex and technologically advanced undertakings, that reliance on strictly quantitative methods does not always produce the most useful or reliable inputs for decision-making, nor are all methods equally suited for all situations. Therefore, it is not surprising that for project evaluations involving complex sets of business criteria, narrowly focused quantitative methods are often supplemented with broad scanning, intuitive processes, and collective, multifunctional decision making such as Delphi, nominal group technology, brainstorming, focus groups, sensitivity analysis, benchmarking, and UCD. Each of these techniques can either be used by itself to determine the best, most successful, or most valuable option, or these techniques are integrated into a comprehensive analytical framework for collective multifunctional decision making, which is being discussed next.
Collective, multifunctional evaluations
This process relies on subject experts from various functional areas for collectively defining and evaluating broad project success criteria, employing both quantitative and qualitative methods. The first step is to define the specific organizational areas critical to project success and to assign expert evaluators. For example, a product or service development project may typically include organizations such as R&D, engineering, testing, manufacturing, marketing, product assurance, and customer/field services. The function experts should be given the time and resources necessary for the evaluation. They also should have the commitment from senior management for full organizational support. Ideally, these evaluators should be members of the core team ultimately responsible for project implementation.
Ten-point judgment scale: This scale ranges from +5 (most favorable) to − 5 (most unfavorable).
Three-point judgment scale: +1 (favorable), 0 (neutral or cannot judge), −1 (unfavorable).
Five-point judgment scale: A (highly favorable), B (favorable), C (marginally favorable), D (most likely unfavorable), F (definitely unfavorable).
Five-point Likert scale: 1 (strongly agree), 2 (agree), 3 (neutral), 4 (disagree), 5 (strongly disagree).
Weighing of criteria is not recommended for most applications as it complicates and often distorts the collective evaluation. Perspective and judgment are part of the strength and value of these qualitative methods, which can be lost by forcing too much of a quantitative framework on the qualitative evaluation.
The evaluation process
Evaluators first assess and then score all of the success factors they feel qualified to judge. Then, collective discussions follow. Initial discussions of project alternatives, their markets, business opportunities, and technologies involved are usually beneficial, but not necessary for the first round of the evaluation process. The objective of this first round of expert judgments is to get calibrated on the opportunities and challenges presented. Further, each evaluator has the opportunity to recommend (1) actions that could improve the quality and accuracy of the project evaluation, (2) additional data needed, and (3) suggestions for increasing project success. Before meeting at the next group session, agreed-on action items and activities for improving the decision process should be completed. The evaluation process is enhanced with each iteration by producing more accurate, refined, and comprehensive data. Typically, between 3 and 5 iterations are required before a go/no-go decision can be reached for a given project.
Mixed approaches, combining both quantitative and qualitative methods
Mixed approaches are the most common method of evaluating and selecting projects in today’s complex business environment. Virtually all evaluations of project proposals include some form of quantitative and qualitative methods. However, to qualify as a mixed approach, the project evaluation and selection process has to contain a fairly balanced array of both classes of analytical and judgmental tools and techniques of the various types discussed above.
The following is a summary description of terms, variables, and abbreviations used in this chapter:
Cross-functional involves actions which span organizational boundaries.
Phase management involves breaking of projects into natural implementation phases, such as development, production, and marketing, as a basis for project planning, integration, and control. Phase management also provides the framework for concurrent engineering and Stage-Gate® processes.
Project success is a comprehensive measure, defined in both quantitative and qualitative terms which includes economic, market, and strategic objectives.
Stage-Gate® process is a framework originally developed by R Cooper and S Edgett for executing projects within predefined stages with measurable deliverables (at gates) at the end of each stage. These gates also provide the review metrics for ensuring successful transition and integration of the project into the next stage.
Weighing of criteria is a a multiplier associated with specific evaluation criteria.
Annuity (A) is the present worth of a revenue or cost at the end of a period n.
Cost benefit (CB) is the net present value of all ROIs in dollars.
Prevailing interest rate (i).
Internal rate of return (IRR) is the average return on investment realized by a firm on its investment capital.
Minimum attractive rate of return (MARR) on new investments acceptable to an organization.
Net present value (NPV) of a stream of future revenues or costs.
Payback period (PBP) is the time period needed to recover the original investment.
Project rating factor (PR) is a measure developed by Carlo Pacifico for predicting project success.
Present worth (PW, also called annuity) is the present value of a revenue or cost at the end of a period n.
Return on investment (ROI)
Project rating factor (z) is a measure developed by Sobelman for predicting project success.
Dr. Hans Thamhain is a Professor of Management and Director of MOT and Project Management Programs at Bentley University, Boston/Waltham. Dr. Thamhain, held management positions with Verizon, General Electric and ITT, has written over seventy research papers and six professional reference books. He received the IEEE Engineering Manager Award in 2001, PMI’s Distinguished Contribution Award in 1998 and PMI’s Research Achievement Award in 2006. He is profiled in Marquis Who's Who in America and certified as NPDP and PMP.
This article is published under license to BioMed Central Ltd. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.