Skip to main content

Review panel consensus and post-decision commercial performance: a study of early stage technologies

The Journal of Technology Transfer volume 35, pages 253–281 (2010)Cite this article

Abstract

We examine the ability of review panels to predict commercial success when evaluating early-stage technologies from small business. Specifically we examined whether a screening process resulted in greater consensus among grant panel members, to what extent certain panel members “stuck” to their evaluations, and whether information sharing and panel consensus resulted in better predictions of commercial success. In general, we found that expert panel members tend to move toward consensus after discussion, with technical experts being the most “sticky”. While information sharing does not lead to better prediction, increasing consensus among panel members does indicate a slight improvement in prediction accuracy.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Notes

  1. Other commonly used measures of with-in group agreement are the rwg (single item) and rwg(j) (multi-item) (James et al. 1984). Since we are looking at the effect of consensus on success over time, and since the theoretical expected random variation between the pre- and post-presentation scores should be the same for each technology, we chose the simpler measures of range and standard deviation.

References

  • Ajamian, G., & Koen, P. A. (2002). Technology stage gate: A structured process for managing high risk, new technology projects. In P. Belliveau, A. Griffin, & S. Sorermeyer (Eds.), PDMA toolbook for new product development (pp. 267–295). New York: Wiley.

    Google Scholar 

  • Astebro, T. (2004). Key success factors for technological entrepreneurs’ R&D projects. IEEE Transactions on Engineering Management, 51(3), 314–321. doi:10.1109/TEM.2004.830863.

    Article  Google Scholar 

  • Barney, J. (1997). On flipping coins and making technology choices: Luck as an explanation of technological foresight and oversight. In R. Garud, P. R. Nayyar, & Z. B. Shapira (Eds.), Technological innovation: Oversights and foresights (pp. 13–19). New York: Cambridge University Press.

    Google Scholar 

  • Baum, J., & Silverman, B. (2004). Picking winners or making them? Alliances, patents and human capital as selection criteria in venture financing and performance of biotechnology start-ups. Journal of Business Venturing, 19, 411–436. doi:10.1016/S0883-9026(03)00038-7.

    Article  Google Scholar 

  • Bonito, J., & Lambert, B. (2005). Information similarity as a moderator of the effect of gender on participation in small groups: A multilevel analysis. Small Group Research, 36, 139–165. doi:10.1177/1046496404266164.

    Article  Google Scholar 

  • Bunderson, J. S., & Sutcliffe, K. M. (2002). Comparing alternative conceptualizations of functional diversity in management teams: Process and performance effects. Academy of Management Journal, 45(5), 875–893. doi:10.2307/3069319.

    Article  Google Scholar 

  • Carland, J., Hoy, F., Boulton, W., & Carland, J. (1984). Differentiating entrepreneurs from small business owners: A conceptualization. Academy of Management Review, 9, 354–359. doi:10.2307/258448.

    Article  Google Scholar 

  • Chalos, P., & Poon, M. C. C. (2000). Participation and performance in capital budgeting teams. Behavioral Research in Accounting, 12, 199–229.

    Google Scholar 

  • Chidambaram, L., & Tung, L. L. (2005). Is out of sight, out of mind? An empirical study of social loafing in technology-supported groups. Information Systems Research, 16, 149–168. doi:10.1287/isre.1050.0051.

    Article  Google Scholar 

  • Christensen-Szalanski, J., & Fobian, C. (1991). The hindsight bias: A meta-analysis. Organizational Behavior and Human Decision Processes, 48, 147–168. doi:10.1016/0749-5978(91)90010-Q.

    Article  Google Scholar 

  • Cooper, R. (1993). Winning at new products (2nd ed.). Reading, MA: Addison-Wesley.

    Google Scholar 

  • Cooper, R. (1998). Product leadership: Creating and launching superior new products. Reading, MA: Perseus Publishing.

    Google Scholar 

  • Cooper, R. (2001). Winning at new products (3rd ed.). Reading, MA: Perseus Publishing.

    Google Scholar 

  • Cooper, R., Edgett, S., & Kleinschmidt, E. (2002). Optimizing the stage-gate process: What best practice companies are doing—Part I. Research-Technology Management, 45(5), 21–27.

    Google Scholar 

  • Cruz, M. G., Boster, F. J., & Rodriguez, J. I. (1997). The impact of group size and proportion of shared information on the exchange and integration of information in groups. Communication Research, 24(3), 291–313. doi:10.1177/009365097024003004.

    Article  Google Scholar 

  • Cummings, J. N. (2004). Work groups, structural diversity, and knowledge sharing in a global organization. Management Science, 50(3), 352–364. doi:10.1287/mnsc.1030.0134.

    Article  Google Scholar 

  • Dahlin, K. B., Weingart, L. R., & Hinds, P. J. (2005). Team diversity and information use. Academy of Management Journal, 48(6), 1107–1123.

    Article  Google Scholar 

  • Danneels, E. (2004). Disruptive technology reconsidered: A critique and research agenda. Journal of Product Innovation Management, 21, 246–258. doi:10.1111/j.0737-6782.2004.00076.x.

    Article  Google Scholar 

  • Das, T., & Teng, B. (1999). Managing risks in strategic alliances. Academy of Management Executive, 13, 50–62.

    Google Scholar 

  • Davidsson, P., & Honig, B. (2003). The role of social and human capital among nascent entrepreneurs. Journal of Business Venturing, 18, 301–331. doi:10.1016/S0883-9026(02)00097-6.

    Article  Google Scholar 

  • Davis, J. (1996). Group decision making and quantitative judgments: A consensus model. In E. Witte & J. Davis (Eds.), Understanding group behavior: Consensual action by small groups (pp. 35–59). Mahwah, NJ: Erlbaum.

    Google Scholar 

  • DeStephan, R. S. (1983). High and low consensus groups: A content and relational interaction analysis. Small Group Behavior, 14, 143–162. doi:10.1177/104649648301400202.

    Article  Google Scholar 

  • Devine, D. J. (1999). Effects of cognitive ability, task knowledge, information sharing, and conflict on group decision-making effectiveness. Small Group Research, 30(5), 608–634. doi:10.1177/104649649903000506.

    Article  Google Scholar 

  • Eldred, E., & McGrath, M. (1997). Commercializing new technology-II. Research Technology Management, 40(2), 29–33.

    Google Scholar 

  • Eldred, E., & Shapiro, A. (1996). Technology management. In M. E. McGrath (Ed.), Setting the PACE in product development. Boston, MA: Butterworth and Heinemann.

    Google Scholar 

  • Elton, E., Martin, J., & Blake, C. (1996). Survivorship bias and mutual fund performance. Review of Financial Studies, 9, 1097–1120. doi:10.1093/rfs/9.4.1097.

    Article  Google Scholar 

  • Farmer, S. M., & Hyatt, C. W. (1994). Effects of task language demands and task complexity on computer-mediated work groups. Small Group Research, 25(3), 331–366. doi:10.1177/1046496494253001.

    Article  Google Scholar 

  • Franz, T. M., & Larson, J. R., Jr. (2002). The impact of experts on information sharing during group discussion. Small Group Research, 33(4), 383–411. doi:10.1177/104649640203300401.

    Article  Google Scholar 

  • Galbraith, C., DeNoble, A., Ehrlich, S., & Kline, D. (2007). Can experts really assess future technology success? A neural network and Bayesian analysis of early stage technology proposals. The Journal of High Technology Management Research, 17, 125–137. doi:10.1016/j.hitech.2006.11.002.

    Article  Google Scholar 

  • Galbraith, C., Ehrlich, S., & DeNoble, A. (2006). Predicting technology success: Identifying key predictors and assessing expert evaluation for advanced technologies. The Journal of Technology Transfer, 32(1), 673–684. doi:10.1007/s10961-006-0022-8.

    Article  Google Scholar 

  • Galbraith, C., & Zeznock, L. (2008). The use of formalized rating systems in early-stage angel investment screening. SE INFORMS (electronic) proceedings, October, Myrtle Beach, NC.

  • Galinsky, A. D., & Kray, L. J. (2004). From thinking about what might have been to sharing what we know: The effects of counterfactual mind-sets on information sharing in groups. Journal of Experimental Social Psychology, 40, 606–618. doi:10.1016/j.jesp.2003.11.005.

    Article  Google Scholar 

  • Govindarajan, V., & Kopalle, P. K. (2006). The usefulness of measuring disruptiveness of innovations ex post in making ex ante predictions. Journal of Product Innovation Management, 23, 12–18. doi:10.1111/j.1540-5885.2005.00176.x.

    Article  Google Scholar 

  • Greenhalgh, L., & Chapman, D. H. (1998). Negotiator relationships: Construct measurement, and demonstration of their impact on the process and outcomes of negotiation. Group Decision and Negotiation, 7, 465–489. doi:10.1023/A:1008694307035.

    Article  Google Scholar 

  • Henningsen, D. D., & Henningsen, M. L. M. (2003). Examining social influence in information-sharing contexts. Small Group Research, 34(4), 391–412. doi:10.1177/1046496403251933.

    Article  Google Scholar 

  • Henningsen, D. D., Henningsen, M. L. M., Jakobsen, L., & Borton, I. (2004). It’s good to be leader: The influence of randomly and systematically selected leaders on decision-making. Group Dynamics: Theory, Research, and Practice, 8(1), 62–76. doi:10.1037/1089-2699.8.1.62.

    Article  Google Scholar 

  • Heslop, L., McGregor, E., & Griffith, M. (2001). Development of a technology readiness assessment measure: The cloverleaf model of technology transfer. Journal of Technology Transfer, 26, 369–384. doi:10.1023/A:1011139021356.

    Article  Google Scholar 

  • Hopthrow, T., & Hulbert, L. (2005). The effect of group decision making on cooperation in social dilemmas. Group Processes & Intergroup Relations, 8(1), 89–100. doi:10.1177/1368430205049253.

    Article  Google Scholar 

  • Jehn, K. A., & Shah, P. P. (1997). Interpersonal relationships and task performance: An examination of mediating processes in friendship and acquaintance groups. Journal of Personality and Social Psychology, 72, 775–790. doi:10.1037/0022-3514.72.4.775.

    Article  Google Scholar 

  • Keck, S. L. (1997). Top management team structure: Differential effects by environmental context. Organization Science, 8, 143–156. doi:10.1287/orsc.8.2.143.

    Article  Google Scholar 

  • Kelly, J., & Loving, T. (2004). Time pressure and group performance: Exploring underlying processes in the attentional focus model. Journal of Experimental Social Psychology, 40(2), 185–198. doi:10.1016/S0022-1031(03)00094-5.

    Article  Google Scholar 

  • Koen, P., Ajamian, G., Boyce, S., Clamen, A., Fisher, E., Fountoulakis, S., et al. (2002). Fuzzy-front end: Effective methods, tools and techniques. In P. Belliveau, A. Griffin, & S. Sorermeyer (Eds.), PDMA toolbook for new product development (pp. 2–35). New York: Wiley.

    Google Scholar 

  • Laughlin, P. R. (1996). Group decision making and collective induction. In E. H. White & J. H. Davis (Eds.), Understanding group behavior: Consensual action by small groups (pp. 61–80). Mahwah, NJ: Erlbaum.

    Google Scholar 

  • Linton, J., Walsh, S., & Morabito, J. (2002). Analysis, ranking and selection of R&D projects in a portfolio. R & D Management, 32(32), 139–148. doi:10.1111/1467-9310.00246.

    Google Scholar 

  • MacMillan, I. C., Seigel, R., & Subba Narasimha, P. N. (1985). Criteria used by venture capitalist to evaluate new venture proposals. Journal of Business Venturing, 1, 119–128. doi:10.1016/0883-9026(85)90011-4.

    Article  Google Scholar 

  • MacMillan, I. C., Zemann, L., & Subba Narasimha, P. N. (1987). Criteria distinguishing unsuccessful ventures in the venture screening process. Journal of Business Venturing, 2, 123–137. doi:10.1016/0883-9026(87)90003-6.

    Article  Google Scholar 

  • Massachusetts Institute of Technology (MIT). (2000). Venture support systems project: Angel investors. Boston, MA: MIT Press.

  • McCollough, R. (1998). The ProGrid system: A customized expert system for evaluation of intellectual capital. In Proceedings of the workshop on evaluating technology for commercial exploitation. Ottawa: Federal Partners for Technology Transfer.

  • McGrath, M. (1996). Setting the PACE in product development. Boston, MA: Butterworth-Heineman.

    Google Scholar 

  • Mesmer-Magnus, J., & DeChurch, L. (2009). Information sharing and team performance: A meta-analysis. Journal of Applied Psychology, 94(2), 535–546.

    Article  Google Scholar 

  • Mock, J., Kenkeremath, D., & Janis, F. (1993). Moving R&D to the marketplace: A guidebook for technology transfer managers. Falls Church, VA: Technology Prospects, Inc.

    Google Scholar 

  • Moye, N. A., & Langfred, C. W. (2004). Information sharing and group conflict: Going beyond decision making to understand the effects of information sharing on group performance. The International Journal of Conflict Management, 15, 381–435. doi:10.1108/eb022919.

    Article  Google Scholar 

  • Murray, A. I. (1989). Top management group heterogeneity and firm performance. Strategic Management Journal, 10, 125–141. doi:10.1002/smj.4250100710.

    Article  Google Scholar 

  • Napier, R. W., & Gershenfeld, M. K. (1999). Groups: Theory and experience (6th ed.). New York: Houghlin Mifflin Company.

    Google Scholar 

  • Ohtsubo, Y., Masuchi, A., & Nakanishi, D. (2002). Majority influence process in group judgment: Test of the social judgment scheme model in a group polarization context. Group Processes & Intergroup Relations, 5(3), 249–261. doi:10.1177/1368430202005003005.

    Article  Google Scholar 

  • Okhuysen, G. A., & Eisenhardt, K. M. (2002). Integrating knowledge in groups: How formal interventions enable flexibility. Organization Science, 13(4), 370–386. doi:10.1287/orsc.13.4.370.2947.

    Article  Google Scholar 

  • Ozer, M. (1999). A survey of new product evaluation models. Journal of Product Innovation Management, 16, 77–94. doi:10.1016/S0737-6782(98)00037-X.

    Article  Google Scholar 

  • Payne, W., & Macarty, M. (2002). The anatomy of an angel investing network: Tech Coast Angels. Venture Capital, 4(4), 331–336. doi:10.1080/1369106022000024950.

    Article  Google Scholar 

  • Phillips, K. W., Northcraft, G. B., & Neale, M. A. (2006). Surface-level diversity and decision-making in groups: When does deep-level similarity help? Group Processes & Intergroup Relations, 9(4), 467–482. doi:10.1177/1368430206067557.

    Article  Google Scholar 

  • Priem, R. L., Harrison, D. A., & Muir, N. K. (1995). Structured conflict and consensus outcomes in group decision making. Journal of Management, 21, 691–710. doi:10.1177/014920639502100406.

    Article  Google Scholar 

  • Quigley, N. R., Tesluk, P. E., Locke, E. A., & Bartol, K. M. (2007). A multilevel investigation of the motivational mechanisms underlying knowledge sharing and performance. Organization Science, 18(1), 71–88. doi:10.1287/orsc.1060.0223.

    Article  Google Scholar 

  • Rothstein, M., & Jackson, D. N. (1980). Decision making in the employment interview: An experimental approach. The Journal of Applied Psychology, 65, 271–283. doi:10.1037/0021-9010.65.3.271.

    Article  Google Scholar 

  • Savadori, L., Van Swol, L. M., & Sniezek, J. A. (2001). Information sampling and confidence within groups and judge advisor systems. Communication Research, 28(6), 737–771. doi:10.1177/009365001028006002.

    Article  Google Scholar 

  • Schittekatte, M., & Van Hiel, A. (1996). Effects of partially shared information and awareness of unshared information on information sampling. Small Group Research, 27(3), 431–449. doi:10.1177/1046496496273006.

    Article  Google Scholar 

  • Schoenborn, T., & Fredette, J. (1997). So you want to set up a technology evaluation program? Cherry Hill, NJ: Federal Laboratory Consortium.

    Google Scholar 

  • Schweiger, P. M., Sandberg, W. R., & Ragan, J. W. (1986). Group approaches for improving strategic decision making: A comparative analysis of dialectical inquiry, devil’s advocacy, and consensus. Academy of Management Journal, 29, 51–71. doi:10.2307/255859.

    Article  Google Scholar 

  • Shane, S., & Stuart, T. (2002). Organizational endowments and the performance of university start-ups. Management Science, 48(1), 154–170. doi:10.1287/mnsc.48.1.154.14280.

    Article  Google Scholar 

  • Shepard, D., Zacharakis, A., & Baron, R. (2003). VCs decision processes: Evidence suggesting more experience may not always be better. Journal of Business Venturing, 18(3), 381–401. doi:10.1016/S0883-9026(02)00099-X.

    Article  Google Scholar 

  • Srivastava, A. (2001). Antecedents and effects of knowledge sharing in teams: A field study. An unpublished doctoral dissertation submitted to the University of Maryland, College Park.

  • Stasser, G. (1992). Information salience and the discovery of hidden profiles by decision-making groups: A “thought” experiment. Organizational Behavior and Human Decision Processes, 52, 156–181. doi:10.1016/0749-5978(92)90049-D.

    Article  Google Scholar 

  • Stasser, G., & Titus, W. (1985). Pooling of unshared information in group decision making: Biased information sampling during discussion. Journal of Personality and Social Psychology, 48(6), 1467–1478. doi:10.1037/0022-3514.48.6.1467.

    Article  Google Scholar 

  • Steiner, I. D. (1982). Heuristic models of groupthink. In M. Brandstatter, J. H. Davis, & G. Stocker-Krichgauer (Eds.), Group decision making (pp. 503–524). New York: Academic Press.

    Google Scholar 

  • Stewart, D. D., Billings, R. S., & Stasser, G. (1998). Accountability and the discussion of unshared, critical information in decision-making groups. Group Dynamics: Theory, Research, and Practice, 2(1), 18–23. doi:10.1037/1089-2699.2.1.18.

    Article  Google Scholar 

  • Thatcher, A., & De La Cour, A. (2003). Small group decision-making in face-to-face and computer-mediated environments: The role of personality. Behaviour & Information Technology, 22(3), 203–218. doi:10.1080/0144929031000117071.

    Article  Google Scholar 

  • Tjosvold, D., & Field, R. (1983). Effects of social context on consensus and majority vote decision making. Academy of Management Journal, 26, 500–506. doi:10.2307/256261.

    Article  Google Scholar 

  • Tyebjee, T., & Bruno, A. (1984). A model of venture capitalist investment activity. Management Science, 30, 1051–1056. doi:10.1287/mnsc.30.9.1051.

    Article  Google Scholar 

  • Wiltbank, R. (2005). Investment practices and outcomes of informal venture investors. Venture Capital, 7(4), 343. doi:10.1080/13691060500348876.

    Article  Google Scholar 

  • Winquist, J. R., & Larson, J. R., Jr. (1998). Information pooling: When it impacts group decision making. Journal of Personality and Social Psychology, 74(2), 371–377. doi:10.1037/0022-3514.74.2.371.

    Article  Google Scholar 

  • Wittenbaum, G. M., & Bowman, J. M. (2004). A social validation explanation for mutual enhancement. Journal of Experimental Social Psychology, 40, 169–184. doi:10.1016/S0022-1031(03)00091-X.

    Article  Google Scholar 

  • Xiao, Y., Seagull, F. J., Mackenzie, C. F., Klein, K. J., & Ziegert, J. (2008). Adaptation of team communication patterns: Exploring the effects of leadership at a distance, task urgency, and shared team experience. In S. P. Weisband (Ed.), Leadership at a distance: Research in technology-supported work (pp. 71–93). London: Lawrence Erlbaum Associates Publishers.

    Google Scholar 

  • Zacharakis, A., & Meyer, G. (1998). A lack of insight: Do venture capitalists really understand their own decision process? Journal of Business Venturing, 13(1), 57–76.

    Article  Google Scholar 

  • Zacharakis, A., & Meyer, G. (2000). The potential for actuarial decision models: Can they improve the venture capital investment decision? Journal of Business Venturing, 15, 323–346. doi:10.1016/S0883-9026(98)00016-0.

    Article  Google Scholar 

  • Zacharakis, A., & Shepard, D. (2001). The nature of information and overconfidence on venture capitalists’ decision making. Journal of Business Venturing, 16, 311–332. doi:10.1016/S0883-9026(99)00052-X.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Management, Cameron School of Business, University of North Carolina Wilmington, 601 South College Road, Wilmington, NC, 28403, USA

    Craig S. Galbraith & Jessica Mesmer-Magnus

  2. Department of Management, College of Business Administration, San Diego State University, 5500 Campanile Drive, San Diego, CA, 92182–8236, USA

    Alex F. DeNoble

  3. Entrepreneurial Management Center, San Diego State University, 5250 Campanile Drive, San Diego, CA, 92182, USA

    Sanford B. Ehrlich

Authors
  1. Craig S. Galbraith
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Alex F. DeNoble
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sanford B. Ehrlich
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jessica Mesmer-Magnus
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Craig S. Galbraith.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Galbraith, C.S., DeNoble, A.F., Ehrlich, S.B. et al. Review panel consensus and post-decision commercial performance: a study of early stage technologies. J Technol Transf 35, 253–281 (2010). https://doi.org/10.1007/s10961-009-9122-6

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10961-009-9122-6

Keywords

  • Grant review
  • Commercialization success
  • Prediction
  • Panel consensus

JEL Classification

  • O32