Abstract
We examine the ability of review panels to predict commercial success when evaluating early-stage technologies from small business. Specifically we examined whether a screening process resulted in greater consensus among grant panel members, to what extent certain panel members “stuck” to their evaluations, and whether information sharing and panel consensus resulted in better predictions of commercial success. In general, we found that expert panel members tend to move toward consensus after discussion, with technical experts being the most “sticky”. While information sharing does not lead to better prediction, increasing consensus among panel members does indicate a slight improvement in prediction accuracy.
Similar content being viewed by others
Notes
Other commonly used measures of with-in group agreement are the rwg (single item) and rwg(j) (multi-item) (James et al. 1984). Since we are looking at the effect of consensus on success over time, and since the theoretical expected random variation between the pre- and post-presentation scores should be the same for each technology, we chose the simpler measures of range and standard deviation.
References
Ajamian, G., & Koen, P. A. (2002). Technology stage gate: A structured process for managing high risk, new technology projects. In P. Belliveau, A. Griffin, & S. Sorermeyer (Eds.), PDMA toolbook for new product development (pp. 267–295). New York: Wiley.
Astebro, T. (2004). Key success factors for technological entrepreneurs’ R&D projects. IEEE Transactions on Engineering Management, 51(3), 314–321. doi:10.1109/TEM.2004.830863.
Barney, J. (1997). On flipping coins and making technology choices: Luck as an explanation of technological foresight and oversight. In R. Garud, P. R. Nayyar, & Z. B. Shapira (Eds.), Technological innovation: Oversights and foresights (pp. 13–19). New York: Cambridge University Press.
Baum, J., & Silverman, B. (2004). Picking winners or making them? Alliances, patents and human capital as selection criteria in venture financing and performance of biotechnology start-ups. Journal of Business Venturing, 19, 411–436. doi:10.1016/S0883-9026(03)00038-7.
Bonito, J., & Lambert, B. (2005). Information similarity as a moderator of the effect of gender on participation in small groups: A multilevel analysis. Small Group Research, 36, 139–165. doi:10.1177/1046496404266164.
Bunderson, J. S., & Sutcliffe, K. M. (2002). Comparing alternative conceptualizations of functional diversity in management teams: Process and performance effects. Academy of Management Journal, 45(5), 875–893. doi:10.2307/3069319.
Carland, J., Hoy, F., Boulton, W., & Carland, J. (1984). Differentiating entrepreneurs from small business owners: A conceptualization. Academy of Management Review, 9, 354–359. doi:10.2307/258448.
Chalos, P., & Poon, M. C. C. (2000). Participation and performance in capital budgeting teams. Behavioral Research in Accounting, 12, 199–229.
Chidambaram, L., & Tung, L. L. (2005). Is out of sight, out of mind? An empirical study of social loafing in technology-supported groups. Information Systems Research, 16, 149–168. doi:10.1287/isre.1050.0051.
Christensen-Szalanski, J., & Fobian, C. (1991). The hindsight bias: A meta-analysis. Organizational Behavior and Human Decision Processes, 48, 147–168. doi:10.1016/0749-5978(91)90010-Q.
Cooper, R. (1993). Winning at new products (2nd ed.). Reading, MA: Addison-Wesley.
Cooper, R. (1998). Product leadership: Creating and launching superior new products. Reading, MA: Perseus Publishing.
Cooper, R. (2001). Winning at new products (3rd ed.). Reading, MA: Perseus Publishing.
Cooper, R., Edgett, S., & Kleinschmidt, E. (2002). Optimizing the stage-gate process: What best practice companies are doing—Part I. Research-Technology Management, 45(5), 21–27.
Cruz, M. G., Boster, F. J., & Rodriguez, J. I. (1997). The impact of group size and proportion of shared information on the exchange and integration of information in groups. Communication Research, 24(3), 291–313. doi:10.1177/009365097024003004.
Cummings, J. N. (2004). Work groups, structural diversity, and knowledge sharing in a global organization. Management Science, 50(3), 352–364. doi:10.1287/mnsc.1030.0134.
Dahlin, K. B., Weingart, L. R., & Hinds, P. J. (2005). Team diversity and information use. Academy of Management Journal, 48(6), 1107–1123.
Danneels, E. (2004). Disruptive technology reconsidered: A critique and research agenda. Journal of Product Innovation Management, 21, 246–258. doi:10.1111/j.0737-6782.2004.00076.x.
Das, T., & Teng, B. (1999). Managing risks in strategic alliances. Academy of Management Executive, 13, 50–62.
Davidsson, P., & Honig, B. (2003). The role of social and human capital among nascent entrepreneurs. Journal of Business Venturing, 18, 301–331. doi:10.1016/S0883-9026(02)00097-6.
Davis, J. (1996). Group decision making and quantitative judgments: A consensus model. In E. Witte & J. Davis (Eds.), Understanding group behavior: Consensual action by small groups (pp. 35–59). Mahwah, NJ: Erlbaum.
DeStephan, R. S. (1983). High and low consensus groups: A content and relational interaction analysis. Small Group Behavior, 14, 143–162. doi:10.1177/104649648301400202.
Devine, D. J. (1999). Effects of cognitive ability, task knowledge, information sharing, and conflict on group decision-making effectiveness. Small Group Research, 30(5), 608–634. doi:10.1177/104649649903000506.
Eldred, E., & McGrath, M. (1997). Commercializing new technology-II. Research Technology Management, 40(2), 29–33.
Eldred, E., & Shapiro, A. (1996). Technology management. In M. E. McGrath (Ed.), Setting the PACE in product development. Boston, MA: Butterworth and Heinemann.
Elton, E., Martin, J., & Blake, C. (1996). Survivorship bias and mutual fund performance. Review of Financial Studies, 9, 1097–1120. doi:10.1093/rfs/9.4.1097.
Farmer, S. M., & Hyatt, C. W. (1994). Effects of task language demands and task complexity on computer-mediated work groups. Small Group Research, 25(3), 331–366. doi:10.1177/1046496494253001.
Franz, T. M., & Larson, J. R., Jr. (2002). The impact of experts on information sharing during group discussion. Small Group Research, 33(4), 383–411. doi:10.1177/104649640203300401.
Galbraith, C., DeNoble, A., Ehrlich, S., & Kline, D. (2007). Can experts really assess future technology success? A neural network and Bayesian analysis of early stage technology proposals. The Journal of High Technology Management Research, 17, 125–137. doi:10.1016/j.hitech.2006.11.002.
Galbraith, C., Ehrlich, S., & DeNoble, A. (2006). Predicting technology success: Identifying key predictors and assessing expert evaluation for advanced technologies. The Journal of Technology Transfer, 32(1), 673–684. doi:10.1007/s10961-006-0022-8.
Galbraith, C., & Zeznock, L. (2008). The use of formalized rating systems in early-stage angel investment screening. SE INFORMS (electronic) proceedings, October, Myrtle Beach, NC.
Galinsky, A. D., & Kray, L. J. (2004). From thinking about what might have been to sharing what we know: The effects of counterfactual mind-sets on information sharing in groups. Journal of Experimental Social Psychology, 40, 606–618. doi:10.1016/j.jesp.2003.11.005.
Govindarajan, V., & Kopalle, P. K. (2006). The usefulness of measuring disruptiveness of innovations ex post in making ex ante predictions. Journal of Product Innovation Management, 23, 12–18. doi:10.1111/j.1540-5885.2005.00176.x.
Greenhalgh, L., & Chapman, D. H. (1998). Negotiator relationships: Construct measurement, and demonstration of their impact on the process and outcomes of negotiation. Group Decision and Negotiation, 7, 465–489. doi:10.1023/A:1008694307035.
Henningsen, D. D., & Henningsen, M. L. M. (2003). Examining social influence in information-sharing contexts. Small Group Research, 34(4), 391–412. doi:10.1177/1046496403251933.
Henningsen, D. D., Henningsen, M. L. M., Jakobsen, L., & Borton, I. (2004). It’s good to be leader: The influence of randomly and systematically selected leaders on decision-making. Group Dynamics: Theory, Research, and Practice, 8(1), 62–76. doi:10.1037/1089-2699.8.1.62.
Heslop, L., McGregor, E., & Griffith, M. (2001). Development of a technology readiness assessment measure: The cloverleaf model of technology transfer. Journal of Technology Transfer, 26, 369–384. doi:10.1023/A:1011139021356.
Hopthrow, T., & Hulbert, L. (2005). The effect of group decision making on cooperation in social dilemmas. Group Processes & Intergroup Relations, 8(1), 89–100. doi:10.1177/1368430205049253.
Jehn, K. A., & Shah, P. P. (1997). Interpersonal relationships and task performance: An examination of mediating processes in friendship and acquaintance groups. Journal of Personality and Social Psychology, 72, 775–790. doi:10.1037/0022-3514.72.4.775.
Keck, S. L. (1997). Top management team structure: Differential effects by environmental context. Organization Science, 8, 143–156. doi:10.1287/orsc.8.2.143.
Kelly, J., & Loving, T. (2004). Time pressure and group performance: Exploring underlying processes in the attentional focus model. Journal of Experimental Social Psychology, 40(2), 185–198. doi:10.1016/S0022-1031(03)00094-5.
Koen, P., Ajamian, G., Boyce, S., Clamen, A., Fisher, E., Fountoulakis, S., et al. (2002). Fuzzy-front end: Effective methods, tools and techniques. In P. Belliveau, A. Griffin, & S. Sorermeyer (Eds.), PDMA toolbook for new product development (pp. 2–35). New York: Wiley.
Laughlin, P. R. (1996). Group decision making and collective induction. In E. H. White & J. H. Davis (Eds.), Understanding group behavior: Consensual action by small groups (pp. 61–80). Mahwah, NJ: Erlbaum.
Linton, J., Walsh, S., & Morabito, J. (2002). Analysis, ranking and selection of R&D projects in a portfolio. R & D Management, 32(32), 139–148. doi:10.1111/1467-9310.00246.
MacMillan, I. C., Seigel, R., & Subba Narasimha, P. N. (1985). Criteria used by venture capitalist to evaluate new venture proposals. Journal of Business Venturing, 1, 119–128. doi:10.1016/0883-9026(85)90011-4.
MacMillan, I. C., Zemann, L., & Subba Narasimha, P. N. (1987). Criteria distinguishing unsuccessful ventures in the venture screening process. Journal of Business Venturing, 2, 123–137. doi:10.1016/0883-9026(87)90003-6.
Massachusetts Institute of Technology (MIT). (2000). Venture support systems project: Angel investors. Boston, MA: MIT Press.
McCollough, R. (1998). The ProGrid system: A customized expert system for evaluation of intellectual capital. In Proceedings of the workshop on evaluating technology for commercial exploitation. Ottawa: Federal Partners for Technology Transfer.
McGrath, M. (1996). Setting the PACE in product development. Boston, MA: Butterworth-Heineman.
Mesmer-Magnus, J., & DeChurch, L. (2009). Information sharing and team performance: A meta-analysis. Journal of Applied Psychology, 94(2), 535–546.
Mock, J., Kenkeremath, D., & Janis, F. (1993). Moving R&D to the marketplace: A guidebook for technology transfer managers. Falls Church, VA: Technology Prospects, Inc.
Moye, N. A., & Langfred, C. W. (2004). Information sharing and group conflict: Going beyond decision making to understand the effects of information sharing on group performance. The International Journal of Conflict Management, 15, 381–435. doi:10.1108/eb022919.
Murray, A. I. (1989). Top management group heterogeneity and firm performance. Strategic Management Journal, 10, 125–141. doi:10.1002/smj.4250100710.
Napier, R. W., & Gershenfeld, M. K. (1999). Groups: Theory and experience (6th ed.). New York: Houghlin Mifflin Company.
Ohtsubo, Y., Masuchi, A., & Nakanishi, D. (2002). Majority influence process in group judgment: Test of the social judgment scheme model in a group polarization context. Group Processes & Intergroup Relations, 5(3), 249–261. doi:10.1177/1368430202005003005.
Okhuysen, G. A., & Eisenhardt, K. M. (2002). Integrating knowledge in groups: How formal interventions enable flexibility. Organization Science, 13(4), 370–386. doi:10.1287/orsc.13.4.370.2947.
Ozer, M. (1999). A survey of new product evaluation models. Journal of Product Innovation Management, 16, 77–94. doi:10.1016/S0737-6782(98)00037-X.
Payne, W., & Macarty, M. (2002). The anatomy of an angel investing network: Tech Coast Angels. Venture Capital, 4(4), 331–336. doi:10.1080/1369106022000024950.
Phillips, K. W., Northcraft, G. B., & Neale, M. A. (2006). Surface-level diversity and decision-making in groups: When does deep-level similarity help? Group Processes & Intergroup Relations, 9(4), 467–482. doi:10.1177/1368430206067557.
Priem, R. L., Harrison, D. A., & Muir, N. K. (1995). Structured conflict and consensus outcomes in group decision making. Journal of Management, 21, 691–710. doi:10.1177/014920639502100406.
Quigley, N. R., Tesluk, P. E., Locke, E. A., & Bartol, K. M. (2007). A multilevel investigation of the motivational mechanisms underlying knowledge sharing and performance. Organization Science, 18(1), 71–88. doi:10.1287/orsc.1060.0223.
Rothstein, M., & Jackson, D. N. (1980). Decision making in the employment interview: An experimental approach. The Journal of Applied Psychology, 65, 271–283. doi:10.1037/0021-9010.65.3.271.
Savadori, L., Van Swol, L. M., & Sniezek, J. A. (2001). Information sampling and confidence within groups and judge advisor systems. Communication Research, 28(6), 737–771. doi:10.1177/009365001028006002.
Schittekatte, M., & Van Hiel, A. (1996). Effects of partially shared information and awareness of unshared information on information sampling. Small Group Research, 27(3), 431–449. doi:10.1177/1046496496273006.
Schoenborn, T., & Fredette, J. (1997). So you want to set up a technology evaluation program? Cherry Hill, NJ: Federal Laboratory Consortium.
Schweiger, P. M., Sandberg, W. R., & Ragan, J. W. (1986). Group approaches for improving strategic decision making: A comparative analysis of dialectical inquiry, devil’s advocacy, and consensus. Academy of Management Journal, 29, 51–71. doi:10.2307/255859.
Shane, S., & Stuart, T. (2002). Organizational endowments and the performance of university start-ups. Management Science, 48(1), 154–170. doi:10.1287/mnsc.48.1.154.14280.
Shepard, D., Zacharakis, A., & Baron, R. (2003). VCs decision processes: Evidence suggesting more experience may not always be better. Journal of Business Venturing, 18(3), 381–401. doi:10.1016/S0883-9026(02)00099-X.
Srivastava, A. (2001). Antecedents and effects of knowledge sharing in teams: A field study. An unpublished doctoral dissertation submitted to the University of Maryland, College Park.
Stasser, G. (1992). Information salience and the discovery of hidden profiles by decision-making groups: A “thought” experiment. Organizational Behavior and Human Decision Processes, 52, 156–181. doi:10.1016/0749-5978(92)90049-D.
Stasser, G., & Titus, W. (1985). Pooling of unshared information in group decision making: Biased information sampling during discussion. Journal of Personality and Social Psychology, 48(6), 1467–1478. doi:10.1037/0022-3514.48.6.1467.
Steiner, I. D. (1982). Heuristic models of groupthink. In M. Brandstatter, J. H. Davis, & G. Stocker-Krichgauer (Eds.), Group decision making (pp. 503–524). New York: Academic Press.
Stewart, D. D., Billings, R. S., & Stasser, G. (1998). Accountability and the discussion of unshared, critical information in decision-making groups. Group Dynamics: Theory, Research, and Practice, 2(1), 18–23. doi:10.1037/1089-2699.2.1.18.
Thatcher, A., & De La Cour, A. (2003). Small group decision-making in face-to-face and computer-mediated environments: The role of personality. Behaviour & Information Technology, 22(3), 203–218. doi:10.1080/0144929031000117071.
Tjosvold, D., & Field, R. (1983). Effects of social context on consensus and majority vote decision making. Academy of Management Journal, 26, 500–506. doi:10.2307/256261.
Tyebjee, T., & Bruno, A. (1984). A model of venture capitalist investment activity. Management Science, 30, 1051–1056. doi:10.1287/mnsc.30.9.1051.
Wiltbank, R. (2005). Investment practices and outcomes of informal venture investors. Venture Capital, 7(4), 343. doi:10.1080/13691060500348876.
Winquist, J. R., & Larson, J. R., Jr. (1998). Information pooling: When it impacts group decision making. Journal of Personality and Social Psychology, 74(2), 371–377. doi:10.1037/0022-3514.74.2.371.
Wittenbaum, G. M., & Bowman, J. M. (2004). A social validation explanation for mutual enhancement. Journal of Experimental Social Psychology, 40, 169–184. doi:10.1016/S0022-1031(03)00091-X.
Xiao, Y., Seagull, F. J., Mackenzie, C. F., Klein, K. J., & Ziegert, J. (2008). Adaptation of team communication patterns: Exploring the effects of leadership at a distance, task urgency, and shared team experience. In S. P. Weisband (Ed.), Leadership at a distance: Research in technology-supported work (pp. 71–93). London: Lawrence Erlbaum Associates Publishers.
Zacharakis, A., & Meyer, G. (1998). A lack of insight: Do venture capitalists really understand their own decision process? Journal of Business Venturing, 13(1), 57–76.
Zacharakis, A., & Meyer, G. (2000). The potential for actuarial decision models: Can they improve the venture capital investment decision? Journal of Business Venturing, 15, 323–346. doi:10.1016/S0883-9026(98)00016-0.
Zacharakis, A., & Shepard, D. (2001). The nature of information and overconfidence on venture capitalists’ decision making. Journal of Business Venturing, 16, 311–332. doi:10.1016/S0883-9026(99)00052-X.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Galbraith, C.S., DeNoble, A.F., Ehrlich, S.B. et al. Review panel consensus and post-decision commercial performance: a study of early stage technologies. J Technol Transf 35, 253–281 (2010). https://doi.org/10.1007/s10961-009-9122-6
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10961-009-9122-6