Technology emergence has become a hot topic in R&D policy and management communities. Various methods of measuring technology emergence have been developed. However, there is little literature discussing how to evaluate the results identified by different methods. This research sharpens a promising Technology Emergence Indicator (TEI) set by assessing alternative formulations on three distinct datasets: Dye-Sensitized Solar Cells, Non-Linear Programming, and Nano-Enabled Drug Delivery. Our TEIs derive from a conceptual foundation including three attributes of emergence: persistence, community, and growth that we systematically address through a 3-dimensional evaluation framework. Comparing TEI behavior through sensitivity analyses shows good robustness for the measures. The TEI serve to distinguish emerging R&D topics in the field under study. They can further be used to identify highly active players publishing on those topics. Importantly, results show that identified emerging terms and topics persist to a strong degree; thus, they serve to predict highly active R&D foci within the technical domain under study.
This is a preview of subscription content,to check access.
Access this article
Similar content being viewed by others
The Emergence script provided in VantagePoint enables one to vary many parameters of EScoring. To alter the 1.77 threshold (chosen based on empirical comparisons—see Porter et al. 2018), one just needs to scan the list of terms ordered by EScore in VantagePoint or MS Excel.
We also tried calculations based on the later two-year test period and last 2 years of the validation period as the prior period to test the predictive utility. Results are similar.
As of January, 2020, we have adjusted the specialized scope procedure. We abandon the cross-dataset comparison as problematic in requiring a representative population or random sample outside the test dataset. We tighten the within-dataset filter to use IDF ≥ 1.
Arora, S. K., Porter, A. L., Youtie, J., & Shapira, P. (2013). Capturing new developments in an emerging technology: an updated search strategy for identifying nanotechnology research outputs. Scientometrics,95(1), 351–370.
Bettencourt, L., Kaiser, D., Kaur, J., Castillo-Chavez, C., & Wojick, D. (2008). Population modeling of the emergence and development of scientific fields. Scientometrics,75(3), 495–518.
Breitzman, A., & Thomas, P. (2015). The Emerging Clusters Model: A tool for identifying emerging technologies across multiple patent systems. Research Policy,44(1), 195–205.
Burmaoglu, S., Porter, A. L., & Souminen, A. What is technology emergence? A micro level definition for improving tech mining practice. In Portland International Conference on Management of Engineering and Technology (PICMET), Honolulu, 2018.
Burmaoglu, S., Sartenaer, O., & Porter, A. (2019a). Conceptual definition of technology emergence: A long journey from philosophy of science to science policy. Technology in Society. https://doi.org/10.1016/j.techsoc.2019.04.002.
Burmaoglu, S., Sartenaer, O., Porter, A., & Li, M. (2019b). Analysing the theoretical roots of technology emergence: an evolutionary perspective. Scientometrics,119(1), 97–118.
Carley, S. F., Newman, N. C., Porter, A. L., & Garner, J. G. (2017). A measure of staying power: Is the persistence of emergent concepts more significantly influenced by technical domain or scale? Scientometrics,111(3), 2077–2087.
Carley, S. F., Newman, N. C., Porter, A. L., & Garner, J. G. (2018). An indicator of technical emergence. Scientometrics,115(1), 35–49.
Carley, S. F., Porter, A. L., & Youtie, J. L. (2019). A multi-match approach to the author uncertainty problem. Journal of Data and Information Science,4(2), 1–18.
Cheng, A. C., Chen, C. J., & Chen, C. Y. (2008). A fuzzy multiple criteria comparison of technology forecasting methods for predicting the new materials development. Technological Forecasting and Social Change,75(1), 131–141. https://doi.org/10.1016/j.techfore.2006.08.002.
Coates, V., Farooque, M., Klavans, R., Lapid, K., Linstone, H. A., Pistorius, C., et al. (2001). On the future of technological forecasting. Technological Forecasting and Social Change,67(1), 1–17.
Corning, P. A. (2002). The re-emergence of “emergence”: A venerable concept in search of a theory. Complexity,7(6), 18–30.
Corrocher, N., Malerba, F., & Montobbio, F. (2003). The emergence of new technologies in the ICT field: Main actors, geographical distribution and knowledge sources. Varese: Department of Economics, University of Insubria.
Cozzens, S., Gatchair, S., Kang, J., Kim, K. S., Lee, H. J., Ordóñez, G., et al. (2010). Emerging technologies: Quantitative identification and measurement. Technology Analysis & Strategic Management,22(3), 361–376.
Crutchfield, J. P. (2013). Is anything ever new? Considering emergence. In M. A. Bedau & P. Humphreys (Eds.), Emergence: Contemporary readings in philosophy and science. Cambridge: MIT Press Scholarship Online, The MIT Press.
Daim, T. U., Rueda, G., Martin, H., & Gerdsri, P. (2006). Forecasting emerging technologies: Use of bibliometrics and patent analysis. Technological Forecasting and Social Change,73(8), 981–1012.
Esmaelian, M., Tavana, M., Di Caprio, D., & Ansari, R. (2017). A multiple correspondence analysis model for evaluating technology foresight methods. Technological Forecasting and Social Change,125, 188–205. https://doi.org/10.1016/j.techfore.2017.07.022.
Foster, J., & Metcalfe, J. S. (2012). Economic emergence: An evolutionary economic perspective. Journal of Economic Behavior & Organization,82(2–3), 420–432.
Goldspink, C., & Kay, R. (2010). Emergence in organizations: The reflexive turn. Emergence: Complexity and Organization,12(3), 47–63.
Guo, Y., Xu, C., Huang, L., & Porter, A. (2012). Empirically informing a technology delivery system model for an emerging technology: Illustrated for dye-sensitized solar cells. R&D Management,42(2), 133–149.
Guston, D. H., & Sarewitz, D. (2002). Real-time technology assessment. Technology in Society,24(1–2), 93–109.
Halaweh, M. (2013). Emerging technology: What is it. Journal of Technology Management and Innovation,8(3), 108–115.
Harper, D. A., & Endres, A. M. (2012). The anatomy of emergence, with a focus upon capital formation. Journal of Economic Behavior & Organization,82(2–3), 352–367.
Huang, Y., Ma, J., Porter, A. L., Kwon, S., & Zhu, D. (2015). Analyzing collaboration networks and developmental patterns of nano-enhanced drug delivery (NEDD) for brain cancer. Beilstein Journal of Nanotechnology,6(Special issue on Nanoinformatics), 1666–1676.
Jun, S., & Lee, S. J. (2012). Emerging technology forecasting using new patent information analysis. International Journal of Software Engineering and Its Applications,6(3), 107–116.
Kajikawa, Y., Yoshikawa, J., Takeda, Y., & Matsushima, K. (2008). Tracking emerging technologies in energy research: Toward a roadmap for sustainable energy. Technological Forecasting and Social Change,75(6), 771–782.
Kim, D. H., Lee, H., & Kwak, J. (2017). Standards as a driving force that influences emerging technological trajectories in the converging world of the Internet and things: An investigation of the M2M/IoT patent network. Research Policy,46(7), 1234–1254. https://doi.org/10.1016/j.respol.2017.05.008.
Li, M., Porter, A. L., & Suominen, A. (2018). Insights into relationships between disruptive technology/innovation and emerging technology: A bibliometric perspective. Technological Forecasting and Social Change,129, 285–296.
Martin, B. R. (1995). Foresight in science and technology. Technology Analysis & Strategic Management,7(2), 139–168.
Martin, R., & Sunley, P. (2012). Forms of emergence and the evolution of economic landscapes. Journal of Economic Behavior & Organization,82(2–3), 338–351. https://doi.org/10.1016/j.jebo.2011.08.005.
Metcalfe, B. (1995). Metcalfe’s law: A network becomes more valuable as it reaches more users. Infoworld,17(40), 53.
Porter, A. L. (2010). Technology foresight: types and methods. International Journal of Foresight and Innovation Policy,6(1–3), 36–45.
Porter, A. L., & Cunningham, S. W. (2005). Tech mining: Exploiting new technologies for competitive advantage. New York: Wiley. (Chinese edition, Tsinghua University Press, 2012).
Porter, A. L., Garner, J., Carley, S. F., & Newman, N. C. (2018). Emergence scoring to identify frontier R&D topics and key players. Technological Forecasting and Social Change,146, 628–643. https://doi.org/10.1016/j.techfore.2018.04.016.
Porter, A. L., Roessner, J. D., Jin, X. Y., & Newman, N. C. (2002). Measuring national ‘emerging technology’ capabilities. Science and Public Policy,29(3), 189–200.
Porter, A. L., Rossini, F. A., Carpenter, S. R., & Roper, A. T. (1980). A guidebook for technology assessment and impact analysis. New York: North Holland.
Rader, M., & Porter, A. (2008). Fitting future-oriented technology analysis methods to study types. In C. Cagnin, M. Keenan, R. Johnston, F. Scapolo, & R. Barre (Eds.), future-oriented technology analysis (pp. 25–40). Berlin: Springer.
Roper, A. T., Cunningham, S. W., Porter, A. L., Mason, T. W., Rossini, F. A., & Banks, J. (2011). Forecasting and management of technology (2d ed.). New York: Wiley.
Rotolo, D., Hicks, D., & Martin, B. R. (2015). What is an emerging technology? Research Policy,44(10), 1827–1843.
Saritas, O., & Burmaoglu, S. (2015). The evolution of the use of Foresight methods: A scientometric analysis of global FTA research output. Scientometrics,105(1), 497–508. https://doi.org/10.1007/s11192-015-1671-x.
Sawyer, R. K. (2001). Emergence in sociology: Contemporary philosophy of mind and some implications for sociological theory. American Journal of Sociology,107(3), 551–585.
Small, H. (1999). Visualizing science by citation mapping. Journal of the American Society for Information Science,50(9), 799–813.
Small, H., Boyack, K. W., & Klavans, R. (2014). Identifying emerging topics in science and technology. Research Policy,43(8), 1450–1467.
Sohn, S. Y., & Ahn, B. J. (2003). Multigeneration diffusion model for economic assessment of new technology. Technological Forecasting and Social Change,70(3), 251–264. https://doi.org/10.1016/S0040-1625(02)00200-7.
Srinivasan, R. (2008). Sources, characteristics and effects of emerging technologies: Research opportunities in innovation. Industrial Marketing Management,37(6), 633–640.
Van Merkerk, R. O., & Smits, R. E. (2008). Tailoring CTA for emerging technologies. Technological Forecasting and Social Change,75(3), 312–333.
Wang, Q. (2018). A bibliometric model for identifying emerging research topics. Journal of the Association for Information Science and Technology,69(2), 290–304.
Wang, Z., Porter, A. L., Kwon, S., Youtie, J., Shapira, P., Carley, S. F., et al. (2019). Updating a search strategy to track emerging nanotechnologies. Journal of Nanoparticle Research. https://doi.org/10.1007/s11051-019-4627-x.
Zhou, X., Porter, A. L., Robinson, D. K., Shim, M. S., & Guo, Y. (2014). Nano-enabled drug delivery: A research profile. Nanomedicine: Nanotechnology, Biology and Medicine,10(5), e889–e896.
This work was undertaken with support from the US National Science Foundation (Award #1759960 – “Indicators of Technological Emergence”) to Search Technology, Inc. and Georgia Tech. The findings and observations contained in this work are those of the authors and do not necessarily reflect the views of the National Science Foundation. We wish to express our appreciation to Jan Youtie for sharing pearls of wisdom with us and to Seokbeom Kwon who provided the sensitivity tests of TEI.
Electronic supplementary material
Below is the link to the electronic supplementary material.
Extensive supplemental materials are available at: http://….. These materials provide the sensitivity test of TEIs in NLP dataset and DSSCs dataset, and the full lists of emerging terms identified by both the new method and the previous method for all datasets. (DOCX 359 kb)
Detailed steps for the term cleaning process are listed below.
Use VantagePoint’s natural language processing to extract noun phrases of abstracts and titles. [We have experimented with use of WOS Keywords-Plus and Keywords-Authors. We don’t use those here so as to make our evaluation processes more generalizable to other data resources lacking such fields.]
Merge the two fields (abstract noun phrases and title noun phrases) together and remove the terms with instances fewer than 2.
Apply Cluster Suite (a VantagePoint script available at www.VPInstitute.org, under Resources) to
eliminate single characters,
remove keywords beginning with non-alpha numeric characters,
remove XML tags,
consolidate chemical compounds and their abbreviations,
consolidate multiple keywords common to scientific and academic publications into one header.
Run a general fuzzy routine in VantagePoint to consolidate name variations in the list.
Divide the terms into unigrams and multigrams:
For unigrams, run a WOS stopwords list to remove common academic words.
For multigrams, use VantagePoint’s Folding NLP macro to consolidate the phrases.
Combine the unigrams and multigrams, and finally get the consolidated terms field as input to identify emerging terms and calculate their emergence scores.
Here is our process to generate annual, random sample datasets for WOS:
Split the total records into several search queries with under 100,000 records each. In recent years, there have been more than 2 million records in the WOS database for each year. Since the WOS could show no more than 100,000 records in one search query, we needed to split the records into several search queries. We first split the data by the capital letters of organizations. If the number of records for one capital letter was also more than 100,000, then we would separate the search by the Web of Science Categories (WCs). This way, we could split the total records for each year into 39 search queries, which constitute the whole record set.
Calculate the number of records we should download for each query. The data were split into 39 search queries, and the number of selections for each query was calculated according to the share of the number of records for each query out of the total number of records. For example, there are 90,079 records that are assigned to organizations beginning with the letter “k”, making 3.52% of the total 2,559,592 records in 2014. Thus, we would download 3.52% * 5000 = 176 records for this query.
Generate a specific amount of random numbers and download the corresponding records. For example, for the search query “OG = K* AND PY = 2014”, we generate 176 random numbers between 1 and 90079, and download the relative records.
About this article
Cite this article
Liu, X., Porter, A.L. A 3-dimensional analysis for evaluating technology emergence indicators. Scientometrics 124, 27–55 (2020). https://doi.org/10.1007/s11192-020-03432-6
- Technology emergence indicators
- Technology forecasting
- Emerging technologies
- Technology emergence assessment
- R&D emergence
- Predictive indicators