1 Introduction

Organizational agility is increasingly seen by researchers as a source of competitive advantage as firms alter their capabilities and business models to better respond to market opportunities and threats (Roberts and Grover 2012; Teece et al. 2016). Agility is essentially an outcome of enacting effective strategic change at the right time to achieve beneficial performance outcomes. According to the dynamic capability literature, agility involves the continuing ability to sense new market conditions, adapt or seize upon opportunities, and alter strategy (Teece 2007; Teece et al. 2016).

While both large and small firms can be agile, small firms are often better able to quickly respond to market needs and changing environments than their larger counterparts (Arora and Gambardella 1994; Rothwell 1989). They have fewer well-defined routines than large firms, and entrepreneurial leadership may alter the course of a firm’s direction more readily than in firms with complex bureaucracies (Covin and Slevin 1988). While the contribution of small firms to economic growth is sometimes overpromoted (c.f., Atkinson and Lind 2018), small firms are often considered catalysts of disruptive innovation (Kassicieh et al. 2002). Investigating relationships between agility and innovation outcomes in small firms is thus valuable from both management and policy perspectives. Yet studying small firm agility and change is particularly difficult for a number of reasons, including low survey participation rates and panel attrition, lack of representation in databases, and other limitations in traditional social science research designs involving primary data collection (Bartholomew and Smith 2006; Baruch 1999; Laaksonen and Peltoniemi 2016; March and Sutton 1997).

Our study develops a non-traditional approach to measuring the evolution of firm capabilities and agility by using firm archived website data from the Wayback Machine. This work relies on publicly-posted web information that is not only essential for many firms’ marketing and sales efforts but also may reflect important strategic content, particularly for small and medium sized enterprises (SMEs) (Li et al. 2016). We also contribute to the innovation literature by (a) using a computational, machine learning approach to quantify firm change over time, and (b) employing distinct, non-overlapping sources to measure dynamic capabilities and performance (c.f., Laaksonen and Peltoniemi 2016).

The case context for this work is the green goods manufacturing industry, which from a qualitative standpoint shows both underlying environmental dynamism and associated needs for changes in internal capabilities. The time period under study follows the 2008 economic crisis and US government incentives to encourage the manufacturing and adoption of sustainability related goods and services, indicating a high-velocity environment requiring market and policy learning as well as incremental strategic adjustments (c.f., Eisenhardt and Martin 2000).

In sum, we ask two related research questions: “Are green goods firms that sense and seize on market opportunities more likely to experience positive performance outcomes?” and “Can website data help us answer this first question?” While only addressing change over one-time period, we find that there is an inverse U-shaped relationship between seizing, operationalized as topical website change, and firm performance. Conversely, changes in sensing, as measured through R&D keyword mentions, are not associated with sales growth. Our exploratory study has implications for the way in which future research on small firm growth and technology commercialization could be undertaken to address market dynamism and firms’ propensity to respond with increasing agility. Although more work is needed to refine the method, we contend that measuring firm change through websites is an appropriate empirical approach for studying firm agility in the dynamic capability framework.

The paper is organized as follows: first, we position our review of the literature within the dynamic capabilities framework. Second, we introduce the case study context in more detail. Third, we submit an extensive overview of the research design to introduce the reader to our website measures and supporting methods. Fourth, we present the findings of our exploratory econometric analysis, which uses an instrumental variable approach to account for endogeneity of website topical change in the sales growth model. Finally, the discussion and conclusion summarizes the results and assess implications for research, policy, and practice.

2 Literature review

Organizational agility and performance can be explained by the dynamic capabilities framework, i.e., sensing, seizing, and managing threats (Roberts and Grover 2012; Teece et al. 2016). Sensing, according to Teece (2007), is the process of ascertaining environmental changes—for example, in science and technology, customer, and supplier landscapes—and directing R&D efforts accordingly. Sensing is related to exploration, which involves the outward examination of new information, applications of existing know-how, and dynamism of markets and science and technology (March 1991; Teece 2017). Seizing, conversely, includes business model definition and operationalization, securing complementary assets, implementing routines for decision making, and instituting leadership, communication, and cultural supports (Teece 2007). Seizing is related to exploitation, which refers to the operational aspects of following through on innovations and realizing efficiencies (Lavie et al. 2010; March 1991).

Research on ambidexterity acknowledges the motivations for, opportunities associated with, and challenges related to engaging in both exploration and exploitation (Raisch and Birkinshaw 2008; Rothaermel and Alexandre 2009; Tushman et al. 2010). A broad body of research shows ambidexterity results in positive performance outcomes, including sales growth, particularly in environments characterized by high market and technological uncertainty (O’Reilly and Tushman 2013). For example, using a longitudinal sample of 605 public technology firms, Caspin-Wagner et al. (2012) find an inverse-U shaped relationships between exploration, as measured by a firm R&D expenditures, and performance. They also find a similar relationship between exploitation activities and performance, suggesting that optimal levels of exploration and exploitation exist simultaneously.

Sensing and seizing, as put forth here, are examples of “first-order” dynamic capabilities (Winter 2003), which allow firms to reallocate resources and pursue market opportunities that may not exist in their current operational purview. First-order capabilities can be contrasted with ordinary capabilities, which enable operations and recurring (present) revenue streams, and second-order dynamic capabilities, which help firms “learn how to learn” (Schilke 2014b).

For a new small firm, dominant or ordinary routines are not likely to be well formed, but this is precisely why smaller firms are often perceived as being more nimble than their large firm counterparts (Arora and Gambardella 1994; Rothwell 1989). While all firms may become more steadfast in response to competitive threats (Killaly 1998; Toh and Kim 2013), young small firms may be more likely to benefit from a proactive approach to growth than from competitive aggressiveness, compared to larger, well-established firms (Lumpkin and Dess 2001), suggesting that they may be more likely to change when encountering environmental uncertainty. In the absence of higher order routines that can alter an SME’s lower-level capabilities, entrepreneurial managers may instead redirect resources and pursue strategic shifts directly (Teece 2012). Kickul and Gundry (2002) find that prospector (i.e., scanning) strategies mediate the relationship between proactive entrepreneur personalities and desirable innovation outcomes.

Ambidexterity in small firms appears more often sequentially rather than simultaneously due to limited resources and a lack of organizational slack (O’Reilly and Tushman 2013). That is, small firms are unlikely to sense and seize at the same time. In a related study, for example, Ebben and Johnson (2005) study efficiency and flexibility strategies and find that small firms that mix both strategies perform less well than those that focus primarily on one approach or another. If ambidexterity is likely to appear sequentially, what might cause a small firm to sense and seize insufficiently—or too much? We consider several possible explanations.

First, entrepreneurs often face resource constraints that limit their ability to act on new information and fully exploit new opportunities (Elston and Audretsch 2007; Newman 2010). Consequently, entrepreneurs restrict the types of goals and growth opportunities they seek based on available resources (Sarasvathy 2001). In some cases, entrepreneurs may not have the means to sense widely, though in other cases, these constraints may catalyze unconventional modes of experimentation, setting the foundation for radical innovations (Keupp and Gassmann 2013).

Second, in the mind of the entrepreneur, ingrained schemas, or ways of organizing and classifying observations about the world (Thagard 2005), may limit the breadth and depth of perceptions, a phenomena likely exacerbated by entrepreneurs who fail to respond appropriately to key developments or who do not wish to grow in the first place (Solymossy 2009; Wiklund et al. 2003). In this case, certain phenomena of critical importance for the firm’s operations and long-term survival may systemically “fail to register”, and as a result the entrepreneur has fewer alternatives to consider (March 1994; Simon 1997).

Too much seizing, on the other hand, may suggest deviations from firm-specific capabilities, i.e., that the firm is spreading itself too thin in the short-run in the pursuit of positive growth outcomes. Some scholars acknowledge that an unfettered propensity to change may result in declining performance outcomes (Schilke 2014b), and this is a particularly salient consideration for small firms which have limited resource to begin with. In addition, as argued by Spender (1989), industries often share a common set of knowledge, norms, and ways of doing things—or an “industry recipe”. The recipe is abstract enough to be translated from one firm to another in the same industry, yet detailed enough to offer opportunities for vicarious learning.

Recent conceptual work by Teece et al. (2016) argues that agility is so in vogue as a management philosophy (e.g., by way of pivoting) that organizations operating in low to moderate uncertainty environments may be likely to over-emphasize the importance of agility and thereby stress resources that could otherwise be used for efficiency building and risk-reduction purposes. Even within highly competitive environments, Yuan et al. (2010) find that flexibility in the coordination of resources is a more important moderator than the development of new resources alone in the (positive) relationship between product innovation and firm performance.

One way to engage in proactive sensing activities is through R&D investments. Cohen and Levinthal (1990) remark that absorptive capacity helps a firm gauge the merit of information and knowledge that it may in turn exploit. Research outputs and intellectual property may be secured from alliance partners, accessed via informal contacts, or developed in-house, with the intention of informing commercialization efforts. By investing in R&D, the small firm is not only able to stay abreast of exogenous scientific and technological changes but also promote the development of important inventions (Rosenberg 1990). Higher levels of R&D have been associated with small firms’ abilities to invest early in the product development lifecycle, which is itself a type of dynamic capability (Stam and Wennberg 2009). Saemundsson and Candi (2017) suggest that identifying opportunities in new high-technology firms depends more on the adaptation of existing technical knowledge to known problem areas than on the identification of new customer wants and needs.

In sum, as we attempt to assess components of agility, we posit that greater activity in sensing through investment and engagement in R&D is likely to lead to improved small firm performance. For seizing, we advance a non-linear expectation. Too much seizing could be one way of deviating from common practices in an industry; that is, whenever firms look to each other to observe the competitive landscape, drastic change is more the exception than the rule. Thus, not enough change or too much change suggests an under- or over-extending of firm resources that negatively impacts firm performance.

3 Case study context: the green goods manufacturing industry

We use the green goods manufacturing industry as a case to study small firm agility because of (1) contemporaneous and rapid changes in policy and incentives (e.g., subsidies and other public investments) impacting the industry in the time period under investigation, i.e., 2008–2011; (2) the underlying dynamism of the technology and market; and (3) the role of local exploration and exploitation processes for small green goods firms.

Green goods firms can be defined as firms that produce manufactured items with environmental or natural resource benefits when used by other businesses, organizations or households (Shapira et al. 2013). Small firms in green goods manufacturing represent an important case, where the survival of small firms is closely tied to their abilities to adapt to rapid technological and market changes as well as government policy shifts (Patzelt and Shepherd 2011). In the policy domain, US energy policy during the George W. Bush administration saw an emphasis on hydrogen fuel and energy security and efficiency through the Energy Independence and Security Act of 2007. The Obama administration continued to support green goods development through a range of policies including R&D investments from the American Recovery and Reinvestment Act (ARRA) of 2009, of which 12% was allocated to the energy area under Title IV of ARRA to build infrastructure and conduct R&D in energy efficiency and renewables. In addition, the Obama administration oversaw the startup of the Advanced Research Projects Agency-Energy (ARPA-E), Energy Frontier Research Centers, Energy Innovation Hubs, and greenhouse gas emission reporting. A range of renewable and nuclear energy options were promoted, although issues such as the Solyndra bankruptcy filing and the Fukushima Daiichi nuclear accident were highlighted in public discussions. In recent years, targets of interest for investment in clean-tech have shifted from biomass to solar to wind to smart grid to water and waste management (PriceWaterhouseCoopers 2014). Since 2017, notwithstanding the Trump administration’s efforts to backtrack from renewable energy and to revitalize coal production, US investment in renewable energy, clean transportation, and other green sectors has continued to increase (Bloomberg New Energy Finance 2019). These fluctuations form the backdrop highlighting the necessity of sensing and seizing by green goods firms if they wish not only to survive but also to grow.

The geographic focus of a firm’s marketing and operations may play an important part in the influence of agility on growth. On one hand, after the economic downturn in 2008, the US government (as well as other governments) expressed interest in the growth of the green goods industry as a means to create new jobs in the economy. Because of the regulated nature of the broader energy sector, and due to the policy agenda of the US government to increase sustainable and green jobs, firms with increases in a local or domestic focus may be more likely to grow. That is, strong connections to local partners, suppliers, customers, and other stakeholders, who encourage information sharing, opportunity seeking, and geographic-based embeddedness, may positively enhance sensing and seizing activities and contribute to better performance (Becheikh et al. 2006; MacKinnon et al. 2002; Moulaert and Sekia 2003; Simmie 2005). On the other hand, the Brookings Institute found that most of these (green) jobs are in mature industries in manufacturing and public services rather than in new industries that could propel economic growth (Muro et al. 2011), and thus an increase in local ties would have limited effect on performance because presumably those local ties already existed and were being exploited to a sufficient extent.

4 Data and methods

Measuring deviations from a firm’s existing course in real-time is challenging, though the need to do so is especially critical in high-velocity markets, where the fast-changing environment requires adaptation (Eisenhardt and Martin 2000). Feedback provided from managers and other respondents via surveys and interviews can be expensive and suffer from low response rates and panel attrition (Bartholomew and Smith 2006; Baruch 1999). Additionally, manager responses about dynamic capabilities and firm performance cannot be evaluated independently of one another because perceptions of performance and actions tend to reinforce each other (Laaksonen and Peltoniemi 2016). Such recall and response bias problems introduce measurement error when key explanatory variables are not observed directly and respondents selectively rationalize behaviors and decisions to conform to known performance outcomes and “conventional storylines” (March and Sutton 1997, p. 702). Finally, the phenomena under study—i.e., how firms position themselves for competitive advantage and what that looks like at any given time—often moves faster than traditional research designs involving primary data collections can track.

Computational social science approaches, including the use of website data and other forms of new media, present new opportunities for measuring firm change and attempts at shaping and responding to the environment. Firms use the web to give away or pilot digital products to gauge market demand; they develop online communities to promote a brand; and they create websites to share information and collect feedback (Choo 2006). Ashurst et al. (2012) show that ICT platforms are increasingly used to solicit feedback from stakeholders both within and external to the organization, while Fischer and Reuber (2011) find that social media furthers entrepreneurial sense-making processes because of its ability to facilitate a wide range of social interactions.

Websites, in particular, offer a view of a firm’s seizing capabilities because websites may show changes in market orientation, product portfolio, and alliance networks (Arora et al. 2013, 2015; Li et al. 2016). Websites as information and marketing vehicles, may also show how small firms sense the market and portray their internal capabilities. For example, Gök et al. (2015) examine the use of research and development related terms on firm websites and determine that derivative website variables, while subject to some measurement error, appear to convey R&D activities not captured in traditional sources (e.g., databases on firm R&D expenditure and grants and patenting and publishing outputs). Ebben and Johnson (2005) use website data to code flexibility and efficiency strategies for small Finnish firms, and when comparing these measures to survey-derived responses, they find convergent validity between measures derived from the two sources.

Using website data, however, is not without limitations, which include sampling biases and endogeneity between website change (as a proxy for evolution in firm capabilities) and growth. These limitations inform the research design and are further discussed in the concluding section of this paper.

4.1 Sampling strategy

The empirical analysis in this paper focuses on US small and medium-sized enterprises (SMEs) that entered the green goods sector with a year of establishment in the 2003–2007 time period. Concerning the definition of green goods, the formal US Bureau of Labor Statistical definitions of green industries were judged to be too broad, and green patents were judged to be too narrow. Instead, we used a multi-phase approach to identify green goods sector companies. The first phase began with a keyword based approach that was applied to the “line of business” field in the Dun & Bradstreet (D&B) Million Dollar database. [For a discussion on the use of D&B Million Dollar database in small firm research, see Carlton (1983), Brown and Medoff (2003), Neumark et al. (2011) and Reynolds (2017)]. More than 100 keywords were used to identify relevant companies in three green goods sub-segments: environmental (e.g., biological treatment, air pollution, environmental monitoring, land remediation, waste management, water treatment, recovery and recycling), renewable energy (e.g., wave and tidal, biomass, wind, geothermal, photovoltaic/solar), and emerging low carbon (e.g., alternative fuel vehicles, alternative fuels, electrochemical processes, batteries, carbon capture and storage, building technologies) (Shapira et al. 2013).

We applied additional filters for manufacturers with US headquarters, year of establishment in the 2003–2007 timeframe, and with 250 or fewer employees. Despite the year-of-establishment criterion, some older firms which re-incorporated between 2003 and 2007 appeared in the results. On further inspection of some of these re-incorporations, the re-incorporated firm exhibited a new orientation towards green goods. In total, this process yielded a frame of 2505 US enterprises. Manufacturers comprised 80% of these enterprises, but enterprises outside out of the primary North American Industry Classification System (NAICS) codes for manufacturing were considered because some of their offerings suggested green goods production.

Further work was undertaken to verify whether enterprises within this broad frame were valid green goods manufacturing firms. This second phase of population refinement involved the application of a four-point coding scheme by two different coders and reviewed by a senior researcher to each of these enterprises based on the text in the Dun and Bradstreet “line of business” field to determine the enterprise’s relevance to green manufacturing. Enterprises not involved in manufacturing or lacking a green product offering were removed from the list. This coding scheme reduced the number of enterprises to 62% of the original list. This coding scheme disproportionately affected manufacturers with primary NAICS classifications in the wood products, printing and furniture industries; these enterprises were more apt to be considered less relevant and removed from the list because although they produced resource-based goods, the goods were not relevant to green goods manufacturing. In addition, a group of nonmanufacturing enterprises in the agriculture, mining, and construction industries were found to not manufacture goods and were removed from the list.

In a third phase, each of these enterprises was examined to determine whether or not it had a website. This criterion reduced the number of enterprises to 700. The addition of a website provided for more information about the enterprises, so another round of green goods relevance scoring was performed. Again, enterprises found not to be doing manufacturing or not having a green product offering were taken out of the list. In addition, those websites comprised of non-HTML pages (e.g., developed in Adobe Flash) were not included in the sample as our analytical software requires textual input. As a result, the sample was reduced to 298.

The proportions of enterprises across the primary NAICS codes were slightly different between the first phase and the second and third phases. As previously indicated, the second and third phases had proportionately fewer enterprises in the wood products/printing/furniture industries and in agriculture/mining/construction (Appendix Table 5). Likewise, the final phase of relevance coding emphasized more durables industries. Nevertheless, the total percentage of enterprises in manufacturing industry codes rose only slightly between the first phase list and the final phase: from 79% in the first phase/list to 86% in the final phase/list.

Next, firms with missing website data, e.g., such that a seizing (i.e., firm change) score could not be computed, were removed from the dataset. A total of 223 observations were available for analysis. Of these 223 firms, 78 were in the emerging low carbon subarea, 124 were in the environmental area, with the remaining 21 being classified as renewables. A weighting approach described below was used to account for over- and under-representation of certain firms in the 2SLS model.

As a final step to examine the representativeness of our sample, we compared the employment growth rate of our final sample of 223 firms with the population of private green goods firms, as defined by the US Bureau of Labor and Statistics (BLS). The rate of growth of private US green goods and services employment was 4.9% between 2010 and 2011. (Manufacturing employment growth during this same period was 2.9%.) Our sample shows an employment growth rate of 12.0%. Due to the smaller database that is used in this analysis compared to the BLS benchmarks, the mean employment growth for this database is skewed by a few outliers such as Tesla Motors, Inc (which started during our study period). The median employment growth rate for individual firms in our database is zero, and 73.5% of the firms have zero employment growth, with another 10.3% having negative employment growth over this period. Overall, the majority of the sample firms are at or below national green goods employment growth benchmarks.

4.2 Dependent variable

The dependent variable, derived from D&B, is logged sales growth. We computed growth, a continuous interval variable, by subtracting logged revenues in 2012 from logged revenues in 2010. Because of data quality and availability issues, in 30 select cases, sales growth was computed by subtracting 2014 revenue figures from 2012 revenue. Fifteen firms lacked sufficient sales data from D&B (in two distinct time periods) to determine sales growth and thus were dropped from the dataset. Four of these fifteen firms were acquired.

4.3 Website method and variables

We gathered data from company websites using a web crawling method adapted for social sciences (Arora et al. 2015). Websites have advantages (including accessibility, timeliness, and scope of data) and limitations (as unstructured company self-reports) for understanding innovation behavior (see consideration of limitations towards the end of this paper, and for further discussion of website mining advantages and limitations, see Gök et al. 2015). Company websites for each firm were extracted from 2008 to 2011 using the Wayback Machine. The Internet Archive, available at http://archive.org, is a repository of historical web pages saved periodically over time. By 2016 over its 20 year history, the Wayback machine held 273 billion webpages consisting of 361 petabytes (Goel 2016).

Leetaru (2015) documents two important considerations for researchers to be aware of in their use of the Wayback Machine. First, the Wayback machine adheres to “robots.txt” directives, which signal crawlers to exclude sites from automated web crawling. Consequently, firms with “robots.txt” exclusion directives are missing from the archive. Second, the Wayback machine is a decentralized architecture of crawlers, and there is no single policy as to how often or how comprehensively websites are crawled. Leetaru quotes Mark Graham, Director of the Wayback Machine, who comments, “I would expect any researcher would be remiss to not take the fluid nature of the web, and the crawls of the [Internet Archive], into consideration.”

Despite its limitations, the Wayback Machine has been used in other research contexts and, despite issues of lack of coverage of some websites in some years, has been found to produce information with content validity over time (Murphy et al. 2007). To ensure sufficient coverage and capture any updates, we recrawled (i.e., scraped again) any website with what appeared to be erroneous data, and as a result, many of these company websites in the Wayback Machine were completely crawled in 2012 and again in 2014. We also account for bias of missing data in our analysis, as described further below.

IBM Content Analytics (ICA)Footnote 1 was employed as a software tool for web extraction because of its ability to efficiently scrape websites, to allow the user to define keyword definition sets via parsers, and to deploy such parsers to collections of documents. After scraping was conducted, and parsing rules had been deployed on the dataset, ICA delivered data regarding the amount of times the words within the model appeared. XML files of these data were exported from IBM Content Analytics to a proprietary Java-based program, and the numbers of terms were summarized on a per-variable basis. Keyword-derived website variables include R&D difference and local difference. R&D difference captures the degree of research and development intensity found on firm websites in 2008–2009, while local difference reflects the change in local geographic focus between 2008–2009 and 2010–2011, relative to the firm’s headquarters location.

Each variable was normalized based on the number of pages collected during a given year (with duplicates excluded). Additionally, because Wayback coverage of firm websites can be sparse from year-to-year, despite our care in recrawling websites, we grouped these variables into two time periods, 2008–2009 and 2010–2011, by summing each normalized-year variable in the period and dividing by two.

The validity of these keyword-based variables is studied elsewhere. For example, the R&D keywords include “lab” OR “laboratory” OR “research” OR “development” OR “R&D” OR “researcher” OR “scientist”. Ongoing work suggests that this set of keywords is statistically albeit weakly correlated with patenting intensity using a non-parametric test of significance (tau = 0.16, z = 7.64) (Arora 2019). Prior work by Gök et al. (2015) finds that various instantiations of R&D keyword combinations deployed on website data, while highly correlated with one another due to the overlapping key terms, are not statistically correlated with other non-website related measures, such as total number of publications, R&D expenditures and number of grants. That said, not all SMEs patent and even fewer publish, so measures of correlation cannot tell a complete story pertaining to the value of R&D keyword-based variables.

For local difference, our method successfully captures different intensities of geographic foci for a given SME; we assure validity for this measure by manually assessing selected firm websites from the distribution of scores in the two time periods, and from the distribution of local difference scores across both periods. For instance, several firms that show higher levels of local geographical presence in the second time period opened local offices in their respective metropolitan statistical area. In contrast, one firm, Tesla Motors, Inc., retained a negative local difference score; the firm developed a strong national and international focus on its website in the second time period, even as it advertised two locations in California in the second time period, but only one in the first.

The key variables of interest in this study are proxies for strategic sensing (R&D difference, as noted above) and seizing (dpq, as explained here). We operationalize seizing through estimating the topical change in website content from one period (2008–2009) to the next (2010–2011). Topical change on websites is an appropriate proxy measure for firm seizing, we argue, because of two reasons: (1) the extent to which a firm changes its topical focus reflects adjustments in the way it organizes, accesses, and markets its capabilities and/or pursues customers (we provide an example of this after explaining our method), and (2) marketing efforts, including those put forth on websites, are unlikely to deviate too much from a firm’s actual capabilities to deliver to its prospective customer base, lest the firm suffer reputational harm. We review the endogenous relationship between topical change and firm growth below, and potential limitations of this approach in the discussion session.

We used latent Dirichlet allocation (LDA), a popular topic modeling algorithm, to identify latent topics in the website data. Topic modeling is based on probabilistic models of the generation of texts. Based on observed words in documents (in this case web pages), topic modeling uncovers a “hidden structure” to identify the probability distribution of words to K topics and the probability distribution of K topics to M documents in a corpus (Blei 2012). By estimating these distributions, it is possible to establish which words belong together to form topics, what mixtures of topics are found in documents, and which topics belong together in a broader corpus of firm-specific website data. The distribution of words and documents can be expressed as:

$$p\left( {{\text{w|}}\alpha ,\beta } \right) = \int {p\left( {\uptheta |\alpha } \right)} \left( {\mathop \prod \limits_{n = 1}^{N} \mathop \sum \limits_{{z_{n} }} p\left( {z_{n} |\uptheta} \right)p\left( {w_{n} |z_{n} ,\beta } \right)} \right)d\uptheta$$
(1)
$$p\left( {D |\propto ,\beta } \right) = \mathop \prod \limits_{d = 1}^{M} \int {p\left( {\uptheta_{d} |\alpha } \right)} \left( {\mathop \prod \limits_{n = 1}^{{N_{d} }} \mathop \sum \limits_{{z_{dn} }} p\left( {z_{dn} |\uptheta_{d} } \right)p\left( {w_{dn} |z_{dn} ,\beta } \right)} \right)d\uptheta_{d}$$
(2)

where w and D represent words and documents respectively, θ represents a mixture of topics zn, N the number of words in a document, M the number of documents in a corpus, and β is a matrix of probabilities indicating the probability that the jth word belongs in the ith topic. Finally α is a parameter defining the distribution of the mixture of topics. In LDA, the distribution \(p\left( {\uptheta |\alpha } \right)\) is chosen to be a Dirichlet distribution with shape given by the vector parameter α (Blei et al. 2003).

To prepare the website data for topical analysis, we followed a number of cleaning and preparation steps (Fig. 1). First, as with the keyword variables, duplicates were removed, along with any non-English textual content. We also removed punctuation characters, company names and company urls, the latter of which were found to bias the topics around company-specific content. At this point, LDA was trained on 2008–2009 website data to develop baseline probability distributions of (a) latent topics-to-web page documents and (b) observed words to web page documents. We employed K = 100 topics over 1000 iterations with smoothing parameters α = 50/K and β = 0.01. We refer the reader to Steyvers and Griffiths (2007), who discuss this parameter setting in producing robust LDA results.

Fig. 1
figure 1

Technology architecture for operationalizing the firm change variable

Full size image

Our overarching goal is not to isolate and present specific topical content; rather, we are interested in a broad measure of topical change over time as a proxy for seizing. For example, small and medium-sized firms report the breadth of their product portfolio, the geographic scope of their operations, and their approach to investor relations, etc. through their websites, and this content changes over time, thus conveying some degree of strategic change. We measured change by creating firm-level documents for the time periods 2008–2009 and 2010–2011, and then using the trained LDA model, we inferred the latent probability distribution of topics to those single firm-level documents. A firm-level document contains all the firm web-pages for the specific time period (excluding duplicates), aggregated in one record.

An asymmetric firm-level measure captures the extent to which the probability distribution of inferred topics in 2008–2009 differs from the inferred probability distribution of topics in 2010–2011 (Steyvers and Griffiths 2007). It is calculated as follows, where D measures the distance between the topical distribution vector p for a given firm i in 2008–2009 and the topical distribution q for the same firm in 2010–2011:

$$D\left( {p,q} \right) = \mathop \sum \limits_{i}^{K = 100} p_{i} * log_{2} \left( {\frac{{p_{i} }}{{q_{i} }}} \right)$$
(3)

Identifying topics is a generative (i.e., bottom-up) process based on observed words. Hence, there is no guarantee that topical order and underlying content will remain consistent between one run and the next. To safeguard against this potential bias in measurement error, we ran the training and inference process (bottom right portion of Fig. 1) 30 times and used the average of the information distance measure as the sample observation for firm sensing and seizing (Canini et al. 2009). While topical content and order will change from one run to another, we contend that this approach deemphasizes outlier observations by incorporating a greater number of distance estimates. The resulting variable representing the extent of firm change is signified by dpq. The squared value is denoted as dpq squared (distance between p and q squared) in the ensuing models.

To better illustrate the intuition behind our measure of firm change via website topical content, we present descriptive results of one LDA run. In this run, 99 of 100 topics have face validity in that a human can easily review the top most frequently occurring words belonging to a topic and offer a label or two to summarize that topic. We exemplify the labeling and its relationship with the underlying words through presenting four illustrative topics: “wastewater treatment”, “engineering systems”, “technology solutions”, and “toxicity and hazardous effects”, chosen to show the breadth of topics captured by LDA (Table 1).

Table 1 Sample topics (4/100) with top ten words trained on all firm website data, 2008–2009, from one run of LDA
Full size table

To exemplify how topic modeling and our information distance measures capture firm change, we describe the evolution of two firms’ websites with respect to the four sample topics in Table 1. Upon founding, one Colorado firm specializing in water treatment solutions began servicing a nearby industrial company in natural gas. As the Colorado firm grew, it expanded its product and service offerings to the food and beverage industry, which also has a need for water purification. As a result, the firm’s website as a whole reduced emphasis on industrial water treatment (topic 0) in favor of a broader portfolio of technical and service solutions (topics 9 and 49). The Colorado firm’s dpq score (3.071) for this LDA run was well above the sample average (1.303).

In contrast, a Pennsylvania firm offering fifteen non-toxic cleaners, degreasers, and washers had a dpq score (0.994) below the sample mean, indicating that the firm’s website content changed very little between the two time periods. However, the LDA inference process was still able to isolate small changes in topical content. For example, in 2008 this firm produced nine material safety data sheets for its fifteen products. In 2010, the company added material safety data sheets for all fifteen products. Because each data sheet stresses the non-toxic and non-hazardous profile of each product, the probability of the toxicity and hazard related topic (64) occurring increased in the second time period. The Pennsylvania firm did not seize any net-new business opportunity, and thus its dpq score was low, less than 1.0. Moreover, whereas the Colorado firm experienced sales growth between the first and second time periods, the Pennsylvania firm did not.

4.4 Instruments

Topical website change is likely endogenous in the sales growth model.Footnote 2 Profitable firms are not only more likely to have the capital on hand to explore and exploit new business opportunities (Fauchart and Keilbach 2009) but also may have more resources to invest in website development, which may introduce new topics in self-disclosed narratives. To address this endogeneity, we instrument on topical change (both dpq and dpqsq) using website variables that are correlated with website topical change over time but uncorrelated (or weakly correlated) with sales growth. These instruments include: (1) year site appears, or the year the site first appears on the Wayback Machine; (2) logged words difference, or the difference between the number of words observed in each time period, 2008–2009 and 2010–2011, logged; and (3) redesigns, the number of site redesigns since 2004. A site redesign is defined as a significant change in the color, imagery, and style presentation of a firm website from 1 year to the next. A site redesign may also incorporate functional enhancements, e.g., adding a substantive capability such as an e-commerce feature. Our review of website redesigns show primarily incremental, aesthetic changes in the final sample of websites (Appendix Table 6). This variable is coded manually.

The rationale behind these instruments is as follows: While firms may change the presentation of their website content over time (e.g., to look more contemporary), this should not significantly add to the long-term operating expense of running such sites; therefore, the number of site redesigns, controlling for the year the site appears, should not correlate with the structural error term. At the same time, a redesign likely introduces some new topics as the firm rebrands itself (e.g., as it seeks to find new markets for its existing capabilities and complimentary assets). We interact, redesigns with year site appears to capture the possibility that sites with longer histories could be more likely to experience redesigns, and together this interactive relationship could instrument for topical change. Similarly, adding new words to a website is a near costless activity; topics may be added or removed as a result, but such an endeavor is unlikely to correlate with the error term in such a way as to explain sales growth. Refer to the correlation table in the results section for confirmation that the instruments appear to pass face validity. Results of more robust statistical checks are also reported below.

4.5 Controls

We collect from D&B data on employee growth from 2010 to 2012 (logged employee growth). A second control variable captures environmental dynamism: the change in industry-level gross output in the second time period (i.e., from 2011 to 2010) (logged industry growth). Put another way, this variable controls for broader economic growth in these industries. To calculate this value, we map a firm’s primary North American Industry Classification System (NAICS) code, as obtained from D&B, to US Bureau of Economic Analysis yearly gross output reports. We include this variable because the literature on dynamic capabilities emphasizes differences in rate of industry growth with consequential implications for firm-level growth (Schilke 2014a). Finally, we include two dichotomous industry controls for environmental and renewable energy, with low carbon as the reference group.

4.6 Model identification

We use a 2SLS regression model to account for endogeneity between topical change and sales growth. Formally, we estimate the reduced form as follows:

$$\begin{aligned} \log \left( {sales\,growth} \right) & = B_{0} + B_{1} dpq + B_{2} dpq^{2} + B_{3} local\,difference \\ & \quad + \,B_{4} R\,\& \,D\,difference + B_{5} \,log\left( {industry\,growth} \right) \\ & \quad + \, B_{6} \,log\,\left( {employees\,growth} \right) \\ & \quad + \,B_{7} environmental + B_{8} {\text{renewable}}\,{\text{energy}} + u \\ \end{aligned}$$
(4)

where u is the error term. In the fully specified 2SLS model, the first stage regresses topical change (both dpq and dpqsq) on all exogenous variables, including the three instruments noted above. The second stage regresses logged sales difference on the estimated parameters of dpq and dpqsq, as well as R&D difference, local-diff, and the control variables.

5 Results

Descriptive statistics and pairwise correlations are shown in Table 2 for all 2SLS variables. The average number of employees in 2010 was 104.63, with the largest firm employing 3061 people. (Note that all firms were classified as being small in 2003–2007 but may have grown, sometimes by a substantial number in subsequent years.) The mean number of pages per firm site in 2008–2009 was 40.87 (min. = 1 and max = 830), while in 2010–2011, the mean was 47.39 (min. = 1 and max = 1467). The seizing proxy website variable, dpq is always positive, with some firms exhibiting large changes in website content and others showing relatively little variation. As shown in Table 2, there are no strong pairwise correlations variables outside of the relationship between dpq and dpqsq, although some correlations are significant at p < 0.05, e.g., as is the case with the positive relationship between logged employee growth and logged sales growth, as well as the positive relationships between the instruments, redesigns and logged word difference, and the topical change variables, dpq and dpq squared.

Table 2 Pairwise correlations and descriptive statistics, n = 223
Full size table

Before examining the results for substantive findings, we evaluated the merits of our instrumental variable approach using traditional tests of partial correlation, endogeneity and over-identification. For model 4 presented below, the first stage in a 2SLS model with multiple instruments can be used to assess Cov(z,x) ≠ 0. An F-statistic in the first stage of a 2SLS regression of greater than seven indicates strong instruments. In models A1 and A2 (Appendix Table 7), we reject the hypothesis H0: all instruments = 0 at the 95% confidence level, though the F-statistics (3.461 for dpq and 4.565 for dpqsq) suggest moderate, not strong, instrumentation.

The coefficient on redesigns in positive and highly significant for both models A1 and A2 with dpq and dqqsq as the dependent variables, respectively, suggesting that each additional redesign introduces some amount of topical change to a website’s content, holding all other variables constant. Contrary to our intuition, however, the coefficient on year site appears is negative but only marginally significant in A1. (This variable is not significant in A2.) The interaction term between year site appears and redesigns is highly significant in both models, and the coefficient is negative, implying that the effect of each additional redesign decreases with firms whose sites appear relatively later vis-à-vis firms whose sites are older. The effect of logged words difference is positive but insignificant in Model A1 and positive and highly significant in A2.

Tests of endogeneity reject H0: instruments are exogenous in the Durbin test (p = 0.012) and in the Wu–Hausman test (p = 0.014) at α = 0.05. Lastly, testing for overidentification shows large p values (p > 0.10) for the Sargan and Basmann tests; consequently, we cannot reject the null hypothesis that the instruments are valid and the model is correctly specified. Overall, we have sufficient evidence that dpq and dpq squared are in fact endogenous and that the instruments, although not as strong as we would like, serve appropriately for estimating logged sales growth.

We now turn to the substantive modeling results. 2SLS model specifications 1–4 in Table 3 reflect an increasing number of variables in their respective specifications. Model 5, as discussed below, introduces a weighting scheme as a robustness check on top of the model 4 specification.

Table 3 2SLS results on logged sales growth, n = 223
Full size table

In Model 1, dpq is significant and positive (p < 0.05) with a coefficient of 1.03, and dpq squared is marginally significant and negative (p < 0.10) with a coefficient of − 0.06, suggesting that topical change as an endogenous proxy to firm seizing is associated with the outcome variable, logged sales growth, in an inverse U-shaped fashion. In other words, a one-unit increase of topical change on websites is associated with a sales growth increase, but this increase declines with increasing levels of topical change.

Model 2 adopts the time varying control variables. Logged employee growth is highly significant at α = 0.01: a 10% increase in a firm’s number of employees results in a 2.1% increase in firm sales growth. Logged industry growth does not impact firm sales growth. In Model 2, the coefficient on dpq becomes significant (p < 0.01) and goes up to 1.13, while dpqsq remains marginally significant and negative. The whole model becomes significant at Wald χ2(4) = 16.48, p = 0.002.

Model 3 includes the R&D difference variable, but it is not significant. Model 3 also introduces the local difference variable, which shows a positive and significant coefficient (0.18, p < 0.05): A one-unit increase in the difference in local mentions by a firm vis-à-vis its headquarters is associated with an 18% increase in firm sales growth. Changes in localization intensity appears to capture some of the explanatory power of the firm seizing proxy variable (i.e., dpq); removing R&D difference from Model 3 does not impact the magnitude of coefficients dpq and dpqsq in Model 2. While these results are not conclusive, it appears that dpq and local difference measure some aspects of firm seizing. Model 3 as a whole remains highly significant.

Model 4 is the fully specified, unweighted 2SLS model, and it is significant at α = 0.01. All explanatory variables of interest are (marginally or better) significant in the same direction as mentioned above. For example, dpq and dpqsq are significant at p < 0.05, though local difference is only marginally significant at p < 0.10. In Fig. 2, we show a graphical depiction of the curvilinear relationship between dpq and dpq squared while holding all other variables in M4 at their means. This curve shows that most firms in our dataset are likely to experience increasing levels of firm sales growth with increasing amounts of website topical change until a dpq score of between 8 and 9; thereafter a select few with very high website change scores are predicted to experience lower relative sales growth. Removing firms with very high-levels of website change produces model results where the (endogenous) relationship between firm seizing and firm growth appears linear in nature.

Fig. 2
figure 2

Using dpq to predict logged sales growth while holding other variables at their means

Full size image

To account for observations with missing values, we introduce an inverse probability weighting scheme to estimate the likelihood of being included in the final sample of 223 firms (c.f., Solon et al. 2015). Recall that some of the 298 firms did not have sufficient website data over the two-time period span to be included in the sample, while others were missing other data derived from D&B. We include this analysis as a post hoc robustness check addressing potential sample bias and to illustrate the representativeness of website data more generally for small firm innovation studies. We estimate the log-odds of being in the final sample of 223 firms vis-à-vis the initial set of 298 firms as:

$$\begin{aligned} ln\frac{{{ \Pr }\left( {in\,sample} \right)}}{{1 - { \Pr }\left( {in\,sample} \right)}} & = B_{0} + B_{1} \log \left( {employees} \right) + B_{2} year\,firm \,founded \\ & \quad + \,B_{3} \,out\,of\,business\, + \, B_{4} northeast\,US \\ & \quad + \, B_{5} midwest\,US + B_{6} \,southern\,US \\ \end{aligned}$$
(5)

Logged employees is the number of employees in 2010; year founded is the year the firm was established; and out of business is coded 1 if the firm experienced a market exit in the time period under study, 0 otherwise. Finally, a firm may be found in any one of the four main US Census Bureau defined regions: Northeast, South, Midwest, and West. The reference group for region is West.

Results of the logit model (Eq. 5, Table 4) suggest that weighting is necessary to account for bias in the sample. The odds ratios suggest that firms that have gone out of business are less likely be to be represented (p < 0.01). Additionally, firms in the Midwest and South are much less likely to be in the final sample than firms in the West (p < 0.01). These findings suggest systemic differences in the likelihood of the existence of firm websites across the sample. Further, this result points to a clear need to better understand differences in firm website publishing behavior, a topic to which we return in the discussion below.

Table 4 Logit results for probability of being included in the final sample, a dichotomous variable
Full size table

Comparing Model 4 with Model 5 in Table 3, which includes probability weights of being included in the final sample, we see that the linear firm change measure of dpq remain positive and significant, but only marginally so. The coefficient on dpqsq is negative and no longer significant, while the coefficients for local difference and logged employee growth remain positive but turn marginally significant.

6 Discussion and conclusions

This paper has put forward a methodological approach for operationalizing the concept of firm agility and examining the relationship between agility and growth outcomes. We build a measure of strategic change through unstructured text mining of firm websites using topic modeling based on LDA. To construct this measure, we collect archived website data for US green goods small manufacturers from 2008 to 2011 using the Wayback Machine and using web scraping methods (Arora et al. 2015). With this relatively novel source of data, we overcome an important barrier in studying small firms: a dearth of data. Among the small green goods manufacturers in our sample, only 20% had any scientific publications or patents, preventing us from constructing any indicators based on this type of public record (Li et al. 2016). Business databases have limited, if not inconsistent, records for these small firms as well. On the contrary, as shown through this work, we are able to learn a great amount about firm activities and strategies from their websites.

The exploratory findings suggest that strategic change (or seizing), as measured by firm website topical change over time, has a curvilinear, inverse U-shaped relationship with sales growth. This is consistent with the expectation indicated following our literature review. We take into account the problem of endogeneity between topical change on firm websites and sales growth by instrumenting topical change with several variables characterizing the firm websites, which are, by definition, correlated with topical changes but not with sales growth. Additionally, increasing levels of local presence as emphasized on firm websites is associated positively with sales growth. In contrast, changes in sensing, as measured by differences in R&D levels between the first time period and the second, do not appear to have an impact on sales growth outcomes. Although, as discussed below, there are methodological limitations related to this finding, it could also be that a longer period of elapsed time (than measured in our study) is needed for firms to see business results from prior R&D activities.

Our use of website topical change as a proxy for firm change captures aspects of both ordinary and dynamic capabilities (Laaksonen and Peltoniemi 2016). The single time-period view of topics comprising a firm’s website conveys substantive, self-reported capabilities ranging from the firm’s product and service portfolio, and marketing approach (e.g., use of science- and technology-based language vis-à-vis verbiage focused on applications and industries), often grounded in a firm’s history and personnel. The longitudinal view of these firm websites and how they change over time may reveal higher-order dynamic capabilities at play, i.e., the extent to which firms are able to modify the substantive routines that may result in enhanced performance outcomes.

However, not all firms are willing and able to invest in dynamic capabilities. Winter (2003) argues that firms may strategize on an ad hoc basis if the cost–benefit analysis does not show a defensible reason for building and maintaining dynamic capabilities. In this sense, high topical change scores could be the result of either substantive changes as a result of seizing or ad hoc strategizing. In any case, larger scores indicate substantial deviations from the firm’s previous capabilities (or at the very least, the way the firm marketed itself), while smaller scores suggest less (or even a lack of) change.Footnote 3 Future work is needed to refine variables that the current topical change measures capture but may not sufficiently isolate. With our topical change measure, our approach has the advantage of allowing for inter-firm comparison across a variety of firm behaviors and activities that comprise what we put forth as strategic movements. Yet, this probabilistic model-based measurement conceals the details of the actual seizing activities. Whether the changes reflect product/process iterations, realignment of market orientations, or changes in product portfolios, these specific activities are not reflected in the final measures, and thus we cannot identify them in our regression models. Admittedly, this limitation has constrained our ability to link the results to a specific context and interpret its policy implications.

Even so, we contend that the need for multiple observations per firm over a longer period of time may enhance the study of strategic and organizational change, and we argue that website and other new media data are well-suited for this purpose. Because agility is an increasingly important driver of competitive advantage (Teece et al. 2016), and because of the aforementioned dearth of data on small innovative firms, researchers need more cost-effective, richer, and unobtrusive sources from which to study the evolving nature of innovation and decision-making. Findings from this line of work, including future studies, will better inform practitioners and policymakers about the context and boundary conditions that make such agility valuable and necessary, given specific industry and macro-economic contexts.

We illustrate our position through the prism of pivoting (O’Connor and Klebahn 2011). This is a popular and influential idea in business schools and among practitioners who claim that the “secret sauce” for successful startups such as Groupon, Pay Pal, and Twitter is decisive, rapid and early shifts in business strategies based on frequent engagement with customers. Through these pivots, startups reach out to customers in early stages of product development and make deliberate as well as often dramatic changes to the business model in response to customer feedback received. To some extent, the idea of pivoting is an antidote to technophile-dominated startups where the teams spend much of their time “in the laboratory” instead of interacting with real users. Thus, a crucial task of startups is to learn “what customers really want” through pivots, which is echoed by Ries in his popular The Lean Startup (2011). However, the concept of pivoting, while appealing to entrepreneurs, is not without criticism. McGinn (2012) argues that an overemphasis on pivoting might encourage a culture of opportunism, in which people starting new companies engage in “hopscotching” between widely different ideas. As McGinn points out, the latest trend of pivoting might overlook passion, commitment and sustained efforts, which are well recognized as key qualities of successful technology entrepreneurship (Stevenson 1983).

The pivoting concept has clear ties to organizational theory: in dynamic capabilities parlance, the pivot is essentially an iterative process of sensing and seizing to manage threats proactively and capitalize on new business opportunities (Teece et al. 2016). Zahra et al. (2006) propose that over time new ventures “calcify” meta routines (i.e., dynamic capabilities facilitating change) through a progression from ad hoc improvisation to trial-and-error to eventual resource-intensive experimentation.

Experimentation is increasingly seen as a viable option as new ventures, especially those that are software based, adopt lean principles emphasizing hypothesis testing and customer-driven approaches to product development (Humble et al. 2015). The green goods manufacturing industry is notably different than software based industries that are amenable to extreme rapid cycle iterations and frequent pivoting, however (Humble et al. 2015). We wish to highlight three distinctions here: First and foremost, we posit that the timeframe between noticeable changes in any manufacturing sector, at least as observed on firm websites, is not on the order of weeks (or sprint cycles in software) but rather months or years: tangible goods, especially those of a green goods nature, necessitate relatively longer lead times for changes in technology, supply chain coordination, and environmental practices (Chung and Wee 2011). Secondly, the consequences of any new pivot are likely more important in green goods manufacturing than in software or Internet related industries, which experience lower fixed costs, potentially less liability for faulty products, and less costly channels for repair.

Our study design addresses change in two periods and does not explicitly reflect the role of customer input. Yet, in line with these cautionary tales, our primary results suggest that there are two kinds of hazards to SMEs operating in uncertain and fluctuating technology-intensive markets such as the green goods manufacturing sector. The first hazard is the lack of ability, willingness, or resources to invest in seizing activities. If these entrants lack the flexibility to experiment with and make changes in response to technological or market fluctuations, they will experience limits in their ability to grow. The second hazard stems from an overemphasis on strategic changes. The risk here is that growing firms become locked-into the notion of change for change’s sake.

Can websites be used to study agility and rapid cycle pivoting? Despite its popularity, the pivoting concept has not been deeply analyzed in business and management literature.Footnote 4 We suggest the answer is yes, but with important caveats. In this work, we used a sample of 223 US-based green goods SMEs and their archived website information to explore the relationship between strategic change and the growth of innovative small manufacturing firms. Taking advantage of recent developments in information technologies including web scraping and topic modeling, our study develops a measure of strategic change by quantifying the evolution in firm website topics. We showed that this measure is reasonably robust in detecting changes in firm behavior.

However, the pivot is not all about seizing; it is sensing, too. We used changes in R&D keyword mentions as a proxy of sensing, but the pivoting concept focuses more on customer feedback processes than absorptive capacity. As a result, it is then necessary to consider how a firm’s customer feedback channels can be detected in the website data collection and variable operationalization process. While there may be some pointers to such channels on the website itself (e.g., in the topics that emerge), we posit that websites themselves may already be fine-tuned to elicit customer feedback and interaction as marketing vehicles. The point of hypothesis-driven testing in the pivoting framework is to solicit feedback and improve value capture of products and services (Humble et al. 2015), and we posit that every time a website substantially changes, it could be a way of improving the customer experience. Alternatively, it could be the case that websites are not able to determine sensing activities through customer channels and thus additional data sources (e.g., such as survey instruments) will be needed.

There are other limitations associated with using company websites as an information source. Websites are essentially self-reported by companies and have not been validated in a consistent way by third party data providers. One issue is that firms differ in what they publish on their sites. To some extent, variations may represent sub-sectoral differences. For example, biotechnology firms may be more apt to report information on patents, publications, and clinical trials while firms in the energy domain may be more apt to report information about customers and partnerships (Youtie et al. 2012). Variations may also exist over time; for example, at one point a firm may desire high visibility to attract visits from potential funders or customers while at another time, the firm may seek to “go dark” and reveal very little information while working on a new application or trade secret (Arora et al. 2012). The variation in information on a website over time is one reason that we re-scraped websites of the green good SMEs in our dataset and normalized results over multiple years. A second issue is that information embedded in images, PDFs, or Flash displays are not captured by our approach, which requires HTML text-based data in order to process. We manually reviewed the websites we scraped and found that non-textual information was not extensive for the SMEs in our dataset, but this type of checking should be performed whenever website data are used. In sum, we regard web scraping (subject to manual verification of websites, c.f. Arora et al. 2015) as a value-adding source particularly when data from official statistics, scholarly publications, or other patent and business information sources are lacking.

Within the confines of these limitations, we position our contributions through an original, systematic, and quantitative analysis of a conceptualization of strategic change and associated small firm behaviors using website data. Our results have research, managerial, and public policy implications for promoting entrepreneurship. The narratives that emerge on websites are self-determined and updated on firm-specific terms and schedules. As a result, we are able to ask questions about the entrepreneurial narratives that emerge (Autio et al. 2014; Zott and Huy 2007). This is an approach that could be used to probe other relevant entrepreneurial research topics, for example, how the adoption of new media channels, including websites and social media, influences a firm’s ability to access complimentary resources, attract and retain talent, and survive and grow. A common feature of this line of research is the use of data-intensive methods to analyze enterprise behaviors exhibited online, with the mapping of these behaviors to existing or new types of entrepreneurial business models and innovation patterns. In addition to web mining, other accessible platforms and sources (such as LinkedIn or Twitter) present themselves as sources of big data for business and innovation analyses. As our study illustrates, careful attention must be given to the reliability of such sources. Consideration of data protection and privacy is a further prerequisite. Nonetheless, as sources of big data grow both in scope as well as scale, we anticipate ongoing value for entrepreneurship research in finding ways to use these sources to understand increasingly dynamic business environments and firms.