Introduction

Technological innovation is increasingly the result of collaborative efforts among various entities [52]. In this scenario, the interaction of researchers from academic institutions with their external environment emerges as a fundamental element in the development of new products and innovative solutions. The open innovation approach emphasizes that the diversity of knowledge and perspectives significantly contributes to innovation, making partnerships between different types of organizations a crucial vector for success [21]. The synergy of these partnerships can generate radical innovations and overcome the “valley of death” of innovation, a challenge widely documented in the literature [31, 45].

The inherent complexity of this collaborative ecosystem, however, presents significant challenges. Previous studies [60, 70] have highlighted various barriers to university–industry collaboration, including differences in goals among parties, low levels of research and development (R&D) application in company activities, and a lack of mutual understanding about expectations and priorities. From the researchers’ perspective, engaging with the industry is not only about advancing theoretical knowledge but also about ensuring that their research has practical applications. This interaction, central to the open innovation paradigm, is key to securing access to industry resources, gaining insights into market needs, and fostering professional development through exposure to real-world challenges and technologies [74]. Against this backdrop, the interaction of researchers with the business environment emerges as an important mechanism for fostering innovative technological solutions, where the formation of robust relationships and the building of trust play a catalytic role in the effectiveness of collaboration [56, 62].

Simultaneously, the need to align with market demands and the ability to respond promptly to them become evident. Market orientation, understood as the generation, dissemination, and responsiveness to market-related information, such as preferences and/or social needs [48, 53], acts to guide R&D activities toward outcomes that not only advance knowledge but also meet the pragmatic needs of the sector [81]. Ultimately, this market orientation can be decisive in overcoming the valley of death, ensuring that innovations have a viable path to practical and commercial application [45].

The concept of the “valley of death” illustrates one of the greatest obstacles in the innovation lifecycle, identified specifically between technology readiness levels (TRLs) 5 and 6, a phase in which a technology is validated in a relevant environment but has not yet been demonstrated in an operational context [36]. This challenging phase, also recognized between TRLs 5 and 7 by other studies, is characterized by the difficulty in advancing technologies due to the industry’s reluctance to adopt innovations that have not reached a higher stage of maturity, typically TRL 7 or 8 [79]. The TRL scale, a well-known nine-level methodology for assessing technological maturity, is widely used in the aerospace industry to guide the development of innovations from initial conception to full commercialization [9].

While the significance of social capital and market orientation in R&D collaborations is recognized, the interaction of these elements with TRLs, especially at this critical juncture, remains underexplored. Social capital, which is essential for building trust networks, and market orientation, which ensures that research efforts are aligned with market demands, are determining factors for overcoming the “valley of death.” However, how these factors interact with TRLs to influence the intensity and efficacy of researcher engagement with the business sector, especially at different stages of technological maturity, represents a significant gap in current knowledge. This study aims to investigate this dynamic, seeking to understand the influence of social capital, market orientation, and TRLs on the intensity of collaboration between researchers from a public research institution in Brazil and companies, and how TRLs moderate this relationship. The research questions addressed are as follows: what is the influence of social capital, market orientation, and technology readiness levels on the intensity of the interaction between researchers from a public research institution and companies? What is the moderating effect of technology readiness levels? Through this investigation, we can devise strategies to enhance researcher engagement at various stages of technological maturity, providing valuable insights to tackle the challenges of the “valley of death” and contribute to the success of technological innovation.

Public researchers play a role, in driving innovation benefiting from resources like funding and strong academic connections that propel technological progress. Their focus on addressing issues aligns well with endeavors geared toward solving real-world problems. The principle of innovation underscores the significance of tapping into expertise and partnerships to hasten technological advancements and boost competitiveness. Therefore, examining how these researchers engage with the business realm can offer insights into enhancing support for innovation bridging the gap between research and market demands and nurturing an environment conducive, to fostering innovations through open innovation.

The methodology adopted in this study employs a quantitative analysis through structural equation modeling with a partial least squares (PLS-SEM) approach, which is suitable for exploring the complexity of interactions among social capital, market orientation, TRLs, and the intensity of researchers’ interaction with companies. This methodological choice allows for a precise evaluation of direct relationships and moderating effects among variables, revealing how technological maturity influences collaboration dynamics. By focusing on the experience of researchers from a public research institution in Brazil, this study highlights the uniqueness of the Brazilian context in contributing to a broader understanding of R&D interactions, reinforcing the importance of the study in the global R&D landscape.

Through this investigation, we make a significant contribution to the theoretical base and administrative practice. Theoretically, this work is expected to provide new insights into how these factors influence the intensity and efficacy of interactions between research institutions and companies. Specifically, by examining the moderating effect of TRLs on these relationships, this study illuminates the conditions under which social capital and market orientation are most effective in fostering fruitful collaborations. This analysis not only enriches the existing literature on R&D interactions, offering a more nuanced understanding of these complex interactions, but also challenges and expands current theoretical models by exploring the role of TRLs as a critical contextual factor. In practice, the findings of this study can offer valuable guidelines for managers in the academic and business sectors. For academic managers, the findings can illuminate strategies to strengthen the social capital and market orientation of their institutions, maximizing the potential for successful collaborations with the business sector. This might include the development of training programs that focus on building effective networks and awareness of market needs. For business sector managers, this study highlights the importance of identifying and engaging academic partners with high social capital and strong market orientation, in addition to the relevance of considering TRLs when setting expectations for R&D interactions. Furthermore, the findings can assist both groups in developing more effective approaches to managing the “valley of death” in technology, facilitating a smoother transition of innovations from the lab to the market.

Literature review and development of the hypotheses

R&D interaction between researchers and companies

The interaction dynamics among universities, governmental research institutions, and the private sector is a vital component of open innovation, a paradigm that acknowledges the necessity for companies to absorb external knowledge to accelerate innovation and enhance competitiveness [21]. Governments around the world encourage the formation of significant partnerships with the private sector [78], recognizing the value of R&D collaborations in enhancing knowledge generation, reducing costs, complementing skills, and minimizing risks [55]. Such collaborations enable the sharing of risks and the optimization of resources among the involved parties, founded not only on contractual agreements but also on trust and reciprocity [76]. In recent years, the proliferation of these partnerships has highlighted their importance for economic growth, with countries demonstrating the benefits of these arrangements [83]. In Australia, formal collaborations with universities generate revenues exceeding AUD 10 billion annually, culminating in an economic impact that surpasses AUD 20 billion per year [35], thus contributing to social development and improving local quality of life.

Open innovation reveals that interactions between researchers and companies are crucial for the absorption and application of external knowledge. Chesbrough [21] argues that companies should open their innovation processes to incorporate external knowledge and explore new opportunities, facilitated by collaborations with research institutions.

Interactions between researchers from research institutions and companies reveal an underlying complexity that encompasses personal, professional, and institutional factors. Bozeman and Gaughan [14] illustrate how industry grants and contracts amplify researchers’ propensity to collaborate with the private sector. Specifically, affiliation with university research centers is highlighted by Craig Boardman and Ponomariov [23] as a significant facilitator for such collaborations, underlining the role of organizational structures in promoting productive interactions. This point is reinforced when considering the diversity of interaction forms, which extend beyond licensing and patenting activities to include informal knowledge exchanges, contracted research, and consulting, suggesting a broader spectrum of academic engagement with the industry.

The motivation for such collaborations is also examined, with studies identifying a range of driving factors, from the desire to apply research outcomes in the real world to the pursuit of additional financial resources. For example, the research by Franco and Haase [33] highlights that researchers’ motivation can be influenced by both intrinsic factors, such as the desire for academic reputation and practical application of their findings, and extrinsic factors, like access to advanced technologies and funding. Moreover, De Fuentes and Dutrénit [26] demonstrated how specific interaction channels, such as joint R&D, have a substantial impact on the benefits perceived by companies, suggesting the need for a more granular approach in analyzing these collaborations.

On the other hand, researchers seek partnerships with the private sector motivated by financial and infrastructure needs, where social capital and project funding play a central role in these collaborations [56]. Proximity to companies allows researchers to quickly align their activities with market needs because of their experience and in-depth understanding of business dynamics [44].

Social capital and its implications for R&D interactions

Social capital constitutes a set of current or potential resources linked to a network of enduring relationships, mutual acquaintance, and recognition, providing each party with support for collective capital [12]. Characteristics such as networks, norms, and trust, which facilitate collaboration among people and institutions, are components of social capital [67]. Nahapiet and Ghoshal ([59], p. 243) defined social capital as the “sum of the actual and potential resources embedded within, available through, and derived from the network of relationships possessed by an individual or social unit”. Social capital increases the efficiency of institutions by allowing them to exchange and combine their resources [82]. This capital is linked to the individual and their power to influence a network of relationships through social norms and trust [12].

From the perspective of open innovation, social capital is essential for facilitating knowledge exchange and collaboration between researchers and companies. Trust and social networks enhance the ability of institutions to engage in effective R&D interactions and to explore innovation opportunities [21].

The intensity of R&D interaction depends on strong relationships involving structural network issues and trust among individuals. Nahapiet and Ghoshal [59] proposed subdividing social capital into three dimensions: structural, relational, and cognitive.

The structural dimension refers to the links between individuals in institutional networks that provide access to information and structures, creating opportunities and being a prerequisite for collaboration formation [72]. Structural social capital involves an individual’s network connections and the resource exchanges that occur among them, with principal characteristics centered on the quantity of links, configuration, and density among network members [55]. Information exchange through socialization is vital for an innovation project [50] and tends to occur at high levels of structural social capital.

The relational dimension is linked to the interactions between individuals and the trust generated over time. The resources generated in this dimension are especially important at the start of collaboration when greater familiarity among partners is required [77]. Relational social capital encompasses various facets that contribute to innovation, primarily in establishing the foundations for collaboration [59]. Trust is associated with the concept of associability, i.e., the way in which individuals align their desires and actions with collective goals [19]. Trust increases people’s willingness to engage in collaborative activities [19, 67] and facilitates the exchange of resources and information necessary for projects [19]. Considering radical innovation in collaborative projects, establishing a trustful environment and deep knowledge of technological domains is imperative, given the high investment risk [56]. The absence of trust can obstruct knowledge transfer, which is a crucial element in the joint development of technologies [69]. Trust by reducing uncertainty and opportunism among external partners [50], promotes commitment and collaboration and minimizes conflicts [83]. Moreover, trust acts as a resource in mitigating the risks of opportunistic behaviors, such as fraud or cheating [61].

The cognitive dimension of social capital encompasses tools and resources that enable the effective sharing of understandings and meanings among participants [59], which is influenced by the closeness of relationships and collaborative experiences. This dimension is particularly relevant to R&D interactions because it facilitates the creation and combination of both tacit and explicit knowledge, which are essential for innovative processes [4]. The ability to share a common language, goals, and cultural norms significantly enhances collaborative efforts by ensuring that all parties have a mutual understanding and aligned objectives, thereby streamlining the innovation process [46, 77]. Furthermore, elements such as unified language, cultural awareness, and shared norms and values are crucial for aligning the efforts of diverse organizational actors [55, 69], such as public research institutions and companies. These elements help to bridge the cultural and objective differences that often exist between these entities, making the cognitive dimension a linchpin in successful R&D collaborations [46]. Therefore, not only does the cognitive dimension facilitate the necessary communication for collaborative innovation, but it also establishes a trust foundation that is vital for managing the high investment risks associated with radical innovation projects [77].

An interconnection between the three dimensions—structural, relational, and cognitive—is consistent across numerous studies. Strong ties in the structural dimension are directly correlated with the development of trust in the relational dimension [86]. The cognitive dimension correlates with the relational dimension, as knowledge transfer and the use of common norms and language broaden trust among actors. During the pre-formation phase of collaboration, all dimensions of social capital play a useful role in mitigating barriers [3].

Delving into the relationship between social capital and R&D interactions, this study introduces the frequency of these interactions with companies as the dependent variable, reflecting the tangibility of collaborative engagement. This quantitative approach allows for an objective analysis of how networks, trust, and knowledge sharing—components of social capital—directly impact the intensity of interaction between researchers and companies. Thus, we outline the following hypothesis:

Hypothesis 1

The social capital of researchers from a public research institution positively influences the intensity of their R&D interactions with companies.

Market orientation and implications for R&D interactions

In an open innovation ecosystem, market orientation plays a critical role, acting as a catalyst for productive partnerships between academic research actors and the industrial sector. This orientation involves not only identifying and understanding market needs but also effectively responding to these demands through collaborative innovation. The absence of this orientation can significantly limit the potential impact and relevance of R&D outcomes, underlining the need for a market-oriented approach aligned with the expectations and needs of industries [80]. This orientation pertains to identifying and understanding society’s needs (market intelligence), disseminating this information among the team (intelligence sharing), and effectively responding to these demands (responsiveness) [48, 81].

Open innovation emphasizes the importance of seeking information external to the organization [21]. At the same time, a strong market orientation within research institutions aligns R&D activities with market demands, maximizing the value of collaborations between academia and industry.

Previous research has shown a positive correlation between the market orientation of academic research groups and the industrial partner’s commitment to collaboration [66]. However, the lack of a focus on value creation can diminish the partner’s intention to maintain the relationship, especially if the partner places high importance on market orientation [65]. A lack of market knowledge can lead to difficulties in meeting industry expectations in research projects, resulting in outcomes that are less desirable than initially expected and inadequate for their needs [81]. Differences in market orientation can negatively affect the integration between partners and impact the continuity of collaboration [37].

Market orientation seeks to understand industry needs, paving the way for the exploration of creative opportunities for the productive sector. The more market-oriented the researcher, the greater the stimulus for knowledge sharing and, consequently, collaboration with the industry [81]. Market orientation is one of the most relevant variables in academia–industry collaboration, as divergences in this aspect can hinder communication, information, and knowledge creation [37], affecting project satisfaction and continuity. Applying the approach of Schlosser and McNaughton [75] emphasizes the importance of market orientation at the individual level (I-Markor), which is crucial for the success of R&D partnerships. Focusing on information acquisition, sharing, and strategic response, this approach illuminates how individual market-oriented behaviors can energize collaborative innovation.

Market orientation is operationalized in three dimensions: market intelligence generation, market intelligence dissemination, and responsiveness. The first encompasses society’s needs and preferences (customers, competitors, etc.) in the present and future, including environmental monitoring and changes in regulations impacting everyday life [47]. The second involves the participation and integration of various organizational teams in the process of sharing information produced by market intelligence [47]. Finally, responsiveness seeks to take what is discovered through market intelligence beyond mere dissemination to effectively meet society’s needs [47].

Individual analysis suggests that researchers who are attentive to market dynamics are more inclined to promote knowledge sharing and effective collaboration with the sector. This individual perspective stimulates innovative behaviors by employing market information to creatively address complex challenges [6]. This dynamic suggests that market orientation not only facilitates the formation of effective partnerships but also drives the intensity of R&D interaction between researchers and companies. Therefore, we propose the following hypothesis:

Hypothesis 2

The market orientation of researchers from a public research institution positively influences the intensity of their R&D interactions with companies.

Technological readiness assessment and the “valley of death”

To enhance communication and create a more effective way of assessing the maturity of technological solutions, the National Aeronautics and Space Administration (NASA) introduced the concept of Technology Readiness Levels (TRLs) in the 1970s [57]. The TRL scale begins with a technological readiness assessment, which is a point at which an organization attempts to determine the maturity of a new technology or capability [57] and initially consisted of six or seven levels. In 1995, the concept was refined and now depicts nine levels of readiness, each with specific characteristics that define them.

TRL 1 (Fig. 1) is the initial level and represents the most basic stage of technological development, such as studies on the fundamental properties of materials [57]. At TRL 2, the knowledge generated at the previous level is applied in a practical manner, e.g., using carbon nanotubes and identifying specific uses for this material [57]. At levels 3, 4, and 5, there are laboratory tests, where the concept is proven (TRL 3), functionality is verified (TRL 4), and critical function (TRL 5) is verified, which must meet relevant test conditions to validate the model [8]. TRL 6 is the prototyping phase in the relevant environment [20]. From level 7 onwards, the manufacturing phase begins, requiring a real demonstration of the prototype to ensure confidence in the technology’s development [57]. At level 8, the technology is finalized and has been successfully developed and tested. The last level (9) involves market adoption and the necessary monitoring of any required corrections [57].

Fig. 1
figure 1

Adapted from Capdeville et al. [20]

Technological maturity levels.

The choice of partner is related to the type of product expected in the partnership [42]. Researchers seek the industrial sector to obtain patents and generate superior academic performance because industries are more interested in solutions for the short and medium term [63]. In contrast, partnerships between research institutions and the public sector are more oriented toward basic research, strengthening the researcher’s role in academic studies [42].

The combination of knowledge between partners in R&D projects tends to generate new and valuable knowledge, resulting in disruptive technological innovations. Arant et al. [5] suggested that the knowledge bases between partners should be quite different for radical innovations to occur. Partnerships between academia and industry are invaluable for crossing the “valley of death” and solving funding problems [11]. The “valley of death” is the period in which a researcher attempts to advanced technology from a laboratory concept to a validated instrument for industrial-scale production. This usually occurs between levels 5 and 6 of the TRL scale and is known as the Technological Valley of Death [84].

Academic research, through public resources and funding agencies, can finance its activities at the early stages of technological readiness (TRLs 1–3). However, from TRL 4 onwards, private investment becomes essential. At this stage, the technological development process is considered too applied for further public funding, but still too risky for industrial financing [45]. However, even with alternative financing to cross the valley of death, other factors influence the success of technological development. Recent research examined five areas of innovation risk that managers should consider when designing and executing their projects [28]. Uncertainty in issues such as technology concept, technology performance, commercial and consumer uncertainty, potential impact, and post-death commercialization strategy directly influences companies’ decisions to invest in venture capital. This scenario highlights the importance of interaction between academia and the productive sector from the beginning of technological development.

In the initial research phase, TRLs 1 to 3 (Fig. 1), the focus is on theoretical models and laboratory proof of concept, often conducted within academic institutions [45]. Researchers typically seek partners from companies interested in exploring new areas and willing to invest in basic research. Interactions at this stage are less frequent but deeply significant, laying the groundwork for future collaborations.

As the technology progresses to the development of prototypes and testing in laboratory or controlled environments, TRLs 4 to 6, the need for collaboration intensifies. It is at this critical point of the “valley of death” that researchers increase their interaction with companies, seeking resources to develop prototypes and test functionalities under conditions that mimic the real usage environment [45, 84]. Collaboration is essential to overcome the technical and financial challenges of this phase; thus, the intensity of interactions is expected to significantly increase.

Finally, in the scaling stage for production and commercialization, TRLs 7 to 9, interactions between researchers and companies become more regular and focused on innovation. Partner companies are actively involved in optimizing the product, preparing for full-scale production, and developing market strategies [45, 84]. Interactions at this level are strategic, focusing on product optimization, compliance with regulations, and commercialization strategies.

Open innovation highlights the importance of academia-industry partnerships in overcoming the challenge of the “valley of death” in technology. As technology progresses through the TRL levels, collaboration becomes crucial in providing the necessary resources for the transition from the laboratory to the market [21].

Thus, understanding how researchers’ involvement in technological development projects at different TRLs affects the intensity of their interactions with companies is crucial. It is anticipated that at lower TRLs, the intensity of interactions will reflect the exploratory potential of the research. In intermediate TRLs, these interactions are vital for overcoming financial and technical hurdles. At higher TRLs, the intensity of interactions aligns with efforts toward commercialization and market application. This dynamic leads us to the following proposed hypotheses:

Hypothesis 3

The involvement of researchers from a public research institution in technological development projects at different levels of technological readiness (low, medium, and high) positively influences the intensity of their interactions with companies, such that:

Hypothesis 3a

Researchers’ participation in projects at low technological readiness levels (TRLs 1–3) is positively associated with the intensity of their interactions with companies, but this association is weaker compared with projects at medium and high TRL levels.

Hypothesis 3b

Researchers’ participation in projects at medium technological readiness levels (TRLs 4–6) is positively associated with the intensity of their interactions with companies, and this association is expected to be stronger than that in projects at low TRL levels but weaker than that in projects at high TRL level.

Hypothesis 3c

Researchers’ participation in projects at high technological readiness levels (TRLs 7–9) is positively associated with the intensity of their interactions with companies, and this association is expected to be the strongest among the three levels of TRL.

All stages of technology development are associated with risks. As technology matures, these risks diminish [17]. The stage characterized by the “valley of death” (TRLs 4–6) carries high risk, and partnerships between academia and industry have the power to substantially reduce these risks [11], considering the social capital levels of researchers [59]. Risk sharing optimizes resource allocation among partners; however, it is a dynamic and complex process that depends on mutual trust and reciprocity [76]. Considering that the risks of investing resources decrease as the level of technological readiness increases, we propose the following hypothesis:

Hypothesis 4

The involvement of researchers from a public research institution in technological development projects at different levels of technological readiness (4-a: low level—TRLs 1 to 3; 4-b: medium level—TRLs 4 to 6; 4-c: high level—TRLs 7 to 9) moderates the influence of the researchers’ social capital on the intensity of their interactions with companies.

The influence of market orientation on researchers’ interactions with companies can also vary because of the level of technological readiness (TRLs) of the developments. In the early stages, it is crucial to align research with emerging market needs [81]. At intermediate stages, it adapts innovation to market changes, and at final stages, it paves the way for commercialization [15, 18]. The literature suggests that this orientation, adapted to different stages of technological development, is fundamental for the success of technology transfer and collaborations between academia and industry [37, 66]. Therefore, we propose the following hypothesis:

Hypothesis 5

The involvement of researchers from a public research institution in technological development projects at different levels of technological readiness (5-a: low level—TRLs 1 to 3; 5-b: medium level—TRLs 4 to 6; 5-c: high level—TRLs 7 to 9) moderates the influence of the researchers’ market orientation on the intensity of their interactions with companies.

Figure 2 presents the conceptual model used in this study, with the five hypotheses grounded in the literature.

Fig. 2
figure 2

Conceptual model

Methodology

Population and sample

This study focused on the Brazilian Agricultural Research Corporation (Embrapa), a leader in agricultural research in Brazil with national and international recognition [43]. With 43 research centers and 2085 researchers, Embrapa was chosen for its influence and tendency toward innovation and collaboration with the productive sector, making it an ideal setting to study the impact of social capital and market orientation on R&D collaboration [16]. Embrapa’s presence in all regions of Brazil also offers the opportunity to explore regional and cultural variations in collaboration practices, enriching the analysis and research results.

In this study, a convenience sample of 2085 Embrapa researchers was used, which was suitable for its specific research context [30]. This technique was chosen because of the representativeness and diversity of Embrapa researchers, aligning with the research questions. Despite potential biases from convenience sampling, measures were taken to minimize them, such as efforts to maximize the response rate and comparative analyses between participants and non-participants to identify biases [10, 27]. The selection of Embrapa researchers elevates the relevance and applicability of the results, given the institution’s importance in agricultural research [73].

This study is based on empirical investigation, supported by a quantitative survey conducted through an online questionnaire between September 1 and 31, 2023. The invitation was sent through the institution’s official communication channels, ensuring authentic and comprehensive communication. A total of 930 researchers interacted with the initial invitation, with 847 engaged in the survey and 83 requesting exclusion, resulting in a response rate of 45% (relative to the 2085 researchers invited). This number was later adjusted to 788 participants, after 40 researchers refused the Informed Consent Form (ICF) and 19 did not meet the inclusion criteria. The final sample of researchers had an average of 27.2 years of R&D experience (SD = 10.8), linked to the 43 Embrapa research centers, ensuring geographic representativeness and diversity of research specialties.

Detailed profiles of the participants, their specialties, research experience, and other control and sociodemographic variables are presented in Table 1, offering a comprehensive view of the respondents’ profiles and ensuring the robustness and applicability of the research results to the context of Embrapa and, potentially, other research institutions.

Table 1 Participant profile (N = 788)

Development of the measurement model

This study aimed to elucidate the relationships between social capital and market orientation, TRLs, and the intensity of R&D interaction with companies by researchers from a public research institution in Brazil. We hypothesized that the intensity of R&D interaction could be influenced by both the social capital and market orientation of researchers, with potential variations depending on the technology development stage, as indicated by the TRLs.

This study employed a quantitative methodology and a cross-sectional research design for data processing and analysis, with the goal of confirming or refuting the hypotheses. This approach captured the researchers’ perceptions at a specific point in time, providing crucial insights to answer the research question [16].

The main method used for analysis was structural equation modeling with a partial least squares (PLS-SEM) approach, which is an advanced statistical technique that allows the evaluation of complex relationships between observed and latent variables [2, 39]. The PLS-SEM approach included two phases: measurement model evaluation to test the reliability and validity of the constructs [41], followed by structural model analysis to verify the hypothetical relationships.

The quantitative and correlational nature of this study, aligned with the application of PLS-SEM, facilitates the identification of systematic patterns and relationships between variables, contributing to the generalization of findings to similar contexts [24]. The study also exhibits case study characteristics, owing to its concentration on a single organization [85], enabling a detailed analysis of specific situations and providing in-depth insights into the examined context. Although focusing on individual researchers, the organization to which they belong is crucial for understanding the dynamics of R&D interactions with companies.

The intensity (INT) of R&D interaction between researchers and companies was established as the dependent variable and measured using a scale adapted from Bozeman and Gaughan [14]. This scale incorporates six specific indicators that encompass various nuances of interactions between the academic and business sectors. The evaluation was conducted using a five-point Likert scale, ranging from 1 (never) to 5 (always). A description of the indicators is provided in Appendix 1.

In terms of independent variables, the study explored social capital (SC) from a three-dimensional perspective: structural, relational, and cognitive, with measures influenced by Martínez-Cañas et al. [58] and Nahapiet and Ghoshal [59]. The three dimensions of social capital were operationalized with four statements each. The structural dimension (SSC) focuses on the researcher’s participation in private sector networks. The relational dimension (RSC) encompasses trust, willingness to support, reciprocity, and mutual commitment. The cognitive dimension (CSC) includes shared values and visions, common interpretations, shared systems of meaning, and common norms/codes.

Market orientation (MO) was another independent variable, measured using an i-Markor adapted for academic researchers based on Kohli et al. [48], Schlosser and McNaughton [75], and Ting et al. [81]. MO included three dimensions: market intelligence generation (MIG) with eight items, intelligence sharing within the organization (IS) with seven items, and response capability to the productive environment (RC) with five items.

Technological Readiness Levels (TRLs) were operationalized as both independent and moderating variables to explore the influence of various stages of technological development on the intensity of researchers’ interactions with companies. Using TRLs as moderators allows for a detailed analysis of how different phases of technological maturity can modulate the interaction between Market Orientation (MO), Social Capital (SC), and interaction intensity [57, 71]. To capture the frequency of researchers’ participation in R&D projects at different technological readiness stages, TRLs were categorized into three groups: low readiness (TRLs 1–3), medium readiness (TRLs 4–6), and high readiness (TRLs 7–9). These groups constituted discrete variables, with measurable frequencies of participant involvement in each category. Participants rated the frequency of their involvement in each TRL category using a 5-point Likert scale ranging from “never” to “always” [25]. The scale was applied separately for each level of technological readiness (low, medium, and high), allowing for the quantification and comparison of researcher participation in R&D projects associated with each stage of technological development. This quantitative approach ensures a precise analysis of the participation frequency of researchers and its potential impact on the intensity of interactions with industry.

To reinforce the analysis and mitigate biases, two control variables were included: the researcher’s experience and the research center to which they were linked. The first assesses how the duration of a research career influences collaboration, and the second examines the impact of geographic context on R&D interaction, as highlighted by Bozeman et al. [13] and Fritsch and Schwirten [34].

Each construct was meticulously operationalized and measured, ensuring a detailed and robust analysis of the proposed relationships and a comprehensive understanding of R&D interactions between academic researchers and companies. A crucial pre-test was conducted from July 3 to 14, 2023, with 22 researchers, focusing on the clarity, relevance, and suitability of the questions and identifying ambiguities or difficulties in interpretation. This preliminary phase refined the study’s methodology, allowing adjustments to the measurement model and alignment of questionnaire items with the research objectives.

In the validation process of the SC and MO constructs, an approach of averaging the indicators for each dimension was chosen based on established methodological practices. This choice is based on the analytical simplicity and efficiency that averaging provides, allowing for an effective synthesis of multiple indicator responses into a single representative value for each dimension. This practice is widely adopted in research using Likert scales, especially when indicators are homogeneous and designed to measure the same theoretical construct [38].

The validity of this approach is supported by reliability and validity outcomes. For SC (Social Capital), Cronbach’s alpha was obtained at 0.786, composite reliability (rho_a) at 0.890, rho_c at 0.866, and AVE (Average Variance Extracted) at 0.683. For MO (Market Orientation), the results were Cronbach’s alpha at 0.868, rho_a at 0.886, rho_c at 0.919, and AVE at 0.790. And for INT (Interaction Intensity), the outcomes were Cronbach’s alpha at 0.774, rho_a at 0.790, rho_c at 0.847, and AVE at 0.527. These indicators demonstrate strong internal consistency and convergent validity for the constructs [38].

Ethical procedures

The research adhered to rigorous ethical standards and was submitted to and approved by an ethics committee under a specified protocol, highlighting the commitment to ethical conduct throughout the study. Authorization was obtained from the relevant authorities, ensuring adherence to ethical guidelines. Confidentiality and anonymity of the participants were strictly maintained, and all collected data were treated with the highest level of secrecy. An Informed Consent Form (ICF) was obtained from all participants, ensuring that they were fully informed about the study’s objectives, benefits, and potential risks. This approach upholds the ethical integrity of the study and agrees with international guidelines for research involving humans.

Data analysis and results

In evaluating the hypotheses and exploring the proposed theoretical model (Fig. 2), the Partial Least Squares Structural Equation Modeling (PLS-SEM) method was applied using SmartPLS software, version 4.0.9.6 [68]. This method is recognized for its effectiveness in investigating complex relationship structures [2]. A notable advantage of this process is its flexibility regarding data distribution and sample sizes, facilitating analysis in various contexts [38, 40].

Discriminant validity

In this study, the assessment of discriminant validity was conducted following the criteria of Fornell and Larcker [32], a widely accepted methodology in research data analysis. This step is crucial to confirm the uniqueness and distinction of each construct within the model. The results presented in Table 2 demonstrate that all constructs meet the established criteria for discriminant validity, thus evidencing the appropriate separation and singularity of each construct in the context of the research model.

Table 2 Discriminant Validity by the Fornell and Larcker Criterion

Discriminant validity in our study was also investigated through the Heterotrait–Monotrait (HTMT) ratio, an alternative method that offers a comparative perspective between correlations of distinct constructs and correlations of the same constructs. This method is based on comparing heterotrait (different constructs) and monotrait (same construct) correlations, where an HTMT value close to 1 indicates a possible lack of discriminant validity. Following the guidelines proposed by [41], we adopted a conservative threshold of 0.85 for the HTMT. According to Table 3, all calculated HTMT values are below this threshold, suggesting a clear distinction between constructs and therefore reinforcing the discriminant validity of the model.

Table 3 Discriminant validity by the heterotrait-monotrait (HTMT) matrix

Path model analysis and hypothesis testing

Our analysis employed the PLS-SEM approach to unravel the structural relations within the model of researchers’ interaction intensities with companies involved in R&D activities. We investigated various model configurations, including causality inversions, to arrive at a robust final model. The fit indices of the final model indicate satisfactory adequacy, with an SRMR of 0.053, reiterating the accuracy of the estimated model. Specifically, paths in the model were significant at various points, with data obtained through bootstrapping with 5000 samples. The relationship between SC and interaction intensity (INT) demonstrated substantial strength and statistical significance (β = 0.226, t = 6.039, p < 0.001), supporting hypothesis 1 and emphasizing the vital role of social capital in interaction dynamics. Likewise, market orientation (MO) showed a positive and significant influence on INT (β = 0.258, t = 6.705, p < 0.001), supporting hypothesis 2 and highlighting the importance of aligning research activities with market trends to foster interaction with companies.

The level of technological readiness, represented by the different stages of TRL, presented distinct effects. For the relations of TRL1_3, TRL4_6, and TRL7_9 with INT, the coefficients were 0.078, 0.099, and 0.240, respectively, all with statistical significance (p values of 0.016, 0.002, and less than 0.001, respectively), validating hypothesis 3. The TRL7_9 level exhibited the most pronounced influence on INT (β = 0.240, t = 6.793, p < 0.001), suggesting that projects at advanced stages of technological development are more likely to engage in interactions with companies.

The interactions TRL1_3 * SC for INT, TRL4_6 * SC for INT, and TRL7_9 * SC for INT were not significant, with coefficients of 0.048,  − 0.021, and 0.024 and p values of 0.209, 0.573, and 0.603, respectively, not confirming hypothesis 4 (moderation).

The interaction TRL4_6 * MO for INT was significant, with a coefficient of 0.104 and a p value of 0.004, partially confirming hypothesis 5 (moderation). However, the interactions TRL7_9 * MO and TRL1_3 * MO for INT were not significant, with coefficients of −0.038 and 0.029 and p values of 0.389 and 0.478, respectively.

The control variables CENTER, corresponding to the research center to which the researcher is affiliated, and EXPERIENCE, identifying the researcher’s length of experience in research, showed no significant effects on the dependent variable INT (Fig. 3).

Fig. 3
figure 3

Structural model results

To contextualize the statistical results within the broader narrative of our discussion, Table 4 presents a systematic summary of these findings. These consolidated data allow us not only to validate our initial hypotheses but also to understand the depth and applicability of the theories of social capital and market orientation in fostering researchers’ interaction with companies.

Table 4 Summary of the structural equation modeling (PLS-SEM) results

Discussion

Our study underscores the importance of social capital in fostering R&D interaction with companies, a finding that resonates with the theories of Bourdieu [12] and Putnam [67]. The positive and significant relationship between social capital and interaction intensity with companies (β = 0.226, p <  0.001) illustrates the power of relational networks, trust, and resource exchange, which are key components of social capital, as described by Nahapiet and Ghoshal [59] and Steinmo and Rasmussen [77].

Market orientation emerges as a significant factor in interaction intensity with companies (β = 0.258, p < 0.001), aligning with studies by Kohli et al. [48] and Ting et al. [81]. This result reflects the importance of understanding market needs and the ability to respond to them, highlighting that researchers with a strong market orientation are more likely to engage in productive partnerships with the business sector. This finding agrees with the literature that emphasizes the role of market orientation in generating market intelligence, sharing information, and effectively responding to societal demands [1, 53]. The discovery supports the notion that a closer alignment with market needs and trends can foster a more conducive environment for collaborative innovation, as it facilitates communication, mutual understanding, and knowledge creation between academic and industrial partners [37].

The influence of social capital and market orientation on interactions with companies presents an equivalence of impact, a finding that provides new insights into the interdependence of these variables in the R&D context. While previous studies often examined these variables in isolation, highlighting the singular role of social capital [12, 67] or market orientation [48] in facilitating innovation, our analysis suggests that a synergistic approach may be more representative of the reality faced by researchers. This understanding of similar weights reflects a balance between a researcher’s ability to effectively engage in networks [59, 67] and to understand and respond to market needs [48, 81]. The combined influence of these factors illuminates the multifaceted dynamics between social integration within the sector and the ability to navigate market demands, an interaction that is corroborated but not fully explored in the current literature [1, 53].

This equal relationship may suggest that social capital not only facilitates collaboration but also enhances the understanding and incorporation of market perspectives into R&D activities. This could explain how robust networking naturally affords researchers improved access to market trends and business needs, thereby informing and enriching their market orientation [37]. Battaglia et al. [7] emphasize that insufficient business networks and a lack of market-focused orientation among researchers restrict the advancement of research-based inventions to higher Technology Readiness Levels (TRLs), where they are prepared for commercial exploitation.

The synergistic analysis of the influence of social capital and market orientation highlights the complexity of interactions in the R&D ecosystem and paves the way for a deeper understanding of the role of Technological Readiness Levels (TRLs). This synergy is emphasized in the analysis by Kruja-Demneri [49], who identifies collaborativity—driven by a deep understanding of market demands—as a significant factor for enterprise performance in the agro-industrial sector. This understanding is broadly supported by findings from Confraria and Vargas [22], who noted that institutions with a greater diversity of research partners, especially in fields such as agriculture and plant and animal sciences, tend to collaborate more intensely with the industrial sector. This pattern is particularly relevant in the context of agribusiness in Latin America, a crucial sector for the economic and environmental sustainability of the region. Research indicates that social capital networks not only facilitate technology transfer but also promote innovations needed to address the specific challenges of agribusiness. In this scenario, Brazil emerges as a leader in scientific production. However, despite this academic leadership, collaboration between science and the business sector is limited, with co-publications representing less than 1% of the total, highlighting a gap between research potential and practical application.

This finding reinforces the need for robust social capital networks that not only connect researchers within and between institutions but also extend these connections to the business sector, thereby facilitating the translation of innovations to the market. The integration of TRLs in our analysis underscores the importance of aligning these research capabilities with stages of technological development that maximize the commercial readiness of innovations, especially considering the challenges of the “valley of death” in technology, which are pronounced in the Brazilian R&D context. Market orientation and the development of effective social capital networks can create a conducive environment for overcoming these challenges, promoting more intense and productive collaboration between academic researchers and companies, as suggested by the dynamics explored by Kruja-Demneri [49] and Confraria and Vargas [22].

As social capital and market orientation are integrated into the innovation process, they shape and are shaped by the technological trajectory of the innovations. Thus, the study of TRLs becomes a natural complement to this discussion, as these levels represent not only the technical progression of innovations but also the context in which social and market interactions occur.

As we explore the association between the TRLs and the intensity of researchers’ interactions with companies, it becomes evident that technological readiness transcends a simple marker of technical progress; it signals to companies the level of researchers’ commitment and the potential commercial readiness of innovations. This link between technological advancement and strategic collaboration leads us to a more detailed discussion of how stages of technological development—from the earliest to the most advanced—influence and are influenced by the dynamics of social capital and market orientation.

The relationship between different levels of technological readiness (TRL) and interaction intensity with companies is notable, with higher TRL levels showing the most substantial effect (β = 0.240, p < 0.001). This result aligns with the theory that technological maturity, represented by TRL levels, is a critical factor in attracting business interest, as suggested by Liu et al. [54] and Mankins [57]. The positive association between higher TRLs and greater interaction intensity with companies reflects the preference of businesses for technologies closer to commercialization and, consequently, a closer connection with researchers developing such technologies. The confirmation of this hypothesis reiterates the importance of considering the technological development stage in R&D partnerships, especially in the context of the technological “valley of death”, where transitioning from concept to industrial-scale production is challenging, as discussed by Upadhyayula et al. [84]. This finding reinforces the need for effective collaboration strategies that integrate technological development and market demands, thereby promoting the successful transition of innovations from the lab to the market.

The role of TRL in moderating the relationship among social capital, market orientation, and interaction intensity with companies reveals a complex scenario. The significant positive moderation of TRL4_6 in the relationship between market orientation and interaction intensity with companies (β = 0.104, p = 0.004) suggests that, at intermediate stages of technological development, market orientation is important for promoting interaction. However, the lack of significance in other interactions indicates that the influence of TRL as a moderator is not uniform and may depend on specific contextual factors, such as the different risks and challenges associated with each TRL stage [64]. This result emphasizes the dynamic and complex interactions between variables in the R&D environment, where the impact of TRL levels can vary significantly depending on the readiness level and specific characteristics of the technologies involved. In this context, it becomes relevant to adopt the countermeasures suggested by Landi and Wei [51]. These include clarifying the roles of research institutions, encouraging their active participation up to advanced TRL phases, and promoting integration with companies from early stages, like TRLs 1–3. Additionally, developing more detailed cooperation plans to strengthen partnerships between academic institutions and the business sector is important. Such strategies can help mitigate challenges in the critical phases of technological development, facilitating more effective and productive collaboration between research institutions and the business sector.

The findings of this study highlighted that the influence of advanced technological readiness levels (TRL7_9, β = 0.240) demonstrated an influence similar to the effects of social capital (β = 0.226) and market orientation (β = 0.258) on the intensity of researchers’ interactions with companies. This observation suggests a tripartite strategy for research institutions and companies aimed at maximizing the effectiveness of R&D collaborations: simultaneously investing in strengthening social capital networks, fine-tuning sensitivity to market demands and trends, and directing research efforts toward technological phases closer to commercialization.

However, it is necessary to recognize that focusing exclusively on advanced stages might neglect the innovation potential in the early phases of technological development (TRL1_3), where market orientation can anticipate and shape R&D guidelines. Thus, we recommend dynamic integration across all levels of TRL. This involves not only promoting intense and productive interactions in the more mature stages (TRL7_9) but also establishing a solid foundation in the early stages, where market-research interactions can effectively guide innovation pathways from their inception. Therefore, rather than a simple linear transition to higher technological readiness phases, we propose a strategy that encompasses the entire spectrum of TRL. This approach involves continuously adjusting market orientation as technology progresses while maintaining and expanding social capital networks. Such a strategy not only prepares the ground to overcome the “Valley of Death” in technology but also ensures that innovations are aligned with actual market needs and that the transition from laboratory to commercial-scale production is both efficient and effective.

This vision recognizes the multifaceted nature of collaboration in R&D, emphasizing that success in this area requires more than just technical excellence or innovative advances. It demands an understanding of the social dynamics, market forces, and strategic planning that underpin successful partnerships. By focusing on these areas, research institutions and companies can better navigate the complexities of introducing new technologies into the market, thus contributing to the advancement of science and stimulating economic growth.

Conclusions and final remarks

R&D interaction between researchers from research institutions and the business sector is a critical lever for technological innovation, constituting an essential vector for global economic and technological advancement. This study aimed to unravel the role of social capital, market orientation, and technology readiness levels (TRLs) in the intensity of this interaction from the perspective of individual researchers, framing an empirical investigation at the Brazilian Agricultural Research Corporation (Embrapa). Through a quantitative methodology, specifically structural equation modeling with a partial least squares (PLS-SEM) approach, we analyzed the complex relationships between these variables, seeking to understand how they influence and are modulated by different TRLs.

The findings of this study, by demonstrating that interaction with companies is not limited to formal projects but permeates a continuous routine of researchers’ engagement, signal a significant innovation in understanding the dynamics of R&D activity interactions. Through analysis at a renowned Brazilian research institution, it was observed that researchers’ competitive advantage, potentially enhanced by a strong corporate image, is significantly enriched by levels of social capital and market orientation. These elements are crucial for the intensity of these researchers’ interactions with the business sector. Moreover, it was noted that participation in R&D projects at various TRL levels distinctly influences the intensity of this interaction, with participation in projects at intermediate TRL stages positively moderating the relationship between market orientation and interaction intensity with companies. This pattern underscores the value of a proactive market perspective in promoting technological advancement beyond the “Valley of Death”.

This study makes valuable contributions to theory and practice. Theoretically, it enriches the literature by integrating interactions among social capital, market orientation, and TRLs within a unified model under the umbrella of the open innovation approach. Refuting the notion that these variables operate in isolation, we propose a framework of interdependence that more accurately reflects the complexity of interactions in R&D environments. The open innovation approach emphasizes the importance of leveraging external knowledge and collaborative networks to enhance innovation outcomes, which aligns well with our findings. This synergistic approach offers new perspectives for understanding how interrelated variables facilitate innovation and effective collaboration. Practically, this study suggests that research institutions and companies should emphasize developing and maintaining high social capital and a robust market orientation. Specifically, it is recommended that 1) research institutions invest in networking activities and the development of strategic alliances to strengthen social capital; 2) training programs be implemented to enhance researchers’ understanding of market needs, aligning innovations with these demands; 3) adaptive strategies be developed to navigate through different TRLs, maximizing the commercial readiness of technological innovations and mitigating the risks associated with early development.

The limitations of this study include its focus on a single public research institution and its reliance on self-reported data, which may introduce biases. Future research could expand the analysis to include multiple institutions and gather data from more varied sources, in addition to investigating how factors such as organizational culture, incentive policies, and governance structures influence this collaboration. Although the research explored the interaction between technological readiness levels (TRLs), social capital, and market orientation, it did not address the influence of external factors such as governmental regulations and macroeconomic conditions, which could moderate these interactions. In addition, the specific context and type of innovation, whether incremental or radical, also significantly alters the dynamics of these interactions, with radical innovations requiring deeper and more intensive collaborations. The potential bidirectional causality between the variables suggests that success in advanced TRL stages may reinforce social capital, creating a virtuous cycle with market orientation. This complexity indicates the need for more sophisticated analytical models and longitudinal studies that observe variable dynamics over time, helping to identify strategic interventions and policies to optimize R&D collaboration. Therefore, the current research serves as a starting point for future investigations that may provide a more holistic and integrated understanding of the forces influencing technological innovation.

Furthermore, future studies could benefit from the application of innovative methodological approaches, such as social network analysis, to better map and understand the interconnections and influence of actors within the innovation ecosystem. Future research could also investigate the impact of emerging technologies, such as artificial intelligence and blockchain, on the dynamics of collaboration between researchers and companies. These advancements have the potential to address communication challenges, ensure adherence to agreements, monitor progress and safeguard intellectual property. Another promising avenue would be the exploration of international comparative studies, which could reveal how different cultural and economic contexts influence the effectiveness of R&D collaborations. Additionally, implementing controlled experiments to test specific interventions, such as innovation management training programs and open innovation practices, could provide more robust evidence on how to optimize these interactions to maximize technological innovation outcomes. These lines of investigation would not only complement the findings of the present study but also provide a solid foundation for developing policies and practices that foster more effective and sustainable collaborations between academia and industry.