Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1 Context

Dutch is ranked as the 40th most widely spoken language of the world’s 6,000 languages. Most of the 23 million Dutch native speakers live in 2 neighbouring territories, the Netherlands and the Flemish region of Belgium, that have Dutch as an official language. As language policy decisions taken on one side of the national border affect citizens at the other side of the border, the Netherlands and Belgium created in 1980 the Dutch Language Union (Nederlandse Taalunie – NTU) as an intergovernmental organisation to conduct a joint language policy. The NTU’s mission is to deal with issues concerning the position of the Dutch language. It includes Dutch language and literature as a scientific subject, Dutch as a cultural language, Dutch as an administrative language, Dutch as a means of communication, and, more in general, the Dutch language as a tool for social interaction.

In an ICT based society a language needs a digital infrastructure (digital corpora and dictionaries, software and lingware modules, etc.) to maintain its position as a “used and useful” language and to avoid what is called an “electronic Gutenberg” effect. However, the market for human language technology for Dutch (HLTD) is seemingly too limited to attract important investments by industry in HLTD. As a consequence, the Flemish and Dutch governments decided in 2004 to spend 11.4 million euros to stimulate the HLTD sector (industry and academia) and thus strengthen the position of Dutch in the modern knowledge based society [11].

2 Historical Background

2.1 Researching Apart

HLT for Dutch started early. It began to flourish in the 1980s thanks to the Eurotra programme of the EC. Eurotra was intended as an EC research and technology development effort targeted at the development of a machine translation system for its internal use. For Dutch, it was mainly a collaboration between the universities of Leuven and Utrecht [18]. In parallel, some private companies also funded research on machine translation. Footnote 1 This initial wave slowly faded away in the beginning of the 1990s. In the Netherlands, the Dutch Organisation for Scientific Research (NWO) initiated some large scale programmes on HLT for Dutch. Footnote 2 In Flanders, no specific HLT research programmes were set up, except for the Flemish Research Initiative in Speech and Language Technology Footnote 3 that ran from 1994 till 1997. In Flanders research funding is mainly organised in a “bottom up” manner, not thematically or programmatically as is mainly the case in the Netherlands.

Of course, researchers in both Flanders and the Netherlands collaborated in cross-border projects, but this happened on a personal and ad hoc basis. In addition, researchers were not always aware of the availability of resources and tools for Dutch developed elsewhere. Systematically sharing and maintaining of resources hardly occurred. How the Eurotra software, which represented a research effort of more than a decade, fell into oblivion is an all too sad example. Clearly a coordinating platform or organisation was lacking.

2.2 Researching Apart Together

Things changed in the 1990s. The Flemish and Dutch governments became interested in HLTD at the start of the 1990s. They initiated research programmes, albeit still separate, and organised some exploratory policy studies. For example, in a large technology foresight exercise of 1998, Footnote 4 HLT was mentioned as a potentially strategic technology domain for the economy of the Netherlands.

In the mid 1990s, the EC sponsored the Euromap Language Technologies project (1996–2003). Euromap aimed at accelerating awareness of the benefits of HLT enabled systems, services and applications within user sectors, policy makers and national administrations and bridge-building and market-enabling services to stimulate market take-up of HLT RTD projects’ results. The Euromap project wanted to determine the status of HLT for the various languages in the participating countries. For each participating territory, a national profile was made as well as a national policy review [13]. In addition, directories of all research groups and companies active in the field of HLT of a country were published – e.g., cf. [10] for the Flemish directory. Initially, the Flemish and Dutch administrations participated in Euromap as separate partners.

The objectives of Euromap ran partly in parallel with the ambitions of the NTU that prepared and published in 1998 a study on the status of Dutch in speech and language technology[ 6]. Footnote 5 The fact that the NTU became the “national” focal point for the entire Dutch language, representing both Flanders and the Netherlands in the second phase of the Euromap project, gave a boost to the implementation of some of the recommendations of this study. The national seminars on various HLT related subjects organised in the framework of Euromap, for example largely contributed to network building and laid the foundation for the future cooperation between academia, industry and governments in the Netherlands and Flanders.

In addition, as these were the booming years of Lernout&Hauspie Speech Products in Flanders, Footnote 6 HLT became very prominent on the public forum in the Low Countries. The study and these (economic) circumstances made the NTU – and the Dutch and Flemish governments – realise the importance of a digital language infrastructure for the Dutch language. At that time such an infrastructure was largely lacking. As a result an HLT for Dutch Platform (HLT Platform) in which the relevant government departments and agencies were represented, was installed in 1999 [2]. The goals of the HLT Platform, which constituted a forum for information exchange, agenda adjusting and joint activities, were:

  • To promote the position of the Dutch language in HLT developments, so that the Dutch language could become and remain a “first class citizen” language within a multilingual European information society;

  • To establish the proper conditions for a successful management and maintenance of basic HLT resources developed with governmental funding;

  • To promote and stimulate the collaboration between the research community and the business community in the field of HLT;

  • To contribute to European collaboration in HLT-relevant areas;

  • To establish a network, both electronic and personal, that brings together demand and supply of knowledge, products and services.

In parallel, the NTU took on the challenge to coordinate the development of high quality resources needed for automated translation from and into Dutch for the Systran translation system. This was the TransLex project [12], funded by the EU MLIS programme with additional contributions by the Flemish and Dutch governments together with the private partner Systran and the translation service of the EC.

In 1998, the construction of a Spoken Corpus for Dutch (CGN)[ 17] started. Again, Flemish and Dutch governments have jointly financed the project. The NTU received the ownership of the corpus and became responsible for its maintenance and exploitation. However, the NTU did not assume any central role in the process. Note that, from the governance point of view, only the CGN board (of funding organisations) and the scientific steering group were organised as a joint endeavour. All other (practical) matters (set-up, funding etc.) were organised separately in Flanders and the Netherlands. The CGN scientific steering group ensured that scientific activities remained compatible (common formats, protocols, tools etc.).

2.3 Researching and Developing Together

The NTU published in 1999 together with the HLT Platform an “action plan for Dutch in speech and language technology”. Four major action lines were defined:

  • Action line A: setting up an information brokering service;

  • Action line B: strengthening the digital language infrastructure;

  • Action line C: defining standards and evaluation criteria;

  • Action line D: developing a management, maintenance and distribution plan.

Several working groups, consisting of researchers from academia and industry, started to write specific plans on how to accomplish these four action lines. Action line A has been taken up by the NTU and resulted in the creation of the HLT Info desk. Footnote 7 The HLT Info desk publishes a newsletter, maintains a website with an overview of HLTD related organisations (academia, industry and government) and HLTD events in Flanders and the Netherlands.

Action line B has eventually materialised in an HLTD R&D programme. Extensive preparatory activities paved the way for this programme. Field surveys resulted in the description of a basic language resource kit (BLARK) for Dutch. A BLARK is defined as the set of basic HLT resources that should be available for both academia and industry [15]. Not only were all the materials (data, modules and tools) available (or at least identified) at that moment listed, but also “missing links” ’ were identified and included in the overview. Prioritisation exercises, including discussions and meetings with the entire HLTD field, led to ranked lists of R&D topics [4, 8, 23]. A longer term road map was sketched [3].

In addition, the Dutch Ministry of Economic Affairs has ordered a specific HL technology forecast to estimate the economic value and potential of HLTD and to determine the ideal government intervention logic [1]. Eventually, a proposal for a joint Flemish-Dutch R&D programme was drafted. The proposal was baptised STEVIN (Essential Resources for Speech and Language Technology for Dutch). The proposal was accepted and STEVIN started in September 2004.

Activities on action line C have been combined with action line B: determining whether materials are available could not be done without a quality evaluation. However, actual evaluation criteria or benchmarks have not been developed – except as (parts of) projects in the STEVIN-programme (e.g., the NBest project [14] – cf. Chap. 15, p. 271).

The working group for action line D has delivered a blueprint for management, maintenance, and distribution of publicly funded HLT resources that eventually resulted in the creation of the HLT Agency for Dutch by the NTU [3]. ELDA and LDC served as examples. This agency acts as a “one-stop-shop for HLTD” and takes care of maintaining, distributing and promoting HLT for Dutch project results (corpora, tools, dictionaries etc.) [25] – cf. Chap. 21, p. 381.

3 The STEVIN Programme

3.1 In a Nutshell

In line with the action plan of the HLT Platform (cf. Sect. 2.2.3 ), the STEVIN-programme aimed to contribute to the progress of human language technology for Dutch (HLTD) in Flanders and the Netherlands and to stimulate innovation in this sector. In addition, it aimed to strengthen the economic and cultural position of the Dutch language in the modern ICT-based society. The mission of the programme was translated into three specific main goals:

  1. 1.

    Build an effective digital language infrastructure for Dutch, based on the BLARK priorities for Dutch;

  2. 2.

    Carry out strategic research in the field of language and speech technology, especially in areas of high demand for specific applications and technologies;

  3. 3.

    Advance the creation of networks and the consolidation of language and speech technology activities, educate new experts, stimulate the demand for HLT products.

The STEVIN HLT programme was comprehensive in many respects. First of all, because it was based on co-operation between government, academia and industry both in Flanders and the Netherlands. For example, projects with partners from both Flanders and the Netherlands were encouraged. Co-operation saves money and effort by avoiding duplication of activities and enhances scientific excellence thanks to an increased competition. Secondly, the programme encompassed the whole range from basic resources to applications for language users. For example, application oriented projects were encouraged to build upon results of the resource oriented projects (cf. Chap. 1, Sect. 1.3.2 p. 12). And thirdly, it paid attention to the distribution, dissemination and valorisation of project results by means of the HLT Agency (cf. Chap. 21, p. 381). To ease the distribution of the resulting resources and tools, the HLT Platform stipulated the obligation to transfer the ownership of the foreground results (i.e. material made in the course of a STEVIN project) to the NTU, which is rather uncommon. The important task of clearing the IPR and issuing licence agreements for further use was delegated to the HLT Agency, which in turn received some extra funding from the STEVIN budget.

3.2 Governance

Various committees were set up around the STEVIN programme, as Fig. 2.1 shows. The HLT Platform (cf. Sect. 2.2.2) became the HLT board supervising STEVIN. It consisted of the NTU and the funding bodies. Footnote 8 The NTU was the overall coordinating instance.

Fig. 2.1
figure 1figure 1

Main goals of STEVIN and distributed responsibilities

Full size image

A programme committee – PC, consisting of both academic and industrial representatives, was responsible for all scientific and content related issues. It consisted of local Flemish and Dutch HLT experts who wrote a detailed multi-annual research programme (topics, expected outcomes, instruments, timing, …). The PC defined the various calls for project proposals. An international assessment panel (IAP) of eight highly respected HLT-experts evaluated the submitted R&D project proposals. The PC added a “local check” to the assessment of the IAP. Divergences of opinion between the IAP and the PC were rare and of minor importance. Several calls (three open calls and two calls for tender) have been issued over time.

Next to the R&D projects, which were supposed to achieve the first two main goals of STEVIN mentioned above, some other (smaller sized) actions were initiated by the “accompanying activities” working group :

  • Demonstration projects had to increase the demand for HLT technology by using proven HLT technologies to build highly visible close-to-market applications;

  • Educational projects aimed at sensitising young students within educational settings (school, museums, etc.) for the possibilities of language and speech technologies;

  • Master class projects targeted high level decision makers within government organisations and the industry to familiarise them with the opportunities offered by HLT.

STEVIN has awarded 19 R&D projects (in total 8.909 K euros), 14 demonstrator projects (1.011 K euros), 3 educational projects (100 K euros), 2 master classes (33 K euros) and 31 networking grants (45 K euros in total). The acceptance rate for the R&D projects was between 26 and 33 %. In the Low Countries, most of the funding agencies consider an acceptance rate of around 30 % sufficiently selective to guarantee scientific excellence and high enough to fund (almost) all the best proposals.

A programme office, a joint collaboration of the Netherlands Organisation for Scientific Research and the Dutch innovation agency called Agency NL, took care of the operational matters, such as the practical organisation of the calls (submission site, related documents etc.)

An important committee was the IPR working group that defined the licence templates. These licences settled the ownership transfer of the foreground results to the NTU, the conditions under which third parties agreed to make their resources available for academia (and if possible also for industry), the permissions for STEVIN researchers to continue to work on their material and the terms of usage for third parties to use the STEVIN resources and tools. As a result, the IPR of all the STEVIN material has been legally cleared, which opened a wide range of different possibilities to distribute and exploit the material. This task, managing the IPR of the STEVIN results, was delegated by the NTU to the HLT Agency [25] – cf. Chap.21, p. 381.

3.3 Monitoring and Evaluation

Some time after the start of the STEVIN programme a baseline was defined [1]. It encompassed the then current state of HLT for Dutch in terms of number of researchers active, turn-over of HLT companies, degree of academia-industry cooperation, degree of Flemish-Dutch cooperation etc. in the spirit of the Euromap country reports ([ 13] – cf. Sect. 2.2.2 ). This base line served as reference point for the final evaluation to determine to which extent the STEVIN programme had a positive impact on HLTD in Flanders and the Netherlands. During the programme, a light weight monitoring process at project level was organised. Each project had to organise two site visits during which two members of the PC attended presentations on the project’s progress and achievements. The members of the PC studied the reports, gave suggestions and made critical remarks – if needed. Additionally, the projects, if appropriate, had to organise an external validation exercise or deliver some “circumstantial evidence” of a quality control check (e.g., a test report by a research group not belonging to the consortium that had used the resource concerned).

Half way through the programme, a scientific mid term evaluation by the IAP was organised to see if the entire programme was on track and if any adjustments had to be made [22]. In addition, the PC made a self evaluation report. A.o. the IAP felt that STEVIN material was worthy of more high profile scientific publications (the projects and the programme in its entirety) than was the case at that moment. Another matter of concern was the lack of projects in the multimodal and/or multimedia and semantic domains. But all in all, the IAP in its report Footnote 9 congratulated the HLTD community in the Low Lands on their achievements within the STEVIN programme [19].

The final evaluation was concluded before the actual end of the programme. As a consequence, some projects still produced an important number of publications that were not taken into account. An important advantage would have been that a smooth continuation had been enabled as the funding authorities already had almost all the necessary information available to decide on follow-up activities before the actual end of the programme. Unfortunately, the aftermath of the 2008 financial crisis decided otherwise.

3.4 The Final Evaluation

The final evaluation of the STEVIN programme was an overall evaluation. Not only the scientific issues but also the governance and economic aspects of the programme were taken into account. A small ad hoc committee did the preparatory work, largely inspired by the evaluation framework and practices of the Flemish Department of Economy, Science and Innovation.

3.4.1 Evaluation Assignment

The HLT board formulated a set of evaluation questions, which can be grouped into four major categories.

  • Efficiency: Were the resources properly and adequately used? Was the management of the programme efficient? and Was the programme adequately monitored?

  • Effectiveness: Did the programme achieve its targets? Was the programme effectively organised? Did the programme influence the policy agenda in Flanders and The Netherlands?

  • Usefulness: Were the problems in the HLT domain identified at the start of the programme successfully addressed? Was there an overlap with other activities/efforts? and Which role did STEVIN play in the HLT field, both nationally and internationally?

  • Relevance: To what extent did STEVIN lead to usable material for the HLT field and user groups? To what extent technological and scientific progress in the HLT field did evolve thanks to STEVIN ? and What was the added value of STEVIN?

These evaluation questions were grouped around the major issues at play in the STEVIN-programme:

  • Governance and management of the programme;

  • Application and selection process of project proposals;

  • Effects and impacts of the programme;

  • Positioning of the STEVIN programme with respect to other programmes;

  • Future of the programme.

In order to obtain an objective evaluation, a call for tender was issued and an external consultant (c.q. the Technopolis group Footnote 10) [9] was selected to perform the evaluation. The same questions were addressed by the PC as well in their self assessment report. Footnote 11

3.4.2 Methodology

A combination of both quantitative and qualitative research methods was employed [9]. During a desk research phase, the consultant analysed all relevant documents (the STEVIN multi-annual work plan, yearly reports, the baseline and mid-term reports, self assessments reports, fact files, meeting minutes of the various committees, call for proposal texts, etc.). The STEVIN programme was also compared with some other (foreign) programmes – in particular concerning the governance structure, the financial management and selection, evaluation and monitoring procedures. An international expert panel (other than the STEVIN IAP mentioned earlier) assessed the scientific output of the programme. In total, 127 relevant contacts were invited to participate in two online surveys (cf. Sect. 2.3.4.3 ). A network analysis was used to map the various co-operation relationships within the STEVIN-programme. Finally, 23 interviews were held that involved the most important STEVIN stakeholders. More details can be found in [21].

3.4.3 Evaluation Outcomes

Of course it is impossible to report here Footnote 12 on all aspects of the STEVIN final evaluation. We limit ourselves to summarising the assessment of the three main goals of the programme (cf. Sect. 2.3.1 ), as well as giving some general comments of the international experts that concern the entire programme. In the subsequent Sect. 2.3.4.3, ten recommendations by the external evaluator are presented. We refer the reader to the chapter on the HLT Agency (cf. Chap. 21, p. 381) for more details on IPR and licence agreement management, and other issues on maintenance, distribution, promotion and utilisation of the STEVIN results. For a more detailed account on the scientific achievements, the reader is referred to the individual chapters on the various STEVIN projects in this volume. The introductory chapter (cf. Chap. 1, p. 1) gives a more global view on the entire programme, while overall conclusions and forward looking statements are provided in Chap. 22, p. 395.

3.4.3.1 Overall Assessment

Figures 2.2 – 2.4 summarise the scores of the two online surveys. The first survey addressed academics in Flanders and the Netherlands, c.q. successful and unsuccessful submitters of STEVIN proposals. Research institutes that had participated in the baseline survey, even without any link with STEVIN, were also invited. Sixty-two research institutes were contacted, of which 56.5 % responded. The second survey concerned the Flemish and Dutch HLT industry. Again, applicants for funding (granted or not) were invited as well as companies that had participated in the baseline survey. Sixty-five companies were contacted with a response rate of 43.2 %. The responses may thus safely be assumed to be representative (overall response rate of 49.6 %).

Fig. 2.2
figure 2figure 2

Reported degree of achieved success for STEVIN main goal 1: setting up the BLARK for Dutch

Full size image
Fig. 2.3
figure 3figure 3

Reported degree of achieved success for STEVIN main goal 2: performing strategic HLTD research

Full size image
Fig. 2.4
figure 4figure 4

Reported degree of achieved success for STEVIN main goal 3: HLTD networking and demand stimulation

Full size image

Applicants for STEVIN funding were asked to rate (on a scale of 1–10) the achievements and “mechanics” of the STEVIN programme – e.g., statements on the transparency of the decision process, the quality of communication, the expectations towards the programme etc. Participants in the baseline survey had to provide data and information on the status of HLT in their organisation – e.g., the number of HLT related staff, HLT turn-over (if applicable), expenditures in HLT R&D etc. A comparison between the situation described in the baseline report and the new situation should allow to assess the impact of the STEVIN programme on the domain. Due to space limitations, we cannot discuss the comparison in this volume.

Figure 2.2 shows that the participants of the STEVIN programme agreed that STEVIN largely succeeded in setting up a digital language infrastructure for Dutch, i.e. creating many of the missing building blocks of the BLARK for Dutch. The overall score is 6.6. Representatives of academia considered this mission as accomplished in a slightly higher degree than companies. And Flemish respondents are slightly more positive than respondents from the Netherlands.

Figure 2.3 reflects how the participants assessed the opportunity offered by STEVIN to perform strategic basic research ( = second main goal of STEVIN). Again, the overall score is 6.6. Academics are slightly less positive (in particular Flemish academics: 5.8). Again, the international experts involved in the evaluation pointed out (as did the IAP members during the mid-term review) that a programme as STEVIN should generate more international high profile publications. Nevertheless, they concluded that many STEVIN deliverables, even if not always cutting edge, were highly important to set up the BLARK for Dutch.

Even if a too low number of high profile scientific publications seems a justified point of critique, one has to take into account that creating elements of a digital language infrastructure does not necessarily imply performing cutting edge research – in some cases, it is rather the contrary. And in all fairness, it should be mentioned that around 55 new papers appeared Footnote 13 after the delivery of the evaluation report. In total, around 200 official STEVIN publications were published. Application oriented projects resulted in more higher impact publications than the resource producing projects.

Many of the corpora, tools, protocols, databases, etc. resulting from STEVIN still are, to an important degree, instrumental for the Flemish and Dutch local CLARIN counterparts of the European CLARIN project [26]. Footnote 14 In addition, some of the STEVIN results are not only important for Dutch, but do also have an impact (practical and theoretical) on research on other languages. For example, the STEVINcanPRAAT project (cf. Chap. 5, p. 79) improved the PRAAT tool [5] that is widely used by an international community.

Figure 2.4 shows that the stakeholders positively judged the impact of STEVIN on the strengthening of networks and co-operations in the Low Countries ( = third main goal of the programme). Even if participants knew each other before their involvement in STEVIN, STEVIN offered the opportunity to actually co-operate. In addition, the intensity of co-operation between Flemish and Dutch parties and between academia and industry increased thanks to STEVIN. The fact that STEVIN created a unique opportunity for all HLT subdisciplines (speech, language, information extraction, dialogue, …) was well appreciated. Flemish and Dutch respondents reacted alike. Respondents from academia (7.4) are more positive than respondents from the private sector (6.9). Jointly performing research and exchanging knowledge were the two most cited types of co-operation.

Overall, researchers apparently felt the need to set up the BLARK for Dutch more strongly than industry, and hence were more happy with the results. Companies were more interested in performing strategic research, acquiring and integrating new technology to improve their products, while researchers preferred to perform more basic research and publish papers instead. Usually academia is more open to co-operation than industry. These statements are confirmed by other findings (not mentioned here) in the survey.

The external evaluator compared STEVIN with a few other international R&D programmes. Footnote 15 The comparison showed that STEVIN is quite unique in its cross-border co-operation (including cross-border funding). STEVIN can be seen as a precursor of the “joint programming” approach of the EC [20]. Its comprehensive approach (ranging from strategic research to demonstrator projects and sensitisation and promotional activities) and the involvement of various ministries and agencies at both sides of the border was rather rare. As such, the programme has achieved its objectives and is regarded by the evaluators as a successful programme.

3.4.3.2 Evaluation Recommendations

An evaluation of an R&D programme focusses not only on ex-post reflections (on how things have happened) but also includes ex-ante suggestions for post STEVIN activities. The external evaluator has condensed his findings in ten recommendations Footnote 16 :

  1. 1.

    The integrated approach of STEVIN was a good approach and should be replicated in a potential follow-up of STEVIN. The focus should then be shifted from the BLARK and strategic research to application-oriented research and demonstration projects. It is important to balance between the different types of research. In the design of the programme, multiple modalities should be possible: basic research combined with more application-oriented research and projects aimed at either strategic or application-oriented research. Maybe less of a priority, but still important are projects aimed at basic language infrastructure.

  2. 2.

    STEVIN is an example of transnational cooperation through “joint programming” that has value for both funders and performers. A possible follow-up to STEVIN should also have a bilateral structure with a “common pot”.

  3. 3.

    The main structure of governance does not need to be adjusted. However, the tasks and responsibilities should be defined more precisely, so that it is clear to everyone what the tasks and roles of the various organisations involved are.

  4. 4.

    The programme office needs to be positioned more closely to the NTU. This could be done by means of a secondment to the NTU from various organisations.

  5. 5.

    The programme office should also be more balanced, in the sense that there is a better Dutch-Flanders balance in governance structure.

  6. 6.

    In general, partly dependent on the focus of a follow-up programme, the composition of different committees and commissions should be reviewed. If its focus is to be more on the application of HLT-knowledge in practice, representation of industry and applicators should be enforced.

  7. 7.

    IPR issues, including how to deal with open source, should be addressed before the start of a follow-up programme. Rules regarding IPR should be clearly defined and availability of standard contracts, etc. should also be taken into consideration. The preparations can build on the work of the IPR Working Group and the experiences of the HLT Agency.

  8. 8.

    A more active collaboration with related programmes at the national level, and at European level is needed in the follow-up programme. In addition, it is to be considered whether a junction is possible with social innovation programmes in the fields of education, care, and safety.

  9. 9.

    If strategic research obtains an important role in a follow-up programme, there should be a greater emphasis on publications in international journals and at international summits.

  10. 10.

    Consider dedicating part of the budget to an international publication in which the results of the STEVIN programme are presented in conjunction. Footnote 17

4 Discussion

In this section, we briefly treat two governing principles that are seemingly very typical of the STEVIN programme. In fact, it is rather uncommon for researchers to have to transfer the ownership of their research results to a governmental organisation (cf. Sect. 2.4.2) and to be funded according to the actual delivery of results specified on beforehand instead of on the basis of a predefined amount of time and resources (cf. Sect. 2.4.1 ).

4.1 Delivering Results

Most of the contracts between a funding agency and research institutes are based on an obligation to perform to the best of one’s abilities ( = a commitment by a researcher to use as well as possible the given means to investigate a topic without any guarantee on success). STEVIN contracts however were based on an obligation to achieve results ( = a commitment by a researcher to deliver well specified results). As the aim of STEVIN was to create a digital language infrastructure, the funding organisations did expect a finalised corpus or a properly working tool to be actually delivered at the end of a project. An obligation of means was considered as an insufficient guarantee for actual delivery. Some university administrations of the participating research groups initially were not so keen of a contract based on an obligation of results. But the universities benefitted from a reduced administrative overhead imposed by STEVIN: the result counted, not how the means were spent (human resources, equipment, …). This implied that if a satisfactory result was delivered using less funding, the researchers could keep the difference. Footnote 18 The challenge for the programme governance was then to determine and monitor the quality of the results – cf. Sect. 2.3.3 .

4.2 Managing IPR

As mentioned earlier, the ownership of all the STEVIN results (or foreground knowledge), except for the few open source exceptions, eventually went to the NTU. Where needed, extra agreements were concluded regarding background knowledge. The main idea was that all STEVIN results had to be made available and re-usable for academia and industry in the Low Countries. Centralising the ownership within one organisation (which has as its mission to promote and support the Dutch language on behalf of the two funding governments) was seen by the HLT board a good guarantee that STEVIN results would become easily accessible and available for wider distribution. Therefore, an dedicated agency for resource maintenance, distribution and promotion, c.q. the HLT Agency, was created earlier on (cf. Chap. 21, p. 381).

Since it was quite unusual for researchers to have to transfer the ownership of their foreground results, some voiced their discontentment and preference for open source variants, in particular when software Footnote 19 was involved. Even if at the start of STEVIN, IPR problems did arise as the HLT board and the programme committee had seriously underestimated the complexity of IPR issues, at the end of STEVIN it became clear that, on the international level, infrastructure projects, resource organisations and even policy makers look enviously at STEVIN as all the IPR rights were legally conclusively cleared (including material of privately hold editing houses), template licence agreements were drafted and ownership was centralised. It still happens all too often that researchers are allowed to use material from commercial third parties only for the specific purpose and duration of a single project. In the case of STEVIN, this limitation, to a very large extent, does not apply thanks to (sometimes time consuming) negotiations and solid agreements with providers. The HLT Agency is now responsible for maintaining and distributing the STEVIN materials and for concluding licence agreements on behalf of the NTU.

From [16], it appears that e.g., open source licences may end up less attractive and more complex for resource distributing agencies or initiatives than initially foreseen. In any case, as the NTU is the proprietor of the STEVIN results, except for some open source materials, any possible way of distribution and exploitation can be applied. For example, a synonym list resulting from the Cornetto (cf. Chap. 10, p. 165) project could be released as open source to be included in the Dutch “language pack” for Firefox, OpenOffice and Chrome distributed by the OpenTaal organisation. Footnote 20 For companies interested in using the Autonomata grapheme-to-phoneme converter, a royalty scheme was set up.

5 Conclusion

According to Technopolis group, the general targets of the STEVIN programme have been reached to a (very) large extent. As a sufficient number of high quality project proposals was funded by STEVIN, a substantial strengthening of the digital language infrastructure for Dutch was achieved. The quality of research within STEVIN was, in general, good, albeit not cutting edge. This can be attributed to the nature of the projects (in particular when addressing the BLARK for Dutch) being less apt for high impact publications. Another strong point of STEVIN was the funding of application oriented projects as these demonstrate the potentialities of HLTD to industry and the general public. It resulted in a network with strong ties between academia and industry that is beneficial for future utilisation of the STEVIN results. Some adaptations in the programme governance structure, more interaction with other similar (inter)national R&D programmes, and a better clarification of the role of open source were recommended by the evaluators for a future programme. All in all, they qualify STEVIN as a successful cross-border R&D programme.

Technopolis group recommends to organise a follow-up programme again as a combination of different types of R&D within the same programme: even if focusing more on application-oriented research and demonstrator projects (and thus strengthening the participation of industry and software integrators in the programme), other types of research (e.g., basic research) should not be overlooked. Health care, education, e-government, safety and cultural heritage are cited as potentially interesting application domains for follow-up R&D activities.

According to the evaluator, the Dutch-Flemish HLT community has been able to retain their top position in the international HLT community thanks to STEVIN, which prepared them for a leading position in the European CLARIN endeavour.