We believe there are three main barriers that have prevented researchers and developers from pursuing such studies. The first pertains to funding. National funding agencies seldom support studies that extend beyond a few years. The European Commission has funded large networks of excellence (see for example http://www.teleurope.eu/pg/frontpage and http://www.galanoe.eu/) but these only last 4 years, and none have yet to result in systemic change and large-scale impact of the type mentioned previously. The American National Science Foundation also funds centers of excellence, and these are expected to continue beyond their original funding (see for example https://www.nsf.gov/about/budget/fy2012/pdf/39_fy2012.pdf and http://www.op-tec.org/nsfate.php). Of all such large-scale, nationally funded efforts, perhaps the one that has achieved long-term impact is the National Center for Research on Evaluation Standards, & Student Testing (CRESST; see https://www.cse.ucla.edu/index.php), but that prestigious and successful national center has not led to large-scale, systemic reform in educational practice although its influence on educational research is clearly positive. National funding agencies need to promote more types of long-term efforts to achieve systemic and sustainable reform in educational practice.
The second barrier concerns the willingness of researchers and developers to conduct replication studies and meta-analyzes, and find funding for large-scale efforts. Researchers have a tendency to want to create their own instruments and engage in an effort that is closely tailored to their personal interests. While such efforts are laudable, they do not lead to progressive improvement of educational technology research as a science. One-off studies may get published and even lead to some notoriety of the investigators. However, they often fail to result in systemic and systematic improvement of educational practice. There is clearly a disconnect between what is personally valued by educational researchers and what is needed to improve learning and instructional practice. Because publication in high-quality refereed journals is typically required for promotion and tenure, ETR&D encourages replication studies, meta-analyses, and large-scale reports of impact on education to be submitted for publication.
The third barrier concerns the willingness of researchers to openly and freely share instruments and details of their research procedures with others. There are a few exceptions when this has been done—notably, by the American National Science Foundation’s ITEST (Innovative Technology Experiences for Students and Teachers; see the most recent NSF ITEST description at http://www.nsf.gov/funding/pgm_summ.jsp?pims_id=5467). However, too many researchers want to retain full rights and ownership of their instruments and all too often charge those who wish to use those instruments a fee that is prohibitive for graduate students and junior faculty.
It is our hope that project investigators and senior faculty will encourage meta-analyses, replication studies and large-scale impact reports so that our enterprise can become more scientific and provide a basis for transforming educational practice. By welcoming these studies, we aim to fulfill the overarching aims of ETR&D, which includes publishing quality research and development findings that will improve learning and instruction around the world.