Developing Education Research Competencies in Mathematics Teachers Through TRAIL: Teacher-Researcher Alliance for Investigating Learning
- 138 Downloads
This theoretical article explores an issue of developing education research competencies in mathematics teachers through their involvement in mathematics education research. We first argue that the development of education research competencies is beneficial for the teachers’ professional growth. We then identify opportunities for mathematics teachers to develop education research competencies through different modes of research-practice partnerships. In the main part of the paper, we present a particular theoretical-organizational framework for large-scope teacher-researcher collaborations in educational research. The framework is called Teacher-Researcher Alliance for Investigating Learning (TRAIL), and consists of a set of theoretically laden premises, design heuristics, and provisional partnerships.
KeywordsResearch competencies Research-practice partnerships Teacher research Teacher professional growth Citizen science
Cet article théorique analyse un aspect du perfectionnement des compétences de recherche en didactique chez les enseignants des mathématiques, en se penchant sur leur engagement dans la recherche en enseignement des mathématiques. D’abord, nous posons comme principe que le perfectionnement des compétences dans les domaines de recherche en enseignement est. bénéfique pour la croissance professionnelle des enseignants. Ensuite, nous décrivons des situations grâce auxquelles les enseignants de mathématiques peuvent acquérir ou perfectionner certaines compétences en recherche par le biais de différents types de partenariats de recherche-pratique. Dans le corps de l’article, nous présentons un cadre théorique-organisationnel favorisant la collaboration entre chercheurs et enseignants dans des projets de recherche didactique de grande envergure. Ce cadre, appelé Alliance chercheurs-enseignants pour la recherche en apprentissage (« TRAIL », soit Teacher-Researcher Alliance for Investigating Learning), consiste en une série de prémisses théoriques, d’heuristiques conceptuelles et de partenariats provisionnels.
Educational systems in many countries have accredited programs for in-service mathematics teachers that include studying and practicing educational research towards a graduate degree or a certificate. Positive effects of such academic programs on mathematics teacher professional growth are pointed out by many scholars (e.g., Even 2003; Jaworski 2003; Taylor 2017). Yet, graduate studies are a time- and effort-consuming enterprise, which suits the aspirations of some, but is not the choice of many practicing mathematics teachers.
Many teachers, however, have a tendency to engage themselves in different forms of educational inquiry and act as informal researchers of their own practice (Taylor 2017). Taylor (2017) calls for the development of tools that can support such teachers and enhance their involvement in educational research beyond a teacher education classroom or a formal coursework. This paper is stimulated by this call. It is also stimulated by the ongoing effort to situate teachers as stakeholders in mathematics education research, either for the sake of their professional development or for the sake of enhancing the extent to which mathematics education research can influence practice (e.g., Krainer 2014; Kieran et al. 2013).
Our first goal in this paper is to theoretically substantiate the following claim: the development of research competencies in mathematics education is pertinent not only to the training and professional growth of mathematics education researchers but also to the professional growth of mathematics teachers. Our second goal is to identify opportunities for mathematics teachers to develop education research competencies through different modes of research-practice partnerships. Our third and main goal is to present a particular theoretical-organizational framework for large-scope teacher-researcher collaborations in educational research. The intended framework is called Teacher-Researcher Alliance for Investigating Learning (TRAIL). It draws on theoretical constructs and ideas developed in three bodies of the professional literature: the literature on mathematics teacher inquiry, the literature on modes of research-practice partnership in education, and the literature on Citizen Science. We demonstrate this framework through several provisional examples.
Teachers and Education Researchers as Professional Inquirers
Both teachers and education researchers spend a substantial amount of their time engaged in various forms of inquiry about teaching and learning. They listen to students and examine their work and try to assess what students know and how they think; they examine teaching materials and explore their pedagogical affordances; they design tasks and teaching sequences and examine how different students engage with them; they try and compare different teaching approaches; they read, and think, and talk to their colleagues about education, and about how to improve it. By all accounts, both teachers and education researchers develop over time competencies and knowledge that support these professional activities (Leikin and Zazkis 2010; Boaler et al. 2003). On the basis of this observation, and while acknowledging that there are profound differences between the world of education researchers and the world of teachers, we substantiate in this section the claim that there is an important overlap between the competencies and knowledge required for being proficient in these two communities of practice.
Much empirical and theoretically-based knowledge is produced by the scientific community and much, mostly unpublished, knowledge is produced by the rich experiences of thousands of teachers. (Kieran et al. 2013, p. 363)
Before turning to explore this overlap and its implications to teacher professional growth, it is important to underline the different nature of inquiry in the communities of research and teaching. The studies conducted in scientific communities are often referred to as “disciplined inquiries” (Cronbach and Suppes 1969; Kilpatrick 1981; Shulman 1981). Disciplined inquiries are characterized, among other things, by their theoretical purposes (Kilpatrick 1981; Schoenfeld 2000), systematic and transparent analytical methods (Shulman 1981), and shared core practices (Boaler et al.2003). According to Labaree (2003), the inquiries teachers engage with routinely in their course of their professional work differ from disciplined inquiries in that they are normative rather than analytical, personal rather than intellectual, particular rather than universal, and experiential rather than theoretical. Scholars (Kontorovich and Rouleau 2018; Krainer 2014; Labaree 2003; Nardi 2015) have pointed to these differences as a source for possible tensions, which may hinder collaboration between teachers and researchers, and reduce the impact of research on practice.
However, cultural differences between teachers’ and researchers’ worlds can also be viewed as opportunities for teacher learning and professional growth (Jaworski 2003; Kieran et al. 2013). We elaborate on this stance in the remainder of this section. We first discuss how different modes of teacher inquiry can inform and guide teaching and support professional growth. We then discuss what teachers can take from disciplined inquiry.
How Teacher Inquiry Informs Teaching and Supports Professional Growth
It is widely accepted that teaching is an active process that relies on strategic enactment of knowledge, in context, often in the moment. As such, teaching has a lot of room for inquiry that informs and guides pedagogical decisions. Over the past few decades, teacher inquiry has also been considered a powerful tool for professional growth. Researchers and teacher educators have advocated for professional development programs that encourage and support teacher inquiry into, and reflection on, their practice (Ball and Cohen 1999; Crawford and Adler 1996; Jaworski 2003; Kieran et al. 2013; Mason 2002; Taylor 2017).
Much of what mathematics teachers do every day comes close to being research; it is just not quite so deliberate, systematic or reflective. (Kilpatrick 1981, p. 26)
Some scholars have gone as far as to regard teaching as a contextual application of inquiry and problem-solving (Stylianides and Stylianides 2010; Watson and Barton 2011). Accordingly, teachers have been regarded in the literature as reflective practitioners (Schön 1983), inquirers (Day 1999; Ellis 1998), and even researchers (Crawford and Adler 1996). Menter et al. (2011) define practitioner research in education as a “systematic inquiry in an educational setting carried out by someone working in that setting, the outcomes of which are shared with other practitioners” (p. 3). Inquiry, for this purpose, is defined as “trying to develop some new knowledge or understanding” (ibid.).
The mathematics education literature includes discussions of many forms of teacher inquiry. Some forms of teacher inquiry are strongly linked to well-documented practices of mathematicians. For example, Watson and Barton (2011) derived a list of mathematical modes of inquiry by examining how teachers “act mathematically” while planning for instruction. Their analysis of the teachers’ work was informed and guided by the literature on mathematicians’ practices and habits of mind, including identifying implicit variables, connections, structures, or patterns; purposefully playing with mathematical ideas and noticing what changes and what stays invariant; considering different representations; and asking “what if” questions, making conjectures and testing them. Watson and Barton concluded that mathematical modes of inquiry are central for competent mathematics teaching, and hypothesized that providing teachers with opportunities to engage in and reflect on different mathematical modes of inquiry would result in better mathematics learning for their students.
Competences of mathematical inquiry were also found to be linked with teachers’ ability to capitalize on contingent learning opportunities that arise in lessons. Rowland et al. (2011) found that the great majority of contingent moments is triggered by unexpected contributions by students, and that teachers’ responses in these situations are typically to ignore or to acknowledge the contribution but put it aside. Rowland and Zazkis (2013) exemplified in three cases how knowledge associated with mathematical inquiry informed and guided teachers and enabled them to capitalize on students’ unexpected ideas. They observed that the teachers’ actions in these cases did not rely on familiarity with university-level content in specific curricular topics, but rather on the teachers’ readiness, willingness, and commitment to inquiry—to making conjectures, experiment, and taking risks—when contingent opportunities arise. Rowland and Zazkis concluded that teachers’ ability to capitalize on contingent situations is underpinned by strategic enactment of knowledge about the nature and mechanisms of mathematical inquiry. They contended that teachers are most likely to develop this kind of knowledge through extended exposure to and engagement with mathematical inquiry.
However, mathematical inquiries support only some aspects of the know-how required for acting in the moment, and capitalizing on contingent situations in the classrooms (Mason 2002; Mason and Spence 1999). According to Mason, knowing to act in the moment depends on a teacher’s modes of attention in the moment, and on his or her network of connections between opportunities and actions. Mason suggests several practices of inquiry that can support teachers’ professional growth by helping them develop the skills of noticing and of acting in the moment, most notably, reflection and labeling. According to Mason, reflective inquiry is the active re-vivifying of recent incidents or moments from the lessons, coupled with actively imagining alternative reactions to similar situations in the future. Mason suggests that through reflective inquiry, teachers inform their future practice by formulating an array of alternative viable responses to contingent situations, and the sensitivity to noticing opportunities in which to try them. Labeling is the formulation of a succinct collection of words, a conceptual framework, which acts as an axis around which experiences can be accumulated. For example, labels can be used to frame experiences as “seeing the general through the particular,” or “seeing the particular in the general,” which in turn can help teachers make better sense of a contingent situation, and trigger a repertoire of alternative responses.
To summarize, the reviewed above literature suggests that teacher inquiry is central in the world of the teacher and that the issue of developing research competencies is pertinent for teacher professional growth. Further, we learn from the literature that the term teacher inquiry is used broadly, so that it embraces mathematical modes of inquiry, practitioner educational research, and also forms of inquiry that teachers engage in their daily work, when preparing the lessons, conducting them, and reflecting on them.
Disciplined Inquiry as a Tool for Teacher Professional Growth
If we agree that teachers benefit from developing knowledge, competences, and attitudes that support inquiry, then the next question is how teachers can be supported in developing this knowledge, competences, and attitudes. As noted in the previous section, one of the recommendations made in the research literature, which is already implemented in some professional development programs (Watson and Mason 2007), is to provide teachers with extended exposure to and engagement with different forms of inquiry in contexts that simulate teaching activities. Furthermore, we argue that teacher learning in this area can be intensified through involvement in disciplined inquiry.
If teachers are to learn about their profession in the active ways that are now known to be empowering and recommended in curriculum documents, then they will need opportunities to explore, investigate, create, reflect and solve problems and answer research questions for themselves. (Crawford and Adler 1996, p. 1201)
In discussing tensions between the worlds of the teacher and the researcher, Labaree (2003) argues that teachers are often reluctant to embrace the analytical practices of educational scholarship because they are used to inquire in order to find practical solutions rather than explanations. Developing competencies that support observing and formulating explanations for pedagogical uncertainties is important not only for analytical and theoretical purposes but also for noticing in the classroom, and for reflecting on and learning from experiences. Thus, the tension between the researchers’ analytical mode of inquiry and teachers’ normative mode of inquiry can provide valuable learning opportunities for the teacher as well as for the researcher.
Another tension Labaree discusses is between the personal and particular to the intellectual and universal. Disciplined inquiries are conducted and communicated so that the data, argument, and reasoning are capable of withstanding careful scrutiny by other members of the scientific community (Shulman 1981). Teacher inquiries are often intended to inform only the inquirer, and most of the knowledge teachers develop through teacher inquiries is not documented or shared with the larger community. Exposure and engagement with disciplined inquiries can help teachers become more conscious and reflective about their inquiry processes and their results. There is evidence that participation in the scientific community’s discourse could lead teachers to integrate in their own discourse new constructs about teaching and learning, so that they can make explicit things they already “felt” but did not have the language to express (Even 1999). Having a richer and more refined discourse to share their experiences with peers is likely to intensify teachers’ interest in participating in such discourse, and hence their ability to learn through their teaching experiences (Leikin and Zazkis 2010).
There is accumulating evidence that teacher engagement in education research as a disciplined inquiry can have a substantial impact on teachers’ confidence in their practice, their sense of agency, their views about teaching and learning, and their teaching practice (Taylor 2017). The impact of exposure or active engagement with education research on teachers’ competences of inquiry is less documented. Moreover, many of the studies and reports in this area address teacher participation in education research in the context of an academic degree. However, there are some alternative and less-demanding options for teacher exposure to and engagement with disciplined educational inquiry. In the next section, we review different modes of partnerships between teachers and researchers and explore the opportunities for learning and professional growth they offer.
Modes of Research-Practice Partnership and Development of Research Competencies
In the previous section, a case has been made that mathematics teachers may benefit professionally from developing education research competences, even if they have no aspirations of becoming education researchers. It has also been argued that the development of such competences in teachers can be intensified through their interactions with people who do possess these competencies, that is, with mathematics education researchers. Different forms of teacher-researcher interactions, however, vary in the opportunities they generate for exposing teachers to, and engaging teachers with, the processes of educational research.
All educational research in schools involves cooperation of one form or another between researchers and practitioners. (Wagner 1997, p. 13)
Based on reflective analysis of many R&D educational projects, Wagner (1997) extracted three forms of research-practice cooperation: data-extraction agreements, clinical partnership, and co-learning agreements. These three forms have their own pragmatic, moral, and political expectations and implications for the involved parties. Wagner (1997) compares these forms of agreements in terms of focal research questions, division of labor in a typical research process, agencies and voices in reporting the projects’ outcomes, and the implied models of educational change.
In data-extraction agreements, the researchers have the full agency over the research-practice cooperation. The teacher-researcher interactions in this kind of agreements revolve around questions about the nature of education and schooling that researchers suggest and investigate. Namely, the researchers formulate research questions and ways of exploring them and are involved in the process of inquiry and reflection, whereas the teachers are engaged in action under exploration. The researchers are in charge of the constructing of the new knowledge and reporting it. The knowledge is reported in the researchers’ voices and in ways suitable for other researchers and for high-level administrators. These administrators are expected to act as mediators between the researchers and the teachers by “translating” the constructed knowledge into decisions and guidelines affecting future teaching practice. Teachers in this form of cooperation have little say or even knowledge about the research processes. The researchers often restrain from providing constructive feedback to the teachers in order to avoid “contamination” of data. Thus, data-extraction agreements generate little opportunities for the teachers to confront and refine their existing knowledge and discourse about inquiry. Moreover, data-extraction agreements are likely to intensify the tensions and lack of trust between practicing mathematics teachers and mathematics education researchers, which has been pointed out by several scholars (Kontorovich and Rouleau 2018; Krainer 2014; Labaree 2003; Nardi 2015).
Investigations conducted under clinical partnership agreements also focus on the nature of education and schooling. However, the formulation of research questions and methods in these agreements is oriented towards a joint work of researchers and teachers striving to improve together the understanding of mathematics teaching and learning in schools. A paradigmatic example of this form of agreement is teaching experimentation, as described in a seminal paper by Cobb (2000); examples of specific projects can be found, for instance, in Kieran et al. (2013). In clinical partnership agreements, the researchers are still the agents of the inquiry, but teachers are involved not only in the action under exploration but in the inquiry itself, at least by assisting their research colleagues. Attention is given to the process of researcher-teacher consultations, and both parties are involved in reflection. Research reports are produced by researchers, but effort is made to give room to the voices of the teachers in the reports. It is intended that the reports would be useful not only to other researchers and high-level administrators but also to educational practitioners.
This type of agreement can involve several opportunities for teachers to develop education research competencies. Through problematizing routine pedagogical situations, teachers gain opportunities to identify implicit patterns of teaching and learning in their classroom and label them. Translating with researchers these problems of practice into researchable questions and testable hypotheses can help teachers learn how to shift from the space of personal-particular inquiries to the space of universal-intellectual inquiries. Gaining knowledge about ways of documentation and interpretation of teaching/learning events could lead teachers to be more aware and reflective of their own inquiry processes. In particular, considering with researchers multiple interpretations to the data can help teachers develop appreciation to the roles of noticing and reflection, and to the variability of data interpretations.
Co-learning agreements share many features of clinical partnership, but differ from it in a subtle yet essential way. Whereas in the clinical partnership the practitioner is invited to the world of the researcher to inquire the practitioner’s world according to the researchers’ rules of inquiry, in co-learning partnerships, researchers and practitioners join forces to inquire together and aid one another in order to learn something new and worthwhile about their worlds and themselves. In co-learning partnerships, the goals, methods, and principles of inquiry are negotiated openly to maximize the learning and the fruits of this learning for both sides. Therefore, co-learning agreements essentially reduce asymmetry in the roles of the researchers and practitioners that characterize data-extraction and clinical partnership modes of cooperation. In addition to research questions of the two previous forms of agreements, research questions pursued within co-learning agreements concern the nature of educational research itself as well as characteristics of the home institutions of the involved parties.
For example, when a school collaborates with a university department, a study designed in a co-learning mode may ask: What characteristics of the school and what characteristics of the university department are beneficial (or otherwise counterproductive) for making the desired educational change? The use of which research design may increase chances for both parties to develop mutually fruitful partnership?
Inquiry processes in a co-learning agreement may occur simultaneously at several levels, to allow the same systematic data collection procedures as in the data-extracting and clinical partnership agreements, and at the same time, it may facilitate some reflective inquiry of the research process and the development of the involved institutions. Co-learning agreements value specialized types of expertise developed by researchers and practitioners in their home institutions, but also value special skills required to cross between institutional boundaries.
Examples of this type of researcher-practitioner agreements can be found within an emerging research paradigm known as DBIR (for Design-Based Implementation Research). According to Fishman et al. 2013), DBIR focuses on problems of practice from multiple stakeholders’ perspectives, concerns with developing capacity for sustaining change in educational systems, and calls for breaking down barriers that isolate those who design and study educational innovations and those who implement them. An additional example comes from Schoenfeld (2009) who argues for rapprochement between the field of educational research and the field of educational design.
Co-learning agreements can be immensely productive for the development of education research competencies in mathematics teachers, and to their professional growth. Whereas clinical partnerships generate mostly opportunities for teachers to learn about inquiry as outsiders (in the sense of Jaworski 2003), co-learning partnerships accommodate them as insiders. Rather than accepting a ready-made research agenda, teachers work with researchers on translating both sides’ goals into a mutually beneficial agenda. This process of establishing a co-learning agreement provides opportunities for both sides to learn about the other side’s needs and cultural orientations. In particular, through this process, teachers can develop some understanding of the researchers’ world, including their objectives and methods. Teachers are invited to work side by side with the researchers in the developing, validating, and implementing of data collection tools, and practice collective reflection and data analysis. A particularly important stage in co-learning inquiry is formulating conclusions and implications. Here, teachers can bring forth an established sense of what works in education and what makes education work, a feel for the breadth, depth, and complexity of education as an institution that cannot be picked up by reading about it or observing it (Labaree 2003). The exchanges between researchers and teachers about the conclusions and implications of the joint inquiry provide valuable opportunities for both sides to revisit their assertions and discourse about mathematics teaching and learning.
This view of co-learning agreements is in accordance with a theoretical framework for conceptualizing mathematics education research proposed by Jaworski (2003). This framework encourages thinking about any mathematics education research, which has the teacher professional growth as one of its goals, in terms of four sets of questions: (1) questions about knowledge and learning (e.g., “who are the people learning and whose learning is studied?” (Jaworski 2003, p. 263); (2) questions about inquiry and reflection (e.g., “Who is inquiring? What forms does reflection take?” ibid, p. 264); (3) questions about insiders and outsiders (e.g., “who is conducting research or inquiry and into whose practices?” ibid); and (4) questions about individual and community (e.g., “In what community did inquiry take place? What is the nature of the community?” ibid). Note that these questions imply diverse answers that can embrace Wagner’s three modes of research-practice partnership.
Our final note in this section is that neither the three types of the agreements offered by Wagner nor Jaworski’s framework necessarily imply that teachers would become proficient in educational theories underlying mathematics education research while collaborating with researchers. Accordingly, the mentioned modes of research-practice cooperation do not necessarily lead to the development of mathematics teachers as independent education researchers but rather to their development as proficient teacher inquirers.
Citizen Science as a Source of Inspiration for Devising Research-Practice Partnerships
[a]ll form of cooperative educational research have the potential to alter the social life of individuals and institutions. (Wagner 1997, p. 20)
Citizen Science: An Introduction
Constructed 20 years ago, Wagner’s (1997) typology of partnership agreements stemmed from and operated with examples of educational projects that involved parties having clear structures and boundaries, such as a partnership between a university department and a school or a partnership that develops within an academic course having a clinical component. However, Wagner’s vision of collaborative educational research as an enterprise that can alter social lives of the individuals and institutions involved, and as a social interaction having its own rights beyond problems that it attempts to resolve, embraces additional types of projects, which have been developed only recently. In particular, the Wagner vision fits well a rapidly developing type of research known as Citizen Science (CS).
Bonney et al. (2009) refer to CS as a form of conducting scientific research that involves members of the public in association with scientists to collectively gather, categorize, or analyze large quantities of data in order to address real-world problems. From the social viewpoint, CS is an open movement that includes hundreds of scientific projects, thousands of scientists, and millions of volunteering members of the public all over the globe (Wiggins and Crowston 2011).
CS flourishes nowadays in many fields, including medicine (e.g., PatientsLikeMe, see www.patientslikeme.com), astronomy (e.g., GalaxyZoo, see www.galaxyzoo.org), environmental sciences (e.g., OPAL—Open Air Laboratories, see www.opalexplorenature.org), biology (e.g., eBird, see http://ebird.org/content/ebird), and more. Many CS projects are considered successful in that they achieve a large-scale response from the public and produce high-quality scientific results. The success of CS projects seems to stem from an ingenious integration of scientific, social, educational, and technological components. In this section, we discuss selected insights from the literature on CS as a source of ideas for devising research-practice partnerships in mathematics education that have the development of research competencies of teachers as one of their goals.
Different scholars map the CS landscape by focusing on different aspects of CS projects. For instance, Bonney et al. (2009) characterize CS projects by levels of participation of the public. They distinguish contributory, collaborative, and co-created projects. These three levels of participation (roughly) correspond to Wagner’s three modes of research-practice partnerships. Of note is that there exist highly successful, in the meaning specified above, projects at each of the above levels of participation, and that some of the projects allow the public to be engaged at different levels. We learn from Bonney et al. (2009) that (1) not only co-created projects (cf. co-learning partnerships) can be of interest to the public and (2) an option to engage different participants in the same study at different levels of participation should be thought through in the future mathematics education research projects.
Wiggins and Crowston (2011) proposed a typology of CS projects based on their goals and types of activities. Two out of five types of CS projects seem to be particularly relevant to mathematics education research: action-oriented projects and investigation projects. Briefly, action-oriented CS projects encourage participants’ involvement in local concerns, using action research as a tool to support these activities. Such projects are usually initiated by members of local communities and engage scientists as consultants. In the context of mathematics education, action-oriented CS projects can be associated with action research collaboratively conducted by members of a school or a district community of mathematics teachers.
Investigation projects specialize in scientific research that relies on massive data collection, by volunteers, from physical environments. As a rule, such projects are initiated by scientists and revolve around ideas that well resonate with the interests of large groups of the public. The public can be involved not only in data collection but also in data analysis and training. Educating the public is a secondary and frequently unarticulated goal of projects of this type. The main motivational factor for the public to participate is the involvement in a science area of interest and the wish to be a part of a community uniting individuals with similar interests.
An Example of a Successful CS Investigation Project: eBird
As a prelude to the sections devoted to the provisional mathematics education projects, let us briefly describe eBird, probably the most famous CS project of the investigation type.1 The eBird goal is to collect and distribute information on bird abundance and distribution in diverse spatial and temporal measures. The project has been launched in 2002 by the Cornell Lab of Ornithology and National Audubon Society, and is to date the largest biodiversity data resource, with over one million bird observations reported every month from tens of thousands of participants, and with more than 100,000 people entering the project every year. Data collection is done by participants from many geographic locations and having a broad range of professional backgrounds, including amateur and expert bird observers.
All data are submitted using an online checklist program. The participants simply enter when, where, and how they went birding, then fill out a checklist of all the birds seen and heard during that time. Before an individual record of observation enters the database, it is validated with an automated data quality filters developed by regional bird experts. Local experts review unusual records that are flagged by the filters. The data are then classified and grouped according to species, location, and time of observation. The project has an active website, Facebook page, and Twitter feed that contain recent reports about the project progress and interesting findings. All birding information is available on the project website and can be accessed in 11 different languages, including English, Spanish, French, and Portuguese. The aggregated data are presented in tables, in bar charts, in line graphs, and on a map. The data are available for download and further use and analysis. The project is remarkably productive scientifically: more than 140 published journal and conference papers are currently listed on the eBird website.
It is our dream to launch mathematics education research-practice partnerships, which architecture and structure would be in line with the eBird architecture and structure. Therefore, it is in place to inquire what makes eBird so successful. Golombic (2015) suggested that eBird was launched based on an assumption that there is an unconsolidated amateur birdwatcher community, which might benefit from providing services that would consolidate it. The online tool built by the eBird team was instrumental to this end, and in addition it provided an infrastructure for conducting scientific research and supporting knowledge construction by the community members. So, the stances “capitalize on what the public needs” and “provide educational opportunities within authentic scientific activity” can be seen as contributors to the eBird success. In-built hierarchy of levels of participation in the project can be seen as an additional contributor. The eBird community is formed so that most of the volunteers take part in the project as data contributors, but some of them act also as reviewers of the submitted data, some are involved in the data classification, and some in preparing the data summaries and research reports. Therefore, a volunteering eBird participant can build a sort of an informal career within the project.
A Provisional Research-Practice Partnership in Mathematics Education
We hypothesized that the factors that influence whether a particular problem would or would not be a challenge for a mathematics learner extend beyond the learner’s background, and include also mathematical characteristics of the problem, situational factors, emotional factors, and more (cf. Koichu 2017, for a detailed discussion). Accordingly, we looked for an opportunity to explore the following query: under which circumstances mathematical problems of certain characteristics are perceived by learners having different mathematical backgrounds as challenging ones?
For the purpose of the Study, we will regard challenge as a question posed deliberately to entice [italics is added] its recipients to attempt its resolution while at the same time stretching their understanding and knowledge of some topic. Whether the question is a challenge depends on the background of the recipient; what may be a genuine puzzle for one person may be a mundane exercise or a matter of recall for another with more experience. (Barbeau 2009, p. 5).
Our working assumption was that this query could be addressed along with the collection of large-size data corpus constructed in collaboration with many mathematics teachers. Furthermore, we assumed that some mathematics teachers could be interested in questions related to the above query, such as the following: How to find or design problems that could become a challenge for students with different characteristics? How to lunch challenging problems in a heterogeneous class in a way that the students will become “enticed”?
A brief explanation of the proposed project with an invitation to continue reading a friendly overview of the literature on mathematical challenge and its role in teaching and learning mathematics
A collection of mathematical problems which are considered, by the participants of the project, as “enticing” for particular categories of learners. The description of each problem includes the formulations of the problem, its source, one or more hints to its solution, one or more full solutions, and a brief explanation of why the problem was chosen as a favorite. In addition, every problem is automatically categorized according to various user rankings (e.g., popularity, difficulty, challenge, teaching time) and pre-determined keywords that allow users to sort and select problems (e.g., “geometry, 8th grade, regular lesson, textbook problem” or “algebra, 11th grade, enrichment class, the first source is unknown”).
A statement of the goals of the project and ways of participation
Submit their own favorite problems in the format described above
Rank the problems presented on the website by different criteria
Comment on the problems and take part in online discussion forums
Assist other participants in the process of submitting problems
Review submitted problems prior to making them visible on the website
Suggest categories for classifying problems and classify them accordingly
Prepare aggregated summaries by problems, by types of problems, by types of situations in which the problems were used, and by additional parameters
Suggest new queries and activities for the project community
We intend that some of the above activities would be facilitated by face-to-face professional development workshops for local communities of mathematics teachers, and some activities would be undertaken purely online.
As this project is still at early stages of development, we can only speculate about the kinds of impact that this project can have. Assume that during the first 6 months of the project operation, about 500 teachers have visited the site, 200 teachers registered and took part in ranking problems, 100 teachers uploaded several of their favorite problems, and 20 teachers became advanced participants. Under these assumptions, it is not unrealistic to suggest that the site would contain a considerable collection of “favorite” problems, and that some of them would be actively ranked and discussed by the participants. We, the initiators of the project, would benefit from a rich data set in relation to our original research question (to recall, the question was “what constitutes a challenging problem for different categories of learners?”). The teacher-mentors would gain valuable information about MOFET teachers’ preferences and needs. All the participating teachers would benefit from having a useful online resource to work with. Furthermore, we expect that some of the participating teachers would offer their own queries to be explored based on the emerging collection of “favorite” problems, or try in their classes problems proposed by the other participants and share their experiences. This activity might initiate additional research directions, for example, of design experimentation nature. Overall, the “My favorite mathematical problems” community may develop so that the initial research question would gradually be addressed, but it can also develop in directions unforeseen, in accordance with the emerging needs and interests of the different participants.
As argued above, active participation in such a project can foster the development of research competencies in its participants. But even when thinking of the most optimistic scenario, we are far from suggesting that the “My favorite mathematics problem” project would lead to the development of education research competencies in all participating teachers. Indeed, participation in the project would be of the data acquisition type for many of them, and only some of the participants would choose to be engaged in activities complying with the clinical partnership and co-learning types of opportunities offered by the project. We therefore suggest that, though all the participants would benefit in some respects (e.g., they would increase their awareness of the importance of challenging their students, get familiar with a rich collection of problems, and take part in a vibrant community of interest), only a few would be engaged in research-related practices, such as classifying problems, analyzing clusters of problems, documenting and sharing classroom events stipulated by the use of some of the problems, preparing aggregated summaries, and formulating questions for further inquiries.
Further, we deem that it would be unrealistic to expect participation at the eBird scope, even in the most optimistic scenario and even on the long run. Indeed, we think about hundreds rather than hundreds of thousands of prospective participants and would be glad to be proved as overcareful.3 This is because eBird capitalizes on the volunteer involvement of individuals who benefit from the infrastructure supporting their hobby, and “My favorite mathematics problems” project capitalizes on curiosity related to professional occupation of its participants. We believe that a system of rewards (e.g., obtaining credit for active online participation, as it is the case for participation in face-to-face professional development workshops) should be considered. This is in addition to the main reward, which is becoming part of a community that suggests interesting activities and enables its participants to develop professionally and build self-identity as contributors to research.
The TRAIL Framework
The above sections provide the background for the main part of this paper, which is the presentation of TRAIL4—a developing theoretical-organizational framework for supporting, enhancing, and scaling up partnerships between education researchers and teachers. In what follows, we present TRAIL as a system consisting of (1) premises, (2) a set of heuristics for guiding the design and conduct of research projects, and (3) a set of provisional examples.
Researchers who collaborate with teachers accept what can be called realistic constraints as they explore what might be possible in students’ mathematics education. (Cobb 2000, p. 330)
Professional Growth Through Involvement in Research premise: Active involvement in the various stages of educational research generates opportunities for teachers to enhance their education research competencies, which in turn could support teachers’ abilities to engage effectively in inquiry, noticing, and reflection as part of the day-to-day practice.
Authenticity premise: Teachers’ engagement in research is more likely to produce positive effects if conducted in the context of an authentic educational research rather than an exercise in doing research. Accordingly, it is advantageous for teachers and researchers to take part in research that is drawn by questions of potential importance to both communities, which is conducted so that it can lead to scientifically sound results and implementable insights.
Shared Agency premise: Alliance of the communities of teachers and education researchers can be stable and productive if the opportunity to share the agency over the partnership is available for both communities. This means that individual members of each community are to be involved in the partnership in ways that can advance their peculiar goals and needs, including the needs to contribute, to develop professionally, and have room for expressing personal creativity.
Choice premise: Teacher participation in educational research can be stable and productive if the teachers can choose in which research projects to take part, in which capacity, and to which extent. This premise implies that TRAIL should consist of a network of research projects that run simultaneously and allow different modes of participation. Additionally, it presumes that TRAIL will enable teacher-participants and researcher-participants to initiate new projects and find partners for their development.
Duality of Creating and Using New Knowledge premise: The more practitioners are actively involved in co-creating new knowledge within TRAIL projects, the more chances there are that this knowledge would be used in practice and impact the field. And vice versa, the more influence TRAIL projects have on teaching practice, the more opportunities for professional growth of its participants (both teacher-participants and researcher-participants are meant) emerge within these projects.
Design Heuristics for TRIAL Studies
It is intended that TRAIL will embrace various studies focusing on mathematics learning, teaching practices, and issues related to implementation of novel pedagogical ideas. The TRAIL projects can be of the action type (i.e., projects initiated by local communities of mathematics teachers), of the investigation type (i.e., projects relying on multi-level participation of large groups of teachers having similar interests), and of the mixed type (e.g., projects that combine characteristics of the previous two types at different stages of their development).
However, not any mathematics education research can be accommodated in TRAIL. Stimulated by the reviewed literature on types of teacher inquiry, Wagner’s (1997) categories for classifying research-practice agreements, Jaworski’s (2003) framework for re-thinking mathematics education research, and our (still limited) experience, we propose several design heuristics that can guide the development of TRAIL projects. Needless to say, the list of heuristics is tentative and, most likely, will be refined with the experience accumulation.
About Appropriate Research Goals and Questions
The research goals and question that underlie TRAIL partnerships deal with issues that have the potential to resonate with dilemmas and challenges that mathematics teachers encounter in their daily work at the level of a class, a small group, or an individual student.
TRAIL partnership must have “clear utility” for practitioners that can be convincingly communicated without heavily relying on the scientific literature in which the research is situated. In a similar vein, a TRAIL partnership must have “clear utility” for education researchers, that is, have the potential to yield insights of importance to the mathematics education research community.
TRAIL partnerships develop around research questions that can be answered gradually and at different levels of comprehensiveness. That is, an initial answer to the question can be obtained based on small-scope data corpus collected by a small group of the participants, but involvement of the larger groups of participants and creating the larger data corpus would lead to revising and refining the initial answer towards the production of a more comprehensive and reliable answer.
About the Conduct of TRAIL Studies
TRAIL partnerships enable teachers to be involved as research assistants or researchers, but not as objects of research. However, both teacher-participants and researcher-participants can be objects of a study about TRAIL.
TRAIL partnerships employ accessible data collection and data analysis procedures. We call a research procedure accessible if it can be mastered by an interested individual with no background in research procedures after a brief training period, and if its use requires reasonable time and effort. Examples include conducting a questionnaire in a classroom, writing a reflective summary of a lesson, responding to a summary by another participant, and using a brief structured protocol for coding a video clip of a lesson.
TRAIL partnerships offer a technological platform for supporting the development of a community of inquiry (Jaworski 2003) that comprises of the project’s participants. Participants at different levels of involvement have different roles in this community, from making occasional contributions to the data corpus of the study to active involvement in data analysis and creating the data summaries and writing reports. Furthermore, TRAIL partnerships have transparent rules for movement from one level of participation to another.
TRAIL partnerships offer channels of interaction among the participants as well as channels for providing both quick and delayed feedback on contributions of the participants. For example, a teacher who contributes a mathematical problem to the database of “My favorite mathematical problems” project will obtain structured feedback on his or her problem from the other participants.
TRAIL partnerships have mechanisms for earning credit (e.g., in terms of professional development hours) for active participation at different scopes and levels.
TRAIL partnerships comply with the existing ethics codes for conducting educational research. In particular, the shared databased of a TRAIL partnership should consist only of properly anonymized data and be used only for educational and research purposes. We anticipate however that TRAIL would require creation of a special ethics code that would attend to its specificity.
“What counts in my class for/as …?” The abovementioned “My favorite mathematical problem” project belongs to this category. Additional examples of partnerships can be developed around such questions as the following ones: “what counts in my classroom as an explanatory proof?” or “what counts for being creative?” This type of questions is associated with the flourishing realm of research questions on social and sociomathematical norms (Yakel and Cobb 1996). Of note is that a typical study on social and sociomathematical norms is based on analysis of classroom observations. The data for addressing this type of questions in a TRIAL study can come from teacher-produced accounts of their lessons, for example, a brief account of an episode, in which a student exhibited, in the eyes of his or her teacher, a creative behavior, or a mathematical justification of an assertion that was tagged in a classroom as an explanatory proof.
“Does a particular phenomenon, described in the literature, occur also in my class?” For example, a group of teachers, who are familiar from their teacher education or professional development course with the paper by Fischbein and Kedem (1982), can inquire: “Is it possible that my students need to see proofs to special cases in order to become convinced that a general theorem is true, even after studying its general proof?” This question can be addressed by several accessible research means. Some teachers may wish to replicate the original Fischbein and Kedem’s questionnaire in their class, whereas other teachers can choose to share accounts of appropriate episodes from their lessons. An additional question of this type might be: “Will my best students also fail to explain the rationale behind the standard procedure for division of fractions?” The question is stimulated by a phenomenon first investigated by Ma (1999) and re-analyzed by Koichu et al. (2013). In both studies, the data collection procedures were quite simple and could be implemented by interested teachers in their classes with proper support.
“What do my students do if/when?” For example, “What types of questions do my students ask me when I explain new material, or when I engage them in problem solving'?” This question is stimulated by the literature that connects the quality of instruction with the quality of students’ questions (e.g., Leikin et al. 2017). Exploring the modes of inquiry that students enact through different types of questions can provide teachers with valuable insights into student learning and into strengths and weaknesses of their own teaching. An added value can be gained when a teacher-participant is exposed to the findings produced by the other participants of the project. Additional TRAIL partnerships can be developed around the following questions: “What do my students do when they don’t know immediately how to solve a given problem?” “What types of events are memorable for the students in my lessons?” Each of these questions is supported by considerable body of past research but has room for further research of importance to both the community of mathematics teachers and of mathematics education researchers.
“How does a particular pedagogical idea work in my classroom?” This type of studies can facilitate implementation and testing of innovative ideas for teaching. For example, the use of social networks as a complementary teaching tool becomes widespread nowadays (Stahl and Rosé 2011), but empirical database on affordances, limitations, and effects of online forums on classroom problem-solving practices is still limited. This observation can give rise to the birth of a TRAIL partnership for inquiring: “How do online forums for collaborative problem-solving can complement classroom teaching?” “What types of problems deserve considerable student response in online forums?” These questions could be researched on a large scale by engaging teachers in collecting and sharing their experiences of supporting students’ online mathematics activities.
Summary and Concluding Remarks
Mathematics teachers are major stakeholders in mathematics education, yet they are often regarded as means for producing scientific knowledge about education, or as passive recipients of the scientific knowledge produced by education researchers (Krainer 2014). Several scholars asserted that research on education has neither the role nor the capacity to influence directly mathematics teaching on a large scale (Kieran et al. 2013). The nature of education as a discipline and as a practice implies that it is unlikely that findings of educational research will ever impact teaching practice the way, say, medical research can impact medical practice (Schoenfeld 2000).
The more researchers regard teachers as stakeholders in the production of knowledge about education, the more researchers’ and teachers’ knowledge and theory bases will overlap.
Good collaboration and mutual trust between researchers and teachers support the professional growth of both parties.
Researchers are better fit than teachers are to assume the responsibility of taking serious steps to promote the negotiation process between the two parties, based on each party’s strengths, and as a two-way street.
We believe that the presented in this paper Teacher-Researcher Alliance for Investigating Learning can be such a “serious step” towards enhancing the collaboration and trust between researchers and teachers, towards greater and more meaningful involvement of teachers in the production of knowledge about education, and towards realizing the potential of mathematics teachers’ involvement in disciplined inquiry for their professional growth. We see TRAIL as a theoretical-organizational framework that can support the much-desired process of building bridges between researchers and practitioners, and bringing research and practice closer. In addition, we suggest that TRAIL can serve as a theoretical means for designing mutually beneficial research-practice partnerships, in which teachers would have diverse opportunities for developing their education research competence. As Goldin (1998) wrote, “human competence refers to the ability to perform a task some of the time, under conditions which are partially but incompletely specified” (p. 147). To this end, TRAIL suggests challenging tasks to pursue and specifies some of the conditions for doing so, but, of course, can lead to successful outcomes only “some of the time.”
In closing, it is important to us to be precise when acknowledging the teachers’ side in TRAIL. After all, calling for greater teacher involvement in educational research can be seen as asking teachers to get closer to the researchers’ world, rather than bringing the teacher and the researcher worlds closer to each other. As it is hopefully clear from the TRAIL premises, design heuristics and examples, not the former but the latter stance is our position. Clearly, the proposed teacher-researcher alliance needs further elaboration and development. It is our hope for this article that it would encourage its readers to bring their own premises, heuristics, and examples into the continuous discussion of the issue of developing education research competences in mathematics teachers, as part of the greater discussion of mathematics teachers’ professional growth.
This section consists of an abridged and modified version of the text that appears in Golombic’s (2015) technical report.
MOFET association in Israel supports teaching and learning advanced level mathematics in secondary school. MOFET is an acronym in Hebrew for “Mathematics and Physics and Culture.” MOFET classes study mathematics for 7–8 h per week since the seventh grade, in accordance with specially designed curriculum that combines acceleration and enrichment. See http://www.reshetmofet.org/en/english-english-home for details.
Our belief that we are not overoptimistic when talking about hundreds of participants is based on the fact that some of the existing projects aimed at supporting mathematics teachers’ reflection and inquiry have achieved the intended scope (e.g., VIDEO-LM project in Israel, see Karsenty and Arcavi 2017; Teacher Researchers’ Network in Thailand, see Wareerat et al. 2016).
The Hebrew name of the proposed framework is MaSHaL—an acronym for Morim SHutafim Lemehkar (Teachers Participate in Research). This name involves a play of words: MaSHaL in Hebrew is an equivalent of the Latin QED (Quod Erat Demonstrandum) and also means “fable.”
- Ball, D. L., & Cohen, D. (1999). Developing practice, developing practitioners. In L. Darling-Hammond & G. Sykes (Eds.), Teaching as the learning profession: Handbook of policy and practice. San Francisco, CA: Jossey Bass.Google Scholar
- Barbeau, E. J. (2009). Introduction. In Barbeau, E. J., & Taylor, P. J. (Eds.). (2009), Challenging mathematics in and beyond the classroom, Study Volume of ICMI Study 16 (pp. 1–9). New York, NY: Springer.Google Scholar
- Barbeau, E. J., & Taylor, P. J. (Eds.). (2009). Challenging mathematics in and beyond the classroom: the 16th ICMI study (Vol. 12). Springer Science & Business Media.Google Scholar
- Boaler, J., Ball, D. L., & Even, R. (2003). Preparing mathematics education researchers for disciplined inquiry: Learning from, in, and for practice. In Second international handbook of mathematics education (pp. 491–521). Springer Netherlands.Google Scholar
- Bonney, R., Ballard, H., Jordan, R., McCallie, E., Phillips, T., Shirk, J., & Wilderman, C. C. (2009). Public Participation in Scientific Research: Defining the Field and Assessing Its Potential for Informal Science Education. A CAISE Inquiry Group Report. Washington, D.C.: Center for Advancement of Informal Science Education (CAISE).Google Scholar
- Cobb, P. (2000). Conducting teaching experiments in collaboration with teachers. In A. Kelly, R. Lesh (Eds.), Handbook of Research Design in Mathematics and Science Education (pp. 307–333). Mahwah, NJ: Lawrence Erlbaum.Google Scholar
- Crawford, K., & Adler, J. (1996). Teachers as researchers in mathematics education. In A. Bishop, K. Clements, C. Keitel, J. Kilpatrick, & C. Laborde (Eds.), International Handbook of Mathematics Education (pp. 1187–1205). Dordrecht, The Netherlands: Kluwer.Google Scholar
- Cronbach, L. J., & Suppes, P. (Eds.). (1969). Research for tomorrow's schools: Disciplined inquiry for education. New York: Macmillan.Google Scholar
- Day, C. (1999). Developing teachers: The challenges of lifelong learning. Psychology Press.Google Scholar
- Ellis, J. L. (1998). Introduction. The teacher as interpretive inquirer. In Ellis, J. L. (Ed.), Teaching from understanding: Teacher as interpretive inquirer (pp. 5–14). Taylor & Francis.Google Scholar
- Even, R. (1999). Integrating academic and practical knowledge in a teacher leaders’ development program. In Forms of Mathematical Knowledge (pp. 235–252). Springer Netherlands.Google Scholar
- Even, R. (2003). What can teachers learn from research in mathematics education? For the Learning of Mathematics, 23(3), 38–42.Google Scholar
- Fischbein, E., & Kedem, I. (1982), Proof and certitude in the development of mathematical thinking, in A. Vermandel (Ed.), Proceedings of the Sixth International Conference for the Psychology of Mathematics Education (pp. 128–131). Antwerp, Belgium.Google Scholar
- Fishman, B. J., Penuel, W. R., Allen, A. R., Cheng, B. H., & Sabelli, N. O. (2013). Design-based implementation research: An emerging model for transforming the relationship of research and practice. National Society for the Study of Education, 112(2), 136–156.Google Scholar
- Golombic, Y. (2015). What makes Citizen Science projects successful, and what can we learn from them for future projects? Unpublished report on the literature review written on request of the steering committee of the Technion Citizen Science Project under the supervision of Boris Koichu.Google Scholar
- Kieran, C., Krainer, K., & Shaughnessy, J. M. (2013). Linking research to practice: Teachers as key stakeholders in mathematics education research. In M.A. Clements, A. Bishop, C. Keitel, J. Kilpatrick, & F. Leung (Eds.) Third International Handbook of Mathematics Education, Volume B (pp. 361–392). Dordrecht, the Netherland: Springer.Google Scholar
- Kilpatrick, J. (1981). The reasonable ineffectiveness of research in mathematics education. For the Learning of Mathematics, 2(2), 22–29.Google Scholar
- Koichu, B. (2017). On mathematics with distinction, a learner-centered conceptualization of challenge and choice-based pedagogies. The Mathematics Enthusiast, 14(1–3), 517–540.Google Scholar
- Koichu, B., & Keller, N. (2018). Creating and sustaining online problem-solving forums: Two perspectives. In P. Liljedahl & L. M. Santos Trigo (Eds.), Mathematical Problem Solving: ICME 13 Monograph (tentative title). Springer. (in press).Google Scholar
- Koichu, B., Harel. G., & Manaster, A. (2013). Ways of thinking associated with mathematics teachers’ problem posing in the context of division of fractions. Instructional Science, 41(4), 681–698.Google Scholar
- Kontorovich, I., & Rouleau, A. (2018). To teach or not to teach? Teacher-researchers cope with misconceptions in interview settings. Canadian Journal of Science, Mathematics and Technology Education. https://doi.org/10.1080/14926156.2016.1243278.
- Krainer, K. (2014). Teachers as stakeholders in mathematics education research. The Mathematics Enthusiast, 11(1), 49.Google Scholar
- Leikin, R., Koichu, B., Berman, A., & Dinur, S. (2017). Does general giftedness play a role in classes of students motivated to study mathematics at a high level? Focus on students' questions. ZDM Mathematics Education, 49, 65-80.Google Scholar
- Leikin R., & Zazkis R. (Eds.) (2010). Learning through Teaching Mathematics. Mathematics Teacher Education, vol 5. Springer, Dordrecht.Google Scholar
- Ma, L. (1999). Knowing and teaching elementary mathematics. Mahwah, NJ: Erlbaum.Google Scholar
- Mason, J. (2002). Researching your own Classroom Practice: From Noticing to Reflection. Routledge Falmer, London.Google Scholar
- Menter, I., Elliot, D., Hulme, M., Lewin, J., & Lowden, K. (2011). A guide to practitioner research in education. Sage publishing.Google Scholar
- Rowland, T., Thwaites, A., & Jared, L. (2011). Triggers of contingency in mathematics teaching. In B. Ubuz (Ed.), Proceedings of the 35th conference of the International Group for the Psychology of Mathematics Education (Vol. 4, pp. 73–80). Ankara: International Group for the Psychology of Mathematics Education.Google Scholar
- Schoenfeld, A. H. (2000). Purposes and methods of research in mathematics education. Notices of the AMS, 47(6), 641–649.Google Scholar
- Schoenfeld, A. H. (2009). Bridging the cultures of educational research and design. Educational Designer: Journal of the International Society for Design and Development in Education, 1(2). Available online at: http://www.educationaldesigner.org/ed/volume1/issue2/article5/index.htm
- Schön, D. (1983). The reflective practitioner: How professionals think in action. London: Temple-Smith.Google Scholar
- Stahl, G., & Rosé, C. P. (2011). Group cognition in online teams. In E. Salas & S. M. Fiore (Eds.), Theories of team cognition: Cross-disciplinary perspectives. New York, NY: Routledge/Taylor & Francis.Google Scholar
- Wareerat, K., Rujroad, K., Skonchai, C., Wanintorn, S., & Sureeporn, S. (2016). The Development of the Teachers’ Researcher Network to Create Instructional Innovation for Raising Students’ Learning Achievement in Science and Mathematics, Thailand. Advanced Science Letters, 22(12), 4514–4518.CrossRefGoogle Scholar
- Wiggins, A., & Crowston, K. (2011). From conservation to crowdsourcing: A typology of citizen science. Proceedings of the 44th Annual Hawaii International Conference on Systems Sciences (pp. 1–10), Koloa, Hawaii. Available online at https://citsci.syr.edu/sites/crowston.syr.edu/files/hicss-44.pdf.